Architecture

How the engine works.

Parametric execution templates, a managed CPU headline path, and incremental GPU offload.

The stack, top to bottom

llmdot pipeline

Your .NET app
     │
     ▼
IChatClient  ────────  Microsoft.Extensions.AI
     │                 (Llmdot.Extensions.AI)
     ▼
LlmModel / ChatSession   (typed, async, IAsyncEnumerable<string>)
     │
     ▼
Execution template       resolved from GGUF metadata →
  ├─ LLaMA-like            TransformerConfig
  ├─ GPT-NeoX-like        (no per-architecture branches)
  ├─ Gemma-like
  └─ LFM2-like
     │
     ▼
Compute backend
  ├─ CPU (managed)         ← default, always present
  ├─ Vulkan  (planned)     ← per-operation offload
  └─ Metal   (planned)
     │
     ▼
GGUF model file           memory-mapped from disk

Metadata in, template out

When you call LlmModel.LoadAsync, llmdot reads the GGUF file's metadata and builds a TransformerConfig — head counts, layer counts, norm placement, activation, rotary settings, and so on. That config selects one of four execution templates. There are no if (arch == "llama") branches in the hot path; the engine is parametric, not architecture-specific.

LLaMA-like

Sequential pre-norm blocks. Covers llama, phi3, qwen2, stablelm, mistral.

GPT-NeoX-like

Parallel residual blocks. Covers gptneox and phi2.

Gemma-like

Embedding scaling plus post-norm. Covers gemma and gemma2.

LFM2-like

Hybrid conv-attention blocks. Covers lfm2 and lfm2_moe.

Managed CPU is the headline, not the fallback

The default compute backend is pure managed .NET. That is a deliberate design choice: the deployment story is the product. A managed core stays trimming-friendly, NativeAOT-friendly, and single-file-publish-friendly, and it runs anywhere .NET runs without shipping per-RID native binaries.

GPU offload is incremental

Optional Vulkan and Metal backends implement an IComputeBackend contract and offload individual operations. There is no all-or-nothing graph rewrite and no conversion step — CPU remains the fallback for any operation a backend does not cover. Both backends are planned; the managed path ships first.

Multimodal plugs onto the backbone

Vision (via a SigLIP2 encoder in the planned Llmdot.Multimodal.Vision package) attaches as a modality encoder on top of the base LLM backbone. The core text runtime is unchanged — multimodal is additive, not a fork of the engine.

Pre-alpha. The specification and template design are stable; the implementation is in active development. See the architecture doc and roadmap for detail.

Read more in Four execution templates and Managed by default, or start with the quickstart.