14. Model families and example applications

The fucina_llm module root (src/llm.zig) exposes each model family as a namespace — llm.qwen3.{model,train}, llm.kimi3.model, llm.qwen35.{model,chat}, llm.gemma.{gemma4,gemma4_train,moe,moe_route,moe_route_tensor}, llm.diffusion_gemma.model, llm.deepseek2.model, llm.glm4moe.model, llm.deepseek4.model, llm.inkling.{model,mmproj,chat}, llm.parakeet.*, llm.speculative.* — while the generic helpers (llm.weights, llm.kv_cache, llm.kv_persist, llm.tokenizer, llm.spm_tokenizer, llm.sampler, llm.logit_processor, llm.llguidance, llm.chat, llm.data, llm.gguf_meta, llm.ptqtp_gguf, llm.cartridge, llm.cartridge_fleet, llm.engram, llm.unicode_categories) stay flat and are covered in §13. This section documents the per-family model APIs, their runner CLIs, and the example applications under examples/. Weight containers (LinearWeight and its quant arms), KvCache, tokenizers, sampling, chat orchestration, and speculative decoding are §13 material; GGUF parsing is §12; LoRA/optimizer/ES mechanics are §11.

14.1 Conventions shared by every family

(src/llm/*/model.zig, src/llm/gguf_meta.zig)

Config from GGUF metadata. Each family's Config.fromGguf(file) (deepseek4/inkling additionally take an allocator; kimi3 is the one non-GGUF family — it loads a reference checkpoint directory, §14.7) reads hyperparameters from the standard GGUF key convention: the value of general.architecture (e.g. "qwen3", "qwen3moe", "qwen35", "gemma4", "diffusion-gemma", "parakeet") prefixes every key — <arch>.block_count, <arch>.embedding_length, <arch>.attention.head_count, <arch>.rope.freq_base, and so on — so one loader covers every size of a family without hardcoding. The vocab size comes from the token_embd.weight tensor shape, not a key. The llm.gguf_meta helpers (metaInt, metaIntOpt, metaFloat, metaFloatOpt) implement the prefixing plus a per-family zero policy: qwen3/qwen35 reject present-but-zero required ints (.reject_zero — every config int is structurally positive), gemma accepts zeros (.accept_zero — keys like attention.shared_kv_layers are legitimately 0). Missing or malformed keys surface as Error.InvalidConfig (parakeet: Error.MissingMetadata; parakeet also departs from the tensor-shape vocab convention — its Config.fromGguf reads parakeet.vocab_size from metadata).

Loader entry points. The qwen3, qwen35, gemma4 and diffusion_gemma families expose the same pair (the deepseek2, glm4moe, deepseek4, and inkling loaders read their Config from the file internally; glm4moe additionally takes max_positions, and deepseek2/glm4moe/deepseek4 take a LoadOptions):

pub fn loadGguf(ctx: *ExecContext, io: std.Io, path: []const u8, config: Config) !Model
pub fn loadGgufFromFile(ctx: *ExecContext, file: *gguf.File, config: Config) !Model
// qwen35 takes `file: *const gguf.File` (it never calls takeMapping);
// a mutable pointer coerces, so caller code is unaffected

loadGguf opens the file itself (mmap via gguf.File.loadMmap for qwen3, gemma4 and diffusion_gemma; qwen35's convenience arm uses the eager gguf.File.load). loadGgufFromFile takes an already-parsed gguf.File so the caller can build a tokenizer from the same file's metadata without a second read — the pattern every runner uses:

var file = try fucina.gguf.File.loadMmap(alloc, io, path);
defer file.deinit();
const config = try llm.qwen3.model.Config.fromGguf(&file);
var model = try llm.qwen3.model.Model.loadGgufFromFile(&ctx, &file, config);
defer model.deinit();
var tok = try llm.tokenizer.Tokenizer.initFromGguf(alloc, &file, .{});
defer tok.deinit();

Weights are materialized through llm.weights.LinearWeight.load (§13), which keeps the GGUF dtype resident — f32/f16/bf16 and the quant forms (Q8_0, Q4_K/Q5_K/Q6_K and the other ggml types) run their own packed kernels; no global dequantization happens at load. Sibling projections that share a dtype and layout are fused at load (weights.fuseLinear): q/k/v into one QKV matrix, gate/up into one gate_up matrix — one wider GEMM per block instead of two or three. Layer loading is parallelized across the work pool (gguf_meta.parallelLoadLayers). Ownership: Model.deinit releases every weight; when expert blocks borrow from the mmap (qwen3 MoE when a single-file GGUF is mmap'd — split GGUFs' experts are copied, and opt-in expert disk streaming (LoadOptions.moe_stream) leaves the mapping with the caller; gemma4/diffusion_gemma under borrow_experts) the model takes the mapping via file.takeMapping() and unmaps it last in deinit. On GPU builds nothing changes at this API level — offload decisions are per-GEMM work-gates inside the shared kernels (§9); the two model-level GPU knobs are gemma-MoE's raw expert representation (14.4) and diffusion_gemma's convertDenseWeightsToF16 (14.5).

Forward/decode surface. The autoregressive families share one contract:

• forwardLastLogits(ctx, token_ids) — cacheless whole-sequence forward; returns the last position's [1, vocab] logits (caller deinits). Empty input is Error.InvalidSequenceLength.
• initKvCache(ctx, capacity) / qwen35's initCache — build the streaming cache sized for capacity positions. This is the duck-typed construction seam the generic llm.chat.Conversation embedder uses (§13).
• forwardStep(ctx, kv, token_ids, pos0) — process token_ids at absolute positions pos0..pos0+len, append their K/V, advance the cache by len, return the last row's logits. Contract: kv.len == pos0 or Error.InvalidSequenceLength; kv.len + len <= kv.capacity or kv_cache.Error.KvCacheOverflow. Prefill is one call on a fresh cache with pos0 == 0 (last-token logits equal forwardLastLogits); decode is a one-token call at pos0 == kv.len.
• forwardStepAllLogits (qwen3, gemma4) — same KV semantics, but returns [len, vocab] logits for every appended position: the speculative-decoding verify entry (§13 — one batched pass scores all draft positions for ~one step's weight traffic).
• forwardStepBatch (qwen3, gemma4) — lockstep multi-stream decode, 14.2.
• generate(ctx, kv, prompt_tokens, out_tokens, options) (qwen3, gemma4; not qwen35) — greedy loop (argmax; GenerateOptions{ .max_new_tokens, .stop_token = null }); resets kv first, returns the count written. diffusion_gemma's block-diffusion generate has its own options and returns a GenerateResult (14.5). Sampled decoding is composed by the callers from llm.sampler (§13).
• forward*Profiled variants take io: std.Io and a family-specific ForwardProfile accumulator (per-block wall-clock buckets; the --profile runner flag).

All forward entries take *const Model and mutate only the ExecContext, the cache, and (profiled) the profile struct; a loaded model is read-only. ExecContext is single-threaded (§6), so concurrent streams over one model need one context and one cache per thread. Returned logits are caller-owned constants (deinit them); no exec scope is required for inference.

14.2 Qwen3 — dense and MoE (src/llm/qwen3/model.zig)

The reference transformer family and the most complete runner: standard GQA attention with per-head q/k RMSNorm and full RoPE, SwiGLU FFN — dense, or a top-k routed mixture (qwen3moe, e.g. 30B-A3B) selected purely by GGUF metadata.

pub const Config = struct {
    vocab_size, hidden_size, intermediate_size, num_layers,
    num_attention_heads, num_key_value_heads, head_dim: usize,
    rms_norm_eps, rope_theta: f32,
    num_experts: usize = 0,          // 0 = dense
    num_experts_used: usize = 0,
    moe_intermediate_size: usize = 0,
    norm_topk_prob: bool = true,
    moe_expert_top_p: f32 = 1.0,     // adaptive expert top-p; 1.0 = full top-k (runtime knob, not GGUF)
    pub fn isMoe(self: Config) bool
    pub fn qwen3_0_6b() Config       // hardcoded 0.6B reference config
    pub fn fromGguf(file: *const gguf.File) !Config
};

fromGguf reads <arch>.{embedding_length, feed_forward_length, block_count, attention.head_count, attention.head_count_kv, attention.key_length, attention.layer_norm_rms_epsilon, rope.freq_base} plus the MoE trio {expert_count, expert_used_count, expert_feed_forward_length}; a model with no expert_count key stays dense. Validation (at load) rejects zero heads, non-divisible GQA grouping, odd head_dim, and inconsistent MoE fields with Error.InvalidConfig.

test "qwen3 reference config" {
    const cfg = llm.qwen3.model.Config.qwen3_0_6b();
    try std.testing.expect(!cfg.isMoe());
    try std.testing.expectEqual(@as(usize, 28), cfg.num_layers);
    try std.testing.expectEqual(@as(usize, 8), cfg.num_key_value_heads);
}

compiled & run in CI ✓

pub const Error = weights.Error || error{ InvalidConfig, InvalidSequenceLength, MismatchedKvCaches }. Public surface on Model: loadGguf, loadGgufOptions, loadGgufFromFile, loadGgufFromFileOptions (opt-in MoE expert disk streaming, LoadOptions.moe_stream), deinit, forwardLastLogits, forwardLastLogitsProfiled, initKvCache, forwardStep, forwardStepProfiled, forwardStepAllLogits, forwardStepBatch, forwardStepBatchSpans, generate, decoratePtqtp, savePtqtpGguf (§10.9); plus GenerateOptions, ForwardProfile, MoeStreamOptions, LoadOptions, and applyExpertTopP at module level.

Load specifics: when the GGUF is mmap'd, MoE expert stacks (ffn_{gate,up,down}_exps.weight) are borrowed zero-copy from the mapping (weights.loadMoeRhs with borrow = file.is_mmap and !file.isSplit() — split GGUFs cannot hand over their multiple mappings, so their experts are copied) instead of copying multi-GB tensors; the model owns the mapping and unmaps it last. A missing output.weight means tied embeddings (token_embedding.cloneView). The MoE FFN routes on the host (routerTopK with normalize_selected = norm_topk_prob) and runs the router-weighted SwiGLU mixture through weights.moeSwiGluFfnSeq: decode (seq 1) uses a fused expert-parallel GEMV, prefill groups tokens by expert so each expert's weights are read once per batch.

initKvCache builds a uniform-geometry f16 KvCache (KvCache.init(ctx, num_layers, num_key_value_heads, head_dim, capacity)). Qwen3 is the only family whose attention also accepts a q8_0 cache (construct it with kv_cache.KvCache.initWithDtype(..., .q8_0), §13; the runner flag is --cache-type q8_0 — half the KV memory, and since the integer q8xq8 score path also the long-context speed option: decode meets f16 by ~8k context and the halved footprint doubles how much context a RAM budget holds).

var file = try fucina.gguf.File.loadMmap(alloc, io, "models/Qwen3-0.6B-Q8_0.gguf");
defer file.deinit();
const config = try llm.qwen3.model.Config.fromGguf(&file);
var model = try llm.qwen3.model.Model.loadGgufFromFile(&ctx, &file, config);
defer model.deinit();

var kv = try model.initKvCache(&ctx, 512);
defer kv.deinit();
var prefill = try model.forwardStep(&ctx, &kv, &.{ 151644, 872, 198 }, 0); // [1, vocab]
defer prefill.deinit();
var step = try model.forwardStep(&ctx, &kv, &.{9707}, kv.len); // one decode step
defer step.deinit();
// requires model assets to run

Lockstep batch decode. forwardStepBatch(ctx, caches, token_ids) decodes one new token per stream, each stream backed by its own sibling cache from this model's initKvCache (same dtype, distinct pointers, layer count matching the model — violations return Error.MismatchedKvCaches; a full cache returns KvCacheOverflow). Row s of the returned [n_streams, vocab] logits is stream s's next-token distribution and every cache advances by one. The dense trunk (QKV/O-proj, FFN or MoE mixture, lm_head) runs as ONE m=n pass — weights are read once for all streams, the batch-decode bandwidth win — while RoPE positions, KV appends and attention stay per-stream (ragged, each row against its own cache at its own position). Per-row numerics match per-stream forwardStep bit-for-bit below the m-dependent kernel thresholds (quantized x4-packed kernels engage at n >= 4, fused FFN at seq >= 12, tiled attention at seq >= 48); beyond them rows can differ by ~1e-6 reassociation drift. The same thresholds bound forwardStepAllLogits against per-token steps.

var kv_a = try model.initKvCache(ctx, 256);
defer kv_a.deinit();
var kv_b = try model.initKvCache(ctx, 256);
defer kv_b.deinit();
var a = try model.forwardStep(ctx, &kv_a, &.{ 151644, 872 }, 0);
a.deinit();
var b = try model.forwardStep(ctx, &kv_b, &.{ 151644, 8948 }, 0);
b.deinit();
// One m=2 weight pass decodes both streams; row s = stream s's logits.
var logits = try model.forwardStepBatch(ctx, &.{ &kv_a, &kv_b }, &.{ 9707, 3838 });
defer logits.deinit();
// requires model assets to run

Speculative decoding is available on this family: the runner's --spec drives the draft-model-free SAM + Token-Recycling cascade from llm.speculative (§13) with forwardStepAllLogits as the verify pass; --spec-ref doc.txt injects a reference document the drafter can copy spans from. Output is lossless (greedy streams verified identical with and without --spec).

Runner (examples/qwen3/main.zig, the full CLI surface is documented in the README):

# chat / REPL, sampling flags, GPU offload
zig build qwen3 -Doptimize=ReleaseFast -- models/Qwen3-0.6B-Q8_0.gguf \
  --chat "What is the capital of France?" --no-think \
  --temp 0.7 --top-k 40 --top-p 0.9 --seed 42
zig build qwen3 -Dgpu=metal -Doptimize=ReleaseFast -- models/Qwen3-0.6B-f16.gguf --repl

# raw completion, speculative decode, lockstep multi-stream bench
zig build qwen3 -Doptimize=ReleaseFast -- models/Qwen3-0.6B-Q4_K_S.gguf \
  --prompt "The capital of France is" --gen 64 --spec
zig build qwen3 -Doptimize=ReleaseFast -- models/Qwen3-0.6B-Q4_K_S.gguf \
  151644,872,198,9707 --gen 64 --bench 3 --streams 4

# q8_0 KV cache; tokenizer / logit parity oracles
zig build qwen3 -Doptimize=ReleaseFast -- models/Qwen3-0.6B-Q8_0.gguf \
  --prompt "..." --gen 256 --cache-type q8_0
zig build qwen3 -Doptimize=ReleaseFast -- models/Qwen3-0.6B-Q8_0.gguf --tokenize input.txt

# constrained decoding (§13.6; needs -Dllguidance=true): the reply must
# satisfy the JSON schema / regex / Lark grammar; composes with --spec
zig build qwen3 -Dllguidance=true -Doptimize=ReleaseFast -- models/Qwen3-0.6B-Q8_0.gguf \
  --chat "Give me facts about Paris." --no-think \
  --json-schema '{"type":"object","properties":{"city":{"type":"string"},"population":{"type":"integer","maximum":99999999}},"required":["city","population"],"additionalProperties":false}'
zig build qwen3 -Dllguidance=true -Doptimize=ReleaseFast -- models/Qwen3-0.6B-Q8_0.gguf \
  --prompt "The answer is" --gen 32 --regex ' (yes|no)\.'
# --json-schema @schema.json / --lark @grammar.lark read the grammar from a file

14.2.1 LoRA fine-tuning (src/llm/qwen3/train.zig)

llm.qwen3.train trains LoRA adapters over a frozen, possibly quantized qwen3.Model (dense only — MoE configs return Error.MoeUnsupported). The trainer mirrors the inference forward op-for-op but routes every frozen projection through the differentiable frozen-RHS dot (gradients flow to f32 activations only; weight memory stays quantized/f16) and adds trainable A/B deltas on the projections selected at comptime. Mechanics — adapters, optimizers, checkpoints, exec scopes — live in §11; this is the entry-point map.

pub const Targets = struct { q: bool = true, k: bool = false, v: bool = true,
                             o, gate, up, down: bool = false };
pub const ignore_index: usize = std.math.maxInt(usize);
pub fn Trainer(comptime targets: Targets) type

Module-level symbols: Error (MoeUnsupported, ExecScopeRequired, InvalidSequenceLength, LabelLengthMismatch, InvalidLayerRange, InvalidInjection, InvalidCartridge, InvalidCapture, InvalidPacking, InvalidEngram, CartridgeCheckpointUnsupported), Targets, ignore_index, ModelLayer (test seam: the model's per-block layer type), Hidden (fucina.Tensor(.{ .seq, .embed })), Injection ({ pos, row } — a differentiable single-row embedding override), ForwardOptions ({ start_layer = 0, layer_count = null, inject = null } plus the cartridge/engram fields — cartridge, cartridges, capture, packed_segments, engram — and the module-level KvCapture/EngramOptions, all §13.10-§13.11 material).

Trainer(targets) members: init(ctx, model, lora.Config, seed) / deinit; registerAllParams(opt) (registers every A/B under layers.<i>.<target>.lora_{a,b} on anything with addParamNamed; the trainer must outlive the optimizer — params and names are borrowed); saveAdapters(writer) / loadAdapters(reader) / loadAdaptersWithOptions(reader, optim.LoadOptions) (clean safetensors state dict, strict one-to-one on load); loss(ctx, tokens, labels) / lossExt(..., LossOptions) (mean CE against pre-shifted labels, ignore_index masks; must run inside an open exec scope — Error.ExecScopeRequired otherwise — and returns a scope-owned borrow; LossOptions{ .reduction = .mean|.sum, .loss_scale = 1 } is the gradient- accumulation seam, TRAINING.md §4); lossInjected(...); evalLastLogits / evalLogits / evalLastLogitsExt (dropout off, no step advance, run under their own scope, return caller-owned constants); forwardHidden(ctx, tokens, step, opts) (raw residual stream, scope required); the cartridge/engram seams (captureKv, initCartridge, distillLoss/distillLossExt, lossForwardExt, evalLogitsExt/evalLogitsRows, embedLastHidden, freeTransientRope) are §13.10-§13.11 material; the checkpoint_layers field enables recompute-in-backward per layer; n_enabled and LayerAdapters are the comptime target plumbing. Dropout is deterministic per (step, layer, projection) from the base seed; RoPE tables are cached per sequence length and freed only in deinit.

const Trainer = llm.qwen3.train.Trainer(.{ .q = true, .v = true });
var trainer = try Trainer.init(ctx, model, .{ .rank = 8, .alpha = 16 }, 42);
defer trainer.deinit();

var opt = fucina.optim.AdamW.init(ctx.allocator, .{ .lr = 1e-3 });
defer opt.deinit();
try trainer.registerAllParams(&opt);

const scope = ctx.openExecScope();
defer ctx.closeExecScope(scope);
const tokens: []const usize = &.{ 1, 2, 3, 4 };
const labels: []const usize = &.{ 2, 3, 4, llm.qwen3.train.ignore_index };
var loss = try trainer.loss(ctx, tokens, labels);
try loss.backward(ctx);
try opt.step(ctx);
opt.zeroGrad();
// requires model assets to run

The end-to-end loop — fine-tune (zig build finetune), merge adapters into dense weights (zig build export-gguf -- --adapters ... --alpha ...), re-quantize, serve — is scripted in examples/finetune/README.md; the gradient-free twin is zig build es-finetune (§11, TRAINING.md §13).

14.3 Qwen3.5 — Gated-DeltaNet hybrid (src/llm/qwen35/model.zig)

The qwen35 GGUF arch is a hybrid linear-attention transformer (sibling of qwen3next, not a Qwen3 variant): every full_attention_interval-th block is full GQA attention (fused Q+gate projection, per-head q/k RMSNorm, multi-section/partial RoPE, sigmoid output gate); the rest are DeltaNet linear blocks — a causal depthwise conv1d feeding a gated delta-rule recurrent scan over per-v-head state matrices. Both feed a SiLU dense FFN.

Config adds, on top of the usual attention keys: rope.dimension_count (rope_n_rot — partial RoPE when < head_dim), rope.dimension_sections (rope_sections: [4]i32), full_attention_interval (default 4), and the SSM dims ssm.{conv_kernel, inner_size, state_size, time_step_rank, group_count} (ssm_d_conv/d_inner/d_state/dt_rank/n_group), plus nextn_predict_layers and expert_count. Config.isRecurrent(il) implements the block schedule; isMoe() mirrors qwen3. Validation rejects qwen35moe and MTP/NextN variants with Error.UnsupportedVariant (dense text path only).

DeltaNet heads may be non-uniform: ssm_dt_rank v-heads over ssm_n_group q/k heads (numVHeads % numKHeads == 0 required), with the q/k heads broadcast onto the v-heads tiled — v-head h reads q/k head h % numKHeads, matching ggml_gated_delta_net's iv1 % ne1 semantics in both the recurrent and the batched chunked scan. Qwen3.5 dense is uniform (1:1); Ternary-Bonsai-27B (a ternarized Qwen3.6-27B, general. architecture = "qwen35", weights in the Q2_0 g128 container — §10.7) runs 48 v-heads over 16 k-heads across 64 blocks (16 full-attention + 48 linear). Its tokenizer declares tokenizer.ggml.pre = "qwen35" (§13.5.1): the qwen2 rules with \p{M} combining marks folded into the word class. Loading it is the ordinary flow — every projection (embeddings and LM head included) is a .q2_0 LinearWeight, logit-parity-validated against the PrismML llama.cpp fork (argmax match at pp1..pp512, cosine ≥ 0.9998 — ≥ 0.99999 on the BLAS prefill arm — token-ID-exact tokenizer).

test "qwen35 hybrid layer pattern" {
    const cfg = llm.qwen35.model.Config{
        .vocab_size = 151_936, .hidden_size = 1024, .intermediate_size = 4096,
        .num_layers = 24, .num_attention_heads = 16, .num_key_value_heads = 2,
        .head_dim = 256, .rms_norm_eps = 1e-6, .rope_theta = 1_000_000,
        .rope_n_rot = 64, .rope_sections = .{ 11, 11, 10, 0 },
        .full_attention_interval = 4,
        .ssm_d_conv = 4, .ssm_d_inner = 4096, .ssm_d_state = 128,
        .ssm_dt_rank = 32, .ssm_n_group = 16,
    };
    // Every 4th block is full attention; the rest run the DeltaNet scan.
    try std.testing.expect(cfg.isRecurrent(0));
    try std.testing.expect(!cfg.isRecurrent(3));
    try std.testing.expect(cfg.isRecurrent(4));
}

compiled & run in CI ✓

pub const Error = weights.Error || error{ InvalidConfig, InvalidSequenceLength, UnsupportedVariant, UnsupportedKvCacheDtype }. Model surface: loadGguf, loadGgufFromFile, deinit, blockCounts (.{ attn, linear } counts for --info), forwardLastLogits (cacheless, chunked DeltaNet scan; logit-parity-validated against llama.cpp on Qwen3.5-0.8B), initCache, forwardStep, forwardStepWithScanMode, forwardStepProfiled, forwardStepProfiledWithScanMode; module-level LinearScanMode and ForwardProfile.

The streaming state is Cache, not a bare KvCache: an f16 attention KV cache (q8_0 caches are rejected with Error.UnsupportedKvCacheDtype) plus, per linear layer, a conv window ((d_conv-1)*conv_dim floats) and the recurrent state matrix (H*Sd*Sd floats) — O(1) state per linear layer regardless of context. Cache.deinit, Cache.reset (zero all carried state), Cache.len() (current position). LinearScanMode selects the DeltaNet prefill path: .chunked (default — exact batched chunked-GEMM) or .recurrent (exact token-by-token scan, forced even for prefill); both are exact, the choice is performance/validation.

var file = try fucina.gguf.File.loadMmap(alloc, io, "models/Qwen3.5-0.8B-Q8_0.gguf");
defer file.deinit();
const config = try llm.qwen35.model.Config.fromGguf(&file);
var model = try llm.qwen35.model.Model.loadGgufFromFile(&ctx, &file, config);
defer model.deinit();

var cache = try model.initCache(&ctx, 256); // KV + conv/SSM state
defer cache.deinit();
var prefill = try model.forwardStep(&ctx, &cache, &.{ 9707, 11, 1879 }, 0);
defer prefill.deinit();
var step = try model.forwardStepWithScanMode(&ctx, &cache, &.{0}, cache.len(), .recurrent);
defer step.deinit();
// requires model assets to run

No training entry, no forwardStepAllLogits/forwardStepBatch, no speculative decoding on this family. Chat lives in llm.qwen35.chat (src/llm/qwen35/chat.zig): renderPrompt renders the shared ChatML template with the Qwen3.6 generation-prompt think prefill (<think>\n opener when thinking is on; the ChatML empty think block when off), and Engine(TokMod).generate runs one sampled reply per call on a fresh Cache (the recurrent state cannot be truncated to a token prefix, so there is no cross-request KV reuse). lmserve serves the family through it (backend_qwen35.zig — reasoning channel, JSON-schema/regex/Lark constrained output; LMSERVER.md); Ternary-Bonsai-27B (README) is the flagship checkpoint. The CLI is a loader/parity harness:

zig build qwen35 -Doptimize=ReleaseFast -- models/Qwen3.5-0.8B-Q8_0.gguf
zig build qwen35 -Doptimize=ReleaseFast -- models/Qwen3.5-0.8B-Q8_0.gguf --info
zig build qwen35 -Doptimize=ReleaseFast -- models/Qwen3.5-0.8B-Q8_0.gguf --linear-scan chunked

14.4 Gemma 4 — text + MoE (src/llm/gemma/)

gemma4 (26B-A4B class) is the geometry-heavy family: 16 query heads over per-layer KV geometry — interleaved local sliding-window (SWA) and global layers with different head dims, KV-head counts and RoPE bases, trailing layers that share an earlier layer's K/V (shared_kv_layers), optional per-layer embeddings (PLE), per-layer output scales, GeGLU FFNs (shared dense MLP + a 128-expert top-8 MoE), and final logit softcapping.

Config keys beyond the common set: attention.key_length_swa, attention.sliding_window, attention.shared_kv_layers, rope.freq_base_swa, expert_count/expert_used_count/ expert_feed_forward_length, embedding_length_per_layer_input (PLE width, 0 = disabled), final_logit_softcapping, plus the per-layer arrays gemma4.attention.sliding_window_pattern and gemma4.attention.head_count_kv (read by gguf_meta.readU32OrBoolArray, which broadcasts a scalar across layers like llama.cpp's get_key_or_arr). Config.fromGguf wraps Config.fromGgufArch(file, "gemma4"); the arch argument exists because diffusion-gemma shares the identical hparam key set under its own prefix. Config.borrow_experts is a load-time policy field, not a GGUF hparam: true (the --experts=borrow flag) borrows MoE experts zero-copy from the mmap on CPU builds — load in seconds at ~half the RSS instead of x4-packing ~20 GB — at some decode-throughput cost; the default packed path favors peak CPU throughput. Numerically identical either way.

deriveGeometry(allocator, n_layer, swa_pattern, kv_heads_in, shared_kv_layers, head_dim_global, head_dim_swa) !LayerGeometry computes the per-layer view (is_swa, head_dim, kv_heads, has_kv, kv_ref; LayerGeometry.deinit(allocator) frees it): the trailing shared_kv_layers layers store no K/V and instead reference the last same-type writer (offset 2 for SWA, 1 for global).

test "gemma4 shared-KV geometry" {
    const alloc = std.testing.allocator;
    var geom = try llm.gemma.gemma4.deriveGeometry(
        alloc,
        4, // n_layer
        &.{ true, true, false, true }, // SWA pattern (false = global)
        &.{ 4, 4, 8, 4 }, // per-layer KV heads
        1, // shared_kv_layers: the last layer stores no K/V
        256, // head_dim_global
        128, // head_dim_swa
    );
    defer geom.deinit(alloc);
    try std.testing.expect(!geom.has_kv[3]);
    try std.testing.expectEqual(@as(usize, 1), geom.kv_ref[3]); // reuses layer 1
    try std.testing.expectEqual(@as(usize, 128), geom.head_dim[3]);
}

compiled & run in CI ✓

pub const Error = weights.Error || error{ InvalidConfig, InvalidSequenceLength, MismatchedKvCaches, MissingMetadata, PleUnsupported, UnsupportedExpertType, UnsupportedKvCacheDtype }. Model surface: loadGguf, loadGgufFromFile, deinit, initKvCache (per-layer geometry: KvCache.initPerLayer(ctx, geom.kv_heads, geom.head_dim, capacity)), forwardLastLogits, forwardLastLogitsProfiled, forwardStep, forwardStepProfiled, forwardStepAllLogits (speculative verify entry — softcapping applies to every row), forwardStepBatch/forwardStepBatchSpans (lockstep multi-stream decode, 14.2 — PLE models rejected with Error.PleUnsupported), generate + GenerateOptions, ForwardProfile. Only f16 caches are accepted (requireF16KvCache returns Error.UnsupportedKvCacheDtype for q8_0). Final logits are softcapped when final_logit_softcapping != 0 (a fused softcap30 kernel serves the model's actual 30.0 value). The remaining public symbols are loader/forward plumbing reused by diffusion_gemma and the trainer: max_heads (64), metaInt/metaIntOpt/metaFloat/metaFloatOpt, LayerGeometry, deriveGeometry, MoeFfn, PerLayerInject, SeparateAttentionProjection, FusedAttentionProjectionKind, FusedAttentionProjection, AttentionProjectionResult, AttentionProjection (with toResidentF16 and project), Layer, loadLayers (the pub wrapper over a file-private LayerLoader), requireF16KvCache, attnBlock, ffnBlock.

var file = try fucina.gguf.File.loadMmap(alloc, io, "models/gemma-4-26B-A4B-it-UD-Q6_K.gguf");
defer file.deinit();
var config = try llm.gemma.gemma4.Config.fromGguf(&file);
config.borrow_experts = true; // zero-copy experts from the mmap (--experts=borrow)
var model = try llm.gemma.gemma4.Model.loadGgufFromFile(&ctx, &file, config);
defer model.deinit();

var kv = try model.initKvCache(&ctx, 512); // per-layer geometry
defer kv.deinit();
var prefill = try model.forwardStep(&ctx, &kv, &.{ 2, 651, 235 }, 0);
defer prefill.deinit();
var step = try model.forwardStep(&ctx, &kv, &.{651}, kv.len);
defer step.deinit();
// requires model assets to run

MoE expert kernels (moe.zig, moe_route.zig, moe_route_tensor.zig, survey depth). The expert FFN has two weight representations: per-expert packed RHS (MoeFfn.gate/up/down, the tested Q6_K/Q8_0 packed kernels — peak CPU throughput) and raw GGUF-layout blocks (RawExpertWeights{ gu: .q6_k|.q4_k, dn_blocks, device_owned, borrowed }, plus guBlockCount), used on -Dgpu=metal builds (grouped dequant-in-kernel Metal GEMMs read them; the loader then keeps ONE representation — tens of seconds and ~24 GB saved at load), on Q4_K-transcoded experts, under --experts=borrow, and by the trainer. Four entry pairs cover the (decode | batch) x (packed | raw) matrix: decodePackedTensor / batchPackedTensor / decodeRawTensor / batchRawTensor (tagged-tensor wrappers) over decodePacked / batchPacked / decodeRaw / batchRaw. Batch entries consume the shared counting-sort route plan re-exported by moe_route (Plan, BuildResult, build) with the gemma-specific expert-major scatter (moe_route.scatterInto, deliberately serial to keep each token's summation order fixed against parity oracles); moe_route_tensor.scatterGrouped / recordBatch are the tensor-level scatter and profile hooks.

LoRA fine-tuning (gemma4_train.zig, pointer depth — §11). llm.gemma.gemma4_train.Trainer(targets) mirrors the qwen3 trainer over the gemma4 forward: identical Targets struct and defaults (q, v), identical ignore_index, and the same member set — init(ctx, model, lora.Config, seed), deinit, registerAllParams, saveAdapters, loadAdapters, loadAdaptersWithOptions, loss, lossExt + LossOptions, evalLastLogits, n_enabled, LayerAdapters (per-layer geometry sizes the k/v adapters); its cartridge seams (captureKv, initCartridge, distillLoss/distillLossExt, evalLogitsExt/evalLogitsRows, embedLastHidden, freeTransientRope, ForwardOptions) are §13.10 material. Its Error set encodes the intentional exclusions checked in init: PleUnsupported (PLE models rejected), SharedKvUnsupported (any layer with has_kv == false), RawMoeWeightsRequired (MoE layers must retain raw expert blocks — load with --experts=borrow or a raw-expert build; the packed inference-only RHS cannot take gradients), plus CartridgeGeometry, ExecScopeRequired, InvalidSequenceLength, LabelLengthMismatch.

Runner (examples/gemma4/main.zig, README — chat/REPL over the SPM tokenizer (llm.spm_tokenizer) and the generic llm.chat.Conversation; sampling defaults come from the GGUF):

zig build gemma4 -Doptimize=ReleaseFast -- models/gemma-4-26B-A4B-it-UD-Q6_K.gguf \
  --chat "Why is the sky blue?" --experts=borrow
zig build gemma4 -Doptimize=ReleaseFast -- models/gemma-4-26B-A4B-it-UD-Q6_K.gguf \
  --repl --system "Answer tersely." --think
zig build gemma4 -Doptimize=ReleaseFast -- models/gemma-4-26B-A4B-it-UD-Q6_K.gguf \
  --chat "Why is the sky blue?" --spec          # lossless speculative decoding
zig build gemma4 -Dllguidance=true -Doptimize=ReleaseFast -- models/gemma-4-26B-A4B-it-UD-Q6_K.gguf \
  --chat "List three facts about the sky." \
  --json-schema '{"type":"array","items":{"type":"string"},"minItems":3,"maxItems":3}'  # constrained reply (§13.6)
zig build gemma4 -Dgpu=metal -Doptimize=ReleaseFast -- models/gemma-4-26B-A4B-it-UD-Q6_K.gguf \
  --chat "..."                                  # MoE expert FFN on the GPU
zig build gemma4 -Doptimize=ReleaseFast -- models/gemma-4-26B-A4B-it-UD-Q6_K.gguf \
  2,651,235 --bench 3 --profile                 # prefill/decode benchmark

14.5 DiffusionGemma — block text-diffusion (src/llm/diffusion_gemma/model.zig)

The diffusion-gemma arch is not autoregressive: the transformer is exactly gemma4 (this module reuses gemma4's layer loader and attn/ffn blocks), but generation denoises fixed-length token canvases and commits them block-autoregressively. Two forward modes share one weight set:

• encodeStep(ctx, kv, token_ids, pos0) !void — causal prefix pass over the prompt or a finalized canvas; exists only for its K/V side effect (the lm head is skipped), appends and advances the cache, applies the per-layer enc_layer_output_scale. Same pos0/capacity contract as forwardStep.
• canvasForward(ctx, kv, canvas_ids, sc) ![seq, vocab] — one bidirectional denoiser pass over the canvas at absolute positions [kv.len, kv.len + C). Canvas K/V are written into the cache's scratch region past kv.len WITHOUT advancing it (the next step overwrites), so the cache is read-only from the caller's perspective; logits are returned for every row (softcapped). sc is the previous step's self-conditioning signal (null on the first step); passing one on a GGUF without the self_cond_* MLP is Error.SelfConditioningUnavailable.

Config = { base: gemma4.Config, canvas_length, eb: EbParams }; Config.fromGguf reads the gemma4 keys under the diffusion-gemma. prefix plus diffusion.canvas_length (required — Error.MissingCanvasLength) and the optional diffusion.eb_* overrides of EbParams (defaults are the reference generation_config: max_steps = 48, t_min = 0.4, t_max = 0.8, entropy_bound = 0.1, stability_threshold = 1, confidence_threshold = 0.005). Loading additionally requires both per-layer scales (layer_output_scale via the gemma4 layer loader, the diffusion-only enc_layer_output_scale into Model.enc_scale) — Error.MissingLayerScale otherwise — and rejects PLE configs. pub const Error = gemma4.Error || error{ MissingCanvasLength, MissingLayerScale, CanvasLengthMismatch, KvCapacityTooSmall, SelfConditioningUnavailable }.

Model surface: loadGguf, loadGgufFromFile, deinit, initKvCache (per-layer geometry; capacity must cover prefix + one canvas), convertDenseWeightsToF16 (dequantize attention q/k/v/o, the shared dense FFN, the self-conditioning MLP and the lm head to resident f16 so the m = 256 canvas GEMMs take the f16 GPU path — the --gpu-f16 flag; ~4.6 GB extra resident on 26B-A4B, pointless without -Dgpu=metal), encodeStep, canvasForward.

The entropy-bound sampler is exposed as free functions over the canvas logits: SamplerOptions / SamplerPass (owns results + an optional ScSignal; deinit(allocator)) / samplerPass(ctx, logits, temp, u, options) (per-position argmax, entropy of softmax(z/t) and one multinomial draw, parallelized over positions with caller-pre-drawn uniforms so results are thread-count independent; also collects the sparse self-conditioning candidate lists), ScSignal (sparse per-row id/prob lists; deinit), entropyBoundAccept(results, entropy_bound, order, accepted) (accept positions by ascending entropy while the cumulative entropy of the strictly-lower set stays within the bound). The loop drivers: denoiseCanvas(model, ctx, kv, canvas, DenoiseOptions) !DenoiseResult (denoise one canvas in place — uniform-random init, temperature schedule t_max→t_min, per-step acceptance + renoise, stable-and-confident adaptive stop; DenoiseOptions{ .eb, .seed = 0, .self_conditioning = true, .sampler, .on_step, .on_step_user } with StepInfo snapshots feeding the runner's live inline visualization) and generate(model, ctx, kv, prompt_tokens, out_tokens, GenerateOptions) !GenerateResult (encode the prompt once, then per block: denoise, trim at the first EOG token — default ids {1, 106, 50} — or a repetition-loop onset, append the kept tokens, encoder-pass the canvas back into the cache; .on_block callback; returns { produced, steps, blocks }).

const dg = llm.diffusion_gemma.model;
var file = try fucina.gguf.File.loadMmap(alloc, io, "models/diffusiongemma-26B-A4B-it-Q6_K.gguf");
defer file.deinit();
const config = try dg.Config.fromGguf(&file); // gemma4 hparams + canvas_length + EB sampler
var model = try dg.Model.loadGgufFromFile(&ctx, &file, config);
defer model.deinit();

const prompt: []const usize = &.{ 2, 651, 235 };
var kv = try model.initKvCache(&ctx, prompt.len + 2 * config.canvas_length);
defer kv.deinit();
var out: [512]usize = undefined;
const result = try dg.generate(&model, &ctx, &kv, prompt, &out, .{
    .denoise = .{ .eb = config.eb, .seed = 42 },
    .max_new_tokens = 256,
});
_ = out[0..result.produced];
// requires model assets to run

No training entry and no speculative decoding (there is no autoregressive draft/verify seam). Runner (examples/diffusion_gemma/main.zig, README; on a TTY the reply denoises live inline — --no-visual disables):

zig build diffusion-gemma -Doptimize=ReleaseFast -- models/diffusiongemma-26B-A4B-it-Q6_K.gguf \
  --chat "Why is the sky blue? Answer in two sentences." --max 256 --seed 42 --experts=borrow
zig build diffusion-gemma -Doptimize=ReleaseFast -- models/diffusiongemma-26B-A4B-it-Q6_K.gguf \
  --repl --system "Answer tersely."
zig build diffusion-gemma -Doptimize=ReleaseFast -- models/diffusiongemma-26B-A4B-it-Q6_K.gguf \
  --chat "..." --steps 32 --entropy-bound 0.2 --t-max 0.9 --t-min 0.4   # sampler knobs
zig build diffusion-gemma -Dgpu=metal -Doptimize=ReleaseFast -- models/diffusiongemma-26B-A4B-it-Q6_K.gguf \
  --chat "..." --gpu-f16

14.6 Parakeet ASR (src/llm/parakeet/)

NVIDIA NeMo FastConformer speech recognition (110M hybrid TDT+CTC through 0.6B multilingual TDT), ported stage-for-stage against parakeet.cpp/NeMo. The pipeline is a chain of free functions over one gguf.File — there is no monolithic Model struct; ParakeetWeights is a lazy name-keyed cache:

stage	module	role
front end	frontend.zig	WAV → 16 kHz mono f32 → preemphasis → STFT power → log-mel (+ per-feature normalization)
subsampling	subsampling.zig	stride-2 conv2d stack (subsampling_factor, 8x on the shipped models), mel → [T/8, d_model]
encoder	encoder.zig	Conformer layers: rel-pos MHA + conv module + macaron half-step FFNs
decoder	decoder.zig	CTC argmax-collapse, or LSTM predictor + joint (RNNT/TDT greedy)
text	tokenizer.zig, transcription.zig	SentencePiece detokenize; word grouping, timestamps, JSON
streaming	streaming.zig	cache-aware chunked encoder + carried-state RNN-T session

Config and loading (loader.zig). Config.fromGguf requires general.architecture == "parakeet" and reads flat parakeet.* keys: arch (a DecoderArch: ctc, rnnt, tdt, hybrid_tdt_ctc, hybrid_rnnt_ctc, with hasCtc/hasTransducer/isTdt predicates — describes which weights exist, not which decoder runs), the encoder set encoder.{d_model, n_layers, n_heads, ff_dim, feat_in, conv_kernel, conv_norm_type, subsampling_factor, subsampling_conv_channels, pos_emb_max_len, xscaling}, the mel front end preprocessor.{sample_rate, n_mels, n_fft, win_length, hop_length, preemph, mag_power, log_zero_guard, normalize}, vocab_size/blank_id, the predictor decoder.{pred_hidden, pred_rnn_layers} + decoding.max_symbols, the joint joint.{joint_hidden, activation}, and the TDT duration table parakeet.tdt.durations (required iff the arch is TDT; max_durations = 16). Derived accessors: vPlus / checkedVPlus (joint output width = vocab + blank + durations), subsampledFreq / checkedSubsampledFreq, durationsSlice. Supporting enums: ConvNorm, Normalize, JointActivation. Streaming-variant models additionally carry StreamingConfig.fromGguf (null for offline models): att_context_left/ right/style (AttContextStyle.regular|chunked_limited), the [step0, step>=1] schedules chunk_size/shift_size/ pre_encode_cache_size (+ stepIdx), cache_drop_size, last_channel_cache_size, valid_out_len, drop_extra_pre_encoded. Multilingual prompt-conditioned models carry PromptConfig.fromGguf (null otherwise) with resolveLang mapping a locale to its one-hot index. expectTensor/TensorClass and validateTensors gate the tensor inventory at load (f32_required vs quantizable — the GGUFs ship f16/q8_0/q6_k/ q5_k/q4_k variants of the big matmuls; norms, biases and the featurizer stay f32). loadFeaturizer returns the Featurizer (mel filterbank fb + window, borrowed zero-copy from the mapping — valid only while the gguf.File lives); loadPieces decodes the SentencePiece table (outer slice caller-freed, pieces borrow the mapping). Error covers NotParakeet, UnsupportedArch, UnsupportedConvNorm, InvalidConfig, MissingMetadata, TensorNotFound, TensorShapeMismatch, TensorDtypeMismatch.

Weights (weights.zig). ParakeetWeights.init(ctx, file) / deinit — a lazy cache mapping tensor names to LinearWeights, built on first use; enableF32Blas pre-converts f32 GEMM operands for the BLAS path (the --f32-cache flag); accessors getLinear, getLinearF32, linear, linearD, linearQkvD (fused QKV), linearPosAllD (all layers' linear_pos in one GEMM); free borrowF32 (alignment-checked zero-copy f32 view of mapped bytes). Sessions borrow the weights struct; it must outlive them.

Front end (frontend.zig): Audio (+ deinit), loadWav16kMono / loadWav16kMonoFile (PCM16/24/32/f32, stereo downmix, linear resample to 16 kHz via resampleLinear), preemphasis, StftParams, Spectrogram, DftBasis (precomputed direct-DFT basis for the melSpectrogramFast* variants), stftPower, MelParams, MelSpectrogram (feat-major feats[m * n_frames + t]), melSpectrogram, melSpectrogramFast, melSpectrogramFastWithBasis. NeMo-exact: constant-pad STFT, f64 accumulation, per-feature z-score over the valid frames.

Subsampling (subsampling.zig): subsample / subsampleWithWeights (the offline stride-2 conv stack + linear proj), streamingSubsample (the causal variant), conv2dPublic (the shared conv2d entry, also exercised by tests). Encoder (encoder.zig): encode / encodeWithWeights (mel [n_mels, T] → [T/8, d_model], caller-owned Tensor(2)), built from conformerLayer = relposAttention (Transformer-XL relative-position attention) + convModule + feedForwardT, with helpers relPosEncoding, layerNorm, layerNormByName, layerNormByNameT, linearWT, f32Data, attnName, convName.

Decoders (decoder.zig): ctcDecode (frame argmax → ctcCollapse / ctcCollapseWithMeta); tdtDecode / tdtDecodeWithWeights (greedy TDT: LSTM Predictor (init/deinit/step) + Joint (init/deinit/encProjAll/step), duration head skips frames); rnntDecodeFrames + RnntDecodeState (init/deinit/reset) — the carried-state per-chunk variant streaming uses. The batch decoders return caller-freed []i32 token ids; rnntDecodeFrames returns !void and appends into a caller-provided *std.ArrayList(i32). TokenInfo/TokenMeta optionally collect per-token frame indices and confidences. Text: tokenizer.detokenize (SentencePiece piece join, ▁ → space); transcription.Word, groupWords, freeWords, toJson (per-word timestamps from token frames x frame_sec).

Offline transcription end-to-end:

const pk = llm.parakeet;
var file = try fucina.gguf.File.loadMmap(alloc, io, "models/parakeet/tdt_ctc-110m-f16.gguf");
defer file.deinit();
const cfg = try pk.loader.Config.fromGguf(&file);
const feat = try pk.loader.loadFeaturizer(&file, cfg); // borrows the mmap

var audio = try pk.frontend.loadWav16kMonoFile(alloc, io, "clip.wav");
defer audio.deinit(alloc);
var mel = try pk.frontend.melSpectrogram(alloc, audio.samples, .{
    .stft = .{ .n_fft = cfg.n_fft, .hop = cfg.hop_length, .win_length = cfg.win_length,
               .mag_power = cfg.mag_power, .preemph = cfg.preemph },
    .n_mels = cfg.n_mels,
    .log_guard = cfg.log_zero_guard,
    .normalize_per_feature = cfg.normalize == .per_feature,
}, feat.fb, feat.window);
defer mel.deinit(alloc);

var w = pk.weights.ParakeetWeights.init(&ctx, &file);
defer w.deinit();
var enc = try pk.encoder.encodeWithWeights(&ctx, &file, cfg, mel.feats, cfg.n_mels, mel.n_frames, &w);
defer enc.deinit();
const ids = try pk.decoder.tdtDecodeWithWeights(&ctx, cfg, &enc, alloc, &w, null);
defer alloc.free(ids);

const pieces = try pk.loader.loadPieces(&file, alloc);
defer alloc.free(pieces);
const text = try pk.tokenizer.detokenize(alloc, pieces, ids);
defer alloc.free(text);
// requires model assets to run

Streaming API (streaming.zig). Two layers:

• StreamingEncoder (init(allocator, cfg, StreamingConfig) / deinit / reset / step / layerStack) runs the full cache-aware encoder on one mel chunk: causal subsampling → drop drop_extra_pre_encoded leading frames (steps >= 1) → the layer stack with carried caches → slice to valid_out_len (all frames on the last chunk). Per-layer state is a ConvCache (depthwise conv tail, init/deinit/reset) and a ChannelCache (attention K/V left-context window, init/deinit/reset/advance); the windowed attention itself is streamingAttnMask + streamingRelposAttention + streamingConformerLayer, with streamingDepthwiseConv for the conv module and applyPromptKernel for the multilingual one-hot projection.
• StreamingSession (init(allocator, file, cfg, sc, weights, lang) / deinit) owns the encoder caches, the LSTM predictor + joint, the carried RnntDecodeState and the accumulated output. Feed granularities: feedMel(ctx, file, w, mel, n_mels, t) windows a whole clip through the chunk schedule; feedMelChunk processes one pre-windowed mel chunk; encodeChunkPrompted returns a chunk's encoder frames (+ prompt kernel); feedEncoderFrames greedy-decodes frames you encoded yourself. Non-special tokens accumulate in session.tokens (set collect_meta = true to align token_meta for timestamps); <EOU>/ <EOB> events are counted in eou_events and reset the decoder state for the next utterance (decoder-only, matching the reference). init returns error.UnknownLang if a prompt-conditioned model cannot resolve lang.

No training entry, no speculative decoding (not autoregressive text). Runner (examples/parakeet/main.zig, README):

zig build parakeet -Doptimize=ReleaseFast -- --model models/parakeet/tdt_ctc-110m-f16.gguf \
  --audio clip.wav --transcribe                       # offline; --json --timestamps for word timing
zig build parakeet -Doptimize=ReleaseFast -- --model models/parakeet/tdt_ctc-110m-f16.gguf \
  --audio clip.wav --stream                           # cache-aware chunked pipeline
zig build parakeet -Dparakeet-mic -Doptimize=ReleaseFast -- \
  --model models/parakeet/tdt_ctc-110m-f16.gguf --mic # live microphone
zig build parakeet -Doptimize=ReleaseFast -- --model ... --manifest files.txt --decoder ctc

--decoder tdt|ctc picks the head on hybrid models; --lang XX selects the prompt locale on multilingual models; --threads N caps the worker team.

14.7 Kimi-K3 — KDA/MLA hybrid, architecture parity (src/llm/kimi3/model.zig)

The kimi3 family (the Kimi-Linear lineage) is a KDA linear-attention / Gated-MLA-NoPE hybrid with a latent sigmoid-routed MoE, cross-layer attention residuals (layers at index % attn_res_block_size == 0 snapshot their entry into a residual bank that later layers depth-mix against), and the SiTU gated activation (§4.5). It is the one family outside the GGUF conventions of §14.1: an architecture-parity model over the tiny f32 reference checkpoint, loaded from a checkpoint directory. Config.fromJsonFile(allocator, io, path) parses the reference config.json (text_config, including the 1-based KDA layer list from linear_attn_config — Config.isKdaLayer(layer_idx) implements the schedule, Config.deinit(allocator) frees the list) and Model.load(allocator, io, ctx, dir) reads <dir>/config.json plus <dir>/model.safetensors (safetensors.File.loadMmap).

Heavy lifting goes through the shared exec ops — matmulTransB, causalDepthwiseConv1dAxisRank, kdaRecurrent (§4.13), gatedRank(.situ), rmsNormMulAxisRank — while the depth mixture and the small MLA core are model-local routines. Model surface: load, deinit, forward(ctx, tokens) (full-sequence forward: token ids, []const u32, to [seq, vocab] logits) and forwardProbed(ctx, tokens, probe) with Probe ({ context, callback }) — a stage observer that receives the same per-layer intermediates the reference dumps. No KV/state cache, no runner CLI, no GGUF path; the golden tests pin the forward against reference activations.

14.8 Example applications

Beyond the family runners, six applications exercise the library end to end.

nanochat (examples/nanochat/, README) is a from-scratch CPU port of karpathy/nanochat: BPE tokenizer training (rustbpe-equivalent), GPT base pretraining (grad-accum loop, Muon+AdamW, checkpoint/resume), supervised fine-tuning on the task mixture, bits-per-byte evaluation, and an interactive chat CLI with a calculator tool. The port is example-local — everything composes from the public facade — and every stage is validated against the fp32 Python reference under a tiered parity ladder. zig build nanochat -- tok-train|base-train|sft|eval-bpb|chat ....

lmserve (examples/lmserve/, README; LMSERVER.md is the full design doc) is an OpenAI- and Anthropic-compatible HTTP server over the in-tree language models: Chat Completions (POST /v1/chat/completions), the stateless Responses API (POST /v1/responses), and the Anthropic Messages API (POST /v1/messages — a translation layer onto the same engine), with SSE streaming in all three dialects, hermes-style function calling on the qwen3-family backends (tool declarations render into the prompt, the client executes; tool_choice is grammar-forced on -Dllguidance=true builds), JSON-schema/regex/Lark constrained output (-Dllguidance=true builds), an optional Host-header allowlist (--allow-host), and a bounded request queue in front of one inference worker — a per-request stream pipe decouples generation from client sockets, so a stalled client never blocks the worker. Serving levers: --kv-slots keeps interleaved conversations warm (guarded against RAM overcommit; --kv-cache-dir adds a disk tier, and a request sharing a prefix with another resident slot copies those KV rows instead of re-prefilling them, §13.4), --batch N lockstep-decodes queued requests together with per-stream failure isolation (§13.8), --spec adds lossless self-draft speculative decoding (composes with reuse and stop sequences, §13.9), and --cartridge/--fleet mount trained KV-prefix cartridges (§13.10). The GGUF's general.architecture picks the backend (qwen3 / qwen3moe / qwen35 / gemma4 / diffusion-gemma / inkling); --nanochat <dir> serves a nanochat checkpoint. zig build lmserve -- <model.gguf> [--host H] [--port N] [flags].

facedetect (examples/facedetect/, README) runs the insightface buffalo_l pack — SCRFD det_10g detection (boxes + 5-point landmarks), ArcFace IResNet-50 recognition (512-d embeddings, 1:1 verification), GenderAge MobileNet-0.25 attributes, a MiniFASNet x2 anti-spoof ensemble, and 2d106det/1k3d68 dense landmarks — from self-contained GGUFs, as a pure-Zig port of face-detect.cpp. It is the CNN workout for the core op set: the hand-mapped nets (recognizer.zig, scrfd.zig, genderage.zig) drive the public tagged-Tensor facade with channel-last [h, w, c] conv2d, pool2d, prelu, channelAffine and upsample (§4), with GGUF dequant, layout repack and BatchNorm folding at load; the anti-spoof and landmark nets replay a GGUF-embedded node list through an app-level graph dispatcher over ExecContext ops. Decision-critical control paths (cv2-exact letterbox, umeyama alignment, NMS) are verbatim scalar ports; detect/analyze JSON is byte-identical to the reference, embeddings agree at cosine >= 0.999999. zig build facedetect -- detect|embed|verify|analyze|landmarks|bench ....

locate_anything (examples/locate_anything/, README) runs NVIDIA LocateAnything-3B open-vocabulary detection — MoonViT vision tower + MLP projector + Qwen2.5-3B, detection in token space via coordinate tokens — from one GGUF, ported from locate-anything.cpp and validated stage by stage (token streams id-exact in all three decode modes, detections JSON byte-identical at f32). Everything numeric runs on stock tensor ops: the interleaved 2D vision RoPE is a hand-filled RopeTable, ViT attention is the bidirectional grouped-attention arm, the MTP block-diffusion mask rides the additive-bias attention arm, and every linear goes through LinearWeight (so f16/quant arms and BLAS/Metal/CUDA GEMM dispatch apply unchanged). Decode modes hybrid (parallel box decoding with AR fallback), slow (pure autoregressive), fast (MTP-only). ~1.2-2.4x faster than the reference CLI depending on ISA and dtype. zig build locate-anything -- detect --model ... --input scene.png --prompt '...'.

nam (examples/nam/, README) is a complete Neural Amp Modeler ecosystem port: load any upstream-format 0.5.0-0.7.x .nam guitar-amp capture (WaveNet incl. gated/grouped/FiLM variants, LSTM, ConvNet, Linear, SlimmableContainer), render offline or play live through real audio devices (vendored miniaudio, allocation-free lock-free callback, MIDI control), append cabinet IRs and multi-stage chains, train new profiles from a reamp pair (classic/A2/packed WaveNet recipes, ESR validation) and exchange them losslessly with upstream tooling (GGUF interchange recovers a byte-identical .nam). It exercises the streaming-convolution regime: tiny L1-resident models where per-block latency dominates — standard WaveNet at ~49 us per 64-frame block on one x86 P-core (~27x realtime), numerically within 2.3e-6 of upstream tools/render with a strict-contract SIMD tanh. zig build nam -- live|render|train|profile|bench|devices ....

omnivoice (examples/omnivoice/, README) is multilingual zero-shot TTS (646 languages): a MaskGIT non-autoregressive decoder on a Qwen3-0.6B backbone drives the Higgs Audio v2 codec — HuBERT semantic encoder + DAC acoustic codec + 8-codebook RVQ at 25 fps / 24 kHz — for auto voice, voice design (attribute prompts) and voice cloning (reference WAV + transcript). RVQ codes and MaskGIT token streams are byte-exact vs omnivoice.cpp at F32 with a fixed seed; decoded audio cosine >= 0.99999; 2.3-4.6x faster than the reference's CPU backend on M1 Max. The example doubles as a library (pipeline.synthesize/synthesizeStream, ring-buffered play.Player) and ships a WAV↔RVQ codec tool. zig build omnivoice -- tts --model ... --codec ... --lang English -o out.wav (see 14.9 for the flag shape).

The didactic set (single-main.zig examples under examples/):

• smoke/main.zig (zig build smoke) — the minimal facade round trip: two variables, dot, sumAll, backward, gradients printed.
• spirals/main.zig (zig build spirals) — two-spirals MLP trained with every optimizer (SGD/AdamW/Muon/APOLLO/APOLLO-Mini) + param groups, lr schedule, clipping; proves bit-exact checkpoint/resume (§11).
• finetune/main.zig (zig build finetune, README) — the qwen3 LoRA loop of 14.2.1 on a built-in pirate SFT set; --data PATH.jsonl, --accum-steps, --verify-grads gradient-evidence audit.
• es_finetune/main.zig (zig build es-finetune, README) — the gradient-free twin: fucina.es over the same trainer forward; --mode lora|full, --reward rule|acc|nll, anchored weight decay.
• es_spirals/main.zig (zig build es-spirals) — from-scratch ES on two spirals; member-parallel evaluation (one ExecContext + model replica per worker); self-verifying (fails below --target accuracy).
• es_ternary_spirals/main.zig (zig build es-ternary-spirals) — the ternary-native ES flagship: packed TQ2_0 genomes are the inference model (§10, §11), trained by trit flips on the real int8 kernels; self-verifying.
• cartridge/main.zig (zig build cartridge, README) — KV-prefix corpus compression on a frozen qwen3 GGUF (§13.10); corpus-init, distillation training, --equiv acceptance gate.
• cartridge_fleet/main.zig (zig build cartridge-fleet, README) — per-document cartridges trained jointly, cosine-retrieval selection (§13.10).
• engram/main.zig (zig build engram, README) — conditional n-gram memory graft trained on a frozen qwen3 GGUF (§13.11, ENGRAM.md).

14.9 Example → features → run command

Weights are never bundled; RUNNING-MODELS.md lists the download source for every model row; each example's README (examples/<name>/README.md) documents its full flag set. The table omits -Doptimize=ReleaseFast for width — add it to every real run.

example	demonstrates	run
smoke	facade: tensors, dot, autograd (§3-§5)	zig build smoke
spirals	optimizers, schedules, checkpoint/resume (§11)	zig build spirals
es_spirals	from-scratch ES, member-parallel eval (§11)	zig build es-spirals
es_ternary_spirals	ternary-native ES on TQ2_0 kernels (§10, §11)	zig build es-ternary-spirals
finetune	qwen3 LoRA SFT, accumulation, data loader (§11, 14.2.1)	zig build finetune -- --model models/Qwen3-0.6B-Q4_K_S.gguf --steps 30 --rank 8 --alpha 16 --save /tmp/qwen3-lora
es_finetune	gradient-free LLM fine-tuning (§11)	zig build es-finetune -- --model models/Qwen3-0.6B-Q4_K_S.gguf --reward acc --iterations 100 --population 8
qwen3	dense+MoE decode, KV cache (f16/q8_0), batch decode, speculative (§13, 14.2)	zig build qwen3 -- models/Qwen3-0.6B-Q8_0.gguf --chat "..." --no-think
qwen35	Gated-DeltaNet hybrid, recurrent cache (14.3)	zig build qwen35 -- models/Qwen3.5-0.8B-Q8_0.gguf --info
gemma4	per-layer KV geometry, MoE experts, SPM chat, speculative (14.4)	zig build gemma4 -- models/gemma-4-26B-A4B-it-UD-Q6_K.gguf --chat "..." --experts=borrow
diffusion_gemma	block text-diffusion, EB sampler, live denoise UI (14.5)	zig build diffusion-gemma -- models/diffusiongemma-26B-A4B-it-Q6_K.gguf --chat "..." --seed 42
parakeet	ASR pipeline, streaming encoder, mic capture (14.6)	zig build parakeet -- --model models/parakeet/tdt_ctc-110m-f16.gguf --audio clip.wav --transcribe
omnivoice	MaskGIT TTS, HuBERT/DAC/RVQ codec, streaming WAV	zig build omnivoice -- tts --model models/omnivoice/omnivoice-base-Q8_0.gguf --codec models/omnivoice/omnivoice-tokenizer-F32.gguf --lang English -o out.wav
nam	streaming conv nets, live audio/MIDI, .nam/GGUF interchange	zig build nam -- live profile.nam --ir cab.wav
facedetect	channel-last conv2d/pool2d/prelu/upsample CNNs (§4)	zig build facedetect -- detect --model models/buffalo_l.gguf --input face.png
locate_anything	VLM: ViT + projector + LM, token-space detection	zig build locate-anything -- detect --model models/locate-anything-f32.gguf --input scene.png --prompt '...'
nanochat	end-to-end GPT: BPE tokenizer training, pretraining, SFT, bpb eval, chat (14.8)	zig build nanochat -- chat -i <ckpt dir> --tokenizer <tokenizer.bin> -p "..."
lmserve	OpenAI- and Anthropic-compatible HTTP server: chat completions + responses + messages, function calling, SSE streaming, constrained output, --spec/--batch/--kv-slots (14.8)	zig build lmserve -- models/Qwen3-0.6B-Q8_0.gguf --port 8080
deepseek2	DeepSeek V2/V3: MLA compressed KV cache, MoE decode (README)	zig build deepseek2 -- models/DeepSeek-V2-Lite-Chat.Q8_0.gguf --prompt "..." --gen 64
glm4moe	GLM-4.5 family: native MTP speculative decode, streamed experts (README)	zig build glm4moe -- models/glm45-air/GLM-4.5-Air-Q6_K-00001-of-00002.gguf --prompt "..." --gen 64 --mtp
deepseek4	DeepSeek V4 Flash: CSA/HCA trunk, streamed experts, MTP sidecar (README)	zig build deepseek4 -- <model.gguf> --chat --prompt "..." --moe-stream
inkling	Inkling: hybrid rel-bias attention, shortconv sites, sink-shared MoE; parity harness (README)	zig build inkling -- <model.gguf> --prompt "..." --gen 64
cartridge	KV-prefix corpus compression on a frozen qwen3 GGUF (§13.10)	zig build cartridge -- --model models/Qwen3-0.6B-f16.gguf --corpus README.md --p 256 --suffix-max 128 --equiv
cartridge_fleet	per-document cartridge fleets: joint training, cosine retrieval (§13.10)	zig build cartridge-fleet -- --model models/Qwen3-0.6B-f16.gguf --docs README.md --equiv --p 256 --suffix-max 128
engram	n-gram memory graft training on a frozen qwen3 GGUF (§13.11)	zig build engram -- --model models/Qwen3-0.6B-f16.gguf --equiv
ptqtp_spirals	float-trained MLP decorated post-training with trit-planes, self-verifying (§10.9)	zig build ptqtp-spirals
ptqtp_qwen3	PTQTP-decorate a Qwen3 GGUF in place, NLL before/after, --save GGUF (§10.9, §13.2.1)	zig build ptqtp-qwen3 -- models/Qwen3-0.6B-Q4_K_S.gguf --planes 2
(tool) export-gguf	transcode/re-emit GGUF, merge LoRA adapters (§11, §12)	zig build export-gguf -- --from-gguf in.gguf --out out.gguf --dtype q8_0