* feat(kvcache): add fp8_ds_mla and nvfp4_ds_mla kv formats for deepseek-v4 / v4.1
Window and compressed KV are stored as fp8_ds_mla rows (584 B instead of
1024 B) and indexer keys as MXFP4, holding the same values as the
reference's quant round-trip. DeepSeek-V4.1's packed layout is named
nvfp4_ds_mla and shares the row-format code.
* refactor(engine): drop the model field from the dsv4 kv formats
* feat(dflash): load dflash2 gguf drafts for qwen3.8-flash-next
* fix(dflash): accept mrope sections and reuse the float qkv buffer
* docs(dflash): scope the dequantized parts to quantized ggufs
* perf(gemma4): read the source layer's kv slab on kv-shared layers
* refactor(gemma4): read kv sharing from kv_shared_layers in the module and the draft view
Gemma4Attention takes its source and shared flags from config.kv_shared_layers,
the list the pool maps, and keys shared_kv by the source layer. The draft's
read-only view routes through the same list. A group whose shared layers have
no earlier layer of their type is refused at parse time.
* test(gemma4): give the tiny shared-layer configs an earlier source layer
* test(gemma4): drop the two tests of branches that already fail at load
* feat(gguf): serve deepseek-v4.1 gguf checkpoints on deepseek_v41
llama.cpp 10-25 added the deepseek41 layout. The reader takes the config from the file, undoes the converter (engram gate stored as the product, one per_layer_token_embd tensor, hash constants as KVs) and streams the Engram rows from the file as the HF path does.
* fix(gguf): refuse deepseek-v4.1 gguf mismatches at config time
Refuse an FTW of the GGUF before the writer creates the output dir and keep
the side-file hook after the dense loop as on main. Check the Engram hash KVs
and the table in parse_gguf_config, refuse a table with a .scale, and share
the clamp and expert_count readers with the deepseek4 GGUF reader.
* perf(attention): run up to 8 query rows per request in the triton decode kernel
Speculative verify (MTP, DFlash) and the MTP draft extend sent q_len > 1 to the extend kernel, which streams the whole context per (request, query head) with no KV split: 6.6-60x the single-row decode time at 32K-128K on sm_120.
- the grouped decode packs query rows x GQA heads in the M axis, limits each row to its own position and window, and keeps the KV split
- the triton backend sends batches of at most 8 rows per request without multimodal blocks to it
- a program's query heads divide the GQA group, which also fixes wrong output for groups above 16 that are not a multiple of 16 (e.g. 48/2)
* fix(attention): keep multi-row triton decode free of register spills on sm_80 to sm_120
- pass maxnreg=255 to multi-row stage-1 launches on CUDA: without .maxnreg ptxas holds these tiles at 80-128 registers for occupancy and spills 8-16 B on sm_80 to sm_120
- pick block_n and warps from a launch sweep under maxnreg, among the variants that do not spill on sm_80, 86, 89, 90, 100 and 120; quantized head_dim 512 takes 16 rows x heads again
* fix(scheduler): return the sliding-window slots when a verify step rolls back
* feat(gemma4): add the assistant draft model and its loader
* feat(spec): serve sliding-window targets with a draft that keeps no KV
* fix(spec): budget verify rows in the swa pool and drop the post-verify sync
* feat(kvcache): add fp8 and int8 per-token-head KV cache formats
* fix(kernel): keep the bf16 m3 sparse k/v loads inline
* fix(attention): refuse a backend that cannot read the kv format
- filter the m3 page-size fallback by kv format
- release the scale buffer when a qsa or bsa rebuild fails
- keep the bf16 qsa and m3 attend code identical to main
- drop docstrings that claimed no register spills
- drop guard tests
* fix(kernel): widen dsa latent row offsets past int32
* feat(spec): store the mtp draft kv in the target's format
* feat(quant)!: compress bf16 weights to df11 at load with --online-quant
GLM-4.7 no longer hardwires DF11 attention and embedding; it serves them
bf16 unless --online-quant df11 is passed, and FTW files converted with the
old DF11 tensors must be reconverted.
* fix(quant): skip layers --online-quant cannot convert instead of failing
* feat(spec): add --speculative-token-map for a smaller mtp draft head
* feat(spec): add scripts/make_token_map.py to write a token map
* fix(spec): correct the token map vram note and seed its test
Serve the HF safetensors and GGUF files. The decoder gains per-layer inputs, KV-shared layers and E2B's double-wide MLP.
- --embed-weights host moves the per-layer table to pinned host RAM, which fits these models on an 8 GB GPU.
- The vision tower does not implement clipped linears, so these checkpoints serve text only.
- Each KV-shared layer still stores its own KV copy.
* feat(spec): hand the pending token to Proposer.propose and return a SpecProposal
* feat(engine): Sampler.probs and sparse-q rejection
* feat(spec): let a proposer declare slot states riding the GDN slots
* feat(models): export layer outputs to the feature sink in the qwen3_5 and qwen4_exp families
* feat(attention): record the source slot of each tracked request
* feat(kernel): DFlash2 draft kernels
* feat(models): DFlash2 draft family
* feat(spec): DFlashWindow
* feat(spec): DFlashProposer
* docs(cli): add dflash to the speculative decoding section
* refactor(engine): share the sampling backend choice and drop the extra renorm in Sampler.probs
* fix(models): build the DFlash rope table uncached and refuse draft scalings it does not apply
* refactor(spec): share the draft path check between the MTP and DFlash proposers
* test(e2e): compare rebuilt outputs exactly only without hybrid MoE
* refactor(spec): keep proposer slot states on EngineConfig instead of replacing model_config
* perf(spec): compute the verify distribution once for sparse-q rejection
* perf(models): let the feature sink add the residual on the rows it keeps
* perf(models): share the DFlash draft K/V weights with the ring projection
* fix(kernel): keep the dflash selector off zero-probability candidates
* test(spec): drop the DFlash tests that the draft forward and Monte Carlo tests already cover
* fix(spec): default and cap DFlash steps at the draft's block_size - 1
* fix(models): reject DFlash draft configs the draft forward cannot serve
* test(e2e): read the MoE strategy from the rebuild test's extra args
* docs(spec): say that slot-state owners write their own prefill track snapshot
* test(models): move the DFlash draft parity tests under tests/models/dflash
* refactor(spec): drop the no-op verify handler of the DFlash window
* refactor(models): fold the DFlash candidate top-k into propose
* refactor(spec): declare the draft step limits through Proposer.num_steps only
* refactor(kernel): make the DFlash ring attention stateless and keep its workspace on the ring
* refactor(models): name the DFlash draft submodules like the other model families
* refactor(spec): move SpecProposal next to DraftContext in spec/proposer.py
* refactor(spec): hand the verify the draft distribution over the whole vocabulary
* refactor(spec): call the DFlashRing object dflash_ring and its length ring_len
* refactor(models): split the DFlash draft's propose into compute_candidates and sample_path
* refactor(spec): keep the draft distribution on its support in SpecProposal
* fix(spec): drop the step's proposal once the verify has read it
* refactor(spec): name the DFlash feature sinks DFlashPrefillSink and DFlashVerifySink
* refactor(models): read the declared Batch.features directly instead of through getattr
* test(dflash): check the window rebuild inside the block forward test
* test(spec): share the plain run of the verify contract and run the DFlash draft once per family
* test(kvcache): drop the window cases from the pool tests and check a layerless state's bytes
* test(spec): check the verify probs of each row against a torch top-k and top-p reference
* test(e2e): add an accept-len floor and a prefill chunk knob to the AIME gate
* refactor(scheduler): let callers take the speculative counts from the status reporter
* test(spec): keep the shared plain runs of the verify contract in their own temp dirs
* test(models): give the ricursive test's stand-in batch a features field
* feat(ricursive): add ripple-65b preview support
* fix(ricursive): read swiglu_clamp as a per-layer gguf array
llama.cpp's Ripple converter (ggml-org/llama.cpp#31692) writes
ricursive.swiglu_clamp with one value per layer, while the first
community GGUFs store a scalar. Accept both and refuse per-layer
values that differ, since the model has a single clamp.
* fix(ricursive): add ripple's own parsers and copy the dense reader
The patch repeats every edit to a copied file and nothing checks that the two agree. The pin and the tree reproduce it. scripts/sync-llama-cpp.sh diffs the copy against the pin and merges a newer llama.cpp from it, and docs/ggml-kernels.md says how to maintain the copy.
* feat(server): report running usage on stream chunks with continuous_usage_stats
* fix(shell): count tokens instead of stream chunks in the live tok/s
* refactor(server): build both openai stream usages from GenUsage
* fix(shell): reset the first-delta count per turn and ignore zero running totals
debug_once / info_once / warning_once log on rank 0, once per distinct message and args. select_kernel, which logged its choice once per layer, and the hand-rolled log-once guards now use them.
* feat(server): report per-request prefill and decode timings in api responses
* fix(server): take timings spans from the scheduler and gate them behind --enable-timings-report
* fix(scheduler): count accepted drafts as min(n, emitted) and stamp reqstats with perf_counter
The VNNI dots were compute bound per core; with fewer instructions per weight they reach the DRAM read bandwidth.
The W4A8 kernel now follows FREETOKEN_CPU_MOE_ISA.
* feat(spec): add the MTP proposer for the qwen3_5 family
* test(models): read the qwen3_5 MTP head of bf16 and block-fp8 checkpoints
* feat(spec): verify qwen4_exp targets on qsa_sparse graphs with PLE verify handlers
* feat(spec): add the qwen4_exp MTP draft from the checkpoint or a Strata pack
* docs(cli): document the speculative decoding flags
* fix(spec): refuse MTP draft sources the draft cannot read before the target loads
* docs(qwen4_exp): describe the verify path in the PLE docstrings
* fix(spec): refuse a missing relative MTP draft path before the target loads
* fix(moe): read MTP head experts on the host and require both stacked tensors
* refactor(spec): tell the final prefill chunk from prompt_len instead of the scheduler's ChunkedReq
* refactor(spec): let each MTP draft family check its own weight source
* fix(qwen4_exp): size the disk PLE eager staging for decode and verify rows
* perf(kernel): read the KV-index gather's chunk count from the grid, not a constexpr
* refactor(moe): let each family name its MTP expert tensors
* refactor(spec): force the MTP draft's MoE resident in the proposer and check it after load
* fix(spec): refuse MTP with dummy weights
* refactor(spec): move HiddenRelay under spec/mtp, its only user
* refactor(spec): drop MTPDraft.config, fold its embed hooks into embed_tokens
* refactor(models): note FeatureSink's graph binding, drop its runtime_checkable
* fix(models): read quantized MTP fc projections under the quant config
* fix(moe): skip inactive gguf routes in hybrid decode
* perf(moe): run the inactive-route memset and zeroing only in hybrid decode
Only hybrid decode hands the GGUF kernel -1 routes, so mul_mat_id takes an inactive flag
and MMQ does the ids_src1 memset and the zeroing pass only when it is set. Prefill and
fused / offload decode go back to the main-branch MMQ calls.
* perf(moe): compare expert ids unsigned in mm_ids_helper
A negative (inactive) id compared unsigned sorts after every expert and equals none, so the
helper needs no remap to INT_MAX; the remap made nvcc unroll its token loop half as far.
* feat(spec): add the speculative verify infrastructure
* feat(scheduler): run decode steps as speculative verify steps
* chore(engine): mark the speculative decoding config fields with one comment
* feat(server): add short aliases for the speculative decoding flags
* refactor(attention): move linear-attention metadata into a LinearAttnBackend
* refactor(attention): make LinearAttnBackend a BaseAttnBackend with FLACaptureData
* refactor(attention): key every linear-attention row by fla_metadata.cache_indices
* refactor: add engine and scheduler seams for multi-row decode steps
* refactor: rename the graph steps argument to rows_per_req
* refactor: drop VerifyInfo and require rows_per_req on verify batches
* refactor: fix stale seam comments and read decode_rows_per_req directly
* refactor: reword two GraphRunner comments for multi-row graphs
* fix(engine): scale the GPU-decode MoE pool limits by decode_rows_per_req
* refactor: drop the unread Context.moe_offload_cache field
* test: trim the multi-row seam tests