* feat(kvcache): add fp8_ds_mla and nvfp4_ds_mla kv formats for deepseek-v4 / v4.1
Window and compressed KV are stored as fp8_ds_mla rows (584 B instead of
1024 B) and indexer keys as MXFP4, holding the same values as the
reference's quant round-trip. DeepSeek-V4.1's packed layout is named
nvfp4_ds_mla and shares the row-format code.
* refactor(engine): drop the model field from the dsv4 kv formats
* perf(gemma4): read the source layer's kv slab on kv-shared layers
* refactor(gemma4): read kv sharing from kv_shared_layers in the module and the draft view
Gemma4Attention takes its source and shared flags from config.kv_shared_layers,
the list the pool maps, and keys shared_kv by the source layer. The draft's
read-only view routes through the same list. A group whose shared layers have
no earlier layer of their type is refused at parse time.
* test(gemma4): give the tiny shared-layer configs an earlier source layer
* test(gemma4): drop the two tests of branches that already fail at load
* fix(scheduler): return the sliding-window slots when a verify step rolls back
* feat(gemma4): add the assistant draft model and its loader
* feat(spec): serve sliding-window targets with a draft that keeps no KV
* fix(spec): budget verify rows in the swa pool and drop the post-verify sync
* feat(kvcache): add fp8 and int8 per-token-head KV cache formats
* fix(kernel): keep the bf16 m3 sparse k/v loads inline
* fix(attention): refuse a backend that cannot read the kv format
- filter the m3 page-size fallback by kv format
- release the scale buffer when a qsa or bsa rebuild fails
- keep the bf16 qsa and m3 attend code identical to main
- drop docstrings that claimed no register spills
- drop guard tests
* fix(kernel): widen dsa latent row offsets past int32
* feat(spec): store the mtp draft kv in the target's format
* feat(spec): hand the pending token to Proposer.propose and return a SpecProposal
* feat(engine): Sampler.probs and sparse-q rejection
* feat(spec): let a proposer declare slot states riding the GDN slots
* feat(models): export layer outputs to the feature sink in the qwen3_5 and qwen4_exp families
* feat(attention): record the source slot of each tracked request
* feat(kernel): DFlash2 draft kernels
* feat(models): DFlash2 draft family
* feat(spec): DFlashWindow
* feat(spec): DFlashProposer
* docs(cli): add dflash to the speculative decoding section
* refactor(engine): share the sampling backend choice and drop the extra renorm in Sampler.probs
* fix(models): build the DFlash rope table uncached and refuse draft scalings it does not apply
* refactor(spec): share the draft path check between the MTP and DFlash proposers
* test(e2e): compare rebuilt outputs exactly only without hybrid MoE
* refactor(spec): keep proposer slot states on EngineConfig instead of replacing model_config
* perf(spec): compute the verify distribution once for sparse-q rejection
* perf(models): let the feature sink add the residual on the rows it keeps
* perf(models): share the DFlash draft K/V weights with the ring projection
* fix(kernel): keep the dflash selector off zero-probability candidates
* test(spec): drop the DFlash tests that the draft forward and Monte Carlo tests already cover
* fix(spec): default and cap DFlash steps at the draft's block_size - 1
* fix(models): reject DFlash draft configs the draft forward cannot serve
* test(e2e): read the MoE strategy from the rebuild test's extra args
* docs(spec): say that slot-state owners write their own prefill track snapshot
* test(models): move the DFlash draft parity tests under tests/models/dflash
* refactor(spec): drop the no-op verify handler of the DFlash window
* refactor(models): fold the DFlash candidate top-k into propose
* refactor(spec): declare the draft step limits through Proposer.num_steps only
* refactor(kernel): make the DFlash ring attention stateless and keep its workspace on the ring
* refactor(models): name the DFlash draft submodules like the other model families
* refactor(spec): move SpecProposal next to DraftContext in spec/proposer.py
* refactor(spec): hand the verify the draft distribution over the whole vocabulary
* refactor(spec): call the DFlashRing object dflash_ring and its length ring_len
* refactor(models): split the DFlash draft's propose into compute_candidates and sample_path
* refactor(spec): keep the draft distribution on its support in SpecProposal
* fix(spec): drop the step's proposal once the verify has read it
* refactor(spec): name the DFlash feature sinks DFlashPrefillSink and DFlashVerifySink
* refactor(models): read the declared Batch.features directly instead of through getattr
* test(dflash): check the window rebuild inside the block forward test
* test(spec): share the plain run of the verify contract and run the DFlash draft once per family
* test(kvcache): drop the window cases from the pool tests and check a layerless state's bytes
* test(spec): check the verify probs of each row against a torch top-k and top-p reference
* test(e2e): add an accept-len floor and a prefill chunk knob to the AIME gate
* refactor(scheduler): let callers take the speculative counts from the status reporter
* test(spec): keep the shared plain runs of the verify contract in their own temp dirs
* test(models): give the ricursive test's stand-in batch a features field
* feat(spec): add the MTP proposer for the qwen3_5 family
* test(models): read the qwen3_5 MTP head of bf16 and block-fp8 checkpoints
* feat(spec): verify qwen4_exp targets on qsa_sparse graphs with PLE verify handlers
* feat(spec): add the qwen4_exp MTP draft from the checkpoint or a Strata pack
* docs(cli): document the speculative decoding flags
* fix(spec): refuse MTP draft sources the draft cannot read before the target loads
* docs(qwen4_exp): describe the verify path in the PLE docstrings
* fix(spec): refuse a missing relative MTP draft path before the target loads
* fix(moe): read MTP head experts on the host and require both stacked tensors
* refactor(spec): tell the final prefill chunk from prompt_len instead of the scheduler's ChunkedReq
* refactor(spec): let each MTP draft family check its own weight source
* fix(qwen4_exp): size the disk PLE eager staging for decode and verify rows
* perf(kernel): read the KV-index gather's chunk count from the grid, not a constexpr
* refactor(moe): let each family name its MTP expert tensors
* refactor(spec): force the MTP draft's MoE resident in the proposer and check it after load
* fix(spec): refuse MTP with dummy weights
* refactor(spec): move HiddenRelay under spec/mtp, its only user
* refactor(spec): drop MTPDraft.config, fold its embed hooks into embed_tokens
* refactor(models): note FeatureSink's graph binding, drop its runtime_checkable
* fix(models): read quantized MTP fc projections under the quant config
* feat(spec): add the speculative verify infrastructure
* feat(scheduler): run decode steps as speculative verify steps
* chore(engine): mark the speculative decoding config fields with one comment
* feat(server): add short aliases for the speculative decoding flags
* feat(moe): split the offload expert cache into slot pools
* perf(moe): stop recompiling the hybrid ensure kernel for each slot pool size
* fix(engine): turn off prefill overlap when --moe-cache-mem leaves under two layers of slots
* fix(engine): load the --moe-cache-split profile before the expert banks
* test(deepseek-v41): add independent text and vision reference
* feat(quant): support block-32 and unrounded inference outputs
* feat(attention): add csa2 packed cache and sparse backend
* feat(deepseek-v41): serve text and images through shared runtime
* perf(mhc): tune batch-stable reduction tiles
* perf(moe): autotune fp4 decode launch geometry
* perf(moe): admit recurring misses against actual lru victims
* revert: perf(moe): admit recurring misses against actual lru victims
This reverts commit 1652d200f0ebb825379cf44c28baf52971e3a461.
* perf(mhc): derive the reduction tile from the width, not a timing
* perf(moe): replace the ds_fp4 decode autotune with a fixed two-warp launch
* fix(deepseek-v41): fence engram staging against in-flight lookups
* fix(moe): return the cpu routed sum in fp32 and round it in python
* fix(deepseek-v41): keep the vision tower and aligner unquantized
* fix(scheduler): admit bounded-replay hits by the history their resume reads
* refactor(deepseek-v41): name the attention family after the model, not csa2
* refactor(kvcache): import the dsv41 pool only for dsv41 models
* refactor(deepseek-v41): name the replay switch --swa-decoder-replay
* fix(server): keep reasoning_effort a string on the wire
* refactor(models): keep the prefill replay hooks off BaseLLMModel
* refactor(deepseek-v41): drop the fp32 routed sum and batch-invariant prefill
* fix(moe): weight ds_fp4 routes in the down epilogue again
* docs(models): anchor the v4.1 replay paragraph on the tech report
* refactor(deepseek-v41): run the decoder source's narrowed prefill on Block
* refactor(kvcache): let the dsv41 pool own the history a resume keeps live
* refactor(deepseek-v41): resume bounded-replay hits by capping the prefix match
* refactor(deepseek-v41): describe the attention group as a dsv4 variant and derive its geometry from the args
* refactor(deepseek-v41): keep the engram table locator in the model package
* refactor(deepseek-v41): drop the fp32 lm head output
- return the head's bf16 logits, as DeepSeek-V4 does
- revert ParallelLMHead(out_dtype=) and LinearKernel.apply_out_dtype, which only this call used
- remove test_head_preserves_fp32_logit_margin
- cast test_vision's prefill logits to fp32 before comparing, as the harness does
* refactor(quant): carry the w4a8 activation block on MoEConfig
- drop ActDesc and QuantScheme.act; the scheme only describes stored tensors
- Fp8BlockConfig sets MoEConfig.act_block from weight_block_size for its fp4 experts via a layer_config hook
- the ds_fp4 GPU kernel and CPU executor read the block from the layer config and require it
* refactor(deepseek-v41): size the kv pool from the args, not a geometry copy
- delete kvcache/dsv41_geometry.py; the pool and cost model read dsv41_args by attribute, as the DSV4 pool reads dsv4_args
- source / ratio aliasing reads args.roles; the duplicated source and ratio validation is gone
- the cost model owns the tier row formats, row bytes and the private window layers
- the pool keeps the format and window/ratio checks, and exposes the formats and is_private_window to the attention backend
* refactor(kvcache): move the dsv4 pool into kvcache/dsv4
* refactor(kvcache): move the dsv41 pool into kvcache/dsv4
* chore(benchmarks): drop the dsv41 selection and fp8 linear benchmarks
* test(deepseek-v41): drop tests that repeat other coverage
* test(deepseek-v41): load the tiny model's resident experts as banks
* fix(server): parse deepseek-v4.1's space-led dsml tool calls
---------
Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
* feat(mm): serve image input on the Qwen VL families
- frontend: OpenAI image_url, Anthropic image blocks and Responses
input_image (http(s), base64, gated file://) become MMItems in the
tokenizer workers; image spans carry content pseudo ids so the radix
cache keys them by content; mrope positions are precomputed per request
- registry: EncoderSpec per family (Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next
register a vision encoder serving image) and ModelSpec.mm_processor;
EngineConfig.active_encoders decides what a process builds,
served_modalities what it accepts; --text-model-only, --mm-disable,
--mm-encoder-weights, --mm-embed-cache-device, --mm-max-pixels
- engine and scheduler: per-request encoder-cache row ownership, chunk
planning on the batch (jobs, gather rows, scatter rows), items encoded
right before the LM forward and gathered into batch.mm_embeds
- models: the Qwen VL vision tower on the text tower's TP layers with
DeepStack, block weights streamed from pinned host banks by default,
merger outputs parked on the host between blocks; SupportsMultimodal
(encode, place_encoder_weights) and a shared embed_input_ids
- server: image inputs rejected with a client-facing reason when no
vision encoder is served; GET /v1/stats reports model.input_modalities
* feat(mm): replace the pixel cap with a token budget and processor kwargs
- --image-min-tokens / --image-max-tokens replace --mm-max-pixels; MMProcessor.get_mm_processor_kwargs turns the MultimodalConfig into the processor call's arguments, the default passes mm.processor_kwargs through and Qwen VL converts the budget to pixel areas in size.shortest_edge / longest_edge
- --mm-processor-kwargs hands a JSON object to the checkpoint's image processor call, applied after the token budget
- MMProcessor is an ABC that owns the model path, the config and the lazily loaded image processor; get_mm_processor returns None when --mm-disable leaves no encoder to serve
- the frontend tokenizer, the tokenizer worker and the offline LLM build one processor each and hand it to TokenizeManager, so the tokenize message carries no per-request pixel cap
Hybrid 45-layer model (34 KDA + 11 DSA) with NVFP4 MoE (288 experts):
- KDA linear-attention kernels and op, hybrid-linear pool dispatch
generalized to any paged family
- NoPE-MLA attention with a kpool DSA indexer backend: 64-token pages,
1/ratio shadow index slab, per-request tail rings, CUDA-graph-safe
decode, prefix-cache snapshot contract
- mHC (x4 residual streams) layer with a fused split-K triton kernel;
torch reference kept as the semantic pin
- clamped-SwiGLU activation across the MoE stack
- weight loading for NVFP4 exports in the multimodal-wrapper layout:
ModelOpt and compressed-tensors (RedHatAI) tensor kinds;
Co-authored-by: Shuo Yang <73746844+andy-yang-1@users.noreply.github.com>
Serve Qwen3.8-Flash-Next (HF model_type qwen4_exp) text-only: 36 GDN +
12 QSA compressed-sparse attention layers on 4 hyper-connection residual
streams, a PLE n-gram embedding layer backed by a 47.7 GiB pinned-host
table with UVA gather, and 512 NVFP4 / block-fp8 routed experts (top-10)
plus a gated shared expert.
- attention: qsa_sparse backend (AttnType.QSA) over QSAKVCache -- paged
GQA K/V, a 1/ratio compressed index-key slab shadowing the KV pages,
and a per-request pending ring sized from index_ratio
- kvcache: declarative slot-sibling states (ModelConfig.slot_states) on
LinearStatePool carry the PLE conv history and n-gram context through
the hybrid-radix snapshot/COW lifecycle
- scheduler: hybrid prefill chunks align to the page size so snapshots
land on donatable boundaries
- kernels: triton kernels adapted from vLLM/SGLang (hc, qsa, ple gather,
moe router / shared gate) plus an original radix block top-k; int64
row addressing throughout
- moe: non-power-of-2 top-k router, deep-K marlin decode config, one
fp8 scale-bank padding rule shared with the AOT row table
- engine: the PLE table load reserves its pinned bytes from the pin
budget before the expert banks plan their residency