* feat(kvcache): add fp8_ds_mla and nvfp4_ds_mla kv formats for deepseek-v4 / v4.1
Window and compressed KV are stored as fp8_ds_mla rows (584 B instead of
1024 B) and indexer keys as MXFP4, holding the same values as the
reference's quant round-trip. DeepSeek-V4.1's packed layout is named
nvfp4_ds_mla and shares the row-format code.
* refactor(engine): drop the model field from the dsv4 kv formats
* feat(gguf): serve deepseek-v4.1 gguf checkpoints on deepseek_v41
llama.cpp 10-25 added the deepseek41 layout. The reader takes the config from the file, undoes the converter (engram gate stored as the product, one per_layer_token_embd tensor, hash constants as KVs) and streams the Engram rows from the file as the HF path does.
* fix(gguf): refuse deepseek-v4.1 gguf mismatches at config time
Refuse an FTW of the GGUF before the writer creates the output dir and keep
the side-file hook after the dense loop as on main. Check the Engram hash KVs
and the table in parse_gguf_config, refuse a table with a .scale, and share
the clamp and expert_count readers with the deepseek4 GGUF reader.
* feat(ricursive): add ripple-65b preview support
* fix(ricursive): read swiglu_clamp as a per-layer gguf array
llama.cpp's Ripple converter (ggml-org/llama.cpp#31692) writes
ricursive.swiglu_clamp with one value per layer, while the first
community GGUFs store a scalar. Accept both and refuse per-layer
values that differ, since the model has a single clamp.
* fix(ricursive): add ripple's own parsers and copy the dense reader
* feat(gguf): load the vision tower from a llama.cpp mmproj
Qwen3.5 / Qwen3.6 / Qwen3.8 (qwen3vl_merger) and Gemma-4 26B-A4B / 31B (gemma4v) / 12B (gemma4uv) take image input from a separate mmproj file: --mmproj, else one in the model's own folder. The tower runs in bf16 (quantized mmproj files are dequantized) and its config comes from the GGUF metadata plus per-arch defaults.
* docs(gguf): describe mmproj image input without a model list
* test(gguf): drop the mmproj tests
* fix(gguf): escape the model folder when globbing for mmproj files
* fix(gguf): skip auto-discovered mmproj files the text model cannot use
* fix(gguf): serve text from a GGUF that carries vision tensors itself
* feat(gguf): serve DeepSeek-V4-Flash GGUF checkpoints on deepseek4
* fix(gguf): refuse reap-style per-layer expert prunes in deepseek4 ggufs
* fix(gguf): apply the deepseek4 review fixes
- route the offload experts through swiglu_clamp so the GGUF MoE path applies the limit
- map model.embed -> embed on the HF row so the bf16 exemption matches
- default a wholly missing rope.scaling block to the family yarn values with a warning;
refuse a partly present one
* feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next
Adds the llama.cpp glm5-next GGUF reader (unsloth GLM-5.3-Flash-GGUF, both
the glm5-next and the older glm5next arch spellings) on the GGUF quant
kind from #690:
- parse_gguf_config derives the text config from the GGUF KV metadata,
with per-block attention.head_count_kv laying out the KDA/DSA hybrid,
the trailing MTP block dropped like the HF loader drops it, and the
MLA head widths from the *_length_mla keys.
- iter_gguf_weights maps every tensor to the glm5_next state dict and
undoes the converter transforms: A_log stored as -exp(A_log) is
inverted, the per-head attn_k_b/attn_v_b pair (k transposed) fuses
into the dense kv_b_proj layout the MLA absorption reads, and the
q|k|v conv kernels concatenate on the channel axis. The mixed-type
fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a) rides #702 segments;
the routed experts load as GGUF expert banks at their shipped types
(unsloth dynamic quants: IQ3_S gate/up + IQ4_XS down, Q8_0 outlier
layers pass through natively).
- passes quant_config to embed_tokens so the Q8_0 token_embd serves
packed; registers Glm5NextGGUFForCausalLM and claims it in the AOT
table.
Verified end to end against unsloth GLM-5.3-Flash-UD-IQ4_XS (5 split
shards): the full engine on a 32GB GPU serves coherent output with the
experts off host banks. Text-only, TP=1.
Part of #671.
* fix(gguf): refuse unread scale tensors in glm5-next ggufs
check_scales at config parse like the llama, qwen3 and gemma4 loaders
(review on #721): a .scale / .input_scale outside an NVFP4 weight now
fails with a clear error instead of loading silently unread.
* fix(gguf): serve the quantized tensors of unsloth dynamic glm5-next quants
unsloth's UD quants quantize tensors the AesSedAI-style files keep float:
the MLA kv_b halves (Q8_0), the mHC mixers and the indexer's wk /
weights_proj / compressor gate. kv_b_proj joins the registry's
unquantized_modules (glm_moe_dsa keeps its kv_b dense for the MLA
absorption for the same reason) and the reader dequantizes through the
vendored gguf-py reference on every float-expecting read; wq_b keeps
serving its checkpoint quant packed (attention.py keeps wk and
weights_proj dense). Regression test stores the kv_b halves Q8_0.
* fix(gguf): treat the glm5 pre-tokenizer spelling like glm4
llama-vocab.cpp maps both spellings to CHATGLM4 with ignore_merges;
unsloth writes glm4 and AesSedAI's writer glm5 for the same vocabulary.
Without the mapping a glm5 file loads but silently mis-tokenizes
(tok_compare: 22,840/153,725 vocab words differ from the HF reference;
0 everywhere with it).
* fix(gguf): size the mmq tail padding by ggml type and k
pad_rows added one zero row of k columns, which covers the bytes MMQ
reads past the last row only when k is at least half a padding unit
(512); glm5-next's KDA f_b / g_b (k = linear head dim 128, Q8_0 in
dynamic quants) are the first packed tensors below that and died at the
first mul_mat. pad_rows now takes the ggml type and returns enough whole
rows to cover mmq_tail_bytes for any (type, k), consistently in the
reader's fused(), the gguf linear and embedding params and the expert
host banks. Found serving the real unsloth UD-IQ4_XS: the dry load
never runs a forward.
* fix(gguf): take the older glm5-next writer variants in stride
Two long-tail writer variants the full check sweep found (the GSQ-RCO
lineage alone is ~57k downloads/month):
- the conv kernels with a trailing unit dim: llama.cpp's glm5-next
loader tries that 4-D shape before the 3-D one, so trailing unit dims
in ne are dropped on float reads (they carry no elements).
- attention.head_count_kv as the generic scalar (all blocks): wrong for
the hybrid -- llama.cpp broadcasts it and then fails tensor creation
-- so only the list is trusted and a scalar falls to the tensor-name
census like the pre-list writer.
* fix(gguf): glm5-next review cleanups
- refuse instead of guessing metadata the model cannot express: the
scalar attention.head_count_kv spelling (llama.cpp broadcasts it and
then fails tensor creation; every public file carries the list),
missing *_length_mla keys through need(), non-uniform
swiglu_clamp_exp / swiglu_clamp_shexp, indexer.types with non-full
layers, expert_gating_func other than sigmoid, and rope dims as a
NotImplementedError instead of an assert.
- call check_scales directly; aot_models comment on its own line.
- drop the GGUF test files; coverage will be handled across families.
NVFP4 weight global scales (amax / 2688, 2e-5 to 4e-4 in the served checkpoints) sit at fp16's
normal minimum; stored as fp16, up to 97% of a checkpoint's expert globals were subnormal, with
up to 1.7e-3 relative error per row.
BREAKING CHANGE: an FTW checkpoint that stores NVFP4 global scales as fp16 is refused at load;
reconvert it with `ft checkpoint`.
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding
* build(deps): vendor the gguf-py reader
* feat(quant): add the GGUF quant kind and an embedding quant method
* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path
* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA
* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16
* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small
* fix(gguf): read F32, F16 and BF16 PLE tables without ggml
* fix(moe): pad the offload slot cache for ggml MMQ tail reads
* fix(gguf): support hip builds and runtime devices
* fix(gguf): copy same-type dequantize and check for missing ggml converters
* fix(gguf): load quantized qwen4exp ple_key and ple_value
* feat(gguf): run float parts of a fused GGUF linear as bf16 segments
* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights
* fix(moe): zero the prefill staging area after the last prefill layer
---------
Co-authored-by: zihaomu <zihaomu@outlook.com>
* test(deepseek-v41): add independent text and vision reference
* feat(quant): support block-32 and unrounded inference outputs
* feat(attention): add csa2 packed cache and sparse backend
* feat(deepseek-v41): serve text and images through shared runtime
* perf(mhc): tune batch-stable reduction tiles
* perf(moe): autotune fp4 decode launch geometry
* perf(moe): admit recurring misses against actual lru victims
* revert: perf(moe): admit recurring misses against actual lru victims
This reverts commit 1652d200f0ebb825379cf44c28baf52971e3a461.
* perf(mhc): derive the reduction tile from the width, not a timing
* perf(moe): replace the ds_fp4 decode autotune with a fixed two-warp launch
* fix(deepseek-v41): fence engram staging against in-flight lookups
* fix(moe): return the cpu routed sum in fp32 and round it in python
* fix(deepseek-v41): keep the vision tower and aligner unquantized
* fix(scheduler): admit bounded-replay hits by the history their resume reads
* refactor(deepseek-v41): name the attention family after the model, not csa2
* refactor(kvcache): import the dsv41 pool only for dsv41 models
* refactor(deepseek-v41): name the replay switch --swa-decoder-replay
* fix(server): keep reasoning_effort a string on the wire
* refactor(models): keep the prefill replay hooks off BaseLLMModel
* refactor(deepseek-v41): drop the fp32 routed sum and batch-invariant prefill
* fix(moe): weight ds_fp4 routes in the down epilogue again
* docs(models): anchor the v4.1 replay paragraph on the tech report
* refactor(deepseek-v41): run the decoder source's narrowed prefill on Block
* refactor(kvcache): let the dsv41 pool own the history a resume keeps live
* refactor(deepseek-v41): resume bounded-replay hits by capping the prefix match
* refactor(deepseek-v41): describe the attention group as a dsv4 variant and derive its geometry from the args
* refactor(deepseek-v41): keep the engram table locator in the model package
* refactor(deepseek-v41): drop the fp32 lm head output
- return the head's bf16 logits, as DeepSeek-V4 does
- revert ParallelLMHead(out_dtype=) and LinearKernel.apply_out_dtype, which only this call used
- remove test_head_preserves_fp32_logit_margin
- cast test_vision's prefill logits to fp32 before comparing, as the harness does
* refactor(quant): carry the w4a8 activation block on MoEConfig
- drop ActDesc and QuantScheme.act; the scheme only describes stored tensors
- Fp8BlockConfig sets MoEConfig.act_block from weight_block_size for its fp4 experts via a layer_config hook
- the ds_fp4 GPU kernel and CPU executor read the block from the layer config and require it
* refactor(deepseek-v41): size the kv pool from the args, not a geometry copy
- delete kvcache/dsv41_geometry.py; the pool and cost model read dsv41_args by attribute, as the DSV4 pool reads dsv4_args
- source / ratio aliasing reads args.roles; the duplicated source and ratio validation is gone
- the cost model owns the tier row formats, row bytes and the private window layers
- the pool keeps the format and window/ratio checks, and exposes the formats and is_private_window to the attention backend
* refactor(kvcache): move the dsv4 pool into kvcache/dsv4
* refactor(kvcache): move the dsv41 pool into kvcache/dsv4
* chore(benchmarks): drop the dsv41 selection and fp8 linear benchmarks
* test(deepseek-v41): drop tests that repeat other coverage
* test(deepseek-v41): load the tiny model's resident experts as banks
* fix(server): parse deepseek-v4.1's space-led dsml tool calls
---------
Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
* feat(mm): serve image input on the Qwen VL families
- frontend: OpenAI image_url, Anthropic image blocks and Responses
input_image (http(s), base64, gated file://) become MMItems in the
tokenizer workers; image spans carry content pseudo ids so the radix
cache keys them by content; mrope positions are precomputed per request
- registry: EncoderSpec per family (Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next
register a vision encoder serving image) and ModelSpec.mm_processor;
EngineConfig.active_encoders decides what a process builds,
served_modalities what it accepts; --text-model-only, --mm-disable,
--mm-encoder-weights, --mm-embed-cache-device, --mm-max-pixels
- engine and scheduler: per-request encoder-cache row ownership, chunk
planning on the batch (jobs, gather rows, scatter rows), items encoded
right before the LM forward and gathered into batch.mm_embeds
- models: the Qwen VL vision tower on the text tower's TP layers with
DeepStack, block weights streamed from pinned host banks by default,
merger outputs parked on the host between blocks; SupportsMultimodal
(encode, place_encoder_weights) and a shared embed_input_ids
- server: image inputs rejected with a client-facing reason when no
vision encoder is served; GET /v1/stats reports model.input_modalities
* feat(mm): replace the pixel cap with a token budget and processor kwargs
- --image-min-tokens / --image-max-tokens replace --mm-max-pixels; MMProcessor.get_mm_processor_kwargs turns the MultimodalConfig into the processor call's arguments, the default passes mm.processor_kwargs through and Qwen VL converts the budget to pixel areas in size.shortest_edge / longest_edge
- --mm-processor-kwargs hands a JSON object to the checkpoint's image processor call, applied after the token budget
- MMProcessor is an ABC that owns the model path, the config and the lazily loaded image processor; get_mm_processor returns None when --mm-disable leaves no encoder to serve
- the frontend tokenizer, the tokenizer worker and the offline LLM build one processor each and hand it to TokenizeManager, so the tokenize message carries no per-request pixel cap
* refactor(quant): config, scheme and method layers for checkpoint quantization
QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.
- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
Hybrid 45-layer model (34 KDA + 11 DSA) with NVFP4 MoE (288 experts):
- KDA linear-attention kernels and op, hybrid-linear pool dispatch
generalized to any paged family
- NoPE-MLA attention with a kpool DSA indexer backend: 64-token pages,
1/ratio shadow index slab, per-request tail rings, CUDA-graph-safe
decode, prefix-cache snapshot contract
- mHC (x4 residual streams) layer with a fused split-K triton kernel;
torch reference kept as the semantic pin
- clamped-SwiGLU activation across the MoE stack
- weight loading for NVFP4 exports in the multimodal-wrapper layout:
ModelOpt and compressed-tensors (RedHatAI) tensor kinds;
Co-authored-by: Shuo Yang <73746844+andy-yang-1@users.noreply.github.com>
* fix(kernel): move fp8_block_scale_pad into aot_models to unbreak the kernel-cache build
* fix(kernel): exclude bank rows fast_index_copy cannot compile from aot specs
* fix(moe): fail fast when fused copy is off and a bank row cannot fall back
Serve Qwen3.8-Flash-Next (HF model_type qwen4_exp) text-only: 36 GDN +
12 QSA compressed-sparse attention layers on 4 hyper-connection residual
streams, a PLE n-gram embedding layer backed by a 47.7 GiB pinned-host
table with UVA gather, and 512 NVFP4 / block-fp8 routed experts (top-10)
plus a gated shared expert.
- attention: qsa_sparse backend (AttnType.QSA) over QSAKVCache -- paged
GQA K/V, a 1/ratio compressed index-key slab shadowing the KV pages,
and a per-request pending ring sized from index_ratio
- kvcache: declarative slot-sibling states (ModelConfig.slot_states) on
LinearStatePool carry the PLE conv history and n-gram context through
the hybrid-radix snapshot/COW lifecycle
- scheduler: hybrid prefill chunks align to the page size so snapshots
land on donatable boundaries
- kernels: triton kernels adapted from vLLM/SGLang (hc, qsa, ple gather,
moe router / shared gate) plus an original radix block top-k; int64
row addressing throughout
- moe: non-power-of-2 top-k router, deep-K marlin decode config, one
fp8 scale-bank padding rule shared with the AOT row table
- engine: the PLE table load reserves its pinned bytes from the pin
budget before the expert banks plan their residency