190 Commits
Author SHA1 Message Date
Xiaoze Fan 2fb371e318 feat(kvcache): add fp8_ds_mla and nvfp4_ds_mla kv formats for deepseek-v4 / v4.1 (#920)
* feat(kvcache): add fp8_ds_mla and nvfp4_ds_mla kv formats for deepseek-v4 / v4.1

Window and compressed KV are stored as fp8_ds_mla rows (584 B instead of
1024 B) and indexer keys as MXFP4, holding the same values as the
reference's quant round-trip. DeepSeek-V4.1's packed layout is named
nvfp4_ds_mla and shares the row-format code.

* refactor(engine): drop the model field from the dsv4 kv formats
2026-11-18 22:04:03 -08:00
Mukkesh Ganesh 1ba3e719c1 feat(dflash): load dflash2 gguf drafts for qwen3.8-flash-next (#914)
* feat(dflash): load dflash2 gguf drafts for qwen3.8-flash-next

* fix(dflash): accept mrope sections and reuse the float qkv buffer

* docs(dflash): scope the dequantized parts to quantized ggufs
2026-11-17 16:53:10 -08:00
Chen Cheng 548f97f51e perf(gemma4): read the source layer's kv slab on kv-shared layers (#898)
* perf(gemma4): read the source layer's kv slab on kv-shared layers

* refactor(gemma4): read kv sharing from kv_shared_layers in the module and the draft view

Gemma4Attention takes its source and shared flags from config.kv_shared_layers,
the list the pool maps, and keys shared_kv by the source layer. The draft's
read-only view routes through the same list. A group whose shared layers have
no earlier layer of their type is refused at parse time.

* test(gemma4): give the tiny shared-layer configs an earlier source layer

* test(gemma4): drop the two tests of branches that already fail at load
2026-11-15 16:08:01 -08:00
cherry77-cloud 6036b9c3c8 fix(spec): preserve kv quantization metadata in gemma4 mtp (#911) 2026-11-15 13:45:49 -08:00
Mukkesh Ganesh f2e7f24a0e perf(spec): capture the gemma4 assistant draft in cuda graphs (#903) 2026-11-14 21:53:47 -08:00
Xiaoze Fan 11729f7d7a fix(attention): bound multi-row triton decode tiles by the LDS size on ROCm (#906) 2026-11-14 19:46:51 -08:00
coco 4b71d4d928 fix(tokenizer): render the developer role as system for templates without it (#904) 2026-11-13 23:12:14 -08:00
Chen Cheng 27c891fe99 feat(gguf): serve deepseek-v4.1 gguf checkpoints on deepseek_v41 (#890)
* feat(gguf): serve deepseek-v4.1 gguf checkpoints on deepseek_v41

llama.cpp 10-25 added the deepseek41 layout. The reader takes the config from the file, undoes the converter (engram gate stored as the product, one per_layer_token_embd tensor, hash constants as KVs) and streams the Engram rows from the file as the HF path does.

* fix(gguf): refuse deepseek-v4.1 gguf mismatches at config time

Refuse an FTW of the GGUF before the writer creates the output dir and keep
the side-file hook after the dense loop as on main. Check the Engram hash KVs
and the table in parse_gguf_config, refuse a table with a .scale, and share
the clamp and expert_count readers with the deepseek4 GGUF reader.
2026-11-13 17:39:17 -08:00
ZihaoMu 140faab6b5 perf(attention): add packed fp8 kv reads on gfx1201 (#897)
* perf(attention): add packed fp8 kv reads on gfx1201

* refactor(attention): encapsulate packed fp8 kv readers

* fix(attention): isolate packed fp8 kv reads to amd
2026-11-13 16:17:51 -08:00
LZ_QAQ 66190a937f fix(moe): pad fp8 calibration scale banks like the loader (#902) 2026-11-13 00:17:32 -08:00
Xiaoze Fan 76d2094965 perf(attention): run up to 8 query rows per request in the triton decode kernel (#900)
* perf(attention): run up to 8 query rows per request in the triton decode kernel

Speculative verify (MTP, DFlash) and the MTP draft extend sent q_len > 1 to the extend kernel, which streams the whole context per (request, query head) with no KV split: 6.6-60x the single-row decode time at 32K-128K on sm_120.

- the grouped decode packs query rows x GQA heads in the M axis, limits each row to its own position and window, and keeps the KV split
- the triton backend sends batches of at most 8 rows per request without multimodal blocks to it
- a program's query heads divide the GQA group, which also fixes wrong output for groups above 16 that are not a multiple of 16 (e.g. 48/2)

* fix(attention): keep multi-row triton decode free of register spills on sm_80 to sm_120

- pass maxnreg=255 to multi-row stage-1 launches on CUDA: without .maxnreg ptxas holds these tiles at 80-128 registers for occupancy and spills 8-16 B on sm_80 to sm_120
- pick block_n and warps from a launch sweep under maxnreg, among the variants that do not spill on sm_80, 86, 89, 90, 100 and 120; quantized head_dim 512 takes 16 rows x heads again
2026-11-12 23:28:34 -08:00
Mukkesh Ganesh 58b3ad7ba7 feat(gemma4): speculative decoding with the assistant draft (#895)
* fix(scheduler): return the sliding-window slots when a verify step rolls back

* feat(gemma4): add the assistant draft model and its loader

* feat(spec): serve sliding-window targets with a draft that keeps no KV

* fix(spec): budget verify rows in the swa pool and drop the post-verify sync
2026-11-12 20:34:28 -08:00
Xiaoze Fan 27740fec66 feat(kvcache): add fp8 and int8 per-token-head KV cache formats (#894)
* feat(kvcache): add fp8 and int8 per-token-head KV cache formats

* fix(kernel): keep the bf16 m3 sparse k/v loads inline

* fix(attention): refuse a backend that cannot read the kv format

- filter the m3 page-size fallback by kv format
- release the scale buffer when a qsa or bsa rebuild fails
- keep the bf16 qsa and m3 attend code identical to main
- drop docstrings that claimed no register spills
- drop guard tests

* fix(kernel): widen dsa latent row offsets past int32

* feat(spec): store the mtp draft kv in the target's format
2026-11-11 15:47:50 -08:00
Xiaoze Fan 56e7f3ae38 feat(quant)!: compress bf16 weights to df11 at load with --online-quant (#891)
* feat(quant)!: compress bf16 weights to df11 at load with --online-quant

GLM-4.7 no longer hardwires DF11 attention and embedding; it serves them
bf16 unless --online-quant df11 is passed, and FTW files converted with the
old DF11 tensors must be reconverted.

* fix(quant): skip layers --online-quant cannot convert instead of failing
2026-11-11 13:24:46 -08:00
Xiaoze Fan 96dbe1dd14 feat(spec): add --speculative-token-map for a smaller mtp draft head (#892)
* feat(spec): add --speculative-token-map for a smaller mtp draft head

* feat(spec): add scripts/make_token_map.py to write a token map

* fix(spec): correct the token map vram note and seed its test
2026-11-11 12:15:09 -08:00
coco 0ad521d1a6 fix(server): send an empty signature when a streamed thinking block starts (#889) 2026-11-10 15:22:14 -08:00
Mukkesh Ganesh f69d50686d feat(server): report timings on every streamed chunk when requested (#887) 2026-11-09 21:44:40 -08:00
mkornreich d9184eb9ce feat(gemma4): serve the E2B and E4B releases (#59)
Serve the HF safetensors and GGUF files. The decoder gains per-layer inputs, KV-shared layers and E2B's double-wide MLP.

- --embed-weights host moves the per-layer table to pinned host RAM, which fits these models on an 8 GB GPU.
- The vision tower does not implement clipped linears, so these checkpoints serve text only.
- Each KV-shared layer still stores its own KV copy.
2026-11-09 20:22:26 -08:00
Xiaoze Fan fff1022a5c fix(quant): accept the modelopt_mixed quant method (#886) 2026-11-09 11:25:16 -08:00
Xiaoze Fan 5a73685b1a chore(release): 0.2.2 (#880) 2026-11-08 17:53:16 -08:00
Xiaoze Fan ef6b303304 fix(ggml): cap the MoE route sort's shared memory at the device limit (#878) 2026-11-07 23:30:06 -08:00
Xiaoze Fan 340db81ec0 fix(engine): send graph-capture KV writes to the dummy page (#877)
* fix(engine): send graph-capture KV writes to the dummy page

* test(engine): check graph capture runs on the dummy page
2026-11-07 16:55:58 -08:00
Saurabh Pathak 8215a94f92 fix(server): reject empty completions prompt and keep error code (#767) (#850) 2026-11-07 13:36:26 -08:00
Xiaoze Fan 08cee44468 feat(spec): add DFlash 2 speculative decoding (#857)
* feat(spec): hand the pending token to Proposer.propose and return a SpecProposal

* feat(engine): Sampler.probs and sparse-q rejection

* feat(spec): let a proposer declare slot states riding the GDN slots

* feat(models): export layer outputs to the feature sink in the qwen3_5 and qwen4_exp families

* feat(attention): record the source slot of each tracked request

* feat(kernel): DFlash2 draft kernels

* feat(models): DFlash2 draft family

* feat(spec): DFlashWindow

* feat(spec): DFlashProposer

* docs(cli): add dflash to the speculative decoding section

* refactor(engine): share the sampling backend choice and drop the extra renorm in Sampler.probs

* fix(models): build the DFlash rope table uncached and refuse draft scalings it does not apply

* refactor(spec): share the draft path check between the MTP and DFlash proposers

* test(e2e): compare rebuilt outputs exactly only without hybrid MoE

* refactor(spec): keep proposer slot states on EngineConfig instead of replacing model_config

* perf(spec): compute the verify distribution once for sparse-q rejection

* perf(models): let the feature sink add the residual on the rows it keeps

* perf(models): share the DFlash draft K/V weights with the ring projection

* fix(kernel): keep the dflash selector off zero-probability candidates

* test(spec): drop the DFlash tests that the draft forward and Monte Carlo tests already cover

* fix(spec): default and cap DFlash steps at the draft's block_size - 1

* fix(models): reject DFlash draft configs the draft forward cannot serve

* test(e2e): read the MoE strategy from the rebuild test's extra args

* docs(spec): say that slot-state owners write their own prefill track snapshot

* test(models): move the DFlash draft parity tests under tests/models/dflash

* refactor(spec): drop the no-op verify handler of the DFlash window

* refactor(models): fold the DFlash candidate top-k into propose

* refactor(spec): declare the draft step limits through Proposer.num_steps only

* refactor(kernel): make the DFlash ring attention stateless and keep its workspace on the ring

* refactor(models): name the DFlash draft submodules like the other model families

* refactor(spec): move SpecProposal next to DraftContext in spec/proposer.py

* refactor(spec): hand the verify the draft distribution over the whole vocabulary

* refactor(spec): call the DFlashRing object dflash_ring and its length ring_len

* refactor(models): split the DFlash draft's propose into compute_candidates and sample_path

* refactor(spec): keep the draft distribution on its support in SpecProposal

* fix(spec): drop the step's proposal once the verify has read it

* refactor(spec): name the DFlash feature sinks DFlashPrefillSink and DFlashVerifySink

* refactor(models): read the declared Batch.features directly instead of through getattr

* test(dflash): check the window rebuild inside the block forward test

* test(spec): share the plain run of the verify contract and run the DFlash draft once per family

* test(kvcache): drop the window cases from the pool tests and check a layerless state's bytes

* test(spec): check the verify probs of each row against a torch top-k and top-p reference

* test(e2e): add an accept-len floor and a prefill chunk knob to the AIME gate

* refactor(scheduler): let callers take the speculative counts from the status reporter

* test(spec): keep the shared plain runs of the verify contract in their own temp dirs

* test(models): give the ricursive test's stand-in batch a features field
2026-11-07 12:29:18 -08:00
Chen Cheng 14119b8cbb feat(ricursive): add ripple-65b preview support (#871)
* feat(ricursive): add ripple-65b preview support

* fix(ricursive): read swiglu_clamp as a per-layer gguf array

llama.cpp's Ripple converter (ggml-org/llama.cpp#31692) writes
ricursive.swiglu_clamp with one value per layer, while the first
community GGUFs store a scalar. Accept both and refuse per-layer
values that differ, since the model has a single clamp.

* fix(ricursive): add ripple's own parsers and copy the dense reader
2026-11-06 16:17:21 -08:00
Chen Cheng 5209c8c327 perf(cpu-moe): give only the routes the cpu computes work items in a hybrid step (#866) 2026-11-06 11:36:59 -08:00
Xiaoze Fan e5856ac471 fix(utils): read stop ids from config.json without generation_config.json (#874) 2026-11-05 19:21:56 -08:00
Xiaoze Fan 60059b41fb chore(ggml): replace freetoken.patch with the llama.cpp pin (#873)
The patch repeats every edit to a copied file and nothing checks that the two agree. The pin and the tree reproduce it. scripts/sync-llama-cpp.sh diffs the copy against the pin and merges a newer llama.cpp from it, and docs/ggml-kernels.md says how to maintain the copy.
2026-11-05 15:13:56 -08:00
Xiaoze Fan 650dd6f6fe perf(ggml): sort MoE MMQ routes with a counting sort (#872)
* perf(ggml): sort MoE MMQ routes with a counting sort

* perf(ggml): fold the MoE MMQ tile map into the route sort
2026-11-05 13:16:03 -08:00
Xiaoze Fan 4503c424ee feat(server): add continuous_usage_stats and count shell tok/s from it (#868)
* feat(server): report running usage on stream chunks with continuous_usage_stats

* fix(shell): count tokens instead of stream chunks in the live tok/s

* refactor(server): build both openai stream usages from GenUsage

* fix(shell): reset the first-delta count per turn and ignore zero running totals
2026-11-04 22:58:51 -08:00
Mukkesh Ganesh dac018e90b fix(gguf): log why mmproj auto-discovery skips a file (#860) 2026-11-04 14:17:02 -08:00
LZ_QAQ 628b88ff79 fix(attention): avoid overflow in triton kv cache offsets (#854)
* fix(attention): avoid overflow in triton kv cache offsets

* fix(attention): simplify kv offset dtype selection
2026-11-04 00:09:40 -08:00
Mukkesh Ganesh 9ee2e5c53f feat(gguf): load mtp drafts for the qwen3.5 family from nextn blocks and mtp-*.gguf sidecars (#856) 2026-11-03 20:27:21 -08:00
wenlive d34b283f0a fix(sampling): bound top-k before int32 conversion (#790) 2026-11-03 18:44:50 -08:00
Chen Cheng e6e079cd04 fix(scheduler): stop the swa prefill commit from freeing pages the request still reads (#651) 2026-11-03 17:52:09 -08:00
Xiaoze Fan 33cbb3c54e feat(utils): add once-per-message logger methods (#858)
debug_once / info_once / warning_once log on rank 0, once per distinct message and args. select_kernel, which logged its choice once per layer, and the hand-rolled log-once guards now use them.
2026-11-03 15:31:08 -08:00
Mukkesh Ganesh 03e5ea0672 feat(server): report per-request prefill and decode timings in api responses (#814)
* feat(server): report per-request prefill and decode timings in api responses

* fix(server): take timings spans from the scheduler and gate them behind --enable-timings-report

* fix(scheduler): count accepted drafts as min(n, emitted) and stamp reqstats with perf_counter
2026-11-03 14:44:25 -08:00
ZihaoMuandLZ-QWQ 9cacd0896c fix(moe): publish rocm cpu expert inputs before notification (#764)
* fix(moe): prevent stale cpu expert inputs in rocm graphs

* fix(moe): publish rocm cpu inputs with graph events

* fix(moe): qualify rocm kernel io with ordered copy fallback

* fix(moe): enable locked hybrid staging on rocm

---------

Co-authored-by: LZ-QWQ <lz1830837889@163.com>
2026-11-02 21:29:42 -08:00
Chen Cheng d026ffd3b9 feat(gguf): read nvfp4 weight scales on qwen4exp ggufs (#845) 2026-11-02 19:49:03 -08:00
Xiaoze Fan 3d23551c5b perf(cpu-moe): speed up the nvfp4 W4A8 kernels (#851)
The VNNI dots were compute bound per core; with fewer instructions per weight they reach the DRAM read bandwidth.
The W4A8 kernel now follows FREETOKEN_CPU_MOE_ISA.
2026-11-02 15:05:17 -08:00
Xiaoze Fan dacac5f5fa feat(spec): add MTP drafts for the Qwen3.5 family and Qwen3.8-Flash-Next (#844)
* feat(spec): add the MTP proposer for the qwen3_5 family

* test(models): read the qwen3_5 MTP head of bf16 and block-fp8 checkpoints

* feat(spec): verify qwen4_exp targets on qsa_sparse graphs with PLE verify handlers

* feat(spec): add the qwen4_exp MTP draft from the checkpoint or a Strata pack

* docs(cli): document the speculative decoding flags

* fix(spec): refuse MTP draft sources the draft cannot read before the target loads

* docs(qwen4_exp): describe the verify path in the PLE docstrings

* fix(spec): refuse a missing relative MTP draft path before the target loads

* fix(moe): read MTP head experts on the host and require both stacked tensors

* refactor(spec): tell the final prefill chunk from prompt_len instead of the scheduler's ChunkedReq

* refactor(spec): let each MTP draft family check its own weight source

* fix(qwen4_exp): size the disk PLE eager staging for decode and verify rows

* perf(kernel): read the KV-index gather's chunk count from the grid, not a constexpr

* refactor(moe): let each family name its MTP expert tensors

* refactor(spec): force the MTP draft's MoE resident in the proposer and check it after load

* fix(spec): refuse MTP with dummy weights

* refactor(spec): move HiddenRelay under spec/mtp, its only user

* refactor(spec): drop MTPDraft.config, fold its embed hooks into embed_tokens

* refactor(models): note FeatureSink's graph binding, drop its runtime_checkable

* fix(models): read quantized MTP fc projections under the quant config
2026-11-02 11:37:09 -08:00
Mukkesh Ganesh ea2916d31c feat(gguf): serve gpt-oss gguf checkpoints on gpt_oss (#838)
* feat(gguf): serve gpt-oss gguf checkpoints on gpt_oss

* fix(gguf): refuse missing gpt-oss bias tensors, honor swa pattern, scale nvfp4 before bias

* fix(gguf): honor gpt-oss yarn truncate, gate the bf16 float cast, scale nvfp4 before the down bias
2026-11-02 08:58:07 -08:00
A5bhinav ee43b2a113 fix(server): refuse n != 1 on /v1/completions instead of ignoring it (#842) 2026-10-29 17:15:32 -07:00
Chen Cheng ecbbce4577 fix(moe): skip inactive gguf routes in hybrid decode (#823)
* fix(moe): skip inactive gguf routes in hybrid decode

* perf(moe): run the inactive-route memset and zeroing only in hybrid decode

Only hybrid decode hands the GGUF kernel -1 routes, so mul_mat_id takes an inactive flag
and MMQ does the ids_src1 memset and the zeroing pass only when it is set. Prefill and
fused / offload decode go back to the main-branch MMQ calls.

* perf(moe): compare expert ids unsigned in mm_ids_helper

A negative (inactive) id compared unsigned sorts after every expert and equals none, so the
helper needs no remap to INT_MAX; the remap made nvcc unroll its token loop half as far.
2026-10-28 22:28:13 -07:00
Xiaoze Fan c2c3e828e8 feat(spec): verify draft tokens on the target side (#836)
* feat(spec): add the speculative verify infrastructure

* feat(scheduler): run decode steps as speculative verify steps

* chore(engine): mark the speculative decoding config fields with one comment

* feat(server): add short aliases for the speculative decoding flags
2026-10-28 21:18:02 -07:00
Chen Cheng d2977fbc13 feat(kernel): add an iq1_m mmq kernel for moe steps above 8 tokens (#835) 2026-10-28 20:48:55 -07:00
festeraeb c94535c909 fix(attention): size the decode KV tile to the shared-memory budget (#840)
Assisted-by: Claude (Anthropic)
2026-10-28 20:05:27 -07:00
Xiaoze Fan d043fb61f8 refactor(layers): share one GatedDeltaNet op in layers/linear_attn (#841)
* refactor(layers): share one GatedDeltaNet op in layers/linear_attn

* test(layers): restore the global context in the GDN test and fix stale comments
2026-10-28 17:22:50 -07:00
Xiaoze Fan a2ce636083 refactor(attention): move linear-attention metadata into a LinearAttnBackend (#837)
* refactor(attention): move linear-attention metadata into a LinearAttnBackend

* refactor(attention): make LinearAttnBackend a BaseAttnBackend with FLACaptureData

* refactor(attention): key every linear-attention row by fla_metadata.cache_indices
2026-10-27 23:56:59 -07:00
Xiaoze Fan e328cf6381 refactor: add engine and scheduler seams for multi-row decode steps (#833)
* refactor: add engine and scheduler seams for multi-row decode steps

* refactor: rename the graph steps argument to rows_per_req

* refactor: drop VerifyInfo and require rows_per_req on verify batches

* refactor: fix stale seam comments and read decode_rows_per_req directly

* refactor: reword two GraphRunner comments for multi-row graphs

* fix(engine): scale the GPU-decode MoE pool limits by decode_rows_per_req

* refactor: drop the unread Context.moe_offload_cache field

* test: trim the multi-row seam tests
2026-10-27 16:03:01 -07:00