Commit Graph
172 Commits
Author SHA1 Message Date
Xiaoze Fan 42553c5463 fix(quant): accept the modelopt_mixed quant method 2026-11-09 19:24:14 +00:00
Xiaoze Fan 5a73685b1a chore(release): 0.2.2 (#880) 2026-11-08 17:53:16 -08:00
Xiaoze Fan ef6b303304 fix(ggml): cap the MoE route sort's shared memory at the device limit (#878) 2026-11-07 23:30:06 -08:00
Xiaoze Fan 340db81ec0 fix(engine): send graph-capture KV writes to the dummy page (#877)
* fix(engine): send graph-capture KV writes to the dummy page

* test(engine): check graph capture runs on the dummy page
2026-11-07 16:55:58 -08:00
Saurabh Pathak 8215a94f92 fix(server): reject empty completions prompt and keep error code (#767) (#850) 2026-11-07 13:36:26 -08:00
Xiaoze Fan 08cee44468 feat(spec): add DFlash 2 speculative decoding (#857)
* feat(spec): hand the pending token to Proposer.propose and return a SpecProposal

* feat(engine): Sampler.probs and sparse-q rejection

* feat(spec): let a proposer declare slot states riding the GDN slots

* feat(models): export layer outputs to the feature sink in the qwen3_5 and qwen4_exp families

* feat(attention): record the source slot of each tracked request

* feat(kernel): DFlash2 draft kernels

* feat(models): DFlash2 draft family

* feat(spec): DFlashWindow

* feat(spec): DFlashProposer

* docs(cli): add dflash to the speculative decoding section

* refactor(engine): share the sampling backend choice and drop the extra renorm in Sampler.probs

* fix(models): build the DFlash rope table uncached and refuse draft scalings it does not apply

* refactor(spec): share the draft path check between the MTP and DFlash proposers

* test(e2e): compare rebuilt outputs exactly only without hybrid MoE

* refactor(spec): keep proposer slot states on EngineConfig instead of replacing model_config

* perf(spec): compute the verify distribution once for sparse-q rejection

* perf(models): let the feature sink add the residual on the rows it keeps

* perf(models): share the DFlash draft K/V weights with the ring projection

* fix(kernel): keep the dflash selector off zero-probability candidates

* test(spec): drop the DFlash tests that the draft forward and Monte Carlo tests already cover

* fix(spec): default and cap DFlash steps at the draft's block_size - 1

* fix(models): reject DFlash draft configs the draft forward cannot serve

* test(e2e): read the MoE strategy from the rebuild test's extra args

* docs(spec): say that slot-state owners write their own prefill track snapshot

* test(models): move the DFlash draft parity tests under tests/models/dflash

* refactor(spec): drop the no-op verify handler of the DFlash window

* refactor(models): fold the DFlash candidate top-k into propose

* refactor(spec): declare the draft step limits through Proposer.num_steps only

* refactor(kernel): make the DFlash ring attention stateless and keep its workspace on the ring

* refactor(models): name the DFlash draft submodules like the other model families

* refactor(spec): move SpecProposal next to DraftContext in spec/proposer.py

* refactor(spec): hand the verify the draft distribution over the whole vocabulary

* refactor(spec): call the DFlashRing object dflash_ring and its length ring_len

* refactor(models): split the DFlash draft's propose into compute_candidates and sample_path

* refactor(spec): keep the draft distribution on its support in SpecProposal

* fix(spec): drop the step's proposal once the verify has read it

* refactor(spec): name the DFlash feature sinks DFlashPrefillSink and DFlashVerifySink

* refactor(models): read the declared Batch.features directly instead of through getattr

* test(dflash): check the window rebuild inside the block forward test

* test(spec): share the plain run of the verify contract and run the DFlash draft once per family

* test(kvcache): drop the window cases from the pool tests and check a layerless state's bytes

* test(spec): check the verify probs of each row against a torch top-k and top-p reference

* test(e2e): add an accept-len floor and a prefill chunk knob to the AIME gate

* refactor(scheduler): let callers take the speculative counts from the status reporter

* test(spec): keep the shared plain runs of the verify contract in their own temp dirs

* test(models): give the ricursive test's stand-in batch a features field
2026-11-07 12:29:18 -08:00
Chen Cheng 14119b8cbb feat(ricursive): add ripple-65b preview support (#871)
* feat(ricursive): add ripple-65b preview support

* fix(ricursive): read swiglu_clamp as a per-layer gguf array

llama.cpp's Ripple converter (ggml-org/llama.cpp#31692) writes
ricursive.swiglu_clamp with one value per layer, while the first
community GGUFs store a scalar. Accept both and refuse per-layer
values that differ, since the model has a single clamp.

* fix(ricursive): add ripple's own parsers and copy the dense reader
2026-11-06 16:17:21 -08:00
Chen Cheng 5209c8c327 perf(cpu-moe): give only the routes the cpu computes work items in a hybrid step (#866) 2026-11-06 11:36:59 -08:00
Xiaoze Fan e5856ac471 fix(utils): read stop ids from config.json without generation_config.json (#874) 2026-11-05 19:21:56 -08:00
Xiaoze Fan 60059b41fb chore(ggml): replace freetoken.patch with the llama.cpp pin (#873)
The patch repeats every edit to a copied file and nothing checks that the two agree. The pin and the tree reproduce it. scripts/sync-llama-cpp.sh diffs the copy against the pin and merges a newer llama.cpp from it, and docs/ggml-kernels.md says how to maintain the copy.
2026-11-05 15:13:56 -08:00
Xiaoze Fan 650dd6f6fe perf(ggml): sort MoE MMQ routes with a counting sort (#872)
* perf(ggml): sort MoE MMQ routes with a counting sort

* perf(ggml): fold the MoE MMQ tile map into the route sort
2026-11-05 13:16:03 -08:00
Xiaoze Fan 4503c424ee feat(server): add continuous_usage_stats and count shell tok/s from it (#868)
* feat(server): report running usage on stream chunks with continuous_usage_stats

* fix(shell): count tokens instead of stream chunks in the live tok/s

* refactor(server): build both openai stream usages from GenUsage

* fix(shell): reset the first-delta count per turn and ignore zero running totals
2026-11-04 22:58:51 -08:00
Mukkesh Ganesh dac018e90b fix(gguf): log why mmproj auto-discovery skips a file (#860) 2026-11-04 14:17:02 -08:00
LZ_QAQ 628b88ff79 fix(attention): avoid overflow in triton kv cache offsets (#854)
* fix(attention): avoid overflow in triton kv cache offsets

* fix(attention): simplify kv offset dtype selection
2026-11-04 00:09:40 -08:00
Mukkesh Ganesh 9ee2e5c53f feat(gguf): load mtp drafts for the qwen3.5 family from nextn blocks and mtp-*.gguf sidecars (#856) 2026-11-03 20:27:21 -08:00
wenlive d34b283f0a fix(sampling): bound top-k before int32 conversion (#790) 2026-11-03 18:44:50 -08:00
Chen Cheng e6e079cd04 fix(scheduler): stop the swa prefill commit from freeing pages the request still reads (#651) 2026-11-03 17:52:09 -08:00
Xiaoze Fan 33cbb3c54e feat(utils): add once-per-message logger methods (#858)
debug_once / info_once / warning_once log on rank 0, once per distinct message and args. select_kernel, which logged its choice once per layer, and the hand-rolled log-once guards now use them.
2026-11-03 15:31:08 -08:00
Mukkesh Ganesh 03e5ea0672 feat(server): report per-request prefill and decode timings in api responses (#814)
* feat(server): report per-request prefill and decode timings in api responses

* fix(server): take timings spans from the scheduler and gate them behind --enable-timings-report

* fix(scheduler): count accepted drafts as min(n, emitted) and stamp reqstats with perf_counter
2026-11-03 14:44:25 -08:00
ZihaoMuandLZ-QWQ 9cacd0896c fix(moe): publish rocm cpu expert inputs before notification (#764)
* fix(moe): prevent stale cpu expert inputs in rocm graphs

* fix(moe): publish rocm cpu inputs with graph events

* fix(moe): qualify rocm kernel io with ordered copy fallback

* fix(moe): enable locked hybrid staging on rocm

---------

Co-authored-by: LZ-QWQ <lz1830837889@163.com>
2026-11-02 21:29:42 -08:00
Chen Cheng d026ffd3b9 feat(gguf): read nvfp4 weight scales on qwen4exp ggufs (#845) 2026-11-02 19:49:03 -08:00
Xiaoze Fan 3d23551c5b perf(cpu-moe): speed up the nvfp4 W4A8 kernels (#851)
The VNNI dots were compute bound per core; with fewer instructions per weight they reach the DRAM read bandwidth.
The W4A8 kernel now follows FREETOKEN_CPU_MOE_ISA.
2026-11-02 15:05:17 -08:00
Xiaoze Fan dacac5f5fa feat(spec): add MTP drafts for the Qwen3.5 family and Qwen3.8-Flash-Next (#844)
* feat(spec): add the MTP proposer for the qwen3_5 family

* test(models): read the qwen3_5 MTP head of bf16 and block-fp8 checkpoints

* feat(spec): verify qwen4_exp targets on qsa_sparse graphs with PLE verify handlers

* feat(spec): add the qwen4_exp MTP draft from the checkpoint or a Strata pack

* docs(cli): document the speculative decoding flags

* fix(spec): refuse MTP draft sources the draft cannot read before the target loads

* docs(qwen4_exp): describe the verify path in the PLE docstrings

* fix(spec): refuse a missing relative MTP draft path before the target loads

* fix(moe): read MTP head experts on the host and require both stacked tensors

* refactor(spec): tell the final prefill chunk from prompt_len instead of the scheduler's ChunkedReq

* refactor(spec): let each MTP draft family check its own weight source

* fix(qwen4_exp): size the disk PLE eager staging for decode and verify rows

* perf(kernel): read the KV-index gather's chunk count from the grid, not a constexpr

* refactor(moe): let each family name its MTP expert tensors

* refactor(spec): force the MTP draft's MoE resident in the proposer and check it after load

* fix(spec): refuse MTP with dummy weights

* refactor(spec): move HiddenRelay under spec/mtp, its only user

* refactor(spec): drop MTPDraft.config, fold its embed hooks into embed_tokens

* refactor(models): note FeatureSink's graph binding, drop its runtime_checkable

* fix(models): read quantized MTP fc projections under the quant config
2026-11-02 11:37:09 -08:00
Mukkesh Ganesh ea2916d31c feat(gguf): serve gpt-oss gguf checkpoints on gpt_oss (#838)
* feat(gguf): serve gpt-oss gguf checkpoints on gpt_oss

* fix(gguf): refuse missing gpt-oss bias tensors, honor swa pattern, scale nvfp4 before bias

* fix(gguf): honor gpt-oss yarn truncate, gate the bf16 float cast, scale nvfp4 before the down bias
2026-11-02 08:58:07 -08:00
A5bhinav ee43b2a113 fix(server): refuse n != 1 on /v1/completions instead of ignoring it (#842) 2026-10-29 17:15:32 -07:00
Chen Cheng ecbbce4577 fix(moe): skip inactive gguf routes in hybrid decode (#823)
* fix(moe): skip inactive gguf routes in hybrid decode

* perf(moe): run the inactive-route memset and zeroing only in hybrid decode

Only hybrid decode hands the GGUF kernel -1 routes, so mul_mat_id takes an inactive flag
and MMQ does the ids_src1 memset and the zeroing pass only when it is set. Prefill and
fused / offload decode go back to the main-branch MMQ calls.

* perf(moe): compare expert ids unsigned in mm_ids_helper

A negative (inactive) id compared unsigned sorts after every expert and equals none, so the
helper needs no remap to INT_MAX; the remap made nvcc unroll its token loop half as far.
2026-10-28 22:28:13 -07:00
Xiaoze Fan c2c3e828e8 feat(spec): verify draft tokens on the target side (#836)
* feat(spec): add the speculative verify infrastructure

* feat(scheduler): run decode steps as speculative verify steps

* chore(engine): mark the speculative decoding config fields with one comment

* feat(server): add short aliases for the speculative decoding flags
2026-10-28 21:18:02 -07:00
Chen Cheng d2977fbc13 feat(kernel): add an iq1_m mmq kernel for moe steps above 8 tokens (#835) 2026-10-28 20:48:55 -07:00
festeraeb c94535c909 fix(attention): size the decode KV tile to the shared-memory budget (#840)
Assisted-by: Claude (Anthropic)
2026-10-28 20:05:27 -07:00
Xiaoze Fan d043fb61f8 refactor(layers): share one GatedDeltaNet op in layers/linear_attn (#841)
* refactor(layers): share one GatedDeltaNet op in layers/linear_attn

* test(layers): restore the global context in the GDN test and fix stale comments
2026-10-28 17:22:50 -07:00
Xiaoze Fan a2ce636083 refactor(attention): move linear-attention metadata into a LinearAttnBackend (#837)
* refactor(attention): move linear-attention metadata into a LinearAttnBackend

* refactor(attention): make LinearAttnBackend a BaseAttnBackend with FLACaptureData

* refactor(attention): key every linear-attention row by fla_metadata.cache_indices
2026-10-27 23:56:59 -07:00
Xiaoze Fan e328cf6381 refactor: add engine and scheduler seams for multi-row decode steps (#833)
* refactor: add engine and scheduler seams for multi-row decode steps

* refactor: rename the graph steps argument to rows_per_req

* refactor: drop VerifyInfo and require rows_per_req on verify batches

* refactor: fix stale seam comments and read decode_rows_per_req directly

* refactor: reword two GraphRunner comments for multi-row graphs

* fix(engine): scale the GPU-decode MoE pool limits by decode_rows_per_req

* refactor: drop the unread Context.moe_offload_cache field

* test: trim the multi-row seam tests
2026-10-27 16:03:01 -07:00
Xiaoze Fan 9221fd7fc2 feat(scheduler): add radix_lookahead keys to the prefix cache (#834)
* feat(scheduler): key the prefix cache by token bigrams behind radix_lookahead

* refactor(scheduler): make radix_lookahead a bool

* test(scheduler): admit radix_lookahead test requests at their prefix hit
2026-10-26 22:38:53 -07:00
Julien Pierre 075cf57296 fix(server): recover Qwen tool calls after unclosed think (#825) 2026-10-26 19:49:11 -07:00
Xiaoze Fan 408e6ae8c7 feat(gguf): serve image input on minimax-m3 ggufs from a minimax_m3 mmproj (#829)
* feat(gguf): serve image input on minimax-m3 ggufs from a minimax_m3 mmproj

* docs(models): drop the minimax-m3 gguf vision note
2026-10-26 17:15:46 -07:00
Xiaoze Fan edab62cdb3 feat(gguf): serve image input on muse-glimmer ggufs from a muse-glimmer mmproj (#830)
* feat(gguf): serve image input on muse-glimmer ggufs from a muse-glimmer mmproj

* docs(models): drop the muse-glimmer gguf vision note
2026-10-26 17:15:34 -07:00
Mukkesh Ganesh 380693af9c feat(gguf): serve image input on qwen3vl ggufs from a deepstack qwen3vl_merger mmproj (#827) 2026-10-26 15:21:04 -07:00
Xiaoze Fan ba31db2ea7 chore(release): 0.2.1 (#824) 2026-10-25 17:36:06 -07:00
Chris Qian 0bf7dbbf23 fix(models): discover safetensors shards from the index (#819)
* fix(models): find index-only checkpoints that keep shards in a subdirectory

* fix(models): make the checkpoint index authoritative for shard discovery

The index is read directly so safetensors_weight_map stays the only
header-based reader, and _host_ram_fits_parallel sizes the shards the
reader will actually open, so the low-RAM serial fallback fires again
for subdirectory layouts.

* fix(models): fall back to root shards when the index is unusable

Share one index parser between iter_weight_files and safetensors_weight_map
instead of a third copy; warn and glob the root when the index is broken,
absent or empty; reject index keys that escape the model folder; let the hub
download fetch the index itself; tell the truth in the diagnose docstring.

* revert(models): back out the defensive index fallback, keep the no-index test

Per review: a broken index is a broken checkpoint, so the loader keeps the plain one rule -- use the index if there is one, otherwise glob the root. The no-index fallback test stays as the only coverage of the root glob and the consolidated.safetensors rule.
2026-10-24 19:10:52 -07:00
Mukkesh Ganesh 271ff07b30 feat(gguf): serve image input on deepseek4 ggufs from a deepseek4v mmproj (#816)
* feat(gguf): serve image input on deepseek4 ggufs from a deepseek4v mmproj

* fix(gguf): skip an unusable deepseek4v mmproj instead of refusing the trunk
2026-10-24 13:31:24 -07:00
Jesús Jódar Piernas 2d93085545 feat(flint): add nvfp4 modelopt and gguf support (#797)
* feat(flint): add nvfp4 modelopt and gguf support

* refactor(flint): address final review scope requests
2026-10-24 12:37:57 -07:00
LZ_QAQ 32056cf460 fix(ple): add rocm disk wait-sync support (#776)
* fix(ple): add rocm disk wait-sync support

* fix(kernel): validate hip wait-sync support before enabling it

* refactor(ple): share decode fills and sync flag allocation
2026-10-23 15:35:47 -07:00
Xiaoze Fan fd863cce3e feat: add --embed-weights host for a pinned host embedding table (#806)
* feat: add --embed-weights host for a pinned host embedding table

* refactor: check the tie from the model config and share the op walk
2026-10-23 14:41:37 -07:00
Chen Chengandzihaomu 6505b6a005 fix(kernel): convert fp32 to e4m3 in one rounding on sm_89 (#805)
* fix(kernel): convert fp32 to e4m3 in one rounding on sm_89

* fix(kernel): avoid nvidia fp8 cast asm on hip

---------

Co-authored-by: zihaomu <zihaomu@outlook.com>
2026-10-23 14:27:34 -07:00
Mukkesh Ganesh 0c50eb9d82 perf(attention): drop redundant activation copies in glm5 dsa prefill (#795)
* perf(attention): drop redundant activation copies in glm5 dsa prefill

* fix(attention): guard the dsa sparse kernels' int32 q/o addressing
2026-10-23 12:39:33 -07:00
Mukkesh Ganesh 0b98e9d900 feat(gguf): serve image input on glm5-next ggufs from a glm5v mmproj (#810) 2026-10-23 12:28:53 -07:00
cherry77-cloud 507a8bfa81 fix(rotary): preserve fp32 inputs in mrope torch fallbacks (#641) 2026-10-23 00:11:06 -07:00
Xiaoze Fan e0a7e23c9d feat(server): add /v1/cache/flush to drop the prefix cache when idle (#811)
* feat(server): add /v1/cache/flush to drop the prefix cache when idle

* refactor(scheduler): run every cache flush at the idle safe point

* test: cover cache flush on the overlap loop and on a live server
2026-10-22 23:43:14 -07:00
Manish Biswal 20d5d93132 feat(gguf): read NVFP4 token_embd rows in the vendored get_rows (#801)
* feat(gguf): read NVFP4 token_embd rows in the vendored get_rows

Signed-off-by: iamanishx <manishbiswal754@gmail.com>

* fix(tests)

Signed-off-by: iamanishx <manishbiswal754@gmail.com>

---------

Signed-off-by: iamanishx <manishbiswal754@gmail.com>
2026-10-22 20:19:24 -07:00
Mukkesh Ganesh ed359eee0a feat(gguf): serve qwen3vl and qwen3vlmoe gguf checkpoints text only (#807) 2026-10-22 18:14:03 -07:00