debug_once / info_once / warning_once log on rank 0, once per distinct message and args. select_kernel, which logged its choice once per layer, and the hand-rolled log-once guards now use them.
* feat(server): report per-request prefill and decode timings in api responses
* fix(server): take timings spans from the scheduler and gate them behind --enable-timings-report
* fix(scheduler): count accepted drafts as min(n, emitted) and stamp reqstats with perf_counter
The VNNI dots were compute bound per core; with fewer instructions per weight they reach the DRAM read bandwidth.
The W4A8 kernel now follows FREETOKEN_CPU_MOE_ISA.
* feat(spec): add the MTP proposer for the qwen3_5 family
* test(models): read the qwen3_5 MTP head of bf16 and block-fp8 checkpoints
* feat(spec): verify qwen4_exp targets on qsa_sparse graphs with PLE verify handlers
* feat(spec): add the qwen4_exp MTP draft from the checkpoint or a Strata pack
* docs(cli): document the speculative decoding flags
* fix(spec): refuse MTP draft sources the draft cannot read before the target loads
* docs(qwen4_exp): describe the verify path in the PLE docstrings
* fix(spec): refuse a missing relative MTP draft path before the target loads
* fix(moe): read MTP head experts on the host and require both stacked tensors
* refactor(spec): tell the final prefill chunk from prompt_len instead of the scheduler's ChunkedReq
* refactor(spec): let each MTP draft family check its own weight source
* fix(qwen4_exp): size the disk PLE eager staging for decode and verify rows
* perf(kernel): read the KV-index gather's chunk count from the grid, not a constexpr
* refactor(moe): let each family name its MTP expert tensors
* refactor(spec): force the MTP draft's MoE resident in the proposer and check it after load
* fix(spec): refuse MTP with dummy weights
* refactor(spec): move HiddenRelay under spec/mtp, its only user
* refactor(spec): drop MTPDraft.config, fold its embed hooks into embed_tokens
* refactor(models): note FeatureSink's graph binding, drop its runtime_checkable
* fix(models): read quantized MTP fc projections under the quant config
* fix(moe): skip inactive gguf routes in hybrid decode
* perf(moe): run the inactive-route memset and zeroing only in hybrid decode
Only hybrid decode hands the GGUF kernel -1 routes, so mul_mat_id takes an inactive flag
and MMQ does the ids_src1 memset and the zeroing pass only when it is set. Prefill and
fused / offload decode go back to the main-branch MMQ calls.
* perf(moe): compare expert ids unsigned in mm_ids_helper
A negative (inactive) id compared unsigned sorts after every expert and equals none, so the
helper needs no remap to INT_MAX; the remap made nvcc unroll its token loop half as far.
* feat(spec): add the speculative verify infrastructure
* feat(scheduler): run decode steps as speculative verify steps
* chore(engine): mark the speculative decoding config fields with one comment
* feat(server): add short aliases for the speculative decoding flags
* refactor(attention): move linear-attention metadata into a LinearAttnBackend
* refactor(attention): make LinearAttnBackend a BaseAttnBackend with FLACaptureData
* refactor(attention): key every linear-attention row by fla_metadata.cache_indices
* refactor: add engine and scheduler seams for multi-row decode steps
* refactor: rename the graph steps argument to rows_per_req
* refactor: drop VerifyInfo and require rows_per_req on verify batches
* refactor: fix stale seam comments and read decode_rows_per_req directly
* refactor: reword two GraphRunner comments for multi-row graphs
* fix(engine): scale the GPU-decode MoE pool limits by decode_rows_per_req
* refactor: drop the unread Context.moe_offload_cache field
* test: trim the multi-row seam tests
* feat(scheduler): key the prefix cache by token bigrams behind radix_lookahead
* refactor(scheduler): make radix_lookahead a bool
* test(scheduler): admit radix_lookahead test requests at their prefix hit
* fix(models): find index-only checkpoints that keep shards in a subdirectory
* fix(models): make the checkpoint index authoritative for shard discovery
The index is read directly so safetensors_weight_map stays the only
header-based reader, and _host_ram_fits_parallel sizes the shards the
reader will actually open, so the low-RAM serial fallback fires again
for subdirectory layouts.
* fix(models): fall back to root shards when the index is unusable
Share one index parser between iter_weight_files and safetensors_weight_map
instead of a third copy; warn and glob the root when the index is broken,
absent or empty; reject index keys that escape the model folder; let the hub
download fetch the index itself; tell the truth in the diagnose docstring.
* revert(models): back out the defensive index fallback, keep the no-index test
Per review: a broken index is a broken checkpoint, so the loader keeps the plain one rule -- use the index if there is one, otherwise glob the root. The no-index fallback test stays as the only coverage of the root glob and the consolidated.safetensors rule.
* feat(gguf): serve image input on deepseek4 ggufs from a deepseek4v mmproj
* fix(gguf): skip an unusable deepseek4v mmproj instead of refusing the trunk
* fix(ple): add rocm disk wait-sync support
* fix(kernel): validate hip wait-sync support before enabling it
* refactor(ple): share decode fills and sync flag allocation
* fix(kernel): convert fp32 to e4m3 in one rounding on sm_89
* fix(kernel): avoid nvidia fp8 cast asm on hip
---------
Co-authored-by: zihaomu <zihaomu@outlook.com>
* feat(server): add /v1/cache/flush to drop the prefix cache when idle
* refactor(scheduler): run every cache flush at the idle safe point
* test: cover cache flush on the overlap loop and on a live server
* perf(moe): fuse sigmoid + selection-bias routing into a triton kernel
BREAKING CHANGE: glm_moe_dsa keeps its router selection bias in fp32. An FTW converted by an older build stores it as bf16 and still loads, but routes with the rounded bias; restore it with `scripts/ftw_hotfix.py --repo nvidia/GLM-5.2-NVFP4` or reconvert.
* fix(hotfix): scope the bias restore to GLM-5.2 and drop new router comments
* feat(gguf): load the vision tower from a llama.cpp mmproj
Qwen3.5 / Qwen3.6 / Qwen3.8 (qwen3vl_merger) and Gemma-4 26B-A4B / 31B (gemma4v) / 12B (gemma4uv) take image input from a separate mmproj file: --mmproj, else one in the model's own folder. The tower runs in bf16 (quantized mmproj files are dequantized) and its config comes from the GGUF metadata plus per-arch defaults.
* docs(gguf): describe mmproj image input without a model list
* test(gguf): drop the mmproj tests
* fix(gguf): escape the model folder when globbing for mmproj files
* fix(gguf): skip auto-discovered mmproj files the text model cannot use
* fix(gguf): serve text from a GGUF that carries vision tensors itself
* docs: set admission rules for new tests
* docs: tighten the tests README
- list where older model-specific tests sit (flat in models/, subsystem dirs, dsv4/) and add dsv4/ to the table
- say that layer kernels and fused ops go in kernels/, whole layer modules in layers/
- drop the silent-bug examples repeated from the admission list
- keep reusing a file's fixtures only when they are not stand-ins for FreeToken objects
* feat(gguf): serve DeepSeek-V4-Flash GGUF checkpoints on deepseek4
* fix(gguf): refuse reap-style per-layer expert prunes in deepseek4 ggufs
* fix(gguf): apply the deepseek4 review fixes
- route the offload experts through swiglu_clamp so the GGUF MoE path applies the limit
- map model.embed -> embed on the HF row so the bf16 exemption matches
- default a wholly missing rope.scaling block to the family yarn values with a warning;
refuse a partly present one
* fix(kernel): order the batch-memcpy probe against the current stream
Adapt the stream dependency and busy-stream regression from #414
(https://github.com/FlashML-org/FreeToken/pull/414). Warm the JIT
before queuing the backlog so compilation cannot mask the race.
* fix(moe): enable prefill hit-d2d on rocm
* feat(moe): fetch locked-layer experts through pinned staging and probe the pin limit
Under hybrid, layers whose banks the pin quota could not take keep GPU slots of their own pools and fetch part of their misses through pinned staging that CPU threads fill (--moe-locked-fetch, --moe-locked-max-fetch). --moe-cpu-layers auto without FREETOKEN_PIN_BUDGET_GB now loads the banks unpinned, pins the staging, then pins layers middle-out until CUDA refuses one and OS-locks the rest.
* fix(moe): resize the locked fetch on cache rebuild and gather staging by device address
* feat(moe): log decode moe stats per layer group with --moe-collect-stats
* perf(moe): default the locked fetch to max(2, min(4, logical cpus / 4)) copy threads
* feat(moe): take the locked fetch share from the ft bench bw profile by default
* fix(moe): fuse the locked fetch into publish and gather kernels and address review findings
* refactor(moe): gather the locked staging with fast_index_copy_multi
* refactor(moe): move the locked fetch setup out of the engine and address review findings
* fix(moe): fail the boot when the locked staging cannot be pinned and keep locked layers out of the hybrid share log
* feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next
Adds the llama.cpp glm5-next GGUF reader (unsloth GLM-5.3-Flash-GGUF, both
the glm5-next and the older glm5next arch spellings) on the GGUF quant
kind from #690:
- parse_gguf_config derives the text config from the GGUF KV metadata,
with per-block attention.head_count_kv laying out the KDA/DSA hybrid,
the trailing MTP block dropped like the HF loader drops it, and the
MLA head widths from the *_length_mla keys.
- iter_gguf_weights maps every tensor to the glm5_next state dict and
undoes the converter transforms: A_log stored as -exp(A_log) is
inverted, the per-head attn_k_b/attn_v_b pair (k transposed) fuses
into the dense kv_b_proj layout the MLA absorption reads, and the
q|k|v conv kernels concatenate on the channel axis. The mixed-type
fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a) rides #702 segments;
the routed experts load as GGUF expert banks at their shipped types
(unsloth dynamic quants: IQ3_S gate/up + IQ4_XS down, Q8_0 outlier
layers pass through natively).
- passes quant_config to embed_tokens so the Q8_0 token_embd serves
packed; registers Glm5NextGGUFForCausalLM and claims it in the AOT
table.
Verified end to end against unsloth GLM-5.3-Flash-UD-IQ4_XS (5 split
shards): the full engine on a 32GB GPU serves coherent output with the
experts off host banks. Text-only, TP=1.
Part of #671.
* fix(gguf): refuse unread scale tensors in glm5-next ggufs
check_scales at config parse like the llama, qwen3 and gemma4 loaders
(review on #721): a .scale / .input_scale outside an NVFP4 weight now
fails with a clear error instead of loading silently unread.
* fix(gguf): serve the quantized tensors of unsloth dynamic glm5-next quants
unsloth's UD quants quantize tensors the AesSedAI-style files keep float:
the MLA kv_b halves (Q8_0), the mHC mixers and the indexer's wk /
weights_proj / compressor gate. kv_b_proj joins the registry's
unquantized_modules (glm_moe_dsa keeps its kv_b dense for the MLA
absorption for the same reason) and the reader dequantizes through the
vendored gguf-py reference on every float-expecting read; wq_b keeps
serving its checkpoint quant packed (attention.py keeps wk and
weights_proj dense). Regression test stores the kv_b halves Q8_0.
* fix(gguf): treat the glm5 pre-tokenizer spelling like glm4
llama-vocab.cpp maps both spellings to CHATGLM4 with ignore_merges;
unsloth writes glm4 and AesSedAI's writer glm5 for the same vocabulary.
Without the mapping a glm5 file loads but silently mis-tokenizes
(tok_compare: 22,840/153,725 vocab words differ from the HF reference;
0 everywhere with it).
* fix(gguf): size the mmq tail padding by ggml type and k
pad_rows added one zero row of k columns, which covers the bytes MMQ
reads past the last row only when k is at least half a padding unit
(512); glm5-next's KDA f_b / g_b (k = linear head dim 128, Q8_0 in
dynamic quants) are the first packed tensors below that and died at the
first mul_mat. pad_rows now takes the ggml type and returns enough whole
rows to cover mmq_tail_bytes for any (type, k), consistently in the
reader's fused(), the gguf linear and embedding params and the expert
host banks. Found serving the real unsloth UD-IQ4_XS: the dry load
never runs a forward.
* fix(gguf): take the older glm5-next writer variants in stride
Two long-tail writer variants the full check sweep found (the GSQ-RCO
lineage alone is ~57k downloads/month):
- the conv kernels with a trailing unit dim: llama.cpp's glm5-next
loader tries that 4-D shape before the 3-D one, so trailing unit dims
in ne are dropped on float reads (they carry no elements).
- attention.head_count_kv as the generic scalar (all blocks): wrong for
the hybrid -- llama.cpp broadcasts it and then fails tensor creation
-- so only the list is trusted and a scalar falls to the tensor-name
census like the pre-list writer.
* fix(gguf): glm5-next review cleanups
- refuse instead of guessing metadata the model cannot express: the
scalar attention.head_count_kv spelling (llama.cpp broadcasts it and
then fails tensor creation; every public file carries the list),
missing *_length_mla keys through need(), non-uniform
swiglu_clamp_exp / swiglu_clamp_shexp, indexer.types with non-full
layers, expert_gating_func other than sigmoid, and rope dims as a
NotImplementedError instead of an assert.
- call check_scales directly; aot_models comment on its own line.
- drop the GGUF test files; coverage will be handled across families.
Resolve the parsed URI path without prepending localhost, while
rejecting non-local authorities and retaining the media root check.
Add regression cases for localhost authority spellings and remote hosts.
Refs FlashML-org/FreeToken#757