* feat(spec): add the speculative verify infrastructure
* feat(scheduler): run decode steps as speculative verify steps
* chore(engine): mark the speculative decoding config fields with one comment
* feat(server): add short aliases for the speculative decoding flags
* refactor: add engine and scheduler seams for multi-row decode steps
* refactor: rename the graph steps argument to rows_per_req
* refactor: drop VerifyInfo and require rows_per_req on verify batches
* refactor: fix stale seam comments and read decode_rows_per_req directly
* refactor: reword two GraphRunner comments for multi-row graphs
* fix(engine): scale the GPU-decode MoE pool limits by decode_rows_per_req
* refactor: drop the unread Context.moe_offload_cache field
* test: trim the multi-row seam tests
* fix(kernel): order the batch-memcpy probe against the current stream
Adapt the stream dependency and busy-stream regression from #414
(https://github.com/FlashML-org/FreeToken/pull/414). Warm the JIT
before queuing the backlog so compilation cannot mask the race.
* fix(moe): enable prefill hit-d2d on rocm
* feat(moe): fetch locked-layer experts through pinned staging and probe the pin limit
Under hybrid, layers whose banks the pin quota could not take keep GPU slots of their own pools and fetch part of their misses through pinned staging that CPU threads fill (--moe-locked-fetch, --moe-locked-max-fetch). --moe-cpu-layers auto without FREETOKEN_PIN_BUDGET_GB now loads the banks unpinned, pins the staging, then pins layers middle-out until CUDA refuses one and OS-locks the rest.
* fix(moe): resize the locked fetch on cache rebuild and gather staging by device address
* feat(moe): log decode moe stats per layer group with --moe-collect-stats
* perf(moe): default the locked fetch to max(2, min(4, logical cpus / 4)) copy threads
* feat(moe): take the locked fetch share from the ft bench bw profile by default
* fix(moe): fuse the locked fetch into publish and gather kernels and address review findings
* refactor(moe): gather the locked staging with fast_index_copy_multi
* refactor(moe): move the locked fetch setup out of the engine and address review findings
* fix(moe): fail the boot when the locked staging cannot be pinned and keep locked layers out of the hybrid share log
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding
* build(deps): vendor the gguf-py reader
* feat(quant): add the GGUF quant kind and an embedding quant method
* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path
* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA
* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16
* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small
* fix(gguf): read F32, F16 and BF16 PLE tables without ggml
* fix(moe): pad the offload slot cache for ggml MMQ tail reads
* fix(gguf): support hip builds and runtime devices
* fix(gguf): copy same-type dequantize and check for missing ggml converters
* fix(gguf): load quantized qwen4exp ple_key and ple_value
* feat(gguf): run float parts of a fused GGUF linear as bf16 segments
* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights
* fix(moe): zero the prefill staging area after the last prefill layer
---------
Co-authored-by: zihaomu <zihaomu@outlook.com>
* feat(moe): split the offload expert cache into slot pools
* perf(moe): stop recompiling the hybrid ensure kernel for each slot pool size
* fix(engine): turn off prefill overlap when --moe-cache-mem leaves under two layers of slots
* fix(engine): load the --moe-cache-split profile before the expert banks
* test(deepseek-v41): add independent text and vision reference
* feat(quant): support block-32 and unrounded inference outputs
* feat(attention): add csa2 packed cache and sparse backend
* feat(deepseek-v41): serve text and images through shared runtime
* perf(mhc): tune batch-stable reduction tiles
* perf(moe): autotune fp4 decode launch geometry
* perf(moe): admit recurring misses against actual lru victims
* revert: perf(moe): admit recurring misses against actual lru victims
This reverts commit 1652d200f0ebb825379cf44c28baf52971e3a461.
* perf(mhc): derive the reduction tile from the width, not a timing
* perf(moe): replace the ds_fp4 decode autotune with a fixed two-warp launch
* fix(deepseek-v41): fence engram staging against in-flight lookups
* fix(moe): return the cpu routed sum in fp32 and round it in python
* fix(deepseek-v41): keep the vision tower and aligner unquantized
* fix(scheduler): admit bounded-replay hits by the history their resume reads
* refactor(deepseek-v41): name the attention family after the model, not csa2
* refactor(kvcache): import the dsv41 pool only for dsv41 models
* refactor(deepseek-v41): name the replay switch --swa-decoder-replay
* fix(server): keep reasoning_effort a string on the wire
* refactor(models): keep the prefill replay hooks off BaseLLMModel
* refactor(deepseek-v41): drop the fp32 routed sum and batch-invariant prefill
* fix(moe): weight ds_fp4 routes in the down epilogue again
* docs(models): anchor the v4.1 replay paragraph on the tech report
* refactor(deepseek-v41): run the decoder source's narrowed prefill on Block
* refactor(kvcache): let the dsv41 pool own the history a resume keeps live
* refactor(deepseek-v41): resume bounded-replay hits by capping the prefix match
* refactor(deepseek-v41): describe the attention group as a dsv4 variant and derive its geometry from the args
* refactor(deepseek-v41): keep the engram table locator in the model package
* refactor(deepseek-v41): drop the fp32 lm head output
- return the head's bf16 logits, as DeepSeek-V4 does
- revert ParallelLMHead(out_dtype=) and LinearKernel.apply_out_dtype, which only this call used
- remove test_head_preserves_fp32_logit_margin
- cast test_vision's prefill logits to fp32 before comparing, as the harness does
* refactor(quant): carry the w4a8 activation block on MoEConfig
- drop ActDesc and QuantScheme.act; the scheme only describes stored tensors
- Fp8BlockConfig sets MoEConfig.act_block from weight_block_size for its fp4 experts via a layer_config hook
- the ds_fp4 GPU kernel and CPU executor read the block from the layer config and require it
* refactor(deepseek-v41): size the kv pool from the args, not a geometry copy
- delete kvcache/dsv41_geometry.py; the pool and cost model read dsv41_args by attribute, as the DSV4 pool reads dsv4_args
- source / ratio aliasing reads args.roles; the duplicated source and ratio validation is gone
- the cost model owns the tier row formats, row bytes and the private window layers
- the pool keeps the format and window/ratio checks, and exposes the formats and is_private_window to the attention backend
* refactor(kvcache): move the dsv4 pool into kvcache/dsv4
* refactor(kvcache): move the dsv41 pool into kvcache/dsv4
* chore(benchmarks): drop the dsv41 selection and fp8 linear benchmarks
* test(deepseek-v41): drop tests that repeat other coverage
* test(deepseek-v41): load the tiny model's resident experts as banks
* fix(server): parse deepseek-v4.1's space-led dsml tool calls
---------
Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
* feat(rotary): size the rope table for YaRN and add --hf-overrides
* fix(rotary): guard mrope's table width and copy --hf-overrides values
- MRotaryEmbedding asserts its cos/sin table is rotary_dim wide. A proportional
rope's table spans the head, and the mrope kernel indexes the rotary_dim // 2
section table with the table's width.
- _with_overrides deep-copies every value it stores: transformers fills rope
defaults into the rope_parameters it is handed, which was the caller's dict.
- The rope-table tests the engine test covers fold into it, the two shared-table
tests are one, and the --hf-overrides test moves to tests/utils/test_hf.py,
where it loads the override against the same values written into config.json,
for a model type transformers knows and one it does not.
- docs/cli.md lists --hf-overrides.
* fix(server): give --hf-overrides a fresh dict per parse
argparse hands its default object to every parse that omits the flag, so
default={} put one shared dict in every ServerArgs. The flag defaults to None and
becomes a new dict where the arguments are assembled, as --mm-processor-kwargs
does.
* docs(rotary): an indexer's heads differ in width from the attention's, not wider
* feat(mm): serve image input on the Qwen VL families
- frontend: OpenAI image_url, Anthropic image blocks and Responses
input_image (http(s), base64, gated file://) become MMItems in the
tokenizer workers; image spans carry content pseudo ids so the radix
cache keys them by content; mrope positions are precomputed per request
- registry: EncoderSpec per family (Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next
register a vision encoder serving image) and ModelSpec.mm_processor;
EngineConfig.active_encoders decides what a process builds,
served_modalities what it accepts; --text-model-only, --mm-disable,
--mm-encoder-weights, --mm-embed-cache-device, --mm-max-pixels
- engine and scheduler: per-request encoder-cache row ownership, chunk
planning on the batch (jobs, gather rows, scatter rows), items encoded
right before the LM forward and gathered into batch.mm_embeds
- models: the Qwen VL vision tower on the text tower's TP layers with
DeepStack, block weights streamed from pinned host banks by default,
merger outputs parked on the host between blocks; SupportsMultimodal
(encode, place_encoder_weights) and a shared embed_input_ids
- server: image inputs rejected with a client-facing reason when no
vision encoder is served; GET /v1/stats reports model.input_modalities
* feat(mm): replace the pixel cap with a token budget and processor kwargs
- --image-min-tokens / --image-max-tokens replace --mm-max-pixels; MMProcessor.get_mm_processor_kwargs turns the MultimodalConfig into the processor call's arguments, the default passes mm.processor_kwargs through and Qwen VL converts the budget to pixel areas in size.shortest_edge / longest_edge
- --mm-processor-kwargs hands a JSON object to the checkpoint's image processor call, applied after the token budget
- MMProcessor is an ABC that owns the model path, the config and the lazily loaded image processor; get_mm_processor returns None when --mm-disable leaves no encoder to serve
- the frontend tokenizer, the tokenizer worker and the offline LLM build one processor each and hand it to TokenizeManager, so the tokenize message carries no per-request pixel cap
* refactor(quant): config, scheme and method layers for checkpoint quantization
QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.
- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
* feat(qwen4_exp): stream the PLE n-gram table from disk (--ple-backend disk)
* fix(qwen4_exp): hash disk PLE rows with the checkpoint-loaded constants
* fix(kernel): handle io_uring partial submission and validate ple_store geometry
* fix(qwen4_exp): validate PLE row coverage, widen deferred-fill signaling, log io/sync choice
* fix(qwen4_exp): zero the eager PLE staging for the warmup prefill
* fix(qwen4_exp): read back the padded decode batch for the disk PLE fill