* feat(kvcache): add fp8_ds_mla and nvfp4_ds_mla kv formats for deepseek-v4 / v4.1
Window and compressed KV are stored as fp8_ds_mla rows (584 B instead of
1024 B) and indexer keys as MXFP4, holding the same values as the
reference's quant round-trip. DeepSeek-V4.1's packed layout is named
nvfp4_ds_mla and shares the row-format code.
* refactor(engine): drop the model field from the dsv4 kv formats
* feat(dflash): load dflash2 gguf drafts for qwen3.8-flash-next
* fix(dflash): accept mrope sections and reuse the float qkv buffer
* docs(dflash): scope the dequantized parts to quantized ggufs
* feat(gguf): serve deepseek-v4.1 gguf checkpoints on deepseek_v41
llama.cpp 10-25 added the deepseek41 layout. The reader takes the config from the file, undoes the converter (engram gate stored as the product, one per_layer_token_embd tensor, hash constants as KVs) and streams the Engram rows from the file as the HF path does.
* fix(gguf): refuse deepseek-v4.1 gguf mismatches at config time
Refuse an FTW of the GGUF before the writer creates the output dir and keep
the side-file hook after the dense loop as on main. Check the Engram hash KVs
and the table in parse_gguf_config, refuse a table with a .scale, and share
the clamp and expert_count readers with the deepseek4 GGUF reader.
* fix(scheduler): return the sliding-window slots when a verify step rolls back
* feat(gemma4): add the assistant draft model and its loader
* feat(spec): serve sliding-window targets with a draft that keeps no KV
* fix(spec): budget verify rows in the swa pool and drop the post-verify sync
* feat(kvcache): add fp8 and int8 per-token-head KV cache formats
* fix(kernel): keep the bf16 m3 sparse k/v loads inline
* fix(attention): refuse a backend that cannot read the kv format
- filter the m3 page-size fallback by kv format
- release the scale buffer when a qsa or bsa rebuild fails
- keep the bf16 qsa and m3 attend code identical to main
- drop docstrings that claimed no register spills
- drop guard tests
* fix(kernel): widen dsa latent row offsets past int32
* feat(spec): store the mtp draft kv in the target's format
* feat(quant)!: compress bf16 weights to df11 at load with --online-quant
GLM-4.7 no longer hardwires DF11 attention and embedding; it serves them
bf16 unless --online-quant df11 is passed, and FTW files converted with the
old DF11 tensors must be reconverted.
* fix(quant): skip layers --online-quant cannot convert instead of failing
* feat(spec): add --speculative-token-map for a smaller mtp draft head
* feat(spec): add scripts/make_token_map.py to write a token map
* fix(spec): correct the token map vram note and seed its test
Serve the HF safetensors and GGUF files. The decoder gains per-layer inputs, KV-shared layers and E2B's double-wide MLP.
- --embed-weights host moves the per-layer table to pinned host RAM, which fits these models on an 8 GB GPU.
- The vision tower does not implement clipped linears, so these checkpoints serve text only.
- Each KV-shared layer still stores its own KV copy.
* feat(spec): hand the pending token to Proposer.propose and return a SpecProposal
* feat(engine): Sampler.probs and sparse-q rejection
* feat(spec): let a proposer declare slot states riding the GDN slots
* feat(models): export layer outputs to the feature sink in the qwen3_5 and qwen4_exp families
* feat(attention): record the source slot of each tracked request
* feat(kernel): DFlash2 draft kernels
* feat(models): DFlash2 draft family
* feat(spec): DFlashWindow
* feat(spec): DFlashProposer
* docs(cli): add dflash to the speculative decoding section
* refactor(engine): share the sampling backend choice and drop the extra renorm in Sampler.probs
* fix(models): build the DFlash rope table uncached and refuse draft scalings it does not apply
* refactor(spec): share the draft path check between the MTP and DFlash proposers
* test(e2e): compare rebuilt outputs exactly only without hybrid MoE
* refactor(spec): keep proposer slot states on EngineConfig instead of replacing model_config
* perf(spec): compute the verify distribution once for sparse-q rejection
* perf(models): let the feature sink add the residual on the rows it keeps
* perf(models): share the DFlash draft K/V weights with the ring projection
* fix(kernel): keep the dflash selector off zero-probability candidates
* test(spec): drop the DFlash tests that the draft forward and Monte Carlo tests already cover
* fix(spec): default and cap DFlash steps at the draft's block_size - 1
* fix(models): reject DFlash draft configs the draft forward cannot serve
* test(e2e): read the MoE strategy from the rebuild test's extra args
* docs(spec): say that slot-state owners write their own prefill track snapshot
* test(models): move the DFlash draft parity tests under tests/models/dflash
* refactor(spec): drop the no-op verify handler of the DFlash window
* refactor(models): fold the DFlash candidate top-k into propose
* refactor(spec): declare the draft step limits through Proposer.num_steps only
* refactor(kernel): make the DFlash ring attention stateless and keep its workspace on the ring
* refactor(models): name the DFlash draft submodules like the other model families
* refactor(spec): move SpecProposal next to DraftContext in spec/proposer.py
* refactor(spec): hand the verify the draft distribution over the whole vocabulary
* refactor(spec): call the DFlashRing object dflash_ring and its length ring_len
* refactor(models): split the DFlash draft's propose into compute_candidates and sample_path
* refactor(spec): keep the draft distribution on its support in SpecProposal
* fix(spec): drop the step's proposal once the verify has read it
* refactor(spec): name the DFlash feature sinks DFlashPrefillSink and DFlashVerifySink
* refactor(models): read the declared Batch.features directly instead of through getattr
* test(dflash): check the window rebuild inside the block forward test
* test(spec): share the plain run of the verify contract and run the DFlash draft once per family
* test(kvcache): drop the window cases from the pool tests and check a layerless state's bytes
* test(spec): check the verify probs of each row against a torch top-k and top-p reference
* test(e2e): add an accept-len floor and a prefill chunk knob to the AIME gate
* refactor(scheduler): let callers take the speculative counts from the status reporter
* test(spec): keep the shared plain runs of the verify contract in their own temp dirs
* test(models): give the ricursive test's stand-in batch a features field
* feat(ricursive): add ripple-65b preview support
* fix(ricursive): read swiglu_clamp as a per-layer gguf array
llama.cpp's Ripple converter (ggml-org/llama.cpp#31692) writes
ricursive.swiglu_clamp with one value per layer, while the first
community GGUFs store a scalar. Accept both and refuse per-layer
values that differ, since the model has a single clamp.
* fix(ricursive): add ripple's own parsers and copy the dense reader
The patch repeats every edit to a copied file and nothing checks that the two agree. The pin and the tree reproduce it. scripts/sync-llama-cpp.sh diffs the copy against the pin and merges a newer llama.cpp from it, and docs/ggml-kernels.md says how to maintain the copy.
* feat(server): report per-request prefill and decode timings in api responses
* fix(server): take timings spans from the scheduler and gate them behind --enable-timings-report
* fix(scheduler): count accepted drafts as min(n, emitted) and stamp reqstats with perf_counter
* feat(spec): add the MTP proposer for the qwen3_5 family
* test(models): read the qwen3_5 MTP head of bf16 and block-fp8 checkpoints
* feat(spec): verify qwen4_exp targets on qsa_sparse graphs with PLE verify handlers
* feat(spec): add the qwen4_exp MTP draft from the checkpoint or a Strata pack
* docs(cli): document the speculative decoding flags
* fix(spec): refuse MTP draft sources the draft cannot read before the target loads
* docs(qwen4_exp): describe the verify path in the PLE docstrings
* fix(spec): refuse a missing relative MTP draft path before the target loads
* fix(moe): read MTP head experts on the host and require both stacked tensors
* refactor(spec): tell the final prefill chunk from prompt_len instead of the scheduler's ChunkedReq
* refactor(spec): let each MTP draft family check its own weight source
* fix(qwen4_exp): size the disk PLE eager staging for decode and verify rows
* perf(kernel): read the KV-index gather's chunk count from the grid, not a constexpr
* refactor(moe): let each family name its MTP expert tensors
* refactor(spec): force the MTP draft's MoE resident in the proposer and check it after load
* fix(spec): refuse MTP with dummy weights
* refactor(spec): move HiddenRelay under spec/mtp, its only user
* refactor(spec): drop MTPDraft.config, fold its embed hooks into embed_tokens
* refactor(models): note FeatureSink's graph binding, drop its runtime_checkable
* fix(models): read quantized MTP fc projections under the quant config
* feat(gguf): serve image input on deepseek4 ggufs from a deepseek4v mmproj
* fix(gguf): skip an unusable deepseek4v mmproj instead of refusing the trunk
* perf(moe): fuse sigmoid + selection-bias routing into a triton kernel
BREAKING CHANGE: glm_moe_dsa keeps its router selection bias in fp32. An FTW converted by an older build stores it as bf16 and still loads, but routes with the rounded bias; restore it with `scripts/ftw_hotfix.py --repo nvidia/GLM-5.2-NVFP4` or reconvert.
* fix(hotfix): scope the bias restore to GLM-5.2 and drop new router comments
* feat(gguf): load the vision tower from a llama.cpp mmproj
Qwen3.5 / Qwen3.6 / Qwen3.8 (qwen3vl_merger) and Gemma-4 26B-A4B / 31B (gemma4v) / 12B (gemma4uv) take image input from a separate mmproj file: --mmproj, else one in the model's own folder. The tower runs in bf16 (quantized mmproj files are dequantized) and its config comes from the GGUF metadata plus per-arch defaults.
* docs(gguf): describe mmproj image input without a model list
* test(gguf): drop the mmproj tests
* fix(gguf): escape the model folder when globbing for mmproj files
* fix(gguf): skip auto-discovered mmproj files the text model cannot use
* fix(gguf): serve text from a GGUF that carries vision tensors itself
* feat(moe): fetch locked-layer experts through pinned staging and probe the pin limit
Under hybrid, layers whose banks the pin quota could not take keep GPU slots of their own pools and fetch part of their misses through pinned staging that CPU threads fill (--moe-locked-fetch, --moe-locked-max-fetch). --moe-cpu-layers auto without FREETOKEN_PIN_BUDGET_GB now loads the banks unpinned, pins the staging, then pins layers middle-out until CUDA refuses one and OS-locks the rest.
* fix(moe): resize the locked fetch on cache rebuild and gather staging by device address
* feat(moe): log decode moe stats per layer group with --moe-collect-stats
* perf(moe): default the locked fetch to max(2, min(4, logical cpus / 4)) copy threads
* feat(moe): take the locked fetch share from the ft bench bw profile by default
* fix(moe): fuse the locked fetch into publish and gather kernels and address review findings
* refactor(moe): gather the locked staging with fast_index_copy_multi
* refactor(moe): move the locked fetch setup out of the engine and address review findings
* fix(moe): fail the boot when the locked staging cannot be pinned and keep locked layers out of the hybrid share log
BREAKING CHANGE: an FTW of Qwen3.5 / Qwen3.6 / Qwen3.8-27B converted by an older build
stores the norms with the 1 added and gives wrong output; reconvert it with `ft checkpoint`.
NVFP4 weight global scales (amax / 2688, 2e-5 to 4e-4 in the served checkpoints) sit at fp16's
normal minimum; stored as fp16, up to 97% of a checkpoint's expert globals were subnormal, with
up to 1.7e-3 relative error per row.
BREAKING CHANGE: an FTW checkpoint that stores NVFP4 global scales as fp16 is refused at load;
reconvert it with `ft checkpoint`.
The NVFP4 release quantizes the attention gate with a different weight global scale than q/k/v, so
one fused weight cannot carry a single global for all four parts.
BREAKING CHANGE: an FTW checkpoint converted with the fused qkvg_proj no longer loads; reconvert it
with `ft checkpoint`.
* feat(moe): bench hybrid fetch fractions per moe layer kind
The ft bench bw profile (version 6) holds only layer_kinds rows, keyed by CPU format, options, expert shape, top_k and CPU worker threads; older profiles are ignored. ft bench bw --model also takes checkpoint paths and benches each of their layer kinds. Hybrid decode reads each layer's own row for its fetch fraction, else keeps a fixed cap of 1. --moe-strategy auto no longer upgrades to hybrid from the profile.
* fix(bench): fill gguf mxfp4 scales, scope act_block to ds_fp4 and skip cpu-less rows
* feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu
* feat(cpu-moe): run float GGUF expert rows on the cpu through ggml-cpu's vec_dot
* feat(cpu-moe): build ggml-cpu for each llama.cpp x86 variant and run the best one the CPU supports
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding
* build(deps): vendor the gguf-py reader
* feat(quant): add the GGUF quant kind and an embedding quant method
* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path
* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA
* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16
* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small
* fix(gguf): read F32, F16 and BF16 PLE tables without ggml
* fix(moe): pad the offload slot cache for ggml MMQ tail reads
* fix(gguf): support hip builds and runtime devices
* fix(gguf): copy same-type dequantize and check for missing ggml converters
* fix(gguf): load quantized qwen4exp ple_key and ple_value
* feat(gguf): run float parts of a fused GGUF linear as bf16 segments
* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights
* fix(moe): zero the prefill staging area after the last prefill layer
---------
Co-authored-by: zihaomu <zihaomu@outlook.com>
* feat(moe): split the offload expert cache into slot pools
* perf(moe): stop recompiling the hybrid ensure kernel for each slot pool size
* fix(engine): turn off prefill overlap when --moe-cache-mem leaves under two layers of slots
* fix(engine): load the --moe-cache-split profile before the expert banks
* refactor(cpu-moe): split the executor into weight formats and gpu sync backends
- move each expert weight format into kernels/<fmt>.cpp behind the MoeFormat
interface in format.h; the executor keeps the threads, the work split and
the pass 2 route sum
- pass each format its banks and integer options by name; MoeParams keeps only
the shared shape and activation fields
- key the formats by name in one table that weight_formats() exports, replacing
the integer format ids and the fp8_block_isa_name hook
- replace the per-(layer, batch size) flag slots, pinned IO buffers and task
table with one doorbell word MoeTask{layer, num_tokens} and one shared pinned
IO block of max_tokens rows
- move the GPU/CPU handshake behind GpuSync with a CUDA and a HIP backend picked
by setup.py; HIP waits on signal memory and adds explicit batch-memop graph
nodes where the runtime does not record the native memops under capture
- raise on graph capture when the handshake is off and host functions are not
replay-safe
- allocate format scratch on 64-byte boundaries
- size max_tokens by the explicit cuda_graph_bs as well
Co-authored-by: zihaomu <zihaomu@outlook.com>
* perf(cpu-moe): skip the worker pool for tasks without routed experts
* perf(cpu-moe): publish and combine the cpu handshake with kernels
* fix(cpu-moe): clean up failed hip capability probes
---------
Co-authored-by: zihaomu <zihaomu@outlook.com>
* feat(cpu-moe): block-FP8 expert GEMV (W8A16) for the cpu/hybrid backends
Block-fp8 checkpoints (Qwen3.5/3.6-FP8, DeepSeek-V3-style, GLM) had no CPU expert
path at all:
NotImplementedError: --moe-backend cpu/hybrid ... supports
['bf16', 'ds_fp4', 'mxfp4_triton', 'nvfp4', 'q4_0'] formats,
but this checkpoint's experts are 'fp8_block'
so every expert miss had to stream over PCIe at ~25 GB/s while the CPU sat idle.
The bank is fp8-e4m3 row-major [rows, K] with one bf16 scale per 128x128 block --
K contiguous per output row, unlike mxfp4's transposed bank, so it maps straight
onto a bf16 dot product. e4m3 -> bf16 is *exact* (3 mantissa bits into 7, and
bf16's exponent range covers e4m3's whole 2^-9..448 span), so the AVX-512 path
widens weights in-register and feeds `_mm512_dpbf16_ps` against the bf16
activations: identical products to the reference's fp32 math, differing only in
summation order -- the same latitude the bf16 dot already takes.
The normal range rebases the exponent with a shift and a constant +120; exp == 0
is subnormal (m * 2^-9) and does not obey that, so those lanes blend in from a
small table. AVX2 does the same in fp32; scalar goes through the existing 256-entry
e4m3 LUT.
`ft bench bw` gains an fp8_block CPU measurement (it could already build the banks).
Measured on main with this commit alone, `--dtype fp8_block --reps 5`, 2x Xeon Gold
6526Y:
CPU MoE PCIe ratio backend picked
main not measurable 25.1 -- offload
main + this 53.7 GB/s 25.1 2.14x hybrid
Before this commit the format reports no CPU number, so the backend picker has
nothing to weigh and selects offload. The ratio is what decides the backend, so
adding the kernel is what lets a block-fp8 deployment use the CPU at all.
End to end on Qwen/Qwen3.6-35B-A3B-FP8, median of 3 generations at matched GPU
expert residency:
residency offload hybrid delta
10% 43.94 50.94 +15.9%
25% 82.65 55.12 -33%
The win is real but conditional: hybrid pays off once the miss rate is high enough
that PCIe is the bottleneck, and costs throughput when enough experts are resident
that the CPU becomes the limiter. See the PR discussion -- the ratio makes
`--moe-backend auto` select hybrid for every block-fp8 checkpoint, which is right
for the oversubscribed case this project targets and wrong for a model that nearly
fits, and the bench models bandwidth but not miss rate.
Correctness is checked against the Triton block-fp8 decode kernel on identical
banks and routing (bs 1/2/5), plus an exactness check on the e4m3 -> bf16 widening
over all 256 codes. A weight-format ABI probe (`max_weight_format_id`) makes a
stale prebuilt `.so` fail loudly instead of falling through to the wrong dequant
branch and returning silently wrong numbers.
Assisted-by: Claude
* fix(cpu-moe): honor padded fp8 scale strides
The scale banks' trailing dimension is padded so per-expert rows land on
a 16-byte boundary for the fused copy (kernel/aot_models.py:
fp8_block_scale_pad), so the row stride is not ceil(K/128). Recomputing
it from the unpadded block count reads scales at the wrong offset.
Qwen3.8 is the case in the tree: models/qwen3_5_moe/weight.py records
down_scale as 20x5 bf16 = 200 B, and 200 % 16 != 0 pads cols 5 -> 6.
The benchbw workload (H=2048, I=512) and DSV4-Flash (H=7168, I=2048)
both happen to give rows*cols divisible by 8, so neither trips it.
Read the stride off the tensor and pass it through to the kernel, and
let the shape check assert the unpadded extent is present rather than an
exact shape. The fixture now allocates padded banks, so the existing GPU
cross-check covers this too; the GPU kernel was already stride-aware.
* test(cpu-moe): add float64 parity reference and an avx512f fp8 tier
The existing GPU-vs-CPU check compares two fp32 accumulations at a 2e-2
tolerance, which cannot see accumulator precision loss. Add a float64
reference over mixed-sign, full-range e4m3 weights and zero-mean
activations, and force each ISA tier explicitly so the sweep is real.
Also add an avx512f tier between avx512bf16 and avx2, so CPUs with
AVX-512 but no BF16 extension (Skylake-SP through Ice Lake-SP) stop
falling back to 8-wide fp32.
Measured on Xeon Platinum 8368 (Ice Lake-SP, no avx512_bf16), three
repetitions each: +19% single-thread cache-resident (3.31 -> 3.94 GB/s,
disjoint intervals), but only +4% under `ft bench bw --dtype fp8` at 75
threads (60.4 -> 62.7 GB/s, overlapping intervals). At that thread count
the kernel is memory-bound, so the project's own benchmark cannot
resolve the tier; the single-thread gain is the honest figure.
* test(cpu-moe): cover fp8 dispatch and cpu-only regressions
The new tier shipped without test coverage: deleting the avx512f branch
from select_fp8dot left every test green, because isa_name() reports
select_dot() (the bf16 chain) rather than the fp8 dispatch. Injecting a
ragged-tail bug into dot_fp8_block_avx2 was invisible for the same
reason -- no test ran numerics under a forced tier.
Force each available fp8 tier through the float64 parity GEMV, expose
the selected fp8 tier so the assertion can see it, and make the padded
bank test run a GEMV instead of only constructing an executor.
Also move the module-level CUDA guard onto the tests that actually need
a GPU. A CPU MoE kernel's most relevant hosts are the ones without a
usable GPU, and all ten tests were skipping there.
* review: force each ISA tier in the fp8_block test, and stop the docstring overclaiming
The comparison test only varied batch size, so all three cases ran
whatever tier this machine auto-selects and the scalar / AVX2 /
AVX-512-BF16 fp8_block kernels were never compared against the GPU at
all.
Parametrise FREETOKEN_CPU_MOE_ISA over the four tiers. Setting the
variable alone would not be enough: pick_isa() caps *downward*, so asking
for a tier the CPU or build lacks silently returns a lower one and the
test would pass four times over the same kernel. The executor latches its
choice into isa_name() at construction, so read it back and skip when it
does not match what was asked for. On a machine with all four, that is 4
x 3 cases against a 8-expert, 256-hidden problem.
test_subnormal_and_sign_decode_exactly said it validated that "every e4m3
byte must widen to the value the reference LUT gives", but it never calls
the CPU kernel -- it checks that bf16 round-trips e4m3 losslessly and
that subnormals are finite and small. That is the premise the widening
rests on, which is worth pinning, but it is not the widening. Renamed to
test_bf16_carries_every_e4m3_byte and the docstring now says what the
body asserts.
Not run here: this file is skipif(not torch.cuda.is_available()), and
this machine has no GPU and no torch. Compiles clean and ruff reports the
same two pre-existing E741 findings before and after.
Assisted-by: Claude
* fix(moe): let fp8_block experts decode on the cpu executor
* fix(cpu-moe): build the bf16 fp8 tier only when the compiler has avx512bf16
* fix(bench): version the bw profile and ignore older ones
* perf(cpu-moe): read fp8 weights as fp16 bits in the cpu dot kernels
* refactor(cpu-moe): write the fp8_block bank checks like the other formats
* refactor(cpu-moe): name the fp8 cpu symbols fp8_block
* docs(server): list fp8_block in the --moe-cpu-layers help
---------
Co-authored-by: Gabriel A. Devenyi <gdevenyi@gmail.com>
Co-authored-by: ChenyuHe <hechenyu@zju.edu.cn>
* feat(server): require an API key on every route but /health
* feat(shell): send the server's API key
* feat(ctl): send the server's API key
* feat(launch): hand the server's API key to the agent
* feat(daemon): pass the serve's API key as apiKey
The daemon reaches the serve's /v1/stats and prepare-stop itself, so the key
travels in the start request, reaches the serve through its environment and is
kept in the state file for re-adoption. The daemon's own FREETOKEN_API_KEY is no
longer passed down to the serve.
* fix(launch): keep the API key out of the variable dsh's web search reads
dsh's web search reads DEEPSEEK_API_KEY and sends it to api.deepseek.com, so
dsh now reads the key from FREETOKEN_API_KEY through the apiKeyEnv of its
llm-deepseek section, and DEEPSEEK_API_KEY stays the dummy.
* fix(server): check the API key on OPTIONS requests
* fix(launch): write agent configs that hold the API key owner-only
* chore: drop comments that restate the code
* feat(deepseek_v4): serve image input on the V4-Flash-Vision release
The block layout depends on where the image lands: lead pads align its first row token to
a 4-token compression stride, so the item hash carries the layout and two alignments never
share a radix key.
The block's tokens all carry one content pad id (the shared pad scheme); the sparse
attention reads its spans from batch.mm_spans, which the scheduler recovers from the flat
block ends its existing plan already reports.
* fix(mm): keep the shared mmprocessor signature and own the dsv4 offset in its apply
* fix(models): build dsv4 image spans in the model and start them at image_start
* fix(models): mount the dsv4 vision tower under visual and keep its weights unquantized
* fix(models): keep dsv4 image blocks whole across chunked prefill
* test(deepseek-v41): add independent text and vision reference
* feat(quant): support block-32 and unrounded inference outputs
* feat(attention): add csa2 packed cache and sparse backend
* feat(deepseek-v41): serve text and images through shared runtime
* perf(mhc): tune batch-stable reduction tiles
* perf(moe): autotune fp4 decode launch geometry
* perf(moe): admit recurring misses against actual lru victims
* revert: perf(moe): admit recurring misses against actual lru victims
This reverts commit 1652d200f0ebb825379cf44c28baf52971e3a461.
* perf(mhc): derive the reduction tile from the width, not a timing
* perf(moe): replace the ds_fp4 decode autotune with a fixed two-warp launch
* fix(deepseek-v41): fence engram staging against in-flight lookups
* fix(moe): return the cpu routed sum in fp32 and round it in python
* fix(deepseek-v41): keep the vision tower and aligner unquantized
* fix(scheduler): admit bounded-replay hits by the history their resume reads
* refactor(deepseek-v41): name the attention family after the model, not csa2
* refactor(kvcache): import the dsv41 pool only for dsv41 models
* refactor(deepseek-v41): name the replay switch --swa-decoder-replay
* fix(server): keep reasoning_effort a string on the wire
* refactor(models): keep the prefill replay hooks off BaseLLMModel
* refactor(deepseek-v41): drop the fp32 routed sum and batch-invariant prefill
* fix(moe): weight ds_fp4 routes in the down epilogue again
* docs(models): anchor the v4.1 replay paragraph on the tech report
* refactor(deepseek-v41): run the decoder source's narrowed prefill on Block
* refactor(kvcache): let the dsv41 pool own the history a resume keeps live
* refactor(deepseek-v41): resume bounded-replay hits by capping the prefix match
* refactor(deepseek-v41): describe the attention group as a dsv4 variant and derive its geometry from the args
* refactor(deepseek-v41): keep the engram table locator in the model package
* refactor(deepseek-v41): drop the fp32 lm head output
- return the head's bf16 logits, as DeepSeek-V4 does
- revert ParallelLMHead(out_dtype=) and LinearKernel.apply_out_dtype, which only this call used
- remove test_head_preserves_fp32_logit_margin
- cast test_vision's prefill logits to fp32 before comparing, as the harness does
* refactor(quant): carry the w4a8 activation block on MoEConfig
- drop ActDesc and QuantScheme.act; the scheme only describes stored tensors
- Fp8BlockConfig sets MoEConfig.act_block from weight_block_size for its fp4 experts via a layer_config hook
- the ds_fp4 GPU kernel and CPU executor read the block from the layer config and require it
* refactor(deepseek-v41): size the kv pool from the args, not a geometry copy
- delete kvcache/dsv41_geometry.py; the pool and cost model read dsv41_args by attribute, as the DSV4 pool reads dsv4_args
- source / ratio aliasing reads args.roles; the duplicated source and ratio validation is gone
- the cost model owns the tier row formats, row bytes and the private window layers
- the pool keeps the format and window/ratio checks, and exposes the formats and is_private_window to the attention backend
* refactor(kvcache): move the dsv4 pool into kvcache/dsv4
* refactor(kvcache): move the dsv41 pool into kvcache/dsv4
* chore(benchmarks): drop the dsv41 selection and fp8 linear benchmarks
* test(deepseek-v41): drop tests that repeat other coverage
* test(deepseek-v41): load the tiny model's resident experts as banks
* fix(server): parse deepseek-v4.1's space-led dsml tool calls
---------
Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
* perf(moe): default fused gather to 32 blocks/bank on NVLink-C2C hosts
* build(aot): prebuild fast_index_copy_multi (1024, 32) variant
* perf(moe): make fused H2D gather grid configurable via env var
* docs: document FREETOKEN_H2D_BLOCKS_PER_BANK for NVLink-C2C hosts
* fix(moe): auto-detect H2D gather grid per device in the copy plan
* chore(docs): revised docs for NVLinkC2C
* feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks
Resident experts skip the state dict and load through the offload expert-bank loader (HF pieces or FTW banks) into per-layer GPU banks, so fused serves bf16, fp8_block, NVFP4 and MXFP4 alike. Unified-memory GPUs auto-select fused for these formats and ignore offload-only flags with a warning.
BREAKING CHANGE: ft checkpoint always writes MoE experts as banks; an FTW converted with --moe-backend fused or triton (dense experts) no longer loads and must be reconverted.
* docs(moe): name only GB10 / DGX Spark for unified memory
* feat(rotary): size the rope table for YaRN and add --hf-overrides
* fix(rotary): guard mrope's table width and copy --hf-overrides values
- MRotaryEmbedding asserts its cos/sin table is rotary_dim wide. A proportional
rope's table spans the head, and the mrope kernel indexes the rotary_dim // 2
section table with the table's width.
- _with_overrides deep-copies every value it stores: transformers fills rope
defaults into the rope_parameters it is handed, which was the caller's dict.
- The rope-table tests the engine test covers fold into it, the two shared-table
tests are one, and the --hf-overrides test moves to tests/utils/test_hf.py,
where it loads the override against the same values written into config.json,
for a model type transformers knows and one it does not.
- docs/cli.md lists --hf-overrides.
* fix(server): give --hf-overrides a fresh dict per parse
argparse hands its default object to every parse that omits the flag, so
default={} put one shared dict in every ServerArgs. The flag defaults to None and
becomes a new dict where the arguments are assembled, as --mm-processor-kwargs
does.
* docs(rotary): an indexer's heads differ in width from the attention's, not wider
* feat: add simple ROCm GPU support for RDNA3 (gfx1100-1103)
- Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents
- Update pinned_tensor.cpp to compile under both nvcc and hipcc
- Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family)
- Guard NVIDIA arch checks to return None on ROCm
- Skip nvcc version check in _toolchain.py when on ROCm
- Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch)
- Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm
- Add is_rocm() and driver_hip_version() in backend.py
- Add rocm-smi fallback in __main__.py for clangd generation
- Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds,
Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache
- Add AMD ROCm classifier in pyproject.toml
* fix(rocm): fall back for single-bank fast index copy
* test(rocm): cover fast index copy fallback
* test(rocm): use explicit fast copy module import
* fix(rocm): complete RDNA3 runtime path
* fix(rocm): preserve CPU MoE implementation
* build(rocm): support RDNA4 discovery and modular SDKs
* fix(rocm): complete modular SDK runtime support
* test(rocm): harden foundation platform contracts
* fix(rocm): pin jit target and split install guide
* perf(kernel): restore ROCm non-temporal index copy
---------
Co-authored-by: bouclem <bouclement2012@gmail.com>
Co-authored-by: Yuu <206304251+nekomario28@users.noreply.github.com>
Co-authored-by: LZ-QWQ <lz1830837889@163.com>
* fix(checkpoint): skip the vision tower before reading an FTW served text-only
- iter_ftw_weights takes a keep(name) predicate and drops the rejected entries before their bytes are read; load_weight passes the vision prefixes through it instead of discarding the tensors after the replay
- an FTW converted before its family served images holds no encoder tensors: the engine now refuses it with the reconvert / --text-model-only choice instead of failing on the first missing key
- ftw_tensor_names lists an index's names by kind; docs/models.md notes the pre-image-support FTWs
* feat(scripts): add the vision encoder to a pre-image-support FTW
- ftw_hotfix.py fetches the encoder's checkpoint tensors by byte range from the Hub or a local checkpoint into a scratch checkpoint dir and lets the family's own weight reader name and fuse them, so the repaired FTW holds exactly what the converter would have written
- every family with an encoder gains iter_vision_weights, the encoder-only reader behind load_vision_weight: the Qwen families and Gemma yield the tower as stored (Gemma without its clipped-linear wrapper), GLM and MiniMax reuse their tower readers, Muse fuses q/k/v as its dense pass does
- the engine's refusal of a tower-less FTW and the models.md note name the tool next to reconverting; docs/ftw-hotfix.md gains the repair row and a Hub example; the ignored-leftover prefixes follow VISION_KEY_PREFIXES