* refactor: add engine and scheduler seams for multi-row decode steps
* refactor: rename the graph steps argument to rows_per_req
* refactor: drop VerifyInfo and require rows_per_req on verify batches
* refactor: fix stale seam comments and read decode_rows_per_req directly
* refactor: reword two GraphRunner comments for multi-row graphs
* fix(engine): scale the GPU-decode MoE pool limits by decode_rows_per_req
* refactor: drop the unread Context.moe_offload_cache field
* test: trim the multi-row seam tests
* fix(kernel): order the batch-memcpy probe against the current stream
Adapt the stream dependency and busy-stream regression from #414
(https://github.com/FlashML-org/FreeToken/pull/414). Warm the JIT
before queuing the backlog so compilation cannot mask the race.
* fix(moe): enable prefill hit-d2d on rocm
* feat(moe): fetch locked-layer experts through pinned staging and probe the pin limit
Under hybrid, layers whose banks the pin quota could not take keep GPU slots of their own pools and fetch part of their misses through pinned staging that CPU threads fill (--moe-locked-fetch, --moe-locked-max-fetch). --moe-cpu-layers auto without FREETOKEN_PIN_BUDGET_GB now loads the banks unpinned, pins the staging, then pins layers middle-out until CUDA refuses one and OS-locks the rest.
* fix(moe): resize the locked fetch on cache rebuild and gather staging by device address
* feat(moe): log decode moe stats per layer group with --moe-collect-stats
* perf(moe): default the locked fetch to max(2, min(4, logical cpus / 4)) copy threads
* feat(moe): take the locked fetch share from the ft bench bw profile by default
* fix(moe): fuse the locked fetch into publish and gather kernels and address review findings
* refactor(moe): gather the locked staging with fast_index_copy_multi
* refactor(moe): move the locked fetch setup out of the engine and address review findings
* fix(moe): fail the boot when the locked staging cannot be pinned and keep locked layers out of the hybrid share log
NVFP4 weight global scales (amax / 2688, 2e-5 to 4e-4 in the served checkpoints) sit at fp16's
normal minimum; stored as fp16, up to 97% of a checkpoint's expert globals were subnormal, with
up to 1.7e-3 relative error per row.
BREAKING CHANGE: an FTW checkpoint that stores NVFP4 global scales as fp16 is refused at load;
reconvert it with `ft checkpoint`.
* feat(moe): bench hybrid fetch fractions per moe layer kind
The ft bench bw profile (version 6) holds only layer_kinds rows, keyed by CPU format, options, expert shape, top_k and CPU worker threads; older profiles are ignored. ft bench bw --model also takes checkpoint paths and benches each of their layer kinds. Hybrid decode reads each layer's own row for its fetch fraction, else keeps a fixed cap of 1. --moe-strategy auto no longer upgrades to hybrid from the profile.
* fix(bench): fill gguf mxfp4 scales, scope act_block to ds_fp4 and skip cpu-less rows
* feat(moe): serve experts of several formats through per-format slot pools
* refactor(cpu-moe): pass each layer's cpu format with its options
* fix(moe): read only each format's layers when loading experts of several formats
---------
Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding
* build(deps): vendor the gguf-py reader
* feat(quant): add the GGUF quant kind and an embedding quant method
* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path
* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA
* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16
* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small
* fix(gguf): read F32, F16 and BF16 PLE tables without ggml
* fix(moe): pad the offload slot cache for ggml MMQ tail reads
* fix(gguf): support hip builds and runtime devices
* fix(gguf): copy same-type dequantize and check for missing ggml converters
* fix(gguf): load quantized qwen4exp ple_key and ple_value
* feat(gguf): run float parts of a fused GGUF linear as bf16 segments
* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights
* fix(moe): zero the prefill staging area after the last prefill layer
---------
Co-authored-by: zihaomu <zihaomu@outlook.com>
* feat(moe): split the offload expert cache into slot pools
* perf(moe): stop recompiling the hybrid ensure kernel for each slot pool size
* fix(engine): turn off prefill overlap when --moe-cache-mem leaves under two layers of slots
* fix(engine): load the --moe-cache-split profile before the expert banks
* perf(moe): split hybrid decode routes inside the ensure kernel
* refactor(moe): always split routes in the hybrid ensure kernel
* test(moe): drop the fetch split test left without its kernel check
* perf(moe): default fused gather to 32 blocks/bank on NVLink-C2C hosts
* build(aot): prebuild fast_index_copy_multi (1024, 32) variant
* perf(moe): make fused H2D gather grid configurable via env var
* docs: document FREETOKEN_H2D_BLOCKS_PER_BANK for NVLink-C2C hosts
* fix(moe): auto-detect H2D gather grid per device in the copy plan
* chore(docs): revised docs for NVLinkC2C
* feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks
Resident experts skip the state dict and load through the offload expert-bank loader (HF pieces or FTW banks) into per-layer GPU banks, so fused serves bf16, fp8_block, NVFP4 and MXFP4 alike. Unified-memory GPUs auto-select fused for these formats and ignore offload-only flags with a warning.
BREAKING CHANGE: ft checkpoint always writes MoE experts as banks; an FTW converted with --moe-backend fused or triton (dense experts) no longer loads and must be reconverted.
* docs(moe): name only GB10 / DGX Spark for unified memory
* refactor(quant): config, scheme and method layers for checkpoint quantization
QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.
- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
* fix(kernel): move fp8_block_scale_pad into aot_models to unbreak the kernel-cache build
* fix(kernel): exclude bank rows fast_index_copy cannot compile from aot specs
* fix(moe): fail fast when fused copy is off and a bank row cannot fall back
Serve Qwen3.8-Flash-Next (HF model_type qwen4_exp) text-only: 36 GDN +
12 QSA compressed-sparse attention layers on 4 hyper-connection residual
streams, a PLE n-gram embedding layer backed by a 47.7 GiB pinned-host
table with UVA gather, and 512 NVFP4 / block-fp8 routed experts (top-10)
plus a gated shared expert.
- attention: qsa_sparse backend (AttnType.QSA) over QSAKVCache -- paged
GQA K/V, a 1/ratio compressed index-key slab shadowing the KV pages,
and a per-request pending ring sized from index_ratio
- kvcache: declarative slot-sibling states (ModelConfig.slot_states) on
LinearStatePool carry the PLE conv history and n-gram context through
the hybrid-radix snapshot/COW lifecycle
- scheduler: hybrid prefill chunks align to the page size so snapshots
land on donatable boundaries
- kernels: triton kernels adapted from vLLM/SGLang (hc, qsa, ple gather,
moe router / shared gate) plus an original radix block top-k; int64
row addressing throughout
- moe: non-power-of-2 top-k router, deep-K marlin decode config, one
fp8 scale-bank padding rule shared with the AOT row table
- engine: the PLE table load reserves its pinned bytes from the pin
budget before the expert banks plan their residency