PyTorch < 2.12 runs row-wise FP8 _scaled_mm on sm_89 through a CUTLASS stream-K kernel whose launch ignored the current stream (pytorch/pytorch#177651, fixed by pytorch/pytorch@252bb4a in 2.12). FreeToken issues the fused per-tensor-FP8 projections (q/k/v and GDN qkv|z of the NVFP4 checkpoints) from a side stream, so on Ada every prefill of >= 256 tokens stalled the GPU and the worker hung or died (#182, #72, #220). Windows torch builds ship no row-wise kernel at all (#227).
Tensor-wise scaling is unaffected. Where row-wise is unsafe (sm_89 on torch < 2.12, or a probe on the default stream raises), a fused projection now runs one tensor-wise GEMM per part over its row slice and concatenates: the same W8A8 scheme (rel ~7e-4 to row-wise, accumulation order), one extra launch per part. The parts' row ranges come from the load-time weight_scale run-lengths; the decision and its probe run at load, never under CUDA-graph capture. FREETOKEN_FP8_ROWWISE_MM=0/1 forces either path for A/B.
Tested on RTX 4070 SUPER (sm_89), driver 591.86, torch 2.11.0+cu130, WSL2. Sweep over M on a side stream: row-wise stalls at M >= 256, the new path completes at every M. tests/kernels/test_fp8_pertensor_linear.py: the side-stream test fails on main (rc=124, 0/128 GEMMs complete) and passes here; the per-part path is compared directly against row-wise at M=1/4/64/300. Three pre-existing test_w8a8_matches_w8a8_reference cases miss the 1e-2 tolerance on this GPU (rel 0.0103-0.0107) on main and on this branch alike.
Assisted-by: Claude Fable 5
Serve Qwen3.8-Flash-Next (HF model_type qwen4_exp) text-only: 36 GDN +
12 QSA compressed-sparse attention layers on 4 hyper-connection residual
streams, a PLE n-gram embedding layer backed by a 47.7 GiB pinned-host
table with UVA gather, and 512 NVFP4 / block-fp8 routed experts (top-10)
plus a gated shared expert.
- attention: qsa_sparse backend (AttnType.QSA) over QSAKVCache -- paged
GQA K/V, a 1/ratio compressed index-key slab shadowing the KV pages,
and a per-request pending ring sized from index_ratio
- kvcache: declarative slot-sibling states (ModelConfig.slot_states) on
LinearStatePool carry the PLE conv history and n-gram context through
the hybrid-radix snapshot/COW lifecycle
- scheduler: hybrid prefill chunks align to the page size so snapshots
land on donatable boundaries
- kernels: triton kernels adapted from vLLM/SGLang (hc, qsa, ple gather,
moe router / shared gate) plus an original radix block top-k; int64
row addressing throughout
- moe: non-power-of-2 top-k router, deep-K marlin decode config, one
fp8 scale-bank padding rule shared with the AOT row table
- engine: the PLE table load reserves its pinned bytes from the pin
budget before the expert banks plan their residency
* feat(server)!: add --gpu to choose the GPU on multi-GPU machines
--gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index.
It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>.
/v1/stats reports the engine's GPU under "gpus".
ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name.
The daemon's /bench/profile returns the running serve's GPU profile.
BREAKING CHANGE: ft checkpoint --device is removed; use --gpu.
* refactor!: resolve --gpu via NVML and bind by UUID, not CUDA_VISIBLE_DEVICES
BREAKING CHANGE: ft bench bw --device is removed; use --gpu.
* refactor: name the id namespaces and drop the multi-device e4m3 scan
Splits the published id into _assigned_physical (UUID) and
_assigned_visible (CUDA ordinal). One process runs on one GPU, so
e4m3_native() judges that card instead of scanning every visible device.
The request ring only carried whole-request duration, so /v1/stats exposed no
first-token latency — the number that says how responsive a local model feels,
independent of decode speed. Streaming generations now stamp the first delta
they emit; non-streaming rows carry None and are skipped by the mean, so a
/health probe can never drag it toward zero.
Surfaced as requests.ttft_mean_ms in /v1/stats and in `ft ctl stats`.