Commit Graph
155 Commits
Author SHA1 Message Date
Xiaoze Fan fd410ee616 perf(moe): pipeline the nvfp4 prefill scale loads on geforce sm_86 / sm_89 (#735)
The pointer path of the Triton prefill GEMM loads the e4m3 scales one byte per thread, too small for cp.async, so every K step waits on that load and a CTA barrier. Read them as int32 words (4 scales each) on GeForce sm_86 / sm_89 so they join the pipeline.

- SCALE_WORDS constexpr in `_prefill_nvfp4_moe_kernel` (off by default)
- Triton prefill tuning as one record per (GPU class, cc); GeForce sm_86 / sm_89 turn scale words on, with 8 warps when K >= 2048, and sm_86 moves to BLOCK_SIZE_M 64 from 100 tokens per expert
2026-10-18 14:39:56 -07:00
Xiaoze Fan 8f49e0af98 feat(moe): bench hybrid fetch fractions per moe layer kind (#734)
* feat(moe): bench hybrid fetch fractions per moe layer kind

The ft bench bw profile (version 6) holds only layer_kinds rows, keyed by CPU format, options, expert shape, top_k and CPU worker threads; older profiles are ignored. ft bench bw --model also takes checkpoint paths and benches each of their layer kinds. Hybrid decode reads each layer's own row for its fetch fraction, else keeps a fixed cap of 1. --moe-strategy auto no longer upgrades to hybrid from the profile.

* fix(bench): fill gguf mxfp4 scales, scope act_block to ds_fp4 and skip cpu-less rows
2026-10-17 23:25:12 -07:00
Xiaoze Fanandandy-yang-1 ac6a15de75 perf(moe): speed up the nvfp4 moe prefill gemm on rtx gpus (#733)
* perf(moe): speed up the nvfp4 moe prefill gemm on rtx gpus

* perf(moe): keep the byte e2m1 decode at every tile size on h100

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-10-17 17:50:59 -07:00
Xiaoze Fan 31c6ef4f77 feat(gguf): apply the weight scales of nvfp4 gguf tensors (#732) 2026-10-17 13:46:50 -07:00
Xiaoze Fan 6b8f200454 feat(gguf): serve GLM-4.5, GLM-4.6 and GLM-4.7 GGUF checkpoints (#731) 2026-10-17 12:56:19 -07:00
Xiaoze Fan 11c1adb88f perf(kernel): split triton decode attention by context length and sm count (#729)
* perf(kernel): split triton decode attention by context length and sm count

* perf(kernel): merge triton decode splits a tile at a time

- stage 2 loads SPLIT_TILE splits per step and merges them with one
  online-softmax update instead of one split per iteration
- SPLIT_TILE is the launch's split cap rounded up to a power of two,
  at most 8192 fp32 elements and 64 splits; num_warps scales with it
2026-10-16 23:56:35 -07:00
Xiaoze Fan c8559fdff6 feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu (#704)
* feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu

* feat(cpu-moe): run float GGUF expert rows on the cpu through ggml-cpu's vec_dot

* feat(cpu-moe): build ggml-cpu for each llama.cpp x86 variant and run the best one the CPU supports
2026-10-16 22:28:34 -07:00
Xiaoze Fan 03c2c9c35e perf(kernel): rewrite the triton nvfp4 moe prefill gemm (#723)
* perf(kernel): rewrite the triton nvfp4 moe prefill gemm

* perf(moe): tune nvfp4 prefill tiles per gpu class

GeForce runs fp32-accumulate tensor MMA at half rate, so RTX parts want smaller BLOCK_SIZE_M than H100 and their own BLOCK_N / BLOCK_KB / stages.
Tables come from tile sweeps on RTX PRO 6000, RTX 5090, RTX 4090 and RTX 3080 Ti; H100-class GPUs and AMD keep the previous tuning.
2026-10-16 22:03:59 -07:00
Xiaoze Fan 4663b2b89e feat(gguf): serve MiniMax-M2 and MiniMax-M3 GGUF checkpoints (#725) 2026-10-16 19:01:30 -07:00
Xiaoze Fan 5d39d6408a feat(gguf): serve Muse-Glimmer GGUF checkpoints (#724) 2026-10-16 18:38:43 -07:00
Xiaoze Fan e0dbb7dee1 fix(gguf): set ignore_merges for llama 3 gguf tokenizers (#728) 2026-10-16 17:41:06 -07:00
Xiaoze Fan bc0e2feb7c perf(attention): copy triton metadata without blocking the host (#726) 2026-10-16 16:48:24 -07:00
Xiaoze Fan 21d9578bde feat(gguf): serve Llama and Mistral llama-arch GGUF checkpoints (#722) 2026-10-16 15:50:29 -07:00
Xiaoze Fanandandy-yang-1 e8c6bf1123 feat(moe): serve experts of several formats through per-format slot pools (#700)
* feat(moe): serve experts of several formats through per-format slot pools

* refactor(cpu-moe): pass each layer's cpu format with its options

* fix(moe): read only each format's layers when loading experts of several formats

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-10-16 15:13:59 -07:00
LZ_QAQ d5eb18febd fix(kernel): correct fp4 cache scale rounding (#720)
* fix(kernel): preserve fp4 scale rounding at subnormal ties

* fix(kernel): preserve native e4m3 scale calculation
2026-10-16 13:49:17 -07:00
Xiaoze Fan 3886ceeebf feat(gguf): serve Gemma 4 dense GGUF checkpoints (#706) 2026-10-16 13:05:08 -07:00
Xiaoze Fan 6c21dd5615 feat(gguf): serve Qwen2, Qwen3 and Qwen3 MoE GGUF checkpoints (#707) 2026-10-16 13:04:29 -07:00
Xiaoze Fan 9b68e00767 fix(engine): upload cuda graphs at capture so a stream wait cannot block the first launch (#697) 2026-10-15 23:23:37 -07:00
Xiaoze Fan 3381b6eb0a feat(gguf): run experts whose gate_up and down mix float and quantized types (#702)
* feat(gguf): run experts whose gate_up and down mix float and quantized types

* fix(kernel): map an fp32 compute type to tl.float32 in fused_moe_kernel_triton

* fix(gguf): write float expert GEMM output in the activation dtype
2026-10-15 23:20:12 -07:00
Xiaoze Fan 18448d29a7 feat(gguf): serve Qwen3.5 dense and MoE GGUF checkpoints (#701) 2026-10-15 22:15:28 -07:00
Xiaoze Fan df8d450c28 fix(gguf): reject ggml weights past the 32-bit block offsets of MMQ / MMVQ (#699) 2026-10-15 21:25:32 -07:00
Xiaoze Fan b6193e86f7 build(deps): upgrade transformers to 5.18 (#698) 2026-10-15 20:35:02 -07:00
Zhang Jason 7d6490c5d6 fix(rocm): clamp QSA KV tile to the device LDS budget on RDNA (#587)
The qsa_sparse_paged_attention tiles are tuned for the GB300's 228KB shared
memory; RDNA gfx11/gfx12 have ~64KB/CU, so k+v staging overflows LDS. Query the
device's actual shared-memory limit and shrink the KV tile only when it doesn't
fit, ROCm-gated so NVIDIA tiles stay byte-identical.

The gguf ROCm-compile fixes this branch also carried are now covered upstream by
the ggml-cuda vendoring in #690 (separate DEFAULT_HIP_CFLAGS + vendors/hip.h), so
this branch reduces to the QSA clamp only.
2026-10-15 20:34:20 -07:00
Xiaoze Fanandzihaomu e5d3bd3faf feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels (#690)
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding

* build(deps): vendor the gguf-py reader

* feat(quant): add the GGUF quant kind and an embedding quant method

* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path

* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA

* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16

* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small

* fix(gguf): read F32, F16 and BF16 PLE tables without ggml

* fix(moe): pad the offload slot cache for ggml MMQ tail reads

* fix(gguf): support hip builds and runtime devices

* fix(gguf): copy same-type dequantize and check for missing ggml converters

* fix(gguf): load quantized qwen4exp ple_key and ple_value

* feat(gguf): run float parts of a fused GGUF linear as bf16 segments

* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights

* fix(moe): zero the prefill staging area after the last prefill layer

---------

Co-authored-by: zihaomu <zihaomu@outlook.com>
2026-10-15 16:33:16 -07:00
LZ_QAQ 0bca88f70c fix(rocm): reuse staged moe alignment on Triton 3.8 (#691)
* fix(rocm): reuse staged moe alignment below triton 3.9

* fix(rocm): restrict moe alignment workaround to Triton 3.8
2026-10-15 02:21:32 -07:00
Xiaoze Fan b416f85c40 feat(moe): split the offload expert cache into slot pools (#689)
* feat(moe): split the offload expert cache into slot pools

* perf(moe): stop recompiling the hybrid ensure kernel for each slot pool size

* fix(engine): turn off prefill overlap when --moe-cache-mem leaves under two layers of slots

* fix(engine): load the --moe-cache-split profile before the expert banks
2026-10-14 19:44:55 -07:00
Xiaoze Fan f02fbb16a2 perf(moe): split hybrid decode routes inside the ensure kernel (#685)
* perf(moe): split hybrid decode routes inside the ensure kernel

* refactor(moe): always split routes in the hybrid ensure kernel

* test(moe): drop the fetch split test left without its kernel check
2026-10-13 20:30:34 -07:00
Xiaoze Fanandzihaomu 2010271266 refactor(cpu-moe): split the executor into weight formats and gpu sync backends (#670)
* refactor(cpu-moe): split the executor into weight formats and gpu sync backends

- move each expert weight format into kernels/<fmt>.cpp behind the MoeFormat
  interface in format.h; the executor keeps the threads, the work split and
  the pass 2 route sum
- pass each format its banks and integer options by name; MoeParams keeps only
  the shared shape and activation fields
- key the formats by name in one table that weight_formats() exports, replacing
  the integer format ids and the fp8_block_isa_name hook
- replace the per-(layer, batch size) flag slots, pinned IO buffers and task
  table with one doorbell word MoeTask{layer, num_tokens} and one shared pinned
  IO block of max_tokens rows
- move the GPU/CPU handshake behind GpuSync with a CUDA and a HIP backend picked
  by setup.py; HIP waits on signal memory and adds explicit batch-memop graph
  nodes where the runtime does not record the native memops under capture
- raise on graph capture when the handshake is off and host functions are not
  replay-safe
- allocate format scratch on 64-byte boundaries
- size max_tokens by the explicit cuda_graph_bs as well

Co-authored-by: zihaomu <zihaomu@outlook.com>

* perf(cpu-moe): skip the worker pool for tasks without routed experts

* perf(cpu-moe): publish and combine the cpu handshake with kernels

* fix(cpu-moe): clean up failed hip capability probes

---------

Co-authored-by: zihaomu <zihaomu@outlook.com>
2026-10-13 20:12:58 -07:00
95ac86f215 feat(cpu-moe): serve block-fp8 experts on the cpu and hybrid backends (#662)
* feat(cpu-moe): block-FP8 expert GEMV (W8A16) for the cpu/hybrid backends

Block-fp8 checkpoints (Qwen3.5/3.6-FP8, DeepSeek-V3-style, GLM) had no CPU expert
path at all:

    NotImplementedError: --moe-backend cpu/hybrid ... supports
    ['bf16', 'ds_fp4', 'mxfp4_triton', 'nvfp4', 'q4_0'] formats,
    but this checkpoint's experts are 'fp8_block'

so every expert miss had to stream over PCIe at ~25 GB/s while the CPU sat idle.

The bank is fp8-e4m3 row-major [rows, K] with one bf16 scale per 128x128 block --
K contiguous per output row, unlike mxfp4's transposed bank, so it maps straight
onto a bf16 dot product. e4m3 -> bf16 is *exact* (3 mantissa bits into 7, and
bf16's exponent range covers e4m3's whole 2^-9..448 span), so the AVX-512 path
widens weights in-register and feeds `_mm512_dpbf16_ps` against the bf16
activations: identical products to the reference's fp32 math, differing only in
summation order -- the same latitude the bf16 dot already takes.

The normal range rebases the exponent with a shift and a constant +120; exp == 0
is subnormal (m * 2^-9) and does not obey that, so those lanes blend in from a
small table. AVX2 does the same in fp32; scalar goes through the existing 256-entry
e4m3 LUT.

`ft bench bw` gains an fp8_block CPU measurement (it could already build the banks).
Measured on main with this commit alone, `--dtype fp8_block --reps 5`, 2x Xeon Gold
6526Y:

                        CPU MoE      PCIe    ratio   backend picked
    main             not measurable  25.1     --     offload
    main + this      53.7 GB/s       25.1    2.14x   hybrid

Before this commit the format reports no CPU number, so the backend picker has
nothing to weigh and selects offload. The ratio is what decides the backend, so
adding the kernel is what lets a block-fp8 deployment use the CPU at all.

End to end on Qwen/Qwen3.6-35B-A3B-FP8, median of 3 generations at matched GPU
expert residency:

    residency   offload   hybrid    delta
    10%           43.94    50.94    +15.9%
    25%           82.65    55.12    -33%

The win is real but conditional: hybrid pays off once the miss rate is high enough
that PCIe is the bottleneck, and costs throughput when enough experts are resident
that the CPU becomes the limiter. See the PR discussion -- the ratio makes
`--moe-backend auto` select hybrid for every block-fp8 checkpoint, which is right
for the oversubscribed case this project targets and wrong for a model that nearly
fits, and the bench models bandwidth but not miss rate.

Correctness is checked against the Triton block-fp8 decode kernel on identical
banks and routing (bs 1/2/5), plus an exactness check on the e4m3 -> bf16 widening
over all 256 codes. A weight-format ABI probe (`max_weight_format_id`) makes a
stale prebuilt `.so` fail loudly instead of falling through to the wrong dequant
branch and returning silently wrong numbers.

Assisted-by: Claude

* fix(cpu-moe): honor padded fp8 scale strides

The scale banks' trailing dimension is padded so per-expert rows land on
a 16-byte boundary for the fused copy (kernel/aot_models.py:
fp8_block_scale_pad), so the row stride is not ceil(K/128). Recomputing
it from the unpadded block count reads scales at the wrong offset.

Qwen3.8 is the case in the tree: models/qwen3_5_moe/weight.py records
down_scale as 20x5 bf16 = 200 B, and 200 % 16 != 0 pads cols 5 -> 6.
The benchbw workload (H=2048, I=512) and DSV4-Flash (H=7168, I=2048)
both happen to give rows*cols divisible by 8, so neither trips it.

Read the stride off the tensor and pass it through to the kernel, and
let the shape check assert the unpadded extent is present rather than an
exact shape. The fixture now allocates padded banks, so the existing GPU
cross-check covers this too; the GPU kernel was already stride-aware.

* test(cpu-moe): add float64 parity reference and an avx512f fp8 tier

The existing GPU-vs-CPU check compares two fp32 accumulations at a 2e-2
tolerance, which cannot see accumulator precision loss. Add a float64
reference over mixed-sign, full-range e4m3 weights and zero-mean
activations, and force each ISA tier explicitly so the sweep is real.

Also add an avx512f tier between avx512bf16 and avx2, so CPUs with
AVX-512 but no BF16 extension (Skylake-SP through Ice Lake-SP) stop
falling back to 8-wide fp32.

Measured on Xeon Platinum 8368 (Ice Lake-SP, no avx512_bf16), three
repetitions each: +19% single-thread cache-resident (3.31 -> 3.94 GB/s,
disjoint intervals), but only +4% under `ft bench bw --dtype fp8` at 75
threads (60.4 -> 62.7 GB/s, overlapping intervals). At that thread count
the kernel is memory-bound, so the project's own benchmark cannot
resolve the tier; the single-thread gain is the honest figure.

* test(cpu-moe): cover fp8 dispatch and cpu-only regressions

The new tier shipped without test coverage: deleting the avx512f branch
from select_fp8dot left every test green, because isa_name() reports
select_dot() (the bf16 chain) rather than the fp8 dispatch. Injecting a
ragged-tail bug into dot_fp8_block_avx2 was invisible for the same
reason -- no test ran numerics under a forced tier.

Force each available fp8 tier through the float64 parity GEMV, expose
the selected fp8 tier so the assertion can see it, and make the padded
bank test run a GEMV instead of only constructing an executor.

Also move the module-level CUDA guard onto the tests that actually need
a GPU. A CPU MoE kernel's most relevant hosts are the ones without a
usable GPU, and all ten tests were skipping there.

* review: force each ISA tier in the fp8_block test, and stop the docstring overclaiming

The comparison test only varied batch size, so all three cases ran
whatever tier this machine auto-selects and the scalar / AVX2 /
AVX-512-BF16 fp8_block kernels were never compared against the GPU at
all.

Parametrise FREETOKEN_CPU_MOE_ISA over the four tiers. Setting the
variable alone would not be enough: pick_isa() caps *downward*, so asking
for a tier the CPU or build lacks silently returns a lower one and the
test would pass four times over the same kernel. The executor latches its
choice into isa_name() at construction, so read it back and skip when it
does not match what was asked for. On a machine with all four, that is 4
x 3 cases against a 8-expert, 256-hidden problem.

test_subnormal_and_sign_decode_exactly said it validated that "every e4m3
byte must widen to the value the reference LUT gives", but it never calls
the CPU kernel -- it checks that bf16 round-trips e4m3 losslessly and
that subnormals are finite and small. That is the premise the widening
rests on, which is worth pinning, but it is not the widening. Renamed to
test_bf16_carries_every_e4m3_byte and the docstring now says what the
body asserts.

Not run here: this file is skipif(not torch.cuda.is_available()), and
this machine has no GPU and no torch. Compiles clean and ruff reports the
same two pre-existing E741 findings before and after.

Assisted-by: Claude

* fix(moe): let fp8_block experts decode on the cpu executor

* fix(cpu-moe): build the bf16 fp8 tier only when the compiler has avx512bf16

* fix(bench): version the bw profile and ignore older ones

* perf(cpu-moe): read fp8 weights as fp16 bits in the cpu dot kernels

* refactor(cpu-moe): write the fp8_block bank checks like the other formats

* refactor(cpu-moe): name the fp8 cpu symbols fp8_block

* docs(server): list fp8_block in the --moe-cpu-layers help

---------

Co-authored-by: Gabriel A. Devenyi <gdevenyi@gmail.com>
Co-authored-by: ChenyuHe <hechenyu@zju.edu.cn>
2026-10-11 22:12:41 -07:00
cherry77-cloud c8186396f5 fix(sampling): apply presence and frequency penalties (#527)
* fix(sampling): apply presence and frequency penalties

* fix(server): reject non-finite sampling penalties

* fix(server): bound presence and frequency penalties
2026-10-11 12:41:07 -07:00
Chen Cheng ee9789ff6c fix(kernel): avoid nan in triton sampling softmax when a chunk starts with -inf (#654) 2026-10-10 01:17:57 -07:00
Xiaoze Fan 9528190454 feat(server): add --api-key authentication (#639)
* feat(server): require an API key on every route but /health

* feat(shell): send the server's API key

* feat(ctl): send the server's API key

* feat(launch): hand the server's API key to the agent

* feat(daemon): pass the serve's API key as apiKey

The daemon reaches the serve's /v1/stats and prepare-stop itself, so the key
travels in the start request, reaches the serve through its environment and is
kept in the state file for re-adoption. The daemon's own FREETOKEN_API_KEY is no
longer passed down to the serve.

* fix(launch): keep the API key out of the variable dsh's web search reads

dsh's web search reads DEEPSEEK_API_KEY and sends it to api.deepseek.com, so
dsh now reads the key from FREETOKEN_API_KEY through the apiKeyEnv of its
llm-deepseek section, and DEEPSEEK_API_KEY stays the dummy.

* fix(server): check the API key on OPTIONS requests

* fix(launch): write agent configs that hold the API key owner-only

* chore: drop comments that restate the code
2026-10-10 00:36:04 -07:00
Xiaoze Fan 2d8b9cc1ea perf(kernel): avoid the activation layout conversion in fp8 moe decode and read mxfp4 gemv weights as column pairs (#653) 2026-10-09 00:15:07 -07:00
ZihaoMu 9b585b72d0 fix(rocm): gate CUDA-only optional backends on ROCm (#134)
* fix(rocm): gate cuda-only optional kernel integrations

* test(rocm): narrow optional-backend regression coverage
2026-10-08 19:55:57 -07:00
Mukkesh Ganesh 23fc922bf8 feat(deepseek_v4): serve image input on the V4-Flash-Vision release (#502)
* feat(deepseek_v4): serve image input on the V4-Flash-Vision release

The block layout depends on where the image lands: lead pads align its first row token to
a 4-token compression stride, so the item hash carries the layout and two alignments never
share a radix key.

The block's tokens all carry one content pad id (the shared pad scheme); the sparse
attention reads its spans from batch.mm_spans, which the scheduler recovers from the flat
block ends its existing plan already reports.

* fix(mm): keep the shared mmprocessor signature and own the dsv4 offset in its apply

* fix(models): build dsv4 image spans in the model and start them at image_start

* fix(models): mount the dsv4 vision tower under visual and keep its weights unquantized

* fix(models): keep dsv4 image blocks whole across chunked prefill
2026-10-08 14:42:39 -07:00
Xiaoze Fanandzihaomu 803a3b4802 fix(kernel): correct jit tensor matcher checks and pdl arch guards (#638)
* fix(kernel): correct jit matcher options and pdl arch guards

- with_dtype<Ts...>(sym) and with_device<Codes...>(sym) applied no options, so their dtype and device checks accepted anything
- index and store list kDLROCM, which the ignored check let through
- emit griddepcontrol only for !FREETOKEN_USE_ROCM && __CUDA_ARCH__ >= 900; it failed ptxas below sm_90 and does not exist on HIP
- declare the launch attribute cache only on CUDA
- use kWarpThreads for the index and store launch geometry

Co-authored-by: zihaomu <zihaomu@outlook.com>

* fix(kernel): treat zero as a matched size in tensor matcher

- a zero dimension counted as unset, so empty inputs failed with "Size value is not set" and a zero-row tensor did not catch a mismatched one

* fix(kernel): return early on empty index and store batches

- an empty batch launched a zero-sized grid, which CUDA and HIP reject; the leftover error then failed the next GPU call

Co-authored-by: zihaomu <zihaomu@outlook.com>

---------

Co-authored-by: zihaomu <zihaomu@outlook.com>
2026-10-08 13:31:44 -07:00
Xiaoze Fan 078132480e perf(kernel): skip masks and out-of-window tiles in triton extend attention (#632)
* perf(kernel): skip masks and out-of-window tiles in triton extend attention

* perf(kernel): drop the sm90-only register cap and k/v pipelining in triton extend attention
2026-10-08 00:36:55 -07:00
Xiaoze Fan 1c3aa0aa4c perf(kernel): keep 16-byte kv loads in dsv4 sparse attention and unroll gated pool (#633) 2026-10-07 23:06:22 -07:00
Xiaoze Fan 799bec39f6 perf(kernel): read dsv41 indexer key rows as int32 words (#634) 2026-10-07 23:04:18 -07:00
Xiaoze Fan 76449847b6 perf(kernel): avoid layout conversions in nvfp4 marlin decode (#631) 2026-10-07 22:42:44 -07:00
Xiaoze Fanandandy-yang-1 127c85665b feat(deepseek-v41): support text and vision serving (#603)
* test(deepseek-v41): add independent text and vision reference

* feat(quant): support block-32 and unrounded inference outputs

* feat(attention): add csa2 packed cache and sparse backend

* feat(deepseek-v41): serve text and images through shared runtime

* perf(mhc): tune batch-stable reduction tiles

* perf(moe): autotune fp4 decode launch geometry

* perf(moe): admit recurring misses against actual lru victims

* revert: perf(moe): admit recurring misses against actual lru victims

This reverts commit 1652d200f0ebb825379cf44c28baf52971e3a461.

* perf(mhc): derive the reduction tile from the width, not a timing

* perf(moe): replace the ds_fp4 decode autotune with a fixed two-warp launch

* fix(deepseek-v41): fence engram staging against in-flight lookups

* fix(moe): return the cpu routed sum in fp32 and round it in python

* fix(deepseek-v41): keep the vision tower and aligner unquantized

* fix(scheduler): admit bounded-replay hits by the history their resume reads

* refactor(deepseek-v41): name the attention family after the model, not csa2

* refactor(kvcache): import the dsv41 pool only for dsv41 models

* refactor(deepseek-v41): name the replay switch --swa-decoder-replay

* fix(server): keep reasoning_effort a string on the wire

* refactor(models): keep the prefill replay hooks off BaseLLMModel

* refactor(deepseek-v41): drop the fp32 routed sum and batch-invariant prefill

* fix(moe): weight ds_fp4 routes in the down epilogue again

* docs(models): anchor the v4.1 replay paragraph on the tech report

* refactor(deepseek-v41): run the decoder source's narrowed prefill on Block

* refactor(kvcache): let the dsv41 pool own the history a resume keeps live

* refactor(deepseek-v41): resume bounded-replay hits by capping the prefix match

* refactor(deepseek-v41): describe the attention group as a dsv4 variant and derive its geometry from the args

* refactor(deepseek-v41): keep the engram table locator in the model package

* refactor(deepseek-v41): drop the fp32 lm head output

- return the head's bf16 logits, as DeepSeek-V4 does
- revert ParallelLMHead(out_dtype=) and LinearKernel.apply_out_dtype, which only this call used
- remove test_head_preserves_fp32_logit_margin
- cast test_vision's prefill logits to fp32 before comparing, as the harness does

* refactor(quant): carry the w4a8 activation block on MoEConfig

- drop ActDesc and QuantScheme.act; the scheme only describes stored tensors
- Fp8BlockConfig sets MoEConfig.act_block from weight_block_size for its fp4 experts via a layer_config hook
- the ds_fp4 GPU kernel and CPU executor read the block from the layer config and require it

* refactor(deepseek-v41): size the kv pool from the args, not a geometry copy

- delete kvcache/dsv41_geometry.py; the pool and cost model read dsv41_args by attribute, as the DSV4 pool reads dsv4_args
- source / ratio aliasing reads args.roles; the duplicated source and ratio validation is gone
- the cost model owns the tier row formats, row bytes and the private window layers
- the pool keeps the format and window/ratio checks, and exposes the formats and is_private_window to the attention backend

* refactor(kvcache): move the dsv4 pool into kvcache/dsv4

* refactor(kvcache): move the dsv41 pool into kvcache/dsv4

* chore(benchmarks): drop the dsv41 selection and fp8 linear benchmarks

* test(deepseek-v41): drop tests that repeat other coverage

* test(deepseek-v41): load the tiny model's resident experts as banks

* fix(server): parse deepseek-v4.1's space-led dsml tool calls

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-10-07 00:15:11 -07:00
Xiaoze Fanandshubham3-ucb 1c183e74ae perf(moe): spin-wait the cpu moe pool between layers (#619)
Co-authored-by: shubham3-ucb <229030647+shubham3-ucb@users.noreply.github.com>
2026-10-06 21:38:10 -07:00
SuperHotDogCat c1a504a50b perf(moe): Improved fast_index_copy_multi_jit for NVLink-C2C hosts (#610)
* perf(moe): default fused gather to 32 blocks/bank on NVLink-C2C hosts

* build(aot): prebuild fast_index_copy_multi (1024, 32) variant

* perf(moe): make fused H2D gather grid configurable via env var

* docs: document FREETOKEN_H2D_BLOCKS_PER_BANK for NVLink-C2C hosts

* fix(moe): auto-detect H2D gather grid per device in the copy plan

* chore(docs): revised docs for NVLinkC2C
2026-10-06 19:54:38 -07:00
cherry77-cloud 555efd8944 fix(kvcache): clone inserted radix keys to release request buffers (#559) 2026-10-05 15:29:44 -07:00
William Xinze Zheng 93b0a59164 fix(server): keep Anthropic inline system messages in place for prefix reuse (#608)
* Implement fix for Claude Code's inline system message

* Fix for Muse
2026-10-05 14:59:59 -07:00
云微 581fdca310 fix(server): report the backend kv page size in stats (#599)
* fix(server): report the backend kv page size in stats

Use resolved backend cache metadata, retaining the frontend fallback.

Refs FlashML-org/FreeToken#598
Assisted-by: Codex

* test(server): cover resolved kv page size and fallback

Refs FlashML-org/FreeToken#598
Assisted-by: Codex
2026-10-05 00:24:33 -07:00
Xiaoze Fan 189c8ad0c3 feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks (#601)
* feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks

Resident experts skip the state dict and load through the offload expert-bank loader (HF pieces or FTW banks) into per-layer GPU banks, so fused serves bf16, fp8_block, NVFP4 and MXFP4 alike. Unified-memory GPUs auto-select fused for these formats and ignore offload-only flags with a warning.

BREAKING CHANGE: ft checkpoint always writes MoE experts as banks; an FTW converted with --moe-backend fused or triton (dense experts) no longer loads and must be reconverted.

* docs(moe): name only GB10 / DGX Spark for unified memory
2026-10-04 13:51:55 -07:00
Xiaoze Fanandandy-yang-1 d3512b43af fix(ple): fence disk staging against in-flight lookups (#588)
Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-10-01 16:59:45 -07:00
YevheniiKotyrlo 235e2016dd feat(rotary): size the rope table for YaRN and add --hf-overrides (#573)
* feat(rotary): size the rope table for YaRN and add --hf-overrides

* fix(rotary): guard mrope's table width and copy --hf-overrides values

- MRotaryEmbedding asserts its cos/sin table is rotary_dim wide. A proportional
  rope's table spans the head, and the mrope kernel indexes the rotary_dim // 2
  section table with the table's width.
- _with_overrides deep-copies every value it stores: transformers fills rope
  defaults into the rope_parameters it is handed, which was the caller's dict.
- The rope-table tests the engine test covers fold into it, the two shared-table
  tests are one, and the --hf-overrides test moves to tests/utils/test_hf.py,
  where it loads the override against the same values written into config.json,
  for a model type transformers knows and one it does not.
- docs/cli.md lists --hf-overrides.

* fix(server): give --hf-overrides a fresh dict per parse

argparse hands its default object to every parse that omits the flag, so
default={} put one shared dict in every ServerArgs. The flag defaults to None and
becomes a new dict where the arguments are assembled, as --mm-processor-kwargs
does.

* docs(rotary): an indexer's heads differ in width from the attention's, not wider
2026-10-01 00:40:27 -07:00
YevheniiKotyrlo 5031e3d578 perf(qsa): reduce block top-k candidates in levels past 16 splits (#571) 2026-09-29 16:29:17 -07:00