Commit Graph
83 Commits
Author SHA1 Message Date
Xiaoze Fan ef6b303304 fix(ggml): cap the MoE route sort's shared memory at the device limit (#878) 2026-11-07 23:30:06 -08:00
Xiaoze Fan 08cee44468 feat(spec): add DFlash 2 speculative decoding (#857)
* feat(spec): hand the pending token to Proposer.propose and return a SpecProposal

* feat(engine): Sampler.probs and sparse-q rejection

* feat(spec): let a proposer declare slot states riding the GDN slots

* feat(models): export layer outputs to the feature sink in the qwen3_5 and qwen4_exp families

* feat(attention): record the source slot of each tracked request

* feat(kernel): DFlash2 draft kernels

* feat(models): DFlash2 draft family

* feat(spec): DFlashWindow

* feat(spec): DFlashProposer

* docs(cli): add dflash to the speculative decoding section

* refactor(engine): share the sampling backend choice and drop the extra renorm in Sampler.probs

* fix(models): build the DFlash rope table uncached and refuse draft scalings it does not apply

* refactor(spec): share the draft path check between the MTP and DFlash proposers

* test(e2e): compare rebuilt outputs exactly only without hybrid MoE

* refactor(spec): keep proposer slot states on EngineConfig instead of replacing model_config

* perf(spec): compute the verify distribution once for sparse-q rejection

* perf(models): let the feature sink add the residual on the rows it keeps

* perf(models): share the DFlash draft K/V weights with the ring projection

* fix(kernel): keep the dflash selector off zero-probability candidates

* test(spec): drop the DFlash tests that the draft forward and Monte Carlo tests already cover

* fix(spec): default and cap DFlash steps at the draft's block_size - 1

* fix(models): reject DFlash draft configs the draft forward cannot serve

* test(e2e): read the MoE strategy from the rebuild test's extra args

* docs(spec): say that slot-state owners write their own prefill track snapshot

* test(models): move the DFlash draft parity tests under tests/models/dflash

* refactor(spec): drop the no-op verify handler of the DFlash window

* refactor(models): fold the DFlash candidate top-k into propose

* refactor(spec): declare the draft step limits through Proposer.num_steps only

* refactor(kernel): make the DFlash ring attention stateless and keep its workspace on the ring

* refactor(models): name the DFlash draft submodules like the other model families

* refactor(spec): move SpecProposal next to DraftContext in spec/proposer.py

* refactor(spec): hand the verify the draft distribution over the whole vocabulary

* refactor(spec): call the DFlashRing object dflash_ring and its length ring_len

* refactor(models): split the DFlash draft's propose into compute_candidates and sample_path

* refactor(spec): keep the draft distribution on its support in SpecProposal

* fix(spec): drop the step's proposal once the verify has read it

* refactor(spec): name the DFlash feature sinks DFlashPrefillSink and DFlashVerifySink

* refactor(models): read the declared Batch.features directly instead of through getattr

* test(dflash): check the window rebuild inside the block forward test

* test(spec): share the plain run of the verify contract and run the DFlash draft once per family

* test(kvcache): drop the window cases from the pool tests and check a layerless state's bytes

* test(spec): check the verify probs of each row against a torch top-k and top-p reference

* test(e2e): add an accept-len floor and a prefill chunk knob to the AIME gate

* refactor(scheduler): let callers take the speculative counts from the status reporter

* test(spec): keep the shared plain runs of the verify contract in their own temp dirs

* test(models): give the ricursive test's stand-in batch a features field
2026-11-07 12:29:18 -08:00
Chen Cheng 14119b8cbb feat(ricursive): add ripple-65b preview support (#871)
* feat(ricursive): add ripple-65b preview support

* fix(ricursive): read swiglu_clamp as a per-layer gguf array

llama.cpp's Ripple converter (ggml-org/llama.cpp#31692) writes
ricursive.swiglu_clamp with one value per layer, while the first
community GGUFs store a scalar. Accept both and refuse per-layer
values that differ, since the model has a single clamp.

* fix(ricursive): add ripple's own parsers and copy the dense reader
2026-11-06 16:17:21 -08:00
Chen Cheng 5209c8c327 perf(cpu-moe): give only the routes the cpu computes work items in a hybrid step (#866) 2026-11-06 11:36:59 -08:00
Xiaoze Fan 60059b41fb chore(ggml): replace freetoken.patch with the llama.cpp pin (#873)
The patch repeats every edit to a copied file and nothing checks that the two agree. The pin and the tree reproduce it. scripts/sync-llama-cpp.sh diffs the copy against the pin and merges a newer llama.cpp from it, and docs/ggml-kernels.md says how to maintain the copy.
2026-11-05 15:13:56 -08:00
Xiaoze Fan 650dd6f6fe perf(ggml): sort MoE MMQ routes with a counting sort (#872)
* perf(ggml): sort MoE MMQ routes with a counting sort

* perf(ggml): fold the MoE MMQ tile map into the route sort
2026-11-05 13:16:03 -08:00
LZ_QAQ 628b88ff79 fix(attention): avoid overflow in triton kv cache offsets (#854)
* fix(attention): avoid overflow in triton kv cache offsets

* fix(attention): simplify kv offset dtype selection
2026-11-04 00:09:40 -08:00
ZihaoMuandLZ-QWQ 9cacd0896c fix(moe): publish rocm cpu expert inputs before notification (#764)
* fix(moe): prevent stale cpu expert inputs in rocm graphs

* fix(moe): publish rocm cpu inputs with graph events

* fix(moe): qualify rocm kernel io with ordered copy fallback

* fix(moe): enable locked hybrid staging on rocm

---------

Co-authored-by: LZ-QWQ <lz1830837889@163.com>
2026-11-02 21:29:42 -08:00
Xiaoze Fan 3d23551c5b perf(cpu-moe): speed up the nvfp4 W4A8 kernels (#851)
The VNNI dots were compute bound per core; with fewer instructions per weight they reach the DRAM read bandwidth.
The W4A8 kernel now follows FREETOKEN_CPU_MOE_ISA.
2026-11-02 15:05:17 -08:00
Xiaoze Fan dacac5f5fa feat(spec): add MTP drafts for the Qwen3.5 family and Qwen3.8-Flash-Next (#844)
* feat(spec): add the MTP proposer for the qwen3_5 family

* test(models): read the qwen3_5 MTP head of bf16 and block-fp8 checkpoints

* feat(spec): verify qwen4_exp targets on qsa_sparse graphs with PLE verify handlers

* feat(spec): add the qwen4_exp MTP draft from the checkpoint or a Strata pack

* docs(cli): document the speculative decoding flags

* fix(spec): refuse MTP draft sources the draft cannot read before the target loads

* docs(qwen4_exp): describe the verify path in the PLE docstrings

* fix(spec): refuse a missing relative MTP draft path before the target loads

* fix(moe): read MTP head experts on the host and require both stacked tensors

* refactor(spec): tell the final prefill chunk from prompt_len instead of the scheduler's ChunkedReq

* refactor(spec): let each MTP draft family check its own weight source

* fix(qwen4_exp): size the disk PLE eager staging for decode and verify rows

* perf(kernel): read the KV-index gather's chunk count from the grid, not a constexpr

* refactor(moe): let each family name its MTP expert tensors

* refactor(spec): force the MTP draft's MoE resident in the proposer and check it after load

* fix(spec): refuse MTP with dummy weights

* refactor(spec): move HiddenRelay under spec/mtp, its only user

* refactor(spec): drop MTPDraft.config, fold its embed hooks into embed_tokens

* refactor(models): note FeatureSink's graph binding, drop its runtime_checkable

* fix(models): read quantized MTP fc projections under the quant config
2026-11-02 11:37:09 -08:00
Mukkesh Ganesh ea2916d31c feat(gguf): serve gpt-oss gguf checkpoints on gpt_oss (#838)
* feat(gguf): serve gpt-oss gguf checkpoints on gpt_oss

* fix(gguf): refuse missing gpt-oss bias tensors, honor swa pattern, scale nvfp4 before bias

* fix(gguf): honor gpt-oss yarn truncate, gate the bf16 float cast, scale nvfp4 before the down bias
2026-11-02 08:58:07 -08:00
Chen Cheng ecbbce4577 fix(moe): skip inactive gguf routes in hybrid decode (#823)
* fix(moe): skip inactive gguf routes in hybrid decode

* perf(moe): run the inactive-route memset and zeroing only in hybrid decode

Only hybrid decode hands the GGUF kernel -1 routes, so mul_mat_id takes an inactive flag
and MMQ does the ids_src1 memset and the zeroing pass only when it is set. Prefill and
fused / offload decode go back to the main-branch MMQ calls.

* perf(moe): compare expert ids unsigned in mm_ids_helper

A negative (inactive) id compared unsigned sorts after every expert and equals none, so the
helper needs no remap to INT_MAX; the remap made nvcc unroll its token loop half as far.
2026-10-28 22:28:13 -07:00
Xiaoze Fan c2c3e828e8 feat(spec): verify draft tokens on the target side (#836)
* feat(spec): add the speculative verify infrastructure

* feat(scheduler): run decode steps as speculative verify steps

* chore(engine): mark the speculative decoding config fields with one comment

* feat(server): add short aliases for the speculative decoding flags
2026-10-28 21:18:02 -07:00
Chen Cheng d2977fbc13 feat(kernel): add an iq1_m mmq kernel for moe steps above 8 tokens (#835) 2026-10-28 20:48:55 -07:00
festeraeb c94535c909 fix(attention): size the decode KV tile to the shared-memory budget (#840)
Assisted-by: Claude (Anthropic)
2026-10-28 20:05:27 -07:00
Jesús Jódar Piernas 2d93085545 feat(flint): add nvfp4 modelopt and gguf support (#797)
* feat(flint): add nvfp4 modelopt and gguf support

* refactor(flint): address final review scope requests
2026-10-24 12:37:57 -07:00
LZ_QAQ 32056cf460 fix(ple): add rocm disk wait-sync support (#776)
* fix(ple): add rocm disk wait-sync support

* fix(kernel): validate hip wait-sync support before enabling it

* refactor(ple): share decode fills and sync flag allocation
2026-10-23 15:35:47 -07:00
Xiaoze Fan fd863cce3e feat: add --embed-weights host for a pinned host embedding table (#806)
* feat: add --embed-weights host for a pinned host embedding table

* refactor: check the tie from the model config and share the op walk
2026-10-23 14:41:37 -07:00
Chen Chengandzihaomu 6505b6a005 fix(kernel): convert fp32 to e4m3 in one rounding on sm_89 (#805)
* fix(kernel): convert fp32 to e4m3 in one rounding on sm_89

* fix(kernel): avoid nvidia fp8 cast asm on hip

---------

Co-authored-by: zihaomu <zihaomu@outlook.com>
2026-10-23 14:27:34 -07:00
Mukkesh Ganesh 0c50eb9d82 perf(attention): drop redundant activation copies in glm5 dsa prefill (#795)
* perf(attention): drop redundant activation copies in glm5 dsa prefill

* fix(attention): guard the dsa sparse kernels' int32 q/o addressing
2026-10-23 12:39:33 -07:00
cherry77-cloud 507a8bfa81 fix(rotary): preserve fp32 inputs in mrope torch fallbacks (#641) 2026-10-23 00:11:06 -07:00
Manish Biswal 20d5d93132 feat(gguf): read NVFP4 token_embd rows in the vendored get_rows (#801)
* feat(gguf): read NVFP4 token_embd rows in the vendored get_rows

Signed-off-by: iamanishx <manishbiswal754@gmail.com>

* fix(tests)

Signed-off-by: iamanishx <manishbiswal754@gmail.com>

---------

Signed-off-by: iamanishx <manishbiswal754@gmail.com>
2026-10-22 20:19:24 -07:00
Mukkesh Ganesh ed359eee0a feat(gguf): serve qwen3vl and qwen3vlmoe gguf checkpoints text only (#807) 2026-10-22 18:14:03 -07:00
Xiaoze Fan 19a60a5b65 perf(moe): fuse sigmoid + selection-bias routing into a triton kernel (#804)
* perf(moe): fuse sigmoid + selection-bias routing into a triton kernel

BREAKING CHANGE: glm_moe_dsa keeps its router selection bias in fp32. An FTW converted by an older build stores it as bf16 and still loads, but routes with the rounded bias; restore it with `scripts/ftw_hotfix.py --repo nvidia/GLM-5.2-NVFP4` or reconvert.

* fix(hotfix): scope the bias restore to GLM-5.2 and drop new router comments
2026-10-22 17:48:21 -07:00
cherry77-cloud 7974c9cc8c perf(moe): skip inactive block-fp8 decode routes (#665) 2026-10-22 15:43:34 -07:00
Xiaoze Fan 609191d81f feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj (#803)
* feat(gguf): load the vision tower from a llama.cpp mmproj

Qwen3.5 / Qwen3.6 / Qwen3.8 (qwen3vl_merger) and Gemma-4 26B-A4B / 31B (gemma4v) / 12B (gemma4uv) take image input from a separate mmproj file: --mmproj, else one in the model's own folder. The tower runs in bf16 (quantized mmproj files are dequantized) and its config comes from the GGUF metadata plus per-arch defaults.

* docs(gguf): describe mmproj image input without a model list

* test(gguf): drop the mmproj tests

* fix(gguf): escape the model folder when globbing for mmproj files

* fix(gguf): skip auto-discovered mmproj files the text model cannot use

* fix(gguf): serve text from a GGUF that carries vision tensors itself
2026-10-22 14:56:19 -07:00
Mukkesh Ganesh 546c5fe2e4 feat(gguf): serve DeepSeek-V4 Flash and Pro GGUF checkpoints on deepseek4 (#774)
* feat(gguf): serve DeepSeek-V4-Flash GGUF checkpoints on deepseek4

* fix(gguf): refuse reap-style per-layer expert prunes in deepseek4 ggufs

* fix(gguf): apply the deepseek4 review fixes

- route the offload experts through swiglu_clamp so the GGUF MoE path applies the limit
- map model.embed -> embed on the HF row so the bf16 exemption matches
- default a wholly missing rope.scaling block to the family yarn values with a warning;
  refuse a partly present one
2026-10-21 18:40:37 -07:00
LZ_QAQ cc4aa085b6 fix(moe): enable prefill hit-d2d on rocm (#771)
* fix(kernel): order the batch-memcpy probe against the current stream

Adapt the stream dependency and busy-stream regression from #414
(https://github.com/FlashML-org/FreeToken/pull/414). Warm the JIT
before queuing the backlog so compilation cannot mask the race.

* fix(moe): enable prefill hit-d2d on rocm
2026-10-20 23:15:17 -07:00
Xiaoze Fan c5b5b7243a feat(moe): fetch locked-layer experts through pinned staging (#742)
* feat(moe): fetch locked-layer experts through pinned staging and probe the pin limit

Under hybrid, layers whose banks the pin quota could not take keep GPU slots of their own pools and fetch part of their misses through pinned staging that CPU threads fill (--moe-locked-fetch, --moe-locked-max-fetch). --moe-cpu-layers auto without FREETOKEN_PIN_BUDGET_GB now loads the banks unpinned, pins the staging, then pins layers middle-out until CUDA refuses one and OS-locks the rest.

* fix(moe): resize the locked fetch on cache rebuild and gather staging by device address

* feat(moe): log decode moe stats per layer group with --moe-collect-stats

* perf(moe): default the locked fetch to max(2, min(4, logical cpus / 4)) copy threads

* feat(moe): take the locked fetch share from the ft bench bw profile by default

* fix(moe): fuse the locked fetch into publish and gather kernels and address review findings

* refactor(moe): gather the locked staging with fast_index_copy_multi

* refactor(moe): move the locked fetch setup out of the engine and address review findings

* fix(moe): fail the boot when the locked staging cannot be pinned and keep locked layers out of the hybrid share log
2026-10-20 14:59:05 -07:00
Mukkesh Ganesh cc56b6bdcf feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next (#721)
* feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next

Adds the llama.cpp glm5-next GGUF reader (unsloth GLM-5.3-Flash-GGUF, both
the glm5-next and the older glm5next arch spellings) on the GGUF quant
kind from #690:

- parse_gguf_config derives the text config from the GGUF KV metadata,
  with per-block attention.head_count_kv laying out the KDA/DSA hybrid,
  the trailing MTP block dropped like the HF loader drops it, and the
  MLA head widths from the *_length_mla keys.
- iter_gguf_weights maps every tensor to the glm5_next state dict and
  undoes the converter transforms: A_log stored as -exp(A_log) is
  inverted, the per-head attn_k_b/attn_v_b pair (k transposed) fuses
  into the dense kv_b_proj layout the MLA absorption reads, and the
  q|k|v conv kernels concatenate on the channel axis. The mixed-type
  fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a) rides #702 segments;
  the routed experts load as GGUF expert banks at their shipped types
  (unsloth dynamic quants: IQ3_S gate/up + IQ4_XS down, Q8_0 outlier
  layers pass through natively).
- passes quant_config to embed_tokens so the Q8_0 token_embd serves
  packed; registers Glm5NextGGUFForCausalLM and claims it in the AOT
  table.

Verified end to end against unsloth GLM-5.3-Flash-UD-IQ4_XS (5 split
shards): the full engine on a 32GB GPU serves coherent output with the
experts off host banks. Text-only, TP=1.

Part of #671.

* fix(gguf): refuse unread scale tensors in glm5-next ggufs

check_scales at config parse like the llama, qwen3 and gemma4 loaders
(review on #721): a .scale / .input_scale outside an NVFP4 weight now
fails with a clear error instead of loading silently unread.

* fix(gguf): serve the quantized tensors of unsloth dynamic glm5-next quants

unsloth's UD quants quantize tensors the AesSedAI-style files keep float:
the MLA kv_b halves (Q8_0), the mHC mixers and the indexer's wk /
weights_proj / compressor gate. kv_b_proj joins the registry's
unquantized_modules (glm_moe_dsa keeps its kv_b dense for the MLA
absorption for the same reason) and the reader dequantizes through the
vendored gguf-py reference on every float-expecting read; wq_b keeps
serving its checkpoint quant packed (attention.py keeps wk and
weights_proj dense). Regression test stores the kv_b halves Q8_0.

* fix(gguf): treat the glm5 pre-tokenizer spelling like glm4

llama-vocab.cpp maps both spellings to CHATGLM4 with ignore_merges;
unsloth writes glm4 and AesSedAI's writer glm5 for the same vocabulary.
Without the mapping a glm5 file loads but silently mis-tokenizes
(tok_compare: 22,840/153,725 vocab words differ from the HF reference;
0 everywhere with it).

* fix(gguf): size the mmq tail padding by ggml type and k

pad_rows added one zero row of k columns, which covers the bytes MMQ
reads past the last row only when k is at least half a padding unit
(512); glm5-next's KDA f_b / g_b (k = linear head dim 128, Q8_0 in
dynamic quants) are the first packed tensors below that and died at the
first mul_mat. pad_rows now takes the ggml type and returns enough whole
rows to cover mmq_tail_bytes for any (type, k), consistently in the
reader's fused(), the gguf linear and embedding params and the expert
host banks. Found serving the real unsloth UD-IQ4_XS: the dry load
never runs a forward.

* fix(gguf): take the older glm5-next writer variants in stride

Two long-tail writer variants the full check sweep found (the GSQ-RCO
lineage alone is ~57k downloads/month):

- the conv kernels with a trailing unit dim: llama.cpp's glm5-next
  loader tries that 4-D shape before the 3-D one, so trailing unit dims
  in ne are dropped on float reads (they carry no elements).
- attention.head_count_kv as the generic scalar (all blocks): wrong for
  the hybrid -- llama.cpp broadcasts it and then fails tensor creation
  -- so only the list is trusted and a scalar falls to the tensor-name
  census like the pre-list writer.

* fix(gguf): glm5-next review cleanups

- refuse instead of guessing metadata the model cannot express: the
  scalar attention.head_count_kv spelling (llama.cpp broadcasts it and
  then fails tensor creation; every public file carries the list),
  missing *_length_mla keys through need(), non-uniform
  swiglu_clamp_exp / swiglu_clamp_shexp, indexer.types with non-full
  layers, expert_gating_func other than sigmoid, and rope dims as a
  NotImplementedError instead of an assert.
- call check_scales directly; aot_models comment on its own line.
- drop the GGUF test files; coverage will be handled across families.
2026-10-19 23:20:40 -07:00
Xiaoze Fan 0db5023b60 fix(quant)!: keep nvfp4 global scales in fp32 (#739)
NVFP4 weight global scales (amax / 2688, 2e-5 to 4e-4 in the served checkpoints) sit at fp16's
normal minimum; stored as fp16, up to 97% of a checkpoint's expert globals were subnormal, with
up to 1.7e-3 relative error per row.

BREAKING CHANGE: an FTW checkpoint that stores NVFP4 global scales as fp16 is refused at load;
reconvert it with `ft checkpoint`.
2026-10-19 12:33:43 -07:00
Xiaoze Fan 4d2e2e2104 perf(linear): run nvfp4 linears on nvfp4 activations on sm_120 (#737)
* perf(linear): run nvfp4 linears on nvfp4 activations on sm_120

* fix(linear): guard the b12x nvfp4 linear against inputs it mishandles

* fix(linear): use the largest per-part global in the w4a4 nvfp4 linears
2026-10-18 23:55:07 -07:00
Xiaoze Fan e4f69d9e23 perf(moe): prefill nvfp4 moe experts on nvfp4 activations on sm_120 (#736) 2026-10-18 20:10:51 -07:00
Xiaoze Fan fd410ee616 perf(moe): pipeline the nvfp4 prefill scale loads on geforce sm_86 / sm_89 (#735)
The pointer path of the Triton prefill GEMM loads the e4m3 scales one byte per thread, too small for cp.async, so every K step waits on that load and a CTA barrier. Read them as int32 words (4 scales each) on GeForce sm_86 / sm_89 so they join the pipeline.

- SCALE_WORDS constexpr in `_prefill_nvfp4_moe_kernel` (off by default)
- Triton prefill tuning as one record per (GPU class, cc); GeForce sm_86 / sm_89 turn scale words on, with 8 warps when K >= 2048, and sm_86 moves to BLOCK_SIZE_M 64 from 100 tokens per expert
2026-10-18 14:39:56 -07:00
Xiaoze Fanandandy-yang-1 ac6a15de75 perf(moe): speed up the nvfp4 moe prefill gemm on rtx gpus (#733)
* perf(moe): speed up the nvfp4 moe prefill gemm on rtx gpus

* perf(moe): keep the byte e2m1 decode at every tile size on h100

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-10-17 17:50:59 -07:00
Xiaoze Fan 31c6ef4f77 feat(gguf): apply the weight scales of nvfp4 gguf tensors (#732) 2026-10-17 13:46:50 -07:00
Xiaoze Fan 6b8f200454 feat(gguf): serve GLM-4.5, GLM-4.6 and GLM-4.7 GGUF checkpoints (#731) 2026-10-17 12:56:19 -07:00
Xiaoze Fan 11c1adb88f perf(kernel): split triton decode attention by context length and sm count (#729)
* perf(kernel): split triton decode attention by context length and sm count

* perf(kernel): merge triton decode splits a tile at a time

- stage 2 loads SPLIT_TILE splits per step and merges them with one
  online-softmax update instead of one split per iteration
- SPLIT_TILE is the launch's split cap rounded up to a power of two,
  at most 8192 fp32 elements and 64 splits; num_warps scales with it
2026-10-16 23:56:35 -07:00
Xiaoze Fan c8559fdff6 feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu (#704)
* feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu

* feat(cpu-moe): run float GGUF expert rows on the cpu through ggml-cpu's vec_dot

* feat(cpu-moe): build ggml-cpu for each llama.cpp x86 variant and run the best one the CPU supports
2026-10-16 22:28:34 -07:00
Xiaoze Fan 03c2c9c35e perf(kernel): rewrite the triton nvfp4 moe prefill gemm (#723)
* perf(kernel): rewrite the triton nvfp4 moe prefill gemm

* perf(moe): tune nvfp4 prefill tiles per gpu class

GeForce runs fp32-accumulate tensor MMA at half rate, so RTX parts want smaller BLOCK_SIZE_M than H100 and their own BLOCK_N / BLOCK_KB / stages.
Tables come from tile sweeps on RTX PRO 6000, RTX 5090, RTX 4090 and RTX 3080 Ti; H100-class GPUs and AMD keep the previous tuning.
2026-10-16 22:03:59 -07:00
Xiaoze Fan 4663b2b89e feat(gguf): serve MiniMax-M2 and MiniMax-M3 GGUF checkpoints (#725) 2026-10-16 19:01:30 -07:00
Xiaoze Fan 5d39d6408a feat(gguf): serve Muse-Glimmer GGUF checkpoints (#724) 2026-10-16 18:38:43 -07:00
Xiaoze Fan 21d9578bde feat(gguf): serve Llama and Mistral llama-arch GGUF checkpoints (#722) 2026-10-16 15:50:29 -07:00
Xiaoze Fanandandy-yang-1 e8c6bf1123 feat(moe): serve experts of several formats through per-format slot pools (#700)
* feat(moe): serve experts of several formats through per-format slot pools

* refactor(cpu-moe): pass each layer's cpu format with its options

* fix(moe): read only each format's layers when loading experts of several formats

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-10-16 15:13:59 -07:00
LZ_QAQ d5eb18febd fix(kernel): correct fp4 cache scale rounding (#720)
* fix(kernel): preserve fp4 scale rounding at subnormal ties

* fix(kernel): preserve native e4m3 scale calculation
2026-10-16 13:49:17 -07:00
Xiaoze Fan 6c21dd5615 feat(gguf): serve Qwen2, Qwen3 and Qwen3 MoE GGUF checkpoints (#707) 2026-10-16 13:04:29 -07:00
Xiaoze Fan 3381b6eb0a feat(gguf): run experts whose gate_up and down mix float and quantized types (#702)
* feat(gguf): run experts whose gate_up and down mix float and quantized types

* fix(kernel): map an fp32 compute type to tl.float32 in fused_moe_kernel_triton

* fix(gguf): write float expert GEMM output in the activation dtype
2026-10-15 23:20:12 -07:00
Xiaoze Fan df8d450c28 fix(gguf): reject ggml weights past the 32-bit block offsets of MMQ / MMVQ (#699) 2026-10-15 21:25:32 -07:00
Zhang Jason 7d6490c5d6 fix(rocm): clamp QSA KV tile to the device LDS budget on RDNA (#587)
The qsa_sparse_paged_attention tiles are tuned for the GB300's 228KB shared
memory; RDNA gfx11/gfx12 have ~64KB/CU, so k+v staging overflows LDS. Query the
device's actual shared-memory limit and shrink the KV tile only when it doesn't
fit, ROCm-gated so NVIDIA tiles stay byte-identical.

The gguf ROCm-compile fixes this branch also carried are now covered upstream by
the ggml-cuda vendoring in #690 (separate DEFAULT_HIP_CFLAGS + vendors/hip.h), so
this branch reduces to the QSA clamp only.
2026-10-15 20:34:20 -07:00
Xiaoze Fanandzihaomu e5d3bd3faf feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels (#690)
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding

* build(deps): vendor the gguf-py reader

* feat(quant): add the GGUF quant kind and an embedding quant method

* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path

* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA

* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16

* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small

* fix(gguf): read F32, F16 and BF16 PLE tables without ggml

* fix(moe): pad the offload slot cache for ggml MMQ tail reads

* fix(gguf): support hip builds and runtime devices

* fix(gguf): copy same-type dequantize and check for missing ggml converters

* fix(gguf): load quantized qwen4exp ple_key and ple_value

* feat(gguf): run float parts of a fused GGUF linear as bf16 segments

* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights

* fix(moe): zero the prefill staging area after the last prefill layer

---------

Co-authored-by: zihaomu <zihaomu@outlook.com>
2026-10-15 16:33:16 -07:00