26 Commits
Author SHA1 Message Date
Xiaoze Fan 56e7f3ae38 feat(quant)!: compress bf16 weights to df11 at load with --online-quant (#891)
* feat(quant)!: compress bf16 weights to df11 at load with --online-quant

GLM-4.7 no longer hardwires DF11 attention and embedding; it serves them
bf16 unless --online-quant df11 is passed, and FTW files converted with the
old DF11 tensors must be reconverted.

* fix(quant): skip layers --online-quant cannot convert instead of failing
2026-11-11 13:24:46 -08:00
Xiaoze Fan fff1022a5c fix(quant): accept the modelopt_mixed quant method (#886) 2026-11-09 11:25:16 -08:00
Xiaoze Fan 33cbb3c54e feat(utils): add once-per-message logger methods (#858)
debug_once / info_once / warning_once log on rank 0, once per distinct message and args. select_kernel, which logged its choice once per layer, and the hand-rolled log-once guards now use them.
2026-11-03 15:31:08 -08:00
Mukkesh Ganesh ea2916d31c feat(gguf): serve gpt-oss gguf checkpoints on gpt_oss (#838)
* feat(gguf): serve gpt-oss gguf checkpoints on gpt_oss

* fix(gguf): refuse missing gpt-oss bias tensors, honor swa pattern, scale nvfp4 before bias

* fix(gguf): honor gpt-oss yarn truncate, gate the bf16 float cast, scale nvfp4 before the down bias
2026-11-02 08:58:07 -08:00
Chen Cheng ecbbce4577 fix(moe): skip inactive gguf routes in hybrid decode (#823)
* fix(moe): skip inactive gguf routes in hybrid decode

* perf(moe): run the inactive-route memset and zeroing only in hybrid decode

Only hybrid decode hands the GGUF kernel -1 routes, so mul_mat_id takes an inactive flag
and MMQ does the ids_src1 memset and the zeroing pass only when it is set. Prefill and
fused / offload decode go back to the main-branch MMQ calls.

* perf(moe): compare expert ids unsigned in mm_ids_helper

A negative (inactive) id compared unsigned sorts after every expert and equals none, so the
helper needs no remap to INT_MAX; the remap made nvcc unroll its token loop half as far.
2026-10-28 22:28:13 -07:00
Xiaoze Fan fd863cce3e feat: add --embed-weights host for a pinned host embedding table (#806)
* feat: add --embed-weights host for a pinned host embedding table

* refactor: check the tie from the model config and share the op walk
2026-10-23 14:41:37 -07:00
Manish Biswal 20d5d93132 feat(gguf): read NVFP4 token_embd rows in the vendored get_rows (#801)
* feat(gguf): read NVFP4 token_embd rows in the vendored get_rows

Signed-off-by: iamanishx <manishbiswal754@gmail.com>

* fix(tests)

Signed-off-by: iamanishx <manishbiswal754@gmail.com>

---------

Signed-off-by: iamanishx <manishbiswal754@gmail.com>
2026-10-22 20:19:24 -07:00
cherry77-cloud 7974c9cc8c perf(moe): skip inactive block-fp8 decode routes (#665) 2026-10-22 15:43:34 -07:00
Mukkesh Ganesh cc56b6bdcf feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next (#721)
* feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next

Adds the llama.cpp glm5-next GGUF reader (unsloth GLM-5.3-Flash-GGUF, both
the glm5-next and the older glm5next arch spellings) on the GGUF quant
kind from #690:

- parse_gguf_config derives the text config from the GGUF KV metadata,
  with per-block attention.head_count_kv laying out the KDA/DSA hybrid,
  the trailing MTP block dropped like the HF loader drops it, and the
  MLA head widths from the *_length_mla keys.
- iter_gguf_weights maps every tensor to the glm5_next state dict and
  undoes the converter transforms: A_log stored as -exp(A_log) is
  inverted, the per-head attn_k_b/attn_v_b pair (k transposed) fuses
  into the dense kv_b_proj layout the MLA absorption reads, and the
  q|k|v conv kernels concatenate on the channel axis. The mixed-type
  fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a) rides #702 segments;
  the routed experts load as GGUF expert banks at their shipped types
  (unsloth dynamic quants: IQ3_S gate/up + IQ4_XS down, Q8_0 outlier
  layers pass through natively).
- passes quant_config to embed_tokens so the Q8_0 token_embd serves
  packed; registers Glm5NextGGUFForCausalLM and claims it in the AOT
  table.

Verified end to end against unsloth GLM-5.3-Flash-UD-IQ4_XS (5 split
shards): the full engine on a 32GB GPU serves coherent output with the
experts off host banks. Text-only, TP=1.

Part of #671.

* fix(gguf): refuse unread scale tensors in glm5-next ggufs

check_scales at config parse like the llama, qwen3 and gemma4 loaders
(review on #721): a .scale / .input_scale outside an NVFP4 weight now
fails with a clear error instead of loading silently unread.

* fix(gguf): serve the quantized tensors of unsloth dynamic glm5-next quants

unsloth's UD quants quantize tensors the AesSedAI-style files keep float:
the MLA kv_b halves (Q8_0), the mHC mixers and the indexer's wk /
weights_proj / compressor gate. kv_b_proj joins the registry's
unquantized_modules (glm_moe_dsa keeps its kv_b dense for the MLA
absorption for the same reason) and the reader dequantizes through the
vendored gguf-py reference on every float-expecting read; wq_b keeps
serving its checkpoint quant packed (attention.py keeps wk and
weights_proj dense). Regression test stores the kv_b halves Q8_0.

* fix(gguf): treat the glm5 pre-tokenizer spelling like glm4

llama-vocab.cpp maps both spellings to CHATGLM4 with ignore_merges;
unsloth writes glm4 and AesSedAI's writer glm5 for the same vocabulary.
Without the mapping a glm5 file loads but silently mis-tokenizes
(tok_compare: 22,840/153,725 vocab words differ from the HF reference;
0 everywhere with it).

* fix(gguf): size the mmq tail padding by ggml type and k

pad_rows added one zero row of k columns, which covers the bytes MMQ
reads past the last row only when k is at least half a padding unit
(512); glm5-next's KDA f_b / g_b (k = linear head dim 128, Q8_0 in
dynamic quants) are the first packed tensors below that and died at the
first mul_mat. pad_rows now takes the ggml type and returns enough whole
rows to cover mmq_tail_bytes for any (type, k), consistently in the
reader's fused(), the gguf linear and embedding params and the expert
host banks. Found serving the real unsloth UD-IQ4_XS: the dry load
never runs a forward.

* fix(gguf): take the older glm5-next writer variants in stride

Two long-tail writer variants the full check sweep found (the GSQ-RCO
lineage alone is ~57k downloads/month):

- the conv kernels with a trailing unit dim: llama.cpp's glm5-next
  loader tries that 4-D shape before the 3-D one, so trailing unit dims
  in ne are dropped on float reads (they carry no elements).
- attention.head_count_kv as the generic scalar (all blocks): wrong for
  the hybrid -- llama.cpp broadcasts it and then fails tensor creation
  -- so only the list is trusted and a scalar falls to the tensor-name
  census like the pre-list writer.

* fix(gguf): glm5-next review cleanups

- refuse instead of guessing metadata the model cannot express: the
  scalar attention.head_count_kv spelling (llama.cpp broadcasts it and
  then fails tensor creation; every public file carries the list),
  missing *_length_mla keys through need(), non-uniform
  swiglu_clamp_exp / swiglu_clamp_shexp, indexer.types with non-full
  layers, expert_gating_func other than sigmoid, and rope dims as a
  NotImplementedError instead of an assert.
- call check_scales directly; aot_models comment on its own line.
- drop the GGUF test files; coverage will be handled across families.
2026-10-19 23:20:40 -07:00
Xiaoze Fan 163d1de5c5 fix(qwen4_exp): read weights under the checkpoint's quant config (#759)
* fix(qwen4_exp): read weights under the checkpoint's quant config

- route each dense Linear tensor by its scheme's stored names, so per-channel fp8 scales (llm-compressor) load and fused parts concatenate per role
- name the routed NVFP4 experts by the checkpoint's dialect, so compressed-tensors weight_packed / weight_global_scale load
- accept a bf16 PLE table without a scale in the pinned and disk backends

* fix(quant): accept weight_scale as the modelopt mxfp8 scale name

ModelOpt exports store the same e8m0 codes as weight_scale_inv (MiniMax-M3) or weight_scale (Qwen3.8-Flash-Next-FP4-Local); Stored gains aliases and the scheme-driven readers match any of them.
2026-10-19 20:18:34 -07:00
Xiaoze Fan 0db5023b60 fix(quant)!: keep nvfp4 global scales in fp32 (#739)
NVFP4 weight global scales (amax / 2688, 2e-5 to 4e-4 in the served checkpoints) sit at fp16's
normal minimum; stored as fp16, up to 97% of a checkpoint's expert globals were subnormal, with
up to 1.7e-3 relative error per row.

BREAKING CHANGE: an FTW checkpoint that stores NVFP4 global scales as fp16 is refused at load;
reconvert it with `ft checkpoint`.
2026-10-19 12:33:43 -07:00
Xiaoze Fan 4d2e2e2104 perf(linear): run nvfp4 linears on nvfp4 activations on sm_120 (#737)
* perf(linear): run nvfp4 linears on nvfp4 activations on sm_120

* fix(linear): guard the b12x nvfp4 linear against inputs it mishandles

* fix(linear): use the largest per-part global in the w4a4 nvfp4 linears
2026-10-18 23:55:07 -07:00
Xiaoze Fan e4f69d9e23 perf(moe): prefill nvfp4 moe experts on nvfp4 activations on sm_120 (#736) 2026-10-18 20:10:51 -07:00
Xiaoze Fan 31c6ef4f77 feat(gguf): apply the weight scales of nvfp4 gguf tensors (#732) 2026-10-17 13:46:50 -07:00
Xiaoze Fan c8559fdff6 feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu (#704)
* feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu

* feat(cpu-moe): run float GGUF expert rows on the cpu through ggml-cpu's vec_dot

* feat(cpu-moe): build ggml-cpu for each llama.cpp x86 variant and run the best one the CPU supports
2026-10-16 22:28:34 -07:00
Xiaoze Fanandandy-yang-1 e8c6bf1123 feat(moe): serve experts of several formats through per-format slot pools (#700)
* feat(moe): serve experts of several formats through per-format slot pools

* refactor(cpu-moe): pass each layer's cpu format with its options

* fix(moe): read only each format's layers when loading experts of several formats

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-10-16 15:13:59 -07:00
Xiaoze Fan 3381b6eb0a feat(gguf): run experts whose gate_up and down mix float and quantized types (#702)
* feat(gguf): run experts whose gate_up and down mix float and quantized types

* fix(kernel): map an fp32 compute type to tl.float32 in fused_moe_kernel_triton

* fix(gguf): write float expert GEMM output in the activation dtype
2026-10-15 23:20:12 -07:00
Xiaoze Fanandzihaomu e5d3bd3faf feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels (#690)
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding

* build(deps): vendor the gguf-py reader

* feat(quant): add the GGUF quant kind and an embedding quant method

* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path

* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA

* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16

* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small

* fix(gguf): read F32, F16 and BF16 PLE tables without ggml

* fix(moe): pad the offload slot cache for ggml MMQ tail reads

* fix(gguf): support hip builds and runtime devices

* fix(gguf): copy same-type dequantize and check for missing ggml converters

* fix(gguf): load quantized qwen4exp ple_key and ple_value

* feat(gguf): run float parts of a fused GGUF linear as bf16 segments

* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights

* fix(moe): zero the prefill staging area after the last prefill layer

---------

Co-authored-by: zihaomu <zihaomu@outlook.com>
2026-10-15 16:33:16 -07:00
95ac86f215 feat(cpu-moe): serve block-fp8 experts on the cpu and hybrid backends (#662)
* feat(cpu-moe): block-FP8 expert GEMV (W8A16) for the cpu/hybrid backends

Block-fp8 checkpoints (Qwen3.5/3.6-FP8, DeepSeek-V3-style, GLM) had no CPU expert
path at all:

    NotImplementedError: --moe-backend cpu/hybrid ... supports
    ['bf16', 'ds_fp4', 'mxfp4_triton', 'nvfp4', 'q4_0'] formats,
    but this checkpoint's experts are 'fp8_block'

so every expert miss had to stream over PCIe at ~25 GB/s while the CPU sat idle.

The bank is fp8-e4m3 row-major [rows, K] with one bf16 scale per 128x128 block --
K contiguous per output row, unlike mxfp4's transposed bank, so it maps straight
onto a bf16 dot product. e4m3 -> bf16 is *exact* (3 mantissa bits into 7, and
bf16's exponent range covers e4m3's whole 2^-9..448 span), so the AVX-512 path
widens weights in-register and feeds `_mm512_dpbf16_ps` against the bf16
activations: identical products to the reference's fp32 math, differing only in
summation order -- the same latitude the bf16 dot already takes.

The normal range rebases the exponent with a shift and a constant +120; exp == 0
is subnormal (m * 2^-9) and does not obey that, so those lanes blend in from a
small table. AVX2 does the same in fp32; scalar goes through the existing 256-entry
e4m3 LUT.

`ft bench bw` gains an fp8_block CPU measurement (it could already build the banks).
Measured on main with this commit alone, `--dtype fp8_block --reps 5`, 2x Xeon Gold
6526Y:

                        CPU MoE      PCIe    ratio   backend picked
    main             not measurable  25.1     --     offload
    main + this      53.7 GB/s       25.1    2.14x   hybrid

Before this commit the format reports no CPU number, so the backend picker has
nothing to weigh and selects offload. The ratio is what decides the backend, so
adding the kernel is what lets a block-fp8 deployment use the CPU at all.

End to end on Qwen/Qwen3.6-35B-A3B-FP8, median of 3 generations at matched GPU
expert residency:

    residency   offload   hybrid    delta
    10%           43.94    50.94    +15.9%
    25%           82.65    55.12    -33%

The win is real but conditional: hybrid pays off once the miss rate is high enough
that PCIe is the bottleneck, and costs throughput when enough experts are resident
that the CPU becomes the limiter. See the PR discussion -- the ratio makes
`--moe-backend auto` select hybrid for every block-fp8 checkpoint, which is right
for the oversubscribed case this project targets and wrong for a model that nearly
fits, and the bench models bandwidth but not miss rate.

Correctness is checked against the Triton block-fp8 decode kernel on identical
banks and routing (bs 1/2/5), plus an exactness check on the e4m3 -> bf16 widening
over all 256 codes. A weight-format ABI probe (`max_weight_format_id`) makes a
stale prebuilt `.so` fail loudly instead of falling through to the wrong dequant
branch and returning silently wrong numbers.

Assisted-by: Claude

* fix(cpu-moe): honor padded fp8 scale strides

The scale banks' trailing dimension is padded so per-expert rows land on
a 16-byte boundary for the fused copy (kernel/aot_models.py:
fp8_block_scale_pad), so the row stride is not ceil(K/128). Recomputing
it from the unpadded block count reads scales at the wrong offset.

Qwen3.8 is the case in the tree: models/qwen3_5_moe/weight.py records
down_scale as 20x5 bf16 = 200 B, and 200 % 16 != 0 pads cols 5 -> 6.
The benchbw workload (H=2048, I=512) and DSV4-Flash (H=7168, I=2048)
both happen to give rows*cols divisible by 8, so neither trips it.

Read the stride off the tensor and pass it through to the kernel, and
let the shape check assert the unpadded extent is present rather than an
exact shape. The fixture now allocates padded banks, so the existing GPU
cross-check covers this too; the GPU kernel was already stride-aware.

* test(cpu-moe): add float64 parity reference and an avx512f fp8 tier

The existing GPU-vs-CPU check compares two fp32 accumulations at a 2e-2
tolerance, which cannot see accumulator precision loss. Add a float64
reference over mixed-sign, full-range e4m3 weights and zero-mean
activations, and force each ISA tier explicitly so the sweep is real.

Also add an avx512f tier between avx512bf16 and avx2, so CPUs with
AVX-512 but no BF16 extension (Skylake-SP through Ice Lake-SP) stop
falling back to 8-wide fp32.

Measured on Xeon Platinum 8368 (Ice Lake-SP, no avx512_bf16), three
repetitions each: +19% single-thread cache-resident (3.31 -> 3.94 GB/s,
disjoint intervals), but only +4% under `ft bench bw --dtype fp8` at 75
threads (60.4 -> 62.7 GB/s, overlapping intervals). At that thread count
the kernel is memory-bound, so the project's own benchmark cannot
resolve the tier; the single-thread gain is the honest figure.

* test(cpu-moe): cover fp8 dispatch and cpu-only regressions

The new tier shipped without test coverage: deleting the avx512f branch
from select_fp8dot left every test green, because isa_name() reports
select_dot() (the bf16 chain) rather than the fp8 dispatch. Injecting a
ragged-tail bug into dot_fp8_block_avx2 was invisible for the same
reason -- no test ran numerics under a forced tier.

Force each available fp8 tier through the float64 parity GEMV, expose
the selected fp8 tier so the assertion can see it, and make the padded
bank test run a GEMV instead of only constructing an executor.

Also move the module-level CUDA guard onto the tests that actually need
a GPU. A CPU MoE kernel's most relevant hosts are the ones without a
usable GPU, and all ten tests were skipping there.

* review: force each ISA tier in the fp8_block test, and stop the docstring overclaiming

The comparison test only varied batch size, so all three cases ran
whatever tier this machine auto-selects and the scalar / AVX2 /
AVX-512-BF16 fp8_block kernels were never compared against the GPU at
all.

Parametrise FREETOKEN_CPU_MOE_ISA over the four tiers. Setting the
variable alone would not be enough: pick_isa() caps *downward*, so asking
for a tier the CPU or build lacks silently returns a lower one and the
test would pass four times over the same kernel. The executor latches its
choice into isa_name() at construction, so read it back and skip when it
does not match what was asked for. On a machine with all four, that is 4
x 3 cases against a 8-expert, 256-hidden problem.

test_subnormal_and_sign_decode_exactly said it validated that "every e4m3
byte must widen to the value the reference LUT gives", but it never calls
the CPU kernel -- it checks that bf16 round-trips e4m3 losslessly and
that subnormals are finite and small. That is the premise the widening
rests on, which is worth pinning, but it is not the widening. Renamed to
test_bf16_carries_every_e4m3_byte and the docstring now says what the
body asserts.

Not run here: this file is skipif(not torch.cuda.is_available()), and
this machine has no GPU and no torch. Compiles clean and ruff reports the
same two pre-existing E741 findings before and after.

Assisted-by: Claude

* fix(moe): let fp8_block experts decode on the cpu executor

* fix(cpu-moe): build the bf16 fp8 tier only when the compiler has avx512bf16

* fix(bench): version the bw profile and ignore older ones

* perf(cpu-moe): read fp8 weights as fp16 bits in the cpu dot kernels

* refactor(cpu-moe): write the fp8_block bank checks like the other formats

* refactor(cpu-moe): name the fp8 cpu symbols fp8_block

* docs(server): list fp8_block in the --moe-cpu-layers help

---------

Co-authored-by: Gabriel A. Devenyi <gdevenyi@gmail.com>
Co-authored-by: ChenyuHe <hechenyu@zju.edu.cn>
2026-10-11 22:12:41 -07:00
ZihaoMu 9b585b72d0 fix(rocm): gate CUDA-only optional backends on ROCm (#134)
* fix(rocm): gate cuda-only optional kernel integrations

* test(rocm): narrow optional-backend regression coverage
2026-10-08 19:55:57 -07:00
Xiaoze Fanandandy-yang-1 127c85665b feat(deepseek-v41): support text and vision serving (#603)
* test(deepseek-v41): add independent text and vision reference

* feat(quant): support block-32 and unrounded inference outputs

* feat(attention): add csa2 packed cache and sparse backend

* feat(deepseek-v41): serve text and images through shared runtime

* perf(mhc): tune batch-stable reduction tiles

* perf(moe): autotune fp4 decode launch geometry

* perf(moe): admit recurring misses against actual lru victims

* revert: perf(moe): admit recurring misses against actual lru victims

This reverts commit 1652d200f0ebb825379cf44c28baf52971e3a461.

* perf(mhc): derive the reduction tile from the width, not a timing

* perf(moe): replace the ds_fp4 decode autotune with a fixed two-warp launch

* fix(deepseek-v41): fence engram staging against in-flight lookups

* fix(moe): return the cpu routed sum in fp32 and round it in python

* fix(deepseek-v41): keep the vision tower and aligner unquantized

* fix(scheduler): admit bounded-replay hits by the history their resume reads

* refactor(deepseek-v41): name the attention family after the model, not csa2

* refactor(kvcache): import the dsv41 pool only for dsv41 models

* refactor(deepseek-v41): name the replay switch --swa-decoder-replay

* fix(server): keep reasoning_effort a string on the wire

* refactor(models): keep the prefill replay hooks off BaseLLMModel

* refactor(deepseek-v41): drop the fp32 routed sum and batch-invariant prefill

* fix(moe): weight ds_fp4 routes in the down epilogue again

* docs(models): anchor the v4.1 replay paragraph on the tech report

* refactor(deepseek-v41): run the decoder source's narrowed prefill on Block

* refactor(kvcache): let the dsv41 pool own the history a resume keeps live

* refactor(deepseek-v41): resume bounded-replay hits by capping the prefix match

* refactor(deepseek-v41): describe the attention group as a dsv4 variant and derive its geometry from the args

* refactor(deepseek-v41): keep the engram table locator in the model package

* refactor(deepseek-v41): drop the fp32 lm head output

- return the head's bf16 logits, as DeepSeek-V4 does
- revert ParallelLMHead(out_dtype=) and LinearKernel.apply_out_dtype, which only this call used
- remove test_head_preserves_fp32_logit_margin
- cast test_vision's prefill logits to fp32 before comparing, as the harness does

* refactor(quant): carry the w4a8 activation block on MoEConfig

- drop ActDesc and QuantScheme.act; the scheme only describes stored tensors
- Fp8BlockConfig sets MoEConfig.act_block from weight_block_size for its fp4 experts via a layer_config hook
- the ds_fp4 GPU kernel and CPU executor read the block from the layer config and require it

* refactor(deepseek-v41): size the kv pool from the args, not a geometry copy

- delete kvcache/dsv41_geometry.py; the pool and cost model read dsv41_args by attribute, as the DSV4 pool reads dsv4_args
- source / ratio aliasing reads args.roles; the duplicated source and ratio validation is gone
- the cost model owns the tier row formats, row bytes and the private window layers
- the pool keeps the format and window/ratio checks, and exposes the formats and is_private_window to the attention backend

* refactor(kvcache): move the dsv4 pool into kvcache/dsv4

* refactor(kvcache): move the dsv41 pool into kvcache/dsv4

* chore(benchmarks): drop the dsv41 selection and fp8 linear benchmarks

* test(deepseek-v41): drop tests that repeat other coverage

* test(deepseek-v41): load the tiny model's resident experts as banks

* fix(server): parse deepseek-v4.1's space-led dsml tool calls

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-10-07 00:15:11 -07:00
Xiaoze Fan 189c8ad0c3 feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks (#601)
* feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks

Resident experts skip the state dict and load through the offload expert-bank loader (HF pieces or FTW banks) into per-layer GPU banks, so fused serves bf16, fp8_block, NVFP4 and MXFP4 alike. Unified-memory GPUs auto-select fused for these formats and ignore offload-only flags with a warning.

BREAKING CHANGE: ft checkpoint always writes MoE experts as banks; an FTW converted with --moe-backend fused or triton (dense experts) no longer loads and must be reconverted.

* docs(moe): name only GB10 / DGX Spark for unified memory
2026-10-04 13:51:55 -07:00
Xiaoze Fan f7dbab7f15 fix(quant): load ModelOpt exports that ship no input_scale (#462) 2026-09-13 17:50:48 -07:00
Xiaoze Fan 0ffd5c8b29 feat(qwen3_5_moe): read every checkpoint layout through the QuantConfig (#438)
* feat(qwen3_5_moe): read every checkpoint layout through the QuantConfig

NVFP4 dense layers now declare the input_scale their scheme carries, so an
FTW written by an older build from an NVFP4 dense checkpoint needs
scripts/ftw_hotfix.py before it loads again.

* fix(quant): address scheme reader review: hotfix dialect names, gemma4 cross-shard scales, ct block-fp8 experts
2026-09-10 23:18:53 -07:00
Xiaoze Fan fb7f732de0 refactor(quant): hand the checkpoint QuantConfig to the weight readers (#427) 2026-09-10 01:46:44 -07:00
Xiaoze Fan 477c8601b2 refactor(quant): config, scheme and method layers for quantization (#418)
* refactor(quant): config, scheme and method layers for checkpoint quantization

QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.

- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
2026-09-08 23:37:23 -07:00