Commit Graph
9 Commits
Author SHA1 Message Date
Xiaoze Fan 0db5023b60 fix(quant)!: keep nvfp4 global scales in fp32 (#739)
NVFP4 weight global scales (amax / 2688, 2e-5 to 4e-4 in the served checkpoints) sit at fp16's
normal minimum; stored as fp16, up to 97% of a checkpoint's expert globals were subnormal, with
up to 1.7e-3 relative error per row.

BREAKING CHANGE: an FTW checkpoint that stores NVFP4 global scales as fp16 is refused at load;
reconvert it with `ft checkpoint`.
2026-10-19 12:33:43 -07:00
Xiaoze Fanandandy-yang-1 e8c6bf1123 feat(moe): serve experts of several formats through per-format slot pools (#700)
* feat(moe): serve experts of several formats through per-format slot pools

* refactor(cpu-moe): pass each layer's cpu format with its options

* fix(moe): read only each format's layers when loading experts of several formats

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-10-16 15:13:59 -07:00
Xiaoze Fan 189c8ad0c3 feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks (#601)
* feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks

Resident experts skip the state dict and load through the offload expert-bank loader (HF pieces or FTW banks) into per-layer GPU banks, so fused serves bf16, fp8_block, NVFP4 and MXFP4 alike. Unified-memory GPUs auto-select fused for these formats and ignore offload-only flags with a warning.

BREAKING CHANGE: ft checkpoint always writes MoE experts as banks; an FTW converted with --moe-backend fused or triton (dense experts) no longer loads and must be reconverted.

* docs(moe): name only GB10 / DGX Spark for unified memory
2026-10-04 13:51:55 -07:00
Xiaoze Fan cade1a9e45 fix(checkpoint): serve and repair FTW checkpoints around the vision encoder (#486)
* fix(checkpoint): skip the vision tower before reading an FTW served text-only

- iter_ftw_weights takes a keep(name) predicate and drops the rejected entries before their bytes are read; load_weight passes the vision prefixes through it instead of discarding the tensors after the replay
- an FTW converted before its family served images holds no encoder tensors: the engine now refuses it with the reconvert / --text-model-only choice instead of failing on the first missing key
- ftw_tensor_names lists an index's names by kind; docs/models.md notes the pre-image-support FTWs

* feat(scripts): add the vision encoder to a pre-image-support FTW

- ftw_hotfix.py fetches the encoder's checkpoint tensors by byte range from the Hub or a local checkpoint into a scratch checkpoint dir and lets the family's own weight reader name and fuse them, so the repaired FTW holds exactly what the converter would have written
- every family with an encoder gains iter_vision_weights, the encoder-only reader behind load_vision_weight: the Qwen families and Gemma yield the tower as stored (Gemma without its clipped-linear wrapper), GLM and MiniMax reuse their tower readers, Muse fuses q/k/v as its dense pass does
- the engine's refusal of a tower-less FTW and the models.md note name the tool next to reconverting; docs/ftw-hotfix.md gains the repair row and a Hub example; the ignored-leftover prefixes follow VISION_KEY_PREFIXES
2026-09-15 12:43:09 -07:00
Xiaoze Fan 477c8601b2 refactor(quant): config, scheme and method layers for quantization (#418)
* refactor(quant): config, scheme and method layers for checkpoint quantization

QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.

- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
2026-09-08 23:37:23 -07:00
Lancer bd372b630a fix(checkpoint): FTW truncated shard raises OSError instead of silent garbage load (#25) 2026-08-23 17:53:27 -07:00
Xiaoze Fan 831d38a66b fix(moe): report the residency banks actually settle at 2026-08-23 23:06:56 +00:00
Xiaoze Fan c41833b717 feat(moe): per-layer host-bank residency -- split lock-CPU/pin-GPU layers, auto selection under capped pin quota 2026-08-23 21:02:59 +00:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00