6 Commits
Author SHA1 Message Date
Chris Qian 0bf7dbbf23 fix(models): discover safetensors shards from the index (#819)
* fix(models): find index-only checkpoints that keep shards in a subdirectory

* fix(models): make the checkpoint index authoritative for shard discovery

The index is read directly so safetensors_weight_map stays the only
header-based reader, and _host_ram_fits_parallel sizes the shards the
reader will actually open, so the low-RAM serial fallback fires again
for subdirectory layouts.

* fix(models): fall back to root shards when the index is unusable

Share one index parser between iter_weight_files and safetensors_weight_map
instead of a third copy; warn and glob the root when the index is broken,
absent or empty; reject index keys that escape the model folder; let the hub
download fetch the index itself; tell the truth in the diagnose docstring.

* revert(models): back out the defensive index fallback, keep the no-index test

Per review: a broken index is a broken checkpoint, so the loader keeps the plain one rule -- use the index if there is one, otherwise glob the root. The no-index fallback test stays as the only coverage of the root glob and the consolidated.safetensors rule.
2026-10-24 19:10:52 -07:00
Xiaoze Fan 0db5023b60 fix(quant)!: keep nvfp4 global scales in fp32 (#739)
NVFP4 weight global scales (amax / 2688, 2e-5 to 4e-4 in the served checkpoints) sit at fp16's
normal minimum; stored as fp16, up to 97% of a checkpoint's expert globals were subnormal, with
up to 1.7e-3 relative error per row.

BREAKING CHANGE: an FTW checkpoint that stores NVFP4 global scales as fp16 is refused at load;
reconvert it with `ft checkpoint`.
2026-10-19 12:33:43 -07:00
Xiaoze Fan 0ffd5c8b29 feat(qwen3_5_moe): read every checkpoint layout through the QuantConfig (#438)
* feat(qwen3_5_moe): read every checkpoint layout through the QuantConfig

NVFP4 dense layers now declare the input_scale their scheme carries, so an
FTW written by an older build from an NVFP4 dense checkpoint needs
scripts/ftw_hotfix.py before it loads again.

* fix(quant): address scheme reader review: hotfix dialect names, gemma4 cross-shard scales, ct block-fp8 experts
2026-09-10 23:18:53 -07:00
Xiaoze Fan 477c8601b2 refactor(quant): config, scheme and method layers for quantization (#418)
* refactor(quant): config, scheme and method layers for checkpoint quantization

QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.

- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
2026-09-08 23:37:23 -07:00
Shuo Yang 6223cda1b0 Support muse glimmer (#4)
* feat(muse_glimmer): serve Muse-Glimmer-30B
2026-08-18 12:26:05 -07:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00