* feat(gguf): serve deepseek-v4.1 gguf checkpoints on deepseek_v41
llama.cpp 10-25 added the deepseek41 layout. The reader takes the config from the file, undoes the converter (engram gate stored as the product, one per_layer_token_embd tensor, hash constants as KVs) and streams the Engram rows from the file as the HF path does.
* fix(gguf): refuse deepseek-v4.1 gguf mismatches at config time
Refuse an FTW of the GGUF before the writer creates the output dir and keep
the side-file hook after the dense loop as on main. Check the Engram hash KVs
and the table in parse_gguf_config, refuse a table with a .scale, and share
the clamp and expert_count readers with the deepseek4 GGUF reader.
* feat(gguf): load the vision tower from a llama.cpp mmproj
Qwen3.5 / Qwen3.6 / Qwen3.8 (qwen3vl_merger) and Gemma-4 26B-A4B / 31B (gemma4v) / 12B (gemma4uv) take image input from a separate mmproj file: --mmproj, else one in the model's own folder. The tower runs in bf16 (quantized mmproj files are dequantized) and its config comes from the GGUF metadata plus per-arch defaults.
* docs(gguf): describe mmproj image input without a model list
* test(gguf): drop the mmproj tests
* fix(gguf): escape the model folder when globbing for mmproj files
* fix(gguf): skip auto-discovered mmproj files the text model cannot use
* fix(gguf): serve text from a GGUF that carries vision tensors itself
NVFP4 weight global scales (amax / 2688, 2e-5 to 4e-4 in the served checkpoints) sit at fp16's
normal minimum; stored as fp16, up to 97% of a checkpoint's expert globals were subnormal, with
up to 1.7e-3 relative error per row.
BREAKING CHANGE: an FTW checkpoint that stores NVFP4 global scales as fp16 is refused at load;
reconvert it with `ft checkpoint`.
* feat(moe): serve experts of several formats through per-format slot pools
* refactor(cpu-moe): pass each layer's cpu format with its options
* fix(moe): read only each format's layers when loading experts of several formats
---------
Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding
* build(deps): vendor the gguf-py reader
* feat(quant): add the GGUF quant kind and an embedding quant method
* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path
* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA
* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16
* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small
* fix(gguf): read F32, F16 and BF16 PLE tables without ggml
* fix(moe): pad the offload slot cache for ggml MMQ tail reads
* fix(gguf): support hip builds and runtime devices
* fix(gguf): copy same-type dequantize and check for missing ggml converters
* fix(gguf): load quantized qwen4exp ple_key and ple_value
* feat(gguf): run float parts of a fused GGUF linear as bf16 segments
* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights
* fix(moe): zero the prefill staging area after the last prefill layer
---------
Co-authored-by: zihaomu <zihaomu@outlook.com>
* feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks
Resident experts skip the state dict and load through the offload expert-bank loader (HF pieces or FTW banks) into per-layer GPU banks, so fused serves bf16, fp8_block, NVFP4 and MXFP4 alike. Unified-memory GPUs auto-select fused for these formats and ignore offload-only flags with a warning.
BREAKING CHANGE: ft checkpoint always writes MoE experts as banks; an FTW converted with --moe-backend fused or triton (dense experts) no longer loads and must be reconverted.
* docs(moe): name only GB10 / DGX Spark for unified memory
* fix(checkpoint): skip the vision tower before reading an FTW served text-only
- iter_ftw_weights takes a keep(name) predicate and drops the rejected entries before their bytes are read; load_weight passes the vision prefixes through it instead of discarding the tensors after the replay
- an FTW converted before its family served images holds no encoder tensors: the engine now refuses it with the reconvert / --text-model-only choice instead of failing on the first missing key
- ftw_tensor_names lists an index's names by kind; docs/models.md notes the pre-image-support FTWs
* feat(scripts): add the vision encoder to a pre-image-support FTW
- ftw_hotfix.py fetches the encoder's checkpoint tensors by byte range from the Hub or a local checkpoint into a scratch checkpoint dir and lets the family's own weight reader name and fuse them, so the repaired FTW holds exactly what the converter would have written
- every family with an encoder gains iter_vision_weights, the encoder-only reader behind load_vision_weight: the Qwen families and Gemma yield the tower as stored (Gemma without its clipped-linear wrapper), GLM and MiniMax reuse their tower readers, Muse fuses q/k/v as its dense pass does
- the engine's refusal of a tower-less FTW and the models.md note name the tool next to reconverting; docs/ftw-hotfix.md gains the repair row and a Hub example; the ignored-leftover prefixes follow VISION_KEY_PREFIXES
* refactor(quant): config, scheme and method layers for checkpoint quantization
QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.
- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
* feat(server)!: add --gpu to choose the GPU on multi-GPU machines
--gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index.
It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>.
/v1/stats reports the engine's GPU under "gpus".
ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name.
The daemon's /bench/profile returns the running serve's GPU profile.
BREAKING CHANGE: ft checkpoint --device is removed; use --gpu.
* refactor!: resolve --gpu via NVML and bind by UUID, not CUDA_VISIBLE_DEVICES
BREAKING CHANGE: ft bench bw --device is removed; use --gpu.
* refactor: name the id namespaces and drop the multi-device e4m3 scan
Splits the published id into _assigned_physical (UUID) and
_assigned_visible (CUDA ordinal). One process runs on one GPU, so
e4m3_native() judges that card instead of scanning every visible device.