* feat(quant)!: compress bf16 weights to df11 at load with --online-quant
GLM-4.7 no longer hardwires DF11 attention and embedding; it serves them
bf16 unless --online-quant df11 is passed, and FTW files converted with the
old DF11 tensors must be reconverted.
* fix(quant): skip layers --online-quant cannot convert instead of failing
* feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next
Adds the llama.cpp glm5-next GGUF reader (unsloth GLM-5.3-Flash-GGUF, both
the glm5-next and the older glm5next arch spellings) on the GGUF quant
kind from #690:
- parse_gguf_config derives the text config from the GGUF KV metadata,
with per-block attention.head_count_kv laying out the KDA/DSA hybrid,
the trailing MTP block dropped like the HF loader drops it, and the
MLA head widths from the *_length_mla keys.
- iter_gguf_weights maps every tensor to the glm5_next state dict and
undoes the converter transforms: A_log stored as -exp(A_log) is
inverted, the per-head attn_k_b/attn_v_b pair (k transposed) fuses
into the dense kv_b_proj layout the MLA absorption reads, and the
q|k|v conv kernels concatenate on the channel axis. The mixed-type
fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a) rides #702 segments;
the routed experts load as GGUF expert banks at their shipped types
(unsloth dynamic quants: IQ3_S gate/up + IQ4_XS down, Q8_0 outlier
layers pass through natively).
- passes quant_config to embed_tokens so the Q8_0 token_embd serves
packed; registers Glm5NextGGUFForCausalLM and claims it in the AOT
table.
Verified end to end against unsloth GLM-5.3-Flash-UD-IQ4_XS (5 split
shards): the full engine on a 32GB GPU serves coherent output with the
experts off host banks. Text-only, TP=1.
Part of #671.
* fix(gguf): refuse unread scale tensors in glm5-next ggufs
check_scales at config parse like the llama, qwen3 and gemma4 loaders
(review on #721): a .scale / .input_scale outside an NVFP4 weight now
fails with a clear error instead of loading silently unread.
* fix(gguf): serve the quantized tensors of unsloth dynamic glm5-next quants
unsloth's UD quants quantize tensors the AesSedAI-style files keep float:
the MLA kv_b halves (Q8_0), the mHC mixers and the indexer's wk /
weights_proj / compressor gate. kv_b_proj joins the registry's
unquantized_modules (glm_moe_dsa keeps its kv_b dense for the MLA
absorption for the same reason) and the reader dequantizes through the
vendored gguf-py reference on every float-expecting read; wq_b keeps
serving its checkpoint quant packed (attention.py keeps wk and
weights_proj dense). Regression test stores the kv_b halves Q8_0.
* fix(gguf): treat the glm5 pre-tokenizer spelling like glm4
llama-vocab.cpp maps both spellings to CHATGLM4 with ignore_merges;
unsloth writes glm4 and AesSedAI's writer glm5 for the same vocabulary.
Without the mapping a glm5 file loads but silently mis-tokenizes
(tok_compare: 22,840/153,725 vocab words differ from the HF reference;
0 everywhere with it).
* fix(gguf): size the mmq tail padding by ggml type and k
pad_rows added one zero row of k columns, which covers the bytes MMQ
reads past the last row only when k is at least half a padding unit
(512); glm5-next's KDA f_b / g_b (k = linear head dim 128, Q8_0 in
dynamic quants) are the first packed tensors below that and died at the
first mul_mat. pad_rows now takes the ggml type and returns enough whole
rows to cover mmq_tail_bytes for any (type, k), consistently in the
reader's fused(), the gguf linear and embedding params and the expert
host banks. Found serving the real unsloth UD-IQ4_XS: the dry load
never runs a forward.
* fix(gguf): take the older glm5-next writer variants in stride
Two long-tail writer variants the full check sweep found (the GSQ-RCO
lineage alone is ~57k downloads/month):
- the conv kernels with a trailing unit dim: llama.cpp's glm5-next
loader tries that 4-D shape before the 3-D one, so trailing unit dims
in ne are dropped on float reads (they carry no elements).
- attention.head_count_kv as the generic scalar (all blocks): wrong for
the hybrid -- llama.cpp broadcasts it and then fails tensor creation
-- so only the list is trusted and a scalar falls to the tensor-name
census like the pre-list writer.
* fix(gguf): glm5-next review cleanups
- refuse instead of guessing metadata the model cannot express: the
scalar attention.head_count_kv spelling (llama.cpp broadcasts it and
then fails tensor creation; every public file carries the list),
missing *_length_mla keys through need(), non-uniform
swiglu_clamp_exp / swiglu_clamp_shexp, indexer.types with non-full
layers, expert_gating_func other than sigmoid, and rope dims as a
NotImplementedError instead of an assert.
- call check_scales directly; aot_models comment on its own line.
- drop the GGUF test files; coverage will be handled across families.
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding
* build(deps): vendor the gguf-py reader
* feat(quant): add the GGUF quant kind and an embedding quant method
* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path
* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA
* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16
* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small
* fix(gguf): read F32, F16 and BF16 PLE tables without ggml
* fix(moe): pad the offload slot cache for ggml MMQ tail reads
* fix(gguf): support hip builds and runtime devices
* fix(gguf): copy same-type dequantize and check for missing ggml converters
* fix(gguf): load quantized qwen4exp ple_key and ple_value
* feat(gguf): run float parts of a fused GGUF linear as bf16 segments
* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights
* fix(moe): zero the prefill staging area after the last prefill layer
---------
Co-authored-by: zihaomu <zihaomu@outlook.com>