Commit Graph
4 Commits
Author SHA1 Message Date
Manish Biswal 20d5d93132 feat(gguf): read NVFP4 token_embd rows in the vendored get_rows (#801)
* feat(gguf): read NVFP4 token_embd rows in the vendored get_rows

Signed-off-by: iamanishx <manishbiswal754@gmail.com>

* fix(tests)

Signed-off-by: iamanishx <manishbiswal754@gmail.com>

---------

Signed-off-by: iamanishx <manishbiswal754@gmail.com>
2026-10-22 20:19:24 -07:00
Mukkesh Ganesh cc56b6bdcf feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next (#721)
* feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next

Adds the llama.cpp glm5-next GGUF reader (unsloth GLM-5.3-Flash-GGUF, both
the glm5-next and the older glm5next arch spellings) on the GGUF quant
kind from #690:

- parse_gguf_config derives the text config from the GGUF KV metadata,
  with per-block attention.head_count_kv laying out the KDA/DSA hybrid,
  the trailing MTP block dropped like the HF loader drops it, and the
  MLA head widths from the *_length_mla keys.
- iter_gguf_weights maps every tensor to the glm5_next state dict and
  undoes the converter transforms: A_log stored as -exp(A_log) is
  inverted, the per-head attn_k_b/attn_v_b pair (k transposed) fuses
  into the dense kv_b_proj layout the MLA absorption reads, and the
  q|k|v conv kernels concatenate on the channel axis. The mixed-type
  fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a) rides #702 segments;
  the routed experts load as GGUF expert banks at their shipped types
  (unsloth dynamic quants: IQ3_S gate/up + IQ4_XS down, Q8_0 outlier
  layers pass through natively).
- passes quant_config to embed_tokens so the Q8_0 token_embd serves
  packed; registers Glm5NextGGUFForCausalLM and claims it in the AOT
  table.

Verified end to end against unsloth GLM-5.3-Flash-UD-IQ4_XS (5 split
shards): the full engine on a 32GB GPU serves coherent output with the
experts off host banks. Text-only, TP=1.

Part of #671.

* fix(gguf): refuse unread scale tensors in glm5-next ggufs

check_scales at config parse like the llama, qwen3 and gemma4 loaders
(review on #721): a .scale / .input_scale outside an NVFP4 weight now
fails with a clear error instead of loading silently unread.

* fix(gguf): serve the quantized tensors of unsloth dynamic glm5-next quants

unsloth's UD quants quantize tensors the AesSedAI-style files keep float:
the MLA kv_b halves (Q8_0), the mHC mixers and the indexer's wk /
weights_proj / compressor gate. kv_b_proj joins the registry's
unquantized_modules (glm_moe_dsa keeps its kv_b dense for the MLA
absorption for the same reason) and the reader dequantizes through the
vendored gguf-py reference on every float-expecting read; wq_b keeps
serving its checkpoint quant packed (attention.py keeps wk and
weights_proj dense). Regression test stores the kv_b halves Q8_0.

* fix(gguf): treat the glm5 pre-tokenizer spelling like glm4

llama-vocab.cpp maps both spellings to CHATGLM4 with ignore_merges;
unsloth writes glm4 and AesSedAI's writer glm5 for the same vocabulary.
Without the mapping a glm5 file loads but silently mis-tokenizes
(tok_compare: 22,840/153,725 vocab words differ from the HF reference;
0 everywhere with it).

* fix(gguf): size the mmq tail padding by ggml type and k

pad_rows added one zero row of k columns, which covers the bytes MMQ
reads past the last row only when k is at least half a padding unit
(512); glm5-next's KDA f_b / g_b (k = linear head dim 128, Q8_0 in
dynamic quants) are the first packed tensors below that and died at the
first mul_mat. pad_rows now takes the ggml type and returns enough whole
rows to cover mmq_tail_bytes for any (type, k), consistently in the
reader's fused(), the gguf linear and embedding params and the expert
host banks. Found serving the real unsloth UD-IQ4_XS: the dry load
never runs a forward.

* fix(gguf): take the older glm5-next writer variants in stride

Two long-tail writer variants the full check sweep found (the GSQ-RCO
lineage alone is ~57k downloads/month):

- the conv kernels with a trailing unit dim: llama.cpp's glm5-next
  loader tries that 4-D shape before the 3-D one, so trailing unit dims
  in ne are dropped on float reads (they carry no elements).
- attention.head_count_kv as the generic scalar (all blocks): wrong for
  the hybrid -- llama.cpp broadcasts it and then fails tensor creation
  -- so only the list is trusted and a scalar falls to the tensor-name
  census like the pre-list writer.

* fix(gguf): glm5-next review cleanups

- refuse instead of guessing metadata the model cannot express: the
  scalar attention.head_count_kv spelling (llama.cpp broadcasts it and
  then fails tensor creation; every public file carries the list),
  missing *_length_mla keys through need(), non-uniform
  swiglu_clamp_exp / swiglu_clamp_shexp, indexer.types with non-full
  layers, expert_gating_func other than sigmoid, and rope dims as a
  NotImplementedError instead of an assert.
- call check_scales directly; aot_models comment on its own line.
- drop the GGUF test files; coverage will be handled across families.
2026-10-19 23:20:40 -07:00
Xiaoze Fan 31c6ef4f77 feat(gguf): apply the weight scales of nvfp4 gguf tensors (#732) 2026-10-17 13:46:50 -07:00
Xiaoze Fanandzihaomu e5d3bd3faf feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels (#690)
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding

* build(deps): vendor the gguf-py reader

* feat(quant): add the GGUF quant kind and an embedding quant method

* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path

* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA

* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16

* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small

* fix(gguf): read F32, F16 and BF16 PLE tables without ggml

* fix(moe): pad the offload slot cache for ggml MMQ tail reads

* fix(gguf): support hip builds and runtime devices

* fix(gguf): copy same-type dequantize and check for missing ggml converters

* fix(gguf): load quantized qwen4exp ple_key and ple_value

* feat(gguf): run float parts of a fused GGUF linear as bf16 segments

* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights

* fix(moe): zero the prefill staging area after the last prefill layer

---------

Co-authored-by: zihaomu <zihaomu@outlook.com>
2026-10-15 16:33:16 -07:00