Files
Mukkesh Ganesh cc56b6bdcf feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next (#721)
* feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next

Adds the llama.cpp glm5-next GGUF reader (unsloth GLM-5.3-Flash-GGUF, both
the glm5-next and the older glm5next arch spellings) on the GGUF quant
kind from #690:

- parse_gguf_config derives the text config from the GGUF KV metadata,
  with per-block attention.head_count_kv laying out the KDA/DSA hybrid,
  the trailing MTP block dropped like the HF loader drops it, and the
  MLA head widths from the *_length_mla keys.
- iter_gguf_weights maps every tensor to the glm5_next state dict and
  undoes the converter transforms: A_log stored as -exp(A_log) is
  inverted, the per-head attn_k_b/attn_v_b pair (k transposed) fuses
  into the dense kv_b_proj layout the MLA absorption reads, and the
  q|k|v conv kernels concatenate on the channel axis. The mixed-type
  fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a) rides #702 segments;
  the routed experts load as GGUF expert banks at their shipped types
  (unsloth dynamic quants: IQ3_S gate/up + IQ4_XS down, Q8_0 outlier
  layers pass through natively).
- passes quant_config to embed_tokens so the Q8_0 token_embd serves
  packed; registers Glm5NextGGUFForCausalLM and claims it in the AOT
  table.

Verified end to end against unsloth GLM-5.3-Flash-UD-IQ4_XS (5 split
shards): the full engine on a 32GB GPU serves coherent output with the
experts off host banks. Text-only, TP=1.

Part of #671.

* fix(gguf): refuse unread scale tensors in glm5-next ggufs

check_scales at config parse like the llama, qwen3 and gemma4 loaders
(review on #721): a .scale / .input_scale outside an NVFP4 weight now
fails with a clear error instead of loading silently unread.

* fix(gguf): serve the quantized tensors of unsloth dynamic glm5-next quants

unsloth's UD quants quantize tensors the AesSedAI-style files keep float:
the MLA kv_b halves (Q8_0), the mHC mixers and the indexer's wk /
weights_proj / compressor gate. kv_b_proj joins the registry's
unquantized_modules (glm_moe_dsa keeps its kv_b dense for the MLA
absorption for the same reason) and the reader dequantizes through the
vendored gguf-py reference on every float-expecting read; wq_b keeps
serving its checkpoint quant packed (attention.py keeps wk and
weights_proj dense). Regression test stores the kv_b halves Q8_0.

* fix(gguf): treat the glm5 pre-tokenizer spelling like glm4

llama-vocab.cpp maps both spellings to CHATGLM4 with ignore_merges;
unsloth writes glm4 and AesSedAI's writer glm5 for the same vocabulary.
Without the mapping a glm5 file loads but silently mis-tokenizes
(tok_compare: 22,840/153,725 vocab words differ from the HF reference;
0 everywhere with it).

* fix(gguf): size the mmq tail padding by ggml type and k

pad_rows added one zero row of k columns, which covers the bytes MMQ
reads past the last row only when k is at least half a padding unit
(512); glm5-next's KDA f_b / g_b (k = linear head dim 128, Q8_0 in
dynamic quants) are the first packed tensors below that and died at the
first mul_mat. pad_rows now takes the ggml type and returns enough whole
rows to cover mmq_tail_bytes for any (type, k), consistently in the
reader's fused(), the gguf linear and embedding params and the expert
host banks. Found serving the real unsloth UD-IQ4_XS: the dry load
never runs a forward.

* fix(gguf): take the older glm5-next writer variants in stride

Two long-tail writer variants the full check sweep found (the GSQ-RCO
lineage alone is ~57k downloads/month):

- the conv kernels with a trailing unit dim: llama.cpp's glm5-next
  loader tries that 4-D shape before the 3-D one, so trailing unit dims
  in ne are dropped on float reads (they carry no elements).
- attention.head_count_kv as the generic scalar (all blocks): wrong for
  the hybrid -- llama.cpp broadcasts it and then fails tensor creation
  -- so only the list is trusted and a scalar falls to the tensor-name
  census like the pre-list writer.

* fix(gguf): glm5-next review cleanups

- refuse instead of guessing metadata the model cannot express: the
  scalar attention.head_count_kv spelling (llama.cpp broadcasts it and
  then fails tensor creation; every public file carries the list),
  missing *_length_mla keys through need(), non-uniform
  swiglu_clamp_exp / swiglu_clamp_shexp, indexer.types with non-full
  layers, expert_gating_func other than sigmoid, and rope dims as a
  NotImplementedError instead of an assert.
- call check_scales directly; aot_models comment on its own line.
- drop the GGUF test files; coverage will be handled across families.
2026-10-19 23:20:40 -07:00

202 lines
8.2 KiB
Python

"""GLM-5.3-Flash (glm5_next) model: hybrid KDA/DSA decoder with mHC residual streams.
Layer layout comes from the checkpoint's ``layer_types`` (34 KDA linear-attention
layers, 11 NoPE-MLA/DSA layers at 3:1) and ``mlp_layer_types`` (3 dense + 42 MoE).
The residual stream is mHC-widened to ``hc_mult`` (4) parallel streams:
layer 0: residual = hc_expand(x); (post, comb, x) = mhc_pre(residual, hc_attn_*)
each sublayer boundary fuses the previous hc_post with the next hc_pre
(mhc_fused_post_pre), and the sublayer input is RMS-normed AFTER the mix
(the reference fuses the norm into its hc kernels; decomposed here, same math).
last layer: x = mhc_post(...); x = hc_contract(x) -> final norm -> lm_head.
The deferred (post, comb) pair threads through the layer loop exactly like
glm_moe_dsa's (x, residual) pair. lm_head quant mirrors glm_moe_dsa (optional
W8A16 fp8 at load; the ~1.2 GiB bf16 head is read every decode step).
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Tuple
import torch
from freetoken.core import get_global_ctx
from freetoken.layers import (
BaseOP,
OPList,
ParallelLMHead,
RMSNorm,
VocabParallelEmbedding,
)
from freetoken.layers.mhc import hc_contract, hc_expand, mhc_fused_post_pre, mhc_post, mhc_pre
from freetoken.models.blocks import BaseLLMModel, embed_input_ids
from freetoken.utils import nvtx_annotate
from .attention import Glm5NextAttention
from .kda import Glm5NextKDA
from .mlp import Glm5NextGatedMLP
from .moe import Glm5NextSparseBlock
from .vision import Glm5NextVisionModel
if TYPE_CHECKING:
from freetoken.message import MMItem
from freetoken.models.config import ModelConfig
class Glm5NextDecoderLayer(BaseOP):
def __init__(self, config: ModelConfig, layer_id: int, *, prefix: str = ""):
args = config.glm5_args
self._layer_id = layer_id
self._is_last = layer_id == config.num_layers - 1
self.mhc = args.mhc
self._n = args.mhc_num_residual_streams
self._hc_eps = args.hc_eps
self._rms_eps = args.norm_eps
self._post_mult = args.mhc_post_mult_value
self._sinkhorn = args.mhc_sinkhorn_iterations
if args.is_kda_layer(layer_id):
self.self_attn: BaseOP = Glm5NextKDA(config, layer_id, prefix=f"{prefix}.self_attn")
else:
self.self_attn = Glm5NextAttention(config, layer_id, prefix=f"{prefix}.self_attn")
if layer_id >= config.first_k_dense_replace:
self.mlp: BaseOP = Glm5NextSparseBlock(config, layer_id, prefix=f"{prefix}.mlp")
else:
self.mlp = Glm5NextGatedMLP(
config.hidden_size, config.intermediate_size, swiglu_limit=config.swiglu_limit,
quant_config=config.quant, prefix=f"{prefix}.mlp",
)
self.input_layernorm = RMSNorm(size=config.hidden_size, eps=args.norm_eps)
self.post_attention_layernorm = RMSNorm(size=config.hidden_size, eps=args.norm_eps)
if self.mhc:
n, hidden = self._n, config.hidden_size
mix = 2 * n + n * n
# fp32 mHC weights (models/weight.py exempts hc_* from the dtype downcast).
self.hc_attn_fn = torch.empty(mix, n * hidden, dtype=torch.float32)
self.hc_attn_base = torch.empty(mix, dtype=torch.float32)
self.hc_attn_scale = torch.empty(3, dtype=torch.float32)
self.hc_ffn_fn = torch.empty(mix, n * hidden, dtype=torch.float32)
self.hc_ffn_base = torch.empty(mix, dtype=torch.float32)
self.hc_ffn_scale = torch.empty(3, dtype=torch.float32)
def _pre(self, residual, fn, scale, base):
# Layer 0's standalone pre rides the fused kernel too (HAS_POST=False
# path; x/post/comb are the no-post sentinels) -- same dispatch, same
# numerics, and the kernel wins at every batch size (see layers/mhc.py).
if residual.is_cuda:
_, post, comb, x = mhc_fused_post_pre(
residual.new_empty(residual.shape[0], residual.shape[-1]),
residual, None, None, fn, scale, base,
self._rms_eps, self._hc_eps, self._post_mult, self._sinkhorn,
)
return post, comb, x
return mhc_pre(
residual, fn, scale, base,
self._rms_eps, self._hc_eps, self._post_mult, self._sinkhorn,
)
def _fused(self, x, residual, post, comb, fn, scale, base):
return mhc_fused_post_pre(
x, residual, post, comb, fn, scale, base,
self._rms_eps, self._hc_eps, self._post_mult, self._sinkhorn,
)
@nvtx_annotate("Layer_{}", layer_id_field="_layer_id")
def forward(
self,
x: torch.Tensor,
residual: torch.Tensor | None,
post: torch.Tensor | None,
comb: torch.Tensor | None,
) -> Tuple[torch.Tensor, torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]:
if post is None:
if residual is None:
residual = hc_expand(x, self._n)
post, comb, x = self._pre(
residual, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base
)
else:
residual, post, comb, x = self._fused(
x, residual, post, comb,
self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base,
)
x = self.input_layernorm.forward(x)
x = self.self_attn.forward(x)
residual, post, comb, x = self._fused(
x, residual, post, comb,
self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base,
)
x = self.post_attention_layernorm.forward(x)
x = self.mlp.forward(x)
if self._is_last:
x = mhc_post(x, residual, post, comb)
return hc_contract(x), None, None, None
return x, residual, post, comb
class Glm5NextModel(BaseOP):
def __init__(self, config: ModelConfig, *, prefix: str = "model"):
self.embed_tokens = VocabParallelEmbedding(
num_embeddings=config.vocab_size,
embedding_dim=config.hidden_size,
quant_config=config.quant,
prefix=f"{prefix}.embed_tokens",
)
self.layers = OPList(
[Glm5NextDecoderLayer(config, i, prefix=f"{prefix}.layers.{i}") for i in range(config.num_layers)]
)
self.norm = RMSNorm(size=config.hidden_size, eps=config.rms_norm_eps)
def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
x = embed_input_ids(self.embed_tokens, input_ids, get_global_ctx().batch)
residual = post = comb = None
for layer in self.layers.op_list:
x, residual, post, comb = layer.forward(x, residual, post, comb)
return self.norm.forward(x)
class Glm5NextForCausalLM(BaseLLMModel):
def __init__(self, config: ModelConfig):
self._config = config
self.model = Glm5NextModel(config)
self.lm_head = ParallelLMHead(
num_embeddings=config.vocab_size,
embedding_dim=config.hidden_size,
tie_word_embeddings=config.tie_word_embeddings,
tied_embedding=self.model.embed_tokens if config.tie_word_embeddings else None,
quant_config=config.quant,
prefix="lm_head",
)
def prepare_for_runtime(self) -> None:
"""Post-load, pre-KV-sizing hook: materialize the DSA layers' bmm-ready
kv_b splits and free the checkpoint-layout originals (glm_moe_dsa
precedent)."""
for layer in self.model.layers.op_list:
if isinstance(layer.self_attn, Glm5NextAttention):
layer.self_attn.prepare_for_runtime()
torch.cuda.empty_cache()
def forward(self) -> torch.Tensor:
output = self.model.forward(get_global_ctx().batch.input_ids)
return self.lm_head.forward(output)
class Glm5NextForConditionalGeneration(Glm5NextForCausalLM):
def __init__(self, config: ModelConfig):
super().__init__(config)
if config.is_multimodal:
self.visual = Glm5NextVisionModel(config.vision_config)
def place_encoder_weights(self, mode: str) -> None:
self.visual.place_weights(mode)
def encode(self, item: MMItem) -> torch.Tensor:
return self.visual.forward(item.feature, [item.grid_thw])
__all__ = ["Glm5NextForCausalLM", "Glm5NextForConditionalGeneration"]