mirror of
https://github.com/FlashML-org/FreeToken.git
synced 2026-11-21 01:52:30 +08:00
* feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next Adds the llama.cpp glm5-next GGUF reader (unsloth GLM-5.3-Flash-GGUF, both the glm5-next and the older glm5next arch spellings) on the GGUF quant kind from #690: - parse_gguf_config derives the text config from the GGUF KV metadata, with per-block attention.head_count_kv laying out the KDA/DSA hybrid, the trailing MTP block dropped like the HF loader drops it, and the MLA head widths from the *_length_mla keys. - iter_gguf_weights maps every tensor to the glm5_next state dict and undoes the converter transforms: A_log stored as -exp(A_log) is inverted, the per-head attn_k_b/attn_v_b pair (k transposed) fuses into the dense kv_b_proj layout the MLA absorption reads, and the q|k|v conv kernels concatenate on the channel axis. The mixed-type fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a) rides #702 segments; the routed experts load as GGUF expert banks at their shipped types (unsloth dynamic quants: IQ3_S gate/up + IQ4_XS down, Q8_0 outlier layers pass through natively). - passes quant_config to embed_tokens so the Q8_0 token_embd serves packed; registers Glm5NextGGUFForCausalLM and claims it in the AOT table. Verified end to end against unsloth GLM-5.3-Flash-UD-IQ4_XS (5 split shards): the full engine on a 32GB GPU serves coherent output with the experts off host banks. Text-only, TP=1. Part of #671. * fix(gguf): refuse unread scale tensors in glm5-next ggufs check_scales at config parse like the llama, qwen3 and gemma4 loaders (review on #721): a .scale / .input_scale outside an NVFP4 weight now fails with a clear error instead of loading silently unread. * fix(gguf): serve the quantized tensors of unsloth dynamic glm5-next quants unsloth's UD quants quantize tensors the AesSedAI-style files keep float: the MLA kv_b halves (Q8_0), the mHC mixers and the indexer's wk / weights_proj / compressor gate. kv_b_proj joins the registry's unquantized_modules (glm_moe_dsa keeps its kv_b dense for the MLA absorption for the same reason) and the reader dequantizes through the vendored gguf-py reference on every float-expecting read; wq_b keeps serving its checkpoint quant packed (attention.py keeps wk and weights_proj dense). Regression test stores the kv_b halves Q8_0. * fix(gguf): treat the glm5 pre-tokenizer spelling like glm4 llama-vocab.cpp maps both spellings to CHATGLM4 with ignore_merges; unsloth writes glm4 and AesSedAI's writer glm5 for the same vocabulary. Without the mapping a glm5 file loads but silently mis-tokenizes (tok_compare: 22,840/153,725 vocab words differ from the HF reference; 0 everywhere with it). * fix(gguf): size the mmq tail padding by ggml type and k pad_rows added one zero row of k columns, which covers the bytes MMQ reads past the last row only when k is at least half a padding unit (512); glm5-next's KDA f_b / g_b (k = linear head dim 128, Q8_0 in dynamic quants) are the first packed tensors below that and died at the first mul_mat. pad_rows now takes the ggml type and returns enough whole rows to cover mmq_tail_bytes for any (type, k), consistently in the reader's fused(), the gguf linear and embedding params and the expert host banks. Found serving the real unsloth UD-IQ4_XS: the dry load never runs a forward. * fix(gguf): take the older glm5-next writer variants in stride Two long-tail writer variants the full check sweep found (the GSQ-RCO lineage alone is ~57k downloads/month): - the conv kernels with a trailing unit dim: llama.cpp's glm5-next loader tries that 4-D shape before the 3-D one, so trailing unit dims in ne are dropped on float reads (they carry no elements). - attention.head_count_kv as the generic scalar (all blocks): wrong for the hybrid -- llama.cpp broadcasts it and then fails tensor creation -- so only the list is trusted and a scalar falls to the tensor-name census like the pre-list writer. * fix(gguf): glm5-next review cleanups - refuse instead of guessing metadata the model cannot express: the scalar attention.head_count_kv spelling (llama.cpp broadcasts it and then fails tensor creation; every public file carries the list), missing *_length_mla keys through need(), non-uniform swiglu_clamp_exp / swiglu_clamp_shexp, indexer.types with non-full layers, expert_gating_func other than sigmoid, and rope dims as a NotImplementedError instead of an assert. - call check_scales directly; aot_models comment on its own line. - drop the GGUF test files; coverage will be handled across families.
202 lines
8.2 KiB
Python
202 lines
8.2 KiB
Python
"""GLM-5.3-Flash (glm5_next) model: hybrid KDA/DSA decoder with mHC residual streams.
|
|
|
|
Layer layout comes from the checkpoint's ``layer_types`` (34 KDA linear-attention
|
|
layers, 11 NoPE-MLA/DSA layers at 3:1) and ``mlp_layer_types`` (3 dense + 42 MoE).
|
|
The residual stream is mHC-widened to ``hc_mult`` (4) parallel streams:
|
|
|
|
layer 0: residual = hc_expand(x); (post, comb, x) = mhc_pre(residual, hc_attn_*)
|
|
each sublayer boundary fuses the previous hc_post with the next hc_pre
|
|
(mhc_fused_post_pre), and the sublayer input is RMS-normed AFTER the mix
|
|
(the reference fuses the norm into its hc kernels; decomposed here, same math).
|
|
last layer: x = mhc_post(...); x = hc_contract(x) -> final norm -> lm_head.
|
|
|
|
The deferred (post, comb) pair threads through the layer loop exactly like
|
|
glm_moe_dsa's (x, residual) pair. lm_head quant mirrors glm_moe_dsa (optional
|
|
W8A16 fp8 at load; the ~1.2 GiB bf16 head is read every decode step).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import TYPE_CHECKING, Tuple
|
|
|
|
import torch
|
|
from freetoken.core import get_global_ctx
|
|
from freetoken.layers import (
|
|
BaseOP,
|
|
OPList,
|
|
ParallelLMHead,
|
|
RMSNorm,
|
|
VocabParallelEmbedding,
|
|
)
|
|
from freetoken.layers.mhc import hc_contract, hc_expand, mhc_fused_post_pre, mhc_post, mhc_pre
|
|
from freetoken.models.blocks import BaseLLMModel, embed_input_ids
|
|
from freetoken.utils import nvtx_annotate
|
|
|
|
from .attention import Glm5NextAttention
|
|
from .kda import Glm5NextKDA
|
|
from .mlp import Glm5NextGatedMLP
|
|
from .moe import Glm5NextSparseBlock
|
|
from .vision import Glm5NextVisionModel
|
|
|
|
if TYPE_CHECKING:
|
|
from freetoken.message import MMItem
|
|
from freetoken.models.config import ModelConfig
|
|
|
|
|
|
class Glm5NextDecoderLayer(BaseOP):
|
|
def __init__(self, config: ModelConfig, layer_id: int, *, prefix: str = ""):
|
|
args = config.glm5_args
|
|
self._layer_id = layer_id
|
|
self._is_last = layer_id == config.num_layers - 1
|
|
self.mhc = args.mhc
|
|
self._n = args.mhc_num_residual_streams
|
|
self._hc_eps = args.hc_eps
|
|
self._rms_eps = args.norm_eps
|
|
self._post_mult = args.mhc_post_mult_value
|
|
self._sinkhorn = args.mhc_sinkhorn_iterations
|
|
|
|
if args.is_kda_layer(layer_id):
|
|
self.self_attn: BaseOP = Glm5NextKDA(config, layer_id, prefix=f"{prefix}.self_attn")
|
|
else:
|
|
self.self_attn = Glm5NextAttention(config, layer_id, prefix=f"{prefix}.self_attn")
|
|
if layer_id >= config.first_k_dense_replace:
|
|
self.mlp: BaseOP = Glm5NextSparseBlock(config, layer_id, prefix=f"{prefix}.mlp")
|
|
else:
|
|
self.mlp = Glm5NextGatedMLP(
|
|
config.hidden_size, config.intermediate_size, swiglu_limit=config.swiglu_limit,
|
|
quant_config=config.quant, prefix=f"{prefix}.mlp",
|
|
)
|
|
self.input_layernorm = RMSNorm(size=config.hidden_size, eps=args.norm_eps)
|
|
self.post_attention_layernorm = RMSNorm(size=config.hidden_size, eps=args.norm_eps)
|
|
|
|
if self.mhc:
|
|
n, hidden = self._n, config.hidden_size
|
|
mix = 2 * n + n * n
|
|
# fp32 mHC weights (models/weight.py exempts hc_* from the dtype downcast).
|
|
self.hc_attn_fn = torch.empty(mix, n * hidden, dtype=torch.float32)
|
|
self.hc_attn_base = torch.empty(mix, dtype=torch.float32)
|
|
self.hc_attn_scale = torch.empty(3, dtype=torch.float32)
|
|
self.hc_ffn_fn = torch.empty(mix, n * hidden, dtype=torch.float32)
|
|
self.hc_ffn_base = torch.empty(mix, dtype=torch.float32)
|
|
self.hc_ffn_scale = torch.empty(3, dtype=torch.float32)
|
|
|
|
def _pre(self, residual, fn, scale, base):
|
|
# Layer 0's standalone pre rides the fused kernel too (HAS_POST=False
|
|
# path; x/post/comb are the no-post sentinels) -- same dispatch, same
|
|
# numerics, and the kernel wins at every batch size (see layers/mhc.py).
|
|
if residual.is_cuda:
|
|
_, post, comb, x = mhc_fused_post_pre(
|
|
residual.new_empty(residual.shape[0], residual.shape[-1]),
|
|
residual, None, None, fn, scale, base,
|
|
self._rms_eps, self._hc_eps, self._post_mult, self._sinkhorn,
|
|
)
|
|
return post, comb, x
|
|
return mhc_pre(
|
|
residual, fn, scale, base,
|
|
self._rms_eps, self._hc_eps, self._post_mult, self._sinkhorn,
|
|
)
|
|
|
|
def _fused(self, x, residual, post, comb, fn, scale, base):
|
|
return mhc_fused_post_pre(
|
|
x, residual, post, comb, fn, scale, base,
|
|
self._rms_eps, self._hc_eps, self._post_mult, self._sinkhorn,
|
|
)
|
|
|
|
@nvtx_annotate("Layer_{}", layer_id_field="_layer_id")
|
|
def forward(
|
|
self,
|
|
x: torch.Tensor,
|
|
residual: torch.Tensor | None,
|
|
post: torch.Tensor | None,
|
|
comb: torch.Tensor | None,
|
|
) -> Tuple[torch.Tensor, torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]:
|
|
if post is None:
|
|
if residual is None:
|
|
residual = hc_expand(x, self._n)
|
|
post, comb, x = self._pre(
|
|
residual, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base
|
|
)
|
|
else:
|
|
residual, post, comb, x = self._fused(
|
|
x, residual, post, comb,
|
|
self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base,
|
|
)
|
|
x = self.input_layernorm.forward(x)
|
|
x = self.self_attn.forward(x)
|
|
|
|
residual, post, comb, x = self._fused(
|
|
x, residual, post, comb,
|
|
self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base,
|
|
)
|
|
x = self.post_attention_layernorm.forward(x)
|
|
x = self.mlp.forward(x)
|
|
|
|
if self._is_last:
|
|
x = mhc_post(x, residual, post, comb)
|
|
return hc_contract(x), None, None, None
|
|
return x, residual, post, comb
|
|
|
|
|
|
class Glm5NextModel(BaseOP):
|
|
def __init__(self, config: ModelConfig, *, prefix: str = "model"):
|
|
self.embed_tokens = VocabParallelEmbedding(
|
|
num_embeddings=config.vocab_size,
|
|
embedding_dim=config.hidden_size,
|
|
quant_config=config.quant,
|
|
prefix=f"{prefix}.embed_tokens",
|
|
)
|
|
self.layers = OPList(
|
|
[Glm5NextDecoderLayer(config, i, prefix=f"{prefix}.layers.{i}") for i in range(config.num_layers)]
|
|
)
|
|
self.norm = RMSNorm(size=config.hidden_size, eps=config.rms_norm_eps)
|
|
|
|
def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
|
|
x = embed_input_ids(self.embed_tokens, input_ids, get_global_ctx().batch)
|
|
residual = post = comb = None
|
|
for layer in self.layers.op_list:
|
|
x, residual, post, comb = layer.forward(x, residual, post, comb)
|
|
return self.norm.forward(x)
|
|
|
|
|
|
class Glm5NextForCausalLM(BaseLLMModel):
|
|
def __init__(self, config: ModelConfig):
|
|
self._config = config
|
|
self.model = Glm5NextModel(config)
|
|
self.lm_head = ParallelLMHead(
|
|
num_embeddings=config.vocab_size,
|
|
embedding_dim=config.hidden_size,
|
|
tie_word_embeddings=config.tie_word_embeddings,
|
|
tied_embedding=self.model.embed_tokens if config.tie_word_embeddings else None,
|
|
quant_config=config.quant,
|
|
prefix="lm_head",
|
|
)
|
|
|
|
def prepare_for_runtime(self) -> None:
|
|
"""Post-load, pre-KV-sizing hook: materialize the DSA layers' bmm-ready
|
|
kv_b splits and free the checkpoint-layout originals (glm_moe_dsa
|
|
precedent)."""
|
|
for layer in self.model.layers.op_list:
|
|
if isinstance(layer.self_attn, Glm5NextAttention):
|
|
layer.self_attn.prepare_for_runtime()
|
|
torch.cuda.empty_cache()
|
|
|
|
def forward(self) -> torch.Tensor:
|
|
output = self.model.forward(get_global_ctx().batch.input_ids)
|
|
return self.lm_head.forward(output)
|
|
|
|
|
|
class Glm5NextForConditionalGeneration(Glm5NextForCausalLM):
|
|
def __init__(self, config: ModelConfig):
|
|
super().__init__(config)
|
|
if config.is_multimodal:
|
|
self.visual = Glm5NextVisionModel(config.vision_config)
|
|
|
|
def place_encoder_weights(self, mode: str) -> None:
|
|
self.visual.place_weights(mode)
|
|
|
|
def encode(self, item: MMItem) -> torch.Tensor:
|
|
return self.visual.forward(item.feature, [item.grid_thw])
|
|
|
|
|
|
__all__ = ["Glm5NextForCausalLM", "Glm5NextForConditionalGeneration"]
|