Commit Graph
6 Commits
Author SHA1 Message Date
Xiaoze Fanandzihaomu e5d3bd3faf feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels (#690)
* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding

* build(deps): vendor the gguf-py reader

* feat(quant): add the GGUF quant kind and an embedding quant method

* feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path

* fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA

* perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16

* build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small

* fix(gguf): read F32, F16 and BF16 PLE tables without ggml

* fix(moe): pad the offload slot cache for ggml MMQ tail reads

* fix(gguf): support hip builds and runtime devices

* fix(gguf): copy same-type dequantize and check for missing ggml converters

* fix(gguf): load quantized qwen4exp ple_key and ple_value

* feat(gguf): run float parts of a fused GGUF linear as bf16 segments

* fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights

* fix(moe): zero the prefill staging area after the last prefill layer

---------

Co-authored-by: zihaomu <zihaomu@outlook.com>
2026-10-15 16:33:16 -07:00
Xiaoze Fan cade1a9e45 fix(checkpoint): serve and repair FTW checkpoints around the vision encoder (#486)
* fix(checkpoint): skip the vision tower before reading an FTW served text-only

- iter_ftw_weights takes a keep(name) predicate and drops the rejected entries before their bytes are read; load_weight passes the vision prefixes through it instead of discarding the tensors after the replay
- an FTW converted before its family served images holds no encoder tensors: the engine now refuses it with the reconvert / --text-model-only choice instead of failing on the first missing key
- ftw_tensor_names lists an index's names by kind; docs/models.md notes the pre-image-support FTWs

* feat(scripts): add the vision encoder to a pre-image-support FTW

- ftw_hotfix.py fetches the encoder's checkpoint tensors by byte range from the Hub or a local checkpoint into a scratch checkpoint dir and lets the family's own weight reader name and fuse them, so the repaired FTW holds exactly what the converter would have written
- every family with an encoder gains iter_vision_weights, the encoder-only reader behind load_vision_weight: the Qwen families and Gemma yield the tower as stored (Gemma without its clipped-linear wrapper), GLM and MiniMax reuse their tower readers, Muse fuses q/k/v as its dense pass does
- the engine's refusal of a tower-less FTW and the models.md note name the tool next to reconverting; docs/ftw-hotfix.md gains the repair row and a Hub example; the ignored-leftover prefixes follow VISION_KEY_PREFIXES
2026-09-15 12:43:09 -07:00
Xiaoze Fan 08d728d5e3 feat(mm): serve image input on the Qwen families (#454)
* feat(mm): serve image input on the Qwen VL families

- frontend: OpenAI image_url, Anthropic image blocks and Responses
  input_image (http(s), base64, gated file://) become MMItems in the
  tokenizer workers; image spans carry content pseudo ids so the radix
  cache keys them by content; mrope positions are precomputed per request
- registry: EncoderSpec per family (Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next
  register a vision encoder serving image) and ModelSpec.mm_processor;
  EngineConfig.active_encoders decides what a process builds,
  served_modalities what it accepts; --text-model-only, --mm-disable,
  --mm-encoder-weights, --mm-embed-cache-device, --mm-max-pixels
- engine and scheduler: per-request encoder-cache row ownership, chunk
  planning on the batch (jobs, gather rows, scatter rows), items encoded
  right before the LM forward and gathered into batch.mm_embeds
- models: the Qwen VL vision tower on the text tower's TP layers with
  DeepStack, block weights streamed from pinned host banks by default,
  merger outputs parked on the host between blocks; SupportsMultimodal
  (encode, place_encoder_weights) and a shared embed_input_ids
- server: image inputs rejected with a client-facing reason when no
  vision encoder is served; GET /v1/stats reports model.input_modalities

* feat(mm): replace the pixel cap with a token budget and processor kwargs

- --image-min-tokens / --image-max-tokens replace --mm-max-pixels; MMProcessor.get_mm_processor_kwargs turns the MultimodalConfig into the processor call's arguments, the default passes mm.processor_kwargs through and Qwen VL converts the budget to pixel areas in size.shortest_edge / longest_edge
- --mm-processor-kwargs hands a JSON object to the checkpoint's image processor call, applied after the token budget
- MMProcessor is an ABC that owns the model path, the config and the lazily loaded image processor; get_mm_processor returns None when --mm-disable leaves no encoder to serve
- the frontend tokenizer, the tokenizer worker and the offline LLM build one processor each and hand it to TokenizeManager, so the tokenize message carries no per-request pixel cap
2026-09-13 15:39:33 -07:00
Xiaoze Fan 0ffd5c8b29 feat(qwen3_5_moe): read every checkpoint layout through the QuantConfig (#438)
* feat(qwen3_5_moe): read every checkpoint layout through the QuantConfig

NVFP4 dense layers now declare the input_scale their scheme carries, so an
FTW written by an older build from an NVFP4 dense checkpoint needs
scripts/ftw_hotfix.py before it loads again.

* fix(quant): address scheme reader review: hotfix dialect names, gemma4 cross-shard scales, ct block-fp8 experts
2026-09-10 23:18:53 -07:00
Xiaoze Fan 477c8601b2 refactor(quant): config, scheme and method layers for quantization (#418)
* refactor(quant): config, scheme and method layers for checkpoint quantization

QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.

- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
2026-09-08 23:37:23 -07:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00