* build(kernel): vendor ggml-cuda from llama.cpp 8c2134a8 behind a tvm-ffi binding * build(deps): vendor the gguf-py reader * feat(quant): add the GGUF quant kind and an embedding quant method * feat(gguf): serve Flash-Next and Gemma4 GGUF on the GGUF quant kind and drop the old Q4_0 path * fix(kernel): build ggml-cuda for plain sm_120 by turning off the native fp4 MMA * perf(gguf): write ggml results in the activation dtype and keep MoE intermediates in bf16 * build(kernel): compress the ggml fatbin to keep the multi-arch kernel cache small * fix(gguf): read F32, F16 and BF16 PLE tables without ggml * fix(moe): pad the offload slot cache for ggml MMQ tail reads * fix(gguf): support hip builds and runtime devices * fix(gguf): copy same-type dequantize and check for missing ggml converters * fix(gguf): load quantized qwen4exp ple_key and ple_value * feat(gguf): run float parts of a fused GGUF linear as bf16 segments * fix(gguf): load quantized qwen4exp hyper-connection and gdn ba weights * fix(moe): zero the prefill staging area after the last prefill layer --------- Co-authored-by: zihaomu <zihaomu@outlook.com>
benchmarks
Run from the repo root with PYTHONPATH=python:., pinned to one GPU
(CUDA_VISIBLE_DEVICES=0). Each script's --help / docstring has the details.
bench_decode_moe.py — bs=1 decode tok/s of a served MoE model. Spawns ft serve
per backend and times token arrivals over streamed /v1/chat/completions, so numbers
include the full serving path. AIME-25 prompt, checkpoint-recommended sampling.
python benchmarks/bench_decode_moe.py --model /path/to/model --backend offload,cpu,hybrid
bench_load_weight_generic.py — expert-bank load time: serial vs parallel O_DIRECT
vs pre-repacked FTW, each mode in its own subprocess. Linux-only; stages the FTW under
/var/tmp (--ftw-dir overrides; roughly checkpoint-sized).
python benchmarks/bench_load_weight_generic.py --model /path/to/model
bench_offload_cache_copy.py — synthetic (no checkpoint): per-layer decode expert
copy cost (ensure_experts + copy_missing), swept over bank layout x cache slots x
batch size x miss rate.
python benchmarks/bench_offload_cache_copy.py
For host RAM vs PCIe bandwidth and the offload/hybrid backend pick, use ft bench bw
instead — it writes the JSON profile the engine reads.