Commit Graph
155 Commits
Author SHA1 Message Date
0d652e73a4 feat(rocm): add RDNA3 and RDNA4 runtime foundation (#132)
* feat: add simple ROCm GPU support for RDNA3 (gfx1100-1103)

- Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents
- Update pinned_tensor.cpp to compile under both nvcc and hipcc
- Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family)
- Guard NVIDIA arch checks to return None on ROCm
- Skip nvcc version check in _toolchain.py when on ROCm
- Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch)
- Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm
- Add is_rocm() and driver_hip_version() in backend.py
- Add rocm-smi fallback in __main__.py for clangd generation
- Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds,
  Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache
- Add AMD ROCm classifier in pyproject.toml

* fix(rocm): fall back for single-bank fast index copy

* test(rocm): cover fast index copy fallback

* test(rocm): use explicit fast copy module import

* fix(rocm): complete RDNA3 runtime path

* fix(rocm): preserve CPU MoE implementation

* build(rocm): support RDNA4 discovery and modular SDKs

* fix(rocm): complete modular SDK runtime support

* test(rocm): harden foundation platform contracts

* fix(rocm): pin jit target and split install guide

* perf(kernel): restore ROCm non-temporal index copy

---------

Co-authored-by: bouclem <bouclement2012@gmail.com>
Co-authored-by: Yuu <206304251+nekomario28@users.noreply.github.com>
Co-authored-by: LZ-QWQ <lz1830837889@163.com>
2026-09-25 17:49:45 -07:00
Xiaoze Fan f5b9700cc1 refactor(ple): extract a shared disk row store (#548) 2026-09-24 20:16:03 -07:00
Xiaoze Fanandandy-yang-1 cee23bf44a fix(runtime): correct overlap token limits and cache safety (#546)
* fix(scheduler): count delivered tokens under overlap

* fix(moe): initialize slots used by inactive hybrid routes

* fix(kvcache): report usable window capacity floors

* refactor(moe): inline the zeroed slot cache allocation

* test: drop the new regression tests

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
2026-09-23 22:22:48 -07:00
cherry77-cloud cab110ecb7 fix(models): preserve glm4 router correction bias in fp32 (#469) 2026-09-21 18:38:43 -07:00
cherry77-cloud cc1f5c2c91 fix(engine): preserve greedy sampling in mixed batches (#471) 2026-09-18 21:00:44 -07:00
Xiaoze Fan 6410c97e15 fix(engine): raise WeightLoadError when the checkpoint cannot be read (#518)
* fix(engine): raise WeightLoadError when the checkpoint cannot be read

* fix(engine): wrap the FTW index read and pass host allocation failures through
2026-09-18 12:15:49 -07:00
Xiaoze Fan 1d8a5417ac fix(kernel): pin tvm-ffi jit arch to the bound gpu (#521) 2026-09-18 12:15:36 -07:00
Xiaoze Fan cac247a860 chore(release): 0.1.3 (#489) 2026-09-15 16:47:35 -07:00
Xiaoze Fan cade1a9e45 fix(checkpoint): serve and repair FTW checkpoints around the vision encoder (#486)
* fix(checkpoint): skip the vision tower before reading an FTW served text-only

- iter_ftw_weights takes a keep(name) predicate and drops the rejected entries before their bytes are read; load_weight passes the vision prefixes through it instead of discarding the tensors after the replay
- an FTW converted before its family served images holds no encoder tensors: the engine now refuses it with the reconvert / --text-model-only choice instead of failing on the first missing key
- ftw_tensor_names lists an index's names by kind; docs/models.md notes the pre-image-support FTWs

* feat(scripts): add the vision encoder to a pre-image-support FTW

- ftw_hotfix.py fetches the encoder's checkpoint tensors by byte range from the Hub or a local checkpoint into a scratch checkpoint dir and lets the family's own weight reader name and fuse them, so the repaired FTW holds exactly what the converter would have written
- every family with an encoder gains iter_vision_weights, the encoder-only reader behind load_vision_weight: the Qwen families and Gemma yield the tower as stored (Gemma without its clipped-linear wrapper), GLM and MiniMax reuse their tower readers, Muse fuses q/k/v as its dense pass does
- the engine's refusal of a tower-less FTW and the models.md note name the tool next to reconverting; docs/ftw-hotfix.md gains the repair row and a Hub example; the ignored-leftover prefixes follow VISION_KEY_PREFIXES
2026-09-15 12:43:09 -07:00
Xiaoze Fan 68a81ffdda feat(minimax_m3): serve image input on MiniMax-M3 (#480) 2026-09-14 21:15:17 -07:00
Xiaoze Fan 63d6471d7c feat(muse_glimmer): serve image input on Muse-Glimmer-30B (#481) 2026-09-14 20:26:39 -07:00
Manish Biswal bea8d06f19 fix(engine): resolve moe-strategy auto to fused on unified-memory GPUs (GB10) (#445)
* fix(engine): resolve moe-strategy auto to fused on unified-memory GPUs (GB10)

* refactor(engine): mark _fused_resident_ok gate FIXME, drop uma test file

Signed-off-by: iamanishx <manishbiswal754@gmail.com>

---------

Signed-off-by: iamanishx <manishbiswal754@gmail.com>
2026-09-14 20:17:54 -07:00
Xiaoze Fan 3faef36b81 feat(glm5_next): serve image input on GLM-5.3-Flash (#479) 2026-09-14 18:25:25 -07:00
Xiaoze Fan 84d236c4c7 feat(gemma4): serve image input on the Gemma-4 releases (#467) 2026-09-14 13:04:33 -07:00
cherry77-cloud e0886ccfb6 fix(models): compute shared experts before in-place routed experts (#463) 2026-09-13 22:25:08 -07:00
Xiaoze Fan db64879de4 feat(server): carry tool-result images on the following user turn
Chat templates render tool messages as text, so an image returned by a tool
(Claude Code's Read as an Anthropic tool_result block, Codex's view_image as a
Responses function_call_output list) had nowhere to go: the Anthropic path
rejected it with a 400, the Responses path stringified it. The tool message
keeps the text and the images join the user turn that follows it, as vLLM does.
2026-09-14 03:25:12 +00:00
Xiaoze Fan 8ff0cce671 feat(launch): declare image input to the agents when the server serves it
Codex, OpenCode, OpenClaw and dsh refuse to send images unless the model entry
declares them, so their image tools and attachments were dead against a vision
server. The declaration follows the server's /v1/stats model.input_modalities;
each agent has its own field (codex input_modalities, opencode modalities,
openclaw input, dsh inputModalities).
2026-09-14 03:25:06 +00:00
Xiaoze Fan f7dbab7f15 fix(quant): load ModelOpt exports that ship no input_scale (#462) 2026-09-13 17:50:48 -07:00
Xiaoze Fan 08d728d5e3 feat(mm): serve image input on the Qwen families (#454)
* feat(mm): serve image input on the Qwen VL families

- frontend: OpenAI image_url, Anthropic image blocks and Responses
  input_image (http(s), base64, gated file://) become MMItems in the
  tokenizer workers; image spans carry content pseudo ids so the radix
  cache keys them by content; mrope positions are precomputed per request
- registry: EncoderSpec per family (Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next
  register a vision encoder serving image) and ModelSpec.mm_processor;
  EngineConfig.active_encoders decides what a process builds,
  served_modalities what it accepts; --text-model-only, --mm-disable,
  --mm-encoder-weights, --mm-embed-cache-device, --mm-max-pixels
- engine and scheduler: per-request encoder-cache row ownership, chunk
  planning on the batch (jobs, gather rows, scatter rows), items encoded
  right before the LM forward and gathered into batch.mm_embeds
- models: the Qwen VL vision tower on the text tower's TP layers with
  DeepStack, block weights streamed from pinned host banks by default,
  merger outputs parked on the host between blocks; SupportsMultimodal
  (encode, place_encoder_weights) and a shared embed_input_ids
- server: image inputs rejected with a client-facing reason when no
  vision encoder is served; GET /v1/stats reports model.input_modalities

* feat(mm): replace the pixel cap with a token budget and processor kwargs

- --image-min-tokens / --image-max-tokens replace --mm-max-pixels; MMProcessor.get_mm_processor_kwargs turns the MultimodalConfig into the processor call's arguments, the default passes mm.processor_kwargs through and Qwen VL converts the budget to pixel areas in size.shortest_edge / longest_edge
- --mm-processor-kwargs hands a JSON object to the checkpoint's image processor call, applied after the token budget
- MMProcessor is an ABC that owns the model path, the config and the lazily loaded image processor; get_mm_processor returns None when --mm-disable leaves no encoder to serve
- the frontend tokenizer, the tokenizer worker and the offline LLM build one processor each and hand it to TokenizeManager, so the tokenize message carries no per-request pixel cap
2026-09-13 15:39:33 -07:00
Earl Co 953565667f fix(server): honor configured output default across APIs (#411) 2026-09-11 20:02:23 -07:00
Xiaoze Fan 0ffd5c8b29 feat(qwen3_5_moe): read every checkpoint layout through the QuantConfig (#438)
* feat(qwen3_5_moe): read every checkpoint layout through the QuantConfig

NVFP4 dense layers now declare the input_scale their scheme carries, so an
FTW written by an older build from an NVFP4 dense checkpoint needs
scripts/ftw_hotfix.py before it loads again.

* fix(quant): address scheme reader review: hotfix dialect names, gemma4 cross-shard scales, ct block-fp8 experts
2026-09-10 23:18:53 -07:00
cherry77-cloud 46d27439a4 fix(scheduler): reserve paged KV at allocation granularity (#367) 2026-09-10 16:43:39 -07:00
Xiaoze Fan ddd2e3abbe feat(qwen4_exp): load block-fp8 dense projections natively (#428) 2026-09-10 16:20:57 -07:00
Xiaoze Fan fb7f732de0 refactor(quant): hand the checkpoint QuantConfig to the weight readers (#427) 2026-09-10 01:46:44 -07:00
Xiaoze Fan fa814ab594 fix(qwen4_exp): take the expert quant kind from the checkpoint QuantConfig (#426)
Fixes #394
2026-09-09 20:00:18 -07:00
Xiaoze Fan 3d919e9bd9 fix(checkpoint): write the Qwen3.8-Flash-Next PLE table next to the FTW (#420) 2026-09-09 03:12:09 -07:00
Xiaoze Fan 477c8601b2 refactor(quant): config, scheme and method layers for quantization (#418)
* refactor(quant): config, scheme and method layers for checkpoint quantization

QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.

- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
2026-09-08 23:37:23 -07:00
cherry77-cloudandXiaoze Fan 03c28d2b15 fix(kernel): make Triton top-k/top-p sampling exact (#329)
* fix(sampling): handle Triton top-k candidate limits

* fix(kernel): exact single-launch triton top-k/top-p sampling

* fix(sampling): make fused filtering exact and portable

* fix(sampling): preserve boundary ties

---------

Co-authored-by: Xiaoze Fan <jasonfxz@sjtu.edu.cn>
2026-09-02 19:01:57 -07:00
Xiaoze Fan 6eca2d7d2b fix(models): detect nvfp4 experts behind a mixed-precision compressed-tensors format (#343) 2026-09-02 00:11:14 -07:00
Xiaoze Fan a80b4d308a fix(hf): download the shards the safetensors index names (#336) 2026-09-01 19:43:19 -07:00
Xiaoze FanandShuo Yang a2538a428b feat(glm5_next): support GLM-5.3-Flash (#332)
Hybrid 45-layer model (34 KDA + 11 DSA) with NVFP4 MoE (288 experts):

- KDA linear-attention kernels and op, hybrid-linear pool dispatch
  generalized to any paged family
- NoPE-MLA attention with a kpool DSA indexer backend: 64-token pages,
  1/ratio shadow index slab, per-request tail rings, CUDA-graph-safe
  decode, prefix-cache snapshot contract
- mHC (x4 residual streams) layer with a fused split-K triton kernel;
  torch reference kept as the semantic pin
- clamped-SwiGLU activation across the MoE stack
- weight loading for NVFP4 exports in the multimodal-wrapper layout:
  ModelOpt and compressed-tensors (RedHatAI) tensor kinds;

Co-authored-by: Shuo Yang <73746844+andy-yang-1@users.noreply.github.com>
2026-09-01 15:18:15 -07:00
Xiaoze Fan 4c0bad3f56 feat(qwen4_exp): stream the PLE n-gram table from disk (#311)
* feat(qwen4_exp): stream the PLE n-gram table from disk (--ple-backend disk)

* fix(qwen4_exp): hash disk PLE rows with the checkpoint-loaded constants

* fix(kernel): handle io_uring partial submission and validate ple_store geometry

* fix(qwen4_exp): validate PLE row coverage, widen deferred-fill signaling, log io/sync choice

* fix(qwen4_exp): zero the eager PLE staging for the warmup prefill

* fix(qwen4_exp): read back the padded decode batch for the disk PLE fill
2026-09-01 13:35:14 -07:00
Xiaoze Fan e05cff83a0 perf(moe): route fused_topk through the in-repo triton router (#319) 2026-08-31 20:21:11 -07:00
Xiaoze Fan 3a20a79038 fix(kernel): unbreak the nightly kernel-cache wheel build (#310)
* fix(kernel): move fp8_block_scale_pad into aot_models to unbreak the kernel-cache build

* fix(kernel): exclude bank rows fast_index_copy cannot compile from aot specs

* fix(moe): fail fast when fused copy is off and a bank row cannot fall back
2026-08-30 22:43:50 -07:00
Berni McCoy 58f4b9ec0e fix(kernel): avoid row-wise _scaled_mm stall on sm_89 with torch<2.12 (#243)
PyTorch < 2.12 runs row-wise FP8 _scaled_mm on sm_89 through a CUTLASS stream-K kernel whose launch ignored the current stream (pytorch/pytorch#177651, fixed by pytorch/pytorch@252bb4a in 2.12). FreeToken issues the fused per-tensor-FP8 projections (q/k/v and GDN qkv|z of the NVFP4 checkpoints) from a side stream, so on Ada every prefill of >= 256 tokens stalled the GPU and the worker hung or died (#182, #72, #220). Windows torch builds ship no row-wise kernel at all (#227).

Tensor-wise scaling is unaffected. Where row-wise is unsafe (sm_89 on torch < 2.12, or a probe on the default stream raises), a fused projection now runs one tensor-wise GEMM per part over its row slice and concatenates: the same W8A8 scheme (rel ~7e-4 to row-wise, accumulation order), one extra launch per part. The parts' row ranges come from the load-time weight_scale run-lengths; the decision and its probe run at load, never under CUDA-graph capture. FREETOKEN_FP8_ROWWISE_MM=0/1 forces either path for A/B.

Tested on RTX 4070 SUPER (sm_89), driver 591.86, torch 2.11.0+cu130, WSL2. Sweep over M on a side stream: row-wise stalls at M >= 256, the new path completes at every M. tests/kernels/test_fp8_pertensor_linear.py: the side-stream test fails on main (rc=124, 0/128 GEMMs complete) and passes here; the per-part path is compared directly against row-wise at M=1/4/64/300. Three pre-existing test_w8a8_matches_w8a8_reference cases miss the 1e-2 tolerance on this GPU (rel 0.0103-0.0107) on main and on this branch alike.

Assisted-by: Claude Fable 5
2026-08-28 21:20:16 -07:00
Xiaoze Fan bd8f3d519a feat(qwen4_exp): support Qwen3.8-Flash-Next (#257)
Serve Qwen3.8-Flash-Next (HF model_type qwen4_exp) text-only: 36 GDN +
12 QSA compressed-sparse attention layers on 4 hyper-connection residual
streams, a PLE n-gram embedding layer backed by a 47.7 GiB pinned-host
table with UVA gather, and 512 NVFP4 / block-fp8 routed experts (top-10)
plus a gated shared expert.

- attention: qsa_sparse backend (AttnType.QSA) over QSAKVCache -- paged
  GQA K/V, a 1/ratio compressed index-key slab shadowing the KV pages,
  and a per-request pending ring sized from index_ratio
- kvcache: declarative slot-sibling states (ModelConfig.slot_states) on
  LinearStatePool carry the PLE conv history and n-gram context through
  the hybrid-radix snapshot/COW lifecycle
- scheduler: hybrid prefill chunks align to the page size so snapshots
  land on donatable boundaries
- kernels: triton kernels adapted from vLLM/SGLang (hc, qsa, ple gather,
  moe router / shared gate) plus an original radix block top-k; int64
  row addressing throughout
- moe: non-power-of-2 top-k router, deep-K marlin decode config, one
  fp8 scale-bank padding rule shared with the AOT row table
- engine: the PLE table load reserves its pinned bytes from the pin
  budget before the expert banks plan their residency
2026-08-28 15:33:15 -07:00
Xiaoze Fan f7c31e92db fix(fla): stop l2norm recompiling per token count 2026-08-25 16:39:32 -07:00
Xiaoze Fan 2757bb5f91 feat(server)!: add --gpu to choose the GPU on multi-GPU machines (#117)
* feat(server)!: add --gpu to choose the GPU on multi-GPU machines

--gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index.

It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>.

/v1/stats reports the engine's GPU under "gpus".

ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name.

The daemon's /bench/profile returns the running serve's GPU profile.

BREAKING CHANGE: ft checkpoint --device is removed; use --gpu.

* refactor!: resolve --gpu via NVML and bind by UUID, not CUDA_VISIBLE_DEVICES

BREAKING CHANGE: ft bench bw --device is removed; use --gpu.

* refactor: name the id namespaces and drop the multi-device e4m3 scan

Splits the published id into _assigned_physical (UUID) and
_assigned_visible (CUDA ordinal). One process runs on one GPU, so
e4m3_native() judges that card instead of scanning every visible device.
2026-08-24 16:44:01 -07:00
Lancer bd372b630a fix(checkpoint): FTW truncated shard raises OSError instead of silent garbage load (#25) 2026-08-23 17:53:27 -07:00
Xiaoze Fan 831d38a66b fix(moe): report the residency banks actually settle at 2026-08-23 23:06:56 +00:00
Xiaoze Fan eebb3f53f3 feat(moe): estimate expert-bank bytes from the model config 2026-08-23 23:06:56 +00:00
Xiaoze Fan c41833b717 feat(moe): per-layer host-bank residency -- split lock-CPU/pin-GPU layers, auto selection under capped pin quota 2026-08-23 21:02:59 +00:00
Xiaoze Fan 9db1a39455 chore(release): 0.1.2 2026-08-19 05:57:18 +00:00
jason-fxz 12ed695386 feat(stats): record per-request TTFT and report its mean
The request ring only carried whole-request duration, so /v1/stats exposed no
first-token latency — the number that says how responsive a local model feels,
independent of decode speed. Streaming generations now stamp the first delta
they emit; non-streaming rows carry None and are skipped by the mean, so a
/health probe can never drag it toward zero.

Surfaced as requests.ttft_mean_ms in /v1/stats and in `ft ctl stats`.
2026-08-18 19:08:52 -07:00
Xiaoze Fan 586a6c16ce perf(fp8): run per-tensor fp8 as W8A8 via scaled_mm (#7) 2026-08-18 18:04:39 -07:00
Shuo Yang 6223cda1b0 Support muse glimmer (#4)
* feat(muse_glimmer): serve Muse-Glimmer-30B
2026-08-18 12:26:05 -07:00
Xiaoze Fan 76edc16282 build(release): pypi-ready wheels, metadata, and publish workflow 2026-08-18 05:00:47 +00:00
jason-fxz 711325d17e fix(server): stop API server when backend dies
Signed-off-by: jason-fxz <jason341132@qq.com>
2026-08-16 14:34:17 -07:00
Xiaoze Fan d7cd6ed88b [Feature] unify reasoning effort on one shared scale (#5)
* feat(server): accept every reasoning-effort dialect

* refactor(server): derive thinking gears from the checkpoint, not a registry
2026-08-15 23:40:52 -07:00
Xiaoze Fan a2572083e9 feat(launch): support launch deepseek harness (#3)
* feat(launch): support launch deepseek harness

* fix(launch): node version gate, cloud key clearing, and dsh docs
2026-08-14 23:50:44 -07:00