3 Commits
Author SHA1 Message Date
Xiaoze FanandShuo Yang a2538a428b feat(glm5_next): support GLM-5.3-Flash (#332)
Hybrid 45-layer model (34 KDA + 11 DSA) with NVFP4 MoE (288 experts):

- KDA linear-attention kernels and op, hybrid-linear pool dispatch
  generalized to any paged family
- NoPE-MLA attention with a kpool DSA indexer backend: 64-token pages,
  1/ratio shadow index slab, per-request tail rings, CUDA-graph-safe
  decode, prefix-cache snapshot contract
- mHC (x4 residual streams) layer with a fused split-K triton kernel;
  torch reference kept as the semantic pin
- clamped-SwiGLU activation across the MoE stack
- weight loading for NVFP4 exports in the multimodal-wrapper layout:
  ModelOpt and compressed-tensors (RedHatAI) tensor kinds;

Co-authored-by: Shuo Yang <73746844+andy-yang-1@users.noreply.github.com>
2026-09-01 15:18:15 -07:00
Xiaoze Fan 2757bb5f91 feat(server)!: add --gpu to choose the GPU on multi-GPU machines (#117)
* feat(server)!: add --gpu to choose the GPU on multi-GPU machines

--gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index.

It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>.

/v1/stats reports the engine's GPU under "gpus".

ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name.

The daemon's /bench/profile returns the running serve's GPU profile.

BREAKING CHANGE: ft checkpoint --device is removed; use --gpu.

* refactor!: resolve --gpu via NVML and bind by UUID, not CUDA_VISIBLE_DEVICES

BREAKING CHANGE: ft bench bw --device is removed; use --gpu.

* refactor: name the id namespaces and drop the multi-device e4m3 scan

Splits the published id into _assigned_physical (UUID) and
_assigned_visible (CUDA ordinal). One process runs on one GPU, so
e4m3_native() judges that card instead of scanning every visible device.
2026-08-24 16:44:01 -07:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00