Commit Graph
4 Commits
Author SHA1 Message Date
Chen Cheng ee9789ff6c fix(kernel): avoid nan in triton sampling softmax when a chunk starts with -inf (#654) 2026-10-10 01:17:57 -07:00
cherry77-cloudandXiaoze Fan 03c28d2b15 fix(kernel): make Triton top-k/top-p sampling exact (#329)
* fix(sampling): handle Triton top-k candidate limits

* fix(kernel): exact single-launch triton top-k/top-p sampling

* fix(sampling): make fused filtering exact and portable

* fix(sampling): preserve boundary ties

---------

Co-authored-by: Xiaoze Fan <jasonfxz@sjtu.edu.cn>
2026-09-02 19:01:57 -07:00
Xiaoze Fan 2757bb5f91 feat(server)!: add --gpu to choose the GPU on multi-GPU machines (#117)
* feat(server)!: add --gpu to choose the GPU on multi-GPU machines

--gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index.

It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>.

/v1/stats reports the engine's GPU under "gpus".

ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name.

The daemon's /bench/profile returns the running serve's GPU profile.

BREAKING CHANGE: ft checkpoint --device is removed; use --gpu.

* refactor!: resolve --gpu via NVML and bind by UUID, not CUDA_VISIBLE_DEVICES

BREAKING CHANGE: ft bench bw --device is removed; use --gpu.

* refactor: name the id namespaces and drop the multi-device e4m3 scan

Splits the published id into _assigned_physical (UUID) and
_assigned_visible (CUDA ordinal). One process runs on one GPU, so
e4m3_native() judges that card instead of scanning every visible device.
2026-08-24 16:44:01 -07:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00