Commit Graph
15 Commits
Author SHA1 Message Date
Xiaoze Fan eebb3f53f3 feat(moe): estimate expert-bank bytes from the model config 2026-08-23 23:06:56 +00:00
Xiaoze Fan c41833b717 feat(moe): per-layer host-bank residency -- split lock-CPU/pin-GPU layers, auto selection under capped pin quota 2026-08-23 21:02:59 +00:00
Xiaoze Fan 9db1a39455 chore(release): 0.1.2 2026-08-19 05:57:18 +00:00
jason-fxz 12ed695386 feat(stats): record per-request TTFT and report its mean
The request ring only carried whole-request duration, so /v1/stats exposed no
first-token latency — the number that says how responsive a local model feels,
independent of decode speed. Streaming generations now stamp the first delta
they emit; non-streaming rows carry None and are skipped by the mean, so a
/health probe can never drag it toward zero.

Surfaced as requests.ttft_mean_ms in /v1/stats and in `ft ctl stats`.
2026-08-18 19:08:52 -07:00
Xiaoze Fan 586a6c16ce perf(fp8): run per-tensor fp8 as W8A8 via scaled_mm (#7) 2026-08-18 18:04:39 -07:00
Shuo Yang 6223cda1b0 Support muse glimmer (#4)
* feat(muse_glimmer): serve Muse-Glimmer-30B
2026-08-18 12:26:05 -07:00
Xiaoze Fan 76edc16282 build(release): pypi-ready wheels, metadata, and publish workflow 2026-08-18 05:00:47 +00:00
jason-fxz 711325d17e fix(server): stop API server when backend dies
Signed-off-by: jason-fxz <jason341132@qq.com>
2026-08-16 14:34:17 -07:00
Xiaoze Fan d7cd6ed88b [Feature] unify reasoning effort on one shared scale (#5)
* feat(server): accept every reasoning-effort dialect

* refactor(server): derive thinking gears from the checkpoint, not a registry
2026-08-15 23:40:52 -07:00
Xiaoze Fan a2572083e9 feat(launch): support launch deepseek harness (#3)
* feat(launch): support launch deepseek harness

* fix(launch): node version gate, cloud key clearing, and dsh docs
2026-08-14 23:50:44 -07:00
Xiaoze Fan 4f510f1ba6 feat(server): enable reasoning_effort on /v1/chat/completions (#2)
* feat(server): enable reasoning_effort on /v1/chat/completions

* fix(server): glm thinking-off mislabeling and reasoning_effort mapping edge cases
2026-08-14 00:02:32 -07:00
Shuo Yang 9dc3ff4ca5 Support minimax m3 (#1)
* feat(minimax_m3): serve MiniMax-M3-NVFP4 single-GPU (block-sparse GQA + MXFP8 dense + swigluoai NVFP4 experts)
2026-08-13 16:22:41 -07:00
Xiaoze Fan ea5348b4e3 feat(kvcache): checkpoint decode state at the tool-call opener token 2026-08-13 18:32:02 +00:00
Xiaoze Fan ee54f638fd fix(gemma4): read head_dim / kv heads per attention type from per_layer_config 2026-08-13 03:57:59 +00:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00