Xiaoze Fan
eebb3f53f3
feat(moe): estimate expert-bank bytes from the model config
2026-08-23 23:06:56 +00:00
Xiaoze Fan
c41833b717
feat(moe): per-layer host-bank residency -- split lock-CPU/pin-GPU layers, auto selection under capped pin quota
2026-08-23 21:02:59 +00:00
Xiaoze Fan
9db1a39455
chore(release): 0.1.2
2026-08-19 05:57:18 +00:00
jason-fxz
12ed695386
feat(stats): record per-request TTFT and report its mean
...
The request ring only carried whole-request duration, so /v1/stats exposed no
first-token latency — the number that says how responsive a local model feels,
independent of decode speed. Streaming generations now stamp the first delta
they emit; non-streaming rows carry None and are skipped by the mean, so a
/health probe can never drag it toward zero.
Surfaced as requests.ttft_mean_ms in /v1/stats and in `ft ctl stats`.
2026-08-18 19:08:52 -07:00
Xiaoze Fan
586a6c16ce
perf(fp8): run per-tensor fp8 as W8A8 via scaled_mm ( #7 )
2026-08-18 18:04:39 -07:00
Shuo Yang
6223cda1b0
Support muse glimmer ( #4 )
...
* feat(muse_glimmer): serve Muse-Glimmer-30B
2026-08-18 12:26:05 -07:00
Xiaoze Fan
76edc16282
build(release): pypi-ready wheels, metadata, and publish workflow
2026-08-18 05:00:47 +00:00
jason-fxz
711325d17e
fix(server): stop API server when backend dies
...
Signed-off-by: jason-fxz <jason341132@qq.com >
2026-08-16 14:34:17 -07:00
Xiaoze Fan
d7cd6ed88b
[Feature] unify reasoning effort on one shared scale ( #5 )
...
* feat(server): accept every reasoning-effort dialect
* refactor(server): derive thinking gears from the checkpoint, not a registry
2026-08-15 23:40:52 -07:00
Xiaoze Fan
a2572083e9
feat(launch): support launch deepseek harness ( #3 )
...
* feat(launch): support launch deepseek harness
* fix(launch): node version gate, cloud key clearing, and dsh docs
2026-08-14 23:50:44 -07:00
Xiaoze Fan
4f510f1ba6
feat(server): enable reasoning_effort on /v1/chat/completions ( #2 )
...
* feat(server): enable reasoning_effort on /v1/chat/completions
* fix(server): glm thinking-off mislabeling and reasoning_effort mapping edge cases
2026-08-14 00:02:32 -07:00
Shuo Yang
9dc3ff4ca5
Support minimax m3 ( #1 )
...
* feat(minimax_m3): serve MiniMax-M3-NVFP4 single-GPU (block-sparse GQA + MXFP8 dense + swigluoai NVFP4 experts)
2026-08-13 16:22:41 -07:00
Xiaoze Fan
ea5348b4e3
feat(kvcache): checkpoint decode state at the tool-call opener token
2026-08-13 18:32:02 +00:00
Xiaoze Fan
ee54f638fd
fix(gemma4): read head_dim / kv heads per attention type from per_layer_config
2026-08-13 03:57:59 +00:00
Xiaoze Fan and Shuo Yang
3af9d90ee5
feat: initial open-source release
...
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu >
2026-08-11 22:53:25 +00:00