mirror of
https://github.com/FlashML-org/FreeToken.git
synced 2026-11-22 00:23:05 +08:00
The request ring only carried whole-request duration, so /v1/stats exposed no first-token latency — the number that says how responsive a local model feels, independent of decode speed. Streaming generations now stamp the first delta they emit; non-streaming rows carry None and are skipped by the mean, so a /health probe can never drag it toward zero. Surfaced as requests.ttft_mean_ms in /v1/stats and in `ft ctl stats`.