Enwei Zhu f8acfc3a30 perf(kda): replace binary prefill with FlashInfer CuTe kernels (#3665)
<!-- Headings and layout are optional. Include change context and
validation
(checks performed or planned, or why testing was not run or is not
applicable)
anywhere in the description. -->

## Summary

Replace the binary KDA prefill dependency with FlashInfer CuTe kernels
under the existing `cutedsl_kda` backend.

- Consume strided Q/K/V directly from convolution output and remove
obsolete preparation and installation code.
- Select ordinary or persistent CuTe using kernel eligibility,
request/head count, and SM count. Build persistent schedules from live
GPU boundaries for eager execution and graph replay, with
invocation-owned scratch and startup warmup.
- Optimize recurrent-state scatter with a request × tile grid and
fixed-layout specialization.

Preserve FP32 scan state, the existing zero-padding contract, and LCM
state ownership.

Persistent integration adapted from
[8fcc2985](https://github.com/lightseekorg/tokenspeed/commit/8fcc298526068cff2840c017837aefcdc3a388f3).

## Test Plan

- [x] Focused numerical, strided-input, graph-replay, and compile-reuse
tests.
- [x] State-scatter correctness, partial-tile, and large-offset tests.
- [x] Memory release/resume with zero ordinary CUDA allocation
occupancy.
- [x] Installer validation and `pre-commit run --all-files`.
- [x] GB200 profiling across 24/48/96/144 heads, 4k–16k tokens, and 1–16
requests.

Example: 96 heads, 16,384 total tokens, BF16 inputs/cache and FP32 scan
state:

| Requests | Ordinary FlashInfer CuTe | New dispatch | Speedup |
|---:|---:|---:|---:|
| 1 | 885.44 µs | 753.25 µs | 1.18× |
| 2 | 899.87 µs | 522.18 µs | 1.72× |

Both paths use the optimized scatter. Timings include gather,
input/metadata preparation, scan, and scatter: median Nsight GPU time
from 21 warmed eager calls, flushing L2 before each sequence.

Separately, the scatter optimization reduces the single-request, 96-head
scatter from ~12.6 µs to ~3.2 µs. These are kernel measurements, not an
end-to-end serving benchmark.

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
2026-11-21 00:15:23 +08:00
2026-07-12 13:37:43 -07:00
2026-05-06 13:27:02 +00:00
2026-05-06 13:27:02 +00:00
2026-05-06 13:27:02 +00:00
2026-10-14 06:11:03 -07:00

TokenSpeed: Tokens at the speed of light

TokenSpeed is a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability. The goal is to be the most performant inference engine for production agentic workloads.

Core components:

  • Scheduler: C++ control plane and Python execution plane. Request lifecycle, KV cache ownership, and overlap timing are encoded as a finite-state machine, with safe KV resource reuse enforced by the type system at compile time.
  • Kernels: pluggable, layered kernel system with a portable public API and a centralized registry including one of the fastest MLA (Multi-head Latent Attention) implementations on Blackwell for agentic workloads.
  • Entrypoint: SMG and Dynamo drive the engine from Rust over its msgpack wire, with no Python on the request path.

Differentiation

TokenSpeed takes a fundamentally different architectural approach from existing LLM inference engines. The PyTorch Foundation highlighted these architectural differentiators when welcoming TokenSpeed to the PyTorch Ecosystem Landscape:

TokenSpeed is an open source LLM inference engine and the first to separate the control plane from the execution plane. The control plane is implemented in C++ as a finite-state machine, using the type system to enforce safe resource management, including request lifecycles and KV cache state, at compile time rather than runtime. The execution plane is implemented in Python, enabling fast iteration and lowering the cognitive load for researchers and engineers. This architecture combines strong correctness guarantees in the core scheduling system with the development velocity of a high-level execution layer.

TokenSpeed also treats kernels as a first-class, modular subsystem, separating them from the core engine through a portable public API, centralized registry and selection model, and an extensible plugin mechanism for heterogeneous accelerators.

More importantly, TokenSpeed is the Switzerland of open-source LLM inference engines: neutral because it's backed by the non-profit LightSeek Foundation rather than a commercial company, with serious hardware and engineering support from vendors such as NVIDIA and AMD.

News

  • [2026/11] 🔥 331 output tokens/s per GPU on AgentX: Serving Kimi K3 on GB300 with TokenSpeed (Kimi K3 Optimization on GB300 — Part II) [blog]
  • [2026/11] Kimi K3.1 Pro at Day 0: Sparse attention on the Kimi K3 stack with TokenSpeed. [blog]
  • [2026/10] EAGLE-4 training and serving with TorchSpec and TokenSpeed. [blog]
  • [2026/10] Reflection's Beam at day 0: 501B MoE serving in NVFP4 on NVIDIA GB200 with TokenSpeed. [blog]
  • [2026/10] 🔥 TokenSpeed MLA 2 for NVIDIA Vera Rubin. [blog]
  • [2026/10] 🔥 685 tokens/s per user: serving Qwen3.8-Flash-Next on NVIDIA GB300 with TokenSpeed. [blog]
  • [2026/10] 🔥 Advancing Kimi K3 inference on AMD Instinct MI355X GPUs with TokenSpeed. [AMD blog]
  • [2026/09] Kimi K3 Optimization on GB300 — Part I. [blog]
  • [2026/08] Qwen3.8 Flash Next at Day 0 and GLM 5.3 Flash at Day 0.
  • [2026/08] Qwen3.8 at Day 0: 2.4T-Scale Inference with TokenSpeed. [blog]
  • [2026/08] TokenSpeed joins the PyTorch Ecosystem.
  • [2026/07] Kimi K3 at Day 0: Frontier Model Enablement on Leading Platforms with TokenSpeed. [blog]
  • [2026/07] TML Inkling at Day 0: FP4 Inference on NVIDIA and AMD with TokenSpeed. [blog]
  • [2026/06] Deep dive into the design and optimization of TokenSpeed-Kernel. [PyTorch blog]
  • [2026/05] 🚀 TokenSpeed hits 580 TPS on Qwen3.5-397B-A17B for agentic workloads. [PyTorch blog]
  • [2026/05] 🚀 TokenSpeed is a brand new inference engine purpose-built for speed-of-light agentic workloads. [NVIDIA AI X]
  • [2026/05] TokenSpeed announced — a speed-of-light LLM inference engine for agentic workloads. [blog]

Blogs and talks

For technical blogs, conference talks, and engineering articles from the LightSeek Foundation, visit the LightSeek Blog.

Sponsors and partners

LightSeek's work is advanced by the support of sponsors and partners across the AI ecosystem.

Performance comparison

TokenSpeed vs. TensorRT-LLM Pareto curves on agentic workloads (Kimi K2.5, B200)

TokenSpeed vs. TensorRT-LLM Pareto curves on agentic workloads (Kimi K2.5, B200)

Documentation

Start here:

Citation

@misc{tokenspeed2026,
  author       = {{TokenSpeed Team}},
  title        = {{TokenSpeed}: A Speed-of-Light {LLM} Inference Engine},
  year         = {2026},
  howpublished = {\url{https://github.com/lightseekorg/tokenspeed}}
}
Languages
Python 86.7%
C++ 7.3%
Rust 5.5%
Shell 0.3%
Cuda 0.1%