Files
Enwei Zhu f8acfc3a30 perf(kda): replace binary prefill with FlashInfer CuTe kernels (#3665)
<!-- Headings and layout are optional. Include change context and
validation
(checks performed or planned, or why testing was not run or is not
applicable)
anywhere in the description. -->

## Summary

Replace the binary KDA prefill dependency with FlashInfer CuTe kernels
under the existing `cutedsl_kda` backend.

- Consume strided Q/K/V directly from convolution output and remove
obsolete preparation and installation code.
- Select ordinary or persistent CuTe using kernel eligibility,
request/head count, and SM count. Build persistent schedules from live
GPU boundaries for eager execution and graph replay, with
invocation-owned scratch and startup warmup.
- Optimize recurrent-state scatter with a request × tile grid and
fixed-layout specialization.

Preserve FP32 scan state, the existing zero-padding contract, and LCM
state ownership.

Persistent integration adapted from
[8fcc2985](https://github.com/lightseekorg/tokenspeed/commit/8fcc298526068cff2840c017837aefcdc3a388f3).

## Test Plan

- [x] Focused numerical, strided-input, graph-replay, and compile-reuse
tests.
- [x] State-scatter correctness, partial-tile, and large-offset tests.
- [x] Memory release/resume with zero ordinary CUDA allocation
occupancy.
- [x] Installer validation and `pre-commit run --all-files`.
- [x] GB200 profiling across 24/48/96/144 heads, 4k–16k tokens, and 1–16
requests.

Example: 96 heads, 16,384 total tokens, BF16 inputs/cache and FP32 scan
state:

| Requests | Ordinary FlashInfer CuTe | New dispatch | Speedup |
|---:|---:|---:|---:|
| 1 | 885.44 µs | 753.25 µs | 1.18× |
| 2 | 899.87 µs | 522.18 µs | 1.72× |

Both paths use the optimized scatter. Timings include gather,
input/metadata preparation, scan, and scatter: median Nsight GPU time
from 21 warmed eager calls, flushing L2 before each sequence.

Separately, the scatter optimization reduces the single-request, 96-head
scatter from ~12.6 µs to ~3.2 µs. These are kernel measurements, not an
end-to-end serving benchmark.

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
2026-11-21 00:15:23 +08:00
..
2026-05-06 13:27:02 +00:00