Files
Xiaoze Fan e3c4e4ab49 perf(attention): run up to 8 query rows per request in the triton decode kernel
Speculative verify (MTP, DFlash) and the MTP draft extend sent q_len > 1 to the extend kernel, which streams the whole context per (request, query head) with no KV split: 6.6-60x the single-row decode time at 32K-128K on sm_120.

- the grouped decode packs query rows x GQA heads in the M axis, limits each row to its own position and window, and keeps the KV split
- the triton backend sends batches of at most 8 rows per request without multimodal blocks to it
- a program's query heads divide the GQA group, which also fixes wrong output for groups above 16 that are not a multiple of 16 (e.g. 48/2)
2026-11-13 01:00:57 +00:00
..