mirror of
https://github.com/FlashML-org/FreeToken.git
synced 2026-11-22 05:49:20 +08:00
Speculative verify (MTP, DFlash) and the MTP draft extend sent q_len > 1 to the extend kernel, which streams the whole context per (request, query head) with no KV split: 6.6-60x the single-row decode time at 32K-128K on sm_120. - the grouped decode packs query rows x GQA heads in the M axis, limits each row to its own position and window, and keeps the KV split - the triton backend sends batches of at most 8 rows per request without multimodal blocks to it - a program's query heads divide the GQA group, which also fixes wrong output for groups above 16 that are not a multiple of 16 (e.g. 48/2)