Files
FreeToken/python/freetoken/spec/dflash/window.py
Xiaoze Fan 08cee44468 feat(spec): add DFlash 2 speculative decoding (#857)
* feat(spec): hand the pending token to Proposer.propose and return a SpecProposal

* feat(engine): Sampler.probs and sparse-q rejection

* feat(spec): let a proposer declare slot states riding the GDN slots

* feat(models): export layer outputs to the feature sink in the qwen3_5 and qwen4_exp families

* feat(attention): record the source slot of each tracked request

* feat(kernel): DFlash2 draft kernels

* feat(models): DFlash2 draft family

* feat(spec): DFlashWindow

* feat(spec): DFlashProposer

* docs(cli): add dflash to the speculative decoding section

* refactor(engine): share the sampling backend choice and drop the extra renorm in Sampler.probs

* fix(models): build the DFlash rope table uncached and refuse draft scalings it does not apply

* refactor(spec): share the draft path check between the MTP and DFlash proposers

* test(e2e): compare rebuilt outputs exactly only without hybrid MoE

* refactor(spec): keep proposer slot states on EngineConfig instead of replacing model_config

* perf(spec): compute the verify distribution once for sparse-q rejection

* perf(models): let the feature sink add the residual on the rows it keeps

* perf(models): share the DFlash draft K/V weights with the ring projection

* fix(kernel): keep the dflash selector off zero-probability candidates

* test(spec): drop the DFlash tests that the draft forward and Monte Carlo tests already cover

* fix(spec): default and cap DFlash steps at the draft's block_size - 1

* fix(models): reject DFlash draft configs the draft forward cannot serve

* test(e2e): read the MoE strategy from the rebuild test's extra args

* docs(spec): say that slot-state owners write their own prefill track snapshot

* test(models): move the DFlash draft parity tests under tests/models/dflash

* refactor(spec): drop the no-op verify handler of the DFlash window

* refactor(models): fold the DFlash candidate top-k into propose

* refactor(spec): declare the draft step limits through Proposer.num_steps only

* refactor(kernel): make the DFlash ring attention stateless and keep its workspace on the ring

* refactor(models): name the DFlash draft submodules like the other model families

* refactor(spec): move SpecProposal next to DraftContext in spec/proposer.py

* refactor(spec): hand the verify the draft distribution over the whole vocabulary

* refactor(spec): call the DFlashRing object dflash_ring and its length ring_len

* refactor(models): split the DFlash draft's propose into compute_candidates and sample_path

* refactor(spec): keep the draft distribution on its support in SpecProposal

* fix(spec): drop the step's proposal once the verify has read it

* refactor(spec): name the DFlash feature sinks DFlashPrefillSink and DFlashVerifySink

* refactor(models): read the declared Batch.features directly instead of through getattr

* test(dflash): check the window rebuild inside the block forward test

* test(spec): share the plain run of the verify contract and run the DFlash draft once per family

* test(kvcache): drop the window cases from the pool tests and check a layerless state's bytes

* test(spec): check the verify probs of each row against a torch top-k and top-p reference

* test(e2e): add an accept-len floor and a prefill chunk knob to the AIME gate

* refactor(scheduler): let callers take the speculative counts from the status reporter

* test(spec): keep the shared plain runs of the verify contract in their own temp dirs

* test(models): give the ricursive test's stand-in batch a features field
2026-11-07 12:29:18 -08:00

46 lines
2.1 KiB
Python

"""DFlashWindow: the draft's context rows of each request's last positions, riding the target's GDN slots so a prefix-cache hit restores them with the GDN state and the ring is rebuilt from them."""
from __future__ import annotations
import torch
from freetoken.models.config import SlotStateSpec
class DFlashWindow:
"""Context rows of positions ``p`` at row ``p % R`` of each GDN slot; ``fmt="unquantized"`` keeps them in the activation dtype, one ``[R, H]`` tensor per slot."""
NAME = "dflash_window"
FORMATS = ("unquantized",)
@classmethod
def length(cls, window: int, block: int) -> int:
"""Rows per slot: the window plus room for a verify's unaccepted block rows and for a prefill's rows past the GDN boundary it tracks."""
from freetoken.kernel.fla.chunk import CHUNK_SIZE
return window + max(block, CHUNK_SIZE)
@classmethod
def slot_specs(cls, R: int, H: int, fmt: str = "unquantized") -> tuple[SlotStateSpec, ...]:
cls._check(fmt)
return (SlotStateSpec(name=cls.NAME, shape=(R, H)),)
@classmethod
def _check(cls, fmt: str) -> None:
if fmt not in cls.FORMATS:
raise ValueError(f"unknown DFlash window format {fmt!r}, known: {cls.FORMATS}")
def __init__(self, tensors: dict[str, torch.Tensor], R: int, H: int, fmt: str = "unquantized") -> None:
self._check(fmt)
self.R, self.H, self.fmt = R, H, fmt
self.tensor = tensors[self.NAME]
assert tuple(self.tensor.shape[1:]) == (R, H), f"window tensor {tuple(self.tensor.shape)} is not [slots, {R}, {H}]"
def write(self, rows: torch.Tensor, positions: torch.Tensor, slots: torch.Tensor) -> None:
"""``rows [n, H]`` of ``positions`` into ``slots``; positions distinct modulo R within one call."""
self.tensor[slots, positions.long() % self.R] = rows
def read(self, slot: int, start: int, n: int) -> torch.Tensor:
"""``[n, H]`` rows of positions ``start .. start+n-1`` held by ``slot``."""
return self.tensor[slot, torch.arange(start, start + n, device=self.tensor.device) % self.R]