mirror of
https://github.com/FlashML-org/FreeToken.git
synced 2026-11-21 10:02:33 +08:00
* feat(spec): add the MTP proposer for the qwen3_5 family * test(models): read the qwen3_5 MTP head of bf16 and block-fp8 checkpoints * feat(spec): verify qwen4_exp targets on qsa_sparse graphs with PLE verify handlers * feat(spec): add the qwen4_exp MTP draft from the checkpoint or a Strata pack * docs(cli): document the speculative decoding flags * fix(spec): refuse MTP draft sources the draft cannot read before the target loads * docs(qwen4_exp): describe the verify path in the PLE docstrings * fix(spec): refuse a missing relative MTP draft path before the target loads * fix(moe): read MTP head experts on the host and require both stacked tensors * refactor(spec): tell the final prefill chunk from prompt_len instead of the scheduler's ChunkedReq * refactor(spec): let each MTP draft family check its own weight source * fix(qwen4_exp): size the disk PLE eager staging for decode and verify rows * perf(kernel): read the KV-index gather's chunk count from the grid, not a constexpr * refactor(moe): let each family name its MTP expert tensors * refactor(spec): force the MTP draft's MoE resident in the proposer and check it after load * fix(spec): refuse MTP with dummy weights * refactor(spec): move HiddenRelay under spec/mtp, its only user * refactor(spec): drop MTPDraft.config, fold its embed hooks into embed_tokens * refactor(models): note FeatureSink's graph binding, drop its runtime_checkable * fix(models): read quantized MTP fc projections under the quant config
15 lines
654 B
Python
15 lines
654 B
Python
"""Per-request draft state carried from one step to the next, by page-table row."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import torch
|
|
|
|
|
|
class HiddenRelay:
|
|
"""For each table row: the draft hidden the next proposal starts from, the first draft token it predicts, and the target hidden that closes a prefill chunk."""
|
|
|
|
def __init__(self, rows: int, hidden_size: int, dtype: torch.dtype, device: torch.device) -> None:
|
|
self.hidden = torch.zeros(rows, hidden_size, dtype=dtype, device=device)
|
|
self.d1 = torch.zeros(rows, dtype=torch.int64, device=device)
|
|
self.carry = torch.zeros(rows, hidden_size, dtype=dtype, device=device)
|