Commit Graph
4 Commits
Author SHA1 Message Date
Xiaoze Fan dacac5f5fa feat(spec): add MTP drafts for the Qwen3.5 family and Qwen3.8-Flash-Next (#844)
* feat(spec): add the MTP proposer for the qwen3_5 family

* test(models): read the qwen3_5 MTP head of bf16 and block-fp8 checkpoints

* feat(spec): verify qwen4_exp targets on qsa_sparse graphs with PLE verify handlers

* feat(spec): add the qwen4_exp MTP draft from the checkpoint or a Strata pack

* docs(cli): document the speculative decoding flags

* fix(spec): refuse MTP draft sources the draft cannot read before the target loads

* docs(qwen4_exp): describe the verify path in the PLE docstrings

* fix(spec): refuse a missing relative MTP draft path before the target loads

* fix(moe): read MTP head experts on the host and require both stacked tensors

* refactor(spec): tell the final prefill chunk from prompt_len instead of the scheduler's ChunkedReq

* refactor(spec): let each MTP draft family check its own weight source

* fix(qwen4_exp): size the disk PLE eager staging for decode and verify rows

* perf(kernel): read the KV-index gather's chunk count from the grid, not a constexpr

* refactor(moe): let each family name its MTP expert tensors

* refactor(spec): force the MTP draft's MoE resident in the proposer and check it after load

* fix(spec): refuse MTP with dummy weights

* refactor(spec): move HiddenRelay under spec/mtp, its only user

* refactor(spec): drop MTPDraft.config, fold its embed hooks into embed_tokens

* refactor(models): note FeatureSink's graph binding, drop its runtime_checkable

* fix(models): read quantized MTP fc projections under the quant config
2026-11-02 11:37:09 -08:00
Xiaoze Fan c2c3e828e8 feat(spec): verify draft tokens on the target side (#836)
* feat(spec): add the speculative verify infrastructure

* feat(scheduler): run decode steps as speculative verify steps

* chore(engine): mark the speculative decoding config fields with one comment

* feat(server): add short aliases for the speculative decoding flags
2026-10-28 21:18:02 -07:00
Xiaoze Fan bd8f3d519a feat(qwen4_exp): support Qwen3.8-Flash-Next (#257)
Serve Qwen3.8-Flash-Next (HF model_type qwen4_exp) text-only: 36 GDN +
12 QSA compressed-sparse attention layers on 4 hyper-connection residual
streams, a PLE n-gram embedding layer backed by a 47.7 GiB pinned-host
table with UVA gather, and 512 NVFP4 / block-fp8 routed experts (top-10)
plus a gated shared expert.

- attention: qsa_sparse backend (AttnType.QSA) over QSAKVCache -- paged
  GQA K/V, a 1/ratio compressed index-key slab shadowing the KV pages,
  and a per-request pending ring sized from index_ratio
- kvcache: declarative slot-sibling states (ModelConfig.slot_states) on
  LinearStatePool carry the PLE conv history and n-gram context through
  the hybrid-radix snapshot/COW lifecycle
- scheduler: hybrid prefill chunks align to the page size so snapshots
  land on donatable boundaries
- kernels: triton kernels adapted from vLLM/SGLang (hc, qsa, ple gather,
  moe router / shared gate) plus an original radix block top-k; int64
  row addressing throughout
- moe: non-power-of-2 top-k router, deep-K marlin decode config, one
  fp8 scale-bank padding rule shared with the AOT row table
- engine: the PLE table load reserves its pinned bytes from the pin
  budget before the expert banks plan their residency
2026-08-28 15:33:15 -07:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00