Commit Graph
5 Commits
Author SHA1 Message Date
Xiaoze Fan c2c3e828e8 feat(spec): verify draft tokens on the target side (#836)
* feat(spec): add the speculative verify infrastructure

* feat(scheduler): run decode steps as speculative verify steps

* chore(engine): mark the speculative decoding config fields with one comment

* feat(server): add short aliases for the speculative decoding flags
2026-10-28 21:18:02 -07:00
Xiaoze Fan d043fb61f8 refactor(layers): share one GatedDeltaNet op in layers/linear_attn (#841)
* refactor(layers): share one GatedDeltaNet op in layers/linear_attn

* test(layers): restore the global context in the GDN test and fix stale comments
2026-10-28 17:22:50 -07:00
Xiaoze Fan a2ce636083 refactor(attention): move linear-attention metadata into a LinearAttnBackend (#837)
* refactor(attention): move linear-attention metadata into a LinearAttnBackend

* refactor(attention): make LinearAttnBackend a BaseAttnBackend with FLACaptureData

* refactor(attention): key every linear-attention row by fla_metadata.cache_indices
2026-10-27 23:56:59 -07:00
Xiaoze Fan bd8f3d519a feat(qwen4_exp): support Qwen3.8-Flash-Next (#257)
Serve Qwen3.8-Flash-Next (HF model_type qwen4_exp) text-only: 36 GDN +
12 QSA compressed-sparse attention layers on 4 hyper-connection residual
streams, a PLE n-gram embedding layer backed by a 47.7 GiB pinned-host
table with UVA gather, and 512 NVFP4 / block-fp8 routed experts (top-10)
plus a gated shared expert.

- attention: qsa_sparse backend (AttnType.QSA) over QSAKVCache -- paged
  GQA K/V, a 1/ratio compressed index-key slab shadowing the KV pages,
  and a per-request pending ring sized from index_ratio
- kvcache: declarative slot-sibling states (ModelConfig.slot_states) on
  LinearStatePool carry the PLE conv history and n-gram context through
  the hybrid-radix snapshot/COW lifecycle
- scheduler: hybrid prefill chunks align to the page size so snapshots
  land on donatable boundaries
- kernels: triton kernels adapted from vLLM/SGLang (hc, qsa, ple gather,
  moe router / shared gate) plus an original radix block top-k; int64
  row addressing throughout
- moe: non-power-of-2 top-k router, deep-K marlin decode config, one
  fp8 scale-bank padding rule shared with the AOT row table
- engine: the PLE table load reserves its pinned bytes from the pin
  budget before the expert banks plan their residency
2026-08-28 15:33:15 -07:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00