1332 Commits
Author SHA1 Message Date
Enwei Zhu f8acfc3a30 perf(kda): replace binary prefill with FlashInfer CuTe kernels (#3665)
<!-- Headings and layout are optional. Include change context and
validation
(checks performed or planned, or why testing was not run or is not
applicable)
anywhere in the description. -->

## Summary

Replace the binary KDA prefill dependency with FlashInfer CuTe kernels
under the existing `cutedsl_kda` backend.

- Consume strided Q/K/V directly from convolution output and remove
obsolete preparation and installation code.
- Select ordinary or persistent CuTe using kernel eligibility,
request/head count, and SM count. Build persistent schedules from live
GPU boundaries for eager execution and graph replay, with
invocation-owned scratch and startup warmup.
- Optimize recurrent-state scatter with a request × tile grid and
fixed-layout specialization.

Preserve FP32 scan state, the existing zero-padding contract, and LCM
state ownership.

Persistent integration adapted from
[8fcc2985](https://github.com/lightseekorg/tokenspeed/commit/8fcc298526068cff2840c017837aefcdc3a388f3).

## Test Plan

- [x] Focused numerical, strided-input, graph-replay, and compile-reuse
tests.
- [x] State-scatter correctness, partial-tile, and large-offset tests.
- [x] Memory release/resume with zero ordinary CUDA allocation
occupancy.
- [x] Installer validation and `pre-commit run --all-files`.
- [x] GB200 profiling across 24/48/96/144 heads, 4k–16k tokens, and 1–16
requests.

Example: 96 heads, 16,384 total tokens, BF16 inputs/cache and FP32 scan
state:

| Requests | Ordinary FlashInfer CuTe | New dispatch | Speedup |
|---:|---:|---:|---:|
| 1 | 885.44 µs | 753.25 µs | 1.18× |
| 2 | 899.87 µs | 522.18 µs | 1.72× |

Both paths use the optimized scatter. Timings include gather,
input/metadata preparation, scan, and scatter: median Nsight GPU time
from 21 warmed eager calls, flushing L2 before each sequence.

Separately, the scatter optimization reduces the single-request, 96-head
scatter from ~12.6 µs to ~3.2 µs. These are kernel measurements, not an
end-to-end serving benchmark.

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
2026-11-21 00:15:23 +08:00
WangBoandlongcat-ia-team a6df89ddce fix(pd): revoke a dead writer's access instead of pinning Decode pages (#3658)
## Problem

A failed Decode receive may reuse its pages only after every Prefill
target acknowledges the cancellation, because a failure report or
timeout does not prove that remote writes stopped
(`docs/design/event-loop.md`, PD cancellation). Sometimes that
acknowledgment never comes:

- the Prefill process died;
- it restarted at a new control port, so cancellations go to a dead
address;
- it holds a native write whose DMA may still land, so it withholds the
acknowledgment.

In all three cases the pages stayed pinned until the Decode rank
restarted. Every Prefill crash leaked the pages of the receives in
flight to it at that moment. Sleep with memory release, cache flush and
weight replacement could not drain either.

```mermaid
sequenceDiagram
    participant D as Decode scheduler
    participant O as Old Prefill
    participant N as Restarted Prefill
    D->>O: manifest (Decode page addresses)
    O->>D: RDMA writes into the pages
    Note over O: process dies, a write may be unfinished
    Note over D: heartbeat lost, room failed
    loop every second
        D->>O: Cancel (to the old control port)
    end
    Note over O: nothing listens on the old port
    Note over N: new random control port, old cache unknown
    Note over D: CancelAck never comes: pages and slot stay pinned
```

## Fix: revoke the writers instead of trusting a timeout

Each Decode rank owns a Mooncake transfer engine (`ts_mooncake::Engine`
in its PD transfer peer). The engine registers the rank's KV cache with
the RDMA NICs and serves the session Prefill handshakes with to obtain
the keys it writes with. A revocation replaces this engine; it is not
the inference engine.

Trigger: a failed receive has waited `waiting_timeout` for
acknowledgments, and its manifest named the current transfer engine. The
Decode peer then starts a revocation on a background thread:

1. Start a fresh Mooncake engine with the cache regions registered (no
lock held).
2. Under the state lock:
- fail every receive still attached to the replaced transport, whether
published or waiting for admission;
- make the fresh engine the one transfers use and registrations name
(`Transport::install_engine`);
   - switch to a new cache identity;
   - clear route registrations.
3. Deregister the cache from the replaced engine.
4. Only if that succeeded, the receives whose manifests named the
replaced engine retire without acknowledgments. Otherwise they keep
waiting and do not trigger revocations of later engines. The replaced
engine may still reach the cache, so the cache is then never released
(sleep with memory release is refused); this is fail-closed on a double
failure.

```mermaid
sequenceDiagram
    participant D as Decode scheduler
    participant R as Revocation thread
    participant E1 as Decode Mooncake engine (old)
    participant E2 as Decode Mooncake engine (new)
    participant W as Old Prefill writer
    Note over E1,E2: Decode's transfer engines:<br/>each registers the KV cache with the RDMA NICs<br/>and hands Prefill the keys it writes with
    Note over D: failed room waited waiting_timeout for CancelAck
    D-)R: start revocation (off the scheduling loop)
    R->>E2: start engine, register the cache
    Note over R: swap engine and cache identity, fail attached receives
    R->>E1: deregister the cache
    alt deregistration succeeded
        E1-->>R: deregistered, old keys invalid
        R-->>D: retire the rooms, reuse their pages
        W->>E1: RDMA write with an old key
        E1--xW: rejected by the NIC
    else deregistration failed
        E1-->>R: deregistration error
        Note over R: some regions may still be registered, so old keys may still work: rooms keep waiting for CancelAck, memory never released
    end
```

The fence, not the timeout, is what authorizes reuse. An RDMA NIC checks
each incoming write against the destination's registration, and
`ibv_dereg_mr` returns only once the NIC no longer uses the MR; the
kernel may unpin the pages right after. The timeout only decides when to
pay for the fence.

Details:

- **Transport:** only `rdma` revokes. With other transports such
receives stay pinned, as before.
- **Replaced engine kept alive:** it stays without the cache registered
for the peer's lifetime, so its session (host and RPC port) is never
reused by a later engine. A revoked writer that resolves its old session
again would otherwise reach the newer engine. The test reproduces
exactly this with a sleep/wake after the revocation when the replaced
engine is dropped.
- **Cache identity per receive:** each room records the cache identity
its route registered (and so the engine its manifest names).
Cancellations and their acknowledgments use that identity, so a receive
failed before admission still reaches Prefill under an identity it
knows. Under `deferred` decode allocation (#3394), a room's
announcements use that identity too.
- **No sleep mid-revocation:** memory release is refused while a
revocation runs, since it installs a fresh engine with the cache
registered. `revoke` marks the fenced rooms and ends the revocation in
the critical section that listed them, so a room retiring in between is
not marked again. The running revocation is also an RAII guard whose
drop ends a revocation that did not end itself (no fresh engine, or a
panic).
- **No stranded rooms:** rooms connecting during a revocation wait and
reconnect under the new identity instead of staying in Bootstrapping. A
receive that already failed never publishes its manifest later.
- **Observability:** each outcome is logged as an event with the rank,
the rooms and the old and fresh sessions:
`event=ts_decode_revocation_fenced`,
`event=ts_decode_revocation_unfenced` (error, with the pinned rooms),
`event=ts_decode_revocation_not_started`. A release refused for an
unfenced cache logs `event=ts_decode_release_refused_unfenced`, so the
lasting effect can be traced back after the first log line is gone.
- **Cost:** each revocation fails every receive attached on that rank at
that moment (its replica peers fail them too, through the agreed
transfer status), and keeps one idle engine.

`docs/design/event-loop.md` gains a "PD receive revocation" paragraph.

## Tests


`tests/cancellation.rs::a_cancellation_unacknowledged_past_the_receive_budget_is_retired_by_revoking_the_transport`
runs real peers, sockets and bootstrap with the CPU native stand-in. The
Prefill write is blocked in native I/O, so it withholds the
acknowledgment as a dead peer would. The test checks:

- the cancelled receive retires after the budget;
- a request waiting for admission fails, and admitting it afterwards
publishes no manifest (no second revocation);
- a sleep/wake while the revoked write is still blocked;
- the released write does not reach the recycled pages;
- the next request transfers through the fresh engine.

The test fails on `main` (the receive never retires). It also fails if:

- a failed receive still publishes;
- the replaced engine keeps its registration;
- the replaced engine is dropped (session port reuse).


`a_revocation_whose_deregistration_fails_keeps_waiting_for_acknowledgments`
makes the stand-in's deregistration fail. It checks that:

- the receive keeps waiting and starts no second revocation;
- it retires on the late acknowledgment;
- the cache can no longer be released.

Unit tests cover cancellations acknowledged under the cache their route
registered, revoked rooms retiring without acknowledgments, rooms marked
as the revocation ends (a room that retires before the guard drops is
not marked again), deferred-allocation announcements after a revocation,
a registration wait interrupted by a revocation reconnecting under the
fresh cache, and the logged outcome events.

`cargo test -p tokenspeed-pd`, `cargo fmt --all --check`, `cargo clippy
--all-targets -- -D warnings` and `pre-commit run --all-files` pass
locally.

## Not covered / review notes

- The fence relies on `ts_mooncake_unregister` returning only after
`ibv_dereg_mr` succeeded on every device. This needs confirming against
the Mooncake build in use, plus a run on real RDMA (kill a Prefill rank
mid-transfer and check the Decode pages are released and not
overwritten).
- Builds on #3657 (merged): a room waiting for its route's registration
acknowledgment gives up that registration when a revocation starts and
connects again under the fresh cache identity once it finished.

Signed-off-by: wangbo981016 <wangbo981016@gmail.com>
Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>
2026-11-20 20:15:44 +08:00
minedec 87a6c2995a perf(ple): optimize rank-local host gather and verify state writes (#3381)
## Summary

Qwen4-Exp PLE can spend over a millisecond reading an offloaded
embedding table during a long prefill. Its convolution also reloads the
same channel weights across tokens, and speculative verification writes
a full convolution-state window for every candidate. This PR reduces
those three costs.

- Batch multiple rows per CTA in the regular UVA gather. For a complete
table and at least 131,072 n-gram IDs, sort the IDs before the
persistent gather and write the results directly into their original
positions.
- Add a rank-local HBM cache for offloaded Qwen4-Exp PLE tables on
NVIDIA Hopper and Blackwell. It supports 16 n-gram heads, head
dimensions of 160 or 512, and BF16 lookup output. Tables can use BF16
storage or per-tensor FP8 storage, including FP8 checkpoints and BF16
weights quantized online. The gather deduplicates IDs within eligible
64-token windows per head and stages cache misses for asynchronous fill.
Lookups below 256 tokens and CUDA Graph capture and replay use the
regular UVA gather.
- Add `--enable-ple-hot-cache` to enable the cache when PLE offloading
is active. The cache is off by default.
`TOKENSPEED_PLE_HOST_CACHE_SLOTS` sets the capacity per rank and
defaults to 524,288 slots. It must be a power of two greater than one.
At the default capacity, a 160-wide FP8 table uses 84 MiB of cache
memory per rank, plus temporary miss staging.
- Tile long BF16 prefill convolution across 8 tokens and 128 channels on
NVIDIA Blackwell so one CTA can reuse the channel weights. For
speculative verify, store the carried state and one normalized input per
candidate, then reconstruct the accepted window in its destination page
with `copy_state_rows_compact`. Regular GPU convolution and the AMD
monokernel share the compact layout.

## Performance

Qwen3.8-Flash-Next-NVFP4 on a single NVIDIA SM103 GPU, TP1 unless noted.

### Agentic Bench with All Optimizations Enabled

The baseline and optimized builds ran sequentially on the same GPU with
MTP3, FP8 KV cache, FP8 PLE offload, CUDA Graph, and 16K prefill chunks.
The optimized run enabled `--enable-ple-hot-cache` with 524,288 slots
and used the branch's UVA, convolution, and compact verify paths as
applicable.

Both variants used `TOKENSPEED_SPEC_SIMULATED_ACCEPT_LEN=2.32`,
including the verification token, matching the InferenceX golden
accept-length distribution for Flash-Next native MTP3 with thinking
enabled.

EvalScope measured one round per variant and concurrency after warmup
and a prefix-cache flush, with 4 conversations at BS1, 32 at BS32, and
500 output tokens per turn. All measured requests succeeded.

| Concurrency | Metric | Baseline | This PR | Change |
| ---: | --- | ---: | ---: | ---: |
| 1 | TTFT (ms) | 282.59 | 280.44 | -0.76% |
| 1 | Decode (tokens per second per request) | 442.48 | 444.44 | +0.44%
|
| 1 | Aggregate output (tokens per second) | 353.65 | 355.46 | +0.51% |
| 1 | Decoded tokens per iteration | 2.2924 | 2.2934 | +0.04% |
| 32 | TTFT (ms) | 4,313.30 | 4,299.87 | -0.31% |
| 32 | Decode (tokens per second per request) | 67.07 | 67.34 | +0.40% |
| 32 | Aggregate output (tokens per second) | 1,284.40 | 1,288.59 |
+0.33% |
| 32 | Decoded tokens per iteration | 2.2934 | 2.2931 | -0.01% |

End-to-end changes were below 1% in this run. At BS1 and BS32, MTP3
verifies at most 128 tokens per batch, below the 256-token hot-cache
threshold. Compact verify saves 1.40 µs at BS1 and 8.31 µs at BS32 for
convolution and state commit together, a small part of the full decode
step. In the earlier 16K prefill profiles below, the host gather already
finished before its consumer in both variants, and convolution tiling
reduced the full target GPU span by 0.23%.

### Prefill convolution

Flash-Next TP1 geometry (10,240 BF16 channels, four taps, dilation 3),
with PDL and CUDA Graph enabled in the microbenchmark. The A and B
comparison isolates the tile change and compares 1 token × 256 channels
with 8 tokens × 128 channels. The convolution and final state matched
bit for bit.

| Prefill case | Baseline (µs) | Tiled (µs) | Change |
| --- | ---: | ---: | ---: |
| 2,048 tokens | 106.58 | 65.50 | −38.5% |
| 8,192 tokens | 395.25 | 217.10 | −45.1% |
| 8,192 tokens, ragged requests | 398.24 | 217.01 | −45.5% |
| 16,384 tokens | 779.66 | 423.94 | −45.6% |

In a separate Nsight Systems capture of one real 16K prefill step with
the same hot-cache path on both sides, convolution took 776.04 → 417.92
µs (−46.1%). The target GPU span was 364.08 → 363.25 ms (−0.23%).

### Host gather

#### Regular UVA gather

Both variants use the same complete FP8 PLE table with hot cache
disabled. The A and B comparison uses six measured Agentic prefill
windows. Each 8K case uses the first half of its 16K window. Timing
includes sorting and output reordering. The table reports means of
per-window medians, and all 12 outputs matched bit for bit.

| Prefill tokens | Direct UVA gather | Sort and persistent UVA gather |
Change |
| --- | ---: | ---: | ---: |
| 8K | 976.07 µs | 744.36 µs | −23.7% |
| 16K | 1,889.63 µs | 1,299.11 µs | −31.3% |

#### HBM hot cache

One Nsight Systems trace per variant measures a 16,384-token prefill
step on a live Flash-Next TP1 server. The comparison uses the baseline
and an early hot-cache implementation.

| Metric | Regular gather | HBM hot cache | Change |
| --- | ---: | ---: | ---: |
| Host gather kernel | 1,462.35 µs | 372.13 µs | −74.6% |
| Cache fill on a separate CUDA stream | — | 14.82 µs | — |

### Decode convolution and state commit

This microbenchmark compares full-window → compact verify using
`_ple_conv_state_kernel`; the state commit replaces `copy_state_rows`
with `copy_state_rows_compact`. Both use 10,240 BF16 channels with CUDA
Graph and PDL enabled. Values are medians of five alternating A and B
trials. The combined column times both launches together. Convolution
output and reconstructed state matched for every accepted length.

| Speculative steps | Batch size | Verify conv (µs) | State commit (µs)
| Combined (µs) | Combined change |
| --- | ---: | ---: | ---: | ---: | ---: |
| MTP3 | 1 | 4.78 → 3.08 | 1.11 → 1.43 | 6.02 → 4.62 | -23.3% |
| MTP3 | 8 | 11.31 → 7.66 | 1.37 → 4.07 | 13.33 → 9.75 | -26.8% |
| MTP3 | 32 | 34.10 → 15.65 | 2.35 → 12.53 | 36.76 → 28.45 | -22.6% |
| MTP7 | 1 | 5.00 → 3.22 | 1.05 → 1.31 | 6.49 → 4.77 | -26.4% |
| MTP7 | 8 | 17.80 → 7.99 | 1.28 → 3.88 | 19.38 → 11.74 | -39.4% |
| MTP7 | 32 | 61.23 → 19.24 | 2.36 → 12.55 | 64.84 → 32.06 | -50.5% |

Compact verify reduces the writes in the convolution kernel. The state
commit does more work to reconstruct the window, and the two launches
together take less time across the measured cases.

For single-token decode, both variants use the same tile geometry. With
identical inputs and output buffers, convolution latency was 2.87 → 2.87
µs at BS1, 4.84 → 4.84 µs at BS8, 14.99 → 14.93 µs at BS32. Outputs and
final states matched bit for bit.

## Test Plan

- With the fast math CUDA library on SM103, ran 39 focused kernel tests
covering FP8 and BF16 cache lookup, shard masking, validity checks,
dynamic prefill bounds, and convolution across request boundaries.
- On SM103, ran the PLE backend and cache setup tests against the
workspace and compact state-commit changes (55 passed). Coverage
includes accepted lengths, invalid pages, mixed per-layer page strides,
cache-pool rebinding, and graph replay.
- During sharded cache integration, ran the TP4 and DP4 lookup tests
against an unsharded reference, including a DP rank with zero local
tokens. Compiled the prefill tiles for SM100 and the compact monokernel
PLE stages for gfx950.
- Compared the CUDA host gather builds with and without fast math. The
Flash-Next checkpoint's FP8 scale produced bitwise-identical BF16 output
across all 256 FP8 byte encodings in the tested cold and prepopulated
caches. BF16 copies also matched. Synthetic scales near the FP32
subnormal range produced differences.

---------

Signed-off-by: minedec <jiandong@lightseek.org>
2026-11-20 00:47:43 -08:00
Nicolas Perrin 68c66eabef perf(comm): fuse scatter projections' GEMM with their reduce-scatter (#3662)
## Summary

Scatter projections (#3651) reduce every rank's partial of all rows into
owner rows. This adds `comm_ops.projection_mm_reduce_scatter`, which
takes the rows and this rank's input-shard weight and returns the owner
rows, so the backend can fuse the GEMM with the reduce-scatter:

- Rows the Lamport reduce-scatter holds keep the GEMM and that
reduce-scatter.
- Above them, groups with NVLink multicast on SM100 and SM103 run NVIDIA
CUTLASS's CuTe DSL GEMM + reduce-scatter, vendored under
`thirdparty/cute_dsl/gemm_reduce_scatter` and registered as
`cute_dsl_gemm_reduce_scatter`. Each rank stores its BF16 partial tiles
in symmetric memory and signals the tile's owner; the owner sums all
ranks' tiles with `multimem.ld_reduce` (FP32 accumulation in the switch,
one rounding) while the remaining tiles compute.
- Other groups keep the GEMM and the backend's reduce-scatter.

The kernel's state is created while the projection is prepared,
collectively and before graph capture; ranks agree on support through
the existing host agreement, so every rank takes the same collective.
Muse Spark's owned attention output projection now calls the new op.

## Results

GB200 x4, TP4 and EP4, NVFP4; #3651 vs this PR, two pairs of A and B
runs on one node.

Prefill only (2 output tokens), mean TTFT ms:

| prompt x concurrency | #3651 | this PR |
|---|---|---|
| 4k x 1 | 120 and 120 | 111 and 116 |
| 8k x 1 | 206 and 206 | 192 and 194 |
| 8k x 16 | 1584 and 1586 | 1521 and 1509 |
| 1k x 128 | 1529 and 1531 | 1474 and 1472 |

1k input and 1k output tokens, output tokens per second and mean TTFT
ms:

| concurrency | #3651 | this PR |
|---|---|---|
| 64 | 3624, 903 and 3645, 886 | 3623, 843 and 3650, 853 |
| 128 | 5342, 1569 and 5328, 1586 | 5319, 1524 and 5353, 1502 |
| 256 | 7896, 3016 and 7867, 3011 | 7894, 2885 and 7920, 2864 |

GSM8K, 512 questions at concurrency 32: #3651 0.977 and 0.965, this PR
0.984 and 0.984. Greedy outputs of six 20-shot GSM8K prompts (about 3.3k
tokens) are identical to #3651.

Alone, on 4 GB200s, projecting 8192 rows of 2048 channels per rank into
8192 outputs takes 229 us against 406 us for the GEMM and NCCL's
reduce-scatter.

## Tests

-
`tokenspeed-kernel/test/nvidia/ops/communication/test_gemm_reduce_scatter.py`:
on every GPU of the node (2, 4 or 8), the fused kernel matches an exact
integer reference within one BF16 unit, for row blocks that fill and pad
the 128-row tile, row counts that change between calls without
recompiling, graph replays with fresh inputs, and owner rows past 2^31
elements of the symmetric storage.
- `test/runtime/distributed/test_owner_row_projections.py`:
`projection_mm_reduce_scatter` matches the reduced reference for one
row, Lamport-sized and fused-kernel-sized batches, and rejects more rows
than the owners hold, with the auto and NCCL backends on 2 and 4 GPUs.

---------

Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
2026-11-20 00:42:35 -08:00
Nicolas Perrin bd8f0d4f4a perf(dsa): score and select KVP decode index candidates in two CuTe DSL launches (#3660)
## Summary

A KVP decode's local index candidates ran the DeepGEMM leaf's query
quantization, schedule metadata, paged logits, non-finite scrub, ragged
top-k, and gather, then copied the scores into the owned top-k's send
buffer: about 30 microseconds per DSA layer at one request.

- The DeepGEMM leaf writes its candidate scores straight into the send
buffer (`scores_out`) and copies the query only when it is strided.
- `dsa_decode_index_candidates` (`cute_dsl` on SM100 and SM103) runs the
fused decode top-k's scorer and select in a KVP mode: every request is
scored over its compact owned pages to each row's own local length, and
the select writes each row's exact local top-k as logical positions and
FP32 logits, recovered from its order keys. The logits are bit for bit
the DeepGEMM leaf's.
- A row that sees at most `topk` keys over every rank keeps all of them
in the merged selection, so the op takes each row's keys over every rank
(`row_lengths`): the scorer schedules no tiles for a request whose last
row fits, and the select writes such a row's local keys in key order,
scored 0, without reading logits.
- The owned decode top-k takes the op for BF16 decode queries without
mandatory windows, up to 4096 requests; other sizes keep
`dsa_index_candidates`.

## Test Plan

- [x]
`tokenspeed-kernel/test/nvidia/ops/attention/test_dsa_decode_index_candidates.py`:
9 passed on GB300 (positions and logits against the DeepGEMM leaf, rows
that fit the top-k, short and empty rows, verify windows, strided send
buffers, and a batch past the scorer's 4096 requests refused); a
position-offset mutant fails it.
- [x] Every touched test file and every test file importing a touched
module, 44 in all, on GB300, among them `test_dsa_fused_decode_topk.py`
(32), `test_cute_dsl_dsa_owned_selection.py` (12),
`test_flashmla_dsa_dcp.py` (73), `test_dsa_dcp_owned_decode_topk.py` (2
and 4 ranks), `test_kimi_k31_dcp.py` (14), `test_dsa_dcp_k31_indexer.py`
(10), and `test_dsa_dcp_index_reuse.py` (24).
- [x] One KVP8 rank's decode candidates, GB300, CUDA graph, top-k 2048,
microseconds:

  | rows | context tokens | DeepGEMM leaf | this PR |
  |-----:|-----:|-----:|-----:|
  | 8 | 1.5K (fits the top-k) | 32.5 | 5.2 |
  | 128 | 1.5K to 2K | 17.6 | 10.6 |
  | 8 | 16K | 15.0 | 8.7 |
  | 128 | 16K | 21.0 | 12.0 |
  | 8 | 64K | 21.2 | 18.2 |
  | 128 | 64K | 32.5 | 27.8 |
- [x] Kimi-K3.1, TP8 with KVP8 on 2x GB300, DSpark with synthetic
acceptance 8, 1024 output tokens, steady ms per token, two alternating
boots per arm:

  | batch | main | this PR | change |
  |-----:|-----:|-----:|-----:|
  | 1 | 1.294 | 1.223 | -5.5% (boots -9.8% and -1.1%) |
  | 2 | 1.640 | 1.602 | -2.4% |
  | 4 | 2.212 | 2.116 | -4.4% |
  | 8 | 2.942 | 2.859 | -2.9% |
  | 16 | 4.404 | 4.326 | -1.8% |
  | 32 | 5.716 | 5.618 | -1.7% |
  | 64 | 7.925 | 7.786 | -1.8% |
- [x] GSM8K, 1000 questions with real acceptance: 959 with this PR, 961
on main (2 and 4 questions right only in one arm, sign test p = 0.69).

---------

Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
2026-11-20 00:41:41 -08:00
Nicolas Perrin 89315406aa perf(dsa): run the head-sharded KVP decode on the Prims-TS core (#3659)
## Summary

A head-sharded KVP decode attends every rank's gathered query heads over
its own keys and hands the partials and their LSEs to the DCP combine.
On SM100 and SM103 it ran the trtllm-gen core, which pads K3.1's 96
gathered heads to 128, computes the LSE in another launch, and needs a
mask after it for rows owning no key. The Prims-TS core
(`flashinfer_prims`) attends the 96 heads as they are and writes the
combine's LSE itself, -inf for an empty row.

- `DSABackend` probes the head-sharded KVP decode's solution once its
own pin has landed (`_probe_kvp_decode_solution`): `flashinfer_prims`
where it serves every decode selection the forward makes (one row per
request and the verify rows) on an FP8 cache, without a solution pin or
kernel plan of the leaf's own; otherwise the leaf's pin, as before.
- The probes, the sparse-leaf log, and the trtllm-gen buffer reservation
follow the decode's solution.
- Other decodes and every prefill keep their selection.

## Test Plan

- [x] `tokenspeed-kernel/test/nvidia/ops/test_dsa_flashinfer_prims.py`
on GB300, now with a length-0 row (zero output, -inf LSE).
- [x] `test/runtime/test_dsa_backend_selection.py`: the probe pins
Prims-TS only where it serves both decode widths, keeps a pin, a plan, a
BF16 cache, and a single rank as before; on the real K3.1 selection it
pins Prims-TS on SM100/SM103.
- [x] `test/runtime/distributed/test_kimi_k31_dcp.py` and every test
file building a `DSABackend` double (their decode doubles now carry the
solution).
- [x] One KVP8 rank's decode core per layer, GB300, 24 layer caches
(cold L2), CUDA graph, 256 owned keys per row, microseconds (the same at
2K, 16K, and 64K-token contexts):

  | rows | trtllm-gen (pad, fmha, LSE, mask) | Prims-TS |
  |-----:|-----:|-----:|
  | 8 | 21.0 | 9.8 |
  | 32 | 29.0 | 9.1 |
  | 64 | 26.5 | 9.7 |
  | 128 | 38.4 | 13.0 |
  | 512 | 117.2 | 30.9 |
- [x] Kimi-K3.1, TP8 with KVP8 on 2x GB300, DSpark with synthetic
acceptance 8, 1024 output tokens, steady ms per token, two alternating
boots per arm:

  | batch | main | this PR | change |
  |-----:|-----:|-----:|-----:|
  | 1 | 1.296 | 1.312 | +1.2% (boots -0.2% and +2.6%) |
  | 2 | 1.665 | 1.664 | -0.1% |
  | 4 | 2.187 | 2.160 | -1.2% |
  | 8 | 2.932 | 2.882 | -1.7% |
  | 16 | 4.457 | 4.335 | -2.7% |
  | 32 | 5.708 | 5.641 | -1.2% |
  | 64 | 7.911 | 7.656 | -3.2% |

Batch 1 moves within the boot-to-boot spread. Profiles of one boot each
settle it: a DSA layer's attention takes 114.8 instead of 129.7
microseconds at batch 1 (the 24 DSA layers 322 microseconds less per
step, the step 1.8% shorter), and a DSA layer 333.1 instead of 355.5
microseconds at batch 8 (538 microseconds less per step, the step 1.9%
shorter).
- [x] GSM8K, 1000 questions with real acceptance: 964 against main's 956
(13 and 5 questions right only in one arm, sign test p = 0.10).

---------

Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
2026-11-20 00:40:58 -08:00
Nicolas Perrin 19b0242f23 perf(fp8): read block-FP8 activation scales in the quantizer's layout (#3654)
## Summary

On SM10x, every block-FP8 dense linear moved its activation scales into
another layout and back:

- The TRT-LLM quantizer writes MN-major scales, then copies them to
token-major (`scale.t().contiguous()`).
- At `M > 16` the CuTe-DSL GEMM copies them back to MN-major
(`a_scales.T.contiguous()`).
- `fp8_quantize_1x128` also copies on SM10x to drop its 4-row padding.

Each copy is a small PyTorch kernel recorded into every captured CUDA
graph, so a profile is the only place they show up. On
Qwen3.5-35B-A3B-FP8 decode with 32 concurrent requests, they were 320 of
the 441 copy kernels per step.

Each GEMM now declares the scale layout it reads, and the linear layer
quantizes into that layout:

- **Registrations:** block-FP8 GEMMs declare `a_scales_layout`,
`mn_major` for CuTe-DSL and `k_major` for FlashInfer. `mm` derives the
trait from the strides of `A_scales`. A one-row or one-column tensor
fits both layouts.
- **Layout query:** `mm_activation_scale_layout()` reports which layout
the kernel `mm` would select reads.
- **Linear layer:** `Fp8LinearMethod` asks that before quantizing FP32
block scales. For MN-major it uses `trtllm_quantize_fp8_mn_major`
(#2427), the same TRT-LLM kernel returning its native storage.
- **CuTe-DSL launcher:**
- It reads MN-major scales in place. Their `stride(1)` becomes the
kernel's `sf_m` row extent, so the padded native layout needs no copy.
  - It raises `ValueError` on any other layout instead of copying.
  - It requires contiguous weight scales.
- **`fp8_quantize_1x128`:** returns the padded `[K/128, M]` view on
SM10x, as it already does on other GPUs.
- **Online quantization** (`mm` with `A_scales=None`): still uses the
Triton quantizer, which writes K-major scales. When the selected kernel
reads MN-major, `mm` now transposes them explicitly. This is the same
copy as before, moved to where the layout is chosen.
- **CuTe tuner:** it builds token-major profile scales, so its input
hook converts them to MN-major. Tuning now measures the same copy-free
launch that serving runs.
- **Numerics and benchmark generators:** they build activation scales in
the layout the selected kernel declares.

Kernel selection and numerics don't change: every shape selects the same
kernel, and the quantizer produces the same values.

Not in this PR:

- The FlashInfer path (`M <= 16`) still receives token-major scales, so
it keeps one copy per linear.
- The FP8 MoE still makes its own scale round trip.

Both need changes on the FlashInfer side.

## Test Plan

On GB200, main (943bef04f) against this branch.

- [x] **Bit-exact linear outputs.** `Fp8LinearMethod.apply` and
`apply_into` covered:
  - shapes N×K of 256×512, 4096×2048, 2048×4096 and 1536×7168;
  - 17 values of M from 1 to 4096;
  - `--dense-gemm-backend` set to `auto` and to `trtllm_cutedsl`.

All 136 outputs are identical to main, and the selected kernel matches
in every case. CUDA graph replay matches eager execution.
- [x] **Fewer copy kernels per linear call**, counted with the PyTorch
profiler:

  | Backend | M | Main | Branch |
  | --- | --- | --- | --- |
  | auto | 17 to 4096 | 2.40 | 0 |
  | auto | 1 to 16 (FlashInfer) | 2.43 | 2.00 |
  | `trtllm_cutedsl` | every M | 2.29 to 2.40 | 0 |
- [x] **Copy kernels in real decode.** Qwen3.5-35B-A3B-FP8 on TP1,
profiled with nsys over 30 steady decode steps with CUDA graphs:

| Concurrency | PyTorch copy kernels per step, main | Branch | GPU busy
per step, main | Branch |
  | --- | --- | --- | --- | --- |
  | 32 | 441 | 121 | 6237 µs | 5884 µs |
  | 16 | 281 | 281 | 5078 µs | 5286 µs |

At 16 the dense GEMMs and the quantizer take the same time on both
sides. The gap there is in the FP8 MoE expert GEMMs, which this PR
doesn't touch; their time follows the sampled routing.
- [x] **Serving A/B.** Same model and GPU, main and branch alternating
for two rounds each. Random prompts of 1024 input and 256 output tokens,
4 × concurrency prompts, a fresh seed per run.
  - Greedy outputs are identical to main: 4 of 4 prompts, both rounds.

| Concurrency | Throughput (tok/s), main | Branch | Median TPOT (ms),
main | Branch |
  | --- | --- | --- | --- | --- |
  | 16 | 2617 / 2581 | 2598 / 2610 | 5.69 / 5.83 | 5.78 / 5.73 |
  | 32 | 4043 / 4022 | 4302 / 4310 | 7.21 / 7.35 | 6.79 / 6.80 |
  | 64 | 6077 / 6071 | 6281 / 6292 | 9.37 / 9.43 | 9.03 / 8.96 |

Throughput rises 6.8% at concurrency 32 and 3.5% at 64; 16 is unchanged.
Median time to first token stays within main's run-to-run range.
- [x] **Tests.** 50 test files covering block-FP8 GEMM, quantization,
selection, numerics, autotuning and FP8 linears:
- Files that assumed token-major input to CuTe-DSL now pass the layout
the kernel reads: the autotune helper, the selection boundary test, and
the numerics and benchmark generators.
- The selection test gains a case: K-major scales at `M = 512` route to
FlashInfer.
- Every other file gives the same per-file result on both sides. The 23
`test_qwen4_exp` failures also fail on main.
- [x] **New tests:**
  - `test_block_fp8_scale_layout.py`:
- CuTe-DSL gives identical output from padded and dense MN-major scales.
    - It rejects K-major scales with `ValueError`.
    - Scales in the reported layout give the canonical result.
    - Online quantization feeds CuTe-DSL MN-major scales.
- `test_fp8_linear_scale_layout.py`, registered for `runtime-1gpu`: the
linear layer launches no PyTorch copy kernel at `M >= 17`, and matches
the canonical-scale GEMM bit for bit.
- [x] All modules under `tokenspeed.runtime.layers` and
`tokenspeed.runtime.models` import: 149 of 149.
- [x] pre-commit on the changed files.

---------

Signed-off-by: Nicolas <nperrin.ai@gmail.com>
2026-11-20 00:40:31 -08:00
ShixianVincentandShixianVincent ba93b55ab6 fix(dflash): guard Iris import on unsupported platforms (#3663)
DFlash startup imports Iris before checking whether the device supports
its fused reduction. On Ascend, Iris reads Gluon dtypes at import time
while `gl` is `None`, so startup fails before the capability check.

Check for CDNA4 before importing Iris. Other devices use the existing
unfused path, and CDNA4 retains the existing eligibility checks.

Validation: `pre-commit run --all-files` passed. An equivalent guard was
exercised on Ascend with ACL Graph capture and a successful chat smoke
request.

Signed-off-by: ShixianVincent <6891657+VEN-DSX@users.noreply.github.com>
Co-authored-by: ShixianVincent <6891657+VEN-DSX@users.noreply.github.com>
2026-11-20 15:38:24 +08:00
dongjiyingdjy 1e433a5b9d feat(scheduler): fill the budget a holding prefill chunk leaves (#3255)
## Summary

This PR ports and adapts
https://github.com/lightseekorg/tokenspeed/pull/2661/changes/7f071ac1cb0c6fb9dc26dfa838f76d30d087d83d
to main branch.

With `--prefill-chunk-yield-tokens N > 0` (P role), a prefill chunk that
cannot finish its prompt gives way to at most N tokens of whole prompts
behind it (#2961). The chunk often still leaves budget idle: a junction
cut (KV cached past the last State checkpoint) ends it short of where it
gave way, and a new prompt cut at its junction never gives way at all.
This PR fills that leftover budget with whole prompts that complete in
the same round. The fill is P-role behaviour whatever N; N only bounds
what a chunk gives way to.

## Behaviour changes (P role; fused and D plans are unchanged)

- The once-per-round whole-prompt probe (still at most 8 prompts) keeps
prompts up to max(N, budget left). Prompts above N are never given way
to, so what the chunk gives way to and where it ends are unchanged; with
N = 0 no chunk gives way, as before.
- Once the chunk is admitted and keeps the head of line, and the prompts
it gave way to are admitted, the remaining budget is filled from the
same list by the same `carryWholePrompts` (must_complete; stops at a
full batch or the first capacity failure), with N = 0 too. A round that
had not probed yet probes after the chunk's admission. A chunk whose
admission compacted a lone request (#3065) fills none, as it carries
none.
- With `--enable-short-prefill-first` (#3179), the fill follows the
ranking instead: a protected prompt first, then least remaining work,
including prompts submitted before the chunk's own, with no probe limit;
a resident chunk ranks them after its admission. The carry keeps #3179's
bounded scan in submission order.
- Invariants kept: at most one incomplete prefill per round, and it runs
last; completing prefills form a prefix of the batch.
- Filled prompts hold their pages until their transfer is acknowledged,
so in a tight pool the chunk's next admission may wait for those ACKs.
They do not depend on the chunk: D allocates destinations before the
request is bootstrapped.
- Docs: `docs/design/scheduler.md` (§1.1, §1.4, §3.1, §5),
`--prefill-chunk-yield-tokens` help, `docs/configuration/server.md`.

## Performance

K3 on GB300, 1P1D (P: TP8 KVP8, prefill graphs, chunk 8192), N = 2048 in
both arms, one pass each. Measured while the fill still required N > 0;
not re-measured with N = 0.

**AgentX traffic** (measured on main, before the last rebase; behind the
gateway, c64, one 1-hour window per arm, 0 request errors). TTFT in ms.

| Arm | Requests | requests per second | TTFT p50 | TTFT p90 |
|---|---|---|---|---|
| main | 7,386 | 2.029 | 1,794 | 8,086 |
| this PR | 7,363 | 2.023 | 1,792 | 7,845 |

No measurable difference: arm-to-arm variation on this setup is about 1%
in requests per second and 80 ms in TTFT p50. This workload gives the
fill little to do. Junctions are 5.0% of first admissions (369 requests
per arm), the prefill side is not saturated (about 72% busy; decode is
the bottleneck), and a cut chunk rarely has a whole prompt waiting
behind it: of 151 cut rounds read from the logs, 3 admitted whole
prompts after the chunk, and 1 of them was beyond what N already allows.

**Earlier measurement** (main vs this change on it, submission order; D:
TP8). TTFT in seconds.

| Workload | Arm | requests per second | TTFT p50 | TTFT p90 | P
forwards |
|---|---|---|---|---|---|
| Junction-heavy (8 trunks x 16 forks + short requests, 616 req) | main
| 6.662 | 2.323 | 7.238 | 298 |
| | this PR | 7.049 (+5.8%) | 2.282 | 6.508 (-10.1%) | 255 |
| Agentic (p_bench P workload, c64, 4137 req) | main | 15.504 | 3.759 |
4.272 | 771 |
| | this PR | 15.459 (-0.3%) | 3.775 | 4.358 | 760 |

Outputs were token-identical across arms (616 of 616 junction-heavy, 124
of 124 junction correctness set, 16 of 16 greedy); gsm8k 0.965 with N =
0 and N = 2048.

## Test plan

- Scheduler gtest 1642 passed of 1643, 1 skipped as on main (17 new
tests; 15 fail with the fill disabled, and the fused-role test fails if
the fill runs on every role), pytest 431 passed, `pre-commit run
--all-files`.
- Closed-loop plan digests on this head: fused and D roles with N = 0
identical to main on 7160 jobs; P role with N = 0 differs from main on
1222 of 1400 jobs (the fill) and no job raises. Before the rebases onto
current main: P role with N = 24 identical to the previous head on 4200
jobs, and randomized P and 1P1D stress with N = 0 in tight pools (17,000
jobs, in submission order and with `--enable-short-prefill-first`)
showed no invariant violation and no hang or stall that main does not
have.
- GB300 K3 1P1D (P: TP8 KVP8, prefill graphs, chunk 8192), before the
rebases onto current main, this PR against main with N = 0 on both:
outputs token-identical on the 16-prompt greedy set and on the
124-request junction correctness set (two passes each; each build
repeated against itself as the control); the fill admitted whole prompts
in 54 plans with this PR, 46 of them inside the junction correctness
set, and in none on main; 108 concurrent requests, 0 errors. This check
ran without speculative decoding: on that base (main, with or without
this PR) the K3 decode engines exit during warm-up when DSpark is
enabled (`positions and loc must hold 320 rows` in the drafter's
context-KV write), a path this PR does not touch.

---------

Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>
2026-11-20 14:46:26 +08:00
Shikhar MishraandTuan Zhang 496598e4ab perf(gdn): optimized FlashInfer backend when ReplaySSM chain is disabled (#3312)
<!-- Headings and layout are optional. Include change context and
validation
(checks performed or planned, or why testing was not run or is not
applicable)
anywhere in the description. -->

## Summary

With `--disable-replay-ssm` and FP32 state, a GDN chain verify goes
through FlashInfer's MTP kernel. Each GDN layer still launches two copy
kernels around the conv update:
- `pack_gdn_projection` copies the projection views into dense tensors.
- `fused_qkv_split_gdn_prefill` copies the conv output into separate q,
k and v.

#2702 maded the conv update write dense q/k/v rows (`out=`) and
ReplaySSM payload (`replay_payload=`). Even after #2583, backend takes
that path when the selected verify kernel reads projection views under
ReplaySSM. FlashInfer backend couldn't read projection views, so it
still does the `pack and split`.

This PR makes every GDN chain verify take the conv path:
- conv update reads the projection views and writes dense `q|k|v` rows.
Under `ReplaySSM` it also fills the payload.
- `q/k/v` are just views of those rows. FlashInfer accepts them, since
`_qkv_rows_strided` doesn't count them as strided, so it doesn't effect
kernel selection.
- FlashInfer reads `a`/`b` as projection views. The MTP runner passes
them with dynamic layout and no alignment assumption.

## Performance

Setup:
- B200.
- Qwen3.5-397B's GDN layer shapes on one GPU at TP4: 4 key and 16 value
heads, dim 128.
- MTP 3 (4 verify rows), FP32 state, `--disable-replay-ssm`.
- One GDN layer's pack, conv, split and FlashInfer verify, in a CUDA
graph.

| Batch | main (µs) | PR (µs) | Speedup | Saved per decode step (×45 GDN
layers) |
|---|---|---|---|---|
| 1 | 10.23 | 7.88 | 1.30× | 106 µs |
| 2 | 11.69 | 8.93 | 1.31× | 124 µs |
| 4 | 14.07 | 12.77 | 1.10× | 58 µs |
| 8 | 17.66 | 14.62 | 1.21× | 136 µs |
| 16 | 24.05 | 23.88 | 1.01× | 8 µs |

Qwen3.5-397B's CI perf and eval configs serve with
`--disable-replay-ssm`.

## Tests

- `test_flashinfer_mtp_reads_conv_rows_and_projection_gates`:
FlashInfer's FP32 verify conv-row q/k/v and projection-view a/b,
including 16-byte gate offsets, equals to the inputs. Covers the inline
and warp-specialized kernels, read-only and per-token state writes.
- `test_chain_verify_conv_rows_match_the_split`: through the backend,
the conv-rows path matches the split path (output and committed states),
for BF16/FP32 state with ReplaySSM on and off.
- The existing
`test_gdn_backend_reads_projection_views_like_dense_inputs`
(`GdnAttnBackend` with FlashInfer, ReplaySSM off, misaligned `a`) now
covers the new path.

Signed-off-by: shikhar <shikhar2807.ace@gmail.com>
Co-authored-by: Tuan Zhang <tuan@lightseek.org>
2026-11-20 14:24:14 +08:00
dongjiyingdjy c20ea8c988 feat(pd): let a prefill compute before its decode allocation (opt-in) (#3394)
## Summary

`--disaggregation-decode-allocation-policy deferred` (prefill servers,
experimental): a prompt is computed without waiting for its decode node
to allocate the destination; only its KV transfer waits. The default,
`early`, keeps today's admission and is marked deprecated in the docs
and the flag's help.

Replaces #3218 with the ordering proposed in its review: a request takes
every page it needs on the prefill node before its decode node
allocates, so the two nodes cannot wait for each other and nothing is
requeued or computed twice. Based on 58e9f197a in #2661.

## Behaviour

- **Order:** a decode rank announces the room; the prefill rank admits
the request once it is announced; when the plan with the last chunk is
dispatched, the prefill rank notifies the decode ranks, which only then
allocate.
- **Scheduler:** a prompt computed before its destination waits in
`PrefillAwaitingDestination` with its pages and slot. It counts as
running.
- **Admission:** unchanged from `early`, with or without a computed
prompt waiting for its destination: new prompts are admitted the same
way, into the pages left, and a smaller prompt passes one that does not
fit.
- **Decode side:** the scheduler does not change. It sees every request
whose notice arrived and chooses among them.
- **Timeouts** (the bootstrap timeout, by what a request waits for): its
decode node's announcement, from registration; its destination, from the
pages-held notice. A backlog on the prefill rank fails nothing there;
the decode node's waiting timeout covers it.
- **Flag:** `deferred` needs attention TP 1, KVP 1, PP 1, layerwise
interval 0, no external encoder and a text-only model; anything else is
refused at start-up.
- **Scheduler config:** a P-role config without a policy is `early`, so
a runtime that does not set the field keeps working.
- **`early`:** plans identical to main.

## Known limits

- With a tight prefill pool `deferred` is slower than `early` even when
every decode node is healthy: computed prompts wait for their
destinations with their pages (seen in a CPU closed-loop probe, not
measured on GPU).
- While computed prompts wait for their destinations, new prompts are
admitted as under `early`: on a tight prefill pool smaller prompts that
keep fitting can pass a larger one until they stop arriving.
- A decode server that does not allocate keeps its computed prompts'
pages on the prefill server until each times out, and prompts for other
decode servers wait for those pages meanwhile.
- On a full decode server a larger request can be passed by later
smaller ones while its prompt's pages stay held on the prefill server,
until it is allocated or times out.
- A prompt waiting for its destination has not published its last chunk;
a prompt with the same prefix admitted meanwhile computes it again.
- Every decode server of a `deferred` prefill server needs this change;
an older one fails each request at once.

## Test plan

- **CPU:** scheduler gtest 1620 (1 skipped: needs ASan), scheduler
pytest 431, cargo tests of ts-control, ts-pd, ts-scheduler, ts-metrics
and ts-host 499, clippy and pre-commit pass; closed-loop plans under
`early` identical to main on 4320 jobs.
- **GB300, Kimi-K3 1P1D (attention DP8), without speculative decoding,
on the head before the rebase onto #3362:** greedy outputs identical
under `early` and `deferred` (16 of 16, twice); 96 concurrent requests
against 16 decode slots complete under `deferred` (37.9 s, no timeout);
cancels end cleanly; with the decode server stopped for 55.6 s and a 30
s bootstrap timeout, 48 of 64 requests complete, 16 fail with that
timeout, and no page stays held afterwards.

---------

Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>
2026-11-20 11:26:22 +08:00
Nicolas Perrin 94397ff099 refactor(comm): add gather and scatter projections for owner rows (#3651)
## Summary

Owner-row projection workspaces (`comm_backend/projection.py`) served
the DP column and row layouts only, so Muse Spark's owned rows on NVIDIA
kept their own copy-engine gather and NCCL collectives in the model.
This adds two layouts beside DP column and row, and moves that
communication behind them:

- `gather`: every owner's rows through this rank's output shard.
`comm_ops.projection_gather_mm` gathers and projects them; the backend
picks Lamport for the rows it holds, copy-engine pulls overlapped with
per-owner GEMMs where the group maps its memory, and its ordinary
all-gather and one GEMM otherwise.
- `scatter`: every rank's partial of all rows reduced into owner rows,
through the existing `acquire_projection_output` and
`projection_reduce_scatter`.

The copy-engine gather writes this rank's rows into symmetric memory,
passes a barrier, and pulls peers' rows on a side stream while its own
rows project. A second barrier after the pulls, which the gather joins
before returning, lets the next gather rewrite the rows, so gathers
carry no state between calls (the previous parity scheme relied on every
forward gathering an even number of times).

Muse Spark prepares four workspaces (gathers for Q, K and V and for the
pre-expert projection, the output-projection scatter and the post-expert
column exchange) on one scratch block and calls the shared ops.
`_CopyEngineRows` and `_NcclOwnedRows` are removed, and the owned path
no longer needs a model-side peer-mapping gate: groups the copy engine
cannot reach take the ordinary path.

## Results

GB200 x4, TP4 and EP4, NVFP4; main vs this PR, two A and B pairs on one
node.

Prefill only (2 output tokens), mean TTFT ms:

| prompt x concurrency | main | this PR |
|---|---|---|
| 4k x 1 | 117 and 119 | 120 and 120 |
| 8k x 1 | 206 and 205 | 202 and 202 |
| 8k x 16 | 1579 and 1582 | 1584 and 1586 |
| 1k x 128 | 1529 and 1521 | 1535 and 1544 |

1k input and 1k output tokens, output tokens per second and mean TTFT
ms:

| concurrency | main | this PR |
|---|---|---|
| 64 | 3646, 913 and 3675, 863 | 3648, 876 and 3646, 864 |
| 128 | 5346, 1561 and 5320, 1584 | 5325, 1596 and 5291, 1686 |
| 256 | 7892, 3012 and 7916, 2977 | 7890, 3003 and 7901, 3004 |

GSM8K, 512 questions at concurrency 32: main 0.982 and 0.979, this PR
0.980 and 0.979.

The second barrier is the cost of carrying no state: isolated on 4
GB200s with ranks in step, a gather plus its GEMMs takes 167 -> 173 us
at 1000 rows per rank and 326 -> 335 us at 2048, about 1% of a 4k
prefill.

Greedy outputs of six 20-shot GSM8K prompts (about 3.3k tokens, so every
forward takes the owned path) are identical to main.

## Tests

- `test/runtime/distributed/test_owner_row_projections.py`: on 2 and 4
GPUs, with the auto and NCCL backends, gathers and scatters match
gathered and reduced references for one row, Lamport-sized and
copy-engine-sized batches with unequal owner rows; three captured
gathers replay on new rows.
- `test/runtime/distributed/test_muse_owned_projections.py` (renamed
from `test_muse_owned_rows_nccl.py`): Muse's four owned projections
match references for odd row counts, eagerly and as two captured layers.
- `test/runtime/models/test_muse_spark.py`, `test_muse_spark_iris.py`:
stubs use the renamed field.

Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
2026-11-19 20:43:47 -05:00
Nicolas Perrin f0fbe27ec6 perf(dcp): merge and project K3.1's KVP decode partials in one CuTe DSL launch (#3645)
## Summary

A KVP decode's head-sharded combine packed every rank's partials into
records, exchanged them, merged them in another launch, and projected
the latent values in a third. For Kimi-K3.1's owned decode top-k, one
CuTe DSL launch (`DcpValue`) now does all three.

- `DcpValue` (`k3_decode/mla/dcp.py`) pushes each rank's BF16 partials
and LSEs to their head owners through Lamport mailboxes in symmetric
memory, merges them exactly as `combine_dcp_attention_partials` does and
projects them as `cute_dsl_mla_project_value` does, with the optional
sigmoid gate. Its output is bit for bit the records all-to-all, combine,
and projection.
- `create_dcp_value_tail` builds a group's tail collectively (SM100 or
SM103, 2, 4, or 8 ranks on one multicast fabric); every refusal is voted
on, so all ranks keep the unfused path together. Its mailbox stays under
2^31 elements (the merge's int32 offsets), and every in-kernel wait
traps at the group's timeout, as the LL exchange's do.
- `prepare_decode_exchange` prepares a tail for the attentions that
project as they merge (K3.1's sparse MLA). The sparse decode then hands
the value projection (`ValueProjection`) to
`combine_project_attention_partials`, which runs the tail when it serves
the call and the combine plus `mla_project_value` otherwise.

## Test Plan

- [x]
`tokenspeed-kernel/test/nvidia/ops/attention/test_cute_dsl_dcp_value.py`:
2 and 4 ranks, bit for bit against the records all-to-all, combine, and
projection over changing batch sizes, gated and ungated, negative zeros,
NaN and infinite LSEs, and CUDA graph replays with new inputs; voted
refusals (one ineligible rank, no timeout, an oversized mailbox) and a
trap within the timeout when a peer never calls.
- [x] `test/runtime/distributed/test_kimi_k31_dcp.py` (14 passed): with
the tail prepared, the owned decode selects and attends like one rank
holding the history, eager and replayed, and the tail runs.
- [x] Test files that import a touched module pass as on main.
- [x] Kimi-K3.1, TP8 with KVP8 on 2x GB300, DSpark with synthetic
acceptance 8 (mean of two fresh boots, ms/token):

| batch | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
|---|---|---|---|---|---|---|---|
| main | 1.368 | 1.678 | 2.238 | 2.950 | 4.406 | 5.730 | 7.931 |
| this PR | 1.360 | 1.677 | 2.221 | 2.872 | 4.305 | 5.607 | 7.713 |
| change | -0.6% | -0.0% | -0.7% | -2.6% | -2.3% | -2.2% | -2.7% |

- [x] GSM8K, 1000 questions with real acceptance: 960 with this PR, 956
on main (9 answers right only with this PR, 5 only on main; sign test p
= 0.42).

---------

Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
2026-11-19 20:43:35 -05:00
Xiangyi ZhangandEnwei Zhu 1983549040 perf(moe): run Kimi-K3 NVFP4 SiTU experts on FlashInfer's CuTe DSL MoE (#3588)
## Summary

Ports 40f0605d0 from #2661 onto current `main`.

Adds `--moe-backend flashinfer_cutedsl` for Kimi-K3 NVFP4 SiTU routed
experts: FlashInfer's CuTe DSL fused MoE on precomputed top-k. `auto`
still selects TRT-LLM-Gen. The kernel does not take expert parallelism,
so it serves EP1 layouts such as the DWDP prefill configuration, where
peer weights stream the experts a worker does not hold.

The preprocessor converts NVFP4 checkpoint experts to the CuTe DSL
layout: W13 rows `[gate | up]` become `[up | gate]` interleaved in
64-row groups, and the linear E4M3 block scales are swizzled per expert.
Storage stays expert-major, so DWDP streams these experts like the
TRT-LLM-Gen layout. The kernel quantizes the GEMM2 input with one global
scale, so the preprocessor rejects checkpoints whose per-expert GEMM2
input scales differ (Kimi-K3-NVFP4 ships 1.0 everywhere). One fixed
tactic compiles each kernel once for any row count.

On GB300 at K3 prefill shapes (896 experts, top-16), #2661 measured the
routed MoE layer 5-6% faster than TRT-LLM-Gen at 16K rows and 11% faster
at 24K rows beside a peer weight copy, and GSM8K strict 96.66% on a K3
PD deployment with this backend for prefill (TRT-LLM-Gen on the same
deployment: 96.89-97.19%).

## Changes

- `ops/moe/flashinfer_cutedsl.py`: the weight preprocessor and
`flashinfer_cutedsl_nvfp4_situ_routed_moe_apply`, NVIDIA SM100-SM103
only: its fixed tactic is in the Blackwell format, and SM107 takes
separate FlashInfer kernels.
- `latent.py`: `Kimi3MoEExecutionPlan.use_cutedsl`, an explicit field
selected by the backend outside MegaMoE, native and Marlin.
- `kimi_k3.py`: K3 MoE accepts the backend, keeps BF16 input for the
kernel to quantize, and configures the latent projection from its
processed input scale.
- `expert.py`: the peer-weight check admits CuTe DSL NVFP4 SiTU.
- `ci_path_filter.py`: `flashinfer_cutedsl` is an NVIDIA-only solution.
- Docs: the kernel README covers the layout and limits; `parallelism.md`
lists the backend for DWDP.

**Adaptations for `main`:** the plan keeps `main`'s `decode_chain`
beside `use_cutedsl`, and the newer plan constructions in the K3 tests
set `use_cutedsl=False`. The preprocessor reads the loader's `[E, 2]`
gate/up `weight_scale_2` directly. #3524 adds another plan construction
in `test_kimi_k3_expert_placement.py`; whichever of the two lands second
sets `use_cutedsl=False` there.

## Tests

- `test_nvfp4_situ_flashinfer_cutedsl.py`:
- the preprocessor layout matches FlashInfer's
`prepare_cute_dsl_weights` byte for byte;
- the apply matches an FP32 SiTU MoE over the dequantized weights,
within 1.5x of TRT-LLM-Gen's error on the same checkpoint, and an NVFP4
input pair gives the BF16-input result;
  - new row counts trigger no CuTe DSL compile;
  - per-expert GEMM2 input scales are rejected;
  - the plan selects CuTe DSL only when asked.
- `test_latent_moe.py`: a forced `flashinfer_cutedsl` backend yields a
CuTe DSL-only plan.

GB300 with FlashInfer 0.7.4: the 6 kernel tests pass; the runtime K3 /
latent MoE tests and the CI path filter tests pass, apart from failures
that `main` shows in the same environment.

Not yet run end to end: no K3 serving run with `--moe-backend
flashinfer_cutedsl` on this branch.

---------

Signed-off-by: Xiangyi Zhang <xiangyiz@nvidia.com>
Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
2026-11-20 09:36:25 +08:00
943bef04fb fix(memory): scope expandable segments around memory saver operations (#3640)
Enabling expandable CUDA segments together with memory saver can fail
initialization or bypass the allocator that tracks managed storage. This
change temporarily disables expandable segments during managed weight/KV
and persistent-buffer allocations, CUDA graph capture, and pause/resume.
It restores the effective allocator configuration and both environment
aliases afterward, including on exceptions and when the last
configuration string omits the active expandable setting.

The allocator setting is process-wide. The adapter serializes its own
scopes, and callers must keep unrelated CUDA allocation threads idle
during startup or drained maintenance.

Validation:
- `pre-commit run --all-files` passed.
- H200 with PyTorch 2.9.0 and `torch_memory_saver` 0.0.10: 19 passed, 1
skipped. The skipped case uses `PYTORCH_ALLOC_CONF`, which does not
enable expandable segments in the tested Torch build.
- GPU tests cover expandable allocations outside managed scopes,
weight/KV/buffer isolation, graph replay, three pause/resume cycles,
exception and sticky-setting restoration, and normal process exit.

---------

Signed-off-by: lyzuu <349859374+lyzuu@users.noreply.github.com>
Co-authored-by: lyzuu <349859374+lyzuu@users.noreply.github.com>
Co-authored-by: longcat-ia-team <312376714+longcat-ia-team@users.noreply.github.com>
2026-11-19 14:55:22 -08:00
Nicolas Perrin fcf48aea6c refactor(models): share the DeepSeek-layout NextN draft loader (#3648)
## Summary

The DeepSeek NextN draft (`deepseek_nextn.py`) and the GLM-MoE-DSA NextN
draft (`glm_moe_dsa_nextn.py`, which `DeepseekV32ForCausalLMNextN`
reuses) each carried a copy of the same code:

- the NextN model's `__init__`;
- the checkpoint-name mapping for the NextN layer;
- the `load_weights` loop: stacked MLP, MoE experts, the fused q/kv
a-projection pair, and plain tensors;
- the q/kv a-projection pairing state and its
`begin`/`abort`/`end_weight_update` hooks.

They now share it:

- `DeepseekModelNextN` builds its decoder from a `decoder_cls` class
attribute. `GlmMoeDsaModelNextN` subclasses it, sets `decoder_cls =
GlmMoeDsaDecoderLayer`, and keeps its own `forward`.
- A new `DeepseekNextNWeights` mixin in `deepseek_nextn.py` holds the
name mapping, the loader loop and the three update hooks. Both draft
classes list it first.
- Per-model behavior stays in each class:
- **DeepSeek** fails on a parameter name it doesn't know. **GLM** looks
it up with `get_param` and skips it if missing. A small `_nextn_param`
hook keeps both behaviors.
- **GLM's `load_weights`** loads its indexer weights as they stream by
and passes every other tensor on to the shared loop, in the same order
as before.
- Each model keeps its own `forward`, LM head,
`checkpoint_weight_name_filter` and `post_load_weights`.

**One behavior change:** a checkpoint name with a non-numeric layer
index, such as `model.layers.x.mlp.gate.weight`, used to make the
DeepSeek draft raise on `int()`. It is now skipped, as the GLM draft
already did. Well-formed names map the same way as before.

No model forward code changes. 172 fewer lines.

## Test Plan

On GB200, main (588408510) against this branch.

- [x] **Same weights after loading.** Tiny real DeepSeek-V3, GLM-MoE-DSA
and DeepSeek-V3.2 NextN drafts were built on one GPU, and each loaded a
synthetic checkpoint in a shuffled order. The checkpoint covered:
- the NextN layer's attention, with q_a/kv_a pairs, norms, the MoE gate
and correction bias, each expert, and the shared experts;
  - the GLM indexer weights;
- names the loader skips: target layers, the LM head, embeddings, rotary
`inv_freq`, an extra bias, and an unknown name for GLM.

  The drafts then went through three more steps:
- a weight-update session whose q_a and kv_a arrive in separate
`load_weights` calls;
  - a session that never delivers kv_a;
  - two malformed names.

After each step, every parameter, buffer and tensor attribute was
hashed, along with the errors and the pending pairs: 365 entries in all.
All match main except the two malformed-name results above. Two runs of
main give identical results.
- [x] **Same structure.** The three drafts have the same parameter and
buffer names, shapes and dtypes, and the same module types, as on main.
- [x] **Same test results.** 34 test files covering these models, from
DeepSeek V3/V4.1, GLM and dots3 to the NextN shard filter and
weight-update sessions, give identical per-file results: 1146 passed and
51 skipped on both sides. 47 tests fail on both sides, in
`test_kimi_k31`, `test_qwen35_nextn_quantization` and
`test_mla_lora_scale`, the same as on main.
- [x] **Every module imports.** All modules under
`tokenspeed.runtime.layers` and `tokenspeed.runtime.models` import: 149
of 149.
- [x] **New test.** `test/runtime/test_nextn_draft_weights.py` builds
real DeepSeek and GLM drafts and checks that:
- the NextN layer lands on the draft and its decoder, and a target
layer's tensor is skipped;
- GLM's indexer `wk` and `weights_proj` fill the fused projection when
they arrive mid-stream;
  - a GLM q_a/kv_a pair split across update chunks is fused.

  It passes on main too.
- [x] pre-commit on the changed files.

Signed-off-by: Nicolas <nperrin.ai@gmail.com>
2026-11-19 14:26:12 -05:00
zhuhaozhecooland7huahua 3786263e1b refactor(npu): rework Ascend attention metadata, Prefill workspaces and MoE execution contracts (#3618)
## Scope

Refactor Ascend runtime attention, cache workspace preparation and MoE
execution contracts. This consolidates #3498, #3573 and #3596 on top of
#3563.

- Separate DSA decode resources from shared context-parallel metadata,
and reuse the metadata lifecycle across DSA and MLA.
- Share recipe-sized QCP Prefill workspace preparation and transport
across attention backends, with explicit layout and capacity contracts.
- Centralize scheduler-derived MoE source counts and refresh the
graph-owned buffer before replay.
- Clarify W4A4 weight preparation and common-registry ownership, while
retaining FP32 normalization scales for BF16 inputs.

Model plugins select their numerical execution paths through these
runtime interfaces. Sampling and logprob reduction retain the existing
runtime policy.

## Review fixes

- Reject DP-local decode graphs for models requiring global MoE source
counts; use the communication layer's scatter calculation. Startup
profiling uploads the selected packed tier's row count and restores
capture metadata, including on failure.
- Keep persistent CUDA count buffers in the memory saver's CPU-backed
runtime region. Cover repeated release/resume, contents and addresses.
Register the packed profiling regression in CUDA CI.
- Pass cache-recipe workspace layouts through the shared QCP Prefill
preparation path, removing the NPU package's reverse dependency. Restore
transport input validation and cover dynamic metadata parameters without
recompilation.
- Make W4A4 layout choices explicit, update the common registry
regression, and test fused DSA operator rejection at binding time.
Restore KVP 2/4/8/16 LSE coverage, exact FP32-scale sensitivity, invalid
FlatQuant clip checks and both cache-probe sizing cases.
- Update the scheduler test fixture for the cache-storage contract
reported by NVIDIA and AMD CI, and refresh the renamed NPU metadata test
entry point.

The existing DSA constructor's optional decode flag remains compatible
with callers that previously supplied only `prefill_enabled`. Low-level
packing helpers retain their separately tested geometry; the model
support constraints remain unchanged.

## Validation

CPU contract tests passed: 413 runtime/kernel tests and a follow-up
selection of 126 runner tests (with overlap). `pre-commit run
--all-files` passes.

Ascend 910B: 297 metadata graph/stream regressions, 25 transport-kernel
cases, 10 normalization cases and 8 real INT4 NZ/GMM cases passed. The
two-rank QCP transport run passed 34 byte-exact scenarios per rank,
covering KV-only and KV+index layouts, empty owners, shared pages and
fragmented history. The two scheduler fixture regressions also passed
with the matching engine package.

CUDA packed profiling regressions passed on NVIDIA B200 and AMD MI350.
The attention contract follow-up passed 160 CPU tests and 25 subtests;
`pre-commit run --all-files` passes.

---------

Signed-off-by: zhuhaozhecool <zhuhaozhecool@163.com>
Signed-off-by: 7huahua <26763681+7huahua@users.noreply.github.com>
Co-authored-by: 7huahua <26763681+7huahua@users.noreply.github.com>
2026-11-20 01:49:56 +08:00
Nicolas Perrin a33c5a1e9a perf(muse): overlap owned-row gathers with their projections on the copy engine (#3644)
## Summary

At >= 4096 tokens, Muse Spark on NVIDIA keeps the residual stream
row-sharded, and each layer gathered its owned rows over NCCL twice
before a column projection: the attention input before the Q, K, and V
projection, and the pre-expert input before the pre-expert projection.
At 8192 rows each gather moved 128 megabytes and ran before its GEMM
could start.

The gathers now run on the copy engine and overlap the GEMMs:

- `_CopyEngineRows` keeps every rank's owned BF16 rows in symmetric
memory, in two parities. A gather writes this rank's rows, passes a
barrier, and pulls the peers' rows on a side stream.
- `_NcclOwnedRows.all_gather_mm` projects this rank's rows while the
peers' rows are in flight, then projects each peer's rows as they land.
The SMs stay on the GEMMs.
- Layers pass owned attention rows to the next layer, and layer zero
projects gathered owned rows too. The Iris path is unchanged.
- `prepare_communication_runtime` sizes the workspace from the runtime's
maximum forward tokens and binds it before cache sizing and graph
capture.

Each rank's rows go through the same `mm` as before. At these shapes
(4096 to 16384 rows; 4608 and 1024 columns) the per-rank products are
bitwise equal to one product over all rows, so outputs match the NCCL
gather followed by one projection.

Isolated on 4 GB200s at 8192 rows: the Q, K, and V gather plus GEMM went
from 610 to 452 microseconds, and the pre-expert one from 315 to 197
microseconds.

## Results

4 GB200s, TP4 and EP4, NVFP4, random prompts with a distinct seed per
run; two A versus B pairs on one node.

Prefill only (2 output tokens), mean TTFT in milliseconds:

| prompt x concurrency | main | this PR |
|---|---|---|
| 1k x 1 | 58 and 57 | 57 and 59 |
| 4k x 1 | 127 and 124 | 120 and 120 |
| 8k x 1 | 222 and 217 | 210 and 205 |
| 8k x 16 | 1714 and 1724 | 1577 and 1577 |
| 1k x 128 | 1660 and 1665 | 1523 and 1527 |

1k input and 1k output tokens, output tokens per second and mean TTFT in
milliseconds:

| concurrency | main | this PR |
|---|---|---|
| 64 | 3619, 932 and 3638, 934 | 3648, 864 and 3632, 906 |
| 128 | 5256, 1699 and 5228, 1798 | 5341, 1589 and 5327, 1589 |
| 256 | 7741, 3258 and 7783, 3238 | 7876, 2989 and 7923, 2967 |

GSM8K, 512 questions at concurrency 32: main 0.979 and 0.977, this PR
0.975 and 0.969 (one standard deviation is about 0.007).

## Tests

- `test/runtime/distributed/test_muse_owned_rows_nccl.py`: on 2 and 4
GPUs, `all_gather_mm` matches the NCCL gather followed by `mm` for odd
and even row counts, and a captured pair of gathers replays with new
rows.
- `test/runtime/models/test_muse_spark.py`, `test_muse_spark_iris.py`:
model stubs declare the new field.

---------

Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
2026-11-19 11:38:32 -05:00
Nicolas Perrin 1df4f4ba28 refactor(attention): pass extend_with_prefix to the MLA prefill hook (#3614)
## Summary

Follow-up to #3605. FlashMLA's prefill hook read `extend_with_prefix`
from a field that only its `init_forward_metadata` wrapper set. A direct
call to the hook, as the dots3 tests make on other leaves, therefore
reused the previous forward's value. A stale `True` on a prefix-free
batch planned paged prefill instead of ragged, with no error.

`extend_with_prefix` is now an argument of the shared
`_init_prefill_metadata` hook:

- `PagedMLABackend.init_forward_metadata` passes it to the hook.
- `FlashMLABackend` reads it from the argument. Its wrapper and the
`_extend_with_prefix` field are removed.
- The MLA, CuTe DSL MLA and TRT-LLM MLA hooks don't use it. They `del`
it with a one-line reason.
- `Dots3NoteSWABackend` passes it on to the MLA hook. Shifting the
prefix into the window keeps a prefixed request prefixed, so the flag
stays true.
- `AscendMLABackend` passes the hook's keywords on when its own prefill
is off. When it is on, its plan reads its own host table and the prefix
lengths, so it now drops `page_table_cpu` and `extend_with_prefix`
before building `MLAPrefillMetadata`, which has neither field. That path
already raised `TypeError` on `page_table_cpu` before this PR.

Callers that invoke the hook directly now pass the flag:
- the dots3 cache tests;
- `run_cutedsl_mla_dcp.py`;
- the two NPU callers, `test_ascend_mla_prefill.py` and
`run_ascend_mla_lifecycle.py`. These also pass their host page table,
which they had been missing, so they failed on main too.

Server forwards get the same flag value as before. No kernel code
changes.

## Test Plan

On GB200, main (1aa75cbc3) against this branch.

- [x] **Same metadata.** The MLA, FlashMLA, CuTe DSL MLA, TRT-LLM MLA
and dots3 window leaves each ran as target and as draft, through these
calls:
  - extend with no prefix, with a prefix, and over several chunks;
  - mixed and idle batches;
  - the decode-mode and query-shard errors;
  - `override_num_extends`.

All 1238 hashes of prefill, chunked-prefill and decode metadata tensors
and leaf tensors match main.
- [x] **Same host time.** Main's entry and hooks were compiled from its
source as subclasses of each leaf. Both versions ran in one process,
alternating call by call, 1000 calls each on extend with no prefix,
extend with a prefix, and a mixed batch. Every median is within 1% of
main (0.990x to 1.004x), and both versions leave identical metadata.
- [x] **Same test results.** 35 runtime test files covering these leaves
pass the same on both sides: 983 passed and 18 skipped on main; 984
passed and 18 skipped here, the extra one being the new test. No
failures.
- [x] **Every module imports.** All modules under
`tokenspeed.runtime.layers` and `tokenspeed.runtime.models` import: 149
of 149.
- [x] **New test.**
`test_flashmla_prefill_plans_from_the_flag_it_is_given` calls FlashMLA's
hook with a prefixed batch and then a prefix-free one. It checks that
each plans from the flag it is given: paged, then ragged.
- [x] **`run_cutedsl_mla_dcp.py --kvp-size 2`** on 4 GPUs, run one stage
at a time. In the prefill stage, the edited hook call and the chunk
checks after it pass. The stage then stops at a later `AttributeError`
(`sparse_topk`), the same as on main. The FP8 decode stage fails on main
too, with the same mismatch: 18 of 24576 elements at the same index.
Neither path changes here.
- [x] pre-commit on the changed files.
- [ ] The two NPU callers and Ascend's prefill-enabled path need an NPU.
They are left to Ascend CI.

---------

Signed-off-by: Nicolas <nperrin.ai@gmail.com>
2026-11-19 11:19:32 -05:00
WangBo 71f4039605 feat(metrics): add request-level speculative accept length (#3641)
## Summary

- Add `tokenspeed:request_spec_decode_accept_length`, with one mean
acceptance-length sample per ended speculative request, including bonus
tokens.
- Drain request samples through the Rust control plane and mirror them
in the Python and Prometheus reporters.
- Keep the existing verify-slot-weighted `spec_decode_accept_length`
metric unchanged and document the distinction.

## Tests

- `pre-commit run --all-files`
- Python metric reporter tests covering request MAL, speculation
counters, and weighted observations: 3 passed.
- Python syntax checks for the modified collector, reporter tests, and
exposition generator.

Rust tests were not run locally because no Rust toolchain is available
in this environment.

Signed-off-by: wangbo981016 <wangbo981016@gmail.com>
2026-11-19 23:12:09 +08:00
FC.Li 415a87aee3 fix: prepare Iris all-reduce before short prefills (#3638)
The [AMD GPT-OSS GPQA
job](https://github.com/lightseekorg/tokenspeed/actions/runs/44522750854/job/135581030558)
exits when a short prefix-cache hit first selects Iris all-reduce.
Startup's large batches use RCCL, leaving ordinary Iris state
unprepared; initializing it during serving synchronizes the device and
trips the data-plane sync guard.

Prepare ordinary reductions after model-specific communication setup and
before KV memory profiling. Execution groups follow the same role order
on every rank so overlapping groups retain symmetric heap offsets, and
preparation reuses existing model capacities. A registered AMD
regression covers large-to-small batches under sync-debug error mode,
repeated preparation, and graph replay. The communication test fixture
now uses the real backend constructor.

Also install `prometheus-client` for the [Rust engine CI metrics
tests](https://github.com/lightseekorg/tokenspeed/actions/runs/44516690947/job/135534552347)
and document that test dependency.

The distributed GPU tests now bind NCCL to each worker's device before
symmetric-memory rendezvous. EAGLE4 uses `spawn`'s exception transport
directly, removing a Manager that forked after Torch started threads.
The MegaMoE graph test also prints live rank stacks after three minutes
and has a ten-minute command limit; this exposed a repeated stall in
PyTorch's symmetric-memory rendezvous.

Validation:

- Related local tests: 377 passed; AMD-only tests require the AMD CI
runners.
- EAGLE4 TP2 test: three independent local runs passed after removing
the Manager.
- `cargo test --locked`: 642 passed for the unchanged native engine
code.
- `pre-commit run --all-files`: passed.
- Targeted AMD GPT-OSS GPQA: 594 samples, 69.4% (original threshold
69%), no data-plane sync failures. NVIDIA GPT-OSS also passed.
- Distributed GPU regression, the remaining post-submit-only model
cases, and PR CI are running on the updated branch.

---------

Signed-off-by: FC-Li <robert_lexis@163.com>
2026-11-19 23:09:44 +08:00
Tuan Zhangandlightseek-bot 23d32440b5 feat(cache): pack FlatKV per Qwen4-Exp pipeline stage (#3564)
## Summary

Stacked on
[#3389](https://github.com/lightseekorg/tokenspeed/pull/3389). Pack one
FlatKV plane per Qwen4-Exp pipeline stage. Compute each cache group's
packing from the stages that own its fields. The stages share logical
block IDs, and the last stage owns the draft fields.

For Qwen3.8-Flash-Next at TP2 with BF16 KV, FP32 GDN state, and MTP, the
PP2 plan uses 28 MiB per parent per GPU on both stages. Stage 0 packs
128 PLE blocks per parent. With a 150 GiB cache budget per GPU and the
int32 slot limit applied, the plan admits 4,681 parents at PP2 and 2,372
at PP1. End-to-end PP2 serving still requires Qwen4-Exp model forward
support.

## Test Plan

- Runtime regression suite with 351 passed and 7 skipped, covering stage
ownership, arena addressing, and target and draft int32 slot limits.
- PP2 planning with the Qwen3.8-Flash-Next checkpoint configuration and
PD wire and address checks on both stages.
- Fresh B300 TP2 and PP1 serving with MTP, forward state checkpoints,
and L2 `write_back`. The short request and both long requests returned
the expected answer. The repeated prompt reused 17,920 of 18,029 tokens.
- `pre-commit run --all-files`

---------

Signed-off-by: tuanzhangCS <tuan@lightseek.org>
Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
2026-11-19 20:19:00 +08:00
FC.Li 6103eb3953 fix(deps): require tokenspeed-mooncake 0.3.13.post20261119 (#3637)
## Summary

TokenSpeed still accepts Mooncake releases whose native C Store library
and Python Store binding abort on duplicate gflags registration when
loaded together. Require `tokenspeed-mooncake>=0.3.13.post20261119`,
which shares one Store core between the bindings, and update the cu129
install constraint to the same version.

Document the shared core required for native L3 clients and Python Store
clients to coexist. The release is available on
[PyPI](https://pypi.org/project/tokenspeed-mooncake/0.3.13.post20261119/)
and both [cu129](https://lightseek.org/whl/cu129/tokenspeed-mooncake/)
and [cu130](https://lightseek.org/whl/cu130/tokenspeed-mooncake/) wheel
indexes.

## Test Plan

- Downloaded the published cu129 and cu130 CPython 3.11 x86_64 wheels
and verified their SHA-256 hashes against the indexes.
- Installed each variant into a separate environment. Both C-first and
Python-first loading passed bidirectional TCP batch reads/writes and
three buffer registration/client teardown cycles; the surviving client
remained healthy after its peer closed.
- Dependency checks passed for both installed variants.
- `pre-commit run --all-files` passed.

Signed-off-by: FC-Li <robert_lexis@163.com>
2026-11-19 19:38:30 +08:00
WangBo 5a8b323430 feat: support SM90 peer expert weights and breakable prefill graph replay (#3574)
DWDP currently requires fabric handles and rejects model CUDA graphs.
Add explicit node-local POSIX FD transport on SM90 and newer GPUs, and
keep peer-weight prefetch, acquire, and release as eager breaks around
captured MoE compute so staging advances on every prefill replay.

The copy worker accepts CUDA 12.8 and CUDA 13 batch-copy signatures. MoE
kernels can declare that their weight preprocessor supplies the existing
peer-weight manifest.

Validation:
- `pre-commit run --all-files`.
- 71 focused CPU tests for ownership, FD exchange and cleanup, mapping
and CLI validation.
- Two H200 GPUs: each rank passed 2 synthetic graph-replay cases (3 and
4 layers, full VMM pages, changing inputs, exact comparison) and 4 POSIX
storage and cleanup cases.
- The graph regression test fails with the original manager and passes
with this change.

---------

Signed-off-by: wangbo981016 <wangbo981016@gmail.com>
2026-11-19 19:36:08 +08:00
Xiangyi ZhangandEnwei Zhu 32d0de3de8 feat(kimi-k3): serve EP sizes that do not divide the experts (#3524)
## Summary

Ports 2cb578c08 from #2661 onto current `main`, with the adaptations
`main` now needs.

Kimi-K3's 896 routed experts do not divide over EP20. This lets K3 run
static expert placement with attention DP and EP-only MoE: with
`--ep-num-redundant-experts R` the routed experts hold 896 + R slots,
the router maps each logical expert to this rank's static replica, and
the loader fills every placed slot. The placement does not depend on how
routes are carried: MegaMoE, the FlashInfer all-to-all, or the AG/RS
dispatch to the local MoE. EP20 runs with `--ep-num-redundant-experts 4
--ep-dispatch-algorithm static` (900 slots, 45 per rank). DeepEP and
attention TP refuse a placement at startup.

The EAGLE3 MLA drafter is one dense layer whose config inherits the
target's expert count. Its loader no longer builds a routed-expert plan,
which failed at EP sizes that do not divide 896.

## Changes

- `kimi_k3.py`: `KimiLinearMoE` sizes its experts from the placement and
routes through its static dispatch map. The FlashInfer all-to-all
partitions ranks by physical slots, which is what the router emits.
`_check_k3_expert_placement` refuses a placement outside attention DP +
EP-only MoE, with weight sharing, with load recording or EPLB, or with a
non-static dispatch algorithm. `KimiK3ForConditionalGeneration` opts in
and reports its text-model geometry.
- `deepseek_v3.py`: `has_routed_experts` lets the EAGLE3 drafter skip
the routed-expert loader plan.
- `expert_location.py`: a model class outside `BaseCausalLM` can opt in
with the same class attribute; host-side dispatch tables are built on
CPU explicitly.
- **Adaptation for `main`:** #2853 builds the static dispatch map with
the placement, outside the weight region a memory-saver discard
releases, but only when an all-to-all backend is set. Attention-DP ranks
route only their own tokens even with `--all2all-backend none` (MegaMoE,
or K3's automatic FlashInfer / AG/RS choice), so
`build_expert_placement` now also builds it under attention DP.
- Docs: `server.md` and `parallelism.md` list K3 (and Inkling2) among
the models that take a placement.

## Tests

- `test/runtime/test_kimi_k3_expert_placement.py`: the K3 MoE layer
builds 900 slots and this rank's static dispatch map under MegaMoE,
FlashInfer all-to-all (which gets the 900-slot count) and AG/RS; the
router writes the physical slot of the chosen expert, including
replicas.
- `test/runtime/layers/test_moe_loader_placement.py`: over EP20, every
slot holds the logical expert placed in it: each expert exactly once,
experts 0..3 also in rank 19's replica slots. Loading into replica slots
is covered by the existing placed-loader tests.
- `test/runtime/test_expert_location.py`: the dispatch-map test gains an
attention-DP case with no all-to-all backend.

GB300, single GPU: 846 passed, 7 skipped (AMD only) across the
expert-placement and K3 MoE test files. Four `test_kimi_k3_config.py`
tests fail on `main` itself in this environment (missing
`tokenspeed_engine.scheduler`). Reverting each change (MegaMoE-only
check, logical count to the all-to-all, logical count to the MoE layer,
no attention-DP map) fails a test.

These tests check construction and routing only: the routed experts are
a stand-in, so no MoE kernel runs at 900 slots / 45 per rank, and no
multi-rank forward checks that the router passes the placement on. Not
yet run end to end at EP20.

---------

Signed-off-by: Xiangyi Zhang <xiangyiz@nvidia.com>
Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
2026-11-19 19:35:28 +08:00
a2be7880e2 feat: metrics for peer prefix sharing (#3623)
<!-- Headings and layout are optional. Include change context and
validation
(checks performed or planned, or why testing was not run or is not
applicable)
anywhere in the description. -->

## Summary
<!-- What changed and why? -->
- Add peer_consumed_tokens and peer_estimation_error_tokens to measure
actual peer-prefix reuse per request and the router’s estimate minus
actual reuse, grouped by outcome.
- Add peer_stage_seconds to measure latency and count completed visits
by protocol stage and outcome. Add peer_hard_pin_remaining_seconds to
record how much lease time remains when a hard pin is released or
expires.
- Add peer_shard_seconds to measure device-fence wait, READ queue wait,
and transfer time per reader rank and holder. Add peer_read_bytes_total
for successful READ payloads and peer_outstanding_read_bytes for bytes
queued or in flight.
- Add peer_scheduler_state to track active leases, pinned blocks, and
requests in lookup or landing. Add peer_scheduler_events_total to count
landed tokens and failed landings.

## Test Plan

<!-- How was this validated? -->
- [x] Ran cargo fmt --all --check, cargo clippy --all-targets --locked
-- -D warnings, and cargo test --locked across the Rust workspace.
- [x] Verified successful reuse, lookup misses, incompatible holders,
and lookup timeouts with engine metrics enabled and transfer metrics
disabled. Checked Prometheus output for token attribution, prediction
error, READ bytes, and outstanding bytes after completion and
cancellation.
- [x] Ran all 15 C++ peer-export tests, including hard-pin release and
expiry.

---------

Signed-off-by: lvyongkang <kangguangli@hotmail.com>
Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>
Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
2026-11-19 19:35:10 +08:00
WangBoandlongcat-ia-team 0e39ed17af fix(dspark): keep the native query extent at every capture forward (#3589)
## Summary

- Restore `ctx.input_num_tokens` at the top of every decode-graph
capture run, so the grammar fill records a non-empty verify-candidate
copy in shortened graphs.
- Pin the contract with a regression test over the sequence a capture
performs (four warmups and the capture over one context).

Packed verification narrows `ctx.input_num_tokens` for the rest of the
forward that selected it, because sampling and the NaN audit consume its
packed target rows. A capture runs four warmups and the capture itself
over a single context, so every run after the first started from the
narrowed extent.

The grammar fill slices this round's verify candidates out of the live
input buffer by that extent (`ModelExecutor._forward_step`), so the
captured copy read an empty slice. Every replay of a shortened graph
then filled its masks by walking the draft tokens a previous forward had
left in the pinned buffer: the first row still followed the request's
matcher, the rows behind it were reset to allow every token, and
verification accepted drafts the grammar forbids.

## Tests

- `pre-commit run --all-files`
- Mutation check: dropping the reset fails the new test with `assert
[12, 8, 8, 8, 8] == [12, 12, 12, 12, 12]` — the first run keeps the
native extent and every later one inherits the narrowed one.
- `test/runtime/test_adaptive_dspark_contract.py
test/runtime/test_capturable_grammar_host.py
test/runtime/test_cudagraph_probe_ladder.py
test/runtime/test_cudagraph_memory_projection.py
test/runtime/test_prefill_graph.py
test/runtime/test_qsa_verify_lifecycle.py
test/runtime/sampling/test_packed_dspark_sampling.py
test/runtime/test_slot_state.py`: **258 passed**.
- DeepSeek-V4.1-Flash 1P1D on 8xB300 (prefill data parallel 4 and expert
parallel 4, decode data parallel 4 and expert parallel 4 with sampled
adaptive DSpark and DP-local CUDA graphs, xgrammar,
`--dsv41-dspark-min-batch-size 4` so shortening engages at low
concurrency). Before: greedy `json_schema` returned malformed JSON in
24-29 of 32 concurrent requests, and 128-per-case thinking requests
validated in 25-75% of cases. After: `json_schema` and `json_object`
validate in **128 out of 128 requests per case** at T=1.0 and T=0, with
`budget decisions` reporting shortened verification on 64-81% of steps.
Rejected completions replayed through a fresh matcher confirm the first
forbidden token sat behind an otherwise valid prefix, consistent with
masks filled from stale drafts.
- Unrelated to grammar, 340 greedy prompts (GSM8K, AIME, 15-65K-token
code contexts) show the same divergence profile as a run without
shortening: every first divergence is a near-tie and none is confident
on both sides.

Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>

---------

Signed-off-by: wangbo981016 <wangbo981016@gmail.com>
2026-11-19 01:59:18 -08:00
minedecandlightseek-bot 697e842eb1 feat(qwen4-exp): use per-tensor FP8 for online PLE quantization (#3575)
## Summary

When `ple_embed_dtype` is `float8_e4m3fn`, Qwen4 now quantizes each BF16
PLE table with one BF16 scale shared across all rows and ranks. This
matches the scalar-scale layout of offline FP8 PLE checkpoints and
removes the per-row scale buffer from online quantization.

For a split BF16 table, each rank scans its own checkpoint rows. A max
reduction across the table's ranks gives `scale = BF16(max(abs(W)) /
448)`. The regular weight loader then quantizes each shard with that
scale. The prescan supports standard single-file and indexed safetensors
checkpoints for both target and MTP models.

## Test Plan

- On B300, all 11 focused PLE tests passed on the final code. Coverage
included target and MTP loading, scale recomputation on reload, all-zero
tables, offline FP8 weights, host and device lookup agreement, and
errors for unsupported layouts or a missing PLE table.
- Before the final review fixes, 60 tests and 5 subtests passed across
PLE quantization, disk weight updates, and MTP checkpoint filtering.

---------

Signed-off-by: minedec <jiandong@lightseek.org>
Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
2026-11-19 17:49:29 +08:00
WangBo c937127aa8 fix(pd): start the decode first token at the first local output (#3627)
## Summary

The decode role used the remote KV landing as `first_token`, so a
request that already had its KV but waited for a decode execution permit
(the E-queue with `--disaggregation-decode-extra-slots`) counted that
wait as decode time. `decode_tps` and TPOT therefore mixed admission
queueing into the decode rate, even though the first output is streamed
only at the first local decode commit.

This change starts the decode window at the role's first local decode
output:

- `first_token` is stamped at the first local decode commit; the KV
landing only stamps `prefill_done`.
- Requests ended by the bootstrap token (no local decode) stamp it at
their terminal stream.
- `ttft_ms` and `tokenspeed:time_to_first_token_seconds` now include the
permit wait; `decode_tps`, `time_per_output_token_seconds`, and the OTLP
`tpot_ms` exclude it.
- `decode_wait_ms` remains the KV landing to first decode dispatch
interval and is documented as a sub-interval of TTFT.
- Updated the metric help text, the server configuration, tracing,
event-loop, and scheduler docs, and the metrics exposition fixture.

## Tests

- `cargo test -p tokenspeed-control --lib` (203 passed)
- `cargo test -p tokenspeed-control --test request_logs`
- `cargo test -p tokenspeed-metrics`
- `pre-commit run --all-files`

---------

Signed-off-by: wangbo981016 <wangbo981016@gmail.com>
2026-11-19 01:42:07 -08:00
Enwei Zhu b6221f7448 fix(moe): migrate FlashInfer all-to-all to the moe_ep API (#3634)
<!-- Headings and layout are optional. Include change context and
validation
(checks performed or planned, or why testing was not run or is not
applicable)
anywhere in the description. -->

## Summary

Use the NVLink one-sided communication backend in FlashInfer 0.7.4,
which owns the workspace and the dispatch and combine state. Remove the
obsolete weights_dtype argument and construct host-side layouts under a
CPU default device.

Adapted from 626c9421e1.

## Test Plan

<!-- How was this validated? -->

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
2026-11-19 01:39:36 -08:00
a6dc79402b feat(scheduler): let the decode role hold peer prefixes and a Host tier (RFC 0005 phase 1) (#3550)
Decode replicas retain both a session's prompt and the reply they
generated. Letting them serve peer prefixes allows the next turn's
prefill replica to reuse that reply. This adds RFC 0005 and implements
its phase 1.

- Allow decode replicas to hold peer prefixes and use Host write-back.
Decode replicas ignore holder hints and continue receiving missing
prompt data from their prefill peers.
- Probe Device history followed by whole-page Host suffixes. Load the
local Host hit below the PD landing; reuse State checkpoints only at the
exact prompt end on Device.
- Return decode history probes explicitly and consume the current
`HostSuffixProbe` interface without output parameters or copying the
Device hit arrays.
- Fence PD receives and peer reads behind page preparation and Host
loads using separate stream events. Wait after launching the current
forward so its layerwise load overlap is preserved.
- Bind the peer endpoint's listener up front and move it into
`Peer::start`, so the first member owns the bound port continuously
through startup, including an OS-selected port, while the replica's name
and scheduler are prepared.
- Retain bounded concurrent peer reads and pin validation from the
merged peer-sharing implementation; update comments, design
documentation, and regression coverage.

Validation:

- Full scheduler C++ suite: 1,596 passed; 1 AddressSanitizer-only test
skipped in the Release build.
- Rust scheduler, control, and host test suites passed.
- Device-handle, peer-sharing, and rank-setup Python tests: 186 passed.
- Restoring the old full-load barrier makes both concurrent-forward
regression cases fail (PD and peer); the corrected implementation
passes.
- `pre-commit run --all-files`.

Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>

---------

Signed-off-by: FC-Li <robert_lexis@163.com>
Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>
Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
2026-11-19 16:48:36 +08:00
Hanyu Weiandlongcat-ia-team 1ff9d0ba97 fix: preserve host preloads and tokenizer specials (#3631)
Nested allocator contexts can repeat an inherited LD_PRELOAD entry.
Deduplicate the child environment while retaining the first occurrence
and library order, leaving the parent environment untouched. Extend the
existing real-child-process test to cover inherited and repeated
entries. Main already captures the environment inside the startup
context; this change preserves that behavior.

Special-token reconciliation now passes the append flag positionally,
avoiding a dependency on its renamed keyword across tokenizer versions.
A real PreTrainedTokenizerFast regression covers preserving existing
specials and vocabulary IDs, repeat calls, and decoding with special
tokens removed.

Validation:
- Based on main 401d29b693. A three-way
merge check against current main
6a5b9947e4 has no content conflicts;
upstream changes do not overlap the four changed files. git diff --check
passes. Tests below were run on the original base.
- Tokenizer tests: 3 passed with Transformers 4.55.0. A separate-process
comparison using the unwrapped upstream tokenizer method reproduces
TypeError on the old keyword before the fix, then passes all 3 tests
after the fix. The pinned Transformers 5.12.0 combination has not been
tested here.
- Host test collection is blocked because the matching
tokenspeed_engine.scheduler module is unavailable in this environment;
no host-test pass is claimed for the current main.
- pre-commit run --all-files was attempted and failed while provisioning
the markdownlint Node environment (nodeenv version parsing error).
Supplemental black-jupyter --all-files passed. All four changed Python
files parse successfully.

The current-main host tests and complete hooks still need to pass in a
matching development environment before merge.

Signed-off-by: Patirck <151747616+Patirck@users.noreply.github.com>
Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>
2026-11-19 00:30:26 -08:00
Enwei Zhu c24bc5545f feat(dwdp): share MXFP4 SiTU expert weights across prefill ranks (#3626)
## Summary

Publish processed MXFP4 expert weights, scales and biases through the
peer-weight manifest while replicating activation parameters. Allow
MXFP4 SiTU experts on the FlashInfer TRT-LLM backend to use DWDP.

Previously the FlashInfer TRT-LLM MXFP4 preprocessor published no
peer-weight manifest and the runtime peer-weight gate accepted only
NVFP4, so MXFP4 experts could not be shared across prefill ranks. The
preprocessor now returns a `PreparedExpertWeights` manifest that streams
the processed weights, scales and biases while replicating the SiTU
activation parameters (`gemm1_alpha`, `gemm1_beta`,
`gemm1_clamp_limit`), and the gate admits MXFP4 with the SiTU
activation. MXFP4 experts store no input scale; the kernel quantizes
their activations on each forward. Serving MXFP4 with DWDP requires
`--moe-backend flashinfer_trtllm`, since `auto` may select another
kernel. The DWDP design and serving docs are updated to match.

Adapted from abd6a11099.

## Test Plan

Tests added or extended in this PR:

- `test/runtime/distributed/test_dwdp_moe.py` parametrizes the
owned-loading and full-expert-compute check over NVFP4 and MXFP4
(launched with torchrun on fabric-connected GPUs).
-
`tokenspeed-kernel/test/nvidia/ops/moe/test_mxfp4_situ_flashinfer_trtllm.py`
adds `test_situ_owner_shards_rebuild_resident_weights`, verifying that
shards preprocessed by separate owners serve identically to resident
weights, with and without precomputed routing.

---------

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
2026-11-19 00:28:12 -08:00
dingyuqing05andlongcat-ia-team af43d67c56 fix(metrics): align request, cache, and transfer observations (#3621)
## Summary
- Distinguish active and resident cache occupancy, scheduler waits, and
prefill-inflight requests.
- Preserve independent completion-token, transfer-size, per-output TPOT,
and request e2e_TPOT samples.
- Complete transfer-failed requests through the common metric path and
exclude cancellation propagation from transfer failures.

## Results
- Output-chunk timing excludes the first chunk and empty outputs;
single-chunk requests do not create timing samples.
- Weighted observations retain correct totals while individual
observations preserve their distributions.
- Multi-rank cancellation and repeated failure notifications no longer
inflate transfer-failure counts.

## Test plan
- pre-commit run --all-files
- cargo fmt --all --check (tokenspeed-engine)
- 62 output tests, 101 executed PD tests, and 69 serving tests passed;
14 PD tests required an unavailable test-library configuration.
- Validated 21 successful requests and one intentional stream
cancellation, including single-token boundaries and drained request
gauges.

---------

Signed-off-by: dingyuqing05 <13126694060@163.com>
Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>
2026-11-19 16:26:17 +08:00
sjqgogogogo 6a5b9947e4 fix(sampling): key chain MTP randomness by token position (#3490)
## Summary

FlashInfer chain verification used per-slot generators that advanced
during prefill preparation. Under `--sampling-stream per-request`,
FlashInfer and FlashInfer Full now derive accept and final-draw coins
from the request seed and the first target position in the verify round.
The two kinds of draws use distinct counters.

Sampled draft proposals use the accepted frontier after the round and
the draft step. The verifier retains its target-only and recorded-q
rejection rules, and uses the same fixed-order residual scan with
replicated and batch-DP logits layouts under per-request sampling.

## Test Plan

- `pre-commit run --all-files` passed.
- On H200, 64 sampler and kernel tests passed and one existing test was
skipped. This covers extra prefill preparation, changed slots and mixed
batches, CUDA graph replay, sampler snapshot restore, and recorded-q
verification.
- Real chain MTP passed ten-bin distribution calibration for 10,000
committed tokens each with greedy draft proposals and with sampled
proposals using recorded q. Both configurations accepted draft tokens.
- Per-request DP layout equivalence remains unverified. End-to-end
`auto` runs already showed first-token logprob differences before
speculative verification. The sampler tests compare runs with matching
numerical inputs.

---------

Signed-off-by: sjqgogogogo <sjqgogogogo@gmail.com>
2026-11-19 13:29:31 +08:00
lipanpan03andFC-Li abb3ce0b5a fix(runtime): offload persistent metadata with weight regions (#3620)
## Summary

Releasing weights can leave ordinary CUDA allocator segments resident
because small persistent runtime tensors were allocated outside the
memory saver's regions. A packed cache table can keep an entire 2 MiB
segment alive through a backend's shared-storage view.

Allocate these tensors in the existing CPU-backed persistent-buffer pool
so weights release unmaps their physical storage and weights resume
restores their original addresses and contents:

| Allocation owner | Change |
| --- | --- |
| Sampling and executor inputs | Scope sampler construction and the
executor sentinel. |
| Attention and cache metadata | Scope arena geometry, capture
placeholders, target/draft metadata, and per-forward packed cache
tables. Live forwards and prefill capture share the managed table
factory. |
| Host transfers | Scope immutable geometry and workspace tables at
initialization and when later submissions grow them. |
| Graph capture | Register generator seed/offset state before entering
discardable graph capture; centralize capture barriers whose WORLD
communicator is initialized before managed regions. |

The existing release/drain order, CPU backup policy, CUDA graph path,
NCCL backend, and worker lifecycle remain in place. Tests cover disabled
mode, workspace growth, and content/address recovery through explicit
metadata inputs. A fresh CUDA subprocess regression checks geometry,
real GPU table packing and capture metadata across table growth and
repeated sleep/wake cycles, retaining old views and requiring zero
ordinary-pool bytes after release plus restored addresses and contents.

## Test Plan

- Current review revision: `pre-commit run --all-files` passed.
- Current review revision: local collection of
`test_torch_memory_saver_adapter.py`, `test_host_executor.py`, and
`test_retraction_image_executor.py` is blocked by missing
`tokenspeed_triton` (three collection errors; no tests executed). The
native metadata test requires CUDA VMM hooks and a fresh subprocess; it
has not been run locally. Runtime/GPU CI is still required on this head.
- Historical validation before the main rebase and this review refactor:
the adapter suite passed **24 tests** in a CUDA runtime. A fresh
**32-GPU** serving/training smoke completed two iterations and a final
weights restore/release; all **128 inference offload observations** had
**0 bytes in ordinary allocator segments** (`segment_pool_id=(0, 0)`),
with unchanged inference worker PIDs. This measures the ordinary Torch
pool, not total process GPU memory.
- That historical smoke used non-default NCCL buffer/channel settings
and training-side checkpoint-group cleanup. It does not validate a
configuration with those settings and cleanup removed, or establish an
isolated performance effect for this patch.
- Historical native single-GPU controls covered barrier, graph RNG, and
Host geometry sleep/wake across three cycles. The geometry control did
not execute its payload transfer kernel. An isolated GPU packed-table
baseline script failed in Triton `loadBinary` before reaching the
candidate and is not counted as passed; the complete serving smoke
exercised the packing path.
- GPU decode step latency has **not** been compared before/after region
entry. No latency or throughput neutrality claim is made. A full GPU
smoke has not been repeated after rebasing or this review refactor.

---------

Signed-off-by: lipanpan03 <41904587+lipanpan03@users.noreply.github.com>
Signed-off-by: FC-Li <robert_lexis@163.com>
Co-authored-by: FC-Li <robert_lexis@163.com>
2026-11-19 12:08:17 +08:00
Enwei Zhu 9a19e45d65 feat(engine): report resident and pending prefill token load (#3572)
## Summary

Report per-rank token counts so SMG can use context size and pending
prefill work for load-aware routing:

- `num_resident_tokens`: prompt plus committed output tokens of every
request held by the rank.
- `num_waiting_uncached_tokens`: waiting prompts minus their last
observed cache hit, plus unscheduled prefill tokens. Before the first
admission probe, a prompt counts in full. Reporting adds no cache scans.

Append both optional fields to the existing ZMQ output schema, declare
them through the handshake, and preserve them in native client and
Python snapshots. Reports from older engines retain `None` for
unavailable counters.

Adapted from
[94b30e81](https://github.com/lightseekorg/tokenspeed/commit/94b30e81acebd55ec8db0c2c1654a81247c0e225)
and
[970f00f9](https://github.com/lightseekorg/tokenspeed/commit/970f00f9c51942305c33dd5ab6507b077f645b22)
for the current Rust engine.

Pairs with merged
[smg-project/smg#6996](https://github.com/smg-project/smg/pull/6996).
Older SMG versions ignore the appended fields; consuming them requires
upgrading the SMG pin and selecting the appropriate routing policy.

## Test Plan

- 275 Rust tests passed across wire and client, scheduler, control-unit,
and focused CPU serving tests.
- 29 Python wire-conformance tests passed.
- Verified cache-hit accounting, partial prefill, cancellation, and live
and idle load reports.
- Verified TokenSpeed’s actual handshake and output encoding against the
paired SMG decoder, including 64-bit counts, zero, and absence.
- Python binding compilation and `pre-commit run --all-files` passed.

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
2026-11-19 11:19:17 +08:00
Max191 1780127a77 feat(Kimi-K3): run each MoE layer's post-attention work as one gfx950 launch in single-request decode (#3613)
## Summary

This PR adds the "MoE layer tail": a persistent gfx950 Gluon kernel for
Kimi-K3 and Kimi-K3.1. It serves decode steps that hold one request,
including speculative verify steps. In such a step, it runs each MoE
layer's work after the attention core as one launch per rank:

- the attention o-projection, and its all-reduce across the 8 TP ranks
through each other's memory
- the MLP-side AttnRes mix and the post-attention RMSNorm
- the router (sigmoid plus correction bias, top-16 of 896), the latent
down projection and the latent RMSNorm
- the routed experts (MXFP4) and the shared experts
- the latent up projection and the layer output
- the next layer's attention input, when that layer mixes it from a
prepared block partial

The kernel is a persistent launch of 256 workgroups. Its stages hand
data to each other through tagged words in an uncached scratch buffer.
The ranks exchange partial sums through an uncached peer buffer that
every rank maps with HIP IPC. Each rank adds the 8 parts in rank order,
so every rank computes the same sums.

How the model uses it:

- The kernel is registered in a new `kimi_k3` family (ops `moe_tail`,
`moe_tail_prepare` and `moe_tail_pack`). Its API is
`tokenspeed_kernel.ops.model.kimi_k3`, and that module's README
describes it.
- While the model builds its layers, it asks the API which expert bank
layout the tail reads. When a tail serves the deployment, the MoE layers
store their routed experts once, in that layout (N16).
- Before graph capture, the model prepares the per-rank buffers in a
collective step. It then checks and packs every MoE layer and binds the
tail to all of them. If any layer fails the checks, no layer uses the
tail.
- A decode forward whose row count equals the verify width, that is a
step with exactly one request, runs each layer's tail. The decision uses
only the forward mode and the row count, so every rank runs the same
layers. Every other forward uses the regular path.
- The regular gfx950 MXFP4 SiTU kernels (decode and both prefill stages)
read N16 banks, so one copy of the experts serves both paths. Their
outputs are bit-identical to the default gdot128 layout: in decode, and
in prefill above 2,048 rows. At or below 2,048 rows, the prefill adds
routes with BF16 atomics, so its output varies from run to run in either
layout. N16 banks are about 19 GB per GPU smaller than gdot128 at
Kimi-K3.1 sizes, and the runtime gives that memory to the KV cache.
- The projection-plus-partials GEMV in `ops/gemm/gluon.py` now accepts
Kimi-K3.1's 3808-row decode projection, so the attention input
projection can also produce the next layer's AttnRes block partials.

Numerics: the tail quantizes the routed experts' activations to MXFP4
(A4W4), where the regular decode MoE uses FP8 (A8W4). A step that the
tail serves therefore does not give the same values as the regular path.
A request's decode numerics depend on whether other requests share its
steps. GSM8K accuracy is unchanged within noise (below).

Limits:

- gfx950 with 256 to 1024 compute units.
- Kimi-K3 sizes at TP8, with one TP group for attention and MoE on one
node (HIP IPC), and 1 to 8 rows, fixed per server.
- The model keeps the regular path in these cases: attention DP, EP or
PP, deterministic collectives, unfused norms, a prefill-only role,
quantized or split attention projections, non-BF16 router, latent or
shared-expert weights, and other expert formats or sizes.
- To turn the tail off, set
`TOKENSPEED_KERNEL_OVERRIDE_KIMI_K3_MOE_TAIL=none`.
- The larger KV pool also grows a host KV tier sized by
`--kvstore-ratio`, by about 2.1× at ratio 8. Set `--kvstore-size` to
keep the previous host-tier size, for example 107 GB per rank at ratio
8.
- Every wait in the kernel has a 60 s deadline. A rank that stalls past
it inside a decode step aborts the server with a GPU memory fault. One
example is a first-time compile with CUDA graphs off.

## Benchmarks and accuracy

Kimi-K3.1 on 8x MI355X (gfx950), TP8, one request in flight, DSpark
speculative decoding (7 draft steps, 8 rows per verify step), greedy
sampling. Serving numbers compare the base with this PR; decode and
GSM8K were measured on one build of this PR, prefill and AgentX on
another. Those builds' tail code compiles to the same instructions as
this branch.

Decode verify step, median client step gap:

| Workload | Regular path | Tail | Change |
|---|---|---|---|
| 50k-token prompts, real acceptance (base vs this PR, 4 interleaved
starts) | 14.66 ms | 13.89 ms | −5.2% (95% CI −5.3% to −5.1%) |
| 200k-token prompts, AgentX flags (one build, tail off vs on) | 15.24
ms | 14.40 ms | −5.5% (95% CI −5.7% to −5.3%) |

With the 50k-token prompts, accepted tokens per step were 3.6% lower
(95% CI −7.0% to −0.5%), and the per-request output rate showed no gain.

InferenceX AgentX, concurrency 1, 30 minutes per build, synthetic
acceptance 4.16 (not quality evidence), one run per build, means:

| Metric | Base | This PR | Change |
|---|---|---|---|
| Output throughput per GPU | 16.36 tokens per second | 16.74 tokens per
second | +2.4% |
| Time per output token | 3.67 ms | 3.47 ms | −5.5% |
| Time to first token | 1.65 s | 1.57 s | −5.0% |
| End-to-end latency | 6.69 s | 6.41 s | −4.2% |

Prefill, time to first token with 1 output token, mean of 20 requests
per arm (2 interleaved starts each):

| Input tokens | Base | This PR | Change |
|---|---|---|---|
| 2,048 | 230.5 ms | 233.9 ms | +1.5% |
| 16,384 | 819.7 ms | 823.4 ms | +0.4% |

GSM8K, first 256 test questions, greedy, up to 2,048 output tokens, real
acceptance:

| Metric | Base | This PR | Change |
|---|---|---|---|
| Accuracy | 243 of 256 (94.9%) | 242 of 256 (94.5%) | −0.4 points (95%
CI −2.4 to +1.6) |
| Decode output rate per request, mean | 404.5 tokens per second | 419.9
tokens per second | +3.8% (95% CI +1.8% to +5.8%) |

Tests on this branch:

- pre-commit (all files) and ruff with undefined-name checks pass.
- CPU model runtime tests: 1,131 passed, 26 failed. The same 26 fail on
the base.
- `test_kernel_api_selection.py`, on one GPU: 499 passed, 122 skipped, 0
failed.
- The gfx950 MoE tests that build the changed prefill stage, including
10 N16-versus-gdot128 bitwise cases: 390 passed.
- The 8-GPU tail test (widths 1, 4 and 8, a block-write case, two steps
across the step-count wrap): 4 of 4 passed.
- The off switch, checked on CPU: no tail layout with it set, `n16`
without it.
- The broader gfx950 kernel tests for the changed MoE, GEMV and shared
extern code ran on an earlier commit with the same tail, GEMV and extern
code. They gave 1,295 passed, 0 failed and 75 skipped, and the 13
skipped torchrun cases passed on all 8 ranks.

---------

Signed-off-by: Max Dawkins <max.dawkins@gmail.com>
2026-11-18 17:22:20 -08:00
Nicolas Perrin 1aa75cbc31 refactor(attention): share the MLA leaves' forward-metadata entry (#3605)
## Summary

Each of the four paged MLA leaves carried its own copy of
`init_forward_metadata` and `override_num_extends`:

- `MLAAttnBackend` (`paged/mla.py`)
- `FlashMLABackend` (`paged/flashmla.py`)
- `CuteDSLMLABackend` (`paged/tokenspeed_mla.py`)
- `TRTLLMMLABackend` (`paged/trtllm_mla.py`)

Every entry took the same four steps:
1. reject a query shard;
2. refuse a decode forward;
3. build prefill metadata for the leading extend requests;
4. refresh decode metadata for a target's mixed or idle batch.

The four overrides were near-identical. Both methods now live in a new
`PagedMLABackend` in `paged/base.py`, and each leaf keeps only its
`_init_prefill_metadata`. `Dots3NoteSWABackend`, `AscendMLABackend` and
the DSA wrapper reach the shared entry through the leaves they extend or
wrap.

Where the copies differed:
- **FlashMLA** is the only leaf that reads `extend_with_prefix`, which
picks ragged or paged prefill. It records the flag in a small
`init_forward_metadata` wrapper before calling the shared entry. This
keeps the hook's arguments the same for every leaf, including the
subclasses and the tests that call it directly.
- **TRT-LLM MLA**'s hook now receives `page_table_cpu` and discards it,
as its entry did before.
- **Error messages** now name the leaf class:
- the decode-forward error reads `TRTLLMMLABackend decode metadata goes
through ...`; it used to say `trtllm_mla`;
- the query-shard error from `Dots3NoteSWABackend` and
`AscendMLABackend` names the subclass instead of `MLAAttnBackend`.

No kernel code changes.

## Test Plan

On GB200, main (5b26453b0) against this branch, with the engine and
scheduler built at that commit.

- [x] **Same metadata.** Each leaf, as target and as draft, ran the same
set of calls:
- extend without a prefix, with a prefix, and with a prefix long enough
for several chunk loops;
  - mixed batches with one and with two decode requests;
  - idle;
  - the decode-mode and query-shard errors;
  - `override_num_extends`, including an exception raised inside it.

After each call, every tensor in the prefill, chunked-prefill and decode
metadata was hashed, along with every tensor attribute of the leaf. That
gave 1238 hashes over the four leaves and the dots3 window leaf, and all
match main. The only differences are the two query-shard messages of the
dots3 window leaf, described above. Two runs of main give identical
hashes.
- [x] **Same host time.** Main's entry was rebuilt from its source as a
subclass of each leaf. Both versions ran in one process and alternated
call by call, 1000 calls each, on an extend with prefixes and on a mixed
batch. Every median is within 0.4% of main (0.998x to 1.004x).
- [x] **Same test results.** 35 runtime test files cover these leaves:
router and cache setup, MLA, FlashMLA, CuTe DSL and TRT-LLM MLA
metadata, block decode, tree verify, DSA selection, dots3, Kimi K3,
DFlash and the drafters. Both sides give 983 passed and 18 skipped, with
no failures.
- [x] **Every module imports.** All modules under
`tokenspeed.runtime.layers` and `tokenspeed.runtime.models` import on
both sides: 149 of 149.
- [x] **New test.** `test/runtime/test_mla_forward_metadata_entry.py`
builds real MLA, FlashMLA and TRT-LLM MLA leaves (12 cases) and checks
that:
  - a target's mixed batch refreshes its decode requests;
  - a draft's mixed batch leaves decode metadata to its wrapper;
  - `override_num_extends` restores the count after an exception;
  - a decode forward is refused.

  It passes on main too.
- [x] pre-commit on the changed files.

Signed-off-by: Nicolas <nperrin.ai@gmail.com>
2026-11-18 18:01:52 -05:00
Nicolas Perrin aa03b0275a perf(dcp): exchange small KVP decode payloads in one NVLink hop (#3586)
## Summary

KVP decode runs a few small collectives per DSA layer: the query head
gather, the candidate gather and the partial exchange. Across eight
ranks, NCCL's ring all-gather takes about 19 us for a few dozen KiB.

- A CuTe DSL one-hop exchange over symmetric memory (`ll_exchange.py`)
serves all-gathers and equal-split all-to-alls up to a prepared per-rank
capacity. Every 16 payload bytes travel as one 32-byte store of four
(word, epoch) pairs, and the receiver polls until each pair carries the
call's epoch, so no flag or fence is needed. Two parity slots let
consecutive calls proceed. All-gathers store once through the multicast
address. Waits are bounded by the distributed timeout and trap past it.
- `prepare_small_exchange` arms it per group after a vote, so every rank
either uses it or keeps its collectives. Engines with a KVP group arm 1
MiB at startup.
- `AutoBackend` sends fitting `all_gather_single`, `all_to_all_single`
and `all_gather` calls to it. Larger, unaligned and uneven calls keep
their current path, and the NCCL-only switch disables it.

## Test Plan

- [x]
`tokenspeed-kernel/test/nvidia/ops/communication/test_ll_exchange.py`: 3
passed on four GB300 GPUs. It covers byte-exact all-gather and
all-to-all against NCCL from 16 B to 1 MiB, graph replays with fresh
inputs, the refusal vote and argument errors.
- [x] `test/runtime/distributed/test_small_exchange.py`: 2 and 4 ranks
passed. It covers routing, NCCL fallbacks, inner-dim gathers and graph
replay.
- [x] Kimi-K3.1, TP8 with KVP8 on 2x GB300, DSpark with synthetic
acceptance 8. Each value is the mean of two fresh boots, in ms/token.
From batch 32, every exchange exceeds the 1 MiB capacity and keeps NCCL.

| batch | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
|---|---|---|---|---|---|---|---|
| main | 1.577 | 1.897 | 2.440 | 3.145 | 4.577 | 6.042 | 8.171 |
| this PR | 1.412 | 1.800 | 2.342 | 3.070 | 4.497 | 6.091 | 8.162 |
| change | -10.5% | -5.1% | -4.0% | -2.4% | -1.8% | +0.8% | -0.1% |

- [x] GSM8K, 200 questions with real acceptance: 196/200.

---------

Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
2026-11-18 18:00:44 -05:00
LightSeek Foundation deab5d7f2e chore(deps): update the tokenspeed-smg pin to 1.16.0.post20261118 (#3608)
Consume the latest published `tokenspeed-smg` gateway release
(`1.16.0.post20261118`).

The `python/pyproject.toml` pins move from the `20261117` snapshot to
the `20261118` snapshot:

- `tokenspeed-smg==1.16.0.post20261118`
- `tokenspeed-smg-grpc-proto==0.5.0.post20261118`
- `tokenspeed-smg-grpc-servicer==0.16.0.post20261118`

---------

Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
2026-11-18 14:31:54 -08:00
Nicolas Perrin 58f7374c07 cleanup(runtime): remove unreferenced layer and model code (#3597)
## Summary

This PR removes runtime code that nothing in the repository calls or
reads. Each name below appears only at its own definition: a whole-repo
`git grep -w` finds no caller, attribute access or string reference in
`python/`, `tokenspeed-kernel/`, `test/` or the configs.

- `layers/rotary_embedding.py`: `MRotaryEmbedding.get_rope_index_glm4v`
(196 lines), and `itertools`, which only it used.
- `layers/layernorm.py`: `FusedRMSNorm.forward_with_allgather_fusion`,
and the `allgather_dual_rmsnorm` import, which only it used.
`distributed/comm_ops.py` keeps its own import.
- `layers/vocab_parallel_embedding.py`:
  - `VocabParallelEmbedding.get_sharded_to_full_mapping`
  - `ParallelLMHead.tie_weights`
  - `VocabParallelEmbeddingShardIndices.num_added_vocab_padding`
- `models/muse_spark.py`: `RESIDUAL_GATE_TEMPERATURE`. The layer reads
`config.residual_gate_temperature`.
- `models/llama_eagle3.py`: `LlamaForCausalLMEagle3.get_embed`, and the
never-read `_eagle3_prefix`.

None of the removed code runs today, so results and speed cannot change.
Left in place on purpose:
- `set_embed_and_head` overrides, which the draft factory calls
generically;
- public `tokenspeed_kernel` functions.

## Test Plan

On GB200, main (8214efc97) against this branch, with the engine and
scheduler built at that commit.

- [x] Every module under `tokenspeed.runtime.layers` and
`tokenspeed.runtime.models` imports: 149 of 149 on both sides.
- [x] The runtime tests that cover these classes give the same result on
both sides: 497 passed and 17 skipped, with no failures. The files are:
- layers: rope scaling, YaRN ramp, vocab-parallel embedding, RMSNorm
kernel, Gemma RMSNorm, sandwich norm;
  - models: Llama EAGLE3, Muse Spark, Inkling2;
- decode TP layout args, DeepSeek V4.1 model, DFlash2, bitwise kernel
providers, DP LM-head argmax.
- [x] pre-commit on the changed files.

Signed-off-by: Nicolas <nperrin.ai@gmail.com>
2026-11-18 14:16:12 -05:00
Nicolas Perrin 8214efc977 perf(startup): freeze the collector once around every graph capture (#3555)
## Summary

Prefill and drafter graph captures (`BreakableCapture`) collect garbage
on entry, because a collection inside a capture would run graph
destructors on a capturing stream; decode captures (`torch.cuda.graph`)
collect only under `force_cudagraph_gc`. Only the decode ladder ran with
the heap frozen (`freeze_gc`). So the prefill buckets and the drafter's
prefill and extend graphs each ran a full collection over the startup
heap: 150-200 ms apiece, about 150 per start.

- `ModelExecutor.capture_graphs` now holds one frozen heap around every
capture: decode, prefill and drafter graphs.
- A boot that captures nothing (enforce-eager) skips the freeze and its
two collections, as on main.
- `--enable-cudagraph-gc` still keeps the heap unfrozen.
- The decode runner no longer freezes on its own. Nested scopes would
break, because an inner `gc.unfreeze()` thaws the outer scope.

## Results

Qwen3.8-Flash-Next-NVFP4, TP1, MTP, 56 prefill buckets plus the drafter
graphs. One GPU, warm compile caches, fresh server per run:

| | main | this PR |
|---|---|---|
| full collections in the scheduler process | 148, 24.0 s | 148, 3.4 s |
| server ready | 204.6, 207.6, 209.6 s (mean 207.3) | 187.5, 190.5 s
(mean 189.0) |

Both arms capture the same graphs with the same memory:

| | both arms |
|---|---|
| KV cache | 182.05 GB |
| activations + graphs | 3.98 GB |
| device free | 5.63 GB |

One consequence: an object that existed before capture and becomes
cyclic garbage during a capture is now freed after the last capture
instead of between capture groups. On this model's boot with the memory
probe enabled, allocated memory is byte-identical at every one of the
5714 captures and the probe series are identical.

Serving is unchanged:

- First 64k-token prefill: 0.88 s; later ones: 0.655 s.
- The same kernels compile on the first requests.
- No collection runs while serving.

## Test Plan

- `test_dispatch_autotune.py` runs `ModelExecutor.capture_graphs` with
its capture doubles:
- `test_capture_freezes_the_heap_once_around_every_capture`: the decode,
prefill and drafter captures see a frozen heap unless
`--enable-cudagraph-gc`; one `gc.freeze` spans them all; a boot that
captures nothing collects nothing.
- `test_a_failed_capture_leaves_the_heap_unfrozen`: a capture failing at
any position leaves the heap as it found it.
- Each of these mutations makes them fail: no `try`/`finally` in
`freeze_gc`, one freeze per capture group, a decode-only freeze, no
early return, `or` in its predicate, the flag ignored, no freeze.
- `pre-commit run --all-files` passes.
- The 11 test files that use `capture_graphs`, the decode runner's
capture or `freeze_gc` fail the same tests on this branch and on main
(28 FlashInfer BF16 adapter tests in `test_dispatch_autotune.py`, and
`test_draft_final_step_follows_the_complete_drafter_run`, which fails on
main since #3534).

---------

Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
2026-11-18 11:08:38 -05:00
dingyuqing05andlongcat-ia-team 401d29b693 fix(pd): propagate hybrid cache readiness to state backends (#3563)
Summary
- Register the hybrid producer step counter on the linear-attention
child.
- Let state backends publish staged prefill fields before the wrapper
signals readiness, without recording the same producer step twice.
- Document the final-write and single-barrier contract for hybrid
producers.

Results
- The rebased source passes 21 focused producer-readiness,
destination-validation and FP8 state-transfer regression cases with real
GPU backends.
- Before the base update, an eight-GPU disaggregated comparison passed
18 requests and matched all 523 output token IDs and FP32 logprob bit
patterns. It covered concurrency, chunked prefill and cached prefixes,
with native layerwise sends.

Test plan
- Passed: pre-commit run --all-files.
- Passed: cargo fmt --all --check in tokenspeed-engine, with
/opt/cargo/bin on PATH and RUSTUP_HOME=/opt/rustup.
- Passed: 21 focused integration regression cases against the rebased
source.
- Full-model comparison was run before the base update, not repeated
afterward.

---------

Signed-off-by: dingyuqing05 <13126694060@163.com>
Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>
2026-11-18 20:05:06 +08:00
Tuan Zhang 25872a5ca0 feat(cache): enable FlatKV by default for Qwen4-Exp (#3389)
## Summary

Reduce padding in Qwen4-Exp caches by packing target and draft KV, GDN,
PLE and QSA into one physical plane. FlatKV is enabled by default.
Qwen3.8-Flash-Next uses five logical groups: full attention, one GDN
state group, PLE, compressed QSA and recent QSA. Each parent remains
owned by one group.

- Translate full-history KV and QSA page IDs and write locations into
atom addresses; retain explicit strides for state and recent-window
caches.
- Select stride-capable GDN kernels, including MTP and tree
verification, using the actual input and output layouts.
- Bound parent capacity by every target and draft field's int32 atom
page and write slot limit, retaining startup overflow checks. Derive L3
namespaces from the actual layout, and preserve existing DeepSeek and
Dots byte-page layouts.

Use `--no-qwen4-exp-flat-kv` for the ordinary layout. FlatKV currently
requires PP=1, uniform per-view K and V heads and non-MXFP8 KV.

## Measurements

NVIDIA Qwen3.8-Flash-Next-NVFP4 on B300, TP1, BF16 KV, FP32 GDN, MTP 3
steps and 4 verify tokens, forward checkpoints enabled and L2
`write_back` (192 GB).

| Metric | FlatKV disabled | FlatKV enabled |
| --- | ---: | ---: |
| GDN checkpoint allocation | 2 × 78 MiB | 110.5 MiB |
| GDN checkpoint padding | 29.42% | 0.35% |
| API L1+L2 hit rate | 86.90% | **88.81%** |
| Output tokens per second | 1078.40 | **1157.56** |
| TTFT P50 | 8.24 s | 6.47 s |
| Mean ITL | 22.68 ms | 22.29 ms |
| Snapshot reservation | 23.56 GB | 37.08 GB |

InferenceX AgentX: concurrency 64, 256K context, seed 42, 702 warmup
requests and a 900-second measurement window per configuration. Hit rate
is cached prompt tokens divided by prompt tokens for completed profiling
requests; both runs had zero request errors.

These are historical runs from separate rounds with different GPUs and
peer workloads. They used FlashInfer 0.7.3 with common compatibility
adapters; current kernel validation uses pinned 0.7.4. Useful KV payload
is unchanged. The larger parent also increases null-parent and
small-group allocation costs, so total memory use depends on the live
group mix.

<details>
<summary>GDN grouping comparison</summary>

| Metric | 1 group (selected) | 6 groups | 18 groups |
| --- | ---: | ---: | ---: |
| Total cache groups | 5 | 10 | 22 |
| Parent size | 110.5 MiB | 19.5 MiB | 6.5 MiB |
| API L1+L2 hit rate | **88.81%** | 88.22% | 88.05% |
| Output tokens per second | 1157.56 | 1189.93 | 1199.22 |
| TTFT P50 | 6.47 s | **4.31 s** | 5.58 s |
| TTFT P90 | **32.44 s** | 34.63 s | 33.09 s |
| Mean ITL | **22.29 ms** | 23.34 ms | 26.57 ms |
| Snapshot reservation | 37.08 GB | 13.09 GB | 9.60 GB |

One group prioritizes checkpoint packing and the observed hit rate, ITL
and P90 TTFT. More groups reduce snapshot reservation and improve
throughput. Each configuration was measured once; the results show this
workload's trade-offs.

</details>

Accuracy comparison: main `cdeb32aa7` vs PR `fb4cc01f9`, using the same
serving configuration and complete test splits.

| Evaluation | Main | PR | PR-only repeat |
| --- | ---: | ---: | ---: |
| GSM8K, 1K output budget | 1276 of 1319 (96.74%) | 1274 of 1319
(96.59%) | — |
| GSM8K, 4K output budget | 1287 of 1319 (97.57%) | 1283 of 1319
(97.27%) | 1287 of 1319 (97.57%) |
| AIME26, 64K output budget | 28 of 30 (93.33%) | 28 of 30 (93.33%) | —
|

GSM8K used 4-shot greedy decoding with thinking disabled; AIME26 used
zero-shot thinking, temperature 1.0, top-p 0.95 and top-k 20. Both used
EvalScope 1.11.1, batch size 16 and seed 42. All requests succeeded.
GSM8K scores varied between PR runs; the repeat matched main's aggregate
score with different per-question results. The cause of this variation
has not been isolated. AIME26 answer extractions matched on all 30
questions, including the same truncated output.

## Test plan

- **444 runtime and cache tests passed** at `4692d6cf6` (7 skipped),
plus 7 subtests. Includes metadata-only capacity boundaries, differing
target and draft payloads and existing layout regressions. Inkling
activation parity also passes across prefill and page-crossing decode.
- **279 GDN and fused decode-metadata kernel tests passed** (36
skipped), including 17 far-page cases beyond 2^31 elements, independent
state strides, padding guards and CUDA Graph execution.
- Real Qwen4 target and draft D2H → L3 → H2D round trips compare every
arena byte, including padding and untouched blocks.
- Fresh B300 TP2 service at 0.95 memory utilization, with forward
checkpoints and L2 `write_back`: capacity safely capped at **2,372
parents and 137.48 GB per GPU**. All three requests returned the
expected answer; a repeated 18,029-token prompt reused **17,920
tokens**.
- `pre-commit run --all-files` passed. Hardware CI status is tracked by
the checks on this PR.

---------

Signed-off-by: tuanzhangCS <tuan@lightseek.org>
2026-11-18 17:19:05 +08:00
3bd08baacc feat: share cached prefixes between engine replicas (peer prefix sharing) (#3531)
## Summary

A replica can reuse a prefix that another replica already caches, in
that replica's GPU or pinned Host cache, by reading it over RDMA instead
of recomputing it. The design is in
`docs/rfcs/0004-peer-prefix-sharing.md`.

There is no central store and no L3. The router names a holder with the
request (`peer_holder`, the holder's `host:port`). The reader then:

1. **Looks up.** It asks the holder how much of the prompt it caches.
The lookup soft-pins those blocks.
2. **Pins.** At admission it hard-pins exactly the pages it will read.
3. **Reads.** It pulls them with one-sided Mooncake Transfer Engine
READs into its own GPU blocks. Each member of the replica reads its own
shard from the holder's member of the same index.
4. **Releases.** If the read was complete, the release asks the holder
to demote its copy, so the two replicas don't both keep it.

Any failure falls back to computing locally. The landing is given up
whole and the request is admitted again from its local hit.

Opt in with `--peer-prefix-port PORT`; attention-DP rank `r` binds `PORT
+ r`. Peer sharing requires the Host tier (`--enable-kvstore`) and
prefix caching. It is refused with `--disaggregation-mode decode`,
`--kvstore-storage-backend`, pipeline stages and query context
parallelism.

## What changes

- **Scheduler (C++).**
- The holder answers lookups, pins and releases from const reads between
rounds. The changes those answers imply are applied by every replica
member at the next round's head.
- The reader gains a `PeerLooking` state, and a `PeerLanding` state that
generalizes the PD decode landing.
- The landing's pages are kept out of the plan's copies, loads and
zeroing.
- Leases are protected by deadlines, incarnations and namespace checks.
- **Control loop (Rust, `ts-control::peer`).**
- The servicer, the per-request landing state machine, read geometry for
Device and Host sources, and the endpoint over ZMQ.
- The loop holds the scheduler under a mutex, released while it waits on
a forward result, an idle frontend or a paused round's sleep, and handed
once per round to a waiting servicer, so the endpoint can answer without
waiting for a round. That last hand-over matters to a loop waiting on a
transfer (a prefill engine's PD sends, a landing's reads), which reaches
neither wait and spins through empty rounds.
  - Peer reports ride the replica relay.
- **Engine host and runtime.**
- Startup: a transfer engine per rank with the Device arena and Host
pool registered, and the replica's geometries exchanged over the rank
mesh.
- Read fence: a plan with landings records an event behind its stores,
copies, zeroing and the forwards in flight. Reads start only after that
event. Every plan with a landing goes to the data plane, also when it
has no other device work, as a prefill engine's landing may not.
- The namespace is the L3 deployment id (`cache_deployment_id`, now
shared by L3 and peer sharing) plus the weight version. A flushed weight
update moves it, under the same weight-version rule L3 already enforces.
- **KV events.** With Host L2, the event stream now also covers Host
blocks, published with their medium in vLLM's layout. Device events gain
trailing `nil` fields (`lora_id`, `medium`), which vLLM-format consumers
already read. Each medium's stream is prefix-closed for a router that
continues a GPU match into the Host tier.

## Testing

- **C++.** The scheduler suite passes, including new peer tests run on
both full-history and hybrid (sliding window + State) layouts. The
closed-loop random test now checks each KV event medium's stream and its
closure rule.
- **Rust.** `ts-control`, `ts-pd`, `ts-mooncake`, `ts-scheduler` and
`ts-host` pass, including in-process two-replica tests over real
sockets. `cargo fmt` and `clippy` are clean.
- **Python.** Tests for the geometry, server args, the device fence, the
L3 factory refactor and the KV event layout pass.
- **GPU.** Two replicas of Qwen3-0.6B on one node, with RDMA loopback
through the NIC, using a locally built transfer engine:
- From the holder's GPU cache: the reader landed 1536 prompt tokens and
computed 1, and its greedy output was token-identical to the holder's.
- From the holder's Host cache (write-back with a small GPU cache): same
result.
  - A failed READ fell back to local compute with correct output.
- **Prefill and decode, end to end through the Dynamo router** (with the
router-side change on top): this found two bugs, fixed in the last two
commits.
- **A servicer starved by a spinning loop.** A loop waiting on a
transfer spun without releasing the scheduler, so its servicer never
answered. A test runs a loop waiting on a PD transfer and checks that
another thread gets the scheduler.
- **A landing that never read.** `plan_needs_device` ignored peer
fetches, so a prefill engine's landing plan, with no other device work,
never reached the data plane and its reads never started. A test lands a
prefix on a prefill replica from such a plan.

  Both tests fail without their fix.
- Some runtime test files need a newer GPU stack than the machine this
was developed on, so CI covers those.

## Not in this PR

- Router-side placement: peer credit with same-node and cross-node cost
models, and the hint itself. That lands separately in the Dynamo router.
- KV-page sharding, the decode role, and pipeline stages.

Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>

---------

Signed-off-by: FC-Li <robert_lexis@163.com>
Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>
Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com>
2026-11-18 16:07:10 +08:00
WangBo e43d3bf6a2 feat(dspark): support grammar and logprobs in adaptive verify (#3539)
## Summary

- Gather adaptive-verification grammar masks into the same CSR-packed
target-logit layout used by DP-local CUDA graphs.
- Copy grammar verify candidates to pinned storage on the main stream
before the fork event, so the side-stream host fill reads pre-packing
tokens even when packed verification rewrites `input_ids_buf` in place.
- Map flat accepted-path indices through packed CSR rows and emit
selected-token output logprobs for greedy and sampled FlashInfer
verification and sampled prefill.
- Allow xGrammar and output logprobs at startup; keep cumulative
probabilities explicitly unsupported and document the packed-layout
contract.

## Tests

- `pre-commit run --all-files`
- Packed greedy and FlashInfer sampling tests: **38 passed**.
- Adaptive layout and startup tests: **21 passed, 49 deselected**.
- DeepSeek-V4.1-Flash on 8×B300 with TP1, DP8 and EP8, MegaMoE and
DP-local CUDA graphs: 32 concurrent JSON-schema requests returned exact
JSON with a finite logprob for every completion token.
- Fresh restart with the committed source, sampled JSON and narrative
validation: **32 of 32 requests passed**. Adaptive replay selected
target query capacities **12 and 8 versus full width 24 on all eight DP
ranks**; responses retained valid grammar and per-token finite logprobs.

---------

Signed-off-by: wangbo981016 <wangbo981016@gmail.com>
2026-11-18 15:10:41 +08:00
FC.Li 5cbc2329a4 feat(cache): visualize recipe layouts in startup logs (#3571)
Cache startup logs show capacity and packing without exposing field
placement or unused bytes. Add an ASCII layout matrix after capacity
agreement and PP narrowing, with cache group names as columns and
resident planes as rows.

The report includes physical packing and usable block counts, field
offsets, shapes, and dtypes, stride padding, unused-plane holes, and
per-parent payload utilization. Each column describes an alternative
binding of the same parent. Serving builds log at INFO; memory probes
log at DEBUG. The design documentation explains byte accounting,
null-parent reservation, and PP and offload interpretation.

Validation:

- 44 CPU planner and visualization tests passed: `python -m pytest
--noconftest -q test/runtime/test_cache_layout_visualization.py
test/runtime/test_cache_memory_plan.py`.
- Coverage includes shared-stride alignment gaps, trailing padding,
unused planes, PP field and plane filtering, null-parent accounting, and
large capacities.
- `pre-commit run --all-files` passed.
- GPU serving was not exercised.

## Example startup logs

These examples use the cache field declarations with the same `pack()`
and `format_cache_layout()` code as the startup report. Capacity is
illustrative: **one usable parent plus one reserved null parent**, with
`pp_rank=0`. Actual serving capacity follows memory budgeting and
scheduler demand. Logger timestamps and rank prefixes are omitted.

### DeepSeek V4.1 Flash with DSpark

Configuration: native `v41` cache format and **three DSpark stages**.
The single `flatkv` plane is **3,225,600 B per parent**. The groups
reuse that plane with different packing, so their columns describe
alternative parent bindings and must not be added together.

<details>
<summary>Full V4.1 Flash + DSpark layout log</summary>

```text
CacheRecipe DeepseekV41Recipe (deepseek_v41): build=serving, pp_rank=0
prefix_granularity=256 tokens, usable_parents=1, reserved_null_parents=1
logical_parent=3,225,600 B, resident_parent=3,225,600 B, layout_arena=6,451,200 B (0.006 GiB)
Groups are alternative bindings of the same parent; do not sum columns.
Packing and block counts are rank-local physical counts.
Ranges [start, end) are bytes within one child stride, repeated packing times.
Padding is unused space in an occupied plane; a hole is an unused plane.
+--------------------+-----------------------------------+---------------------------------+-----------------------------------+------------------------------------------+
| plane              | v41.compressor_tail_r2            | v41.global_r1                   | v41.global_r2                     | v41.swa                                  |
|                    | packing=126                       | packing=140                     | packing=45                        | packing=2                                |
|                    | usable_blocks=126                 | usable_blocks=140               | usable_blocks=45                  | usable_blocks=2                          |
+--------------------+-----------------------------------+---------------------------------+-----------------------------------+------------------------------------------+
| flatkv             | stride=25,600 B                   | stride=23,040 B                 | stride=71,680 B                   | stride=1,612,800 B                       |
| parent=3,225,600 B | layer.14.compressor_tail          | layer.20.global_kv              | layer.14.global_kv                | layer.0.swa                              |
| arena_offset=0 B   |   [0, 8,192): 8,192 B             |   [0, 18,432): 18,432 B         |   [0, 18,432): 18,432 B           |   [0, 33,792): 33,792 B                  |
|                    |   float32 (2, 2, 512)             |   uint8 (64, 288)               |   uint8 (64, 288)                 |   uint8 (64, 528)                        |
|                    | layer.2.compressor_tail           | layer.20.index_k                | layer.14.index_k                  | layer.1.swa                              |
|                    |   [8,192, 16,384): 8,192 B        |   [18,432, 22,784): 4,352 B     |   [18,432, 22,784): 4,352 B       |   [33,792, 67,584): 33,792 B             |
|                    |   float32 (2, 2, 512)             |   uint8 (64, 68)                |   uint8 (64, 68)                  |   uint8 (64, 528)                        |
|                    | layer.8.compressor_tail           | padding [22,784, 23,040): 256 B | layer.2.global_kv                 | layer.10.swa                             |
|                    |   [16,384, 24,576): 8,192 B       | payload=3,189,760 B/parent      |   [22,784, 41,216): 18,432 B      |   [67,584, 101,376): 33,792 B            |
|                    |   float32 (2, 2, 512)             | padding=35,840 B/parent         |   uint8 (64, 288)                 |   uint8 (64, 528)                        |
|                    | padding [24,576, 25,600): 1,024 B |                                 | layer.2.index_k                   | layer.11.swa                             |
|                    | payload=3,096,576 B/parent        |                                 |   [41,216, 45,568): 4,352 B       |   [101,376, 135,168): 33,792 B           |
|                    | padding=129,024 B/parent          |                                 |   uint8 (64, 68)                  |   uint8 (64, 528)                        |
|                    |                                   |                                 | layer.8.global_kv                 | layer.12.swa                             |
|                    |                                   |                                 |   [45,568, 64,000): 18,432 B      |   [135,168, 168,960): 33,792 B           |
|                    |                                   |                                 |   uint8 (64, 288)                 |   uint8 (64, 528)                        |
|                    |                                   |                                 | layer.8.index_k                   | layer.13.swa                             |
|                    |                                   |                                 |   [64,000, 68,352): 4,352 B       |   [168,960, 202,752): 33,792 B           |
|                    |                                   |                                 |   uint8 (64, 68)                  |   uint8 (64, 528)                        |
|                    |                                   |                                 | padding [68,352, 71,680): 3,328 B | layer.14.swa                             |
|                    |                                   |                                 | payload=3,075,840 B/parent        |   [202,752, 236,544): 33,792 B           |
|                    |                                   |                                 | padding=149,760 B/parent          |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.15.swa                             |
|                    |                                   |                                 |                                   |   [236,544, 270,336): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.16.swa                             |
|                    |                                   |                                 |                                   |   [270,336, 304,128): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.17.swa                             |
|                    |                                   |                                 |                                   |   [304,128, 337,920): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.18.swa                             |
|                    |                                   |                                 |                                   |   [337,920, 371,712): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.19.swa                             |
|                    |                                   |                                 |                                   |   [371,712, 405,504): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.2.swa                              |
|                    |                                   |                                 |                                   |   [405,504, 439,296): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.20.swa                             |
|                    |                                   |                                 |                                   |   [439,296, 473,088): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.21.swa                             |
|                    |                                   |                                 |                                   |   [473,088, 506,880): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.22.swa                             |
|                    |                                   |                                 |                                   |   [506,880, 540,672): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.23.swa                             |
|                    |                                   |                                 |                                   |   [540,672, 574,464): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.24.swa                             |
|                    |                                   |                                 |                                   |   [574,464, 608,256): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.25.swa                             |
|                    |                                   |                                 |                                   |   [608,256, 642,048): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.26.swa                             |
|                    |                                   |                                 |                                   |   [642,048, 675,840): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.27.swa                             |
|                    |                                   |                                 |                                   |   [675,840, 709,632): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.28.swa                             |
|                    |                                   |                                 |                                   |   [709,632, 743,424): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.29.swa                             |
|                    |                                   |                                 |                                   |   [743,424, 777,216): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.3.swa                              |
|                    |                                   |                                 |                                   |   [777,216, 811,008): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.30.swa                             |
|                    |                                   |                                 |                                   |   [811,008, 844,800): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.31.swa                             |
|                    |                                   |                                 |                                   |   [844,800, 878,592): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.32.swa                             |
|                    |                                   |                                 |                                   |   [878,592, 912,384): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.33.swa                             |
|                    |                                   |                                 |                                   |   [912,384, 946,176): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.34.swa                             |
|                    |                                   |                                 |                                   |   [946,176, 979,968): 33,792 B           |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.35.swa                             |
|                    |                                   |                                 |                                   |   [979,968, 1,013,760): 33,792 B         |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.36.swa                             |
|                    |                                   |                                 |                                   |   [1,013,760, 1,047,552): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.37.swa                             |
|                    |                                   |                                 |                                   |   [1,047,552, 1,081,344): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.38.swa                             |
|                    |                                   |                                 |                                   |   [1,081,344, 1,115,136): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.39.dspark_kv0                      |
|                    |                                   |                                 |                                   |   [1,115,136, 1,180,672): 65,536 B       |
|                    |                                   |                                 |                                   |   bfloat16 (64, 512)                     |
|                    |                                   |                                 |                                   | layer.39.dspark_kv1                      |
|                    |                                   |                                 |                                   |   [1,180,672, 1,246,208): 65,536 B       |
|                    |                                   |                                 |                                   |   bfloat16 (64, 512)                     |
|                    |                                   |                                 |                                   | layer.39.dspark_kv2                      |
|                    |                                   |                                 |                                   |   [1,246,208, 1,311,744): 65,536 B       |
|                    |                                   |                                 |                                   |   bfloat16 (64, 512)                     |
|                    |                                   |                                 |                                   | layer.39.swa                             |
|                    |                                   |                                 |                                   |   [1,311,744, 1,345,536): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.4.swa                              |
|                    |                                   |                                 |                                   |   [1,345,536, 1,379,328): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.5.swa                              |
|                    |                                   |                                 |                                   |   [1,379,328, 1,413,120): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.6.swa                              |
|                    |                                   |                                 |                                   |   [1,413,120, 1,446,912): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.7.swa                              |
|                    |                                   |                                 |                                   |   [1,446,912, 1,480,704): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.8.swa                              |
|                    |                                   |                                 |                                   |   [1,480,704, 1,514,496): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | layer.9.swa                              |
|                    |                                   |                                 |                                   |   [1,514,496, 1,548,288): 33,792 B       |
|                    |                                   |                                 |                                   |   uint8 (64, 528)                        |
|                    |                                   |                                 |                                   | padding [1,548,288, 1,612,800): 64,512 B |
|                    |                                   |                                 |                                   | payload=3,096,576 B/parent               |
|                    |                                   |                                 |                                   | padding=129,024 B/parent                 |
+--------------------+-----------------------------------+---------------------------------+-----------------------------------+------------------------------------------+
| TOTAL / parent     | payload=3,096,576 B               | payload=3,189,760 B             | payload=3,075,840 B               | payload=3,096,576 B                      |
| 3,225,600 B        | padding=129,024 B                 | padding=35,840 B                | padding=149,760 B                 | padding=129,024 B                        |
|                    | hole=0 B                          | hole=0 B                        | hole=0 B                          | hole=0 B                                 |
|                    | unused=129,024 B                  | unused=35,840 B                 | unused=149,760 B                  | unused=129,024 B                         |
|                    | utilization=96.00%                | utilization=98.89%              | utilization=95.36%                | utilization=96.00%                       |
+--------------------+-----------------------------------+---------------------------------+-----------------------------------+------------------------------------------+
```

</details>

### Kimi K3 with DSpark

Configuration: **TP8, five MLA draft layers, FP8 KV for both target and
draft, and FP32 KDA recurrent state**. The parent has 24 target MLA
planes and five draft planes, totaling **25,657,344 B**. The reported
`uint8` dtype is the FP8 KV storage view.

Each KDA group has 23 layers and uses `slot.0` through `slot.22`.
`slot.23` and the five draft planes (`slot.24` through `slot.28`) are
unused by a KDA binding: **5,308,416 B of holes**, plus **1,625,088 B of
stride padding**, giving **72.98% payload utilization**.

The excerpt below preserves the original table formatting and includes
`slot.0`, `slot.23`, `slot.24`, `slot.28`, and the totals. Other plane
rows are omitted; totals cover all 29 planes.

<details>
<summary>K3 TP8 + DSpark layout log (selected planes and complete
totals)</summary>

```text
CacheRecipe KimiK3Recipe (kimi_k3): build=serving, pp_rank=0
prefix_granularity=128 tokens, usable_parents=1, reserved_null_parents=1
logical_parent=25,657,344 B, resident_parent=25,657,344 B, layout_arena=51,314,688 B (0.048 GiB)
Groups are alternative bindings of the same parent; do not sum columns.
Packing and block counts are rank-local physical counts.
Ranges [start, end) are bytes within one child stride, repeated packing times.
Padding is unused space in an occupied plane; a hole is an unused plane.
+---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| plane                     | full_attention           | linear_attention_0                   | linear_attention_1                   | linear_attention_2                   |
|                           | packing=12               | packing=1                            | packing=1                            | packing=1                            |
|                           | usable_blocks=12         | usable_blocks=1                      | usable_blocks=1                      | usable_blocks=1                      |
+---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| slot.0                    | stride=73,728 B          | stride=884,736 B                     | stride=884,736 B                     | stride=884,736 B                     |
| parent=884,736 B          | layer.3.latent_kv        | layer.0.conv_state                   | layer.30.conv_state                  | layer.61.conv_state                  |
| arena_offset=0 B          |   [0, 73,728): 73,728 B  |   [0, 27,648): 27,648 B              |   [0, 27,648): 27,648 B              |   [0, 27,648): 27,648 B              |
|                           |   uint8 (128, 1, 576)    |   bfloat16 (4608, 3)                 |   bfloat16 (4608, 3)                 |   bfloat16 (4608, 3)                 |
|                           | payload=884,736 B/parent | layer.0.recurrent_state              | layer.30.recurrent_state             | layer.61.recurrent_state             |
|                           | padding=0 B/parent       |   [27,648, 814,080): 786,432 B       |   [27,648, 814,080): 786,432 B       |   [27,648, 814,080): 786,432 B       |
|                           |                          |   float32 (12, 128, 128)             |   float32 (12, 128, 128)             |   float32 (12, 128, 128)             |
|                           |                          | padding [814,080, 884,736): 70,656 B | padding [814,080, 884,736): 70,656 B | padding [814,080, 884,736): 70,656 B |
|                           |                          | payload=814,080 B/parent             | payload=814,080 B/parent             | payload=814,080 B/parent             |
|                           |                          | padding=70,656 B/parent              | padding=70,656 B/parent              | padding=70,656 B/parent              |
+---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| slot.23                   | stride=73,728 B          | hole=884,736 B/parent                | hole=884,736 B/parent                | hole=884,736 B/parent                |
| parent=884,736 B          | layer.92.latent_kv       |                                      |                                      |                                      |
| arena_offset=28,311,552 B |   [0, 73,728): 73,728 B  |                                      |                                      |                                      |
|                           |   uint8 (128, 1, 576)    |                                      |                                      |                                      |
|                           | payload=884,736 B/parent |                                      |                                      |                                      |
|                           | padding=0 B/parent       |                                      |                                      |                                      |
+---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| slot.24                   | stride=73,728 B          | hole=884,736 B/parent                | hole=884,736 B/parent                | hole=884,736 B/parent                |
| parent=884,736 B          | layer.93.latent_kv       |                                      |                                      |                                      |
| arena_offset=30,081,024 B |   [0, 73,728): 73,728 B  |                                      |                                      |                                      |
|                           |   uint8 (128, 1, 576)    |                                      |                                      |                                      |
|                           | payload=884,736 B/parent |                                      |                                      |                                      |
|                           | padding=0 B/parent       |                                      |                                      |                                      |
+---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| slot.28                   | stride=73,728 B          | hole=884,736 B/parent                | hole=884,736 B/parent                | hole=884,736 B/parent                |
| parent=884,736 B          | layer.97.latent_kv       |                                      |                                      |                                      |
| arena_offset=37,158,912 B |   [0, 73,728): 73,728 B  |                                      |                                      |                                      |
|                           |   uint8 (128, 1, 576)    |                                      |                                      |                                      |
|                           | payload=884,736 B/parent |                                      |                                      |                                      |
|                           | padding=0 B/parent       |                                      |                                      |                                      |
+---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
| TOTAL / parent            | payload=25,657,344 B     | payload=18,723,840 B                 | payload=18,723,840 B                 | payload=18,723,840 B                 |
| 25,657,344 B              | padding=0 B              | padding=1,625,088 B                  | padding=1,625,088 B                  | padding=1,625,088 B                  |
|                           | hole=0 B                 | hole=5,308,416 B                     | hole=5,308,416 B                     | hole=5,308,416 B                     |
|                           | unused=0 B               | unused=6,933,504 B                   | unused=6,933,504 B                   | unused=6,933,504 B                   |
|                           | utilization=100.00%      | utilization=72.98%                   | utilization=72.98%                   | utilization=72.98%                   |
+---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+
```

</details>

Signed-off-by: FC-Li <robert_lexis@163.com>
2026-11-18 14:41:55 +08:00
FC.Li 85266e5764 refactor(scheduler): make cache planning data flow explicit (#3566)
## Summary

Cache admission and prefix probing carried intermediate results through
parallel arrays, output references, and preparation-dependent members.
This refactor makes request inputs, returned decisions, and per-attempt
working state explicit.

- Group placement requirements in `BlockPlacementDemand`; return
admission capacity, constraints, copy-in reservations, and decisions as
values.
- Construct candidate traversals directly and derive checkpoint aging
from selected State blocks, removing mirrored checkpoint-removal
counters. Derive group-scan counts from cursor state and L3 completion
from per-key results, including all-failed results.
- Return prefix-probe results together while reusing hit-array storage;
make derived cache metadata and allocator configuration immutable, and
strengthen const views.
- Clarify block counts versus indices, available tail capacity, and
interfaces that record statistics or acquire owners. Document explicit
data flow, functional style, const correctness, and consistent naming in
`AGENTS.md`.
- Install the exact clang-format revision pinned by pre-commit in CI.
The runner's newer distro formatter otherwise rejects code that passes
the pinned formatter.

## Validation

- `pre-commit run --all-files`: passed.
- Build and run the full scheduler C++ suite: 1,550 passed; 1
AddressSanitizer-only test skipped in the Release build.
- CI-equivalent clang-format 18.1.8 dry run over all 170 scheduler C++
files: passed.

---------

Signed-off-by: FC-Li <robert_lexis@163.com>
2026-11-18 13:51:27 +08:00
dingyuqing05andlongcat-ia-team 2a2faef07c fix: preserve explicit CUDA device connection limits (#3562)
Summary
- Preserve CUDA_DEVICE_MAX_CONNECTIONS supplied by the caller.
- Retain the default of 4 when unset and document the override.
- Add regression coverage for unset, 1, 4, 8, and 32.

Results
- All 5 environment regression cases pass.
- Disaggregated inference passes 6 requests, including concurrent and
streaming requests.
- All 8 GPU workers inherit CUDA_DEVICE_MAX_CONNECTIONS=8.

Test plan
- Passed: targeted launcher regression tests, Black, isort, and git diff
--check.
- Passed: pre-commit run --all-files and cargo fmt --all --check.
- Attempted: cargo clippy --all-targets --locked -- -D warnings; blocked
by CMake failing to resolve Python Interpreter and Development.Module.
Further checks were stopped at the author's request.

Signed-off-by: dingyuqing05 <13126694060@163.com>
Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>
2026-11-17 21:27:06 -08:00