mirror of
https://github.com/lightseekorg/tokenspeed.git
synced 2026-11-21 01:53:23 +08:00
main
1332
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f8acfc3a30 |
perf(kda): replace binary prefill with FlashInfer CuTe kernels (#3665)
<!-- Headings and layout are optional. Include change context and
validation
(checks performed or planned, or why testing was not run or is not
applicable)
anywhere in the description. -->
## Summary
Replace the binary KDA prefill dependency with FlashInfer CuTe kernels
under the existing `cutedsl_kda` backend.
- Consume strided Q/K/V directly from convolution output and remove
obsolete preparation and installation code.
- Select ordinary or persistent CuTe using kernel eligibility,
request/head count, and SM count. Build persistent schedules from live
GPU boundaries for eager execution and graph replay, with
invocation-owned scratch and startup warmup.
- Optimize recurrent-state scatter with a request × tile grid and
fixed-layout specialization.
Preserve FP32 scan state, the existing zero-padding contract, and LCM
state ownership.
Persistent integration adapted from
[
|
||
|
|
a6df89ddce |
fix(pd): revoke a dead writer's access instead of pinning Decode pages (#3658)
## Problem
A failed Decode receive may reuse its pages only after every Prefill
target acknowledges the cancellation, because a failure report or
timeout does not prove that remote writes stopped
(`docs/design/event-loop.md`, PD cancellation). Sometimes that
acknowledgment never comes:
- the Prefill process died;
- it restarted at a new control port, so cancellations go to a dead
address;
- it holds a native write whose DMA may still land, so it withholds the
acknowledgment.
In all three cases the pages stayed pinned until the Decode rank
restarted. Every Prefill crash leaked the pages of the receives in
flight to it at that moment. Sleep with memory release, cache flush and
weight replacement could not drain either.
```mermaid
sequenceDiagram
participant D as Decode scheduler
participant O as Old Prefill
participant N as Restarted Prefill
D->>O: manifest (Decode page addresses)
O->>D: RDMA writes into the pages
Note over O: process dies, a write may be unfinished
Note over D: heartbeat lost, room failed
loop every second
D->>O: Cancel (to the old control port)
end
Note over O: nothing listens on the old port
Note over N: new random control port, old cache unknown
Note over D: CancelAck never comes: pages and slot stay pinned
```
## Fix: revoke the writers instead of trusting a timeout
Each Decode rank owns a Mooncake transfer engine (`ts_mooncake::Engine`
in its PD transfer peer). The engine registers the rank's KV cache with
the RDMA NICs and serves the session Prefill handshakes with to obtain
the keys it writes with. A revocation replaces this engine; it is not
the inference engine.
Trigger: a failed receive has waited `waiting_timeout` for
acknowledgments, and its manifest named the current transfer engine. The
Decode peer then starts a revocation on a background thread:
1. Start a fresh Mooncake engine with the cache regions registered (no
lock held).
2. Under the state lock:
- fail every receive still attached to the replaced transport, whether
published or waiting for admission;
- make the fresh engine the one transfers use and registrations name
(`Transport::install_engine`);
- switch to a new cache identity;
- clear route registrations.
3. Deregister the cache from the replaced engine.
4. Only if that succeeded, the receives whose manifests named the
replaced engine retire without acknowledgments. Otherwise they keep
waiting and do not trigger revocations of later engines. The replaced
engine may still reach the cache, so the cache is then never released
(sleep with memory release is refused); this is fail-closed on a double
failure.
```mermaid
sequenceDiagram
participant D as Decode scheduler
participant R as Revocation thread
participant E1 as Decode Mooncake engine (old)
participant E2 as Decode Mooncake engine (new)
participant W as Old Prefill writer
Note over E1,E2: Decode's transfer engines:<br/>each registers the KV cache with the RDMA NICs<br/>and hands Prefill the keys it writes with
Note over D: failed room waited waiting_timeout for CancelAck
D-)R: start revocation (off the scheduling loop)
R->>E2: start engine, register the cache
Note over R: swap engine and cache identity, fail attached receives
R->>E1: deregister the cache
alt deregistration succeeded
E1-->>R: deregistered, old keys invalid
R-->>D: retire the rooms, reuse their pages
W->>E1: RDMA write with an old key
E1--xW: rejected by the NIC
else deregistration failed
E1-->>R: deregistration error
Note over R: some regions may still be registered, so old keys may still work: rooms keep waiting for CancelAck, memory never released
end
```
The fence, not the timeout, is what authorizes reuse. An RDMA NIC checks
each incoming write against the destination's registration, and
`ibv_dereg_mr` returns only once the NIC no longer uses the MR; the
kernel may unpin the pages right after. The timeout only decides when to
pay for the fence.
Details:
- **Transport:** only `rdma` revokes. With other transports such
receives stay pinned, as before.
- **Replaced engine kept alive:** it stays without the cache registered
for the peer's lifetime, so its session (host and RPC port) is never
reused by a later engine. A revoked writer that resolves its old session
again would otherwise reach the newer engine. The test reproduces
exactly this with a sleep/wake after the revocation when the replaced
engine is dropped.
- **Cache identity per receive:** each room records the cache identity
its route registered (and so the engine its manifest names).
Cancellations and their acknowledgments use that identity, so a receive
failed before admission still reaches Prefill under an identity it
knows. Under `deferred` decode allocation (#3394), a room's
announcements use that identity too.
- **No sleep mid-revocation:** memory release is refused while a
revocation runs, since it installs a fresh engine with the cache
registered. `revoke` marks the fenced rooms and ends the revocation in
the critical section that listed them, so a room retiring in between is
not marked again. The running revocation is also an RAII guard whose
drop ends a revocation that did not end itself (no fresh engine, or a
panic).
- **No stranded rooms:** rooms connecting during a revocation wait and
reconnect under the new identity instead of staying in Bootstrapping. A
receive that already failed never publishes its manifest later.
- **Observability:** each outcome is logged as an event with the rank,
the rooms and the old and fresh sessions:
`event=ts_decode_revocation_fenced`,
`event=ts_decode_revocation_unfenced` (error, with the pinned rooms),
`event=ts_decode_revocation_not_started`. A release refused for an
unfenced cache logs `event=ts_decode_release_refused_unfenced`, so the
lasting effect can be traced back after the first log line is gone.
- **Cost:** each revocation fails every receive attached on that rank at
that moment (its replica peers fail them too, through the agreed
transfer status), and keeps one idle engine.
`docs/design/event-loop.md` gains a "PD receive revocation" paragraph.
## Tests
`tests/cancellation.rs::a_cancellation_unacknowledged_past_the_receive_budget_is_retired_by_revoking_the_transport`
runs real peers, sockets and bootstrap with the CPU native stand-in. The
Prefill write is blocked in native I/O, so it withholds the
acknowledgment as a dead peer would. The test checks:
- the cancelled receive retires after the budget;
- a request waiting for admission fails, and admitting it afterwards
publishes no manifest (no second revocation);
- a sleep/wake while the revoked write is still blocked;
- the released write does not reach the recycled pages;
- the next request transfers through the fresh engine.
The test fails on `main` (the receive never retires). It also fails if:
- a failed receive still publishes;
- the replaced engine keeps its registration;
- the replaced engine is dropped (session port reuse).
`a_revocation_whose_deregistration_fails_keeps_waiting_for_acknowledgments`
makes the stand-in's deregistration fail. It checks that:
- the receive keeps waiting and starts no second revocation;
- it retires on the late acknowledgment;
- the cache can no longer be released.
Unit tests cover cancellations acknowledged under the cache their route
registered, revoked rooms retiring without acknowledgments, rooms marked
as the revocation ends (a room that retires before the guard drops is
not marked again), deferred-allocation announcements after a revocation,
a registration wait interrupted by a revocation reconnecting under the
fresh cache, and the logged outcome events.
`cargo test -p tokenspeed-pd`, `cargo fmt --all --check`, `cargo clippy
--all-targets -- -D warnings` and `pre-commit run --all-files` pass
locally.
## Not covered / review notes
- The fence relies on `ts_mooncake_unregister` returning only after
`ibv_dereg_mr` succeeded on every device. This needs confirming against
the Mooncake build in use, plus a run on real RDMA (kill a Prefill rank
mid-transfer and check the Decode pages are released and not
overwritten).
- Builds on #3657 (merged): a room waiting for its route's registration
acknowledgment gives up that registration when a revocation starts and
connects again under the fresh cache identity once it finished.
Signed-off-by: wangbo981016 <wangbo981016@gmail.com>
Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com>
|
||
|
|
87a6c2995a |
perf(ple): optimize rank-local host gather and verify state writes (#3381)
## Summary Qwen4-Exp PLE can spend over a millisecond reading an offloaded embedding table during a long prefill. Its convolution also reloads the same channel weights across tokens, and speculative verification writes a full convolution-state window for every candidate. This PR reduces those three costs. - Batch multiple rows per CTA in the regular UVA gather. For a complete table and at least 131,072 n-gram IDs, sort the IDs before the persistent gather and write the results directly into their original positions. - Add a rank-local HBM cache for offloaded Qwen4-Exp PLE tables on NVIDIA Hopper and Blackwell. It supports 16 n-gram heads, head dimensions of 160 or 512, and BF16 lookup output. Tables can use BF16 storage or per-tensor FP8 storage, including FP8 checkpoints and BF16 weights quantized online. The gather deduplicates IDs within eligible 64-token windows per head and stages cache misses for asynchronous fill. Lookups below 256 tokens and CUDA Graph capture and replay use the regular UVA gather. - Add `--enable-ple-hot-cache` to enable the cache when PLE offloading is active. The cache is off by default. `TOKENSPEED_PLE_HOST_CACHE_SLOTS` sets the capacity per rank and defaults to 524,288 slots. It must be a power of two greater than one. At the default capacity, a 160-wide FP8 table uses 84 MiB of cache memory per rank, plus temporary miss staging. - Tile long BF16 prefill convolution across 8 tokens and 128 channels on NVIDIA Blackwell so one CTA can reuse the channel weights. For speculative verify, store the carried state and one normalized input per candidate, then reconstruct the accepted window in its destination page with `copy_state_rows_compact`. Regular GPU convolution and the AMD monokernel share the compact layout. ## Performance Qwen3.8-Flash-Next-NVFP4 on a single NVIDIA SM103 GPU, TP1 unless noted. ### Agentic Bench with All Optimizations Enabled The baseline and optimized builds ran sequentially on the same GPU with MTP3, FP8 KV cache, FP8 PLE offload, CUDA Graph, and 16K prefill chunks. The optimized run enabled `--enable-ple-hot-cache` with 524,288 slots and used the branch's UVA, convolution, and compact verify paths as applicable. Both variants used `TOKENSPEED_SPEC_SIMULATED_ACCEPT_LEN=2.32`, including the verification token, matching the InferenceX golden accept-length distribution for Flash-Next native MTP3 with thinking enabled. EvalScope measured one round per variant and concurrency after warmup and a prefix-cache flush, with 4 conversations at BS1, 32 at BS32, and 500 output tokens per turn. All measured requests succeeded. | Concurrency | Metric | Baseline | This PR | Change | | ---: | --- | ---: | ---: | ---: | | 1 | TTFT (ms) | 282.59 | 280.44 | -0.76% | | 1 | Decode (tokens per second per request) | 442.48 | 444.44 | +0.44% | | 1 | Aggregate output (tokens per second) | 353.65 | 355.46 | +0.51% | | 1 | Decoded tokens per iteration | 2.2924 | 2.2934 | +0.04% | | 32 | TTFT (ms) | 4,313.30 | 4,299.87 | -0.31% | | 32 | Decode (tokens per second per request) | 67.07 | 67.34 | +0.40% | | 32 | Aggregate output (tokens per second) | 1,284.40 | 1,288.59 | +0.33% | | 32 | Decoded tokens per iteration | 2.2934 | 2.2931 | -0.01% | End-to-end changes were below 1% in this run. At BS1 and BS32, MTP3 verifies at most 128 tokens per batch, below the 256-token hot-cache threshold. Compact verify saves 1.40 µs at BS1 and 8.31 µs at BS32 for convolution and state commit together, a small part of the full decode step. In the earlier 16K prefill profiles below, the host gather already finished before its consumer in both variants, and convolution tiling reduced the full target GPU span by 0.23%. ### Prefill convolution Flash-Next TP1 geometry (10,240 BF16 channels, four taps, dilation 3), with PDL and CUDA Graph enabled in the microbenchmark. The A and B comparison isolates the tile change and compares 1 token × 256 channels with 8 tokens × 128 channels. The convolution and final state matched bit for bit. | Prefill case | Baseline (µs) | Tiled (µs) | Change | | --- | ---: | ---: | ---: | | 2,048 tokens | 106.58 | 65.50 | −38.5% | | 8,192 tokens | 395.25 | 217.10 | −45.1% | | 8,192 tokens, ragged requests | 398.24 | 217.01 | −45.5% | | 16,384 tokens | 779.66 | 423.94 | −45.6% | In a separate Nsight Systems capture of one real 16K prefill step with the same hot-cache path on both sides, convolution took 776.04 → 417.92 µs (−46.1%). The target GPU span was 364.08 → 363.25 ms (−0.23%). ### Host gather #### Regular UVA gather Both variants use the same complete FP8 PLE table with hot cache disabled. The A and B comparison uses six measured Agentic prefill windows. Each 8K case uses the first half of its 16K window. Timing includes sorting and output reordering. The table reports means of per-window medians, and all 12 outputs matched bit for bit. | Prefill tokens | Direct UVA gather | Sort and persistent UVA gather | Change | | --- | ---: | ---: | ---: | | 8K | 976.07 µs | 744.36 µs | −23.7% | | 16K | 1,889.63 µs | 1,299.11 µs | −31.3% | #### HBM hot cache One Nsight Systems trace per variant measures a 16,384-token prefill step on a live Flash-Next TP1 server. The comparison uses the baseline and an early hot-cache implementation. | Metric | Regular gather | HBM hot cache | Change | | --- | ---: | ---: | ---: | | Host gather kernel | 1,462.35 µs | 372.13 µs | −74.6% | | Cache fill on a separate CUDA stream | — | 14.82 µs | — | ### Decode convolution and state commit This microbenchmark compares full-window → compact verify using `_ple_conv_state_kernel`; the state commit replaces `copy_state_rows` with `copy_state_rows_compact`. Both use 10,240 BF16 channels with CUDA Graph and PDL enabled. Values are medians of five alternating A and B trials. The combined column times both launches together. Convolution output and reconstructed state matched for every accepted length. | Speculative steps | Batch size | Verify conv (µs) | State commit (µs) | Combined (µs) | Combined change | | --- | ---: | ---: | ---: | ---: | ---: | | MTP3 | 1 | 4.78 → 3.08 | 1.11 → 1.43 | 6.02 → 4.62 | -23.3% | | MTP3 | 8 | 11.31 → 7.66 | 1.37 → 4.07 | 13.33 → 9.75 | -26.8% | | MTP3 | 32 | 34.10 → 15.65 | 2.35 → 12.53 | 36.76 → 28.45 | -22.6% | | MTP7 | 1 | 5.00 → 3.22 | 1.05 → 1.31 | 6.49 → 4.77 | -26.4% | | MTP7 | 8 | 17.80 → 7.99 | 1.28 → 3.88 | 19.38 → 11.74 | -39.4% | | MTP7 | 32 | 61.23 → 19.24 | 2.36 → 12.55 | 64.84 → 32.06 | -50.5% | Compact verify reduces the writes in the convolution kernel. The state commit does more work to reconstruct the window, and the two launches together take less time across the measured cases. For single-token decode, both variants use the same tile geometry. With identical inputs and output buffers, convolution latency was 2.87 → 2.87 µs at BS1, 4.84 → 4.84 µs at BS8, 14.99 → 14.93 µs at BS32. Outputs and final states matched bit for bit. ## Test Plan - With the fast math CUDA library on SM103, ran 39 focused kernel tests covering FP8 and BF16 cache lookup, shard masking, validity checks, dynamic prefill bounds, and convolution across request boundaries. - On SM103, ran the PLE backend and cache setup tests against the workspace and compact state-commit changes (55 passed). Coverage includes accepted lengths, invalid pages, mixed per-layer page strides, cache-pool rebinding, and graph replay. - During sharded cache integration, ran the TP4 and DP4 lookup tests against an unsharded reference, including a DP rank with zero local tokens. Compiled the prefill tiles for SM100 and the compact monokernel PLE stages for gfx950. - Compared the CUDA host gather builds with and without fast math. The Flash-Next checkpoint's FP8 scale produced bitwise-identical BF16 output across all 256 FP8 byte encodings in the tested cold and prepopulated caches. BF16 copies also matched. Synthetic scales near the FP32 subnormal range produced differences. --------- Signed-off-by: minedec <jiandong@lightseek.org> |
||
|
|
68c66eabef |
perf(comm): fuse scatter projections' GEMM with their reduce-scatter (#3662)
## Summary Scatter projections (#3651) reduce every rank's partial of all rows into owner rows. This adds `comm_ops.projection_mm_reduce_scatter`, which takes the rows and this rank's input-shard weight and returns the owner rows, so the backend can fuse the GEMM with the reduce-scatter: - Rows the Lamport reduce-scatter holds keep the GEMM and that reduce-scatter. - Above them, groups with NVLink multicast on SM100 and SM103 run NVIDIA CUTLASS's CuTe DSL GEMM + reduce-scatter, vendored under `thirdparty/cute_dsl/gemm_reduce_scatter` and registered as `cute_dsl_gemm_reduce_scatter`. Each rank stores its BF16 partial tiles in symmetric memory and signals the tile's owner; the owner sums all ranks' tiles with `multimem.ld_reduce` (FP32 accumulation in the switch, one rounding) while the remaining tiles compute. - Other groups keep the GEMM and the backend's reduce-scatter. The kernel's state is created while the projection is prepared, collectively and before graph capture; ranks agree on support through the existing host agreement, so every rank takes the same collective. Muse Spark's owned attention output projection now calls the new op. ## Results GB200 x4, TP4 and EP4, NVFP4; #3651 vs this PR, two pairs of A and B runs on one node. Prefill only (2 output tokens), mean TTFT ms: | prompt x concurrency | #3651 | this PR | |---|---|---| | 4k x 1 | 120 and 120 | 111 and 116 | | 8k x 1 | 206 and 206 | 192 and 194 | | 8k x 16 | 1584 and 1586 | 1521 and 1509 | | 1k x 128 | 1529 and 1531 | 1474 and 1472 | 1k input and 1k output tokens, output tokens per second and mean TTFT ms: | concurrency | #3651 | this PR | |---|---|---| | 64 | 3624, 903 and 3645, 886 | 3623, 843 and 3650, 853 | | 128 | 5342, 1569 and 5328, 1586 | 5319, 1524 and 5353, 1502 | | 256 | 7896, 3016 and 7867, 3011 | 7894, 2885 and 7920, 2864 | GSM8K, 512 questions at concurrency 32: #3651 0.977 and 0.965, this PR 0.984 and 0.984. Greedy outputs of six 20-shot GSM8K prompts (about 3.3k tokens) are identical to #3651. Alone, on 4 GB200s, projecting 8192 rows of 2048 channels per rank into 8192 outputs takes 229 us against 406 us for the GEMM and NCCL's reduce-scatter. ## Tests - `tokenspeed-kernel/test/nvidia/ops/communication/test_gemm_reduce_scatter.py`: on every GPU of the node (2, 4 or 8), the fused kernel matches an exact integer reference within one BF16 unit, for row blocks that fill and pad the 128-row tile, row counts that change between calls without recompiling, graph replays with fresh inputs, and owner rows past 2^31 elements of the symmetric storage. - `test/runtime/distributed/test_owner_row_projections.py`: `projection_mm_reduce_scatter` matches the reduced reference for one row, Lamport-sized and fused-kernel-sized batches, and rejects more rows than the owners hold, with the auto and NCCL backends on 2 and 4 GPUs. --------- Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com> |
||
|
|
bd8f0d4f4a |
perf(dsa): score and select KVP decode index candidates in two CuTe DSL launches (#3660)
## Summary A KVP decode's local index candidates ran the DeepGEMM leaf's query quantization, schedule metadata, paged logits, non-finite scrub, ragged top-k, and gather, then copied the scores into the owned top-k's send buffer: about 30 microseconds per DSA layer at one request. - The DeepGEMM leaf writes its candidate scores straight into the send buffer (`scores_out`) and copies the query only when it is strided. - `dsa_decode_index_candidates` (`cute_dsl` on SM100 and SM103) runs the fused decode top-k's scorer and select in a KVP mode: every request is scored over its compact owned pages to each row's own local length, and the select writes each row's exact local top-k as logical positions and FP32 logits, recovered from its order keys. The logits are bit for bit the DeepGEMM leaf's. - A row that sees at most `topk` keys over every rank keeps all of them in the merged selection, so the op takes each row's keys over every rank (`row_lengths`): the scorer schedules no tiles for a request whose last row fits, and the select writes such a row's local keys in key order, scored 0, without reading logits. - The owned decode top-k takes the op for BF16 decode queries without mandatory windows, up to 4096 requests; other sizes keep `dsa_index_candidates`. ## Test Plan - [x] `tokenspeed-kernel/test/nvidia/ops/attention/test_dsa_decode_index_candidates.py`: 9 passed on GB300 (positions and logits against the DeepGEMM leaf, rows that fit the top-k, short and empty rows, verify windows, strided send buffers, and a batch past the scorer's 4096 requests refused); a position-offset mutant fails it. - [x] Every touched test file and every test file importing a touched module, 44 in all, on GB300, among them `test_dsa_fused_decode_topk.py` (32), `test_cute_dsl_dsa_owned_selection.py` (12), `test_flashmla_dsa_dcp.py` (73), `test_dsa_dcp_owned_decode_topk.py` (2 and 4 ranks), `test_kimi_k31_dcp.py` (14), `test_dsa_dcp_k31_indexer.py` (10), and `test_dsa_dcp_index_reuse.py` (24). - [x] One KVP8 rank's decode candidates, GB300, CUDA graph, top-k 2048, microseconds: | rows | context tokens | DeepGEMM leaf | this PR | |-----:|-----:|-----:|-----:| | 8 | 1.5K (fits the top-k) | 32.5 | 5.2 | | 128 | 1.5K to 2K | 17.6 | 10.6 | | 8 | 16K | 15.0 | 8.7 | | 128 | 16K | 21.0 | 12.0 | | 8 | 64K | 21.2 | 18.2 | | 128 | 64K | 32.5 | 27.8 | - [x] Kimi-K3.1, TP8 with KVP8 on 2x GB300, DSpark with synthetic acceptance 8, 1024 output tokens, steady ms per token, two alternating boots per arm: | batch | main | this PR | change | |-----:|-----:|-----:|-----:| | 1 | 1.294 | 1.223 | -5.5% (boots -9.8% and -1.1%) | | 2 | 1.640 | 1.602 | -2.4% | | 4 | 2.212 | 2.116 | -4.4% | | 8 | 2.942 | 2.859 | -2.9% | | 16 | 4.404 | 4.326 | -1.8% | | 32 | 5.716 | 5.618 | -1.7% | | 64 | 7.925 | 7.786 | -1.8% | - [x] GSM8K, 1000 questions with real acceptance: 959 with this PR, 961 on main (2 and 4 questions right only in one arm, sign test p = 0.69). --------- Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com> |
||
|
|
89315406aa |
perf(dsa): run the head-sharded KVP decode on the Prims-TS core (#3659)
## Summary A head-sharded KVP decode attends every rank's gathered query heads over its own keys and hands the partials and their LSEs to the DCP combine. On SM100 and SM103 it ran the trtllm-gen core, which pads K3.1's 96 gathered heads to 128, computes the LSE in another launch, and needs a mask after it for rows owning no key. The Prims-TS core (`flashinfer_prims`) attends the 96 heads as they are and writes the combine's LSE itself, -inf for an empty row. - `DSABackend` probes the head-sharded KVP decode's solution once its own pin has landed (`_probe_kvp_decode_solution`): `flashinfer_prims` where it serves every decode selection the forward makes (one row per request and the verify rows) on an FP8 cache, without a solution pin or kernel plan of the leaf's own; otherwise the leaf's pin, as before. - The probes, the sparse-leaf log, and the trtllm-gen buffer reservation follow the decode's solution. - Other decodes and every prefill keep their selection. ## Test Plan - [x] `tokenspeed-kernel/test/nvidia/ops/test_dsa_flashinfer_prims.py` on GB300, now with a length-0 row (zero output, -inf LSE). - [x] `test/runtime/test_dsa_backend_selection.py`: the probe pins Prims-TS only where it serves both decode widths, keeps a pin, a plan, a BF16 cache, and a single rank as before; on the real K3.1 selection it pins Prims-TS on SM100/SM103. - [x] `test/runtime/distributed/test_kimi_k31_dcp.py` and every test file building a `DSABackend` double (their decode doubles now carry the solution). - [x] One KVP8 rank's decode core per layer, GB300, 24 layer caches (cold L2), CUDA graph, 256 owned keys per row, microseconds (the same at 2K, 16K, and 64K-token contexts): | rows | trtllm-gen (pad, fmha, LSE, mask) | Prims-TS | |-----:|-----:|-----:| | 8 | 21.0 | 9.8 | | 32 | 29.0 | 9.1 | | 64 | 26.5 | 9.7 | | 128 | 38.4 | 13.0 | | 512 | 117.2 | 30.9 | - [x] Kimi-K3.1, TP8 with KVP8 on 2x GB300, DSpark with synthetic acceptance 8, 1024 output tokens, steady ms per token, two alternating boots per arm: | batch | main | this PR | change | |-----:|-----:|-----:|-----:| | 1 | 1.296 | 1.312 | +1.2% (boots -0.2% and +2.6%) | | 2 | 1.665 | 1.664 | -0.1% | | 4 | 2.187 | 2.160 | -1.2% | | 8 | 2.932 | 2.882 | -1.7% | | 16 | 4.457 | 4.335 | -2.7% | | 32 | 5.708 | 5.641 | -1.2% | | 64 | 7.911 | 7.656 | -3.2% | Batch 1 moves within the boot-to-boot spread. Profiles of one boot each settle it: a DSA layer's attention takes 114.8 instead of 129.7 microseconds at batch 1 (the 24 DSA layers 322 microseconds less per step, the step 1.8% shorter), and a DSA layer 333.1 instead of 355.5 microseconds at batch 8 (538 microseconds less per step, the step 1.9% shorter). - [x] GSM8K, 1000 questions with real acceptance: 964 against main's 956 (13 and 5 questions right only in one arm, sign test p = 0.10). --------- Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com> |
||
|
|
19b0242f23 |
perf(fp8): read block-FP8 activation scales in the quantizer's layout (#3654)
## Summary
On SM10x, every block-FP8 dense linear moved its activation scales into
another layout and back:
- The TRT-LLM quantizer writes MN-major scales, then copies them to
token-major (`scale.t().contiguous()`).
- At `M > 16` the CuTe-DSL GEMM copies them back to MN-major
(`a_scales.T.contiguous()`).
- `fp8_quantize_1x128` also copies on SM10x to drop its 4-row padding.
Each copy is a small PyTorch kernel recorded into every captured CUDA
graph, so a profile is the only place they show up. On
Qwen3.5-35B-A3B-FP8 decode with 32 concurrent requests, they were 320 of
the 441 copy kernels per step.
Each GEMM now declares the scale layout it reads, and the linear layer
quantizes into that layout:
- **Registrations:** block-FP8 GEMMs declare `a_scales_layout`,
`mn_major` for CuTe-DSL and `k_major` for FlashInfer. `mm` derives the
trait from the strides of `A_scales`. A one-row or one-column tensor
fits both layouts.
- **Layout query:** `mm_activation_scale_layout()` reports which layout
the kernel `mm` would select reads.
- **Linear layer:** `Fp8LinearMethod` asks that before quantizing FP32
block scales. For MN-major it uses `trtllm_quantize_fp8_mn_major`
(#2427), the same TRT-LLM kernel returning its native storage.
- **CuTe-DSL launcher:**
- It reads MN-major scales in place. Their `stride(1)` becomes the
kernel's `sf_m` row extent, so the padded native layout needs no copy.
- It raises `ValueError` on any other layout instead of copying.
- It requires contiguous weight scales.
- **`fp8_quantize_1x128`:** returns the padded `[K/128, M]` view on
SM10x, as it already does on other GPUs.
- **Online quantization** (`mm` with `A_scales=None`): still uses the
Triton quantizer, which writes K-major scales. When the selected kernel
reads MN-major, `mm` now transposes them explicitly. This is the same
copy as before, moved to where the layout is chosen.
- **CuTe tuner:** it builds token-major profile scales, so its input
hook converts them to MN-major. Tuning now measures the same copy-free
launch that serving runs.
- **Numerics and benchmark generators:** they build activation scales in
the layout the selected kernel declares.
Kernel selection and numerics don't change: every shape selects the same
kernel, and the quantizer produces the same values.
Not in this PR:
- The FlashInfer path (`M <= 16`) still receives token-major scales, so
it keeps one copy per linear.
- The FP8 MoE still makes its own scale round trip.
Both need changes on the FlashInfer side.
## Test Plan
On GB200, main (
|
||
|
|
ba93b55ab6 |
fix(dflash): guard Iris import on unsupported platforms (#3663)
DFlash startup imports Iris before checking whether the device supports its fused reduction. On Ascend, Iris reads Gluon dtypes at import time while `gl` is `None`, so startup fails before the capability check. Check for CDNA4 before importing Iris. Other devices use the existing unfused path, and CDNA4 retains the existing eligibility checks. Validation: `pre-commit run --all-files` passed. An equivalent guard was exercised on Ascend with ACL Graph capture and a successful chat smoke request. Signed-off-by: ShixianVincent <6891657+VEN-DSX@users.noreply.github.com> Co-authored-by: ShixianVincent <6891657+VEN-DSX@users.noreply.github.com> |
||
|
|
1e433a5b9d |
feat(scheduler): fill the budget a holding prefill chunk leaves (#3255)
## Summary This PR ports and adapts https://github.com/lightseekorg/tokenspeed/pull/2661/changes/7f071ac1cb0c6fb9dc26dfa838f76d30d087d83d to main branch. With `--prefill-chunk-yield-tokens N > 0` (P role), a prefill chunk that cannot finish its prompt gives way to at most N tokens of whole prompts behind it (#2961). The chunk often still leaves budget idle: a junction cut (KV cached past the last State checkpoint) ends it short of where it gave way, and a new prompt cut at its junction never gives way at all. This PR fills that leftover budget with whole prompts that complete in the same round. The fill is P-role behaviour whatever N; N only bounds what a chunk gives way to. ## Behaviour changes (P role; fused and D plans are unchanged) - The once-per-round whole-prompt probe (still at most 8 prompts) keeps prompts up to max(N, budget left). Prompts above N are never given way to, so what the chunk gives way to and where it ends are unchanged; with N = 0 no chunk gives way, as before. - Once the chunk is admitted and keeps the head of line, and the prompts it gave way to are admitted, the remaining budget is filled from the same list by the same `carryWholePrompts` (must_complete; stops at a full batch or the first capacity failure), with N = 0 too. A round that had not probed yet probes after the chunk's admission. A chunk whose admission compacted a lone request (#3065) fills none, as it carries none. - With `--enable-short-prefill-first` (#3179), the fill follows the ranking instead: a protected prompt first, then least remaining work, including prompts submitted before the chunk's own, with no probe limit; a resident chunk ranks them after its admission. The carry keeps #3179's bounded scan in submission order. - Invariants kept: at most one incomplete prefill per round, and it runs last; completing prefills form a prefix of the batch. - Filled prompts hold their pages until their transfer is acknowledged, so in a tight pool the chunk's next admission may wait for those ACKs. They do not depend on the chunk: D allocates destinations before the request is bootstrapped. - Docs: `docs/design/scheduler.md` (§1.1, §1.4, §3.1, §5), `--prefill-chunk-yield-tokens` help, `docs/configuration/server.md`. ## Performance K3 on GB300, 1P1D (P: TP8 KVP8, prefill graphs, chunk 8192), N = 2048 in both arms, one pass each. Measured while the fill still required N > 0; not re-measured with N = 0. **AgentX traffic** (measured on main, before the last rebase; behind the gateway, c64, one 1-hour window per arm, 0 request errors). TTFT in ms. | Arm | Requests | requests per second | TTFT p50 | TTFT p90 | |---|---|---|---|---| | main | 7,386 | 2.029 | 1,794 | 8,086 | | this PR | 7,363 | 2.023 | 1,792 | 7,845 | No measurable difference: arm-to-arm variation on this setup is about 1% in requests per second and 80 ms in TTFT p50. This workload gives the fill little to do. Junctions are 5.0% of first admissions (369 requests per arm), the prefill side is not saturated (about 72% busy; decode is the bottleneck), and a cut chunk rarely has a whole prompt waiting behind it: of 151 cut rounds read from the logs, 3 admitted whole prompts after the chunk, and 1 of them was beyond what N already allows. **Earlier measurement** (main vs this change on it, submission order; D: TP8). TTFT in seconds. | Workload | Arm | requests per second | TTFT p50 | TTFT p90 | P forwards | |---|---|---|---|---|---| | Junction-heavy (8 trunks x 16 forks + short requests, 616 req) | main | 6.662 | 2.323 | 7.238 | 298 | | | this PR | 7.049 (+5.8%) | 2.282 | 6.508 (-10.1%) | 255 | | Agentic (p_bench P workload, c64, 4137 req) | main | 15.504 | 3.759 | 4.272 | 771 | | | this PR | 15.459 (-0.3%) | 3.775 | 4.358 | 760 | Outputs were token-identical across arms (616 of 616 junction-heavy, 124 of 124 junction correctness set, 16 of 16 greedy); gsm8k 0.965 with N = 0 and N = 2048. ## Test plan - Scheduler gtest 1642 passed of 1643, 1 skipped as on main (17 new tests; 15 fail with the fill disabled, and the fused-role test fails if the fill runs on every role), pytest 431 passed, `pre-commit run --all-files`. - Closed-loop plan digests on this head: fused and D roles with N = 0 identical to main on 7160 jobs; P role with N = 0 differs from main on 1222 of 1400 jobs (the fill) and no job raises. Before the rebases onto current main: P role with N = 24 identical to the previous head on 4200 jobs, and randomized P and 1P1D stress with N = 0 in tight pools (17,000 jobs, in submission order and with `--enable-short-prefill-first`) showed no invariant violation and no hang or stall that main does not have. - GB300 K3 1P1D (P: TP8 KVP8, prefill graphs, chunk 8192), before the rebases onto current main, this PR against main with N = 0 on both: outputs token-identical on the 16-prompt greedy set and on the 124-request junction correctness set (two passes each; each build repeated against itself as the control); the fill admitted whole prompts in 54 plans with this PR, 46 of them inside the junction correctness set, and in none on main; 108 concurrent requests, 0 errors. This check ran without speculative decoding: on that base (main, with or without this PR) the K3 decode engines exit during warm-up when DSpark is enabled (`positions and loc must hold 320 rows` in the drafter's context-KV write), a path this PR does not touch. --------- Signed-off-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com> |
||
|
|
496598e4ab |
perf(gdn): optimized FlashInfer backend when ReplaySSM chain is disabled (#3312)
<!-- Headings and layout are optional. Include change context and validation (checks performed or planned, or why testing was not run or is not applicable) anywhere in the description. --> ## Summary With `--disable-replay-ssm` and FP32 state, a GDN chain verify goes through FlashInfer's MTP kernel. Each GDN layer still launches two copy kernels around the conv update: - `pack_gdn_projection` copies the projection views into dense tensors. - `fused_qkv_split_gdn_prefill` copies the conv output into separate q, k and v. #2702 maded the conv update write dense q/k/v rows (`out=`) and ReplaySSM payload (`replay_payload=`). Even after #2583, backend takes that path when the selected verify kernel reads projection views under ReplaySSM. FlashInfer backend couldn't read projection views, so it still does the `pack and split`. This PR makes every GDN chain verify take the conv path: - conv update reads the projection views and writes dense `q|k|v` rows. Under `ReplaySSM` it also fills the payload. - `q/k/v` are just views of those rows. FlashInfer accepts them, since `_qkv_rows_strided` doesn't count them as strided, so it doesn't effect kernel selection. - FlashInfer reads `a`/`b` as projection views. The MTP runner passes them with dynamic layout and no alignment assumption. ## Performance Setup: - B200. - Qwen3.5-397B's GDN layer shapes on one GPU at TP4: 4 key and 16 value heads, dim 128. - MTP 3 (4 verify rows), FP32 state, `--disable-replay-ssm`. - One GDN layer's pack, conv, split and FlashInfer verify, in a CUDA graph. | Batch | main (µs) | PR (µs) | Speedup | Saved per decode step (×45 GDN layers) | |---|---|---|---|---| | 1 | 10.23 | 7.88 | 1.30× | 106 µs | | 2 | 11.69 | 8.93 | 1.31× | 124 µs | | 4 | 14.07 | 12.77 | 1.10× | 58 µs | | 8 | 17.66 | 14.62 | 1.21× | 136 µs | | 16 | 24.05 | 23.88 | 1.01× | 8 µs | Qwen3.5-397B's CI perf and eval configs serve with `--disable-replay-ssm`. ## Tests - `test_flashinfer_mtp_reads_conv_rows_and_projection_gates`: FlashInfer's FP32 verify conv-row q/k/v and projection-view a/b, including 16-byte gate offsets, equals to the inputs. Covers the inline and warp-specialized kernels, read-only and per-token state writes. - `test_chain_verify_conv_rows_match_the_split`: through the backend, the conv-rows path matches the split path (output and committed states), for BF16/FP32 state with ReplaySSM on and off. - The existing `test_gdn_backend_reads_projection_views_like_dense_inputs` (`GdnAttnBackend` with FlashInfer, ReplaySSM off, misaligned `a`) now covers the new path. Signed-off-by: shikhar <shikhar2807.ace@gmail.com> Co-authored-by: Tuan Zhang <tuan@lightseek.org> |
||
|
|
c20ea8c988 |
feat(pd): let a prefill compute before its decode allocation (opt-in) (#3394)
## Summary
`--disaggregation-decode-allocation-policy deferred` (prefill servers,
experimental): a prompt is computed without waiting for its decode node
to allocate the destination; only its KV transfer waits. The default,
`early`, keeps today's admission and is marked deprecated in the docs
and the flag's help.
Replaces #3218 with the ordering proposed in its review: a request takes
every page it needs on the prefill node before its decode node
allocates, so the two nodes cannot wait for each other and nothing is
requeued or computed twice. Based on
|
||
|
|
94397ff099 |
refactor(comm): add gather and scatter projections for owner rows (#3651)
## Summary Owner-row projection workspaces (`comm_backend/projection.py`) served the DP column and row layouts only, so Muse Spark's owned rows on NVIDIA kept their own copy-engine gather and NCCL collectives in the model. This adds two layouts beside DP column and row, and moves that communication behind them: - `gather`: every owner's rows through this rank's output shard. `comm_ops.projection_gather_mm` gathers and projects them; the backend picks Lamport for the rows it holds, copy-engine pulls overlapped with per-owner GEMMs where the group maps its memory, and its ordinary all-gather and one GEMM otherwise. - `scatter`: every rank's partial of all rows reduced into owner rows, through the existing `acquire_projection_output` and `projection_reduce_scatter`. The copy-engine gather writes this rank's rows into symmetric memory, passes a barrier, and pulls peers' rows on a side stream while its own rows project. A second barrier after the pulls, which the gather joins before returning, lets the next gather rewrite the rows, so gathers carry no state between calls (the previous parity scheme relied on every forward gathering an even number of times). Muse Spark prepares four workspaces (gathers for Q, K and V and for the pre-expert projection, the output-projection scatter and the post-expert column exchange) on one scratch block and calls the shared ops. `_CopyEngineRows` and `_NcclOwnedRows` are removed, and the owned path no longer needs a model-side peer-mapping gate: groups the copy engine cannot reach take the ordinary path. ## Results GB200 x4, TP4 and EP4, NVFP4; main vs this PR, two A and B pairs on one node. Prefill only (2 output tokens), mean TTFT ms: | prompt x concurrency | main | this PR | |---|---|---| | 4k x 1 | 117 and 119 | 120 and 120 | | 8k x 1 | 206 and 205 | 202 and 202 | | 8k x 16 | 1579 and 1582 | 1584 and 1586 | | 1k x 128 | 1529 and 1521 | 1535 and 1544 | 1k input and 1k output tokens, output tokens per second and mean TTFT ms: | concurrency | main | this PR | |---|---|---| | 64 | 3646, 913 and 3675, 863 | 3648, 876 and 3646, 864 | | 128 | 5346, 1561 and 5320, 1584 | 5325, 1596 and 5291, 1686 | | 256 | 7892, 3012 and 7916, 2977 | 7890, 3003 and 7901, 3004 | GSM8K, 512 questions at concurrency 32: main 0.982 and 0.979, this PR 0.980 and 0.979. The second barrier is the cost of carrying no state: isolated on 4 GB200s with ranks in step, a gather plus its GEMMs takes 167 -> 173 us at 1000 rows per rank and 326 -> 335 us at 2048, about 1% of a 4k prefill. Greedy outputs of six 20-shot GSM8K prompts (about 3.3k tokens, so every forward takes the owned path) are identical to main. ## Tests - `test/runtime/distributed/test_owner_row_projections.py`: on 2 and 4 GPUs, with the auto and NCCL backends, gathers and scatters match gathered and reduced references for one row, Lamport-sized and copy-engine-sized batches with unequal owner rows; three captured gathers replay on new rows. - `test/runtime/distributed/test_muse_owned_projections.py` (renamed from `test_muse_owned_rows_nccl.py`): Muse's four owned projections match references for odd row counts, eagerly and as two captured layers. - `test/runtime/models/test_muse_spark.py`, `test_muse_spark_iris.py`: stubs use the renamed field. Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com> |
||
|
|
f0fbe27ec6 |
perf(dcp): merge and project K3.1's KVP decode partials in one CuTe DSL launch (#3645)
## Summary A KVP decode's head-sharded combine packed every rank's partials into records, exchanged them, merged them in another launch, and projected the latent values in a third. For Kimi-K3.1's owned decode top-k, one CuTe DSL launch (`DcpValue`) now does all three. - `DcpValue` (`k3_decode/mla/dcp.py`) pushes each rank's BF16 partials and LSEs to their head owners through Lamport mailboxes in symmetric memory, merges them exactly as `combine_dcp_attention_partials` does and projects them as `cute_dsl_mla_project_value` does, with the optional sigmoid gate. Its output is bit for bit the records all-to-all, combine, and projection. - `create_dcp_value_tail` builds a group's tail collectively (SM100 or SM103, 2, 4, or 8 ranks on one multicast fabric); every refusal is voted on, so all ranks keep the unfused path together. Its mailbox stays under 2^31 elements (the merge's int32 offsets), and every in-kernel wait traps at the group's timeout, as the LL exchange's do. - `prepare_decode_exchange` prepares a tail for the attentions that project as they merge (K3.1's sparse MLA). The sparse decode then hands the value projection (`ValueProjection`) to `combine_project_attention_partials`, which runs the tail when it serves the call and the combine plus `mla_project_value` otherwise. ## Test Plan - [x] `tokenspeed-kernel/test/nvidia/ops/attention/test_cute_dsl_dcp_value.py`: 2 and 4 ranks, bit for bit against the records all-to-all, combine, and projection over changing batch sizes, gated and ungated, negative zeros, NaN and infinite LSEs, and CUDA graph replays with new inputs; voted refusals (one ineligible rank, no timeout, an oversized mailbox) and a trap within the timeout when a peer never calls. - [x] `test/runtime/distributed/test_kimi_k31_dcp.py` (14 passed): with the tail prepared, the owned decode selects and attends like one rank holding the history, eager and replayed, and the tail runs. - [x] Test files that import a touched module pass as on main. - [x] Kimi-K3.1, TP8 with KVP8 on 2x GB300, DSpark with synthetic acceptance 8 (mean of two fresh boots, ms/token): | batch | 1 | 2 | 4 | 8 | 16 | 32 | 64 | |---|---|---|---|---|---|---|---| | main | 1.368 | 1.678 | 2.238 | 2.950 | 4.406 | 5.730 | 7.931 | | this PR | 1.360 | 1.677 | 2.221 | 2.872 | 4.305 | 5.607 | 7.713 | | change | -0.6% | -0.0% | -0.7% | -2.6% | -2.3% | -2.2% | -2.7% | - [x] GSM8K, 1000 questions with real acceptance: 960 with this PR, 956 on main (9 answers right only with this PR, 5 only on main; sign test p = 0.42). --------- Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com> |
||
|
|
1983549040 |
perf(moe): run Kimi-K3 NVFP4 SiTU experts on FlashInfer's CuTe DSL MoE (#3588)
## Summary
Ports
|
||
|
|
943bef04fb |
fix(memory): scope expandable segments around memory saver operations (#3640)
Enabling expandable CUDA segments together with memory saver can fail initialization or bypass the allocator that tracks managed storage. This change temporarily disables expandable segments during managed weight/KV and persistent-buffer allocations, CUDA graph capture, and pause/resume. It restores the effective allocator configuration and both environment aliases afterward, including on exceptions and when the last configuration string omits the active expandable setting. The allocator setting is process-wide. The adapter serializes its own scopes, and callers must keep unrelated CUDA allocation threads idle during startup or drained maintenance. Validation: - `pre-commit run --all-files` passed. - H200 with PyTorch 2.9.0 and `torch_memory_saver` 0.0.10: 19 passed, 1 skipped. The skipped case uses `PYTORCH_ALLOC_CONF`, which does not enable expandable segments in the tested Torch build. - GPU tests cover expandable allocations outside managed scopes, weight/KV/buffer isolation, graph replay, three pause/resume cycles, exception and sticky-setting restoration, and normal process exit. --------- Signed-off-by: lyzuu <349859374+lyzuu@users.noreply.github.com> Co-authored-by: lyzuu <349859374+lyzuu@users.noreply.github.com> Co-authored-by: longcat-ia-team <312376714+longcat-ia-team@users.noreply.github.com> |
||
|
|
fcf48aea6c |
refactor(models): share the DeepSeek-layout NextN draft loader (#3648)
## Summary
The DeepSeek NextN draft (`deepseek_nextn.py`) and the GLM-MoE-DSA NextN
draft (`glm_moe_dsa_nextn.py`, which `DeepseekV32ForCausalLMNextN`
reuses) each carried a copy of the same code:
- the NextN model's `__init__`;
- the checkpoint-name mapping for the NextN layer;
- the `load_weights` loop: stacked MLP, MoE experts, the fused q/kv
a-projection pair, and plain tensors;
- the q/kv a-projection pairing state and its
`begin`/`abort`/`end_weight_update` hooks.
They now share it:
- `DeepseekModelNextN` builds its decoder from a `decoder_cls` class
attribute. `GlmMoeDsaModelNextN` subclasses it, sets `decoder_cls =
GlmMoeDsaDecoderLayer`, and keeps its own `forward`.
- A new `DeepseekNextNWeights` mixin in `deepseek_nextn.py` holds the
name mapping, the loader loop and the three update hooks. Both draft
classes list it first.
- Per-model behavior stays in each class:
- **DeepSeek** fails on a parameter name it doesn't know. **GLM** looks
it up with `get_param` and skips it if missing. A small `_nextn_param`
hook keeps both behaviors.
- **GLM's `load_weights`** loads its indexer weights as they stream by
and passes every other tensor on to the shared loop, in the same order
as before.
- Each model keeps its own `forward`, LM head,
`checkpoint_weight_name_filter` and `post_load_weights`.
**One behavior change:** a checkpoint name with a non-numeric layer
index, such as `model.layers.x.mlp.gate.weight`, used to make the
DeepSeek draft raise on `int()`. It is now skipped, as the GLM draft
already did. Well-formed names map the same way as before.
No model forward code changes. 172 fewer lines.
## Test Plan
On GB200, main (
|
||
|
|
3786263e1b |
refactor(npu): rework Ascend attention metadata, Prefill workspaces and MoE execution contracts (#3618)
## Scope Refactor Ascend runtime attention, cache workspace preparation and MoE execution contracts. This consolidates #3498, #3573 and #3596 on top of #3563. - Separate DSA decode resources from shared context-parallel metadata, and reuse the metadata lifecycle across DSA and MLA. - Share recipe-sized QCP Prefill workspace preparation and transport across attention backends, with explicit layout and capacity contracts. - Centralize scheduler-derived MoE source counts and refresh the graph-owned buffer before replay. - Clarify W4A4 weight preparation and common-registry ownership, while retaining FP32 normalization scales for BF16 inputs. Model plugins select their numerical execution paths through these runtime interfaces. Sampling and logprob reduction retain the existing runtime policy. ## Review fixes - Reject DP-local decode graphs for models requiring global MoE source counts; use the communication layer's scatter calculation. Startup profiling uploads the selected packed tier's row count and restores capture metadata, including on failure. - Keep persistent CUDA count buffers in the memory saver's CPU-backed runtime region. Cover repeated release/resume, contents and addresses. Register the packed profiling regression in CUDA CI. - Pass cache-recipe workspace layouts through the shared QCP Prefill preparation path, removing the NPU package's reverse dependency. Restore transport input validation and cover dynamic metadata parameters without recompilation. - Make W4A4 layout choices explicit, update the common registry regression, and test fused DSA operator rejection at binding time. Restore KVP 2/4/8/16 LSE coverage, exact FP32-scale sensitivity, invalid FlatQuant clip checks and both cache-probe sizing cases. - Update the scheduler test fixture for the cache-storage contract reported by NVIDIA and AMD CI, and refresh the renamed NPU metadata test entry point. The existing DSA constructor's optional decode flag remains compatible with callers that previously supplied only `prefill_enabled`. Low-level packing helpers retain their separately tested geometry; the model support constraints remain unchanged. ## Validation CPU contract tests passed: 413 runtime/kernel tests and a follow-up selection of 126 runner tests (with overlap). `pre-commit run --all-files` passes. Ascend 910B: 297 metadata graph/stream regressions, 25 transport-kernel cases, 10 normalization cases and 8 real INT4 NZ/GMM cases passed. The two-rank QCP transport run passed 34 byte-exact scenarios per rank, covering KV-only and KV+index layouts, empty owners, shared pages and fragmented history. The two scheduler fixture regressions also passed with the matching engine package. CUDA packed profiling regressions passed on NVIDIA B200 and AMD MI350. The attention contract follow-up passed 160 CPU tests and 25 subtests; `pre-commit run --all-files` passes. --------- Signed-off-by: zhuhaozhecool <zhuhaozhecool@163.com> Signed-off-by: 7huahua <26763681+7huahua@users.noreply.github.com> Co-authored-by: 7huahua <26763681+7huahua@users.noreply.github.com> |
||
|
|
a33c5a1e9a |
perf(muse): overlap owned-row gathers with their projections on the copy engine (#3644)
## Summary At >= 4096 tokens, Muse Spark on NVIDIA keeps the residual stream row-sharded, and each layer gathered its owned rows over NCCL twice before a column projection: the attention input before the Q, K, and V projection, and the pre-expert input before the pre-expert projection. At 8192 rows each gather moved 128 megabytes and ran before its GEMM could start. The gathers now run on the copy engine and overlap the GEMMs: - `_CopyEngineRows` keeps every rank's owned BF16 rows in symmetric memory, in two parities. A gather writes this rank's rows, passes a barrier, and pulls the peers' rows on a side stream. - `_NcclOwnedRows.all_gather_mm` projects this rank's rows while the peers' rows are in flight, then projects each peer's rows as they land. The SMs stay on the GEMMs. - Layers pass owned attention rows to the next layer, and layer zero projects gathered owned rows too. The Iris path is unchanged. - `prepare_communication_runtime` sizes the workspace from the runtime's maximum forward tokens and binds it before cache sizing and graph capture. Each rank's rows go through the same `mm` as before. At these shapes (4096 to 16384 rows; 4608 and 1024 columns) the per-rank products are bitwise equal to one product over all rows, so outputs match the NCCL gather followed by one projection. Isolated on 4 GB200s at 8192 rows: the Q, K, and V gather plus GEMM went from 610 to 452 microseconds, and the pre-expert one from 315 to 197 microseconds. ## Results 4 GB200s, TP4 and EP4, NVFP4, random prompts with a distinct seed per run; two A versus B pairs on one node. Prefill only (2 output tokens), mean TTFT in milliseconds: | prompt x concurrency | main | this PR | |---|---|---| | 1k x 1 | 58 and 57 | 57 and 59 | | 4k x 1 | 127 and 124 | 120 and 120 | | 8k x 1 | 222 and 217 | 210 and 205 | | 8k x 16 | 1714 and 1724 | 1577 and 1577 | | 1k x 128 | 1660 and 1665 | 1523 and 1527 | 1k input and 1k output tokens, output tokens per second and mean TTFT in milliseconds: | concurrency | main | this PR | |---|---|---| | 64 | 3619, 932 and 3638, 934 | 3648, 864 and 3632, 906 | | 128 | 5256, 1699 and 5228, 1798 | 5341, 1589 and 5327, 1589 | | 256 | 7741, 3258 and 7783, 3238 | 7876, 2989 and 7923, 2967 | GSM8K, 512 questions at concurrency 32: main 0.979 and 0.977, this PR 0.975 and 0.969 (one standard deviation is about 0.007). ## Tests - `test/runtime/distributed/test_muse_owned_rows_nccl.py`: on 2 and 4 GPUs, `all_gather_mm` matches the NCCL gather followed by `mm` for odd and even row counts, and a captured pair of gathers replays with new rows. - `test/runtime/models/test_muse_spark.py`, `test_muse_spark_iris.py`: model stubs declare the new field. --------- Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com> |
||
|
|
1df4f4ba28 |
refactor(attention): pass extend_with_prefix to the MLA prefill hook (#3614)
## Summary
Follow-up to #3605. FlashMLA's prefill hook read `extend_with_prefix`
from a field that only its `init_forward_metadata` wrapper set. A direct
call to the hook, as the dots3 tests make on other leaves, therefore
reused the previous forward's value. A stale `True` on a prefix-free
batch planned paged prefill instead of ragged, with no error.
`extend_with_prefix` is now an argument of the shared
`_init_prefill_metadata` hook:
- `PagedMLABackend.init_forward_metadata` passes it to the hook.
- `FlashMLABackend` reads it from the argument. Its wrapper and the
`_extend_with_prefix` field are removed.
- The MLA, CuTe DSL MLA and TRT-LLM MLA hooks don't use it. They `del`
it with a one-line reason.
- `Dots3NoteSWABackend` passes it on to the MLA hook. Shifting the
prefix into the window keeps a prefixed request prefixed, so the flag
stays true.
- `AscendMLABackend` passes the hook's keywords on when its own prefill
is off. When it is on, its plan reads its own host table and the prefix
lengths, so it now drops `page_table_cpu` and `extend_with_prefix`
before building `MLAPrefillMetadata`, which has neither field. That path
already raised `TypeError` on `page_table_cpu` before this PR.
Callers that invoke the hook directly now pass the flag:
- the dots3 cache tests;
- `run_cutedsl_mla_dcp.py`;
- the two NPU callers, `test_ascend_mla_prefill.py` and
`run_ascend_mla_lifecycle.py`. These also pass their host page table,
which they had been missing, so they failed on main too.
Server forwards get the same flag value as before. No kernel code
changes.
## Test Plan
On GB200, main (
|
||
|
|
71f4039605 |
feat(metrics): add request-level speculative accept length (#3641)
## Summary - Add `tokenspeed:request_spec_decode_accept_length`, with one mean acceptance-length sample per ended speculative request, including bonus tokens. - Drain request samples through the Rust control plane and mirror them in the Python and Prometheus reporters. - Keep the existing verify-slot-weighted `spec_decode_accept_length` metric unchanged and document the distinction. ## Tests - `pre-commit run --all-files` - Python metric reporter tests covering request MAL, speculation counters, and weighted observations: 3 passed. - Python syntax checks for the modified collector, reporter tests, and exposition generator. Rust tests were not run locally because no Rust toolchain is available in this environment. Signed-off-by: wangbo981016 <wangbo981016@gmail.com> |
||
|
|
415a87aee3 |
fix: prepare Iris all-reduce before short prefills (#3638)
The [AMD GPT-OSS GPQA job](https://github.com/lightseekorg/tokenspeed/actions/runs/44522750854/job/135581030558) exits when a short prefix-cache hit first selects Iris all-reduce. Startup's large batches use RCCL, leaving ordinary Iris state unprepared; initializing it during serving synchronizes the device and trips the data-plane sync guard. Prepare ordinary reductions after model-specific communication setup and before KV memory profiling. Execution groups follow the same role order on every rank so overlapping groups retain symmetric heap offsets, and preparation reuses existing model capacities. A registered AMD regression covers large-to-small batches under sync-debug error mode, repeated preparation, and graph replay. The communication test fixture now uses the real backend constructor. Also install `prometheus-client` for the [Rust engine CI metrics tests](https://github.com/lightseekorg/tokenspeed/actions/runs/44516690947/job/135534552347) and document that test dependency. The distributed GPU tests now bind NCCL to each worker's device before symmetric-memory rendezvous. EAGLE4 uses `spawn`'s exception transport directly, removing a Manager that forked after Torch started threads. The MegaMoE graph test also prints live rank stacks after three minutes and has a ten-minute command limit; this exposed a repeated stall in PyTorch's symmetric-memory rendezvous. Validation: - Related local tests: 377 passed; AMD-only tests require the AMD CI runners. - EAGLE4 TP2 test: three independent local runs passed after removing the Manager. - `cargo test --locked`: 642 passed for the unchanged native engine code. - `pre-commit run --all-files`: passed. - Targeted AMD GPT-OSS GPQA: 594 samples, 69.4% (original threshold 69%), no data-plane sync failures. NVIDIA GPT-OSS also passed. - Distributed GPU regression, the remaining post-submit-only model cases, and PR CI are running on the updated branch. --------- Signed-off-by: FC-Li <robert_lexis@163.com> |
||
|
|
23d32440b5 |
feat(cache): pack FlatKV per Qwen4-Exp pipeline stage (#3564)
## Summary Stacked on [#3389](https://github.com/lightseekorg/tokenspeed/pull/3389). Pack one FlatKV plane per Qwen4-Exp pipeline stage. Compute each cache group's packing from the stages that own its fields. The stages share logical block IDs, and the last stage owns the draft fields. For Qwen3.8-Flash-Next at TP2 with BF16 KV, FP32 GDN state, and MTP, the PP2 plan uses 28 MiB per parent per GPU on both stages. Stage 0 packs 128 PLE blocks per parent. With a 150 GiB cache budget per GPU and the int32 slot limit applied, the plan admits 4,681 parents at PP2 and 2,372 at PP1. End-to-end PP2 serving still requires Qwen4-Exp model forward support. ## Test Plan - Runtime regression suite with 351 passed and 7 skipped, covering stage ownership, arena addressing, and target and draft int32 slot limits. - PP2 planning with the Qwen3.8-Flash-Next checkpoint configuration and PD wire and address checks on both stages. - Fresh B300 TP2 and PP1 serving with MTP, forward state checkpoints, and L2 `write_back`. The short request and both long requests returned the expected answer. The repeated prompt reused 17,920 of 18,029 tokens. - `pre-commit run --all-files` --------- Signed-off-by: tuanzhangCS <tuan@lightseek.org> Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> |
||
|
|
6103eb3953 |
fix(deps): require tokenspeed-mooncake 0.3.13.post20261119 (#3637)
## Summary TokenSpeed still accepts Mooncake releases whose native C Store library and Python Store binding abort on duplicate gflags registration when loaded together. Require `tokenspeed-mooncake>=0.3.13.post20261119`, which shares one Store core between the bindings, and update the cu129 install constraint to the same version. Document the shared core required for native L3 clients and Python Store clients to coexist. The release is available on [PyPI](https://pypi.org/project/tokenspeed-mooncake/0.3.13.post20261119/) and both [cu129](https://lightseek.org/whl/cu129/tokenspeed-mooncake/) and [cu130](https://lightseek.org/whl/cu130/tokenspeed-mooncake/) wheel indexes. ## Test Plan - Downloaded the published cu129 and cu130 CPython 3.11 x86_64 wheels and verified their SHA-256 hashes against the indexes. - Installed each variant into a separate environment. Both C-first and Python-first loading passed bidirectional TCP batch reads/writes and three buffer registration/client teardown cycles; the surviving client remained healthy after its peer closed. - Dependency checks passed for both installed variants. - `pre-commit run --all-files` passed. Signed-off-by: FC-Li <robert_lexis@163.com> |
||
|
|
5a8b323430 |
feat: support SM90 peer expert weights and breakable prefill graph replay (#3574)
DWDP currently requires fabric handles and rejects model CUDA graphs. Add explicit node-local POSIX FD transport on SM90 and newer GPUs, and keep peer-weight prefetch, acquire, and release as eager breaks around captured MoE compute so staging advances on every prefill replay. The copy worker accepts CUDA 12.8 and CUDA 13 batch-copy signatures. MoE kernels can declare that their weight preprocessor supplies the existing peer-weight manifest. Validation: - `pre-commit run --all-files`. - 71 focused CPU tests for ownership, FD exchange and cleanup, mapping and CLI validation. - Two H200 GPUs: each rank passed 2 synthetic graph-replay cases (3 and 4 layers, full VMM pages, changing inputs, exact comparison) and 4 POSIX storage and cleanup cases. - The graph regression test fails with the original manager and passes with this change. --------- Signed-off-by: wangbo981016 <wangbo981016@gmail.com> |
||
|
|
32d0de3de8 |
feat(kimi-k3): serve EP sizes that do not divide the experts (#3524)
## Summary
Ports
|
||
|
|
a2be7880e2 |
feat: metrics for peer prefix sharing (#3623)
<!-- Headings and layout are optional. Include change context and validation (checks performed or planned, or why testing was not run or is not applicable) anywhere in the description. --> ## Summary <!-- What changed and why? --> - Add peer_consumed_tokens and peer_estimation_error_tokens to measure actual peer-prefix reuse per request and the router’s estimate minus actual reuse, grouped by outcome. - Add peer_stage_seconds to measure latency and count completed visits by protocol stage and outcome. Add peer_hard_pin_remaining_seconds to record how much lease time remains when a hard pin is released or expires. - Add peer_shard_seconds to measure device-fence wait, READ queue wait, and transfer time per reader rank and holder. Add peer_read_bytes_total for successful READ payloads and peer_outstanding_read_bytes for bytes queued or in flight. - Add peer_scheduler_state to track active leases, pinned blocks, and requests in lookup or landing. Add peer_scheduler_events_total to count landed tokens and failed landings. ## Test Plan <!-- How was this validated? --> - [x] Ran cargo fmt --all --check, cargo clippy --all-targets --locked -- -D warnings, and cargo test --locked across the Rust workspace. - [x] Verified successful reuse, lookup misses, incompatible holders, and lookup timeouts with engine metrics enabled and transfer metrics disabled. Checked Prometheus output for token attribution, prediction error, READ bytes, and outstanding bytes after completion and cancellation. - [x] Ran all 15 C++ peer-export tests, including hard-pin release and expiry. --------- Signed-off-by: lvyongkang <kangguangli@hotmail.com> Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com> Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> |
||
|
|
0e39ed17af |
fix(dspark): keep the native query extent at every capture forward (#3589)
## Summary - Restore `ctx.input_num_tokens` at the top of every decode-graph capture run, so the grammar fill records a non-empty verify-candidate copy in shortened graphs. - Pin the contract with a regression test over the sequence a capture performs (four warmups and the capture over one context). Packed verification narrows `ctx.input_num_tokens` for the rest of the forward that selected it, because sampling and the NaN audit consume its packed target rows. A capture runs four warmups and the capture itself over a single context, so every run after the first started from the narrowed extent. The grammar fill slices this round's verify candidates out of the live input buffer by that extent (`ModelExecutor._forward_step`), so the captured copy read an empty slice. Every replay of a shortened graph then filled its masks by walking the draft tokens a previous forward had left in the pinned buffer: the first row still followed the request's matcher, the rows behind it were reset to allow every token, and verification accepted drafts the grammar forbids. ## Tests - `pre-commit run --all-files` - Mutation check: dropping the reset fails the new test with `assert [12, 8, 8, 8, 8] == [12, 12, 12, 12, 12]` — the first run keeps the native extent and every later one inherits the narrowed one. - `test/runtime/test_adaptive_dspark_contract.py test/runtime/test_capturable_grammar_host.py test/runtime/test_cudagraph_probe_ladder.py test/runtime/test_cudagraph_memory_projection.py test/runtime/test_prefill_graph.py test/runtime/test_qsa_verify_lifecycle.py test/runtime/sampling/test_packed_dspark_sampling.py test/runtime/test_slot_state.py`: **258 passed**. - DeepSeek-V4.1-Flash 1P1D on 8xB300 (prefill data parallel 4 and expert parallel 4, decode data parallel 4 and expert parallel 4 with sampled adaptive DSpark and DP-local CUDA graphs, xgrammar, `--dsv41-dspark-min-batch-size 4` so shortening engages at low concurrency). Before: greedy `json_schema` returned malformed JSON in 24-29 of 32 concurrent requests, and 128-per-case thinking requests validated in 25-75% of cases. After: `json_schema` and `json_object` validate in **128 out of 128 requests per case** at T=1.0 and T=0, with `budget decisions` reporting shortened verification on 64-81% of steps. Rejected completions replayed through a fresh matcher confirm the first forbidden token sat behind an otherwise valid prefix, consistent with masks filled from stale drafts. - Unrelated to grammar, 340 greedy prompts (GSM8K, AIME, 15-65K-token code contexts) show the same divergence profile as a run without shortening: every first divergence is a near-tie and none is confident on both sides. Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com> --------- Signed-off-by: wangbo981016 <wangbo981016@gmail.com> |
||
|
|
697e842eb1 |
feat(qwen4-exp): use per-tensor FP8 for online PLE quantization (#3575)
## Summary When `ple_embed_dtype` is `float8_e4m3fn`, Qwen4 now quantizes each BF16 PLE table with one BF16 scale shared across all rows and ranks. This matches the scalar-scale layout of offline FP8 PLE checkpoints and removes the per-row scale buffer from online quantization. For a split BF16 table, each rank scans its own checkpoint rows. A max reduction across the table's ranks gives `scale = BF16(max(abs(W)) / 448)`. The regular weight loader then quantizes each shard with that scale. The prescan supports standard single-file and indexed safetensors checkpoints for both target and MTP models. ## Test Plan - On B300, all 11 focused PLE tests passed on the final code. Coverage included target and MTP loading, scale recomputation on reload, all-zero tables, offline FP8 weights, host and device lookup agreement, and errors for unsupported layouts or a missing PLE table. - Before the final review fixes, 60 tests and 5 subtests passed across PLE quantization, disk weight updates, and MTP checkpoint filtering. --------- Signed-off-by: minedec <jiandong@lightseek.org> Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> |
||
|
|
c937127aa8 |
fix(pd): start the decode first token at the first local output (#3627)
## Summary The decode role used the remote KV landing as `first_token`, so a request that already had its KV but waited for a decode execution permit (the E-queue with `--disaggregation-decode-extra-slots`) counted that wait as decode time. `decode_tps` and TPOT therefore mixed admission queueing into the decode rate, even though the first output is streamed only at the first local decode commit. This change starts the decode window at the role's first local decode output: - `first_token` is stamped at the first local decode commit; the KV landing only stamps `prefill_done`. - Requests ended by the bootstrap token (no local decode) stamp it at their terminal stream. - `ttft_ms` and `tokenspeed:time_to_first_token_seconds` now include the permit wait; `decode_tps`, `time_per_output_token_seconds`, and the OTLP `tpot_ms` exclude it. - `decode_wait_ms` remains the KV landing to first decode dispatch interval and is documented as a sub-interval of TTFT. - Updated the metric help text, the server configuration, tracing, event-loop, and scheduler docs, and the metrics exposition fixture. ## Tests - `cargo test -p tokenspeed-control --lib` (203 passed) - `cargo test -p tokenspeed-control --test request_logs` - `cargo test -p tokenspeed-metrics` - `pre-commit run --all-files` --------- Signed-off-by: wangbo981016 <wangbo981016@gmail.com> |
||
|
|
b6221f7448 |
fix(moe): migrate FlashInfer all-to-all to the moe_ep API (#3634)
<!-- Headings and layout are optional. Include change context and
validation
(checks performed or planned, or why testing was not run or is not
applicable)
anywhere in the description. -->
## Summary
Use the NVLink one-sided communication backend in FlashInfer 0.7.4,
which owns the workspace and the dispatch and combine state. Remove the
obsolete weights_dtype argument and construct host-side layouts under a
CPU default device.
Adapted from
|
||
|
|
a6dc79402b |
feat(scheduler): let the decode role hold peer prefixes and a Host tier (RFC 0005 phase 1) (#3550)
Decode replicas retain both a session's prompt and the reply they generated. Letting them serve peer prefixes allows the next turn's prefill replica to reuse that reply. This adds RFC 0005 and implements its phase 1. - Allow decode replicas to hold peer prefixes and use Host write-back. Decode replicas ignore holder hints and continue receiving missing prompt data from their prefill peers. - Probe Device history followed by whole-page Host suffixes. Load the local Host hit below the PD landing; reuse State checkpoints only at the exact prompt end on Device. - Return decode history probes explicitly and consume the current `HostSuffixProbe` interface without output parameters or copying the Device hit arrays. - Fence PD receives and peer reads behind page preparation and Host loads using separate stream events. Wait after launching the current forward so its layerwise load overlap is preserved. - Bind the peer endpoint's listener up front and move it into `Peer::start`, so the first member owns the bound port continuously through startup, including an OS-selected port, while the replica's name and scheduler are prepared. - Retain bounded concurrent peer reads and pin validation from the merged peer-sharing implementation; update comments, design documentation, and regression coverage. Validation: - Full scheduler C++ suite: 1,596 passed; 1 AddressSanitizer-only test skipped in the Release build. - Rust scheduler, control, and host test suites passed. - Device-handle, peer-sharing, and rank-setup Python tests: 186 passed. - Restoring the old full-load barrier makes both concurrent-forward regression cases fail (PD and peer); the corrected implementation passes. - `pre-commit run --all-files`. Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com> --------- Signed-off-by: FC-Li <robert_lexis@163.com> Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com> Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> |
||
|
|
1ff9d0ba97 |
fix: preserve host preloads and tokenizer specials (#3631)
Nested allocator contexts can repeat an inherited LD_PRELOAD entry. Deduplicate the child environment while retaining the first occurrence and library order, leaving the parent environment untouched. Extend the existing real-child-process test to cover inherited and repeated entries. Main already captures the environment inside the startup context; this change preserves that behavior. Special-token reconciliation now passes the append flag positionally, avoiding a dependency on its renamed keyword across tokenizer versions. A real PreTrainedTokenizerFast regression covers preserving existing specials and vocabulary IDs, repeat calls, and decoding with special tokens removed. Validation: - Based on main |
||
|
|
c24bc5545f |
feat(dwdp): share MXFP4 SiTU expert weights across prefill ranks (#3626)
## Summary
Publish processed MXFP4 expert weights, scales and biases through the
peer-weight manifest while replicating activation parameters. Allow
MXFP4 SiTU experts on the FlashInfer TRT-LLM backend to use DWDP.
Previously the FlashInfer TRT-LLM MXFP4 preprocessor published no
peer-weight manifest and the runtime peer-weight gate accepted only
NVFP4, so MXFP4 experts could not be shared across prefill ranks. The
preprocessor now returns a `PreparedExpertWeights` manifest that streams
the processed weights, scales and biases while replicating the SiTU
activation parameters (`gemm1_alpha`, `gemm1_beta`,
`gemm1_clamp_limit`), and the gate admits MXFP4 with the SiTU
activation. MXFP4 experts store no input scale; the kernel quantizes
their activations on each forward. Serving MXFP4 with DWDP requires
`--moe-backend flashinfer_trtllm`, since `auto` may select another
kernel. The DWDP design and serving docs are updated to match.
Adapted from
|
||
|
|
af43d67c56 |
fix(metrics): align request, cache, and transfer observations (#3621)
## Summary - Distinguish active and resident cache occupancy, scheduler waits, and prefill-inflight requests. - Preserve independent completion-token, transfer-size, per-output TPOT, and request e2e_TPOT samples. - Complete transfer-failed requests through the common metric path and exclude cancellation propagation from transfer failures. ## Results - Output-chunk timing excludes the first chunk and empty outputs; single-chunk requests do not create timing samples. - Weighted observations retain correct totals while individual observations preserve their distributions. - Multi-rank cancellation and repeated failure notifications no longer inflate transfer-failure counts. ## Test plan - pre-commit run --all-files - cargo fmt --all --check (tokenspeed-engine) - 62 output tests, 101 executed PD tests, and 69 serving tests passed; 14 PD tests required an unavailable test-library configuration. - Validated 21 successful requests and one intentional stream cancellation, including single-token boundaries and drained request gauges. --------- Signed-off-by: dingyuqing05 <13126694060@163.com> Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com> |
||
|
|
6a5b9947e4 |
fix(sampling): key chain MTP randomness by token position (#3490)
## Summary FlashInfer chain verification used per-slot generators that advanced during prefill preparation. Under `--sampling-stream per-request`, FlashInfer and FlashInfer Full now derive accept and final-draw coins from the request seed and the first target position in the verify round. The two kinds of draws use distinct counters. Sampled draft proposals use the accepted frontier after the round and the draft step. The verifier retains its target-only and recorded-q rejection rules, and uses the same fixed-order residual scan with replicated and batch-DP logits layouts under per-request sampling. ## Test Plan - `pre-commit run --all-files` passed. - On H200, 64 sampler and kernel tests passed and one existing test was skipped. This covers extra prefill preparation, changed slots and mixed batches, CUDA graph replay, sampler snapshot restore, and recorded-q verification. - Real chain MTP passed ten-bin distribution calibration for 10,000 committed tokens each with greedy draft proposals and with sampled proposals using recorded q. Both configurations accepted draft tokens. - Per-request DP layout equivalence remains unverified. End-to-end `auto` runs already showed first-token logprob differences before speculative verification. The sampler tests compare runs with matching numerical inputs. --------- Signed-off-by: sjqgogogogo <sjqgogogogo@gmail.com> |
||
|
|
abb3ce0b5a |
fix(runtime): offload persistent metadata with weight regions (#3620)
## Summary Releasing weights can leave ordinary CUDA allocator segments resident because small persistent runtime tensors were allocated outside the memory saver's regions. A packed cache table can keep an entire 2 MiB segment alive through a backend's shared-storage view. Allocate these tensors in the existing CPU-backed persistent-buffer pool so weights release unmaps their physical storage and weights resume restores their original addresses and contents: | Allocation owner | Change | | --- | --- | | Sampling and executor inputs | Scope sampler construction and the executor sentinel. | | Attention and cache metadata | Scope arena geometry, capture placeholders, target/draft metadata, and per-forward packed cache tables. Live forwards and prefill capture share the managed table factory. | | Host transfers | Scope immutable geometry and workspace tables at initialization and when later submissions grow them. | | Graph capture | Register generator seed/offset state before entering discardable graph capture; centralize capture barriers whose WORLD communicator is initialized before managed regions. | The existing release/drain order, CPU backup policy, CUDA graph path, NCCL backend, and worker lifecycle remain in place. Tests cover disabled mode, workspace growth, and content/address recovery through explicit metadata inputs. A fresh CUDA subprocess regression checks geometry, real GPU table packing and capture metadata across table growth and repeated sleep/wake cycles, retaining old views and requiring zero ordinary-pool bytes after release plus restored addresses and contents. ## Test Plan - Current review revision: `pre-commit run --all-files` passed. - Current review revision: local collection of `test_torch_memory_saver_adapter.py`, `test_host_executor.py`, and `test_retraction_image_executor.py` is blocked by missing `tokenspeed_triton` (three collection errors; no tests executed). The native metadata test requires CUDA VMM hooks and a fresh subprocess; it has not been run locally. Runtime/GPU CI is still required on this head. - Historical validation before the main rebase and this review refactor: the adapter suite passed **24 tests** in a CUDA runtime. A fresh **32-GPU** serving/training smoke completed two iterations and a final weights restore/release; all **128 inference offload observations** had **0 bytes in ordinary allocator segments** (`segment_pool_id=(0, 0)`), with unchanged inference worker PIDs. This measures the ordinary Torch pool, not total process GPU memory. - That historical smoke used non-default NCCL buffer/channel settings and training-side checkpoint-group cleanup. It does not validate a configuration with those settings and cleanup removed, or establish an isolated performance effect for this patch. - Historical native single-GPU controls covered barrier, graph RNG, and Host geometry sleep/wake across three cycles. The geometry control did not execute its payload transfer kernel. An isolated GPU packed-table baseline script failed in Triton `loadBinary` before reaching the candidate and is not counted as passed; the complete serving smoke exercised the packing path. - GPU decode step latency has **not** been compared before/after region entry. No latency or throughput neutrality claim is made. A full GPU smoke has not been repeated after rebasing or this review refactor. --------- Signed-off-by: lipanpan03 <41904587+lipanpan03@users.noreply.github.com> Signed-off-by: FC-Li <robert_lexis@163.com> Co-authored-by: FC-Li <robert_lexis@163.com> |
||
|
|
9a19e45d65 |
feat(engine): report resident and pending prefill token load (#3572)
## Summary Report per-rank token counts so SMG can use context size and pending prefill work for load-aware routing: - `num_resident_tokens`: prompt plus committed output tokens of every request held by the rank. - `num_waiting_uncached_tokens`: waiting prompts minus their last observed cache hit, plus unscheduled prefill tokens. Before the first admission probe, a prompt counts in full. Reporting adds no cache scans. Append both optional fields to the existing ZMQ output schema, declare them through the handshake, and preserve them in native client and Python snapshots. Reports from older engines retain `None` for unavailable counters. Adapted from [ |
||
|
|
1780127a77 |
feat(Kimi-K3): run each MoE layer's post-attention work as one gfx950 launch in single-request decode (#3613)
## Summary This PR adds the "MoE layer tail": a persistent gfx950 Gluon kernel for Kimi-K3 and Kimi-K3.1. It serves decode steps that hold one request, including speculative verify steps. In such a step, it runs each MoE layer's work after the attention core as one launch per rank: - the attention o-projection, and its all-reduce across the 8 TP ranks through each other's memory - the MLP-side AttnRes mix and the post-attention RMSNorm - the router (sigmoid plus correction bias, top-16 of 896), the latent down projection and the latent RMSNorm - the routed experts (MXFP4) and the shared experts - the latent up projection and the layer output - the next layer's attention input, when that layer mixes it from a prepared block partial The kernel is a persistent launch of 256 workgroups. Its stages hand data to each other through tagged words in an uncached scratch buffer. The ranks exchange partial sums through an uncached peer buffer that every rank maps with HIP IPC. Each rank adds the 8 parts in rank order, so every rank computes the same sums. How the model uses it: - The kernel is registered in a new `kimi_k3` family (ops `moe_tail`, `moe_tail_prepare` and `moe_tail_pack`). Its API is `tokenspeed_kernel.ops.model.kimi_k3`, and that module's README describes it. - While the model builds its layers, it asks the API which expert bank layout the tail reads. When a tail serves the deployment, the MoE layers store their routed experts once, in that layout (N16). - Before graph capture, the model prepares the per-rank buffers in a collective step. It then checks and packs every MoE layer and binds the tail to all of them. If any layer fails the checks, no layer uses the tail. - A decode forward whose row count equals the verify width, that is a step with exactly one request, runs each layer's tail. The decision uses only the forward mode and the row count, so every rank runs the same layers. Every other forward uses the regular path. - The regular gfx950 MXFP4 SiTU kernels (decode and both prefill stages) read N16 banks, so one copy of the experts serves both paths. Their outputs are bit-identical to the default gdot128 layout: in decode, and in prefill above 2,048 rows. At or below 2,048 rows, the prefill adds routes with BF16 atomics, so its output varies from run to run in either layout. N16 banks are about 19 GB per GPU smaller than gdot128 at Kimi-K3.1 sizes, and the runtime gives that memory to the KV cache. - The projection-plus-partials GEMV in `ops/gemm/gluon.py` now accepts Kimi-K3.1's 3808-row decode projection, so the attention input projection can also produce the next layer's AttnRes block partials. Numerics: the tail quantizes the routed experts' activations to MXFP4 (A4W4), where the regular decode MoE uses FP8 (A8W4). A step that the tail serves therefore does not give the same values as the regular path. A request's decode numerics depend on whether other requests share its steps. GSM8K accuracy is unchanged within noise (below). Limits: - gfx950 with 256 to 1024 compute units. - Kimi-K3 sizes at TP8, with one TP group for attention and MoE on one node (HIP IPC), and 1 to 8 rows, fixed per server. - The model keeps the regular path in these cases: attention DP, EP or PP, deterministic collectives, unfused norms, a prefill-only role, quantized or split attention projections, non-BF16 router, latent or shared-expert weights, and other expert formats or sizes. - To turn the tail off, set `TOKENSPEED_KERNEL_OVERRIDE_KIMI_K3_MOE_TAIL=none`. - The larger KV pool also grows a host KV tier sized by `--kvstore-ratio`, by about 2.1× at ratio 8. Set `--kvstore-size` to keep the previous host-tier size, for example 107 GB per rank at ratio 8. - Every wait in the kernel has a 60 s deadline. A rank that stalls past it inside a decode step aborts the server with a GPU memory fault. One example is a first-time compile with CUDA graphs off. ## Benchmarks and accuracy Kimi-K3.1 on 8x MI355X (gfx950), TP8, one request in flight, DSpark speculative decoding (7 draft steps, 8 rows per verify step), greedy sampling. Serving numbers compare the base with this PR; decode and GSM8K were measured on one build of this PR, prefill and AgentX on another. Those builds' tail code compiles to the same instructions as this branch. Decode verify step, median client step gap: | Workload | Regular path | Tail | Change | |---|---|---|---| | 50k-token prompts, real acceptance (base vs this PR, 4 interleaved starts) | 14.66 ms | 13.89 ms | −5.2% (95% CI −5.3% to −5.1%) | | 200k-token prompts, AgentX flags (one build, tail off vs on) | 15.24 ms | 14.40 ms | −5.5% (95% CI −5.7% to −5.3%) | With the 50k-token prompts, accepted tokens per step were 3.6% lower (95% CI −7.0% to −0.5%), and the per-request output rate showed no gain. InferenceX AgentX, concurrency 1, 30 minutes per build, synthetic acceptance 4.16 (not quality evidence), one run per build, means: | Metric | Base | This PR | Change | |---|---|---|---| | Output throughput per GPU | 16.36 tokens per second | 16.74 tokens per second | +2.4% | | Time per output token | 3.67 ms | 3.47 ms | −5.5% | | Time to first token | 1.65 s | 1.57 s | −5.0% | | End-to-end latency | 6.69 s | 6.41 s | −4.2% | Prefill, time to first token with 1 output token, mean of 20 requests per arm (2 interleaved starts each): | Input tokens | Base | This PR | Change | |---|---|---|---| | 2,048 | 230.5 ms | 233.9 ms | +1.5% | | 16,384 | 819.7 ms | 823.4 ms | +0.4% | GSM8K, first 256 test questions, greedy, up to 2,048 output tokens, real acceptance: | Metric | Base | This PR | Change | |---|---|---|---| | Accuracy | 243 of 256 (94.9%) | 242 of 256 (94.5%) | −0.4 points (95% CI −2.4 to +1.6) | | Decode output rate per request, mean | 404.5 tokens per second | 419.9 tokens per second | +3.8% (95% CI +1.8% to +5.8%) | Tests on this branch: - pre-commit (all files) and ruff with undefined-name checks pass. - CPU model runtime tests: 1,131 passed, 26 failed. The same 26 fail on the base. - `test_kernel_api_selection.py`, on one GPU: 499 passed, 122 skipped, 0 failed. - The gfx950 MoE tests that build the changed prefill stage, including 10 N16-versus-gdot128 bitwise cases: 390 passed. - The 8-GPU tail test (widths 1, 4 and 8, a block-write case, two steps across the step-count wrap): 4 of 4 passed. - The off switch, checked on CPU: no tail layout with it set, `n16` without it. - The broader gfx950 kernel tests for the changed MoE, GEMV and shared extern code ran on an earlier commit with the same tail, GEMV and extern code. They gave 1,295 passed, 0 failed and 75 skipped, and the 13 skipped torchrun cases passed on all 8 ranks. --------- Signed-off-by: Max Dawkins <max.dawkins@gmail.com> |
||
|
|
1aa75cbc31 |
refactor(attention): share the MLA leaves' forward-metadata entry (#3605)
## Summary
Each of the four paged MLA leaves carried its own copy of
`init_forward_metadata` and `override_num_extends`:
- `MLAAttnBackend` (`paged/mla.py`)
- `FlashMLABackend` (`paged/flashmla.py`)
- `CuteDSLMLABackend` (`paged/tokenspeed_mla.py`)
- `TRTLLMMLABackend` (`paged/trtllm_mla.py`)
Every entry took the same four steps:
1. reject a query shard;
2. refuse a decode forward;
3. build prefill metadata for the leading extend requests;
4. refresh decode metadata for a target's mixed or idle batch.
The four overrides were near-identical. Both methods now live in a new
`PagedMLABackend` in `paged/base.py`, and each leaf keeps only its
`_init_prefill_metadata`. `Dots3NoteSWABackend`, `AscendMLABackend` and
the DSA wrapper reach the shared entry through the leaves they extend or
wrap.
Where the copies differed:
- **FlashMLA** is the only leaf that reads `extend_with_prefix`, which
picks ragged or paged prefill. It records the flag in a small
`init_forward_metadata` wrapper before calling the shared entry. This
keeps the hook's arguments the same for every leaf, including the
subclasses and the tests that call it directly.
- **TRT-LLM MLA**'s hook now receives `page_table_cpu` and discards it,
as its entry did before.
- **Error messages** now name the leaf class:
- the decode-forward error reads `TRTLLMMLABackend decode metadata goes
through ...`; it used to say `trtllm_mla`;
- the query-shard error from `Dots3NoteSWABackend` and
`AscendMLABackend` names the subclass instead of `MLAAttnBackend`.
No kernel code changes.
## Test Plan
On GB200, main (
|
||
|
|
aa03b0275a |
perf(dcp): exchange small KVP decode payloads in one NVLink hop (#3586)
## Summary KVP decode runs a few small collectives per DSA layer: the query head gather, the candidate gather and the partial exchange. Across eight ranks, NCCL's ring all-gather takes about 19 us for a few dozen KiB. - A CuTe DSL one-hop exchange over symmetric memory (`ll_exchange.py`) serves all-gathers and equal-split all-to-alls up to a prepared per-rank capacity. Every 16 payload bytes travel as one 32-byte store of four (word, epoch) pairs, and the receiver polls until each pair carries the call's epoch, so no flag or fence is needed. Two parity slots let consecutive calls proceed. All-gathers store once through the multicast address. Waits are bounded by the distributed timeout and trap past it. - `prepare_small_exchange` arms it per group after a vote, so every rank either uses it or keeps its collectives. Engines with a KVP group arm 1 MiB at startup. - `AutoBackend` sends fitting `all_gather_single`, `all_to_all_single` and `all_gather` calls to it. Larger, unaligned and uneven calls keep their current path, and the NCCL-only switch disables it. ## Test Plan - [x] `tokenspeed-kernel/test/nvidia/ops/communication/test_ll_exchange.py`: 3 passed on four GB300 GPUs. It covers byte-exact all-gather and all-to-all against NCCL from 16 B to 1 MiB, graph replays with fresh inputs, the refusal vote and argument errors. - [x] `test/runtime/distributed/test_small_exchange.py`: 2 and 4 ranks passed. It covers routing, NCCL fallbacks, inner-dim gathers and graph replay. - [x] Kimi-K3.1, TP8 with KVP8 on 2x GB300, DSpark with synthetic acceptance 8. Each value is the mean of two fresh boots, in ms/token. From batch 32, every exchange exceeds the 1 MiB capacity and keeps NCCL. | batch | 1 | 2 | 4 | 8 | 16 | 32 | 64 | |---|---|---|---|---|---|---|---| | main | 1.577 | 1.897 | 2.440 | 3.145 | 4.577 | 6.042 | 8.171 | | this PR | 1.412 | 1.800 | 2.342 | 3.070 | 4.497 | 6.091 | 8.162 | | change | -10.5% | -5.1% | -4.0% | -2.4% | -1.8% | +0.8% | -0.1% | - [x] GSM8K, 200 questions with real acceptance: 196/200. --------- Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com> |
||
|
|
deab5d7f2e |
chore(deps): update the tokenspeed-smg pin to 1.16.0.post20261118 (#3608)
Consume the latest published `tokenspeed-smg` gateway release (`1.16.0.post20261118`). The `python/pyproject.toml` pins move from the `20261117` snapshot to the `20261118` snapshot: - `tokenspeed-smg==1.16.0.post20261118` - `tokenspeed-smg-grpc-proto==0.5.0.post20261118` - `tokenspeed-smg-grpc-servicer==0.16.0.post20261118` --------- Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> |
||
|
|
58f7374c07 |
cleanup(runtime): remove unreferenced layer and model code (#3597)
## Summary
This PR removes runtime code that nothing in the repository calls or
reads. Each name below appears only at its own definition: a whole-repo
`git grep -w` finds no caller, attribute access or string reference in
`python/`, `tokenspeed-kernel/`, `test/` or the configs.
- `layers/rotary_embedding.py`: `MRotaryEmbedding.get_rope_index_glm4v`
(196 lines), and `itertools`, which only it used.
- `layers/layernorm.py`: `FusedRMSNorm.forward_with_allgather_fusion`,
and the `allgather_dual_rmsnorm` import, which only it used.
`distributed/comm_ops.py` keeps its own import.
- `layers/vocab_parallel_embedding.py`:
- `VocabParallelEmbedding.get_sharded_to_full_mapping`
- `ParallelLMHead.tie_weights`
- `VocabParallelEmbeddingShardIndices.num_added_vocab_padding`
- `models/muse_spark.py`: `RESIDUAL_GATE_TEMPERATURE`. The layer reads
`config.residual_gate_temperature`.
- `models/llama_eagle3.py`: `LlamaForCausalLMEagle3.get_embed`, and the
never-read `_eagle3_prefix`.
None of the removed code runs today, so results and speed cannot change.
Left in place on purpose:
- `set_embed_and_head` overrides, which the draft factory calls
generically;
- public `tokenspeed_kernel` functions.
## Test Plan
On GB200, main (
|
||
|
|
8214efc977 |
perf(startup): freeze the collector once around every graph capture (#3555)
## Summary Prefill and drafter graph captures (`BreakableCapture`) collect garbage on entry, because a collection inside a capture would run graph destructors on a capturing stream; decode captures (`torch.cuda.graph`) collect only under `force_cudagraph_gc`. Only the decode ladder ran with the heap frozen (`freeze_gc`). So the prefill buckets and the drafter's prefill and extend graphs each ran a full collection over the startup heap: 150-200 ms apiece, about 150 per start. - `ModelExecutor.capture_graphs` now holds one frozen heap around every capture: decode, prefill and drafter graphs. - A boot that captures nothing (enforce-eager) skips the freeze and its two collections, as on main. - `--enable-cudagraph-gc` still keeps the heap unfrozen. - The decode runner no longer freezes on its own. Nested scopes would break, because an inner `gc.unfreeze()` thaws the outer scope. ## Results Qwen3.8-Flash-Next-NVFP4, TP1, MTP, 56 prefill buckets plus the drafter graphs. One GPU, warm compile caches, fresh server per run: | | main | this PR | |---|---|---| | full collections in the scheduler process | 148, 24.0 s | 148, 3.4 s | | server ready | 204.6, 207.6, 209.6 s (mean 207.3) | 187.5, 190.5 s (mean 189.0) | Both arms capture the same graphs with the same memory: | | both arms | |---|---| | KV cache | 182.05 GB | | activations + graphs | 3.98 GB | | device free | 5.63 GB | One consequence: an object that existed before capture and becomes cyclic garbage during a capture is now freed after the last capture instead of between capture groups. On this model's boot with the memory probe enabled, allocated memory is byte-identical at every one of the 5714 captures and the probe series are identical. Serving is unchanged: - First 64k-token prefill: 0.88 s; later ones: 0.655 s. - The same kernels compile on the first requests. - No collection runs while serving. ## Test Plan - `test_dispatch_autotune.py` runs `ModelExecutor.capture_graphs` with its capture doubles: - `test_capture_freezes_the_heap_once_around_every_capture`: the decode, prefill and drafter captures see a frozen heap unless `--enable-cudagraph-gc`; one `gc.freeze` spans them all; a boot that captures nothing collects nothing. - `test_a_failed_capture_leaves_the_heap_unfrozen`: a capture failing at any position leaves the heap as it found it. - Each of these mutations makes them fail: no `try`/`finally` in `freeze_gc`, one freeze per capture group, a decode-only freeze, no early return, `or` in its predicate, the flag ignored, no freeze. - `pre-commit run --all-files` passes. - The 11 test files that use `capture_graphs`, the decode runner's capture or `freeze_gc` fail the same tests on this branch and on main (28 FlashInfer BF16 adapter tests in `test_dispatch_autotune.py`, and `test_draft_final_step_follows_the_complete_drafter_run`, which fails on main since #3534). --------- Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com> |
||
|
|
401d29b693 |
fix(pd): propagate hybrid cache readiness to state backends (#3563)
Summary - Register the hybrid producer step counter on the linear-attention child. - Let state backends publish staged prefill fields before the wrapper signals readiness, without recording the same producer step twice. - Document the final-write and single-barrier contract for hybrid producers. Results - The rebased source passes 21 focused producer-readiness, destination-validation and FP8 state-transfer regression cases with real GPU backends. - Before the base update, an eight-GPU disaggregated comparison passed 18 requests and matched all 523 output token IDs and FP32 logprob bit patterns. It covered concurrency, chunked prefill and cached prefixes, with native layerwise sends. Test plan - Passed: pre-commit run --all-files. - Passed: cargo fmt --all --check in tokenspeed-engine, with /opt/cargo/bin on PATH and RUSTUP_HOME=/opt/rustup. - Passed: 21 focused integration regression cases against the rebased source. - Full-model comparison was run before the base update, not repeated afterward. --------- Signed-off-by: dingyuqing05 <13126694060@163.com> Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com> |
||
|
|
25872a5ca0 |
feat(cache): enable FlatKV by default for Qwen4-Exp (#3389)
## Summary Reduce padding in Qwen4-Exp caches by packing target and draft KV, GDN, PLE and QSA into one physical plane. FlatKV is enabled by default. Qwen3.8-Flash-Next uses five logical groups: full attention, one GDN state group, PLE, compressed QSA and recent QSA. Each parent remains owned by one group. - Translate full-history KV and QSA page IDs and write locations into atom addresses; retain explicit strides for state and recent-window caches. - Select stride-capable GDN kernels, including MTP and tree verification, using the actual input and output layouts. - Bound parent capacity by every target and draft field's int32 atom page and write slot limit, retaining startup overflow checks. Derive L3 namespaces from the actual layout, and preserve existing DeepSeek and Dots byte-page layouts. Use `--no-qwen4-exp-flat-kv` for the ordinary layout. FlatKV currently requires PP=1, uniform per-view K and V heads and non-MXFP8 KV. ## Measurements NVIDIA Qwen3.8-Flash-Next-NVFP4 on B300, TP1, BF16 KV, FP32 GDN, MTP 3 steps and 4 verify tokens, forward checkpoints enabled and L2 `write_back` (192 GB). | Metric | FlatKV disabled | FlatKV enabled | | --- | ---: | ---: | | GDN checkpoint allocation | 2 × 78 MiB | 110.5 MiB | | GDN checkpoint padding | 29.42% | 0.35% | | API L1+L2 hit rate | 86.90% | **88.81%** | | Output tokens per second | 1078.40 | **1157.56** | | TTFT P50 | 8.24 s | 6.47 s | | Mean ITL | 22.68 ms | 22.29 ms | | Snapshot reservation | 23.56 GB | 37.08 GB | InferenceX AgentX: concurrency 64, 256K context, seed 42, 702 warmup requests and a 900-second measurement window per configuration. Hit rate is cached prompt tokens divided by prompt tokens for completed profiling requests; both runs had zero request errors. These are historical runs from separate rounds with different GPUs and peer workloads. They used FlashInfer 0.7.3 with common compatibility adapters; current kernel validation uses pinned 0.7.4. Useful KV payload is unchanged. The larger parent also increases null-parent and small-group allocation costs, so total memory use depends on the live group mix. <details> <summary>GDN grouping comparison</summary> | Metric | 1 group (selected) | 6 groups | 18 groups | | --- | ---: | ---: | ---: | | Total cache groups | 5 | 10 | 22 | | Parent size | 110.5 MiB | 19.5 MiB | 6.5 MiB | | API L1+L2 hit rate | **88.81%** | 88.22% | 88.05% | | Output tokens per second | 1157.56 | 1189.93 | 1199.22 | | TTFT P50 | 6.47 s | **4.31 s** | 5.58 s | | TTFT P90 | **32.44 s** | 34.63 s | 33.09 s | | Mean ITL | **22.29 ms** | 23.34 ms | 26.57 ms | | Snapshot reservation | 37.08 GB | 13.09 GB | 9.60 GB | One group prioritizes checkpoint packing and the observed hit rate, ITL and P90 TTFT. More groups reduce snapshot reservation and improve throughput. Each configuration was measured once; the results show this workload's trade-offs. </details> Accuracy comparison: main `cdeb32aa7` vs PR `fb4cc01f9`, using the same serving configuration and complete test splits. | Evaluation | Main | PR | PR-only repeat | | --- | ---: | ---: | ---: | | GSM8K, 1K output budget | 1276 of 1319 (96.74%) | 1274 of 1319 (96.59%) | — | | GSM8K, 4K output budget | 1287 of 1319 (97.57%) | 1283 of 1319 (97.27%) | 1287 of 1319 (97.57%) | | AIME26, 64K output budget | 28 of 30 (93.33%) | 28 of 30 (93.33%) | — | GSM8K used 4-shot greedy decoding with thinking disabled; AIME26 used zero-shot thinking, temperature 1.0, top-p 0.95 and top-k 20. Both used EvalScope 1.11.1, batch size 16 and seed 42. All requests succeeded. GSM8K scores varied between PR runs; the repeat matched main's aggregate score with different per-question results. The cause of this variation has not been isolated. AIME26 answer extractions matched on all 30 questions, including the same truncated output. ## Test plan - **444 runtime and cache tests passed** at `4692d6cf6` (7 skipped), plus 7 subtests. Includes metadata-only capacity boundaries, differing target and draft payloads and existing layout regressions. Inkling activation parity also passes across prefill and page-crossing decode. - **279 GDN and fused decode-metadata kernel tests passed** (36 skipped), including 17 far-page cases beyond 2^31 elements, independent state strides, padding guards and CUDA Graph execution. - Real Qwen4 target and draft D2H → L3 → H2D round trips compare every arena byte, including padding and untouched blocks. - Fresh B300 TP2 service at 0.95 memory utilization, with forward checkpoints and L2 `write_back`: capacity safely capped at **2,372 parents and 137.48 GB per GPU**. All three requests returned the expected answer; a repeated 18,029-token prompt reused **17,920 tokens**. - `pre-commit run --all-files` passed. Hardware CI status is tracked by the checks on this PR. --------- Signed-off-by: tuanzhangCS <tuan@lightseek.org> |
||
|
|
3bd08baacc |
feat: share cached prefixes between engine replicas (peer prefix sharing) (#3531)
## Summary A replica can reuse a prefix that another replica already caches, in that replica's GPU or pinned Host cache, by reading it over RDMA instead of recomputing it. The design is in `docs/rfcs/0004-peer-prefix-sharing.md`. There is no central store and no L3. The router names a holder with the request (`peer_holder`, the holder's `host:port`). The reader then: 1. **Looks up.** It asks the holder how much of the prompt it caches. The lookup soft-pins those blocks. 2. **Pins.** At admission it hard-pins exactly the pages it will read. 3. **Reads.** It pulls them with one-sided Mooncake Transfer Engine READs into its own GPU blocks. Each member of the replica reads its own shard from the holder's member of the same index. 4. **Releases.** If the read was complete, the release asks the holder to demote its copy, so the two replicas don't both keep it. Any failure falls back to computing locally. The landing is given up whole and the request is admitted again from its local hit. Opt in with `--peer-prefix-port PORT`; attention-DP rank `r` binds `PORT + r`. Peer sharing requires the Host tier (`--enable-kvstore`) and prefix caching. It is refused with `--disaggregation-mode decode`, `--kvstore-storage-backend`, pipeline stages and query context parallelism. ## What changes - **Scheduler (C++).** - The holder answers lookups, pins and releases from const reads between rounds. The changes those answers imply are applied by every replica member at the next round's head. - The reader gains a `PeerLooking` state, and a `PeerLanding` state that generalizes the PD decode landing. - The landing's pages are kept out of the plan's copies, loads and zeroing. - Leases are protected by deadlines, incarnations and namespace checks. - **Control loop (Rust, `ts-control::peer`).** - The servicer, the per-request landing state machine, read geometry for Device and Host sources, and the endpoint over ZMQ. - The loop holds the scheduler under a mutex, released while it waits on a forward result, an idle frontend or a paused round's sleep, and handed once per round to a waiting servicer, so the endpoint can answer without waiting for a round. That last hand-over matters to a loop waiting on a transfer (a prefill engine's PD sends, a landing's reads), which reaches neither wait and spins through empty rounds. - Peer reports ride the replica relay. - **Engine host and runtime.** - Startup: a transfer engine per rank with the Device arena and Host pool registered, and the replica's geometries exchanged over the rank mesh. - Read fence: a plan with landings records an event behind its stores, copies, zeroing and the forwards in flight. Reads start only after that event. Every plan with a landing goes to the data plane, also when it has no other device work, as a prefill engine's landing may not. - The namespace is the L3 deployment id (`cache_deployment_id`, now shared by L3 and peer sharing) plus the weight version. A flushed weight update moves it, under the same weight-version rule L3 already enforces. - **KV events.** With Host L2, the event stream now also covers Host blocks, published with their medium in vLLM's layout. Device events gain trailing `nil` fields (`lora_id`, `medium`), which vLLM-format consumers already read. Each medium's stream is prefix-closed for a router that continues a GPU match into the Host tier. ## Testing - **C++.** The scheduler suite passes, including new peer tests run on both full-history and hybrid (sliding window + State) layouts. The closed-loop random test now checks each KV event medium's stream and its closure rule. - **Rust.** `ts-control`, `ts-pd`, `ts-mooncake`, `ts-scheduler` and `ts-host` pass, including in-process two-replica tests over real sockets. `cargo fmt` and `clippy` are clean. - **Python.** Tests for the geometry, server args, the device fence, the L3 factory refactor and the KV event layout pass. - **GPU.** Two replicas of Qwen3-0.6B on one node, with RDMA loopback through the NIC, using a locally built transfer engine: - From the holder's GPU cache: the reader landed 1536 prompt tokens and computed 1, and its greedy output was token-identical to the holder's. - From the holder's Host cache (write-back with a small GPU cache): same result. - A failed READ fell back to local compute with correct output. - **Prefill and decode, end to end through the Dynamo router** (with the router-side change on top): this found two bugs, fixed in the last two commits. - **A servicer starved by a spinning loop.** A loop waiting on a transfer spun without releasing the scheduler, so its servicer never answered. A test runs a loop waiting on a PD transfer and checks that another thread gets the scheduler. - **A landing that never read.** `plan_needs_device` ignored peer fetches, so a prefill engine's landing plan, with no other device work, never reached the data plane and its reads never started. A test lands a prefix on a prefill replica from such a plan. Both tests fail without their fix. - Some runtime test files need a newer GPU stack than the machine this was developed on, so CI covers those. ## Not in this PR - Router-side placement: peer credit with same-node and cross-node cost models, and the hint itself. That lands separately in the Dynamo router. - KV-page sharding, the decode role, and pipeline stages. Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com> --------- Signed-off-by: FC-Li <robert_lexis@163.com> Signed-off-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com> Co-authored-by: lightseek-bot <243258330+lightseek-bot@users.noreply.github.com> |
||
|
|
e43d3bf6a2 |
feat(dspark): support grammar and logprobs in adaptive verify (#3539)
## Summary - Gather adaptive-verification grammar masks into the same CSR-packed target-logit layout used by DP-local CUDA graphs. - Copy grammar verify candidates to pinned storage on the main stream before the fork event, so the side-stream host fill reads pre-packing tokens even when packed verification rewrites `input_ids_buf` in place. - Map flat accepted-path indices through packed CSR rows and emit selected-token output logprobs for greedy and sampled FlashInfer verification and sampled prefill. - Allow xGrammar and output logprobs at startup; keep cumulative probabilities explicitly unsupported and document the packed-layout contract. ## Tests - `pre-commit run --all-files` - Packed greedy and FlashInfer sampling tests: **38 passed**. - Adaptive layout and startup tests: **21 passed, 49 deselected**. - DeepSeek-V4.1-Flash on 8×B300 with TP1, DP8 and EP8, MegaMoE and DP-local CUDA graphs: 32 concurrent JSON-schema requests returned exact JSON with a finite logprob for every completion token. - Fresh restart with the committed source, sampled JSON and narrative validation: **32 of 32 requests passed**. Adaptive replay selected target query capacities **12 and 8 versus full width 24 on all eight DP ranks**; responses retained valid grammar and per-token finite logprobs. --------- Signed-off-by: wangbo981016 <wangbo981016@gmail.com> |
||
|
|
5cbc2329a4 |
feat(cache): visualize recipe layouts in startup logs (#3571)
Cache startup logs show capacity and packing without exposing field placement or unused bytes. Add an ASCII layout matrix after capacity agreement and PP narrowing, with cache group names as columns and resident planes as rows. The report includes physical packing and usable block counts, field offsets, shapes, and dtypes, stride padding, unused-plane holes, and per-parent payload utilization. Each column describes an alternative binding of the same parent. Serving builds log at INFO; memory probes log at DEBUG. The design documentation explains byte accounting, null-parent reservation, and PP and offload interpretation. Validation: - 44 CPU planner and visualization tests passed: `python -m pytest --noconftest -q test/runtime/test_cache_layout_visualization.py test/runtime/test_cache_memory_plan.py`. - Coverage includes shared-stride alignment gaps, trailing padding, unused planes, PP field and plane filtering, null-parent accounting, and large capacities. - `pre-commit run --all-files` passed. - GPU serving was not exercised. ## Example startup logs These examples use the cache field declarations with the same `pack()` and `format_cache_layout()` code as the startup report. Capacity is illustrative: **one usable parent plus one reserved null parent**, with `pp_rank=0`. Actual serving capacity follows memory budgeting and scheduler demand. Logger timestamps and rank prefixes are omitted. ### DeepSeek V4.1 Flash with DSpark Configuration: native `v41` cache format and **three DSpark stages**. The single `flatkv` plane is **3,225,600 B per parent**. The groups reuse that plane with different packing, so their columns describe alternative parent bindings and must not be added together. <details> <summary>Full V4.1 Flash + DSpark layout log</summary> ```text CacheRecipe DeepseekV41Recipe (deepseek_v41): build=serving, pp_rank=0 prefix_granularity=256 tokens, usable_parents=1, reserved_null_parents=1 logical_parent=3,225,600 B, resident_parent=3,225,600 B, layout_arena=6,451,200 B (0.006 GiB) Groups are alternative bindings of the same parent; do not sum columns. Packing and block counts are rank-local physical counts. Ranges [start, end) are bytes within one child stride, repeated packing times. Padding is unused space in an occupied plane; a hole is an unused plane. +--------------------+-----------------------------------+---------------------------------+-----------------------------------+------------------------------------------+ | plane | v41.compressor_tail_r2 | v41.global_r1 | v41.global_r2 | v41.swa | | | packing=126 | packing=140 | packing=45 | packing=2 | | | usable_blocks=126 | usable_blocks=140 | usable_blocks=45 | usable_blocks=2 | +--------------------+-----------------------------------+---------------------------------+-----------------------------------+------------------------------------------+ | flatkv | stride=25,600 B | stride=23,040 B | stride=71,680 B | stride=1,612,800 B | | parent=3,225,600 B | layer.14.compressor_tail | layer.20.global_kv | layer.14.global_kv | layer.0.swa | | arena_offset=0 B | [0, 8,192): 8,192 B | [0, 18,432): 18,432 B | [0, 18,432): 18,432 B | [0, 33,792): 33,792 B | | | float32 (2, 2, 512) | uint8 (64, 288) | uint8 (64, 288) | uint8 (64, 528) | | | layer.2.compressor_tail | layer.20.index_k | layer.14.index_k | layer.1.swa | | | [8,192, 16,384): 8,192 B | [18,432, 22,784): 4,352 B | [18,432, 22,784): 4,352 B | [33,792, 67,584): 33,792 B | | | float32 (2, 2, 512) | uint8 (64, 68) | uint8 (64, 68) | uint8 (64, 528) | | | layer.8.compressor_tail | padding [22,784, 23,040): 256 B | layer.2.global_kv | layer.10.swa | | | [16,384, 24,576): 8,192 B | payload=3,189,760 B/parent | [22,784, 41,216): 18,432 B | [67,584, 101,376): 33,792 B | | | float32 (2, 2, 512) | padding=35,840 B/parent | uint8 (64, 288) | uint8 (64, 528) | | | padding [24,576, 25,600): 1,024 B | | layer.2.index_k | layer.11.swa | | | payload=3,096,576 B/parent | | [41,216, 45,568): 4,352 B | [101,376, 135,168): 33,792 B | | | padding=129,024 B/parent | | uint8 (64, 68) | uint8 (64, 528) | | | | | layer.8.global_kv | layer.12.swa | | | | | [45,568, 64,000): 18,432 B | [135,168, 168,960): 33,792 B | | | | | uint8 (64, 288) | uint8 (64, 528) | | | | | layer.8.index_k | layer.13.swa | | | | | [64,000, 68,352): 4,352 B | [168,960, 202,752): 33,792 B | | | | | uint8 (64, 68) | uint8 (64, 528) | | | | | padding [68,352, 71,680): 3,328 B | layer.14.swa | | | | | payload=3,075,840 B/parent | [202,752, 236,544): 33,792 B | | | | | padding=149,760 B/parent | uint8 (64, 528) | | | | | | layer.15.swa | | | | | | [236,544, 270,336): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.16.swa | | | | | | [270,336, 304,128): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.17.swa | | | | | | [304,128, 337,920): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.18.swa | | | | | | [337,920, 371,712): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.19.swa | | | | | | [371,712, 405,504): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.2.swa | | | | | | [405,504, 439,296): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.20.swa | | | | | | [439,296, 473,088): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.21.swa | | | | | | [473,088, 506,880): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.22.swa | | | | | | [506,880, 540,672): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.23.swa | | | | | | [540,672, 574,464): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.24.swa | | | | | | [574,464, 608,256): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.25.swa | | | | | | [608,256, 642,048): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.26.swa | | | | | | [642,048, 675,840): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.27.swa | | | | | | [675,840, 709,632): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.28.swa | | | | | | [709,632, 743,424): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.29.swa | | | | | | [743,424, 777,216): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.3.swa | | | | | | [777,216, 811,008): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.30.swa | | | | | | [811,008, 844,800): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.31.swa | | | | | | [844,800, 878,592): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.32.swa | | | | | | [878,592, 912,384): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.33.swa | | | | | | [912,384, 946,176): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.34.swa | | | | | | [946,176, 979,968): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.35.swa | | | | | | [979,968, 1,013,760): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.36.swa | | | | | | [1,013,760, 1,047,552): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.37.swa | | | | | | [1,047,552, 1,081,344): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.38.swa | | | | | | [1,081,344, 1,115,136): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.39.dspark_kv0 | | | | | | [1,115,136, 1,180,672): 65,536 B | | | | | | bfloat16 (64, 512) | | | | | | layer.39.dspark_kv1 | | | | | | [1,180,672, 1,246,208): 65,536 B | | | | | | bfloat16 (64, 512) | | | | | | layer.39.dspark_kv2 | | | | | | [1,246,208, 1,311,744): 65,536 B | | | | | | bfloat16 (64, 512) | | | | | | layer.39.swa | | | | | | [1,311,744, 1,345,536): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.4.swa | | | | | | [1,345,536, 1,379,328): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.5.swa | | | | | | [1,379,328, 1,413,120): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.6.swa | | | | | | [1,413,120, 1,446,912): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.7.swa | | | | | | [1,446,912, 1,480,704): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.8.swa | | | | | | [1,480,704, 1,514,496): 33,792 B | | | | | | uint8 (64, 528) | | | | | | layer.9.swa | | | | | | [1,514,496, 1,548,288): 33,792 B | | | | | | uint8 (64, 528) | | | | | | padding [1,548,288, 1,612,800): 64,512 B | | | | | | payload=3,096,576 B/parent | | | | | | padding=129,024 B/parent | +--------------------+-----------------------------------+---------------------------------+-----------------------------------+------------------------------------------+ | TOTAL / parent | payload=3,096,576 B | payload=3,189,760 B | payload=3,075,840 B | payload=3,096,576 B | | 3,225,600 B | padding=129,024 B | padding=35,840 B | padding=149,760 B | padding=129,024 B | | | hole=0 B | hole=0 B | hole=0 B | hole=0 B | | | unused=129,024 B | unused=35,840 B | unused=149,760 B | unused=129,024 B | | | utilization=96.00% | utilization=98.89% | utilization=95.36% | utilization=96.00% | +--------------------+-----------------------------------+---------------------------------+-----------------------------------+------------------------------------------+ ``` </details> ### Kimi K3 with DSpark Configuration: **TP8, five MLA draft layers, FP8 KV for both target and draft, and FP32 KDA recurrent state**. The parent has 24 target MLA planes and five draft planes, totaling **25,657,344 B**. The reported `uint8` dtype is the FP8 KV storage view. Each KDA group has 23 layers and uses `slot.0` through `slot.22`. `slot.23` and the five draft planes (`slot.24` through `slot.28`) are unused by a KDA binding: **5,308,416 B of holes**, plus **1,625,088 B of stride padding**, giving **72.98% payload utilization**. The excerpt below preserves the original table formatting and includes `slot.0`, `slot.23`, `slot.24`, `slot.28`, and the totals. Other plane rows are omitted; totals cover all 29 planes. <details> <summary>K3 TP8 + DSpark layout log (selected planes and complete totals)</summary> ```text CacheRecipe KimiK3Recipe (kimi_k3): build=serving, pp_rank=0 prefix_granularity=128 tokens, usable_parents=1, reserved_null_parents=1 logical_parent=25,657,344 B, resident_parent=25,657,344 B, layout_arena=51,314,688 B (0.048 GiB) Groups are alternative bindings of the same parent; do not sum columns. Packing and block counts are rank-local physical counts. Ranges [start, end) are bytes within one child stride, repeated packing times. Padding is unused space in an occupied plane; a hole is an unused plane. +---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | plane | full_attention | linear_attention_0 | linear_attention_1 | linear_attention_2 | | | packing=12 | packing=1 | packing=1 | packing=1 | | | usable_blocks=12 | usable_blocks=1 | usable_blocks=1 | usable_blocks=1 | +---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | slot.0 | stride=73,728 B | stride=884,736 B | stride=884,736 B | stride=884,736 B | | parent=884,736 B | layer.3.latent_kv | layer.0.conv_state | layer.30.conv_state | layer.61.conv_state | | arena_offset=0 B | [0, 73,728): 73,728 B | [0, 27,648): 27,648 B | [0, 27,648): 27,648 B | [0, 27,648): 27,648 B | | | uint8 (128, 1, 576) | bfloat16 (4608, 3) | bfloat16 (4608, 3) | bfloat16 (4608, 3) | | | payload=884,736 B/parent | layer.0.recurrent_state | layer.30.recurrent_state | layer.61.recurrent_state | | | padding=0 B/parent | [27,648, 814,080): 786,432 B | [27,648, 814,080): 786,432 B | [27,648, 814,080): 786,432 B | | | | float32 (12, 128, 128) | float32 (12, 128, 128) | float32 (12, 128, 128) | | | | padding [814,080, 884,736): 70,656 B | padding [814,080, 884,736): 70,656 B | padding [814,080, 884,736): 70,656 B | | | | payload=814,080 B/parent | payload=814,080 B/parent | payload=814,080 B/parent | | | | padding=70,656 B/parent | padding=70,656 B/parent | padding=70,656 B/parent | +---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | slot.23 | stride=73,728 B | hole=884,736 B/parent | hole=884,736 B/parent | hole=884,736 B/parent | | parent=884,736 B | layer.92.latent_kv | | | | | arena_offset=28,311,552 B | [0, 73,728): 73,728 B | | | | | | uint8 (128, 1, 576) | | | | | | payload=884,736 B/parent | | | | | | padding=0 B/parent | | | | +---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | slot.24 | stride=73,728 B | hole=884,736 B/parent | hole=884,736 B/parent | hole=884,736 B/parent | | parent=884,736 B | layer.93.latent_kv | | | | | arena_offset=30,081,024 B | [0, 73,728): 73,728 B | | | | | | uint8 (128, 1, 576) | | | | | | payload=884,736 B/parent | | | | | | padding=0 B/parent | | | | +---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | slot.28 | stride=73,728 B | hole=884,736 B/parent | hole=884,736 B/parent | hole=884,736 B/parent | | parent=884,736 B | layer.97.latent_kv | | | | | arena_offset=37,158,912 B | [0, 73,728): 73,728 B | | | | | | uint8 (128, 1, 576) | | | | | | payload=884,736 B/parent | | | | | | padding=0 B/parent | | | | +---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ | TOTAL / parent | payload=25,657,344 B | payload=18,723,840 B | payload=18,723,840 B | payload=18,723,840 B | | 25,657,344 B | padding=0 B | padding=1,625,088 B | padding=1,625,088 B | padding=1,625,088 B | | | hole=0 B | hole=5,308,416 B | hole=5,308,416 B | hole=5,308,416 B | | | unused=0 B | unused=6,933,504 B | unused=6,933,504 B | unused=6,933,504 B | | | utilization=100.00% | utilization=72.98% | utilization=72.98% | utilization=72.98% | +---------------------------+--------------------------+--------------------------------------+--------------------------------------+--------------------------------------+ ``` </details> Signed-off-by: FC-Li <robert_lexis@163.com> |
||
|
|
85266e5764 |
refactor(scheduler): make cache planning data flow explicit (#3566)
## Summary Cache admission and prefix probing carried intermediate results through parallel arrays, output references, and preparation-dependent members. This refactor makes request inputs, returned decisions, and per-attempt working state explicit. - Group placement requirements in `BlockPlacementDemand`; return admission capacity, constraints, copy-in reservations, and decisions as values. - Construct candidate traversals directly and derive checkpoint aging from selected State blocks, removing mirrored checkpoint-removal counters. Derive group-scan counts from cursor state and L3 completion from per-key results, including all-failed results. - Return prefix-probe results together while reusing hit-array storage; make derived cache metadata and allocator configuration immutable, and strengthen const views. - Clarify block counts versus indices, available tail capacity, and interfaces that record statistics or acquire owners. Document explicit data flow, functional style, const correctness, and consistent naming in `AGENTS.md`. - Install the exact clang-format revision pinned by pre-commit in CI. The runner's newer distro formatter otherwise rejects code that passes the pinned formatter. ## Validation - `pre-commit run --all-files`: passed. - Build and run the full scheduler C++ suite: 1,550 passed; 1 AddressSanitizer-only test skipped in the Release build. - CI-equivalent clang-format 18.1.8 dry run over all 170 scheduler C++ files: passed. --------- Signed-off-by: FC-Li <robert_lexis@163.com> |
||
|
|
2a2faef07c |
fix: preserve explicit CUDA device connection limits (#3562)
Summary - Preserve CUDA_DEVICE_MAX_CONNECTIONS supplied by the caller. - Retain the default of 4 when unset and document the override. - Add regression coverage for unset, 1, 4, 8, and 32. Results - All 5 environment regression cases pass. - Disaggregated inference passes 6 requests, including concurrent and streaming requests. - All 8 GPU workers inherit CUDA_DEVICE_MAX_CONNECTIONS=8. Test plan - Passed: targeted launcher regression tests, Black, isort, and git diff --check. - Passed: pre-commit run --all-files and cargo fmt --all --check. - Attempted: cargo clippy --all-targets --locked -- -D warnings; blocked by CMake failing to resolve Python Interpreter and Development.Module. Further checks were stopped at the author's request. Signed-off-by: dingyuqing05 <13126694060@163.com> Co-authored-by: longcat-ia-team <longcat_ia_team@meituan.com> |