fix(pd): let decode allocation gate KV transfer instead of prefill

Start prefill admission independently while decode follows its existing
allocation path. Remember destination readiness before or during prefill,
and retain completed prefill resources until KV transfer can start.

Select early or independent admission explicitly on prefill. Preserve the
decode protocol and existing pause/cancellation paths; handle cancellation
of the new destination-waiting state through the existing transfer events.

Cover readiness ordering, capacity backpressure, frozen scheduling, and
cancellation received during the final prefill forward.

Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com>
This commit is contained in:
Enwei Zhu
2026-10-29 13:04:24 +00:00
parent 965705ed8b
commit 58e9f197a0
65 changed files with 1156 additions and 112 deletions
+11
View File
@@ -529,6 +529,17 @@ Use `--tensor-parallel-size` for simple launches. When attention, dense,
and MoE layers need different process groups, use the TokenSpeed-specific
split knobs.
### P/D admission
Prefill servers explicitly select `--disaggregation-decode-allocation-policy`.
Decode servers leave it unset and use their existing allocation path.
Both `early` and `independent` start decode KV allocation on request arrival.
`early` waits for that allocation before admitting prefill. `independent` lets
prefill schedule immediately and requires both a completed prefill and a ready
destination only when sending KV. A destination grant is retained if it arrives
before prefill finishes. Independent admission currently requires Mooncake,
attention TP1/KVP1, PP1, and `--disaggregation-layerwise-interval 0`.
### Mixed prefill/decode placement
Prefill and decode instances can occupy disjoint GPUs on the same hosts. Each
+7 -2
View File
@@ -84,7 +84,7 @@ KV-pool device memory over RDMA rather than through a CUDA kernel, but it
needs the same ordering against forwards and page zeroing — so its execution
face lives behind the handle too, attached once at startup. Its control face
(bootstrap register/abort, event polling, `pop_*`) stays on the control
plane, where it feeds Principle 3's tail advance.
plane, where it feeds Principle 3's explicit feedback points.
Consequences:
@@ -332,7 +332,12 @@ There are exactly two call sites, each with a documented reason:
* **Head of the round** — completed Host cache-op events
(`_cache_hooks.poll_ready_events()`: `WriteBackDone`, `LoadBackDone`,
`SnapshotDone`, `RestoreDone`, each replica-intersected), together with
the L3 hooks' agreed storage facts.
the L3 hooks' agreed storage facts and local prefill readiness under
`independent` P/D admission. This includes destination-ready grants, which
the native request remembers independently of its prefill state. Local P
readiness does not depend on D allocation and must be visible to the first
batch planned after arrival. Transfer completions retain their tail feedback
point, where grants arriving during the forward are also processed.
These must advance *before* `next_execution_plan`; otherwise, cache-gated
admissions -- a prefetched request's admission, a suspended request's
restore, which waits for both of its store ACKs, then its own -- are
+24 -2
View File
@@ -744,6 +744,26 @@ outcomes but never compose the batch.
### 3.1 P — prefill worker
Prefill workers select an explicit `decode_allocation_policy`: `early` or
`independent`; decode and fused engines leave it unset. Both start normal D KV
allocation when the request arrives. Early allocation gates P admission on the
destination manifest. Independent admission makes P eligible before batch
planning using only its own cache capacity, while D admits the request in parallel.
A `DestinationReadyEvent` is remembered on the native request even while P is
queued or computing. After the final P result commits, a ready destination makes
the request `PrefillDone` immediately. Otherwise, `PrefillAwaitingDestination`
retains its normal ForwardResources until the grant arrives. Only `PrefillDone`
can enter `remote_decode`, so sending KV requires both completed computation and
an allocated destination. The runtime polls readiness before planning and with
forward completion, allowing a grant received during a forward to release that
forward's completed prompt without another model iteration. A cancelled request
waiting for its destination uses the existing transfer-failure event to release
its resources. The prefill executor remembers cancellations received during a
forward so they can take effect if the request enters this waiting state. Other
request states retain their existing cancellation behavior.
Duplicate or late grants do not revive finished requests.
**Phases:** completed prompts out on `plan.remote_decode` (their pages stay
pinned until the transfer finishes, so releasing them outranks feeding more
prompt work), then the shared local-prefill phases
@@ -766,8 +786,10 @@ prompt whose KV is mid-transfer would strand the decode side.
The PD pin is not recorded anywhere; it is a function of the FSM
(`Scheduler::pdTransferInFlight`). On this role every page-holding state is
pinned — the peer's decode reads the pages from the first scheduled chunk until
the PD ACK finishes or aborts the request. On the D role the pin is exactly
pinned from the first scheduled chunk until the PD ACK finishes or aborts the
request. Early allocation may transfer chunks before prefill completes;
independent admission waits for the final result and destination grant.
On the D role the pin is exactly
`RemotePrefilling`: the peer's prefill is writing the destination pages, and
`RemotePrefillDone` ends it by leaving that state. A fused engine never
transfers. Because the pin is the state, no event handler can forget to clear
+12 -10
View File
@@ -198,6 +198,7 @@ def scheduler_config_from_args(
disable_l2_cache=not server_args.enable_kvstore,
enable_l3_storage=has_l3,
role=server_args.disaggregation_mode,
decode_allocation_policy=server_args.disaggregation_decode_allocation_policy,
num_snapshot_pages=specs.num_snapshot_pages,
max_retracted_requests=specs.max_retracted_requests,
storage_config=storage_config,
@@ -849,6 +850,8 @@ class EventLoop:
self.output_processor.mark_abort(request_id, notify_client=True)
else:
self.output_processor.mark_abort(request_id)
if self._pd_hooks.cancel_request(request_id, _reason):
return
if self._l3_hooks.cancel_request(request_id):
state = self.output_processor.rid_to_state.get(request_id)
if state is not None:
@@ -1201,8 +1204,8 @@ class EventLoop:
Scheduler feedback is only ever an explicit ``advance_scheduler`` call
in this loop body — helpers return events, never advance. Two calls:
cache-op completions at the head of the round (so this round's plan
sees them) and forward results at the tail (they only exist after
admission readiness at the head of the round (so this round's plan
sees it) and forward results at the tail (they only exist after
dispatch); everything else funnels into ``request_changes``.
"""
in_flight: deque = deque()
@@ -1217,17 +1220,16 @@ class EventLoop:
# rank-identical every cycle. A no-op without an EPD admission
# controller (every non-EPD deployment).
self._epd_hooks.drain_ready_embeddings()
cache_events = self._l3_hooks.poll_ready_events(
ready_events = self._l3_hooks.poll_ready_events(
allow_forward=not self._pause.forward_blocked,
allow_storage=not self._pause.forward_blocked,
)
cache_events.extend(self._cache_hooks.poll_ready_events())
if cache_events:
# Advanced at the HEAD of the round (not funneled into the
# tail advance) so completed cache ops are visible to this
# round's next_execution_plan — deferring them would delay
# cache-gated admissions by a full round.
advance_scheduler(self.scheduler, cache_events)
ready_events.extend(self._cache_hooks.poll_ready_events())
ready_events.extend(self._pd_hooks.poll_admission_events())
if ready_events:
# Apply admission readiness before planning so ready
# requests can join this round's batch.
advance_scheduler(self.scheduler, ready_events)
self._l3_hooks.record_metrics()
# Every path in this round appends its committed results here;
@@ -350,6 +350,7 @@ def make_config(
num_snapshot_pages: int,
max_retracted_requests: int,
storage_config: StorageConfig | None,
decode_allocation_policy: str | None,
debug_force_retraction_interval: int = 0,
enable_kv_cache_events: bool = False,
decode_input_tokens: int = 1,
@@ -385,6 +386,8 @@ def make_config(
A victim whose image does not fit the Host (L2 pins, pool, rows)
is aborted by the scheduler instead of imaged.
storage_config: Explicit asynchronous L3 budgets, or None without L3.
decode_allocation_policy: Explicit early/independent prefill admission choice,
or None for decode and fused serving.
debug_force_retraction_interval: ``--debug-force-retraction-interval``:
0 is off; every ``|N|`` plans the scheduler retracts the oldest
quiescent Decoding (``N > 0``) or Prefilling (``N < 0``) request
@@ -435,6 +438,20 @@ def make_config(
cfg.enable_kv_cache_events = enable_kv_cache_events
cfg.role = scheduler_role(role)
if cfg.role != SchedulerConfig.Role.P:
if decode_allocation_policy is not None:
raise ValueError("Decode allocation policy is only valid for prefill")
cfg.decode_allocation_policy = None
else:
policies = {
"early": SchedulerConfig.DecodeAllocationPolicy.Early,
"independent": SchedulerConfig.DecodeAllocationPolicy.Independent,
}
if decode_allocation_policy not in policies:
raise ValueError(
"Prefill scheduling requires an explicit decode allocation policy"
)
cfg.decode_allocation_policy = policies[decode_allocation_policy]
cfg.decode_input_tokens = decode_input_tokens
cfg.overlap_schedule_depth = overlap_schedule_depth
cfg.disable_prefix_cache = disable_prefix_cache
@@ -1838,6 +1838,7 @@ def _build_kv_transfer(
logical_plan=logical_plan,
),
gloo_group=sync_group,
prefill_admission_policy=server_args.disaggregation_decode_allocation_policy,
)
interval = server_args.disaggregation_layerwise_interval
if isinstance(kv_transfer, DisaggPrefillExecutor) and interval > 0:
+7 -1
View File
@@ -104,12 +104,18 @@ def create_kv_transfer(
args: KVManagerArgs,
kv_args: KVArgs,
gloo_group,
*,
prefill_admission_policy: str | None,
):
if backend not in (TransferBackend.MOONCAKE, TransferBackend.MOONCAKE.value):
raise NotImplementedError("CachePD supports only the Mooncake backend")
if mode == "prefill":
return DisaggPrefillExecutor(args, kv_args, gloo_group)
return DisaggPrefillExecutor(
args, kv_args, gloo_group, admission_policy=prefill_admission_policy
)
elif mode == "decode":
if prefill_admission_policy is not None:
raise ValueError("Decode does not select prefill admission policy")
return DisaggDecodeExecutor(args, kv_args, gloo_group)
else:
raise NotImplementedError(f"Unsupported disaggregation mode: {mode}")
@@ -230,12 +230,20 @@ class MooncakeKVManagerPrefill(MooncakeKVManagerBase):
self.bootstrap_logprobs[room] = logprob
def begin_room(self, room: int) -> None:
"""Reset request metadata before publishing a room."""
"""Register the local attempt before making it eligible for prefill."""
with self.bootstrap_token_cond:
self.prefill_metadata.pop(room, None)
self.cached_tokens.pop(room, None)
self.bootstrap_logprobs.pop(room, None)
self.update_status(room, TransferPoll.Bootstrapping)
self.update_status(room, TransferPoll.Bootstrapping)
def cancel_before_transfer(self, room: int, reason: str) -> None:
with self.bootstrap_token_cond:
if self.request_status.get(room) in (None, TransferPoll.Success):
return
# Intentional cancellation uses terminal notification without a transfer-failure sample.
self.update_status(room, TransferPoll.Failed)
self.abort_room(room, reason)
def discard_room(self, room: int) -> None:
"""Drop all Prefill-manager state owned by one terminal request."""
@@ -243,9 +251,9 @@ class MooncakeKVManagerPrefill(MooncakeKVManagerBase):
# chunk (Success) do: a thread that finds no destinations for a room
# reads its status (room_status) to tell an ended room from missing
# metadata.
self.request_status.pop(room, None)
self.transfer_infos.pop(room, None)
with self.bootstrap_token_cond:
self.request_status.pop(room, None)
self.transfer_infos.pop(room, None)
self.prefill_metadata.pop(room, None)
self.cached_tokens.pop(room, None)
self.bootstrap_logprobs.pop(room, None)
@@ -930,8 +938,9 @@ class MooncakeKVManagerPrefill(MooncakeKVManagerBase):
already_counted = room in self.failure_records
if not already_counted:
self.kv_transfer_metrics.record_kv_transfer_failure()
self.record_failure(room, reason)
self.update_status(room, TransferPoll.Failed)
with self.bootstrap_token_cond:
self.record_failure(room, reason)
self.update_status(room, TransferPoll.Failed)
for req in list(self.transfer_infos.get(room, {}).values()):
if not req.is_dummy:
try:
@@ -1201,12 +1210,13 @@ class MooncakeKVManagerPrefill(MooncakeKVManagerBase):
)
return
self.transfer_infos[parsed_room] = candidate_infos
# A transfer worker can fail and remove the room after the precheck but
# before this bootstrap-thread commit. Do not let that interleaving
# resurrect destination state for a sticky-Failed room.
if self.request_status.get(parsed_room) == TransferPoll.Failed:
self.transfer_infos.pop(parsed_room, None)
with self.bootstrap_token_cond:
rejected = self.request_status.get(parsed_room) == TransferPoll.Failed
if not rejected:
self.transfer_infos[parsed_room] = candidate_infos
if len(candidate_infos) == expected_fanout:
self.update_status(parsed_room, TransferPoll.Bootstrapped)
if rejected:
self.sync_status_to_decode_endpoint(
registration.endpoint,
registration.dst_port,
@@ -1214,10 +1224,6 @@ class MooncakeKVManagerPrefill(MooncakeKVManagerBase):
TransferPoll.Failed,
self._status_prefill_rank,
)
return
complete = len(candidate_infos) == expected_fanout
if complete:
self.update_status(parsed_room, TransferPoll.Bootstrapped)
def start_prefill_thread(self):
self.bind_server_socket(get_local_ip_by_remote())
@@ -45,16 +45,25 @@ _BOOTSTRAP_STALL_SECONDS = 10.0
class DisaggPrefillExecutor:
def __init__(self, args, kv_args, gloo_group):
def __init__(self, args, kv_args, gloo_group, *, admission_policy: str):
if admission_policy not in ("early", "independent"):
raise ValueError("Prefill requires an explicit admission policy")
self._admission_policy = admission_policy
self.cache_layout = kv_args.cache_layout
self.senders: dict[str, MooncakeKVSender] = {}
self.kv_manager = MooncakeKVManagerPrefill(args, kv_args)
self.gloo_group = gloo_group
self._local_states = {}
self._destination_ready: set[str] = set()
self.pending_cancellations: dict[str, str] = {}
self._layerwise_enabled = False
self._layerwise_interval = 1
self._bootstrap_stall_logged_at: float | None = None
@property
def independent_admission(self) -> bool:
return self._admission_policy == "independent"
def has_inflight_transfers(self) -> bool:
"""Whether a KV send is still registered (bootstrapping or sending)."""
return bool(self.senders)
@@ -100,6 +109,8 @@ class DisaggPrefillExecutor:
sender.clear()
self.kv_manager.discard_room(sender.bootstrap_room)
self._local_states.pop(req_id, None)
self._destination_ready.discard(req_id)
self.pending_cancellations.pop(req_id, None)
def prepare_prefill(self, op) -> None:
if not self._layerwise_enabled or op.num_extends() == 0:
@@ -366,6 +377,8 @@ class DisaggPrefillExecutor:
Bootstrapping is waiting for another rank's pre-allocation; that rank
never receives it when the decode side lost the room.
"""
if self._admission_policy == "independent":
return
now = time.time()
stalled = [
f"{req_id}(room={sender.bootstrap_room}, age={now - sender.init_time:.0f}s)"
@@ -388,28 +401,62 @@ class DisaggPrefillExecutor:
stalled[:8],
)
def cancel_awaiting_destination(self, request_id: str, reason: str) -> None:
self.pending_cancellations.pop(request_id, None)
sender = self.senders.get(request_id)
if sender is not None:
self.kv_manager.cancel_before_transfer(sender.bootstrap_room, reason)
def _readiness_events(self, request_ids, polls):
independent = self._admission_policy == "independent"
events = []
for request_id, poll in zip(request_ids, polls, strict=True):
if poll in (TransferPoll.Failed, TransferPoll.Success):
continue
if self._local_states[request_id] == TransferPoll.Bootstrapping and (
independent or poll == TransferPoll.Bootstrapped
):
events.append(PD.BootstrappedEvent(request_id))
self._local_states[request_id] = TransferPoll.Bootstrapped
logger.debug(
f"[prefill][generate_events] rid={request_id!s} -> BootstrappedEvent"
)
if (
independent
and poll == TransferPoll.Bootstrapped
and request_id not in self._destination_ready
):
self._destination_ready.add(request_id)
events.append(PD.DestinationReadyEvent(request_id))
return events
def generate_admission_events(self):
if self._admission_policy != "independent":
return []
request_ids = [
request_id
for request_id in self.senders
if request_id not in self._destination_ready
]
if not request_ids:
return []
polls = poll_and_all_reduce(
[self.senders[request_id] for request_id in request_ids], self.gloo_group
)
return self._readiness_events(request_ids, polls)
def generate_events(self):
if not self.senders:
return []
request_ids = list(self.senders)
polls = poll_and_all_reduce(self.senders.values(), self.gloo_group)
self._log_bootstrap_stalls(request_ids, polls)
events = []
events = self._readiness_events(request_ids, polls)
to_remove = []
for req_id, poll in zip(request_ids, polls):
if (
self._local_states[req_id] == TransferPoll.Bootstrapping
and poll == TransferPoll.Bootstrapped
):
logger.debug(
f"[prefill][generate_events] rid={req_id!s} -> BootstrappedEvent",
)
events.append(PD.BootstrappedEvent(req_id))
self._local_states[req_id] = TransferPoll.Bootstrapped
elif poll == TransferPoll.Failed:
for req_id, poll in zip(request_ids, polls, strict=True):
if poll == TransferPoll.Failed:
logger.warning(
f"[prefill][generate_events] rid={req_id!s} -> FailedEvent",
f"[prefill][generate_events] rid={req_id!s} -> FailedEvent"
)
events.append(PD.FailedEvent(req_id))
to_remove.append(req_id)
@@ -419,13 +466,10 @@ class DisaggPrefillExecutor:
):
self._local_states[req_id] = TransferPoll.Success
logger.debug(
f"[prefill][generate_events] rid={req_id!s} -> SucceededEvent",
f"[prefill][generate_events] rid={req_id!s} -> SucceededEvent"
)
events.append(PD.SucceededEvent(req_id))
to_remove.append(req_id)
else:
pass
for req_id in to_remove:
self._drop_request_state(req_id)
return events
@@ -46,6 +46,20 @@ class PdTransferHooks:
# Injected, not reached through the loop — see PauseHooks.
self._device = device
def cancel_request(self, request_id: str, reason: str) -> bool:
transfer = self._loop.kv_transfer
if (
not isinstance(transfer, DisaggPrefillExecutor)
or not transfer.independent_admission
or request_id not in transfer.senders
):
return False
transfer.pending_cancellations[request_id] = reason
if not self._loop.scheduler.can_abort_awaiting_destination(request_id):
return False
transfer.cancel_awaiting_destination(request_id, reason)
return True
def record_prefill_usage(self, request_ids: list[str]) -> None:
"""Forward committed host-side usage before remote-decode dispatch.
@@ -67,6 +81,19 @@ class PdTransferHooks:
request_id, state.output_token_logprobs_val[0]
)
def poll_admission_events(self) -> list:
transfer = self._loop.kv_transfer
if self._loop._pause.forward_blocked or not isinstance(
transfer, DisaggPrefillExecutor
):
return []
for request_id, reason in list(transfer.pending_cancellations.items()):
if self._loop.scheduler.request_token_size(request_id) < 0:
transfer.pending_cancellations.pop(request_id)
elif self._loop.scheduler.can_abort_awaiting_destination(request_id):
transfer.cancel_awaiting_destination(request_id, reason)
return transfer.generate_admission_events()
def poll_transfer_events(self) -> list:
"""Poll the KV transfer executor, act on its events, and return the
(possibly enriched) event list for the scheduler advance. Empty when
@@ -735,6 +735,7 @@ class ServerArgs:
# For PD/EPD disaggregation: "null", "prefill", "decode", or "encode" (vision-tower-only).
disaggregation_mode: str = "null"
disaggregation_decode_allocation_policy: str | None = None
disaggregation_bootstrap_port: int = 8998
disaggregation_transfer_backend: str = "mooncake"
disaggregation_ib_device: str | None = None
@@ -1922,6 +1923,30 @@ class ServerArgs:
)
def resolve_disaggregation(self):
policy = self.disaggregation_decode_allocation_policy
if self.disaggregation_mode == "prefill":
if policy not in ("early", "independent"):
raise ValueError(
"Prefill serving requires explicit "
"--disaggregation-decode-allocation-policy early or independent"
)
elif policy is not None:
raise ValueError("Decode allocation policy is only valid for prefill")
if policy == "independent":
if (
self.mapping.attn.tp_size != 1
or self.mapping.attn.kvp_size != 1
or self.mapping.has_pp
or self.disaggregation_transfer_backend != "mooncake"
):
raise ValueError(
"Independent P/D admission requires attention TP1/KVP1, PP1 and Mooncake"
)
if self.disaggregation_layerwise_interval != 0:
raise ValueError(
"Independent P/D admission requires explicit "
"--disaggregation-layerwise-interval 0"
)
# Pipeline parallelism is a prefill-node-only capability: the chunk
# pipeline needs the P role's structural guarantees (no decode token
# feedback, non-overlap loop).
@@ -4079,6 +4104,13 @@ class ServerArgs:
action="store_true",
help="Enable allreduce fusion for improved decode performance. Auto-enabled on supported single-node TP configurations, and for Beam on any number of nodes (its boundaries fuse where an MNNVL workspace serves them).",
)
parser.add_argument(
"--disaggregation-decode-allocation-policy",
type=str,
choices=["early", "independent"],
default=None,
help="Required on prefill servers: early waits for decode allocation before prefill; independent gates only KV transfer. Decode allocation starts on request arrival in both cases.",
)
parser.add_argument(
"--disaggregation-bootstrap-port",
type=int,
@@ -477,6 +477,7 @@ def test_failed_room_fanout_never_restores_state(
9: (TransferPoll.WaitingForInput if fail_during_commit else TransferPoll.Failed)
}
manager.topology = _topology()
manager.bootstrap_token_cond = threading.Condition()
if fail_during_commit:
manager._validate_cache_room_fanout = lambda _requests: (
manager.request_status.__setitem__(9, TransferPoll.Failed)
@@ -694,6 +695,8 @@ def test_prefill_submits_manifest_through_contract_sender() -> None:
calls = []
executor = object.__new__(prefill_module.DisaggPrefillExecutor)
executor._destination_ready = set()
executor.pending_cancellations = {}
executor._layerwise_enabled = False
executor.cache_layout = layout
executor.senders = {"request-0": _RecordingSender(calls)}
@@ -718,6 +721,8 @@ def test_idle_prefill_rank_submits_final_dummy_rendezvous() -> None:
calls = []
executor = object.__new__(prefill_module.DisaggPrefillExecutor)
executor._destination_ready = set()
executor.pending_cancellations = {}
executor._layerwise_enabled = False
executor.senders = {"request-0": _RecordingSender(calls)}
executor.kv_manager = SimpleNamespace(
@@ -741,6 +746,8 @@ def test_idle_layerwise_prefill_rank_submits_final_dummy_rendezvous() -> None:
calls = []
executor = object.__new__(prefill_module.DisaggPrefillExecutor)
executor._destination_ready = set()
executor.pending_cancellations = {}
executor._layerwise_enabled = True
executor.senders = {"request-0": _RecordingSender(calls)}
executor.kv_manager = SimpleNamespace(
@@ -763,6 +770,8 @@ def test_layerwise_final_preserves_speculative_candidates() -> None:
metadata_calls = []
executor = object.__new__(prefill_module.DisaggPrefillExecutor)
executor._destination_ready = set()
executor.pending_cancellations = {}
executor._layerwise_enabled = True
executor.senders = {"request-0": _FinalLayerwiseSender([])}
executor.kv_manager = SimpleNamespace(
@@ -795,6 +804,8 @@ def test_submission_skips_a_room_aborted_while_queued(layerwise: bool) -> None:
manager.transfer_queues = [FastQueue()]
manager.step_counter = None
executor = object.__new__(DisaggPrefillExecutor)
executor._destination_ready = set()
executor.pending_cancellations = {}
executor.cache_layout = _layout()
executor.kv_manager = manager
executor._layerwise_enabled = layerwise
@@ -883,6 +894,8 @@ def test_submitting_a_live_room_without_destinations_raises(layerwise: bool) ->
manager.request_status = {9: TransferPoll.Bootstrapped}
manager.transfer_infos = {}
executor = object.__new__(DisaggPrefillExecutor)
executor._destination_ready = set()
executor.pending_cancellations = {}
executor._layerwise_enabled = layerwise
executor.senders = {"request-0": _RecordingSender([])}
executor.kv_manager = manager
@@ -236,6 +236,8 @@ def test_prepare_cache_prefill_preflights_batch_and_reserves_once() -> None:
block_tables_arrays=lambda: tables,
)
executor = object.__new__(DisaggPrefillExecutor)
executor._destination_ready = set()
executor.pending_cancellations = {}
executor.cache_layout = layout
executor.senders = {"request-0": _Sender(9), "request-1": _Sender(10)}
executor.kv_manager = manager
@@ -289,6 +291,8 @@ def test_first_layerwise_chunk_includes_deeper_prefill_cache_hit() -> None:
"state-b": np.asarray([[11, 12, 13, 14, 15]], dtype=np.int32),
}
executor = object.__new__(DisaggPrefillExecutor)
executor._destination_ready = set()
executor.pending_cancellations = {}
executor.cache_layout = layout
executor.senders = {"request-0": sender}
executor.kv_manager = manager
+287 -22
View File
@@ -47,6 +47,19 @@ register_cuda_ci(est_time=5, suite="runtime-1gpu")
from test.runtime import cache_pd_test_utils as cache_pd # noqa: E402
from tokenspeed.runtime.engine.cache_hooks import CacheOpHooks # noqa: E402
from tokenspeed.runtime.engine.event_loop import EventLoop # noqa: E402
from tokenspeed.runtime.engine.generation_output_processor import ( # noqa: E402
OutputProcesser,
)
from tokenspeed.runtime.engine.l3_cache_hooks import L3CacheHooks # noqa: E402
from tokenspeed.runtime.engine.pause import ( # noqa: E402
PauseController,
PauseHooks,
PauseState,
)
from tokenspeed.runtime.epd.prefill_hooks import EpdPrefillHooks # noqa: E402
from tokenspeed.runtime.metrics.collector import EngineMetrics # noqa: E402
from tokenspeed.runtime.pd.base.bootstrap import BootstrapInfo # noqa: E402
from tokenspeed.runtime.pd.base.manager import ( # noqa: E402
DisaggManagerBase,
@@ -62,6 +75,7 @@ from tokenspeed.runtime.pd.mooncake.conn import MooncakeKVBootstrapServer # noq
from tokenspeed.runtime.pd.mooncake.entities import KVArgs, KVManagerArgs # noqa: E402
from tokenspeed.runtime.pd.prefill_executor import DisaggPrefillExecutor # noqa: E402
from tokenspeed.runtime.pd.topology import PDParallelTopology # noqa: E402
from tokenspeed.runtime.pd.transfer_hooks import PdTransferHooks # noqa: E402
from tokenspeed.runtime.utils.network import get_free_port, join_host_port # noqa: E402
@@ -238,6 +252,94 @@ def _wait_for(condition) -> None:
time.sleep(0.01)
def _loopback_scheduler(role, allocation_policy):
import tokenspeed_scheduler as native
from tokenspeed.runtime.engine.scheduler_utils import make_config
config = make_config(
num_device_pages=16,
max_scheduled_tokens=8,
max_batch_size=4,
prefix_granularity=2,
num_host_pages=0,
disable_l2_cache=True,
enable_l3_storage=False,
role=role,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
decode_allocation_policy=allocation_policy if role == "prefill" else None,
cache_groups=[
native.CacheGroupConfig(
"history",
2,
16,
native.AttnKind.Full,
transfer_policy=native.CacheTransferPolicy.FullSuffix,
)
],
)
core = native.Scheduler(config)
spec = native.RequestSpec()
spec.request_id, spec.tokens, spec.max_new_tokens = "req", [1, 2, 3, 4], 8
core.submit_requests([spec])
return core
class _PlanCaptured(Exception):
def __init__(self, plan) -> None:
super().__init__("Model execution is replaced by the loopback test")
self.plan = plan
class _PrefillPlanningLoop(EventLoop):
"""Use real admission hooks and planning without constructing GPU models."""
def __init__(self, scheduler, transfer) -> None:
self.scheduler = scheduler
self.kv_transfer = transfer
self.in_flight_depth = 0
self._pause = PauseController(send_func=lambda _message: None)
self._l3 = None
self._epd_hooks = EpdPrefillHooks(self, admission=None)
self._pd_hooks = PdTransferHooks(self, device=None)
self._cache_hooks = CacheOpHooks(
None,
speculative_algorithm=None,
attn_tp_rank=0,
attn_tp_size=1,
attn_tp_cpu_group=None,
pp_size=1,
pp_cpu_group=None,
global_rank=0,
)
self._l3_hooks = L3CacheHooks(
scheduler,
None,
config=None,
attn_tp_size=1,
attn_tp_cpu_group=None,
pp_size=1,
pp_cpu_group=None,
metrics=None,
)
self.output_processor = OutputProcesser(
send_to_tokenizer=lambda _message: None,
metrics=EngineMetrics(labels={}, enabled=False, reporters=[]),
)
def _shutdown_complete(self) -> bool:
return False
def _process_new_requests(self) -> None:
# The test supplies the request directly instead of a tokenizer socket.
pass
def _get_forward_op(self, execution_plan):
raise _PlanCaptured(execution_plan)
@pytest.mark.parametrize(
("host", "mc_use_ipv6", "decode_session"),
[
@@ -245,8 +347,26 @@ def _wait_for(condition) -> None:
pytest.param("::1", "1", "[::1]:15432", marks=requires_ipv6_loopback),
],
)
@pytest.mark.parametrize(
("allocation_policy", "readiness_order"),
[
("early", "decode_first"),
("independent", "decode_first"),
("independent", "frozen_before_grant"),
("independent", "prefill_first"),
("independent", "during_forward"),
("independent", "cancel_before_grant"),
("independent", "cancel_during_forward"),
],
)
def test_request_crosses_pd_control_plane_over_loopback(
monkeypatch, fake_mooncake, host, mc_use_ipv6, decode_session
monkeypatch,
fake_mooncake,
host,
mc_use_ipv6,
decode_session,
allocation_policy,
readiness_order,
):
"""Bootstrap, Decode registration, the request manifest, the transfer and
Prefill's final status between real executors on ``host``, which both roles
@@ -296,6 +416,10 @@ def test_request_crosses_pd_control_plane_over_loopback(
cache_layout=cache_layout,
cache_fields_by_stage=(("layer.0.kv",),),
)
if executor_cls is DisaggPrefillExecutor:
return executor_cls(
args, kv_args, gloo_group=None, admission_policy=allocation_policy
)
return executor_cls(args, kv_args, gloo_group=None)
try:
@@ -306,30 +430,171 @@ def test_request_crosses_pd_control_plane_over_loopback(
assert decode.kv_manager.get_session_id() == decode_session
info = BootstrapInfo(
bootstrap_host=host, bootstrap_port=bootstrap_port, bootstrap_room=7
bootstrap_host=host,
bootstrap_port=bootstrap_port,
bootstrap_room=7,
)
prefill.register("req", info)
# Decode finds Prefill through the bootstrap server and registers.
decode.register("req", info)
status = decode.kv_manager.check_status(7)
assert status == TransferPoll.Bootstrapped, decode.kv_manager.failure_records
decode.receivers["req"].prefill(
block_manifest=cache_pd.block_manifest(("history", (1,)))
)
_wait_for(
lambda: prefill.kv_manager.check_status(7) == TransferPoll.Bootstrapped
import tokenspeed_scheduler as native
from tokenspeed.runtime.engine.scheduler_utils import make_config
# One-rank process-group consensus is identity; only GPU computation and
# the missing RDMA/process-group environment are replaced in this test.
monkeypatch.setattr(
"tokenspeed.runtime.pd.utils.dist.all_reduce", lambda *_a, **_k: None
)
prefill.senders["req"].send(
is_last=True,
bootstrap_token=42,
block_manifest=cache_pd.block_manifest(("history", (2,))),
)
_wait_for(
lambda: decode.kv_manager.check_status(7)
in (TransferPoll.Success, TransferPoll.Failed)
)
assert decode.kv_manager.check_status(7) == TransferPoll.Success
p_core, d_core = _loopback_scheduler(
"prefill", allocation_policy
), _loopback_scheduler("decode", None)
planning_loop = _PrefillPlanningLoop(p_core, prefill)
source_computed = False
def plan_prefill():
with pytest.raises(_PlanCaptured) as captured:
planning_loop.event_loop()
return captured.value.plan
def complete_prefill(plan):
nonlocal source_computed
if plan.remote_decode is not None:
assert source_computed
prefill.execute(plan.remote_decode)
for batch in plan.forward:
for rid in batch.request_ids:
result = native.ForwardEvent.ExtendResult()
result.request_id, result.tokens = rid, [42]
p_core.advance(native.ExecutionEvent().add_event(result))
source_computed = True
events = native.ExecutionEvent()
for event in prefill.generate_events():
events.add_event(event)
p_core.advance(events)
def step_decode():
events = native.ExecutionEvent()
for event in decode.generate_events():
events.add_event(event)
d_core.advance(events)
plan = d_core.next_execution_plan()
if plan.remote_prefill is not None:
decode.execute(plan.remote_prefill)
if readiness_order in ("decode_first", "frozen_before_grant"):
decode.register("req", info)
step_decode()
assert d_core.active_lcm_blocks() > 0
assert not source_computed
_wait_for(
lambda: prefill.kv_manager.room_status(7) == TransferPoll.Bootstrapped
)
# The manifest may arrive before the local request is registered.
prefill.register("req", info)
assert prefill.kv_manager.room_status(7) == TransferPoll.Bootstrapped
if readiness_order == "frozen_before_grant":
frozen = _FrozenPlanningLoop(p_core, prefill)
with pytest.raises(_FrozenRoundComplete):
frozen.event_loop()
assert p_core.bootstrapping_size() == 1
assert p_core.active_lcm_blocks() == 0
if allocation_policy == "early":
assert prefill.generate_admission_events() == []
complete_prefill(plan_prefill())
compute = plan_prefill()
assert any(batch.request_ids for batch in compute.forward)
assert compute.remote_decode is None
assert not written_sessions
complete_prefill(compute)
assert p_core.awaiting_destination_size() == 0
send = plan_prefill()
assert send.remote_decode is not None
assert not any(batch.request_ids for batch in send.forward)
complete_prefill(send)
else:
prefill.register("req", info)
compute = plan_prefill()
assert any(batch.request_ids for batch in compute.forward)
assert d_core.active_lcm_blocks() == 0
assert not written_sessions
if readiness_order == "cancel_during_forward":
assert not planning_loop._pd_hooks.cancel_request("req", "cancelled")
assert p_core.active_lcm_blocks() > 0
if readiness_order in (
"prefill_first",
"cancel_before_grant",
"cancel_during_forward",
):
complete_prefill(compute)
assert p_core.awaiting_destination_size() == 1
assert plan_prefill().remote_decode is None
if readiness_order in ("cancel_before_grant", "cancel_during_forward"):
if readiness_order == "cancel_before_grant":
assert planning_loop._pd_hooks.cancel_request(
"req", "cancelled"
)
events = native.ExecutionEvent()
for event in planning_loop._pd_hooks.poll_transfer_events():
events.add_event(event)
p_core.advance(events)
assert p_core.active_lcm_blocks() == 0
assert not prefill.has_inflight_transfers()
assert not written_sessions
return
decode.register("req", info)
step_decode()
assert d_core.active_lcm_blocks() > 0
_wait_for(
lambda: prefill.kv_manager.room_status(7) == TransferPoll.Bootstrapped
)
if readiness_order == "during_forward":
assert not source_computed
assert not written_sessions
complete_prefill(compute)
assert p_core.awaiting_destination_size() == 0
send = plan_prefill()
assert send.remote_decode is not None
assert not any(batch.request_ids for batch in send.forward)
complete_prefill(send)
def advance_pair():
complete_prefill(plan_prefill())
step_decode()
return d_core.decoding_size() == 1 and not prefill.has_inflight_transfers()
_wait_for(advance_pair)
assert source_computed
assert p_core.active_lcm_blocks() == 0
assert d_core.active_lcm_blocks() > 0
assert written_sessions == [decode_session]
finally:
bootstrap.close()
class _FrozenRoundComplete(Exception):
pass
class _StopAfterPause(PauseHooks):
def paused_idle_step(self):
super().paused_idle_step()
raise _FrozenRoundComplete
class _FrozenPlanningLoop(_PrefillPlanningLoop):
def __init__(self, scheduler, transfer):
super().__init__(scheduler, transfer)
self.requires_dp_sync = False
self._pause.state = PauseState.PAUSED_ALL
self._pause_hooks = _StopAfterPause(self, self._pause, device=None)
def test_frozen_loop_preserves_queued_request_without_l3():
core = _loopback_scheduler("fused", None)
loop = _FrozenPlanningLoop(core, None)
with pytest.raises(_FrozenRoundComplete):
loop.event_loop()
assert core.waiting_size() == 1
assert core.prefilling_size() == 0
assert core.active_lcm_blocks() == 0
plan = core.next_execution_plan()
assert any(batch.request_ids == ["req"] for batch in plan.forward)
+3 -1
View File
@@ -150,7 +150,8 @@ class _EventLoopHarness:
count_ops=lambda _plan: self.trace.append("count_cache"),
)
self._pd_hooks = SimpleNamespace(
poll_transfer_events=lambda: (self.trace.append("poll_pd"), [])[1]
poll_admission_events=lambda: [],
poll_transfer_events=lambda: (self.trace.append("poll_pd"), [])[1],
)
self._eplb_hooks = SimpleNamespace(
note_round=lambda *, forwarded: self.trace.append("eplb_round"),
@@ -237,6 +238,7 @@ def test_abort_uses_the_output_marker() -> None:
)
loop = SimpleNamespace(
output_processor=output,
_pd_hooks=SimpleNamespace(cancel_request=lambda _request_id, _reason: False),
_l3_hooks=L3CacheHooks(
None,
None,
+1
View File
@@ -239,6 +239,7 @@ class TestFinishTailHitCopyCudaTest:
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
+6 -1
View File
@@ -70,7 +70,12 @@ class TestServerArgs:
with pytest.raises(ValueError, match="disaggregation-mode decode"):
ServerArgs(**DP8, attn_head_tp_size=8)
with pytest.raises(ValueError, match="disaggregation-mode decode"):
ServerArgs(**DP8, attn_head_tp_size=8, disaggregation_mode="prefill")
ServerArgs(
**DP8,
attn_head_tp_size=8,
disaggregation_mode="prefill",
disaggregation_decode_allocation_policy="early",
)
def test_head_tp_needs_attention_tp_1(self):
with pytest.raises(ValueError, match="attention TP 1"):
+7 -1
View File
@@ -1081,7 +1081,13 @@ def test_tp1_ep2_draft_checkpoint_and_idle_schedule(monkeypatch, rank, mode):
# The runner's GPU initialization is outside this CPU checkpoint test.
runner = Mock(spec=ModelRunner)
runner.model, runner.mapping, runner.device = adapter, mapping, "cpu"
runner.server_args = ServerArgs(model="unused", disaggregation_mode=mode)
runner.server_args = ServerArgs(
model="unused",
disaggregation_mode=mode,
disaggregation_decode_allocation_policy=(
"early" if mode == "prefill" else None
),
)
kwargs = dict(
spec_num_tokens=6,
spec_num_steps=5,
@@ -0,0 +1,76 @@
# Copyright (c) 2026 LightSeek Foundation
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
"""Decode destination allocation is an explicit serving choice."""
import pytest
from tokenspeed.runtime.utils.server_args import ServerArgs
def test_prefill_requires_explicit_allocation_policy():
with pytest.raises(ValueError, match="requires explicit"):
ServerArgs(model="unused", disaggregation_mode="prefill")
@pytest.mark.parametrize("policy", ["early", "independent"])
def test_prefill_preserves_selected_allocation_policy(policy):
args = ServerArgs(
model="unused",
disaggregation_mode="prefill",
disaggregation_decode_allocation_policy=policy,
disaggregation_layerwise_interval=0,
)
assert args.disaggregation_decode_allocation_policy == policy
assert args.disaggregation_layerwise_interval == 0
@pytest.mark.parametrize("role", ["null", "decode"])
def test_non_prefill_uses_original_admission_without_a_policy(role):
args = ServerArgs(model="unused", disaggregation_mode=role)
assert args.disaggregation_decode_allocation_policy is None
with pytest.raises(ValueError, match="only valid for prefill"):
ServerArgs(
model="unused",
disaggregation_mode=role,
disaggregation_decode_allocation_policy="early",
)
def test_independent_admission_refuses_layerwise_transfer_without_overriding_it():
with pytest.raises(ValueError, match="layerwise-interval 0"):
ServerArgs(
model="unused",
disaggregation_mode="prefill",
disaggregation_decode_allocation_policy="independent",
disaggregation_layerwise_interval=1,
)
def test_independent_admission_refuses_unsupported_attention_topology():
with pytest.raises(ValueError, match="TP1/KVP1"):
ServerArgs(
model="unused",
world_size=2,
attn_tp_size=2,
disaggregation_mode="prefill",
disaggregation_decode_allocation_policy="independent",
disaggregation_layerwise_interval=0,
)
+2
View File
@@ -1318,6 +1318,8 @@ def test_prefill_usage_hook_records_committed_totals_and_skips_retired_requests(
recorded = []
logprobs = []
transfer = object.__new__(DisaggPrefillExecutor)
transfer._destination_ready = set()
transfer.pending_cancellations = {}
transfer.senders = {"hit": SimpleNamespace(bootstrap_room=9)}
transfer.kv_manager = SimpleNamespace(
record_cached_tokens=lambda room, count: recorded.append((room, count)),
+1
View File
@@ -451,6 +451,7 @@ def _decode_batch(bs: int, width: int):
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
@@ -110,6 +110,9 @@ def test_pd_roles_configure_their_pools_as_resolved(role):
model="x",
max_num_seqs=8,
disaggregation_mode=role,
disaggregation_decode_allocation_policy=(
"early" if role == "prefill" else None
),
debug_force_retraction_interval=3,
disaggregation_transfer_backend="mooncake",
)
@@ -79,6 +79,7 @@ def _scheduler(ts, make_config, *, in_forward: bool):
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
@@ -79,6 +79,7 @@ def _scheduler(ts, make_config):
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
+2
View File
@@ -142,6 +142,7 @@ def test_prefill_abort_room_records_failure() -> None:
manager.request_status = {}
manager.failure_records = {}
manager.failure_lock = threading.Lock()
manager.bootstrap_token_cond = threading.Condition()
manager.transfer_infos = {}
manager.abort_room(9, "mooncake transfer ret=-1")
@@ -163,6 +164,7 @@ def test_prefill_abort_room_counts_once_per_room() -> None:
manager.request_status = {}
manager.failure_records = {}
manager.failure_lock = threading.Lock()
manager.bootstrap_token_cond = threading.Condition()
manager.transfer_infos = {}
# Later chunks of the same room funnel back into abort_room after the
@@ -28,6 +28,7 @@ def _make_config(*, prefix_replay_tokens: int | None = None):
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
@@ -143,6 +144,7 @@ def test_make_config_preserves_explicit_l3_storage(enable_l3_storage: bool) -> N
disable_l2_cache=False,
enable_l3_storage=enable_l3_storage,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=(
@@ -82,6 +82,7 @@ def _scheduler(*, l2: bool, force_interval: int) -> ts.Scheduler:
disable_l2_cache=not l2,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=POOL_LCM_BLOCKS + 1,
max_retracted_requests=2,
storage_config=None,
@@ -389,6 +390,7 @@ def test_a_capacity_blocked_round_with_no_fitting_image_aborts_the_newest_reside
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
+22 -3
View File
@@ -129,7 +129,12 @@ def test_non_retracting_roles_resolve_to_no_pool_with_a_log(role, caplog, monkey
monkeypatch.setattr(logging.getLogger(ServerArgs.__module__), "propagate", True)
with caplog.at_level(logging.INFO):
args = ServerArgs(
model="x", disaggregation_mode=role, retraction_snapshot_host_gb=2.0
model="x",
disaggregation_mode=role,
disaggregation_decode_allocation_policy=(
"early" if role == "prefill" else None
),
retraction_snapshot_host_gb=2.0,
)
assert args.retraction_snapshot_pool_disabled
assert (args.retraction_snapshot_host_gb, args.retraction_snapshot_ratio) == (
@@ -143,7 +148,12 @@ def test_non_retracting_roles_resolve_to_no_pool_with_a_log(role, caplog, monkey
caplog.clear()
with caplog.at_level(logging.INFO):
args = ServerArgs(
model="x", disaggregation_mode=role, debug_force_retraction_interval=3
model="x",
disaggregation_mode=role,
disaggregation_decode_allocation_policy=(
"early" if role == "prefill" else None
),
debug_force_retraction_interval=3,
)
assert args.debug_force_retraction_interval == 0
assert any(
@@ -154,7 +164,13 @@ def test_non_retracting_roles_resolve_to_no_pool_with_a_log(role, caplog, monkey
# Nothing to say when nothing was asked for.
caplog.clear()
with caplog.at_level(logging.INFO):
args = ServerArgs(model="x", disaggregation_mode=role)
args = ServerArgs(
model="x",
disaggregation_mode=role,
disaggregation_decode_allocation_policy=(
"early" if role == "prefill" else None
),
)
assert args.retraction_snapshot_pool_disabled
assert not any("never retracts" in record.message for record in caplog.records)
@@ -194,6 +210,9 @@ def test_l3_accepts_kvp_in_every_serving_role(role, dcp_size):
model="x",
world_size=2,
disaggregation_mode=role,
disaggregation_decode_allocation_policy=(
"early" if role == "prefill" else None
),
kv_parallel_size=2,
decode_context_parallel_size=dcp_size,
kvstore_storage_backend="mooncake",
+1
View File
@@ -221,6 +221,7 @@ def _config(**overrides):
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
@@ -116,6 +116,7 @@ class _Engine:
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
@@ -67,7 +67,10 @@ def test_decode_role_accepts_the_flag_and_the_prefill_role_refuses_it():
args = _args(disaggregation_mode="decode")
assert args.enable_speculative_sampling
with pytest.raises(ValueError, match="prefill role"):
_args(disaggregation_mode="prefill")
_args(
disaggregation_mode="prefill",
disaggregation_decode_allocation_policy="early",
)
def test_rl_bitwise_accepts_the_flag_with_a_flashinfer_verifier():
@@ -59,6 +59,7 @@ def _scheduler(*, token_budget: int, with_state: bool, disable_prefix_cache: boo
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
@@ -73,6 +73,7 @@ def _scheduler(*, with_state: bool = False) -> ts.Scheduler:
disable_l2_cache=True,
enable_l3_storage=False,
role="fused",
decode_allocation_policy=None,
num_snapshot_pages=1,
max_retracted_requests=0,
storage_config=None,
+1
View File
@@ -130,6 +130,7 @@ if(TOKENSPEED_SCHEDULER_BUILD_TESTS)
tests/cpp/test_kv_cache_lifecycle.cpp
tests/cpp/test_cache_scenarios.cpp
tests/cpp/test_scheduler_plan.cpp
tests/cpp/test_delayed_decode_admission.cpp
tests/cpp/test_token_level_hits.cpp
tests/cpp/test_outside_event_handler.cpp
tests/cpp/test_cache_operations.cpp
@@ -95,6 +95,10 @@ NB_MODULE(tokenspeed_scheduler_ext, m) {
.value("D", tokenspeed::Role::kD)
.value("Fused", tokenspeed::Role::kFused);
nb::enum_<tokenspeed::DecodeAllocationPolicy>(scheduler_config, "DecodeAllocationPolicy")
.value("Early", tokenspeed::DecodeAllocationPolicy::kEarly)
.value("Independent", tokenspeed::DecodeAllocationPolicy::kIndependent);
nb::enum_<tokenspeed::AttnKind>(m, "AttnKind")
.value("Full", tokenspeed::AttnKind::kFull)
.value("SlidingWindow", tokenspeed::AttnKind::kSlidingWindow)
@@ -141,6 +145,7 @@ NB_MODULE(tokenspeed_scheduler_ext, m) {
.def_rw("decode_input_tokens", &tokenspeed::SchedulerConfig::decode_input_tokens)
.def_rw("overlap_schedule_depth", &tokenspeed::SchedulerConfig::overlap_schedule_depth)
.def_rw("role", &tokenspeed::SchedulerConfig::role)
.def_rw("decode_allocation_policy", &tokenspeed::SchedulerConfig::decode_allocation_policy)
.def_prop_rw(
"num_device_pages", [](const tokenspeed::SchedulerConfig& c) { return c.device_allocator.total_pages; },
[](tokenspeed::SchedulerConfig& c, std::int32_t v) { c.device_allocator.total_pages = v; })
@@ -341,6 +346,10 @@ NB_MODULE(tokenspeed_scheduler_ext, m) {
.def(nb::init<std::string>(), nb::arg("request_id"))
.def_ro("request_id", &tokenspeed::pd::BootstrappedEvent::request_id);
nb::class_<tokenspeed::pd::DestinationReadyEvent>(pd, "DestinationReadyEvent")
.def(nb::init<std::string>(), nb::arg("request_id"))
.def_ro("request_id", &tokenspeed::pd::DestinationReadyEvent::request_id);
nb::class_<tokenspeed::pd::FailedEvent>(pd, "FailedEvent")
.def(nb::init<std::string>(), nb::arg("request_id"))
.def_ro("request_id", &tokenspeed::pd::FailedEvent::request_id);
@@ -570,6 +579,8 @@ NB_MODULE(tokenspeed_scheduler_ext, m) {
.def("decoding_size", &tokenspeed::Scheduler::DecodingSize, "Count requests in the Decoding FSM state.")
.def("prefilling_size", &tokenspeed::Scheduler::PrefillSize,
"Count local/remote prefills, PrefillAwaitingResult, and PrefillDone requests.")
.def("awaiting_destination_size", &tokenspeed::Scheduler::AwaitingDestinationSize)
.def("awaiting_destination_lcm_blocks", &tokenspeed::Scheduler::AwaitingDestinationLcmBlocks)
.def("remote_prefilling_size", &tokenspeed::Scheduler::RemotePrefillSize,
"Count RemotePrefilling requests; these are also included in prefilling_size().")
.def("pd_transfer_size", &tokenspeed::Scheduler::PdTransferSize,
@@ -587,6 +598,7 @@ NB_MODULE(tokenspeed_scheduler_ext, m) {
.def("tail_cap_evictions", &tokenspeed::Scheduler::TailCapEvictions,
"Tails dropped because more than eight were published under one prior page.")
.def("can_abort_before_admission", &tokenspeed::Scheduler::CanAbortBeforeAdmission, nb::arg("id"))
.def("can_abort_awaiting_destination", &tokenspeed::Scheduler::CanAbortAwaitingDestination, nb::arg("id"))
.def("request_token_size", &tokenspeed::Scheduler::RequestTokenSize, nb::arg("id"))
.def("max_single_request_tokens", &tokenspeed::Scheduler::MaxSingleRequestTokens)
.def("clear_l1_cache", &tokenspeed::Scheduler::ClearL1Cache)
@@ -164,6 +164,10 @@ Finished AbortEvent::operator()(PrefillAwaitingResult&& state) {
return abortForward(std::move(state));
}
Finished AbortEvent::operator()(PrefillAwaitingDestination&& state) {
return abortForward(std::move(state));
}
Finished AbortEvent::operator()(Decoding&& state) {
return abortForward(std::move(state));
}
+10 -2
View File
@@ -168,6 +168,7 @@ struct AbortEvent : InvalidTransitionHandler<AbortEvent> {
Finished operator()(RemotePrefilling&& state);
Finished operator()(PrefillDone&& state);
Finished operator()(PrefillAwaitingResult&& state);
Finished operator()(PrefillAwaitingDestination&& state);
Finished operator()(Decoding&& state);
Finished operator()(Retracted&& state);
Finished operator()(Restoring&& state);
@@ -300,7 +301,8 @@ private:
struct ExtendResultEvent : InvalidTransitionHandler<ExtendResultEvent> {
using InvalidTransitionHandler<ExtendResultEvent>::operator();
explicit ExtendResultEvent(std::vector<std::int32_t> result_tokens) : result_tokens_{std::move(result_tokens)} {}
ExtendResultEvent(std::vector<std::int32_t> result_tokens, bool await_destination)
: result_tokens_{std::move(result_tokens)}, await_destination_{await_destination} {}
template <typename State>
requires CanExtendTokenContainer<State>
@@ -317,11 +319,16 @@ struct ExtendResultEvent : InvalidTransitionHandler<ExtendResultEvent> {
// waiting FOR: the prompt's last token is now real, so the request
// becomes schedulable (P: its remote decode can carry the bootstrap
// token).
std::variant<PrefillDone, PrefillAwaitingResult> operator()(PrefillAwaitingResult&& state) {
std::variant<PrefillDone, PrefillAwaitingResult, PrefillAwaitingDestination> operator()(
PrefillAwaitingResult&& state) {
if (result_tokens_.empty()) {
return std::move(state);
}
state.ExtendResultTokens(result_tokens_);
if (await_destination_) {
return PrefillAwaitingDestination{std::move(state.resources), state.window,
state.ReserveNumTokensInNextScheduleEvent()};
}
// Older intermediate chunk results may still be landing on these
// pages (see above): the bundle, in-flight count included, moves on.
return PrefillDone{std::move(state.resources), state.window, state.ReserveNumTokensInNextScheduleEvent()};
@@ -341,6 +348,7 @@ struct ExtendResultEvent : InvalidTransitionHandler<ExtendResultEvent> {
private:
std::vector<std::int32_t> result_tokens_;
bool await_destination_;
};
} // namespace tokenspeed::fsm
@@ -30,6 +30,10 @@
namespace tokenspeed {
namespace fsm {
PrefillDone DestinationReadyEvent::operator()(PrefillAwaitingDestination&& state) {
return PrefillDone{std::move(state.resources), state.window, state.ReserveNumTokensInNextScheduleEvent()};
}
Submitted BootstrappedEvent::operator()(Bootstrapping&& state) {
return Submitted{std::move(state.admission)};
}
@@ -31,6 +31,12 @@ namespace fsm {
struct Bootstrapping;
struct RemotePrefilling;
struct Prefetching;
struct PrefillAwaitingDestination;
struct DestinationReadyEvent : InvalidTransitionHandler<DestinationReadyEvent> {
using InvalidTransitionHandler<DestinationReadyEvent>::operator();
PrefillDone operator()(PrefillAwaitingDestination&& state);
};
struct BootstrappedEvent : InvalidTransitionHandler<BootstrappedEvent> {
using InvalidTransitionHandler<BootstrappedEvent>::operator();
+19
View File
@@ -82,5 +82,24 @@ private:
std::int32_t reserve_num_tokens_in_next_schedule_event_{};
};
// Completed P KV stays pinned until the peer grants destination memory.
struct PrefillAwaitingDestination {
PrefillAwaitingDestination(ForwardResources resources, TokenContainer::Window window,
std::int32_t reserve_num_tokens_in_next_schedule_event)
: resources{std::move(resources)},
window{window},
reserve_num_tokens_in_next_schedule_event_{reserve_num_tokens_in_next_schedule_event} {}
PrefillInfo CurrentPrefillInfo() const { return MakePrefillInfo(resources, window); }
std::int32_t ReserveNumTokensInNextScheduleEvent() const { return reserve_num_tokens_in_next_schedule_event_; }
void ExtendResultTokens(const std::vector<std::int32_t>& result_tokens) { resources.ExtendTokens(result_tokens); }
ForwardResources resources;
TokenContainer::Window window{};
private:
std::int32_t reserve_num_tokens_in_next_schedule_event_{};
};
} // namespace fsm
} // namespace tokenspeed
+3 -2
View File
@@ -30,8 +30,9 @@
// Put resources into each particular state
namespace tokenspeed::fsm {
using State = std::variant<Bootstrapping, Submitted, Prefetching, Prefilling, RemotePrefilling, PrefillAwaitingResult,
PrefillDone, Decoding, Retracted, Restoring, Finishing, Finished>;
using State =
std::variant<Bootstrapping, Submitted, Prefetching, Prefilling, RemotePrefilling, PrefillAwaitingResult,
PrefillAwaitingDestination, PrefillDone, Decoding, Retracted, Restoring, Finishing, Finished>;
inline State ToState(State state) {
return state;
@@ -44,6 +44,21 @@ void Scheduler::handleEvent(const pd::BootstrappedEvent& event) {
}
}
void Scheduler::handleEvent(const pd::DestinationReadyEvent& event) {
Request* request = findRequest(event.request_id);
if (request == nullptr || request->IsAnyOf<fsm::PrefillDone, fsm::Decoding, fsm::Finishing, fsm::Finished>()) {
return;
}
if (config_.role == Role::kP && config_.decode_allocation_policy == DecodeAllocationPolicy::kIndependent) {
request->MarkDestinationReady();
if (request->Is<fsm::PrefillAwaitingDestination>()) {
request->Apply(fsm::DestinationReadyEvent{});
}
return;
}
throw std::logic_error("Destination grant requires independent prefill admission");
}
void Scheduler::handleEvent(const pd::FailedEvent& event) {
Request* request = findRequest(event.request_id);
if (request == nullptr || request->Is<fsm::Finished>()) {
@@ -147,7 +162,10 @@ void Scheduler::handleEvent(const forward::UpdateReserveNumTokens& event) {
void Scheduler::handleEvent(const forward::ExtendResult& event) {
if (Request* request = findRequest(event.request_id)) {
request->NoteResultLanded();
request->Apply(fsm::ExtendResultEvent{event.tokens});
request->Apply(fsm::ExtendResultEvent{
event.tokens, config_.role == Role::kP &&
config_.decode_allocation_policy == DecodeAllocationPolicy::kIndependent &&
!request->DestinationReady()});
if (!event.spec_candidate_ids.empty()) {
request->StoreSpecCandidates(event.spec_candidate_ids);
}
@@ -35,6 +35,6 @@ namespace tokenspeed {
using Event = std::variant<cache::StorageTick, cache::StorageDetach, cache::StorageDone, cache::WriteBackDone,
cache::LoadBackDone, cache::SnapshotDone, cache::RestoreDone, forward::ExtendResult,
forward::Finish, forward::Abort, forward::UpdateReserveNumTokens, pd::BootstrappedEvent,
pd::FailedEvent, pd::SucceededEvent, pd::RemotePrefillDoneEvent>;
pd::DestinationReadyEvent, pd::FailedEvent, pd::SucceededEvent, pd::RemotePrefillDoneEvent>;
} // namespace tokenspeed
@@ -32,6 +32,11 @@ struct BootstrappedEvent {
explicit BootstrappedEvent(std::string id) : request_id(std::move(id)) {}
};
struct DestinationReadyEvent {
std::string request_id;
explicit DestinationReadyEvent(std::string id) : request_id(std::move(id)) {}
};
struct FailedEvent {
std::string request_id;
explicit FailedEvent(std::string id) : request_id(std::move(id)) {}
@@ -51,6 +56,7 @@ struct RemotePrefillDoneEvent {
} // namespace pd
using PDEvent = std::variant<pd::BootstrappedEvent, pd::FailedEvent, pd::SucceededEvent, pd::RemotePrefillDoneEvent>;
using PDEvent = std::variant<pd::BootstrappedEvent, pd::DestinationReadyEvent, pd::FailedEvent, pd::SucceededEvent,
pd::RemotePrefillDoneEvent>;
} // namespace tokenspeed
@@ -53,6 +53,7 @@ PrefillInfo Request::CurrentPrefillInfo() const {
[](const fsm::RemotePrefilling& state) { return state.CurrentPrefillInfo(); },
[](const fsm::PrefillDone& state) { return state.CurrentPrefillInfo(); },
[](const fsm::PrefillAwaitingResult& state) { return state.CurrentPrefillInfo(); },
[](const fsm::PrefillAwaitingDestination& state) { return state.CurrentPrefillInfo(); },
[this](const auto&) -> PrefillInfo {
throw std::logic_error(
"Request::CurrentPrefillInfo: expected Prefilling, RemotePrefilling or PrefillDone; got " +
@@ -81,6 +82,7 @@ std::int32_t Request::NumComputedTokens() const {
Overloaded{
[](const fsm::Prefilling& state) { return state.window.begin + state.window.size; },
[](const fsm::PrefillDone& state) { return state.window.begin + state.window.size; },
[](const fsm::PrefillAwaitingDestination& state) { return state.window.begin + state.window.size; },
// Every token but the last is computed; the last is the sampled
// token no landed forward consumed.
[this](const fsm::Decoding&) { return TokenSize() - 1; },
+23 -15
View File
@@ -47,6 +47,8 @@ public:
Request(const RequestSpec& spec, std::int32_t prefix_granularity, Role role);
const std::string& Id() const { return id_; }
bool DestinationReady() const { return destination_ready_; }
void MarkDestinationReady() { destination_ready_ = true; }
// Decode headroom an admission must secure before the prefill starts:
// one safe-step window up front, plus one more per retraction suffered
@@ -283,6 +285,9 @@ public:
Overloaded{
[](const fsm::PrefillDone& state) { return state.ReserveNumTokensInNextScheduleEvent(); },
[](const fsm::PrefillAwaitingResult& state) { return state.ReserveNumTokensInNextScheduleEvent(); },
[](const fsm::PrefillAwaitingDestination& state) {
return state.ReserveNumTokensInNextScheduleEvent();
},
[](const fsm::Decoding& state) { return state.ReserveNumTokensInNextScheduleEvent(); },
[this](const auto&) -> std::int32_t {
throw std::logic_error(
@@ -294,27 +299,30 @@ public:
}
std::string StateName() const {
return std::visit(Overloaded{
[](const fsm::Bootstrapping&) -> std::string { return "Bootstrapping"; },
[](const fsm::Submitted&) -> std::string { return "Submitted"; },
[](const fsm::Prefetching&) -> std::string { return "Prefetching"; },
[](const fsm::Prefilling&) -> std::string { return "Prefilling"; },
[](const fsm::RemotePrefilling&) -> std::string { return "RemotePrefilling"; },
[](const fsm::PrefillAwaitingResult&) -> std::string { return "PrefillAwaitingResult"; },
[](const fsm::PrefillDone&) -> std::string { return "PrefillDone"; },
[](const fsm::Decoding&) -> std::string { return "Decoding"; },
[](const fsm::Retracted&) -> std::string { return "Retracted"; },
[](const fsm::Restoring&) -> std::string { return "Restoring"; },
[](const fsm::Finishing&) -> std::string { return "Finishing"; },
[](const fsm::Finished&) -> std::string { return "Finished"; },
},
state_);
return std::visit(
Overloaded{
[](const fsm::Bootstrapping&) -> std::string { return "Bootstrapping"; },
[](const fsm::Submitted&) -> std::string { return "Submitted"; },
[](const fsm::Prefetching&) -> std::string { return "Prefetching"; },
[](const fsm::Prefilling&) -> std::string { return "Prefilling"; },
[](const fsm::RemotePrefilling&) -> std::string { return "RemotePrefilling"; },
[](const fsm::PrefillAwaitingResult&) -> std::string { return "PrefillAwaitingResult"; },
[](const fsm::PrefillAwaitingDestination&) -> std::string { return "PrefillAwaitingDestination"; },
[](const fsm::PrefillDone&) -> std::string { return "PrefillDone"; },
[](const fsm::Decoding&) -> std::string { return "Decoding"; },
[](const fsm::Retracted&) -> std::string { return "Retracted"; },
[](const fsm::Restoring&) -> std::string { return "Restoring"; },
[](const fsm::Finishing&) -> std::string { return "Finishing"; },
[](const fsm::Finished&) -> std::string { return "Finished"; },
},
state_);
}
private:
fsm::ForwardResources& forwardResources(const char* operation);
const fsm::ForwardResources& forwardResources(const char* operation) const;
bool destination_ready_{false};
std::string id_;
TokenContainer token_container_;
std::int32_t submitted_prompt_size_{0};
@@ -401,10 +401,24 @@ std::size_t Scheduler::DecodingSize() const {
std::size_t Scheduler::PrefillSize() const {
return static_cast<std::size_t>(std::ranges::count_if(requests_, [](const std::unique_ptr<Request>& request) {
return request->IsAnyOf<fsm::Prefilling, fsm::RemotePrefilling, fsm::PrefillAwaitingResult, fsm::PrefillDone>();
return request->IsAnyOf<fsm::Prefilling, fsm::RemotePrefilling, fsm::PrefillAwaitingResult,
fsm::PrefillAwaitingDestination, fsm::PrefillDone>();
}));
}
std::size_t Scheduler::AwaitingDestinationSize() const {
return static_cast<std::size_t>(std::ranges::count_if(
requests_, [](const auto& request) { return request->template Is<fsm::PrefillAwaitingDestination>(); }));
}
std::int32_t Scheduler::AwaitingDestinationLcmBlocks() const {
std::vector<std::span<const BlockTable>> tables;
for (const auto& request : requests_) {
if (request->Is<fsm::PrefillAwaitingDestination>()) tables.emplace_back(request->BlockTablesRef());
}
return coordinator_.NumActiveLcmBlocks(tables);
}
std::size_t Scheduler::RemotePrefillSize() const {
return static_cast<std::size_t>(std::ranges::count_if(
requests_, [](const std::unique_ptr<Request>& request) { return request->Is<fsm::RemotePrefilling>(); }));
@@ -458,6 +472,12 @@ bool Scheduler::CanAbortBeforeAdmission(const std::string& id) const {
it->second->IsAnyOf<fsm::Submitted, fsm::Prefetching>();
}
bool Scheduler::CanAbortAwaitingDestination(const std::string& id) const {
const auto it = requests_by_id_.find(id);
return it != requests_by_id_.end() && it->second->ResultsInFlight() == 0 &&
it->second->Is<fsm::PrefillAwaitingDestination>();
}
ExecutionPlan Scheduler::NextExecutionPlan() {
std::vector<WriteBackOperation> write_back_operations = std::exchange(pending_write_back_operations_, {});
std::erase_if(requests_, [this](const auto& request) {
@@ -85,6 +85,8 @@ public:
std::size_t WaitingSize() const;
std::size_t DecodingSize() const;
std::size_t PrefillSize() const;
std::size_t AwaitingDestinationSize() const;
std::int32_t AwaitingDestinationLcmBlocks() const;
// Device pool parents that are empty or hold only unpinned cache entries.
// Scans the pool and every prefix index: a leak check for tests and
// diagnostics, not a per-step gauge.
@@ -102,6 +104,7 @@ public:
// Only fused, page-less requests can finish without the normal forward/PD
// completion fence. This is a read-only check for the head event channel.
bool CanAbortBeforeAdmission(const std::string& id) const;
bool CanAbortAwaitingDestination(const std::string& id) const;
// Maximum logical request extent that one request can reserve in an
// otherwise reclaimable device pool. The runtime must enforce this limit
// before submitting requests.
@@ -259,6 +262,7 @@ private:
void handleEvent(const cache::SnapshotDone& event);
void handleEvent(const cache::RestoreDone& event);
void handleEvent(const pd::BootstrappedEvent& event);
void handleEvent(const pd::DestinationReadyEvent& event);
void handleEvent(const pd::FailedEvent& event);
void handleEvent(const pd::SucceededEvent& event);
void handleEvent(const pd::RemotePrefillDoneEvent& event);
@@ -53,6 +53,9 @@ void validateGroupCapacityInputs(const SchedulerConfig& config, const CacheGroup
void SchedulerConfig::Validate() const {
ValidateCapacityInputs();
if ((role == Role::kP) != decode_allocation_policy.has_value()) {
throw std::invalid_argument("Decode allocation policy is required only for prefill");
}
if (device_allocator.total_pages <= 1) {
throw std::invalid_argument("Scheduler: device cache must contain a null page and usable capacity");
}
@@ -85,6 +85,7 @@ struct SchedulerConfig {
// (transfer pins, destination layouts, transfer_policy validation) keys
// on the role directly.
Role role{Role::kFused};
std::optional<DecodeAllocationPolicy> decode_allocation_policy;
bool disable_prefix_cache{false};
// Minimum prompt tail that must be recomputed after a prefix-cache hit.
+5
View File
@@ -147,6 +147,11 @@ template <typename T>
concept CanExtendTokenContainer =
requires(T& state, std::vector<std::int32_t> tokens) { state.ExtendResultTokens(tokens); };
enum class DecodeAllocationPolicy {
kEarly,
kIndependent,
};
enum class Role {
kP,
kD,
@@ -28,6 +28,9 @@ from conftest import _advance, _spec, storage_ops
def scheduler(*, role, groups=1, host=32, timeout=100):
cfg = ts.SchedulerConfig()
cfg.role = role
cfg.decode_allocation_policy = (
type(cfg).DecodeAllocationPolicy.Early if cfg.role == type(cfg).Role.P else None
)
cfg.prefix_granularity = 2
cfg.num_snapshot_pages = 1
cfg.num_device_pages = 64
@@ -427,6 +430,9 @@ def test_prefill_storage_preparation_survives_bootstrap(bootstrap_before_get):
def mixed_replay_scheduler():
cfg = ts.SchedulerConfig()
cfg.role = ts.SchedulerConfig.Role.Fused
cfg.decode_allocation_policy = (
type(cfg).DecodeAllocationPolicy.Early if cfg.role == type(cfg).Role.P else None
)
cfg.prefix_granularity = 2
cfg.num_device_pages = 64
cfg.num_host_pages = 32
@@ -154,6 +154,7 @@ def test_snapshot_pool_is_stated_explicitly():
assert scheduler.retracted_size() == 0
assert scheduler.host_pool_pinned_blocks() == 0
cfg.role = ts.SchedulerConfig.Role.P
cfg.decode_allocation_policy = type(cfg).DecodeAllocationPolicy.Early
cfg.cache_groups[0].transfer_policy = ts.CacheTransferPolicy.FullSuffix
with pytest.raises(ValueError, match="P role"):
ts.Scheduler(cfg)
@@ -28,6 +28,9 @@ def _sizing_config(
"""A config as the recipes hand it to the model: no page counts yet."""
cfg = ts.SchedulerConfig()
cfg.role = role
cfg.decode_allocation_policy = (
type(cfg).DecodeAllocationPolicy.Early if cfg.role == type(cfg).Role.P else None
)
cfg.prefix_granularity = prefix_granularity
cfg.max_scheduled_tokens = max_scheduled_tokens
cfg.max_batch_size = max_batch_size
@@ -60,6 +60,9 @@ def _config(
cfg.decode_input_tokens = 1
cfg.overlap_schedule_depth = 1
cfg.role = role
cfg.decode_allocation_policy = (
type(cfg).DecodeAllocationPolicy.Early if cfg.role == type(cfg).Role.P else None
)
groups = [
ts.CacheGroupConfig(
group_id=HISTORY,
@@ -300,6 +300,7 @@ class TestPrefillFirst:
dispatches in one round."""
cfg = make_config(max_scheduled_tokens=8)
cfg.role = SchedulerConfig.Role.P
cfg.decode_allocation_policy = type(cfg).DecodeAllocationPolicy.Early
cfg.decode_input_tokens = 0
cfg.cache_groups[0].transfer_policy = CacheTransferPolicy.FullSuffix
s = Scheduler(cfg)
@@ -126,6 +126,7 @@ def test_prefill_role_reserves_the_decode_window_on_the_completing_chunk():
like a decoding role; intermediate chunks hold only their own tokens."""
cfg = SchedulerConfig()
cfg.role = SchedulerConfig.Role.P
cfg.decode_allocation_policy = type(cfg).DecodeAllocationPolicy.Early
cfg.prefix_granularity = 2
cfg.max_scheduled_tokens = 4
cfg.max_batch_size = 1
@@ -165,6 +166,7 @@ def test_prefill_role_hands_off_without_the_draft_window():
verify window: a pool sized for the prompt and that window still hands off."""
cfg = SchedulerConfig()
cfg.role = SchedulerConfig.Role.P
cfg.decode_allocation_policy = type(cfg).DecodeAllocationPolicy.Early
cfg.prefix_granularity = 1
cfg.max_scheduled_tokens = 8
cfg.max_batch_size = 1
@@ -60,6 +60,9 @@ def _config(
cfg.decode_input_tokens = 1
cfg.overlap_schedule_depth = overlap_schedule_depth
cfg.role = ts.SchedulerConfig.Role.Fused
cfg.decode_allocation_policy = (
type(cfg).DecodeAllocationPolicy.Early if cfg.role == type(cfg).Role.P else None
)
cfg.disable_prefix_cache = not prefix_cache
cfg.cache_groups = [
ts.CacheGroupConfig(
@@ -555,6 +555,7 @@ TEST(CacheOperationTest, PrefillAcceptsPromptThatFitsWithoutReservingDecodeToken
config.max_scheduled_tokens = 8;
config.max_batch_size = 2;
config.role = Role::kP;
config.decode_allocation_policy = DecodeAllocationPolicy::kEarly;
config.cache_groups.push_back(CacheGroupConfig{
.group_id = "full",
.block_granularity = 2,
@@ -3015,10 +3015,10 @@ TEST(ExtendResultEvent, AwaitingResultAbsorbsEmptyIntermediateResults) {
/*awaits_result=*/true});
ASSERT_TRUE(request.Is<fsm::PrefillAwaitingResult>());
request.Apply(fsm::ExtendResultEvent{{}}); // an older intermediate chunk's empty result
request.Apply(fsm::ExtendResultEvent{{}, false}); // an older intermediate chunk's empty result
EXPECT_TRUE(request.Is<fsm::PrefillAwaitingResult>()) << "an empty result must not end the wait";
request.Apply(fsm::ExtendResultEvent{{42}}); // the final chunk's token
request.Apply(fsm::ExtendResultEvent{{42}, false}); // the final chunk's token
EXPECT_TRUE(request.Is<fsm::PrefillDone>());
EXPECT_EQ(request.LastToken(), 42);
}
@@ -3059,12 +3059,12 @@ TEST(ExtendResultEvent, AwaitingResultCarriesForwardsInFlightIntoPrefillDone) {
ASSERT_EQ(request.ResultsInFlight(), 2);
request.NoteResultLanded();
request.Apply(fsm::ExtendResultEvent{{42}}); // the final chunk's token, ahead of chunk 1's result
request.Apply(fsm::ExtendResultEvent{{42}, false}); // the final chunk's token, ahead of chunk 1's result
EXPECT_TRUE(request.Is<fsm::PrefillDone>());
EXPECT_EQ(request.ResultsInFlight(), 1) << "chunk 1's forward is still out against these pages";
request.NoteResultLanded();
request.Apply(fsm::ExtendResultEvent{{}}); // chunk 1's empty result, late
request.Apply(fsm::ExtendResultEvent{{}, false}); // chunk 1's empty result, late
EXPECT_TRUE(request.Is<fsm::PrefillDone>());
EXPECT_EQ(request.ResultsInFlight(), 0);
}
@@ -3098,7 +3098,7 @@ TEST(SnapshotRetractEvent, StampsResumePriorityAndKeepsTheResumePoint) {
ASSERT_TRUE(request.Is<fsm::Prefilling>());
request.TrackScheduledForward();
request.NoteResultLanded();
request.Apply(fsm::ExtendResultEvent{{}}); // the first chunk landed
request.Apply(fsm::ExtendResultEvent{{}, false}); // the first chunk landed
RetractForTest(request, coordinator, slots, /*epoch=*/1);
const auto* retracted = request.GetIf<fsm::Retracted>();
@@ -3164,9 +3164,9 @@ TEST(RetractionHeadroom, ReservesOnlyTheRemainingGenerationBudget) {
fsm::PrefillSource::kLocal, &coordinator, std::move(tables),
/*hit_tokens=*/0, fsm::CacheProgress{},
/*load_pairs=*/{}, /*awaits_result=*/false});
request.Apply(fsm::ExtendResultEvent{{42}});
request.Apply(fsm::ExtendResultEvent{{42}, false});
request.Apply(fsm::ScheduleDecodeEvent{/*decode_input_tokens=*/1});
request.Apply(fsm::ExtendResultEvent{std::vector<std::int32_t>(999, 7)});
request.Apply(fsm::ExtendResultEvent{std::vector<std::int32_t>(999, 7), false});
ASSERT_EQ(request.GeneratedTokens(), 1000);
constexpr std::int32_t kSafeSteps = 4096;
@@ -3224,9 +3224,9 @@ TEST(RetractionHeadroom, SpendingTheWindowNeverMakesItCoverTheRemainder) {
fsm::PrefillSource::kLocal, &coordinator, std::move(tables),
/*hit_tokens=*/0, fsm::CacheProgress{},
/*load_pairs=*/{}, /*awaits_result=*/false});
request.Apply(fsm::ExtendResultEvent{{42}});
request.Apply(fsm::ExtendResultEvent{{42}, false});
request.Apply(fsm::ScheduleDecodeEvent{/*decode_input_tokens=*/1});
request.Apply(fsm::ExtendResultEvent{std::vector<std::int32_t>(4096, 7)});
request.Apply(fsm::ExtendResultEvent{std::vector<std::int32_t>(4096, 7), false});
ASSERT_EQ(request.GeneratedTokens(), 4097);
constexpr std::int32_t kSafeSteps = 4096;
@@ -3275,6 +3275,7 @@ protected:
SchedulerConfig MakeConfig() override {
SchedulerConfig cfg = RetractSuite::MakeConfig();
cfg.role = Role::kP;
cfg.decode_allocation_policy = DecodeAllocationPolicy::kEarly;
for (CacheGroupConfig& group : cfg.cache_groups) {
group.transfer_policy = CacheTransferPolicy::FullSuffix;
}
@@ -3967,7 +3968,7 @@ TEST(SnapshotRetractEvent, APrefillDoneVictimResumesAsPrefillDone) {
ASSERT_LT(pool.NumEmptyLcmBlocks(), 8);
// The last chunk's ExtendResult lands while still PrefillDone.
request.Apply(fsm::ExtendResultEvent{{42}});
request.Apply(fsm::ExtendResultEvent{{42}, false});
RetractForTest(request, coordinator, slots, /*epoch=*/1);
EXPECT_TRUE(request.Is<fsm::Retracted>());
@@ -6367,6 +6368,7 @@ protected:
SchedulerConfig MakeConfig() override {
SchedulerConfig cfg = BoundedReplaySuite::MakeConfig();
cfg.role = RoleUnderTest();
if (cfg.role == Role::kP) cfg.decode_allocation_policy = DecodeAllocationPolicy::kEarly;
for (CacheGroupConfig& group : cfg.cache_groups) {
group.transfer_policy = CacheTransferPolicy::FullSuffix;
}
@@ -72,6 +72,7 @@ SchedulerConfig SizingConfig(Role role, std::int32_t prefix_granularity, std::in
std::vector<CacheGroupConfig> groups) {
SchedulerConfig cfg{};
cfg.role = role;
cfg.decode_allocation_policy = role == Role::kP ? std::optional{DecodeAllocationPolicy::kEarly} : std::nullopt;
cfg.prefix_granularity = prefix_granularity;
cfg.max_scheduled_tokens = chunk_tokens;
cfg.max_batch_size = 1;
@@ -200,6 +201,7 @@ TEST(CapacityModelTest, ConcurrentDemandByKind) {
// role.
SchedulerConfig prefill = cfg;
prefill.role = Role::kP;
prefill.decode_allocation_policy = DecodeAllocationPolicy::kEarly;
for (CacheGroupConfig& group : prefill.cache_groups) {
group.transfer_policy =
group.attn_kind == AttnKind::kState ? CacheTransferPolicy::LatestSnapshot : CacheTransferPolicy::FullSuffix;
@@ -0,0 +1,308 @@
// Copyright (c) 2026 LightSeek Foundation
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#include "integration_test_helper.h"
namespace tokenspeed::test {
class IndependentAdmissionTest : public SchedulerTestSuite {
protected:
SchedulerConfig MakeConfig() override {
auto cfg = SchedulerTestSuite::MakeConfig();
cfg.role = Role::kP;
cfg.decode_allocation_policy = DecodeAllocationPolicy::kIndependent;
cfg.disable_l2_cache = true;
cfg.host_allocator.total_pages = 0;
cfg.max_scheduled_tokens = 4;
cfg.decode_input_tokens = 4;
cfg.cache_groups.front().transfer_policy = CacheTransferPolicy::FullSuffix;
cfg.cache_groups.push_back(CacheGroupConfig{
.group_id = "state",
.block_granularity = 0,
.total_pages = cfg.device_allocator.total_pages,
.attn_kind = AttnKind::kState,
.transfer_policy = CacheTransferPolicy::LatestSnapshot,
});
SetTestSnapshotPool(cfg);
return cfg;
}
void SubmitReady(const std::string& id, int length, int start = 1) {
auto request = MakeRequestSpec(id, (length + 1) / 2, start);
request.tokens.resize(length);
request.max_new_tokens = 64;
Submit(request);
scheduler_->Advance(ExecutionEvent{}.With(pd::BootstrappedEvent{id}));
}
void Grant(const std::string& id) { scheduler_->Advance(ExecutionEvent{}.With(pd::DestinationReadyEvent{id})); }
void Ack(const std::string& id) { scheduler_->Advance(ExecutionEvent{}.With(pd::SucceededEvent{id})); }
};
TEST_F(IndependentAdmissionTest, FinalResultRetainsEveryGroupUntilGrant) {
SubmitReady("r", 6);
const auto first = PlanOnce();
ASSERT_EQ(FindForwardBatch(first)->input_lengths, std::vector<std::int32_t>{4});
SendForwardDone("r");
const auto last = PlanOnce();
ASSERT_EQ(FindForwardBatch(last)->input_lengths, std::vector<std::int32_t>{2});
const auto source_blocks = FindForwardBatch(last)->block_tables.at("full_attention").front();
EXPECT_FALSE(last.remote_decode.has_value());
PlanOnce();
scheduler_->Advance(ExecutionEvent{}.With(
forward::ExtendResult{.request_id = "r", .tokens = {42}, .spec_candidate_ids = {42, 43, 44}}));
const auto ready = PlanOnce();
EXPECT_FALSE(ready.remote_decode.has_value());
EXPECT_EQ(scheduler_->AwaitingDestinationSize(), 1u);
EXPECT_EQ(scheduler_->AwaitingDestinationLcmBlocks(), scheduler_->ActiveLcmBlocks());
EXPECT_TRUE(scheduler_->PdTransferPinned("r"));
EXPECT_FALSE(scheduler_->ClearCache());
const auto held = scheduler_->ActiveLcmBlocks();
for (int i = 0; i < 3; ++i) {
const auto wait = PlanOnce();
EXPECT_FALSE(wait.remote_decode.has_value());
EXPECT_EQ(scheduler_->ActiveLcmBlocks(), held);
}
Grant("r");
Grant("r");
const auto send = PlanOnce();
ASSERT_TRUE(send.remote_decode.has_value());
EXPECT_EQ(send.remote_decode->request_ids, std::vector<std::string>{"r"});
EXPECT_EQ(send.remote_decode->decode_input_ids, std::vector<std::int32_t>{42});
EXPECT_EQ(send.remote_decode->spec_candidate_ids.front(), (std::vector<std::int32_t>{42, 43, 44}));
EXPECT_EQ(send.remote_decode->block_tables.at("full_attention").front(), source_blocks);
EXPECT_EQ(scheduler_->AwaitingDestinationSize(), 0u);
EXPECT_TRUE(scheduler_->PdTransferPinned("r"));
for (const auto& [group, blocks] : send.pages_to_zero) EXPECT_TRUE(blocks.empty());
Ack("r");
EXPECT_EQ(scheduler_->ActiveLcmBlocks(), 0);
EXPECT_TRUE(scheduler_->ClearCache());
}
TEST_F(IndependentAdmissionTest, CancellationIgnoresLateGrant) {
SubmitReady("r", 4);
PlanOnce();
SendForwardDone("r", {42});
SendAbortEvent("r");
Grant("r");
const auto cancelled = PlanOnce();
EXPECT_FALSE(cancelled.remote_decode.has_value());
EXPECT_EQ(scheduler_->ActiveLcmBlocks(), 0);
EXPECT_EQ(scheduler_->AwaitingDestinationSize(), 0u);
}
TEST_F(IndependentAdmissionTest, CancellingWhileWaitingReleasesResourcesWithoutTransfer) {
SubmitReady("r", 4);
PlanOnce();
SendForwardDone("r", {42});
PlanOnce();
SendAbortEvent("r");
const auto cancelled = PlanOnce();
EXPECT_FALSE(cancelled.remote_decode.has_value());
EXPECT_EQ(scheduler_->ActiveLcmBlocks(), 0);
Grant("r");
}
TEST_F(IndependentAdmissionTest, EarlyGrantIsRememberedWithoutSendingUncomputedKv) {
SubmitReady("r", 6);
Grant("r");
Grant("r");
const auto first = PlanOnce();
ASSERT_EQ(FindForwardBatch(first)->input_lengths, std::vector<std::int32_t>{4});
EXPECT_FALSE(first.remote_decode.has_value());
SendForwardDone("r");
const auto last = PlanOnce();
ASSERT_EQ(FindForwardBatch(last)->input_lengths, std::vector<std::int32_t>{2});
EXPECT_FALSE(last.remote_decode.has_value());
SendForwardDone("r", {42});
EXPECT_EQ(scheduler_->AwaitingDestinationSize(), 0u);
const auto send = PlanOnce();
ASSERT_TRUE(send.remote_decode.has_value());
EXPECT_EQ(send.remote_decode->request_ids, std::vector<std::string>{"r"});
Ack("r");
EXPECT_EQ(scheduler_->ActiveLcmBlocks(), 0);
}
class IndependentAdmissionFullPoolTest : public IndependentAdmissionTest, public ::testing::WithParamInterface<bool> {
protected:
SchedulerConfig MakeConfig() override {
auto cfg = IndependentAdmissionTest::MakeConfig();
cfg.cache_groups.resize(GetParam() ? 2 : 1);
cfg.device_allocator.total_pages = GetParam() ? 11 : 7;
for (auto& group : cfg.cache_groups) group.total_pages = cfg.device_allocator.total_pages;
cfg.disable_prefix_cache = true;
cfg.max_scheduled_tokens = 8;
return cfg;
}
};
TEST_P(IndependentAdmissionFullPoolTest, GrantDrainsCompletedPromptAndUnblocksPartialPromptInAFullPool) {
SubmitReady("first", 4);
SubmitReady("partial", 6, 100);
const auto compute = PlanOnce();
ASSERT_EQ(FindForwardBatch(compute)->request_ids, (std::vector<std::string>{"first", "partial"}));
EXPECT_EQ(FindForwardBatch(compute)->input_lengths, (std::vector<std::int32_t>{4, 4}));
EXPECT_EQ(scheduler_->EmptyLcmBlocks(), 0);
SendForwardDone("first", {42});
SendForwardDone("partial");
const auto ready = PlanOnce();
EXPECT_TRUE(FindForwardBatch(ready)->request_ids.empty());
Grant("first");
const auto send = PlanOnce();
ASSERT_TRUE(send.remote_decode.has_value());
EXPECT_EQ(send.remote_decode->request_ids, std::vector<std::string>{"first"});
EXPECT_TRUE(send.pages_to_zero.at("full_attention").empty());
Ack("first");
const auto resume = PlanOnce();
ASSERT_EQ(FindForwardBatch(resume)->request_ids, std::vector<std::string>{"partial"});
EXPECT_EQ(FindForwardBatch(resume)->input_lengths, std::vector<std::int32_t>{2});
SendForwardDone("partial", {43});
PlanOnce();
Grant("partial");
ASSERT_TRUE(PlanOnce().remote_decode.has_value());
Ack("partial");
EXPECT_EQ(scheduler_->ActiveLcmBlocks(), 0);
}
INSTANTIATE_TEST_SUITE_P(HistoryAndState, IndependentAdmissionFullPoolTest, ::testing::Bool());
TEST_F(IndependentAdmissionTest, GrantDuringFinalForwardSendsAsSoonAsItsResultLands) {
SubmitReady("r", 4);
PlanOnce();
Grant("r");
EXPECT_FALSE(PlanOnce().remote_decode.has_value());
SendForwardDone("r", {42});
const auto send = PlanOnce();
ASSERT_TRUE(send.remote_decode.has_value());
EXPECT_EQ(send.remote_decode->decode_input_ids, std::vector<std::int32_t>{42});
Ack("r");
EXPECT_EQ(scheduler_->ActiveLcmBlocks(), 0);
}
TEST_F(IndependentAdmissionTest, DecodeFailureWhileAwaitingGrantReleasesAllSourceGroups) {
SubmitReady("r", 4);
PlanOnce();
SendForwardDone("r", {42});
PlanOnce();
scheduler_->Advance(ExecutionEvent{}.With(pd::FailedEvent{"r"}));
const auto failed = PlanOnce();
EXPECT_FALSE(failed.remote_decode.has_value());
EXPECT_EQ(scheduler_->ActiveLcmBlocks(), 0);
}
TEST_F(IndependentAdmissionTest, EarlyPolicySendsWithoutASeparateDestinationGrant) {
auto cfg = MakeConfig();
cfg.decode_allocation_policy = DecodeAllocationPolicy::kEarly;
Scheduler early{cfg};
early.SubmitRequests({RequestSpec{.request_id = "r", .tokens = {1, 2, 3, 4}, .max_new_tokens = 8}});
early.Advance(ExecutionEvent{}.With(pd::BootstrappedEvent{"r"}));
early.NextExecutionPlan();
early.Advance(ExecutionEvent{}.With(forward::ExtendResult{.request_id = "r", .tokens = {42}}));
const auto send = early.NextExecutionPlan();
EXPECT_TRUE(send.remote_decode.has_value());
}
TEST_F(IndependentAdmissionTest, DecodeAllocatesWithoutWaitingForPrefillCompletion) {
auto cfg = MakeConfig();
cfg.role = Role::kD;
cfg.decode_allocation_policy.reset();
SetTestSnapshotPool(cfg);
Scheduler decode{cfg};
decode.SubmitRequests({RequestSpec{.request_id = "r", .tokens = {1, 2, 3, 4}, .max_new_tokens = 8}});
for (int i = 0; i < 3; ++i) {
const auto plan = decode.NextExecutionPlan();
EXPECT_FALSE(plan.remote_prefill.has_value());
EXPECT_EQ(decode.ActiveLcmBlocks(), 0);
EXPECT_EQ(decode.BootstrappingSize(), 1u);
}
decode.Advance(ExecutionEvent{}.With(pd::BootstrappedEvent{"r"}));
const auto receive = decode.NextExecutionPlan();
ASSERT_TRUE(receive.remote_prefill.has_value());
EXPECT_GT(decode.ActiveLcmBlocks(), 0);
EXPECT_EQ(decode.RemotePrefillSize(), 1u);
}
TEST_F(IndependentAdmissionTest, DecodeAdmissionPressureDoesNotBlockPrefillComputation) {
auto cfg = MakeConfig();
cfg.role = Role::kD;
cfg.decode_allocation_policy.reset();
cfg.max_batch_size = 1;
SetTestSnapshotPool(cfg);
Scheduler decode{cfg};
decode.SubmitRequests({RequestSpec{.request_id = "first", .tokens = {1, 2, 3, 4}, .max_new_tokens = 8},
RequestSpec{.request_id = "later", .tokens = {101, 102, 103, 104}, .max_new_tokens = 8}});
decode.Advance(ExecutionEvent{}.With(pd::BootstrappedEvent{"first"}).With(pd::BootstrappedEvent{"later"}));
const auto first = decode.NextExecutionPlan();
ASSERT_TRUE(first.remote_prefill.has_value());
EXPECT_EQ(first.remote_prefill->request_ids, std::vector<std::string>{"first"});
EXPECT_FALSE(decode.NextExecutionPlan().remote_prefill.has_value());
EXPECT_EQ(decode.WaitingSize(), 1u);
SubmitReady("later", 4, 100);
ASSERT_EQ(FindForwardBatch(PlanOnce())->input_lengths, std::vector<std::int32_t>{4});
SendForwardDone("later", {43});
EXPECT_EQ(scheduler_->AwaitingDestinationSize(), 1u);
EXPECT_FALSE(PlanOnce().remote_decode.has_value());
EXPECT_FALSE(decode.NextExecutionPlan().remote_prefill.has_value());
decode.Advance(ExecutionEvent{}.With(pd::RemotePrefillDoneEvent{"first", 42}));
decode.Advance(ExecutionEvent{}.With(forward::Finish{.request_id = "first"}));
const auto later = decode.NextExecutionPlan();
ASSERT_TRUE(later.remote_prefill.has_value());
EXPECT_EQ(later.remote_prefill->request_ids, std::vector<std::string>{"later"});
Grant("later");
ASSERT_TRUE(PlanOnce().remote_decode.has_value());
Ack("later");
EXPECT_EQ(scheduler_->ActiveLcmBlocks(), 0);
}
TEST_F(IndependentAdmissionTest, ImmediateCancellationOnlyAppliesWhileAwaitingDestination) {
SubmitReady("r", 4);
EXPECT_FALSE(scheduler_->CanAbortAwaitingDestination("r"));
PlanOnce();
EXPECT_FALSE(scheduler_->CanAbortAwaitingDestination("r"));
SendForwardDone("r", {42});
EXPECT_TRUE(scheduler_->CanAbortAwaitingDestination("r"));
Grant("r");
EXPECT_FALSE(scheduler_->CanAbortAwaitingDestination("r"));
PlanOnce();
EXPECT_FALSE(scheduler_->CanAbortAwaitingDestination("r"));
Ack("r");
EXPECT_FALSE(scheduler_->CanAbortAwaitingDestination("r"));
}
TEST_F(IndependentAdmissionTest, AllocationPolicyIsExplicitOnlyForPrefill) {
auto prefill = MakeConfig();
prefill.decode_allocation_policy.reset();
EXPECT_THROW(Scheduler{prefill}, std::invalid_argument);
for (const auto role : {Role::kD, Role::kFused}) {
auto cfg = MakeConfig();
cfg.role = role;
SetTestSnapshotPool(cfg);
EXPECT_THROW(Scheduler{cfg}, std::invalid_argument);
cfg.decode_allocation_policy.reset();
EXPECT_NO_THROW(Scheduler{cfg});
}
}
} // namespace tokenspeed::test
@@ -656,6 +656,7 @@ TEST(SchedulerConfigValidateTest, ReplayNeedsBudgetForAWindowOnEveryRole) {
for (const Role role : {Role::kP, Role::kD}) {
SchedulerConfig pd = config;
pd.role = role;
pd.decode_allocation_policy = role == Role::kP ? std::optional{DecodeAllocationPolicy::kEarly} : std::nullopt;
for (CacheGroupConfig& group : pd.cache_groups) {
group.transfer_policy = CacheTransferPolicy::FullSuffix;
}
@@ -681,6 +682,7 @@ TEST(SchedulerConfigValidateTest, StateCheckpointInForwardRequiresPrefixCaching)
for (const Role role : {Role::kP, Role::kD}) {
SchedulerConfig pd = MakeValidConfig();
pd.role = role;
pd.decode_allocation_policy = role == Role::kP ? std::optional{DecodeAllocationPolicy::kEarly} : std::nullopt;
for (CacheGroupConfig& group : pd.cache_groups) {
group.transfer_policy = CacheTransferPolicy::FullSuffix;
}
@@ -717,6 +719,7 @@ TEST(SchedulerConfigValidateTest, SnapshotPoolAndRetractedSlotsAreStatedTogether
config.max_retracted_requests = 4;
config.role = Role::kP;
config.decode_allocation_policy = DecodeAllocationPolicy::kEarly;
config.cache_groups[0].transfer_policy = CacheTransferPolicy::FullSuffix;
EXPECT_THROW(config.Validate(), std::invalid_argument) << "the P role never retracts";
config.snapshot_allocator.total_pages = 1;
@@ -472,6 +472,7 @@ protected:
SchedulerConfig MakeConfig() override {
SchedulerConfig cfg = SchedulerKvCacheEventTestSuite::MakeConfig();
cfg.role = Role::kP;
cfg.decode_allocation_policy = DecodeAllocationPolicy::kEarly;
cfg.max_scheduled_tokens = 9;
cfg.device_allocator.total_pages = 8;
cfg.cache_groups.front().total_pages = cfg.device_allocator.total_pages;
@@ -421,6 +421,7 @@ protected:
SchedulerConfig MakeConfig() override {
SchedulerConfig cfg = StateCheckpointInForwardSuite::MakeConfig();
cfg.role = Role::kP;
cfg.decode_allocation_policy = DecodeAllocationPolicy::kEarly;
for (CacheGroupConfig& group : cfg.cache_groups) {
group.transfer_policy = group.attn_kind == AttnKind::kState ? CacheTransferPolicy::LatestSnapshot
: CacheTransferPolicy::FullSuffix;