mirror of
https://github.com/lightseekorg/tokenspeed.git
synced 2026-11-21 01:53:23 +08:00
perf(startup): freeze the collector once around every graph capture (#3555)
## Summary Prefill and drafter graph captures (`BreakableCapture`) collect garbage on entry, because a collection inside a capture would run graph destructors on a capturing stream; decode captures (`torch.cuda.graph`) collect only under `force_cudagraph_gc`. Only the decode ladder ran with the heap frozen (`freeze_gc`). So the prefill buckets and the drafter's prefill and extend graphs each ran a full collection over the startup heap: 150-200 ms apiece, about 150 per start. - `ModelExecutor.capture_graphs` now holds one frozen heap around every capture: decode, prefill and drafter graphs. - A boot that captures nothing (enforce-eager) skips the freeze and its two collections, as on main. - `--enable-cudagraph-gc` still keeps the heap unfrozen. - The decode runner no longer freezes on its own. Nested scopes would break, because an inner `gc.unfreeze()` thaws the outer scope. ## Results Qwen3.8-Flash-Next-NVFP4, TP1, MTP, 56 prefill buckets plus the drafter graphs. One GPU, warm compile caches, fresh server per run: | | main | this PR | |---|---|---| | full collections in the scheduler process | 148, 24.0 s | 148, 3.4 s | | server ready | 204.6, 207.6, 209.6 s (mean 207.3) | 187.5, 190.5 s (mean 189.0) | Both arms capture the same graphs with the same memory: | | both arms | |---|---| | KV cache | 182.05 GB | | activations + graphs | 3.98 GB | | device free | 5.63 GB | One consequence: an object that existed before capture and becomes cyclic garbage during a capture is now freed after the last capture instead of between capture groups. On this model's boot with the memory probe enabled, allocated memory is byte-identical at every one of the 5714 captures and the probe series are identical. Serving is unchanged: - First 64k-token prefill: 0.88 s; later ones: 0.655 s. - The same kernels compile on the first requests. - No collection runs while serving. ## Test Plan - `test_dispatch_autotune.py` runs `ModelExecutor.capture_graphs` with its capture doubles: - `test_capture_freezes_the_heap_once_around_every_capture`: the decode, prefill and drafter captures see a frozen heap unless `--enable-cudagraph-gc`; one `gc.freeze` spans them all; a boot that captures nothing collects nothing. - `test_a_failed_capture_leaves_the_heap_unfrozen`: a capture failing at any position leaves the heap as it found it. - Each of these mutations makes them fail: no `try`/`finally` in `freeze_gc`, one freeze per capture group, a decode-only freeze, no early return, `or` in its predicate, the flag ignored, no freeze. - `pre-commit run --all-files` passes. - The 11 test files that use `capture_graphs`, the decode runner's capture or `freeze_gc` fail the same tests on this branch and on main (28 FlashInfer BF16 adapter tests in `test_dispatch_autotune.py`, and `test_draft_final_step_follows_the_complete_drafter_run`, which fails on main since #3534). --------- Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
This commit is contained in:
@@ -439,6 +439,17 @@ alive. An EPD encode rank does the same and counts its vision-tower batches,
|
||||
which every rank of its attention-TP group forms alike. The young
|
||||
generations still collect automatically.
|
||||
|
||||
Graph capture runs before that mark and collects for its own reason: a
|
||||
collection inside a capture would run CUDA graph destructors on a capturing
|
||||
stream, so each prefill and drafter capture (`BreakableCapture`) collects on
|
||||
entry and keeps automatic collection off until it exits. Decode captures
|
||||
(`torch.cuda.graph`) collect on entry only under PyTorch's
|
||||
`force_cudagraph_gc`. When the decode or the prefill graph captures,
|
||||
`ModelExecutor.capture_graphs` freezes the heap once around all captures, so
|
||||
those collections, and any automatic one during a decode capture, walk only
|
||||
what the captures created; an enforce-eager boot runs none of this.
|
||||
`--enable-cudagraph-gc` keeps the heap unfrozen.
|
||||
|
||||
### Profiling the issuing thread
|
||||
|
||||
Profiling must include the forward thread that issues device work. Batch and
|
||||
|
||||
@@ -374,7 +374,6 @@ class ForwardStepRunner:
|
||||
self.eager_grammar_buffers = eager_grammar_buffers
|
||||
self.runtime_states = runtime_states
|
||||
self.disable_padding = config.disable_cuda_graph_padding
|
||||
self.enable_cudagraph_gc = config.enable_cudagraph_gc
|
||||
self.device = config.device
|
||||
self.device_module = torch.get_device_module(self.device)
|
||||
# Decode graphs capture into the memory saver's ``cuda_graph`` region
|
||||
@@ -591,36 +590,35 @@ class ForwardStepRunner:
|
||||
observer: Measured around each capture, for the same caller.
|
||||
"""
|
||||
rank = self.global_rank
|
||||
with freeze_gc(self.enable_cudagraph_gc):
|
||||
# Capture backend-declared sampler variants explicitly.
|
||||
ladders = self.capture_ladders(entries)
|
||||
capture_items = [
|
||||
(variant, ladders[f"decode:{variant}"].widths[i])
|
||||
for variant in self.capture_plan
|
||||
for i in ladders[f"decode:{variant}"].sampled
|
||||
]
|
||||
capture_range = tqdm.tqdm(capture_items) if rank == 0 else capture_items
|
||||
# Capture backend-declared sampler variants explicitly.
|
||||
ladders = self.capture_ladders(entries)
|
||||
capture_items = [
|
||||
(variant, ladders[f"decode:{variant}"].widths[i])
|
||||
for variant in self.capture_plan
|
||||
for i in ladders[f"decode:{variant}"].sampled
|
||||
]
|
||||
capture_range = tqdm.tqdm(capture_items) if rank == 0 else capture_items
|
||||
if rank == 0:
|
||||
batch_sizes = list(dict.fromkeys(bs for _variant, bs in capture_items))
|
||||
logger.info(f"Capturing batches: {batch_sizes!s}")
|
||||
for variant, bs in capture_range:
|
||||
if rank == 0:
|
||||
batch_sizes = list(dict.fromkeys(bs for _variant, bs in capture_items))
|
||||
logger.info(f"Capturing batches: {batch_sizes!s}")
|
||||
for variant, bs in capture_range:
|
||||
if rank == 0:
|
||||
avail_mem = get_available_gpu_memory(
|
||||
self.device, self.gpu_id, empty_cache=False
|
||||
)
|
||||
variant_desc = (
|
||||
""
|
||||
if variant == CUDA_GRAPH_VARIANT_DEFAULT
|
||||
else f" variant={variant}"
|
||||
)
|
||||
capture_range.set_description(
|
||||
f"Capturing batches ({bs=}{variant_desc} {avail_mem=:.2f} GB)"
|
||||
)
|
||||
graph, output_buffers = self._capture_one(
|
||||
bs, variant=variant, observer=observer.measure(f"decode:{variant}")
|
||||
avail_mem = get_available_gpu_memory(
|
||||
self.device, self.gpu_id, empty_cache=False
|
||||
)
|
||||
self.graphs[(variant, bs)] = graph
|
||||
self.output_buffers[(variant, bs)] = output_buffers
|
||||
variant_desc = (
|
||||
""
|
||||
if variant == CUDA_GRAPH_VARIANT_DEFAULT
|
||||
else f" variant={variant}"
|
||||
)
|
||||
capture_range.set_description(
|
||||
f"Capturing batches ({bs=}{variant_desc} {avail_mem=:.2f} GB)"
|
||||
)
|
||||
graph, output_buffers = self._capture_one(
|
||||
bs, variant=variant, observer=observer.measure(f"decode:{variant}")
|
||||
)
|
||||
self.graphs[(variant, bs)] = graph
|
||||
self.output_buffers[(variant, bs)] = output_buffers
|
||||
|
||||
@property
|
||||
def capture_plan(self) -> dict[str, list[int]]:
|
||||
|
||||
@@ -77,7 +77,7 @@ from tokenspeed.runtime.execution.forward_batch_info import (
|
||||
CaptureHiddenMode,
|
||||
ForwardMode,
|
||||
)
|
||||
from tokenspeed.runtime.execution.forward_step import ForwardStepRunner
|
||||
from tokenspeed.runtime.execution.forward_step import ForwardStepRunner, freeze_gc
|
||||
from tokenspeed.runtime.execution.input_buffer import InputBuffers
|
||||
from tokenspeed.runtime.execution.memory_delta import MemoryDeltaObserver
|
||||
from tokenspeed.runtime.execution.model_runner import ModelRunner
|
||||
@@ -1101,34 +1101,39 @@ class ModelExecutor:
|
||||
captures every graph.
|
||||
"""
|
||||
workspace_pool(self.device).freeze()
|
||||
# Enforce-eager boots capture nothing and skip freeze_gc's collections.
|
||||
if self.forward_step.disable and self.prefill_graph.disable:
|
||||
return
|
||||
|
||||
if not self.forward_step.disable:
|
||||
self.forward_step.capture(entries=entries, observer=observer)
|
||||
if not self.prefill_graph.disable:
|
||||
self.prefill_graph.capture(
|
||||
self.forward_step, entries=entries, observer=observer
|
||||
)
|
||||
# Only a drafter that declared the ladder gets a window to fill.
|
||||
if self.captures_drafter_prefill_graph:
|
||||
self.drafter.capture_prefill_graph(
|
||||
self.forward_step.stream, observer.measure("prefill:drafter")
|
||||
)
|
||||
# The drafter's extend graphs: the entries the projection prices.
|
||||
plan = self._drafter_extend_plan()
|
||||
if plan:
|
||||
width, dtype = self.prefill_graph.captured_hidden_spec()
|
||||
self.drafter.capture_extend_graphs(
|
||||
DrafterExtendCapture(
|
||||
plan=plan,
|
||||
ladders=self.drafter_extend_ladders(entries),
|
||||
stream=self.forward_step.stream,
|
||||
num_warmup=self.prefill_graph.num_warmup,
|
||||
target_hidden_width=width,
|
||||
target_hidden_dtype=dtype,
|
||||
prepare_batch=self._prepare_drafter_capture_batch,
|
||||
),
|
||||
observer=observer,
|
||||
# Freeze once for all captures; prefill and drafter captures collect on entry.
|
||||
with freeze_gc(self.config.enable_cudagraph_gc):
|
||||
if not self.forward_step.disable:
|
||||
self.forward_step.capture(entries=entries, observer=observer)
|
||||
if not self.prefill_graph.disable:
|
||||
self.prefill_graph.capture(
|
||||
self.forward_step, entries=entries, observer=observer
|
||||
)
|
||||
# Only a drafter that declared the ladder gets a window to fill.
|
||||
if self.captures_drafter_prefill_graph:
|
||||
self.drafter.capture_prefill_graph(
|
||||
self.forward_step.stream, observer.measure("prefill:drafter")
|
||||
)
|
||||
# The drafter's extend graphs: the entries the projection prices.
|
||||
plan = self._drafter_extend_plan()
|
||||
if plan:
|
||||
width, dtype = self.prefill_graph.captured_hidden_spec()
|
||||
self.drafter.capture_extend_graphs(
|
||||
DrafterExtendCapture(
|
||||
plan=plan,
|
||||
ladders=self.drafter_extend_ladders(entries),
|
||||
stream=self.forward_step.stream,
|
||||
num_warmup=self.prefill_graph.num_warmup,
|
||||
target_hidden_width=width,
|
||||
target_hidden_dtype=dtype,
|
||||
prepare_batch=self._prepare_drafter_capture_batch,
|
||||
),
|
||||
observer=observer,
|
||||
)
|
||||
|
||||
@property
|
||||
def captures_drafter_prefill_graph(self) -> bool:
|
||||
|
||||
@@ -506,6 +506,7 @@ def test_the_drafters_own_pool_is_declared_captured_and_labelled() -> None:
|
||||
)
|
||||
executor = ModelExecutor.__new__(ModelExecutor)
|
||||
executor.device = "cuda"
|
||||
executor.config = SimpleNamespace(enable_cudagraph_gc=False)
|
||||
executor.drafter = drafter
|
||||
executor.forward_step = SimpleNamespace(
|
||||
disable=True, capture_ladders=lambda entries: {}, stream="stream"
|
||||
@@ -555,6 +556,7 @@ def test_the_drafter_extend_graphs_are_declared_captured_and_labelled() -> None:
|
||||
buckets = [16, 32, 48, 64, 80, 96, 112]
|
||||
executor = ModelExecutor.__new__(ModelExecutor)
|
||||
executor.device = "cuda"
|
||||
executor.config = SimpleNamespace(enable_cudagraph_gc=False)
|
||||
executor.drafter = Drafter()
|
||||
executor.draft_attn_backend = "draft backend"
|
||||
executor.forward_step = SimpleNamespace(
|
||||
|
||||
@@ -80,7 +80,6 @@ def _decode_runner(capture_bs, variants=None):
|
||||
runner.dp_local_graph = False
|
||||
runner.capture_bs = list(capture_bs)
|
||||
runner.global_rank = 1
|
||||
runner.enable_cudagraph_gc = False
|
||||
runner.disable = False
|
||||
runner.max_tokens_per_req = 1
|
||||
runner.drafter = None
|
||||
|
||||
@@ -30,9 +30,11 @@ order of the startup steps -- not the shape of the production source.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gc
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from collections.abc import Callable, Iterator
|
||||
from contextlib import nullcontext
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal, get_args
|
||||
@@ -765,6 +767,13 @@ class _ObserverDouble:
|
||||
return nullcontext()
|
||||
|
||||
|
||||
@dataclass
|
||||
class _CaptureConfig:
|
||||
"""The ModelExecutorConfig field capture_graphs() reads."""
|
||||
|
||||
enable_cudagraph_gc: bool
|
||||
|
||||
|
||||
class _CaptureExecutor:
|
||||
"""The ModelExecutor state capture_graphs() reads, without the GPU boot."""
|
||||
|
||||
@@ -774,7 +783,8 @@ class _CaptureExecutor:
|
||||
drafter_extend_ladders = ModelExecutor.drafter_extend_ladders
|
||||
_prepare_drafter_capture_batch = ModelExecutor._prepare_drafter_capture_batch
|
||||
|
||||
def __init__(self, *, forward_step, prefill_graph, drafter):
|
||||
def __init__(self, *, forward_step, prefill_graph, drafter, enable_cudagraph_gc):
|
||||
self.config = _CaptureConfig(enable_cudagraph_gc=enable_cudagraph_gc)
|
||||
self.device = "cpu"
|
||||
self.forward_step = forward_step
|
||||
self.prefill_graph = prefill_graph
|
||||
@@ -823,6 +833,7 @@ def test_capture_lifecycle_preserves_main_order(
|
||||
extend_graphs=drafter_graphs == "extend", on_capture=capture_draft
|
||||
)
|
||||
),
|
||||
enable_cudagraph_gc=False,
|
||||
)
|
||||
ModelExecutor.capture_graphs(
|
||||
executor,
|
||||
@@ -838,6 +849,133 @@ def test_capture_lifecycle_preserves_main_order(
|
||||
reset_workspace_pools()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def unfrozen_heap() -> Iterator[int]:
|
||||
"""Yield the freeze count of a heap nothing froze, with fresh workspace pools."""
|
||||
gc.unfreeze()
|
||||
# A collection parks CPython's immortal objects in the permanent generation.
|
||||
gc.collect()
|
||||
reset_workspace_pools()
|
||||
yield gc.get_freeze_count()
|
||||
reset_workspace_pools()
|
||||
|
||||
|
||||
def _record_gc_calls(monkeypatch: pytest.MonkeyPatch) -> list[str]:
|
||||
calls: list[str] = []
|
||||
for name, real in (
|
||||
("collect", gc.collect),
|
||||
("freeze", gc.freeze),
|
||||
("unfreeze", gc.unfreeze),
|
||||
):
|
||||
|
||||
def record(
|
||||
*args: int, name: str = name, real: Callable[..., object] = real
|
||||
) -> object:
|
||||
calls.append(name)
|
||||
return real(*args)
|
||||
|
||||
monkeypatch.setattr(gc, name, record)
|
||||
return calls
|
||||
|
||||
|
||||
def _capture_executor(
|
||||
capture: Callable[[str], Callable[..., None]],
|
||||
*,
|
||||
decode_disabled: bool,
|
||||
prefill_disabled: bool,
|
||||
drafter_graphs: Literal["own", "extend"],
|
||||
enable_cudagraph_gc: bool,
|
||||
) -> _CaptureExecutor:
|
||||
return _CaptureExecutor(
|
||||
forward_step=_DecodeCaptureDouble(
|
||||
disable=decode_disabled, stream=object(), on_capture=capture("decode")
|
||||
),
|
||||
prefill_graph=_PrefillCaptureDouble(
|
||||
disable=prefill_disabled, on_capture=capture("prefill")
|
||||
),
|
||||
drafter=_DrafterCaptureDouble(
|
||||
extend_graphs=drafter_graphs == "extend", on_capture=capture("draft")
|
||||
),
|
||||
enable_cudagraph_gc=enable_cudagraph_gc,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("enable_cudagraph_gc", [False, True])
|
||||
@pytest.mark.parametrize("decode_disabled", [False, True])
|
||||
@pytest.mark.parametrize("prefill_disabled", [False, True])
|
||||
@pytest.mark.parametrize("drafter_graphs", ["own", "extend"])
|
||||
def test_capture_freezes_the_heap_once_around_every_capture(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
unfrozen_heap: int,
|
||||
enable_cudagraph_gc: bool,
|
||||
decode_disabled: bool,
|
||||
prefill_disabled: bool,
|
||||
drafter_graphs: Literal["own", "extend"],
|
||||
) -> None:
|
||||
"""One freeze spans the decode, prefill and drafter captures unless
|
||||
--enable-cudagraph-gc, and a boot that captures nothing collects nothing."""
|
||||
frozen: dict[str, bool] = {}
|
||||
|
||||
def capture(name: str) -> Callable[..., None]:
|
||||
def run(*_args: object) -> None:
|
||||
frozen[name] = gc.get_freeze_count() > unfrozen_heap
|
||||
|
||||
return run
|
||||
|
||||
executor = _capture_executor(
|
||||
capture,
|
||||
decode_disabled=decode_disabled,
|
||||
prefill_disabled=prefill_disabled,
|
||||
drafter_graphs=drafter_graphs,
|
||||
enable_cudagraph_gc=enable_cudagraph_gc,
|
||||
)
|
||||
calls = _record_gc_calls(monkeypatch)
|
||||
ModelExecutor.capture_graphs(executor, entries=None, observer=_ObserverDouble())
|
||||
|
||||
captured = [
|
||||
*([] if decode_disabled else ["decode"]),
|
||||
*([] if prefill_disabled else ["prefill", "draft"]),
|
||||
]
|
||||
assert frozen == {name: not enable_cudagraph_gc for name in captured}
|
||||
if not captured:
|
||||
assert calls == []
|
||||
elif enable_cudagraph_gc:
|
||||
assert calls == ["collect"]
|
||||
else:
|
||||
assert calls == ["collect", "freeze", "unfreeze", "collect"]
|
||||
assert gc.get_freeze_count() == unfrozen_heap
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("failing", "drafter_graphs"),
|
||||
[("decode", "own"), ("prefill", "own"), ("draft", "own"), ("draft", "extend")],
|
||||
)
|
||||
def test_a_failed_capture_leaves_the_heap_unfrozen(
|
||||
unfrozen_heap: int, failing: str, drafter_graphs: Literal["own", "extend"]
|
||||
) -> None:
|
||||
frozen: list[bool] = []
|
||||
|
||||
def capture(name: str) -> Callable[..., None]:
|
||||
def run(*_args: object) -> None:
|
||||
frozen.append(gc.get_freeze_count() > unfrozen_heap)
|
||||
if name == failing:
|
||||
raise RuntimeError(f"{name} capture failed")
|
||||
|
||||
return run
|
||||
|
||||
executor = _capture_executor(
|
||||
capture,
|
||||
decode_disabled=False,
|
||||
prefill_disabled=False,
|
||||
drafter_graphs=drafter_graphs,
|
||||
enable_cudagraph_gc=False,
|
||||
)
|
||||
with pytest.raises(RuntimeError, match=f"{failing} capture failed"):
|
||||
ModelExecutor.capture_graphs(executor, entries=None, observer=_ObserverDouble())
|
||||
assert frozen and all(frozen)
|
||||
assert gc.get_freeze_count() == unfrozen_heap
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _autotune_cache_key
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -65,6 +65,7 @@ def test_draft_prefill_capture_uses_the_resolved_prefill_gate(
|
||||
# A real instance, so the gate resolves the production property.
|
||||
executor = model_executor.ModelExecutor.__new__(model_executor.ModelExecutor)
|
||||
executor.device = "cuda"
|
||||
executor.config = SimpleNamespace(enable_cudagraph_gc=False)
|
||||
executor.forward_step = Mock(disable=decode_disabled)
|
||||
executor.prefill_graph = Mock(disable=prefill_disabled)
|
||||
# Explicit: a bare Mock answers truthily, so the gate below would hold anyway.
|
||||
|
||||
@@ -207,6 +207,7 @@ def test_capture_hands_both_owners_the_entries_and_the_drafter_its_window(
|
||||
seen = []
|
||||
executor = ModelExecutor.__new__(ModelExecutor)
|
||||
executor.device = "cuda"
|
||||
executor.config = SimpleNamespace(enable_cudagraph_gc=False)
|
||||
executor.forward_step = SimpleNamespace(
|
||||
disable=False,
|
||||
stream="stream",
|
||||
|
||||
@@ -227,47 +227,6 @@ def test_draft_final_step_follows_the_complete_drafter_run():
|
||||
]
|
||||
|
||||
|
||||
def test_cudagraph_gc_flag_reaches_the_capture_context():
|
||||
"""The operator flag must survive ServerArgs -> config -> capture.
|
||||
|
||||
Freezing the collector for the duration of capture is the default; the
|
||||
flag is the escape hatch. It previously never arrived -- the wrapper read
|
||||
it off a config that never carried it -- so capture never froze and the
|
||||
flag moved nothing in either direction. Pin the whole path, not the
|
||||
default: read the value the capture context would actually see.
|
||||
"""
|
||||
import dataclasses
|
||||
import gc
|
||||
|
||||
from tokenspeed.runtime.execution.forward_step import (
|
||||
ForwardStepRunner,
|
||||
freeze_gc,
|
||||
)
|
||||
from tokenspeed.runtime.execution.model_executor import ModelExecutorConfig
|
||||
|
||||
fields = {f.name for f in dataclasses.fields(ModelExecutorConfig)}
|
||||
assert "enable_cudagraph_gc" in fields, "config must carry the flag"
|
||||
|
||||
for flag in (False, True):
|
||||
config = SimpleNamespace(enable_cudagraph_gc=flag)
|
||||
wrapper = ForwardStepRunner.__new__(ForwardStepRunner)
|
||||
wrapper.dp_local_graph = False
|
||||
# Only the flag plumbing is under test; __init__ needs a live model.
|
||||
wrapper.enable_cudagraph_gc = config.enable_cudagraph_gc
|
||||
assert wrapper.enable_cudagraph_gc is flag
|
||||
|
||||
# get_freeze_count() is process-global and other code freezes too, so
|
||||
# compare against the count on entry rather than against zero.
|
||||
before = gc.get_freeze_count()
|
||||
canary = [object() for _ in range(64)]
|
||||
with freeze_gc(wrapper.enable_cudagraph_gc):
|
||||
during = gc.get_freeze_count()
|
||||
after = gc.get_freeze_count()
|
||||
assert (during > before) is (not flag), (flag, before, during)
|
||||
assert after <= before, (before, after)
|
||||
assert len(canary) == 64
|
||||
|
||||
|
||||
def test_non_spec_decode_routes_through_verify():
|
||||
"""The unified sampling rule: decode rows verify even without a drafter.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user