perf(startup): freeze the collector once around every graph capture (#3555)

## Summary

Prefill and drafter graph captures (`BreakableCapture`) collect garbage
on entry, because a collection inside a capture would run graph
destructors on a capturing stream; decode captures (`torch.cuda.graph`)
collect only under `force_cudagraph_gc`. Only the decode ladder ran with
the heap frozen (`freeze_gc`). So the prefill buckets and the drafter's
prefill and extend graphs each ran a full collection over the startup
heap: 150-200 ms apiece, about 150 per start.

- `ModelExecutor.capture_graphs` now holds one frozen heap around every
capture: decode, prefill and drafter graphs.
- A boot that captures nothing (enforce-eager) skips the freeze and its
two collections, as on main.
- `--enable-cudagraph-gc` still keeps the heap unfrozen.
- The decode runner no longer freezes on its own. Nested scopes would
break, because an inner `gc.unfreeze()` thaws the outer scope.

## Results

Qwen3.8-Flash-Next-NVFP4, TP1, MTP, 56 prefill buckets plus the drafter
graphs. One GPU, warm compile caches, fresh server per run:

| | main | this PR |
|---|---|---|
| full collections in the scheduler process | 148, 24.0 s | 148, 3.4 s |
| server ready | 204.6, 207.6, 209.6 s (mean 207.3) | 187.5, 190.5 s
(mean 189.0) |

Both arms capture the same graphs with the same memory:

| | both arms |
|---|---|
| KV cache | 182.05 GB |
| activations + graphs | 3.98 GB |
| device free | 5.63 GB |

One consequence: an object that existed before capture and becomes
cyclic garbage during a capture is now freed after the last capture
instead of between capture groups. On this model's boot with the memory
probe enabled, allocated memory is byte-identical at every one of the
5714 captures and the probe series are identical.

Serving is unchanged:

- First 64k-token prefill: 0.88 s; later ones: 0.655 s.
- The same kernels compile on the first requests.
- No collection runs while serving.

## Test Plan

- `test_dispatch_autotune.py` runs `ModelExecutor.capture_graphs` with
its capture doubles:
- `test_capture_freezes_the_heap_once_around_every_capture`: the decode,
prefill and drafter captures see a frozen heap unless
`--enable-cudagraph-gc`; one `gc.freeze` spans them all; a boot that
captures nothing collects nothing.
- `test_a_failed_capture_leaves_the_heap_unfrozen`: a capture failing at
any position leaves the heap as it found it.
- Each of these mutations makes them fail: no `try`/`finally` in
`freeze_gc`, one freeze per capture group, a decode-only freeze, no
early return, `or` in its predicate, the flag ignored, no freeze.
- `pre-commit run --all-files` passes.
- The 11 test files that use `capture_graphs`, the decode runner's
capture or `freeze_gc` fail the same tests on this branch and on main
(28 FlashInfer BF16 adapter tests in `test_dispatch_autotune.py`, and
`test_draft_final_step_follows_the_complete_drafter_run`, which fails on
main since #3534).

---------

Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
This commit is contained in:
Nicolas Perrin
2026-11-18 11:08:38 -05:00
committed by GitHub
parent 4749638bd8
commit 8214efc977
9 changed files with 213 additions and 99 deletions
+11
View File
@@ -439,6 +439,17 @@ alive. An EPD encode rank does the same and counts its vision-tower batches,
which every rank of its attention-TP group forms alike. The young
generations still collect automatically.
Graph capture runs before that mark and collects for its own reason: a
collection inside a capture would run CUDA graph destructors on a capturing
stream, so each prefill and drafter capture (`BreakableCapture`) collects on
entry and keeps automatic collection off until it exits. Decode captures
(`torch.cuda.graph`) collect on entry only under PyTorch's
`force_cudagraph_gc`. When the decode or the prefill graph captures,
`ModelExecutor.capture_graphs` freezes the heap once around all captures, so
those collections, and any automatic one during a decode capture, walk only
what the captures created; an enforce-eager boot runs none of this.
`--enable-cudagraph-gc` keeps the heap unfrozen.
### Profiling the issuing thread
Profiling must include the forward thread that issues device work. Batch and
@@ -374,7 +374,6 @@ class ForwardStepRunner:
self.eager_grammar_buffers = eager_grammar_buffers
self.runtime_states = runtime_states
self.disable_padding = config.disable_cuda_graph_padding
self.enable_cudagraph_gc = config.enable_cudagraph_gc
self.device = config.device
self.device_module = torch.get_device_module(self.device)
# Decode graphs capture into the memory saver's ``cuda_graph`` region
@@ -591,36 +590,35 @@ class ForwardStepRunner:
observer: Measured around each capture, for the same caller.
"""
rank = self.global_rank
with freeze_gc(self.enable_cudagraph_gc):
# Capture backend-declared sampler variants explicitly.
ladders = self.capture_ladders(entries)
capture_items = [
(variant, ladders[f"decode:{variant}"].widths[i])
for variant in self.capture_plan
for i in ladders[f"decode:{variant}"].sampled
]
capture_range = tqdm.tqdm(capture_items) if rank == 0 else capture_items
# Capture backend-declared sampler variants explicitly.
ladders = self.capture_ladders(entries)
capture_items = [
(variant, ladders[f"decode:{variant}"].widths[i])
for variant in self.capture_plan
for i in ladders[f"decode:{variant}"].sampled
]
capture_range = tqdm.tqdm(capture_items) if rank == 0 else capture_items
if rank == 0:
batch_sizes = list(dict.fromkeys(bs for _variant, bs in capture_items))
logger.info(f"Capturing batches: {batch_sizes!s}")
for variant, bs in capture_range:
if rank == 0:
batch_sizes = list(dict.fromkeys(bs for _variant, bs in capture_items))
logger.info(f"Capturing batches: {batch_sizes!s}")
for variant, bs in capture_range:
if rank == 0:
avail_mem = get_available_gpu_memory(
self.device, self.gpu_id, empty_cache=False
)
variant_desc = (
""
if variant == CUDA_GRAPH_VARIANT_DEFAULT
else f" variant={variant}"
)
capture_range.set_description(
f"Capturing batches ({bs=}{variant_desc} {avail_mem=:.2f} GB)"
)
graph, output_buffers = self._capture_one(
bs, variant=variant, observer=observer.measure(f"decode:{variant}")
avail_mem = get_available_gpu_memory(
self.device, self.gpu_id, empty_cache=False
)
self.graphs[(variant, bs)] = graph
self.output_buffers[(variant, bs)] = output_buffers
variant_desc = (
""
if variant == CUDA_GRAPH_VARIANT_DEFAULT
else f" variant={variant}"
)
capture_range.set_description(
f"Capturing batches ({bs=}{variant_desc} {avail_mem=:.2f} GB)"
)
graph, output_buffers = self._capture_one(
bs, variant=variant, observer=observer.measure(f"decode:{variant}")
)
self.graphs[(variant, bs)] = graph
self.output_buffers[(variant, bs)] = output_buffers
@property
def capture_plan(self) -> dict[str, list[int]]:
@@ -77,7 +77,7 @@ from tokenspeed.runtime.execution.forward_batch_info import (
CaptureHiddenMode,
ForwardMode,
)
from tokenspeed.runtime.execution.forward_step import ForwardStepRunner
from tokenspeed.runtime.execution.forward_step import ForwardStepRunner, freeze_gc
from tokenspeed.runtime.execution.input_buffer import InputBuffers
from tokenspeed.runtime.execution.memory_delta import MemoryDeltaObserver
from tokenspeed.runtime.execution.model_runner import ModelRunner
@@ -1101,34 +1101,39 @@ class ModelExecutor:
captures every graph.
"""
workspace_pool(self.device).freeze()
# Enforce-eager boots capture nothing and skip freeze_gc's collections.
if self.forward_step.disable and self.prefill_graph.disable:
return
if not self.forward_step.disable:
self.forward_step.capture(entries=entries, observer=observer)
if not self.prefill_graph.disable:
self.prefill_graph.capture(
self.forward_step, entries=entries, observer=observer
)
# Only a drafter that declared the ladder gets a window to fill.
if self.captures_drafter_prefill_graph:
self.drafter.capture_prefill_graph(
self.forward_step.stream, observer.measure("prefill:drafter")
)
# The drafter's extend graphs: the entries the projection prices.
plan = self._drafter_extend_plan()
if plan:
width, dtype = self.prefill_graph.captured_hidden_spec()
self.drafter.capture_extend_graphs(
DrafterExtendCapture(
plan=plan,
ladders=self.drafter_extend_ladders(entries),
stream=self.forward_step.stream,
num_warmup=self.prefill_graph.num_warmup,
target_hidden_width=width,
target_hidden_dtype=dtype,
prepare_batch=self._prepare_drafter_capture_batch,
),
observer=observer,
# Freeze once for all captures; prefill and drafter captures collect on entry.
with freeze_gc(self.config.enable_cudagraph_gc):
if not self.forward_step.disable:
self.forward_step.capture(entries=entries, observer=observer)
if not self.prefill_graph.disable:
self.prefill_graph.capture(
self.forward_step, entries=entries, observer=observer
)
# Only a drafter that declared the ladder gets a window to fill.
if self.captures_drafter_prefill_graph:
self.drafter.capture_prefill_graph(
self.forward_step.stream, observer.measure("prefill:drafter")
)
# The drafter's extend graphs: the entries the projection prices.
plan = self._drafter_extend_plan()
if plan:
width, dtype = self.prefill_graph.captured_hidden_spec()
self.drafter.capture_extend_graphs(
DrafterExtendCapture(
plan=plan,
ladders=self.drafter_extend_ladders(entries),
stream=self.forward_step.stream,
num_warmup=self.prefill_graph.num_warmup,
target_hidden_width=width,
target_hidden_dtype=dtype,
prepare_batch=self._prepare_drafter_capture_batch,
),
observer=observer,
)
@property
def captures_drafter_prefill_graph(self) -> bool:
@@ -506,6 +506,7 @@ def test_the_drafters_own_pool_is_declared_captured_and_labelled() -> None:
)
executor = ModelExecutor.__new__(ModelExecutor)
executor.device = "cuda"
executor.config = SimpleNamespace(enable_cudagraph_gc=False)
executor.drafter = drafter
executor.forward_step = SimpleNamespace(
disable=True, capture_ladders=lambda entries: {}, stream="stream"
@@ -555,6 +556,7 @@ def test_the_drafter_extend_graphs_are_declared_captured_and_labelled() -> None:
buckets = [16, 32, 48, 64, 80, 96, 112]
executor = ModelExecutor.__new__(ModelExecutor)
executor.device = "cuda"
executor.config = SimpleNamespace(enable_cudagraph_gc=False)
executor.drafter = Drafter()
executor.draft_attn_backend = "draft backend"
executor.forward_step = SimpleNamespace(
@@ -80,7 +80,6 @@ def _decode_runner(capture_bs, variants=None):
runner.dp_local_graph = False
runner.capture_bs = list(capture_bs)
runner.global_rank = 1
runner.enable_cudagraph_gc = False
runner.disable = False
runner.max_tokens_per_req = 1
runner.drafter = None
+139 -1
View File
@@ -30,9 +30,11 @@ order of the startup steps -- not the shape of the production source.
from __future__ import annotations
import gc
import json
import os
import sys
from collections.abc import Callable, Iterator
from contextlib import nullcontext
from dataclasses import dataclass
from typing import Literal, get_args
@@ -765,6 +767,13 @@ class _ObserverDouble:
return nullcontext()
@dataclass
class _CaptureConfig:
"""The ModelExecutorConfig field capture_graphs() reads."""
enable_cudagraph_gc: bool
class _CaptureExecutor:
"""The ModelExecutor state capture_graphs() reads, without the GPU boot."""
@@ -774,7 +783,8 @@ class _CaptureExecutor:
drafter_extend_ladders = ModelExecutor.drafter_extend_ladders
_prepare_drafter_capture_batch = ModelExecutor._prepare_drafter_capture_batch
def __init__(self, *, forward_step, prefill_graph, drafter):
def __init__(self, *, forward_step, prefill_graph, drafter, enable_cudagraph_gc):
self.config = _CaptureConfig(enable_cudagraph_gc=enable_cudagraph_gc)
self.device = "cpu"
self.forward_step = forward_step
self.prefill_graph = prefill_graph
@@ -823,6 +833,7 @@ def test_capture_lifecycle_preserves_main_order(
extend_graphs=drafter_graphs == "extend", on_capture=capture_draft
)
),
enable_cudagraph_gc=False,
)
ModelExecutor.capture_graphs(
executor,
@@ -838,6 +849,133 @@ def test_capture_lifecycle_preserves_main_order(
reset_workspace_pools()
@pytest.fixture
def unfrozen_heap() -> Iterator[int]:
"""Yield the freeze count of a heap nothing froze, with fresh workspace pools."""
gc.unfreeze()
# A collection parks CPython's immortal objects in the permanent generation.
gc.collect()
reset_workspace_pools()
yield gc.get_freeze_count()
reset_workspace_pools()
def _record_gc_calls(monkeypatch: pytest.MonkeyPatch) -> list[str]:
calls: list[str] = []
for name, real in (
("collect", gc.collect),
("freeze", gc.freeze),
("unfreeze", gc.unfreeze),
):
def record(
*args: int, name: str = name, real: Callable[..., object] = real
) -> object:
calls.append(name)
return real(*args)
monkeypatch.setattr(gc, name, record)
return calls
def _capture_executor(
capture: Callable[[str], Callable[..., None]],
*,
decode_disabled: bool,
prefill_disabled: bool,
drafter_graphs: Literal["own", "extend"],
enable_cudagraph_gc: bool,
) -> _CaptureExecutor:
return _CaptureExecutor(
forward_step=_DecodeCaptureDouble(
disable=decode_disabled, stream=object(), on_capture=capture("decode")
),
prefill_graph=_PrefillCaptureDouble(
disable=prefill_disabled, on_capture=capture("prefill")
),
drafter=_DrafterCaptureDouble(
extend_graphs=drafter_graphs == "extend", on_capture=capture("draft")
),
enable_cudagraph_gc=enable_cudagraph_gc,
)
@pytest.mark.parametrize("enable_cudagraph_gc", [False, True])
@pytest.mark.parametrize("decode_disabled", [False, True])
@pytest.mark.parametrize("prefill_disabled", [False, True])
@pytest.mark.parametrize("drafter_graphs", ["own", "extend"])
def test_capture_freezes_the_heap_once_around_every_capture(
monkeypatch: pytest.MonkeyPatch,
unfrozen_heap: int,
enable_cudagraph_gc: bool,
decode_disabled: bool,
prefill_disabled: bool,
drafter_graphs: Literal["own", "extend"],
) -> None:
"""One freeze spans the decode, prefill and drafter captures unless
--enable-cudagraph-gc, and a boot that captures nothing collects nothing."""
frozen: dict[str, bool] = {}
def capture(name: str) -> Callable[..., None]:
def run(*_args: object) -> None:
frozen[name] = gc.get_freeze_count() > unfrozen_heap
return run
executor = _capture_executor(
capture,
decode_disabled=decode_disabled,
prefill_disabled=prefill_disabled,
drafter_graphs=drafter_graphs,
enable_cudagraph_gc=enable_cudagraph_gc,
)
calls = _record_gc_calls(monkeypatch)
ModelExecutor.capture_graphs(executor, entries=None, observer=_ObserverDouble())
captured = [
*([] if decode_disabled else ["decode"]),
*([] if prefill_disabled else ["prefill", "draft"]),
]
assert frozen == {name: not enable_cudagraph_gc for name in captured}
if not captured:
assert calls == []
elif enable_cudagraph_gc:
assert calls == ["collect"]
else:
assert calls == ["collect", "freeze", "unfreeze", "collect"]
assert gc.get_freeze_count() == unfrozen_heap
@pytest.mark.parametrize(
("failing", "drafter_graphs"),
[("decode", "own"), ("prefill", "own"), ("draft", "own"), ("draft", "extend")],
)
def test_a_failed_capture_leaves_the_heap_unfrozen(
unfrozen_heap: int, failing: str, drafter_graphs: Literal["own", "extend"]
) -> None:
frozen: list[bool] = []
def capture(name: str) -> Callable[..., None]:
def run(*_args: object) -> None:
frozen.append(gc.get_freeze_count() > unfrozen_heap)
if name == failing:
raise RuntimeError(f"{name} capture failed")
return run
executor = _capture_executor(
capture,
decode_disabled=False,
prefill_disabled=False,
drafter_graphs=drafter_graphs,
enable_cudagraph_gc=False,
)
with pytest.raises(RuntimeError, match=f"{failing} capture failed"):
ModelExecutor.capture_graphs(executor, entries=None, observer=_ObserverDouble())
assert frozen and all(frozen)
assert gc.get_freeze_count() == unfrozen_heap
# ---------------------------------------------------------------------------
# _autotune_cache_key
# ---------------------------------------------------------------------------
+1
View File
@@ -65,6 +65,7 @@ def test_draft_prefill_capture_uses_the_resolved_prefill_gate(
# A real instance, so the gate resolves the production property.
executor = model_executor.ModelExecutor.__new__(model_executor.ModelExecutor)
executor.device = "cuda"
executor.config = SimpleNamespace(enable_cudagraph_gc=False)
executor.forward_step = Mock(disable=decode_disabled)
executor.prefill_graph = Mock(disable=prefill_disabled)
# Explicit: a bare Mock answers truthily, so the gate below would hold anyway.
+1
View File
@@ -207,6 +207,7 @@ def test_capture_hands_both_owners_the_entries_and_the_drafter_its_window(
seen = []
executor = ModelExecutor.__new__(ModelExecutor)
executor.device = "cuda"
executor.config = SimpleNamespace(enable_cudagraph_gc=False)
executor.forward_step = SimpleNamespace(
disable=False,
stream="stream",
@@ -227,47 +227,6 @@ def test_draft_final_step_follows_the_complete_drafter_run():
]
def test_cudagraph_gc_flag_reaches_the_capture_context():
"""The operator flag must survive ServerArgs -> config -> capture.
Freezing the collector for the duration of capture is the default; the
flag is the escape hatch. It previously never arrived -- the wrapper read
it off a config that never carried it -- so capture never froze and the
flag moved nothing in either direction. Pin the whole path, not the
default: read the value the capture context would actually see.
"""
import dataclasses
import gc
from tokenspeed.runtime.execution.forward_step import (
ForwardStepRunner,
freeze_gc,
)
from tokenspeed.runtime.execution.model_executor import ModelExecutorConfig
fields = {f.name for f in dataclasses.fields(ModelExecutorConfig)}
assert "enable_cudagraph_gc" in fields, "config must carry the flag"
for flag in (False, True):
config = SimpleNamespace(enable_cudagraph_gc=flag)
wrapper = ForwardStepRunner.__new__(ForwardStepRunner)
wrapper.dp_local_graph = False
# Only the flag plumbing is under test; __init__ needs a live model.
wrapper.enable_cudagraph_gc = config.enable_cudagraph_gc
assert wrapper.enable_cudagraph_gc is flag
# get_freeze_count() is process-global and other code freezes too, so
# compare against the count on entry rather than against zero.
before = gc.get_freeze_count()
canary = [object() for _ in range(64)]
with freeze_gc(wrapper.enable_cudagraph_gc):
during = gc.get_freeze_count()
after = gc.get_freeze_count()
assert (during > before) is (not flag), (flag, before, during)
assert after <= before, (before, after)
assert len(canary) == 64
def test_non_spec_decode_routes_through_verify():
"""The unified sampling rule: decode rows verify even without a drafter.