mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(kompress): bound the inference deadline to the request, not each block (#3693)
## Description `HEADROOM_COMPRESSION_DEADLINE_MS` (default 20s) is a budget on kompress inference, checked at chunk boundaries against a clock that `compress()` starts itself when the caller passes no `_deadline_started_at`. Nothing outside `kompress_compressor.py` ever passes one, so the budget bounds a **block**, not a request: a request with a dozen compressible blocks gets a dozen full deadlines. That is how one request runs past the pipeline's own `COMPRESSION_TIMEOUT_SECONDS` (default 30s). The overrunning worker cannot be preempted, so it becomes timeout debt and opens the quarantine, and every request behind it forwards with NO compression until that worker exits or the 60s cap lapses. Same cascade as #2360, reached on the Anthropic path by ordinary large agentic requests rather than by a Codex WS 5s deadline. `apply()` now stamps one `time.perf_counter()` origin into the existing per-request `_PerRequestRuntimeState` (#3486) and `_try_ml_compressor` hands it to every kompress call for that request, so a big request degrades to partial compression inside its budget instead of overrunning and starving the requests queued behind it. Related: #3653 (the thread/fd leak each quarantine event also causes) and #2451 (the concurrency-1 execution slot that makes blocks serialize). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `_PerRequestRuntimeState` carries `kompress_deadline_started_at`, stamped once per `apply()` alongside the other per-request fields. - `_try_ml_compressor` forwards that origin as `_deadline_started_at` on every kompress call. - `KompressCompressor.shares_request_deadline = True` gates the kwarg; `RemoteKompressCompressor.compress()` has no such parameter, so it is not handed one. - Three tests in `tests/test_transforms/test_kompress_deadline.py`. ## Testing - [x] Unit tests pass (`pytest`) ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_transforms/ tests/test_router_registry_smartcrusher.py -q 404 passed, 67 skipped in 32.37s $ uv run --frozen --extra dev pytest tests/test_transforms/test_kompress_deadline.py -q 6 passed in 2.00s # Against the unfixed source, the two behavioural tests fail: FAILED test_every_block_of_one_request_draws_down_the_same_deadline FAILED test_a_second_request_gets_a_fresh_deadline 2 failed, 4 passed ``` ## Real Behavior Proof - Environment: macOS 24.6 arm64, Python 3.12, headroom-ai 0.37.0 installed wheel, ONNX kompress (`chopratejas/kompress-v2-base`), backend anthropic, one heavy Claude Code user over 2026-09-17..21. - Exact command / steps: Measured the installed runtime directly. Timed `KompressCompressor.compress()` on a real 20 KB block pulled from `/transformations/feed` (2622 words, 8 chunks), four calls. Parsed 4100 `PERF` lines and every `CompressionQuarantinedError` request id out of `~/.headroom/logs/proxy.log*`, bucketed by `tok_before`, and priced the quarantined requests at the savings rate achieved by non-quarantined requests in the same size bucket. - Observed result: Inference costs ~1.6 ms/word: the 20 KB block takes 4.2s per call, reproducible (4548/4162/4392/4237 ms). `opt_ms` by size: 0-50k tokens n=376 avg 113 ms; 200-250k tokens n=101 avg 5192 ms, max 67247 ms. One day carried 266 "Kompress slow compress" markers totalling 682s of inference, max single inference 21.9s. 250 requests were refused compression by the quarantine, costing ~3.56M tokens, 8.7% of that day's achievable savings (per-day: 2.2%, 19.4%, 3.7%, 8.7%). With the deadline shared, the same request stops kompressing at the budget and returns instead of overrunning the 30s pipeline timeout. - Not tested: The fleet-wide prevalence of the quarantine cascade, which no counter reports today; and the CUDA/MPS batched path, which shares one deadline inside `compress_batch` already and was not re-measured on this change. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes. A request whose kompress inference exceeds `HEADROOM_COMPRESSION_DEADLINE_MS` in total now stops compressing further blocks, where before each block restarted the budget. That is a reduction in per-request compression for requests that were previously overrunning the pipeline timeout, and an increase for every request that was being quarantined behind them. - Kill switch / disable path: Set `HEADROOM_COMPRESSION_DEADLINE_MS` high (for example `600000`) to restore effectively unbounded per-request inference. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit; the deadline returns to being per-block. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review
This commit is contained in:
@@ -1866,6 +1866,13 @@ class _PerRequestRuntimeState:
|
||||
# a compression-policy misapplication.
|
||||
protect_read_tool_ids: set[str] = field(default_factory=set)
|
||||
protect_read_msg_indices: set[int] = field(default_factory=set)
|
||||
# ``time.perf_counter()`` origin shared by every kompress call this
|
||||
# request makes. ``HEADROOM_COMPRESSION_DEADLINE_MS`` is a deadline on
|
||||
# kompress inference, but each ``compress()`` call started its own clock,
|
||||
# so the budget bounded a BLOCK and a request with a dozen blocks could
|
||||
# spend a dozen budgets. Set once per ``apply()``; see
|
||||
# ``_try_ml_compressor``.
|
||||
kompress_deadline_started_at: float | None = None
|
||||
|
||||
|
||||
# Monotonic counter so each ContentRouter instance gets a uniquely named
|
||||
@@ -4232,6 +4239,18 @@ class ContentRouter(Transform):
|
||||
# don't accept the kwarg are unaffected on the common path.
|
||||
if protected:
|
||||
compress_kwargs["ccr_original"] = content
|
||||
# One deadline for the whole request, not one per
|
||||
# block. Without this a request with N compressible
|
||||
# blocks gets N full deadlines and can run far past
|
||||
# the pipeline's own compression timeout; the worker
|
||||
# that overruns cannot be preempted, so it opens the
|
||||
# timeout-debt quarantine and every request behind it
|
||||
# forwards with no compression at all.
|
||||
deadline_origin = self._runtime_state_var.get().kompress_deadline_started_at
|
||||
if deadline_origin is not None and getattr(
|
||||
compressor, "shares_request_deadline", False
|
||||
):
|
||||
compress_kwargs["_deadline_started_at"] = deadline_origin
|
||||
result = compressor.compress(text_to_compress, **compress_kwargs)
|
||||
compressed = result.compressed
|
||||
compressed_tokens = result.compressed_tokens
|
||||
@@ -5141,7 +5160,12 @@ class ContentRouter(Transform):
|
||||
# call does. No `reset()` is needed: every `apply()` call installs
|
||||
# its own fresh object up front, so the next call on a reused worker
|
||||
# thread simply overwrites the ambient value before reading it.
|
||||
self._runtime_state_var.set(_PerRequestRuntimeState())
|
||||
# The kompress deadline origin is stamped HERE, with the rest of the
|
||||
# per-request state, so every block this call compresses draws down one
|
||||
# shared budget instead of restarting it.
|
||||
self._runtime_state_var.set(
|
||||
_PerRequestRuntimeState(kompress_deadline_started_at=time.perf_counter())
|
||||
)
|
||||
|
||||
# Pre-process: Read lifecycle management (stale/superseded detection)
|
||||
if self.config.read_lifecycle.enabled:
|
||||
|
||||
@@ -1416,6 +1416,14 @@ class KompressCompressor(Transform):
|
||||
|
||||
name: str = "kompress_compressor"
|
||||
|
||||
# ``compress()`` accepts ``_deadline_started_at``, so a caller that
|
||||
# compresses many blocks for ONE request can hand every call the same
|
||||
# origin and have the deadline bound the request. Duck-typed rather than
|
||||
# isinstance-checked at the call site because ``RemoteKompressCompressor``
|
||||
# is the other compressor the router may get back and its ``compress()``
|
||||
# does not take the argument.
|
||||
shares_request_deadline: bool = True
|
||||
|
||||
def __init__(self, config: KompressConfig | None = None):
|
||||
self.config = config or KompressConfig()
|
||||
# Set by the preload canary when inference is too slow to be useful, or by
|
||||
|
||||
@@ -108,3 +108,113 @@ def test_compress_partial_run_keeps_processed_head_plus_verbatim_tail(
|
||||
# chunk 1 tripped the deadline -> its words kept verbatim (all present)
|
||||
for i in range(half, n_words):
|
||||
assert f"w{i}" in out
|
||||
|
||||
|
||||
def _two_block_messages(salt: str = "a") -> list[dict]:
|
||||
"""Two tool results, each big enough to reach the kompress stage.
|
||||
|
||||
``salt`` varies the text so a second request is not served from the
|
||||
router's content cache.
|
||||
"""
|
||||
return [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": f"toolu_{n}",
|
||||
"content": " ".join(
|
||||
f'{{"file":"src/mod_{i}.py","line":{i},"text":"payload {salt}{n}"}}'
|
||||
for i in range(160)
|
||||
),
|
||||
}
|
||||
for n in range(2)
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def _tokenizer():
|
||||
from headroom.providers import OpenAIProvider
|
||||
from headroom.tokenizer import Tokenizer
|
||||
|
||||
provider = OpenAIProvider()
|
||||
return Tokenizer(provider.get_token_counter("gpt-4o"), "gpt-4o")
|
||||
|
||||
|
||||
class _RecordingKompress:
|
||||
"""Stands in for KompressCompressor, recording the deadline it is handed."""
|
||||
|
||||
shares_request_deadline = True
|
||||
|
||||
def __init__(self):
|
||||
self.deadlines: list[float | None] = []
|
||||
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def ensure_background_load(self) -> None: # pragma: no cover - never reached
|
||||
pass
|
||||
|
||||
def compress(self, content, **kwargs):
|
||||
from types import SimpleNamespace
|
||||
|
||||
self.deadlines.append(kwargs.get("_deadline_started_at"))
|
||||
compressed = " ".join(content.split()[:20])
|
||||
return SimpleNamespace(compressed=compressed, compressed_tokens=len(compressed.split()))
|
||||
|
||||
|
||||
def test_every_block_of_one_request_draws_down_the_same_deadline(monkeypatch):
|
||||
"""The budget must bound the REQUEST, not each block.
|
||||
|
||||
``compress()`` starts its own clock when no origin is passed, so a request
|
||||
with N compressible blocks used to get N full deadlines. That is how a
|
||||
single request runs past the pipeline's compression timeout; the worker
|
||||
that overruns cannot be preempted, so it opens the timeout-debt quarantine
|
||||
and every request queued behind it forwards with no compression at all.
|
||||
"""
|
||||
from headroom.transforms.content_router import ContentRouter
|
||||
|
||||
router = ContentRouter()
|
||||
fake = _RecordingKompress()
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: fake)
|
||||
|
||||
router.apply(_two_block_messages(), _tokenizer(), force_kompress=True)
|
||||
|
||||
assert len(fake.deadlines) >= 2, f"expected both blocks to reach kompress: {fake.deadlines}"
|
||||
assert all(d is not None for d in fake.deadlines), fake.deadlines
|
||||
assert len(set(fake.deadlines)) == 1, f"each block restarted the clock: {fake.deadlines}"
|
||||
|
||||
|
||||
def test_a_second_request_gets_a_fresh_deadline(monkeypatch):
|
||||
"""Sharing is per-request: the next request must not inherit a spent budget."""
|
||||
from headroom.transforms.content_router import ContentRouter
|
||||
|
||||
router = ContentRouter()
|
||||
fake = _RecordingKompress()
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: fake)
|
||||
tokenizer = _tokenizer()
|
||||
|
||||
router.apply(_two_block_messages("first"), tokenizer, force_kompress=True)
|
||||
first = set(fake.deadlines)
|
||||
router.apply(_two_block_messages("second"), tokenizer, force_kompress=True)
|
||||
|
||||
assert len(set(fake.deadlines) - first) == 1, (
|
||||
f"second request reused a deadline: {fake.deadlines}"
|
||||
)
|
||||
|
||||
|
||||
def test_a_compressor_that_cannot_take_the_deadline_is_not_handed_one(monkeypatch):
|
||||
"""``RemoteKompressCompressor.compress()`` has no ``_deadline_started_at``
|
||||
parameter, so passing it would raise TypeError and fail compression open."""
|
||||
from headroom.transforms.content_router import ContentRouter
|
||||
|
||||
router = ContentRouter()
|
||||
fake = _RecordingKompress()
|
||||
fake.shares_request_deadline = False
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: fake)
|
||||
|
||||
router.apply(_two_block_messages(), _tokenizer(), force_kompress=True)
|
||||
|
||||
assert fake.deadlines, "the stub should still have been called"
|
||||
assert all(d is None for d in fake.deadlines), fake.deadlines
|
||||
|
||||
Reference in New Issue
Block a user