fix(kompress): bound the inference deadline to the request, not each block (#3693)

## Description

`HEADROOM_COMPRESSION_DEADLINE_MS` (default 20s) is a budget on kompress
inference, checked at chunk boundaries against a clock that `compress()`
starts itself when the caller passes no `_deadline_started_at`. Nothing
outside `kompress_compressor.py` ever passes one, so the budget bounds a
**block**, not a request: a request with a dozen compressible blocks
gets a dozen full deadlines.

That is how one request runs past the pipeline's own
`COMPRESSION_TIMEOUT_SECONDS` (default 30s). The overrunning worker
cannot be preempted, so it becomes timeout debt and opens the
quarantine, and every request behind it forwards with NO compression
until that worker exits or the 60s cap lapses. Same cascade as #2360,
reached on the Anthropic path by ordinary large agentic requests rather
than by a Codex WS 5s deadline.

`apply()` now stamps one `time.perf_counter()` origin into the existing
per-request `_PerRequestRuntimeState` (#3486) and `_try_ml_compressor`
hands it to every kompress call for that request, so a big request
degrades to partial compression inside its budget instead of overrunning
and starving the requests queued behind it.

Related: #3653 (the thread/fd leak each quarantine event also causes)
and #2451 (the concurrency-1 execution slot that makes blocks
serialize).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `_PerRequestRuntimeState` carries `kompress_deadline_started_at`,
stamped once per `apply()` alongside the other per-request fields.
- `_try_ml_compressor` forwards that origin as `_deadline_started_at` on
every kompress call.
- `KompressCompressor.shares_request_deadline = True` gates the kwarg;
`RemoteKompressCompressor.compress()` has no such parameter, so it is
not handed one.
- Three tests in `tests/test_transforms/test_kompress_deadline.py`.

## Testing

- [x] Unit tests pass (`pytest`)

### Test Output


```text
$ uv run --frozen --extra dev pytest tests/test_transforms/ tests/test_router_registry_smartcrusher.py -q
404 passed, 67 skipped in 32.37s

$ uv run --frozen --extra dev pytest tests/test_transforms/test_kompress_deadline.py -q
6 passed in 2.00s

# Against the unfixed source, the two behavioural tests fail:
FAILED test_every_block_of_one_request_draws_down_the_same_deadline
FAILED test_a_second_request_gets_a_fresh_deadline
2 failed, 4 passed
```


## Real Behavior Proof

- Environment: macOS 24.6 arm64, Python 3.12, headroom-ai 0.37.0
installed wheel, ONNX kompress (`chopratejas/kompress-v2-base`), backend
anthropic, one heavy Claude Code user over 2026-09-17..21.
- Exact command / steps: Measured the installed runtime directly. Timed
`KompressCompressor.compress()` on a real 20 KB block pulled from
`/transformations/feed` (2622 words, 8 chunks), four calls. Parsed 4100
`PERF` lines and every `CompressionQuarantinedError` request id out of
`~/.headroom/logs/proxy.log*`, bucketed by `tok_before`, and priced the
quarantined requests at the savings rate achieved by non-quarantined
requests in the same size bucket.
- Observed result: Inference costs ~1.6 ms/word: the 20 KB block takes
4.2s per call, reproducible (4548/4162/4392/4237 ms). `opt_ms` by size:
0-50k tokens n=376 avg 113 ms; 200-250k tokens n=101 avg 5192 ms, max
67247 ms. One day carried 266 "Kompress slow compress" markers totalling
682s of inference, max single inference 21.9s. 250 requests were refused
compression by the quarantine, costing ~3.56M tokens, 8.7% of that day's
achievable savings (per-day: 2.2%, 19.4%, 3.7%, 8.7%). With the deadline
shared, the same request stops kompressing at the budget and returns
instead of overrunning the 30s pipeline timeout.
- Not tested: The fleet-wide prevalence of the quarantine cascade, which
no counter reports today; and the CUDA/MPS batched path, which shares
one deadline inside `compress_batch` already and was not re-measured on
this change.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. A request whose kompress
inference exceeds `HEADROOM_COMPRESSION_DEADLINE_MS` in total now stops
compressing further blocks, where before each block restarted the
budget. That is a reduction in per-request compression for requests that
were previously overrunning the pipeline timeout, and an increase for
every request that was being quarantined behind them.
- Kill switch / disable path: Set `HEADROOM_COMPRESSION_DEADLINE_MS`
high (for example `600000`) to restore effectively unbounded per-request
inference.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit; the deadline returns to being
per-block.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
This commit is contained in:
gglucass
2026-09-23 23:43:08 -05:00
committed by GitHub
parent 1a7c5bbf52
commit 7f2766ca4b
3 changed files with 143 additions and 1 deletions
+25 -1
View File
@@ -1866,6 +1866,13 @@ class _PerRequestRuntimeState:
# a compression-policy misapplication.
protect_read_tool_ids: set[str] = field(default_factory=set)
protect_read_msg_indices: set[int] = field(default_factory=set)
# ``time.perf_counter()`` origin shared by every kompress call this
# request makes. ``HEADROOM_COMPRESSION_DEADLINE_MS`` is a deadline on
# kompress inference, but each ``compress()`` call started its own clock,
# so the budget bounded a BLOCK and a request with a dozen blocks could
# spend a dozen budgets. Set once per ``apply()``; see
# ``_try_ml_compressor``.
kompress_deadline_started_at: float | None = None
# Monotonic counter so each ContentRouter instance gets a uniquely named
@@ -4232,6 +4239,18 @@ class ContentRouter(Transform):
# don't accept the kwarg are unaffected on the common path.
if protected:
compress_kwargs["ccr_original"] = content
# One deadline for the whole request, not one per
# block. Without this a request with N compressible
# blocks gets N full deadlines and can run far past
# the pipeline's own compression timeout; the worker
# that overruns cannot be preempted, so it opens the
# timeout-debt quarantine and every request behind it
# forwards with no compression at all.
deadline_origin = self._runtime_state_var.get().kompress_deadline_started_at
if deadline_origin is not None and getattr(
compressor, "shares_request_deadline", False
):
compress_kwargs["_deadline_started_at"] = deadline_origin
result = compressor.compress(text_to_compress, **compress_kwargs)
compressed = result.compressed
compressed_tokens = result.compressed_tokens
@@ -5141,7 +5160,12 @@ class ContentRouter(Transform):
# call does. No `reset()` is needed: every `apply()` call installs
# its own fresh object up front, so the next call on a reused worker
# thread simply overwrites the ambient value before reading it.
self._runtime_state_var.set(_PerRequestRuntimeState())
# The kompress deadline origin is stamped HERE, with the rest of the
# per-request state, so every block this call compresses draws down one
# shared budget instead of restarting it.
self._runtime_state_var.set(
_PerRequestRuntimeState(kompress_deadline_started_at=time.perf_counter())
)
# Pre-process: Read lifecycle management (stale/superseded detection)
if self.config.read_lifecycle.enabled:
@@ -1416,6 +1416,14 @@ class KompressCompressor(Transform):
name: str = "kompress_compressor"
# ``compress()`` accepts ``_deadline_started_at``, so a caller that
# compresses many blocks for ONE request can hand every call the same
# origin and have the deadline bound the request. Duck-typed rather than
# isinstance-checked at the call site because ``RemoteKompressCompressor``
# is the other compressor the router may get back and its ``compress()``
# does not take the argument.
shares_request_deadline: bool = True
def __init__(self, config: KompressConfig | None = None):
self.config = config or KompressConfig()
# Set by the preload canary when inference is too slow to be useful, or by
@@ -108,3 +108,113 @@ def test_compress_partial_run_keeps_processed_head_plus_verbatim_tail(
# chunk 1 tripped the deadline -> its words kept verbatim (all present)
for i in range(half, n_words):
assert f"w{i}" in out
def _two_block_messages(salt: str = "a") -> list[dict]:
"""Two tool results, each big enough to reach the kompress stage.
``salt`` varies the text so a second request is not served from the
router's content cache.
"""
return [
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": f"toolu_{n}",
"content": " ".join(
f'{{"file":"src/mod_{i}.py","line":{i},"text":"payload {salt}{n}"}}'
for i in range(160)
),
}
for n in range(2)
],
}
]
def _tokenizer():
from headroom.providers import OpenAIProvider
from headroom.tokenizer import Tokenizer
provider = OpenAIProvider()
return Tokenizer(provider.get_token_counter("gpt-4o"), "gpt-4o")
class _RecordingKompress:
"""Stands in for KompressCompressor, recording the deadline it is handed."""
shares_request_deadline = True
def __init__(self):
self.deadlines: list[float | None] = []
def is_ready(self) -> bool:
return True
def ensure_background_load(self) -> None: # pragma: no cover - never reached
pass
def compress(self, content, **kwargs):
from types import SimpleNamespace
self.deadlines.append(kwargs.get("_deadline_started_at"))
compressed = " ".join(content.split()[:20])
return SimpleNamespace(compressed=compressed, compressed_tokens=len(compressed.split()))
def test_every_block_of_one_request_draws_down_the_same_deadline(monkeypatch):
"""The budget must bound the REQUEST, not each block.
``compress()`` starts its own clock when no origin is passed, so a request
with N compressible blocks used to get N full deadlines. That is how a
single request runs past the pipeline's compression timeout; the worker
that overruns cannot be preempted, so it opens the timeout-debt quarantine
and every request queued behind it forwards with no compression at all.
"""
from headroom.transforms.content_router import ContentRouter
router = ContentRouter()
fake = _RecordingKompress()
monkeypatch.setattr(router, "_get_kompress", lambda: fake)
router.apply(_two_block_messages(), _tokenizer(), force_kompress=True)
assert len(fake.deadlines) >= 2, f"expected both blocks to reach kompress: {fake.deadlines}"
assert all(d is not None for d in fake.deadlines), fake.deadlines
assert len(set(fake.deadlines)) == 1, f"each block restarted the clock: {fake.deadlines}"
def test_a_second_request_gets_a_fresh_deadline(monkeypatch):
"""Sharing is per-request: the next request must not inherit a spent budget."""
from headroom.transforms.content_router import ContentRouter
router = ContentRouter()
fake = _RecordingKompress()
monkeypatch.setattr(router, "_get_kompress", lambda: fake)
tokenizer = _tokenizer()
router.apply(_two_block_messages("first"), tokenizer, force_kompress=True)
first = set(fake.deadlines)
router.apply(_two_block_messages("second"), tokenizer, force_kompress=True)
assert len(set(fake.deadlines) - first) == 1, (
f"second request reused a deadline: {fake.deadlines}"
)
def test_a_compressor_that_cannot_take_the_deadline_is_not_handed_one(monkeypatch):
"""``RemoteKompressCompressor.compress()`` has no ``_deadline_started_at``
parameter, so passing it would raise TypeError and fail compression open."""
from headroom.transforms.content_router import ContentRouter
router = ContentRouter()
fake = _RecordingKompress()
fake.shares_request_deadline = False
monkeypatch.setattr(router, "_get_kompress", lambda: fake)
router.apply(_two_block_messages(), _tokenizer(), force_kompress=True)
assert fake.deadlines, "the stub should still have been called"
assert all(d is None for d in fake.deadlines), fake.deadlines