fix(proxy): share one compression deadline across a Responses request (#3938)

## Description

#3693 made `HEADROOM_COMPRESSION_DEADLINE_MS` bound a request instead of
a block, but only inside `ContentRouter.apply()`. OpenAI Responses
requests (Codex HTTP, the Codex WebSocket path, the Responses
passthrough) never call `apply()`: the adapter compresses each text unit
and each small-unit batch with its own `ContentRouter.compress()` call,
several at a time on `_openai_responses_unit_executor()`. No deadline
origin exists on that path, so every Kompress call starts a fresh 20s
clock. A request with many tool outputs runs far past its compression
timeout; the awaiter gives up and forwards it uncompressed while the
worker keeps running inference, and that timeout debt quarantines
compression for every other request until it finishes.

This PR reuses #3693's mechanism on the Responses path: one deadline
origin per request, bound as the router's per-request
`kompress_deadline_started_at`, for every unit and batch, including the
ones on the unit pool.

Closes #3937

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_compress_openai_responses_payload_in_executor` stamps one origin per
request (`_openai_responses_deadline_started_at`) before the job is
queued, because the caller's `timeout` counts queue wait too, and passes
it down through `_compress_openai_responses_payload` and
`_compress_openai_responses_live_text_units_with_router`.
`deadline_started_at` joins the existing list of optional keywords the
TypeError retry drops, so a narrow compressor override keeps working.
- The budget is `min(HEADROOM_COMPRESSION_DEADLINE_MS, 0.75 * timeout)`.
With the defaults (20s, 30s) that is the 20s deadline unchanged. On the
Codex WebSocket path, which awaits at most 5s, it is 3.75s; before, each
Kompress call allowed itself 20s there. Kompress checks the deadline
only at chunk boundaries, so the chunk in flight at expiry, the splice
and the output serialization need the remaining 25%. Kompress measures
`now - origin > deadline`, so the cap is expressed as a backdated origin
and Kompress itself is unchanged. `HEADROOM_COMPRESSION_DEADLINE_MS=0`
still disables the deadline (no origin).
- `ContentRouter.share_request_deadline(started_at)` binds a fresh
`_PerRequestRuntimeState(kompress_deadline_started_at=started_at)` in
the current Context, the same state `apply()` binds, so
`_try_ml_compressor` forwards the origin to Kompress unchanged. It
returns `False` and binds nothing once the deadline has passed.
- `compress_unit_with_router` and `compress_batch_with_router` take an
optional `deadline_started_at` and call it before entering the router. A
unit or batch that has not started by the deadline comes back unchanged
with reason `deadline_exceeded` (new unit reason category `deadline`),
without entering the router. Default `None` keeps every other caller as
before.
- The two unit-pool `submit` calls run each task in
`contextvars.copy_context()`. Each task binds fresh state (a shared
state object would race on the `_runtime_target_ratio` set/restore in
`compress_unit_with_router`), and the copy keeps that binding from
outliving the task on a long-lived pool thread.
- Finished units are kept, no thread is interrupted, and the unit result
cache and replay policy are untouched: the cache still stores exactly
what was forwarded, including a unit passed through because of the
deadline, so the next turn forwards the same bytes.
- Tests: `test_openai_responses_units_share_one_kompress_deadline`
(parallel fan-out, every Kompress call gets one non-`None` origin),
`test_openai_responses_units_after_the_deadline_pass_through_unchanged`
(finished unit kept, later units byte-identical, Kompress entered once),
`test_openai_responses_deadline_budget_stays_inside_the_timeout` (cap
and opt-out),
`test_batch_after_the_request_deadline_passes_through_without_routing`.
Three exact-kwargs assertions in `tests/test_proxy_openai.py` now also
ignore `deadline_started_at`, the same way they already ignore `timing`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Unit tests: targeted files only (the new tests plus the existing suites
for every module touched), not the whole suite. Lint and types: `ruff
check`, `ruff format --check` and `mypy` on the changed files, not the
whole tree.

### Test Output

New behavioural tests with only the `headroom/` changes reverted
(upstream `main` d5318ac2):

```text
$ python -m pytest "tests/test_openai_responses_compression_units.py::test_openai_responses_units_share_one_kompress_deadline" "tests/test_openai_responses_compression_units.py::test_openai_responses_units_after_the_deadline_pass_through_unchanged" -q -p no:cacheprovider
E   AssertionError: [None, None, None, None, None, None]
E   assert None not in [None, None, None, None, None, None]
E   assert 4 == 1
E    +  where 4 = len([None, None, None, None])
FAILED tests/test_openai_responses_compression_units.py::test_openai_responses_units_share_one_kompress_deadline
FAILED tests/test_openai_responses_compression_units.py::test_openai_responses_units_after_the_deadline_pass_through_unchanged
============================== 2 failed in 1.84s ===============================
```

With the fix:

```text
$ python -m pytest tests/test_openai_responses_compression_units.py tests/test_compression_batches.py tests/test_compression_units.py tests/test_transforms/test_kompress_deadline.py tests/test_content_router_concurrency.py tests/test_proxy_openai.py tests/test_openai_responses_additional_tools.py tests/test_codex_ws_compression_scheduler.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py tests/test_openai_codex_ws_timings.py tests/test_openai_codex_routing.py tests/test_responses_savings_denominator.py tests/test_compress_passthrough.py tests/test_openai_responses_read_protection.py tests/test_openai_responses_context_compaction.py tests/test_openai_responses_t3_replay_regression.py tests/test_compression_determinism.py tests/test_html_tool_reversibility.py tests/test_proxy_compression_executor.py tests/test_agent_savings.py -q -p no:cacheprovider
================== 334 passed, 2 skipped, 1 warning in 12.26s ==================

$ ruff check <7 changed files> && ruff format --check <7 changed files>   # ruff 0.16.3
All checks passed!
7 files already formatted

$ mypy headroom/proxy/handlers/openai.py headroom/transforms/compression_batches.py headroom/transforms/compression_units.py headroom/transforms/content_router.py
Success: no issues found in 4 source files
```

## Real Behavior Proof

- Environment: macOS 27.0.1 arm64, Python 3.11.13. Upstream `main`
d5318ac2 versus this branch, same venv.
- Exact command / steps: an offline probe (script below) builds the real
proxy with `create_app(ProxyConfig(...))` and calls
`HeadroomProxy._compress_openai_responses_payload_in_executor` with 60
Codex `function_call_output` items of 700 words. The path is all real:
`_run_compression_in_executor`, `_compress_openai_responses_payload`,
the unit fan-out on `_openai_responses_unit_executor` (parallelism 4),
`ContentRouter`, and the `KompressCompressor` chunk loop with its
execution slot and deadline checks. Only the ONNX model is replaced, by
a fake that sleeps per 350-word chunk, and the token counter is a word
counter. Network is blocked in-process. Times are scaled down 10x:
deadline 2s (default 20s); timeout 3s (HTTP default 30s) or 0.5s (Codex
WS 5s). After the request it waits for the worker to drain, then sends
an unrelated request. `PROBE_DEADLINE_MS=2000 PROBE_TIMEOUT=<t>
PROBE_DELAY_MS=<ms> PYTHONPATH=. python responses_budget_probe.py`
- Observed result: on main the request times out and the worker keeps
running inference for 6.5-10s, quarantining the next request; on this
branch the request returns inside its budget, nothing runs after it, and
nothing is quarantined (output below).

```text
=== main @ d5318ac2  PROBE_TIMEOUT=3.0 PROBE_DELAY_MS=100   (HTTP proportions)
request: TIMED OUT after 3002ms
inference calls: total=120, started after the 2000ms budget=101 (latest +10955.6ms), after the awaiter returned=93
worker busy until 13078ms (10075ms after the awaiter returned)
timeout debt: leaked_threads=1, quarantine_activations=1
next request right after a slow one (timed out): CompressionQuarantinedError: compression quarantined: 1 timed-out worker(s) still running

=== branch @ cfa775a1  PROBE_TIMEOUT=3.0 PROBE_DELAY_MS=100
request: completed after 2069ms
inference calls: total=19, started after the 2000ms budget=0, after the awaiter returned=0
worker busy until 2069ms (0ms after the awaiter returned)
units compressed=10/60, passed through byte-identical=50, tokens_saved=3175
timeout debt: leaked_threads=0, quarantine_activations=0
next request right after a slow one (completed): ran, not quarantined

=== main @ d5318ac2  PROBE_TIMEOUT=0.5 PROBE_DELAY_MS=50   (Codex WS proportions)
request: TIMED OUT after 501ms
inference calls: total=120, started after the 375ms budget=113 (latest +6570.6ms), after the awaiter returned=112
worker busy until 7019ms (6518ms after the awaiter returned)
timeout debt: leaked_threads=1, quarantine_activations=1
next request right after a slow one (timed out): CompressionQuarantinedError: compression quarantined: 1 timed-out worker(s) still running

=== branch @ cfa775a1  PROBE_TIMEOUT=0.5 PROBE_DELAY_MS=50
request: completed after 415ms
inference calls: total=7, started after the 375ms budget=0, after the awaiter returned=0
worker busy until 415ms (0ms after the awaiter returned)
units compressed=4/60, passed through byte-identical=56, tokens_saved=1165
timeout debt: leaked_threads=0, quarantine_activations=0
next request right after a slow one (completed): ran, not quarantined
```

On main the request is forwarded uncompressed at the timeout, the worker
then runs another 93 to 112 inference calls for 6.5 to 10s, and the next
request is quarantined. On the branch the request returns inside its
timeout with the units it finished, nothing runs after it returns, and
nothing is quarantined. The budget column is measured from just before
the call, so it slightly precedes the handler's own origin; in one of
three extra WS-proportion runs a single call started 9.4ms after that
clock's expiry. That call had passed Kompress's own pre-acquire deadline
check (#3693) just before. With 100ms chunks at the 0.5s timeout (a 1s
chunk at production scale, the worst case for the 25% margin) one branch
run still timed out, at 511ms; the worker had finished at that same
moment, so no debt was recorded and nothing was quarantined.

- Not tested: no live Codex or OpenAI traffic was measured, and no real
Kompress/ONNX model was run (none is downloaded on this machine and the
probe stays offline). `headroom proxy` was not booted as a process; the
probe drives the same handler method the HTTP and WebSocket routes call,
on an app from `create_app`. `RemoteKompressCompressor` does not take
`_deadline_started_at` (unchanged from #3693): with a remote endpoint
the deadline still stops new units, but not a unit already in flight.
Linux and Windows were not run.

<details>
<summary>responses_budget_probe.py</summary>

```python
"""Offline probe: does one Responses request stay inside its compression budget?

Drives the real handler path (create_app -> HeadroomProxy ->
_compress_openai_responses_payload_in_executor -> _compress_openai_responses_payload
-> unit fan-out -> real ContentRouter -> real KompressCompressor chunk loop).
Only the ONNX model is replaced, by a fake whose inference sleeps DELAY seconds
per 350-word chunk. No network, no model download, no provider call.

PROBE_DEADLINE_MS, PROBE_TIMEOUT (seconds) and PROBE_DELAY_MS (per chunk) scale
the production defaults down (deadline 20s, HTTP timeout 30s, Codex WS timeout 5s).
"""

import asyncio
import os
import socket
import sys
import threading
import time
from types import SimpleNamespace


def deny_network(*args, **kwargs):
    raise AssertionError("probe must stay offline")


socket.socket.connect = socket.socket.connect_ex = deny_network
socket.create_connection = socket.getaddrinfo = deny_network
os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
os.environ["HEADROOM_COMPRESSION_DEADLINE_MS"] = os.environ.get("PROBE_DEADLINE_MS", "400")
os.environ.setdefault("HEADROOM_TOOL_OUTPUT_COMPRESSION_PARALLELISM", "4")

from headroom.proxy.server import ProxyConfig, create_app  # noqa: E402
from headroom.transforms import kompress_compressor as kc  # noqa: E402
from headroom.transforms.content_router import ContentRouter  # noqa: E402

DELAY = float(os.environ.get("PROBE_DELAY_MS", "20")) / 1000
TIMEOUT = float(os.environ.get("PROBE_TIMEOUT", "0.6"))
UNITS = 60
WORDS = 700  # two 350-word Kompress chunks per unit

calls: list[tuple[float, float]] = []  # (start, end) perf_counter of each inference
calls_lock = threading.Lock()


class Enc(dict):
    def word_ids(self, batch_index=0):
        return self["_word_ids"]


class Tok:
    def __call__(self, chunk_words, **kw):
        n = len(chunk_words)
        return Enc(input_ids=[[0] * n], attention_mask=[[1] * n], _word_ids=list(range(n)))


class Model:
    def _infer(self, input_ids):
        started = time.perf_counter()
        time.sleep(DELAY)
        with calls_lock:
            calls.append((started, time.perf_counter()))
        return len(input_ids[0])

    def get_keep_mask(self, input_ids, attention_mask):
        n = self._infer(input_ids)
        return [[i < n // 2 for i in range(n)]]

    def get_scores(self, input_ids, attention_mask):
        n = self._infer(input_ids)
        return [[1.0 - i / n for i in range(n)]]


class TokenCounter:
    def count_text(self, text):
        return len(text.split())


def make_proxy():
    proxy = create_app(
        ProxyConfig(
            optimize=False,
            cache_enabled=False,
            rate_limit_enabled=False,
            cost_tracking_enabled=False,
            log_requests=False,
            ccr_inject_tool=False,
            ccr_handle_responses=False,
            ccr_context_tracking=False,
            image_optimize=False,
            compression_max_workers=4,
        )
    ).state.proxy
    compressor = kc.KompressCompressor()
    kc._kompress_cache[compressor.config.model_id] = (Model(), Tok(), "onnx")
    router = ContentRouter()
    router._get_kompress = lambda: compressor
    proxy.openai_pipeline = SimpleNamespace(transforms=[router])
    proxy.openai_provider = SimpleNamespace(get_token_counter=lambda _model: TokenCounter())
    return proxy


def payload(salt):
    return {
        "model": "gpt-5",
        "input": [
            {
                "type": "function_call_output",
                "call_id": f"call_{i}",
                "output": "\n".join(
                    " ".join(f"{salt}{i}line{line}word{j}" for j in range(10))
                    for line in range(WORDS // 10)
                ),
            }
            for i in range(UNITS)
        ],
    }


async def warm_up(proxy):
    # First-use imports and detector setup must not count against the request.
    small = {"model": "gpt-5", "input": payload("warm")["input"][:1]}
    await proxy._compress_openai_responses_payload_in_executor(
        small, model="gpt-5", request_id="warm-up", timeout=30
    )
    calls.clear()


async def main():
    proxy = make_proxy()
    await warm_up(proxy)
    deadline_s = float(os.environ["HEADROOM_COMPRESSION_DEADLINE_MS"]) / 1000
    budget_s = min(deadline_s, 0.75 * TIMEOUT)
    source = payload("a")
    started = time.perf_counter()
    try:
        result = await proxy._compress_openai_responses_payload_in_executor(
            source, model="gpt-5", request_id="probe-1", timeout=TIMEOUT
        )
        outcome = "completed"
    except asyncio.TimeoutError:
        result = None
        outcome = "TIMED OUT"
    returned = time.perf_counter()
    with calls_lock:
        at_return = len(calls)
    # Is the worker still doing inference after the awaiter gave up / returned?
    while proxy._compression_in_flight:
        await asyncio.sleep(0.01)
    worker_done = time.perf_counter()
    with calls_lock:
        snapshot = list(calls)
    expiry = started + budget_s
    print(f"deadline={deadline_s * 1000:.0f}ms outer_timeout={TIMEOUT * 1000:.0f}ms "
          f"units={UNITS} chunks/unit={WORDS // 350} inference={DELAY * 1000:.0f}ms/chunk")
    print(f"request: {outcome} after {(returned - started) * 1000:.0f}ms")
    late = [s - expiry for s, _ in snapshot if s > expiry]
    print(f"inference calls: total={len(snapshot)}, "
          f"started after the {budget_s * 1000:.0f}ms budget={len(late)}"
          + (f" (latest +{max(late) * 1000:.1f}ms)" if late else "")
          + f", after the awaiter returned={len(snapshot) - at_return}")
    print(f"worker busy until {(worker_done - started) * 1000:.0f}ms "
          f"({(worker_done - returned) * 1000:.0f}ms after the awaiter returned)")
    if result is not None:
        changed = sum(a != b for a, b in zip(source["input"], result[0]["input"]))
        print(f"units compressed={changed}/{UNITS}, passed through byte-identical={UNITS - changed}, "
              f"tokens_saved={result[2]}")
        if os.environ.get("PROBE_TIMINGS"):
            for name, ms in sorted(result[8].items(), key=lambda kv: -kv[1])[:8]:
                print(f"  timing {name}={ms:.1f}ms")
    print(f"timeout debt: leaked_threads={proxy._compression_leaked_threads}, "
          f"quarantine_activations={proxy._compression_quarantine_activations}")

    # An unrelated request that arrives while the first one's worker is running.
    proxy2 = make_proxy()
    await warm_up(proxy2)
    first = asyncio.ensure_future(
        proxy2._compress_openai_responses_payload_in_executor(
            payload("b"), model="gpt-5", request_id="probe-2", timeout=TIMEOUT
        )
    )
    try:
        await first
        first_outcome = "completed"
    except asyncio.TimeoutError:
        first_outcome = "timed out"
    small = {"model": "gpt-5", "input": payload("c")["input"][:1]}
    try:
        await proxy2._compress_openai_responses_payload_in_executor(
            small, model="gpt-5", request_id="probe-3", timeout=TIMEOUT
        )
        follow = "ran, not quarantined"
    except asyncio.TimeoutError as exc:
        follow = f"{type(exc).__name__}: {exc}"
    print(f"next request right after a slow one ({first_outcome}): {follow}")
    while proxy2._compression_in_flight:
        await asyncio.sleep(0.01)
    for p in (proxy, proxy2):
        p._compression_executor.shutdown(wait=True)


if __name__ == "__main__":
    print(f"python {sys.version.split()[0]}, headroom from {os.path.dirname(kc.__file__)}")
    asyncio.run(main())
```

</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, on the OpenAI Responses
compression path only. A request whose Kompress work exceeds
`min(HEADROOM_COMPRESSION_DEADLINE_MS, 0.75 * timeout)` in total now
stops starting new inference and forwards the units it had not started
unchanged. Before, each unit restarted the deadline, so such a request
either finished inside the timeout with more units compressed (fan-outs
that took between the budget and the timeout), or timed out, was
forwarded with no compression at all, and quarantined compression for
every request behind it. Codex WS requests now budget 3.75s of the 5s
they are awaited for, instead of 20s per Kompress call. Requests that
finish inside the budget are unchanged, byte for byte. The chat and
Anthropic paths (`apply()`) are not touched.
- Kill switch / disable path: `HEADROOM_COMPRESSION_DEADLINE_MS=0`
disables the deadline (the same switch #3693 has), and a higher value or
`HEADROOM_COMPRESSION_TIMEOUT_SECONDS` widens the budget. No new
environment variable.
- Unsafe override required: No.
- Qualification impact: Responses compression only, and only on requests
whose fan-out outruns the budget. Those that used to finish between the
budget and the timeout now compress fewer units (the rest pass through
unchanged); those that used to time out and go out with no compression
now keep the units finished in time, and no longer quarantine the
requests behind them. Requests inside the budget are byte-identical. No
provider cache or replay change.
- Rollback path: Revert this commit; Responses units go back to a fresh
deadline per Kompress call.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: no docs page describes
`HEADROOM_COMPRESSION_DEADLINE_MS` per path, so nothing to update.
- Why 0.75 and not #3721's 0.5: #3721 (open) caps cumulative ML time
inside `apply()` at half the outer timeout; this caps wall time from
enqueue on the Responses path, which already includes the non-ML work.
0.75 leaves the default HTTP budget at the existing 20s deadline and
only binds when the timeout is short (Codex WS) or lowered by the
operator. The probe above shows the remaining 25% covering the chunk in
flight plus splice and serialization. #3721 arms only `apply()` and
leaves direct `compress()` callers unbounded, so the two do not overlap;
if it merges, the Responses path still gets its bound from this change.
- Related: #2360 (Codex WS worker hang -> permanent quarantine; the
quarantine time cap from #2412 limits how long one leaked worker can
block, this removes the leak on this path), #3653 (a thread/fd per
quarantine event; fewer quarantine events here), #3711 (the chat-path
counterpart).
- Headroom Desktop has shipped an equivalent downstream patch since
2026-09-07. This PR re-implements it on the existing #3693 state instead
of a separate budget.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3938"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
This commit is contained in:
gglucass
2026-10-05 08:25:17 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis Tejas Chopra
parent befdb52fd5
commit 4d27d024b0
7 changed files with 233 additions and 6 deletions
+51 -3
View File
@@ -7,6 +7,7 @@ from __future__ import annotations
import asyncio
import contextlib
import contextvars
import copy
import hashlib
import hmac
@@ -664,6 +665,31 @@ def _openai_responses_unit_parallelism() -> int:
return max(1, min(_OPENAI_RESPONSES_UNIT_PARALLELISM_MAX, requested))
def _openai_responses_deadline_started_at(timeout: float) -> float | None:
"""Deadline origin shared by every compress call of one Responses request.
A Responses request fans out over many ``ContentRouter.compress`` calls,
and Kompress starts a fresh ``HEADROOM_COMPRESSION_DEADLINE_MS`` clock in
each one unless it is handed an origin. Without one, a request past its
``timeout`` keeps running inference long after the awaiter gave up, and
that timeout debt quarantines compression for every request behind it.
Stamped before the job is queued, because ``timeout`` counts queue wait
too. ``None`` when the deadline is disabled (``0``).
The budget is also capped at 75% of ``timeout``: the deadline only stops
the NEXT Kompress chunk, so the chunk already running when it expires, the
splice and the output serialization need the rest. Kompress checks
``now - origin > deadline``, so a shorter budget is a backdated origin.
"""
from headroom.transforms.content_router import _compression_deadline_seconds
deadline_s = _compression_deadline_seconds()
if deadline_s <= 0:
return None
budget_s = min(deadline_s, 0.75 * timeout)
return time.perf_counter() - (deadline_s - budget_s)
def _openai_responses_unit_executor() -> ThreadPoolExecutor:
global _OPENAI_RESPONSES_UNIT_EXECUTOR
with _OPENAI_RESPONSES_UNIT_EXECUTOR_LOCK:
@@ -2130,6 +2156,7 @@ class OpenAIHandlerMixin:
request_id: str,
pass_id: str | None = None,
timing: dict[str, float] | None = None,
deadline_started_at: float | None = None,
) -> tuple[dict[str, Any], bool, int, list[str], dict[str, int], list[str], int]:
"""Run ContentRouter on OpenAI Responses text units.
@@ -2723,6 +2750,7 @@ class OpenAIHandlerMixin:
router=router,
tokenizer=tokenizer,
target_ratio=unit_target_ratio,
deadline_started_at=deadline_started_at,
)
elapsed_ms = (time.perf_counter() - unit_started) * 1000.0
return routed.slot, result, elapsed_ms
@@ -2792,12 +2820,18 @@ class OpenAIHandlerMixin:
0.0,
)
# Each pool task runs in its own copy of this Context: the router binds
# the request's deadline per task (see share_request_deadline), and
# pool threads would otherwise keep that binding between requests.
parallelism = _openai_responses_unit_parallelism()
if len(cache_misses) > 1 and parallelism > 1:
executor = _openai_responses_unit_executor()
for start in range(0, len(cache_misses), parallelism):
batch = cache_misses[start : start + parallelism]
futures = [executor.submit(_compress_and_store, *item) for item in batch]
futures = [
executor.submit(contextvars.copy_context().run, _compress_and_store, *item)
for item in batch
]
for future in as_completed(futures):
unit_idx, cache_key, routed_result = future.result()
_record_routed_result(unit_idx, cache_key, routed_result)
@@ -2822,6 +2856,7 @@ class OpenAIHandlerMixin:
router=router,
tokenizer=tokenizer,
target_ratio=unit_target_ratio,
deadline_started_at=deadline_started_at,
)
return results, (time.perf_counter() - batch_started) * 1000.0
@@ -2835,7 +2870,10 @@ class OpenAIHandlerMixin:
executor = _openai_responses_unit_executor()
for start in range(0, len(small_batches), parallelism):
batch_group = small_batches[start : start + parallelism]
futures = [executor.submit(_compress_batch, batch) for batch in batch_group]
futures = [
executor.submit(contextvars.copy_context().run, _compress_batch, batch)
for batch in batch_group
]
for future in as_completed(futures):
_record_batch_result(future.result())
else:
@@ -3023,6 +3061,7 @@ class OpenAIHandlerMixin:
timing: dict[str, float] | None = None,
client: str | None = None,
savings_tags: dict[str, Any] | None = None,
deadline_started_at: float | None = None,
) -> tuple[dict[str, Any], bool, int, list[str], str | None, int, int, int]:
"""Compress an OpenAI Responses payload through the shared router.
@@ -3290,6 +3329,7 @@ class OpenAIHandlerMixin:
request_id=request_id,
pass_id=pass_id,
timing=timing_sink,
deadline_started_at=deadline_started_at,
)
_add_timing("compression_live_units_total", live_units_started)
if router_modified:
@@ -3437,6 +3477,8 @@ class OpenAIHandlerMixin:
exc_info=True,
)
deadline_started_at = _openai_responses_deadline_started_at(timeout)
def _compress(): # noqa: ANN202
# Output shaping (opt-in via HEADROOM_OUTPUT_SHAPER) runs before
# compression so the turn classifier sees the client's input as
@@ -3456,6 +3498,7 @@ class OpenAIHandlerMixin:
"request_id": request_id,
"timing": timing,
"client": client,
"deadline_started_at": deadline_started_at,
}
if savings_tags is not None:
compression_kwargs["savings_tags"] = savings_tags
@@ -3470,7 +3513,12 @@ class OpenAIHandlerMixin:
unsupported_kwarg = next(
(
name
for name in ("savings_tags", "client", "timing")
for name in (
"savings_tags",
"client",
"timing",
"deadline_started_at",
)
if f"unexpected keyword argument '{name}'" in str(exc)
and name in compression_kwargs
),
+8 -1
View File
@@ -230,8 +230,15 @@ def compress_batch_with_router(
router: ContentRouter,
tokenizer: TokenCounterLike,
target_ratio: float | None = None,
deadline_started_at: float | None = None,
) -> list[tuple[object, UnitCompressionResult]]:
"""Compress one tagged batch and split only structurally valid output."""
"""Compress one tagged batch and split only structurally valid output.
``deadline_started_at`` works as in ``compress_unit_with_router``.
"""
if deadline_started_at is not None and not router.share_request_deadline(deadline_started_at):
return _passthrough_batch_results(batch, tokenizer=tokenizer, reason="deadline_exceeded")
nonce = _batch_nonce(batch)
batch_texts, marker_blocks = _protect_ccr_markers(batch, nonce)
+11
View File
@@ -56,6 +56,8 @@ class CompressionUnit:
# Responses handler's unit-result cache) and is surfaced
# via RouterCompressionResult.cache_hit, not as a reason
# here — no code path produces this category today
# - deadline: the request's compression deadline had passed before
# this unit started, so it was left unchanged
UNIT_REASON_CATEGORIES = {
None: "applied",
"protected_user_message": "protected_role",
@@ -66,6 +68,7 @@ UNIT_REASON_CATEGORIES = {
"router_no_change": "compressor_noop",
"already_compressed": "already_compressed",
"rejected_not_smaller": "rejected_not_smaller",
"deadline_exceeded": "deadline",
}
@@ -221,11 +224,17 @@ def compress_unit_with_router(
router: ContentRouter,
tokenizer: TokenCounterLike,
target_ratio: float | None = None,
deadline_started_at: float | None = None,
) -> UnitCompressionResult:
"""Compress one safe text unit through ContentRouter.
The final accept/reject gate uses the provider/model tokenizer, not the
router's internal word-count estimates.
``deadline_started_at`` is the deadline origin of the request this unit
belongs to (see ``ContentRouter.share_request_deadline``). Once that
deadline has passed the unit is returned unchanged without entering the
router.
"""
tokens_before = tokenizer.count_text(unit.text)
@@ -266,6 +275,8 @@ def compress_unit_with_router(
return _with_reason(reason=f"cache_zone_{unit.cache_zone}")
if text_bytes < unit.min_bytes:
return _with_reason(reason="below_unit_floor")
if deadline_started_at is not None and not router.share_request_deadline(deadline_started_at):
return _with_reason(reason="deadline_exceeded")
prior_target_ratio = getattr(router, "_runtime_target_ratio", None)
if target_ratio is not None:
+21
View File
@@ -2394,6 +2394,27 @@ class ContentRouter(Transform):
self._runtime_state_var.set(state)
return state
def share_request_deadline(self, started_at: float) -> bool:
"""Join a request whose kompress deadline started at ``started_at``.
``apply()`` stamps its own origin. A caller that fans ONE request out
over many ``compress()`` calls (the OpenAI Responses unit adapter) calls
this before each one instead, so every call draws down the same
``HEADROOM_COMPRESSION_DEADLINE_MS`` budget rather than restarting it.
Binds a fresh ``_PerRequestRuntimeState`` in the CURRENT Context, so a
worker-pool task must run in its own ``contextvars.copy_context()``.
Returns ``False``, binding nothing, once the deadline has passed: the
caller should leave the content unchanged rather than start new work.
"""
deadline_s = _compression_deadline_seconds()
if deadline_s and time.perf_counter() - started_at > deadline_s:
return False
self._runtime_state_var.set(
_PerRequestRuntimeState(kompress_deadline_started_at=started_at)
)
return True
@property
def _runtime_compression_policy(self) -> Any:
return self._runtime_state_var.get().compression_policy
+25
View File
@@ -1,6 +1,7 @@
from __future__ import annotations
import re
import time
from headroom.transforms.compression_batches import (
CompressionBatchEntry,
@@ -229,3 +230,27 @@ def test_batch_uses_utf8_bytes_for_cjk_small_units():
assert batches[0].text_bytes == 1800
assert all(result.modified for _, result in results)
assert [result.compressed for _, result in results] == ["短"] * 4
def test_batch_after_the_request_deadline_passes_through_without_routing(monkeypatch):
from headroom.transforms.content_router import ContentRouter
monkeypatch.setenv("HEADROOM_COMPRESSION_DEADLINE_MS", "20000")
entries = [_entry(index, "x" * 150) for index in range(4)]
batches, _ = build_compression_batches(entries, min_batch_bytes=512)
router = ContentRouter()
def _not_called(*_args, **_kwargs):
raise AssertionError("a batch past the deadline must not enter the router")
router.compress = _not_called
results = compress_batch_with_router(
batches[0],
router=router,
tokenizer=_CharacterCounter(),
deadline_started_at=time.perf_counter() - 21,
)
assert [result.compressed for _, result in results] == ["x" * 150] * 4
assert {result.reason for _, result in results} == {"deadline_exceeded"}
@@ -1,7 +1,9 @@
from __future__ import annotations
import asyncio
import json
import threading
import time
from types import MethodType, SimpleNamespace
import pytest
@@ -1417,6 +1419,114 @@ def test_openai_responses_adapter_compresses_historical_messages_not_current_use
assert any(t.startswith("router:openai:responses:message:") for t in transforms)
class _RecordingKompress:
"""Stands in for KompressCompressor, recording the deadline origin it gets."""
shares_request_deadline = True
def __init__(self, delay: float = 0.0) -> None:
self.delay = delay
self.deadlines: list[float | None] = []
def is_ready(self) -> bool:
return True
def compress(self, content: str, **kwargs):
self.deadlines.append(kwargs.get("_deadline_started_at"))
time.sleep(self.delay)
compressed = " ".join(content.split()[:20])
return SimpleNamespace(compressed=compressed, compressed_tokens=20)
def _kompress_payload(units: int) -> dict:
return {
"model": "gpt-5",
"input": [
{
"type": "function_call_output",
"call_id": f"call_{i}",
"output": "\n".join(
" ".join(f"unit{i}line{line}word{j}" for j in range(10)) for line in range(70)
),
}
for i in range(units)
],
}
def _compress_through_executor(router: ContentRouter, payload: dict) -> tuple:
handler = _handler_with_router(router)
async def _run_compression(fn, *, timeout): # noqa: ANN001, ANN202
return fn()
handler._run_compression_in_executor = _run_compression
return asyncio.run(
handler._compress_openai_responses_payload_in_executor(
payload, model="gpt-5", request_id="req_deadline"
)
)
def test_openai_responses_units_share_one_kompress_deadline(monkeypatch):
"""The fan-out must draw down ONE request deadline (#3693), pool threads too.
Each unit is its own ContentRouter.compress call, and Kompress started a
fresh HEADROOM_COMPRESSION_DEADLINE_MS clock in every one, so a request
that blew its timeout kept running inference for minutes afterwards.
"""
monkeypatch.setenv("HEADROOM_TOOL_OUTPUT_COMPRESSION_PARALLELISM", "4")
monkeypatch.setenv("HEADROOM_COMPRESSION_DEADLINE_MS", "20000")
router = ContentRouter()
kompress = _RecordingKompress()
monkeypatch.setattr(router, "_get_kompress", lambda: kompress)
_compress_through_executor(router, _kompress_payload(6))
assert len(kompress.deadlines) >= 6, kompress.deadlines
assert None not in kompress.deadlines, kompress.deadlines
assert len(set(kompress.deadlines)) == 1, kompress.deadlines
def test_openai_responses_units_after_the_deadline_pass_through_unchanged(monkeypatch):
"""Finished units are kept; units not started by the deadline are not started."""
monkeypatch.setenv("HEADROOM_TOOL_OUTPUT_COMPRESSION_PARALLELISM", "1")
monkeypatch.setenv("HEADROOM_COMPRESSION_DEADLINE_MS", "200")
router = ContentRouter()
kompress = _RecordingKompress(delay=0.3)
monkeypatch.setattr(router, "_get_kompress", lambda: kompress)
payload = _kompress_payload(4)
original = json.loads(json.dumps(payload))
result = _compress_through_executor(router, payload)
assert len(kompress.deadlines) == 1
outputs = [item["output"] for item in result[0]["input"]]
assert outputs[0] != original["input"][0]["output"]
assert outputs[1:] == [item["output"] for item in original["input"][1:]]
assert result[2] > 0
@pytest.mark.parametrize(
("deadline_ms", "timeout", "budget"),
[("20000", 30.0, 20.0), ("20000", 5.0, 3.75), ("0", 30.0, None)],
ids=["deadline-binds", "timeout-binds", "disabled"],
)
def test_openai_responses_deadline_budget_stays_inside_the_timeout(
monkeypatch, deadline_ms, timeout, budget
):
monkeypatch.setenv("HEADROOM_COMPRESSION_DEADLINE_MS", deadline_ms)
before = time.perf_counter()
origin = openai_handler._openai_responses_deadline_started_at(timeout)
after = time.perf_counter()
if budget is None:
assert origin is None
return
expires_at = origin + int(deadline_ms) / 1000
assert before + budget <= expires_at <= after + budget
@pytest.mark.parametrize(
("compress_user_messages", "expected_user_text"),
[(False, None), (None, "compressed history")],
+7 -2
View File
@@ -132,7 +132,10 @@ def test_native_responses_compressor_scopes_the_exclusion_per_call() -> None:
asyncio.run(_run())
assert [{key: value for key, value in call.items() if key != "timing"} for call in seen] == [
assert [
{key: value for key, value in call.items() if key not in {"timing", "deadline_started_at"}}
for call in seen
] == [
{"client": "opencode"},
{"client": None},
]
@@ -445,4 +448,6 @@ def test_websocket_route_carries_the_client_decision(
asyncio.run(handler.handle_openai_responses_ws(client_ws))
assert seen, "the Responses compressor was never reached on the WS path"
assert {k: v for k, v in seen[0].items() if k != "timing"} == expected
assert {
k: v for k, v in seen[0].items() if k not in {"timing", "deadline_started_at"}
} == expected