mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): share one compression deadline across a Responses request (#3938)
## Description #3693 made `HEADROOM_COMPRESSION_DEADLINE_MS` bound a request instead of a block, but only inside `ContentRouter.apply()`. OpenAI Responses requests (Codex HTTP, the Codex WebSocket path, the Responses passthrough) never call `apply()`: the adapter compresses each text unit and each small-unit batch with its own `ContentRouter.compress()` call, several at a time on `_openai_responses_unit_executor()`. No deadline origin exists on that path, so every Kompress call starts a fresh 20s clock. A request with many tool outputs runs far past its compression timeout; the awaiter gives up and forwards it uncompressed while the worker keeps running inference, and that timeout debt quarantines compression for every other request until it finishes. This PR reuses #3693's mechanism on the Responses path: one deadline origin per request, bound as the router's per-request `kompress_deadline_started_at`, for every unit and batch, including the ones on the unit pool. Closes #3937 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `_compress_openai_responses_payload_in_executor` stamps one origin per request (`_openai_responses_deadline_started_at`) before the job is queued, because the caller's `timeout` counts queue wait too, and passes it down through `_compress_openai_responses_payload` and `_compress_openai_responses_live_text_units_with_router`. `deadline_started_at` joins the existing list of optional keywords the TypeError retry drops, so a narrow compressor override keeps working. - The budget is `min(HEADROOM_COMPRESSION_DEADLINE_MS, 0.75 * timeout)`. With the defaults (20s, 30s) that is the 20s deadline unchanged. On the Codex WebSocket path, which awaits at most 5s, it is 3.75s; before, each Kompress call allowed itself 20s there. Kompress checks the deadline only at chunk boundaries, so the chunk in flight at expiry, the splice and the output serialization need the remaining 25%. Kompress measures `now - origin > deadline`, so the cap is expressed as a backdated origin and Kompress itself is unchanged. `HEADROOM_COMPRESSION_DEADLINE_MS=0` still disables the deadline (no origin). - `ContentRouter.share_request_deadline(started_at)` binds a fresh `_PerRequestRuntimeState(kompress_deadline_started_at=started_at)` in the current Context, the same state `apply()` binds, so `_try_ml_compressor` forwards the origin to Kompress unchanged. It returns `False` and binds nothing once the deadline has passed. - `compress_unit_with_router` and `compress_batch_with_router` take an optional `deadline_started_at` and call it before entering the router. A unit or batch that has not started by the deadline comes back unchanged with reason `deadline_exceeded` (new unit reason category `deadline`), without entering the router. Default `None` keeps every other caller as before. - The two unit-pool `submit` calls run each task in `contextvars.copy_context()`. Each task binds fresh state (a shared state object would race on the `_runtime_target_ratio` set/restore in `compress_unit_with_router`), and the copy keeps that binding from outliving the task on a long-lived pool thread. - Finished units are kept, no thread is interrupted, and the unit result cache and replay policy are untouched: the cache still stores exactly what was forwarded, including a unit passed through because of the deadline, so the next turn forwards the same bytes. - Tests: `test_openai_responses_units_share_one_kompress_deadline` (parallel fan-out, every Kompress call gets one non-`None` origin), `test_openai_responses_units_after_the_deadline_pass_through_unchanged` (finished unit kept, later units byte-identical, Kompress entered once), `test_openai_responses_deadline_budget_stays_inside_the_timeout` (cap and opt-out), `test_batch_after_the_request_deadline_passes_through_without_routing`. Three exact-kwargs assertions in `tests/test_proxy_openai.py` now also ignore `deadline_started_at`, the same way they already ignore `timing`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Unit tests: targeted files only (the new tests plus the existing suites for every module touched), not the whole suite. Lint and types: `ruff check`, `ruff format --check` and `mypy` on the changed files, not the whole tree. ### Test Output New behavioural tests with only the `headroom/` changes reverted (upstream `main`d5318ac2): ```text $ python -m pytest "tests/test_openai_responses_compression_units.py::test_openai_responses_units_share_one_kompress_deadline" "tests/test_openai_responses_compression_units.py::test_openai_responses_units_after_the_deadline_pass_through_unchanged" -q -p no:cacheprovider E AssertionError: [None, None, None, None, None, None] E assert None not in [None, None, None, None, None, None] E assert 4 == 1 E + where 4 = len([None, None, None, None]) FAILED tests/test_openai_responses_compression_units.py::test_openai_responses_units_share_one_kompress_deadline FAILED tests/test_openai_responses_compression_units.py::test_openai_responses_units_after_the_deadline_pass_through_unchanged ============================== 2 failed in 1.84s =============================== ``` With the fix: ```text $ python -m pytest tests/test_openai_responses_compression_units.py tests/test_compression_batches.py tests/test_compression_units.py tests/test_transforms/test_kompress_deadline.py tests/test_content_router_concurrency.py tests/test_proxy_openai.py tests/test_openai_responses_additional_tools.py tests/test_codex_ws_compression_scheduler.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_codex_ws_lifecycle.py tests/test_openai_codex_ws_timings.py tests/test_openai_codex_routing.py tests/test_responses_savings_denominator.py tests/test_compress_passthrough.py tests/test_openai_responses_read_protection.py tests/test_openai_responses_context_compaction.py tests/test_openai_responses_t3_replay_regression.py tests/test_compression_determinism.py tests/test_html_tool_reversibility.py tests/test_proxy_compression_executor.py tests/test_agent_savings.py -q -p no:cacheprovider ================== 334 passed, 2 skipped, 1 warning in 12.26s ================== $ ruff check <7 changed files> && ruff format --check <7 changed files> # ruff 0.16.3 All checks passed! 7 files already formatted $ mypy headroom/proxy/handlers/openai.py headroom/transforms/compression_batches.py headroom/transforms/compression_units.py headroom/transforms/content_router.py Success: no issues found in 4 source files ``` ## Real Behavior Proof - Environment: macOS 27.0.1 arm64, Python 3.11.13. Upstream `main`d5318ac2versus this branch, same venv. - Exact command / steps: an offline probe (script below) builds the real proxy with `create_app(ProxyConfig(...))` and calls `HeadroomProxy._compress_openai_responses_payload_in_executor` with 60 Codex `function_call_output` items of 700 words. The path is all real: `_run_compression_in_executor`, `_compress_openai_responses_payload`, the unit fan-out on `_openai_responses_unit_executor` (parallelism 4), `ContentRouter`, and the `KompressCompressor` chunk loop with its execution slot and deadline checks. Only the ONNX model is replaced, by a fake that sleeps per 350-word chunk, and the token counter is a word counter. Network is blocked in-process. Times are scaled down 10x: deadline 2s (default 20s); timeout 3s (HTTP default 30s) or 0.5s (Codex WS 5s). After the request it waits for the worker to drain, then sends an unrelated request. `PROBE_DEADLINE_MS=2000 PROBE_TIMEOUT=<t> PROBE_DELAY_MS=<ms> PYTHONPATH=. python responses_budget_probe.py` - Observed result: on main the request times out and the worker keeps running inference for 6.5-10s, quarantining the next request; on this branch the request returns inside its budget, nothing runs after it, and nothing is quarantined (output below). ```text === main @d5318ac2PROBE_TIMEOUT=3.0 PROBE_DELAY_MS=100 (HTTP proportions) request: TIMED OUT after 3002ms inference calls: total=120, started after the 2000ms budget=101 (latest +10955.6ms), after the awaiter returned=93 worker busy until 13078ms (10075ms after the awaiter returned) timeout debt: leaked_threads=1, quarantine_activations=1 next request right after a slow one (timed out): CompressionQuarantinedError: compression quarantined: 1 timed-out worker(s) still running === branch @cfa775a1PROBE_TIMEOUT=3.0 PROBE_DELAY_MS=100 request: completed after 2069ms inference calls: total=19, started after the 2000ms budget=0, after the awaiter returned=0 worker busy until 2069ms (0ms after the awaiter returned) units compressed=10/60, passed through byte-identical=50, tokens_saved=3175 timeout debt: leaked_threads=0, quarantine_activations=0 next request right after a slow one (completed): ran, not quarantined === main @d5318ac2PROBE_TIMEOUT=0.5 PROBE_DELAY_MS=50 (Codex WS proportions) request: TIMED OUT after 501ms inference calls: total=120, started after the 375ms budget=113 (latest +6570.6ms), after the awaiter returned=112 worker busy until 7019ms (6518ms after the awaiter returned) timeout debt: leaked_threads=1, quarantine_activations=1 next request right after a slow one (timed out): CompressionQuarantinedError: compression quarantined: 1 timed-out worker(s) still running === branch @cfa775a1PROBE_TIMEOUT=0.5 PROBE_DELAY_MS=50 request: completed after 415ms inference calls: total=7, started after the 375ms budget=0, after the awaiter returned=0 worker busy until 415ms (0ms after the awaiter returned) units compressed=4/60, passed through byte-identical=56, tokens_saved=1165 timeout debt: leaked_threads=0, quarantine_activations=0 next request right after a slow one (completed): ran, not quarantined ``` On main the request is forwarded uncompressed at the timeout, the worker then runs another 93 to 112 inference calls for 6.5 to 10s, and the next request is quarantined. On the branch the request returns inside its timeout with the units it finished, nothing runs after it returns, and nothing is quarantined. The budget column is measured from just before the call, so it slightly precedes the handler's own origin; in one of three extra WS-proportion runs a single call started 9.4ms after that clock's expiry. That call had passed Kompress's own pre-acquire deadline check (#3693) just before. With 100ms chunks at the 0.5s timeout (a 1s chunk at production scale, the worst case for the 25% margin) one branch run still timed out, at 511ms; the worker had finished at that same moment, so no debt was recorded and nothing was quarantined. - Not tested: no live Codex or OpenAI traffic was measured, and no real Kompress/ONNX model was run (none is downloaded on this machine and the probe stays offline). `headroom proxy` was not booted as a process; the probe drives the same handler method the HTTP and WebSocket routes call, on an app from `create_app`. `RemoteKompressCompressor` does not take `_deadline_started_at` (unchanged from #3693): with a remote endpoint the deadline still stops new units, but not a unit already in flight. Linux and Windows were not run. <details> <summary>responses_budget_probe.py</summary> ```python """Offline probe: does one Responses request stay inside its compression budget? Drives the real handler path (create_app -> HeadroomProxy -> _compress_openai_responses_payload_in_executor -> _compress_openai_responses_payload -> unit fan-out -> real ContentRouter -> real KompressCompressor chunk loop). Only the ONNX model is replaced, by a fake whose inference sleeps DELAY seconds per 350-word chunk. No network, no model download, no provider call. PROBE_DEADLINE_MS, PROBE_TIMEOUT (seconds) and PROBE_DELAY_MS (per chunk) scale the production defaults down (deadline 20s, HTTP timeout 30s, Codex WS timeout 5s). """ import asyncio import os import socket import sys import threading import time from types import SimpleNamespace def deny_network(*args, **kwargs): raise AssertionError("probe must stay offline") socket.socket.connect = socket.socket.connect_ex = deny_network socket.create_connection = socket.getaddrinfo = deny_network os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True" os.environ["HEADROOM_COMPRESSION_DEADLINE_MS"] = os.environ.get("PROBE_DEADLINE_MS", "400") os.environ.setdefault("HEADROOM_TOOL_OUTPUT_COMPRESSION_PARALLELISM", "4") from headroom.proxy.server import ProxyConfig, create_app # noqa: E402 from headroom.transforms import kompress_compressor as kc # noqa: E402 from headroom.transforms.content_router import ContentRouter # noqa: E402 DELAY = float(os.environ.get("PROBE_DELAY_MS", "20")) / 1000 TIMEOUT = float(os.environ.get("PROBE_TIMEOUT", "0.6")) UNITS = 60 WORDS = 700 # two 350-word Kompress chunks per unit calls: list[tuple[float, float]] = [] # (start, end) perf_counter of each inference calls_lock = threading.Lock() class Enc(dict): def word_ids(self, batch_index=0): return self["_word_ids"] class Tok: def __call__(self, chunk_words, **kw): n = len(chunk_words) return Enc(input_ids=[[0] * n], attention_mask=[[1] * n], _word_ids=list(range(n))) class Model: def _infer(self, input_ids): started = time.perf_counter() time.sleep(DELAY) with calls_lock: calls.append((started, time.perf_counter())) return len(input_ids[0]) def get_keep_mask(self, input_ids, attention_mask): n = self._infer(input_ids) return [[i < n // 2 for i in range(n)]] def get_scores(self, input_ids, attention_mask): n = self._infer(input_ids) return [[1.0 - i / n for i in range(n)]] class TokenCounter: def count_text(self, text): return len(text.split()) def make_proxy(): proxy = create_app( ProxyConfig( optimize=False, cache_enabled=False, rate_limit_enabled=False, cost_tracking_enabled=False, log_requests=False, ccr_inject_tool=False, ccr_handle_responses=False, ccr_context_tracking=False, image_optimize=False, compression_max_workers=4, ) ).state.proxy compressor = kc.KompressCompressor() kc._kompress_cache[compressor.config.model_id] = (Model(), Tok(), "onnx") router = ContentRouter() router._get_kompress = lambda: compressor proxy.openai_pipeline = SimpleNamespace(transforms=[router]) proxy.openai_provider = SimpleNamespace(get_token_counter=lambda _model: TokenCounter()) return proxy def payload(salt): return { "model": "gpt-5", "input": [ { "type": "function_call_output", "call_id": f"call_{i}", "output": "\n".join( " ".join(f"{salt}{i}line{line}word{j}" for j in range(10)) for line in range(WORDS // 10) ), } for i in range(UNITS) ], } async def warm_up(proxy): # First-use imports and detector setup must not count against the request. small = {"model": "gpt-5", "input": payload("warm")["input"][:1]} await proxy._compress_openai_responses_payload_in_executor( small, model="gpt-5", request_id="warm-up", timeout=30 ) calls.clear() async def main(): proxy = make_proxy() await warm_up(proxy) deadline_s = float(os.environ["HEADROOM_COMPRESSION_DEADLINE_MS"]) / 1000 budget_s = min(deadline_s, 0.75 * TIMEOUT) source = payload("a") started = time.perf_counter() try: result = await proxy._compress_openai_responses_payload_in_executor( source, model="gpt-5", request_id="probe-1", timeout=TIMEOUT ) outcome = "completed" except asyncio.TimeoutError: result = None outcome = "TIMED OUT" returned = time.perf_counter() with calls_lock: at_return = len(calls) # Is the worker still doing inference after the awaiter gave up / returned? while proxy._compression_in_flight: await asyncio.sleep(0.01) worker_done = time.perf_counter() with calls_lock: snapshot = list(calls) expiry = started + budget_s print(f"deadline={deadline_s * 1000:.0f}ms outer_timeout={TIMEOUT * 1000:.0f}ms " f"units={UNITS} chunks/unit={WORDS // 350} inference={DELAY * 1000:.0f}ms/chunk") print(f"request: {outcome} after {(returned - started) * 1000:.0f}ms") late = [s - expiry for s, _ in snapshot if s > expiry] print(f"inference calls: total={len(snapshot)}, " f"started after the {budget_s * 1000:.0f}ms budget={len(late)}" + (f" (latest +{max(late) * 1000:.1f}ms)" if late else "") + f", after the awaiter returned={len(snapshot) - at_return}") print(f"worker busy until {(worker_done - started) * 1000:.0f}ms " f"({(worker_done - returned) * 1000:.0f}ms after the awaiter returned)") if result is not None: changed = sum(a != b for a, b in zip(source["input"], result[0]["input"])) print(f"units compressed={changed}/{UNITS}, passed through byte-identical={UNITS - changed}, " f"tokens_saved={result[2]}") if os.environ.get("PROBE_TIMINGS"): for name, ms in sorted(result[8].items(), key=lambda kv: -kv[1])[:8]: print(f" timing {name}={ms:.1f}ms") print(f"timeout debt: leaked_threads={proxy._compression_leaked_threads}, " f"quarantine_activations={proxy._compression_quarantine_activations}") # An unrelated request that arrives while the first one's worker is running. proxy2 = make_proxy() await warm_up(proxy2) first = asyncio.ensure_future( proxy2._compress_openai_responses_payload_in_executor( payload("b"), model="gpt-5", request_id="probe-2", timeout=TIMEOUT ) ) try: await first first_outcome = "completed" except asyncio.TimeoutError: first_outcome = "timed out" small = {"model": "gpt-5", "input": payload("c")["input"][:1]} try: await proxy2._compress_openai_responses_payload_in_executor( small, model="gpt-5", request_id="probe-3", timeout=TIMEOUT ) follow = "ran, not quarantined" except asyncio.TimeoutError as exc: follow = f"{type(exc).__name__}: {exc}" print(f"next request right after a slow one ({first_outcome}): {follow}") while proxy2._compression_in_flight: await asyncio.sleep(0.01) for p in (proxy, proxy2): p._compression_executor.shutdown(wait=True) if __name__ == "__main__": print(f"python {sys.version.split()[0]}, headroom from {os.path.dirname(kc.__file__)}") asyncio.run(main()) ``` </details> ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes, on the OpenAI Responses compression path only. A request whose Kompress work exceeds `min(HEADROOM_COMPRESSION_DEADLINE_MS, 0.75 * timeout)` in total now stops starting new inference and forwards the units it had not started unchanged. Before, each unit restarted the deadline, so such a request either finished inside the timeout with more units compressed (fan-outs that took between the budget and the timeout), or timed out, was forwarded with no compression at all, and quarantined compression for every request behind it. Codex WS requests now budget 3.75s of the 5s they are awaited for, instead of 20s per Kompress call. Requests that finish inside the budget are unchanged, byte for byte. The chat and Anthropic paths (`apply()`) are not touched. - Kill switch / disable path: `HEADROOM_COMPRESSION_DEADLINE_MS=0` disables the deadline (the same switch #3693 has), and a higher value or `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` widens the budget. No new environment variable. - Unsafe override required: No. - Qualification impact: Responses compression only, and only on requests whose fan-out outruns the budget. Those that used to finish between the budget and the timeout now compress fewer units (the rest pass through unchanged); those that used to time out and go out with no compression now keep the units finished in time, and no longer quarantine the requests behind them. Requests inside the budget are byte-identical. No provider cache or replay change. - Rollback path: Revert this commit; Responses units go back to a fresh deadline per Kompress call. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Documentation: no docs page describes `HEADROOM_COMPRESSION_DEADLINE_MS` per path, so nothing to update. - Why 0.75 and not #3721's 0.5: #3721 (open) caps cumulative ML time inside `apply()` at half the outer timeout; this caps wall time from enqueue on the Responses path, which already includes the non-ML work. 0.75 leaves the default HTTP budget at the existing 20s deadline and only binds when the timeout is short (Codex WS) or lowered by the operator. The probe above shows the remaining 25% covering the chunk in flight plus splice and serialization. #3721 arms only `apply()` and leaves direct `compress()` callers unbounded, so the two do not overlap; if it merges, the Responses path still gets its bound from this change. - Related: #2360 (Codex WS worker hang -> permanent quarantine; the quarantine time cap from #2412 limits how long one leaked worker can block, this removes the leak on this path), #3653 (a thread/fd per quarantine event; fewer quarantine events here), #3711 (the chat-path counterpart). - Headroom Desktop has shipped an equivalent downstream patch since 2026-09-07. This PR re-implements it on the existing #3693 state instead of a separate budget. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3938" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
Tejas Chopra
parent
befdb52fd5
commit
4d27d024b0
@@ -7,6 +7,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import contextvars
|
||||
import copy
|
||||
import hashlib
|
||||
import hmac
|
||||
@@ -664,6 +665,31 @@ def _openai_responses_unit_parallelism() -> int:
|
||||
return max(1, min(_OPENAI_RESPONSES_UNIT_PARALLELISM_MAX, requested))
|
||||
|
||||
|
||||
def _openai_responses_deadline_started_at(timeout: float) -> float | None:
|
||||
"""Deadline origin shared by every compress call of one Responses request.
|
||||
|
||||
A Responses request fans out over many ``ContentRouter.compress`` calls,
|
||||
and Kompress starts a fresh ``HEADROOM_COMPRESSION_DEADLINE_MS`` clock in
|
||||
each one unless it is handed an origin. Without one, a request past its
|
||||
``timeout`` keeps running inference long after the awaiter gave up, and
|
||||
that timeout debt quarantines compression for every request behind it.
|
||||
Stamped before the job is queued, because ``timeout`` counts queue wait
|
||||
too. ``None`` when the deadline is disabled (``0``).
|
||||
|
||||
The budget is also capped at 75% of ``timeout``: the deadline only stops
|
||||
the NEXT Kompress chunk, so the chunk already running when it expires, the
|
||||
splice and the output serialization need the rest. Kompress checks
|
||||
``now - origin > deadline``, so a shorter budget is a backdated origin.
|
||||
"""
|
||||
from headroom.transforms.content_router import _compression_deadline_seconds
|
||||
|
||||
deadline_s = _compression_deadline_seconds()
|
||||
if deadline_s <= 0:
|
||||
return None
|
||||
budget_s = min(deadline_s, 0.75 * timeout)
|
||||
return time.perf_counter() - (deadline_s - budget_s)
|
||||
|
||||
|
||||
def _openai_responses_unit_executor() -> ThreadPoolExecutor:
|
||||
global _OPENAI_RESPONSES_UNIT_EXECUTOR
|
||||
with _OPENAI_RESPONSES_UNIT_EXECUTOR_LOCK:
|
||||
@@ -2130,6 +2156,7 @@ class OpenAIHandlerMixin:
|
||||
request_id: str,
|
||||
pass_id: str | None = None,
|
||||
timing: dict[str, float] | None = None,
|
||||
deadline_started_at: float | None = None,
|
||||
) -> tuple[dict[str, Any], bool, int, list[str], dict[str, int], list[str], int]:
|
||||
"""Run ContentRouter on OpenAI Responses text units.
|
||||
|
||||
@@ -2723,6 +2750,7 @@ class OpenAIHandlerMixin:
|
||||
router=router,
|
||||
tokenizer=tokenizer,
|
||||
target_ratio=unit_target_ratio,
|
||||
deadline_started_at=deadline_started_at,
|
||||
)
|
||||
elapsed_ms = (time.perf_counter() - unit_started) * 1000.0
|
||||
return routed.slot, result, elapsed_ms
|
||||
@@ -2792,12 +2820,18 @@ class OpenAIHandlerMixin:
|
||||
0.0,
|
||||
)
|
||||
|
||||
# Each pool task runs in its own copy of this Context: the router binds
|
||||
# the request's deadline per task (see share_request_deadline), and
|
||||
# pool threads would otherwise keep that binding between requests.
|
||||
parallelism = _openai_responses_unit_parallelism()
|
||||
if len(cache_misses) > 1 and parallelism > 1:
|
||||
executor = _openai_responses_unit_executor()
|
||||
for start in range(0, len(cache_misses), parallelism):
|
||||
batch = cache_misses[start : start + parallelism]
|
||||
futures = [executor.submit(_compress_and_store, *item) for item in batch]
|
||||
futures = [
|
||||
executor.submit(contextvars.copy_context().run, _compress_and_store, *item)
|
||||
for item in batch
|
||||
]
|
||||
for future in as_completed(futures):
|
||||
unit_idx, cache_key, routed_result = future.result()
|
||||
_record_routed_result(unit_idx, cache_key, routed_result)
|
||||
@@ -2822,6 +2856,7 @@ class OpenAIHandlerMixin:
|
||||
router=router,
|
||||
tokenizer=tokenizer,
|
||||
target_ratio=unit_target_ratio,
|
||||
deadline_started_at=deadline_started_at,
|
||||
)
|
||||
return results, (time.perf_counter() - batch_started) * 1000.0
|
||||
|
||||
@@ -2835,7 +2870,10 @@ class OpenAIHandlerMixin:
|
||||
executor = _openai_responses_unit_executor()
|
||||
for start in range(0, len(small_batches), parallelism):
|
||||
batch_group = small_batches[start : start + parallelism]
|
||||
futures = [executor.submit(_compress_batch, batch) for batch in batch_group]
|
||||
futures = [
|
||||
executor.submit(contextvars.copy_context().run, _compress_batch, batch)
|
||||
for batch in batch_group
|
||||
]
|
||||
for future in as_completed(futures):
|
||||
_record_batch_result(future.result())
|
||||
else:
|
||||
@@ -3023,6 +3061,7 @@ class OpenAIHandlerMixin:
|
||||
timing: dict[str, float] | None = None,
|
||||
client: str | None = None,
|
||||
savings_tags: dict[str, Any] | None = None,
|
||||
deadline_started_at: float | None = None,
|
||||
) -> tuple[dict[str, Any], bool, int, list[str], str | None, int, int, int]:
|
||||
"""Compress an OpenAI Responses payload through the shared router.
|
||||
|
||||
@@ -3290,6 +3329,7 @@ class OpenAIHandlerMixin:
|
||||
request_id=request_id,
|
||||
pass_id=pass_id,
|
||||
timing=timing_sink,
|
||||
deadline_started_at=deadline_started_at,
|
||||
)
|
||||
_add_timing("compression_live_units_total", live_units_started)
|
||||
if router_modified:
|
||||
@@ -3437,6 +3477,8 @@ class OpenAIHandlerMixin:
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
deadline_started_at = _openai_responses_deadline_started_at(timeout)
|
||||
|
||||
def _compress(): # noqa: ANN202
|
||||
# Output shaping (opt-in via HEADROOM_OUTPUT_SHAPER) runs before
|
||||
# compression so the turn classifier sees the client's input as
|
||||
@@ -3456,6 +3498,7 @@ class OpenAIHandlerMixin:
|
||||
"request_id": request_id,
|
||||
"timing": timing,
|
||||
"client": client,
|
||||
"deadline_started_at": deadline_started_at,
|
||||
}
|
||||
if savings_tags is not None:
|
||||
compression_kwargs["savings_tags"] = savings_tags
|
||||
@@ -3470,7 +3513,12 @@ class OpenAIHandlerMixin:
|
||||
unsupported_kwarg = next(
|
||||
(
|
||||
name
|
||||
for name in ("savings_tags", "client", "timing")
|
||||
for name in (
|
||||
"savings_tags",
|
||||
"client",
|
||||
"timing",
|
||||
"deadline_started_at",
|
||||
)
|
||||
if f"unexpected keyword argument '{name}'" in str(exc)
|
||||
and name in compression_kwargs
|
||||
),
|
||||
|
||||
@@ -230,8 +230,15 @@ def compress_batch_with_router(
|
||||
router: ContentRouter,
|
||||
tokenizer: TokenCounterLike,
|
||||
target_ratio: float | None = None,
|
||||
deadline_started_at: float | None = None,
|
||||
) -> list[tuple[object, UnitCompressionResult]]:
|
||||
"""Compress one tagged batch and split only structurally valid output."""
|
||||
"""Compress one tagged batch and split only structurally valid output.
|
||||
|
||||
``deadline_started_at`` works as in ``compress_unit_with_router``.
|
||||
"""
|
||||
|
||||
if deadline_started_at is not None and not router.share_request_deadline(deadline_started_at):
|
||||
return _passthrough_batch_results(batch, tokenizer=tokenizer, reason="deadline_exceeded")
|
||||
|
||||
nonce = _batch_nonce(batch)
|
||||
batch_texts, marker_blocks = _protect_ccr_markers(batch, nonce)
|
||||
|
||||
@@ -56,6 +56,8 @@ class CompressionUnit:
|
||||
# Responses handler's unit-result cache) and is surfaced
|
||||
# via RouterCompressionResult.cache_hit, not as a reason
|
||||
# here — no code path produces this category today
|
||||
# - deadline: the request's compression deadline had passed before
|
||||
# this unit started, so it was left unchanged
|
||||
UNIT_REASON_CATEGORIES = {
|
||||
None: "applied",
|
||||
"protected_user_message": "protected_role",
|
||||
@@ -66,6 +68,7 @@ UNIT_REASON_CATEGORIES = {
|
||||
"router_no_change": "compressor_noop",
|
||||
"already_compressed": "already_compressed",
|
||||
"rejected_not_smaller": "rejected_not_smaller",
|
||||
"deadline_exceeded": "deadline",
|
||||
}
|
||||
|
||||
|
||||
@@ -221,11 +224,17 @@ def compress_unit_with_router(
|
||||
router: ContentRouter,
|
||||
tokenizer: TokenCounterLike,
|
||||
target_ratio: float | None = None,
|
||||
deadline_started_at: float | None = None,
|
||||
) -> UnitCompressionResult:
|
||||
"""Compress one safe text unit through ContentRouter.
|
||||
|
||||
The final accept/reject gate uses the provider/model tokenizer, not the
|
||||
router's internal word-count estimates.
|
||||
|
||||
``deadline_started_at`` is the deadline origin of the request this unit
|
||||
belongs to (see ``ContentRouter.share_request_deadline``). Once that
|
||||
deadline has passed the unit is returned unchanged without entering the
|
||||
router.
|
||||
"""
|
||||
|
||||
tokens_before = tokenizer.count_text(unit.text)
|
||||
@@ -266,6 +275,8 @@ def compress_unit_with_router(
|
||||
return _with_reason(reason=f"cache_zone_{unit.cache_zone}")
|
||||
if text_bytes < unit.min_bytes:
|
||||
return _with_reason(reason="below_unit_floor")
|
||||
if deadline_started_at is not None and not router.share_request_deadline(deadline_started_at):
|
||||
return _with_reason(reason="deadline_exceeded")
|
||||
|
||||
prior_target_ratio = getattr(router, "_runtime_target_ratio", None)
|
||||
if target_ratio is not None:
|
||||
|
||||
@@ -2394,6 +2394,27 @@ class ContentRouter(Transform):
|
||||
self._runtime_state_var.set(state)
|
||||
return state
|
||||
|
||||
def share_request_deadline(self, started_at: float) -> bool:
|
||||
"""Join a request whose kompress deadline started at ``started_at``.
|
||||
|
||||
``apply()`` stamps its own origin. A caller that fans ONE request out
|
||||
over many ``compress()`` calls (the OpenAI Responses unit adapter) calls
|
||||
this before each one instead, so every call draws down the same
|
||||
``HEADROOM_COMPRESSION_DEADLINE_MS`` budget rather than restarting it.
|
||||
Binds a fresh ``_PerRequestRuntimeState`` in the CURRENT Context, so a
|
||||
worker-pool task must run in its own ``contextvars.copy_context()``.
|
||||
|
||||
Returns ``False``, binding nothing, once the deadline has passed: the
|
||||
caller should leave the content unchanged rather than start new work.
|
||||
"""
|
||||
deadline_s = _compression_deadline_seconds()
|
||||
if deadline_s and time.perf_counter() - started_at > deadline_s:
|
||||
return False
|
||||
self._runtime_state_var.set(
|
||||
_PerRequestRuntimeState(kompress_deadline_started_at=started_at)
|
||||
)
|
||||
return True
|
||||
|
||||
@property
|
||||
def _runtime_compression_policy(self) -> Any:
|
||||
return self._runtime_state_var.get().compression_policy
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import time
|
||||
|
||||
from headroom.transforms.compression_batches import (
|
||||
CompressionBatchEntry,
|
||||
@@ -229,3 +230,27 @@ def test_batch_uses_utf8_bytes_for_cjk_small_units():
|
||||
assert batches[0].text_bytes == 1800
|
||||
assert all(result.modified for _, result in results)
|
||||
assert [result.compressed for _, result in results] == ["短"] * 4
|
||||
|
||||
|
||||
def test_batch_after_the_request_deadline_passes_through_without_routing(monkeypatch):
|
||||
from headroom.transforms.content_router import ContentRouter
|
||||
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_DEADLINE_MS", "20000")
|
||||
entries = [_entry(index, "x" * 150) for index in range(4)]
|
||||
batches, _ = build_compression_batches(entries, min_batch_bytes=512)
|
||||
router = ContentRouter()
|
||||
|
||||
def _not_called(*_args, **_kwargs):
|
||||
raise AssertionError("a batch past the deadline must not enter the router")
|
||||
|
||||
router.compress = _not_called
|
||||
|
||||
results = compress_batch_with_router(
|
||||
batches[0],
|
||||
router=router,
|
||||
tokenizer=_CharacterCounter(),
|
||||
deadline_started_at=time.perf_counter() - 21,
|
||||
)
|
||||
|
||||
assert [result.compressed for _, result in results] == ["x" * 150] * 4
|
||||
assert {result.reason for _, result in results} == {"deadline_exceeded"}
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
from types import MethodType, SimpleNamespace
|
||||
|
||||
import pytest
|
||||
@@ -1417,6 +1419,114 @@ def test_openai_responses_adapter_compresses_historical_messages_not_current_use
|
||||
assert any(t.startswith("router:openai:responses:message:") for t in transforms)
|
||||
|
||||
|
||||
class _RecordingKompress:
|
||||
"""Stands in for KompressCompressor, recording the deadline origin it gets."""
|
||||
|
||||
shares_request_deadline = True
|
||||
|
||||
def __init__(self, delay: float = 0.0) -> None:
|
||||
self.delay = delay
|
||||
self.deadlines: list[float | None] = []
|
||||
|
||||
def is_ready(self) -> bool:
|
||||
return True
|
||||
|
||||
def compress(self, content: str, **kwargs):
|
||||
self.deadlines.append(kwargs.get("_deadline_started_at"))
|
||||
time.sleep(self.delay)
|
||||
compressed = " ".join(content.split()[:20])
|
||||
return SimpleNamespace(compressed=compressed, compressed_tokens=20)
|
||||
|
||||
|
||||
def _kompress_payload(units: int) -> dict:
|
||||
return {
|
||||
"model": "gpt-5",
|
||||
"input": [
|
||||
{
|
||||
"type": "function_call_output",
|
||||
"call_id": f"call_{i}",
|
||||
"output": "\n".join(
|
||||
" ".join(f"unit{i}line{line}word{j}" for j in range(10)) for line in range(70)
|
||||
),
|
||||
}
|
||||
for i in range(units)
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _compress_through_executor(router: ContentRouter, payload: dict) -> tuple:
|
||||
handler = _handler_with_router(router)
|
||||
|
||||
async def _run_compression(fn, *, timeout): # noqa: ANN001, ANN202
|
||||
return fn()
|
||||
|
||||
handler._run_compression_in_executor = _run_compression
|
||||
return asyncio.run(
|
||||
handler._compress_openai_responses_payload_in_executor(
|
||||
payload, model="gpt-5", request_id="req_deadline"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_openai_responses_units_share_one_kompress_deadline(monkeypatch):
|
||||
"""The fan-out must draw down ONE request deadline (#3693), pool threads too.
|
||||
|
||||
Each unit is its own ContentRouter.compress call, and Kompress started a
|
||||
fresh HEADROOM_COMPRESSION_DEADLINE_MS clock in every one, so a request
|
||||
that blew its timeout kept running inference for minutes afterwards.
|
||||
"""
|
||||
monkeypatch.setenv("HEADROOM_TOOL_OUTPUT_COMPRESSION_PARALLELISM", "4")
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_DEADLINE_MS", "20000")
|
||||
router = ContentRouter()
|
||||
kompress = _RecordingKompress()
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: kompress)
|
||||
|
||||
_compress_through_executor(router, _kompress_payload(6))
|
||||
|
||||
assert len(kompress.deadlines) >= 6, kompress.deadlines
|
||||
assert None not in kompress.deadlines, kompress.deadlines
|
||||
assert len(set(kompress.deadlines)) == 1, kompress.deadlines
|
||||
|
||||
|
||||
def test_openai_responses_units_after_the_deadline_pass_through_unchanged(monkeypatch):
|
||||
"""Finished units are kept; units not started by the deadline are not started."""
|
||||
monkeypatch.setenv("HEADROOM_TOOL_OUTPUT_COMPRESSION_PARALLELISM", "1")
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_DEADLINE_MS", "200")
|
||||
router = ContentRouter()
|
||||
kompress = _RecordingKompress(delay=0.3)
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: kompress)
|
||||
payload = _kompress_payload(4)
|
||||
original = json.loads(json.dumps(payload))
|
||||
|
||||
result = _compress_through_executor(router, payload)
|
||||
|
||||
assert len(kompress.deadlines) == 1
|
||||
outputs = [item["output"] for item in result[0]["input"]]
|
||||
assert outputs[0] != original["input"][0]["output"]
|
||||
assert outputs[1:] == [item["output"] for item in original["input"][1:]]
|
||||
assert result[2] > 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("deadline_ms", "timeout", "budget"),
|
||||
[("20000", 30.0, 20.0), ("20000", 5.0, 3.75), ("0", 30.0, None)],
|
||||
ids=["deadline-binds", "timeout-binds", "disabled"],
|
||||
)
|
||||
def test_openai_responses_deadline_budget_stays_inside_the_timeout(
|
||||
monkeypatch, deadline_ms, timeout, budget
|
||||
):
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_DEADLINE_MS", deadline_ms)
|
||||
before = time.perf_counter()
|
||||
origin = openai_handler._openai_responses_deadline_started_at(timeout)
|
||||
after = time.perf_counter()
|
||||
|
||||
if budget is None:
|
||||
assert origin is None
|
||||
return
|
||||
expires_at = origin + int(deadline_ms) / 1000
|
||||
assert before + budget <= expires_at <= after + budget
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("compress_user_messages", "expected_user_text"),
|
||||
[(False, None), (None, "compressed history")],
|
||||
|
||||
@@ -132,7 +132,10 @@ def test_native_responses_compressor_scopes_the_exclusion_per_call() -> None:
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
assert [{key: value for key, value in call.items() if key != "timing"} for call in seen] == [
|
||||
assert [
|
||||
{key: value for key, value in call.items() if key not in {"timing", "deadline_started_at"}}
|
||||
for call in seen
|
||||
] == [
|
||||
{"client": "opencode"},
|
||||
{"client": None},
|
||||
]
|
||||
@@ -445,4 +448,6 @@ def test_websocket_route_carries_the_client_decision(
|
||||
asyncio.run(handler.handle_openai_responses_ws(client_ws))
|
||||
|
||||
assert seen, "the Responses compressor was never reached on the WS path"
|
||||
assert {k: v for k, v in seen[0].items() if k != "timing"} == expected
|
||||
assert {
|
||||
k: v for k, v in seen[0].items() if k not in {"timing", "deadline_started_at"}
|
||||
} == expected
|
||||
|
||||
Reference in New Issue
Block a user