mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(openai): recompress cached Responses units whose CCR entries expired (#4146)
## Description The Responses unit-result cache has no TTL, but the CCR entries behind the markers in its results expire after the CCR TTL and are written only when a unit is compressed. After expiry, a cache hit reuses a result whose marker no retrieval can redeem (`Content not found`). The result is silent lossy compression on long-running proxies. `_get_openai_responses_cached_unit` now checks, on a hit, that every CCR hash in the cached `compressed` text still `exists()` in the compression store. Hashes come from the existing `_CCR_HASH_RE`, which matches `<<ccr:...>>`, `Retrieve more: hash=` and `Retrieve original: hash=`. If one is missing, the entry is evicted and the lookup returns `None`. The unit then goes through `_compress_and_store`, which writes the CCR entry again. Results without markers and an unreachable store keep the old behaviour. Fixes #4145. ## Type of Change - [x] Bug fix ## Changes Made - `headroom/proxy/handlers/openai.py`: `_openai_responses_cached_ccr_markers_live()` added; `_get_openai_responses_cached_unit()` evicts and misses when it returns False. - `tests/test_openai_responses_compression_units.py`: two tests. - `..._recompresses_when_ccr_entry_expired`: the entry is deleted after the first request; the second request compresses again (2 router calls), produces the same output and restores the entry. - `..._reuses_result_while_ccr_entry_live`: a live entry still gives a cache hit (1 router call). - Review follow-up: inherited markers (already present in the original) are excluded from the liveness check by count, so an extra generated occurrence of the same hash is still checked. The store-failure fallback logs a fixed message without `exc_info`; `test_cached_ccr_store_failure_does_not_log_sensitive_details` guards it with an exception-message canary. ## Testing - On head `a740e380`: `pytest tests/test_openai_responses_compression_units.py tests/test_compression_units.py tests/test_openai_codex_routing.py`: 106 passed. `ruff check` and `ruff format --check` clean on both changed files. First round, on `976aa714`: - `pytest tests/test_openai_responses_compression_units.py`: 46 passed (44 before plus the 2 new). - `pytest tests/test_openai_codex_routing.py tests/test_openai_responses_compression_units.py`: 87 passed. - Red check: with the handler change reverted, `..._recompresses_when_ccr_entry_expired` fails (`assert 1 == 2`); the live-entry test passes either way. - `ruff check` and `ruff format --check` clean on both files. - Built with `maturin` from `main` at `976aa714` (Python 3.14, macOS arm64). ## Real Behavior Proof Run on the PR head `a740e380` (upstream `main` at `2800932e` merged, plus the logging fix). Baseline is `main` at `976aa714`, which does not have this change. - Setup: macOS (Darwin 25.6.0), Python 3.13.15. The real Headroom proxy (`headroom proxy --mode token`), compression pipeline, CCR store and `GET /v1/retrieve/{hash}` route, with `HEADROOM_CCR_TTL_SECONDS=2`. `--openai-api-url` points at a loopback server that returns a fixed Responses reply for model `gpt-5`, so no provider credentials are involved. CCR storage, expiry and retrieval all happen inside the proxy and do not depend on the upstream reply. Each variant gets its own workspace, CCR database and logs; non-loopback connections are blocked. - Exact steps, run against each variant in turn: 1. Start the proxy on a free loopback port. 2. `POST /v1/responses` with a 61,089-byte tool output. Read the CCR hash from the compressed body forwarded upstream. 3. `GET /v1/retrieve/{hash}`. 4. Sleep 2.2 s (past the TTL), then `GET /v1/retrieve/{hash}`. 5. Send the identical `/v1/responses` request again (unit-cache hit). 6. `GET /v1/retrieve/{hash}`. 7. Patched only: send the same tool output with an appended `<<ccr:deadbeefdead>>` marker that has no store entry (an inherited marker), then repeat that request twice. - Observed result: | Variant | Step 3 | Step 4 | Step 6 | | --- | --- | --- | --- | | Baseline `976aa714` | 200 | 404 | 404, entry not restored | | Patched `a740e380` | 200 | 404 | 200, entry restored, all 61,089 bytes equal the original | Both variants emit the same hash, `74aed4a863705704370accdd`. Step 7: the inherited marker is kept in the output, no store entry is invented for it, and both repeats return the identical output from the unit cache without rewriting the generated entry (its `created_at` is unchanged). So a dead inherited marker does not cause a recompression loop. Proxy log, retrieval lines (request ids and timings removed): ```text baseline GET /v1/retrieve/74aed4a863705704370accdd status=200 response_content_length=61855 GET /v1/retrieve/74aed4a863705704370accdd status=404 response_content_length=63 GET /v1/retrieve/74aed4a863705704370accdd status=404 response_content_length=49 patched GET /v1/retrieve/74aed4a863705704370accdd status=200 response_content_length=61855 GET /v1/retrieve/74aed4a863705704370accdd status=404 response_content_length=63 GET /v1/retrieve/74aed4a863705704370accdd status=200 response_content_length=61855 ``` - Logging change: the runtime run above does not reach the store-failure fallback. That path is covered by `test_cached_ccr_store_failure_does_not_log_sensitive_details`, which failed before the change (the canary appeared in the traceback) and passes after it. - Not tested: a real provider or Codex end to end on this head; the same-hash inherited/generated case at runtime (unit tests only); the full repository test suite. Original reproduction (unpatched, before this PR): headroom-ai 0.39.1 proxy (cache mode), Codex CLI 0.161.0 over the ChatGPT subscription WebSocket route, macOS. The same `cat` of three prose files (38.7 KB) through Codex on the same model, more than 1800 s apart, without restarting the proxy. The second run received the same marker hashes as the first (`0a5f6643447032a6ec5a2528`, `9498590711d246862a70f1cf`); `headroom_retrieve` returned `Content not found` and `ccr_entries` had no row. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: only on unit-cache hits whose CCR entry has expired. Those now recompress instead of shipping an unredeemable marker. - Kill switch / disable path: none added; revert the commit. - Unsafe override required: no. - Qualification impact: none expected; recompression cost on expired hits only. - Rollback path: revert. ## Notes for review - The check runs outside the cache lock. Eviction re-checks identity (`cache.get(key) is result`), so a concurrent re-store is not dropped. - Only the Responses unit cache is touched. If other paths reuse compressed text across requests, the same question applies there; not audited. - Marker hashes are the store keys by design (`store(..., explicit_hash=...)` for producers with their own hash). If any producer emits a marker hash that is not its exact store key, this check would turn its hits into misses. That is a performance cost, not a correctness one. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (no user-facing behaviour or setting changed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4146" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: stefan <stefan@Macbook-Pro-14M2-Stefan.local> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
stefan
JerrettDavis
parent
0fe0a597c1
commit
8f7b419980
@@ -18,7 +18,7 @@ import ssl
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from collections import OrderedDict
|
||||
from collections import Counter, OrderedDict
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from dataclasses import replace
|
||||
from datetime import datetime
|
||||
@@ -743,6 +743,35 @@ def _openai_responses_result_with_cache_hit(result: Any) -> Any:
|
||||
return replace(result, router_result=replace(router_result, cache_hit=True))
|
||||
|
||||
|
||||
def _openai_responses_cached_ccr_markers_live(result: Any) -> bool:
|
||||
"""Return whether CCR markers generated for a cached unit are still retrievable.
|
||||
|
||||
The unit-result cache outlives CCR entries: it has no TTL, while a CCR entry
|
||||
expires after the CCR TTL and is written only when the unit is compressed.
|
||||
Treat an expired generated entry as a miss so compression stores it again.
|
||||
Inherited markers cannot be restored by recompressing the unit. A store that
|
||||
cannot be consulted keeps the previous behaviour (reuse).
|
||||
"""
|
||||
compressed = getattr(result, "compressed", None)
|
||||
if not isinstance(compressed, str):
|
||||
return True
|
||||
hashes = Counter(match.group(1).lower() for match in _CCR_HASH_RE.finditer(compressed))
|
||||
original = getattr(result, "original", "")
|
||||
if isinstance(original, str):
|
||||
# Inherited markers are preserved byte-for-byte; extra occurrences are generated.
|
||||
hashes -= Counter(match.group(1).lower() for match in _CCR_HASH_RE.finditer(original))
|
||||
if not hashes:
|
||||
return True
|
||||
try:
|
||||
from headroom.cache.compression_store import get_compression_store
|
||||
|
||||
store = get_compression_store()
|
||||
return all(store.exists(hash_key) for hash_key in hashes)
|
||||
except Exception:
|
||||
logger.debug("CCR liveness check for a cached Responses unit failed")
|
||||
return True
|
||||
|
||||
|
||||
def _codex_ws_text_shape(text: str) -> str:
|
||||
stripped = text.strip()
|
||||
if not stripped:
|
||||
@@ -1812,6 +1841,11 @@ class OpenAIHandlerMixin:
|
||||
if result is None:
|
||||
return None
|
||||
cache.move_to_end(key)
|
||||
if not _openai_responses_cached_ccr_markers_live(result):
|
||||
with lock:
|
||||
if cache.get(key) is result:
|
||||
del cache[key]
|
||||
return None
|
||||
return _openai_responses_result_with_cache_hit(result)
|
||||
|
||||
def _store_openai_responses_cached_unit(self, key: str, result: Any) -> None:
|
||||
|
||||
@@ -18,6 +18,24 @@ from headroom.transforms.content_router import (
|
||||
)
|
||||
|
||||
|
||||
def test_cached_ccr_store_failure_does_not_log_sensitive_details(monkeypatch, caplog):
|
||||
from headroom.cache import compression_store
|
||||
|
||||
def exists(hash_key):
|
||||
raise RuntimeError("PRIVATE_CREDENTIAL_CANARY")
|
||||
|
||||
monkeypatch.setattr(
|
||||
compression_store, "get_compression_store", lambda: SimpleNamespace(exists=exists)
|
||||
)
|
||||
result = SimpleNamespace(original="tool output", compressed="<<ccr:abcdef123456>>")
|
||||
with caplog.at_level("DEBUG", logger="headroom.proxy"):
|
||||
assert openai_handler._openai_responses_cached_ccr_markers_live(result) is True
|
||||
|
||||
assert "CCR liveness check for a cached Responses unit failed" in caplog.text
|
||||
assert "PRIVATE_CREDENTIAL_CANARY" not in caplog.text
|
||||
assert all(record.exc_info is None for record in caplog.records)
|
||||
|
||||
|
||||
class TokenCounter:
|
||||
def count_text(self, text: str) -> int:
|
||||
return len(text.split())
|
||||
@@ -466,6 +484,165 @@ def test_openai_responses_adapter_reuses_exact_tool_output_cache():
|
||||
assert new_payload_two["input"][1]["output"] == "cached output summary"
|
||||
|
||||
|
||||
def _marker_compressing_handler(store):
|
||||
"""Handler whose router stores each original in ``store`` and emits a kompress marker."""
|
||||
router = ContentRouter()
|
||||
calls = {"count": 0}
|
||||
|
||||
def compress(self, content: str, **_kwargs):
|
||||
calls["count"] += 1
|
||||
hash_key = store.store(content, "summary", compression_strategy="kompress")
|
||||
return RouterCompressionResult(
|
||||
compressed=f"summary [Retrieve more: hash={hash_key}]",
|
||||
original=content,
|
||||
strategy_used=CompressionStrategy.KOMPRESS,
|
||||
)
|
||||
|
||||
router.compress = MethodType(compress, router)
|
||||
return _handler_with_router(router), calls
|
||||
|
||||
|
||||
def _compress_shell_output(handler, text: str, request_id: str):
|
||||
payload = {
|
||||
"model": "gpt-5",
|
||||
"input": [{"type": "local_shell_call_output", "call_id": request_id, "output": text}],
|
||||
}
|
||||
new_payload, *_ = handler._compress_openai_responses_live_text_units_with_router(
|
||||
payload,
|
||||
model="gpt-5",
|
||||
request_id=request_id,
|
||||
)
|
||||
return new_payload["input"][0]["output"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"marker",
|
||||
["<<ccr:deadbeefdead>>", "[Retrieve more: hash=deadbeefdead]"],
|
||||
)
|
||||
def test_openai_responses_unit_cache_reuses_result_with_missing_inherited_marker(marker):
|
||||
from headroom.cache.compression_store import (
|
||||
CompressionStore,
|
||||
clear_request_compression_store,
|
||||
set_request_compression_store,
|
||||
)
|
||||
|
||||
store = CompressionStore(max_entries=100, default_ttl=60)
|
||||
set_request_compression_store(store)
|
||||
try:
|
||||
handler, calls = _marker_compressing_handler(store)
|
||||
text = " ".join(f"word{i}" for i in range(180)) + "\n" + marker
|
||||
outputs = [_compress_shell_output(handler, text, f"req_inherited_{i}") for i in range(3)]
|
||||
|
||||
assert outputs[0] == outputs[1] == outputs[2]
|
||||
assert marker in outputs[0]
|
||||
assert not store.exists("deadbeefdead")
|
||||
assert calls["count"] == 1
|
||||
finally:
|
||||
clear_request_compression_store()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("same_hash", [True, False])
|
||||
@pytest.mark.parametrize("marker_format", ["<<ccr:{hash}>>", "[Retrieve more: hash={hash}]"])
|
||||
def test_openai_responses_unit_cache_renews_generated_hash_with_inherited_marker(
|
||||
same_hash, marker_format
|
||||
):
|
||||
from headroom.cache.compression_store import (
|
||||
CompressionStore,
|
||||
clear_request_compression_store,
|
||||
set_request_compression_store,
|
||||
)
|
||||
|
||||
store = CompressionStore(max_entries=100, default_ttl=60)
|
||||
set_request_compression_store(store)
|
||||
try:
|
||||
handler, calls = _marker_compressing_handler(store)
|
||||
text = " ".join(f"word{i}" for i in range(180))
|
||||
seeded = _compress_shell_output(handler, text, "req_seed")
|
||||
generated_hash = seeded.rsplit("hash=", 1)[1].rstrip("]")
|
||||
inherited_hash = generated_hash if same_hash else store.store("other original", "other")
|
||||
marker = marker_format.format(hash=inherited_hash)
|
||||
# Both marker-free spans generate the same hash; neither may be hidden
|
||||
# by the preserved inherited marker or the last router result.
|
||||
mixed = f"{text}\n{marker}\n{text}"
|
||||
first = _compress_shell_output(handler, mixed, "req_mixed_one")
|
||||
assert first == f"{seeded}\n{marker}\n{seeded}"
|
||||
assert calls["count"] == 3
|
||||
assert _compress_shell_output(handler, mixed, "req_mixed_cached") == first
|
||||
assert calls["count"] == 3
|
||||
|
||||
entry = store.retrieve(generated_hash)
|
||||
assert entry is not None
|
||||
assert entry.original_content == text
|
||||
entry.created_at -= entry.ttl + 1
|
||||
store._backend.set(generated_hash, entry)
|
||||
assert store.retrieve(generated_hash) is None
|
||||
if not same_hash:
|
||||
assert store.exists(inherited_hash)
|
||||
|
||||
second = _compress_shell_output(handler, mixed, "req_mixed_expired")
|
||||
assert calls["count"] == 5
|
||||
assert second == first
|
||||
renewed = store.retrieve(generated_hash)
|
||||
assert renewed is not None
|
||||
assert renewed.original_content == text
|
||||
finally:
|
||||
clear_request_compression_store()
|
||||
|
||||
|
||||
def test_openai_responses_unit_cache_recompresses_when_ccr_entry_expired():
|
||||
# The unit cache has no TTL; the CCR entry behind its marker does. Reusing the
|
||||
# cached result after the entry expired would ship a marker no retrieval can
|
||||
# redeem, so the unit must be compressed (and its entry stored) again.
|
||||
from headroom.cache.compression_store import (
|
||||
CompressionStore,
|
||||
clear_request_compression_store,
|
||||
set_request_compression_store,
|
||||
)
|
||||
|
||||
store = CompressionStore(max_entries=100, default_ttl=60)
|
||||
set_request_compression_store(store)
|
||||
try:
|
||||
handler, calls = _marker_compressing_handler(store)
|
||||
long_text = " ".join(f"word{i}" for i in range(180))
|
||||
|
||||
first = _compress_shell_output(handler, long_text, "req_expire_one")
|
||||
hash_key = first.rsplit("hash=", 1)[1].rstrip("]")
|
||||
assert store.exists(hash_key)
|
||||
|
||||
store._backend.delete(hash_key) # what TTL expiry does to the entry
|
||||
assert not store.exists(hash_key)
|
||||
|
||||
second = _compress_shell_output(handler, long_text, "req_expire_two")
|
||||
|
||||
assert calls["count"] == 2
|
||||
assert second == first
|
||||
assert store.exists(hash_key)
|
||||
finally:
|
||||
clear_request_compression_store()
|
||||
|
||||
|
||||
def test_openai_responses_unit_cache_reuses_result_while_ccr_entry_live():
|
||||
from headroom.cache.compression_store import (
|
||||
CompressionStore,
|
||||
clear_request_compression_store,
|
||||
set_request_compression_store,
|
||||
)
|
||||
|
||||
store = CompressionStore(max_entries=100, default_ttl=60)
|
||||
set_request_compression_store(store)
|
||||
try:
|
||||
handler, calls = _marker_compressing_handler(store)
|
||||
long_text = " ".join(f"word{i}" for i in range(180))
|
||||
|
||||
first = _compress_shell_output(handler, long_text, "req_live_one")
|
||||
second = _compress_shell_output(handler, long_text, "req_live_two")
|
||||
|
||||
assert calls["count"] == 1
|
||||
assert second == first
|
||||
finally:
|
||||
clear_request_compression_store()
|
||||
|
||||
|
||||
def test_openai_responses_adapter_reuses_identical_tool_output_in_same_request():
|
||||
router = ContentRouter()
|
||||
calls = {"count": 0}
|
||||
|
||||
Reference in New Issue
Block a user