fix(openai): recompress cached Responses units whose CCR entries expired (#4146)

## Description

The Responses unit-result cache has no TTL, but the CCR entries behind
the markers in its results expire after the CCR TTL and are written only
when a unit is compressed. After expiry, a cache hit reuses a result
whose marker no retrieval can redeem (`Content not found`). The result
is silent lossy compression on long-running proxies.

`_get_openai_responses_cached_unit` now checks, on a hit, that every CCR
hash in the cached `compressed` text still `exists()` in the compression
store. Hashes come from the existing `_CCR_HASH_RE`, which matches
`<<ccr:...>>`, `Retrieve more: hash=` and `Retrieve original: hash=`. If
one is missing, the entry is evicted and the lookup returns `None`. The
unit then goes through `_compress_and_store`, which writes the CCR entry
again. Results without markers and an unreachable store keep the old
behaviour.

Fixes #4145.

## Type of Change

- [x] Bug fix

## Changes Made

- `headroom/proxy/handlers/openai.py`:
`_openai_responses_cached_ccr_markers_live()` added;
`_get_openai_responses_cached_unit()` evicts and misses when it returns
False.
- `tests/test_openai_responses_compression_units.py`: two tests.
- `..._recompresses_when_ccr_entry_expired`: the entry is deleted after
the first request; the second request compresses again (2 router calls),
produces the same output and restores the entry.
- `..._reuses_result_while_ccr_entry_live`: a live entry still gives a
cache hit (1 router call).
- Review follow-up: inherited markers (already present in the original)
are excluded from the liveness check by count, so an extra generated
occurrence of the same hash is still checked. The store-failure fallback
logs a fixed message without `exc_info`;
`test_cached_ccr_store_failure_does_not_log_sensitive_details` guards it
with an exception-message canary.

## Testing

- On head `a740e380`: `pytest
tests/test_openai_responses_compression_units.py
tests/test_compression_units.py tests/test_openai_codex_routing.py`: 106
passed. `ruff check` and `ruff format --check` clean on both changed
files.

First round, on `976aa714`:

- `pytest tests/test_openai_responses_compression_units.py`: 46 passed
(44 before plus the 2 new).
- `pytest tests/test_openai_codex_routing.py
tests/test_openai_responses_compression_units.py`: 87 passed.
- Red check: with the handler change reverted,
`..._recompresses_when_ccr_entry_expired` fails (`assert 1 == 2`); the
live-entry test passes either way.
- `ruff check` and `ruff format --check` clean on both files.
- Built with `maturin` from `main` at `976aa714` (Python 3.14, macOS
arm64).

## Real Behavior Proof

Run on the PR head `a740e380` (upstream `main` at `2800932e` merged,
plus the logging fix). Baseline is `main` at `976aa714`, which does not
have this change.

- Setup: macOS (Darwin 25.6.0), Python 3.13.15. The real Headroom proxy
(`headroom proxy --mode token`), compression pipeline, CCR store and
`GET /v1/retrieve/{hash}` route, with `HEADROOM_CCR_TTL_SECONDS=2`.
`--openai-api-url` points at a loopback server that returns a fixed
Responses reply for model `gpt-5`, so no provider credentials are
involved. CCR storage, expiry and retrieval all happen inside the proxy
and do not depend on the upstream reply. Each variant gets its own
workspace, CCR database and logs; non-loopback connections are blocked.
- Exact steps, run against each variant in turn:
  1. Start the proxy on a free loopback port.
2. `POST /v1/responses` with a 61,089-byte tool output. Read the CCR
hash from the compressed body forwarded upstream.
  3. `GET /v1/retrieve/{hash}`.
  4. Sleep 2.2 s (past the TTL), then `GET /v1/retrieve/{hash}`.
  5. Send the identical `/v1/responses` request again (unit-cache hit).
  6. `GET /v1/retrieve/{hash}`.
7. Patched only: send the same tool output with an appended
`<<ccr:deadbeefdead>>` marker that has no store entry (an inherited
marker), then repeat that request twice.
- Observed result:

| Variant | Step 3 | Step 4 | Step 6 |
| --- | --- | --- | --- |
| Baseline `976aa714` | 200 | 404 | 404, entry not restored |
| Patched `a740e380` | 200 | 404 | 200, entry restored, all 61,089 bytes
equal the original |

Both variants emit the same hash, `74aed4a863705704370accdd`. Step 7:
the inherited marker is kept in the output, no store entry is invented
for it, and both repeats return the identical output from the unit cache
without rewriting the generated entry (its `created_at` is unchanged).
So a dead inherited marker does not cause a recompression loop.

Proxy log, retrieval lines (request ids and timings removed):

```text
baseline
GET /v1/retrieve/74aed4a863705704370accdd status=200 response_content_length=61855
GET /v1/retrieve/74aed4a863705704370accdd status=404 response_content_length=63
GET /v1/retrieve/74aed4a863705704370accdd status=404 response_content_length=49

patched
GET /v1/retrieve/74aed4a863705704370accdd status=200 response_content_length=61855
GET /v1/retrieve/74aed4a863705704370accdd status=404 response_content_length=63
GET /v1/retrieve/74aed4a863705704370accdd status=200 response_content_length=61855
```

- Logging change: the runtime run above does not reach the store-failure
fallback. That path is covered by
`test_cached_ccr_store_failure_does_not_log_sensitive_details`, which
failed before the change (the canary appeared in the traceback) and
passes after it.
- Not tested: a real provider or Codex end to end on this head; the
same-hash inherited/generated case at runtime (unit tests only); the
full repository test suite.

Original reproduction (unpatched, before this PR): headroom-ai 0.39.1
proxy (cache mode), Codex CLI 0.161.0 over the ChatGPT subscription
WebSocket route, macOS. The same `cat` of three prose files (38.7 KB)
through Codex on the same model, more than 1800 s apart, without
restarting the proxy. The second run received the same marker hashes as
the first (`0a5f6643447032a6ec5a2528`, `9498590711d246862a70f1cf`);
`headroom_retrieve` returned `Content not found` and `ccr_entries` had
no row.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only on unit-cache hits whose CCR
entry has expired. Those now recompress instead of shipping an
unredeemable marker.
- Kill switch / disable path: none added; revert the commit.
- Unsafe override required: no.
- Qualification impact: none expected; recompression cost on expired
hits only.
- Rollback path: revert.

## Notes for review

- The check runs outside the cache lock. Eviction re-checks identity
(`cache.get(key) is result`), so a concurrent re-store is not dropped.
- Only the Responses unit cache is touched. If other paths reuse
compressed text across requests, the same question applies there; not
audited.
- Marker hashes are the store keys by design (`store(...,
explicit_hash=...)` for producers with their own hash). If any producer
emits a marker hash that is not its exact store key, this check would
turn its hits into misses. That is a performance cost, not a correctness
one.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (no
user-facing behaviour or setting changed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4146"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: stefan <stefan@Macbook-Pro-14M2-Stefan.local>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Stefan Nesiu Bedreag
2026-10-24 08:38:38 -07:00
committed by GitHub
co-authored by stefan JerrettDavis
parent 0fe0a597c1
commit 8f7b419980
2 changed files with 212 additions and 1 deletions
+35 -1
View File
@@ -18,7 +18,7 @@ import ssl
import threading
import time
import uuid
from collections import OrderedDict
from collections import Counter, OrderedDict
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import replace
from datetime import datetime
@@ -743,6 +743,35 @@ def _openai_responses_result_with_cache_hit(result: Any) -> Any:
return replace(result, router_result=replace(router_result, cache_hit=True))
def _openai_responses_cached_ccr_markers_live(result: Any) -> bool:
"""Return whether CCR markers generated for a cached unit are still retrievable.
The unit-result cache outlives CCR entries: it has no TTL, while a CCR entry
expires after the CCR TTL and is written only when the unit is compressed.
Treat an expired generated entry as a miss so compression stores it again.
Inherited markers cannot be restored by recompressing the unit. A store that
cannot be consulted keeps the previous behaviour (reuse).
"""
compressed = getattr(result, "compressed", None)
if not isinstance(compressed, str):
return True
hashes = Counter(match.group(1).lower() for match in _CCR_HASH_RE.finditer(compressed))
original = getattr(result, "original", "")
if isinstance(original, str):
# Inherited markers are preserved byte-for-byte; extra occurrences are generated.
hashes -= Counter(match.group(1).lower() for match in _CCR_HASH_RE.finditer(original))
if not hashes:
return True
try:
from headroom.cache.compression_store import get_compression_store
store = get_compression_store()
return all(store.exists(hash_key) for hash_key in hashes)
except Exception:
logger.debug("CCR liveness check for a cached Responses unit failed")
return True
def _codex_ws_text_shape(text: str) -> str:
stripped = text.strip()
if not stripped:
@@ -1812,6 +1841,11 @@ class OpenAIHandlerMixin:
if result is None:
return None
cache.move_to_end(key)
if not _openai_responses_cached_ccr_markers_live(result):
with lock:
if cache.get(key) is result:
del cache[key]
return None
return _openai_responses_result_with_cache_hit(result)
def _store_openai_responses_cached_unit(self, key: str, result: Any) -> None:
@@ -18,6 +18,24 @@ from headroom.transforms.content_router import (
)
def test_cached_ccr_store_failure_does_not_log_sensitive_details(monkeypatch, caplog):
from headroom.cache import compression_store
def exists(hash_key):
raise RuntimeError("PRIVATE_CREDENTIAL_CANARY")
monkeypatch.setattr(
compression_store, "get_compression_store", lambda: SimpleNamespace(exists=exists)
)
result = SimpleNamespace(original="tool output", compressed="<<ccr:abcdef123456>>")
with caplog.at_level("DEBUG", logger="headroom.proxy"):
assert openai_handler._openai_responses_cached_ccr_markers_live(result) is True
assert "CCR liveness check for a cached Responses unit failed" in caplog.text
assert "PRIVATE_CREDENTIAL_CANARY" not in caplog.text
assert all(record.exc_info is None for record in caplog.records)
class TokenCounter:
def count_text(self, text: str) -> int:
return len(text.split())
@@ -466,6 +484,165 @@ def test_openai_responses_adapter_reuses_exact_tool_output_cache():
assert new_payload_two["input"][1]["output"] == "cached output summary"
def _marker_compressing_handler(store):
"""Handler whose router stores each original in ``store`` and emits a kompress marker."""
router = ContentRouter()
calls = {"count": 0}
def compress(self, content: str, **_kwargs):
calls["count"] += 1
hash_key = store.store(content, "summary", compression_strategy="kompress")
return RouterCompressionResult(
compressed=f"summary [Retrieve more: hash={hash_key}]",
original=content,
strategy_used=CompressionStrategy.KOMPRESS,
)
router.compress = MethodType(compress, router)
return _handler_with_router(router), calls
def _compress_shell_output(handler, text: str, request_id: str):
payload = {
"model": "gpt-5",
"input": [{"type": "local_shell_call_output", "call_id": request_id, "output": text}],
}
new_payload, *_ = handler._compress_openai_responses_live_text_units_with_router(
payload,
model="gpt-5",
request_id=request_id,
)
return new_payload["input"][0]["output"]
@pytest.mark.parametrize(
"marker",
["<<ccr:deadbeefdead>>", "[Retrieve more: hash=deadbeefdead]"],
)
def test_openai_responses_unit_cache_reuses_result_with_missing_inherited_marker(marker):
from headroom.cache.compression_store import (
CompressionStore,
clear_request_compression_store,
set_request_compression_store,
)
store = CompressionStore(max_entries=100, default_ttl=60)
set_request_compression_store(store)
try:
handler, calls = _marker_compressing_handler(store)
text = " ".join(f"word{i}" for i in range(180)) + "\n" + marker
outputs = [_compress_shell_output(handler, text, f"req_inherited_{i}") for i in range(3)]
assert outputs[0] == outputs[1] == outputs[2]
assert marker in outputs[0]
assert not store.exists("deadbeefdead")
assert calls["count"] == 1
finally:
clear_request_compression_store()
@pytest.mark.parametrize("same_hash", [True, False])
@pytest.mark.parametrize("marker_format", ["<<ccr:{hash}>>", "[Retrieve more: hash={hash}]"])
def test_openai_responses_unit_cache_renews_generated_hash_with_inherited_marker(
same_hash, marker_format
):
from headroom.cache.compression_store import (
CompressionStore,
clear_request_compression_store,
set_request_compression_store,
)
store = CompressionStore(max_entries=100, default_ttl=60)
set_request_compression_store(store)
try:
handler, calls = _marker_compressing_handler(store)
text = " ".join(f"word{i}" for i in range(180))
seeded = _compress_shell_output(handler, text, "req_seed")
generated_hash = seeded.rsplit("hash=", 1)[1].rstrip("]")
inherited_hash = generated_hash if same_hash else store.store("other original", "other")
marker = marker_format.format(hash=inherited_hash)
# Both marker-free spans generate the same hash; neither may be hidden
# by the preserved inherited marker or the last router result.
mixed = f"{text}\n{marker}\n{text}"
first = _compress_shell_output(handler, mixed, "req_mixed_one")
assert first == f"{seeded}\n{marker}\n{seeded}"
assert calls["count"] == 3
assert _compress_shell_output(handler, mixed, "req_mixed_cached") == first
assert calls["count"] == 3
entry = store.retrieve(generated_hash)
assert entry is not None
assert entry.original_content == text
entry.created_at -= entry.ttl + 1
store._backend.set(generated_hash, entry)
assert store.retrieve(generated_hash) is None
if not same_hash:
assert store.exists(inherited_hash)
second = _compress_shell_output(handler, mixed, "req_mixed_expired")
assert calls["count"] == 5
assert second == first
renewed = store.retrieve(generated_hash)
assert renewed is not None
assert renewed.original_content == text
finally:
clear_request_compression_store()
def test_openai_responses_unit_cache_recompresses_when_ccr_entry_expired():
# The unit cache has no TTL; the CCR entry behind its marker does. Reusing the
# cached result after the entry expired would ship a marker no retrieval can
# redeem, so the unit must be compressed (and its entry stored) again.
from headroom.cache.compression_store import (
CompressionStore,
clear_request_compression_store,
set_request_compression_store,
)
store = CompressionStore(max_entries=100, default_ttl=60)
set_request_compression_store(store)
try:
handler, calls = _marker_compressing_handler(store)
long_text = " ".join(f"word{i}" for i in range(180))
first = _compress_shell_output(handler, long_text, "req_expire_one")
hash_key = first.rsplit("hash=", 1)[1].rstrip("]")
assert store.exists(hash_key)
store._backend.delete(hash_key) # what TTL expiry does to the entry
assert not store.exists(hash_key)
second = _compress_shell_output(handler, long_text, "req_expire_two")
assert calls["count"] == 2
assert second == first
assert store.exists(hash_key)
finally:
clear_request_compression_store()
def test_openai_responses_unit_cache_reuses_result_while_ccr_entry_live():
from headroom.cache.compression_store import (
CompressionStore,
clear_request_compression_store,
set_request_compression_store,
)
store = CompressionStore(max_entries=100, default_ttl=60)
set_request_compression_store(store)
try:
handler, calls = _marker_compressing_handler(store)
long_text = " ".join(f"word{i}" for i in range(180))
first = _compress_shell_output(handler, long_text, "req_live_one")
second = _compress_shell_output(handler, long_text, "req_live_two")
assert calls["count"] == 1
assert second == first
finally:
clear_request_compression_store()
def test_openai_responses_adapter_reuses_identical_tool_output_in_same_request():
router = ContentRouter()
calls = {"count": 0}