Commit Graph
13 Commits
Author SHA1 Message Date
Yuchuan HeandJerrettDavis 973c12d2af fix(cache): replay every parallel tool result (#4818)
## Description

An Anthropic user message can hold several parallel `tool_result`
blocks. The session compression cache recorded and replayed only the
first one, but a hit for that result could freeze the whole message.
Later results then reached the provider in their raw form instead of the
previously compressed form.

Visit every result when recording, replaying and marking stable content.
Freeze a message only if every result is replayable or textless. Reuse
the existing single-result shape guard, so images, tool references,
multiblock boundaries and text-block metadata retain their established
protection. Cache seeding checks the paired tool-call IDs before
associating original and compressed text.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Cache and replay each parallel result independently without mutating
caller input.
- Stop the frozen prefix on a later miss or unsupported text layout.
- Cover a textless first result, later text-block metadata, changed IDs
and partial replay. Update the existing later-result limitation test to
assert actual replay.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
New regressions against original production code: 5 failed, 1 passed
pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_cache_replay_shapes.py tests/test_token_headroom_mode.py tests/test_cache_prefix_safety.py tests/test_proxy_anthropic_cache_stability.py tests/test_prefix_tracker_restart.py:
  189 passed
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: rebuilt, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

Optional-model skips and the existing token-count approximation warning
remain visible in the gate logs. The restart tests also report the
environment's existing Starlette/httpx deprecation warning.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
baseline `c05fbdddc873078dd97cb9ede841e75ef7ed0bef`, patch
`a09ad47dc8ceb04b42858a67eecb83a7ee246485`. Explicit checkout on
PYTHONPATH; no provider credentials. Headroom runs offline; the parallel
proof uses a real SDK to send only to its bound loopback receiver.
- Exact command / steps: Save the following script as `proof.py` and run
`python proof.py` from either checkout with `PYTHONPATH=.`. Identical
before/after scripts ran outside pytest.

```python
"""Production cache/session APIs followed by real Anthropic SDK loopback HTTP."""
import copy
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import threading
from anthropic import Anthropic
from headroom.cache.compression_cache import CompressionCache
from headroom.proxy.session_engine import FREEZE_POLICY_REPLAYABLE, prepare_turn

captured = []
class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        captured.append(json.loads(self.rfile.read(int(self.headers['Content-Length']))))
        data = json.dumps({'id':'local','type':'message','role':'assistant','model':'local-fixture','content':[{'type':'text','text':'received'}],'stop_reason':'end_turn','stop_sequence':None,'usage':{'input_tokens':1,'output_tokens':1}}).encode()
        self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
    def log_message(self,*args): pass

def message(a,b):
    return {'role':'user','content':[{'type':'tool_result','tool_use_id':'t1','content':a},{'type':'tool_result','tool_use_id':'t2','content':b}]}

server = ThreadingHTTPServer(('127.0.0.1',0),Handler)
thread = threading.Thread(target=server.serve_forever,daemon=True);thread.start()
raw = message('first original '*20,[{'type':'text','text':'second original '*20,'cache_control':{'type':'ephemeral'}}])
short = message('first compressed',[{'type':'text','text':'second compressed','cache_control':{'type':'ephemeral'}}])
messages=[{'role':'assistant','content':[{'type':'tool_use','id':'t1','name':'inspect','input':{}},{'type':'tool_use','id':'t2','name':'inspect','input':{}}]},raw,{'role':'user','content':'continue'}]
before = copy.deepcopy(messages)
cache = CompressionCache();cache.update_from_result([raw],[short])
prep = prepare_turn(cache,messages,policy=FREEZE_POLICY_REPLAYABLE)
try:
    with Anthropic(api_key='local-fixture',base_url=f'http://127.0.0.1:{server.server_port}',max_retries=0) as client:
        client.messages.create(model='local-fixture',max_tokens=16,messages=prep.pipeline_input)
    wire = captured[-1]['messages'][1]
    print(json.dumps({'all_results_replayed':wire==short,'second_result':wire['content'][1]['content'],'frozen_count':prep.frozen_message_count,'caller_unchanged':messages==before,'requests':len(captured)},indent=2))
finally:
    server.shutdown();server.server_close();thread.join(5)
```

- Observed result: With the same original/compressed message pair, clean
main freezes the two-message prefix but the real Anthropic SDK sends the
second result as its original text. With the patch, the HTTP receiver
gets both compressed results, including the second text block's
cache_control; frozen_count remains 2 and caller_unchanged remains true.
- Not tested: Live Anthropic service acceptance, model inference,
billing, task accuracy, or the complete Python suite. The receiver
records actual SDK JSON and returns a synthetic response.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Cached replay covers every tool
result in a message; a later cache miss or unsupported shape keeps that
message mutable.
- Kill switch / disable path: Revert this commit to restore the prior
replay behavior; no new configuration is introduced.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Developed and validated with Codex assistance. I reviewed the changes
and runtime evidence.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4818"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:00:39 -06:00
Yuchuan He 292b90530d fix(cache): preserve Anthropic tool-result blocks on replay (#4765)
## Description

A cached string replacement for an Anthropic `tool_result` with list
content rebuilt that list as one text block. A matching text hash could
therefore remove an accompanying image or tool reference; even a single
text block lost its `cache_control` and other metadata.

Apply the existing shape-safety contract to Anthropic replay: edit the
text of a single text block in place on the copied message, and leave
text-bearing multiblock layouts unchanged. Those layouts also stop the
replayable frozen prefix, since a text hash does not prove their shape
can be reconstructed. Textless tool-search/image results retain their
stable pass-through path. The shared guard also protects interceptor
rewrites.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Preserve metadata on single-block Anthropic cached replacements.
- Decline joined-string replay for text-bearing multiblock results.
- Cover images, tool references, multiple text blocks, unknown blocks,
input immutability and frozen-prefix limits.

- Align the existing direct swap test with image-only pass-through and
deep-copy preservation. The review follow-up changes tests only.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
Initial new regressions against original production code: 6 failed
pytest tests/test_token_headroom_mode.py tests/test_compression_cache.py tests/test_session_engine.py tests/test_tool_result_interceptors.py tests/test_cache tests/test_cold_prefix.py:
  554 passed, 13 skipped
Review follow-up on the initial PR head: 1 failed, 31 passed
Original pre-commit and commit-msg hooks (including follow-up): passed
Whole-tree Ruff check / format --check: passed (1932 files)
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: built, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

The three cache skips require optional spaCy/embedding models. Ten
interceptor skips cover POSIX filesystem/permission behavior unavailable
on Windows. Critical-suite skips require optional relevance models. The
existing token-count approximation warning remains visible in the gate
log. Initial broader runs failed because the new worktrees lacked the
native extension; these were rerun successfully after building it.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing development venv;
clean baseline `200c838c7393d20ae56a0c338f832ed27133c398`, patch
`d0cc1d7abfd70cd36aaada3f633ef97f44041c3e`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH=.` for the selected checkout. No paid/provider credentials.
- Exact command / steps: Save the following identical script as
`proof.py` and run `python proof.py` from either checkout. Before and
after runs were outside pytest.

```python
"""Production session preparation through the real Anthropic SDK to local HTTP."""
import copy
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import threading
from anthropic import Anthropic
from headroom.cache.compression_cache import CompressionCache
from headroom.proxy.session_engine import FREEZE_POLICY_REPLAYABLE, prepare_turn

captured = []

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        captured.append(json.loads(self.rfile.read(int(self.headers["Content-Length"]))))
        body = json.dumps({"id": "local", "type": "message", "role": "assistant",
            "model": "local-fixture", "content": [{"type": "text", "text": "received"}],
            "stop_reason": "end_turn", "stop_sequence": None,
            "usage": {"input_tokens": 1, "output_tokens": 1}}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
image = {"type": "image", "source": {"type": "base64", "media_type": "image/png",
    "data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Wl6LzsAAAAASUVORK5CYII="}}
reports = []
try:
    with Anthropic(api_key="local-fixture", base_url=f"http://127.0.0.1:{server.server_port}", max_retries=0) as client:
        for name, blocks in [
            ("mixed-image", [{"type": "text", "text": "records"}, image]),
            ("single-text", [{"type": "text", "text": "records", "cache_control": {"type": "ephemeral"}}]),
            ("textless-image", [image]),
        ]:
            cache = CompressionCache()
            cache.store_compressed(cache.content_hash("records"), "short", tokens_saved=1)
            messages = [
                {"role": "user", "content": "Read screenshot"},
                {"role": "assistant", "content": [{"type": "tool_use", "id": "t1", "name": "screenshot", "input": {}}]},
                {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t1", "content": blocks}]},
                {"role": "user", "content": "Continue"},
            ]
            snapshot = copy.deepcopy(messages)
            prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
            response = client.messages.create(model="local-fixture", max_tokens=32, messages=prep.pipeline_input)
            forwarded = captured[-1]["messages"][2]["content"][0]["content"]
            reports.append({"case": name, "forwarded_blocks": forwarded,
                "frozen_count": prep.frozen_message_count, "input_unchanged": messages == snapshot,
                "http_response": response.content[0].text})
finally:
    server.shutdown()
    server.server_close()
    thread.join(5)
print(json.dumps(reports, indent=2))
```

- Observed result: On clean main, the real session preparation plus
Anthropic SDK sent only one text block for the mixed text/image result,
and dropped cache_control from the single text block. With this patch,
the local HTTP server receives both original mixed blocks, the
single-text replacement retains cache_control, and the caller's input
remains unchanged. The mixed message stops freezing at 2 instead of 3;
the textless-image control remains at 3.
- Not tested: Real Anthropic service/model traffic, billing, task
accuracy, multiple-result per-block cache redesign, and the complete
Python suite. The HTTP server is local and records the real SDK request;
it does not validate provider acceptance or model behavior.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Anthropic cached replay preserves
typed content and metadata. A layout that cannot be reconstructed is
left for the existing mutable pipeline instead of treated as safely
frozen.
- Kill switch / disable path: Use the existing cache-mode/session
configuration to avoid token-mode cached replay, or revert this patch.
- Unsafe override required: No.
- Qualification impact: Related cache tests and original local gates
passed; hosted CI is tracked separately.
- Rollback path: Revert this commit. No dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Implementation and local validation were completed with Codex
assistance. The author reviewed the initial code and validation record;
the review follow-up was checked with Codex.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4765"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
2026-11-11 09:59:52 -06:00
Alicia f1035ac737 fix(cache): preserve OpenAI tool block metadata on cached replay (#4334)
Current status (2026-10-24 20:50 UTC): exact head
`2333a5ccf7700f86f8767d1c5d02fbfb903c966c` is conflict-free and
[approved by
JerrettDavis](https://github.com/headroomlabs-ai/headroom/pull/4334#pullrequestreview-5639834650).
All seven held workflows are now successful:
[CI](https://github.com/headroomlabs-ai/headroom/actions/runs/40128152028)
has 17 successful jobs and five skips;
[Security](https://github.com/headroomlabs-ai/headroom/actions/runs/40128151955)
has 12 successful jobs. Both [Wrap
E2E](https://github.com/headroomlabs-ai/headroom/actions/runs/40128151965/job/120974772335)
and CI's Docker-native job actually execute the new npm hook: two tests
pass, zero fail or skip. Logs check out synthetic merge `91669e2` (this
head into `3fef15d3`). The maintainer independently reports 146 tests
passed with ten optional skips, including the nine new regressions, plus
Ruff lint/formatting. These hosted and maintainer results supplement,
rather than replace or relabel, the author's limited local checks.
Security passes upstream policy, including three excepted bundled npm
advisories until 2026-11-15; this is not a zero-findings claim.
Live-provider billing/cache behavior remains unqualified. The prior
approval-pending statement and local Docker limitation in the [author
response](https://github.com/headroomlabs-ai/headroom/pull/4334#issuecomment-6388924792)
describe the earlier observation, now superseded by these hosted
results.

### Previous-head qualification (historical)

Current status (2026-10-24 19:54 UTC): exact head
`a618561542a8bb48fdb29b07c66c3c6647b7a690` is conflict-free and
[approved by
JerrettDavis](https://github.com/headroomlabs-ai/headroom/pull/4334#pullrequestreview-5639593849);
the maintainer explicitly cleared the earlier dependency qualification
hold. All seven previously held workflows completed successfully,
including
[CI](https://github.com/headroomlabs-ai/headroom/actions/runs/40116172110)
(17 successful jobs, five skipped),
[Security](https://github.com/headroomlabs-ai/headroom/actions/runs/40116172235)
(12 successful jobs), Rust and Wrap E2E. Logs check out synthetic merge
`2ff995d67ef4bb52590c1c0ca95514c85e9693ab` (this head into `3fef15d3`).
Maintainer qualification reports 137 tests passed with ten optional
skips plus Ruff/whitespace; these are their results, not another
author-run suite. Security passes upstream policy, not a zero-findings
claim: E2E npm retains three excepted bundled advisories under
upstream's existing policy until 2026-11-15. No additional exception was
introduced here. Live-provider behavior remains unqualified. Earlier
dated results below are historical.

## Description

Fixes #4332. Replaying a cached compression for an OpenAI `role: tool`
message currently converts list-form content to a string, discarding its
client-supplied `cache_control` and other block metadata. A single text
block now keeps its shape and metadata; only its text is replaced in the
existing deep copy.

For multiple blocks or non-text/malformed block shapes, replay leaves
the message unchanged, before a cache lookup is counted. The cached
joined string does not contain enough information to safely reconstruct
block boundaries or preserve media. String-form OpenAI and Anthropic
behavior are unchanged. This does not duplicate #4333, which explicitly
excludes the token-mode lost-breakpoint issue.

## Type of Change

- [x] Bug fix

## Changes Made

- Preserve the deep-copied single-text OpenAI content block and replace
text only.
- Leave ambiguous lists unchanged before cache hit accounting.
- Share the eligibility guard across cache replay, the rewrite helper
and the real interceptor path before callbacks, disclosure state and
savings accounting.
- Add direct-helper/cache/interceptor regressions for unsupported
shapes, metadata isolation and progressive-disclosure isolation.
- Earlier compatible dependency updates are now incorporated in upstream
main; the final PR delta keeps the cache fix and regressions, with
dependency constraints and locks unchanged from main; the tools manifest
now adds only a test script.

## Testing

- [x] Focused regression tests pass.
- [x] Lint / formatting and changed-module type checks passed at
d74aa5c; runtime/test code is unchanged by a00f387.
- [x] Independent review performed.
- [x] Current 2333a5cc hosted CI/native and Docker regression-hook
checks passed; live-provider qualification is not claimed.
- [x] Current 2333a5cc Security gates pass under upstream policy, with
existing exceptions disclosed above.

### Test Output

```text
# Final lockfile bytes published as a00f387; Linux / Python 3.12.14
PYTHONPATH=. python -m pytest -q tests/test_compression_cache.py tests/test_tool_result_interceptors.py --disable-warnings
115 passed in 31.60s
uv lock --check: passed
cargo metadata --locked: passed
cargo audit (0.22.2): zero reported vulnerabilities
pip-audit of frozen production [all] export: 24 vulnerabilities in Pillow 12.3.0 remain
```

The Rust audit retains the repository's existing RUSTSEC-2024-0436
ignore, unchanged; no new ignores were added. certifi's updated CA
bundle successfully loads 120 CA certificates. The exact remote
Cargo.lock / uv.lock Git blob hashes match the tested files.

At d74aa5c, the five adjacent
cache/registry/interceptor/session/token-mode suites passed 169 tests
with zero skips. Whole-repository Ruff and formatting (1805 files), mypy
for both modified source modules and whitespace checks passed. The
maintainer independently reported 144 Windows/Python 3.12 tests passing
with ten optional skips; that is maintainer evidence, not my local run.

Reproduce focused coverage at d74aa5c (compare added executable lines
against base 4c7acba):

```sh
python -m pytest tests/test_compression_cache.py tests/test_tool_result_interceptors.py --cov=headroom.cache.compression_cache --cov=headroom.proxy.interceptors.base --cov-branch --cov-report=json:coverage.json --cov-report=term-missing -q
```

Independent local coverage at d74aa5c: 115 cache/interceptor tests
passed; all 19 changed executable lines were covered and changed-line
branches had no misses. Whole-module coverage was 92% / 86%, not a
whole-project coverage claim. Codecov's 00:01:55 UTC comment preceded
the relevant shard uploads at 00:02:59 and 00:05:00; final service
aggregation remains unverified.

Historical hosted CI run39657491771 and rust run39657491768 passed for
d74aa5c; Security run39657491814 failed. These are not final a00f387
checks. Maintainers verify hosted checks; no contributor workflow rerun
was requested or initiated.

### Historical dependency blocker (cleared on a618561)

a00f387 updates rustls-webpki to 0.103.16, webpki-roots to 1.0.10 and
certifi to 2026.10.19, only in the lockfiles. Pillow 12.3.0 remains
vulnerable. Pillow 13 requires Python >=3.11; Headroom supports >=3.10.
fastembed 0.8.0 also requires Pillow <13. An actual uv resolver attempt
using conditional Pillow >=13 failed; the attempted manifest edit was
reverted. No overrides, audit suppressions or Python-support removal
were introduced. A compatible fastembed/Pillow release or a separately
agreed dependency/support-policy change is still required.

The initial environment's full-suite collection failed on missing
optional/native dependencies; full native/runtime/live-provider
qualification has not been established locally. Focused checks do not
imply complete qualification.

## Real Behavior Proof

- Environment: Linux, Python 3.12.14, pytest 9.1.1, Ruff 0.17.1.
Synthetic data only, HEADROOM_OFFLINE=1, HEADROOM_BEACON=off,
DO_NOT_TRACK=1; no LLM/provider calls or mocks.

- Exact command / steps: Run the following Python snippet in the project
environment. I called the real public
`CompressionCache.update_from_result` API to seed a cached string
compression, then the real shared `prepare_turn` session engine with a
new list-form tool result. Minimal reproduction:

```python
from headroom.cache.compression_cache import CompressionCache
from headroom.proxy.session_engine import prepare_turn, FREEZE_POLICY_REPLAYABLE

cache = CompressionCache()
raw = "synthetic records " * 50
older = {"role": "tool", "tool_call_id": "c1", "content": raw}
cache.update_from_result([older], [{**older, "content": "<compressed>"}])
newest = {"role": "tool", "tool_call_id": "c2", "content": [
    {"type": "text", "text": raw,
     "cache_control": {"type": "ephemeral", "ttl": "1h"}}
]}
prepared = prepare_turn(cache, [
    {"role": "user", "content": "synthetic request"}, newest
], policy=FREEZE_POLICY_REPLAYABLE)
print(prepared.pipeline_input[-1])
```

- Observed result: Before: content is the string `<compressed>`; the
breakpoint is absent.
After:

```text
{'role': 'tool', 'tool_call_id': 'c2', 'content': [{'type': 'text', 'text': '<compressed>', 'cache_control': {'type': 'ephemeral', 'ttl': '1h'}}]}
original_unchanged: True
frozen_message_count: 1
```

- Not tested: live OpenCode/gateway/Bedrock requests, provider cache
billing or hit rates, author-run Windows/macOS, final-head
native/runtime behavior, full suite. Maintainer Windows results are
separately attributed above. This proof establishes session-preparation
output, not a live provider outcome.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; no new feature/configuration.
Compatible certificate dependency lock versions are refreshed; no
manifest constraints changed.
- Minimum rollout channel: not applicable; this is a cache correctness
fix, not a rollout-managed feature.
- Kill switch / disable path: no new switch; revert this patch to
restore previous behavior.
- Qualification impact: no qualification settings changed; hosted
Rust/native checks passed on the current merge tree; live-provider
qualification remains unverified.
- Stable/default behavior changed: cached replay and direct interception
preserve single-text-block shape/metadata; unsupported layouts pass
through before transformations or savings accounting.
- Unsafe override required: no.
- Rollback path: revert the change.

## Review Readiness

- [x] I have performed a self-review

Independent review has also been performed.
- [x] This PR is ready for human review

Current-head hosted checks and the regression hook passed; maintainer
approval and real behavior proof are recorded above.
- [x] No CHANGELOG edits.
2026-10-29 20:27:26 -05:00
fba09f674c fix(cache): count a tool_result with no text as replayable in the frozen prefix (#4523)
## Description

In Claude Code sessions that load deferred tools, the Anthropic
token-mode pipeline freezes only the messages before the first
ToolSearch result, however far the provider's cache has moved. The log
shows `Pipeline: freezing first 11/166 messages`, and the count stays
there for the rest of the session. Every turn then re-runs compression
over the whole history after that point, only for `finalize_turn` to
overwrite the result with its replay of the provider-confirmed prefix.
Read maturation also trusts that freeze, so it matures Reads the
provider has already cached and the cache re-writes from the Read onward
(#4522).

Root cause: `CompressionCache.compute_frozen_count`
(`headroom/cache/compression_cache.py:293`) stops at a tool_result whose
extracted text is None (`compression_cache.py:323`, "treat as
unstable"). `_extract_text_from_blocks` returns None for a block list
without text (`compression_cache.py:52`): a ToolSearch `tool_reference`
list, or an image. Nothing ever stores such a result or marks it stable,
because `update_from_result` and `mark_stable_from_messages` skip None
too. So it stays a miss for the whole session, and `prepare_turn`'s
`min(tracker_frozen, cache_count)`
(`headroom/proxy/session_engine.py:134`) clamps every turn's freeze to
it.

With this change, a tool_result message with no text in any of its
tool_results counts as stable: the cache has nothing to replay and the
text pipeline has nothing to compress. A message that mixes a textless
first result with a text one still stops the count. The cache keys a
message by its first tool_result, so that text was never stored and
cannot be byte-replayed; fixing that means keying every tool_result,
which belongs in its own change.

What it changes: in such sessions the pipeline freeze follows the
provider-confirmed prefix again, so the pipeline compresses only the new
turn instead of re-running the whole history each turn. Forwarded bytes
inside the confirmed prefix are unchanged, because `finalize_turn`
already replays them. Compression of new content is unchanged.

Evidence from one Extra Headroom user's macOS proxy (wheel 0.41.0, token
mode):
- 299 of the 484 Anthropic requests in its current log froze more than
12 messages below the request size.
- In a sample of that proxy's feed, every long session stopped at a
ToolSearch result. Those holding only `tool_reference` blocks (11/139,
11/166, 11/173) are fixed here. One whose ToolSearch result shares a
message with a text result (13/566 to 13/573) is the mixed case this PR
leaves alone.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/compression_cache.py`: `compute_frozen_count` treats a
tool_result message with no text as stable, through a new
`_carries_tool_result_text` helper (both formats). It still stops at
text it cannot replay, including text beside a textless first result.
- `tests/test_compression_cache.py`: `tool_reference`, image and empty
results no longer stop the count, and a text result beside a textless
one still does.
- `tests/test_session_engine.py`: `prepare_turn` follows the tracker
past a ToolSearch result. The new tests fail on main.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev --extra proxy pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_token_headroom_mode.py -q
90 passed in 4.28s
# without the change, the 4 new tests fail

$ uv run --frozen --extra dev --extra proxy pytest <the 22 tests that fail in a full local run on main>
16 failed, 6 passed
# the same 16 fail on main (f10725e6) on this machine: tests/test_proxy_uds.py (11),
# tests/test_cli/test_wrap_persistent.py (4), test_image_compression.py::TestOnnxRouter::test_full_classify_with_image
# The full suite was not run locally on this branch; CI runs it.

$ ruff check . && ruff format --check .
All checks passed!
1834 files already formatted

$ mypy headroom --ignore-missing-imports
Found 2 errors in 2 files (checked 578 source files)
# the same 2 as main, in files this PR does not touch:
# integrations/langchain/chat_model.py:380, memory/adapters/embedders.py:389
```

## Real Behavior Proof

- Environment: macOS 27 on an M2; this branch and upstream main
(f10725e6), each from its own worktree; `headroom proxy` started with
Extra Headroom's flags and env (token mode, read maturation on via the
beta channel, `--no-http2 --no-rate-limit --log-messages`, isolated
HOME, telemetry off), in front of a local mock Anthropic upstream that
reports cache_read for the longest message prefix shared with an earlier
request and cache_write for the rest.
- Exact command / steps: one scripted Claude Code session per build: a
ToolSearch answered with a `tool_reference` block, a Read of a 7.7 KB
file, then 8 Bash turns, each a 300-entry JSON listing that the router
compresses. The pipeline freeze per request comes from the `Pipeline:
freezing first N/M` log lines. Each forwarded request is diffed against
the previous one with `cache_control` stripped.
- Observed result: the freeze follows the confirmed prefix instead of
staying at the ToolSearch result, and on the last turn the router
processes 1 content-block message instead of 17.

| Build | Pipeline freeze per request | Router content-block msgs, last
turn | Cached prefix changed | cache_read/forwarded | Cache writes |
tok_saved/tok_before |
  |---|---|---|---|---|---|---|
| main | 2, 2, 2, 2, 2, 2, 2, 2, 2 | 17 | request 8, message 4 (a Read
matured) | 67.0% | 39,360 | 43.2% |
| this PR | 2, 4, 4, 8, 10, 12, 14, 16, 18 | 1 | never | 79.5% | 25,994
| 42.2% |
- Not tested: Linux and Windows; real Anthropic traffic on this branch;
the OpenAI token path and `/v1/compress`, which use the replayable
policy, where a larger count only freezes more and cannot bust (their
suites pass); a session where nothing compresses, where the freeze moves
on only as far as the first uncompressed text result (2 to 4 in the same
session with short Bash output), which is why #4522 is needed as well.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, not rollout-managed
- Stable/default behavior changed: yes; in token mode the Anthropic
pipeline now freezes past a tool_result with no text, up to the
provider-confirmed prefix, instead of stopping there for the session.
- Kill switch / disable path: none; revert the commit
- Unsafe override required: no
- Qualification impact: none expected; forwarded bytes inside the
confirmed prefix were already replayed by finalize_turn
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Related: #4522 is the fix for the maturation rewrite and works on its
own; this PR removes one of the two reasons the freeze sticks. The other
is a text tool_result that never went through a pipeline run that
changed something: `update_from_result` only marks unchanged results
stable on such a run (`headroom/proxy/handlers/anthropic.py`, `if
result.messages != working_messages`). That is a broader change, left
for its own PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4523"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-28 23:45:04 -05:00
Joseph Benno 0ae948c151 fix(cache): bound compression cache bookkeeping
## Description

`CompressionCache.max_entries` bounded the main compression cache, but
not `_stable_hashes` or `_first_seen`. A long-lived session could
therefore retain every unique tool-result hash even while `_cache`
stayed empty.

This change applies the same bounded retention to both side tables. It
also cleans up expired first-seen entries and resets the timing window
when compression occurs near the TTL boundary.

Fixes #2874

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Store stable hashes and first-seen timestamps in ordered mappings.
- Evict oldest entries when either side table exceeds `max_entries`.
- Keep all bookkeeping under the existing reentrant lock.
- Reset first-seen timing after compression near the TTL boundary.
- Add tests covering size limits, TTL behavior, frozen-prefix safety,
and concurrency.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run ruff format --check .
Passed

uv run ruff check .
All checks passed!

uv run mypy headroom
Success: no issues found in 515 source files

uv run pytest
Passed
```

Focused cache tests on macOS 26.5.2 arm64 with Python 3.11.14:

```text
uv run pytest tests/test_compression_cache.py::TestCompressionCacheRetention -v
5 passed in 0.30s

uv run pytest tests/test_compression_cache.py -q
38 passed in 5.76s
```

After the final formatting-only commit, the cache test file was also run
on Linux with Python 3.12.13:

```text
37 passed, 1 skipped in 32.70s
```

## Real Behavior Proof

- Environment: Linux 6.18 x86_64, Python 3.12.13,
`CompressionCache(max_entries=100)`.
- Exact command / steps: Created a `CompressionCache(max_entries=100)`,
generated 20,000 unique content hashes, and passed each hash through
`mark_stable()` and `should_defer_compression()`. Store sizes were
sampled after 100, 1,000, 5,000, and 20,000 results.
- Observed result: `_cache=0`, `_stable_hashes=100`, and
`_first_seen=100` at every sample after reaching the configured limit.
At 20,000 results, traced memory was approximately 0.03 MB current and
0.04 MB peak. Before the fix, the same workload retained all 20,000
hashes and timestamps.
- Not tested: A live multi-hour proxy/provider session.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented the code where retention behavior is not obvious
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove the fix is effective
- [x] New and existing unit tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Screenshots

N/A — internal cache bookkeeping change.

## Additional Notes

No changes to dependencies, public APIs, or configuration.

No user-facing behavior changes.
2026-08-11 09:52:27 -07:00
uliasandClaude Opus 4.8 e36fccd8cf refactor: DRY cache logic, add thread safety, fix Bash exclusion (#704)
## Description

Four targeted improvements to ContentRouter and configuration,
refactoring ~120 lines of duplicated cache logic into a shared helper
and fixing several correctness issues.

### 1. DRY: Extract `_compress_block_content` helper
The two-tier cache lookup + compression logic was duplicated ~60 lines
per path (tool_result blocks and text blocks in
`_process_content_blocks`). Extracted into a single, shared helper
method. Net reduction of ~80 lines; no behavioural change.

### 2. Thread-safe `CompressionCache`
`CompressionCache` is read/modified from `ThreadPoolExecutor` workers
during parallel compression in `apply()`. Added a `threading.Lock`
guarding all read-modify-write operations so concurrent cache misses for
the same content do not produce duplicate compression work and metrics
counters stay consistent.

### 3. Remove duplicate Kompress fallback for SmartCrusher
The SMART_CRUSHER strategy block had an inline Kompress fallback that
ran when SmartCrusher produced no savings. The unified post-strategy
fallback block already covers the same case — the inline copy was a
duplicate Kompress invocation. Removed it; the post-strategy handler now
owns all fallback decisions for both SMART_CRUSHER and CODE_AWARE. Also
added a guard preventing duplicate Kompress when CODE_AWARE's inline
fallback fires alongside the unified block.

### 4. Fix Bash exclusion contradiction in `DEFAULT_EXCLUDE_TOOLS`
The docstring on `DEFAULT_EXCLUDE_TOOLS` explicitly states "Bash is NOT
excluded — its outputs (build logs, test output) are ideal compression
targets." But both "Bash" and "bash" were still in the frozenset.
Removed them so code matches the documented intent.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/config.py`: Remove Bash/bash from `DEFAULT_EXCLUDE_TOOLS`
- `headroom/transforms/content_router.py`: Extract
`_compress_block_content` helper; unified post-strategy fallback block;
threading.Lock on CompressionCache; CODE_AWARE duplicate guard
- `headroom/client.py`: Replace silent `except Exception: pass` with
`logger.debug(..., exc_info=True)`
- `tests/test_compression_cache.py`: Add 2 concurrency regression tests
- `tests/test_transforms/test_content_router.py`: Add 14 tests covering
Bash exclusion, SmartCrusher fallback chain, and
`_compress_block_content` shared path

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# 14 new tests added across 3 test classes:
# TestExcludeTools: 3 tests (Bash not in DEFAULT_EXCLUDE_TOOLS)
# TestSmartCrusherFallback: 4 tests (fallback chain, no duplicate Kompress, JSON direct hit, CODE_AWARE path)
# TestCompressBlockContent: 5 tests (skip set, result cache, ratio gating, route counts, transforms tracking)
# TestCompressionCache: 2 tests (concurrent hits/misses consistency, stable hash ops no race)

# Local run (43 tests pass):
$ pytest tests/test_compression_cache.py tests/test_transforms/test_content_router.py -v
...43 passed...

# ruff check:
$ ruff check headroom/client.py headroom/config.py headroom/transforms/content_router.py tests/test_compression_cache.py tests/test_transforms/test_content_router.py
All checks passed!

# ruff format:
$ ruff format --check headroom/client.py headroom/config.py headroom/transforms/content_router.py tests/test_compression_cache.py tests/test_transforms/test_content_router.py
5 files already formatted
```

## Real Behavior Proof

- Environment: Python 3.12, Linux (CI), headroom with headroom._core
Rust extension compiled
- Exact command / steps: CI run
https://github.com/chopratejas/headroom/actions/runs/27326150021 — 13/16
jobs pass; 2 failures were lint+commitlint (both fixed in subsequent
commits); 1 failure is pre-existing test(4) which monkeypatches
time.time() but the CompressionCache uses time.monotonic() — unrelated
to our changes
- Observed result: All 14 new tests pass in CI; SmartCrusher fallback
chain deterministically shows [smart_crusher, kompress] or
[smart_crusher, kompress, log] when SmartCrusher produces no savings,
with no duplicate entries
- Not tested: fork-PR CI path where GitHub secrets are not available;
local Windows environment where headroom._core Rust extension is not
built

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

The pre-existing CI failure in `test (4)` is
`test_compression_cache_handles_hits_skips_evictions_and_clear` in
`tests/test_transforms_content_router.py`. It monkeypatches
`time.time()` but the `CompressionCache` (content_router-local, line
191) uses `time.monotonic()` for TTL — the monkeypatched clock never
advances, and `is_skipped()` always returns True. This failure exists on
`main` and is unrelated to our changes (we only modified the other
CompressionCache in `headroom/cache/compression_cache.py`).

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 14:50:04 -05:00
chopratejas 89f7b6c2dd fix: complete /v1/responses compression telemetry, multi-frame WS, frozen-count cap
Builds on PR #406 (HTTP /v1/responses PyO3) and PR #410 (WS first-frame
PyO3) which landed the binding for `compress_openai_responses_live_zone`.
This change closes the remaining gaps so every (provider × endpoint ×
auth-mode × streaming) combination compresses AND surfaces in the
dashboard.

Telemetry: extend the PyO3 binding return tuple from `(bytes, modified)`
to `(bytes, modified, tokens_saved, transforms_applied)` by adding
`CompressionManifest::tokens_saved()` and `transforms_applied()`
accessors on the existing manifest. The Python proxy populates
request-log telemetry from the binding output instead of recounting
tokens. Updates the existing 2-tuple call sites in HTTP and WS
first-frame, plus the unpacks in tests.

WebSocket multi-frame compression: subscription Codex users keep a
long-lived WS open and send multiple `response.create` events per
session. PR #410 only compressed the first frame; subsequent frames
went raw. Added `_maybe_compress_response_create_frame` closure inside
`_client_to_upstream` that runs the same Rust dispatcher on every
client→upstream `response.create` text frame, passes other event
types (response.cancel, session.update, etc.) through unchanged, and
accumulates `tokens_saved` / `transforms_applied` /
`ws_frames_compressed` counters across the session.

Pre-existing dashboard gap: `streaming.py` and `anthropic.py` write
`RequestLog` entries; the non-streaming OpenAI HTTP and WS handlers
did not. Result: /transformations/feed was invisible for every Codex
turn and every Cline / OpenClaude / Aider turn. Added the same wiring
in `handle_openai_chat` (non-streaming), `handle_openai_responses`
(non-streaming HTTP), and `handle_openai_responses_ws` (session-end).
All three populate `auth_mode` + `endpoint` tags so the dashboard can
break compression activity down by client class (PAYG / OAuth /
Subscription) and surface (`chat_completions` / `responses_http` /
`responses_ws`). The WS metric record is now unconditional — was
previously gated on `tokens_saved > 0`, so first-frame no-changes
never registered.

compute_frozen_count over-freeze for prose-format clients:
`compute_frozen_count` walked until it found an unstable
`tool_result` / `role: "tool"` block. Cline / OpenClaude / Aider —
clients that embed tool calls as XML inside plain text — never
produce such a boundary, so the function returned `len(messages)` and
the pipeline froze 100% of messages including the brand-new user
turn. Live zone empty → `Transform content_router: 16414 → 16414
tokens (saved 0)`. Reported on Discord 2026-05-07 with Cline+DeepSeek.
Fix: cap at `max(0, len(messages) - 1)`. Updates 3 existing test
assertions whose expected values encoded the old over-freeze. Adds 6
new prose-format invariant tests.

CodeQL "clear-text logging of sensitive information" fix:
`tests/e2e_real_compression.py` previously stored API keys in local
variables in the same scope as diagnostic prints, which CodeQL flagged
via data-flow analysis. Refactored to read keys from `os.environ`
inside the request helper — the credentials never enter the runner's
main scope, so the taint flow never reaches the print.

End-to-end verification with real keys (.env):

  /v1/messages         (PAYG, non-stream)  tok 14109 → 969    saved 13140
  /v1/messages         (PAYG, stream)      tok 14109 → 969    saved 13140
  /v1/chat/completions (PAYG, non-stream)  tok 18460 → 1374   saved 17086
  /v1/chat/completions (PAYG, stream)      tok 18460 → 1374   saved 17086 (cache_hit=100%)
  /v1/responses HTTP   (PAYG, non-stream)  bytes 50138 → 597  saved 18391
  /v1/responses WS     (frame 1)           bytes 46429 → 488  saved 16791
  /v1/responses WS     (frame 2 multi)     bytes 46429 → 488  saved 16791
  /v1/responses WS     (response.cancel)   passthrough untouched

Tests: cargo workspace + pytest (4846 pass, 0 fail), make ci-precheck
passed, two E2E scripts (multi-turn HTTP, WS fake-upstream) all green.
2026-05-07 14:50:03 -07:00
chopratejas ea78cf6252 fix(proxy): cache concurrency lock, multi-worker docs, bounded compression executor
Three audit follow-ups from issue #327's deep-dive review.

C1 — CompressionCache concurrency lock
======================================

`CompressionCache` instances are shared per `session_id` and accessed from
async-dispatched threadpool workers. Pre-fix, concurrent requests for the
same session raced on `_cache`, `_stable_hashes`, `_first_seen`, and
`_total_tokens_saved` with no synchronization. Observable failures:

* Lost-update on `_total_tokens_saved` (read-modify-write).
* `RuntimeError: OrderedDict mutated during iteration` from `apply_cached`
  when a concurrent `store_compressed` evicts during the walk.
* Lost stable-hash records — next-turn compute_frozen_count reads
  inconsistent state.

May also explain part of SvenMeyer's `_cache: 0 entries / 1003 misses`
observation: the cache was being clobbered concurrently.

Added `threading.RLock` guarding all mutating methods. `RLock` (not `Lock`)
so future code can call locked methods from inside another locked method
without self-deadlock. Also locked `HeadroomProxy._compression_caches`
dict-of-caches access via a separate `_compression_caches_lock` so two
concurrent calls for the same session_id can't each create distinct
CompressionCache objects (which would split the cache state between them).
The `/stats` endpoint snapshots the cache list under the dict lock before
iterating to avoid eviction-during-iteration.

C2 — Multi-worker CCR fragmentation: documented + startup warning
=================================================================

The in-memory `InMemoryCcrStore` (Rust), `_compression_caches` (Python),
`session_tracker_store` (Python), and TOIN learner state are ALL
per-process. Multi-worker uvicorn round-robins requests across workers,
so a session whose turn-1 lands on worker A may have turn-2 land on
worker B. Worker B has zero knowledge of A's CCR markers, replay cache,
or prefix-cache state. Result: `Retrieve original: hash=X` markers stay
in-context as opaque directives, every fresh tool_result is recompressed
from scratch, and `frozen_message_count=0` causes Anthropic prefix-cache
busts on every cross-worker turn.

Added a "Multi-worker deployment — CCR fragmentation" section in
`RUST_DEV.md` documenting the failure modes, the supported configuration
(`--workers 1`), and the sticky-session workaround for horizontal scale.
The proxy emits a `WARNING`-level log line on startup if `workers > 1` is
detected, pointing at the doc section.

C3 — Bounded compression executor with cancel-aware metrics
===========================================================

`asyncio.wait_for(asyncio.to_thread(pipeline.apply), timeout=...)`
cancellation does NOT propagate into the threadpool worker that's running
Rust code. Once the worker has picked up the task,
`concurrent.futures.Future.cancel()` returns False and the thread runs to
completion. Stuck threads accumulated invisibly on asyncio's default
executor, contending with unrelated `to_thread` callers (file IO, etc.).

Replaced all 7 `asyncio.to_thread` call sites for `pipeline.apply()`
across `proxy/handlers/anthropic.py` (3) and `proxy/handlers/openai.py` (4)
with a new `HeadroomProxy._run_compression_in_executor(fn, *, timeout)`
helper that:

  1. Submits to a dedicated bounded `ThreadPoolExecutor` named
     `headroom-compress` (configurable via
     `ProxyConfig.compression_max_workers`; defaults to
     `min(32, (cpu_count or 1) * 4)`).
  2. Increments `_compression_in_flight` (gauge) when work starts and
     decrements when work completes; tracks `_compression_in_flight_max`
     as a high-water mark.
  3. Detects "leaked threads" by comparing wall-clock elapsed against the
     timeout in the worker's `finally` block. Increments
     `_compression_leaked_threads` when a worker finishes after its
     asyncio future was cancelled. Operators can see the leaked-thread
     rate climbing in `/stats runtime.compression_executor` BEFORE the
     pool fills up.

Tests
=====

* `TestCompressionCacheConcurrency` (3 tests) — many threads
  store_compressed / apply_cached / update_from_result on a single
  CompressionCache; assert no exceptions, no lost updates, no partial
  state.
* `test_get_compression_cache_returns_same_instance_under_contention` —
  32 concurrent `_get_compression_cache(same_id)` calls return the
  identical instance (would split pre-lock).
* `test_proxy_compression_executor.py` (8 tests) — pool size respects
  config, in-flight gauge tracks running compressions, high-water mark
  is monotonic, timeout propagates to awaiter, leaked-thread counter
  increments on post-deadline completion, `/stats` surfaces all three
  gauges.

Verification
============

* All 123 targeted regression tests pass.
* `make ci-precheck` clean.
* No `Co-Authored-By` trailer; conventional `fix:` prefix; no
  `--no-verify`.
2026-05-01 15:25:18 -07:00
chopratejas 44944fb3fe fix(proxy): restore Anthropic compression on token mode (issue #327)
Three bugs combined to drive end-to-end compression on the Anthropic
backend to ~0% in token mode (the default). User report #327 saw a
~9× drop in dashboard savings from one day to the next on Claude
Code traffic; the dashboard headline was technically correct but the
underlying compression genuinely was not running. After this change
the same Claude Code-shape multi-turn conversation goes from
14987 → 14371 tokens at the request boundary on turn 1 and only
recompresses the freshest tool_result on subsequent turns, with the
prior turns frozen byte-identical to preserve the upstream prefix
cache.

Bug 1 — IntelligentContextManager inner ContentRouter has no observer

PR #302 (commit cf979958, 2026-04-28) wired CompressionObserver onto
the outer ContentRouter in proxy/server.py and onto SmartCrusher.
The inner ContentRouter constructed lazily inside
IntelligentContextManager._get_content_router (added Jan 18, 2026
in 57b2de5 alongside the COMPRESS_FIRST strategy) was missed. That
inner router handles the bulk of Claude Code's tool_result-block
compression, so per-strategy counters surfaced by PR #314 in v0.15.0
showed compressions_by_strategy={"text": 6} while
summary.compression.total_tokens_removed=1.3M — math-impossible.

Fix: add observer= parameter to IntelligentContextManager.__init__,
forward it to the inner ContentRouter at intelligent_context.py:525,
and pass observer=self.metrics from proxy/server.py.

Bug 2 — TTL deferral marks every fresh tool_result as stable

should_defer_compression in compression_cache.py returned True on
first-sight (added 2026-04-07 in commit 22dad13 with the intent of
batching first-time compressions near the 5-min cache TTL boundary
to trade many small busts for one). The token-mode walker at
anthropic.py:766-787 walks every message past frozen_message_count,
calls should_defer_compression on each fresh tool_result, gets True,
and advances ttl_frozen += 1 — every iteration. Result:
frozen_message_count grows to len(messages), the pipeline freezes
the entire request, and nothing reaches a real compressor.

The defer-first-sight rationale assumes recurring content within
TTL. Real Claude Code traffic produces unique content per turn, so
"defer until next sight" defers forever. Compressing fresh content
on first sight does not bust any prefix cache because Anthropic has
not cached that byte position yet — it's a cache write either way.

Fix: should_defer_compression returns False on first-sight (record
the timestamp; compress now). Subsequent sightings within TTL still
defer (batch window preserved for genuinely repeating content).
Updated tests in test_compression_cache.py to assert the corrected
semantics and verify _first_seen is recorded on first call.

Bug 3 — cross-tokenizer comparison in token-mode inflation guard

anthropic.py:634 sets original_tokens = tokenizer.count_messages(...)
using the proxy-side EstimatingTokenCounter. The token-mode branch
at line 816 set optimized_tokens = result.tokens_after from
pipeline, which uses the provider-side AnthropicProvider tiktoken
estimator. The two tokenizers disagree by ~25% on the same payload.

The inflation guard at line 901
(if optimized_tokens > original_tokens: revert to originals) treats
those two numbers as comparable. After a real 12% compression the
provider-tokenizer figure was still higher than the proxy-tokenizer
baseline, so the guard fired, optimized_messages was reset to the
original input, transforms_applied was emptied, and tokens_saved
went to 0. The dashboard showed no compression even when the
pipeline successfully compressed.

Fix: recount optimized_tokens with the proxy tokenizer right after
the pipeline returns, so the guard compares apples-to-apples. The
recount cost is a few ms on a 50K-token request and is dwarfed by
upstream call latency.

Verification

* 80 targeted tests across test_compression_cache,
  test_compression_observability, test_proxy_anthropic_cache_stability,
  test_proxy_intelligent_context pass.
* make ci-precheck clean.
* End-to-end real-API run against api.anthropic.com via local proxy:
  - Turn 1 fresh: 14987 → 14371 (4.1%) on a 3-tool-round payload;
    smart_crusher and diff strategies fired with non-zero savings.
  - Turn 2 (turn 1 history + 1 new tool_result): 23161 → 21928 (5.3%);
    only the new tool_result compressed; older turns marked
    router:protected:user_message; Anthropic returned
    cache_creation_input_tokens > 0 confirming the prefix was not
    busted.

Two new regression tests in test_compression_observability lock down
the inner ContentRouter observer wiring so a future copy of Bug 1
fails the suite the day it lands.
2026-04-30 12:59:19 -07:00
JerrettDavisandCopilot ca728a4d35 fix(ci): harden wrap e2e validation
Make the Docker wrap e2e harness validate live proxy env wiring for Codex and Aider, start a real OpenClaw gateway in-container, and clear the repo-wide Ruff issues that were keeping the Python 3.12 CI job red.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-07 23:27:31 -05:00
Tejas Chopra 22dad133e2 fix: eliminate prefix cache busts from frozen count underestimation
Root cause: CompressionCache.compute_frozen_count() stopped at the first
tool_result not in its cache, capping frozen_message_count at 2. Tool
results excluded by content_router (Read/Glob) or skipped (ratio too
high) never entered the cache, so every subsequent message was eligible
for recompression — causing 192 cache busts per session.

Four fixes:
1. Add _stable_hashes set to CompressionCache so excluded/skipped
   tool_results don't block the frozen count walk
2. Fix _estimate_message_tokens to count tool_result content and
   tool_use input fields (were counted as 0 tokens in Anthropic format)
3. Fix streaming handler to include assistant response and
   original_messages in prefix tracker updates (parity with non-streaming)
4. TTL-aware batch recompression: defer first-time compressions within
   the 5-min cache TTL window, batching them at the boundary to trade
   many small busts for one
2026-04-07 17:10:20 -07:00
chopratejasandClaude Opus 4.6 0f2f992132 feat: add frozen count, apply_cached, update_from_result to CompressionCache
Add three methods and supporting helpers for token headroom mode:
- compute_frozen_count: counts consecutive stable messages from start
- apply_cached: swaps cached compressions into tool results (immutable)
- update_from_result: learns new compressions from original/compressed pairs

Supports both Anthropic and OpenAI tool result formats.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 09:52:33 -07:00
chopratejasandClaude Opus 4.6 ee1bcc93bd feat: add CompressionCache with LRU eviction for token headroom mode
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 09:48:52 -07:00