Files
headroom/tests/test_proxy_anthropic_cache_stability.py
Mansurshakh JaparovandJD Davis 0f3b0d8a21 fix(memory): resolve pi cwd before injecting project tools (#4618)
## Description

Closes #4442.

pi's system `<cwd>` block now resolves the memory project. If project
scope is still unresolved with the `empty` fallback, the Anthropic
handler no longer advertises unusable memory tools, including tools
recorded earlier in the session.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Parse pi cwd only in system text. Keep main's line parser, header/CLI
precedence and limited user-message fallback.
- Reuse `is_project_unresolved` for context and sticky-tool replay.
Client tools and global/user storage behavior stay intact.
- Add initialized-handler scope regressions, replace
source-inspection/forwarding-only checks, and document project scope in
the published memory page.

## Testing

- [x] Unit tests pass (`pytest`) — related suite and the normal critical
gate, not the entire Python suite locally.
- [x] Linting passes (`ruff check .`) — hosted lint passed on 5ace63e2;
local filesystem limitation below.
- [x] Type checking passes (`mypy headroom`) — normal full hook with its
existing `--ignore-missing-imports` setting.
- [x] New tests added for the scope regression
- [x] Manual testing performed

### Test Output

```text
# 16 related test files
329 passed, 12 skipped, 1 warning in 43.07s

# ordinary pre-push: make ci-precheck
178 passed, 5 skipped, 1 warning in 10.87s
✅ ci-precheck PASSED — safe to push.

uv run ruff format --check .
1860 files already formatted
```

Normal commit hooks (Ruff, formatter, full mypy, commitlint) and the
complete pre-push path passed: Rust fmt/clippy/workspace tests, native
build/import verification, critical Python tests and commitlint. The
existing Starlette test-client warning was not filtered.

Local whole-tree Ruff reports EXE002 on five **unchanged** OAuth plugin
files: my `fuseblk` checkout exposes them as executable despite Git
modes `100644`; `chmod` returns zero without changing the mode. The
uncached check agrees. I did not change those files, add shebangs or
ignore the rule. Hosted Ruff, formatting and full mypy passed on
5ace63e2; the local filesystem limitation remains.

The first full CI caught a missing scope predicate on the existing
lifecycle test double. Fixed in
[5ace63e2](https://github.com/headroomlabs-ai/headroom/commit/5ace63e2cbd2295f9594d1e0442a679aad99979c);
its complete file passes (`5 passed, 1 warning in 13.01s`). This
follow-up changes no production code or assertions and adds no runtime
fallback. The behavior proof below ran on 7aeb83cf with the same
production source; that qualification is historical; current-head checks
are recorded below.

### Author follow-up: 80453a03

Moved the project-scope guidance out of the unpublished wiki and into
`docs/content/docs/memory.mdx`; this commit changes no production code
or tests. Viewed the actual Next.js `/docs/memory#proxy-project-scope`
page and its table-of-contents entry; no browser errors.

The existing router/scope tests pass: 65 passed, with four SQLite
ResourceWarnings. All seven pi-parser lines and all six branch edges are
covered, so I did not add duplicate tests for the old Codecov
percentage.

Normal commit and pre-push hooks passed, including Rust checks, native
build/import, 180 critical Python tests (5 skipped, 1
tokenizer-approximation warning), and commitlint. My first push timed
out with the wrong local ONNX path; the next found missing pip in the
fresh uv environment. Both setup issues were fixed without bypassing
checks. Current-head hosted CI is not yet qualified.

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, freshly built native core,
real TCP/Uvicorn proxy and SQLite/ONNX memory backend. Only the upstream
is a local Anthropic-shaped HTTP server; no provider charges.
- Exact command / steps: after `uv sync --locked --extra dev --extra
proxy --extra memory` and `make verify-rust-core`, run this from the
checkout. It prints the actual loaded native module/hash and leaves a
disposable `.headroom-4442-*` directory for inspecting SQLite. Both HTTP
listeners shut down before exit.

<details>
<summary>Standalone HTTP/SQLite reproduction</summary>

```bash
./.venv/bin/python - <<'PY'
import hashlib,http.client,json,os,socket,tempfile,threading,time
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from pathlib import Path
import uvicorn
import headroom._core as core
assert Path(core.__file__).resolve().parent == Path.cwd().resolve()/"headroom"
native_sha=hashlib.sha256(Path(core.__file__).read_bytes()).hexdigest()
from headroom.memory.storage_router import ProjectResolver,RequestContext,extract_system_prompt
from headroom.proxy.server import ProxyConfig,create_app
from headroom.proxy.memory_handler import MEMORY_TOOL_NAMES
from headroom.proxy.helpers import get_session_tool_tracker
root=Path(tempfile.mkdtemp(prefix=".headroom-4442-",dir=Path.cwd()))
project_a=root/"pi%2Fproject-a";project_b=root/"project-b"
project_a.mkdir(exist_ok=True);project_b.mkdir(exist_ok=True)
os.environ["HEADROOM_WORKSPACE_DIR"]=str(root/"workspace")
os.environ["HEADROOM_BEACON"]="off"
os.environ["HEADROOM_TELEMETRY"]="off"
os.environ["NO_PROXY"]="127.0.0.1,localhost"
forwarded=[]
class Gateway(BaseHTTPRequestHandler):
    def do_POST(self):
        body=json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        forwarded.append({"path":self.path,"body":body})
        payload=json.dumps({"id":"msg_local_scope","type":"message","role":"assistant","model":"claude-haiku-4-5","content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn","stop_sequence":None,"usage":{"input_tokens":10,"output_tokens":1}}).encode()
        self.send_response(200);self.send_header("Content-Type","application/json");self.send_header("Content-Length",str(len(payload)));self.end_headers();self.wfile.write(payload)
    def log_message(self,*args):pass
upstream=ThreadingHTTPServer(("127.0.0.1",0),Gateway)
ut=threading.Thread(target=upstream.serve_forever,daemon=True);ut.start()
config=ProxyConfig(mode="cache",anthropic_api_url=f"http://127.0.0.1:{upstream.server_port}",memory_enabled=True,memory_db_path=str(root/"legacy.db"),traffic_learning_enabled=True,subscription_tracking_enabled=False,cost_tracking_enabled=False,cache_enabled=False,rate_limit_enabled=False,log_requests=False,image_optimize=False,ccr_inject_tool=False,ccr_handle_responses=False,ccr_context_tracking=False,anthropic_pre_upstream_memory_context_timeout_seconds=30.0)
app=create_app(config)
sock=socket.socket();sock.bind(("127.0.0.1",0));sock.listen(128)
port=sock.getsockname()[1]
server=uvicorn.Server(uvicorn.Config(app,log_level="warning",lifespan="on",timeout_graceful_shutdown=3))
st=threading.Thread(target=server.run,kwargs={"sockets":[sock]},daemon=True);st.start()
try:
    deadline=time.monotonic()+180
    while not server.started:
        if not st.is_alive() or time.monotonic()>deadline:raise RuntimeError("real proxy startup failed")
        time.sleep(.02)
    session="headroom4442-native-project-scope"
    common={"Content-Type":"application/json","x-api-key":"local-unbilled-key","anthropic-version":"2023-06-01","x-headroom-user-id":"native-scope-user","x-headroom-session-id":session}
    client_tools=[{"name":"get_weather","description":"Weather","input_schema":{"type":"object","properties":{}}}]
    rows=[]
    cases=[("fresh_unresolved","You are helpful.",{},False),("pi_cwd",f"<cwd>\n{project_a}\n</cwd>",{},True),("warm_unresolved","You are helpful.",{},False),("pi_cwd_restored",f"<cwd>\n{project_a}\n</cwd>",{},True),("explicit_header_wins",f"<cwd>\n{project_b}\n</cwd>",{"x-headroom-project-id":"explicit-project"},True),("ordinary_user_cwd_ignored","You are helpful.",{},False)]
    for label,system,extra,expected in cases:
        body={"model":"claude-haiku-4-5","max_tokens":16,"system":system,"messages":[{"role":"user","content":f"cwd:\n<cwd>\n{project_b}\n</cwd>" if label=="ordinary_user_cwd_ignored" else "hello scope"}],"tools":client_tools}
        before=len(forwarded)
        conn=http.client.HTTPConnection("127.0.0.1",port,timeout=90)
        conn.request("POST","/v1/messages",body=json.dumps(body).encode(),headers={**common,**extra})
        response=conn.getresponse();payload=response.read();conn.close()
        assert response.status==200,(label,response.status,payload.decode())
        assert len(forwarded)==before+1,(label,"unexpected upstream request count")
        actual=forwarded[-1]["body"]
        names={tool.get("name") for tool in actual.get("tools",[])}
        assert names==({"get_weather"}|MEMORY_TOOL_NAMES if expected else {"get_weather"}),(label,names)
        scope=ProjectResolver().resolve(RequestContext(headers={**common,**extra},system_prompt=extract_system_prompt(body),base_user_id="native-scope-user"))
        rows.append({"case":label,"status":response.status,"memoryTools":sorted(names & MEMORY_TOOL_NAMES),"project":scope,"gatewayPath":forwarded[-1]["path"]})
    assert forwarded[1]["body"]["tools"]==forwarded[3]["body"]["tools"],"restored definitions changed"
    tracker=get_session_tool_tracker()
    assert tracker.should_inject("anthropic",session),"probe did not exercise an actual recorded sticky session"
    from urllib.parse import quote
    key_a=ProjectResolver().resolve(RequestContext(headers={"x-headroom-cwd":quote(str(project_a),safe="/")},system_prompt="",base_user_id="native-scope-user"))[0]
    assert rows[1]["project"][0]==key_a
    assert rows[4]["project"][1]=="explicit-project"
    router=app.state.proxy.memory_handler._router
    assert router is not None,"real memory backend never initialized"
    opened=router.open_backends()
    project_db=root/"memories"/"projects"/key_a/"memory.db"
    assert project_db in opened and project_db.is_file(),("real pi project database not opened",[str(x) for x in opened])
    print(json.dumps({"mode":config.mode,"realHttpListener":f"http://127.0.0.1:{port}","localGateway":f"http://127.0.0.1:{upstream.server_port}","rows":rows,"realStickyGoldenSession":True,"realPiProjectDatabase":str(project_db),"openedDatabases":[str(x) for x in opened],"nativeModule":core.__file__,"nativeSha256":native_sha,"noLivePiOrPaidProvider":True},indent=2))
finally:
    server.should_exit=True;st.join(20)
    upstream.shutdown();upstream.server_close();ut.join(5);sock.close()
    assert not st.is_alive() and not ut.is_alive(),"native proof leaked a server"


PY
```

</details>

- Observed result: unchanged `main` fc1ba891 fails at the first
unresolved request: five proxy memory tools reach upstream. With the
fix, all six requests return HTTP 200:

| Same-session request | Proxy memory tools |
| --- | ---: |
| Fresh, no cwd | 0 |
| pi system cwd | 5 |
| Cwd absent again | 0 |
| Same pi cwd restored | 5 |
| Explicit project header, different system cwd | 5 |
| Cwd block only in ordinary user text | 0 |

The client weather tool stays present. Restored definitions are
structurally equal to the first resolved turn; the explicit header wins.
The literal-percent cwd agrees with its encoded-header form, and the
actual project SQLite database opens at the expected key.

- Not tested: live pi/Caveman, paid-provider cache accounting/billing,
Windows, or other provider protocols. This does not claim to fix Caveman
header forwarding.

## Runtime Rollout Safety

- Rollout-managed feature(s): none added; existing Anthropic memory
injection.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: pi cwd resolves; unresolved
project/empty scope no longer adds unusable proxy memory tools.
- Kill switch / disable path: existing `--no-memory`; explicit
project/cwd headers still work.
- Unsafe override required: no.
- Qualification impact: no storage migration or new dependency; scoped
HTTP/SQLite behavior checked.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — related
suite and critical gate, not a whole-suite claim
- [x] I did **not** edit `CHANGELOG.md` — release-please owns it

## Screenshots (if applicable)

N/A; the observable surface is the HTTP payload and project database.

## Additional Notes

AI-assisted; no subagents. OpenAI injection policy is unchanged. Keeping
this draft until the current hosted checks are qualified.

One earlier post-build probe completed the requests and closed both
listeners but hit its whole-process shutdown deadline. The same program
and native artifact later exited normally with faulthandler enabled; its
diagnostic abort timer did not fire. I have not identified the
intermittent cause and am not claiming a shutdown fix. No production
timeout or feature flag was changed to get that result.


The two npm audits now pass on 5ace63e2, along with the other Security
jobs. The earlier [unchanged-base
failure](https://github.com/headroomlabs-ai/headroom/actions/runs/41399323584)
is historical evidence, not a current blocker. This PR changes no
dependency manifests, lockfiles, workflow or audit exceptions.


### Maintainer current-main requalification

Exact head `60597623e00967153d947a46bf197950250f310e`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Clean mechanical update from current upstream main, preserving the pi
system-cwd resolver, request-scoped project guard, main's cache-mode
search skip/logging, and client-tool preservation. Ordinary user cwd
blocks still do not enable the pi fallback; unresolved project scope
still blocks both context injection and replayed memory tools. No
contributor production logic was rewritten and no new production finding
remains in this reviewed scope.

The initial combined tree dropped main's memory-invariant file and two
stream/timeout regression cases. Maintainer test-only repairs restore
those tests rather than reducing coverage: the invariant file is
byte-identical to main, every main assertion in the four
restored/updated test files is preserved by AST comparison, and three
fixtures now supply the handler protocol methods needed for resolved
scope and beta headers. The lifecycle main control passes without that
fixture extension; no claim is made that this particular fixture failure
independently fails main. All 222 affected
scope/invariant/stream/timeout/lifecycle/backpressure/cache-stability/system-prompt
tests pass with one skip. Ten shared Gemini request-fixture cases also
pass after the separate test-only repair for independently verified main
failures. All 12 main-relative changed Python files pass Ruff/format;
whitespace checks pass.

The author's existing real TCP/SQLite/ONNX reproduction and explicit
limits were read and remain in the body. These fresh Windows checks
qualify the combined request-scope regressions; they are not a claim of
rerunning that Linux proof, live pi/client/provider behavior, or a fresh
native build. An existing binary was supplied solely for test startup.
Draft status is preserved; readiness remains the author's decision and
nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `60e01cf7c25e90abb9f5cb66fa221da83e7c5177`, upstream main
`ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`.

Clean integration with upstream/main
ea59ac3eef. Re-reviewed project-scoping
delta against intervening memory/backend/stream changes: pi cwd remains
limited to system context; ordinary user cwd blocks do not enable
fallback; unresolved projects skip context/tool injection and sticky
tool replay while retaining client tools and main cache-mode no-search
behavior.232
scope/invariant/stream/timeout/lifecycle/backpressure/cache/system-prompt/Gemini
tests pass,1optional skip; touched-source Ruff and whitespace pass. No
contributor production policy rewritten. Earlier author TCP/SQLite/ONNX
evidence remains scoped, with no fresh native build or live-pi/provider
claim. Existing draft preserved; final approval awaits exact-head hosted
qualification. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `e93ceb35df7f78bfa2dcc0611609131cf2b08b0a`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 232 passed, 1 skipped, 1 warning in
47.62s. Selected real dependency overlays and native DLL paths are
recorded in final4618-fresh-native-tests.log; the DLL provenance/input
hashes are in final-native-artifact-proof.json. No redundant rerun is
implied after this Cargo-only merge. Whitespace passes. Prior source
review and scoped real-behavior evidence remain applicable to unchanged
contributor inputs.

No new source finding in the reviewed scope. Final formal approval
awaits exact-head hosted qualification. Project resolution and
unresolved-scope injection/replay gates remain unchanged. Existing draft
status is preserved.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `d693dd2b6e33fd47ff853f7e28d938fc1dbc30ac`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Current-main requalification
against2b0315b2d4b0d54947050bbdad2de9e89df06378. Clean source
integration preserves pi systemcwd resolution and unresolved-scope
context/tool/sticky-replay gates.70integration patches exactly matchmain
delta;Anthropic exception patch differs only byclosing-parenthesis diff
alignment and its exception AST equalsmain. Broad
memory/router/projectscope/continuation/backpressure/cache/lifecycle/doctor/nativeops
run:405passed,1skipped,one newmain fake-handler fixture failed
missingis_project_unresolved. Maintainer adds only that fake method
returningFalse,respecting resolvedfixture scope;all39backpressure tests
nowpass,including unchanged
injectedSQLiteexception,HTTP200andsemaphorerelease assertions. No
production logic changed.33changedPythonlint/format,offline
lock/whitespacepass;fixedtestlint/formatrechecked.
Existingpr3982nativeDLLverified fullinputhashes/checksum;no
freshbuildclaimed. Currentauthor/Devin discussion checked,inline
endpoint empty;prior scoped realbehavior evidence remains applicable to
unchanged contributor inputs. No new sourcefinding;no
realprovider/agentlaunch claim. Draftpreserved.
Exactnewheadhostedqualification pending;maintainers
handlechecks,nothingmerged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `0da0b5a01958eaaac1d625227cb4bdb9477c5d3b`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main delta after diff-location
normalization. Contributor pi system-cwd resolution and
unresolved-project memory context/tool/sticky-replay gates are
unchanged. Fresh
memory-router/project-scope/streaming/continuation/backpressure/cache/lifecycle
and incoming Vertex/backend/CLI tests: 381 passed, 12 skipped, 1
warning. Twelve changed Python files pass Ruff lint/format; offline lock
and whitespace pass.

Static production/test scope and security review considered user versus
system cwd inputs, malformed/empty cwd, missing scope and stale tool
replay: tests keep pi cwd tags in ordinary user content from resolving
project scope, and unresolved scope blocks replay. No unrelated
executable, workflow, dependency or setup changes. Flagged inherited
workflow-validation job is excluded by unchanged workflow paths. Author
discussion checked; collected inline endpoint has no findings. Prior
scoped behavior evidence applies to unchanged contributor inputs; no new
real agent/provider launch claimed. Native DLL reused only after
complete tracked input-set/hash and artifact checksum verification, no
fresh build claimed. Draft preserved. Formal approval awaits exact
published-head hosted qualification. Maintainers handle checks; nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-10 22:28:01 -06:00

2065 lines
78 KiB
Python

"""Regression tests for Anthropic prefix-cache stability in proxy mode."""
from __future__ import annotations
import copy
from types import SimpleNamespace
from unittest.mock import AsyncMock
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
from headroom.proxy.server import ProxyConfig, create_app
def _force_compression(monkeypatch) -> None: # noqa: ANN001
decision = SimpleNamespace(should_compress=True, passthrough_reason=None)
decision.apply_to_tags = lambda tags: None
monkeypatch.setattr(
"headroom.proxy.handlers.anthropic.CompressionDecision.decide",
lambda **kwargs: decision,
)
class _FakePrefixTracker:
def __init__(self, frozen_count: int):
self._frozen_count = frozen_count
self._cached_token_count = 0
self._last_original_messages = []
self._last_forwarded_messages = []
def get_frozen_message_count(self) -> int:
return self._frozen_count
def get_last_original_messages(self): # noqa: ANN201
return self._last_original_messages.copy()
def get_last_forwarded_messages(self): # noqa: ANN201
return self._last_forwarded_messages.copy()
def update_from_response(self, **kwargs): # noqa: ANN003
self._cached_token_count = kwargs.get("cache_read_tokens", 0) + kwargs.get(
"cache_write_tokens", 0
)
self._last_original_messages = kwargs.get(
"original_messages", kwargs.get("messages", [])
).copy()
self._last_forwarded_messages = kwargs.get("messages", []).copy()
return None
class _FakeImageCompressor:
def __init__(self):
self.last_result = None
def has_images(self, messages): # noqa: ANN001
return True
def compress(self, messages, provider="anthropic"): # noqa: ANN001
assert provider == "anthropic"
assert len(messages) == 1
msg = messages[0]
content = msg["content"]
updated_content = []
for block in content:
if isinstance(block, dict) and block.get("type") == "image":
src = block.get("source", {})
updated_content.append(
{
"type": "image",
"source": {**src, "data": "COMPRESSED_IMAGE_BYTES"},
}
)
else:
updated_content.append(block)
return [{**msg, "content": updated_content}]
def _make_proxy_client() -> TestClient:
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=True,
)
app = create_app(config)
return TestClient(app)
@pytest.mark.parametrize(
("optimize", "expected_names"),
[
(False, ["zeta", "alpha", "mu"]),
(True, ["alpha", "mu", "zeta"]),
],
)
def test_anthropic_tools_forwarding_order_matches_optimization_mode(
optimize: bool,
expected_names: list[str],
) -> None:
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = optimize
proxy.config.mode = "token"
if optimize:
proxy.anthropic_pipeline.apply = lambda **kwargs: SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
timing={},
tokens_before=100,
tokens_after=100,
waste_signals=None,
)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msg_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 10,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 128,
"messages": [{"role": "user", "content": "hello"}],
"tools": [
{"name": "zeta", "description": "z", "input_schema": {"type": "object"}},
{"name": "alpha", "description": "a", "input_schema": {"type": "object"}},
{"name": "mu", "description": "m", "input_schema": {"type": "object"}},
],
},
)
assert response.status_code == 200
sent_tools = captured["body"]["tools"]
assert [t["name"] for t in sent_tools] == expected_names
def test_image_compression_only_applies_to_latest_non_frozen_user_turn() -> None:
fake_compressor = _FakeImageCompressor()
old_image = {
"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": "OLD_IMAGE_BYTES"},
}
new_image = {
"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": "NEW_IMAGE_BYTES"},
}
messages = [
{"role": "user", "content": [old_image, {"type": "text", "text": "old image turn"}]},
{"role": "assistant", "content": "ack"},
{"role": "user", "content": [new_image, {"type": "text", "text": "new image turn"}]},
]
result = AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
messages,
frozen_message_count=1,
compressor=fake_compressor,
)
# Frozen prefix must remain byte-identical.
assert result[0]["content"][0]["source"]["data"] == "OLD_IMAGE_BYTES"
# Latest non-frozen user turn is eligible for compression.
assert result[2]["content"][0]["source"]["data"] == "COMPRESSED_IMAGE_BYTES"
def test_image_compression_does_not_touch_previous_turns_if_last_message_not_user() -> None:
fake_compressor = _FakeImageCompressor()
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": "OLD_IMAGE_BYTES",
},
}
],
},
{"role": "assistant", "content": "last turn is assistant"},
]
result = AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
messages,
frozen_message_count=0,
compressor=fake_compressor,
)
assert result[0]["content"][0]["source"]["data"] == "OLD_IMAGE_BYTES"
@pytest.mark.parametrize(
("optimize", "expected_names"),
[
(False, ["zeta", "alpha", "mu"]),
(True, ["alpha", "mu", "zeta"]),
],
)
def test_anthropic_batch_tools_forwarding_order_matches_optimization_mode(
optimize: bool,
expected_names: list[str],
) -> None:
captured = {}
config = ProxyConfig(
optimize=optimize,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
)
app = create_app(config)
with TestClient(app) as client:
proxy = client.app.state.proxy
proxy.config.mode = "token"
if optimize:
proxy.anthropic_pipeline.apply = lambda **kwargs: SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
timing={},
tokens_before=100,
tokens_after=100,
waste_signals=None,
)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msgbatch_1",
"type": "message_batch",
"processing_status": "in_progress",
"request_counts": {
"processing": 1,
"succeeded": 0,
"errored": 0,
"canceled": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages/batches",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"requests": [
{
"custom_id": "req-1",
"params": {
"model": "claude-sonnet-4-6",
"max_tokens": 128,
"messages": [{"role": "user", "content": "hello"}],
"tools": [
{
"name": "zeta",
"description": "z",
"input_schema": {"type": "object"},
},
{
"name": "alpha",
"description": "a",
"input_schema": {"type": "object"},
},
{
"name": "mu",
"description": "m",
"input_schema": {"type": "object"},
},
],
},
}
]
},
)
assert response.status_code == 200
sent_tools = captured["body"]["requests"][0]["params"]["tools"]
assert [t["name"] for t in sent_tools] == expected_names
def test_append_context_targets_latest_non_frozen_user_turn() -> None:
messages = [
{"role": "user", "content": "frozen prefix"},
{"role": "assistant", "content": "ack"},
{"role": "user", "content": "active turn"},
]
result = AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn(
messages,
"CTX",
frozen_message_count=1,
)
assert result[0]["content"] == "frozen prefix"
assert result[2]["content"].endswith("CTX")
def test_append_context_does_not_touch_previous_turns_if_last_message_not_user() -> None:
messages = [
{"role": "user", "content": "previous user turn"},
{"role": "assistant", "content": "assistant last"},
]
result = AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn(
messages,
"CTX",
frozen_message_count=0,
)
assert result[0]["content"] == "previous user turn"
def test_token_mode_freeze_is_capped_by_prefix_tracker() -> None:
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "token"
proxy.config.image_optimize = False
fake_tracker = _FakePrefixTracker(frozen_count=1)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
class _FakeCompressionCache:
def apply_cached(self, messages): # noqa: ANN001
return messages
def compute_frozen_count(self, messages): # noqa: ANN001
return 99
def update_from_result(self, originals, compressed): # noqa: ANN001
return None
def mark_stable_from_messages(self, messages, up_to): # noqa: ANN001
pass
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache()
def _fake_apply(**kwargs):
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
return SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
timing={},
tokens_before=50,
tokens_after=50,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "msg_tc_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 50,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [{"role": "user", "content": "hello"}],
},
)
assert response.status_code == 200
assert captured["frozen_message_count"] == 1
def test_memory_context_avoids_system_mutation_when_prefix_frozen() -> None:
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = False
proxy.config.image_optimize = False
proxy.config.ccr_proactive_expansion = False
fake_tracker = _FakePrefixTracker(frozen_count=1)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy.memory_handler = SimpleNamespace(
config=SimpleNamespace(inject_context=True, inject_tools=False),
is_project_unresolved=lambda request_context: False,
search_and_format_context=AsyncMock(return_value="MEMCTX"),
has_memory_tool_calls=lambda resp, provider: False,
)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msg_mem_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 20,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={
"x-api-key": "test-key",
"anthropic-version": "2023-06-01",
"x-headroom-user-id": "u1",
},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"system": "base system",
"messages": [
{"role": "user", "content": "frozen prefix"},
{"role": "assistant", "content": "ack"},
{"role": "user", "content": "latest user"},
],
},
)
assert response.status_code == 200
sent = captured["body"]
assert sent["system"] == "base system"
assert sent["messages"][2]["content"].endswith("MEMCTX")
def test_ccr_system_instruction_injection_disabled_when_prefix_frozen(monkeypatch) -> None:
captured = {"inject_system": None}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = False
proxy.config.image_optimize = False
proxy.config.ccr_inject_tool = False
proxy.config.ccr_inject_system_instructions = True
fake_tracker = _FakePrefixTracker(frozen_count=1)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
class _FakeInjector:
def __init__(
self,
provider, # noqa: ANN001
inject_tool, # noqa: ANN001
inject_system_instructions, # noqa: ANN001
):
captured["inject_system"] = inject_system_instructions
self.has_compressed_content = False
self.detected_hashes = []
def process_request(self, messages, tools): # noqa: ANN001
return messages, tools, False
def scan_for_markers(self, messages): # noqa: ANN001
return []
def verify_ownership(self, store=None): # noqa: ANN001
return self.detected_hashes
monkeypatch.setattr("headroom.ccr.CCRToolInjector", _FakeInjector)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "msg_ccr_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 20,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [{"role": "user", "content": "hello"}],
},
)
assert response.status_code == 200
assert captured["inject_system"] is False
def test_ccr_tool_injection_disabled_when_prefix_frozen(monkeypatch) -> None:
captured = {"inject_tool": None}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = False
proxy.config.image_optimize = False
proxy.config.ccr_inject_tool = True
proxy.config.ccr_inject_system_instructions = False
fake_tracker = _FakePrefixTracker(frozen_count=1)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
class _FakeInjector:
def __init__(
self,
provider, # noqa: ANN001
inject_tool, # noqa: ANN001
inject_system_instructions, # noqa: ANN001
):
captured["inject_tool"] = inject_tool
self.has_compressed_content = False
self.detected_hashes = []
def process_request(self, messages, tools): # noqa: ANN001
return messages, tools, False
def scan_for_markers(self, messages): # noqa: ANN001
return []
def verify_ownership(self, store=None): # noqa: ANN001
return self.detected_hashes
monkeypatch.setattr("headroom.ccr.CCRToolInjector", _FakeInjector)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "msg_ccr_tool_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 20,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [{"role": "user", "content": "hello"}],
},
)
assert response.status_code == 200
assert captured["inject_tool"] is False
def test_ccr_tool_stays_in_forwarded_tools_across_frozen_transition() -> None:
"""``tools`` identity must survive the ``frozen 0 -> >0`` transition.
``tools`` is the head of Anthropic's cache key, so adding or removing
``headroom_retrieve`` between turns invalidates 100% of the provider-cached
prefix — in both directions. Turn 1 (cold prefix, fresh markers) injects the
tool; turn 2 (warm prefix, no *new* markers) must forward the same bytes
rather than dropping it.
Asserts on the forwarded request body, not on a policy function's return
value: unit-testing the old policy in isolation is exactly what let a
wrong-but-self-consistent decision pass.
"""
from headroom.cache.compression_store import get_compression_store, reset_compression_store
from headroom.ccr.tool_injection import CCR_TOOL_NAME
from headroom.proxy.helpers import (
_reset_session_ccr_tracker_for_test,
serialize_tool_definition_canonical,
)
# verify_ownership() (issue #2836) requires the marker's hash to be a
# real store entry — seed one with the exact hash the marker text below
# references, via explicit_hash (the store's own hash generation from
# `original` content wouldn't match this hand-typed literal).
reset_compression_store()
get_compression_store().store(
original="original tool output",
compressed="[50 items compressed to 5]",
explicit_hash="abc123def456abc123def456",
)
marker_message = {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_bash_x",
"content": (
"[50 items compressed to 5. Retrieve more: hash=abc123def456abc123def456]"
),
}
],
}
forwarded: list[dict] = []
_reset_session_ccr_tracker_for_test()
try:
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = False
proxy.config.image_optimize = False
proxy.config.ccr_inject_tool = True
proxy.config.ccr_inject_system_instructions = False
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"frozen-transition-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
forwarded.append(body)
return httpx.Response(
200,
json={
"id": "msg_frozen_transition",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 20,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
def _post():
return client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [marker_message],
},
)
# Turn 1 — cold prefix, marker is new: first-time injection.
assert _post().status_code == 200
# Turn 2 — the provider cached turn 1's prefix (tool included), and
# the marker is now historical rather than new. Seed both facts
# explicitly instead of relying on ``update_from_response`` plumbing:
# if the marker still counted as new, the old code would have
# injected via its override path and this test would pass against
# the defect.
fake_tracker._frozen_count = 3
fake_tracker._last_forwarded_messages = [marker_message]
assert _post().status_code == 200
finally:
_reset_session_ccr_tracker_for_test()
reset_compression_store()
assert len(forwarded) == 2, "expected exactly two forwarded requests"
def _ccr_tools(body: dict) -> list[dict]:
return [t for t in (body.get("tools") or []) if t.get("name") == CCR_TOOL_NAME]
turn1 = _ccr_tools(forwarded[0])
turn2 = _ccr_tools(forwarded[1])
assert turn1, "test setup: turn 1 should inject headroom_retrieve on fresh markers"
assert turn2, (
"headroom_retrieve was dropped from the forwarded tools array once the "
"prefix went warm — that removes a tool already inside the cached prefix "
"and busts 100% of it"
)
# Byte-identity, not ``==``: a re-serialized definition with a different key
# order compares equal as a dict but busts the cache just as hard.
assert serialize_tool_definition_canonical(turn1[0]) == serialize_tool_definition_canonical(
turn2[0]
), "headroom_retrieve was re-serialized rather than replayed byte-for-byte"
def test_previous_turns_always_frozen_only_final_turn_mutable() -> None:
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "cache"
proxy.config.image_optimize = False
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy.anthropic_pipeline.apply = lambda **kwargs: (_ for _ in ()).throw(
AssertionError("cache mode should not invoke anthropic pipeline")
)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msg_frz_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 80,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "current turn"},
],
},
)
assert response.status_code == 200
assert captured["body"]["messages"] == [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "current turn"},
]
def test_batch_optimization_freezes_previous_turns_only() -> None:
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "cache"
proxy.config.image_optimize = False
proxy.config.ccr_inject_tool = False
proxy.anthropic_pipeline.apply = lambda **kwargs: (_ for _ in ()).throw(
AssertionError("cache mode batch path should not invoke anthropic pipeline")
)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msgbatch_2",
"type": "message_batch",
"processing_status": "in_progress",
"request_counts": {
"processing": 1,
"succeeded": 0,
"errored": 0,
"canceled": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages/batches",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"requests": [
{
"custom_id": "req-1",
"params": {
"model": "claude-sonnet-4-6",
"max_tokens": 128,
"messages": [
{"role": "user", "content": "old turn"},
{"role": "assistant", "content": "old assistant"},
{"role": "user", "content": "current turn"},
],
},
}
]
},
)
assert response.status_code == 200
assert captured["body"]["requests"][0]["params"]["messages"] == [
{"role": "user", "content": "old turn"},
{"role": "assistant", "content": "old assistant"},
{"role": "user", "content": "current turn"},
]
def test_batch_optimization_passes_savings_profile_kwargs() -> None:
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "token"
proxy.config.savings_profile = "agent-90"
proxy.config.ccr_inject_tool = False
def _fake_apply(**kwargs):
captured["pipeline_kwargs"] = kwargs
return SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
timing={},
tokens_before=100,
tokens_after=80,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msgbatch_profile",
"type": "message_batch",
"processing_status": "in_progress",
"request_counts": {
"processing": 1,
"succeeded": 0,
"errored": 0,
"canceled": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages/batches",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"requests": [
{
"custom_id": "req-1",
"params": {
"model": "claude-sonnet-4-6",
"max_tokens": 128,
"messages": [{"role": "user", "content": "compress me"}],
},
}
]
},
)
assert response.status_code == 200
pipeline_kwargs = captured["pipeline_kwargs"]
assert pipeline_kwargs["force_kompress"] is True
assert pipeline_kwargs["target_ratio"] == 0.10
assert pipeline_kwargs["compress_user_messages"] is True
def test_token_mode_does_not_force_freeze_all_previous_turns() -> None:
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "token"
proxy.config.image_optimize = False
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
class _FakeCompressionCache:
def apply_cached(self, messages): # noqa: ANN001
return messages
def compute_frozen_count(self, messages): # noqa: ANN001
return 0
def update_from_result(self, originals, compressed): # noqa: ANN001
return None
def mark_stable_from_messages(self, messages, up_to): # noqa: ANN001
pass
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache()
def _fake_apply(**kwargs):
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
return SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
timing={},
tokens_before=70,
tokens_after=70,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "msg_tok_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 70,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "current turn"},
],
},
)
assert response.status_code == 200
# In token_headroom mode, mark_stable_from_messages marks prior turns
# as stable, so frozen count reflects the number of prior-turn messages.
# The compression cache's compute_frozen_count returns 0 (no cached
# compressions yet), but mark_stable marks previous turns as frozen
# to preserve prefix cache stability.
assert captured["frozen_message_count"] >= 0
def test_cache_mode_restores_frozen_prefix_if_transform_mutates_history(monkeypatch) -> None:
_force_compression(monkeypatch)
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "cache"
proxy.config.image_optimize = False
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
original_messages = [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "current turn"},
]
# Mid-session: the first two messages were already forwarded last turn,
# so they form the byte-stable cached prefix the handler must replay
# even if a transform tries to mutate them.
fake_tracker._last_original_messages = original_messages[:2]
fake_tracker._last_forwarded_messages = original_messages[:2]
def _fake_apply(**kwargs):
mutated = list(kwargs["messages"])
mutated[0] = {**mutated[0], "content": "MUTATED_PREFIX"}
return SimpleNamespace(
messages=mutated,
transforms_applied=["fake:mutated"],
timing={},
tokens_before=80,
tokens_after=70,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msg_cache_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 70,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": original_messages,
},
)
assert response.status_code == 200
sent_messages = captured["body"]["messages"]
assert sent_messages[0] == original_messages[0]
assert sent_messages[1] == original_messages[1]
def test_cache_mode_cold_start_forwards_pipeline_rewrites(monkeypatch) -> None:
_force_compression(monkeypatch)
# Issue #2357: on a true session cold start (no prior forwarded messages,
# no frozen prefix) there is no provider cache to protect, so the full
# pipeline output is forwarded and recorded for byte-identical replay.
captured = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "cache"
proxy.config.image_optimize = False
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
original_messages = [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "current turn"},
]
def _fake_apply(**kwargs):
mutated = list(kwargs["messages"])
mutated[2] = {**mutated[2], "content": "REWRITTEN_CURRENT_TURN"}
return SimpleNamespace(
messages=mutated,
transforms_applied=["fake:mutated-latest"],
timing={},
tokens_before=80,
tokens_after=60,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msg_cache_2",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 80,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": original_messages,
},
)
assert response.status_code == 200
sent_messages = captured["body"]["messages"]
assert sent_messages[:2] == original_messages[:2]
assert sent_messages[2]["content"] == "REWRITTEN_CURRENT_TURN"
# Recorded as forwarded so later turns replay this prefix verbatim
# (the tracker may append the assistant reply after the forwarded turns).
assert fake_tracker._last_forwarded_messages[: len(sent_messages)] == sent_messages
def test_cache_mode_reuses_prior_forwarded_prefix_and_compresses_only_new_suffix() -> None:
captured = {"calls": []}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "cache"
proxy.config.image_optimize = False
tracker = _FakePrefixTracker(frozen_count=0)
tracker._last_original_messages = [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "turn2"},
{"role": "assistant", "content": "turn2-assistant"},
]
tracker._last_forwarded_messages = [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "COMPRESSED_TURN2"},
{"role": "assistant", "content": "turn2-assistant"},
]
tracker.get_last_original_messages = lambda: tracker._last_original_messages.copy()
tracker.get_last_forwarded_messages = lambda: tracker._last_forwarded_messages.copy()
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
def _fake_apply(**kwargs):
captured["calls"].append(kwargs["messages"])
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
# fix-6 contract: the compressor is handed the frozen forwarded prefix
# + the new delta and only compresses indices >= frozen_message_count
# (so a lone tool_result can resolve its tool_name from the prefix).
# Mirror the real router: pass the frozen prefix through verbatim and
# compress only the tail — the handler splices result.messages[prefix_n:].
fz = kwargs.get("frozen_message_count") or 0
msgs = kwargs["messages"]
return SimpleNamespace(
messages=list(msgs[:fz]) + [{"role": "user", "content": "COMPRESSED_TURN3"}],
transforms_applied=["fake:delta"],
timing={},
tokens_before=40,
tokens_after=20,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msg_cache_3",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 80,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "turn2"},
{"role": "assistant", "content": "turn2-assistant"},
{"role": "user", "content": "turn3"},
],
},
)
assert response.status_code == 200
# fix-6 contract: the compressor receives the frozen FORWARDED prefix
# (with COMPRESSED_TURN2, the byte-stable cached form) + the raw new
# delta (turn3), so tool_name resolution / dedup stay consistent with
# what is actually cached. frozen_message_count = prefix length pins
# compression to the delta ONLY — the prefix is never re-compressed.
assert captured["calls"] == [
[
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "COMPRESSED_TURN2"},
{"role": "assistant", "content": "turn2-assistant"},
{"role": "user", "content": "turn3"},
]
]
assert captured["frozen_message_count"] == 4 # only the delta (turn3) is compressed
# Forwarded body = byte-identical cached prefix + the compressed delta.
assert captured["body"]["messages"] == [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "COMPRESSED_TURN2"},
{"role": "assistant", "content": "turn2-assistant"},
{"role": "user", "content": "COMPRESSED_TURN3"},
]
def test_cache_mode_delta_keeps_newest_user_prompt_verbatim() -> None:
"""The cache-mode delta call must tell the router the prefix is replayed.
Without ``prefix_replay_guaranteed`` the router treats the newest user
message as compressible text, so under the coding profile (which turns on
user-message compression) a follow-up prompt -- e.g. a Claude Code
teammate's ``<teammate-message>`` -- reached the model lossily rewritten
on every turn after the first (#1174). Every other Anthropic
``pipeline.apply`` call already passes the flag.
"""
captured: dict = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "cache"
proxy.config.image_optimize = False
tracker = _FakePrefixTracker(frozen_count=0)
tracker._last_original_messages = [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
]
tracker._last_forwarded_messages = list(tracker._last_original_messages)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
def _fake_apply(**kwargs):
captured.update(kwargs)
return SimpleNamespace(
messages=list(kwargs["messages"]),
transforms_applied=[],
timing={},
tokens_before=10,
tokens_after=10,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "msg_delta_prompt",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {"input_tokens": 10, "output_tokens": 1},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "<teammate-message>next task</teammate-message>"},
],
},
)
assert response.status_code == 200
assert captured["frozen_message_count"] == 2 # the delta path ran
assert captured.get("prefix_replay_guaranteed") is True
def test_anthropic_handler_anchors_a_proven_rewritten_tail_to_stable_blocks() -> None:
"""The real handler must feed last turn's bytes into normalization."""
bodies = []
def _history(turn: int, churn: int) -> list[dict]:
content = [{"type": "text", "text": f"stable-{index}"} for index in range(30)]
content.extend(
{"type": "text", "text": f"turn-{turn}-changing-{index}"} for index in range(churn)
)
content.extend(
[
{"type": "text", "text": "instruction: summarize"},
{"type": "text", "text": "fixed end-of-transcript reminder"},
]
)
return [{"role": "user", "content": content}]
with _make_proxy_client() as client:
proxy = client.app.state.proxy
# Breakpoint ownership and lineage tracking apply even in passthrough
# mode. Keeping optimization off isolates that real handler wiring from
# the compression pipeline.
proxy.config.optimize = False
proxy.config.image_optimize = False
# This regression models a client whose next request replaces the same
# aggregate message. Do not synthesize an assistant history entry in
# the tracker, because that is a separate response-reconstruction path.
proxy._assistant_message_from_response_json = lambda _body: None
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
bodies.append(copy.deepcopy(body))
return httpx.Response(
200,
json={
"id": "msg_rewrite",
"type": "message",
"role": "assistant",
"content": [],
"usage": {
"input_tokens": 10_000,
"output_tokens": 1,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 10_000,
},
},
)
proxy._retry_request = _fake_retry
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
for turn, churn in enumerate((3, 5, 8), start=1):
response = client.post(
"/v1/messages",
headers=headers,
json={
"model": "claude-sonnet-4-6",
"max_tokens": 32,
"messages": _history(turn, churn),
},
)
assert response.status_code == 200
breakpoint_indices = []
for body in bodies:
marked = [
index
for index, block in enumerate(body["messages"][0]["content"])
if "cache_control" in block
]
assert len(marked) == 1
breakpoint_indices.append(marked[0])
# Cold request caches through its newest block. Once rewrite is proven,
# all subsequent calls pin the same 30-block boundary instead of creating
# an ever-growing full write on each turn.
assert breakpoint_indices == [34, 29, 29]
def test_cache_mode_skips_same_message_append_rewrite_to_preserve_stability() -> None:
captured = {"calls": []}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "cache"
proxy.config.image_optimize = False
tracker = _FakePrefixTracker(frozen_count=0)
tracker._last_original_messages = [
{"role": "user", "content": "shared-prefix"},
]
tracker._last_forwarded_messages = [
{"role": "user", "content": "COMPRESSED_PREFIX"},
]
tracker.get_last_original_messages = lambda: tracker._last_original_messages.copy()
tracker.get_last_forwarded_messages = lambda: tracker._last_forwarded_messages.copy()
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
def _fake_apply(**kwargs):
captured["calls"].append(kwargs["messages"])
return SimpleNamespace(
messages=[{"role": "user", "content": " + COMPRESSED_SUFFIX"}],
transforms_applied=["fake:suffix"],
timing={},
tokens_before=20,
tokens_after=10,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
captured["body"] = body
return httpx.Response(
200,
json={
"id": "msg_cache_suffix",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 80,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [
{"role": "user", "content": "shared-prefix + raw suffix"},
],
},
)
assert response.status_code == 200
assert captured["calls"] == []
assert captured["body"]["messages"] == [
{"role": "user", "content": "shared-prefix + raw suffix"},
]
# ─── Issue #327 regression tests ─────────────────────────────────────────────
#
# Lock down the post-fix invariant: the Anthropic token-mode handler must
# never extend `frozen_message_count` past the smaller of
# `prefix_tracker.frozen_message_count` and `compute_frozen_count(messages)`.
# The deleted walker (anthropic.py:756-787 pre-fix) advanced past those bounds
# whenever a fresh tool_result's content-hash matched any prior `_stable_hashes`
# entry or the TTL deferral fired. That conflated content equality with
# positional cache membership; for SvenMeyer's reported session it forced 73%
# of requests into `transforms_applied=[]` even though the corresponding byte
# positions were not actually in Anthropic's prefix cache.
class _IssueFakeCompCache:
"""Mock CompressionCache supporting the post-fix surface.
Records calls so tests can assert which methods fire and in what order.
Provides a populated `_stable_hashes` set for tests that need to prove
`_stable_hashes` membership no longer pushes `frozen_message_count` past
the prefix-tracker bound.
"""
def __init__(self, frozen_via_compute: int = 0, prepopulated_hashes: set | None = None):
self._frozen_via_compute = frozen_via_compute
self._stable_hashes: set[str] = set(prepopulated_hashes or set())
self._cache: dict = {}
self.calls: list[tuple[str, tuple, dict]] = []
self.applied_cached_with: list = []
def apply_cached(self, messages): # noqa: ANN001
self.calls.append(("apply_cached", (), {}))
self.applied_cached_with = list(messages)
return list(messages)
def compute_frozen_count(self, messages): # noqa: ANN001
self.calls.append(("compute_frozen_count", (), {}))
return self._frozen_via_compute
def mark_stable_from_messages(self, messages, up_to): # noqa: ANN001
self.calls.append(("mark_stable_from_messages", (up_to,), {}))
def update_from_result(self, originals, compressed): # noqa: ANN001
self.calls.append(("update_from_result", (), {}))
# Methods that should NOT be called in the post-fix code path. If any of
# these fire, the walker has resurrected.
def should_defer_compression(self, *args, **kwargs): # noqa: ANN001, ANN002, ANN003
self.calls.append(("should_defer_compression", args, kwargs))
return False
def mark_stable(self, content_hash): # noqa: ANN001
self.calls.append(("mark_stable", (content_hash,), {}))
@staticmethod
def content_hash(content): # noqa: ANN001
# Deterministic hash so prepopulated_hashes works in tests.
if isinstance(content, str):
return f"H({content[:40]})"
return f"H(list:{len(content)})"
def _make_optimize_proxy_client(mode: str = "token") -> TestClient:
"""Build a proxy client wired for optimization (issue-#327 tests)."""
config = ProxyConfig(
optimize=True,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
mode=mode,
)
app = create_app(config)
return TestClient(app)
def _drive_request(
client: TestClient,
*,
fake_comp_cache: _IssueFakeCompCache,
prefix_tracker_frozen: int,
messages: list,
captured: dict,
) -> httpx.Response:
"""Common test driver — wire fakes and submit a /v1/messages request."""
proxy = client.app.state.proxy
fake_tracker = _FakePrefixTracker(frozen_count=prefix_tracker_frozen)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"issue-327-session"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda session_id: fake_comp_cache
def _fake_apply(**kwargs): # noqa: ANN003
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
captured["pipeline_messages"] = list(kwargs["messages"])
# Record the byte-shape of the frozen prefix (deep snapshot via repr —
# tests below assert byte-stability with input).
captured["frozen_prefix_repr"] = repr(
list(kwargs["messages"])[: kwargs.get("frozen_message_count", 0)]
)
return SimpleNamespace(
messages=list(kwargs["messages"]),
transforms_applied=[],
timing={},
tokens_before=100,
tokens_after=100,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "msg_327",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 100,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
return client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": messages,
},
)
def _build_messages_with_repeat(repeat_at_idx: int = None) -> list: # noqa: ANN001
"""Build a 20-message session ending in a fresh tool_result.
If `repeat_at_idx` is given, the tool_result at that index has the same
content as the LAST tool_result, so its hash collides with `_stable_hashes`.
"""
msgs: list = []
for turn in range(7):
msgs.append({"role": "user", "content": f"turn-{turn}-user-question"})
if turn % 2 == 0:
msgs.append({"role": "assistant", "content": f"turn-{turn}-assistant"})
else:
msgs.append(
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"id": f"toolu_{turn}",
"name": "lookup",
"input": {"q": str(turn)},
}
],
}
)
msgs.append(
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": f"toolu_{turn}",
"content": f"unique-fresh-tool-output-for-turn-{turn}-AAAAA" * 20,
}
],
}
)
# msgs is currently length 17. Add a final tool_result at index 17.
fresh_content = "FINAL-FRESH-tool-output-content-XXXXX" * 30
msgs.append(
{
"role": "assistant",
"content": [
{"type": "tool_use", "id": "toolu_final", "name": "lookup", "input": {"q": "f"}}
],
}
)
msgs.append(
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_final",
"content": fresh_content,
}
],
}
)
if repeat_at_idx is not None:
# Overwrite the tool_result at repeat_at_idx with `fresh_content` so
# its hash collides with the final message's hash.
target = msgs[repeat_at_idx]
if isinstance(target.get("content"), list):
for blk in target["content"]:
if isinstance(blk, dict) and blk.get("type") == "tool_result":
blk["content"] = fresh_content
break
return msgs
def test_issue_327_walker_removed_does_not_advance_past_prefix_tracker() -> None:
"""frozen_message_count must equal min(prefix_tracker, compute_frozen_count).
The deleted walker (anthropic.py:766-787 pre-fix) advanced past
`prefix_tracker.frozen_message_count` whenever upcoming tool_results had
hashes in `_stable_hashes` or returned True from `should_defer_compression`.
Even with `_stable_hashes` populated with 50 entries, the post-fix code
must clamp to the smaller of the two positional sources.
"""
captured: dict = {}
with _make_optimize_proxy_client(mode="token") as client:
prepopulated = {f"H(synthetic-old-content-{i})" for i in range(50)}
fake_cache = _IssueFakeCompCache(frozen_via_compute=8, prepopulated_hashes=prepopulated)
response = _drive_request(
client,
fake_comp_cache=fake_cache,
prefix_tracker_frozen=15, # bigger than compute_frozen_count
messages=_build_messages_with_repeat(),
captured=captured,
)
assert response.status_code == 200
# Post-fix invariant: clamped to min(15, 8) = 8.
assert captured["frozen_message_count"] == 8, (
f"Expected frozen_message_count=8 (min of prefix_tracker=15 and "
f"compute_frozen_count=8); got {captured['frozen_message_count']}. "
f"If this is higher, the deleted walker has resurrected."
)
# The walker functions must not have been called.
method_names = [c[0] for c in fake_cache.calls]
assert "should_defer_compression" not in method_names, (
f"should_defer_compression was called from production handler; calls={fake_cache.calls}"
)
assert "mark_stable" not in method_names, (
f"mark_stable was called as walker side-effect; calls={fake_cache.calls}"
)
def test_issue_327_repeated_content_new_position_is_not_frozen() -> None:
"""A fresh tool_result whose content-hash matches an old `_stable_hashes`
entry must NOT be frozen on its new position. Pre-fix: walker would skip
past it on hash equality. Post-fix: only positional bounds matter."""
captured: dict = {}
with _make_optimize_proxy_client(mode="token") as client:
# Pre-populate _stable_hashes with the hash of the trailing tool_result.
# Since _IssueFakeCompCache.content_hash is deterministic on the first
# 40 chars of the string, pre-seed the same key.
fresh_content = "FINAL-FRESH-tool-output-content-XXXXX" * 30
prepopulated = {_IssueFakeCompCache.content_hash(fresh_content)}
fake_cache = _IssueFakeCompCache(frozen_via_compute=8, prepopulated_hashes=prepopulated)
response = _drive_request(
client,
fake_comp_cache=fake_cache,
prefix_tracker_frozen=8,
messages=_build_messages_with_repeat(repeat_at_idx=8),
captured=captured,
)
assert response.status_code == 200
# Pipeline got everything from index 8 onward — including the trailing
# repeat-content tool_result at index 18. Post-fix, frozen_message_count
# is exactly 8 regardless of any hash matches in _stable_hashes.
assert captured["frozen_message_count"] == 8
def test_issue_327_pipeline_preserves_frozen_prefix_byte_for_byte() -> None:
"""Invariant: messages[:frozen_message_count] passed to the pipeline are
byte-identical to the messages received from the client (modulo the
`apply_cached` swap, which is byte-stable). Lock the cache-floor."""
captured: dict = {}
with _make_optimize_proxy_client(mode="token") as client:
fake_cache = _IssueFakeCompCache(frozen_via_compute=10)
msgs = _build_messages_with_repeat()
response = _drive_request(
client,
fake_comp_cache=fake_cache,
prefix_tracker_frozen=10,
messages=msgs,
captured=captured,
)
assert response.status_code == 200
# apply_cached returned `list(messages)` (no swap) so the prefix should be
# byte-equal to the input.
frozen_prefix = captured["pipeline_messages"][: captured["frozen_message_count"]]
assert frozen_prefix == msgs[: captured["frozen_message_count"]], (
"Frozen prefix mutated between client request and pipeline call — "
"this would bust Anthropic's prefix cache."
)
def test_issue_327_multi_turn_session_compresses_each_turns_tail() -> None:
"""Simulate a 10-turn loop and assert that each turn the pipeline gets a
suffix of the messages to compress (frozen_message_count < len(messages)).
Pre-fix, after a few turns of accumulation, the walker would advance
`frozen_message_count` to `len(messages)` and the pipeline would get an
empty suffix → transforms_applied=[] on every turn (the SvenMeyer
fingerprint).
"""
frozen_per_turn: list = []
suffix_size_per_turn: list = []
with _make_optimize_proxy_client(mode="token") as client:
# Same comp_cache shared across all turns — `_stable_hashes` accumulates.
fake_cache = _IssueFakeCompCache()
for turn in range(10):
captured: dict = {}
# Each turn: prefix_tracker advances by 2 (one assistant + one new
# tool_result observed last turn). compute_frozen_count returns
# the same value to simulate "local cache covers what tracker says".
prefix_tracker_frozen = max(0, turn * 2 - 1)
fake_cache._frozen_via_compute = prefix_tracker_frozen
msgs = _build_messages_with_repeat()
# Append turn-specific filler so each request has a different shape.
for t in range(turn):
msgs.append({"role": "user", "content": f"continuation-{t}"})
response = _drive_request(
client,
fake_comp_cache=fake_cache,
prefix_tracker_frozen=prefix_tracker_frozen,
messages=msgs,
captured=captured,
)
assert response.status_code == 200
frozen_per_turn.append(captured["frozen_message_count"])
suffix_size_per_turn.append(
len(captured["pipeline_messages"]) - captured["frozen_message_count"]
)
# Every turn the pipeline must see a non-empty suffix to compress.
# Pre-fix, this would be 0 for most turns (the SvenMeyer 73%-frozen).
empty_suffix_turns = [i for i, s in enumerate(suffix_size_per_turn) if s == 0]
assert len(empty_suffix_turns) == 0, (
f"{len(empty_suffix_turns)}/10 turns had empty suffix to compress; "
f"sizes={suffix_size_per_turn}, frozen={frozen_per_turn}. "
f"Pre-fix walker behavior detected."
)
def test_issue_327_streaming_and_non_streaming_compute_same_frozen_count() -> None:
"""The optimization path is upstream of the stream/non-stream branch.
Whatever `frozen_message_count` token-mode produces for `stream=False`
must match what it produces for `stream=True` on identical inputs.
Locks the safety property that streaming has no separate, divergent
walker logic.
"""
msgs = _build_messages_with_repeat()
# Run 1: stream=False
captured_a: dict = {}
with _make_optimize_proxy_client(mode="token") as client:
fake_cache_a = _IssueFakeCompCache(frozen_via_compute=10)
proxy = client.app.state.proxy
fake_tracker = _FakePrefixTracker(frozen_count=12)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stream-parity-A"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda s: fake_cache_a
def _fake_apply_a(**kwargs): # noqa: ANN003
captured_a["frozen_message_count"] = kwargs.get("frozen_message_count")
return SimpleNamespace(
messages=list(kwargs["messages"]),
transforms_applied=[],
timing={},
tokens_before=100,
tokens_after=100,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply_a
async def _fake_retry_a(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "msg",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 100,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry_a
r_a = client.post(
"/v1/messages",
headers={"x-api-key": "test", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"stream": False,
"messages": msgs,
},
)
assert r_a.status_code == 200
# Run 2: stream=True (same inputs)
captured_b: dict = {}
with _make_optimize_proxy_client(mode="token") as client:
fake_cache_b = _IssueFakeCompCache(frozen_via_compute=10)
proxy = client.app.state.proxy
fake_tracker = _FakePrefixTracker(frozen_count=12)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stream-parity-B"
)
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda s: fake_cache_b
def _fake_apply_b(**kwargs): # noqa: ANN003
captured_b["frozen_message_count"] = kwargs.get("frozen_message_count")
return SimpleNamespace(
messages=list(kwargs["messages"]),
transforms_applied=[],
timing={},
tokens_before=100,
tokens_after=100,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply_b
# Streaming path: return a minimal SSE body
sse_body = (
b"event: message_start\n"
b'data: {"type":"message_start","message":{"id":"msg",'
b'"role":"assistant","content":[],"model":"claude",'
b'"usage":{"input_tokens":100,"output_tokens":0,'
b'"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}\n\n'
b"event: content_block_start\n"
b'data: {"type":"content_block_start","index":0,'
b'"content_block":{"type":"text","text":""}}\n\n'
b"event: content_block_stop\n"
b'data: {"type":"content_block_stop","index":0}\n\n'
b"event: message_delta\n"
b'data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},'
b'"usage":{"output_tokens":3}}\n\n'
b"event: message_stop\n"
b'data: {"type":"message_stop"}\n\n'
)
async def _fake_retry_b(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
content=sse_body,
headers={"content-type": "text/event-stream"},
)
proxy._retry_request = _fake_retry_b
# Streaming dispatch uses _stream_response (not _retry_request). Stub
# it to a no-op streaming response so we can inspect what
# pipeline.apply received without being responsible for the SSE
# plumbing — the optimization runs before _stream_response is called.
from fastapi.responses import StreamingResponse
async def _fake_stream_response(*args, **kwargs): # noqa: ANN001, ANN002, ANN003
async def _gen():
yield b""
return StreamingResponse(_gen(), media_type="text/event-stream")
proxy._stream_response = _fake_stream_response
client.post(
"/v1/messages",
headers={"x-api-key": "test", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"stream": True,
"messages": msgs,
},
)
# Status code is incidental — what matters is that pipeline.apply ran
# and captured_b was populated.
assert "frozen_message_count" in captured_a, "Non-streaming path didn't reach pipeline.apply()"
assert "frozen_message_count" in captured_b, "Streaming path didn't reach pipeline.apply()"
assert captured_a["frozen_message_count"] == captured_b["frozen_message_count"], (
f"Streaming/non-streaming divergence: stream=False produced "
f"frozen={captured_a['frozen_message_count']}, stream=True produced "
f"{captured_b['frozen_message_count']}. The optimization path must "
f"be identical for both."
)