mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Closes #4442. pi's system `<cwd>` block now resolves the memory project. If project scope is still unresolved with the `empty` fallback, the Anthropic handler no longer advertises unusable memory tools, including tools recorded earlier in the session. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Parse pi cwd only in system text. Keep main's line parser, header/CLI precedence and limited user-message fallback. - Reuse `is_project_unresolved` for context and sticky-tool replay. Client tools and global/user storage behavior stay intact. - Add initialized-handler scope regressions, replace source-inspection/forwarding-only checks, and document project scope in the published memory page. ## Testing - [x] Unit tests pass (`pytest`) — related suite and the normal critical gate, not the entire Python suite locally. - [x] Linting passes (`ruff check .`) — hosted lint passed on 5ace63e2; local filesystem limitation below. - [x] Type checking passes (`mypy headroom`) — normal full hook with its existing `--ignore-missing-imports` setting. - [x] New tests added for the scope regression - [x] Manual testing performed ### Test Output ```text # 16 related test files 329 passed, 12 skipped, 1 warning in 43.07s # ordinary pre-push: make ci-precheck 178 passed, 5 skipped, 1 warning in 10.87s ✅ ci-precheck PASSED — safe to push. uv run ruff format --check . 1860 files already formatted ``` Normal commit hooks (Ruff, formatter, full mypy, commitlint) and the complete pre-push path passed: Rust fmt/clippy/workspace tests, native build/import verification, critical Python tests and commitlint. The existing Starlette test-client warning was not filtered. Local whole-tree Ruff reports EXE002 on five **unchanged** OAuth plugin files: my `fuseblk` checkout exposes them as executable despite Git modes `100644`; `chmod` returns zero without changing the mode. The uncached check agrees. I did not change those files, add shebangs or ignore the rule. Hosted Ruff, formatting and full mypy passed on 5ace63e2; the local filesystem limitation remains. The first full CI caught a missing scope predicate on the existing lifecycle test double. Fixed in [5ace63e2](https://github.com/headroomlabs-ai/headroom/commit/5ace63e2cbd2295f9594d1e0442a679aad99979c); its complete file passes (`5 passed, 1 warning in 13.01s`). This follow-up changes no production code or assertions and adds no runtime fallback. The behavior proof below ran on7aeb83cfwith the same production source; that qualification is historical; current-head checks are recorded below. ### Author follow-up:80453a03Moved the project-scope guidance out of the unpublished wiki and into `docs/content/docs/memory.mdx`; this commit changes no production code or tests. Viewed the actual Next.js `/docs/memory#proxy-project-scope` page and its table-of-contents entry; no browser errors. The existing router/scope tests pass: 65 passed, with four SQLite ResourceWarnings. All seven pi-parser lines and all six branch edges are covered, so I did not add duplicate tests for the old Codecov percentage. Normal commit and pre-push hooks passed, including Rust checks, native build/import, 180 critical Python tests (5 skipped, 1 tokenizer-approximation warning), and commitlint. My first push timed out with the wrong local ONNX path; the next found missing pip in the fresh uv environment. Both setup issues were fixed without bypassing checks. Current-head hosted CI is not yet qualified. ## Real Behavior Proof - Environment: Linux x86-64, Python 3.13.14, freshly built native core, real TCP/Uvicorn proxy and SQLite/ONNX memory backend. Only the upstream is a local Anthropic-shaped HTTP server; no provider charges. - Exact command / steps: after `uv sync --locked --extra dev --extra proxy --extra memory` and `make verify-rust-core`, run this from the checkout. It prints the actual loaded native module/hash and leaves a disposable `.headroom-4442-*` directory for inspecting SQLite. Both HTTP listeners shut down before exit. <details> <summary>Standalone HTTP/SQLite reproduction</summary> ```bash ./.venv/bin/python - <<'PY' import hashlib,http.client,json,os,socket,tempfile,threading,time from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer from pathlib import Path import uvicorn import headroom._core as core assert Path(core.__file__).resolve().parent == Path.cwd().resolve()/"headroom" native_sha=hashlib.sha256(Path(core.__file__).read_bytes()).hexdigest() from headroom.memory.storage_router import ProjectResolver,RequestContext,extract_system_prompt from headroom.proxy.server import ProxyConfig,create_app from headroom.proxy.memory_handler import MEMORY_TOOL_NAMES from headroom.proxy.helpers import get_session_tool_tracker root=Path(tempfile.mkdtemp(prefix=".headroom-4442-",dir=Path.cwd())) project_a=root/"pi%2Fproject-a";project_b=root/"project-b" project_a.mkdir(exist_ok=True);project_b.mkdir(exist_ok=True) os.environ["HEADROOM_WORKSPACE_DIR"]=str(root/"workspace") os.environ["HEADROOM_BEACON"]="off" os.environ["HEADROOM_TELEMETRY"]="off" os.environ["NO_PROXY"]="127.0.0.1,localhost" forwarded=[] class Gateway(BaseHTTPRequestHandler): def do_POST(self): body=json.loads(self.rfile.read(int(self.headers["Content-Length"]))) forwarded.append({"path":self.path,"body":body}) payload=json.dumps({"id":"msg_local_scope","type":"message","role":"assistant","model":"claude-haiku-4-5","content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn","stop_sequence":None,"usage":{"input_tokens":10,"output_tokens":1}}).encode() self.send_response(200);self.send_header("Content-Type","application/json");self.send_header("Content-Length",str(len(payload)));self.end_headers();self.wfile.write(payload) def log_message(self,*args):pass upstream=ThreadingHTTPServer(("127.0.0.1",0),Gateway) ut=threading.Thread(target=upstream.serve_forever,daemon=True);ut.start() config=ProxyConfig(mode="cache",anthropic_api_url=f"http://127.0.0.1:{upstream.server_port}",memory_enabled=True,memory_db_path=str(root/"legacy.db"),traffic_learning_enabled=True,subscription_tracking_enabled=False,cost_tracking_enabled=False,cache_enabled=False,rate_limit_enabled=False,log_requests=False,image_optimize=False,ccr_inject_tool=False,ccr_handle_responses=False,ccr_context_tracking=False,anthropic_pre_upstream_memory_context_timeout_seconds=30.0) app=create_app(config) sock=socket.socket();sock.bind(("127.0.0.1",0));sock.listen(128) port=sock.getsockname()[1] server=uvicorn.Server(uvicorn.Config(app,log_level="warning",lifespan="on",timeout_graceful_shutdown=3)) st=threading.Thread(target=server.run,kwargs={"sockets":[sock]},daemon=True);st.start() try: deadline=time.monotonic()+180 while not server.started: if not st.is_alive() or time.monotonic()>deadline:raise RuntimeError("real proxy startup failed") time.sleep(.02) session="headroom4442-native-project-scope" common={"Content-Type":"application/json","x-api-key":"local-unbilled-key","anthropic-version":"2023-06-01","x-headroom-user-id":"native-scope-user","x-headroom-session-id":session} client_tools=[{"name":"get_weather","description":"Weather","input_schema":{"type":"object","properties":{}}}] rows=[] cases=[("fresh_unresolved","You are helpful.",{},False),("pi_cwd",f"<cwd>\n{project_a}\n</cwd>",{},True),("warm_unresolved","You are helpful.",{},False),("pi_cwd_restored",f"<cwd>\n{project_a}\n</cwd>",{},True),("explicit_header_wins",f"<cwd>\n{project_b}\n</cwd>",{"x-headroom-project-id":"explicit-project"},True),("ordinary_user_cwd_ignored","You are helpful.",{},False)] for label,system,extra,expected in cases: body={"model":"claude-haiku-4-5","max_tokens":16,"system":system,"messages":[{"role":"user","content":f"cwd:\n<cwd>\n{project_b}\n</cwd>" if label=="ordinary_user_cwd_ignored" else "hello scope"}],"tools":client_tools} before=len(forwarded) conn=http.client.HTTPConnection("127.0.0.1",port,timeout=90) conn.request("POST","/v1/messages",body=json.dumps(body).encode(),headers={**common,**extra}) response=conn.getresponse();payload=response.read();conn.close() assert response.status==200,(label,response.status,payload.decode()) assert len(forwarded)==before+1,(label,"unexpected upstream request count") actual=forwarded[-1]["body"] names={tool.get("name") for tool in actual.get("tools",[])} assert names==({"get_weather"}|MEMORY_TOOL_NAMES if expected else {"get_weather"}),(label,names) scope=ProjectResolver().resolve(RequestContext(headers={**common,**extra},system_prompt=extract_system_prompt(body),base_user_id="native-scope-user")) rows.append({"case":label,"status":response.status,"memoryTools":sorted(names & MEMORY_TOOL_NAMES),"project":scope,"gatewayPath":forwarded[-1]["path"]}) assert forwarded[1]["body"]["tools"]==forwarded[3]["body"]["tools"],"restored definitions changed" tracker=get_session_tool_tracker() assert tracker.should_inject("anthropic",session),"probe did not exercise an actual recorded sticky session" from urllib.parse import quote key_a=ProjectResolver().resolve(RequestContext(headers={"x-headroom-cwd":quote(str(project_a),safe="/")},system_prompt="",base_user_id="native-scope-user"))[0] assert rows[1]["project"][0]==key_a assert rows[4]["project"][1]=="explicit-project" router=app.state.proxy.memory_handler._router assert router is not None,"real memory backend never initialized" opened=router.open_backends() project_db=root/"memories"/"projects"/key_a/"memory.db" assert project_db in opened and project_db.is_file(),("real pi project database not opened",[str(x) for x in opened]) print(json.dumps({"mode":config.mode,"realHttpListener":f"http://127.0.0.1:{port}","localGateway":f"http://127.0.0.1:{upstream.server_port}","rows":rows,"realStickyGoldenSession":True,"realPiProjectDatabase":str(project_db),"openedDatabases":[str(x) for x in opened],"nativeModule":core.__file__,"nativeSha256":native_sha,"noLivePiOrPaidProvider":True},indent=2)) finally: server.should_exit=True;st.join(20) upstream.shutdown();upstream.server_close();ut.join(5);sock.close() assert not st.is_alive() and not ut.is_alive(),"native proof leaked a server" PY ``` </details> - Observed result: unchanged `main`fc1ba891fails at the first unresolved request: five proxy memory tools reach upstream. With the fix, all six requests return HTTP 200: | Same-session request | Proxy memory tools | | --- | ---: | | Fresh, no cwd | 0 | | pi system cwd | 5 | | Cwd absent again | 0 | | Same pi cwd restored | 5 | | Explicit project header, different system cwd | 5 | | Cwd block only in ordinary user text | 0 | The client weather tool stays present. Restored definitions are structurally equal to the first resolved turn; the explicit header wins. The literal-percent cwd agrees with its encoded-header form, and the actual project SQLite database opens at the expected key. - Not tested: live pi/Caveman, paid-provider cache accounting/billing, Windows, or other provider protocols. This does not claim to fix Caveman header forwarding. ## Runtime Rollout Safety - Rollout-managed feature(s): none added; existing Anthropic memory injection. - Minimum rollout channel: unchanged. - Stable/default behavior changed: pi cwd resolves; unresolved project/empty scope no longer adds unusable proxy memory tools. - Kill switch / disable path: existing `--no-memory`; explicit project/cwd headers still work. - Unsafe override required: no. - Qualification impact: no storage migration or new dependency; scoped HTTP/SQLite behavior checked. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes — related suite and critical gate, not a whole-suite claim - [x] I did **not** edit `CHANGELOG.md` — release-please owns it ## Screenshots (if applicable) N/A; the observable surface is the HTTP payload and project database. ## Additional Notes AI-assisted; no subagents. OpenAI injection policy is unchanged. Keeping this draft until the current hosted checks are qualified. One earlier post-build probe completed the requests and closed both listeners but hit its whole-process shutdown deadline. The same program and native artifact later exited normally with faulthandler enabled; its diagnostic abort timer did not fire. I have not identified the intermittent cause and am not claiming a shutdown fix. No production timeout or feature flag was changed to get that result. The two npm audits now pass on5ace63e2, along with the other Security jobs. The earlier [unchanged-base failure](https://github.com/headroomlabs-ai/headroom/actions/runs/41399323584) is historical evidence, not a current blocker. This PR changes no dependency manifests, lockfiles, workflow or audit exceptions. ### Maintainer current-main requalification Exact head `60597623e00967153d947a46bf197950250f310e`, upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. Clean mechanical update from current upstream main, preserving the pi system-cwd resolver, request-scoped project guard, main's cache-mode search skip/logging, and client-tool preservation. Ordinary user cwd blocks still do not enable the pi fallback; unresolved project scope still blocks both context injection and replayed memory tools. No contributor production logic was rewritten and no new production finding remains in this reviewed scope. The initial combined tree dropped main's memory-invariant file and two stream/timeout regression cases. Maintainer test-only repairs restore those tests rather than reducing coverage: the invariant file is byte-identical to main, every main assertion in the four restored/updated test files is preserved by AST comparison, and three fixtures now supply the handler protocol methods needed for resolved scope and beta headers. The lifecycle main control passes without that fixture extension; no claim is made that this particular fixture failure independently fails main. All 222 affected scope/invariant/stream/timeout/lifecycle/backpressure/cache-stability/system-prompt tests pass with one skip. Ten shared Gemini request-fixture cases also pass after the separate test-only repair for independently verified main failures. All 12 main-relative changed Python files pass Ruff/format; whitespace checks pass. The author's existing real TCP/SQLite/ONNX reproduction and explicit limits were read and remain in the body. These fresh Windows checks qualify the combined request-scope regressions; they are not a claim of rerunning that Linux proof, live pi/client/provider behavior, or a fresh native build. An existing binary was supplied solely for test startup. Draft status is preserved; readiness remains the author's decision and nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `60e01cf7c25e90abb9f5cb66fa221da83e7c5177`, upstream main `ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`. Clean integration with upstream/mainea59ac3eef. Re-reviewed project-scoping delta against intervening memory/backend/stream changes: pi cwd remains limited to system context; ordinary user cwd blocks do not enable fallback; unresolved projects skip context/tool injection and sticky tool replay while retaining client tools and main cache-mode no-search behavior.232 scope/invariant/stream/timeout/lifecycle/backpressure/cache/system-prompt/Gemini tests pass,1optional skip; touched-source Ruff and whitespace pass. No contributor production policy rewritten. Earlier author TCP/SQLite/ONNX evidence remains scoped, with no fresh native build or live-pi/provider claim. Existing draft preserved; final approval awaits exact-head hosted qualification. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `e93ceb35df7f78bfa2dcc0611609131cf2b08b0a`, upstream main `c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`. Final post-Dependabot requalification against upstream/mainc01cf07d94. All six Dependabot updates are now merged. Relative to this branch's freshly tested prepared tree, the final main sync changes only the three intended Cargo.lock records; Python/source/test inputs are unchanged. Re-reviewed the integration and confirmed complete native source/lock/toolchain objects match the tested combined build and exact fresh native DLL used for this branch's local tests. Focused fresh-native qualification: 232 passed, 1 skipped, 1 warning in 47.62s. Selected real dependency overlays and native DLL paths are recorded in final4618-fresh-native-tests.log; the DLL provenance/input hashes are in final-native-artifact-proof.json. No redundant rerun is implied after this Cargo-only merge. Whitespace passes. Prior source review and scoped real-behavior evidence remain applicable to unchanged contributor inputs. No new source finding in the reviewed scope. Final formal approval awaits exact-head hosted qualification. Project resolution and unresolved-scope injection/replay gates remain unchanged. Existing draft status is preserved. Maintainers handle hosted checks; no external-provider claim, contributor semantic rewrite or non-Dependabot merge. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `d693dd2b6e33fd47ff853f7e28d938fc1dbc30ac`, upstream main `2b0315b2d4b0d54947050bbdad2de9e89df06378`. Current-main requalification against2b0315b2d4b0d54947050bbdad2de9e89df06378. Clean source integration preserves pi systemcwd resolution and unresolved-scope context/tool/sticky-replay gates.70integration patches exactly matchmain delta;Anthropic exception patch differs only byclosing-parenthesis diff alignment and its exception AST equalsmain. Broad memory/router/projectscope/continuation/backpressure/cache/lifecycle/doctor/nativeops run:405passed,1skipped,one newmain fake-handler fixture failed missingis_project_unresolved. Maintainer adds only that fake method returningFalse,respecting resolvedfixture scope;all39backpressure tests nowpass,including unchanged injectedSQLiteexception,HTTP200andsemaphorerelease assertions. No production logic changed.33changedPythonlint/format,offline lock/whitespacepass;fixedtestlint/formatrechecked. Existingpr3982nativeDLLverified fullinputhashes/checksum;no freshbuildclaimed. Currentauthor/Devin discussion checked,inline endpoint empty;prior scoped realbehavior evidence remains applicable to unchanged contributor inputs. No new sourcefinding;no realprovider/agentlaunch claim. Draftpreserved. Exactnewheadhostedqualification pending;maintainers handlechecks,nothingmerged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `0da0b5a01958eaaac1d625227cb4bdb9477c5d3b`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean integration with upstream/main0ddaa56c21. All 17 integration patches exactly match the prior-reviewed-main delta after diff-location normalization. Contributor pi system-cwd resolution and unresolved-project memory context/tool/sticky-replay gates are unchanged. Fresh memory-router/project-scope/streaming/continuation/backpressure/cache/lifecycle and incoming Vertex/backend/CLI tests: 381 passed, 12 skipped, 1 warning. Twelve changed Python files pass Ruff lint/format; offline lock and whitespace pass. Static production/test scope and security review considered user versus system cwd inputs, malformed/empty cwd, missing scope and stale tool replay: tests keep pi cwd tags in ordinary user content from resolving project scope, and unresolved scope blocks replay. No unrelated executable, workflow, dependency or setup changes. Flagged inherited workflow-validation job is excluded by unchanged workflow paths. Author discussion checked; collected inline endpoint has no findings. Prior scoped behavior evidence applies to unchanged contributor inputs; no new real agent/provider launch claimed. Native DLL reused only after complete tracked input-set/hash and artifact checksum verification, no fresh build claimed. Draft preserved. Formal approval awaits exact published-head hosted qualification. Maintainers handle checks; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
2065 lines
78 KiB
Python
2065 lines
78 KiB
Python
"""Regression tests for Anthropic prefix-cache stability in proxy mode."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import copy
|
|
from types import SimpleNamespace
|
|
from unittest.mock import AsyncMock
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
|
|
def _force_compression(monkeypatch) -> None: # noqa: ANN001
|
|
decision = SimpleNamespace(should_compress=True, passthrough_reason=None)
|
|
decision.apply_to_tags = lambda tags: None
|
|
monkeypatch.setattr(
|
|
"headroom.proxy.handlers.anthropic.CompressionDecision.decide",
|
|
lambda **kwargs: decision,
|
|
)
|
|
|
|
|
|
class _FakePrefixTracker:
|
|
def __init__(self, frozen_count: int):
|
|
self._frozen_count = frozen_count
|
|
self._cached_token_count = 0
|
|
self._last_original_messages = []
|
|
self._last_forwarded_messages = []
|
|
|
|
def get_frozen_message_count(self) -> int:
|
|
return self._frozen_count
|
|
|
|
def get_last_original_messages(self): # noqa: ANN201
|
|
return self._last_original_messages.copy()
|
|
|
|
def get_last_forwarded_messages(self): # noqa: ANN201
|
|
return self._last_forwarded_messages.copy()
|
|
|
|
def update_from_response(self, **kwargs): # noqa: ANN003
|
|
self._cached_token_count = kwargs.get("cache_read_tokens", 0) + kwargs.get(
|
|
"cache_write_tokens", 0
|
|
)
|
|
self._last_original_messages = kwargs.get(
|
|
"original_messages", kwargs.get("messages", [])
|
|
).copy()
|
|
self._last_forwarded_messages = kwargs.get("messages", []).copy()
|
|
return None
|
|
|
|
|
|
class _FakeImageCompressor:
|
|
def __init__(self):
|
|
self.last_result = None
|
|
|
|
def has_images(self, messages): # noqa: ANN001
|
|
return True
|
|
|
|
def compress(self, messages, provider="anthropic"): # noqa: ANN001
|
|
assert provider == "anthropic"
|
|
assert len(messages) == 1
|
|
msg = messages[0]
|
|
content = msg["content"]
|
|
updated_content = []
|
|
for block in content:
|
|
if isinstance(block, dict) and block.get("type") == "image":
|
|
src = block.get("source", {})
|
|
updated_content.append(
|
|
{
|
|
"type": "image",
|
|
"source": {**src, "data": "COMPRESSED_IMAGE_BYTES"},
|
|
}
|
|
)
|
|
else:
|
|
updated_content.append(block)
|
|
return [{**msg, "content": updated_content}]
|
|
|
|
|
|
def _make_proxy_client() -> TestClient:
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=True,
|
|
)
|
|
app = create_app(config)
|
|
return TestClient(app)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("optimize", "expected_names"),
|
|
[
|
|
(False, ["zeta", "alpha", "mu"]),
|
|
(True, ["alpha", "mu", "zeta"]),
|
|
],
|
|
)
|
|
def test_anthropic_tools_forwarding_order_matches_optimization_mode(
|
|
optimize: bool,
|
|
expected_names: list[str],
|
|
) -> None:
|
|
captured = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = optimize
|
|
proxy.config.mode = "token"
|
|
|
|
if optimize:
|
|
proxy.anthropic_pipeline.apply = lambda **kwargs: SimpleNamespace(
|
|
messages=kwargs["messages"],
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=100,
|
|
tokens_after=100,
|
|
waste_signals=None,
|
|
)
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 10,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 128,
|
|
"messages": [{"role": "user", "content": "hello"}],
|
|
"tools": [
|
|
{"name": "zeta", "description": "z", "input_schema": {"type": "object"}},
|
|
{"name": "alpha", "description": "a", "input_schema": {"type": "object"}},
|
|
{"name": "mu", "description": "m", "input_schema": {"type": "object"}},
|
|
],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
sent_tools = captured["body"]["tools"]
|
|
assert [t["name"] for t in sent_tools] == expected_names
|
|
|
|
|
|
def test_image_compression_only_applies_to_latest_non_frozen_user_turn() -> None:
|
|
fake_compressor = _FakeImageCompressor()
|
|
|
|
old_image = {
|
|
"type": "image",
|
|
"source": {"type": "base64", "media_type": "image/png", "data": "OLD_IMAGE_BYTES"},
|
|
}
|
|
new_image = {
|
|
"type": "image",
|
|
"source": {"type": "base64", "media_type": "image/png", "data": "NEW_IMAGE_BYTES"},
|
|
}
|
|
messages = [
|
|
{"role": "user", "content": [old_image, {"type": "text", "text": "old image turn"}]},
|
|
{"role": "assistant", "content": "ack"},
|
|
{"role": "user", "content": [new_image, {"type": "text", "text": "new image turn"}]},
|
|
]
|
|
|
|
result = AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
|
|
messages,
|
|
frozen_message_count=1,
|
|
compressor=fake_compressor,
|
|
)
|
|
|
|
# Frozen prefix must remain byte-identical.
|
|
assert result[0]["content"][0]["source"]["data"] == "OLD_IMAGE_BYTES"
|
|
# Latest non-frozen user turn is eligible for compression.
|
|
assert result[2]["content"][0]["source"]["data"] == "COMPRESSED_IMAGE_BYTES"
|
|
|
|
|
|
def test_image_compression_does_not_touch_previous_turns_if_last_message_not_user() -> None:
|
|
fake_compressor = _FakeImageCompressor()
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "image",
|
|
"source": {
|
|
"type": "base64",
|
|
"media_type": "image/png",
|
|
"data": "OLD_IMAGE_BYTES",
|
|
},
|
|
}
|
|
],
|
|
},
|
|
{"role": "assistant", "content": "last turn is assistant"},
|
|
]
|
|
result = AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
|
|
messages,
|
|
frozen_message_count=0,
|
|
compressor=fake_compressor,
|
|
)
|
|
assert result[0]["content"][0]["source"]["data"] == "OLD_IMAGE_BYTES"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("optimize", "expected_names"),
|
|
[
|
|
(False, ["zeta", "alpha", "mu"]),
|
|
(True, ["alpha", "mu", "zeta"]),
|
|
],
|
|
)
|
|
def test_anthropic_batch_tools_forwarding_order_matches_optimization_mode(
|
|
optimize: bool,
|
|
expected_names: list[str],
|
|
) -> None:
|
|
captured = {}
|
|
config = ProxyConfig(
|
|
optimize=optimize,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
)
|
|
app = create_app(config)
|
|
|
|
with TestClient(app) as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.mode = "token"
|
|
|
|
if optimize:
|
|
proxy.anthropic_pipeline.apply = lambda **kwargs: SimpleNamespace(
|
|
messages=kwargs["messages"],
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=100,
|
|
tokens_after=100,
|
|
waste_signals=None,
|
|
)
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msgbatch_1",
|
|
"type": "message_batch",
|
|
"processing_status": "in_progress",
|
|
"request_counts": {
|
|
"processing": 1,
|
|
"succeeded": 0,
|
|
"errored": 0,
|
|
"canceled": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages/batches",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"requests": [
|
|
{
|
|
"custom_id": "req-1",
|
|
"params": {
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 128,
|
|
"messages": [{"role": "user", "content": "hello"}],
|
|
"tools": [
|
|
{
|
|
"name": "zeta",
|
|
"description": "z",
|
|
"input_schema": {"type": "object"},
|
|
},
|
|
{
|
|
"name": "alpha",
|
|
"description": "a",
|
|
"input_schema": {"type": "object"},
|
|
},
|
|
{
|
|
"name": "mu",
|
|
"description": "m",
|
|
"input_schema": {"type": "object"},
|
|
},
|
|
],
|
|
},
|
|
}
|
|
]
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
sent_tools = captured["body"]["requests"][0]["params"]["tools"]
|
|
assert [t["name"] for t in sent_tools] == expected_names
|
|
|
|
|
|
def test_append_context_targets_latest_non_frozen_user_turn() -> None:
|
|
messages = [
|
|
{"role": "user", "content": "frozen prefix"},
|
|
{"role": "assistant", "content": "ack"},
|
|
{"role": "user", "content": "active turn"},
|
|
]
|
|
result = AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn(
|
|
messages,
|
|
"CTX",
|
|
frozen_message_count=1,
|
|
)
|
|
assert result[0]["content"] == "frozen prefix"
|
|
assert result[2]["content"].endswith("CTX")
|
|
|
|
|
|
def test_append_context_does_not_touch_previous_turns_if_last_message_not_user() -> None:
|
|
messages = [
|
|
{"role": "user", "content": "previous user turn"},
|
|
{"role": "assistant", "content": "assistant last"},
|
|
]
|
|
result = AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn(
|
|
messages,
|
|
"CTX",
|
|
frozen_message_count=0,
|
|
)
|
|
assert result[0]["content"] == "previous user turn"
|
|
|
|
|
|
def test_token_mode_freeze_is_capped_by_prefix_tracker() -> None:
|
|
captured = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "token"
|
|
proxy.config.image_optimize = False
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=1)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
|
|
class _FakeCompressionCache:
|
|
def apply_cached(self, messages): # noqa: ANN001
|
|
return messages
|
|
|
|
def compute_frozen_count(self, messages): # noqa: ANN001
|
|
return 99
|
|
|
|
def update_from_result(self, originals, compressed): # noqa: ANN001
|
|
return None
|
|
|
|
def mark_stable_from_messages(self, messages, up_to): # noqa: ANN001
|
|
pass
|
|
|
|
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache()
|
|
|
|
def _fake_apply(**kwargs):
|
|
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
|
|
return SimpleNamespace(
|
|
messages=kwargs["messages"],
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=50,
|
|
tokens_after=50,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_tc_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 50,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [{"role": "user", "content": "hello"}],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert captured["frozen_message_count"] == 1
|
|
|
|
|
|
def test_memory_context_avoids_system_mutation_when_prefix_frozen() -> None:
|
|
captured = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = False
|
|
proxy.config.image_optimize = False
|
|
proxy.config.ccr_proactive_expansion = False
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=1)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
|
|
proxy.memory_handler = SimpleNamespace(
|
|
config=SimpleNamespace(inject_context=True, inject_tools=False),
|
|
is_project_unresolved=lambda request_context: False,
|
|
search_and_format_context=AsyncMock(return_value="MEMCTX"),
|
|
has_memory_tool_calls=lambda resp, provider: False,
|
|
)
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_mem_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 20,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={
|
|
"x-api-key": "test-key",
|
|
"anthropic-version": "2023-06-01",
|
|
"x-headroom-user-id": "u1",
|
|
},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"system": "base system",
|
|
"messages": [
|
|
{"role": "user", "content": "frozen prefix"},
|
|
{"role": "assistant", "content": "ack"},
|
|
{"role": "user", "content": "latest user"},
|
|
],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
sent = captured["body"]
|
|
assert sent["system"] == "base system"
|
|
assert sent["messages"][2]["content"].endswith("MEMCTX")
|
|
|
|
|
|
def test_ccr_system_instruction_injection_disabled_when_prefix_frozen(monkeypatch) -> None:
|
|
captured = {"inject_system": None}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = False
|
|
proxy.config.image_optimize = False
|
|
proxy.config.ccr_inject_tool = False
|
|
proxy.config.ccr_inject_system_instructions = True
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=1)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
|
|
class _FakeInjector:
|
|
def __init__(
|
|
self,
|
|
provider, # noqa: ANN001
|
|
inject_tool, # noqa: ANN001
|
|
inject_system_instructions, # noqa: ANN001
|
|
):
|
|
captured["inject_system"] = inject_system_instructions
|
|
self.has_compressed_content = False
|
|
self.detected_hashes = []
|
|
|
|
def process_request(self, messages, tools): # noqa: ANN001
|
|
return messages, tools, False
|
|
|
|
def scan_for_markers(self, messages): # noqa: ANN001
|
|
return []
|
|
|
|
def verify_ownership(self, store=None): # noqa: ANN001
|
|
return self.detected_hashes
|
|
|
|
monkeypatch.setattr("headroom.ccr.CCRToolInjector", _FakeInjector)
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_ccr_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 20,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [{"role": "user", "content": "hello"}],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert captured["inject_system"] is False
|
|
|
|
|
|
def test_ccr_tool_injection_disabled_when_prefix_frozen(monkeypatch) -> None:
|
|
captured = {"inject_tool": None}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = False
|
|
proxy.config.image_optimize = False
|
|
proxy.config.ccr_inject_tool = True
|
|
proxy.config.ccr_inject_system_instructions = False
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=1)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
|
|
class _FakeInjector:
|
|
def __init__(
|
|
self,
|
|
provider, # noqa: ANN001
|
|
inject_tool, # noqa: ANN001
|
|
inject_system_instructions, # noqa: ANN001
|
|
):
|
|
captured["inject_tool"] = inject_tool
|
|
self.has_compressed_content = False
|
|
self.detected_hashes = []
|
|
|
|
def process_request(self, messages, tools): # noqa: ANN001
|
|
return messages, tools, False
|
|
|
|
def scan_for_markers(self, messages): # noqa: ANN001
|
|
return []
|
|
|
|
def verify_ownership(self, store=None): # noqa: ANN001
|
|
return self.detected_hashes
|
|
|
|
monkeypatch.setattr("headroom.ccr.CCRToolInjector", _FakeInjector)
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_ccr_tool_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 20,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [{"role": "user", "content": "hello"}],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert captured["inject_tool"] is False
|
|
|
|
|
|
def test_ccr_tool_stays_in_forwarded_tools_across_frozen_transition() -> None:
|
|
"""``tools`` identity must survive the ``frozen 0 -> >0`` transition.
|
|
|
|
``tools`` is the head of Anthropic's cache key, so adding or removing
|
|
``headroom_retrieve`` between turns invalidates 100% of the provider-cached
|
|
prefix — in both directions. Turn 1 (cold prefix, fresh markers) injects the
|
|
tool; turn 2 (warm prefix, no *new* markers) must forward the same bytes
|
|
rather than dropping it.
|
|
|
|
Asserts on the forwarded request body, not on a policy function's return
|
|
value: unit-testing the old policy in isolation is exactly what let a
|
|
wrong-but-self-consistent decision pass.
|
|
"""
|
|
from headroom.cache.compression_store import get_compression_store, reset_compression_store
|
|
from headroom.ccr.tool_injection import CCR_TOOL_NAME
|
|
from headroom.proxy.helpers import (
|
|
_reset_session_ccr_tracker_for_test,
|
|
serialize_tool_definition_canonical,
|
|
)
|
|
|
|
# verify_ownership() (issue #2836) requires the marker's hash to be a
|
|
# real store entry — seed one with the exact hash the marker text below
|
|
# references, via explicit_hash (the store's own hash generation from
|
|
# `original` content wouldn't match this hand-typed literal).
|
|
reset_compression_store()
|
|
get_compression_store().store(
|
|
original="original tool output",
|
|
compressed="[50 items compressed to 5]",
|
|
explicit_hash="abc123def456abc123def456",
|
|
)
|
|
|
|
marker_message = {
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "toolu_bash_x",
|
|
"content": (
|
|
"[50 items compressed to 5. Retrieve more: hash=abc123def456abc123def456]"
|
|
),
|
|
}
|
|
],
|
|
}
|
|
forwarded: list[dict] = []
|
|
|
|
_reset_session_ccr_tracker_for_test()
|
|
try:
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = False
|
|
proxy.config.image_optimize = False
|
|
proxy.config.ccr_inject_tool = True
|
|
proxy.config.ccr_inject_system_instructions = False
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"frozen-transition-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
forwarded.append(body)
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_frozen_transition",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 20,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
def _post():
|
|
return client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [marker_message],
|
|
},
|
|
)
|
|
|
|
# Turn 1 — cold prefix, marker is new: first-time injection.
|
|
assert _post().status_code == 200
|
|
|
|
# Turn 2 — the provider cached turn 1's prefix (tool included), and
|
|
# the marker is now historical rather than new. Seed both facts
|
|
# explicitly instead of relying on ``update_from_response`` plumbing:
|
|
# if the marker still counted as new, the old code would have
|
|
# injected via its override path and this test would pass against
|
|
# the defect.
|
|
fake_tracker._frozen_count = 3
|
|
fake_tracker._last_forwarded_messages = [marker_message]
|
|
|
|
assert _post().status_code == 200
|
|
finally:
|
|
_reset_session_ccr_tracker_for_test()
|
|
reset_compression_store()
|
|
|
|
assert len(forwarded) == 2, "expected exactly two forwarded requests"
|
|
|
|
def _ccr_tools(body: dict) -> list[dict]:
|
|
return [t for t in (body.get("tools") or []) if t.get("name") == CCR_TOOL_NAME]
|
|
|
|
turn1 = _ccr_tools(forwarded[0])
|
|
turn2 = _ccr_tools(forwarded[1])
|
|
|
|
assert turn1, "test setup: turn 1 should inject headroom_retrieve on fresh markers"
|
|
assert turn2, (
|
|
"headroom_retrieve was dropped from the forwarded tools array once the "
|
|
"prefix went warm — that removes a tool already inside the cached prefix "
|
|
"and busts 100% of it"
|
|
)
|
|
# Byte-identity, not ``==``: a re-serialized definition with a different key
|
|
# order compares equal as a dict but busts the cache just as hard.
|
|
assert serialize_tool_definition_canonical(turn1[0]) == serialize_tool_definition_canonical(
|
|
turn2[0]
|
|
), "headroom_retrieve was re-serialized rather than replayed byte-for-byte"
|
|
|
|
|
|
def test_previous_turns_always_frozen_only_final_turn_mutable() -> None:
|
|
captured = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "cache"
|
|
proxy.config.image_optimize = False
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = lambda **kwargs: (_ for _ in ()).throw(
|
|
AssertionError("cache mode should not invoke anthropic pipeline")
|
|
)
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_frz_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 80,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "current turn"},
|
|
],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert captured["body"]["messages"] == [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "current turn"},
|
|
]
|
|
|
|
|
|
def test_batch_optimization_freezes_previous_turns_only() -> None:
|
|
captured = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "cache"
|
|
proxy.config.image_optimize = False
|
|
proxy.config.ccr_inject_tool = False
|
|
|
|
proxy.anthropic_pipeline.apply = lambda **kwargs: (_ for _ in ()).throw(
|
|
AssertionError("cache mode batch path should not invoke anthropic pipeline")
|
|
)
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msgbatch_2",
|
|
"type": "message_batch",
|
|
"processing_status": "in_progress",
|
|
"request_counts": {
|
|
"processing": 1,
|
|
"succeeded": 0,
|
|
"errored": 0,
|
|
"canceled": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages/batches",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"requests": [
|
|
{
|
|
"custom_id": "req-1",
|
|
"params": {
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 128,
|
|
"messages": [
|
|
{"role": "user", "content": "old turn"},
|
|
{"role": "assistant", "content": "old assistant"},
|
|
{"role": "user", "content": "current turn"},
|
|
],
|
|
},
|
|
}
|
|
]
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert captured["body"]["requests"][0]["params"]["messages"] == [
|
|
{"role": "user", "content": "old turn"},
|
|
{"role": "assistant", "content": "old assistant"},
|
|
{"role": "user", "content": "current turn"},
|
|
]
|
|
|
|
|
|
def test_batch_optimization_passes_savings_profile_kwargs() -> None:
|
|
captured = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "token"
|
|
proxy.config.savings_profile = "agent-90"
|
|
proxy.config.ccr_inject_tool = False
|
|
|
|
def _fake_apply(**kwargs):
|
|
captured["pipeline_kwargs"] = kwargs
|
|
return SimpleNamespace(
|
|
messages=kwargs["messages"],
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=100,
|
|
tokens_after=80,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msgbatch_profile",
|
|
"type": "message_batch",
|
|
"processing_status": "in_progress",
|
|
"request_counts": {
|
|
"processing": 1,
|
|
"succeeded": 0,
|
|
"errored": 0,
|
|
"canceled": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages/batches",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"requests": [
|
|
{
|
|
"custom_id": "req-1",
|
|
"params": {
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 128,
|
|
"messages": [{"role": "user", "content": "compress me"}],
|
|
},
|
|
}
|
|
]
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
pipeline_kwargs = captured["pipeline_kwargs"]
|
|
assert pipeline_kwargs["force_kompress"] is True
|
|
assert pipeline_kwargs["target_ratio"] == 0.10
|
|
assert pipeline_kwargs["compress_user_messages"] is True
|
|
|
|
|
|
def test_token_mode_does_not_force_freeze_all_previous_turns() -> None:
|
|
captured = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "token"
|
|
proxy.config.image_optimize = False
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
|
|
class _FakeCompressionCache:
|
|
def apply_cached(self, messages): # noqa: ANN001
|
|
return messages
|
|
|
|
def compute_frozen_count(self, messages): # noqa: ANN001
|
|
return 0
|
|
|
|
def update_from_result(self, originals, compressed): # noqa: ANN001
|
|
return None
|
|
|
|
def mark_stable_from_messages(self, messages, up_to): # noqa: ANN001
|
|
pass
|
|
|
|
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache()
|
|
|
|
def _fake_apply(**kwargs):
|
|
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
|
|
return SimpleNamespace(
|
|
messages=kwargs["messages"],
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=70,
|
|
tokens_after=70,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_tok_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 70,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "current turn"},
|
|
],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
# In token_headroom mode, mark_stable_from_messages marks prior turns
|
|
# as stable, so frozen count reflects the number of prior-turn messages.
|
|
# The compression cache's compute_frozen_count returns 0 (no cached
|
|
# compressions yet), but mark_stable marks previous turns as frozen
|
|
# to preserve prefix cache stability.
|
|
assert captured["frozen_message_count"] >= 0
|
|
|
|
|
|
def test_cache_mode_restores_frozen_prefix_if_transform_mutates_history(monkeypatch) -> None:
|
|
_force_compression(monkeypatch)
|
|
captured = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "cache"
|
|
proxy.config.image_optimize = False
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
|
|
original_messages = [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "current turn"},
|
|
]
|
|
# Mid-session: the first two messages were already forwarded last turn,
|
|
# so they form the byte-stable cached prefix the handler must replay
|
|
# even if a transform tries to mutate them.
|
|
fake_tracker._last_original_messages = original_messages[:2]
|
|
fake_tracker._last_forwarded_messages = original_messages[:2]
|
|
|
|
def _fake_apply(**kwargs):
|
|
mutated = list(kwargs["messages"])
|
|
mutated[0] = {**mutated[0], "content": "MUTATED_PREFIX"}
|
|
return SimpleNamespace(
|
|
messages=mutated,
|
|
transforms_applied=["fake:mutated"],
|
|
timing={},
|
|
tokens_before=80,
|
|
tokens_after=70,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_cache_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 70,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": original_messages,
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
sent_messages = captured["body"]["messages"]
|
|
assert sent_messages[0] == original_messages[0]
|
|
assert sent_messages[1] == original_messages[1]
|
|
|
|
|
|
def test_cache_mode_cold_start_forwards_pipeline_rewrites(monkeypatch) -> None:
|
|
_force_compression(monkeypatch)
|
|
# Issue #2357: on a true session cold start (no prior forwarded messages,
|
|
# no frozen prefix) there is no provider cache to protect, so the full
|
|
# pipeline output is forwarded and recorded for byte-identical replay.
|
|
captured = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "cache"
|
|
proxy.config.image_optimize = False
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
|
|
original_messages = [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "current turn"},
|
|
]
|
|
|
|
def _fake_apply(**kwargs):
|
|
mutated = list(kwargs["messages"])
|
|
mutated[2] = {**mutated[2], "content": "REWRITTEN_CURRENT_TURN"}
|
|
return SimpleNamespace(
|
|
messages=mutated,
|
|
transforms_applied=["fake:mutated-latest"],
|
|
timing={},
|
|
tokens_before=80,
|
|
tokens_after=60,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_cache_2",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 80,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": original_messages,
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
sent_messages = captured["body"]["messages"]
|
|
assert sent_messages[:2] == original_messages[:2]
|
|
assert sent_messages[2]["content"] == "REWRITTEN_CURRENT_TURN"
|
|
# Recorded as forwarded so later turns replay this prefix verbatim
|
|
# (the tracker may append the assistant reply after the forwarded turns).
|
|
assert fake_tracker._last_forwarded_messages[: len(sent_messages)] == sent_messages
|
|
|
|
|
|
def test_cache_mode_reuses_prior_forwarded_prefix_and_compresses_only_new_suffix() -> None:
|
|
captured = {"calls": []}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "cache"
|
|
proxy.config.image_optimize = False
|
|
|
|
tracker = _FakePrefixTracker(frozen_count=0)
|
|
tracker._last_original_messages = [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "turn2"},
|
|
{"role": "assistant", "content": "turn2-assistant"},
|
|
]
|
|
tracker._last_forwarded_messages = [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "COMPRESSED_TURN2"},
|
|
{"role": "assistant", "content": "turn2-assistant"},
|
|
]
|
|
tracker.get_last_original_messages = lambda: tracker._last_original_messages.copy()
|
|
tracker.get_last_forwarded_messages = lambda: tracker._last_forwarded_messages.copy()
|
|
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
|
)
|
|
|
|
def _fake_apply(**kwargs):
|
|
captured["calls"].append(kwargs["messages"])
|
|
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
|
|
# fix-6 contract: the compressor is handed the frozen forwarded prefix
|
|
# + the new delta and only compresses indices >= frozen_message_count
|
|
# (so a lone tool_result can resolve its tool_name from the prefix).
|
|
# Mirror the real router: pass the frozen prefix through verbatim and
|
|
# compress only the tail — the handler splices result.messages[prefix_n:].
|
|
fz = kwargs.get("frozen_message_count") or 0
|
|
msgs = kwargs["messages"]
|
|
return SimpleNamespace(
|
|
messages=list(msgs[:fz]) + [{"role": "user", "content": "COMPRESSED_TURN3"}],
|
|
transforms_applied=["fake:delta"],
|
|
timing={},
|
|
tokens_before=40,
|
|
tokens_after=20,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_cache_3",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 80,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "turn2"},
|
|
{"role": "assistant", "content": "turn2-assistant"},
|
|
{"role": "user", "content": "turn3"},
|
|
],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
# fix-6 contract: the compressor receives the frozen FORWARDED prefix
|
|
# (with COMPRESSED_TURN2, the byte-stable cached form) + the raw new
|
|
# delta (turn3), so tool_name resolution / dedup stay consistent with
|
|
# what is actually cached. frozen_message_count = prefix length pins
|
|
# compression to the delta ONLY — the prefix is never re-compressed.
|
|
assert captured["calls"] == [
|
|
[
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "COMPRESSED_TURN2"},
|
|
{"role": "assistant", "content": "turn2-assistant"},
|
|
{"role": "user", "content": "turn3"},
|
|
]
|
|
]
|
|
assert captured["frozen_message_count"] == 4 # only the delta (turn3) is compressed
|
|
# Forwarded body = byte-identical cached prefix + the compressed delta.
|
|
assert captured["body"]["messages"] == [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "COMPRESSED_TURN2"},
|
|
{"role": "assistant", "content": "turn2-assistant"},
|
|
{"role": "user", "content": "COMPRESSED_TURN3"},
|
|
]
|
|
|
|
|
|
def test_cache_mode_delta_keeps_newest_user_prompt_verbatim() -> None:
|
|
"""The cache-mode delta call must tell the router the prefix is replayed.
|
|
|
|
Without ``prefix_replay_guaranteed`` the router treats the newest user
|
|
message as compressible text, so under the coding profile (which turns on
|
|
user-message compression) a follow-up prompt -- e.g. a Claude Code
|
|
teammate's ``<teammate-message>`` -- reached the model lossily rewritten
|
|
on every turn after the first (#1174). Every other Anthropic
|
|
``pipeline.apply`` call already passes the flag.
|
|
"""
|
|
captured: dict = {}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "cache"
|
|
proxy.config.image_optimize = False
|
|
|
|
tracker = _FakePrefixTracker(frozen_count=0)
|
|
tracker._last_original_messages = [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
]
|
|
tracker._last_forwarded_messages = list(tracker._last_original_messages)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
|
)
|
|
|
|
def _fake_apply(**kwargs):
|
|
captured.update(kwargs)
|
|
return SimpleNamespace(
|
|
messages=list(kwargs["messages"]),
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=10,
|
|
tokens_after=10,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_delta_prompt",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {"input_tokens": 10, "output_tokens": 1},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [
|
|
{"role": "user", "content": "turn1"},
|
|
{"role": "assistant", "content": "turn1-assistant"},
|
|
{"role": "user", "content": "<teammate-message>next task</teammate-message>"},
|
|
],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert captured["frozen_message_count"] == 2 # the delta path ran
|
|
assert captured.get("prefix_replay_guaranteed") is True
|
|
|
|
|
|
def test_anthropic_handler_anchors_a_proven_rewritten_tail_to_stable_blocks() -> None:
|
|
"""The real handler must feed last turn's bytes into normalization."""
|
|
bodies = []
|
|
|
|
def _history(turn: int, churn: int) -> list[dict]:
|
|
content = [{"type": "text", "text": f"stable-{index}"} for index in range(30)]
|
|
content.extend(
|
|
{"type": "text", "text": f"turn-{turn}-changing-{index}"} for index in range(churn)
|
|
)
|
|
content.extend(
|
|
[
|
|
{"type": "text", "text": "instruction: summarize"},
|
|
{"type": "text", "text": "fixed end-of-transcript reminder"},
|
|
]
|
|
)
|
|
return [{"role": "user", "content": content}]
|
|
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
# Breakpoint ownership and lineage tracking apply even in passthrough
|
|
# mode. Keeping optimization off isolates that real handler wiring from
|
|
# the compression pipeline.
|
|
proxy.config.optimize = False
|
|
proxy.config.image_optimize = False
|
|
# This regression models a client whose next request replaces the same
|
|
# aggregate message. Do not synthesize an assistant history entry in
|
|
# the tracker, because that is a separate response-reconstruction path.
|
|
proxy._assistant_message_from_response_json = lambda _body: None
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
bodies.append(copy.deepcopy(body))
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_rewrite",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [],
|
|
"usage": {
|
|
"input_tokens": 10_000,
|
|
"output_tokens": 1,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 10_000,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
|
|
for turn, churn in enumerate((3, 5, 8), start=1):
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers=headers,
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 32,
|
|
"messages": _history(turn, churn),
|
|
},
|
|
)
|
|
assert response.status_code == 200
|
|
|
|
breakpoint_indices = []
|
|
for body in bodies:
|
|
marked = [
|
|
index
|
|
for index, block in enumerate(body["messages"][0]["content"])
|
|
if "cache_control" in block
|
|
]
|
|
assert len(marked) == 1
|
|
breakpoint_indices.append(marked[0])
|
|
|
|
# Cold request caches through its newest block. Once rewrite is proven,
|
|
# all subsequent calls pin the same 30-block boundary instead of creating
|
|
# an ever-growing full write on each turn.
|
|
assert breakpoint_indices == [34, 29, 29]
|
|
|
|
|
|
def test_cache_mode_skips_same_message_append_rewrite_to_preserve_stability() -> None:
|
|
captured = {"calls": []}
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.config.optimize = True
|
|
proxy.config.mode = "cache"
|
|
proxy.config.image_optimize = False
|
|
|
|
tracker = _FakePrefixTracker(frozen_count=0)
|
|
tracker._last_original_messages = [
|
|
{"role": "user", "content": "shared-prefix"},
|
|
]
|
|
tracker._last_forwarded_messages = [
|
|
{"role": "user", "content": "COMPRESSED_PREFIX"},
|
|
]
|
|
tracker.get_last_original_messages = lambda: tracker._last_original_messages.copy()
|
|
tracker.get_last_forwarded_messages = lambda: tracker._last_forwarded_messages.copy()
|
|
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stable-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
|
)
|
|
|
|
def _fake_apply(**kwargs):
|
|
captured["calls"].append(kwargs["messages"])
|
|
return SimpleNamespace(
|
|
messages=[{"role": "user", "content": " + COMPRESSED_SUFFIX"}],
|
|
transforms_applied=["fake:suffix"],
|
|
timing={},
|
|
tokens_before=20,
|
|
tokens_after=10,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
captured["body"] = body
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_cache_suffix",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 80,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [
|
|
{"role": "user", "content": "shared-prefix + raw suffix"},
|
|
],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert captured["calls"] == []
|
|
assert captured["body"]["messages"] == [
|
|
{"role": "user", "content": "shared-prefix + raw suffix"},
|
|
]
|
|
|
|
|
|
# ─── Issue #327 regression tests ─────────────────────────────────────────────
|
|
#
|
|
# Lock down the post-fix invariant: the Anthropic token-mode handler must
|
|
# never extend `frozen_message_count` past the smaller of
|
|
# `prefix_tracker.frozen_message_count` and `compute_frozen_count(messages)`.
|
|
# The deleted walker (anthropic.py:756-787 pre-fix) advanced past those bounds
|
|
# whenever a fresh tool_result's content-hash matched any prior `_stable_hashes`
|
|
# entry or the TTL deferral fired. That conflated content equality with
|
|
# positional cache membership; for SvenMeyer's reported session it forced 73%
|
|
# of requests into `transforms_applied=[]` even though the corresponding byte
|
|
# positions were not actually in Anthropic's prefix cache.
|
|
|
|
|
|
class _IssueFakeCompCache:
|
|
"""Mock CompressionCache supporting the post-fix surface.
|
|
|
|
Records calls so tests can assert which methods fire and in what order.
|
|
Provides a populated `_stable_hashes` set for tests that need to prove
|
|
`_stable_hashes` membership no longer pushes `frozen_message_count` past
|
|
the prefix-tracker bound.
|
|
"""
|
|
|
|
def __init__(self, frozen_via_compute: int = 0, prepopulated_hashes: set | None = None):
|
|
self._frozen_via_compute = frozen_via_compute
|
|
self._stable_hashes: set[str] = set(prepopulated_hashes or set())
|
|
self._cache: dict = {}
|
|
self.calls: list[tuple[str, tuple, dict]] = []
|
|
self.applied_cached_with: list = []
|
|
|
|
def apply_cached(self, messages): # noqa: ANN001
|
|
self.calls.append(("apply_cached", (), {}))
|
|
self.applied_cached_with = list(messages)
|
|
return list(messages)
|
|
|
|
def compute_frozen_count(self, messages): # noqa: ANN001
|
|
self.calls.append(("compute_frozen_count", (), {}))
|
|
return self._frozen_via_compute
|
|
|
|
def mark_stable_from_messages(self, messages, up_to): # noqa: ANN001
|
|
self.calls.append(("mark_stable_from_messages", (up_to,), {}))
|
|
|
|
def update_from_result(self, originals, compressed): # noqa: ANN001
|
|
self.calls.append(("update_from_result", (), {}))
|
|
|
|
# Methods that should NOT be called in the post-fix code path. If any of
|
|
# these fire, the walker has resurrected.
|
|
def should_defer_compression(self, *args, **kwargs): # noqa: ANN001, ANN002, ANN003
|
|
self.calls.append(("should_defer_compression", args, kwargs))
|
|
return False
|
|
|
|
def mark_stable(self, content_hash): # noqa: ANN001
|
|
self.calls.append(("mark_stable", (content_hash,), {}))
|
|
|
|
@staticmethod
|
|
def content_hash(content): # noqa: ANN001
|
|
# Deterministic hash so prepopulated_hashes works in tests.
|
|
if isinstance(content, str):
|
|
return f"H({content[:40]})"
|
|
return f"H(list:{len(content)})"
|
|
|
|
|
|
def _make_optimize_proxy_client(mode: str = "token") -> TestClient:
|
|
"""Build a proxy client wired for optimization (issue-#327 tests)."""
|
|
config = ProxyConfig(
|
|
optimize=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
mode=mode,
|
|
)
|
|
app = create_app(config)
|
|
return TestClient(app)
|
|
|
|
|
|
def _drive_request(
|
|
client: TestClient,
|
|
*,
|
|
fake_comp_cache: _IssueFakeCompCache,
|
|
prefix_tracker_frozen: int,
|
|
messages: list,
|
|
captured: dict,
|
|
) -> httpx.Response:
|
|
"""Common test driver — wire fakes and submit a /v1/messages request."""
|
|
proxy = client.app.state.proxy
|
|
|
|
fake_tracker = _FakePrefixTracker(frozen_count=prefix_tracker_frozen)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"issue-327-session"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
proxy._get_compression_cache = lambda session_id: fake_comp_cache
|
|
|
|
def _fake_apply(**kwargs): # noqa: ANN003
|
|
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
|
|
captured["pipeline_messages"] = list(kwargs["messages"])
|
|
# Record the byte-shape of the frozen prefix (deep snapshot via repr —
|
|
# tests below assert byte-stability with input).
|
|
captured["frozen_prefix_repr"] = repr(
|
|
list(kwargs["messages"])[: kwargs.get("frozen_message_count", 0)]
|
|
)
|
|
return SimpleNamespace(
|
|
messages=list(kwargs["messages"]),
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=100,
|
|
tokens_after=100,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_327",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 100,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
return client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": messages,
|
|
},
|
|
)
|
|
|
|
|
|
def _build_messages_with_repeat(repeat_at_idx: int = None) -> list: # noqa: ANN001
|
|
"""Build a 20-message session ending in a fresh tool_result.
|
|
|
|
If `repeat_at_idx` is given, the tool_result at that index has the same
|
|
content as the LAST tool_result, so its hash collides with `_stable_hashes`.
|
|
"""
|
|
msgs: list = []
|
|
for turn in range(7):
|
|
msgs.append({"role": "user", "content": f"turn-{turn}-user-question"})
|
|
if turn % 2 == 0:
|
|
msgs.append({"role": "assistant", "content": f"turn-{turn}-assistant"})
|
|
else:
|
|
msgs.append(
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": f"toolu_{turn}",
|
|
"name": "lookup",
|
|
"input": {"q": str(turn)},
|
|
}
|
|
],
|
|
}
|
|
)
|
|
msgs.append(
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": f"toolu_{turn}",
|
|
"content": f"unique-fresh-tool-output-for-turn-{turn}-AAAAA" * 20,
|
|
}
|
|
],
|
|
}
|
|
)
|
|
# msgs is currently length 17. Add a final tool_result at index 17.
|
|
fresh_content = "FINAL-FRESH-tool-output-content-XXXXX" * 30
|
|
msgs.append(
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{"type": "tool_use", "id": "toolu_final", "name": "lookup", "input": {"q": "f"}}
|
|
],
|
|
}
|
|
)
|
|
msgs.append(
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "toolu_final",
|
|
"content": fresh_content,
|
|
}
|
|
],
|
|
}
|
|
)
|
|
if repeat_at_idx is not None:
|
|
# Overwrite the tool_result at repeat_at_idx with `fresh_content` so
|
|
# its hash collides with the final message's hash.
|
|
target = msgs[repeat_at_idx]
|
|
if isinstance(target.get("content"), list):
|
|
for blk in target["content"]:
|
|
if isinstance(blk, dict) and blk.get("type") == "tool_result":
|
|
blk["content"] = fresh_content
|
|
break
|
|
return msgs
|
|
|
|
|
|
def test_issue_327_walker_removed_does_not_advance_past_prefix_tracker() -> None:
|
|
"""frozen_message_count must equal min(prefix_tracker, compute_frozen_count).
|
|
|
|
The deleted walker (anthropic.py:766-787 pre-fix) advanced past
|
|
`prefix_tracker.frozen_message_count` whenever upcoming tool_results had
|
|
hashes in `_stable_hashes` or returned True from `should_defer_compression`.
|
|
Even with `_stable_hashes` populated with 50 entries, the post-fix code
|
|
must clamp to the smaller of the two positional sources.
|
|
"""
|
|
captured: dict = {}
|
|
with _make_optimize_proxy_client(mode="token") as client:
|
|
prepopulated = {f"H(synthetic-old-content-{i})" for i in range(50)}
|
|
fake_cache = _IssueFakeCompCache(frozen_via_compute=8, prepopulated_hashes=prepopulated)
|
|
|
|
response = _drive_request(
|
|
client,
|
|
fake_comp_cache=fake_cache,
|
|
prefix_tracker_frozen=15, # bigger than compute_frozen_count
|
|
messages=_build_messages_with_repeat(),
|
|
captured=captured,
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
# Post-fix invariant: clamped to min(15, 8) = 8.
|
|
assert captured["frozen_message_count"] == 8, (
|
|
f"Expected frozen_message_count=8 (min of prefix_tracker=15 and "
|
|
f"compute_frozen_count=8); got {captured['frozen_message_count']}. "
|
|
f"If this is higher, the deleted walker has resurrected."
|
|
)
|
|
# The walker functions must not have been called.
|
|
method_names = [c[0] for c in fake_cache.calls]
|
|
assert "should_defer_compression" not in method_names, (
|
|
f"should_defer_compression was called from production handler; calls={fake_cache.calls}"
|
|
)
|
|
assert "mark_stable" not in method_names, (
|
|
f"mark_stable was called as walker side-effect; calls={fake_cache.calls}"
|
|
)
|
|
|
|
|
|
def test_issue_327_repeated_content_new_position_is_not_frozen() -> None:
|
|
"""A fresh tool_result whose content-hash matches an old `_stable_hashes`
|
|
entry must NOT be frozen on its new position. Pre-fix: walker would skip
|
|
past it on hash equality. Post-fix: only positional bounds matter."""
|
|
captured: dict = {}
|
|
with _make_optimize_proxy_client(mode="token") as client:
|
|
# Pre-populate _stable_hashes with the hash of the trailing tool_result.
|
|
# Since _IssueFakeCompCache.content_hash is deterministic on the first
|
|
# 40 chars of the string, pre-seed the same key.
|
|
fresh_content = "FINAL-FRESH-tool-output-content-XXXXX" * 30
|
|
prepopulated = {_IssueFakeCompCache.content_hash(fresh_content)}
|
|
fake_cache = _IssueFakeCompCache(frozen_via_compute=8, prepopulated_hashes=prepopulated)
|
|
|
|
response = _drive_request(
|
|
client,
|
|
fake_comp_cache=fake_cache,
|
|
prefix_tracker_frozen=8,
|
|
messages=_build_messages_with_repeat(repeat_at_idx=8),
|
|
captured=captured,
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
# Pipeline got everything from index 8 onward — including the trailing
|
|
# repeat-content tool_result at index 18. Post-fix, frozen_message_count
|
|
# is exactly 8 regardless of any hash matches in _stable_hashes.
|
|
assert captured["frozen_message_count"] == 8
|
|
|
|
|
|
def test_issue_327_pipeline_preserves_frozen_prefix_byte_for_byte() -> None:
|
|
"""Invariant: messages[:frozen_message_count] passed to the pipeline are
|
|
byte-identical to the messages received from the client (modulo the
|
|
`apply_cached` swap, which is byte-stable). Lock the cache-floor."""
|
|
captured: dict = {}
|
|
with _make_optimize_proxy_client(mode="token") as client:
|
|
fake_cache = _IssueFakeCompCache(frozen_via_compute=10)
|
|
|
|
msgs = _build_messages_with_repeat()
|
|
response = _drive_request(
|
|
client,
|
|
fake_comp_cache=fake_cache,
|
|
prefix_tracker_frozen=10,
|
|
messages=msgs,
|
|
captured=captured,
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
# apply_cached returned `list(messages)` (no swap) so the prefix should be
|
|
# byte-equal to the input.
|
|
frozen_prefix = captured["pipeline_messages"][: captured["frozen_message_count"]]
|
|
assert frozen_prefix == msgs[: captured["frozen_message_count"]], (
|
|
"Frozen prefix mutated between client request and pipeline call — "
|
|
"this would bust Anthropic's prefix cache."
|
|
)
|
|
|
|
|
|
def test_issue_327_multi_turn_session_compresses_each_turns_tail() -> None:
|
|
"""Simulate a 10-turn loop and assert that each turn the pipeline gets a
|
|
suffix of the messages to compress (frozen_message_count < len(messages)).
|
|
|
|
Pre-fix, after a few turns of accumulation, the walker would advance
|
|
`frozen_message_count` to `len(messages)` and the pipeline would get an
|
|
empty suffix → transforms_applied=[] on every turn (the SvenMeyer
|
|
fingerprint).
|
|
"""
|
|
frozen_per_turn: list = []
|
|
suffix_size_per_turn: list = []
|
|
|
|
with _make_optimize_proxy_client(mode="token") as client:
|
|
# Same comp_cache shared across all turns — `_stable_hashes` accumulates.
|
|
fake_cache = _IssueFakeCompCache()
|
|
|
|
for turn in range(10):
|
|
captured: dict = {}
|
|
# Each turn: prefix_tracker advances by 2 (one assistant + one new
|
|
# tool_result observed last turn). compute_frozen_count returns
|
|
# the same value to simulate "local cache covers what tracker says".
|
|
prefix_tracker_frozen = max(0, turn * 2 - 1)
|
|
fake_cache._frozen_via_compute = prefix_tracker_frozen
|
|
|
|
msgs = _build_messages_with_repeat()
|
|
# Append turn-specific filler so each request has a different shape.
|
|
for t in range(turn):
|
|
msgs.append({"role": "user", "content": f"continuation-{t}"})
|
|
|
|
response = _drive_request(
|
|
client,
|
|
fake_comp_cache=fake_cache,
|
|
prefix_tracker_frozen=prefix_tracker_frozen,
|
|
messages=msgs,
|
|
captured=captured,
|
|
)
|
|
assert response.status_code == 200
|
|
|
|
frozen_per_turn.append(captured["frozen_message_count"])
|
|
suffix_size_per_turn.append(
|
|
len(captured["pipeline_messages"]) - captured["frozen_message_count"]
|
|
)
|
|
|
|
# Every turn the pipeline must see a non-empty suffix to compress.
|
|
# Pre-fix, this would be 0 for most turns (the SvenMeyer 73%-frozen).
|
|
empty_suffix_turns = [i for i, s in enumerate(suffix_size_per_turn) if s == 0]
|
|
assert len(empty_suffix_turns) == 0, (
|
|
f"{len(empty_suffix_turns)}/10 turns had empty suffix to compress; "
|
|
f"sizes={suffix_size_per_turn}, frozen={frozen_per_turn}. "
|
|
f"Pre-fix walker behavior detected."
|
|
)
|
|
|
|
|
|
def test_issue_327_streaming_and_non_streaming_compute_same_frozen_count() -> None:
|
|
"""The optimization path is upstream of the stream/non-stream branch.
|
|
Whatever `frozen_message_count` token-mode produces for `stream=False`
|
|
must match what it produces for `stream=True` on identical inputs.
|
|
Locks the safety property that streaming has no separate, divergent
|
|
walker logic.
|
|
"""
|
|
msgs = _build_messages_with_repeat()
|
|
|
|
# Run 1: stream=False
|
|
captured_a: dict = {}
|
|
with _make_optimize_proxy_client(mode="token") as client:
|
|
fake_cache_a = _IssueFakeCompCache(frozen_via_compute=10)
|
|
|
|
proxy = client.app.state.proxy
|
|
fake_tracker = _FakePrefixTracker(frozen_count=12)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stream-parity-A"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
proxy._get_compression_cache = lambda s: fake_cache_a
|
|
|
|
def _fake_apply_a(**kwargs): # noqa: ANN003
|
|
captured_a["frozen_message_count"] = kwargs.get("frozen_message_count")
|
|
return SimpleNamespace(
|
|
messages=list(kwargs["messages"]),
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=100,
|
|
tokens_after=100,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply_a
|
|
|
|
async def _fake_retry_a(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 100,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry_a
|
|
|
|
r_a = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"stream": False,
|
|
"messages": msgs,
|
|
},
|
|
)
|
|
assert r_a.status_code == 200
|
|
|
|
# Run 2: stream=True (same inputs)
|
|
captured_b: dict = {}
|
|
with _make_optimize_proxy_client(mode="token") as client:
|
|
fake_cache_b = _IssueFakeCompCache(frozen_via_compute=10)
|
|
|
|
proxy = client.app.state.proxy
|
|
fake_tracker = _FakePrefixTracker(frozen_count=12)
|
|
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
|
"stream-parity-B"
|
|
)
|
|
proxy.session_tracker_store.get_or_create = (
|
|
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
|
)
|
|
proxy._get_compression_cache = lambda s: fake_cache_b
|
|
|
|
def _fake_apply_b(**kwargs): # noqa: ANN003
|
|
captured_b["frozen_message_count"] = kwargs.get("frozen_message_count")
|
|
return SimpleNamespace(
|
|
messages=list(kwargs["messages"]),
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=100,
|
|
tokens_after=100,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply_b
|
|
|
|
# Streaming path: return a minimal SSE body
|
|
sse_body = (
|
|
b"event: message_start\n"
|
|
b'data: {"type":"message_start","message":{"id":"msg",'
|
|
b'"role":"assistant","content":[],"model":"claude",'
|
|
b'"usage":{"input_tokens":100,"output_tokens":0,'
|
|
b'"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}\n\n'
|
|
b"event: content_block_start\n"
|
|
b'data: {"type":"content_block_start","index":0,'
|
|
b'"content_block":{"type":"text","text":""}}\n\n'
|
|
b"event: content_block_stop\n"
|
|
b'data: {"type":"content_block_stop","index":0}\n\n'
|
|
b"event: message_delta\n"
|
|
b'data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},'
|
|
b'"usage":{"output_tokens":3}}\n\n'
|
|
b"event: message_stop\n"
|
|
b'data: {"type":"message_stop"}\n\n'
|
|
)
|
|
|
|
async def _fake_retry_b(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return httpx.Response(
|
|
200,
|
|
content=sse_body,
|
|
headers={"content-type": "text/event-stream"},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry_b
|
|
# Streaming dispatch uses _stream_response (not _retry_request). Stub
|
|
# it to a no-op streaming response so we can inspect what
|
|
# pipeline.apply received without being responsible for the SSE
|
|
# plumbing — the optimization runs before _stream_response is called.
|
|
from fastapi.responses import StreamingResponse
|
|
|
|
async def _fake_stream_response(*args, **kwargs): # noqa: ANN001, ANN002, ANN003
|
|
async def _gen():
|
|
yield b""
|
|
|
|
return StreamingResponse(_gen(), media_type="text/event-stream")
|
|
|
|
proxy._stream_response = _fake_stream_response
|
|
|
|
client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"stream": True,
|
|
"messages": msgs,
|
|
},
|
|
)
|
|
# Status code is incidental — what matters is that pipeline.apply ran
|
|
# and captured_b was populated.
|
|
|
|
assert "frozen_message_count" in captured_a, "Non-streaming path didn't reach pipeline.apply()"
|
|
assert "frozen_message_count" in captured_b, "Streaming path didn't reach pipeline.apply()"
|
|
assert captured_a["frozen_message_count"] == captured_b["frozen_message_count"], (
|
|
f"Streaming/non-streaming divergence: stream=False produced "
|
|
f"frozen={captured_a['frozen_message_count']}, stream=True produced "
|
|
f"{captured_b['frozen_message_count']}. The optimization path must "
|
|
f"be identical for both."
|
|
)
|