Files
headroom/tests/test_proxy_semantic_cache_key_integration.py
C0sm0catsandJerrettDavis dfdc72511e fix(proxy): partition response cache by upstream (#3349)
## Description

Include the resolved custom upstream base URL in semantic response-cache
identity for Anthropic and OpenAI chat handlers. Identical prompts sent
to different compatible gateways can legitimately produce different
responses and must not share cached output.

Closes #3346

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `upstream_base_url` to Anthropic response-cache key fields.
- Add `upstream_base_url` to OpenAI chat response-cache key fields.
- Add handler-level regressions for two custom gateways receiving
identical requests.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/pytest -q tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py
8 passed, 1 warning in 2.76s

$ .venv/bin/ruff check <changed files>
All checks passed!

$ .venv/bin/ruff format --check <changed files>
4 files already formatted

$ .venv/bin/mypy headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Linux, Python 3.14, real FastAPI proxy app and real HTTP
transport, two local OpenAI-compatible HTTP upstreams, semantic cache
enabled.
- Exact command / steps: allowlisted two loopback origins with
`HEADROOM_ALLOWED_BASE_URLS`, started an HTTP server on each origin
returning its gateway identity, then posted the same
`/v1/chat/completions` body twice with only `x-headroom-base-url`
changed.
- Observed result: each request reached its selected upstream and
returned that gateway's response.

```text
HTTP Request: POST http://127.0.0.1:41521/v1/chat/completions "HTTP/1.0 200 OK"
HTTP Request: POST http://127.0.0.1:37439/v1/chat/completions "HTTP/1.0 200 OK"
responses=['gateway-A', 'gateway-B']
upstream_calls=[1, 1]
```

- Not tested: external/cloud OpenAI-compatible gateways; streaming
requests; full repository test suite.

## Runtime Rollout Safety

- Rollout-managed feature(s): Semantic response caching with custom
upstream routing.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: cache partitioning changes only when
the resolved upstream differs.
- Kill switch / disable path: disable semantic caching with
`cache_enabled=False`.
- Unsafe override required: No.
- Qualification impact: existing cache entries remain readable only
under their previous key; affected requests repopulate separate entries.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

Documentation is unchanged because no public configuration or API
contract changes. The warning above is the existing Starlette
`TestClient` deprecation warning. The refreshed branch also updates the
seeded cache-hit regression to include the new `upstream_base_url` key
field; all hosted CI test shards passed on `6cc2d648`.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-25 09:51:09 -05:00

229 lines
8.1 KiB
Python

"""Integration RBP for the SemanticCache key fix.
Drives the real ``/v1/messages`` handler path with the cache enabled and a mocked
upstream, proving end to end that a second request with the same messages but a
different ``system`` prompt is NOT served the first request's cached response
(no cross-request contamination), while a repeat of the first request IS served
from cache. This is the deterministic stand-in for a live real-upstream e2e
(no API credits, fully reproducible) and covers what the cache-key unit tests
cannot: that the handler actually threads the response-shaping fields into the
cache get/set calls.
Before the fix the cache key omitted ``system``, so request B collided with
request A: it returned A's response and the upstream was never called.
"""
from __future__ import annotations
import json
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
def _make_cached_proxy_client() -> TestClient:
config = ProxyConfig(
optimize=False,
cache_enabled=True,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
)
return TestClient(create_app(config))
def _body(system: str) -> dict:
return {
"model": "claude-haiku-4-5",
"max_tokens": 64,
"system": system,
"messages": [{"role": "user", "content": "Say hi."}],
"stream": False,
}
def _text(response: httpx.Response) -> str:
return response.json()["content"][0]["text"]
def test_different_system_not_served_from_cache() -> None:
calls = {"n": 0}
with _make_cached_proxy_client() as client:
proxy = client.app.state.proxy
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
calls["n"] += 1
system = body.get("system")
sys_text = system if isinstance(system, str) else json.dumps(system)
text = "Bonjour" if "French" in sys_text else "Hello"
return httpx.Response(
200,
json={
"id": "msg_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": text}],
"usage": {
"input_tokens": 10,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
# A: French system -> upstream call 1, cached under the French key.
a = client.post("/v1/messages", headers=headers, json=_body("Answer only in French."))
assert a.status_code == 200
assert _text(a) == "Bonjour"
assert calls["n"] == 1
# B: English system, SAME messages -> must reach the upstream again, not
# be served A's cached French response. Before the fix this returned
# "Bonjour" with calls["n"] still 1 (the bug).
b = client.post("/v1/messages", headers=headers, json=_body("Answer only in English."))
assert b.status_code == 200
assert _text(b) == "Hello"
assert calls["n"] == 2
# A again: French system -> served from cache, upstream NOT called.
a2 = client.post("/v1/messages", headers=headers, json=_body("Answer only in French."))
assert a2.status_code == 200
assert _text(a2) == "Bonjour"
assert calls["n"] == 2
def _body_thinking(thinking: dict) -> dict:
return {
"model": "claude-haiku-4-5",
"max_tokens": 64,
"system": "You are helpful.",
"messages": [{"role": "user", "content": "Say hi."}],
"thinking": thinking,
"stream": False,
}
def test_different_thinking_not_served_from_cache() -> None:
"""Same system + messages, different ``thinking`` config -> B must reach the
upstream, not be served A's cached response. ``thinking`` is the field the
#1473 review called out as still missing from the Anthropic key."""
calls = {"n": 0}
with _make_cached_proxy_client() as client:
proxy = client.app.state.proxy
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
calls["n"] += 1
return httpx.Response(
200,
json={
"id": "msg_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": f"resp-{calls['n']}"}],
"usage": {
"input_tokens": 10,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
enabled = {"type": "enabled", "budget_tokens": 2048}
disabled = {"type": "disabled"}
# A: thinking enabled -> upstream call 1, cached under A's key.
a = client.post("/v1/messages", headers=headers, json=_body_thinking(enabled))
assert a.status_code == 200
assert _text(a) == "resp-1"
assert calls["n"] == 1
# B: thinking disabled, SAME messages -> must reach the upstream again.
b = client.post("/v1/messages", headers=headers, json=_body_thinking(disabled))
assert b.status_code == 200
assert _text(b) == "resp-2"
assert calls["n"] == 2
# A again -> served from cache, upstream NOT called.
a2 = client.post("/v1/messages", headers=headers, json=_body_thinking(enabled))
assert a2.status_code == 200
assert _text(a2) == "resp-1"
assert calls["n"] == 2
def test_different_custom_upstream_not_served_from_cache(monkeypatch) -> None:
"""The response producer is part of cache identity (#3346)."""
calls: list[str] = []
async def allow_example_upstream(_url: str) -> bool:
return True
monkeypatch.setattr(
"headroom.providers.proxy_routes.is_safe_upstream_url_async",
allow_example_upstream,
)
with _make_cached_proxy_client() as client:
proxy = client.app.state.proxy
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
calls.append(str(url))
text = "gateway-A" if "gateway-a" in str(url) else "gateway-B"
return httpx.Response(
200,
json={
"id": f"msg_{len(calls)}",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": text}],
"usage": {
"input_tokens": 10,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
body = _body("Answer with the gateway identity.")
first = client.post(
"/v1/messages",
headers={**headers, "x-headroom-base-url": "https://gateway-a.example"},
json=body,
)
second = client.post(
"/v1/messages",
headers={**headers, "x-headroom-base-url": "https://gateway-b.example"},
json=body,
)
assert first.status_code == 200
assert second.status_code == 200
assert _text(first) == "gateway-A"
assert _text(second) == "gateway-B"
assert calls == [
"https://gateway-a.example/v1/messages",
"https://gateway-b.example/v1/messages",
]