mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Include the resolved custom upstream base URL in semantic response-cache identity for Anthropic and OpenAI chat handlers. Identical prompts sent to different compatible gateways can legitimately produce different responses and must not share cached output. Closes #3346 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `upstream_base_url` to Anthropic response-cache key fields. - Add `upstream_base_url` to OpenAI chat response-cache key fields. - Add handler-level regressions for two custom gateways receiving identical requests. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest -q tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py 8 passed, 1 warning in 2.76s $ .venv/bin/ruff check <changed files> All checks passed! $ .venv/bin/ruff format --check <changed files> 4 files already formatted $ .venv/bin/mypy headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Linux, Python 3.14, real FastAPI proxy app and real HTTP transport, two local OpenAI-compatible HTTP upstreams, semantic cache enabled. - Exact command / steps: allowlisted two loopback origins with `HEADROOM_ALLOWED_BASE_URLS`, started an HTTP server on each origin returning its gateway identity, then posted the same `/v1/chat/completions` body twice with only `x-headroom-base-url` changed. - Observed result: each request reached its selected upstream and returned that gateway's response. ```text HTTP Request: POST http://127.0.0.1:41521/v1/chat/completions "HTTP/1.0 200 OK" HTTP Request: POST http://127.0.0.1:37439/v1/chat/completions "HTTP/1.0 200 OK" responses=['gateway-A', 'gateway-B'] upstream_calls=[1, 1] ``` - Not tested: external/cloud OpenAI-compatible gateways; streaming requests; full repository test suite. ## Runtime Rollout Safety - Rollout-managed feature(s): Semantic response caching with custom upstream routing. - Minimum rollout channel: N/A. - Stable/default behavior changed: cache partitioning changes only when the resolved upstream differs. - Kill switch / disable path: disable semantic caching with `cache_enabled=False`. - Unsafe override required: No. - Qualification impact: existing cache entries remain readable only under their previous key; affected requests repopulate separate entries. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes Documentation is unchanged because no public configuration or API contract changes. The warning above is the existing Starlette `TestClient` deprecation warning. The refreshed branch also updates the seeded cache-hit regression to include the new `upstream_base_url` key field; all hosted CI test shards passed on `6cc2d648`. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
229 lines
8.1 KiB
Python
229 lines
8.1 KiB
Python
"""Integration RBP for the SemanticCache key fix.
|
|
|
|
Drives the real ``/v1/messages`` handler path with the cache enabled and a mocked
|
|
upstream, proving end to end that a second request with the same messages but a
|
|
different ``system`` prompt is NOT served the first request's cached response
|
|
(no cross-request contamination), while a repeat of the first request IS served
|
|
from cache. This is the deterministic stand-in for a live real-upstream e2e
|
|
(no API credits, fully reproducible) and covers what the cache-key unit tests
|
|
cannot: that the handler actually threads the response-shaping fields into the
|
|
cache get/set calls.
|
|
|
|
Before the fix the cache key omitted ``system``, so request B collided with
|
|
request A: it returned A's response and the upstream was never called.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
|
|
def _make_cached_proxy_client() -> TestClient:
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=True,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
)
|
|
return TestClient(create_app(config))
|
|
|
|
|
|
def _body(system: str) -> dict:
|
|
return {
|
|
"model": "claude-haiku-4-5",
|
|
"max_tokens": 64,
|
|
"system": system,
|
|
"messages": [{"role": "user", "content": "Say hi."}],
|
|
"stream": False,
|
|
}
|
|
|
|
|
|
def _text(response: httpx.Response) -> str:
|
|
return response.json()["content"][0]["text"]
|
|
|
|
|
|
def test_different_system_not_served_from_cache() -> None:
|
|
calls = {"n": 0}
|
|
|
|
with _make_cached_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
calls["n"] += 1
|
|
system = body.get("system")
|
|
sys_text = system if isinstance(system, str) else json.dumps(system)
|
|
text = "Bonjour" if "French" in sys_text else "Hello"
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": text}],
|
|
"usage": {
|
|
"input_tokens": 10,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
|
|
|
|
# A: French system -> upstream call 1, cached under the French key.
|
|
a = client.post("/v1/messages", headers=headers, json=_body("Answer only in French."))
|
|
assert a.status_code == 200
|
|
assert _text(a) == "Bonjour"
|
|
assert calls["n"] == 1
|
|
|
|
# B: English system, SAME messages -> must reach the upstream again, not
|
|
# be served A's cached French response. Before the fix this returned
|
|
# "Bonjour" with calls["n"] still 1 (the bug).
|
|
b = client.post("/v1/messages", headers=headers, json=_body("Answer only in English."))
|
|
assert b.status_code == 200
|
|
assert _text(b) == "Hello"
|
|
assert calls["n"] == 2
|
|
|
|
# A again: French system -> served from cache, upstream NOT called.
|
|
a2 = client.post("/v1/messages", headers=headers, json=_body("Answer only in French."))
|
|
assert a2.status_code == 200
|
|
assert _text(a2) == "Bonjour"
|
|
assert calls["n"] == 2
|
|
|
|
|
|
def _body_thinking(thinking: dict) -> dict:
|
|
return {
|
|
"model": "claude-haiku-4-5",
|
|
"max_tokens": 64,
|
|
"system": "You are helpful.",
|
|
"messages": [{"role": "user", "content": "Say hi."}],
|
|
"thinking": thinking,
|
|
"stream": False,
|
|
}
|
|
|
|
|
|
def test_different_thinking_not_served_from_cache() -> None:
|
|
"""Same system + messages, different ``thinking`` config -> B must reach the
|
|
upstream, not be served A's cached response. ``thinking`` is the field the
|
|
#1473 review called out as still missing from the Anthropic key."""
|
|
calls = {"n": 0}
|
|
|
|
with _make_cached_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
calls["n"] += 1
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": f"resp-{calls['n']}"}],
|
|
"usage": {
|
|
"input_tokens": 10,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
|
|
enabled = {"type": "enabled", "budget_tokens": 2048}
|
|
disabled = {"type": "disabled"}
|
|
|
|
# A: thinking enabled -> upstream call 1, cached under A's key.
|
|
a = client.post("/v1/messages", headers=headers, json=_body_thinking(enabled))
|
|
assert a.status_code == 200
|
|
assert _text(a) == "resp-1"
|
|
assert calls["n"] == 1
|
|
|
|
# B: thinking disabled, SAME messages -> must reach the upstream again.
|
|
b = client.post("/v1/messages", headers=headers, json=_body_thinking(disabled))
|
|
assert b.status_code == 200
|
|
assert _text(b) == "resp-2"
|
|
assert calls["n"] == 2
|
|
|
|
# A again -> served from cache, upstream NOT called.
|
|
a2 = client.post("/v1/messages", headers=headers, json=_body_thinking(enabled))
|
|
assert a2.status_code == 200
|
|
assert _text(a2) == "resp-1"
|
|
assert calls["n"] == 2
|
|
|
|
|
|
def test_different_custom_upstream_not_served_from_cache(monkeypatch) -> None:
|
|
"""The response producer is part of cache identity (#3346)."""
|
|
calls: list[str] = []
|
|
|
|
async def allow_example_upstream(_url: str) -> bool:
|
|
return True
|
|
|
|
monkeypatch.setattr(
|
|
"headroom.providers.proxy_routes.is_safe_upstream_url_async",
|
|
allow_example_upstream,
|
|
)
|
|
|
|
with _make_cached_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
calls.append(str(url))
|
|
text = "gateway-A" if "gateway-a" in str(url) else "gateway-B"
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": f"msg_{len(calls)}",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": text}],
|
|
"usage": {
|
|
"input_tokens": 10,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
|
|
body = _body("Answer with the gateway identity.")
|
|
|
|
first = client.post(
|
|
"/v1/messages",
|
|
headers={**headers, "x-headroom-base-url": "https://gateway-a.example"},
|
|
json=body,
|
|
)
|
|
second = client.post(
|
|
"/v1/messages",
|
|
headers={**headers, "x-headroom-base-url": "https://gateway-b.example"},
|
|
json=body,
|
|
)
|
|
|
|
assert first.status_code == 200
|
|
assert second.status_code == 200
|
|
assert _text(first) == "gateway-A"
|
|
assert _text(second) == "gateway-B"
|
|
assert calls == [
|
|
"https://gateway-a.example/v1/messages",
|
|
"https://gateway-b.example/v1/messages",
|
|
]
|