mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
The response cache stores any non-streaming reply whose upstream status
is 200. An Anthropic- or OpenAI-compatible gateway can answer 200 with
an error object (`{"type": "error", ...}` / `{"error": {...}}`) or an
empty object, and that body was then replayed from the cache to every
matching request for the full TTL (1h by default) without the upstream
being asked again. Only a proxy restart or `POST /cache/clear`
recovered.
`SemanticCache.set` now refuses an empty body and any JSON body that is
not a non-empty object or that carries an error. Every store goes
through `set` (the Anthropic and OpenAI chat handlers are its only
callers), so one check covers both instead of a guard at each call site.
Closes #3929
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/semantic_cache.py`: new
`_is_cacheable_reply(response_body)`; `SemanticCache.set` returns
without storing when it is false. Refused: empty bytes, JSON that is not
a dict, `{}`, a dict with `"type": "error"`, a dict with a non-null
`"error"`. A reply with `"error": null` is still cached. Bytes that are
not JSON are stored as before: both handlers only store a reply they
already decoded, and existing cache tests store opaque bytes.
- `tests/test_proxy_response_cache_store_guard.py`: parametrized unit
test of `_is_cacheable_reply`, plus an end-to-end test per handler
(`/v1/messages`, `/v1/chat/completions`) with the cache on and an
upstream that answers 200 + error body once, then a good body. It
asserts the second request reaches the upstream and the good reply is
then served from cache.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Lint, format and type checks ran on the changed files only. Pytest ran
the new file plus the existing response-cache test files, not the whole
suite.
### Test Output
Before the fix (`semantic_cache.py` at upstream/main d5318ac2, plus a
pass-through `_is_cacheable_reply` stub so the new test module imports):
```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py -q -p no:cacheprovider -k replayed
E assert 1 == 2
E assert 1 == 2
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/messages-headers0-claude-haiku-4-5-error_body0-ok_body0]
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/chat/completions-headers1-gpt-4o-mini-error_body1-ok_body1]
================= 2 failed, 10 deselected, 1 warning in 9.28s ==================
```
After the fix:
```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py tests/test_proxy_semantic_cache_key.py \
tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py \
tests/test_openai_response_cache_key.py tests/test_proxy_semantic_cache_key_policy.py \
tests/test_semantic_cache_key_policy.py -q -x -p no:cacheprovider
======================== 61 passed, 1 warning in 16.55s ========================
$ python -m pytest tests/test_proxy_response_cache_replay.py tests/test_ccr_buffered_stream_signed_thinking.py \
tests/test_handler_outcome_tag_invariant.py tests/test_read_maturation_handler_nobust.py \
tests/test_observability_metrics.py -q -p no:cacheprovider
=================== 45 passed, 1 warning in 69.84s (0:01:09) ===================
$ ruff check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
All checks passed!
$ ruff format --check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
2 files already formatted
$ mypy headroom/proxy/semantic_cache.py
Success: no issues found in 1 source file
```
The warning is the existing `StarletteDeprecationWarning` from
`fastapi.testclient`.
## Real Behavior Proof
- Environment: macOS, Python 3.11.13. The real `headroom proxy` CLI,
booted from the source tree with the default `ProxyConfig` (cache on,
TTL 3600s, `Mode: cache`), pointed at a local stub upstream on 127.0.0.1
via `--anthropic-api-url` / `--openai-api-url`. The stub answers the
first POST per path with HTTP 200 + an error body (`{"type": "error",
"error": {"type": "overloaded_error", ...}}` for `/v1/messages`,
`{"error": {"message": "Upstream overloaded", "type": "server_error",
"code": null}}` for `/v1/chat/completions`) and every later POST with a
normal reply. The scratch venv lacks the `mcp`, `magika` and
`onnxruntime` extras, so the CLI was started through a three-line
launcher that skips `ensure_proxy_dependencies()` and then calls the
stock `headroom.cli.main.main(["proxy", ...])`. None of those modules is
on the response-cache path.
- Exact command / steps: `python launch.py --port <p>
--anthropic-api-url http://127.0.0.1:<s> --openai-api-url
http://127.0.0.1:<s>/v1`, then three identical non-streaming requests
per endpoint, then `GET /stats`. Run once with upstream/main d5318ac2
and once with this branch, same venv and stub.
- Observed result: on main the stub is hit once per path and requests 2
and 3 replay the stored error from cache; on this branch request 2
reaches the stub and gets the good reply, and request 3 is served that
reply from cache (output below).
upstream/main (d5318ac2): the stub is hit once per path; requests 2 and
3 get the stored error from cache.
```text
stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 668ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 2 -> 200 in 26ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 3 -> 200 in 22ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 61ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 2 -> 200 in 12ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 3 -> 200 in 24ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 4,
"ttl_seconds": 3600}
```
This branch: request 2 reaches the stub and gets the good reply; request
3 is served that good reply from cache.
```text
stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 1010ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
stub: /v1/messages hit #2 -> 200 good body
client: /v1/messages request 2 -> 200 in 304ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
client: /v1/messages request 3 -> 200 in 115ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 51ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
stub: /v1/chat/completions hit #2 -> 200 good body
client: /v1/chat/completions request 2 -> 200 in 21ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
client: /v1/chat/completions request 3 -> 200 in 20ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 2,
"ttl_seconds": 3600}
```
Proxy `PERF` lines match: on main requests 2-3 log `tok_before=0` (cache
hit) on both models; on this branch request 2 logs `tok_before=10`
(forwarded) and only request 3 logs `tok_before=0`.
- Not tested: real Anthropic/OpenAI or a real third-party gateway (stub
only, no credentials used); streaming requests (they never read or write
this cache); the full test suite (targeted files only); the empty /
unparseable 200 seen in production, because on main neither handler can
pass such a body to `set` any more (Anthropic requires `resp_json is not
None`, OpenAI raises to its 502 path), so that case is covered by the
unit test only.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is a correctness guard in
`SemanticCache.set`, not a rollout feature.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes, narrowly. With the cache on (the
default), an HTTP 200 reply that is empty, an empty JSON object, a
non-object JSON value, or a JSON object carrying `"type": "error"` or a
non-null `"error"` is no longer stored, so the next identical request
goes upstream instead of replaying it. The reply itself is still
returned to the client unchanged. Successful replies are cached exactly
as before.
- Kill switch / disable path: none added; there is no case where
replaying a stored error is wanted. `--no-cache` still disables the
cache as a whole.
- Unsafe override required: no.
- Qualification impact: none expected. Compression, prompt-cache prefix
handling and token accounting are untouched; only which bodies the
response cache keeps changes. Cache-hit counts drop only where the hits
were replayed errors.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A.
## Additional Notes
- Documentation: N/A. No user-facing option or documented behavior
changes.
- Overlap with #3862 (partition the response cache by credential and
principal): both touch `SemanticCache.set`, but this change only adds
lines after the key computation and does not touch the signature,
`_compute_key` or `get`. A three-way merge of `semantic_cache.py`
against #3862's head is clean. The new tests call `_is_cacheable_reply`
directly and go through the handlers for the end-to-end case, so they do
not depend on whether `set`/`get` take `partition=`.
- History: the production case that surfaced this was the malformed
buffered-CCR 200 that #2997 turned into a real 502. That source is gone
on main, but a 200 with a JSON error payload from a compatible gateway
still reaches `set`, as the proof above shows.
- The check parses the body once per store. Stores only happen on
non-streaming 200 replies, so the cost is one `json.loads` per such
reply.
- Headroom Desktop has shipped an equivalent downstream patch since
2026-08-13.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3930"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
123 lines
3.8 KiB
Python
123 lines
3.8 KiB
Python
"""The response cache must not store an error delivered as HTTP 200.
|
|
|
|
Both handlers gate ``SemanticCache.set`` on ``status_code == 200``. Anthropic- and
|
|
OpenAI-compatible gateways can answer 200 with an error object (or an empty
|
|
object), and once stored that body is replayed to every matching non-streaming
|
|
request for the full TTL without the upstream ever being asked again.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.semantic_cache import _is_cacheable_reply
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
MESSAGES = [{"role": "user", "content": "Say hi."}]
|
|
|
|
ANTHROPIC_ERROR = {"type": "error", "error": {"type": "api_error", "message": "boom"}}
|
|
OPENAI_ERROR = {"error": {"message": "boom", "type": "server_error", "code": None}}
|
|
|
|
ANTHROPIC_OK = {
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "Hello"}],
|
|
"usage": {"input_tokens": 10, "output_tokens": 3},
|
|
}
|
|
OPENAI_OK = {
|
|
"id": "chatcmpl_1",
|
|
"object": "chat.completion",
|
|
"choices": [
|
|
{"index": 0, "message": {"role": "assistant", "content": "Hello"}, "finish_reason": "stop"}
|
|
],
|
|
"usage": {"prompt_tokens": 10, "completion_tokens": 3, "total_tokens": 13},
|
|
}
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"body,cacheable",
|
|
[
|
|
(json.dumps(ANTHROPIC_OK).encode(), True),
|
|
(json.dumps(OPENAI_OK).encode(), True),
|
|
(json.dumps({**OPENAI_OK, "error": None}).encode(), True),
|
|
(json.dumps(ANTHROPIC_ERROR).encode(), False),
|
|
(json.dumps(OPENAI_ERROR).encode(), False),
|
|
(b"", False),
|
|
(b"opaque-bytes", True),
|
|
(b"{}", False),
|
|
(b"null", False),
|
|
(b"[]", False),
|
|
],
|
|
)
|
|
def test_is_cacheable_reply(body: bytes, cacheable: bool) -> None:
|
|
assert _is_cacheable_reply(body) is cacheable
|
|
|
|
|
|
def _client() -> TestClient:
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=True,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
)
|
|
return TestClient(create_app(config))
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"path,headers,model,error_body,ok_body",
|
|
[
|
|
(
|
|
"/v1/messages",
|
|
{"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
"claude-haiku-4-5",
|
|
ANTHROPIC_ERROR,
|
|
ANTHROPIC_OK,
|
|
),
|
|
(
|
|
"/v1/chat/completions",
|
|
{"authorization": "Bearer test-key"},
|
|
"gpt-4o-mini",
|
|
OPENAI_ERROR,
|
|
OPENAI_OK,
|
|
),
|
|
],
|
|
)
|
|
def test_error_200_is_not_replayed_from_cache(path, headers, model, error_body, ok_body) -> None:
|
|
calls = {"n": 0}
|
|
|
|
with _client() as client:
|
|
|
|
async def _fake_retry(method, url, req_headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
calls["n"] += 1
|
|
return httpx.Response(200, json=error_body if calls["n"] == 1 else ok_body)
|
|
|
|
client.app.state.proxy._retry_request = _fake_retry
|
|
request = {"model": model, "max_tokens": 64, "messages": MESSAGES, "stream": False}
|
|
|
|
first = client.post(path, headers=headers, json=request)
|
|
assert first.json() == error_body
|
|
|
|
# Before the fix the error body was cached: this came back from the
|
|
# cache with the upstream never called a second time.
|
|
second = client.post(path, headers=headers, json=request)
|
|
assert calls["n"] == 2
|
|
assert second.json() == ok_body
|
|
|
|
# A real reply is still cached.
|
|
third = client.post(path, headers=headers, json=request)
|
|
assert calls["n"] == 2
|
|
assert third.json() == ok_body
|