Files
headroom/tests/test_proxy_response_cache_store_guard.py
1132a54b21 fix(proxy): do not cache error replies delivered as http 200 (#3930)
## Description

The response cache stores any non-streaming reply whose upstream status
is 200. An Anthropic- or OpenAI-compatible gateway can answer 200 with
an error object (`{"type": "error", ...}` / `{"error": {...}}`) or an
empty object, and that body was then replayed from the cache to every
matching request for the full TTL (1h by default) without the upstream
being asked again. Only a proxy restart or `POST /cache/clear`
recovered.

`SemanticCache.set` now refuses an empty body and any JSON body that is
not a non-empty object or that carries an error. Every store goes
through `set` (the Anthropic and OpenAI chat handlers are its only
callers), so one check covers both instead of a guard at each call site.

Closes #3929

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/semantic_cache.py`: new
`_is_cacheable_reply(response_body)`; `SemanticCache.set` returns
without storing when it is false. Refused: empty bytes, JSON that is not
a dict, `{}`, a dict with `"type": "error"`, a dict with a non-null
`"error"`. A reply with `"error": null` is still cached. Bytes that are
not JSON are stored as before: both handlers only store a reply they
already decoded, and existing cache tests store opaque bytes.
- `tests/test_proxy_response_cache_store_guard.py`: parametrized unit
test of `_is_cacheable_reply`, plus an end-to-end test per handler
(`/v1/messages`, `/v1/chat/completions`) with the cache on and an
upstream that answers 200 + error body once, then a good body. It
asserts the second request reaches the upstream and the good reply is
then served from cache.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Lint, format and type checks ran on the changed files only. Pytest ran
the new file plus the existing response-cache test files, not the whole
suite.

### Test Output

Before the fix (`semantic_cache.py` at upstream/main d5318ac2, plus a
pass-through `_is_cacheable_reply` stub so the new test module imports):

```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py -q -p no:cacheprovider -k replayed
E   assert 1 == 2
E   assert 1 == 2
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/messages-headers0-claude-haiku-4-5-error_body0-ok_body0]
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/chat/completions-headers1-gpt-4o-mini-error_body1-ok_body1]
================= 2 failed, 10 deselected, 1 warning in 9.28s ==================
```

After the fix:

```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py tests/test_proxy_semantic_cache_key.py \
    tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py \
    tests/test_openai_response_cache_key.py tests/test_proxy_semantic_cache_key_policy.py \
    tests/test_semantic_cache_key_policy.py -q -x -p no:cacheprovider
======================== 61 passed, 1 warning in 16.55s ========================

$ python -m pytest tests/test_proxy_response_cache_replay.py tests/test_ccr_buffered_stream_signed_thinking.py \
    tests/test_handler_outcome_tag_invariant.py tests/test_read_maturation_handler_nobust.py \
    tests/test_observability_metrics.py -q -p no:cacheprovider
=================== 45 passed, 1 warning in 69.84s (0:01:09) ===================

$ ruff check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
All checks passed!
$ ruff format --check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
2 files already formatted
$ mypy headroom/proxy/semantic_cache.py
Success: no issues found in 1 source file
```

The warning is the existing `StarletteDeprecationWarning` from
`fastapi.testclient`.

## Real Behavior Proof

- Environment: macOS, Python 3.11.13. The real `headroom proxy` CLI,
booted from the source tree with the default `ProxyConfig` (cache on,
TTL 3600s, `Mode: cache`), pointed at a local stub upstream on 127.0.0.1
via `--anthropic-api-url` / `--openai-api-url`. The stub answers the
first POST per path with HTTP 200 + an error body (`{"type": "error",
"error": {"type": "overloaded_error", ...}}` for `/v1/messages`,
`{"error": {"message": "Upstream overloaded", "type": "server_error",
"code": null}}` for `/v1/chat/completions`) and every later POST with a
normal reply. The scratch venv lacks the `mcp`, `magika` and
`onnxruntime` extras, so the CLI was started through a three-line
launcher that skips `ensure_proxy_dependencies()` and then calls the
stock `headroom.cli.main.main(["proxy", ...])`. None of those modules is
on the response-cache path.
- Exact command / steps: `python launch.py --port <p>
--anthropic-api-url http://127.0.0.1:<s> --openai-api-url
http://127.0.0.1:<s>/v1`, then three identical non-streaming requests
per endpoint, then `GET /stats`. Run once with upstream/main d5318ac2
and once with this branch, same venv and stub.
- Observed result: on main the stub is hit once per path and requests 2
and 3 replay the stored error from cache; on this branch request 2
reaches the stub and gets the good reply, and request 3 is served that
reply from cache (output below).

upstream/main (d5318ac2): the stub is hit once per path; requests 2 and
3 get the stored error from cache.

  ```text
  stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 668ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 2 -> 200 in 26ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 3 -> 200 in 22ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
  stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 61ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 2 -> 200 in 12ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 3 -> 200 in 24ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 4,
"ttl_seconds": 3600}
  ```

This branch: request 2 reaches the stub and gets the good reply; request
3 is served that good reply from cache.

  ```text
  stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 1010ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
  stub: /v1/messages hit #2 -> 200 good body
client: /v1/messages request 2 -> 200 in 304ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
client: /v1/messages request 3 -> 200 in 115ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
  stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 51ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
  stub: /v1/chat/completions hit #2 -> 200 good body
client: /v1/chat/completions request 2 -> 200 in 21ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
client: /v1/chat/completions request 3 -> 200 in 20ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 2,
"ttl_seconds": 3600}
  ```

Proxy `PERF` lines match: on main requests 2-3 log `tok_before=0` (cache
hit) on both models; on this branch request 2 logs `tok_before=10`
(forwarded) and only request 3 logs `tok_before=0`.
- Not tested: real Anthropic/OpenAI or a real third-party gateway (stub
only, no credentials used); streaming requests (they never read or write
this cache); the full test suite (targeted files only); the empty /
unparseable 200 seen in production, because on main neither handler can
pass such a body to `set` any more (Anthropic requires `resp_json is not
None`, OpenAI raises to its 502 path), so that case is covered by the
unit test only.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a correctness guard in
`SemanticCache.set`, not a rollout feature.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes, narrowly. With the cache on (the
default), an HTTP 200 reply that is empty, an empty JSON object, a
non-object JSON value, or a JSON object carrying `"type": "error"` or a
non-null `"error"` is no longer stored, so the next identical request
goes upstream instead of replaying it. The reply itself is still
returned to the client unchanged. Successful replies are cached exactly
as before.
- Kill switch / disable path: none added; there is no case where
replaying a stored error is wanted. `--no-cache` still disables the
cache as a whole.
- Unsafe override required: no.
- Qualification impact: none expected. Compression, prompt-cache prefix
handling and token accounting are untouched; only which bodies the
response cache keeps changes. Cache-hit counts drop only where the hits
were replayed errors.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- Documentation: N/A. No user-facing option or documented behavior
changes.
- Overlap with #3862 (partition the response cache by credential and
principal): both touch `SemanticCache.set`, but this change only adds
lines after the key computation and does not touch the signature,
`_compute_key` or `get`. A three-way merge of `semantic_cache.py`
against #3862's head is clean. The new tests call `_is_cacheable_reply`
directly and go through the handlers for the end-to-end case, so they do
not depend on whether `set`/`get` take `partition=`.
- History: the production case that surfaced this was the malformed
buffered-CCR 200 that #2997 turned into a real 502. That source is gone
on main, but a 200 with a JSON error payload from a compatible gateway
still reaches `set`, as the proof above shows.
- The check parses the body once per store. Stores only happen on
non-streaming 200 replies, so the cost is one `json.loads` per such
reply.
- Headroom Desktop has shipped an equivalent downstream patch since
2026-08-13.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3930"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:56:40 -07:00

123 lines
3.8 KiB
Python

"""The response cache must not store an error delivered as HTTP 200.
Both handlers gate ``SemanticCache.set`` on ``status_code == 200``. Anthropic- and
OpenAI-compatible gateways can answer 200 with an error object (or an empty
object), and once stored that body is replayed to every matching non-streaming
request for the full TTL without the upstream ever being asked again.
"""
from __future__ import annotations
import json
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from headroom.proxy.semantic_cache import _is_cacheable_reply
from headroom.proxy.server import ProxyConfig, create_app
MESSAGES = [{"role": "user", "content": "Say hi."}]
ANTHROPIC_ERROR = {"type": "error", "error": {"type": "api_error", "message": "boom"}}
OPENAI_ERROR = {"error": {"message": "boom", "type": "server_error", "code": None}}
ANTHROPIC_OK = {
"id": "msg_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "Hello"}],
"usage": {"input_tokens": 10, "output_tokens": 3},
}
OPENAI_OK = {
"id": "chatcmpl_1",
"object": "chat.completion",
"choices": [
{"index": 0, "message": {"role": "assistant", "content": "Hello"}, "finish_reason": "stop"}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 3, "total_tokens": 13},
}
@pytest.mark.parametrize(
"body,cacheable",
[
(json.dumps(ANTHROPIC_OK).encode(), True),
(json.dumps(OPENAI_OK).encode(), True),
(json.dumps({**OPENAI_OK, "error": None}).encode(), True),
(json.dumps(ANTHROPIC_ERROR).encode(), False),
(json.dumps(OPENAI_ERROR).encode(), False),
(b"", False),
(b"opaque-bytes", True),
(b"{}", False),
(b"null", False),
(b"[]", False),
],
)
def test_is_cacheable_reply(body: bytes, cacheable: bool) -> None:
assert _is_cacheable_reply(body) is cacheable
def _client() -> TestClient:
config = ProxyConfig(
optimize=False,
cache_enabled=True,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
)
return TestClient(create_app(config))
@pytest.mark.parametrize(
"path,headers,model,error_body,ok_body",
[
(
"/v1/messages",
{"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
"claude-haiku-4-5",
ANTHROPIC_ERROR,
ANTHROPIC_OK,
),
(
"/v1/chat/completions",
{"authorization": "Bearer test-key"},
"gpt-4o-mini",
OPENAI_ERROR,
OPENAI_OK,
),
],
)
def test_error_200_is_not_replayed_from_cache(path, headers, model, error_body, ok_body) -> None:
calls = {"n": 0}
with _client() as client:
async def _fake_retry(method, url, req_headers, body, stream=False, **kwargs): # noqa: ANN001
calls["n"] += 1
return httpx.Response(200, json=error_body if calls["n"] == 1 else ok_body)
client.app.state.proxy._retry_request = _fake_retry
request = {"model": model, "max_tokens": 64, "messages": MESSAGES, "stream": False}
first = client.post(path, headers=headers, json=request)
assert first.json() == error_body
# Before the fix the error body was cached: this came back from the
# cache with the upstream never called a second time.
second = client.post(path, headers=headers, json=request)
assert calls["n"] == 2
assert second.json() == ok_body
# A real reply is still cached.
third = client.post(path, headers=headers, json=request)
assert calls["n"] == 2
assert third.json() == ok_body