fix(proxy): do not cache error replies delivered as http 200 (#3930)

## Description

The response cache stores any non-streaming reply whose upstream status
is 200. An Anthropic- or OpenAI-compatible gateway can answer 200 with
an error object (`{"type": "error", ...}` / `{"error": {...}}`) or an
empty object, and that body was then replayed from the cache to every
matching request for the full TTL (1h by default) without the upstream
being asked again. Only a proxy restart or `POST /cache/clear`
recovered.

`SemanticCache.set` now refuses an empty body and any JSON body that is
not a non-empty object or that carries an error. Every store goes
through `set` (the Anthropic and OpenAI chat handlers are its only
callers), so one check covers both instead of a guard at each call site.

Closes #3929

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/semantic_cache.py`: new
`_is_cacheable_reply(response_body)`; `SemanticCache.set` returns
without storing when it is false. Refused: empty bytes, JSON that is not
a dict, `{}`, a dict with `"type": "error"`, a dict with a non-null
`"error"`. A reply with `"error": null` is still cached. Bytes that are
not JSON are stored as before: both handlers only store a reply they
already decoded, and existing cache tests store opaque bytes.
- `tests/test_proxy_response_cache_store_guard.py`: parametrized unit
test of `_is_cacheable_reply`, plus an end-to-end test per handler
(`/v1/messages`, `/v1/chat/completions`) with the cache on and an
upstream that answers 200 + error body once, then a good body. It
asserts the second request reaches the upstream and the good reply is
then served from cache.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Lint, format and type checks ran on the changed files only. Pytest ran
the new file plus the existing response-cache test files, not the whole
suite.

### Test Output

Before the fix (`semantic_cache.py` at upstream/main d5318ac2, plus a
pass-through `_is_cacheable_reply` stub so the new test module imports):

```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py -q -p no:cacheprovider -k replayed
E   assert 1 == 2
E   assert 1 == 2
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/messages-headers0-claude-haiku-4-5-error_body0-ok_body0]
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/chat/completions-headers1-gpt-4o-mini-error_body1-ok_body1]
================= 2 failed, 10 deselected, 1 warning in 9.28s ==================
```

After the fix:

```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py tests/test_proxy_semantic_cache_key.py \
    tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py \
    tests/test_openai_response_cache_key.py tests/test_proxy_semantic_cache_key_policy.py \
    tests/test_semantic_cache_key_policy.py -q -x -p no:cacheprovider
======================== 61 passed, 1 warning in 16.55s ========================

$ python -m pytest tests/test_proxy_response_cache_replay.py tests/test_ccr_buffered_stream_signed_thinking.py \
    tests/test_handler_outcome_tag_invariant.py tests/test_read_maturation_handler_nobust.py \
    tests/test_observability_metrics.py -q -p no:cacheprovider
=================== 45 passed, 1 warning in 69.84s (0:01:09) ===================

$ ruff check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
All checks passed!
$ ruff format --check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
2 files already formatted
$ mypy headroom/proxy/semantic_cache.py
Success: no issues found in 1 source file
```

The warning is the existing `StarletteDeprecationWarning` from
`fastapi.testclient`.

## Real Behavior Proof

- Environment: macOS, Python 3.11.13. The real `headroom proxy` CLI,
booted from the source tree with the default `ProxyConfig` (cache on,
TTL 3600s, `Mode: cache`), pointed at a local stub upstream on 127.0.0.1
via `--anthropic-api-url` / `--openai-api-url`. The stub answers the
first POST per path with HTTP 200 + an error body (`{"type": "error",
"error": {"type": "overloaded_error", ...}}` for `/v1/messages`,
`{"error": {"message": "Upstream overloaded", "type": "server_error",
"code": null}}` for `/v1/chat/completions`) and every later POST with a
normal reply. The scratch venv lacks the `mcp`, `magika` and
`onnxruntime` extras, so the CLI was started through a three-line
launcher that skips `ensure_proxy_dependencies()` and then calls the
stock `headroom.cli.main.main(["proxy", ...])`. None of those modules is
on the response-cache path.
- Exact command / steps: `python launch.py --port <p>
--anthropic-api-url http://127.0.0.1:<s> --openai-api-url
http://127.0.0.1:<s>/v1`, then three identical non-streaming requests
per endpoint, then `GET /stats`. Run once with upstream/main d5318ac2
and once with this branch, same venv and stub.
- Observed result: on main the stub is hit once per path and requests 2
and 3 replay the stored error from cache; on this branch request 2
reaches the stub and gets the good reply, and request 3 is served that
reply from cache (output below).

upstream/main (d5318ac2): the stub is hit once per path; requests 2 and
3 get the stored error from cache.

  ```text
  stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 668ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 2 -> 200 in 26ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 3 -> 200 in 22ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
  stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 61ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 2 -> 200 in 12ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 3 -> 200 in 24ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 4,
"ttl_seconds": 3600}
  ```

This branch: request 2 reaches the stub and gets the good reply; request
3 is served that good reply from cache.

  ```text
  stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 1010ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
  stub: /v1/messages hit #2 -> 200 good body
client: /v1/messages request 2 -> 200 in 304ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
client: /v1/messages request 3 -> 200 in 115ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
  stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 51ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
  stub: /v1/chat/completions hit #2 -> 200 good body
client: /v1/chat/completions request 2 -> 200 in 21ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
client: /v1/chat/completions request 3 -> 200 in 20ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 2,
"ttl_seconds": 3600}
  ```

Proxy `PERF` lines match: on main requests 2-3 log `tok_before=0` (cache
hit) on both models; on this branch request 2 logs `tok_before=10`
(forwarded) and only request 3 logs `tok_before=0`.
- Not tested: real Anthropic/OpenAI or a real third-party gateway (stub
only, no credentials used); streaming requests (they never read or write
this cache); the full test suite (targeted files only); the empty /
unparseable 200 seen in production, because on main neither handler can
pass such a body to `set` any more (Anthropic requires `resp_json is not
None`, OpenAI raises to its 502 path), so that case is covered by the
unit test only.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a correctness guard in
`SemanticCache.set`, not a rollout feature.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes, narrowly. With the cache on (the
default), an HTTP 200 reply that is empty, an empty JSON object, a
non-object JSON value, or a JSON object carrying `"type": "error"` or a
non-null `"error"` is no longer stored, so the next identical request
goes upstream instead of replaying it. The reply itself is still
returned to the client unchanged. Successful replies are cached exactly
as before.
- Kill switch / disable path: none added; there is no case where
replaying a stored error is wanted. `--no-cache` still disables the
cache as a whole.
- Unsafe override required: no.
- Qualification impact: none expected. Compression, prompt-cache prefix
handling and token accounting are untouched; only which bodies the
response cache keeps changes. Cache-hit counts drop only where the hits
were replayed errors.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- Documentation: N/A. No user-facing option or documented behavior
changes.
- Overlap with #3862 (partition the response cache by credential and
principal): both touch `SemanticCache.set`, but this change only adds
lines after the key computation and does not touch the signature,
`_compute_key` or `get`. A three-way merge of `semantic_cache.py`
against #3862's head is clean. The new tests call `_is_cacheable_reply`
directly and go through the handlers for the end-to-end case, so they do
not depend on whether `set`/`get` take `partition=`.
- History: the production case that surfaced this was the malformed
buffered-CCR 200 that #2997 turned into a real 502. That source is gone
on main, but a 200 with a JSON error payload from a compatible gateway
still reaches `set`, as the proof above shows.
- The check parses the body once per store. Stores only happen on
non-streaming 200 replies, so the cost is one `json.loads` per such
reply.
- Headroom Desktop has shipped an equivalent downstream patch since
2026-08-13.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3930"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-10-04 13:56:40 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 4ba231ab3d
commit 1132a54b21
2 changed files with 151 additions and 0 deletions
+29
View File
@@ -8,6 +8,7 @@ Extracted from server.py for maintainability.
from __future__ import annotations
import asyncio
import json
import sys
from collections import OrderedDict
from datetime import datetime
@@ -22,6 +23,31 @@ from headroom.proxy.semantic_cache_key_policy import compute_semantic_cache_key,
_strip_cache_control = strip_cache_control
def _is_cacheable_reply(response_body: bytes) -> bool:
"""Whether a 200 body is a reply the cache may replay.
The status code alone does not say so: an Anthropic- or OpenAI-compatible
gateway can answer 200 with an error object, or with nothing. Stored, that
body is replayed to every matching request for the full TTL and the
upstream is never asked again. Refuse an empty body, and JSON that is not a
non-empty object or that carries an error (Anthropic ``"type": "error"``,
OpenAI ``"error": {...}``). Bytes that are not JSON pass: the cache is
format-agnostic, and both handlers only store a reply they have decoded.
"""
if not response_body:
return False
try:
reply = json.loads(response_body)
except ValueError:
return True
return (
isinstance(reply, dict)
and bool(reply)
and reply.get("type") != "error"
and reply.get("error") is None
)
class SemanticCache:
"""Simple semantic cache based on message content hash.
@@ -84,6 +110,9 @@ class SemanticCache:
"""Cache a response."""
key = self._compute_key(messages, model, **key_fields)
if not _is_cacheable_reply(response_body):
return
async with self._lock:
# If key already exists, remove it first to update position
if key in self._cache:
@@ -0,0 +1,122 @@
"""The response cache must not store an error delivered as HTTP 200.
Both handlers gate ``SemanticCache.set`` on ``status_code == 200``. Anthropic- and
OpenAI-compatible gateways can answer 200 with an error object (or an empty
object), and once stored that body is replayed to every matching non-streaming
request for the full TTL without the upstream ever being asked again.
"""
from __future__ import annotations
import json
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from headroom.proxy.semantic_cache import _is_cacheable_reply
from headroom.proxy.server import ProxyConfig, create_app
MESSAGES = [{"role": "user", "content": "Say hi."}]
ANTHROPIC_ERROR = {"type": "error", "error": {"type": "api_error", "message": "boom"}}
OPENAI_ERROR = {"error": {"message": "boom", "type": "server_error", "code": None}}
ANTHROPIC_OK = {
"id": "msg_1",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "Hello"}],
"usage": {"input_tokens": 10, "output_tokens": 3},
}
OPENAI_OK = {
"id": "chatcmpl_1",
"object": "chat.completion",
"choices": [
{"index": 0, "message": {"role": "assistant", "content": "Hello"}, "finish_reason": "stop"}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 3, "total_tokens": 13},
}
@pytest.mark.parametrize(
"body,cacheable",
[
(json.dumps(ANTHROPIC_OK).encode(), True),
(json.dumps(OPENAI_OK).encode(), True),
(json.dumps({**OPENAI_OK, "error": None}).encode(), True),
(json.dumps(ANTHROPIC_ERROR).encode(), False),
(json.dumps(OPENAI_ERROR).encode(), False),
(b"", False),
(b"opaque-bytes", True),
(b"{}", False),
(b"null", False),
(b"[]", False),
],
)
def test_is_cacheable_reply(body: bytes, cacheable: bool) -> None:
assert _is_cacheable_reply(body) is cacheable
def _client() -> TestClient:
config = ProxyConfig(
optimize=False,
cache_enabled=True,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
)
return TestClient(create_app(config))
@pytest.mark.parametrize(
"path,headers,model,error_body,ok_body",
[
(
"/v1/messages",
{"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
"claude-haiku-4-5",
ANTHROPIC_ERROR,
ANTHROPIC_OK,
),
(
"/v1/chat/completions",
{"authorization": "Bearer test-key"},
"gpt-4o-mini",
OPENAI_ERROR,
OPENAI_OK,
),
],
)
def test_error_200_is_not_replayed_from_cache(path, headers, model, error_body, ok_body) -> None:
calls = {"n": 0}
with _client() as client:
async def _fake_retry(method, url, req_headers, body, stream=False, **kwargs): # noqa: ANN001
calls["n"] += 1
return httpx.Response(200, json=error_body if calls["n"] == 1 else ok_body)
client.app.state.proxy._retry_request = _fake_retry
request = {"model": model, "max_tokens": 64, "messages": MESSAGES, "stream": False}
first = client.post(path, headers=headers, json=request)
assert first.json() == error_body
# Before the fix the error body was cached: this came back from the
# cache with the upstream never called a second time.
second = client.post(path, headers=headers, json=request)
assert calls["n"] == 2
assert second.json() == ok_body
# A real reply is still cached.
third = client.post(path, headers=headers, json=request)
assert calls["n"] == 2
assert third.json() == ok_body