Commit Graph
1 Commits
Author SHA1 Message Date
1132a54b21 fix(proxy): do not cache error replies delivered as http 200 (#3930)
## Description

The response cache stores any non-streaming reply whose upstream status
is 200. An Anthropic- or OpenAI-compatible gateway can answer 200 with
an error object (`{"type": "error", ...}` / `{"error": {...}}`) or an
empty object, and that body was then replayed from the cache to every
matching request for the full TTL (1h by default) without the upstream
being asked again. Only a proxy restart or `POST /cache/clear`
recovered.

`SemanticCache.set` now refuses an empty body and any JSON body that is
not a non-empty object or that carries an error. Every store goes
through `set` (the Anthropic and OpenAI chat handlers are its only
callers), so one check covers both instead of a guard at each call site.

Closes #3929

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/semantic_cache.py`: new
`_is_cacheable_reply(response_body)`; `SemanticCache.set` returns
without storing when it is false. Refused: empty bytes, JSON that is not
a dict, `{}`, a dict with `"type": "error"`, a dict with a non-null
`"error"`. A reply with `"error": null` is still cached. Bytes that are
not JSON are stored as before: both handlers only store a reply they
already decoded, and existing cache tests store opaque bytes.
- `tests/test_proxy_response_cache_store_guard.py`: parametrized unit
test of `_is_cacheable_reply`, plus an end-to-end test per handler
(`/v1/messages`, `/v1/chat/completions`) with the cache on and an
upstream that answers 200 + error body once, then a good body. It
asserts the second request reaches the upstream and the good reply is
then served from cache.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Lint, format and type checks ran on the changed files only. Pytest ran
the new file plus the existing response-cache test files, not the whole
suite.

### Test Output

Before the fix (`semantic_cache.py` at upstream/main d5318ac2, plus a
pass-through `_is_cacheable_reply` stub so the new test module imports):

```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py -q -p no:cacheprovider -k replayed
E   assert 1 == 2
E   assert 1 == 2
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/messages-headers0-claude-haiku-4-5-error_body0-ok_body0]
FAILED tests/test_proxy_response_cache_store_guard.py::test_error_200_is_not_replayed_from_cache[/v1/chat/completions-headers1-gpt-4o-mini-error_body1-ok_body1]
================= 2 failed, 10 deselected, 1 warning in 9.28s ==================
```

After the fix:

```text
$ python -m pytest tests/test_proxy_response_cache_store_guard.py tests/test_proxy_semantic_cache_key.py \
    tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py \
    tests/test_openai_response_cache_key.py tests/test_proxy_semantic_cache_key_policy.py \
    tests/test_semantic_cache_key_policy.py -q -x -p no:cacheprovider
======================== 61 passed, 1 warning in 16.55s ========================

$ python -m pytest tests/test_proxy_response_cache_replay.py tests/test_ccr_buffered_stream_signed_thinking.py \
    tests/test_handler_outcome_tag_invariant.py tests/test_read_maturation_handler_nobust.py \
    tests/test_observability_metrics.py -q -p no:cacheprovider
=================== 45 passed, 1 warning in 69.84s (0:01:09) ===================

$ ruff check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
All checks passed!
$ ruff format --check headroom/proxy/semantic_cache.py tests/test_proxy_response_cache_store_guard.py
2 files already formatted
$ mypy headroom/proxy/semantic_cache.py
Success: no issues found in 1 source file
```

The warning is the existing `StarletteDeprecationWarning` from
`fastapi.testclient`.

## Real Behavior Proof

- Environment: macOS, Python 3.11.13. The real `headroom proxy` CLI,
booted from the source tree with the default `ProxyConfig` (cache on,
TTL 3600s, `Mode: cache`), pointed at a local stub upstream on 127.0.0.1
via `--anthropic-api-url` / `--openai-api-url`. The stub answers the
first POST per path with HTTP 200 + an error body (`{"type": "error",
"error": {"type": "overloaded_error", ...}}` for `/v1/messages`,
`{"error": {"message": "Upstream overloaded", "type": "server_error",
"code": null}}` for `/v1/chat/completions`) and every later POST with a
normal reply. The scratch venv lacks the `mcp`, `magika` and
`onnxruntime` extras, so the CLI was started through a three-line
launcher that skips `ensure_proxy_dependencies()` and then calls the
stock `headroom.cli.main.main(["proxy", ...])`. None of those modules is
on the response-cache path.
- Exact command / steps: `python launch.py --port <p>
--anthropic-api-url http://127.0.0.1:<s> --openai-api-url
http://127.0.0.1:<s>/v1`, then three identical non-streaming requests
per endpoint, then `GET /stats`. Run once with upstream/main d5318ac2
and once with this branch, same venv and stub.
- Observed result: on main the stub is hit once per path and requests 2
and 3 replay the stored error from cache; on this branch request 2
reaches the stub and gets the good reply, and request 3 is served that
reply from cache (output below).

upstream/main (d5318ac2): the stub is hit once per path; requests 2 and
3 get the stored error from cache.

  ```text
  stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 668ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 2 -> 200 in 26ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
client: /v1/messages request 3 -> 200 in 22ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
  stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 61ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 2 -> 200 in 12ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
client: /v1/chat/completions request 3 -> 200 in 24ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 4,
"ttl_seconds": 3600}
  ```

This branch: request 2 reaches the stub and gets the good reply; request
3 is served that good reply from cache.

  ```text
  stub: /v1/messages hit #1 -> 200 error body
client: /v1/messages request 1 -> 200 in 1010ms: {"type": "error",
"error": {"type": "overloaded_error", "message": "Overloaded"}}
  stub: /v1/messages hit #2 -> 200 good body
client: /v1/messages request 2 -> 200 in 304ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
client: /v1/messages request 3 -> 200 in 115ms: {"id": "msg_1", "type":
"message", "role": "assistant", "model": "claude-haiku-4-5", "content":
[...]
  stub: /v1/chat/completions hit #1 -> 200 error body
client: /v1/chat/completions request 1 -> 200 in 51ms: {"error":
{"message": "Upstream overloaded", "type": "server_error", "code":
null}}
  stub: /v1/chat/completions hit #2 -> 200 good body
client: /v1/chat/completions request 2 -> 200 in 21ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
client: /v1/chat/completions request 3 -> 200 in 20ms: {"id":
"chatcmpl_1", "object": "chat.completion", "model": "gpt-4o-mini",
"choices": [...]
proxy /stats cache: {"entries": 2, "max_entries": 1000, "total_hits": 2,
"ttl_seconds": 3600}
  ```

Proxy `PERF` lines match: on main requests 2-3 log `tok_before=0` (cache
hit) on both models; on this branch request 2 logs `tok_before=10`
(forwarded) and only request 3 logs `tok_before=0`.
- Not tested: real Anthropic/OpenAI or a real third-party gateway (stub
only, no credentials used); streaming requests (they never read or write
this cache); the full test suite (targeted files only); the empty /
unparseable 200 seen in production, because on main neither handler can
pass such a body to `set` any more (Anthropic requires `resp_json is not
None`, OpenAI raises to its 502 path), so that case is covered by the
unit test only.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a correctness guard in
`SemanticCache.set`, not a rollout feature.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes, narrowly. With the cache on (the
default), an HTTP 200 reply that is empty, an empty JSON object, a
non-object JSON value, or a JSON object carrying `"type": "error"` or a
non-null `"error"` is no longer stored, so the next identical request
goes upstream instead of replaying it. The reply itself is still
returned to the client unchanged. Successful replies are cached exactly
as before.
- Kill switch / disable path: none added; there is no case where
replaying a stored error is wanted. `--no-cache` still disables the
cache as a whole.
- Unsafe override required: no.
- Qualification impact: none expected. Compression, prompt-cache prefix
handling and token accounting are untouched; only which bodies the
response cache keeps changes. Cache-hit counts drop only where the hits
were replayed errors.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- Documentation: N/A. No user-facing option or documented behavior
changes.
- Overlap with #3862 (partition the response cache by credential and
principal): both touch `SemanticCache.set`, but this change only adds
lines after the key computation and does not touch the signature,
`_compute_key` or `get`. A three-way merge of `semantic_cache.py`
against #3862's head is clean. The new tests call `_is_cacheable_reply`
directly and go through the handlers for the end-to-end case, so they do
not depend on whether `set`/`get` take `partition=`.
- History: the production case that surfaced this was the malformed
buffered-CCR 200 that #2997 turned into a real 502. That source is gone
on main, but a 200 with a JSON error payload from a compatible gateway
still reaches `set`, as the proof above shows.
- The check parses the body once per store. Stores only happen on
non-streaming 200 replies, so the cost is one `json.loads` per such
reply.
- Headroom Desktop has shipped an equivalent downstream patch since
2026-08-13.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3930"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:56:40 -07:00