mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3871b3664a |
fix(proxy): drop headers named by Connection when forwarding responses (#4674)
## Description `sanitize_forwarded_response_headers` strips `Connection` but not the connection-local headers its value names, and `TE`, `Trailer` and `Upgrade` also survived. Per RFC 9110 section 7.6.1 these must not be forwarded by an intermediary. The helper now drops every token listed in any `Connection` value (case-insensitive) and adds `te`, `trailer`, `upgrade` to `FRAMING_RESPONSE_HEADERS`. Closes #4265 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/helpers.py`: collect names from `Connection`, extend the framing header list. - `tests/test_proxy_response_cache_replay.py`: regression test with real `httpx.Headers` (duplicate `Connection` fields), also asserting the input is not mutated. Fails on `main`, passes now. ## Testing - [x] Unit tests pass: `TestSanitizeForwardedResponseHeaders` 5 passed (new one failed before the fix) - [x] Linting passes (`ruff check` / `ruff format --check` on touched files) - [ ] Type checking (`mypy headroom`): not run - [x] New tests added Note: in my environment 2 other tests in `tests/test_proxy_response_cache_replay.py` (`test_cache_hit_replays_a_body_the_client_can_actually_read`, `test_buffered_ccr_turn_does_not_write_the_response_cache`) fail with `CancelledError` identically on unmodified `main`; my setup is a partial install (no Rust extension build), so I did not investigate. ## Real Behavior Proof - Environment: macOS, Python 3.12, real `httpx.Headers`. - Steps: called the helper with `Connection: X-Upstream-Hop, Upgrade`, `X-Upstream-Hop`, `Upgrade: h2c`, `TE: trailers`, `Trailer: Digest`, `X-Request-ID`. - Result: before, `x-upstream-hop`, `upgrade`, `te` and `trailer` were all kept; after, only `x-request-id` remains. - Not tested: end-to-end through a running proxy and upstream, streaming paths, full suite, mypy, pre-push hooks. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I did **not** edit `CHANGELOG.md` This change was prepared with AI assistance (Claude Code). <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4674" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
ca6ba3486c |
fix(ccr): stream the buffered CCR upstream request so idle-killing networks cannot cut it (#4206)
## Description The buffered CCR path flips a streaming Claude Code turn to `stream: false` upstream so the proxy holds the whole reply before serving `headroom_retrieve`. With `stream: false`, Anthropic sends no byte until the generation is complete. A middlebox that drops HTTP connections idle for ~60s (TLS-inspecting antivirus, VPN, corporate proxy; TCP keepalive does not reset an HTTP-layer idle timer) closes the connection first: each attempt fails at ~60s with `Server disconnected without sending a response`, the client gets a 502 after ~186s, and Claude Code resends the same body forever. The CCR continuation that serves the retrieval (`anthropic_ccr_continuation`, its own `http_client.post`) is a second `stream: false` call and is cut the same way. Without Headroom the client streams and the same turns complete. This PR keeps the buffered design and makes only its upstream leg stream. The buffered request and every continuation built from it (CCR, memory, turn hook) go to first-party Anthropic as `stream: true`, and the event stream is folded back into the Message JSON the buffered path already reads, so retrieval, memory, turn hooks, usage accounting and SSE resynthesis see exactly what they saw before. Bytes (deltas and pings) now flow for the whole generation, so an idle timer never fires. The fold reuses `AnthropicSSEEnvelope`, the reconstruction #3130 and #3266 already use on this path. Closes #4205 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/anthropic.py`: new `_send_buffered_as_stream(send, headers, body, *, url, request_id)`. It sends `{**body, "stream": true}` with `Accept: text/event-stream` (one Accept value, any casing replaced) and folds the reply: - not a 200 event stream (an HTTP error, or an upstream that answered JSON anyway): returned unchanged; - a whole stream: a 200 JSON response holding the folded Message; - an in-band `error` event: the event as the body, with its own HTTP status (529 `overloaded_error`, 429 `rate_limit_error`, other documented types to their status, else 500); - a stream that ended before `message_stop` or with a block open: the existing strict 502 `upstream_protocol_error` of `_adapt_event_stream_to_json` (#3266); - a complete stream holding a known event the fold cannot read (an unknown delta type): logged, and the original `stream: false` body is sent instead. One more generation, but never a 502 the client would resend forever. A connection dropped mid-stream raises inside `send`, exactly as for a `stream: false` reply. An in-band error arrives after upstream already answered 200, so `_retry_request` does not retry it; the client gets the status and retries as it would on a direct stream. - It is used at the four places Headroom sends a body it made non-streaming: the buffered CCR request (only when `buffered_stream_ccr`; a client's own `stream: false` request still goes through `_retry_request` with its original bytes), the CCR continuation (the `prepare_outbound_body_bytes` + `log_outbound_request` + `http_client.post` sequence moved into a local `_post_continuation`, unchanged otherwise, so a fallback attempt logs its own outbound line), the memory continuation and the turn-hook continuation. - Gates: the body is not already streaming, the upstream is first-party Anthropic (`is_custom_anthropic_base_url(url)` is false; the host-equality check tool-search gating already uses), and the kill switch `HEADROOM_BUFFERED_UPSTREAM_STREAM` is not `0`/`false`/`no`/`off`. Copilot and other gateways keep the `stream: false` request: the fold is checked against Anthropic's own stream, and a gateway dialect it cannot read would cost every turn a second generation. - `headroom/proxy/runtime_env.py`: `HEADROOM_BUFFERED_UPSTREAM_STREAM` is a registered knob in `RUNTIME_ENV_KNOBS`, read per request through `runtime_env.getenv`, so a reusing `headroom wrap` pushes it to the running proxy and `POST /admin/runtime-env` sets it with no restart. Listed in the `security-model.mdx` allowlist. - `headroom/proxy/anthropic_wire.py`: `fold_anthropic_stream(raw)` parses with `AnthropicSSEEnvelope.parse` and trims each block to the non-streaming Message shape: a null member leaves every block, `thinking` keeps `type`/`thinking`/`signature`, `redacted_thinking` keeps `type`/`data`. Claude Opus 5.5 streams a thinking `content_block_start` with `estimated_tokens: null`; the CCR continuation resends the folded turn as history, and Anthropic answers any block carrying that member with 400 `Extra inputs are not permitted`. - `headroom/proxy/anthropic_wire.py` `_response_from_events`: `message_delta.usage` is merged only where non-null, as the Anthropic SDK's `MessageStream` does. Before, a null count there overwrote the real `message_start` count (input and cache tokens) in the reconstructed message. - `headroom/proxy/buffered_ccr_response.py`: the status-to-error-type table moves out of `_anthropic_error_type` into `ANTHROPIC_ERROR_TYPES` so the handler can invert it; `_anthropic_error_type` returns the same values. - Unchanged on purpose: `body["stream"] = False` and its mutation reason `ccr_streaming_retrieve_buffered_non_stream` stay the buffered path's internal representation (every later step, the response-cache guard and the #3587 log counts key on them), and the `using buffered stream:false upstream request` log line keeps its text. - Tests: new `tests/test_buffered_ccr_upstream_stream.py` (16 tests): the fold equals the Anthropic Python SDK's `accumulate_event` result on a stream with thinking, signature, redacted thinking, text, citations, split `input_json_delta`, pings and a null usage count; stream-only members are dropped; truncated, in-band error and unknown-delta outcomes; the send helper's gates (kill switch, gateway URL, already streaming), passthrough of non-stream replies, 529 for an in-band overload, 502 for a truncated stream, the `stream: false` fallback; and through the handler, the buffered turn and its CCR continuation both go upstream `stream: true` with `Accept: text/event-stream` and the continuation's resent thinking block carries only its schema, while a client's own `stream: false` request goes out `stream: false` with `Accept: application/json` and its original bytes. - Existing tests that pinned `stream: false` on the wire for a buffered turn now pin `stream: true` (`test_anthropic_streaming_ccr_retrieve.py`, `test_ccr_buffered_stream_signed_thinking.py`, `test_proxy_response_cache_replay.py`, `test_no_ccr_disables_response_handling.py`). `test_anthropic_buffered_ccr_usage.py` now answers the buffered turn with an event stream, so usage booking is checked through the fold. `test_buffered_ccr_accept_header.py` (#3078) pins the `stream: false` leg it is about with the kill switch; custom upstreams such as Copilot always take that leg. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - ran on the 3 changed source files only, see below - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_buffered_ccr_upstream_stream.py -q 16 passed, 1 warning in 2.89s # every test file that touches the buffered CCR path, the CCR continuation or SSE reconstruction $ pytest tests/test_buffered_ccr_upstream_stream.py tests/test_sse_safeguards_passthrough.py \ tests/test_buffered_ccr_salvage.py tests/test_anthropic_buffered_ccr_usage.py \ tests/test_buffered_ccr_grace_window.py tests/test_anthropic_buffered_sse.py \ tests/test_ccr_buffered_stream_signed_thinking.py tests/test_toin_feedback.py \ tests/test_proxy_response_cache_replay.py tests/test_proxy/test_anthropic_buffered_timeout.py \ tests/test_anthropic_auto_mode_passthrough.py tests/test_proxy/test_openai_responses_ccr.py \ tests/test_sse_thinking_blocks.py tests/test_streaming_sse_malformed_events.py \ tests/test_route_advice.py tests/test_memory_tool_stream.py \ tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy_byte_faithful_forwarding.py \ tests/test_buffered_ccr_accept_header.py tests/test_anthropic_buffered_memory_continuation.py \ tests/test_anthropic_ccr_workspace_unbound.py tests/test_no_ccr_disables_response_handling.py \ tests/test_proxy_ccr.py tests/test_ccr_inline_resolve_handlers.py \ tests/test_proxy/test_anthropic_ccr_raise.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py \ tests/test_nonstream_sse_policy.py tests/test_ccr_response_handler_extra.py \ tests/test_compress_route_tokenizer_by_model.py -q 855 passed, 6 warnings in 57.94s # the updated assertions fail on main's source (they pin stream:true on the wire) $ git checkout upstream/main -- headroom/ && pytest tests/test_anthropic_buffered_ccr_usage.py \ tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_ccr_buffered_stream_signed_thinking.py \ tests/test_no_ccr_disables_response_handling.py tests/test_proxy_response_cache_replay.py -q --tb=no 12 failed, 33 passed, 1 warning in 13.64s # whole suite $ pytest tests -q 18 failed, 16435 passed, 583 skipped, 119 warnings in 1226.31s (0:20:26) # 16 of the 18 fail the same way on unmodified main in the same environment # (tests/test_proxy_uds.py x11, tests/test_cli/test_wrap_persistent.py x4, # tests/test_pricing_from_litellm.py groq/llama-guard-3-8b x1); the other 2 # (test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request, # test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children) pass when run alone on this branch. $ ruff check . All checks passed! $ ruff format --check . 1790 files already formatted $ mypy headroom/proxy/anthropic_wire.py headroom/proxy/buffered_ccr_response.py headroom/proxy/handlers/anthropic.py Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS 27.0.1 arm64, Python 3.11.13. Upstream `main` |
||
|
|
f78e66f7cc |
fix(proxy): stop headroom logger from suppressing propagation to stdout/stderr (#3096)
## Description `_setup_file_logging()` in `headroom/proxy/helpers.py` unconditionally set `headroom_logger.propagate = False`, on the theory that `wrap.py` redirects stderr into the same `proxy.log` file the rotating handler writes to, so propagation would double-write records. That premise is wrong: `wrap.py` redirects the proxy subprocess's stdout/stderr to a separate sibling file, `proxy-stdio.log` (see `_get_proxy_stdio_log_path()`), never `proxy.log`. Severing propagation bought no deduplication — it only stopped every `headroom.*` log record from reaching root's `logging.basicConfig()` handler (`proxy/server.py`), which is what writes to stderr/stdout. In containerized deployments (Docker/Kubernetes), stdout/stderr is the only channel the platform collects, so this silently dropped all application logs. Closes #3087 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed `headroom_logger.propagate = False` from `_setup_file_logging()` in `headroom/proxy/helpers.py`. - Updated the surrounding comment to explain why propagation is safe to leave on: `proxy.log` (file handler) and stderr/stdout (root's `basicConfig` handler) are two distinct destinations, so nothing is duplicated in either file. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check . All checks passed! $ pytest tests/test_corrupt_golden_bytes_recovery.py -q collected 9 items tests/test_corrupt_golden_bytes_recovery.py ......... [100%] 9 passed in 0.47s ``` `mypy headroom` was not run to a clean result: it fails before reaching this file with a pre-existing, unrelated error in the installed `numpy` stub (`Type statement is only supported in Python 3.12 and greater`) in this sandbox's dependency set. Not caused by this change (a 4-line pure-Python logging tweak). The broader `pytest tests/` collection also hits ~170 pre-existing, unrelated errors in this sandbox: `headroom._core`, the compiled Rust extension, isn't built here (no `cargo`/`maturin` available), so every module importing it fails at collection. This change doesn't touch Rust code or `_core` and isn't implicated. No new tests added: this restores an existing, already-tested code path (see `tests/conftest.py`'s autouse fixture and `tests/test_corrupt_golden_bytes_recovery.py`, both of which already assert/require `propagate=True` for `caplog` to work) to its intended default rather than adding new behavior. ## Real Behavior Proof - Environment: local sandbox, Python 3.12.3, unable to build/import `headroom.proxy.server` directly (missing compiled `headroom._core`). Verified with an isolated script that copies `_setup_file_logging()`'s exact logic byte-for-byte plus the same `logging.basicConfig()` call `proxy/server.py` makes at import time. - Exact command / steps: ran the fixed logic, then emitted `logging.getLogger("headroom.proxy").info("compression timeout: request abc123 exceeded 30s budget")`. - Observed result: the `headroom.proxy` record was written to stderr AND to `proxy.log` simultaneously (full output below); re-running the same script with the old `headroom_logger.propagate = False` line restored reproduces the bug — the record disappears from stderr entirely and only lands in `proxy.log`. ```text propagate flag on 'headroom' logger: True --- emitting a headroom.proxy log record, watch for it below on stderr --- 2026-08-17 21:17:39,545 - headroom.proxy - INFO - compression timeout: request abc123 exceeded 30s budget --- also present in proxy.log (file handler): OK --- ``` - Not tested: end-to-end through the actual compiled proxy binary/server process (blocked by missing `headroom._core` in this sandbox) and real container log collection (Docker/Kubernetes) — the isolated repro exercises the exact same logging configuration code the real process runs, so this is considered sufficient proxy proof for a pure logging-config change. ## Runtime Rollout Safety - Rollout-managed feature(s): None — this is unconditional startup logging configuration, not behind a rollout flag. - Minimum rollout channel: N/A - Stable/default behavior changed: Yes — `headroom.*` log records now also reach stderr/stdout (previously file-only). No change to request/response handling, compression, or any user-facing proxy behavior. - Kill switch / disable path: None needed; this restores default Python logging propagation. If a deployment truly wants file-only logging, `HEADROOM_LOG_TO_STDOUT`-style suppression could be added later, but no such request exists today. - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this commit to restore `propagate = False`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — logging configuration change, no UI impact. ## Additional Notes - "New and existing unit tests pass locally" is left unchecked because the full `pytest tests/` collection can't complete in this sandbox (missing compiled `headroom._core` Rust extension, unrelated to this change) — see Test Output above for what was actually run and passed. - "I have added tests" is left unchecked: existing tests (`tests/conftest.py` autouse fixture, `tests/test_corrupt_golden_bytes_recovery.py`) already assert the code path this restores; happy to add a focused regression test asserting `logging.getLogger("headroom").propagate is True` after `_setup_file_logging()` if maintainers want one. --------- Co-authored-by: = <=> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
c502087db7 |
fix(ccr): only buffer a stream when a marker is actually redeemable (#3092)
## Description Closes #3071 `headroom_retrieve` is injected once and kept resident for the session so the tools array stays byte-stable and the prompt cache survives. The buffered-CCR path keyed on that tool merely being **present**, so once a session went sticky, *every* later streaming turn was silently converted to `stream: false`, buffered whole, and resynthesized as SSE: ``` CCR: stream:true request has headroom_retrieve available; using buffered stream:false upstream request ``` Buffering leaves time-to-last-byte roughly unchanged but makes **time-to-first-byte the entire generation**. The reporter measured 8s average and up to 100s across 234 requests in one day — turns that would have streamed a first token in ~1s instead delivered nothing until done. Retrieval can only expand a `<<ccr:...>>` marker present in the outgoing body, so a turn carrying none cannot benefit from the buffered path at all. Gate on that instead of on the tool. This is also the root cause #3082 traced independently from the OpenCode side — its plugin registers `headroom_retrieve` unconditionally, so *every* turn buffered and neither `--no-ccr` nor `HEADROOM_NO_CCR` stopped it. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - New `_outgoing_body_has_redeemable_marker()` scans the body **about to go on the wire** and verifies ownership against the compression store — the same `exists()` check the retrieve endpoint performs, so a same-shaped marker from another context tool is not adopted (#2836). Unexpected shapes answer `True`, keeping the long-standing behavior. - The buffered-stream decision site gates on it, and logs at INFO when it skips buffering. - The correctness detail worth reviewing: the check reads `body`, **not** the earlier `scan_for_markers(optimized_messages)` result. `optimized_messages` is reassigned five times after that scan (memory hooks, pre-send extensions, tool-search repair, CCR repair), so reusing it would have been stale. - Two existing test files encoded the very coupling this removes and had to be repaired — see Testing. Scope: this narrows *when* buffering happens; it does not make buffered turns stream. A turn that genuinely carries a marker still loses incremental delivery — restoring streaming there means wiring `StreamingCCRHandler`, which is #3069's scope. It does not fix #3088 either, whose requests do carry markers. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] Manual testing performed Two existing files built a request with `headroom_retrieve` and **no** marker, relying on the tool alone to trigger buffering: - `tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py` (11 tests) fell through to the live streaming path, where only `_retry_request` is stubbed — so the requests reached the network and the file **hung indefinitely** rather than failing. Seeded real markers; now passes in ~5s. - `tests/test_proxy_response_cache_replay.py::test_buffered_ccr_turn_does_not_write_the_response_cache` asserts its own premise (*"the conversion really happened — otherwise this test proves nothing"*), so it failed loudly instead of passing vacuously. Seeded a marker. New `test_buffering_is_gated_on_a_redeemable_marker` pins all three directions: owned marker → buffered, no marker → streaming, foreign marker → streaming. ### Test Output ```text $ pytest tests/test_ccr_buffered_stream_signed_thinking.py -q 8 passed, 1 warning in 4.45s $ pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py -q 11 passed, 1 warning in 4.94s # was: hung indefinitely $ pytest tests/test_proxy_response_cache_replay.py -q 9 passed, 1 warning in 1.69s $ pytest tests/ -q 3 failed, 11151 passed, 581 skipped in 398.28s (0:06:38) Same 3 failures as a clean-main baseline run on this machine: tests/test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter tests/test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline tests/test_release_workflows.py::test_no_native_tls_in_wheel_build_tree $ ruff check . && ruff format --check . All checks passed! ``` ## Real Behavior Proof - Environment: this branch driven through the real FastAPI app with the outbound HTTP client captured; macOS arm64, Python 3.12. - Exact command / steps: posted a `stream: true` `/v1/messages` request carrying a resident `headroom_retrieve` tool in three variants — no marker, a marker seeded into the compression store, and a correctly-shaped marker the store does not own — recording whether `_retry_request` saw a `stream: false` body. - Observed result: no marker → **streams**, `_retry_request` never sees a flipped body; owned marker → **buffers**, exactly as before; foreign marker → streams, honoring #2836 rather than adopting another tool's hash. - Not tested: the latency improvement against live client traffic. The mechanism is verified (the buffered conversion no longer occurs), but the reported 8s → ~1s TTFB needs the reporter's traffic to confirm. ## Runtime Rollout Safety - Rollout-managed feature(s): none — this narrows an existing code path and is not behind a rollout channel. - Minimum rollout channel: n/a (ships to stable with the fix). - Stable/default behavior changed: yes. A streaming turn whose body carries no redeemable marker now stays streaming instead of being buffered. Turns carrying a marker are unchanged. - Kill switch / disable path: no new switch. Existing CCR controls still apply — disabling the CCR response handler bypasses this decision site entirely, and the helper fails open (returns `True`, i.e. the old behavior) on any unexpected message shape. - Unsafe override required: no. - Qualification impact: none — no qualification-gated surface is touched. - Rollback path: revert this commit. Note it also carries two test repairs; reverting the production change alone would leave those tests passing but vacuous. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Documentation update is marked N/A: no user-facing flag or endpoint changes. Type checking (`mypy headroom`) was not run separately; `ruff` is the gate this repo's CI enforces. Related: #2836 (marker ownership), #3069 (streaming CCR handler), #3082, #3088. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
9d370592b0 |
fix(proxy): stop cached responses replaying the producing turn's wire framing (#3024)
## Description Closes #3019 A response-cache hit could hand the client an HTTP 200 that the client could not read, and nothing in the logs marked the turn as anything other than normal. Two separate problems combine to produce the reported failure. **The unreadable 200.** A cache entry stores the producing upstream's response headers verbatim. When the entry is replayed, the Anthropic handler removed only `content-encoding`, `content-length` and `content-type` before handing those headers to a brand-new `Response`. Anything else describing how that *other* connection framed its body rode along — most damagingly `transfer-encoding: chunked`. RFC 9112 §6.1 makes `Transfer-Encoding` override `Content-Length`, so the client is told to parse a plain JSON body as chunked frames, finds no valid chunk-size line, and reads an empty body out of a 200. Every other response-forwarding site in the Python proxy already strips that header; the two cache-hit sites were the only ones that did not. **How a CCR turn could put a foreign response in the cache.** On the Anthropic path, `cache.get` is gated on `not stream` but `cache.set` was not, and the cache key has no `stream` component. A CCR buffered-stream conversion takes a request the client sent with `stream: true`, forces `stream: false` upstream, and — unlike every other streaming turn, which returns via `_stream_response` and never touches the cache — falls through to the store site. The stored reply was shaped by that forced flip plus CCR tool injection, and the key cannot distinguish it from an ordinary non-streaming reply, so a later non-streaming caller could be served a response built for a request it never made. This is why the reporters saw the failures pair with CCR activity and stop under `--lossless` / `--no-ccr`. **Why it was invisible.** The cache-hit block emitted no log line at all, and the `PERF` line rendered no field for `RequestOutcome.from_response_cache`. A cache-served turn contacts no upstream, so it has no `outbound_request` line, no upstream stage timings, and all-zero token counters — byte-for-byte what a turn that died would look like. That is why `headroom doctor` reported zero failures while turns were dying. ### Scope note The header fix also lands on the OpenAI cache-hit site, which additionally never received the `content-type` fix from #2952. The `not stream` gate is added to the OpenAI store site too, where it is currently redundant — a streaming chat request returns via `_stream_response` long before that point — purely to state the invariant, since the Anthropic handler had exactly that shape until a buffered-CCR branch began falling through to it. Because the strip list now lives in one shared helper, the OpenAI handler's other five forwarding sites strip the three added headers as well. That is a widening, so it is worth being explicit about: each of those sites builds a fresh fixed-length `Response` (or, at `openai.py:6122`, synthesises SSE) from `response.content`, so replaying the upstream's framing there was the same latent bug, just without a cache to make it outlive the request that produced it. The precedent is already in the file — `openai.py:9865` passes `"transfer-encoding", "connection"` as extra names by hand, which is exactly the gap this PR closes centrally. That call site keeps its now-redundant arguments; removing them is a cleanup for another PR. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `sanitize_forwarded_response_headers` to `headroom/proxy/helpers.py`, promoting the private helper that already lived in `headroom/proxy/handlers/openai.py` and extending it with the remaining wire-framing headers (`transfer-encoding`, `connection`, `keep-alive`). Matching is now case-insensitive; surviving headers keep their original casing. `openai.py`'s `_sanitize_forwarded_response_headers` is now a thin alias so its six call sites and the Anthropic handler strip an identical set. - `headroom/proxy/handlers/anthropic.py`: the response-cache hit now sanitises through that helper (passing `content-type` as an extra name, preserving #2952) instead of three hand-rolled `pop` calls. - `headroom/proxy/handlers/openai.py`: the response-cache hit sanitises the same way, gains the `content-type` handling it was missing, and sets `media_type="application/json"` explicitly. - `headroom/proxy/handlers/anthropic.py`: `cache.set` is now gated on `not stream`, mirroring the read gate. `stream` still holds the client's original flag at that point — the buffered-CCR conversion flips `body["stream"]`, never the local variable. - `headroom/proxy/handlers/openai.py`: the same `not stream` gate on its store site, as an invariant guard. - Both cache-hit sites now log `RESPONSE-CACHE-HIT: model=… bytes=… age_s=… hits=…`, following the existing `CACHE-MISS-ATTRIBUTION` line style. - `headroom/proxy/outcome.py`: the `PERF` line appends `cached=1` on a response-cache hit. It is appended only on a hit, so every other PERF line is byte-identical to before and existing parsers are unaffected. - `headroom/perf/analyzer.py`: `PerfRecord.from_response_cache` reads that field, so `headroom perf` can tell a cache-served turn from a dead one. It defaults to `False`, so older logs still parse. `PERF_RECORD_FIELDS` gains the name at the end of the list, which is what `headroom perf --format csv --raw` uses as its column set; appending keeps every existing column at its current position. `--format json --raw` gains the key too. - `tests/test_anthropic_pre_upstream_backpressure.py`: its cache-hit double was a partial hand-rolled stand-in for `CacheEntry` carrying only a body and headers, so it broke once the hit path started reading the entry's age and hit count. It now constructs a real `CacheEntry`, which is what the cache actually returns. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ python -m pytest tests/test_proxy_response_cache_replay.py -q tests\test_proxy_response_cache_replay.py ......... [100%] ============================== 9 passed in 4.22s ============================== # Everything that mentions PERF, the sanitiser, cache.set, PerfRecord or # response_headers, plus the whole proxy suite. $ python -m pytest tests/test_proxy/ tests/test_proxy_compression_headers.py \ tests/test_agent_savings.py tests/test_anthropic_pre_upstream_backpressure.py \ tests/test_backend_nonstreaming_cache_metrics.py tests/test_backend_streaming_cache_metrics.py \ tests/test_ccr_buffered_stream_signed_thinking.py tests/test_cli_perf_format.py \ tests/test_codex_ws_compression_scheduler.py tests/test_handler_outcome_tag_invariant.py \ tests/test_openai_codex_ws_lifecycle.py tests/test_provider_codex_images.py \ tests/test_proxy_handlers_batch.py tests/test_proxy_passthrough_transient_retry.py \ tests/test_proxy_response_cache_replay.py tests/test_proxy_semantic_cache_key.py \ tests/test_proxy_streaming_request_logger.py tests/test_request_outcome.py \ tests/test_savings_tool_search_aggregation.py -q ================== 555 passed, 1 skipped in 88.60s (0:01:28) ================== # Full suite, 16 workers. See "Real Behavior Proof" below for how every # failure here was traced to a pre-existing failure or a parallelism flake. $ python -m pytest tests scripts/tests -n 16 -q -p no:randomly --timeout=300 83 failed, 10493 passed, 657 skipped, 80 errors in 437.00s (0:07:17) $ ruff check . All checks passed! $ ruff format --check <the 7 changed files> 7 files already formatted $ python -m mypy headroom --ignore-missing-imports --python-version 3.13 Found 12 errors in 3 files (checked 520 source files) # All 12 are pre-existing MCP-SDK/tomllib drift in release_version.py, # ccr/mcp_server.py and memory/mcp_server.py; identical count before and # after this change, none in the files it touches. ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.11, pytest 9.1.1, ruff 0.16.2, branch based on `upstream/main` at `2d88e31a`. - Exact command / steps: Two experiments. (1) Revert-and-rerun: I reverted both fixes in place (dropped the three framing headers from `FRAMING_RESPONSE_HEADERS`, restored `cache.set` to `if self.cache and response.status_code == 200 and resp_json is not None:`), ran `python -m pytest tests/test_proxy_response_cache_replay.py -q`, then restored the fixes and re-ran. (2) Regression sweep: ran the full suite on this branch, then checked out `upstream/main` into a second worktree and re-ran, in that worktree, exactly the tests that failed here and not there. - Observed result: With the fixes reverted, 5 of 9 new tests fail and reproduce both halves of the bug. `test_buffered_ccr_turn_does_not_write_the_response_cache` fails with `AssertionError: Expected mock to not have been awaited. Awaited 1 times.` — a turn the client sent as `stream: true` really does reach `cache.set` through the buffered-CCR branch. `test_cache_hit_replays_a_body_the_client_can_actually_read` fails with `AssertionError: assert 'transfer-encoding' not in {'transfer-encoding': 'chunked', 'connection': 'keep-alive', 'request-id': ..., 'content-length': '228', ...}` — the replayed 200 carries the producing turn's chunked framing alongside a fresh `content-length`, which is the exact framing conflict a client cannot parse. With the fixes restored, all 9 pass, the replayed body arrives intact as `application/json`, and the run logs both `RESPONSE-CACHE-HIT` and a `PERF … cached=1` line. The full suite on this branch gives `83 failed, 10493 passed, 657 skipped, 80 errors`; 33 of those failures were not in my baseline list, so I ran those 33 in the `upstream/main` worktree and 20 failed there identically (Windows-specific: `sqlite:///C:\…` path handling, private-directory permissions, fsync, ONNX thread caps, serena config discovery). Re-running the remaining 13 serially on this branch gave `1 failed, 25 passed` — the other 12 were xdist parallelism flakes, including all four `tests/test_proxy/test_anthropic_ccr_deferred_injection.py` tests, which are the only ones in this change's blast radius and which pass serially. The one real serial failure, `tests/test_savings_ledger_offload.py::test_concurrent_requests_all_land_their_events` (`AssertionError: a concurrent append was lost / assert 23 == 24`), fails the same way on `upstream/main` run serially. The 80 errors are dashboard-template collection errors unrelated to the proxy. Net: no failure attributable to this change. - Not tested: I could not reproduce against live upstream traffic, so I have not confirmed which upstream in the reporters' setups emits `transfer-encoding: chunked`. Anthropic direct is HTTP/2, where the header is forbidden, but any HTTP/1.1 hop (corporate proxy, third-party gateway, local relay) reintroduces it. I have also not measured whether the `not stream` gate reduces the cache hit rate in practice; by construction it can only drop entries that were unsafe to serve. A reporter running unmodified 0.35.0 with `headroom proxy --no-cache` would confirm the cache path is the one involved, and that flag is a lighter workaround than `--lossless` or `--no-ccr` because it keeps CCR and compression enabled. ## Runtime Rollout Safety - Rollout-managed feature(s): none — this is a correctness fix on the always-on response-cache path (`cache_enabled` defaults to `True`). - Minimum rollout channel: stable. - Stable/default behavior changed: yes, in four ways. Replayed cached responses no longer carry the producing upstream's framing headers (or `server`, on the Anthropic side). Forwarded responses on the OpenAI handler's other five sanitiser call sites no longer carry `transfer-encoding`, `connection` or `keep-alive` either, since the strip list is now shared; all five build a fixed-length response from `response.content`, so none of them could legitimately replay that framing. A turn whose client asked for `stream: true` no longer writes the response cache on the Anthropic path. `PERF` lines gain a trailing `cached=1` on a response-cache hit only; all other PERF lines are unchanged. - Kill switch / disable path: `headroom proxy --no-cache` disables the response cache entirely and bypasses every path this PR touches. - Unsafe override required: no. - Qualification impact: low. No public API, config key, CLI flag or wire format changes. Two additive output changes: the `cached=1` PERF field, which `_parse_kv` already handles the same way it handles the existing trailing `client=` field, and a `from_response_cache` column appended to `headroom perf --format csv --raw` (plus the matching key in `--format json --raw`). Anything consuming that CSV positionally keeps working because the column is last; anything reading it by name is unaffected. - Rollback path: revert this commit. It is self-contained with no migration, no persisted state and no schema change; cache entries written before or after behave identically on read. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Documentation is marked N/A: no user-facing surface changes, and the new `cached=` PERF field is additive and self-describing. Relationship to nearby open PRs, since several touch adjacent code: - **#2953** (already merged, unreleased) added the `resp_json is not None` guard at the same Anthropic store site. That stops an SSE *body* being stored; it does not stop a JSON-bodied response storing chunked framing headers, and it does not add the `stream` gate. The two changes are complementary. - **#2959** and **#2968** both touch the buffered-CCR response path but address when and how the status is committed. Neither reaches the cache-hit replay. - **#3013** rewrites CCR into event-level stream splicing and keeps `buffered_stream_ccr` as a fallback, so the store site this PR gates remains reachable. If #3013 lands first I am happy to rebase. `mypy headroom --ignore-missing-imports` reports 12 pre-existing errors in `headroom/release_version.py`, `headroom/ccr/mcp_server.py` and `headroom/memory/mcp_server.py` from MCP SDK version drift in my local environment. None are in the files this PR touches, and the count is identical before and after the change. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |