Commit Graph
5 Commits
Author SHA1 Message Date
3871b3664a fix(proxy): drop headers named by Connection when forwarding responses (#4674)
## Description

`sanitize_forwarded_response_headers` strips `Connection` but not the
connection-local headers its value names, and `TE`, `Trailer` and
`Upgrade` also survived. Per RFC 9110 section 7.6.1 these must not be
forwarded by an intermediary. The helper now drops every token listed in
any `Connection` value (case-insensitive) and adds `te`, `trailer`,
`upgrade` to `FRAMING_RESPONSE_HEADERS`.

Closes #4265

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/helpers.py`: collect names from `Connection`, extend
the framing header list.
- `tests/test_proxy_response_cache_replay.py`: regression test with real
`httpx.Headers` (duplicate `Connection` fields), also asserting the
input is not mutated. Fails on `main`, passes now.

## Testing

- [x] Unit tests pass: `TestSanitizeForwardedResponseHeaders` 5 passed
(new one failed before the fix)
- [x] Linting passes (`ruff check` / `ruff format --check` on touched
files)
- [ ] Type checking (`mypy headroom`): not run
- [x] New tests added

Note: in my environment 2 other tests in
`tests/test_proxy_response_cache_replay.py`
(`test_cache_hit_replays_a_body_the_client_can_actually_read`,
`test_buffered_ccr_turn_does_not_write_the_response_cache`) fail with
`CancelledError` identically on unmodified `main`; my setup is a partial
install (no Rust extension build), so I did not investigate.

## Real Behavior Proof

- Environment: macOS, Python 3.12, real `httpx.Headers`.
- Steps: called the helper with `Connection: X-Upstream-Hop, Upgrade`,
`X-Upstream-Hop`, `Upgrade: h2c`, `TE: trailers`, `Trailer: Digest`,
`X-Request-ID`.
- Result: before, `x-upstream-hop`, `upgrade`, `te` and `trailer` were
all kept; after, only `x-request-id` remains.
- Not tested: end-to-end through a running proxy and upstream, streaming
paths, full suite, mypy, pre-push hooks.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I did **not** edit `CHANGELOG.md`

This change was prepared with AI assistance (Claude Code).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4674"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-06 21:29:45 -06:00
ca6ba3486c fix(ccr): stream the buffered CCR upstream request so idle-killing networks cannot cut it (#4206)
## Description

The buffered CCR path flips a streaming Claude Code turn to `stream:
false` upstream so the proxy holds the whole reply before serving
`headroom_retrieve`. With `stream: false`, Anthropic sends no byte until
the generation is complete. A middlebox that drops HTTP connections idle
for ~60s (TLS-inspecting antivirus, VPN, corporate proxy; TCP keepalive
does not reset an HTTP-layer idle timer) closes the connection first:
each attempt fails at ~60s with `Server disconnected without sending a
response`, the client gets a 502 after ~186s, and Claude Code resends
the same body forever. The CCR continuation that serves the retrieval
(`anthropic_ccr_continuation`, its own `http_client.post`) is a second
`stream: false` call and is cut the same way. Without Headroom the
client streams and the same turns complete.

This PR keeps the buffered design and makes only its upstream leg
stream. The buffered request and every continuation built from it (CCR,
memory, turn hook) go to first-party Anthropic as `stream: true`, and
the event stream is folded back into the Message JSON the buffered path
already reads, so retrieval, memory, turn hooks, usage accounting and
SSE resynthesis see exactly what they saw before. Bytes (deltas and
pings) now flow for the whole generation, so an idle timer never fires.
The fold reuses `AnthropicSSEEnvelope`, the reconstruction #3130 and
#3266 already use on this path.

Closes #4205

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: new
`_send_buffered_as_stream(send, headers, body, *, url, request_id)`. It
sends `{**body, "stream": true}` with `Accept: text/event-stream` (one
Accept value, any casing replaced) and folds the reply:
- not a 200 event stream (an HTTP error, or an upstream that answered
JSON anyway): returned unchanged;
  - a whole stream: a 200 JSON response holding the folded Message;
- an in-band `error` event: the event as the body, with its own HTTP
status (529 `overloaded_error`, 429 `rate_limit_error`, other documented
types to their status, else 500);
- a stream that ended before `message_stop` or with a block open: the
existing strict 502 `upstream_protocol_error` of
`_adapt_event_stream_to_json` (#3266);
- a complete stream holding a known event the fold cannot read (an
unknown delta type): logged, and the original `stream: false` body is
sent instead. One more generation, but never a 502 the client would
resend forever.
A connection dropped mid-stream raises inside `send`, exactly as for a
`stream: false` reply. An in-band error arrives after upstream already
answered 200, so `_retry_request` does not retry it; the client gets the
status and retries as it would on a direct stream.
- It is used at the four places Headroom sends a body it made
non-streaming: the buffered CCR request (only when
`buffered_stream_ccr`; a client's own `stream: false` request still goes
through `_retry_request` with its original bytes), the CCR continuation
(the `prepare_outbound_body_bytes` + `log_outbound_request` +
`http_client.post` sequence moved into a local `_post_continuation`,
unchanged otherwise, so a fallback attempt logs its own outbound line),
the memory continuation and the turn-hook continuation.
- Gates: the body is not already streaming, the upstream is first-party
Anthropic (`is_custom_anthropic_base_url(url)` is false; the
host-equality check tool-search gating already uses), and the kill
switch `HEADROOM_BUFFERED_UPSTREAM_STREAM` is not
`0`/`false`/`no`/`off`. Copilot and other gateways keep the `stream:
false` request: the fold is checked against Anthropic's own stream, and
a gateway dialect it cannot read would cost every turn a second
generation.
- `headroom/proxy/runtime_env.py`: `HEADROOM_BUFFERED_UPSTREAM_STREAM`
is a registered knob in `RUNTIME_ENV_KNOBS`, read per request through
`runtime_env.getenv`, so a reusing `headroom wrap` pushes it to the
running proxy and `POST /admin/runtime-env` sets it with no restart.
Listed in the `security-model.mdx` allowlist.
- `headroom/proxy/anthropic_wire.py`: `fold_anthropic_stream(raw)`
parses with `AnthropicSSEEnvelope.parse` and trims each block to the
non-streaming Message shape: a null member leaves every block,
`thinking` keeps `type`/`thinking`/`signature`, `redacted_thinking`
keeps `type`/`data`. Claude Opus 5.5 streams a thinking
`content_block_start` with `estimated_tokens: null`; the CCR
continuation resends the folded turn as history, and Anthropic answers
any block carrying that member with 400 `Extra inputs are not
permitted`.
- `headroom/proxy/anthropic_wire.py` `_response_from_events`:
`message_delta.usage` is merged only where non-null, as the Anthropic
SDK's `MessageStream` does. Before, a null count there overwrote the
real `message_start` count (input and cache tokens) in the reconstructed
message.
- `headroom/proxy/buffered_ccr_response.py`: the status-to-error-type
table moves out of `_anthropic_error_type` into `ANTHROPIC_ERROR_TYPES`
so the handler can invert it; `_anthropic_error_type` returns the same
values.
- Unchanged on purpose: `body["stream"] = False` and its mutation reason
`ccr_streaming_retrieve_buffered_non_stream` stay the buffered path's
internal representation (every later step, the response-cache guard and
the #3587 log counts key on them), and the `using buffered stream:false
upstream request` log line keeps its text.
- Tests: new `tests/test_buffered_ccr_upstream_stream.py` (16 tests):
the fold equals the Anthropic Python SDK's `accumulate_event` result on
a stream with thinking, signature, redacted thinking, text, citations,
split `input_json_delta`, pings and a null usage count; stream-only
members are dropped; truncated, in-band error and unknown-delta
outcomes; the send helper's gates (kill switch, gateway URL, already
streaming), passthrough of non-stream replies, 529 for an in-band
overload, 502 for a truncated stream, the `stream: false` fallback; and
through the handler, the buffered turn and its CCR continuation both go
upstream `stream: true` with `Accept: text/event-stream` and the
continuation's resent thinking block carries only its schema, while a
client's own `stream: false` request goes out `stream: false` with
`Accept: application/json` and its original bytes.
- Existing tests that pinned `stream: false` on the wire for a buffered
turn now pin `stream: true` (`test_anthropic_streaming_ccr_retrieve.py`,
`test_ccr_buffered_stream_signed_thinking.py`,
`test_proxy_response_cache_replay.py`,
`test_no_ccr_disables_response_handling.py`).
`test_anthropic_buffered_ccr_usage.py` now answers the buffered turn
with an event stream, so usage booking is checked through the fold.
`test_buffered_ccr_accept_header.py` (#3078) pins the `stream: false`
leg it is about with the kill switch; custom upstreams such as Copilot
always take that leg.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) - ran on the 3 changed
source files only, see below
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_buffered_ccr_upstream_stream.py -q
16 passed, 1 warning in 2.89s

# every test file that touches the buffered CCR path, the CCR continuation or SSE reconstruction
$ pytest tests/test_buffered_ccr_upstream_stream.py tests/test_sse_safeguards_passthrough.py \
    tests/test_buffered_ccr_salvage.py tests/test_anthropic_buffered_ccr_usage.py \
    tests/test_buffered_ccr_grace_window.py tests/test_anthropic_buffered_sse.py \
    tests/test_ccr_buffered_stream_signed_thinking.py tests/test_toin_feedback.py \
    tests/test_proxy_response_cache_replay.py tests/test_proxy/test_anthropic_buffered_timeout.py \
    tests/test_anthropic_auto_mode_passthrough.py tests/test_proxy/test_openai_responses_ccr.py \
    tests/test_sse_thinking_blocks.py tests/test_streaming_sse_malformed_events.py \
    tests/test_route_advice.py tests/test_memory_tool_stream.py \
    tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy_byte_faithful_forwarding.py \
    tests/test_buffered_ccr_accept_header.py tests/test_anthropic_buffered_memory_continuation.py \
    tests/test_anthropic_ccr_workspace_unbound.py tests/test_no_ccr_disables_response_handling.py \
    tests/test_proxy_ccr.py tests/test_ccr_inline_resolve_handlers.py \
    tests/test_proxy/test_anthropic_ccr_raise.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py \
    tests/test_nonstream_sse_policy.py tests/test_ccr_response_handler_extra.py \
    tests/test_compress_route_tokenizer_by_model.py -q
855 passed, 6 warnings in 57.94s

# the updated assertions fail on main's source (they pin stream:true on the wire)
$ git checkout upstream/main -- headroom/ && pytest tests/test_anthropic_buffered_ccr_usage.py \
    tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_ccr_buffered_stream_signed_thinking.py \
    tests/test_no_ccr_disables_response_handling.py tests/test_proxy_response_cache_replay.py -q --tb=no
12 failed, 33 passed, 1 warning in 13.64s

# whole suite
$ pytest tests -q
18 failed, 16435 passed, 583 skipped, 119 warnings in 1226.31s (0:20:26)
# 16 of the 18 fail the same way on unmodified main in the same environment
# (tests/test_proxy_uds.py x11, tests/test_cli/test_wrap_persistent.py x4,
#  tests/test_pricing_from_litellm.py groq/llama-guard-3-8b x1); the other 2
# (test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request,
#  test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children) pass when run alone on this branch.

$ ruff check .
All checks passed!
$ ruff format --check .
1790 files already formatted
$ mypy headroom/proxy/anthropic_wire.py headroom/proxy/buffered_ccr_response.py headroom/proxy/handlers/anthropic.py
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 27.0.1 arm64, Python 3.11.13. Upstream `main`
f1bc2609 versus this branch, same venv.
- Exact command / steps: `repro_buffered_idle_cut.py` (full script in
#4205) runs the real proxy app (`create_app(ProxyConfig(...))` +
`TestClient`, real `_retry_request`, real `http_client`) against a fake
Anthropic upstream that takes 5s per reply (stream: a delta or ping
every 0.5s; non-stream: silent for 5s, then JSON), reached through a TCP
relay that closes any connection silent for 2s, a scaled-down 60s
middlebox. Scenario A: a streaming turn with a redeemable marker
(buffered path), slow reply. Scenario B: the model calls
`headroom_retrieve` in 0.3s and the continuation is slow. The script
points the first-party host check at the fake upstream and isolates
`HOME` and the savings ledger. `PYTHONPATH=<checkout> python
repro_buffered_idle_cut.py 2>/dev/null`
- Observed result: on main both scenarios fail and every upstream
request goes out `stream: false` and is cut; on the branch both complete
in about the generation time with no cut; with
`HEADROOM_BUFFERED_UPSTREAM_STREAM=0` on the branch the main behaviour
returns.

```text
# main f1bc2609
Scenario A (slow buffered turn)
  client: HTTP 200 after 10.1s -> FAIL: 20 keepalive pings, then {"type": "error", "error": {"type": "api_error", "message": "An error occurred while processing your request. Please try again."}}
  upstream requests: ['turn stream=false', 'turn stream=false', 'turn stream=false']
  connections the relay cut for idling: 3

Scenario B (slow CCR continuation)
  client: HTTP 502 after 2.3s -> FAIL: 0 keepalive pings, then {"type": "error", "error": {"type": "api_error", "message": "Unable to safely complete streamed CCR retrieval."}}
  upstream requests: ['retrieve call stream=false', 'continuation stream=false']
  connections the relay cut for idling: 1

# proxy log on main
WARNING - Request failed (attempt 1), retrying in 1322ms: Server disconnected without sending a response.
WARNING - Request failed (attempt 2), retrying in 2536ms: Server disconnected without sending a response.
ERROR - [hr_1791973393_000001] Request failed: RemoteProtocolError: Server disconnected without sending a response.
ERROR - CCR: Continuation API call failed: RemoteProtocolError: RemoteProtocolError('Server disconnected without sending a response.')
WARNING - [hr_1791973403_000002] CCR: Buffered streaming response still contains an unresolved headroom_retrieve after handling; failing closed

# this branch
Scenario A (slow buffered turn)
  client: HTTP 200 after 5.2s -> OK: the client got the final answer
  upstream requests: ['turn stream=true']
  connections the relay cut for idling: 0

Scenario B (slow CCR continuation)
  client: HTTP 200 after 5.3s -> OK: the client got the final answer
  upstream requests: ['retrieve call stream=true', 'continuation stream=true']
  connections the relay cut for idling: 0

# this branch, HEADROOM_BUFFERED_UPSTREAM_STREAM=0
Scenario A (slow buffered turn)
  client: HTTP 200 after 9.0s -> FAIL: 16 keepalive pings, then {"type": "error", "error": {"type": "api_error", "message": "An error occurred while processing your request. Please try again."}}
  upstream requests: ['turn stream=false', 'turn stream=false', 'turn stream=false']
  connections the relay cut for idling: 3

Scenario B (slow CCR continuation)
  client: HTTP 502 after 2.4s -> FAIL: 0 keepalive pings, then {"type": "error", "error": {"type": "api_error", "message": "Unable to safely complete streamed CCR retrieval."}}
  upstream requests: ['retrieve call stream=false', 'continuation stream=false']
  connections the relay cut for idling: 1
```

- Not tested: this branch has not sent traffic to api.anthropic.com. The
fold is checked against the Anthropic Python SDK's own accumulator and
against hand-built streams, not against live replies. For context, the
same transport change (a different fold implementation with the same
contract) has run as a downstream patch on the 0.39.0 wheel and was
checked live: 56 turns across Opus, Sonnet, Haiku and Fable models
(thinking, parallel `tool_use`, `web_search` with citations,
`max_tokens` cutoffs) matched the non-streaming reply's top-level, block
and usage shapes; 28 folded replies resent as history all returned 200;
behind a relay that cuts a connection after 20s of silence, a 409s
thinking turn completed (longest silence 2.1s) while the `stream: false`
path looped on 502s. The OpenAI Responses buffered path has the same
flip and is not changed here. Gateways (Copilot, custom base URLs) are
not changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This is a transport fix, not a
`rollout.py` feature.
- Minimum rollout channel: N/A; it applies on every channel.
- Stable/default behavior changed: Yes, for first-party Anthropic only.
A buffered CCR turn and its CCR, memory and turn-hook continuations now
go upstream `stream: true` with `Accept: text/event-stream`, and the
proxy folds the reply into the same Message JSON. Request content is
otherwise identical, so there is no compression, cache or token-count
effect. The client still gets the buffered response it got before
(keepalive pings, then the resynthesized stream), so time to first byte
(#3587) is unchanged. A client's own `stream: false` request, every
custom upstream, and the OpenAI paths are unchanged. One shared change:
`message_delta` usage nulls no longer overwrite `message_start` counts
in `AnthropicSSEEnvelope`, so a re-rendered stream on the streaming path
now carries the real count where it carried `null`.
- Kill switch / disable path: `HEADROOM_BUFFERED_UPSTREAM_STREAM=0`
(also `false`, `no`, `off`), a runtime-env knob read on every request:
set it in the proxy environment, let a reusing `headroom wrap` push it,
or `POST /admin/runtime-env` (no restart). Every buffered request then
goes out `stream: false` exactly as on main
(`test_a_propagated_kill_switch_keeps_the_request_non_streaming`).
- Unsafe override required: No.
- Qualification impact: transport only. A complete stream the fold
cannot read costs one extra generation (logged as `Could not fold the
upstream event stream ...; sending the request again as stream:false`);
that should not happen with the first-party API today and is the safe
answer to a future delta type.
- Rollback path: set `HEADROOM_BUFFERED_UPSTREAM_STREAM=0` (no restart
needed), or revert these commits.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, proxy transport change.

## Additional Notes

- Documentation: no user-facing doc describes the buffered request's
wire shape, so none changed. The kill switch is documented in the
helper's docstring.
- Relation to #3587: that issue is the latency side of the same flip
(time to first byte becomes the whole generation, and the flip fires on
turns where retrieval never redeems). This PR does not change when the
buffered path is taken or what the client sees first; it fixes the
availability side, where a turn longer than a middlebox's idle limit
never arrives. #3587's asks (a stricter redeemability gate, a switch
that disables buffering) are independent of this change.
- Relation to #3013: #3013 replaces the buffered path with event-level
splicing of the client stream behind `ccr_event_streaming` (default on)
and keeps the buffered `stream: false` request as the qualified fallback
when that flag is off. This PR is the small transport fix for the
buffered design as it is on main: it does not touch the
`buffered_stream_ccr` gating or the log line #3013 rewrites, and it adds
no new conflict with #3013 (a `git merge-tree` of #3013's head against
main and against this branch shows the same 3 conflict hunks in
`anthropic.py` and 2 in `test_anthropic_streaming_ccr_retrieve.py`). If
#3013 lands, its buffered fallback gets a streamed upstream leg from
this change for free.
- OpenAI Responses: that handler has the same flip (`openai.py`, `using
buffered stream:false`) and the same exposure; it is left for a
follow-up because its fold is a different event format.
- `compaction_delta` and other delta types `AnthropicSSEEnvelope` does
not reconstruct take the `stream: false` fallback rather than a guess;
teaching the envelope a new delta type removes that extra generation.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4206"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
- Review fix 61b42465: the kill switch is a registered runtime-env knob
(see Changes Made), so the rollback reaches a proxy that `headroom wrap`
reuses. Useful here because a fold failure can cost a second generation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-15 16:29:46 -05:00
f78e66f7cc fix(proxy): stop headroom logger from suppressing propagation to stdout/stderr (#3096)
## Description

`_setup_file_logging()` in `headroom/proxy/helpers.py` unconditionally
set `headroom_logger.propagate = False`, on the theory that `wrap.py`
redirects stderr into the same `proxy.log` file the rotating handler
writes to, so propagation would double-write records. That premise is
wrong: `wrap.py` redirects the proxy subprocess's stdout/stderr to a
separate sibling file, `proxy-stdio.log` (see
`_get_proxy_stdio_log_path()`), never `proxy.log`. Severing propagation
bought no deduplication — it only stopped every `headroom.*` log record
from reaching root's `logging.basicConfig()` handler
(`proxy/server.py`), which is what writes to stderr/stdout. In
containerized deployments (Docker/Kubernetes), stdout/stderr is the only
channel the platform collects, so this silently dropped all application
logs.

Closes #3087

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed `headroom_logger.propagate = False` from
`_setup_file_logging()` in `headroom/proxy/helpers.py`.
- Updated the surrounding comment to explain why propagation is safe to
leave on: `proxy.log` (file handler) and stderr/stdout (root's
`basicConfig` handler) are two distinct destinations, so nothing is
duplicated in either file.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check .
All checks passed!

$ pytest tests/test_corrupt_golden_bytes_recovery.py -q
collected 9 items
tests/test_corrupt_golden_bytes_recovery.py .........                    [100%]
9 passed in 0.47s
```

`mypy headroom` was not run to a clean result: it fails before reaching
this file with a pre-existing, unrelated error in the installed `numpy`
stub (`Type statement is only supported in Python 3.12 and greater`) in
this sandbox's dependency set. Not caused by this change (a 4-line
pure-Python logging tweak).

The broader `pytest tests/` collection also hits ~170 pre-existing,
unrelated errors in this sandbox: `headroom._core`, the compiled Rust
extension, isn't built here (no `cargo`/`maturin` available), so every
module importing it fails at collection. This change doesn't touch Rust
code or `_core` and isn't implicated.

No new tests added: this restores an existing, already-tested code path
(see `tests/conftest.py`'s autouse fixture and
`tests/test_corrupt_golden_bytes_recovery.py`, both of which already
assert/require `propagate=True` for `caplog` to work) to its intended
default rather than adding new behavior.

## Real Behavior Proof

- Environment: local sandbox, Python 3.12.3, unable to build/import
`headroom.proxy.server` directly (missing compiled `headroom._core`).
Verified with an isolated script that copies `_setup_file_logging()`'s
exact logic byte-for-byte plus the same `logging.basicConfig()` call
`proxy/server.py` makes at import time.
- Exact command / steps: ran the fixed logic, then emitted
`logging.getLogger("headroom.proxy").info("compression timeout: request
abc123 exceeded 30s budget")`.
- Observed result: the `headroom.proxy` record was written to stderr AND
to `proxy.log` simultaneously (full output below); re-running the same
script with the old `headroom_logger.propagate = False` line restored
reproduces the bug — the record disappears from stderr entirely and only
lands in `proxy.log`.
  ```text
  propagate flag on 'headroom' logger: True
--- emitting a headroom.proxy log record, watch for it below on stderr
---
2026-08-17 21:17:39,545 - headroom.proxy - INFO - compression timeout:
request abc123 exceeded 30s budget
  --- also present in proxy.log (file handler): OK ---
  ```
- Not tested: end-to-end through the actual compiled proxy binary/server
process (blocked by missing `headroom._core` in this sandbox) and real
container log collection (Docker/Kubernetes) — the isolated repro
exercises the exact same logging configuration code the real process
runs, so this is considered sufficient proxy proof for a pure
logging-config change.

## Runtime Rollout Safety

- Rollout-managed feature(s): None — this is unconditional startup
logging configuration, not behind a rollout flag.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes — `headroom.*` log records now
also reach stderr/stdout (previously file-only). No change to
request/response handling, compression, or any user-facing proxy
behavior.
- Kill switch / disable path: None needed; this restores default Python
logging propagation. If a deployment truly wants file-only logging,
`HEADROOM_LOG_TO_STDOUT`-style suppression could be added later, but no
such request exists today.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this commit to restore `propagate = False`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — logging configuration change, no UI impact.

## Additional Notes

- "New and existing unit tests pass locally" is left unchecked because
the full `pytest tests/` collection can't complete in this sandbox
(missing compiled `headroom._core` Rust extension, unrelated to this
change) — see Test Output above for what was actually run and passed.
- "I have added tests" is left unchecked: existing tests
(`tests/conftest.py` autouse fixture,
`tests/test_corrupt_golden_bytes_recovery.py`) already assert the code
path this restores; happy to add a focused regression test asserting
`logging.getLogger("headroom").propagate is True` after
`_setup_file_logging()` if maintainers want one.

---------

Co-authored-by: = <=>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 00:03:53 -05:00
c502087db7 fix(ccr): only buffer a stream when a marker is actually redeemable (#3092)
## Description

Closes #3071

`headroom_retrieve` is injected once and kept resident for the session
so the tools array stays byte-stable and the prompt cache survives. The
buffered-CCR path keyed on that tool merely being **present**, so once a
session went sticky, *every* later streaming turn was silently converted
to `stream: false`, buffered whole, and resynthesized as SSE:

```
CCR: stream:true request has headroom_retrieve available; using buffered stream:false upstream request
```

Buffering leaves time-to-last-byte roughly unchanged but makes
**time-to-first-byte the entire generation**. The reporter measured 8s
average and up to 100s across 234 requests in one day — turns that would
have streamed a first token in ~1s instead delivered nothing until done.

Retrieval can only expand a `<<ccr:...>>` marker present in the outgoing
body, so a turn carrying none cannot benefit from the buffered path at
all. Gate on that instead of on the tool.

This is also the root cause #3082 traced independently from the OpenCode
side — its plugin registers `headroom_retrieve` unconditionally, so
*every* turn buffered and neither `--no-ccr` nor `HEADROOM_NO_CCR`
stopped it.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- New `_outgoing_body_has_redeemable_marker()` scans the body **about to
go on the wire** and verifies ownership against the compression store —
the same `exists()` check the retrieve endpoint performs, so a
same-shaped marker from another context tool is not adopted (#2836).
Unexpected shapes answer `True`, keeping the long-standing behavior.
- The buffered-stream decision site gates on it, and logs at INFO when
it skips buffering.
- The correctness detail worth reviewing: the check reads `body`,
**not** the earlier `scan_for_markers(optimized_messages)` result.
`optimized_messages` is reassigned five times after that scan (memory
hooks, pre-send extensions, tool-search repair, CCR repair), so reusing
it would have been stale.
- Two existing test files encoded the very coupling this removes and had
to be repaired — see Testing.

Scope: this narrows *when* buffering happens; it does not make buffered
turns stream. A turn that genuinely carries a marker still loses
incremental delivery — restoring streaming there means wiring
`StreamingCCRHandler`, which is #3069's scope. It does not fix #3088
either, whose requests do carry markers.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Two existing files built a request with `headroom_retrieve` and **no**
marker, relying on the tool alone to trigger buffering:

- `tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py` (11 tests)
fell through to the live streaming path, where only `_retry_request` is
stubbed — so the requests reached the network and the file **hung
indefinitely** rather than failing. Seeded real markers; now passes in
~5s.
-
`tests/test_proxy_response_cache_replay.py::test_buffered_ccr_turn_does_not_write_the_response_cache`
asserts its own premise (*"the conversion really happened — otherwise
this test proves nothing"*), so it failed loudly instead of passing
vacuously. Seeded a marker.

New `test_buffering_is_gated_on_a_redeemable_marker` pins all three
directions: owned marker → buffered, no marker → streaming, foreign
marker → streaming.

### Test Output

```text
$ pytest tests/test_ccr_buffered_stream_signed_thinking.py -q
8 passed, 1 warning in 4.45s

$ pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py -q
11 passed, 1 warning in 4.94s        # was: hung indefinitely

$ pytest tests/test_proxy_response_cache_replay.py -q
9 passed, 1 warning in 1.69s

$ pytest tests/ -q
3 failed, 11151 passed, 581 skipped in 398.28s (0:06:38)

Same 3 failures as a clean-main baseline run on this machine:
  tests/test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter
  tests/test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline
  tests/test_release_workflows.py::test_no_native_tls_in_wheel_build_tree

$ ruff check . && ruff format --check .
All checks passed!
```

## Real Behavior Proof

- Environment: this branch driven through the real FastAPI app with the
outbound HTTP client captured; macOS arm64, Python 3.12.
- Exact command / steps: posted a `stream: true` `/v1/messages` request
carrying a resident `headroom_retrieve` tool in three variants — no
marker, a marker seeded into the compression store, and a
correctly-shaped marker the store does not own — recording whether
`_retry_request` saw a `stream: false` body.
- Observed result: no marker → **streams**, `_retry_request` never sees
a flipped body; owned marker → **buffers**, exactly as before; foreign
marker → streams, honoring #2836 rather than adopting another tool's
hash.
- Not tested: the latency improvement against live client traffic. The
mechanism is verified (the buffered conversion no longer occurs), but
the reported 8s → ~1s TTFB needs the reporter's traffic to confirm.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this narrows an existing code path
and is not behind a rollout channel.
- Minimum rollout channel: n/a (ships to stable with the fix).
- Stable/default behavior changed: yes. A streaming turn whose body
carries no redeemable marker now stays streaming instead of being
buffered. Turns carrying a marker are unchanged.
- Kill switch / disable path: no new switch. Existing CCR controls still
apply — disabling the CCR response handler bypasses this decision site
entirely, and the helper fails open (returns `True`, i.e. the old
behavior) on any unexpected message shape.
- Unsafe override required: no.
- Qualification impact: none — no qualification-gated surface is
touched.
- Rollback path: revert this commit. Note it also carries two test
repairs; reverting the production change alone would leave those tests
passing but vacuous.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Documentation update is marked N/A: no user-facing flag or endpoint
changes. Type checking (`mypy headroom`) was not run separately; `ruff`
is the gate this repo's CI enforces.

Related: #2836 (marker ownership), #3069 (streaming CCR handler), #3082,
#3088.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 11:19:26 -07:00
9d370592b0 fix(proxy): stop cached responses replaying the producing turn's wire framing (#3024)
## Description

Closes #3019

A response-cache hit could hand the client an HTTP 200 that the client
could not read, and nothing in the logs marked the turn as anything
other than normal.

Two separate problems combine to produce the reported failure.

**The unreadable 200.** A cache entry stores the producing upstream's
response headers verbatim. When the entry is replayed, the Anthropic
handler removed only `content-encoding`, `content-length` and
`content-type` before handing those headers to a brand-new `Response`.
Anything else describing how that *other* connection framed its body
rode along — most damagingly `transfer-encoding: chunked`. RFC 9112 §6.1
makes `Transfer-Encoding` override `Content-Length`, so the client is
told to parse a plain JSON body as chunked frames, finds no valid
chunk-size line, and reads an empty body out of a 200. Every other
response-forwarding site in the Python proxy already strips that header;
the two cache-hit sites were the only ones that did not.

**How a CCR turn could put a foreign response in the cache.** On the
Anthropic path, `cache.get` is gated on `not stream` but `cache.set` was
not, and the cache key has no `stream` component. A CCR buffered-stream
conversion takes a request the client sent with `stream: true`, forces
`stream: false` upstream, and — unlike every other streaming turn, which
returns via `_stream_response` and never touches the cache — falls
through to the store site. The stored reply was shaped by that forced
flip plus CCR tool injection, and the key cannot distinguish it from an
ordinary non-streaming reply, so a later non-streaming caller could be
served a response built for a request it never made. This is why the
reporters saw the failures pair with CCR activity and stop under
`--lossless` / `--no-ccr`.

**Why it was invisible.** The cache-hit block emitted no log line at
all, and the `PERF` line rendered no field for
`RequestOutcome.from_response_cache`. A cache-served turn contacts no
upstream, so it has no `outbound_request` line, no upstream stage
timings, and all-zero token counters — byte-for-byte what a turn that
died would look like. That is why `headroom doctor` reported zero
failures while turns were dying.

### Scope note

The header fix also lands on the OpenAI cache-hit site, which
additionally never received the `content-type` fix from #2952. The `not
stream` gate is added to the OpenAI store site too, where it is
currently redundant — a streaming chat request returns via
`_stream_response` long before that point — purely to state the
invariant, since the Anthropic handler had exactly that shape until a
buffered-CCR branch began falling through to it.

Because the strip list now lives in one shared helper, the OpenAI
handler's other five forwarding sites strip the three added headers as
well. That is a widening, so it is worth being explicit about: each of
those sites builds a fresh fixed-length `Response` (or, at
`openai.py:6122`, synthesises SSE) from `response.content`, so replaying
the upstream's framing there was the same latent bug, just without a
cache to make it outlive the request that produced it. The precedent is
already in the file — `openai.py:9865` passes `"transfer-encoding",
"connection"` as extra names by hand, which is exactly the gap this PR
closes centrally. That call site keeps its now-redundant arguments;
removing them is a cleanup for another PR.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `sanitize_forwarded_response_headers` to
`headroom/proxy/helpers.py`, promoting the private helper that already
lived in `headroom/proxy/handlers/openai.py` and extending it with the
remaining wire-framing headers (`transfer-encoding`, `connection`,
`keep-alive`). Matching is now case-insensitive; surviving headers keep
their original casing. `openai.py`'s
`_sanitize_forwarded_response_headers` is now a thin alias so its six
call sites and the Anthropic handler strip an identical set.
- `headroom/proxy/handlers/anthropic.py`: the response-cache hit now
sanitises through that helper (passing `content-type` as an extra name,
preserving #2952) instead of three hand-rolled `pop` calls.
- `headroom/proxy/handlers/openai.py`: the response-cache hit sanitises
the same way, gains the `content-type` handling it was missing, and sets
`media_type="application/json"` explicitly.
- `headroom/proxy/handlers/anthropic.py`: `cache.set` is now gated on
`not stream`, mirroring the read gate. `stream` still holds the client's
original flag at that point — the buffered-CCR conversion flips
`body["stream"]`, never the local variable.
- `headroom/proxy/handlers/openai.py`: the same `not stream` gate on its
store site, as an invariant guard.
- Both cache-hit sites now log `RESPONSE-CACHE-HIT: model=… bytes=…
age_s=… hits=…`, following the existing `CACHE-MISS-ATTRIBUTION` line
style.
- `headroom/proxy/outcome.py`: the `PERF` line appends `cached=1` on a
response-cache hit. It is appended only on a hit, so every other PERF
line is byte-identical to before and existing parsers are unaffected.
- `headroom/perf/analyzer.py`: `PerfRecord.from_response_cache` reads
that field, so `headroom perf` can tell a cache-served turn from a dead
one. It defaults to `False`, so older logs still parse.
`PERF_RECORD_FIELDS` gains the name at the end of the list, which is
what `headroom perf --format csv --raw` uses as its column set;
appending keeps every existing column at its current position. `--format
json --raw` gains the key too.
- `tests/test_anthropic_pre_upstream_backpressure.py`: its cache-hit
double was a partial hand-rolled stand-in for `CacheEntry` carrying only
a body and headers, so it broke once the hit path started reading the
entry's age and hit count. It now constructs a real `CacheEntry`, which
is what the cache actually returns.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
$ python -m pytest tests/test_proxy_response_cache_replay.py -q
tests\test_proxy_response_cache_replay.py .........                      [100%]
============================== 9 passed in 4.22s ==============================

# Everything that mentions PERF, the sanitiser, cache.set, PerfRecord or
# response_headers, plus the whole proxy suite.
$ python -m pytest tests/test_proxy/ tests/test_proxy_compression_headers.py \
    tests/test_agent_savings.py tests/test_anthropic_pre_upstream_backpressure.py \
    tests/test_backend_nonstreaming_cache_metrics.py tests/test_backend_streaming_cache_metrics.py \
    tests/test_ccr_buffered_stream_signed_thinking.py tests/test_cli_perf_format.py \
    tests/test_codex_ws_compression_scheduler.py tests/test_handler_outcome_tag_invariant.py \
    tests/test_openai_codex_ws_lifecycle.py tests/test_provider_codex_images.py \
    tests/test_proxy_handlers_batch.py tests/test_proxy_passthrough_transient_retry.py \
    tests/test_proxy_response_cache_replay.py tests/test_proxy_semantic_cache_key.py \
    tests/test_proxy_streaming_request_logger.py tests/test_request_outcome.py \
    tests/test_savings_tool_search_aggregation.py -q
================== 555 passed, 1 skipped in 88.60s (0:01:28) ==================

# Full suite, 16 workers. See "Real Behavior Proof" below for how every
# failure here was traced to a pre-existing failure or a parallelism flake.
$ python -m pytest tests scripts/tests -n 16 -q -p no:randomly --timeout=300
83 failed, 10493 passed, 657 skipped, 80 errors in 437.00s (0:07:17)

$ ruff check .
All checks passed!

$ ruff format --check <the 7 changed files>
7 files already formatted

$ python -m mypy headroom --ignore-missing-imports --python-version 3.13
Found 12 errors in 3 files (checked 520 source files)
# All 12 are pre-existing MCP-SDK/tomllib drift in release_version.py,
# ccr/mcp_server.py and memory/mcp_server.py; identical count before and
# after this change, none in the files it touches.
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, ruff 0.16.2,
branch based on `upstream/main` at `2d88e31a`.
- Exact command / steps: Two experiments. (1) Revert-and-rerun: I
reverted both fixes in place (dropped the three framing headers from
`FRAMING_RESPONSE_HEADERS`, restored `cache.set` to `if self.cache and
response.status_code == 200 and resp_json is not None:`), ran `python -m
pytest tests/test_proxy_response_cache_replay.py -q`, then restored the
fixes and re-ran. (2) Regression sweep: ran the full suite on this
branch, then checked out `upstream/main` into a second worktree and
re-ran, in that worktree, exactly the tests that failed here and not
there.
- Observed result: With the fixes reverted, 5 of 9 new tests fail and
reproduce both halves of the bug.
`test_buffered_ccr_turn_does_not_write_the_response_cache` fails with
`AssertionError: Expected mock to not have been awaited. Awaited 1
times.` — a turn the client sent as `stream: true` really does reach
`cache.set` through the buffered-CCR branch.
`test_cache_hit_replays_a_body_the_client_can_actually_read` fails with
`AssertionError: assert 'transfer-encoding' not in {'transfer-encoding':
'chunked', 'connection': 'keep-alive', 'request-id': ...,
'content-length': '228', ...}` — the replayed 200 carries the producing
turn's chunked framing alongside a fresh `content-length`, which is the
exact framing conflict a client cannot parse. With the fixes restored,
all 9 pass, the replayed body arrives intact as `application/json`, and
the run logs both `RESPONSE-CACHE-HIT` and a `PERF … cached=1` line. The
full suite on this branch gives `83 failed, 10493 passed, 657 skipped,
80 errors`; 33 of those failures were not in my baseline list, so I ran
those 33 in the `upstream/main` worktree and 20 failed there identically
(Windows-specific: `sqlite:///C:\…` path handling, private-directory
permissions, fsync, ONNX thread caps, serena config discovery).
Re-running the remaining 13 serially on this branch gave `1 failed, 25
passed` — the other 12 were xdist parallelism flakes, including all four
`tests/test_proxy/test_anthropic_ccr_deferred_injection.py` tests, which
are the only ones in this change's blast radius and which pass serially.
The one real serial failure,
`tests/test_savings_ledger_offload.py::test_concurrent_requests_all_land_their_events`
(`AssertionError: a concurrent append was lost / assert 23 == 24`),
fails the same way on `upstream/main` run serially. The 80 errors are
dashboard-template collection errors unrelated to the proxy. Net: no
failure attributable to this change.
- Not tested: I could not reproduce against live upstream traffic, so I
have not confirmed which upstream in the reporters' setups emits
`transfer-encoding: chunked`. Anthropic direct is HTTP/2, where the
header is forbidden, but any HTTP/1.1 hop (corporate proxy, third-party
gateway, local relay) reintroduces it. I have also not measured whether
the `not stream` gate reduces the cache hit rate in practice; by
construction it can only drop entries that were unsafe to serve. A
reporter running unmodified 0.35.0 with `headroom proxy --no-cache`
would confirm the cache path is the one involved, and that flag is a
lighter workaround than `--lossless` or `--no-ccr` because it keeps CCR
and compression enabled.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this is a correctness fix on the
always-on response-cache path (`cache_enabled` defaults to `True`).
- Minimum rollout channel: stable.
- Stable/default behavior changed: yes, in four ways. Replayed cached
responses no longer carry the producing upstream's framing headers (or
`server`, on the Anthropic side). Forwarded responses on the OpenAI
handler's other five sanitiser call sites no longer carry
`transfer-encoding`, `connection` or `keep-alive` either, since the
strip list is now shared; all five build a fixed-length response from
`response.content`, so none of them could legitimately replay that
framing. A turn whose client asked for `stream: true` no longer writes
the response cache on the Anthropic path. `PERF` lines gain a trailing
`cached=1` on a response-cache hit only; all other PERF lines are
unchanged.
- Kill switch / disable path: `headroom proxy --no-cache` disables the
response cache entirely and bypasses every path this PR touches.
- Unsafe override required: no.
- Qualification impact: low. No public API, config key, CLI flag or wire
format changes. Two additive output changes: the `cached=1` PERF field,
which `_parse_kv` already handles the same way it handles the existing
trailing `client=` field, and a `from_response_cache` column appended to
`headroom perf --format csv --raw` (plus the matching key in `--format
json --raw`). Anything consuming that CSV positionally keeps working
because the column is last; anything reading it by name is unaffected.
- Rollback path: revert this commit. It is self-contained with no
migration, no persisted state and no schema change; cache entries
written before or after behave identically on read.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Documentation is marked N/A: no user-facing surface changes, and the new
`cached=` PERF field is additive and self-describing.

Relationship to nearby open PRs, since several touch adjacent code:

- **#2953** (already merged, unreleased) added the `resp_json is not
None` guard at the same Anthropic store site. That stops an SSE *body*
being stored; it does not stop a JSON-bodied response storing chunked
framing headers, and it does not add the `stream` gate. The two changes
are complementary.
- **#2959** and **#2968** both touch the buffered-CCR response path but
address when and how the status is committed. Neither reaches the
cache-hit replay.
- **#3013** rewrites CCR into event-level stream splicing and keeps
`buffered_stream_ccr` as a fallback, so the store site this PR gates
remains reachable. If #3013 lands first I am happy to rebase.

`mypy headroom --ignore-missing-imports` reports 12 pre-existing errors
in `headroom/release_version.py`, `headroom/ccr/mcp_server.py` and
`headroom/memory/mcp_server.py` from MCP SDK version drift in my local
environment. None are in the files this PR touches, and the count is
identical before and after the change.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-08-16 15:04:01 -07:00