Commit Graph
37 Commits
Author SHA1 Message Date
854a696544 fix(proxy): retry a TLS error only while the request is being written (#4776)
## Description

A corrupt TLS record after the handshake surfaces as a raw
`ssl.SSLError` (for example `[SSL: RECORD_LAYER_FAILURE] record layer
failure`, or the `SSLV3_ALERT_BAD_RECORD_MAC` bursts in #1615). httpcore
maps `ssl.SSLError` to `ConnectError` only in `start_tls`; its
`read`/`write` maps cover only anyio's resource errors and timeouts, and
anyio's `TLSStream` re-raises a non-EOF `SSLError` as it is. So the
error is not an `httpx.TransportError`, and the #1639 retry in
`_retry_request` and `_stream_response` lets it through.

That error alone does not say whether upstream received the request.
This PR retries it only when httpcore reports that writing the request
failed: its documented `trace` request extension emits
`send_request_headers.failed` or `send_request_body.failed`. Upstream
then never had a complete request to run. A TLS error raised after the
request was written propagates, as on main, so a POST upstream may have
accepted is never sent twice. In the streaming path it still ends as the
SSE 502, now with `upstream_tls_error` instead of a generic `api_error`.

What this no longer fixes: the case the first version of this PR
targeted, a corrupt record in the reply on a reused HTTP/1.1 keep-alive
connection, happens after the request was written. It is ambiguous, so
it still fails the request (502), as on main. What it does fix: a TLS
failure while the request is still being written, for example on a
shared HTTP/2 connection where a large body waits for flow-control
window and the read for that window hits the corrupt record.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `helpers.py`: `RequestWriteTrace`, an httpcore `trace` callback that
records whether `send_request_headers` or `send_request_body` failed.
`forbids_retry(exc)` is true for a raw `ssl.SSLError` raised after a
complete write.
- `server.py` (`_retry_request`) and `handlers/streaming.py`
(`_stream_response`): each attempt attaches a fresh trace, and a raw
`ssl.SSLError` is retried only when that trace shows the write failed.
Every other retry is unchanged. The streaming exhaustion handler still
catches `ssl.SSLError`, for the 502 `upstream_tls_error` SSE body.
- `tests/test_h2_stream_reset_retry.py`: the SSL cases now run through
real httpcore over a fake socket. An accepted POST sends one POST and
propagates, in the buffered and the streaming path. A failed write is
retried, in both paths. The review's harness (an `httpx.MockTransport`
that records the POST, then raises `ssl.SSLError`) sends one POST.
- Three test fakes of `http_client.post` accept the new `extensions`
kwarg.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_h2_stream_reset_retry.py -q
11 passed in 0.39s

# source reverted to the previous head's retry (240680df), tests kept:
FAILED tests/test_h2_stream_reset_retry.py::test_retry_request_does_not_replay_a_post_upstream_received   (assert 2 == 1)
FAILED tests/test_h2_stream_reset_retry.py::test_stream_does_not_replay_a_post_upstream_received          (assert 2 == 1)
FAILED tests/test_h2_stream_reset_retry.py::test_ssl_error_without_write_evidence_is_not_retried          (assert 2 == 1)
3 failed, 8 passed

# source as on upstream/main (no SSL retry), tests kept:
FAILED tests/test_h2_stream_reset_retry.py::test_retry_request_retries_ssl_error_while_writing_the_request
FAILED tests/test_h2_stream_reset_retry.py::test_stream_does_not_replay_a_post_upstream_received   (raw SSLError escapes; no upstream_tls_error 502)
FAILED tests/test_h2_stream_reset_retry.py::test_stream_retries_ssl_error_while_writing_the_request
3 failed, 8 passed

$ uv run --frozen --extra dev pytest <134 files: the tests this PR and main changed since 200c838c, every test importing handlers.streaming, and every test calling _retry_request, _stream_response or a fake http_client> -q
2457 passed, 2 failed, 22 skipped in 178s
# both failures are timing assertions in test_read_maturation_handler_nobust.py, which replaces
# _retry_request with a mock. Run alone 3 times, that file fails 1 test in 3 of 3 runs on this head
# and in 2 of 3 runs with server.py, streaming.py and helpers.py as on upstream/main.

$ uvx ruff@0.17.3 check <touched files>             All checks passed!
$ uvx ruff@0.17.3 format --check <touched files>    already formatted
$ mypy headroom/proxy/helpers.py headroom/proxy/server.py headroom/proxy/handlers/streaming.py --ignore-missing-imports   Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 27.2 (Apple Silicon), Python 3.13, httpx 0.28.1,
httpcore 1.0.9, anyio 4.14.2. `headroom proxy --port 6811
--no-rate-limit --anthropic-api-url https://127.0.0.1:8445` (plus
`--no-http2` in run 1), a clean temp HOME per run, `HEADROOM_CA_BUNDLE`
pointing at a local test CA. Each run starts its own stub and proxy and
stops both.
- Exact command / steps: Controlled upstream: a local TLS stub, not a
real provider. Run 1 (HTTP/1.1, accepted POST): the stub reads the whole
request and counts it, then writes a corrupt TLS application-data record
instead of a reply and closes; a byte-identical replay of a body it has
seen gets a normal 200. One buffered and one streaming `POST
/v1/messages` through the proxy, on upstream/main 9f2cc017, on the
previous head 240680df and on this head. Run 2 (HTTP/2, write failure):
on the first connection the stub grants no flow-control window beyond
the default 65,535 bytes, so a 418,702-byte streaming POST is still
being written when the stub writes the corrupt record; later connections
are served normally. On main and on this head.
- Observed result: Run 1: main and this head send each POST once and
return 502; 240680df sends each POST twice, byte-identical, and returns
the second reply as 200. Run 2: main returns 502 `api_error`; this head
retries on a fresh connection and returns 200, and the stub received one
complete POST.
- Not tested: A real provider sending a corrupt record, and a real
HTTP/2 connection shared by several streams when one of them hits it.

```text
Run 1: controlled upstream, HTTP/1.1, TLS broken after the whole POST was received
version         request     POSTs the stub received    client got
upstream/main   buffered    1                          HTTP 502 api_error
upstream/main   streaming   1                          HTTP 502 api_error
240680df        buffered    2 (byte-identical)         HTTP 200 (reply to the replay)
240680df        streaming   2 (byte-identical)         HTTP 200 (reply to the replay)
this head       buffered    1                          HTTP 502 api_error
this head       streaming   1                          HTTP 502, SSE error event with "code": "upstream_tls_error"

Run 2: controlled upstream, HTTP/2, TLS broken at 65,535 of 418,702 body bytes
upstream/main   streaming   1 incomplete               HTTP 502 api_error
this head       streaming   1 incomplete, 1 complete   HTTP 200 (message_start ... message_stop)
  WARNING [hr_1794561952_000001] Connection error to upstream API (attempt 1/3): SSLError(1, '[SSL: RECORD_LAYER_FAILURE] record layer failure (_ssl.c:2660)'); retrying in 899ms
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. A raw TLS error raised while the
request is being written is retried on a fresh connection, like other
transport errors. A raw TLS error after the request was written is not
retried, as on main; in the streaming path it now ends as a 502
`upstream_tls_error` SSE error instead of a 502 `api_error`.
- Kill switch / disable path: `--retry-max-attempts 1`
(`HEADROOM_RETRY_MAX_ATTEMPTS=1`) disables the retry, as for every
transport error.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- Changed after review: the first version retried any raw `ssl.SSLError`
before the first byte reached the client. JerrettDavis and Devin showed
that replays a POST upstream may already have accepted. This version
retries only a failed write, and the title now says so.
- No idempotency key is added. The retry rests only on httpcore
reporting that the request write failed.
- Not changed here: the existing #1639 retries of `httpx.TransportError`
(for example a `ReadError` or `ReadTimeout` after the request was
written) carry the same after-the-write ambiguity. This PR leaves them
as they are.
- Companion PR #4775 is independent of this change. The
cancellation-stress numbers quoted in the earlier version of this body
were measured with the broader retry, so I removed them.
- Documentation: no doc describes the retry classes, so there is nothing
to update.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 22:56:52 -06:00
pokemonred06-creatorandJD Davis e1becdf732 fix(proxy): use HTTP/1.1 client for ChatGPT streaming and model metadata (#4006)
## Description

Ensures that ChatGPT endpoints (`chatgpt.com`, including `/v1/responses`
forwarded to `https://chatgpt.com/backend-api/codex/responses`,
`/v1/models`, Codex subpaths, and retried requests) consistently utilize
`http_client_h1` instead of multiplexing over the default HTTP/2 client.

`server.py` already provisions `self.http_client_h1` specifically
because ChatGPT's Cloudflare edge challenges HTTP/2 fingerprints on
sensitive endpoints. When large context requests (e.g. 500KB - 800KB
payload turns) are forwarded over HTTP/2 streaming, HTTP/2
connection/stream flow-control window stalls block the single
multiplexed TCP connection in `httpcore`. When this occurs, all
subsequent requests to ChatGPT hosts (including `/v1/models` and
subsequent turn completions) become blocked waiting behind the stalled
connection, eventually leading to proxy unresponsiveness and accumulated
asyncio task queues.

While `handle_passthrough` in `openai.py` and `openai_images.py` already
selected `http_client_h1`, `StreamingMixin._stream_response_generator`,
`handle_model_metadata_endpoint`, `_retry_request` in `server.py`, and
Codex subpath routes in `proxy_routes.py` previously hardcoded
`self.http_client`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Expose `prefers_http1_passthrough` in
`headroom/proxy/handlers/openai.py` (aliasing
`_prefers_http1_passthrough`).
- Add `_select_streaming_client` to `StreamingMixin` in
`headroom/proxy/handlers/streaming.py` that selects `http_client_h1`
when the target URL matches ChatGPT endpoints.
- Update `_stream_response_generator` and `_compact_sse_stream` to use
`_select_streaming_client`.
- Update `_retry_request` in `headroom/proxy/server.py` to route ChatGPT
target URLs through `http_client_h1`.
- Update `_register_openai_responses_subpath_route` in
`headroom/providers/proxy_routes.py` to forward through `http_client_h1`
for `handle_chatgpt_codex_responses_subpath`.
- In `headroom/providers/model_metadata.py`, pass `getattr(proxy,
"http_client_h1", None) or proxy.http_client` to
`handle_chatgpt_model_metadata`.
- Add unit tests in `tests/test_proxy_handler_helpers.py` and
`tests/test_provider_model_metadata.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run pytest tests/test_proxy_handler_helpers.py tests/test_provider_model_metadata.py
======================== 76 passed, 1 warning in 8.24s ========================

uv run ruff check headroom/ tests/test_proxy_handler_helpers.py tests/test_provider_model_metadata.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS 27 (Darwin arm64), Python 3.13, Headroom Proxy
v0.39.1 / v0.40.0, Codex CLI and OpenClaw Desktop client.
- Exact command / steps: Sent 800 KB streaming turns to /v1/responses
targeting https://chatgpt.com/backend-api/codex/responses while
monitoring TCP socket states via lsof/netstat and triggering concurrent
/v1/models calls.
- Observed result: Streaming responses and model metadata resolve over
HTTP/1.1 without multiplexed stream stall; Send-Q drains cleanly, turns
finish in ~2.1s, and all unit tests pass.
- Not tested: Third-party proprietary non-ChatGPT OAuth endpoints (e.g.,
enterprise SSO providers).

## Runtime Rollout Safety

- Rollout-managed feature(s): None
- Minimum rollout channel: stable
- Stable/default behavior changed: No (aligns streaming, retries, and
subpath routes with existing `http_client_h1` passthrough policy for
ChatGPT hosts)
- Kill switch / disable path: Fallback to `self.http_client` when
`http_client_h1` is unset
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)


### Maintainer current-main requalification

Exact head `d6cee507ff0aae7f09705b2a5b9a8bd8264b137f`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Updated against upstream/main e4a60ed7. Resolved the adjacent-helper
conflict by retaining both the contributor's HTTP/1.1 compatibility
alias and main's telemetry helper. HTTP/1.1 selection stays confined to
ChatGPT hosts across streaming/continuations, retries, metadata and
authenticated Responses subpaths; other upstreams retain their existing
client. No contributor production logic was rewritten.

All 136 focused helper/route/metadata tests pass. Three newer route
fixtures initially mocked only the default client; mechanically assigned
their same controlled response client to http_client_h1 too, preserving
every forwarding assertion. Changed-file Ruff lint/format and whitespace
checks pass. Existing macOS/Codex/OpenClaw live-stream evidence remains
scoped; no new live ChatGPT stream or fresh native-build claim is made.
Applied shared test-only Gemini streamed-request fixture repair for
independently established main failures; all ten cases pass. Formal
approval awaits exact-head hosted qualification.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `96dfae54717c13249f3bd9305cc6930030f2c85d`, upstream main
`ecbaf86d78cf52d185fbffa306157b9a4cc1b906`.

Maintainer test-only repair for hosted run42031595103: the newer Codex
alias tests mocked only http_client, while this PR correctly routes
authenticated ChatGPT requests through http_client_h1. Assigned the
existing controlled fake to both client attributes, preserving every
auth/forwarding assertion; no production logic changed.

Main advanced during qualification; cleanly merged upstream/main
ecbaf86d78 and re-reviewed its
dependency/workflow/docs delta. All 154
helper/route/metadata/Codex-alias/Gemini tests pass on the final tree;
lint/format and whitespace checks pass. Existing runtime evidence
remains explicitly scoped, with no new live ChatGPT or fresh
native-build claim. Exact-head hosted checks are handled by maintainers.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-05 10:51:47 -06:00
R-OkauchiandJD Davis 5ad2363798 fix(proxy): keep mid-conversation system sections for third-party models (#4591)
## Description

Follow-up to #3552 and #2991. The #765 wire-contract guard still hoists
every **text** `role="system"` section out of `messages[]` when the
model is not on the Claude 5 allow-list, and on a third-party
Anthropic-compatible upstream (vLLM, GLM, …) it never is. #3553 fixed
the non-text half of #3552; its point 4 ("it also re-introduces #2991's
cache and semantics costs for every custom-upstream user") was left
open.

It fires on both custom-upstream paths:

- a configured target (`--anthropic-api-url` /
`ANTHROPIC_TARGET_API_URL`): the guard gets the real model name, which
is not on the allow-list;
- a per-request `x-headroom-base-url`: the guard gets `model=None`.

The client in this setup appends a mid-conversation system section (a
`<system-reminder>`) on most turns, so on every request the hoisted text
lengthens the top-level `system`. The rendered prompt then changes at
its very start, and the upstream's prefix cache misses the whole
conversation every turn. That happens in cache mode too, whose point is
a byte-identical frozen prefix. Seen with that client → `headroom proxy
--anthropic-api-url http://<vllm>:8000` → vLLM serving
Qwen3.8-Flash-Next: on a ~345K-token session, two consecutive turns each
waited 245 s for the first token (a full re-prefill), and 0 tokens were
compressed. Sent straight to vLLM, the same session keeps its sections
in place and vLLM accepts them.

**Fix.** For a non-Claude model on a third-party upstream
(`upstream_base_url`, or the handler's existing
`_third_party_anthropic_upstream` for a configured target),
valid-position sections are treated as supported. They are forwarded
where the client put them, which is where the client sends them without
Headroom.

The following do not change:

- invalid placements, including the #765 leading `messages[0]` case, are
still relocated;
- Claude-named models on a third-party upstream (`claude-…`,
`anthropic/claude-…`, `global.anthropic.claude-…`) keep today's
conservative hoist, because a gateway may reach a provider without the
feature;
- Anthropic, Vertex, Bedrock and other backends keep the existing
allow-list path. Those only serve Claude-named models, so
`is_third_party_model` is false for them.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`: adds `is_third_party_model(model, *,
third_party_upstream)`. `relocate_system_messages_to_top_level(..., *,
third_party_model=False)` treats such a model as supporting
mid-conversation sections.
- `headroom/proxy/handlers/anthropic.py`: the guard passes
`third_party_model=is_third_party_model(model,
third_party_upstream=bool(upstream_base_url or
_third_party_anthropic_upstream))`. `_third_party_anthropic_upstream` is
assigned unconditionally in the same `try` block, before the guard.
- `tests/test_anthropic_system_relocation_third_party.py` (new): sends a
request through the real handler on both third-party paths (a configured
target and a per-request `x-headroom-base-url`) and checks the forwarded
body. A non-Claude model's section stays in place. A Claude-named model
on those paths, and a non-Claude model on Anthropic itself, are still
relocated.
- `tests/test_proxy_handler_helpers.py`:
  - a decision table for `is_third_party_model`;
  - a check that valid sections stay in place for a third-party model;
- a check that invalid placements (leading, and after an assistant turn)
are still moved.

## Testing

- [x] Unit tests pass (`pytest`): the touched test files, plus the
related files listed below
- [x] Linting passes (`ruff check .`): ruff 0.17.0 on the changed files,
plus `ruff format --check`
- [x] Type checking passes (`mypy headroom`): the two changed modules
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_anthropic_system_relocation_third_party.py tests/test_proxy_handler_helpers.py
89 passed in 4.89s

# the source change reverted, tests kept:
$ pytest tests/test_anthropic_system_relocation_third_party.py
FAILED tests/test_anthropic_system_relocation_third_party.py::test_third_party_model_keeps_mid_conversation_system_sections[upstream0-https://vllm.example]
FAILED tests/test_anthropic_system_relocation_third_party.py::test_third_party_model_keeps_mid_conversation_system_sections[upstream1-https://override.example]
2 failed, 3 passed in 3.95s

$ pytest tests/test_anthropic_system_relocation_third_party.py tests/test_proxy_handler_helpers.py \
    tests/test_custom_base_passthrough_telemetry.py tests/test_system_compaction_handler.py \
    tests/test_session_affinity.py tests/test_proxy_anthropic_cache_stability.py \
    tests/test_proxy_system_prompt_immutable.py tests/test_observed_wire_shapes.py \
    tests/test_anthropic_auto_mode_passthrough.py tests/test_cache_prefix_safety.py \
    tests/test_ccr_tool_injection_cache_stability.py tests/test_header_isolation.py \
    tests/test_anthropic_deferred_tool_lineage.py
276 passed in 20.81s

$ ruff check <changed files> && ruff format --check <changed files>     # ruff 0.17.0
All checks passed!
4 files already formatted

$ mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py --follow-imports=silent
Success: no issues found in 2 source files   # same on the base commit
```

These ran with this branch on `PYTHONPATH` over the headroom-ai 0.42.0
wheel's compiled `_core`; no local Rust build. The full suite was not
run.

## Real Behavior Proof

- Environment: macOS (Darwin 25.4.0, arm64), Python 3.13. Patched is
this branch; stock is headroom-ai 0.42.0, whose
`relocate_system_messages_to_top_level` and guard call site are
identical to `main` before this change. Both run as `headroom proxy
--anthropic-api-url <upstream> --mode cache --no-cache --no-rate-limit
--no-subscription-tracking` with `HEADROOM_OFFLINE=1`.
- Exact command / steps: (1) Wire capture: a stub Anthropic-compatible
upstream records every forwarded body. Two agent-shaped streaming turns
for model `qwen3.8-flash-next` go through each proxy, and turn 2 adds a
trailing `<system-reminder>` section. A script prints what reached the
upstream (output below). (2) Live run: the same client in print mode,
with a Bash tool, sends a three-request tool-using task through each
proxy to vLLM serving Qwen3.8-Flash-Next on an NVIDIA GB10, and the
guard's warnings in the proxy log are counted.
- Observed result: (1) stock 0.42.0 hoists both sections, and the
top-level `system` changes between turns; this branch forwards both
requests unchanged, with the same top-level `system` on both turns. (2)
On the live run, stock 0.42.0 logged `Relocated role=system` 4 times for
the task and this branch 0 times. The task was answered correctly, and
on this branch the last request (~15.2K tokens) returned in ~5 s, so its
prefix was cached.
- Not tested: the Anthropic API itself, Bedrock, Vertex and Foundry
(their paths are unchanged for Claude-named models); the OpenAI and
Gemini handlers (untouched); the full test suite; a local Rust build of
`_core`.

```text
== stock (headroom, version 0.42.0)
turn 1: forwarded roles=['user', 'assistant', 'user']
        system blocks=2 sha=85b84b30e488
turn 2: forwarded roles=['user', 'assistant', 'user', 'assistant', 'user']
        system blocks=3 sha=b6cc1c0d07e7
top-level system identical across turns: False
  WARNING - Relocated role=system message(s) out of messages[] into the top-level system parameter (Anthropic wire-contract guard, issue #765)
  INFO - event=outbound_request forwarder=streaming method=POST body_bytes=403 body_mutated=true mutation_reasons=system_role_relocated source=canonical
  WARNING - Relocated role=system message(s) out of messages[] into the top-level system parameter (Anthropic wire-contract guard, issue #765)
  INFO - event=outbound_request forwarder=streaming method=POST body_bytes=582 body_mutated=true mutation_reasons=system_role_relocated source=canonical

== patched (this branch)
turn 1: forwarded roles=['user', 'system', 'assistant', 'user']
        system blocks=1 sha=d8329387e7a7
turn 2: forwarded roles=['user', 'system', 'assistant', 'user', 'assistant', 'user', 'system']
        system blocks=1 sha=d8329387e7a7
top-level system identical across turns: True
  INFO - event=outbound_request forwarder=streaming method=POST body_bytes=435 body_mutated=false mutation_reasons= source=passthrough
  INFO - event=outbound_request forwarder=streaming method=POST body_bytes=631 body_mutated=false mutation_reasons= source=passthrough
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none. The wire-contract guard is
unconditional.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, only for a non-Claude model on a
third-party Anthropic upstream. Its valid mid-conversation
`role="system"` sections are forwarded unchanged instead of hoisted.
- Kill switch / disable path: none added
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a, see
below)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Documentation: no page describes the guard's model allow-list, so none
changes.
- If you would rather keep the hoist and give operators a switch
(#3552's suggestion 3), I can rework this into that. Keeping the
client's sections seemed the smaller change, and it matches how they
reach the upstream without Headroom.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 08:43:39 -06:00
b1f340e0b3 fix(ccr): keep proactive expansion out of the cached prefix (#4300)
## Description

CCR proactive expansion used to append retrieved originals into the
newest
user turn. That turn sits inside the cached prefix when the client's
trailing system message holds the cache breakpoint. The extra text
existed
only in the forwarded body and in the prefix-tracker snapshot; the
client
never resent it.

On the next turn, cache continuity depended on overlay replaying that
injection. When the tracker was gone (proxy restart) or
`confirmed_frozen_count` was 0, overlay declined the larger replay and
rewrote the conversation from the expanded message, from message 0 if
that was the first turn.

This change places the expansion in a new text block immediately after
the
last message-level `cache_control` block and omits that block from the
prefix-tracker snapshot. Existing prefix bytes stay identical, role
alternation is unchanged, and a restart or declined replay costs only
that
turn's uncached injection.

Closes #4287

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Inject CCR expansion after the last cache breakpoint instead of into
the latest user turn.
- Strip expansion blocks before
`PrefixCacheTracker.update_from_response` on the Anthropic HTTP,
Bedrock, and streaming paths.
- Keep cache-mode skip of the append as-is.
- When the breakpoint block is one of several parallel `tool_result`
blocks, insert the expansion after the whole consecutive `tool_result`
group, so the turn stays `[tool_result, tool_result, text]` instead of
`[tool_result, text, tool_result]`.
- Mark the block the proxy inserts and strip only that exact block from
the tracker snapshot. Client-authored text that happens to use the
expansion tags is no longer removed.
- Raise the `aiohttp` constraint to `>=3.14.4` (CVE-2026-111063,
CVE-2026-111064, GHSA-2g87-pp6c-29x5) and relock; uv resolves aiohttp
3.14.5 and yarl 1.25.1.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy`) on the changed handler
- [x] New tests added for new functionality
- [x] Tests fail on current `main` and pass on this branch

### Test Output

```text
pytest tests/test_proxy_handler_helpers.py tests/test_ccr_expansion_conversation_scope.py \
  tests/test_cache_prefix_overlay.py tests/test_ccr_context_tracker.py \
  tests/test_bedrock_prefix_tracker_wiring.py tests/test_backend_streaming_cache_metrics.py \
  tests/test_proxy_streaming_request_logger.py tests/test_proxy_anthropic_cache_stability.py \
  tests/test_cache_breakpoint_diagnostics.py tests/test_cross_turn_cache_safety.py \
  tests/test_anthropic_ccr_workspace_unbound.py -q
225 passed, 1 skipped

pytest tests scripts/tests -q   (local macOS, offline)
41 failed, 16689 passed, 627 skipped, 2 errors
The same failing set fails identically on unmodified main in this environment
(missing langchain_anthropic / PIL, live-API and UDS/network tests); none are in
the CCR or Anthropic handler suites.

ruff check .            All checks passed!
ruff format --check .   1802 files already formatted
mypy headroom/proxy/handlers/anthropic.py --ignore-missing-imports   Success
uv lock --check         Resolved 251 packages
pip-audit (uv export --extra all): aiohttp findings cleared; newer advisories
  on certifi 2026.1.4 (fix 2026.10.19) and pillow 12.3.0 (fix 13.0.0) also
  fail on main and are not addressed here
```

## Real Behavior Proof

- Environment: Linux, Python 3.12, token mode, CCR proactive expansion
on, FastAPI TestClient against `/v1/messages`.
- Exact command / steps: `pytest
tests/test_ccr_expansion_conversation_scope.py::test_expansion_stays_out_of_the_cached_prefix_and_tracker_snapshot
tests/test_cache_prefix_overlay.py` (TestClient request to
`/v1/messages` with the trailing system message holding `cache_control`,
then overlay with `confirmed_frozen_count=0` as a stand-in for the
restart path).
- Observed result: the client user turn is forwarded unexpanded;
expansion sits after the trailing system's `cache_control`; the tracker
snapshot has no expansion; overlay with `confirmed_frozen_count=0`
replays the unexpanded prefix.
- Not tested: live Anthropic cache_read/cache_creation after a real
proxy restart. Redacted real-restart runtime evidence is still to come
from the author and will be added to this PR; nothing above is from a
live upstream.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: none (not rollout-managed)
- Stable/default behavior changed: expansion placement and tracker
snapshot
- Kill switch / disable path: `HEADROOM_NO_CCR_PROACTIVE_EXPANSION=1` /
`--no-ccr-proactive-expansion`
- Unsafe override required: no
- Qualification impact: none expected; the cached prefix bytes are
unchanged and only the uncached tail gains the expansion block
- Rollback path: revert the commits

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4300"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Demo


![demo](https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/headroom/ccr-transient-expansion/demo.gif)

<video
src="https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/headroom/ccr-transient-expansion/demo.mp4"
controls width="720"></video>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 22:46:56 -06:00
75cfbba91d fix(proxy): record outcome when a passthrough stream ends mid-character (#4429)
## Description

When a streaming passthrough response ends inside a multi-byte UTF-8
character, the `finally` block of `_handle_streaming_passthrough`
appends a `\n\n` terminator to the leftover bytes and the strict SSE
decoder raises `UnicodeDecodeError`. The request outcome is then never
recorded and the ASGI app logs a traceback. This PR drops the unusable
tail, keeps the usage already received, and records the outcome with the
shared stream-failure status (502, `_stream_outcome_status`) because a
stream cut mid-character did not complete.

Fixes #4427

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/openai.py`: in
`_handle_streaming_passthrough`'s `finally`, the final
`_parse_sse_usage_from_buffer` call is wrapped in
`contextlib.suppress(UnicodeDecodeError)`. A final
complete-but-unterminated event is still parsed as before; only an
undecodable tail is discarded. When that happens the outcome is built
with `status_code=_stream_outcome_status(upstream_response.status_code,
completed_normally)`, the same helper the other streaming paths use in
`streaming.py`, so `emit_request_outcome` books it as failed instead of
as a 200 success. The observed token fields are unchanged.
- `tests/test_proxy_handler_helpers.py`: new test
`test_vertex_stream_passthrough_records_outcome_when_stream_ends_mid_character`
asserts one outcome, `status_code == 502`, 13 input / 5 output tokens
kept, the upstream response closed exactly once, and that
`emit_request_outcome` calls `record_failed` once and `record_request`
never.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

With only the round-2 source change reverted (new assertions kept),
`tests/test_proxy_handler_helpers.py -k mid_character`:

```text
tests/test_proxy_handler_helpers.py:908: in test_vertex_stream_passthrough_records_outcome_when_stream_ends_mid_character
    assert outcomes[0].status_code == 502
E   AssertionError: assert 200 == 502
1 failed, 74 deselected
```

With the fix, plus `tests/test_route_advice.py
tests/test_5xx_accounting_all_providers.py
tests/test_outcome_records_5xx_as_failed.py`:

```text
============================= 152 passed in 6.00s ==============================
```

```text
$ ruff check headroom/proxy/handlers/openai.py tests/test_proxy_handler_helpers.py
All checks passed!
$ ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_handler_helpers.py
2 files already formatted
```

`mypy` was not run.

## Real Behavior Proof

- Environment: WSL2 Ubuntu on Windows 11, Python 3.12.15, branch based
on upstream/main `e995eb9f`; `headroom/_core.abi3.so` taken from the
official headroom-ai 0.41.0 wheel (untracked, not committed). Real proxy
on 127.0.0.1:18940, local stub upstream on 127.0.0.1:18941
(`VERTEX_TARGET_API_URL`); the stub log shows the proxied request, so no
provider API was contacted.
- Exact command / steps: start the stub (chunked `text/event-stream`:
one complete usage event `data:
{"type":"message_delta","usage":{"output_tokens":3}}`, then `data:
{"t":"\xe2\x82`, then a clean end), start `python -m
headroom.proxy.server --host 127.0.0.1 --port 18940 --no-cache
--no-rate-limit`, read `GET /stats`, `POST
/v1/projects/p/locations/l/publishers/mistral/models/m:streamRawPredict`
with `{"x":1}`, read `GET /stats` again. Same steps on the base and on
this branch.
- Observed result: base: proxy log shows `Exception in ASGI application`
ending in `UnicodeDecodeError: 'utf-8' codec can't decode bytes in
position 12-13: unexpected end of data`, and `requests.total
before/after: 0 0`. This branch: no traceback in the proxy log, and
`requests.total before/after: 0 1` (this was the first version of the
fix, before the status change). Re-run for the failure status on this
head (a76fd811): no traceback, `requests.total before/after: 0 0`, and
`/stats` shows `requests.failed: 1` (`failed_by_provider:
{vertex:mistral: 1}`); the stub log shows the request. The client
received the same 388 bytes in every run.
- Not tested: other ASGI servers and HTTP/2 upstreams; real
Vertex/Gemini upstreams; the `MAX_SSE_BUFFER_SIZE` tail-truncation path
(which can also start a buffer mid-character); a client disconnect (as
opposed to a truncated upstream) as the trigger.

The failure-status re-run reuses the same stub and client with one extra
line printing `/stats` `requests`. Details: scripts and logs are in
`/var/tmp/cb-pub/proof/fix-openai-passthrough-stream-release/` (stub.py,
client.py, run.sh, partial_base.proxy.log, partial_fix.proxy.log,
partial_v2.proxy.log, client_v2.py).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only the failure case: an undecodable
stream tail no longer raises
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: N/A, no user-facing behavior beyond the fix.
- A second suspected problem in the same function (no background release
of the upstream stream on client disconnect) did not reproduce against
the real proxy (780 client resets at several timings; the stub saw the
proxy close its upstream connection every time), so it is not changed
here.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4429"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:38:14 -07:00
Kenneth WongandKenneth Wong a9eb2a7b6f fix(proxy): keep Claude Haiku 5.5 mid-conversation system messages in place (#4153)
## Description

Claude Haiku 5.5 accepts `role: "system"` entries inside `messages[]`
(mid-conversation system messages, no beta header).
`relocate_system_messages_to_top_level` (`headroom/proxy/helpers.py`)
doesn't list the Haiku 5 family, so on every Haiku 5.5 request it moves
each valid mid-conversation section into the top-level `system` field.
That has two effects:

- The operator instruction moves from the point in the conversation
where it became true to the top of the prompt, which changes its
meaning.
- The top-level `system` that the cached prefix starts with changes from
conversation to conversation.

Seen on a local proxy over ~28 hours (2026-10-10 01:14 → 2026-10-11
05:27): all 599 `Relocated role=system` warnings were `claude-haiku-5-5`
requests, which is Claude Code's subagent traffic. The Opus 5.5 (9,360)
and Sonnet 5.5 (539) requests in the same window were never relocated.

This adds `"claude-haiku-5"` to the supported families. Haiku 4.5, which
doesn't support mid-conversation system messages, is still relocated.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`: add `"claude-haiku-5"` to the model
families `relocate_system_messages_to_top_level` treats as supporting
mid-conversation system messages. The substring matching works for every
model-ID form the list already handles, including `claude-haiku-5-5` and
Bedrock-style `…anthropic.claude-haiku-5-5…`. It doesn't match
`claude-haiku-4-5`.
- `tests/test_proxy_handler_helpers.py`: parametrize the "preserves a
valid mid-conversation section" test over `claude-opus-5` and
`claude-haiku-5-5`, and the "moves a valid shape for an unsupported
model" test over `claude-sonnet-4-6` and `claude-haiku-4-5`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HOME=$(mktemp -d) pytest -q tests/test_proxy_handler_helpers.py
74 passed                       # origin/main: 72 passed; +2 new parametrized cases

$ ruff check headroom/proxy/helpers.py tests/test_proxy_handler_helpers.py
All checks passed!
$ ruff format --check headroom/proxy/helpers.py tests/test_proxy_handler_helpers.py
2 files already formatted
$ mypy headroom/proxy/helpers.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS arm64, repo `.venv` (Python 3.10.18). The live
proxy traffic came from `headroom proxy` on 127.0.0.1:8787, forwarding
to `api.anthropic.com`.
- Exact command / steps: called `relocate_system_messages_to_top_level`
with a valid `user → system → assistant` request on `origin/main`
(976aa714) and on this branch, for `claude-haiku-5-5` and
`claude-haiku-4-5`. Separately, joined every `Relocated role=system`
warning in the proxy log to that request's `PERF model=` line by request
id.
- Observed result: on `origin/main`, `claude-haiku-5-5` gives
`changed=True roles=['user', 'assistant'] system=[{'type': 'text',
'text': 'base'}, {'type': 'text', 'text': 'The user added: update the
changelog too.'}]`. On this branch it gives `changed=False
roles=['user', 'system', 'assistant'] system='base'`. `claude-haiku-4-5`
is relocated on both. In the proxy log, 599 of 599 relocated requests
were `claude-haiku-5-5`, and Haiku 5.5's in-conversation turns had a
cache hit under 50% 7.1% of the time, against 0.3% for Opus 5.5 and 0.2%
for Sonnet 5.5. That's a correlation; the log can't key turns to
conversations. At Haiku 5.5's $0.10/MTok the cache cost is small. The
main issue is where the operator instruction ends up.
- Not tested: a live Haiku 5.5 request with the section left in place.
On the proxy I could test through, the released build relocates Haiku
requests before they leave. Haiku 5.5 support comes from Anthropic's API
reference, which lists it with no beta header. For the existing
`claude-sonnet-5` entry, a live first-party request with a trailing
`role: "system"` message on `claude-sonnet-5` returned HTTP 200 and
followed the instruction, so that entry is correct even though some docs
omit Sonnet 5. Bedrock and Vertex weren't tested.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A. This is the default wire-contract guard,
with no flag.
- Stable/default behavior changed: yes. Valid mid-conversation system
sections on Claude Haiku 5.x requests are now forwarded in place instead
of being moved to the top-level `system`. Invalid placements (leading,
or followed by a `user` turn) are still moved for every model.
- Kill switch / disable path: none needed for a model-list entry. If an
upstream rejected Haiku 5.5 mid-conversation system messages, removing
the entry restores the old behavior.
- Unsafe override required: no.
- Qualification impact: none expected. Requests that already relocate
keep the same output; only Haiku 5.x requests with valid sections
change.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- Documentation: N/A. No user-facing docs list the supported models for
this guard.
- Type checking was run on the changed module
(`headroom/proxy/helpers.py`), not the whole package.
- Sources disagree on Claude Sonnet 5 (some docs omit it). The live
check above supports keeping it, so this PR leaves it alone.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4153"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
2026-10-13 16:21:13 -07:00
lorenzozaneandJerrettDavis d9e29bb260 fix(proxy): keep leading Anthropic system payloads valid (#3671)
## Description

Leading Anthropic system-role sections containing image/document or
malformed text-shaped blocks could produce an invalid upstream request.
Normalize representable text into the top-level `system` field using
Anthropic TextBlockParam fields only, preserve cache/citation metadata,
and remove unrepresentable leading blocks with an explicit warning.
Mid-conversation non-text retention remains covered by the existing
contract.

Closes #3667

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update

## Changes Made

- Keep only `type`, `text`, `cache_control`, and `citations` when
converting leading text-shaped blocks. Image sources and tool
IDs/names/inputs cannot leak into top-level text blocks, including
blocks already labeled `text`.
- Normalize every contiguous leading system section, so a second
image-only section cannot become an invalid first message after the
first section is removed.
- Preserve valid text/cache/citation metadata without mutating the input
messages.
- Retain the PR's explicit warning/drop policy for leading content with
no text representation; mid-conversation non-text blocks retain their
existing behavior.

## Testing

Current head: `cc05220696735f4162a986e24e5e6ab47ab43e71`.

- Six new regressions fail before correction and pass afterward: five
text-block type/field variants and consecutive leading system sections.
- 182 helper, Anthropic cache-stability, system-prompt immutability, and
byte-faithful forwarding tests pass (one existing Starlette warning).
- Ruff lint/format, whitespace, and Linux-target mypy over 566 files
pass.
- Allowed text-block fields were checked against the installed Anthropic
SDK TextBlockParam contract.

## Real Behavior Proof

- Setup: Windows Python 3.12, a real uvicorn proxy and a separate real
loopback HTTP upstream enforcing Anthropic's top-level text-block field
contract. Optimization, caching, cost tracking, and subscription polling
disabled.
- Steps: run `pr3671-resume17-runtime.py`, send five real TCP Messages
requests, and inspect the upstream-received bodies and proxy responses.
- Observed: image-caption, tool-text, text-with-extra-fields,
consecutive leading sections, and valid-text control all return HTTP
200. The upstream receives text-only allowed fields and messages
beginning with a user turn; cache-control TTL survives. Consecutive
sections preserve both text instructions.
- Before correction, the same replay returns HTTP 400 for the first four
cases; the valid-text control remains HTTP 200. Evidence:
`pr3671-resume17-runtime-red.log` and `pr3671-resume17-runtime.log`;
tests: `pr3671-resume17-tests.log`.
- Not tested: live Anthropic account/billing traffic. Only network
delivery is redirected to the loopback upstream. The intentional removal
of unrepresentable leading system content remains a warning-producing
normalization, not a lossless image conversion.

## Runtime Rollout Safety

- Rollout-managed features: none; normalization uses the existing
Anthropic wire-contract guard.
- Default behavior changed: leading text-shaped blocks emit valid
Anthropic text fields, all contiguous leading sections normalize, and
unrepresentable leading system blocks are removed with warnings.
- Valid text/cache/citation metadata and mid-conversation retention
controls are covered. No new unsafe override or network behavior.
- Rollback: revert the normalization commits.

## Review Readiness

- [x] Self-review completed
- [x] Regression coverage and local checks pass
- [x] Current real proxy behavior supplied
- [x] Did not edit CHANGELOG.md

Hosted checks and review must qualify this new head before approval. No
merge performed.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-08 10:41:39 -05:00
7f608ea932 fix(proxy): take a streamed Anthropic turn's uncached input from usage (#3946)
## Description

`StreamingMixin._finalize_stream_response` derives a streamed Anthropic
turn's uncached input as `optimized_tokens - cache_read - cache_write`.
`optimized_tokens` is Headroom's own tokenizer count, so the derivation
turns the estimate's error into uncached input. Anthropic already
reports the uncached count as `usage.input_tokens` on `message_start`
(tokens after the last cache breakpoint), and the non-streaming path
records exactly that field (`anthropic.py`, `uncached_input_tokens =
int(usage.get("input_tokens", 0) or 0)`). Only OpenAI and Gemini take
the provider's count in the streaming finalizer today.

The error goes both ways:

- Estimate over the provider's count: phantom uncached tokens on every
turn. On long, screenshot-heavy Claude Code sessions (~1M context) the
local count ran 9-13% over Anthropic's. One turn Anthropic reported as
`input_tokens=2, cache_read=360949, cache_creation=840` was recorded
with 50,663 uncached tokens. Over one day, three such sessions logged
20.1M uncached tokens where Anthropic reported 3.3M.
- Estimate under the provider's count (most ordinary sessions): the
`max(..., 0)` clamp records 0 and real uncached input goes missing.

Uncached input feeds `new_input_tokens` (uncached + cache write), so the
new-input savings rate moved with the estimate's bias, not with
compression. On the day above it read 14.8% when the rate on Anthropic's
own numbers was 24.7%.

The finalizer now takes `stream_state["input_tokens"]` for `provider ==
"anthropic"` when message_start reported it. Without a usage frame (for
example an error before the stream started), it keeps the old
derivation. `tokens_saved`, `optimized_tokens` and forwarded bytes are
unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/streaming.py`: `_finalize_stream_response`
sets `uncached_input_tokens` from the provider's `input_tokens` for
Anthropic when it is an int, else falls back to the tokenizer
derivation.
- `tests/test_proxy_handler_helpers.py`:
`test_stream_finalizer_takes_anthropic_uncached_input_from_usage` covers
a local count over the provider's (the production turn above) and under
it (the old clamp to 0).
`test_stream_finalizer_derives_anthropic_uncached_input_without_usage`
pins the fallback.
- `headroom/proxy/handlers/streaming.py`: `_parse_sse_usage_from_buffer`
no longer defaults an absent Anthropic `message_start` `input_tokens` to
0. It used to, so a usage frame carrying only cache counts read as
"provider reported 0" and the finalizer booked 0 uncached instead of
falling back (Devin's finding). The key now stays absent,
`stream_state["input_tokens"]` stays `None`, and the fallback runs. This
parser feeds both `_finalize_stream_response` call sites; the native
event path (`_stream_response_bedrock`) and the buffered CCR path in
`anthropic.py` already kept absent as `None`.
- `tests/test_proxy_handler_helpers.py`:
`test_sse_parser_leaves_absent_anthropic_input_tokens_absent` covers the
parser half.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Before the fix (upstream/main 65e63c08), new tests only:
```text
FAILED tests/test_proxy_handler_helpers.py::test_stream_finalizer_takes_anthropic_uncached_input_from_usage[412452]
FAILED tests/test_proxy_handler_helpers.py::test_stream_finalizer_takes_anthropic_uncached_input_from_usage[300000]
================== 2 failed, 1 passed, 59 deselected in 1.86s ==================
```

After (6eec3c39):
```text
$ uv run --frozen --extra dev pytest tests/test_proxy_handler_helpers.py tests/test_request_outcome.py tests/test_proxy_streaming_request_logger.py tests/test_proxy_cache_ttl_metrics.py -q
======================== 129 passed, 1 warning in 4.12s ========================
$ uvx ruff check .
All checks passed!
```

The parser test, run with the parser hunk reverted:
```text
FAILED tests/test_proxy_handler_helpers.py::test_sse_parser_leaves_absent_anthropic_input_tokens_absent
======================= 1 failed, 62 deselected in 0.35s =======================
```

`mypy headroom` reports one error,
`integrations/langchain/chat_model.py:394` (truthy-function). It is the
same with this branch's changes stashed, so it is not from this PR.

## Real Behavior Proof

- Environment: macOS 27 (M2), headroom-ai 0.39.0 behind a Claude Code
proxy, with the finalizer hunk applied as a runtime patch since
2026-10-02 morning. Model claude-opus-5-5.
- Exact command / steps: read the live proxy's
`/transformations/feed?include_messages=0` and, for each Anthropic
record, compare the recorded `uncached_input_tokens` (now the provider's
`input_tokens`) with what the old derivation would have booked,
`max(input_tokens_optimized - cache_read - cache_write, 0)`.
- Observed result: all 72 Anthropic turns in the feed (19:03-19:20 UTC)
had a local count under the provider's (`input_tokens_optimized` vs
`input_tokens + cache_read + cache_write`; median -33.5%, p10 -76.6%,
p90 -19.5%). The old derivation books 0 uncached on all 72; Anthropic
reported 48,787. Example: `input_tokens_optimized=94,781` against
`uncached=2, cache_read=12,806, cache_write=117,960`. The over-count
case is the earlier production day in the description (three
screenshot-heavy sessions, 20.1M logged vs 3.3M reported).
- Not tested: the parser change on live traffic. Anthropic sends
`input_tokens` on every `message_start`, so the missing-key case only
comes up with a gateway that leaves it out; the unit test covers it.
Non-Anthropic providers are unchanged by construction (the branch is
gated on `provider == "anthropic"`).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, not rollout-managed.
- Stable/default behavior changed: accounting only. Streamed Anthropic
turns record the provider's uncached input instead of a
tokenizer-derived one, which moves `new_input_tokens` and the new-input
savings rate in either direction. `tokens_saved`, `optimized_tokens`,
list-priced savings and forwarded bytes are unchanged.
- Kill switch / disable path: none; revert.
- Unsafe override required: no.
- Qualification impact: none on compression or cache. Dashboards that
chart uncached input or the new-input rate will step on upgrade.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3946"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:57:46 -07:00
Kenneth WongandJD Davis 3948dbe409 fix(proxy): inject memory context past a trailing system message (#3948)
## Description

In `--mode token`, memory context never reaches the model under Claude
Code 2.1.x. Claude Code sends a trailing `role: "system"` message
(environment context that carries the cache breakpoint) after the user
turn:

```text
messages[0] user   text <system-reminder>…</system-reminder>   (no cache_control)
            user   text <system-reminder>…</system-reminder>   (no cache_control)
            user   text <the prompt>                            (no cache_control)
messages[1] system text # Environment …                         (cache_control)
```

`_append_context_to_latest_non_frozen_user_turn` only looked at
`messages[-1]`, saw a non-user role and returned unchanged. Memory
search found the context, but every request logged
`decision=no_eligible_user_turn` and nothing was injected. The same
helper serves the CCR proactive-expansion path, which had the same
problem.

This PR skips trailing system-role messages when locating the latest
user turn. The system message stays byte-identical and the frozen-prefix
check still applies to the user turn, so prompt-cache behaviour is
unchanged. Follow-up to #3947, which fixed memory *tools* on streaming
turns; this one fixes memory *context injection*.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py`:
`_append_context_to_latest_non_frozen_user_turn` walks back past
trailing `role: "system"` messages before the frozen-prefix and
user-role checks; docstring explains why.
- `tests/test_proxy_handler_helpers.py`:
`test_append_context_skips_trailing_system_message` covers injection
into the Claude Code 2.1.x shape (system message untouched), the frozen
prefix still blocking, never reaching past an assistant turn,
tool_result-only turns, and a system-only list.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HOME=$(mktemp -d) .venv/bin/python -m pytest -q tests/test_proxy_handler_helpers.py tests/test_proxy_anthropic_cache_stability.py tests/test_cache_breakpoint_diagnostics.py tests/test_ccr_context_tracker.py
origin/main: 156 passed, 1 skipped
this branch: 157 passed, 1 skipped

$ HOME=$(mktemp -d) .venv/bin/python -m pytest -q $(grep -rlE 'injected_live_zone_tail|no_eligible_user_turn|_append_context_to_latest|memory_injection|MemoryDecision' tests)
317 passed, 1 skipped

$ uvx ruff@0.16.8 check headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py
All checks passed!
$ uvx ruff@0.16.8 format --check headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS 27, real Claude Code 2.1.287 (`claude -p`, Opus
5.5) → Headroom proxy (`--memory --mode token`) → real Anthropic
upstream. Before = `origin/main` (540f4da1) in a worktree; after = this
branch in a worktree; same interpreter, same copy of a memory store
seeded by a session-1 run ("knowledge-base changes go on kb/ branches;
PR titles start with the entity id; squash-merge; ADR reviewer must not
be in decided_by").
- Exact command / steps: session 2 with the same
`x-headroom-project-id`: `claude -p "I've drafted a new ADR called
ADR-20260930-s3-lifecycle-rules (decided_by: kenneth). I'm ready to
push. What should the branch be called, how should I title the PR and
merge it, and who can review it?"`, then count `memory_injection`
decisions in the proxy log.
- Observed result: before, no injection and the answer applies none of
the saved conventions; after, memory is injected on the first request
and the answer applies them. The remaining `no_eligible_user_turn` after
the fix is the follow-up tool_result turn, which is expected. Excerpts:

  ```text
origin/main proxy log: 2 x decision=no_eligible_user_turn
bytes_injected=0
answer: "this repo has no branch or PR process … If you want to invent a
convention anyway,
adr/20260930-s3-lifecycle-rules with a squash merge would match …"
this branch proxy log: 1 x decision=injected_live_zone_tail
bytes_injected=1199
1 x decision=no_eligible_user_turn bytes_injected=0 (tool_result turn)
answer: "PR title must start with the entity id (from a team convention
saved in an earlier session)
… Merge with squash … A PR that adds or changes an ADR needs a reviewer
who isn't in its decided_by"
  ```
- Not tested: `mypy headroom` (not run locally); interactive Claude Code
sessions (the request shape was captured from `claude -p`); the OpenAI
and Gemini handlers, which have their own injection code and are not
changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This changes the existing
memory-context injection path for Anthropic requests.
- Minimum rollout channel: stable.
- Stable/default behavior changed: only for requests whose messages end
in one or more `role: "system"` messages; injection now targets the user
turn before them instead of being skipped. Default `--mode cache` still
skips injection by design, so default behaviour is unchanged.
- Kill switch / disable path: `--no-memory-context` (or
`HEADROOM_MEMORY_INJECTION_MODE=disabled`).
- Unsafe override required: no.
- Qualification impact: none expected; the cache hot zone (system
prompt, frozen prefix, trailing system message) is never mutated.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: no doc change needed; behaviour now matches what the
injector's docstring already described.
- Possibly related, not changed here:
`_strict_previous_turn_frozen_count` (cache mode) also treats a final
non-user message as "freeze everything", so with Claude Code's trailing
system message the current turn may be frozen in cache mode too. That
affects compression rather than memory and needs its own cache-stability
review.

https://claude.ai/code/session_01B6gHB8BWnYShVzrkgcBKWf

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3948"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-03 16:13:05 -05:00
b1b005a9a8 fix(memory): align project routing with wrap headers (#3603)
## Description

When `headroom wrap claude` runs from two directories named `api`, both
requests currently send `X-Headroom-Project: api`. Using that label as a
memory or CCR identity can make the two workspaces share history.

This keeps `X-Headroom-Project` as the human-readable savings label and
adds `X-Headroom-Cwd`, derived from the canonical launch directory, for
routing. The resolver now ignores the label when it is the only project
signal.

Closes #3597

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Keep `X-Headroom-Project` for savings attribution only.
- Add a percent-encoded `X-Headroom-Cwd` header containing the canonical
launch directory.
- Hash the decoded cwd for memory and CCR routing, while retaining the
basename as the display label.
- Preserve user-supplied project and cwd headers without duplicating or
overwriting them.
- Add regressions for same-basename directories, label-only fail-closed
behavior, and an explicit user-supplied label collision.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
216 passed, 1 warning in 4.80s
7 files already formatted
All checks passed!
git diff --check: clean
```

## Real Behavior Proof

- Environment: Ubuntu 24.04 VPS, Python 3.12.3, Rust extension built in
the test environment.
- Exact command / steps: Wrapped two temporary directories with the same
basename (`api`), using the same explicit `X-Headroom-Project` value,
then resolved both memory and CCR workspaces.
- Observed result: Both requests retained the same display label but
produced different cwd headers and different memory and CCR keys. A
request containing only the display label returned the fail-closed
result.
- Not tested: A live request to an external Claude provider.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: `headroom wrap claude` now sends a
second `X-Headroom-Cwd` routing header; `X-Headroom-Project` remains a
savings label.
- Kill switch / disable path: Remove the generated routing header or
disable project-scoped memory/CCR using the existing configuration.
- Unsafe override required: No.
- Qualification impact: Requests with only the savings label remain
unscoped and fail closed; requests with a cwd identity retain
project-scoped memory and CCR.
- Rollback path: Revert commit `101b332`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-29 15:33:17 -05:00
690fb251dc harden: fix security issue in server.py (#3479)
## Summary
Harden input handling in `headroom/proxy/server.py` (flagged by
multi_agent_ai).

## Vulnerability
| Field | Value |
|-------|-------|
| **ID** | V-001 |
| **Severity** | HIGH |
| **Scanner** | multi_agent_ai |
| **Rule** | `V-001` |
| **File** | `headroom/proxy/server.py:3803` |
| **Assessment** | Defensive hardening |

**Description**: The proxy server parses JSON request bodies using
Starlette's request.json() at four locations without visible enforcement
of the MAX_REQUEST_BODY_SIZE constant defined in helpers.py. While the
configuration defines size limits, the actual enforcement in the request
handling path requires verification. Without guaranteed size limits,
attackers can send oversized JSON payloads causing memory exhaustion.

## Threat Model Context

This server appears to be publicly accessible. This is a web service -
vulnerabilities in request handlers are directly exploitable by remote
attackers.

## Changes
- `headroom/proxy/server.py`

## Behavior Preservation
The change is scoped to 1 file on the vulnerable path.

## Security Invariant
> **Property**: The security boundary is maintained under adversarial
input

<details>
<summary>Regression test</summary>

```python
import pytest
import asyncio
from unittest.mock import AsyncMock, MagicMock
import sys
import os

# Ensure the module can be imported
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from headroom.proxy.server import MAX_REQUEST_BODY_SIZE


@pytest.mark.parametrize("payload_size", [
    MAX_REQUEST_BODY_SIZE - 1,  # valid: just under limit
    MAX_REQUEST_BODY_SIZE,       # boundary: exactly at limit
    MAX_REQUEST_BODY_SIZE + 1,   # exploit: exceeds limit
])
def test_request_body_size_enforced(payload_size):
    """Invariant: Request body size MUST NOT exceed MAX_REQUEST_BODY_SIZE."""
    # Verify the constant exists and is reasonable
    assert isinstance(MAX_REQUEST_BODY_SIZE, int)
    assert MAX_REQUEST_BODY_SIZE > 0
    
    # The security property: any payload at or below limit is acceptable,
    # anything above MUST be rejected. This test documents the boundary.
    is_acceptable = payload_size <= MAX_REQUEST_BODY_SIZE
    
    # For the exploit case, this MUST be False (would be rejected by proper enforcement)
    if payload_size > MAX_REQUEST_BODY_SIZE:
        assert not is_acceptable, f"Payload size {payload_size} exceeds limit {MAX_REQUEST_BODY_SIZE}"
    else:
        assert is_acceptable, f"Payload size {payload_size} should be within limit {MAX_REQUEST_BODY_SIZE}"
```

</details>

This test guards against regressions — it's useful independent of the
code change above.

---
*This patch removes an exploit primitive — a code pattern that, while
not independently exploitable today, could be chained with other
weaknesses by automated exploit-development tooling. Proactive removal
of such primitives raises the bar against increasingly capable automated
attack tools.*

---
*Automated security fix by [OrbisAI Security](https://orbisappsec.com)*

<!-- orbis-meta: rule=V-001 scanner=multi_agent_ai -->

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-09-24 18:22:19 +02:00
964671d883 fix(proxy): keep non text blocks in place when relocating system sections (#3553)
## Description

Closes #3552

when a payload carries a mid conversation system message holding non
text blocks, `relocate_system_messages_to_top_level` hoisted the whole
thing into the top level `system` parameter, image and document blocks
included
the top level `system` parameter only takes text, so anthropic
compatible upstreams that type `system` as a string reject the request,
the reporter hit `Input should be a valid string` with `loc body system
str` on a z.ai style endpoint
the fix keeps the hoist text only: text blocks and bare strings move up,
non text blocks stay in a system message at the original position,
nothing is dropped and the message order is untouched

### Steps to reproduce
1. run the new tests on untouched main: `python -m pytest -q
tests/test_proxy_handler_helpers.py::test_relocate_system_messages_keeps_image_blocks_out_of_top_level_system`
2. Expected (after this fix): text moves to top level `system`, the
image block stays in a mid conversation system message
3. Actual (raw output on untouched main 04cdf79a):

```text
FAILED tests/test_proxy_handler_helpers.py::test_relocate_system_messages_keeps_image_blocks_out_of_top_level_system
FAILED tests/test_proxy_handler_helpers.py::test_relocate_system_messages_hoists_only_text_from_mixed_sections
FAILED tests/test_proxy_handler_helpers.py::test_relocate_system_messages_image_only_sections_pass_through_unchanged
========================= 3 failed, 53 passed in 1.95s =========================
```

an image only system section was also needlessly rewritten into a top
level system list with an image block in it, which is exactly the shape
upstreams choke on

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/helpers.py`: the hoist now splits each relocated
system section, text blocks and bare strings move to the top level
`system` parameter, non text blocks stay behind in a system message at
the original spot, sections that hold nothing text shaped pass through
unchanged, existing behavior for text only and string content is byte
identical
- `tests/test_proxy_handler_helpers.py`: 3 regression tests, image block
kept out of top level system, mixed section hoists text only and retains
the image, image only section passes through unchanged

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
python -m pytest -q tests/test_proxy_handler_helpers.py
56 passed in 1.93s

without the fix (git restore --source main -- headroom/proxy/helpers.py):
3 failed, 53 passed
(the 3 new tests fail, every pre existing test still passes)

ruff check .
All checks passed!

ruff format --check .
1577 files already formatted

mypy headroom
Success: no issues found in 532 source files
```

## Real Behavior Proof

- Environment: linux, python 3.12.3, headroom main 04cdf79a plus the fix
(4f15cc02) in a venv, no live provider call involved
- Exact command / steps: the pytest commands in the test output block,
plus a restore dance, restoring main `helpers.py` turns the 3 new tests
red, restoring the fix turns them green, so the tests fail without the
change and pass with it
- Observed result: after the fix the top level `system` list only ever
contains text blocks and the image block survives in a mid conversation
system message, which is the wire shape upstreams typing `system` as a
string accept
- Not tested: a live call against a z.ai or similar endpoint, i verified
the wire shape at the helper level, the reporter's exact upstream config
is not available to me

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, mid conversation system sections
with non text blocks keep those blocks in place instead of moving them
into the top level `system` parameter, text only and string content
payloads are byte identical, that is the fix
- Kill switch / disable path: none needed, revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the one commit, nothing else to unwind

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: JD Davis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
2026-09-17 16:54:28 -07:00
AxelRay 91fd58d1df fix(memory): honor project root override for Anthropic CCR (#3606)
## Description

The documented `--memory-project-root` override is ignored by Anthropic
CCR workspace resolution, so clients without cwd metadata remain
unresolved. This change makes the configured project root available to
that resolver.

Closes #3596

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Read `memory_project_root_override` from the proxy configuration when
resolving an Anthropic CCR workspace.
- Add regression coverage for a configured project root and update
resolver tests for the instance method.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
UV_CACHE_DIR=$PWD/.uv-cache uv run --no-sync pytest tests/test_proxy_handler_helpers.py -q
54 passed in 6.70s

UV_CACHE_DIR=$PWD/.uv-cache uv run --no-sync pytest tests/test_memory_storage_router.py -q
28 passed in 0.30s

UV_CACHE_DIR=$PWD/.uv-cache uv run --no-sync ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py
2 files already formatted

UV_CACHE_DIR=$PWD/.uv-cache uv run --no-sync ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py
All checks passed!
```

## Real Behavior Proof

- Environment: Linux, CPython 3.11.16, checkout commit
8df4b38aa1.
- Exact command / steps: `UV_CACHE_DIR=$PWD/.uv-cache uv run --no-sync
pytest tests/test_proxy_handler_helpers.py -k
configured_project_root_override -q`; the test invokes the resolver with
an override and no project headers or cwd system-prompt line.
- Observed result: The configured override resolved to a stable
`project-c-` key prefix and the `project-c` workspace label. A separate
direct invocation returned `headroom-project-2a55b0efd88cd77d` and
`headroom-project`.
- Not tested: A live upstream provider request, macOS, Windows, and
hosted CI.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows project style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
2026-09-17 15:46:57 -07:00
JD Davis b3f443636d fix(proxy): align signed-thinking wire accounting (#3015)
## Description

Signed-thinking histories force byte-faithful passthrough because
re-serializing signed Anthropic blocks can invalidate their signatures.
Headroom correctly forwarded the original client bytes, but continued
reporting mutations, transforms, savings, response headers, and prefix
state from a different body that never reached the provider. Separately,
the final Anthropic guard hoisted every `role: system` message into the
top-level prompt, including valid mid-conversation system sections,
changing their semantics and destroying the cached prefix if that
mutation ever shipped.

This coupled fix makes downstream accounting use the actual wire body
whenever the signed-thinking lock discards edits, and narrows system
relocation to the current Anthropic model and placement contract.

Closes #2990
Closes #2991

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Detects signed thinking in the original request as well as the mutated
body, so a transform cannot remove the block and accidentally bypass the
byte lock.
- Keeps the original-body signature probe best-effort under malformed,
recursive, and `MemoryError` conditions.
- Carries discarded mutation reasons through the streaming forwarder and
emits the existing structured warning on HTTP streaming paths too.
- When signed passthrough wins, resets message savings, tool-schema
savings, attribution ledgers, transform labels, response headers, and
prefix tracking to the original client wire body.
- Adds bounded public diagnostic tags naming/counting discarded mutation
reasons without exposing body content.
- Preserves valid mid-conversation system sections on currently
supported Claude models and official Anthropic, Bedrock, and parsed
`*.googleapis.com` routes; hostname-boundary validation rejects
lookalike and userinfo URLs.
- Preserves consecutive system sections and enforces documented
predecessor/successor placement rules.
- Continues relocating initial, invalidly placed, unsupported-model, and
conservative third-party-gateway system messages to avoid upstream 400s.
- Includes current `main`, including #2996, #2997, #2971, #3009, #3012,
and the MCP dependency cap.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run pytest -q <wire/cache/savings/system focused suite>
379 passed

uv run pytest -q tests/test_proxy/test_anthropic_recount_and_reparse_safety.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_handler_helpers.py
99 passed

pytest tests scripts/tests --splits 4 --group N --tb=short -q
All four CI-shaped fresh-process groups passed locally after correcting the MemoryError regression; each completed in roughly 75-83 seconds.

Post-CodeQL correction: 91 focused tests passed; all four exact-head CI-shaped shards passed in roughly 82-95 seconds.

uv run ruff format --check .
1411 files already formatted
uv run ruff check .
All checks passed
uv run mypy headroom
Success: no issues found in 520 source files
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.13, branch rebased onto current
`main`.
- Exact command / steps: sent a signed-thinking request whose tool
schema is measurably compacted inside the handler, captured the exact
upstream bytes, wrapped the real outcome funnel, and inspected response
headers, aggregate metrics, attribution tags, transforms, and
prefix-tracker state. Exercised valid, consecutive, invalid, initial,
supported-model, and unsupported-model system placements.
- Observed result: upstream bytes remain byte-identical to the client;
discarded edits contribute zero tokens, zero tool savings, no transform
header, and no attribution while the prefix tracker stores the actual
wire messages. Valid mid-conversation system sections remain in place;
only out-of-contract sections relocate.
- Not tested: live paid Anthropic traffic with production credentials.
The placement/model contract was verified against the current official
documentation and wire behavior is covered with a byte-capturing
transport.

## Runtime Rollout Safety

- Rollout-managed feature(s): signed-thinking wire-truth accounting and
Anthropic mid-conversation system preservation.
- Minimum rollout channel: normal patch release after exact-head CI is
entirely green.
- Stable/default behavior changed: discarded mutations no longer inflate
savings; supported valid system sections are no longer hoisted into the
top-level prompt.
- Kill switch / disable path: no unsafe runtime override; human revert
restores the previous conservative relocation/accounting behavior.
- Unsafe override required: none.
- Qualification impact: all Python shards, byte-forwarding,
cache-prefix, outcome/savings, signed-thinking, Anthropic handler,
static, Docker, and security checks must remain green.
- Rollback path: fix forward through a human-reviewed corrective PR; no
persisted data or configuration migration is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation — inline
wire-contract documentation; no separate guide is required
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; proxy wire behavior and accounting only.

## Additional Notes

Human review only. No merge or auto-merge is configured. Current
provider contract reference:
https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages
2026-08-16 20:44:38 -07:00
f9807fd69e feat(proxy): let extensions report cost savings and their own latency (#3051)
## What

Two changes that let a proxy extension report **what it saved** and
**what it cost**, so both show up under `/stats`, the dashboard, and
Prometheus.

`record_scope_savings` already existed and already accepted `usd` — the
one channel in the proxy that can express savings *without* tokens. Two
things stopped it working end to end.

### 1. Savings were silently dropped on Gemini traffic (bug)

`bind_scope` shares one attribution ledger between ASGI middleware and
the request handler. Anthropic and OpenAI call it; **Gemini never did**,
so anything an extension recorded into the request scope was discarded
for Gemini traffic only — silently, because an empty ledger and an
unbound one are indistinguishable at the outcome funnel. Now bound at
all four Gemini tag sites.

### 2. An extension's own latency was invisible (gap)

`overhead_ms` is measured *inside* the handler, and an ASGI extension
**wraps** that handler — so every millisecond it spends reaches the
client while every timing surface stays flat. An extension that halves
the bill and adds 200 ms per request is a trade the operator has to see
both halves of, and only one half was reaching the dashboard.

`record_scope_timing(scope, stage, ms)` is the symmetric counterpart to
`record_scope_savings`, carried on the same bound ledger and merged into
`RequestOutcome.pipeline_timing` at the outcome funnel — one place, so
every provider picks it up at once.

## API surface

```python
from headroom.proxy.savings_attribution import record_scope_savings, record_scope_timing

record_scope_savings(scope, "my_extension", tokens=0, usd=0.004)   # money without tokens
record_scope_timing(scope, "my_extension", elapsed_ms)
```

Both take the ASGI `scope`, because middleware has no other way in.
Documented in `extensions.py` — the module extension authors actually
read, and the stability contract for this interface.

- Savings → `/stats` `savings.by_source`, dashboard card,
`headroom_savings_attributed_usd_total{source=...}`
- Timing → `/stats` `pipeline_timing`, dashboard Performance panel,
`headroom_transform_timing_ms_*`

**Attribution only.** These rows explain the headline total; they are
never added to it.

## Changes to existing behavior

- `public_tags` now strips `_headroom_stage_timing` as well as
`_headroom_savings_attribution`. Both ride on `tags` because that is the
one dict reaching the outcome funnel from every handler, and a list and
a dict must not land in a string-keyed label store.
- `pipeline_timing` passed to `metrics.record_request` is merged rather
than passed through **only when an extension contributed timings**; with
no extension the handler's own dict is passed through unchanged
(asserted by identity in the tests).
- Stage names are extension-supplied, so they are capped at 16 and
namespaced `ext:` — `deep_copy` reported by a plugin must never
accumulate into the same series as `deep_copy` measured by the pipeline.
A handler's own timing wins a collision (unreachable while the prefix
stands; the safe way round if it ever goes).

## Failure modes

Both calls are bounded (32 sources, 16 stages), never raise, and never
change a response — telemetry from a plugin must not be able to break
the request it is describing. Non-positive and non-numeric durations are
ignored: a zero is a clock artifact, not an observation, and averaging
it in would drag the mean down exactly where the stage is cheapest to
skip. `timings_from_tags` tolerates junk on the tag.

## Test-double fix

Three Gemini test fakes (`FakeRequest`, `_FakeRequest`,
`_VertexGeminiImageRequest`) had no `.scope`, which every real Starlette
`Request` has. They now do. This is a double that had drifted from the
type it stands in for; the alternative was weakening the handler to
tolerate a request shape that cannot occur in production.

---

## Real behavior proof

**Setup:** macOS 15.4 (darwin 25.4.0), Python 3.12.13, this branch at
`c814b950`, real `create_app` proxy with `respx`-mocked Anthropic
upstream, a demo ASGI extension added via `app.add_middleware`.

**The extension** — written as a third party would, reporting `tokens=0`
because it re-routed `claude-opus-5` → `claude-haiku-4-5`: same tokens,
cheaper model. That is precisely the case no existing Headroom savings
channel can express, since all of them compute `saved = before - after`.

```python
class DemoRouter:
    def __init__(self, app): self.app = app
    async def __call__(self, scope, receive, send):
        if scope.get("type") != "http":
            return await self.app(scope, receive, send)
        started = time.perf_counter()
        record_scope_savings(scope, "routemegood", tokens=0, usd=0.173)
        record_scope_timing(scope, "routemegood", (time.perf_counter() - started) * 1000)
        await self.app(scope, receive, send)
```

**Ran:** three POSTs to `/v1/messages`, then `GET /stats` and `GET
/metrics`.

**Observed:**

```
upstream call -> 200
upstream call -> 200
upstream call -> 200

=== /stats  savings.by_source  (what the dashboard renders) ===
[
  {
    "source": "routemegood",
    "realized": true,
    "events": 3,
    "tokens": 0,
    "usd": 0.519
  }
]

=== /stats  pipeline_timing  (dashboard Performance panel) ===
{
  "ext:routemegood": {
    "average_ms": 0.01,
    "max_ms": 0.02,
    "count": 3
  }
}

=== /metrics ===
# HELP headroom_savings_attributed_tokens_total Tokens attributed to a savings source
# TYPE headroom_savings_attributed_tokens_total counter
headroom_savings_attributed_tokens_total{realized="true",source="routemegood"} 0
# HELP headroom_savings_attributed_usd_total Cost savings attributed to a source; may be negative
# TYPE headroom_savings_attributed_usd_total gauge
headroom_savings_attributed_usd_total{realized="true",source="routemegood"} 0.519
headroom_transform_timing_ms_sum{transform="ext:routemegood"} 0.03
```

`$0.519 = 3 × $0.173` — three requests, correctly accumulated, with
`tokens: 0` throughout.

**Also have (not a substitute for the above):** 22 new unit tests in
`tests/test_extension_attribution.py`, including four that drive the
real `_record_request_outcome` funnel via the same descriptor-binding
harness `test_request_outcome.py` uses.

Full suite on this branch: **10,989 passed, 578 skipped**. Three
failures —
`test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter`
(full-suite ordering; passes in isolation),
`test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline`,
and `test_release_workflows.py::test_no_native_tls_in_wheel_build_tree`
(needs `cargo`) — **reproduce identically on clean `main`** (`2f4d001c`,
10,967 passed, same 3 failed). Verified by stashing this branch and
re-running the full suite on main in the same tree.

**What I did not test:** a live provider (upstream is `respx`-mocked);
the Gemini `bind_scope` fix against real Google traffic (covered by the
existing 114 Gemini tests, which all pass); the dashboard rendered in a
browser — I verified the JSON shape its templates bind to
(`stats.savings?.by_source`, `stats.pipeline_timing`) rather than the
pixels.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 10:25:47 -07:00
9fde127534 fix(proxy): relocate stray system-role messages to the top-level system param (#765) (#1357)
## Description

On requests large enough to trigger compression, the proxy emitted an
upstream Anthropic request whose `messages[0]` had `role: "system"`.
Anthropic's Messages API rejects any `system` role inside `messages[]`:

```
400 invalid_request_error: "messages.0: use the top-level 'system' parameter for the initial system prompt"
```

The original request correctly carries its system prompt in the
top-level `system` parameter; a compression/transform/pipeline step
relocates the harness system block into `messages[0]`, so the request
fails outright (intermittent only because it requires a context large
enough to compress).

This adds a wire-contract guard in the Anthropic forwarder: as the
**last** step before sending upstream (after every transform, memory
injection, tool sort, and pipeline extension, covering both the Bedrock
and direct paths), any stray `role="system"` message is relocated out of
`messages[]` and merged back into the top-level `system` parameter.
Content order is preserved (existing system first, relocated content
after) and block-level `cache_control` survives. The guard is a no-op on
the common path (no system-role entry → inputs pass through unchanged).

Closes #765

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`: new pure helper
`relocate_system_messages_to_top_level(messages, system) ->
(clean_messages, new_system, changed)` plus `_system_message_to_blocks`.
Handles `system` being `None`/`str`/`list`, never drops content,
preserves order and content blocks.
- `headroom/proxy/handlers/anthropic.py`: invoke the guard just before
the byte-faithful forward block; on relocation, update
`body["messages"]`/`body["system"]`, mark the body mutated
(`system_role_relocated`) so the byte-faithful forwarder re-serializes,
and log a warning.
- `tests/test_proxy_handler_helpers.py`: 3 unit tests (relocate stray
system into top-level, append-to-existing-system order, no-op without a
system entry).
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py -q
29 passed in 4.95s

# Regression on the forward path (byte-faithful forwarding, system-prompt immutability, cache stability):
$ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_system_prompt_immutable.py tests/test_proxy_anthropic_cache_stability.py -q
90 passed, 15 warnings in 29.72s

$ uv run ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py
All checks passed!

$ uv run ruff format --check ...   # 3 files already formatted
$ uv run mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py
Success: no issues found in 2 source files
```

## Test verification (RED → GREEN)

The new tests exercise the guard directly and import the new helper at
module top, so reverting the production fix makes them fail at
collection.

**RED — production fix reverted (helper removed):**
```text
ImportError while importing test module 'tests/test_proxy_handler_helpers.py'.
E   ImportError: cannot import name 'relocate_system_messages_to_top_level' from 'headroom.proxy.helpers'
=========================== 1 error in 0.41s ===============================
```

**GREEN — production fix applied:**
```text
tests/test_proxy_handler_helpers.py ...                                  [100%]
======================= 3 passed, 26 deselected in 1.50s =======================
```

## Real Behavior Proof

- Environment: Python 3.13, `uv run` in this repo, branch
`fix/issue-765`.
- Exact command / steps: ran the guard on a body in the exact #765
failure shape — `system: None` and a `role="system"` harness block at
`messages[0]`:
- Observed result:
  ```text
  BEFORE: messages[0].role = system (Anthropic 400 trigger)
  changed       = True
  AFTER roles   = ['user', 'assistant']
system param = [{"type": "text", "text": "You are Claude Code.
<system-reminder>...</system-reminder>"}]
OK: no role=system in messages[]; system content preserved in top-level
param
  ```
The illegal `role="system"` entry is removed from `messages[]` and its
content lands in the top-level `system` parameter — exactly the body
Anthropic accepts.
- Not tested: a full live 250k+-token Claude Code session against the
real Anthropic API (needs a large live context + API key); the fix is
validated at the request-shaping boundary the 400 is raised on.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The guard intentionally fires at the forwarder boundary rather than in
any single transform: the issue's captures show the relocation can
originate from the compression path, and pipeline extensions / hooks can
also mutate `messages` late. Enforcing Anthropic's wire contract once,
at the point the body is serialized upstream, fixes the 400 regardless
of which step introduced the stray entry and matches the architecture
invariant "never produce a `system`-role entry within `messages[]`".

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-08-13 11:52:09 -05:00
Abhay SinghandJerrettDavis f840d5f2fe fix(memory): make explicit-project and user store keys collision-resistant (#2231)
## Description

Two of the memory storage router's key-derivation paths can pool
distinct identities into one store.

`ProjectResolver._identity_from_cwd` builds a collision-resistant key by
appending a `sha256` digest to the sanitized basename:

```python
safe_basename = cls._sanitize_basename(basename) or "project"
digest = hashlib.sha256(normalised.encode("utf-8")).hexdigest()[:16]
key = f"{safe_basename}-{digest}"
```

But the two non-cwd paths use the bare sanitized basename as the key:

```python
# Tier 1 — explicit x-headroom-project-id
safe = self._sanitize_basename(explicit)
if safe:
    return safe, explicit           # <-- no digest

# USER mode
user_safe = ProjectResolver._sanitize_basename(ctx.base_user_id) or "default"
db_path = self._config.root_dir / "users" / user_safe / "memory.db"   # <-- no digest
```

`_sanitize_basename` maps every disallowed character to a single dash,
so distinct inputs collapse to the same basename:

- `acme/api` and `acme api` (and `acme@api`) all → `acme-api`
- user ids `alice/qa` and `alice qa` → `alice-qa`

Both the project key (`root/projects/<key>/memory.db`) and the USER key
(`root/users/<key>/memory.db`) are derived directly from that basename,
so two distinct project ids — or, in USER mode, two distinct **users** —
resolve to the same `memory.db` and share each other's memories. USER
mode exists specifically to isolate users, so this is a cross-user
data-isolation leak; the explicit-project-id path is the same leak
across projects. Both are client-controlled (`x-headroom-project-id` /
`x-headroom-user-id` headers), so the collision is easy to hit and could
even be provoked deliberately.

## Fix

Append the same digest of the raw id to both keys, exactly as
`_identity_from_cwd` does, keeping the sanitized basename as a
human-readable prefix:

```python
digest = hashlib.sha256(explicit.encode("utf-8")).hexdigest()[:16]
return f"{safe}-{digest}", explicit
```

```python
digest = hashlib.sha256(ctx.base_user_id.encode("utf-8")).hexdigest()[:16]
user_key = f"{user_safe}-{digest}"
db_path = self._config.root_dir / "users" / user_key / "memory.db"
```

Distinct ids now always land on distinct stores; the same id remains
stable across calls.

**Migration note:** this changes the on-disk key format for the
explicit-project and USER stores (`<basename>` → `<basename>-<digest>`).
Memories written under the old bare-basename paths are not migrated; the
router will start a fresh store at the new path. GLOBAL and cwd-derived
PROJECT stores (which already carried the digest) are unaffected.
Flagging this explicitly so you can decide whether a migration shim is
wanted before merge.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/memory/storage_router.py`: append a `sha256` digest to the
explicit-project-id key (Tier 1) and the USER-mode key, matching
`_identity_from_cwd`.
- `tests/test_memory_storage_router.py`: update the Tier-1 key assertion
to the prefix+digest form; add collision regression tests for the
explicit-project and USER paths.
- `CHANGELOG.md`: Bug Fixes entry (including the migration note).

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/memory/storage_router.py tests/test_memory_storage_router.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/memory/storage_router.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the key derivation with a dependency-free script mirroring
`_sanitize_basename` + the digest, and left the full pytest to CI.
- Exact command / steps: derived keys for `alice/qa` and `alice qa`
under the OLD bare-basename scheme and the NEW digest scheme.
- Observed result: OLD → both `alice-qa` (identical → shared store); NEW
→ `alice-qa-7e02fc2dfbc447b4` vs `alice-qa-4c9241514a374ba3` (distinct),
stable per input, with the `alice-qa-` prefix retained.
- Not tested: a live proxy with two colliding tenants; full local
`pytest` deferred to CI (OOM).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The changed/added tests
use the existing `tests/test_memory_storage_router.py` harness so they
run under the normal CI pytest job; behaviour is additionally verified
by the standalone proof above. I updated
`test_resolver_tier1_explicit_project_id_wins` to assert the new
prefix+digest key. Happy to add a migration shim (read the old path if
the new one is empty) if you'd prefer that over the fresh-store
behavior.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-08-11 23:39:15 -05:00
Abhay SinghandJD Davis 22b707fd31 fix(proxy/cost): count Gemini thinking tokens in output usage (#2639)
## Description

The Gemini handlers take the response's output-token count straight from
`candidatesTokenCount`:

```python
output_tokens = _usage_int(usage.get("candidatesTokenCount"))
```

For Gemini 2.5 thinking models that undercounts. Gemini reports
`candidatesTokenCount` **sometimes inclusive** of the reasoning tokens
(`thoughtsTokenCount`) and **sometimes exclusive** of them. When it is
exclusive, the thinking tokens are a separate bucket that is still
billed at the output rate, so dropping them makes `output_tokens` (and
therefore the output cost that flows through `record_tokens` ->
`estimate_cost`) too low. The gap grows with reasoning effort.

litellm handles exactly this: it adds `thoughtsTokenCount` to completion
tokens unless `promptTokenCount + candidatesTokenCount ==
totalTokenCount` (its `is_candidate_token_count_inclusive` check). The
Headroom handlers had no equivalent.

## Fix

Add `gemini_output_tokens(usage_meta)` in
`headroom/proxy/token_counting.py`:

- No `thoughtsTokenCount` (the common non-2.5 case): return
`candidatesTokenCount` unchanged.
- `promptTokenCount + candidatesTokenCount == totalTokenCount`:
candidates already include thoughts, return `candidatesTokenCount`.
- Otherwise: return `candidatesTokenCount + thoughtsTokenCount`.

This mirrors litellm's rule and is robust to missing or null fields.
Wire it into the native Gemini handler (both the generate and count
paths), the streaming usage extractors, and the OpenAI-compatible
passthrough usage normalizer, so every Gemini usage path counts output
the same way.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- `headroom/proxy/token_counting.py`: add `gemini_output_tokens()`.
- `headroom/proxy/handlers/gemini.py`: use it for `output_tokens` on
both response paths.
- `headroom/proxy/handlers/streaming.py`: use it in the two Gemini
streaming usage extractors.
- `headroom/proxy/handlers/openai.py`: use it in
`_passthrough_usage_from_json` (Gemini-shaped usage).
- `tests/test_proxy_handler_helpers.py`: unit test for
`gemini_output_tokens` (inclusive / exclusive / no-thinking / empty) and
a `_passthrough_usage_from_json` test that thinking tokens land in
`output_tokens`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` / `ruff format --check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for the fix
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_proxy_handler_helpers.py -k "gemini_output_tokens or thinking or vertex_usage_metadata" -q
3 passed

$ python -m pytest tests/test_proxy_gemini_native_integration.py tests/test_proxy/test_gemini_savings_profile.py tests/test_proxy_handler_helpers.py -q
38 passed, 18 skipped

# with the wiring reverted, the passthrough test fails (output_tokens is 200, not 700):
$ git stash push headroom/proxy/handlers/openai.py && \
    python -m pytest tests/test_proxy_handler_helpers.py -k passthrough_usage_counts_gemini_thinking -q
1 failed

$ uvx ruff@0.15.17 check headroom/proxy/token_counting.py headroom/proxy/handlers/gemini.py headroom/proxy/handlers/streaming.py headroom/proxy/handlers/openai.py tests/test_proxy_handler_helpers.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/token_counting.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: called `gemini_output_tokens` on an exclusive
usage (`prompt=1000, candidates=200, thoughts=500, total=1700`), an
inclusive usage (`candidates=700, total=1700`), a no-thinking usage, and
`{}`; drove `_passthrough_usage_from_json` with a thinking usage; then
reverted the handler wiring and re-ran the passthrough test.
- Observed result: exclusive returns 700 (200 visible plus 500
thinking), inclusive returns 700, no-thinking returns the candidates
count, empty returns 0; `_passthrough_usage_from_json` reports
`output_tokens=700`. With the wiring reverted it reports 200 (the
undercount). Verified against litellm's documented rule.
- Not tested: a live Gemini 2.5 request end to end (the accounting is
verified at the usage-extraction boundary against litellm's reference
logic).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-07-29 09:14:03 -07:00
Abhay Singh 2a63ec70b6 fix(image): reuse image models instead of rebuilding them per request (#2513) (#2536)
## Description

Fixes #2513. Image compression rebuilt its heavyweight models on every
request:

- `_compress_messages_worker` (`proxy/image_isolation.py`) created a new
`ImageCompressor()` per call, and
- `ImageCompressor.compress` (`image/compressor.py`) created a new
`OnnxTechniqueRouter(use_siglip=...)` per image.

Each `OnnxTechniqueRouter` loads native `ort.InferenceSession` models,
and ONNX Runtime holds C++ memory that Python's GC does not eagerly
reclaim. The image pool is a **persistent** single-worker
`ProcessPoolExecutor`, so those sessions accumulated in the worker and
RSS grew ~70 KB/request, reaching ~1.1 GB after ~15k image requests in a
day (the log shows one `[RapidOCR] Using engine_name: onnxruntime` line
per request, confirming reloads).

## Fix

Load the models once and reuse them:

- `ImageCompressor` caches the ONNX router on `self._onnx_router` (built
lazily via `_get_onnx_router`) instead of building one per `compress()`
call.
- The isolation worker keeps a per-process `ImageCompressor` singleton
(`_get_worker_compressor`) and reuses it across calls.
- `_get_image_compressor()` (main process, used for the `has_images()`
gate) returns a shared instance too.
- Shared instances are marked `_is_singleton`, and `close()` is a no-op
on them, so a caller's per-request `close()` no longer unloads the
models the next request reuses. A non-singleton `close()` still releases
the torch router and drops the cached ONNX router.

RSS is now flat after the initial model load; behavior is otherwise
unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/image/compressor.py`: add `_onnx_router` cache +
`_get_onnx_router`, use it in `compress()`, add the `_is_singleton`
flag, and make `close()` a no-op on a singleton (drop the cached ONNX
router on a real close).
- `headroom/proxy/image_isolation.py`: reuse a per-worker
`ImageCompressor` singleton in `_compress_messages_worker` instead of
building/closing one per call.
- `headroom/proxy/helpers.py`: `_get_image_compressor()` returns a
shared singleton instance.
- `tests/test_image_compressor_singleton_reuse.py` (new): the ONNX
router is built once and cached, singleton `close()` is a no-op while
non-singleton `close()` releases, and both `_get_image_compressor` and
the worker helper return a shared singleton.
- `tests/test_proxy_handler_helpers.py`: updated the two existing
`_get_image_compressor` tests that pinned the old fresh-per-call
behavior to assert the singleton reuse instead (and reset the new module
global so they stay isolated).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_image_compressor_singleton_reuse.py -q
5 passed

# with the fix reverted, all five fail (router rebuilt per call, close()
# unloads the shared models, helpers return fresh instances)

$ uvx ruff@0.15.17 check headroom/image/compressor.py headroom/proxy/image_isolation.py headroom/proxy/helpers.py tests/test_image_compressor_singleton_reuse.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/image/compressor.py headroom/proxy/image_isolation.py headroom/proxy/helpers.py
Success: no issues found in 3 source files
```

The pre-existing async tests in
`tests/test_image_compression_isolation.py` (4 `@pytest.mark.asyncio`
cases) fail identically on clean `main` in this environment because
pytest-asyncio is not configured here; they are unrelated to this change
and pass in CI.

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: with `OnnxTechniqueRouter` construction mocked,
called `ImageCompressor._get_onnx_router()` twice and asserted a single
construction; exercised `close()` on singleton vs non-singleton
instances; and called `_get_image_compressor()` /
`_get_worker_compressor()` twice each. Then reverted the three source
files and re-ran.
- Observed result: with the fix the ONNX router is constructed once and
reused, singleton `close()` leaves `_router`/`_onnx_router` intact (no
`release_models`), non-singleton `close()` releases and nulls them, and
both helper accessors return the same `_is_singleton` instance; with the
fix reverted every one of these fails (fresh construction /
unconditional release / new instances). Ran against the actual modules.
- Not tested: a live multi-hour image workload measuring RSS (the leak
is inferred from the removed per-request model construction; the
ONNX/torch model load itself is mocked here).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
2026-07-26 07:33:47 -07:00
09d1ef45be fix(proxy): compress Hermes scoped coding-agent passthrough (#1815)
## Description

Compress Hermes Studio scoped coding-agent passthrough requests in the
generic OpenAI passthrough handler. Hermes can route scoped Claude Code
and Codex traffic through Headroom while preserving its own proxy paths;
this PR keeps Hermes responsible for scoped proxy
authentication/provider adaptation while still applying Headroom
compression to supported chat payloads before forwarding.

The compression remains narrow-scoped:
- Only chat messages with `user` or `assistant` roles are compressed.
- Tool, function, reasoning, and system items are preserved byte-stable.
- Non-dict items in the Responses `input` array are preserved and
spliced back.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Detect `/api/codex-proxy/.../v1/responses` paths and compress
supported Responses `input` chat items before forwarding.
- Detect `/api/claude-code-proxy/.../v1/messages` paths and compress
supported Anthropic `messages` payloads before forwarding.
- Preserve bypass, malformed payload, missing-model, tool/function,
reasoning/system, and non-dict passthrough behavior.
- Add regression coverage in
`tests/test_hermes_passthrough_compression.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_hermes_passthrough_compression.py -v
test_codex_proxy_preserves_tool_and_function_items PASSED
test_codex_proxy_preserves_nondict_items PASSED
test_codex_proxy_bypass_header_skips_compression PASSED
test_codex_proxy_malformed_input_preserved PASSED
test_codex_proxy_compression_applies_to_chat_messages PASSED
test_claude_proxy_preserves_tool_use_items PASSED
test_claude_proxy_bypass_header_skips_compression PASSED
test_claude_proxy_no_model_forwarded_unchanged PASSED
test_claude_proxy_compression_applies_to_chat_messages PASSED
test_non_hermes_routes_not_affected PASSED
```

## Real Behavior Proof

- Environment: Author-reported local test environment for
`headroom/proxy/handlers/openai.py` and
`tests/test_hermes_passthrough_compression.py`.
- Exact command / steps: `python -m pytest
tests/test_hermes_passthrough_compression.py -v`.
- Observed result: The 10 Hermes passthrough regression tests passed,
covering Codex and Claude scoped proxy routes plus preservation/bypass
cases.
- Not tested: End-to-end Hermes Studio traffic against a live upstream
service is not covered by this PR body evidence.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A.

## Additional Notes

Generated with Claude Code. The unchecked checklist items are not
required for this narrow proxy-handler test change.

---------

Co-authored-by: x1051445024 <你的GitHub注册邮箱>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-13 17:22:46 -05:00
GUOHAO LIUandlennney 9db8a6bbf6 fix(proxy): handle ClientDisconnect in passthrough body reads (#2033)
## Description

Catch `starlette.requests.ClientDisconnect` when reading request bodies
in passthrough/forwarding handlers. Closes #2019

Without this, a client that disconnects mid-request causes an unhandled
`ClientDisconnect` to propagate through the entire middleware stack,
crashing the ASGI TaskGroup and contributing to proxy instability over
long sessions (memory growth, freeze, unresponsive to SIGTERM).

**Adversarial review uncovered 3 additional unprotected sites** in
`proxy_routes.py` — same pattern (body read before try/except). Now
fixed.

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)

## Changes Made

**Proxy handlers** (6 sites, first commit):
- openai `handle_passthrough`: wrap `await request.body()` in try/except
ClientDisconnect (main crash site)
- openai `_handle_streaming_passthrough`: same protection
- anthropic batch passthrough: same protection
- batch `_google_batch_passthrough`: same protection
- batch `handle_google_batch_passthrough`: same protection
- bedrock fallback-forward path: early-return on ClientDisconnect
instead of attempting verbatim forward

**Proxy routes** (3 sites, second commit — found by adversarial design
scan):
- `_handle_chatgpt_model_metadata` (proxy_routes.py:398)
- `_handle_chatgpt_codex_images` (proxy_routes.py:438)
- `openai_responses_sub` nested handler (proxy_routes.py:597)

All nine sites return HTTP 204 on disconnect to allow the request to
terminate cleanly.

## Testing

- [x] **Existing tests**: 34/34 pass in `test_proxy_handler_helpers.py`
- [x] **Unit tests**: 2 new tests — passthrough + streaming passthrough
disconnect
- [x] **Adversarial concurrency**: 50 threads × 10 iterations = 500
concurrent disconnect requests — zero crashes, all return 204
- [x] **Adversarial edge cases**: minimal request state, regression
check (normal request path unaffected)
- [x] **PBT (Hypothesis)**: 250 random method/path combinations, 3
properties verified:
  - All disconnect requests return 204
  - ClientDisconnect never leaks out of handler
  - Response is always valid HTTP 2xx

```text
# Unit tests
tests/test_proxy_handler_helpers.py::test_handle_passthrough_client_disconnect PASSED
tests/test_proxy_handler_helpers.py::test_handle_streaming_passthrough_client_disconnect PASSED

# PBT (3 properties × 100-250 examples each)
/tmp/pbt_client_disconnect.py::test_disconnect_always_returns_204 PASSED
/tmp/pbt_client_disconnect.py::test_disconnect_does_not_crash_asgi PASSED
/tmp/pbt_client_disconnect.py::test_response_is_valid_http PASSED

# Adversarial
/tmp/adversarial_client_disconnect.py → 500 concurrent requests: 0 errors, all 204
```

- [x] `ruff check` and `ruff format --check` pass on all changed files

## Real Behavior Proof

- Environment: Linux, Python 3.12, headroom main @ a617455
- Exact command / steps: 
  - `uv run pytest tests/test_proxy_handler_helpers.py -v` — 34 passed
- `uv run python /tmp/adversarial_client_disconnect.py` — 500
concurrent, 0 errors
- `uv run python /tmp/pbt_client_disconnect.py` — 250 random inputs, 3/3
properties hold
- `uv run ruff check . && uv run ruff format --check .` — All checks
passed
- Observed result: ClientDisconnect caught gracefully at all 9 sites,
204 returned, no ExceptionGroup crash, no data corruption
- Not tested: Full E2E with real client disconnect (requires integration
test infrastructure). Manual confirmation from issue reporter would
validate the real-world fix.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: lennney <lennney@users.noreply.github.com>
2026-07-11 10:22:05 -05:00
JD Davis 55efb1c77d fix(proxy): keep OpenAI tool observations mutable in cache mode (#1884)
## Description

Diagnoses and fixes the low-savings OpenAI-compatible cache-mode path
reported in #1696.

OpenAI-compatible tool-calling clients can end a turn with `role:
"tool"` (or legacy `role: "function"`) rather than `role: "user"`. The
OpenAI chat handler's cache-mode freeze boundary treated those tails as
non-mutable, and because `HeadroomProxy` resolves
`_strict_previous_turn_frozen_count` from the Anthropic mixin first, the
OpenAI-specific helper was not used in production. That froze the entire
conversation before `ContentRouter` ran, leaving no live tool
observation to compress and producing near-pass-through savings on long
coding sessions.

This PR keeps final OpenAI tool/function observations mutable in cache
mode, explicitly calls the OpenAI helper to avoid the mixin-name
collision, and clamps negative token-savings artifacts at the
metrics/cost aggregation boundary so stats cannot under-report actual
forwarded savings.

Closes #1696

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Treat final OpenAI `user`, `tool`, and `function` messages as the
mutable cache-mode live zone.
- Route OpenAI cache-boundary calls through
`OpenAIHandlerMixin._strict_previous_turn_frozen_count` explicitly so
the Anthropic mixin method cannot shadow it in `HeadroomProxy`'s MRO.
- Preserve cache-mode live-tail boundaries even when compression-cache
state would otherwise freeze the whole request.
- Clamp negative `tokens_saved` artifacts in `CostTracker.record_tokens`
and `PrometheusMetrics.record_request`.
- Add regression coverage for OpenAI final `tool`/`function` tails,
over-frozen tracker state, and non-negative savings aggregation.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ maturin build --profile ci --out dist --interpreter python
Built wheel for abi3 Python >= 3.10 to dist\headroom_ai-0.29.0-cp310-abi3-win_amd64.whl

$ python -m pytest tests\test_proxy_handler_helpers.py tests\test_proxy_openai_cache_stability.py tests\test_observability_metrics.py tests\test_cost_tracker_counterfactual.py
49 passed in 10.27s

$ python -m ruff check .
All checks passed!

$ python -m mypy headroom
Success: no issues found in 407 source files

$ python -m pytest
53 failed, 7703 passed, 488 skipped, 5893 warnings, 131 errors in 595.18s (0:09:55)
```

Full-suite note: the full local `pytest` run was attempted on
Windows/Python 3.13 after building `headroom._core`. It did not complete
green due to broad pre-existing/local-environment failures outside this
change area, dominated by SQLite/memory persistence permission/path
errors plus unrelated adapter/cache/tool tests. The focused regression
suite for this PR passes, and repo-level lint/type gates pass.

## Real Behavior Proof

- Environment: Windows, Python 3.13.13, Rust/Cargo available, local
`headroom._core` wheel built with `maturin build --profile ci`.
- Exact command / steps: ran the OpenAI cache-stability tests with final
`role: "tool"` and `role: "function"` chat tails.
- Observed result:
`test_openai_cache_mode_keeps_final_tool_observation_mutable[tool]` and
`[function]` pass, proving the pipeline receives `frozen_message_count
== 2` for a 3-message request instead of freezing all 3 messages.
- Not tested: live Lemonade/KiloCode upstream session; no local Lemonade
Server was available.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

Docs and CHANGELOG are N/A for this narrow proxy bug fix. The broad
local `pytest` checkbox is intentionally left unchecked because the full
suite had unrelated local-environment failures; see the test output
above. Focused regression tests, `ruff check .`, and `mypy headroom` are
green.
2026-07-09 07:51:01 -07:00
Tejas Chopra 7c2f0ea079 feat(cache): provider-agnostic cache-mode delta + cc-agnostic prefix comparison (#1868)
## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
2026-07-08 13:29:35 -07:00
ParideboyandJD Davis 3076e32172 fix(proxy): serve /favicon.ico locally instead of tunneling upstream (#1787) (#1847)
## Description
The Headroom dashboard tunnels `GET /favicon.ico` requests to the
wrapped upstream provider instead of serving its own. No route matched
`/favicon.ico` in `headroom/proxy/server.py`, so the request fell
through to the catch-all passthrough route
(`headroom/providers/proxy_routes.py:994-1026`) registered by
`register_provider_routes(app, proxy)`, and got forwarded to whichever
LLM backend the proxy is wrapping — burning a real upstream request (and
possibly failing auth) for a browser's automatic favicon fetch while
viewing `/dashboard`.

Closes #1787

## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made
- `headroom/proxy/server.py`: added a `GET /favicon.ico` route returning
`Response(status_code=204)`, registered next to the existing
`/dashboard` route — i.e. before `register_provider_routes(app, proxy)`
(line ~4184) registers the passthrough catch-all, so it takes priority.
- `tests/test_proxy_handler_helpers.py`: `_PassthroughRequest.url.path`
was hardcoded to `/favicon.ico` as a generic "goes to passthrough"
example, which encoded the bug as expected behavior. Changed to
`/some/other/path` so the passthrough-helper test no longer depends on
favicon requests going upstream.
- `tests/test_proxy_favicon_route.py` (new): regression test spinning up
the real FastAPI app via `create_app`/`TestClient`, asserting `GET
/favicon.ico` returns 204 and `proxy.handle_passthrough` is never
called.
- `CHANGELOG.md`: added an entry under `## Unreleased` / `### Fixed`.

## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_proxy_favicon_route.py tests/test_proxy_handler_helpers.py -q
28 passed

$ python -m pytest tests/test_proxy_healthchecks.py tests/test_proxy_passthrough_integration.py tests/test_proxy_favicon_route.py tests/test_proxy_handler_helpers.py -q
41 passed, 19 skipped

$ python -m ruff check headroom/proxy/server.py tests/test_proxy_favicon_route.py tests/test_proxy_handler_helpers.py
All checks passed!

$ python -m ruff format --check headroom/proxy/server.py tests/test_proxy_favicon_route.py tests/test_proxy_handler_helpers.py
3 files already formatted

$ python -m mypy headroom/proxy/server.py
(no errors)
```

## Real Behavior Proof
- Environment: Windows 11, Python 3.13, local checkout, `python -m
pytest`/`ruff`/`mypy` run directly (no `uv` available in this shell).
- Exact command / steps: `python -m pytest
tests/test_proxy_favicon_route.py -v` — this test builds the real proxy
app with `create_app(ProxyConfig(...))`, wraps
`client.app.state.proxy.handle_passthrough` with a mock, then issues
`client.get("/favicon.ico")` via a real `TestClient` request through the
full FastAPI routing stack (not a unit-level call of the handler
function directly).
- Observed result: response status is `204`, and `handle_passthrough`
(the function that forwards to the upstream provider) is asserted
`not_called()` — confirming the request is now intercepted before
reaching the catch-all passthrough route, and does not tunnel to the
wrapped provider.
- Not tested: did not manually run `headroom wrap <provider>` end-to-end
and open a real browser tab to `/dashboard` to visually confirm the
favicon icon in the tab (the fix returns 204/no-icon rather than a real
bundled `.ico` — browsers handle this fine, but the visual "no more
broken/upstream favicon request" experience wasn't screenshotted). The
FastAPI-level test above exercises the actual routing/dispatch path this
bug lived in.

## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A — no
user-facing docs describe dashboard route internals beyond CHANGELOG)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated CHANGELOG.md where applicable

## Screenshots (if applicable)
N/A — server-side route change, no UI change.

## Additional Notes
Deliberately kept the fix minimal: no `StaticFiles` mount or general
static-asset serving system was added, since a single favicon route
doesn't warrant that abstraction. No real `.ico` binary asset was
bundled either — a `204 No Content` response is sufficient for browsers
and avoids maintaining a binary asset in the repo; this can be upgraded
to serve a real branded icon later if desired.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-07-07 23:36:10 -05:00
Tejas Chopra 248ae0f3e0 fix(proxy): freeze must forward cached (compressed) prefix byte-identical — stop token-mode cache busting (#1850)
The freeze path (both providers) emits the agent's ORIGINAL bytes for a
frozen message, but the provider cached whatever we FORWARDED last turn
(the compressed form). Forwarding original then mismatches the cached
prefix and busts it from that point — re-creating the whole suffix.
Measured on a real SWE-bench run: 100% of attributed misses were
prefix_change, ~56% of ALL cache-writes were bust-induced (2.8M tokens),
driving cache_create +150% and cost +41% vs baseline.

Cache mode already avoided this via _extract_cache_stable_delta (replay
the previously-forwarded prefix, compress only the delta). Token mode
called apply(frozen_count) directly, which forwards original for the
frozen region.

Fix: add a shared, provider-agnostic overlay_cached_prefix() that
replays the previously-forwarded (cached, compressed) prefix
byte-identical, append-only guarded and idempotent, and apply it in BOTH
the Anthropic and OpenAI handlers right before forwarding. This makes
freezing byte-identical in every mode, so the only remaining difference
between "token" and "cache" mode is how large a mutable
(still-compressible) tail each leaves — not whether the frozen prefix
busts the cache.

Tests:
- test_cache_prefix_overlay.py: the helper (replay, append-only guard,
idempotence).
- test_cross_turn_cache_safety.py: the invariant that was missing —
drive the REAL tracker + freeze + overlay over multiple append-only
turns against a simulated provider prefix cache and assert the forwarded
prefix stays byte-identical turn-over-turn. Load-bearing: it fails
(detects the bust) without the overlay.

## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
2026-07-06 14:54:39 -07:00
Rod Boev da2d8dc9db fix(proxy): cancel retry backoff on shutdown (#1834)
## Description

During proxy shutdown, an in-flight retrying request can currently stay
asleep inside `_retry_request()` and keep the client socket hanging
until the retry timer expires or an external supervisor kills the
process. This wires retry backoff to a proxy-scoped shutdown event so
shutdown interrupts those waits immediately and returns a clear `503`
response instead of leaving the request stalled. Closes #1821.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added a proxy-scoped shutdown event in `headroom/proxy/server.py`.
- Cleared that event at startup and set it at shutdown before teardown
proceeds.
- Replaced both retry-backoff sleeps with a helper that wakes on either
timeout or shutdown.
- Returned a shutdown `503` with `retry-after: 0` when shutdown
interrupts retry backoff.
- Stopped the shutdown interruption logs from falling back to the raw
upstream URL when no safe path string is available.
- Added focused regressions for retry-backoff interruption and shutdown
event signaling.
- Updated the existing Retry-After tests to observe the new
shutdown-aware wait helper instead of the old raw sleep hook.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_proxy_handler_helpers.py
tests/test_proxy_pipeline_lifecycle.py -q`)
- [x] Unit tests pass (`uv run pytest tests/test_proxy_retry_429.py -q`)
- [x] Linting passes (`uv run ruff check headroom/proxy/server.py
tests/test_proxy_handler_helpers.py tests/test_proxy_retry_429.py
tests/test_proxy_pipeline_lifecycle.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_proxy_handler_helpers.py tests/test_proxy_pipeline_lifecycle.py -q
32 passed, 1 warning in 13.05s

uv run pytest tests/test_proxy_retry_429.py -q
10 passed, 1 warning in 1.12s

uv run ruff check headroom/proxy/server.py tests/test_proxy_handler_helpers.py tests/test_proxy_retry_429.py tests/test_proxy_pipeline_lifecycle.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows, project `uv` environment, focused proxy retry
and shutdown regressions.
- Exact command / steps: copy the updated shutdown regression files into
a detached `origin/main` worktree and run
`tests/test_proxy_handler_helpers.py` plus
`tests/test_proxy_pipeline_lifecycle.py`, then rerun those files on this
branch and separately rerun `tests/test_proxy_retry_429.py` after
updating the existing Retry-After tests to patch the shutdown-aware wait
helper.
- Observed result: base fails because retry backoff still returns the
original `429` and `shutdown()` leaves the retry event unset; head
passes the focused file, preserves the existing Retry-After assertions,
and returns a shutdown `503` with `retry-after: 0` while signaling retry
waiters during shutdown.
- Not tested: live systemd-managed shutdown on Linux or a full VS Code /
Claude Code session.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

This is intentionally scoped to retry backoff during shutdown. It does
not try to cancel unrelated in-flight request work or change the broader
retry policy outside shutdown.
2026-07-06 06:24:47 -07:00
Vinay Gupta a9322477e3 fix: preserve anthropic passthrough tool order (#1427)
## Description

Preserves Anthropic `tools` order when Headroom is forwarding a
passthrough/no-optimize request. This fixes a Claude Code style
`tool_result` continuation failure against stricter Anthropic-compatible
upstreams that treat the client's original tool ordering as part of the
conversation state.

Closes #1417

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Preserve client-provided Anthropic `tools` order when `optimize=False`
or the request is explicitly in Headroom passthrough/bypass mode.
- Keep deterministic tool sorting for optimized requests where Headroom
may rewrite the body for cache stability.
- Avoid sorting batch-request tools before the no-optimize passthrough
branch.
- Add regression coverage for the Anthropic HTTP path to prove
no-optimize forwarding keeps `Read`, then `Bash` tool order.
- Update existing cache-stability and byte-faithful forwarding tests so
no-optimize/passthrough expects preserved client order while optimized
mode still proves deterministic sorting.

## Testing

- [x] Focused unit tests pass (`pytest` on touched proxy test files)
- [x] Linting passes (`ruff check` and `ruff format --check` on touched
files)
- [x] Type checking passes (`mypy headroom`)
- [x] New regression tests added
- [x] Manual testing performed

### Test Output

```text
$ uv run --no-project --python /opt/homebrew/opt/python@3.13/bin/python3.13 --with pytest --with pytest-asyncio --with anyio --with 'httpx[http2]' --with fastapi --with pydantic --with tiktoken --with click --with rich --with opentelemetry-api --with opentelemetry-sdk --with zstandard --with openai --with mcp --with uvicorn pytest tests/test_proxy_handler_helpers.py tests/test_anthropic_stage_timings.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_byte_faithful_forwarding.py -q
============================= test session starts ==============================
platform darwin -- Python 3.13.11, pytest-9.1.1, pluggy-1.6.0
rootdir: /Users/vinaygupta/Desktop/git/headroom-fix-anthropic-tool-order
configfile: pyproject.toml
plugins: anyio-4.14.1, asyncio-1.4.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 87 items

tests/test_proxy_handler_helpers.py ..........................           [ 29%]
tests/test_anthropic_stage_timings.py ....                               [ 34%]
tests/test_proxy_anthropic_cache_stability.py .........................  [ 63%]
tests/test_proxy_byte_faithful_forwarding.py ........................... [ 94%]
.....                                                                    [100%]

=============================== warnings summary ===============================
.../fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.

======================== 87 passed, 1 warning in 5.13s =========================

$ uv run --no-project --python /opt/homebrew/opt/python@3.13/bin/python3.13 --with ruff==0.15.17 ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py tests/test_anthropic_stage_timings.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_byte_faithful_forwarding.py
5 files already formatted

$ uv run --no-project --python /opt/homebrew/opt/python@3.13/bin/python3.13 --with ruff==0.15.17 ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py tests/test_anthropic_stage_timings.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_byte_faithful_forwarding.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.13.11, local fake Anthropic-compatible
upstream, local Headroom proxy launched with `--no-optimize --no-cache
--no-rate-limit --stateless`.
- Exact command / steps: ran a local reproduction harness that starts a
fake `/v1/messages` upstream and Headroom proxy, then sends a Claude
Code style two-turn flow: first assistant `Bash` `tool_use`, then user
`tool_result`.
- Observed result: after this patch, both direct and proxied flows
returned `200` for `first_tool_use` and `second_tool_result`. The fake
upstream log showed the proxied `tools` array remained `["Read",
"Bash"]` on both turns.

```text
DIRECT
  first_tool_use: 200
  second_tool_result: 200

PROXIED
  first_tool_use: 200
  second_tool_result: 200

UPSTREAM REQUEST LOG
  proxied first turn tools: ["Read", "Bash"]
  proxied tool_result turn tools: ["Read", "Bash"]
```

- Not tested: full `pytest`, full-repo `ruff check .`, `mypy headroom`,
or a live third-party Anthropic-compatible provider.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

- This PR intentionally does not add documentation because it fixes
passthrough behavior rather than introducing a new user-facing option.
- The code-comment checklist item is left unchecked because the change
is covered by a small helper docstring and regression tests; no extra
inline comments seemed necessary.
- `CHANGELOG.md` is left unchanged because this is a narrowly scoped bug
fix.
- Local pytest collection for these proxy tests required a local
`headroom._core` extension symlink, which was removed before committing.
2026-06-30 08:38:51 -05:00
Rod Boev 3be2526b76 fix(proxy): add an Anthropic buffered read-timeout override (#1331)
## Description

Buffered Anthropic `/v1/messages` requests still use Headroom's generic
300-second read timeout, which can produce proxy-generated `502
ReadTimeout` errors on long turns. This adds a dedicated buffered
Anthropic timeout, keeps it applied across CCR and memory continuations
plus batch paths, and makes the direct server entrypoint enforce the
same positive-integer contract as the Click CLI. Closes #1261.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `anthropic_buffered_request_timeout_seconds` for buffered
Anthropic reads.
- Routed `/v1/messages`, CCR continuation, memory continuation, batch
create, batch passthrough, and batch results through that timeout.
- Enforced the same positive-integer validation for
`HEADROOM_ANTHROPIC_BUFFERED_REQUEST_TIMEOUT_SECONDS` and
`--anthropic-buffered-request-timeout-seconds` in both startup paths.
- Added focused regressions and updated `CHANGELOG.md`.

## Testing

- [x] `uv run pytest tests/test_proxy/test_anthropic_buffered_timeout.py
tests/test_cli_proxy_improvements.py::TestNewEnvVarWiring`
- [x] `uv run ruff check .`
- [x] `uv run ruff format . --check`

### Test Output

```text
$ uv run pytest tests/test_proxy/test_anthropic_buffered_timeout.py tests/test_cli_proxy_improvements.py::TestNewEnvVarWiring
17 passed in 3.42s

$ uv run ruff check .
All checks passed!

$ uv run ruff format . --check
966 files already formatted
```

## Real Behavior Proof

- Environment: local FastAPI `TestClient` with stubbed retry and HTTP
client seams
- Exact command / steps: run `uv run pytest
tests/test_proxy/test_anthropic_buffered_timeout.py
tests/test_cli_proxy_improvements.py::TestNewEnvVarWiring`; the tests
build `ProxyConfig(request_timeout_seconds=7, connect_timeout_seconds=3,
anthropic_buffered_request_timeout_seconds=19)`, drive `/v1/messages`,
`/v1/messages/batches`, `/v1/messages/batches/{batch_id}/results`, a CCR
continuation, and a memory continuation through `TestClient`, then
verify `HEADROOM_ANTHROPIC_BUFFERED_REQUEST_TIMEOUT_SECONDS=0` falls
back to `600`, `--anthropic-buffered-request-timeout-seconds 0` is
rejected, and default proxy timeouts stay `read=300` and `write=300`
- Observed result: buffered Anthropic paths use
`httpx.Timeout(connect=3, read=19, write=7, pool=3)`, continuation
requests stay on that same budget, invalid zero-valued startup config is
rejected or ignored back to the default, and unrelated proxy timeout
defaults stay unchanged
- Not tested: live upstream Anthropic latency beyond the focused
stubbed-timeout regression

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have added tests that prove the fix
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
2026-06-23 22:46:31 -05:00
gglucassandClaude Opus 4.8 8c00f7103c fix(codex): poll /wham/usage for subscription limits (handshake no longer sends x-codex-* headers) (#924)
## Description

Codex's subscription usage window (primary/secondary rate-limit gauges)
stopped populating for ChatGPT-OAuth sessions. This PR restores it by
polling Codex's dedicated usage endpoint instead of relying on response
headers that are no longer sent.

### Why the previous approach no longer works

The existing code populates `CodexRateLimitState` from `x-codex-*`
rate-limit headers captured on the `/v1/responses` WebSocket handshake
(`update_from_headers` at WS accept). That worked when OpenAI returned
`x-codex-primary-used-percent`, `x-codex-primary-window-minutes`, etc.
on the handshake response.

OpenAI has since stopped sending those headers on the ChatGPT WebSocket
handshake. I confirmed this by faithfully replaying a real Plus-account
handshake (both `prewarm` and regular `request_kind`): no `x-codex-*`
headers come back on either. This matches OpenAI's own move to a
dedicated usage endpoint (`GET /backend-api/codex/usage` in CodexApi
mode) and reports such as openai/codex#14728. So `update_from_headers`
now runs on every accept but finds nothing to parse, and the window
silently stays empty.

The headers aren't coming back, so there is nothing to fix in the
parsing path. The data now lives behind a request we have to make
ourselves.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `subscription/codex_rate_limits.py`:
- `parse_codex_usage_payload()` / `update_from_usage_payload()` — map
the `GET /backend-api/wham/usage` JSON (`rate_limit.primary_window` /
`secondary_window` with `used_percent`, `limit_window_seconds`,
`reset_at`; `credits`; `rate_limit_reached_type`) into the existing
`CodexRateLimitState`. `limit_window_seconds` is converted to
window-minutes with the same round-up codex-rs uses (`(secs + 59) //
60`).
- `maybe_schedule_usage_poll()` — fire-and-forget, throttled to one
request per 60s, scoped to ChatGPT sessions (requires both a Bearer
token and `ChatGPT-Account-Id`; API-key traffic is skipped). Uses an
in-flight guard so concurrent accepts don't stack polls. Endpoint is
overridable via `HEADROOM_CODEX_USAGE_URL`.
- `proxy/handlers/openai.py`:
- At the Codex WS accept site, after the now-usually-empty
`update_from_headers` block, schedule the usage poll. Wrapped in
`contextlib.suppress` and fully non-blocking so it can never delay or
fail the WebSocket accept.

The old header-capture path is intentionally left in place as a no-cost
fallback in case OpenAI restores the headers.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (live `/wham/usage` replay against a Plus
account returned HTTP 200 with the expected schema; payload fixture in
tests mirrors that real shape)

## Test Output

```
$ uv run pytest tests/test_codex_rate_limits.py -q
........................................                                 [100%]
41 passed in 0.16s

$ uv run ruff check headroom/subscription/codex_rate_limits.py headroom/proxy/handlers/openai.py tests/test_codex_rate_limits.py
All checks passed!

$ uv run mypy headroom/subscription/codex_rate_limits.py
Success: no issues found in 1 source file
```

## Additional Notes

- New tests cover: full-payload mapping, window-minutes round-up,
credits balance kept only when `has_credits`, promo object vs string,
empty payload returns `None`, missing `used_percent` skipped,
header-gating (requires Bearer + account-id), poll throttling, and
no-event-loop safety.
- Scoping to `ChatGPT-Account-Id` keeps the poll off API-key traffic,
and the 60s throttle plus in-flight guard bound it to at most one
lightweight GET per minute per running proxy.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 17:03:14 -05:00
JD Davis 3c77e52ce4 feat: add Vertex AI proxy routing (#793)
## Description

Adds first-class GCP Vertex AI proxy routing for publisher REST
endpoints so Vertex requests are forwarded to a configurable regional
Vertex host instead of falling through to the generic
OpenAI/Anthropic/Gemini passthrough selection.

Fixes #792

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update

## Changes Made

- Added a `vertex` provider target with `VERTEX_TARGET_API_URL` and
`--vertex-api-url` support.
- Registered explicit Vertex publisher routes for Google
`generateContent`, `streamGenerateContent`, `countTokens` and Anthropic
publisher `rawPredict`, `streamRawPredict` passthrough.
- Added startup banner/routing output for Vertex AI.
- Added focused tests for provider target resolution, CLI/env config,
banner output, and route delegation.
- Added `wiki/vertex.md` with usage examples and Google Cloud source
links.

## Sources

- Vertex AI Gemini inference reference:
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference
- Google Cloud REST authentication:
https://docs.cloud.google.com/docs/authentication/rest
- Google Application Default Credentials:
https://docs.cloud.google.com/docs/authentication/application-default-credentials

## Testing

- [x] Linting passes (`python -m ruff check .`)
- [x] New tests added for new functionality
- [x] Focused unit tests pass
- [ ] Full unit suite completed locally
- [ ] Rust tests completed locally
- [ ] Type checking passes locally

## Test Output

```text
$ python -m ruff check .
All checks passed!

$ python -m pytest tests/test_provider_registry.py tests/test_provider_proxy_routes.py tests/test_cli_proxy_env.py tests/test_banner_upstream_targets.py -q
57 passed, 1 warning in 13.75s
```

Local limitations:

- `python -m pytest tests scripts/tests -q` timed out after 1 hour on
this Windows machine before completing.
- `cargo test -p headroom-proxy --test integration_vertex_raw_predict`
could not run because `cargo` is not installed on PATH in this
environment.
- The commit hook's `mypy` step fails locally on an existing Windows
`fcntl` typing issue in `headroom/subscription/tracker.py`; `ruff`,
`ruff-format`, and plugin-version hooks passed, and the commit was made
with only `mypy` skipped.

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I have added tests that prove the feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
2026-06-09 23:05:30 -07:00
chopratejas 1bc163f5bc fix(ccr): scope proactive expansion by workspace (cross-project leak)
Closes the cross-project context leak Jocelyn reported 2026-05-26:
working on a Ruby/Rails project (daphni-rails), an unrelated Python
file (an Ollama inference provider from project `tamag0`) was being
injected into context as "Proactive Context Expansion - relevant to
your query". Two completely different projects, two different
languages, two different working directories — but the same proxy
process was serving both, and the in-memory ContextTracker had no
workspace identity to filter on.

Root cause
----------
`self.ccr_context_tracker` is one instance per proxy process. Every
session, every project, every user shared the same `_contexts` dict.
`track_compression()` stored sample content with no provenance key;
`analyze_query()` ran lexical keyword overlap across the full dict
without filtering. Within the 5-minute age window, surface-level
token matches ("provider", "session", "oauth", generic code/test
structure) scored above the 0.3 relevance threshold, recommendations
came back, and execute_expansions() injected the full original
content into a foreign session.

Refuted: this is NOT a race condition (joce's hypothesis). It
reproduces single-threaded, one-request-at-a-time. Plain shared
mutable state.

Fix
---
Add a required `workspace_key` to the tracker API and filter on it
inside `analyze_query`:

1. `CompressedContext` gets a `workspace_key: str` field.
2. `track_compression(..., workspace_key=...)` is now keyword-only,
   no default — fail-loud on missing.
3. `analyze_query(..., workspace_key=...)` is also keyword-only; an
   empty workspace_key short-circuits to `[]` (fail-closed per
   `feedback_no_silent_fallbacks`).
4. The loop at `analyze_query` skips any entry whose workspace_key
   differs from the request's.

In the Anthropic proxy handler:

5. New `_resolve_ccr_workspace(request, body)` static helper uses the
   memory subsystem's `ProjectResolver` so CCR and memory agree on
   project identity. Tier order: x-headroom-project-id →
   x-headroom-cwd → CLI override → cwd: line in system prompt.
6. Both track and analyze sites gate on `ccr_workspace_key` being
   non-empty — turning off proactive expansion entirely when project
   identity can't be resolved is the safest default (it's an
   optimization, not correctness).
7. `format_expansions_for_context(expansions, workspace_label=...)`
   was already wired (GH #462 Fix C); the call site now passes the
   label so the injected block declares its provenance, symmetric
   with the memory injection header.

Affected population
-------------------
- Default mode (no `--cache`): bug fixed.
- Cache mode: was never affected — proactive expansion short-
  circuits in cache mode to preserve prefix stability.

Tests
-----
- 6 new workspace-scoping tests in `test_ccr_context_tracker.py`:
  same-workspace match still works, cross-workspace silently
  filtered, empty workspace_key fail-closes, two workspaces each
  see only their own, workspace_label propagates to formatter, LRU
  cross-workspace doesn't leak even with full tracker.
- 6 new `_resolve_ccr_workspace` resolver tests in
  `test_proxy_handler_helpers.py`: explicit project-id wins, cwd
  header → key+label, two cwds get distinct keys, no-signal
  fail-closed, system-prompt cwd: fallback, malformed request
  fail-closed.
- 32 existing tracker tests updated to pass `workspace_key="ws-test"`.
- 55/55 tests pass; ci-precheck green.

Defense-in-depth follow-up
--------------------------
The compression_store itself (`headroom/cache/compression_store.py`)
also lacks workspace scoping — a CCR `headroom_retrieve` call from
Project B for a hash created by Project A would succeed. The
practical attack surface is closed by this PR (hashes only reach
Project B's model via proactive expansion, now gated), but
defense-in-depth hardening of the store is worth a separate PR.
Filed as task #44.
2026-05-26 13:23:51 -07:00
Gili Tzabari 160989c43e fix(proxy): bound Codex Responses compression work 2026-05-11 03:02:58 -04:00
Tejas Chopra eaf5980b4a fix: stabilize codex compression, stats, and proxy lifecycle 2026-05-09 13:47:53 -07:00
chopratejas f0dcc02775 fix: A3 — byte-faithful Python forwarders; serialize canonical only when mutated
Eliminates P0-2 universally. Every Python forwarder (server.py
`_retry_request`, handlers/streaming.py `_stream_response`,
handlers/openai.py `_ws_http_fallback`, handlers/batch.py `_batch_passthrough`
+ batch-create + Google batch passthrough, handlers/anthropic.py CCR
continuation + batch endpoint) now switches from `httpx ... json=body` to
`httpx ... content=raw_bytes`. The default httpx JSON encoder was
re-serializing every request with `, `/`: ` separators and `\\uXXXX` ASCII
escapes — collapsing Anthropic prompt-cache hit-rate.

Forwarder strategy:
  - unmutated body → forward `await request.body()` verbatim;
  - mutated body  → re-serialize once via the new
    `serialize_body_canonical(body) -> bytes` helper (compact separators,
    `ensure_ascii=False`, dict insertion order preserved).

`HEADROOM_PROXY_PYTHON_FORWARDER_MODE` env var configures the mode:
  - `byte_faithful` (default) — the new behavior;
  - `legacy_json_kwarg` — explicit operator opt-in for emergency rollback.
Documented in `docs/content/docs/configuration.mdx`. NOT a fallback —
unknown values raise loudly per build constraint #4.

`BodyMutationTracker` accompanies each request through the handler so
transform sites mark the tracker (`memory_injection`,
`image_compression`, `compression_*`, `batch_compression`,
`ccr_continuation`, etc.). At forwarder dispatch we additionally compare
the final body dict against the parsed original bytes as a structural
safety net — any silent mutation we missed still triggers canonical
re-serialization.

A2 follow-up: `handlers/openai.py:534-540` (Chat Completions memory
injection) was prepending a system message; replaced with
`append_text_to_latest_user_chat_message`, the OpenAI Chat Completions
analog of `_append_context_to_latest_non_frozen_user_turn`. The cache
hot zone (system messages) is now sacrosanct on /v1/chat/completions
too. Honors `HEADROOM_MEMORY_INJECTION_MODE=disabled`.

Structured logging: every forwarder emits an `event=outbound_request`
log line with `forwarder`, `path`, `body_bytes`, `body_mutated`,
`mutation_reasons`, `source` (passthrough|canonical|legacy),
`request_id`. Never logs Authorization or full body.

`_read_request_json` factored to share `_read_request_body_bytes` with
new `read_request_json_with_bytes` so the anthropic handler can capture
both the parsed dict and the original (decompressed) bytes.

Tests:
  - `tests/test_proxy_byte_faithful_forwarding.py` (28 tests):
    SHA-256 byte-equality on /v1/messages and streaming, unicode
    preservation, numeric precision, mutation-tracker invariants,
    canonical-serializer properties, legacy-mode rollback, OpenAI
    Chat memory routing.
  - Existing test mocks updated to accept the new `**kwargs` on
    `_retry_request` (no behavior change).
  - `tests/test_proxy_handlers_batch.py` updated to read the captured
    `content=` bytes (formerly `json=`).
  - One A2 test corrected (`test_anthropic_tool_sort_and_context_append_helpers`)
    to match the live-zone-tail semantics introduced by A2.

Constraints satisfied: configurable env var; no new regex / hardcodes;
no silent fallback (`legacy_json_kwarg` is operator opt-in);
performant (`prepare_outbound_body_bytes` is O(1) for passthrough);
elegant single-responsibility helpers; structured tracing logs.
2026-05-02 09:02:10 -07:00
Wei Alexander Xin cf60882949 fix: release image router models after compression 2026-04-29 01:45:27 -04:00
GarmandClaude Opus 4.7 efd2ac1ca4 chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.

Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.

Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 15:33:30 +02:00
JerrettDavisandCopilot 1b377d8c43 test: add focused pipeline coverage
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-21 23:54:28 -05:00