Commit Graph
106 Commits
Author SHA1 Message Date
2f58877732 fix(proxy): check x-headroom-base-url off the event loop in chat and responses (#4856)
## Description

`handle_openai_chat` and `handle_openai_responses` validated
`x-headroom-base-url` with the synchronous `is_safe_upstream_url`, which
waits for the DNS lookup (up to `HEADROOM_UPSTREAM_RESOLVE_TIMEOUT_S`, 3
s by default) on the event-loop thread. Chat did this three times per
request and responses twice, so one slow-resolving host held up every
other in-flight request for several seconds. The other routes in
`headroom/providers/proxy_routes.py` already use
`is_safe_upstream_url_async`.

Both handlers now resolve the header once, through a new
`_resolve_openai_upstream_base_async` that uses the async guard (lookup
on a worker thread, pin scope opened on the request's task), and reuse
that result for the destination and the extra-header decision.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/openai.py`: header parsing split into
`_openai_base_url_candidate`; sync `_resolve_openai_upstream_base` (kept
for `_resolve_openai_upstream`) and new
`_resolve_openai_upstream_base_async` share it;
`_openai_upstream_from(request, custom)` picks the upstream from an
already-resolved override; chat and responses call the async resolver
once.
- `tests/test_proxy/test_openai_upstream_guard_off_loop.py`: asserts the
guard runs once and not on the event-loop thread, for chat and
responses.
- `tests/test_proxy_openai_cache_key_integration.py`: its stub now
patches the async resolver the handler calls.

**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-idna-pin-once` adds the identical `_openai_upstream_from`
and resolve-once hunks (this PR is a superset of that part);
`fix/openai-base-url-reject` changes the rejection branch of
`_resolve_openai_upstream_base`, which after this PR must be mirrored in
`_resolve_openai_upstream_base_async`. Suggested order: idna-pin-once,
then this PR, then reject (rebasing reject means raising the 400 from
the shared rejection path in both resolvers).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New test on origin/main (c21b0813)
E   AssertionError: the guard ran on the event-loop thread
E   assert [True, True, True] == [False]      (chat)
E   assert [True, True] == [False]            (responses)
============================== 2 failed in 0.72s ===============================

# This branch: new test + cache-key integration
============================== 7 passed in 7.30s ===============================

# Area suites: test_upstream_guard, test_grok_shared_proxy_call_sites, test_proxy/test_openai_upstream_header,
# test_proxy_openai_cache_key_integration, test_upstream_credential_scoping, test_header_isolation,
# test_proxy_hardening, test_openai_codex_routing, test_proxy/test_openai_upstream_guard_off_loop
======================= 282 passed, 5 warnings in 56.32s =======================

$ ruff check / ruff format --check (0.17.3) on changed files
All checks passed!
3 files already formatted
$ mypy headroom/proxy/handlers/openai.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, full app from `create_app` driven
through `httpx.ASGITransport` on one event loop; `socket.getaddrinfo`
replaced so `slow.example` takes 2 s to answer (everything else real).
- Exact command / steps: start a `/v1/chat/completions` request with
`x-headroom-base-url: https://slow.example/v1`, then 0.2 s later send
`GET /livez` and time it.
- Observed result:
  ```text
  == origin/main
/livez sent 0.2s after a chat request naming a 2s-to-resolve host: HTTP
200 after 6.03s (0.20s if the loop is free)
  == this branch
/livez sent 0.2s after a chat request naming a 2s-to-resolve host: HTTP
200 after 0.20s (0.20s if the loop is free)
  ```
- Not tested: under real slow DNS.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no change in which upstream is
chosen; the lookup moves to a worker thread and happens once per
request.
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

Follow-up, not in this PR: `select_passthrough_base_url`
(`headroom/providers/proxy_targets.py`, Azure `api-key` branch) and
`headroom/providers/registry.py` still call the sync guard; the
passthrough route already checks the same header with the async guard
first, so the sync call there is a second lookup on the loop.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4856"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-15 01:52:37 +05:30
6d8121cfd4 fix(proxy): send operator extra headers only to a designated origin (#4858)
## Description

`upstream_trust.is_trusted_upstream` decides whether operator-configured
extra headers (`OPENAI_TARGET_API_HEADERS` /
`ANTHROPIC_TARGET_API_HEADERS`, stored as secrets) may be merged for a
client-chosen `x-headroom-base-url`. It compared hostnames only, so
`http://<designated host>` or `https://<designated host>:<other port>`
also received those headers, over cleartext in the http case.

It also used one trusted set for every provider, so a chat/responses
request with `x-headroom-base-url: https://api.anthropic.com` received
the OpenAI extras, and `/v1/messages` pointed at
`https://api.openai.com` or the configured OpenAI gateway received the
Anthropic extras.

Trust is now decided per provider, on the origin (scheme, host, port).
The OpenAI extras go only to the OpenAI target / `api.openai.com`; the
Anthropic extras only to the Anthropic target, the oMLX target or
`api.anthropic.com`; `HEADROOM_UPSTREAM_ALLOWED_HOSTS` entries count for
both.

Origin rules:
- a bare designation (`api.openai.com`, or a scheme-less configured URL)
means `https` on port 443;
- configured provider API URLs (`OPENAI_TARGET_API_URL`,
`ANTHROPIC_TARGET_API_URL`, oMLX, Gemini/Vertex/Cloud Code/Bedrock)
designate their own origin, e.g. `http://omlx.internal:8888`;
- `HEADROOM_UPSTREAM_ALLOWED_HOSTS` entries may be a bare host,
`host:port` (https) or a full URL (`http://gateway.internal:4000`).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update

## Changes Made

- `headroom/proxy/upstream_trust.py`: `_origin()`,
`trusted_upstream_origins()`, `is_trusted_upstream` matches on origin;
malformed ports and bracketed hosts are refused instead of raising; the
withheld-headers warning is keyed and logged by origin
(`host=http://gateway.internal:4000`) so a listed bare host reached over
http still warns; the now-unused host-only helpers
`trusted_upstream_hosts` / `_env_allowed_hosts` are removed (only one
test used them).
- `merge_extra_headers` / `is_trusted_upstream` take `provider`; the
OpenAI chat and responses handlers pass `"openai"`, the Anthropic
handler (`_build_outbound_anthropic_headers`) passes `"anthropic"`.
Without `provider` the union of all targets is used, as before.
- Docs: `configuration.mdx` (designation rules, example, upgrade note),
`security-model.mdx`, `pipeline-extensions.mdx`.
- Tests: new `tests/test_upstream_trust_origin.py` (8 cases, through
`merge_extra_headers` and `is_trusted_upstream`);
`tests/test_proxy/test_omlx_routing.py` asserts through
`is_trusted_upstream` instead of the removed helper.

**Sibling PRs:** `fix/upstream-idna-pin-once`,
`fix/upstream-guard-off-loop` and `fix/openai-base-url-reject` change
how `handlers/openai.py` resolves the URL passed to
`merge_extra_headers`; this PR only changes `upstream_trust.py`, so
there is no textual overlap and it can merge in any order.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New tests on origin/main (c21b0813)
FAILED tests/test_upstream_trust_origin.py::test_secret_extra_headers_are_not_merged_for_another_scheme_or_port
FAILED tests/test_upstream_trust_origin.py::test_a_configured_api_url_designates_its_own_origin
FAILED tests/test_upstream_trust_origin.py::test_a_scheme_less_configured_api_url_means_https
FAILED tests/test_upstream_trust_origin.py::test_allowed_hosts_entries_designate_one_origin[llm.corp-...]
FAILED tests/test_upstream_trust_origin.py::test_allowed_hosts_entries_designate_one_origin[llm.corp:8443-...]
FAILED tests/test_upstream_trust_origin.py::test_allowed_hosts_entries_designate_one_origin[http://llm.corp:8080-...]
FAILED tests/test_upstream_trust_origin.py::test_malformed_destinations_are_not_trusted[https://[api.openai.com]/v1]
FAILED tests/test_upstream_trust_origin.py::test_malformed_destinations_are_not_trusted[https://api.openai.com:notaport/v1]
============================== 8 failed in 1.13s ===============================

# This branch
============================== 8 passed in 1.67s ===============================

# Every test file that mentions extra_headers / UPSTREAM_ALLOWED_HOSTS / upstream_trust (24 files)
=========== 571 passed, 12 skipped, 8 warnings in 166.82s (0:02:46) ============
# after review follow-ups (origin-keyed warning, helper removal)
tests/test_upstream_trust_origin.py tests/test_proxy/test_omlx_routing.py tests/test_upstream_credential_scoping.py
tests/test_proxy/test_factory_passthrough.py tests/test_grok_shared_proxy_call_sites.py tests/test_header_isolation.py
======================= 145 passed, 1 warning in 37.22s ========================

$ ruff check / ruff format --check (0.17.3) on changed files
All checks passed!
$ mypy headroom/proxy/upstream_trust.py --ignore-missing-imports
Success: no issues found in 1 source file
```

### Per-provider scoping (follow-up commit 37ff98d0)

```text
# 3 new tests in tests/test_upstream_trust_origin.py, before the commit
FAILED tests/test_upstream_trust_origin.py::test_each_provider_extras_stay_with_that_provider
FAILED tests/test_upstream_trust_origin.py::test_an_allowed_host_is_designated_for_both_providers
FAILED tests/test_upstream_trust_origin.py::test_the_handlers_scope_extras_to_their_own_provider
# after
============================== 11 passed in 0.71s ==============================
# every test file mentioning extra_headers / UPSTREAM_ALLOWED_HOSTS / upstream_trust
============ 574 passed, 12 skipped, 8 warnings in 86.12s (0:01:26) ============
```

Full app (`create_app`, upstream replaced by an httpx MockTransport that
records headers; `openai_api_url=https://gateway.example.invalid`,
OpenAI extra `x-operator-oai`, Anthropic extra `x-operator-ant`):

```text
before:
chat-xprov base=https://api.anthropic.com        -> 200 oai=OAI-MARKER ant=-
resp-xprov base=https://api.anthropic.com        -> 200 oai=OAI-MARKER ant=-
msgs-xprov base=https://api.openai.com           -> 200 oai=-          ant=ANT-MARKER
msgs-xprov base=https://gateway.example.invalid  -> 200 oai=-          ant=ANT-MARKER
after:
chat-xprov base=https://api.anthropic.com        -> 200 oai=-          ant=-
resp-xprov base=https://api.anthropic.com        -> 200 oai=-          ant=-
msgs-xprov base=https://api.openai.com           -> 200 oai=-          ant=-
msgs-xprov base=https://gateway.example.invalid  -> 200 oai=-          ant=-
```
Same-provider cases are unchanged (chat to the OpenAI gateway and
messages to `https://api.anthropic.com` still carry their own extras).

## Real Behavior Proof

- Environment: macOS, Python 3.12, `python -m headroom.cli proxy
--no-optimize` from origin/main and from this branch with
`HEADROOM_UPSTREAM_ALLOWED_HOSTS=localtest.me`,
`HEADROOM_ALLOWED_BASE_URLS=localtest.me`,
`OPENAI_TARGET_API_HEADERS='{"X-Operator-Key": "operator-secret"}'`;
`localtest.me` resolves to 127.0.0.1 where a local server records
request headers.
- Exact command / steps: `curl
http://127.0.0.1:$PORT/v1/chat/completions -H 'x-headroom-base-url:
http://localtest.me:$REC/v1' ...` (plain http, non-443 port of the
designated bare host).
- Observed result:
  ```text
  == origin/main
  HTTP 200
recorder: /v1/v1/chat/completions auth= Bearer client-key
x-operator-key= operator-secret
  == this branch
  HTTP 200
recorder: /v1/v1/chat/completions auth= Bearer client-key
x-operator-key= None
  log: upstream_extra_headers_withheld host=http://localtest.me:21154
  ```
  The request is still proxied; only the operator's header is withheld.
- Not tested: Copilot / Bedrock live upstreams (they pass
`upstream_url=None` or do not merge extras, so they are unaffected).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: **yes, for one setup.** An operator
who listed a bare host in `HEADROOM_UPSTREAM_ALLOWED_HOSTS` and has
clients reach it over plain `http://` or a non-443 port (via
`x-headroom-base-url`) stops sending the configured extra headers there
after upgrading. The request is still proxied, and the proxy logs
`upstream_extra_headers_withheld host=<scheme>://<host>:<port>` once per
origin. Also, a deployment that relied on one provider's extras reaching
the other provider's configured target (for example the Anthropic extras
sent to `OPENAI_TARGET_API_URL` through `/v1/messages`) stops sending
them; list that origin in `HEADROOM_UPSTREAM_ALLOWED_HOSTS` if that is
intended. Remedy for the scheme/port case: list the exact origin, e.g.
`HEADROOM_UPSTREAM_ALLOWED_HOSTS=http://gateway.internal:4000` (or
`gateway.internal:8443` for https on another port). Configured provider
targets keep working because they designate their own origin.
- Kill switch / disable path: the env entry above.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4858"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-14 10:46:28 +05:30
b9f6255395 fix(proxy): log client-supplied model ids as one escaped token (#4854)
## Description

The Anthropic handler logged the request's `model` field raw in three
places (`RESPONSE-CACHE-HIT`, the `cache_mode_prefix_mismatch` skip
line, `UPSTREAM_ERROR`). `sanitize_anthropic_model_id` strips ANSI
styling but keeps an interior newline, so a model id with a newline
split one log record into two, and the second line could look like any
other record. The OpenAI handler had the same raw `model=` in its
cache-hit, upstream-error, `/v1/responses` passthrough (HTTP and WS) and
token-recount lines, and the Bedrock handler logged the model id from
the URL path the same way.

All of them now go through `headroom.log_safety.safe_id`, the helper
`providers/openai.py` already uses for model ids: the value is rendered
with `repr` (newlines escaped) and capped in length.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: `safe_id(model)` in the
cache-hit, prefix-mismatch and upstream-error log lines.
- `headroom/proxy/handlers/openai.py`: `safe_id(...)` in the cache-hit,
upstream-error, `/v1/responses` HTTP and WS passthrough, and raw-payload
recount log lines.
- `headroom/proxy/handlers/bedrock.py`: `safe_id(model_id)` in the
compression log line.
- `tests/test_proxy/test_log_model_id.py`: drives `/v1/messages` and
`/v1/chat/completions` through `create_app` with a model id that
contains a newline and checks the `UPSTREAM_ERROR` and
`RESPONSE-CACHE-HIT` records stay on one line.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new test, with the fix
$ pytest -q tests/test_proxy/test_log_model_id.py
3 passed in 9.18s

# same test with the handler changes stashed (origin/main code)
FAILED tests/test_proxy/test_log_model_id.py::test_upstream_error_log_keeps_model_on_one_line[/v1/messages-headers0]
FAILED tests/test_proxy/test_log_model_id.py::test_upstream_error_log_keeps_model_on_one_line[/v1/chat/completions-headers1]
FAILED tests/test_proxy/test_log_model_id.py::test_anthropic_cache_hit_log_keeps_model_on_one_line
3 failed in 9.36s

# area tests
$ pytest -q tests/test_proxy tests/test_bedrock_*.py tests/test_log_safety.py
738 passed, 1 warning in 251.77s

$ ruff check / ruff format --check (0.17.3) on the 4 changed files: All checks passed, 4 files already formatted
$ mypy headroom/proxy/handlers/{anthropic,openai,bedrock}.py --ignore-missing-imports: Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, the real `headroom proxy` CLI
(`--stateless --no-telemetry`, isolated `HOME`) with
`--anthropic-api-url` pointed at a local HTTP server that answers every
request with an Anthropic-format 400. No mocks in the proxy process.
- Exact command / steps: `curl -X POST
http://127.0.0.1:<port>/v1/messages` with `"model": "claude-x\nINJECTED
fake log line"`; grep the proxy's stderr.
- Observed result:
- origin/main: `UPSTREAM_ERROR status=400 ... model=claude-x` followed
by a forged line starting `INJECTED fake log line compressed=no ...`;
the provider warning `Unknown Anthropic model 'claude-x` is also split,
with `INJECTED fake log line': using default Claude limit ...` on its
own line.
- this branch: `UPSTREAM_ERROR status=400 ... model='claude-x\nINJECTED
fake log line' compressed=no ...` and `Unknown Anthropic model
'claude-x\nINJECTED fake log line': using default Claude limit ...`,
each on one line; no line starts with the injected text.
- The live run found the provider warning, which the handler tests did
not cover; fixed in the second commit with its own regression test.
- Not tested: a live provider call. Separately observed on both trees
and not changed here: handlers echo the raw model into the
`x-headroom-model` response header, so a model id containing a newline
makes uvicorn/h11 drop the connection for that request.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: log format only: `model=` values are
now quoted (`model='claude-sonnet-4-6'`).
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none; anything that greps these lines for
`model=<id>` sees the id in quotes.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4854"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-14 10:46:11 +05:30
Mansurshakh JaparovandJD Davis 0f3b0d8a21 fix(memory): resolve pi cwd before injecting project tools (#4618)
## Description

Closes #4442.

pi's system `<cwd>` block now resolves the memory project. If project
scope is still unresolved with the `empty` fallback, the Anthropic
handler no longer advertises unusable memory tools, including tools
recorded earlier in the session.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Parse pi cwd only in system text. Keep main's line parser, header/CLI
precedence and limited user-message fallback.
- Reuse `is_project_unresolved` for context and sticky-tool replay.
Client tools and global/user storage behavior stay intact.
- Add initialized-handler scope regressions, replace
source-inspection/forwarding-only checks, and document project scope in
the published memory page.

## Testing

- [x] Unit tests pass (`pytest`) — related suite and the normal critical
gate, not the entire Python suite locally.
- [x] Linting passes (`ruff check .`) — hosted lint passed on 5ace63e2;
local filesystem limitation below.
- [x] Type checking passes (`mypy headroom`) — normal full hook with its
existing `--ignore-missing-imports` setting.
- [x] New tests added for the scope regression
- [x] Manual testing performed

### Test Output

```text
# 16 related test files
329 passed, 12 skipped, 1 warning in 43.07s

# ordinary pre-push: make ci-precheck
178 passed, 5 skipped, 1 warning in 10.87s
✅ ci-precheck PASSED — safe to push.

uv run ruff format --check .
1860 files already formatted
```

Normal commit hooks (Ruff, formatter, full mypy, commitlint) and the
complete pre-push path passed: Rust fmt/clippy/workspace tests, native
build/import verification, critical Python tests and commitlint. The
existing Starlette test-client warning was not filtered.

Local whole-tree Ruff reports EXE002 on five **unchanged** OAuth plugin
files: my `fuseblk` checkout exposes them as executable despite Git
modes `100644`; `chmod` returns zero without changing the mode. The
uncached check agrees. I did not change those files, add shebangs or
ignore the rule. Hosted Ruff, formatting and full mypy passed on
5ace63e2; the local filesystem limitation remains.

The first full CI caught a missing scope predicate on the existing
lifecycle test double. Fixed in
[5ace63e2](https://github.com/headroomlabs-ai/headroom/commit/5ace63e2cbd2295f9594d1e0442a679aad99979c);
its complete file passes (`5 passed, 1 warning in 13.01s`). This
follow-up changes no production code or assertions and adds no runtime
fallback. The behavior proof below ran on 7aeb83cf with the same
production source; that qualification is historical; current-head checks
are recorded below.

### Author follow-up: 80453a03

Moved the project-scope guidance out of the unpublished wiki and into
`docs/content/docs/memory.mdx`; this commit changes no production code
or tests. Viewed the actual Next.js `/docs/memory#proxy-project-scope`
page and its table-of-contents entry; no browser errors.

The existing router/scope tests pass: 65 passed, with four SQLite
ResourceWarnings. All seven pi-parser lines and all six branch edges are
covered, so I did not add duplicate tests for the old Codecov
percentage.

Normal commit and pre-push hooks passed, including Rust checks, native
build/import, 180 critical Python tests (5 skipped, 1
tokenizer-approximation warning), and commitlint. My first push timed
out with the wrong local ONNX path; the next found missing pip in the
fresh uv environment. Both setup issues were fixed without bypassing
checks. Current-head hosted CI is not yet qualified.

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, freshly built native core,
real TCP/Uvicorn proxy and SQLite/ONNX memory backend. Only the upstream
is a local Anthropic-shaped HTTP server; no provider charges.
- Exact command / steps: after `uv sync --locked --extra dev --extra
proxy --extra memory` and `make verify-rust-core`, run this from the
checkout. It prints the actual loaded native module/hash and leaves a
disposable `.headroom-4442-*` directory for inspecting SQLite. Both HTTP
listeners shut down before exit.

<details>
<summary>Standalone HTTP/SQLite reproduction</summary>

```bash
./.venv/bin/python - <<'PY'
import hashlib,http.client,json,os,socket,tempfile,threading,time
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from pathlib import Path
import uvicorn
import headroom._core as core
assert Path(core.__file__).resolve().parent == Path.cwd().resolve()/"headroom"
native_sha=hashlib.sha256(Path(core.__file__).read_bytes()).hexdigest()
from headroom.memory.storage_router import ProjectResolver,RequestContext,extract_system_prompt
from headroom.proxy.server import ProxyConfig,create_app
from headroom.proxy.memory_handler import MEMORY_TOOL_NAMES
from headroom.proxy.helpers import get_session_tool_tracker
root=Path(tempfile.mkdtemp(prefix=".headroom-4442-",dir=Path.cwd()))
project_a=root/"pi%2Fproject-a";project_b=root/"project-b"
project_a.mkdir(exist_ok=True);project_b.mkdir(exist_ok=True)
os.environ["HEADROOM_WORKSPACE_DIR"]=str(root/"workspace")
os.environ["HEADROOM_BEACON"]="off"
os.environ["HEADROOM_TELEMETRY"]="off"
os.environ["NO_PROXY"]="127.0.0.1,localhost"
forwarded=[]
class Gateway(BaseHTTPRequestHandler):
    def do_POST(self):
        body=json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        forwarded.append({"path":self.path,"body":body})
        payload=json.dumps({"id":"msg_local_scope","type":"message","role":"assistant","model":"claude-haiku-4-5","content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn","stop_sequence":None,"usage":{"input_tokens":10,"output_tokens":1}}).encode()
        self.send_response(200);self.send_header("Content-Type","application/json");self.send_header("Content-Length",str(len(payload)));self.end_headers();self.wfile.write(payload)
    def log_message(self,*args):pass
upstream=ThreadingHTTPServer(("127.0.0.1",0),Gateway)
ut=threading.Thread(target=upstream.serve_forever,daemon=True);ut.start()
config=ProxyConfig(mode="cache",anthropic_api_url=f"http://127.0.0.1:{upstream.server_port}",memory_enabled=True,memory_db_path=str(root/"legacy.db"),traffic_learning_enabled=True,subscription_tracking_enabled=False,cost_tracking_enabled=False,cache_enabled=False,rate_limit_enabled=False,log_requests=False,image_optimize=False,ccr_inject_tool=False,ccr_handle_responses=False,ccr_context_tracking=False,anthropic_pre_upstream_memory_context_timeout_seconds=30.0)
app=create_app(config)
sock=socket.socket();sock.bind(("127.0.0.1",0));sock.listen(128)
port=sock.getsockname()[1]
server=uvicorn.Server(uvicorn.Config(app,log_level="warning",lifespan="on",timeout_graceful_shutdown=3))
st=threading.Thread(target=server.run,kwargs={"sockets":[sock]},daemon=True);st.start()
try:
    deadline=time.monotonic()+180
    while not server.started:
        if not st.is_alive() or time.monotonic()>deadline:raise RuntimeError("real proxy startup failed")
        time.sleep(.02)
    session="headroom4442-native-project-scope"
    common={"Content-Type":"application/json","x-api-key":"local-unbilled-key","anthropic-version":"2023-06-01","x-headroom-user-id":"native-scope-user","x-headroom-session-id":session}
    client_tools=[{"name":"get_weather","description":"Weather","input_schema":{"type":"object","properties":{}}}]
    rows=[]
    cases=[("fresh_unresolved","You are helpful.",{},False),("pi_cwd",f"<cwd>\n{project_a}\n</cwd>",{},True),("warm_unresolved","You are helpful.",{},False),("pi_cwd_restored",f"<cwd>\n{project_a}\n</cwd>",{},True),("explicit_header_wins",f"<cwd>\n{project_b}\n</cwd>",{"x-headroom-project-id":"explicit-project"},True),("ordinary_user_cwd_ignored","You are helpful.",{},False)]
    for label,system,extra,expected in cases:
        body={"model":"claude-haiku-4-5","max_tokens":16,"system":system,"messages":[{"role":"user","content":f"cwd:\n<cwd>\n{project_b}\n</cwd>" if label=="ordinary_user_cwd_ignored" else "hello scope"}],"tools":client_tools}
        before=len(forwarded)
        conn=http.client.HTTPConnection("127.0.0.1",port,timeout=90)
        conn.request("POST","/v1/messages",body=json.dumps(body).encode(),headers={**common,**extra})
        response=conn.getresponse();payload=response.read();conn.close()
        assert response.status==200,(label,response.status,payload.decode())
        assert len(forwarded)==before+1,(label,"unexpected upstream request count")
        actual=forwarded[-1]["body"]
        names={tool.get("name") for tool in actual.get("tools",[])}
        assert names==({"get_weather"}|MEMORY_TOOL_NAMES if expected else {"get_weather"}),(label,names)
        scope=ProjectResolver().resolve(RequestContext(headers={**common,**extra},system_prompt=extract_system_prompt(body),base_user_id="native-scope-user"))
        rows.append({"case":label,"status":response.status,"memoryTools":sorted(names & MEMORY_TOOL_NAMES),"project":scope,"gatewayPath":forwarded[-1]["path"]})
    assert forwarded[1]["body"]["tools"]==forwarded[3]["body"]["tools"],"restored definitions changed"
    tracker=get_session_tool_tracker()
    assert tracker.should_inject("anthropic",session),"probe did not exercise an actual recorded sticky session"
    from urllib.parse import quote
    key_a=ProjectResolver().resolve(RequestContext(headers={"x-headroom-cwd":quote(str(project_a),safe="/")},system_prompt="",base_user_id="native-scope-user"))[0]
    assert rows[1]["project"][0]==key_a
    assert rows[4]["project"][1]=="explicit-project"
    router=app.state.proxy.memory_handler._router
    assert router is not None,"real memory backend never initialized"
    opened=router.open_backends()
    project_db=root/"memories"/"projects"/key_a/"memory.db"
    assert project_db in opened and project_db.is_file(),("real pi project database not opened",[str(x) for x in opened])
    print(json.dumps({"mode":config.mode,"realHttpListener":f"http://127.0.0.1:{port}","localGateway":f"http://127.0.0.1:{upstream.server_port}","rows":rows,"realStickyGoldenSession":True,"realPiProjectDatabase":str(project_db),"openedDatabases":[str(x) for x in opened],"nativeModule":core.__file__,"nativeSha256":native_sha,"noLivePiOrPaidProvider":True},indent=2))
finally:
    server.should_exit=True;st.join(20)
    upstream.shutdown();upstream.server_close();ut.join(5);sock.close()
    assert not st.is_alive() and not ut.is_alive(),"native proof leaked a server"


PY
```

</details>

- Observed result: unchanged `main` fc1ba891 fails at the first
unresolved request: five proxy memory tools reach upstream. With the
fix, all six requests return HTTP 200:

| Same-session request | Proxy memory tools |
| --- | ---: |
| Fresh, no cwd | 0 |
| pi system cwd | 5 |
| Cwd absent again | 0 |
| Same pi cwd restored | 5 |
| Explicit project header, different system cwd | 5 |
| Cwd block only in ordinary user text | 0 |

The client weather tool stays present. Restored definitions are
structurally equal to the first resolved turn; the explicit header wins.
The literal-percent cwd agrees with its encoded-header form, and the
actual project SQLite database opens at the expected key.

- Not tested: live pi/Caveman, paid-provider cache accounting/billing,
Windows, or other provider protocols. This does not claim to fix Caveman
header forwarding.

## Runtime Rollout Safety

- Rollout-managed feature(s): none added; existing Anthropic memory
injection.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: pi cwd resolves; unresolved
project/empty scope no longer adds unusable proxy memory tools.
- Kill switch / disable path: existing `--no-memory`; explicit
project/cwd headers still work.
- Unsafe override required: no.
- Qualification impact: no storage migration or new dependency; scoped
HTTP/SQLite behavior checked.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — related
suite and critical gate, not a whole-suite claim
- [x] I did **not** edit `CHANGELOG.md` — release-please owns it

## Screenshots (if applicable)

N/A; the observable surface is the HTTP payload and project database.

## Additional Notes

AI-assisted; no subagents. OpenAI injection policy is unchanged. Keeping
this draft until the current hosted checks are qualified.

One earlier post-build probe completed the requests and closed both
listeners but hit its whole-process shutdown deadline. The same program
and native artifact later exited normally with faulthandler enabled; its
diagnostic abort timer did not fire. I have not identified the
intermittent cause and am not claiming a shutdown fix. No production
timeout or feature flag was changed to get that result.


The two npm audits now pass on 5ace63e2, along with the other Security
jobs. The earlier [unchanged-base
failure](https://github.com/headroomlabs-ai/headroom/actions/runs/41399323584)
is historical evidence, not a current blocker. This PR changes no
dependency manifests, lockfiles, workflow or audit exceptions.


### Maintainer current-main requalification

Exact head `60597623e00967153d947a46bf197950250f310e`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Clean mechanical update from current upstream main, preserving the pi
system-cwd resolver, request-scoped project guard, main's cache-mode
search skip/logging, and client-tool preservation. Ordinary user cwd
blocks still do not enable the pi fallback; unresolved project scope
still blocks both context injection and replayed memory tools. No
contributor production logic was rewritten and no new production finding
remains in this reviewed scope.

The initial combined tree dropped main's memory-invariant file and two
stream/timeout regression cases. Maintainer test-only repairs restore
those tests rather than reducing coverage: the invariant file is
byte-identical to main, every main assertion in the four
restored/updated test files is preserved by AST comparison, and three
fixtures now supply the handler protocol methods needed for resolved
scope and beta headers. The lifecycle main control passes without that
fixture extension; no claim is made that this particular fixture failure
independently fails main. All 222 affected
scope/invariant/stream/timeout/lifecycle/backpressure/cache-stability/system-prompt
tests pass with one skip. Ten shared Gemini request-fixture cases also
pass after the separate test-only repair for independently verified main
failures. All 12 main-relative changed Python files pass Ruff/format;
whitespace checks pass.

The author's existing real TCP/SQLite/ONNX reproduction and explicit
limits were read and remain in the body. These fresh Windows checks
qualify the combined request-scope regressions; they are not a claim of
rerunning that Linux proof, live pi/client/provider behavior, or a fresh
native build. An existing binary was supplied solely for test startup.
Draft status is preserved; readiness remains the author's decision and
nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `60e01cf7c25e90abb9f5cb66fa221da83e7c5177`, upstream main
`ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`.

Clean integration with upstream/main
ea59ac3eef. Re-reviewed project-scoping
delta against intervening memory/backend/stream changes: pi cwd remains
limited to system context; ordinary user cwd blocks do not enable
fallback; unresolved projects skip context/tool injection and sticky
tool replay while retaining client tools and main cache-mode no-search
behavior.232
scope/invariant/stream/timeout/lifecycle/backpressure/cache/system-prompt/Gemini
tests pass,1optional skip; touched-source Ruff and whitespace pass. No
contributor production policy rewritten. Earlier author TCP/SQLite/ONNX
evidence remains scoped, with no fresh native build or live-pi/provider
claim. Existing draft preserved; final approval awaits exact-head hosted
qualification. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `e93ceb35df7f78bfa2dcc0611609131cf2b08b0a`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 232 passed, 1 skipped, 1 warning in
47.62s. Selected real dependency overlays and native DLL paths are
recorded in final4618-fresh-native-tests.log; the DLL provenance/input
hashes are in final-native-artifact-proof.json. No redundant rerun is
implied after this Cargo-only merge. Whitespace passes. Prior source
review and scoped real-behavior evidence remain applicable to unchanged
contributor inputs.

No new source finding in the reviewed scope. Final formal approval
awaits exact-head hosted qualification. Project resolution and
unresolved-scope injection/replay gates remain unchanged. Existing draft
status is preserved.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `d693dd2b6e33fd47ff853f7e28d938fc1dbc30ac`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Current-main requalification
against2b0315b2d4b0d54947050bbdad2de9e89df06378. Clean source
integration preserves pi systemcwd resolution and unresolved-scope
context/tool/sticky-replay gates.70integration patches exactly matchmain
delta;Anthropic exception patch differs only byclosing-parenthesis diff
alignment and its exception AST equalsmain. Broad
memory/router/projectscope/continuation/backpressure/cache/lifecycle/doctor/nativeops
run:405passed,1skipped,one newmain fake-handler fixture failed
missingis_project_unresolved. Maintainer adds only that fake method
returningFalse,respecting resolvedfixture scope;all39backpressure tests
nowpass,including unchanged
injectedSQLiteexception,HTTP200andsemaphorerelease assertions. No
production logic changed.33changedPythonlint/format,offline
lock/whitespacepass;fixedtestlint/formatrechecked.
Existingpr3982nativeDLLverified fullinputhashes/checksum;no
freshbuildclaimed. Currentauthor/Devin discussion checked,inline
endpoint empty;prior scoped realbehavior evidence remains applicable to
unchanged contributor inputs. No new sourcefinding;no
realprovider/agentlaunch claim. Draftpreserved.
Exactnewheadhostedqualification pending;maintainers
handlechecks,nothingmerged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `0da0b5a01958eaaac1d625227cb4bdb9477c5d3b`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main delta after diff-location
normalization. Contributor pi system-cwd resolution and
unresolved-project memory context/tool/sticky-replay gates are
unchanged. Fresh
memory-router/project-scope/streaming/continuation/backpressure/cache/lifecycle
and incoming Vertex/backend/CLI tests: 381 passed, 12 skipped, 1
warning. Twelve changed Python files pass Ruff lint/format; offline lock
and whitespace pass.

Static production/test scope and security review considered user versus
system cwd inputs, malformed/empty cwd, missing scope and stale tool
replay: tests keep pi cwd tags in ordinary user content from resolving
project scope, and unresolved scope blocks replay. No unrelated
executable, workflow, dependency or setup changes. Flagged inherited
workflow-validation job is excluded by unchanged workflow paths. Author
discussion checked; collected inline endpoint has no findings. Prior
scoped behavior evidence applies to unchanged contributor inputs; no new
real agent/provider launch claimed. Native DLL reused only after
complete tracked input-set/hash and artifact checksum verification, no
fresh build claimed. Draft preserved. Formal approval awaits exact
published-head hosted qualification. Maintainers handle checks; nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-10 22:28:01 -06:00
M Barry a2bcbd101b feat(pipeline): say whose request an OutcomeSnapshot is (#4698)
## Description

`OutcomeSnapshot` is the one public seam a pipeline extension gets at
`OUTCOME_OBSERVED`, but it says nothing about *whose* request it
measured. An extension that wants to split what it observes (an
exporter, a chargeback job, a budget alert) cannot tell projects,
clients or conversations apart without patching core. This adds four
read-only fields and fills them from values `emit_request_outcome`
already has:

- `attribution`: `{dimension: label}`. For now `project` and `client`,
from the same values the savings tracker books, and only when known.
- `conversation_key`: the handler's key, already on `RequestOutcome`.
- `original_tokens`: the pre-compression size, on the same tokenizer as
`tokens_saved`.
- `output_tokens_saved`: the output-shaping estimate this request
booked.

`attribution` is frozen into a read-only `dict` subclass
(`FrozenLabels`) in `__post_init__`. The dataclass is already frozen,
but a frozen field can still hold a mutable dict, so without this one
extension could rewrite the labels the next one reads. A `dict` subclass
rather than a `MappingProxyType` keeps the snapshot hashable, picklable
and safe for `dataclasses.asdict`, `copy.deepcopy` and `json.dumps`, as
it was before.

It also adds `PipelineExtensionManager.add()`. A proxy extension
(`headroom.proxy_extension`) that also wants lifecycle events is
installed after the manager exists, so it cannot be passed to the
constructor. Adding the same object twice is a no-op.

This is PR 1 of the split proposed in #4676 (generic attribution
labels). It stands on its own, and the later PRs only widen what goes
into `attribution`.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/pipeline.py`: `OutcomeSnapshot.attribution`,
`conversation_key`, `original_tokens`, `output_tokens_saved`, and
`PipelineExtensionManager.add()`. All new fields have defaults, so
existing constructors are unchanged.
- `headroom/proxy/outcome.py`: `emit_request_outcome` fills the new
fields.
- Tests: the fields and their immutability, `add()` (late attach, no
duplicates) in `tests/test_pipeline_outcome_stage.py`, and end-to-end
through `emit_request_outcome` in
`tests/test_proxy/test_conversation_savings.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy tests/test_pipeline.py tests/test_pipeline_outcome_stage.py tests/test_5xx_accounting_all_providers.py
615 passed, 7 warnings in 62.72s          # upstream/main at bb8d18a5: 608 passed

$ ruff check <changed files> && ruff format --check <changed files>     # ruff 0.17.3
All checks passed!
4 files already formatted

$ mypy headroom
No errors in the changed files. The 12 errors reported are in ccr/mcp_server.py,
memory/mcp_server.py, memory/adapters/embedders.py and providers/model_metadata.py,
identical on upstream/main, from local mcp/library versions.
```

## Real Behavior Proof

- Environment: macOS, a Headroom proxy serving as a LAN gateway for
three machines.
- Exact command / steps: a superset of this change (the full #4676
proposal) has run there since 2026-11-06 with an `OUTCOME_OBSERVED`
extension reading these fields.
- Observed result: the extension reads per-request project, client and
conversation from the snapshot with no core patch.
- Not tested: this exact subset was not deployed on its own; it is
covered by the unit tests above.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no. The fields are new and additive,
and nothing reads them unless an extension is enabled.
- Kill switch / disable path: n/a (extensions are already opt-in)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- No docs change here: the fields are documented on the dataclass.
User-facing attribution docs come with PR 2 of #4676.
- `attribution` uses a mapping rather than more scalar fields so that
later dimensions (host, team, …) need no further change to this
contract.
2026-11-09 15:15:05 -06:00
bb8d18a56a feat(wrap): add headroom wrap droid for Factory Droid (#4143)
## Description

Add `headroom wrap droid` so Factory subscription traffic, including
Droid Core, can be compressed on the existing Factory plan.

Droid ignores `ANTHROPIC_BASE_URL` / `OPENAI_BASE_URL` for
Factory-served models. It honors `FACTORY_API_BASE_URL`. This wrapper
points that at the local proxy and gives the proxy a Factory upstream
(`--factory-api-url` / `FACTORY_TARGET_API_URL`, default
`https://api.factory.ai`).

With a Factory upstream configured the proxy registers:

- `POST /api/llm/a/v1/messages` — compressed with the existing Anthropic
pipeline and forwarded to `{factory}/api/llm/a/v1/messages`. A
client-sent `x-headroom-base-url` is ignored. `x-headroom-*` headers are
stripped. Operator Anthropic extra headers are withheld from the
untrusted Factory host.
- Every other `/api/*` path — forwarded verbatim to Factory, except
Hermes Studio's scoped prefixes and requests that name their own
`x-headroom-base-url`.

Nothing outside `/api/` is sent to Factory.
`select_passthrough_base_url` and `upstream_trust` are untouched, so a
leftover Codex, Claude, or Copilot config pointed at a Factory-mode
proxy keeps its own upstream and credentials. That is the deliberate
difference from https://github.com/headroomlabs-ai/headroom/pull/2352,
which is still open and which this does not close. #2352 sends every
unrouted request on a Factory-mode proxy to Factory.

`wrap droid` writes no Droid config. It threads `factory_api_url`
through the existing proxy-reuse logic. With `--no-proxy` it refuses
unless `/health` shows a Headroom proxy serving the same Factory
upstream. OpenAI-shaped `/api/llm/o/*` traffic is forwarded
uncompressed. Those routes are deferred.

This commit is `74c889fb` rebased onto current `main` (`0dd8e0b5`). The
only conflicts were `docs/content/docs/meta.json` (kept both the new
`hermes` page and `droid`) and the `--no-proxy` branch in
`headroom/cli/wrap.py` (kept upstream's OpenAI upstream check and the
Factory check).

Refs #2351

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom wrap droid` sets `FACTORY_API_BASE_URL` to the local proxy
and resolves the Factory upstream from `--factory-api-url`, then
`FACTORY_API_BASE_URL`, then `https://api.factory.ai`. Loopback,
credential, query, and fragment URLs are rejected.
- Factory inference and `/api/*` passthrough routes are registered only
when a Factory upstream is configured. The default proxy is unchanged.
- `/health` reports `factory_api_url`.
- Docs: `docs/content/docs/droid.mdx`.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Rebased tree `/home/hermes/workspace/headroom-rebase` at `0dd8e0b5`,
importing that tree via `PYTHONPATH` and the existing venv at
`/home/hermes/workspace/headroom/.venv` (Python 3.11.16).

```text
tests/test_cli/test_wrap_droid.py ............................           [ 31%]
tests/test_provider_droid_runtime.py ................................    [ 66%]
```

60 passed. Those are the wrap launch-env tests and the Factory runtime
tests.

`tests/test_proxy/test_factory_decoy_proof.py` did not complete in this
worktree. The proxy refuses to start because `headroom._core` is not
built here (`ModuleNotFoundError`, exit 78). Setting
`HEADROOM_REQUIRE_RUST_CORE=false` still did not finish within 180s. The
decoy proof (decoy upstream stays silent, Factory stand-in receives the
inference request, spoofed `x-headroom-*` headers, REST passthrough,
`--no-proxy` refusals) was written for this change and passed in the
pre-rebase review session. It was not re-run to a green result after
this rebase.

`ruff` and `mypy` were not re-run after the rebase.

## Real Behavior Proof

- Environment: Linux, Python 3.11.16, rebased commit `0dd8e0b5` on top
of `upstream/main` `8c1beb20`. No Factory token. No call to
`api.factory.ai`.
- Exact command / steps:
`PYTHONPATH=/home/hermes/workspace/headroom-rebase
/home/hermes/workspace/headroom/.venv/bin/python -m pytest
tests/test_cli/test_wrap_droid.py tests/test_provider_droid_runtime.py`
- Observed result: 60 passed.
- Not tested: a live Factory subscription, `droid` itself, the socket
decoy proof on this rebased commit (blocked locally by the missing Rust
extension), `ruff`, `mypy`, and CI on GitHub.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. Factory routes register only when
`factory_api_url` / `--factory-api-url` is set.
- Minimum rollout channel: stable. Default proxy behavior is unchanged.
- Stable/default behavior changed: no, unless a Factory upstream is
configured.
- Kill switch / disable path: do not pass `--factory-api-url` and do not
set `FACTORY_TARGET_API_URL`. `wrap droid` is opt-in.
- Unsafe override required: no.
- Qualification impact: none on the default proxy.
- Rollback path: revert this commit. No config is written into Droid.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

The pre-rebase commit `74c889fb` was approved by a fresh Claude Opus 5.5
high review with an empty blocking list. This commit is that change
rebased onto current main. The two conflict resolutions are described
above and have not had a second Opus pass.

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Does not close #2352. That PR is the prior attempt. This one is a
current-main rebuild with the catch-all leak removed: only `/api/*` is
forwarded to Factory.

akbarkanso is not a Headroom maintainer. Please do not expect this
account to merge.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4143"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `e28dfe6422dfcb9b2cbd5ce0fe673196d3b6ad40`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Updated against upstream/main e4a60ed7. Resolved the OpenAI passthrough
overlap by retaining main's prepared-request/model/timing parameters and
soft-passthrough guard alongside the contributor's verbatim flag;
generic compression requires both guards to permit it. Re-reviewed
Factory direct-routing/verbatim semantics and existing discussion. All
93 wrapper/runtime/Factory passthrough/socket-decoy tests pass with
existing native extension; changed-file Ruff lint/format and whitespace
checks pass. No contributor routing policy was rewritten.

Independently reran the official Droid1.3.0 CLI with isolated home,
dummy API key, actual Headroom/Uvicorn proxy and controlled LAN HTTP
upstream on this exact combined head. The CLI consumed the controlled
Anthropic SSE reply and exited zero with HEADROOM_QUALIFICATION_OK. The
message request preserved dummy Authorization and forwarded no
x-headroom headers. This is controlled transport/runtime qualification,
not a live Factory subscription or savings benchmark, and not a fresh
native build. Applied shared test-only Gemini fixture repair for
independently established main failures; all ten cases pass. Formal
approval awaits exact-head hosted qualification.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `7dd0725a9ac4dd8103dedf78552ceca886fc3620`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical merge with upstream main
c01cf07d94. Re-reviewed current
Factory/Droid routing and author/Devin discussion. Factory inference
retains force_direct to bypass translating backends; Factory passthrough
retains verbatim alongside main's soft-passthrough/prepared-request
guards. Hermes scoped proxy prefixes remain excluded from Factory
dispatch; the existing standalone environment precedence is
intentionally consistent with sibling targets, while Click/wrap retain
flag precedence. No contributor routing policy rewritten.

Fresh wrapper/runtime/Factory passthrough/socket-decoy/Hermes
verification: 100 passed, with test-client/runpy warnings only. All 14
changed Python files pass Ruff lint/format; main-relative whitespace
clean and owned tree clean. Tests use the current-main native DLL for
local startup. Earlier official Droid 1.3.0 controlled-upstream runtime
proof remains scoped to its original run; no fresh official CLI replay,
Factory subscription inference, savings benchmark or branch-native build
is claimed. No new blocker found; approval awaits exact-head hosted
checks. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-07 20:48:53 -06:00
Abhay SinghandJD Davis 861ef95620 feat(proxy/model-router): add shadow mode (HEADROOM_MODEL_ROUTER_LOG_ONLY) (#2395)
## Description

Adds a shadow mode to the cost-aware model router (#1706) so operators
can validate their routes against real traffic before switching routing
on for real.

Enabling routing today immediately rewrites the upstream model. That is
a big first step to take on faith: an operator wants to see which
requests each rule would catch (and confirm nothing important gets
downgraded) before letting it change live traffic. Shadow mode provides
exactly that dry run.

## How it works

Set `HEADROOM_MODEL_ROUTER_LOG_ONLY=1` (alongside the usual
`HEADROOM_MODEL_ROUTER_ENABLED` / `HEADROOM_MODEL_ROUTES`). Rules are
evaluated normally and the intended route is logged, but the outgoing
model is left unchanged.

It is implemented entirely inside the pure `ModelRouter.select()` and
needs no handler changes: in `log_only` mode a matched rule returns a
`ModelDecision` whose `routed_model` equals the original model, so
`changed` is `False` and every caller (the Anthropic and OpenAI routing
helpers) already leaves the model untouched. The decision `reason`
carries the shadow intent, e.g.

```
[shadow] would route gpt-5.4 -> gpt-5.4-mini via rule 'small' (input_tokens=1200, has_tools=False); log_only, model unchanged
```

which the handlers already log at INFO, so the would-be routing is fully
observable.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/model_router.py`: add `ModelRouterConfig.log_only`
(parsed from `HEADROOM_MODEL_ROUTER_LOG_ONLY` in `from_env`), a
`ModelRouter.log_only` property, and the shadow branch in `select()`
that returns a matched-but-unchanged decision with a `[shadow]` reason.
- `headroom/proxy/server.py`: pass `HEADROOM_MODEL_ROUTER_LOG_ONLY` into
`from_env`.
- `tests/test_proxy/test_model_router.py`: cover env parsing, the
default-off case, that a matched rule leaves the model unchanged in
shadow mode (with the shadow reason), and that shadow off still routes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/proxy/model_router.py headroom/proxy/server.py tests/test_proxy/test_model_router.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/model_router.py
Success: no issues found in 1 source file
# model_router is a pure module, so I ran its test file against the real code
# in the project venv (uv sync): 33 no-arg tests pass, including the new
# shadow-mode ones.
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`.
- Exact command / steps: built a config via
`ModelRouterConfig.from_env("1", "[{...}]", "true")` and called
`ModelRouter.select(...)` on a request a rule matches, plus the same
with `log_only` off.
- Observed result: in shadow mode the decision is `matched=True`,
`changed=False`, `routed_model` equals the original, and `reason` starts
with `[shadow] would route strong -> cheap`; with shadow off the same
request is routed (`changed=True`, `routed_model="cheap"`); `log_only`
defaults to `False`. Because `model_router` has no heavy imports, this
ran against the actual module, not a replica.
- Not tested: a live proxy request in shadow mode; the router is pure
and the handlers already consume `ModelDecision`/log its reason
unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

`model_router` is a pure component, so I ran its unit tests against the
real code in the venv (33 pass). No handler changes are required: shadow
mode works by returning a non-changing decision, which the existing
Anthropic and OpenAI routing helpers already handle. Disabled by
default; only takes effect when routing is enabled and
`HEADROOM_MODEL_ROUTER_LOG_ONLY` is truthy.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2395"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `aff3be8e38c361809d98ca19414bfb019c265cef`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean current-main0ddaa56c integration; contributor ModelRouter.select
and ModelRouterConfig.from_env ASTs unchanged. Main-relative scope
remains three intended source/test files. Fresh
validation349passed,11skipped,1warning:60router/wiring tests
plus289incoming-main regressions. Three changed Python files pass
Rufflint/format,candidate offline lock and whitespace pass. Complete
native tracked-input set/all recorded hashes/artifact checksum verified
before3982DLL reuse; no freshbuild.

Fresh runtime proof closes the outstanding select-only evidence gap:
real local proxy TCP/ASGI lifespan forwards to loopback HTTP upstream
with synthetic auth/model/content. From canonical environment-factory
config, enabled+shadow forwards original claude-sonnet-4-6 and logs
exactly one '[shadow] would route ... -> claude-haiku-4-5'; active
forwards claude-haiku-4-5 and logs one matched route; disabled forwards
original and emits no routing decision. All three HTTP200. Each proxy
child stopped and upstream server closed. These are actual local
forwarding/log observations, not cloud/provider-account,
model-inference, billing or CLI-env parsing proof. Main's separate CLI
construction was not changed; do not infer CLI env wiring from
environment-factory coverage. Synthetic fixture/log artifacts preserved
in pr2395-0dd-runtime-proof.json and associated scripts/logs.

Read author/other-human discussion and current Devin runtime-proof note.
Current fresh REST audit contains no new external findings. Only three
reviewed source/test files differ from main; no
workflow/setup/dependency/native/CLI delta, central publication gate
passes. Contributor shadow semantics/defaults preserved. Exact
published-head CI/Actions/rollup,main ancestry,mergeability and fresh
discussion remain before approval. Maintainers handle CI; no PR merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


<!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 -->

Maintainer current-main refresh: exact head
`41a0f5b920f48f669e6e8ea0a85593c02d6622c0`, authoritative main
`12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches
exactly match incoming main; contributor behavior preserved. Fresh
focused source tests plus tokenizer/response-header/HF/JSON-parser
incoming-main controls: 149 passed, 1 warning in 15.46s.
Ruff/format/whitespace pass. Router tests and mocked proxy wiring are
not live provider/billing proof; shadow mode preserves the actual
requested model. Native artifact reuse verified against full recorded
input set/every hash and artifact checksum; no fresh native build
claimed. Static publication gate excludes inherited
workflow-validation/native Docker/Init/Wrap installer paths, with no
workflow/setup/dependency/native/CLI changes in PR diff. Actual
exact-head hosted workflows and final independent review remain pending.
No merge performed.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-07 20:41:30 -06:00
Aleksei TcelishchevandJD Davis f896161340 feat(proxy): unified multi-upstream routing for Anthropic Cloud and local oMLX (#4597)
## Description

Adds unified multi-upstream routing for Anthropic Cloud and local Apple
Silicon MLX (`oMLX`) inference on a single Headroom proxy instance, with
strict allowlist-based local model selection, post-rewrite destination
resolution, cloud credential isolation, and trusted upstream
extra-header support.

## Why (Rationale & Problem)

Developers running hybrid cloud + local Apple Silicon MLX workflows
(`oMLX` / `jundot/omlx` serving `/v1/messages` alongside Anthropic
Cloud) previously had to run separate Headroom proxy instances or
reconfigure upstream URLs when switching between Claude and local `oMLX`
models.

## What For (Use Cases & Value)

Allows a single Headroom proxy instance to dynamically route explicit
local `omlx/*` / `local/*` / MLX model requests (or requests to
`/omlx/v1/messages`) to a local `oMLX` server (`HEADROOM_OMLX_BASE_URL`
/ `OMLX_TARGET_API_URL`) while keeping all Claude models
(`claude-haiku-4-5-20251001`, `claude-haiku-5-5`, `claude-fable-5`,
`claude-sonnet-4-5`, etc.) and unmatched models on `api.anthropic.com`
with full context compression and telemetry.

## How (Implementation, Invariants & Verification)

- **`headroom/proxy/handlers/anthropic.py`**:
- `is_omlx_local_model`: Only routes to `omlx_target` when the model
explicitly matches local model prefixes (`omlx/`, `mlx/`, `local/`,
`mlx-community/`, `gemma`, `qwen`, `llama`, `mistral`, `deepseek`,
`phi`, `lfm`, `smollm`, `starcoder`, `codellama`, `yi-`, `nemotron`) or
the operator-configured `HEADROOM_OMLX_MODELS` allowlist. Never routes
`claude*` models (unless explicitly prefixed with `omlx/` or `local/`)
to oMLX.
- Evaluates `_maybe_route_model` **before** local/cloud target selection
so cloud-to-local and local-to-cloud model rewrites select the matching
destination URL and credential policy.
- Strips leading `omlx/` or `local/` routing prefixes from
`body["model"]` (recording `omlx_strip_model_prefix` on
`body_mutation_tracker`) before forwarding to `oMLX`'s native
`/v1/messages` endpoint.
- Strips inbound Anthropic cloud credentials (`x-api-key` and
`authorization`) when forwarding to local oMLX so cloud tokens never
leak to the local process, optionally substituting
`HEADROOM_OMLX_API_KEY` / `OMLX_API_KEY` when configured.
- **`headroom/config.py`, `headroom/proxy/models.py`,
`headroom/cli/proxy.py`, `headroom/proxy/server.py`,
`headroom/providers/proxy_routes.py`**:
- Consistent resolution of `OMLX_TARGET_API_URL`,
`HEADROOM_OMLX_BASE_URL`, and `--omlx-api-url` across SDK config,
`ProxyConfig`, Click CLI, and server factory startup.
- **`headroom/proxy/upstream_trust.py`**:
- Includes `omlx_target_api_url` in `trusted_upstream_hosts` so
operator-configured `anthropic_extra_headers` are permitted for the
designated oMLX gateway.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `resolve_omlx_target_api_url` in `headroom/config.py` and wired
`--omlx-api-url` / `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL`
across `headroom/cli/proxy.py`, `headroom/proxy/models.py`,
`headroom/proxy/server.py`, and `headroom/providers/proxy_routes.py`.
- Replaced open-ended `cloud_prefixes` fallback in
`headroom/proxy/handlers/anthropic.py` with explicit local-prefix +
`HEADROOM_OMLX_MODELS` allowlist matching (`is_omlx_local_model`) and
`strip_omlx_model_prefix`.
- Moved local/cloud target resolution after `_maybe_route_model` and
stripped Anthropic cloud credentials (`x-api-key`, `authorization`) on
local oMLX routes.
- Added `omlx_target_api_url` to `_API_URL_ATTRS` and
`trusted_upstream_hosts` in `headroom/proxy/upstream_trust.py`.
- Added 42 unit tests in `tests/test_proxy/test_omlx_routing.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/pytest tests/test_proxy/test_omlx_routing.py tests/test_proxy/test_model_router_wiring.py tests/test_proxy/test_anthropic_upstream_header.py tests/test_upstream_guard.py -v
121 passed in 91.28s

$ .venv/bin/ruff check headroom/config.py headroom/proxy/models.py headroom/proxy/upstream_trust.py headroom/providers/proxy_routes.py headroom/cli/proxy.py headroom/proxy/server.py headroom/proxy/handlers/anthropic.py tests/test_proxy/test_omlx_routing.py
All checks passed!

$ .venv/bin/mypy headroom/config.py headroom/proxy/models.py headroom/proxy/upstream_trust.py headroom/providers/proxy_routes.py headroom/cli/proxy.py
Success: no issues found in 5 source files
```

## Live Verification & Proof (CONTRIBUTING.md)

- **Setup**:
- macOS Apple Silicon (M4 Max), Python 3.14.10 in `.venv`, Headroom
proxy configured with `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888` and
`ANTHROPIC_API_KEY=sk-ant-api03-...`.
- **Commands**:
1. `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888 .venv/bin/pytest
tests/test_proxy/test_omlx_routing.py -v`
2. POST `/v1/messages` with `model="claude-haiku-4-5-20251001"`,
`model="claude-haiku-5-5"`, `model="claude-fable-5"`, and
`model="claude-sonnet-4-5"` while
`HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888` is active.
3. POST `/v1/messages` with `model="omlx/qwen3-8b"` and
`headers={"x-api-key": "sk-ant-cloud-secret"}`.
- **Observed results**:
- All 33 supported Claude model IDs (including
`claude-haiku-4-5-20251001`, `claude-haiku-5-5`, and `claude-fable-5`)
routed to `https://api.anthropic.com/v1/messages` with cloud credentials
preserved.
- `model="omlx/qwen3-8b"` routed to `http://127.0.0.1:8888/v1/messages`
with `body["model"]` normalized to `"qwen3-8b"` and inbound Anthropic
`x-api-key` / `Authorization` headers stripped (or replaced by
`HEADROOM_OMLX_API_KEY` when set).
- **Untested cases**:
- Multi-node remote oMLX cluster failover (single target URL per proxy
instance is supported).

## Real Behavior Proof

- Environment: macOS Darwin arm64 (Apple Silicon), Python 3.14.10,
`HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888`
- Exact command / steps: `.venv/bin/pytest
tests/test_proxy/test_omlx_routing.py -v`
- Observed result: 42 passed in 15.28s — verified cloud routing for all
Claude models, local routing and prefix stripping for `omlx/*` /
`local/*` / `gemma*` / `qwen*`, pre-target evaluation of
`_maybe_route_model`, cloud credential stripping, and
`trusted_upstream_hosts` inclusion.
- Not tested: Windows named-pipe transport with local oMLX.

## Runtime Rollout Safety

- Rollout-managed feature(s): None (opt-in via `HEADROOM_OMLX_BASE_URL`
/ `OMLX_TARGET_API_URL` / `--omlx-api-url`).
- Minimum rollout channel: stable (no-op unless operator sets
`HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL`).
- Stable/default behavior changed: No — when `omlx_target_api_url` is
unset, `/v1/messages` routing is 100% unchanged.
- Kill switch / disable path: Unset `HEADROOM_OMLX_BASE_URL` and
`OMLX_TARGET_API_URL`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Unset `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL`
or revert commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)


### Maintainer re-review of the credential-isolation update

Reviewed updated author head 5c45c8f964,
already based on upstream main c01cf07d94.
The previous semantic merge hold no longer applies: the contributor
rebased and integrated destination-aware header construction with main's
earlier enforcement flow. Configured x-api-key/Authorization are
filtered from local routes without overriding the local key; cloud
routes retain configured headers, and cloud subscription notification is
suppressed for local targets. Current discussion and prior Devin
findings considered; scope remains Anthropic Messages-compatible oMLX.

Fresh routing/backpressure/rate-limit/subscription verification: 134
passed, one skip, deprecation warnings only. All eight changed Python
files pass Ruff lint/format and main-relative whitespace. A
diagnostic-only external control restoring unfiltered extra-header
selection makes all eight requested credential-isolation regressions
fail; this does not modify the PR or claim an old-head test execution.

Independent actual Uvicorn/TCP forwarding with controlled local
Anthropic-compatible upstreams passed 12 cases: eight local
credential-isolation combinations plus four cloud-retention controls.
Dummy credentials only; subscription polling disabled for this
header-only probe. No real oMLX model/server, cloud inference, or
branch-specific native build claimed; current-main DLL supports local
startup. The prior configured-header and current-main integration
findings are addressed. No new source blocker found; formal approval
awaits exact-head hosted qualification. No contributor code changed and
nothing merged.



### Maintainer current-main requalification

Exact head `1b9082fb4baf4cb2cf8984e241fdae77a4494e83`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Requalified against upstream main
2b0315b2d4. All 71 integration patches
match the main delta after diff-location normalization; the
contributor's destination-aware header policy is unchanged. 266 routing,
backpressure, rate-limit, subscription and doctor tests pass, with one
skip and three deprecation warnings. Twelve fresh Uvicorn/TCP cases
pass: eight local credential-isolation combinations and four
cloud-retention controls. These use dummy credentials and local
Anthropic-compatible upstream fixtures, with subscription polling
disabled. They do not qualify a real oMLX model/server or cloud
inference.

Ruff lint/format passes for all 33 changed Python files; offline lock
and whitespace checks pass. The existing native DLL matches the recorded
build-input hashes and artifact checksum; no fresh build is claimed.
Current author and Devin discussions were considered, and the prior
credential-isolation finding remains addressed.

A direct fork push was denied with HTTP 403. GitHub's PR branch-update
API applied the clean main merge instead. Its published tree,
116abaccb5, exactly matches the tested
candidate's complete tree. The test evidence therefore applies to this
head. Approval awaits exact-head hosted checks; maintainers handle CI.
No PR was merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-07 20:40:09 -06:00
alan blountandJD Davis 0ddaa56c21 fix(vertex): explain Vertex failures in-band, correct the model/region matrix (#2640)
## Description

Vertex rejects onboarding requests for a handful of recurring reasons —
expired ADC, a partner model never enabled in Model Garden, a model not
served in the requested location, a missing optional SDK — and its own
error text names none of the fixes. Users see a bare `404` or an opaque
`500` through their SDK and cannot tell a proxy bug from a
project-config gap.

This makes the proxy explain those failures where it happens, adds a
`[vertex]` extra so the LiteLLM Vertex backend can actually start, and
corrects an example, a test matrix, and three documentation claims that
could not have worked as written.

Closes #2632

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

**Explain Vertex failures in-band**
(`headroom/providers/vertex/diagnostics.py`, new)

- On `401/403/404/429`, attach an actionable hint three ways: a WARNING
in the proxy log, an `x-headroom-hint` response header, and a note
appended to `error.message`. The last is load-bearing — SDKs raise with
the message string and discard headers.
- Successful responses are never modified; annotation is idempotent;
statuses we cannot explain get no hint rather than a plausible-sounding
wrong one.
- Routes opt in via a one-line `@with_vertex_diagnostics` decorator
reading path params FastAPI already injects (**+8 lines** in
`proxy_routes.py`, no re-indentation).
- Same treatment for backend setup failures, where `--backend vertex`
surfaced LiteLLM's raw `No module named 'vertexai'` as an opaque 500.

**Make `--backend vertex` able to start** (`pyproject.toml`,
`registry.py`, `cli/proxy.py`)

- New `[vertex]` extra carrying `google-cloud-aiplatform>=1.38`.
Deliberately **not** folded into `[proxy]`: measured at ~175 MB
installed, and the native passthrough routes never needed it. Follows
the existing `proxy-prod` split.
- Startup preflight keyed on the **resolved provider**, so every alias
normalized at `registry.py:229` (`vertex`, `vertex_ai`,
`litellm-vertex`, `google-vertex`, `googlevertex`) is covered. Skipped
when a backend class is injected, so existing fake-based tests are
unaffected.
- New `BackendUnavailableError` so the CLI prints the remedy and exits
`2` instead of dumping a traceback through startup internals.

**Correct things that could not have worked**

| | was | is |
|---|---|---|
| SDK client | `http_options={"api_endpoint": ...}` | `{"base_url":
"http://..."}` — `api_endpoint` is not a field on `HttpOptions`;
pydantic rejects it before any request is sent |
| Thinking config | `ThinkingConfig(thinking_budget_tokens=)` |
`thinking_budget=` |
| Request payload | `contents: [{parts: [...]}]` | adds `role: "user"` —
the `global` endpoint enforces it, regional was silently lenient |
| Auth guidance | `gcloud auth print-access-token` |
`application-default` — the plain user token gets `401
ACCESS_TOKEN_TYPE_UNSUPPORTED` |
| Test matrix | model × location cross-product | explicit servable pairs
|

- Test matrix is now four pairs that can actually serve:
`global`/`gemini-flash-latest`, `global`/`claude-sonnet-4-6`,
`europe-west2`/`gemini-3.5-flash`, `us-east5`/`claude-sonnet-4-6`.
Gemini 3.x has no US regional endpoint and `-latest` aliases are
global-only, so a pinned `us-central1` Gemini case can never pass. No
Gemini 2.5 — it retires 2026-10-16.
- `test_thinking_levels` asserted only `status_code < 500` while sending
an ignored `thinkingBudgetTokens` key: it passed on 400s and proved
nothing. Now asserts 200 plus scored usage metadata with the correct
`thinkingBudget` spelling.
- Cases the caller's project genuinely cannot serve now **skip** with an
actionable reason instead of reporting a red proxy.

**Docs** (`proxy.mdx`, `installation.mdx`, `wiki/vertex.md`)

- `proxy.mdx` claimed `publisher=anthropic` passthrough "routes through
the same LiteLLM-Vertex path as `--backend vertex_ai`". It does not —
without the flag it is a direct forward, which is why it works on a bare
`[proxy]` install where the LiteLLM path cannot even import.
- Documented the trap of combining `--backend vertex` with native Vertex
URLs: LiteLLM takes project/region/model from its own config and ignores
the URL.
- `installation.mdx` and `wiki/vertex.md` said the SDK is "not included
in any extra".
- Cleared 15 pre-existing `ruff` findings in the two files this PR
rewrites (4 remain in `plugins/headroom-oauth2`, untouched by this
work).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ mypy headroom
Success: no issues found in 511 source files

$ ruff check .
Found 4 errors.        # all in plugins/headroom-oauth2, pre-existing, untouched by this PR

$ pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q
44 passed, 1 warning in 69.86s          # 8 live integration + 36 offline diagnostics, 0 skipped

$ pytest tests/ -k "anthropic_messages or handler_anthropic or backend or registry or cli_proxy or vertex or proxy_routes"
853 passed, 49 skipped, 9345 deselected, 2 warnings in 28.41s
```

New tests in `tests/test_vertex_diagnostics.py` (36, no credentials
required): status coverage, per-publisher remedy divergence,
idempotency, `content-length` consistency after body rewrite, non-JSON
and body-less/streaming responses, every backend alias preflighted,
other providers unaffected, injected backend bypasses the preflight.

## Real Behavior Proof

- Environment: macOS 14, Python 3.11.15, live GCP project
(`alanblount-sandbox`) with `aiplatform.googleapis.com` enabled and
Claude enabled in Model Garden; ADC via `gcloud auth application-default
login`
- Exact command / steps: `export GCP_PROJECT_ID=alanblount-sandbox &&
export GCP_ACCESS_TOKEN=$(gcloud auth application-default
print-access-token) && uv run --extra proxy --with google-genai python3
examples/vertex_genai_sdk_demo.py && uv run --extra proxy --with
pytest-asyncio pytest tests/test_proxy_vertex_native_integration.py
tests/test_vertex_diagnostics.py -q && headroom proxy --backend vertex`
(exits 2, then retried with `[proxy,vertex]` extra installed)
- Observed result: demo printed `✓ Standard response received
successfully! > 1, 2, 3, 4, 5` and `✓ Thinking response received
successfully!`; 44 tests passed (8 live integration, 36 diagnostics), 0
skipped; `--backend vertex` exited 2 with the remedy message; same
request with the extra installed returned 200; invalid region produced
`[headroom] hint:` in the response body and `x-headroom-hint` header;
success responses were byte-clean
- Not tested: Windows; `us`/`eu` multi-region endpoints and Claude 4.7+
(project cannot serve them); Provisioned Throughput; Rust proxy path;
`--backend vertex` dropping URL project/model on native Vertex routes
(documented as known gap in Additional Notes)


## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — proxy and CLI behavior; terminal output is quoted above.

## Additional Notes

- `uv.lock` intentionally not included: the new extra is opt-in and does
not change default resolution.
- Error-body rewriting is confined to already-failing responses and
wrapped so a diagnostic can never turn a clean upstream error into a
proxy 500.
- The preflight raises where the provider is resolved (`registry.py`)
rather than at the CLI, so programmatic embedders get the same
guarantee; the CLI only formats it.
- Three test files fail to collect on this branch
(`test_hermes_passthrough_compression`, `test_observability_metrics`,
`test_observability_tracing`). They predate this work and are unrelated,
so they were excluded rather than fixed — happy to open a separate
issue.
- `[vertex]` was chosen over adding the dependency to `[proxy]` after
measuring the footprint at ~175 MB. If maintainers prefer it in
`[proxy]`, that is a one-line change.
## Runtime Rollout Safety

- Rollout-managed feature(s): none; diagnostics annotate already-failing
Vertex responses and the backend dependency is opt-in through the vertex
extra.
- Minimum rollout channel: normal stable release after Vertex
diagnostics, route, and backend-path tests pass.
- Stable/default behavior changed: successful responses and native
passthrough remain unchanged; supported Vertex failures gain an
actionable header/message, and selecting the LiteLLM Vertex backend
without its SDK now fails once at startup.
- Kill switch / disable path: use native Vertex passthrough without the
LiteLLM backend, or human-revert the diagnostic/preflight change.
- Unsafe override required: none.
- Qualification impact: offline diagnostics, native integration, and
mocked LiteLLM Vertex request-shaping tests must remain green.
- Rollback path: human-revert the PR; no persisted state or migration is
involved.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2640"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `5d7071f348d2462f666fd7ac1ceb8ac18050169a`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Re-reviewed latest integration head c70c1a497 and current author
responses, including fixed public error hints, log escaping, body/header
consistency, all four backend error sites, and native-route versus
LiteLLM SDK dependency behavior. Clean integration with upstream/main
2d3ee51846. No new PR-specific source
blocker found in reviewed scope; no contributor diagnostic/routing logic
rewritten.

232 Vertex diagnostics/native-route/backend/CLI/release/version/MCP
tests pass with11 skips. Offline lock validation passes (268 packages);
all12 changed Python files pass Ruff check/format, and whitespace
passes. These checks cover synthetic error/transport paths and do not
establish live cloud regional availability or billing behavior. Earlier
author live evidence remains scoped to its stated environment; no fresh
cloud request or native build claimed here. Historical shared
cached-read fixture failure was main-reproduced and is not a
contributor-code blocker. Maintainers handle current exact-head hosted
qualification before approval; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `486a716adbf642ae5e43276df4c2dc1e63b901d7`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical sync with upstream main
c01cf07d94. Re-reviewed current
author/Devin discussion and public diagnostic/error boundaries,
native-versus-SDK route separation and startup preflight. This sync does
not alter the PR Vertex diagnostics, backend or route-registry
modules.158 focused diagnostics/native-route/backend/streaming/CLI tests
pass,11live-environment skips,1deprecation warning. Offline uv lock
--check passes;all12 changed Python files pass Ruff lint/format and
main-relative whitespace passes. Fixed hints remain separate from raw
provider error text; multi-worker preflight and non-Google partner
guidance retain resolved regressions. No contributor diagnostic/routing
correctness logic rewritten. Tests use existing current-main DLL and
synthetic errors/transports; no fresh native build, live regional
availability/provider inference or billing qualification claimed. Formal
approval awaits exact-head hosted qualification; maintainers handle CI.
Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `0b75c609f24562c8b6ed2aaeaf58239984b4810c`, upstream main
`ce4275491299231bc9d2907833f72f72ea22b8a9`.

Requalified after human merges #4667,#4669,#4670 against upstream main
ce42754912. Earlier approval and
publication guards stopped stale candidates before either was posted.
Current clean merge retains Vertex diagnostics/native-versus-SDK routing
and main enable_kompress checks, lossless cold-prefix disable, and
Python token-window boundary behavior.190 combined
Vertex/backend/CLI/Kompress-disable/must-keep tests
pass,11live-environment skips,1warning. Offline lock validation,12-file
Ruff lint/format and main-relative whitespace pass. No contributor
diagnostic/routing logic rewritten. Native Rust/Cargo sources are
identical to c01 across these merges; existing matching-core DLL
supports tests, not a fresh native build. No new live Vertex
regional/provider/billing qualification claimed. All prior-main hosted
results are superseded for readiness; approval awaits exact checks on
this published merge head. Maintainers handle CI; no agent merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `1fe62702ce614315679d68d9d126e31c57208c67`, upstream main
`00d4f441d3f1d48c463fea57e6c574c2f3671302`.

Requalified after human merges #4667,#4669,#4670 against upstream main
ce42754912. Earlier approval and
publication guards stopped stale candidates before either was posted.
Current clean merge retains Vertex diagnostics/native-versus-SDK routing
and main enable_kompress checks, lossless cold-prefix disable, and
Python token-window boundary behavior.190 combined
Vertex/backend/CLI/Kompress-disable/must-keep tests
pass,11live-environment skips,1warning. Offline lock validation,12-file
Ruff lint/format and main-relative whitespace pass. No contributor
diagnostic/routing logic rewritten. Native Rust/Cargo sources are
identical to c01 across these merges; existing matching-core DLL
supports tests, not a fresh native build. No new live Vertex
regional/provider/billing qualification claimed. All prior-main hosted
results are superseded for readiness; approval awaits exact checks on
this published merge head. Maintainers handle CI; no agent merge.

Final current-main requalification:
07d5add5ba, including human merges #4671
action pinning and #4648 multimodal memory preservation. Clean
mechanical merge; Vertex/backend/CLI inputs retain the preceding
190-test qualification. Fresh workflow/release suite: 62 passed. Fresh
memory wrapper suite: 9 passed. New wrapper retains list payload parts
and extracts textual search inputs; no contributor Vertex logic
rewritten. Existing native input provenance remains unchanged. Hosted
qualification must run on this final head before approval.

Newer human merge #4646 advanced authoritative main to
8cdce172ea before publication. Cleanly
integrated quote-aware read detection; reviewed new masking/segmentation
overlap. Fresh read-command/content-router suite: 188 passed, 1 optional
SOCKS warning. Vertex code remains unchanged by this merge.

Final additional human merge #4067 packaging changes advanced main to
a8905214d: cleanly integrated explicit torch-extras split, retaining
Vertex optional dependencies. Fresh ONNX/optional-dependency contracts:
12 passed. Offline lock check and whitespace pass. No live clean-install
or Torch backend claim. Earlier publication candidates were stopped
before push by fresh-main guard.

Latest human merge #4042 advances authoritative main to
cde584d7fe. Clean merge preserves optional
LangChain availability handling and CI dependency setup. Fresh
binding/compatibility/startup-log/xAI regression suite: 29 passed. New
main integration re-reviewed; no contributor Vertex behavior rewritten.

Subsequent #4049 main merge 00d4f441d adds graceful context lookup/init
failure handling. Cleanly integrated with Vertex handler diagnostics
intact. Fresh Anthropic pre-upstream/memory native ops/OpenAI Codex
routing/WebSocket lifecycle suite: 192 passed, 1 deprecation warning.
Reviewed exception handling and transport overlap; no live provider
qualification claim.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `6c63811d6de7491ebce9a16f89e197f5f5dd86dd`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Requalified against authoritative
main2b0315b2d4b0d54947050bbdad2de9e89df06378 after further human merges.
All10 integration patches exactly match00d4f441-to-current-main changes
after diff-location normalization; contributor Vertex
diagnostics/routing policy unchanged.539combined
Vertex/backend/CLI/Kompress-disable/must-keep/binary/doctor/Windows-supervisor/provider/search/text
tests pass;13platform/live-environment skips,2warnings. Nine changed
Python files pass Ruff lint/format; offline lock and whitespace pass.
Existing pr3982 native DLL verified against every tracked native
build-input hash and artifact checksum; no fresh native build claimed.
Current author/Devin discussion checked; prior fixes remain intact. No
live Vertex regional/provider/billing,IPv6 listener or unsupported cloud
qualification claimed. No new finding in reviewed scope. Exact-head
hosted checks must qualify this published head before formal approval;
maintainers handle them. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-06 12:27:47 -06:00
Mansurshakh JaparovandJerrettDavis 50bf775e8c fix(proxy): bypass compression for large admitted histories (#4142)
## Description

Closes #4139.

A history that is too large to compress should still reach the provider.
This keeps the 100 MiB hard admission limit and bypasses request-side
transforms above 16 MiB decoded size. The hard limit still applies
before forwarding.

Current head: `62ffe3defef1684f70d96f1640818d99445a43bb`, merging
current main `a2d4fe72` into `1902d8ef`. The
42-test/native/manual/pre-push results below qualify `1902d8ef`, not
this new merge. New-head shard 4 has a cached-prefix fixture failure
under investigation; coverage uploads are still incomplete.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
- [x] Documentation update

## Changes Made

- Keep the startup override and both wire/decompressed hard bounds.
Large admitted histories skip normalization, image processing, memory
injection, tokenization and compression on Messages, Chat/Responses,
Gemini and Bedrock HTTP routes.
- Keep provider routing and backend translation, including Copilot Chat,
Codex `store:false` and Vertex route-forced streaming. Native forwarding
retains decoded bytes unless the provider's wire contract requires a
rewrite; translation retains the full message text.
- Report actual provider usage, with zero compression savings or
attempted tokens for the bypass. Keep `/v1/compress` and WebSocket
contracts unchanged.
- Add real-provider coverage for large Gemini JSON/SSE requests with
query-key auth, and OpenAI JSON/SSE through the installed LiteLLM
adapter. No admission-policy or CI changes in this follow-up.

## Testing

- [x] Unit tests pass (focused suite and normal critical pre-push suite)
- [x] Linting passes (normal hooks)
- [x] New tests added for admission, recovery and passthrough accounting
- [x] Manual testing performed against the real native build

### Test Output

```sh
python -m pytest -q tests/test_proxy_compression_admission.py tests/test_proxy/test_gemini_plugin_routed_streaming.py --cov=headroom --cov-branch
# 42 passed, 41 warnings
```

Normal pre-push `make ci-precheck` passed: 1668 Rust tests (3 ignored),
181 Python tests, lint/type checks and commitlint. No hook bypass. The
initial deep-module coverage command hit a Pydantic import-isolation
error; using the repo's `--cov=headroom` convention passed both complete
modules without disabling plugins or tests.

The earlier `96c5fe7e` qualification covered the other native protocols,
configured/per-request backends, hard-ceiling recovery and complete
local shards 2/3. Those runs are historical, not a new whole-suite
claim. New-head hosted success is not claimed.

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.13.14, freshly built real
`headroom._core`, public Headroom ASGI routes, a real localhost HTTP
provider and installed LiteLLM. Synthetic credentials, offline mode and
`DO_NOT_TRACK=1`; no live provider calls.
- Exact command / steps: Start
`create_app(ProxyConfig(backend="litellm-openai", ...))` with all
upstream URLs set to the localhost peer and
`litellm.api_base=peer+"/v1"`. Leave the production 16 MiB soft/100 MiB
hard limits unchanged. POST `"history " * (16777216 // 8 + 1024) + "LAST
TURN"` as a user message to Chat Completions, then as a Gemini content
part to generateContent/streamGenerateContent, with streaming off/on.
The peer rejects a changed history, requires Gemini query-key auth and
one `alt=sse`, and returns protocol-valid JSON/SSE with 101 input/1
output tokens. The maintained four small-ceiling cases are in
`tests/test_proxy_compression_admission.py` and run with the command
above.
- Observed result: All four >16 MiB requests returned 200 with the full
history intact. JSON/SSE and Gemini auth survived; streamGenerateContent
forced SSE even with incoming `alt=json`. Each request recorded exactly
101 provider input/1 output tokens, zero compression attempts/savings
and no failures; `/livez` stayed 200. Earlier recorded qualification
also covered all five native HTTP protocols, real translating backends
and 413 recovery.
- Not tested: Live cloud accounts, the reporter's deployed 4 GB VPS,
screenshot/image quality or a zero-JSON-parse path. One envelope/model
JSON parse remains; no proxy-only RSS guarantee is claimed.

## Runtime Rollout Safety

- Rollout-managed feature(s): None; this is the admission/compatibility
fix.
- Minimum rollout channel: Stable/default.
- Stable/default behavior changed: Keep the 100 MiB hard ceiling; bypass
request-side transforms above 16 MiB decoded size on the named HTTP
routes.
- Kill switch / disable path: Set
`HEADROOM_REQUEST_BODY_MAX_BYTES=16777216` (or lower) and restart,
accepting local 413s instead of the large-body path.
- Unsafe override required: No.
- Qualification impact: Local native protocols, backend translation,
hard bounds and recovery exercised; live-provider/VPS qualification
remains external.
- Rollback path: Revert the admission revision and restart; no storage
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I have made corresponding changes to the documentation
- [x] I did not edit `CHANGELOG.md`

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-04 22:37:21 -06:00
Chen YufeiyangandJD Davis dfcac78119 fix(proxy): strip upstream transport framing headers from Gemini response replays (#4605)
## Description

The Gemini `generateContent` and `countTokens` response replays (both
the optimized fast path, the main path and the continuation-error
replay) copied the upstream headers by hand and only removed
`content-encoding`/`content-length`. A chunked upstream therefore leaked
`transfer-encoding: chunked` to a client that receives an
already-decoded, re-framed JSON body — the RFC 9112 wrong-framing class
this codebase already avoids elsewhere — along with `connection`,
`keep-alive` and the upstream `server` banner. Other paths in the same
file and other providers already use the shared
`sanitize_forwarded_response_headers`; these five sites were missed.

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Refactor (non-breaking change that only restructures code)
- [ ] Tests (no production code change)
- [ ] Documentation (no production code change)
- [ ] Chore (dependencies, CI, tooling, etc.)

## Changes Made

- Import and use `sanitize_forwarded_response_headers` in the five
Gemini response-replay sites instead of hand-copying headers.
- Add regressions for `generateContent` and `countTokens` that assert
`server`, `keep-alive`, `connection` and `transfer-encoding` from the
upstream do not reach the client while `content-type`, body and
`x-headroom-*` stay intact.

## Testing

- [x] Unit tests pass (`tests/test_proxy` 491 passed)
- [x] New tests added
- [x] Manual testing performed (real uvicorn proxy + real httpx against
a local fake upstream)

### Test Output

```text
pytest tests/test_proxy/test_gemini_response_framing_headers.py -q
# parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client
# this branch:     2 passed
pytest tests/test_proxy -q                                     # 491 passed
ruff check / ruff format --check / mypy <changed file>         # pass
```

## Real Behavior Proof

<details>
<summary>real behavior proof details</summary>

- Exact command / steps: python -m pytest
tests/test_proxy/test_gemini_response_framing_headers.py -q with a local
uvicorn proxy, a local http.server fake upstream and a real httpx client
comparing the before/after response headers.
- Environment: macOS aarch64, Python 3.14.5 worktree venv, local uvicorn
proxy with a Python `http.server` fake upstream, real httpx client, no
provider keys.
- Observed result: before the change the client received `server:
uvicorn, BaseHTTP/0.6 ..., Upstream-Marker/9.9`, `keep-alive`,
`connection: keep-alive` and `transfer-encoding: chunked`; after the
change only uvicorn's own `server` header remains, with `content-type`,
body and `x-headroom-tokens-*` intact for `generateContent` and
`countTokens`.
- Not tested: a real Gemini API call, the optimized/compression branch
as a separate e2e, the `_GeminiContinuationError` branch as its own
case, Windows/Linux, and the other providers that still hand-copy
headers (left as a follow-up to keep this PR scoped).
</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Gemini response replays no longer
forward upstream transport framing or server headers (intended).
- Kill switch / disable path: revert this commit.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] No unrelated changes
- [x] Tests added/pass
- [ ] CHANGELOG updated — no changelog entry for this internal header
fix; say so if you want one
- [x] No secrets or local paths

## Notes

No linked issue (source-derived, with a reproduction); other replay
paths use the same hand-copy pattern and are left as a follow-up.
Duplicate `date` headers (upstream Date plus uvicorn Date) are the
shared sanitizer's existing behavior and out of scope.

## Paste relevant command output or artifact links here

<details>
<summary>Commands and output</summary>

```text
pytest tests/test_proxy/test_gemini_response_framing_headers.py -q
# parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client
# this branch:    2 passed
pytest tests/test_proxy -q                                  # 491 passed
ruff check / ruff format --check / mypy <changed file>      # pass
```
</details>

## Screenshots (if applicable)

N/A - not user-visible; the evidence is the header comparison in the
Real Behavior Proof section (real uvicorn proxy + httpx client + local
fake upstream).

## Additional Notes

Other providers/buffered replay paths still hand-copy headers; left as a
follow-up to keep this PR scoped. Duplicate `date` headers are the
shared sanitizer's existing behavior and out of scope. No linked issue
(source-derived bug with a reproduction).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4605"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 09:13:56 -06:00
Younes BerianeandJD Davis 6ef52c2047 fix(proxy): route /v1/models to the x-headroom-base-url upstream (#4308)
## Description

Fixes #4302.

`GET /v1/models` and `GET /v1/models/{id}` always queried the default
provider upstream, so a request carrying `x-headroom-base-url` for a
custom OpenAI-compatible gateway got that gateway's key sent to
`api.openai.com` and a 401,. Both routes now resolve the upstream
through one helper that uses the header and applies the same unsafe-URL
rejection (400) as the catch-all.

Closes #4302

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Shared upstream resolver in `proxy_routes.py`, plus a test file.
- 5d7ecd1a5 (review follow-up): the resolver keeps the provider the
request's auth selects and overrides only the destination, so
`x-api-key` requests still get Anthropic model-id sanitization in
`handle_passthrough`. New
`tests/test_proxy/test_model_metadata_passthrough_sanitization.py` runs
the real `create_app` and `handle_passthrough` with an
`httpx.MockTransport` upstream and asserts the outbound URL.

## Testing

- [x] Unit tests pass (`pytest`) - the two metadata test files (7 tests)
- [x] Linting passes (`ruff check .`) - on the changed files
- [x] Type checking passes (`mypy headroom`) - `mypy headroom
--ignore-missing-imports` clean in the verify run
- [x] New tests added for new functionality
- [ ] Manual testing performed - not done

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_proxy/test_model_metadata_passthrough_sanitization.py tests/test_proxy/test_model_metadata_upstream_header.py -q -p no:cacheprovider
platform linux -- Python 3.14.7, pytest-9.2.0, pluggy-1.7.0
collected 7 items

tests/test_proxy/test_model_metadata_passthrough_sanitization.py ..      [ 28%]
tests/test_proxy/test_model_metadata_upstream_header.py .....            [100%]

============================== 7 passed in 6.76s ===============================

$ ruff check headroom/providers/proxy_routes.py tests/test_proxy/test_model_metadata_passthrough_sanitization.py && ruff format --check <same files>
All checks passed!
2 files already formatted

Discrimination: with proxy_routes.py reverted to a62ec010, test_anthropic_auth_with_header_upstream_sanitizes_model_id fails (outbound URL keeps `[1m]`) and the bearer test passes; with the fix, both pass.
Wider run (12 proxy/provider test files, same minimal venv): 282 passed / 17 failed at a62ec010 vs 283 passed / 16 failed at the fix; the 16 are environmental (offline-egress exporters and embedders absent from the minimal install, Gemini tests needing a key) and identical before and after.
```

## Real Behavior Proof

- Environment: FastAPI `TestClient` on the real `create_app`, real
`handle_passthrough`, only the outbound HTTP transport faked with
`httpx.MockTransport`
- Exact command / steps: the pytest command above
- Observed result: header honored on both routes; unsafe URLs get 400;
`GET /v1/models/claude-opus-4-8[1m]` with `x-api-key` and
`x-headroom-base-url: https://gateway.example` sends
`https://gateway.example/v1/models/claude-opus-4-8`; bearer auth sends
the id unchanged
- Not tested: live provider

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: none
- Stable/default behavior changed: no
- Kill switch / disable path: none
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines - ruff check and
format on the changed files
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings - not run
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes - see
Test Output; 16 failures are environmental and identical at base
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

N/A

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4308"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 22:47:46 -06:00
e2350bc8fd fix(proxy): warm deferred kompress model so /health stops reporting backend=null (#2799)
## Description

On an idle proxy with the ML extras installed and the Kompress model
cached, `/health` and `/readyz` reported `kompress: {"ready": false,
"status": "unhealthy", "backend": null}` forever (#2730).

The cause was not missing wiring: `enable_kompress` defaults to `True`
and `ContentRouter` sits in both pipelines. The chain was:

1. `ContentRouter.eager_load_compressors()` deliberately skips
`preload()`, so a native load never runs before the port binds (#790
hang on Windows, #1908 segfault on RHEL/CentOS 7). It returns `kompress
= "deferred"`.
2. `WarmupRegistry.merge_transform_status` leaves a deferred slot at
`status="null"`.
3. `_reconcile_kompress_health()` promotes the slot only once a
compressor reports `is_ready()`, and nothing loaded the model without a
kompress-routed request.

#3619 (merged after this PR was opened) added
`_start_kompress_background_warmup()`, which loads the model on a daemon
thread shortly after startup. That fixes the load itself. Rebased onto
it, this PR covers what is left:

1. **`/health` says why `ready` is false.** The kompress check gains a
`detail` field. `ready: false` alone cannot tell "still warming" from
"extras not installed" from "warm failed", which is the confusion in
#2730.
2. **The warm thread records its outcome on `warmup.kompress`.** Success
promotes the slot immediately with `source_status="background"`, instead
of leaving `source_status="deferred"` on a loaded slot until the next
reconcile. Failure leaves a bounded reason.
3. **The detail is a closed vocabulary.** `/health` and `/readyz` are
auth-exempt, and loader exceptions carry paths, model ids and URLs, so
anything outside the allowlist collapses to `unavailable`.

Closes #2730

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/kompress_health_detail_policy.py` (new, pure, matching
the existing `*_policy.py` modules): `KOMPRESS_HEALTH_DETAILS` lists the
values the field may report (`loaded`, `warming`, `model not cached`,
`warm failed`, `not installed`, plus the `eager_load_compressors()`
statuses `enabled` / `deferred` / `unavailable`). `public_detail()`
passes those through and maps anything else to `unavailable`.
- `headroom/proxy/server.py`:
- `_start_kompress_background_warmup()` (from #3619, still
`preload(allow_download=True)` on thread `kompress-warmup`) now writes
its result to the slot. On success it calls `mark_loaded(...,
source_status="background")`. On any exception it sets detail `warm
failed` and logs the real cause with `exc_info=True`; the exception text
never reaches the slot. A compressor that raises
`KompressModelNotCached` gets `model not cached`. The local loader only
raises that with downloads off, so on a cold cache it downloads and
`/health` reports `warming` in the meantime.
- `_kompress_detail()` feeds the new field. Every candidate, including
raw `slot.error` and `source_status`, goes through `public_detail()`, so
the allowlist has a single choke point.
- Router/compressor lookup moves to module-level `_kompress_routers()` /
`_kompress_compressors()`, shared by startup and `/health`.
- `enabled`, `ready` and `backend` keep their meaning, so the `/readyz`
aggregate exclusion of optional Kompress still holds.
- Tests:
- `tests/test_kompress_preload_deferral.py`: the warm thread promotes a
cached model with no request (`source_status == "background"`), a
compressor-reported cache miss stays deferred with `model not cached`,
and a failure carrying a path and a token leaves exactly `warm failed`.
- `tests/test_proxy_health.py`: sentinel tests over `/health` and
`/readyz` for both leak paths (`slot.info["detail"]` and `slot.error`).
- `tests/test_kompress_health_detail_policy.py`: covers the policy
directly.
- `tests/test_proxy/test_kompress_background_warmup.py` and
`tests/test_proxy_eager_preload_bind.py`: expectations updated for the
slot promotion.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Type checking: `mypy headroom --ignore-missing-imports` (mypy 1.20.2)
reports one error, in `headroom/fileperms.py` (`os.fchmod` does not
exist on Windows). That file is not touched by this PR, and there are no
errors in `server.py` or the new module.

### Test Output

```text
$ pytest tests/test_kompress_preload_deferral.py tests/test_proxy_health.py tests/test_proxy/test_kompress_background_warmup.py tests/test_kompress_health_detail_policy.py tests/test_proxy_eager_preload_bind.py tests/test_proxy_warmup.py -q
73 passed in 22.36s

$ ruff check headroom tests
All checks passed!

$ ruff format --check headroom tests
1519 files already formatted

$ mypy headroom --ignore-missing-imports --no-site-packages
headroom\fileperms.py:71: error: Module has no attribute "fchmod"; maybe "chmod"?  [attr-defined]
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, headroom installed from this
branch, Kompress model `chopratejas/kompress-v2-base` in the local HF
cache, `HF_HUB_OFFLINE=1`, branch rebased on `upstream/main` 540f4da1
- Exact command / steps: built the real app with
`create_app(ProxyConfig(optimize=True, cache_enabled=False,
rate_limit_enabled=False, cost_tracking_enabled=False))`, started it
through the FastAPI lifespan with
`HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS=0`, sent no compression request,
and polled `GET /health` every 250 ms until Kompress reported ready,
then read `/readyz` and the warmup slot
- Observed result: at t=0.25 s `/health` returned `{"ready": false,
"status": "degraded", "backend": null, "detail": "warming"}`. The log
showed `Kompress: warmed in the background in 7618 ms (backend onnx)`,
and at t=6.44 s `/health` returned `{"ready": true, "status": "healthy",
"backend": "onnx", "detail": "loaded"}`. `/readyz` matched, and the slot
was `{"status": "loaded", "info": {"source_status": "background",
"backend": "onnx"}}`. No request was needed.
- Not tested: macOS arm64 (the reporter's platform), a real cold-cache
download through the warm thread, and glibc < 2.28 hosts, where #3619
skips the warm-up and `/health` keeps reporting `deferred`.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Health reporting for the existing
background warm-up from #3619.
- Minimum rollout channel: stable
- Stable/default behavior changed: Yes, additively. The kompress health
check gains a `detail` field, and a successful background warm marks the
slot `loaded` straight away instead of at the next `/health` call.
`enabled`, `ready`, `status` and `backend` are unchanged.
- Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0` disables the
background warm-up (from #3619). The `detail` field has no separate
switch; it only reads state.
- Unsafe override required: No
- Qualification impact: None expected. Health payloads gain one field,
and every value comes from a fixed allowlist.
- Rollback path: Revert the PR. There is no stored state or migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`. It is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A. No user-facing UI change.

## Additional Notes

- No documentation change: the `detail` field is diagnostic.
`HEADROOM_KOMPRESS_WARMUP` (from #3619) is not in the env-var reference
yet either; happy to add it here if maintainers prefer.
- An earlier revision of this PR added its own
`HEADROOM_KOMPRESS_BACKGROUND_WARM` switch. It was removed after the
rebase because it did the same job as `HEADROOM_KOMPRESS_WARMUP` and was
never released.
- Follow-up, out of scope: the #3619 warm thread is not told about
`shutdown()`. It is a daemon thread and production shutdown ends the
process, but an in-process restart could still run one load after stop.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 22:30:37 -06:00
gglucassandClaude Opus 5.5 78736a16bc fix(proxy): keep health probes and the Codex attestation out of the proxy log (#4611)
## Description

Two things fill and leak through the always-on proxy log
(`~/.headroom/logs/proxy-<port>.log`):

1. Health probes. Every `/readyz`, `/livez` and `/health` poll writes an
`event=proxy_inbound_request` and an `event=proxy_inbound_response` line
at INFO. A desktop wrapper or orchestrator polls these every few
seconds: in an Extra Headroom user's proxy log (Codex + Claude Code,
2026-10-27) 23,201 `/readyz` probes were 19% of 54 MB, and on a lighter
machine they are 48% of the file. With 10 MB x 5 rotation, that log held
only 14 hours; at the busiest hour one 10 MB file lasted about an hour.
2. The Codex attestation. The inbound request line runs headers through
`redact_for_wire_debug`, whose key list has no entry for
`x-oai-attestation`. Codex sends that ~4 KB signed device attestation on
every request, so every Codex request wrote it to the log in clear. That
is 3,040 copies (12 MB) in the log above, in a file users are routinely
asked to attach to bug reports. #2612 proposed the same key for the
opt-in wire-debug capture and was closed for inactivity. The default
proxy log uses the same policy, so this fix covers both.

Health probes now log at DEBUG (both lines), and `x-oai-attestation` is
a redacted key. Every other path still logs at INFO exactly as before.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: a module-level `_HEALTH_PROBE_PATHS`
(`/health`, `/livez`, `/readyz`). The inbound request/response lines use
DEBUG for a GET to one of them, INFO otherwise. `_AUTH_EXEMPT_PATHS` now
refers to the same set instead of repeating it. The aborted/error lines
are unchanged.
- `headroom/proxy/wire_debug_redaction_policy.py`: `x-oai-attestation`
added to `WIRE_DEBUG_SECRET_KEYS`.
- Tests: `test_health_probes_log_at_debug_and_traffic_at_info` and
`test_wire_debug_redacts_codex_attestation_header`. Both fail on `main`
and pass here.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest -q tests/test_wire_debug_redaction_policy.py tests/test_proxy/test_inbound_response_size.py
11 passed, 1 warning in 5.36s

# every test file that mentions readyz / livez / proxy_token / inbound_request (36 files)
$ uv run --frozen --extra dev pytest -q $(git grep -l -E 'readyz|livez|proxy_token|inbound_request' -- 'tests/*.py' 'tests/**/*.py')
4 failed, 1245 passed, 2 skipped
# the 4 are tests/test_cli/test_wrap_persistent.py and fail identically on unmodified main on this machine

$ uvx ruff check <changed files>   -> All checks passed!
$ uvx ruff format --check <changed files>   -> 4 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/server.py headroom/proxy/wire_debug_redaction_policy.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 27.2, this branch via `uv run --frozen --extra
proxy headroom proxy --port 18799 --no-optimize`,
`HEADROOM_SKIP_UPSTREAM_CHECK=1`.
- Exact command / steps: polled `GET /livez` until ready, then `curl
/readyz`, then `curl -H 'x-oai-attestation:
{"v":1,"s":1,"t":"SIGNED-DEVICE-TOKEN"}' -H 'x-codex-window-id: w:1'
/stats`, then read stderr and `~/.headroom/logs/proxy-18799.log`.
- Observed result: neither output has a `/livez` or `/readyz` line; the
`/stats` request line shows `"x-oai-attestation": "[REDACTED]"` next to
the visible `x-codex-window-id`, and `SIGNED-DEVICE-TOKEN` occurs 0
times in either output.
- Not tested: a Codex client end to end; the header value is never used
by the proxy, only logged.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, logging only
- Stable/default behavior changed: yes, health-probe inbound lines move
from INFO to DEBUG and `x-oai-attestation` is redacted in logged
headers; forwarding is unchanged
- Kill switch / disable path: run with DEBUG logging to see probe lines
again
- Unsafe override required: no
- Qualification impact: none, no request or response bytes change
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Docs: no doc describes these log lines. Credit to #2612 for the
redaction key.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4611"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 17:00:02 -06:00
28172a921f fix(proxy): don't fail startup when libc_ver() can't read the interpreter (#4519)
## Description

On Windows with Microsoft Store Python, `headroom proxy` exits during
startup with code 3:

```text
  File ".../headroom/proxy/server.py", line 2007, in _start_kompress_background_warmup
    libc, version = platform.libc_ver()
  File "...\Lib\platform.py", line 200, in libc_ver
    with open(executable, 'rb') as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\<user>\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\\python.exe'
ERROR:    Application startup failed. Exiting.
```

Here's how it happens:
1. When `HEADROOM_KOMPRESS_WARMUP` is unset,
`_start_kompress_background_warmup` calls `platform.libc_ver()` to skip
warm-up on glibc < 2.28 (#1908). It does this on every OS.
2. `platform.libc_ver()` works by opening `sys.executable` and scanning
its bytes.
3. On Store Python, `sys.executable` is the
`...\WindowsApps\...\python.exe` app-execution alias, which can't be
opened for reading, so `open()` raises `OSError` (Errno 22).
4. Nothing catches it, so lifespan startup fails.

This PR catches `OSError` from `libc_ver()` and treats the interpreter
as non-glibc, which is correct on Windows. The warm-up then runs as it
does on any other non-glibc platform. The glibc < 2.28 skip is
unchanged: on glibc, `libc_ver()` returns the version from
`os.confstr('CS_GNU_LIBC_VERSION')` and never opens the executable, so
the new `except` can't fire on the hosts #1908 protects.

Workaround until this ships: set `HEADROOM_KOMPRESS_WARMUP=1` (or `0`),
which skips the `libc_ver()` call.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/server.py`: wrap `platform.libc_ver()` in `try/except
OSError`, falling back to "not glibc".
- `tests/test_proxy/test_kompress_background_warmup.py`: regression
test. When `libc_ver()` raises `OSError`, the warm-up still starts
instead of the error propagating.

## Testing

- [x] Unit tests pass (`pytest
tests/test_proxy/test_kompress_background_warmup.py`)
- [x] Linting passes (`ruff check`, `ruff format --check` on the changed
files)
- [x] New tests added
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_proxy/test_kompress_background_warmup.py -q
7 passed, 1 warning in 3.16s

# new test with the server.py change reverted:
E   OSError: [Errno 22] Invalid argument
FAILED tests/test_proxy/test_kompress_background_warmup.py::test_unreadable_interpreter_does_not_fail_startup
1 failed, 6 passed, 1 warning in 2.40s
```

## Real Behavior Proof

- Environment: Windows 11 Pro (10.0.26200), Python 3.12 from the
Microsoft Store (`sys.executable` =
`...\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe`),
headroom 0.42.0 from PyPI, default config, `HEADROOM_KOMPRESS_WARMUP`
unset.
- Exact command / steps: `headroom proxy --port 8798` on the installed
0.42.0 (before), then `headroom proxy --port 8799` on 0.42.0 plus this
branch's `server.py` (after), polling `GET /health` each time. In
detail:
1. Interpreter check: `python -c "import platform; platform.libc_ver()"`
raises `OSError: [Errno 22] Invalid argument` on this machine.
  2. Before: `headroom proxy --port 8798` using the installed 0.42.0.
3. After: the installed 0.42.0 package copied to a temp directory, with
only `headroom/proxy/server.py` replaced by this branch's version
(0.42.0's `server.py` is byte-identical to `main`'s). Then `headroom
proxy --port 8799` from that copy, polling `GET /health`. I used the
copy because a source checkout has no built `headroom._core`, and the
proxy refuses to start without it.
- Observed result: 0.42.0 aborts startup with the traceback above. With
the patch, the proxy starts, `/health` returns 200, the process stays
up, and the Kompress warm-up thread starts.

```text
=== installed 0.42.0 ===
    libc, version = platform.libc_ver()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.2800.0_x64__qbz5n2kfra8p0\Lib\platform.py", line 200, in libc_ver
OSError: [Errno 22] Invalid argument: 'C:\\Users\\<user>\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\\python.exe'
ERROR:    Application startup failed. Exiting.

=== 0.42.0 + this patch ===
headroom.proxy - INFO - Headroom Proxy started (version 0.42.0)
headroom.proxy - INFO - Kompress: DEFERRED (warming in the background after startup)
GET /health -> 200 {"service":"headroom-proxy","status":"healthy","ready":true,"version":"0.42.0",...}
process alive after health check: True
```

- Not tested: python.org Windows Python (its `python.exe` is a real
file, so `libc_ver()` doesn't raise there), Linux/macOS (behavior
unchanged unless `libc_ver()` raises), and a request routed through the
proxy to a provider. I didn't wait for the background warm-up to finish;
it only had to start without breaking startup.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This changes the Kompress warm-up
gate at proxy startup.
- Minimum rollout channel: N/A
- Stable/default behavior changed: only where `platform.libc_ver()`
raises `OSError`. Startup used to fail there; now it proceeds and warms
Kompress as on any non-glibc platform. Everywhere else, behavior is
unchanged.
- Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: no
documented behavior changes)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-29 20:26:21 -05:00
HetCreepandClaude Opus 5.5 d9958bfdff fix(proxy): recognise every local spelling of the proxy in the cc-switch loop guard (#4405)
## Description

The cc-switch reconciler's loop guard compared `ANTHROPIC_BASE_URL` with
the exact string `http://127.0.0.1:{port}`. `http://localhost:{port}`,
`http://LOCALHOST:{port}/v1/` and `http://127.0.0.1:{port}/v1` were
therefore captured as the proxy's own upstream, so requests looped back
to the proxy. The guard now compares scheme, port and host, where host
is `localhost` or the proxy's own bind address. `[::1]` is a different
socket from the `127.0.0.1` the proxy binds, so a provider on
`[::1]:{port}` is still captured (review feedback, second commit).

Fixes #4400

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/cc_switch_reconciler.py`: add
`CCSwitchReconciler._is_self()` (urlsplit; host is `localhost` or the
host of `proxy_url`, same scheme and port) and use it in the loop guard.
- `tests/test_proxy/test_cc_switch_reconciler.py`: 5 spellings that must
not be captured; 5 URLs (other port, https, `[::1]` provider on the same
port, third party) that must still be captured; one test that `[::1]` is
the proxy only when `proxy_url` is `[::1]`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

First commit, with only its source change reverted (new tests kept):

```text
FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_proxy_own_url_spellings_are_not_captured[http://127.0.0.1:8787/v1]
FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_proxy_own_url_spellings_are_not_captured[http://localhost:8787]
FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_proxy_own_url_spellings_are_not_captured[http://LOCALHOST:8787/v1/]
FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_proxy_own_url_spellings_are_not_captured[http://[::1]:8787]
========================= 4 failed, 22 passed in 1.33s =========================
```

First commit with the fix: 26 passed.

Second commit (`[::1]` follow-up), with only its source change reverted
(its test changes kept):

```text
FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_other_ports_and_hosts_are_still_captured[http://[::1]:8787]
FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_ipv6_proxy_address_is_self_only_for_ipv6_bind
========================= 2 failed, 25 passed in 0.84s =========================
```

With the second commit:

```text
============================== 27 passed in 0.76s ==============================
```

`ruff check` and `ruff format --check` on both changed files: All checks
passed / 2 files already formatted. `mypy` was not run.

## Real Behavior Proof

- Environment: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn
0.54.0, fastapi 0.158.1, starlette 1.9.0; base `e995eb9f`;
`headroom/_core.abi3.so` copied (untracked, not committed) from the
official `headroom_ai-0.41.0` wheel.
- Exact command / steps: stub upstream on 127.0.0.1:18821 that logs
every POST; `CLAUDE_CONFIG_DIR=<dir> HEADROOM_CC_SWITCH_RECONCILE=1
ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18821 python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18820 --no-cache
--no-rate-limit`, with `<dir>/settings.json` = `{"env":
{"ANTHROPIC_BASE_URL": "http://localhost:18820", ...}}`; then one `POST
/v1/messages` to 127.0.0.1:18820 (40 s client timeout). Before = parent
source (this change reverted with `git apply -R`), after = this change.
- Observed result: before, the reconciler logs `captured
upstream=http://localhost:18820`, rewrites settings.json to
`http://127.0.0.1:18820`, the client times out and the stub receives 0
requests. After, settings.json is left unchanged, the client gets HTTP
200 and the stub receives 1 `POST /v1/messages`. Control with the fix
and a real third-party URL (`http://127.0.0.1:18821`): still captured,
settings.json rewritten to the proxy, HTTP 200, stub receives 1 request.
- Second proof (review feedback): a provider stub on `[::1]:18820`, the
proxy on `127.0.0.1:18820`, a different stub on `127.0.0.1:18821` as
`ANTHROPIC_TARGET_API_URL`, settings.json
`ANTHROPIC_BASE_URL=http://[::1]:18820`, same client request. Before the
second commit settings.json is left on `[::1]` and the request reaches
the 18821 default upstream, not the provider (provider received none).
After it, the reconciler logs `captured upstream=http://[::1]:18820`,
settings.json is rewritten to `http://127.0.0.1:18820`, and the provider
on `[::1]` receives the `POST /v1/messages` (default upstream: none).
- Not tested: real cc-switch overwriting settings.json; a real Anthropic
upstream; Windows/macOS; the Rust proxy.

Raw output:

```text
== BEFORE
settings.json after reconcile: {"env":{"ANTHROPIC_BASE_URL":"http://127.0.0.1:18820",...}}
client: TimeoutError timed out
cc-switch reconciler: captured upstream=http://localhost:18820, base_url -> http://127.0.0.1:18820
stub upstream received: (count: 0)
== AFTER
settings.json after reconcile: {"env":{"ANTHROPIC_BASE_URL":"http://localhost:18820",...}}
client: HTTP 200 {"id": "msg_stub", ...}
stub upstream received: STUB-UPSTREAM POST /v1/messages (count: 1)
== CONTROL (fix, third-party URL)
cc-switch reconciler: captured upstream=http://127.0.0.1:18821, base_url -> http://127.0.0.1:18820
client: HTTP 200 {"id": "msg_stub", ...}
stub upstream received: STUB-UPSTREAM POST /v1/messages (count: 1)

== SECOND PROOF, BEFORE the [::1] commit
settings.json after reconcile: {"env":{"ANTHROPIC_BASE_URL":"http://[::1]:18820",...}}
provider on [::1]:18820 received: none
default upstream 127.0.0.1:18821 received: STUB-DEFAULT-UPSTREAM POST /v1/messages
== SECOND PROOF, AFTER
cc-switch reconciler: captured upstream=http://[::1]:18820, base_url -> http://127.0.0.1:18820
settings.json after reconcile: {"env":{"ANTHROPIC_BASE_URL":"http://127.0.0.1:18820",...}}
provider on [::1]:18820 received: STUB-PROVIDER-ON-IPV6 POST /v1/messages
default upstream 127.0.0.1:18821 received: none
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none (opt-in
`HEADROOM_CC_SWITCH_RECONCILE=1`)
- Minimum rollout channel: n/a
- Stable/default behavior changed: no; the reconciler is off by default
- Kill switch / disable path: leave `HEADROOM_CC_SWITCH_RECONCILE` unset
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation checklist item: N/A, no user-facing doc changes.
- Scope: only the loop guard; the stale-upstream handling and
`_atomic_write` are untouched.
- Produced with AI assistance (Claude Code) operated by me (HetCreep);
the reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 23:29:19 -05:00
Vaibhav HajareandJD Davis b1aaf5b366 fix: do not report a seeded stored setting as an env override (#4326)
## Description

`apply_to_environ()` seeds each stored value into its own `HEADROOM_*`
env var at startup (`os.environ.setdefault(field.env, _serialize(field,
value))`). `to_schema()` then set `env_override` from the mere
*presence* of that env var:

```python
"env_override": bool(os.environ.get(field.env)) and field.env not in env_override_exclusions,
```

So a setting saved from the dashboard appeared as "overridden by an
environment variable" and the UI treated it as locked, even though
nothing outside headroom ever exported it. That state survived a
restart, because `apply_to_environ` re-seeds on every start.

Closes #4320

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `to_schema()` now reports `env_override` only when the env var's value
**differs** from the value the settings file would serialize to. A var
that merely matches the stored value is headroom's own seeded value, not
an operator export.
- The existing `env_override_exclusions` check is folded into the same
condition, so profile-seeded keys keep behaving as before.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Not run locally: this machine has no opentelemetry install, so headroom does
not import and the suite cannot be collected:

    ModuleNotFoundError: No module named 'opentelemetry'

Installing a toolchain is out of scope here, so I did not run pytest. GitHub
CI is the judge.
```

## Real Behavior Proof

- Environment: headroom source checkout, no third-party deps installed
- Exact command / steps: read `settings_store.py`, traced
`apply_to_environ` (line 1002) into `to_schema` (line 1097)
- Observed result: a stored value's own env var satisfied
`bool(os.environ.get(field.env))` and produced `env_override: True`
- Not tested: running the test suite (no `opentelemetry`), and the
dashboard UI itself

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only how `env_override` is computed
in the settings schema; effective values are unchanged
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- "Unit tests pass locally" is unchecked because the suite cannot be
collected without `opentelemetry`; the new test lives in
`tests/test_proxy/test_settings_store.py` next to the existing
env-override tests and asserts both directions (seeded value → `False`,
genuinely differing value → `True`).
- The regression test is committed; CI will exercise it.

## Model Used

MiniMax M3.1 Flash Free via the opencode agent harness
(`minimax-m3.1-flash-free`)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4326"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 21:13:22 -05:00
gglucassandClaude Opus 5.5 11f743dd90 fix(savings): key a Responses conversation on its thread, not the shared session (#4440)
## Description

`savings_conversation_key` picks `session_id` before `thread_id`. Codex
sends both, flat in `client_metadata`
(`codex-rs/core/src/responses_metadata.rs`, `client_metadata()`), and
its `session_id` is the root thread's id on every request of the
session: the auto-review guardian and spawned subagents carry the root's
`session_id` with their own `thread_id`. All of them land on one key.

`ConversationSavings.novel()` stores each request's running total under
that key, so a guardian request (total 0) resets the root's total, and
the root's next turn books its whole history as new again. With the
guardian running every few turns, a Codex conversation is booked several
times over.

This changes the precedence so a `thread_id` wins over a `session_id`
beside it, at the top level and in `client_metadata`/`metadata`. A body
with only one of the two keys keys exactly as before.

It matters for #4238 too: `prefix_key` in `headroom/proxy/savings.py`
builds on `savings_conversation_key`, so a subagent that shares the
root's model, tools and instructions would still share its baseline.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/conversation_savings.py`: `thread_id` before
`session_id` in both lookup orders; docstring says why.
- `tests/test_proxy/test_conversation_savings.py`: the Codex shape
(shared `session_id`, distinct `thread_id`) yields one key per thread; a
guardian request between two root turns no longer makes the root re-book
its history (`[1000, 0, 200]` instead of `[1000, 0, 1200]`).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_openai_codex_ws_lifecycle.py tests/test_output_savings.py tests/test_codex_project_context.py tests/test_savings_items.py tests/test_proxy/test_conversation_savings.py -q
327 passed, 1 warning in 12.64s

# the two new tests against the old lookup order
$ git checkout upstream/main -- headroom/proxy/conversation_savings.py && pytest tests/test_proxy/test_conversation_savings.py -q
2 failed, 20 passed

$ uvx ruff check / ruff format --check (changed files)
All checks passed! / 2 files already formatted
```

## Real Behavior Proof

- Environment: macOS, headroom-ai 0.41.0 as installed by Headroom
Desktop, Codex CLI 0.156.1 over HTTP SSE, real `~/.codex/sessions`
rollouts and a real `~/.headroom/savings_events.jsonl`.
- Exact command / steps: replayed two real Codex sessions request by
request through `_compress_openai_responses_payload` with Kompress warm,
fed each request's running total to `ConversationSavings.novel()` under
per-thread keys vs one session key with a guardian request interleaved,
and compared against the fresh content each request added.
- Observed result: per-thread keys book 16.1% and 11.5% of new input;
with a guardian request after every 5th turn the same sessions book 58%
and 69%. The local ledger on one machine showed a root conversation
re-booking about 45.5k tokens after each guardian call.
- Not tested: the WebSocket path end to end (same lookup order; turns
carrying `previous_response_id` already book per request), and `mypy
headroom`.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, Codex savings totals drop to
once per thread; nothing else reads the key.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- Id-less Codex bodies still key to `None` and book per request; a
fallback key for them is a separate change.
- Related: #4238 (savings redesign), #3480 (the per-conversation
ledger).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4440"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 08:44:03 -07:00
4fd9cddfd2 fix(proxy): honor bypass header and license gate in Anthropic batch create (#4411)
## Description

`handle_anthropic_batch_create` ignored `x-headroom-bypass: true`,
`x-headroom-mode: passthrough` and the `usage_reporter.should_compress`
license gate, and always re-serialized the body. The OpenAI and Google
batch paths enforce all of these. This PR makes the Anthropic batch path
do the same: compression is skipped for each item in either case, and on
bypass the client's original bytes are forwarded.

Fixes #4404

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: read the body with
`read_request_json_with_bytes`; compute `_bypass`
(`_headroom_bypass_enabled`) and `license_ok` (same expression as
`batch.py`, fails open without a reporter); tag `passthrough_reason`;
skip the per-item compression loop body when either gate applies; on
bypass call `_retry_request` with `original_body_bytes` and
`body_mutated=False`.
- `tests/test_proxy/test_anthropic_batch_bypass.py`: 4 tests (bypass
header, `x-headroom-mode: passthrough`, license denied, unchanged
compression path).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

With only the source change reverted (new test file kept):

```text
FAILED tests/test_proxy/test_anthropic_batch_bypass.py::test_batch_create_bypass_skips_compression_and_forwards_original_bytes[bypass_headers0]
FAILED tests/test_proxy/test_anthropic_batch_bypass.py::test_batch_create_bypass_skips_compression_and_forwards_original_bytes[bypass_headers1]
FAILED tests/test_proxy/test_anthropic_batch_bypass.py::test_batch_create_license_denied_skips_compression
==================== 3 failed, 1 passed, 1 warning in 6.77s ====================
```

With the fix, new tests plus neighbouring files
(`tests/test_proxy/test_anthropic_batch_bypass.py
tests/test_proxy/test_anthropic_buffered_timeout.py
tests/test_proxy_anthropic_cache_stability.py
tests/test_proxy_handlers_batch.py`):

```text
======================== 92 passed, 1 warning in 13.36s ========================
```

```text
$ ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_batch_bypass.py
All checks passed!
$ ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_batch_bypass.py
2 files already formatted
```

## Real Behavior Proof

- Environment: WSL2 Ubuntu on Windows 11, Python 3.12.15, branch based
on upstream/main `e995eb9f`; `headroom/_core.abi3.so` taken from the
official headroom-ai 0.41.0 wheel (untracked, not committed). Real proxy
on 127.0.0.1:18860, local stub Anthropic upstream on 127.0.0.1:18861
that logs the byte count and sha256 of each request body.
- Exact command / steps:
`ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18861 python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18860 --no-cache
--no-rate-limit`, then a stdlib client POSTs the same 47340-byte
`/v1/messages/batches` body twice (non-canonical whitespace, one large
tool_result): once without headers, once with `x-headroom-bypass: true`.
Run on the code without the fix, then with it.
- Observed result: without the fix, the bypass request reaches the stub
as 47284 bytes, sha256 `d2e7efe52f09facb` (re-serialized; the sent body
was `09cbbe29568d899d`). With the fix, the bypass request reaches the
stub as 47340 bytes, sha256 `09cbbe29568d899d`, identical to what the
client sent; the request without the header is unchanged by the fix
(47284 bytes, `d2e7efe52f09facb`).

  Without the fix:
  ```text
  STUB POST /v1/messages/batches bytes=47284 sha256=d2e7efe52f09facb
  client sent bytes=47340 sha256=09cbbe29568d899d headers={}
  STUB POST /v1/messages/batches bytes=47284 sha256=d2e7efe52f09facb
client sent bytes=47340 sha256=09cbbe29568d899d
headers={'x-headroom-bypass': 'true'}
  ```
  With the fix:
  ```text
  STUB POST /v1/messages/batches bytes=47284 sha256=d2e7efe52f09facb
  client sent bytes=47340 sha256=09cbbe29568d899d headers={}
  STUB POST /v1/messages/batches bytes=47340 sha256=09cbbe29568d899d
client sent bytes=47340 sha256=09cbbe29568d899d
headers={'x-headroom-bypass': 'true'}
  ```
In this run the compression pipeline did not shrink the body (the
56-byte difference is canonical re-serialization), so the real-process
check demonstrates the byte-faithful forward, not a compression change.
- Not tested: the license-denied branch in a real process (covered only
by the unit test with a stub reporter); a real Anthropic API; batch
results processing (`handle_anthropic_batch_results`) after a bypassed
create; compressed (gzip/zstd) request bodies; `mypy headroom` was not
run.

Scripts and raw output:
`/var/tmp/cb-pub/proof/fix-anthropic-batch-bypass/` (WSL).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for requests without the bypass
header and with a license that permits compression; with the header or a
denied license, batch items are no longer compressed
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- On bypass the CCR batch context store may still run if
`ccr_inject_tool` is on; it is harmless because no markers are injected.
Left unchanged to keep the diff small.
- Documentation: none changed; the bypass contract is already documented
for the other handlers.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4411"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:38:24 -07:00
51e4ae36a5 fix(proxy): import Response at runtime in the Bedrock handler (#4397)
## Description

`handle_bedrock_invoke` returns `Response(status_code=204)` when the
client disconnects during the body read, but `Response` was imported
only under `TYPE_CHECKING`, so that branch raised `NameError`. This
imports it next to `JSONResponse` inside the handler.

Closes #4393

Fixes #4393

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/bedrock.py`: `from fastapi.responses import
JSONResponse, Response` in `handle_bedrock_invoke`.
- `tests/test_proxy/test_bedrock_passthrough.py`:
`test_client_disconnect_during_body_read_returns_204`.

## Testing

- [x] Unit tests pass (`pytest`) (the five Bedrock test files only)
- [x] Linting passes (`ruff check .`) (changed files only)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# source change reverted, new test only
E   starlette.requests.ClientDisconnect
E   NameError: name 'Response' is not defined
1 failed, 23 deselected

# with the fix, 5 Bedrock test files
52 passed, 1 warning in 24.57s

ruff check: All checks passed!
ruff format --check: 2 files already formatted
```

## Real Behavior Proof

- Environment: Linux (WSL2), Python 3.12.15, the real `python -m
headroom.proxy.server --bedrock-api-url http://127.0.0.1:18792`, Rust
core from the headroom-ai 0.41.0 wheel, a local stub upstream
- Exact command / steps: a raw-socket client sends `POST
/model/anthropic.claude-3-haiku/invoke` with `content-length: 1000`,
writes about 50 bytes and closes; then a normal request; run with the
import change reverted and on this branch (details below)
- Observed result: reverted, the log shows `NameError: name 'Response'
is not defined` while handling `ClientDisconnect`; this branch logs
`status=204` with no traceback, the partial request is not forwarded,
and the next request returns 200
- Not tested: real Bedrock or a gateway, the streaming route, a client
behind a TCP proxy, a Rust core built from this checkout

Details:

Setup: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn 0.54.0,
fastapi 0.158.1, starlette 1.9.0. Worktree at PR head `c7d9d59e` (base
`e995eb9f`).
`headroom/_core.abi3.so` was copied (untracked, not committed) from the
official `headroom_ai-0.41.0-cp310-abi3-manylinux_2_28_x86_64.whl`
because the checkout has no built Rust core. The core is from 0.41.0,
the Python source is upstream/main plus the PR.
Stub upstream: a 20-line `http.server` on 127.0.0.1:18792 that logs
every POST it receives. Proxy: the real server, `python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18791 --bedrock-api-url
http://127.0.0.1:18792`, started with `HEADROOM_LOG_LEVEL=DEBUG` (the
debug line goes to `~/.headroom/logs/proxy-18791.log`).

Client (raw socket, `bedrock_client.py`): connect, send `POST
/model/anthropic.claude-3-haiku/invoke` headers with `content-length:
1000`, write about 50 body bytes, wait 0.5 s, close the socket. Then, on
the same proxy, send one normal full request.

Before = `headroom/proxy/handlers/bedrock.py` with the PR's one-line
change reverted (`git apply -R`). After = PR head.

### Before (PR change reverted)

```
proxy /livez -> 200
client: sent 1000-byte Content-Length, wrote ~50 bytes, closed socket
client: follow-up full request -> 200 {"stub": "ok"}
--- proxy log
42: DEBUG - [hr_1792672318_000001] bedrock_invoke client disconnected during body read
43: ERROR - event=proxy_inbound_request_aborted id=inbound-1792672318828183190 method=POST path=/model/anthropic.claude-3-haiku/invoke reason=NameError duration_ms=500.61
45:  File ".../headroom/proxy/handlers/bedrock.py", line 132, in handle_bedrock_invoke
54:    raise ClientDisconnect()
55: starlette.requests.ClientDisconnect
...
110: NameError: name 'Response' is not defined          (bedrock.py line 138, during handling of the ClientDisconnect)
--- stub upstream received
STUB-UPSTREAM POST /model/anthropic.claude-3-haiku/invoke body_bytes=49
```

The disconnect branch is reached (the DEBUG line), then `return
Response(status_code=204)` raises `NameError` (`Response` is imported
only under `TYPE_CHECKING`, line 38), so the request ends as
`proxy_inbound_request_aborted` with a traceback instead of a handled
disconnect.

### After (PR head c7d9d59e)

```
proxy /livez -> 200
client: sent 1000-byte Content-Length, wrote ~50 bytes, closed socket
client: follow-up full request -> 200 {"stub": "ok"}
--- proxy log
42: DEBUG - [hr_1792672327_000001] bedrock_invoke client disconnected during body read
43: INFO  - event=proxy_inbound_response id=inbound-1792672327942098751 method=POST path=/model/anthropic.claude-3-haiku/invoke status=204 duration_ms=501.52
48: INFO  - event=proxy_inbound_response ... method=POST path=/model/anthropic.claude-3-haiku/invoke status=200 duration_ms=55.82
--- stub upstream received
STUB-UPSTREAM POST /model/anthropic.claude-3-haiku/invoke body_bytes=49
```

The disconnect is logged and answered with status 204, no traceback,
nothing is forwarded for the partial request (the stub saw exactly one
POST: the 49-byte follow-up), and the proxy keeps serving.

### Commands

```
# stub upstream
python -P stub.py 18792 bedrock.stub.log &
# real proxy
HEADROOM_LOG_LEVEL=DEBUG PYTHONPATH=. python -P -m headroom.proxy.server --host 127.0.0.1 --port 18791 --bedrock-api-url http://127.0.0.1:18792 &
# client
python -P bedrock_client.py 18791
```

Scripts and raw outputs: `/var/tmp/cb-pub/proof/` (WSL):
`run_bedrock.sh`, `stub.py`, `bedrock_client.py`, `before_bedrock.out`,
`after_bedrock.out`.

### Not tested

- Real AWS Bedrock or a re-signing gateway (stub upstream only).
- A client behind a real TCP proxy or load balancer; the disconnect is a
local socket close.
- The Rust core is the 0.41.0 wheel's, not one built from this checkout.
- Streaming (`invoke-with-response-stream`) route; only `invoke` was
driven.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no (the branch previously raised
NameError)
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Only the Bedrock tests were run, not the full suite. Produced with AI
assistance (Claude Code) operated by me (HetCreep); the reproduction was
run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4397"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:37:36 -07:00
8cedf7be4d feat(pipeline): put per-request saved tokens on OutcomeSnapshot (#4231)
## Description

Expose per-request message-compression savings and the novel share
booked to conversation totals on the immutable OutcomeSnapshot passed to
pipeline extensions. Extensions can consume the same readings as the
proxy without parsing PERF logs.

The previous body described the separate remote-Kompress startup fix in
#4234. This description has been corrected by the maintainer to match
this PR's diff.

## Type of Change
- [x] New feature (non-breaking additive outcome fields)

## Changes Made
- Add tokens_saved and novel_tokens_saved to OutcomeSnapshot, with
defaults preserving existing construction.
- Emit OUTCOME_OBSERVED after the proxy computes the novel share.
- Populate both fields from the proxy's accounting result.
- Add a regression observing successive Responses outcomes through
PipelineExtensionManager.

## Testing
- [x] Focused conversation-savings tests pass.
- [x] Touched-file Ruff and whitespace checks pass.

### Test Output
```text
python -m pytest tests/test_proxy/test_conversation_savings.py tests/test_original_snapshot_in_place_hook.py -q
25 passed, 1 warning in 8.72s

ruff check headroom/pipeline.py headroom/proxy/outcome.py tests/test_proxy/test_conversation_savings.py
All checks passed!
```

Maintainer verification used the contributor's unchanged outcome
implementation. The latest clean main sync incorporates #4241's
snapshot-preservation fix; 25 combined conversation/snapshot tests
passed. Exact-head hosted CI remains in progress.

## Real Behavior Proof
- Environment: maintainer isolated Windows/Python 3.12.13 worktree;
actual outcome emitter and PipelineExtensionManager exercised by the
focused suite.
- Exact command / steps: python -m pytest
tests/test_proxy/test_conversation_savings.py
tests/test_original_snapshot_in_place_hook.py -q; inspect
test_pipeline_extensions_read_both_figures_from_the_snapshot.
- Observed result: 25 tests passed, including the regression asserting
that extension snapshots carry per-request and novel savings across
successive Responses outcomes.
- Not tested: maintainer live provider round trip with an installed
entry-point extension. The author separately reports replaying recorded
Codex outcomes through a real create_app proxy and observing
tokens_saved=8575 and novel_tokens_saved=791, matching the recorded PERF
figures; this is author-supplied replay evidence, not an independent
live inference claim.

## Runtime Rollout Safety
- Rollout-managed feature(s): no additional rollout-managed switch
introduced.
- Minimum rollout channel: existing pipeline-extension policy applies.
- Stable/default behavior changed: enabled extensions receive two
additive fields; the outcome event is emitted after novelty accounting.
- Kill switch / disable path: disable the consuming pipeline extension.
- Unsafe override required: no.
- Qualification impact: exact-head hosted CI and final human review
remain required.
- Rollback path: revert this change or retain the previous qualified
release.

## Review Readiness
- [ ] I performed self-review
- [ ] This PR is ready for human review

Maintainer qualification above does not invent an author self-review
attestation. Final merge readiness remains pending hosted CI and review.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-16 14:31:31 -07:00
89b229ce5f fix(kompress): time each model-load stage and stop false not-ready warnings during warm-up (#4195)
## Description

After every proxy start, the post-startup Kompress warm-up
(`HeadroomProxy._start_kompress_background_warmup`) takes several
seconds; 22.8 s was seen on one start. Requests that arrive during that
time caused three problems:

1. **A false connectivity warning.** `ContentRouter._try_ml_compressor`
logged `Kompress model not ready; requests will not be compressed. Check
HuggingFace connectivity ...` at WARNING, although the model was only
still loading.
2. **A second loader thread.** The warm-up thread was not in
`kompress_compressor`'s per-model loader registry (`_download_threads`).
`ensure_background_download` therefore started a second thread, which
logged `downloading model ... in the background`, waited on
`_kompress_lock` until the warm-up finished, and then returned the
warm-up's cache entry.
3. **Misleading or missing load logs.** `_load_kompress_onnx` logged
`Downloading Kompress ONNX model` before any cache lookup, even when
every file came from disk. The `Kompress ONNX loaded` line gave no
breakdown, so there was no way to tell why a load took 22.8 s.

This PR has one commit for each problem, plus a fourth that fixes an
issue Devin Review found in the second:

- `feat(kompress)`: the `Kompress ONNX loaded` line now gives wall-clock
and loading-thread CPU milliseconds for each load stage, and the start
message says "Loading".
- `fix(kompress)`: the warm-up thread is registered as the model's
loader before it starts.
- `fix(router)`: while a loader thread is running, the router logs at
INFO, once per router, that the model is still loading. The WARNING
remains for the case where nothing is loading the model.
- `fix(kompress)`: a warm-up that replaces a live prefetch in the
registry waits for it, so the prefetch stays covered.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/kompress_compressor.py`:
- `_timed_stage` / `StageTimings`: record wall-clock and
`time.thread_time()` milliseconds for each stage.
- `_create_onnx_session(stage_ms=...)` times `artifact_resolve`,
`onnxruntime_import`, `session_build` and `smoke_run`. When a candidate
fails its smoke run and the next one is tried, both attempts are added
to the totals.
- `_load_kompress_onnx` also times `transformers_import` and
`tokenizer_load`, and logs the stages on the `Kompress ONNX loaded`
line. Its start message now says "Loading" instead of "Downloading".
- `start_kompress_warmup_thread(model_id, warm)` (new): starts the
`kompress-warmup` thread and registers it in `_download_threads` under
`_download_threads_lock` before it runs. This covers the warm-up's
initial delay too. If a prefetch or download thread is already
registered, the warm-up takes its slot but waits for that thread to
finish before running `warm()`. A live prefetch therefore can't drop out
of the registry and be duplicated after a failed warm-up, and the two
don't fetch the same files at once (Devin Review). After the warm-up
ends, even if it failed, the next `ensure_background_download` replaces
the entry as before.
- `headroom/proxy/server.py`: `_start_kompress_background_warmup` starts
its thread with `start_kompress_warmup_thread` instead of
`threading.Thread(...)`.
- `is_kompress_load_in_progress(model_id)` and
`KompressCompressor.is_loading()` (new): true while a registered
prefetch, download or warm-up thread is alive.
- `headroom/transforms/content_router.py`: in the not-ready branch, the
router logs at INFO once while `is_loading()` is true, and keeps the
original WARNING once when no loader is running (for example, a failed
download in its retry backoff; `_background_download` has already logged
the actual error). Compressors without `is_loading` keep the WARNING.
- Tests:
- `tests/test_kompress_failsafe.py`: stage totals include a candidate
rejected by its smoke run, and the load log names every stage.
- `tests/test_proxy/test_kompress_background_warmup.py`: a request
during the warm-up starts no second loader, and one after a failed
warm-up still does.
- `tests/test_kompress_download_backoff.py`: a warm-up that replaces a
live prefetch waits for it. No download starts while the prefetch runs,
and one can start once both have ended without a model.
- `tests/test_content_router_kompress_not_ready_log.py` (new): INFO
while loading, WARNING with no loader, INFO and then WARNING after a
failed load, WARNING for a compressor without `is_loading`, and a
registered warm-up thread counting as loading.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_content_router_kompress_not_ready_log.py tests/test_kompress_failsafe.py tests/test_proxy/test_kompress_background_warmup.py tests/test_kompress_download_backoff.py tests/test_kompress_request_nonblocking.py tests/test_kompress_preload_deferral.py tests/test_router_registry_smartcrusher.py tests/test_content_router_ml_stage_deadline.py tests/test_content_router_concurrency.py tests/test_content_router_exclude_tools.py tests/test_transforms/test_content_router.py -q
======================= 184 passed, 3 warnings in 21.25s =======================

$ ruff check headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py headroom/proxy/server.py tests/test_kompress_failsafe.py tests/test_proxy/test_kompress_background_warmup.py tests/test_content_router_kompress_not_ready_log.py tests/test_kompress_download_backoff.py
All checks passed!
$ ruff format --check (same 7 files)
7 files already formatted
$ mypy headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py headroom/proxy/server.py
Found 9 errors in 2 files (checked 3 source files)
# All 9 are in headroom/relevance/embedding.py and headroom/providers/model_metadata.py,
# which mypy reaches through imports. origin/main (f1bc2609) gives the same 9;
# the three changed files report no errors.
```

Regression check: with each fix reverted, its new tests fail.

- Without the `server.py` change,
`test_request_during_warm_up_does_not_start_a_second_loader` fails with
`assert ['test/warmup-model'] == []`.
- Without the `content_router.py` change,
`test_still_loading_is_info_not_a_connectivity_warning` and
`test_warns_once_the_load_ends_without_a_model` fail.
- Without the wait in `start_kompress_warmup_thread`,
`test_warm_up_waits_for_a_live_loader_it_replaces` fails with `warm-up
ran while the prefetch was still live`.

Each commit was also tested on its own; at each one the only failure was
`TestLazyImports::test_unloadable_native_dep_degrades_instead_of_raising`,
which fails the same way on `origin/main` in the same run.

## Real Behavior Proof

- Environment: macOS 27 on Apple M5, Python 3.11.x (uv tool install of
headroom-ai with onnxruntime 1.31.0), real
`chopratejas/kompress-v2-base` ONNX model already in the HF cache.
`origin/main` (f1bc2609) and the PR head (4d881f47, rerun on 6b9393e9
with the same output) were run in separate worktrees with the same
interpreter.
- Exact command / steps: a script calls the real
`HeadroomProxy._start_kompress_background_warmup` with a real
`ContentRouter` (`HEADROOM_KOMPRESS_WARMUP=1`,
`HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS=0`). After 0.5 s it sends one
block through `router._try_ml_compressor` (a request arriving
mid-warm-up), waits for the load, and prints the Kompress log lines plus
any loader threads still alive. Run as `PYTHONPATH=<worktree> python
warmup_proof.py` in each worktree.
- Observed result: on `origin/main`, the request logs the connectivity
WARNING and starts a second "background download" loader, and the load
says "Downloading". On the PR head there is no WARNING and no second
loader, the load says "Loading", and the loaded line gives each stage.
Output below.
- Not tested: a full `headroom proxy` restart with live client traffic,
the CoreML and PyTorch backends (the stage timing covers the ONNX path
only), and a cold HF cache.

```text
===== before (origin/main f1bc2609)
INFO    headroom.transforms.kompress_compressor - Downloading Kompress ONNX model from chopratejas/kompress-v2-base ...
INFO    headroom.transforms.kompress_compressor - Kompress: downloading model chopratejas/kompress-v2-base in the background ...
WARNING headroom.transforms.content_router - Kompress model not ready; requests will not be compressed. Check HuggingFace connectivity or pre-download: headroom-ai[ml] + first-run warmup.
INFO    headroom.transforms.kompress_compressor - Kompress ONNX loaded: chopratejas/kompress-v2-base backend=onnx
INFO    headroom.transforms.kompress_compressor - Kompress: background model download complete for chopratejas/kompress-v2-base
INFO    headroom.proxy - Kompress: warmed in the background in 5241 ms (backend onnx)
ready=True loader threads still alive: ['kompress-canary']
===== after (PR head 4d881f47)
INFO    headroom.transforms.kompress_compressor - Loading Kompress ONNX model from chopratejas/kompress-v2-base ...
INFO    headroom.transforms.content_router - Kompress model still loading in the background; requests pass through uncompressed until it is ready.
INFO    headroom.transforms.kompress_compressor - Kompress ONNX loaded: chopratejas/kompress-v2-base backend=onnx total_ms=3212/2212 stages_ms(wall/cpu): artifact_resolve=2/1 onnxruntime_import=0/0 session_build=424/344 smoke_run=27/27 transformers_import=2687/1770 tokenizer_load=71/68
INFO    headroom.proxy - Kompress: warmed in the background in 3212 ms (backend onnx)
ready=True loader threads still alive: ['kompress-canary']
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This changes log lines and
loader-thread bookkeeping inside the existing Kompress warm-up; there is
no new flag.
- Minimum rollout channel: N/A (nothing rollout-managed).
- Stable/default behavior changed: Log output only, plus no duplicate
loader thread during warm-up. One WARNING becomes INFO while a load is
in progress. When, how and what Kompress compresses is unchanged.
- Kill switch / disable path: Existing `HEADROOM_KOMPRESS_WARMUP=0`
still disables the warm-up; no new switch is needed.
- Unsafe override required: No.
- Qualification impact: None. Compression results and request handling
are unchanged.
- Rollback path: Revert the PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A (log output is shown above).

## Additional Notes

- Documentation: no docs describe these log lines, so there is nothing
to update.
- Type checking: unchecked because `mypy headroom` was not run in full;
on the changed files, the only errors are the 9 that `origin/main` also
has (see Test Output).
- What the timings showed: in a process doing nothing else, `from
transformers import AutoTokenizer` (which also imports torch) takes most
of the ~3 s load. On the proxy start that took 22.8 s, the load
overlapped with request processing in the same process. The new wall/cpu
figures should show whether that time went to contention or to work; if
it was contention, importing `AutoTokenizer` earlier could help, subject
to the #1908 constraint on native init at startup.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-15 16:34:21 -05:00
ca6ba3486c fix(ccr): stream the buffered CCR upstream request so idle-killing networks cannot cut it (#4206)
## Description

The buffered CCR path flips a streaming Claude Code turn to `stream:
false` upstream so the proxy holds the whole reply before serving
`headroom_retrieve`. With `stream: false`, Anthropic sends no byte until
the generation is complete. A middlebox that drops HTTP connections idle
for ~60s (TLS-inspecting antivirus, VPN, corporate proxy; TCP keepalive
does not reset an HTTP-layer idle timer) closes the connection first:
each attempt fails at ~60s with `Server disconnected without sending a
response`, the client gets a 502 after ~186s, and Claude Code resends
the same body forever. The CCR continuation that serves the retrieval
(`anthropic_ccr_continuation`, its own `http_client.post`) is a second
`stream: false` call and is cut the same way. Without Headroom the
client streams and the same turns complete.

This PR keeps the buffered design and makes only its upstream leg
stream. The buffered request and every continuation built from it (CCR,
memory, turn hook) go to first-party Anthropic as `stream: true`, and
the event stream is folded back into the Message JSON the buffered path
already reads, so retrieval, memory, turn hooks, usage accounting and
SSE resynthesis see exactly what they saw before. Bytes (deltas and
pings) now flow for the whole generation, so an idle timer never fires.
The fold reuses `AnthropicSSEEnvelope`, the reconstruction #3130 and
#3266 already use on this path.

Closes #4205

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: new
`_send_buffered_as_stream(send, headers, body, *, url, request_id)`. It
sends `{**body, "stream": true}` with `Accept: text/event-stream` (one
Accept value, any casing replaced) and folds the reply:
- not a 200 event stream (an HTTP error, or an upstream that answered
JSON anyway): returned unchanged;
  - a whole stream: a 200 JSON response holding the folded Message;
- an in-band `error` event: the event as the body, with its own HTTP
status (529 `overloaded_error`, 429 `rate_limit_error`, other documented
types to their status, else 500);
- a stream that ended before `message_stop` or with a block open: the
existing strict 502 `upstream_protocol_error` of
`_adapt_event_stream_to_json` (#3266);
- a complete stream holding a known event the fold cannot read (an
unknown delta type): logged, and the original `stream: false` body is
sent instead. One more generation, but never a 502 the client would
resend forever.
A connection dropped mid-stream raises inside `send`, exactly as for a
`stream: false` reply. An in-band error arrives after upstream already
answered 200, so `_retry_request` does not retry it; the client gets the
status and retries as it would on a direct stream.
- It is used at the four places Headroom sends a body it made
non-streaming: the buffered CCR request (only when
`buffered_stream_ccr`; a client's own `stream: false` request still goes
through `_retry_request` with its original bytes), the CCR continuation
(the `prepare_outbound_body_bytes` + `log_outbound_request` +
`http_client.post` sequence moved into a local `_post_continuation`,
unchanged otherwise, so a fallback attempt logs its own outbound line),
the memory continuation and the turn-hook continuation.
- Gates: the body is not already streaming, the upstream is first-party
Anthropic (`is_custom_anthropic_base_url(url)` is false; the
host-equality check tool-search gating already uses), and the kill
switch `HEADROOM_BUFFERED_UPSTREAM_STREAM` is not
`0`/`false`/`no`/`off`. Copilot and other gateways keep the `stream:
false` request: the fold is checked against Anthropic's own stream, and
a gateway dialect it cannot read would cost every turn a second
generation.
- `headroom/proxy/runtime_env.py`: `HEADROOM_BUFFERED_UPSTREAM_STREAM`
is a registered knob in `RUNTIME_ENV_KNOBS`, read per request through
`runtime_env.getenv`, so a reusing `headroom wrap` pushes it to the
running proxy and `POST /admin/runtime-env` sets it with no restart.
Listed in the `security-model.mdx` allowlist.
- `headroom/proxy/anthropic_wire.py`: `fold_anthropic_stream(raw)`
parses with `AnthropicSSEEnvelope.parse` and trims each block to the
non-streaming Message shape: a null member leaves every block,
`thinking` keeps `type`/`thinking`/`signature`, `redacted_thinking`
keeps `type`/`data`. Claude Opus 5.5 streams a thinking
`content_block_start` with `estimated_tokens: null`; the CCR
continuation resends the folded turn as history, and Anthropic answers
any block carrying that member with 400 `Extra inputs are not
permitted`.
- `headroom/proxy/anthropic_wire.py` `_response_from_events`:
`message_delta.usage` is merged only where non-null, as the Anthropic
SDK's `MessageStream` does. Before, a null count there overwrote the
real `message_start` count (input and cache tokens) in the reconstructed
message.
- `headroom/proxy/buffered_ccr_response.py`: the status-to-error-type
table moves out of `_anthropic_error_type` into `ANTHROPIC_ERROR_TYPES`
so the handler can invert it; `_anthropic_error_type` returns the same
values.
- Unchanged on purpose: `body["stream"] = False` and its mutation reason
`ccr_streaming_retrieve_buffered_non_stream` stay the buffered path's
internal representation (every later step, the response-cache guard and
the #3587 log counts key on them), and the `using buffered stream:false
upstream request` log line keeps its text.
- Tests: new `tests/test_buffered_ccr_upstream_stream.py` (16 tests):
the fold equals the Anthropic Python SDK's `accumulate_event` result on
a stream with thinking, signature, redacted thinking, text, citations,
split `input_json_delta`, pings and a null usage count; stream-only
members are dropped; truncated, in-band error and unknown-delta
outcomes; the send helper's gates (kill switch, gateway URL, already
streaming), passthrough of non-stream replies, 529 for an in-band
overload, 502 for a truncated stream, the `stream: false` fallback; and
through the handler, the buffered turn and its CCR continuation both go
upstream `stream: true` with `Accept: text/event-stream` and the
continuation's resent thinking block carries only its schema, while a
client's own `stream: false` request goes out `stream: false` with
`Accept: application/json` and its original bytes.
- Existing tests that pinned `stream: false` on the wire for a buffered
turn now pin `stream: true` (`test_anthropic_streaming_ccr_retrieve.py`,
`test_ccr_buffered_stream_signed_thinking.py`,
`test_proxy_response_cache_replay.py`,
`test_no_ccr_disables_response_handling.py`).
`test_anthropic_buffered_ccr_usage.py` now answers the buffered turn
with an event stream, so usage booking is checked through the fold.
`test_buffered_ccr_accept_header.py` (#3078) pins the `stream: false`
leg it is about with the kill switch; custom upstreams such as Copilot
always take that leg.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) - ran on the 3 changed
source files only, see below
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_buffered_ccr_upstream_stream.py -q
16 passed, 1 warning in 2.89s

# every test file that touches the buffered CCR path, the CCR continuation or SSE reconstruction
$ pytest tests/test_buffered_ccr_upstream_stream.py tests/test_sse_safeguards_passthrough.py \
    tests/test_buffered_ccr_salvage.py tests/test_anthropic_buffered_ccr_usage.py \
    tests/test_buffered_ccr_grace_window.py tests/test_anthropic_buffered_sse.py \
    tests/test_ccr_buffered_stream_signed_thinking.py tests/test_toin_feedback.py \
    tests/test_proxy_response_cache_replay.py tests/test_proxy/test_anthropic_buffered_timeout.py \
    tests/test_anthropic_auto_mode_passthrough.py tests/test_proxy/test_openai_responses_ccr.py \
    tests/test_sse_thinking_blocks.py tests/test_streaming_sse_malformed_events.py \
    tests/test_route_advice.py tests/test_memory_tool_stream.py \
    tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy_byte_faithful_forwarding.py \
    tests/test_buffered_ccr_accept_header.py tests/test_anthropic_buffered_memory_continuation.py \
    tests/test_anthropic_ccr_workspace_unbound.py tests/test_no_ccr_disables_response_handling.py \
    tests/test_proxy_ccr.py tests/test_ccr_inline_resolve_handlers.py \
    tests/test_proxy/test_anthropic_ccr_raise.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py \
    tests/test_nonstream_sse_policy.py tests/test_ccr_response_handler_extra.py \
    tests/test_compress_route_tokenizer_by_model.py -q
855 passed, 6 warnings in 57.94s

# the updated assertions fail on main's source (they pin stream:true on the wire)
$ git checkout upstream/main -- headroom/ && pytest tests/test_anthropic_buffered_ccr_usage.py \
    tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_ccr_buffered_stream_signed_thinking.py \
    tests/test_no_ccr_disables_response_handling.py tests/test_proxy_response_cache_replay.py -q --tb=no
12 failed, 33 passed, 1 warning in 13.64s

# whole suite
$ pytest tests -q
18 failed, 16435 passed, 583 skipped, 119 warnings in 1226.31s (0:20:26)
# 16 of the 18 fail the same way on unmodified main in the same environment
# (tests/test_proxy_uds.py x11, tests/test_cli/test_wrap_persistent.py x4,
#  tests/test_pricing_from_litellm.py groq/llama-guard-3-8b x1); the other 2
# (test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request,
#  test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children) pass when run alone on this branch.

$ ruff check .
All checks passed!
$ ruff format --check .
1790 files already formatted
$ mypy headroom/proxy/anthropic_wire.py headroom/proxy/buffered_ccr_response.py headroom/proxy/handlers/anthropic.py
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 27.0.1 arm64, Python 3.11.13. Upstream `main`
f1bc2609 versus this branch, same venv.
- Exact command / steps: `repro_buffered_idle_cut.py` (full script in
#4205) runs the real proxy app (`create_app(ProxyConfig(...))` +
`TestClient`, real `_retry_request`, real `http_client`) against a fake
Anthropic upstream that takes 5s per reply (stream: a delta or ping
every 0.5s; non-stream: silent for 5s, then JSON), reached through a TCP
relay that closes any connection silent for 2s, a scaled-down 60s
middlebox. Scenario A: a streaming turn with a redeemable marker
(buffered path), slow reply. Scenario B: the model calls
`headroom_retrieve` in 0.3s and the continuation is slow. The script
points the first-party host check at the fake upstream and isolates
`HOME` and the savings ledger. `PYTHONPATH=<checkout> python
repro_buffered_idle_cut.py 2>/dev/null`
- Observed result: on main both scenarios fail and every upstream
request goes out `stream: false` and is cut; on the branch both complete
in about the generation time with no cut; with
`HEADROOM_BUFFERED_UPSTREAM_STREAM=0` on the branch the main behaviour
returns.

```text
# main f1bc2609
Scenario A (slow buffered turn)
  client: HTTP 200 after 10.1s -> FAIL: 20 keepalive pings, then {"type": "error", "error": {"type": "api_error", "message": "An error occurred while processing your request. Please try again."}}
  upstream requests: ['turn stream=false', 'turn stream=false', 'turn stream=false']
  connections the relay cut for idling: 3

Scenario B (slow CCR continuation)
  client: HTTP 502 after 2.3s -> FAIL: 0 keepalive pings, then {"type": "error", "error": {"type": "api_error", "message": "Unable to safely complete streamed CCR retrieval."}}
  upstream requests: ['retrieve call stream=false', 'continuation stream=false']
  connections the relay cut for idling: 1

# proxy log on main
WARNING - Request failed (attempt 1), retrying in 1322ms: Server disconnected without sending a response.
WARNING - Request failed (attempt 2), retrying in 2536ms: Server disconnected without sending a response.
ERROR - [hr_1791973393_000001] Request failed: RemoteProtocolError: Server disconnected without sending a response.
ERROR - CCR: Continuation API call failed: RemoteProtocolError: RemoteProtocolError('Server disconnected without sending a response.')
WARNING - [hr_1791973403_000002] CCR: Buffered streaming response still contains an unresolved headroom_retrieve after handling; failing closed

# this branch
Scenario A (slow buffered turn)
  client: HTTP 200 after 5.2s -> OK: the client got the final answer
  upstream requests: ['turn stream=true']
  connections the relay cut for idling: 0

Scenario B (slow CCR continuation)
  client: HTTP 200 after 5.3s -> OK: the client got the final answer
  upstream requests: ['retrieve call stream=true', 'continuation stream=true']
  connections the relay cut for idling: 0

# this branch, HEADROOM_BUFFERED_UPSTREAM_STREAM=0
Scenario A (slow buffered turn)
  client: HTTP 200 after 9.0s -> FAIL: 16 keepalive pings, then {"type": "error", "error": {"type": "api_error", "message": "An error occurred while processing your request. Please try again."}}
  upstream requests: ['turn stream=false', 'turn stream=false', 'turn stream=false']
  connections the relay cut for idling: 3

Scenario B (slow CCR continuation)
  client: HTTP 502 after 2.4s -> FAIL: 0 keepalive pings, then {"type": "error", "error": {"type": "api_error", "message": "Unable to safely complete streamed CCR retrieval."}}
  upstream requests: ['retrieve call stream=false', 'continuation stream=false']
  connections the relay cut for idling: 1
```

- Not tested: this branch has not sent traffic to api.anthropic.com. The
fold is checked against the Anthropic Python SDK's own accumulator and
against hand-built streams, not against live replies. For context, the
same transport change (a different fold implementation with the same
contract) has run as a downstream patch on the 0.39.0 wheel and was
checked live: 56 turns across Opus, Sonnet, Haiku and Fable models
(thinking, parallel `tool_use`, `web_search` with citations,
`max_tokens` cutoffs) matched the non-streaming reply's top-level, block
and usage shapes; 28 folded replies resent as history all returned 200;
behind a relay that cuts a connection after 20s of silence, a 409s
thinking turn completed (longest silence 2.1s) while the `stream: false`
path looped on 502s. The OpenAI Responses buffered path has the same
flip and is not changed here. Gateways (Copilot, custom base URLs) are
not changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This is a transport fix, not a
`rollout.py` feature.
- Minimum rollout channel: N/A; it applies on every channel.
- Stable/default behavior changed: Yes, for first-party Anthropic only.
A buffered CCR turn and its CCR, memory and turn-hook continuations now
go upstream `stream: true` with `Accept: text/event-stream`, and the
proxy folds the reply into the same Message JSON. Request content is
otherwise identical, so there is no compression, cache or token-count
effect. The client still gets the buffered response it got before
(keepalive pings, then the resynthesized stream), so time to first byte
(#3587) is unchanged. A client's own `stream: false` request, every
custom upstream, and the OpenAI paths are unchanged. One shared change:
`message_delta` usage nulls no longer overwrite `message_start` counts
in `AnthropicSSEEnvelope`, so a re-rendered stream on the streaming path
now carries the real count where it carried `null`.
- Kill switch / disable path: `HEADROOM_BUFFERED_UPSTREAM_STREAM=0`
(also `false`, `no`, `off`), a runtime-env knob read on every request:
set it in the proxy environment, let a reusing `headroom wrap` push it,
or `POST /admin/runtime-env` (no restart). Every buffered request then
goes out `stream: false` exactly as on main
(`test_a_propagated_kill_switch_keeps_the_request_non_streaming`).
- Unsafe override required: No.
- Qualification impact: transport only. A complete stream the fold
cannot read costs one extra generation (logged as `Could not fold the
upstream event stream ...; sending the request again as stream:false`);
that should not happen with the first-party API today and is the safe
answer to a future delta type.
- Rollback path: set `HEADROOM_BUFFERED_UPSTREAM_STREAM=0` (no restart
needed), or revert these commits.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, proxy transport change.

## Additional Notes

- Documentation: no user-facing doc describes the buffered request's
wire shape, so none changed. The kill switch is documented in the
helper's docstring.
- Relation to #3587: that issue is the latency side of the same flip
(time to first byte becomes the whole generation, and the flip fires on
turns where retrieval never redeems). This PR does not change when the
buffered path is taken or what the client sees first; it fixes the
availability side, where a turn longer than a middlebox's idle limit
never arrives. #3587's asks (a stricter redeemability gate, a switch
that disables buffering) are independent of this change.
- Relation to #3013: #3013 replaces the buffered path with event-level
splicing of the client stream behind `ccr_event_streaming` (default on)
and keeps the buffered `stream: false` request as the qualified fallback
when that flag is off. This PR is the small transport fix for the
buffered design as it is on main: it does not touch the
`buffered_stream_ccr` gating or the log line #3013 rewrites, and it adds
no new conflict with #3013 (a `git merge-tree` of #3013's head against
main and against this branch shows the same 3 conflict hunks in
`anthropic.py` and 2 in `test_anthropic_streaming_ccr_retrieve.py`). If
#3013 lands, its buffered fallback gets a streamed upstream leg from
this change for free.
- OpenAI Responses: that handler has the same flip (`openai.py`, `using
buffered stream:false`) and the same exposure; it is left for a
follow-up because its fold is a different event format.
- `compaction_delta` and other delta types `AnthropicSSEEnvelope` does
not reconstruct take the `stream: false` fallback rather than a guess;
teaching the envelope a new delta type removes that extra generation.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4206"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
- Review fix 61b42465: the kill switch is a registered runtime-env knob
(see Changes Made), so the rollback reaches a proxy that `headroom wrap`
reuses. Useful here because a fold failure can cost a second generation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-15 16:29:46 -05:00
22a1296cfa fix(copilot): send VS Code next-edit requests to the completions host (#4188)
## Description

VS Code Copilot Chat serves ghost text and next edits by posting
`copilot-nes-*` models (seen live: `copilot-nes-pandia-25`, Copilot Chat
0.69) to
`{github.copilot.advanced.debug.overrideProxyUrl}/chat/completions`.
`headroom wrap vscode` points that override and the CAPI override at the
same proxy, so these requests land on the OpenAI chat handler and were
forwarded to CAPI. CAPI answers `400 model_not_supported` for them on
both the per-plan host and the generic host; only the Copilot
completions host serves them. Result: behind Headroom, VS Code shows no
ghost text and no next-edit suggestions, with nothing in the UI saying
why.

Found during the live Copilot Business end-to-end run for #4155 / #4154;
it is on `main` and independent of both. #4144 does not cover it (it
handles the legacy `/v1/engines/*/completions` path only).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/copilot_auth.py`: `copilot_nes_base_url(model,
upstream_base_url)` returns the completions host for a `copilot-nes-*`
request aimed at a public CAPI host, via the existing
`copilot_completions_base_url()` resolver (operator
`GITHUB_COPILOT_PROXY_URL` > advertised `endpoints.proxy` >
`copilot-proxy.githubusercontent.com`), the same resolver legacy
`/v1/engines` completions routing uses. It returns `None` for any other
model, for an upstream that already is a completions host, and for GHE /
custom deployments, whose traffic must stay inside them.
- `headroom/proxy/handlers/openai.py`: the chat-completions handler
applies it right after resolving the upstream, unless the caller chose
one with `x-headroom-base-url` (same rule as
`select_passthrough_base_url`). URL, auth, cache key, and both buffered
and streaming arms use the rerouted base.
- `tests/test_proxy/test_copilot_nes_routing.py`: through `create_app`
at the send seams: NES goes to the completions host with minted Copilot
credentials (buffered and streaming); follows an advertised
`endpoints.proxy`; chat models stay on the plan host; a GHE tenant keeps
its NES traffic; a completions-host target is not moved; a caller-chosen
base URL is left alone.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_proxy/test_copilot_nes_routing.py
7 passed
# on main (fix reverted): the two NES routing tests fail with
#   assert 'https://api.business.githubcopilot.com/chat/completions' == 'https://copilot-proxy.githubusercontent.com/chat/completions'

$ pytest -q tests/test_proxy/ tests/test_copilot_vscode_completions_routing.py tests/test_copilot_auth.py \
    tests/test_proxy_copilot_auth_hooks.py tests/test_provider_proxy_routes.py tests/test_copilot_native_mode.py \
    tests/test_copilot_subscription_smoke.py tests/test_cli/test_wrap_vscode.py
698 passed

$ uvx ruff@0.16.8 check <touched files> && uvx ruff@0.16.8 format --check <touched files>
All checks passed! / 3 files already formatted
$ mypy headroom/copilot_auth.py headroom/proxy/handlers/openai.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, native core, live GitHub Copilot
Business seat (2026-10-13), VS Code with Copilot Chat 0.69, isolated
`HEADROOM_WORKSPACE_DIR`, branch merged with #4155 and #4154 on current
`main`.
- Exact command / steps: `headroom copilot-auth login`; `headroom wrap
vscode --configure` (dedicated proxy on the plan host
`api.business.githubcopilot.com`); in VS Code, type in a Python function
body and wait for ghost text. Then the same `copilot-nes-pandia-25`
request sent directly to each host with the exchanged Copilot token.
- Observed result: without the fix, the proxy forwarded NES to
`https://api.business.githubcopilot.com/chat/completions` → `400
{"error":{"message":"The requested model is not
supported.","code":"model_not_supported"}}` (3 of 3), no ghost text.
Direct probe: `api.business.githubcopilot.com` 400,
`api.githubcopilot.com` 400, `proxy.business.githubcopilot.com` 200,
`copilot-proxy.githubusercontent.com` 200. With the fix, NES went to
`https://copilot-proxy.githubusercontent.com/chat/completions` → 200 (4
of 4), ghost text appeared in VS Code and was accepted; chat, Claude and
Agent-mode turns stayed on the plan host.
- Not tested: an Enterprise-plan seat (same code path); a GHE
(`*.ghe.com`) tenant live (covered by unit test).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: `copilot-nes-*` chat requests aimed
at a public CAPI host now go to the Copilot completions host instead of
failing with 400.
- Kill switch / disable path: `GITHUB_COPILOT_PROXY_URL` pins the
completions host; revert the commit.
- Unsafe override required: no.
- Qualification impact: Copilot routing and proxy suites.
- Rollback path: revert.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Docs: no doc change; `docs/content/docs/vscode-copilot.mdx` does not
describe NES routing.
- Follow-ups, not in this PR: (1) in the `wrap vscode` lane the
dedicated proxy does no token exchange of its own, so no
`endpoints.proxy` is recorded and NES uses
`copilot-proxy.githubusercontent.com` rather than the plan's
`proxy.business.githubcopilot.com`; GitHub's subscription-routing
guidance only blocks `*.individual.githubcopilot.com`, so this should
pass a plan-routed firewall, but wrap could pin the advertised proxy
host into the dedicated proxy as it does the API host in #4155. (2) NES
requests now run through the chat pipeline (compression, memory
injection when `--memory` is on, turn hooks); next-edit prompts are
latency-sensitive and memory context is probably unwanted there.
- Independent review (code-reviewer subagent) found no blockers; its
three SHOULDs (GHE tenant leak, caller-chosen base URL override,
completions-host target moved) are fixed and each has a test.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4188"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-13 15:49:02 -07:00
Tejas ChopraandClaude Opus 5.5 72302a3998 fix(proxy): say why proxy.log is missing and why a knob or extension was ignored (#4088)
## Description

Several failures an operator needs to see are swallowed silently or
logged only at DEBUG. This PR makes each of them visible with one
warning. It does not change defaults or control flow.

1. **Missing runtime log.** `_setup_file_logging` caught `OSError` with
`pass`. On a read-only workspace, a wrong owner, or a file where the
logs directory belongs, `proxy.log` simply never appeared and nothing
said why. It now logs `Headroom runtime log disabled: cannot write the
log file: <OSError>. Point HEADROOM_WORKSPACE_DIR at a writable
directory to restore proxy.log.` The OSError text names the path. The
symlink case just above it already warned this way.
2. **Six import-time numeric knobs.** Each one fell back to its default
on an unparseable value, without a word:
   - `HEADROOM_COMPRESSION_TIMEOUT_SECONDS`
   - `HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS`
   - `HEADROOM_EAGER_PRELOAD_TIMEOUT_SECONDS`
   - `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS`
   - `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS`
   - `HEADROOM_COMPRESSION_CACHE_MAX_ENTRIES`

Each now logs `Ignoring invalid NAME='value'; using DEFAULT` once, at
import. That matches the idiom in `hard_watchdog.py`.
3. **Extension discovery failure.** When entry-point enumeration fails,
every proxy extension is dropped, Enterprise plugins included. That was
logged at DEBUG. It is now a warning: `proxy extensions: entry-point
enumeration failed (<describe_exception>), so no proxy extensions will
be installed`. `describe_exception` gives the exception types and code
locations, never the message, and no traceback is logged at any level.
It runs at startup only: once in `install_all`, and once more for the
CLI banner.
4. **Pricing lookups in `cost.py`.** `_get_list_price`,
`_get_output_price`, `_get_cache_prices` and the cache-rate lookup in
the savings summary returned `None` on any exception. A pricing bug
therefore showed up as a $0 cost with no trace. All four, plus the
existing `estimate_cost` path, go through one helper,
`_warn_pricing_once(lookup, what, model, exc)`. Example line: `Failed to
get pricing for model 'gpt-4o': RuntimeError at
headroom/proxy/cost.py:... in estimate_cost`. It logs the lookup name,
`safe_id(model)` and `describe_exception(exc)` (types and code
locations, never the message), at WARNING only. No level carries the raw
exception or a traceback, because a pricing backend error can quote
provider responses or credentials. Dedupe is by `(lookup, model)`
through the shared `WarnOnce(1024, "unpriceable models")`. Each pair
warns at most once an hour (the `WarnOnce` window). Past 1024 models,
one overflow notice is logged, and a key is not used up while WARNING is
disabled.
5. **Malformed `HEADROOM_OPENAI_TOOL_SEARCH_MODELS` regex.** The
override was ignored on every request with no log. It now warns once per
bad pattern, and still falls back to the version gate as before. A valid
pattern re-arms the warning, so bad A, then valid B, then bad A again
warns twice.

None of the new lines logs header values, request content or
credentials. They log only knob names, their raw values (operator-set
env), model names, regex patterns, exception type names, the `re.error`
text, and `OSError` text naming a local path. No exception message or
traceback from pricing or extension discovery reaches any log level.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`:
- add `_warn_invalid_env_number(name, default)` and call it from the six
import-time `except ValueError` branches;
  - log the `OSError` in `_setup_file_logging`;
- add a once-per-pattern warning for a malformed
`HEADROOM_OPENAI_TOOL_SEARCH_MODELS`.
- `headroom/proxy/extensions.py`: entry-point enumeration failure logs
at warning with `describe_exception(exc)` instead of at debug.
- `headroom/proxy/cost.py`:
- the four silent pricing `except Exception` branches call
`_warn_pricing_once`;
- `_warn_pricing_once(lookup, what, model, exc)` logs `safe_id(model)`
plus `describe_exception(exc)` at WARNING, with nothing at DEBUG. It
dedupes through `log_safety.WarnOnce`.
- `tests/test_cost_pricing_warning_dedup.py`: the tests reset with a
fresh `WarnOnce`, and match the quoted model id (`for model
'glm-5.2':`).
- Regression tests. Each one attaches a handler to the module logger
directly, because caplog misses `headroom.*` records after proxy
startup. Each positive test fails on main and is paired with a negative
case that passes on both.
  - `tests/test_cache_breakpoint_diagnostics.py`:
- `test_runtime_log_says_why_it_is_missing` makes the logs directory a
regular file. It asserts one warning that names the path and
`HEADROOM_WORKSPACE_DIR`.
    - Negative: `test_runtime_log_stays_quiet_when_writable`.
  - `tests/test_openai_tool_search_deferral.py`:
- `test_malformed_override_warns_once_not_per_request` calls the gate 5
times with a bad pattern. It asserts exactly one warning and that the
version-gate result is unchanged.
    - Negative: a valid override is silent (same test).
- `test_valid_override_rearms_the_malformed_warning`: bad, valid, bad
gives two warnings.
  - `tests/test_proxy/test_compression_timeout_config.py`:
- `test_bad_env_says_it_was_ignored` reloads `helpers` with
`HEADROOM_COMPRESSION_TIMEOUT_SECONDS=not-a-number` and asserts the
exact warning.
- Negative: `test_valid_env_is_silent`. The existing fixture reloads
`helpers` afterwards.
  - `tests/test_proxy_extensions.py`:
- `test_discover_warns_when_entry_point_enumeration_fails` makes
`importlib.metadata.entry_points` raise. It asserts one WARNING that
says no extensions will be installed and names `RuntimeError` but not
the exception text.
    - Negative: `test_discover_is_quiet_when_nothing_is_installed`.
  - `tests/test_cost_cache_prices.py`:
- `test_cache_rate_failure_warns_once_per_model` makes `resolve_rates`
raise. It asserts two `_get_cache_prices` calls give one warning naming
the model and `RuntimeError`, without the exception text.
    - Negative: a resolvable model is silent (same test).
- `test_pricing_error_never_logs_its_text_at_any_level` runs the real
`CostTracker.estimate_cost('gpt-4o', 100, 10)` with a pricing error
whose text holds an Authorization credential canary, with
`headroom.proxy` at DEBUG. It formats every captured record with
`logging.Formatter` (which renders `exc_info`), asserts the canary is
absent and the result is `None`, and asserts the warning is present. On
the previous commit it fails: the formatted DEBUG traceback contains the
canary.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_cost_cache_prices.py -k never_logs   # previous commit 3202058c (test adapted to its old dedupe set)
E     RuntimeError: upstream rejected request (Authorization: Bearer sk-canary-7f3a9c)']
======================= 1 failed, 25 deselected in 3.48s =======================
$ pytest -q -p no:cacheprovider tests/test_pr208_changes.py tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_cost_tracker_totals.py tests/test_tool_search_client_deferral.py tests/test_cache_breakpoint_diagnostics.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_byte_faithful_forwarding.py tests/test_memory_tool_mode.py tests/test_codex_ws_compression_scheduler.py tests/test_proxy_savings_history.py tests/test_compression_cache_registry.py tests/test_proxy_extension_ordering.py tests/test_proxy_eager_preload_bind.py tests/test_perf_per_port_logs.py tests/test_cost_cache_prices.py tests/test_openai_tool_search_deferral.py tests/test_proxy_extensions.py tests/test_direct_chat_ccr_resolution.py tests/test_org_scale_limits.py tests/test_savings_ledger.py tests/test_cost_pricing_warning_dedup.py tests/test_proxy/test_compression_timeout_config.py tests/test_log_safety.py   # 64ff3f03
================== 517 passed, 1 skipped in 85.70s (0:01:25) ===================
$ pytest -q (same pricing/extension/regex/log/knob/log_safety files) after merging log-safety-helpers @ 0ce74a63   # 4ff57af4
======================== 139 passed, 1 skipped in 4.23s ========================
$ uvx ruff@0.16.8 check / format --check headroom/proxy/cost.py headroom/proxy/extensions.py tests/test_cost_cache_prices.py tests/test_cost_pricing_warning_dedup.py tests/test_proxy_extensions.py
All checks passed! / 5 files already formatted
$ mypy headroom/proxy/cost.py headroom/proxy/extensions.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, branch checkout at 4ff57af4
(log-safety-helpers merged at 0ce74a63), using the repo .venv. A temp
HOME, workspace and CODEX_HOME were used each time; the real HOME was
not touched.
- Exact command / steps:
1. Ran the shared smoke harness, which starts a real proxy from this
checkout against a local fake upstream and sends one Anthropic and one
OpenAI request.
2. Ran the real `CostTracker.estimate_cost('gpt-4o', 100, 10)` twice,
with every logger at DEBUG and a LiteLLM stand-in whose `cost_per_token`
raises an error quoting `Authorization: Bearer sk-canary-7f3a9c`.
3. The earlier forced-failure run for the knob, log-directory and regex
warnings (d83e7d5c, below) is unchanged by this commit.
- Observed result:
```text
branch: cleanup/proxy-core-a-surface-ignored-config @ 4ff57af4
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>):
(smoke dir: <tmp>)
```
```text
WARNING headroom.proxy: Failed to get pricing for model 'gpt-4o': RuntimeError at <string>:8 in cost_per_token <- headroom/proxy/cost.py:999 in estimate_cost
result: None None
```
- One payload-safe WARNING appears, with the exception type and code
location, and both calls return `None`.
- The canary appears in no output line at any level: `grep -c canary`
over the whole run, LiteLLM's own DEBUG chatter included, gives 0.
  - Earlier forced-failure run:
```text
branch: cleanup/proxy-core-a-surface-ignored-config @ d83e7d5c
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>):
(smoke dir: <tmp>)
```
The harness also printed two pre-existing WARNING lines, which I left
out above: offline-mode air-gap active, and the Kompress model prefetch
refused offline. Neither comes from this PR. Output of the direct
command:
```text
WARNING headroom.proxy: Ignoring invalid HEADROOM_COMPRESSION_TIMEOUT_SECONDS='abc'; using 30.0
WARNING headroom.proxy: Headroom runtime log disabled: cannot write the log file: [Errno 17] File exists: '<tmp>/ws/logs'. Point HEADROOM_WORKSPACE_DIR at a writable directory to restore proxy.log.
WARNING headroom.proxy: Ignoring invalid HEADROOM_OPENAI_TOOL_SEARCH_MODELS regex '[bad' (unterminated character set at position 0); using the gpt-5.4+ version gate
gate: True True
```
- The proxy serves both providers, and the normal path adds no new
warnings.
- Each forced failure gives exactly one warning, even though the regex
gate ran twice. The fallbacks are unchanged: 30.0, and the version gate
is True for gpt-5.4.
- Not tested: A real read-only filesystem, a real broken
`importlib.metadata` and a live pricing-catalog failure (the stand-in
above raises from the same call). Each goes through the same `except`
branch that the unit tests force.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; log output only.
- Stable/default behavior changed: No. Every fallback value and code
path is the same; only new WARNING lines appear, when something was
already failing.
- Kill switch / disable path: N/A. A pricing warning fires at most once
an hour per (lookup, model), through `WarnOnce`; the regex warning fires
once per bad pattern.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of a maintainer-requested codebase cleanup (dead code, duplication,
debuggability), split into small PRs for review. It is part of
maintainer-requested cleanup #4136.

Depends on #4138; its two files show in this diff until it merges. #4137
also edits `_setup_file_logging`, so whichever of the two merges second
rebases.

Docs are unchanged because no knob's meaning changed. The doc impacts
are recorded in the cleanup docs ledger for the follow-up docs PR. The
six import-time knob warnings have no test: testing them would need
`importlib.reload(helpers)`, which would reset the module globals other
tests depend on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 11:51:06 -07:00
01d9194854 fix(ccr): serve headroom_retrieve when the model calls it alongside client tools (#4034)
## Description

When the model calls `headroom_retrieve` in parallel with client tools,
`CCRResponseHandler.handle_response` hands the whole turn back for the
client to resolve (#839). The client has no `headroom_retrieve`, so it
can never resolve that call: Claude Code answers `Error: No such tool
available: headroom_retrieve`, and the model never gets the content it
asked for. Parallel calls are the normal shape for agentic clients. A
Claude Code subagent that greps and reads several files in one turn hits
this regularly, and users report the error as "happens sometimes,
usually with subagents".

Measured on one machine's Claude Code transcripts
(`~/.claude/projects`): 501 leaked `headroom_retrieve` calls since
2026-09-21. 484 of them were mixed turns (retrieve plus at least one
client tool in the same assistant message), and 435 of those were in
subagents. The other 17 were lone retrieves on the #3071 streaming path,
which this PR does not touch.

The client tools in a mixed turn have not run yet. This PR drops them
from the turn, serves the retrieval in the continuation exactly as for a
lone call, and lets the model decide again there, with the retrieved
content in hand. The continuation is what the client receives, whatever
it does: re-issue the client calls, change them, or withdraw them
because the retrieved content says not to. The client never sees
`headroom_retrieve` and never runs a call the model withdrew. Only when
the continuation request fails does the client get the model's own mixed
turn back (the #839 behaviour), since the model made no newer decision;
a continuation that succeeded is kept even if a later retrieval round
fails. A `headroom_retrieve` without a valid hash is never treated as a
client call: such a turn is handed back unchanged.

This applies to Anthropic and OpenAI chat only. In the Responses API a
`function_call` can be the item a `reasoning` item requires next, and a
Gemini `functionCall` part can carry the turn's thought signature, so
dropping a sibling there could make the continuation invalid. Those
providers keep the #839 pass-through unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `ccr/tool_calls.py`: new `drop_tool_calls(response, provider, calls)`.
It returns a shallow copy of the response without the given tool-call
objects, matched by identity. It handles Anthropic and OpenAI chat and
returns the response unchanged for any other provider.
- `ccr/response_handler.py`: on a mixed turn, `handle_response` trims
the client calls and continues the normal retrieval loop. A successful
continuation is returned as is. If the continuation request for a mixed
round fails, it returns the original mixed turn instead of the trimmed
one. When the provider is not trimmable, or one of the extra calls is a
`headroom_retrieve` without a valid hash, it keeps the old
skip-and-return. The model's own response object is not mutated.
- Tests: the #839 handler test now asserts the new behaviour for
Anthropic and OpenAI chat: one continuation carrying only the retrieve
pair, and the re-issuing continuation returned. New handler tests return
the continuation when it withdraws the client call in text, changes its
arguments, or re-issues it unchanged; return the original turn when the
continuation request fails; and keep a successful continuation when a
later retrieval round fails. A malformed `headroom_retrieve` keeps the
turn unchanged with no continuation. The Responses pass-through is
pinned as unchanged. On the streaming SSE path, a re-issued client tool
streams without `headroom_retrieve`, a continuation that withdraws the
client tool streams the continuation, and a failed continuation streams
both original blocks as a 200.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/ -q -k "ccr or CCR or retrieve or mixed"
1168 passed, 24 skipped, 15064 deselected, 3 warnings in 178.17s

$ uv run --frozen --extra dev pytest <every test file naming headroom_retrieve, CCR_TOOL_NAME or ccr_response_handler>  # at 03e8d876
1101 passed, 63 skipped, 3 warnings in 86.75s

# against main (tests kept): the serve tests fail
FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_serves_retrieval
FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_serves_retrieval_openai
FAILED tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py::test_mixed_ccr_and_client_tool_serves_retrieval_then_streams_reissued_tool

# against 95dc4975 (handed the original turn back unless every client call was re-issued unchanged): the continuation tests fail
FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_returns_the_continuation[withdrawn]
FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_returns_the_continuation[changed]
FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_later_failure_keeps_the_continuation
FAILED tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py::test_mixed_turn_streams_the_continuation_or_the_turn_on_failure[withdrawn]

# against the first commit of this PR (trimmed turn returned on a failed continuation, malformed retrieve trimmed)
FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_failed_continuation_returns_turn
FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_malformed_ccr_call_keeps_turn

$ uv run --frozen --extra dev ruff check <changed files> && ruff format --check <changed files>
All checks passed!
$ uv run --frozen --extra dev mypy headroom/ccr/tool_calls.py headroom/ccr/response_handler.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, the released `headroom-ai==0.39.0` wheel
(byte-identical `response_handler.py` and `tool_calls.py` to main
3c418f6f) with this logic applied to the installed `handle_response`,
the way Headroom Desktop ships it.
- Exact command / steps: ran `CCRResponseHandler().handle_response` on a
stored hash with an Anthropic turn `[tool_use headroom_retrieve,
tool_use Read]` and a fake continuation that re-issues the Read, then
one that answers in text, then the same turn shaped as Responses
`function_call` items.
- Observed result: re-issued: one continuation carrying only the
retrieve `tool_use` and a `tool_result` with the original content, and
the client gets the re-issued `[Read]`. Answered in text: the client
gets that text, no calls. The reviewer's loopback-HTTP reproducer
(#issuecomment-6062651317) on 03e8d876, with its fallback assertion
inverted: a cancelling text continuation returns no call, a changed
`write_file(new.txt)` returns that call, an unchanged re-issue returns
it, and the original `write_file(old.txt)` is never handed back. Without
the change, no continuation and the client gets `[headroom_retrieve,
Read]`. The Responses turn comes back unchanged in every case.
- Not tested: a live Anthropic call where the trimmed assistant turn
also carries a signed `thinking` block. Only `tool_use` blocks are
removed and thinking blocks are kept unmodified, which is the shape the
lone-retrieve continuation already sends.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; CCR response handling
(`ccr_handle_responses`) is existing behaviour.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes; a mixed retrieve plus
client-tool turn on Anthropic or OpenAI chat is now served by a
continuation instead of being returned with an unresolvable
`headroom_retrieve`.
- Kill switch / disable path: `--no-ccr` (`ccr_handle_responses=False`)
disables CCR response handling entirely, as before.
- Unsafe override required: no
- Qualification impact: none on compression; one extra upstream
continuation per mixed turn, the same cost a lone retrieve already pays.
If the continuation request fails, the turn costs what it does today
plus that failed request.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- The model does not see the client calls it dropped: the continuation
carries the trimmed assistant turn (any text and thinking, plus the
retrieve) and its tool_result. If its text announced a client call, the
model re-issues it in the continuation, or decides not to.
- If the model calls a mixed turn again in the last allowed retrieval
round, the handler returns that turn as before
(`RESIDUAL_CCR_SKIPPED_MIXED`).
- Responses and Gemini still leak a mixed `headroom_retrieve` to the
client. Fixing them needs the reasoning item or thought-signature
pairing handled, which is out of scope here.
- Headroom Desktop vendors this logic against 0.39.0 until a release
carries it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-08 22:42:10 -05:00
Hill PatelandClaude Sonnet 5 159daa25a0 fix(proxy/bedrock): record the real upstream status on InvokeModel outcomes (#4025)
## Description

`handle_bedrock_invoke` (the Bedrock `InvokeModel` REST passthrough
handler) builds a `RequestOutcome` without ever passing `status_code`,
so every outcome it records keeps the dataclass default of `200`
regardless of what the Bedrock upstream actually returned. A throttled
(429), unauthorized (403), bad-request (400), or 5xx response from the
configured `--bedrock-api-url` gateway is forwarded to the client with
its real status, but is recorded in Headroom's own
savings/cost/request-log funnel as a successful `200` request -- so
`emit_request_outcome`'s `status_code >= 400` short-circuit (added in
#2682 / #3010) never fires for this path, and a rejected Bedrock turn
still feeds `metrics.record_request`, the cost tracker, and the PERF log
exactly like a served one.

This is a different gap from the one #2682 documented and deliberately
left alone: that PR's scope note says `_stream_response_bedrock` (the
SDK/boto3-based direct-to-AWS streaming path in `streaming.py`) "never
holds an HTTP response object" so it can't pass a real status.
`handle_bedrock_invoke` is a separate code path -- the REST gateway
passthrough described in `bedrock.py`'s own module docstring -- and it
does forward via raw `httpx`, so `response.status_code` (the real
upstream status, already used to build the
`StreamingResponse`/`JSONResponse` returned to the client) was sitting
right there and simply never got threaded into the outcome.

Every other `RequestOutcome` call site in the raw-httpx handlers
(`anthropic.py`, `batch.py`, `gemini.py`, most of `openai.py`) passes
`status_code=response.status_code` (or equivalent); `bedrock.py` was the
one exception.

Closes: none filed -- found by direct source review, not from an
existing issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/bedrock.py`: pass
`status_code=response.status_code` when constructing the
`RequestOutcome` in `handle_bedrock_invoke`, with a comment
distinguishing this path (real httpx response available) from the
SDK-based streaming path that genuinely cannot.
- `tests/test_proxy/test_bedrock_passthrough.py`: added
`test_outcome_carries_real_upstream_status_on_error`, parametrized over
`[400, 403, 429, 500, 503]`, asserting the recorded
`RequestOutcome.status_code` matches the real upstream status rather
than the `200` default.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) -- see "Not tested" below
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_proxy/test_bedrock_passthrough.py -q
collected 23 items
tests\test_proxy\test_bedrock_passthrough.py .......................     [100%]
23 passed in 23.15s

$ uv run ruff check headroom/proxy/handlers/bedrock.py tests/test_proxy/test_bedrock_passthrough.py
All checks passed!

$ uv run ruff format --check headroom/proxy/handlers/bedrock.py tests/test_proxy/test_bedrock_passthrough.py
2 files already formatted
```

RED proof -- new test fails against the pre-fix handler (stashed only
`bedrock.py`, kept the new test):

```text
$ uv run pytest tests/test_proxy/test_bedrock_passthrough.py::test_outcome_carries_real_upstream_status_on_error -q
...
    assert outcome.status_code == upstream_status
E   AssertionError: assert 200 == 400
E    +  where 200 = RequestOutcome(request_id='hr_...', provider='bedrock', ...).status_code
7 failed (one per parametrized status), 0 passed
```

GREEN -- same command with the fix restored: all 5 parametrized cases
(400/403/429/500/503) pass, full file 23/23 passed.

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.4, headroom `main` @ `3c418f6f`,
`uv run` with the `proxy` extra installed.
- Exact command / steps: A standalone script (not committed, not a unit
test) started a real `http.server.ThreadingHTTPServer` bound to
`127.0.0.1:<ephemeral-port>` -- an actual TCP socket, a genuine stand-in
for a Bedrock re-signing gateway -- that answers `POST
/model/.../invoke` with a crafted upstream `429 ThrottlingException`.
`ProxyConfig.bedrock_api_url` was pointed at that server; nothing else
was touched. The script then booted the real FastAPI app via
`TestClient`, which runs the real ASGI lifespan, so `proxy.http_client`
is the genuine, **unmodified** `httpx.AsyncClient` created by
`proxy.startup()` -- it was never replaced or mocked, unlike the unit
test and the original proof script. It POSTed a real request through the
live route to `/model/{model_id}/invoke`, so the request traveled over
an actual socket through the real `_forward_bedrock`/httpx code path to
the real local server and back. `proxy._record_request_outcome` was
wrapped only to capture the `RequestOutcome` object for printing -- it
still called through to the real implementation, so the real funnel
(`emit_request_outcome`, Prometheus counters, the proxy's own end-of-run
session-summary log) executed for real. Ran once with
`headroom/proxy/handlers/bedrock.py` checked out from `main` (pre-fix)
and once with the fix restored (post-fix); the fix was never left in the
reverted state (verified with `git status` / `git diff` after
restoring).
- Observed result: Pre-fix: `HTTP response returned to client: 429` /
`RequestOutcome.status_code recorded for metrics/savings funnel: 200` /
`Would emit_request_outcome short-circuit (status_code >= 400)? False`.
The proxy's own session-summary log for that run printed `Total
requests: 1`, `Rate limited: 0`, `Failed: 0` -- the throttled request
was silently counted as an ordinary served request. Post-fix, identical
request/server: `HTTP response returned to client: 429` /
`RequestOutcome.status_code recorded for metrics/savings funnel: 429` /
`Would emit_request_outcome short-circuit (status_code >= 400)? True`,
and the session-summary log changed to `Total requests: 0`, `Rate
limited: 1`, ` upstream 429s: 1`. In both runs the client-facing
response was correctly 429 -- only the internal accounting silently
diverged from it pre-fix, which is exactly the bug this PR fixes.
- Not tested: `mypy headroom` -- this Windows dev box's `uv sync --extra
dev` fails building the `hnswlib` native extension (missing MSVC
toolchain, unrelated to this change); only the `proxy` extra was synced.
No live AWS Bedrock or a real LiteLLM/LocalStack gateway was exercised
-- the upstream in this proof is a local `http.server` instance standing
in for that gateway. To be explicit about what is and isn't real here:
the real parts are the TCP socket, the HTTP round trip, headroom's own
`httpx.AsyncClient`, and the full
`handle_bedrock_invoke`/`_forward_bedrock`/`emit_request_outcome` code
path, all unmodified and unmocked; the one stand-in is the endpoint
itself (a local server playing the role of the AWS/gateway upstream, not
AWS or a real gateway). The streaming route
(`invoke-with-response-stream`) was not re-run under this real-socket
proof; it shares the identical `_forward_bedrock` call and
outcome-recording code with `invoke`, and is already covered by the
existing mocked `test_invoke_with_response_stream_is_byte_faithful`
test.

## Runtime Rollout Safety

- Rollout-managed feature(s): none -- Bedrock passthrough is already
gated behind `--bedrock-api-url`/`BEDROCK_TARGET_API_URL` being set;
this change doesn't add or touch a rollout flag.
- Minimum rollout channel: N/A, ships on stable with no gate.
- Stable/default behavior changed: yes, but only for already-failing
requests -- a Bedrock `InvokeModel` call that upstream rejects (4xx/5xx)
will now correctly show up under
`requests_rate_limited`/`requests_failed` instead of inflating
`requests_total`/`tokens_saved_total`/cost. Successful (2xx) requests
are unaffected (`status_code=200` either way).
- Kill switch / disable path: none added; revert is a one-line removal
of the `status_code=response.status_code` kwarg.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: `git revert` this commit; no persisted state to
migrate.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Found by direct source review of `headroom/proxy/handlers/bedrock.py`
while auditing provider-handler divergence (comparing how each provider
path populates `RequestOutcome`), not from the issue queue.
Cross-checked against #2682 and #3010 (which both touch
`RequestOutcome.status_code` plumbing on other paths) to confirm this
specific gap -- the REST gateway passthrough, as opposed to the
SDK-based streaming path #2682 explicitly left alone -- was not already
covered.

**2026-10-08 update:** addressed Devin's review comment (mocked-upstream
proof) and the governance bot's template-field check:
- The `Real Behavior Proof` section above now documents a genuine
local-socket proof: a real `http.server` TCP listener stands in for the
Bedrock gateway, and headroom's own `httpx.AsyncClient` /
`handle_bedrock_invoke` / `_forward_bedrock` / `emit_request_outcome`
code is exercised completely unmocked against it. The existing mocked
unit test (`test_outcome_carries_real_upstream_status_on_error`) is
unchanged and still serves as the fast, deterministic correctness check;
the new evidence is additional runtime proof, not a replacement.
- `Real Behavior Proof` and `Runtime Rollout Safety` now use the exact
plain-text field labels (`Environment:`, `Exact command / steps:`,
`Observed result:`, `Not tested:`) the governance script matches on,
instead of the differently-worded bold labels the previous revision
used.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4025"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-08 10:31:05 -05:00
1d4b357ca7 feat(opencode): support the OpenCode 2.x plugin API (#3999)
## Description

Takeover of #3719 by @laurence-diack-pk, who has been unavailable since
Sep 23 and whose branch was left conflicted with `main`. The original
commit is preserved and @laurence-diack-pk remains its author; this
branch adds follow-up commits addressing @JerrettDavis's review and the
findings from live OpenCode 2.x testing.

It makes the OpenCode plugin and `headroom wrap opencode` work on both
OpenCode 1.x and 2.x. On 2.x the 1.x function plugin was silently
dropped (the loader requires a `{ id, setup }` default export loaded
from a directory), and Effect's `FetchHttpClient` keeps its own `fetch`
reference from before plugins load — so `wrap opencode` loaded no plugin
and routed nothing.

Closes #3669. Supersedes #3719.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)

## Changes Made

Carried from #3719:
- One plugin for both versions: default export `{ id: "headroom",
server, setup }`; 1.x calls `server`, 2.x calls `setup`.
- 2.x `setup`: registers `headroom_retrieve` via `ctx.tool.transform`
(`codemode: false`), sets `HEADROOM_*` shell env through the
`create.before` hook, and returns a cleanup that releases the refcounted
transport so hot reloads do not stack interceptors.
- 2.x model routing: `ctx.model.transform` points remote OpenAI-wire
models at the proxy with `x-headroom-base-url` /
`x-headroom-original-path` headers, since 2.x keeps its own `fetch`
reference and the transport patch alone misses model traffic.
- Wheel ships the bundle as `_dist/index.js`;
`headroom_opencode_plugin_path()` returns the directory; `wrap opencode`
adds `--standalone` on 2.x so the private server sees
`OPENCODE_CONFIG_CONTENT`.

Follow-up commits (review from #3719 + live testing):
- Rebased on current `main` (0.40.0). Fixed the routing-contract
conflict: replaced `routesThroughProxy` (TS2554 against `shouldRoute`'s
3-arg signature) with `modelBaseRoutesThroughProxy`, a deliberate
model-base eligibility check that keeps the shared protocol / loopback /
proxy-origin / excluded-host contract but drops the endpoint-suffix
requirement, since 2.x hands us model base URLs (e.g. `/v1`), not
inference endpoints.
- `setup` now resolves the effective `excludeHosts` once (option over
`HEADROOM_OPENCODE_EXCLUDE_HOSTS`) and applies it to both the transport
install and the model rewrite, with new tests for both paths.
- Fixed `test_build_launch_env_with_project` to decode
`OPENCODE_CONFIG_CONTENT` instead of substring-matching the raw path,
which JSON backslash escaping makes impossible on Windows.
- `opencode_major_version` now reads the `v2` channel marker in dev
builds (`opencode v0.0.0-v2-<stamp>`), which previously parsed as major
0, so `wrap` skips `--standalone` and the private server never loads
Headroom's config.
- Rebuilt `_dist/index.js` and `hook-shim/handler.js`; drift check
clean.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .` — run on the changed paths)
- [x] Type checking passes (`mypy headroom` — run on the changed paths)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# plugins/opencode
$ npm ci && npm run typecheck && npm run build:standalone && npm test
tsc: clean (no TS2554)
vitest: Test Files 3 passed (3) | Tests 45 passed (45)
$ cmp dist-standalone/index.js ../../headroom/providers/opencode/_dist/index.js          # identical
$ cmp dist-standalone/hook-shim/handler.js ../../headroom/providers/opencode/hook-shim/handler.js  # identical

# focused Python suites (same set as #3719, plus the 2.x dev-build version case)
$ pytest tests/test_providers_opencode_plugin_path.py tests/test_providers_opencode_config.py \
    tests/test_providers_opencode_install.py tests/test_cli/test_wrap_opencode.py \
    tests/test_mcp_registry_opencode.py tests/test_learn/test_opencode_scanner.py -q
193 passed in 1.10s

$ ruff check / ruff format --check on changed paths: clean
$ mypy headroom/providers/opencode headroom/cli/wrap.py: Success: no issues found
```

## Real Behavior Proof

- Environment: macOS arm64, Node v26.10.0, Python 3.13, headroom from
this branch (0.41.0-dev). Tested against two real OpenCode 2.x clients,
each in throwaway XDG dirs with its own proxy port:
  - `opencode 2.0.19` (official 2.x binary);
- OpenCode 2.x head built from `anomalyco/opencode@v2`
(`0.0.0-v2-202610061051`, Bun 1.4.2, `bun run script/build.ts --single`
in `packages/cli`).
- Exact command / steps: throwaway XDG dirs and a dedicated proxy port;
fake key first (routing proof), then the real key (end-to-end proof):
  ```bash
  # fake key first (routing proof), then the real key (end-to-end proof)
OPENCODE_API_KEY=<key> headroom wrap opencode --port 8799 --no-serena --
\
run -m opencode-go/deepseek-v4.1-flash "reply with the single word ok"
  # exclusion check
HEADROOM_OPENCODE_EXCLUDE_HOSTS=opencode.ai OPENCODE_API_KEY=<key>
headroom wrap opencode ...
  ```
(`opencode-go` declares `OPENCODE_API_KEY` as its env credential in the
catalog, so no interactive login is needed.)
- Observed result: routed 401 with the fake key and 200 with the real
key through the proxy, `--standalone` added automatically, exclusion
honored — details:
- **Fake key** — proxy log: `proxy_inbound_request POST
/v1/chat/completions` with `user-agent: opencode/latest/2.0.19/cli` (and
`opencode/v2/0.0.0-v2-202610061051/cli` for the head), headers
`x-headroom-base-url: https://opencode.ai` and
`x-headroom-original-path: /zen/go/v1/chat/completions`; forwarded
upstream, `401` from the fake key — routed through Headroom.
- **Real key** — the model answered `ok`; proxy `status=200` with
upstream `https://opencode.ai/inference/go/openai/v1/chat/completions`.
- `wrap` added `--standalone` automatically on 2.0.19, and again on the
v2 head after the dev-build version fix (explicit `--standalone` also
works, per the kill-switch path).
- **Exclusion** — with `HEADROOM_OPENCODE_EXCLUDE_HOSTS=opencode.ai` the
model still answered while the proxy saw **zero** new chat requests: the
2.x model transform honors the exclusion list.
- Not tested: Windows; OpenAI "responses"-wire against a live 2.x
server; Anthropic-wire custom providers on 2.x; loading the plugin by
npm name (`headroom-opencode`), which needs a published release;
OpenCode 1.x older than 1.2.20.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: on OpenCode 2.x, `wrap opencode`
launches with `--standalone` (a private server per session); on 1.x
behaviour is unchanged apart from the plugin path now being a directory.
- Kill switch / disable path: pass `--server <url>` or `--standalone`
explicitly; `HEADROOM_OPENCODE_PLUGIN_PATH` still overrides the plugin
location.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR; the wheel bundle and plugin path revert
with it.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

@laurence-diack-pk — thank you for the original implementation; the
commit is preserved verbatim at the base of this branch. @JerrettDavis —
this addresses the routing-contract conflict you described in #3719 (the
missing `excludeHosts` argument and the model-base vs endpoint
mismatch), the Windows assertion, and a dev-build version-detection gap
found while validating on macOS. Live 2.x validation is done on both
2.0.19 and today's `v2` head; details above.

---------

Co-authored-by: laurence <laurence.diack@peekrobotics.co.nz>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-07 20:52:53 -05:00
Mansurshakh JaparovandJD Davis 9553d4292f fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description

Session cleanup currently drops prefix trackers after the default
600-second session lifetime even when Anthropic's request selects a
one-hour cache TTL. Losing the tracker also loses the frozen forwarded
prefix and causes the cache-preserving overlay to replay different
bytes.

Closes #4017.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Resolve the request's existing Anthropic cache TTL before selecting
the prefix lineage and pass that TTL through tracker selection/creation.
- Retain each lineage for the greater of the configured session floor
and its observed/current provider TTL plus the existing 60-second safety
margin.
- Keep the lifetime per lineage rather than changing global
configuration: a five-minute session does not inherit another session's
one-hour lifetime.
- Preserve normal expiry beyond the provider lifetime and margin.
- Honor configured cache TTL before a confirmed cache write; discard
expired state before accepting a later request's longer TTL.
- Re-register the live lineage family after lookup/cleanup so recreated
sessions cannot share a tracker's frozen state across sibling
conversations or tool profiles.
- Update existing fake callbacks to the changed tracker contract,
preserve all consumer byte-stability assertions, and remove one
handler-to-store wiring-only test rather than re-pinning its internal
identity assertions.
- Document local tracker retention separately from provider cache
availability.

## Testing

- [x] Unit tests pass (`pytest`), including frozen-prefix byte stability
and independent TTL expiry.
- [x] Changed-file lint, configured pre-commit mypy and manual
production timestamp/overlay checks pass.

### Test Output

```text
Review correction: 273 passed, 2 dependency warnings in 83.79s
{"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true}
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate
explicitly used the installed CPU `libonnxruntime.so.1.26.0` via
`ORT_DYLIB_PATH`, without skipping cached-model assertions.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 12.87s ==================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_cache/test_prefix_tracker.py
python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py
```

**74 passed** in tracker tests; **54 passed** in request/cache tests
(one existing Starlette TestClient deprecation warning). Two new
regressions fail on baseline because the request TTL is not accepted; a
separate production runtime comparison demonstrates the actual pre-fix
loss, not just the missing parameter.

After rebasing onto current main
`1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together
with `tests/test_prefix_tracker_client_rewrote_reply.py`,
`tests/test_cache_mode_delta_marker.py`, and
`tests/test_content_router_mcp_small_result.py`: **150 passed**. The
actual production timestamp/byte-retention smoke was repeated
successfully on the rebased code, covering compatibility with main's
newly merged client-reply normalization.

Review regression and affected HTTP-consumer suite: **273 passed**,
including tracker expiry, Anthropic/OpenAI cache stability,
byte-faithful forwarding, header isolation, buffered timeouts, CCR
injection, and immutable system prompts. All affected two-argument test
callbacks now accept the TTL contract. The obsolete walker-call-order
test was removed; wire-byte assertions remain. The new
configured-TTL/expiry cases fail against the published baseline; the
post-cleanup regression also verifies that an unrelated conversation
gets a different tracker after recreation.

Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and
Commitlint hooks passed. Full-tree Ruff is locally blocked by executable
filesystem permissions on five unchanged OAuth2 plugin files (tracked
100644). A separate full dev-environment mypy run found an unchanged
optional LangChain `truthy-function` error; the configured isolated
pre-commit hook passed. No unrelated source/permissions changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real
`SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token
estimator and `overlay_cached_prefix`, no mocked store or mocked clock.
- Exact command / steps: parse `cache_control: {type: ephemeral, ttl:
1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed
frozen message with different original/forwarded prefix bytes; age
`_last_activity` to `time.time()-660` and make periodic cleanup due;
resume the same lineage and compare the tracker, frozen count, and
forwarded-prefix bytes. Also age beyond 3901 seconds and compare
independent 300/3600-second sessions.
- Before: resolved TTL 3600; after 660 seconds, tracker identity
changes, frozen count drops **1→0**, and forwarded-prefix bytes differ.
- Observed result: after the fix, after 660 seconds the same tracker
remains, frozen count stays **1**, and forwarded-prefix bytes are
identical. The one-hour session survives independently, the five-minute
session expires, and the one-hour lineage still expires beyond TTL plus
margin.
- Review runtime proof against published head `e1a107df`: actual stores
and prefix overlays confirm configured TTL retention before the first
cache write, no resurrection of expired state by a later one-hour
request, eventual expiry, and separate trackers after the maintainer's
3901-second cleanup/recreation sequence. On that exact baseline the
unrelated lookup reuses the wrong tracker, although the overlay guard
preserves the particular unrelated input bytes; after the fix both
identity isolation and byte preservation hold. Identical messages under
different tool profiles also receive a cold, separate tracker after the
expired bare-key transition.
- Not tested: Anthropic cloud cache availability, billed savings, or an
actual one-hour wait. The local production state and byte invariants
were exercised by aging actual timestamps, not by a paid upstream
request. Preserving a tracker cannot guarantee that the provider retains
its cache.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; repair of existing TTL-aware
cache-prefix behavior.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: eligible local prefix trackers
survive for their provider cache lifetime. Short-TTL/default sessions
remain bounded by their own session floor.
- Kill switch / disable path: the existing prefix-freeze disable path
remains available; no new flag.
- Unsafe override required: no.
- Qualification impact: local Anthropic prefix lineage
selection/retention, not cache pricing or subscription quota policy.
- Rollback path: revert the commit to restore previous retention. No
persistent schema change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing cache TTL parser and safety margin reused; cache docs
updated.
- [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes,
and existing request byte-stability behavior.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

Remote CI results are not inferred from local verification.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 17:46:21 -05:00
Sudhindra DesaiandJD Davis 01185e32d3 feat(router): add max_output_tokens condition to ModelRoute (#3898)
## Description

`ModelRoute` could bound the estimated input size (`max_input_tokens`)
but had no condition on the request's own `max_tokens`. That meant a
tiny prompt asking for a long answer (for example, `max_tokens: 4000`)
still matched a "trivial turn" downshift rule and was sent to the
cheaper model.

This PR adds `max_output_tokens`, using the shape proposed in the issue.
A rule with it set matches only when the request's `max_tokens` is `<=
N`. As with the other conditions, `None` means "ignore this condition",
set conditions AND together, and the first matching rule wins.

**Open question from the issue, decided here:** a request with no
`max_tokens`, or with a non-integer or boolean value, does **not**
match. An unbounded response is not a trivial turn, so the rule can
never widen to cover unbounded requests. This follows the same
fail-closed approach as the existing parser.

Closes #2765

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/model_router.py`: new `ModelRoute.max_output_tokens`
field. `matches()` and `ModelRouter.select()` take an optional
`max_tokens` keyword, which defaults to `None`, so existing callers are
unchanged. `max_tokens` is added to the decision reason.
`max_output_tokens` is added to the allowed route keys and parsed with
the existing `_strict_opt_int`, so a malformed, boolean or negative
value skips the route. New `request_max_tokens(body)` helper.
- `headroom/proxy/handlers/anthropic.py`: `_maybe_route_model` passes
`request_max_tokens(body)` to the router.
- `docs/content/docs/configuration.mdx`: new field in the rule table,
plus a note on combining it with `max_input_tokens`.
- Tests for matching, absent budget, select and reason, env parsing
(valid and malformed), the helper, and handler wiring.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): not run successfully. My
local mypy stops on an unrelated numpy stub syntax error in this env
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy/test_model_router.py tests/test_proxy/test_model_router_wiring.py -q
54 passed, 1 warning in 5.22s

$ pytest tests/test_proxy -q -k "anthropic or router"
119 passed, 237 deselected, 1 warning in 14.41s

$ ruff check headroom/proxy/model_router.py headroom/proxy/handlers/anthropic.py tests/test_proxy/test_model_router*.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, branch head
- Exact command / steps: the rule from the issue (`from_models:
[claude-opus-4-8]`, `max_input_tokens: 4096`, `max_output_tokens: 512`,
`require_no_tools`, `to_model: claude-haiku-4-5`) built via
`ModelRouterConfig.from_env`. I then ran `select()` for the issue's
"Reply with the single word OK." prompt with three different
`max_tokens` values.
- Observed result: `max_tokens=4000` stays on `claude-opus-4-8`,
`max_tokens=256` routes to `claude-haiku-4-5`, and a request with no
`max_tokens` stays on `claude-opus-4-8`:
  ```text
  max_tokens=4000: routed_model=claude-opus-4-8 reason=no rule matched
max_tokens=256: routed_model=claude-haiku-4-5 reason=matched rule
'trivial-opus->haiku': claude-opus-4-8 -> claude-haiku-4-5
(input_tokens=7, has_tools=False, max_tokens=256)
  max_tokens=None: routed_model=claude-opus-4-8 reason=no rule matched
  ```
- Not tested: an end-to-end request through a running proxy against a
live upstream.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This only extends the opt-in model
router (`HEADROOM_MODEL_ROUTER_ENABLED`).
- Minimum rollout channel: N/A
- Stable/default behavior changed: No. The router is still off by
default, and rules without `max_output_tokens` behave exactly as before.
- Kill switch / disable path: unset `HEADROOM_MODEL_ROUTER_ENABLED`, or
remove the field from the rule.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Routing still only applies on the Anthropic `/v1/messages` path, where
`max_tokens` is required. If routing is later extended to the OpenAI
paths (#2354), `request_max_tokens` is the single place to add
`max_completion_tokens` / `max_output_tokens`.


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3898"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-06 10:30:01 -05:00
76cceeb487 fix(proxy): break the compact-every-turn loop near the real context window (#2942)
## Description

Closes #3998

Compression makes the usage reported back to the client reflect the
*forwarded* (compressed) request, not the client's raw transcript.
Clients whose auto-compaction keys off reported usage (Claude Code)
therefore defer compaction — the selling point — but with no floor: the
raw transcript can grow until even the compressed request exceeds the
model's real context window. From that point every turn fails with a 400
`prompt is too long`, the client force-compacts on each error, and the
session degrades into a compact-every-other-prompt loop. This came from
a churned-user field report (2026-08-12): "Headroom cut off my context
and started compacting every two prompts."

A second, related bug: `get_context_limit()` resolves purely from the
model id, so a session running with the `context-1m` beta is treated as
5x over budget by compression pressure (`context_pressure` pins at 1.0)
and gets maximum-aggression lossy compression from early on — the "cut
off my context" half of the report.

This PR adds `headroom/proxy/context_guard.py` with three cooperating
pieces, all passthrough until the danger zone:

1. `effective_context_limit` — the window the forwarded request is
really subject to: the registry limit, raised to 1M when the client sent
a `context-1m` beta token, capped by any limit learned from a real
`prompt is too long` error. Also used for compression pressure at the
`context_limit` call site.
2. `note_prompt_too_long` — parses `prompt is too long: N tokens > M
maximum` from upstream 400 bodies (streaming and buffered error paths)
and records M keyed by (model, has-1m-beta). This is how a client that
believes 1M but whose account is capped at 200k stops looping after
exactly one error — and why the learned limit must not leak between beta
shapes.
3. `MessageStartGuard` — an SSE transformer applied to the client-bound
bytes only: when the `message_start` usage total reaches 90% of the
effective limit, it inflates the reported `input_tokens` so the client's
gauge reads 95% of the window the client believes in, which trips the
client's own graceful compaction *before* the 400 wall. Inert after a
single decision (steady-state overhead is one `if`); savings metrics and
telemetry parse the original upstream bytes, never the nudged ones.

Kill switch: `HEADROOM_CONTEXT_GUARD=0`.

Out of scope, deliberately: the buffered (non-streaming) success path is
not nudged — the affected clients (Claude Code) always stream, and
rewriting buffered bodies would break response byte-fidelity for a path
with no known symptom. The batch endpoint's `context_limit` site is also
untouched (batch has no interactive compaction loop).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `headroom/proxy/context_guard.py`: limit resolution
(`believed_context_limit`, `effective_context_limit`), 400-learning
(`note_prompt_too_long`), and the `MessageStartGuard` SSE usage
rewriter, plus the `HEADROOM_CONTEXT_GUARD` kill switch.
- `headroom/proxy/handlers/streaming.py`: build the guard for anthropic
streams (inside try/except, passthrough on any setup failure), route
yielded chunks through `guard.feed()`, flush held bytes at stream end,
and learn from streaming 400 bodies.
- `headroom/proxy/handlers/anthropic.py`: compression `context_limit`
now uses `effective_context_limit` (1m-beta aware, learned-cap aware);
the existing UPSTREAM_ERROR block also feeds `note_prompt_too_long`.
- `tests/test_proxy/test_context_guard.py`: 25 tests — unit coverage of
limits/learning/rewriter (chunk splits, pings, malformed JSON, buffer
cap, never-deflate) and four integration tests driving
`_stream_response` with a mocked upstream (nudge on the wire,
byte-identical passthrough far from the limit, kill switch, 400
learning).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_context_guard.py -q
25 passed in 0.53s

$ uv run --frozen --extra dev pytest tests/test_proxy/ -q
272 passed, 1 warning in 71.42s (0:01:11)

$ uv run --frozen --extra dev pytest tests/test_sse_thinking_blocks.py tests/test_h2_stream_reset_retry.py \
    tests/test_proxy_streaming_ratelimit_headers.py tests/test_request_outcome.py tests/test_proxy_retry_429.py -q
81 passed in 1.53s

$ uv run --frozen --extra dev mypy headroom/proxy/context_guard.py
Success: no issues found in 1 source file

$ uvx ruff check headroom/proxy/context_guard.py headroom/proxy/handlers/streaming.py \
    headroom/proxy/handlers/anthropic.py tests/test_proxy/test_context_guard.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS 15 (Darwin 24.6.0), Python 3.10 project venv via
uv, repo at this branch; live runs used a sandboxed proxy instance (this
module injected) with Claude Code 2.1.206 as the real client.
- Exact command / steps: ran the four integration tests in
`TestStreamResponseIntegration`, which drive the real `_stream_response`
code path with a mocked upstream httpx response: (a) a `message_start`
carrying a 185k-token usage against a 200k model limit (above the 90%
trigger, below the 95% target, so the assertion can only pass if the
rewrite happened), (b) the same stream at 50k tokens, (c) a 190k stream
with `HEADROOM_CONTEXT_GUARD=0`, (d) a 400 body containing `prompt is
too long: 213021 tokens > 200000 maximum` with `anthropic-beta:
context-1m-2025-08-07`.
- Observed result: (a) the client-received `message_start` reports
`input_tokens` 185000 inflated to 190000 (the 95% target) while the
`message_stop` bytes pass through unmodified; (b) the client-received
bytes are byte-identical to the upstream bytes; (c) byte-identical
passthrough with the kill switch set; (d) the 400 is forwarded unchanged
and `effective_context_limit("claude-sonnet-4-5", 200000,
"context-1m-2025-08-07")` returns 200000 afterwards.
- Live end-to-end (added after the initial PR): drove a real Claude Code
2.1.206 session through a proxy running this code, with a learned limit
seeded to the state a real prompt-too-long 400 produces. Result: the
client-visible usage total read exactly 190000 (95% of the believed 200k
window) and the interactive TUI showed "100% context used". This run is
also why the second commit exists: with only message_start rewritten,
the client kept displaying the raw values because it merges the final
cumulative-usage message_delta over message_start. Forcing a
prompt-too-long 400 on the next turn produced a clean "Compacting
conversation..." -> summarization request -> retried turn -> correct
response, with no user-facing error.
- Re-validated 2026-10-06 at `739bacfa`, merged with upstream/main
`855390d6` (0.40.0): the repro in #3998 runs the real app with only the
upstream HTTP call mocked. On main a forwarded 185k turn reaches the
client as 185000 in both `message_start` and `message_delta`, a `prompt
is too long ... > 200000 maximum` 400 is forgotten (the next 185k turn
reads 18.5% of the 1M the client believes in), and a `context-1m`
request is compressed against 200k (`context_pressure` 1.00 at 400k). On
this branch: 190000 (95% of 200k), 950000 (95% of the believed 1M) after
the learned limit, and 1M (`context_pressure` 0.40).
- Merge note: main's memory tool stream filter (#3947) changed the same
`yield chunk` line. The guard now runs last, on the frames the filter
releases, including the filter's closing frames and the memory
continuation rounds, and the end-of-stream drain moved after the memory
handling so the `message_delta` the filter holds back is still nudged.
`test_guard_nudges_what_the_memory_filter_releases` covers it (fails
with `185000 == 190000` if the drain moves back). `tests/test_proxy/`
432 passed, the memory tool suites 95 passed.
- Not tested: an organic (unseeded) 180k+-token conversation against the
production API; the buffered non-streaming success path is deliberately
not nudged (see Description). Note for reviewers: Claude Code 2.1.206
compacts reactively at prompt-too-long (with a threshold-armed
precomputed summary) rather than proactively pre-send, so the nudge's
role there is an honest gauge/warnings and threshold arming; older
clients and other harnesses that compact proactively off reported usage
get the pre-wall compaction directly.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Not in the rollout FEATURES
registry; gated by its own env switch, `HEADROOM_CONTEXT_GUARD` (default
on).
- Minimum rollout channel: N/A (not rollout-managed; applies on every
channel)
- Stable/default behavior changed: Yes, for Anthropic requests only.
Compression pressure uses the effective context window (1M when the
client sent the `context-1m` beta, capped by any limit learned from a
real `prompt is too long` 400), and a client-bound `message_start` whose
usage reaches 90% of that window reports `input_tokens` raised to 95% of
it so the client compacts before the wall. Below 90% the client receives
byte-identical upstream bytes. Upstream requests and error bodies are
never modified.
- Kill switch / disable path: `HEADROOM_CONTEXT_GUARD=0` (or `false`).
Every call site checks it before using the effective limit or rewriting
usage, so with it off the proxy behaves exactly as before this PR
(covered by the kill-switch integration test).
- Unsafe override required: No
- Qualification impact: Adds `tests/test_proxy/test_context_guard.py`;
no existing test changed or relaxed.
- Rollback path: Set `HEADROOM_CONTEXT_GUARD=0` or revert the PR.
Learned limits live in process memory only, so nothing persisted needs
cleanup.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from the Conventional Commit PR title (a CI guard
enforces this)




<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2942"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-06 10:27:09 -05:00
Rod BoevandJD Davis 0ad996a82e feat(proxy): route Gemini plugin traffic through native transforms (#2697)
## Description

OpenCode-routed Gemini requests now reach the tagged gateway and use the
shared transform pipeline used by direct Gemini requests. Unsafe
caller-supplied destinations stop with 400 before a handler can connect
to them. Previously, the transport preserved `x-headroom-base-url` and
the upstream pathname, but route matching sent native
`:streamGenerateContent` through a handler that did not accept the
tagged base and skipped transforms.

`handle_gemini_generate_content` already accepts an `upstream_base_url`,
runs the OpenAI pipeline over converted `contents`, and handles
streaming by URL shape. `handle_gemini_count_tokens` already accepts one
too. This adds an opt-in capability on the route spec, checks the
caller-supplied base with the shared upstream guard, and sends both
direct and plugin-routed native streaming through the shared
transform-capable handler. Native parts, incoming query parameters, and
the SSE selector are preserved. Wire formats with no native handler keep
the existing custom-base passthrough branch. Loopback and private
destinations receive a 400 response unless the operator explicitly
allowlists them.

Closes #2602

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `ProviderHandlerRoute` gains `supports_custom_base_url`, defaulting to
`False` so every existing handler route keeps its current behavior
- `GEMINI_HANDLER_ROUTES` opts in all three model endpoints:
`:generateContent`, `:streamGenerateContent`, and `:countTokens`.
Counting has to follow the same upstream as generating, or the client
gets counts from a host that is not serving the request
- `_register_provider_handler_route` reads `x-headroom-base-url` and
forwards it as `upstream_base_url` for routes that opted in
- The route validates each custom base with the shared upstream guard
before handler dispatch, rejecting unsafe destinations unless an
operator allowlist permits them
- `handle_gemini_stream_generate_content` delegates unconditionally to
`handle_gemini_generate_content`, which runs the transform pipeline and
builds the SSE URL from the request path for both direct and
plugin-routed traffic
- regressions for direct and plugin production-composed routes, mixed
native parts, raw query preservation with one `alt=sse`, and adjacent
paths that stay on passthrough

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

`uv run pytest tests/test_proxy/test_gemini_plugin_routed_streaming.py
tests/test_provider_proxy_routes.py tests/test_provider_route_specs.py
tests/test_tokenizer_count_offload.py -q`

```text
62 passed
```

`uv run ruff check . && uv run ruff format . --check`

```text
All checks passed!
1600 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.13, no live Google credentials;
production `create_app` routes use fake upstream callbacks
- Exact command / steps: `uv run pytest
tests/test_proxy/test_gemini_plugin_routed_streaming.py
tests/test_provider_proxy_routes.py tests/test_provider_route_specs.py
tests/test_tokenizer_count_offload.py -q`
- Observed result: direct and plugin-routed `:streamGenerateContent`
requests reach the same transform pipeline, preserve native parts and
query parameters, add one `alt=sse`, and strip `x-headroom-*` headers.
All three native Gemini routes reject loopback when the allowlist is
unset without calling their handlers; an explicitly allowlisted fake
gateway still reaches the handlers. `:countTokens` follows the tagged
upstream, and adjacent custom-base routes remain passthrough.
- Not tested: live Google endpoint

## Runtime Rollout Safety

- Rollout-managed feature(s): None; upstream validation applies to every
opted-in Gemini request
- Minimum rollout channel: Standard proxy release containing this change
- Stable/default behavior changed: Native Gemini routes reject loopback
and private custom bases by default; public gateways and explicitly
allowlisted destinations continue to work
- Kill switch / disable path: No runtime kill switch;
`HEADROOM_ALLOWED_BASE_URLS` allows an operator to approve a private
gateway
- Unsafe override required: Only when a private deployment gateway must
be used
- Qualification impact: None
- Rollback path: Revert this change; the prior route behavior returns

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; this change has no rendered UI.

## Additional Notes

Scope note: this covers Gemini's three model endpoints, including direct
and plugin-routed `:streamGenerateContent`. Gemini's other routes still
use `generativelanguage.googleapis.com` regardless of the tag. Batch
routes need handler support for a custom upstream; model listing and
passthrough routes use a different registrar. `cachedContents` stays
separate because a tagged client could create a cache entry at Google
and reference it from a `:generateContent` request sent to a gateway. A
follow-up can cover those routes.

The tokenizer and pricing entries the issue also asks for, and adapters
for other non-Anthropic/non-OpenAI wire formats, are not here.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2697"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 15:34:32 -05:00
6fb7cadfe4 fix(proxy): key rate limits by peer unless the caller authenticated (#3860)
## Why

The proxy's request and token rate limits could be walked around by any
caller the proxy cannot authenticate (finding 01-F3). Each provider
handler keyed its buckets on a client-supplied header:

- **OpenAI:** an HMAC of the full `Authorization` / `api-key` value,
else one shared `"default"` bucket. Rotating the header gave a fresh
bucket per request, and all callers without a credential shared one
bucket.
- **Anthropic:** the first 16 characters of the key plus the peer. That
is mostly the common `sk-ant-api03-` prefix.
- **Gemini:** the first 20 characters of `x-goog-api-key`, with no peer
at all.

## What changes

- New `headroom/proxy/rate_limit_identity.py` with one rule for the
Anthropic, OpenAI and Gemini handlers. `rate_limit_identity()` returns a
plain string key:
- Every key starts with the peer from `resolve_client_ip`, which honours
`X-Forwarded-For` only from `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`
peers. IPv6 peers are grouped by `/64`; an IPv4-mapped IPv6 address
counts as the IPv4 peer.
- **Trusted** requests get one bucket per provider credential:
`peer:<addr>|cred:<hmac>`, an HMAC of the whole credential with a
process-local key, so bucket names hold no credential material. A
request is trusted if it presented a valid `HEADROOM_PROXY_TOKEN` or
came directly from loopback. A trusted-gateway peer is trusted to supply
the client address, not to vouch that the caller authenticated, so a
forwarded request without the token is untrusted and keyed by the
forwarded client address. A request from a gateway address that carries
no `X-Forwarded-For` is judged as a direct request, so a loopback
gateway CIDR does not change how direct localhost callers are keyed.
Distinct principals behind one address keep separate limits, as #3364
intended.
- **Untrusted** requests (no token, remote caller) are charged per peer
(`peer:<addr>`). The credential header is ignored.
- The security gate sets `request.state.proxy_authenticated = True` when
a caller presents a valid proxy token.
- The per-handler key code is removed (`_openai_rate_limit_key` and the
Anthropic and Gemini inline keys).

`rate_limiter.py` is unchanged, and so are token-limit semantics.

## Behaviour change

Requests the proxy cannot authenticate (no `HEADROOM_PROXY_TOKEN`,
remote caller) are now limited per client address. Changing the API-key
header no longer gives a new allowance. Authenticated and
direct-loopback callers keep one allowance per provider credential.
Behind a gateway, set `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` so each
forwarded client address gets its own limit; to keep one allowance per
credential there, callers must also send the proxy token.

The limiter is quota smoothing and abuse control for unauthenticated
callers. It is not a security boundary for authenticated callers, and
this PR does not make it one.

## Tests

- `tests/test_proxy/test_rate_limit_identity.py`: untrusted keys ignore
the credential; trusted keys are per credential and include the peer;
the whole credential is used, not a prefix; bucket names carry no
credential material; loopback is trusted; a request relayed by a
trusted-gateway peer (including one on loopback) is not trusted without
the proxy token; with `127.0.0.0/8` configured as a gateway, direct
loopback callers stay per credential, and a relayed request with an
unusable `X-Forwarded-For` still counts as relayed; IPv6 `/64` grouping;
IPv4-mapped addresses.
- `tests/test_proxy_openai_rate_limit_key.py`, end to end through
`create_app` with `--rpm 1`: authenticated and loopback callers with
distinct keys do not share a bucket; an unauthenticated remote caller
who rotates `api-key` or `Authorization`, or drops the credential, gets
429 on the second request; two different unauthenticated peers do not
share a bucket; through a trusted gateway (`10.0.0.2`, fixed
`X-Forwarded-For: 203.0.113.9`) a caller rotating `api-key` gets 429,
distinct forwarded clients do not share, and token-authenticated callers
stay per credential; with `127.0.0.0/8` as the gateway CIDR, two direct
loopback callers with different keys both pass while a caller relayed
through it with a fixed forwarded address gets 429 on the second key.
The unauthenticated cases fail on `main`.
- Existing rate-limiter and TPM suites pass unchanged.

Manual run: `headroom proxy --host 0.0.0.0 --rpm 2` with six requests
from a LAN address, each with a different `Authorization` key. On `main`
all six reach the upstream. On this branch without a proxy token, the
third and later get 429. With the proxy token sent, all six pass, one
bucket per credential.

Not tested: the Rust `headroom-proxy` binary, which has its own limiter.

## Merge order / conflicts

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. #3852 makes the untrusted case rare by refusing token-less
non-loopback binds; this PR makes it safe when an operator opts out of
that.

Textual conflict: #3862 adds an import next to the one this PR adds in
`handlers/anthropic.py` and `handlers/openai.py`. Whichever merges
second keeps both imports.

Revised 2026-10-01 after self-review: `rate_limiter.py` is back to
`main`; the separate trusted/untrusted pools, per-owner cap and overflow
bucket were removed.
Revised 2026-10-02 after review: gateway membership only drops the
credential from the key when the gateway actually forwarded a client
address, so direct localhost callers keep per-credential buckets under a
loopback gateway CIDR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3860"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:26:56 -05:00
im47cnandJerrettDavis cfa479af8e fix(proxy): log response_content_length in proxy_inbound_response (#2701)
## Description

The `proxy_inbound_response` log line recorded only `duration_ms` and
`status` — **not the response payload size**. This made it impossible to
correlate slow responses with payload size or diagnose "large response →
high latency" patterns.

Meanwhile, the inbound *request* line already logs `content_length`
(from the HTTP header), and the *outbound* request line already logs
`body_bytes`. The response size was the missing piece.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `response_content_length` field to the `proxy_inbound_response`
log line in `headroom/proxy/server.py`, reading from
`response.headers.get("content-length", "")`
- For streaming/chunked responses where Content-Length is absent, the
field appears as an empty string — no crash, no omitted field
- Added `tests/test_proxy/test_inbound_response_size.py` with 2 tests
covering both JSON (with Content-Length) and StreamingResponse (without
Content-Length) scenarios

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python3.11 -m pytest test_inbound_response_size.py -v --tb=short

test_inbound_response_size.py::test_json PASSED                                [ 50%]
test_inbound_response_size.py::test_streaming PASSED                           [100%]

======================== 2 passed, 2 warnings in 0.61s =========================
```

Manual verification against a live proxy instance:

```text
# JSON response (has Content-Length)
event=proxy_inbound_response id=inbound-1785548469603571000 method=POST path=/v1/messages status=200 duration_ms=0.65 response_content_length=221

# Streaming response (no Content-Length — empty value, no crash)
event=proxy_inbound_response ... response_content_length=
```

Note: Tests were run against the installed headroom package
(site-packages) which includes the compiled Rust extension
`headroom._core`. The source-tree `pyproject.toml` uses a maturin build
backend — running `pytest` from the source tree without `maturin
develop` will fail on `ModuleNotFoundError: No module named
'headroom._core'`. The test logic itself is environment-agnostic.

## Real Behavior Proof

- Environment: macOS 15, Python 3.11.15, headroom-ai 0.32.1 (installed
via pip)
- Exact command / steps: `python3.11 -m pytest
tests/test_proxy/test_inbound_response_size.py -v`
- Observed result: 2 passed, `response_content_length=221` confirmed in
captured log output for JSON responses; empty string for streaming
responses
- Not tested: ruff linting and mypy type checking (not installed in test
environment); e2e against a real upstream provider

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my diff
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — text log format change only.

## Additional Notes

- The fix is intentionally minimal: one new `%s` placeholder + one new
argument to an existing `logger.info()` call. No control flow changes,
no new dependencies.
- The field name `response_content_length` (not `response_body_size`) is
chosen to mirror the existing `content_length` field name used on the
inbound request side, keeping the log schema symmetric.
- `ruff` and `mypy` were not run because they are not installed in the
test environment. The change is a single-string-literal + single-arg
addition to a `logger.info()` call — no new types, no new imports.
## Runtime Rollout Safety

- Rollout-managed feature(s): None — this is a log-field addition, no
feature flag involved.
- Minimum rollout channel: N/A — no gated rollout; ships with the
regular proxy release.
- Stable/default behavior changed: No. The change only appends
`response_content_length` to an existing log line; the log format
remains backwards-compatible and no default behavior is altered.
- Kill switch / disable path: N/A — no new runtime behavior to disable;
removing the field is a one-line revert.
- Unsafe override required: No.
- Qualification impact: Low. Covered by 2 new unit tests (JSON with
Content-Length, StreamingResponse without) plus manual verification
against a live proxy; streaming responses without Content-Length produce
an empty field rather than an error.
- Rollback path: Revert the single commit — the previous log format is
restored immediately, no data migration or config change needed.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-03 00:01:39 -05:00
Tejas ChopraandClaude Fable 5.1 861e94d8c1 fix(proxy): keep exception text and the proxy token out of client output (#3851)
## Why

Two leaks of internal detail to clients and logs (findings 01-F4 and
10-F4).

- When an upstream connection failed, several response paths returned
`str(e)` from `httpx`, `ssl` or the OS in 500/502/503 bodies and SSE
`error` events. That text names the resolved upstream host, its address
and the failing library. A client that can steer the upstream
(`x-headroom-base-url`) could learn internal hostnames one error at a
time.
- `extract_tags` copied every `x-headroom-*` header into
`RequestOutcome.tags`, including `x-headroom-proxy-token`, which
gateways send on every turn. The token then appeared in the request log
(`--log-file`), dashboard facets and any export that carries tags.

## What changes

- New module `headroom/proxy/public_errors.py`. `classify_exception()`
maps transport and OS failures to fixed codes: `upstream_unreachable`,
`upstream_timeout`, `upstream_tls_error`, `upstream_protocol_error`;
anything else is `internal_error`. Every client- or model-visible path
fails closed: unrecognised exception text (`RuntimeError`, `ValueError`,
parser, database or library errors) never leaves the proxy. Provider
error text is forwarded only when the exception carries a provider HTTP
error response (`openai.APIStatusError`, which LiteLLM's provider errors
subclass, or `anthropic.APIStatusError`). Helpers build the existing
wire shapes with the same `error.type`, plus `code` and `request_id`.
The full exception stays in the server log under that request id.
- Every site that built a client-visible message from an exception now
uses it: `handlers/streaming.py` (direct, Bedrock and OpenAI-via-backend
SSE), `handlers/openai.py` (backend 500, `/v1/compress` 503, both
passthrough connect paths), `handlers/anthropic.py` (Bedrock backend
500), `handlers/bedrock.py` (connect 502), the memory tool handlers
(tool results returned to the model), and the `litellm` / `anyllm`
backends. The existing corporate-TLS hint is kept.
- `internal_header_policy.py` gains `is_credential_header()` and
`is_credential_tag_key()`. They match whole name components (`token`,
`secret`, `password`, `authorization`, `cookie`, `apikey`, `key`, and
similar), so `X-Headroom-Key` and `X-Headroom-Api-Key` are covered, so
numeric tags such as `tool-search-deferred-tokens` are not caught.
- `extract_tags` skips credential headers.
`savings_attribution.public_tags`, the last step before the request log,
also drops credential-shaped keys.

## Behaviour change

- Transport-layer and internal failures now return a fixed sentence plus
`code` and `request_id`. Clients that parsed the old free-text messages
should read `error.code`. Provider HTTP error responses (and their
status codes) are unchanged. In the LiteLLM / any-llm backends, an
exception that is not a provider response now returns `The proxy could
not complete the request.`; the HTTP status mapping is unchanged.
- Memory tool results for any failure return `{"status": "error",
"error": "<code>", "message": ...}` instead of the exception text.
- `x-headroom-*` headers whose name contains a credential component are
no longer recorded as tags. A benign label that happens to use such a
name should be renamed.

## Tests

- `tests/test_public_errors.py`: classification of transport exceptions
(including TLS wrapped in a connect error), non-transport exceptions
left alone, wire shapes carry `code` and `request_id`; the passthrough,
direct-streaming and `/v1/compress` error paths contain no upstream
detail; memory tool results hide transport and internal text;
`client_message` / `tool_result_error` / the LiteLLM backend return
`internal_error` for `RuntimeError`, `ValueError`, `KeyError` and
`JSONDecodeError` carrying a path, host and token (10 tests in this file
fail without the fail-closed change), while an
`openai.AuthenticationError` keeps its text; `extract_tags` and
`public_tags` never carry credentials. Six of these fail on `main`.
- Existing tests that asserted the old exception text in client output
now assert the new contract: the client gets the fixed `code` and
`request_id`, the exception text is absent, and the `headroom.proxy` log
has the full text under the same `request_id`
(`test_litellm_transport_errors.py`, `test_openai_backend_path.py`,
`test_proxy_compress_endpoint.py`,
`test_proxy_streaming_ratelimit_headers.py`).
- Manual run with every upstream connect failing: streaming
`/v1/messages`, streaming `/v1/chat/completions` and passthrough
`/v1/models` return `code: upstream_unreachable` with no address. A
request with `x-headroom-proxy-token` and `x-headroom-license-token`
leaves neither value in the `--log-file` output.

Not tested live: the Bedrock and LiteLLM / any-llm backend paths (unit
tests only) and OTEL/Langfuse export (tags are filtered before they
reach it).

## Open review points

- Resolved: unclassified exception text (JerrettDavis, 2026-09-29) now
fails closed in `client_message` and `tool_result_error`.
- Resolved: `X-Headroom-Key` is matched (`key` is a credential
component; the `api`+`key` pair rule is gone). No internal or test tag
key has a `key` component.
- `is_credential_header` special-cases `authorization`, `cookie`,
`api-key` and `x-api-key`, but its only caller passes only
`x-headroom-*` names, so those branches never run.
- `_is_tls_failure` also matches on message text. Provider HTTP error
responses are now checked first in the backends, so this only affects
non-provider exceptions.
- any-llm: only provider errors raised as `openai`/`anthropic`
`APIStatusError` keep their text; any other any-llm exception type
returns `internal_error`.

Revised 2026-10-01 after review: unrecognised exceptions now return a
fixed `internal_error` message and only provider HTTP error responses
keep their text.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-01 19:28:34 -05:00
Sabee Ur Rehman KhanandJerrettDavis 58b1454282 fix(settings): report live proxy configuration (#3177)
## Description

The settings schema re-reads process environment values after the CLI
has already resolved the live proxy configuration. Savings-profile
defaults seeded later in startup can therefore make `/settings/schema`
report values that are not actually active. For example, `--lossless`
can produce `ProxyConfig.lossless=True` while the schema reports the
seeded `HEADROOM_LOSSLESS=0`.

This change makes the live `ProxyConfig` the highest-precedence settings
source, preserves provenance for profile-generated environment defaults,
and adds the missing `HEADROOM_MODE` registry field.

Closes #3090

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `HEADROOM_MODE` with canonical `token` and `cache` choices.
- Made live `ProxyConfig` values take precedence in `/settings/schema`.
- Added `runtime_override` metadata for live values that differ from
lower tiers.
- Prevented savings-profile defaults from being labeled as explicit
environment overrides.
- Preserved provenance through multi-worker configuration serialization.
- Added regression tests for precedence, provenance, endpoint reporting,
and worker round-tripping.

## Testing

- [ ] Unit tests pass (`pytest`) — focused settings tests pass; full
suite not run
- [x] Linting passes (`ruff check .`) — changed files checked
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ focused settings-store test run
36 passed, 1 deselected in 0.09s

$ direct savings-profile provenance check
seed provenance check passed (18 generated keys)

$ ruff check <8 changed files>
All checks passed!

$ ruff format --check <8 changed files>
8 files already formatted

$ python -m compileall -q <8 changed files>
# No output

$ git diff --check
# No output
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.12.13, source based on upstream
`5e0ce242` (v0.36.2).
- Exact command / steps: Set profile-seeded `HEADROOM_MODE=cache` and
`HEADROOM_LOSSLESS=0`, then generated the schema with live values
`mode=token` and `lossless=True`, excluding those generated keys from
explicit-environment provenance.
- Observed result: The schema reports `mode=token` and `lossless=true`;
both generated variables report `env_override=false`, and both
differences report `runtime_override=true`.
- Not tested: Complete repository and FastAPI endpoint suites. This
environment lacks the compiled `headroom._core` extension and Rust
toolchain, and dependency downloads are unavailable.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Proxy compression behavior is
unchanged; only settings registry/schema reporting changes.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: The dashboard receives the actual live proxy
values.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A; no
user documentation change required
- [ ] My changes generate no new warnings — focused checks pass; full
suite not run
- [x] I have added tests that prove my fix is effective
- [ ] New and existing unit tests pass locally — focused tests pass;
full suite not run
- [x] I did **not** edit `CHANGELOG.md` — generated by release-please

## Screenshots (if applicable)

N/A — settings API behavior only.

## Additional Notes

This is intentionally narrow for #3090. PR #2346 overlaps by adding
`HEADROOM_MODE`, but currently includes unrelated dashboard/runtime
work; this PR isolates the live-config precedence and provenance
correction.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 12:25:26 -05:00
PeterandJerrett Davis ef1c528e92 fix(grok): route CLI traffic to api.x.ai on shared proxies (#2693)
## Description

`headroom wrap grok` points the Grok CLI at the local proxy via
`GROK_MODELS_BASE_URL` and, when it starts the proxy, sets
`openai_api_url=https://api.x.ai`. When Claude/Codex already own port
8787, wrap reuses that process without restart (other wrappers
attached), so process-wide `OPENAI_API_URL` stays
`https://api.openai.com`.

Grok CLI cannot stamp `x-headroom-base-url` (no custom attribution
headers). Inference then goes to OpenAI with xAI session tokens and
returns 401. Observed on the wire:

- `x-xai-token-auth: xai-grok-cli`
- `user-agent: grok-pager/... grok-shell/...`
- outbound `POST https://api.openai.com/v1/chat/completions` status 401

This PR adds per-request Grok CLI detection and routes chat / models /
responses to `https://api.x.ai` when those signals are present and the
OpenAI target is still its default, without restarting the shared proxy
or changing Claude/Codex paths. An operator-configured gateway is never
bypassed.

Not covered by #2626 (attribution / UA map only when custom base is
already `api.x.ai`).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `is_grok_cli_request()` in `providers/grok/runtime.py` using
narrow wire signals: exact `x-xai-token-auth: xai-grok-cli` and the
`grok-pager/` / `grok-shell/` UA product tokens, compared per
whitespace-split token so wrappers like `litellm-grok/1.0` cannot match
- Resolve OpenAI chat and responses final upstream as:
`x-headroom-base-url` then Grok CLI then process `OPENAI_API_URL`
- Route `GET /v1/models` and OpenAI-classified passthrough through the
same Grok to xAI base
- Add unit tests for detector, resolver priority, and passthrough/models
base selection
- Add `tests/test_grok_shared_proxy_call_sites.py`, which drives
`handle_openai_chat` and `handle_openai_responses` against a stubbed
transport and asserts the outbound URL — reverting either call site to
the old `custom_only or OPENAI_API_URL` formula turns these red
- Gate Grok to xAI routing on the OpenAI target still being its default,
via one shared `route_grok_to_xai()` predicate used by both the
chat/responses resolver and the models/passthrough path, so an
operator-configured gateway is never bypassed

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` / `ruff format`)
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed

### Test Output

```text
# Grok routing: call sites, detector, resolver priority, passthrough/models
uv run pytest tests/test_grok_shared_proxy_call_sites.py \
  tests/test_provider_grok.py tests/test_provider_proxy_targets.py \
  tests/test_proxy/test_openai_upstream_header.py \
  tests/test_openai_codex_routing.py -q
============================== 48 passed in 0.46s ==============================

uv run ruff check .
All checks passed!

uv run ruff format --check .
1331 files already formatted

uv run mypy headroom --ignore-missing-imports
Success: no issues found in 506 source files

# Full suite (uv, Python 3.14, tests + scripts/tests) — honesty note
==== 15 failed, 9480 passed, 592 skipped in 553.64s ====
# Replaying those 15 as a set gives an identical 11 failed / 4 passed at the
# merge base, at HEAD~1 and at HEAD, so this branch introduces no failure.
# Causes are local-env: missing cargo, missing litellm extra, Windows-only
# asyncio path, AF_UNIX path length. The 4 test_proxy_health.py kompress tests
# pass in isolation at every ref and fail only under the full suite
# (pre-existing ordering pollution; the new test file is not the cause --
# it runs clean immediately before test_proxy_health.py).
# tests/test_memory_eval.py excluded: it imports litellm at collection time.
# Fork CI pytest not yet run (action_required until maintainer approves).
```

## Real Behavior Proof

- Environment: macOS arm64; Grok CLI 0.2.117; PR branch
`fix/grok-shared-proxy-xai-routing` (0.33.1-dev) running as the only
proxy on 8787 with `openai_api_url=null` and `--backend anthropic` (same
shared-proxy shape as Claude/Codex-started 8787). Stock Headroom 0.33.0
was first used to capture the before failure.
- Exact command / steps: 1) Before: with stock 0.33.0 on 8787, run
`headroom wrap grok` (or attach via `GROK_MODELS_BASE_URL`) and inspect
`~/.headroom/logs/proxy.log`. 2) After: replace the process with this
branch (`uv run python -m headroom.cli proxy --host 127.0.0.1 --port
8787 --backend anthropic`, no OpenAI target override). 3)
`GROK_MODELS_BASE_URL=http://127.0.0.1:8787/v1 grok -p "Reply with
exactly one word: pong" --always-approve`. 4) Confirm health still shows
`openai_api_url: null` and inspect proxy log outbound lines.
- Observed result: Before (stock): Grok chat with `x-xai-token-auth:
xai-grok-cli` forwarded to `https://api.openai.com/v1/chat/completions`
→ 401. After (this branch): same Grok CLI on the same port; `/v1/models`
for `user-agent: grok-shell/0.2.117` returned 200; chat outbound was
`POST path=https://api.x.ai/v1/chat/completions` with status 200; CLI
printed `pong` and exited 0. Codex models list against the same proxy
still returned 200.
- Not tested: multi-day soak with Claude + Codex + Grok all attached for
hours; production PyPI release build; full-suite green under fork CI
(local full run had 7 env-only failures unrelated to this change);
savings attribution still labels provider as openai without custom base
(orthogonal, #2626).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title

## Screenshots (if applicable)

N/A — evidence is CLI output and proxy logs (redacted auth).

## Additional Notes

- Explicitly out of scope: savings attribution as `provider=xai` without
custom base (see #2626), multi-provider matrix for Kimi/Mistral,
hard-coded per-agent ports.
- ChatGPT Codex auth still wins on `/v1/responses` before this chain
runs.
- Priority: `x-headroom-base-url` > Grok CLI signals (only while
`OPENAI_API_URL` is still the default) > process OpenAI default. The
gate matters because an operator who points the proxy at a gateway
(LiteLLM, Azure, self-hosted vLLM) chose it for every OpenAI-compatible
client: a client User-Agent must not silently bypass it, nor carry that
gateway's `OPENAI_TARGET_API_HEADERS` to a different vendor. The
shared-proxy case this PR fixes runs with `openai_api_url: null`, so it
is unaffected.
- Fork PR workflows may show `action_required` until a maintainer
approves first-time external CI runs.
## Runtime Rollout Safety

- Rollout-managed feature(s): none; routing is selected per request from
narrow Grok CLI wire signals.
- Minimum rollout channel: normal stable release after focused routing
and credential-isolation coverage passes.
- Stable/default behavior changed: only Grok-fingerprinted requests on a
proxy whose OpenAI target remains the default route to api.x.ai;
explicit client base URLs and operator-configured gateways retain
priority.
- Kill switch / disable path: use an explicit non-default OpenAI
gateway/target, or human-revert this routing change.
- Unsafe override required: none.
- Qualification impact: Grok chat, Responses, models, passthrough, and
cross-vendor header-isolation regressions must remain green.
- Rollback path: human-revert the PR; no persisted state or migration is
involved.

---------

Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
2026-10-01 00:11:46 -05:00
AxelRayandJD Davis 94bb055810 fix(anthropic): preserve failed CCR continuations (#3843)
## Description

Anthropic CCR continuations currently parse non-2xx upstream error
bodies as successful model responses. This change treats every non-2xx
continuation as a failed continuation and forwards the original upstream
response without re-serializing it.

Closes #3836

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Reject non-2xx Anthropic CCR continuation responses before parsing
their bodies.
- Preserve the original response object and raw bytes when a
continuation fails.
- Add regression coverage for 401, 429, and 500 continuation responses.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --no-sync pytest tests/test_proxy/test_anthropic_ccr_raise.py -q
4 passed, 2 warnings in 4.10s

$ uv run --no-sync pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_direct_chat_ccr_resolution.py -q
28 passed, 1 warning in 9.31s

$ uv run --no-sync ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_ccr_raise.py
2 files already formatted

$ uv run --no-sync ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_ccr_raise.py
All checks passed!
```

## Real Behavior Proof

- Environment: Linux, Python 3.11.16, local FastAPI TestClient with a
mocked Anthropic upstream, commit
d05cf39b94
- Exact command / steps: `uv run --no-sync pytest
tests/test_proxy/test_anthropic_ccr_raise.py -q`; send a `/v1/messages`
request whose initial response contains `headroom_retrieve` and whose
continuation returns 401, 429, or 500
- Observed result: All three continuation statuses preserve the original
HTTP 200 response body and raw bytes instead of returning the upstream
error object.
- Not tested: Live Anthropic provider/network behavior and full
repository CI.

## Runtime Rollout Safety

- Rollout-managed feature(s): None
- Minimum rollout channel: Not applicable
- Stable/default behavior changed: Failed non-2xx CCR continuations now
retain the original upstream response.
- Kill switch / disable path: Disable CCR response handling with the
existing configuration.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-30 22:43:47 -05:00
gglucassandClaude Opus 5 fed7281120 fix(proxy): reset the cc-switch upstream when Claude Official is selected (#3166)
## Description

The cc-switch reconciler (#1030) captures a third-party
`ANTHROPIC_BASE_URL` as the proxy's upstream and points Claude Code back
at Headroom. The "Claude Official" branch (cc-switch writes `{"env":
{}}`) returned early without dropping that captured upstream.

`HeadroomProxy.ANTHROPIC_API_URL` is a **process-wide class attr** read
per request, so switching back to Official left the previous provider
live. Any Anthropic client still routed through that proxy - a second
Claude Code instance, the VS Code connector, a desktop app that
re-asserts `ANTHROPIC_BASE_URL` on launch - kept reaching e.g.
`api.deepseek.com`, now with Anthropic OAuth credentials attached.

Related: #1264 (Anthropic-compatible third-party upstream support).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `cc_switch_reconciler.tick()`: before leaving Official direct, reset a
captured third-party upstream back to `default_upstream`.
- Guarded on `self.current_upstream not in (None,
self.default_upstream)`, so nothing captured means nothing reset - an
operator-configured upstream is not clobbered by a passing official
tick.
- `tick()` still returns `False` for the direct-official case:
`settings.json` is not rewritten, only the in-process upstream is
corrected.
- Two regression tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_cc_switch_reconciler.py -q
collected 16 items
tests/test_proxy/test_cc_switch_reconciler.py ................           [100%]
16 passed in 0.82s

$ uv run --frozen --extra dev ruff check headroom/proxy/cc_switch_reconciler.py tests/test_proxy/test_cc_switch_reconciler.py
All checks passed!

$ uv run --frozen --extra dev ruff format --check headroom/proxy/cc_switch_reconciler.py tests/test_proxy/test_cc_switch_reconciler.py
2 files already formatted

$ uv run --frozen --extra dev mypy headroom/proxy/cc_switch_reconciler.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 24.6.0, Python 3.10.18, pytest 9.0.3, branch off
`main` @ 5e0ce242 (0.36.2).
- Exact command / steps: applied the source hunk in reverse (`git show
HEAD -- headroom/proxy/cc_switch_reconciler.py | git apply -R -`),
re-ran the suite, then restored the fix and re-ran.
- Observed result: without the fix,
`test_official_resets_a_captured_third_party_upstream` fails with `-
https://api.anthropic.com` / `+ https://api.kimi.com/anthropic` (1
failed, 15 passed) - the captured Kimi upstream stayed live after the
switch back to Official. With the fix, 16 passed.
- Not tested: no live end-to-end run against a real cc-switch install or
a real third-party provider. The reconciler is gated behind
`HEADROOM_CC_SWITCH_RECONCILE=1`, which is off by default, so the change
is inert for anyone not opted in. Verified at the `tick()` level only.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. The cc-switch reconciler is gated by
the `HEADROOM_CC_SWITCH_RECONCILE` env flag (off by default), not by a
rollout channel.
- Minimum rollout channel: N/A - not rollout-managed.
- Stable/default behavior changed: no. With the flag unset the
reconciler never starts and the changed branch is unreachable. With it
set, the change only corrects the in-process upstream attr;
`settings.json` is not touched and `tick()` still returns `False` on
that path.
- Kill switch / disable path: unset `HEADROOM_CC_SWITCH_RECONCILE` (the
default), which stops the whole reconciler including this branch.
- Unsafe override required: none.
- Qualification impact: none. No new config surface, no wire/schema
change, no new dependency, no persisted state.
- Rollback path: revert the commit. Behavior returns to the previous
sticky-upstream state; nothing to migrate or clean up.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 11:29:10 -05:00
charlieanna ffc6edb4ba fix(proxy): preserve Bedrock body-limit error dialect (#3871)
## Description

Bedrock InvokeModel requests that exceed the request-body ceiling were
returned with `error.type=invalid_request_error`, even though the
Bedrock handler reports the same condition as `request_too_large`. This
makes the middleware response inconsistent with the route's wire
contract.

Closes #3867

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Added explicit dialect handling for `/model/{id}/invoke` and
`/model/{id}/invoke-with-response-stream`.
- Preserved the existing Anthropic Messages and OpenAI payload formats.
- Added regression coverage for both Bedrock routes, the Anthropic
route, and the OpenAI fallback, including response-once behavior.

## Testing

- [x] Unit tests pass
- [x] Linting passes
- [x] Type checking passes
- [x] New tests added
- [ ] Manual testing performed

### Test Output

```text
41 passed: tests/test_proxy/test_bedrock_passthrough.py tests/test_proxy_extension_ordering.py
55 passed, 2 skipped: related body-limit suites
ruff check: passed
mypy headroom/proxy/request_body_limit.py: passed
```

## Real Behavior Proof

On main, the oversized Bedrock request returned:

```json
{"error":{"message":"Request body too large...","type":"invalid_request_error","code":"request_too_large"}}
```

With this change, the same request returns:

```json
{"error":{"type":"request_too_large","message":"Request body too large..."}}
```

The request is rejected before any upstream call.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Stable/default behavior changed: only the 413 error payload for
Bedrock InvokeModel routes.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review
- [x] I have added tests proving the fix
- [x] New and existing tests pass locally
- [x] I did not edit `CHANGELOG.md`

## Screenshots (if applicable)

Not applicable.
2026-09-29 22:29:46 -07:00
Amir FathiandJD Davis c946b6b0b0 fix(proxy): gate the raw request body, not just its Content-Length header (#3338)
## Description

Closes #3326.

The four handlers named in the issue (`openai.py`, `batch.py`,
`gemini.py`,
`anthropic.py`) all gate on the `Content-Length` header before calling
`_read_request_json`, but that header is only present when the client
sends
one. A chunked-transfer-encoding request has no `Content-Length`, so it
walks straight past every one of those checks, and
`_read_request_body_bytes`
then reads the whole body through the unbounded `request.body()`.

`_read_request_json` and `_read_request_body_bytes` are the single
chokepoint all six call sites route through (confirmed by reading each
of
the six named lines, plus a repo-wide grep for other
`_read_request_json`
callers), so one change at that chokepoint covers all of them.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `_read_request_body_bounded` in `headroom/proxy/helpers.py`:
reads
the body via `Request.stream()` instead of `Request.body()`, refusing
the
  request with `RequestBodyTooLarge` once the running total crosses
  `MAX_REQUEST_BODY_SIZE`, before the rest of the body arrives.
- `_read_request_body_bytes` now reads through it instead of calling
  `request.body()` directly.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_request_body_decompression_limits.py -q
25 passed, 2 skipped in 4.25s   # 2 skipped = brotli not installed in this environment

$ pytest tests/ -k "helpers or request_body or decompress or proxy_body or size_limit or content_length" -q
246 passed, 12 skipped, 3 failed in 62.57s
# The 3 failures are in tests/test_cli/test_wrap_helpers.py::TestProxyClientRefCounting
# (process ref-counting / PID checks) and reproduce identically against
# unmodified main in this sandboxed container, unrelated to this change.

$ ruff check headroom/proxy/helpers.py tests/test_request_body_decompression_limits.py tests/test_proxy_handler_helpers.py
All checks passed!
$ ruff format --check headroom/proxy/helpers.py tests/test_request_body_decompression_limits.py tests/test_proxy_handler_helpers.py
3 files already formatted
```

`mypy headroom --ignore-missing-imports` did not run cleanly in this
environment: it fails at `numpy/__init__.pyi` with "Type statement is
only
supported in Python 3.12 and greater" before reaching this file, a
pre-existing mismatch between the installed numpy stubs and this repo's
pinned `python_version = "3.10"` mypy target, not something this diff
introduces.

One existing test needed a fixture update:
`test_proxy_handler_helpers.py`'s
`_VertexGeminiImageRequest` stand-in only implemented `.body()`, and the
new
code path calls `.stream()` (a real Starlette `Request` always has
both).
Added a `.stream()` method that yields the same body in one chunk.

## Real Behavior Proof

- Environment: Docker `python:3.12-slim`, headroom-ai 0.37.0's published
  wheel (for the compiled `_core` extension) with this branch's
  `headroom/proxy/helpers.py` overlaid on top.
- Exact command / steps: built a genuine Starlette `Request` over a
hand-rolled
chunked ASGI `receive()` channel (no `content-length` header, matching
what
a real chunked-transfer-encoding request looks like at the ASGI layer),
and
called the actual `_read_request_body_bytes` with
`MAX_REQUEST_BODY_SIZE`
turned down to 10KB so the demonstration does not need to stream tens of
  megabytes.
- Observed result: against unmodified `main`, a 50,000-byte chunked body
with
  no `Content-Length` header is read in full with no error. Against this
branch, the same request raises `RequestBodyTooLarge` before the stream
finishes, and a 5,000-byte chunked body under the cap still round-trips
  byte for byte.
- Not tested: brotli-encoded bodies (package not installed in this
sandbox,
same as the two skipped unit tests above), and the repo's own CI matrix
  (Rust extension build, HuggingFace model prefetch) end to end.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: no for any request under
`MAX_REQUEST_BODY_SIZE` (100MB by default); a chunked request that would
previously have been read unbounded is now refused past that size, which
  is the fix.
- Kill switch / disable path: none by design; raising
`MAX_REQUEST_BODY_SIZE`
  widens the ceiling if an operator needs to.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert this commit, one function plus a two-line
call-site
  change in a single file.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit `CHANGELOG.md`

## Additional Notes

Not in scope: a handful of other call sites in these same handler files
(`batch.py`'s and `bedrock.py`'s passthrough forwarders, and two
streaming
passthrough paths in `openai.py`/`anthropic.py`) also call
`request.body()` directly rather than through this chokepoint. None of
them
are among the six lines the issue names, they exist to forward a request
byte-for-byte rather than to parse it as JSON, and fixing them is a
larger,
separate change than this one. Flagging it here rather than leaving it
implicit.

---------

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-29 15:34:50 -05:00
rNoz 4227bd2419 fix(proxy): reap wrap-spawned proxies once no wrap clients remain (#3202)
## Description

Nothing in Headroom ever stops a wrap-spawned proxy whose wrappers are
gone. Two leak paths, both by-products of deliberate hardening without a
counterpart:

1. **Unclean wrapper death.** `_start_proxy` detaches the proxy
(`start_new_session` on POSIX, breakaway console flags on Windows) on
purpose, so a crashed wrapper cannot tree-kill a shared proxy out from
under other clients. But when the wrapper dies without running cleanup
(SIGKILL, crash, terminal close), its client marker goes stale,
`_live_proxy_clients` unlinks it, and the proxy itself is never stopped.
2. **Graceful last-exit leak.** `_make_cleanup` only terminates the
proxy when the exiting wrapper *started* it (`proxy_holder[0] is not
None`). When the starter exits while other clients remain (correct),
every later wrapper exits with no handle, so even a perfectly behaved
multi-wrap topology leaks the proxy at the end.

Observed in production: three `headroom proxy` processes still alive 21
days after their last session, holding ports and being silently reused
by later wraps (which then inherited their stale version and captive
routing; see the two companion PRs).

The fix: wrap-spawned proxies carry `HEADROOM_WRAP_OWNED=1` and run an
orphan-watchdog task that polls the per-port client-marker dir every 30s
and gracefully stops the process once no live markers, no active
WebSocket sessions, and no served-request activity remain for a grace
period (default 900s, floor 60s, `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS`).
Standalone `headroom proxy` instances and persistent installs never get
the marker env var and are never affected; multi-worker runs are
excluded (WS sessions are per-worker, and wrap never requests multiple
workers).

Closes #3199

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/orphan_watchdog.py` (new): env config
(`WRAP_OWNED_ENV`, grace parsing with floor), marker liveness
(`pid_alive` + start-time identity, pruning stale markers), and
`orphan_watchdog_loop` (idle clock reset by live markers, active WS
sessions, or movement of the proxy's served-request counter — the last
covers direct-HTTP and SSH-forwarded clients that write no marker). Stop
is `signal.raise_signal(SIGTERM)`, which reaches uvicorn's handler on
Unix and the emulated handler table on Windows (unlike `os.kill(pid,
SIGTERM)`, which is `TerminateProcess` there).
- `headroom/proxy/server.py`: lifespan spawns the watchdog when
`orphan_watchdog_enabled()` and not a multi-worker run
(`HEADROOM_PROXY_CONFIG_JSON` absent), and cancels it in teardown like
the other periodic tasks.
- `headroom/cli/wrap.py`: `_start_proxy` sets `HEADROOM_WRAP_OWNED=1` in
the proxy subprocess env; `_proc_identity`/`_identity_mismatch` moved to
`headroom/_subprocess.py` (`proc_identity`/`identity_mismatch`) so proxy
code does not import the CLI layer — thin module-level delegates keep
the existing monkeypatch-based tests working unchanged.
- `tests/test_proxy/test_orphan_watchdog.py` (new, 13 tests): env
parsing (default/override/floor/garbage), marker pruning (live, dead
PID, recycled PID, garbage JSON, missing dir), loop semantics (exit
after grace, hold on live client, hold on active WS session, hold on
served-request movement, grace reset on returning client, missing
registry counts as idle).
- `tests/test_cli/test_wrap_claude_vertex_proxy_env.py`: `_start_proxy`
marks the subprocess env wrap-owned.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_proxy/test_orphan_watchdog.py -q
13 passed in 2.77s
$ uv run pytest tests/test_proxy tests/test_cli -q
973 passed, 1 skipped in 49.85s   # 1 pre-existing unrelated failure
                                  # (test_litellm_vertex_backend_path_...) fails
                                  # identically on clean upstream/main
$ uv run ruff format / ruff check on touched files
All checks passed!
$ uv run mypy headroom/_subprocess.py headroom/cli/wrap.py headroom/proxy/orphan_watchdog.py
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 26.5 arm64, Python 3.13, local source build
0.37.0-dev (integration overlay with this PR applied).
- Exact command / steps: start wrap-spawned proxies via
`wrap._ensure_proxy(...)` (which goes through `_start_proxy` and thus
`HEADROOM_WRAP_OWNED=1`), with zero registered clients; wait; inspect
`~/.headroom/logs/proxy.log`.
- Observed result: two such proxies (plain on 8789 PID 48365,
augment-mode on 8791 PID 49729) self-exited at the 900s default grace:
`event=proxy_shutdown reason=signal pid=49729` at 09:51:49 against a
~09:36 start, port released afterwards (`lsof` clean). A machine-wide
sweep beforehand showed three 21-day-old orphans (PIDs 73974, 63454,
79339); with this build they can no longer accumulate.
- Not tested: Windows signal path (reasoned from CPython/uvicorn
semantics; unit-injected `stop` callable covers the loop side); the
multi-worker exclusion with a real `--workers 2` run (guarded by env
presence only); marker pruning under active PID recycling on Linux
without psutil (`/proc` fallback is unit-tested via mocks).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for wrap-spawned proxies, which
now self-exit after 15 clientless minutes. Standalone and persistent
proxies are untouched.
- Kill switch / disable path: omit `HEADROOM_WRAP_OWNED` (not settable
by users of wrap; standalone proxies never have it), or set
`HEADROOM_WRAP_ORPHAN_GRACE_SECONDS` very high.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit; orphaned proxies leak again (status
quo ante).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Adversarially reviewed by a second model (glm-5.2, high reasoning);
initial verdict needs-work, all accepted findings applied (multi-worker
exclusion, Windows-safe stop signal, served-request activity hold for
marker-less clients, wider test timing margins) and re-tested green. The
reviewer independently verified the boot-race ordering, teardown
cancellation, and the kill-vs-self-exit race as already correct.
Companion PRs (same incident): `-dev` version-restart fix,
routing-config reuse check.
2026-09-28 13:31:49 -05:00
iuiuiu 000fefceaa fix(anthropic): reject ping-only SSE from buffered CCR (#3682)
## Description

Buffered CCR deliberately converts a client `stream:true` request into
an upstream `stream:false` request. The SSE recovery gate used the
client-facing `stream` value, however, so an upstream that answered the
JSON request with `200 text/event-stream` bypassed SSE-to-JSON
adaptation. A body containing only ping keepalives could therefore be
relayed as a successful but contentless stream.

Fixes #3266

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Treat the upstream contract as non-streaming while a streaming client
is in buffered CCR mode.
- Let `_adapt_event_stream_to_json()` handle non-compliant upstream SSE
replies in that path.
- A ping-only SSE body now becomes a deterministic `502
upstream_protocol_error` response instead of an empty `200` stream.
- Added a regression test with a `200` SSE response containing only
`event: ping`.

## Testing

- [x] New tests added for the bug fix
- [x] Linting passes (`ruff check`)
- [x] Relevant unit tests pass locally

### Test Output

```text
# before the fix
test_buffered_ccr_rejects_ping_only_sse_as_502 failed:
assert 200 == 502

# after the fix
$ pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_nonstream_sse_policy.py -q
30 passed

$ pytest tests/test_proxy_response_cache_replay.py tests/test_proxy/test_anthropic_buffered_timeout.py -q
18 passed

$ ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py
All checks passed!

$ ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Windows x64, Python 3.13.13, pytest 9.1.1.
- Before: the new regression test simulated a `200 text/event-stream`
reply containing only `event: ping`; the buffered CCR path returned HTTP
200 instead of reporting an upstream protocol error.
- After: the same reply produces HTTP 502 with JSON error type
`upstream_protocol_error`, so the proxy no longer presents a
keepalive-only stream as a successful answer.
- Not tested here: every upstream provider's actual SSE behavior. The
test models the non-compliant ping-only reply described in the linked
issue.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: yes, a previously accepted invalid
upstream response shell is now reported as HTTP 502.
- Kill switch / disable path: not needed; normal JSON and SSE recovery
paths are unchanged.
- Unsafe override required: no.
- Qualification impact: clients receive an explicit recoverable upstream
failure instead of waiting on or accepting an empty response.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove my fix is effective
- [x] Relevant new and existing unit tests pass locally with my changes
- [x] My changes generate no new warnings
- [x] I did **not** edit `CHANGELOG.md`

## AI Disclosure

This change and its tests were prepared with AI assistance and reviewed
by the submitter.
2026-09-24 10:53:28 -05:00
994a89f169 fix(proxy): bound request-log message payloads to the feed window; trim glibc too (#3717)
## Description

A user running an overnight Claude Code session through the proxy on
Linux reported it consuming 100 GB of a 128 GB host (about 3,500
requests at ~92k input tokens each, `--log-messages` on). Two stacked
causes, both fixed here:

1. `RequestLogger` keeps message payloads on all 10,000 entries. With
`--log-messages`, every `RequestLog` in the `MAX_LOG_ENTRIES` deque
holds `request_messages`, `compressed_messages` and `response_content`:
two parsed copies of the whole conversation, ~1.9x their JSON size as
Python objects. The only reader of those payloads is
`/transformations/feed`, which is hard-capped at 100 entries, so the
other 9,900 are retained for nothing and grow linearly for the life of
the process.
2. The periodic allocator trim (#2820, #2879) defaults on only for
macOS. glibc retains freed pages the same way, and per-thread arenas on
a many-core host multiply the high-water mark.

Not changed: `MAX_LOG_ENTRIES` itself (`_build_stats_payload` reads the
light fields of all 10,000), and `get_recent_with_messages(n)` for `n >
100`, which now returns bodies only for the newest 100, matching what
the feed could ever serve.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/request_logger.py`: `MESSAGE_WINDOW = 100`; on every
append, null the three payload fields on the entry that just left the
window. Light fields stay on every entry, so `get_recent` and `/stats`
are unchanged.
- `headroom/proxy/models.py`, `headroom/proxy/server.py`:
`periodic_malloc_trim_enabled` defaults on for `darwin` and `linux` (was
`darwin` only). Platforms without a trim call already disable the task
themselves; `HEADROOM_MALLOC_TRIM=0` still opts out.
- `tests/test_proxy/test_request_logger.py`: window tests (payloads
nulled beyond the newest 100, light fields intact and ordered, feed path
fully populated, short logs untouched).
- `tests/test_malloc_tuning.py`: default-scope test asserts darwin and
linux on, win32 off.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py -q
22 passed in 1.10s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/models.py headroom/proxy/server.py tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py
All checks passed!
$ uvx ruff format --check <same files>
5 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/models.py
Success: no issues found
```

## Real Behavior Proof

- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, this branch at
ba5f99d7, `uv run --frozen`.
- Exact command / steps: Logged 2,000 `RequestLog` entries through a
real `RequestLogger(log_full_messages=True)`, each carrying a
300-message, ~128 KB conversation in both `request_messages` and
`compressed_messages`, under `tracemalloc`; once with the fix
(`MESSAGE_WINDOW = 100`) and once with the window set to 10,000 to
reproduce the pre-fix retention.
- Observed result: fix: 41 MB live, 100 entries with bodies; pre-fix:
802 MB live, 2,000 entries with bodies. Same 2,000 entries remain in the
deque either way. Extrapolated to the reporter's 10,000-entry cap and
~1.5 MB per entry, the pre-fix retention alone is ~15 GB before
allocator bloat.
- Not tested: end-to-end RSS on a Linux host with the trim default
flipped (no Linux machine available; the trim path itself is covered by
the existing `test_malloc_tuning.py` suite and was measured on macOS in
#2820).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, two defaults. (1) Request-log
entries older than the newest 100 no longer carry message payloads; only
`/transformations/feed` reads them and it is capped at 100, so no served
response changes. (2) The periodic allocator trim is now on by default
on glibc Linux, as it already was on macOS.
- Kill switch / disable path: `HEADROOM_MALLOC_TRIM=0` disables the
trim. The payload window has no switch; it changes no output.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from the Conventional Commit PR title (a CI guard
enforces this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation: no user-facing docs describe `MAX_LOG_ENTRIES` or the
trim default, so nothing to update. Two commits, one per cause, in case
the maintainers want the Linux trim default separately.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-24 10:51:02 -05:00
gglucass 46ac52d39d feat(proxy): let /transformations/feed omit message bodies for number-only pollers (#3672)
## Description

`/transformations/feed` always returns `request_messages`,
`compressed_messages` and `response_content` for every entry, and
`RequestLogger.get_recent_with_messages` builds them with `asdict()`,
which deep-copies each transcript first. A poller that only reads the
per-request numbers (tokens, model, transforms) still pays for a full
copy and a multi-MB JSON serialization on the event loop for each pull.
Measured on a desktop client polling every 20 s: a `limit=100` pull is
~44 MB, of which ~400 bytes per entry are read, and three concurrent
pulls take `/stats?cached=1` from 45 ms to 1.3 s.

This adds `?include_messages=0` (default `1`, byte-identical output to
today). When off, the three body fields are omitted from the response
and never walked server-side.

Closes #

## Type of Change

- [x] Performance improvement

## Changes Made

- `RequestLogger.get_recent_with_messages(n, include_messages=True)`:
when `False`, builds each entry from the dataclass fields minus the
three payload fields, deep-copying only what it returns (same idiom as
#3613 for `get_recent`).
- `/transformations/feed` accepts `include_messages: bool = True` and
omits the three body keys when it is off. Everything else in the item,
including `log_full_messages`, is unchanged.
- Each item also carries `uncached_input_tokens`, `cache_write_tokens`
and `cache_read_tokens` (already stored on `RequestLog`, never exposed
by the feed). A number-only poller can then rate `tokens_saved` against
new input (uncached + cache_write) the way `/stats`
`new_input_savings_percent` does; the item's full-transcript
`savings_percent` is unchanged. Present in both the full and the slim
response.
- Tests: logger test proves the payloads are not traversed (a leaf whose
`__deepcopy__` raises); endpoint test proves the slim item equals the
full item minus the three keys and carries the cache split.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_transformations_feed.py tests/test_proxy/test_request_logger.py -q
9 passed in 4.53s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/server.py tests/test_proxy/test_transformations_feed.py tests/test_proxy/test_request_logger.py
All checks passed!
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/server.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 15, Python 3.12, this branch, in-process
`RequestLogger` seeded with 100 entries carrying ~160 KB of messages
each (the size of a mid-session Claude Code turn).
- Exact command / steps: `get_recent_with_messages(100)` and
`get_recent_with_messages(100, include_messages=False)`, timed, then
`json.dumps` of each result, timed and sized.
- Observed result: include_messages=True: copy 29.4 ms, json 56.5 ms,
24.45 MB. include_messages=False: copy 1.3 ms, json 1.9 ms, 0.05 MB.
Same 100 entries, same numeric fields.
- Not tested: the dashboard UI (it does not pass the parameter, so it
gets the default, unchanged response).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a (opt-in query parameter)
- Stable/default behavior changed: additive only; without the parameter
every existing key is byte-identical, and three integer keys are added
to each item
- Kill switch / disable path: callers stop passing `include_messages=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit; no persisted state or schema
involved

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: the endpoint has no reference page; the parameter is
documented in the handler docstring.
- Overlaps #3613 on the `_HEAVY_FIELDS` constant and the import lines;
both hunks are identical so either merge order applies cleanly.
2026-09-23 23:12:15 -05:00
44b8e7ba87 fix(proxy): stop get_recent deep-copying dropped message payloads (#3613)
## Description

`RequestLogger.get_recent()` built each returned row with
`dataclasses.asdict(entry)` and only then dropped `request_messages`,
`compressed_messages` and `response_content`. `asdict` deep-copies every
field first, so every call walked the full request and compressed
transcripts of every entry it returned, for fields it was about to
discard.

`/stats` calls `get_recent(10_000)` synchronously on the event loop on
every build (both the uncached path and the `?cached=1` snapshot
rebuild). Measured against 0.37.0 with a 300-message Claude Code
transcript shape (~400 KB per entry): 3.4 ms per entry, so a full 10k
deque costs ~34 s per `/stats` call, during which the loop serves
nothing else. A desktop dashboard polling `/stats` reproduced this in
production: 4,769 `/stats` calls in 55 min with a mean latency of 517 s,
`/v1/messages` mean 135 s (max 1,839 s), 61k `Exceeded concurrency
limit` warnings and a watchdog restart storm (Headroom Desktop user
report, 2026-09-16).

The fix builds the row from `fields()` while skipping the heavy fields,
so the payloads are never touched. Kept fields are scalars plus flat
`dict`/`list` containers; those get the same shallow copy `asdict`
produced for them, so callers still get copies rather than aliases into
the log entry.

Fixes the backend half of the desktop report; the desktop side stops
polling uncached `/stats` for connector verification separately.

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- `headroom/proxy/request_logger.py`: `get_recent` iterates `fields()`
and skips `_HEAVY_FIELDS` instead of calling `asdict()` and filtering
afterwards. Shallow-copies the kept `dict`/`list` fields.
- `tests/test_proxy/test_request_logger.py`: new test plants a leaf
whose `__deepcopy__` raises inside both message payloads and asserts
`get_recent` still returns the row with copied (not aliased) containers.
Fails on the previous implementation, passes on this one.

## Testing

- [x] Unit tests pass locally (`uv run --frozen --extra dev pytest
tests/test_proxy/test_request_logger.py`, 5 passed; `pytest
tests/test_proxy -k "stats and not history"`, 6 passed)
- [x] `uvx ruff check` and `uvx ruff format --check` clean on both files
- [x] New test verified to fail on the old code (`git stash` the source
change: 1 failed, 4 passed)

### Test Output

```
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py -q
tests/test_proxy/test_request_logger.py .....                            [100%]

============================== 5 passed in 0.23s ===============================

$ uvx ruff check headroom/proxy/request_logger.py tests/test_proxy/test_request_logger.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS 15, Python 3.12, headroom-ai 0.37.0 venv (the
desktop app's pinned runtime), plus this branch.
- Exact command / steps: Built a `RequestLog` with
`request_messages`/`compressed_messages` set to a 300-message transcript
(3 content blocks each, ~416 KB serialized) and timed
`dataclasses.asdict(entry)` in a loop; then ran the new unit test
against the old and new `get_recent`.
- Observed result: 3.4 ms per `asdict` call on 0.37.0, i.e. ~34 s for
the 10,000-entry `get_recent` that `/stats` performs on the event loop.
With this change the per-row cost is a `fields()` walk plus shallow
copies of two small containers (microseconds), and the new test passes
only on the new code.
- Not tested: an end-to-end `/stats` latency comparison on a long-lived
proxy with a full 10k deque; the production numbers above come from a
user's `proxy.log`, not from a controlled run.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no; `/stats` returns the same rows
and keys, they are just built without walking the dropped payloads
- Kill switch / disable path: none needed; revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes

## Screenshots

n/a

## Additional Notes

`get_recent_with_messages` keeps `asdict` on purpose: it returns the
payloads, so the walk is the point there.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-21 09:37:34 -07:00
Tejas ChopraandClaude Opus 5 aceff2ea9d fix(proxy): gateway turn fixes found by benchmarking behind a gateway (#3619)
## Description

Fixes five defects found by running real Claude Code sessions through a
gateway that uses the `/v1/compress` turn contract (#3605).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- **Prompt text is never compressed on replaying callers.** The coding
profile compresses user messages, and on turn 1 that rewrote a plain
agent's task ("Fix this bug in the library source" became "Fix bug in
source", and test ids lost their `tests/test_` prefix). When
`prefix_replay_guaranteed` is set, `content_router` protects the opening
user messages and the newest user message, for string and list content
alike. The one exception is a newest user message that answers a fenced
shell command in the assistant turn before it: that is a text harness's
tool observation and still compresses. Tool results still compress, and
cache freeze is unchanged.
- **Anthropic's four cache breakpoint limit.** One benchmark request
failed with `max 4 cache_control blocks, found 5`. Session turns in the
OpenAI handler now re-place markers with
`normalize_message_cache_control`. A last-stop guard,
`enforce_cache_breakpoint_budget`, runs in the gateway extension for
Anthropic bodies: it re-places markers, then drops the oldest extras,
and logs `event=cache_breakpoint_budget` when it repairs.
- **Failure relay without a body.** `POST /v1/compress/response`
rejected an upstream 4xx/5xx without a response body as
`missing_response`. A body is now required only on 2xx.
- **Usage totals after a re-drive.** The done payload carries
`billed_usage` summed across rounds. After one or more re-drives,
`response` is never null: it is the hook's replacement or the latest
provider response (including when the hook hands back that same object),
with `usage` set to the totals in its own key shape (chat or Anthropic).
Turns with no re-drive keep `response: null`. Before, the client saw the
last round only.
- **Cold Kompress stall.** When native Kompress init is deferred at
startup, a daemon thread preloads it after a short delay.
`HEADROOM_KOMPRESS_WARMUP=0` disables it. It is off under pytest and
skipped on glibc < 2.28 (the #1908 segfault hosts).
- Docs: `proxy.mdx` and `wiki/proxy.md` describe the 2xx-only body rule
and `billed_usage`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) on the changed modules
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_transforms_content_router.py tests/test_compress_session_mode.py \
    tests/test_compress_route_tokenizer_by_model.py tests/gateway tests/test_proxy \
    tests/test_kompress_preload_deferral.py
672 passed, 4 skipped in 228.49s

$ uvx ruff@0.16.3 format --check <changed files> && uvx ruff@0.16.3 check <changed files>
12 files already formatted
All checks passed!

$ mypy <changed modules>
Success: no issues found in 6 source files
```

New tests: 5 prompt-protection tests in
`test_transforms_content_router.py` (3 fail without the fix),
`tests/gateway/test_cache_breakpoint_budget.py`, failure-relay and
billed-usage tests in `test_redrive_loop.py`, `billed_usage` in the
response-half contract test,
`tests/test_proxy/test_kompress_background_warmup.py`.

## Real Behavior Proof

- Environment: headroom proxy on this branch behind a gateway, Claude
Code with claude-sonnet-5, real Anthropic calls.
- Exact command / steps: 8 smoke sessions (32 calls) on SWE-bench-style
tasks, with a spend logger recording every provider round.
- Observed result: 0 errors. Task statements reached the provider
verbatim. On every re-driven call (2 and 3 rounds) the client's usage
equalled the sum of the per-round provider usage. Kompress warmed in the
background in about 4 s at startup; in a replay of the slow session the
first code-aware turn went from 5.4 s to 1.3 s.
- Not tested: glibc < 2.28 hosts (warm-up is skipped there by design).
The original body behind the 5-breakpoint error was not captured, so the
guard is covered by unit tests rather than a live repro.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: yes. Prompt text on replaying callers
is no longer compressed, the done payload adds `billed_usage`, and
Kompress preloads in the background when deferred.
- Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0` for the
warm-up. The other changes are correctness fixes with no switch.
- Unsafe override required: no.
- Qualification impact: none expected; the breakpoint guard only acts on
bodies that would otherwise get a 400.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Open follow-up, not in this PR: steady-state code-aware compression
falls back to Kompress at 0.3 to 1.3 s per tool result even when it
saves nothing. A gate for that is worth a separate change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 08:24:53 -07:00
Abhay SinghandJerrettDavis ae75ca49a0 feat(proxy/model-router): add require_tools route condition (#2362)
## Description

Extends the cost-aware model router (#1706) with a `require_tools` route
condition — the inverse of the existing `require_no_tools`.

Today an operator can route **tool-free** turns to a cheaper model
(`require_no_tools`), but there is no way to express the opposite: keep
plain chat on a cheap model while sending **agentic, tool-using** turns
to a more capable one. That is a very common cost/quality split (simple
Q&A on a small model, tool-loop/agent work on a strong model), and it is
exactly symmetric to what the router already supports.

## Fix

Add a `require_tools: bool = False` field to `ModelRoute`:

- `matches()` gains `if self.require_tools and not has_tools: return
False`, so a rule with `require_tools` matches only when the request
declares tools.
- `_route_from_entry` parses `"require_tools"` with the same strict,
fail-open validation as `require_no_tools` (a non-boolean skips just
that rule rather than silently widening it), and `require_tools` is
added to `_ALLOWED_ROUTE_KEYS` so it is not rejected as an unknown key.
- Setting both `require_tools` and `require_no_tools` on one rule is an
AND of contradictory conditions, so the rule simply never matches — a
harmless operator error, not a crash.

Example:

```json
[
  {"name": "agentic->strong", "require_tools": true, "to_model": "gpt-5.4"},
  {"name": "chat->cheap", "require_no_tools": true, "max_input_tokens": 8000, "to_model": "gpt-5.4-mini"}
]
```

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/model_router.py`: add the `require_tools` field to
`ModelRoute`, honor it in `matches()`, allow + strictly parse the key in
`_route_from_entry`/`_ALLOWED_ROUTE_KEYS`.
- `tests/test_proxy/test_model_router.py`: cover `require_tools`
matching, the contradictory both-set case, and env parsing (valid +
non-boolean fail-open).
- `CHANGELOG.md`: Features entry.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_proxy/test_model_router.py -q
33 passed in 0.53s
$ uvx ruff@0.15.17 check headroom/proxy/model_router.py tests/test_proxy/test_model_router.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/model_router.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. `model_router.py` is a pure, dependency-free component, so
its full pytest runs locally (no ML stack) — the 33 tests above include
the new ones.
- Exact command / steps: `ModelRoute(to_model="strong",
require_tools=True).matches(...)` for tool/no-tool requests; a rule with
both `require_tools` and `require_no_tools`; and
`ModelRouterConfig.from_env(...)` with valid and non-boolean
`require_tools`.
- Observed result: `require_tools` matches only when `has_tools` is
True; the contradictory rule never matches; a valid `require_tools`
parses into the route; a non-boolean `require_tools` fails open (the
route is skipped). `require_no_tools` behavior is unchanged.
- Not tested: a live proxy round trip; the router is wired into the
handlers separately, and this change is confined to the pure
route-matching/parsing component that those handlers already call.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

`model_router.py` is a pure component with no heavy imports, so unlike
most of my recent PRs I was able to run its full unit-test file locally
(33 passed). This change is purely additive: existing routes without
`require_tools` behave exactly as before.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-09 23:11:00 -05:00
gglucassandClaude Fable 5.1 427fa76f60 fix(savings): count compression savings once per conversation, not once per turn (#3480)
## Description

On the OpenAI `/v1/responses` path, a request's `tokens_saved` is the
running total for the whole **conversation**, not for that turn. Codex
re-sends the entire transcript every turn and the router recompresses
all of it, so the same removed token is booked again on every remaining
turn. Anthropic's frozen cached prefix means its `tokens_saved` is
already novel-only. The two providers have therefore never meant the
same thing by the same field, and everything that sums it inherits the
difference.

Measured across one machine's retained proxy logs on 2026-09-08:
gpt/codex models reported 8.10M tokens saved against 3.90M tokens of new
input (2.1x), where Claude reported 4.18M against 43.3M. Two requests
from the same machine and the same minute:

```text
gpt-6-astra   tok_before=156260 tok_after=80240 tok_saved=76020 cache_read=66304
claude-opus-5 tok_before=114024 tok_after=113566 tok_saved=458   cache_read=98503
```

What inherits the inflation: lifetime savings totals,
`by_model.savings_percent`, the dollar figure (repeat removals priced at
the full uncached input rate when the alternative was a cache read at
roughly 0.1x), and any rate dividing savings by new input, which then
has no 100% ceiling.


## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `headroom/proxy/conversation_savings.py`: a bounded, thread-safe
per-conversation ledger that converts the cumulative-per-turn series
into the incremental one. 512 conversations, LRU; a forgotten
conversation re-counts its transcript once rather than leaking.
- `RequestOutcome` gains `conversation_key` and
`conversation_tokens_saved`, both defaulting to `None`. `None` means
"this path does not distinguish", and the funnel falls back to
`tokens_saved` unchanged.
- `emit_request_outcome` derives the novel figure once and uses it for
the two cumulative consumers (`metrics.record_request`,
`cost_tracker.record_tokens`). The request log and the transformations
feed keep `outcome.tokens_saved`: those tokens really did leave this
request's payload, so per-request surfaces still describe the wire.
- The PERF line gains `tok_novel=` only when it differs from
`tok_saved`.
- The three `/v1/responses` emit sites populate the pair: HTTP
non-streaming, HTTP streaming (via `_stream_response` ->
`_finalize_stream_response` -> `RequestOutcome.from_stream`), and the
WebSocket per-turn and session-residual records. On WS the conversation
total is the last frame's figure, not the session sum, because the
session sum adds cumulative values together.
- The key is derived with the existing `conversation_key_from_body` and
taken **before** compression, so a rewritten first user message cannot
move it mid-conversation. It is left unset when compression did not run,
so a bypassed turn cannot reset the running total.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_conversation_savings.py -q
...............                                                          [100%]
15 passed in 0.30s

$ uv run --frozen --extra dev pytest tests/test_request_outcome.py tests/test_openai_codex_ws_lifecycle.py \
    tests/test_openai_codex_routing.py tests/test_openai_responses_output_shaper.py \
    tests/test_proxy_project_savings.py tests/test_handler_outcome_tag_invariant.py tests/test_proxy/ -q
250 passed, 1 warning in 89.55s

$ uv run --frozen --extra dev pytest tests/ -q
5 failed, 7538 passed, 514 skipped, 6 warnings in 178.47s

$ uv run --frozen --extra dev mypy headroom/proxy/outcome.py headroom/proxy/conversation_savings.py headroom/proxy/handlers/streaming.py
Success: no issues found in 3 source files

$ uvx ruff check headroom/proxy/
All checks passed!
```

The 5 full-suite failures are pre-existing on `origin/main`. Verified by
stashing this branch and re-running the full suite on the base commit,
which fails the same `tests/test_providers/test_deepseek.py` cases
(stale pricing registry) plus the same order-dependent
`tests/test_lossless_mode.py::test_router_lossless_search_no_marker_and_recoverable`.
Both files pass in isolation with this change applied.

## Real Behavior Proof

- Environment: macOS 24.6.0, Python 3.11, `uv run --frozen --extra dev`,
base commit 88f935a1. Field evidence from a separate machine running the
released 0.37.0 wheel with live Codex and Claude Code traffic.
- Exact command / steps: aggregated every `PERF` line in
`~/.headroom/logs/proxy.log*` by model family, summing `tok_saved`
against per-request new input (`tok_after - cache_read + cache_write`);
cross-checked against `/stats` `prefix_cache.by_provider` and
`persistent_savings.by_model`; then ran the suites above against this
branch.
- Observed result: gpt/codex 443 requests, 8,098,522 `tok_saved` against
3,896,164 new input (2.08x, i.e. a 67.5% "share of new input"); claude
2,163 requests, 4,180,023 against 43,296,720 (8.8%). With this change
the same twenty-turn conversation books its removals once:
`test_a_long_conversation_counts_each_removed_token_once` sums twenty
turns reporting 10,000 each and asserts 10,000, which fails on `main`
with 200,000.
- Not tested: no live end-to-end replay against the real ChatGPT
WebSocket transport; the WS path is covered by unit tests and the
existing `tests/test_openai_codex_ws_lifecycle.py` suite only. Bedrock
and Gemini are untouched by construction (they never set the pair) and
were not exercised beyond the full suite.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This is unconditional accounting on
the OpenAI `/v1/responses` path; it is not gated by `rollout.py`
FEATURES and adds no new feature name.
- Minimum rollout channel: N/A (not rollout-managed). Behaves
identically on stable, beta and dev.
- Stable/default behavior changed: Yes, and this is the point.
Cumulative savings totals on `/v1/responses` stop counting the same
removed token once per remaining turn, so reported Codex savings fall by
roughly an order of magnitude on long conversations. Affected surfaces:
`SavingsTracker` / Prometheus totals, `by_model.savings_percent`, the
cost dashboard's savings dollars, and the `headroom savings` ledger.
Per-request surfaces (`tok_before`/`tok_after`, request log,
transformations feed) are unchanged. No provider other than OpenAI
`/v1/responses` is affected: every other handler leaves the new fields
`None` and takes the unchanged `tokens_saved` path, which is covered by
`test_paths_without_conversation_identity_are_unchanged`.
- Kill switch / disable path: None added. Disabling would mean restoring
a known-wrong number, so a flag would only preserve the bug. Reverting
the commit is the disable path; the ledger is a leaf module with no
other callers.
- Unsafe override required: No. No new env vars, no config keys, no
unsafe overrides.
- Qualification impact: Anyone tracking Codex savings across this
version boundary will see a step change that is a correction, not a
regression. A same-window before/after comparison of Codex savings
totals is not valid across this commit; compare per-conversation unique
removals instead. Suggest calling it out in the release notes rather
than shipping it silently.
- Rollback path: Revert the single commit. State is in-process only (a
bounded in-memory `OrderedDict`, no file, no schema, nothing persisted),
so a rollback needs no migration and leaves nothing behind. Records
already written under either accounting are unaffected either way.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Deliberate scope limits, happy to extend if you would rather have them
here:

- **Per-request surfaces keep the wire figure on purpose.**
`tok_before`/`tok_after`, the request log and the transformations feed
still say what left this process for this request. Only quantities that
accumulate across turns de-duplicate. If you would rather the feed also
show the novel figure, it is one line.
- **This lowers reported Codex savings, substantially.** That is the
point, but it is a visible drop for anyone whose traffic is mostly
Codex, so it may deserve a release note rather than a silent bump.
- **The 512-conversation cap is a memory bound, not a correctness one.**
A conversation evicted while still live re-counts its transcript once.
Sized well above what any one client keeps warm; happy to make it
configurable.
- Documentation and CHANGELOG are unchecked above because I did not want
to guess your release framing for a user-visible metric change. Say the
word and I will add both.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-09 11:29:14 -05:00
Tejas ChopraandClaude Opus 5 a507249b92 fix(proxy): give the upstream write phase its own timeout (#3327)
## Description

Partially addresses #3259.

`write` inherited `request_timeout_seconds`, so pushing request bytes
got the same 300s budget as waiting for a model to answer:

```python
httpx.Timeout(
    connect=config.connect_timeout_seconds,   # 10
    read=config.request_timeout_seconds,      # 300
    write=config.request_timeout_seconds,     # 300  <- same knob
    pool=config.connect_timeout_seconds,      # 10
)
```

Sending a request and waiting for a model to think are different
operations, and conflating them left the send **effectively unbounded in
practice**. When an upstream stops draining — a pooled socket whose peer
went away over a laptop sleep, a network change, a NAT timeout — the
send blocks until the OS abandons retransmission, ~180-220s on macOS.
That is *under* the inherited 300s, so httpx never timed out and the OS
ended the request instead. The retry then hit the same wall and stalled
again, which is why one incident blocked the reporter's client for 5-10
minutes.

`--connect-timeout-seconds` cannot help, exactly as the reporter says:
it only guards establishing a *fresh* connection, and a reused one never
connects.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Added `ProxyConfig.write_timeout_seconds` (default **150**), with
`--write-timeout-seconds` / `HEADROOM_WRITE_TIMEOUT_SECONDS` and a
`SettingField` so it appears in the settings registry.
- Used it in `_provider_httpx_client_options` **and** in
`_anthropic_buffered_request_timeout` — the buffered path sets its own
timeout and is the path most likely to carry a large body, so it needed
the split too or it would have kept the unbounded write.

Behaviour after:

```
BEFORE:  connect=10  read=300  write=300  pool=10
AFTER:   connect=10  read=300  write=150  pool=10
```

### What `write` actually bounds, and why the default is 150s

The first revision of this PR used 60s and claimed httpx applies `write`
per write operation, so a large body would be unaffected. **That was
wrong, and I measured it before shipping the correction.** httpx hands a
bytes body to the transport as a *single* write (`ByteStream.__aiter__`
yields the whole body; httpcore's h11 `_send_request_body` wraps each
yielded chunk in one `anyio.fail_after(write)`), so on HTTP/1.1 the
entire upload runs inside one timer.

Against a local peer draining a 16MB body at a steady ~1MB/s:

```
write=3.0:   WriteTimeout after 3.0s      <- healthy peer, still cut off
write=60.0:  send completed in 19.1s
```

So this is a bound on total upload time, not a stall detector. 60s would
have cut off the 7-15MB bodies the reporter sends over a slow uplink —
turning a fix into an outage for exactly the person who filed the issue.

150s is the value that satisfies both real constraints: it carries 15MB
(the largest body #3259 reports) over a ~1 Mbps uplink, and still fires
well before the ~180s OS retransmit ceiling that made the inherited 300s
unreachable. `tests/test_upstream_write_timeout.py` pins both ends of
that range so neither can drift.

On HTTP/2 the body is split by flow control and the waiting-for-window
part is charged to `read` (httpcore `_wait_for_outgoing_flow` →
`_receive_events`), so only real socket writes count against the bound
there.

A genuinely per-chunk bound would separate "dead peer" from "slow peer"
perfectly, but it needs the body fed as an iterator without changing the
wire framing. That is a larger change than this fix warrants; worth a
follow-up if slow-uplink users hit the 150s cap.

### Why the read timeout is deliberately left alone

Time-to-first-byte is a legitimately long wait for a model, and the
buffered Anthropic path already sets its own 600s read for exactly that
reason. Lowering `read` globally would break slow legitimate responses
to fix a rare hang — the wrong trade. Operators who want a tighter read
bound already have `--request-timeout-seconds`; before this PR that knob
also silently shortened the write budget, which is the conflation being
removed.

### Failover

`_retry_request` catches `httpx.TransportError`, and `WriteTimeout` is a
`TimeoutException` under it — so a bounded write drops the bad
connection and re-sends on a fresh one instead of stalling behind a
socket whose peer is gone.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (CI-pinned `ruff` 0.16.3)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
$ pytest tests/test_proxy/ tests/test_proxy_hardening.py tests/test_upstream_write_timeout.py
302 passed in 40.46s

$ pytest tests/test_upstream_write_timeout.py tests/test_proxy/test_anthropic_buffered_timeout.py \
         tests/test_proxy/test_settings_store.py tests/test_cli_proxy_env.py tests/test_cli_proxy_improvements.py
181 passed, 1 skipped in 11.65s
```

## Scope

This does **not** explain the reporter's follow-up: consistent ~60s
stalls on 7-15MB bodies with `HEADROOM_MAX_KEEPALIVE=0`, where there is
no pooled socket to inherit. I would not close #3259 on this PR.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 18:16:58 +05:30