mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
fix/plugin-launcher-isolated-python
106
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2f58877732 |
fix(proxy): check x-headroom-base-url off the event loop in chat and responses (#4856)
## Description
`handle_openai_chat` and `handle_openai_responses` validated
`x-headroom-base-url` with the synchronous `is_safe_upstream_url`, which
waits for the DNS lookup (up to `HEADROOM_UPSTREAM_RESOLVE_TIMEOUT_S`, 3
s by default) on the event-loop thread. Chat did this three times per
request and responses twice, so one slow-resolving host held up every
other in-flight request for several seconds. The other routes in
`headroom/providers/proxy_routes.py` already use
`is_safe_upstream_url_async`.
Both handlers now resolve the header once, through a new
`_resolve_openai_upstream_base_async` that uses the async guard (lookup
on a worker thread, pin scope opened on the request's task), and reuse
that result for the destination and the extra-header decision.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
## Changes Made
- `headroom/proxy/handlers/openai.py`: header parsing split into
`_openai_base_url_candidate`; sync `_resolve_openai_upstream_base` (kept
for `_resolve_openai_upstream`) and new
`_resolve_openai_upstream_base_async` share it;
`_openai_upstream_from(request, custom)` picks the upstream from an
already-resolved override; chat and responses call the async resolver
once.
- `tests/test_proxy/test_openai_upstream_guard_off_loop.py`: asserts the
guard runs once and not on the event-loop thread, for chat and
responses.
- `tests/test_proxy_openai_cache_key_integration.py`: its stub now
patches the async resolver the handler calls.
**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-idna-pin-once` adds the identical `_openai_upstream_from`
and resolve-once hunks (this PR is a superset of that part);
`fix/openai-base-url-reject` changes the rejection branch of
`_resolve_openai_upstream_base`, which after this PR must be mirrored in
`_resolve_openai_upstream_base_async`. Suggested order: idna-pin-once,
then this PR, then reject (rebasing reject means raising the 400 from
the shared rejection path in both resolvers).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# New test on origin/main (
|
||
|
|
6d8121cfd4 |
fix(proxy): send operator extra headers only to a designated origin (#4858)
## Description `upstream_trust.is_trusted_upstream` decides whether operator-configured extra headers (`OPENAI_TARGET_API_HEADERS` / `ANTHROPIC_TARGET_API_HEADERS`, stored as secrets) may be merged for a client-chosen `x-headroom-base-url`. It compared hostnames only, so `http://<designated host>` or `https://<designated host>:<other port>` also received those headers, over cleartext in the http case. It also used one trusted set for every provider, so a chat/responses request with `x-headroom-base-url: https://api.anthropic.com` received the OpenAI extras, and `/v1/messages` pointed at `https://api.openai.com` or the configured OpenAI gateway received the Anthropic extras. Trust is now decided per provider, on the origin (scheme, host, port). The OpenAI extras go only to the OpenAI target / `api.openai.com`; the Anthropic extras only to the Anthropic target, the oMLX target or `api.anthropic.com`; `HEADROOM_UPSTREAM_ALLOWED_HOSTS` entries count for both. Origin rules: - a bare designation (`api.openai.com`, or a scheme-less configured URL) means `https` on port 443; - configured provider API URLs (`OPENAI_TARGET_API_URL`, `ANTHROPIC_TARGET_API_URL`, oMLX, Gemini/Vertex/Cloud Code/Bedrock) designate their own origin, e.g. `http://omlx.internal:8888`; - `HEADROOM_UPSTREAM_ALLOWED_HOSTS` entries may be a bare host, `host:port` (https) or a full URL (`http://gateway.internal:4000`). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update ## Changes Made - `headroom/proxy/upstream_trust.py`: `_origin()`, `trusted_upstream_origins()`, `is_trusted_upstream` matches on origin; malformed ports and bracketed hosts are refused instead of raising; the withheld-headers warning is keyed and logged by origin (`host=http://gateway.internal:4000`) so a listed bare host reached over http still warns; the now-unused host-only helpers `trusted_upstream_hosts` / `_env_allowed_hosts` are removed (only one test used them). - `merge_extra_headers` / `is_trusted_upstream` take `provider`; the OpenAI chat and responses handlers pass `"openai"`, the Anthropic handler (`_build_outbound_anthropic_headers`) passes `"anthropic"`. Without `provider` the union of all targets is used, as before. - Docs: `configuration.mdx` (designation rules, example, upgrade note), `security-model.mdx`, `pipeline-extensions.mdx`. - Tests: new `tests/test_upstream_trust_origin.py` (8 cases, through `merge_extra_headers` and `is_trusted_upstream`); `tests/test_proxy/test_omlx_routing.py` asserts through `is_trusted_upstream` instead of the removed helper. **Sibling PRs:** `fix/upstream-idna-pin-once`, `fix/upstream-guard-off-loop` and `fix/openai-base-url-reject` change how `handlers/openai.py` resolves the URL passed to `merge_extra_headers`; this PR only changes `upstream_trust.py`, so there is no textual overlap and it can merge in any order. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New tests on origin/main ( |
||
|
|
b9f6255395 |
fix(proxy): log client-supplied model ids as one escaped token (#4854)
## Description
The Anthropic handler logged the request's `model` field raw in three
places (`RESPONSE-CACHE-HIT`, the `cache_mode_prefix_mismatch` skip
line, `UPSTREAM_ERROR`). `sanitize_anthropic_model_id` strips ANSI
styling but keeps an interior newline, so a model id with a newline
split one log record into two, and the second line could look like any
other record. The OpenAI handler had the same raw `model=` in its
cache-hit, upstream-error, `/v1/responses` passthrough (HTTP and WS) and
token-recount lines, and the Bedrock handler logged the model id from
the URL path the same way.
All of them now go through `headroom.log_safety.safe_id`, the helper
`providers/openai.py` already uses for model ids: the value is rendered
with `repr` (newlines escaped) and capped in length.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: `safe_id(model)` in the
cache-hit, prefix-mismatch and upstream-error log lines.
- `headroom/proxy/handlers/openai.py`: `safe_id(...)` in the cache-hit,
upstream-error, `/v1/responses` HTTP and WS passthrough, and raw-payload
recount log lines.
- `headroom/proxy/handlers/bedrock.py`: `safe_id(model_id)` in the
compression log line.
- `tests/test_proxy/test_log_model_id.py`: drives `/v1/messages` and
`/v1/chat/completions` through `create_app` with a model id that
contains a newline and checks the `UPSTREAM_ERROR` and
`RESPONSE-CACHE-HIT` records stay on one line.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# new test, with the fix
$ pytest -q tests/test_proxy/test_log_model_id.py
3 passed in 9.18s
# same test with the handler changes stashed (origin/main code)
FAILED tests/test_proxy/test_log_model_id.py::test_upstream_error_log_keeps_model_on_one_line[/v1/messages-headers0]
FAILED tests/test_proxy/test_log_model_id.py::test_upstream_error_log_keeps_model_on_one_line[/v1/chat/completions-headers1]
FAILED tests/test_proxy/test_log_model_id.py::test_anthropic_cache_hit_log_keeps_model_on_one_line
3 failed in 9.36s
# area tests
$ pytest -q tests/test_proxy tests/test_bedrock_*.py tests/test_log_safety.py
738 passed, 1 warning in 251.77s
$ ruff check / ruff format --check (0.17.3) on the 4 changed files: All checks passed, 4 files already formatted
$ mypy headroom/proxy/handlers/{anthropic,openai,bedrock}.py --ignore-missing-imports: Success: no issues found in 3 source files
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, the real `headroom proxy` CLI
(`--stateless --no-telemetry`, isolated `HOME`) with
`--anthropic-api-url` pointed at a local HTTP server that answers every
request with an Anthropic-format 400. No mocks in the proxy process.
- Exact command / steps: `curl -X POST
http://127.0.0.1:<port>/v1/messages` with `"model": "claude-x\nINJECTED
fake log line"`; grep the proxy's stderr.
- Observed result:
- origin/main: `UPSTREAM_ERROR status=400 ... model=claude-x` followed
by a forged line starting `INJECTED fake log line compressed=no ...`;
the provider warning `Unknown Anthropic model 'claude-x` is also split,
with `INJECTED fake log line': using default Claude limit ...` on its
own line.
- this branch: `UPSTREAM_ERROR status=400 ... model='claude-x\nINJECTED
fake log line' compressed=no ...` and `Unknown Anthropic model
'claude-x\nINJECTED fake log line': using default Claude limit ...`,
each on one line; no line starts with the injected text.
- The live run found the provider warning, which the handler tests did
not cover; fixed in the second commit with its own regression test.
- Not tested: a live provider call. Separately observed on both trees
and not changed here: handlers echo the raw model into the
`x-headroom-model` response header, so a model id containing a newline
makes uvicorn/h11 drop the connection for that request.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: log format only: `model=` values are
now quoted (`model='claude-sonnet-4-6'`).
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none; anything that greps these lines for
`model=<id>` sees the id in quotes.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Fix adapted from a proposal by HetCreep.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4854"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
0f3b0d8a21 |
fix(memory): resolve pi cwd before injecting project tools (#4618)
## Description Closes #4442. pi's system `<cwd>` block now resolves the memory project. If project scope is still unresolved with the `empty` fallback, the Anthropic handler no longer advertises unusable memory tools, including tools recorded earlier in the session. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Parse pi cwd only in system text. Keep main's line parser, header/CLI precedence and limited user-message fallback. - Reuse `is_project_unresolved` for context and sticky-tool replay. Client tools and global/user storage behavior stay intact. - Add initialized-handler scope regressions, replace source-inspection/forwarding-only checks, and document project scope in the published memory page. ## Testing - [x] Unit tests pass (`pytest`) — related suite and the normal critical gate, not the entire Python suite locally. - [x] Linting passes (`ruff check .`) — hosted lint passed on 5ace63e2; local filesystem limitation below. - [x] Type checking passes (`mypy headroom`) — normal full hook with its existing `--ignore-missing-imports` setting. - [x] New tests added for the scope regression - [x] Manual testing performed ### Test Output ```text # 16 related test files 329 passed, 12 skipped, 1 warning in 43.07s # ordinary pre-push: make ci-precheck 178 passed, 5 skipped, 1 warning in 10.87s ✅ ci-precheck PASSED — safe to push. uv run ruff format --check . 1860 files already formatted ``` Normal commit hooks (Ruff, formatter, full mypy, commitlint) and the complete pre-push path passed: Rust fmt/clippy/workspace tests, native build/import verification, critical Python tests and commitlint. The existing Starlette test-client warning was not filtered. Local whole-tree Ruff reports EXE002 on five **unchanged** OAuth plugin files: my `fuseblk` checkout exposes them as executable despite Git modes `100644`; `chmod` returns zero without changing the mode. The uncached check agrees. I did not change those files, add shebangs or ignore the rule. Hosted Ruff, formatting and full mypy passed on 5ace63e2; the local filesystem limitation remains. The first full CI caught a missing scope predicate on the existing lifecycle test double. Fixed in [ |
||
|
|
a2bcbd101b |
feat(pipeline): say whose request an OutcomeSnapshot is (#4698)
## Description
`OutcomeSnapshot` is the one public seam a pipeline extension gets at
`OUTCOME_OBSERVED`, but it says nothing about *whose* request it
measured. An extension that wants to split what it observes (an
exporter, a chargeback job, a budget alert) cannot tell projects,
clients or conversations apart without patching core. This adds four
read-only fields and fills them from values `emit_request_outcome`
already has:
- `attribution`: `{dimension: label}`. For now `project` and `client`,
from the same values the savings tracker books, and only when known.
- `conversation_key`: the handler's key, already on `RequestOutcome`.
- `original_tokens`: the pre-compression size, on the same tokenizer as
`tokens_saved`.
- `output_tokens_saved`: the output-shaping estimate this request
booked.
`attribution` is frozen into a read-only `dict` subclass
(`FrozenLabels`) in `__post_init__`. The dataclass is already frozen,
but a frozen field can still hold a mutable dict, so without this one
extension could rewrite the labels the next one reads. A `dict` subclass
rather than a `MappingProxyType` keeps the snapshot hashable, picklable
and safe for `dataclasses.asdict`, `copy.deepcopy` and `json.dumps`, as
it was before.
It also adds `PipelineExtensionManager.add()`. A proxy extension
(`headroom.proxy_extension`) that also wants lifecycle events is
installed after the manager exists, so it cannot be passed to the
constructor. Adding the same object twice is a no-op.
This is PR 1 of the split proposed in #4676 (generic attribution
labels). It stands on its own, and the later PRs only widen what goes
into `attribution`.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/pipeline.py`: `OutcomeSnapshot.attribution`,
`conversation_key`, `original_tokens`, `output_tokens_saved`, and
`PipelineExtensionManager.add()`. All new fields have defaults, so
existing constructors are unchanged.
- `headroom/proxy/outcome.py`: `emit_request_outcome` fills the new
fields.
- Tests: the fields and their immutability, `add()` (late attach, no
duplicates) in `tests/test_pipeline_outcome_stage.py`, and end-to-end
through `emit_request_outcome` in
`tests/test_proxy/test_conversation_savings.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy tests/test_pipeline.py tests/test_pipeline_outcome_stage.py tests/test_5xx_accounting_all_providers.py
615 passed, 7 warnings in 62.72s # upstream/main at
|
||
|
|
bb8d18a56a |
feat(wrap): add headroom wrap droid for Factory Droid (#4143)
## Description Add `headroom wrap droid` so Factory subscription traffic, including Droid Core, can be compressed on the existing Factory plan. Droid ignores `ANTHROPIC_BASE_URL` / `OPENAI_BASE_URL` for Factory-served models. It honors `FACTORY_API_BASE_URL`. This wrapper points that at the local proxy and gives the proxy a Factory upstream (`--factory-api-url` / `FACTORY_TARGET_API_URL`, default `https://api.factory.ai`). With a Factory upstream configured the proxy registers: - `POST /api/llm/a/v1/messages` — compressed with the existing Anthropic pipeline and forwarded to `{factory}/api/llm/a/v1/messages`. A client-sent `x-headroom-base-url` is ignored. `x-headroom-*` headers are stripped. Operator Anthropic extra headers are withheld from the untrusted Factory host. - Every other `/api/*` path — forwarded verbatim to Factory, except Hermes Studio's scoped prefixes and requests that name their own `x-headroom-base-url`. Nothing outside `/api/` is sent to Factory. `select_passthrough_base_url` and `upstream_trust` are untouched, so a leftover Codex, Claude, or Copilot config pointed at a Factory-mode proxy keeps its own upstream and credentials. That is the deliberate difference from https://github.com/headroomlabs-ai/headroom/pull/2352, which is still open and which this does not close. #2352 sends every unrouted request on a Factory-mode proxy to Factory. `wrap droid` writes no Droid config. It threads `factory_api_url` through the existing proxy-reuse logic. With `--no-proxy` it refuses unless `/health` shows a Headroom proxy serving the same Factory upstream. OpenAI-shaped `/api/llm/o/*` traffic is forwarded uncompressed. Those routes are deferred. This commit is `74c889fb` rebased onto current `main` (`0dd8e0b5`). The only conflicts were `docs/content/docs/meta.json` (kept both the new `hermes` page and `droid`) and the `--no-proxy` branch in `headroom/cli/wrap.py` (kept upstream's OpenAI upstream check and the Factory check). Refs #2351 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom wrap droid` sets `FACTORY_API_BASE_URL` to the local proxy and resolves the Factory upstream from `--factory-api-url`, then `FACTORY_API_BASE_URL`, then `https://api.factory.ai`. Loopback, credential, query, and fragment URLs are rejected. - Factory inference and `/api/*` passthrough routes are registered only when a Factory upstream is configured. The default proxy is unchanged. - `/health` reports `factory_api_url`. - Docs: `docs/content/docs/droid.mdx`. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Rebased tree `/home/hermes/workspace/headroom-rebase` at `0dd8e0b5`, importing that tree via `PYTHONPATH` and the existing venv at `/home/hermes/workspace/headroom/.venv` (Python 3.11.16). ```text tests/test_cli/test_wrap_droid.py ............................ [ 31%] tests/test_provider_droid_runtime.py ................................ [ 66%] ``` 60 passed. Those are the wrap launch-env tests and the Factory runtime tests. `tests/test_proxy/test_factory_decoy_proof.py` did not complete in this worktree. The proxy refuses to start because `headroom._core` is not built here (`ModuleNotFoundError`, exit 78). Setting `HEADROOM_REQUIRE_RUST_CORE=false` still did not finish within 180s. The decoy proof (decoy upstream stays silent, Factory stand-in receives the inference request, spoofed `x-headroom-*` headers, REST passthrough, `--no-proxy` refusals) was written for this change and passed in the pre-rebase review session. It was not re-run to a green result after this rebase. `ruff` and `mypy` were not re-run after the rebase. ## Real Behavior Proof - Environment: Linux, Python 3.11.16, rebased commit `0dd8e0b5` on top of `upstream/main` `8c1beb20`. No Factory token. No call to `api.factory.ai`. - Exact command / steps: `PYTHONPATH=/home/hermes/workspace/headroom-rebase /home/hermes/workspace/headroom/.venv/bin/python -m pytest tests/test_cli/test_wrap_droid.py tests/test_provider_droid_runtime.py` - Observed result: 60 passed. - Not tested: a live Factory subscription, `droid` itself, the socket decoy proof on this rebased commit (blocked locally by the missing Rust extension), `ruff`, `mypy`, and CI on GitHub. ## Runtime Rollout Safety - Rollout-managed feature(s): none. Factory routes register only when `factory_api_url` / `--factory-api-url` is set. - Minimum rollout channel: stable. Default proxy behavior is unchanged. - Stable/default behavior changed: no, unless a Factory upstream is configured. - Kill switch / disable path: do not pass `--factory-api-url` and do not set `FACTORY_TARGET_API_URL`. `wrap droid` is opt-in. - Unsafe override required: no. - Qualification impact: none on the default proxy. - Rollback path: revert this commit. No config is written into Droid. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review The pre-rebase commit `74c889fb` was approved by a fresh Claude Opus 5.5 high review with an empty blocking list. This commit is that change rebased onto current main. The two conflict resolutions are described above and have not had a second Opus pass. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Does not close #2352. That PR is the prior attempt. This one is a current-main rebuild with the catch-all leak removed: only `/api/*` is forwarded to Factory. akbarkanso is not a Headroom maintainer. Please do not expect this account to merge. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4143" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `e28dfe6422dfcb9b2cbd5ce0fe673196d3b6ad40`, upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. Updated against upstream/main |
||
|
|
861ef95620 |
feat(proxy/model-router): add shadow mode (HEADROOM_MODEL_ROUTER_LOG_ONLY) (#2395)
## Description Adds a shadow mode to the cost-aware model router (#1706) so operators can validate their routes against real traffic before switching routing on for real. Enabling routing today immediately rewrites the upstream model. That is a big first step to take on faith: an operator wants to see which requests each rule would catch (and confirm nothing important gets downgraded) before letting it change live traffic. Shadow mode provides exactly that dry run. ## How it works Set `HEADROOM_MODEL_ROUTER_LOG_ONLY=1` (alongside the usual `HEADROOM_MODEL_ROUTER_ENABLED` / `HEADROOM_MODEL_ROUTES`). Rules are evaluated normally and the intended route is logged, but the outgoing model is left unchanged. It is implemented entirely inside the pure `ModelRouter.select()` and needs no handler changes: in `log_only` mode a matched rule returns a `ModelDecision` whose `routed_model` equals the original model, so `changed` is `False` and every caller (the Anthropic and OpenAI routing helpers) already leaves the model untouched. The decision `reason` carries the shadow intent, e.g. ``` [shadow] would route gpt-5.4 -> gpt-5.4-mini via rule 'small' (input_tokens=1200, has_tools=False); log_only, model unchanged ``` which the handlers already log at INFO, so the would-be routing is fully observable. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/model_router.py`: add `ModelRouterConfig.log_only` (parsed from `HEADROOM_MODEL_ROUTER_LOG_ONLY` in `from_env`), a `ModelRouter.log_only` property, and the shadow branch in `select()` that returns a matched-but-unchanged decision with a `[shadow]` reason. - `headroom/proxy/server.py`: pass `HEADROOM_MODEL_ROUTER_LOG_ONLY` into `from_env`. - `tests/test_proxy/test_model_router.py`: cover env parsing, the default-off case, that a matched rule leaves the model unchanged in shadow mode (with the shadow reason), and that shadow off still routes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/model_router.py headroom/proxy/server.py tests/test_proxy/test_model_router.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/model_router.py Success: no issues found in 1 source file # model_router is a pure module, so I ran its test file against the real code # in the project venv (uv sync): 33 no-arg tests pass, including the new # shadow-mode ones. ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. - Exact command / steps: built a config via `ModelRouterConfig.from_env("1", "[{...}]", "true")` and called `ModelRouter.select(...)` on a request a rule matches, plus the same with `log_only` off. - Observed result: in shadow mode the decision is `matched=True`, `changed=False`, `routed_model` equals the original, and `reason` starts with `[shadow] would route strong -> cheap`; with shadow off the same request is routed (`changed=True`, `routed_model="cheap"`); `log_only` defaults to `False`. Because `model_router` has no heavy imports, this ran against the actual module, not a replica. - Not tested: a live proxy request in shadow mode; the router is pure and the handlers already consume `ModelDecision`/log its reason unchanged. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `model_router` is a pure component, so I ran its unit tests against the real code in the venv (33 pass). No handler changes are required: shadow mode works by returning a non-changing decision, which the existing Anthropic and OpenAI routing helpers already handle. Disabled by default; only takes effect when routing is enabled and `HEADROOM_MODEL_ROUTER_LOG_ONLY` is truthy. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2395" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `aff3be8e38c361809d98ca19414bfb019c265cef`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean current-main0ddaa56c integration; contributor ModelRouter.select and ModelRouterConfig.from_env ASTs unchanged. Main-relative scope remains three intended source/test files. Fresh validation349passed,11skipped,1warning:60router/wiring tests plus289incoming-main regressions. Three changed Python files pass Rufflint/format,candidate offline lock and whitespace pass. Complete native tracked-input set/all recorded hashes/artifact checksum verified before3982DLL reuse; no freshbuild. Fresh runtime proof closes the outstanding select-only evidence gap: real local proxy TCP/ASGI lifespan forwards to loopback HTTP upstream with synthetic auth/model/content. From canonical environment-factory config, enabled+shadow forwards original claude-sonnet-4-6 and logs exactly one '[shadow] would route ... -> claude-haiku-4-5'; active forwards claude-haiku-4-5 and logs one matched route; disabled forwards original and emits no routing decision. All three HTTP200. Each proxy child stopped and upstream server closed. These are actual local forwarding/log observations, not cloud/provider-account, model-inference, billing or CLI-env parsing proof. Main's separate CLI construction was not changed; do not infer CLI env wiring from environment-factory coverage. Synthetic fixture/log artifacts preserved in pr2395-0dd-runtime-proof.json and associated scripts/logs. Read author/other-human discussion and current Devin runtime-proof note. Current fresh REST audit contains no new external findings. Only three reviewed source/test files differ from main; no workflow/setup/dependency/native/CLI delta, central publication gate passes. Contributor shadow semantics/defaults preserved. Exact published-head CI/Actions/rollup,main ancestry,mergeability and fresh discussion remain before approval. Maintainers handle CI; no PR merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. <!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 --> Maintainer current-main refresh: exact head `41a0f5b920f48f669e6e8ea0a85593c02d6622c0`, authoritative main `12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches exactly match incoming main; contributor behavior preserved. Fresh focused source tests plus tokenizer/response-header/HF/JSON-parser incoming-main controls: 149 passed, 1 warning in 15.46s. Ruff/format/whitespace pass. Router tests and mocked proxy wiring are not live provider/billing proof; shadow mode preserves the actual requested model. Native artifact reuse verified against full recorded input set/every hash and artifact checksum; no fresh native build claimed. Static publication gate excludes inherited workflow-validation/native Docker/Init/Wrap installer paths, with no workflow/setup/dependency/native/CLI changes in PR diff. Actual exact-head hosted workflows and final independent review remain pending. No merge performed. Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
f896161340 |
feat(proxy): unified multi-upstream routing for Anthropic Cloud and local oMLX (#4597)
## Description Adds unified multi-upstream routing for Anthropic Cloud and local Apple Silicon MLX (`oMLX`) inference on a single Headroom proxy instance, with strict allowlist-based local model selection, post-rewrite destination resolution, cloud credential isolation, and trusted upstream extra-header support. ## Why (Rationale & Problem) Developers running hybrid cloud + local Apple Silicon MLX workflows (`oMLX` / `jundot/omlx` serving `/v1/messages` alongside Anthropic Cloud) previously had to run separate Headroom proxy instances or reconfigure upstream URLs when switching between Claude and local `oMLX` models. ## What For (Use Cases & Value) Allows a single Headroom proxy instance to dynamically route explicit local `omlx/*` / `local/*` / MLX model requests (or requests to `/omlx/v1/messages`) to a local `oMLX` server (`HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL`) while keeping all Claude models (`claude-haiku-4-5-20251001`, `claude-haiku-5-5`, `claude-fable-5`, `claude-sonnet-4-5`, etc.) and unmatched models on `api.anthropic.com` with full context compression and telemetry. ## How (Implementation, Invariants & Verification) - **`headroom/proxy/handlers/anthropic.py`**: - `is_omlx_local_model`: Only routes to `omlx_target` when the model explicitly matches local model prefixes (`omlx/`, `mlx/`, `local/`, `mlx-community/`, `gemma`, `qwen`, `llama`, `mistral`, `deepseek`, `phi`, `lfm`, `smollm`, `starcoder`, `codellama`, `yi-`, `nemotron`) or the operator-configured `HEADROOM_OMLX_MODELS` allowlist. Never routes `claude*` models (unless explicitly prefixed with `omlx/` or `local/`) to oMLX. - Evaluates `_maybe_route_model` **before** local/cloud target selection so cloud-to-local and local-to-cloud model rewrites select the matching destination URL and credential policy. - Strips leading `omlx/` or `local/` routing prefixes from `body["model"]` (recording `omlx_strip_model_prefix` on `body_mutation_tracker`) before forwarding to `oMLX`'s native `/v1/messages` endpoint. - Strips inbound Anthropic cloud credentials (`x-api-key` and `authorization`) when forwarding to local oMLX so cloud tokens never leak to the local process, optionally substituting `HEADROOM_OMLX_API_KEY` / `OMLX_API_KEY` when configured. - **`headroom/config.py`, `headroom/proxy/models.py`, `headroom/cli/proxy.py`, `headroom/proxy/server.py`, `headroom/providers/proxy_routes.py`**: - Consistent resolution of `OMLX_TARGET_API_URL`, `HEADROOM_OMLX_BASE_URL`, and `--omlx-api-url` across SDK config, `ProxyConfig`, Click CLI, and server factory startup. - **`headroom/proxy/upstream_trust.py`**: - Includes `omlx_target_api_url` in `trusted_upstream_hosts` so operator-configured `anthropic_extra_headers` are permitted for the designated oMLX gateway. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `resolve_omlx_target_api_url` in `headroom/config.py` and wired `--omlx-api-url` / `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL` across `headroom/cli/proxy.py`, `headroom/proxy/models.py`, `headroom/proxy/server.py`, and `headroom/providers/proxy_routes.py`. - Replaced open-ended `cloud_prefixes` fallback in `headroom/proxy/handlers/anthropic.py` with explicit local-prefix + `HEADROOM_OMLX_MODELS` allowlist matching (`is_omlx_local_model`) and `strip_omlx_model_prefix`. - Moved local/cloud target resolution after `_maybe_route_model` and stripped Anthropic cloud credentials (`x-api-key`, `authorization`) on local oMLX routes. - Added `omlx_target_api_url` to `_API_URL_ATTRS` and `trusted_upstream_hosts` in `headroom/proxy/upstream_trust.py`. - Added 42 unit tests in `tests/test_proxy/test_omlx_routing.py`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/test_proxy/test_omlx_routing.py tests/test_proxy/test_model_router_wiring.py tests/test_proxy/test_anthropic_upstream_header.py tests/test_upstream_guard.py -v 121 passed in 91.28s $ .venv/bin/ruff check headroom/config.py headroom/proxy/models.py headroom/proxy/upstream_trust.py headroom/providers/proxy_routes.py headroom/cli/proxy.py headroom/proxy/server.py headroom/proxy/handlers/anthropic.py tests/test_proxy/test_omlx_routing.py All checks passed! $ .venv/bin/mypy headroom/config.py headroom/proxy/models.py headroom/proxy/upstream_trust.py headroom/providers/proxy_routes.py headroom/cli/proxy.py Success: no issues found in 5 source files ``` ## Live Verification & Proof (CONTRIBUTING.md) - **Setup**: - macOS Apple Silicon (M4 Max), Python 3.14.10 in `.venv`, Headroom proxy configured with `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888` and `ANTHROPIC_API_KEY=sk-ant-api03-...`. - **Commands**: 1. `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888 .venv/bin/pytest tests/test_proxy/test_omlx_routing.py -v` 2. POST `/v1/messages` with `model="claude-haiku-4-5-20251001"`, `model="claude-haiku-5-5"`, `model="claude-fable-5"`, and `model="claude-sonnet-4-5"` while `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888` is active. 3. POST `/v1/messages` with `model="omlx/qwen3-8b"` and `headers={"x-api-key": "sk-ant-cloud-secret"}`. - **Observed results**: - All 33 supported Claude model IDs (including `claude-haiku-4-5-20251001`, `claude-haiku-5-5`, and `claude-fable-5`) routed to `https://api.anthropic.com/v1/messages` with cloud credentials preserved. - `model="omlx/qwen3-8b"` routed to `http://127.0.0.1:8888/v1/messages` with `body["model"]` normalized to `"qwen3-8b"` and inbound Anthropic `x-api-key` / `Authorization` headers stripped (or replaced by `HEADROOM_OMLX_API_KEY` when set). - **Untested cases**: - Multi-node remote oMLX cluster failover (single target URL per proxy instance is supported). ## Real Behavior Proof - Environment: macOS Darwin arm64 (Apple Silicon), Python 3.14.10, `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888` - Exact command / steps: `.venv/bin/pytest tests/test_proxy/test_omlx_routing.py -v` - Observed result: 42 passed in 15.28s — verified cloud routing for all Claude models, local routing and prefix stripping for `omlx/*` / `local/*` / `gemma*` / `qwen*`, pre-target evaluation of `_maybe_route_model`, cloud credential stripping, and `trusted_upstream_hosts` inclusion. - Not tested: Windows named-pipe transport with local oMLX. ## Runtime Rollout Safety - Rollout-managed feature(s): None (opt-in via `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL` / `--omlx-api-url`). - Minimum rollout channel: stable (no-op unless operator sets `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL`). - Stable/default behavior changed: No — when `omlx_target_api_url` is unset, `/v1/messages` routing is 100% unchanged. - Kill switch / disable path: Unset `HEADROOM_OMLX_BASE_URL` and `OMLX_TARGET_API_URL`. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Unset `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL` or revert commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ### Maintainer re-review of the credential-isolation update Reviewed updated author head |
||
|
|
0ddaa56c21 |
fix(vertex): explain Vertex failures in-band, correct the model/region matrix (#2640)
## Description Vertex rejects onboarding requests for a handful of recurring reasons — expired ADC, a partner model never enabled in Model Garden, a model not served in the requested location, a missing optional SDK — and its own error text names none of the fixes. Users see a bare `404` or an opaque `500` through their SDK and cannot tell a proxy bug from a project-config gap. This makes the proxy explain those failures where it happens, adds a `[vertex]` extra so the LiteLLM Vertex backend can actually start, and corrects an example, a test matrix, and three documentation claims that could not have worked as written. Closes #2632 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Explain Vertex failures in-band** (`headroom/providers/vertex/diagnostics.py`, new) - On `401/403/404/429`, attach an actionable hint three ways: a WARNING in the proxy log, an `x-headroom-hint` response header, and a note appended to `error.message`. The last is load-bearing — SDKs raise with the message string and discard headers. - Successful responses are never modified; annotation is idempotent; statuses we cannot explain get no hint rather than a plausible-sounding wrong one. - Routes opt in via a one-line `@with_vertex_diagnostics` decorator reading path params FastAPI already injects (**+8 lines** in `proxy_routes.py`, no re-indentation). - Same treatment for backend setup failures, where `--backend vertex` surfaced LiteLLM's raw `No module named 'vertexai'` as an opaque 500. **Make `--backend vertex` able to start** (`pyproject.toml`, `registry.py`, `cli/proxy.py`) - New `[vertex]` extra carrying `google-cloud-aiplatform>=1.38`. Deliberately **not** folded into `[proxy]`: measured at ~175 MB installed, and the native passthrough routes never needed it. Follows the existing `proxy-prod` split. - Startup preflight keyed on the **resolved provider**, so every alias normalized at `registry.py:229` (`vertex`, `vertex_ai`, `litellm-vertex`, `google-vertex`, `googlevertex`) is covered. Skipped when a backend class is injected, so existing fake-based tests are unaffected. - New `BackendUnavailableError` so the CLI prints the remedy and exits `2` instead of dumping a traceback through startup internals. **Correct things that could not have worked** | | was | is | |---|---|---| | SDK client | `http_options={"api_endpoint": ...}` | `{"base_url": "http://..."}` — `api_endpoint` is not a field on `HttpOptions`; pydantic rejects it before any request is sent | | Thinking config | `ThinkingConfig(thinking_budget_tokens=)` | `thinking_budget=` | | Request payload | `contents: [{parts: [...]}]` | adds `role: "user"` — the `global` endpoint enforces it, regional was silently lenient | | Auth guidance | `gcloud auth print-access-token` | `application-default` — the plain user token gets `401 ACCESS_TOKEN_TYPE_UNSUPPORTED` | | Test matrix | model × location cross-product | explicit servable pairs | - Test matrix is now four pairs that can actually serve: `global`/`gemini-flash-latest`, `global`/`claude-sonnet-4-6`, `europe-west2`/`gemini-3.5-flash`, `us-east5`/`claude-sonnet-4-6`. Gemini 3.x has no US regional endpoint and `-latest` aliases are global-only, so a pinned `us-central1` Gemini case can never pass. No Gemini 2.5 — it retires 2026-10-16. - `test_thinking_levels` asserted only `status_code < 500` while sending an ignored `thinkingBudgetTokens` key: it passed on 400s and proved nothing. Now asserts 200 plus scored usage metadata with the correct `thinkingBudget` spelling. - Cases the caller's project genuinely cannot serve now **skip** with an actionable reason instead of reporting a red proxy. **Docs** (`proxy.mdx`, `installation.mdx`, `wiki/vertex.md`) - `proxy.mdx` claimed `publisher=anthropic` passthrough "routes through the same LiteLLM-Vertex path as `--backend vertex_ai`". It does not — without the flag it is a direct forward, which is why it works on a bare `[proxy]` install where the LiteLLM path cannot even import. - Documented the trap of combining `--backend vertex` with native Vertex URLs: LiteLLM takes project/region/model from its own config and ignores the URL. - `installation.mdx` and `wiki/vertex.md` said the SDK is "not included in any extra". - Cleared 15 pre-existing `ruff` findings in the two files this PR rewrites (4 remain in `plugins/headroom-oauth2`, untouched by this work). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ mypy headroom Success: no issues found in 511 source files $ ruff check . Found 4 errors. # all in plugins/headroom-oauth2, pre-existing, untouched by this PR $ pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q 44 passed, 1 warning in 69.86s # 8 live integration + 36 offline diagnostics, 0 skipped $ pytest tests/ -k "anthropic_messages or handler_anthropic or backend or registry or cli_proxy or vertex or proxy_routes" 853 passed, 49 skipped, 9345 deselected, 2 warnings in 28.41s ``` New tests in `tests/test_vertex_diagnostics.py` (36, no credentials required): status coverage, per-publisher remedy divergence, idempotency, `content-length` consistency after body rewrite, non-JSON and body-less/streaming responses, every backend alias preflighted, other providers unaffected, injected backend bypasses the preflight. ## Real Behavior Proof - Environment: macOS 14, Python 3.11.15, live GCP project (`alanblount-sandbox`) with `aiplatform.googleapis.com` enabled and Claude enabled in Model Garden; ADC via `gcloud auth application-default login` - Exact command / steps: `export GCP_PROJECT_ID=alanblount-sandbox && export GCP_ACCESS_TOKEN=$(gcloud auth application-default print-access-token) && uv run --extra proxy --with google-genai python3 examples/vertex_genai_sdk_demo.py && uv run --extra proxy --with pytest-asyncio pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q && headroom proxy --backend vertex` (exits 2, then retried with `[proxy,vertex]` extra installed) - Observed result: demo printed `✓ Standard response received successfully! > 1, 2, 3, 4, 5` and `✓ Thinking response received successfully!`; 44 tests passed (8 live integration, 36 diagnostics), 0 skipped; `--backend vertex` exited 2 with the remedy message; same request with the extra installed returned 200; invalid region produced `[headroom] hint:` in the response body and `x-headroom-hint` header; success responses were byte-clean - Not tested: Windows; `us`/`eu` multi-region endpoints and Claude 4.7+ (project cannot serve them); Provisioned Throughput; Rust proxy path; `--backend vertex` dropping URL project/model on native Vertex routes (documented as known gap in Additional Notes) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy and CLI behavior; terminal output is quoted above. ## Additional Notes - `uv.lock` intentionally not included: the new extra is opt-in and does not change default resolution. - Error-body rewriting is confined to already-failing responses and wrapped so a diagnostic can never turn a clean upstream error into a proxy 500. - The preflight raises where the provider is resolved (`registry.py`) rather than at the CLI, so programmatic embedders get the same guarantee; the CLI only formats it. - Three test files fail to collect on this branch (`test_hermes_passthrough_compression`, `test_observability_metrics`, `test_observability_tracing`). They predate this work and are unrelated, so they were excluded rather than fixed — happy to open a separate issue. - `[vertex]` was chosen over adding the dependency to `[proxy]` after measuring the footprint at ~175 MB. If maintainers prefer it in `[proxy]`, that is a one-line change. ## Runtime Rollout Safety - Rollout-managed feature(s): none; diagnostics annotate already-failing Vertex responses and the backend dependency is opt-in through the vertex extra. - Minimum rollout channel: normal stable release after Vertex diagnostics, route, and backend-path tests pass. - Stable/default behavior changed: successful responses and native passthrough remain unchanged; supported Vertex failures gain an actionable header/message, and selecting the LiteLLM Vertex backend without its SDK now fails once at startup. - Kill switch / disable path: use native Vertex passthrough without the LiteLLM backend, or human-revert the diagnostic/preflight change. - Unsafe override required: none. - Qualification impact: offline diagnostics, native integration, and mocked LiteLLM Vertex request-shaping tests must remain green. - Rollback path: human-revert the PR; no persisted state or migration is involved. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2640" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `5d7071f348d2462f666fd7ac1ceb8ac18050169a`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration head |
||
|
|
50bf775e8c |
fix(proxy): bypass compression for large admitted histories (#4142)
## Description Closes #4139. A history that is too large to compress should still reach the provider. This keeps the 100 MiB hard admission limit and bypasses request-side transforms above 16 MiB decoded size. The hard limit still applies before forwarding. Current head: `62ffe3defef1684f70d96f1640818d99445a43bb`, merging current main `a2d4fe72` into `1902d8ef`. The 42-test/native/manual/pre-push results below qualify `1902d8ef`, not this new merge. New-head shard 4 has a cached-prefix fixture failure under investigation; coverage uploads are still incomplete. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement - [x] Documentation update ## Changes Made - Keep the startup override and both wire/decompressed hard bounds. Large admitted histories skip normalization, image processing, memory injection, tokenization and compression on Messages, Chat/Responses, Gemini and Bedrock HTTP routes. - Keep provider routing and backend translation, including Copilot Chat, Codex `store:false` and Vertex route-forced streaming. Native forwarding retains decoded bytes unless the provider's wire contract requires a rewrite; translation retains the full message text. - Report actual provider usage, with zero compression savings or attempted tokens for the bypass. Keep `/v1/compress` and WebSocket contracts unchanged. - Add real-provider coverage for large Gemini JSON/SSE requests with query-key auth, and OpenAI JSON/SSE through the installed LiteLLM adapter. No admission-policy or CI changes in this follow-up. ## Testing - [x] Unit tests pass (focused suite and normal critical pre-push suite) - [x] Linting passes (normal hooks) - [x] New tests added for admission, recovery and passthrough accounting - [x] Manual testing performed against the real native build ### Test Output ```sh python -m pytest -q tests/test_proxy_compression_admission.py tests/test_proxy/test_gemini_plugin_routed_streaming.py --cov=headroom --cov-branch # 42 passed, 41 warnings ``` Normal pre-push `make ci-precheck` passed: 1668 Rust tests (3 ignored), 181 Python tests, lint/type checks and commitlint. No hook bypass. The initial deep-module coverage command hit a Pydantic import-isolation error; using the repo's `--cov=headroom` convention passed both complete modules without disabling plugins or tests. The earlier `96c5fe7e` qualification covered the other native protocols, configured/per-request backends, hard-ceiling recovery and complete local shards 2/3. Those runs are historical, not a new whole-suite claim. New-head hosted success is not claimed. ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14, freshly built real `headroom._core`, public Headroom ASGI routes, a real localhost HTTP provider and installed LiteLLM. Synthetic credentials, offline mode and `DO_NOT_TRACK=1`; no live provider calls. - Exact command / steps: Start `create_app(ProxyConfig(backend="litellm-openai", ...))` with all upstream URLs set to the localhost peer and `litellm.api_base=peer+"/v1"`. Leave the production 16 MiB soft/100 MiB hard limits unchanged. POST `"history " * (16777216 // 8 + 1024) + "LAST TURN"` as a user message to Chat Completions, then as a Gemini content part to generateContent/streamGenerateContent, with streaming off/on. The peer rejects a changed history, requires Gemini query-key auth and one `alt=sse`, and returns protocol-valid JSON/SSE with 101 input/1 output tokens. The maintained four small-ceiling cases are in `tests/test_proxy_compression_admission.py` and run with the command above. - Observed result: All four >16 MiB requests returned 200 with the full history intact. JSON/SSE and Gemini auth survived; streamGenerateContent forced SSE even with incoming `alt=json`. Each request recorded exactly 101 provider input/1 output tokens, zero compression attempts/savings and no failures; `/livez` stayed 200. Earlier recorded qualification also covered all five native HTTP protocols, real translating backends and 413 recovery. - Not tested: Live cloud accounts, the reporter's deployed 4 GB VPS, screenshot/image quality or a zero-JSON-parse path. One envelope/model JSON parse remains; no proxy-only RSS guarantee is claimed. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is the admission/compatibility fix. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Keep the 100 MiB hard ceiling; bypass request-side transforms above 16 MiB decoded size on the named HTTP routes. - Kill switch / disable path: Set `HEADROOM_REQUEST_BODY_MAX_BYTES=16777216` (or lower) and restart, accepting local 413s instead of the large-body path. - Unsafe override required: No. - Qualification impact: Local native protocols, backend translation, hard bounds and recovery exercised; live-provider/VPS qualification remains external. - Rollback path: Revert the admission revision and restart; no storage migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I did not edit `CHANGELOG.md` --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
dfcac78119 |
fix(proxy): strip upstream transport framing headers from Gemini response replays (#4605)
## Description The Gemini `generateContent` and `countTokens` response replays (both the optimized fast path, the main path and the continuation-error replay) copied the upstream headers by hand and only removed `content-encoding`/`content-length`. A chunked upstream therefore leaked `transfer-encoding: chunked` to a client that receives an already-decoded, re-framed JSON body — the RFC 9112 wrong-framing class this codebase already avoids elsewhere — along with `connection`, `keep-alive` and the upstream `server` banner. Other paths in the same file and other providers already use the shared `sanitize_forwarded_response_headers`; these five sites were missed. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Refactor (non-breaking change that only restructures code) - [ ] Tests (no production code change) - [ ] Documentation (no production code change) - [ ] Chore (dependencies, CI, tooling, etc.) ## Changes Made - Import and use `sanitize_forwarded_response_headers` in the five Gemini response-replay sites instead of hand-copying headers. - Add regressions for `generateContent` and `countTokens` that assert `server`, `keep-alive`, `connection` and `transfer-encoding` from the upstream do not reach the client while `content-type`, body and `x-headroom-*` stay intact. ## Testing - [x] Unit tests pass (`tests/test_proxy` 491 passed) - [x] New tests added - [x] Manual testing performed (real uvicorn proxy + real httpx against a local fake upstream) ### Test Output ```text pytest tests/test_proxy/test_gemini_response_framing_headers.py -q # parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client # this branch: 2 passed pytest tests/test_proxy -q # 491 passed ruff check / ruff format --check / mypy <changed file> # pass ``` ## Real Behavior Proof <details> <summary>real behavior proof details</summary> - Exact command / steps: python -m pytest tests/test_proxy/test_gemini_response_framing_headers.py -q with a local uvicorn proxy, a local http.server fake upstream and a real httpx client comparing the before/after response headers. - Environment: macOS aarch64, Python 3.14.5 worktree venv, local uvicorn proxy with a Python `http.server` fake upstream, real httpx client, no provider keys. - Observed result: before the change the client received `server: uvicorn, BaseHTTP/0.6 ..., Upstream-Marker/9.9`, `keep-alive`, `connection: keep-alive` and `transfer-encoding: chunked`; after the change only uvicorn's own `server` header remains, with `content-type`, body and `x-headroom-tokens-*` intact for `generateContent` and `countTokens`. - Not tested: a real Gemini API call, the optimized/compression branch as a separate e2e, the `_GeminiContinuationError` branch as its own case, Windows/Linux, and the other providers that still hand-copy headers (left as a follow-up to keep this PR scoped). </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: Gemini response replays no longer forward upstream transport framing or server headers (intended). - Kill switch / disable path: revert this commit. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] No unrelated changes - [x] Tests added/pass - [ ] CHANGELOG updated — no changelog entry for this internal header fix; say so if you want one - [x] No secrets or local paths ## Notes No linked issue (source-derived, with a reproduction); other replay paths use the same hand-copy pattern and are left as a follow-up. Duplicate `date` headers (upstream Date plus uvicorn Date) are the shared sanitizer's existing behavior and out of scope. ## Paste relevant command output or artifact links here <details> <summary>Commands and output</summary> ```text pytest tests/test_proxy/test_gemini_response_framing_headers.py -q # parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client # this branch: 2 passed pytest tests/test_proxy -q # 491 passed ruff check / ruff format --check / mypy <changed file> # pass ``` </details> ## Screenshots (if applicable) N/A - not user-visible; the evidence is the header comparison in the Real Behavior Proof section (real uvicorn proxy + httpx client + local fake upstream). ## Additional Notes Other providers/buffered replay paths still hand-copy headers; left as a follow-up to keep this PR scoped. Duplicate `date` headers are the shared sanitizer's existing behavior and out of scope. No linked issue (source-derived bug with a reproduction). <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4605" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
6ef52c2047 |
fix(proxy): route /v1/models to the x-headroom-base-url upstream (#4308)
## Description Fixes #4302. `GET /v1/models` and `GET /v1/models/{id}` always queried the default provider upstream, so a request carrying `x-headroom-base-url` for a custom OpenAI-compatible gateway got that gateway's key sent to `api.openai.com` and a 401,. Both routes now resolve the upstream through one helper that uses the header and applies the same unsafe-URL rejection (400) as the catch-all. Closes #4302 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Shared upstream resolver in `proxy_routes.py`, plus a test file. - |
||
|
|
e2350bc8fd |
fix(proxy): warm deferred kompress model so /health stops reporting backend=null (#2799)
## Description
On an idle proxy with the ML extras installed and the Kompress model
cached, `/health` and `/readyz` reported `kompress: {"ready": false,
"status": "unhealthy", "backend": null}` forever (#2730).
The cause was not missing wiring: `enable_kompress` defaults to `True`
and `ContentRouter` sits in both pipelines. The chain was:
1. `ContentRouter.eager_load_compressors()` deliberately skips
`preload()`, so a native load never runs before the port binds (#790
hang on Windows, #1908 segfault on RHEL/CentOS 7). It returns `kompress
= "deferred"`.
2. `WarmupRegistry.merge_transform_status` leaves a deferred slot at
`status="null"`.
3. `_reconcile_kompress_health()` promotes the slot only once a
compressor reports `is_ready()`, and nothing loaded the model without a
kompress-routed request.
#3619 (merged after this PR was opened) added
`_start_kompress_background_warmup()`, which loads the model on a daemon
thread shortly after startup. That fixes the load itself. Rebased onto
it, this PR covers what is left:
1. **`/health` says why `ready` is false.** The kompress check gains a
`detail` field. `ready: false` alone cannot tell "still warming" from
"extras not installed" from "warm failed", which is the confusion in
#2730.
2. **The warm thread records its outcome on `warmup.kompress`.** Success
promotes the slot immediately with `source_status="background"`, instead
of leaving `source_status="deferred"` on a loaded slot until the next
reconcile. Failure leaves a bounded reason.
3. **The detail is a closed vocabulary.** `/health` and `/readyz` are
auth-exempt, and loader exceptions carry paths, model ids and URLs, so
anything outside the allowlist collapses to `unavailable`.
Closes #2730
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/kompress_health_detail_policy.py` (new, pure, matching
the existing `*_policy.py` modules): `KOMPRESS_HEALTH_DETAILS` lists the
values the field may report (`loaded`, `warming`, `model not cached`,
`warm failed`, `not installed`, plus the `eager_load_compressors()`
statuses `enabled` / `deferred` / `unavailable`). `public_detail()`
passes those through and maps anything else to `unavailable`.
- `headroom/proxy/server.py`:
- `_start_kompress_background_warmup()` (from #3619, still
`preload(allow_download=True)` on thread `kompress-warmup`) now writes
its result to the slot. On success it calls `mark_loaded(...,
source_status="background")`. On any exception it sets detail `warm
failed` and logs the real cause with `exc_info=True`; the exception text
never reaches the slot. A compressor that raises
`KompressModelNotCached` gets `model not cached`. The local loader only
raises that with downloads off, so on a cold cache it downloads and
`/health` reports `warming` in the meantime.
- `_kompress_detail()` feeds the new field. Every candidate, including
raw `slot.error` and `source_status`, goes through `public_detail()`, so
the allowlist has a single choke point.
- Router/compressor lookup moves to module-level `_kompress_routers()` /
`_kompress_compressors()`, shared by startup and `/health`.
- `enabled`, `ready` and `backend` keep their meaning, so the `/readyz`
aggregate exclusion of optional Kompress still holds.
- Tests:
- `tests/test_kompress_preload_deferral.py`: the warm thread promotes a
cached model with no request (`source_status == "background"`), a
compressor-reported cache miss stays deferred with `model not cached`,
and a failure carrying a path and a token leaves exactly `warm failed`.
- `tests/test_proxy_health.py`: sentinel tests over `/health` and
`/readyz` for both leak paths (`slot.info["detail"]` and `slot.error`).
- `tests/test_kompress_health_detail_policy.py`: covers the policy
directly.
- `tests/test_proxy/test_kompress_background_warmup.py` and
`tests/test_proxy_eager_preload_bind.py`: expectations updated for the
slot promotion.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Type checking: `mypy headroom --ignore-missing-imports` (mypy 1.20.2)
reports one error, in `headroom/fileperms.py` (`os.fchmod` does not
exist on Windows). That file is not touched by this PR, and there are no
errors in `server.py` or the new module.
### Test Output
```text
$ pytest tests/test_kompress_preload_deferral.py tests/test_proxy_health.py tests/test_proxy/test_kompress_background_warmup.py tests/test_kompress_health_detail_policy.py tests/test_proxy_eager_preload_bind.py tests/test_proxy_warmup.py -q
73 passed in 22.36s
$ ruff check headroom tests
All checks passed!
$ ruff format --check headroom tests
1519 files already formatted
$ mypy headroom --ignore-missing-imports --no-site-packages
headroom\fileperms.py:71: error: Module has no attribute "fchmod"; maybe "chmod"? [attr-defined]
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, headroom installed from this
branch, Kompress model `chopratejas/kompress-v2-base` in the local HF
cache, `HF_HUB_OFFLINE=1`, branch rebased on `upstream/main`
|
||
|
|
78736a16bc |
fix(proxy): keep health probes and the Codex attestation out of the proxy log (#4611)
## Description Two things fill and leak through the always-on proxy log (`~/.headroom/logs/proxy-<port>.log`): 1. Health probes. Every `/readyz`, `/livez` and `/health` poll writes an `event=proxy_inbound_request` and an `event=proxy_inbound_response` line at INFO. A desktop wrapper or orchestrator polls these every few seconds: in an Extra Headroom user's proxy log (Codex + Claude Code, 2026-10-27) 23,201 `/readyz` probes were 19% of 54 MB, and on a lighter machine they are 48% of the file. With 10 MB x 5 rotation, that log held only 14 hours; at the busiest hour one 10 MB file lasted about an hour. 2. The Codex attestation. The inbound request line runs headers through `redact_for_wire_debug`, whose key list has no entry for `x-oai-attestation`. Codex sends that ~4 KB signed device attestation on every request, so every Codex request wrote it to the log in clear. That is 3,040 copies (12 MB) in the log above, in a file users are routinely asked to attach to bug reports. #2612 proposed the same key for the opt-in wire-debug capture and was closed for inactivity. The default proxy log uses the same policy, so this fix covers both. Health probes now log at DEBUG (both lines), and `x-oai-attestation` is a redacted key. Every other path still logs at INFO exactly as before. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: a module-level `_HEALTH_PROBE_PATHS` (`/health`, `/livez`, `/readyz`). The inbound request/response lines use DEBUG for a GET to one of them, INFO otherwise. `_AUTH_EXEMPT_PATHS` now refers to the same set instead of repeating it. The aborted/error lines are unchanged. - `headroom/proxy/wire_debug_redaction_policy.py`: `x-oai-attestation` added to `WIRE_DEBUG_SECRET_KEYS`. - Tests: `test_health_probes_log_at_debug_and_traffic_at_info` and `test_wire_debug_redacts_codex_attestation_header`. Both fail on `main` and pass here. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest -q tests/test_wire_debug_redaction_policy.py tests/test_proxy/test_inbound_response_size.py 11 passed, 1 warning in 5.36s # every test file that mentions readyz / livez / proxy_token / inbound_request (36 files) $ uv run --frozen --extra dev pytest -q $(git grep -l -E 'readyz|livez|proxy_token|inbound_request' -- 'tests/*.py' 'tests/**/*.py') 4 failed, 1245 passed, 2 skipped # the 4 are tests/test_cli/test_wrap_persistent.py and fail identically on unmodified main on this machine $ uvx ruff check <changed files> -> All checks passed! $ uvx ruff format --check <changed files> -> 4 files already formatted $ uv run --frozen --extra dev mypy headroom/proxy/server.py headroom/proxy/wire_debug_redaction_policy.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS 27.2, this branch via `uv run --frozen --extra proxy headroom proxy --port 18799 --no-optimize`, `HEADROOM_SKIP_UPSTREAM_CHECK=1`. - Exact command / steps: polled `GET /livez` until ready, then `curl /readyz`, then `curl -H 'x-oai-attestation: {"v":1,"s":1,"t":"SIGNED-DEVICE-TOKEN"}' -H 'x-codex-window-id: w:1' /stats`, then read stderr and `~/.headroom/logs/proxy-18799.log`. - Observed result: neither output has a `/livez` or `/readyz` line; the `/stats` request line shows `"x-oai-attestation": "[REDACTED]"` next to the visible `x-codex-window-id`, and `SIGNED-DEVICE-TOKEN` occurs 0 times in either output. - Not tested: a Codex client end to end; the header value is never used by the proxy, only logged. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a, logging only - Stable/default behavior changed: yes, health-probe inbound lines move from INFO to DEBUG and `x-oai-attestation` is redacted in logged headers; forwarding is unchanged - Kill switch / disable path: run with DEBUG logging to see probe lines again - Unsafe override required: no - Qualification impact: none, no request or response bytes change - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Docs: no doc describes these log lines. Credit to #2612 for the redaction key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4611" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
28172a921f |
fix(proxy): don't fail startup when libc_ver() can't read the interpreter (#4519)
## Description
On Windows with Microsoft Store Python, `headroom proxy` exits during
startup with code 3:
```text
File ".../headroom/proxy/server.py", line 2007, in _start_kompress_background_warmup
libc, version = platform.libc_ver()
File "...\Lib\platform.py", line 200, in libc_ver
with open(executable, 'rb') as f:
OSError: [Errno 22] Invalid argument: 'C:\\Users\\<user>\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\\python.exe'
ERROR: Application startup failed. Exiting.
```
Here's how it happens:
1. When `HEADROOM_KOMPRESS_WARMUP` is unset,
`_start_kompress_background_warmup` calls `platform.libc_ver()` to skip
warm-up on glibc < 2.28 (#1908). It does this on every OS.
2. `platform.libc_ver()` works by opening `sys.executable` and scanning
its bytes.
3. On Store Python, `sys.executable` is the
`...\WindowsApps\...\python.exe` app-execution alias, which can't be
opened for reading, so `open()` raises `OSError` (Errno 22).
4. Nothing catches it, so lifespan startup fails.
This PR catches `OSError` from `libc_ver()` and treats the interpreter
as non-glibc, which is correct on Windows. The warm-up then runs as it
does on any other non-glibc platform. The glibc < 2.28 skip is
unchanged: on glibc, `libc_ver()` returns the version from
`os.confstr('CS_GNU_LIBC_VERSION')` and never opens the executable, so
the new `except` can't fire on the hosts #1908 protects.
Workaround until this ships: set `HEADROOM_KOMPRESS_WARMUP=1` (or `0`),
which skips the `libc_ver()` call.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/server.py`: wrap `platform.libc_ver()` in `try/except
OSError`, falling back to "not glibc".
- `tests/test_proxy/test_kompress_background_warmup.py`: regression
test. When `libc_ver()` raises `OSError`, the warm-up still starts
instead of the error propagating.
## Testing
- [x] Unit tests pass (`pytest
tests/test_proxy/test_kompress_background_warmup.py`)
- [x] Linting passes (`ruff check`, `ruff format --check` on the changed
files)
- [x] New tests added
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy/test_kompress_background_warmup.py -q
7 passed, 1 warning in 3.16s
# new test with the server.py change reverted:
E OSError: [Errno 22] Invalid argument
FAILED tests/test_proxy/test_kompress_background_warmup.py::test_unreadable_interpreter_does_not_fail_startup
1 failed, 6 passed, 1 warning in 2.40s
```
## Real Behavior Proof
- Environment: Windows 11 Pro (10.0.26200), Python 3.12 from the
Microsoft Store (`sys.executable` =
`...\WindowsApps\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\python.exe`),
headroom 0.42.0 from PyPI, default config, `HEADROOM_KOMPRESS_WARMUP`
unset.
- Exact command / steps: `headroom proxy --port 8798` on the installed
0.42.0 (before), then `headroom proxy --port 8799` on 0.42.0 plus this
branch's `server.py` (after), polling `GET /health` each time. In
detail:
1. Interpreter check: `python -c "import platform; platform.libc_ver()"`
raises `OSError: [Errno 22] Invalid argument` on this machine.
2. Before: `headroom proxy --port 8798` using the installed 0.42.0.
3. After: the installed 0.42.0 package copied to a temp directory, with
only `headroom/proxy/server.py` replaced by this branch's version
(0.42.0's `server.py` is byte-identical to `main`'s). Then `headroom
proxy --port 8799` from that copy, polling `GET /health`. I used the
copy because a source checkout has no built `headroom._core`, and the
proxy refuses to start without it.
- Observed result: 0.42.0 aborts startup with the traceback above. With
the patch, the proxy starts, `/health` returns 200, the process stays
up, and the Kompress warm-up thread starts.
```text
=== installed 0.42.0 ===
libc, version = platform.libc_ver()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.2800.0_x64__qbz5n2kfra8p0\Lib\platform.py", line 200, in libc_ver
OSError: [Errno 22] Invalid argument: 'C:\\Users\\<user>\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\\python.exe'
ERROR: Application startup failed. Exiting.
=== 0.42.0 + this patch ===
headroom.proxy - INFO - Headroom Proxy started (version 0.42.0)
headroom.proxy - INFO - Kompress: DEFERRED (warming in the background after startup)
GET /health -> 200 {"service":"headroom-proxy","status":"healthy","ready":true,"version":"0.42.0",...}
process alive after health check: True
```
- Not tested: python.org Windows Python (its `python.exe` is a real
file, so `libc_ver()` doesn't raise there), Linux/macOS (behavior
unchanged unless `libc_ver()` raises), and a request routed through the
proxy to a provider. I didn't wait for the background warm-up to finish;
it only had to start without breaking startup.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This changes the Kompress warm-up
gate at proxy startup.
- Minimum rollout channel: N/A
- Stable/default behavior changed: only where `platform.libc_ver()`
raises `OSError`. Startup used to fail there; now it proceeds and warms
Kompress as on any non-glibc platform. Everywhere else, behavior is
unchanged.
- Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: no
documented behavior changes)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
d9958bfdff |
fix(proxy): recognise every local spelling of the proxy in the cc-switch loop guard (#4405)
## Description The cc-switch reconciler's loop guard compared `ANTHROPIC_BASE_URL` with the exact string `http://127.0.0.1:{port}`. `http://localhost:{port}`, `http://LOCALHOST:{port}/v1/` and `http://127.0.0.1:{port}/v1` were therefore captured as the proxy's own upstream, so requests looped back to the proxy. The guard now compares scheme, port and host, where host is `localhost` or the proxy's own bind address. `[::1]` is a different socket from the `127.0.0.1` the proxy binds, so a provider on `[::1]:{port}` is still captured (review feedback, second commit). Fixes #4400 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/cc_switch_reconciler.py`: add `CCSwitchReconciler._is_self()` (urlsplit; host is `localhost` or the host of `proxy_url`, same scheme and port) and use it in the loop guard. - `tests/test_proxy/test_cc_switch_reconciler.py`: 5 spellings that must not be captured; 5 URLs (other port, https, `[::1]` provider on the same port, third party) that must still be captured; one test that `[::1]` is the proxy only when `proxy_url` is `[::1]`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output First commit, with only its source change reverted (new tests kept): ```text FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_proxy_own_url_spellings_are_not_captured[http://127.0.0.1:8787/v1] FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_proxy_own_url_spellings_are_not_captured[http://localhost:8787] FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_proxy_own_url_spellings_are_not_captured[http://LOCALHOST:8787/v1/] FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_proxy_own_url_spellings_are_not_captured[http://[::1]:8787] ========================= 4 failed, 22 passed in 1.33s ========================= ``` First commit with the fix: 26 passed. Second commit (`[::1]` follow-up), with only its source change reverted (its test changes kept): ```text FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_other_ports_and_hosts_are_still_captured[http://[::1]:8787] FAILED tests/test_proxy/test_cc_switch_reconciler.py::test_ipv6_proxy_address_is_self_only_for_ipv6_bind ========================= 2 failed, 25 passed in 0.84s ========================= ``` With the second commit: ```text ============================== 27 passed in 0.76s ============================== ``` `ruff check` and `ruff format --check` on both changed files: All checks passed / 2 files already formatted. `mypy` was not run. ## Real Behavior Proof - Environment: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn 0.54.0, fastapi 0.158.1, starlette 1.9.0; base `e995eb9f`; `headroom/_core.abi3.so` copied (untracked, not committed) from the official `headroom_ai-0.41.0` wheel. - Exact command / steps: stub upstream on 127.0.0.1:18821 that logs every POST; `CLAUDE_CONFIG_DIR=<dir> HEADROOM_CC_SWITCH_RECONCILE=1 ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18821 python -P -m headroom.proxy.server --host 127.0.0.1 --port 18820 --no-cache --no-rate-limit`, with `<dir>/settings.json` = `{"env": {"ANTHROPIC_BASE_URL": "http://localhost:18820", ...}}`; then one `POST /v1/messages` to 127.0.0.1:18820 (40 s client timeout). Before = parent source (this change reverted with `git apply -R`), after = this change. - Observed result: before, the reconciler logs `captured upstream=http://localhost:18820`, rewrites settings.json to `http://127.0.0.1:18820`, the client times out and the stub receives 0 requests. After, settings.json is left unchanged, the client gets HTTP 200 and the stub receives 1 `POST /v1/messages`. Control with the fix and a real third-party URL (`http://127.0.0.1:18821`): still captured, settings.json rewritten to the proxy, HTTP 200, stub receives 1 request. - Second proof (review feedback): a provider stub on `[::1]:18820`, the proxy on `127.0.0.1:18820`, a different stub on `127.0.0.1:18821` as `ANTHROPIC_TARGET_API_URL`, settings.json `ANTHROPIC_BASE_URL=http://[::1]:18820`, same client request. Before the second commit settings.json is left on `[::1]` and the request reaches the 18821 default upstream, not the provider (provider received none). After it, the reconciler logs `captured upstream=http://[::1]:18820`, settings.json is rewritten to `http://127.0.0.1:18820`, and the provider on `[::1]` receives the `POST /v1/messages` (default upstream: none). - Not tested: real cc-switch overwriting settings.json; a real Anthropic upstream; Windows/macOS; the Rust proxy. Raw output: ```text == BEFORE settings.json after reconcile: {"env":{"ANTHROPIC_BASE_URL":"http://127.0.0.1:18820",...}} client: TimeoutError timed out cc-switch reconciler: captured upstream=http://localhost:18820, base_url -> http://127.0.0.1:18820 stub upstream received: (count: 0) == AFTER settings.json after reconcile: {"env":{"ANTHROPIC_BASE_URL":"http://localhost:18820",...}} client: HTTP 200 {"id": "msg_stub", ...} stub upstream received: STUB-UPSTREAM POST /v1/messages (count: 1) == CONTROL (fix, third-party URL) cc-switch reconciler: captured upstream=http://127.0.0.1:18821, base_url -> http://127.0.0.1:18820 client: HTTP 200 {"id": "msg_stub", ...} stub upstream received: STUB-UPSTREAM POST /v1/messages (count: 1) == SECOND PROOF, BEFORE the [::1] commit settings.json after reconcile: {"env":{"ANTHROPIC_BASE_URL":"http://[::1]:18820",...}} provider on [::1]:18820 received: none default upstream 127.0.0.1:18821 received: STUB-DEFAULT-UPSTREAM POST /v1/messages == SECOND PROOF, AFTER cc-switch reconciler: captured upstream=http://[::1]:18820, base_url -> http://127.0.0.1:18820 settings.json after reconcile: {"env":{"ANTHROPIC_BASE_URL":"http://127.0.0.1:18820",...}} provider on [::1]:18820 received: STUB-PROVIDER-ON-IPV6 POST /v1/messages default upstream 127.0.0.1:18821 received: none ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none (opt-in `HEADROOM_CC_SWITCH_RECONCILE=1`) - Minimum rollout channel: n/a - Stable/default behavior changed: no; the reconciler is off by default - Kill switch / disable path: leave `HEADROOM_CC_SWITCH_RECONCILE` unset - Unsafe override required: no - Qualification impact: none expected - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Documentation checklist item: N/A, no user-facing doc changes. - Scope: only the loop guard; the stale-upstream handling and `_atomic_write` are untouched. - Produced with AI assistance (Claude Code) operated by me (HetCreep); the reproduction was run on my machine on a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b1aaf5b366 |
fix: do not report a seeded stored setting as an env override (#4326)
## Description `apply_to_environ()` seeds each stored value into its own `HEADROOM_*` env var at startup (`os.environ.setdefault(field.env, _serialize(field, value))`). `to_schema()` then set `env_override` from the mere *presence* of that env var: ```python "env_override": bool(os.environ.get(field.env)) and field.env not in env_override_exclusions, ``` So a setting saved from the dashboard appeared as "overridden by an environment variable" and the UI treated it as locked, even though nothing outside headroom ever exported it. That state survived a restart, because `apply_to_environ` re-seeds on every start. Closes #4320 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `to_schema()` now reports `env_override` only when the env var's value **differs** from the value the settings file would serialize to. A var that merely matches the stored value is headroom's own seeded value, not an operator export. - The existing `env_override_exclusions` check is folded into the same condition, so profile-seeded keys keep behaving as before. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Not run locally: this machine has no opentelemetry install, so headroom does not import and the suite cannot be collected: ModuleNotFoundError: No module named 'opentelemetry' Installing a toolchain is out of scope here, so I did not run pytest. GitHub CI is the judge. ``` ## Real Behavior Proof - Environment: headroom source checkout, no third-party deps installed - Exact command / steps: read `settings_store.py`, traced `apply_to_environ` (line 1002) into `to_schema` (line 1097) - Observed result: a stored value's own env var satisfied `bool(os.environ.get(field.env))` and produced `env_override: True` - Not tested: running the test suite (no `opentelemetry`), and the dashboard UI itself ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: only how `env_override` is computed in the settings schema; effective values are unchanged - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - "Unit tests pass locally" is unchecked because the suite cannot be collected without `opentelemetry`; the new test lives in `tests/test_proxy/test_settings_store.py` next to the existing env-override tests and asserts both directions (seeded value → `False`, genuinely differing value → `True`). - The regression test is committed; CI will exercise it. ## Model Used MiniMax M3.1 Flash Free via the opencode agent harness (`minimax-m3.1-flash-free`) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4326" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
11f743dd90 |
fix(savings): key a Responses conversation on its thread, not the shared session (#4440)
## Description `savings_conversation_key` picks `session_id` before `thread_id`. Codex sends both, flat in `client_metadata` (`codex-rs/core/src/responses_metadata.rs`, `client_metadata()`), and its `session_id` is the root thread's id on every request of the session: the auto-review guardian and spawned subagents carry the root's `session_id` with their own `thread_id`. All of them land on one key. `ConversationSavings.novel()` stores each request's running total under that key, so a guardian request (total 0) resets the root's total, and the root's next turn books its whole history as new again. With the guardian running every few turns, a Codex conversation is booked several times over. This changes the precedence so a `thread_id` wins over a `session_id` beside it, at the top level and in `client_metadata`/`metadata`. A body with only one of the two keys keys exactly as before. It matters for #4238 too: `prefix_key` in `headroom/proxy/savings.py` builds on `savings_conversation_key`, so a subagent that shares the root's model, tools and instructions would still share its baseline. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/conversation_savings.py`: `thread_id` before `session_id` in both lookup orders; docstring says why. - `tests/test_proxy/test_conversation_savings.py`: the Codex shape (shared `session_id`, distinct `thread_id`) yields one key per thread; a guardian request between two root turns no longer makes the root re-book its history (`[1000, 0, 200]` instead of `[1000, 0, 1200]`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_openai_codex_ws_lifecycle.py tests/test_output_savings.py tests/test_codex_project_context.py tests/test_savings_items.py tests/test_proxy/test_conversation_savings.py -q 327 passed, 1 warning in 12.64s # the two new tests against the old lookup order $ git checkout upstream/main -- headroom/proxy/conversation_savings.py && pytest tests/test_proxy/test_conversation_savings.py -q 2 failed, 20 passed $ uvx ruff check / ruff format --check (changed files) All checks passed! / 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS, headroom-ai 0.41.0 as installed by Headroom Desktop, Codex CLI 0.156.1 over HTTP SSE, real `~/.codex/sessions` rollouts and a real `~/.headroom/savings_events.jsonl`. - Exact command / steps: replayed two real Codex sessions request by request through `_compress_openai_responses_payload` with Kompress warm, fed each request's running total to `ConversationSavings.novel()` under per-thread keys vs one session key with a guardian request interleaved, and compared against the fresh content each request added. - Observed result: per-thread keys book 16.1% and 11.5% of new input; with a guardian request after every 5th turn the same sessions book 58% and 69%. The local ledger on one machine showed a root conversation re-booking about 45.5k tokens after each guardian call. - Not tested: the WebSocket path end to end (same lookup order; turns carrying `previous_response_id` already book per request), and `mypy headroom`. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes, Codex savings totals drop to once per thread; nothing else reads the key. - Kill switch / disable path: Not applicable. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes - Id-less Codex bodies still key to `None` and book per request; a fallback key for them is a separate change. - Related: #4238 (savings redesign), #3480 (the per-conversation ledger). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4440" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4fd9cddfd2 |
fix(proxy): honor bypass header and license gate in Anthropic batch create (#4411)
## Description `handle_anthropic_batch_create` ignored `x-headroom-bypass: true`, `x-headroom-mode: passthrough` and the `usage_reporter.should_compress` license gate, and always re-serialized the body. The OpenAI and Google batch paths enforce all of these. This PR makes the Anthropic batch path do the same: compression is skipped for each item in either case, and on bypass the client's original bytes are forwarded. Fixes #4404 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/anthropic.py`: read the body with `read_request_json_with_bytes`; compute `_bypass` (`_headroom_bypass_enabled`) and `license_ok` (same expression as `batch.py`, fails open without a reporter); tag `passthrough_reason`; skip the per-item compression loop body when either gate applies; on bypass call `_retry_request` with `original_body_bytes` and `body_mutated=False`. - `tests/test_proxy/test_anthropic_batch_bypass.py`: 4 tests (bypass header, `x-headroom-mode: passthrough`, license denied, unchanged compression path). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output With only the source change reverted (new test file kept): ```text FAILED tests/test_proxy/test_anthropic_batch_bypass.py::test_batch_create_bypass_skips_compression_and_forwards_original_bytes[bypass_headers0] FAILED tests/test_proxy/test_anthropic_batch_bypass.py::test_batch_create_bypass_skips_compression_and_forwards_original_bytes[bypass_headers1] FAILED tests/test_proxy/test_anthropic_batch_bypass.py::test_batch_create_license_denied_skips_compression ==================== 3 failed, 1 passed, 1 warning in 6.77s ==================== ``` With the fix, new tests plus neighbouring files (`tests/test_proxy/test_anthropic_batch_bypass.py tests/test_proxy/test_anthropic_buffered_timeout.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_handlers_batch.py`): ```text ======================== 92 passed, 1 warning in 13.36s ======================== ``` ```text $ ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_batch_bypass.py All checks passed! $ ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_batch_bypass.py 2 files already formatted ``` ## Real Behavior Proof - Environment: WSL2 Ubuntu on Windows 11, Python 3.12.15, branch based on upstream/main `e995eb9f`; `headroom/_core.abi3.so` taken from the official headroom-ai 0.41.0 wheel (untracked, not committed). Real proxy on 127.0.0.1:18860, local stub Anthropic upstream on 127.0.0.1:18861 that logs the byte count and sha256 of each request body. - Exact command / steps: `ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18861 python -P -m headroom.proxy.server --host 127.0.0.1 --port 18860 --no-cache --no-rate-limit`, then a stdlib client POSTs the same 47340-byte `/v1/messages/batches` body twice (non-canonical whitespace, one large tool_result): once without headers, once with `x-headroom-bypass: true`. Run on the code without the fix, then with it. - Observed result: without the fix, the bypass request reaches the stub as 47284 bytes, sha256 `d2e7efe52f09facb` (re-serialized; the sent body was `09cbbe29568d899d`). With the fix, the bypass request reaches the stub as 47340 bytes, sha256 `09cbbe29568d899d`, identical to what the client sent; the request without the header is unchanged by the fix (47284 bytes, `d2e7efe52f09facb`). Without the fix: ```text STUB POST /v1/messages/batches bytes=47284 sha256=d2e7efe52f09facb client sent bytes=47340 sha256=09cbbe29568d899d headers={} STUB POST /v1/messages/batches bytes=47284 sha256=d2e7efe52f09facb client sent bytes=47340 sha256=09cbbe29568d899d headers={'x-headroom-bypass': 'true'} ``` With the fix: ```text STUB POST /v1/messages/batches bytes=47284 sha256=d2e7efe52f09facb client sent bytes=47340 sha256=09cbbe29568d899d headers={} STUB POST /v1/messages/batches bytes=47340 sha256=09cbbe29568d899d client sent bytes=47340 sha256=09cbbe29568d899d headers={'x-headroom-bypass': 'true'} ``` In this run the compression pipeline did not shrink the body (the 56-byte difference is canonical re-serialization), so the real-process check demonstrates the byte-faithful forward, not a compression change. - Not tested: the license-denied branch in a real process (covered only by the unit test with a stub reporter); a real Anthropic API; batch results processing (`handle_anthropic_batch_results`) after a bypassed create; compressed (gzip/zstd) request bodies; `mypy headroom` was not run. Scripts and raw output: `/var/tmp/cb-pub/proof/fix-anthropic-batch-bypass/` (WSL). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: no for requests without the bypass header and with a license that permits compression; with the header or a denied license, batch items are no longer compressed - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none expected - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - On bypass the CCR batch context store may still run if `ccr_inject_tool` is on; it is harmless because no markers are injected. Left unchanged to keep the diff small. - Documentation: none changed; the bypass contract is already documented for the other handlers. Produced with AI assistance (Claude Code) operated by me (HetCreep); the reproduction was run on my machine on a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4411" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
51e4ae36a5 |
fix(proxy): import Response at runtime in the Bedrock handler (#4397)
## Description
`handle_bedrock_invoke` returns `Response(status_code=204)` when the
client disconnects during the body read, but `Response` was imported
only under `TYPE_CHECKING`, so that branch raised `NameError`. This
imports it next to `JSONResponse` inside the handler.
Closes #4393
Fixes #4393
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/bedrock.py`: `from fastapi.responses import
JSONResponse, Response` in `handle_bedrock_invoke`.
- `tests/test_proxy/test_bedrock_passthrough.py`:
`test_client_disconnect_during_body_read_returns_204`.
## Testing
- [x] Unit tests pass (`pytest`) (the five Bedrock test files only)
- [x] Linting passes (`ruff check .`) (changed files only)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# source change reverted, new test only
E starlette.requests.ClientDisconnect
E NameError: name 'Response' is not defined
1 failed, 23 deselected
# with the fix, 5 Bedrock test files
52 passed, 1 warning in 24.57s
ruff check: All checks passed!
ruff format --check: 2 files already formatted
```
## Real Behavior Proof
- Environment: Linux (WSL2), Python 3.12.15, the real `python -m
headroom.proxy.server --bedrock-api-url http://127.0.0.1:18792`, Rust
core from the headroom-ai 0.41.0 wheel, a local stub upstream
- Exact command / steps: a raw-socket client sends `POST
/model/anthropic.claude-3-haiku/invoke` with `content-length: 1000`,
writes about 50 bytes and closes; then a normal request; run with the
import change reverted and on this branch (details below)
- Observed result: reverted, the log shows `NameError: name 'Response'
is not defined` while handling `ClientDisconnect`; this branch logs
`status=204` with no traceback, the partial request is not forwarded,
and the next request returns 200
- Not tested: real Bedrock or a gateway, the streaming route, a client
behind a TCP proxy, a Rust core built from this checkout
Details:
Setup: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn 0.54.0,
fastapi 0.158.1, starlette 1.9.0. Worktree at PR head `c7d9d59e` (base
`e995eb9f`).
`headroom/_core.abi3.so` was copied (untracked, not committed) from the
official `headroom_ai-0.41.0-cp310-abi3-manylinux_2_28_x86_64.whl`
because the checkout has no built Rust core. The core is from 0.41.0,
the Python source is upstream/main plus the PR.
Stub upstream: a 20-line `http.server` on 127.0.0.1:18792 that logs
every POST it receives. Proxy: the real server, `python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18791 --bedrock-api-url
http://127.0.0.1:18792`, started with `HEADROOM_LOG_LEVEL=DEBUG` (the
debug line goes to `~/.headroom/logs/proxy-18791.log`).
Client (raw socket, `bedrock_client.py`): connect, send `POST
/model/anthropic.claude-3-haiku/invoke` headers with `content-length:
1000`, write about 50 body bytes, wait 0.5 s, close the socket. Then, on
the same proxy, send one normal full request.
Before = `headroom/proxy/handlers/bedrock.py` with the PR's one-line
change reverted (`git apply -R`). After = PR head.
### Before (PR change reverted)
```
proxy /livez -> 200
client: sent 1000-byte Content-Length, wrote ~50 bytes, closed socket
client: follow-up full request -> 200 {"stub": "ok"}
--- proxy log
42: DEBUG - [hr_1792672318_000001] bedrock_invoke client disconnected during body read
43: ERROR - event=proxy_inbound_request_aborted id=inbound-1792672318828183190 method=POST path=/model/anthropic.claude-3-haiku/invoke reason=NameError duration_ms=500.61
45: File ".../headroom/proxy/handlers/bedrock.py", line 132, in handle_bedrock_invoke
54: raise ClientDisconnect()
55: starlette.requests.ClientDisconnect
...
110: NameError: name 'Response' is not defined (bedrock.py line 138, during handling of the ClientDisconnect)
--- stub upstream received
STUB-UPSTREAM POST /model/anthropic.claude-3-haiku/invoke body_bytes=49
```
The disconnect branch is reached (the DEBUG line), then `return
Response(status_code=204)` raises `NameError` (`Response` is imported
only under `TYPE_CHECKING`, line 38), so the request ends as
`proxy_inbound_request_aborted` with a traceback instead of a handled
disconnect.
### After (PR head
|
||
|
|
8cedf7be4d |
feat(pipeline): put per-request saved tokens on OutcomeSnapshot (#4231)
## Description Expose per-request message-compression savings and the novel share booked to conversation totals on the immutable OutcomeSnapshot passed to pipeline extensions. Extensions can consume the same readings as the proxy without parsing PERF logs. The previous body described the separate remote-Kompress startup fix in #4234. This description has been corrected by the maintainer to match this PR's diff. ## Type of Change - [x] New feature (non-breaking additive outcome fields) ## Changes Made - Add tokens_saved and novel_tokens_saved to OutcomeSnapshot, with defaults preserving existing construction. - Emit OUTCOME_OBSERVED after the proxy computes the novel share. - Populate both fields from the proxy's accounting result. - Add a regression observing successive Responses outcomes through PipelineExtensionManager. ## Testing - [x] Focused conversation-savings tests pass. - [x] Touched-file Ruff and whitespace checks pass. ### Test Output ```text python -m pytest tests/test_proxy/test_conversation_savings.py tests/test_original_snapshot_in_place_hook.py -q 25 passed, 1 warning in 8.72s ruff check headroom/pipeline.py headroom/proxy/outcome.py tests/test_proxy/test_conversation_savings.py All checks passed! ``` Maintainer verification used the contributor's unchanged outcome implementation. The latest clean main sync incorporates #4241's snapshot-preservation fix; 25 combined conversation/snapshot tests passed. Exact-head hosted CI remains in progress. ## Real Behavior Proof - Environment: maintainer isolated Windows/Python 3.12.13 worktree; actual outcome emitter and PipelineExtensionManager exercised by the focused suite. - Exact command / steps: python -m pytest tests/test_proxy/test_conversation_savings.py tests/test_original_snapshot_in_place_hook.py -q; inspect test_pipeline_extensions_read_both_figures_from_the_snapshot. - Observed result: 25 tests passed, including the regression asserting that extension snapshots carry per-request and novel savings across successive Responses outcomes. - Not tested: maintainer live provider round trip with an installed entry-point extension. The author separately reports replaying recorded Codex outcomes through a real create_app proxy and observing tokens_saved=8575 and novel_tokens_saved=791, matching the recorded PERF figures; this is author-supplied replay evidence, not an independent live inference claim. ## Runtime Rollout Safety - Rollout-managed feature(s): no additional rollout-managed switch introduced. - Minimum rollout channel: existing pipeline-extension policy applies. - Stable/default behavior changed: enabled extensions receive two additive fields; the outcome event is emitted after novelty accounting. - Kill switch / disable path: disable the consuming pipeline extension. - Unsafe override required: no. - Qualification impact: exact-head hosted CI and final human review remain required. - Rollback path: revert this change or retain the previous qualified release. ## Review Readiness - [ ] I performed self-review - [ ] This PR is ready for human review Maintainer qualification above does not invent an author self-review attestation. Final merge readiness remains pending hosted CI and review. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
89b229ce5f |
fix(kompress): time each model-load stage and stop false not-ready warnings during warm-up (#4195)
## Description After every proxy start, the post-startup Kompress warm-up (`HeadroomProxy._start_kompress_background_warmup`) takes several seconds; 22.8 s was seen on one start. Requests that arrive during that time caused three problems: 1. **A false connectivity warning.** `ContentRouter._try_ml_compressor` logged `Kompress model not ready; requests will not be compressed. Check HuggingFace connectivity ...` at WARNING, although the model was only still loading. 2. **A second loader thread.** The warm-up thread was not in `kompress_compressor`'s per-model loader registry (`_download_threads`). `ensure_background_download` therefore started a second thread, which logged `downloading model ... in the background`, waited on `_kompress_lock` until the warm-up finished, and then returned the warm-up's cache entry. 3. **Misleading or missing load logs.** `_load_kompress_onnx` logged `Downloading Kompress ONNX model` before any cache lookup, even when every file came from disk. The `Kompress ONNX loaded` line gave no breakdown, so there was no way to tell why a load took 22.8 s. This PR has one commit for each problem, plus a fourth that fixes an issue Devin Review found in the second: - `feat(kompress)`: the `Kompress ONNX loaded` line now gives wall-clock and loading-thread CPU milliseconds for each load stage, and the start message says "Loading". - `fix(kompress)`: the warm-up thread is registered as the model's loader before it starts. - `fix(router)`: while a loader thread is running, the router logs at INFO, once per router, that the model is still loading. The WARNING remains for the case where nothing is loading the model. - `fix(kompress)`: a warm-up that replaces a live prefetch in the registry waits for it, so the prefetch stays covered. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/kompress_compressor.py`: - `_timed_stage` / `StageTimings`: record wall-clock and `time.thread_time()` milliseconds for each stage. - `_create_onnx_session(stage_ms=...)` times `artifact_resolve`, `onnxruntime_import`, `session_build` and `smoke_run`. When a candidate fails its smoke run and the next one is tried, both attempts are added to the totals. - `_load_kompress_onnx` also times `transformers_import` and `tokenizer_load`, and logs the stages on the `Kompress ONNX loaded` line. Its start message now says "Loading" instead of "Downloading". - `start_kompress_warmup_thread(model_id, warm)` (new): starts the `kompress-warmup` thread and registers it in `_download_threads` under `_download_threads_lock` before it runs. This covers the warm-up's initial delay too. If a prefetch or download thread is already registered, the warm-up takes its slot but waits for that thread to finish before running `warm()`. A live prefetch therefore can't drop out of the registry and be duplicated after a failed warm-up, and the two don't fetch the same files at once (Devin Review). After the warm-up ends, even if it failed, the next `ensure_background_download` replaces the entry as before. - `headroom/proxy/server.py`: `_start_kompress_background_warmup` starts its thread with `start_kompress_warmup_thread` instead of `threading.Thread(...)`. - `is_kompress_load_in_progress(model_id)` and `KompressCompressor.is_loading()` (new): true while a registered prefetch, download or warm-up thread is alive. - `headroom/transforms/content_router.py`: in the not-ready branch, the router logs at INFO once while `is_loading()` is true, and keeps the original WARNING once when no loader is running (for example, a failed download in its retry backoff; `_background_download` has already logged the actual error). Compressors without `is_loading` keep the WARNING. - Tests: - `tests/test_kompress_failsafe.py`: stage totals include a candidate rejected by its smoke run, and the load log names every stage. - `tests/test_proxy/test_kompress_background_warmup.py`: a request during the warm-up starts no second loader, and one after a failed warm-up still does. - `tests/test_kompress_download_backoff.py`: a warm-up that replaces a live prefetch waits for it. No download starts while the prefetch runs, and one can start once both have ended without a model. - `tests/test_content_router_kompress_not_ready_log.py` (new): INFO while loading, WARNING with no loader, INFO and then WARNING after a failed load, WARNING for a compressor without `is_loading`, and a registered warm-up thread counting as loading. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_content_router_kompress_not_ready_log.py tests/test_kompress_failsafe.py tests/test_proxy/test_kompress_background_warmup.py tests/test_kompress_download_backoff.py tests/test_kompress_request_nonblocking.py tests/test_kompress_preload_deferral.py tests/test_router_registry_smartcrusher.py tests/test_content_router_ml_stage_deadline.py tests/test_content_router_concurrency.py tests/test_content_router_exclude_tools.py tests/test_transforms/test_content_router.py -q ======================= 184 passed, 3 warnings in 21.25s ======================= $ ruff check headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py headroom/proxy/server.py tests/test_kompress_failsafe.py tests/test_proxy/test_kompress_background_warmup.py tests/test_content_router_kompress_not_ready_log.py tests/test_kompress_download_backoff.py All checks passed! $ ruff format --check (same 7 files) 7 files already formatted $ mypy headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py headroom/proxy/server.py Found 9 errors in 2 files (checked 3 source files) # All 9 are in headroom/relevance/embedding.py and headroom/providers/model_metadata.py, # which mypy reaches through imports. origin/main ( |
||
|
|
ca6ba3486c |
fix(ccr): stream the buffered CCR upstream request so idle-killing networks cannot cut it (#4206)
## Description The buffered CCR path flips a streaming Claude Code turn to `stream: false` upstream so the proxy holds the whole reply before serving `headroom_retrieve`. With `stream: false`, Anthropic sends no byte until the generation is complete. A middlebox that drops HTTP connections idle for ~60s (TLS-inspecting antivirus, VPN, corporate proxy; TCP keepalive does not reset an HTTP-layer idle timer) closes the connection first: each attempt fails at ~60s with `Server disconnected without sending a response`, the client gets a 502 after ~186s, and Claude Code resends the same body forever. The CCR continuation that serves the retrieval (`anthropic_ccr_continuation`, its own `http_client.post`) is a second `stream: false` call and is cut the same way. Without Headroom the client streams and the same turns complete. This PR keeps the buffered design and makes only its upstream leg stream. The buffered request and every continuation built from it (CCR, memory, turn hook) go to first-party Anthropic as `stream: true`, and the event stream is folded back into the Message JSON the buffered path already reads, so retrieval, memory, turn hooks, usage accounting and SSE resynthesis see exactly what they saw before. Bytes (deltas and pings) now flow for the whole generation, so an idle timer never fires. The fold reuses `AnthropicSSEEnvelope`, the reconstruction #3130 and #3266 already use on this path. Closes #4205 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/anthropic.py`: new `_send_buffered_as_stream(send, headers, body, *, url, request_id)`. It sends `{**body, "stream": true}` with `Accept: text/event-stream` (one Accept value, any casing replaced) and folds the reply: - not a 200 event stream (an HTTP error, or an upstream that answered JSON anyway): returned unchanged; - a whole stream: a 200 JSON response holding the folded Message; - an in-band `error` event: the event as the body, with its own HTTP status (529 `overloaded_error`, 429 `rate_limit_error`, other documented types to their status, else 500); - a stream that ended before `message_stop` or with a block open: the existing strict 502 `upstream_protocol_error` of `_adapt_event_stream_to_json` (#3266); - a complete stream holding a known event the fold cannot read (an unknown delta type): logged, and the original `stream: false` body is sent instead. One more generation, but never a 502 the client would resend forever. A connection dropped mid-stream raises inside `send`, exactly as for a `stream: false` reply. An in-band error arrives after upstream already answered 200, so `_retry_request` does not retry it; the client gets the status and retries as it would on a direct stream. - It is used at the four places Headroom sends a body it made non-streaming: the buffered CCR request (only when `buffered_stream_ccr`; a client's own `stream: false` request still goes through `_retry_request` with its original bytes), the CCR continuation (the `prepare_outbound_body_bytes` + `log_outbound_request` + `http_client.post` sequence moved into a local `_post_continuation`, unchanged otherwise, so a fallback attempt logs its own outbound line), the memory continuation and the turn-hook continuation. - Gates: the body is not already streaming, the upstream is first-party Anthropic (`is_custom_anthropic_base_url(url)` is false; the host-equality check tool-search gating already uses), and the kill switch `HEADROOM_BUFFERED_UPSTREAM_STREAM` is not `0`/`false`/`no`/`off`. Copilot and other gateways keep the `stream: false` request: the fold is checked against Anthropic's own stream, and a gateway dialect it cannot read would cost every turn a second generation. - `headroom/proxy/runtime_env.py`: `HEADROOM_BUFFERED_UPSTREAM_STREAM` is a registered knob in `RUNTIME_ENV_KNOBS`, read per request through `runtime_env.getenv`, so a reusing `headroom wrap` pushes it to the running proxy and `POST /admin/runtime-env` sets it with no restart. Listed in the `security-model.mdx` allowlist. - `headroom/proxy/anthropic_wire.py`: `fold_anthropic_stream(raw)` parses with `AnthropicSSEEnvelope.parse` and trims each block to the non-streaming Message shape: a null member leaves every block, `thinking` keeps `type`/`thinking`/`signature`, `redacted_thinking` keeps `type`/`data`. Claude Opus 5.5 streams a thinking `content_block_start` with `estimated_tokens: null`; the CCR continuation resends the folded turn as history, and Anthropic answers any block carrying that member with 400 `Extra inputs are not permitted`. - `headroom/proxy/anthropic_wire.py` `_response_from_events`: `message_delta.usage` is merged only where non-null, as the Anthropic SDK's `MessageStream` does. Before, a null count there overwrote the real `message_start` count (input and cache tokens) in the reconstructed message. - `headroom/proxy/buffered_ccr_response.py`: the status-to-error-type table moves out of `_anthropic_error_type` into `ANTHROPIC_ERROR_TYPES` so the handler can invert it; `_anthropic_error_type` returns the same values. - Unchanged on purpose: `body["stream"] = False` and its mutation reason `ccr_streaming_retrieve_buffered_non_stream` stay the buffered path's internal representation (every later step, the response-cache guard and the #3587 log counts key on them), and the `using buffered stream:false upstream request` log line keeps its text. - Tests: new `tests/test_buffered_ccr_upstream_stream.py` (16 tests): the fold equals the Anthropic Python SDK's `accumulate_event` result on a stream with thinking, signature, redacted thinking, text, citations, split `input_json_delta`, pings and a null usage count; stream-only members are dropped; truncated, in-band error and unknown-delta outcomes; the send helper's gates (kill switch, gateway URL, already streaming), passthrough of non-stream replies, 529 for an in-band overload, 502 for a truncated stream, the `stream: false` fallback; and through the handler, the buffered turn and its CCR continuation both go upstream `stream: true` with `Accept: text/event-stream` and the continuation's resent thinking block carries only its schema, while a client's own `stream: false` request goes out `stream: false` with `Accept: application/json` and its original bytes. - Existing tests that pinned `stream: false` on the wire for a buffered turn now pin `stream: true` (`test_anthropic_streaming_ccr_retrieve.py`, `test_ccr_buffered_stream_signed_thinking.py`, `test_proxy_response_cache_replay.py`, `test_no_ccr_disables_response_handling.py`). `test_anthropic_buffered_ccr_usage.py` now answers the buffered turn with an event stream, so usage booking is checked through the fold. `test_buffered_ccr_accept_header.py` (#3078) pins the `stream: false` leg it is about with the kill switch; custom upstreams such as Copilot always take that leg. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - ran on the 3 changed source files only, see below - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_buffered_ccr_upstream_stream.py -q 16 passed, 1 warning in 2.89s # every test file that touches the buffered CCR path, the CCR continuation or SSE reconstruction $ pytest tests/test_buffered_ccr_upstream_stream.py tests/test_sse_safeguards_passthrough.py \ tests/test_buffered_ccr_salvage.py tests/test_anthropic_buffered_ccr_usage.py \ tests/test_buffered_ccr_grace_window.py tests/test_anthropic_buffered_sse.py \ tests/test_ccr_buffered_stream_signed_thinking.py tests/test_toin_feedback.py \ tests/test_proxy_response_cache_replay.py tests/test_proxy/test_anthropic_buffered_timeout.py \ tests/test_anthropic_auto_mode_passthrough.py tests/test_proxy/test_openai_responses_ccr.py \ tests/test_sse_thinking_blocks.py tests/test_streaming_sse_malformed_events.py \ tests/test_route_advice.py tests/test_memory_tool_stream.py \ tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy_byte_faithful_forwarding.py \ tests/test_buffered_ccr_accept_header.py tests/test_anthropic_buffered_memory_continuation.py \ tests/test_anthropic_ccr_workspace_unbound.py tests/test_no_ccr_disables_response_handling.py \ tests/test_proxy_ccr.py tests/test_ccr_inline_resolve_handlers.py \ tests/test_proxy/test_anthropic_ccr_raise.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py \ tests/test_nonstream_sse_policy.py tests/test_ccr_response_handler_extra.py \ tests/test_compress_route_tokenizer_by_model.py -q 855 passed, 6 warnings in 57.94s # the updated assertions fail on main's source (they pin stream:true on the wire) $ git checkout upstream/main -- headroom/ && pytest tests/test_anthropic_buffered_ccr_usage.py \ tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_ccr_buffered_stream_signed_thinking.py \ tests/test_no_ccr_disables_response_handling.py tests/test_proxy_response_cache_replay.py -q --tb=no 12 failed, 33 passed, 1 warning in 13.64s # whole suite $ pytest tests -q 18 failed, 16435 passed, 583 skipped, 119 warnings in 1226.31s (0:20:26) # 16 of the 18 fail the same way on unmodified main in the same environment # (tests/test_proxy_uds.py x11, tests/test_cli/test_wrap_persistent.py x4, # tests/test_pricing_from_litellm.py groq/llama-guard-3-8b x1); the other 2 # (test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request, # test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children) pass when run alone on this branch. $ ruff check . All checks passed! $ ruff format --check . 1790 files already formatted $ mypy headroom/proxy/anthropic_wire.py headroom/proxy/buffered_ccr_response.py headroom/proxy/handlers/anthropic.py Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS 27.0.1 arm64, Python 3.11.13. Upstream `main` |
||
|
|
22a1296cfa |
fix(copilot): send VS Code next-edit requests to the completions host (#4188)
## Description
VS Code Copilot Chat serves ghost text and next edits by posting
`copilot-nes-*` models (seen live: `copilot-nes-pandia-25`, Copilot Chat
0.69) to
`{github.copilot.advanced.debug.overrideProxyUrl}/chat/completions`.
`headroom wrap vscode` points that override and the CAPI override at the
same proxy, so these requests land on the OpenAI chat handler and were
forwarded to CAPI. CAPI answers `400 model_not_supported` for them on
both the per-plan host and the generic host; only the Copilot
completions host serves them. Result: behind Headroom, VS Code shows no
ghost text and no next-edit suggestions, with nothing in the UI saying
why.
Found during the live Copilot Business end-to-end run for #4155 / #4154;
it is on `main` and independent of both. #4144 does not cover it (it
handles the legacy `/v1/engines/*/completions` path only).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/copilot_auth.py`: `copilot_nes_base_url(model,
upstream_base_url)` returns the completions host for a `copilot-nes-*`
request aimed at a public CAPI host, via the existing
`copilot_completions_base_url()` resolver (operator
`GITHUB_COPILOT_PROXY_URL` > advertised `endpoints.proxy` >
`copilot-proxy.githubusercontent.com`), the same resolver legacy
`/v1/engines` completions routing uses. It returns `None` for any other
model, for an upstream that already is a completions host, and for GHE /
custom deployments, whose traffic must stay inside them.
- `headroom/proxy/handlers/openai.py`: the chat-completions handler
applies it right after resolving the upstream, unless the caller chose
one with `x-headroom-base-url` (same rule as
`select_passthrough_base_url`). URL, auth, cache key, and both buffered
and streaming arms use the rerouted base.
- `tests/test_proxy/test_copilot_nes_routing.py`: through `create_app`
at the send seams: NES goes to the completions host with minted Copilot
credentials (buffered and streaming); follows an advertised
`endpoints.proxy`; chat models stay on the plan host; a GHE tenant keeps
its NES traffic; a completions-host target is not moved; a caller-chosen
base URL is left alone.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_proxy/test_copilot_nes_routing.py
7 passed
# on main (fix reverted): the two NES routing tests fail with
# assert 'https://api.business.githubcopilot.com/chat/completions' == 'https://copilot-proxy.githubusercontent.com/chat/completions'
$ pytest -q tests/test_proxy/ tests/test_copilot_vscode_completions_routing.py tests/test_copilot_auth.py \
tests/test_proxy_copilot_auth_hooks.py tests/test_provider_proxy_routes.py tests/test_copilot_native_mode.py \
tests/test_copilot_subscription_smoke.py tests/test_cli/test_wrap_vscode.py
698 passed
$ uvx ruff@0.16.8 check <touched files> && uvx ruff@0.16.8 format --check <touched files>
All checks passed! / 3 files already formatted
$ mypy headroom/copilot_auth.py headroom/proxy/handlers/openai.py --ignore-missing-imports
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, native core, live GitHub Copilot
Business seat (2026-10-13), VS Code with Copilot Chat 0.69, isolated
`HEADROOM_WORKSPACE_DIR`, branch merged with #4155 and #4154 on current
`main`.
- Exact command / steps: `headroom copilot-auth login`; `headroom wrap
vscode --configure` (dedicated proxy on the plan host
`api.business.githubcopilot.com`); in VS Code, type in a Python function
body and wait for ghost text. Then the same `copilot-nes-pandia-25`
request sent directly to each host with the exchanged Copilot token.
- Observed result: without the fix, the proxy forwarded NES to
`https://api.business.githubcopilot.com/chat/completions` → `400
{"error":{"message":"The requested model is not
supported.","code":"model_not_supported"}}` (3 of 3), no ghost text.
Direct probe: `api.business.githubcopilot.com` 400,
`api.githubcopilot.com` 400, `proxy.business.githubcopilot.com` 200,
`copilot-proxy.githubusercontent.com` 200. With the fix, NES went to
`https://copilot-proxy.githubusercontent.com/chat/completions` → 200 (4
of 4), ghost text appeared in VS Code and was accepted; chat, Claude and
Agent-mode turns stayed on the plan host.
- Not tested: an Enterprise-plan seat (same code path); a GHE
(`*.ghe.com`) tenant live (covered by unit test).
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: `copilot-nes-*` chat requests aimed
at a public CAPI host now go to the Copilot completions host instead of
failing with 400.
- Kill switch / disable path: `GITHUB_COPILOT_PROXY_URL` pins the
completions host; revert the commit.
- Unsafe override required: no.
- Qualification impact: Copilot routing and proxy suites.
- Rollback path: revert.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
- Docs: no doc change; `docs/content/docs/vscode-copilot.mdx` does not
describe NES routing.
- Follow-ups, not in this PR: (1) in the `wrap vscode` lane the
dedicated proxy does no token exchange of its own, so no
`endpoints.proxy` is recorded and NES uses
`copilot-proxy.githubusercontent.com` rather than the plan's
`proxy.business.githubcopilot.com`; GitHub's subscription-routing
guidance only blocks `*.individual.githubcopilot.com`, so this should
pass a plan-routed firewall, but wrap could pin the advertised proxy
host into the dedicated proxy as it does the API host in #4155. (2) NES
requests now run through the chat pipeline (compression, memory
injection when `--memory` is on, turn hooks); next-edit prompts are
latency-sensitive and memory context is probably unwanted there.
- Independent review (code-reviewer subagent) found no blockers; its
three SHOULDs (GHE tenant leak, caller-chosen base URL override,
completions-host target moved) are fixed and each has a test.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4188"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
72302a3998 |
fix(proxy): say why proxy.log is missing and why a knob or extension was ignored (#4088)
## Description Several failures an operator needs to see are swallowed silently or logged only at DEBUG. This PR makes each of them visible with one warning. It does not change defaults or control flow. 1. **Missing runtime log.** `_setup_file_logging` caught `OSError` with `pass`. On a read-only workspace, a wrong owner, or a file where the logs directory belongs, `proxy.log` simply never appeared and nothing said why. It now logs `Headroom runtime log disabled: cannot write the log file: <OSError>. Point HEADROOM_WORKSPACE_DIR at a writable directory to restore proxy.log.` The OSError text names the path. The symlink case just above it already warned this way. 2. **Six import-time numeric knobs.** Each one fell back to its default on an unparseable value, without a word: - `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` - `HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS` - `HEADROOM_EAGER_PRELOAD_TIMEOUT_SECONDS` - `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS` - `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS` - `HEADROOM_COMPRESSION_CACHE_MAX_ENTRIES` Each now logs `Ignoring invalid NAME='value'; using DEFAULT` once, at import. That matches the idiom in `hard_watchdog.py`. 3. **Extension discovery failure.** When entry-point enumeration fails, every proxy extension is dropped, Enterprise plugins included. That was logged at DEBUG. It is now a warning: `proxy extensions: entry-point enumeration failed (<describe_exception>), so no proxy extensions will be installed`. `describe_exception` gives the exception types and code locations, never the message, and no traceback is logged at any level. It runs at startup only: once in `install_all`, and once more for the CLI banner. 4. **Pricing lookups in `cost.py`.** `_get_list_price`, `_get_output_price`, `_get_cache_prices` and the cache-rate lookup in the savings summary returned `None` on any exception. A pricing bug therefore showed up as a $0 cost with no trace. All four, plus the existing `estimate_cost` path, go through one helper, `_warn_pricing_once(lookup, what, model, exc)`. Example line: `Failed to get pricing for model 'gpt-4o': RuntimeError at headroom/proxy/cost.py:... in estimate_cost`. It logs the lookup name, `safe_id(model)` and `describe_exception(exc)` (types and code locations, never the message), at WARNING only. No level carries the raw exception or a traceback, because a pricing backend error can quote provider responses or credentials. Dedupe is by `(lookup, model)` through the shared `WarnOnce(1024, "unpriceable models")`. Each pair warns at most once an hour (the `WarnOnce` window). Past 1024 models, one overflow notice is logged, and a key is not used up while WARNING is disabled. 5. **Malformed `HEADROOM_OPENAI_TOOL_SEARCH_MODELS` regex.** The override was ignored on every request with no log. It now warns once per bad pattern, and still falls back to the version gate as before. A valid pattern re-arms the warning, so bad A, then valid B, then bad A again warns twice. None of the new lines logs header values, request content or credentials. They log only knob names, their raw values (operator-set env), model names, regex patterns, exception type names, the `re.error` text, and `OSError` text naming a local path. No exception message or traceback from pricing or extension discovery reaches any log level. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: - add `_warn_invalid_env_number(name, default)` and call it from the six import-time `except ValueError` branches; - log the `OSError` in `_setup_file_logging`; - add a once-per-pattern warning for a malformed `HEADROOM_OPENAI_TOOL_SEARCH_MODELS`. - `headroom/proxy/extensions.py`: entry-point enumeration failure logs at warning with `describe_exception(exc)` instead of at debug. - `headroom/proxy/cost.py`: - the four silent pricing `except Exception` branches call `_warn_pricing_once`; - `_warn_pricing_once(lookup, what, model, exc)` logs `safe_id(model)` plus `describe_exception(exc)` at WARNING, with nothing at DEBUG. It dedupes through `log_safety.WarnOnce`. - `tests/test_cost_pricing_warning_dedup.py`: the tests reset with a fresh `WarnOnce`, and match the quoted model id (`for model 'glm-5.2':`). - Regression tests. Each one attaches a handler to the module logger directly, because caplog misses `headroom.*` records after proxy startup. Each positive test fails on main and is paired with a negative case that passes on both. - `tests/test_cache_breakpoint_diagnostics.py`: - `test_runtime_log_says_why_it_is_missing` makes the logs directory a regular file. It asserts one warning that names the path and `HEADROOM_WORKSPACE_DIR`. - Negative: `test_runtime_log_stays_quiet_when_writable`. - `tests/test_openai_tool_search_deferral.py`: - `test_malformed_override_warns_once_not_per_request` calls the gate 5 times with a bad pattern. It asserts exactly one warning and that the version-gate result is unchanged. - Negative: a valid override is silent (same test). - `test_valid_override_rearms_the_malformed_warning`: bad, valid, bad gives two warnings. - `tests/test_proxy/test_compression_timeout_config.py`: - `test_bad_env_says_it_was_ignored` reloads `helpers` with `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=not-a-number` and asserts the exact warning. - Negative: `test_valid_env_is_silent`. The existing fixture reloads `helpers` afterwards. - `tests/test_proxy_extensions.py`: - `test_discover_warns_when_entry_point_enumeration_fails` makes `importlib.metadata.entry_points` raise. It asserts one WARNING that says no extensions will be installed and names `RuntimeError` but not the exception text. - Negative: `test_discover_is_quiet_when_nothing_is_installed`. - `tests/test_cost_cache_prices.py`: - `test_cache_rate_failure_warns_once_per_model` makes `resolve_rates` raise. It asserts two `_get_cache_prices` calls give one warning naming the model and `RuntimeError`, without the exception text. - Negative: a resolvable model is silent (same test). - `test_pricing_error_never_logs_its_text_at_any_level` runs the real `CostTracker.estimate_cost('gpt-4o', 100, 10)` with a pricing error whose text holds an Authorization credential canary, with `headroom.proxy` at DEBUG. It formats every captured record with `logging.Formatter` (which renders `exc_info`), asserts the canary is absent and the result is `None`, and asserts the warning is present. On the previous commit it fails: the formatted DEBUG traceback contains the canary. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_cost_cache_prices.py -k never_logs # previous commit |
||
|
|
01d9194854 |
fix(ccr): serve headroom_retrieve when the model calls it alongside client tools (#4034)
## Description When the model calls `headroom_retrieve` in parallel with client tools, `CCRResponseHandler.handle_response` hands the whole turn back for the client to resolve (#839). The client has no `headroom_retrieve`, so it can never resolve that call: Claude Code answers `Error: No such tool available: headroom_retrieve`, and the model never gets the content it asked for. Parallel calls are the normal shape for agentic clients. A Claude Code subagent that greps and reads several files in one turn hits this regularly, and users report the error as "happens sometimes, usually with subagents". Measured on one machine's Claude Code transcripts (`~/.claude/projects`): 501 leaked `headroom_retrieve` calls since 2026-09-21. 484 of them were mixed turns (retrieve plus at least one client tool in the same assistant message), and 435 of those were in subagents. The other 17 were lone retrieves on the #3071 streaming path, which this PR does not touch. The client tools in a mixed turn have not run yet. This PR drops them from the turn, serves the retrieval in the continuation exactly as for a lone call, and lets the model decide again there, with the retrieved content in hand. The continuation is what the client receives, whatever it does: re-issue the client calls, change them, or withdraw them because the retrieved content says not to. The client never sees `headroom_retrieve` and never runs a call the model withdrew. Only when the continuation request fails does the client get the model's own mixed turn back (the #839 behaviour), since the model made no newer decision; a continuation that succeeded is kept even if a later retrieval round fails. A `headroom_retrieve` without a valid hash is never treated as a client call: such a turn is handed back unchanged. This applies to Anthropic and OpenAI chat only. In the Responses API a `function_call` can be the item a `reasoning` item requires next, and a Gemini `functionCall` part can carry the turn's thought signature, so dropping a sibling there could make the continuation invalid. Those providers keep the #839 pass-through unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `ccr/tool_calls.py`: new `drop_tool_calls(response, provider, calls)`. It returns a shallow copy of the response without the given tool-call objects, matched by identity. It handles Anthropic and OpenAI chat and returns the response unchanged for any other provider. - `ccr/response_handler.py`: on a mixed turn, `handle_response` trims the client calls and continues the normal retrieval loop. A successful continuation is returned as is. If the continuation request for a mixed round fails, it returns the original mixed turn instead of the trimmed one. When the provider is not trimmable, or one of the extra calls is a `headroom_retrieve` without a valid hash, it keeps the old skip-and-return. The model's own response object is not mutated. - Tests: the #839 handler test now asserts the new behaviour for Anthropic and OpenAI chat: one continuation carrying only the retrieve pair, and the re-issuing continuation returned. New handler tests return the continuation when it withdraws the client call in text, changes its arguments, or re-issues it unchanged; return the original turn when the continuation request fails; and keep a successful continuation when a later retrieval round fails. A malformed `headroom_retrieve` keeps the turn unchanged with no continuation. The Responses pass-through is pinned as unchanged. On the streaming SSE path, a re-issued client tool streams without `headroom_retrieve`, a continuation that withdraws the client tool streams the continuation, and a failed continuation streams both original blocks as a 200. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/ -q -k "ccr or CCR or retrieve or mixed" 1168 passed, 24 skipped, 15064 deselected, 3 warnings in 178.17s $ uv run --frozen --extra dev pytest <every test file naming headroom_retrieve, CCR_TOOL_NAME or ccr_response_handler> # at |
||
|
|
159daa25a0 |
fix(proxy/bedrock): record the real upstream status on InvokeModel outcomes (#4025)
## Description `handle_bedrock_invoke` (the Bedrock `InvokeModel` REST passthrough handler) builds a `RequestOutcome` without ever passing `status_code`, so every outcome it records keeps the dataclass default of `200` regardless of what the Bedrock upstream actually returned. A throttled (429), unauthorized (403), bad-request (400), or 5xx response from the configured `--bedrock-api-url` gateway is forwarded to the client with its real status, but is recorded in Headroom's own savings/cost/request-log funnel as a successful `200` request -- so `emit_request_outcome`'s `status_code >= 400` short-circuit (added in #2682 / #3010) never fires for this path, and a rejected Bedrock turn still feeds `metrics.record_request`, the cost tracker, and the PERF log exactly like a served one. This is a different gap from the one #2682 documented and deliberately left alone: that PR's scope note says `_stream_response_bedrock` (the SDK/boto3-based direct-to-AWS streaming path in `streaming.py`) "never holds an HTTP response object" so it can't pass a real status. `handle_bedrock_invoke` is a separate code path -- the REST gateway passthrough described in `bedrock.py`'s own module docstring -- and it does forward via raw `httpx`, so `response.status_code` (the real upstream status, already used to build the `StreamingResponse`/`JSONResponse` returned to the client) was sitting right there and simply never got threaded into the outcome. Every other `RequestOutcome` call site in the raw-httpx handlers (`anthropic.py`, `batch.py`, `gemini.py`, most of `openai.py`) passes `status_code=response.status_code` (or equivalent); `bedrock.py` was the one exception. Closes: none filed -- found by direct source review, not from an existing issue. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/bedrock.py`: pass `status_code=response.status_code` when constructing the `RequestOutcome` in `handle_bedrock_invoke`, with a comment distinguishing this path (real httpx response available) from the SDK-based streaming path that genuinely cannot. - `tests/test_proxy/test_bedrock_passthrough.py`: added `test_outcome_carries_real_upstream_status_on_error`, parametrized over `[400, 403, 429, 500, 503]`, asserting the recorded `RequestOutcome.status_code` matches the real upstream status rather than the `200` default. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) -- see "Not tested" below - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_bedrock_passthrough.py -q collected 23 items tests\test_proxy\test_bedrock_passthrough.py ....................... [100%] 23 passed in 23.15s $ uv run ruff check headroom/proxy/handlers/bedrock.py tests/test_proxy/test_bedrock_passthrough.py All checks passed! $ uv run ruff format --check headroom/proxy/handlers/bedrock.py tests/test_proxy/test_bedrock_passthrough.py 2 files already formatted ``` RED proof -- new test fails against the pre-fix handler (stashed only `bedrock.py`, kept the new test): ```text $ uv run pytest tests/test_proxy/test_bedrock_passthrough.py::test_outcome_carries_real_upstream_status_on_error -q ... assert outcome.status_code == upstream_status E AssertionError: assert 200 == 400 E + where 200 = RequestOutcome(request_id='hr_...', provider='bedrock', ...).status_code 7 failed (one per parametrized status), 0 passed ``` GREEN -- same command with the fix restored: all 5 parametrized cases (400/403/429/500/503) pass, full file 23/23 passed. ## Real Behavior Proof - Environment: Windows 11, Python 3.12.4, headroom `main` @ `3c418f6f`, `uv run` with the `proxy` extra installed. - Exact command / steps: A standalone script (not committed, not a unit test) started a real `http.server.ThreadingHTTPServer` bound to `127.0.0.1:<ephemeral-port>` -- an actual TCP socket, a genuine stand-in for a Bedrock re-signing gateway -- that answers `POST /model/.../invoke` with a crafted upstream `429 ThrottlingException`. `ProxyConfig.bedrock_api_url` was pointed at that server; nothing else was touched. The script then booted the real FastAPI app via `TestClient`, which runs the real ASGI lifespan, so `proxy.http_client` is the genuine, **unmodified** `httpx.AsyncClient` created by `proxy.startup()` -- it was never replaced or mocked, unlike the unit test and the original proof script. It POSTed a real request through the live route to `/model/{model_id}/invoke`, so the request traveled over an actual socket through the real `_forward_bedrock`/httpx code path to the real local server and back. `proxy._record_request_outcome` was wrapped only to capture the `RequestOutcome` object for printing -- it still called through to the real implementation, so the real funnel (`emit_request_outcome`, Prometheus counters, the proxy's own end-of-run session-summary log) executed for real. Ran once with `headroom/proxy/handlers/bedrock.py` checked out from `main` (pre-fix) and once with the fix restored (post-fix); the fix was never left in the reverted state (verified with `git status` / `git diff` after restoring). - Observed result: Pre-fix: `HTTP response returned to client: 429` / `RequestOutcome.status_code recorded for metrics/savings funnel: 200` / `Would emit_request_outcome short-circuit (status_code >= 400)? False`. The proxy's own session-summary log for that run printed `Total requests: 1`, `Rate limited: 0`, `Failed: 0` -- the throttled request was silently counted as an ordinary served request. Post-fix, identical request/server: `HTTP response returned to client: 429` / `RequestOutcome.status_code recorded for metrics/savings funnel: 429` / `Would emit_request_outcome short-circuit (status_code >= 400)? True`, and the session-summary log changed to `Total requests: 0`, `Rate limited: 1`, ` upstream 429s: 1`. In both runs the client-facing response was correctly 429 -- only the internal accounting silently diverged from it pre-fix, which is exactly the bug this PR fixes. - Not tested: `mypy headroom` -- this Windows dev box's `uv sync --extra dev` fails building the `hnswlib` native extension (missing MSVC toolchain, unrelated to this change); only the `proxy` extra was synced. No live AWS Bedrock or a real LiteLLM/LocalStack gateway was exercised -- the upstream in this proof is a local `http.server` instance standing in for that gateway. To be explicit about what is and isn't real here: the real parts are the TCP socket, the HTTP round trip, headroom's own `httpx.AsyncClient`, and the full `handle_bedrock_invoke`/`_forward_bedrock`/`emit_request_outcome` code path, all unmodified and unmocked; the one stand-in is the endpoint itself (a local server playing the role of the AWS/gateway upstream, not AWS or a real gateway). The streaming route (`invoke-with-response-stream`) was not re-run under this real-socket proof; it shares the identical `_forward_bedrock` call and outcome-recording code with `invoke`, and is already covered by the existing mocked `test_invoke_with_response_stream_is_byte_faithful` test. ## Runtime Rollout Safety - Rollout-managed feature(s): none -- Bedrock passthrough is already gated behind `--bedrock-api-url`/`BEDROCK_TARGET_API_URL` being set; this change doesn't add or touch a rollout flag. - Minimum rollout channel: N/A, ships on stable with no gate. - Stable/default behavior changed: yes, but only for already-failing requests -- a Bedrock `InvokeModel` call that upstream rejects (4xx/5xx) will now correctly show up under `requests_rate_limited`/`requests_failed` instead of inflating `requests_total`/`tokens_saved_total`/cost. Successful (2xx) requests are unaffected (`status_code=200` either way). - Kill switch / disable path: none added; revert is a one-line removal of the `status_code=response.status_code` kwarg. - Unsafe override required: no. - Qualification impact: none. - Rollback path: `git revert` this commit; no persisted state to migrate. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Found by direct source review of `headroom/proxy/handlers/bedrock.py` while auditing provider-handler divergence (comparing how each provider path populates `RequestOutcome`), not from the issue queue. Cross-checked against #2682 and #3010 (which both touch `RequestOutcome.status_code` plumbing on other paths) to confirm this specific gap -- the REST gateway passthrough, as opposed to the SDK-based streaming path #2682 explicitly left alone -- was not already covered. **2026-10-08 update:** addressed Devin's review comment (mocked-upstream proof) and the governance bot's template-field check: - The `Real Behavior Proof` section above now documents a genuine local-socket proof: a real `http.server` TCP listener stands in for the Bedrock gateway, and headroom's own `httpx.AsyncClient` / `handle_bedrock_invoke` / `_forward_bedrock` / `emit_request_outcome` code is exercised completely unmocked against it. The existing mocked unit test (`test_outcome_carries_real_upstream_status_on_error`) is unchanged and still serves as the fast, deterministic correctness check; the new evidence is additional runtime proof, not a replacement. - `Real Behavior Proof` and `Runtime Rollout Safety` now use the exact plain-text field labels (`Environment:`, `Exact command / steps:`, `Observed result:`, `Not tested:`) the governance script matches on, instead of the differently-worded bold labels the previous revision used. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4025" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
1d4b357ca7 |
feat(opencode): support the OpenCode 2.x plugin API (#3999)
## Description Takeover of #3719 by @laurence-diack-pk, who has been unavailable since Sep 23 and whose branch was left conflicted with `main`. The original commit is preserved and @laurence-diack-pk remains its author; this branch adds follow-up commits addressing @JerrettDavis's review and the findings from live OpenCode 2.x testing. It makes the OpenCode plugin and `headroom wrap opencode` work on both OpenCode 1.x and 2.x. On 2.x the 1.x function plugin was silently dropped (the loader requires a `{ id, setup }` default export loaded from a directory), and Effect's `FetchHttpClient` keeps its own `fetch` reference from before plugins load — so `wrap opencode` loaded no plugin and routed nothing. Closes #3669. Supersedes #3719. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) ## Changes Made Carried from #3719: - One plugin for both versions: default export `{ id: "headroom", server, setup }`; 1.x calls `server`, 2.x calls `setup`. - 2.x `setup`: registers `headroom_retrieve` via `ctx.tool.transform` (`codemode: false`), sets `HEADROOM_*` shell env through the `create.before` hook, and returns a cleanup that releases the refcounted transport so hot reloads do not stack interceptors. - 2.x model routing: `ctx.model.transform` points remote OpenAI-wire models at the proxy with `x-headroom-base-url` / `x-headroom-original-path` headers, since 2.x keeps its own `fetch` reference and the transport patch alone misses model traffic. - Wheel ships the bundle as `_dist/index.js`; `headroom_opencode_plugin_path()` returns the directory; `wrap opencode` adds `--standalone` on 2.x so the private server sees `OPENCODE_CONFIG_CONTENT`. Follow-up commits (review from #3719 + live testing): - Rebased on current `main` (0.40.0). Fixed the routing-contract conflict: replaced `routesThroughProxy` (TS2554 against `shouldRoute`'s 3-arg signature) with `modelBaseRoutesThroughProxy`, a deliberate model-base eligibility check that keeps the shared protocol / loopback / proxy-origin / excluded-host contract but drops the endpoint-suffix requirement, since 2.x hands us model base URLs (e.g. `/v1`), not inference endpoints. - `setup` now resolves the effective `excludeHosts` once (option over `HEADROOM_OPENCODE_EXCLUDE_HOSTS`) and applies it to both the transport install and the model rewrite, with new tests for both paths. - Fixed `test_build_launch_env_with_project` to decode `OPENCODE_CONFIG_CONTENT` instead of substring-matching the raw path, which JSON backslash escaping makes impossible on Windows. - `opencode_major_version` now reads the `v2` channel marker in dev builds (`opencode v0.0.0-v2-<stamp>`), which previously parsed as major 0, so `wrap` skips `--standalone` and the private server never loads Headroom's config. - Rebuilt `_dist/index.js` and `hook-shim/handler.js`; drift check clean. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .` — run on the changed paths) - [x] Type checking passes (`mypy headroom` — run on the changed paths) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # plugins/opencode $ npm ci && npm run typecheck && npm run build:standalone && npm test tsc: clean (no TS2554) vitest: Test Files 3 passed (3) | Tests 45 passed (45) $ cmp dist-standalone/index.js ../../headroom/providers/opencode/_dist/index.js # identical $ cmp dist-standalone/hook-shim/handler.js ../../headroom/providers/opencode/hook-shim/handler.js # identical # focused Python suites (same set as #3719, plus the 2.x dev-build version case) $ pytest tests/test_providers_opencode_plugin_path.py tests/test_providers_opencode_config.py \ tests/test_providers_opencode_install.py tests/test_cli/test_wrap_opencode.py \ tests/test_mcp_registry_opencode.py tests/test_learn/test_opencode_scanner.py -q 193 passed in 1.10s $ ruff check / ruff format --check on changed paths: clean $ mypy headroom/providers/opencode headroom/cli/wrap.py: Success: no issues found ``` ## Real Behavior Proof - Environment: macOS arm64, Node v26.10.0, Python 3.13, headroom from this branch (0.41.0-dev). Tested against two real OpenCode 2.x clients, each in throwaway XDG dirs with its own proxy port: - `opencode 2.0.19` (official 2.x binary); - OpenCode 2.x head built from `anomalyco/opencode@v2` (`0.0.0-v2-202610061051`, Bun 1.4.2, `bun run script/build.ts --single` in `packages/cli`). - Exact command / steps: throwaway XDG dirs and a dedicated proxy port; fake key first (routing proof), then the real key (end-to-end proof): ```bash # fake key first (routing proof), then the real key (end-to-end proof) OPENCODE_API_KEY=<key> headroom wrap opencode --port 8799 --no-serena -- \ run -m opencode-go/deepseek-v4.1-flash "reply with the single word ok" # exclusion check HEADROOM_OPENCODE_EXCLUDE_HOSTS=opencode.ai OPENCODE_API_KEY=<key> headroom wrap opencode ... ``` (`opencode-go` declares `OPENCODE_API_KEY` as its env credential in the catalog, so no interactive login is needed.) - Observed result: routed 401 with the fake key and 200 with the real key through the proxy, `--standalone` added automatically, exclusion honored — details: - **Fake key** — proxy log: `proxy_inbound_request POST /v1/chat/completions` with `user-agent: opencode/latest/2.0.19/cli` (and `opencode/v2/0.0.0-v2-202610061051/cli` for the head), headers `x-headroom-base-url: https://opencode.ai` and `x-headroom-original-path: /zen/go/v1/chat/completions`; forwarded upstream, `401` from the fake key — routed through Headroom. - **Real key** — the model answered `ok`; proxy `status=200` with upstream `https://opencode.ai/inference/go/openai/v1/chat/completions`. - `wrap` added `--standalone` automatically on 2.0.19, and again on the v2 head after the dev-build version fix (explicit `--standalone` also works, per the kill-switch path). - **Exclusion** — with `HEADROOM_OPENCODE_EXCLUDE_HOSTS=opencode.ai` the model still answered while the proxy saw **zero** new chat requests: the 2.x model transform honors the exclusion list. - Not tested: Windows; OpenAI "responses"-wire against a live 2.x server; Anthropic-wire custom providers on 2.x; loading the plugin by npm name (`headroom-opencode`), which needs a published release; OpenCode 1.x older than 1.2.20. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: on OpenCode 2.x, `wrap opencode` launches with `--standalone` (a private server per session); on 1.x behaviour is unchanged apart from the plugin path now being a directory. - Kill switch / disable path: pass `--server <url>` or `--standalone` explicitly; `HEADROOM_OPENCODE_PLUGIN_PATH` still overrides the plugin location. - Unsafe override required: no - Qualification impact: none - Rollback path: revert this PR; the wheel bundle and plugin path revert with it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes @laurence-diack-pk — thank you for the original implementation; the commit is preserved verbatim at the base of this branch. @JerrettDavis — this addresses the routing-contract conflict you described in #3719 (the missing `excludeHosts` argument and the model-base vs endpoint mismatch), the Windows assertion, and a dev-build version-detection gap found while validating on macOS. Live 2.x validation is done on both 2.0.19 and today's `v2` head; details above. --------- Co-authored-by: laurence <laurence.diack@peekrobotics.co.nz> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
9553d4292f |
fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description Session cleanup currently drops prefix trackers after the default 600-second session lifetime even when Anthropic's request selects a one-hour cache TTL. Losing the tracker also loses the frozen forwarded prefix and causes the cache-preserving overlay to replay different bytes. Closes #4017. ## Type of Change - [x] Bug fix - [x] Documentation update ## Changes Made - Resolve the request's existing Anthropic cache TTL before selecting the prefix lineage and pass that TTL through tracker selection/creation. - Retain each lineage for the greater of the configured session floor and its observed/current provider TTL plus the existing 60-second safety margin. - Keep the lifetime per lineage rather than changing global configuration: a five-minute session does not inherit another session's one-hour lifetime. - Preserve normal expiry beyond the provider lifetime and margin. - Honor configured cache TTL before a confirmed cache write; discard expired state before accepting a later request's longer TTL. - Re-register the live lineage family after lookup/cleanup so recreated sessions cannot share a tracker's frozen state across sibling conversations or tool profiles. - Update existing fake callbacks to the changed tracker contract, preserve all consumer byte-stability assertions, and remove one handler-to-store wiring-only test rather than re-pinning its internal identity assertions. - Document local tracker retention separately from provider cache availability. ## Testing - [x] Unit tests pass (`pytest`), including frozen-prefix byte stability and independent TTL expiry. - [x] Changed-file lint, configured pre-commit mypy and manual production timestamp/overlay checks pass. ### Test Output ```text Review correction: 273 passed, 2 dependency warnings in 83.79s {"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true} ``` The mandatory pre-push `make ci-precheck` passed: workspace Rust fmt/clippy/tests, native extension build/import verification, the complete configured Python gate, and Commitlint. The native gate explicitly used the installed CPU `libonnxruntime.so.1.26.0` via `ORT_DYLIB_PATH`, without skipping cached-model assertions. ```text [build_rust_extension] headroom._core build + install + verify: OK ================== 177 passed, 4 skipped, 1 warning in 12.87s ================== ✅ ci-precheck PASSED — safe to push. ``` ```sh python -m pytest -q tests/test_cache/test_prefix_tracker.py python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py ``` **74 passed** in tracker tests; **54 passed** in request/cache tests (one existing Starlette TestClient deprecation warning). Two new regressions fail on baseline because the request TTL is not accepted; a separate production runtime comparison demonstrates the actual pre-fix loss, not just the missing parameter. After rebasing onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together with `tests/test_prefix_tracker_client_rewrote_reply.py`, `tests/test_cache_mode_delta_marker.py`, and `tests/test_content_router_mcp_small_result.py`: **150 passed**. The actual production timestamp/byte-retention smoke was repeated successfully on the rebased code, covering compatibility with main's newly merged client-reply normalization. Review regression and affected HTTP-consumer suite: **273 passed**, including tracker expiry, Anthropic/OpenAI cache stability, byte-faithful forwarding, header isolation, buffered timeouts, CCR injection, and immutable system prompts. All affected two-argument test callbacks now accept the TTL contract. The obsolete walker-call-order test was removed; wire-byte assertions remain. The new configured-TTL/expiry cases fail against the published baseline; the post-cleanup regression also verifies that an unrelated conversation gets a different tracker after recreation. Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and Commitlint hooks passed. Full-tree Ruff is locally blocked by executable filesystem permissions on five unchanged OAuth2 plugin files (tracked 100644). A separate full dev-environment mypy run found an unchanged optional LangChain `truthy-function` error; the configured isolated pre-commit hook passed. No unrelated source/permissions changed. ## Real Behavior Proof - Environment: Linux x86_64, CPython 3.13.14; unchanged `dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real `SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token estimator and `overlay_cached_prefix`, no mocked store or mocked clock. - Exact command / steps: parse `cache_control: {type: ephemeral, ttl: 1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed frozen message with different original/forwarded prefix bytes; age `_last_activity` to `time.time()-660` and make periodic cleanup due; resume the same lineage and compare the tracker, frozen count, and forwarded-prefix bytes. Also age beyond 3901 seconds and compare independent 300/3600-second sessions. - Before: resolved TTL 3600; after 660 seconds, tracker identity changes, frozen count drops **1→0**, and forwarded-prefix bytes differ. - Observed result: after the fix, after 660 seconds the same tracker remains, frozen count stays **1**, and forwarded-prefix bytes are identical. The one-hour session survives independently, the five-minute session expires, and the one-hour lineage still expires beyond TTL plus margin. - Review runtime proof against published head `e1a107df`: actual stores and prefix overlays confirm configured TTL retention before the first cache write, no resurrection of expired state by a later one-hour request, eventual expiry, and separate trackers after the maintainer's 3901-second cleanup/recreation sequence. On that exact baseline the unrelated lookup reuses the wrong tracker, although the overlay guard preserves the particular unrelated input bytes; after the fix both identity isolation and byte preservation hold. Identical messages under different tool profiles also receive a cold, separate tracker after the expired bare-key transition. - Not tested: Anthropic cloud cache availability, billed savings, or an actual one-hour wait. The local production state and byte invariants were exercised by aging actual timestamps, not by a paid upstream request. Preserving a tracker cannot guarantee that the provider retains its cache. ## Runtime Rollout Safety - Rollout-managed feature(s): none; repair of existing TTL-aware cache-prefix behavior. - Minimum rollout channel: stable; no rollout-managed feature is enabled. - Stable/default behavior changed: eligible local prefix trackers survive for their provider cache lifetime. Short-TTL/default sessions remain bounded by their own session floor. - Kill switch / disable path: the existing prefix-freeze disable path remains available; no new flag. - Unsafe override required: no. - Qualification impact: local Anthropic prefix lineage selection/retention, not cache pricing or subscription quota policy. - Rollback path: revert the commit to restore previous retention. No persistent schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Existing cache TTL parser and safety margin reused; cache docs updated. - [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes, and existing request byte-stability behavior. - [x] `CHANGELOG.md` not edited; Conventional Commit title supplied. Remote CI results are not inferred from local verification. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
01185e32d3 |
feat(router): add max_output_tokens condition to ModelRoute (#3898)
## Description `ModelRoute` could bound the estimated input size (`max_input_tokens`) but had no condition on the request's own `max_tokens`. That meant a tiny prompt asking for a long answer (for example, `max_tokens: 4000`) still matched a "trivial turn" downshift rule and was sent to the cheaper model. This PR adds `max_output_tokens`, using the shape proposed in the issue. A rule with it set matches only when the request's `max_tokens` is `<= N`. As with the other conditions, `None` means "ignore this condition", set conditions AND together, and the first matching rule wins. **Open question from the issue, decided here:** a request with no `max_tokens`, or with a non-integer or boolean value, does **not** match. An unbounded response is not a trivial turn, so the rule can never widen to cover unbounded requests. This follows the same fail-closed approach as the existing parser. Closes #2765 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/model_router.py`: new `ModelRoute.max_output_tokens` field. `matches()` and `ModelRouter.select()` take an optional `max_tokens` keyword, which defaults to `None`, so existing callers are unchanged. `max_tokens` is added to the decision reason. `max_output_tokens` is added to the allowed route keys and parsed with the existing `_strict_opt_int`, so a malformed, boolean or negative value skips the route. New `request_max_tokens(body)` helper. - `headroom/proxy/handlers/anthropic.py`: `_maybe_route_model` passes `request_max_tokens(body)` to the router. - `docs/content/docs/configuration.mdx`: new field in the rule table, plus a note on combining it with `max_input_tokens`. - Tests for matching, absent budget, select and reason, env parsing (valid and malformed), the helper, and handler wiring. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`): not run successfully. My local mypy stops on an unrelated numpy stub syntax error in this env - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy/test_model_router.py tests/test_proxy/test_model_router_wiring.py -q 54 passed, 1 warning in 5.22s $ pytest tests/test_proxy -q -k "anthropic or router" 119 passed, 237 deselected, 1 warning in 14.41s $ ruff check headroom/proxy/model_router.py headroom/proxy/handlers/anthropic.py tests/test_proxy/test_model_router*.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, branch head - Exact command / steps: the rule from the issue (`from_models: [claude-opus-4-8]`, `max_input_tokens: 4096`, `max_output_tokens: 512`, `require_no_tools`, `to_model: claude-haiku-4-5`) built via `ModelRouterConfig.from_env`. I then ran `select()` for the issue's "Reply with the single word OK." prompt with three different `max_tokens` values. - Observed result: `max_tokens=4000` stays on `claude-opus-4-8`, `max_tokens=256` routes to `claude-haiku-4-5`, and a request with no `max_tokens` stays on `claude-opus-4-8`: ```text max_tokens=4000: routed_model=claude-opus-4-8 reason=no rule matched max_tokens=256: routed_model=claude-haiku-4-5 reason=matched rule 'trivial-opus->haiku': claude-opus-4-8 -> claude-haiku-4-5 (input_tokens=7, has_tools=False, max_tokens=256) max_tokens=None: routed_model=claude-opus-4-8 reason=no rule matched ``` - Not tested: an end-to-end request through a running proxy against a live upstream. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This only extends the opt-in model router (`HEADROOM_MODEL_ROUTER_ENABLED`). - Minimum rollout channel: N/A - Stable/default behavior changed: No. The router is still off by default, and rules without `max_output_tokens` behave exactly as before. - Kill switch / disable path: unset `HEADROOM_MODEL_ROUTER_ENABLED`, or remove the field from the rule. - Unsafe override required: No - Qualification impact: None - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Routing still only applies on the Anthropic `/v1/messages` path, where `max_tokens` is required. If routing is later extended to the OpenAI paths (#2354), `request_max_tokens` is the single place to add `max_completion_tokens` / `max_output_tokens`. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3898" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
76cceeb487 |
fix(proxy): break the compact-every-turn loop near the real context window (#2942)
## Description Closes #3998 Compression makes the usage reported back to the client reflect the *forwarded* (compressed) request, not the client's raw transcript. Clients whose auto-compaction keys off reported usage (Claude Code) therefore defer compaction — the selling point — but with no floor: the raw transcript can grow until even the compressed request exceeds the model's real context window. From that point every turn fails with a 400 `prompt is too long`, the client force-compacts on each error, and the session degrades into a compact-every-other-prompt loop. This came from a churned-user field report (2026-08-12): "Headroom cut off my context and started compacting every two prompts." A second, related bug: `get_context_limit()` resolves purely from the model id, so a session running with the `context-1m` beta is treated as 5x over budget by compression pressure (`context_pressure` pins at 1.0) and gets maximum-aggression lossy compression from early on — the "cut off my context" half of the report. This PR adds `headroom/proxy/context_guard.py` with three cooperating pieces, all passthrough until the danger zone: 1. `effective_context_limit` — the window the forwarded request is really subject to: the registry limit, raised to 1M when the client sent a `context-1m` beta token, capped by any limit learned from a real `prompt is too long` error. Also used for compression pressure at the `context_limit` call site. 2. `note_prompt_too_long` — parses `prompt is too long: N tokens > M maximum` from upstream 400 bodies (streaming and buffered error paths) and records M keyed by (model, has-1m-beta). This is how a client that believes 1M but whose account is capped at 200k stops looping after exactly one error — and why the learned limit must not leak between beta shapes. 3. `MessageStartGuard` — an SSE transformer applied to the client-bound bytes only: when the `message_start` usage total reaches 90% of the effective limit, it inflates the reported `input_tokens` so the client's gauge reads 95% of the window the client believes in, which trips the client's own graceful compaction *before* the 400 wall. Inert after a single decision (steady-state overhead is one `if`); savings metrics and telemetry parse the original upstream bytes, never the nudged ones. Kill switch: `HEADROOM_CONTEXT_GUARD=0`. Out of scope, deliberately: the buffered (non-streaming) success path is not nudged — the affected clients (Claude Code) always stream, and rewriting buffered bodies would break response byte-fidelity for a path with no known symptom. The batch endpoint's `context_limit` site is also untouched (batch has no interactive compaction loop). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - New `headroom/proxy/context_guard.py`: limit resolution (`believed_context_limit`, `effective_context_limit`), 400-learning (`note_prompt_too_long`), and the `MessageStartGuard` SSE usage rewriter, plus the `HEADROOM_CONTEXT_GUARD` kill switch. - `headroom/proxy/handlers/streaming.py`: build the guard for anthropic streams (inside try/except, passthrough on any setup failure), route yielded chunks through `guard.feed()`, flush held bytes at stream end, and learn from streaming 400 bodies. - `headroom/proxy/handlers/anthropic.py`: compression `context_limit` now uses `effective_context_limit` (1m-beta aware, learned-cap aware); the existing UPSTREAM_ERROR block also feeds `note_prompt_too_long`. - `tests/test_proxy/test_context_guard.py`: 25 tests — unit coverage of limits/learning/rewriter (chunk splits, pings, malformed JSON, buffer cap, never-deflate) and four integration tests driving `_stream_response` with a mocked upstream (nudge on the wire, byte-identical passthrough far from the limit, kill switch, 400 learning). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_proxy/test_context_guard.py -q 25 passed in 0.53s $ uv run --frozen --extra dev pytest tests/test_proxy/ -q 272 passed, 1 warning in 71.42s (0:01:11) $ uv run --frozen --extra dev pytest tests/test_sse_thinking_blocks.py tests/test_h2_stream_reset_retry.py \ tests/test_proxy_streaming_ratelimit_headers.py tests/test_request_outcome.py tests/test_proxy_retry_429.py -q 81 passed in 1.53s $ uv run --frozen --extra dev mypy headroom/proxy/context_guard.py Success: no issues found in 1 source file $ uvx ruff check headroom/proxy/context_guard.py headroom/proxy/handlers/streaming.py \ headroom/proxy/handlers/anthropic.py tests/test_proxy/test_context_guard.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS 15 (Darwin 24.6.0), Python 3.10 project venv via uv, repo at this branch; live runs used a sandboxed proxy instance (this module injected) with Claude Code 2.1.206 as the real client. - Exact command / steps: ran the four integration tests in `TestStreamResponseIntegration`, which drive the real `_stream_response` code path with a mocked upstream httpx response: (a) a `message_start` carrying a 185k-token usage against a 200k model limit (above the 90% trigger, below the 95% target, so the assertion can only pass if the rewrite happened), (b) the same stream at 50k tokens, (c) a 190k stream with `HEADROOM_CONTEXT_GUARD=0`, (d) a 400 body containing `prompt is too long: 213021 tokens > 200000 maximum` with `anthropic-beta: context-1m-2025-08-07`. - Observed result: (a) the client-received `message_start` reports `input_tokens` 185000 inflated to 190000 (the 95% target) while the `message_stop` bytes pass through unmodified; (b) the client-received bytes are byte-identical to the upstream bytes; (c) byte-identical passthrough with the kill switch set; (d) the 400 is forwarded unchanged and `effective_context_limit("claude-sonnet-4-5", 200000, "context-1m-2025-08-07")` returns 200000 afterwards. - Live end-to-end (added after the initial PR): drove a real Claude Code 2.1.206 session through a proxy running this code, with a learned limit seeded to the state a real prompt-too-long 400 produces. Result: the client-visible usage total read exactly 190000 (95% of the believed 200k window) and the interactive TUI showed "100% context used". This run is also why the second commit exists: with only message_start rewritten, the client kept displaying the raw values because it merges the final cumulative-usage message_delta over message_start. Forcing a prompt-too-long 400 on the next turn produced a clean "Compacting conversation..." -> summarization request -> retried turn -> correct response, with no user-facing error. - Re-validated 2026-10-06 at `739bacfa`, merged with upstream/main `855390d6` (0.40.0): the repro in #3998 runs the real app with only the upstream HTTP call mocked. On main a forwarded 185k turn reaches the client as 185000 in both `message_start` and `message_delta`, a `prompt is too long ... > 200000 maximum` 400 is forgotten (the next 185k turn reads 18.5% of the 1M the client believes in), and a `context-1m` request is compressed against 200k (`context_pressure` 1.00 at 400k). On this branch: 190000 (95% of 200k), 950000 (95% of the believed 1M) after the learned limit, and 1M (`context_pressure` 0.40). - Merge note: main's memory tool stream filter (#3947) changed the same `yield chunk` line. The guard now runs last, on the frames the filter releases, including the filter's closing frames and the memory continuation rounds, and the end-of-stream drain moved after the memory handling so the `message_delta` the filter holds back is still nudged. `test_guard_nudges_what_the_memory_filter_releases` covers it (fails with `185000 == 190000` if the drain moves back). `tests/test_proxy/` 432 passed, the memory tool suites 95 passed. - Not tested: an organic (unseeded) 180k+-token conversation against the production API; the buffered non-streaming success path is deliberately not nudged (see Description). Note for reviewers: Claude Code 2.1.206 compacts reactively at prompt-too-long (with a threshold-armed precomputed summary) rather than proactively pre-send, so the nudge's role there is an honest gauge/warnings and threshold arming; older clients and other harnesses that compact proactively off reported usage get the pre-wall compaction directly. ## Runtime Rollout Safety - Rollout-managed feature(s): None. Not in the rollout FEATURES registry; gated by its own env switch, `HEADROOM_CONTEXT_GUARD` (default on). - Minimum rollout channel: N/A (not rollout-managed; applies on every channel) - Stable/default behavior changed: Yes, for Anthropic requests only. Compression pressure uses the effective context window (1M when the client sent the `context-1m` beta, capped by any limit learned from a real `prompt is too long` 400), and a client-bound `message_start` whose usage reaches 90% of that window reports `input_tokens` raised to 95% of it so the client compacts before the wall. Below 90% the client receives byte-identical upstream bytes. Upstream requests and error bodies are never modified. - Kill switch / disable path: `HEADROOM_CONTEXT_GUARD=0` (or `false`). Every call site checks it before using the effective limit or rewriting usage, so with it off the proxy behaves exactly as before this PR (covered by the kill-switch integration test). - Unsafe override required: No - Qualification impact: Adds `tests/test_proxy/test_context_guard.py`; no existing test changed or relaxed. - Rollback path: Set `HEADROOM_CONTEXT_GUARD=0` or revert the PR. Learned limits live in process memory only, so nothing persisted needs cleanup. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from the Conventional Commit PR title (a CI guard enforces this) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2942" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
0ad996a82e |
feat(proxy): route Gemini plugin traffic through native transforms (#2697)
## Description OpenCode-routed Gemini requests now reach the tagged gateway and use the shared transform pipeline used by direct Gemini requests. Unsafe caller-supplied destinations stop with 400 before a handler can connect to them. Previously, the transport preserved `x-headroom-base-url` and the upstream pathname, but route matching sent native `:streamGenerateContent` through a handler that did not accept the tagged base and skipped transforms. `handle_gemini_generate_content` already accepts an `upstream_base_url`, runs the OpenAI pipeline over converted `contents`, and handles streaming by URL shape. `handle_gemini_count_tokens` already accepts one too. This adds an opt-in capability on the route spec, checks the caller-supplied base with the shared upstream guard, and sends both direct and plugin-routed native streaming through the shared transform-capable handler. Native parts, incoming query parameters, and the SSE selector are preserved. Wire formats with no native handler keep the existing custom-base passthrough branch. Loopback and private destinations receive a 400 response unless the operator explicitly allowlists them. Closes #2602 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `ProviderHandlerRoute` gains `supports_custom_base_url`, defaulting to `False` so every existing handler route keeps its current behavior - `GEMINI_HANDLER_ROUTES` opts in all three model endpoints: `:generateContent`, `:streamGenerateContent`, and `:countTokens`. Counting has to follow the same upstream as generating, or the client gets counts from a host that is not serving the request - `_register_provider_handler_route` reads `x-headroom-base-url` and forwards it as `upstream_base_url` for routes that opted in - The route validates each custom base with the shared upstream guard before handler dispatch, rejecting unsafe destinations unless an operator allowlist permits them - `handle_gemini_stream_generate_content` delegates unconditionally to `handle_gemini_generate_content`, which runs the transform pipeline and builds the SSE URL from the request path for both direct and plugin-routed traffic - regressions for direct and plugin production-composed routes, mixed native parts, raw query preservation with one `alt=sse`, and adjacent paths that stay on passthrough ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output `uv run pytest tests/test_proxy/test_gemini_plugin_routed_streaming.py tests/test_provider_proxy_routes.py tests/test_provider_route_specs.py tests/test_tokenizer_count_offload.py -q` ```text 62 passed ``` `uv run ruff check . && uv run ruff format . --check` ```text All checks passed! 1600 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.13, no live Google credentials; production `create_app` routes use fake upstream callbacks - Exact command / steps: `uv run pytest tests/test_proxy/test_gemini_plugin_routed_streaming.py tests/test_provider_proxy_routes.py tests/test_provider_route_specs.py tests/test_tokenizer_count_offload.py -q` - Observed result: direct and plugin-routed `:streamGenerateContent` requests reach the same transform pipeline, preserve native parts and query parameters, add one `alt=sse`, and strip `x-headroom-*` headers. All three native Gemini routes reject loopback when the allowlist is unset without calling their handlers; an explicitly allowlisted fake gateway still reaches the handlers. `:countTokens` follows the tagged upstream, and adjacent custom-base routes remain passthrough. - Not tested: live Google endpoint ## Runtime Rollout Safety - Rollout-managed feature(s): None; upstream validation applies to every opted-in Gemini request - Minimum rollout channel: Standard proxy release containing this change - Stable/default behavior changed: Native Gemini routes reject loopback and private custom bases by default; public gateways and explicitly allowlisted destinations continue to work - Kill switch / disable path: No runtime kill switch; `HEADROOM_ALLOWED_BASE_URLS` allows an operator to approve a private gateway - Unsafe override required: Only when a private deployment gateway must be used - Qualification impact: None - Rollback path: Revert this change; the prior route behavior returns ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; this change has no rendered UI. ## Additional Notes Scope note: this covers Gemini's three model endpoints, including direct and plugin-routed `:streamGenerateContent`. Gemini's other routes still use `generativelanguage.googleapis.com` regardless of the tag. Batch routes need handler support for a custom upstream; model listing and passthrough routes use a different registrar. `cachedContents` stays separate because a tagged client could create a cache entry at Google and reference it from a `:generateContent` request sent to a gateway. A follow-up can cover those routes. The tokenizer and pricing entries the issue also asks for, and adapters for other non-Anthropic/non-OpenAI wire formats, are not here. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2697" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
6fb7cadfe4 |
fix(proxy): key rate limits by peer unless the caller authenticated (#3860)
## Why The proxy's request and token rate limits could be walked around by any caller the proxy cannot authenticate (finding 01-F3). Each provider handler keyed its buckets on a client-supplied header: - **OpenAI:** an HMAC of the full `Authorization` / `api-key` value, else one shared `"default"` bucket. Rotating the header gave a fresh bucket per request, and all callers without a credential shared one bucket. - **Anthropic:** the first 16 characters of the key plus the peer. That is mostly the common `sk-ant-api03-` prefix. - **Gemini:** the first 20 characters of `x-goog-api-key`, with no peer at all. ## What changes - New `headroom/proxy/rate_limit_identity.py` with one rule for the Anthropic, OpenAI and Gemini handlers. `rate_limit_identity()` returns a plain string key: - Every key starts with the peer from `resolve_client_ip`, which honours `X-Forwarded-For` only from `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` peers. IPv6 peers are grouped by `/64`; an IPv4-mapped IPv6 address counts as the IPv4 peer. - **Trusted** requests get one bucket per provider credential: `peer:<addr>|cred:<hmac>`, an HMAC of the whole credential with a process-local key, so bucket names hold no credential material. A request is trusted if it presented a valid `HEADROOM_PROXY_TOKEN` or came directly from loopback. A trusted-gateway peer is trusted to supply the client address, not to vouch that the caller authenticated, so a forwarded request without the token is untrusted and keyed by the forwarded client address. A request from a gateway address that carries no `X-Forwarded-For` is judged as a direct request, so a loopback gateway CIDR does not change how direct localhost callers are keyed. Distinct principals behind one address keep separate limits, as #3364 intended. - **Untrusted** requests (no token, remote caller) are charged per peer (`peer:<addr>`). The credential header is ignored. - The security gate sets `request.state.proxy_authenticated = True` when a caller presents a valid proxy token. - The per-handler key code is removed (`_openai_rate_limit_key` and the Anthropic and Gemini inline keys). `rate_limiter.py` is unchanged, and so are token-limit semantics. ## Behaviour change Requests the proxy cannot authenticate (no `HEADROOM_PROXY_TOKEN`, remote caller) are now limited per client address. Changing the API-key header no longer gives a new allowance. Authenticated and direct-loopback callers keep one allowance per provider credential. Behind a gateway, set `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` so each forwarded client address gets its own limit; to keep one allowance per credential there, callers must also send the proxy token. The limiter is quota smoothing and abuse control for unauthenticated callers. It is not a security boundary for authenticated callers, and this PR does not make it one. ## Tests - `tests/test_proxy/test_rate_limit_identity.py`: untrusted keys ignore the credential; trusted keys are per credential and include the peer; the whole credential is used, not a prefix; bucket names carry no credential material; loopback is trusted; a request relayed by a trusted-gateway peer (including one on loopback) is not trusted without the proxy token; with `127.0.0.0/8` configured as a gateway, direct loopback callers stay per credential, and a relayed request with an unusable `X-Forwarded-For` still counts as relayed; IPv6 `/64` grouping; IPv4-mapped addresses. - `tests/test_proxy_openai_rate_limit_key.py`, end to end through `create_app` with `--rpm 1`: authenticated and loopback callers with distinct keys do not share a bucket; an unauthenticated remote caller who rotates `api-key` or `Authorization`, or drops the credential, gets 429 on the second request; two different unauthenticated peers do not share a bucket; through a trusted gateway (`10.0.0.2`, fixed `X-Forwarded-For: 203.0.113.9`) a caller rotating `api-key` gets 429, distinct forwarded clients do not share, and token-authenticated callers stay per credential; with `127.0.0.0/8` as the gateway CIDR, two direct loopback callers with different keys both pass while a caller relayed through it with a fixed forwarded address gets 429 on the second key. The unauthenticated cases fail on `main`. - Existing rate-limiter and TPM suites pass unchanged. Manual run: `headroom proxy --host 0.0.0.0 --rpm 2` with six requests from a LAN address, each with a different `Authorization` key. On `main` all six reach the upstream. On this branch without a proxy token, the third and later get 429. With the proxy token sent, all six pass, one bucket per credential. Not tested: the Rust `headroom-proxy` binary, which has its own limiter. ## Merge order / conflicts Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 → #3863. #3852 makes the untrusted case rare by refusing token-less non-loopback binds; this PR makes it safe when an operator opts out of that. Textual conflict: #3862 adds an import next to the one this PR adds in `handlers/anthropic.py` and `handlers/openai.py`. Whichever merges second keeps both imports. Revised 2026-10-01 after self-review: `rate_limiter.py` is back to `main`; the separate trusted/untrusted pools, per-owner cap and overflow bucket were removed. Revised 2026-10-02 after review: gateway membership only drops the credential from the key when the gateway actually forwarded a client address, so direct localhost callers keep per-credential buckets under a loopback gateway CIDR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3860" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
cfa479af8e |
fix(proxy): log response_content_length in proxy_inbound_response (#2701)
## Description
The `proxy_inbound_response` log line recorded only `duration_ms` and
`status` — **not the response payload size**. This made it impossible to
correlate slow responses with payload size or diagnose "large response →
high latency" patterns.
Meanwhile, the inbound *request* line already logs `content_length`
(from the HTTP header), and the *outbound* request line already logs
`body_bytes`. The response size was the missing piece.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added `response_content_length` field to the `proxy_inbound_response`
log line in `headroom/proxy/server.py`, reading from
`response.headers.get("content-length", "")`
- For streaming/chunked responses where Content-Length is absent, the
field appears as an empty string — no crash, no omitted field
- Added `tests/test_proxy/test_inbound_response_size.py` with 2 tests
covering both JSON (with Content-Length) and StreamingResponse (without
Content-Length) scenarios
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python3.11 -m pytest test_inbound_response_size.py -v --tb=short
test_inbound_response_size.py::test_json PASSED [ 50%]
test_inbound_response_size.py::test_streaming PASSED [100%]
======================== 2 passed, 2 warnings in 0.61s =========================
```
Manual verification against a live proxy instance:
```text
# JSON response (has Content-Length)
event=proxy_inbound_response id=inbound-1785548469603571000 method=POST path=/v1/messages status=200 duration_ms=0.65 response_content_length=221
# Streaming response (no Content-Length — empty value, no crash)
event=proxy_inbound_response ... response_content_length=
```
Note: Tests were run against the installed headroom package
(site-packages) which includes the compiled Rust extension
`headroom._core`. The source-tree `pyproject.toml` uses a maturin build
backend — running `pytest` from the source tree without `maturin
develop` will fail on `ModuleNotFoundError: No module named
'headroom._core'`. The test logic itself is environment-agnostic.
## Real Behavior Proof
- Environment: macOS 15, Python 3.11.15, headroom-ai 0.32.1 (installed
via pip)
- Exact command / steps: `python3.11 -m pytest
tests/test_proxy/test_inbound_response_size.py -v`
- Observed result: 2 passed, `response_content_length=221` confirmed in
captured log output for JSON responses; empty string for streaming
responses
- Not tested: ruff linting and mypy type checking (not installed in test
environment); e2e against a real upstream provider
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my diff
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — text log format change only.
## Additional Notes
- The fix is intentionally minimal: one new `%s` placeholder + one new
argument to an existing `logger.info()` call. No control flow changes,
no new dependencies.
- The field name `response_content_length` (not `response_body_size`) is
chosen to mirror the existing `content_length` field name used on the
inbound request side, keeping the log schema symmetric.
- `ruff` and `mypy` were not run because they are not installed in the
test environment. The change is a single-string-literal + single-arg
addition to a `logger.info()` call — no new types, no new imports.
## Runtime Rollout Safety
- Rollout-managed feature(s): None — this is a log-field addition, no
feature flag involved.
- Minimum rollout channel: N/A — no gated rollout; ships with the
regular proxy release.
- Stable/default behavior changed: No. The change only appends
`response_content_length` to an existing log line; the log format
remains backwards-compatible and no default behavior is altered.
- Kill switch / disable path: N/A — no new runtime behavior to disable;
removing the field is a one-line revert.
- Unsafe override required: No.
- Qualification impact: Low. Covered by 2 new unit tests (JSON with
Content-Length, StreamingResponse without) plus manual verification
against a live proxy; streaming responses without Content-Length produce
an empty field rather than an error.
- Rollback path: Revert the single commit — the previous log format is
restored immediately, no data migration or config change needed.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
861e94d8c1 |
fix(proxy): keep exception text and the proxy token out of client output (#3851)
## Why
Two leaks of internal detail to clients and logs (findings 01-F4 and
10-F4).
- When an upstream connection failed, several response paths returned
`str(e)` from `httpx`, `ssl` or the OS in 500/502/503 bodies and SSE
`error` events. That text names the resolved upstream host, its address
and the failing library. A client that can steer the upstream
(`x-headroom-base-url`) could learn internal hostnames one error at a
time.
- `extract_tags` copied every `x-headroom-*` header into
`RequestOutcome.tags`, including `x-headroom-proxy-token`, which
gateways send on every turn. The token then appeared in the request log
(`--log-file`), dashboard facets and any export that carries tags.
## What changes
- New module `headroom/proxy/public_errors.py`. `classify_exception()`
maps transport and OS failures to fixed codes: `upstream_unreachable`,
`upstream_timeout`, `upstream_tls_error`, `upstream_protocol_error`;
anything else is `internal_error`. Every client- or model-visible path
fails closed: unrecognised exception text (`RuntimeError`, `ValueError`,
parser, database or library errors) never leaves the proxy. Provider
error text is forwarded only when the exception carries a provider HTTP
error response (`openai.APIStatusError`, which LiteLLM's provider errors
subclass, or `anthropic.APIStatusError`). Helpers build the existing
wire shapes with the same `error.type`, plus `code` and `request_id`.
The full exception stays in the server log under that request id.
- Every site that built a client-visible message from an exception now
uses it: `handlers/streaming.py` (direct, Bedrock and OpenAI-via-backend
SSE), `handlers/openai.py` (backend 500, `/v1/compress` 503, both
passthrough connect paths), `handlers/anthropic.py` (Bedrock backend
500), `handlers/bedrock.py` (connect 502), the memory tool handlers
(tool results returned to the model), and the `litellm` / `anyllm`
backends. The existing corporate-TLS hint is kept.
- `internal_header_policy.py` gains `is_credential_header()` and
`is_credential_tag_key()`. They match whole name components (`token`,
`secret`, `password`, `authorization`, `cookie`, `apikey`, `key`, and
similar), so `X-Headroom-Key` and `X-Headroom-Api-Key` are covered, so
numeric tags such as `tool-search-deferred-tokens` are not caught.
- `extract_tags` skips credential headers.
`savings_attribution.public_tags`, the last step before the request log,
also drops credential-shaped keys.
## Behaviour change
- Transport-layer and internal failures now return a fixed sentence plus
`code` and `request_id`. Clients that parsed the old free-text messages
should read `error.code`. Provider HTTP error responses (and their
status codes) are unchanged. In the LiteLLM / any-llm backends, an
exception that is not a provider response now returns `The proxy could
not complete the request.`; the HTTP status mapping is unchanged.
- Memory tool results for any failure return `{"status": "error",
"error": "<code>", "message": ...}` instead of the exception text.
- `x-headroom-*` headers whose name contains a credential component are
no longer recorded as tags. A benign label that happens to use such a
name should be renamed.
## Tests
- `tests/test_public_errors.py`: classification of transport exceptions
(including TLS wrapped in a connect error), non-transport exceptions
left alone, wire shapes carry `code` and `request_id`; the passthrough,
direct-streaming and `/v1/compress` error paths contain no upstream
detail; memory tool results hide transport and internal text;
`client_message` / `tool_result_error` / the LiteLLM backend return
`internal_error` for `RuntimeError`, `ValueError`, `KeyError` and
`JSONDecodeError` carrying a path, host and token (10 tests in this file
fail without the fail-closed change), while an
`openai.AuthenticationError` keeps its text; `extract_tags` and
`public_tags` never carry credentials. Six of these fail on `main`.
- Existing tests that asserted the old exception text in client output
now assert the new contract: the client gets the fixed `code` and
`request_id`, the exception text is absent, and the `headroom.proxy` log
has the full text under the same `request_id`
(`test_litellm_transport_errors.py`, `test_openai_backend_path.py`,
`test_proxy_compress_endpoint.py`,
`test_proxy_streaming_ratelimit_headers.py`).
- Manual run with every upstream connect failing: streaming
`/v1/messages`, streaming `/v1/chat/completions` and passthrough
`/v1/models` return `code: upstream_unreachable` with no address. A
request with `x-headroom-proxy-token` and `x-headroom-license-token`
leaves neither value in the `--log-file` output.
Not tested live: the Bedrock and LiteLLM / any-llm backend paths (unit
tests only) and OTEL/Langfuse export (tags are filtered before they
reach it).
## Open review points
- Resolved: unclassified exception text (JerrettDavis, 2026-09-29) now
fails closed in `client_message` and `tool_result_error`.
- Resolved: `X-Headroom-Key` is matched (`key` is a credential
component; the `api`+`key` pair rule is gone). No internal or test tag
key has a `key` component.
- `is_credential_header` special-cases `authorization`, `cookie`,
`api-key` and `x-api-key`, but its only caller passes only
`x-headroom-*` names, so those branches never run.
- `_is_tls_failure` also matches on message text. Provider HTTP error
responses are now checked first in the backends, so this only affects
non-provider exceptions.
- any-llm: only provider errors raised as `openai`/`anthropic`
`APIStatusError` keep their text; any other any-llm exception type
returns `internal_error`.
Revised 2026-10-01 after review: unrecognised exceptions now return a
fixed `internal_error` message and only provider HTTP error responses
keep their text.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
|
58b1454282 |
fix(settings): report live proxy configuration (#3177)
## Description The settings schema re-reads process environment values after the CLI has already resolved the live proxy configuration. Savings-profile defaults seeded later in startup can therefore make `/settings/schema` report values that are not actually active. For example, `--lossless` can produce `ProxyConfig.lossless=True` while the schema reports the seeded `HEADROOM_LOSSLESS=0`. This change makes the live `ProxyConfig` the highest-precedence settings source, preserves provenance for profile-generated environment defaults, and adds the missing `HEADROOM_MODE` registry field. Closes #3090 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `HEADROOM_MODE` with canonical `token` and `cache` choices. - Made live `ProxyConfig` values take precedence in `/settings/schema`. - Added `runtime_override` metadata for live values that differ from lower tiers. - Prevented savings-profile defaults from being labeled as explicit environment overrides. - Preserved provenance through multi-worker configuration serialization. - Added regression tests for precedence, provenance, endpoint reporting, and worker round-tripping. ## Testing - [ ] Unit tests pass (`pytest`) — focused settings tests pass; full suite not run - [x] Linting passes (`ruff check .`) — changed files checked - [ ] Type checking passes (`mypy headroom`) — not run - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ focused settings-store test run 36 passed, 1 deselected in 0.09s $ direct savings-profile provenance check seed provenance check passed (18 generated keys) $ ruff check <8 changed files> All checks passed! $ ruff format --check <8 changed files> 8 files already formatted $ python -m compileall -q <8 changed files> # No output $ git diff --check # No output ``` ## Real Behavior Proof - Environment: Linux x86_64, Python 3.12.13, source based on upstream `5e0ce242` (v0.36.2). - Exact command / steps: Set profile-seeded `HEADROOM_MODE=cache` and `HEADROOM_LOSSLESS=0`, then generated the schema with live values `mode=token` and `lossless=True`, excluding those generated keys from explicit-environment provenance. - Observed result: The schema reports `mode=token` and `lossless=true`; both generated variables report `env_override=false`, and both differences report `runtime_override=true`. - Not tested: Complete repository and FastAPI endpoint suites. This environment lacks the compiled `headroom._core` extension and Rust toolchain, and dependency downloads are unavailable. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A. - Stable/default behavior changed: Proxy compression behavior is unchanged; only settings registry/schema reporting changes. - Kill switch / disable path: N/A. - Unsafe override required: No. - Qualification impact: The dashboard receives the actual live proxy values. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A; no user documentation change required - [ ] My changes generate no new warnings — focused checks pass; full suite not run - [x] I have added tests that prove my fix is effective - [ ] New and existing unit tests pass locally — focused tests pass; full suite not run - [x] I did **not** edit `CHANGELOG.md` — generated by release-please ## Screenshots (if applicable) N/A — settings API behavior only. ## Additional Notes This is intentionally narrow for #3090. PR #2346 overlaps by adding `HEADROOM_MODE`, but currently includes unrelated dashboard/runtime work; this PR isolates the live-config precedence and provenance correction. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
ef1c528e92 |
fix(grok): route CLI traffic to api.x.ai on shared proxies (#2693)
## Description `headroom wrap grok` points the Grok CLI at the local proxy via `GROK_MODELS_BASE_URL` and, when it starts the proxy, sets `openai_api_url=https://api.x.ai`. When Claude/Codex already own port 8787, wrap reuses that process without restart (other wrappers attached), so process-wide `OPENAI_API_URL` stays `https://api.openai.com`. Grok CLI cannot stamp `x-headroom-base-url` (no custom attribution headers). Inference then goes to OpenAI with xAI session tokens and returns 401. Observed on the wire: - `x-xai-token-auth: xai-grok-cli` - `user-agent: grok-pager/... grok-shell/...` - outbound `POST https://api.openai.com/v1/chat/completions` status 401 This PR adds per-request Grok CLI detection and routes chat / models / responses to `https://api.x.ai` when those signals are present and the OpenAI target is still its default, without restarting the shared proxy or changing Claude/Codex paths. An operator-configured gateway is never bypassed. Not covered by #2626 (attribution / UA map only when custom base is already `api.x.ai`). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `is_grok_cli_request()` in `providers/grok/runtime.py` using narrow wire signals: exact `x-xai-token-auth: xai-grok-cli` and the `grok-pager/` / `grok-shell/` UA product tokens, compared per whitespace-split token so wrappers like `litellm-grok/1.0` cannot match - Resolve OpenAI chat and responses final upstream as: `x-headroom-base-url` then Grok CLI then process `OPENAI_API_URL` - Route `GET /v1/models` and OpenAI-classified passthrough through the same Grok to xAI base - Add unit tests for detector, resolver priority, and passthrough/models base selection - Add `tests/test_grok_shared_proxy_call_sites.py`, which drives `handle_openai_chat` and `handle_openai_responses` against a stubbed transport and asserts the outbound URL — reverting either call site to the old `custom_only or OPENAI_API_URL` formula turns these red - Gate Grok to xAI routing on the OpenAI target still being its default, via one shared `route_grok_to_xai()` predicate used by both the chat/responses resolver and the models/passthrough path, so an operator-configured gateway is never bypassed ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` / `ruff format`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text # Grok routing: call sites, detector, resolver priority, passthrough/models uv run pytest tests/test_grok_shared_proxy_call_sites.py \ tests/test_provider_grok.py tests/test_provider_proxy_targets.py \ tests/test_proxy/test_openai_upstream_header.py \ tests/test_openai_codex_routing.py -q ============================== 48 passed in 0.46s ============================== uv run ruff check . All checks passed! uv run ruff format --check . 1331 files already formatted uv run mypy headroom --ignore-missing-imports Success: no issues found in 506 source files # Full suite (uv, Python 3.14, tests + scripts/tests) — honesty note ==== 15 failed, 9480 passed, 592 skipped in 553.64s ==== # Replaying those 15 as a set gives an identical 11 failed / 4 passed at the # merge base, at HEAD~1 and at HEAD, so this branch introduces no failure. # Causes are local-env: missing cargo, missing litellm extra, Windows-only # asyncio path, AF_UNIX path length. The 4 test_proxy_health.py kompress tests # pass in isolation at every ref and fail only under the full suite # (pre-existing ordering pollution; the new test file is not the cause -- # it runs clean immediately before test_proxy_health.py). # tests/test_memory_eval.py excluded: it imports litellm at collection time. # Fork CI pytest not yet run (action_required until maintainer approves). ``` ## Real Behavior Proof - Environment: macOS arm64; Grok CLI 0.2.117; PR branch `fix/grok-shared-proxy-xai-routing` (0.33.1-dev) running as the only proxy on 8787 with `openai_api_url=null` and `--backend anthropic` (same shared-proxy shape as Claude/Codex-started 8787). Stock Headroom 0.33.0 was first used to capture the before failure. - Exact command / steps: 1) Before: with stock 0.33.0 on 8787, run `headroom wrap grok` (or attach via `GROK_MODELS_BASE_URL`) and inspect `~/.headroom/logs/proxy.log`. 2) After: replace the process with this branch (`uv run python -m headroom.cli proxy --host 127.0.0.1 --port 8787 --backend anthropic`, no OpenAI target override). 3) `GROK_MODELS_BASE_URL=http://127.0.0.1:8787/v1 grok -p "Reply with exactly one word: pong" --always-approve`. 4) Confirm health still shows `openai_api_url: null` and inspect proxy log outbound lines. - Observed result: Before (stock): Grok chat with `x-xai-token-auth: xai-grok-cli` forwarded to `https://api.openai.com/v1/chat/completions` → 401. After (this branch): same Grok CLI on the same port; `/v1/models` for `user-agent: grok-shell/0.2.117` returned 200; chat outbound was `POST path=https://api.x.ai/v1/chat/completions` with status 200; CLI printed `pong` and exited 0. Codex models list against the same proxy still returned 200. - Not tested: multi-day soak with Claude + Codex + Grok all attached for hours; production PyPI release build; full-suite green under fork CI (local full run had 7 env-only failures unrelated to this change); savings attribution still labels provider as openai without custom base (orthogonal, #2626). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title ## Screenshots (if applicable) N/A — evidence is CLI output and proxy logs (redacted auth). ## Additional Notes - Explicitly out of scope: savings attribution as `provider=xai` without custom base (see #2626), multi-provider matrix for Kimi/Mistral, hard-coded per-agent ports. - ChatGPT Codex auth still wins on `/v1/responses` before this chain runs. - Priority: `x-headroom-base-url` > Grok CLI signals (only while `OPENAI_API_URL` is still the default) > process OpenAI default. The gate matters because an operator who points the proxy at a gateway (LiteLLM, Azure, self-hosted vLLM) chose it for every OpenAI-compatible client: a client User-Agent must not silently bypass it, nor carry that gateway's `OPENAI_TARGET_API_HEADERS` to a different vendor. The shared-proxy case this PR fixes runs with `openai_api_url: null`, so it is unaffected. - Fork PR workflows may show `action_required` until a maintainer approves first-time external CI runs. ## Runtime Rollout Safety - Rollout-managed feature(s): none; routing is selected per request from narrow Grok CLI wire signals. - Minimum rollout channel: normal stable release after focused routing and credential-isolation coverage passes. - Stable/default behavior changed: only Grok-fingerprinted requests on a proxy whose OpenAI target remains the default route to api.x.ai; explicit client base URLs and operator-configured gateways retain priority. - Kill switch / disable path: use an explicit non-default OpenAI gateway/target, or human-revert this routing change. - Unsafe override required: none. - Qualification impact: Grok chat, Responses, models, passthrough, and cross-vendor header-isolation regressions must remain green. - Rollback path: human-revert the PR; no persisted state or migration is involved. --------- Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
|
94bb055810 |
fix(anthropic): preserve failed CCR continuations (#3843)
## Description
Anthropic CCR continuations currently parse non-2xx upstream error
bodies as successful model responses. This change treats every non-2xx
continuation as a failed continuation and forwards the original upstream
response without re-serializing it.
Closes #3836
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Reject non-2xx Anthropic CCR continuation responses before parsing
their bodies.
- Preserve the original response object and raw bytes when a
continuation fails.
- Add regression coverage for 401, 429, and 500 continuation responses.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --no-sync pytest tests/test_proxy/test_anthropic_ccr_raise.py -q
4 passed, 2 warnings in 4.10s
$ uv run --no-sync pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_direct_chat_ccr_resolution.py -q
28 passed, 1 warning in 9.31s
$ uv run --no-sync ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_ccr_raise.py
2 files already formatted
$ uv run --no-sync ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_ccr_raise.py
All checks passed!
```
## Real Behavior Proof
- Environment: Linux, Python 3.11.16, local FastAPI TestClient with a
mocked Anthropic upstream, commit
|
||
|
|
fed7281120 |
fix(proxy): reset the cc-switch upstream when Claude Official is selected (#3166)
## Description
The cc-switch reconciler (#1030) captures a third-party
`ANTHROPIC_BASE_URL` as the proxy's upstream and points Claude Code back
at Headroom. The "Claude Official" branch (cc-switch writes `{"env":
{}}`) returned early without dropping that captured upstream.
`HeadroomProxy.ANTHROPIC_API_URL` is a **process-wide class attr** read
per request, so switching back to Official left the previous provider
live. Any Anthropic client still routed through that proxy - a second
Claude Code instance, the VS Code connector, a desktop app that
re-asserts `ANTHROPIC_BASE_URL` on launch - kept reaching e.g.
`api.deepseek.com`, now with Anthropic OAuth credentials attached.
Related: #1264 (Anthropic-compatible third-party upstream support).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `cc_switch_reconciler.tick()`: before leaving Official direct, reset a
captured third-party upstream back to `default_upstream`.
- Guarded on `self.current_upstream not in (None,
self.default_upstream)`, so nothing captured means nothing reset - an
operator-configured upstream is not clobbered by a passing official
tick.
- `tick()` still returns `False` for the direct-official case:
`settings.json` is not rewritten, only the in-process upstream is
corrected.
- Two regression tests.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_cc_switch_reconciler.py -q
collected 16 items
tests/test_proxy/test_cc_switch_reconciler.py ................ [100%]
16 passed in 0.82s
$ uv run --frozen --extra dev ruff check headroom/proxy/cc_switch_reconciler.py tests/test_proxy/test_cc_switch_reconciler.py
All checks passed!
$ uv run --frozen --extra dev ruff format --check headroom/proxy/cc_switch_reconciler.py tests/test_proxy/test_cc_switch_reconciler.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/cc_switch_reconciler.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 24.6.0, Python 3.10.18, pytest 9.0.3, branch off
`main` @
|
||
|
|
ffc6edb4ba |
fix(proxy): preserve Bedrock body-limit error dialect (#3871)
## Description Bedrock InvokeModel requests that exceed the request-body ceiling were returned with `error.type=invalid_request_error`, even though the Bedrock handler reports the same condition as `request_too_large`. This makes the middleware response inconsistent with the route's wire contract. Closes #3867 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Added explicit dialect handling for `/model/{id}/invoke` and `/model/{id}/invoke-with-response-stream`. - Preserved the existing Anthropic Messages and OpenAI payload formats. - Added regression coverage for both Bedrock routes, the Anthropic route, and the OpenAI fallback, including response-once behavior. ## Testing - [x] Unit tests pass - [x] Linting passes - [x] Type checking passes - [x] New tests added - [ ] Manual testing performed ### Test Output ```text 41 passed: tests/test_proxy/test_bedrock_passthrough.py tests/test_proxy_extension_ordering.py 55 passed, 2 skipped: related body-limit suites ruff check: passed mypy headroom/proxy/request_body_limit.py: passed ``` ## Real Behavior Proof On main, the oversized Bedrock request returned: ```json {"error":{"message":"Request body too large...","type":"invalid_request_error","code":"request_too_large"}} ``` With this change, the same request returns: ```json {"error":{"type":"request_too_large","message":"Request body too large..."}} ``` The request is rejected before any upstream call. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Stable/default behavior changed: only the 413 error payload for Bedrock InvokeModel routes. - Kill switch / disable path: Not applicable. - Unsafe override required: No. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review - [x] I have added tests proving the fix - [x] New and existing tests pass locally - [x] I did not edit `CHANGELOG.md` ## Screenshots (if applicable) Not applicable. |
||
|
|
c946b6b0b0 |
fix(proxy): gate the raw request body, not just its Content-Length header (#3338)
## Description Closes #3326. The four handlers named in the issue (`openai.py`, `batch.py`, `gemini.py`, `anthropic.py`) all gate on the `Content-Length` header before calling `_read_request_json`, but that header is only present when the client sends one. A chunked-transfer-encoding request has no `Content-Length`, so it walks straight past every one of those checks, and `_read_request_body_bytes` then reads the whole body through the unbounded `request.body()`. `_read_request_json` and `_read_request_body_bytes` are the single chokepoint all six call sites route through (confirmed by reading each of the six named lines, plus a repo-wide grep for other `_read_request_json` callers), so one change at that chokepoint covers all of them. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `_read_request_body_bounded` in `headroom/proxy/helpers.py`: reads the body via `Request.stream()` instead of `Request.body()`, refusing the request with `RequestBodyTooLarge` once the running total crosses `MAX_REQUEST_BODY_SIZE`, before the rest of the body arrives. - `_read_request_body_bytes` now reads through it instead of calling `request.body()` directly. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_request_body_decompression_limits.py -q 25 passed, 2 skipped in 4.25s # 2 skipped = brotli not installed in this environment $ pytest tests/ -k "helpers or request_body or decompress or proxy_body or size_limit or content_length" -q 246 passed, 12 skipped, 3 failed in 62.57s # The 3 failures are in tests/test_cli/test_wrap_helpers.py::TestProxyClientRefCounting # (process ref-counting / PID checks) and reproduce identically against # unmodified main in this sandboxed container, unrelated to this change. $ ruff check headroom/proxy/helpers.py tests/test_request_body_decompression_limits.py tests/test_proxy_handler_helpers.py All checks passed! $ ruff format --check headroom/proxy/helpers.py tests/test_request_body_decompression_limits.py tests/test_proxy_handler_helpers.py 3 files already formatted ``` `mypy headroom --ignore-missing-imports` did not run cleanly in this environment: it fails at `numpy/__init__.pyi` with "Type statement is only supported in Python 3.12 and greater" before reaching this file, a pre-existing mismatch between the installed numpy stubs and this repo's pinned `python_version = "3.10"` mypy target, not something this diff introduces. One existing test needed a fixture update: `test_proxy_handler_helpers.py`'s `_VertexGeminiImageRequest` stand-in only implemented `.body()`, and the new code path calls `.stream()` (a real Starlette `Request` always has both). Added a `.stream()` method that yields the same body in one chunk. ## Real Behavior Proof - Environment: Docker `python:3.12-slim`, headroom-ai 0.37.0's published wheel (for the compiled `_core` extension) with this branch's `headroom/proxy/helpers.py` overlaid on top. - Exact command / steps: built a genuine Starlette `Request` over a hand-rolled chunked ASGI `receive()` channel (no `content-length` header, matching what a real chunked-transfer-encoding request looks like at the ASGI layer), and called the actual `_read_request_body_bytes` with `MAX_REQUEST_BODY_SIZE` turned down to 10KB so the demonstration does not need to stream tens of megabytes. - Observed result: against unmodified `main`, a 50,000-byte chunked body with no `Content-Length` header is read in full with no error. Against this branch, the same request raises `RequestBodyTooLarge` before the stream finishes, and a 5,000-byte chunked body under the cap still round-trips byte for byte. - Not tested: brotli-encoded bodies (package not installed in this sandbox, same as the two skipped unit tests above), and the repo's own CI matrix (Rust extension build, HuggingFace model prefetch) end to end. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: no for any request under `MAX_REQUEST_BODY_SIZE` (100MB by default); a chunked request that would previously have been read unbounded is now refused past that size, which is the fix. - Kill switch / disable path: none by design; raising `MAX_REQUEST_BODY_SIZE` widens the ceiling if an operator needs to. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert this commit, one function plus a two-line call-site change in a single file. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes Not in scope: a handful of other call sites in these same handler files (`batch.py`'s and `bedrock.py`'s passthrough forwarders, and two streaming passthrough paths in `openai.py`/`anthropic.py`) also call `request.body()` directly rather than through this chokepoint. None of them are among the six lines the issue names, they exist to forward a request byte-for-byte rather than to parse it as JSON, and fixing them is a larger, separate change than this one. Flagging it here rather than leaving it implicit. --------- Signed-off-by: Amir Fathi <amirfathi.me@gmail.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
4227bd2419 |
fix(proxy): reap wrap-spawned proxies once no wrap clients remain (#3202)
## Description Nothing in Headroom ever stops a wrap-spawned proxy whose wrappers are gone. Two leak paths, both by-products of deliberate hardening without a counterpart: 1. **Unclean wrapper death.** `_start_proxy` detaches the proxy (`start_new_session` on POSIX, breakaway console flags on Windows) on purpose, so a crashed wrapper cannot tree-kill a shared proxy out from under other clients. But when the wrapper dies without running cleanup (SIGKILL, crash, terminal close), its client marker goes stale, `_live_proxy_clients` unlinks it, and the proxy itself is never stopped. 2. **Graceful last-exit leak.** `_make_cleanup` only terminates the proxy when the exiting wrapper *started* it (`proxy_holder[0] is not None`). When the starter exits while other clients remain (correct), every later wrapper exits with no handle, so even a perfectly behaved multi-wrap topology leaks the proxy at the end. Observed in production: three `headroom proxy` processes still alive 21 days after their last session, holding ports and being silently reused by later wraps (which then inherited their stale version and captive routing; see the two companion PRs). The fix: wrap-spawned proxies carry `HEADROOM_WRAP_OWNED=1` and run an orphan-watchdog task that polls the per-port client-marker dir every 30s and gracefully stops the process once no live markers, no active WebSocket sessions, and no served-request activity remain for a grace period (default 900s, floor 60s, `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS`). Standalone `headroom proxy` instances and persistent installs never get the marker env var and are never affected; multi-worker runs are excluded (WS sessions are per-worker, and wrap never requests multiple workers). Closes #3199 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/orphan_watchdog.py` (new): env config (`WRAP_OWNED_ENV`, grace parsing with floor), marker liveness (`pid_alive` + start-time identity, pruning stale markers), and `orphan_watchdog_loop` (idle clock reset by live markers, active WS sessions, or movement of the proxy's served-request counter — the last covers direct-HTTP and SSH-forwarded clients that write no marker). Stop is `signal.raise_signal(SIGTERM)`, which reaches uvicorn's handler on Unix and the emulated handler table on Windows (unlike `os.kill(pid, SIGTERM)`, which is `TerminateProcess` there). - `headroom/proxy/server.py`: lifespan spawns the watchdog when `orphan_watchdog_enabled()` and not a multi-worker run (`HEADROOM_PROXY_CONFIG_JSON` absent), and cancels it in teardown like the other periodic tasks. - `headroom/cli/wrap.py`: `_start_proxy` sets `HEADROOM_WRAP_OWNED=1` in the proxy subprocess env; `_proc_identity`/`_identity_mismatch` moved to `headroom/_subprocess.py` (`proc_identity`/`identity_mismatch`) so proxy code does not import the CLI layer — thin module-level delegates keep the existing monkeypatch-based tests working unchanged. - `tests/test_proxy/test_orphan_watchdog.py` (new, 13 tests): env parsing (default/override/floor/garbage), marker pruning (live, dead PID, recycled PID, garbage JSON, missing dir), loop semantics (exit after grace, hold on live client, hold on active WS session, hold on served-request movement, grace reset on returning client, missing registry counts as idle). - `tests/test_cli/test_wrap_claude_vertex_proxy_env.py`: `_start_proxy` marks the subprocess env wrap-owned. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_orphan_watchdog.py -q 13 passed in 2.77s $ uv run pytest tests/test_proxy tests/test_cli -q 973 passed, 1 skipped in 49.85s # 1 pre-existing unrelated failure # (test_litellm_vertex_backend_path_...) fails # identically on clean upstream/main $ uv run ruff format / ruff check on touched files All checks passed! $ uv run mypy headroom/_subprocess.py headroom/cli/wrap.py headroom/proxy/orphan_watchdog.py Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS 26.5 arm64, Python 3.13, local source build 0.37.0-dev (integration overlay with this PR applied). - Exact command / steps: start wrap-spawned proxies via `wrap._ensure_proxy(...)` (which goes through `_start_proxy` and thus `HEADROOM_WRAP_OWNED=1`), with zero registered clients; wait; inspect `~/.headroom/logs/proxy.log`. - Observed result: two such proxies (plain on 8789 PID 48365, augment-mode on 8791 PID 49729) self-exited at the 900s default grace: `event=proxy_shutdown reason=signal pid=49729` at 09:51:49 against a ~09:36 start, port released afterwards (`lsof` clean). A machine-wide sweep beforehand showed three 21-day-old orphans (PIDs 73974, 63454, 79339); with this build they can no longer accumulate. - Not tested: Windows signal path (reasoned from CPython/uvicorn semantics; unit-injected `stop` callable covers the loop side); the multi-worker exclusion with a real `--workers 2` run (guarded by env presence only); marker pruning under active PID recycling on Linux without psutil (`/proc` fallback is unit-tested via mocks). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: only for wrap-spawned proxies, which now self-exit after 15 clientless minutes. Standalone and persistent proxies are untouched. - Kill switch / disable path: omit `HEADROOM_WRAP_OWNED` (not settable by users of wrap; standalone proxies never have it), or set `HEADROOM_WRAP_ORPHAN_GRACE_SECONDS` very high. - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit; orphaned proxies leak again (status quo ante). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Adversarially reviewed by a second model (glm-5.2, high reasoning); initial verdict needs-work, all accepted findings applied (multi-worker exclusion, Windows-safe stop signal, served-request activity hold for marker-less clients, wider test timing margins) and re-tested green. The reviewer independently verified the boot-race ordering, teardown cancellation, and the kill-vs-self-exit race as already correct. Companion PRs (same incident): `-dev` version-restart fix, routing-config reuse check. |
||
|
|
000fefceaa |
fix(anthropic): reject ping-only SSE from buffered CCR (#3682)
## Description Buffered CCR deliberately converts a client `stream:true` request into an upstream `stream:false` request. The SSE recovery gate used the client-facing `stream` value, however, so an upstream that answered the JSON request with `200 text/event-stream` bypassed SSE-to-JSON adaptation. A body containing only ping keepalives could therefore be relayed as a successful but contentless stream. Fixes #3266 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Treat the upstream contract as non-streaming while a streaming client is in buffered CCR mode. - Let `_adapt_event_stream_to_json()` handle non-compliant upstream SSE replies in that path. - A ping-only SSE body now becomes a deterministic `502 upstream_protocol_error` response instead of an empty `200` stream. - Added a regression test with a `200` SSE response containing only `event: ping`. ## Testing - [x] New tests added for the bug fix - [x] Linting passes (`ruff check`) - [x] Relevant unit tests pass locally ### Test Output ```text # before the fix test_buffered_ccr_rejects_ping_only_sse_as_502 failed: assert 200 == 502 # after the fix $ pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_nonstream_sse_policy.py -q 30 passed $ pytest tests/test_proxy_response_cache_replay.py tests/test_proxy/test_anthropic_buffered_timeout.py -q 18 passed $ ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py All checks passed! $ ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows x64, Python 3.13.13, pytest 9.1.1. - Before: the new regression test simulated a `200 text/event-stream` reply containing only `event: ping`; the buffered CCR path returned HTTP 200 instead of reporting an upstream protocol error. - After: the same reply produces HTTP 502 with JSON error type `upstream_protocol_error`, so the proxy no longer presents a keepalive-only stream as a successful answer. - Not tested here: every upstream provider's actual SSE behavior. The test models the non-compliant ping-only reply described in the linked issue. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: stable. - Stable/default behavior changed: yes, a previously accepted invalid upstream response shell is now reported as HTTP 502. - Kill switch / disable path: not needed; normal JSON and SSE recovery paths are unchanged. - Unsafe override required: no. - Qualification impact: clients receive an explicit recoverable upstream failure instead of waiting on or accepting an empty response. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove my fix is effective - [x] Relevant new and existing unit tests pass locally with my changes - [x] My changes generate no new warnings - [x] I did **not** edit `CHANGELOG.md` ## AI Disclosure This change and its tests were prepared with AI assistance and reviewed by the submitter. |
||
|
|
994a89f169 |
fix(proxy): bound request-log message payloads to the feed window; trim glibc too (#3717)
## Description
A user running an overnight Claude Code session through the proxy on
Linux reported it consuming 100 GB of a 128 GB host (about 3,500
requests at ~92k input tokens each, `--log-messages` on). Two stacked
causes, both fixed here:
1. `RequestLogger` keeps message payloads on all 10,000 entries. With
`--log-messages`, every `RequestLog` in the `MAX_LOG_ENTRIES` deque
holds `request_messages`, `compressed_messages` and `response_content`:
two parsed copies of the whole conversation, ~1.9x their JSON size as
Python objects. The only reader of those payloads is
`/transformations/feed`, which is hard-capped at 100 entries, so the
other 9,900 are retained for nothing and grow linearly for the life of
the process.
2. The periodic allocator trim (#2820, #2879) defaults on only for
macOS. glibc retains freed pages the same way, and per-thread arenas on
a many-core host multiply the high-water mark.
Not changed: `MAX_LOG_ENTRIES` itself (`_build_stats_payload` reads the
light fields of all 10,000), and `get_recent_with_messages(n)` for `n >
100`, which now returns bodies only for the newest 100, matching what
the feed could ever serve.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/request_logger.py`: `MESSAGE_WINDOW = 100`; on every
append, null the three payload fields on the entry that just left the
window. Light fields stay on every entry, so `get_recent` and `/stats`
are unchanged.
- `headroom/proxy/models.py`, `headroom/proxy/server.py`:
`periodic_malloc_trim_enabled` defaults on for `darwin` and `linux` (was
`darwin` only). Platforms without a trim call already disable the task
themselves; `HEADROOM_MALLOC_TRIM=0` still opts out.
- `tests/test_proxy/test_request_logger.py`: window tests (payloads
nulled beyond the newest 100, light fields intact and ordered, feed path
fully populated, short logs untouched).
- `tests/test_malloc_tuning.py`: default-scope test asserts darwin and
linux on, win32 off.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py -q
22 passed in 1.10s
$ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/models.py headroom/proxy/server.py tests/test_proxy/test_request_logger.py tests/test_malloc_tuning.py
All checks passed!
$ uvx ruff format --check <same files>
5 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/models.py
Success: no issues found
```
## Real Behavior Proof
- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, this branch at
|
||
|
|
46ac52d39d |
feat(proxy): let /transformations/feed omit message bodies for number-only pollers (#3672)
## Description `/transformations/feed` always returns `request_messages`, `compressed_messages` and `response_content` for every entry, and `RequestLogger.get_recent_with_messages` builds them with `asdict()`, which deep-copies each transcript first. A poller that only reads the per-request numbers (tokens, model, transforms) still pays for a full copy and a multi-MB JSON serialization on the event loop for each pull. Measured on a desktop client polling every 20 s: a `limit=100` pull is ~44 MB, of which ~400 bytes per entry are read, and three concurrent pulls take `/stats?cached=1` from 45 ms to 1.3 s. This adds `?include_messages=0` (default `1`, byte-identical output to today). When off, the three body fields are omitted from the response and never walked server-side. Closes # ## Type of Change - [x] Performance improvement ## Changes Made - `RequestLogger.get_recent_with_messages(n, include_messages=True)`: when `False`, builds each entry from the dataclass fields minus the three payload fields, deep-copying only what it returns (same idiom as #3613 for `get_recent`). - `/transformations/feed` accepts `include_messages: bool = True` and omits the three body keys when it is off. Everything else in the item, including `log_full_messages`, is unchanged. - Each item also carries `uncached_input_tokens`, `cache_write_tokens` and `cache_read_tokens` (already stored on `RequestLog`, never exposed by the feed). A number-only poller can then rate `tokens_saved` against new input (uncached + cache_write) the way `/stats` `new_input_savings_percent` does; the item's full-transcript `savings_percent` is unchanged. Present in both the full and the slim response. - Tests: logger test proves the payloads are not traversed (a leaf whose `__deepcopy__` raises); endpoint test proves the slim item equals the full item minus the three keys and carries the cache split. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_proxy/test_transformations_feed.py tests/test_proxy/test_request_logger.py -q 9 passed in 4.53s $ uvx ruff check headroom/proxy/request_logger.py headroom/proxy/server.py tests/test_proxy/test_transformations_feed.py tests/test_proxy/test_request_logger.py All checks passed! $ uv run --frozen --extra dev mypy headroom/proxy/request_logger.py headroom/proxy/server.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS 15, Python 3.12, this branch, in-process `RequestLogger` seeded with 100 entries carrying ~160 KB of messages each (the size of a mid-session Claude Code turn). - Exact command / steps: `get_recent_with_messages(100)` and `get_recent_with_messages(100, include_messages=False)`, timed, then `json.dumps` of each result, timed and sized. - Observed result: include_messages=True: copy 29.4 ms, json 56.5 ms, 24.45 MB. include_messages=False: copy 1.3 ms, json 1.9 ms, 0.05 MB. Same 100 entries, same numeric fields. - Not tested: the dashboard UI (it does not pass the parameter, so it gets the default, unchanged response). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a (opt-in query parameter) - Stable/default behavior changed: additive only; without the parameter every existing key is byte-identical, and three integer keys are added to each item - Kill switch / disable path: callers stop passing `include_messages=0` - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit; no persisted state or schema involved ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Documentation: the endpoint has no reference page; the parameter is documented in the handler docstring. - Overlaps #3613 on the `_HEAVY_FIELDS` constant and the import lines; both hunks are identical so either merge order applies cleanly. |
||
|
|
44b8e7ba87 |
fix(proxy): stop get_recent deep-copying dropped message payloads (#3613)
## Description `RequestLogger.get_recent()` built each returned row with `dataclasses.asdict(entry)` and only then dropped `request_messages`, `compressed_messages` and `response_content`. `asdict` deep-copies every field first, so every call walked the full request and compressed transcripts of every entry it returned, for fields it was about to discard. `/stats` calls `get_recent(10_000)` synchronously on the event loop on every build (both the uncached path and the `?cached=1` snapshot rebuild). Measured against 0.37.0 with a 300-message Claude Code transcript shape (~400 KB per entry): 3.4 ms per entry, so a full 10k deque costs ~34 s per `/stats` call, during which the loop serves nothing else. A desktop dashboard polling `/stats` reproduced this in production: 4,769 `/stats` calls in 55 min with a mean latency of 517 s, `/v1/messages` mean 135 s (max 1,839 s), 61k `Exceeded concurrency limit` warnings and a watchdog restart storm (Headroom Desktop user report, 2026-09-16). The fix builds the row from `fields()` while skipping the heavy fields, so the payloads are never touched. Kept fields are scalars plus flat `dict`/`list` containers; those get the same shallow copy `asdict` produced for them, so callers still get copies rather than aliases into the log entry. Fixes the backend half of the desktop report; the desktop side stops polling uncached `/stats` for connector verification separately. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - `headroom/proxy/request_logger.py`: `get_recent` iterates `fields()` and skips `_HEAVY_FIELDS` instead of calling `asdict()` and filtering afterwards. Shallow-copies the kept `dict`/`list` fields. - `tests/test_proxy/test_request_logger.py`: new test plants a leaf whose `__deepcopy__` raises inside both message payloads and asserts `get_recent` still returns the row with copied (not aliased) containers. Fails on the previous implementation, passes on this one. ## Testing - [x] Unit tests pass locally (`uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py`, 5 passed; `pytest tests/test_proxy -k "stats and not history"`, 6 passed) - [x] `uvx ruff check` and `uvx ruff format --check` clean on both files - [x] New test verified to fail on the old code (`git stash` the source change: 1 failed, 4 passed) ### Test Output ``` $ uv run --frozen --extra dev pytest tests/test_proxy/test_request_logger.py -q tests/test_proxy/test_request_logger.py ..... [100%] ============================== 5 passed in 0.23s =============================== $ uvx ruff check headroom/proxy/request_logger.py tests/test_proxy/test_request_logger.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS 15, Python 3.12, headroom-ai 0.37.0 venv (the desktop app's pinned runtime), plus this branch. - Exact command / steps: Built a `RequestLog` with `request_messages`/`compressed_messages` set to a 300-message transcript (3 content blocks each, ~416 KB serialized) and timed `dataclasses.asdict(entry)` in a loop; then ran the new unit test against the old and new `get_recent`. - Observed result: 3.4 ms per `asdict` call on 0.37.0, i.e. ~34 s for the 10,000-entry `get_recent` that `/stats` performs on the event loop. With this change the per-row cost is a `fields()` walk plus shallow copies of two small containers (microseconds), and the new test passes only on the new code. - Not tested: an end-to-end `/stats` latency comparison on a long-lived proxy with a full 10k deque; the production numbers above come from a user's `proxy.log`, not from a controlled run. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: no; `/stats` returns the same rows and keys, they are just built without walking the dropped payloads - Kill switch / disable path: none needed; revert the commit - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes ## Screenshots n/a ## Additional Notes `get_recent_with_messages` keeps `asdict` on purpose: it returns the payloads, so the walk is the point there. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
aceff2ea9d |
fix(proxy): gateway turn fixes found by benchmarking behind a gateway (#3619)
## Description Fixes five defects found by running real Claude Code sessions through a gateway that uses the `/v1/compress` turn contract (#3605). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **Prompt text is never compressed on replaying callers.** The coding profile compresses user messages, and on turn 1 that rewrote a plain agent's task ("Fix this bug in the library source" became "Fix bug in source", and test ids lost their `tests/test_` prefix). When `prefix_replay_guaranteed` is set, `content_router` protects the opening user messages and the newest user message, for string and list content alike. The one exception is a newest user message that answers a fenced shell command in the assistant turn before it: that is a text harness's tool observation and still compresses. Tool results still compress, and cache freeze is unchanged. - **Anthropic's four cache breakpoint limit.** One benchmark request failed with `max 4 cache_control blocks, found 5`. Session turns in the OpenAI handler now re-place markers with `normalize_message_cache_control`. A last-stop guard, `enforce_cache_breakpoint_budget`, runs in the gateway extension for Anthropic bodies: it re-places markers, then drops the oldest extras, and logs `event=cache_breakpoint_budget` when it repairs. - **Failure relay without a body.** `POST /v1/compress/response` rejected an upstream 4xx/5xx without a response body as `missing_response`. A body is now required only on 2xx. - **Usage totals after a re-drive.** The done payload carries `billed_usage` summed across rounds. After one or more re-drives, `response` is never null: it is the hook's replacement or the latest provider response (including when the hook hands back that same object), with `usage` set to the totals in its own key shape (chat or Anthropic). Turns with no re-drive keep `response: null`. Before, the client saw the last round only. - **Cold Kompress stall.** When native Kompress init is deferred at startup, a daemon thread preloads it after a short delay. `HEADROOM_KOMPRESS_WARMUP=0` disables it. It is off under pytest and skipped on glibc < 2.28 (the #1908 segfault hosts). - Docs: `proxy.mdx` and `wiki/proxy.md` describe the 2xx-only body rule and `billed_usage`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) on the changed modules - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_transforms_content_router.py tests/test_compress_session_mode.py \ tests/test_compress_route_tokenizer_by_model.py tests/gateway tests/test_proxy \ tests/test_kompress_preload_deferral.py 672 passed, 4 skipped in 228.49s $ uvx ruff@0.16.3 format --check <changed files> && uvx ruff@0.16.3 check <changed files> 12 files already formatted All checks passed! $ mypy <changed modules> Success: no issues found in 6 source files ``` New tests: 5 prompt-protection tests in `test_transforms_content_router.py` (3 fail without the fix), `tests/gateway/test_cache_breakpoint_budget.py`, failure-relay and billed-usage tests in `test_redrive_loop.py`, `billed_usage` in the response-half contract test, `tests/test_proxy/test_kompress_background_warmup.py`. ## Real Behavior Proof - Environment: headroom proxy on this branch behind a gateway, Claude Code with claude-sonnet-5, real Anthropic calls. - Exact command / steps: 8 smoke sessions (32 calls) on SWE-bench-style tasks, with a spend logger recording every provider round. - Observed result: 0 errors. Task statements reached the provider verbatim. On every re-driven call (2 and 3 rounds) the client's usage equalled the sum of the per-round provider usage. Kompress warmed in the background in about 4 s at startup; in a replay of the slow session the first code-aware turn went from 5.4 s to 1.3 s. - Not tested: glibc < 2.28 hosts (warm-up is skipped there by design). The original body behind the 5-breakpoint error was not captured, so the guard is covered by unit tests rather than a live repro. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: stable. - Stable/default behavior changed: yes. Prompt text on replaying callers is no longer compressed, the done payload adds `billed_usage`, and Kompress preloads in the background when deferred. - Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0` for the warm-up. The other changes are correctness fixes with no switch. - Unsafe override required: no. - Qualification impact: none expected; the breakpoint guard only acts on bodies that would otherwise get a 400. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Open follow-up, not in this PR: steady-state code-aware compression falls back to Kompress at 0.3 to 1.3 s per tool result even when it saves nothing. A gate for that is worth a separate change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
ae75ca49a0 |
feat(proxy/model-router): add require_tools route condition (#2362)
## Description Extends the cost-aware model router (#1706) with a `require_tools` route condition — the inverse of the existing `require_no_tools`. Today an operator can route **tool-free** turns to a cheaper model (`require_no_tools`), but there is no way to express the opposite: keep plain chat on a cheap model while sending **agentic, tool-using** turns to a more capable one. That is a very common cost/quality split (simple Q&A on a small model, tool-loop/agent work on a strong model), and it is exactly symmetric to what the router already supports. ## Fix Add a `require_tools: bool = False` field to `ModelRoute`: - `matches()` gains `if self.require_tools and not has_tools: return False`, so a rule with `require_tools` matches only when the request declares tools. - `_route_from_entry` parses `"require_tools"` with the same strict, fail-open validation as `require_no_tools` (a non-boolean skips just that rule rather than silently widening it), and `require_tools` is added to `_ALLOWED_ROUTE_KEYS` so it is not rejected as an unknown key. - Setting both `require_tools` and `require_no_tools` on one rule is an AND of contradictory conditions, so the rule simply never matches — a harmless operator error, not a crash. Example: ```json [ {"name": "agentic->strong", "require_tools": true, "to_model": "gpt-5.4"}, {"name": "chat->cheap", "require_no_tools": true, "max_input_tokens": 8000, "to_model": "gpt-5.4-mini"} ] ``` ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/model_router.py`: add the `require_tools` field to `ModelRoute`, honor it in `matches()`, allow + strictly parse the key in `_route_from_entry`/`_ALLOWED_ROUTE_KEYS`. - `tests/test_proxy/test_model_router.py`: cover `require_tools` matching, the contradictory both-set case, and env parsing (valid + non-boolean fail-open). - `CHANGELOG.md`: Features entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_proxy/test_model_router.py -q 33 passed in 0.53s $ uvx ruff@0.15.17 check headroom/proxy/model_router.py tests/test_proxy/test_model_router.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/model_router.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. `model_router.py` is a pure, dependency-free component, so its full pytest runs locally (no ML stack) — the 33 tests above include the new ones. - Exact command / steps: `ModelRoute(to_model="strong", require_tools=True).matches(...)` for tool/no-tool requests; a rule with both `require_tools` and `require_no_tools`; and `ModelRouterConfig.from_env(...)` with valid and non-boolean `require_tools`. - Observed result: `require_tools` matches only when `has_tools` is True; the contradictory rule never matches; a valid `require_tools` parses into the route; a non-boolean `require_tools` fails open (the route is skipped). `require_no_tools` behavior is unchanged. - Not tested: a live proxy round trip; the router is wired into the handlers separately, and this change is confined to the pure route-matching/parsing component that those handlers already call. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes `model_router.py` is a pure component with no heavy imports, so unlike most of my recent PRs I was able to run its full unit-test file locally (33 passed). This change is purely additive: existing routes without `require_tools` behave exactly as before. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
427fa76f60 |
fix(savings): count compression savings once per conversation, not once per turn (#3480)
## Description
On the OpenAI `/v1/responses` path, a request's `tokens_saved` is the
running total for the whole **conversation**, not for that turn. Codex
re-sends the entire transcript every turn and the router recompresses
all of it, so the same removed token is booked again on every remaining
turn. Anthropic's frozen cached prefix means its `tokens_saved` is
already novel-only. The two providers have therefore never meant the
same thing by the same field, and everything that sums it inherits the
difference.
Measured across one machine's retained proxy logs on 2026-09-08:
gpt/codex models reported 8.10M tokens saved against 3.90M tokens of new
input (2.1x), where Claude reported 4.18M against 43.3M. Two requests
from the same machine and the same minute:
```text
gpt-6-astra tok_before=156260 tok_after=80240 tok_saved=76020 cache_read=66304
claude-opus-5 tok_before=114024 tok_after=113566 tok_saved=458 cache_read=98503
```
What inherits the inflation: lifetime savings totals,
`by_model.savings_percent`, the dollar figure (repeat removals priced at
the full uncached input rate when the alternative was a cache read at
roughly 0.1x), and any rate dividing savings by new input, which then
has no 100% ceiling.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `headroom/proxy/conversation_savings.py`: a bounded, thread-safe
per-conversation ledger that converts the cumulative-per-turn series
into the incremental one. 512 conversations, LRU; a forgotten
conversation re-counts its transcript once rather than leaking.
- `RequestOutcome` gains `conversation_key` and
`conversation_tokens_saved`, both defaulting to `None`. `None` means
"this path does not distinguish", and the funnel falls back to
`tokens_saved` unchanged.
- `emit_request_outcome` derives the novel figure once and uses it for
the two cumulative consumers (`metrics.record_request`,
`cost_tracker.record_tokens`). The request log and the transformations
feed keep `outcome.tokens_saved`: those tokens really did leave this
request's payload, so per-request surfaces still describe the wire.
- The PERF line gains `tok_novel=` only when it differs from
`tok_saved`.
- The three `/v1/responses` emit sites populate the pair: HTTP
non-streaming, HTTP streaming (via `_stream_response` ->
`_finalize_stream_response` -> `RequestOutcome.from_stream`), and the
WebSocket per-turn and session-residual records. On WS the conversation
total is the last frame's figure, not the session sum, because the
session sum adds cumulative values together.
- The key is derived with the existing `conversation_key_from_body` and
taken **before** compression, so a rewritten first user message cannot
move it mid-conversation. It is left unset when compression did not run,
so a bypassed turn cannot reset the running total.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_proxy/test_conversation_savings.py -q
............... [100%]
15 passed in 0.30s
$ uv run --frozen --extra dev pytest tests/test_request_outcome.py tests/test_openai_codex_ws_lifecycle.py \
tests/test_openai_codex_routing.py tests/test_openai_responses_output_shaper.py \
tests/test_proxy_project_savings.py tests/test_handler_outcome_tag_invariant.py tests/test_proxy/ -q
250 passed, 1 warning in 89.55s
$ uv run --frozen --extra dev pytest tests/ -q
5 failed, 7538 passed, 514 skipped, 6 warnings in 178.47s
$ uv run --frozen --extra dev mypy headroom/proxy/outcome.py headroom/proxy/conversation_savings.py headroom/proxy/handlers/streaming.py
Success: no issues found in 3 source files
$ uvx ruff check headroom/proxy/
All checks passed!
```
The 5 full-suite failures are pre-existing on `origin/main`. Verified by
stashing this branch and re-running the full suite on the base commit,
which fails the same `tests/test_providers/test_deepseek.py` cases
(stale pricing registry) plus the same order-dependent
`tests/test_lossless_mode.py::test_router_lossless_search_no_marker_and_recoverable`.
Both files pass in isolation with this change applied.
## Real Behavior Proof
- Environment: macOS 24.6.0, Python 3.11, `uv run --frozen --extra dev`,
base commit
|
||
|
|
a507249b92 |
fix(proxy): give the upstream write phase its own timeout (#3327)
## Description Partially addresses #3259. `write` inherited `request_timeout_seconds`, so pushing request bytes got the same 300s budget as waiting for a model to answer: ```python httpx.Timeout( connect=config.connect_timeout_seconds, # 10 read=config.request_timeout_seconds, # 300 write=config.request_timeout_seconds, # 300 <- same knob pool=config.connect_timeout_seconds, # 10 ) ``` Sending a request and waiting for a model to think are different operations, and conflating them left the send **effectively unbounded in practice**. When an upstream stops draining — a pooled socket whose peer went away over a laptop sleep, a network change, a NAT timeout — the send blocks until the OS abandons retransmission, ~180-220s on macOS. That is *under* the inherited 300s, so httpx never timed out and the OS ended the request instead. The retry then hit the same wall and stalled again, which is why one incident blocked the reporter's client for 5-10 minutes. `--connect-timeout-seconds` cannot help, exactly as the reporter says: it only guards establishing a *fresh* connection, and a reused one never connects. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `ProxyConfig.write_timeout_seconds` (default **150**), with `--write-timeout-seconds` / `HEADROOM_WRITE_TIMEOUT_SECONDS` and a `SettingField` so it appears in the settings registry. - Used it in `_provider_httpx_client_options` **and** in `_anthropic_buffered_request_timeout` — the buffered path sets its own timeout and is the path most likely to carry a large body, so it needed the split too or it would have kept the unbounded write. Behaviour after: ``` BEFORE: connect=10 read=300 write=300 pool=10 AFTER: connect=10 read=300 write=150 pool=10 ``` ### What `write` actually bounds, and why the default is 150s The first revision of this PR used 60s and claimed httpx applies `write` per write operation, so a large body would be unaffected. **That was wrong, and I measured it before shipping the correction.** httpx hands a bytes body to the transport as a *single* write (`ByteStream.__aiter__` yields the whole body; httpcore's h11 `_send_request_body` wraps each yielded chunk in one `anyio.fail_after(write)`), so on HTTP/1.1 the entire upload runs inside one timer. Against a local peer draining a 16MB body at a steady ~1MB/s: ``` write=3.0: WriteTimeout after 3.0s <- healthy peer, still cut off write=60.0: send completed in 19.1s ``` So this is a bound on total upload time, not a stall detector. 60s would have cut off the 7-15MB bodies the reporter sends over a slow uplink — turning a fix into an outage for exactly the person who filed the issue. 150s is the value that satisfies both real constraints: it carries 15MB (the largest body #3259 reports) over a ~1 Mbps uplink, and still fires well before the ~180s OS retransmit ceiling that made the inherited 300s unreachable. `tests/test_upstream_write_timeout.py` pins both ends of that range so neither can drift. On HTTP/2 the body is split by flow control and the waiting-for-window part is charged to `read` (httpcore `_wait_for_outgoing_flow` → `_receive_events`), so only real socket writes count against the bound there. A genuinely per-chunk bound would separate "dead peer" from "slow peer" perfectly, but it needs the body fed as an iterator without changing the wire framing. That is a larger change than this fix warrants; worth a follow-up if slow-uplink users hit the 150s cap. ### Why the read timeout is deliberately left alone Time-to-first-byte is a legitimately long wait for a model, and the buffered Anthropic path already sets its own 600s read for exactly that reason. Lowering `read` globally would break slow legitimate responses to fix a rare hang — the wrong trade. Operators who want a tighter read bound already have `--request-timeout-seconds`; before this PR that knob also silently shortened the write budget, which is the conflation being removed. ### Failover `_retry_request` catches `httpx.TransportError`, and `WriteTimeout` is a `TimeoutException` under it — so a bounded write drops the bad connection and re-sends on a fresh one instead of stalling behind a socket whose peer is gone. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (CI-pinned `ruff` 0.16.3) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_proxy/ tests/test_proxy_hardening.py tests/test_upstream_write_timeout.py 302 passed in 40.46s $ pytest tests/test_upstream_write_timeout.py tests/test_proxy/test_anthropic_buffered_timeout.py \ tests/test_proxy/test_settings_store.py tests/test_cli_proxy_env.py tests/test_cli_proxy_improvements.py 181 passed, 1 skipped in 11.65s ``` ## Scope This does **not** explain the reporter's follow-up: consistent ~60s stalls on 7-15MB bodies with `HEADROOM_MAX_KEEPALIVE=0`, where there is no pooled socket to inherit. I would not close #3259 on this PR. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |