mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
18
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2f58877732 |
fix(proxy): check x-headroom-base-url off the event loop in chat and responses (#4856)
## Description
`handle_openai_chat` and `handle_openai_responses` validated
`x-headroom-base-url` with the synchronous `is_safe_upstream_url`, which
waits for the DNS lookup (up to `HEADROOM_UPSTREAM_RESOLVE_TIMEOUT_S`, 3
s by default) on the event-loop thread. Chat did this three times per
request and responses twice, so one slow-resolving host held up every
other in-flight request for several seconds. The other routes in
`headroom/providers/proxy_routes.py` already use
`is_safe_upstream_url_async`.
Both handlers now resolve the header once, through a new
`_resolve_openai_upstream_base_async` that uses the async guard (lookup
on a worker thread, pin scope opened on the request's task), and reuse
that result for the destination and the extra-header decision.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
## Changes Made
- `headroom/proxy/handlers/openai.py`: header parsing split into
`_openai_base_url_candidate`; sync `_resolve_openai_upstream_base` (kept
for `_resolve_openai_upstream`) and new
`_resolve_openai_upstream_base_async` share it;
`_openai_upstream_from(request, custom)` picks the upstream from an
already-resolved override; chat and responses call the async resolver
once.
- `tests/test_proxy/test_openai_upstream_guard_off_loop.py`: asserts the
guard runs once and not on the event-loop thread, for chat and
responses.
- `tests/test_proxy_openai_cache_key_integration.py`: its stub now
patches the async resolver the handler calls.
**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-idna-pin-once` adds the identical `_openai_upstream_from`
and resolve-once hunks (this PR is a superset of that part);
`fix/openai-base-url-reject` changes the rejection branch of
`_resolve_openai_upstream_base`, which after this PR must be mirrored in
`_resolve_openai_upstream_base_async`. Suggested order: idna-pin-once,
then this PR, then reject (rebasing reject means raising the 400 from
the shared rejection path in both resolvers).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# New test on origin/main (
|
||
|
|
b1fb162b62 |
fix(grok): attribute api.x.ai chat outcomes as provider xai, recognize grok-shell UA (#2626)
## Description Grok Build chat requests were recorded as `custom` or `openai`, preventing xAI provider rollups. This change records custom-base and directly routed Grok chat traffic as `xai`, recognizes both `grok/` and `grok-shell/` clients, and keeps API-key requests on credential-driven PAYG policy. Closes #3996 ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Attribute supported xAI chat routes, including trailing-dot hosts, to `xai`. - Preserve per-model catalog cache prices and share the xAI cache fallback between dashboard costs and persisted savings. - Exclude inferred cache writes from billed aggregate totals; those tokens already belong to uncached input. Explicit provider-reported writes remain billable. - Cover custom-base outcome integration, catalog pricing precedence, legacy and current client credentials, and inferred writes with and without TTL splits. ## Testing At `29fd4f9d7a24a76f3a3f53edacc5832e357cae58`: 130 focused pricing, accounting, auth, cache, and routing tests pass (one existing Starlette warning). Ruff lint and format, whitespace checks, and Linux-target mypy over 567 source files pass. Main refresh retains `3c418f6f` ancestry. Latest documentation dependency audit found zero vulnerabilities. ## Real Behavior Proof - Environment: Windows, Python 3.12, a real uvicorn proxy and a separate loopback HTTP upstream. Subscription polling disabled. Only upstream network delivery is redirected; this is not live xAI billing traffic. - Steps: send four real TCP chat POSTs with custom-base, direct, legacy Grok, and trailing-dot routing; query the running proxy's `/stats` endpoint. The upstream returns 10,000 input tokens, 5,000 cached input tokens, and 1,000 output tokens per request. A priced xAI catalog fixture has $1/M input and $2/M output, with no explicit cache-read rate. - Observed: all four requests return HTTP 200, outcomes record `provider=xai` and `client=grok_build`, and `/stats` records `{"xai":4}`. Input spend is $0.0232 using the xAI 0.16 cache fallback. Before the aggregate-accounting correction this same replay reported $0.0432. - Evidence: `pr2626-resume15-pricing-runtime.log` and `pr2626-resume15-final-tests.log`; regressions reproduce the pricing/auth defects before correction. ## Runtime Rollout Safety - Rollout-managed features: none; attribution, credential classification, and accounting use the existing request paths. - Default behavior: supported xAI routes receive the xAI label; API keys use PAYG policy, JWT credentials retain OAuth policy. Unknown gateways retain the bounded custom provider bucket. - Pricing behavior: when a priced catalog row has no cache rates and the caller omits `provider`, `resolve_rates` now identifies the provider from `litellm_provider` and uses the shared fallback. For xAI this means cached input costs 0.16 times list input; explicit model cache rates still win. Dashboard aggregates also exclude inferred writes already counted as uncached input. These corrections change reported input costs and persisted savings, not upstream requests or billing. - Rollback: revert the pricing/auth/attribution commits; no new rollout switch or unsafe override is introduced. - Coverage: real HTTP routing and `/stats` replay plus focused default-policy and accounting tests. ## Checklist - [x] Self-review completed - [x] Added regression coverage - [x] Relevant local tests and static checks pass - [x] Did not edit CHANGELOG.md ## Additional Notes Hosted checks and automated review must qualify this newly published head before approval. No PR merge performed. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
9553d4292f |
fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description Session cleanup currently drops prefix trackers after the default 600-second session lifetime even when Anthropic's request selects a one-hour cache TTL. Losing the tracker also loses the frozen forwarded prefix and causes the cache-preserving overlay to replay different bytes. Closes #4017. ## Type of Change - [x] Bug fix - [x] Documentation update ## Changes Made - Resolve the request's existing Anthropic cache TTL before selecting the prefix lineage and pass that TTL through tracker selection/creation. - Retain each lineage for the greater of the configured session floor and its observed/current provider TTL plus the existing 60-second safety margin. - Keep the lifetime per lineage rather than changing global configuration: a five-minute session does not inherit another session's one-hour lifetime. - Preserve normal expiry beyond the provider lifetime and margin. - Honor configured cache TTL before a confirmed cache write; discard expired state before accepting a later request's longer TTL. - Re-register the live lineage family after lookup/cleanup so recreated sessions cannot share a tracker's frozen state across sibling conversations or tool profiles. - Update existing fake callbacks to the changed tracker contract, preserve all consumer byte-stability assertions, and remove one handler-to-store wiring-only test rather than re-pinning its internal identity assertions. - Document local tracker retention separately from provider cache availability. ## Testing - [x] Unit tests pass (`pytest`), including frozen-prefix byte stability and independent TTL expiry. - [x] Changed-file lint, configured pre-commit mypy and manual production timestamp/overlay checks pass. ### Test Output ```text Review correction: 273 passed, 2 dependency warnings in 83.79s {"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true} ``` The mandatory pre-push `make ci-precheck` passed: workspace Rust fmt/clippy/tests, native extension build/import verification, the complete configured Python gate, and Commitlint. The native gate explicitly used the installed CPU `libonnxruntime.so.1.26.0` via `ORT_DYLIB_PATH`, without skipping cached-model assertions. ```text [build_rust_extension] headroom._core build + install + verify: OK ================== 177 passed, 4 skipped, 1 warning in 12.87s ================== ✅ ci-precheck PASSED — safe to push. ``` ```sh python -m pytest -q tests/test_cache/test_prefix_tracker.py python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py ``` **74 passed** in tracker tests; **54 passed** in request/cache tests (one existing Starlette TestClient deprecation warning). Two new regressions fail on baseline because the request TTL is not accepted; a separate production runtime comparison demonstrates the actual pre-fix loss, not just the missing parameter. After rebasing onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together with `tests/test_prefix_tracker_client_rewrote_reply.py`, `tests/test_cache_mode_delta_marker.py`, and `tests/test_content_router_mcp_small_result.py`: **150 passed**. The actual production timestamp/byte-retention smoke was repeated successfully on the rebased code, covering compatibility with main's newly merged client-reply normalization. Review regression and affected HTTP-consumer suite: **273 passed**, including tracker expiry, Anthropic/OpenAI cache stability, byte-faithful forwarding, header isolation, buffered timeouts, CCR injection, and immutable system prompts. All affected two-argument test callbacks now accept the TTL contract. The obsolete walker-call-order test was removed; wire-byte assertions remain. The new configured-TTL/expiry cases fail against the published baseline; the post-cleanup regression also verifies that an unrelated conversation gets a different tracker after recreation. Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and Commitlint hooks passed. Full-tree Ruff is locally blocked by executable filesystem permissions on five unchanged OAuth2 plugin files (tracked 100644). A separate full dev-environment mypy run found an unchanged optional LangChain `truthy-function` error; the configured isolated pre-commit hook passed. No unrelated source/permissions changed. ## Real Behavior Proof - Environment: Linux x86_64, CPython 3.13.14; unchanged `dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real `SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token estimator and `overlay_cached_prefix`, no mocked store or mocked clock. - Exact command / steps: parse `cache_control: {type: ephemeral, ttl: 1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed frozen message with different original/forwarded prefix bytes; age `_last_activity` to `time.time()-660` and make periodic cleanup due; resume the same lineage and compare the tracker, frozen count, and forwarded-prefix bytes. Also age beyond 3901 seconds and compare independent 300/3600-second sessions. - Before: resolved TTL 3600; after 660 seconds, tracker identity changes, frozen count drops **1→0**, and forwarded-prefix bytes differ. - Observed result: after the fix, after 660 seconds the same tracker remains, frozen count stays **1**, and forwarded-prefix bytes are identical. The one-hour session survives independently, the five-minute session expires, and the one-hour lineage still expires beyond TTL plus margin. - Review runtime proof against published head `e1a107df`: actual stores and prefix overlays confirm configured TTL retention before the first cache write, no resurrection of expired state by a later one-hour request, eventual expiry, and separate trackers after the maintainer's 3901-second cleanup/recreation sequence. On that exact baseline the unrelated lookup reuses the wrong tracker, although the overlay guard preserves the particular unrelated input bytes; after the fix both identity isolation and byte preservation hold. Identical messages under different tool profiles also receive a cold, separate tracker after the expired bare-key transition. - Not tested: Anthropic cloud cache availability, billed savings, or an actual one-hour wait. The local production state and byte invariants were exercised by aging actual timestamps, not by a paid upstream request. Preserving a tracker cannot guarantee that the provider retains its cache. ## Runtime Rollout Safety - Rollout-managed feature(s): none; repair of existing TTL-aware cache-prefix behavior. - Minimum rollout channel: stable; no rollout-managed feature is enabled. - Stable/default behavior changed: eligible local prefix trackers survive for their provider cache lifetime. Short-TTL/default sessions remain bounded by their own session floor. - Kill switch / disable path: the existing prefix-freeze disable path remains available; no new flag. - Unsafe override required: no. - Qualification impact: local Anthropic prefix lineage selection/retention, not cache pricing or subscription quota policy. - Rollback path: revert the commit to restore previous retention. No persistent schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Existing cache TTL parser and safety margin reused; cache docs updated. - [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes, and existing request byte-stability behavior. - [x] `CHANGELOG.md` not edited; Conventional Commit title supplied. Remote CI results are not inferred from local verification. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
ee731d7ea8 |
fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why Four related gaps in the proxy's network gate (findings 01-F1, 01-F2, 01-F5, 01-F9): - **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only logged a warning, then served the `/v1/*` relay, with the operator's upstream credentials, to every peer that could reach the port. - **Missing peer address.** `is_loopback_host(None)` returned `True`. A Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset, or a hand-built request therefore turned off the token gate, the `/admin` and `/debug` guards and the WebSocket gate. - **Operator data routes.** `/stats-history` (including CSV export), `/quota`, `/subscription-window` and `/metrics` were served to any network caller. - **WebSocket loopback exemption** checked only the peer IP, not the `Host` header, so it had no DNS-rebinding defence. ## What changes - New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`, `::`, a LAN address or a hostname) with no token is refused at three places: `create_app` (embedders and the multi-worker factory), `run_server` (one exit with code 2 before uvicorn forks) and the `headroom proxy` CLI (a clear error before the banner, exit code 1). An absent host means uvicorn's default `127.0.0.1`. - `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when the runtime already restricts reachability, such as a container published on `127.0.0.1`. The `proxy_open_bind` warning is still logged on that path, and the banner says the bind is open. - `is_loopback_host(None)` now returns `False` (fail closed), matching `identity.py`. - `/stats-history`, `/quota` and `/subscription-window` answer only: a non-loopback caller that passed the token gate, a loopback caller (peer and `Host` header both loopback), a trusted dashboard client, or, under an acknowledged token-less bind, the container's own default-gateway TCP peer with a loopback `Host` (the host browser reaching a `127.0.0.1`-published container). Everyone else gets 404, as with `/admin`. Settings writes keep their stricter rule with no token short-cut. - `/metrics` answers a non-loopback caller that passed the token gate, a loopback caller, a connecting peer in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host` is fine, so Prometheus scrapers work; a cross-origin browser request is still refused. - The token short-cut applies only to non-loopback requests. The gate exempts loopback peers without checking `Host`, so a loopback request has not proven it holds the token and must pass the loopback `Host` check. This blocks a DNS-rebound browser on the operator's machine. - The `/dashboard` shell stays ungated. It is a static template, like the already public `/dashboard/static` assets. Gating it would break the dashboard for containers published on host loopback, where the peer is the bridge gateway and `Host` is `localhost`. The data routes behind it are gated. - The WebSocket loopback exemption requires a loopback peer and a loopback `Host` header. A loopback peer with a foreign `Host` can still connect with the token. - Launchers that always publish on `127.0.0.1` set the acknowledgement themselves: `headroom install` (docker preset), `scripts/install.sh` and `scripts/install.ps1` (through one helper that adds the acknowledgement only together with the `127.0.0.1` publication), and `docker/docker-compose.native.yml`. - Docs: `proxy.mdx` documents the new variable and which routes the token also gates; `docker-install` (docs and wiki) and the `docker-compose.yml` comment show the acknowledgement in the loopback-published example. `e2e/docker-bind-security.sh` sets it and checks that the same launch without it refuses to start. ## Behaviour change (breaking) - `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to start. The error names both remedies. Set a token, bind loopback, or set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already confined. Anyone running the image by hand with `--host 0.0.0.0` and no token must add one of these. - Without a token, `/stats-history`, `/quota`, `/subscription-window` and `/metrics` return 404 to non-loopback callers outside a trusted CIDR. Prometheus scrapers on another host must be in `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token. - Code that relied on a request with no peer address being treated as loopback now gets the non-loopback path. - The default loopback run is unchanged. ## Tests - `tests/test_proxy_bind_policy.py`: the refuse/allow matrix (non-loopback, loopback, unset host, token, acknowledgement values), the warning on the acknowledged path, `create_app`, the `run_server` exit before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env, acknowledged start, offline banner, start with a token). - `tests/test_proxy_loopback_gating.py`: each data route against non-loopback, loopback, trusted-CIDR and token callers; the dashboard shell stays reachable; the host of an acknowledged, loopback-published container reads the data routes while another bridge peer, a forwarded gateway address and a foreign `Host` do not; CSV export is not served to network callers; `/metrics` across loopback, both CIDR lists, out-of-range peers, cross-origin browsers and token callers, including a trusted gateway relaying a scraper outside its own range. A regression test shows that a token on the proxy does not exempt a loopback peer from the `Host` check; it fails without the fix. - `tests/test_proxy_hardening.py`: the HTTP gate with no peer address, WebSocket `Host` cases, `require_loopback` and `/debug` with no peer. - Installer tests: `headroom install` adds the acknowledgement, and the shell and PowerShell wrappers add it only together with the loopback publication. Launcher-level tests start the proxy with the environment and `--host` that `build_runtime_command()` and `docker-compose.native.yml` produce and check that the host browser (bridge-gateway peer) gets `/stats-history` and another container does not. - Tests that bound `0.0.0.0` for unrelated reasons now set a token or the acknowledgement. Tests that read operator routes use an explicit loopback peer. On `main`, 14 of the gating and hardening tests fail and the bind-policy module does not exist. Manual runs: token-less `--host 0.0.0.0` exits with the refusal from both the CLI and `run_server`; with the acknowledgement, the data routes are 404 to a LAN peer and to a loopback peer with a foreign `Host`; with a token, a LAN peer gets 401 without it and 200 with it; the default loopback bind starts as before. Not tested: a real Unix-domain-socket deployment and a real Prometheus scrape (both covered by tests only). ## Merge order Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 → #3863. This PR goes first: it is the gate the others assume, and it is the release-note event. ## Not in this PR - Enforcing the token for loopback callers when one is configured. Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs against the TCP peer, and the operator data routes serve the host of a loopback-published container (bridge-gateway peer under the launchers' bind acknowledgement) so the default Docker dashboard works. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
6fb7cadfe4 |
fix(proxy): key rate limits by peer unless the caller authenticated (#3860)
## Why The proxy's request and token rate limits could be walked around by any caller the proxy cannot authenticate (finding 01-F3). Each provider handler keyed its buckets on a client-supplied header: - **OpenAI:** an HMAC of the full `Authorization` / `api-key` value, else one shared `"default"` bucket. Rotating the header gave a fresh bucket per request, and all callers without a credential shared one bucket. - **Anthropic:** the first 16 characters of the key plus the peer. That is mostly the common `sk-ant-api03-` prefix. - **Gemini:** the first 20 characters of `x-goog-api-key`, with no peer at all. ## What changes - New `headroom/proxy/rate_limit_identity.py` with one rule for the Anthropic, OpenAI and Gemini handlers. `rate_limit_identity()` returns a plain string key: - Every key starts with the peer from `resolve_client_ip`, which honours `X-Forwarded-For` only from `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` peers. IPv6 peers are grouped by `/64`; an IPv4-mapped IPv6 address counts as the IPv4 peer. - **Trusted** requests get one bucket per provider credential: `peer:<addr>|cred:<hmac>`, an HMAC of the whole credential with a process-local key, so bucket names hold no credential material. A request is trusted if it presented a valid `HEADROOM_PROXY_TOKEN` or came directly from loopback. A trusted-gateway peer is trusted to supply the client address, not to vouch that the caller authenticated, so a forwarded request without the token is untrusted and keyed by the forwarded client address. A request from a gateway address that carries no `X-Forwarded-For` is judged as a direct request, so a loopback gateway CIDR does not change how direct localhost callers are keyed. Distinct principals behind one address keep separate limits, as #3364 intended. - **Untrusted** requests (no token, remote caller) are charged per peer (`peer:<addr>`). The credential header is ignored. - The security gate sets `request.state.proxy_authenticated = True` when a caller presents a valid proxy token. - The per-handler key code is removed (`_openai_rate_limit_key` and the Anthropic and Gemini inline keys). `rate_limiter.py` is unchanged, and so are token-limit semantics. ## Behaviour change Requests the proxy cannot authenticate (no `HEADROOM_PROXY_TOKEN`, remote caller) are now limited per client address. Changing the API-key header no longer gives a new allowance. Authenticated and direct-loopback callers keep one allowance per provider credential. Behind a gateway, set `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` so each forwarded client address gets its own limit; to keep one allowance per credential there, callers must also send the proxy token. The limiter is quota smoothing and abuse control for unauthenticated callers. It is not a security boundary for authenticated callers, and this PR does not make it one. ## Tests - `tests/test_proxy/test_rate_limit_identity.py`: untrusted keys ignore the credential; trusted keys are per credential and include the peer; the whole credential is used, not a prefix; bucket names carry no credential material; loopback is trusted; a request relayed by a trusted-gateway peer (including one on loopback) is not trusted without the proxy token; with `127.0.0.0/8` configured as a gateway, direct loopback callers stay per credential, and a relayed request with an unusable `X-Forwarded-For` still counts as relayed; IPv6 `/64` grouping; IPv4-mapped addresses. - `tests/test_proxy_openai_rate_limit_key.py`, end to end through `create_app` with `--rpm 1`: authenticated and loopback callers with distinct keys do not share a bucket; an unauthenticated remote caller who rotates `api-key` or `Authorization`, or drops the credential, gets 429 on the second request; two different unauthenticated peers do not share a bucket; through a trusted gateway (`10.0.0.2`, fixed `X-Forwarded-For: 203.0.113.9`) a caller rotating `api-key` gets 429, distinct forwarded clients do not share, and token-authenticated callers stay per credential; with `127.0.0.0/8` as the gateway CIDR, two direct loopback callers with different keys both pass while a caller relayed through it with a fixed forwarded address gets 429 on the second key. The unauthenticated cases fail on `main`. - Existing rate-limiter and TPM suites pass unchanged. Manual run: `headroom proxy --host 0.0.0.0 --rpm 2` with six requests from a LAN address, each with a different `Authorization` key. On `main` all six reach the upstream. On this branch without a proxy token, the third and later get 429. With the proxy token sent, all six pass, one bucket per credential. Not tested: the Rust `headroom-proxy` binary, which has its own limiter. ## Merge order / conflicts Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 → #3863. #3852 makes the untrusted case rare by refusing token-less non-loopback binds; this PR makes it safe when an operator opts out of that. Textual conflict: #3862 adds an import next to the one this PR adds in `handlers/anthropic.py` and `handlers/openai.py`. Whichever merges second keeps both imports. Revised 2026-10-01 after self-review: `rate_limiter.py` is back to `main`; the separate trusted/untrusted pools, per-owner cap and overflow bucket were removed. Revised 2026-10-02 after review: gateway membership only drops the credential from the key when the gateway actually forwarded a client address, so direct localhost callers keep per-credential buckets under a loopback gateway CIDR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3860" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
7df8bd87b5 |
fix(proxy): label plain OpenAI chat traffic openai, not custom (#3912)
## Description Three tests fail on `main`. Each of the PRs below passed CI on its own branch; the failures come from how they combine. - **Real bug (#3759 + #2693):** #3759 labels the chat outcome provider with `CUSTOM_BASE_PROVIDER if upstream_base_url else "openai"`. That was written against a handler where `upstream_base_url` held the `x-headroom-base-url` override. #2693 had already renamed the override to `custom_upstream_base_url` and made `upstream_base_url` the resolved upstream, which is never empty. Since both merged, **every chat request without a custom base is labelled `custom` instead of `openai`**. That affects `/stats`, `/metrics`, OTel and provider-dependent savings estimates. Not in any release yet. - **Stale test (#3365):** #3365 keys OpenAI rate limits by an HMAC of the credential. The flood test's stub limiter still compared against the raw `Authorization` value, so it never refused and the "limited" requests got 200 instead of 429. The code is right; the test is updated. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: the fallback label checks `custom_upstream_base_url`. - `tests/test_proxy_openai_cache_stability.py`: the stub limiter refuses `_openai_rate_limit_key({"authorization": limited_auth})`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — one-identifier change - [ ] New tests added for new functionality — the existing tests already cover it and fail on `main` - [x] Manual testing performed ### Test Output ```text # on main |
||
|
|
f0ec2bb376 |
fix(telemetry): label custom-base chat upstreams from a fixed provider set (#3759)
## Description
Unrecognized custom-base chat upstreams report
`outcome_provider="openai"`.
Dashboards and per-provider stores merge every OpenAI-compatible vendor
except Zen into `openai`.
### Fix
Labels come from a fixed set in `custom_base_passthrough_telemetry()`:
| Custom base host | Label |
| --- | --- |
| `api.z.ai` | `zai` |
| `api.meta.ai` | `meta` |
| `api.openai.com` | `openai` |
| anything else | `custom` |
- **Match:** exact host, `chat/completions` path
- **Zen:** unchanged
- **No base header:** `openai`, as before
- **Bounded:** the hostname is never a label
Formatting and the SSRF guard are unchanged.
Pricing tables are unchanged; provider-dependent savings estimates can
change.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Fix:** `CUSTOM_BASE_CHAT_PROVIDERS`, exact host to label
- **Fix:** `CUSTOM_BASE_PROVIDER = "custom"` fallback
- **Fix:** chat handler reports the fixed-set label
- **Removed:** `_custom_base_provider_label()`
- **Test:** known hosts, subpath bases, host case
- **Test:** lookalike hosts, non-chat paths, bad URLs
- **Test:** `zai`, `meta`, `custom`, no-header control
- **Test:** 1,000-request flood across every sink
### Flood regression
Through the real chat handler:
1. 980 distinct hosts: ok, upstream 502, local 429, upstream 429
2. 20 `api.z.ai` requests: ok and local 429
3. Exact counts: in-memory, lifetime, OTel
4. `/metrics` labels: inside `{openai, zen, zai, meta, custom}`
The pre-seeded zero `unknown` failure bucket is the one extra label, on
`main` too.
## Testing
- [x] Unit tests pass (`pytest`): no new failures; see baseline below
- [x] Linting passes (`ruff check .`), plus `ruff format --check .`
- [x] Type checking passes (`mypy headroom`):
`--ignore-missing-imports`, mypy 1.20.2 as in CI
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
new tests against main
|
||
|
|
deca575c63 |
fix(cache): preserve the forwarded OpenAI chat prefix across turns (#3757)
## Description In cache mode, `/v1/chat/completions` changes its forwarded prefix every turn, so the provider's prompt cache misses from last turn's tool result onward. Two causes, and each one alone busts the prefix: ### Tracker originals (all modes) - **Write:** tracker writes omit `original_messages` - **Stored:** compressed bytes, as if the client's - **Next turn:** `finalize_turn` finds no match - **Replay:** none ### Restore after replay (cache mode) - **Order:** restore runs after `finalize_turn` - **Effect:** replayed bytes revert to raw ### Fix - **Originals:** passed on every chat tracker write - **Order:** restore, then `finalize_turn` (cache) - **Token mode:** shared replay now sees originals Folds in #3758: the live run below showed neither half fixes the prefix alone. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Fix:** restore, then `finalize_turn` - **Fix:** originals on every chat tracker write - **Test:** 3-turn lifecycle, buffered and stream - **Test:** originals reach the tracker, per path - **Test:** replay survives restore - **Test:** partial replay, block append, divergence ## Testing - [x] Unit tests pass (`pytest`): no new failures; 8 fail on `main` too - [x] Linting passes (`ruff check .`), plus `ruff format --check .` - [x] Type checking passes (`mypy headroom`): `--ignore-missing-imports`, mypy 1.20.2 as in CI - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text test_openai_cache_mode_forwards_last_turns_bytes_unchanged[buffered|stream]: main |
||
|
|
aebe98954e |
fix(cache): keep cached-prefix replay after background recompression (#3380)
## Description Since the non-inflation size bound added in #3052 (shipped 0.36.x), the Anthropic proxy token path repeatedly busts the provider prompt cache: the bound declines the cached-prefix replay exactly when background compression lands a SMALLER form of an already-forwarded message, so the forwarded bytes change mid-history and the provider cache misses from the first changed byte. Steady state on a busy session is a 160-210k-token cache re-write every 1-2 requests (measured across a 0.35.0 -> 0.37.0 swap on identical workload: dollar cache-hit rate 90% -> 52%, billable input $/M up 2.2x; full data in #3379). Reworked per review (thanks @chopratejas - the direction is yours): rather than disabling the size bound wholesale, `overlay_cached_prefix` now takes `confirmed_frozen_count` and splits at that floor. - Inside the provider-confirmed prefix: replay is unconditional. The replay source is exactly what the provider hashed, so changing those bytes can only bust the cache; an "inflating" replay there bills at the ~0.1x cache-read rate and is still the cheaper request. - Beyond the floor: the size bound arbitrates each turn exactly as on main - a shrinking replay repairs freeze drift (#1850), an inflating one is declined so a fresh compression improvement reaches the wire. - When the provider count collapses (cold cache, TTL lapse) the floor collapses with it and every accumulated improvement lands at once: the natural re-baselining that bounds long-session growth (the #3026 direction of travel) by the TTL-lapse cadence instead of pinning forever. The #3052 alignment/append-only guards stay enforced unconditionally. The OpenAI chat path now carries the same floor (reviewer question: is the root cause live there too?). It is: the tracker is provider-generic and is fed OpenAI's own `prompt_tokens_details.cached_tokens`, the forwarded-bytes feedback loop is identical, both handlers compress through the same ContentRouter, and OpenAI's automatic prompt caching busts on the first changed byte exactly like Anthropic's. Exposure is narrower - the token path's `FREEZE_POLICY_REPLAYABLE` already stops most recompression of already-forwarded messages - but not zero: an entry evicted from the local byte-replay cache drops out of that freeze, and cache mode freezes from the tracker only. Remaining callers with no provider-confirmed count in scope (sidecar, `openai.py:9940`) pass nothing and keep today's fully size-bounded behavior; `bedrock.py` / `gemini.py` never call `finalize_turn` at all, tracked in #3391. Closes #3379 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `overlay_cached_prefix` gains keyword-only `confirmed_frozen_count: int | None = None`: an unconditional-replay floor for the provider-confirmed prefix; the size bound still governs everything beyond it, and alignment guards are untouched. - `finalize_turn` forwards the floor (default `None`, so all existing callers are unchanged). - The OpenAI chat call site passes the same floor, snapshotted before `prepare_turn`'s replayable clamp (that count runs over the LOCAL byte-replay cache and can exceed what the provider confirmed; a cold prefix zeroes it, so cold re-baselining matches Anthropic's). - The Anthropic confirmed-clamp call site stashes the pre-clamp tracker count (the post-`prepare_turn` min under-floors when the local byte-replay cache evicted entries) and passes it; the stash defaults to 0 on paths that never compute a tracker count (backpressure). - Tests: floor replays a recompressed confirmed prefix; improvement beyond the floor lands while the confirmed region replays; originals-drift beyond the floor is still repaired when the replay shrinks; alignment guards unaffected by the floor; block-append within the floor; `finalize_turn` pass-through; the reviewer-requested multi-turn harness test - a background improvement to an already-forwarded message never busts the warm cache across 8 turns and lands on the first cold turn; the reviewer-requested growth bound - forwarded size grows by only its own delta each warm turn and the first cold turn re-baselines to exactly the fresh pipeline's output; and an end-to-end OpenAI chat test asserting the provider-confirmed prefix is replayed even when the replay inflates (fails without the new argument). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py -q ============================== 58 passed in 0.44s ============================== $ uvx ruff check headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py headroom/proxy/handlers/anthropic.py tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py All checks passed! $ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS 15 (arm64), Python 3.12; regression measured on the released headroom-ai 0.37.0 under real Claude Code traffic through `headroom proxy` (token mode); this branch validated with the repo test suite. - Exact command / steps: diffed per-request PERF logs across a 0.35.0 -> 0.37.0 swap on the same machine and workload (sequences and aggregates in #3379); then ran the pytest/ruff/mypy commands above on this branch. A deployment carrying the unconditional-replay behavior on the confirmed prefix restored the healthy pattern live: the broken wheel capped cached reads at 157k across 32 large requests (prefix perpetually re-written) while the patched deployment accumulated to 305k cached reads within 15 minutes. - Observed result: 58 passed including the new multi-turn harness test; on stock 0.37.0 every background-compression completion produced a cache_read collapse to the ~7.7k tools+system head plus a 160-210k cache_write. - Not tested: a live OpenAI `cached_tokens` comparison for the OpenAI half (the Anthropic measurement is what #3379 carries; the OpenAI change is argued from shared mechanism plus an end-to-end handler test, not from its own dollar data). A live provider run of this exact branch (the wheel comparison plus the live patched-deployment measurement isolate the bound as the trigger; the unit and multi-turn regressions reproduce decline and fix deterministically). `mypy headroom` was run only on the two touched modules, not the whole package. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: On the Anthropic proxy confirmed-clamp path and the OpenAI chat path, both of which now replay the provider-confirmed prefix unconditionally (the pre-0.36 posture) while keeping the size bound beyond the confirmed floor. The overlay's default and every other caller keep the fully size-bounded behavior. - Kill switch / disable path: remove the `confirmed_frozen_count=` argument at the two call sites (anthropic.py, openai.py chat); the parameter defaults to `None`. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert the commits; the parameter is additive and default-preserving. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes Lineage for review: #1850 made freeze forward the cached prefix byte-identical, #3052 added alignment guards plus the size bound after #3026's misalignment blowup, #3271 unified both modes on the session engine. The first cut of this PR disabled the bound on the confirmed-clamp path wholesale; the reviewed rework scopes unconditional replay to the provider-confirmed prefix, which additionally gives cold-lapse re-baselining (improvements land whenever the floor collapses), addressing the growth analysis raised in review. Economics discussion in the review thread. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
c16be9bbbe |
fix(proxy/anthropic): don't replay recorded prefix over live history (#3026) (#3052)
## Description A Claude Code session that reads a large tool result through `headroom proxy` can fail on turn 2 with Anthropic's `400 prompt_too_long`. The reporter's controlled comparison completed eight turns through 0.33.0 with 187,986 input tokens, while 0.35.0 failed after five requests with 753,077 input tokens. The local regression uses an actual prior optimized request to populate tracker state, then a decision-false bypass turn with Claude-shaped tool-result content. The old unconditional replay path substitutes the compressed prefix; the eligibility gate preserves the client's outbound body without claiming a live provider reproduction. The Anthropic `/v1/messages` route computes whether a request should be compressed, but cached-prefix replay currently runs outside that decision. The replay helper also derives its prefix length from the original message list and applies that index to the optimized list without proving the two lists still align. A stale forwarded prefix can therefore be grafted onto the wrong positions and enlarge later requests. This change limits replay to requests whose existing compression decision permits it and whose pre-upstream backpressure path is inactive. It also makes `overlay_cached_prefix()` decline misaligned or inflating candidates while preserving normal append-only replay. Reported by @itsumonotakumi, whose controlled comparison isolated the failure from compression, headers, one-request serialization, memory, code graph, and CCR. Closes #3026 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Gate Anthropic cached-prefix replay on the existing `CompressionDecision.should_compress` result and the existing pre-upstream backpressure state. - Require positional alignment between optimized and original message arrays before replay. - Reject replay candidates that would serialize larger than the current optimized messages. - Add focused handler coverage for the decision-false tool-result regression, bypass and backpressure paths, and outbound optimize-on preservation. - Add direct unit coverage for positional mismatch, no-inflation, and JSON sizing-failure bailouts. - Update the moved-cache-control and pure-block-append regression fixtures to keep the no-inflation contract explicit. - Run the unchanged OpenAI cache-stability preservation proof; no OpenAI production code was edited. ## Testing - [x] Unit tests pass (153 focused proxy, helper, cache-control, block-append, cross-turn, byte-faithful, Anthropic, OpenAI, and backpressure tests) - [x] Linting passes (Ruff check and format validation on the seven changed repository files) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed with the in-process proxy and local stub upstream ### Test Output ```text python -m pytest tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_anthropic_cache_stability.py tests/test_anthropic_pre_upstream_backpressure.py -q python -m pytest tests/test_proxy_openai_cache_stability.py -q python -m pytest tests/test_issue_2671_block_growth_cache.py::test_pure_append_replays_forwarded_blocks_and_advances_breakpoint -q 153 passed across focused invocations, exit code 0 optimize_off turn2_message_count=3 marker_count=1 outbound_compact_utf8_bytes=2293 client_compact_utf8_bytes=2293 optimize_on turn2_message_count=3 client_message_count=3 marker_count=0 outbound_compact_utf8_bytes=171 client_compact_utf8_bytes=182 python -m ruff check headroom/proxy/handlers/anthropic.py headroom/cache/prefix_tracker.py tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_proxy_openai_cache_stability.py All checks passed!, exit code 0 python -m ruff format headroom/proxy/handlers/anthropic.py headroom/cache/prefix_tracker.py tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_proxy_openai_cache_stability.py --check 7 files already formatted, exit code 0 git diff --check clean, exit code 0 ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12 via `uv`, real Headroom proxy app with a local stub Anthropic upstream - Exact command / steps: send an actual optimize-on first request through the in-process proxy with a deterministic production-pipeline seam, then send a decision-false bypass turn containing a large Claude-shaped `tool_result` with moved `cache_control`; separately send an aligned optimize-on turn with a new suffix - Observed result: the exact base checkout fails with `AssertionError: assert 'compressed-tool-result' == 'large-tool-result-marker ...'`; the guarded path passes with the client marker present once and outbound compact JSON no larger than the client body. The optimize-on preservation run records `optimize_on turn2_message_count=3 client_message_count=3 marker_count=0 outbound_compact_utf8_bytes=171 client_compact_utf8_bytes=182`, proving the actual compressed prefix is outbound before the new suffix without turn-2 growth. - Not tested: live Claude Code session against api.anthropic.com on this host ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: cached-prefix replay now follows the existing compression and backpressure decision and rejects misaligned or inflating candidates. - Kill switch / disable path: no new switch; the existing optimize and bypass controls remain available. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert the implementation commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes ## Additional Notes `CHANGELOG.md` is not modified because Headroom's release automation generates it from conventional commits. This change does not add a context-limit guard or alter compression, streaming tracker provenance, outbound-body selection, OpenAI behavior, or provider limits. Local tests prove request-body ownership and replay bounds. The reporter's live Claude Code completion and Anthropic token acceptance remain external to this local proof. |
||
|
|
05932d7165 |
fix(proxy): compress OpenCode tool schemas and embedded JSON (#1535)
## Description Fixes two remaining OpenCode/OpenAI Chat compression gaps after `main` incorporated the original savings-profile threading and user content-block work from this PR. OpenCode requests can still report very low savings when most input tokens live in verbose `tools` schemas rather than messages. They can also route poorly when a short instruction wraps a valid JSON block but does not satisfy the existing long-prose heuristic. Closes #1534 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that causes existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Compact OpenAI Chat Completions `tools` schemas whenever request compression is active, reusing the existing OpenAI Responses schema compactor. The outbound tool invocation shape is preserved while non-semantic annotations such as `$schema`, `title`, and `examples` are removed. - Include the tool-schema token delta in Headroom's savings accounting and expose `openai:chat:tool_schema_compaction` in the applied transforms. - Detect valid JSON blocks surrounded by prose or log text as mixed content, so short OpenCode instructions route through mixed/SmartCrusher handling instead of falling through or producing a no-op. - Adapt the mixed-content change to the new `headroom.transforms.mixed_content` module introduced on `main` by #1939. ## Why the Focus Changed The original headline fix—threading savings-profile kwargs into `/v1/chat/completions`—is now already present on `main`, as is the user content-block opt-in behavior. Those duplicate changes were removed during the merge. The branch also no longer changes developer/system role protection or forced-Kompress semantics. It follows `main` for both, so the earlier instruction-role safety concern is outside the current diff. The resulting PR is limited to two OpenCode-specific compression gaps that remain reproducible on current `main`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` on changed files) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for the fixed behavior - [ ] Manual live-upstream testing performed after the latest rebase ### Test Output ```text 59 passed, 1 warning in 83.53s All checks passed! # ruff check 4 files already formatted # ruff format --check python -m py_compile: passed git diff --check: passed ``` Focused test coverage includes: - OpenAI Chat tool-schema compaction, transform reporting, outbound schema shape, and positive token savings. - Embedded JSON mixed-content detection, SmartCrusher routing, positive savings, and preservation of a critical sentinel value. - Current `main` regressions for savings-profile threading, user content blocks, turn hooks, and forced-Kompress behavior. ## Real Behavior Proof - Environment: Linux ARM64, Python 3.13.12, current `main` at `9bacf481` merged into the branch. - Exact command / steps: focused pytest run across the OpenAI cache-stability, content-router, mixed-content, savings-profile, user-block, turn-hook, and forced-Kompress suites. - Observed result: 59 tests passed; the chat request test forwarded compacted tools and reported positive savings, while the embedded-JSON fixture used mixed routing and preserved `CRITICAL_NEEDLE_42`. - Not tested: full repository suite and a live external OpenCode request after the latest merge; those remain for CI/live follow-up. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented the non-obvious behavior - [ ] I have made corresponding documentation changes — N/A; internal routing behavior only - [x] My changes generate no new warnings - [x] I have added tests that prove the fixes are effective - [x] New and existing focused tests pass locally - [ ] I have updated the changelog — N/A; release automation handles fix entries ## Screenshots N/A — proxy/transform behavior only. ## Additional Notes - Current diff versus `main`: 4 files, 172 insertions, no role-policy or forced-Kompress changes. - The mixed-content conflict was resolved by extending the new isolated parser module rather than reintroducing parsing code into `ContentRouter`. |
||
|
|
55efb1c77d |
fix(proxy): keep OpenAI tool observations mutable in cache mode (#1884)
## Description Diagnoses and fixes the low-savings OpenAI-compatible cache-mode path reported in #1696. OpenAI-compatible tool-calling clients can end a turn with `role: "tool"` (or legacy `role: "function"`) rather than `role: "user"`. The OpenAI chat handler's cache-mode freeze boundary treated those tails as non-mutable, and because `HeadroomProxy` resolves `_strict_previous_turn_frozen_count` from the Anthropic mixin first, the OpenAI-specific helper was not used in production. That froze the entire conversation before `ContentRouter` ran, leaving no live tool observation to compress and producing near-pass-through savings on long coding sessions. This PR keeps final OpenAI tool/function observations mutable in cache mode, explicitly calls the OpenAI helper to avoid the mixin-name collision, and clamps negative token-savings artifacts at the metrics/cost aggregation boundary so stats cannot under-report actual forwarded savings. Closes #1696 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Treat final OpenAI `user`, `tool`, and `function` messages as the mutable cache-mode live zone. - Route OpenAI cache-boundary calls through `OpenAIHandlerMixin._strict_previous_turn_frozen_count` explicitly so the Anthropic mixin method cannot shadow it in `HeadroomProxy`'s MRO. - Preserve cache-mode live-tail boundaries even when compression-cache state would otherwise freeze the whole request. - Clamp negative `tokens_saved` artifacts in `CostTracker.record_tokens` and `PrometheusMetrics.record_request`. - Add regression coverage for OpenAI final `tool`/`function` tails, over-frozen tracker state, and non-negative savings aggregation. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ maturin build --profile ci --out dist --interpreter python Built wheel for abi3 Python >= 3.10 to dist\headroom_ai-0.29.0-cp310-abi3-win_amd64.whl $ python -m pytest tests\test_proxy_handler_helpers.py tests\test_proxy_openai_cache_stability.py tests\test_observability_metrics.py tests\test_cost_tracker_counterfactual.py 49 passed in 10.27s $ python -m ruff check . All checks passed! $ python -m mypy headroom Success: no issues found in 407 source files $ python -m pytest 53 failed, 7703 passed, 488 skipped, 5893 warnings, 131 errors in 595.18s (0:09:55) ``` Full-suite note: the full local `pytest` run was attempted on Windows/Python 3.13 after building `headroom._core`. It did not complete green due to broad pre-existing/local-environment failures outside this change area, dominated by SQLite/memory persistence permission/path errors plus unrelated adapter/cache/tool tests. The focused regression suite for this PR passes, and repo-level lint/type gates pass. ## Real Behavior Proof - Environment: Windows, Python 3.13.13, Rust/Cargo available, local `headroom._core` wheel built with `maturin build --profile ci`. - Exact command / steps: ran the OpenAI cache-stability tests with final `role: "tool"` and `role: "function"` chat tails. - Observed result: `test_openai_cache_mode_keeps_final_tool_observation_mutable[tool]` and `[function]` pass, proving the pipeline receives `frozen_message_count == 2` for a 3-message request instead of freezing all 3 messages. - Not tested: live Lemonade/KiloCode upstream session; no local Lemonade Server was available. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Docs and CHANGELOG are N/A for this narrow proxy bug fix. The broad local `pytest` checkbox is intentionally left unchecked because the full suite had unrelated local-environment failures; see the test output above. Focused regression tests, `ruff check .`, and `mypy headroom` are green. |
||
|
|
248ae0f3e0 |
fix(proxy): freeze must forward cached (compressed) prefix byte-identical — stop token-mode cache busting (#1850)
The freeze path (both providers) emits the agent's ORIGINAL bytes for a frozen message, but the provider cached whatever we FORWARDED last turn (the compressed form). Forwarding original then mismatches the cached prefix and busts it from that point — re-creating the whole suffix. Measured on a real SWE-bench run: 100% of attributed misses were prefix_change, ~56% of ALL cache-writes were bust-induced (2.8M tokens), driving cache_create +150% and cost +41% vs baseline. Cache mode already avoided this via _extract_cache_stable_delta (replay the previously-forwarded prefix, compress only the delta). Token mode called apply(frozen_count) directly, which forwards original for the frozen region. Fix: add a shared, provider-agnostic overlay_cached_prefix() that replays the previously-forwarded (cached, compressed) prefix byte-identical, append-only guarded and idempotent, and apply it in BOTH the Anthropic and OpenAI handlers right before forwarding. This makes freezing byte-identical in every mode, so the only remaining difference between "token" and "cache" mode is how large a mutable (still-compressible) tail each leaves — not whether the frozen prefix busts the cache. Tests: - test_cache_prefix_overlay.py: the helper (replay, append-only guard, idempotence). - test_cross_turn_cache_safety.py: the invariant that was missing — drive the REAL tracker + freeze + overlay over multiple append-only turns against a simulated provider prefix cache and assert the forwarded prefix stays byte-identical turn-over-turn. Load-bearing: it fails (detects the bust) without the overlay. ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
|
f0dcc02775 |
fix: A3 — byte-faithful Python forwarders; serialize canonical only when mutated
Eliminates P0-2 universally. Every Python forwarder (server.py
`_retry_request`, handlers/streaming.py `_stream_response`,
handlers/openai.py `_ws_http_fallback`, handlers/batch.py `_batch_passthrough`
+ batch-create + Google batch passthrough, handlers/anthropic.py CCR
continuation + batch endpoint) now switches from `httpx ... json=body` to
`httpx ... content=raw_bytes`. The default httpx JSON encoder was
re-serializing every request with `, `/`: ` separators and `\\uXXXX` ASCII
escapes — collapsing Anthropic prompt-cache hit-rate.
Forwarder strategy:
- unmutated body → forward `await request.body()` verbatim;
- mutated body → re-serialize once via the new
`serialize_body_canonical(body) -> bytes` helper (compact separators,
`ensure_ascii=False`, dict insertion order preserved).
`HEADROOM_PROXY_PYTHON_FORWARDER_MODE` env var configures the mode:
- `byte_faithful` (default) — the new behavior;
- `legacy_json_kwarg` — explicit operator opt-in for emergency rollback.
Documented in `docs/content/docs/configuration.mdx`. NOT a fallback —
unknown values raise loudly per build constraint #4.
`BodyMutationTracker` accompanies each request through the handler so
transform sites mark the tracker (`memory_injection`,
`image_compression`, `compression_*`, `batch_compression`,
`ccr_continuation`, etc.). At forwarder dispatch we additionally compare
the final body dict against the parsed original bytes as a structural
safety net — any silent mutation we missed still triggers canonical
re-serialization.
A2 follow-up: `handlers/openai.py:534-540` (Chat Completions memory
injection) was prepending a system message; replaced with
`append_text_to_latest_user_chat_message`, the OpenAI Chat Completions
analog of `_append_context_to_latest_non_frozen_user_turn`. The cache
hot zone (system messages) is now sacrosanct on /v1/chat/completions
too. Honors `HEADROOM_MEMORY_INJECTION_MODE=disabled`.
Structured logging: every forwarder emits an `event=outbound_request`
log line with `forwarder`, `path`, `body_bytes`, `body_mutated`,
`mutation_reasons`, `source` (passthrough|canonical|legacy),
`request_id`. Never logs Authorization or full body.
`_read_request_json` factored to share `_read_request_body_bytes` with
new `read_request_json_with_bytes` so the anthropic handler can capture
both the parsed dict and the original (decompressed) bytes.
Tests:
- `tests/test_proxy_byte_faithful_forwarding.py` (28 tests):
SHA-256 byte-equality on /v1/messages and streaming, unicode
preservation, numeric precision, mutation-tracker invariants,
canonical-serializer properties, legacy-mode rollback, OpenAI
Chat memory routing.
- Existing test mocks updated to accept the new `**kwargs` on
`_retry_request` (no behavior change).
- `tests/test_proxy_handlers_batch.py` updated to read the captured
`content=` bytes (formerly `json=`).
- One A2 test corrected (`test_anthropic_tool_sort_and_context_append_helpers`)
to match the live-zone-tail semantics introduced by A2.
Constraints satisfied: configurable env var; no new regex / hardcodes;
no silent fallback (`legacy_json_kwarg` is operator opt-in);
performant (`prepare_outbound_body_bytes` is O(1) for passthrough);
elegant single-responsibility helpers; structured tracing logs.
|
||
|
|
35eaf8de7f |
fix(proxy): remove content-keyed TTL walker that conflated content with positional cache (#327)
The Anthropic token-mode handler walked past prefix_tracker.frozen_message_count
whenever an upcoming tool_result's content-hash matched comp_cache._stable_hashes
or should_defer_compression returned True. That conflated content equality with
positional cache membership.
Anthropic's prefix cache is POSITIONAL: bytes 0..K cached, anything past K is
fresh. _stable_hashes is content-keyed and grows unbounded. In long Claude Code
sessions where tool_result content rhymes across turns (repeated system prompts,
repeated file reads, repeated tool descriptions), the walker advanced
frozen_message_count to len(messages) on every turn and the pipeline produced
transforms_applied=[] on 73% of requests in user SvenMeyer's reported session
(headroom-stats-2026-05-01.json: 74 of 101 eligible requests "prefix_frozen") —
even after the prior fix in
|
||
|
|
d00c6739e1 | Fix CI regressions for cache benchmark work | ||
|
|
83b730f2b9 | Fix CI lint/format failures after proxy mode hardening | ||
|
|
2625789a28 | Harden cache-mode immutability for OpenAI and fix stats mode reporting |