Commit Graph
18 Commits
Author SHA1 Message Date
2f58877732 fix(proxy): check x-headroom-base-url off the event loop in chat and responses (#4856)
## Description

`handle_openai_chat` and `handle_openai_responses` validated
`x-headroom-base-url` with the synchronous `is_safe_upstream_url`, which
waits for the DNS lookup (up to `HEADROOM_UPSTREAM_RESOLVE_TIMEOUT_S`, 3
s by default) on the event-loop thread. Chat did this three times per
request and responses twice, so one slow-resolving host held up every
other in-flight request for several seconds. The other routes in
`headroom/providers/proxy_routes.py` already use
`is_safe_upstream_url_async`.

Both handlers now resolve the header once, through a new
`_resolve_openai_upstream_base_async` that uses the async guard (lookup
on a worker thread, pin scope opened on the request's task), and reuse
that result for the destination and the extra-header decision.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/openai.py`: header parsing split into
`_openai_base_url_candidate`; sync `_resolve_openai_upstream_base` (kept
for `_resolve_openai_upstream`) and new
`_resolve_openai_upstream_base_async` share it;
`_openai_upstream_from(request, custom)` picks the upstream from an
already-resolved override; chat and responses call the async resolver
once.
- `tests/test_proxy/test_openai_upstream_guard_off_loop.py`: asserts the
guard runs once and not on the event-loop thread, for chat and
responses.
- `tests/test_proxy_openai_cache_key_integration.py`: its stub now
patches the async resolver the handler calls.

**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-idna-pin-once` adds the identical `_openai_upstream_from`
and resolve-once hunks (this PR is a superset of that part);
`fix/openai-base-url-reject` changes the rejection branch of
`_resolve_openai_upstream_base`, which after this PR must be mirrored in
`_resolve_openai_upstream_base_async`. Suggested order: idna-pin-once,
then this PR, then reject (rebasing reject means raising the 400 from
the shared rejection path in both resolvers).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New test on origin/main (c21b0813)
E   AssertionError: the guard ran on the event-loop thread
E   assert [True, True, True] == [False]      (chat)
E   assert [True, True] == [False]            (responses)
============================== 2 failed in 0.72s ===============================

# This branch: new test + cache-key integration
============================== 7 passed in 7.30s ===============================

# Area suites: test_upstream_guard, test_grok_shared_proxy_call_sites, test_proxy/test_openai_upstream_header,
# test_proxy_openai_cache_key_integration, test_upstream_credential_scoping, test_header_isolation,
# test_proxy_hardening, test_openai_codex_routing, test_proxy/test_openai_upstream_guard_off_loop
======================= 282 passed, 5 warnings in 56.32s =======================

$ ruff check / ruff format --check (0.17.3) on changed files
All checks passed!
3 files already formatted
$ mypy headroom/proxy/handlers/openai.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, full app from `create_app` driven
through `httpx.ASGITransport` on one event loop; `socket.getaddrinfo`
replaced so `slow.example` takes 2 s to answer (everything else real).
- Exact command / steps: start a `/v1/chat/completions` request with
`x-headroom-base-url: https://slow.example/v1`, then 0.2 s later send
`GET /livez` and time it.
- Observed result:
  ```text
  == origin/main
/livez sent 0.2s after a chat request naming a 2s-to-resolve host: HTTP
200 after 6.03s (0.20s if the loop is free)
  == this branch
/livez sent 0.2s after a chat request naming a 2s-to-resolve host: HTTP
200 after 0.20s (0.20s if the loop is free)
  ```
- Not tested: under real slow DNS.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no change in which upstream is
chosen; the lookup moves to a worker thread and happens once per
request.
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

Follow-up, not in this PR: `select_passthrough_base_url`
(`headroom/providers/proxy_targets.py`, Azure `api-key` branch) and
`headroom/providers/registry.py` still call the sync guard; the
passthrough route already checks the same header with the async guard
first, so the sync call there is a second lookup on the loop.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4856"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-15 01:52:37 +05:30
b1fb162b62 fix(grok): attribute api.x.ai chat outcomes as provider xai, recognize grok-shell UA (#2626)
## Description

Grok Build chat requests were recorded as `custom` or `openai`,
preventing xAI provider rollups. This change records custom-base and
directly routed Grok chat traffic as `xai`, recognizes both `grok/` and
`grok-shell/` clients, and keeps API-key requests on credential-driven
PAYG policy.

Closes #3996

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Attribute supported xAI chat routes, including trailing-dot hosts, to
`xai`.
- Preserve per-model catalog cache prices and share the xAI cache
fallback between dashboard costs and persisted savings.
- Exclude inferred cache writes from billed aggregate totals; those
tokens already belong to uncached input. Explicit provider-reported
writes remain billable.
- Cover custom-base outcome integration, catalog pricing precedence,
legacy and current client credentials, and inferred writes with and
without TTL splits.

## Testing

At `29fd4f9d7a24a76f3a3f53edacc5832e357cae58`: 130 focused pricing,
accounting, auth, cache, and routing tests pass (one existing Starlette
warning). Ruff lint and format, whitespace checks, and Linux-target mypy
over 567 source files pass. Main refresh retains `3c418f6f` ancestry.
Latest documentation dependency audit found zero vulnerabilities.

## Real Behavior Proof

- Environment: Windows, Python 3.12, a real uvicorn proxy and a separate
loopback HTTP upstream. Subscription polling disabled. Only upstream
network delivery is redirected; this is not live xAI billing traffic.
- Steps: send four real TCP chat POSTs with custom-base, direct, legacy
Grok, and trailing-dot routing; query the running proxy's `/stats`
endpoint. The upstream returns 10,000 input tokens, 5,000 cached input
tokens, and 1,000 output tokens per request. A priced xAI catalog
fixture has $1/M input and $2/M output, with no explicit cache-read
rate.
- Observed: all four requests return HTTP 200, outcomes record
`provider=xai` and `client=grok_build`, and `/stats` records
`{"xai":4}`. Input spend is $0.0232 using the xAI 0.16 cache fallback.
Before the aggregate-accounting correction this same replay reported
$0.0432.
- Evidence: `pr2626-resume15-pricing-runtime.log` and
`pr2626-resume15-final-tests.log`; regressions reproduce the
pricing/auth defects before correction.

## Runtime Rollout Safety

- Rollout-managed features: none; attribution, credential
classification, and accounting use the existing request paths.
- Default behavior: supported xAI routes receive the xAI label; API keys
use PAYG policy, JWT credentials retain OAuth policy. Unknown gateways
retain the bounded custom provider bucket.
- Pricing behavior: when a priced catalog row has no cache rates and the
caller omits `provider`, `resolve_rates` now identifies the provider
from `litellm_provider` and uses the shared fallback. For xAI this means
cached input costs 0.16 times list input; explicit model cache rates
still win. Dashboard aggregates also exclude inferred writes already
counted as uncached input. These corrections change reported input costs
and persisted savings, not upstream requests or billing.
- Rollback: revert the pricing/auth/attribution commits; no new rollout
switch or unsafe override is introduced.
- Coverage: real HTTP routing and `/stats` replay plus focused
default-policy and accounting tests.

## Checklist

- [x] Self-review completed
- [x] Added regression coverage
- [x] Relevant local tests and static checks pass
- [x] Did not edit CHANGELOG.md

## Additional Notes

Hosted checks and automated review must qualify this newly published
head before approval. No PR merge performed.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-08 10:56:22 -05:00
Mansurshakh JaparovandJD Davis 9553d4292f fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description

Session cleanup currently drops prefix trackers after the default
600-second session lifetime even when Anthropic's request selects a
one-hour cache TTL. Losing the tracker also loses the frozen forwarded
prefix and causes the cache-preserving overlay to replay different
bytes.

Closes #4017.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Resolve the request's existing Anthropic cache TTL before selecting
the prefix lineage and pass that TTL through tracker selection/creation.
- Retain each lineage for the greater of the configured session floor
and its observed/current provider TTL plus the existing 60-second safety
margin.
- Keep the lifetime per lineage rather than changing global
configuration: a five-minute session does not inherit another session's
one-hour lifetime.
- Preserve normal expiry beyond the provider lifetime and margin.
- Honor configured cache TTL before a confirmed cache write; discard
expired state before accepting a later request's longer TTL.
- Re-register the live lineage family after lookup/cleanup so recreated
sessions cannot share a tracker's frozen state across sibling
conversations or tool profiles.
- Update existing fake callbacks to the changed tracker contract,
preserve all consumer byte-stability assertions, and remove one
handler-to-store wiring-only test rather than re-pinning its internal
identity assertions.
- Document local tracker retention separately from provider cache
availability.

## Testing

- [x] Unit tests pass (`pytest`), including frozen-prefix byte stability
and independent TTL expiry.
- [x] Changed-file lint, configured pre-commit mypy and manual
production timestamp/overlay checks pass.

### Test Output

```text
Review correction: 273 passed, 2 dependency warnings in 83.79s
{"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true}
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate
explicitly used the installed CPU `libonnxruntime.so.1.26.0` via
`ORT_DYLIB_PATH`, without skipping cached-model assertions.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 12.87s ==================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_cache/test_prefix_tracker.py
python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py
```

**74 passed** in tracker tests; **54 passed** in request/cache tests
(one existing Starlette TestClient deprecation warning). Two new
regressions fail on baseline because the request TTL is not accepted; a
separate production runtime comparison demonstrates the actual pre-fix
loss, not just the missing parameter.

After rebasing onto current main
`1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together
with `tests/test_prefix_tracker_client_rewrote_reply.py`,
`tests/test_cache_mode_delta_marker.py`, and
`tests/test_content_router_mcp_small_result.py`: **150 passed**. The
actual production timestamp/byte-retention smoke was repeated
successfully on the rebased code, covering compatibility with main's
newly merged client-reply normalization.

Review regression and affected HTTP-consumer suite: **273 passed**,
including tracker expiry, Anthropic/OpenAI cache stability,
byte-faithful forwarding, header isolation, buffered timeouts, CCR
injection, and immutable system prompts. All affected two-argument test
callbacks now accept the TTL contract. The obsolete walker-call-order
test was removed; wire-byte assertions remain. The new
configured-TTL/expiry cases fail against the published baseline; the
post-cleanup regression also verifies that an unrelated conversation
gets a different tracker after recreation.

Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and
Commitlint hooks passed. Full-tree Ruff is locally blocked by executable
filesystem permissions on five unchanged OAuth2 plugin files (tracked
100644). A separate full dev-environment mypy run found an unchanged
optional LangChain `truthy-function` error; the configured isolated
pre-commit hook passed. No unrelated source/permissions changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real
`SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token
estimator and `overlay_cached_prefix`, no mocked store or mocked clock.
- Exact command / steps: parse `cache_control: {type: ephemeral, ttl:
1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed
frozen message with different original/forwarded prefix bytes; age
`_last_activity` to `time.time()-660` and make periodic cleanup due;
resume the same lineage and compare the tracker, frozen count, and
forwarded-prefix bytes. Also age beyond 3901 seconds and compare
independent 300/3600-second sessions.
- Before: resolved TTL 3600; after 660 seconds, tracker identity
changes, frozen count drops **1→0**, and forwarded-prefix bytes differ.
- Observed result: after the fix, after 660 seconds the same tracker
remains, frozen count stays **1**, and forwarded-prefix bytes are
identical. The one-hour session survives independently, the five-minute
session expires, and the one-hour lineage still expires beyond TTL plus
margin.
- Review runtime proof against published head `e1a107df`: actual stores
and prefix overlays confirm configured TTL retention before the first
cache write, no resurrection of expired state by a later one-hour
request, eventual expiry, and separate trackers after the maintainer's
3901-second cleanup/recreation sequence. On that exact baseline the
unrelated lookup reuses the wrong tracker, although the overlay guard
preserves the particular unrelated input bytes; after the fix both
identity isolation and byte preservation hold. Identical messages under
different tool profiles also receive a cold, separate tracker after the
expired bare-key transition.
- Not tested: Anthropic cloud cache availability, billed savings, or an
actual one-hour wait. The local production state and byte invariants
were exercised by aging actual timestamps, not by a paid upstream
request. Preserving a tracker cannot guarantee that the provider retains
its cache.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; repair of existing TTL-aware
cache-prefix behavior.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: eligible local prefix trackers
survive for their provider cache lifetime. Short-TTL/default sessions
remain bounded by their own session floor.
- Kill switch / disable path: the existing prefix-freeze disable path
remains available; no new flag.
- Unsafe override required: no.
- Qualification impact: local Anthropic prefix lineage
selection/retention, not cache pricing or subscription quota policy.
- Rollback path: revert the commit to restore previous retention. No
persistent schema change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing cache TTL parser and safety margin reused; cache docs
updated.
- [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes,
and existing request byte-stability behavior.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

Remote CI results are not inferred from local verification.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 17:46:21 -05:00
ee731d7ea8 fix(proxy): refuse token-less non-loopback binds, gate operator routes (#3852)
## Why

Four related gaps in the proxy's network gate (findings 01-F1, 01-F2,
01-F5, 01-F9):

- **Open bind.** A non-loopback bind with no `HEADROOM_PROXY_TOKEN` only
logged a warning, then served the `/v1/*` relay, with the operator's
upstream credentials, to every peer that could reach the port.
- **Missing peer address.** `is_loopback_host(None)` returned `True`. A
Unix-domain socket, an ASGI adapter that leaves `scope["client"]` unset,
or a hand-built request therefore turned off the token gate, the
`/admin` and `/debug` guards and the WebSocket gate.
- **Operator data routes.** `/stats-history` (including CSV export),
`/quota`, `/subscription-window` and `/metrics` were served to any
network caller.
- **WebSocket loopback exemption** checked only the peer IP, not the
`Host` header, so it had no DNS-rebinding defence.

## What changes

- New `headroom/proxy/bind_policy.py`. A non-loopback bind (`0.0.0.0`,
`::`, a LAN address or a hostname) with no token is refused at three
places: `create_app` (embedders and the multi-worker factory),
`run_server` (one exit with code 2 before uvicorn forks) and the
`headroom proxy` CLI (a clear error before the banner, exit code 1). An
absent host means uvicorn's default `127.0.0.1`.
- `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` acknowledges an open bind when
the runtime already restricts reachability, such as a container
published on `127.0.0.1`. The `proxy_open_bind` warning is still logged
on that path, and the banner says the bind is open.
- `is_loopback_host(None)` now returns `False` (fail closed), matching
`identity.py`.
- `/stats-history`, `/quota` and `/subscription-window` answer only: a
non-loopback caller that passed the token gate, a loopback caller (peer
and `Host` header both loopback), a trusted dashboard client, or, under
an acknowledged token-less bind, the container's own default-gateway TCP
peer with a loopback `Host` (the host browser reaching a
`127.0.0.1`-published container). Everyone else gets 404, as with
`/admin`. Settings writes keep their stricter rule with no token
short-cut.
- `/metrics` answers a non-loopback caller that passed the token gate, a
loopback caller, a connecting peer in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` (whatever it forwards), or a
resolved client in the dashboard CIDRs. For CIDR peers a hostname `Host`
is fine, so Prometheus scrapers work; a cross-origin browser request is
still refused.
- The token short-cut applies only to non-loopback requests. The gate
exempts loopback peers without checking `Host`, so a loopback request
has not proven it holds the token and must pass the loopback `Host`
check. This blocks a DNS-rebound browser on the operator's machine.
- The `/dashboard` shell stays ungated. It is a static template, like
the already public `/dashboard/static` assets. Gating it would break the
dashboard for containers published on host loopback, where the peer is
the bridge gateway and `Host` is `localhost`. The data routes behind it
are gated.
- The WebSocket loopback exemption requires a loopback peer and a
loopback `Host` header. A loopback peer with a foreign `Host` can still
connect with the token.
- Launchers that always publish on `127.0.0.1` set the acknowledgement
themselves: `headroom install` (docker preset), `scripts/install.sh` and
`scripts/install.ps1` (through one helper that adds the acknowledgement
only together with the `127.0.0.1` publication), and
`docker/docker-compose.native.yml`.
- Docs: `proxy.mdx` documents the new variable and which routes the
token also gates; `docker-install` (docs and wiki) and the
`docker-compose.yml` comment show the acknowledgement in the
loopback-published example. `e2e/docker-bind-security.sh` sets it and
checks that the same launch without it refuses to start.

## Behaviour change (breaking)

- `headroom proxy --host 0.0.0.0`, or `HEADROOM_HOST` set to any
non-loopback address, without `HEADROOM_PROXY_TOKEN` now refuses to
start. The error names both remedies. Set a token, bind loopback, or set
`HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` when the port is already
confined. Anyone running the image by hand with `--host 0.0.0.0` and no
token must add one of these.
- Without a token, `/stats-history`, `/quota`, `/subscription-window`
and `/metrics` return 404 to non-loopback callers outside a trusted
CIDR. Prometheus scrapers on another host must be in
`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` or send the token.
- Code that relied on a request with no peer address being treated as
loopback now gets the non-loopback path.
- The default loopback run is unchanged.

## Tests

- `tests/test_proxy_bind_policy.py`: the refuse/allow matrix
(non-loopback, loopback, unset host, token, acknowledgement values), the
warning on the acknowledged path, `create_app`, the `run_server` exit
before uvicorn, and the CLI (refusal, `HEADROOM_HOST` from env,
acknowledged start, offline banner, start with a token).
- `tests/test_proxy_loopback_gating.py`: each data route against
non-loopback, loopback, trusted-CIDR and token callers; the dashboard
shell stays reachable; the host of an acknowledged, loopback-published
container reads the data routes while another bridge peer, a forwarded
gateway address and a foreign `Host` do not; CSV export is not served to
network callers; `/metrics` across loopback, both CIDR lists,
out-of-range peers, cross-origin browsers and token callers, including a
trusted gateway relaying a scraper outside its own range. A regression
test shows that a token on the proxy does not exempt a loopback peer
from the `Host` check; it fails without the fix.
- `tests/test_proxy_hardening.py`: the HTTP gate with no peer address,
WebSocket `Host` cases, `require_loopback` and `/debug` with no peer.
- Installer tests: `headroom install` adds the acknowledgement, and the
shell and PowerShell wrappers add it only together with the loopback
publication. Launcher-level tests start the proxy with the environment
and `--host` that `build_runtime_command()` and
`docker-compose.native.yml` produce and check that the host browser
(bridge-gateway peer) gets `/stats-history` and another container does
not.
- Tests that bound `0.0.0.0` for unrelated reasons now set a token or
the acknowledgement. Tests that read operator routes use an explicit
loopback peer.

On `main`, 14 of the gating and hardening tests fail and the bind-policy
module does not exist. Manual runs: token-less `--host 0.0.0.0` exits
with the refusal from both the CLI and `run_server`; with the
acknowledgement, the data routes are 404 to a LAN peer and to a loopback
peer with a foreign `Host`; with a token, a LAN peer gets 401 without it
and 200 with it; the default loopback bind starts as before.

Not tested: a real Unix-domain-socket deployment and a real Prometheus
scrape (both covered by tests only).

## Merge order

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. This PR goes first: it is the gate the others assume, and it is
the release-note event.

## Not in this PR

- Enforcing the token for loopback callers when one is configured.

Revised 2026-10-02 after review: `/metrics` matches gateway CIDRs
against the TCP peer, and the operator data routes serve the host of a
loopback-published container (bridge-gateway peer under the launchers'
bind acknowledgement) so the default Docker dashboard works.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3852"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:27:35 -05:00
6fb7cadfe4 fix(proxy): key rate limits by peer unless the caller authenticated (#3860)
## Why

The proxy's request and token rate limits could be walked around by any
caller the proxy cannot authenticate (finding 01-F3). Each provider
handler keyed its buckets on a client-supplied header:

- **OpenAI:** an HMAC of the full `Authorization` / `api-key` value,
else one shared `"default"` bucket. Rotating the header gave a fresh
bucket per request, and all callers without a credential shared one
bucket.
- **Anthropic:** the first 16 characters of the key plus the peer. That
is mostly the common `sk-ant-api03-` prefix.
- **Gemini:** the first 20 characters of `x-goog-api-key`, with no peer
at all.

## What changes

- New `headroom/proxy/rate_limit_identity.py` with one rule for the
Anthropic, OpenAI and Gemini handlers. `rate_limit_identity()` returns a
plain string key:
- Every key starts with the peer from `resolve_client_ip`, which honours
`X-Forwarded-For` only from `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`
peers. IPv6 peers are grouped by `/64`; an IPv4-mapped IPv6 address
counts as the IPv4 peer.
- **Trusted** requests get one bucket per provider credential:
`peer:<addr>|cred:<hmac>`, an HMAC of the whole credential with a
process-local key, so bucket names hold no credential material. A
request is trusted if it presented a valid `HEADROOM_PROXY_TOKEN` or
came directly from loopback. A trusted-gateway peer is trusted to supply
the client address, not to vouch that the caller authenticated, so a
forwarded request without the token is untrusted and keyed by the
forwarded client address. A request from a gateway address that carries
no `X-Forwarded-For` is judged as a direct request, so a loopback
gateway CIDR does not change how direct localhost callers are keyed.
Distinct principals behind one address keep separate limits, as #3364
intended.
- **Untrusted** requests (no token, remote caller) are charged per peer
(`peer:<addr>`). The credential header is ignored.
- The security gate sets `request.state.proxy_authenticated = True` when
a caller presents a valid proxy token.
- The per-handler key code is removed (`_openai_rate_limit_key` and the
Anthropic and Gemini inline keys).

`rate_limiter.py` is unchanged, and so are token-limit semantics.

## Behaviour change

Requests the proxy cannot authenticate (no `HEADROOM_PROXY_TOKEN`,
remote caller) are now limited per client address. Changing the API-key
header no longer gives a new allowance. Authenticated and
direct-loopback callers keep one allowance per provider credential.
Behind a gateway, set `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` so each
forwarded client address gets its own limit; to keep one allowance per
credential there, callers must also send the proxy token.

The limiter is quota smoothing and abuse control for unauthenticated
callers. It is not a security boundary for authenticated callers, and
this PR does not make it one.

## Tests

- `tests/test_proxy/test_rate_limit_identity.py`: untrusted keys ignore
the credential; trusted keys are per credential and include the peer;
the whole credential is used, not a prefix; bucket names carry no
credential material; loopback is trusted; a request relayed by a
trusted-gateway peer (including one on loopback) is not trusted without
the proxy token; with `127.0.0.0/8` configured as a gateway, direct
loopback callers stay per credential, and a relayed request with an
unusable `X-Forwarded-For` still counts as relayed; IPv6 `/64` grouping;
IPv4-mapped addresses.
- `tests/test_proxy_openai_rate_limit_key.py`, end to end through
`create_app` with `--rpm 1`: authenticated and loopback callers with
distinct keys do not share a bucket; an unauthenticated remote caller
who rotates `api-key` or `Authorization`, or drops the credential, gets
429 on the second request; two different unauthenticated peers do not
share a bucket; through a trusted gateway (`10.0.0.2`, fixed
`X-Forwarded-For: 203.0.113.9`) a caller rotating `api-key` gets 429,
distinct forwarded clients do not share, and token-authenticated callers
stay per credential; with `127.0.0.0/8` as the gateway CIDR, two direct
loopback callers with different keys both pass while a caller relayed
through it with a fixed forwarded address gets 429 on the second key.
The unauthenticated cases fail on `main`.
- Existing rate-limiter and TPM suites pass unchanged.

Manual run: `headroom proxy --host 0.0.0.0 --rpm 2` with six requests
from a LAN address, each with a different `Authorization` key. On `main`
all six reach the upstream. On this branch without a proxy token, the
third and later get 429. With the proxy token sent, all six pass, one
bucket per credential.

Not tested: the Rust `headroom-proxy` binary, which has its own limiter.

## Merge order / conflicts

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. #3852 makes the untrusted case rare by refusing token-less
non-loopback binds; this PR makes it safe when an operator opts out of
that.

Textual conflict: #3862 adds an import next to the one this PR adds in
`handlers/anthropic.py` and `handlers/openai.py`. Whichever merges
second keeps both imports.

Revised 2026-10-01 after self-review: `rate_limiter.py` is back to
`main`; the separate trusted/untrusted pools, per-owner cap and overflow
bucket were removed.
Revised 2026-10-02 after review: gateway membership only drops the
credential from the key when the gateway actually forwarded a client
address, so direct localhost callers keep per-credential buckets under a
loopback gateway CIDR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3860"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:26:56 -05:00
Tejas ChopraandClaude Opus 5.5 7df8bd87b5 fix(proxy): label plain OpenAI chat traffic openai, not custom (#3912)
## Description

Three tests fail on `main`. Each of the PRs below passed CI on its own
branch; the failures come from how they combine.

- **Real bug (#3759 + #2693):** #3759 labels the chat outcome provider
with `CUSTOM_BASE_PROVIDER if upstream_base_url else "openai"`. That was
written against a handler where `upstream_base_url` held the
`x-headroom-base-url` override. #2693 had already renamed the override
to `custom_upstream_base_url` and made `upstream_base_url` the resolved
upstream, which is never empty. Since both merged, **every chat request
without a custom base is labelled `custom` instead of `openai`**. That
affects `/stats`, `/metrics`, OTel and provider-dependent savings
estimates. Not in any release yet.
- **Stale test (#3365):** #3365 keys OpenAI rate limits by an HMAC of
the credential. The flood test's stub limiter still compared against the
raw `Authorization` value, so it never refused and the "limited"
requests got 200 instead of 429. The code is right; the test is updated.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/openai.py`: the fallback label checks
`custom_upstream_base_url`.
- `tests/test_proxy_openai_cache_stability.py`: the stub limiter refuses
`_openai_rate_limit_key({"authorization": limited_auth})`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — one-identifier change
- [ ] New tests added for new functionality — the existing tests already
cover it and fail on `main`
- [x] Manual testing performed

### Test Output

```text
# on main 59b8cefc
FAILED tests/gateway/test_parity_proxy_vs_gateway.py::test_provider_bound_bytes_and_outcomes_match
FAILED tests/test_proxy_openai_cache_stability.py::test_openai_chat_outcome_provider_uses_the_fixed_taxonomy
FAILED tests/test_proxy_openai_cache_stability.py::test_openai_chat_custom_base_flood_cannot_grow_the_provider_set

# this branch: those files plus rate-limit, passthrough and Grok call-site tests
$ pytest tests/gateway/test_parity_proxy_vs_gateway.py tests/test_proxy_openai_cache_stability.py \
    tests/test_proxy_openai_rate_limit_key.py tests/test_proxy_passthrough.py tests/test_grok_shared_proxy_call_sites.py
49 passed
```

## Real Behavior Proof

- Environment: macOS, Python 3.14, `main` at 59b8cefc.
- Exact command / steps: bisected with #3759's head merged onto each
`main` commit. Taxonomy and parity tests first fail at ef1c528e (#2693);
the flood test first fails at b9e8462a (#3365).
- Observed result: with this change all three pass.
- Not tested: a live proxy's `/stats` output.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: restores the `openai` label for plain
OpenAI chat traffic
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
— not needed
- [ ] I have made corresponding changes to the documentation — not
needed
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works — existing tests fail without it
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Both PRs passed CI on their own branches because neither was retested
against the other. Requiring branches to be up to date before merge, or
a merge queue, would catch this class of break.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 14:25:56 -05:00
donjorandJerrettDavis f0ec2bb376 fix(telemetry): label custom-base chat upstreams from a fixed provider set (#3759)
## Description

Unrecognized custom-base chat upstreams report
`outcome_provider="openai"`.
Dashboards and per-provider stores merge every OpenAI-compatible vendor
except Zen into `openai`.

### Fix

Labels come from a fixed set in `custom_base_passthrough_telemetry()`:

| Custom base host | Label |
| --- | --- |
| `api.z.ai` | `zai` |
| `api.meta.ai` | `meta` |
| `api.openai.com` | `openai` |
| anything else | `custom` |

- **Match:** exact host, `chat/completions` path
- **Zen:** unchanged
- **No base header:** `openai`, as before
- **Bounded:** the hostname is never a label

Formatting and the SSRF guard are unchanged.
Pricing tables are unchanged; provider-dependent savings estimates can
change.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Fix:** `CUSTOM_BASE_CHAT_PROVIDERS`, exact host to label
- **Fix:** `CUSTOM_BASE_PROVIDER = "custom"` fallback
- **Fix:** chat handler reports the fixed-set label
- **Removed:** `_custom_base_provider_label()`
- **Test:** known hosts, subpath bases, host case
- **Test:** lookalike hosts, non-chat paths, bad URLs
- **Test:** `zai`, `meta`, `custom`, no-header control
- **Test:** 1,000-request flood across every sink

### Flood regression

Through the real chat handler:

1. 980 distinct hosts: ok, upstream 502, local 429, upstream 429
2. 20 `api.z.ai` requests: ok and local 429
3. Exact counts: in-memory, lifetime, OTel
4. `/metrics` labels: inside `{openai, zen, zai, meta, custom}`

The pre-seeded zero `unknown` failure bucket is the one extra label, on
`main` too.

## Testing

- [x] Unit tests pass (`pytest`): no new failures; see baseline below
- [x] Linting passes (`ruff check .`), plus `ruff format --check .`
- [x] Type checking passes (`mypy headroom`):
`--ignore-missing-imports`, mypy 1.20.2 as in CI
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
new tests against main c0292984 code:
FAILED tests/test_proxy_openai_cache_stability.py::test_openai_chat_custom_base_reports_zai_as_outcome_provider
FAILED tests/test_proxy_openai_cache_stability.py::test_openai_chat_outcome_provider_uses_the_fixed_taxonomy
FAILED tests/test_proxy_openai_cache_stability.py::test_openai_chat_custom_base_flood_cannot_grow_the_provider_set
FAILED tests/test_proxy_passthrough.py::test_custom_base_passthrough_telemetry_names_known_chat_hosts
FAILED tests/test_proxy_passthrough.py::test_custom_base_passthrough_telemetry_requires_a_chat_path_segment
E   AssertionError: assert {'openai': 260} == {'zai': 10, 'custom': 250}
5 failed, 13 passed

branch: 18 passed

pytest, full suite:
  main c0292984   11 failed, 12975 passed, 580 skipped
  branch          11 failed, 12981 passed, 580 skipped
same 11 fail on both:
  tests/test_proxy_health.py (4 readyz Kompress tests)
  tests/test_read_maturation_handler_nobust.py (3)
  tests/test_upstream_guard.py::test_no_route_forwards_to_a_loopback_upstream
  tests/test_cli_proxy_improvements.py::TestMissingProxyDepsError (2)
  tests/test_image_compression.py::TestOnnxRouter::test_full_classify_with_image

ruff check .: All checks passed!
ruff format --check .: 1634 files already formatted
mypy 1.20.2 headroom --ignore-missing-imports: Success: no issues found in 539 source files
```

## Real Behavior Proof

- Environment: Linux, Python 3.12.3, model `glm-5`
- Exact command / steps: see Commands below
- Observed result: see table
- Not tested: live OTel collector; `api.meta.ai` live

### Setup

1. **Trees:** `main` c0292984, this branch
2. **Proxy:** one per tree, own workspace
3. **Allowlist:** every host in `HEADROOM_ALLOWED_BASE_URLS`

The SSRF guard drops a base header whose host does not resolve and falls
back to `openai`.
The flood hosts do not resolve, so without the allowlist they would
never reach the label code.

### Commands

Per tree, from its checkout:

```bash
hosts="api.z.ai,$UPSTREAM_HOST"
for i in $(seq -f %04g 1 100); do hosts="$hosts,flood-$i.label-flood.test"; done
HEADROOM_ALLOWED_BASE_URLS="$hosts" HEADROOM_WORKSPACE_DIR=ws PYTHONPATH=. \
  headroom proxy --port 8787 --log-file ws/requests.jsonl
```

Traffic:

```bash
chat() {  # chat [base-url] [key] [content]
  curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8787/v1/chat/completions \
    -H 'content-type: application/json' \
    ${1:+-H "x-headroom-base-url: $1"} ${2:+-H "authorization: Bearer $2"} \
    -d '{"model":"glm-5","messages":[{"role":"user","content":"'"${3:-provider label check}"'"}]}'
}
chat https://api.z.ai/api/coding/paas/v4        # 401, no key
chat "$UPSTREAM_BASE_URL" "$UPSTREAM_KEY"       # 200, OpenAI-compatible
chat "" sk-control                              # 200, Headroom response-cache hit
for i in $(seq -f %04g 1 100); do chat "https://flood-$i.label-flood.test/v1" sk-flood flood; done  # 502
curl -s http://127.0.0.1:8787/stats | jq '.requests | {by_provider, failed_by_provider}'
curl -s http://127.0.0.1:8787/metrics | grep -o 'provider="[a-z]*"' | sort | uniq -c
```

### Result

| | `main` c0292984 | this branch |
| --- | --- | --- |
| `by_provider` | `openai` 2 | `custom` 1, `openai` 1 |
| `failed_by_provider` | `openai` 101, `unknown` 0 | `custom` 100, `zai`
1, `unknown` 0 |
| `/metrics` provider series | 13 `openai`, 1 `unknown` | 13 `custom`, 1
`zai`, 1 `openai`, 1 `unknown` |

On `main`, z.ai, the real upstream and all 100 flood hosts count as
`openai`.
On this branch, z.ai stays `zai` and the flood lands in one `custom`
bucket.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, telemetry only. Custom-base chat
reports `zai`, `meta`, `openai` or `custom` instead of always `openai`.
Routing, auth, pricing tables and the SSRF guard are unchanged;
provider-dependent savings estimates can change.
- Kill switch / disable path: None dedicated.
- Unsafe override required: No.
- Qualification impact: None. Dashboards keyed on `openai` will see
`zai`, `meta` and `custom` series.
- Rollback path: Revert.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (no new
failures)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

**main**


https://github.com/user-attachments/assets/3a0790f0-36a2-4906-ab3b-049273547fa5

**this branch**


https://github.com/user-attachments/assets/c4bc678c-4c59-47d2-94b2-2e942a9e061c

## Additional Notes

- **Closes:** none; no issue tracks this
- **Related:** #2626, same convention for `xai`
- **Docs:** unchecked; no documented behavior
- **Trade-off:** proxied z.ai reports `custom`
- **Baseline:** 11 fail on `main` too; listed above

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 12:17:49 -05:00
donjorandJD Davis deca575c63 fix(cache): preserve the forwarded OpenAI chat prefix across turns (#3757)
## Description

In cache mode, `/v1/chat/completions` changes its forwarded prefix every
turn, so the provider's prompt cache misses from last turn's tool result
onward.

Two causes, and each one alone busts the prefix:

### Tracker originals (all modes)

- **Write:** tracker writes omit `original_messages`
- **Stored:** compressed bytes, as if the client's
- **Next turn:** `finalize_turn` finds no match
- **Replay:** none

### Restore after replay (cache mode)

- **Order:** restore runs after `finalize_turn`
- **Effect:** replayed bytes revert to raw

### Fix

- **Originals:** passed on every chat tracker write
- **Order:** restore, then `finalize_turn` (cache)
- **Token mode:** shared replay now sees originals

Folds in #3758: the live run below showed neither half fixes the prefix
alone.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Fix:** restore, then `finalize_turn`
- **Fix:** originals on every chat tracker write
- **Test:** 3-turn lifecycle, buffered and stream
- **Test:** originals reach the tracker, per path
- **Test:** replay survives restore
- **Test:** partial replay, block append, divergence

## Testing

- [x] Unit tests pass (`pytest`): no new failures; 8 fail on `main` too
- [x] Linting passes (`ruff check .`), plus `ruff format --check .`
- [x] Type checking passes (`mypy headroom`):
`--ignore-missing-imports`, mypy 1.20.2 as in CI
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
test_openai_cache_mode_forwards_last_turns_bytes_unchanged[buffered|stream]:
  main d17ac6a1       2 failed
  restore fix only    2 failed
  originals fix only  2 failed
  both                2 passed
E     At index 2 diff: {'role': 'tool', 'tool_call_id': 'call_1', 'content': 'build 1 log line ...
E     -         'content': '[compressed call_1]',

pytest, full suite:
  main d17ac6a1   8 failed, 12961 passed, 591 skipped, 6137 warnings
  branch          8 failed, 12972 passed, 591 skipped, 6137 warnings
same 8 fail on both:
  tests/test_proxy_health.py (4 readyz Kompress tests)
  tests/test_read_maturation_handler_nobust.py (3)
  tests/test_upstream_guard.py::test_no_route_forwards_to_a_loopback_upstream

ruff check .: All checks passed!
ruff format --check .: 1634 files already formatted
mypy 1.20.2 headroom --ignore-missing-imports: Success: no issues found in 539 source files (same on main)
```

## Real Behavior Proof

- Environment: Linux, Python 3.12.3, `gpt-6-sol`, Kompress on ONNX
Runtime
- Exact command / steps: see Commands below
- Observed result: see table
- Not tested: `/v1/responses`; token mode live (shares the originals
change); backend paths live

### Setup

1. **Trees:** `main`, restore, originals, both
2. **Proxy:** one per tree, own workspace
3. **Client:** `repro-3757.py` (stdlib), below
4. **Head:** ~9 KB system prompt
5. **Salt:** unique per run: new session, cold cache

### Commands

Per tree, from its checkout:

```bash
HEADROOM_WORKSPACE_DIR=ws PYTHONPATH=. headroom proxy --mode cache --port 8787 --log-file ws/requests.jsonl --log-messages
```

Client, buffered then streaming:

```bash
export OPENAI_API_KEY=... OPENAI_BASE_URL=...  # upstream, sent as x-headroom-base-url
python3 repro-3757.py --proxy http://127.0.0.1:8787 --model gpt-6-sol --log ws/requests.jsonl
python3 repro-3757.py --proxy http://127.0.0.1:8787 --model gpt-6-sol --log ws/requests.jsonl --stream
```

### `repro-3757.py`

```python
"""Send a 4-turn tool-calling chat through a headroom proxy; print usage per turn.

Each turn adds a user ask, an assistant tool call, and a ~48 KB JSON log result
behind a ~9 KB system prompt. --salt leads the system prompt, so each run is its
own headroom session and a cold provider cache. With --log (the proxy's
--log-file, run with --log-messages), each turn also compares the forwarded
messages with last turn's.
"""

import argparse, json, os, random, time, urllib.request, uuid

SERVICES = ["checkout", "billing", "search", "inventory", "auth", "shipping", "ledger", "notify"]
TOOL = {"type": "function", "function": {
    "name": "read_log", "description": "Read one service's structured request log.",
    "parameters": {"type": "object", "properties": {"service": {"type": "string"}},
                   "required": ["service"]}}}


def system_prompt(salt):
    rules = (f"Rule {i:03d}: when a {SERVICES[i % 8]} request exceeds its latency budget, check "
             "the upstream pool, the retry policy, and the last deploy before paging on-call."
             for i in range(60))
    return f"Run {salt}. You are an SRE assistant reviewing service logs.\n" + "\n".join(rules)


def tool_log(turn):
    rng, svc = random.Random(3757_3758 + turn), SERVICES[turn % 8]
    rows = []
    for i in range(300):
        err = i in (41, 173, 260)
        rows.append({
            "ts": f"2026-09-25T10:{(i // 60) % 60:02d}:{i % 60:02d}.{rng.randint(0, 999):03d}Z",
            "service": svc, "level": "ERROR" if err else "INFO",
            "route": f"/api/v1/{svc}/{rng.choice(['get', 'list', 'update'])}",
            "status": 502 if err else 200,
            "latency_ms": rng.randint(900, 4000) if err else rng.randint(8, 60),
            "msg": f"upstream reset by peer (pool={svc}-db)" if err else "request ok"})
    return json.dumps(rows)


def forwarded(log, n):
    for _ in range(40):  # streaming logs just after [DONE]
        lines = open(log).read().splitlines() if os.path.exists(log) else []
        if len(lines) >= n:
            entry = json.loads(lines[n - 1])
            return entry.get("compressed_messages") or entry.get("request_messages"), entry
        time.sleep(0.25)
    return None, {}


def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--proxy", required=True, help="headroom URL, e.g. http://127.0.0.1:8787")
    ap.add_argument("--model", required=True)
    ap.add_argument("--salt", default=uuid.uuid4().hex[:12], help="default: random")
    ap.add_argument("--stream", action="store_true", help="stream, with include_usage")
    ap.add_argument("--turns", type=int, default=4)
    ap.add_argument("--upstream", default=os.environ.get("OPENAI_BASE_URL"),
                    help="sent as x-headroom-base-url (env: OPENAI_BASE_URL)")
    ap.add_argument("--api-key", default=os.environ.get("OPENAI_API_KEY"), help="env: OPENAI_API_KEY")
    ap.add_argument("--log", help="proxy --log-file path, for the prefix check")
    a = ap.parse_args()
    headers = {"authorization": f"Bearer {a.api_key}", "content-type": "application/json"}
    if a.upstream:
        headers["x-headroom-base-url"] = a.upstream
    msgs = [{"role": "system", "content": system_prompt(a.salt)}]
    base = len(open(a.log).read().splitlines()) if a.log and os.path.exists(a.log) else 0
    prev = None
    for turn in range(1, a.turns + 1):
        svc, cid = SERVICES[turn % 8], f"call_{turn:02d}"
        msgs += [{"role": "user", "content": f"Turn {turn}: check the {svc} log. Reply in one short sentence."},
                 {"role": "assistant", "content": None, "tool_calls": [{"id": cid, "type": "function",
                  "function": {"name": "read_log", "arguments": json.dumps({"service": svc})}}]},
                 {"role": "tool", "tool_call_id": cid, "content": tool_log(turn)}]
        body = {"model": a.model, "messages": msgs, "tools": [TOOL], "tool_choice": "none"}
        if a.stream:
            body |= {"stream": True, "stream_options": {"include_usage": True}}
        req = urllib.request.Request(f"{a.proxy.rstrip('/')}/v1/chat/completions",
                                     json.dumps(body).encode(), headers)
        with urllib.request.urlopen(req, timeout=300) as resp:
            text = resp.read().decode()
        if a.stream:
            chunks = [json.loads(l[5:]) for l in text.splitlines()
                      if l.startswith("data:") and l[5:].strip() not in ("", "[DONE]")]
            reply = "".join((c.get("delta") or {}).get("content") or ""
                            for ch in chunks for c in ch.get("choices") or [])
            usage = next((ch["usage"] for ch in reversed(chunks) if ch.get("usage")), {})
        else:
            data = json.loads(text)
            reply, usage = data["choices"][0]["message"].get("content") or "", data.get("usage") or {}
        msgs.append({"role": "assistant", "content": reply})
        cached = (usage.get("prompt_tokens_details") or {}).get("cached_tokens", 0)
        line = f"turn {turn}  prompt {usage.get('prompt_tokens', 0):>7,}  cached {cached:>7,}"
        if a.log:
            sent, entry = forwarded(a.log, base + turn)
            if not sent:
                state = "unavailable (no log entry)"
            elif prev is None:
                state = "first turn"
            else:
                diff = next((i for i, (x, y) in enumerate(zip(prev, sent)) if x != y),
                            None if len(sent) >= len(prev) else len(sent))
                state = f"changed at msg {diff}/{len(prev)}" if diff is not None else f"stable {len(prev)}/{len(prev)}"
            line += f"  saved {entry.get('tokens_saved')}  prefix {state}"
            prev = sent
        print(line, flush=True)


if __name__ == "__main__":
    main()
```

### Each turn

1. **Ask:** user message
2. **Tool:** call returns ~48 KB JSON log
3. **Cached:** `prompt_tokens_details.cached_tokens`
4. **Saved:** `tokens_saved`, proxy log
5. **Prefix:** forwarded vs last turn's

### Result, turn 4, buffered

| | `main` d17ac6a1 | restore only | originals only | both |
| --- | --- | --- | --- | --- |
| Prompt tokens | 69,848 | 69,862 | 69,856 | 42,944 |
| Cached tokens | 40,448 (58%) | 40,448 (58%) | 40,448 (58%) | 32,768
(76%) |
| Tokens saved | 9,214 | 9,214 | 9,214 | 37,226 |
| Forwarded prefix | changed at msg 11/12 | changed at msg 11/12 |
changed at msg 11/12 | stable 12/12 |

Turn-4 streaming results: same cached tokens, saved tokens, and prefix
result; prompt tokens within 18.
Earlier turns differ between modes, e.g. `main` turn 2 cached 2,048
buffered vs 0 streaming.

In this workload, unchanged history that the overlay validates stays
byte-stable.
Divergent history is not replayed, by design.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, OpenAI chat only. All modes: the
prefix tracker stores client originals, so the shared replay can match
them. Cache mode only: restore runs before replay.
- Kill switch / disable path: None dedicated.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (no new
failures)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

**main**


https://github.com/user-attachments/assets/46dcd2a1-7246-4d0b-aa33-256bbe5109da

**both fixes** (`headroom/` identical to this PR's head)


https://github.com/user-attachments/assets/5c687b18-93cd-4c1a-a675-72deb6579cac

## Additional Notes

- **Closes:** none; no issue tracks this bug
- **Related:** #3391, other replay gaps; not fixed here
- **Docs:** unchecked; no documented behavior
- **Baseline:** 8 fail on `main` too; listed above

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-24 16:16:58 -05:00
gglucassandClaude Fable 5 aebe98954e fix(cache): keep cached-prefix replay after background recompression (#3380)
## Description

Since the non-inflation size bound added in #3052 (shipped 0.36.x), the
Anthropic proxy token path repeatedly busts the provider prompt cache:
the bound declines the cached-prefix replay exactly when background
compression lands a SMALLER form of an already-forwarded message, so the
forwarded bytes change mid-history and the provider cache misses from
the first changed byte. Steady state on a busy session is a
160-210k-token cache re-write every 1-2 requests (measured across a
0.35.0 -> 0.37.0 swap on identical workload: dollar cache-hit rate 90%
-> 52%, billable input $/M up 2.2x; full data in #3379).

Reworked per review (thanks @chopratejas - the direction is yours):
rather than disabling the size bound wholesale, `overlay_cached_prefix`
now takes `confirmed_frozen_count` and splits at that floor.

- Inside the provider-confirmed prefix: replay is unconditional. The
replay source is exactly what the provider hashed, so changing those
bytes can only bust the cache; an "inflating" replay there bills at the
~0.1x cache-read rate and is still the cheaper request.
- Beyond the floor: the size bound arbitrates each turn exactly as on
main - a shrinking replay repairs freeze drift (#1850), an inflating one
is declined so a fresh compression improvement reaches the wire.
- When the provider count collapses (cold cache, TTL lapse) the floor
collapses with it and every accumulated improvement lands at once: the
natural re-baselining that bounds long-session growth (the #3026
direction of travel) by the TTL-lapse cadence instead of pinning
forever.

The #3052 alignment/append-only guards stay enforced unconditionally.

The OpenAI chat path now carries the same floor (reviewer question: is
the root cause live there too?). It is: the tracker is provider-generic
and is fed OpenAI's own `prompt_tokens_details.cached_tokens`, the
forwarded-bytes feedback loop is identical, both handlers compress
through the same ContentRouter, and OpenAI's automatic prompt caching
busts on the first changed byte exactly like Anthropic's. Exposure is
narrower - the token path's `FREEZE_POLICY_REPLAYABLE` already stops
most recompression of already-forwarded messages - but not zero: an
entry evicted from the local byte-replay cache drops out of that freeze,
and cache mode freezes from the tracker only. Remaining callers with no
provider-confirmed count in scope (sidecar, `openai.py:9940`) pass
nothing and keep today's fully size-bounded behavior; `bedrock.py` /
`gemini.py` never call `finalize_turn` at all, tracked in #3391.

Closes #3379

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `overlay_cached_prefix` gains keyword-only `confirmed_frozen_count:
int | None = None`: an unconditional-replay floor for the
provider-confirmed prefix; the size bound still governs everything
beyond it, and alignment guards are untouched.
- `finalize_turn` forwards the floor (default `None`, so all existing
callers are unchanged).
- The OpenAI chat call site passes the same floor, snapshotted before
`prepare_turn`'s replayable clamp (that count runs over the LOCAL
byte-replay cache and can exceed what the provider confirmed; a cold
prefix zeroes it, so cold re-baselining matches Anthropic's).
- The Anthropic confirmed-clamp call site stashes the pre-clamp tracker
count (the post-`prepare_turn` min under-floors when the local
byte-replay cache evicted entries) and passes it; the stash defaults to
0 on paths that never compute a tracker count (backpressure).
- Tests: floor replays a recompressed confirmed prefix; improvement
beyond the floor lands while the confirmed region replays;
originals-drift beyond the floor is still repaired when the replay
shrinks; alignment guards unaffected by the floor; block-append within
the floor; `finalize_turn` pass-through; the reviewer-requested
multi-turn harness test - a background improvement to an
already-forwarded message never busts the warm cache across 8 turns and
lands on the first cold turn; the reviewer-requested growth bound -
forwarded size grows by only its own delta each warm turn and the first
cold turn re-baselines to exactly the fresh pipeline's output; and an
end-to-end OpenAI chat test asserting the provider-confirmed prefix is
replayed even when the replay inflates (fails without the new argument).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py -q
============================== 58 passed in 0.44s ==============================
$ uvx ruff check headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py headroom/proxy/handlers/anthropic.py tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py
All checks passed!
$ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.12; regression measured on the
released headroom-ai 0.37.0 under real Claude Code traffic through
`headroom proxy` (token mode); this branch validated with the repo test
suite.
- Exact command / steps: diffed per-request PERF logs across a 0.35.0 ->
0.37.0 swap on the same machine and workload (sequences and aggregates
in #3379); then ran the pytest/ruff/mypy commands above on this branch.
A deployment carrying the unconditional-replay behavior on the confirmed
prefix restored the healthy pattern live: the broken wheel capped cached
reads at 157k across 32 large requests (prefix perpetually re-written)
while the patched deployment accumulated to 305k cached reads within 15
minutes.
- Observed result: 58 passed including the new multi-turn harness test;
on stock 0.37.0 every background-compression completion produced a
cache_read collapse to the ~7.7k tools+system head plus a 160-210k
cache_write.
- Not tested: a live OpenAI `cached_tokens` comparison for the OpenAI
half (the Anthropic measurement is what #3379 carries; the OpenAI change
is argued from shared mechanism plus an end-to-end handler test, not
from its own dollar data). A live provider run of this exact branch (the
wheel comparison plus the live patched-deployment measurement isolate
the bound as the trigger; the unit and multi-turn regressions reproduce
decline and fix deterministically). `mypy headroom` was run only on the
two touched modules, not the whole package.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: On the Anthropic proxy
confirmed-clamp path and the OpenAI chat path, both of which now replay
the provider-confirmed prefix unconditionally (the pre-0.36 posture)
while keeping the size bound beyond the confirmed floor. The overlay's
default and every other caller keep the fully size-bounded behavior.
- Kill switch / disable path: remove the `confirmed_frozen_count=`
argument at the two call sites (anthropic.py, openai.py chat); the
parameter defaults to `None`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the commits; the parameter is additive and
default-preserving.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Lineage for review: #1850 made freeze forward the cached prefix
byte-identical, #3052 added alignment guards plus the size bound after
#3026's misalignment blowup, #3271 unified both modes on the session
engine. The first cut of this PR disabled the bound on the
confirmed-clamp path wholesale; the reviewed rework scopes unconditional
replay to the provider-confirmed prefix, which additionally gives
cold-lapse re-baselining (improvements land whenever the floor
collapses), addressing the growth analysis raised in review. Economics
discussion in the review thread.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:50:58 +05:30
Rod Boev c16be9bbbe fix(proxy/anthropic): don't replay recorded prefix over live history (#3026) (#3052)
## Description

A Claude Code session that reads a large tool result through `headroom
proxy` can fail on turn 2 with Anthropic's `400 prompt_too_long`. The
reporter's controlled comparison completed eight turns through 0.33.0
with 187,986 input tokens, while 0.35.0 failed after five requests with
753,077 input tokens. The local regression uses an actual prior
optimized request to populate tracker state, then a decision-false
bypass turn with Claude-shaped tool-result content. The old
unconditional replay path substitutes the compressed prefix; the
eligibility gate preserves the client's outbound body without claiming a
live provider reproduction.

The Anthropic `/v1/messages` route computes whether a request should be
compressed, but cached-prefix replay currently runs outside that
decision. The replay helper also derives its prefix length from the
original message list and applies that index to the optimized list
without proving the two lists still align. A stale forwarded prefix can
therefore be grafted onto the wrong positions and enlarge later
requests.

This change limits replay to requests whose existing compression
decision permits it and whose pre-upstream backpressure path is
inactive. It also makes `overlay_cached_prefix()` decline misaligned or
inflating candidates while preserving normal append-only replay.

Reported by @itsumonotakumi, whose controlled comparison isolated the
failure from compression, headers, one-request serialization, memory,
code graph, and CCR.

Closes #3026

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Gate Anthropic cached-prefix replay on the existing
`CompressionDecision.should_compress` result and the existing
pre-upstream backpressure state.
- Require positional alignment between optimized and original message
arrays before replay.
- Reject replay candidates that would serialize larger than the current
optimized messages.
- Add focused handler coverage for the decision-false tool-result
regression, bypass and backpressure paths, and outbound optimize-on
preservation.
- Add direct unit coverage for positional mismatch, no-inflation, and
JSON sizing-failure bailouts.
- Update the moved-cache-control and pure-block-append regression
fixtures to keep the no-inflation contract explicit.
- Run the unchanged OpenAI cache-stability preservation proof; no OpenAI
production code was edited.

## Testing

- [x] Unit tests pass (153 focused proxy, helper, cache-control,
block-append, cross-turn, byte-faithful, Anthropic, OpenAI, and
backpressure tests)
- [x] Linting passes (Ruff check and format validation on the seven
changed repository files)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed with the in-process proxy and local stub
upstream

### Test Output

```text
python -m pytest tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_anthropic_cache_stability.py tests/test_anthropic_pre_upstream_backpressure.py -q
python -m pytest tests/test_proxy_openai_cache_stability.py -q
python -m pytest tests/test_issue_2671_block_growth_cache.py::test_pure_append_replays_forwarded_blocks_and_advances_breakpoint -q
153 passed across focused invocations, exit code 0
optimize_off turn2_message_count=3 marker_count=1 outbound_compact_utf8_bytes=2293 client_compact_utf8_bytes=2293
optimize_on turn2_message_count=3 client_message_count=3 marker_count=0 outbound_compact_utf8_bytes=171 client_compact_utf8_bytes=182
python -m ruff check headroom/proxy/handlers/anthropic.py headroom/cache/prefix_tracker.py tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_proxy_openai_cache_stability.py
All checks passed!, exit code 0

python -m ruff format headroom/proxy/handlers/anthropic.py headroom/cache/prefix_tracker.py tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_proxy_openai_cache_stability.py --check
7 files already formatted, exit code 0

git diff --check
clean, exit code 0
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12 via `uv`, real Headroom proxy app
with a local stub Anthropic upstream
- Exact command / steps: send an actual optimize-on first request
through the in-process proxy with a deterministic production-pipeline
seam, then send a decision-false bypass turn containing a large
Claude-shaped `tool_result` with moved `cache_control`; separately send
an aligned optimize-on turn with a new suffix
- Observed result: the exact base checkout fails with `AssertionError:
assert 'compressed-tool-result' == 'large-tool-result-marker ...'`; the
guarded path passes with the client marker present once and outbound
compact JSON no larger than the client body. The optimize-on
preservation run records `optimize_on turn2_message_count=3
client_message_count=3 marker_count=0 outbound_compact_utf8_bytes=171
client_compact_utf8_bytes=182`, proving the actual compressed prefix is
outbound before the new suffix without turn-2 growth.
- Not tested: live Claude Code session against api.anthropic.com on this
host

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: cached-prefix replay now follows the
existing compression and backpressure decision and rejects misaligned or
inflating candidates.
- Kill switch / disable path: no new switch; the existing optimize and
bypass controls remain available.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the implementation commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes

## Additional Notes

`CHANGELOG.md` is not modified because Headroom's release automation
generates it from conventional commits.

This change does not add a context-limit guard or alter compression,
streaming tracker provenance, outbound-body selection, OpenAI behavior,
or provider limits. Local tests prove request-body ownership and replay
bounds. The reporter's live Claude Code completion and Anthropic token
acceptance remain external to this local proof.
2026-08-17 15:02:18 -07:00
vscunha 05932d7165 fix(proxy): compress OpenCode tool schemas and embedded JSON (#1535)
## Description

Fixes two remaining OpenCode/OpenAI Chat compression gaps after `main`
incorporated the original savings-profile threading and user
content-block work from this PR.

OpenCode requests can still report very low savings when most input
tokens live in verbose `tools` schemas rather than messages. They can
also route poorly when a short instruction wraps a valid JSON block but
does not satisfy the existing long-prose heuristic.

Closes #1534

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that causes existing functionality
to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Compact OpenAI Chat Completions `tools` schemas whenever request
compression is active, reusing the existing OpenAI Responses schema
compactor. The outbound tool invocation shape is preserved while
non-semantic annotations such as `$schema`, `title`, and `examples` are
removed.
- Include the tool-schema token delta in Headroom's savings accounting
and expose `openai:chat:tool_schema_compaction` in the applied
transforms.
- Detect valid JSON blocks surrounded by prose or log text as mixed
content, so short OpenCode instructions route through mixed/SmartCrusher
handling instead of falling through or producing a no-op.
- Adapt the mixed-content change to the new
`headroom.transforms.mixed_content` module introduced on `main` by
#1939.

## Why the Focus Changed

The original headline fix—threading savings-profile kwargs into
`/v1/chat/completions`—is now already present on `main`, as is the user
content-block opt-in behavior. Those duplicate changes were removed
during the merge.

The branch also no longer changes developer/system role protection or
forced-Kompress semantics. It follows `main` for both, so the earlier
instruction-role safety concern is outside the current diff.

The resulting PR is limited to two OpenCode-specific compression gaps
that remain reproducible on current `main`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` on changed files)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for the fixed behavior
- [ ] Manual live-upstream testing performed after the latest rebase

### Test Output

```text
59 passed, 1 warning in 83.53s
All checks passed!  # ruff check
4 files already formatted  # ruff format --check
python -m py_compile: passed
git diff --check: passed
```

Focused test coverage includes:

- OpenAI Chat tool-schema compaction, transform reporting, outbound
schema shape, and positive token savings.
- Embedded JSON mixed-content detection, SmartCrusher routing, positive
savings, and preservation of a critical sentinel value.
- Current `main` regressions for savings-profile threading, user content
blocks, turn hooks, and forced-Kompress behavior.

## Real Behavior Proof

- Environment: Linux ARM64, Python 3.13.12, current `main` at `9bacf481`
merged into the branch.
- Exact command / steps: focused pytest run across the OpenAI
cache-stability, content-router, mixed-content, savings-profile,
user-block, turn-hook, and forced-Kompress suites.
- Observed result: 59 tests passed; the chat request test forwarded
compacted tools and reported positive savings, while the embedded-JSON
fixture used mixed routing and preserved `CRITICAL_NEEDLE_42`.
- Not tested: full repository suite and a live external OpenCode request
after the latest merge; those remain for CI/live follow-up.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented the non-obvious behavior
- [ ] I have made corresponding documentation changes — N/A; internal
routing behavior only
- [x] My changes generate no new warnings
- [x] I have added tests that prove the fixes are effective
- [x] New and existing focused tests pass locally
- [ ] I have updated the changelog — N/A; release automation handles fix
entries

## Screenshots

N/A — proxy/transform behavior only.

## Additional Notes

- Current diff versus `main`: 4 files, 172 insertions, no role-policy or
forced-Kompress changes.
- The mixed-content conflict was resolved by extending the new isolated
parser module rather than reintroducing parsing code into
`ContentRouter`.
2026-07-15 19:58:42 +00:00
JD Davis 55efb1c77d fix(proxy): keep OpenAI tool observations mutable in cache mode (#1884)
## Description

Diagnoses and fixes the low-savings OpenAI-compatible cache-mode path
reported in #1696.

OpenAI-compatible tool-calling clients can end a turn with `role:
"tool"` (or legacy `role: "function"`) rather than `role: "user"`. The
OpenAI chat handler's cache-mode freeze boundary treated those tails as
non-mutable, and because `HeadroomProxy` resolves
`_strict_previous_turn_frozen_count` from the Anthropic mixin first, the
OpenAI-specific helper was not used in production. That froze the entire
conversation before `ContentRouter` ran, leaving no live tool
observation to compress and producing near-pass-through savings on long
coding sessions.

This PR keeps final OpenAI tool/function observations mutable in cache
mode, explicitly calls the OpenAI helper to avoid the mixin-name
collision, and clamps negative token-savings artifacts at the
metrics/cost aggregation boundary so stats cannot under-report actual
forwarded savings.

Closes #1696

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Treat final OpenAI `user`, `tool`, and `function` messages as the
mutable cache-mode live zone.
- Route OpenAI cache-boundary calls through
`OpenAIHandlerMixin._strict_previous_turn_frozen_count` explicitly so
the Anthropic mixin method cannot shadow it in `HeadroomProxy`'s MRO.
- Preserve cache-mode live-tail boundaries even when compression-cache
state would otherwise freeze the whole request.
- Clamp negative `tokens_saved` artifacts in `CostTracker.record_tokens`
and `PrometheusMetrics.record_request`.
- Add regression coverage for OpenAI final `tool`/`function` tails,
over-frozen tracker state, and non-negative savings aggregation.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ maturin build --profile ci --out dist --interpreter python
Built wheel for abi3 Python >= 3.10 to dist\headroom_ai-0.29.0-cp310-abi3-win_amd64.whl

$ python -m pytest tests\test_proxy_handler_helpers.py tests\test_proxy_openai_cache_stability.py tests\test_observability_metrics.py tests\test_cost_tracker_counterfactual.py
49 passed in 10.27s

$ python -m ruff check .
All checks passed!

$ python -m mypy headroom
Success: no issues found in 407 source files

$ python -m pytest
53 failed, 7703 passed, 488 skipped, 5893 warnings, 131 errors in 595.18s (0:09:55)
```

Full-suite note: the full local `pytest` run was attempted on
Windows/Python 3.13 after building `headroom._core`. It did not complete
green due to broad pre-existing/local-environment failures outside this
change area, dominated by SQLite/memory persistence permission/path
errors plus unrelated adapter/cache/tool tests. The focused regression
suite for this PR passes, and repo-level lint/type gates pass.

## Real Behavior Proof

- Environment: Windows, Python 3.13.13, Rust/Cargo available, local
`headroom._core` wheel built with `maturin build --profile ci`.
- Exact command / steps: ran the OpenAI cache-stability tests with final
`role: "tool"` and `role: "function"` chat tails.
- Observed result:
`test_openai_cache_mode_keeps_final_tool_observation_mutable[tool]` and
`[function]` pass, proving the pipeline receives `frozen_message_count
== 2` for a 3-message request instead of freezing all 3 messages.
- Not tested: live Lemonade/KiloCode upstream session; no local Lemonade
Server was available.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

Docs and CHANGELOG are N/A for this narrow proxy bug fix. The broad
local `pytest` checkbox is intentionally left unchecked because the full
suite had unrelated local-environment failures; see the test output
above. Focused regression tests, `ruff check .`, and `mypy headroom` are
green.
2026-07-09 07:51:01 -07:00
Tejas Chopra 248ae0f3e0 fix(proxy): freeze must forward cached (compressed) prefix byte-identical — stop token-mode cache busting (#1850)
The freeze path (both providers) emits the agent's ORIGINAL bytes for a
frozen message, but the provider cached whatever we FORWARDED last turn
(the compressed form). Forwarding original then mismatches the cached
prefix and busts it from that point — re-creating the whole suffix.
Measured on a real SWE-bench run: 100% of attributed misses were
prefix_change, ~56% of ALL cache-writes were bust-induced (2.8M tokens),
driving cache_create +150% and cost +41% vs baseline.

Cache mode already avoided this via _extract_cache_stable_delta (replay
the previously-forwarded prefix, compress only the delta). Token mode
called apply(frozen_count) directly, which forwards original for the
frozen region.

Fix: add a shared, provider-agnostic overlay_cached_prefix() that
replays the previously-forwarded (cached, compressed) prefix
byte-identical, append-only guarded and idempotent, and apply it in BOTH
the Anthropic and OpenAI handlers right before forwarding. This makes
freezing byte-identical in every mode, so the only remaining difference
between "token" and "cache" mode is how large a mutable
(still-compressible) tail each leaves — not whether the frozen prefix
busts the cache.

Tests:
- test_cache_prefix_overlay.py: the helper (replay, append-only guard,
idempotence).
- test_cross_turn_cache_safety.py: the invariant that was missing —
drive the REAL tracker + freeze + overlay over multiple append-only
turns against a simulated provider prefix cache and assert the forwarded
prefix stays byte-identical turn-over-turn. Load-bearing: it fails
(detects the bust) without the overlay.

## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
2026-07-06 14:54:39 -07:00
chopratejas f0dcc02775 fix: A3 — byte-faithful Python forwarders; serialize canonical only when mutated
Eliminates P0-2 universally. Every Python forwarder (server.py
`_retry_request`, handlers/streaming.py `_stream_response`,
handlers/openai.py `_ws_http_fallback`, handlers/batch.py `_batch_passthrough`
+ batch-create + Google batch passthrough, handlers/anthropic.py CCR
continuation + batch endpoint) now switches from `httpx ... json=body` to
`httpx ... content=raw_bytes`. The default httpx JSON encoder was
re-serializing every request with `, `/`: ` separators and `\\uXXXX` ASCII
escapes — collapsing Anthropic prompt-cache hit-rate.

Forwarder strategy:
  - unmutated body → forward `await request.body()` verbatim;
  - mutated body  → re-serialize once via the new
    `serialize_body_canonical(body) -> bytes` helper (compact separators,
    `ensure_ascii=False`, dict insertion order preserved).

`HEADROOM_PROXY_PYTHON_FORWARDER_MODE` env var configures the mode:
  - `byte_faithful` (default) — the new behavior;
  - `legacy_json_kwarg` — explicit operator opt-in for emergency rollback.
Documented in `docs/content/docs/configuration.mdx`. NOT a fallback —
unknown values raise loudly per build constraint #4.

`BodyMutationTracker` accompanies each request through the handler so
transform sites mark the tracker (`memory_injection`,
`image_compression`, `compression_*`, `batch_compression`,
`ccr_continuation`, etc.). At forwarder dispatch we additionally compare
the final body dict against the parsed original bytes as a structural
safety net — any silent mutation we missed still triggers canonical
re-serialization.

A2 follow-up: `handlers/openai.py:534-540` (Chat Completions memory
injection) was prepending a system message; replaced with
`append_text_to_latest_user_chat_message`, the OpenAI Chat Completions
analog of `_append_context_to_latest_non_frozen_user_turn`. The cache
hot zone (system messages) is now sacrosanct on /v1/chat/completions
too. Honors `HEADROOM_MEMORY_INJECTION_MODE=disabled`.

Structured logging: every forwarder emits an `event=outbound_request`
log line with `forwarder`, `path`, `body_bytes`, `body_mutated`,
`mutation_reasons`, `source` (passthrough|canonical|legacy),
`request_id`. Never logs Authorization or full body.

`_read_request_json` factored to share `_read_request_body_bytes` with
new `read_request_json_with_bytes` so the anthropic handler can capture
both the parsed dict and the original (decompressed) bytes.

Tests:
  - `tests/test_proxy_byte_faithful_forwarding.py` (28 tests):
    SHA-256 byte-equality on /v1/messages and streaming, unicode
    preservation, numeric precision, mutation-tracker invariants,
    canonical-serializer properties, legacy-mode rollback, OpenAI
    Chat memory routing.
  - Existing test mocks updated to accept the new `**kwargs` on
    `_retry_request` (no behavior change).
  - `tests/test_proxy_handlers_batch.py` updated to read the captured
    `content=` bytes (formerly `json=`).
  - One A2 test corrected (`test_anthropic_tool_sort_and_context_append_helpers`)
    to match the live-zone-tail semantics introduced by A2.

Constraints satisfied: configurable env var; no new regex / hardcodes;
no silent fallback (`legacy_json_kwarg` is operator opt-in);
performant (`prepare_outbound_body_bytes` is O(1) for passthrough);
elegant single-responsibility helpers; structured tracing logs.
2026-05-02 09:02:10 -07:00
chopratejas 35eaf8de7f fix(proxy): remove content-keyed TTL walker that conflated content with positional cache (#327)
The Anthropic token-mode handler walked past prefix_tracker.frozen_message_count
whenever an upcoming tool_result's content-hash matched comp_cache._stable_hashes
or should_defer_compression returned True. That conflated content equality with
positional cache membership.

Anthropic's prefix cache is POSITIONAL: bytes 0..K cached, anything past K is
fresh. _stable_hashes is content-keyed and grows unbounded. In long Claude Code
sessions where tool_result content rhymes across turns (repeated system prompts,
repeated file reads, repeated tool descriptions), the walker advanced
frozen_message_count to len(messages) on every turn and the pipeline produced
transforms_applied=[] on 73% of requests in user SvenMeyer's reported session
(headroom-stats-2026-05-01.json: 74 of 101 eligible requests "prefix_frozen") —
even after the prior fix in 44944fb. The 15 requests that did compress averaged
21%, proving compression itself works when reached.

Fix: delete the walker. The freeze boundary is now

    frozen_message_count = min(
        prefix_tracker.frozen_message_count,    # positional ground truth
        comp_cache.compute_frozen_count(messages),  # local cache lower bound
    )

compute_frozen_count's use of _stable_hashes can only LOWER the freeze via the
min clamp, never raise it past prefix_tracker's value. For any position in the
gap [compute_frozen_count, prefix_tracker.frozen_count], recompressing produces
byte-stable output (compression is deterministic on input content), so
Anthropic's prefix cache stays valid.

Cross-handler verification:
* OpenAI handler (proxy/handlers/openai.py:358-382) does not have this walker
  — uses only compute_frozen_count. Codex routes through OpenAI handler. Both
  unaffected.
* Streaming and non-streaming both invoke anthropic_pipeline.apply() before the
  upstream call. One fix covers both paths.
* Cache mode (is_cache_mode) takes the _extract_cache_stable_delta path and is
  independent of the walker. Unaffected.

Tests: six new regression tests lock down the post-fix invariants — clamp to
min(prefix_tracker, compute_frozen_count); fresh tool_result whose hash matches
old _stable_hashes entry is not frozen; frozen prefix byte-stable across the
pipeline; 10-turn session produces non-empty compression suffix every turn;
streaming and non-streaming compute identical frozen_message_count; OpenAI
handler never calls the walker functions. Plus scripts/smoke_issue_327.py
(gated by RUN_LIVE_API=1) drives a 10-turn conversation against
api.anthropic.com in both shapes (string + list-of-blocks) and both modes
(streaming + non-streaming).

ci-precheck clean. 191 tests pass.

Follow-ups (separate PRs):
* Fix _cache perpetually empty (anthropic.py result.messages != working_messages
  comparison rarely fires in token mode).
* Cap _stable_hashes with bounded LRU + 1h TTL — hygiene only after the freeze
  gate is removed.
* List-shape tool_result content gates at content_router.py:1975 and
  intelligent_context.py:657 (cluster A from the audit).
2026-05-01 12:04:28 -07:00
JerrettDavis d00c6739e1 Fix CI regressions for cache benchmark work 2026-04-04 22:33:44 -05:00
JerrettDavis 83b730f2b9 Fix CI lint/format failures after proxy mode hardening 2026-04-04 14:42:32 -05:00
JerrettDavis 2625789a28 Harden cache-mode immutability for OpenAI and fix stats mode reporting 2026-04-04 14:36:29 -05:00