mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
99021af20a |
fix(proxy): make the Rust front's native Bedrock and Vertex routes opt-in (#4851)
## Description
Two routes in the Rust front (`crates/headroom-proxy`) attach the host's
own cloud credentials to whatever request reaches them, without checking
the caller:
- **Bedrock native** (`/model/{id}/invoke`, `/converse`,
`/invoke-with-response-stream`, `/converse-stream`) is on by default and
signs every request with SigV4 using the AWS credentials found on the
host.
- **Vertex** (`…/publishers/anthropic/models/{m}:rawPredict` /
`:streamRawPredict`) is mounted unconditionally, replaces the caller's
`Authorization` with a bearer from the host's Google ADC, and forwards
to the generic `--upstream`, whatever host that is. So with a non-Google
upstream (the Python proxy, another provider) the host's Google token is
sent there. `--vertex-adc-scope` was parsed but never used.
This PR makes both routes explicit opt-ins, only sends the ADC bearer to
a Google API host, and uses the configured scope.
**Dependency:** these routes still do not authenticate callers
themselves. Callers are expected to be authenticated by the inbound
`HEADROOM_PROXY_TOKEN` gate added in #4840 (which also moves the default
listen address to loopback). This PR is based on `main` and does not
require #4840 to merge first, but operators who enable either route on a
non-loopback address need #4840 to restrict who can use them.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `config.rs`: `--enable-bedrock-native` /
`HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` now defaults to `false`. New
`--enable-vertex-native` / `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE`
(default `false`) and `--vertex-allow-non-google-upstream` /
`HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` (default `false`).
`Config::for_test` keeps both routes on so the existing route tests run
unchanged.
- `proxy.rs`: the Vertex route is mounted only when enabled; with it
off, Vertex paths go through the catch-all with the caller's own headers
(as the Python proxy's Vertex passthrough expects). The ADC token source
is built with `--vertex-adc-scope`. The "Bedrock route not mounted"
startup line is now `info`, since off is the default.
- `vertex/raw_predict.rs`: before fetching the ADC bearer, the handler
refuses (502 `vertex upstream is not a Google API host`,
`vertex_upstream_not_google` error log) unless `--upstream` is
`https://` on a `*.googleapis.com` host (regional, global and
`*.p.googleapis.com` endpoints), or the operator set
`--vertex-allow-non-google-upstream=true` (local emulators, gateways).
No token is fetched on that path.
- `RUST_DEV.md`: the three flags in the configuration table, with what
enabling them means.
**Behaviour change for operators:** deployments that used the Rust
front's native Bedrock or Vertex signing must now pass
`--enable-bedrock-native true` / `--enable-vertex-native true` (or the
env vars). Vertex with a non-Google `--upstream` additionally needs
`--vertex-allow-non-google-upstream true`. Without the flags, those
paths are forwarded to `--upstream` unsigned with the client's own
credentials (Claude Code in Vertex mode keeps sending its own ADC token,
which is what `docs/content/docs/claude-code-vertex.mdx` describes).
`scripts/qualify_cargo_dependencies.py` already passes
`--enable-bedrock-native true`.
## Testing
- [x] `cargo test -p headroom-proxy` (45 test binaries, 596 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes
New tests in `tests/integration_cloud_routes_opt_in.rs` build the config
from the real CLI defaults (`CliArgs` → `Config::from_cli`), give the
app host AWS credentials and a host ADC bearer, and record what a
`wiremock` upstream receives:
- `default_config_does_not_sign_bedrock_requests`: no `AWS4-HMAC-SHA256`
Authorization on `invoke`, `invoke-with-response-stream`, `converse`.
- `default_config_never_attaches_host_adc_bearer`: `rawPredict` and
`streamRawPredict` reach the upstream without the host bearer.
- `vertex_bearer_is_not_sent_to_a_non_google_upstream`: with the route
enabled and an `http://127.0.0.1` upstream, 502 and the upstream sees
nothing.
- `vertex_opt_in_still_signs_when_non_google_upstream_is_allowed`: the
explicit escape hatch still works.
- `configured_adc_scope_reaches_token_source`.
- Unit test `vertex::raw_predict::tests::google_api_host_check`
(`http://`, `googleapis.com.example.net`, `example-googleapis.com`,
loopback are rejected).
### Test Output
Before (the three tests that compile against `main`, run with this PR's
`src/` changes stashed):
```text
test configured_adc_scope_reaches_token_source ... FAILED
test default_config_never_attaches_host_adc_bearer ... FAILED
test default_config_does_not_sign_bedrock_requests ... FAILED
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s
```
After:
```text
$ cargo test -p headroom-proxy --test integration_cloud_routes_opt_in
test configured_adc_scope_reaches_token_source ... ok
test vertex_opt_in_still_signs_when_non_google_upstream_is_allowed ... ok
test vertex_bearer_is_not_sent_to_a_non_google_upstream ... ok
test default_config_never_attaches_host_adc_bearer ... ok
test default_config_does_not_sign_bedrock_requests ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s
```
## Real Behavior Proof
- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (
|
||
|
|
f5c4217648 |
fix(proxy): enforce HEADROOM_PROXY_TOKEN in the Rust front and listen on loopback by default (#4840)
## Description
The Rust front (`crates/headroom-proxy`) relays every request to the
Python proxy from `127.0.0.1`. The Python proxy treats a loopback caller
as local: it skips the `HEADROOM_PROXY_TOKEN` check and serves `/admin`,
`/debug` and `/settings` to it. The front had no inbound check of its
own, listened on `0.0.0.0:8787` by default, and dropped
`x-headroom-proxy-token` (as an `x-headroom-*` header), so a token set
on the Python side neither protected the stack nor could be presented
through the front.
This PR makes the Rust front apply the same token model as the Python
proxy (`read_proxy_token` / `scrub_proxy_token_headers` in
`headroom/proxy/server.py`, `headroom/proxy/bind_policy.py`) before
anything is forwarded, and makes its default listen address loopback.
Fix adapted from a proposal by HetCreep.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `crates/headroom-proxy/src/inbound_auth.rs`, installed as a
router-wide layer (all routes, the catch-all and WebSocket upgrades)
when `HEADROOM_PROXY_TOKEN` / `--proxy-token` is set:
- a non-loopback peer (or one with no peer address) must send the token
as `X-Headroom-Proxy-Token` or `Authorization: Bearer`; the explicit
header wins, as in Python. Constant-time compare. Otherwise `401
{"error":"unauthorized"}` and no upstream contact; a
`proxy_auth_rejected` warning is logged (no credential values).
- loopback peers are exempt (same as Python); `GET /healthz`,
`/healthz/upstream`, `/health`, `/livez`, `/readyz` stay open;
`GET`/`HEAD /api/hello` (also under `/p/<project>`) is answered locally
with an empty 200, as Python does, instead of being relayed.
- the token is removed before forwarding: `x-headroom-proxy-token`
always, `Authorization` only when it carries the token (a provider
credential is left alone).
- the token is redacted from `Debug` output of `Config`.
- `main.rs`: startup bind policy, same rule and variable as Python: a
non-loopback `--listen` with no token exits with code 2 and a message
naming both remedies, unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
(then a warning is logged).
- `config.rs`: `--listen` / `HEADROOM_PROXY_LISTEN` default changes from
`0.0.0.0:8787` to `127.0.0.1:8787`; new `--proxy-token` (env
`HEADROOM_PROXY_TOKEN`, value hidden from `--help`).
- `RUST_DEV.md`: runbook exports a token for the `0.0.0.0` cutover, flag
table updated, new "Inbound authentication" section.
**Behaviour change for operators:** a Rust front started with `--listen
0.0.0.0:...` (or relying on the old default to be reachable from other
hosts) now needs `HEADROOM_PROXY_TOKEN` set, and clients on other hosts
must send it. Setups that front the port with something else that
restricts access can set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`.
Loopback clients (Claude Code, Codex, SDKs on the same host) are
unaffected. The Docker image's entrypoint is the Python proxy, so it is
unaffected.
## Testing
- [x] `cargo test -p headroom-proxy` (46 test binaries, 608 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes
New tests:
- `tests/integration_inbound_auth.rs` (11 tests, `oneshot` with an
injected `ConnectInfo` so a non-loopback peer can be simulated,
`wiremock` upstream): missing/wrong token is 401 and the upstream sees
nothing; correct token via header or Bearer is forwarded without the
token; provider `Authorization` kept when the header carries the token;
loopback and IPv4-mapped loopback peers pass; missing peer address fails
closed; health probes open but `POST /readyz` is 401; `/api/hello`
answered locally; no token configured keeps passthrough.
- `tests/integration_bind_policy.rs` (3 tests, real binary): `0.0.0.0`
with no token refuses to start; loopback starts; default `--listen` is
loopback.
- unit tests in `inbound_auth.rs` for token parsing, constant-time
compare, bind policy and exemptions.
### Test Output
Before (the bind-policy test file copied onto a clean `origin/main`
worktree; `integration_inbound_auth.rs` does not compile there since
main has no token setting at all):
```text
test default_listen_address_is_loopback ... FAILED
test loopback_listen_without_token_still_starts ... ok
test non_loopback_listen_without_token_refuses_to_start ... FAILED
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.01s
```
After:
```text
$ cargo test -p headroom-proxy --test integration_inbound_auth --test integration_bind_policy
test default_listen_address_is_loopback ... ok
test non_loopback_listen_without_token_refuses_to_start ... ok
test loopback_listen_without_token_still_starts ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.01s
test missing_peer_address_fails_closed ... ok
test remote_peer_with_bearer_token_is_forwarded_without_it ... ok
test remote_peer_without_token_gets_401_and_upstream_is_not_contacted ... ok
test token_header_wins_over_a_provider_bearer ... ok
test no_token_configured_keeps_todays_passthrough ... ok
test loopback_peer_without_token_is_still_forwarded ... ok
test ipv4_mapped_loopback_peer_counts_as_loopback ... ok
test remote_peer_with_token_header_is_forwarded_without_it ... ok
test hello_probe_is_answered_locally_without_token ... ok
test remote_peer_with_wrong_token_gets_401 ... ok
test health_probes_stay_reachable_without_token ... ok
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.74s
```
## Real Behavior Proof
- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (
|
||
|
|
70b21c12c4 |
feat(proxy): native spend/savings stats + dashboard for the Rust proxy (#2321)
# feat(proxy): native spend/savings stats + dashboard for the Rust proxy ## Description The Rust proxy has no cost observability of its own today: `/stats` and `/dashboard` only exist on the Python proxy, so anyone running `headroom-proxy` standalone (the Bedrock/Vertex lanes especially) is flying blind on spend. This adds it natively. Every lane the Rust proxy forwards — Anthropic `/v1/messages`, OpenAI Chat + Responses, all four native Bedrock routes, Vertex `rawPredict`/`streamRawPredict` — records per-request cost/usage telemetry into one in-process ledger, served from `GET /stats`, `GET /stats/timeseries`, `GET /stats/events`, and an embedded, dependency-free `GET /dashboard`. This is a ground-up redesign of #1057 (approved but never merged, now badly stale against `main`), not a rebase of it. Supersedes #1057. The "Design" section below states exactly what was kept and what was deliberately dropped, so it can be reviewed on its own terms. ### What it measures today Spend, input/output/cache-read/cache-write tokens, prompt-cache hit rate and its NET dollar benefit, per-provider and per-model breakdowns, failures, latency, and hourly/daily history that survives restarts. Two numbers are computed carefully rather than the obvious way, because the obvious way flatters the product: - **Spend counts output.** Input and output are priced separately (`input_cost_usd` / `output_cost_usd`, summed into `total_cost_usd`) because output bills at its own rate — typically 4-5x input. Counting input alone would understate a short-prompt/long-answer turn by most of its cost. - **Cache savings are NET**, not the gross read discount: the read discount minus the cache-write premium (writes bill *above* list price), floored at $0 per request. Reporting gross reads overstates the benefit on write-heavy traffic. ### What reads 0 today — stated up front `tokens_saved` / `compression_savings_usd` will read **0 on real traffic**, because the Rust live-zone per-type compressors are still no-ops on `main` — there is nothing to record yet. Verified against the real proxy: a 13.8 KB body with a large `tool_result`, sent with `--compression --compression-mode live_zone`, produced `tokens_before: 0` (dispatcher returned `NoCompression`), which is the invariant `live_zone_mode_with_valid_body_returns_no_compression_pr_b2` pins. The recording path is wired and unit-tested against `Outcome::Compressed`, and starts populating on its own the moment those compressors land — no changes needed here. Stating it up front so `Saved $0` isn't read as a bug. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Design — how this differs from #1057 - **One fold, many views.** A single `Totals::fold` is the only place request accounting happens; lifetime, since-boot session, per-provider, per-model, hourly and daily buckets are that same fold under different keys, so the views cannot disagree. #1057 (like the Python tracker it ported) maintained each aggregate separately. - **Its own schema, not Python emulation.** `/stats` is a documented Rust-native payload (`version: 1` on disk). No `schema_version: 5` file compat, no `display_session` idle-window emulation, no `projects` passthrough. The five keys external unified-stats consumers already read from this endpoint (`summary.api_requests`, `requests.failed`, `requests.cached`, `tokens.saved`, `tokens.proxy_compression_saved`) are kept and pinned by a test. - **No federation flag.** #1057's `--upstream-stats-url` supplemental merge is dropped (YAGNI — the deployed consumer merges in the opposite direction). - **No new price data file.** USD valuation reuses the LiteLLM table already vendored for `compression::model_limits`. #1057 shipped a second, models.dev-shaped price book. - **Purpose-built dashboard.** One self-contained HTML file designed for what the Rust proxy actually measures, instead of embedding the Python template and rendering its ~20 Python-only panels as zeros. - **Not gated on compression.** `--compression` defaults to **off**, so recording gated behind the compression buffer (as #1057 had it) records nothing under default config. Here the model is sourced from the *response* when the request body isn't buffered, so spend/cache observability works on a pure passthrough deployment — with zero change to the byte path. ## Correctness rules (carried from #1057's review, all test-pinned) - **Deferred recording** — a request records only when its response is fully observed, so streaming token counts come from real SSE/EventStream usage frames instead of 0. - **Failed upstreams accrue nothing** — counted in `requests`/`failed` only; retries can't inflate savings. - **No disk I/O on the request path** — `record()` folds under a mutex and sets a dirty flag; a background task (and shutdown) writes an atomic snapshot (temp → fsync → rename → fsync dir), re-arming the dirty flag on failure. - **Bounded everything** — per-model cardinality caps with `"other"` overflow, length-clamped client strings, a 50M-token per-event ceiling, bounded capture queues/buffers, bounded warn-once set. - **Honest pricing** — exact-match only; a generic id misses (and WARNs) rather than mispricing. Unknown ⇒ $0, never a blended fallback. - **Bedrock regional pricing respected** — the unmodified id is tried first, because `eu.anthropic.claude-3-5-haiku…` genuinely bills $0.25/M against the base id's $0.80/M. Geo-stripping is only a fallback for regions the table doesn't list. ## Pre-existing bugs fixed along the way Both were found while wiring the recording hooks; both affect `main` today, independently of stats. - **Bedrock geo-prefix coverage**: `au.`, `jp.`, and `us-gov.` cross-region inference profiles weren't recognized, so Anthropic models invoked through them silently skipped compression — the same failure mode #953 reported and #999 fixed for `eu.`/`us.`/`apac.`/`global.`. The list is now complete (all seven published geos) and shared (`bedrock::vendor::GEO_PREFIXES`) between the vendor gate and pricing, so the two can't drift apart again. - **EventStream `Skip` frame stranding**: the streaming parser pulled a fresh upstream chunk after each unknown (`Skip`ped) event type, delaying — and at stream end **dropping** — frames already buffered. It now drains the buffer to exhaustion before awaiting more bytes. Regression-pinned by `skip_in_final_chunk_does_not_strand_buffered_frames`. ## Changes Made - `crates/headroom-proxy/src/observability/ledger.rs` — the store: `Totals` fold, bounded maps/buckets, atomic snapshot persistence, payload builders - `crates/headroom-proxy/src/observability/pricing.rs` — deterministic lookup over the vendored LiteLLM table (no new data file) - `crates/headroom-proxy/src/observability/capture.rs` — `PendingRecord` (deferred finalization), usage extraction for every wire shape, bounded tee tasks off the byte path - `crates/headroom-proxy/src/observability/stats_http.rs` + `dashboard.html` — HTTP surface + embedded self-contained UI - `crates/headroom-proxy/src/proxy.rs`, `bedrock/invoke.rs`, `bedrock/invoke_streaming.rs`, `vertex/raw_predict.rs` — per-lane recording hooks - `crates/headroom-proxy/src/config.rs` — `--stats` (default on), `--stats-path` - `crates/headroom-proxy/src/bedrock/vendor.rs` — shared `GEO_PREFIXES` (the geo-coverage fix above) - `crates/headroom-proxy/src/sse/framing.rs` — 4 MiB accumulator cap (telemetry-side; a stream with no event terminator can no longer grow the buffer unboundedly) `--stats=false` restores byte-for-byte pre-feature behaviour (paths tunnel upstream, nothing recorded) — pinned by `stats_disabled_falls_through_to_upstream`. ## Testing - [x] `cargo fmt --all -- --check` passes - [x] `cargo clippy --workspace -- -D warnings` passes (0 warnings) - [x] `cargo test --workspace` passes - [x] New tests added for new functionality (53 new tests across unit + integration) - [x] Manual testing performed (live AWS Bedrock, all four native lanes — see Real Behavior Proof) Unit: fold-consistency across views, failed-no-savings, pricing (regional/revision/path forms, collision-miss, untracked-region fallback), USD math, bounded maps, token-ceiling clamp, bidi/control-char stripping, atomic flush with no temp litter, corrupt-file preservation, oversized-state rejection, negative-USD clamping on load, restart round-trip, week/month rollups, hostile field lengths. Integration (wiremock): endpoint contracts, `--stats=false` tunneling, deferred SSE capture, non-streaming capture, OpenAI cached-token normalisation, failed-upstream accounting, hostile-usage clamping, all four Bedrock lanes, Vertex, dashboard self-containment. The persisted-state tests are deliberately adversarial rather than round-trip only, since the ledger reads a file an operator can hand-edit: malformed bucket keys (including multi-byte ones that would panic a naive `&str` slice) are dropped before the age-trim runs, hostile model-id keys are re-clamped on load, and an implausibly large state file is treated as corrupt instead of allocated. ### Test Output ```text $ cargo fmt --all -- --check (clean) $ cargo clippy -p headroom-proxy --all-targets -- -D warnings exit=0 0 warnings, 0 errors $ cargo nextest run -p headroom-proxy Summary [4.396s] 488 tests run: 488 passed, 0 skipped $ cargo nextest run -p headroom-core -p headroom-simulators -p headroom-parity Summary [49.535s] 967 tests run: 967 passed, 1 skipped $ cargo clippy --workspace -- -D warnings exit=0 0 warnings, 0 errors $ cargo test --workspace --doc ok, 0 failed ``` ## Real Behavior Proof - Environment: live AWS Bedrock, staging account (SSO profile — name redacted), `eu-central-1`, `anthropic.claude-3-haiku-20240307-v1:0`, release binary, compression off (shipped default) - Exact command / steps: `headroom-proxy --listen 127.0.0.1:8790 --enable-bedrock-native --bedrock-region eu-central-1` then one request per lane (`/invoke`, `/converse`, `/invoke-with-response-stream` with `Accept: application/vnd.amazon.eventstream`, `/converse-stream`), one deliberate failure (nonexistent model), `SIGTERM`, restart, re-read `/stats` — full transcript, raw proxy logs, and the persisted ledger are attached below - Observed result: all four lanes captured usage (input tokens = 48 = 4 × 12; spend $0.000012 = 48 × $0.25/M, exact for Bedrock Haiku 3); the failure recorded `failed: true` with 0 tokens and $0; lifetime totals survived the restart while session totals reset; dashboard rendered against this live data (screenshots attached) - Not tested: live Vertex (covered by wiremock integration tests only — no GCP staging account available); live OpenAI Chat/Responses lanes (same wiremock coverage); dashboard on Windows browsers The four Bedrock lanes are the ones worth proving live, because two of them had no parsing before this PR: | Lane | Wire shape | Captured | |---|---|---| | `/invoke` | sync, snake_case `usage` | 12 in / 4 out | | `/converse` | sync, **camelCase** `usage` | 12 in / 4 out | | `/invoke-with-response-stream` | **binary EventStream passthrough**, base64-wrapped chunks | 12 in / 4 out | | `/converse-stream` | SSE-translated; usage lives in the camelCase `metadata` frame the Anthropic state machine drops | 12 in / 4 out | ```text requests : 5 (failed: 1) input tokens : 48 output tokens : 16 cache read/wr : 0 / 0 input spend : $0.00001200 # 48 × $0.25/M output spend : $0.00002000 # 16 × $1.25/M — output bills 5x here TOTAL spend : $0.00003200 # what /dashboard reports as "Spend" savings : $0.00000000 # compression off -> 0 by design avg latency : 256 ms by provider : {'bedrock': 5} by model : {'anthropic.claude-3-haiku-20240307-v1:0': 4, 'anthropic.claude-nonexistent-v99:0': 1} persistence : True | healthy: True ``` Those two spend lines are the reason output pricing is in this PR: **output was 62.5% of the real bill** on this run. Reporting input alone would have called a $0.000032 spend `$0.000012`. The exposure split was exercised against the same live proxy: ```text $ curl /stats/events -H 'Host: 127.0.0.1:18787' -> 200 (local tooling works) $ curl /stats/events -H 'Host: attacker.com' -> 404 (DNS-rebound: peer IS loopback) $ curl /stats -H 'Host: attacker.com' -> recent_requests: null persistence.path: null requests.total still served ``` ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (RUST_DEV.md, CHANGELOG.md) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Known limitations 1. Compression savings read 0 until the live-zone compressors land (above). 2. Hourly resolution is 48 h, daily ~13 months; older traffic folds into lifetime totals only. `/stats/events` is an in-memory ring of 100 and is not persisted — this is a cost ledger, not an audit log. 3. The vendored price table lacks some legacy direct-API ids (`claude-3-5-sonnet-20241022` is absent while its Bedrock form is present); those record $0 with a WARN until refreshed via `scripts/refresh_model_limits.sh`. 4. `output_cost_usd` is absent from state files written before this change, so it loads as 0: historical spend keeps reading input-only until those buckets age out (48 h hourly / ~400 d daily). New traffic is priced in full immediately. Deliberately no migration shim — the field defaults cleanly and the alternative is retro-pricing history against today's price table, which would invent numbers. 5. A non-streaming response body over 2 MiB isn't buffered for usage parsing (the request still records, compression-side numbers only). Streaming responses have no such cap — they parse incrementally. 6. The endpoints carry no auth, so exposure is split by tier instead (matching the Python proxy): aggregates are public like `/metrics`, while per-request rows (`recent_requests`, `/stats/events`) and the ledger's filesystem path require a local caller — loopback peer **and** loopback `Host`, the latter being the DNS-rebinding defence. Remote viewers of `/dashboard` therefore see totals but not the request feed, and the page says so. Opening that tier to a trusted reverse proxy is #1959's call to make for both proxies; this PR doesn't invent a second scheme. 7. Copilot traffic isn't proxied by the Rust binary; it reaches this dashboard only via external unified-stats layers (unchanged). ## Screenshots (if applicable) Attached below: live-test terminal transcript, dashboard rendered against the live Bedrock data (session + lifetime scopes, before/after restart), raw proxy logs, and the persisted ledger file. ## Additional Notes - Related issues/PRs: supersedes #1057; completes the geo-prefix coverage from #999 (design discussion in #953 — the multi-vendor adapter work proposed there remains out of scope). - `requests.cached` counts requests whose response reported `cache_read_tokens > 0`, matching the Python tracker's derivation, so unified-stats consumers see consistent semantics across both proxies. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2321" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
a05717f6ec |
fix(proxy): redact upstream error detail and add opt-in /metrics loopback gate (#2589)
## Description Two response-path information-exposure hardenings for the native Rust reverse proxy (`crates/headroom-proxy`). Both are defensive and non-breaking; neither adds a user-facing capability nor changes forwarding behaviour, and defaults are unchanged. 1. **Redact upstream error detail from client responses.** `ProxyError::into_response` returned `self.to_string()` as the client body, which for `reqwest` upstream errors interpolates the upstream URL/host/port. It now returns a generic, stable message per status while the full detail is retained in the server-side `tracing::warn!` log. `413` intentionally stays descriptive (no internal detail; retry-on-413 clients rely on recognising it). 2. **Opt-in loopback gate for `/metrics`.** New `--metrics-require-loopback` / `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK` (default `false`). When enabled, the Prometheus scrape returns `403` to non-loopback peers. This is defense-in-depth for deployments that bind a non-loopback `--listen`. Default-off preserves current behaviour exactly. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `error.rs`: client-facing bodies are now generic per status; full detail stays in the log only. - `config.rs`: add `--metrics-require-loopback` / `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK` (default `false`). - `proxy.rs`: mount `/metrics` in its own sub-router so the optional `require_metrics_loopback` layer gates only that route (403 to non-loopback peers when enabled). - `RUST_DEV.md`: document the new flag in the configuration table. - Tests: new `tests/integration_metrics_loopback.rs` plus unit tests for the gate (v4/v6 loopback and non-loopback) and for error-body redaction. ## Testing <!-- This PR touches only the Rust `headroom-proxy` crate; the pytest/ruff/mypy items are Python and N/A here (see Additional Notes). Validated with cargo on the repo-pinned toolchain. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo fmt --all -- --check # clean $ cargo clippy --workspace -- -D warnings # 0 warnings $ cargo test --workspace test result: ok. 1432 passed; 0 failed; 0 ignored (aggregate across all suites) $ make test-parity [tokenizer/smart_crusher/content_detector/text_crusher/diff/log/search] diffed=0 ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Rust 1.95.0 (repo-pinned `rust-toolchain.toml`), `headroom-proxy` debug binary run directly (no mocks), dead upstream `http://127.0.0.1:9`. - Exact command / steps: started with `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK=true --listen 0.0.0.0:18787`, then `curl http://127.0.0.1:18787/metrics`, `curl http://192.168.1.184:18787/metrics` (LAN IP, non-loopback), and `curl -X POST http://127.0.0.1:18787/v1/messages -d '{"model":"x","messages":[]}'`; then restarted with the gate off and repeated the non-loopback `/metrics` scrape. - Observed result: loopback `/metrics` returned `200`; non-loopback `/metrics` returned `403` with body `metrics endpoint restricted to loopback`; the dead-upstream POST returned `502` with client body `upstream connection failed` while the server log retained the full detail (`... error sending request for url (http://127.0.0.1:9/v1/messages)`), i.e. the upstream URL is logged, not returned to the client; with the gate off, the non-loopback `/metrics` scrape returned `200` (default behaviour unchanged). - Not tested: live cloud upstreams (Bedrock/Vertex/OpenAI/Anthropic real endpoints); the Python proxy path (unchanged by this PR); Windows/Linux (macOS only). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A (server-side behaviour; see Real Behavior Proof for command output). ## Additional Notes - This PR modifies only the Rust `headroom-proxy` crate (plus its docs), so the Python-oriented Testing checkboxes (`pytest`, `ruff`, `mypy`) are N/A; validation was via `cargo fmt`/`clippy`/`test` and `make test-parity` on the repo-pinned toolchain, per the Test Output above. - No new dependencies. Both changes are non-breaking and leave all defaults unchanged (the `/metrics` gate is opt-in). <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2589" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
96dfc7005f |
chore: finish the org and docs-domain migration (#3389)
Two stale identities left over from moves that were only half applied. ## GitHub org `install/state.py` already migrated Docker images to `ghcr.io/headroomlabs-ai/headroom` and carries a `_DEPRECATED_IMAGE_REPO` constant that rewrites the old path on load — so this move was decided long ago. Everything else still said `chopratejas/headroom`. Rewritten across packaging (`pyproject.toml`, `Cargo.toml`), both plugin marketplaces, the agent-hooks manifests, install scripts, CI templates, crate docs and tests. **Two deliberate exclusions:** - **`ghcr.io/chopratejas/headroom` is untouched everywhere.** It is the *input* to `_migrate_deprecated_image()` and to the tests covering it. Rewriting it would silently disable the migration. - **`CHANGELOG.md` keeps its 238 historical references.** GitHub's rename redirect resolves them, and rewriting would bury this diff in noise. **HuggingFace model ids are NOT affected and were never in scope.** `chopratejas/kompress-base`, `kompress-v2-base`, `kompress-finance`, `technique-router`, `technique-router-onnx` and `siglip-image-encoder-onnx` are account-scoped identifiers on a different service. All six were validated against the HF API before this change: | Model | HTTP | |---|---| | `kompress-base` | 200 | | `kompress-v2-base` | 200 | | `technique-router` | 200 | | `technique-router-onnx` | 200 | | `siglip-image-encoder-onnx` | 200 | | `kompress-finance` | **401 (private)** | The 401 is unrelated to this migration, but worth knowing: `kompress_compressor.py:1255` references it, so that path is not reachable by an OSS user without access. **One functional fix, not cosmetic:** `_marketplace_source()` (`headroom/cli/init.py`) returned the old org as its fallback, so plugin installs were pointed at the retired repo. ## Docs domain The site moved to `docs.headroomlabs.ai`, but eleven references still named the vercel.app host — including **`llms.txt` (27 links)**, which is served at `docs.headroomlabs.ai/llms.txt` and read by LLM crawlers, the **dashboard's own Documentation link**, and a `headroom wrap` help message. The docs app's canonical-URL fallbacks were fixed separately in #3388. ## Also folded in `RUST_DEV.md` corrections found while verifying the above: - `make build-wheel` points at `Cargo.toml`, not `pyproject.toml` - the Rust toolchain is pinned, not tracking `stable` - `headroom-simulators` was missing from the workspace listing - the Python `CompressionStore` defaults to **SQLite**, not in-memory, when `HEADROOM_CCR_BACKEND` is unset (verified by execution) — this one matters for multi-worker reasoning And the install scripts pointed at `docs/docker-install.md`, a path that no longer exists; they now point at the live docs page. ## Verification 297 tests pass, including the plugin-manifest, install-state and release-workflow suites that assert on these exact strings. `ruff check` clean. Unrelated lockfile churn (`uv.lock`, both `package-lock.json`) is deliberately excluded. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
a639540959 |
chore: remove committed node_modules + stray/internal markdown (repo hygiene) (#1528)
## Description Repo hygiene for a public OSS project: removes committed `node_modules`, stray/internal/draft markdown, and commercial-surface references — keeping every real doc (the published docs site, the wiki guides, and all component READMEs) intact. Every file was content-audited before removal, and load-bearing files were verified against the code/CI and kept. Net: **1,695 files changed, +23 / −266,409** (the deletions are dominated by a committed `node_modules` tree). Closes # (no tracking issue) ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made **Removed (verified to have no code/CI dependencies):** - `examples/vercel-ai-sdk-pr/` — 1,649 committed `node_modules` files (zero example source); `node_modules/` added to `.gitignore`. - `docs/spec/` (23 draft "Living Specification" files — orphaned, `1.0.0-draft`, drifted from the code), `docs/superpowers/` (2 agent plans), `docs/proposals/` (2 internal/commercial memos). - 6 orphan `docs/*.md` (auth-modes, bedrock, claude-code-vertex-headroom, cortex-code, output-token-reduction-guide, rtk-loop-weighting). - `PR.md` (committed PR draft), `ENTERPRISE.md`, `.github/FUNDING.yml`. **Content scrubs:** - Removed unreleased "Headroom Cloud" / `api.headroom.ai` / `hr_` references from `configuration.mdx`, `wiki/configuration.md`, `wiki/typescript-sdk.md`, `sdk/typescript/README.md` (reworded to neutral, accurate phrasing). - Dropped a stale "awaiting maintainer before merge" line from `plugins/headroom-oauth2/SPEC.md`; tidied `.gitignore` comments (kept the protective `headroom-managed/` ignore rule). - Fixed the now-dangling links into removed files (README nav/`output-token-reduction` link, `scripts/README`, `wiki/vertex`). **Explicitly KEPT (load-bearing — would orphan in-code citations if removed):** - `.changelog.md` — consumed by `.github/workflows/release.yml` (read as the release-notes file). - `REALIGNMENT/`, `docs/observability.md`, `docs/rtk-architecture.md`, `wiki/plans/`, `TESTING-copilot-subscription.md` — referenced by the Rust core / Python / tests as design docs. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [ ] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Docs/markdown + .gitignore only — no Python/Rust source changed, so the # behavioral test suite is unaffected. Verified the cleanup did not orphan # references or break the published docs site: $ git ls-files 'docs/content/docs/*.mdx' | wc -l # published site intact 42 $ # meta.json nav unchanged; no published page removed. $ grep -rnI "Headroom Cloud|api.headroom.ai|'hr_" $(git ls-files '*.md' '*.mdx') >>> none $ # dangling refs to removed files (excl pre-existing P0/P2 spec stubs that $ # never existed in git): none remaining. ``` ## Real Behavior Proof - Environment: macOS, local git clone of the repo (markdown/.gitignore changes only — no runtime). - Exact command / steps: 4 read-only content-audit agents classified every `.md`/`.mdx` file; each removal candidate was cross-checked against the codebase (`grep` for citations in `.rs`/`.py`/tests, workflows, and configs); only files with no dependents were removed; the tree was re-grepped after removal to confirm no new dangling references; verified the published docs site page count (`git ls-files 'docs/content/docs/*.mdx' | wc -l` = 42, unchanged). - Observed result: the 42-page published docs site and all wiki guides are untouched; no source or workflow references a removed file; `.changelog.md` (consumed by release.yml) and the code-cited design docs were detected as dependencies and kept; the committed `node_modules` tree is removed and `node_modules/` is gitignored so it can't be re-committed; zero "Headroom Cloud"/`headroom.dev` references remain. - Not tested: N/A — no executable code changed (only markdown, `.mdx`, and `.gitignore`), so the behavioral test suite is unaffected. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - This branch deletes `.github/FUNDING.yml` while PR #1526 edits it — the two will be sequenced at merge (delete wins). - A follow-up option (not in this PR): also remove the internal design docs that are currently cited by the code (`REALIGNMENT/`, `docs/observability.md`, `docs/rtk-architecture.md`, `wiki/plans/`) — that requires scrubbing ~15–20 in-code citations so nothing dangles, so it's deliberately deferred. - Untracked local working files (`benchmarks/hf_pilot/`, `tools/copilot-test/`) are intentionally left out of git (not committed). |
||
|
|
d76a7296df |
fix(proxy): make CCR multi-worker warning conditional on backend (#770)
## Problem The multi-worker startup warning always mentioned CCR retrieval failures, even when the operator had already configured a cross-worker backend via `HEADROOM_CCR_BACKEND`. That's noise — if they've set `HEADROOM_CCR_BACKEND=sqlite` or `redis`, CCR fragmentation is already resolved. This was surfaced during review of #628 (now closed): the reviewer correctly noted that Python `CompressionStore` defaults to `InMemoryBackend`, which is per-process — each uvicorn worker has its own singleton, so CCR markers written on worker A are invisible to worker B unless a shared backend is configured. ## Changes ### `headroom/proxy/server.py` The `workers > 1` warning is now conditional on `HEADROOM_CCR_BACKEND`: - **Backend unset (default `InMemoryBackend`, per-process):** warning includes CCR retrieval failures and suggests `HEADROOM_CCR_BACKEND=sqlite` to use a shared cross-worker store. - **Backend configured (`sqlite`/`redis`):** warning covers only the remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker) — CCR fragmentation is already resolved. ### `RUST_DEV.md` Updated the multi-worker fragmentation section: - Removed the incorrect parenthetical claiming this only applies when the operator *explicitly* chooses `CcrBackendConfig::InMemory` (Python defaults to InMemory) - Added Python `CompressionStore` as item 1 in the fragmented-state list, with a note that setting `HEADROOM_CCR_BACKEND=sqlite` resolves it - Restored TOIN to the fragmented list with a note that its file-backed snapshots do not make it coherently shared across workers - Updated "Detecting it in the wild" to document the conditional warning behaviour ## Files changed | File | Change | |---|---| | `headroom/proxy/server.py` | Conditional two-branch warning based on `HEADROOM_CCR_BACKEND` | | `RUST_DEV.md` | Accurate per-process description of Python `CompressionStore`; restored TOIN | | `CHANGELOG.md` | Entry under `[Unreleased]` | Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
42b1cd24de |
docs: fix env var discrepancies across markdown files
Audit all .md files against codebase; fix wrong names, remove phantom
variables, and correct outdated values:
- HEADROOM_PROXY_PORT → HEADROOM_PORT (proxy.py envvar="HEADROOM_PORT")
- HEADROOM_BIND → HEADROOM_HOST + HEADROOM_PORT (RUST_DEV.md)
- HEADROOM_LEARN_{CLAUDE,CODEX,GEMINI}_ENABLED → HEADROOM_LEARN_CLI
(only HEADROOM_LEARN_CLI exists in learn/analyzer.py)
- HEADROOM_TRACING_ENABLED → HEADROOM_LANGFUSE_ENABLED=1 with correct
LANGFUSE_PUBLIC_KEY/SECRET_KEY vars (tracing.py)
- HEADROOM_LOG_LEVEL/LOG_FORMAT → --log-level CLI flag / RUST_LOG
(no HEADROOM_LOG_LEVEL var exists in code)
- HEADROOM_LOG_LEVEL/HEADROOM_STORE_URL/HEADROOM_DEFAULT_MODE rows
removed from wiki/configuration.md (all phantom)
- HEADROOM_SUMMARY_{ENABLED,THRESHOLD,RATIO} noted as not yet
implemented (no code exists)
- HEADROOM_DB_URL/HEADROOM_CACHE_BACKEND → explanatory notes pointing
to HEADROOM_WORKSPACE_DIR (no external DB support in code)
- HEADROOM_DB_PATH/HEADROOM_CACHE_PATH table rows replaced with actual
HEADROOM_WORKSPACE_DIR/CONFIG_DIR (paths.py)
|
||
|
|
00902b8fea |
fix: B7 — CCR hardening: persistent backends + always-on tool
P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store
that fragmented across uvicorn workers and was wiped on restart, and
the `headroom_retrieve` tool was registered/unregistered per-request
based on whether the latest body happened to contain compression
markers — every flip busted the prompt cache. Both are sticky
side-channels: once a session has done CCR, the tool list bytes and
the retrieval store must stay stable. This PR fixes both.
Rust:
* Split `ccr.rs` into `ccr/` with `backends/` submodule
(`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated).
* `SqliteCcrStore` (production default): WAL mode, prepared upsert,
lazy TTL purge on read, persistent across worker restarts and
shareable across workers on the same host via SQLite file locking.
* `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with
startup PING smoke-test, no key-prefix collision risk, no sticky
session required at the LB.
* `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)`
factory — every init failure surfaces (no silent fallback per
`feedback_no_silent_fallbacks.md`).
* `ccr::compute_key` (BLAKE3 → first 24 hex chars) and
`ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash +
marker format; one definition for the live-zone dispatcher and the
Python regex (`headroom/ccr/tool_injection.py:211`).
* `compress_anthropic_live_zone_with_ccr` accepts
`Option<&dyn CcrStore>`. When wired, every accepted compression
puts the original bytes into the backend and appends `<<ccr:HASH>>`
to the compressed string. The token-validation gate runs on the
marker-augmented string so the `compressed_tokens >=
original_tokens` rejection stays honest.
Python:
* `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the
PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools`
pattern: once a session has done CCR, every subsequent request
injects the recorded golden tool-definition bytes. Tool list bytes
are byte-stable across turns (snapshot test pins them).
* `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a
new `session_has_done_ccr` kwarg per the PR-B7 spec change at line
302-328. The legacy per-request path stays intact for callers that
don't yet thread a session id (e.g. Google handler).
* Anthropic + OpenAI handlers route their CCR tool-list updates
through `apply_session_sticky_ccr_tool`, keyed off the existing
`session_tracker_store.compute_session_id(...)` plumbing.
Backend selection model: `CcrBackendConfig::Sqlite { path }` is the
production default — single host, persistent, multi-worker safe with
sticky session. `CcrBackendConfig::Redis { url }` is the multi-host
scale-out option — no stickiness needed. `InMemory` is for tests
and single-worker dev only. RUST_DEV.md "Multi-worker deployment —
CCR fragmentation" rewritten around this matrix.
Tests:
* `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering
SQLite round-trip, TTL purge, proxy-restart survival, cross-backend
byte-equal keys, `from_config` paths, and the no-redis-feature
loud-failure check (+ 2 redis tests gated behind the feature).
* `crates/headroom-core/tests/live_zone_ccr.rs` — confirms
`<<ccr:HASH>>` marker injection, store population, and
no-marker-when-no-store invariants end-to-end.
* `tests/test_ccr_tool_always_on.py` — 12 tests pinning the
always-on behaviour, session/provider isolation, LRU bound, no-
session-id fallback, and (per-acceptance-criterion) the byte-stable
tool-definition snapshot.
Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
|
ea78cf6252 |
fix(proxy): cache concurrency lock, multi-worker docs, bounded compression executor
Three audit follow-ups from issue #327's deep-dive review. C1 — CompressionCache concurrency lock ====================================== `CompressionCache` instances are shared per `session_id` and accessed from async-dispatched threadpool workers. Pre-fix, concurrent requests for the same session raced on `_cache`, `_stable_hashes`, `_first_seen`, and `_total_tokens_saved` with no synchronization. Observable failures: * Lost-update on `_total_tokens_saved` (read-modify-write). * `RuntimeError: OrderedDict mutated during iteration` from `apply_cached` when a concurrent `store_compressed` evicts during the walk. * Lost stable-hash records — next-turn compute_frozen_count reads inconsistent state. May also explain part of SvenMeyer's `_cache: 0 entries / 1003 misses` observation: the cache was being clobbered concurrently. Added `threading.RLock` guarding all mutating methods. `RLock` (not `Lock`) so future code can call locked methods from inside another locked method without self-deadlock. Also locked `HeadroomProxy._compression_caches` dict-of-caches access via a separate `_compression_caches_lock` so two concurrent calls for the same session_id can't each create distinct CompressionCache objects (which would split the cache state between them). The `/stats` endpoint snapshots the cache list under the dict lock before iterating to avoid eviction-during-iteration. C2 — Multi-worker CCR fragmentation: documented + startup warning ================================================================= The in-memory `InMemoryCcrStore` (Rust), `_compression_caches` (Python), `session_tracker_store` (Python), and TOIN learner state are ALL per-process. Multi-worker uvicorn round-robins requests across workers, so a session whose turn-1 lands on worker A may have turn-2 land on worker B. Worker B has zero knowledge of A's CCR markers, replay cache, or prefix-cache state. Result: `Retrieve original: hash=X` markers stay in-context as opaque directives, every fresh tool_result is recompressed from scratch, and `frozen_message_count=0` causes Anthropic prefix-cache busts on every cross-worker turn. Added a "Multi-worker deployment — CCR fragmentation" section in `RUST_DEV.md` documenting the failure modes, the supported configuration (`--workers 1`), and the sticky-session workaround for horizontal scale. The proxy emits a `WARNING`-level log line on startup if `workers > 1` is detected, pointing at the doc section. C3 — Bounded compression executor with cancel-aware metrics =========================================================== `asyncio.wait_for(asyncio.to_thread(pipeline.apply), timeout=...)` cancellation does NOT propagate into the threadpool worker that's running Rust code. Once the worker has picked up the task, `concurrent.futures.Future.cancel()` returns False and the thread runs to completion. Stuck threads accumulated invisibly on asyncio's default executor, contending with unrelated `to_thread` callers (file IO, etc.). Replaced all 7 `asyncio.to_thread` call sites for `pipeline.apply()` across `proxy/handlers/anthropic.py` (3) and `proxy/handlers/openai.py` (4) with a new `HeadroomProxy._run_compression_in_executor(fn, *, timeout)` helper that: 1. Submits to a dedicated bounded `ThreadPoolExecutor` named `headroom-compress` (configurable via `ProxyConfig.compression_max_workers`; defaults to `min(32, (cpu_count or 1) * 4)`). 2. Increments `_compression_in_flight` (gauge) when work starts and decrements when work completes; tracks `_compression_in_flight_max` as a high-water mark. 3. Detects "leaked threads" by comparing wall-clock elapsed against the timeout in the worker's `finally` block. Increments `_compression_leaked_threads` when a worker finishes after its asyncio future was cancelled. Operators can see the leaked-thread rate climbing in `/stats runtime.compression_executor` BEFORE the pool fills up. Tests ===== * `TestCompressionCacheConcurrency` (3 tests) — many threads store_compressed / apply_cached / update_from_result on a single CompressionCache; assert no exceptions, no lost updates, no partial state. * `test_get_compression_cache_returns_same_instance_under_contention` — 32 concurrent `_get_compression_cache(same_id)` calls return the identical instance (would split pre-lock). * `test_proxy_compression_executor.py` (8 tests) — pool size respects config, in-flight gauge tracks running compressions, high-water mark is monotonic, timeout propagates to awaiter, leaked-thread counter increments on post-deadline completion, `/stats` surfaces all three gauges. Verification ============ * All 123 targeted regression tests pass. * `make ci-precheck` clean. * No `Co-Authored-By` trailer; conventional `fix:` prefix; no `--no-verify`. |
||
|
|
378d8a0f05 |
fix(rust): audit cleanup — DiffCompressor CCR leak, CCR TOCTOU race, clippy debt, dep dedup
Closes findings from the post-Phase-3g audit. Five surgical fixes plus telemetry-discoverability docs. PyO3 0.22 → 0.24 security upgrade is its own PR (issue #335). 1. DiffCompressor cache_key persistence (production bug) --------------------------------------------------------- Pre-fix: `RustDiffCompressor.compress()` minted a `cache_key`, embedded `[... hash=abc123]` in the wire marker, and returned without storing the original anywhere. Python ContentRouter then returned the compressed text with a dangling marker — every retrieval tool call from the LLM 404'd. Sibling compressors (LogCompressor, SearchCompressor) already had the right pattern: Rust mints the key, Python's `_persist_to_python_ccr` writes the original to the production `CompressionStore`. DiffCompressor was the asymmetric one. Fix: - Rust: add `DiffCompressor::compress_with_store(content, context, Option<&dyn CcrStore>)` mirroring siblings. Calls `store.put` when a key is minted; legacy `compress()` and `compress_with_stats()` delegate with `None` for parity. - Python: add `_persist_to_python_ccr` helper to `headroom/transforms/diff_compressor.py.compress()` mirroring `log_compressor.py` and `search_compressor.py`. - Pipeline `DiffOffload`: switch to `compress_with_store(Some(store))` and drop the post-hoc double-store hack that papered over this bug at the orchestrator boundary. 2. CCR store TOCTOU race in `get()` ----------------------------------- `InMemoryCcrStore::get()` checked TTL under a read lock, dropped the lock, then called `remove()`. Between drop and remove a concurrent `put()` of the same hash with fresh data could land — and our `remove` would then wipe that fresh entry. Under multi-worker proxy load this manifested as "I just stored it; why is it gone?" Fix: use `DashMap::remove_if`. Predicate runs under the shard write lock so check-and-remove is atomic. New regression test exercises a tight contention loop between writer and reader on the same key. 3. Pre-existing clippy debt in smart_crusher -------------------------------------------- - 3× `field_reassign_with_default` in `crusher.rs` test setup — switch to struct-update syntax `Config { field: x, ..Default }`. - `hash_array_for_ccr` was `#[cfg(test)]` but unused; deleted with a comment so a future test can reintroduce it as a one-liner. `cargo clippy --workspace --all-targets -- -D warnings` is now clean across the whole workspace; previous CI patches that allowed these warnings can be removed in a follow-up. 4. Tokenizers dependency dedup ------------------------------ `tokenizers 0.21` (direct dep) + `tokenizers 0.22` (transitive via fastembed) compiled twice into the binary. Bumped direct dep to `0.22` to align; API is compatible (verified by full tokenizer test suite). Saves compile time + binary bloat. 5. Telemetry-discoverability doc (no new code) ---------------------------------------------- The audit recommended a per-transform invocation counter to inform the next Python → Rust port. Discovered the infrastructure already exists at `/stats`: - `compressions_by_strategy` — invocation count per strategy - `pipeline_timing` — count + avg/max ms per transform name - `tokens_saved_by_strategy` — savings attribution Added a section to `RUST_DEV.md` showing the `curl + jq` recipes to read this data, with example output highlighting how to spot zero-invocation deferral candidates (e.g. `code_compressor`). Verification: workspace tests 734 + 14 + 5 + 4 + 6 + 5 + 2 + 2 + 3 + 4 + 2 + 2 + 1 = all green; cargo fmt clean; cargo clippy --all-targets clean; Python tests 185 pass; commitlint clean. |
||
|
|
12c2665531 |
feat(rust): signals trait module + KeywordDetector (Phase 3e.1)
Establish `crates/headroom-core/src/signals/` as a top-level module holding cross-cutting detection traits. Phase 3e.1 ports `error_detection.py` to a `LineImportanceDetector` trait + a `Tiered<T>` combinator + a single concrete `KeywordDetector` impl backed by aho-corasick. Three traits at three granularities are sketched (line / blob / item); only line-importance is implemented today. Two bug fixes from the Python source bake into both the Rust impl and the Python regex shim: 1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but `ERROR_PATTERN` regex omitted them. Lines like `"Connection timeout"` were silently neutral despite the keyword being canonical. Both surfaces now flag them. 2. `SECURITY_KEYWORDS` carried `token`, which false-positived on every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...) in our own product. Dropped from the security set. The Python `error_detection.py` shim now reflects keyword data out of Rust via `keyword_registry_snapshot()` and recompiles the legacy `re.Pattern` objects on the fly. Existing callers (text_compressor, search_compressor, intelligent_context) continue to import the same names with no source changes; caller migration to the trait API happens in their own port PRs. The trait architecture is the seam where a future ML detector slots in without touching `KeywordDetector` or any caller. The canonical extension is documented in `signals/README.md` as a classifier head on the existing `bge-small-en-v1.5` embedder loaded by `relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax, ~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives (distilled tinyBERT in ONNX, logistic regression on lexical features) are kept open in case BGE-head underfits. Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a concrete impl. No NoOp, no MockDetector, no stub-ML -- those will arrive with their real implementations. Phase 3g (Compression Pipeline Formalization, issue #315) is queued as the cross-cutting follow-up that will make lossless-then-lossy- then-CCR ordering an explicit, observable architecture rather than implicit per-compressor logic. Trait shapes there will reuse the signals primitive landed in this PR. |
||
|
|
3f8de4e117 |
fix(smart_crusher): re-land orphaned audit close-out — CCR knob + scorer fail-loud
Re-lands two audit fixes that were marked "merged" on GitHub but never reached main: squash-merging the parent stack changed its commit SHA, which silently dropped the contents of the stacked PRs (#301, #305). Single PR this time — no stacking risk. What lands: 1. **`enable_ccr_marker` Rust gate** — new field on `SmartCrusherConfig` (default `true`). `crush_array` checks it before emitting the `<<ccr:HASH>>` marker text and the CCR store write. PyO3 surface + parity-fixture tolerance updated; recorded fixtures predate the field and inherit the `true` default. 2. **Python shim collapses both flags to the gate** — both `ccr_config.enabled=False` and `ccr_config.inject_retrieval_marker =False` now flip the Rust gate off. Storing a payload nothing in the prompt can reference is pointless, and storing under `enabled=False` would be a surprise side effect the user explicitly opted out of. 3. **Custom `scorer` / `relevance_config` fails loud** — replaces the prior WARNING-and-drop. Silently dropping a user-supplied scorer is a textbook silent fallback. `NotImplementedError` instead. Verified zero production callers pass these args; full plumbing arrives with Stage-3c.2's relevance-crate Python bridge. Tests: - 2 new Rust unit tests in `crusher.rs::tests` - 6 new Python tests in `test_smart_crusher_toin_attachment.py` (3 CCR marker-knob behaviors + 3 scorer fail-loud) - Removed `test_ccr_inject_marker_false_logs_warning` (the WARNING is gone now that the flag is honored) - `make ci-precheck` green; eval suite + observability tests run twice consecutively to verify no TOIN file pollution leaks into the regular+coverage double-run on Python 3.11 RUST_DEV.md audit table reflects both gaps closed. |
||
|
|
049ca9cab2 |
fix(smart_crusher): re-attach TOIN learning loop + audit known regressions
The Stage 3c.1b retirement of the Python SmartCrusher silently disconnected three subsystems. The audit on 2026-04-28 caught them; this commit fixes what's fixable today and labels the rest visibly. ## TOIN learning loop — fixed Before: `ContentRouter._record_to_toin` skipped SmartCrusher on the assumption SmartCrusher recorded its own TOIN events. The retired Python class did. The Rust port doesn't know about TOIN. Net result: the highest-traffic compression strategy stopped fueling the learning loop, silently. Fix: shim's `crush()` and `_smart_crush_content()` now call `toin.record_compression()` after a real compression. Filtered on `strategy != "passthrough"` because the Rust port flips `was_modified=True` from JSON whitespace re-canonicalization. Best effort: TOIN failures are logged at debug level and never break compression. Token estimates use `len(json) // 4` (the rule the retired Python used) because the router doesn't pass a tokenizer down to this layer and re-tokenizing here would dominate the recording cost. 7 tests in `tests/test_smart_crusher_toin_attachment.py`: - crush() records on real compression, doesn't on passthrough - structurally-similar inputs land on the same pattern - _smart_crush_content() records (legacy apply() path) - TOIN errors don't break compression - non-JSON input doesn't record - inject_retrieval_marker=False emits a WARNING ## CCR marker emission knob — labelled, not yet fixed `ccr_config.inject_retrieval_marker=False` is not honored — the Rust port emits `<<ccr:HASH N_rows_offloaded>>` markers in `dropped_summary` unconditionally. Today the production default has the flag True so no one is hitting the gap, but the silent-disconnect was a real regression. Shim now logs a WARNING when callers pass `False` so the mismatch is visible. Fix needs a Rust-side gate; tracked in `RUST_DEV.md`. ## Custom relevance scorer — labelled, not yet fixed `relevance_config` and `scorer` constructor args are accepted for source compatibility but the Rust default `HybridScorer` always runs. Shim was logging this at debug; bumped to WARNING. Tracked. ## RUST_DEV.md New "Known regressions in retired-Python components" section with a table per retired component. The intent is that this section gets updated whenever a regression closes (or a new one is found), so the duplicate-codebase tax stays visible instead of decaying into folklore. |
||
|
|
c2749c0fb6 |
docs(rust): lockfile + RUST_DEV.md for proxy CLI
Cargo.lock: pick up tokio-util added in the WS half-close fix. RUST_DEV.md: document how to run headroom-proxy in passthrough mode (listen + upstream flags, e2e test gate, env vars). |
||
|
|
0414cb70e4 |
feat(rust): scaffold workspace + parity harness (phase-0)
Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |