Commit Graph
16 Commits
Author SHA1 Message Date
99021af20a fix(proxy): make the Rust front's native Bedrock and Vertex routes opt-in (#4851)
## Description

Two routes in the Rust front (`crates/headroom-proxy`) attach the host's
own cloud credentials to whatever request reaches them, without checking
the caller:

- **Bedrock native** (`/model/{id}/invoke`, `/converse`,
`/invoke-with-response-stream`, `/converse-stream`) is on by default and
signs every request with SigV4 using the AWS credentials found on the
host.
- **Vertex** (`…/publishers/anthropic/models/{m}:rawPredict` /
`:streamRawPredict`) is mounted unconditionally, replaces the caller's
`Authorization` with a bearer from the host's Google ADC, and forwards
to the generic `--upstream`, whatever host that is. So with a non-Google
upstream (the Python proxy, another provider) the host's Google token is
sent there. `--vertex-adc-scope` was parsed but never used.

This PR makes both routes explicit opt-ins, only sends the ADC bearer to
a Google API host, and uses the configured scope.

**Dependency:** these routes still do not authenticate callers
themselves. Callers are expected to be authenticated by the inbound
`HEADROOM_PROXY_TOKEN` gate added in #4840 (which also moves the default
listen address to loopback). This PR is based on `main` and does not
require #4840 to merge first, but operators who enable either route on a
non-loopback address need #4840 to restrict who can use them.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `config.rs`: `--enable-bedrock-native` /
`HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` now defaults to `false`. New
`--enable-vertex-native` / `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE`
(default `false`) and `--vertex-allow-non-google-upstream` /
`HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` (default `false`).
`Config::for_test` keeps both routes on so the existing route tests run
unchanged.
- `proxy.rs`: the Vertex route is mounted only when enabled; with it
off, Vertex paths go through the catch-all with the caller's own headers
(as the Python proxy's Vertex passthrough expects). The ADC token source
is built with `--vertex-adc-scope`. The "Bedrock route not mounted"
startup line is now `info`, since off is the default.
- `vertex/raw_predict.rs`: before fetching the ADC bearer, the handler
refuses (502 `vertex upstream is not a Google API host`,
`vertex_upstream_not_google` error log) unless `--upstream` is
`https://` on a `*.googleapis.com` host (regional, global and
`*.p.googleapis.com` endpoints), or the operator set
`--vertex-allow-non-google-upstream=true` (local emulators, gateways).
No token is fetched on that path.
- `RUST_DEV.md`: the three flags in the configuration table, with what
enabling them means.

**Behaviour change for operators:** deployments that used the Rust
front's native Bedrock or Vertex signing must now pass
`--enable-bedrock-native true` / `--enable-vertex-native true` (or the
env vars). Vertex with a non-Google `--upstream` additionally needs
`--vertex-allow-non-google-upstream true`. Without the flags, those
paths are forwarded to `--upstream` unsigned with the client's own
credentials (Claude Code in Vertex mode keeps sending its own ADC token,
which is what `docs/content/docs/claude-code-vertex.mdx` describes).
`scripts/qualify_cargo_dependencies.py` already passes
`--enable-bedrock-native true`.

## Testing

- [x] `cargo test -p headroom-proxy` (45 test binaries, 596 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes

New tests in `tests/integration_cloud_routes_opt_in.rs` build the config
from the real CLI defaults (`CliArgs` → `Config::from_cli`), give the
app host AWS credentials and a host ADC bearer, and record what a
`wiremock` upstream receives:
- `default_config_does_not_sign_bedrock_requests`: no `AWS4-HMAC-SHA256`
Authorization on `invoke`, `invoke-with-response-stream`, `converse`.
- `default_config_never_attaches_host_adc_bearer`: `rawPredict` and
`streamRawPredict` reach the upstream without the host bearer.
- `vertex_bearer_is_not_sent_to_a_non_google_upstream`: with the route
enabled and an `http://127.0.0.1` upstream, 502 and the upstream sees
nothing.
- `vertex_opt_in_still_signs_when_non_google_upstream_is_allowed`: the
explicit escape hatch still works.
- `configured_adc_scope_reaches_token_source`.
- Unit test `vertex::raw_predict::tests::google_api_host_check`
(`http://`, `googleapis.com.example.net`, `example-googleapis.com`,
loopback are rejected).

### Test Output

Before (the three tests that compile against `main`, run with this PR's
`src/` changes stashed):

```text
test configured_adc_scope_reaches_token_source ... FAILED
test default_config_never_attaches_host_adc_bearer ... FAILED
test default_config_does_not_sign_bedrock_requests ... FAILED
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s
```

After:

```text
$ cargo test -p headroom-proxy --test integration_cloud_routes_opt_in
test configured_adc_scope_reaches_token_source ... ok
test vertex_opt_in_still_signs_when_non_google_upstream_is_allowed ... ok
test vertex_bearer_is_not_sent_to_a_non_google_upstream ... ok
test default_config_never_attaches_host_adc_bearer ... ok
test default_config_does_not_sign_bedrock_requests ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s
```

## Real Behavior Proof

- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (c21b0813) and this branch. AWS credentials supplied
to the proxy through `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (fake
values), IMDS disabled. Upstream and Bedrock endpoint are a small Python
recorder on `127.0.0.1:18788` that prints the `Authorization` it
receives.
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:18787
--upstream http://127.0.0.1:18788 --bedrock-endpoint
http://127.0.0.1:18788` (no Bedrock/Vertex flags), then `curl -d
'<anthropic body>'
http://127.0.0.1:18787/model/anthropic.claude-3-haiku-20240307-v1:0/invoke`
with no client credentials. Repeated on this branch with
`--enable-bedrock-native true`.
- Observed result (key id and signature redacted):

```text
== origin/main (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}

== this branch (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": null}

== this branch, --enable-bedrock-native true
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}
```

- Not tested: Vertex against real Google ADC and a real
`*.googleapis.com` upstream (covered by the integration tests with a
static token source; no GCP credentials on the test machine).

## Runtime Rollout Safety

- Rollout-managed feature(s): `native_bedrock` (existing). The CLI flag
already maps `false` to a rollout disable, so with the new default the
feature reports disabled unless `--enable-bedrock-native true` is
passed. Vertex is not rollout-managed.
- Minimum rollout channel: unchanged
- Stable/default behavior changed: yes, both native routes are off by
default
- Kill switch / disable path: the routes are now off unless enabled
- Unsafe override required: no
- Qualification impact: none, the qualification script already passes
`--enable-bedrock-native true`
- Rollback path: pass `--enable-bedrock-native true` /
`--enable-vertex-native true` (plus `--vertex-allow-non-google-upstream
true` if the upstream is not a Google host), or revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4851"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:07:49 -06:00
f5c4217648 fix(proxy): enforce HEADROOM_PROXY_TOKEN in the Rust front and listen on loopback by default (#4840)
## Description

The Rust front (`crates/headroom-proxy`) relays every request to the
Python proxy from `127.0.0.1`. The Python proxy treats a loopback caller
as local: it skips the `HEADROOM_PROXY_TOKEN` check and serves `/admin`,
`/debug` and `/settings` to it. The front had no inbound check of its
own, listened on `0.0.0.0:8787` by default, and dropped
`x-headroom-proxy-token` (as an `x-headroom-*` header), so a token set
on the Python side neither protected the stack nor could be presented
through the front.

This PR makes the Rust front apply the same token model as the Python
proxy (`read_proxy_token` / `scrub_proxy_token_headers` in
`headroom/proxy/server.py`, `headroom/proxy/bind_policy.py`) before
anything is forwarded, and makes its default listen address loopback.

Fix adapted from a proposal by HetCreep.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `crates/headroom-proxy/src/inbound_auth.rs`, installed as a
router-wide layer (all routes, the catch-all and WebSocket upgrades)
when `HEADROOM_PROXY_TOKEN` / `--proxy-token` is set:
- a non-loopback peer (or one with no peer address) must send the token
as `X-Headroom-Proxy-Token` or `Authorization: Bearer`; the explicit
header wins, as in Python. Constant-time compare. Otherwise `401
{"error":"unauthorized"}` and no upstream contact; a
`proxy_auth_rejected` warning is logged (no credential values).
- loopback peers are exempt (same as Python); `GET /healthz`,
`/healthz/upstream`, `/health`, `/livez`, `/readyz` stay open;
`GET`/`HEAD /api/hello` (also under `/p/<project>`) is answered locally
with an empty 200, as Python does, instead of being relayed.
- the token is removed before forwarding: `x-headroom-proxy-token`
always, `Authorization` only when it carries the token (a provider
credential is left alone).
  - the token is redacted from `Debug` output of `Config`.
- `main.rs`: startup bind policy, same rule and variable as Python: a
non-loopback `--listen` with no token exits with code 2 and a message
naming both remedies, unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
(then a warning is logged).
- `config.rs`: `--listen` / `HEADROOM_PROXY_LISTEN` default changes from
`0.0.0.0:8787` to `127.0.0.1:8787`; new `--proxy-token` (env
`HEADROOM_PROXY_TOKEN`, value hidden from `--help`).
- `RUST_DEV.md`: runbook exports a token for the `0.0.0.0` cutover, flag
table updated, new "Inbound authentication" section.

**Behaviour change for operators:** a Rust front started with `--listen
0.0.0.0:...` (or relying on the old default to be reachable from other
hosts) now needs `HEADROOM_PROXY_TOKEN` set, and clients on other hosts
must send it. Setups that front the port with something else that
restricts access can set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`.
Loopback clients (Claude Code, Codex, SDKs on the same host) are
unaffected. The Docker image's entrypoint is the Python proxy, so it is
unaffected.

## Testing

- [x] `cargo test -p headroom-proxy` (46 test binaries, 608 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes

New tests:
- `tests/integration_inbound_auth.rs` (11 tests, `oneshot` with an
injected `ConnectInfo` so a non-loopback peer can be simulated,
`wiremock` upstream): missing/wrong token is 401 and the upstream sees
nothing; correct token via header or Bearer is forwarded without the
token; provider `Authorization` kept when the header carries the token;
loopback and IPv4-mapped loopback peers pass; missing peer address fails
closed; health probes open but `POST /readyz` is 401; `/api/hello`
answered locally; no token configured keeps passthrough.
- `tests/integration_bind_policy.rs` (3 tests, real binary): `0.0.0.0`
with no token refuses to start; loopback starts; default `--listen` is
loopback.
- unit tests in `inbound_auth.rs` for token parsing, constant-time
compare, bind policy and exemptions.

### Test Output

Before (the bind-policy test file copied onto a clean `origin/main`
worktree; `integration_inbound_auth.rs` does not compile there since
main has no token setting at all):

```text
test default_listen_address_is_loopback ... FAILED
test loopback_listen_without_token_still_starts ... ok
test non_loopback_listen_without_token_refuses_to_start ... FAILED
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.01s
```

After:

```text
$ cargo test -p headroom-proxy --test integration_inbound_auth --test integration_bind_policy
test default_listen_address_is_loopback ... ok
test non_loopback_listen_without_token_refuses_to_start ... ok
test loopback_listen_without_token_still_starts ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.01s
test missing_peer_address_fails_closed ... ok
test remote_peer_with_bearer_token_is_forwarded_without_it ... ok
test remote_peer_without_token_gets_401_and_upstream_is_not_contacted ... ok
test token_header_wins_over_a_provider_bearer ... ok
test no_token_configured_keeps_todays_passthrough ... ok
test loopback_peer_without_token_is_still_forwarded ... ok
test ipv4_mapped_loopback_peer_counts_as_loopback ... ok
test remote_peer_with_token_header_is_forwarded_without_it ... ok
test hello_probe_is_answered_locally_without_token ... ok
test remote_peer_with_wrong_token_gets_401 ... ok
test health_probes_stay_reachable_without_token ... ok
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.74s
```

## Real Behavior Proof

- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (c21b0813) and from this branch. Upstream is a small
Python HTTP recorder on `127.0.0.1:18788` that prints each request it
receives. Requests are sent with curl to the machine's LAN address
(non-loopback peer) and to `127.0.0.1`.
- Exact command / steps: `HEADROOM_PROXY_TOKEN=s3cret
HEADROOM_PROXY_STATS=false headroom-proxy --listen 0.0.0.0:18787
--upstream http://127.0.0.1:18788`, then `curl
http://<lan-ip>:18787/admin/upstream` with no token, a wrong token and
the right token in `x-headroom-proxy-token`, `GET /readyz` with no token
from the LAN address, and `GET /v1/models` from `127.0.0.1` with no
token.
- Observed result:

```text
== origin/main
GET /admin/upstream (no token)                     -> 200
GET /admin/upstream -H x-headroom-proxy-token:wrong -> 200
GET /admin/upstream -H x-headroom-proxy-token:s3cret -> 200
GET /readyz (no token)                             -> 200
GET /v1/models from 127.0.0.1 (no token)           -> 200
UPSTREAM SAW {"path": "/admin/upstream", "peer": "127.0.0.1", "token_hdr": null, "authorization": null}   (x3)
UPSTREAM SAW {"path": "/readyz", ...}
UPSTREAM SAW {"path": "/v1/models", ...}

== this branch
GET /admin/upstream (no token)                     -> 401
GET /admin/upstream -H x-headroom-proxy-token:wrong -> 401
GET /admin/upstream -H x-headroom-proxy-token:s3cret -> 200
GET /readyz (no token)                             -> 200
GET /v1/models from 127.0.0.1 (no token)           -> 200
UPSTREAM SAW {"path": "/admin/upstream", "peer": "127.0.0.1", "token_hdr": null, "authorization": null}   (only the authenticated one)
UPSTREAM SAW {"path": "/readyz", ...}
UPSTREAM SAW {"path": "/v1/models", ...}

$ headroom-proxy --listen 0.0.0.0:18787 --upstream http://127.0.0.1:18788   # no token
headroom-proxy: Refusing to listen on 0.0.0.0:18787: with no HEADROOM_PROXY_TOKEN this proxy would relay every network peer to the upstream Headroom proxy from 127.0.0.1, where it is treated as a trusted local caller. Either set HEADROOM_PROXY_TOKEN ... or ... set HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1 to acknowledge the open bind.
(exit 2)
```

- Not tested: a full chain with the real Python proxy behind the Rust
front (the recorder stands in for it); WebSocket upgrades through the
gate are covered only by the layer wrapping the catch-all, not by a
dedicated test.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes, default listen address is now
loopback, and a non-loopback listen needs a token or the acknowledgement
variable
- Kill switch / disable path: `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
restores starting on a non-loopback address without a token; `--listen
0.0.0.0:8787` restores the old address
- Unsafe override required: no
- Qualification impact: none (qualification scripts already pass
`--listen 127.0.0.1:<port>`)
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

A companion PR makes the Rust front's native Bedrock and Vertex routes
opt-in; it relies on this token gate for callers on those routes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-17 09:47:48 +05:30
Yasser SheikhandJerrettDavis 70b21c12c4 feat(proxy): native spend/savings stats + dashboard for the Rust proxy (#2321)
# feat(proxy): native spend/savings stats + dashboard for the Rust proxy

## Description

The Rust proxy has no cost observability of its own today: `/stats` and
`/dashboard` only exist on the Python proxy, so anyone running
`headroom-proxy` standalone (the Bedrock/Vertex lanes especially) is
flying blind on spend.

This adds it natively. Every lane the Rust proxy forwards — Anthropic
`/v1/messages`, OpenAI Chat + Responses, all four native Bedrock routes,
Vertex `rawPredict`/`streamRawPredict` — records per-request cost/usage
telemetry into one in-process ledger, served from `GET /stats`, `GET
/stats/timeseries`, `GET /stats/events`, and an embedded,
dependency-free `GET /dashboard`.

This is a ground-up redesign of #1057 (approved but never merged, now
badly stale against `main`), not a rebase of it. Supersedes #1057. The
"Design" section below states exactly what was kept and what was
deliberately dropped, so it can be reviewed on its own terms.

### What it measures today

Spend, input/output/cache-read/cache-write tokens, prompt-cache hit rate
and its NET dollar benefit, per-provider and per-model breakdowns,
failures, latency, and hourly/daily history that survives restarts.

Two numbers are computed carefully rather than the obvious way, because
the obvious way flatters the product:

- **Spend counts output.** Input and output are priced separately
(`input_cost_usd` / `output_cost_usd`, summed into `total_cost_usd`)
because output bills at its own rate — typically 4-5x input. Counting
input alone would understate a short-prompt/long-answer turn by most of
its cost.
- **Cache savings are NET**, not the gross read discount: the read
discount minus the cache-write premium (writes bill *above* list price),
floored at $0 per request. Reporting gross reads overstates the benefit
on write-heavy traffic.

### What reads 0 today — stated up front

`tokens_saved` / `compression_savings_usd` will read **0 on real
traffic**, because the Rust live-zone per-type compressors are still
no-ops on `main` — there is nothing to record yet. Verified against the
real proxy: a 13.8 KB body with a large `tool_result`, sent with
`--compression --compression-mode live_zone`, produced `tokens_before:
0` (dispatcher returned `NoCompression`), which is the invariant
`live_zone_mode_with_valid_body_returns_no_compression_pr_b2` pins. The
recording path is wired and unit-tested against `Outcome::Compressed`,
and starts populating on its own the moment those compressors land — no
changes needed here. Stating it up front so `Saved $0` isn't read as a
bug.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Design — how this differs from #1057

- **One fold, many views.** A single `Totals::fold` is the only place
request accounting happens; lifetime, since-boot session, per-provider,
per-model, hourly and daily buckets are that same fold under different
keys, so the views cannot disagree. #1057 (like the Python tracker it
ported) maintained each aggregate separately.
- **Its own schema, not Python emulation.** `/stats` is a documented
Rust-native payload (`version: 1` on disk). No `schema_version: 5` file
compat, no `display_session` idle-window emulation, no `projects`
passthrough. The five keys external unified-stats consumers already read
from this endpoint (`summary.api_requests`, `requests.failed`,
`requests.cached`, `tokens.saved`, `tokens.proxy_compression_saved`) are
kept and pinned by a test.
- **No federation flag.** #1057's `--upstream-stats-url` supplemental
merge is dropped (YAGNI — the deployed consumer merges in the opposite
direction).
- **No new price data file.** USD valuation reuses the LiteLLM table
already vendored for `compression::model_limits`. #1057 shipped a
second, models.dev-shaped price book.
- **Purpose-built dashboard.** One self-contained HTML file designed for
what the Rust proxy actually measures, instead of embedding the Python
template and rendering its ~20 Python-only panels as zeros.
- **Not gated on compression.** `--compression` defaults to **off**, so
recording gated behind the compression buffer (as #1057 had it) records
nothing under default config. Here the model is sourced from the
*response* when the request body isn't buffered, so spend/cache
observability works on a pure passthrough deployment — with zero change
to the byte path.

## Correctness rules (carried from #1057's review, all test-pinned)

- **Deferred recording** — a request records only when its response is
fully observed, so streaming token counts come from real SSE/EventStream
usage frames instead of 0.
- **Failed upstreams accrue nothing** — counted in `requests`/`failed`
only; retries can't inflate savings.
- **No disk I/O on the request path** — `record()` folds under a mutex
and sets a dirty flag; a background task (and shutdown) writes an atomic
snapshot (temp → fsync → rename → fsync dir), re-arming the dirty flag
on failure.
- **Bounded everything** — per-model cardinality caps with `"other"`
overflow, length-clamped client strings, a 50M-token per-event ceiling,
bounded capture queues/buffers, bounded warn-once set.
- **Honest pricing** — exact-match only; a generic id misses (and WARNs)
rather than mispricing. Unknown ⇒ $0, never a blended fallback.
- **Bedrock regional pricing respected** — the unmodified id is tried
first, because `eu.anthropic.claude-3-5-haiku…` genuinely bills $0.25/M
against the base id's $0.80/M. Geo-stripping is only a fallback for
regions the table doesn't list.

## Pre-existing bugs fixed along the way

Both were found while wiring the recording hooks; both affect `main`
today, independently of stats.

- **Bedrock geo-prefix coverage**: `au.`, `jp.`, and `us-gov.`
cross-region inference profiles weren't recognized, so Anthropic models
invoked through them silently skipped compression — the same failure
mode #953 reported and #999 fixed for `eu.`/`us.`/`apac.`/`global.`. The
list is now complete (all seven published geos) and shared
(`bedrock::vendor::GEO_PREFIXES`) between the vendor gate and pricing,
so the two can't drift apart again.
- **EventStream `Skip` frame stranding**: the streaming parser pulled a
fresh upstream chunk after each unknown (`Skip`ped) event type, delaying
— and at stream end **dropping** — frames already buffered. It now
drains the buffer to exhaustion before awaiting more bytes.
Regression-pinned by
`skip_in_final_chunk_does_not_strand_buffered_frames`.

## Changes Made

- `crates/headroom-proxy/src/observability/ledger.rs` — the store:
`Totals` fold, bounded maps/buckets, atomic snapshot persistence,
payload builders
- `crates/headroom-proxy/src/observability/pricing.rs` — deterministic
lookup over the vendored LiteLLM table (no new data file)
- `crates/headroom-proxy/src/observability/capture.rs` — `PendingRecord`
(deferred finalization), usage extraction for every wire shape, bounded
tee tasks off the byte path
- `crates/headroom-proxy/src/observability/stats_http.rs` +
`dashboard.html` — HTTP surface + embedded self-contained UI
- `crates/headroom-proxy/src/proxy.rs`, `bedrock/invoke.rs`,
`bedrock/invoke_streaming.rs`, `vertex/raw_predict.rs` — per-lane
recording hooks
- `crates/headroom-proxy/src/config.rs` — `--stats` (default on),
`--stats-path`
- `crates/headroom-proxy/src/bedrock/vendor.rs` — shared `GEO_PREFIXES`
(the geo-coverage fix above)
- `crates/headroom-proxy/src/sse/framing.rs` — 4 MiB accumulator cap
(telemetry-side; a stream with no event terminator can no longer grow
the buffer unboundedly)

`--stats=false` restores byte-for-byte pre-feature behaviour (paths
tunnel upstream, nothing recorded) — pinned by
`stats_disabled_falls_through_to_upstream`.

## Testing

- [x] `cargo fmt --all -- --check` passes
- [x] `cargo clippy --workspace -- -D warnings` passes (0 warnings)
- [x] `cargo test --workspace` passes
- [x] New tests added for new functionality (53 new tests across unit +
integration)
- [x] Manual testing performed (live AWS Bedrock, all four native lanes
— see Real Behavior Proof)

Unit: fold-consistency across views, failed-no-savings, pricing
(regional/revision/path forms, collision-miss, untracked-region
fallback), USD math, bounded maps, token-ceiling clamp,
bidi/control-char stripping, atomic flush with no temp litter,
corrupt-file preservation, oversized-state rejection, negative-USD
clamping on load, restart round-trip, week/month rollups, hostile field
lengths. Integration (wiremock): endpoint contracts, `--stats=false`
tunneling, deferred SSE capture, non-streaming capture, OpenAI
cached-token normalisation, failed-upstream accounting, hostile-usage
clamping, all four Bedrock lanes, Vertex, dashboard self-containment.

The persisted-state tests are deliberately adversarial rather than
round-trip only, since the ledger reads a file an operator can
hand-edit: malformed bucket keys (including multi-byte ones that would
panic a naive `&str` slice) are dropped before the age-trim runs,
hostile model-id keys are re-clamped on load, and an implausibly large
state file is treated as corrupt instead of allocated.

### Test Output

```text
$ cargo fmt --all -- --check
  (clean)

$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
  exit=0   0 warnings, 0 errors

$ cargo nextest run -p headroom-proxy
     Summary [4.396s] 488 tests run: 488 passed, 0 skipped

$ cargo nextest run -p headroom-core -p headroom-simulators -p headroom-parity
     Summary [49.535s] 967 tests run: 967 passed, 1 skipped

$ cargo clippy --workspace -- -D warnings
  exit=0   0 warnings, 0 errors

$ cargo test --workspace --doc
  ok, 0 failed
```

## Real Behavior Proof

- Environment: live AWS Bedrock, staging account (SSO profile — name
redacted), `eu-central-1`, `anthropic.claude-3-haiku-20240307-v1:0`,
release binary, compression off (shipped default)
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:8790
--enable-bedrock-native --bedrock-region eu-central-1` then one request
per lane (`/invoke`, `/converse`, `/invoke-with-response-stream` with
`Accept: application/vnd.amazon.eventstream`, `/converse-stream`), one
deliberate failure (nonexistent model), `SIGTERM`, restart, re-read
`/stats` — full transcript, raw proxy logs, and the persisted ledger are
attached below
- Observed result: all four lanes captured usage (input tokens = 48 = 4
× 12; spend $0.000012 = 48 × $0.25/M, exact for Bedrock Haiku 3); the
failure recorded `failed: true` with 0 tokens and $0; lifetime totals
survived the restart while session totals reset; dashboard rendered
against this live data (screenshots attached)
- Not tested: live Vertex (covered by wiremock integration tests only —
no GCP staging account available); live OpenAI Chat/Responses lanes
(same wiremock coverage); dashboard on Windows browsers

The four Bedrock lanes are the ones worth proving live, because two of
them had no parsing before this PR:

| Lane | Wire shape | Captured |
|---|---|---|
| `/invoke` | sync, snake_case `usage` | 12 in / 4 out |
| `/converse` | sync, **camelCase** `usage` | 12 in / 4 out |
| `/invoke-with-response-stream` | **binary EventStream passthrough**,
base64-wrapped chunks | 12 in / 4 out |
| `/converse-stream` | SSE-translated; usage lives in the camelCase
`metadata` frame the Anthropic state machine drops | 12 in / 4 out |

```text
requests      : 5 (failed: 1)
input tokens  : 48
output tokens : 16
cache read/wr : 0 / 0
input spend   : $0.00001200      # 48 × $0.25/M
output spend  : $0.00002000      # 16 × $1.25/M — output bills 5x here
TOTAL spend   : $0.00003200      # what /dashboard reports as "Spend"
savings       : $0.00000000      # compression off -> 0 by design
avg latency   : 256 ms
by provider   : {'bedrock': 5}
by model      : {'anthropic.claude-3-haiku-20240307-v1:0': 4,
                 'anthropic.claude-nonexistent-v99:0': 1}
persistence   : True | healthy: True
```

Those two spend lines are the reason output pricing is in this PR:
**output was 62.5% of the real bill** on this run. Reporting input alone
would have called a $0.000032 spend `$0.000012`.

The exposure split was exercised against the same live proxy:

```text
$ curl /stats/events  -H 'Host: 127.0.0.1:18787'   -> 200   (local tooling works)
$ curl /stats/events  -H 'Host: attacker.com'      -> 404   (DNS-rebound: peer IS loopback)
$ curl /stats         -H 'Host: attacker.com'      -> recent_requests: null
                                                      persistence.path: null
                                                      requests.total still served
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(RUST_DEV.md, CHANGELOG.md)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Known limitations

1. Compression savings read 0 until the live-zone compressors land
(above).
2. Hourly resolution is 48 h, daily ~13 months; older traffic folds into
lifetime totals only. `/stats/events` is an in-memory ring of 100 and is
not persisted — this is a cost ledger, not an audit log.
3. The vendored price table lacks some legacy direct-API ids
(`claude-3-5-sonnet-20241022` is absent while its Bedrock form is
present); those record $0 with a WARN until refreshed via
`scripts/refresh_model_limits.sh`.
4. `output_cost_usd` is absent from state files written before this
change, so it loads as 0: historical spend keeps reading input-only
until those buckets age out (48 h hourly / ~400 d daily). New traffic is
priced in full immediately. Deliberately no migration shim — the field
defaults cleanly and the alternative is retro-pricing history against
today's price table, which would invent numbers.
5. A non-streaming response body over 2 MiB isn't buffered for usage
parsing (the request still records, compression-side numbers only).
Streaming responses have no such cap — they parse incrementally.
6. The endpoints carry no auth, so exposure is split by tier instead
(matching the Python proxy): aggregates are public like `/metrics`,
while per-request rows (`recent_requests`, `/stats/events`) and the
ledger's filesystem path require a local caller — loopback peer **and**
loopback `Host`, the latter being the DNS-rebinding defence. Remote
viewers of `/dashboard` therefore see totals but not the request feed,
and the page says so. Opening that tier to a trusted reverse proxy is
#1959's call to make for both proxies; this PR doesn't invent a second
scheme.
7. Copilot traffic isn't proxied by the Rust binary; it reaches this
dashboard only via external unified-stats layers (unchanged).

## Screenshots (if applicable)

Attached below: live-test terminal transcript, dashboard rendered
against the live Bedrock data (session + lifetime scopes, before/after
restart), raw proxy logs, and the persisted ledger file.

## Additional Notes

- Related issues/PRs: supersedes #1057; completes the geo-prefix
coverage from #999 (design discussion in #953 — the multi-vendor adapter
work proposed there remains out of scope).
- `requests.cached` counts requests whose response reported
`cache_read_tokens > 0`, matching the Python tracker's derivation, so
unified-stats consumers see consistent semantics across both proxies.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2321"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-07 20:58:22 -05:00
Laura MoneyandJerrettDavis a05717f6ec fix(proxy): redact upstream error detail and add opt-in /metrics loopback gate (#2589)
## Description

Two response-path information-exposure hardenings for the native Rust
reverse proxy (`crates/headroom-proxy`). Both are defensive and
non-breaking; neither adds a user-facing capability nor changes
forwarding behaviour, and defaults are unchanged.

1. **Redact upstream error detail from client responses.**
`ProxyError::into_response` returned `self.to_string()` as the client
body, which for `reqwest` upstream errors interpolates the upstream
URL/host/port. It now returns a generic, stable message per status while
the full detail is retained in the server-side `tracing::warn!` log.
`413` intentionally stays descriptive (no internal detail; retry-on-413
clients rely on recognising it).
2. **Opt-in loopback gate for `/metrics`.** New
`--metrics-require-loopback` / `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK`
(default `false`). When enabled, the Prometheus scrape returns `403` to
non-loopback peers. This is defense-in-depth for deployments that bind a
non-loopback `--listen`. Default-off preserves current behaviour
exactly.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `error.rs`: client-facing bodies are now generic per status; full
detail stays in the log only.
- `config.rs`: add `--metrics-require-loopback` /
`HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK` (default `false`).
- `proxy.rs`: mount `/metrics` in its own sub-router so the optional
`require_metrics_loopback` layer gates only that route (403 to
non-loopback peers when enabled).
- `RUST_DEV.md`: document the new flag in the configuration table.
- Tests: new `tests/integration_metrics_loopback.rs` plus unit tests for
the gate (v4/v6 loopback and non-loopback) and for error-body redaction.

## Testing

<!-- This PR touches only the Rust `headroom-proxy` crate; the
pytest/ruff/mypy items are Python and N/A here (see Additional Notes).
Validated with cargo on the repo-pinned toolchain. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo fmt --all -- --check          # clean
$ cargo clippy --workspace -- -D warnings   # 0 warnings
$ cargo test --workspace
test result: ok. 1432 passed; 0 failed; 0 ignored   (aggregate across all suites)
$ make test-parity
[tokenizer/smart_crusher/content_detector/text_crusher/diff/log/search] diffed=0
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.3.0), Rust 1.95.0 (repo-pinned
`rust-toolchain.toml`), `headroom-proxy` debug binary run directly (no
mocks), dead upstream `http://127.0.0.1:9`.
- Exact command / steps: started with
`HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK=true --listen 0.0.0.0:18787`,
then `curl http://127.0.0.1:18787/metrics`, `curl
http://192.168.1.184:18787/metrics` (LAN IP, non-loopback), and `curl -X
POST http://127.0.0.1:18787/v1/messages -d
'{"model":"x","messages":[]}'`; then restarted with the gate off and
repeated the non-loopback `/metrics` scrape.
- Observed result: loopback `/metrics` returned `200`; non-loopback
`/metrics` returned `403` with body `metrics endpoint restricted to
loopback`; the dead-upstream POST returned `502` with client body
`upstream connection failed` while the server log retained the full
detail (`... error sending request for url
(http://127.0.0.1:9/v1/messages)`), i.e. the upstream URL is logged, not
returned to the client; with the gate off, the non-loopback `/metrics`
scrape returned `200` (default behaviour unchanged).
- Not tested: live cloud upstreams (Bedrock/Vertex/OpenAI/Anthropic real
endpoints); the Python proxy path (unchanged by this PR); Windows/Linux
(macOS only).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A (server-side behaviour; see Real Behavior Proof for command output).

## Additional Notes

- This PR modifies only the Rust `headroom-proxy` crate (plus its docs),
so the Python-oriented Testing checkboxes (`pytest`, `ruff`, `mypy`) are
N/A; validation was via `cargo fmt`/`clippy`/`test` and `make
test-parity` on the repo-pinned toolchain, per the Test Output above.
- No new dependencies. Both changes are non-breaking and leave all
defaults unchanged (the `/metrics` gate is opt-in).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2589"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-05 15:29:31 -05:00
Tejas ChopraandClaude Opus 5 96dfc7005f chore: finish the org and docs-domain migration (#3389)
Two stale identities left over from moves that were only half applied.

## GitHub org

`install/state.py` already migrated Docker images to
`ghcr.io/headroomlabs-ai/headroom` and carries a
`_DEPRECATED_IMAGE_REPO` constant that rewrites the old path on load —
so this move was decided long ago. Everything else still said
`chopratejas/headroom`.

Rewritten across packaging (`pyproject.toml`, `Cargo.toml`), both plugin
marketplaces, the agent-hooks manifests, install scripts, CI templates,
crate docs and tests.

**Two deliberate exclusions:**
- **`ghcr.io/chopratejas/headroom` is untouched everywhere.** It is the
*input* to `_migrate_deprecated_image()` and to the tests covering it.
Rewriting it would silently disable the migration.
- **`CHANGELOG.md` keeps its 238 historical references.** GitHub's
rename redirect resolves them, and rewriting would bury this diff in
noise.

**HuggingFace model ids are NOT affected and were never in scope.**
`chopratejas/kompress-base`, `kompress-v2-base`, `kompress-finance`,
`technique-router`, `technique-router-onnx` and
`siglip-image-encoder-onnx` are account-scoped identifiers on a
different service. All six were validated against the HF API before this
change:

| Model | HTTP |
|---|---|
| `kompress-base` | 200 |
| `kompress-v2-base` | 200 |
| `technique-router` | 200 |
| `technique-router-onnx` | 200 |
| `siglip-image-encoder-onnx` | 200 |
| `kompress-finance` | **401 (private)** |

The 401 is unrelated to this migration, but worth knowing:
`kompress_compressor.py:1255` references it, so that path is not
reachable by an OSS user without access.

**One functional fix, not cosmetic:** `_marketplace_source()`
(`headroom/cli/init.py`) returned the old org as its fallback, so plugin
installs were pointed at the retired repo.

## Docs domain

The site moved to `docs.headroomlabs.ai`, but eleven references still
named the vercel.app host — including **`llms.txt` (27 links)**, which
is served at `docs.headroomlabs.ai/llms.txt` and read by LLM crawlers,
the **dashboard's own Documentation link**, and a `headroom wrap` help
message. The docs app's canonical-URL fallbacks were fixed separately in
#3388.

## Also folded in

`RUST_DEV.md` corrections found while verifying the above:
- `make build-wheel` points at `Cargo.toml`, not `pyproject.toml`
- the Rust toolchain is pinned, not tracking `stable`
- `headroom-simulators` was missing from the workspace listing
- the Python `CompressionStore` defaults to **SQLite**, not in-memory,
when `HEADROOM_CCR_BACKEND` is unset (verified by execution) — this one
matters for multi-worker reasoning

And the install scripts pointed at `docs/docker-install.md`, a path that
no longer exists; they now point at the live docs page.

## Verification

297 tests pass, including the plugin-manifest, install-state and
release-workflow suites that assert on these exact strings. `ruff check`
clean. Unrelated lockfile churn (`uv.lock`, both `package-lock.json`) is
deliberately excluded.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 15:53:35 +05:30
Tejas Chopra a639540959 chore: remove committed node_modules + stray/internal markdown (repo hygiene) (#1528)
## Description

Repo hygiene for a public OSS project: removes committed `node_modules`,
stray/internal/draft markdown, and commercial-surface references —
keeping every real doc (the published docs site, the wiki guides, and
all component READMEs) intact. Every file was content-audited before
removal, and load-bearing files were verified against the code/CI and
kept.

Net: **1,695 files changed, +23 / −266,409** (the deletions are
dominated by a committed `node_modules` tree).

Closes # (no tracking issue)

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

**Removed (verified to have no code/CI dependencies):**
- `examples/vercel-ai-sdk-pr/` — 1,649 committed `node_modules` files
(zero example source); `node_modules/` added to `.gitignore`.
- `docs/spec/` (23 draft "Living Specification" files — orphaned,
`1.0.0-draft`, drifted from the code), `docs/superpowers/` (2 agent
plans), `docs/proposals/` (2 internal/commercial memos).
- 6 orphan `docs/*.md` (auth-modes, bedrock,
claude-code-vertex-headroom, cortex-code, output-token-reduction-guide,
rtk-loop-weighting).
- `PR.md` (committed PR draft), `ENTERPRISE.md`, `.github/FUNDING.yml`.

**Content scrubs:**
- Removed unreleased "Headroom Cloud" / `api.headroom.ai` / `hr_`
references from `configuration.mdx`, `wiki/configuration.md`,
`wiki/typescript-sdk.md`, `sdk/typescript/README.md` (reworded to
neutral, accurate phrasing).
- Dropped a stale "awaiting maintainer before merge" line from
`plugins/headroom-oauth2/SPEC.md`; tidied `.gitignore` comments (kept
the protective `headroom-managed/` ignore rule).
- Fixed the now-dangling links into removed files (README
nav/`output-token-reduction` link, `scripts/README`, `wiki/vertex`).

**Explicitly KEPT (load-bearing — would orphan in-code citations if
removed):**
- `.changelog.md` — consumed by `.github/workflows/release.yml` (read as
the release-notes file).
- `REALIGNMENT/`, `docs/observability.md`, `docs/rtk-architecture.md`,
`wiki/plans/`, `TESTING-copilot-subscription.md` — referenced by the
Rust core / Python / tests as design docs.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [ ] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Docs/markdown + .gitignore only — no Python/Rust source changed, so the
# behavioral test suite is unaffected. Verified the cleanup did not orphan
# references or break the published docs site:

$ git ls-files 'docs/content/docs/*.mdx' | wc -l      # published site intact
42
$ # meta.json nav unchanged; no published page removed.

$ grep -rnI "Headroom Cloud|api.headroom.ai|'hr_" $(git ls-files '*.md' '*.mdx')
>>> none

$ # dangling refs to removed files (excl pre-existing P0/P2 spec stubs that
$ # never existed in git): none remaining.
```

## Real Behavior Proof

- Environment: macOS, local git clone of the repo (markdown/.gitignore
changes only — no runtime).
- Exact command / steps: 4 read-only content-audit agents classified
every `.md`/`.mdx` file; each removal candidate was cross-checked
against the codebase (`grep` for citations in `.rs`/`.py`/tests,
workflows, and configs); only files with no dependents were removed; the
tree was re-grepped after removal to confirm no new dangling references;
verified the published docs site page count (`git ls-files
'docs/content/docs/*.mdx' | wc -l` = 42, unchanged).
- Observed result: the 42-page published docs site and all wiki guides
are untouched; no source or workflow references a removed file;
`.changelog.md` (consumed by release.yml) and the code-cited design docs
were detected as dependencies and kept; the committed `node_modules`
tree is removed and `node_modules/` is gitignored so it can't be
re-committed; zero "Headroom Cloud"/`headroom.dev` references remain.
- Not tested: N/A — no executable code changed (only markdown, `.mdx`,
and `.gitignore`), so the behavioral test suite is unaffected.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- This branch deletes `.github/FUNDING.yml` while PR #1526 edits it —
the two will be sequenced at merge (delete wins).
- A follow-up option (not in this PR): also remove the internal design
docs that are currently cited by the code (`REALIGNMENT/`,
`docs/observability.md`, `docs/rtk-architecture.md`, `wiki/plans/`) —
that requires scrubbing ~15–20 in-code citations so nothing dangles, so
it's deliberately deferred.
- Untracked local working files (`benchmarks/hf_pilot/`,
`tools/copilot-test/`) are intentionally left out of git (not
committed).
2026-06-27 23:32:54 -07:00
Patrick AandJD Davis d76a7296df fix(proxy): make CCR multi-worker warning conditional on backend (#770)
## Problem

The multi-worker startup warning always mentioned CCR retrieval
failures, even when the operator had already configured a cross-worker
backend via `HEADROOM_CCR_BACKEND`. That's noise — if they've set
`HEADROOM_CCR_BACKEND=sqlite` or `redis`, CCR fragmentation is already
resolved.

This was surfaced during review of #628 (now closed): the reviewer
correctly noted that Python `CompressionStore` defaults to
`InMemoryBackend`, which is per-process — each uvicorn worker has its
own singleton, so CCR markers written on worker A are invisible to
worker B unless a shared backend is configured.

## Changes

### `headroom/proxy/server.py`

The `workers > 1` warning is now conditional on `HEADROOM_CCR_BACKEND`:

- **Backend unset (default `InMemoryBackend`, per-process):** warning
includes CCR retrieval failures and suggests
`HEADROOM_CCR_BACKEND=sqlite` to use a shared cross-worker store.
- **Backend configured (`sqlite`/`redis`):** warning covers only the
remaining per-worker stores (compression cache, prefix tracker, TOIN,
CostTracker) — CCR fragmentation is already resolved.

### `RUST_DEV.md`

Updated the multi-worker fragmentation section:
- Removed the incorrect parenthetical claiming this only applies when
the operator *explicitly* chooses `CcrBackendConfig::InMemory` (Python
defaults to InMemory)
- Added Python `CompressionStore` as item 1 in the fragmented-state
list, with a note that setting `HEADROOM_CCR_BACKEND=sqlite` resolves it
- Restored TOIN to the fragmented list with a note that its file-backed
snapshots do not make it coherently shared across workers
- Updated "Detecting it in the wild" to document the conditional warning
behaviour

## Files changed

| File | Change |
|---|---|
| `headroom/proxy/server.py` | Conditional two-branch warning based on
`HEADROOM_CCR_BACKEND` |
| `RUST_DEV.md` | Accurate per-process description of Python
`CompressionStore`; restored TOIN |
| `CHANGELOG.md` | Entry under `[Unreleased]` |

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-06-11 18:59:11 -05:00
pratikbin 42b1cd24de docs: fix env var discrepancies across markdown files
Audit all .md files against codebase; fix wrong names, remove phantom
variables, and correct outdated values:

- HEADROOM_PROXY_PORT → HEADROOM_PORT (proxy.py envvar="HEADROOM_PORT")
- HEADROOM_BIND → HEADROOM_HOST + HEADROOM_PORT (RUST_DEV.md)
- HEADROOM_LEARN_{CLAUDE,CODEX,GEMINI}_ENABLED → HEADROOM_LEARN_CLI
  (only HEADROOM_LEARN_CLI exists in learn/analyzer.py)
- HEADROOM_TRACING_ENABLED → HEADROOM_LANGFUSE_ENABLED=1 with correct
  LANGFUSE_PUBLIC_KEY/SECRET_KEY vars (tracing.py)
- HEADROOM_LOG_LEVEL/LOG_FORMAT → --log-level CLI flag / RUST_LOG
  (no HEADROOM_LOG_LEVEL var exists in code)
- HEADROOM_LOG_LEVEL/HEADROOM_STORE_URL/HEADROOM_DEFAULT_MODE rows
  removed from wiki/configuration.md (all phantom)
- HEADROOM_SUMMARY_{ENABLED,THRESHOLD,RATIO} noted as not yet
  implemented (no code exists)
- HEADROOM_DB_URL/HEADROOM_CACHE_BACKEND → explanatory notes pointing
  to HEADROOM_WORKSPACE_DIR (no external DB support in code)
- HEADROOM_DB_PATH/HEADROOM_CACHE_PATH table rows replaced with actual
  HEADROOM_WORKSPACE_DIR/CONFIG_DIR (paths.py)
2026-05-29 15:04:13 +05:30
chopratejas 00902b8fea fix: B7 — CCR hardening: persistent backends + always-on tool
P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store
that fragmented across uvicorn workers and was wiped on restart, and
the `headroom_retrieve` tool was registered/unregistered per-request
based on whether the latest body happened to contain compression
markers — every flip busted the prompt cache. Both are sticky
side-channels: once a session has done CCR, the tool list bytes and
the retrieval store must stay stable. This PR fixes both.

Rust:
* Split `ccr.rs` into `ccr/` with `backends/` submodule
  (`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated).
* `SqliteCcrStore` (production default): WAL mode, prepared upsert,
  lazy TTL purge on read, persistent across worker restarts and
  shareable across workers on the same host via SQLite file locking.
* `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with
  startup PING smoke-test, no key-prefix collision risk, no sticky
  session required at the LB.
* `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)`
  factory — every init failure surfaces (no silent fallback per
  `feedback_no_silent_fallbacks.md`).
* `ccr::compute_key` (BLAKE3 → first 24 hex chars) and
  `ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash +
  marker format; one definition for the live-zone dispatcher and the
  Python regex (`headroom/ccr/tool_injection.py:211`).
* `compress_anthropic_live_zone_with_ccr` accepts
  `Option<&dyn CcrStore>`. When wired, every accepted compression
  puts the original bytes into the backend and appends `<<ccr:HASH>>`
  to the compressed string. The token-validation gate runs on the
  marker-augmented string so the `compressed_tokens >=
  original_tokens` rejection stays honest.

Python:
* `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the
  PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools`
  pattern: once a session has done CCR, every subsequent request
  injects the recorded golden tool-definition bytes. Tool list bytes
  are byte-stable across turns (snapshot test pins them).
* `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a
  new `session_has_done_ccr` kwarg per the PR-B7 spec change at line
  302-328. The legacy per-request path stays intact for callers that
  don't yet thread a session id (e.g. Google handler).
* Anthropic + OpenAI handlers route their CCR tool-list updates
  through `apply_session_sticky_ccr_tool`, keyed off the existing
  `session_tracker_store.compute_session_id(...)` plumbing.

Backend selection model: `CcrBackendConfig::Sqlite { path }` is the
production default — single host, persistent, multi-worker safe with
sticky session. `CcrBackendConfig::Redis { url }` is the multi-host
scale-out option — no stickiness needed. `InMemory` is for tests
and single-worker dev only. RUST_DEV.md "Multi-worker deployment —
CCR fragmentation" rewritten around this matrix.

Tests:
* `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering
  SQLite round-trip, TTL purge, proxy-restart survival, cross-backend
  byte-equal keys, `from_config` paths, and the no-redis-feature
  loud-failure check (+ 2 redis tests gated behind the feature).
* `crates/headroom-core/tests/live_zone_ccr.rs` — confirms
  `<<ccr:HASH>>` marker injection, store population, and
  no-marker-when-no-store invariants end-to-end.
* `tests/test_ccr_tool_always_on.py` — 12 tests pinning the
  always-on behaviour, session/provider isolation, LRU bound, no-
  session-id fallback, and (per-acceptance-criterion) the byte-stable
  tool-definition snapshot.

Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 16:52:33 -07:00
chopratejas ea78cf6252 fix(proxy): cache concurrency lock, multi-worker docs, bounded compression executor
Three audit follow-ups from issue #327's deep-dive review.

C1 — CompressionCache concurrency lock
======================================

`CompressionCache` instances are shared per `session_id` and accessed from
async-dispatched threadpool workers. Pre-fix, concurrent requests for the
same session raced on `_cache`, `_stable_hashes`, `_first_seen`, and
`_total_tokens_saved` with no synchronization. Observable failures:

* Lost-update on `_total_tokens_saved` (read-modify-write).
* `RuntimeError: OrderedDict mutated during iteration` from `apply_cached`
  when a concurrent `store_compressed` evicts during the walk.
* Lost stable-hash records — next-turn compute_frozen_count reads
  inconsistent state.

May also explain part of SvenMeyer's `_cache: 0 entries / 1003 misses`
observation: the cache was being clobbered concurrently.

Added `threading.RLock` guarding all mutating methods. `RLock` (not `Lock`)
so future code can call locked methods from inside another locked method
without self-deadlock. Also locked `HeadroomProxy._compression_caches`
dict-of-caches access via a separate `_compression_caches_lock` so two
concurrent calls for the same session_id can't each create distinct
CompressionCache objects (which would split the cache state between them).
The `/stats` endpoint snapshots the cache list under the dict lock before
iterating to avoid eviction-during-iteration.

C2 — Multi-worker CCR fragmentation: documented + startup warning
=================================================================

The in-memory `InMemoryCcrStore` (Rust), `_compression_caches` (Python),
`session_tracker_store` (Python), and TOIN learner state are ALL
per-process. Multi-worker uvicorn round-robins requests across workers,
so a session whose turn-1 lands on worker A may have turn-2 land on
worker B. Worker B has zero knowledge of A's CCR markers, replay cache,
or prefix-cache state. Result: `Retrieve original: hash=X` markers stay
in-context as opaque directives, every fresh tool_result is recompressed
from scratch, and `frozen_message_count=0` causes Anthropic prefix-cache
busts on every cross-worker turn.

Added a "Multi-worker deployment — CCR fragmentation" section in
`RUST_DEV.md` documenting the failure modes, the supported configuration
(`--workers 1`), and the sticky-session workaround for horizontal scale.
The proxy emits a `WARNING`-level log line on startup if `workers > 1` is
detected, pointing at the doc section.

C3 — Bounded compression executor with cancel-aware metrics
===========================================================

`asyncio.wait_for(asyncio.to_thread(pipeline.apply), timeout=...)`
cancellation does NOT propagate into the threadpool worker that's running
Rust code. Once the worker has picked up the task,
`concurrent.futures.Future.cancel()` returns False and the thread runs to
completion. Stuck threads accumulated invisibly on asyncio's default
executor, contending with unrelated `to_thread` callers (file IO, etc.).

Replaced all 7 `asyncio.to_thread` call sites for `pipeline.apply()`
across `proxy/handlers/anthropic.py` (3) and `proxy/handlers/openai.py` (4)
with a new `HeadroomProxy._run_compression_in_executor(fn, *, timeout)`
helper that:

  1. Submits to a dedicated bounded `ThreadPoolExecutor` named
     `headroom-compress` (configurable via
     `ProxyConfig.compression_max_workers`; defaults to
     `min(32, (cpu_count or 1) * 4)`).
  2. Increments `_compression_in_flight` (gauge) when work starts and
     decrements when work completes; tracks `_compression_in_flight_max`
     as a high-water mark.
  3. Detects "leaked threads" by comparing wall-clock elapsed against the
     timeout in the worker's `finally` block. Increments
     `_compression_leaked_threads` when a worker finishes after its
     asyncio future was cancelled. Operators can see the leaked-thread
     rate climbing in `/stats runtime.compression_executor` BEFORE the
     pool fills up.

Tests
=====

* `TestCompressionCacheConcurrency` (3 tests) — many threads
  store_compressed / apply_cached / update_from_result on a single
  CompressionCache; assert no exceptions, no lost updates, no partial
  state.
* `test_get_compression_cache_returns_same_instance_under_contention` —
  32 concurrent `_get_compression_cache(same_id)` calls return the
  identical instance (would split pre-lock).
* `test_proxy_compression_executor.py` (8 tests) — pool size respects
  config, in-flight gauge tracks running compressions, high-water mark
  is monotonic, timeout propagates to awaiter, leaked-thread counter
  increments on post-deadline completion, `/stats` surfaces all three
  gauges.

Verification
============

* All 123 targeted regression tests pass.
* `make ci-precheck` clean.
* No `Co-Authored-By` trailer; conventional `fix:` prefix; no
  `--no-verify`.
2026-05-01 15:25:18 -07:00
chopratejas 378d8a0f05 fix(rust): audit cleanup — DiffCompressor CCR leak, CCR TOCTOU race, clippy debt, dep dedup
Closes findings from the post-Phase-3g audit. Five surgical fixes
plus telemetry-discoverability docs. PyO3 0.22 → 0.24 security
upgrade is its own PR (issue #335).

1. DiffCompressor cache_key persistence (production bug)
---------------------------------------------------------
Pre-fix: `RustDiffCompressor.compress()` minted a `cache_key`,
embedded `[... hash=abc123]` in the wire marker, and returned
without storing the original anywhere. Python ContentRouter then
returned the compressed text with a dangling marker — every
retrieval tool call from the LLM 404'd.

Sibling compressors (LogCompressor, SearchCompressor) already had
the right pattern: Rust mints the key, Python's
`_persist_to_python_ccr` writes the original to the production
`CompressionStore`. DiffCompressor was the asymmetric one.

Fix:
- Rust: add `DiffCompressor::compress_with_store(content, context,
  Option<&dyn CcrStore>)` mirroring siblings. Calls `store.put`
  when a key is minted; legacy `compress()` and
  `compress_with_stats()` delegate with `None` for parity.
- Python: add `_persist_to_python_ccr` helper to
  `headroom/transforms/diff_compressor.py.compress()` mirroring
  `log_compressor.py` and `search_compressor.py`.
- Pipeline `DiffOffload`: switch to `compress_with_store(Some(store))`
  and drop the post-hoc double-store hack that papered over this
  bug at the orchestrator boundary.

2. CCR store TOCTOU race in `get()`
-----------------------------------
`InMemoryCcrStore::get()` checked TTL under a read lock, dropped
the lock, then called `remove()`. Between drop and remove a
concurrent `put()` of the same hash with fresh data could land —
and our `remove` would then wipe that fresh entry. Under
multi-worker proxy load this manifested as "I just stored it; why
is it gone?"

Fix: use `DashMap::remove_if`. Predicate runs under the shard
write lock so check-and-remove is atomic. New regression test
exercises a tight contention loop between writer and reader on
the same key.

3. Pre-existing clippy debt in smart_crusher
--------------------------------------------
- 3× `field_reassign_with_default` in `crusher.rs` test setup —
  switch to struct-update syntax `Config { field: x, ..Default }`.
- `hash_array_for_ccr` was `#[cfg(test)]` but unused; deleted with
  a comment so a future test can reintroduce it as a one-liner.

`cargo clippy --workspace --all-targets -- -D warnings` is now
clean across the whole workspace; previous CI patches that allowed
these warnings can be removed in a follow-up.

4. Tokenizers dependency dedup
------------------------------
`tokenizers 0.21` (direct dep) + `tokenizers 0.22` (transitive via
fastembed) compiled twice into the binary. Bumped direct dep to
`0.22` to align; API is compatible (verified by full tokenizer
test suite). Saves compile time + binary bloat.

5. Telemetry-discoverability doc (no new code)
----------------------------------------------
The audit recommended a per-transform invocation counter to
inform the next Python → Rust port. Discovered the infrastructure
already exists at `/stats`:
- `compressions_by_strategy` — invocation count per strategy
- `pipeline_timing` — count + avg/max ms per transform name
- `tokens_saved_by_strategy` — savings attribution

Added a section to `RUST_DEV.md` showing the `curl + jq` recipes
to read this data, with example output highlighting how to spot
zero-invocation deferral candidates (e.g. `code_compressor`).

Verification: workspace tests 734 + 14 + 5 + 4 + 6 + 5 + 2 + 2 +
3 + 4 + 2 + 2 + 1 = all green; cargo fmt clean; cargo clippy
--all-targets clean; Python tests 185 pass; commitlint clean.
2026-04-30 20:54:22 -07:00
chopratejas 12c2665531 feat(rust): signals trait module + KeywordDetector (Phase 3e.1)
Establish `crates/headroom-core/src/signals/` as a top-level module
holding cross-cutting detection traits. Phase 3e.1 ports
`error_detection.py` to a `LineImportanceDetector` trait + a
`Tiered<T>` combinator + a single concrete `KeywordDetector` impl
backed by aho-corasick. Three traits at three granularities are
sketched (line / blob / item); only line-importance is implemented
today.

Two bug fixes from the Python source bake into both the Rust impl
and the Python regex shim:

1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but
   `ERROR_PATTERN` regex omitted them. Lines like `"Connection
   timeout"` were silently neutral despite the keyword being canonical.
   Both surfaces now flag them.
2. `SECURITY_KEYWORDS` carried `token`, which false-positived on
   every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...)
   in our own product. Dropped from the security set.

The Python `error_detection.py` shim now reflects keyword data out of
Rust via `keyword_registry_snapshot()` and recompiles the legacy
`re.Pattern` objects on the fly. Existing callers (text_compressor,
search_compressor, intelligent_context) continue to import the same
names with no source changes; caller migration to the trait API
happens in their own port PRs.

The trait architecture is the seam where a future ML detector slots
in without touching `KeywordDetector` or any caller. The canonical
extension is documented in `signals/README.md` as a classifier head
on the existing `bge-small-en-v1.5` embedder loaded by
`relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax,
~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives
(distilled tinyBERT in ONNX, logistic regression on lexical
features) are kept open in case BGE-head underfits.

Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a
concrete impl. No NoOp, no MockDetector, no stub-ML -- those will
arrive with their real implementations.

Phase 3g (Compression Pipeline Formalization, issue #315) is queued
as the cross-cutting follow-up that will make lossless-then-lossy-
then-CCR ordering an explicit, observable architecture rather than
implicit per-compressor logic. Trait shapes there will reuse the
signals primitive landed in this PR.
2026-04-29 15:55:13 -07:00
chopratejas 3f8de4e117 fix(smart_crusher): re-land orphaned audit close-out — CCR knob + scorer fail-loud
Re-lands two audit fixes that were marked "merged" on GitHub but never
reached main: squash-merging the parent stack changed its commit SHA,
which silently dropped the contents of the stacked PRs (#301, #305).
Single PR this time — no stacking risk.

What lands:

1. **`enable_ccr_marker` Rust gate** — new field on `SmartCrusherConfig`
   (default `true`). `crush_array` checks it before emitting the
   `<<ccr:HASH>>` marker text and the CCR store write. PyO3 surface +
   parity-fixture tolerance updated; recorded fixtures predate the
   field and inherit the `true` default.

2. **Python shim collapses both flags to the gate** — both
   `ccr_config.enabled=False` and `ccr_config.inject_retrieval_marker
   =False` now flip the Rust gate off. Storing a payload nothing in
   the prompt can reference is pointless, and storing under
   `enabled=False` would be a surprise side effect the user
   explicitly opted out of.

3. **Custom `scorer` / `relevance_config` fails loud** — replaces the
   prior WARNING-and-drop. Silently dropping a user-supplied scorer
   is a textbook silent fallback. `NotImplementedError` instead.
   Verified zero production callers pass these args; full plumbing
   arrives with Stage-3c.2's relevance-crate Python bridge.

Tests:
- 2 new Rust unit tests in `crusher.rs::tests`
- 6 new Python tests in `test_smart_crusher_toin_attachment.py`
  (3 CCR marker-knob behaviors + 3 scorer fail-loud)
- Removed `test_ccr_inject_marker_false_logs_warning` (the WARNING is
  gone now that the flag is honored)
- `make ci-precheck` green; eval suite + observability tests run
  twice consecutively to verify no TOIN file pollution leaks into the
  regular+coverage double-run on Python 3.11

RUST_DEV.md audit table reflects both gaps closed.
2026-04-28 18:37:52 -07:00
chopratejas 049ca9cab2 fix(smart_crusher): re-attach TOIN learning loop + audit known regressions
The Stage 3c.1b retirement of the Python SmartCrusher silently disconnected
three subsystems. The audit on 2026-04-28 caught them; this commit fixes
what's fixable today and labels the rest visibly.

## TOIN learning loop — fixed

Before: `ContentRouter._record_to_toin` skipped SmartCrusher on the
assumption SmartCrusher recorded its own TOIN events. The retired Python
class did. The Rust port doesn't know about TOIN. Net result: the
highest-traffic compression strategy stopped fueling the learning loop,
silently.

Fix: shim's `crush()` and `_smart_crush_content()` now call
`toin.record_compression()` after a real compression. Filtered on
`strategy != "passthrough"` because the Rust port flips
`was_modified=True` from JSON whitespace re-canonicalization. Best
effort: TOIN failures are logged at debug level and never break
compression.

Token estimates use `len(json) // 4` (the rule the retired Python used)
because the router doesn't pass a tokenizer down to this layer and
re-tokenizing here would dominate the recording cost.

7 tests in `tests/test_smart_crusher_toin_attachment.py`:
- crush() records on real compression, doesn't on passthrough
- structurally-similar inputs land on the same pattern
- _smart_crush_content() records (legacy apply() path)
- TOIN errors don't break compression
- non-JSON input doesn't record
- inject_retrieval_marker=False emits a WARNING

## CCR marker emission knob — labelled, not yet fixed

`ccr_config.inject_retrieval_marker=False` is not honored — the Rust
port emits `<<ccr:HASH N_rows_offloaded>>` markers in `dropped_summary`
unconditionally. Today the production default has the flag True so no
one is hitting the gap, but the silent-disconnect was a real
regression. Shim now logs a WARNING when callers pass `False` so the
mismatch is visible. Fix needs a Rust-side gate; tracked in
`RUST_DEV.md`.

## Custom relevance scorer — labelled, not yet fixed

`relevance_config` and `scorer` constructor args are accepted for
source compatibility but the Rust default `HybridScorer` always
runs. Shim was logging this at debug; bumped to WARNING. Tracked.

## RUST_DEV.md

New "Known regressions in retired-Python components" section with a
table per retired component. The intent is that this section gets
updated whenever a regression closes (or a new one is found), so the
duplicate-codebase tax stays visible instead of decaying into folklore.
2026-04-28 11:28:25 -07:00
chopratejas c2749c0fb6 docs(rust): lockfile + RUST_DEV.md for proxy CLI
Cargo.lock: pick up tokio-util added in the WS half-close fix.
RUST_DEV.md: document how to run headroom-proxy in passthrough mode
(listen + upstream flags, e2e test gate, env vars).
2026-04-25 12:49:36 -07:00
chopratejasandClaude Opus 4.7 0414cb70e4 feat(rust): scaffold workspace + parity harness (phase-0)
Bootstrap the Rust port of Headroom. Additive only — no existing Python
code modified. Ships the workshop, not the widgets.

Layout
  Cargo.toml (workspace) + rust-toolchain.toml
  crates/headroom-core    — transform library, stub only
  crates/headroom-proxy   — axum binary, /healthz only
  crates/headroom-py      — PyO3 cdylib, exposes headroom._core.hello()
  crates/headroom-parity  — Rust-vs-Python oracle harness + parity-run CLI

Tooling
  Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint
  .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly
  deny.toml for cargo-deny

Parity corpus
  tests/parity/recorder.py + scripts/record_fixtures.py
  125 recorded fixtures across 5 leaf transforms (ccr, tokenizer,
  log_compressor, diff_compressor, cache_aligner)

Docs
  RUST_DEV.md — developer setup and workspace reference
  docs/spec/022-rust-migration.md — migration plan and stage breakdown

.gitignore: whitelist scripts/record_fixtures.py; ignore target/

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00