Files
headroom/RUST_DEV.md
99021af20a fix(proxy): make the Rust front's native Bedrock and Vertex routes opt-in (#4851)
## Description

Two routes in the Rust front (`crates/headroom-proxy`) attach the host's
own cloud credentials to whatever request reaches them, without checking
the caller:

- **Bedrock native** (`/model/{id}/invoke`, `/converse`,
`/invoke-with-response-stream`, `/converse-stream`) is on by default and
signs every request with SigV4 using the AWS credentials found on the
host.
- **Vertex** (`…/publishers/anthropic/models/{m}:rawPredict` /
`:streamRawPredict`) is mounted unconditionally, replaces the caller's
`Authorization` with a bearer from the host's Google ADC, and forwards
to the generic `--upstream`, whatever host that is. So with a non-Google
upstream (the Python proxy, another provider) the host's Google token is
sent there. `--vertex-adc-scope` was parsed but never used.

This PR makes both routes explicit opt-ins, only sends the ADC bearer to
a Google API host, and uses the configured scope.

**Dependency:** these routes still do not authenticate callers
themselves. Callers are expected to be authenticated by the inbound
`HEADROOM_PROXY_TOKEN` gate added in #4840 (which also moves the default
listen address to loopback). This PR is based on `main` and does not
require #4840 to merge first, but operators who enable either route on a
non-loopback address need #4840 to restrict who can use them.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `config.rs`: `--enable-bedrock-native` /
`HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` now defaults to `false`. New
`--enable-vertex-native` / `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE`
(default `false`) and `--vertex-allow-non-google-upstream` /
`HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` (default `false`).
`Config::for_test` keeps both routes on so the existing route tests run
unchanged.
- `proxy.rs`: the Vertex route is mounted only when enabled; with it
off, Vertex paths go through the catch-all with the caller's own headers
(as the Python proxy's Vertex passthrough expects). The ADC token source
is built with `--vertex-adc-scope`. The "Bedrock route not mounted"
startup line is now `info`, since off is the default.
- `vertex/raw_predict.rs`: before fetching the ADC bearer, the handler
refuses (502 `vertex upstream is not a Google API host`,
`vertex_upstream_not_google` error log) unless `--upstream` is
`https://` on a `*.googleapis.com` host (regional, global and
`*.p.googleapis.com` endpoints), or the operator set
`--vertex-allow-non-google-upstream=true` (local emulators, gateways).
No token is fetched on that path.
- `RUST_DEV.md`: the three flags in the configuration table, with what
enabling them means.

**Behaviour change for operators:** deployments that used the Rust
front's native Bedrock or Vertex signing must now pass
`--enable-bedrock-native true` / `--enable-vertex-native true` (or the
env vars). Vertex with a non-Google `--upstream` additionally needs
`--vertex-allow-non-google-upstream true`. Without the flags, those
paths are forwarded to `--upstream` unsigned with the client's own
credentials (Claude Code in Vertex mode keeps sending its own ADC token,
which is what `docs/content/docs/claude-code-vertex.mdx` describes).
`scripts/qualify_cargo_dependencies.py` already passes
`--enable-bedrock-native true`.

## Testing

- [x] `cargo test -p headroom-proxy` (45 test binaries, 596 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes

New tests in `tests/integration_cloud_routes_opt_in.rs` build the config
from the real CLI defaults (`CliArgs` → `Config::from_cli`), give the
app host AWS credentials and a host ADC bearer, and record what a
`wiremock` upstream receives:
- `default_config_does_not_sign_bedrock_requests`: no `AWS4-HMAC-SHA256`
Authorization on `invoke`, `invoke-with-response-stream`, `converse`.
- `default_config_never_attaches_host_adc_bearer`: `rawPredict` and
`streamRawPredict` reach the upstream without the host bearer.
- `vertex_bearer_is_not_sent_to_a_non_google_upstream`: with the route
enabled and an `http://127.0.0.1` upstream, 502 and the upstream sees
nothing.
- `vertex_opt_in_still_signs_when_non_google_upstream_is_allowed`: the
explicit escape hatch still works.
- `configured_adc_scope_reaches_token_source`.
- Unit test `vertex::raw_predict::tests::google_api_host_check`
(`http://`, `googleapis.com.example.net`, `example-googleapis.com`,
loopback are rejected).

### Test Output

Before (the three tests that compile against `main`, run with this PR's
`src/` changes stashed):

```text
test configured_adc_scope_reaches_token_source ... FAILED
test default_config_never_attaches_host_adc_bearer ... FAILED
test default_config_does_not_sign_bedrock_requests ... FAILED
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s
```

After:

```text
$ cargo test -p headroom-proxy --test integration_cloud_routes_opt_in
test configured_adc_scope_reaches_token_source ... ok
test vertex_opt_in_still_signs_when_non_google_upstream_is_allowed ... ok
test vertex_bearer_is_not_sent_to_a_non_google_upstream ... ok
test default_config_never_attaches_host_adc_bearer ... ok
test default_config_does_not_sign_bedrock_requests ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s
```

## Real Behavior Proof

- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (c21b0813) and this branch. AWS credentials supplied
to the proxy through `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (fake
values), IMDS disabled. Upstream and Bedrock endpoint are a small Python
recorder on `127.0.0.1:18788` that prints the `Authorization` it
receives.
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:18787
--upstream http://127.0.0.1:18788 --bedrock-endpoint
http://127.0.0.1:18788` (no Bedrock/Vertex flags), then `curl -d
'<anthropic body>'
http://127.0.0.1:18787/model/anthropic.claude-3-haiku-20240307-v1:0/invoke`
with no client credentials. Repeated on this branch with
`--enable-bedrock-native true`.
- Observed result (key id and signature redacted):

```text
== origin/main (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}

== this branch (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": null}

== this branch, --enable-bedrock-native true
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}
```

- Not tested: Vertex against real Google ADC and a real
`*.googleapis.com` upstream (covered by the integration tests with a
static token source; no GCP credentials on the test machine).

## Runtime Rollout Safety

- Rollout-managed feature(s): `native_bedrock` (existing). The CLI flag
already maps `false` to a rollout disable, so with the new default the
feature reports disabled unless `--enable-bedrock-native true` is
passed. Vertex is not rollout-managed.
- Minimum rollout channel: unchanged
- Stable/default behavior changed: yes, both native routes are off by
default
- Kill switch / disable path: the routes are now off unless enabled
- Unsafe override required: no
- Qualification impact: none, the qualification script already passes
`--enable-bedrock-native true`
- Rollback path: pass `--enable-bedrock-native true` /
`--enable-vertex-native true` (plus `--vertex-allow-non-google-upstream
true` if the upstream is not a Google host), or revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4851"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:07:49 -06:00

477 lines
26 KiB
Markdown

# Headroom Rust Rewrite — Developer Guide
This document covers the Rust port of Headroom. It is the only new top-level
doc created in Phase 0; longer-form design/plan writeups live elsewhere and
are not versioned in this repo.
## Workspace layout
```
Cargo.toml # workspace root
rust-toolchain.toml # pins stable rustc with rustfmt+clippy
crates/
headroom-core/ # library: shared types + transform trait surface
headroom-proxy/ # binary: axum /healthz (Phase 2 grows this)
headroom-py/ # PyO3 cdylib exposing `headroom._core`
headroom-parity/ # lib + `parity-run` CLI for Python parity tests
headroom-simulators/ # binary: deterministic local upstream stub for proxy tests
tests/parity/
fixtures/<transform>/*.json # recorded Python outputs (Phase 1 ports match)
recorder.py # Python-side fixture recorder
scripts/record_fixtures.py # entry point for running the recorder
```
`cargo build --workspace` builds every crate. `default-members` drops
`headroom-py` from `cargo run`/bare-`cargo test` flows so that `cargo test
--workspace` does not try to execute the PyO3 cdylib standalone (it can't
find `libpython` without a Python interpreter hosting it).
## Common commands
`just` is not installed on dev boxes here; a `Makefile` at the repo root
exposes the same targets:
| Target | What it does |
| --- | --- |
| `make test` | `cargo test --workspace` |
| `make test-parity` | Builds `headroom-py` via maturin, runs `parity-run run` |
| `make bench` | `cargo bench --workspace` |
| `make build-proxy` | Release-builds `headroom-proxy`, strips, prints size |
| `make build-wheel` | `maturin build --release -m crates/headroom-py/Cargo.toml` |
| `make fmt` | `cargo fmt --all` |
| `make lint` | `cargo fmt --check` + `cargo clippy --workspace -- -D warnings` |
## Running the proxy
`headroom-proxy` is a transparent reverse proxy. Phase 1 forwards HTTP/1.1,
HTTP/2, SSE, and WebSocket traffic verbatim to a configured upstream — no
provider logic yet. The intent is that operators run the existing Python
proxy on a private port and put `headroom-proxy` on the public port pointed
at it; end users notice nothing.
```bash
# Build
make build-proxy
./target/release/headroom-proxy --help
# Run against a local upstream (listens on 127.0.0.1:8787 by default)
./target/release/headroom-proxy \
--upstream http://127.0.0.1:8788
# Health checks
curl -s http://127.0.0.1:8787/healthz # => {"ok":true,...}
curl -s http://127.0.0.1:8787/healthz/upstream # => 200 if upstream reachable
```
### Operator runbook (Phase 1 cutover)
```bash
# 1. Move the Python proxy to a private port (e.g. 8788)
HEADROOM_HOST=127.0.0.1 HEADROOM_PORT=8788 python -m headroom.proxy & # or your existing launcher
# 2. Run the Rust proxy on the previously-public port (8787) pointing at it.
# The Python proxy sees every relayed request as coming from 127.0.0.1,
# so the Rust proxy enforces HEADROOM_PROXY_TOKEN itself (see below).
export HEADROOM_PROXY_TOKEN="$(openssl rand -hex 32)"
./target/release/headroom-proxy --listen 0.0.0.0:8787 --upstream http://127.0.0.1:8788 &
# 3. End users keep hitting :8787 unchanged.
# 4. Confirm passthrough:
curl -si http://127.0.0.1:8787/v1/models
# 5. Rollback = stop the Rust proxy and rebind Python back to 8787.
```
### Configuration flags
| Flag | Env var | Default | Notes |
| --- | --- | --- | --- |
| `--listen` | `HEADROOM_PROXY_LISTEN` | `127.0.0.1:8787` | bind address; a non-loopback address refuses to start without `HEADROOM_PROXY_TOKEN` unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` |
| `--proxy-token` | `HEADROOM_PROXY_TOKEN` | (unset) | inbound token required from non-loopback peers (below); prefer the env var |
| `--metrics-require-loopback` | `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK` | `false` | when set, `/metrics` is served only to loopback peers (403 otherwise); recommended on non-loopback binds |
| `--upstream` | `HEADROOM_PROXY_UPSTREAM` | (required) | base URL the proxy forwards to |
| `--upstream-timeout` | | `600s` | end-to-end request timeout (long for streams) |
| `--upstream-connect-timeout` | | `10s` | TCP/TLS connect timeout |
| `--max-body-bytes` | | `100MB` | for buffered cases; streams bypass |
| `--log-level` | | `info` | `RUST_LOG`-style filter |
| `--rewrite-host` / `--no-rewrite-host` | | rewrite | rewrite Host to upstream (default) |
| `--graceful-shutdown-timeout` | | `30s` | wait for in-flight on SIGTERM/SIGINT |
| `--stats` | `HEADROOM_PROXY_STATS` | `true` | native savings stats + `/dashboard` (below) |
| `--stats-path` | `HEADROOM_PROXY_STATS_PATH` | `~/.headroom/native_stats.json` | ledger persistence (honours `HEADROOM_WORKSPACE_DIR`) |
| `--enable-bedrock-native` | `HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` | (unset: off) | `true` or `--features native_bedrock` mounts `/model/{id}/invoke`, `/converse` and the streaming variants; they sign every request with this host's AWS credentials and do not authenticate the caller |
| `--enable-vertex-native` | `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE` | `false` | mount Vertex `:rawPredict` / `:streamRawPredict`; replaces the caller's `Authorization` with this host's Google ADC bearer (scope `--vertex-adc-scope`) and does not authenticate the caller |
| `--vertex-allow-non-google-upstream` | `HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` | `false` | by default the Vertex route only sends the ADC bearer when `--upstream` is `https://*.googleapis.com`, and answers 502 otherwise |
### Inbound authentication
The Rust proxy connects to the Python proxy from `127.0.0.1`, and the
Python proxy trusts loopback callers (no token needed, and access to
`/admin`, `/debug` and `/settings`). So the Rust proxy applies the same
`HEADROOM_PROXY_TOKEN` rules itself, before anything is forwarded:
- With a token set, a non-loopback peer must send it as
`X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`,
or gets `401 {"error":"unauthorized"}`. Loopback peers are exempt, as
in the Python proxy. The token is removed from the request before it
is forwarded (`Authorization` only when it carries the token).
- A caller that is not local here (a non-loopback peer, or a loopback
peer whose `Host` is not a loopback name) is never forwarded with a
`Host` that may name loopback (including scoped IPv6 such as
`[::1%lo]`), whatever `--rewrite-host` is and whatever the upstream
name resolves to: that `Host` is replaced with
`headroom-remote.invalid`, so the Python proxy's local-only routes
(`/admin`, `/debug`, `/settings`, ...) stay closed to it, with or
without a token.
- `GET /healthz`, `/healthz/upstream`, `/health`, `/livez` and `/readyz`
stay reachable without the token, and `GET`/`HEAD /api/hello` is
answered locally with an empty `200`.
- A non-loopback `--listen` with no token refuses to start. Set
`HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` only when something else
already restricts who can reach the port.
### Savings stats & dashboard (native)
The Rust proxy records per-request savings/cost telemetry on every
lane it forwards — Anthropic `/v1/messages`, OpenAI Chat/Responses,
all four native Bedrock routes, and Vertex `rawPredict` /
`streamRawPredict` when those are enabled — and serves it locally (never tunnelled
upstream):
```bash
curl -s http://127.0.0.1:8787/stats | jq '.session' # since-boot totals
curl -s http://127.0.0.1:8787/stats | jq '.lifetime_by_model'
curl -s "http://127.0.0.1:8787/stats/timeseries?bucket=day" | jq '.points[-7:]'
curl -s "http://127.0.0.1:8787/stats/events?limit=5" # recent-request feed
open http://127.0.0.1:8787/dashboard # embedded UI, zero deps
```
Recording is deferred until the response is fully observed, so
streaming output/cache token counts come from the real SSE /
EventStream usage frames. USD figures come from the vendored LiteLLM
price table (`data/model_prices_and_context_window.json`); a model
missing from it records $0 and logs one WARN — refresh via
`scripts/refresh_model_limits.sh`. Input and output are priced
separately (`input_cost_usd` / `output_cost_usd`) because they bill
at different rates — output is typically 4-5x input, so a spend
figure that counted input alone would understate a short-prompt,
long-answer turn by most of its cost. `total_cost_usd` is the sum
and is what `/dashboard` shows as "Spend". Cache savings are NET: the
cache-read discount minus the cache-write premium (writes bill
above list price), floored at $0 per request — a warm-up turn that
only writes is not counted as negative savings, and gross-read
figures that overstate the benefit are never shown. Failed upstreams count as
failures and accrue no savings — as do proxy-side rejections
(missing AWS credentials, SigV4/ADC failures, oversized bodies), so
an operator-side outage reads as a failure spike rather than as
zero traffic. Aggregates (lifetime, per-model, 48 h hourly +
~13 mo daily buckets) persist as one atomic snapshot written off
the hot path every 10 s and on shutdown.
> **What reads 0 today.** Spend, token counts, and prompt-cache
> savings are live. `tokens_saved` / `compression_savings_usd` are
> wired end-to-end but will report 0 on real traffic until the
> live-zone per-type compressors are implemented — the dispatcher
> currently returns `NoCompression` for every well-formed body
> (`live_zone_mode_with_valid_body_returns_no_compression_pr_b2`
> pins that invariant). Nothing needs changing here when they land;
> the numbers start flowing on their own.
> **No auth; exposure is split by tier.** These endpoints bind
> wherever `--listen` points (default `127.0.0.1:8787`) and, for
> loopback peers or with no `HEADROOM_PROXY_TOKEN`, carry no auth, so the payload is split the same way the Python proxy splits
> its own `/stats`:
>
> - **Aggregates** — spend, tokens, per-model/provider rollups,
> history — are served to any caller that gets past the inbound gate
> (above), like the Prometheus `/metrics` endpoint next door: with
> `HEADROOM_PROXY_TOKEN` set, remote callers need the token for these
> too; without it, they are open to anyone who can reach the port.
> - **Per-request rows** — `recent_requests`, all of `/stats/events`
> — plus the ledger's filesystem path are served **only to local
> callers**: loopback peer *and* a loopback `Host` header (the
> second is the DNS-rebinding defence — a rebound request reaches
> the proxy *from* loopback, so the peer check alone doesn't hold).
> Remote callers get `recent_requests: null` and a 404 from
> `/stats/events`; the dashboard says so rather than erroring.
>
> A request id + model + timestamp is a different sensitivity tier
> from a counter, and `/metrics` never exposes it — so "same as
> /metrics" isn't a licence to hand it to the open internet. Opening
> the local-only tier to a trusted reverse proxy is tracked in
> [#1959](https://github.com/headroomlabs-ai/headroom/issues/1959),
> which is settling that policy for the Python dashboard first; this
> surface should adopt it rather than grow a second scheme.
> `--stats=false` removes the routes entirely.
### Picking the next port: invocation telemetry
Before porting another Python compressor to Rust, check what's actually
running. The Python proxy already exposes per-transform telemetry on
`/stats` (`headroom.proxy.prometheus_metrics`):
```bash
# Top compressors by invocation count (last process lifetime)
curl -s http://127.0.0.1:8788/stats | jq '.compressions_by_strategy'
# {
# "intelligent_context": 12453,
# "smart_crusher": 487,
# "search": 312,
# "diff": 28,
# "code": 0, # ← never fires; safe to defer porting
# ...
# }
# Per-transform timing (avg/max/count by transform name)
curl -s http://127.0.0.1:8788/stats | jq '.pipeline_timing'
# Token savings attributable to each strategy
curl -s http://127.0.0.1:8788/stats | jq '.tokens_saved_by_strategy'
```
This is the data the audit-cleanup PR (2026-04-30) recommended for
prioritizing the next Python → Rust port. Strategies with zero or
near-zero invocations are deferral candidates; strategies on the hot
path are porting candidates regardless of LOC count.
### Reserved paths
`/healthz` and `/healthz/upstream` are intercepted by the Rust proxy and
**not** forwarded. Operators must not name a real upstream route either of
these. Everything else is a catch-all forward.
## Maturin + Python wiring
`headroom-py` is a PyO3 cdylib that exposes `headroom._core` in Python. The
`extension-module` feature is opt-in so plain `cargo build --workspace` does
not try to link against `libpython` on systems that don't have it.
### First-time setup (clean venv recommended)
```bash
python3.11 -m venv /tmp/hr-rust-venv
source /tmp/hr-rust-venv/bin/activate
pip install maturin
cd crates/headroom-py
maturin develop # editable dev build, installs headroom._core
cd /tmp # IMPORTANT: step out of the repo root first
python -c "from headroom._core import hello; print(hello())"
# => headroom-core
```
> Why `cd /tmp`? The repo root also contains the Python `headroom/` package.
> Running the smoke import from the repo root makes Python resolve `headroom`
> to `./headroom/__init__.py` (the full SDK, which pulls in heavy deps) instead
> of the lightweight namespace package installed by maturin. Tests should
> either run outside the repo root, or ensure `headroom` is installed into
> the same venv (then the maturin-installed `_core.so` lands alongside it and
> both imports resolve).
### Release wheels
```bash
make build-wheel
# wheels land under target/wheels/
```
CI (`.github/workflows/rust.yml`) builds linux-x86_64, macos-arm64, and
macos-x86_64 wheels via `PyO3/maturin-action` and uploads them as artifacts.
## Parity harness
`crates/headroom-parity` owns the Rust-vs-Python oracle:
- JSON fixtures under `tests/parity/fixtures/<transform>/` (schema:
`{ transform, input, config, output, recorded_at, input_sha256 }`).
- `TransformComparator` trait — one impl per transform. Phase 0 stubs return
`Err(...)`; the harness flags those as `Skipped`, not panics.
- `parity-run` CLI: `cargo run -p headroom-parity -- run [--only TRANSFORM]`.
- Unit tests in `crates/headroom-parity/src/lib.rs` include a **negative
test** (`harness_reports_diff_for_divergent_comparator`) proving the
harness detects mismatched output before any real port lands.
### Recording fresh fixtures
```bash
source .venv/bin/activate # the main Python SDK venv
python scripts/record_fixtures.py # uses tests/parity/recorder.py
ls tests/parity/fixtures/*/ | sort | uniq -c
```
The recorder monkey-patches the in-process transform classes (see
`record_all()` in `tests/parity/recorder.py`). It does **not** modify any
file under `headroom/`.
## Known regressions in retired-Python components
The Stage 3b/3c.1b retirements deleted Python source for `DiffCompressor`
and `SmartCrusher` and replaced them with PyO3-delegating shims. The
2026-04-28 audit found that the retirements shipped with subsystems
silently disconnected. This section tracks each gap and its disposition
so they don't regress further or get forgotten.
### SmartCrusher
| Subsystem | State | Tracked by |
|---|---|---|
| TOIN learning loop | **Re-attached 2026-04-28.** Shim's `crush()` and `_smart_crush_content()` now call `toin.record_compression()` after a real compression. Filtered on `strategy != "passthrough"` to ignore JSON re-canonicalization. Best-effort: TOIN failures are logged at debug level and don't break compression. | `tests/test_smart_crusher_toin_attachment.py` |
| CCR marker emission knob | **Honored end-to-end 2026-04-29.** New `enable_ccr_marker: bool` field on Rust `SmartCrusherConfig`; `crush_array` checks it before emitting the `<<ccr:HASH>>` marker text and the CCR store write. Python shim flips it from `ccr_config.enabled and ccr_config.inject_retrieval_marker` — both flags collapse to the same Rust gate, since storing payloads under either off-switch makes no sense. Scope: gates only the row-drop sentinel path; Stage-3c.2 opaque-string CCR substitutions still emit always (no Python equivalent, no production caller asks for suppression). | `tests/test_smart_crusher_toin_attachment.py` + `crates/headroom-core/.../crusher.rs::tests::enable_ccr_marker_*` |
| Custom relevance scorer | **Closed (fail-loud) 2026-04-29.** `relevance_config` and `scorer` constructor args remain in the signature for source compat, but the shim raises `NotImplementedError` when either is non-None — silently dropping a user-supplied scorer is a textbook silent-fallback bug. Full plumbing waits on Stage-3c.2's relevance-crate Python bridge. | `tests/test_smart_crusher_toin_attachment.py::test_custom_*_arg_raises_not_implemented` |
| Per-tool TOIN learning hook | **Re-attached partially.** `_smart_crush_content` accepts `tool_name` and now threads it into the TOIN record. The hook is best-effort — it improves `query_context` aggregation but doesn't drive per-tool overrides yet. | `tests/test_smart_crusher_toin_attachment.py::test_smart_crush_content_records_to_toin` |
### DiffCompressor
| Subsystem | State |
|---|---|
| Adaptive context windows | Honored byte-for-byte (parity fixture-locked). |
| TOIN integration | Never had one — DiffCompressor records via `_record_to_toin` in ContentRouter, which already runs for non-SmartCrusher strategies. No regression. |
### Phase 3e.1 — `signals/` trait module + KeywordDetector (2026-04-29)
The Python `error_detection.py` regex registry was retired and reborn as a
trait + tier system in `crates/headroom-core/src/signals/`. See
`signals/README.md` for the full architecture; the highlights:
- **Per-granularity traits.** `LineImportanceDetector` ships today; future
`ContentTypeDetector` and `ItemImportanceDetector<I>` will follow as their
consumers get touched.
- **`Tiered<T>` combinator.** Composition, not inheritance. Future ML
detectors slot in as new tiers without changes to `KeywordDetector` or
any caller.
- **One concrete impl.** `KeywordDetector` (aho-corasick) is the only tier
registered today. **No NoOp/stub impls** — per project no-silent-fallbacks
rule, future tiers land with their real implementations.
- **Bug fixes baked in.** `ERROR_KEYWORDS` regex now includes
`timeout|abort|denied|rejected` (previously drifted from the keyword set);
`token` dropped from `SECURITY_KEYWORDS` (false-positived on every LLM
metric reference). Both fixed in the Python regex too via the shim that
recompiles patterns from the Rust-exposed keyword tables.
- **Companion canonical extension path.** `signals/README.md` documents
the BGE classifier head — a 384-dim → 4-class softmax on top of the
already-loaded `bge-small-en-v1.5` embedder — as the natural ML tier.
Two alternatives kept open: distilled tinyBERT in ONNX, logistic
regression on lexical features.
### Phase 3g (queued) — Compression Pipeline Formalization (issue #315)
Strategic decision 2026-04-29: after Phase 3e (compressor ports) and
Phase 3f (Rust MCP scaffold) wrap, formalize the lossless-then-lossy-
then-CCR ordering as a cross-cutting `CompressionPipeline` orchestrator
+ `LosslessTransform` / `LossyTransform` traits in
`crates/headroom-core/src/pipeline/`. Existing compressors get
refactored as compositions of pluggable transforms. The crucial design
choice — **parsers for structure, models at the prose/structure
boundary** — is captured in issue #315 and
`memory/project_lossless_first_pipeline.md`. Do NOT start coding before
3e/3f finish.
### Watch list (potential regressions, not yet audited)
- `CCRConfig.enabled=False` end-to-end — **closed 2026-04-29**. Both `enabled=False` and `inject_retrieval_marker=False` collapse to the same Rust `enable_ccr_marker=False` gate (no marker, no store write). See the SmartCrusher table above.
- `SmartCrusherConfig.use_feedback_hints=False` — config field is forwarded to Rust but its honoring inside the Rust crusher hasn't been verified against a parity fixture for the disabled path.
When any item above changes, update both this section and the test file. The shim's docstring also references this section — keep them aligned.
## Phase 0 Blockers
These are known limitations for Phase 0. They are tracked here so Phase 1
doesn't rediscover them.
- **`cache_aligner` fixtures**: `CacheAligner.apply()` takes
`(messages, tokenizer, **kwargs)` — a `Tokenizer` is provider-specific and
its cheapest `NoopTokenCounter` / `TiktokenTokenCounter` construction still
requires pulling `headroom.providers.*` which imports the full observability
stack (opentelemetry, etc). The recorder records `cache_aligner` only if a
usable tokenizer is cheaply available; otherwise it logs a blocker and
skips. See `recorder.py::_build_cache_aligner_tokenizer`.
- **`ccr` is not a single class**: The repo has `CCRToolInjector`,
`CCRResponseHandler`, `CCRToolCall`, `CCRToolResult` etc. rather than a
single `CCR` class. The recorder targets the encoder-style entry point
most analogous to the Rust port (`CCRToolInjector.inject_tool` and
`CCRResponseHandler.parse_response`). If Phase 1 wants a different split
it should update `recorder.py::record_all` accordingly.
- **Pre-commit hook noise**: `scripts/sync-plugin-versions.py` mutates
`.claude-plugin/marketplace.json`, `.github/plugin/marketplace.json`, and
`plugins/headroom-agent-hooks/**/plugin.json` on every commit. Those
changes are harmless but each commit in Phase 0 picks them up. Phase 1
does not need to do anything special — just let the hook run.
- **`rust-toolchain.toml`** now pins `channel = "1.95.0"` (previously tracked
`"stable"`, which let CI drift ahead of local dev boxes — see the file's own
comment for the 2026-04-27 incident this fixed). Resolved; kept here for
history.
## Multi-worker deployment — CCR fragmentation
**Status:** two persistent CCR backends are available. The single-`--workers`
recommendation no longer applies once you select a persistent backend.
### Backend selection
`crates/headroom-core/src/ccr/backends/` ships three implementations of
the `CcrStore` trait:
| Backend | When to use | Persistence | Multi-worker safe |
| ---------------------- | ------------------------------------------- | ----------- | -------------------------- |
| `InMemoryCcrStore` | Tests, single-worker prototyping | No | No |
| `SqliteCcrStore` (default) | Single-instance prod / single-host fleet | Yes (file) | Yes (sticky session) |
| `RedisCcrStore` (opt-in) | Multi-host / horizontally-scaled prod | Yes (Redis) | Yes (no stickiness needed) |
`backends::from_config` picks one at startup from the operator's
`CcrBackendConfig`. **Init failures surface to the caller**
(`feedback_no_silent_fallbacks.md`) — a misconfigured DB path or
unreachable Redis URL aborts startup rather than silently degrading to
in-memory.
### When does what work?
- **`SqliteCcrStore`** is the default for new deploys. The DB file lives
on the local disk; multiple workers on the **same host** share it via
SQLite's WAL-mode locking, so `--workers N` works as long as a sticky
load balancer routes each session to the same host. Survives proxy
restarts: a new worker that opens the same DB file recovers every
in-flight `<<ccr:HASH>>` marker.
- **`RedisCcrStore`** (cfg-gated behind the `redis` feature) is the
drop-in for **horizontally-scaled** deployments. Every worker on
every host hits the same Redis instance; no sticky session is
required at any layer of the LB. Enable with `--features redis` in
the proxy crate's Cargo build.
- **`InMemoryCcrStore`** is fine for tests and single-worker
development. Production deployments using it lose every
`<<ccr:HASH>>` marker on restart and fragment across workers — keep
it confined to local boxes.
### What goes wrong with the in-memory backend on `--workers N > 1`
Each uvicorn worker is a separate Python process. The following state is
fragmented across workers:
1. **Python `CompressionStore`** — defaults to `SQLiteBackend` at
`workspace_dir()/ccr_store.db` (restart-safe, shared across workers) when
`HEADROOM_CCR_BACKEND` is unset or `"sqlite"` (`_create_default_ccr_backend`
in `headroom/cache/compression_store.py`). Set `HEADROOM_CCR_BACKEND=memory`
to opt into the per-process `InMemoryBackend` instead — that is the setting
this section's fragmentation risk actually applies to; the log messages in
`headroom/proxy/server.py` (see below) still assume in-memory-by-default and
are stale on this point.
2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py`)
— per-session `CompressionCache` dict (instance var, always per-worker).
3. **`HeadroomProxy.session_tracker_store`** — per-session prefix-tracker
state derived from Anthropic's `cache_read_input_tokens` responses
(instance var, always per-worker).
4. **TOIN learner state** — writes snapshots to `~/.headroom/toin.json` but
keeps per-process in-memory state; pattern statistics on one worker are not
visible to others until the next disk flush.
When uvicorn round-robins requests across workers, a session whose
turn-1 landed on worker A may have turn-2 land on worker B. Worker B has
zero knowledge of what worker A did, the `<<ccr:HASH>>` marker resolves
to `None`, and the model sees an opaque directive it can't act on.
Switching to `SqliteCcrStore` (default) or `RedisCcrStore` resolves the
CCR fragmentation; a sticky-session load balancer resolves all of them.
### Detecting it in the wild
The proxy emits a `WARNING`-level log line on startup when `--workers N > 1`.
When `HEADROOM_CCR_BACKEND` is unset (default InMemoryBackend), the warning
includes CCR retrieval failures and suggests setting `HEADROOM_CCR_BACKEND=sqlite`.
When a cross-worker backend is already configured, the warning covers only the
remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker).