mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
Two routes in the Rust front (`crates/headroom-proxy`) attach the host's
own cloud credentials to whatever request reaches them, without checking
the caller:
- **Bedrock native** (`/model/{id}/invoke`, `/converse`,
`/invoke-with-response-stream`, `/converse-stream`) is on by default and
signs every request with SigV4 using the AWS credentials found on the
host.
- **Vertex** (`…/publishers/anthropic/models/{m}:rawPredict` /
`:streamRawPredict`) is mounted unconditionally, replaces the caller's
`Authorization` with a bearer from the host's Google ADC, and forwards
to the generic `--upstream`, whatever host that is. So with a non-Google
upstream (the Python proxy, another provider) the host's Google token is
sent there. `--vertex-adc-scope` was parsed but never used.
This PR makes both routes explicit opt-ins, only sends the ADC bearer to
a Google API host, and uses the configured scope.
**Dependency:** these routes still do not authenticate callers
themselves. Callers are expected to be authenticated by the inbound
`HEADROOM_PROXY_TOKEN` gate added in #4840 (which also moves the default
listen address to loopback). This PR is based on `main` and does not
require #4840 to merge first, but operators who enable either route on a
non-loopback address need #4840 to restrict who can use them.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `config.rs`: `--enable-bedrock-native` /
`HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` now defaults to `false`. New
`--enable-vertex-native` / `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE`
(default `false`) and `--vertex-allow-non-google-upstream` /
`HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` (default `false`).
`Config::for_test` keeps both routes on so the existing route tests run
unchanged.
- `proxy.rs`: the Vertex route is mounted only when enabled; with it
off, Vertex paths go through the catch-all with the caller's own headers
(as the Python proxy's Vertex passthrough expects). The ADC token source
is built with `--vertex-adc-scope`. The "Bedrock route not mounted"
startup line is now `info`, since off is the default.
- `vertex/raw_predict.rs`: before fetching the ADC bearer, the handler
refuses (502 `vertex upstream is not a Google API host`,
`vertex_upstream_not_google` error log) unless `--upstream` is
`https://` on a `*.googleapis.com` host (regional, global and
`*.p.googleapis.com` endpoints), or the operator set
`--vertex-allow-non-google-upstream=true` (local emulators, gateways).
No token is fetched on that path.
- `RUST_DEV.md`: the three flags in the configuration table, with what
enabling them means.
**Behaviour change for operators:** deployments that used the Rust
front's native Bedrock or Vertex signing must now pass
`--enable-bedrock-native true` / `--enable-vertex-native true` (or the
env vars). Vertex with a non-Google `--upstream` additionally needs
`--vertex-allow-non-google-upstream true`. Without the flags, those
paths are forwarded to `--upstream` unsigned with the client's own
credentials (Claude Code in Vertex mode keeps sending its own ADC token,
which is what `docs/content/docs/claude-code-vertex.mdx` describes).
`scripts/qualify_cargo_dependencies.py` already passes
`--enable-bedrock-native true`.
## Testing
- [x] `cargo test -p headroom-proxy` (45 test binaries, 596 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes
New tests in `tests/integration_cloud_routes_opt_in.rs` build the config
from the real CLI defaults (`CliArgs` → `Config::from_cli`), give the
app host AWS credentials and a host ADC bearer, and record what a
`wiremock` upstream receives:
- `default_config_does_not_sign_bedrock_requests`: no `AWS4-HMAC-SHA256`
Authorization on `invoke`, `invoke-with-response-stream`, `converse`.
- `default_config_never_attaches_host_adc_bearer`: `rawPredict` and
`streamRawPredict` reach the upstream without the host bearer.
- `vertex_bearer_is_not_sent_to_a_non_google_upstream`: with the route
enabled and an `http://127.0.0.1` upstream, 502 and the upstream sees
nothing.
- `vertex_opt_in_still_signs_when_non_google_upstream_is_allowed`: the
explicit escape hatch still works.
- `configured_adc_scope_reaches_token_source`.
- Unit test `vertex::raw_predict::tests::google_api_host_check`
(`http://`, `googleapis.com.example.net`, `example-googleapis.com`,
loopback are rejected).
### Test Output
Before (the three tests that compile against `main`, run with this PR's
`src/` changes stashed):
```text
test configured_adc_scope_reaches_token_source ... FAILED
test default_config_never_attaches_host_adc_bearer ... FAILED
test default_config_does_not_sign_bedrock_requests ... FAILED
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s
```
After:
```text
$ cargo test -p headroom-proxy --test integration_cloud_routes_opt_in
test configured_adc_scope_reaches_token_source ... ok
test vertex_opt_in_still_signs_when_non_google_upstream_is_allowed ... ok
test vertex_bearer_is_not_sent_to_a_non_google_upstream ... ok
test default_config_never_attaches_host_adc_bearer ... ok
test default_config_does_not_sign_bedrock_requests ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s
```
## Real Behavior Proof
- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (c21b0813) and this branch. AWS credentials supplied
to the proxy through `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (fake
values), IMDS disabled. Upstream and Bedrock endpoint are a small Python
recorder on `127.0.0.1:18788` that prints the `Authorization` it
receives.
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:18787
--upstream http://127.0.0.1:18788 --bedrock-endpoint
http://127.0.0.1:18788` (no Bedrock/Vertex flags), then `curl -d
'<anthropic body>'
http://127.0.0.1:18787/model/anthropic.claude-3-haiku-20240307-v1:0/invoke`
with no client credentials. Repeated on this branch with
`--enable-bedrock-native true`.
- Observed result (key id and signature redacted):
```text
== origin/main (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}
== this branch (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": null}
== this branch, --enable-bedrock-native true
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}
```
- Not tested: Vertex against real Google ADC and a real
`*.googleapis.com` upstream (covered by the integration tests with a
static token source; no GCP credentials on the test machine).
## Runtime Rollout Safety
- Rollout-managed feature(s): `native_bedrock` (existing). The CLI flag
already maps `false` to a rollout disable, so with the new default the
feature reports disabled unless `--enable-bedrock-native true` is
passed. Vertex is not rollout-managed.
- Minimum rollout channel: unchanged
- Stable/default behavior changed: yes, both native routes are off by
default
- Kill switch / disable path: the routes are now off unless enabled
- Unsafe override required: no
- Qualification impact: none, the qualification script already passes
`--enable-bedrock-native true`
- Rollback path: pass `--enable-bedrock-native true` /
`--enable-vertex-native true` (plus `--vertex-allow-non-google-upstream
true` if the upstream is not a Google host), or revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Fix adapted from a proposal by HetCreep.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4851"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Reported by @JoaoMarcos44. Thank you for your contribution! 🩵
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
477 lines
26 KiB
Markdown
477 lines
26 KiB
Markdown
# Headroom Rust Rewrite — Developer Guide
|
|
|
|
This document covers the Rust port of Headroom. It is the only new top-level
|
|
doc created in Phase 0; longer-form design/plan writeups live elsewhere and
|
|
are not versioned in this repo.
|
|
|
|
## Workspace layout
|
|
|
|
```
|
|
Cargo.toml # workspace root
|
|
rust-toolchain.toml # pins stable rustc with rustfmt+clippy
|
|
crates/
|
|
headroom-core/ # library: shared types + transform trait surface
|
|
headroom-proxy/ # binary: axum /healthz (Phase 2 grows this)
|
|
headroom-py/ # PyO3 cdylib exposing `headroom._core`
|
|
headroom-parity/ # lib + `parity-run` CLI for Python parity tests
|
|
headroom-simulators/ # binary: deterministic local upstream stub for proxy tests
|
|
tests/parity/
|
|
fixtures/<transform>/*.json # recorded Python outputs (Phase 1 ports match)
|
|
recorder.py # Python-side fixture recorder
|
|
scripts/record_fixtures.py # entry point for running the recorder
|
|
```
|
|
|
|
`cargo build --workspace` builds every crate. `default-members` drops
|
|
`headroom-py` from `cargo run`/bare-`cargo test` flows so that `cargo test
|
|
--workspace` does not try to execute the PyO3 cdylib standalone (it can't
|
|
find `libpython` without a Python interpreter hosting it).
|
|
|
|
## Common commands
|
|
|
|
`just` is not installed on dev boxes here; a `Makefile` at the repo root
|
|
exposes the same targets:
|
|
|
|
| Target | What it does |
|
|
| --- | --- |
|
|
| `make test` | `cargo test --workspace` |
|
|
| `make test-parity` | Builds `headroom-py` via maturin, runs `parity-run run` |
|
|
| `make bench` | `cargo bench --workspace` |
|
|
| `make build-proxy` | Release-builds `headroom-proxy`, strips, prints size |
|
|
| `make build-wheel` | `maturin build --release -m crates/headroom-py/Cargo.toml` |
|
|
| `make fmt` | `cargo fmt --all` |
|
|
| `make lint` | `cargo fmt --check` + `cargo clippy --workspace -- -D warnings` |
|
|
|
|
## Running the proxy
|
|
|
|
`headroom-proxy` is a transparent reverse proxy. Phase 1 forwards HTTP/1.1,
|
|
HTTP/2, SSE, and WebSocket traffic verbatim to a configured upstream — no
|
|
provider logic yet. The intent is that operators run the existing Python
|
|
proxy on a private port and put `headroom-proxy` on the public port pointed
|
|
at it; end users notice nothing.
|
|
|
|
```bash
|
|
# Build
|
|
make build-proxy
|
|
./target/release/headroom-proxy --help
|
|
|
|
# Run against a local upstream (listens on 127.0.0.1:8787 by default)
|
|
./target/release/headroom-proxy \
|
|
--upstream http://127.0.0.1:8788
|
|
|
|
# Health checks
|
|
curl -s http://127.0.0.1:8787/healthz # => {"ok":true,...}
|
|
curl -s http://127.0.0.1:8787/healthz/upstream # => 200 if upstream reachable
|
|
```
|
|
|
|
### Operator runbook (Phase 1 cutover)
|
|
|
|
```bash
|
|
# 1. Move the Python proxy to a private port (e.g. 8788)
|
|
HEADROOM_HOST=127.0.0.1 HEADROOM_PORT=8788 python -m headroom.proxy & # or your existing launcher
|
|
|
|
# 2. Run the Rust proxy on the previously-public port (8787) pointing at it.
|
|
# The Python proxy sees every relayed request as coming from 127.0.0.1,
|
|
# so the Rust proxy enforces HEADROOM_PROXY_TOKEN itself (see below).
|
|
export HEADROOM_PROXY_TOKEN="$(openssl rand -hex 32)"
|
|
./target/release/headroom-proxy --listen 0.0.0.0:8787 --upstream http://127.0.0.1:8788 &
|
|
|
|
# 3. End users keep hitting :8787 unchanged.
|
|
# 4. Confirm passthrough:
|
|
curl -si http://127.0.0.1:8787/v1/models
|
|
# 5. Rollback = stop the Rust proxy and rebind Python back to 8787.
|
|
```
|
|
|
|
### Configuration flags
|
|
|
|
| Flag | Env var | Default | Notes |
|
|
| --- | --- | --- | --- |
|
|
| `--listen` | `HEADROOM_PROXY_LISTEN` | `127.0.0.1:8787` | bind address; a non-loopback address refuses to start without `HEADROOM_PROXY_TOKEN` unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` |
|
|
| `--proxy-token` | `HEADROOM_PROXY_TOKEN` | (unset) | inbound token required from non-loopback peers (below); prefer the env var |
|
|
| `--metrics-require-loopback` | `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK` | `false` | when set, `/metrics` is served only to loopback peers (403 otherwise); recommended on non-loopback binds |
|
|
| `--upstream` | `HEADROOM_PROXY_UPSTREAM` | (required) | base URL the proxy forwards to |
|
|
| `--upstream-timeout` | | `600s` | end-to-end request timeout (long for streams) |
|
|
| `--upstream-connect-timeout` | | `10s` | TCP/TLS connect timeout |
|
|
| `--max-body-bytes` | | `100MB` | for buffered cases; streams bypass |
|
|
| `--log-level` | | `info` | `RUST_LOG`-style filter |
|
|
| `--rewrite-host` / `--no-rewrite-host` | | rewrite | rewrite Host to upstream (default) |
|
|
| `--graceful-shutdown-timeout` | | `30s` | wait for in-flight on SIGTERM/SIGINT |
|
|
| `--stats` | `HEADROOM_PROXY_STATS` | `true` | native savings stats + `/dashboard` (below) |
|
|
| `--stats-path` | `HEADROOM_PROXY_STATS_PATH` | `~/.headroom/native_stats.json` | ledger persistence (honours `HEADROOM_WORKSPACE_DIR`) |
|
|
| `--enable-bedrock-native` | `HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` | (unset: off) | `true` or `--features native_bedrock` mounts `/model/{id}/invoke`, `/converse` and the streaming variants; they sign every request with this host's AWS credentials and do not authenticate the caller |
|
|
| `--enable-vertex-native` | `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE` | `false` | mount Vertex `:rawPredict` / `:streamRawPredict`; replaces the caller's `Authorization` with this host's Google ADC bearer (scope `--vertex-adc-scope`) and does not authenticate the caller |
|
|
| `--vertex-allow-non-google-upstream` | `HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` | `false` | by default the Vertex route only sends the ADC bearer when `--upstream` is `https://*.googleapis.com`, and answers 502 otherwise |
|
|
|
|
### Inbound authentication
|
|
|
|
The Rust proxy connects to the Python proxy from `127.0.0.1`, and the
|
|
Python proxy trusts loopback callers (no token needed, and access to
|
|
`/admin`, `/debug` and `/settings`). So the Rust proxy applies the same
|
|
`HEADROOM_PROXY_TOKEN` rules itself, before anything is forwarded:
|
|
|
|
- With a token set, a non-loopback peer must send it as
|
|
`X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`,
|
|
or gets `401 {"error":"unauthorized"}`. Loopback peers are exempt, as
|
|
in the Python proxy. The token is removed from the request before it
|
|
is forwarded (`Authorization` only when it carries the token).
|
|
- A caller that is not local here (a non-loopback peer, or a loopback
|
|
peer whose `Host` is not a loopback name) is never forwarded with a
|
|
`Host` that may name loopback (including scoped IPv6 such as
|
|
`[::1%lo]`), whatever `--rewrite-host` is and whatever the upstream
|
|
name resolves to: that `Host` is replaced with
|
|
`headroom-remote.invalid`, so the Python proxy's local-only routes
|
|
(`/admin`, `/debug`, `/settings`, ...) stay closed to it, with or
|
|
without a token.
|
|
- `GET /healthz`, `/healthz/upstream`, `/health`, `/livez` and `/readyz`
|
|
stay reachable without the token, and `GET`/`HEAD /api/hello` is
|
|
answered locally with an empty `200`.
|
|
- A non-loopback `--listen` with no token refuses to start. Set
|
|
`HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1` only when something else
|
|
already restricts who can reach the port.
|
|
|
|
### Savings stats & dashboard (native)
|
|
|
|
The Rust proxy records per-request savings/cost telemetry on every
|
|
lane it forwards — Anthropic `/v1/messages`, OpenAI Chat/Responses,
|
|
all four native Bedrock routes, and Vertex `rawPredict` /
|
|
`streamRawPredict` when those are enabled — and serves it locally (never tunnelled
|
|
upstream):
|
|
|
|
```bash
|
|
curl -s http://127.0.0.1:8787/stats | jq '.session' # since-boot totals
|
|
curl -s http://127.0.0.1:8787/stats | jq '.lifetime_by_model'
|
|
curl -s "http://127.0.0.1:8787/stats/timeseries?bucket=day" | jq '.points[-7:]'
|
|
curl -s "http://127.0.0.1:8787/stats/events?limit=5" # recent-request feed
|
|
open http://127.0.0.1:8787/dashboard # embedded UI, zero deps
|
|
```
|
|
|
|
Recording is deferred until the response is fully observed, so
|
|
streaming output/cache token counts come from the real SSE /
|
|
EventStream usage frames. USD figures come from the vendored LiteLLM
|
|
price table (`data/model_prices_and_context_window.json`); a model
|
|
missing from it records $0 and logs one WARN — refresh via
|
|
`scripts/refresh_model_limits.sh`. Input and output are priced
|
|
separately (`input_cost_usd` / `output_cost_usd`) because they bill
|
|
at different rates — output is typically 4-5x input, so a spend
|
|
figure that counted input alone would understate a short-prompt,
|
|
long-answer turn by most of its cost. `total_cost_usd` is the sum
|
|
and is what `/dashboard` shows as "Spend". Cache savings are NET: the
|
|
cache-read discount minus the cache-write premium (writes bill
|
|
above list price), floored at $0 per request — a warm-up turn that
|
|
only writes is not counted as negative savings, and gross-read
|
|
figures that overstate the benefit are never shown. Failed upstreams count as
|
|
failures and accrue no savings — as do proxy-side rejections
|
|
(missing AWS credentials, SigV4/ADC failures, oversized bodies), so
|
|
an operator-side outage reads as a failure spike rather than as
|
|
zero traffic. Aggregates (lifetime, per-model, 48 h hourly +
|
|
~13 mo daily buckets) persist as one atomic snapshot written off
|
|
the hot path every 10 s and on shutdown.
|
|
|
|
> **What reads 0 today.** Spend, token counts, and prompt-cache
|
|
> savings are live. `tokens_saved` / `compression_savings_usd` are
|
|
> wired end-to-end but will report 0 on real traffic until the
|
|
> live-zone per-type compressors are implemented — the dispatcher
|
|
> currently returns `NoCompression` for every well-formed body
|
|
> (`live_zone_mode_with_valid_body_returns_no_compression_pr_b2`
|
|
> pins that invariant). Nothing needs changing here when they land;
|
|
> the numbers start flowing on their own.
|
|
|
|
> **No auth; exposure is split by tier.** These endpoints bind
|
|
> wherever `--listen` points (default `127.0.0.1:8787`) and, for
|
|
> loopback peers or with no `HEADROOM_PROXY_TOKEN`, carry no auth, so the payload is split the same way the Python proxy splits
|
|
> its own `/stats`:
|
|
>
|
|
> - **Aggregates** — spend, tokens, per-model/provider rollups,
|
|
> history — are served to any caller that gets past the inbound gate
|
|
> (above), like the Prometheus `/metrics` endpoint next door: with
|
|
> `HEADROOM_PROXY_TOKEN` set, remote callers need the token for these
|
|
> too; without it, they are open to anyone who can reach the port.
|
|
> - **Per-request rows** — `recent_requests`, all of `/stats/events`
|
|
> — plus the ledger's filesystem path are served **only to local
|
|
> callers**: loopback peer *and* a loopback `Host` header (the
|
|
> second is the DNS-rebinding defence — a rebound request reaches
|
|
> the proxy *from* loopback, so the peer check alone doesn't hold).
|
|
> Remote callers get `recent_requests: null` and a 404 from
|
|
> `/stats/events`; the dashboard says so rather than erroring.
|
|
>
|
|
> A request id + model + timestamp is a different sensitivity tier
|
|
> from a counter, and `/metrics` never exposes it — so "same as
|
|
> /metrics" isn't a licence to hand it to the open internet. Opening
|
|
> the local-only tier to a trusted reverse proxy is tracked in
|
|
> [#1959](https://github.com/headroomlabs-ai/headroom/issues/1959),
|
|
> which is settling that policy for the Python dashboard first; this
|
|
> surface should adopt it rather than grow a second scheme.
|
|
> `--stats=false` removes the routes entirely.
|
|
|
|
### Picking the next port: invocation telemetry
|
|
|
|
Before porting another Python compressor to Rust, check what's actually
|
|
running. The Python proxy already exposes per-transform telemetry on
|
|
`/stats` (`headroom.proxy.prometheus_metrics`):
|
|
|
|
```bash
|
|
# Top compressors by invocation count (last process lifetime)
|
|
curl -s http://127.0.0.1:8788/stats | jq '.compressions_by_strategy'
|
|
# {
|
|
# "intelligent_context": 12453,
|
|
# "smart_crusher": 487,
|
|
# "search": 312,
|
|
# "diff": 28,
|
|
# "code": 0, # ← never fires; safe to defer porting
|
|
# ...
|
|
# }
|
|
|
|
# Per-transform timing (avg/max/count by transform name)
|
|
curl -s http://127.0.0.1:8788/stats | jq '.pipeline_timing'
|
|
|
|
# Token savings attributable to each strategy
|
|
curl -s http://127.0.0.1:8788/stats | jq '.tokens_saved_by_strategy'
|
|
```
|
|
|
|
This is the data the audit-cleanup PR (2026-04-30) recommended for
|
|
prioritizing the next Python → Rust port. Strategies with zero or
|
|
near-zero invocations are deferral candidates; strategies on the hot
|
|
path are porting candidates regardless of LOC count.
|
|
|
|
### Reserved paths
|
|
|
|
`/healthz` and `/healthz/upstream` are intercepted by the Rust proxy and
|
|
**not** forwarded. Operators must not name a real upstream route either of
|
|
these. Everything else is a catch-all forward.
|
|
|
|
## Maturin + Python wiring
|
|
|
|
`headroom-py` is a PyO3 cdylib that exposes `headroom._core` in Python. The
|
|
`extension-module` feature is opt-in so plain `cargo build --workspace` does
|
|
not try to link against `libpython` on systems that don't have it.
|
|
|
|
### First-time setup (clean venv recommended)
|
|
|
|
```bash
|
|
python3.11 -m venv /tmp/hr-rust-venv
|
|
source /tmp/hr-rust-venv/bin/activate
|
|
pip install maturin
|
|
cd crates/headroom-py
|
|
maturin develop # editable dev build, installs headroom._core
|
|
cd /tmp # IMPORTANT: step out of the repo root first
|
|
python -c "from headroom._core import hello; print(hello())"
|
|
# => headroom-core
|
|
```
|
|
|
|
> Why `cd /tmp`? The repo root also contains the Python `headroom/` package.
|
|
> Running the smoke import from the repo root makes Python resolve `headroom`
|
|
> to `./headroom/__init__.py` (the full SDK, which pulls in heavy deps) instead
|
|
> of the lightweight namespace package installed by maturin. Tests should
|
|
> either run outside the repo root, or ensure `headroom` is installed into
|
|
> the same venv (then the maturin-installed `_core.so` lands alongside it and
|
|
> both imports resolve).
|
|
|
|
### Release wheels
|
|
|
|
```bash
|
|
make build-wheel
|
|
# wheels land under target/wheels/
|
|
```
|
|
|
|
CI (`.github/workflows/rust.yml`) builds linux-x86_64, macos-arm64, and
|
|
macos-x86_64 wheels via `PyO3/maturin-action` and uploads them as artifacts.
|
|
|
|
## Parity harness
|
|
|
|
`crates/headroom-parity` owns the Rust-vs-Python oracle:
|
|
|
|
- JSON fixtures under `tests/parity/fixtures/<transform>/` (schema:
|
|
`{ transform, input, config, output, recorded_at, input_sha256 }`).
|
|
- `TransformComparator` trait — one impl per transform. Phase 0 stubs return
|
|
`Err(...)`; the harness flags those as `Skipped`, not panics.
|
|
- `parity-run` CLI: `cargo run -p headroom-parity -- run [--only TRANSFORM]`.
|
|
- Unit tests in `crates/headroom-parity/src/lib.rs` include a **negative
|
|
test** (`harness_reports_diff_for_divergent_comparator`) proving the
|
|
harness detects mismatched output before any real port lands.
|
|
|
|
### Recording fresh fixtures
|
|
|
|
```bash
|
|
source .venv/bin/activate # the main Python SDK venv
|
|
python scripts/record_fixtures.py # uses tests/parity/recorder.py
|
|
ls tests/parity/fixtures/*/ | sort | uniq -c
|
|
```
|
|
|
|
The recorder monkey-patches the in-process transform classes (see
|
|
`record_all()` in `tests/parity/recorder.py`). It does **not** modify any
|
|
file under `headroom/`.
|
|
|
|
## Known regressions in retired-Python components
|
|
|
|
The Stage 3b/3c.1b retirements deleted Python source for `DiffCompressor`
|
|
and `SmartCrusher` and replaced them with PyO3-delegating shims. The
|
|
2026-04-28 audit found that the retirements shipped with subsystems
|
|
silently disconnected. This section tracks each gap and its disposition
|
|
so they don't regress further or get forgotten.
|
|
|
|
### SmartCrusher
|
|
|
|
| Subsystem | State | Tracked by |
|
|
|---|---|---|
|
|
| TOIN learning loop | **Re-attached 2026-04-28.** Shim's `crush()` and `_smart_crush_content()` now call `toin.record_compression()` after a real compression. Filtered on `strategy != "passthrough"` to ignore JSON re-canonicalization. Best-effort: TOIN failures are logged at debug level and don't break compression. | `tests/test_smart_crusher_toin_attachment.py` |
|
|
| CCR marker emission knob | **Honored end-to-end 2026-04-29.** New `enable_ccr_marker: bool` field on Rust `SmartCrusherConfig`; `crush_array` checks it before emitting the `<<ccr:HASH>>` marker text and the CCR store write. Python shim flips it from `ccr_config.enabled and ccr_config.inject_retrieval_marker` — both flags collapse to the same Rust gate, since storing payloads under either off-switch makes no sense. Scope: gates only the row-drop sentinel path; Stage-3c.2 opaque-string CCR substitutions still emit always (no Python equivalent, no production caller asks for suppression). | `tests/test_smart_crusher_toin_attachment.py` + `crates/headroom-core/.../crusher.rs::tests::enable_ccr_marker_*` |
|
|
| Custom relevance scorer | **Closed (fail-loud) 2026-04-29.** `relevance_config` and `scorer` constructor args remain in the signature for source compat, but the shim raises `NotImplementedError` when either is non-None — silently dropping a user-supplied scorer is a textbook silent-fallback bug. Full plumbing waits on Stage-3c.2's relevance-crate Python bridge. | `tests/test_smart_crusher_toin_attachment.py::test_custom_*_arg_raises_not_implemented` |
|
|
| Per-tool TOIN learning hook | **Re-attached partially.** `_smart_crush_content` accepts `tool_name` and now threads it into the TOIN record. The hook is best-effort — it improves `query_context` aggregation but doesn't drive per-tool overrides yet. | `tests/test_smart_crusher_toin_attachment.py::test_smart_crush_content_records_to_toin` |
|
|
|
|
### DiffCompressor
|
|
|
|
| Subsystem | State |
|
|
|---|---|
|
|
| Adaptive context windows | Honored byte-for-byte (parity fixture-locked). |
|
|
| TOIN integration | Never had one — DiffCompressor records via `_record_to_toin` in ContentRouter, which already runs for non-SmartCrusher strategies. No regression. |
|
|
|
|
### Phase 3e.1 — `signals/` trait module + KeywordDetector (2026-04-29)
|
|
|
|
The Python `error_detection.py` regex registry was retired and reborn as a
|
|
trait + tier system in `crates/headroom-core/src/signals/`. See
|
|
`signals/README.md` for the full architecture; the highlights:
|
|
|
|
- **Per-granularity traits.** `LineImportanceDetector` ships today; future
|
|
`ContentTypeDetector` and `ItemImportanceDetector<I>` will follow as their
|
|
consumers get touched.
|
|
- **`Tiered<T>` combinator.** Composition, not inheritance. Future ML
|
|
detectors slot in as new tiers without changes to `KeywordDetector` or
|
|
any caller.
|
|
- **One concrete impl.** `KeywordDetector` (aho-corasick) is the only tier
|
|
registered today. **No NoOp/stub impls** — per project no-silent-fallbacks
|
|
rule, future tiers land with their real implementations.
|
|
- **Bug fixes baked in.** `ERROR_KEYWORDS` regex now includes
|
|
`timeout|abort|denied|rejected` (previously drifted from the keyword set);
|
|
`token` dropped from `SECURITY_KEYWORDS` (false-positived on every LLM
|
|
metric reference). Both fixed in the Python regex too via the shim that
|
|
recompiles patterns from the Rust-exposed keyword tables.
|
|
- **Companion canonical extension path.** `signals/README.md` documents
|
|
the BGE classifier head — a 384-dim → 4-class softmax on top of the
|
|
already-loaded `bge-small-en-v1.5` embedder — as the natural ML tier.
|
|
Two alternatives kept open: distilled tinyBERT in ONNX, logistic
|
|
regression on lexical features.
|
|
|
|
### Phase 3g (queued) — Compression Pipeline Formalization (issue #315)
|
|
|
|
Strategic decision 2026-04-29: after Phase 3e (compressor ports) and
|
|
Phase 3f (Rust MCP scaffold) wrap, formalize the lossless-then-lossy-
|
|
then-CCR ordering as a cross-cutting `CompressionPipeline` orchestrator
|
|
+ `LosslessTransform` / `LossyTransform` traits in
|
|
`crates/headroom-core/src/pipeline/`. Existing compressors get
|
|
refactored as compositions of pluggable transforms. The crucial design
|
|
choice — **parsers for structure, models at the prose/structure
|
|
boundary** — is captured in issue #315 and
|
|
`memory/project_lossless_first_pipeline.md`. Do NOT start coding before
|
|
3e/3f finish.
|
|
|
|
### Watch list (potential regressions, not yet audited)
|
|
|
|
- `CCRConfig.enabled=False` end-to-end — **closed 2026-04-29**. Both `enabled=False` and `inject_retrieval_marker=False` collapse to the same Rust `enable_ccr_marker=False` gate (no marker, no store write). See the SmartCrusher table above.
|
|
- `SmartCrusherConfig.use_feedback_hints=False` — config field is forwarded to Rust but its honoring inside the Rust crusher hasn't been verified against a parity fixture for the disabled path.
|
|
|
|
When any item above changes, update both this section and the test file. The shim's docstring also references this section — keep them aligned.
|
|
|
|
## Phase 0 Blockers
|
|
|
|
These are known limitations for Phase 0. They are tracked here so Phase 1
|
|
doesn't rediscover them.
|
|
|
|
- **`cache_aligner` fixtures**: `CacheAligner.apply()` takes
|
|
`(messages, tokenizer, **kwargs)` — a `Tokenizer` is provider-specific and
|
|
its cheapest `NoopTokenCounter` / `TiktokenTokenCounter` construction still
|
|
requires pulling `headroom.providers.*` which imports the full observability
|
|
stack (opentelemetry, etc). The recorder records `cache_aligner` only if a
|
|
usable tokenizer is cheaply available; otherwise it logs a blocker and
|
|
skips. See `recorder.py::_build_cache_aligner_tokenizer`.
|
|
- **`ccr` is not a single class**: The repo has `CCRToolInjector`,
|
|
`CCRResponseHandler`, `CCRToolCall`, `CCRToolResult` etc. rather than a
|
|
single `CCR` class. The recorder targets the encoder-style entry point
|
|
most analogous to the Rust port (`CCRToolInjector.inject_tool` and
|
|
`CCRResponseHandler.parse_response`). If Phase 1 wants a different split
|
|
it should update `recorder.py::record_all` accordingly.
|
|
- **Pre-commit hook noise**: `scripts/sync-plugin-versions.py` mutates
|
|
`.claude-plugin/marketplace.json`, `.github/plugin/marketplace.json`, and
|
|
`plugins/headroom-agent-hooks/**/plugin.json` on every commit. Those
|
|
changes are harmless but each commit in Phase 0 picks them up. Phase 1
|
|
does not need to do anything special — just let the hook run.
|
|
- **`rust-toolchain.toml`** now pins `channel = "1.95.0"` (previously tracked
|
|
`"stable"`, which let CI drift ahead of local dev boxes — see the file's own
|
|
comment for the 2026-04-27 incident this fixed). Resolved; kept here for
|
|
history.
|
|
|
|
## Multi-worker deployment — CCR fragmentation
|
|
|
|
**Status:** two persistent CCR backends are available. The single-`--workers`
|
|
recommendation no longer applies once you select a persistent backend.
|
|
|
|
### Backend selection
|
|
|
|
`crates/headroom-core/src/ccr/backends/` ships three implementations of
|
|
the `CcrStore` trait:
|
|
|
|
| Backend | When to use | Persistence | Multi-worker safe |
|
|
| ---------------------- | ------------------------------------------- | ----------- | -------------------------- |
|
|
| `InMemoryCcrStore` | Tests, single-worker prototyping | No | No |
|
|
| `SqliteCcrStore` (default) | Single-instance prod / single-host fleet | Yes (file) | Yes (sticky session) |
|
|
| `RedisCcrStore` (opt-in) | Multi-host / horizontally-scaled prod | Yes (Redis) | Yes (no stickiness needed) |
|
|
|
|
`backends::from_config` picks one at startup from the operator's
|
|
`CcrBackendConfig`. **Init failures surface to the caller**
|
|
(`feedback_no_silent_fallbacks.md`) — a misconfigured DB path or
|
|
unreachable Redis URL aborts startup rather than silently degrading to
|
|
in-memory.
|
|
|
|
### When does what work?
|
|
|
|
- **`SqliteCcrStore`** is the default for new deploys. The DB file lives
|
|
on the local disk; multiple workers on the **same host** share it via
|
|
SQLite's WAL-mode locking, so `--workers N` works as long as a sticky
|
|
load balancer routes each session to the same host. Survives proxy
|
|
restarts: a new worker that opens the same DB file recovers every
|
|
in-flight `<<ccr:HASH>>` marker.
|
|
- **`RedisCcrStore`** (cfg-gated behind the `redis` feature) is the
|
|
drop-in for **horizontally-scaled** deployments. Every worker on
|
|
every host hits the same Redis instance; no sticky session is
|
|
required at any layer of the LB. Enable with `--features redis` in
|
|
the proxy crate's Cargo build.
|
|
- **`InMemoryCcrStore`** is fine for tests and single-worker
|
|
development. Production deployments using it lose every
|
|
`<<ccr:HASH>>` marker on restart and fragment across workers — keep
|
|
it confined to local boxes.
|
|
|
|
### What goes wrong with the in-memory backend on `--workers N > 1`
|
|
|
|
Each uvicorn worker is a separate Python process. The following state is
|
|
fragmented across workers:
|
|
|
|
1. **Python `CompressionStore`** — defaults to `SQLiteBackend` at
|
|
`workspace_dir()/ccr_store.db` (restart-safe, shared across workers) when
|
|
`HEADROOM_CCR_BACKEND` is unset or `"sqlite"` (`_create_default_ccr_backend`
|
|
in `headroom/cache/compression_store.py`). Set `HEADROOM_CCR_BACKEND=memory`
|
|
to opt into the per-process `InMemoryBackend` instead — that is the setting
|
|
this section's fragmentation risk actually applies to; the log messages in
|
|
`headroom/proxy/server.py` (see below) still assume in-memory-by-default and
|
|
are stale on this point.
|
|
2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py`)
|
|
— per-session `CompressionCache` dict (instance var, always per-worker).
|
|
3. **`HeadroomProxy.session_tracker_store`** — per-session prefix-tracker
|
|
state derived from Anthropic's `cache_read_input_tokens` responses
|
|
(instance var, always per-worker).
|
|
4. **TOIN learner state** — writes snapshots to `~/.headroom/toin.json` but
|
|
keeps per-process in-memory state; pattern statistics on one worker are not
|
|
visible to others until the next disk flush.
|
|
|
|
When uvicorn round-robins requests across workers, a session whose
|
|
turn-1 landed on worker A may have turn-2 land on worker B. Worker B has
|
|
zero knowledge of what worker A did, the `<<ccr:HASH>>` marker resolves
|
|
to `None`, and the model sees an opaque directive it can't act on.
|
|
Switching to `SqliteCcrStore` (default) or `RedisCcrStore` resolves the
|
|
CCR fragmentation; a sticky-session load balancer resolves all of them.
|
|
|
|
### Detecting it in the wild
|
|
|
|
The proxy emits a `WARNING`-level log line on startup when `--workers N > 1`.
|
|
When `HEADROOM_CCR_BACKEND` is unset (default InMemoryBackend), the warning
|
|
includes CCR retrieval failures and suggests setting `HEADROOM_CCR_BACKEND=sqlite`.
|
|
When a cross-worker backend is already configured, the warning covers only the
|
|
remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker).
|