Commit Graph
9 Commits
Author SHA1 Message Date
9b2dec56f9 feat(proxy): show per-request /stats metadata to token-authenticated callers (#4168)
## Description

`/stats` and `/stats-lifetime` withhold the per-request tail
(`recent_requests`, `request_logs`) and project names from every
non-loopback caller, unless the request's `Host` is an IP literal
**and** the peer is inside
`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS`.

That rule cannot be met by a common deployment: the proxy runs on a
container platform with `HEADROOM_PROXY_TOKEN` set, and the dashboard
sits on its own host behind an authenticating reverse proxy (here Entra
ID sign-in in front of a small Caddy that adds `x-headroom-proxy-token`
after the user signed in). That reverse proxy reaches Headroom with a
domain `Host` (the platform routes internal calls by Host, so it cannot
be an IP) and from a peer address that is not stable enough to list. The
dashboard loads, the aggregate counters work, and the "Recent Requests"
table stays empty with no indication why.

This PR also admits callers whose proxy token the security gate has
already verified (`request.state.proxy_authenticated`, which the gate
sets only after the constant-time token comparison).

**This is a nice-to-have, not a bug fix.** The current behaviour is
documented and deliberate, and the existing trusted-CIDR path keeps
working unchanged. If you would rather not widen who sees this metadata,
closing the PR is fine; we carry the change on our fork either way.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: add `_request_can_view_stats_metadata`:
true when the gate verified the proxy token, otherwise the existing
`_request_can_view_dashboard_metadata` result. `/stats` and
`/stats-lifetime` use it. The helper is deliberately not used by
`/settings*` or `/transformations/feed`, which keep their loopback /
trusted-client guards.
- `tests/test_proxy_stats_recent_requests.py`: add a regression test
(domain `Host`, non-loopback peer, token sent) for `/stats` and
`/stats?cached=1`, asserting the tail is served with the token and that
the gate still answers 401 without it or with a wrong one.
- `docs/content/docs/security-model.mdx`: new section **What the proxy
token unlocks**. It lists exactly which fields a token holder now also
sees: `recent_requests`, `request_logs` and `config` on `/stats` (and
`?cached=1`), and `projects` and `persistence.error` on
`/stats-lifetime`. It says the settings routes and the
`require_loopback` routes stay closed to a token. It also warns that the
token is one shared secret, not a per-caller identity: every holder sees
every caller's request metadata, so a deployment that hands one token to
several clients that must not see each other's metadata should run one
proxy per client. The dashboard-CIDR table row now points to the
section.
- `docs/content/docs/proxy.mdx` (the `HEADROOM_PROXY_TOKEN` row) and
`SECURITY.md` (the local-control-surfaces bullet): one line each
pointing to the same consequence.
- `headroom/proxy/server.py` docstrings: the `/stats` handler and
`_request_can_view_stats_metadata` now describe the actual access rule.
The helper no longer claims the change widens nothing.

Who gains access: holders of `HEADROOM_PROXY_TOKEN`. This does widen
what the token grants. Before, it granted inference and the read-only
operator routes. Now it also grants the request metadata of every caller
on the proxy. With no token configured, nothing changes (the flag is
never set).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom/proxy/server.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

At `1c6666d` merged with current `main` (`4abbf07`):

```text
$ python -m pytest -q tests/test_proxy_stats_recent_requests.py tests/test_proxy_loopback_gating.py \
    tests/test_dashboard_accessibility.py tests/test_proxy_hardening.py
214 passed, 1 skipped in 44.60s

$ python -m mypy headroom/proxy/server.py
Success: no issues found in 1 source file

$ ruff check headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, this branch at `4abbf07`
(`1c6666d` merged with current `main`). A real `headroom proxy` with a
throwaway `HEADROOM_PROXY_TOKEN`, bound to a non-loopback LAN address,
with `HEADROOM_OFFLINE=1` and `--anthropic-api-url` pointed at a local
stub that answers `/v1/messages` with a fixed Anthropic message:
  ```text
headroom proxy --host 192.168.0.48 --port 18788 --no-optimize
--anthropic-api-url http://127.0.0.1:18999
  ```
- Exact command / steps: a client on `192.168.0.48` (not loopback, in no
trusted CIDR) sends what the dashboard's reverse proxy sends. That is
`Host: headroom-dashboard.example.test` (a domain, as the platform
routes by Host) plus `x-headroom-proxy-token`. It first makes one
token-authenticated `POST /v1/messages` so there is a request to list,
then reads the stats endpoints with the token, without it, and with a
wrong one. It also tries the settings and loopback-only routes with the
token. Same script against the same build with only
`headroom/proxy/server.py` swapped back to `main`'s, for the before
column.
- Observed result: verified token holders receive shared stats metadata,
while missing or incorrect tokens are rejected; exact before/after
responses follow.

  ```text
request (domain Host, non-loopback peer) before (main) after (this PR)
POST /v1/messages token 200 200
GET /stats token 200, recent_requests/request_logs/ 200,
recent_requests=1 request_logs=1
config absent config present, first={"provider":
"anthropic","model":"claude-sonnet-5-5"}
GET /stats?cached=1 token 200, same fields absent 200, same as /stats
GET /stats-lifetime token 200, projects absent 200, projects present
GET /stats no token 401 401
GET /stats wrong token 401 401
GET /stats-lifetime no token 401 401
GET /settings token 404
GET /settings/schema token 404
GET /dashboard/settings token 404
GET /transformations/feed token 404
  ```

The proxy log shows `event=proxy_auth_rejected path=/stats …
reason=missing_token`, `… reason=bad_token` and `path=/stats-lifetime …
reason=missing_token` for the three rejections.
- Not tested: the full suite (only the four files above). The deployed
dashboard end to end: the run above reproduces its request shape against
a real proxy, without the Entra/Caddy hop in front.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only when `HEADROOM_PROXY_TOKEN` is
set. Token-authenticated remote callers now receive `recent_requests`,
`request_logs` and `config` from `/stats`, and `projects` and
`persistence.error` from `/stats-lifetime`, for every caller of the
proxy. Loopback and trusted-CIDR behaviour is unchanged, and the
settings and loopback-only routes are not opened to the token.
- Kill switch / disable path: unset the token (then nothing changes), or
revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: done; see the security-model section above.
- A possible alternative that avoids widening the rule: an opt-in env
var (e.g. `HEADROOM_STATS_METADATA_FOR_TOKEN=1`). Happy to switch to
that if you prefer the default to stay strict.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013MpfLG8sVzqPhD34teqdLD

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4168"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `09689b73ddeb8879a2f2c68726f9e885cd9bfdee`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Mechanically merged upstream/main e4a60ed7 and re-reviewed
stats-metadata authorization plus the existing security/runtime
discussion. Only callers whose proxy token authentication succeeded gain
the additional per-request stats metadata; settings and loopback-only
routes retain their separate guards. Documentation explicitly states
that token holders share every caller's traffic metadata. No contributor
production logic was rewritten.

All 260 focused stats/recent-request, loopback-gating and hardening
tests pass; Ruff lint/format and whitespace checks pass. Existing
before/after real HTTP evidence remains explicitly scoped; local tests
use the existing native extension, not a fresh native build. Applied the
shared test-only Gemini streamed-request fixture repair for
independently established main failures; all ten cases pass. Formal
approval awaits exact-head hosted qualification.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `fa07187e666ca7895d1cc5708a9d930f38399fe1`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical merge with upstream main
c01cf07d94. Reviewed current author
discussion and Devin runtime/documentation findings. Additional stats
metadata is allowed only after successful proxy-token authentication or
the existing dashboard authorization; settings and loopback-only routes
retain separate guards. The security documentation explicitly states
that one shared token exposes every caller's traffic metadata to every
holder. No contributor authorization policy rewritten.

Fresh focused stats/recent-request, loopback-gating and hardening
verification: 260 tests passed with six existing deprecation/OpenAPI
warnings. Changed-file Ruff lint/format and main-relative whitespace
checks pass; owned tree clean. Existing non-loopback before/after HTTP
evidence remains scoped; no fresh live replay or branch-native build is
claimed. Python tests use the current-main native DLL. No new source
blocker found; approval awaits exact-head hosted checks. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `20b62cc35f7a005b91a6c9054c41cfd51ac29c37`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean main integration against 0ddaa56c21.
315 stats/recent-request, loopback/security gating and incoming-main
tests passed, 11 optional skips, 6 warnings. Exact incoming-main
patches, lint/format, offline lock and whitespace checks pass. Complete
recorded native input-set/hash and artifact checksum verified before
existing DLL reuse; no fresh native build claimed.

Current author discussion and both resolved Devin threads/replies
reviewed. Verified token-authenticated state expands only stats metadata
authorization; settings and require_loopback guards remain separate.
Documentation explicitly lists fields and warns that the shared token
exposes every caller's request metadata to every holder, with separate
proxies required for isolation. This is an intentional documented
shared-trust policy, not a per-user identity. Prior author's
non-loopback real-proxy before/after evidence is retained with its
scope; this refresh adds local tests, not a fresh live replay or
browser/reverse-proxy proof. No new source blocker found.

Source/test/prose delta has no
workflow/setup/dependency/CLI/install/provider/OpenClaw/SDK path
changes. Inherited flagged workflow-validation and Init/Wrap bootstrap
paths are excluded. Docs build uses unchanged-main
workflow/package/lock/config and this PR changes only prose there.
Exact-head Actions and rollup remain pending before approval. Nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-07 20:42:27 -06:00
M BarryandJD Davis ba824539d6 fix(proxy): report read maturation as inert in cache mode (#4198)
## Description

`--read-maturation` combined with `--mode cache` starts cleanly, and
`/stats.rollout` reports the feature as requested and enabled. It can
never fire, and nothing says so.

Here is why:

- In cache mode the Anthropic handler sets `frozen_message_count =
_strict_previous_turn_frozen_count(...)` (`handlers/anthropic.py`). That
freezes every message except the final user turn.
- Cache mode never reaches the token-mode `prepare_turn` clamp, so that
count is passed straight to `ReadMaturationManager.apply(...,
frozen_message_count=...)`.
- `apply` skips every message with `i < frozen_message_count`.
- The only Reads it ever examines are therefore the tool results in the
latest user turn. Those answer the assistant message just before, so
`quiet_turns = held_turns = 0`. That is below `quiesce_turns` (default
5) and `max_hold_turns` (default 25), so they are always held.
- By the time a Read could have quiesced, it sits in the frozen prefix.

The handler's per-request summary is logged at DEBUG, so a deployment
sees no output either way. One persistent proxy ran this combination for
over five weeks, about 51,000 real requests, with zero
`read_maturation:` transforms. The existing handler tests all use
`mode="token"`, so no test caught it.

Holding is harmless in cache mode, because a "held" Read is forwarded
verbatim. So this PR makes the inert state visible rather than changing
request handling:

- At startup, enabled read maturation is reported once: a **WARNING**
`Read maturation: INERT (cache mode freezes a Read before it can
quiesce); use --mode token or disable read_maturation` in cache mode,
otherwise an INFO line with the effective quiesce, hold and size
settings. The hint says "disable read_maturation" rather than naming a
flag, because the feature can come from `--read-maturation` or from
`HEADROOM_FEATURES`.
- `/stats` gains `read_maturation: {enabled, effective, inert_reason}`,
beside `rollout`, which only records that the feature was granted.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/modes.py`: `read_maturation_inert_reason(mode, *,
quiesce_turns, max_hold_turns)` returns the reason string under cache
mode, else `None`. A threshold of 0, which the env path
(`HEADROOM_READ_MATURATION_*_TURNS`) accepts even though the CLI
requires at least 1, matures a Read in the still-mutable latest turn, so
it is not reported as inert. The rule lives in one place for both
callers.
- `headroom/proxy/server.py` (startup banner): after the cache-mode
lines, log the INERT WARNING or an ENABLED INFO line with
`quiesce_turns` / `max_hold_turns` / `min_size_bytes`.
- `headroom/proxy/server.py` (`/stats`): add `read_maturation: {enabled,
effective, inert_reason}` through a small `_read_maturation_status(cfg)`
helper.
- `tests/test_proxy_stats_recent_requests.py`:
`test_read_maturation_reports_whether_it_can_fire`, parametrized over
cache, token, and cache with `quiesce_turns=0`. It asserts the `/stats`
fields and that exactly one `Read maturation:` startup record is logged,
at WARNING (inert) or INFO (effective).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy_stats_recent_requests.py -k read_maturation -v
tests/test_proxy_stats_recent_requests.py::test_read_maturation_reports_whether_it_can_fire[cache-5-False-WARNING] PASSED [ 33%]
tests/test_proxy_stats_recent_requests.py::test_read_maturation_reports_whether_it_can_fire[token-5-True-INFO] PASSED [ 66%]
tests/test_proxy_stats_recent_requests.py::test_read_maturation_reports_whether_it_can_fire[cache-0-True-INFO] PASSED [100%]
================= 3 passed, 5 deselected, 2 warnings in 1.74s ==================
# all three cases fail on main (no "read_maturation" key in /stats, no startup record)

$ pytest tests/test_proxy_stats_recent_requests.py tests/test_read_maturation*.py
======================== 40 passed, 3 warnings in 4.86s ========================

$ ruff check headroom/proxy/modes.py headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
All checks passed!
$ ruff format --check headroom/proxy/modes.py headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
3 files already formatted
$ mypy headroom/proxy/modes.py headroom/proxy/server.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.13. A persistent `headroom proxy` (0.38 →
0.40.0) serving Anthropic Messages traffic from an agentic coding
client, with `--mode cache --lossless --code-aware --read-maturation`,
`HEADROOM_ROLLOUT_CHANNEL=beta` and default maturation thresholds.
- Exact command / steps: (1) `GET /stats` on the running proxy. (2)
Counted `PERF` records and `read_maturation:` transforms in the proxy's
archived PERF log from 2026-09-05 to 2026-10-13. (3) Traced
`frozen_message_count` from `_strict_previous_turn_frozen_count` into
`ReadMaturationManager.apply`.
- Observed result: (1) `rollout` reports `read_maturation:
requested=True, enabled=True, decision=explicit` on channel `beta`, so
gating is not the cause. (2) 50,970 PERF records and **0**
`read_maturation:` transforms from real traffic; the only 5 hits in the
window were test-fixture requests. (3) Only the final user turn is ever
unfrozen, so every Read examined is at 0 quiet turns and is held.
- Not tested: a full restart of that deployment on this branch. The
startup line and the `/stats` block were exercised through `create_app`
plus `TestClient` in the new test. I also did not try to make maturation
actually work in cache mode; see Additional Notes.

## Runtime Rollout Safety

- Rollout-managed feature(s): `read_maturation` (beta). Gating is
unchanged; this only reports on it.
- Minimum rollout channel: unchanged (beta).
- Stable/default behavior changed: no request-path change. `effective`
describes the mode and thresholds only; maturation runs in the Anthropic
messages handler and skips bypassed requests. One new startup log line
appears when read maturation is enabled, and `/stats` gains one additive
key.
- Kill switch / disable path: the existing one; drop `--read-maturation`
/ `HEADROOM_FEATURES`.
- Unsafe override required: no.
- Qualification impact: `/stats` gains `read_maturation`; existing keys,
including `rollout`, are unchanged. Black-box tools reading `rollout`
see the same payload.
- Rollback path: revert the commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- **Why a warning, not a refusal:** existing deployments run this
combination, and failing startup on upgrade would take them down for a
feature that is a no-op there. Happy to switch to refusing the
combination if maintainers prefer.
- **Possible follow-up, not in this PR:** cache mode does have one
moment when rewriting the prefix is free, which is cold-prefix
recompaction (`HEADROOM_COLD_RECOMPACT`, after the cache has lapsed).
Letting maturation run there would give it a real effect in cache mode.
That is a design change, and I'd rather discuss it first.
- **Also possible:** per-reason skip counters (frozen, below min size,
holding, already marked) in `MaturationResult`, surfaced in `/stats`. I
left this out to keep the PR small.
- **Docs:** I didn't find a page describing `--read-maturation`'s mode
interaction. Happy to add a line wherever maintainers want it.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4198"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-15 16:31:05 -05:00
Tejas ChopraandTejas Chopra 3145242645 Unify savings attribution across stats, perf, metrics, and dashboard (#2976)
## Summary

Adds a small provider-neutral savings attribution seam. Named sources
can attach realized or projected token/USD deltas to a request without
changing headline arithmetic or introducing private-package inventory
into OSS.

Also fixes the Anthropic buffered lifecycle so normal successful
responses run response hooks, applies stream-safety filtering, includes
tool savings in per-model perf totals, and surfaces the same breakdown
in request logs, `/stats`, `headroom perf`, Prometheus, OTEL, and the
dashboard.

## Why

Request-local savings were split between canonical token deltas,
process-global extension counters, and tool-only tags. This made correct
headline totals possible while losing attribution in perf, recent
requests, metrics, and the dashboard. Normal Anthropic responses also
skipped response hooks unless CCR ran.

## Validation

- 74 focused tests passed: turn hooks, OpenAI hook lifecycle, outcome
funnel, perf formats, and tool-search repair
- Ruff passes on all changed Python files
- Existing compression-observability suite: 11 passed; 2 tokenizer-cache
tests require network access to fetch the tiktoken vocabulary

## Compatibility

No named private packages or private inventory are encoded in OSS.
Existing hooks remain source-compatible because all new TurnContext
fields are optional.

---------

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
2026-08-13 17:13:23 -07:00
3077ac81e8 feat: add deterministic runtime rollout controls (#1490)
## Description

Establish one centrally resolved, observable, deterministic, versioned
runtime rollout-control mechanism for Headroom. Runtime rollout controls
which behaviors an already-built artifact may expose; it does not select
or qualify a Headroom release/version.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Bug fix (non-breaking change that fixes rollout enforcement
regressions)
- [x] Documentation update
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`,
`--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared
by Python configuration boundaries.
- Added schema/policy versions, canonical registry and snapshot SHA-256
identities, per-feature decision reasons, disable precedence, unsafe
qualification poisoning, strict CLI validation, and fail-closed
environment handling.
- Added `headroom rollout status --json`, Python `/stats.rollout`, and
Rust `/rollout/status` runtime provenance.
- Added equivalent Rust snapshot semantics and shared Python/Rust policy
vectors while retaining language-specific feature registries.
- Enforced rollout policy at alternate Python server composition roots
so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate.
- Preserved typed rollout snapshots across multi-worker serialization
with schema, policy, registry, snapshot-digest, type, and feature-name
validation.
- Made loopback runtime output-shaper updates replace the immutable
snapshot atomically for request readers, retain explicit request/disable
provenance, preserve channel and kill-switch precedence, invalidate
cached stats, and return the effective rollout decision.
- Made `headroom learn --verbosity --apply` report a channel-blocked
update instead of claiming the shaper is live.
- Made explicit CLI feature flags fail loudly when their current channel
blocks them.
- Made persistent interceptor installation select canary automatically,
or reject an explicitly insufficient channel unless the break-glass
override is set.
- Updated architecture, proxy, rollout, learn, and output-shaper
documentation with required channels and hot-reload semantics.

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .` and `ruff format --check .`)
- [x] Type checking passes (`mypy headroom --ignore-missing-imports`)
- [x] New regression tests added for every corrected behavior
- [x] Rust tests and production-target Clippy pass
- [x] Documentation build passes

### Test Output

```text
Focused rollout coverage suite
57 passed; headroom.rollout + rollout CLI: 98% coverage

Affected proxy/rollout/transform/governance suites
222 passed; 0 failed

Final changed regression suites
100 passed; 0 failed

Cross-module hot-reload isolation regression
6 passed; 0 failed

cargo test -p headroom-core -p headroom-proxy --quiet
headroom-core: 924 passed; 1 ignored
headroom-proxy and integration suites: all passed

cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings
cargo fmt --all -- --check
ruff check .
ruff format --check .
mypy headroom --ignore-missing-imports
git diff --check
All passed

cd docs && npm run build
Compiled successfully; 164 static pages generated
```

The unsharded Windows-only CI selection exposed unrelated baseline
failures, principally the existing `sqlite:///C:\\...` URL parser
producing an invalid `\\C:\\...` path. At commit `8e793a80`, all 52
completed GitHub checks passed; the only other conclusions are expected
skips and superseded governance jobs.

## Real Behavior Proof

- **Environment:** Windows checkout on Python 3.13.3 and the current
Rust workspace, based on upstream `main` at `93f2d7a2`.
- **Exact command / steps:** Exercised canary and beta feature requests
through CLI status, Python `/stats.rollout`, Rust `/rollout/status`,
multi-worker payload round trips, loopback `/admin/runtime-env`, real
proxy request shaping before/after hot reload, installer manifest
generation, and shared Python/Rust policy vectors.
- **Observed result:** Stable blocks unstable requests; disable wins
over explicit/default/legacy/unsafe paths; unsafe state reports
`qualification_eligible=false`; worker handoff rejects tampering;
running output shaping changes only when the effective beta policy
permits it; explicit blocked flags fail with actionable diagnostics.
- **Not tested:** Live production traffic requiring provider
credentials, or future artifact qualification/promotion automation
(intentionally out of scope).

## Runtime Rollout Safety

- **Rollout-managed features:** Python `tool_result_interceptors`,
`proxy_output_shaper`, `read_maturation`; Rust `native_bedrock`,
`openai_responses_streaming`, `canary_probe`.
- **Minimum rollout channel:** Registry-defined per feature; process
default is `stable`.
- **Stable/default behavior changed:** No unstable feature becomes
enabled by default. Explicit blocked CLI flags now fail instead of
silently doing nothing.
- **Kill switch / disable path:**
`HEADROOM_DISABLE_FEATURES=<comma-separated feature names>`; explicit
disable has highest precedence, including over the unsafe override.
- **Unsafe override required:** No.
`HEADROOM_UNSAFE_ALLOW_UNSTABLE_FEATURES=1` is break-glass only and
makes qualification evidence ineligible.
- **Qualification impact:** Adds machine-readable policy/snapshot
identities and eligibility; does not implement qualification itself.
- **Rollback path:** Set the named disable list for operational
rollback, lower the channel, or revert this PR.

## Review Readiness

- [x] I have performed a full diff review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have made corresponding documentation changes
- [x] My changes generate no new warnings
- [x] I added tests that reproduce and prevent every regression fixed
during review
- [x] New and existing affected tests pass locally
- [x] I did **not** edit `CHANGELOG.md`; release-please generates it
from the Conventional Commit PR title

## Additional Notes

Out of scope: artifact candidates, benchmark orchestration,
qualification manifests/gates, promotion automation, release branches,
publication guards, and release-risk classification. Those workflows can
consume the rollout registry digest, runtime snapshot digest, decision
reasons, and qualification eligibility through supported black-box
interfaces.

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JD Davis <jd@JDH-AIR-00.local>
2026-08-12 23:16:54 -05:00
Tejas Chopra e0ce4b1d48 fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description

Removes both third-party CLI context tools — **rtk** and **lean-ctx** —
and with them the context-tool selector itself. Headroom no longer
downloads, installs or configures either one, and there is no
replacement.

The previous pass (#2344) gated only three entry points inside
`headroom/cli/wrap.py`. That left the feature reachable in practice:

| Gap | Effect |
|---|---|
| `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global
--auto-patch` from bash/PowerShell, **bypassing the Python gate
entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook
regardless of `HEADROOM_RTK` |
| `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was
broken by default**: `rtk_required=True` met a gate returning `None` →
`SystemExit(1)`. Invisible because all 8 openhands tests patched
`_ensure_rtk_binary` to a fake path |
| `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to
`rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker
polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) |
| No cleanup path | Nothing removed artifacts an earlier default had
installed, so a machine that once ran the old default kept rtk in the
loop forever (#1669, #1955) |

Also worth noting: the rtk binary download had **no SHA or signature
verification** — only `rtk --version` as a smoke test.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

**Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages,
`headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` /
`_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` /
`--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap
subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the
dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine
getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers,
`benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path
filters.

**Fails loudly, not silently** — `--context-tool` / `--no-context-tool`
/ `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in
shell profiles, aliases and CI jobs, and accepting them as a no-op would
read as Headroom having quietly stopped working. The installers reject
them too, which matters more than it looks: their arg parsers forward
the first unknown flag **and everything after it** to the wrapped tool,
so a leftover `--no-rtk` would have silently swallowed a following
`--port` and then been ignored downstream.

**New `headroom/context_tool_cleanup.py`** — deleting the code cannot
help a machine that already ran the old default, since the hooks,
binaries and injected guidance are durable on disk.
`purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and
removes the registered hook entries, the generated hook scripts, the
Headroom-managed `~/.local/bin` symlinks, the vendored
`~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server
entry and the marker-fenced instruction blocks. Deliberately
conservative: idempotent, **skips** a malformed config rather than
overwriting it, and only unlinks a symlink resolving inside Headroom's
own bin dir so a user's own build is untouched. It reports on
**stderr**, because `wrap/unwrap openclaw --prepare-only` emit
machine-readable JSON on stdout as their entire contract. Skipped for
`wrap selfheal` (runs from a SessionStart hook; must not race Claude
Code's writer for `~/.claude.json`) and for `--help`, which must stay
read-only.

**Client-config hardening** (discovered while investigating a "corrupted
Serena settings file" report) — `wrap.py` reset a settings file to `{}`
when an existing file would not parse, then wrote that back. One
hand-edited typo or a transient `EACCES`/`EINTR` on a valid file
destroyed the user's `permissions`, `env` and `hooks`, on **every
`headroom wrap claude`**. It now refuses to write. Separately,
`fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`),
fixing all 14 non-atomic client-config writes at once; it follows
symlinks rather than replacing them (dotfile managers) and preserves an
existing file's mode.

**Deliberately kept** — `rtk` stays in the wrapper-peel list in
`transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as
shell-command grammar, so `rtk cat f` is still classified as a file read
for anyone running their own rtk install, which the purge intentionally
leaves alone.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
All checks passed!

$ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
1255 files already formatted

$ mypy headroom/
Success: no issues found in 508 source files

$ pytest tests/test_context_tool_cleanup.py -q
11 passed

$ pytest tests/test_fsutil.py -q
12 passed

$ pytest tests/test_cli/test_wrap_codex.py -q            # 89 tests
89 passed in 431.68s
$ pytest tests/test_cli/test_wrap_opencode.py -q
39 passed in 257.46s
$ pytest tests/test_cli/test_wrap_helpers.py -q
45 passed
$ pytest tests/test_paths.py -q
75 passed
$ pytest tests/test_cli/test_unwrap_claude.py -q
14 passed
$ pytest tests/test_proxy_savings_history.py -q
39 passed
$ pytest tests/test_cli/test_wrap_copilot.py -q
27 passed
$ pytest tests/test_cli/test_wrap_zcode.py -q
20 passed
$ pytest tests/test_subscription_tracker.py -q
9 passed
$ pytest tests/test_proxy_dashboard_stats_cache.py -q
5 passed, 1 skipped
```

Repo-wide grep for 14 removed symbols (`headroom.rtk`,
`headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`,
`_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`,
`wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`,
`tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`,
`*.html`: **zero hits**.

Notable test changes: `test_wrap_openhands.py` no longer patches
`_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0
unpatched — the regression that was previously masked.
`test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed
(every test drove RTK instruction injection). A new
`test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed`
proves a pre-removal `subscription_state.json` still loads.

## Real Behavior Proof

- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @
this branch, real `~/.headroom` and `~/.claude` on the dev machine.
- **Exact command / steps and observed result:**

```text
# 1. Retired flag fails loudly instead of silently no-op'ing
$ headroom wrap codex --prepare-only --context-tool rtk
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they
rewrote shell commands through a third-party binary Headroom no longer manages.
Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL;
`headroom wrap` uninstalls what they left behind on first run.

$ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ...

# 2. install.sh rejects the retired flags (extracted parse_wrap_args harness)
['--no-rtk', '--port', '9999']   rc=1  ERROR: CLI context tools ... Drop --no-rtk
['--context-tool=rtk']           rc=1  ERROR: CLI context tools ... Drop --context-tool
$ bash -n scripts/install.sh   # syntax OK

# 3. Purge ran against the real machine, which had all the orphaned artifacts
$ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..."
  removed ~/.headroom/bin/lean-ctx        (51 MB)
  removed ~/.headroom/bin/rtk             (7.7 MB)
  removed ~/.local/bin/rtk                (symlink into ~/.headroom/bin)
  removed ~/.claude/hooks/rtk-rewrite.sh
  removed 8 lean-ctx-* hook scripts
# ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged
# → ~59 MB reclaimed, no unrelated key touched

# 4. stdout stays machine-readable while the purge reports (planted a fake artifact)
$ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err
$ cat out
{"enabled":true,"config":{"proxyPort":8787,...}}     # parses as JSON
$ cat err
Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk

# 5. --help is inert (planted artifact survives), a real run purges
$ headroom wrap codex --help   → artifact survived: CORRECT
$ headroom wrap openclaw --prepare-only → purged: CORRECT

# 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json
top-level keys 90 -> 90;  projects 19 -> 19;  LOST keys: none
all content outside mcpServers byte-identical: True
```

Dashboard rendered via the Playwright test after the panel removal:
"Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`,
and "Token Usage" reads Before Compression → Proxy Removed → After
Compression with no "Filtered (this session)" row. Nothing below the
removed panel broke.

- **Not tested:** Windows and Linux (macOS only) — `install.ps1` is
verified by brace-balance and inspection, not executed, since no `pwsh`
is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated
but not run; it needs the Docker e2e image. `serena project index`
interaction is exercised in the stacked base PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

**Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge
that first; this PR's base should then be retargeted to `main`, or it
will read as containing that fix too.

**Breaking-change migration for users:**
- Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`,
`--context-tool`, `--no-context-tool` from any alias, script or CI job,
and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error
rather than being ignored, so the failure is immediate and
self-explaining.
- Previously-installed artifacts are purged automatically on the next
`wrap`/`unwrap`; no manual cleanup needed.
- `headroom perf --json` no longer carries a `cli_filtering` key, and
`/stats` no longer returns a `context_tool` section.

**Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from
`README.md`,
`docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`,
`docs/observability.md` and the matching `wiki/` pages.
`REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED
rather than deleted, to keep the planning record.

**Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as
dead code — its only caller was the `except KeyboardInterrupt` guarding
the binary download, so with no download there is nothing slow left to
interrupt.
2026-07-30 22:59:41 -07:00
JervisandJerrettDavis 0537cbfde4 feat(dashboard): persist lifetime proxy metrics (#2198)
## Description

Persist bounded, aggregate-only Lifetime dashboard metrics across proxy
restarts and expose them through a new `/stats-lifetime` endpoint. The
change keeps session/runtime stats separate from durable lifetime stats,
gates sensitive dashboard metadata for loopback or explicitly trusted
dashboard clients, and updates the dashboard Lifetime view to consume
the new endpoint.

Closes #2137

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Added bounded persistent lifetime metrics state and wired proxy metric
events into it.
- Added `/stats-lifetime` with sensitive project/persistence details
gated behind dashboard metadata access checks.
- Extended loopback/dashboard metadata access policy for trusted
dashboard client CIDRs without widening admin/debug endpoints.
- Reorganized dashboard session/lifetime presentation around runtime
counters versus durable aggregates.
- Added focused tests for persistent aggregation, persistence, endpoint
registration, loopback gating, trusted dashboard CIDRs, and recent
request ordering.
- Fixed current Ruff/mypy issues in the lifetime metrics normalization
code.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q
53 passed, 1 warning

uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py
All checks passed!

uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows review worktree, Python 3.13.3 via uv.
- Exact command / steps: Ran the focused persistent metrics,
persistence, loopback gating, and recent request tests; ran CI-matching
Ruff on touched files; ran mypy on the new persistent metrics module.
- Observed result: `/stats-lifetime` is registered, non-loopback callers
receive only non-sensitive aggregate data, loopback/trusted dashboard
clients receive the full lifetime payload, admin/debug endpoints remain
loopback-only, and persistent metrics normalize malformed stored state
without type/lint errors.
- Not tested: Full repository pytest suite, full dashboard browser
screenshot pass, or live long-running proxy traffic.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

No changelog entry is required for this dashboard/internal metrics
iteration. The endpoint intentionally exposes only aggregate lifetime
data to ordinary network callers and strips project/persistence details
unless the caller passes the dashboard metadata access policy.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 18:18:13 +00:00
Vinay Gupta bd8de9f382 fix(proxy): keep recent stats request rows (#1922)
## Description

`/stats.recent_requests` was built from the request-log tail, but then
filtered out any row whose compact token fields were incomplete. That
made the dashboard/API summary diverge from the raw request counters:
`requests.by_model` could show many recent requests for a model while
`recent_requests` only showed the few rows with complete
compression-token accounting.

This fixes the stats payload so the compact `recent_requests` table
mirrors the latest request-log rows without masking unknown token
accounting as measured zero. Missing/non-finite compact numeric fields
now remain `null`, and each row exposes `token_accounting_status` plus
`has_exact_tokens` so API clients and the dashboard can distinguish
complete, partial, and missing token accounting.

Closes #1914

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Removed the token-completeness filter from
`_build_recent_request_payload()`.
- Preserved unknown compact recent-request numeric fields as `null`.
- Added `token_accounting_status` and `has_exact_tokens` to compact
recent request rows.
- Updated the dashboard recent-requests table to render unknown
token/latency fields as `unknown`.
- Hardened `build_session_summary()` against token-incomplete
request-log rows and surfaced an `unknown_token_accounting` bucket.
- Added TypeScript SDK fields for the compact recent-request stats
contract.
- Added a regression test covering missing, partial, and complete
token-accounting rows.

## Testing

- [x] Focused stats regression passes
- [x] Adjacent summary/MCP tests pass
- [x] TypeScript SDK typecheck/tests pass
- [x] Ruff check passes
- [x] Ruff format check passes
- [x] Diff whitespace check passes
- [ ] Full suite not run

### Test Output

```text
$ uv run --no-project ... pytest tests/test_proxy_stats_recent_requests.py -q
4 passed, 1 warning

$ uv run --no-project ... pytest tests/test_proxy_dashboard_stats_cache.py::test_session_summary_uses_generic_cli_filtering_keys tests/test_proxy_dashboard_stats_cache.py::test_session_summary_surfaces_codex_ws_counters tests/test_ccr_mcp_server.py -q
19 passed, 1 skipped

$ npm run typecheck
tsc --noEmit

$ npm test
296 passed, 33 skipped

$ uv run --no-project --with ruff ruff check headroom/proxy/server.py headroom/proxy/cost.py headroom/ccr/mcp_server.py tests/test_proxy_stats_recent_requests.py
All checks passed!

$ uv run --no-project --with ruff ruff format --check headroom/proxy/server.py headroom/proxy/cost.py headroom/ccr/mcp_server.py tests/test_proxy_stats_recent_requests.py
4 files already formatted

$ git diff --check
clean
```

## Verification

- Principal engineer review: no blockers after the final
finite-number/accounting-status alignment.
- Senior developer review: no blockers; implementation and focused
coverage look solid for #1914.
- Architect/design review: original API/UI contract blocker resolved;
unknowns remain visible as `null`/`unknown` instead of measured zero.

## Notes

The normal editable `uv run pytest ...` path is blocked locally by the
known native build issue in `esaxx-rs` (`fatal error: 'cstdint' file not
found`) while building the Rust extension on this machine. I validated
the Python-only stats path with a temporary `headroom._core` import stub
and `HEADROOM_REQUIRE_RUST_CORE=false`; no repository files were changed
for that stub.

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review
2026-07-13 10:09:51 -04:00
Tejas Chopra bd55a426bc fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226)
## Description

Locks down the proxy's browser- and network-facing attack surface, which
matters most under a `--host 0.0.0.0` bind (the Docker default). The
wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`)
let any web page the user had open read the proxy's content endpoints —
`/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets)
— via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator
endpoints additionally leaked sensitive data or allowed unauthenticated
state mutation to any network-reachable client.

This PR scopes CORS to loopback origins and extends the project's
existing `require_loopback` trust boundary (already used for `/admin/*`
and `/debug/*`) to the remaining exposed endpoints.

Closes #863.

Supersedes #864 and #758 — see "Additional Notes".

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True`
with a port-agnostic loopback origin regex
(`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`),
`allow_credentials=False`, and methods/headers narrowed to `GET/POST` +
`Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated)
pins an explicit allowlist for Docker/remote dashboards; `*` opts back
into the old wildcard.
- **`/transformations/feed`** and **`/cache/clear`** gated behind
`require_loopback` → 404 for non-loopback callers. The feed returns full
prompt/completion bodies when `log_full_messages` is on; `/cache/clear`
is unauthenticated state mutation (cache-eviction DoS / cost
amplification).
- **`/health`**: the `config` block (upstream API URLs, savings profile)
is now served only to loopback callers; network callers get the
`/readyz`-shape body (status/checks). `/livez` and `/readyz` remain
unauthenticated probes for orchestration.
- **`/stats`**: `recent_requests` / `request_logs` (per-request ids,
providers, models, errors) and `config` are served only to loopback
callers; aggregate counters stay public for remote monitoring.
- Added `_request_is_loopback()` helper mirroring `require_loopback`'s
two-gate check (loopback peer IP + loopback `Host` header, the
DNS-rebinding defence) but degrading the payload instead of returning
404, so monitors keep the non-sensitive fields.
- Tests: new `tests/test_proxy_cors.py` and
`tests/test_proxy_loopback_gating.py`; updated 4 existing tests that
assert the now-loopback-only data to use loopback clients.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \
    tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \
    tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py
All checks passed!

$ mypy headroom --ignore-missing-imports
Success: no issues found in 380 source files

$ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \
    tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \
    tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \
    tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q
======================== 82 passed, 1 skipped in 17.75s ========================
```

## Real Behavior Proof

- Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI
`TestClient` driving the real `create_app()` ASGI app
- Exact command / steps: issued requests as a non-loopback caller
(`client.host=testclient`) vs a loopback caller
(`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS
preflights with varying `Origin` headers
- Observed result: CORS — `http://evil.com` → no
`access-control-allow-origin`; `http://localhost:8787` and
`http://localhost:9000` → echoed (loopback allowed on any port);
`access-control-allow-credentials` → absent. `/cache/clear` and
`/transformations/feed` → 404 (network) / 200 (loopback). `/health`
`config` block present for loopback only. `/stats` `recent_requests`
present for loopback only, while the aggregate `tokens` block stays
present for network callers.
- Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached
from a second host (simulated via ASGI peer/Host instead); end-to-end
browser DNS-rebinding (covered by the `Host`-header gate and its unit
test)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — proxy/middleware change; behavior is captured under "Real Behavior
Proof".

## Additional Notes

**Supersedes two stale PRs that target the same issue but have drifted
from `main`:**

- **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) —
correct instinct and the source of the tighter `GET/POST` +
`Content-Type/Authorization` scoping kept here, but it derived the
allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is
passed as a CLI flag), carried ~40 lines of unrelated punctuation churn,
and is ~125 commits behind `main`. The port-agnostic regex used here
resolves the reviewer's port concern.
- **#758** (`security: adversarial review`, @neogenix) — bundled these
same application-layer fixes with a large CI/CD + Docker supply-chain
pass. It is a ~160-commit-behind draft whose `server.py` no longer
merges cleanly (`main` independently adopted the same `require_loopback`
pattern). The application-layer fixes are rebased onto current `main`
here; the CI/Docker/supply-chain hardening from #758 is still valuable
and would be welcome as a separate, rebased PR.

Thanks to @gabiudrescu and @neogenix for the original analysis (#863).

**Deliberate scope / follow-ups (not in this PR):**
- `/stats` aggregate counters and the basic `/health` body remain
readable on a `0.0.0.0` bind by design, so remote monitoring keeps
working. Full lock-down is a one-line `Depends(require_loopback)` each
if preferred.
- The `/v1/retrieve*` family stays network-reachable; it can't be
loopback-gated without breaking legitimate remote/containerized agents
and needs auth instead — tracked separately.
- `ruff check .` is scoped to changed paths above because the dashboard
HTML template trips ruff's `invalid-syntax` (a known repo
false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
Michael Sam 6d3f39f213 feat: add dashboard agent usage stats (#814)
## Description

Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.

Fixes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### New Files

**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.

### Modified Files

- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.

## Testing

- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
2026-06-12 14:12:22 -05:00