Files
headroom/.dockerignore
Yasser SheikhandJerrettDavis 70b21c12c4 feat(proxy): native spend/savings stats + dashboard for the Rust proxy (#2321)
# feat(proxy): native spend/savings stats + dashboard for the Rust proxy

## Description

The Rust proxy has no cost observability of its own today: `/stats` and
`/dashboard` only exist on the Python proxy, so anyone running
`headroom-proxy` standalone (the Bedrock/Vertex lanes especially) is
flying blind on spend.

This adds it natively. Every lane the Rust proxy forwards — Anthropic
`/v1/messages`, OpenAI Chat + Responses, all four native Bedrock routes,
Vertex `rawPredict`/`streamRawPredict` — records per-request cost/usage
telemetry into one in-process ledger, served from `GET /stats`, `GET
/stats/timeseries`, `GET /stats/events`, and an embedded,
dependency-free `GET /dashboard`.

This is a ground-up redesign of #1057 (approved but never merged, now
badly stale against `main`), not a rebase of it. Supersedes #1057. The
"Design" section below states exactly what was kept and what was
deliberately dropped, so it can be reviewed on its own terms.

### What it measures today

Spend, input/output/cache-read/cache-write tokens, prompt-cache hit rate
and its NET dollar benefit, per-provider and per-model breakdowns,
failures, latency, and hourly/daily history that survives restarts.

Two numbers are computed carefully rather than the obvious way, because
the obvious way flatters the product:

- **Spend counts output.** Input and output are priced separately
(`input_cost_usd` / `output_cost_usd`, summed into `total_cost_usd`)
because output bills at its own rate — typically 4-5x input. Counting
input alone would understate a short-prompt/long-answer turn by most of
its cost.
- **Cache savings are NET**, not the gross read discount: the read
discount minus the cache-write premium (writes bill *above* list price),
floored at $0 per request. Reporting gross reads overstates the benefit
on write-heavy traffic.

### What reads 0 today — stated up front

`tokens_saved` / `compression_savings_usd` will read **0 on real
traffic**, because the Rust live-zone per-type compressors are still
no-ops on `main` — there is nothing to record yet. Verified against the
real proxy: a 13.8 KB body with a large `tool_result`, sent with
`--compression --compression-mode live_zone`, produced `tokens_before:
0` (dispatcher returned `NoCompression`), which is the invariant
`live_zone_mode_with_valid_body_returns_no_compression_pr_b2` pins. The
recording path is wired and unit-tested against `Outcome::Compressed`,
and starts populating on its own the moment those compressors land — no
changes needed here. Stating it up front so `Saved $0` isn't read as a
bug.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Design — how this differs from #1057

- **One fold, many views.** A single `Totals::fold` is the only place
request accounting happens; lifetime, since-boot session, per-provider,
per-model, hourly and daily buckets are that same fold under different
keys, so the views cannot disagree. #1057 (like the Python tracker it
ported) maintained each aggregate separately.
- **Its own schema, not Python emulation.** `/stats` is a documented
Rust-native payload (`version: 1` on disk). No `schema_version: 5` file
compat, no `display_session` idle-window emulation, no `projects`
passthrough. The five keys external unified-stats consumers already read
from this endpoint (`summary.api_requests`, `requests.failed`,
`requests.cached`, `tokens.saved`, `tokens.proxy_compression_saved`) are
kept and pinned by a test.
- **No federation flag.** #1057's `--upstream-stats-url` supplemental
merge is dropped (YAGNI — the deployed consumer merges in the opposite
direction).
- **No new price data file.** USD valuation reuses the LiteLLM table
already vendored for `compression::model_limits`. #1057 shipped a
second, models.dev-shaped price book.
- **Purpose-built dashboard.** One self-contained HTML file designed for
what the Rust proxy actually measures, instead of embedding the Python
template and rendering its ~20 Python-only panels as zeros.
- **Not gated on compression.** `--compression` defaults to **off**, so
recording gated behind the compression buffer (as #1057 had it) records
nothing under default config. Here the model is sourced from the
*response* when the request body isn't buffered, so spend/cache
observability works on a pure passthrough deployment — with zero change
to the byte path.

## Correctness rules (carried from #1057's review, all test-pinned)

- **Deferred recording** — a request records only when its response is
fully observed, so streaming token counts come from real SSE/EventStream
usage frames instead of 0.
- **Failed upstreams accrue nothing** — counted in `requests`/`failed`
only; retries can't inflate savings.
- **No disk I/O on the request path** — `record()` folds under a mutex
and sets a dirty flag; a background task (and shutdown) writes an atomic
snapshot (temp → fsync → rename → fsync dir), re-arming the dirty flag
on failure.
- **Bounded everything** — per-model cardinality caps with `"other"`
overflow, length-clamped client strings, a 50M-token per-event ceiling,
bounded capture queues/buffers, bounded warn-once set.
- **Honest pricing** — exact-match only; a generic id misses (and WARNs)
rather than mispricing. Unknown ⇒ $0, never a blended fallback.
- **Bedrock regional pricing respected** — the unmodified id is tried
first, because `eu.anthropic.claude-3-5-haiku…` genuinely bills $0.25/M
against the base id's $0.80/M. Geo-stripping is only a fallback for
regions the table doesn't list.

## Pre-existing bugs fixed along the way

Both were found while wiring the recording hooks; both affect `main`
today, independently of stats.

- **Bedrock geo-prefix coverage**: `au.`, `jp.`, and `us-gov.`
cross-region inference profiles weren't recognized, so Anthropic models
invoked through them silently skipped compression — the same failure
mode #953 reported and #999 fixed for `eu.`/`us.`/`apac.`/`global.`. The
list is now complete (all seven published geos) and shared
(`bedrock::vendor::GEO_PREFIXES`) between the vendor gate and pricing,
so the two can't drift apart again.
- **EventStream `Skip` frame stranding**: the streaming parser pulled a
fresh upstream chunk after each unknown (`Skip`ped) event type, delaying
— and at stream end **dropping** — frames already buffered. It now
drains the buffer to exhaustion before awaiting more bytes.
Regression-pinned by
`skip_in_final_chunk_does_not_strand_buffered_frames`.

## Changes Made

- `crates/headroom-proxy/src/observability/ledger.rs` — the store:
`Totals` fold, bounded maps/buckets, atomic snapshot persistence,
payload builders
- `crates/headroom-proxy/src/observability/pricing.rs` — deterministic
lookup over the vendored LiteLLM table (no new data file)
- `crates/headroom-proxy/src/observability/capture.rs` — `PendingRecord`
(deferred finalization), usage extraction for every wire shape, bounded
tee tasks off the byte path
- `crates/headroom-proxy/src/observability/stats_http.rs` +
`dashboard.html` — HTTP surface + embedded self-contained UI
- `crates/headroom-proxy/src/proxy.rs`, `bedrock/invoke.rs`,
`bedrock/invoke_streaming.rs`, `vertex/raw_predict.rs` — per-lane
recording hooks
- `crates/headroom-proxy/src/config.rs` — `--stats` (default on),
`--stats-path`
- `crates/headroom-proxy/src/bedrock/vendor.rs` — shared `GEO_PREFIXES`
(the geo-coverage fix above)
- `crates/headroom-proxy/src/sse/framing.rs` — 4 MiB accumulator cap
(telemetry-side; a stream with no event terminator can no longer grow
the buffer unboundedly)

`--stats=false` restores byte-for-byte pre-feature behaviour (paths
tunnel upstream, nothing recorded) — pinned by
`stats_disabled_falls_through_to_upstream`.

## Testing

- [x] `cargo fmt --all -- --check` passes
- [x] `cargo clippy --workspace -- -D warnings` passes (0 warnings)
- [x] `cargo test --workspace` passes
- [x] New tests added for new functionality (53 new tests across unit +
integration)
- [x] Manual testing performed (live AWS Bedrock, all four native lanes
— see Real Behavior Proof)

Unit: fold-consistency across views, failed-no-savings, pricing
(regional/revision/path forms, collision-miss, untracked-region
fallback), USD math, bounded maps, token-ceiling clamp,
bidi/control-char stripping, atomic flush with no temp litter,
corrupt-file preservation, oversized-state rejection, negative-USD
clamping on load, restart round-trip, week/month rollups, hostile field
lengths. Integration (wiremock): endpoint contracts, `--stats=false`
tunneling, deferred SSE capture, non-streaming capture, OpenAI
cached-token normalisation, failed-upstream accounting, hostile-usage
clamping, all four Bedrock lanes, Vertex, dashboard self-containment.

The persisted-state tests are deliberately adversarial rather than
round-trip only, since the ledger reads a file an operator can
hand-edit: malformed bucket keys (including multi-byte ones that would
panic a naive `&str` slice) are dropped before the age-trim runs,
hostile model-id keys are re-clamped on load, and an implausibly large
state file is treated as corrupt instead of allocated.

### Test Output

```text
$ cargo fmt --all -- --check
  (clean)

$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
  exit=0   0 warnings, 0 errors

$ cargo nextest run -p headroom-proxy
     Summary [4.396s] 488 tests run: 488 passed, 0 skipped

$ cargo nextest run -p headroom-core -p headroom-simulators -p headroom-parity
     Summary [49.535s] 967 tests run: 967 passed, 1 skipped

$ cargo clippy --workspace -- -D warnings
  exit=0   0 warnings, 0 errors

$ cargo test --workspace --doc
  ok, 0 failed
```

## Real Behavior Proof

- Environment: live AWS Bedrock, staging account (SSO profile — name
redacted), `eu-central-1`, `anthropic.claude-3-haiku-20240307-v1:0`,
release binary, compression off (shipped default)
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:8790
--enable-bedrock-native --bedrock-region eu-central-1` then one request
per lane (`/invoke`, `/converse`, `/invoke-with-response-stream` with
`Accept: application/vnd.amazon.eventstream`, `/converse-stream`), one
deliberate failure (nonexistent model), `SIGTERM`, restart, re-read
`/stats` — full transcript, raw proxy logs, and the persisted ledger are
attached below
- Observed result: all four lanes captured usage (input tokens = 48 = 4
× 12; spend $0.000012 = 48 × $0.25/M, exact for Bedrock Haiku 3); the
failure recorded `failed: true` with 0 tokens and $0; lifetime totals
survived the restart while session totals reset; dashboard rendered
against this live data (screenshots attached)
- Not tested: live Vertex (covered by wiremock integration tests only —
no GCP staging account available); live OpenAI Chat/Responses lanes
(same wiremock coverage); dashboard on Windows browsers

The four Bedrock lanes are the ones worth proving live, because two of
them had no parsing before this PR:

| Lane | Wire shape | Captured |
|---|---|---|
| `/invoke` | sync, snake_case `usage` | 12 in / 4 out |
| `/converse` | sync, **camelCase** `usage` | 12 in / 4 out |
| `/invoke-with-response-stream` | **binary EventStream passthrough**,
base64-wrapped chunks | 12 in / 4 out |
| `/converse-stream` | SSE-translated; usage lives in the camelCase
`metadata` frame the Anthropic state machine drops | 12 in / 4 out |

```text
requests      : 5 (failed: 1)
input tokens  : 48
output tokens : 16
cache read/wr : 0 / 0
input spend   : $0.00001200      # 48 × $0.25/M
output spend  : $0.00002000      # 16 × $1.25/M — output bills 5x here
TOTAL spend   : $0.00003200      # what /dashboard reports as "Spend"
savings       : $0.00000000      # compression off -> 0 by design
avg latency   : 256 ms
by provider   : {'bedrock': 5}
by model      : {'anthropic.claude-3-haiku-20240307-v1:0': 4,
                 'anthropic.claude-nonexistent-v99:0': 1}
persistence   : True | healthy: True
```

Those two spend lines are the reason output pricing is in this PR:
**output was 62.5% of the real bill** on this run. Reporting input alone
would have called a $0.000032 spend `$0.000012`.

The exposure split was exercised against the same live proxy:

```text
$ curl /stats/events  -H 'Host: 127.0.0.1:18787'   -> 200   (local tooling works)
$ curl /stats/events  -H 'Host: attacker.com'      -> 404   (DNS-rebound: peer IS loopback)
$ curl /stats         -H 'Host: attacker.com'      -> recent_requests: null
                                                      persistence.path: null
                                                      requests.total still served
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(RUST_DEV.md, CHANGELOG.md)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Known limitations

1. Compression savings read 0 until the live-zone compressors land
(above).
2. Hourly resolution is 48 h, daily ~13 months; older traffic folds into
lifetime totals only. `/stats/events` is an in-memory ring of 100 and is
not persisted — this is a cost ledger, not an audit log.
3. The vendored price table lacks some legacy direct-API ids
(`claude-3-5-sonnet-20241022` is absent while its Bedrock form is
present); those record $0 with a WARN until refreshed via
`scripts/refresh_model_limits.sh`.
4. `output_cost_usd` is absent from state files written before this
change, so it loads as 0: historical spend keeps reading input-only
until those buckets age out (48 h hourly / ~400 d daily). New traffic is
priced in full immediately. Deliberately no migration shim — the field
defaults cleanly and the alternative is retro-pricing history against
today's price table, which would invent numbers.
5. A non-streaming response body over 2 MiB isn't buffered for usage
parsing (the request still records, compression-side numbers only).
Streaming responses have no such cap — they parse incrementally.
6. The endpoints carry no auth, so exposure is split by tier instead
(matching the Python proxy): aggregates are public like `/metrics`,
while per-request rows (`recent_requests`, `/stats/events`) and the
ledger's filesystem path require a local caller — loopback peer **and**
loopback `Host`, the latter being the DNS-rebinding defence. Remote
viewers of `/dashboard` therefore see totals but not the request feed,
and the page says so. Opening that tier to a trusted reverse proxy is
#1959's call to make for both proxies; this PR doesn't invent a second
scheme.
7. Copilot traffic isn't proxied by the Rust binary; it reaches this
dashboard only via external unified-stats layers (unchanged).

## Screenshots (if applicable)

Attached below: live-test terminal transcript, dashboard rendered
against the live Bedrock data (session + lifetime scopes, before/after
restart), raw proxy logs, and the persisted ledger file.

## Additional Notes

- Related issues/PRs: supersedes #1057; completes the geo-prefix
coverage from #999 (design discussion in #953 — the multi-vendor adapter
work proposed there remains out of scope).
- `requests.cached` counts requests whose response reported
`cache_read_tokens > 0`, matching the Python tracker's derivation, so
unified-stats consumers see consistent semantics across both proxies.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2321"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-07 20:58:22 -05:00

85 lines
1.2 KiB
Plaintext

# VCS
.git/*
!.git/HEAD
!.git/packed-refs
!.git/refs/
!.git/refs/**
.github
.github/*
!.github/plugin/
!.github/plugin/**
.gitignore
# Python artifacts
__pycache__
*.pyc
*.pyo
*.egg-info
dist/
build/
# Rust artifacts. The Dockerfile runs its own `cargo build` inside the
# builder stage, so the host's target/ is never needed — and after a
# local `cargo build` it is 100GB+, which the daemon dutifully copies
# on every `docker build`. CI never noticed because a fresh checkout
# has no target/; it only ever bites local builds.
target/
# Dev/test tooling
.pytest_cache
.coverage
.mypy_cache
.ruff_cache
.pre-commit-config.yaml
.venv
venv
# Tests & docs (not needed in image)
tests/
docs/
mkdocs.yml
CHANGELOG.md
LICENSE
NOTICE
# JS/TS artifacts (dashboard, SDK — not part of proxy image)
apps/
sdk/
!sdk/
!sdk/typescript/
!sdk/typescript/**
plugins/
!plugins/
!plugins/openclaw/
!plugins/openclaw/**
!plugins/headroom-agent-hooks/
!plugins/headroom-agent-hooks/**
node_modules/
*.tgz
# Secrets & local config
.env
.env.*
*.log
# IDE
.vscode
.idea
*.swp
# Docker
Dockerfile
docker-compose*.yml
.dockerignore
# Misc
.pi-lens/
.superpowers/
examples/
node-compile-cache/
!e2e/
!e2e/init/
!e2e/init/**
!.claude-plugin/
!.claude-plugin/**