mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
# feat(proxy): native spend/savings stats + dashboard for the Rust proxy ## Description The Rust proxy has no cost observability of its own today: `/stats` and `/dashboard` only exist on the Python proxy, so anyone running `headroom-proxy` standalone (the Bedrock/Vertex lanes especially) is flying blind on spend. This adds it natively. Every lane the Rust proxy forwards — Anthropic `/v1/messages`, OpenAI Chat + Responses, all four native Bedrock routes, Vertex `rawPredict`/`streamRawPredict` — records per-request cost/usage telemetry into one in-process ledger, served from `GET /stats`, `GET /stats/timeseries`, `GET /stats/events`, and an embedded, dependency-free `GET /dashboard`. This is a ground-up redesign of #1057 (approved but never merged, now badly stale against `main`), not a rebase of it. Supersedes #1057. The "Design" section below states exactly what was kept and what was deliberately dropped, so it can be reviewed on its own terms. ### What it measures today Spend, input/output/cache-read/cache-write tokens, prompt-cache hit rate and its NET dollar benefit, per-provider and per-model breakdowns, failures, latency, and hourly/daily history that survives restarts. Two numbers are computed carefully rather than the obvious way, because the obvious way flatters the product: - **Spend counts output.** Input and output are priced separately (`input_cost_usd` / `output_cost_usd`, summed into `total_cost_usd`) because output bills at its own rate — typically 4-5x input. Counting input alone would understate a short-prompt/long-answer turn by most of its cost. - **Cache savings are NET**, not the gross read discount: the read discount minus the cache-write premium (writes bill *above* list price), floored at $0 per request. Reporting gross reads overstates the benefit on write-heavy traffic. ### What reads 0 today — stated up front `tokens_saved` / `compression_savings_usd` will read **0 on real traffic**, because the Rust live-zone per-type compressors are still no-ops on `main` — there is nothing to record yet. Verified against the real proxy: a 13.8 KB body with a large `tool_result`, sent with `--compression --compression-mode live_zone`, produced `tokens_before: 0` (dispatcher returned `NoCompression`), which is the invariant `live_zone_mode_with_valid_body_returns_no_compression_pr_b2` pins. The recording path is wired and unit-tested against `Outcome::Compressed`, and starts populating on its own the moment those compressors land — no changes needed here. Stating it up front so `Saved $0` isn't read as a bug. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Design — how this differs from #1057 - **One fold, many views.** A single `Totals::fold` is the only place request accounting happens; lifetime, since-boot session, per-provider, per-model, hourly and daily buckets are that same fold under different keys, so the views cannot disagree. #1057 (like the Python tracker it ported) maintained each aggregate separately. - **Its own schema, not Python emulation.** `/stats` is a documented Rust-native payload (`version: 1` on disk). No `schema_version: 5` file compat, no `display_session` idle-window emulation, no `projects` passthrough. The five keys external unified-stats consumers already read from this endpoint (`summary.api_requests`, `requests.failed`, `requests.cached`, `tokens.saved`, `tokens.proxy_compression_saved`) are kept and pinned by a test. - **No federation flag.** #1057's `--upstream-stats-url` supplemental merge is dropped (YAGNI — the deployed consumer merges in the opposite direction). - **No new price data file.** USD valuation reuses the LiteLLM table already vendored for `compression::model_limits`. #1057 shipped a second, models.dev-shaped price book. - **Purpose-built dashboard.** One self-contained HTML file designed for what the Rust proxy actually measures, instead of embedding the Python template and rendering its ~20 Python-only panels as zeros. - **Not gated on compression.** `--compression` defaults to **off**, so recording gated behind the compression buffer (as #1057 had it) records nothing under default config. Here the model is sourced from the *response* when the request body isn't buffered, so spend/cache observability works on a pure passthrough deployment — with zero change to the byte path. ## Correctness rules (carried from #1057's review, all test-pinned) - **Deferred recording** — a request records only when its response is fully observed, so streaming token counts come from real SSE/EventStream usage frames instead of 0. - **Failed upstreams accrue nothing** — counted in `requests`/`failed` only; retries can't inflate savings. - **No disk I/O on the request path** — `record()` folds under a mutex and sets a dirty flag; a background task (and shutdown) writes an atomic snapshot (temp → fsync → rename → fsync dir), re-arming the dirty flag on failure. - **Bounded everything** — per-model cardinality caps with `"other"` overflow, length-clamped client strings, a 50M-token per-event ceiling, bounded capture queues/buffers, bounded warn-once set. - **Honest pricing** — exact-match only; a generic id misses (and WARNs) rather than mispricing. Unknown ⇒ $0, never a blended fallback. - **Bedrock regional pricing respected** — the unmodified id is tried first, because `eu.anthropic.claude-3-5-haiku…` genuinely bills $0.25/M against the base id's $0.80/M. Geo-stripping is only a fallback for regions the table doesn't list. ## Pre-existing bugs fixed along the way Both were found while wiring the recording hooks; both affect `main` today, independently of stats. - **Bedrock geo-prefix coverage**: `au.`, `jp.`, and `us-gov.` cross-region inference profiles weren't recognized, so Anthropic models invoked through them silently skipped compression — the same failure mode #953 reported and #999 fixed for `eu.`/`us.`/`apac.`/`global.`. The list is now complete (all seven published geos) and shared (`bedrock::vendor::GEO_PREFIXES`) between the vendor gate and pricing, so the two can't drift apart again. - **EventStream `Skip` frame stranding**: the streaming parser pulled a fresh upstream chunk after each unknown (`Skip`ped) event type, delaying — and at stream end **dropping** — frames already buffered. It now drains the buffer to exhaustion before awaiting more bytes. Regression-pinned by `skip_in_final_chunk_does_not_strand_buffered_frames`. ## Changes Made - `crates/headroom-proxy/src/observability/ledger.rs` — the store: `Totals` fold, bounded maps/buckets, atomic snapshot persistence, payload builders - `crates/headroom-proxy/src/observability/pricing.rs` — deterministic lookup over the vendored LiteLLM table (no new data file) - `crates/headroom-proxy/src/observability/capture.rs` — `PendingRecord` (deferred finalization), usage extraction for every wire shape, bounded tee tasks off the byte path - `crates/headroom-proxy/src/observability/stats_http.rs` + `dashboard.html` — HTTP surface + embedded self-contained UI - `crates/headroom-proxy/src/proxy.rs`, `bedrock/invoke.rs`, `bedrock/invoke_streaming.rs`, `vertex/raw_predict.rs` — per-lane recording hooks - `crates/headroom-proxy/src/config.rs` — `--stats` (default on), `--stats-path` - `crates/headroom-proxy/src/bedrock/vendor.rs` — shared `GEO_PREFIXES` (the geo-coverage fix above) - `crates/headroom-proxy/src/sse/framing.rs` — 4 MiB accumulator cap (telemetry-side; a stream with no event terminator can no longer grow the buffer unboundedly) `--stats=false` restores byte-for-byte pre-feature behaviour (paths tunnel upstream, nothing recorded) — pinned by `stats_disabled_falls_through_to_upstream`. ## Testing - [x] `cargo fmt --all -- --check` passes - [x] `cargo clippy --workspace -- -D warnings` passes (0 warnings) - [x] `cargo test --workspace` passes - [x] New tests added for new functionality (53 new tests across unit + integration) - [x] Manual testing performed (live AWS Bedrock, all four native lanes — see Real Behavior Proof) Unit: fold-consistency across views, failed-no-savings, pricing (regional/revision/path forms, collision-miss, untracked-region fallback), USD math, bounded maps, token-ceiling clamp, bidi/control-char stripping, atomic flush with no temp litter, corrupt-file preservation, oversized-state rejection, negative-USD clamping on load, restart round-trip, week/month rollups, hostile field lengths. Integration (wiremock): endpoint contracts, `--stats=false` tunneling, deferred SSE capture, non-streaming capture, OpenAI cached-token normalisation, failed-upstream accounting, hostile-usage clamping, all four Bedrock lanes, Vertex, dashboard self-containment. The persisted-state tests are deliberately adversarial rather than round-trip only, since the ledger reads a file an operator can hand-edit: malformed bucket keys (including multi-byte ones that would panic a naive `&str` slice) are dropped before the age-trim runs, hostile model-id keys are re-clamped on load, and an implausibly large state file is treated as corrupt instead of allocated. ### Test Output ```text $ cargo fmt --all -- --check (clean) $ cargo clippy -p headroom-proxy --all-targets -- -D warnings exit=0 0 warnings, 0 errors $ cargo nextest run -p headroom-proxy Summary [4.396s] 488 tests run: 488 passed, 0 skipped $ cargo nextest run -p headroom-core -p headroom-simulators -p headroom-parity Summary [49.535s] 967 tests run: 967 passed, 1 skipped $ cargo clippy --workspace -- -D warnings exit=0 0 warnings, 0 errors $ cargo test --workspace --doc ok, 0 failed ``` ## Real Behavior Proof - Environment: live AWS Bedrock, staging account (SSO profile — name redacted), `eu-central-1`, `anthropic.claude-3-haiku-20240307-v1:0`, release binary, compression off (shipped default) - Exact command / steps: `headroom-proxy --listen 127.0.0.1:8790 --enable-bedrock-native --bedrock-region eu-central-1` then one request per lane (`/invoke`, `/converse`, `/invoke-with-response-stream` with `Accept: application/vnd.amazon.eventstream`, `/converse-stream`), one deliberate failure (nonexistent model), `SIGTERM`, restart, re-read `/stats` — full transcript, raw proxy logs, and the persisted ledger are attached below - Observed result: all four lanes captured usage (input tokens = 48 = 4 × 12; spend $0.000012 = 48 × $0.25/M, exact for Bedrock Haiku 3); the failure recorded `failed: true` with 0 tokens and $0; lifetime totals survived the restart while session totals reset; dashboard rendered against this live data (screenshots attached) - Not tested: live Vertex (covered by wiremock integration tests only — no GCP staging account available); live OpenAI Chat/Responses lanes (same wiremock coverage); dashboard on Windows browsers The four Bedrock lanes are the ones worth proving live, because two of them had no parsing before this PR: | Lane | Wire shape | Captured | |---|---|---| | `/invoke` | sync, snake_case `usage` | 12 in / 4 out | | `/converse` | sync, **camelCase** `usage` | 12 in / 4 out | | `/invoke-with-response-stream` | **binary EventStream passthrough**, base64-wrapped chunks | 12 in / 4 out | | `/converse-stream` | SSE-translated; usage lives in the camelCase `metadata` frame the Anthropic state machine drops | 12 in / 4 out | ```text requests : 5 (failed: 1) input tokens : 48 output tokens : 16 cache read/wr : 0 / 0 input spend : $0.00001200 # 48 × $0.25/M output spend : $0.00002000 # 16 × $1.25/M — output bills 5x here TOTAL spend : $0.00003200 # what /dashboard reports as "Spend" savings : $0.00000000 # compression off -> 0 by design avg latency : 256 ms by provider : {'bedrock': 5} by model : {'anthropic.claude-3-haiku-20240307-v1:0': 4, 'anthropic.claude-nonexistent-v99:0': 1} persistence : True | healthy: True ``` Those two spend lines are the reason output pricing is in this PR: **output was 62.5% of the real bill** on this run. Reporting input alone would have called a $0.000032 spend `$0.000012`. The exposure split was exercised against the same live proxy: ```text $ curl /stats/events -H 'Host: 127.0.0.1:18787' -> 200 (local tooling works) $ curl /stats/events -H 'Host: attacker.com' -> 404 (DNS-rebound: peer IS loopback) $ curl /stats -H 'Host: attacker.com' -> recent_requests: null persistence.path: null requests.total still served ``` ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (RUST_DEV.md, CHANGELOG.md) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Known limitations 1. Compression savings read 0 until the live-zone compressors land (above). 2. Hourly resolution is 48 h, daily ~13 months; older traffic folds into lifetime totals only. `/stats/events` is an in-memory ring of 100 and is not persisted — this is a cost ledger, not an audit log. 3. The vendored price table lacks some legacy direct-API ids (`claude-3-5-sonnet-20241022` is absent while its Bedrock form is present); those record $0 with a WARN until refreshed via `scripts/refresh_model_limits.sh`. 4. `output_cost_usd` is absent from state files written before this change, so it loads as 0: historical spend keeps reading input-only until those buckets age out (48 h hourly / ~400 d daily). New traffic is priced in full immediately. Deliberately no migration shim — the field defaults cleanly and the alternative is retro-pricing history against today's price table, which would invent numbers. 5. A non-streaming response body over 2 MiB isn't buffered for usage parsing (the request still records, compression-side numbers only). Streaming responses have no such cap — they parse incrementally. 6. The endpoints carry no auth, so exposure is split by tier instead (matching the Python proxy): aggregates are public like `/metrics`, while per-request rows (`recent_requests`, `/stats/events`) and the ledger's filesystem path require a local caller — loopback peer **and** loopback `Host`, the latter being the DNS-rebinding defence. Remote viewers of `/dashboard` therefore see totals but not the request feed, and the page says so. Opening that tier to a trusted reverse proxy is #1959's call to make for both proxies; this PR doesn't invent a second scheme. 7. Copilot traffic isn't proxied by the Rust binary; it reaches this dashboard only via external unified-stats layers (unchanged). ## Screenshots (if applicable) Attached below: live-test terminal transcript, dashboard rendered against the live Bedrock data (session + lifetime scopes, before/after restart), raw proxy logs, and the persisted ledger file. ## Additional Notes - Related issues/PRs: supersedes #1057; completes the geo-prefix coverage from #999 (design discussion in #953 — the multi-vendor adapter work proposed there remains out of scope). - `requests.cached` counts requests whose response reported `cache_read_tokens > 0`, matching the Python tracker's derivation, so unified-stats consumers see consistent semantics across both proxies. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2321" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
85 lines
1.2 KiB
Plaintext
85 lines
1.2 KiB
Plaintext
# VCS
|
|
.git/*
|
|
!.git/HEAD
|
|
!.git/packed-refs
|
|
!.git/refs/
|
|
!.git/refs/**
|
|
.github
|
|
.github/*
|
|
!.github/plugin/
|
|
!.github/plugin/**
|
|
.gitignore
|
|
|
|
# Python artifacts
|
|
__pycache__
|
|
*.pyc
|
|
*.pyo
|
|
*.egg-info
|
|
dist/
|
|
build/
|
|
|
|
# Rust artifacts. The Dockerfile runs its own `cargo build` inside the
|
|
# builder stage, so the host's target/ is never needed — and after a
|
|
# local `cargo build` it is 100GB+, which the daemon dutifully copies
|
|
# on every `docker build`. CI never noticed because a fresh checkout
|
|
# has no target/; it only ever bites local builds.
|
|
target/
|
|
|
|
# Dev/test tooling
|
|
.pytest_cache
|
|
.coverage
|
|
.mypy_cache
|
|
.ruff_cache
|
|
.pre-commit-config.yaml
|
|
.venv
|
|
venv
|
|
|
|
# Tests & docs (not needed in image)
|
|
tests/
|
|
docs/
|
|
mkdocs.yml
|
|
CHANGELOG.md
|
|
LICENSE
|
|
NOTICE
|
|
|
|
# JS/TS artifacts (dashboard, SDK — not part of proxy image)
|
|
apps/
|
|
sdk/
|
|
!sdk/
|
|
!sdk/typescript/
|
|
!sdk/typescript/**
|
|
plugins/
|
|
!plugins/
|
|
!plugins/openclaw/
|
|
!plugins/openclaw/**
|
|
!plugins/headroom-agent-hooks/
|
|
!plugins/headroom-agent-hooks/**
|
|
node_modules/
|
|
*.tgz
|
|
|
|
# Secrets & local config
|
|
.env
|
|
.env.*
|
|
*.log
|
|
|
|
# IDE
|
|
.vscode
|
|
.idea
|
|
*.swp
|
|
|
|
# Docker
|
|
Dockerfile
|
|
docker-compose*.yml
|
|
.dockerignore
|
|
|
|
# Misc
|
|
.pi-lens/
|
|
.superpowers/
|
|
examples/
|
|
node-compile-cache/
|
|
!e2e/
|
|
!e2e/init/
|
|
!e2e/init/**
|
|
!.claude-plugin/
|
|
!.claude-plugin/**
|