feat(proxy): native spend/savings stats + dashboard for the Rust proxy (#2321)

# feat(proxy): native spend/savings stats + dashboard for the Rust proxy

## Description

The Rust proxy has no cost observability of its own today: `/stats` and
`/dashboard` only exist on the Python proxy, so anyone running
`headroom-proxy` standalone (the Bedrock/Vertex lanes especially) is
flying blind on spend.

This adds it natively. Every lane the Rust proxy forwards — Anthropic
`/v1/messages`, OpenAI Chat + Responses, all four native Bedrock routes,
Vertex `rawPredict`/`streamRawPredict` — records per-request cost/usage
telemetry into one in-process ledger, served from `GET /stats`, `GET
/stats/timeseries`, `GET /stats/events`, and an embedded,
dependency-free `GET /dashboard`.

This is a ground-up redesign of #1057 (approved but never merged, now
badly stale against `main`), not a rebase of it. Supersedes #1057. The
"Design" section below states exactly what was kept and what was
deliberately dropped, so it can be reviewed on its own terms.

### What it measures today

Spend, input/output/cache-read/cache-write tokens, prompt-cache hit rate
and its NET dollar benefit, per-provider and per-model breakdowns,
failures, latency, and hourly/daily history that survives restarts.

Two numbers are computed carefully rather than the obvious way, because
the obvious way flatters the product:

- **Spend counts output.** Input and output are priced separately
(`input_cost_usd` / `output_cost_usd`, summed into `total_cost_usd`)
because output bills at its own rate — typically 4-5x input. Counting
input alone would understate a short-prompt/long-answer turn by most of
its cost.
- **Cache savings are NET**, not the gross read discount: the read
discount minus the cache-write premium (writes bill *above* list price),
floored at $0 per request. Reporting gross reads overstates the benefit
on write-heavy traffic.

### What reads 0 today — stated up front

`tokens_saved` / `compression_savings_usd` will read **0 on real
traffic**, because the Rust live-zone per-type compressors are still
no-ops on `main` — there is nothing to record yet. Verified against the
real proxy: a 13.8 KB body with a large `tool_result`, sent with
`--compression --compression-mode live_zone`, produced `tokens_before:
0` (dispatcher returned `NoCompression`), which is the invariant
`live_zone_mode_with_valid_body_returns_no_compression_pr_b2` pins. The
recording path is wired and unit-tested against `Outcome::Compressed`,
and starts populating on its own the moment those compressors land — no
changes needed here. Stating it up front so `Saved $0` isn't read as a
bug.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Design — how this differs from #1057

- **One fold, many views.** A single `Totals::fold` is the only place
request accounting happens; lifetime, since-boot session, per-provider,
per-model, hourly and daily buckets are that same fold under different
keys, so the views cannot disagree. #1057 (like the Python tracker it
ported) maintained each aggregate separately.
- **Its own schema, not Python emulation.** `/stats` is a documented
Rust-native payload (`version: 1` on disk). No `schema_version: 5` file
compat, no `display_session` idle-window emulation, no `projects`
passthrough. The five keys external unified-stats consumers already read
from this endpoint (`summary.api_requests`, `requests.failed`,
`requests.cached`, `tokens.saved`, `tokens.proxy_compression_saved`) are
kept and pinned by a test.
- **No federation flag.** #1057's `--upstream-stats-url` supplemental
merge is dropped (YAGNI — the deployed consumer merges in the opposite
direction).
- **No new price data file.** USD valuation reuses the LiteLLM table
already vendored for `compression::model_limits`. #1057 shipped a
second, models.dev-shaped price book.
- **Purpose-built dashboard.** One self-contained HTML file designed for
what the Rust proxy actually measures, instead of embedding the Python
template and rendering its ~20 Python-only panels as zeros.
- **Not gated on compression.** `--compression` defaults to **off**, so
recording gated behind the compression buffer (as #1057 had it) records
nothing under default config. Here the model is sourced from the
*response* when the request body isn't buffered, so spend/cache
observability works on a pure passthrough deployment — with zero change
to the byte path.

## Correctness rules (carried from #1057's review, all test-pinned)

- **Deferred recording** — a request records only when its response is
fully observed, so streaming token counts come from real SSE/EventStream
usage frames instead of 0.
- **Failed upstreams accrue nothing** — counted in `requests`/`failed`
only; retries can't inflate savings.
- **No disk I/O on the request path** — `record()` folds under a mutex
and sets a dirty flag; a background task (and shutdown) writes an atomic
snapshot (temp → fsync → rename → fsync dir), re-arming the dirty flag
on failure.
- **Bounded everything** — per-model cardinality caps with `"other"`
overflow, length-clamped client strings, a 50M-token per-event ceiling,
bounded capture queues/buffers, bounded warn-once set.
- **Honest pricing** — exact-match only; a generic id misses (and WARNs)
rather than mispricing. Unknown ⇒ $0, never a blended fallback.
- **Bedrock regional pricing respected** — the unmodified id is tried
first, because `eu.anthropic.claude-3-5-haiku…` genuinely bills $0.25/M
against the base id's $0.80/M. Geo-stripping is only a fallback for
regions the table doesn't list.

## Pre-existing bugs fixed along the way

Both were found while wiring the recording hooks; both affect `main`
today, independently of stats.

- **Bedrock geo-prefix coverage**: `au.`, `jp.`, and `us-gov.`
cross-region inference profiles weren't recognized, so Anthropic models
invoked through them silently skipped compression — the same failure
mode #953 reported and #999 fixed for `eu.`/`us.`/`apac.`/`global.`. The
list is now complete (all seven published geos) and shared
(`bedrock::vendor::GEO_PREFIXES`) between the vendor gate and pricing,
so the two can't drift apart again.
- **EventStream `Skip` frame stranding**: the streaming parser pulled a
fresh upstream chunk after each unknown (`Skip`ped) event type, delaying
— and at stream end **dropping** — frames already buffered. It now
drains the buffer to exhaustion before awaiting more bytes.
Regression-pinned by
`skip_in_final_chunk_does_not_strand_buffered_frames`.

## Changes Made

- `crates/headroom-proxy/src/observability/ledger.rs` — the store:
`Totals` fold, bounded maps/buckets, atomic snapshot persistence,
payload builders
- `crates/headroom-proxy/src/observability/pricing.rs` — deterministic
lookup over the vendored LiteLLM table (no new data file)
- `crates/headroom-proxy/src/observability/capture.rs` — `PendingRecord`
(deferred finalization), usage extraction for every wire shape, bounded
tee tasks off the byte path
- `crates/headroom-proxy/src/observability/stats_http.rs` +
`dashboard.html` — HTTP surface + embedded self-contained UI
- `crates/headroom-proxy/src/proxy.rs`, `bedrock/invoke.rs`,
`bedrock/invoke_streaming.rs`, `vertex/raw_predict.rs` — per-lane
recording hooks
- `crates/headroom-proxy/src/config.rs` — `--stats` (default on),
`--stats-path`
- `crates/headroom-proxy/src/bedrock/vendor.rs` — shared `GEO_PREFIXES`
(the geo-coverage fix above)
- `crates/headroom-proxy/src/sse/framing.rs` — 4 MiB accumulator cap
(telemetry-side; a stream with no event terminator can no longer grow
the buffer unboundedly)

`--stats=false` restores byte-for-byte pre-feature behaviour (paths
tunnel upstream, nothing recorded) — pinned by
`stats_disabled_falls_through_to_upstream`.

## Testing

- [x] `cargo fmt --all -- --check` passes
- [x] `cargo clippy --workspace -- -D warnings` passes (0 warnings)
- [x] `cargo test --workspace` passes
- [x] New tests added for new functionality (53 new tests across unit +
integration)
- [x] Manual testing performed (live AWS Bedrock, all four native lanes
— see Real Behavior Proof)

Unit: fold-consistency across views, failed-no-savings, pricing
(regional/revision/path forms, collision-miss, untracked-region
fallback), USD math, bounded maps, token-ceiling clamp,
bidi/control-char stripping, atomic flush with no temp litter,
corrupt-file preservation, oversized-state rejection, negative-USD
clamping on load, restart round-trip, week/month rollups, hostile field
lengths. Integration (wiremock): endpoint contracts, `--stats=false`
tunneling, deferred SSE capture, non-streaming capture, OpenAI
cached-token normalisation, failed-upstream accounting, hostile-usage
clamping, all four Bedrock lanes, Vertex, dashboard self-containment.

The persisted-state tests are deliberately adversarial rather than
round-trip only, since the ledger reads a file an operator can
hand-edit: malformed bucket keys (including multi-byte ones that would
panic a naive `&str` slice) are dropped before the age-trim runs,
hostile model-id keys are re-clamped on load, and an implausibly large
state file is treated as corrupt instead of allocated.

### Test Output

```text
$ cargo fmt --all -- --check
  (clean)

$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
  exit=0   0 warnings, 0 errors

$ cargo nextest run -p headroom-proxy
     Summary [4.396s] 488 tests run: 488 passed, 0 skipped

$ cargo nextest run -p headroom-core -p headroom-simulators -p headroom-parity
     Summary [49.535s] 967 tests run: 967 passed, 1 skipped

$ cargo clippy --workspace -- -D warnings
  exit=0   0 warnings, 0 errors

$ cargo test --workspace --doc
  ok, 0 failed
```

## Real Behavior Proof

- Environment: live AWS Bedrock, staging account (SSO profile — name
redacted), `eu-central-1`, `anthropic.claude-3-haiku-20240307-v1:0`,
release binary, compression off (shipped default)
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:8790
--enable-bedrock-native --bedrock-region eu-central-1` then one request
per lane (`/invoke`, `/converse`, `/invoke-with-response-stream` with
`Accept: application/vnd.amazon.eventstream`, `/converse-stream`), one
deliberate failure (nonexistent model), `SIGTERM`, restart, re-read
`/stats` — full transcript, raw proxy logs, and the persisted ledger are
attached below
- Observed result: all four lanes captured usage (input tokens = 48 = 4
× 12; spend $0.000012 = 48 × $0.25/M, exact for Bedrock Haiku 3); the
failure recorded `failed: true` with 0 tokens and $0; lifetime totals
survived the restart while session totals reset; dashboard rendered
against this live data (screenshots attached)
- Not tested: live Vertex (covered by wiremock integration tests only —
no GCP staging account available); live OpenAI Chat/Responses lanes
(same wiremock coverage); dashboard on Windows browsers

The four Bedrock lanes are the ones worth proving live, because two of
them had no parsing before this PR:

| Lane | Wire shape | Captured |
|---|---|---|
| `/invoke` | sync, snake_case `usage` | 12 in / 4 out |
| `/converse` | sync, **camelCase** `usage` | 12 in / 4 out |
| `/invoke-with-response-stream` | **binary EventStream passthrough**,
base64-wrapped chunks | 12 in / 4 out |
| `/converse-stream` | SSE-translated; usage lives in the camelCase
`metadata` frame the Anthropic state machine drops | 12 in / 4 out |

```text
requests      : 5 (failed: 1)
input tokens  : 48
output tokens : 16
cache read/wr : 0 / 0
input spend   : $0.00001200      # 48 × $0.25/M
output spend  : $0.00002000      # 16 × $1.25/M — output bills 5x here
TOTAL spend   : $0.00003200      # what /dashboard reports as "Spend"
savings       : $0.00000000      # compression off -> 0 by design
avg latency   : 256 ms
by provider   : {'bedrock': 5}
by model      : {'anthropic.claude-3-haiku-20240307-v1:0': 4,
                 'anthropic.claude-nonexistent-v99:0': 1}
persistence   : True | healthy: True
```

Those two spend lines are the reason output pricing is in this PR:
**output was 62.5% of the real bill** on this run. Reporting input alone
would have called a $0.000032 spend `$0.000012`.

The exposure split was exercised against the same live proxy:

```text
$ curl /stats/events  -H 'Host: 127.0.0.1:18787'   -> 200   (local tooling works)
$ curl /stats/events  -H 'Host: attacker.com'      -> 404   (DNS-rebound: peer IS loopback)
$ curl /stats         -H 'Host: attacker.com'      -> recent_requests: null
                                                      persistence.path: null
                                                      requests.total still served
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(RUST_DEV.md, CHANGELOG.md)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Known limitations

1. Compression savings read 0 until the live-zone compressors land
(above).
2. Hourly resolution is 48 h, daily ~13 months; older traffic folds into
lifetime totals only. `/stats/events` is an in-memory ring of 100 and is
not persisted — this is a cost ledger, not an audit log.
3. The vendored price table lacks some legacy direct-API ids
(`claude-3-5-sonnet-20241022` is absent while its Bedrock form is
present); those record $0 with a WARN until refreshed via
`scripts/refresh_model_limits.sh`.
4. `output_cost_usd` is absent from state files written before this
change, so it loads as 0: historical spend keeps reading input-only
until those buckets age out (48 h hourly / ~400 d daily). New traffic is
priced in full immediately. Deliberately no migration shim — the field
defaults cleanly and the alternative is retro-pricing history against
today's price table, which would invent numbers.
5. A non-streaming response body over 2 MiB isn't buffered for usage
parsing (the request still records, compression-side numbers only).
Streaming responses have no such cap — they parse incrementally.
6. The endpoints carry no auth, so exposure is split by tier instead
(matching the Python proxy): aggregates are public like `/metrics`,
while per-request rows (`recent_requests`, `/stats/events`) and the
ledger's filesystem path require a local caller — loopback peer **and**
loopback `Host`, the latter being the DNS-rebinding defence. Remote
viewers of `/dashboard` therefore see totals but not the request feed,
and the page says so. Opening that tier to a trusted reverse proxy is
#1959's call to make for both proxies; this PR doesn't invent a second
scheme.
7. Copilot traffic isn't proxied by the Rust binary; it reaches this
dashboard only via external unified-stats layers (unchanged).

## Screenshots (if applicable)

Attached below: live-test terminal transcript, dashboard rendered
against the live Bedrock data (session + lifetime scopes, before/after
restart), raw proxy logs, and the persisted ledger file.

## Additional Notes

- Related issues/PRs: supersedes #1057; completes the geo-prefix
coverage from #999 (design discussion in #953 — the multi-vendor adapter
work proposed there remains out of scope).
- `requests.cached` counts requests whose response reported
`cache_read_tokens > 0`, matching the Python tracker's derivation, so
unified-stats consumers see consistent semantics across both proxies.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2321"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Yasser Sheikh
2026-10-07 20:58:22 -05:00
committed by GitHub
co-authored by JerrettDavis
parent 205668d953
commit 70b21c12c4
27 changed files with 5513 additions and 294 deletions
+7
View File
@@ -18,6 +18,13 @@ __pycache__
dist/
build/
# Rust artifacts. The Dockerfile runs its own `cargo build` inside the
# builder stage, so the host's target/ is never needed — and after a
# local `cargo build` it is 100GB+, which the daemon dutifully copies
# on every `docker build`. CI never noticed because a fresh checkout
# has no target/; it only ever bites local builds.
target/
# Dev/test tooling
.pytest_cache
.coverage
Generated
+1
View File
@@ -2009,6 +2009,7 @@ dependencies = [
"aws-sigv4",
"aws-smithy-runtime-api",
"axum",
"base64 0.22.1",
"bytes",
"bytesize",
"clap",
+74
View File
@@ -92,6 +92,80 @@ curl -si http://127.0.0.1:8787/v1/models
| `--log-level` | | `info` | `RUST_LOG`-style filter |
| `--rewrite-host` / `--no-rewrite-host` | | rewrite | rewrite Host to upstream (default) |
| `--graceful-shutdown-timeout` | | `30s` | wait for in-flight on SIGTERM/SIGINT |
| `--stats` | `HEADROOM_PROXY_STATS` | `true` | native savings stats + `/dashboard` (below) |
| `--stats-path` | `HEADROOM_PROXY_STATS_PATH` | `~/.headroom/native_stats.json` | ledger persistence (honours `HEADROOM_WORKSPACE_DIR`) |
### Savings stats & dashboard (native)
The Rust proxy records per-request savings/cost telemetry on every
lane it forwards — Anthropic `/v1/messages`, OpenAI Chat/Responses,
all four native Bedrock routes, and Vertex `rawPredict` /
`streamRawPredict` — and serves it locally (never tunnelled
upstream):
```bash
curl -s http://127.0.0.1:8787/stats | jq '.session' # since-boot totals
curl -s http://127.0.0.1:8787/stats | jq '.lifetime_by_model'
curl -s "http://127.0.0.1:8787/stats/timeseries?bucket=day" | jq '.points[-7:]'
curl -s "http://127.0.0.1:8787/stats/events?limit=5" # recent-request feed
open http://127.0.0.1:8787/dashboard # embedded UI, zero deps
```
Recording is deferred until the response is fully observed, so
streaming output/cache token counts come from the real SSE /
EventStream usage frames. USD figures come from the vendored LiteLLM
price table (`data/model_prices_and_context_window.json`); a model
missing from it records $0 and logs one WARN — refresh via
`scripts/refresh_model_limits.sh`. Input and output are priced
separately (`input_cost_usd` / `output_cost_usd`) because they bill
at different rates — output is typically 4-5x input, so a spend
figure that counted input alone would understate a short-prompt,
long-answer turn by most of its cost. `total_cost_usd` is the sum
and is what `/dashboard` shows as "Spend". Cache savings are NET: the
cache-read discount minus the cache-write premium (writes bill
above list price), floored at $0 per request — a warm-up turn that
only writes is not counted as negative savings, and gross-read
figures that overstate the benefit are never shown. Failed upstreams count as
failures and accrue no savings — as do proxy-side rejections
(missing AWS credentials, SigV4/ADC failures, oversized bodies), so
an operator-side outage reads as a failure spike rather than as
zero traffic. Aggregates (lifetime, per-model, 48 h hourly +
~13 mo daily buckets) persist as one atomic snapshot written off
the hot path every 10 s and on shutdown.
> **What reads 0 today.** Spend, token counts, and prompt-cache
> savings are live. `tokens_saved` / `compression_savings_usd` are
> wired end-to-end but will report 0 on real traffic until the
> live-zone per-type compressors are implemented — the dispatcher
> currently returns `NoCompression` for every well-formed body
> (`live_zone_mode_with_valid_body_returns_no_compression_pr_b2`
> pins that invariant). Nothing needs changing here when they land;
> the numbers start flowing on their own.
> **No auth; exposure is split by tier.** These endpoints bind
> wherever `--listen` points (default `0.0.0.0:8787`) and carry no
> auth, so the payload is split the same way the Python proxy splits
> its own `/stats`:
>
> - **Aggregates** — spend, tokens, per-model/provider rollups,
> history — are served to anyone, like the Prometheus `/metrics`
> endpoint next door. Same data class, same stance.
> - **Per-request rows** — `recent_requests`, all of `/stats/events`
> — plus the ledger's filesystem path are served **only to local
> callers**: loopback peer *and* a loopback `Host` header (the
> second is the DNS-rebinding defence — a rebound request reaches
> the proxy *from* loopback, so the peer check alone doesn't hold).
> Remote callers get `recent_requests: null` and a 404 from
> `/stats/events`; the dashboard says so rather than erroring.
>
> A request id + model + timestamp is a different sensitivity tier
> from a counter, and `/metrics` never exposes it — so "same as
> /metrics" isn't a licence to hand it to the open internet. Opening
> the local-only tier to a trusted reverse proxy is tracked in
> [#1959](https://github.com/headroomlabs-ai/headroom/issues/1959),
> which is settling that policy for the Python dashboard first; this
> surface should adopt it rather than grow a second scheme.
> `--stats=false` removes the routes entirely.
### Picking the next port: invocation telemetry
+8
View File
@@ -56,6 +56,14 @@ aws-smithy-runtime-api = { workspace = true }
# transitively pulled in by `aws-smithy-*` (so this is not an
# additional cold dep — promoting to a direct one for clarity).
crc32fast = "1"
# Native savings stats: Bedrock InvokeModel EventStream chunks wrap
# their Anthropic SSE payloads in base64 (`{"bytes": "<b64>"}`);
# the usage-capture tee decodes them. Already in the tree via the
# HTTP stack (reqwest/axum/hyper-util/gcp_auth all pull base64
# 0.22) — promoted to direct for the explicit use, so no new
# major-version bucket. (The AWS crates use `base64-simd`, a
# different crate — this is not that one.)
base64 = "0.22"
# Phase D PR-D3: Prometheus metrics for Bedrock observability. The
# `prometheus` crate's default-features pull in `protobuf`, which we
# don't need (we serve text-format scrapes only), so we disable
+73 -13
View File
@@ -51,10 +51,12 @@ use url::Url;
use crate::bedrock::envelope::BedrockEnvelope;
use crate::bedrock::sigv4::{sign_request, SigningInputs};
use crate::bedrock::CompressionStats;
use crate::compression::{
compress_anthropic_request, Outcome as AnthropicOutcome, PassthroughReason,
};
use crate::headers::filter_response_headers;
use crate::observability::{capture, ledger};
use crate::observability::{observe_bedrock_invoke_latency, record_bedrock_invoke};
use crate::proxy::AppState;
use crate::upstream_path::append_segments;
@@ -175,8 +177,17 @@ pub async fn handle_invoke(
"bedrock invoke route received request"
);
// Savings ledger: constructed at handler ENTRY so every exit —
// including proxy-side rejections (bad action path, missing
// credentials, SigV4 failure) — produces exactly one record. A
// credentials outage must show in /stats as a failure spike,
// not as zero traffic. `forward_http` gives its 413 rejections
// the same treatment.
let mut pending_stats =
capture::start_pending(&state, ledger::provider::BEDROCK, &model_id, &request_id);
let is_anthropic = is_anthropic_model_id(&model_id);
let outbound_body: Bytes = if is_anthropic {
let (outbound_body, compression_stats): (Bytes, CompressionStats) = if is_anthropic {
run_anthropic_compression(&body, &state, auth_mode, &request_id)
} else {
tracing::info!(
@@ -186,8 +197,15 @@ pub async fn handle_invoke(
reason = "non_anthropic_vendor",
"bedrock invoke: skipping live-zone compression for non-anthropic vendor"
);
body.clone()
(body.clone(), CompressionStats::default())
};
if let Some(p) = pending_stats.as_mut() {
p.set_compression(
compression_stats.tokens_before,
compression_stats.tokens_after,
compression_stats.transforms,
);
}
// Resolve the Bedrock action from the inbound path so `/converse`
// forwards to the upstream Converse endpoint instead of `/invoke`.
@@ -202,6 +220,7 @@ pub async fn handle_invoke(
path = %uri.path(),
"bedrock invoke: unrecognized action path"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::BAD_REQUEST,
"bedrock_invoke_action_invalid",
@@ -235,6 +254,7 @@ pub async fn handle_invoke(
error = %msg,
"bedrock invoke: failed to construct upstream URL"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::INTERNAL_SERVER_ERROR,
"bedrock_endpoint_invalid",
@@ -253,6 +273,7 @@ pub async fn handle_invoke(
model_id = %model_id,
"bedrock invoke: refusing to forward without AWS credentials"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::INTERNAL_SERVER_ERROR,
"bedrock_credentials_missing",
@@ -288,6 +309,7 @@ pub async fn handle_invoke(
error = %e,
"bedrock invoke: SigV4 signing failed; refusing to forward"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::INTERNAL_SERVER_ERROR,
"bedrock_sigv4_failed",
@@ -329,6 +351,7 @@ pub async fn handle_invoke(
error = %e,
"bedrock invoke: invalid HTTP method"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::BAD_REQUEST,
"bedrock_invalid_method",
@@ -348,6 +371,9 @@ pub async fn handle_invoke(
let upstream_resp = match upstream_resp {
Ok(r) => r,
Err(e) => {
if let Some(p) = pending_stats.take() {
p.finalize_failed();
}
tracing::warn!(
event = "bedrock_upstream_error",
request_id = %request_id,
@@ -365,6 +391,11 @@ pub async fn handle_invoke(
let status =
StatusCode::from_u16(upstream_resp.status().as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
if !status.is_success() {
if let Some(p) = pending_stats.as_mut() {
p.mark_failed();
}
}
let resp_headers = filter_response_headers(upstream_resp.headers());
tracing::info!(
@@ -376,10 +407,23 @@ pub async fn handle_invoke(
"bedrock invoke: response forwarded"
);
// Stream the response body back without buffering.
let stream = upstream_resp
.bytes_stream()
.map(|r| r.map_err(std::io::Error::other));
// Stream the response body back without buffering. When the
// savings ledger is recording, tee the bytes into a bounded
// capture task that parses the sync response's usage block
// (InvokeModel snake_case / Converse camelCase) at body end —
// best-effort, never blocking the client byte path.
let capture_tx = pending_stats
.take()
.map(|p| capture::spawn_json_usage_capture(capture::ResponseShape::Bedrock, p));
let stream = upstream_resp.bytes_stream().map(move |r| match r {
Ok(b) => {
if let Some(tx) = &capture_tx {
let _ = tx.try_send(b.clone());
}
Ok(b)
}
Err(e) => Err(std::io::Error::other(e)),
});
let body_out = Body::from_stream(stream);
let mut builder = Response::builder().status(status);
@@ -417,7 +461,7 @@ fn run_anthropic_compression(
state: &AppState,
_auth_mode: AuthMode,
request_id: &str,
) -> Bytes {
) -> (Bytes, CompressionStats) {
// Detect envelope shape. A parseable InvokeModel envelope takes the
// re-emit path below (anthropic_version pinned first); a non-envelope
// body (e.g. a Converse-shaped payload) still runs through the
@@ -453,7 +497,7 @@ fn run_anthropic_compression(
request_id,
);
match outcome {
AnthropicOutcome::NoCompression => body.clone(),
AnthropicOutcome::NoCompression => (body.clone(), CompressionStats::default()),
AnthropicOutcome::Passthrough { reason } => {
tracing::info!(
event = "bedrock_compression_passthrough",
@@ -464,15 +508,26 @@ fn run_anthropic_compression(
// The compressor's passthrough variants all leave bytes
// unchanged. Forward the original.
let _ = (PassthroughReason::ModeOff, PassthroughReason::NoMessages); // pin types
body.clone()
(body.clone(), CompressionStats::default())
}
AnthropicOutcome::Compressed { body: new_body, .. } => {
AnthropicOutcome::Compressed {
body: new_body,
tokens_before,
tokens_after,
strategies_applied,
..
} => {
let stats = CompressionStats {
tokens_before: tokens_before as u64,
tokens_after: tokens_after as u64,
transforms: strategies_applied.iter().map(|s| s.to_string()).collect(),
};
if parsed_envelope {
// Defence-in-depth: re-emit so anthropic_version is the
// first key. With preserve_order this is a no-op on the
// happy path.
match BedrockEnvelope::ensure_anthropic_version_first(&new_body) {
Ok(b) => b,
Ok(b) => (b, stats),
Err(e) => {
tracing::error!(
event = "bedrock_envelope_reemit_failed",
@@ -480,11 +535,13 @@ fn run_anthropic_compression(
error = %e,
"bedrock invoke: failed to re-emit envelope; falling back to original body"
);
body.clone()
// Original bytes forwarded ⇒ no compression
// happened as far as the wire is concerned.
(body.clone(), CompressionStats::default())
}
}
} else {
new_body
(new_body, stats)
}
}
}
@@ -631,6 +688,7 @@ mod tests {
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
"test".to_string(),
)),
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
};
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/converse"
.parse()
@@ -669,6 +727,7 @@ mod tests {
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
"test".to_string(),
)),
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
};
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/invoke"
.parse()
@@ -705,6 +764,7 @@ mod tests {
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
"test".to_string(),
)),
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
};
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/invoke"
.parse()
@@ -61,10 +61,13 @@ use crate::bedrock::eventstream_to_sse::{
translate_message, OutputMode, TranslateError, TranslateOutcome,
};
use crate::bedrock::sigv4::{sign_request, SigningInputs};
use crate::bedrock::CompressionStats;
use crate::compression::{
compress_anthropic_request, Outcome as AnthropicOutcome, PassthroughReason,
};
use crate::headers::filter_response_headers;
use crate::observability::capture::TEE_QUEUE_DEPTH as SSE_PARSER_QUEUE_DEPTH;
use crate::observability::{capture, ledger};
use crate::observability::{
observe_bedrock_invoke_latency, record_bedrock_eventstream_message, record_bedrock_invoke,
};
@@ -153,9 +156,14 @@ pub async fn handle_invoke_streaming(
"bedrock invoke-with-response-stream route received request"
);
// Constructed at handler ENTRY so every exit — including
// proxy-side rejections — records exactly once.
let mut pending_stats =
capture::start_pending(&state, ledger::provider::BEDROCK, &model_id, &request_id);
// 1. Live-zone compression for Anthropic-shape bodies (same as D1).
let is_anthropic = is_anthropic_model_id(&model_id);
let outbound_body: Bytes = if is_anthropic {
let (outbound_body, compression_stats): (Bytes, CompressionStats) = if is_anthropic {
run_anthropic_compression(&body, &state, auth_mode, &request_id)
} else {
tracing::info!(
@@ -165,8 +173,15 @@ pub async fn handle_invoke_streaming(
reason = "non_anthropic_vendor",
"bedrock invoke-streaming: skipping compression for non-anthropic vendor"
);
body.clone()
(body.clone(), CompressionStats::default())
};
if let Some(p) = pending_stats.as_mut() {
p.set_compression(
compression_stats.tokens_before,
compression_stats.tokens_after,
compression_stats.transforms,
);
}
// 2. Resolve the Bedrock streaming action from the inbound path and
// build the upstream URL.
@@ -179,6 +194,7 @@ pub async fn handle_invoke_streaming(
path = %uri.path(),
"bedrock invoke-streaming: unrecognized streaming action path"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::BAD_REQUEST,
"bedrock_streaming_action_invalid",
@@ -209,6 +225,7 @@ pub async fn handle_invoke_streaming(
error = %msg,
"bedrock invoke-streaming: failed to construct upstream URL"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::INTERNAL_SERVER_ERROR,
"bedrock_endpoint_invalid",
@@ -227,6 +244,7 @@ pub async fn handle_invoke_streaming(
model_id = %model_id,
"bedrock invoke-streaming: refusing to forward without AWS credentials"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::INTERNAL_SERVER_ERROR,
"bedrock_credentials_missing",
@@ -261,6 +279,7 @@ pub async fn handle_invoke_streaming(
error = %e,
"bedrock invoke-streaming: SigV4 signing failed"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::INTERNAL_SERVER_ERROR,
"bedrock_sigv4_failed",
@@ -298,6 +317,7 @@ pub async fn handle_invoke_streaming(
error = %e,
"bedrock invoke-streaming: invalid HTTP method"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::BAD_REQUEST,
"bedrock_invalid_method",
@@ -317,6 +337,9 @@ pub async fn handle_invoke_streaming(
let upstream_resp = match upstream_resp {
Ok(r) => r,
Err(e) => {
if let Some(p) = pending_stats.take() {
p.finalize_failed();
}
tracing::warn!(
event = "bedrock_upstream_error",
request_id = %request_id,
@@ -334,6 +357,11 @@ pub async fn handle_invoke_streaming(
let status =
StatusCode::from_u16(upstream_resp.status().as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
if !status.is_success() {
if let Some(p) = pending_stats.as_mut() {
p.mark_failed();
}
}
let upstream_content_type = upstream_resp
.headers()
.get(http::header::CONTENT_TYPE)
@@ -373,10 +401,22 @@ pub async fn handle_invoke_streaming(
// Even here we drop content-length: the proxy streams the
// body and reqwest's transparent decompression may already
// have changed the byte length on the way in.
// Usage capture: this is typically an error envelope (the
// pending event is already marked failed on non-2xx); parse
// it as JSON anyway so a 2xx JSON body still yields usage.
resp_headers.remove(http::header::CONTENT_LENGTH);
let stream = upstream_resp
.bytes_stream()
.map(|r| r.map_err(std::io::Error::other));
let capture_tx = pending_stats
.take()
.map(|p| capture::spawn_json_usage_capture(capture::ResponseShape::Bedrock, p));
let stream = upstream_resp.bytes_stream().map(move |r| match r {
Ok(b) => {
if let Some(tx) = &capture_tx {
let _ = tx.try_send(b.clone());
}
Ok(b)
}
Err(e) => Err(std::io::Error::other(e)),
});
let body_out = Body::from_stream(stream);
return finish(status, resp_headers, body_out, &request_id);
}
@@ -410,7 +450,20 @@ pub async fn handle_invoke_streaming(
resp_headers.insert(http::header::CONTENT_TYPE, v);
}
}
let body_out = Body::from_stream(upstream_stream);
// Usage capture: nothing else parses the frames in
// passthrough mode, so tee the raw bytes into a bounded
// EventStream telemetry parser (chunk / metadata usage;
// finalizes the ledger event when the stream closes).
let capture_tx = pending_stats
.take()
.map(capture::spawn_eventstream_usage_capture);
let teed = upstream_stream.map(move |r| {
if let (Ok(b), Some(tx)) = (&r, &capture_tx) {
let _ = tx.try_send(b.clone());
}
r
});
let body_out = Body::from_stream(teed);
finish(status, resp_headers, body_out, &request_id)
}
OutputMode::Sse => {
@@ -429,7 +482,8 @@ pub async fn handle_invoke_streaming(
model_id.clone(),
region.clone(),
);
let translated = tee_to_anthropic_state(translated, request_id.clone());
let translated =
tee_to_anthropic_state(translated, request_id.clone(), pending_stats.take());
let body_out = Body::from_stream(translated);
finish(status, resp_headers, body_out, &request_id)
}
@@ -475,201 +529,125 @@ where
if done {
return None;
}
// First, drain any complete messages already in the
// parser's buffer (bytes from the previous chunk).
loop {
match parser.next_message() {
Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {
Ok(TranslateOutcome::Emit(frame)) => {
// PR-D3: per-message Prometheus metric.
// The label `event_type` is bounded by
// AWS's documented vocabulary (chunk,
// metadata, exception variants); not
// customer-controlled.
let event_type = msg.event_type().unwrap_or("unknown").to_string();
record_bedrock_eventstream_message(&model_id, &region, &event_type);
tracing::debug!(
event = "bedrock_eventstream_message",
request_id = %request_id,
event_type = %event_type,
payload_bytes = msg.payload.len(),
"translated bedrock eventstream message"
);
return Some((
Ok(frame),
(parser, upstream, false, request_id, model_id, region),
));
}
Ok(TranslateOutcome::Skip { event_type }) => {
tracing::warn!(
event = "bedrock_eventstream_unknown_event_type",
request_id = %request_id,
event_type = %event_type,
"skipping unknown bedrock eventstream message"
);
// Loop and try the next message in the buffer.
continue;
}
Err(TranslateError::UpstreamException { payload_preview }) => {
tracing::warn!(
event = "bedrock_eventstream_upstream_exception",
request_id = %request_id,
payload_preview = %payload_preview,
"bedrock eventstream upstream exception"
);
// Emit the exception as an Anthropic-shape
// SSE error frame so the client sees it.
let json = serde_json::json!({
"type": "error",
"error": {
"type": "bedrock_upstream_exception",
"message": payload_preview,
}
})
.to_string();
let mut frame = Vec::with_capacity(json.len() + 32);
frame.extend_from_slice(b"event: error\ndata: ");
frame.extend_from_slice(json.as_bytes());
frame.extend_from_slice(b"\n\n");
return Some((
Ok(Bytes::from(frame)),
(parser, upstream, true, request_id, model_id, region),
));
}
Err(TranslateError::MissingEventType) => {
tracing::warn!(
event = "bedrock_eventstream_missing_event_type",
request_id = %request_id,
"bedrock eventstream message missing :event-type; emitting error frame"
);
let frame = error_sse_frame(
"bedrock_eventstream_missing_event_type",
"Bedrock message missing :event-type header",
);
return Some((
Ok(frame),
(parser, upstream, true, request_id, model_id, region),
));
}
},
Ok(None) => break,
Err(parse_err) => {
let event_name = match &parse_err {
ParseError::PreludeCrcMismatch { .. }
| ParseError::MessageCrcMismatch { .. } => {
"bedrock_eventstream_crc_mismatch"
// Drain-first alternation: after a chunk is pushed we
// re-enter the drain phase rather than pulling again,
// so a Skip'd message can never strand complete
// messages in the parser buffer. (The previous shape
// pulled a fresh upstream chunk after a Skip —
// delaying, and at stream end DROPPING, frames that
// were already buffered behind an unknown event type.)
'parse: loop {
// Phase 1: emit complete messages already buffered.
loop {
match parser.next_message() {
Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {
Ok(TranslateOutcome::Emit(frame)) => {
// PR-D3: per-message Prometheus metric.
// The label `event_type` is bounded by
// AWS's documented vocabulary (chunk,
// metadata, exception variants); not
// customer-controlled.
let event_type =
msg.event_type().unwrap_or("unknown").to_string();
record_bedrock_eventstream_message(
&model_id,
&region,
&event_type,
);
tracing::debug!(
event = "bedrock_eventstream_message",
request_id = %request_id,
event_type = %event_type,
payload_bytes = msg.payload.len(),
"translated bedrock eventstream message"
);
return Some((
Ok(frame),
(parser, upstream, false, request_id, model_id, region),
));
}
Ok(TranslateOutcome::Skip { event_type }) => {
tracing::warn!(
event = "bedrock_eventstream_unknown_event_type",
request_id = %request_id,
event_type = %event_type,
"skipping unknown bedrock eventstream message"
);
// Loop and try the next message in the buffer.
continue;
}
_ => "bedrock_eventstream_parse_failed",
};
tracing::warn!(
event = event_name,
request_id = %request_id,
error = %parse_err,
"bedrock eventstream parse failure; closing translated stream"
);
let frame = error_sse_frame(event_name, &parse_err.to_string());
return Some((
Ok(frame),
(parser, upstream, true, request_id, model_id, region),
));
}
}
}
// Buffer drained; pull the next chunk from upstream.
loop {
match upstream.next().await {
Some(Ok(chunk)) => {
parser.push(&chunk);
// Loop back through the parser to emit any
// newly-complete messages.
match parser.next_message() {
Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {
Ok(TranslateOutcome::Emit(frame)) => {
// PR-D3: per-message Prometheus
// metric (mirror of the parser-buffer
// drain branch above).
let event_type =
msg.event_type().unwrap_or("unknown").to_string();
record_bedrock_eventstream_message(
&model_id,
&region,
&event_type,
);
tracing::debug!(
event = "bedrock_eventstream_message",
request_id = %request_id,
event_type = %event_type,
payload_bytes = msg.payload.len(),
"translated bedrock eventstream message"
);
return Some((
Ok(frame),
(parser, upstream, false, request_id, model_id, region),
));
}
Ok(TranslateOutcome::Skip { event_type }) => {
tracing::warn!(
event = "bedrock_eventstream_unknown_event_type",
request_id = %request_id,
event_type = %event_type,
"skipping unknown bedrock eventstream message"
);
// Continue draining the parser /
// pulling more chunks.
continue;
}
Err(TranslateError::UpstreamException { payload_preview }) => {
let json = serde_json::json!({
"type": "error",
"error": {
"type": "bedrock_upstream_exception",
"message": payload_preview,
}
})
.to_string();
let mut frame = Vec::with_capacity(json.len() + 32);
frame.extend_from_slice(b"event: error\ndata: ");
frame.extend_from_slice(json.as_bytes());
frame.extend_from_slice(b"\n\n");
return Some((
Ok(Bytes::from(frame)),
(parser, upstream, true, request_id, model_id, region),
));
}
Err(TranslateError::MissingEventType) => {
let frame = error_sse_frame(
"bedrock_eventstream_missing_event_type",
"Bedrock message missing :event-type header",
);
return Some((
Ok(frame),
(parser, upstream, true, request_id, model_id, region),
));
}
},
Ok(None) => continue,
Err(parse_err) => {
let event_name = match &parse_err {
ParseError::PreludeCrcMismatch { .. }
| ParseError::MessageCrcMismatch { .. } => {
"bedrock_eventstream_crc_mismatch"
Err(TranslateError::UpstreamException { payload_preview }) => {
tracing::warn!(
event = "bedrock_eventstream_upstream_exception",
request_id = %request_id,
payload_preview = %payload_preview,
"bedrock eventstream upstream exception"
);
// Emit the exception as an Anthropic-shape
// SSE error frame so the client sees it.
let json = serde_json::json!({
"type": "error",
"error": {
"type": "bedrock_upstream_exception",
"message": payload_preview,
}
_ => "bedrock_eventstream_parse_failed",
};
})
.to_string();
let mut frame = Vec::with_capacity(json.len() + 32);
frame.extend_from_slice(b"event: error\ndata: ");
frame.extend_from_slice(json.as_bytes());
frame.extend_from_slice(b"\n\n");
return Some((
Ok(Bytes::from(frame)),
(parser, upstream, true, request_id, model_id, region),
));
}
Err(TranslateError::MissingEventType) => {
tracing::warn!(
event = event_name,
event = "bedrock_eventstream_missing_event_type",
request_id = %request_id,
error = %parse_err,
"bedrock eventstream parse failure"
"bedrock eventstream message missing :event-type; emitting error frame"
);
let frame = error_sse_frame(
"bedrock_eventstream_missing_event_type",
"Bedrock message missing :event-type header",
);
let frame = error_sse_frame(event_name, &parse_err.to_string());
return Some((
Ok(frame),
(parser, upstream, true, request_id, model_id, region),
));
}
},
Ok(None) => break,
Err(parse_err) => {
let event_name = match &parse_err {
ParseError::PreludeCrcMismatch { .. }
| ParseError::MessageCrcMismatch { .. } => {
"bedrock_eventstream_crc_mismatch"
}
_ => "bedrock_eventstream_parse_failed",
};
tracing::warn!(
event = event_name,
request_id = %request_id,
error = %parse_err,
"bedrock eventstream parse failure; closing translated stream"
);
let frame = error_sse_frame(event_name, &parse_err.to_string());
return Some((
Ok(frame),
(parser, upstream, true, request_id, model_id, region),
));
}
}
}
// Phase 2: buffer fully drained — pull exactly
// one chunk, then loop back to the drain phase.
match upstream.next().await {
Some(Ok(chunk)) => {
parser.push(&chunk);
continue 'parse;
}
Some(Err(e)) => {
tracing::warn!(
event = "bedrock_eventstream_upstream_io_error",
@@ -683,10 +661,11 @@ where
));
}
None => {
// End of upstream stream. If buffered bytes
// remain that did not parse into a message,
// log loudly — we are NOT silently dropping
// them.
// End of upstream stream. The drain phase
// above ran to Ok(None) first, so no
// complete message can be stranded here;
// leftover bytes are a truncated partial
// message — log loudly, never silently.
if parser.buffered_len() > 0 {
tracing::warn!(
event = "bedrock_eventstream_truncated",
@@ -713,6 +692,7 @@ where
fn tee_to_anthropic_state<S>(
upstream: S,
request_id: String,
pending_stats: Option<capture::PendingRecord>,
) -> impl Stream<Item = Result<Bytes, std::io::Error>>
where
S: Stream<Item = Result<Bytes, std::io::Error>> + Send + 'static,
@@ -720,7 +700,7 @@ where
let (tx, rx) = tokio::sync::mpsc::channel::<Bytes>(SSE_PARSER_QUEUE_DEPTH);
let parser_request_id = request_id.clone();
tokio::spawn(async move {
run_anthropic_state_machine(rx, parser_request_id).await;
run_anthropic_state_machine(rx, parser_request_id, pending_stats).await;
});
upstream.map(move |r| {
@@ -738,8 +718,6 @@ where
})
}
const SSE_PARSER_QUEUE_DEPTH: usize = 256;
/// Dedicated state-machine task. Mirrors the
/// `run_sse_state_machine(SseStreamKind::Anthropic, ...)` arm in
/// `proxy.rs` so usage extraction works identically for direct
@@ -747,17 +725,31 @@ const SSE_PARSER_QUEUE_DEPTH: usize = 256;
async fn run_anthropic_state_machine(
mut rx: tokio::sync::mpsc::Receiver<Bytes>,
request_id: String,
pending_stats: Option<capture::PendingRecord>,
) {
use crate::sse::anthropic::AnthropicStreamState;
use crate::sse::framing::SseFramer;
let mut framer = SseFramer::new();
let mut state = AnthropicStreamState::new();
// Converse-native frames (`metadata`, `messageStart`, …) carry no
// Anthropic `type`, so they arrive here as event-name-less frames
// the Anthropic state machine drops. Their camelCase usage block
// is the ONLY usage signal on a native Converse stream — scan
// them separately so `/converse-stream` requests don't record
// zero tokens.
let mut converse_usage = ledger::Usage::default();
while let Some(chunk) = rx.recv().await {
framer.push(&chunk);
while let Some(ev_result) = framer.next_event() {
match ev_result {
Ok(ev) => {
if ev.event_name.is_none() {
if let Ok(payload) = serde_json::from_slice::<serde_json::Value>(&ev.data) {
capture::merge_stream_usage(&mut converse_usage, &payload);
}
continue;
}
if let Err(e) = state.apply(ev) {
tracing::warn!(
request_id = %request_id,
@@ -787,6 +779,16 @@ async fn run_anthropic_state_machine(
blocks = state.blocks.len(),
"bedrock translated stream: closed"
);
if let Some(mut p) = pending_stats {
// In-band `{"type":"error"}` on a 200 OK stream is not a
// billable success (same rule as the direct Anthropic lane).
if matches!(state.status, crate::sse::anthropic::StreamStatus::Errored) {
p.mark_failed();
}
let mut usage = capture::usage_from_anthropic_state(&state);
capture::merge_max(&mut usage, converse_usage);
p.finalize(usage);
}
}
/// True when the content-type is `application/vnd.amazon.eventstream`
@@ -866,7 +868,7 @@ fn run_anthropic_compression(
state: &AppState,
_auth_mode: AuthMode,
request_id: &str,
) -> Bytes {
) -> (Bytes, CompressionStats) {
use crate::bedrock::envelope::BedrockEnvelope;
let parsed_envelope = BedrockEnvelope::parse(body).is_ok();
@@ -888,7 +890,7 @@ fn run_anthropic_compression(
request_id,
);
match outcome {
AnthropicOutcome::NoCompression => body.clone(),
AnthropicOutcome::NoCompression => (body.clone(), CompressionStats::default()),
AnthropicOutcome::Passthrough { reason } => {
tracing::info!(
event = "bedrock_compression_passthrough",
@@ -897,12 +899,23 @@ fn run_anthropic_compression(
"bedrock invoke-streaming: passthrough"
);
let _ = (PassthroughReason::ModeOff, PassthroughReason::NoMessages);
body.clone()
(body.clone(), CompressionStats::default())
}
AnthropicOutcome::Compressed { body: new_body, .. } => {
AnthropicOutcome::Compressed {
body: new_body,
tokens_before,
tokens_after,
strategies_applied,
..
} => {
let stats = CompressionStats {
tokens_before: tokens_before as u64,
tokens_after: tokens_after as u64,
transforms: strategies_applied.iter().map(|s| s.to_string()).collect(),
};
if parsed_envelope {
match BedrockEnvelope::ensure_anthropic_version_first(&new_body) {
Ok(b) => b,
Ok(b) => (b, stats),
Err(e) => {
tracing::error!(
event = "bedrock_envelope_reemit_failed",
@@ -910,11 +923,13 @@ fn run_anthropic_compression(
error = %e,
"bedrock invoke-streaming: failed to re-emit envelope"
);
body.clone()
// Original bytes forwarded ⇒ no compression
// happened as far as the wire is concerned.
(body.clone(), CompressionStats::default())
}
}
} else {
new_body
(new_body, stats)
}
}
}
@@ -1035,6 +1050,7 @@ mod tests {
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
"test".to_string(),
)),
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
};
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/invoke-with-response-stream"
.parse()
@@ -1075,6 +1091,7 @@ mod tests {
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
"test".to_string(),
)),
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
};
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/converse-stream"
.parse()
+9
View File
@@ -54,6 +54,15 @@ pub mod invoke_streaming;
pub mod sigv4;
pub mod vendor;
/// Compression-side numbers the Bedrock handlers hand to the savings
/// ledger alongside the (possibly compressed) outbound body.
#[derive(Debug, Default, Clone)]
pub(crate) struct CompressionStats {
pub tokens_before: u64,
pub tokens_after: u64,
pub transforms: Vec<String>,
}
pub use auth_mode_layer::classify_and_attach_auth_mode;
pub use envelope::{BedrockEnvelope, EnvelopeError};
pub use eventstream::{
+34 -5
View File
@@ -13,7 +13,28 @@
/// vendor segment. Stripped (once) before vendor resolution. Kept to a
/// closed, known set so an unrelated `something.anthropic.x` id is not
/// mistaken for an Anthropic inference profile.
const GEO_PREFIXES: [&str; 4] = ["eu.", "us.", "apac.", "global."];
///
/// The set is the prefixes present in the vendored LiteLLM price
/// table (`data/model_prices_and_context_window.json`) — every one of
/// these fronts real `<geo>.anthropic.…` ids there. A missing entry
/// is not cosmetic: an unrecognized geo prefix makes
/// `is_anthropic_model_id` return false, silently skipping
/// compression for that region's inference profiles (`au.`/`jp.`/
/// `us-gov.` were missing until review of the stats work caught it).
pub(crate) const GEO_PREFIXES: [&str; 7] =
["eu.", "us.", "us-gov.", "apac.", "au.", "jp.", "global."];
/// Strip a cross-region inference-profile geo prefix, if present.
/// `eu.anthropic.claude-…` → `anthropic.claude-…`; ids without a
/// known geo prefix return unchanged. Shared with
/// `observability::pricing` so price lookups see the same canonical
/// id the vendor gate sees.
pub(crate) fn strip_geo_prefix(model_id: &str) -> &str {
GEO_PREFIXES
.iter()
.find_map(|p| model_id.strip_prefix(p))
.unwrap_or(model_id)
}
/// Resolve the canonical vendor of a Bedrock model id, stripping a
/// cross-region inference-profile geo prefix first.
@@ -21,10 +42,7 @@ const GEO_PREFIXES: [&str; 4] = ["eu.", "us.", "apac.", "global."];
/// `eu.anthropic.claude-…` → `anthropic`; `amazon.titan-…` → `amazon`;
/// `global.amazon.nova-…` → `amazon`.
pub fn canonical_vendor(model_id: &str) -> &str {
let stripped = GEO_PREFIXES
.iter()
.find_map(|p| model_id.strip_prefix(p))
.unwrap_or(model_id);
let stripped = strip_geo_prefix(model_id);
stripped.split('.').next().unwrap_or(stripped)
}
@@ -73,6 +91,17 @@ mod tests {
assert!(is_anthropic_model_id(
"global.anthropic.claude-haiku-4-5-20251001-v1:0"
));
// au./jp./us-gov. are real AWS geo prefixes (present in the
// vendored price table); they must gate as Anthropic too.
assert!(is_anthropic_model_id(
"au.anthropic.claude-sonnet-4-5-20250929-v1:0"
));
assert!(is_anthropic_model_id(
"jp.anthropic.claude-haiku-4-5-20251001-v1:0"
));
assert!(is_anthropic_model_id(
"us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0"
));
// Non-Anthropic vendors, including geo-prefixed, stay false.
assert!(!is_anthropic_model_id("amazon.titan-text-express-v1"));
assert!(!is_anthropic_model_id("meta.llama3-70b-instruct-v1:0"));
@@ -66,7 +66,10 @@ pub(crate) const DEFAULT_CONTEXT_WINDOW: u32 = 128_000;
/// LiteLLM's vendored model price + context-window table. Refreshed
/// via `scripts/refresh_model_limits.sh`. ~1.4MB; embedded into the
/// binary so the proxy ships with no startup network dependency.
const VENDORED_JSON: &str = include_str!("../../data/model_prices_and_context_window.json");
// Shared with `observability::pricing`, which reads the per-token
// cost fields of the same table — one embedded copy, two consumers.
pub(crate) const VENDORED_JSON: &str =
include_str!("../../data/model_prices_and_context_window.json");
/// Parsed lookup: model id → max input tokens. Built lazily on
/// first call; subsequent calls reuse the same `HashMap`.
+90
View File
@@ -592,6 +592,34 @@ pub struct CliArgs {
default_value = "https://www.googleapis.com/auth/cloud-platform"
)]
pub vertex_adc_scope: String,
/// Native savings stats: record per-request savings/cost
/// telemetry and serve `/stats`, `/stats/timeseries`,
/// `/stats/events`, and `/dashboard`. When disabled, nothing is
/// recorded and those paths fall through to the catch-all
/// forwarder (i.e. they reach the upstream, matching the
/// pre-feature behaviour).
///
/// Source priority: CLI flag → `HEADROOM_PROXY_STATS` env var →
/// default (`false`).
#[arg(
long = "stats",
env = "HEADROOM_PROXY_STATS",
default_value_t = false,
action = clap::ArgAction::Set,
)]
pub stats: bool,
/// Where the savings ledger persists its state. When unset, the
/// proxy uses `$HEADROOM_WORKSPACE_DIR/native_stats.json` when
/// that env var is set (how the Python proxy resolves its
/// workspace), else `~/.headroom/native_stats.json`; if no home
/// directory can be resolved, stats stay in-memory (logged).
///
/// Source priority: CLI flag → `HEADROOM_PROXY_STATS_PATH`
/// env var → workspace-derived default.
#[arg(long = "stats-path", env = "HEADROOM_PROXY_STATS_PATH")]
pub stats_path: Option<std::path::PathBuf>,
}
fn parse_duration(s: &str) -> Result<Duration, String> {
@@ -708,6 +736,11 @@ pub struct Config {
/// PR-D4: GCP ADC OAuth scope used when fetching the bearer
/// token. Default `https://www.googleapis.com/auth/cloud-platform`.
pub vertex_adc_scope: String,
/// Native savings stats: record per-request telemetry and serve
/// `/stats`, `/stats/timeseries`, `/stats/events`, `/dashboard`.
pub stats: bool,
/// Ledger persistence path. `None` keeps stats in-memory.
pub stats_path: Option<std::path::PathBuf>,
}
impl Config {
@@ -775,6 +808,8 @@ impl Config {
bedrock_validate_eventstream_crc: args.bedrock_validate_eventstream_crc,
vertex_region: args.vertex_region,
vertex_adc_scope: args.vertex_adc_scope,
stats: args.stats,
stats_path: args.stats_path.or_else(default_stats_path),
}
}
@@ -838,10 +873,47 @@ impl Config {
// only; the upstream URL is `upstream`).
vertex_region: "us-central1".to_string(),
vertex_adc_scope: "https://www.googleapis.com/auth/cloud-platform".to_string(),
// Stats on (routes mounted, recording active) but
// in-memory: tests never touch a real home directory.
stats: true,
stats_path: None,
}
}
}
/// Default persistence location for the savings ledger. Honours the
/// same `HEADROOM_WORKSPACE_DIR` override the Python proxy uses for
/// its workspace, falling back to `~/.headroom`. `None` (no home
/// resolvable) keeps stats in-memory.
fn default_stats_path() -> Option<std::path::PathBuf> {
if let Ok(dir) = std::env::var("HEADROOM_WORKSPACE_DIR") {
let dir = dir.trim();
if !dir.is_empty() {
// Tilde-expand, matching the Python proxy's
// `workspace_dir()` semantics (its tests pin
// `HEADROOM_WORKSPACE_DIR=~/custom` → `$HOME/custom`).
let expanded = if let Some(rest) = dir.strip_prefix("~/") {
match home_dir() {
Some(home) => home.join(rest),
None => return None,
}
} else {
std::path::PathBuf::from(dir)
};
return Some(expanded.join("native_stats.json"));
}
}
Some(home_dir()?.join(".headroom").join("native_stats.json"))
}
fn home_dir() -> Option<std::path::PathBuf> {
std::env::var("HOME")
.ok()
.filter(|h| !h.trim().is_empty())
.or_else(|| std::env::var("USERPROFILE").ok())
.map(std::path::PathBuf::from)
}
#[cfg(test)]
mod rollout_input_tests {
use super::*;
@@ -885,4 +957,22 @@ mod rollout_input_tests {
assert!(!config.enable_responses_streaming);
assert!(!config.enable_bedrock_native);
}
#[test]
fn native_stats_require_explicit_opt_in() {
let base = ["headroom-proxy", "--upstream", "http://127.0.0.1:9"];
let default_config = Config::from_cli(CliArgs::try_parse_from(base).unwrap());
assert!(!default_config.stats);
let enabled = Config::from_cli(
CliArgs::try_parse_from([
"headroom-proxy",
"--upstream",
"http://127.0.0.1:9",
"--stats=true",
])
.unwrap(),
);
assert!(enabled.stats);
}
}
+38
View File
@@ -105,6 +105,42 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
}
}
// Savings observability: with the master compression switch on
// but the mode off, the dispatcher passes every body through and
// tokens_saved reads 0 on the dashboard with no hint why. Say so
// once at startup instead of letting the zero look like a bug.
if config.compression
&& matches!(
config.compression_mode,
headroom_proxy::config::CompressionMode::Off
)
{
tracing::warn!(
event = "compression_mode_off",
"--compression is on but --compression-mode is `off`: bodies \
pass through unmodified and the dashboard's compression \
savings will read 0. Set --compression-mode live_zone to \
engage the dispatcher."
);
}
// Savings ledger: background persistence while serving, plus a
// best-effort final flush after the server drains (capture tasks
// are detached, so a request finishing in the same instant as
// shutdown may still miss it — see observability::capture docs).
let stats_ledger = state.stats.clone();
if config.stats {
// Pay the vendored price-table parse now, not on the first
// recorded request.
headroom_proxy::observability::pricing::warm();
tracing::info!(
event = "stats_enabled",
path = ?stats_ledger.path().map(|p| p.display().to_string()),
"native savings stats active (/stats, /dashboard)"
);
headroom_proxy::observability::ledger::spawn_flusher(stats_ledger.clone());
}
let app = build_app(state).into_make_service_with_connect_info::<SocketAddr>();
let listener = tokio::net::TcpListener::bind(config.listen).await?;
@@ -122,6 +158,8 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
})
.await?;
stats_ledger.flush().await;
Ok(())
}
@@ -0,0 +1,594 @@
//! Response-usage capture — Phase H. Shared by every lane.
//!
//! Recording a request's savings needs the *response's* token usage,
//! which arrives in four wire shapes:
//!
//! - JSON bodies (non-streaming Anthropic / OpenAI / Bedrock /
//! Vertex responses),
//! - Anthropic-style SSE streams (direct, Vertex, and Bedrock's
//! SSE-translated mode) — already accumulated by
//! [`crate::sse::anthropic::AnthropicStreamState`],
//! - Converse-native frames whose usage keys are camelCase and carry
//! no Anthropic `type` (the state machine drops them),
//! - Bedrock binary EventStream passthrough, where nothing parses
//! the frames today.
//!
//! This module provides one [`PendingRecord`] (a request's
//! half-built ledger event plus the deferred-finalization contract:
//! **record only after the response is fully observed**, so
//! streaming token counts are real instead of zero) and the capture
//! tasks that feed it. Every capture path is a bounded, best-effort
//! tee off the client byte path — a stalled or hostile response can
//! delay telemetry, never the client.
//!
//! Best-effort has one deliberate edge: capture tasks are plain
//! `tokio::spawn`s with no shutdown drain, so requests still
//! in-flight when the process exits are dropped without recording.
//! Tracking them against shutdown would couple the byte path to
//! telemetry lifetime — the wrong trade for a stats ledger.
use std::sync::Arc;
use std::time::Instant;
use base64::Engine as _;
use serde_json::Value;
use super::ledger::{Ledger, RequestEvent, Usage};
/// Cap for buffered non-streaming response bodies awaiting a usage
/// parse. Bigger bodies finalize with zero usage (tokens/USD from
/// compression estimates only) rather than growing memory.
const JSON_CAPTURE_CAP: usize = 2 * 1024 * 1024;
/// Queue depth for every telemetry tee (capture bodies and the SSE
/// parser feeds in proxy/bedrock/vertex). Bounded so a stalled
/// parser task can never make the byte path wait: senders `try_send`
/// and drop on a full queue.
pub(crate) const TEE_QUEUE_DEPTH: usize = 256;
/// A request's ledger event, waiting for its response to finish.
///
/// Constructed when the request is forwarded; finalized exactly once
/// when the response is fully observed (stream closed / body ended /
/// transport failed). Latency is measured here so every lane reports
/// the same thing: time from forwarding decision to response fully
/// observed.
pub struct PendingRecord {
ledger: Arc<Ledger>,
event: RequestEvent,
started: Instant,
}
impl PendingRecord {
pub fn new(ledger: Arc<Ledger>, event: RequestEvent, started: Instant) -> Self {
Self {
ledger,
event,
started,
}
}
/// Mark the upstream outcome failed (non-2xx or transport error)
/// before finalization.
pub fn mark_failed(&mut self) {
self.event.failed = true;
}
/// Supply the model id the request side knows (path parameter,
/// or the buffered request body when compression is on).
pub fn set_model(&mut self, model: &str) {
self.event.set_model(model);
}
/// Supply the model id learned from the *response*. Recording is
/// not gated on the compression buffer (compression defaults
/// off), so on a pure passthrough request this is the only place
/// the model is known. Never overrides a request-side value.
pub fn set_model_if_unknown(&mut self, model: &str) {
if self.event.model_is_unknown() {
self.event.set_model(model);
}
}
/// Attach the compression dispatcher's before/after token
/// estimate and the strategies it applied.
pub fn set_compression(&mut self, before: u64, after: u64, transforms: Vec<String>) {
self.event.tokens_before = before;
self.event.tokens_after = after;
self.event.transforms = transforms;
}
/// Record with the captured usage. Consumes the record — each
/// request records exactly once.
pub fn finalize(mut self, usage: Usage) {
self.event.usage = usage;
self.event.latency_ms = self.started.elapsed().as_millis() as u64;
self.ledger.record(self.event);
}
/// Record a failure (zero usage, no savings — the ledger's
/// failed-path accounting).
pub fn finalize_failed(mut self) {
self.mark_failed();
self.finalize(Usage::default());
}
}
/// Build the pending ledger event for a request entering a handler
/// (`None` when `--stats` is off). One call site per lane keeps the
/// "constructed at handler entry so every exit records exactly once"
/// rule easy to audit.
pub fn start_pending(
state: &crate::proxy::AppState,
provider: &'static str,
model: &str,
request_id: &str,
) -> Option<PendingRecord> {
state.config.stats.then(|| {
PendingRecord::new(
state.stats.clone(),
RequestEvent::new(provider, model, request_id),
Instant::now(),
)
})
}
/// Finalize a pending record as failed on a proxy-side rejection
/// (bad path, missing credentials, signing failure, oversized body,
/// …). Sugar for handler early-return paths, so "every request that
/// enters a handler produces exactly one record" survives rejects: a
/// credentials outage must show in `/stats` as a failure spike, not
/// as zero traffic.
pub fn finalize_rejected(pending: Option<PendingRecord>) {
if let Some(p) = pending {
p.finalize_failed();
}
}
/// Which JSON body shape to expect when parsing a non-streaming
/// response for usage.
#[derive(Debug, Clone, Copy)]
pub enum ResponseShape {
Anthropic,
OpenAiChat,
OpenAiResponses,
/// Bedrock sync responses: InvokeModel bodies are Anthropic
/// snake_case, Converse bodies are camelCase — try both.
Bedrock,
}
/// Extract usage from a complete response body. `None` when the
/// body has no recognisable usage block (error envelopes, non-JSON).
fn usage_from_response_json(shape: ResponseShape, v: &Value) -> Option<Usage> {
usage_from_usage_block(shape, v.get("usage")?)
}
/// Parse a bare `usage` object (already extracted from its
/// envelope) — the SSE state machines hold usage in this form.
pub fn usage_from_usage_block(shape: ResponseShape, usage: &Value) -> Option<Usage> {
match shape {
ResponseShape::Anthropic => snake_usage(usage),
ResponseShape::OpenAiChat => openai_chat_usage(usage),
ResponseShape::OpenAiResponses => openai_responses_usage(usage),
ResponseShape::Bedrock => snake_usage(usage).or_else(|| camel_usage(usage)),
}
}
fn u64_at(v: &Value, key: &str) -> Option<u64> {
v.get(key).and_then(Value::as_u64)
}
/// Anthropic `usage` block (also Bedrock InvokeModel for Anthropic
/// models): `input_tokens` already EXCLUDES cache reads/writes.
fn snake_usage(usage: &Value) -> Option<Usage> {
let input = u64_at(usage, "input_tokens");
let output = u64_at(usage, "output_tokens");
if input.is_none() && output.is_none() {
return None;
}
Some(Usage {
input_tokens: input.unwrap_or(0),
output_tokens: output.unwrap_or(0),
cache_read_tokens: u64_at(usage, "cache_read_input_tokens").unwrap_or(0),
cache_write_tokens: u64_at(usage, "cache_creation_input_tokens").unwrap_or(0),
})
}
/// Bedrock Converse `usage` block: camelCase, and `inputTokens`
/// likewise excludes cache tokens.
///
/// The cache fields fall back to the `*InputTokenCount` spelling
/// after `*InputTokens`. Reading only one name would silently report
/// zero cache tokens against a response that uses the other, and a
/// miss costs one map lookup — cheap insurance on a field whose
/// whole job is to be believed.
fn camel_usage(usage: &Value) -> Option<Usage> {
let input = u64_at(usage, "inputTokens");
let output = u64_at(usage, "outputTokens");
if input.is_none() && output.is_none() {
return None;
}
Some(Usage {
input_tokens: input.unwrap_or(0),
output_tokens: output.unwrap_or(0),
cache_read_tokens: u64_at(usage, "cacheReadInputTokens")
.or_else(|| u64_at(usage, "cacheReadInputTokenCount"))
.unwrap_or(0),
cache_write_tokens: u64_at(usage, "cacheWriteInputTokens")
.or_else(|| u64_at(usage, "cacheWriteInputTokenCount"))
.unwrap_or(0),
})
}
/// OpenAI Chat Completions: `prompt_tokens` INCLUDES cached tokens —
/// normalise to the ledger's Anthropic semantics (input = uncached).
fn openai_chat_usage(usage: &Value) -> Option<Usage> {
let prompt = u64_at(usage, "prompt_tokens")?;
let cached = usage
.get("prompt_tokens_details")
.and_then(|d| u64_at(d, "cached_tokens"))
.unwrap_or(0)
.min(prompt);
Some(Usage {
input_tokens: prompt - cached,
output_tokens: u64_at(usage, "completion_tokens").unwrap_or(0),
cache_read_tokens: cached,
cache_write_tokens: 0,
})
}
/// OpenAI Responses: `input_tokens` INCLUDES cached tokens — same
/// normalisation.
fn openai_responses_usage(usage: &Value) -> Option<Usage> {
let input = u64_at(usage, "input_tokens")?;
let cached = usage
.get("input_tokens_details")
.and_then(|d| u64_at(d, "cached_tokens"))
.unwrap_or(0)
.min(input);
Some(Usage {
input_tokens: input - cached,
output_tokens: u64_at(usage, "output_tokens").unwrap_or(0),
cache_read_tokens: cached,
cache_write_tokens: 0,
})
}
/// Usage accumulated by the shared Anthropic SSE state machine.
pub fn usage_from_anthropic_state(state: &crate::sse::anthropic::AnthropicStreamState) -> Usage {
Usage {
input_tokens: state.usage.input_tokens,
output_tokens: state.usage.output_tokens,
cache_read_tokens: state.usage.cache_read_input_tokens,
cache_write_tokens: state.usage.cache_creation_input_tokens,
}
}
/// Fold one streaming frame's JSON payload into a usage accumulator.
/// Handles every Bedrock/Anthropic stream shape (fields are monotone
/// within a stream, so `max` merge is safe against repeats):
///
/// - Anthropic SSE events: `message_start.message.usage`,
/// `message_delta.usage` (snake_case),
/// - Converse-native frames: bare `usage` with camelCase keys
/// (`metadata` events — no Anthropic `type` field at all),
/// - Bedrock InvokeModel EventStream chunks: `{"bytes": "<base64>"}`
/// wrapping an Anthropic SSE event (recursed after decode).
pub fn merge_stream_usage(acc: &mut Usage, payload: &Value) {
merge_stream_usage_depth(acc, payload, 0);
}
/// Real Bedrock wraps exactly once (`{"bytes": <b64 of the Anthropic
/// event>}`); the recursion exists only for that unwrap. Cap it so a
/// crafted deeply-nested base64 payload can't blow the stack — a
/// stack overflow aborts the whole process, not just this task.
const MAX_BYTES_UNWRAP_DEPTH: u8 = 4;
fn merge_stream_usage_depth(acc: &mut Usage, payload: &Value, depth: u8) {
// InvokeModel EventStream chunk: base64-wrapped inner event.
if let Some(b64) = payload.get("bytes").and_then(Value::as_str) {
if depth >= MAX_BYTES_UNWRAP_DEPTH {
tracing::warn!(
event = "stats_capture_bytes_depth_exceeded",
depth = depth,
"nested {{\"bytes\": …}} wrapping exceeded the unwrap cap; \
ignoring the payload (real Bedrock wraps exactly once)"
);
return;
}
if let Ok(decoded) = base64::engine::general_purpose::STANDARD.decode(b64) {
if let Ok(inner) = serde_json::from_slice::<Value>(&decoded) {
merge_stream_usage_depth(acc, &inner, depth + 1);
}
}
return;
}
// message_start carries usage nested under `message`.
if let Some(u) = payload
.get("message")
.and_then(|m| m.get("usage"))
.and_then(snake_usage)
{
merge_max(acc, u);
}
// message_delta (and some terminal frames) carry a bare `usage`.
if let Some(usage) = payload.get("usage") {
if let Some(u) = snake_usage(usage).or_else(|| camel_usage(usage)) {
merge_max(acc, u);
}
}
}
/// Take the field-wise maximum of two usage snapshots (usage fields
/// are monotone within a stream, so `max` merge is repeat-safe).
pub(crate) fn merge_max(acc: &mut Usage, u: Usage) {
acc.input_tokens = acc.input_tokens.max(u.input_tokens);
acc.output_tokens = acc.output_tokens.max(u.output_tokens);
acc.cache_read_tokens = acc.cache_read_tokens.max(u.cache_read_tokens);
acc.cache_write_tokens = acc.cache_write_tokens.max(u.cache_write_tokens);
}
/// Bounded tee → JSON-body usage capture. Returns the sender to tee
/// response chunks into; finalizes `pending` when the channel
/// closes (i.e. the response body finished or the client hung up).
pub fn spawn_json_usage_capture(
shape: ResponseShape,
mut pending: PendingRecord,
) -> tokio::sync::mpsc::Sender<bytes::Bytes> {
let (tx, mut rx) = tokio::sync::mpsc::channel::<bytes::Bytes>(TEE_QUEUE_DEPTH);
tokio::spawn(async move {
let mut buf: Vec<u8> = Vec::new();
let mut truncated = false;
while let Some(chunk) = rx.recv().await {
if truncated {
continue; // keep draining so the tee never backpressures
}
if buf.len() + chunk.len() > JSON_CAPTURE_CAP {
// No silent fallbacks: the request still records, but
// its usage will read zero — say why, once.
tracing::warn!(
event = "stats_capture_body_too_large",
cap_bytes = JSON_CAPTURE_CAP,
"response body exceeded the usage-capture buffer; \
this request records with compression-side numbers \
only (tokens/USD from usage read as 0)"
);
truncated = true;
buf.clear();
continue;
}
buf.extend_from_slice(&chunk);
}
let mut usage = Usage::default();
if !truncated {
match serde_json::from_slice::<Value>(&buf) {
Ok(v) => {
if let Some(model) = v.get("model").and_then(Value::as_str) {
pending.set_model_if_unknown(model);
}
// A 2xx from an LLM endpoint carries a usage block
// in every shape we know. A miss means the wire
// format moved (or this isn't the body we think it
// is), and the request would silently record $0 —
// the phantom-zero this module exists to avoid.
// Same no-silent-fallbacks rule as the cap above.
match usage_from_response_json(shape, &v) {
Some(u) => usage = u,
None => tracing::warn!(
event = "stats_capture_no_usage_block",
shape = ?shape,
"response parsed as JSON but carried no recognisable \
usage block; recording this request with zero tokens \
and $0 spend"
),
}
}
Err(e) => tracing::warn!(
event = "stats_capture_body_not_json",
error = %e,
"response body did not parse as JSON; recording this request \
with zero tokens and $0 spend"
),
}
}
pending.finalize(usage);
});
tx
}
/// Bounded tee → Bedrock binary EventStream usage capture, for the
/// passthrough mode where no other parser sees the frames. CRC
/// validation is off — this is the telemetry side; enforcement
/// happens (or not, per config) on the byte path.
pub fn spawn_eventstream_usage_capture(
pending: PendingRecord,
) -> tokio::sync::mpsc::Sender<bytes::Bytes> {
use crate::bedrock::eventstream::{CrcValidation, EventStreamParser};
let (tx, mut rx) = tokio::sync::mpsc::channel::<bytes::Bytes>(TEE_QUEUE_DEPTH);
tokio::spawn(async move {
let mut parser = EventStreamParser::new().with_crc_validation(CrcValidation::No);
let mut usage = Usage::default();
'outer: while let Some(chunk) = rx.recv().await {
parser.push(&chunk);
loop {
match parser.next_message() {
Ok(Some(msg)) => {
if let Ok(payload) = serde_json::from_slice::<Value>(&msg.payload) {
merge_stream_usage(&mut usage, &payload);
}
}
Ok(None) => break,
Err(_) => {
// Telemetry parser out of sync — stop and
// record what accumulated so far. The channel
// is simply dropped; the tee's `try_send` on a
// closed channel is a no-op, so the byte path
// never notices.
break 'outer;
}
}
}
}
pending.finalize(usage);
});
tx
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn anthropic_json_usage_parses() {
let v = json!({"usage": {"input_tokens": 10, "output_tokens": 5,
"cache_read_input_tokens": 7, "cache_creation_input_tokens": 3}});
let u = usage_from_response_json(ResponseShape::Anthropic, &v).unwrap();
assert_eq!(
u,
Usage {
input_tokens: 10,
output_tokens: 5,
cache_read_tokens: 7,
cache_write_tokens: 3
}
);
}
#[test]
fn openai_chat_normalises_cached_out_of_prompt() {
let v = json!({"usage": {"prompt_tokens": 100, "completion_tokens": 20,
"prompt_tokens_details": {"cached_tokens": 60}}});
let u = usage_from_response_json(ResponseShape::OpenAiChat, &v).unwrap();
assert_eq!(u.input_tokens, 40, "prompt minus cached");
assert_eq!(u.cache_read_tokens, 60);
assert_eq!(u.output_tokens, 20);
// Pathological cached > prompt clamps instead of underflowing.
let v = json!({"usage": {"prompt_tokens": 10,
"prompt_tokens_details": {"cached_tokens": 60}}});
let u = usage_from_response_json(ResponseShape::OpenAiChat, &v).unwrap();
assert_eq!(u.input_tokens, 0);
assert_eq!(u.cache_read_tokens, 10);
}
#[test]
fn openai_responses_normalises_cached_out_of_input() {
let v = json!({"usage": {"input_tokens": 80, "output_tokens": 9,
"input_tokens_details": {"cached_tokens": 50}}});
let u = usage_from_response_json(ResponseShape::OpenAiResponses, &v).unwrap();
assert_eq!(u.input_tokens, 30);
assert_eq!(u.cache_read_tokens, 50);
}
#[test]
fn bedrock_shape_accepts_snake_and_camel() {
let snake = json!({"usage": {"input_tokens": 10, "output_tokens": 5}});
let camel = json!({"usage": {"inputTokens": 11, "outputTokens": 6,
"cacheReadInputTokens": 4, "cacheWriteInputTokens": 2}});
assert_eq!(
usage_from_response_json(ResponseShape::Bedrock, &snake)
.unwrap()
.input_tokens,
10
);
let u = usage_from_response_json(ResponseShape::Bedrock, &camel).unwrap();
assert_eq!(u.input_tokens, 11);
assert_eq!(u.cache_read_tokens, 4);
assert_eq!(u.cache_write_tokens, 2);
}
/// The cache fields accept the `*InputTokenCount` spelling too.
/// Untested, this fallback reads as dead code and invites
/// deletion — at which point any response using that spelling
/// silently reports zero cache tokens.
#[test]
fn bedrock_camel_usage_accepts_token_count_suffix() {
let v = json!({"usage": {"inputTokens": 11, "outputTokens": 6,
"cacheReadInputTokenCount": 4, "cacheWriteInputTokenCount": 2}});
let u = usage_from_response_json(ResponseShape::Bedrock, &v).unwrap();
assert_eq!(u.cache_read_tokens, 4);
assert_eq!(u.cache_write_tokens, 2);
// The unsuffixed spelling wins when both are present.
let both = json!({"usage": {"inputTokens": 11, "outputTokens": 6,
"cacheReadInputTokens": 9, "cacheReadInputTokenCount": 4}});
let u = usage_from_response_json(ResponseShape::Bedrock, &both).unwrap();
assert_eq!(u.cache_read_tokens, 9);
}
#[test]
fn error_envelope_yields_none() {
let v = json!({"error": {"type": "overloaded_error"}});
assert!(usage_from_response_json(ResponseShape::Anthropic, &v).is_none());
let v = json!({"usage": {"totalTokens": 5}});
assert!(usage_from_response_json(ResponseShape::Bedrock, &v).is_none());
}
#[test]
fn stream_merge_handles_all_three_wire_shapes() {
let mut acc = Usage::default();
// 1. Anthropic message_start (nested under message).
merge_stream_usage(
&mut acc,
&json!({"type": "message_start",
"message": {"usage": {"input_tokens": 25, "output_tokens": 1,
"cache_read_input_tokens": 9}}}),
);
// 2. Anthropic message_delta (bare usage, growing output).
merge_stream_usage(
&mut acc,
&json!({"type": "message_delta", "usage": {"output_tokens": 42}}),
);
// 3. Converse metadata (camelCase, no `type`).
merge_stream_usage(
&mut acc,
&json!({"usage": {"inputTokens": 25, "outputTokens": 42,
"cacheWriteInputTokens": 3}}),
);
assert_eq!(acc.input_tokens, 25);
assert_eq!(acc.output_tokens, 42);
assert_eq!(acc.cache_read_tokens, 9);
assert_eq!(acc.cache_write_tokens, 3);
}
/// A crafted payload nesting `{"bytes": …}` inside itself must
/// hit the unwrap cap instead of recursing until the stack blows
/// (a stack overflow aborts the whole process). Real Bedrock
/// wraps exactly once.
#[test]
fn stream_merge_bounds_nested_base64_depth() {
let engine = &base64::engine::general_purpose::STANDARD;
// usage buried 10 wraps deep — far past the cap.
let mut v = json!({"type": "message_delta", "usage": {"output_tokens": 99}});
for _ in 0..10 {
v = json!({"bytes": engine.encode(v.to_string())});
}
let mut acc = Usage::default();
merge_stream_usage(&mut acc, &v); // must return, not overflow
assert_eq!(
acc.output_tokens, 0,
"usage beyond the unwrap cap is ignored, not chased"
);
// …while the real single-wrap shape still works.
let single = json!({"bytes": engine.encode(
json!({"type": "message_delta", "usage": {"output_tokens": 7}}).to_string()
)});
merge_stream_usage(&mut acc, &single);
assert_eq!(acc.output_tokens, 7);
}
#[test]
fn stream_merge_decodes_invoke_model_base64_chunks() {
let inner = json!({"type": "message_delta",
"usage": {"output_tokens": 17, "input_tokens": 12}});
let b64 =
base64::engine::general_purpose::STANDARD.encode(serde_json::to_vec(&inner).unwrap());
let mut acc = Usage::default();
merge_stream_usage(&mut acc, &json!({"bytes": b64, "p": "pad"}));
assert_eq!(acc.output_tokens, 17);
assert_eq!(acc.input_tokens, 12);
// Garbage base64 is ignored, never panics.
merge_stream_usage(&mut acc, &json!({"bytes": "!!!not-base64!!!"}));
assert_eq!(acc.output_tokens, 17);
}
}
@@ -0,0 +1,722 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Headroom Proxy — Savings</title>
<!-- Inline icon: keeps the page fully self-contained (the proxy's
/favicon.ico route serves a generic icon; this one is the
dashboard's own). -->
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Crect width='16' height='16' rx='3' fill='%232a78d6'/%3E%3Cpath d='M4 11.5V4.5M8 11.5V7M12 11.5V6' stroke='white' stroke-width='1.8' stroke-linecap='round'/%3E%3C/svg%3E">
<style>
:root {
color-scheme: light;
--page: #f9f9f7;
--surface: #fcfcfb;
--ink: #0b0b0b;
--ink-2: #52514e;
--muted: #898781;
--grid: #e1e0d9;
--baseline: #c3c2b7;
--border: rgba(11, 11, 11, 0.10);
--series-1: #2a78d6; /* spent (also anthropic/bedrock keys) */
--series-2: #008300; /* saved (also openai/vertex keys) */
--good: #0ca30c;
--good-text: #006300;
--critical: #d03b3b;
--wash: rgba(11, 11, 11, 0.04);
--shadow: rgba(0, 0, 0, 0.18);
}
@media (prefers-color-scheme: dark) {
:root {
color-scheme: dark;
--page: #0d0d0d;
--surface: #1a1a19;
--ink: #ffffff;
--ink-2: #c3c2b7;
--muted: #898781;
--grid: #2c2c2a;
--baseline: #383835;
--border: rgba(255, 255, 255, 0.10);
--series-1: #3987e5;
--series-2: #008300;
--good-text: #0ca30c;
/* #d03b3b is 3.6:1 on this surface — under AA for text, and
--critical IS text here (the "✕ N failed" badge), not a dot.
Lightened to clear 4.5:1 on #1a1a19. */
--critical: #f26a6a;
--wash: rgba(255, 255, 255, 0.06);
--shadow: rgba(0, 0, 0, 0.6);
}
}
* { box-sizing: border-box; }
body {
margin: 0;
background: var(--page);
color: var(--ink);
font: 14px/1.45 system-ui, -apple-system, "Segoe UI", sans-serif;
}
.wrap { max-width: 1180px; margin: 0 auto; padding: 20px 24px 48px; }
header { display: flex; align-items: baseline; gap: 12px; flex-wrap: wrap; margin: 4px 0 16px; }
header h1 { font-size: 17px; font-weight: 650; margin: 0; }
header .meta { color: var(--muted); font-size: 12.5px; }
header .spacer { flex: 1; }
.dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; margin-right: 5px; vertical-align: 1px; }
.filters { display: flex; align-items: center; gap: 16px; flex-wrap: wrap; margin-bottom: 14px; }
.seg { display: inline-flex; border: 1px solid var(--border); border-radius: 8px; overflow: hidden; background: var(--surface); }
.seg button {
appearance: none; border: 0; background: transparent; color: var(--ink-2);
font: inherit; font-size: 12.5px; padding: 5px 12px; cursor: pointer;
}
.seg button:hover { background: var(--wash); }
.seg button[aria-pressed="true"] { color: var(--ink); font-weight: 650; background: var(--wash); }
.seg button[aria-pressed="true"]::before { content: "✓ "; font-weight: 700; }
.filters .flabel { color: var(--muted); font-size: 12px; }
.cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(215px, 1fr)); gap: 12px; margin-bottom: 14px; }
.card {
background: var(--surface); border: 1px solid var(--border);
border-radius: 10px; padding: 14px 16px; min-width: 0;
}
.card .label { color: var(--ink-2); font-size: 12.5px; }
.card .value { font-size: 26px; font-weight: 650; margin-top: 2px; }
.card .value.hero { font-size: 48px; line-height: 1.1; letter-spacing: -0.5px; }
.card .sub { color: var(--muted); font-size: 12px; margin-top: 2px; }
.card .sub .up { color: var(--good-text); }
.card .sub .bad { color: var(--critical); }
.card svg.spark { display: block; margin-top: 8px; }
section.panel {
background: var(--surface); border: 1px solid var(--border);
border-radius: 10px; padding: 16px; margin-bottom: 14px;
}
.panel h2 { font-size: 13.5px; font-weight: 650; margin: 0 0 2px; }
.panel .desc { color: var(--muted); font-size: 12px; margin: 0 0 10px; }
.legend { display: flex; gap: 16px; align-items: center; font-size: 12px; color: var(--ink-2); margin: 2px 0 8px; }
.legend .key { display: inline-flex; align-items: center; gap: 6px; }
.legend .swatch { width: 10px; height: 10px; border-radius: 3px; }
.chart-holder { position: relative; }
#chart { display: block; width: 100%; }
/* The hovered/focused column lifts so the mark visibly responds.
Bars are <path> (rounded data-end), so key off the class alone. */
#chart .bar { transition: filter 90ms ease-out; }
#chart g.col:hover .bar, #chart g.col:focus .bar { filter: brightness(1.15); }
/* Keyboard focus needs a visible indicator beyond the brightness
lift (WCAG 2.4.7): stroke the focused column's marks in ink. */
#chart g.col:focus { outline: none; }
#chart g.col:focus .bar { stroke: var(--ink); stroke-width: 1.5; }
@media (prefers-reduced-motion: reduce) { #chart .bar { transition: none; } }
.tooltip {
position: absolute; pointer-events: none; z-index: 5; display: none;
background: var(--surface); border: 1px solid var(--border); border-radius: 8px;
padding: 8px 10px; box-shadow: 0 4px 14px var(--shadow); font-size: 12px; min-width: 170px;
}
.tooltip .t-when { color: var(--muted); margin-bottom: 4px; }
.tooltip .t-row { display: flex; align-items: center; gap: 7px; margin-top: 2px; }
.tooltip .t-key { width: 12px; height: 0; border-top: 3px solid; border-radius: 2px; }
.tooltip .t-val { font-weight: 650; margin-left: auto; font-variant-numeric: tabular-nums; }
.tooltip .t-name { color: var(--ink-2); }
.grid2 { display: grid; grid-template-columns: 1fr 1.4fr; gap: 12px; }
@media (max-width: 900px) { .grid2 { grid-template-columns: 1fr; } }
table { width: 100%; border-collapse: collapse; font-size: 12.5px; }
th {
text-align: right; color: var(--muted); font-weight: 500; font-size: 11.5px;
padding: 4px 8px; border-bottom: 1px solid var(--grid); white-space: nowrap;
}
th:first-child, td:first-child { text-align: left; padding-left: 2px; }
td {
text-align: right; padding: 5px 8px; border-bottom: 1px solid var(--grid);
font-variant-numeric: tabular-nums; white-space: nowrap;
}
tr:last-child td { border-bottom: 0; }
tr:hover td { background: var(--wash); }
td.name { max-width: 260px; overflow: hidden; text-overflow: ellipsis; color: var(--ink); }
td .provider-key { display: inline-block; width: 8px; height: 8px; border-radius: 2px; margin-right: 7px; vertical-align: 0; }
.badge-fail { color: var(--critical); font-weight: 650; }
.empty { color: var(--muted); font-size: 12.5px; padding: 14px 2px; }
.footnote { color: var(--muted); font-size: 11.5px; margin-top: 8px; }
</style>
</head>
<body>
<div class="wrap">
<header>
<h1>Headroom Proxy — Savings</h1>
<span class="meta" id="meta-version"></span>
<span class="meta" id="meta-uptime"></span>
<span class="spacer"></span>
<span class="meta" id="meta-persist"></span>
<span class="meta" id="meta-updated"></span>
</header>
<div class="filters" role="toolbar" aria-label="Scope and history filters">
<span class="flabel">Scope</span>
<div class="seg" id="scope-seg">
<button data-scope="session" aria-pressed="true">Session</button>
<button data-scope="lifetime" aria-pressed="false">Lifetime</button>
</div>
<span class="flabel">History</span>
<div class="seg" id="bucket-seg">
<button data-bucket="hour" aria-pressed="false">48 hours</button>
<button data-bucket="day" aria-pressed="true">Daily</button>
<button data-bucket="week" aria-pressed="false">Weekly</button>
<button data-bucket="month" aria-pressed="false">Monthly</button>
</div>
</div>
<div class="cards" id="kpis"></div>
<section class="panel">
<h2>Spend vs savings over time</h2>
<p class="desc" id="chart-desc">Bar height is what this traffic would have cost without Headroom.</p>
<div class="legend" id="chart-legend"></div>
<div class="chart-holder">
<svg id="chart" height="230" role="img" aria-label="Savings over time"></svg>
<div class="tooltip" id="tooltip"></div>
</div>
<div class="empty" id="chart-empty" style="display:none">No traffic recorded yet.</div>
</section>
<div class="grid2">
<section class="panel">
<h2>By provider</h2>
<p class="desc">Requests routed through each lane (<span id="scope-label-1"></span>).</p>
<div id="providers"></div>
</section>
<section class="panel">
<h2>By model</h2>
<p class="desc">Top models by USD saved (lifetime).</p>
<div id="models"></div>
</section>
</div>
<section class="panel">
<h2>Recent requests</h2>
<p class="desc">Last requests through the proxy (newest first). Failed requests accrue no savings.</p>
<div id="feed"></div>
<p class="footnote">Savings are valued from the vendored LiteLLM price table; models missing from it record $0 (logged by the proxy).</p>
</section>
</div>
<script>
"use strict";
(() => {
const $ = (id) => document.getElementById(id);
const state = { scope: "session", bucket: "day", stats: null, series: null, events: null, seq: 0 };
// ── formatting ──────────────────────────────────────────────
const COMPACT = new Intl.NumberFormat("en-US", {
notation: "compact",
maximumFractionDigits: 1,
});
// Full digits below 10k — "9,999 tokens" reads better than "10K",
// and compact notation alone would abbreviate from 1,000.
const fmtInt = (n) => {
n = Number(n) || 0;
return Math.abs(n) >= 1e4 ? COMPACT.format(n) : n.toLocaleString("en-US");
};
// Spend spans many orders of magnitude here: a dev smoke-testing
// sees fractions of a cent, a fleet sees dollars. Fixed decimals
// collapse the small end to a useless "$0.0000", so sub-cent
// values fall back to significant digits.
const fmtUsd = (n) => {
n = Number(n) || 0;
if (n < 0) return "-" + fmtUsd(-n);
if (n === 0) return "$0";
if (n >= 1000) return "$" + (n / 1000).toFixed(1) + "K";
if (n >= 1) return "$" + n.toFixed(2);
if (n >= 0.01) return "$" + n.toFixed(3);
return "$" + n.toPrecision(2);
};
const fmtPct = (n) => (Number(n) || 0).toFixed(1) + "%";
const fmtUptime = (s) => {
s = Number(s) || 0;
if (s < 90) return s + "s";
if (s < 5400) return Math.round(s / 60) + "m";
if (s < 172800) return (s / 3600).toFixed(1) + "h";
return Math.round(s / 86400) + "d";
};
const el = (tag, cls, text) => {
const e = document.createElement(tag);
if (cls) e.className = cls;
if (text !== undefined) e.textContent = text; // untrusted-safe
return e;
};
// ── data ────────────────────────────────────────────────────
async function getJson(url) {
const r = await fetch(url, { cache: "no-store" });
if (!r.ok) throw new Error(url + " → " + r.status);
return r.json();
}
async function refresh() {
// Monotonic sequence guard: a bucket-toggle fetch racing a slow
// interval fetch must not let the stale response win.
const seq = ++state.seq;
// allSettled, not all: the three endpoints fail independently.
// /stats/events is loopback-only, so viewing this page from
// another host 404s it by design — that must not blank the spend
// panels that answered perfectly well.
const [stats, series, events] = await Promise.allSettled([
getJson("/stats"),
getJson("/stats/timeseries?bucket=" + state.bucket),
getJson("/stats/events?limit=30"),
]);
if (seq !== state.seq) return; // superseded by a newer refresh
if (stats.status === "fulfilled") state.stats = stats.value;
if (series.status === "fulfilled") state.series = series.value;
state.events = events.status === "fulfilled" ? events.value : null;
// The feed is the only endpoint whose 404 is a policy, not a
// fault — tell the reader which it was.
state.feedLocalOnly = events.status === "rejected" && /→ 404$/.test(events.reason.message);
if (stats.status === "fulfilled") {
render();
$("meta-updated").textContent = "updated " + new Date().toLocaleTimeString();
return;
}
$("meta-updated").textContent = "unreachable — retrying";
if (!state.stats) {
// Cold load with the proxy down: say so in the page body,
// not just the header corner.
const kpis = $("kpis");
kpis.textContent = "";
kpis.appendChild(el("div", "empty", "Waiting for the proxy at " + location.origin + " …"));
}
}
// ── filters ─────────────────────────────────────────────────
function wireSeg(segId, attr, onPick) {
$(segId).addEventListener("click", (ev) => {
const b = ev.target.closest("button"); if (!b) return;
for (const btn of $(segId).querySelectorAll("button"))
btn.setAttribute("aria-pressed", String(btn === b));
onPick(b.dataset[attr]);
});
}
wireSeg("scope-seg", "scope", (v) => { state.scope = v; render(); });
wireSeg("bucket-seg", "bucket", (v) => { state.bucket = v; refresh(); });
// ── header + KPIs ───────────────────────────────────────────
function scoped() {
const s = state.stats;
return state.scope === "lifetime" ? s.lifetime : s.session;
}
function renderHeader() {
const s = state.stats;
$("meta-version").textContent = "v" + (s.version || "?") + " · rust-native";
$("meta-uptime").textContent = "up " + fmtUptime(s.uptime_seconds);
const p = s.persistence || {};
const persist = $("meta-persist");
persist.textContent = "";
const dot = el("span", "dot");
dot.style.background = p.enabled && p.healthy ? "var(--good)" : (p.enabled ? "var(--critical)" : "var(--muted)");
persist.appendChild(dot);
persist.appendChild(document.createTextNode(
p.enabled ? (p.healthy ? "persisted" : "persistence error") : "memory only"
));
persist.title = p.path || "";
}
// Shared SVG element factory (sparklines + the main chart).
const SVG_NS = "http://www.w3.org/2000/svg";
function mk(tag, attrs) {
const e = document.createElementNS(SVG_NS, tag);
for (const [k, v] of Object.entries(attrs)) e.setAttribute(k, v);
return e;
}
function sparkline(points) {
const vals = points.slice(-12).map(
(p) => (Number(p.input_cost_usd) || 0) + (Number(p.output_cost_usd) || 0));
if (vals.length < 2) return null;
const w = 120, h = 28, max = Math.max(...vals, 1);
const step = w / (vals.length - 1);
const y = (v) => h - 3 - (v / max) * (h - 6);
const d = vals.map((v, i) => (i ? "L" : "M") + (i * step).toFixed(1) + " " + y(v).toFixed(1)).join(" ");
const svg = mk("svg", { class: "spark", width: w, height: h, "aria-hidden": "true" });
svg.appendChild(mk("path", {
d, fill: "none", stroke: "var(--muted)", "stroke-width": 2,
"stroke-linecap": "round", "stroke-linejoin": "round",
}));
svg.appendChild(mk("circle", {
cx: (vals.length - 1) * step, cy: y(vals[vals.length - 1]), r: 4,
fill: "var(--series-1)", stroke: "var(--surface)", "stroke-width": 2,
}));
return svg;
}
function tile({ label, value, hero, subParts, spark }) {
const card = el("div", "card");
card.appendChild(el("div", "label", label));
card.appendChild(el("div", "value" + (hero ? " hero" : ""), value));
if (subParts && subParts.length) {
const sub = el("div", "sub");
subParts.forEach((p, i) => {
if (i) sub.appendChild(document.createTextNode(" · "));
sub.appendChild(typeof p === "string" ? document.createTextNode(p) : p);
});
card.appendChild(sub);
}
if (spark) card.appendChild(spark);
return card;
}
function renderKpis() {
const t = scoped();
const hours = (state.seriesHourly && state.seriesHourly.points) || [];
const kpis = $("kpis");
kpis.textContent = "";
kpis.appendChild(tile({
label: "Saved (" + state.scope + ")",
value: fmtUsd(t.savings_usd),
hero: true,
subParts: [
fmtInt(t.tokens_saved) + " tokens removed",
el("span", "up", fmtPct(t.savings_percent) + " of would-be input"),
],
}));
kpis.appendChild(tile({
label: "Spend",
value: fmtUsd(t.total_cost_usd),
subParts: [
fmtUsd(t.input_cost_usd) + " in",
fmtUsd(t.output_cost_usd) + " out",
],
spark: hours.length ? sparkline(hours) : null,
}));
const failed = Number(t.failed) || 0;
kpis.appendChild(tile({
label: "Requests",
value: fmtInt(t.requests),
subParts: [
fmtInt(t.compressed) + " compressed",
failed ? el("span", "bad", "✕ " + fmtInt(failed) + " failed") : "0 failed",
],
}));
kpis.appendChild(tile({
label: "Cache reads",
value: fmtInt(t.cache_read_tokens),
subParts: [
fmtPct(t.cache_hit_percent) + " of input from cache",
fmtUsd(t.cache_savings_usd) + " net saved",
],
}));
$("scope-label-1").textContent = state.scope;
}
// ── the chart ───────────────────────────────────────────────
//
// Stacked part-to-whole: what you paid + what Headroom saved =
// what the traffic would have cost without it. Charting savings
// alone would leave the panel empty on the default config
// (compression is off by default), which is exactly the
// deployment that most wants to see its spend.
const SERIES = [
{
name: "Spent", color: "var(--series-1)",
usd: (p) => (Number(p.input_cost_usd) || 0) + (Number(p.output_cost_usd) || 0),
tok: (p) => (Number(p.input_tokens) || 0) + (Number(p.cache_read_tokens) || 0) +
(Number(p.cache_write_tokens) || 0),
},
{
name: "Saved", color: "var(--series-2)",
usd: (p) => (Number(p.compression_savings_usd) || 0) + (Number(p.cache_savings_usd) || 0),
tok: (p) => Number(p.tokens_saved) || 0,
},
];
function niceMax(v) {
if (v <= 0) return 1;
const exp = Math.pow(10, Math.floor(Math.log10(v)));
const m = v / exp;
return (m <= 1 ? 1 : m <= 2 ? 2 : m <= 5 ? 5 : 10) * exp;
}
function renderChart() {
const points = (state.series && state.series.points) || [];
const svg = $("chart");
svg.textContent = "";
const legend = $("chart-legend");
legend.textContent = "";
const hasTraffic = points.some((p) => (Number(p.requests) || 0) > 0);
$("chart-empty").style.display = hasTraffic ? "none" : "";
svg.style.display = hasTraffic ? "" : "none";
legend.style.display = hasTraffic ? "" : "none";
if (!hasTraffic) return;
// Nothing priced yet (e.g. every model missing from the price
// table)? Chart tokens rather than a row of flat $0 bars.
const usdTotal = points.reduce((a, p) => a + SERIES.reduce((b, s) => b + s.usd(p), 0), 0);
const mode = usdTotal > 0 ? "usd" : "tokens";
$("chart-desc").textContent = mode === "usd"
? "Bar height is what this traffic would have cost without Headroom."
: "Input tokens vs tokens saved (nothing priced yet — see the note below).";
const valOf = (p, s) => (mode === "usd" ? s.usd(p) : s.tok(p));
const fmtVal = mode === "usd" ? fmtUsd : fmtInt;
for (const s of SERIES) {
const k = el("span", "key");
const sw = el("span", "swatch"); sw.style.background = s.color;
k.appendChild(sw); k.appendChild(el("span", null, s.name));
legend.appendChild(k);
}
const W = Math.max(svg.clientWidth || 900, 320), H = 230;
// Left margin fits the widest tick: sub-cent USD renders with
// significant digits (e.g. "$0.000025"), which overflows a
// narrow gutter and clips the top label.
const m = { top: 10, right: 8, bottom: 24, left: 72 };
svg.setAttribute("viewBox", "0 0 " + W + " " + H);
const iw = W - m.left - m.right, ih = H - m.top - m.bottom;
const max = niceMax(Math.max(...points.map((p) => SERIES.reduce((a, s) => a + valOf(p, s), 0))));
// Gridlines + ticks (hairline, recessive; clean numbers).
// Token counts are integers, so quartering a niceMax of 10 would
// print "2.5 tokens". Drop to whole steps when the axis is small
// enough that quarters aren't whole.
const ticks = mode === "usd" || max % 4 === 0 ? 4 : Math.min(4, max);
for (let i = 0; i <= ticks; i++) {
const v = (max / ticks) * i, y = m.top + ih - (v / max) * ih;
if (i > 0) svg.appendChild(mk("line", { x1: m.left, x2: W - m.right, y1: y, y2: y, stroke: "var(--grid)", "stroke-width": 1 }));
const t = mk("text", { x: m.left - 8, y: y + 4, "text-anchor": "end", fill: "var(--muted)", "font-size": 11 });
t.textContent = mode === "usd" ? fmtUsd(v) : fmtInt(Math.round(v));
svg.appendChild(t);
}
svg.appendChild(mk("line", { x1: m.left, x2: W - m.right, y1: m.top + ih, y2: m.top + ih, stroke: "var(--baseline)", "stroke-width": 1 }));
const n = points.length;
const band = iw / n;
const barW = Math.min(24, Math.max(3, band * 0.6));
const gap = 2; // surface gap between stack segments
const tipEl = $("tooltip");
// Per-column keyboard stops are useful at hour/week/month density
// but hostile at daily density (up to 400 Tab presses to cross
// the chart). Past this cap, columns stay hoverable and the data
// remains reachable via /stats/timeseries; the svg's own
// aria-label summarizes the series.
const focusableColumns = n <= 90;
svg.setAttribute(
"aria-label",
"Spend vs savings, " + n + " " + state.bucket + " buckets" +
(focusableColumns ? "" : " — full data at /stats/timeseries?bucket=" + state.bucket)
);
points.forEach((p, i) => {
const x = m.left + i * band + (band - barW) / 2;
const colLabel = p.timestamp + ": " + fmtInt(p.requests) + " requests, " +
SERIES.map((s) => s.name + " " + fmtVal(valOf(p, s))).join(", ");
const colAttrs = { class: "col", role: "img", "aria-label": colLabel };
if (focusableColumns) colAttrs.tabindex = "0";
const col = mk("g", colAttrs);
let yCursor = m.top + ih;
const segs = [];
SERIES.forEach((s, si) => {
const v = valOf(p, s);
const hpx = (v / max) * ih;
if (hpx <= 0) return;
const isTop = si === SERIES.length - 1 ||
SERIES.slice(si + 1).every((s2) => valOf(p, s2) <= 0);
// Segment occupies [stackTop, bottom]; the 2px surface gap is
// shaved off THIS segment's top when another segment sits
// above it, so the sliver of background appears BETWEEN
// segments — never between the stack and the axis. The next
// segment still starts from the un-shaved stackTop.
const bottom = yCursor;
const stackTop = yCursor - hpx;
const y = stackTop + (isTop ? 0 : Math.min(gap, hpx - 1));
const drawH = Math.max(1, bottom - y);
const r = isTop ? Math.min(4, barW / 2, drawH) : 0;
// Rounded data-end (top) only; square at the baseline.
const path = "M" + x + " " + (y + drawH) +
" L" + x + " " + (y + r) +
" Q" + x + " " + y + " " + (x + r) + " " + y +
" L" + (x + barW - r) + " " + y +
" Q" + (x + barW) + " " + y + " " + (x + barW) + " " + (y + r) +
" L" + (x + barW) + " " + (y + drawH) + " Z";
segs.push(mk("path", { d: path, fill: s.color, class: "bar" }));
yCursor = stackTop;
});
segs.reverse().forEach((sEl) => col.appendChild(sEl));
// Hit target wider than the mark (full band, full height).
col.appendChild(mk("rect", { x: m.left + i * band, y: m.top, width: band, height: ih, fill: "transparent" }));
const show = () => {
tipEl.textContent = "";
tipEl.appendChild(el("div", "t-when", p.timestamp));
SERIES.forEach((s) => {
const row = el("div", "t-row");
const key = el("span", "t-key"); key.style.borderTopColor = s.color;
row.appendChild(key);
row.appendChild(el("span", "t-name", s.name));
row.appendChild(el("span", "t-val", fmtVal(valOf(p, s))));
tipEl.appendChild(row);
});
const meta = el("div", "t-when");
meta.style.marginTop = "5px";
meta.textContent = fmtInt(p.requests) + " req · " + fmtInt(p.tokens_saved) +
" tok saved" + (Number(p.failed) ? " · " + fmtInt(p.failed) + " failed" : "");
tipEl.appendChild(meta);
tipEl.style.display = "block";
const holder = svg.parentElement.getBoundingClientRect();
const frac = (m.left + i * band + band / 2) / W;
let left = frac * holder.width - tipEl.offsetWidth / 2;
left = Math.max(0, Math.min(left, holder.width - tipEl.offsetWidth));
tipEl.style.left = left + "px";
tipEl.style.top = "6px";
};
const hide = () => { tipEl.style.display = "none"; };
col.addEventListener("pointerenter", show);
col.addEventListener("pointerleave", hide);
col.addEventListener("focus", show);
col.addEventListener("blur", hide);
svg.appendChild(col);
// Sparse x labels: first, last, and every ~quarter.
if (i === 0 || i === n - 1 || (n > 6 && i % Math.ceil(n / 4) === 0)) {
const t = mk("text", {
x: m.left + i * band + band / 2, y: H - 7,
"text-anchor": "middle", fill: "var(--muted)", "font-size": 11,
});
t.textContent = shortLabel(p.timestamp);
svg.appendChild(t);
}
});
}
function shortLabel(ts) {
if (!ts) return "";
if (state.bucket === "hour") return ts.slice(11, 16) || ts;
if (state.bucket === "month") return ts;
return ts.slice(5); // MM-DD
}
// ── tables ──────────────────────────────────────────────────
function table(headers, rows, emptyMsg) {
if (!rows.length) return el("div", "empty", emptyMsg);
const t = el("table");
const tr = el("tr");
headers.forEach((h) => {
const th = el("th", null, h);
th.setAttribute("scope", "col");
tr.appendChild(th);
});
t.appendChild(tr);
rows.forEach((cells) => {
const r = el("tr");
cells.forEach((c) => r.appendChild(c));
t.appendChild(r);
});
return t;
}
const PROVIDER_COLOR = {
anthropic: "var(--series-1)", openai_chat: "var(--series-2)",
openai_responses: "var(--series-2)", bedrock: "var(--series-1)", vertex: "var(--series-2)",
};
function renderProviders() {
const s = state.stats;
const map = state.scope === "lifetime" ? s.lifetime_by_provider : s.session_by_provider;
const rows = Object.entries(map || {}).map(([name, t]) => {
const nameCell = el("td", "name");
nameCell.appendChild(el("span", null, name));
return [
nameCell,
el("td", null, fmtInt(t.requests)),
el("td", null, fmtInt(t.tokens_saved)),
el("td", null, fmtUsd(t.savings_usd)),
el("td", null, fmtInt(t.failed)),
];
});
const holder = $("providers");
holder.textContent = "";
holder.appendChild(table(
["Provider", "Requests", "Tok saved", "Saved", "Failed"],
rows, "No traffic in this scope yet."));
}
function renderModels() {
const s = state.stats;
const entries = Object.entries(s.lifetime_by_model || {})
.sort((a, b) => (b[1].savings_usd + b[1].total_cost_usd) - (a[1].savings_usd + a[1].total_cost_usd))
.slice(0, 10);
const rows = entries.map(([name, t]) => [
el("td", "name", name),
el("td", null, fmtInt(t.requests)),
el("td", null, fmtInt(t.tokens_saved)),
el("td", null, fmtInt(t.cache_read_tokens)),
el("td", null, fmtUsd(t.savings_usd)),
el("td", null, fmtUsd(t.total_cost_usd)),
]);
const holder = $("models");
holder.textContent = "";
holder.appendChild(table(
["Model", "Requests", "Tok saved", "Cache reads", "Saved", "Spend"],
rows, "No models recorded yet."));
}
function renderFeed() {
const events = (state.events && state.events.events) || [];
const rows = events.map((e) => {
const when = el("td", null, (e.timestamp || "").slice(11, 19));
const model = el("td", "name");
const pk = el("span", "provider-key");
pk.style.background = PROVIDER_COLOR[e.provider] || "var(--muted)";
model.appendChild(pk);
model.appendChild(el("span", null, (e.provider || "?") + " · " + (e.model || "?")));
model.title = e.request_id || "";
const status = el("td");
if (e.failed) status.appendChild(el("span", "badge-fail", "✕ failed"));
else status.appendChild(el("span", null, fmtPct(e.compression_percent)));
// Input column: the provider-billed (uncached) input is the
// number that always exists. Only show the compression
// before→after arrow when compression actually ran — otherwise
// it reads a meaningless "0 → 0".
const inputCell = Number(e.tokens_before) > 0
? el("td", null, fmtInt(e.tokens_before) + " → " + fmtInt(e.tokens_after))
: el("td", null, fmtInt(e.input_tokens));
return [
when, model,
inputCell,
el("td", null, fmtInt(e.tokens_saved)),
el("td", null, fmtInt(e.cache_read_tokens) + " / " + fmtInt(e.cache_write_tokens)),
el("td", null, fmtInt(e.output_tokens)),
el("td", null, fmtUsd((Number(e.compression_savings_usd) || 0) + (Number(e.cache_savings_usd) || 0))),
el("td", null, (Number(e.latency_ms) || 0) + "ms"),
status,
];
});
const holder = $("feed");
holder.textContent = "";
if (state.feedLocalOnly) {
// Not an error: the per-request feed is served to local callers
// only. Say which, so nobody debugs a working proxy.
holder.appendChild(el("div", "empty",
"The per-request feed is available to local callers only. " +
"Totals above cover all traffic; open this dashboard on the proxy host " +
"(or over a tunnel to it) to see individual requests."));
return;
}
holder.appendChild(table(
["Time", "Lane · model", "Input tok", "Saved", "Cache r/w", "Out tok", "Saved $", "Latency", "Compr %"],
rows, "No requests yet — point a client at this proxy and refresh."));
}
function render() {
if (!state.stats) return;
renderHeader();
renderKpis();
renderChart();
renderProviders();
renderModels();
renderFeed();
}
// Keep an hourly series around for the KPI sparkline regardless of
// the selected history bucket.
async function refreshHourly() {
try { state.seriesHourly = await getJson("/stats/timeseries?bucket=hour"); } catch (e) { /* keep last */ }
}
refreshHourly().then(refresh);
setInterval(refresh, 3000);
setInterval(refreshHourly, 15000);
window.addEventListener("resize", () => state.stats && renderChart());
})();
</script>
</body>
</html>
File diff suppressed because it is too large Load Diff
@@ -39,10 +39,14 @@
//! revisit if multi-process aggregation lands.
pub mod cache_hit_rate;
pub mod capture;
pub mod compression_ratio;
pub mod ledger;
pub mod metric_names;
pub mod pricing;
pub mod prometheus;
pub mod proxy_metrics;
pub mod stats_http;
pub use prometheus::{
handle_metrics, observe_bedrock_invoke_latency, record_bedrock_eventstream_message,
@@ -0,0 +1,414 @@
//! Per-token USD pricing — Phase H. Used to value spend and savings.
//!
//! Reads the LiteLLM price table already vendored for
//! [`crate::compression::model_limits`] — one embedded copy, two
//! consumers. No new data file, and no startup network dependency:
//! the table ships inside the binary and parses lazily on first
//! lookup.
//!
//! # Lookup discipline
//!
//! Lookup is exact-match over a small, deterministic candidate list
//! derived from the request's model id (lowercased, then: geo prefix
//! stripped, `:rev` suffix trimmed, provider path segment dropped).
//! We deliberately do NOT substring-scan the table: a short or
//! generic id must miss (and be logged) rather than misprice against
//! whichever stored id happens to contain it.
//!
//! **The unmodified id is always tried first**, because Bedrock
//! cross-region pricing is genuinely region-specific — e.g.
//! `eu.anthropic.claude-3-5-haiku-20241022-v1:0` bills at $0.25/M
//! against `anthropic.claude-3-5-haiku-20241022-v1:0`'s $0.80/M. Any
//! region the table tracks therefore gets its true price. The
//! geo-stripped candidate is only a *fallback* for a (region, model)
//! pair the table does not list, where the base price is a far
//! better estimate than $0 — approximate, but never silently
//! preferred over an exact regional entry.
//!
//! # Coverage
//!
//! The vendored snapshot tracks current model generations plus every
//! Bedrock/Vertex-prefixed id, but it does not carry every legacy
//! direct-API id (`claude-3-5-sonnet-20241022`, for one, is absent
//! while its Bedrock form is present). Those price at $0 with a WARN
//! until the table is refreshed — see the module note below.
//!
//! # Unknown models price at $0
//!
//! A miss returns `None`; the ledger then records $0 for every USD
//! field and this module logs one WARN naming the model. We
//! deliberately do NOT substitute a blended fallback rate: phantom
//! dollars on a cost dashboard are worse than an obvious zero, and
//! the WARN tells an operator exactly which id to add by refreshing
//! the vendored table (`scripts/refresh_model_limits.sh`).
use std::collections::HashMap;
use std::sync::{OnceLock, RwLock};
use crate::bedrock::vendor::strip_geo_prefix;
use crate::compression::model_limits::VENDORED_JSON;
/// Per-token USD prices for one model. All fields are $/token
/// (LiteLLM stores them that way natively — no per-million
/// conversion happens here).
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct ModelPrice {
pub input: f64,
pub output: f64,
pub cache_read: f64,
pub cache_write: f64,
}
impl ModelPrice {
/// USD saved by removing `tokens_saved` input tokens before they
/// were sent (compression savings).
pub fn compression_savings_usd(&self, tokens_saved: u64) -> f64 {
tokens_saved as f64 * self.input
}
/// NET USD saved by prompt caching for one request: the read
/// discount (`reads × (input − cache_read)`) minus the write
/// premium (`writes × (cache_write − input)` — cache writes bill
/// ABOVE list price, typically 1.25×). Reporting the gross read
/// discount alone overstates the benefit on write-heavy traffic,
/// which is exactly the phantom-dollars failure mode this module
/// exists to avoid. Floored at $0 per request: a warm-up turn
/// that only writes is an investment, not negative savings —
/// its real cost is already carried by [`Self::input_cost_usd`].
///
/// Known bias, stated because it's the honest thing to state:
/// the ledger sums these per-request values, and a floored
/// request contributes 0 rather than its true negative, so
/// `sum(max(0, net))` ≥ `max(0, sum(net))`. Aggregate cache
/// savings therefore skew slightly high. The gap is one write
/// premium per floored request, which in practice means the
/// first turn of a session: once a prefix is cached, later turns
/// read far more than they write and net positive well clear of
/// the floor (100k reads + 5k writes on Sonnet nets +$0.266, vs
/// a $0.004 premium). Fixing it properly means accumulating a
/// signed net and flooring once at the aggregate — worth doing
/// if cache-heavy traffic ever makes the skew material, but not
/// worth a signed accumulator (and a persisted-negative
/// migration) for single-digit thousandths of a cent today.
pub fn cache_savings_usd(&self, cache_read_tokens: u64, cache_write_tokens: u64) -> f64 {
let read_discount = (self.input - self.cache_read).max(0.0);
let write_premium = (self.cache_write - self.input).max(0.0);
let net =
cache_read_tokens as f64 * read_discount - cache_write_tokens as f64 * write_premium;
net.max(0.0)
}
/// USD spent on generated output for a request. Output bills at
/// its own rate — typically 4-5x input — so a "spend" figure that
/// omits it understates the bill by more than it reports on a
/// short-prompt/long-answer turn.
pub fn output_cost_usd(&self, output_tokens: u64) -> f64 {
output_tokens as f64 * self.output
}
/// USD actually spent on input for a request, using the cache
/// breakdown when any segment is non-zero (never adding the
/// total on top of the breakdown — that would double-count).
pub fn input_cost_usd(
&self,
uncached_input_tokens: u64,
cache_read_tokens: u64,
cache_write_tokens: u64,
) -> f64 {
uncached_input_tokens as f64 * self.input
+ cache_read_tokens as f64 * self.cache_read
+ cache_write_tokens as f64 * self.cache_write
}
}
static BOOK: OnceLock<HashMap<String, ModelPrice>> = OnceLock::new();
/// Bounded once-per-model warn set so an unknown model logs one WARN,
/// not one per request. Capped so attacker-controlled model ids can't
/// grow it without bound; once full, further unknown models simply
/// stop logging (they still price at $0). RwLock so the steady-state
/// path for an already-known-unknown model (every miss after the
/// first) is a shared read, not serialized writes on the record path.
static WARNED: OnceLock<RwLock<std::collections::HashSet<String>>> = OnceLock::new();
const WARNED_CAP: usize = 256;
/// Parse the vendored table now instead of on the first recorded
/// request. Called once at startup (main.rs) so the ~1.4 MB JSON
/// parse never stalls tokio workers when a burst of first requests
/// lands together.
pub fn warm() {
let _ = book();
}
fn book() -> &'static HashMap<String, ModelPrice> {
BOOK.get_or_init(parse_vendored)
}
fn parse_vendored() -> HashMap<String, ModelPrice> {
let raw: serde_json::Value = serde_json::from_str(VENDORED_JSON)
.expect("vendored LiteLLM JSON must parse — same invariant as model_limits");
let mut out = HashMap::new();
let Some(obj) = raw.as_object() else {
return out;
};
for (model_id, spec) in obj {
if model_id == "sample_spec" {
continue;
}
let Some(spec) = spec.as_object() else {
continue;
};
let input = spec.get("input_cost_per_token").and_then(|v| v.as_f64());
let output = spec.get("output_cost_per_token").and_then(|v| v.as_f64());
let (Some(input), Some(output)) = (input, output) else {
continue;
};
if !input.is_finite() || !output.is_finite() || input < 0.0 || output < 0.0 {
continue;
}
let cache_read = spec
.get("cache_read_input_token_cost")
.and_then(|v| v.as_f64())
.filter(|c| c.is_finite() && *c >= 0.0)
.unwrap_or(input);
let cache_write = spec
.get("cache_creation_input_token_cost")
.and_then(|v| v.as_f64())
.filter(|c| c.is_finite() && *c >= 0.0)
.unwrap_or(input);
out.insert(
model_id.to_ascii_lowercase(),
ModelPrice {
input,
output,
cache_read,
cache_write,
},
);
}
out
}
/// Candidate keys tried, in order, for a request model id. Exact
/// matches only — see module doc for why there is no substring scan.
fn candidates(model: &str) -> Vec<String> {
let lower = model.trim().to_ascii_lowercase();
let mut out = Vec::with_capacity(6);
let mut push = |s: String| {
if !s.is_empty() && !out.contains(&s) {
out.push(s);
}
};
push(lower.clone());
// Bedrock cross-region profile: `eu.anthropic.claude…` falls back
// to `anthropic.claude…` when the region itself isn't in the
// table. GEO_PREFIXES (shared with the compression vendor gate)
// covers every geo the vendored table fronts, `us-gov.` included.
let geo = strip_geo_prefix(&lower).to_string();
push(geo.clone());
// Bedrock revision suffix: `…-v1:0`. ONLY a numeric suffix is
// trimmed. A non-numeric colon suffix is a routing/tier variant
// (`:exacto`, `:free`, `:nitro`) that can genuinely bill at its
// own rate — the vendored table lists
// `openrouter/z-ai/glm-4.6` at $0.40/M and its `:exacto` variant
// at $0.45/M. Trimming those would hand an uncatalogued variant
// its sibling's price with no warning: a confident wrong number,
// which this module exists to avoid. They miss and WARN instead.
push_revision_trimmed(&mut push, &geo);
push_revision_trimmed(&mut push, &lower);
// Provider-routed ids (`openai/gpt-4o`, `github-copilot/claude…`):
// the bare tail is how LiteLLM stores first-party models. Try the
// tail both verbatim and `:rev`-trimmed, so a combined
// `vendor/model:rev` id can still reach a fully-stripped entry.
if let Some((_, tail)) = lower.rsplit_once('/') {
push(tail.to_string());
push_revision_trimmed(&mut push, tail);
}
out
}
/// Push `id` minus a trailing `:<digits>` revision, if it has one.
fn push_revision_trimmed(push: &mut impl FnMut(String), id: &str) {
if let Some((head, rev)) = id.rsplit_once(':') {
if !rev.is_empty() && rev.bytes().all(|b| b.is_ascii_digit()) {
push(head.to_string());
}
}
}
/// Look up per-token pricing for a model id. `None` means "not in
/// the vendored table" — the caller records $0 for USD fields and
/// this module logs one WARN per distinct model id.
pub fn lookup(model: &str) -> Option<ModelPrice> {
let table = book();
for key in candidates(model) {
if let Some(p) = table.get(&key) {
return Some(*p);
}
}
warn_once(model);
None
}
fn warn_once(model: &str) {
let set = WARNED.get_or_init(|| RwLock::new(std::collections::HashSet::new()));
{
let read = set
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if read.len() >= WARNED_CAP || read.contains(model) {
return;
}
}
let mut write = set
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if write.len() >= WARNED_CAP || !write.insert(model.to_string()) {
return;
}
drop(write);
tracing::warn!(
event = "stats_price_unknown_model",
model = %model,
"model id not in the vendored price table; both spend AND savings \
for it record as $0 — refresh data/model_prices_and_context_window.json \
via scripts/refresh_model_limits.sh to add it"
);
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn known_anthropic_direct_id_prices() {
let p = lookup("claude-sonnet-4-5-20250929").expect("priced");
assert!(p.input > 0.0 && p.output > p.input);
assert!(p.cache_read < p.input, "cache reads are discounted");
assert!(p.cache_write > p.input, "cache writes carry a premium");
}
/// Bedrock cross-region pricing is region-specific, so an exact
/// regional entry must win over the geo-stripped base id.
#[test]
fn exact_regional_bedrock_entry_wins_over_base() {
let base = lookup("anthropic.claude-3-5-haiku-20241022-v1:0").expect("base priced");
let eu = lookup("eu.anthropic.claude-3-5-haiku-20241022-v1:0").expect("eu priced");
assert!(
(base.input - 8e-7).abs() < 1e-12,
"base id prices at list rate, got {}",
base.input
);
assert!(
(eu.input - 2.5e-7).abs() < 1e-12,
"eu id must use its own regional rate, got {}",
eu.input
);
assert_ne!(
base, eu,
"geo-stripping must not clobber a tracked regional price"
);
}
/// A region the table does NOT list falls back to the base id —
/// approximate, but far better than $0.
/// Every published geo prefix must fall back to the base entry
/// when the table doesn't carry that exact (region, model) pair.
/// Table-driven over GEO_PREFIXES itself, so adding a prefix
/// without its pricing fallback fails here — the `au.`/`jp.`/
/// `us-gov.` gap this PR fixes was exactly that.
#[test]
fn every_geo_prefix_falls_back_to_base_price() {
const BASE: &str = "anthropic.claude-3-5-haiku-20241022-v1:0";
let base = lookup(BASE).expect("base priced");
for geo in crate::bedrock::vendor::GEO_PREFIXES {
let id = format!("{geo}{BASE}");
if book().get(&id).is_some() {
continue; // table lists this region explicitly — priced on its own terms
}
assert_eq!(
lookup(&id).unwrap_or_else(|| panic!("{id} must fall back")),
base,
"{geo} must strip to the base entry when unlisted"
);
}
}
#[test]
fn provider_path_segment_falls_back_to_bare_tail() {
let bare = lookup("gpt-4o").expect("bare id priced");
let routed = lookup("openai/gpt-4o").expect("routed id priced");
assert_eq!(bare, routed);
}
#[test]
fn unknown_model_misses_instead_of_mispricing() {
assert!(lookup("claude").is_none(), "generic id must not match");
assert!(lookup("totally-unknown-model-xyz").is_none());
assert!(lookup("").is_none());
}
#[test]
fn case_insensitive_lookup() {
assert_eq!(
lookup("GPT-4o").expect("priced"),
lookup("gpt-4o").expect("priced")
);
}
/// A combined `vendor/model:rev` id must reach a table entry
/// stored fully stripped — the slash-tail is also tried with its
/// `:rev` suffix trimmed.
#[test]
fn provider_path_and_revision_compose() {
let bare = lookup("gpt-4o").expect("bare id priced");
let combined = lookup("openai/gpt-4o:2").expect("combined form priced");
assert_eq!(bare, combined, "provider path + numeric revision compose");
}
/// A non-numeric colon suffix is a pricing-relevant variant, not a
/// revision. It must NOT inherit the bare model's rate: the table
/// prices `openrouter/z-ai/glm-4.6` at $0.40/M and `:exacto` at
/// $0.45/M, so borrowing across that boundary invents dollars.
#[test]
fn variant_suffix_never_borrows_the_bare_models_price() {
let bare = lookup("openrouter/z-ai/glm-4.6").expect("bare variant priced");
let exacto = lookup("openrouter/z-ai/glm-4.6:exacto").expect("listed variant priced");
assert_ne!(bare.input, exacto.input, "fixture guard: these must differ");
// An UNLISTED variant must miss loudly, not silently take the
// bare price.
assert!(
lookup("openrouter/z-ai/glm-4.6:priority").is_none(),
"an uncatalogued variant must record $0 + WARN, never a guessed rate"
);
}
#[test]
fn usd_helpers_compute_list_price_math() {
let p = ModelPrice {
input: 3e-6,
output: 15e-6,
cache_read: 3e-7,
cache_write: 3.75e-6,
};
assert!((p.compression_savings_usd(1_000_000) - 3.0).abs() < 1e-9);
// Cache savings = read discount, NET of the write premium.
assert!((p.cache_savings_usd(1_000_000, 0) - 2.7).abs() < 1e-9);
// 1M reads save $2.70; 1M writes cost a $0.75 premium → net $1.95.
assert!((p.cache_savings_usd(1_000_000, 1_000_000) - 1.95).abs() < 1e-9);
// Write-only warm-up floors at $0, never negative savings
// (the premium itself is billed via input_cost_usd).
assert_eq!(p.cache_savings_usd(0, 1_000_000), 0.0);
// Breakdown input cost: uncached + cache_read + cache_write.
let usd = p.input_cost_usd(1_000_000, 1_000_000, 1_000_000);
assert!((usd - (3.0 + 0.3 + 3.75)).abs() < 1e-9);
// Inverted cache pricing clamps to $0, never negative savings.
let inverted = ModelPrice {
input: 1e-6,
output: 1e-6,
cache_read: 2e-6,
cache_write: 1e-6,
};
assert_eq!(inverted.cache_savings_usd(1000, 0), 0.0);
}
}
@@ -0,0 +1,216 @@
//! HTTP surface for the native savings ledger — Phase H:
//! `GET /stats`, `GET /stats/timeseries`, `GET /stats/events`, and
//! the embedded `GET /dashboard` page.
//!
//! All handlers are read-only snapshots over
//! [`super::ledger::Ledger`] — no handler can mutate accounting
//! state. Mounted by [`crate::proxy::build_app`] when
//! `Config::stats` is explicitly enabled; when off (the default), the paths fall through
//! to the catch-all forwarder like any other route.
//!
//! # Exposure: aggregate tier is open, per-request tier is local-only
//!
//! There is no auth here, and `--listen` defaults to `0.0.0.0:8787`,
//! so the payloads are split by sensitivity the same way the Python
//! proxy splits its own `/stats`:
//!
//! - **Aggregates** (spend, tokens, per-model/provider rollups,
//! history) are served to anyone, like the Prometheus `/metrics`
//! endpoint next door — same data class, same stance.
//! - **Per-request rows** (`recent_requests`, all of
//! `/stats/events`) and the ledger's filesystem path are served
//! only to local callers ([`caller_is_local`]). A request id plus a
//! model plus a timestamp is a different tier from a counter, and
//! `/metrics` never exposes it — so "same as /metrics" would be the
//! wrong argument for handing it to the open internet.
//!
//! `--stats=false` removes the routes entirely. Making the local-only
//! tier reachable from a trusted reverse proxy is tracked upstream in
//! headroomlabs-ai/headroom#1959, which is defining that policy for
//! the Python dashboard first; this surface should adopt whatever
//! lands there rather than invent a second scheme.
use std::net::{IpAddr, SocketAddr};
use axum::extract::{ConnectInfo, Query, State};
use axum::http::header::HOST;
use axum::http::{HeaderMap, StatusCode};
use axum::response::{Html, IntoResponse, Response};
use axum::Json;
use serde::Deserialize;
use serde_json::Value;
use crate::proxy::AppState;
/// True when the caller is genuinely local — gates the per-request
/// tier of `/stats` and the whole of `/stats/events`.
///
/// BOTH conditions are required, mirroring the Python proxy's
/// `loopback_guard`:
///
/// 1. the TCP peer is a loopback address, and
/// 2. the `Host` header names loopback.
///
/// (2) is not redundant with (1) — it is the DNS-rebinding defence. A
/// hostile page can rebind its own domain to `127.0.0.1`, so the
/// victim's browser connects to this proxy *from* loopback and (1)
/// passes, while `Host` still reads the attacker's name and their JS
/// reads the response. A real local tool always sends a loopback
/// `Host`; a rebound request does not.
pub(crate) fn caller_is_local(peer: SocketAddr, headers: &HeaderMap) -> bool {
peer.ip().is_loopback()
&& headers
.get(HOST)
.and_then(|h| h.to_str().ok())
.is_some_and(host_is_loopback)
}
/// `127.0.0.1`, `127.0.0.1:8787`, `[::1]:8787`, `localhost:8787`, …
fn host_is_loopback(host: &str) -> bool {
let host = host.trim();
// A bare IP with no port must be recognised before any port
// split: `::1` is all colons.
if let Ok(ip) = host.parse::<IpAddr>() {
return ip.is_loopback();
}
let name = match host.strip_prefix('[') {
Some(rest) => match rest.split_once(']') {
Some((inner, _)) => inner,
None => return false,
},
None => host.rsplit_once(':').map_or(host, |(h, _)| h),
};
name.eq_ignore_ascii_case("localhost")
|| name.parse::<IpAddr>().is_ok_and(|ip| ip.is_loopback())
}
/// The dashboard page, embedded at compile time so the binary is
/// self-contained (no template dir to deploy, no CDN dependency).
const DASHBOARD_HTML: &str = include_str!("dashboard.html");
pub async fn handle_stats(
State(state): State<AppState>,
ConnectInfo(peer): ConnectInfo<SocketAddr>,
headers: HeaderMap,
) -> Json<Value> {
Json(state.stats.stats_payload(caller_is_local(peer, &headers)))
}
#[derive(Deserialize)]
pub struct TimeseriesParams {
/// hour | day | week | month (default: day)
bucket: Option<String>,
}
pub async fn handle_timeseries(
State(state): State<AppState>,
Query(params): Query<TimeseriesParams>,
) -> Response {
let bucket = params.bucket.as_deref().unwrap_or("day");
match state.stats.timeseries_payload(bucket) {
Some(payload) => Json(payload).into_response(),
None => (
StatusCode::BAD_REQUEST,
// The Anthropic-style envelope the Bedrock lanes emit
// (`bedrock::invoke::error_response`). Note this is NOT
// universal: `ProxyError` renders plain text, and axum's
// `Query` extractor rejects a malformed `?bucket=` with
// its own plain-text 400 before this handler runs — so a
// client can't assume JSON for every 4xx from this path.
Json(serde_json::json!({
"error": {
"type": "unknown_bucket",
"message": "bucket must be one of: hour, day, week, month",
}
})),
)
.into_response(),
}
}
#[derive(Deserialize)]
pub struct EventsParams {
limit: Option<usize>,
}
/// This endpoint is per-request metadata end to end — there is no
/// aggregate tier to hand a network caller, so the whole thing is
/// gated. 404 rather than 403, matching the Python guard's stance:
/// keep the surface invisible to scanners rather than merely refuse
/// them.
pub async fn handle_events(
State(state): State<AppState>,
ConnectInfo(peer): ConnectInfo<SocketAddr>,
headers: HeaderMap,
Query(params): Query<EventsParams>,
) -> Response {
if !caller_is_local(peer, &headers) {
return (StatusCode::NOT_FOUND, "not found").into_response();
}
Json(state.stats.events_payload(params.limit.unwrap_or(50))).into_response()
}
/// `Html` already sets `text/html; charset=utf-8`.
pub async fn handle_dashboard() -> Html<&'static str> {
Html(DASHBOARD_HTML)
}
#[cfg(test)]
mod tests {
use super::*;
fn headers_with_host(host: &str) -> HeaderMap {
let mut h = HeaderMap::new();
h.insert(HOST, host.parse().unwrap());
h
}
fn peer(addr: &str) -> SocketAddr {
addr.parse().unwrap()
}
#[test]
fn local_caller_needs_loopback_peer_and_loopback_host() {
for host in [
"127.0.0.1:8787",
"127.0.0.1",
"localhost:8787",
"LOCALHOST",
"[::1]:8787",
"::1",
"127.5.6.7:8787", // the whole 127/8 block is loopback
] {
assert!(
caller_is_local(peer("127.0.0.1:5555"), &headers_with_host(host)),
"{host} should read as local"
);
}
}
/// The DNS-rebinding case, and the entire reason the `Host` check
/// exists: the victim's browser really is on loopback, so the peer
/// check alone would wave this through while attacker JS reads the
/// per-request feed.
#[test]
fn loopback_peer_with_foreign_host_is_not_local() {
for host in ["attacker.com", "attacker.com:8787", "evil.localhost.dev"] {
assert!(
!caller_is_local(peer("127.0.0.1:5555"), &headers_with_host(host)),
"{host} must not read as local — DNS rebinding"
);
}
}
#[test]
fn remote_peer_is_never_local_even_claiming_a_loopback_host() {
assert!(!caller_is_local(
peer("10.1.2.3:5555"),
&headers_with_host("127.0.0.1:8787")
));
}
#[test]
fn missing_host_header_fails_closed() {
assert!(!caller_is_local(peer("127.0.0.1:5555"), &HeaderMap::new()));
}
}
+227 -14
View File
@@ -26,6 +26,7 @@ use crate::config::Config;
use crate::error::ProxyError;
use crate::headers::{build_forward_request_headers, filter_response_headers};
use crate::health::{healthz, healthz_upstream, rollout_status};
use crate::observability::{capture, ledger};
use crate::websocket::ws_handler;
// Phase F PR-F1: imported as `classify_auth_mode` to make the call
// site self-documenting. `AuthMode` is re-exported under the same
@@ -80,6 +81,13 @@ pub struct AppState {
/// route hits `bearer()`. Tests override via
/// [`AppState::with_token_source`].
pub vertex_token_source: Arc<dyn crate::vertex::TokenSource>,
/// Native savings ledger (one per process). Loaded from
/// `Config::stats_path` when persistence is configured, else
/// in-memory. Handlers record into it after each finished
/// request; `/stats*` and `/dashboard` read from it. Recording
/// call sites gate on `Config::stats` — when stats are disabled
/// this is an inert in-memory ledger.
pub stats: Arc<ledger::Ledger>,
}
/// PR-E6: maximum number of sessions tracked by the drift detector
@@ -126,6 +134,15 @@ impl AppState {
let vertex_token_source: Arc<dyn crate::vertex::TokenSource> =
Arc::new(crate::vertex::adc::GcpAdcTokenSource::new());
let stats = if config.stats {
match config.stats_path.clone() {
Some(path) => Arc::new(ledger::Ledger::load(path)),
None => Arc::new(ledger::Ledger::in_memory()),
}
} else {
Arc::new(ledger::Ledger::in_memory())
};
Ok(Self {
config: Arc::new(config),
client,
@@ -135,6 +152,7 @@ impl AppState {
cache_stabilization::beta_sticky::BETA_TRACKER_CAPACITY,
),
vertex_token_source,
stats,
})
}
@@ -199,7 +217,39 @@ pub fn build_app(state: AppState) -> Router {
let mut router = Router::new()
.route("/healthz", get(healthz))
.route("/healthz/upstream", get(healthz_upstream))
.route("/rollout/status", get(rollout_status))
.route("/rollout/status", get(rollout_status));
// Native savings stats + dashboard. Served locally (never
// tunnelled upstream) so the Rust proxy is self-sufficient for
// savings observability. Gated on `--stats` so an operator can
// restore pure-passthrough behaviour for these paths.
if state.config.stats {
router = router
.route(
"/stats",
get(crate::observability::stats_http::handle_stats),
)
.route(
"/stats/timeseries",
get(crate::observability::stats_http::handle_timeseries),
)
.route(
"/stats/events",
get(crate::observability::stats_http::handle_events),
)
.route(
"/dashboard",
get(crate::observability::stats_http::handle_dashboard),
);
} else {
tracing::warn!(
event = "stats_disabled",
"native savings stats disabled by --stats=false; \
/stats*, /dashboard fall through to the upstream forwarder \
and no request telemetry is recorded"
);
}
router = router
// PR-C2: explicit POST route for /v1/chat/completions. The
// handler buffers the body and re-injects it into
// `forward_http`, which runs the OpenAI live-zone gate
@@ -614,10 +664,10 @@ pub(crate) async fn forward_http(
// controls what the buffered branch does (currently both `Off`
// and `LiveZone` passthrough); Phase B will branch on it inside
// `compress_anthropic_request`.
let should_intercept = state.config.compression
&& method == axum::http::Method::POST
let is_llm_json_post = method == axum::http::Method::POST
&& compression::is_compressible_path(uri.path())
&& is_application_json(req.headers());
let should_intercept = state.config.compression && is_llm_json_post;
// PR-E6: capture a header snapshot BEFORE the body is consumed so
// the drift detector can derive a per-session key from
@@ -635,6 +685,33 @@ pub(crate) async fn forward_http(
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())
.map_err(|e| ProxyError::InvalidHeader(e.to_string()))?;
// Savings ledger. Deliberately NOT gated on `should_intercept`:
// spend/caching stats are worth answering on a pure passthrough
// deployment too (compression savings are truthfully 0 there),
// so the gate is the LLM-endpoint shape alone and the model id
// is learned from the response when the request didn't carry it.
// Finalized exactly once, when the response is fully observed.
let should_record = state.config.stats && is_llm_json_post;
let mut pending_stats = should_record
.then(|| compression::classify_compressible_path(uri.path()))
.flatten()
.map(|endpoint| {
let provider = match endpoint {
compression::CompressibleEndpoint::AnthropicMessages => ledger::provider::ANTHROPIC,
compression::CompressibleEndpoint::OpenAiChatCompletions => {
ledger::provider::OPENAI_CHAT
}
compression::CompressibleEndpoint::OpenAiResponses => {
ledger::provider::OPENAI_RESPONSES
}
};
capture::PendingRecord::new(
state.stats.clone(),
ledger::RequestEvent::new(provider, "", &request_id),
start,
)
});
let upstream_resp = if should_intercept {
// Buffer up to `compression_max_body_bytes`. If the body
// exceeds this, the body is already partially consumed and
@@ -660,6 +737,9 @@ pub(crate) async fn forward_http(
"compression: Content-Length exceeds buffer limit; \
returning 413 without consuming body"
);
// A proxy-side rejection is still an outcome:
// record it as failed rather than dropping it.
capture::finalize_rejected(pending_stats.take());
return Err(ProxyError::PayloadTooLarge(format!(
"request Content-Length {len} exceeds compression \
buffer limit ({max} bytes)"
@@ -677,6 +757,7 @@ pub(crate) async fn forward_http(
"compression: body exceeds buffer limit; failing loudly (cannot \
resume streaming once the body has been partially consumed)"
);
capture::finalize_rejected(pending_stats.take());
return Err(ProxyError::PayloadTooLarge(format!(
"request body exceeds compression buffer limit ({max} bytes): {e}"
)));
@@ -735,7 +816,14 @@ pub(crate) async fn forward_http(
// enough that a canonical-bytes hash would compare apples
// to oranges. The volatile detector handles its own
// shape-dispatch via `ApiKind::from_endpoint`.
let mut stats_model: Option<String> = None;
if let Ok(parsed) = serde_json::from_slice::<serde_json::Value>(&buffered) {
// Model id for the savings ledger — read from the same
// sanitized bytes the dispatcher compresses.
stats_model = parsed
.get("model")
.and_then(|m| m.as_str())
.map(str::to_owned);
// PR-E5: volatile-content detector. Emits one WARN per
// finding (capped at 10) for content that busts cache
// (timestamps, UUIDs, ID-named fields).
@@ -807,6 +895,10 @@ pub(crate) async fn forward_http(
}
}
}
// Compression-side numbers for the savings ledger, captured
// out of the Compressed arm below (the outcome is consumed
// by the body-selection match).
let mut stats_compression: (u64, u64, Vec<String>) = (0, 0, Vec::new());
let outcome = match endpoint {
compression::CompressibleEndpoint::AnthropicMessages => {
// PR-E3: thread the F1-classified auth_mode into the
@@ -892,6 +984,11 @@ pub(crate) async fn forward_http(
markers_inserted,
per_strategy_tokens,
} => {
stats_compression = (
tokens_before as u64,
tokens_after as u64,
strategies_applied.iter().map(|s| s.to_string()).collect(),
);
tracing::info!(
request_id = %request_id,
path = %path_for_log,
@@ -1012,34 +1109,71 @@ pub(crate) async fn forward_http(
compression::CompressibleEndpoint::AnthropicMessages => body_to_send,
};
if let Some(p) = pending_stats.as_mut() {
// Request-side model (already `[1m]`-sanitized above, so
// it matches what upstream bills) wins over the response
// echo; the compression numbers are only knowable here.
if let Some(m) = stats_model.as_deref() {
p.set_model(m);
}
let (before, after, transforms) = stats_compression;
p.set_compression(before, after, transforms);
}
// Forward the (Phase A: identical) buffered bytes. reqwest
// sets its own Content-Length from the body bytes — the
// existing `build_forward_request_headers` already strips
// the client-supplied Content-Length for us.
state
match state
.client
.request(reqwest_method, upstream_url.clone())
.headers(outgoing_headers)
.body(body_to_send)
.send()
.await?
.await
{
Ok(resp) => resp,
Err(e) => {
// Transport/connect failures are request outcomes
// too — record them so outages are visible in /stats.
capture::finalize_rejected(pending_stats.take());
return Err(e.into());
}
}
} else {
// Pure streaming path — the original passthrough behaviour.
let body_stream =
TryStreamExt::map_err(req.into_body().into_data_stream(), std::io::Error::other);
let reqwest_body = reqwest::Body::wrap_stream(body_stream);
state
match state
.client
.request(reqwest_method, upstream_url.clone())
.headers(outgoing_headers)
.body(reqwest_body)
.send()
.await?
.await
{
Ok(resp) => resp,
Err(e) => {
// Same contract as the buffered arm.
capture::finalize_rejected(pending_stats.take());
return Err(e.into());
}
}
};
let upstream_status = upstream_resp.status();
let status = StatusCode::from_u16(upstream_status.as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
// Savings honesty rule: a non-2xx upstream is a failed request —
// it counts, but accrues no savings (the ledger enforces this on
// the failed flag).
if !upstream_status.is_success() {
if let Some(p) = pending_stats.as_mut() {
p.mark_failed();
}
}
// PR-A8 / P5-57: capture the upstream request id BEFORE we move
// `upstream_resp.headers()` into the response filter. Anthropic
// emits `request-id` (lowercase, no `x-`); OpenAI emits
@@ -1155,8 +1289,32 @@ pub(crate) async fn forward_http(
let parser_tx = if !matches!(sse_kind, SseStreamKind::None) {
let (tx, rx) = tokio::sync::mpsc::channel::<bytes::Bytes>(SSE_PARSER_QUEUE_DEPTH);
let rid_for_parser = request_id.clone();
tokio::spawn(run_sse_state_machine(sse_kind, rx, rid_for_parser));
// Deferred recording: the pending stats event rides into the
// state machine and is finalized when the stream closes, so
// output/cache token counts come from the real SSE usage
// frames instead of recording zero at header time.
tokio::spawn(run_sse_state_machine(
sse_kind,
rx,
rid_for_parser,
pending_stats.take(),
));
Some(tx)
} else if let Some(p) = pending_stats.take() {
// Non-SSE response on a recorded LLM path: tee the JSON body
// into a bounded buffer and parse usage when it completes.
let shape = match SseStreamKind::for_request_path(&path_for_log) {
SseStreamKind::OpenAiChat => capture::ResponseShape::OpenAiChat,
SseStreamKind::OpenAiResponses => capture::ResponseShape::OpenAiResponses,
// Anthropic (/v1/messages). Note: /v1/responses with the
// streaming state machine disabled still matches the
// OpenAiResponses arm above — for_request_path classifies
// by path, independent of the enable flag. An SSE body
// teed here simply fails the JSON parse and records
// compression-side numbers alone.
_ => capture::ResponseShape::Anthropic,
};
Some(capture::spawn_json_usage_capture(shape, p))
} else {
None
};
@@ -1221,11 +1379,9 @@ pub(crate) async fn forward_http(
}
/// Bound on the in-flight queue between the byte-passthrough and the
/// SSE state-machine task. Picked so that under steady-state streaming
/// load (~5 events/100ms typical) the parser is never blocked on
/// queue space, yet a stalled parser can't grow memory unboundedly.
/// Tunable via `proxy.toml` if a deployment finds this insufficient.
const SSE_PARSER_QUEUE_DEPTH: usize = 256;
/// SSE state-machine task — see [`TEE_QUEUE_DEPTH`] for the sizing
/// rationale it shares with every other telemetry tee.
use crate::observability::capture::TEE_QUEUE_DEPTH as SSE_PARSER_QUEUE_DEPTH;
/// Which provider's state machine should run on this stream. Picked
/// from the *request* path because the response content-type
@@ -1381,10 +1537,15 @@ fn maybe_inject_openai_prompt_cache_key(
/// Drive the per-provider state machine over a stream of byte chunks.
/// Lives in its own task; the byte path never waits on it.
///
/// `pending_stats` (when the savings ledger is recording this
/// request) is finalized here — after the stream closes — with the
/// usage the state machine accumulated.
async fn run_sse_state_machine(
kind: SseStreamKind,
mut rx: tokio::sync::mpsc::Receiver<bytes::Bytes>,
request_id: String,
pending_stats: Option<capture::PendingRecord>,
) {
use crate::sse::framing::SseFramer;
@@ -1455,6 +1616,20 @@ async fn run_sse_state_machine(
blocks = state.blocks.len(),
"sse stream closed"
);
if let Some(mut p) = pending_stats {
if let Some(model) = state.model.as_deref() {
p.set_model_if_unknown(model);
}
// A 200 OK stream can still fail in-band: Anthropic
// emits `{"type":"error"}` mid-stream (e.g. overloaded
// during generation) and the state machine lands in
// `Errored`. That is not a billable success — record
// it as failed so retries can't inflate savings.
if matches!(state.status, crate::sse::anthropic::StreamStatus::Errored) {
p.mark_failed();
}
p.finalize(capture::usage_from_anthropic_state(&state));
}
}
SseStreamKind::OpenAiChat => {
let mut state = crate::sse::openai_chat::ChunkState::new();
@@ -1558,6 +1733,19 @@ async fn run_sse_state_machine(
has_usage = state.usage.is_some(),
"sse stream closed"
);
if let Some(mut p) = pending_stats {
if let Some(model) = state.model.as_deref() {
p.set_model_if_unknown(model);
}
let usage = state
.usage
.as_ref()
.and_then(|u| {
capture::usage_from_usage_block(capture::ResponseShape::OpenAiChat, u)
})
.unwrap_or_default();
p.finalize(usage);
}
}
SseStreamKind::OpenAiResponses => {
let mut state = crate::sse::openai_responses::ResponseState::new();
@@ -1689,8 +1877,33 @@ async fn run_sse_state_machine(
incomplete_reason = state.incomplete_reason.as_deref().unwrap_or(""),
"sse stream closed"
);
if let Some(mut p) = pending_stats {
if let Some(model) = state.model.as_deref() {
p.set_model_if_unknown(model);
}
// `response.failed` / `response.incomplete` are not
// billable successes — count them as failures even
// though the HTTP status was 2xx.
if matches!(state.terminal_status(), Some("failed") | Some("incomplete")) {
p.mark_failed();
}
let usage = state
.usage
.as_ref()
.and_then(|u| {
capture::usage_from_usage_block(capture::ResponseShape::OpenAiResponses, u)
})
.unwrap_or_default();
p.finalize(usage);
}
}
SseStreamKind::None => {
if let Some(p) = pending_stats {
// No parser for this stream — record what the
// request side knew (compression estimates only).
p.finalize(ledger::Usage::default());
}
}
SseStreamKind::None => {}
}
}
+29
View File
@@ -124,6 +124,35 @@ impl SseFramer {
if chunk.is_empty() {
return;
}
// Cap the accumulator: a stream that never emits a blank-line
// terminator (a non-SSE body routed here, or a hostile
// upstream) would otherwise grow this buffer for the life of
// the stream. A real SSE event is a few KB; 4 MiB is orders
// of magnitude past any legitimate frame.
//
// On overflow we drop the buffer and the triggering chunk, so
// parsing resumes mid-stream at an arbitrary byte offset.
// Usually that garbage forms a prefix-less line the parser
// ignores; it CAN also resume on a syntactically valid
// `data:` line and yield one truncated event. That is
// acceptable only because every consumer is a telemetry tee
// off the byte path (`SseFramer` is never on the client
// path — see the three callers), so the worst case is a wrong
// usage number for one stream, bounded further by the
// ledger's own per-event token ceiling. Client bytes are
// never touched. Revisit this cap before framing anything
// client-facing with it.
const MAX_BUFFERED_BYTES: usize = 4 * 1024 * 1024;
if self.buf.len() + chunk.len() > MAX_BUFFERED_BYTES {
tracing::warn!(
event = "sse_framer_buffer_overflow",
buffered = self.buf.len(),
"no event terminator within 4 MiB; discarding buffered \
bytes (telemetry-only consumer — client stream unaffected)"
);
self.buf.clear();
return;
}
self.buf.extend_from_slice(chunk);
}
+16
View File
@@ -90,6 +90,7 @@ use axum::http::{HeaderMap, Method, StatusCode, Uri};
use axum::response::Response;
use std::net::SocketAddr;
use crate::observability::{capture, ledger};
use crate::proxy::AppState;
/// Single axum handler mounted at the
@@ -157,6 +158,15 @@ pub async fn handle_vertex_predict_dispatch(
segment = %model_action,
"vertex path final segment missing `:verb` separator"
);
// A proxy-side rejection is still an outcome for this
// request. The model id is whatever the caller sent (it
// is what failed to parse); the ledger clamps it.
capture::finalize_rejected(capture::start_pending(
&state,
ledger::provider::VERTEX,
&model_action,
&request_id,
));
return Response::builder()
.status(StatusCode::NOT_FOUND)
.body(Body::from("vertex path: bad model_action"))
@@ -173,6 +183,12 @@ pub async fn handle_vertex_predict_dispatch(
verb = %verb_str,
"vertex path verb not recognized; only rawPredict / streamRawPredict are supported"
);
capture::finalize_rejected(capture::start_pending(
&state,
ledger::provider::VERTEX,
model_id,
&request_id,
));
return Response::builder()
.status(StatusCode::NOT_FOUND)
.body(Body::from("vertex: unknown verb"))
@@ -36,6 +36,8 @@ use std::net::SocketAddr;
use crate::compression;
use crate::headers::{build_forward_request_headers, filter_response_headers};
use crate::observability::capture::TEE_QUEUE_DEPTH as VERTEX_SSE_QUEUE_DEPTH;
use crate::observability::{capture, ledger};
use crate::proxy::AppState;
use crate::vertex::{adc::TokenSourceError, envelope, VertexVerb};
@@ -76,6 +78,13 @@ pub(crate) async fn forward_vertex_request(
) -> Response {
let path_for_log = uri.path().to_string();
// Constructed at handler ENTRY so every exit — including
// proxy-side rejections (oversized body, envelope mismatch, ADC
// failure) — records exactly once. An ADC outage must show in
// /stats as a failure spike, not zero traffic.
let mut pending_stats =
capture::start_pending(&state, ledger::provider::VERTEX, &ctx.model_id, &request_id);
// ─── 1. BUFFER BODY ────────────────────────────────────────────────
let max = state.config.compression_max_body_bytes as usize;
let buffered = match to_bytes(body, max).await {
@@ -89,6 +98,7 @@ pub(crate) async fn forward_vertex_request(
error = %e,
"vertex request body exceeds compression buffer limit; failing loudly"
);
capture::finalize_rejected(pending_stats.take());
return error_response(
StatusCode::PAYLOAD_TOO_LARGE,
"request body exceeds buffer limit",
@@ -122,6 +132,7 @@ pub(crate) async fn forward_vertex_request(
error = %e,
"vertex envelope did not match expected shape; rejecting with 400"
);
capture::finalize_rejected(pending_stats.take());
return error_response(StatusCode::BAD_REQUEST, "vertex envelope invalid");
}
}
@@ -133,6 +144,7 @@ pub(crate) async fn forward_vertex_request(
// The dispatcher uses RawValue-based surgery so `anthropic_version`
// (and any other non-`messages` top-level field) round-trips
// byte-equal. Compression off → buffered bytes used unchanged.
let mut stats_compression: (u64, u64, Vec<String>) = (0, 0, Vec::new());
let body_to_send = if state.config.compression {
// PR-E3: Vertex uses GCP ADC bearer-token auth downstream, not
// Anthropic credentials, so the PAYG/OAuth/subscription
@@ -177,6 +189,11 @@ pub(crate) async fn forward_vertex_request(
markers = markers_inserted.len(),
"vertex live-zone compression applied"
);
stats_compression = (
tokens_before as u64,
tokens_after as u64,
strategies_applied.iter().map(|s| s.to_string()).collect(),
);
body
}
compression::Outcome::Passthrough { reason } => {
@@ -201,6 +218,11 @@ pub(crate) async fn forward_vertex_request(
buffered
};
if let Some(p) = pending_stats.as_mut() {
let (before, after, transforms) = stats_compression;
p.set_compression(before, after, transforms);
}
// ─── 4. RESOLVE BEARER TOKEN ───────────────────────────────────────
let bearer = match state.vertex_token_source.bearer().await {
Ok(t) => t,
@@ -221,6 +243,7 @@ pub(crate) async fn forward_vertex_request(
TokenSourceError::ProviderInit(_) => StatusCode::BAD_GATEWAY,
TokenSourceError::Fetch(_) => StatusCode::BAD_GATEWAY,
};
capture::finalize_rejected(pending_stats.take());
return error_response(status, "vertex ADC token fetch failed");
}
};
@@ -248,6 +271,7 @@ pub(crate) async fn forward_vertex_request(
error = %e,
"could not construct vertex upstream URL"
);
capture::finalize_rejected(pending_stats.take());
return error_response(StatusCode::BAD_GATEWAY, "vertex upstream URL build failed");
}
};
@@ -293,6 +317,7 @@ pub(crate) async fn forward_vertex_request(
error = %e,
"ADC bearer token contained invalid header bytes; refusing to forward"
);
capture::finalize_rejected(pending_stats.take());
return error_response(StatusCode::BAD_GATEWAY, "vertex auth header build failed");
}
}
@@ -308,9 +333,11 @@ pub(crate) async fn forward_vertex_request(
error = %e,
"could not convert axum method to reqwest method"
);
capture::finalize_rejected(pending_stats.take());
return error_response(StatusCode::BAD_REQUEST, "vertex method invalid");
}
};
let upstream_resp = match state
.client
.request(reqwest_method, upstream_url.clone())
@@ -321,6 +348,9 @@ pub(crate) async fn forward_vertex_request(
{
Ok(r) => r,
Err(e) => {
if let Some(p) = pending_stats.take() {
p.finalize_failed();
}
tracing::warn!(
event = "vertex_upstream_error",
request_id = %request_id,
@@ -335,6 +365,11 @@ pub(crate) async fn forward_vertex_request(
// ─── 8. STREAM RESPONSE ────────────────────────────────────────────
let upstream_status = upstream_resp.status();
let status = StatusCode::from_u16(upstream_status.as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
if !upstream_status.is_success() {
if let Some(p) = pending_stats.as_mut() {
p.mark_failed();
}
}
let resp_headers = filter_response_headers(upstream_resp.headers());
// PR-C1 reuse: when `attach_sse_tee` is set AND the upstream
@@ -355,7 +390,11 @@ pub(crate) async fn forward_vertex_request(
let parser_tx = if attach_sse_tee && is_sse {
let (tx, rx) = tokio::sync::mpsc::channel::<bytes::Bytes>(VERTEX_SSE_QUEUE_DEPTH);
let rid = request_id.clone();
tokio::spawn(run_anthropic_sse_state_machine(rx, rid));
tokio::spawn(run_anthropic_sse_state_machine(
rx,
rid,
pending_stats.take(),
));
tracing::info!(
event = "vertex_sse_tee_engaged",
request_id = %request_id,
@@ -363,7 +402,12 @@ pub(crate) async fn forward_vertex_request(
);
Some(tx)
} else {
None
// Non-SSE (rawPredict, or a JSON error on the streaming
// verb): tee the body into a bounded buffer and parse the
// Anthropic-shape usage block when it completes.
pending_stats
.take()
.map(|p| capture::spawn_json_usage_capture(capture::ResponseShape::Anthropic, p))
};
use futures_util::StreamExt as _;
@@ -434,13 +478,6 @@ fn error_response(status: StatusCode, msg: &'static str) -> Response {
.expect("static error response")
}
/// Bound on the in-flight queue between the byte-passthrough and the
/// SSE state-machine task. Mirrors the
/// `crate::proxy::SSE_PARSER_QUEUE_DEPTH` rationale (256 events ≈ 5
/// seconds of typical Anthropic streaming under the per-100ms event
/// rate; keeps memory bounded even if the parser stalls).
const VERTEX_SSE_QUEUE_DEPTH: usize = 256;
/// Drive the Anthropic SSE state machine over a stream of byte
/// chunks. Lives in its own spawned task; the byte path is fed via a
/// best-effort tee from [`forward_vertex_request`] and never blocks
@@ -448,6 +485,7 @@ const VERTEX_SSE_QUEUE_DEPTH: usize = 256;
async fn run_anthropic_sse_state_machine(
mut rx: tokio::sync::mpsc::Receiver<bytes::Bytes>,
request_id: String,
pending_stats: Option<capture::PendingRecord>,
) {
use crate::sse::framing::SseFramer;
let mut framer = SseFramer::new();
@@ -487,4 +525,12 @@ async fn run_anthropic_sse_state_machine(
blocks = state.blocks.len(),
"vertex sse stream closed"
);
if let Some(mut p) = pending_stats {
// In-band `{"type":"error"}` on a 200 OK stream is not a
// billable success (same rule as the direct Anthropic lane).
if matches!(state.status, crate::sse::anthropic::StreamStatus::Errored) {
p.mark_failed();
}
p.finalize(capture::usage_from_anthropic_state(&state));
}
}
+14
View File
@@ -111,3 +111,17 @@ pub fn install_static_token_source(mut state: AppState, bearer: &str) -> AppStat
pub fn _config_ref() -> Arc<Config> {
Arc::new(Config::for_test(Url::parse("http://127.0.0.1:1").unwrap()))
}
/// Static, obviously-fake AWS credentials for SigV4-signing tests.
/// Shared by every Bedrock test binary — the mock upstream never
/// validates the signature, only that one is present.
#[allow(dead_code)]
pub fn test_credentials() -> aws_credential_types::Credentials {
aws_credential_types::Credentials::new(
"AKIAEXAMPLEAKIDFORTEST",
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
None,
None,
"test",
)
}
@@ -20,14 +20,13 @@
mod common;
use aws_credential_types::Credentials;
use axum::body::Body;
use axum::extract::{Extension, State};
use axum::http::StatusCode;
use axum::routing::post;
use axum::Router;
use bytes::Bytes;
use common::start_proxy_with_state;
use common::{start_proxy_with_state, test_credentials};
use headroom_core::auth_mode::AuthMode;
use headroom_proxy::AppState;
use serde_json::{json, Value};
@@ -41,16 +40,6 @@ use wiremock::{Mock, MockServer, ResponseTemplate};
const TEST_MODEL: &str = "anthropic.claude-3-haiku-20240307-v1:0";
fn test_credentials() -> Credentials {
Credentials::new(
"AKIAEXAMPLEAKIDFORTEST",
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
None,
None,
"test",
)
}
#[derive(Default, Clone, Debug)]
struct CapturedRequest {
body: Option<Vec<u8>>,
@@ -35,8 +35,7 @@
mod common;
use aws_credential_types::Credentials;
use common::start_proxy_with_state;
use common::{start_proxy_with_state, test_credentials};
use serde_json::{json, Value};
use sha2::{Digest, Sha256};
use std::sync::{Arc, Mutex};
@@ -132,16 +131,6 @@ fn assert_byte_equal_sha256(inbound: &[u8], received: &[u8]) {
);
}
fn test_credentials() -> Credentials {
Credentials::new(
"AKIAEXAMPLEAKIDFORTEST",
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
None,
None,
"test",
)
}
/// Boot a proxy pointed at the wiremock upstream as the Bedrock
/// endpoint. The fake-upstream URL goes into `bedrock_endpoint`;
/// the regular `upstream` field is set to a sentinel because the
@@ -23,9 +23,8 @@
mod common;
use aws_credential_types::Credentials;
use bytes::{Bytes, BytesMut};
use common::start_proxy_with_state;
use common::{start_proxy_with_state, test_credentials};
use headroom_proxy::bedrock::MessageBuilder;
use serde_json::json;
use url::Url;
@@ -49,16 +48,6 @@ const TEST_REGION_LATENCY: &str = "us-test-latency-1";
const TEST_REGION_EVENTSTREAM: &str = "us-test-eventstream-1";
const TEST_REGION_SCRAPE: &str = "us-test-scrape-1";
fn test_credentials() -> Credentials {
Credentials::new(
"AKIAEXAMPLEAKIDFORTEST",
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
None,
None,
"test",
)
}
async fn bedrock_proxy_with_region(
upstream: &MockServer,
region: &str,
@@ -22,9 +22,8 @@
mod common;
use aws_credential_types::Credentials;
use bytes::{Bytes, BytesMut};
use common::start_proxy_with_state;
use common::{start_proxy_with_state, test_credentials};
use headroom_proxy::bedrock::{
parse_eventstream, CrcValidation, EventStreamParser, HeaderValue, MessageBuilder, ParseError,
};
@@ -36,16 +35,6 @@ use wiremock::{Mock, MockServer, ResponseTemplate};
const TEST_MODEL: &str = "anthropic.claude-3-haiku-20240307-v1:0";
fn test_credentials() -> Credentials {
Credentials::new(
"AKIAEXAMPLEAKIDFORTEST",
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
None,
None,
"test",
)
}
/// Synthesise a minimal Bedrock-shape Anthropic stream as binary
/// EventStream bytes. The exact JSON payload mirrors what real
/// Bedrock emits for a 2-token completion.
@@ -577,3 +566,77 @@ proptest! {
}
}
}
/// Regression (caught in review of the first stats PR, #1057; the fix
/// never landed with it): in SSE-translation mode, an unknown
/// `:event-type` (`TranslateOutcome::Skip`) arriving in the same
/// upstream chunk as known messages must not strand the known
/// messages in the parser buffer. The pre-fix drain shape pulled a
/// FRESH upstream chunk after a Skip, so when the Skip'd frame sat in
/// the final chunk, every complete message buffered behind it was
/// dropped and the client's stream ended early.
#[tokio::test]
async fn skip_in_final_chunk_does_not_strand_buffered_frames() {
// One chunk, three messages: known → unknown → known.
let mut buf = bytes::BytesMut::new();
let frame = |event_type: &str, payload: serde_json::Value| {
MessageBuilder::new()
.header_string(":event-type", event_type)
.header_string(":content-type", "application/json")
.header_string(":message-type", "event")
.payload(Bytes::from(serde_json::to_string(&payload).unwrap()))
.build()
};
buf.extend_from_slice(&frame(
"chunk",
serde_json::json!({"type": "message_start",
"message": {"id": "m1", "usage": {"input_tokens": 3}}}),
));
buf.extend_from_slice(&frame(
"somethingNewFromAws",
serde_json::json!({"novel": true}),
));
buf.extend_from_slice(&frame("chunk", serde_json::json!({"type": "message_stop"})));
let body = buf.freeze();
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.and(path(
"/model/anthropic.claude-3-haiku-20240307-v1:0/invoke-with-response-stream",
))
.respond_with(
ResponseTemplate::new(200)
.insert_header("content-type", "application/vnd.amazon.eventstream")
.set_body_raw(body.to_vec(), "application/vnd.amazon.eventstream"),
)
.mount(&upstream)
.await;
let proxy = bedrock_proxy(&upstream, |_| {}).await;
let resp = reqwest::Client::new()
.post(format!(
"{}/model/anthropic.claude-3-haiku-20240307-v1:0/invoke-with-response-stream",
proxy.url()
))
.header("content-type", "application/json")
.header("accept", "text/event-stream")
.json(&serde_json::json!({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 8,
"messages": [{"role": "user", "content": "hi"}]
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let text = resp.text().await.unwrap();
assert!(
text.contains("message_start"),
"frame before the Skip must arrive: {text}"
);
assert!(
text.contains("message_stop"),
"frame buffered BEHIND the Skip'd message must still arrive \
(pre-fix behaviour dropped it): {text}"
);
}
@@ -0,0 +1,911 @@
//! Integration tests for the native savings stats surface:
//! `/stats`, `/stats/timeseries`, `/stats/events`, `/dashboard`,
//! and the per-lane recording hooks (deferred SSE finalization,
//! non-streaming JSON usage capture, failed-request accounting,
//! Bedrock camelCase usage).
mod common;
use std::time::Duration;
use common::{
install_static_token_source, start_proxy_with, start_proxy_with_state, test_credentials,
};
use headroom_proxy::bedrock::MessageBuilder;
use serde_json::{json, Value};
use url::Url;
use wiremock::matchers::{method, path, path_regex};
use wiremock::{Mock, MockServer, ResponseTemplate};
/// The capture tasks finalize asynchronously after the client sees
/// the response — poll /stats until the predicate holds.
async fn wait_for_stats<F>(base: &str, pred: F) -> Value
where
F: Fn(&Value) -> bool,
{
let client = reqwest::Client::new();
for _ in 0..100 {
let v: Value = client
.get(format!("{base}/stats"))
.send()
.await
.expect("GET /stats")
.json()
.await
.expect("stats json");
if pred(&v) {
return v;
}
tokio::time::sleep(Duration::from_millis(50)).await;
}
panic!("stats predicate not satisfied within 5s");
}
/// POST a JSON body the way every recording test does.
async fn post_json(url: String, body: &Value) -> reqwest::Response {
reqwest::Client::new()
.post(url)
.header("content-type", "application/json")
.json(body)
.send()
.await
.expect("POST")
}
/// One binary EventStream frame with the standard event headers.
fn event_frame(event_type: &str, payload: &Value) -> bytes::Bytes {
MessageBuilder::new()
.header_string(":message-type", "event")
.header_string(":event-type", event_type)
.header_string(":content-type", "application/json")
.payload(payload.to_string().into())
.build()
}
#[tokio::test]
async fn stats_endpoints_serve_locally_and_validate_input() {
let upstream = MockServer::start().await;
let proxy = start_proxy_with(&upstream.uri(), |_| {}).await;
let client = reqwest::Client::new();
let stats: Value = client
.get(format!("{}/stats", proxy.url()))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(stats["proxy"], "headroom-rust");
assert_eq!(stats["requests"]["total"], 0);
// headway's unified-stats reader contract.
assert!(stats["summary"]["api_requests"].is_u64());
assert!(stats["tokens"]["proxy_compression_saved"].is_u64());
assert!(stats["requests"]["cached"].is_u64());
let ts: Value = client
.get(format!("{}/stats/timeseries?bucket=day", proxy.url()))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(ts["bucket"], "day");
let bad = client
.get(format!("{}/stats/timeseries?bucket=fortnight", proxy.url()))
.send()
.await
.unwrap();
assert_eq!(bad.status(), 400);
let events: Value = client
.get(format!("{}/stats/events", proxy.url()))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert!(events["events"].as_array().unwrap().is_empty());
let dash = client
.get(format!("{}/dashboard", proxy.url()))
.send()
.await
.unwrap();
assert_eq!(dash.status(), 200);
let ct = dash
.headers()
.get("content-type")
.unwrap()
.to_str()
.unwrap();
assert!(ct.starts_with("text/html"), "content-type: {ct}");
let body = dash.text().await.unwrap();
assert!(body.contains("Headroom Proxy — Savings"));
// Self-contained page: no network resource references of any
// kind. The only URLs allowed are XML namespaces and data: URIs.
for needle in [
"src=\"http",
"src=\"//",
"href=\"http",
"href=\"//",
"@import",
"fonts.googleapis",
"cdn.",
] {
assert!(
!body.contains(needle),
"dashboard must be self-contained; found {needle:?}"
);
}
}
#[tokio::test]
async fn stats_disabled_falls_through_to_upstream() {
let upstream = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/stats"))
.respond_with(ResponseTemplate::new(200).set_body_string("python-proxy-stats"))
.mount(&upstream)
.await;
let proxy = start_proxy_with(&upstream.uri(), |c| c.stats = false).await;
let body = reqwest::Client::new()
.get(format!("{}/stats", proxy.url()))
.send()
.await
.unwrap()
.text()
.await
.unwrap();
assert_eq!(
body, "python-proxy-stats",
"with --stats=false the path must tunnel upstream (pre-feature behaviour)"
);
}
#[tokio::test]
async fn anthropic_non_streaming_response_records_usage() {
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/messages"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"id": "msg_1",
"type": "message",
"model": "claude-sonnet-4-5-20250929",
"content": [{"type": "text", "text": "hi"}],
"usage": {
"input_tokens": 100,
"output_tokens": 25,
"cache_read_input_tokens": 40,
"cache_creation_input_tokens": 10
}
})))
.mount(&upstream)
.await;
// compression=true engages the buffered arm (mode stays Off —
// recording must work in pure-passthrough deployments too).
let proxy = start_proxy_with(&upstream.uri(), |c| c.compression = true).await;
let resp = post_json(
format!("{}/v1/messages", proxy.url()),
&json!({
"model": "claude-sonnet-4-5-20250929",
"max_tokens": 32,
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap(); // drain so the capture tee closes
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(stats["tokens"]["input"], 100);
assert_eq!(stats["tokens"]["output"], 25);
assert_eq!(stats["tokens"]["cache_read"], 40);
assert_eq!(stats["tokens"]["cache_write"], 10);
assert_eq!(stats["requests"]["failed"], 0);
// 40 cache-read tokens ⇒ this request counts as a cache hit.
assert_eq!(stats["requests"]["cached"], 1);
assert_eq!(stats["session_by_provider"]["anthropic"]["requests"], 1);
// 40 cache reads on sonnet pricing → non-zero cache savings.
assert!(stats["session"]["cache_savings_usd"].as_f64().unwrap() > 0.0);
assert!(stats["session"]["input_cost_usd"].as_f64().unwrap() > 0.0);
// Model attribution landed.
assert_eq!(
stats["lifetime_by_model"]["claude-sonnet-4-5-20250929"]["requests"],
1
);
// Timeseries picked it up.
let ts: Value = reqwest::Client::new()
.get(format!("{}/stats/timeseries?bucket=hour", proxy.url()))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(ts["points"][0]["requests"], 1);
}
/// Recording must not depend on the compression master switch —
/// it is OFF by default, and spend/cache observability is the whole
/// point of the feature on a passthrough deployment. The model is
/// learned from the response there (the request body is never
/// buffered), and compression savings are legitimately 0.
#[tokio::test]
async fn records_with_compression_off_using_model_from_response() {
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/messages"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"id": "msg_1",
"type": "message",
"model": "claude-sonnet-4-5-20250929",
"content": [{"type": "text", "text": "hi"}],
"usage": {"input_tokens": 200, "output_tokens": 10,
"cache_read_input_tokens": 80}
})))
.mount(&upstream)
.await;
// Config::for_test defaults compression to false — same as the
// shipped binary's default.
let proxy = start_proxy_with(&upstream.uri(), |c| {
assert!(!c.compression, "guard: this test covers compression OFF");
})
.await;
let resp = post_json(
format!("{}/v1/messages", proxy.url()),
&json!({
"model": "claude-sonnet-4-5-20250929",
"max_tokens": 32,
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap();
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(stats["tokens"]["input"], 200);
assert_eq!(stats["tokens"]["cache_read"], 80);
assert_eq!(stats["tokens"]["saved"], 0, "no compression ran → 0 saved");
// Model came from the response envelope, so spend is priced.
assert_eq!(
stats["lifetime_by_model"]["claude-sonnet-4-5-20250929"]["requests"], 1,
"model must be learned from the response when the request isn't buffered"
);
assert!(stats["session"]["input_cost_usd"].as_f64().unwrap() > 0.0);
assert!(stats["session"]["cache_savings_usd"].as_f64().unwrap() > 0.0);
}
#[tokio::test]
async fn anthropic_sse_stream_records_deferred_usage() {
let upstream = MockServer::start().await;
let sse_body = concat!(
"event: message_start\n",
"data: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_1\",\"usage\":",
"{\"input_tokens\":50,\"output_tokens\":1,\"cache_read_input_tokens\":12}}}\n\n",
"event: content_block_delta\n",
"data: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":",
"{\"type\":\"text_delta\",\"text\":\"hey\"}}\n\n",
"event: message_delta\n",
"data: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\"},",
"\"usage\":{\"output_tokens\":30}}\n\n",
"event: message_stop\n",
"data: {\"type\":\"message_stop\"}\n\n",
);
Mock::given(method("POST"))
.and(path("/v1/messages"))
.respond_with(
ResponseTemplate::new(200)
.insert_header("content-type", "text/event-stream")
.set_body_raw(sse_body.as_bytes().to_vec(), "text/event-stream"),
)
.mount(&upstream)
.await;
let proxy = start_proxy_with(&upstream.uri(), |c| c.compression = true).await;
let resp = post_json(
format!("{}/v1/messages", proxy.url()),
&json!({
"model": "claude-haiku-4-5-20251001",
"max_tokens": 32,
"stream": true,
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap();
// Deferred recording: counts come from the stream's usage
// frames, observed only after the stream closed.
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(stats["tokens"]["input"], 50);
assert_eq!(
stats["tokens"]["output"], 30,
"output from message_delta, not 0"
);
assert_eq!(stats["tokens"]["cache_read"], 12);
}
#[tokio::test]
async fn failed_upstream_counts_but_accrues_no_savings() {
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/messages"))
.respond_with(ResponseTemplate::new(529).set_body_json(json!({
"type": "error",
"error": {"type": "overloaded_error", "message": "overloaded"}
})))
.mount(&upstream)
.await;
let proxy = start_proxy_with(&upstream.uri(), |c| c.compression = true).await;
let resp = post_json(
format!("{}/v1/messages", proxy.url()),
&json!({
"model": "claude-sonnet-4-5-20250929",
"max_tokens": 32,
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 529);
let _ = resp.bytes().await.unwrap();
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(stats["requests"]["failed"], 1);
assert_eq!(stats["tokens"]["saved"], 0);
assert_eq!(stats["session"]["savings_usd"], 0.0);
assert_eq!(stats["session"]["input_cost_usd"], 0.0);
// Visible in the feed, flagged.
let events: Value = reqwest::Client::new()
.get(format!("{}/stats/events?limit=5", proxy.url()))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(events["events"][0]["failed"], true);
assert_eq!(events["events"][0]["tokens_saved"], 0);
}
#[tokio::test]
async fn openai_chat_response_normalises_cached_tokens() {
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/chat/completions"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"id": "chatcmpl-1",
"object": "chat.completion",
"model": "gpt-4o",
"choices": [{"index": 0, "message": {"role": "assistant", "content": "hi"},
"finish_reason": "stop"}],
"usage": {
"prompt_tokens": 90,
"completion_tokens": 12,
"prompt_tokens_details": {"cached_tokens": 60}
}
})))
.mount(&upstream)
.await;
let proxy = start_proxy_with(&upstream.uri(), |c| c.compression = true).await;
let resp = post_json(
format!("{}/v1/chat/completions", proxy.url()),
&json!({
"model": "gpt-4o",
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap();
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
// prompt_tokens INCLUDES cached — the ledger stores the split.
assert_eq!(stats["tokens"]["input"], 30);
assert_eq!(stats["tokens"]["cache_read"], 60);
assert_eq!(stats["tokens"]["output"], 12);
// Provider label matches the Prometheus vocabulary
// (`metric_names`): the chat lane is `openai_chat`, not `openai`.
assert_eq!(stats["session_by_provider"]["openai_chat"]["requests"], 1);
}
#[tokio::test]
async fn bedrock_converse_sync_records_camelcase_usage() {
let upstream = MockServer::start().await;
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
Mock::given(method("POST"))
.and(path(format!("/model/{model_id}/converse")))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"output": {"message": {"role": "assistant",
"content": [{"text": "hello"}]}},
"stopReason": "end_turn",
"usage": {"inputTokens": 9, "outputTokens": 4,
"cacheReadInputTokens": 2, "cacheWriteInputTokens": 1}
})))
.mount(&upstream)
.await;
let endpoint: Url = upstream.uri().parse().unwrap();
let proxy = start_proxy_with_state(
&upstream.uri(),
|c| c.bedrock_endpoint = Some(endpoint),
|s| s.with_bedrock_credentials(test_credentials()),
)
.await;
let resp = post_json(
format!("{}/model/{model_id}/converse", proxy.url()),
&json!({
"messages": [{"role": "user", "content": [{"text": "hello"}]}]
}),
)
.await;
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap();
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(stats["session_by_provider"]["bedrock"]["requests"], 1);
assert_eq!(stats["tokens"]["input"], 9);
assert_eq!(stats["tokens"]["output"], 4);
assert_eq!(stats["tokens"]["cache_read"], 2);
assert_eq!(stats["tokens"]["cache_write"], 1);
// Bedrock model ids price via the vendored table (cache reads
// discounted vs list input price).
assert!(stats["session"]["cache_savings_usd"].as_f64().unwrap() > 0.0);
assert_eq!(stats["lifetime_by_model"][model_id]["requests"], 1);
}
#[tokio::test]
async fn bedrock_invoke_sync_records_snake_case_usage() {
let upstream = MockServer::start().await;
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
Mock::given(method("POST"))
.and(path(format!("/model/{model_id}/invoke")))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"id": "msg_1", "type": "message", "role": "assistant",
"model": "claude-3-5-haiku-20241022",
"content": [{"type": "text", "text": "ok"}],
"usage": {"input_tokens": 12, "output_tokens": 4}
})))
.mount(&upstream)
.await;
let endpoint: Url = upstream.uri().parse().unwrap();
let proxy = start_proxy_with_state(
&upstream.uri(),
|c| c.bedrock_endpoint = Some(endpoint),
|s| s.with_bedrock_credentials(test_credentials()),
)
.await;
let resp = post_json(
format!("{}/model/{model_id}/invoke", proxy.url()),
&json!({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 16,
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap();
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(stats["tokens"]["input"], 12);
assert_eq!(stats["tokens"]["output"], 4);
assert_eq!(stats["session_by_provider"]["bedrock"]["requests"], 1);
// The request-side model id (with region/revision) wins over the
// response echo's bare id.
assert_eq!(stats["lifetime_by_model"][model_id]["requests"], 1);
}
/// The passthrough lane nothing else parses: binary EventStream
/// (`Accept: application/vnd.amazon.eventstream`), with InvokeModel's
/// base64-wrapped Anthropic events. Usage must come out of the tee's
/// telemetry parser while the client receives byte-equal frames.
#[tokio::test]
async fn bedrock_eventstream_passthrough_records_usage() {
use base64::Engine as _;
let b64 = |v: &Value| base64::engine::general_purpose::STANDARD.encode(v.to_string());
let frame = |inner: &Value| event_frame("chunk", &json!({"bytes": b64(inner)}));
let mut body_bytes: Vec<u8> = Vec::new();
for inner in [
json!({"type": "message_start",
"message": {"id": "msg_1", "usage": {"input_tokens": 21, "output_tokens": 1,
"cache_read_input_tokens": 7}}}),
json!({"type": "content_block_delta", "index": 0,
"delta": {"type": "text_delta", "text": "ok"}}),
json!({"type": "message_delta", "delta": {"stop_reason": "end_turn"},
"usage": {"output_tokens": 9}}),
json!({"type": "message_stop"}),
] {
body_bytes.extend_from_slice(&frame(&inner));
}
let upstream = MockServer::start().await;
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
Mock::given(method("POST"))
.and(path(format!(
"/model/{model_id}/invoke-with-response-stream"
)))
.respond_with(
ResponseTemplate::new(200)
.insert_header("content-type", "application/vnd.amazon.eventstream")
// Cloned: the assertion below compares the client's
// bytes against these exact frames.
.set_body_raw(body_bytes.clone(), "application/vnd.amazon.eventstream"),
)
.mount(&upstream)
.await;
let endpoint: Url = upstream.uri().parse().unwrap();
let proxy = start_proxy_with_state(
&upstream.uri(),
|c| c.bedrock_endpoint = Some(endpoint),
|s| s.with_bedrock_credentials(test_credentials()),
)
.await;
let resp = reqwest::Client::new()
.post(format!(
"{}/model/{model_id}/invoke-with-response-stream",
proxy.url()
))
.header("content-type", "application/json")
// EventStream passthrough mode — the client wants raw frames.
.header("accept", "application/vnd.amazon.eventstream")
.json(&json!({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 16,
"messages": [{"role": "user", "content": "hello"}]
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let echoed = resp.bytes().await.unwrap();
// The whole claim of this lane is that telemetry rides alongside
// the bytes without touching them. Assert that literally: the
// client's stream must be byte-identical to what the upstream
// sent, tee or no tee. `!is_empty()` would pass a mangled body.
assert_eq!(
echoed.as_ref(),
body_bytes.as_slice(),
"EventStream passthrough must be byte-identical through the tee"
);
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(
stats["tokens"]["input"], 21,
"from base64-wrapped message_start"
);
assert_eq!(
stats["tokens"]["output"], 9,
"monotone max from message_delta"
);
assert_eq!(stats["tokens"]["cache_read"], 7);
assert_eq!(stats["requests"]["cached"], 1);
}
/// Converse-stream in SSE-translation mode: usage arrives ONLY in the
/// camelCase `metadata` frame, which carries no Anthropic `type` and
/// is dropped by the Anthropic state machine — the fallback scan must
/// pick it up.
#[tokio::test]
async fn bedrock_converse_stream_sse_records_camel_metadata_usage() {
let mut body_bytes: Vec<u8> = Vec::new();
for (et, payload) in [
("messageStart", json!({"role": "assistant"})),
(
"contentBlockDelta",
json!({"contentBlockIndex": 0, "delta": {"text": "ok"}}),
),
("messageStop", json!({"stopReason": "end_turn"})),
(
"metadata",
json!({"usage": {"inputTokens": 15, "outputTokens": 6,
"cacheWriteInputTokens": 11},
"metrics": {"latencyMs": 250}}),
),
] {
body_bytes.extend_from_slice(&event_frame(et, &payload));
}
let upstream = MockServer::start().await;
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
Mock::given(method("POST"))
.and(path(format!("/model/{model_id}/converse-stream")))
.respond_with(
ResponseTemplate::new(200)
.insert_header("content-type", "application/vnd.amazon.eventstream")
.set_body_raw(body_bytes, "application/vnd.amazon.eventstream"),
)
.mount(&upstream)
.await;
let endpoint: Url = upstream.uri().parse().unwrap();
let proxy = start_proxy_with_state(
&upstream.uri(),
|c| c.bedrock_endpoint = Some(endpoint),
|s| s.with_bedrock_credentials(test_credentials()),
)
.await;
let resp = reqwest::Client::new()
.post(format!("{}/model/{model_id}/converse-stream", proxy.url()))
.header("content-type", "application/json")
.header("accept", "text/event-stream") // SSE translation mode
.json(&json!({
"messages": [{"role": "user", "content": [{"text": "hello"}]}]
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap();
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(
stats["tokens"]["input"], 15,
"camelCase metadata usage captured"
);
assert_eq!(stats["tokens"]["output"], 6);
assert_eq!(stats["tokens"]["cache_write"], 11);
}
#[tokio::test]
async fn vertex_raw_predict_records_usage() {
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.and(path_regex(
r"^/v1beta1/projects/[^/]+/locations/[^/]+/publishers/anthropic/models/.+:rawPredict$",
))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"id": "msg_1", "type": "message", "role": "assistant",
"model": "claude-sonnet-4-5",
"content": [{"type": "text", "text": "ok"}],
"usage": {"input_tokens": 33, "output_tokens": 8,
"cache_read_input_tokens": 5}
})))
.mount(&upstream)
.await;
let proxy = start_proxy_with_state(
&upstream.uri(),
|_| {},
|s| install_static_token_source(s, "ya29.stats-test-bearer"),
)
.await;
let resp = reqwest::Client::new()
.post(format!(
"{}/v1beta1/projects/p1/locations/us-central1/publishers/anthropic/models/claude-sonnet-4-5@20250929:rawPredict",
proxy.url()
))
.header("content-type", "application/json")
.json(&json!({
"anthropic_version": "vertex-2023-10-16",
"max_tokens": 16,
"messages": [{"role": "user", "content": "hello"}]
}))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap();
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(stats["session_by_provider"]["vertex"]["requests"], 1);
assert_eq!(stats["tokens"]["input"], 33);
assert_eq!(stats["tokens"]["output"], 8);
assert_eq!(stats["tokens"]["cache_read"], 5);
}
/// A proxy-side rejection (here: no AWS credentials configured) is an
/// outcome too — it must show in /stats as a failure, not vanish. An
/// operator-side credentials outage reading as "zero traffic" would
/// hide exactly the incident this dashboard exists to surface.
#[tokio::test]
async fn bedrock_proxy_side_rejection_records_failure() {
let upstream = MockServer::start().await;
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
let endpoint: Url = upstream.uri().parse().unwrap();
// No .with_bedrock_credentials(...) → handler must 5xx before
// any upstream attempt.
let proxy = start_proxy_with(&upstream.uri(), |c| c.bedrock_endpoint = Some(endpoint)).await;
let resp = post_json(
format!("{}/model/{model_id}/invoke", proxy.url()),
&json!({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 16,
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 500);
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(stats["requests"]["failed"], 1);
assert_eq!(stats["tokens"]["input"], 0);
assert_eq!(stats["session"]["savings_usd"], 0.0);
assert_eq!(stats["session_by_provider"]["bedrock"]["failed"], 1);
}
/// Persistence through the real config surface: proxy A (started with
/// a stats path) records and flushes; proxy B on the same path serves
/// the lifetime numbers proxy A earned.
#[tokio::test]
async fn stats_path_survives_across_proxy_instances() {
let dir = std::env::temp_dir().join(format!("stats-restart-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
let stats_path = dir.join("native_stats.json");
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/messages"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"id": "msg_1", "type": "message",
"model": "claude-sonnet-4-5-20250929",
"content": [{"type": "text", "text": "hi"}],
"usage": {"input_tokens": 60, "output_tokens": 6}
})))
.mount(&upstream)
.await;
// Proxy A: record one request, then flush via the ledger handle
// (the background flusher + shutdown flush live in main.rs, which
// this in-process harness deliberately doesn't run).
let ledger_slot: std::sync::Arc<
std::sync::Mutex<Option<std::sync::Arc<headroom_proxy::observability::ledger::Ledger>>>,
> = Default::default();
let slot = ledger_slot.clone();
let sp = stats_path.clone();
let proxy_a = start_proxy_with_state(
&upstream.uri(),
move |c| c.stats_path = Some(sp),
move |s| {
*slot.lock().unwrap() = Some(s.stats.clone());
s
},
)
.await;
let resp = post_json(
format!("{}/v1/messages", proxy_a.url()),
&json!({
"model": "claude-sonnet-4-5-20250929",
"max_tokens": 16,
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap();
wait_for_stats(&proxy_a.url(), |v| v["requests"]["total"] == 1).await;
let ledger = ledger_slot.lock().unwrap().take().unwrap();
ledger.flush().await;
proxy_a.shutdown().await;
assert!(stats_path.exists(), "flush persisted the ledger");
// Proxy B on the same path: lifetime carries over, session fresh.
let sp = stats_path.clone();
let proxy_b = start_proxy_with(&upstream.uri(), move |c| c.stats_path = Some(sp)).await;
let stats: Value = reqwest::Client::new()
.get(format!("{}/stats", proxy_b.url()))
.send()
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(
stats["lifetime"]["requests"], 1,
"lifetime survived restart"
);
assert_eq!(stats["lifetime"]["input_tokens"], 60);
assert_eq!(stats["session"]["requests"], 0, "session resets per boot");
let _ = std::fs::remove_dir_all(&dir);
}
/// A 200 OK SSE stream that fails IN-BAND (`{"type":"error"}` mid-
/// stream — e.g. Anthropic overloading during generation) is not a
/// billable success. The state machine lands in `Errored`; the
/// ledger must count the request as failed and accrue nothing.
#[tokio::test]
async fn in_stream_error_event_records_failure() {
let sse_body = concat!(
"event: message_start\n",
"data: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_1\",\"usage\":",
"{\"input_tokens\":50,\"output_tokens\":1}}}\n\n",
"event: error\n",
"data: {\"type\":\"error\",\"error\":{\"type\":\"overloaded_error\",",
"\"message\":\"Overloaded\"}}\n\n",
);
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/messages"))
.respond_with(
ResponseTemplate::new(200)
.insert_header("content-type", "text/event-stream")
.set_body_raw(sse_body.as_bytes().to_vec(), "text/event-stream"),
)
.mount(&upstream)
.await;
let proxy = start_proxy_with(&upstream.uri(), |_| {}).await;
let resp = post_json(
format!("{}/v1/messages", proxy.url()),
&json!({
"model": "claude-sonnet-4-5-20250929",
"max_tokens": 32,
"stream": true,
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 200, "the HTTP status was a success");
let _ = resp.bytes().await.unwrap();
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
assert_eq!(
stats["requests"]["failed"], 1,
"in-band stream error must count as a failed request"
);
assert_eq!(
stats["tokens"]["input"], 0,
"failed requests accrue nothing"
);
assert_eq!(stats["session"]["input_cost_usd"], 0.0);
}
/// A hostile (or buggy) upstream reporting absurd token counts must
/// not poison the lifetime aggregates: the ledger clamps every
/// per-event token field to its 50M ceiling before pricing.
#[tokio::test]
async fn hostile_usage_counts_are_clamped_before_recording() {
let upstream = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/messages"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"id": "msg_1",
"type": "message",
"model": "claude-sonnet-4-5-20250929",
"content": [{"type": "text", "text": "hi"}],
"usage": {
"input_tokens": u64::MAX,
"output_tokens": 9_000_000_000_000u64,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0
}
})))
.mount(&upstream)
.await;
let proxy = start_proxy_with(&upstream.uri(), |_| {}).await;
let resp = post_json(
format!("{}/v1/messages", proxy.url()),
&json!({
"model": "claude-sonnet-4-5-20250929",
"max_tokens": 32,
"messages": [{"role": "user", "content": "hello"}]
}),
)
.await;
assert_eq!(resp.status(), 200);
let _ = resp.bytes().await.unwrap();
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
const CEILING: u64 = 50_000_000;
assert_eq!(stats["tokens"]["input"], CEILING, "input clamped");
assert_eq!(stats["tokens"]["output"], CEILING, "output clamped");
let cost = stats["session"]["input_cost_usd"].as_f64().unwrap();
assert!(
cost.is_finite() && cost < 1_000.0,
"clamped pricing stays sane: {cost}"
);
}