mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(proxy): native spend/savings stats + dashboard for the Rust proxy (#2321)
# feat(proxy): native spend/savings stats + dashboard for the Rust proxy ## Description The Rust proxy has no cost observability of its own today: `/stats` and `/dashboard` only exist on the Python proxy, so anyone running `headroom-proxy` standalone (the Bedrock/Vertex lanes especially) is flying blind on spend. This adds it natively. Every lane the Rust proxy forwards — Anthropic `/v1/messages`, OpenAI Chat + Responses, all four native Bedrock routes, Vertex `rawPredict`/`streamRawPredict` — records per-request cost/usage telemetry into one in-process ledger, served from `GET /stats`, `GET /stats/timeseries`, `GET /stats/events`, and an embedded, dependency-free `GET /dashboard`. This is a ground-up redesign of #1057 (approved but never merged, now badly stale against `main`), not a rebase of it. Supersedes #1057. The "Design" section below states exactly what was kept and what was deliberately dropped, so it can be reviewed on its own terms. ### What it measures today Spend, input/output/cache-read/cache-write tokens, prompt-cache hit rate and its NET dollar benefit, per-provider and per-model breakdowns, failures, latency, and hourly/daily history that survives restarts. Two numbers are computed carefully rather than the obvious way, because the obvious way flatters the product: - **Spend counts output.** Input and output are priced separately (`input_cost_usd` / `output_cost_usd`, summed into `total_cost_usd`) because output bills at its own rate — typically 4-5x input. Counting input alone would understate a short-prompt/long-answer turn by most of its cost. - **Cache savings are NET**, not the gross read discount: the read discount minus the cache-write premium (writes bill *above* list price), floored at $0 per request. Reporting gross reads overstates the benefit on write-heavy traffic. ### What reads 0 today — stated up front `tokens_saved` / `compression_savings_usd` will read **0 on real traffic**, because the Rust live-zone per-type compressors are still no-ops on `main` — there is nothing to record yet. Verified against the real proxy: a 13.8 KB body with a large `tool_result`, sent with `--compression --compression-mode live_zone`, produced `tokens_before: 0` (dispatcher returned `NoCompression`), which is the invariant `live_zone_mode_with_valid_body_returns_no_compression_pr_b2` pins. The recording path is wired and unit-tested against `Outcome::Compressed`, and starts populating on its own the moment those compressors land — no changes needed here. Stating it up front so `Saved $0` isn't read as a bug. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Design — how this differs from #1057 - **One fold, many views.** A single `Totals::fold` is the only place request accounting happens; lifetime, since-boot session, per-provider, per-model, hourly and daily buckets are that same fold under different keys, so the views cannot disagree. #1057 (like the Python tracker it ported) maintained each aggregate separately. - **Its own schema, not Python emulation.** `/stats` is a documented Rust-native payload (`version: 1` on disk). No `schema_version: 5` file compat, no `display_session` idle-window emulation, no `projects` passthrough. The five keys external unified-stats consumers already read from this endpoint (`summary.api_requests`, `requests.failed`, `requests.cached`, `tokens.saved`, `tokens.proxy_compression_saved`) are kept and pinned by a test. - **No federation flag.** #1057's `--upstream-stats-url` supplemental merge is dropped (YAGNI — the deployed consumer merges in the opposite direction). - **No new price data file.** USD valuation reuses the LiteLLM table already vendored for `compression::model_limits`. #1057 shipped a second, models.dev-shaped price book. - **Purpose-built dashboard.** One self-contained HTML file designed for what the Rust proxy actually measures, instead of embedding the Python template and rendering its ~20 Python-only panels as zeros. - **Not gated on compression.** `--compression` defaults to **off**, so recording gated behind the compression buffer (as #1057 had it) records nothing under default config. Here the model is sourced from the *response* when the request body isn't buffered, so spend/cache observability works on a pure passthrough deployment — with zero change to the byte path. ## Correctness rules (carried from #1057's review, all test-pinned) - **Deferred recording** — a request records only when its response is fully observed, so streaming token counts come from real SSE/EventStream usage frames instead of 0. - **Failed upstreams accrue nothing** — counted in `requests`/`failed` only; retries can't inflate savings. - **No disk I/O on the request path** — `record()` folds under a mutex and sets a dirty flag; a background task (and shutdown) writes an atomic snapshot (temp → fsync → rename → fsync dir), re-arming the dirty flag on failure. - **Bounded everything** — per-model cardinality caps with `"other"` overflow, length-clamped client strings, a 50M-token per-event ceiling, bounded capture queues/buffers, bounded warn-once set. - **Honest pricing** — exact-match only; a generic id misses (and WARNs) rather than mispricing. Unknown ⇒ $0, never a blended fallback. - **Bedrock regional pricing respected** — the unmodified id is tried first, because `eu.anthropic.claude-3-5-haiku…` genuinely bills $0.25/M against the base id's $0.80/M. Geo-stripping is only a fallback for regions the table doesn't list. ## Pre-existing bugs fixed along the way Both were found while wiring the recording hooks; both affect `main` today, independently of stats. - **Bedrock geo-prefix coverage**: `au.`, `jp.`, and `us-gov.` cross-region inference profiles weren't recognized, so Anthropic models invoked through them silently skipped compression — the same failure mode #953 reported and #999 fixed for `eu.`/`us.`/`apac.`/`global.`. The list is now complete (all seven published geos) and shared (`bedrock::vendor::GEO_PREFIXES`) between the vendor gate and pricing, so the two can't drift apart again. - **EventStream `Skip` frame stranding**: the streaming parser pulled a fresh upstream chunk after each unknown (`Skip`ped) event type, delaying — and at stream end **dropping** — frames already buffered. It now drains the buffer to exhaustion before awaiting more bytes. Regression-pinned by `skip_in_final_chunk_does_not_strand_buffered_frames`. ## Changes Made - `crates/headroom-proxy/src/observability/ledger.rs` — the store: `Totals` fold, bounded maps/buckets, atomic snapshot persistence, payload builders - `crates/headroom-proxy/src/observability/pricing.rs` — deterministic lookup over the vendored LiteLLM table (no new data file) - `crates/headroom-proxy/src/observability/capture.rs` — `PendingRecord` (deferred finalization), usage extraction for every wire shape, bounded tee tasks off the byte path - `crates/headroom-proxy/src/observability/stats_http.rs` + `dashboard.html` — HTTP surface + embedded self-contained UI - `crates/headroom-proxy/src/proxy.rs`, `bedrock/invoke.rs`, `bedrock/invoke_streaming.rs`, `vertex/raw_predict.rs` — per-lane recording hooks - `crates/headroom-proxy/src/config.rs` — `--stats` (default on), `--stats-path` - `crates/headroom-proxy/src/bedrock/vendor.rs` — shared `GEO_PREFIXES` (the geo-coverage fix above) - `crates/headroom-proxy/src/sse/framing.rs` — 4 MiB accumulator cap (telemetry-side; a stream with no event terminator can no longer grow the buffer unboundedly) `--stats=false` restores byte-for-byte pre-feature behaviour (paths tunnel upstream, nothing recorded) — pinned by `stats_disabled_falls_through_to_upstream`. ## Testing - [x] `cargo fmt --all -- --check` passes - [x] `cargo clippy --workspace -- -D warnings` passes (0 warnings) - [x] `cargo test --workspace` passes - [x] New tests added for new functionality (53 new tests across unit + integration) - [x] Manual testing performed (live AWS Bedrock, all four native lanes — see Real Behavior Proof) Unit: fold-consistency across views, failed-no-savings, pricing (regional/revision/path forms, collision-miss, untracked-region fallback), USD math, bounded maps, token-ceiling clamp, bidi/control-char stripping, atomic flush with no temp litter, corrupt-file preservation, oversized-state rejection, negative-USD clamping on load, restart round-trip, week/month rollups, hostile field lengths. Integration (wiremock): endpoint contracts, `--stats=false` tunneling, deferred SSE capture, non-streaming capture, OpenAI cached-token normalisation, failed-upstream accounting, hostile-usage clamping, all four Bedrock lanes, Vertex, dashboard self-containment. The persisted-state tests are deliberately adversarial rather than round-trip only, since the ledger reads a file an operator can hand-edit: malformed bucket keys (including multi-byte ones that would panic a naive `&str` slice) are dropped before the age-trim runs, hostile model-id keys are re-clamped on load, and an implausibly large state file is treated as corrupt instead of allocated. ### Test Output ```text $ cargo fmt --all -- --check (clean) $ cargo clippy -p headroom-proxy --all-targets -- -D warnings exit=0 0 warnings, 0 errors $ cargo nextest run -p headroom-proxy Summary [4.396s] 488 tests run: 488 passed, 0 skipped $ cargo nextest run -p headroom-core -p headroom-simulators -p headroom-parity Summary [49.535s] 967 tests run: 967 passed, 1 skipped $ cargo clippy --workspace -- -D warnings exit=0 0 warnings, 0 errors $ cargo test --workspace --doc ok, 0 failed ``` ## Real Behavior Proof - Environment: live AWS Bedrock, staging account (SSO profile — name redacted), `eu-central-1`, `anthropic.claude-3-haiku-20240307-v1:0`, release binary, compression off (shipped default) - Exact command / steps: `headroom-proxy --listen 127.0.0.1:8790 --enable-bedrock-native --bedrock-region eu-central-1` then one request per lane (`/invoke`, `/converse`, `/invoke-with-response-stream` with `Accept: application/vnd.amazon.eventstream`, `/converse-stream`), one deliberate failure (nonexistent model), `SIGTERM`, restart, re-read `/stats` — full transcript, raw proxy logs, and the persisted ledger are attached below - Observed result: all four lanes captured usage (input tokens = 48 = 4 × 12; spend $0.000012 = 48 × $0.25/M, exact for Bedrock Haiku 3); the failure recorded `failed: true` with 0 tokens and $0; lifetime totals survived the restart while session totals reset; dashboard rendered against this live data (screenshots attached) - Not tested: live Vertex (covered by wiremock integration tests only — no GCP staging account available); live OpenAI Chat/Responses lanes (same wiremock coverage); dashboard on Windows browsers The four Bedrock lanes are the ones worth proving live, because two of them had no parsing before this PR: | Lane | Wire shape | Captured | |---|---|---| | `/invoke` | sync, snake_case `usage` | 12 in / 4 out | | `/converse` | sync, **camelCase** `usage` | 12 in / 4 out | | `/invoke-with-response-stream` | **binary EventStream passthrough**, base64-wrapped chunks | 12 in / 4 out | | `/converse-stream` | SSE-translated; usage lives in the camelCase `metadata` frame the Anthropic state machine drops | 12 in / 4 out | ```text requests : 5 (failed: 1) input tokens : 48 output tokens : 16 cache read/wr : 0 / 0 input spend : $0.00001200 # 48 × $0.25/M output spend : $0.00002000 # 16 × $1.25/M — output bills 5x here TOTAL spend : $0.00003200 # what /dashboard reports as "Spend" savings : $0.00000000 # compression off -> 0 by design avg latency : 256 ms by provider : {'bedrock': 5} by model : {'anthropic.claude-3-haiku-20240307-v1:0': 4, 'anthropic.claude-nonexistent-v99:0': 1} persistence : True | healthy: True ``` Those two spend lines are the reason output pricing is in this PR: **output was 62.5% of the real bill** on this run. Reporting input alone would have called a $0.000032 spend `$0.000012`. The exposure split was exercised against the same live proxy: ```text $ curl /stats/events -H 'Host: 127.0.0.1:18787' -> 200 (local tooling works) $ curl /stats/events -H 'Host: attacker.com' -> 404 (DNS-rebound: peer IS loopback) $ curl /stats -H 'Host: attacker.com' -> recent_requests: null persistence.path: null requests.total still served ``` ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (RUST_DEV.md, CHANGELOG.md) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Known limitations 1. Compression savings read 0 until the live-zone compressors land (above). 2. Hourly resolution is 48 h, daily ~13 months; older traffic folds into lifetime totals only. `/stats/events` is an in-memory ring of 100 and is not persisted — this is a cost ledger, not an audit log. 3. The vendored price table lacks some legacy direct-API ids (`claude-3-5-sonnet-20241022` is absent while its Bedrock form is present); those record $0 with a WARN until refreshed via `scripts/refresh_model_limits.sh`. 4. `output_cost_usd` is absent from state files written before this change, so it loads as 0: historical spend keeps reading input-only until those buckets age out (48 h hourly / ~400 d daily). New traffic is priced in full immediately. Deliberately no migration shim — the field defaults cleanly and the alternative is retro-pricing history against today's price table, which would invent numbers. 5. A non-streaming response body over 2 MiB isn't buffered for usage parsing (the request still records, compression-side numbers only). Streaming responses have no such cap — they parse incrementally. 6. The endpoints carry no auth, so exposure is split by tier instead (matching the Python proxy): aggregates are public like `/metrics`, while per-request rows (`recent_requests`, `/stats/events`) and the ledger's filesystem path require a local caller — loopback peer **and** loopback `Host`, the latter being the DNS-rebinding defence. Remote viewers of `/dashboard` therefore see totals but not the request feed, and the page says so. Opening that tier to a trusted reverse proxy is #1959's call to make for both proxies; this PR doesn't invent a second scheme. 7. Copilot traffic isn't proxied by the Rust binary; it reaches this dashboard only via external unified-stats layers (unchanged). ## Screenshots (if applicable) Attached below: live-test terminal transcript, dashboard rendered against the live Bedrock data (session + lifetime scopes, before/after restart), raw proxy logs, and the persisted ledger file. ## Additional Notes - Related issues/PRs: supersedes #1057; completes the geo-prefix coverage from #999 (design discussion in #953 — the multi-vendor adapter work proposed there remains out of scope). - `requests.cached` counts requests whose response reported `cache_read_tokens > 0`, matching the Python tracker's derivation, so unified-stats consumers see consistent semantics across both proxies. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2321" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
205668d953
commit
70b21c12c4
@@ -18,6 +18,13 @@ __pycache__
|
||||
dist/
|
||||
build/
|
||||
|
||||
# Rust artifacts. The Dockerfile runs its own `cargo build` inside the
|
||||
# builder stage, so the host's target/ is never needed — and after a
|
||||
# local `cargo build` it is 100GB+, which the daemon dutifully copies
|
||||
# on every `docker build`. CI never noticed because a fresh checkout
|
||||
# has no target/; it only ever bites local builds.
|
||||
target/
|
||||
|
||||
# Dev/test tooling
|
||||
.pytest_cache
|
||||
.coverage
|
||||
|
||||
Generated
+1
@@ -2009,6 +2009,7 @@ dependencies = [
|
||||
"aws-sigv4",
|
||||
"aws-smithy-runtime-api",
|
||||
"axum",
|
||||
"base64 0.22.1",
|
||||
"bytes",
|
||||
"bytesize",
|
||||
"clap",
|
||||
|
||||
+74
@@ -92,6 +92,80 @@ curl -si http://127.0.0.1:8787/v1/models
|
||||
| `--log-level` | | `info` | `RUST_LOG`-style filter |
|
||||
| `--rewrite-host` / `--no-rewrite-host` | | rewrite | rewrite Host to upstream (default) |
|
||||
| `--graceful-shutdown-timeout` | | `30s` | wait for in-flight on SIGTERM/SIGINT |
|
||||
| `--stats` | `HEADROOM_PROXY_STATS` | `true` | native savings stats + `/dashboard` (below) |
|
||||
| `--stats-path` | `HEADROOM_PROXY_STATS_PATH` | `~/.headroom/native_stats.json` | ledger persistence (honours `HEADROOM_WORKSPACE_DIR`) |
|
||||
|
||||
### Savings stats & dashboard (native)
|
||||
|
||||
The Rust proxy records per-request savings/cost telemetry on every
|
||||
lane it forwards — Anthropic `/v1/messages`, OpenAI Chat/Responses,
|
||||
all four native Bedrock routes, and Vertex `rawPredict` /
|
||||
`streamRawPredict` — and serves it locally (never tunnelled
|
||||
upstream):
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:8787/stats | jq '.session' # since-boot totals
|
||||
curl -s http://127.0.0.1:8787/stats | jq '.lifetime_by_model'
|
||||
curl -s "http://127.0.0.1:8787/stats/timeseries?bucket=day" | jq '.points[-7:]'
|
||||
curl -s "http://127.0.0.1:8787/stats/events?limit=5" # recent-request feed
|
||||
open http://127.0.0.1:8787/dashboard # embedded UI, zero deps
|
||||
```
|
||||
|
||||
Recording is deferred until the response is fully observed, so
|
||||
streaming output/cache token counts come from the real SSE /
|
||||
EventStream usage frames. USD figures come from the vendored LiteLLM
|
||||
price table (`data/model_prices_and_context_window.json`); a model
|
||||
missing from it records $0 and logs one WARN — refresh via
|
||||
`scripts/refresh_model_limits.sh`. Input and output are priced
|
||||
separately (`input_cost_usd` / `output_cost_usd`) because they bill
|
||||
at different rates — output is typically 4-5x input, so a spend
|
||||
figure that counted input alone would understate a short-prompt,
|
||||
long-answer turn by most of its cost. `total_cost_usd` is the sum
|
||||
and is what `/dashboard` shows as "Spend". Cache savings are NET: the
|
||||
cache-read discount minus the cache-write premium (writes bill
|
||||
above list price), floored at $0 per request — a warm-up turn that
|
||||
only writes is not counted as negative savings, and gross-read
|
||||
figures that overstate the benefit are never shown. Failed upstreams count as
|
||||
failures and accrue no savings — as do proxy-side rejections
|
||||
(missing AWS credentials, SigV4/ADC failures, oversized bodies), so
|
||||
an operator-side outage reads as a failure spike rather than as
|
||||
zero traffic. Aggregates (lifetime, per-model, 48 h hourly +
|
||||
~13 mo daily buckets) persist as one atomic snapshot written off
|
||||
the hot path every 10 s and on shutdown.
|
||||
|
||||
> **What reads 0 today.** Spend, token counts, and prompt-cache
|
||||
> savings are live. `tokens_saved` / `compression_savings_usd` are
|
||||
> wired end-to-end but will report 0 on real traffic until the
|
||||
> live-zone per-type compressors are implemented — the dispatcher
|
||||
> currently returns `NoCompression` for every well-formed body
|
||||
> (`live_zone_mode_with_valid_body_returns_no_compression_pr_b2`
|
||||
> pins that invariant). Nothing needs changing here when they land;
|
||||
> the numbers start flowing on their own.
|
||||
|
||||
> **No auth; exposure is split by tier.** These endpoints bind
|
||||
> wherever `--listen` points (default `0.0.0.0:8787`) and carry no
|
||||
> auth, so the payload is split the same way the Python proxy splits
|
||||
> its own `/stats`:
|
||||
>
|
||||
> - **Aggregates** — spend, tokens, per-model/provider rollups,
|
||||
> history — are served to anyone, like the Prometheus `/metrics`
|
||||
> endpoint next door. Same data class, same stance.
|
||||
> - **Per-request rows** — `recent_requests`, all of `/stats/events`
|
||||
> — plus the ledger's filesystem path are served **only to local
|
||||
> callers**: loopback peer *and* a loopback `Host` header (the
|
||||
> second is the DNS-rebinding defence — a rebound request reaches
|
||||
> the proxy *from* loopback, so the peer check alone doesn't hold).
|
||||
> Remote callers get `recent_requests: null` and a 404 from
|
||||
> `/stats/events`; the dashboard says so rather than erroring.
|
||||
>
|
||||
> A request id + model + timestamp is a different sensitivity tier
|
||||
> from a counter, and `/metrics` never exposes it — so "same as
|
||||
> /metrics" isn't a licence to hand it to the open internet. Opening
|
||||
> the local-only tier to a trusted reverse proxy is tracked in
|
||||
> [#1959](https://github.com/headroomlabs-ai/headroom/issues/1959),
|
||||
> which is settling that policy for the Python dashboard first; this
|
||||
> surface should adopt it rather than grow a second scheme.
|
||||
> `--stats=false` removes the routes entirely.
|
||||
|
||||
### Picking the next port: invocation telemetry
|
||||
|
||||
|
||||
@@ -56,6 +56,14 @@ aws-smithy-runtime-api = { workspace = true }
|
||||
# transitively pulled in by `aws-smithy-*` (so this is not an
|
||||
# additional cold dep — promoting to a direct one for clarity).
|
||||
crc32fast = "1"
|
||||
# Native savings stats: Bedrock InvokeModel EventStream chunks wrap
|
||||
# their Anthropic SSE payloads in base64 (`{"bytes": "<b64>"}`);
|
||||
# the usage-capture tee decodes them. Already in the tree via the
|
||||
# HTTP stack (reqwest/axum/hyper-util/gcp_auth all pull base64
|
||||
# 0.22) — promoted to direct for the explicit use, so no new
|
||||
# major-version bucket. (The AWS crates use `base64-simd`, a
|
||||
# different crate — this is not that one.)
|
||||
base64 = "0.22"
|
||||
# Phase D PR-D3: Prometheus metrics for Bedrock observability. The
|
||||
# `prometheus` crate's default-features pull in `protobuf`, which we
|
||||
# don't need (we serve text-format scrapes only), so we disable
|
||||
|
||||
@@ -51,10 +51,12 @@ use url::Url;
|
||||
|
||||
use crate::bedrock::envelope::BedrockEnvelope;
|
||||
use crate::bedrock::sigv4::{sign_request, SigningInputs};
|
||||
use crate::bedrock::CompressionStats;
|
||||
use crate::compression::{
|
||||
compress_anthropic_request, Outcome as AnthropicOutcome, PassthroughReason,
|
||||
};
|
||||
use crate::headers::filter_response_headers;
|
||||
use crate::observability::{capture, ledger};
|
||||
use crate::observability::{observe_bedrock_invoke_latency, record_bedrock_invoke};
|
||||
use crate::proxy::AppState;
|
||||
use crate::upstream_path::append_segments;
|
||||
@@ -175,8 +177,17 @@ pub async fn handle_invoke(
|
||||
"bedrock invoke route received request"
|
||||
);
|
||||
|
||||
// Savings ledger: constructed at handler ENTRY so every exit —
|
||||
// including proxy-side rejections (bad action path, missing
|
||||
// credentials, SigV4 failure) — produces exactly one record. A
|
||||
// credentials outage must show in /stats as a failure spike,
|
||||
// not as zero traffic. `forward_http` gives its 413 rejections
|
||||
// the same treatment.
|
||||
let mut pending_stats =
|
||||
capture::start_pending(&state, ledger::provider::BEDROCK, &model_id, &request_id);
|
||||
|
||||
let is_anthropic = is_anthropic_model_id(&model_id);
|
||||
let outbound_body: Bytes = if is_anthropic {
|
||||
let (outbound_body, compression_stats): (Bytes, CompressionStats) = if is_anthropic {
|
||||
run_anthropic_compression(&body, &state, auth_mode, &request_id)
|
||||
} else {
|
||||
tracing::info!(
|
||||
@@ -186,8 +197,15 @@ pub async fn handle_invoke(
|
||||
reason = "non_anthropic_vendor",
|
||||
"bedrock invoke: skipping live-zone compression for non-anthropic vendor"
|
||||
);
|
||||
body.clone()
|
||||
(body.clone(), CompressionStats::default())
|
||||
};
|
||||
if let Some(p) = pending_stats.as_mut() {
|
||||
p.set_compression(
|
||||
compression_stats.tokens_before,
|
||||
compression_stats.tokens_after,
|
||||
compression_stats.transforms,
|
||||
);
|
||||
}
|
||||
|
||||
// Resolve the Bedrock action from the inbound path so `/converse`
|
||||
// forwards to the upstream Converse endpoint instead of `/invoke`.
|
||||
@@ -202,6 +220,7 @@ pub async fn handle_invoke(
|
||||
path = %uri.path(),
|
||||
"bedrock invoke: unrecognized action path"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::BAD_REQUEST,
|
||||
"bedrock_invoke_action_invalid",
|
||||
@@ -235,6 +254,7 @@ pub async fn handle_invoke(
|
||||
error = %msg,
|
||||
"bedrock invoke: failed to construct upstream URL"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"bedrock_endpoint_invalid",
|
||||
@@ -253,6 +273,7 @@ pub async fn handle_invoke(
|
||||
model_id = %model_id,
|
||||
"bedrock invoke: refusing to forward without AWS credentials"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"bedrock_credentials_missing",
|
||||
@@ -288,6 +309,7 @@ pub async fn handle_invoke(
|
||||
error = %e,
|
||||
"bedrock invoke: SigV4 signing failed; refusing to forward"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"bedrock_sigv4_failed",
|
||||
@@ -329,6 +351,7 @@ pub async fn handle_invoke(
|
||||
error = %e,
|
||||
"bedrock invoke: invalid HTTP method"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::BAD_REQUEST,
|
||||
"bedrock_invalid_method",
|
||||
@@ -348,6 +371,9 @@ pub async fn handle_invoke(
|
||||
let upstream_resp = match upstream_resp {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
if let Some(p) = pending_stats.take() {
|
||||
p.finalize_failed();
|
||||
}
|
||||
tracing::warn!(
|
||||
event = "bedrock_upstream_error",
|
||||
request_id = %request_id,
|
||||
@@ -365,6 +391,11 @@ pub async fn handle_invoke(
|
||||
|
||||
let status =
|
||||
StatusCode::from_u16(upstream_resp.status().as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
|
||||
if !status.is_success() {
|
||||
if let Some(p) = pending_stats.as_mut() {
|
||||
p.mark_failed();
|
||||
}
|
||||
}
|
||||
let resp_headers = filter_response_headers(upstream_resp.headers());
|
||||
|
||||
tracing::info!(
|
||||
@@ -376,10 +407,23 @@ pub async fn handle_invoke(
|
||||
"bedrock invoke: response forwarded"
|
||||
);
|
||||
|
||||
// Stream the response body back without buffering.
|
||||
let stream = upstream_resp
|
||||
.bytes_stream()
|
||||
.map(|r| r.map_err(std::io::Error::other));
|
||||
// Stream the response body back without buffering. When the
|
||||
// savings ledger is recording, tee the bytes into a bounded
|
||||
// capture task that parses the sync response's usage block
|
||||
// (InvokeModel snake_case / Converse camelCase) at body end —
|
||||
// best-effort, never blocking the client byte path.
|
||||
let capture_tx = pending_stats
|
||||
.take()
|
||||
.map(|p| capture::spawn_json_usage_capture(capture::ResponseShape::Bedrock, p));
|
||||
let stream = upstream_resp.bytes_stream().map(move |r| match r {
|
||||
Ok(b) => {
|
||||
if let Some(tx) = &capture_tx {
|
||||
let _ = tx.try_send(b.clone());
|
||||
}
|
||||
Ok(b)
|
||||
}
|
||||
Err(e) => Err(std::io::Error::other(e)),
|
||||
});
|
||||
let body_out = Body::from_stream(stream);
|
||||
|
||||
let mut builder = Response::builder().status(status);
|
||||
@@ -417,7 +461,7 @@ fn run_anthropic_compression(
|
||||
state: &AppState,
|
||||
_auth_mode: AuthMode,
|
||||
request_id: &str,
|
||||
) -> Bytes {
|
||||
) -> (Bytes, CompressionStats) {
|
||||
// Detect envelope shape. A parseable InvokeModel envelope takes the
|
||||
// re-emit path below (anthropic_version pinned first); a non-envelope
|
||||
// body (e.g. a Converse-shaped payload) still runs through the
|
||||
@@ -453,7 +497,7 @@ fn run_anthropic_compression(
|
||||
request_id,
|
||||
);
|
||||
match outcome {
|
||||
AnthropicOutcome::NoCompression => body.clone(),
|
||||
AnthropicOutcome::NoCompression => (body.clone(), CompressionStats::default()),
|
||||
AnthropicOutcome::Passthrough { reason } => {
|
||||
tracing::info!(
|
||||
event = "bedrock_compression_passthrough",
|
||||
@@ -464,15 +508,26 @@ fn run_anthropic_compression(
|
||||
// The compressor's passthrough variants all leave bytes
|
||||
// unchanged. Forward the original.
|
||||
let _ = (PassthroughReason::ModeOff, PassthroughReason::NoMessages); // pin types
|
||||
body.clone()
|
||||
(body.clone(), CompressionStats::default())
|
||||
}
|
||||
AnthropicOutcome::Compressed { body: new_body, .. } => {
|
||||
AnthropicOutcome::Compressed {
|
||||
body: new_body,
|
||||
tokens_before,
|
||||
tokens_after,
|
||||
strategies_applied,
|
||||
..
|
||||
} => {
|
||||
let stats = CompressionStats {
|
||||
tokens_before: tokens_before as u64,
|
||||
tokens_after: tokens_after as u64,
|
||||
transforms: strategies_applied.iter().map(|s| s.to_string()).collect(),
|
||||
};
|
||||
if parsed_envelope {
|
||||
// Defence-in-depth: re-emit so anthropic_version is the
|
||||
// first key. With preserve_order this is a no-op on the
|
||||
// happy path.
|
||||
match BedrockEnvelope::ensure_anthropic_version_first(&new_body) {
|
||||
Ok(b) => b,
|
||||
Ok(b) => (b, stats),
|
||||
Err(e) => {
|
||||
tracing::error!(
|
||||
event = "bedrock_envelope_reemit_failed",
|
||||
@@ -480,11 +535,13 @@ fn run_anthropic_compression(
|
||||
error = %e,
|
||||
"bedrock invoke: failed to re-emit envelope; falling back to original body"
|
||||
);
|
||||
body.clone()
|
||||
// Original bytes forwarded ⇒ no compression
|
||||
// happened as far as the wire is concerned.
|
||||
(body.clone(), CompressionStats::default())
|
||||
}
|
||||
}
|
||||
} else {
|
||||
new_body
|
||||
(new_body, stats)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -631,6 +688,7 @@ mod tests {
|
||||
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
|
||||
"test".to_string(),
|
||||
)),
|
||||
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
|
||||
};
|
||||
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/converse"
|
||||
.parse()
|
||||
@@ -669,6 +727,7 @@ mod tests {
|
||||
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
|
||||
"test".to_string(),
|
||||
)),
|
||||
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
|
||||
};
|
||||
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/invoke"
|
||||
.parse()
|
||||
@@ -705,6 +764,7 @@ mod tests {
|
||||
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
|
||||
"test".to_string(),
|
||||
)),
|
||||
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
|
||||
};
|
||||
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/invoke"
|
||||
.parse()
|
||||
|
||||
@@ -61,10 +61,13 @@ use crate::bedrock::eventstream_to_sse::{
|
||||
translate_message, OutputMode, TranslateError, TranslateOutcome,
|
||||
};
|
||||
use crate::bedrock::sigv4::{sign_request, SigningInputs};
|
||||
use crate::bedrock::CompressionStats;
|
||||
use crate::compression::{
|
||||
compress_anthropic_request, Outcome as AnthropicOutcome, PassthroughReason,
|
||||
};
|
||||
use crate::headers::filter_response_headers;
|
||||
use crate::observability::capture::TEE_QUEUE_DEPTH as SSE_PARSER_QUEUE_DEPTH;
|
||||
use crate::observability::{capture, ledger};
|
||||
use crate::observability::{
|
||||
observe_bedrock_invoke_latency, record_bedrock_eventstream_message, record_bedrock_invoke,
|
||||
};
|
||||
@@ -153,9 +156,14 @@ pub async fn handle_invoke_streaming(
|
||||
"bedrock invoke-with-response-stream route received request"
|
||||
);
|
||||
|
||||
// Constructed at handler ENTRY so every exit — including
|
||||
// proxy-side rejections — records exactly once.
|
||||
let mut pending_stats =
|
||||
capture::start_pending(&state, ledger::provider::BEDROCK, &model_id, &request_id);
|
||||
|
||||
// 1. Live-zone compression for Anthropic-shape bodies (same as D1).
|
||||
let is_anthropic = is_anthropic_model_id(&model_id);
|
||||
let outbound_body: Bytes = if is_anthropic {
|
||||
let (outbound_body, compression_stats): (Bytes, CompressionStats) = if is_anthropic {
|
||||
run_anthropic_compression(&body, &state, auth_mode, &request_id)
|
||||
} else {
|
||||
tracing::info!(
|
||||
@@ -165,8 +173,15 @@ pub async fn handle_invoke_streaming(
|
||||
reason = "non_anthropic_vendor",
|
||||
"bedrock invoke-streaming: skipping compression for non-anthropic vendor"
|
||||
);
|
||||
body.clone()
|
||||
(body.clone(), CompressionStats::default())
|
||||
};
|
||||
if let Some(p) = pending_stats.as_mut() {
|
||||
p.set_compression(
|
||||
compression_stats.tokens_before,
|
||||
compression_stats.tokens_after,
|
||||
compression_stats.transforms,
|
||||
);
|
||||
}
|
||||
|
||||
// 2. Resolve the Bedrock streaming action from the inbound path and
|
||||
// build the upstream URL.
|
||||
@@ -179,6 +194,7 @@ pub async fn handle_invoke_streaming(
|
||||
path = %uri.path(),
|
||||
"bedrock invoke-streaming: unrecognized streaming action path"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::BAD_REQUEST,
|
||||
"bedrock_streaming_action_invalid",
|
||||
@@ -209,6 +225,7 @@ pub async fn handle_invoke_streaming(
|
||||
error = %msg,
|
||||
"bedrock invoke-streaming: failed to construct upstream URL"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"bedrock_endpoint_invalid",
|
||||
@@ -227,6 +244,7 @@ pub async fn handle_invoke_streaming(
|
||||
model_id = %model_id,
|
||||
"bedrock invoke-streaming: refusing to forward without AWS credentials"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"bedrock_credentials_missing",
|
||||
@@ -261,6 +279,7 @@ pub async fn handle_invoke_streaming(
|
||||
error = %e,
|
||||
"bedrock invoke-streaming: SigV4 signing failed"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"bedrock_sigv4_failed",
|
||||
@@ -298,6 +317,7 @@ pub async fn handle_invoke_streaming(
|
||||
error = %e,
|
||||
"bedrock invoke-streaming: invalid HTTP method"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::BAD_REQUEST,
|
||||
"bedrock_invalid_method",
|
||||
@@ -317,6 +337,9 @@ pub async fn handle_invoke_streaming(
|
||||
let upstream_resp = match upstream_resp {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
if let Some(p) = pending_stats.take() {
|
||||
p.finalize_failed();
|
||||
}
|
||||
tracing::warn!(
|
||||
event = "bedrock_upstream_error",
|
||||
request_id = %request_id,
|
||||
@@ -334,6 +357,11 @@ pub async fn handle_invoke_streaming(
|
||||
|
||||
let status =
|
||||
StatusCode::from_u16(upstream_resp.status().as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
|
||||
if !status.is_success() {
|
||||
if let Some(p) = pending_stats.as_mut() {
|
||||
p.mark_failed();
|
||||
}
|
||||
}
|
||||
let upstream_content_type = upstream_resp
|
||||
.headers()
|
||||
.get(http::header::CONTENT_TYPE)
|
||||
@@ -373,10 +401,22 @@ pub async fn handle_invoke_streaming(
|
||||
// Even here we drop content-length: the proxy streams the
|
||||
// body and reqwest's transparent decompression may already
|
||||
// have changed the byte length on the way in.
|
||||
// Usage capture: this is typically an error envelope (the
|
||||
// pending event is already marked failed on non-2xx); parse
|
||||
// it as JSON anyway so a 2xx JSON body still yields usage.
|
||||
resp_headers.remove(http::header::CONTENT_LENGTH);
|
||||
let stream = upstream_resp
|
||||
.bytes_stream()
|
||||
.map(|r| r.map_err(std::io::Error::other));
|
||||
let capture_tx = pending_stats
|
||||
.take()
|
||||
.map(|p| capture::spawn_json_usage_capture(capture::ResponseShape::Bedrock, p));
|
||||
let stream = upstream_resp.bytes_stream().map(move |r| match r {
|
||||
Ok(b) => {
|
||||
if let Some(tx) = &capture_tx {
|
||||
let _ = tx.try_send(b.clone());
|
||||
}
|
||||
Ok(b)
|
||||
}
|
||||
Err(e) => Err(std::io::Error::other(e)),
|
||||
});
|
||||
let body_out = Body::from_stream(stream);
|
||||
return finish(status, resp_headers, body_out, &request_id);
|
||||
}
|
||||
@@ -410,7 +450,20 @@ pub async fn handle_invoke_streaming(
|
||||
resp_headers.insert(http::header::CONTENT_TYPE, v);
|
||||
}
|
||||
}
|
||||
let body_out = Body::from_stream(upstream_stream);
|
||||
// Usage capture: nothing else parses the frames in
|
||||
// passthrough mode, so tee the raw bytes into a bounded
|
||||
// EventStream telemetry parser (chunk / metadata usage;
|
||||
// finalizes the ledger event when the stream closes).
|
||||
let capture_tx = pending_stats
|
||||
.take()
|
||||
.map(capture::spawn_eventstream_usage_capture);
|
||||
let teed = upstream_stream.map(move |r| {
|
||||
if let (Ok(b), Some(tx)) = (&r, &capture_tx) {
|
||||
let _ = tx.try_send(b.clone());
|
||||
}
|
||||
r
|
||||
});
|
||||
let body_out = Body::from_stream(teed);
|
||||
finish(status, resp_headers, body_out, &request_id)
|
||||
}
|
||||
OutputMode::Sse => {
|
||||
@@ -429,7 +482,8 @@ pub async fn handle_invoke_streaming(
|
||||
model_id.clone(),
|
||||
region.clone(),
|
||||
);
|
||||
let translated = tee_to_anthropic_state(translated, request_id.clone());
|
||||
let translated =
|
||||
tee_to_anthropic_state(translated, request_id.clone(), pending_stats.take());
|
||||
let body_out = Body::from_stream(translated);
|
||||
finish(status, resp_headers, body_out, &request_id)
|
||||
}
|
||||
@@ -475,201 +529,125 @@ where
|
||||
if done {
|
||||
return None;
|
||||
}
|
||||
// First, drain any complete messages already in the
|
||||
// parser's buffer (bytes from the previous chunk).
|
||||
loop {
|
||||
match parser.next_message() {
|
||||
Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {
|
||||
Ok(TranslateOutcome::Emit(frame)) => {
|
||||
// PR-D3: per-message Prometheus metric.
|
||||
// The label `event_type` is bounded by
|
||||
// AWS's documented vocabulary (chunk,
|
||||
// metadata, exception variants); not
|
||||
// customer-controlled.
|
||||
let event_type = msg.event_type().unwrap_or("unknown").to_string();
|
||||
record_bedrock_eventstream_message(&model_id, ®ion, &event_type);
|
||||
tracing::debug!(
|
||||
event = "bedrock_eventstream_message",
|
||||
request_id = %request_id,
|
||||
event_type = %event_type,
|
||||
payload_bytes = msg.payload.len(),
|
||||
"translated bedrock eventstream message"
|
||||
);
|
||||
return Some((
|
||||
Ok(frame),
|
||||
(parser, upstream, false, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
Ok(TranslateOutcome::Skip { event_type }) => {
|
||||
tracing::warn!(
|
||||
event = "bedrock_eventstream_unknown_event_type",
|
||||
request_id = %request_id,
|
||||
event_type = %event_type,
|
||||
"skipping unknown bedrock eventstream message"
|
||||
);
|
||||
// Loop and try the next message in the buffer.
|
||||
continue;
|
||||
}
|
||||
Err(TranslateError::UpstreamException { payload_preview }) => {
|
||||
tracing::warn!(
|
||||
event = "bedrock_eventstream_upstream_exception",
|
||||
request_id = %request_id,
|
||||
payload_preview = %payload_preview,
|
||||
"bedrock eventstream upstream exception"
|
||||
);
|
||||
// Emit the exception as an Anthropic-shape
|
||||
// SSE error frame so the client sees it.
|
||||
let json = serde_json::json!({
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "bedrock_upstream_exception",
|
||||
"message": payload_preview,
|
||||
}
|
||||
})
|
||||
.to_string();
|
||||
let mut frame = Vec::with_capacity(json.len() + 32);
|
||||
frame.extend_from_slice(b"event: error\ndata: ");
|
||||
frame.extend_from_slice(json.as_bytes());
|
||||
frame.extend_from_slice(b"\n\n");
|
||||
return Some((
|
||||
Ok(Bytes::from(frame)),
|
||||
(parser, upstream, true, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
Err(TranslateError::MissingEventType) => {
|
||||
tracing::warn!(
|
||||
event = "bedrock_eventstream_missing_event_type",
|
||||
request_id = %request_id,
|
||||
"bedrock eventstream message missing :event-type; emitting error frame"
|
||||
);
|
||||
let frame = error_sse_frame(
|
||||
"bedrock_eventstream_missing_event_type",
|
||||
"Bedrock message missing :event-type header",
|
||||
);
|
||||
return Some((
|
||||
Ok(frame),
|
||||
(parser, upstream, true, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
},
|
||||
Ok(None) => break,
|
||||
Err(parse_err) => {
|
||||
let event_name = match &parse_err {
|
||||
ParseError::PreludeCrcMismatch { .. }
|
||||
| ParseError::MessageCrcMismatch { .. } => {
|
||||
"bedrock_eventstream_crc_mismatch"
|
||||
// Drain-first alternation: after a chunk is pushed we
|
||||
// re-enter the drain phase rather than pulling again,
|
||||
// so a Skip'd message can never strand complete
|
||||
// messages in the parser buffer. (The previous shape
|
||||
// pulled a fresh upstream chunk after a Skip —
|
||||
// delaying, and at stream end DROPPING, frames that
|
||||
// were already buffered behind an unknown event type.)
|
||||
'parse: loop {
|
||||
// Phase 1: emit complete messages already buffered.
|
||||
loop {
|
||||
match parser.next_message() {
|
||||
Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {
|
||||
Ok(TranslateOutcome::Emit(frame)) => {
|
||||
// PR-D3: per-message Prometheus metric.
|
||||
// The label `event_type` is bounded by
|
||||
// AWS's documented vocabulary (chunk,
|
||||
// metadata, exception variants); not
|
||||
// customer-controlled.
|
||||
let event_type =
|
||||
msg.event_type().unwrap_or("unknown").to_string();
|
||||
record_bedrock_eventstream_message(
|
||||
&model_id,
|
||||
®ion,
|
||||
&event_type,
|
||||
);
|
||||
tracing::debug!(
|
||||
event = "bedrock_eventstream_message",
|
||||
request_id = %request_id,
|
||||
event_type = %event_type,
|
||||
payload_bytes = msg.payload.len(),
|
||||
"translated bedrock eventstream message"
|
||||
);
|
||||
return Some((
|
||||
Ok(frame),
|
||||
(parser, upstream, false, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
Ok(TranslateOutcome::Skip { event_type }) => {
|
||||
tracing::warn!(
|
||||
event = "bedrock_eventstream_unknown_event_type",
|
||||
request_id = %request_id,
|
||||
event_type = %event_type,
|
||||
"skipping unknown bedrock eventstream message"
|
||||
);
|
||||
// Loop and try the next message in the buffer.
|
||||
continue;
|
||||
}
|
||||
_ => "bedrock_eventstream_parse_failed",
|
||||
};
|
||||
tracing::warn!(
|
||||
event = event_name,
|
||||
request_id = %request_id,
|
||||
error = %parse_err,
|
||||
"bedrock eventstream parse failure; closing translated stream"
|
||||
);
|
||||
let frame = error_sse_frame(event_name, &parse_err.to_string());
|
||||
return Some((
|
||||
Ok(frame),
|
||||
(parser, upstream, true, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
// Buffer drained; pull the next chunk from upstream.
|
||||
loop {
|
||||
match upstream.next().await {
|
||||
Some(Ok(chunk)) => {
|
||||
parser.push(&chunk);
|
||||
// Loop back through the parser to emit any
|
||||
// newly-complete messages.
|
||||
match parser.next_message() {
|
||||
Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {
|
||||
Ok(TranslateOutcome::Emit(frame)) => {
|
||||
// PR-D3: per-message Prometheus
|
||||
// metric (mirror of the parser-buffer
|
||||
// drain branch above).
|
||||
let event_type =
|
||||
msg.event_type().unwrap_or("unknown").to_string();
|
||||
record_bedrock_eventstream_message(
|
||||
&model_id,
|
||||
®ion,
|
||||
&event_type,
|
||||
);
|
||||
tracing::debug!(
|
||||
event = "bedrock_eventstream_message",
|
||||
request_id = %request_id,
|
||||
event_type = %event_type,
|
||||
payload_bytes = msg.payload.len(),
|
||||
"translated bedrock eventstream message"
|
||||
);
|
||||
return Some((
|
||||
Ok(frame),
|
||||
(parser, upstream, false, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
Ok(TranslateOutcome::Skip { event_type }) => {
|
||||
tracing::warn!(
|
||||
event = "bedrock_eventstream_unknown_event_type",
|
||||
request_id = %request_id,
|
||||
event_type = %event_type,
|
||||
"skipping unknown bedrock eventstream message"
|
||||
);
|
||||
// Continue draining the parser /
|
||||
// pulling more chunks.
|
||||
continue;
|
||||
}
|
||||
Err(TranslateError::UpstreamException { payload_preview }) => {
|
||||
let json = serde_json::json!({
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "bedrock_upstream_exception",
|
||||
"message": payload_preview,
|
||||
}
|
||||
})
|
||||
.to_string();
|
||||
let mut frame = Vec::with_capacity(json.len() + 32);
|
||||
frame.extend_from_slice(b"event: error\ndata: ");
|
||||
frame.extend_from_slice(json.as_bytes());
|
||||
frame.extend_from_slice(b"\n\n");
|
||||
return Some((
|
||||
Ok(Bytes::from(frame)),
|
||||
(parser, upstream, true, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
Err(TranslateError::MissingEventType) => {
|
||||
let frame = error_sse_frame(
|
||||
"bedrock_eventstream_missing_event_type",
|
||||
"Bedrock message missing :event-type header",
|
||||
);
|
||||
return Some((
|
||||
Ok(frame),
|
||||
(parser, upstream, true, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
},
|
||||
Ok(None) => continue,
|
||||
Err(parse_err) => {
|
||||
let event_name = match &parse_err {
|
||||
ParseError::PreludeCrcMismatch { .. }
|
||||
| ParseError::MessageCrcMismatch { .. } => {
|
||||
"bedrock_eventstream_crc_mismatch"
|
||||
Err(TranslateError::UpstreamException { payload_preview }) => {
|
||||
tracing::warn!(
|
||||
event = "bedrock_eventstream_upstream_exception",
|
||||
request_id = %request_id,
|
||||
payload_preview = %payload_preview,
|
||||
"bedrock eventstream upstream exception"
|
||||
);
|
||||
// Emit the exception as an Anthropic-shape
|
||||
// SSE error frame so the client sees it.
|
||||
let json = serde_json::json!({
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "bedrock_upstream_exception",
|
||||
"message": payload_preview,
|
||||
}
|
||||
_ => "bedrock_eventstream_parse_failed",
|
||||
};
|
||||
})
|
||||
.to_string();
|
||||
let mut frame = Vec::with_capacity(json.len() + 32);
|
||||
frame.extend_from_slice(b"event: error\ndata: ");
|
||||
frame.extend_from_slice(json.as_bytes());
|
||||
frame.extend_from_slice(b"\n\n");
|
||||
return Some((
|
||||
Ok(Bytes::from(frame)),
|
||||
(parser, upstream, true, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
Err(TranslateError::MissingEventType) => {
|
||||
tracing::warn!(
|
||||
event = event_name,
|
||||
event = "bedrock_eventstream_missing_event_type",
|
||||
request_id = %request_id,
|
||||
error = %parse_err,
|
||||
"bedrock eventstream parse failure"
|
||||
"bedrock eventstream message missing :event-type; emitting error frame"
|
||||
);
|
||||
let frame = error_sse_frame(
|
||||
"bedrock_eventstream_missing_event_type",
|
||||
"Bedrock message missing :event-type header",
|
||||
);
|
||||
let frame = error_sse_frame(event_name, &parse_err.to_string());
|
||||
return Some((
|
||||
Ok(frame),
|
||||
(parser, upstream, true, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
},
|
||||
Ok(None) => break,
|
||||
Err(parse_err) => {
|
||||
let event_name = match &parse_err {
|
||||
ParseError::PreludeCrcMismatch { .. }
|
||||
| ParseError::MessageCrcMismatch { .. } => {
|
||||
"bedrock_eventstream_crc_mismatch"
|
||||
}
|
||||
_ => "bedrock_eventstream_parse_failed",
|
||||
};
|
||||
tracing::warn!(
|
||||
event = event_name,
|
||||
request_id = %request_id,
|
||||
error = %parse_err,
|
||||
"bedrock eventstream parse failure; closing translated stream"
|
||||
);
|
||||
let frame = error_sse_frame(event_name, &parse_err.to_string());
|
||||
return Some((
|
||||
Ok(frame),
|
||||
(parser, upstream, true, request_id, model_id, region),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
// Phase 2: buffer fully drained — pull exactly
|
||||
// one chunk, then loop back to the drain phase.
|
||||
match upstream.next().await {
|
||||
Some(Ok(chunk)) => {
|
||||
parser.push(&chunk);
|
||||
continue 'parse;
|
||||
}
|
||||
Some(Err(e)) => {
|
||||
tracing::warn!(
|
||||
event = "bedrock_eventstream_upstream_io_error",
|
||||
@@ -683,10 +661,11 @@ where
|
||||
));
|
||||
}
|
||||
None => {
|
||||
// End of upstream stream. If buffered bytes
|
||||
// remain that did not parse into a message,
|
||||
// log loudly — we are NOT silently dropping
|
||||
// them.
|
||||
// End of upstream stream. The drain phase
|
||||
// above ran to Ok(None) first, so no
|
||||
// complete message can be stranded here;
|
||||
// leftover bytes are a truncated partial
|
||||
// message — log loudly, never silently.
|
||||
if parser.buffered_len() > 0 {
|
||||
tracing::warn!(
|
||||
event = "bedrock_eventstream_truncated",
|
||||
@@ -713,6 +692,7 @@ where
|
||||
fn tee_to_anthropic_state<S>(
|
||||
upstream: S,
|
||||
request_id: String,
|
||||
pending_stats: Option<capture::PendingRecord>,
|
||||
) -> impl Stream<Item = Result<Bytes, std::io::Error>>
|
||||
where
|
||||
S: Stream<Item = Result<Bytes, std::io::Error>> + Send + 'static,
|
||||
@@ -720,7 +700,7 @@ where
|
||||
let (tx, rx) = tokio::sync::mpsc::channel::<Bytes>(SSE_PARSER_QUEUE_DEPTH);
|
||||
let parser_request_id = request_id.clone();
|
||||
tokio::spawn(async move {
|
||||
run_anthropic_state_machine(rx, parser_request_id).await;
|
||||
run_anthropic_state_machine(rx, parser_request_id, pending_stats).await;
|
||||
});
|
||||
|
||||
upstream.map(move |r| {
|
||||
@@ -738,8 +718,6 @@ where
|
||||
})
|
||||
}
|
||||
|
||||
const SSE_PARSER_QUEUE_DEPTH: usize = 256;
|
||||
|
||||
/// Dedicated state-machine task. Mirrors the
|
||||
/// `run_sse_state_machine(SseStreamKind::Anthropic, ...)` arm in
|
||||
/// `proxy.rs` so usage extraction works identically for direct
|
||||
@@ -747,17 +725,31 @@ const SSE_PARSER_QUEUE_DEPTH: usize = 256;
|
||||
async fn run_anthropic_state_machine(
|
||||
mut rx: tokio::sync::mpsc::Receiver<Bytes>,
|
||||
request_id: String,
|
||||
pending_stats: Option<capture::PendingRecord>,
|
||||
) {
|
||||
use crate::sse::anthropic::AnthropicStreamState;
|
||||
use crate::sse::framing::SseFramer;
|
||||
|
||||
let mut framer = SseFramer::new();
|
||||
let mut state = AnthropicStreamState::new();
|
||||
// Converse-native frames (`metadata`, `messageStart`, …) carry no
|
||||
// Anthropic `type`, so they arrive here as event-name-less frames
|
||||
// the Anthropic state machine drops. Their camelCase usage block
|
||||
// is the ONLY usage signal on a native Converse stream — scan
|
||||
// them separately so `/converse-stream` requests don't record
|
||||
// zero tokens.
|
||||
let mut converse_usage = ledger::Usage::default();
|
||||
while let Some(chunk) = rx.recv().await {
|
||||
framer.push(&chunk);
|
||||
while let Some(ev_result) = framer.next_event() {
|
||||
match ev_result {
|
||||
Ok(ev) => {
|
||||
if ev.event_name.is_none() {
|
||||
if let Ok(payload) = serde_json::from_slice::<serde_json::Value>(&ev.data) {
|
||||
capture::merge_stream_usage(&mut converse_usage, &payload);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if let Err(e) = state.apply(ev) {
|
||||
tracing::warn!(
|
||||
request_id = %request_id,
|
||||
@@ -787,6 +779,16 @@ async fn run_anthropic_state_machine(
|
||||
blocks = state.blocks.len(),
|
||||
"bedrock translated stream: closed"
|
||||
);
|
||||
if let Some(mut p) = pending_stats {
|
||||
// In-band `{"type":"error"}` on a 200 OK stream is not a
|
||||
// billable success (same rule as the direct Anthropic lane).
|
||||
if matches!(state.status, crate::sse::anthropic::StreamStatus::Errored) {
|
||||
p.mark_failed();
|
||||
}
|
||||
let mut usage = capture::usage_from_anthropic_state(&state);
|
||||
capture::merge_max(&mut usage, converse_usage);
|
||||
p.finalize(usage);
|
||||
}
|
||||
}
|
||||
|
||||
/// True when the content-type is `application/vnd.amazon.eventstream`
|
||||
@@ -866,7 +868,7 @@ fn run_anthropic_compression(
|
||||
state: &AppState,
|
||||
_auth_mode: AuthMode,
|
||||
request_id: &str,
|
||||
) -> Bytes {
|
||||
) -> (Bytes, CompressionStats) {
|
||||
use crate::bedrock::envelope::BedrockEnvelope;
|
||||
|
||||
let parsed_envelope = BedrockEnvelope::parse(body).is_ok();
|
||||
@@ -888,7 +890,7 @@ fn run_anthropic_compression(
|
||||
request_id,
|
||||
);
|
||||
match outcome {
|
||||
AnthropicOutcome::NoCompression => body.clone(),
|
||||
AnthropicOutcome::NoCompression => (body.clone(), CompressionStats::default()),
|
||||
AnthropicOutcome::Passthrough { reason } => {
|
||||
tracing::info!(
|
||||
event = "bedrock_compression_passthrough",
|
||||
@@ -897,12 +899,23 @@ fn run_anthropic_compression(
|
||||
"bedrock invoke-streaming: passthrough"
|
||||
);
|
||||
let _ = (PassthroughReason::ModeOff, PassthroughReason::NoMessages);
|
||||
body.clone()
|
||||
(body.clone(), CompressionStats::default())
|
||||
}
|
||||
AnthropicOutcome::Compressed { body: new_body, .. } => {
|
||||
AnthropicOutcome::Compressed {
|
||||
body: new_body,
|
||||
tokens_before,
|
||||
tokens_after,
|
||||
strategies_applied,
|
||||
..
|
||||
} => {
|
||||
let stats = CompressionStats {
|
||||
tokens_before: tokens_before as u64,
|
||||
tokens_after: tokens_after as u64,
|
||||
transforms: strategies_applied.iter().map(|s| s.to_string()).collect(),
|
||||
};
|
||||
if parsed_envelope {
|
||||
match BedrockEnvelope::ensure_anthropic_version_first(&new_body) {
|
||||
Ok(b) => b,
|
||||
Ok(b) => (b, stats),
|
||||
Err(e) => {
|
||||
tracing::error!(
|
||||
event = "bedrock_envelope_reemit_failed",
|
||||
@@ -910,11 +923,13 @@ fn run_anthropic_compression(
|
||||
error = %e,
|
||||
"bedrock invoke-streaming: failed to re-emit envelope"
|
||||
);
|
||||
body.clone()
|
||||
// Original bytes forwarded ⇒ no compression
|
||||
// happened as far as the wire is concerned.
|
||||
(body.clone(), CompressionStats::default())
|
||||
}
|
||||
}
|
||||
} else {
|
||||
new_body
|
||||
(new_body, stats)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1035,6 +1050,7 @@ mod tests {
|
||||
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
|
||||
"test".to_string(),
|
||||
)),
|
||||
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
|
||||
};
|
||||
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/invoke-with-response-stream"
|
||||
.parse()
|
||||
@@ -1075,6 +1091,7 @@ mod tests {
|
||||
vertex_token_source: std::sync::Arc::new(crate::vertex::StaticTokenSource::new(
|
||||
"test".to_string(),
|
||||
)),
|
||||
stats: std::sync::Arc::new(ledger::Ledger::in_memory()),
|
||||
};
|
||||
let uri: Uri = "/model/anthropic.claude-3-haiku-20240307-v1:0/converse-stream"
|
||||
.parse()
|
||||
|
||||
@@ -54,6 +54,15 @@ pub mod invoke_streaming;
|
||||
pub mod sigv4;
|
||||
pub mod vendor;
|
||||
|
||||
/// Compression-side numbers the Bedrock handlers hand to the savings
|
||||
/// ledger alongside the (possibly compressed) outbound body.
|
||||
#[derive(Debug, Default, Clone)]
|
||||
pub(crate) struct CompressionStats {
|
||||
pub tokens_before: u64,
|
||||
pub tokens_after: u64,
|
||||
pub transforms: Vec<String>,
|
||||
}
|
||||
|
||||
pub use auth_mode_layer::classify_and_attach_auth_mode;
|
||||
pub use envelope::{BedrockEnvelope, EnvelopeError};
|
||||
pub use eventstream::{
|
||||
|
||||
@@ -13,7 +13,28 @@
|
||||
/// vendor segment. Stripped (once) before vendor resolution. Kept to a
|
||||
/// closed, known set so an unrelated `something.anthropic.x` id is not
|
||||
/// mistaken for an Anthropic inference profile.
|
||||
const GEO_PREFIXES: [&str; 4] = ["eu.", "us.", "apac.", "global."];
|
||||
///
|
||||
/// The set is the prefixes present in the vendored LiteLLM price
|
||||
/// table (`data/model_prices_and_context_window.json`) — every one of
|
||||
/// these fronts real `<geo>.anthropic.…` ids there. A missing entry
|
||||
/// is not cosmetic: an unrecognized geo prefix makes
|
||||
/// `is_anthropic_model_id` return false, silently skipping
|
||||
/// compression for that region's inference profiles (`au.`/`jp.`/
|
||||
/// `us-gov.` were missing until review of the stats work caught it).
|
||||
pub(crate) const GEO_PREFIXES: [&str; 7] =
|
||||
["eu.", "us.", "us-gov.", "apac.", "au.", "jp.", "global."];
|
||||
|
||||
/// Strip a cross-region inference-profile geo prefix, if present.
|
||||
/// `eu.anthropic.claude-…` → `anthropic.claude-…`; ids without a
|
||||
/// known geo prefix return unchanged. Shared with
|
||||
/// `observability::pricing` so price lookups see the same canonical
|
||||
/// id the vendor gate sees.
|
||||
pub(crate) fn strip_geo_prefix(model_id: &str) -> &str {
|
||||
GEO_PREFIXES
|
||||
.iter()
|
||||
.find_map(|p| model_id.strip_prefix(p))
|
||||
.unwrap_or(model_id)
|
||||
}
|
||||
|
||||
/// Resolve the canonical vendor of a Bedrock model id, stripping a
|
||||
/// cross-region inference-profile geo prefix first.
|
||||
@@ -21,10 +42,7 @@ const GEO_PREFIXES: [&str; 4] = ["eu.", "us.", "apac.", "global."];
|
||||
/// `eu.anthropic.claude-…` → `anthropic`; `amazon.titan-…` → `amazon`;
|
||||
/// `global.amazon.nova-…` → `amazon`.
|
||||
pub fn canonical_vendor(model_id: &str) -> &str {
|
||||
let stripped = GEO_PREFIXES
|
||||
.iter()
|
||||
.find_map(|p| model_id.strip_prefix(p))
|
||||
.unwrap_or(model_id);
|
||||
let stripped = strip_geo_prefix(model_id);
|
||||
stripped.split('.').next().unwrap_or(stripped)
|
||||
}
|
||||
|
||||
@@ -73,6 +91,17 @@ mod tests {
|
||||
assert!(is_anthropic_model_id(
|
||||
"global.anthropic.claude-haiku-4-5-20251001-v1:0"
|
||||
));
|
||||
// au./jp./us-gov. are real AWS geo prefixes (present in the
|
||||
// vendored price table); they must gate as Anthropic too.
|
||||
assert!(is_anthropic_model_id(
|
||||
"au.anthropic.claude-sonnet-4-5-20250929-v1:0"
|
||||
));
|
||||
assert!(is_anthropic_model_id(
|
||||
"jp.anthropic.claude-haiku-4-5-20251001-v1:0"
|
||||
));
|
||||
assert!(is_anthropic_model_id(
|
||||
"us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0"
|
||||
));
|
||||
// Non-Anthropic vendors, including geo-prefixed, stay false.
|
||||
assert!(!is_anthropic_model_id("amazon.titan-text-express-v1"));
|
||||
assert!(!is_anthropic_model_id("meta.llama3-70b-instruct-v1:0"));
|
||||
|
||||
@@ -66,7 +66,10 @@ pub(crate) const DEFAULT_CONTEXT_WINDOW: u32 = 128_000;
|
||||
/// LiteLLM's vendored model price + context-window table. Refreshed
|
||||
/// via `scripts/refresh_model_limits.sh`. ~1.4MB; embedded into the
|
||||
/// binary so the proxy ships with no startup network dependency.
|
||||
const VENDORED_JSON: &str = include_str!("../../data/model_prices_and_context_window.json");
|
||||
// Shared with `observability::pricing`, which reads the per-token
|
||||
// cost fields of the same table — one embedded copy, two consumers.
|
||||
pub(crate) const VENDORED_JSON: &str =
|
||||
include_str!("../../data/model_prices_and_context_window.json");
|
||||
|
||||
/// Parsed lookup: model id → max input tokens. Built lazily on
|
||||
/// first call; subsequent calls reuse the same `HashMap`.
|
||||
|
||||
@@ -592,6 +592,34 @@ pub struct CliArgs {
|
||||
default_value = "https://www.googleapis.com/auth/cloud-platform"
|
||||
)]
|
||||
pub vertex_adc_scope: String,
|
||||
|
||||
/// Native savings stats: record per-request savings/cost
|
||||
/// telemetry and serve `/stats`, `/stats/timeseries`,
|
||||
/// `/stats/events`, and `/dashboard`. When disabled, nothing is
|
||||
/// recorded and those paths fall through to the catch-all
|
||||
/// forwarder (i.e. they reach the upstream, matching the
|
||||
/// pre-feature behaviour).
|
||||
///
|
||||
/// Source priority: CLI flag → `HEADROOM_PROXY_STATS` env var →
|
||||
/// default (`false`).
|
||||
#[arg(
|
||||
long = "stats",
|
||||
env = "HEADROOM_PROXY_STATS",
|
||||
default_value_t = false,
|
||||
action = clap::ArgAction::Set,
|
||||
)]
|
||||
pub stats: bool,
|
||||
|
||||
/// Where the savings ledger persists its state. When unset, the
|
||||
/// proxy uses `$HEADROOM_WORKSPACE_DIR/native_stats.json` when
|
||||
/// that env var is set (how the Python proxy resolves its
|
||||
/// workspace), else `~/.headroom/native_stats.json`; if no home
|
||||
/// directory can be resolved, stats stay in-memory (logged).
|
||||
///
|
||||
/// Source priority: CLI flag → `HEADROOM_PROXY_STATS_PATH`
|
||||
/// env var → workspace-derived default.
|
||||
#[arg(long = "stats-path", env = "HEADROOM_PROXY_STATS_PATH")]
|
||||
pub stats_path: Option<std::path::PathBuf>,
|
||||
}
|
||||
|
||||
fn parse_duration(s: &str) -> Result<Duration, String> {
|
||||
@@ -708,6 +736,11 @@ pub struct Config {
|
||||
/// PR-D4: GCP ADC OAuth scope used when fetching the bearer
|
||||
/// token. Default `https://www.googleapis.com/auth/cloud-platform`.
|
||||
pub vertex_adc_scope: String,
|
||||
/// Native savings stats: record per-request telemetry and serve
|
||||
/// `/stats`, `/stats/timeseries`, `/stats/events`, `/dashboard`.
|
||||
pub stats: bool,
|
||||
/// Ledger persistence path. `None` keeps stats in-memory.
|
||||
pub stats_path: Option<std::path::PathBuf>,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
@@ -775,6 +808,8 @@ impl Config {
|
||||
bedrock_validate_eventstream_crc: args.bedrock_validate_eventstream_crc,
|
||||
vertex_region: args.vertex_region,
|
||||
vertex_adc_scope: args.vertex_adc_scope,
|
||||
stats: args.stats,
|
||||
stats_path: args.stats_path.or_else(default_stats_path),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -838,10 +873,47 @@ impl Config {
|
||||
// only; the upstream URL is `upstream`).
|
||||
vertex_region: "us-central1".to_string(),
|
||||
vertex_adc_scope: "https://www.googleapis.com/auth/cloud-platform".to_string(),
|
||||
// Stats on (routes mounted, recording active) but
|
||||
// in-memory: tests never touch a real home directory.
|
||||
stats: true,
|
||||
stats_path: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Default persistence location for the savings ledger. Honours the
|
||||
/// same `HEADROOM_WORKSPACE_DIR` override the Python proxy uses for
|
||||
/// its workspace, falling back to `~/.headroom`. `None` (no home
|
||||
/// resolvable) keeps stats in-memory.
|
||||
fn default_stats_path() -> Option<std::path::PathBuf> {
|
||||
if let Ok(dir) = std::env::var("HEADROOM_WORKSPACE_DIR") {
|
||||
let dir = dir.trim();
|
||||
if !dir.is_empty() {
|
||||
// Tilde-expand, matching the Python proxy's
|
||||
// `workspace_dir()` semantics (its tests pin
|
||||
// `HEADROOM_WORKSPACE_DIR=~/custom` → `$HOME/custom`).
|
||||
let expanded = if let Some(rest) = dir.strip_prefix("~/") {
|
||||
match home_dir() {
|
||||
Some(home) => home.join(rest),
|
||||
None => return None,
|
||||
}
|
||||
} else {
|
||||
std::path::PathBuf::from(dir)
|
||||
};
|
||||
return Some(expanded.join("native_stats.json"));
|
||||
}
|
||||
}
|
||||
Some(home_dir()?.join(".headroom").join("native_stats.json"))
|
||||
}
|
||||
|
||||
fn home_dir() -> Option<std::path::PathBuf> {
|
||||
std::env::var("HOME")
|
||||
.ok()
|
||||
.filter(|h| !h.trim().is_empty())
|
||||
.or_else(|| std::env::var("USERPROFILE").ok())
|
||||
.map(std::path::PathBuf::from)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod rollout_input_tests {
|
||||
use super::*;
|
||||
@@ -885,4 +957,22 @@ mod rollout_input_tests {
|
||||
assert!(!config.enable_responses_streaming);
|
||||
assert!(!config.enable_bedrock_native);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn native_stats_require_explicit_opt_in() {
|
||||
let base = ["headroom-proxy", "--upstream", "http://127.0.0.1:9"];
|
||||
let default_config = Config::from_cli(CliArgs::try_parse_from(base).unwrap());
|
||||
assert!(!default_config.stats);
|
||||
|
||||
let enabled = Config::from_cli(
|
||||
CliArgs::try_parse_from([
|
||||
"headroom-proxy",
|
||||
"--upstream",
|
||||
"http://127.0.0.1:9",
|
||||
"--stats=true",
|
||||
])
|
||||
.unwrap(),
|
||||
);
|
||||
assert!(enabled.stats);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -105,6 +105,42 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
}
|
||||
}
|
||||
|
||||
// Savings observability: with the master compression switch on
|
||||
// but the mode off, the dispatcher passes every body through and
|
||||
// tokens_saved reads 0 on the dashboard with no hint why. Say so
|
||||
// once at startup instead of letting the zero look like a bug.
|
||||
if config.compression
|
||||
&& matches!(
|
||||
config.compression_mode,
|
||||
headroom_proxy::config::CompressionMode::Off
|
||||
)
|
||||
{
|
||||
tracing::warn!(
|
||||
event = "compression_mode_off",
|
||||
"--compression is on but --compression-mode is `off`: bodies \
|
||||
pass through unmodified and the dashboard's compression \
|
||||
savings will read 0. Set --compression-mode live_zone to \
|
||||
engage the dispatcher."
|
||||
);
|
||||
}
|
||||
|
||||
// Savings ledger: background persistence while serving, plus a
|
||||
// best-effort final flush after the server drains (capture tasks
|
||||
// are detached, so a request finishing in the same instant as
|
||||
// shutdown may still miss it — see observability::capture docs).
|
||||
let stats_ledger = state.stats.clone();
|
||||
if config.stats {
|
||||
// Pay the vendored price-table parse now, not on the first
|
||||
// recorded request.
|
||||
headroom_proxy::observability::pricing::warm();
|
||||
tracing::info!(
|
||||
event = "stats_enabled",
|
||||
path = ?stats_ledger.path().map(|p| p.display().to_string()),
|
||||
"native savings stats active (/stats, /dashboard)"
|
||||
);
|
||||
headroom_proxy::observability::ledger::spawn_flusher(stats_ledger.clone());
|
||||
}
|
||||
|
||||
let app = build_app(state).into_make_service_with_connect_info::<SocketAddr>();
|
||||
|
||||
let listener = tokio::net::TcpListener::bind(config.listen).await?;
|
||||
@@ -122,6 +158,8 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
})
|
||||
.await?;
|
||||
|
||||
stats_ledger.flush().await;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,594 @@
|
||||
//! Response-usage capture — Phase H. Shared by every lane.
|
||||
//!
|
||||
//! Recording a request's savings needs the *response's* token usage,
|
||||
//! which arrives in four wire shapes:
|
||||
//!
|
||||
//! - JSON bodies (non-streaming Anthropic / OpenAI / Bedrock /
|
||||
//! Vertex responses),
|
||||
//! - Anthropic-style SSE streams (direct, Vertex, and Bedrock's
|
||||
//! SSE-translated mode) — already accumulated by
|
||||
//! [`crate::sse::anthropic::AnthropicStreamState`],
|
||||
//! - Converse-native frames whose usage keys are camelCase and carry
|
||||
//! no Anthropic `type` (the state machine drops them),
|
||||
//! - Bedrock binary EventStream passthrough, where nothing parses
|
||||
//! the frames today.
|
||||
//!
|
||||
//! This module provides one [`PendingRecord`] (a request's
|
||||
//! half-built ledger event plus the deferred-finalization contract:
|
||||
//! **record only after the response is fully observed**, so
|
||||
//! streaming token counts are real instead of zero) and the capture
|
||||
//! tasks that feed it. Every capture path is a bounded, best-effort
|
||||
//! tee off the client byte path — a stalled or hostile response can
|
||||
//! delay telemetry, never the client.
|
||||
//!
|
||||
//! Best-effort has one deliberate edge: capture tasks are plain
|
||||
//! `tokio::spawn`s with no shutdown drain, so requests still
|
||||
//! in-flight when the process exits are dropped without recording.
|
||||
//! Tracking them against shutdown would couple the byte path to
|
||||
//! telemetry lifetime — the wrong trade for a stats ledger.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
use base64::Engine as _;
|
||||
use serde_json::Value;
|
||||
|
||||
use super::ledger::{Ledger, RequestEvent, Usage};
|
||||
|
||||
/// Cap for buffered non-streaming response bodies awaiting a usage
|
||||
/// parse. Bigger bodies finalize with zero usage (tokens/USD from
|
||||
/// compression estimates only) rather than growing memory.
|
||||
const JSON_CAPTURE_CAP: usize = 2 * 1024 * 1024;
|
||||
|
||||
/// Queue depth for every telemetry tee (capture bodies and the SSE
|
||||
/// parser feeds in proxy/bedrock/vertex). Bounded so a stalled
|
||||
/// parser task can never make the byte path wait: senders `try_send`
|
||||
/// and drop on a full queue.
|
||||
pub(crate) const TEE_QUEUE_DEPTH: usize = 256;
|
||||
|
||||
/// A request's ledger event, waiting for its response to finish.
|
||||
///
|
||||
/// Constructed when the request is forwarded; finalized exactly once
|
||||
/// when the response is fully observed (stream closed / body ended /
|
||||
/// transport failed). Latency is measured here so every lane reports
|
||||
/// the same thing: time from forwarding decision to response fully
|
||||
/// observed.
|
||||
pub struct PendingRecord {
|
||||
ledger: Arc<Ledger>,
|
||||
event: RequestEvent,
|
||||
started: Instant,
|
||||
}
|
||||
|
||||
impl PendingRecord {
|
||||
pub fn new(ledger: Arc<Ledger>, event: RequestEvent, started: Instant) -> Self {
|
||||
Self {
|
||||
ledger,
|
||||
event,
|
||||
started,
|
||||
}
|
||||
}
|
||||
|
||||
/// Mark the upstream outcome failed (non-2xx or transport error)
|
||||
/// before finalization.
|
||||
pub fn mark_failed(&mut self) {
|
||||
self.event.failed = true;
|
||||
}
|
||||
|
||||
/// Supply the model id the request side knows (path parameter,
|
||||
/// or the buffered request body when compression is on).
|
||||
pub fn set_model(&mut self, model: &str) {
|
||||
self.event.set_model(model);
|
||||
}
|
||||
|
||||
/// Supply the model id learned from the *response*. Recording is
|
||||
/// not gated on the compression buffer (compression defaults
|
||||
/// off), so on a pure passthrough request this is the only place
|
||||
/// the model is known. Never overrides a request-side value.
|
||||
pub fn set_model_if_unknown(&mut self, model: &str) {
|
||||
if self.event.model_is_unknown() {
|
||||
self.event.set_model(model);
|
||||
}
|
||||
}
|
||||
|
||||
/// Attach the compression dispatcher's before/after token
|
||||
/// estimate and the strategies it applied.
|
||||
pub fn set_compression(&mut self, before: u64, after: u64, transforms: Vec<String>) {
|
||||
self.event.tokens_before = before;
|
||||
self.event.tokens_after = after;
|
||||
self.event.transforms = transforms;
|
||||
}
|
||||
|
||||
/// Record with the captured usage. Consumes the record — each
|
||||
/// request records exactly once.
|
||||
pub fn finalize(mut self, usage: Usage) {
|
||||
self.event.usage = usage;
|
||||
self.event.latency_ms = self.started.elapsed().as_millis() as u64;
|
||||
self.ledger.record(self.event);
|
||||
}
|
||||
|
||||
/// Record a failure (zero usage, no savings — the ledger's
|
||||
/// failed-path accounting).
|
||||
pub fn finalize_failed(mut self) {
|
||||
self.mark_failed();
|
||||
self.finalize(Usage::default());
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the pending ledger event for a request entering a handler
|
||||
/// (`None` when `--stats` is off). One call site per lane keeps the
|
||||
/// "constructed at handler entry so every exit records exactly once"
|
||||
/// rule easy to audit.
|
||||
pub fn start_pending(
|
||||
state: &crate::proxy::AppState,
|
||||
provider: &'static str,
|
||||
model: &str,
|
||||
request_id: &str,
|
||||
) -> Option<PendingRecord> {
|
||||
state.config.stats.then(|| {
|
||||
PendingRecord::new(
|
||||
state.stats.clone(),
|
||||
RequestEvent::new(provider, model, request_id),
|
||||
Instant::now(),
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
/// Finalize a pending record as failed on a proxy-side rejection
|
||||
/// (bad path, missing credentials, signing failure, oversized body,
|
||||
/// …). Sugar for handler early-return paths, so "every request that
|
||||
/// enters a handler produces exactly one record" survives rejects: a
|
||||
/// credentials outage must show in `/stats` as a failure spike, not
|
||||
/// as zero traffic.
|
||||
pub fn finalize_rejected(pending: Option<PendingRecord>) {
|
||||
if let Some(p) = pending {
|
||||
p.finalize_failed();
|
||||
}
|
||||
}
|
||||
|
||||
/// Which JSON body shape to expect when parsing a non-streaming
|
||||
/// response for usage.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub enum ResponseShape {
|
||||
Anthropic,
|
||||
OpenAiChat,
|
||||
OpenAiResponses,
|
||||
/// Bedrock sync responses: InvokeModel bodies are Anthropic
|
||||
/// snake_case, Converse bodies are camelCase — try both.
|
||||
Bedrock,
|
||||
}
|
||||
|
||||
/// Extract usage from a complete response body. `None` when the
|
||||
/// body has no recognisable usage block (error envelopes, non-JSON).
|
||||
fn usage_from_response_json(shape: ResponseShape, v: &Value) -> Option<Usage> {
|
||||
usage_from_usage_block(shape, v.get("usage")?)
|
||||
}
|
||||
|
||||
/// Parse a bare `usage` object (already extracted from its
|
||||
/// envelope) — the SSE state machines hold usage in this form.
|
||||
pub fn usage_from_usage_block(shape: ResponseShape, usage: &Value) -> Option<Usage> {
|
||||
match shape {
|
||||
ResponseShape::Anthropic => snake_usage(usage),
|
||||
ResponseShape::OpenAiChat => openai_chat_usage(usage),
|
||||
ResponseShape::OpenAiResponses => openai_responses_usage(usage),
|
||||
ResponseShape::Bedrock => snake_usage(usage).or_else(|| camel_usage(usage)),
|
||||
}
|
||||
}
|
||||
|
||||
fn u64_at(v: &Value, key: &str) -> Option<u64> {
|
||||
v.get(key).and_then(Value::as_u64)
|
||||
}
|
||||
|
||||
/// Anthropic `usage` block (also Bedrock InvokeModel for Anthropic
|
||||
/// models): `input_tokens` already EXCLUDES cache reads/writes.
|
||||
fn snake_usage(usage: &Value) -> Option<Usage> {
|
||||
let input = u64_at(usage, "input_tokens");
|
||||
let output = u64_at(usage, "output_tokens");
|
||||
if input.is_none() && output.is_none() {
|
||||
return None;
|
||||
}
|
||||
Some(Usage {
|
||||
input_tokens: input.unwrap_or(0),
|
||||
output_tokens: output.unwrap_or(0),
|
||||
cache_read_tokens: u64_at(usage, "cache_read_input_tokens").unwrap_or(0),
|
||||
cache_write_tokens: u64_at(usage, "cache_creation_input_tokens").unwrap_or(0),
|
||||
})
|
||||
}
|
||||
|
||||
/// Bedrock Converse `usage` block: camelCase, and `inputTokens`
|
||||
/// likewise excludes cache tokens.
|
||||
///
|
||||
/// The cache fields fall back to the `*InputTokenCount` spelling
|
||||
/// after `*InputTokens`. Reading only one name would silently report
|
||||
/// zero cache tokens against a response that uses the other, and a
|
||||
/// miss costs one map lookup — cheap insurance on a field whose
|
||||
/// whole job is to be believed.
|
||||
fn camel_usage(usage: &Value) -> Option<Usage> {
|
||||
let input = u64_at(usage, "inputTokens");
|
||||
let output = u64_at(usage, "outputTokens");
|
||||
if input.is_none() && output.is_none() {
|
||||
return None;
|
||||
}
|
||||
Some(Usage {
|
||||
input_tokens: input.unwrap_or(0),
|
||||
output_tokens: output.unwrap_or(0),
|
||||
cache_read_tokens: u64_at(usage, "cacheReadInputTokens")
|
||||
.or_else(|| u64_at(usage, "cacheReadInputTokenCount"))
|
||||
.unwrap_or(0),
|
||||
cache_write_tokens: u64_at(usage, "cacheWriteInputTokens")
|
||||
.or_else(|| u64_at(usage, "cacheWriteInputTokenCount"))
|
||||
.unwrap_or(0),
|
||||
})
|
||||
}
|
||||
|
||||
/// OpenAI Chat Completions: `prompt_tokens` INCLUDES cached tokens —
|
||||
/// normalise to the ledger's Anthropic semantics (input = uncached).
|
||||
fn openai_chat_usage(usage: &Value) -> Option<Usage> {
|
||||
let prompt = u64_at(usage, "prompt_tokens")?;
|
||||
let cached = usage
|
||||
.get("prompt_tokens_details")
|
||||
.and_then(|d| u64_at(d, "cached_tokens"))
|
||||
.unwrap_or(0)
|
||||
.min(prompt);
|
||||
Some(Usage {
|
||||
input_tokens: prompt - cached,
|
||||
output_tokens: u64_at(usage, "completion_tokens").unwrap_or(0),
|
||||
cache_read_tokens: cached,
|
||||
cache_write_tokens: 0,
|
||||
})
|
||||
}
|
||||
|
||||
/// OpenAI Responses: `input_tokens` INCLUDES cached tokens — same
|
||||
/// normalisation.
|
||||
fn openai_responses_usage(usage: &Value) -> Option<Usage> {
|
||||
let input = u64_at(usage, "input_tokens")?;
|
||||
let cached = usage
|
||||
.get("input_tokens_details")
|
||||
.and_then(|d| u64_at(d, "cached_tokens"))
|
||||
.unwrap_or(0)
|
||||
.min(input);
|
||||
Some(Usage {
|
||||
input_tokens: input - cached,
|
||||
output_tokens: u64_at(usage, "output_tokens").unwrap_or(0),
|
||||
cache_read_tokens: cached,
|
||||
cache_write_tokens: 0,
|
||||
})
|
||||
}
|
||||
|
||||
/// Usage accumulated by the shared Anthropic SSE state machine.
|
||||
pub fn usage_from_anthropic_state(state: &crate::sse::anthropic::AnthropicStreamState) -> Usage {
|
||||
Usage {
|
||||
input_tokens: state.usage.input_tokens,
|
||||
output_tokens: state.usage.output_tokens,
|
||||
cache_read_tokens: state.usage.cache_read_input_tokens,
|
||||
cache_write_tokens: state.usage.cache_creation_input_tokens,
|
||||
}
|
||||
}
|
||||
|
||||
/// Fold one streaming frame's JSON payload into a usage accumulator.
|
||||
/// Handles every Bedrock/Anthropic stream shape (fields are monotone
|
||||
/// within a stream, so `max` merge is safe against repeats):
|
||||
///
|
||||
/// - Anthropic SSE events: `message_start.message.usage`,
|
||||
/// `message_delta.usage` (snake_case),
|
||||
/// - Converse-native frames: bare `usage` with camelCase keys
|
||||
/// (`metadata` events — no Anthropic `type` field at all),
|
||||
/// - Bedrock InvokeModel EventStream chunks: `{"bytes": "<base64>"}`
|
||||
/// wrapping an Anthropic SSE event (recursed after decode).
|
||||
pub fn merge_stream_usage(acc: &mut Usage, payload: &Value) {
|
||||
merge_stream_usage_depth(acc, payload, 0);
|
||||
}
|
||||
|
||||
/// Real Bedrock wraps exactly once (`{"bytes": <b64 of the Anthropic
|
||||
/// event>}`); the recursion exists only for that unwrap. Cap it so a
|
||||
/// crafted deeply-nested base64 payload can't blow the stack — a
|
||||
/// stack overflow aborts the whole process, not just this task.
|
||||
const MAX_BYTES_UNWRAP_DEPTH: u8 = 4;
|
||||
|
||||
fn merge_stream_usage_depth(acc: &mut Usage, payload: &Value, depth: u8) {
|
||||
// InvokeModel EventStream chunk: base64-wrapped inner event.
|
||||
if let Some(b64) = payload.get("bytes").and_then(Value::as_str) {
|
||||
if depth >= MAX_BYTES_UNWRAP_DEPTH {
|
||||
tracing::warn!(
|
||||
event = "stats_capture_bytes_depth_exceeded",
|
||||
depth = depth,
|
||||
"nested {{\"bytes\": …}} wrapping exceeded the unwrap cap; \
|
||||
ignoring the payload (real Bedrock wraps exactly once)"
|
||||
);
|
||||
return;
|
||||
}
|
||||
if let Ok(decoded) = base64::engine::general_purpose::STANDARD.decode(b64) {
|
||||
if let Ok(inner) = serde_json::from_slice::<Value>(&decoded) {
|
||||
merge_stream_usage_depth(acc, &inner, depth + 1);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
// message_start carries usage nested under `message`.
|
||||
if let Some(u) = payload
|
||||
.get("message")
|
||||
.and_then(|m| m.get("usage"))
|
||||
.and_then(snake_usage)
|
||||
{
|
||||
merge_max(acc, u);
|
||||
}
|
||||
// message_delta (and some terminal frames) carry a bare `usage`.
|
||||
if let Some(usage) = payload.get("usage") {
|
||||
if let Some(u) = snake_usage(usage).or_else(|| camel_usage(usage)) {
|
||||
merge_max(acc, u);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Take the field-wise maximum of two usage snapshots (usage fields
|
||||
/// are monotone within a stream, so `max` merge is repeat-safe).
|
||||
pub(crate) fn merge_max(acc: &mut Usage, u: Usage) {
|
||||
acc.input_tokens = acc.input_tokens.max(u.input_tokens);
|
||||
acc.output_tokens = acc.output_tokens.max(u.output_tokens);
|
||||
acc.cache_read_tokens = acc.cache_read_tokens.max(u.cache_read_tokens);
|
||||
acc.cache_write_tokens = acc.cache_write_tokens.max(u.cache_write_tokens);
|
||||
}
|
||||
|
||||
/// Bounded tee → JSON-body usage capture. Returns the sender to tee
|
||||
/// response chunks into; finalizes `pending` when the channel
|
||||
/// closes (i.e. the response body finished or the client hung up).
|
||||
pub fn spawn_json_usage_capture(
|
||||
shape: ResponseShape,
|
||||
mut pending: PendingRecord,
|
||||
) -> tokio::sync::mpsc::Sender<bytes::Bytes> {
|
||||
let (tx, mut rx) = tokio::sync::mpsc::channel::<bytes::Bytes>(TEE_QUEUE_DEPTH);
|
||||
tokio::spawn(async move {
|
||||
let mut buf: Vec<u8> = Vec::new();
|
||||
let mut truncated = false;
|
||||
while let Some(chunk) = rx.recv().await {
|
||||
if truncated {
|
||||
continue; // keep draining so the tee never backpressures
|
||||
}
|
||||
if buf.len() + chunk.len() > JSON_CAPTURE_CAP {
|
||||
// No silent fallbacks: the request still records, but
|
||||
// its usage will read zero — say why, once.
|
||||
tracing::warn!(
|
||||
event = "stats_capture_body_too_large",
|
||||
cap_bytes = JSON_CAPTURE_CAP,
|
||||
"response body exceeded the usage-capture buffer; \
|
||||
this request records with compression-side numbers \
|
||||
only (tokens/USD from usage read as 0)"
|
||||
);
|
||||
truncated = true;
|
||||
buf.clear();
|
||||
continue;
|
||||
}
|
||||
buf.extend_from_slice(&chunk);
|
||||
}
|
||||
let mut usage = Usage::default();
|
||||
if !truncated {
|
||||
match serde_json::from_slice::<Value>(&buf) {
|
||||
Ok(v) => {
|
||||
if let Some(model) = v.get("model").and_then(Value::as_str) {
|
||||
pending.set_model_if_unknown(model);
|
||||
}
|
||||
// A 2xx from an LLM endpoint carries a usage block
|
||||
// in every shape we know. A miss means the wire
|
||||
// format moved (or this isn't the body we think it
|
||||
// is), and the request would silently record $0 —
|
||||
// the phantom-zero this module exists to avoid.
|
||||
// Same no-silent-fallbacks rule as the cap above.
|
||||
match usage_from_response_json(shape, &v) {
|
||||
Some(u) => usage = u,
|
||||
None => tracing::warn!(
|
||||
event = "stats_capture_no_usage_block",
|
||||
shape = ?shape,
|
||||
"response parsed as JSON but carried no recognisable \
|
||||
usage block; recording this request with zero tokens \
|
||||
and $0 spend"
|
||||
),
|
||||
}
|
||||
}
|
||||
Err(e) => tracing::warn!(
|
||||
event = "stats_capture_body_not_json",
|
||||
error = %e,
|
||||
"response body did not parse as JSON; recording this request \
|
||||
with zero tokens and $0 spend"
|
||||
),
|
||||
}
|
||||
}
|
||||
pending.finalize(usage);
|
||||
});
|
||||
tx
|
||||
}
|
||||
|
||||
/// Bounded tee → Bedrock binary EventStream usage capture, for the
|
||||
/// passthrough mode where no other parser sees the frames. CRC
|
||||
/// validation is off — this is the telemetry side; enforcement
|
||||
/// happens (or not, per config) on the byte path.
|
||||
pub fn spawn_eventstream_usage_capture(
|
||||
pending: PendingRecord,
|
||||
) -> tokio::sync::mpsc::Sender<bytes::Bytes> {
|
||||
use crate::bedrock::eventstream::{CrcValidation, EventStreamParser};
|
||||
let (tx, mut rx) = tokio::sync::mpsc::channel::<bytes::Bytes>(TEE_QUEUE_DEPTH);
|
||||
tokio::spawn(async move {
|
||||
let mut parser = EventStreamParser::new().with_crc_validation(CrcValidation::No);
|
||||
let mut usage = Usage::default();
|
||||
'outer: while let Some(chunk) = rx.recv().await {
|
||||
parser.push(&chunk);
|
||||
loop {
|
||||
match parser.next_message() {
|
||||
Ok(Some(msg)) => {
|
||||
if let Ok(payload) = serde_json::from_slice::<Value>(&msg.payload) {
|
||||
merge_stream_usage(&mut usage, &payload);
|
||||
}
|
||||
}
|
||||
Ok(None) => break,
|
||||
Err(_) => {
|
||||
// Telemetry parser out of sync — stop and
|
||||
// record what accumulated so far. The channel
|
||||
// is simply dropped; the tee's `try_send` on a
|
||||
// closed channel is a no-op, so the byte path
|
||||
// never notices.
|
||||
break 'outer;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
pending.finalize(usage);
|
||||
});
|
||||
tx
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
|
||||
#[test]
|
||||
fn anthropic_json_usage_parses() {
|
||||
let v = json!({"usage": {"input_tokens": 10, "output_tokens": 5,
|
||||
"cache_read_input_tokens": 7, "cache_creation_input_tokens": 3}});
|
||||
let u = usage_from_response_json(ResponseShape::Anthropic, &v).unwrap();
|
||||
assert_eq!(
|
||||
u,
|
||||
Usage {
|
||||
input_tokens: 10,
|
||||
output_tokens: 5,
|
||||
cache_read_tokens: 7,
|
||||
cache_write_tokens: 3
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn openai_chat_normalises_cached_out_of_prompt() {
|
||||
let v = json!({"usage": {"prompt_tokens": 100, "completion_tokens": 20,
|
||||
"prompt_tokens_details": {"cached_tokens": 60}}});
|
||||
let u = usage_from_response_json(ResponseShape::OpenAiChat, &v).unwrap();
|
||||
assert_eq!(u.input_tokens, 40, "prompt minus cached");
|
||||
assert_eq!(u.cache_read_tokens, 60);
|
||||
assert_eq!(u.output_tokens, 20);
|
||||
// Pathological cached > prompt clamps instead of underflowing.
|
||||
let v = json!({"usage": {"prompt_tokens": 10,
|
||||
"prompt_tokens_details": {"cached_tokens": 60}}});
|
||||
let u = usage_from_response_json(ResponseShape::OpenAiChat, &v).unwrap();
|
||||
assert_eq!(u.input_tokens, 0);
|
||||
assert_eq!(u.cache_read_tokens, 10);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn openai_responses_normalises_cached_out_of_input() {
|
||||
let v = json!({"usage": {"input_tokens": 80, "output_tokens": 9,
|
||||
"input_tokens_details": {"cached_tokens": 50}}});
|
||||
let u = usage_from_response_json(ResponseShape::OpenAiResponses, &v).unwrap();
|
||||
assert_eq!(u.input_tokens, 30);
|
||||
assert_eq!(u.cache_read_tokens, 50);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bedrock_shape_accepts_snake_and_camel() {
|
||||
let snake = json!({"usage": {"input_tokens": 10, "output_tokens": 5}});
|
||||
let camel = json!({"usage": {"inputTokens": 11, "outputTokens": 6,
|
||||
"cacheReadInputTokens": 4, "cacheWriteInputTokens": 2}});
|
||||
assert_eq!(
|
||||
usage_from_response_json(ResponseShape::Bedrock, &snake)
|
||||
.unwrap()
|
||||
.input_tokens,
|
||||
10
|
||||
);
|
||||
let u = usage_from_response_json(ResponseShape::Bedrock, &camel).unwrap();
|
||||
assert_eq!(u.input_tokens, 11);
|
||||
assert_eq!(u.cache_read_tokens, 4);
|
||||
assert_eq!(u.cache_write_tokens, 2);
|
||||
}
|
||||
|
||||
/// The cache fields accept the `*InputTokenCount` spelling too.
|
||||
/// Untested, this fallback reads as dead code and invites
|
||||
/// deletion — at which point any response using that spelling
|
||||
/// silently reports zero cache tokens.
|
||||
#[test]
|
||||
fn bedrock_camel_usage_accepts_token_count_suffix() {
|
||||
let v = json!({"usage": {"inputTokens": 11, "outputTokens": 6,
|
||||
"cacheReadInputTokenCount": 4, "cacheWriteInputTokenCount": 2}});
|
||||
let u = usage_from_response_json(ResponseShape::Bedrock, &v).unwrap();
|
||||
assert_eq!(u.cache_read_tokens, 4);
|
||||
assert_eq!(u.cache_write_tokens, 2);
|
||||
|
||||
// The unsuffixed spelling wins when both are present.
|
||||
let both = json!({"usage": {"inputTokens": 11, "outputTokens": 6,
|
||||
"cacheReadInputTokens": 9, "cacheReadInputTokenCount": 4}});
|
||||
let u = usage_from_response_json(ResponseShape::Bedrock, &both).unwrap();
|
||||
assert_eq!(u.cache_read_tokens, 9);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn error_envelope_yields_none() {
|
||||
let v = json!({"error": {"type": "overloaded_error"}});
|
||||
assert!(usage_from_response_json(ResponseShape::Anthropic, &v).is_none());
|
||||
let v = json!({"usage": {"totalTokens": 5}});
|
||||
assert!(usage_from_response_json(ResponseShape::Bedrock, &v).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stream_merge_handles_all_three_wire_shapes() {
|
||||
let mut acc = Usage::default();
|
||||
// 1. Anthropic message_start (nested under message).
|
||||
merge_stream_usage(
|
||||
&mut acc,
|
||||
&json!({"type": "message_start",
|
||||
"message": {"usage": {"input_tokens": 25, "output_tokens": 1,
|
||||
"cache_read_input_tokens": 9}}}),
|
||||
);
|
||||
// 2. Anthropic message_delta (bare usage, growing output).
|
||||
merge_stream_usage(
|
||||
&mut acc,
|
||||
&json!({"type": "message_delta", "usage": {"output_tokens": 42}}),
|
||||
);
|
||||
// 3. Converse metadata (camelCase, no `type`).
|
||||
merge_stream_usage(
|
||||
&mut acc,
|
||||
&json!({"usage": {"inputTokens": 25, "outputTokens": 42,
|
||||
"cacheWriteInputTokens": 3}}),
|
||||
);
|
||||
assert_eq!(acc.input_tokens, 25);
|
||||
assert_eq!(acc.output_tokens, 42);
|
||||
assert_eq!(acc.cache_read_tokens, 9);
|
||||
assert_eq!(acc.cache_write_tokens, 3);
|
||||
}
|
||||
|
||||
/// A crafted payload nesting `{"bytes": …}` inside itself must
|
||||
/// hit the unwrap cap instead of recursing until the stack blows
|
||||
/// (a stack overflow aborts the whole process). Real Bedrock
|
||||
/// wraps exactly once.
|
||||
#[test]
|
||||
fn stream_merge_bounds_nested_base64_depth() {
|
||||
let engine = &base64::engine::general_purpose::STANDARD;
|
||||
// usage buried 10 wraps deep — far past the cap.
|
||||
let mut v = json!({"type": "message_delta", "usage": {"output_tokens": 99}});
|
||||
for _ in 0..10 {
|
||||
v = json!({"bytes": engine.encode(v.to_string())});
|
||||
}
|
||||
let mut acc = Usage::default();
|
||||
merge_stream_usage(&mut acc, &v); // must return, not overflow
|
||||
assert_eq!(
|
||||
acc.output_tokens, 0,
|
||||
"usage beyond the unwrap cap is ignored, not chased"
|
||||
);
|
||||
// …while the real single-wrap shape still works.
|
||||
let single = json!({"bytes": engine.encode(
|
||||
json!({"type": "message_delta", "usage": {"output_tokens": 7}}).to_string()
|
||||
)});
|
||||
merge_stream_usage(&mut acc, &single);
|
||||
assert_eq!(acc.output_tokens, 7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stream_merge_decodes_invoke_model_base64_chunks() {
|
||||
let inner = json!({"type": "message_delta",
|
||||
"usage": {"output_tokens": 17, "input_tokens": 12}});
|
||||
let b64 =
|
||||
base64::engine::general_purpose::STANDARD.encode(serde_json::to_vec(&inner).unwrap());
|
||||
let mut acc = Usage::default();
|
||||
merge_stream_usage(&mut acc, &json!({"bytes": b64, "p": "pad"}));
|
||||
assert_eq!(acc.output_tokens, 17);
|
||||
assert_eq!(acc.input_tokens, 12);
|
||||
// Garbage base64 is ignored, never panics.
|
||||
merge_stream_usage(&mut acc, &json!({"bytes": "!!!not-base64!!!"}));
|
||||
assert_eq!(acc.output_tokens, 17);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,722 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Headroom Proxy — Savings</title>
|
||||
<!-- Inline icon: keeps the page fully self-contained (the proxy's
|
||||
/favicon.ico route serves a generic icon; this one is the
|
||||
dashboard's own). -->
|
||||
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Crect width='16' height='16' rx='3' fill='%232a78d6'/%3E%3Cpath d='M4 11.5V4.5M8 11.5V7M12 11.5V6' stroke='white' stroke-width='1.8' stroke-linecap='round'/%3E%3C/svg%3E">
|
||||
<style>
|
||||
:root {
|
||||
color-scheme: light;
|
||||
--page: #f9f9f7;
|
||||
--surface: #fcfcfb;
|
||||
--ink: #0b0b0b;
|
||||
--ink-2: #52514e;
|
||||
--muted: #898781;
|
||||
--grid: #e1e0d9;
|
||||
--baseline: #c3c2b7;
|
||||
--border: rgba(11, 11, 11, 0.10);
|
||||
--series-1: #2a78d6; /* spent (also anthropic/bedrock keys) */
|
||||
--series-2: #008300; /* saved (also openai/vertex keys) */
|
||||
--good: #0ca30c;
|
||||
--good-text: #006300;
|
||||
--critical: #d03b3b;
|
||||
--wash: rgba(11, 11, 11, 0.04);
|
||||
--shadow: rgba(0, 0, 0, 0.18);
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
color-scheme: dark;
|
||||
--page: #0d0d0d;
|
||||
--surface: #1a1a19;
|
||||
--ink: #ffffff;
|
||||
--ink-2: #c3c2b7;
|
||||
--muted: #898781;
|
||||
--grid: #2c2c2a;
|
||||
--baseline: #383835;
|
||||
--border: rgba(255, 255, 255, 0.10);
|
||||
--series-1: #3987e5;
|
||||
--series-2: #008300;
|
||||
--good-text: #0ca30c;
|
||||
/* #d03b3b is 3.6:1 on this surface — under AA for text, and
|
||||
--critical IS text here (the "✕ N failed" badge), not a dot.
|
||||
Lightened to clear 4.5:1 on #1a1a19. */
|
||||
--critical: #f26a6a;
|
||||
--wash: rgba(255, 255, 255, 0.06);
|
||||
--shadow: rgba(0, 0, 0, 0.6);
|
||||
}
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0;
|
||||
background: var(--page);
|
||||
color: var(--ink);
|
||||
font: 14px/1.45 system-ui, -apple-system, "Segoe UI", sans-serif;
|
||||
}
|
||||
.wrap { max-width: 1180px; margin: 0 auto; padding: 20px 24px 48px; }
|
||||
|
||||
header { display: flex; align-items: baseline; gap: 12px; flex-wrap: wrap; margin: 4px 0 16px; }
|
||||
header h1 { font-size: 17px; font-weight: 650; margin: 0; }
|
||||
header .meta { color: var(--muted); font-size: 12.5px; }
|
||||
header .spacer { flex: 1; }
|
||||
.dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; margin-right: 5px; vertical-align: 1px; }
|
||||
|
||||
.filters { display: flex; align-items: center; gap: 16px; flex-wrap: wrap; margin-bottom: 14px; }
|
||||
.seg { display: inline-flex; border: 1px solid var(--border); border-radius: 8px; overflow: hidden; background: var(--surface); }
|
||||
.seg button {
|
||||
appearance: none; border: 0; background: transparent; color: var(--ink-2);
|
||||
font: inherit; font-size: 12.5px; padding: 5px 12px; cursor: pointer;
|
||||
}
|
||||
.seg button:hover { background: var(--wash); }
|
||||
.seg button[aria-pressed="true"] { color: var(--ink); font-weight: 650; background: var(--wash); }
|
||||
.seg button[aria-pressed="true"]::before { content: "✓ "; font-weight: 700; }
|
||||
.filters .flabel { color: var(--muted); font-size: 12px; }
|
||||
|
||||
.cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(215px, 1fr)); gap: 12px; margin-bottom: 14px; }
|
||||
.card {
|
||||
background: var(--surface); border: 1px solid var(--border);
|
||||
border-radius: 10px; padding: 14px 16px; min-width: 0;
|
||||
}
|
||||
.card .label { color: var(--ink-2); font-size: 12.5px; }
|
||||
.card .value { font-size: 26px; font-weight: 650; margin-top: 2px; }
|
||||
.card .value.hero { font-size: 48px; line-height: 1.1; letter-spacing: -0.5px; }
|
||||
.card .sub { color: var(--muted); font-size: 12px; margin-top: 2px; }
|
||||
.card .sub .up { color: var(--good-text); }
|
||||
.card .sub .bad { color: var(--critical); }
|
||||
.card svg.spark { display: block; margin-top: 8px; }
|
||||
|
||||
section.panel {
|
||||
background: var(--surface); border: 1px solid var(--border);
|
||||
border-radius: 10px; padding: 16px; margin-bottom: 14px;
|
||||
}
|
||||
.panel h2 { font-size: 13.5px; font-weight: 650; margin: 0 0 2px; }
|
||||
.panel .desc { color: var(--muted); font-size: 12px; margin: 0 0 10px; }
|
||||
.legend { display: flex; gap: 16px; align-items: center; font-size: 12px; color: var(--ink-2); margin: 2px 0 8px; }
|
||||
.legend .key { display: inline-flex; align-items: center; gap: 6px; }
|
||||
.legend .swatch { width: 10px; height: 10px; border-radius: 3px; }
|
||||
|
||||
.chart-holder { position: relative; }
|
||||
#chart { display: block; width: 100%; }
|
||||
/* The hovered/focused column lifts so the mark visibly responds.
|
||||
Bars are <path> (rounded data-end), so key off the class alone. */
|
||||
#chart .bar { transition: filter 90ms ease-out; }
|
||||
#chart g.col:hover .bar, #chart g.col:focus .bar { filter: brightness(1.15); }
|
||||
/* Keyboard focus needs a visible indicator beyond the brightness
|
||||
lift (WCAG 2.4.7): stroke the focused column's marks in ink. */
|
||||
#chart g.col:focus { outline: none; }
|
||||
#chart g.col:focus .bar { stroke: var(--ink); stroke-width: 1.5; }
|
||||
@media (prefers-reduced-motion: reduce) { #chart .bar { transition: none; } }
|
||||
.tooltip {
|
||||
position: absolute; pointer-events: none; z-index: 5; display: none;
|
||||
background: var(--surface); border: 1px solid var(--border); border-radius: 8px;
|
||||
padding: 8px 10px; box-shadow: 0 4px 14px var(--shadow); font-size: 12px; min-width: 170px;
|
||||
}
|
||||
.tooltip .t-when { color: var(--muted); margin-bottom: 4px; }
|
||||
.tooltip .t-row { display: flex; align-items: center; gap: 7px; margin-top: 2px; }
|
||||
.tooltip .t-key { width: 12px; height: 0; border-top: 3px solid; border-radius: 2px; }
|
||||
.tooltip .t-val { font-weight: 650; margin-left: auto; font-variant-numeric: tabular-nums; }
|
||||
.tooltip .t-name { color: var(--ink-2); }
|
||||
|
||||
.grid2 { display: grid; grid-template-columns: 1fr 1.4fr; gap: 12px; }
|
||||
@media (max-width: 900px) { .grid2 { grid-template-columns: 1fr; } }
|
||||
|
||||
table { width: 100%; border-collapse: collapse; font-size: 12.5px; }
|
||||
th {
|
||||
text-align: right; color: var(--muted); font-weight: 500; font-size: 11.5px;
|
||||
padding: 4px 8px; border-bottom: 1px solid var(--grid); white-space: nowrap;
|
||||
}
|
||||
th:first-child, td:first-child { text-align: left; padding-left: 2px; }
|
||||
td {
|
||||
text-align: right; padding: 5px 8px; border-bottom: 1px solid var(--grid);
|
||||
font-variant-numeric: tabular-nums; white-space: nowrap;
|
||||
}
|
||||
tr:last-child td { border-bottom: 0; }
|
||||
tr:hover td { background: var(--wash); }
|
||||
td.name { max-width: 260px; overflow: hidden; text-overflow: ellipsis; color: var(--ink); }
|
||||
td .provider-key { display: inline-block; width: 8px; height: 8px; border-radius: 2px; margin-right: 7px; vertical-align: 0; }
|
||||
.badge-fail { color: var(--critical); font-weight: 650; }
|
||||
.empty { color: var(--muted); font-size: 12.5px; padding: 14px 2px; }
|
||||
.footnote { color: var(--muted); font-size: 11.5px; margin-top: 8px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
<header>
|
||||
<h1>Headroom Proxy — Savings</h1>
|
||||
<span class="meta" id="meta-version"></span>
|
||||
<span class="meta" id="meta-uptime"></span>
|
||||
<span class="spacer"></span>
|
||||
<span class="meta" id="meta-persist"></span>
|
||||
<span class="meta" id="meta-updated"></span>
|
||||
</header>
|
||||
|
||||
<div class="filters" role="toolbar" aria-label="Scope and history filters">
|
||||
<span class="flabel">Scope</span>
|
||||
<div class="seg" id="scope-seg">
|
||||
<button data-scope="session" aria-pressed="true">Session</button>
|
||||
<button data-scope="lifetime" aria-pressed="false">Lifetime</button>
|
||||
</div>
|
||||
<span class="flabel">History</span>
|
||||
<div class="seg" id="bucket-seg">
|
||||
<button data-bucket="hour" aria-pressed="false">48 hours</button>
|
||||
<button data-bucket="day" aria-pressed="true">Daily</button>
|
||||
<button data-bucket="week" aria-pressed="false">Weekly</button>
|
||||
<button data-bucket="month" aria-pressed="false">Monthly</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="cards" id="kpis"></div>
|
||||
|
||||
<section class="panel">
|
||||
<h2>Spend vs savings over time</h2>
|
||||
<p class="desc" id="chart-desc">Bar height is what this traffic would have cost without Headroom.</p>
|
||||
<div class="legend" id="chart-legend"></div>
|
||||
<div class="chart-holder">
|
||||
<svg id="chart" height="230" role="img" aria-label="Savings over time"></svg>
|
||||
<div class="tooltip" id="tooltip"></div>
|
||||
</div>
|
||||
<div class="empty" id="chart-empty" style="display:none">No traffic recorded yet.</div>
|
||||
</section>
|
||||
|
||||
<div class="grid2">
|
||||
<section class="panel">
|
||||
<h2>By provider</h2>
|
||||
<p class="desc">Requests routed through each lane (<span id="scope-label-1"></span>).</p>
|
||||
<div id="providers"></div>
|
||||
</section>
|
||||
<section class="panel">
|
||||
<h2>By model</h2>
|
||||
<p class="desc">Top models by USD saved (lifetime).</p>
|
||||
<div id="models"></div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<section class="panel">
|
||||
<h2>Recent requests</h2>
|
||||
<p class="desc">Last requests through the proxy (newest first). Failed requests accrue no savings.</p>
|
||||
<div id="feed"></div>
|
||||
<p class="footnote">Savings are valued from the vendored LiteLLM price table; models missing from it record $0 (logged by the proxy).</p>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
"use strict";
|
||||
(() => {
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const state = { scope: "session", bucket: "day", stats: null, series: null, events: null, seq: 0 };
|
||||
|
||||
// ── formatting ──────────────────────────────────────────────
|
||||
const COMPACT = new Intl.NumberFormat("en-US", {
|
||||
notation: "compact",
|
||||
maximumFractionDigits: 1,
|
||||
});
|
||||
// Full digits below 10k — "9,999 tokens" reads better than "10K",
|
||||
// and compact notation alone would abbreviate from 1,000.
|
||||
const fmtInt = (n) => {
|
||||
n = Number(n) || 0;
|
||||
return Math.abs(n) >= 1e4 ? COMPACT.format(n) : n.toLocaleString("en-US");
|
||||
};
|
||||
// Spend spans many orders of magnitude here: a dev smoke-testing
|
||||
// sees fractions of a cent, a fleet sees dollars. Fixed decimals
|
||||
// collapse the small end to a useless "$0.0000", so sub-cent
|
||||
// values fall back to significant digits.
|
||||
const fmtUsd = (n) => {
|
||||
n = Number(n) || 0;
|
||||
if (n < 0) return "-" + fmtUsd(-n);
|
||||
if (n === 0) return "$0";
|
||||
if (n >= 1000) return "$" + (n / 1000).toFixed(1) + "K";
|
||||
if (n >= 1) return "$" + n.toFixed(2);
|
||||
if (n >= 0.01) return "$" + n.toFixed(3);
|
||||
return "$" + n.toPrecision(2);
|
||||
};
|
||||
const fmtPct = (n) => (Number(n) || 0).toFixed(1) + "%";
|
||||
const fmtUptime = (s) => {
|
||||
s = Number(s) || 0;
|
||||
if (s < 90) return s + "s";
|
||||
if (s < 5400) return Math.round(s / 60) + "m";
|
||||
if (s < 172800) return (s / 3600).toFixed(1) + "h";
|
||||
return Math.round(s / 86400) + "d";
|
||||
};
|
||||
const el = (tag, cls, text) => {
|
||||
const e = document.createElement(tag);
|
||||
if (cls) e.className = cls;
|
||||
if (text !== undefined) e.textContent = text; // untrusted-safe
|
||||
return e;
|
||||
};
|
||||
|
||||
// ── data ────────────────────────────────────────────────────
|
||||
async function getJson(url) {
|
||||
const r = await fetch(url, { cache: "no-store" });
|
||||
if (!r.ok) throw new Error(url + " → " + r.status);
|
||||
return r.json();
|
||||
}
|
||||
async function refresh() {
|
||||
// Monotonic sequence guard: a bucket-toggle fetch racing a slow
|
||||
// interval fetch must not let the stale response win.
|
||||
const seq = ++state.seq;
|
||||
// allSettled, not all: the three endpoints fail independently.
|
||||
// /stats/events is loopback-only, so viewing this page from
|
||||
// another host 404s it by design — that must not blank the spend
|
||||
// panels that answered perfectly well.
|
||||
const [stats, series, events] = await Promise.allSettled([
|
||||
getJson("/stats"),
|
||||
getJson("/stats/timeseries?bucket=" + state.bucket),
|
||||
getJson("/stats/events?limit=30"),
|
||||
]);
|
||||
if (seq !== state.seq) return; // superseded by a newer refresh
|
||||
if (stats.status === "fulfilled") state.stats = stats.value;
|
||||
if (series.status === "fulfilled") state.series = series.value;
|
||||
state.events = events.status === "fulfilled" ? events.value : null;
|
||||
// The feed is the only endpoint whose 404 is a policy, not a
|
||||
// fault — tell the reader which it was.
|
||||
state.feedLocalOnly = events.status === "rejected" && /→ 404$/.test(events.reason.message);
|
||||
|
||||
if (stats.status === "fulfilled") {
|
||||
render();
|
||||
$("meta-updated").textContent = "updated " + new Date().toLocaleTimeString();
|
||||
return;
|
||||
}
|
||||
$("meta-updated").textContent = "unreachable — retrying";
|
||||
if (!state.stats) {
|
||||
// Cold load with the proxy down: say so in the page body,
|
||||
// not just the header corner.
|
||||
const kpis = $("kpis");
|
||||
kpis.textContent = "";
|
||||
kpis.appendChild(el("div", "empty", "Waiting for the proxy at " + location.origin + " …"));
|
||||
}
|
||||
}
|
||||
|
||||
// ── filters ─────────────────────────────────────────────────
|
||||
function wireSeg(segId, attr, onPick) {
|
||||
$(segId).addEventListener("click", (ev) => {
|
||||
const b = ev.target.closest("button"); if (!b) return;
|
||||
for (const btn of $(segId).querySelectorAll("button"))
|
||||
btn.setAttribute("aria-pressed", String(btn === b));
|
||||
onPick(b.dataset[attr]);
|
||||
});
|
||||
}
|
||||
wireSeg("scope-seg", "scope", (v) => { state.scope = v; render(); });
|
||||
wireSeg("bucket-seg", "bucket", (v) => { state.bucket = v; refresh(); });
|
||||
|
||||
// ── header + KPIs ───────────────────────────────────────────
|
||||
function scoped() {
|
||||
const s = state.stats;
|
||||
return state.scope === "lifetime" ? s.lifetime : s.session;
|
||||
}
|
||||
function renderHeader() {
|
||||
const s = state.stats;
|
||||
$("meta-version").textContent = "v" + (s.version || "?") + " · rust-native";
|
||||
$("meta-uptime").textContent = "up " + fmtUptime(s.uptime_seconds);
|
||||
const p = s.persistence || {};
|
||||
const persist = $("meta-persist");
|
||||
persist.textContent = "";
|
||||
const dot = el("span", "dot");
|
||||
dot.style.background = p.enabled && p.healthy ? "var(--good)" : (p.enabled ? "var(--critical)" : "var(--muted)");
|
||||
persist.appendChild(dot);
|
||||
persist.appendChild(document.createTextNode(
|
||||
p.enabled ? (p.healthy ? "persisted" : "persistence error") : "memory only"
|
||||
));
|
||||
persist.title = p.path || "";
|
||||
}
|
||||
// Shared SVG element factory (sparklines + the main chart).
|
||||
const SVG_NS = "http://www.w3.org/2000/svg";
|
||||
function mk(tag, attrs) {
|
||||
const e = document.createElementNS(SVG_NS, tag);
|
||||
for (const [k, v] of Object.entries(attrs)) e.setAttribute(k, v);
|
||||
return e;
|
||||
}
|
||||
function sparkline(points) {
|
||||
const vals = points.slice(-12).map(
|
||||
(p) => (Number(p.input_cost_usd) || 0) + (Number(p.output_cost_usd) || 0));
|
||||
if (vals.length < 2) return null;
|
||||
const w = 120, h = 28, max = Math.max(...vals, 1);
|
||||
const step = w / (vals.length - 1);
|
||||
const y = (v) => h - 3 - (v / max) * (h - 6);
|
||||
const d = vals.map((v, i) => (i ? "L" : "M") + (i * step).toFixed(1) + " " + y(v).toFixed(1)).join(" ");
|
||||
const svg = mk("svg", { class: "spark", width: w, height: h, "aria-hidden": "true" });
|
||||
svg.appendChild(mk("path", {
|
||||
d, fill: "none", stroke: "var(--muted)", "stroke-width": 2,
|
||||
"stroke-linecap": "round", "stroke-linejoin": "round",
|
||||
}));
|
||||
svg.appendChild(mk("circle", {
|
||||
cx: (vals.length - 1) * step, cy: y(vals[vals.length - 1]), r: 4,
|
||||
fill: "var(--series-1)", stroke: "var(--surface)", "stroke-width": 2,
|
||||
}));
|
||||
return svg;
|
||||
}
|
||||
function tile({ label, value, hero, subParts, spark }) {
|
||||
const card = el("div", "card");
|
||||
card.appendChild(el("div", "label", label));
|
||||
card.appendChild(el("div", "value" + (hero ? " hero" : ""), value));
|
||||
if (subParts && subParts.length) {
|
||||
const sub = el("div", "sub");
|
||||
subParts.forEach((p, i) => {
|
||||
if (i) sub.appendChild(document.createTextNode(" · "));
|
||||
sub.appendChild(typeof p === "string" ? document.createTextNode(p) : p);
|
||||
});
|
||||
card.appendChild(sub);
|
||||
}
|
||||
if (spark) card.appendChild(spark);
|
||||
return card;
|
||||
}
|
||||
function renderKpis() {
|
||||
const t = scoped();
|
||||
const hours = (state.seriesHourly && state.seriesHourly.points) || [];
|
||||
const kpis = $("kpis");
|
||||
kpis.textContent = "";
|
||||
kpis.appendChild(tile({
|
||||
label: "Saved (" + state.scope + ")",
|
||||
value: fmtUsd(t.savings_usd),
|
||||
hero: true,
|
||||
subParts: [
|
||||
fmtInt(t.tokens_saved) + " tokens removed",
|
||||
el("span", "up", fmtPct(t.savings_percent) + " of would-be input"),
|
||||
],
|
||||
}));
|
||||
kpis.appendChild(tile({
|
||||
label: "Spend",
|
||||
value: fmtUsd(t.total_cost_usd),
|
||||
subParts: [
|
||||
fmtUsd(t.input_cost_usd) + " in",
|
||||
fmtUsd(t.output_cost_usd) + " out",
|
||||
],
|
||||
spark: hours.length ? sparkline(hours) : null,
|
||||
}));
|
||||
const failed = Number(t.failed) || 0;
|
||||
kpis.appendChild(tile({
|
||||
label: "Requests",
|
||||
value: fmtInt(t.requests),
|
||||
subParts: [
|
||||
fmtInt(t.compressed) + " compressed",
|
||||
failed ? el("span", "bad", "✕ " + fmtInt(failed) + " failed") : "0 failed",
|
||||
],
|
||||
}));
|
||||
kpis.appendChild(tile({
|
||||
label: "Cache reads",
|
||||
value: fmtInt(t.cache_read_tokens),
|
||||
subParts: [
|
||||
fmtPct(t.cache_hit_percent) + " of input from cache",
|
||||
fmtUsd(t.cache_savings_usd) + " net saved",
|
||||
],
|
||||
}));
|
||||
$("scope-label-1").textContent = state.scope;
|
||||
}
|
||||
|
||||
// ── the chart ───────────────────────────────────────────────
|
||||
//
|
||||
// Stacked part-to-whole: what you paid + what Headroom saved =
|
||||
// what the traffic would have cost without it. Charting savings
|
||||
// alone would leave the panel empty on the default config
|
||||
// (compression is off by default), which is exactly the
|
||||
// deployment that most wants to see its spend.
|
||||
const SERIES = [
|
||||
{
|
||||
name: "Spent", color: "var(--series-1)",
|
||||
usd: (p) => (Number(p.input_cost_usd) || 0) + (Number(p.output_cost_usd) || 0),
|
||||
tok: (p) => (Number(p.input_tokens) || 0) + (Number(p.cache_read_tokens) || 0) +
|
||||
(Number(p.cache_write_tokens) || 0),
|
||||
},
|
||||
{
|
||||
name: "Saved", color: "var(--series-2)",
|
||||
usd: (p) => (Number(p.compression_savings_usd) || 0) + (Number(p.cache_savings_usd) || 0),
|
||||
tok: (p) => Number(p.tokens_saved) || 0,
|
||||
},
|
||||
];
|
||||
function niceMax(v) {
|
||||
if (v <= 0) return 1;
|
||||
const exp = Math.pow(10, Math.floor(Math.log10(v)));
|
||||
const m = v / exp;
|
||||
return (m <= 1 ? 1 : m <= 2 ? 2 : m <= 5 ? 5 : 10) * exp;
|
||||
}
|
||||
function renderChart() {
|
||||
const points = (state.series && state.series.points) || [];
|
||||
const svg = $("chart");
|
||||
svg.textContent = "";
|
||||
const legend = $("chart-legend");
|
||||
legend.textContent = "";
|
||||
const hasTraffic = points.some((p) => (Number(p.requests) || 0) > 0);
|
||||
$("chart-empty").style.display = hasTraffic ? "none" : "";
|
||||
svg.style.display = hasTraffic ? "" : "none";
|
||||
legend.style.display = hasTraffic ? "" : "none";
|
||||
if (!hasTraffic) return;
|
||||
|
||||
// Nothing priced yet (e.g. every model missing from the price
|
||||
// table)? Chart tokens rather than a row of flat $0 bars.
|
||||
const usdTotal = points.reduce((a, p) => a + SERIES.reduce((b, s) => b + s.usd(p), 0), 0);
|
||||
const mode = usdTotal > 0 ? "usd" : "tokens";
|
||||
$("chart-desc").textContent = mode === "usd"
|
||||
? "Bar height is what this traffic would have cost without Headroom."
|
||||
: "Input tokens vs tokens saved (nothing priced yet — see the note below).";
|
||||
const valOf = (p, s) => (mode === "usd" ? s.usd(p) : s.tok(p));
|
||||
const fmtVal = mode === "usd" ? fmtUsd : fmtInt;
|
||||
|
||||
for (const s of SERIES) {
|
||||
const k = el("span", "key");
|
||||
const sw = el("span", "swatch"); sw.style.background = s.color;
|
||||
k.appendChild(sw); k.appendChild(el("span", null, s.name));
|
||||
legend.appendChild(k);
|
||||
}
|
||||
|
||||
const W = Math.max(svg.clientWidth || 900, 320), H = 230;
|
||||
// Left margin fits the widest tick: sub-cent USD renders with
|
||||
// significant digits (e.g. "$0.000025"), which overflows a
|
||||
// narrow gutter and clips the top label.
|
||||
const m = { top: 10, right: 8, bottom: 24, left: 72 };
|
||||
svg.setAttribute("viewBox", "0 0 " + W + " " + H);
|
||||
const iw = W - m.left - m.right, ih = H - m.top - m.bottom;
|
||||
const max = niceMax(Math.max(...points.map((p) => SERIES.reduce((a, s) => a + valOf(p, s), 0))));
|
||||
// Gridlines + ticks (hairline, recessive; clean numbers).
|
||||
// Token counts are integers, so quartering a niceMax of 10 would
|
||||
// print "2.5 tokens". Drop to whole steps when the axis is small
|
||||
// enough that quarters aren't whole.
|
||||
const ticks = mode === "usd" || max % 4 === 0 ? 4 : Math.min(4, max);
|
||||
for (let i = 0; i <= ticks; i++) {
|
||||
const v = (max / ticks) * i, y = m.top + ih - (v / max) * ih;
|
||||
if (i > 0) svg.appendChild(mk("line", { x1: m.left, x2: W - m.right, y1: y, y2: y, stroke: "var(--grid)", "stroke-width": 1 }));
|
||||
const t = mk("text", { x: m.left - 8, y: y + 4, "text-anchor": "end", fill: "var(--muted)", "font-size": 11 });
|
||||
t.textContent = mode === "usd" ? fmtUsd(v) : fmtInt(Math.round(v));
|
||||
svg.appendChild(t);
|
||||
}
|
||||
svg.appendChild(mk("line", { x1: m.left, x2: W - m.right, y1: m.top + ih, y2: m.top + ih, stroke: "var(--baseline)", "stroke-width": 1 }));
|
||||
|
||||
const n = points.length;
|
||||
const band = iw / n;
|
||||
const barW = Math.min(24, Math.max(3, band * 0.6));
|
||||
const gap = 2; // surface gap between stack segments
|
||||
const tipEl = $("tooltip");
|
||||
|
||||
// Per-column keyboard stops are useful at hour/week/month density
|
||||
// but hostile at daily density (up to 400 Tab presses to cross
|
||||
// the chart). Past this cap, columns stay hoverable and the data
|
||||
// remains reachable via /stats/timeseries; the svg's own
|
||||
// aria-label summarizes the series.
|
||||
const focusableColumns = n <= 90;
|
||||
svg.setAttribute(
|
||||
"aria-label",
|
||||
"Spend vs savings, " + n + " " + state.bucket + " buckets" +
|
||||
(focusableColumns ? "" : " — full data at /stats/timeseries?bucket=" + state.bucket)
|
||||
);
|
||||
points.forEach((p, i) => {
|
||||
const x = m.left + i * band + (band - barW) / 2;
|
||||
const colLabel = p.timestamp + ": " + fmtInt(p.requests) + " requests, " +
|
||||
SERIES.map((s) => s.name + " " + fmtVal(valOf(p, s))).join(", ");
|
||||
const colAttrs = { class: "col", role: "img", "aria-label": colLabel };
|
||||
if (focusableColumns) colAttrs.tabindex = "0";
|
||||
const col = mk("g", colAttrs);
|
||||
let yCursor = m.top + ih;
|
||||
const segs = [];
|
||||
SERIES.forEach((s, si) => {
|
||||
const v = valOf(p, s);
|
||||
const hpx = (v / max) * ih;
|
||||
if (hpx <= 0) return;
|
||||
const isTop = si === SERIES.length - 1 ||
|
||||
SERIES.slice(si + 1).every((s2) => valOf(p, s2) <= 0);
|
||||
// Segment occupies [stackTop, bottom]; the 2px surface gap is
|
||||
// shaved off THIS segment's top when another segment sits
|
||||
// above it, so the sliver of background appears BETWEEN
|
||||
// segments — never between the stack and the axis. The next
|
||||
// segment still starts from the un-shaved stackTop.
|
||||
const bottom = yCursor;
|
||||
const stackTop = yCursor - hpx;
|
||||
const y = stackTop + (isTop ? 0 : Math.min(gap, hpx - 1));
|
||||
const drawH = Math.max(1, bottom - y);
|
||||
const r = isTop ? Math.min(4, barW / 2, drawH) : 0;
|
||||
// Rounded data-end (top) only; square at the baseline.
|
||||
const path = "M" + x + " " + (y + drawH) +
|
||||
" L" + x + " " + (y + r) +
|
||||
" Q" + x + " " + y + " " + (x + r) + " " + y +
|
||||
" L" + (x + barW - r) + " " + y +
|
||||
" Q" + (x + barW) + " " + y + " " + (x + barW) + " " + (y + r) +
|
||||
" L" + (x + barW) + " " + (y + drawH) + " Z";
|
||||
segs.push(mk("path", { d: path, fill: s.color, class: "bar" }));
|
||||
yCursor = stackTop;
|
||||
});
|
||||
segs.reverse().forEach((sEl) => col.appendChild(sEl));
|
||||
// Hit target wider than the mark (full band, full height).
|
||||
col.appendChild(mk("rect", { x: m.left + i * band, y: m.top, width: band, height: ih, fill: "transparent" }));
|
||||
|
||||
const show = () => {
|
||||
tipEl.textContent = "";
|
||||
tipEl.appendChild(el("div", "t-when", p.timestamp));
|
||||
SERIES.forEach((s) => {
|
||||
const row = el("div", "t-row");
|
||||
const key = el("span", "t-key"); key.style.borderTopColor = s.color;
|
||||
row.appendChild(key);
|
||||
row.appendChild(el("span", "t-name", s.name));
|
||||
row.appendChild(el("span", "t-val", fmtVal(valOf(p, s))));
|
||||
tipEl.appendChild(row);
|
||||
});
|
||||
const meta = el("div", "t-when");
|
||||
meta.style.marginTop = "5px";
|
||||
meta.textContent = fmtInt(p.requests) + " req · " + fmtInt(p.tokens_saved) +
|
||||
" tok saved" + (Number(p.failed) ? " · " + fmtInt(p.failed) + " failed" : "");
|
||||
tipEl.appendChild(meta);
|
||||
tipEl.style.display = "block";
|
||||
const holder = svg.parentElement.getBoundingClientRect();
|
||||
const frac = (m.left + i * band + band / 2) / W;
|
||||
let left = frac * holder.width - tipEl.offsetWidth / 2;
|
||||
left = Math.max(0, Math.min(left, holder.width - tipEl.offsetWidth));
|
||||
tipEl.style.left = left + "px";
|
||||
tipEl.style.top = "6px";
|
||||
};
|
||||
const hide = () => { tipEl.style.display = "none"; };
|
||||
col.addEventListener("pointerenter", show);
|
||||
col.addEventListener("pointerleave", hide);
|
||||
col.addEventListener("focus", show);
|
||||
col.addEventListener("blur", hide);
|
||||
svg.appendChild(col);
|
||||
|
||||
// Sparse x labels: first, last, and every ~quarter.
|
||||
if (i === 0 || i === n - 1 || (n > 6 && i % Math.ceil(n / 4) === 0)) {
|
||||
const t = mk("text", {
|
||||
x: m.left + i * band + band / 2, y: H - 7,
|
||||
"text-anchor": "middle", fill: "var(--muted)", "font-size": 11,
|
||||
});
|
||||
t.textContent = shortLabel(p.timestamp);
|
||||
svg.appendChild(t);
|
||||
}
|
||||
});
|
||||
}
|
||||
function shortLabel(ts) {
|
||||
if (!ts) return "";
|
||||
if (state.bucket === "hour") return ts.slice(11, 16) || ts;
|
||||
if (state.bucket === "month") return ts;
|
||||
return ts.slice(5); // MM-DD
|
||||
}
|
||||
|
||||
// ── tables ──────────────────────────────────────────────────
|
||||
function table(headers, rows, emptyMsg) {
|
||||
if (!rows.length) return el("div", "empty", emptyMsg);
|
||||
const t = el("table");
|
||||
const tr = el("tr");
|
||||
headers.forEach((h) => {
|
||||
const th = el("th", null, h);
|
||||
th.setAttribute("scope", "col");
|
||||
tr.appendChild(th);
|
||||
});
|
||||
t.appendChild(tr);
|
||||
rows.forEach((cells) => {
|
||||
const r = el("tr");
|
||||
cells.forEach((c) => r.appendChild(c));
|
||||
t.appendChild(r);
|
||||
});
|
||||
return t;
|
||||
}
|
||||
const PROVIDER_COLOR = {
|
||||
anthropic: "var(--series-1)", openai_chat: "var(--series-2)",
|
||||
openai_responses: "var(--series-2)", bedrock: "var(--series-1)", vertex: "var(--series-2)",
|
||||
};
|
||||
function renderProviders() {
|
||||
const s = state.stats;
|
||||
const map = state.scope === "lifetime" ? s.lifetime_by_provider : s.session_by_provider;
|
||||
const rows = Object.entries(map || {}).map(([name, t]) => {
|
||||
const nameCell = el("td", "name");
|
||||
nameCell.appendChild(el("span", null, name));
|
||||
return [
|
||||
nameCell,
|
||||
el("td", null, fmtInt(t.requests)),
|
||||
el("td", null, fmtInt(t.tokens_saved)),
|
||||
el("td", null, fmtUsd(t.savings_usd)),
|
||||
el("td", null, fmtInt(t.failed)),
|
||||
];
|
||||
});
|
||||
const holder = $("providers");
|
||||
holder.textContent = "";
|
||||
holder.appendChild(table(
|
||||
["Provider", "Requests", "Tok saved", "Saved", "Failed"],
|
||||
rows, "No traffic in this scope yet."));
|
||||
}
|
||||
function renderModels() {
|
||||
const s = state.stats;
|
||||
const entries = Object.entries(s.lifetime_by_model || {})
|
||||
.sort((a, b) => (b[1].savings_usd + b[1].total_cost_usd) - (a[1].savings_usd + a[1].total_cost_usd))
|
||||
.slice(0, 10);
|
||||
const rows = entries.map(([name, t]) => [
|
||||
el("td", "name", name),
|
||||
el("td", null, fmtInt(t.requests)),
|
||||
el("td", null, fmtInt(t.tokens_saved)),
|
||||
el("td", null, fmtInt(t.cache_read_tokens)),
|
||||
el("td", null, fmtUsd(t.savings_usd)),
|
||||
el("td", null, fmtUsd(t.total_cost_usd)),
|
||||
]);
|
||||
const holder = $("models");
|
||||
holder.textContent = "";
|
||||
holder.appendChild(table(
|
||||
["Model", "Requests", "Tok saved", "Cache reads", "Saved", "Spend"],
|
||||
rows, "No models recorded yet."));
|
||||
}
|
||||
function renderFeed() {
|
||||
const events = (state.events && state.events.events) || [];
|
||||
const rows = events.map((e) => {
|
||||
const when = el("td", null, (e.timestamp || "").slice(11, 19));
|
||||
const model = el("td", "name");
|
||||
const pk = el("span", "provider-key");
|
||||
pk.style.background = PROVIDER_COLOR[e.provider] || "var(--muted)";
|
||||
model.appendChild(pk);
|
||||
model.appendChild(el("span", null, (e.provider || "?") + " · " + (e.model || "?")));
|
||||
model.title = e.request_id || "";
|
||||
const status = el("td");
|
||||
if (e.failed) status.appendChild(el("span", "badge-fail", "✕ failed"));
|
||||
else status.appendChild(el("span", null, fmtPct(e.compression_percent)));
|
||||
// Input column: the provider-billed (uncached) input is the
|
||||
// number that always exists. Only show the compression
|
||||
// before→after arrow when compression actually ran — otherwise
|
||||
// it reads a meaningless "0 → 0".
|
||||
const inputCell = Number(e.tokens_before) > 0
|
||||
? el("td", null, fmtInt(e.tokens_before) + " → " + fmtInt(e.tokens_after))
|
||||
: el("td", null, fmtInt(e.input_tokens));
|
||||
return [
|
||||
when, model,
|
||||
inputCell,
|
||||
el("td", null, fmtInt(e.tokens_saved)),
|
||||
el("td", null, fmtInt(e.cache_read_tokens) + " / " + fmtInt(e.cache_write_tokens)),
|
||||
el("td", null, fmtInt(e.output_tokens)),
|
||||
el("td", null, fmtUsd((Number(e.compression_savings_usd) || 0) + (Number(e.cache_savings_usd) || 0))),
|
||||
el("td", null, (Number(e.latency_ms) || 0) + "ms"),
|
||||
status,
|
||||
];
|
||||
});
|
||||
const holder = $("feed");
|
||||
holder.textContent = "";
|
||||
if (state.feedLocalOnly) {
|
||||
// Not an error: the per-request feed is served to local callers
|
||||
// only. Say which, so nobody debugs a working proxy.
|
||||
holder.appendChild(el("div", "empty",
|
||||
"The per-request feed is available to local callers only. " +
|
||||
"Totals above cover all traffic; open this dashboard on the proxy host " +
|
||||
"(or over a tunnel to it) to see individual requests."));
|
||||
return;
|
||||
}
|
||||
holder.appendChild(table(
|
||||
["Time", "Lane · model", "Input tok", "Saved", "Cache r/w", "Out tok", "Saved $", "Latency", "Compr %"],
|
||||
rows, "No requests yet — point a client at this proxy and refresh."));
|
||||
}
|
||||
|
||||
function render() {
|
||||
if (!state.stats) return;
|
||||
renderHeader();
|
||||
renderKpis();
|
||||
renderChart();
|
||||
renderProviders();
|
||||
renderModels();
|
||||
renderFeed();
|
||||
}
|
||||
|
||||
// Keep an hourly series around for the KPI sparkline regardless of
|
||||
// the selected history bucket.
|
||||
async function refreshHourly() {
|
||||
try { state.seriesHourly = await getJson("/stats/timeseries?bucket=hour"); } catch (e) { /* keep last */ }
|
||||
}
|
||||
|
||||
refreshHourly().then(refresh);
|
||||
setInterval(refresh, 3000);
|
||||
setInterval(refreshHourly, 15000);
|
||||
window.addEventListener("resize", () => state.stats && renderChart());
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because it is too large
Load Diff
@@ -39,10 +39,14 @@
|
||||
//! revisit if multi-process aggregation lands.
|
||||
|
||||
pub mod cache_hit_rate;
|
||||
pub mod capture;
|
||||
pub mod compression_ratio;
|
||||
pub mod ledger;
|
||||
pub mod metric_names;
|
||||
pub mod pricing;
|
||||
pub mod prometheus;
|
||||
pub mod proxy_metrics;
|
||||
pub mod stats_http;
|
||||
|
||||
pub use prometheus::{
|
||||
handle_metrics, observe_bedrock_invoke_latency, record_bedrock_eventstream_message,
|
||||
|
||||
@@ -0,0 +1,414 @@
|
||||
//! Per-token USD pricing — Phase H. Used to value spend and savings.
|
||||
//!
|
||||
//! Reads the LiteLLM price table already vendored for
|
||||
//! [`crate::compression::model_limits`] — one embedded copy, two
|
||||
//! consumers. No new data file, and no startup network dependency:
|
||||
//! the table ships inside the binary and parses lazily on first
|
||||
//! lookup.
|
||||
//!
|
||||
//! # Lookup discipline
|
||||
//!
|
||||
//! Lookup is exact-match over a small, deterministic candidate list
|
||||
//! derived from the request's model id (lowercased, then: geo prefix
|
||||
//! stripped, `:rev` suffix trimmed, provider path segment dropped).
|
||||
//! We deliberately do NOT substring-scan the table: a short or
|
||||
//! generic id must miss (and be logged) rather than misprice against
|
||||
//! whichever stored id happens to contain it.
|
||||
//!
|
||||
//! **The unmodified id is always tried first**, because Bedrock
|
||||
//! cross-region pricing is genuinely region-specific — e.g.
|
||||
//! `eu.anthropic.claude-3-5-haiku-20241022-v1:0` bills at $0.25/M
|
||||
//! against `anthropic.claude-3-5-haiku-20241022-v1:0`'s $0.80/M. Any
|
||||
//! region the table tracks therefore gets its true price. The
|
||||
//! geo-stripped candidate is only a *fallback* for a (region, model)
|
||||
//! pair the table does not list, where the base price is a far
|
||||
//! better estimate than $0 — approximate, but never silently
|
||||
//! preferred over an exact regional entry.
|
||||
//!
|
||||
//! # Coverage
|
||||
//!
|
||||
//! The vendored snapshot tracks current model generations plus every
|
||||
//! Bedrock/Vertex-prefixed id, but it does not carry every legacy
|
||||
//! direct-API id (`claude-3-5-sonnet-20241022`, for one, is absent
|
||||
//! while its Bedrock form is present). Those price at $0 with a WARN
|
||||
//! until the table is refreshed — see the module note below.
|
||||
//!
|
||||
//! # Unknown models price at $0
|
||||
//!
|
||||
//! A miss returns `None`; the ledger then records $0 for every USD
|
||||
//! field and this module logs one WARN naming the model. We
|
||||
//! deliberately do NOT substitute a blended fallback rate: phantom
|
||||
//! dollars on a cost dashboard are worse than an obvious zero, and
|
||||
//! the WARN tells an operator exactly which id to add by refreshing
|
||||
//! the vendored table (`scripts/refresh_model_limits.sh`).
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{OnceLock, RwLock};
|
||||
|
||||
use crate::bedrock::vendor::strip_geo_prefix;
|
||||
use crate::compression::model_limits::VENDORED_JSON;
|
||||
|
||||
/// Per-token USD prices for one model. All fields are $/token
|
||||
/// (LiteLLM stores them that way natively — no per-million
|
||||
/// conversion happens here).
|
||||
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||
pub struct ModelPrice {
|
||||
pub input: f64,
|
||||
pub output: f64,
|
||||
pub cache_read: f64,
|
||||
pub cache_write: f64,
|
||||
}
|
||||
|
||||
impl ModelPrice {
|
||||
/// USD saved by removing `tokens_saved` input tokens before they
|
||||
/// were sent (compression savings).
|
||||
pub fn compression_savings_usd(&self, tokens_saved: u64) -> f64 {
|
||||
tokens_saved as f64 * self.input
|
||||
}
|
||||
|
||||
/// NET USD saved by prompt caching for one request: the read
|
||||
/// discount (`reads × (input − cache_read)`) minus the write
|
||||
/// premium (`writes × (cache_write − input)` — cache writes bill
|
||||
/// ABOVE list price, typically 1.25×). Reporting the gross read
|
||||
/// discount alone overstates the benefit on write-heavy traffic,
|
||||
/// which is exactly the phantom-dollars failure mode this module
|
||||
/// exists to avoid. Floored at $0 per request: a warm-up turn
|
||||
/// that only writes is an investment, not negative savings —
|
||||
/// its real cost is already carried by [`Self::input_cost_usd`].
|
||||
///
|
||||
/// Known bias, stated because it's the honest thing to state:
|
||||
/// the ledger sums these per-request values, and a floored
|
||||
/// request contributes 0 rather than its true negative, so
|
||||
/// `sum(max(0, net))` ≥ `max(0, sum(net))`. Aggregate cache
|
||||
/// savings therefore skew slightly high. The gap is one write
|
||||
/// premium per floored request, which in practice means the
|
||||
/// first turn of a session: once a prefix is cached, later turns
|
||||
/// read far more than they write and net positive well clear of
|
||||
/// the floor (100k reads + 5k writes on Sonnet nets +$0.266, vs
|
||||
/// a $0.004 premium). Fixing it properly means accumulating a
|
||||
/// signed net and flooring once at the aggregate — worth doing
|
||||
/// if cache-heavy traffic ever makes the skew material, but not
|
||||
/// worth a signed accumulator (and a persisted-negative
|
||||
/// migration) for single-digit thousandths of a cent today.
|
||||
pub fn cache_savings_usd(&self, cache_read_tokens: u64, cache_write_tokens: u64) -> f64 {
|
||||
let read_discount = (self.input - self.cache_read).max(0.0);
|
||||
let write_premium = (self.cache_write - self.input).max(0.0);
|
||||
let net =
|
||||
cache_read_tokens as f64 * read_discount - cache_write_tokens as f64 * write_premium;
|
||||
net.max(0.0)
|
||||
}
|
||||
|
||||
/// USD spent on generated output for a request. Output bills at
|
||||
/// its own rate — typically 4-5x input — so a "spend" figure that
|
||||
/// omits it understates the bill by more than it reports on a
|
||||
/// short-prompt/long-answer turn.
|
||||
pub fn output_cost_usd(&self, output_tokens: u64) -> f64 {
|
||||
output_tokens as f64 * self.output
|
||||
}
|
||||
|
||||
/// USD actually spent on input for a request, using the cache
|
||||
/// breakdown when any segment is non-zero (never adding the
|
||||
/// total on top of the breakdown — that would double-count).
|
||||
pub fn input_cost_usd(
|
||||
&self,
|
||||
uncached_input_tokens: u64,
|
||||
cache_read_tokens: u64,
|
||||
cache_write_tokens: u64,
|
||||
) -> f64 {
|
||||
uncached_input_tokens as f64 * self.input
|
||||
+ cache_read_tokens as f64 * self.cache_read
|
||||
+ cache_write_tokens as f64 * self.cache_write
|
||||
}
|
||||
}
|
||||
|
||||
static BOOK: OnceLock<HashMap<String, ModelPrice>> = OnceLock::new();
|
||||
|
||||
/// Bounded once-per-model warn set so an unknown model logs one WARN,
|
||||
/// not one per request. Capped so attacker-controlled model ids can't
|
||||
/// grow it without bound; once full, further unknown models simply
|
||||
/// stop logging (they still price at $0). RwLock so the steady-state
|
||||
/// path for an already-known-unknown model (every miss after the
|
||||
/// first) is a shared read, not serialized writes on the record path.
|
||||
static WARNED: OnceLock<RwLock<std::collections::HashSet<String>>> = OnceLock::new();
|
||||
const WARNED_CAP: usize = 256;
|
||||
|
||||
/// Parse the vendored table now instead of on the first recorded
|
||||
/// request. Called once at startup (main.rs) so the ~1.4 MB JSON
|
||||
/// parse never stalls tokio workers when a burst of first requests
|
||||
/// lands together.
|
||||
pub fn warm() {
|
||||
let _ = book();
|
||||
}
|
||||
|
||||
fn book() -> &'static HashMap<String, ModelPrice> {
|
||||
BOOK.get_or_init(parse_vendored)
|
||||
}
|
||||
|
||||
fn parse_vendored() -> HashMap<String, ModelPrice> {
|
||||
let raw: serde_json::Value = serde_json::from_str(VENDORED_JSON)
|
||||
.expect("vendored LiteLLM JSON must parse — same invariant as model_limits");
|
||||
let mut out = HashMap::new();
|
||||
let Some(obj) = raw.as_object() else {
|
||||
return out;
|
||||
};
|
||||
for (model_id, spec) in obj {
|
||||
if model_id == "sample_spec" {
|
||||
continue;
|
||||
}
|
||||
let Some(spec) = spec.as_object() else {
|
||||
continue;
|
||||
};
|
||||
let input = spec.get("input_cost_per_token").and_then(|v| v.as_f64());
|
||||
let output = spec.get("output_cost_per_token").and_then(|v| v.as_f64());
|
||||
let (Some(input), Some(output)) = (input, output) else {
|
||||
continue;
|
||||
};
|
||||
if !input.is_finite() || !output.is_finite() || input < 0.0 || output < 0.0 {
|
||||
continue;
|
||||
}
|
||||
let cache_read = spec
|
||||
.get("cache_read_input_token_cost")
|
||||
.and_then(|v| v.as_f64())
|
||||
.filter(|c| c.is_finite() && *c >= 0.0)
|
||||
.unwrap_or(input);
|
||||
let cache_write = spec
|
||||
.get("cache_creation_input_token_cost")
|
||||
.and_then(|v| v.as_f64())
|
||||
.filter(|c| c.is_finite() && *c >= 0.0)
|
||||
.unwrap_or(input);
|
||||
out.insert(
|
||||
model_id.to_ascii_lowercase(),
|
||||
ModelPrice {
|
||||
input,
|
||||
output,
|
||||
cache_read,
|
||||
cache_write,
|
||||
},
|
||||
);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Candidate keys tried, in order, for a request model id. Exact
|
||||
/// matches only — see module doc for why there is no substring scan.
|
||||
fn candidates(model: &str) -> Vec<String> {
|
||||
let lower = model.trim().to_ascii_lowercase();
|
||||
let mut out = Vec::with_capacity(6);
|
||||
let mut push = |s: String| {
|
||||
if !s.is_empty() && !out.contains(&s) {
|
||||
out.push(s);
|
||||
}
|
||||
};
|
||||
push(lower.clone());
|
||||
// Bedrock cross-region profile: `eu.anthropic.claude…` falls back
|
||||
// to `anthropic.claude…` when the region itself isn't in the
|
||||
// table. GEO_PREFIXES (shared with the compression vendor gate)
|
||||
// covers every geo the vendored table fronts, `us-gov.` included.
|
||||
let geo = strip_geo_prefix(&lower).to_string();
|
||||
push(geo.clone());
|
||||
// Bedrock revision suffix: `…-v1:0`. ONLY a numeric suffix is
|
||||
// trimmed. A non-numeric colon suffix is a routing/tier variant
|
||||
// (`:exacto`, `:free`, `:nitro`) that can genuinely bill at its
|
||||
// own rate — the vendored table lists
|
||||
// `openrouter/z-ai/glm-4.6` at $0.40/M and its `:exacto` variant
|
||||
// at $0.45/M. Trimming those would hand an uncatalogued variant
|
||||
// its sibling's price with no warning: a confident wrong number,
|
||||
// which this module exists to avoid. They miss and WARN instead.
|
||||
push_revision_trimmed(&mut push, &geo);
|
||||
push_revision_trimmed(&mut push, &lower);
|
||||
// Provider-routed ids (`openai/gpt-4o`, `github-copilot/claude…`):
|
||||
// the bare tail is how LiteLLM stores first-party models. Try the
|
||||
// tail both verbatim and `:rev`-trimmed, so a combined
|
||||
// `vendor/model:rev` id can still reach a fully-stripped entry.
|
||||
if let Some((_, tail)) = lower.rsplit_once('/') {
|
||||
push(tail.to_string());
|
||||
push_revision_trimmed(&mut push, tail);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Push `id` minus a trailing `:<digits>` revision, if it has one.
|
||||
fn push_revision_trimmed(push: &mut impl FnMut(String), id: &str) {
|
||||
if let Some((head, rev)) = id.rsplit_once(':') {
|
||||
if !rev.is_empty() && rev.bytes().all(|b| b.is_ascii_digit()) {
|
||||
push(head.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Look up per-token pricing for a model id. `None` means "not in
|
||||
/// the vendored table" — the caller records $0 for USD fields and
|
||||
/// this module logs one WARN per distinct model id.
|
||||
pub fn lookup(model: &str) -> Option<ModelPrice> {
|
||||
let table = book();
|
||||
for key in candidates(model) {
|
||||
if let Some(p) = table.get(&key) {
|
||||
return Some(*p);
|
||||
}
|
||||
}
|
||||
warn_once(model);
|
||||
None
|
||||
}
|
||||
|
||||
fn warn_once(model: &str) {
|
||||
let set = WARNED.get_or_init(|| RwLock::new(std::collections::HashSet::new()));
|
||||
{
|
||||
let read = set
|
||||
.read()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if read.len() >= WARNED_CAP || read.contains(model) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
let mut write = set
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if write.len() >= WARNED_CAP || !write.insert(model.to_string()) {
|
||||
return;
|
||||
}
|
||||
drop(write);
|
||||
tracing::warn!(
|
||||
event = "stats_price_unknown_model",
|
||||
model = %model,
|
||||
"model id not in the vendored price table; both spend AND savings \
|
||||
for it record as $0 — refresh data/model_prices_and_context_window.json \
|
||||
via scripts/refresh_model_limits.sh to add it"
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn known_anthropic_direct_id_prices() {
|
||||
let p = lookup("claude-sonnet-4-5-20250929").expect("priced");
|
||||
assert!(p.input > 0.0 && p.output > p.input);
|
||||
assert!(p.cache_read < p.input, "cache reads are discounted");
|
||||
assert!(p.cache_write > p.input, "cache writes carry a premium");
|
||||
}
|
||||
|
||||
/// Bedrock cross-region pricing is region-specific, so an exact
|
||||
/// regional entry must win over the geo-stripped base id.
|
||||
#[test]
|
||||
fn exact_regional_bedrock_entry_wins_over_base() {
|
||||
let base = lookup("anthropic.claude-3-5-haiku-20241022-v1:0").expect("base priced");
|
||||
let eu = lookup("eu.anthropic.claude-3-5-haiku-20241022-v1:0").expect("eu priced");
|
||||
assert!(
|
||||
(base.input - 8e-7).abs() < 1e-12,
|
||||
"base id prices at list rate, got {}",
|
||||
base.input
|
||||
);
|
||||
assert!(
|
||||
(eu.input - 2.5e-7).abs() < 1e-12,
|
||||
"eu id must use its own regional rate, got {}",
|
||||
eu.input
|
||||
);
|
||||
assert_ne!(
|
||||
base, eu,
|
||||
"geo-stripping must not clobber a tracked regional price"
|
||||
);
|
||||
}
|
||||
|
||||
/// A region the table does NOT list falls back to the base id —
|
||||
/// approximate, but far better than $0.
|
||||
/// Every published geo prefix must fall back to the base entry
|
||||
/// when the table doesn't carry that exact (region, model) pair.
|
||||
/// Table-driven over GEO_PREFIXES itself, so adding a prefix
|
||||
/// without its pricing fallback fails here — the `au.`/`jp.`/
|
||||
/// `us-gov.` gap this PR fixes was exactly that.
|
||||
#[test]
|
||||
fn every_geo_prefix_falls_back_to_base_price() {
|
||||
const BASE: &str = "anthropic.claude-3-5-haiku-20241022-v1:0";
|
||||
let base = lookup(BASE).expect("base priced");
|
||||
for geo in crate::bedrock::vendor::GEO_PREFIXES {
|
||||
let id = format!("{geo}{BASE}");
|
||||
if book().get(&id).is_some() {
|
||||
continue; // table lists this region explicitly — priced on its own terms
|
||||
}
|
||||
assert_eq!(
|
||||
lookup(&id).unwrap_or_else(|| panic!("{id} must fall back")),
|
||||
base,
|
||||
"{geo} must strip to the base entry when unlisted"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn provider_path_segment_falls_back_to_bare_tail() {
|
||||
let bare = lookup("gpt-4o").expect("bare id priced");
|
||||
let routed = lookup("openai/gpt-4o").expect("routed id priced");
|
||||
assert_eq!(bare, routed);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_model_misses_instead_of_mispricing() {
|
||||
assert!(lookup("claude").is_none(), "generic id must not match");
|
||||
assert!(lookup("totally-unknown-model-xyz").is_none());
|
||||
assert!(lookup("").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn case_insensitive_lookup() {
|
||||
assert_eq!(
|
||||
lookup("GPT-4o").expect("priced"),
|
||||
lookup("gpt-4o").expect("priced")
|
||||
);
|
||||
}
|
||||
|
||||
/// A combined `vendor/model:rev` id must reach a table entry
|
||||
/// stored fully stripped — the slash-tail is also tried with its
|
||||
/// `:rev` suffix trimmed.
|
||||
#[test]
|
||||
fn provider_path_and_revision_compose() {
|
||||
let bare = lookup("gpt-4o").expect("bare id priced");
|
||||
let combined = lookup("openai/gpt-4o:2").expect("combined form priced");
|
||||
assert_eq!(bare, combined, "provider path + numeric revision compose");
|
||||
}
|
||||
|
||||
/// A non-numeric colon suffix is a pricing-relevant variant, not a
|
||||
/// revision. It must NOT inherit the bare model's rate: the table
|
||||
/// prices `openrouter/z-ai/glm-4.6` at $0.40/M and `:exacto` at
|
||||
/// $0.45/M, so borrowing across that boundary invents dollars.
|
||||
#[test]
|
||||
fn variant_suffix_never_borrows_the_bare_models_price() {
|
||||
let bare = lookup("openrouter/z-ai/glm-4.6").expect("bare variant priced");
|
||||
let exacto = lookup("openrouter/z-ai/glm-4.6:exacto").expect("listed variant priced");
|
||||
assert_ne!(bare.input, exacto.input, "fixture guard: these must differ");
|
||||
// An UNLISTED variant must miss loudly, not silently take the
|
||||
// bare price.
|
||||
assert!(
|
||||
lookup("openrouter/z-ai/glm-4.6:priority").is_none(),
|
||||
"an uncatalogued variant must record $0 + WARN, never a guessed rate"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn usd_helpers_compute_list_price_math() {
|
||||
let p = ModelPrice {
|
||||
input: 3e-6,
|
||||
output: 15e-6,
|
||||
cache_read: 3e-7,
|
||||
cache_write: 3.75e-6,
|
||||
};
|
||||
assert!((p.compression_savings_usd(1_000_000) - 3.0).abs() < 1e-9);
|
||||
// Cache savings = read discount, NET of the write premium.
|
||||
assert!((p.cache_savings_usd(1_000_000, 0) - 2.7).abs() < 1e-9);
|
||||
// 1M reads save $2.70; 1M writes cost a $0.75 premium → net $1.95.
|
||||
assert!((p.cache_savings_usd(1_000_000, 1_000_000) - 1.95).abs() < 1e-9);
|
||||
// Write-only warm-up floors at $0, never negative savings
|
||||
// (the premium itself is billed via input_cost_usd).
|
||||
assert_eq!(p.cache_savings_usd(0, 1_000_000), 0.0);
|
||||
// Breakdown input cost: uncached + cache_read + cache_write.
|
||||
let usd = p.input_cost_usd(1_000_000, 1_000_000, 1_000_000);
|
||||
assert!((usd - (3.0 + 0.3 + 3.75)).abs() < 1e-9);
|
||||
// Inverted cache pricing clamps to $0, never negative savings.
|
||||
let inverted = ModelPrice {
|
||||
input: 1e-6,
|
||||
output: 1e-6,
|
||||
cache_read: 2e-6,
|
||||
cache_write: 1e-6,
|
||||
};
|
||||
assert_eq!(inverted.cache_savings_usd(1000, 0), 0.0);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,216 @@
|
||||
//! HTTP surface for the native savings ledger — Phase H:
|
||||
//! `GET /stats`, `GET /stats/timeseries`, `GET /stats/events`, and
|
||||
//! the embedded `GET /dashboard` page.
|
||||
//!
|
||||
//! All handlers are read-only snapshots over
|
||||
//! [`super::ledger::Ledger`] — no handler can mutate accounting
|
||||
//! state. Mounted by [`crate::proxy::build_app`] when
|
||||
//! `Config::stats` is explicitly enabled; when off (the default), the paths fall through
|
||||
//! to the catch-all forwarder like any other route.
|
||||
//!
|
||||
//! # Exposure: aggregate tier is open, per-request tier is local-only
|
||||
//!
|
||||
//! There is no auth here, and `--listen` defaults to `0.0.0.0:8787`,
|
||||
//! so the payloads are split by sensitivity the same way the Python
|
||||
//! proxy splits its own `/stats`:
|
||||
//!
|
||||
//! - **Aggregates** (spend, tokens, per-model/provider rollups,
|
||||
//! history) are served to anyone, like the Prometheus `/metrics`
|
||||
//! endpoint next door — same data class, same stance.
|
||||
//! - **Per-request rows** (`recent_requests`, all of
|
||||
//! `/stats/events`) and the ledger's filesystem path are served
|
||||
//! only to local callers ([`caller_is_local`]). A request id plus a
|
||||
//! model plus a timestamp is a different tier from a counter, and
|
||||
//! `/metrics` never exposes it — so "same as /metrics" would be the
|
||||
//! wrong argument for handing it to the open internet.
|
||||
//!
|
||||
//! `--stats=false` removes the routes entirely. Making the local-only
|
||||
//! tier reachable from a trusted reverse proxy is tracked upstream in
|
||||
//! headroomlabs-ai/headroom#1959, which is defining that policy for
|
||||
//! the Python dashboard first; this surface should adopt whatever
|
||||
//! lands there rather than invent a second scheme.
|
||||
|
||||
use std::net::{IpAddr, SocketAddr};
|
||||
|
||||
use axum::extract::{ConnectInfo, Query, State};
|
||||
use axum::http::header::HOST;
|
||||
use axum::http::{HeaderMap, StatusCode};
|
||||
use axum::response::{Html, IntoResponse, Response};
|
||||
use axum::Json;
|
||||
use serde::Deserialize;
|
||||
use serde_json::Value;
|
||||
|
||||
use crate::proxy::AppState;
|
||||
|
||||
/// True when the caller is genuinely local — gates the per-request
|
||||
/// tier of `/stats` and the whole of `/stats/events`.
|
||||
///
|
||||
/// BOTH conditions are required, mirroring the Python proxy's
|
||||
/// `loopback_guard`:
|
||||
///
|
||||
/// 1. the TCP peer is a loopback address, and
|
||||
/// 2. the `Host` header names loopback.
|
||||
///
|
||||
/// (2) is not redundant with (1) — it is the DNS-rebinding defence. A
|
||||
/// hostile page can rebind its own domain to `127.0.0.1`, so the
|
||||
/// victim's browser connects to this proxy *from* loopback and (1)
|
||||
/// passes, while `Host` still reads the attacker's name and their JS
|
||||
/// reads the response. A real local tool always sends a loopback
|
||||
/// `Host`; a rebound request does not.
|
||||
pub(crate) fn caller_is_local(peer: SocketAddr, headers: &HeaderMap) -> bool {
|
||||
peer.ip().is_loopback()
|
||||
&& headers
|
||||
.get(HOST)
|
||||
.and_then(|h| h.to_str().ok())
|
||||
.is_some_and(host_is_loopback)
|
||||
}
|
||||
|
||||
/// `127.0.0.1`, `127.0.0.1:8787`, `[::1]:8787`, `localhost:8787`, …
|
||||
fn host_is_loopback(host: &str) -> bool {
|
||||
let host = host.trim();
|
||||
// A bare IP with no port must be recognised before any port
|
||||
// split: `::1` is all colons.
|
||||
if let Ok(ip) = host.parse::<IpAddr>() {
|
||||
return ip.is_loopback();
|
||||
}
|
||||
let name = match host.strip_prefix('[') {
|
||||
Some(rest) => match rest.split_once(']') {
|
||||
Some((inner, _)) => inner,
|
||||
None => return false,
|
||||
},
|
||||
None => host.rsplit_once(':').map_or(host, |(h, _)| h),
|
||||
};
|
||||
name.eq_ignore_ascii_case("localhost")
|
||||
|| name.parse::<IpAddr>().is_ok_and(|ip| ip.is_loopback())
|
||||
}
|
||||
|
||||
/// The dashboard page, embedded at compile time so the binary is
|
||||
/// self-contained (no template dir to deploy, no CDN dependency).
|
||||
const DASHBOARD_HTML: &str = include_str!("dashboard.html");
|
||||
|
||||
pub async fn handle_stats(
|
||||
State(state): State<AppState>,
|
||||
ConnectInfo(peer): ConnectInfo<SocketAddr>,
|
||||
headers: HeaderMap,
|
||||
) -> Json<Value> {
|
||||
Json(state.stats.stats_payload(caller_is_local(peer, &headers)))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct TimeseriesParams {
|
||||
/// hour | day | week | month (default: day)
|
||||
bucket: Option<String>,
|
||||
}
|
||||
|
||||
pub async fn handle_timeseries(
|
||||
State(state): State<AppState>,
|
||||
Query(params): Query<TimeseriesParams>,
|
||||
) -> Response {
|
||||
let bucket = params.bucket.as_deref().unwrap_or("day");
|
||||
match state.stats.timeseries_payload(bucket) {
|
||||
Some(payload) => Json(payload).into_response(),
|
||||
None => (
|
||||
StatusCode::BAD_REQUEST,
|
||||
// The Anthropic-style envelope the Bedrock lanes emit
|
||||
// (`bedrock::invoke::error_response`). Note this is NOT
|
||||
// universal: `ProxyError` renders plain text, and axum's
|
||||
// `Query` extractor rejects a malformed `?bucket=` with
|
||||
// its own plain-text 400 before this handler runs — so a
|
||||
// client can't assume JSON for every 4xx from this path.
|
||||
Json(serde_json::json!({
|
||||
"error": {
|
||||
"type": "unknown_bucket",
|
||||
"message": "bucket must be one of: hour, day, week, month",
|
||||
}
|
||||
})),
|
||||
)
|
||||
.into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct EventsParams {
|
||||
limit: Option<usize>,
|
||||
}
|
||||
|
||||
/// This endpoint is per-request metadata end to end — there is no
|
||||
/// aggregate tier to hand a network caller, so the whole thing is
|
||||
/// gated. 404 rather than 403, matching the Python guard's stance:
|
||||
/// keep the surface invisible to scanners rather than merely refuse
|
||||
/// them.
|
||||
pub async fn handle_events(
|
||||
State(state): State<AppState>,
|
||||
ConnectInfo(peer): ConnectInfo<SocketAddr>,
|
||||
headers: HeaderMap,
|
||||
Query(params): Query<EventsParams>,
|
||||
) -> Response {
|
||||
if !caller_is_local(peer, &headers) {
|
||||
return (StatusCode::NOT_FOUND, "not found").into_response();
|
||||
}
|
||||
Json(state.stats.events_payload(params.limit.unwrap_or(50))).into_response()
|
||||
}
|
||||
|
||||
/// `Html` already sets `text/html; charset=utf-8`.
|
||||
pub async fn handle_dashboard() -> Html<&'static str> {
|
||||
Html(DASHBOARD_HTML)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn headers_with_host(host: &str) -> HeaderMap {
|
||||
let mut h = HeaderMap::new();
|
||||
h.insert(HOST, host.parse().unwrap());
|
||||
h
|
||||
}
|
||||
|
||||
fn peer(addr: &str) -> SocketAddr {
|
||||
addr.parse().unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn local_caller_needs_loopback_peer_and_loopback_host() {
|
||||
for host in [
|
||||
"127.0.0.1:8787",
|
||||
"127.0.0.1",
|
||||
"localhost:8787",
|
||||
"LOCALHOST",
|
||||
"[::1]:8787",
|
||||
"::1",
|
||||
"127.5.6.7:8787", // the whole 127/8 block is loopback
|
||||
] {
|
||||
assert!(
|
||||
caller_is_local(peer("127.0.0.1:5555"), &headers_with_host(host)),
|
||||
"{host} should read as local"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The DNS-rebinding case, and the entire reason the `Host` check
|
||||
/// exists: the victim's browser really is on loopback, so the peer
|
||||
/// check alone would wave this through while attacker JS reads the
|
||||
/// per-request feed.
|
||||
#[test]
|
||||
fn loopback_peer_with_foreign_host_is_not_local() {
|
||||
for host in ["attacker.com", "attacker.com:8787", "evil.localhost.dev"] {
|
||||
assert!(
|
||||
!caller_is_local(peer("127.0.0.1:5555"), &headers_with_host(host)),
|
||||
"{host} must not read as local — DNS rebinding"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remote_peer_is_never_local_even_claiming_a_loopback_host() {
|
||||
assert!(!caller_is_local(
|
||||
peer("10.1.2.3:5555"),
|
||||
&headers_with_host("127.0.0.1:8787")
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_host_header_fails_closed() {
|
||||
assert!(!caller_is_local(peer("127.0.0.1:5555"), &HeaderMap::new()));
|
||||
}
|
||||
}
|
||||
@@ -26,6 +26,7 @@ use crate::config::Config;
|
||||
use crate::error::ProxyError;
|
||||
use crate::headers::{build_forward_request_headers, filter_response_headers};
|
||||
use crate::health::{healthz, healthz_upstream, rollout_status};
|
||||
use crate::observability::{capture, ledger};
|
||||
use crate::websocket::ws_handler;
|
||||
// Phase F PR-F1: imported as `classify_auth_mode` to make the call
|
||||
// site self-documenting. `AuthMode` is re-exported under the same
|
||||
@@ -80,6 +81,13 @@ pub struct AppState {
|
||||
/// route hits `bearer()`. Tests override via
|
||||
/// [`AppState::with_token_source`].
|
||||
pub vertex_token_source: Arc<dyn crate::vertex::TokenSource>,
|
||||
/// Native savings ledger (one per process). Loaded from
|
||||
/// `Config::stats_path` when persistence is configured, else
|
||||
/// in-memory. Handlers record into it after each finished
|
||||
/// request; `/stats*` and `/dashboard` read from it. Recording
|
||||
/// call sites gate on `Config::stats` — when stats are disabled
|
||||
/// this is an inert in-memory ledger.
|
||||
pub stats: Arc<ledger::Ledger>,
|
||||
}
|
||||
|
||||
/// PR-E6: maximum number of sessions tracked by the drift detector
|
||||
@@ -126,6 +134,15 @@ impl AppState {
|
||||
let vertex_token_source: Arc<dyn crate::vertex::TokenSource> =
|
||||
Arc::new(crate::vertex::adc::GcpAdcTokenSource::new());
|
||||
|
||||
let stats = if config.stats {
|
||||
match config.stats_path.clone() {
|
||||
Some(path) => Arc::new(ledger::Ledger::load(path)),
|
||||
None => Arc::new(ledger::Ledger::in_memory()),
|
||||
}
|
||||
} else {
|
||||
Arc::new(ledger::Ledger::in_memory())
|
||||
};
|
||||
|
||||
Ok(Self {
|
||||
config: Arc::new(config),
|
||||
client,
|
||||
@@ -135,6 +152,7 @@ impl AppState {
|
||||
cache_stabilization::beta_sticky::BETA_TRACKER_CAPACITY,
|
||||
),
|
||||
vertex_token_source,
|
||||
stats,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -199,7 +217,39 @@ pub fn build_app(state: AppState) -> Router {
|
||||
let mut router = Router::new()
|
||||
.route("/healthz", get(healthz))
|
||||
.route("/healthz/upstream", get(healthz_upstream))
|
||||
.route("/rollout/status", get(rollout_status))
|
||||
.route("/rollout/status", get(rollout_status));
|
||||
|
||||
// Native savings stats + dashboard. Served locally (never
|
||||
// tunnelled upstream) so the Rust proxy is self-sufficient for
|
||||
// savings observability. Gated on `--stats` so an operator can
|
||||
// restore pure-passthrough behaviour for these paths.
|
||||
if state.config.stats {
|
||||
router = router
|
||||
.route(
|
||||
"/stats",
|
||||
get(crate::observability::stats_http::handle_stats),
|
||||
)
|
||||
.route(
|
||||
"/stats/timeseries",
|
||||
get(crate::observability::stats_http::handle_timeseries),
|
||||
)
|
||||
.route(
|
||||
"/stats/events",
|
||||
get(crate::observability::stats_http::handle_events),
|
||||
)
|
||||
.route(
|
||||
"/dashboard",
|
||||
get(crate::observability::stats_http::handle_dashboard),
|
||||
);
|
||||
} else {
|
||||
tracing::warn!(
|
||||
event = "stats_disabled",
|
||||
"native savings stats disabled by --stats=false; \
|
||||
/stats*, /dashboard fall through to the upstream forwarder \
|
||||
and no request telemetry is recorded"
|
||||
);
|
||||
}
|
||||
router = router
|
||||
// PR-C2: explicit POST route for /v1/chat/completions. The
|
||||
// handler buffers the body and re-injects it into
|
||||
// `forward_http`, which runs the OpenAI live-zone gate
|
||||
@@ -614,10 +664,10 @@ pub(crate) async fn forward_http(
|
||||
// controls what the buffered branch does (currently both `Off`
|
||||
// and `LiveZone` passthrough); Phase B will branch on it inside
|
||||
// `compress_anthropic_request`.
|
||||
let should_intercept = state.config.compression
|
||||
&& method == axum::http::Method::POST
|
||||
let is_llm_json_post = method == axum::http::Method::POST
|
||||
&& compression::is_compressible_path(uri.path())
|
||||
&& is_application_json(req.headers());
|
||||
let should_intercept = state.config.compression && is_llm_json_post;
|
||||
|
||||
// PR-E6: capture a header snapshot BEFORE the body is consumed so
|
||||
// the drift detector can derive a per-session key from
|
||||
@@ -635,6 +685,33 @@ pub(crate) async fn forward_http(
|
||||
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())
|
||||
.map_err(|e| ProxyError::InvalidHeader(e.to_string()))?;
|
||||
|
||||
// Savings ledger. Deliberately NOT gated on `should_intercept`:
|
||||
// spend/caching stats are worth answering on a pure passthrough
|
||||
// deployment too (compression savings are truthfully 0 there),
|
||||
// so the gate is the LLM-endpoint shape alone and the model id
|
||||
// is learned from the response when the request didn't carry it.
|
||||
// Finalized exactly once, when the response is fully observed.
|
||||
let should_record = state.config.stats && is_llm_json_post;
|
||||
let mut pending_stats = should_record
|
||||
.then(|| compression::classify_compressible_path(uri.path()))
|
||||
.flatten()
|
||||
.map(|endpoint| {
|
||||
let provider = match endpoint {
|
||||
compression::CompressibleEndpoint::AnthropicMessages => ledger::provider::ANTHROPIC,
|
||||
compression::CompressibleEndpoint::OpenAiChatCompletions => {
|
||||
ledger::provider::OPENAI_CHAT
|
||||
}
|
||||
compression::CompressibleEndpoint::OpenAiResponses => {
|
||||
ledger::provider::OPENAI_RESPONSES
|
||||
}
|
||||
};
|
||||
capture::PendingRecord::new(
|
||||
state.stats.clone(),
|
||||
ledger::RequestEvent::new(provider, "", &request_id),
|
||||
start,
|
||||
)
|
||||
});
|
||||
|
||||
let upstream_resp = if should_intercept {
|
||||
// Buffer up to `compression_max_body_bytes`. If the body
|
||||
// exceeds this, the body is already partially consumed and
|
||||
@@ -660,6 +737,9 @@ pub(crate) async fn forward_http(
|
||||
"compression: Content-Length exceeds buffer limit; \
|
||||
returning 413 without consuming body"
|
||||
);
|
||||
// A proxy-side rejection is still an outcome:
|
||||
// record it as failed rather than dropping it.
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return Err(ProxyError::PayloadTooLarge(format!(
|
||||
"request Content-Length {len} exceeds compression \
|
||||
buffer limit ({max} bytes)"
|
||||
@@ -677,6 +757,7 @@ pub(crate) async fn forward_http(
|
||||
"compression: body exceeds buffer limit; failing loudly (cannot \
|
||||
resume streaming once the body has been partially consumed)"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return Err(ProxyError::PayloadTooLarge(format!(
|
||||
"request body exceeds compression buffer limit ({max} bytes): {e}"
|
||||
)));
|
||||
@@ -735,7 +816,14 @@ pub(crate) async fn forward_http(
|
||||
// enough that a canonical-bytes hash would compare apples
|
||||
// to oranges. The volatile detector handles its own
|
||||
// shape-dispatch via `ApiKind::from_endpoint`.
|
||||
let mut stats_model: Option<String> = None;
|
||||
if let Ok(parsed) = serde_json::from_slice::<serde_json::Value>(&buffered) {
|
||||
// Model id for the savings ledger — read from the same
|
||||
// sanitized bytes the dispatcher compresses.
|
||||
stats_model = parsed
|
||||
.get("model")
|
||||
.and_then(|m| m.as_str())
|
||||
.map(str::to_owned);
|
||||
// PR-E5: volatile-content detector. Emits one WARN per
|
||||
// finding (capped at 10) for content that busts cache
|
||||
// (timestamps, UUIDs, ID-named fields).
|
||||
@@ -807,6 +895,10 @@ pub(crate) async fn forward_http(
|
||||
}
|
||||
}
|
||||
}
|
||||
// Compression-side numbers for the savings ledger, captured
|
||||
// out of the Compressed arm below (the outcome is consumed
|
||||
// by the body-selection match).
|
||||
let mut stats_compression: (u64, u64, Vec<String>) = (0, 0, Vec::new());
|
||||
let outcome = match endpoint {
|
||||
compression::CompressibleEndpoint::AnthropicMessages => {
|
||||
// PR-E3: thread the F1-classified auth_mode into the
|
||||
@@ -892,6 +984,11 @@ pub(crate) async fn forward_http(
|
||||
markers_inserted,
|
||||
per_strategy_tokens,
|
||||
} => {
|
||||
stats_compression = (
|
||||
tokens_before as u64,
|
||||
tokens_after as u64,
|
||||
strategies_applied.iter().map(|s| s.to_string()).collect(),
|
||||
);
|
||||
tracing::info!(
|
||||
request_id = %request_id,
|
||||
path = %path_for_log,
|
||||
@@ -1012,34 +1109,71 @@ pub(crate) async fn forward_http(
|
||||
compression::CompressibleEndpoint::AnthropicMessages => body_to_send,
|
||||
};
|
||||
|
||||
if let Some(p) = pending_stats.as_mut() {
|
||||
// Request-side model (already `[1m]`-sanitized above, so
|
||||
// it matches what upstream bills) wins over the response
|
||||
// echo; the compression numbers are only knowable here.
|
||||
if let Some(m) = stats_model.as_deref() {
|
||||
p.set_model(m);
|
||||
}
|
||||
let (before, after, transforms) = stats_compression;
|
||||
p.set_compression(before, after, transforms);
|
||||
}
|
||||
|
||||
// Forward the (Phase A: identical) buffered bytes. reqwest
|
||||
// sets its own Content-Length from the body bytes — the
|
||||
// existing `build_forward_request_headers` already strips
|
||||
// the client-supplied Content-Length for us.
|
||||
state
|
||||
match state
|
||||
.client
|
||||
.request(reqwest_method, upstream_url.clone())
|
||||
.headers(outgoing_headers)
|
||||
.body(body_to_send)
|
||||
.send()
|
||||
.await?
|
||||
.await
|
||||
{
|
||||
Ok(resp) => resp,
|
||||
Err(e) => {
|
||||
// Transport/connect failures are request outcomes
|
||||
// too — record them so outages are visible in /stats.
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return Err(e.into());
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Pure streaming path — the original passthrough behaviour.
|
||||
let body_stream =
|
||||
TryStreamExt::map_err(req.into_body().into_data_stream(), std::io::Error::other);
|
||||
let reqwest_body = reqwest::Body::wrap_stream(body_stream);
|
||||
state
|
||||
match state
|
||||
.client
|
||||
.request(reqwest_method, upstream_url.clone())
|
||||
.headers(outgoing_headers)
|
||||
.body(reqwest_body)
|
||||
.send()
|
||||
.await?
|
||||
.await
|
||||
{
|
||||
Ok(resp) => resp,
|
||||
Err(e) => {
|
||||
// Same contract as the buffered arm.
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return Err(e.into());
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
let upstream_status = upstream_resp.status();
|
||||
let status = StatusCode::from_u16(upstream_status.as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
|
||||
|
||||
// Savings honesty rule: a non-2xx upstream is a failed request —
|
||||
// it counts, but accrues no savings (the ledger enforces this on
|
||||
// the failed flag).
|
||||
if !upstream_status.is_success() {
|
||||
if let Some(p) = pending_stats.as_mut() {
|
||||
p.mark_failed();
|
||||
}
|
||||
}
|
||||
|
||||
// PR-A8 / P5-57: capture the upstream request id BEFORE we move
|
||||
// `upstream_resp.headers()` into the response filter. Anthropic
|
||||
// emits `request-id` (lowercase, no `x-`); OpenAI emits
|
||||
@@ -1155,8 +1289,32 @@ pub(crate) async fn forward_http(
|
||||
let parser_tx = if !matches!(sse_kind, SseStreamKind::None) {
|
||||
let (tx, rx) = tokio::sync::mpsc::channel::<bytes::Bytes>(SSE_PARSER_QUEUE_DEPTH);
|
||||
let rid_for_parser = request_id.clone();
|
||||
tokio::spawn(run_sse_state_machine(sse_kind, rx, rid_for_parser));
|
||||
// Deferred recording: the pending stats event rides into the
|
||||
// state machine and is finalized when the stream closes, so
|
||||
// output/cache token counts come from the real SSE usage
|
||||
// frames instead of recording zero at header time.
|
||||
tokio::spawn(run_sse_state_machine(
|
||||
sse_kind,
|
||||
rx,
|
||||
rid_for_parser,
|
||||
pending_stats.take(),
|
||||
));
|
||||
Some(tx)
|
||||
} else if let Some(p) = pending_stats.take() {
|
||||
// Non-SSE response on a recorded LLM path: tee the JSON body
|
||||
// into a bounded buffer and parse usage when it completes.
|
||||
let shape = match SseStreamKind::for_request_path(&path_for_log) {
|
||||
SseStreamKind::OpenAiChat => capture::ResponseShape::OpenAiChat,
|
||||
SseStreamKind::OpenAiResponses => capture::ResponseShape::OpenAiResponses,
|
||||
// Anthropic (/v1/messages). Note: /v1/responses with the
|
||||
// streaming state machine disabled still matches the
|
||||
// OpenAiResponses arm above — for_request_path classifies
|
||||
// by path, independent of the enable flag. An SSE body
|
||||
// teed here simply fails the JSON parse and records
|
||||
// compression-side numbers alone.
|
||||
_ => capture::ResponseShape::Anthropic,
|
||||
};
|
||||
Some(capture::spawn_json_usage_capture(shape, p))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
@@ -1221,11 +1379,9 @@ pub(crate) async fn forward_http(
|
||||
}
|
||||
|
||||
/// Bound on the in-flight queue between the byte-passthrough and the
|
||||
/// SSE state-machine task. Picked so that under steady-state streaming
|
||||
/// load (~5 events/100ms typical) the parser is never blocked on
|
||||
/// queue space, yet a stalled parser can't grow memory unboundedly.
|
||||
/// Tunable via `proxy.toml` if a deployment finds this insufficient.
|
||||
const SSE_PARSER_QUEUE_DEPTH: usize = 256;
|
||||
/// SSE state-machine task — see [`TEE_QUEUE_DEPTH`] for the sizing
|
||||
/// rationale it shares with every other telemetry tee.
|
||||
use crate::observability::capture::TEE_QUEUE_DEPTH as SSE_PARSER_QUEUE_DEPTH;
|
||||
|
||||
/// Which provider's state machine should run on this stream. Picked
|
||||
/// from the *request* path because the response content-type
|
||||
@@ -1381,10 +1537,15 @@ fn maybe_inject_openai_prompt_cache_key(
|
||||
|
||||
/// Drive the per-provider state machine over a stream of byte chunks.
|
||||
/// Lives in its own task; the byte path never waits on it.
|
||||
///
|
||||
/// `pending_stats` (when the savings ledger is recording this
|
||||
/// request) is finalized here — after the stream closes — with the
|
||||
/// usage the state machine accumulated.
|
||||
async fn run_sse_state_machine(
|
||||
kind: SseStreamKind,
|
||||
mut rx: tokio::sync::mpsc::Receiver<bytes::Bytes>,
|
||||
request_id: String,
|
||||
pending_stats: Option<capture::PendingRecord>,
|
||||
) {
|
||||
use crate::sse::framing::SseFramer;
|
||||
|
||||
@@ -1455,6 +1616,20 @@ async fn run_sse_state_machine(
|
||||
blocks = state.blocks.len(),
|
||||
"sse stream closed"
|
||||
);
|
||||
if let Some(mut p) = pending_stats {
|
||||
if let Some(model) = state.model.as_deref() {
|
||||
p.set_model_if_unknown(model);
|
||||
}
|
||||
// A 200 OK stream can still fail in-band: Anthropic
|
||||
// emits `{"type":"error"}` mid-stream (e.g. overloaded
|
||||
// during generation) and the state machine lands in
|
||||
// `Errored`. That is not a billable success — record
|
||||
// it as failed so retries can't inflate savings.
|
||||
if matches!(state.status, crate::sse::anthropic::StreamStatus::Errored) {
|
||||
p.mark_failed();
|
||||
}
|
||||
p.finalize(capture::usage_from_anthropic_state(&state));
|
||||
}
|
||||
}
|
||||
SseStreamKind::OpenAiChat => {
|
||||
let mut state = crate::sse::openai_chat::ChunkState::new();
|
||||
@@ -1558,6 +1733,19 @@ async fn run_sse_state_machine(
|
||||
has_usage = state.usage.is_some(),
|
||||
"sse stream closed"
|
||||
);
|
||||
if let Some(mut p) = pending_stats {
|
||||
if let Some(model) = state.model.as_deref() {
|
||||
p.set_model_if_unknown(model);
|
||||
}
|
||||
let usage = state
|
||||
.usage
|
||||
.as_ref()
|
||||
.and_then(|u| {
|
||||
capture::usage_from_usage_block(capture::ResponseShape::OpenAiChat, u)
|
||||
})
|
||||
.unwrap_or_default();
|
||||
p.finalize(usage);
|
||||
}
|
||||
}
|
||||
SseStreamKind::OpenAiResponses => {
|
||||
let mut state = crate::sse::openai_responses::ResponseState::new();
|
||||
@@ -1689,8 +1877,33 @@ async fn run_sse_state_machine(
|
||||
incomplete_reason = state.incomplete_reason.as_deref().unwrap_or(""),
|
||||
"sse stream closed"
|
||||
);
|
||||
if let Some(mut p) = pending_stats {
|
||||
if let Some(model) = state.model.as_deref() {
|
||||
p.set_model_if_unknown(model);
|
||||
}
|
||||
// `response.failed` / `response.incomplete` are not
|
||||
// billable successes — count them as failures even
|
||||
// though the HTTP status was 2xx.
|
||||
if matches!(state.terminal_status(), Some("failed") | Some("incomplete")) {
|
||||
p.mark_failed();
|
||||
}
|
||||
let usage = state
|
||||
.usage
|
||||
.as_ref()
|
||||
.and_then(|u| {
|
||||
capture::usage_from_usage_block(capture::ResponseShape::OpenAiResponses, u)
|
||||
})
|
||||
.unwrap_or_default();
|
||||
p.finalize(usage);
|
||||
}
|
||||
}
|
||||
SseStreamKind::None => {
|
||||
if let Some(p) = pending_stats {
|
||||
// No parser for this stream — record what the
|
||||
// request side knew (compression estimates only).
|
||||
p.finalize(ledger::Usage::default());
|
||||
}
|
||||
}
|
||||
SseStreamKind::None => {}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -124,6 +124,35 @@ impl SseFramer {
|
||||
if chunk.is_empty() {
|
||||
return;
|
||||
}
|
||||
// Cap the accumulator: a stream that never emits a blank-line
|
||||
// terminator (a non-SSE body routed here, or a hostile
|
||||
// upstream) would otherwise grow this buffer for the life of
|
||||
// the stream. A real SSE event is a few KB; 4 MiB is orders
|
||||
// of magnitude past any legitimate frame.
|
||||
//
|
||||
// On overflow we drop the buffer and the triggering chunk, so
|
||||
// parsing resumes mid-stream at an arbitrary byte offset.
|
||||
// Usually that garbage forms a prefix-less line the parser
|
||||
// ignores; it CAN also resume on a syntactically valid
|
||||
// `data:` line and yield one truncated event. That is
|
||||
// acceptable only because every consumer is a telemetry tee
|
||||
// off the byte path (`SseFramer` is never on the client
|
||||
// path — see the three callers), so the worst case is a wrong
|
||||
// usage number for one stream, bounded further by the
|
||||
// ledger's own per-event token ceiling. Client bytes are
|
||||
// never touched. Revisit this cap before framing anything
|
||||
// client-facing with it.
|
||||
const MAX_BUFFERED_BYTES: usize = 4 * 1024 * 1024;
|
||||
if self.buf.len() + chunk.len() > MAX_BUFFERED_BYTES {
|
||||
tracing::warn!(
|
||||
event = "sse_framer_buffer_overflow",
|
||||
buffered = self.buf.len(),
|
||||
"no event terminator within 4 MiB; discarding buffered \
|
||||
bytes (telemetry-only consumer — client stream unaffected)"
|
||||
);
|
||||
self.buf.clear();
|
||||
return;
|
||||
}
|
||||
self.buf.extend_from_slice(chunk);
|
||||
}
|
||||
|
||||
|
||||
@@ -90,6 +90,7 @@ use axum::http::{HeaderMap, Method, StatusCode, Uri};
|
||||
use axum::response::Response;
|
||||
use std::net::SocketAddr;
|
||||
|
||||
use crate::observability::{capture, ledger};
|
||||
use crate::proxy::AppState;
|
||||
|
||||
/// Single axum handler mounted at the
|
||||
@@ -157,6 +158,15 @@ pub async fn handle_vertex_predict_dispatch(
|
||||
segment = %model_action,
|
||||
"vertex path final segment missing `:verb` separator"
|
||||
);
|
||||
// A proxy-side rejection is still an outcome for this
|
||||
// request. The model id is whatever the caller sent (it
|
||||
// is what failed to parse); the ledger clamps it.
|
||||
capture::finalize_rejected(capture::start_pending(
|
||||
&state,
|
||||
ledger::provider::VERTEX,
|
||||
&model_action,
|
||||
&request_id,
|
||||
));
|
||||
return Response::builder()
|
||||
.status(StatusCode::NOT_FOUND)
|
||||
.body(Body::from("vertex path: bad model_action"))
|
||||
@@ -173,6 +183,12 @@ pub async fn handle_vertex_predict_dispatch(
|
||||
verb = %verb_str,
|
||||
"vertex path verb not recognized; only rawPredict / streamRawPredict are supported"
|
||||
);
|
||||
capture::finalize_rejected(capture::start_pending(
|
||||
&state,
|
||||
ledger::provider::VERTEX,
|
||||
model_id,
|
||||
&request_id,
|
||||
));
|
||||
return Response::builder()
|
||||
.status(StatusCode::NOT_FOUND)
|
||||
.body(Body::from("vertex: unknown verb"))
|
||||
|
||||
@@ -36,6 +36,8 @@ use std::net::SocketAddr;
|
||||
|
||||
use crate::compression;
|
||||
use crate::headers::{build_forward_request_headers, filter_response_headers};
|
||||
use crate::observability::capture::TEE_QUEUE_DEPTH as VERTEX_SSE_QUEUE_DEPTH;
|
||||
use crate::observability::{capture, ledger};
|
||||
use crate::proxy::AppState;
|
||||
use crate::vertex::{adc::TokenSourceError, envelope, VertexVerb};
|
||||
|
||||
@@ -76,6 +78,13 @@ pub(crate) async fn forward_vertex_request(
|
||||
) -> Response {
|
||||
let path_for_log = uri.path().to_string();
|
||||
|
||||
// Constructed at handler ENTRY so every exit — including
|
||||
// proxy-side rejections (oversized body, envelope mismatch, ADC
|
||||
// failure) — records exactly once. An ADC outage must show in
|
||||
// /stats as a failure spike, not zero traffic.
|
||||
let mut pending_stats =
|
||||
capture::start_pending(&state, ledger::provider::VERTEX, &ctx.model_id, &request_id);
|
||||
|
||||
// ─── 1. BUFFER BODY ────────────────────────────────────────────────
|
||||
let max = state.config.compression_max_body_bytes as usize;
|
||||
let buffered = match to_bytes(body, max).await {
|
||||
@@ -89,6 +98,7 @@ pub(crate) async fn forward_vertex_request(
|
||||
error = %e,
|
||||
"vertex request body exceeds compression buffer limit; failing loudly"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(
|
||||
StatusCode::PAYLOAD_TOO_LARGE,
|
||||
"request body exceeds buffer limit",
|
||||
@@ -122,6 +132,7 @@ pub(crate) async fn forward_vertex_request(
|
||||
error = %e,
|
||||
"vertex envelope did not match expected shape; rejecting with 400"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(StatusCode::BAD_REQUEST, "vertex envelope invalid");
|
||||
}
|
||||
}
|
||||
@@ -133,6 +144,7 @@ pub(crate) async fn forward_vertex_request(
|
||||
// The dispatcher uses RawValue-based surgery so `anthropic_version`
|
||||
// (and any other non-`messages` top-level field) round-trips
|
||||
// byte-equal. Compression off → buffered bytes used unchanged.
|
||||
let mut stats_compression: (u64, u64, Vec<String>) = (0, 0, Vec::new());
|
||||
let body_to_send = if state.config.compression {
|
||||
// PR-E3: Vertex uses GCP ADC bearer-token auth downstream, not
|
||||
// Anthropic credentials, so the PAYG/OAuth/subscription
|
||||
@@ -177,6 +189,11 @@ pub(crate) async fn forward_vertex_request(
|
||||
markers = markers_inserted.len(),
|
||||
"vertex live-zone compression applied"
|
||||
);
|
||||
stats_compression = (
|
||||
tokens_before as u64,
|
||||
tokens_after as u64,
|
||||
strategies_applied.iter().map(|s| s.to_string()).collect(),
|
||||
);
|
||||
body
|
||||
}
|
||||
compression::Outcome::Passthrough { reason } => {
|
||||
@@ -201,6 +218,11 @@ pub(crate) async fn forward_vertex_request(
|
||||
buffered
|
||||
};
|
||||
|
||||
if let Some(p) = pending_stats.as_mut() {
|
||||
let (before, after, transforms) = stats_compression;
|
||||
p.set_compression(before, after, transforms);
|
||||
}
|
||||
|
||||
// ─── 4. RESOLVE BEARER TOKEN ───────────────────────────────────────
|
||||
let bearer = match state.vertex_token_source.bearer().await {
|
||||
Ok(t) => t,
|
||||
@@ -221,6 +243,7 @@ pub(crate) async fn forward_vertex_request(
|
||||
TokenSourceError::ProviderInit(_) => StatusCode::BAD_GATEWAY,
|
||||
TokenSourceError::Fetch(_) => StatusCode::BAD_GATEWAY,
|
||||
};
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(status, "vertex ADC token fetch failed");
|
||||
}
|
||||
};
|
||||
@@ -248,6 +271,7 @@ pub(crate) async fn forward_vertex_request(
|
||||
error = %e,
|
||||
"could not construct vertex upstream URL"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(StatusCode::BAD_GATEWAY, "vertex upstream URL build failed");
|
||||
}
|
||||
};
|
||||
@@ -293,6 +317,7 @@ pub(crate) async fn forward_vertex_request(
|
||||
error = %e,
|
||||
"ADC bearer token contained invalid header bytes; refusing to forward"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(StatusCode::BAD_GATEWAY, "vertex auth header build failed");
|
||||
}
|
||||
}
|
||||
@@ -308,9 +333,11 @@ pub(crate) async fn forward_vertex_request(
|
||||
error = %e,
|
||||
"could not convert axum method to reqwest method"
|
||||
);
|
||||
capture::finalize_rejected(pending_stats.take());
|
||||
return error_response(StatusCode::BAD_REQUEST, "vertex method invalid");
|
||||
}
|
||||
};
|
||||
|
||||
let upstream_resp = match state
|
||||
.client
|
||||
.request(reqwest_method, upstream_url.clone())
|
||||
@@ -321,6 +348,9 @@ pub(crate) async fn forward_vertex_request(
|
||||
{
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
if let Some(p) = pending_stats.take() {
|
||||
p.finalize_failed();
|
||||
}
|
||||
tracing::warn!(
|
||||
event = "vertex_upstream_error",
|
||||
request_id = %request_id,
|
||||
@@ -335,6 +365,11 @@ pub(crate) async fn forward_vertex_request(
|
||||
// ─── 8. STREAM RESPONSE ────────────────────────────────────────────
|
||||
let upstream_status = upstream_resp.status();
|
||||
let status = StatusCode::from_u16(upstream_status.as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
|
||||
if !upstream_status.is_success() {
|
||||
if let Some(p) = pending_stats.as_mut() {
|
||||
p.mark_failed();
|
||||
}
|
||||
}
|
||||
let resp_headers = filter_response_headers(upstream_resp.headers());
|
||||
|
||||
// PR-C1 reuse: when `attach_sse_tee` is set AND the upstream
|
||||
@@ -355,7 +390,11 @@ pub(crate) async fn forward_vertex_request(
|
||||
let parser_tx = if attach_sse_tee && is_sse {
|
||||
let (tx, rx) = tokio::sync::mpsc::channel::<bytes::Bytes>(VERTEX_SSE_QUEUE_DEPTH);
|
||||
let rid = request_id.clone();
|
||||
tokio::spawn(run_anthropic_sse_state_machine(rx, rid));
|
||||
tokio::spawn(run_anthropic_sse_state_machine(
|
||||
rx,
|
||||
rid,
|
||||
pending_stats.take(),
|
||||
));
|
||||
tracing::info!(
|
||||
event = "vertex_sse_tee_engaged",
|
||||
request_id = %request_id,
|
||||
@@ -363,7 +402,12 @@ pub(crate) async fn forward_vertex_request(
|
||||
);
|
||||
Some(tx)
|
||||
} else {
|
||||
None
|
||||
// Non-SSE (rawPredict, or a JSON error on the streaming
|
||||
// verb): tee the body into a bounded buffer and parse the
|
||||
// Anthropic-shape usage block when it completes.
|
||||
pending_stats
|
||||
.take()
|
||||
.map(|p| capture::spawn_json_usage_capture(capture::ResponseShape::Anthropic, p))
|
||||
};
|
||||
|
||||
use futures_util::StreamExt as _;
|
||||
@@ -434,13 +478,6 @@ fn error_response(status: StatusCode, msg: &'static str) -> Response {
|
||||
.expect("static error response")
|
||||
}
|
||||
|
||||
/// Bound on the in-flight queue between the byte-passthrough and the
|
||||
/// SSE state-machine task. Mirrors the
|
||||
/// `crate::proxy::SSE_PARSER_QUEUE_DEPTH` rationale (256 events ≈ 5
|
||||
/// seconds of typical Anthropic streaming under the per-100ms event
|
||||
/// rate; keeps memory bounded even if the parser stalls).
|
||||
const VERTEX_SSE_QUEUE_DEPTH: usize = 256;
|
||||
|
||||
/// Drive the Anthropic SSE state machine over a stream of byte
|
||||
/// chunks. Lives in its own spawned task; the byte path is fed via a
|
||||
/// best-effort tee from [`forward_vertex_request`] and never blocks
|
||||
@@ -448,6 +485,7 @@ const VERTEX_SSE_QUEUE_DEPTH: usize = 256;
|
||||
async fn run_anthropic_sse_state_machine(
|
||||
mut rx: tokio::sync::mpsc::Receiver<bytes::Bytes>,
|
||||
request_id: String,
|
||||
pending_stats: Option<capture::PendingRecord>,
|
||||
) {
|
||||
use crate::sse::framing::SseFramer;
|
||||
let mut framer = SseFramer::new();
|
||||
@@ -487,4 +525,12 @@ async fn run_anthropic_sse_state_machine(
|
||||
blocks = state.blocks.len(),
|
||||
"vertex sse stream closed"
|
||||
);
|
||||
if let Some(mut p) = pending_stats {
|
||||
// In-band `{"type":"error"}` on a 200 OK stream is not a
|
||||
// billable success (same rule as the direct Anthropic lane).
|
||||
if matches!(state.status, crate::sse::anthropic::StreamStatus::Errored) {
|
||||
p.mark_failed();
|
||||
}
|
||||
p.finalize(capture::usage_from_anthropic_state(&state));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -111,3 +111,17 @@ pub fn install_static_token_source(mut state: AppState, bearer: &str) -> AppStat
|
||||
pub fn _config_ref() -> Arc<Config> {
|
||||
Arc::new(Config::for_test(Url::parse("http://127.0.0.1:1").unwrap()))
|
||||
}
|
||||
|
||||
/// Static, obviously-fake AWS credentials for SigV4-signing tests.
|
||||
/// Shared by every Bedrock test binary — the mock upstream never
|
||||
/// validates the signature, only that one is present.
|
||||
#[allow(dead_code)]
|
||||
pub fn test_credentials() -> aws_credential_types::Credentials {
|
||||
aws_credential_types::Credentials::new(
|
||||
"AKIAEXAMPLEAKIDFORTEST",
|
||||
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
None,
|
||||
None,
|
||||
"test",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -20,14 +20,13 @@
|
||||
|
||||
mod common;
|
||||
|
||||
use aws_credential_types::Credentials;
|
||||
use axum::body::Body;
|
||||
use axum::extract::{Extension, State};
|
||||
use axum::http::StatusCode;
|
||||
use axum::routing::post;
|
||||
use axum::Router;
|
||||
use bytes::Bytes;
|
||||
use common::start_proxy_with_state;
|
||||
use common::{start_proxy_with_state, test_credentials};
|
||||
use headroom_core::auth_mode::AuthMode;
|
||||
use headroom_proxy::AppState;
|
||||
use serde_json::{json, Value};
|
||||
@@ -41,16 +40,6 @@ use wiremock::{Mock, MockServer, ResponseTemplate};
|
||||
|
||||
const TEST_MODEL: &str = "anthropic.claude-3-haiku-20240307-v1:0";
|
||||
|
||||
fn test_credentials() -> Credentials {
|
||||
Credentials::new(
|
||||
"AKIAEXAMPLEAKIDFORTEST",
|
||||
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
None,
|
||||
None,
|
||||
"test",
|
||||
)
|
||||
}
|
||||
|
||||
#[derive(Default, Clone, Debug)]
|
||||
struct CapturedRequest {
|
||||
body: Option<Vec<u8>>,
|
||||
|
||||
@@ -35,8 +35,7 @@
|
||||
|
||||
mod common;
|
||||
|
||||
use aws_credential_types::Credentials;
|
||||
use common::start_proxy_with_state;
|
||||
use common::{start_proxy_with_state, test_credentials};
|
||||
use serde_json::{json, Value};
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::sync::{Arc, Mutex};
|
||||
@@ -132,16 +131,6 @@ fn assert_byte_equal_sha256(inbound: &[u8], received: &[u8]) {
|
||||
);
|
||||
}
|
||||
|
||||
fn test_credentials() -> Credentials {
|
||||
Credentials::new(
|
||||
"AKIAEXAMPLEAKIDFORTEST",
|
||||
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
None,
|
||||
None,
|
||||
"test",
|
||||
)
|
||||
}
|
||||
|
||||
/// Boot a proxy pointed at the wiremock upstream as the Bedrock
|
||||
/// endpoint. The fake-upstream URL goes into `bedrock_endpoint`;
|
||||
/// the regular `upstream` field is set to a sentinel because the
|
||||
|
||||
@@ -23,9 +23,8 @@
|
||||
|
||||
mod common;
|
||||
|
||||
use aws_credential_types::Credentials;
|
||||
use bytes::{Bytes, BytesMut};
|
||||
use common::start_proxy_with_state;
|
||||
use common::{start_proxy_with_state, test_credentials};
|
||||
use headroom_proxy::bedrock::MessageBuilder;
|
||||
use serde_json::json;
|
||||
use url::Url;
|
||||
@@ -49,16 +48,6 @@ const TEST_REGION_LATENCY: &str = "us-test-latency-1";
|
||||
const TEST_REGION_EVENTSTREAM: &str = "us-test-eventstream-1";
|
||||
const TEST_REGION_SCRAPE: &str = "us-test-scrape-1";
|
||||
|
||||
fn test_credentials() -> Credentials {
|
||||
Credentials::new(
|
||||
"AKIAEXAMPLEAKIDFORTEST",
|
||||
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
None,
|
||||
None,
|
||||
"test",
|
||||
)
|
||||
}
|
||||
|
||||
async fn bedrock_proxy_with_region(
|
||||
upstream: &MockServer,
|
||||
region: &str,
|
||||
|
||||
@@ -22,9 +22,8 @@
|
||||
|
||||
mod common;
|
||||
|
||||
use aws_credential_types::Credentials;
|
||||
use bytes::{Bytes, BytesMut};
|
||||
use common::start_proxy_with_state;
|
||||
use common::{start_proxy_with_state, test_credentials};
|
||||
use headroom_proxy::bedrock::{
|
||||
parse_eventstream, CrcValidation, EventStreamParser, HeaderValue, MessageBuilder, ParseError,
|
||||
};
|
||||
@@ -36,16 +35,6 @@ use wiremock::{Mock, MockServer, ResponseTemplate};
|
||||
|
||||
const TEST_MODEL: &str = "anthropic.claude-3-haiku-20240307-v1:0";
|
||||
|
||||
fn test_credentials() -> Credentials {
|
||||
Credentials::new(
|
||||
"AKIAEXAMPLEAKIDFORTEST",
|
||||
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
None,
|
||||
None,
|
||||
"test",
|
||||
)
|
||||
}
|
||||
|
||||
/// Synthesise a minimal Bedrock-shape Anthropic stream as binary
|
||||
/// EventStream bytes. The exact JSON payload mirrors what real
|
||||
/// Bedrock emits for a 2-token completion.
|
||||
@@ -577,3 +566,77 @@ proptest! {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Regression (caught in review of the first stats PR, #1057; the fix
|
||||
/// never landed with it): in SSE-translation mode, an unknown
|
||||
/// `:event-type` (`TranslateOutcome::Skip`) arriving in the same
|
||||
/// upstream chunk as known messages must not strand the known
|
||||
/// messages in the parser buffer. The pre-fix drain shape pulled a
|
||||
/// FRESH upstream chunk after a Skip, so when the Skip'd frame sat in
|
||||
/// the final chunk, every complete message buffered behind it was
|
||||
/// dropped and the client's stream ended early.
|
||||
#[tokio::test]
|
||||
async fn skip_in_final_chunk_does_not_strand_buffered_frames() {
|
||||
// One chunk, three messages: known → unknown → known.
|
||||
let mut buf = bytes::BytesMut::new();
|
||||
let frame = |event_type: &str, payload: serde_json::Value| {
|
||||
MessageBuilder::new()
|
||||
.header_string(":event-type", event_type)
|
||||
.header_string(":content-type", "application/json")
|
||||
.header_string(":message-type", "event")
|
||||
.payload(Bytes::from(serde_json::to_string(&payload).unwrap()))
|
||||
.build()
|
||||
};
|
||||
buf.extend_from_slice(&frame(
|
||||
"chunk",
|
||||
serde_json::json!({"type": "message_start",
|
||||
"message": {"id": "m1", "usage": {"input_tokens": 3}}}),
|
||||
));
|
||||
buf.extend_from_slice(&frame(
|
||||
"somethingNewFromAws",
|
||||
serde_json::json!({"novel": true}),
|
||||
));
|
||||
buf.extend_from_slice(&frame("chunk", serde_json::json!({"type": "message_stop"})));
|
||||
let body = buf.freeze();
|
||||
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path(
|
||||
"/model/anthropic.claude-3-haiku-20240307-v1:0/invoke-with-response-stream",
|
||||
))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.insert_header("content-type", "application/vnd.amazon.eventstream")
|
||||
.set_body_raw(body.to_vec(), "application/vnd.amazon.eventstream"),
|
||||
)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let proxy = bedrock_proxy(&upstream, |_| {}).await;
|
||||
|
||||
let resp = reqwest::Client::new()
|
||||
.post(format!(
|
||||
"{}/model/anthropic.claude-3-haiku-20240307-v1:0/invoke-with-response-stream",
|
||||
proxy.url()
|
||||
))
|
||||
.header("content-type", "application/json")
|
||||
.header("accept", "text/event-stream")
|
||||
.json(&serde_json::json!({
|
||||
"anthropic_version": "bedrock-2023-05-31",
|
||||
"max_tokens": 8,
|
||||
"messages": [{"role": "user", "content": "hi"}]
|
||||
}))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), 200);
|
||||
let text = resp.text().await.unwrap();
|
||||
assert!(
|
||||
text.contains("message_start"),
|
||||
"frame before the Skip must arrive: {text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains("message_stop"),
|
||||
"frame buffered BEHIND the Skip'd message must still arrive \
|
||||
(pre-fix behaviour dropped it): {text}"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,911 @@
|
||||
//! Integration tests for the native savings stats surface:
|
||||
//! `/stats`, `/stats/timeseries`, `/stats/events`, `/dashboard`,
|
||||
//! and the per-lane recording hooks (deferred SSE finalization,
|
||||
//! non-streaming JSON usage capture, failed-request accounting,
|
||||
//! Bedrock camelCase usage).
|
||||
|
||||
mod common;
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use common::{
|
||||
install_static_token_source, start_proxy_with, start_proxy_with_state, test_credentials,
|
||||
};
|
||||
use headroom_proxy::bedrock::MessageBuilder;
|
||||
use serde_json::{json, Value};
|
||||
use url::Url;
|
||||
use wiremock::matchers::{method, path, path_regex};
|
||||
use wiremock::{Mock, MockServer, ResponseTemplate};
|
||||
|
||||
/// The capture tasks finalize asynchronously after the client sees
|
||||
/// the response — poll /stats until the predicate holds.
|
||||
async fn wait_for_stats<F>(base: &str, pred: F) -> Value
|
||||
where
|
||||
F: Fn(&Value) -> bool,
|
||||
{
|
||||
let client = reqwest::Client::new();
|
||||
for _ in 0..100 {
|
||||
let v: Value = client
|
||||
.get(format!("{base}/stats"))
|
||||
.send()
|
||||
.await
|
||||
.expect("GET /stats")
|
||||
.json()
|
||||
.await
|
||||
.expect("stats json");
|
||||
if pred(&v) {
|
||||
return v;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
}
|
||||
panic!("stats predicate not satisfied within 5s");
|
||||
}
|
||||
|
||||
/// POST a JSON body the way every recording test does.
|
||||
async fn post_json(url: String, body: &Value) -> reqwest::Response {
|
||||
reqwest::Client::new()
|
||||
.post(url)
|
||||
.header("content-type", "application/json")
|
||||
.json(body)
|
||||
.send()
|
||||
.await
|
||||
.expect("POST")
|
||||
}
|
||||
|
||||
/// One binary EventStream frame with the standard event headers.
|
||||
fn event_frame(event_type: &str, payload: &Value) -> bytes::Bytes {
|
||||
MessageBuilder::new()
|
||||
.header_string(":message-type", "event")
|
||||
.header_string(":event-type", event_type)
|
||||
.header_string(":content-type", "application/json")
|
||||
.payload(payload.to_string().into())
|
||||
.build()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stats_endpoints_serve_locally_and_validate_input() {
|
||||
let upstream = MockServer::start().await;
|
||||
let proxy = start_proxy_with(&upstream.uri(), |_| {}).await;
|
||||
let client = reqwest::Client::new();
|
||||
|
||||
let stats: Value = client
|
||||
.get(format!("{}/stats", proxy.url()))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.json()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(stats["proxy"], "headroom-rust");
|
||||
assert_eq!(stats["requests"]["total"], 0);
|
||||
// headway's unified-stats reader contract.
|
||||
assert!(stats["summary"]["api_requests"].is_u64());
|
||||
assert!(stats["tokens"]["proxy_compression_saved"].is_u64());
|
||||
assert!(stats["requests"]["cached"].is_u64());
|
||||
|
||||
let ts: Value = client
|
||||
.get(format!("{}/stats/timeseries?bucket=day", proxy.url()))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.json()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(ts["bucket"], "day");
|
||||
let bad = client
|
||||
.get(format!("{}/stats/timeseries?bucket=fortnight", proxy.url()))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(bad.status(), 400);
|
||||
|
||||
let events: Value = client
|
||||
.get(format!("{}/stats/events", proxy.url()))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.json()
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(events["events"].as_array().unwrap().is_empty());
|
||||
|
||||
let dash = client
|
||||
.get(format!("{}/dashboard", proxy.url()))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(dash.status(), 200);
|
||||
let ct = dash
|
||||
.headers()
|
||||
.get("content-type")
|
||||
.unwrap()
|
||||
.to_str()
|
||||
.unwrap();
|
||||
assert!(ct.starts_with("text/html"), "content-type: {ct}");
|
||||
let body = dash.text().await.unwrap();
|
||||
assert!(body.contains("Headroom Proxy — Savings"));
|
||||
// Self-contained page: no network resource references of any
|
||||
// kind. The only URLs allowed are XML namespaces and data: URIs.
|
||||
for needle in [
|
||||
"src=\"http",
|
||||
"src=\"//",
|
||||
"href=\"http",
|
||||
"href=\"//",
|
||||
"@import",
|
||||
"fonts.googleapis",
|
||||
"cdn.",
|
||||
] {
|
||||
assert!(
|
||||
!body.contains(needle),
|
||||
"dashboard must be self-contained; found {needle:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stats_disabled_falls_through_to_upstream() {
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("GET"))
|
||||
.and(path("/stats"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_string("python-proxy-stats"))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let proxy = start_proxy_with(&upstream.uri(), |c| c.stats = false).await;
|
||||
|
||||
let body = reqwest::Client::new()
|
||||
.get(format!("{}/stats", proxy.url()))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.text()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
body, "python-proxy-stats",
|
||||
"with --stats=false the path must tunnel upstream (pre-feature behaviour)"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn anthropic_non_streaming_response_records_usage() {
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/messages"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
|
||||
"id": "msg_1",
|
||||
"type": "message",
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"content": [{"type": "text", "text": "hi"}],
|
||||
"usage": {
|
||||
"input_tokens": 100,
|
||||
"output_tokens": 25,
|
||||
"cache_read_input_tokens": 40,
|
||||
"cache_creation_input_tokens": 10
|
||||
}
|
||||
})))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
// compression=true engages the buffered arm (mode stays Off —
|
||||
// recording must work in pure-passthrough deployments too).
|
||||
let proxy = start_proxy_with(&upstream.uri(), |c| c.compression = true).await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/v1/messages", proxy.url()),
|
||||
&json!({
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"max_tokens": 32,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap(); // drain so the capture tee closes
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(stats["tokens"]["input"], 100);
|
||||
assert_eq!(stats["tokens"]["output"], 25);
|
||||
assert_eq!(stats["tokens"]["cache_read"], 40);
|
||||
assert_eq!(stats["tokens"]["cache_write"], 10);
|
||||
assert_eq!(stats["requests"]["failed"], 0);
|
||||
// 40 cache-read tokens ⇒ this request counts as a cache hit.
|
||||
assert_eq!(stats["requests"]["cached"], 1);
|
||||
assert_eq!(stats["session_by_provider"]["anthropic"]["requests"], 1);
|
||||
// 40 cache reads on sonnet pricing → non-zero cache savings.
|
||||
assert!(stats["session"]["cache_savings_usd"].as_f64().unwrap() > 0.0);
|
||||
assert!(stats["session"]["input_cost_usd"].as_f64().unwrap() > 0.0);
|
||||
// Model attribution landed.
|
||||
assert_eq!(
|
||||
stats["lifetime_by_model"]["claude-sonnet-4-5-20250929"]["requests"],
|
||||
1
|
||||
);
|
||||
// Timeseries picked it up.
|
||||
let ts: Value = reqwest::Client::new()
|
||||
.get(format!("{}/stats/timeseries?bucket=hour", proxy.url()))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.json()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(ts["points"][0]["requests"], 1);
|
||||
}
|
||||
|
||||
/// Recording must not depend on the compression master switch —
|
||||
/// it is OFF by default, and spend/cache observability is the whole
|
||||
/// point of the feature on a passthrough deployment. The model is
|
||||
/// learned from the response there (the request body is never
|
||||
/// buffered), and compression savings are legitimately 0.
|
||||
#[tokio::test]
|
||||
async fn records_with_compression_off_using_model_from_response() {
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/messages"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
|
||||
"id": "msg_1",
|
||||
"type": "message",
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"content": [{"type": "text", "text": "hi"}],
|
||||
"usage": {"input_tokens": 200, "output_tokens": 10,
|
||||
"cache_read_input_tokens": 80}
|
||||
})))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
// Config::for_test defaults compression to false — same as the
|
||||
// shipped binary's default.
|
||||
let proxy = start_proxy_with(&upstream.uri(), |c| {
|
||||
assert!(!c.compression, "guard: this test covers compression OFF");
|
||||
})
|
||||
.await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/v1/messages", proxy.url()),
|
||||
&json!({
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"max_tokens": 32,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(stats["tokens"]["input"], 200);
|
||||
assert_eq!(stats["tokens"]["cache_read"], 80);
|
||||
assert_eq!(stats["tokens"]["saved"], 0, "no compression ran → 0 saved");
|
||||
// Model came from the response envelope, so spend is priced.
|
||||
assert_eq!(
|
||||
stats["lifetime_by_model"]["claude-sonnet-4-5-20250929"]["requests"], 1,
|
||||
"model must be learned from the response when the request isn't buffered"
|
||||
);
|
||||
assert!(stats["session"]["input_cost_usd"].as_f64().unwrap() > 0.0);
|
||||
assert!(stats["session"]["cache_savings_usd"].as_f64().unwrap() > 0.0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn anthropic_sse_stream_records_deferred_usage() {
|
||||
let upstream = MockServer::start().await;
|
||||
let sse_body = concat!(
|
||||
"event: message_start\n",
|
||||
"data: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_1\",\"usage\":",
|
||||
"{\"input_tokens\":50,\"output_tokens\":1,\"cache_read_input_tokens\":12}}}\n\n",
|
||||
"event: content_block_delta\n",
|
||||
"data: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":",
|
||||
"{\"type\":\"text_delta\",\"text\":\"hey\"}}\n\n",
|
||||
"event: message_delta\n",
|
||||
"data: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\"},",
|
||||
"\"usage\":{\"output_tokens\":30}}\n\n",
|
||||
"event: message_stop\n",
|
||||
"data: {\"type\":\"message_stop\"}\n\n",
|
||||
);
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/messages"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.insert_header("content-type", "text/event-stream")
|
||||
.set_body_raw(sse_body.as_bytes().to_vec(), "text/event-stream"),
|
||||
)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let proxy = start_proxy_with(&upstream.uri(), |c| c.compression = true).await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/v1/messages", proxy.url()),
|
||||
&json!({
|
||||
"model": "claude-haiku-4-5-20251001",
|
||||
"max_tokens": 32,
|
||||
"stream": true,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
// Deferred recording: counts come from the stream's usage
|
||||
// frames, observed only after the stream closed.
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(stats["tokens"]["input"], 50);
|
||||
assert_eq!(
|
||||
stats["tokens"]["output"], 30,
|
||||
"output from message_delta, not 0"
|
||||
);
|
||||
assert_eq!(stats["tokens"]["cache_read"], 12);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn failed_upstream_counts_but_accrues_no_savings() {
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/messages"))
|
||||
.respond_with(ResponseTemplate::new(529).set_body_json(json!({
|
||||
"type": "error",
|
||||
"error": {"type": "overloaded_error", "message": "overloaded"}
|
||||
})))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let proxy = start_proxy_with(&upstream.uri(), |c| c.compression = true).await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/v1/messages", proxy.url()),
|
||||
&json!({
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"max_tokens": 32,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 529);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(stats["requests"]["failed"], 1);
|
||||
assert_eq!(stats["tokens"]["saved"], 0);
|
||||
assert_eq!(stats["session"]["savings_usd"], 0.0);
|
||||
assert_eq!(stats["session"]["input_cost_usd"], 0.0);
|
||||
// Visible in the feed, flagged.
|
||||
let events: Value = reqwest::Client::new()
|
||||
.get(format!("{}/stats/events?limit=5", proxy.url()))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.json()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(events["events"][0]["failed"], true);
|
||||
assert_eq!(events["events"][0]["tokens_saved"], 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn openai_chat_response_normalises_cached_tokens() {
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/chat/completions"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
|
||||
"id": "chatcmpl-1",
|
||||
"object": "chat.completion",
|
||||
"model": "gpt-4o",
|
||||
"choices": [{"index": 0, "message": {"role": "assistant", "content": "hi"},
|
||||
"finish_reason": "stop"}],
|
||||
"usage": {
|
||||
"prompt_tokens": 90,
|
||||
"completion_tokens": 12,
|
||||
"prompt_tokens_details": {"cached_tokens": 60}
|
||||
}
|
||||
})))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let proxy = start_proxy_with(&upstream.uri(), |c| c.compression = true).await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/v1/chat/completions", proxy.url()),
|
||||
&json!({
|
||||
"model": "gpt-4o",
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
// prompt_tokens INCLUDES cached — the ledger stores the split.
|
||||
assert_eq!(stats["tokens"]["input"], 30);
|
||||
assert_eq!(stats["tokens"]["cache_read"], 60);
|
||||
assert_eq!(stats["tokens"]["output"], 12);
|
||||
// Provider label matches the Prometheus vocabulary
|
||||
// (`metric_names`): the chat lane is `openai_chat`, not `openai`.
|
||||
assert_eq!(stats["session_by_provider"]["openai_chat"]["requests"], 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn bedrock_converse_sync_records_camelcase_usage() {
|
||||
let upstream = MockServer::start().await;
|
||||
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
|
||||
Mock::given(method("POST"))
|
||||
.and(path(format!("/model/{model_id}/converse")))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
|
||||
"output": {"message": {"role": "assistant",
|
||||
"content": [{"text": "hello"}]}},
|
||||
"stopReason": "end_turn",
|
||||
"usage": {"inputTokens": 9, "outputTokens": 4,
|
||||
"cacheReadInputTokens": 2, "cacheWriteInputTokens": 1}
|
||||
})))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
|
||||
let endpoint: Url = upstream.uri().parse().unwrap();
|
||||
let proxy = start_proxy_with_state(
|
||||
&upstream.uri(),
|
||||
|c| c.bedrock_endpoint = Some(endpoint),
|
||||
|s| s.with_bedrock_credentials(test_credentials()),
|
||||
)
|
||||
.await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/model/{model_id}/converse", proxy.url()),
|
||||
&json!({
|
||||
"messages": [{"role": "user", "content": [{"text": "hello"}]}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(stats["session_by_provider"]["bedrock"]["requests"], 1);
|
||||
assert_eq!(stats["tokens"]["input"], 9);
|
||||
assert_eq!(stats["tokens"]["output"], 4);
|
||||
assert_eq!(stats["tokens"]["cache_read"], 2);
|
||||
assert_eq!(stats["tokens"]["cache_write"], 1);
|
||||
// Bedrock model ids price via the vendored table (cache reads
|
||||
// discounted vs list input price).
|
||||
assert!(stats["session"]["cache_savings_usd"].as_f64().unwrap() > 0.0);
|
||||
assert_eq!(stats["lifetime_by_model"][model_id]["requests"], 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn bedrock_invoke_sync_records_snake_case_usage() {
|
||||
let upstream = MockServer::start().await;
|
||||
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
|
||||
Mock::given(method("POST"))
|
||||
.and(path(format!("/model/{model_id}/invoke")))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
|
||||
"id": "msg_1", "type": "message", "role": "assistant",
|
||||
"model": "claude-3-5-haiku-20241022",
|
||||
"content": [{"type": "text", "text": "ok"}],
|
||||
"usage": {"input_tokens": 12, "output_tokens": 4}
|
||||
})))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let endpoint: Url = upstream.uri().parse().unwrap();
|
||||
let proxy = start_proxy_with_state(
|
||||
&upstream.uri(),
|
||||
|c| c.bedrock_endpoint = Some(endpoint),
|
||||
|s| s.with_bedrock_credentials(test_credentials()),
|
||||
)
|
||||
.await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/model/{model_id}/invoke", proxy.url()),
|
||||
&json!({
|
||||
"anthropic_version": "bedrock-2023-05-31",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(stats["tokens"]["input"], 12);
|
||||
assert_eq!(stats["tokens"]["output"], 4);
|
||||
assert_eq!(stats["session_by_provider"]["bedrock"]["requests"], 1);
|
||||
// The request-side model id (with region/revision) wins over the
|
||||
// response echo's bare id.
|
||||
assert_eq!(stats["lifetime_by_model"][model_id]["requests"], 1);
|
||||
}
|
||||
|
||||
/// The passthrough lane nothing else parses: binary EventStream
|
||||
/// (`Accept: application/vnd.amazon.eventstream`), with InvokeModel's
|
||||
/// base64-wrapped Anthropic events. Usage must come out of the tee's
|
||||
/// telemetry parser while the client receives byte-equal frames.
|
||||
#[tokio::test]
|
||||
async fn bedrock_eventstream_passthrough_records_usage() {
|
||||
use base64::Engine as _;
|
||||
let b64 = |v: &Value| base64::engine::general_purpose::STANDARD.encode(v.to_string());
|
||||
let frame = |inner: &Value| event_frame("chunk", &json!({"bytes": b64(inner)}));
|
||||
let mut body_bytes: Vec<u8> = Vec::new();
|
||||
for inner in [
|
||||
json!({"type": "message_start",
|
||||
"message": {"id": "msg_1", "usage": {"input_tokens": 21, "output_tokens": 1,
|
||||
"cache_read_input_tokens": 7}}}),
|
||||
json!({"type": "content_block_delta", "index": 0,
|
||||
"delta": {"type": "text_delta", "text": "ok"}}),
|
||||
json!({"type": "message_delta", "delta": {"stop_reason": "end_turn"},
|
||||
"usage": {"output_tokens": 9}}),
|
||||
json!({"type": "message_stop"}),
|
||||
] {
|
||||
body_bytes.extend_from_slice(&frame(&inner));
|
||||
}
|
||||
|
||||
let upstream = MockServer::start().await;
|
||||
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
|
||||
Mock::given(method("POST"))
|
||||
.and(path(format!(
|
||||
"/model/{model_id}/invoke-with-response-stream"
|
||||
)))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.insert_header("content-type", "application/vnd.amazon.eventstream")
|
||||
// Cloned: the assertion below compares the client's
|
||||
// bytes against these exact frames.
|
||||
.set_body_raw(body_bytes.clone(), "application/vnd.amazon.eventstream"),
|
||||
)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let endpoint: Url = upstream.uri().parse().unwrap();
|
||||
let proxy = start_proxy_with_state(
|
||||
&upstream.uri(),
|
||||
|c| c.bedrock_endpoint = Some(endpoint),
|
||||
|s| s.with_bedrock_credentials(test_credentials()),
|
||||
)
|
||||
.await;
|
||||
|
||||
let resp = reqwest::Client::new()
|
||||
.post(format!(
|
||||
"{}/model/{model_id}/invoke-with-response-stream",
|
||||
proxy.url()
|
||||
))
|
||||
.header("content-type", "application/json")
|
||||
// EventStream passthrough mode — the client wants raw frames.
|
||||
.header("accept", "application/vnd.amazon.eventstream")
|
||||
.json(&json!({
|
||||
"anthropic_version": "bedrock-2023-05-31",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), 200);
|
||||
let echoed = resp.bytes().await.unwrap();
|
||||
// The whole claim of this lane is that telemetry rides alongside
|
||||
// the bytes without touching them. Assert that literally: the
|
||||
// client's stream must be byte-identical to what the upstream
|
||||
// sent, tee or no tee. `!is_empty()` would pass a mangled body.
|
||||
assert_eq!(
|
||||
echoed.as_ref(),
|
||||
body_bytes.as_slice(),
|
||||
"EventStream passthrough must be byte-identical through the tee"
|
||||
);
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(
|
||||
stats["tokens"]["input"], 21,
|
||||
"from base64-wrapped message_start"
|
||||
);
|
||||
assert_eq!(
|
||||
stats["tokens"]["output"], 9,
|
||||
"monotone max from message_delta"
|
||||
);
|
||||
assert_eq!(stats["tokens"]["cache_read"], 7);
|
||||
assert_eq!(stats["requests"]["cached"], 1);
|
||||
}
|
||||
|
||||
/// Converse-stream in SSE-translation mode: usage arrives ONLY in the
|
||||
/// camelCase `metadata` frame, which carries no Anthropic `type` and
|
||||
/// is dropped by the Anthropic state machine — the fallback scan must
|
||||
/// pick it up.
|
||||
#[tokio::test]
|
||||
async fn bedrock_converse_stream_sse_records_camel_metadata_usage() {
|
||||
let mut body_bytes: Vec<u8> = Vec::new();
|
||||
for (et, payload) in [
|
||||
("messageStart", json!({"role": "assistant"})),
|
||||
(
|
||||
"contentBlockDelta",
|
||||
json!({"contentBlockIndex": 0, "delta": {"text": "ok"}}),
|
||||
),
|
||||
("messageStop", json!({"stopReason": "end_turn"})),
|
||||
(
|
||||
"metadata",
|
||||
json!({"usage": {"inputTokens": 15, "outputTokens": 6,
|
||||
"cacheWriteInputTokens": 11},
|
||||
"metrics": {"latencyMs": 250}}),
|
||||
),
|
||||
] {
|
||||
body_bytes.extend_from_slice(&event_frame(et, &payload));
|
||||
}
|
||||
|
||||
let upstream = MockServer::start().await;
|
||||
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
|
||||
Mock::given(method("POST"))
|
||||
.and(path(format!("/model/{model_id}/converse-stream")))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.insert_header("content-type", "application/vnd.amazon.eventstream")
|
||||
.set_body_raw(body_bytes, "application/vnd.amazon.eventstream"),
|
||||
)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let endpoint: Url = upstream.uri().parse().unwrap();
|
||||
let proxy = start_proxy_with_state(
|
||||
&upstream.uri(),
|
||||
|c| c.bedrock_endpoint = Some(endpoint),
|
||||
|s| s.with_bedrock_credentials(test_credentials()),
|
||||
)
|
||||
.await;
|
||||
|
||||
let resp = reqwest::Client::new()
|
||||
.post(format!("{}/model/{model_id}/converse-stream", proxy.url()))
|
||||
.header("content-type", "application/json")
|
||||
.header("accept", "text/event-stream") // SSE translation mode
|
||||
.json(&json!({
|
||||
"messages": [{"role": "user", "content": [{"text": "hello"}]}]
|
||||
}))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(
|
||||
stats["tokens"]["input"], 15,
|
||||
"camelCase metadata usage captured"
|
||||
);
|
||||
assert_eq!(stats["tokens"]["output"], 6);
|
||||
assert_eq!(stats["tokens"]["cache_write"], 11);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn vertex_raw_predict_records_usage() {
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path_regex(
|
||||
r"^/v1beta1/projects/[^/]+/locations/[^/]+/publishers/anthropic/models/.+:rawPredict$",
|
||||
))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
|
||||
"id": "msg_1", "type": "message", "role": "assistant",
|
||||
"model": "claude-sonnet-4-5",
|
||||
"content": [{"type": "text", "text": "ok"}],
|
||||
"usage": {"input_tokens": 33, "output_tokens": 8,
|
||||
"cache_read_input_tokens": 5}
|
||||
})))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let proxy = start_proxy_with_state(
|
||||
&upstream.uri(),
|
||||
|_| {},
|
||||
|s| install_static_token_source(s, "ya29.stats-test-bearer"),
|
||||
)
|
||||
.await;
|
||||
|
||||
let resp = reqwest::Client::new()
|
||||
.post(format!(
|
||||
"{}/v1beta1/projects/p1/locations/us-central1/publishers/anthropic/models/claude-sonnet-4-5@20250929:rawPredict",
|
||||
proxy.url()
|
||||
))
|
||||
.header("content-type", "application/json")
|
||||
.json(&json!({
|
||||
"anthropic_version": "vertex-2023-10-16",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}))
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(stats["session_by_provider"]["vertex"]["requests"], 1);
|
||||
assert_eq!(stats["tokens"]["input"], 33);
|
||||
assert_eq!(stats["tokens"]["output"], 8);
|
||||
assert_eq!(stats["tokens"]["cache_read"], 5);
|
||||
}
|
||||
|
||||
/// A proxy-side rejection (here: no AWS credentials configured) is an
|
||||
/// outcome too — it must show in /stats as a failure, not vanish. An
|
||||
/// operator-side credentials outage reading as "zero traffic" would
|
||||
/// hide exactly the incident this dashboard exists to surface.
|
||||
#[tokio::test]
|
||||
async fn bedrock_proxy_side_rejection_records_failure() {
|
||||
let upstream = MockServer::start().await;
|
||||
let model_id = "anthropic.claude-3-5-haiku-20241022-v1:0";
|
||||
let endpoint: Url = upstream.uri().parse().unwrap();
|
||||
// No .with_bedrock_credentials(...) → handler must 5xx before
|
||||
// any upstream attempt.
|
||||
let proxy = start_proxy_with(&upstream.uri(), |c| c.bedrock_endpoint = Some(endpoint)).await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/model/{model_id}/invoke", proxy.url()),
|
||||
&json!({
|
||||
"anthropic_version": "bedrock-2023-05-31",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 500);
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(stats["requests"]["failed"], 1);
|
||||
assert_eq!(stats["tokens"]["input"], 0);
|
||||
assert_eq!(stats["session"]["savings_usd"], 0.0);
|
||||
assert_eq!(stats["session_by_provider"]["bedrock"]["failed"], 1);
|
||||
}
|
||||
|
||||
/// Persistence through the real config surface: proxy A (started with
|
||||
/// a stats path) records and flushes; proxy B on the same path serves
|
||||
/// the lifetime numbers proxy A earned.
|
||||
#[tokio::test]
|
||||
async fn stats_path_survives_across_proxy_instances() {
|
||||
let dir = std::env::temp_dir().join(format!("stats-restart-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
let stats_path = dir.join("native_stats.json");
|
||||
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/messages"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
|
||||
"id": "msg_1", "type": "message",
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"content": [{"type": "text", "text": "hi"}],
|
||||
"usage": {"input_tokens": 60, "output_tokens": 6}
|
||||
})))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
|
||||
// Proxy A: record one request, then flush via the ledger handle
|
||||
// (the background flusher + shutdown flush live in main.rs, which
|
||||
// this in-process harness deliberately doesn't run).
|
||||
let ledger_slot: std::sync::Arc<
|
||||
std::sync::Mutex<Option<std::sync::Arc<headroom_proxy::observability::ledger::Ledger>>>,
|
||||
> = Default::default();
|
||||
let slot = ledger_slot.clone();
|
||||
let sp = stats_path.clone();
|
||||
let proxy_a = start_proxy_with_state(
|
||||
&upstream.uri(),
|
||||
move |c| c.stats_path = Some(sp),
|
||||
move |s| {
|
||||
*slot.lock().unwrap() = Some(s.stats.clone());
|
||||
s
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let resp = post_json(
|
||||
format!("{}/v1/messages", proxy_a.url()),
|
||||
&json!({
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
wait_for_stats(&proxy_a.url(), |v| v["requests"]["total"] == 1).await;
|
||||
let ledger = ledger_slot.lock().unwrap().take().unwrap();
|
||||
ledger.flush().await;
|
||||
proxy_a.shutdown().await;
|
||||
assert!(stats_path.exists(), "flush persisted the ledger");
|
||||
|
||||
// Proxy B on the same path: lifetime carries over, session fresh.
|
||||
let sp = stats_path.clone();
|
||||
let proxy_b = start_proxy_with(&upstream.uri(), move |c| c.stats_path = Some(sp)).await;
|
||||
let stats: Value = reqwest::Client::new()
|
||||
.get(format!("{}/stats", proxy_b.url()))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.json()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
stats["lifetime"]["requests"], 1,
|
||||
"lifetime survived restart"
|
||||
);
|
||||
assert_eq!(stats["lifetime"]["input_tokens"], 60);
|
||||
assert_eq!(stats["session"]["requests"], 0, "session resets per boot");
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
/// A 200 OK SSE stream that fails IN-BAND (`{"type":"error"}` mid-
|
||||
/// stream — e.g. Anthropic overloading during generation) is not a
|
||||
/// billable success. The state machine lands in `Errored`; the
|
||||
/// ledger must count the request as failed and accrue nothing.
|
||||
#[tokio::test]
|
||||
async fn in_stream_error_event_records_failure() {
|
||||
let sse_body = concat!(
|
||||
"event: message_start\n",
|
||||
"data: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_1\",\"usage\":",
|
||||
"{\"input_tokens\":50,\"output_tokens\":1}}}\n\n",
|
||||
"event: error\n",
|
||||
"data: {\"type\":\"error\",\"error\":{\"type\":\"overloaded_error\",",
|
||||
"\"message\":\"Overloaded\"}}\n\n",
|
||||
);
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/messages"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.insert_header("content-type", "text/event-stream")
|
||||
.set_body_raw(sse_body.as_bytes().to_vec(), "text/event-stream"),
|
||||
)
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let proxy = start_proxy_with(&upstream.uri(), |_| {}).await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/v1/messages", proxy.url()),
|
||||
&json!({
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"max_tokens": 32,
|
||||
"stream": true,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 200, "the HTTP status was a success");
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
assert_eq!(
|
||||
stats["requests"]["failed"], 1,
|
||||
"in-band stream error must count as a failed request"
|
||||
);
|
||||
assert_eq!(
|
||||
stats["tokens"]["input"], 0,
|
||||
"failed requests accrue nothing"
|
||||
);
|
||||
assert_eq!(stats["session"]["input_cost_usd"], 0.0);
|
||||
}
|
||||
|
||||
/// A hostile (or buggy) upstream reporting absurd token counts must
|
||||
/// not poison the lifetime aggregates: the ledger clamps every
|
||||
/// per-event token field to its 50M ceiling before pricing.
|
||||
#[tokio::test]
|
||||
async fn hostile_usage_counts_are_clamped_before_recording() {
|
||||
let upstream = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/messages"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
|
||||
"id": "msg_1",
|
||||
"type": "message",
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"content": [{"type": "text", "text": "hi"}],
|
||||
"usage": {
|
||||
"input_tokens": u64::MAX,
|
||||
"output_tokens": 9_000_000_000_000u64,
|
||||
"cache_read_input_tokens": 0,
|
||||
"cache_creation_input_tokens": 0
|
||||
}
|
||||
})))
|
||||
.mount(&upstream)
|
||||
.await;
|
||||
let proxy = start_proxy_with(&upstream.uri(), |_| {}).await;
|
||||
|
||||
let resp = post_json(
|
||||
format!("{}/v1/messages", proxy.url()),
|
||||
&json!({
|
||||
"model": "claude-sonnet-4-5-20250929",
|
||||
"max_tokens": 32,
|
||||
"messages": [{"role": "user", "content": "hello"}]
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status(), 200);
|
||||
let _ = resp.bytes().await.unwrap();
|
||||
|
||||
let stats = wait_for_stats(&proxy.url(), |v| v["requests"]["total"] == 1).await;
|
||||
const CEILING: u64 = 50_000_000;
|
||||
assert_eq!(stats["tokens"]["input"], CEILING, "input clamped");
|
||||
assert_eq!(stats["tokens"]["output"], CEILING, "output clamped");
|
||||
let cost = stats["session"]["input_cost_usd"].as_f64().unwrap();
|
||||
assert!(
|
||||
cost.is_finite() && cost < 1_000.0,
|
||||
"clamped pricing stays sane: {cost}"
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user