fix(metrics): split rate-limited by source and label failures by provider (#3707)

Closes #3696. #3615 moved the outcome-funnel short-circuit from `>= 500`
to `>= 400` — correct, and staying — but it silently changed what two
metrics mean and the docs were never updated.

**⚠️ This changes exported series. See Migration.**

## What was wrong

- `headroom_requests_failed_total` was documented as "Upstream 5xx
errors"; it now counts 4xx too.
- `headroom_requests_rate_limited_total` was documented as "Requests
**Headroom** rejected via its own rate limiter (**not upstream 429s**)"
— #3615 inverted exactly that.
- 4xx also dropped out of `headroom_requests_by_provider` (only
`record_request` touches it), leaving Prometheus with no way to tell
which upstream was failing.

## Changes


**`headroom_requests_rate_limited_total{source="headroom"|"upstream"}`**
— two 429s mean opposite things to an operator: `headroom` is our
limiter refusing (raise the cap), `upstream` is the provider refusing
(back off / shard keys). Closed label set, so cardinality stays bounded;
unknown values clamp to `headroom`, the counter's pre-#3615 meaning, so
an un-updated caller cannot invent a label. Threaded at all five call
sites (`openai.py:3554, 5681`, `anthropic.py:1256`, `gemini.py:405` →
`headroom`; `outcome.py:543`, the #3615 funnel → `upstream`).

**`headroom_requests_failed_total{provider}`** — uses
`outcome.provider`, already Copilot-relabelled by the funnel, so no
second convention. Matches the existing
`upstream_connection_errors_by_provider` shape.

**All four backends**, so no sink reports a differently-shaped counter:
Prometheus, persistent/lifetime
(`SavingsTracker.record_lifetime_rate_limited`), OTel (attribute on
`headroom.proxy.requests.rate_limited`), and `/stats` JSON.
`SCHEMA_VERSION` stays 5 — the new maps are purely additive, and old
state files load with them empty rather than back-filling invented
history.

**Docs**: `docs/metrics-technical-guide.md`,
`docs/content/docs/metrics.mdx`, `wiki/metrics.md`, plus tooltips on the
dashboard's Failed / Rate Limited tiles so the UI stops implying "Rate
Limited" means Headroom only. `requests_total` is now documented as a
completed-request denominator with `inbound_requests_total` named as the
all-traffic counter, and the failure-rate PromQL carries a "why" on its
denominator so nobody "simplifies" it. Checked and found clean:
`examples/grafana/headroom-dashboard.json`, `docs/observability.md`,
`docs/content/docs/proxy.mdx`, `wiki/proxy.md`, `README.md`.

## Second commit: a bug found while reviewing the first

Adding the label turned `headroom_requests_failed_total` from a counter
that **always** exported `0` into one that exports **no sample at all**
on a healthy proxy — confirmed by rendering `/metrics` on a fresh
instance:

```
headroom_requests_failed_total:       0 sample(s)
headroom_requests_rate_limited_total: 2 sample(s)
```

That breaks the failure-rate query documented in this very PR: an empty
numerator empties the whole PromQL expression, so the panel reads "No
data" instead of 0% while everything is fine, and `absent()` alerts
would fire on healthy proxies. The rate-limit series avoided this by
seeding its closed label set; providers are open-ended so the same trick
does not apply.

Fix: seed the `unknown` bucket at zero. It is a real bucket —
`record_failed` uses it for unattributed failures — so this invents no
provider. Guarded by a new test.

## Migration

Both counters now export **only** labelled series; an unlabelled sample
alongside would double-count under `sum()`. Unaggregated queries return
multiple series — `sum without (source)` / `sum without (provider)`
reproduces the previous value exactly. Noted in the docs and an mdx
Callout.

## Verification

- **16 tests fail on pre-change source** (verified by reverting only
`headroom/`), pass after.
- Targeted metrics suites → **58 passed**. Broad sweep `-k "metric or
stats or outcome or rate_limit or savings_tracker or persistent"` → 634
passed / 44 skipped. Handler, dashboard and savings-history suites → 90
+ 79 passed.
- Ruff 0.16.7 (the CI pin, via `scripts/verify-ruff-version.py
--print-version`): clean.

`CHANGELOG.md` intentionally untouched — `changelog-guard.yml` fails any
PR that edits it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-09-22 07:26:40 -07:00
committed by GitHub
co-authored by Claude Opus 5
parent 94206e2652
commit 5ff4ea1ef9
18 changed files with 573 additions and 43 deletions
+32 -2
View File
@@ -25,7 +25,9 @@ curl http://localhost:8787/stats
"total": 42,
"cached": 5,
"rate_limited": 0,
"failed": 0
"rate_limited_by_source": { "headroom": 0, "upstream": 0 },
"failed": 0,
"failed_by_provider": {}
},
"tokens": {
"input": 50000,
@@ -44,6 +46,14 @@ curl http://localhost:8787/stats
}
```
`requests.total` is the **completed**-request count, not total traffic: a request that
ended in a 4xx, 5xx or 429 is counted under `requests.failed` or `requests.rate_limited`
instead and never reaches `total`. `rate_limited_by_source` splits 429s into
`headroom` (Headroom's own rate limiter refused the request) and `upstream` (the provider
refused it) — different problems, different fixes. `failed_by_provider` attributes failures
to the upstream that produced them. The unlabelled `rate_limited` / `failed` totals are
unchanged.
Persistent savings are stored at `~/.headroom/proxy_savings.json` and survive proxy restarts. Override the path with `HEADROOM_SAVINGS_PATH`.
### Historical Savings
@@ -82,8 +92,27 @@ headroom_latency_ms_count 1234
# HELP headroom_provider_cache_hit_requests_total Requests that read from the provider's prompt cache
headroom_provider_cache_hit_requests_total{provider="anthropic"} 456
# HELP headroom_requests_failed_total Requests that failed upstream (4xx and 5xx, excluding 429), by provider
headroom_requests_failed_total{provider="anthropic"} 3
# HELP headroom_requests_rate_limited_total Requests rejected with 429, by who rejected them (headroom=our own limiter, upstream=the provider)
headroom_requests_rate_limited_total{source="headroom"} 0
headroom_requests_rate_limited_total{source="upstream"} 12
```
<Callout type="warn" title="requests_total is completed requests, not all traffic">
`headroom_requests_total` counts requests that **completed**. A request that ended in a 4xx,
5xx or 429 lands in `headroom_requests_failed_total` / `headroom_requests_rate_limited_total`
and is not in `requests_total` — so a failure rate must add `failed` back into its own
denominator. `headroom_inbound_requests_total` is the all-traffic counter.
`headroom_requests_failed_total` gained a `provider` label and
`headroom_requests_rate_limited_total` a `source` label, so unaggregated queries on them now
return several series. `sum without (provider) (...)` and `sum without (source) (...)`
reproduce the previous single-series behaviour.
</Callout>
<Callout type="warn" title="No percentile buckets on /metrics">
There are no Prometheus histogram buckets for compression ratio or latency — build percentiles or averages from `_sum` / `_count` pairs (`headroom_latency_ms_sum`, `headroom_latency_ms_count`, and so on), or use the `headroom perf` CLI for real p95/p99.
</Callout>
@@ -351,7 +380,8 @@ Env: `HEADROOM_BUDGET_ESTIMATED_BASIS`. `headroom doctor` reports the estimated
| `headroom_persistent_savings_tokens_saved_total` | Durable lifetime tokens saved from `/stats.persistent_savings` | Higher is better |
| `headroom_overhead_ms_sum` / `headroom_overhead_ms_count` | Mean latency Headroom itself adds (there are no percentile buckets — see the callout above) | Low is better |
| `headroom_provider_cache_hit_requests_total` / `headroom_provider_cache_requests_total` | Cache effectiveness, by provider | >20% is good |
| `headroom_requests_failed_total` | Reliability (upstream 5xx errors) | 0 |
| `headroom_requests_failed_total{provider}` | Reliability — upstream **4xx and 5xx** (429s are counted separately), attributed to the provider that failed | 0 |
| `headroom_requests_rate_limited_total{source}` | Throttling — `source="headroom"` is Headroom's own rate limiter, `source="upstream"` is the provider's | 0 |
## Grafana Dashboard
+56 -6
View File
@@ -103,21 +103,55 @@ rate(headroom_cache_bust_total[5m])
| Metric | What it shows |
|---|---|
| `headroom_requests_total` | Requests handled. Unlabelled. |
| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… |
| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. |
| `headroom_requests_failed_total` | Upstream 5xx errors. |
| `headroom_requests_rate_limited_total` | Requests **Headroom** rejected via its own rate limiter (not upstream 429s). |
| `headroom_requests_total` | **Completed** requests — the denominator for success-side stats, not total traffic. A request that ended in a 4xx, 5xx or 429 never reaches it. Unlabelled. |
| `headroom_inbound_requests_total` | Every inbound HTTP request the proxy accepted, whatever the outcome. This is the all-traffic counter. |
| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… **Completed requests only**, same denominator as `headroom_requests_total`. Use `headroom_requests_failed_total{provider}` for the failure side. |
| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. Completed requests only. |
| `headroom_requests_failed_total{provider}` | Requests that failed upstream — **4xx and 5xx**, excluding 429. Excluded from `headroom_requests_total`, so add it back when you compute a rate. |
| `headroom_requests_rate_limited_total{source}` | Requests rejected with 429, by **either** Headroom's own rate limiter (`source="headroom"`) or the upstream provider (`source="upstream"`). Excluded from `headroom_requests_total`. |
| `headroom_compression_failed_total{reason}` | Compression failures — `timeout` or `error`. Fails open, so traffic keeps flowing but savings quietly stop. **Worth an alert.** |
| `headroom_compression_quarantine_total{event}` | Compression disabled after repeated timeouts — `activated`, `skipped`, `released`. |
| `headroom_inbound_requests_active` | In-flight requests, gauge. Counts all HTTP including `/metrics`. |
| `headroom_active_ws_sessions` | Live Codex WebSocket sessions, gauge. |
### `headroom_requests_total` is a denominator, not traffic
`headroom_requests_total` counts requests that **completed** — it has always been
incremented only on the success path, and since the outcome funnel started short-circuiting
at `>= 400`, every 4xx, 5xx and 429 drops out of it. So:
- `headroom_requests_total` + `headroom_requests_failed_total` + `headroom_requests_rate_limited_total`
≈ the requests the proxy forwarded upstream.
- `headroom_inbound_requests_total` is the honest all-traffic counter (it also counts
`/metrics`, `/stats` and other non-proxy HTTP).
That is why the failure-rate query below adds `failed` back into its own denominator, and
why you should not "simplify" it to `… / rate(headroom_requests_total[5m])`: failures are
not in `requests_total`, so that form divides failures by successes and over-reports the
rate — badly, during exactly the incident you wrote it for (at 100% failures the denominator
goes to zero).
```promql
# Failure rate
# Failure rate. The denominator deliberately re-adds `failed`: a failed request
# is NOT in headroom_requests_total, so requests_total alone is successes-only.
rate(headroom_requests_failed_total[5m])
/ clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)
# Failure rate by provider — which upstream is actually broken.
sum by (provider) (rate(headroom_requests_failed_total[5m]))
/ clamp_min(
sum by (provider) (rate(headroom_requests_by_provider[5m]))
+ sum by (provider) (rate(headroom_requests_failed_total[5m])),
1
)
# Who is throttling you: your own limiter (raise the cap) vs the provider
# (back off, shard keys). Alert on these separately — they are different actions.
sum by (source) (rate(headroom_requests_rate_limited_total[5m]))
# Provider throttling only — the one that means "slow down".
rate(headroom_requests_rate_limited_total{source="upstream"}[5m])
# Savings silently stopped
sum by (reason) (rate(headroom_compression_failed_total[5m]))
@@ -125,6 +159,22 @@ sum by (reason) (rate(headroom_compression_failed_total[5m]))
sum by (provider) (rate(headroom_requests_by_provider[5m]))
```
> **Migration — these two counters gained labels.** `headroom_requests_failed_total` is now
> labelled by `provider` and `headroom_requests_rate_limited_total` by `source`, so an
> unaggregated query that used to return one series now returns several, and a panel or
> alert that graphed the bare counter will fan out into one line per label value.
> `sum without (provider) (rate(headroom_requests_failed_total[5m]))` and
> `sum without (source) (rate(headroom_requests_rate_limited_total[5m]))` reproduce the old
> single-series behaviour exactly. Both `source` series are exported from process start
> (including at zero); `failed` exports one series per provider that has actually failed, so
> it has no series at all until the first failure.
>
> The same split is available outside Prometheus: `/stats` carries
> `requests.rate_limited_by_source` and `requests.failed_by_provider` alongside the
> unlabelled `requests.rate_limited` / `requests.failed` totals (which are unchanged), the
> lifetime aggregate carries the same two maps, and the OTel counters
> `headroom.proxy.requests.rate_limited` / `.failed` carry `source` / `provider` attributes.
---
## Anthropic subscription panel
+4 -4
View File
@@ -210,7 +210,7 @@
<div class="grid grid-cols-1 gap-4 mb-6 lg:grid-cols-2">
<section class="bg-surface rounded-lg p-4 border border-border">
<h3 class="mb-3 text-sm font-medium text-gray-300">Request Health</h3>
<div class="grid grid-cols-2 gap-3 text-sm"><div>Completed <span class="float-right tabular-nums" x-text="formatNumber(stats.requests?.total || 0)"></span></div><div>Failed <span class="float-right tabular-nums text-red-400" x-text="formatNumber(stats.requests?.failed || 0)"></span></div><div>Rate Limited <span class="float-right tabular-nums text-amber-400" x-text="formatNumber(stats.requests?.rate_limited || 0)"></span></div><div>Cached <span class="float-right tabular-nums text-cyan-400" x-text="formatNumber(stats.requests?.cached || 0)"></span></div></div>
<div class="grid grid-cols-2 gap-3 text-sm"><div>Completed <span class="float-right tabular-nums" x-text="formatNumber(stats.requests?.total || 0)"></span></div><div title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed <span class="float-right tabular-nums text-red-400" x-text="formatNumber(stats.requests?.failed || 0)"></span></div><div title="Requests rejected with 429 — by Headroom&#39;s own rate limiter or by the upstream provider. /stats splits them under requests.rate_limited_by_source.">Rate Limited <span class="float-right tabular-nums text-amber-400" x-text="formatNumber(stats.requests?.rate_limited || 0)"></span></div><div>Cached <span class="float-right tabular-nums text-cyan-400" x-text="formatNumber(stats.requests?.cached || 0)"></span></div></div>
</section>
<section class="bg-surface rounded-lg p-4 border border-border">
<h3 class="mb-3 text-sm font-medium text-gray-300">Live Activity</h3>
@@ -418,7 +418,7 @@
<span class="font-mono text-sm" x-text="((stats.ttfb?.min_ms || 0) / 1000).toFixed(2) + ' - ' + ((stats.ttfb?.max_ms || 0) / 1000).toFixed(2) + 's'"></span>
</div>
<div class="flex justify-between items-center">
<span class="text-sm text-gray-400">Failed Requests</span>
<span class="text-sm text-gray-400" title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed Requests</span>
<span class="font-mono text-sm" x-text="stats.requests?.failed || 0"></span>
</div>
<!-- Per-transform timing breakdown -->
@@ -1470,8 +1470,8 @@
</div>
<div class="grid grid-cols-2 gap-4 mb-6 md:grid-cols-4">
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Requests</div><div class="text-2xl tabular-nums" x-text="formatNumber(lifetimeStats.requests?.total || 0)"></div></div>
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Failed</div><div class="text-2xl tabular-nums text-red-400" x-text="formatNumber(lifetimeStats.requests?.failed || 0)"></div></div>
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Rate Limited</div><div class="text-2xl tabular-nums text-amber-400" x-text="formatNumber(lifetimeStats.requests?.rate_limited || 0)"></div></div>
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500" title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed</div><div class="text-2xl tabular-nums text-red-400" x-text="formatNumber(lifetimeStats.requests?.failed || 0)"></div></div>
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500" title="Requests rejected with 429 — by Headroom&#39;s own rate limiter or by the upstream provider. /stats splits them under requests.rate_limited_by_source.">Rate Limited</div><div class="text-2xl tabular-nums text-amber-400" x-text="formatNumber(lifetimeStats.requests?.rate_limited || 0)"></div></div>
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Cached</div><div class="text-2xl tabular-nums text-cyan-400" x-text="formatNumber(lifetimeStats.requests?.cached || 0)"></div></div>
</div>
+13 -3
View File
@@ -207,12 +207,15 @@ class HeadroomOtelMetrics:
)
self._proxy_failed_requests = self._meter.create_counter(
"headroom.proxy.requests.failed",
description="Proxy requests that failed.",
description="Proxy requests that failed upstream (4xx and 5xx, excluding 429).",
unit="1",
)
self._proxy_rate_limited_requests = self._meter.create_counter(
"headroom.proxy.requests.rate_limited",
description="Proxy requests rejected by rate limiting.",
description=(
"Proxy requests rejected with 429, by source "
"(headroom=our own limiter, upstream=the provider)."
),
unit="1",
)
self._proxy_input_tokens = self._meter.create_counter(
@@ -567,8 +570,15 @@ class HeadroomOtelMetrics:
*,
provider: str | None = None,
model: str | None = None,
source: str = "headroom",
) -> None:
self._proxy_rate_limited_requests.add(1, self._attrs(provider=provider, model=model))
# ``source`` mirrors the headroom_requests_rate_limited_total{source}
# Prometheus label: "headroom" = our own limiter refused the request,
# "upstream" = the provider did. Same split in both backends so an
# operator's query means the same thing whichever one they scrape.
self._proxy_rate_limited_requests.add(
1, self._attrs(provider=provider, model=model, source=source)
)
def record_proxy_cache_bust(self, *, tokens_lost: int) -> None:
self._proxy_cache_busts.add(1)
+3 -1
View File
@@ -1253,7 +1253,9 @@ class AnthropicHandlerMixin:
rate_key = f"{api_key[:16]}:{client_ip}" if api_key else client_ip
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(provider=provider_name)
await self.metrics.record_rate_limited(
provider=provider_name, source="headroom"
)
# Unit 4: release the pre-upstream semaphore before we
# bail out of the handler via HTTPException — FastAPI's
# exception handler will NOT run our ``finally``.
+1 -1
View File
@@ -402,7 +402,7 @@ class GeminiHandlerMixin:
rate_key = headers.get("x-goog-api-key", "default")[:20]
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(provider=provider_name)
await self.metrics.record_rate_limited(provider=provider_name, source="headroom")
raise HTTPException(
status_code=429,
detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
+4 -2
View File
@@ -3551,7 +3551,9 @@ class OpenAIHandlerMixin:
rate_key = headers.get("authorization", "default")[:20]
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(provider=openai_chat_outcome_provider)
await self.metrics.record_rate_limited(
provider=openai_chat_outcome_provider, source="headroom"
)
raise HTTPException(
status_code=429,
detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
@@ -5676,7 +5678,7 @@ class OpenAIHandlerMixin:
rate_key = headers.get("authorization", "default")[:20]
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(provider="openai")
await self.metrics.record_rate_limited(provider="openai", source="headroom")
raise HTTPException(
status_code=429,
detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
+8 -1
View File
@@ -531,9 +531,16 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
# limit is the one 4xx a user is expected to act on (back off, raise a cap),
# and folding it into a generic failure count hides exactly that. Both
# counters are already exported and neither feeds savings.
#
# source="upstream": this funnel only ever sees a 429 the PROVIDER returned.
# Headroom's own limiter rejects before a request is ever sent and records
# source="headroom" from the handler. The two are acted on differently —
# raise our cap vs. back off / shard keys — so they must stay separable
# (issue #3696). ``outcome.provider`` is the already-Copilot-relabelled
# value, matching every other provider-labelled metric on this path.
if outcome.status_code >= 400:
if outcome.status_code == 429:
await handler.metrics.record_rate_limited(provider=outcome.provider)
await handler.metrics.record_rate_limited(provider=outcome.provider, source="upstream")
else:
await handler.metrics.record_failed(provider=outcome.provider)
return
+39 -2
View File
@@ -16,6 +16,9 @@ MAX_EXPOSED_MODELS = 100
MAX_LABEL_LENGTH = 128
KNOWN_MISS_REASONS = frozenset({"ttl_expiry", "prefix_change", "unknown"})
# Mirrors RATE_LIMIT_SOURCES in prometheus_metrics: "headroom" is our own
# limiter refusing the request, "upstream" is the provider refusing it.
RATE_LIMIT_SOURCES = frozenset({"headroom", "upstream"})
# Names must match ``WasteSignals.to_dict()`` in headroom/config.py — the
# parser emits ``json_bloat``; an allowlist that says ``json_noise`` silently
# shoves the largest waste category into the catch-all bucket.
@@ -91,6 +94,12 @@ def _empty_state() -> dict[str, Any]:
"cached": 0,
"failed": 0,
"rate_limited": 0,
# Who produced the failure / who refused the request. The totals
# above stay the unlabelled figures the dashboard reads; these
# split them the same way the Prometheus labels do, so no backend
# reports a differently-shaped counter (issue #3696).
"failed_by_provider": {},
"rate_limited_by_source": {},
"by_provider": {},
"by_stack": {},
},
@@ -182,6 +191,17 @@ class PersistentMetricsState:
result["requests"]["by_provider"] = self._normalize_count_map(
raw_requests.get("by_provider"), MAX_PROVIDER_VALUES
)
result["requests"]["failed_by_provider"] = self._normalize_count_map(
raw_requests.get("failed_by_provider"), MAX_PROVIDER_VALUES
)
# A state file written before #3696 has no split, and back-filling the
# legacy total into one bucket would invent history we never observed —
# so it loads empty and only new events populate it. The unlabelled
# ``rate_limited`` total is untouched and stays comparable across the
# upgrade.
result["requests"]["rate_limited_by_source"] = self._normalize_enum_map(
raw_requests.get("rate_limited_by_source"), RATE_LIMIT_SOURCES, fallback="headroom"
)
result["requests"]["by_stack"] = self._normalize_count_map(
raw_requests.get("by_stack"), MAX_STACK_VALUES
)
@@ -474,12 +494,29 @@ class PersistentMetricsState:
self._record_activity()
self._state["requests"]["failed"] += 1
self._increment_count(
self._state["requests"]["failed_by_provider"], _label(provider), MAX_PROVIDER_VALUES
)
def record_rate_limited(self, *, provider: str | None = None, model: str | None = None) -> None:
"""Record a rate-limited request without redefining total request semantics."""
def record_rate_limited(
self,
*,
provider: str | None = None,
model: str | None = None,
source: str = "headroom",
) -> None:
"""Record a rate-limited request without redefining total request semantics.
``source`` is ``headroom`` (our own limiter) or ``upstream`` (the
provider's 429); anything else is clamped to ``headroom``, which is what
this counter meant before upstream 429s started reaching it.
"""
self._record_activity()
self._state["requests"]["rate_limited"] += 1
bucket = source if source in RATE_LIMIT_SOURCES else "headroom"
by_source = self._state["requests"]["rate_limited_by_source"]
by_source[bucket] = by_source.get(bucket, 0) + 1
def record_cache_bust(self, *, tokens_lost: Any = 0) -> None:
self._record_activity()
+107 -15
View File
@@ -30,6 +30,33 @@ logger = logging.getLogger("headroom.proxy")
# client-supplied model cardinality stays bounded (see record_request).
_OTHER_MODEL = "other"
# Closed label set for headroom_requests_rate_limited_total{source}. Two 429s
# mean opposite things to an operator: "headroom" is OUR limiter refusing the
# request (raise the cap), "upstream" is the provider refusing it (back off or
# shard keys). Before #3615 only the first could ever increment this counter;
# the outcome funnel now routes provider 429s here too, so the split has to be
# queryable instead of silently merged. Closed set => bounded cardinality, and
# both series are exported from startup so a rate() never goes from absent to
# present mid-incident.
RATE_LIMIT_SOURCE_HEADROOM = "headroom"
RATE_LIMIT_SOURCE_UPSTREAM = "upstream"
RATE_LIMIT_SOURCES = (RATE_LIMIT_SOURCE_HEADROOM, RATE_LIMIT_SOURCE_UPSTREAM)
# Bucket for a failure with no attributed provider. Also seeded at zero so
# headroom_requests_failed_total always exports at least one sample.
_PROVIDER_UNKNOWN = "unknown"
def _rate_limit_source(source: str | None) -> str:
"""Clamp ``source`` to the closed label set, defaulting to Headroom's limiter.
Defaults to ``headroom`` because that is what this counter meant for its
whole life before the outcome funnel started feeding it upstream 429s: an
un-updated caller keeps the historical reading rather than inventing a new
label value.
"""
return source if source in RATE_LIMIT_SOURCES else RATE_LIMIT_SOURCE_HEADROOM
def _escape_label_value(value: str) -> str:
# The /metrics body is emitted whole with .encode("utf-8") (server.py). A
@@ -100,7 +127,26 @@ class PrometheusMetrics:
self.requests_by_stack: dict[str, int] = defaultdict(int)
self.requests_cached = 0
self.requests_rate_limited = 0
# Seeded with both sources at zero so /metrics exports the full series
# set from the first scrape; ``requests_rate_limited`` stays the
# unlabelled total that /stats and the session summary read.
self.requests_rate_limited_by_source: dict[str, int] = dict.fromkeys(RATE_LIMIT_SOURCES, 0)
self.requests_failed = 0
# Per-provider failure attribution. #3615 moved 4xx out of the success
# funnel, which also took them out of ``requests_by_provider`` (only
# record_request touches that), leaving Prometheus with no way to tell
# which upstream was failing. Kept alongside the unlabelled total.
#
# Seeded with the "unknown" bucket at zero so the metric always exports
# at least one sample. Before this counter carried a label it was always
# present as a bare ``headroom_requests_failed_total 0``; a labelled map
# that starts empty would emit NO sample on a healthy proxy, which turns
# the documented failure-rate query into an empty vector (an empty
# numerator makes the whole expression empty, so the panel reads
# "No data" instead of 0%) and would make ``absent()`` alerts fire on
# healthy proxies. "unknown" is a real bucket -- record_failed uses it
# when no provider is attributed -- so seeding it invents no provider.
self.requests_failed_by_provider: dict[str, int] = defaultdict(int, {_PROVIDER_UNKNOWN: 0})
self.inbound_requests_total = 0
self.inbound_requests_completed = 0
self.inbound_requests_active = 0
@@ -343,7 +389,11 @@ class PrometheusMetrics:
self.requests_by_stack.clear()
self.requests_cached = 0
self.requests_rate_limited = 0
self.requests_rate_limited_by_source = dict.fromkeys(RATE_LIMIT_SOURCES, 0)
self.requests_failed = 0
self.requests_failed_by_provider.clear()
# Re-seed so /metrics keeps exporting a sample after a reset.
self.requests_failed_by_provider[_PROVIDER_UNKNOWN] = 0
self.inbound_requests_total = 0
self.inbound_requests_completed = 0
self.inbound_requests_active = 0
@@ -1175,15 +1225,38 @@ class PrometheusMetrics:
if ms_val > self.ws_session_duration_max_ms[cause]:
self.ws_session_duration_max_ms[cause] = ms_val
async def record_rate_limited(self, *, provider: str | None = None, model: str | None = None):
async def record_rate_limited(
self,
*,
provider: str | None = None,
model: str | None = None,
source: str = RATE_LIMIT_SOURCE_HEADROOM,
):
"""Record a 429.
``source`` says WHO refused the request: ``headroom`` for our own
limiter (the handlers, which raise HTTPException and never emit an
outcome) or ``upstream`` for a provider 429 arriving through the
outcome funnel. Threaded to every sink — Prometheus, lifetime/persistent
and OTel — so no backend reports a differently-shaped counter.
"""
source = _rate_limit_source(source)
async with self._lock:
self.requests_rate_limited += 1
self.savings_tracker.record_lifetime_rate_limited(provider=provider, model=model)
self._get_otel_metrics().record_proxy_rate_limited(provider=provider, model=model)
self.requests_rate_limited_by_source[source] = (
self.requests_rate_limited_by_source.get(source, 0) + 1
)
self.savings_tracker.record_lifetime_rate_limited(
provider=provider, model=model, source=source
)
self._get_otel_metrics().record_proxy_rate_limited(
provider=provider, model=model, source=source
)
async def record_failed(self, *, provider: str | None = None, model: str | None = None):
async with self._lock:
self.requests_failed += 1
self.requests_failed_by_provider[provider or _PROVIDER_UNKNOWN] += 1
self.savings_tracker.record_lifetime_failed(provider=provider, model=model)
self._get_otel_metrics().record_proxy_failed(provider=provider, model=model)
@@ -1215,20 +1288,39 @@ class PrometheusMetrics:
help_text="Cached request count",
value=self.requests_cached,
)
_append_metric(
lines,
name="headroom_requests_rate_limited_total",
metric_type="counter",
help_text="Rate limited requests",
value=self.requests_rate_limited,
# Labelled series only — an unlabelled sample alongside these would
# double-count under sum(). `sum without (source)` reproduces the
# pre-label value exactly.
lines.extend(
[
"# HELP headroom_requests_rate_limited_total Requests rejected with 429, "
"by who rejected them (headroom=our own limiter, upstream=the provider)",
"# TYPE headroom_requests_rate_limited_total counter",
]
)
_append_metric(
lines,
name="headroom_requests_failed_total",
metric_type="counter",
help_text="Failed requests",
value=self.requests_failed,
for _source in RATE_LIMIT_SOURCES:
_count = self.requests_rate_limited_by_source.get(_source, 0)
lines.append(
f'headroom_requests_rate_limited_total{{source="{_escape_label_value(_source)}"}}'
f" {_count}"
)
lines.append("")
# Failures carry the provider that produced them. `sum without
# (provider)` reproduces the pre-label value.
lines.extend(
[
"# HELP headroom_requests_failed_total Requests that failed upstream "
"(4xx and 5xx, excluding 429), by provider",
"# TYPE headroom_requests_failed_total counter",
]
)
for _provider, _count in self.requests_failed_by_provider.items():
lines.append(
f'headroom_requests_failed_total{{provider="'
f'{_escape_label_value(str(_provider))}"}} {_count}'
)
lines.append("")
_append_metric(
lines,
name="headroom_inbound_requests_total",
+13 -3
View File
@@ -1210,12 +1210,22 @@ class SavingsTracker:
self._maybe_save_locked()
def record_lifetime_rate_limited(
self, *, provider: str | None = None, model: str | None = None
self,
*,
provider: str | None = None,
model: str | None = None,
source: str = "headroom",
) -> None:
"""Record a rate-limited proxy request without changing legacy history."""
"""Record a rate-limited proxy request without changing legacy history.
``source`` distinguishes Headroom's own limiter from an upstream 429 —
see ``PrometheusMetrics.record_rate_limited``.
"""
with self._lock:
self._persistent_metrics.record_rate_limited(provider=provider, model=model)
self._persistent_metrics.record_rate_limited(
provider=provider, model=model, source=source
)
self._maybe_save_locked()
def record_lifetime_cache_bust(self, *, tokens_lost: int) -> None:
+13
View File
@@ -2263,6 +2263,12 @@ class HeadroomProxy(
logger.info(f"Total requests: {m.requests_total}")
logger.info(f"Cached responses: {m.requests_cached}")
logger.info(f"Rate limited: {m.requests_rate_limited}")
if m.requests_rate_limited:
# The split an operator acts on differently: our limiter firing
# means raise the cap, the provider's means back off / shard keys.
by_source = m.requests_rate_limited_by_source
logger.info(f" headroom limiter: {by_source.get('headroom', 0)}")
logger.info(f" upstream 429s: {by_source.get('upstream', 0)}")
logger.info(f"Failed: {m.requests_failed}")
logger.info(f"Input tokens: {m.tokens_input_total:,}")
logger.info(f"Output tokens: {m.tokens_output_total:,}")
@@ -4572,7 +4578,14 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
"total": m.requests_total,
"cached": m.requests_cached,
"rate_limited": m.requests_rate_limited,
# Who issued the 429: "headroom" is our own limiter, "upstream"
# is the provider (issue #3696). The totals above stay the
# unlabelled figures existing consumers read.
"rate_limited_by_source": dict(m.requests_rate_limited_by_source),
"failed": m.requests_failed,
"failed_by_provider": _remap_provider_counts(
dict(m.requests_failed_by_provider), proxy.config
),
"by_provider": _remap_provider_counts(dict(m.requests_by_provider), proxy.config),
"by_model": dict(m.requests_by_model),
"by_stack": dict(m.requests_by_stack),
+46 -1
View File
@@ -356,11 +356,56 @@ async def test_prometheus_metrics_reads_late_configured_otel_metrics() -> None:
await metrics.record_rate_limited(provider="anthropic", model="claude-sonnet")
assert spy.failed_calls == [{"provider": "openai", "model": None}]
assert spy.rate_limited_calls == [{"provider": "anthropic", "model": "claude-sonnet"}]
# ``source`` reaches OTel too — the split must not exist in Prometheus only.
assert spy.rate_limited_calls == [
{"provider": "anthropic", "model": "claude-sonnet", "source": "headroom"}
]
finally:
reset_otel_metrics()
@pytest.mark.asyncio
async def test_prometheus_metrics_forwards_rate_limit_source_to_otel() -> None:
"""An upstream 429 must reach OTel labelled as upstream, not as our own limiter."""
spy = _SpyProxyMetrics()
metrics = PrometheusMetrics(stateless=True)
set_otel_metrics(spy) # type: ignore[arg-type]
try:
await metrics.record_rate_limited(provider="anthropic", source="upstream")
await metrics.record_rate_limited(provider="anthropic", source="headroom")
# An unrecognised value is clamped rather than exported as a new label.
await metrics.record_rate_limited(provider="anthropic", source="nonsense")
assert [call["source"] for call in spy.rate_limited_calls] == [
"upstream",
"headroom",
"headroom",
]
finally:
reset_otel_metrics()
def test_otel_rate_limited_counter_carries_source_attribute() -> None:
reader = InMemoryMetricReader()
provider = MeterProvider(metric_readers=[reader])
otel_metrics = HeadroomOtelMetrics(meter_provider=provider)
otel_metrics.record_proxy_rate_limited(provider="anthropic", source="upstream")
otel_metrics.record_proxy_rate_limited(provider="openai", source="headroom")
metrics = _collect_metrics(reader)
upstream = _find_point(
metrics["headroom.proxy.requests.rate_limited"], provider="anthropic", source="upstream"
)
headroom_side = _find_point(
metrics["headroom.proxy.requests.rate_limited"], provider="openai", source="headroom"
)
assert upstream.value == 1
assert headroom_side.value == 1
@pytest.mark.asyncio
async def test_prometheus_metrics_forwards_savings_drilldown_fields_to_otel(
monkeypatch: pytest.MonkeyPatch,
+12 -1
View File
@@ -25,13 +25,19 @@ class _Metrics:
def __init__(self):
self.failed = []
self.rate_limited = []
self.rate_limited_sources = []
self.requested = []
async def record_failed(self, provider):
self.failed.append(provider)
async def record_rate_limited(self, provider):
# ``source`` deliberately has NO default here: the funnel only ever sees
# 429s the provider returned, so it must pass source="upstream" explicitly.
# If it ever stops doing so, these tests record None and fail loudly rather
# than silently re-merging upstream throttling into Headroom's own counter.
async def record_rate_limited(self, provider, source=None):
self.rate_limited.append(provider)
self.rate_limited_sources.append(source)
async def record_request(self, **kwargs):
self.requested.append(kwargs)
@@ -78,6 +84,11 @@ def test_429_recorded_as_rate_limited_and_skips_success_funnel():
handler = _Handler()
asyncio.run(emit_request_outcome(handler, _outcome(429, tokens_saved=6380)))
assert handler.metrics.rate_limited == ["anthropic"]
# Labelled as the PROVIDER's 429, not ours. Headroom's own limiter rejects
# before a request is ever sent and records source="headroom" from the
# handler; merging the two would tell an operator to raise a cap that is
# not the one being hit (issue #3696).
assert handler.metrics.rate_limited_sources == ["upstream"]
assert handler.metrics.failed == [] # a rate limit is not a generic failure
assert handler.metrics.requested == [] # its 6,380 "saved" tokens are not savings
+56
View File
@@ -50,6 +50,9 @@ def test_snapshot_accumulates_request_token_cache_cost_and_waste_metrics() -> No
"cached": 1,
"failed": 1,
"rate_limited": 1,
# Lifetime carries the same splits the Prometheus labels do.
"failed_by_provider": {"anthropic": 1},
"rate_limited_by_source": {"headroom": 1},
"by_provider": {"anthropic": 1},
"by_stack": {"codex": 1},
}
@@ -205,3 +208,56 @@ def test_miss_reasons_still_fall_back_to_unknown_on_reload() -> None:
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
assert snapshot["prefix_cache"]["misses_by_reason"] == {"ttl_expiry": 2, "unknown": 1}
def test_rate_limited_is_split_by_source_and_failures_by_provider() -> None:
"""Headroom's own limiter and an upstream 429 must stay distinguishable.
Both land in the same ``rate_limited`` total (unchanged), but an operator
acts on them differently: ours firing means raise the cap, the provider's
means back off. See issue #3696.
"""
state = _new_state()
state.record_rate_limited(provider="anthropic", source="headroom")
state.record_rate_limited(provider="anthropic", source="upstream")
state.record_rate_limited(provider="openai", source="upstream")
state.record_failed(provider="anthropic")
state.record_failed(provider="openai")
state.record_failed(provider="openai")
requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
assert requests["rate_limited"] == 3
assert requests["rate_limited_by_source"] == {"headroom": 1, "upstream": 2}
assert requests["failed"] == 3
assert requests["failed_by_provider"] == {"anthropic": 1, "openai": 2}
def test_rate_limit_source_defaults_to_headroom_and_clamps_unknown_values() -> None:
state = _new_state()
state.record_rate_limited(provider="anthropic")
state.record_rate_limited(provider="anthropic", source="bogus")
requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
assert requests["rate_limited_by_source"] == {"headroom": 2}
def test_pre_3696_state_loads_without_backfilling_invented_history() -> None:
"""A state file written before the split keeps its totals and starts the maps empty.
Back-filling the legacy ``rate_limited`` total into one bucket would claim we
observed a split we never recorded.
"""
state = PersistentMetricsState(
{"requests": {"total": 10, "failed": 4, "rate_limited": 7}},
now=lambda: FIXED_NOW,
)
requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
assert requests["rate_limited"] == 7
assert requests["failed"] == 4
assert requests["rate_limited_by_source"] == {}
assert requests["failed_by_provider"] == {}
@@ -39,6 +39,10 @@ def test_runtime_metric_events_feed_lifetime_without_resetting_runtime_counters(
assert lifetime["requests"]["cached"] == 1
assert lifetime["requests"]["failed"] == 1
assert lifetime["requests"]["rate_limited"] == 1
# The Prometheus labels must not stop at Prometheus: lifetime carries the
# same two splits (issue #3696). No explicit source here, so it is ours.
assert lifetime["requests"]["failed_by_provider"] == {"anthropic": 1}
assert lifetime["requests"]["rate_limited_by_source"] == {"headroom": 1}
assert lifetime["requests"]["by_provider"] == {"anthropic": 1}
assert lifetime["requests"]["by_stack"] == {"codex": 1}
assert lifetime["tokens"]["output"] == 3
@@ -47,3 +51,27 @@ def test_runtime_metric_events_feed_lifetime_without_resetting_runtime_counters(
assert lifetime["prefix_cache"]["misses_by_reason"] == {"prefix_change": 1}
assert lifetime["waste_signals"] == {"repetition": 4}
assert metrics.requests_total == 1
def test_upstream_rate_limit_reaches_lifetime_labelled_upstream(tmp_path) -> None:
"""An upstream 429 must not be filed under Headroom's own limiter in lifetime.
The outcome funnel passes ``source="upstream"``; the whole chain
(PrometheusMetrics -> SavingsTracker -> PersistentMetricsState) has to carry
it, or the durable view re-merges what Prometheus splits (issue #3696).
"""
tracker = SavingsTracker(path=str(tmp_path / "proxy_savings.json"), save_flush_every=25)
metrics = PrometheusMetrics(savings_tracker=tracker)
asyncio.run(metrics.record_rate_limited(provider="anthropic", source="upstream"))
asyncio.run(metrics.record_rate_limited(provider="anthropic", source="headroom"))
asyncio.run(metrics.record_failed(provider="openai"))
lifetime = tracker.lifetime_response()
assert lifetime["requests"]["rate_limited"] == 2
assert lifetime["requests"]["rate_limited_by_source"] == {"upstream": 1, "headroom": 1}
assert lifetime["requests"]["failed_by_provider"] == {"openai": 1}
# Neither counter may touch the completed-request denominator.
assert lifetime["requests"]["total"] == 0
assert metrics.requests_total == 0
+127
View File
@@ -176,3 +176,130 @@ async def test_gate_counter_is_thread_safe_under_concurrent_export() -> None:
assert not errors, f"export() raced the writer: {errors[:3]}"
totals = dict(metrics.kompress_size_gate_by_outcome)
assert sum(totals.values()) == n_threads * per_thread
# ---------------------------------------------------------------------------
# headroom_requests_rate_limited_total{source} / headroom_requests_failed_total{provider}
#
# Issue #3696: #3615 routed upstream 429s and 4xx into these two counters, which
# silently changed what they mean. The labels make the new meaning queryable —
# "my limiter throttled me" vs "the provider throttled me" are acted on
# differently, and a failure with no provider attached names no culprit.
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_rate_limited_export_splits_headroom_from_upstream() -> None:
metrics = PrometheusMetrics(stateless=True)
# What the four handler sites do when Headroom's OWN limiter rejects.
await metrics.record_rate_limited(provider="anthropic", source="headroom")
# What the outcome funnel does with a provider 429.
await metrics.record_rate_limited(provider="anthropic", source="upstream")
await metrics.record_rate_limited(provider="openai", source="upstream")
text = await metrics.export()
assert "# TYPE headroom_requests_rate_limited_total counter" in text
assert 'headroom_requests_rate_limited_total{source="headroom"} 1' in text
assert 'headroom_requests_rate_limited_total{source="upstream"} 2' in text
# `sum without (source)` must reproduce the pre-label value.
assert metrics.requests_rate_limited == 3
# No unlabelled sample alongside the labelled ones — that would double-count.
assert "\nheadroom_requests_rate_limited_total 3" not in text
@pytest.mark.asyncio
async def test_rate_limited_source_series_exist_before_any_429() -> None:
"""Both series export from startup, so a rate() never appears mid-incident."""
metrics = PrometheusMetrics(stateless=True)
text = await metrics.export()
assert 'headroom_requests_rate_limited_total{source="headroom"} 0' in text
assert 'headroom_requests_rate_limited_total{source="upstream"} 0' in text
@pytest.mark.asyncio
async def test_rate_limited_source_defaults_to_headroom_and_clamps_unknown() -> None:
metrics = PrometheusMetrics(stateless=True)
await metrics.record_rate_limited(provider="anthropic")
await metrics.record_rate_limited(provider="anthropic", source="bogus")
text = await metrics.export()
assert 'headroom_requests_rate_limited_total{source="headroom"} 2' in text
assert 'headroom_requests_rate_limited_total{source="upstream"} 0' in text
assert "bogus" not in text
@pytest.mark.asyncio
async def test_failed_export_is_attributed_to_the_provider_that_failed() -> None:
metrics = PrometheusMetrics(stateless=True)
await metrics.record_failed(provider="anthropic")
await metrics.record_failed(provider="openai")
await metrics.record_failed(provider="openai")
text = await metrics.export()
assert "# TYPE headroom_requests_failed_total counter" in text
assert 'headroom_requests_failed_total{provider="anthropic"} 1' in text
assert 'headroom_requests_failed_total{provider="openai"} 2' in text
# `sum without (provider)` must reproduce the pre-label value.
assert metrics.requests_failed == 3
assert "\nheadroom_requests_failed_total 3" not in text
@pytest.mark.asyncio
async def test_failed_without_a_provider_lands_in_unknown() -> None:
metrics = PrometheusMetrics(stateless=True)
await metrics.record_failed()
text = await metrics.export()
assert 'headroom_requests_failed_total{provider="unknown"} 1' in text
@pytest.mark.asyncio
async def test_reset_runtime_clears_rate_limit_and_failure_splits() -> None:
metrics = PrometheusMetrics(stateless=True)
await metrics.record_rate_limited(provider="anthropic", source="upstream")
await metrics.record_failed(provider="anthropic")
await metrics.reset_runtime()
assert metrics.requests_rate_limited == 0
assert metrics.requests_failed == 0
assert metrics.requests_rate_limited_by_source == {"headroom": 0, "upstream": 0}
# Re-seeded, not emptied: the metric must keep exporting a sample after a
# reset, same as the rate-limit source series above.
assert dict(metrics.requests_failed_by_provider) == {"unknown": 0}
@pytest.mark.asyncio
async def test_failed_total_exports_a_sample_before_any_failure() -> None:
"""A healthy proxy must still export ``headroom_requests_failed_total``.
Before the provider label this counter was always present as a bare
``headroom_requests_failed_total 0``. A labelled map that starts empty
emits no sample at all, which makes the documented failure-rate query
return an empty vector (an empty numerator empties the whole expression),
so the panel reads "No data" instead of 0% while everything is fine --
and `absent()` alerts fire on healthy proxies.
"""
metrics = PrometheusMetrics(stateless=True)
text = await metrics.export()
assert 'headroom_requests_failed_total{provider="unknown"} 0' in text
# The seed must not survive as a duplicate once a real provider fails.
await metrics.record_failed(provider="anthropic")
text = await metrics.export()
assert 'headroom_requests_failed_total{provider="anthropic"} 1' in text
assert 'headroom_requests_failed_total{provider="unknown"} 0' in text
assert metrics.requests_failed == 1
+11 -1
View File
@@ -29,7 +29,9 @@ curl http://localhost:8787/stats
"total": 42,
"cached": 5,
"rate_limited": 0,
"failed": 0
"rate_limited_by_source": { "headroom": 0, "upstream": 0 },
"failed": 0,
"failed_by_provider": {}
},
"tokens": {
"input": 50000,
@@ -48,6 +50,14 @@ curl http://localhost:8787/stats
}
```
`requests.total` is the **completed**-request count, not total traffic: a request that
ended in a 4xx, 5xx or 429 is counted under `requests.failed` or `requests.rate_limited`
instead and never reaches `total`. `rate_limited_by_source` splits 429s into
`headroom` (Headroom's own rate limiter refused the request) and `upstream` (the provider
refused it) — different problems, different fixes. `failed_by_provider` attributes failures
to the upstream that produced them. The unlabelled `rate_limited` / `failed` totals are
unchanged.
`/stats` keeps the existing live/session fields, including `savings_history`,
for backward compatibility. The new `persistent_savings` block is durable local
proxy compression history stored by default at