mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(metrics): split rate-limited by source and label failures by provider (#3707)
Closes #3696. #3615 moved the outcome-funnel short-circuit from `>= 500` to `>= 400` — correct, and staying — but it silently changed what two metrics mean and the docs were never updated. **⚠️ This changes exported series. See Migration.** ## What was wrong - `headroom_requests_failed_total` was documented as "Upstream 5xx errors"; it now counts 4xx too. - `headroom_requests_rate_limited_total` was documented as "Requests **Headroom** rejected via its own rate limiter (**not upstream 429s**)" — #3615 inverted exactly that. - 4xx also dropped out of `headroom_requests_by_provider` (only `record_request` touches it), leaving Prometheus with no way to tell which upstream was failing. ## Changes **`headroom_requests_rate_limited_total{source="headroom"|"upstream"}`** — two 429s mean opposite things to an operator: `headroom` is our limiter refusing (raise the cap), `upstream` is the provider refusing (back off / shard keys). Closed label set, so cardinality stays bounded; unknown values clamp to `headroom`, the counter's pre-#3615 meaning, so an un-updated caller cannot invent a label. Threaded at all five call sites (`openai.py:3554, 5681`, `anthropic.py:1256`, `gemini.py:405` → `headroom`; `outcome.py:543`, the #3615 funnel → `upstream`). **`headroom_requests_failed_total{provider}`** — uses `outcome.provider`, already Copilot-relabelled by the funnel, so no second convention. Matches the existing `upstream_connection_errors_by_provider` shape. **All four backends**, so no sink reports a differently-shaped counter: Prometheus, persistent/lifetime (`SavingsTracker.record_lifetime_rate_limited`), OTel (attribute on `headroom.proxy.requests.rate_limited`), and `/stats` JSON. `SCHEMA_VERSION` stays 5 — the new maps are purely additive, and old state files load with them empty rather than back-filling invented history. **Docs**: `docs/metrics-technical-guide.md`, `docs/content/docs/metrics.mdx`, `wiki/metrics.md`, plus tooltips on the dashboard's Failed / Rate Limited tiles so the UI stops implying "Rate Limited" means Headroom only. `requests_total` is now documented as a completed-request denominator with `inbound_requests_total` named as the all-traffic counter, and the failure-rate PromQL carries a "why" on its denominator so nobody "simplifies" it. Checked and found clean: `examples/grafana/headroom-dashboard.json`, `docs/observability.md`, `docs/content/docs/proxy.mdx`, `wiki/proxy.md`, `README.md`. ## Second commit: a bug found while reviewing the first Adding the label turned `headroom_requests_failed_total` from a counter that **always** exported `0` into one that exports **no sample at all** on a healthy proxy — confirmed by rendering `/metrics` on a fresh instance: ``` headroom_requests_failed_total: 0 sample(s) headroom_requests_rate_limited_total: 2 sample(s) ``` That breaks the failure-rate query documented in this very PR: an empty numerator empties the whole PromQL expression, so the panel reads "No data" instead of 0% while everything is fine, and `absent()` alerts would fire on healthy proxies. The rate-limit series avoided this by seeding its closed label set; providers are open-ended so the same trick does not apply. Fix: seed the `unknown` bucket at zero. It is a real bucket — `record_failed` uses it for unattributed failures — so this invents no provider. Guarded by a new test. ## Migration Both counters now export **only** labelled series; an unlabelled sample alongside would double-count under `sum()`. Unaggregated queries return multiple series — `sum without (source)` / `sum without (provider)` reproduces the previous value exactly. Noted in the docs and an mdx Callout. ## Verification - **16 tests fail on pre-change source** (verified by reverting only `headroom/`), pass after. - Targeted metrics suites → **58 passed**. Broad sweep `-k "metric or stats or outcome or rate_limit or savings_tracker or persistent"` → 634 passed / 44 skipped. Handler, dashboard and savings-history suites → 90 + 79 passed. - Ruff 0.16.7 (the CI pin, via `scripts/verify-ruff-version.py --print-version`): clean. `CHANGELOG.md` intentionally untouched — `changelog-guard.yml` fails any PR that edits it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
94206e2652
commit
5ff4ea1ef9
@@ -25,7 +25,9 @@ curl http://localhost:8787/stats
|
||||
"total": 42,
|
||||
"cached": 5,
|
||||
"rate_limited": 0,
|
||||
"failed": 0
|
||||
"rate_limited_by_source": { "headroom": 0, "upstream": 0 },
|
||||
"failed": 0,
|
||||
"failed_by_provider": {}
|
||||
},
|
||||
"tokens": {
|
||||
"input": 50000,
|
||||
@@ -44,6 +46,14 @@ curl http://localhost:8787/stats
|
||||
}
|
||||
```
|
||||
|
||||
`requests.total` is the **completed**-request count, not total traffic: a request that
|
||||
ended in a 4xx, 5xx or 429 is counted under `requests.failed` or `requests.rate_limited`
|
||||
instead and never reaches `total`. `rate_limited_by_source` splits 429s into
|
||||
`headroom` (Headroom's own rate limiter refused the request) and `upstream` (the provider
|
||||
refused it) — different problems, different fixes. `failed_by_provider` attributes failures
|
||||
to the upstream that produced them. The unlabelled `rate_limited` / `failed` totals are
|
||||
unchanged.
|
||||
|
||||
Persistent savings are stored at `~/.headroom/proxy_savings.json` and survive proxy restarts. Override the path with `HEADROOM_SAVINGS_PATH`.
|
||||
|
||||
### Historical Savings
|
||||
@@ -82,8 +92,27 @@ headroom_latency_ms_count 1234
|
||||
|
||||
# HELP headroom_provider_cache_hit_requests_total Requests that read from the provider's prompt cache
|
||||
headroom_provider_cache_hit_requests_total{provider="anthropic"} 456
|
||||
|
||||
# HELP headroom_requests_failed_total Requests that failed upstream (4xx and 5xx, excluding 429), by provider
|
||||
headroom_requests_failed_total{provider="anthropic"} 3
|
||||
|
||||
# HELP headroom_requests_rate_limited_total Requests rejected with 429, by who rejected them (headroom=our own limiter, upstream=the provider)
|
||||
headroom_requests_rate_limited_total{source="headroom"} 0
|
||||
headroom_requests_rate_limited_total{source="upstream"} 12
|
||||
```
|
||||
|
||||
<Callout type="warn" title="requests_total is completed requests, not all traffic">
|
||||
`headroom_requests_total` counts requests that **completed**. A request that ended in a 4xx,
|
||||
5xx or 429 lands in `headroom_requests_failed_total` / `headroom_requests_rate_limited_total`
|
||||
and is not in `requests_total` — so a failure rate must add `failed` back into its own
|
||||
denominator. `headroom_inbound_requests_total` is the all-traffic counter.
|
||||
|
||||
`headroom_requests_failed_total` gained a `provider` label and
|
||||
`headroom_requests_rate_limited_total` a `source` label, so unaggregated queries on them now
|
||||
return several series. `sum without (provider) (...)` and `sum without (source) (...)`
|
||||
reproduce the previous single-series behaviour.
|
||||
</Callout>
|
||||
|
||||
<Callout type="warn" title="No percentile buckets on /metrics">
|
||||
There are no Prometheus histogram buckets for compression ratio or latency — build percentiles or averages from `_sum` / `_count` pairs (`headroom_latency_ms_sum`, `headroom_latency_ms_count`, and so on), or use the `headroom perf` CLI for real p95/p99.
|
||||
</Callout>
|
||||
@@ -351,7 +380,8 @@ Env: `HEADROOM_BUDGET_ESTIMATED_BASIS`. `headroom doctor` reports the estimated
|
||||
| `headroom_persistent_savings_tokens_saved_total` | Durable lifetime tokens saved from `/stats.persistent_savings` | Higher is better |
|
||||
| `headroom_overhead_ms_sum` / `headroom_overhead_ms_count` | Mean latency Headroom itself adds (there are no percentile buckets — see the callout above) | Low is better |
|
||||
| `headroom_provider_cache_hit_requests_total` / `headroom_provider_cache_requests_total` | Cache effectiveness, by provider | >20% is good |
|
||||
| `headroom_requests_failed_total` | Reliability (upstream 5xx errors) | 0 |
|
||||
| `headroom_requests_failed_total{provider}` | Reliability — upstream **4xx and 5xx** (429s are counted separately), attributed to the provider that failed | 0 |
|
||||
| `headroom_requests_rate_limited_total{source}` | Throttling — `source="headroom"` is Headroom's own rate limiter, `source="upstream"` is the provider's | 0 |
|
||||
|
||||
## Grafana Dashboard
|
||||
|
||||
|
||||
@@ -103,21 +103,55 @@ rate(headroom_cache_bust_total[5m])
|
||||
|
||||
| Metric | What it shows |
|
||||
|---|---|
|
||||
| `headroom_requests_total` | Requests handled. Unlabelled. |
|
||||
| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… |
|
||||
| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. |
|
||||
| `headroom_requests_failed_total` | Upstream 5xx errors. |
|
||||
| `headroom_requests_rate_limited_total` | Requests **Headroom** rejected via its own rate limiter (not upstream 429s). |
|
||||
| `headroom_requests_total` | **Completed** requests — the denominator for success-side stats, not total traffic. A request that ended in a 4xx, 5xx or 429 never reaches it. Unlabelled. |
|
||||
| `headroom_inbound_requests_total` | Every inbound HTTP request the proxy accepted, whatever the outcome. This is the all-traffic counter. |
|
||||
| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… **Completed requests only**, same denominator as `headroom_requests_total`. Use `headroom_requests_failed_total{provider}` for the failure side. |
|
||||
| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. Completed requests only. |
|
||||
| `headroom_requests_failed_total{provider}` | Requests that failed upstream — **4xx and 5xx**, excluding 429. Excluded from `headroom_requests_total`, so add it back when you compute a rate. |
|
||||
| `headroom_requests_rate_limited_total{source}` | Requests rejected with 429, by **either** Headroom's own rate limiter (`source="headroom"`) or the upstream provider (`source="upstream"`). Excluded from `headroom_requests_total`. |
|
||||
| `headroom_compression_failed_total{reason}` | Compression failures — `timeout` or `error`. Fails open, so traffic keeps flowing but savings quietly stop. **Worth an alert.** |
|
||||
| `headroom_compression_quarantine_total{event}` | Compression disabled after repeated timeouts — `activated`, `skipped`, `released`. |
|
||||
| `headroom_inbound_requests_active` | In-flight requests, gauge. Counts all HTTP including `/metrics`. |
|
||||
| `headroom_active_ws_sessions` | Live Codex WebSocket sessions, gauge. |
|
||||
|
||||
### `headroom_requests_total` is a denominator, not traffic
|
||||
|
||||
`headroom_requests_total` counts requests that **completed** — it has always been
|
||||
incremented only on the success path, and since the outcome funnel started short-circuiting
|
||||
at `>= 400`, every 4xx, 5xx and 429 drops out of it. So:
|
||||
|
||||
- `headroom_requests_total` + `headroom_requests_failed_total` + `headroom_requests_rate_limited_total`
|
||||
≈ the requests the proxy forwarded upstream.
|
||||
- `headroom_inbound_requests_total` is the honest all-traffic counter (it also counts
|
||||
`/metrics`, `/stats` and other non-proxy HTTP).
|
||||
|
||||
That is why the failure-rate query below adds `failed` back into its own denominator, and
|
||||
why you should not "simplify" it to `… / rate(headroom_requests_total[5m])`: failures are
|
||||
not in `requests_total`, so that form divides failures by successes and over-reports the
|
||||
rate — badly, during exactly the incident you wrote it for (at 100% failures the denominator
|
||||
goes to zero).
|
||||
|
||||
```promql
|
||||
# Failure rate
|
||||
# Failure rate. The denominator deliberately re-adds `failed`: a failed request
|
||||
# is NOT in headroom_requests_total, so requests_total alone is successes-only.
|
||||
rate(headroom_requests_failed_total[5m])
|
||||
/ clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)
|
||||
|
||||
# Failure rate by provider — which upstream is actually broken.
|
||||
sum by (provider) (rate(headroom_requests_failed_total[5m]))
|
||||
/ clamp_min(
|
||||
sum by (provider) (rate(headroom_requests_by_provider[5m]))
|
||||
+ sum by (provider) (rate(headroom_requests_failed_total[5m])),
|
||||
1
|
||||
)
|
||||
|
||||
# Who is throttling you: your own limiter (raise the cap) vs the provider
|
||||
# (back off, shard keys). Alert on these separately — they are different actions.
|
||||
sum by (source) (rate(headroom_requests_rate_limited_total[5m]))
|
||||
|
||||
# Provider throttling only — the one that means "slow down".
|
||||
rate(headroom_requests_rate_limited_total{source="upstream"}[5m])
|
||||
|
||||
# Savings silently stopped
|
||||
sum by (reason) (rate(headroom_compression_failed_total[5m]))
|
||||
|
||||
@@ -125,6 +159,22 @@ sum by (reason) (rate(headroom_compression_failed_total[5m]))
|
||||
sum by (provider) (rate(headroom_requests_by_provider[5m]))
|
||||
```
|
||||
|
||||
> **Migration — these two counters gained labels.** `headroom_requests_failed_total` is now
|
||||
> labelled by `provider` and `headroom_requests_rate_limited_total` by `source`, so an
|
||||
> unaggregated query that used to return one series now returns several, and a panel or
|
||||
> alert that graphed the bare counter will fan out into one line per label value.
|
||||
> `sum without (provider) (rate(headroom_requests_failed_total[5m]))` and
|
||||
> `sum without (source) (rate(headroom_requests_rate_limited_total[5m]))` reproduce the old
|
||||
> single-series behaviour exactly. Both `source` series are exported from process start
|
||||
> (including at zero); `failed` exports one series per provider that has actually failed, so
|
||||
> it has no series at all until the first failure.
|
||||
>
|
||||
> The same split is available outside Prometheus: `/stats` carries
|
||||
> `requests.rate_limited_by_source` and `requests.failed_by_provider` alongside the
|
||||
> unlabelled `requests.rate_limited` / `requests.failed` totals (which are unchanged), the
|
||||
> lifetime aggregate carries the same two maps, and the OTel counters
|
||||
> `headroom.proxy.requests.rate_limited` / `.failed` carry `source` / `provider` attributes.
|
||||
|
||||
---
|
||||
|
||||
## Anthropic subscription panel
|
||||
|
||||
@@ -210,7 +210,7 @@
|
||||
<div class="grid grid-cols-1 gap-4 mb-6 lg:grid-cols-2">
|
||||
<section class="bg-surface rounded-lg p-4 border border-border">
|
||||
<h3 class="mb-3 text-sm font-medium text-gray-300">Request Health</h3>
|
||||
<div class="grid grid-cols-2 gap-3 text-sm"><div>Completed <span class="float-right tabular-nums" x-text="formatNumber(stats.requests?.total || 0)"></span></div><div>Failed <span class="float-right tabular-nums text-red-400" x-text="formatNumber(stats.requests?.failed || 0)"></span></div><div>Rate Limited <span class="float-right tabular-nums text-amber-400" x-text="formatNumber(stats.requests?.rate_limited || 0)"></span></div><div>Cached <span class="float-right tabular-nums text-cyan-400" x-text="formatNumber(stats.requests?.cached || 0)"></span></div></div>
|
||||
<div class="grid grid-cols-2 gap-3 text-sm"><div>Completed <span class="float-right tabular-nums" x-text="formatNumber(stats.requests?.total || 0)"></span></div><div title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed <span class="float-right tabular-nums text-red-400" x-text="formatNumber(stats.requests?.failed || 0)"></span></div><div title="Requests rejected with 429 — by Headroom's own rate limiter or by the upstream provider. /stats splits them under requests.rate_limited_by_source.">Rate Limited <span class="float-right tabular-nums text-amber-400" x-text="formatNumber(stats.requests?.rate_limited || 0)"></span></div><div>Cached <span class="float-right tabular-nums text-cyan-400" x-text="formatNumber(stats.requests?.cached || 0)"></span></div></div>
|
||||
</section>
|
||||
<section class="bg-surface rounded-lg p-4 border border-border">
|
||||
<h3 class="mb-3 text-sm font-medium text-gray-300">Live Activity</h3>
|
||||
@@ -418,7 +418,7 @@
|
||||
<span class="font-mono text-sm" x-text="((stats.ttfb?.min_ms || 0) / 1000).toFixed(2) + ' - ' + ((stats.ttfb?.max_ms || 0) / 1000).toFixed(2) + 's'"></span>
|
||||
</div>
|
||||
<div class="flex justify-between items-center">
|
||||
<span class="text-sm text-gray-400">Failed Requests</span>
|
||||
<span class="text-sm text-gray-400" title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed Requests</span>
|
||||
<span class="font-mono text-sm" x-text="stats.requests?.failed || 0"></span>
|
||||
</div>
|
||||
<!-- Per-transform timing breakdown -->
|
||||
@@ -1470,8 +1470,8 @@
|
||||
</div>
|
||||
<div class="grid grid-cols-2 gap-4 mb-6 md:grid-cols-4">
|
||||
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Requests</div><div class="text-2xl tabular-nums" x-text="formatNumber(lifetimeStats.requests?.total || 0)"></div></div>
|
||||
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Failed</div><div class="text-2xl tabular-nums text-red-400" x-text="formatNumber(lifetimeStats.requests?.failed || 0)"></div></div>
|
||||
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Rate Limited</div><div class="text-2xl tabular-nums text-amber-400" x-text="formatNumber(lifetimeStats.requests?.rate_limited || 0)"></div></div>
|
||||
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500" title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed</div><div class="text-2xl tabular-nums text-red-400" x-text="formatNumber(lifetimeStats.requests?.failed || 0)"></div></div>
|
||||
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500" title="Requests rejected with 429 — by Headroom's own rate limiter or by the upstream provider. /stats splits them under requests.rate_limited_by_source.">Rate Limited</div><div class="text-2xl tabular-nums text-amber-400" x-text="formatNumber(lifetimeStats.requests?.rate_limited || 0)"></div></div>
|
||||
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Cached</div><div class="text-2xl tabular-nums text-cyan-400" x-text="formatNumber(lifetimeStats.requests?.cached || 0)"></div></div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -207,12 +207,15 @@ class HeadroomOtelMetrics:
|
||||
)
|
||||
self._proxy_failed_requests = self._meter.create_counter(
|
||||
"headroom.proxy.requests.failed",
|
||||
description="Proxy requests that failed.",
|
||||
description="Proxy requests that failed upstream (4xx and 5xx, excluding 429).",
|
||||
unit="1",
|
||||
)
|
||||
self._proxy_rate_limited_requests = self._meter.create_counter(
|
||||
"headroom.proxy.requests.rate_limited",
|
||||
description="Proxy requests rejected by rate limiting.",
|
||||
description=(
|
||||
"Proxy requests rejected with 429, by source "
|
||||
"(headroom=our own limiter, upstream=the provider)."
|
||||
),
|
||||
unit="1",
|
||||
)
|
||||
self._proxy_input_tokens = self._meter.create_counter(
|
||||
@@ -567,8 +570,15 @@ class HeadroomOtelMetrics:
|
||||
*,
|
||||
provider: str | None = None,
|
||||
model: str | None = None,
|
||||
source: str = "headroom",
|
||||
) -> None:
|
||||
self._proxy_rate_limited_requests.add(1, self._attrs(provider=provider, model=model))
|
||||
# ``source`` mirrors the headroom_requests_rate_limited_total{source}
|
||||
# Prometheus label: "headroom" = our own limiter refused the request,
|
||||
# "upstream" = the provider did. Same split in both backends so an
|
||||
# operator's query means the same thing whichever one they scrape.
|
||||
self._proxy_rate_limited_requests.add(
|
||||
1, self._attrs(provider=provider, model=model, source=source)
|
||||
)
|
||||
|
||||
def record_proxy_cache_bust(self, *, tokens_lost: int) -> None:
|
||||
self._proxy_cache_busts.add(1)
|
||||
|
||||
@@ -1253,7 +1253,9 @@ class AnthropicHandlerMixin:
|
||||
rate_key = f"{api_key[:16]}:{client_ip}" if api_key else client_ip
|
||||
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
|
||||
if not allowed:
|
||||
await self.metrics.record_rate_limited(provider=provider_name)
|
||||
await self.metrics.record_rate_limited(
|
||||
provider=provider_name, source="headroom"
|
||||
)
|
||||
# Unit 4: release the pre-upstream semaphore before we
|
||||
# bail out of the handler via HTTPException — FastAPI's
|
||||
# exception handler will NOT run our ``finally``.
|
||||
|
||||
@@ -402,7 +402,7 @@ class GeminiHandlerMixin:
|
||||
rate_key = headers.get("x-goog-api-key", "default")[:20]
|
||||
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
|
||||
if not allowed:
|
||||
await self.metrics.record_rate_limited(provider=provider_name)
|
||||
await self.metrics.record_rate_limited(provider=provider_name, source="headroom")
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
|
||||
|
||||
@@ -3551,7 +3551,9 @@ class OpenAIHandlerMixin:
|
||||
rate_key = headers.get("authorization", "default")[:20]
|
||||
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
|
||||
if not allowed:
|
||||
await self.metrics.record_rate_limited(provider=openai_chat_outcome_provider)
|
||||
await self.metrics.record_rate_limited(
|
||||
provider=openai_chat_outcome_provider, source="headroom"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
|
||||
@@ -5676,7 +5678,7 @@ class OpenAIHandlerMixin:
|
||||
rate_key = headers.get("authorization", "default")[:20]
|
||||
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
|
||||
if not allowed:
|
||||
await self.metrics.record_rate_limited(provider="openai")
|
||||
await self.metrics.record_rate_limited(provider="openai", source="headroom")
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
|
||||
|
||||
@@ -531,9 +531,16 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
# limit is the one 4xx a user is expected to act on (back off, raise a cap),
|
||||
# and folding it into a generic failure count hides exactly that. Both
|
||||
# counters are already exported and neither feeds savings.
|
||||
#
|
||||
# source="upstream": this funnel only ever sees a 429 the PROVIDER returned.
|
||||
# Headroom's own limiter rejects before a request is ever sent and records
|
||||
# source="headroom" from the handler. The two are acted on differently —
|
||||
# raise our cap vs. back off / shard keys — so they must stay separable
|
||||
# (issue #3696). ``outcome.provider`` is the already-Copilot-relabelled
|
||||
# value, matching every other provider-labelled metric on this path.
|
||||
if outcome.status_code >= 400:
|
||||
if outcome.status_code == 429:
|
||||
await handler.metrics.record_rate_limited(provider=outcome.provider)
|
||||
await handler.metrics.record_rate_limited(provider=outcome.provider, source="upstream")
|
||||
else:
|
||||
await handler.metrics.record_failed(provider=outcome.provider)
|
||||
return
|
||||
|
||||
@@ -16,6 +16,9 @@ MAX_EXPOSED_MODELS = 100
|
||||
MAX_LABEL_LENGTH = 128
|
||||
|
||||
KNOWN_MISS_REASONS = frozenset({"ttl_expiry", "prefix_change", "unknown"})
|
||||
# Mirrors RATE_LIMIT_SOURCES in prometheus_metrics: "headroom" is our own
|
||||
# limiter refusing the request, "upstream" is the provider refusing it.
|
||||
RATE_LIMIT_SOURCES = frozenset({"headroom", "upstream"})
|
||||
# Names must match ``WasteSignals.to_dict()`` in headroom/config.py — the
|
||||
# parser emits ``json_bloat``; an allowlist that says ``json_noise`` silently
|
||||
# shoves the largest waste category into the catch-all bucket.
|
||||
@@ -91,6 +94,12 @@ def _empty_state() -> dict[str, Any]:
|
||||
"cached": 0,
|
||||
"failed": 0,
|
||||
"rate_limited": 0,
|
||||
# Who produced the failure / who refused the request. The totals
|
||||
# above stay the unlabelled figures the dashboard reads; these
|
||||
# split them the same way the Prometheus labels do, so no backend
|
||||
# reports a differently-shaped counter (issue #3696).
|
||||
"failed_by_provider": {},
|
||||
"rate_limited_by_source": {},
|
||||
"by_provider": {},
|
||||
"by_stack": {},
|
||||
},
|
||||
@@ -182,6 +191,17 @@ class PersistentMetricsState:
|
||||
result["requests"]["by_provider"] = self._normalize_count_map(
|
||||
raw_requests.get("by_provider"), MAX_PROVIDER_VALUES
|
||||
)
|
||||
result["requests"]["failed_by_provider"] = self._normalize_count_map(
|
||||
raw_requests.get("failed_by_provider"), MAX_PROVIDER_VALUES
|
||||
)
|
||||
# A state file written before #3696 has no split, and back-filling the
|
||||
# legacy total into one bucket would invent history we never observed —
|
||||
# so it loads empty and only new events populate it. The unlabelled
|
||||
# ``rate_limited`` total is untouched and stays comparable across the
|
||||
# upgrade.
|
||||
result["requests"]["rate_limited_by_source"] = self._normalize_enum_map(
|
||||
raw_requests.get("rate_limited_by_source"), RATE_LIMIT_SOURCES, fallback="headroom"
|
||||
)
|
||||
result["requests"]["by_stack"] = self._normalize_count_map(
|
||||
raw_requests.get("by_stack"), MAX_STACK_VALUES
|
||||
)
|
||||
@@ -474,12 +494,29 @@ class PersistentMetricsState:
|
||||
|
||||
self._record_activity()
|
||||
self._state["requests"]["failed"] += 1
|
||||
self._increment_count(
|
||||
self._state["requests"]["failed_by_provider"], _label(provider), MAX_PROVIDER_VALUES
|
||||
)
|
||||
|
||||
def record_rate_limited(self, *, provider: str | None = None, model: str | None = None) -> None:
|
||||
"""Record a rate-limited request without redefining total request semantics."""
|
||||
def record_rate_limited(
|
||||
self,
|
||||
*,
|
||||
provider: str | None = None,
|
||||
model: str | None = None,
|
||||
source: str = "headroom",
|
||||
) -> None:
|
||||
"""Record a rate-limited request without redefining total request semantics.
|
||||
|
||||
``source`` is ``headroom`` (our own limiter) or ``upstream`` (the
|
||||
provider's 429); anything else is clamped to ``headroom``, which is what
|
||||
this counter meant before upstream 429s started reaching it.
|
||||
"""
|
||||
|
||||
self._record_activity()
|
||||
self._state["requests"]["rate_limited"] += 1
|
||||
bucket = source if source in RATE_LIMIT_SOURCES else "headroom"
|
||||
by_source = self._state["requests"]["rate_limited_by_source"]
|
||||
by_source[bucket] = by_source.get(bucket, 0) + 1
|
||||
|
||||
def record_cache_bust(self, *, tokens_lost: Any = 0) -> None:
|
||||
self._record_activity()
|
||||
|
||||
@@ -30,6 +30,33 @@ logger = logging.getLogger("headroom.proxy")
|
||||
# client-supplied model cardinality stays bounded (see record_request).
|
||||
_OTHER_MODEL = "other"
|
||||
|
||||
# Closed label set for headroom_requests_rate_limited_total{source}. Two 429s
|
||||
# mean opposite things to an operator: "headroom" is OUR limiter refusing the
|
||||
# request (raise the cap), "upstream" is the provider refusing it (back off or
|
||||
# shard keys). Before #3615 only the first could ever increment this counter;
|
||||
# the outcome funnel now routes provider 429s here too, so the split has to be
|
||||
# queryable instead of silently merged. Closed set => bounded cardinality, and
|
||||
# both series are exported from startup so a rate() never goes from absent to
|
||||
# present mid-incident.
|
||||
RATE_LIMIT_SOURCE_HEADROOM = "headroom"
|
||||
RATE_LIMIT_SOURCE_UPSTREAM = "upstream"
|
||||
RATE_LIMIT_SOURCES = (RATE_LIMIT_SOURCE_HEADROOM, RATE_LIMIT_SOURCE_UPSTREAM)
|
||||
|
||||
# Bucket for a failure with no attributed provider. Also seeded at zero so
|
||||
# headroom_requests_failed_total always exports at least one sample.
|
||||
_PROVIDER_UNKNOWN = "unknown"
|
||||
|
||||
|
||||
def _rate_limit_source(source: str | None) -> str:
|
||||
"""Clamp ``source`` to the closed label set, defaulting to Headroom's limiter.
|
||||
|
||||
Defaults to ``headroom`` because that is what this counter meant for its
|
||||
whole life before the outcome funnel started feeding it upstream 429s: an
|
||||
un-updated caller keeps the historical reading rather than inventing a new
|
||||
label value.
|
||||
"""
|
||||
return source if source in RATE_LIMIT_SOURCES else RATE_LIMIT_SOURCE_HEADROOM
|
||||
|
||||
|
||||
def _escape_label_value(value: str) -> str:
|
||||
# The /metrics body is emitted whole with .encode("utf-8") (server.py). A
|
||||
@@ -100,7 +127,26 @@ class PrometheusMetrics:
|
||||
self.requests_by_stack: dict[str, int] = defaultdict(int)
|
||||
self.requests_cached = 0
|
||||
self.requests_rate_limited = 0
|
||||
# Seeded with both sources at zero so /metrics exports the full series
|
||||
# set from the first scrape; ``requests_rate_limited`` stays the
|
||||
# unlabelled total that /stats and the session summary read.
|
||||
self.requests_rate_limited_by_source: dict[str, int] = dict.fromkeys(RATE_LIMIT_SOURCES, 0)
|
||||
self.requests_failed = 0
|
||||
# Per-provider failure attribution. #3615 moved 4xx out of the success
|
||||
# funnel, which also took them out of ``requests_by_provider`` (only
|
||||
# record_request touches that), leaving Prometheus with no way to tell
|
||||
# which upstream was failing. Kept alongside the unlabelled total.
|
||||
#
|
||||
# Seeded with the "unknown" bucket at zero so the metric always exports
|
||||
# at least one sample. Before this counter carried a label it was always
|
||||
# present as a bare ``headroom_requests_failed_total 0``; a labelled map
|
||||
# that starts empty would emit NO sample on a healthy proxy, which turns
|
||||
# the documented failure-rate query into an empty vector (an empty
|
||||
# numerator makes the whole expression empty, so the panel reads
|
||||
# "No data" instead of 0%) and would make ``absent()`` alerts fire on
|
||||
# healthy proxies. "unknown" is a real bucket -- record_failed uses it
|
||||
# when no provider is attributed -- so seeding it invents no provider.
|
||||
self.requests_failed_by_provider: dict[str, int] = defaultdict(int, {_PROVIDER_UNKNOWN: 0})
|
||||
self.inbound_requests_total = 0
|
||||
self.inbound_requests_completed = 0
|
||||
self.inbound_requests_active = 0
|
||||
@@ -343,7 +389,11 @@ class PrometheusMetrics:
|
||||
self.requests_by_stack.clear()
|
||||
self.requests_cached = 0
|
||||
self.requests_rate_limited = 0
|
||||
self.requests_rate_limited_by_source = dict.fromkeys(RATE_LIMIT_SOURCES, 0)
|
||||
self.requests_failed = 0
|
||||
self.requests_failed_by_provider.clear()
|
||||
# Re-seed so /metrics keeps exporting a sample after a reset.
|
||||
self.requests_failed_by_provider[_PROVIDER_UNKNOWN] = 0
|
||||
self.inbound_requests_total = 0
|
||||
self.inbound_requests_completed = 0
|
||||
self.inbound_requests_active = 0
|
||||
@@ -1175,15 +1225,38 @@ class PrometheusMetrics:
|
||||
if ms_val > self.ws_session_duration_max_ms[cause]:
|
||||
self.ws_session_duration_max_ms[cause] = ms_val
|
||||
|
||||
async def record_rate_limited(self, *, provider: str | None = None, model: str | None = None):
|
||||
async def record_rate_limited(
|
||||
self,
|
||||
*,
|
||||
provider: str | None = None,
|
||||
model: str | None = None,
|
||||
source: str = RATE_LIMIT_SOURCE_HEADROOM,
|
||||
):
|
||||
"""Record a 429.
|
||||
|
||||
``source`` says WHO refused the request: ``headroom`` for our own
|
||||
limiter (the handlers, which raise HTTPException and never emit an
|
||||
outcome) or ``upstream`` for a provider 429 arriving through the
|
||||
outcome funnel. Threaded to every sink — Prometheus, lifetime/persistent
|
||||
and OTel — so no backend reports a differently-shaped counter.
|
||||
"""
|
||||
source = _rate_limit_source(source)
|
||||
async with self._lock:
|
||||
self.requests_rate_limited += 1
|
||||
self.savings_tracker.record_lifetime_rate_limited(provider=provider, model=model)
|
||||
self._get_otel_metrics().record_proxy_rate_limited(provider=provider, model=model)
|
||||
self.requests_rate_limited_by_source[source] = (
|
||||
self.requests_rate_limited_by_source.get(source, 0) + 1
|
||||
)
|
||||
self.savings_tracker.record_lifetime_rate_limited(
|
||||
provider=provider, model=model, source=source
|
||||
)
|
||||
self._get_otel_metrics().record_proxy_rate_limited(
|
||||
provider=provider, model=model, source=source
|
||||
)
|
||||
|
||||
async def record_failed(self, *, provider: str | None = None, model: str | None = None):
|
||||
async with self._lock:
|
||||
self.requests_failed += 1
|
||||
self.requests_failed_by_provider[provider or _PROVIDER_UNKNOWN] += 1
|
||||
self.savings_tracker.record_lifetime_failed(provider=provider, model=model)
|
||||
self._get_otel_metrics().record_proxy_failed(provider=provider, model=model)
|
||||
|
||||
@@ -1215,20 +1288,39 @@ class PrometheusMetrics:
|
||||
help_text="Cached request count",
|
||||
value=self.requests_cached,
|
||||
)
|
||||
_append_metric(
|
||||
lines,
|
||||
name="headroom_requests_rate_limited_total",
|
||||
metric_type="counter",
|
||||
help_text="Rate limited requests",
|
||||
value=self.requests_rate_limited,
|
||||
# Labelled series only — an unlabelled sample alongside these would
|
||||
# double-count under sum(). `sum without (source)` reproduces the
|
||||
# pre-label value exactly.
|
||||
lines.extend(
|
||||
[
|
||||
"# HELP headroom_requests_rate_limited_total Requests rejected with 429, "
|
||||
"by who rejected them (headroom=our own limiter, upstream=the provider)",
|
||||
"# TYPE headroom_requests_rate_limited_total counter",
|
||||
]
|
||||
)
|
||||
_append_metric(
|
||||
lines,
|
||||
name="headroom_requests_failed_total",
|
||||
metric_type="counter",
|
||||
help_text="Failed requests",
|
||||
value=self.requests_failed,
|
||||
for _source in RATE_LIMIT_SOURCES:
|
||||
_count = self.requests_rate_limited_by_source.get(_source, 0)
|
||||
lines.append(
|
||||
f'headroom_requests_rate_limited_total{{source="{_escape_label_value(_source)}"}}'
|
||||
f" {_count}"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# Failures carry the provider that produced them. `sum without
|
||||
# (provider)` reproduces the pre-label value.
|
||||
lines.extend(
|
||||
[
|
||||
"# HELP headroom_requests_failed_total Requests that failed upstream "
|
||||
"(4xx and 5xx, excluding 429), by provider",
|
||||
"# TYPE headroom_requests_failed_total counter",
|
||||
]
|
||||
)
|
||||
for _provider, _count in self.requests_failed_by_provider.items():
|
||||
lines.append(
|
||||
f'headroom_requests_failed_total{{provider="'
|
||||
f'{_escape_label_value(str(_provider))}"}} {_count}'
|
||||
)
|
||||
lines.append("")
|
||||
_append_metric(
|
||||
lines,
|
||||
name="headroom_inbound_requests_total",
|
||||
|
||||
@@ -1210,12 +1210,22 @@ class SavingsTracker:
|
||||
self._maybe_save_locked()
|
||||
|
||||
def record_lifetime_rate_limited(
|
||||
self, *, provider: str | None = None, model: str | None = None
|
||||
self,
|
||||
*,
|
||||
provider: str | None = None,
|
||||
model: str | None = None,
|
||||
source: str = "headroom",
|
||||
) -> None:
|
||||
"""Record a rate-limited proxy request without changing legacy history."""
|
||||
"""Record a rate-limited proxy request without changing legacy history.
|
||||
|
||||
``source`` distinguishes Headroom's own limiter from an upstream 429 —
|
||||
see ``PrometheusMetrics.record_rate_limited``.
|
||||
"""
|
||||
|
||||
with self._lock:
|
||||
self._persistent_metrics.record_rate_limited(provider=provider, model=model)
|
||||
self._persistent_metrics.record_rate_limited(
|
||||
provider=provider, model=model, source=source
|
||||
)
|
||||
self._maybe_save_locked()
|
||||
|
||||
def record_lifetime_cache_bust(self, *, tokens_lost: int) -> None:
|
||||
|
||||
@@ -2263,6 +2263,12 @@ class HeadroomProxy(
|
||||
logger.info(f"Total requests: {m.requests_total}")
|
||||
logger.info(f"Cached responses: {m.requests_cached}")
|
||||
logger.info(f"Rate limited: {m.requests_rate_limited}")
|
||||
if m.requests_rate_limited:
|
||||
# The split an operator acts on differently: our limiter firing
|
||||
# means raise the cap, the provider's means back off / shard keys.
|
||||
by_source = m.requests_rate_limited_by_source
|
||||
logger.info(f" headroom limiter: {by_source.get('headroom', 0)}")
|
||||
logger.info(f" upstream 429s: {by_source.get('upstream', 0)}")
|
||||
logger.info(f"Failed: {m.requests_failed}")
|
||||
logger.info(f"Input tokens: {m.tokens_input_total:,}")
|
||||
logger.info(f"Output tokens: {m.tokens_output_total:,}")
|
||||
@@ -4572,7 +4578,14 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
"total": m.requests_total,
|
||||
"cached": m.requests_cached,
|
||||
"rate_limited": m.requests_rate_limited,
|
||||
# Who issued the 429: "headroom" is our own limiter, "upstream"
|
||||
# is the provider (issue #3696). The totals above stay the
|
||||
# unlabelled figures existing consumers read.
|
||||
"rate_limited_by_source": dict(m.requests_rate_limited_by_source),
|
||||
"failed": m.requests_failed,
|
||||
"failed_by_provider": _remap_provider_counts(
|
||||
dict(m.requests_failed_by_provider), proxy.config
|
||||
),
|
||||
"by_provider": _remap_provider_counts(dict(m.requests_by_provider), proxy.config),
|
||||
"by_model": dict(m.requests_by_model),
|
||||
"by_stack": dict(m.requests_by_stack),
|
||||
|
||||
@@ -356,11 +356,56 @@ async def test_prometheus_metrics_reads_late_configured_otel_metrics() -> None:
|
||||
await metrics.record_rate_limited(provider="anthropic", model="claude-sonnet")
|
||||
|
||||
assert spy.failed_calls == [{"provider": "openai", "model": None}]
|
||||
assert spy.rate_limited_calls == [{"provider": "anthropic", "model": "claude-sonnet"}]
|
||||
# ``source`` reaches OTel too — the split must not exist in Prometheus only.
|
||||
assert spy.rate_limited_calls == [
|
||||
{"provider": "anthropic", "model": "claude-sonnet", "source": "headroom"}
|
||||
]
|
||||
finally:
|
||||
reset_otel_metrics()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prometheus_metrics_forwards_rate_limit_source_to_otel() -> None:
|
||||
"""An upstream 429 must reach OTel labelled as upstream, not as our own limiter."""
|
||||
|
||||
spy = _SpyProxyMetrics()
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
set_otel_metrics(spy) # type: ignore[arg-type]
|
||||
|
||||
try:
|
||||
await metrics.record_rate_limited(provider="anthropic", source="upstream")
|
||||
await metrics.record_rate_limited(provider="anthropic", source="headroom")
|
||||
# An unrecognised value is clamped rather than exported as a new label.
|
||||
await metrics.record_rate_limited(provider="anthropic", source="nonsense")
|
||||
|
||||
assert [call["source"] for call in spy.rate_limited_calls] == [
|
||||
"upstream",
|
||||
"headroom",
|
||||
"headroom",
|
||||
]
|
||||
finally:
|
||||
reset_otel_metrics()
|
||||
|
||||
|
||||
def test_otel_rate_limited_counter_carries_source_attribute() -> None:
|
||||
reader = InMemoryMetricReader()
|
||||
provider = MeterProvider(metric_readers=[reader])
|
||||
otel_metrics = HeadroomOtelMetrics(meter_provider=provider)
|
||||
|
||||
otel_metrics.record_proxy_rate_limited(provider="anthropic", source="upstream")
|
||||
otel_metrics.record_proxy_rate_limited(provider="openai", source="headroom")
|
||||
|
||||
metrics = _collect_metrics(reader)
|
||||
upstream = _find_point(
|
||||
metrics["headroom.proxy.requests.rate_limited"], provider="anthropic", source="upstream"
|
||||
)
|
||||
headroom_side = _find_point(
|
||||
metrics["headroom.proxy.requests.rate_limited"], provider="openai", source="headroom"
|
||||
)
|
||||
assert upstream.value == 1
|
||||
assert headroom_side.value == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prometheus_metrics_forwards_savings_drilldown_fields_to_otel(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
|
||||
@@ -25,13 +25,19 @@ class _Metrics:
|
||||
def __init__(self):
|
||||
self.failed = []
|
||||
self.rate_limited = []
|
||||
self.rate_limited_sources = []
|
||||
self.requested = []
|
||||
|
||||
async def record_failed(self, provider):
|
||||
self.failed.append(provider)
|
||||
|
||||
async def record_rate_limited(self, provider):
|
||||
# ``source`` deliberately has NO default here: the funnel only ever sees
|
||||
# 429s the provider returned, so it must pass source="upstream" explicitly.
|
||||
# If it ever stops doing so, these tests record None and fail loudly rather
|
||||
# than silently re-merging upstream throttling into Headroom's own counter.
|
||||
async def record_rate_limited(self, provider, source=None):
|
||||
self.rate_limited.append(provider)
|
||||
self.rate_limited_sources.append(source)
|
||||
|
||||
async def record_request(self, **kwargs):
|
||||
self.requested.append(kwargs)
|
||||
@@ -78,6 +84,11 @@ def test_429_recorded_as_rate_limited_and_skips_success_funnel():
|
||||
handler = _Handler()
|
||||
asyncio.run(emit_request_outcome(handler, _outcome(429, tokens_saved=6380)))
|
||||
assert handler.metrics.rate_limited == ["anthropic"]
|
||||
# Labelled as the PROVIDER's 429, not ours. Headroom's own limiter rejects
|
||||
# before a request is ever sent and records source="headroom" from the
|
||||
# handler; merging the two would tell an operator to raise a cap that is
|
||||
# not the one being hit (issue #3696).
|
||||
assert handler.metrics.rate_limited_sources == ["upstream"]
|
||||
assert handler.metrics.failed == [] # a rate limit is not a generic failure
|
||||
assert handler.metrics.requested == [] # its 6,380 "saved" tokens are not savings
|
||||
|
||||
|
||||
@@ -50,6 +50,9 @@ def test_snapshot_accumulates_request_token_cache_cost_and_waste_metrics() -> No
|
||||
"cached": 1,
|
||||
"failed": 1,
|
||||
"rate_limited": 1,
|
||||
# Lifetime carries the same splits the Prometheus labels do.
|
||||
"failed_by_provider": {"anthropic": 1},
|
||||
"rate_limited_by_source": {"headroom": 1},
|
||||
"by_provider": {"anthropic": 1},
|
||||
"by_stack": {"codex": 1},
|
||||
}
|
||||
@@ -205,3 +208,56 @@ def test_miss_reasons_still_fall_back_to_unknown_on_reload() -> None:
|
||||
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
|
||||
|
||||
assert snapshot["prefix_cache"]["misses_by_reason"] == {"ttl_expiry": 2, "unknown": 1}
|
||||
|
||||
|
||||
def test_rate_limited_is_split_by_source_and_failures_by_provider() -> None:
|
||||
"""Headroom's own limiter and an upstream 429 must stay distinguishable.
|
||||
|
||||
Both land in the same ``rate_limited`` total (unchanged), but an operator
|
||||
acts on them differently: ours firing means raise the cap, the provider's
|
||||
means back off. See issue #3696.
|
||||
"""
|
||||
state = _new_state()
|
||||
|
||||
state.record_rate_limited(provider="anthropic", source="headroom")
|
||||
state.record_rate_limited(provider="anthropic", source="upstream")
|
||||
state.record_rate_limited(provider="openai", source="upstream")
|
||||
state.record_failed(provider="anthropic")
|
||||
state.record_failed(provider="openai")
|
||||
state.record_failed(provider="openai")
|
||||
|
||||
requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
|
||||
|
||||
assert requests["rate_limited"] == 3
|
||||
assert requests["rate_limited_by_source"] == {"headroom": 1, "upstream": 2}
|
||||
assert requests["failed"] == 3
|
||||
assert requests["failed_by_provider"] == {"anthropic": 1, "openai": 2}
|
||||
|
||||
|
||||
def test_rate_limit_source_defaults_to_headroom_and_clamps_unknown_values() -> None:
|
||||
state = _new_state()
|
||||
|
||||
state.record_rate_limited(provider="anthropic")
|
||||
state.record_rate_limited(provider="anthropic", source="bogus")
|
||||
|
||||
requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
|
||||
assert requests["rate_limited_by_source"] == {"headroom": 2}
|
||||
|
||||
|
||||
def test_pre_3696_state_loads_without_backfilling_invented_history() -> None:
|
||||
"""A state file written before the split keeps its totals and starts the maps empty.
|
||||
|
||||
Back-filling the legacy ``rate_limited`` total into one bucket would claim we
|
||||
observed a split we never recorded.
|
||||
"""
|
||||
state = PersistentMetricsState(
|
||||
{"requests": {"total": 10, "failed": 4, "rate_limited": 7}},
|
||||
now=lambda: FIXED_NOW,
|
||||
)
|
||||
|
||||
requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
|
||||
|
||||
assert requests["rate_limited"] == 7
|
||||
assert requests["failed"] == 4
|
||||
assert requests["rate_limited_by_source"] == {}
|
||||
assert requests["failed_by_provider"] == {}
|
||||
|
||||
@@ -39,6 +39,10 @@ def test_runtime_metric_events_feed_lifetime_without_resetting_runtime_counters(
|
||||
assert lifetime["requests"]["cached"] == 1
|
||||
assert lifetime["requests"]["failed"] == 1
|
||||
assert lifetime["requests"]["rate_limited"] == 1
|
||||
# The Prometheus labels must not stop at Prometheus: lifetime carries the
|
||||
# same two splits (issue #3696). No explicit source here, so it is ours.
|
||||
assert lifetime["requests"]["failed_by_provider"] == {"anthropic": 1}
|
||||
assert lifetime["requests"]["rate_limited_by_source"] == {"headroom": 1}
|
||||
assert lifetime["requests"]["by_provider"] == {"anthropic": 1}
|
||||
assert lifetime["requests"]["by_stack"] == {"codex": 1}
|
||||
assert lifetime["tokens"]["output"] == 3
|
||||
@@ -47,3 +51,27 @@ def test_runtime_metric_events_feed_lifetime_without_resetting_runtime_counters(
|
||||
assert lifetime["prefix_cache"]["misses_by_reason"] == {"prefix_change": 1}
|
||||
assert lifetime["waste_signals"] == {"repetition": 4}
|
||||
assert metrics.requests_total == 1
|
||||
|
||||
|
||||
def test_upstream_rate_limit_reaches_lifetime_labelled_upstream(tmp_path) -> None:
|
||||
"""An upstream 429 must not be filed under Headroom's own limiter in lifetime.
|
||||
|
||||
The outcome funnel passes ``source="upstream"``; the whole chain
|
||||
(PrometheusMetrics -> SavingsTracker -> PersistentMetricsState) has to carry
|
||||
it, or the durable view re-merges what Prometheus splits (issue #3696).
|
||||
"""
|
||||
tracker = SavingsTracker(path=str(tmp_path / "proxy_savings.json"), save_flush_every=25)
|
||||
metrics = PrometheusMetrics(savings_tracker=tracker)
|
||||
|
||||
asyncio.run(metrics.record_rate_limited(provider="anthropic", source="upstream"))
|
||||
asyncio.run(metrics.record_rate_limited(provider="anthropic", source="headroom"))
|
||||
asyncio.run(metrics.record_failed(provider="openai"))
|
||||
|
||||
lifetime = tracker.lifetime_response()
|
||||
|
||||
assert lifetime["requests"]["rate_limited"] == 2
|
||||
assert lifetime["requests"]["rate_limited_by_source"] == {"upstream": 1, "headroom": 1}
|
||||
assert lifetime["requests"]["failed_by_provider"] == {"openai": 1}
|
||||
# Neither counter may touch the completed-request denominator.
|
||||
assert lifetime["requests"]["total"] == 0
|
||||
assert metrics.requests_total == 0
|
||||
|
||||
@@ -176,3 +176,130 @@ async def test_gate_counter_is_thread_safe_under_concurrent_export() -> None:
|
||||
assert not errors, f"export() raced the writer: {errors[:3]}"
|
||||
totals = dict(metrics.kompress_size_gate_by_outcome)
|
||||
assert sum(totals.values()) == n_threads * per_thread
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# headroom_requests_rate_limited_total{source} / headroom_requests_failed_total{provider}
|
||||
#
|
||||
# Issue #3696: #3615 routed upstream 429s and 4xx into these two counters, which
|
||||
# silently changed what they mean. The labels make the new meaning queryable —
|
||||
# "my limiter throttled me" vs "the provider throttled me" are acted on
|
||||
# differently, and a failure with no provider attached names no culprit.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rate_limited_export_splits_headroom_from_upstream() -> None:
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
|
||||
# What the four handler sites do when Headroom's OWN limiter rejects.
|
||||
await metrics.record_rate_limited(provider="anthropic", source="headroom")
|
||||
# What the outcome funnel does with a provider 429.
|
||||
await metrics.record_rate_limited(provider="anthropic", source="upstream")
|
||||
await metrics.record_rate_limited(provider="openai", source="upstream")
|
||||
|
||||
text = await metrics.export()
|
||||
|
||||
assert "# TYPE headroom_requests_rate_limited_total counter" in text
|
||||
assert 'headroom_requests_rate_limited_total{source="headroom"} 1' in text
|
||||
assert 'headroom_requests_rate_limited_total{source="upstream"} 2' in text
|
||||
# `sum without (source)` must reproduce the pre-label value.
|
||||
assert metrics.requests_rate_limited == 3
|
||||
# No unlabelled sample alongside the labelled ones — that would double-count.
|
||||
assert "\nheadroom_requests_rate_limited_total 3" not in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rate_limited_source_series_exist_before_any_429() -> None:
|
||||
"""Both series export from startup, so a rate() never appears mid-incident."""
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
|
||||
text = await metrics.export()
|
||||
|
||||
assert 'headroom_requests_rate_limited_total{source="headroom"} 0' in text
|
||||
assert 'headroom_requests_rate_limited_total{source="upstream"} 0' in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_rate_limited_source_defaults_to_headroom_and_clamps_unknown() -> None:
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
|
||||
await metrics.record_rate_limited(provider="anthropic")
|
||||
await metrics.record_rate_limited(provider="anthropic", source="bogus")
|
||||
|
||||
text = await metrics.export()
|
||||
|
||||
assert 'headroom_requests_rate_limited_total{source="headroom"} 2' in text
|
||||
assert 'headroom_requests_rate_limited_total{source="upstream"} 0' in text
|
||||
assert "bogus" not in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_export_is_attributed_to_the_provider_that_failed() -> None:
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
|
||||
await metrics.record_failed(provider="anthropic")
|
||||
await metrics.record_failed(provider="openai")
|
||||
await metrics.record_failed(provider="openai")
|
||||
|
||||
text = await metrics.export()
|
||||
|
||||
assert "# TYPE headroom_requests_failed_total counter" in text
|
||||
assert 'headroom_requests_failed_total{provider="anthropic"} 1' in text
|
||||
assert 'headroom_requests_failed_total{provider="openai"} 2' in text
|
||||
# `sum without (provider)` must reproduce the pre-label value.
|
||||
assert metrics.requests_failed == 3
|
||||
assert "\nheadroom_requests_failed_total 3" not in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_without_a_provider_lands_in_unknown() -> None:
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
|
||||
await metrics.record_failed()
|
||||
|
||||
text = await metrics.export()
|
||||
|
||||
assert 'headroom_requests_failed_total{provider="unknown"} 1' in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reset_runtime_clears_rate_limit_and_failure_splits() -> None:
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
|
||||
await metrics.record_rate_limited(provider="anthropic", source="upstream")
|
||||
await metrics.record_failed(provider="anthropic")
|
||||
|
||||
await metrics.reset_runtime()
|
||||
|
||||
assert metrics.requests_rate_limited == 0
|
||||
assert metrics.requests_failed == 0
|
||||
assert metrics.requests_rate_limited_by_source == {"headroom": 0, "upstream": 0}
|
||||
# Re-seeded, not emptied: the metric must keep exporting a sample after a
|
||||
# reset, same as the rate-limit source series above.
|
||||
assert dict(metrics.requests_failed_by_provider) == {"unknown": 0}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_total_exports_a_sample_before_any_failure() -> None:
|
||||
"""A healthy proxy must still export ``headroom_requests_failed_total``.
|
||||
|
||||
Before the provider label this counter was always present as a bare
|
||||
``headroom_requests_failed_total 0``. A labelled map that starts empty
|
||||
emits no sample at all, which makes the documented failure-rate query
|
||||
return an empty vector (an empty numerator empties the whole expression),
|
||||
so the panel reads "No data" instead of 0% while everything is fine --
|
||||
and `absent()` alerts fire on healthy proxies.
|
||||
"""
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
|
||||
text = await metrics.export()
|
||||
|
||||
assert 'headroom_requests_failed_total{provider="unknown"} 0' in text
|
||||
|
||||
# The seed must not survive as a duplicate once a real provider fails.
|
||||
await metrics.record_failed(provider="anthropic")
|
||||
text = await metrics.export()
|
||||
assert 'headroom_requests_failed_total{provider="anthropic"} 1' in text
|
||||
assert 'headroom_requests_failed_total{provider="unknown"} 0' in text
|
||||
assert metrics.requests_failed == 1
|
||||
|
||||
+11
-1
@@ -29,7 +29,9 @@ curl http://localhost:8787/stats
|
||||
"total": 42,
|
||||
"cached": 5,
|
||||
"rate_limited": 0,
|
||||
"failed": 0
|
||||
"rate_limited_by_source": { "headroom": 0, "upstream": 0 },
|
||||
"failed": 0,
|
||||
"failed_by_provider": {}
|
||||
},
|
||||
"tokens": {
|
||||
"input": 50000,
|
||||
@@ -48,6 +50,14 @@ curl http://localhost:8787/stats
|
||||
}
|
||||
```
|
||||
|
||||
`requests.total` is the **completed**-request count, not total traffic: a request that
|
||||
ended in a 4xx, 5xx or 429 is counted under `requests.failed` or `requests.rate_limited`
|
||||
instead and never reaches `total`. `rate_limited_by_source` splits 429s into
|
||||
`headroom` (Headroom's own rate limiter refused the request) and `upstream` (the provider
|
||||
refused it) — different problems, different fixes. `failed_by_provider` attributes failures
|
||||
to the upstream that produced them. The unlabelled `rate_limited` / `failed` totals are
|
||||
unchanged.
|
||||
|
||||
`/stats` keeps the existing live/session fields, including `savings_history`,
|
||||
for backward compatibility. The new `persistent_savings` block is durable local
|
||||
proxy compression history stored by default at
|
||||
|
||||
Reference in New Issue
Block a user