perf(metrics): cap inbound request-path cardinality to bound memory (#3340)

## Description

`PrometheusMetrics.record_inbound_request` incremented
`inbound_requests_by_path[path]` with no cardinality cap:

```python
self.inbound_requests_by_path[path] += 1
```

`path` is client-controlled (`request.url.path`, recorded in middleware
on **every** inbound request before routing), and ID-bearing passthrough
routes mint a distinct key per request: `/v1/files/{id}`,
`/v1/batches/{id}`, `/v1/responses/{id}`, model-scoped paths, plus
arbitrary 404 paths. `PrometheusMetrics` is the process-lifetime
singleton (`proxy.metrics`), so this dict — and the Prometheus series it
exports (`by_path` in `inbound_snapshot`) — grows without bound. That's
both a slow heap leak and a label-cardinality explosion.

The sibling counters in the same class already guard against exactly
this: `requests_by_model` buckets past `MAX_DISTINCT_MODELS` into
`"other"`, and `requests_by_stack` past `MAX_DISTINCT_STACKS`.
`inbound_requests_by_path` was simply missed.

This applies the same bounded-bucketing. Past `MAX_DISTINCT_PATHS` (256
— well above a proxy's real route count) further distinct paths collapse
into `"other"` via a membership test that never materializes an over-cap
key (a `defaultdict` index would defeat the cap), logging exactly one
warning. `reset_runtime` clears the dict and re-arms the warning.

Reproduction (before): 50,000 distinct paths → 50,000 dict keys. After:
257 keys (256 real + `"other"`), overflow counted in `"other"`, real
paths keep their own key and count.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/prometheus_metrics.py`: added `MAX_DISTINCT_PATHS =
256` and `_OTHER_PATH = "other"`; `record_inbound_request` now buckets
over-cap paths into `"other"` via a membership test and logs a one-shot
warning; added `self._path_cardinality_warned` (initialized in
`__init__`, re-armed and the dict cleared in `reset_runtime`).
- `tests/test_observability_metrics.py`: added three tests mirroring the
model-cap tests — the path dict is bounded to `MAX_DISTINCT_PATHS + 1`
with accounting preserved and a tracked path still incrementing; the cap
warns exactly once; `reset_runtime` clears the dict and re-arms the
warning.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_observability_metrics.py  ->  16 passed
uvx ruff@0.16.2 check headroom/proxy/prometheus_metrics.py tests/test_observability_metrics.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/proxy/prometheus_metrics.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: called `record_inbound_request` with 50,000
distinct ID-bearing paths; before the fix `len(inbound_requests_by_path)
== 50000` (unbounded), after the fix `== 257` (256 real + `"other"`)
with the overflow (49,744) counted in `"other"` and a known path like
`/v1/messages` keeping its own key. Ran the full
`tests/test_observability_metrics.py` suite.
- Observed result: the counter and its exported Prometheus series are
bounded regardless of client path cardinality; real routes are
unaffected below the cap.
- Not tested: a live long-running proxy heap-growth measurement (the
unbounded growth is demonstrated directly by the key-count reproduction
above).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only past 256 distinct inbound paths,
where the exported per-path series switches from unbounded distinct
labels to an `"other"` bucket. Below the cap, behavior is identical.
- Kill switch / disable path: N/A (raise `MAX_DISTINCT_PATHS` to relax).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; the path counter goes back to
unbounded.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal metric-cardinality bound; matches the existing model/stack
caps)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Mirrors the existing `MAX_DISTINCT_MODELS` / `MAX_DISTINCT_STACKS`
"other"-bucketing already in this class, so the cardinality-bounding
approach and its observability trade-off are consistent across the
counters.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Abhay Singh
2026-10-01 16:02:32 -05:00
committed by GitHub
co-authored by JerrettDavis
parent 746038991c
commit 88a1f4e52b
2 changed files with 84 additions and 2 deletions
+28 -1
View File
@@ -30,6 +30,15 @@ logger = logging.getLogger("headroom.proxy")
# client-supplied model cardinality stays bounded (see record_request).
_OTHER_MODEL = "other"
# Same idea for inbound request paths (see record_inbound_request). ``path`` is
# client-controlled and effectively unbounded — ID-bearing passthrough routes
# such as ``/v1/files/{id}`` or ``/v1/responses/{id}`` mint a distinct key per
# request. Past this many distinct paths, further paths collapse into
# ``_OTHER_PATH`` so the counter (and its exported Prometheus series) can't grow
# without bound. The cap sits well above a proxy's real route count.
MAX_DISTINCT_PATHS = 256
_OTHER_PATH = "other"
# Closed label set for headroom_requests_rate_limited_total{source}. Two 429s
# mean opposite things to an operator: "headroom" is OUR limiter refusing the
# request (raise the cap), "upstream" is the provider refusing it (back off or
@@ -152,6 +161,7 @@ class PrometheusMetrics:
self.inbound_requests_active = 0
self.inbound_requests_by_method: dict[str, int] = defaultdict(int)
self.inbound_requests_by_path: dict[str, int] = defaultdict(int)
self._path_cardinality_warned = False
self.inbound_responses_by_status: dict[str, int] = defaultdict(int)
self.tokens_input_total = 0
@@ -413,6 +423,7 @@ class PrometheusMetrics:
self.inbound_requests_active = 0
self.inbound_requests_by_method.clear()
self.inbound_requests_by_path.clear()
self._path_cardinality_warned = False
self.inbound_responses_by_status.clear()
self.tokens_input_total = 0
@@ -773,7 +784,23 @@ class PrometheusMetrics:
self.inbound_requests_total += 1
self.inbound_requests_active += 1
self.inbound_requests_by_method[method.upper()] += 1
self.inbound_requests_by_path[path] += 1
# Cap client-controlled path cardinality, mirroring the model cap in
# record_request. A membership test (never a defaultdict index) keeps an
# over-cap path from materializing a new key and defeating the bound.
if path in self.inbound_requests_by_path or (
len(self.inbound_requests_by_path) < MAX_DISTINCT_PATHS
):
bounded_path = path
else:
bounded_path = _OTHER_PATH
if not self._path_cardinality_warned:
self._path_cardinality_warned = True
logger.warning(
"metrics.record: inbound path cardinality cap (%d) reached; "
'bucketing further paths into "other"',
MAX_DISTINCT_PATHS,
)
self.inbound_requests_by_path[bounded_path] += 1
def record_inbound_response(self, *, status_code: int | str) -> None:
self.inbound_requests_completed += 1
+56 -1
View File
@@ -18,7 +18,7 @@ from headroom.observability import (
set_otel_metrics,
unregister_otel_metric_attribute_provider,
)
from headroom.proxy.prometheus_metrics import PrometheusMetrics
from headroom.proxy.prometheus_metrics import MAX_DISTINCT_PATHS, PrometheusMetrics
from headroom.telemetry.context import MAX_DISTINCT_MODELS
from headroom.transforms.pipeline import TransformPipeline
@@ -552,6 +552,61 @@ async def test_prometheus_metrics_reset_rearms_cardinality_warning() -> None:
assert len(metrics._cache_requests_by_model) == 0
def test_prometheus_metrics_caps_inbound_path_cardinality() -> None:
"""A client hitting unbounded distinct paths (ID-bearing passthrough routes)
cannot grow inbound_requests_by_path past MAX_DISTINCT_PATHS + the "other"
sentinel, while the total request count stays exact."""
metrics = PrometheusMetrics(stateless=True)
# Fill exactly to the cap with distinct paths: no bucketing yet.
for i in range(MAX_DISTINCT_PATHS):
metrics.record_inbound_request(method="GET", path=f"/v1/files/file_{i}")
assert len(metrics.inbound_requests_by_path) == MAX_DISTINCT_PATHS
assert "other" not in metrics.inbound_requests_by_path
# New distinct paths past the cap bucket into "other", never their own key.
for i in range(5):
metrics.record_inbound_request(method="GET", path=f"/v1/responses/resp_{i}")
assert "/v1/responses/resp_0" not in metrics.inbound_requests_by_path
assert metrics.inbound_requests_by_path["other"] == 5
assert len(metrics.inbound_requests_by_path) == MAX_DISTINCT_PATHS + 1
# An already-tracked path keeps incrementing after the cap is reached.
metrics.record_inbound_request(method="GET", path="/v1/files/file_0")
assert metrics.inbound_requests_by_path["/v1/files/file_0"] == 2
# Accounting is preserved: every inbound request is counted somewhere.
total_calls = MAX_DISTINCT_PATHS + 5 + 1
assert metrics.inbound_requests_total == total_calls
assert sum(metrics.inbound_requests_by_path.values()) == total_calls
def test_prometheus_metrics_path_cardinality_warns_once(
caplog: pytest.LogCaptureFixture,
) -> None:
"""Bucketing inbound paths into "other" logs exactly one warning."""
metrics = PrometheusMetrics(stateless=True)
with caplog.at_level(logging.WARNING, logger="headroom.proxy"):
for i in range(MAX_DISTINCT_PATHS + 10):
metrics.record_inbound_request(method="GET", path=f"/v1/files/file_{i}")
cap_warnings = [r for r in caplog.records if "path cardinality cap" in r.getMessage()]
assert len(cap_warnings) == 1
@pytest.mark.asyncio
async def test_prometheus_metrics_reset_rearms_path_cardinality_warning() -> None:
"""reset_runtime clears the path dict and re-arms the one-shot cap warning."""
metrics = PrometheusMetrics(stateless=True)
for i in range(MAX_DISTINCT_PATHS + 5):
metrics.record_inbound_request(method="GET", path=f"/v1/files/file_{i}")
assert metrics._path_cardinality_warned is True
await metrics.reset_runtime()
assert metrics._path_cardinality_warned is False
assert len(metrics.inbound_requests_by_path) == 0
@pytest.mark.asyncio
async def test_prometheus_metrics_export_bounds_model_series() -> None:
"""export() emits at most MAX_DISTINCT_MODELS model series plus the 'other' bucket."""