mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
perf(metrics): cap inbound request-path cardinality to bound memory (#3340)
## Description
`PrometheusMetrics.record_inbound_request` incremented
`inbound_requests_by_path[path]` with no cardinality cap:
```python
self.inbound_requests_by_path[path] += 1
```
`path` is client-controlled (`request.url.path`, recorded in middleware
on **every** inbound request before routing), and ID-bearing passthrough
routes mint a distinct key per request: `/v1/files/{id}`,
`/v1/batches/{id}`, `/v1/responses/{id}`, model-scoped paths, plus
arbitrary 404 paths. `PrometheusMetrics` is the process-lifetime
singleton (`proxy.metrics`), so this dict — and the Prometheus series it
exports (`by_path` in `inbound_snapshot`) — grows without bound. That's
both a slow heap leak and a label-cardinality explosion.
The sibling counters in the same class already guard against exactly
this: `requests_by_model` buckets past `MAX_DISTINCT_MODELS` into
`"other"`, and `requests_by_stack` past `MAX_DISTINCT_STACKS`.
`inbound_requests_by_path` was simply missed.
This applies the same bounded-bucketing. Past `MAX_DISTINCT_PATHS` (256
— well above a proxy's real route count) further distinct paths collapse
into `"other"` via a membership test that never materializes an over-cap
key (a `defaultdict` index would defeat the cap), logging exactly one
warning. `reset_runtime` clears the dict and re-arms the warning.
Reproduction (before): 50,000 distinct paths → 50,000 dict keys. After:
257 keys (256 real + `"other"`), overflow counted in `"other"`, real
paths keep their own key and count.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/prometheus_metrics.py`: added `MAX_DISTINCT_PATHS =
256` and `_OTHER_PATH = "other"`; `record_inbound_request` now buckets
over-cap paths into `"other"` via a membership test and logs a one-shot
warning; added `self._path_cardinality_warned` (initialized in
`__init__`, re-armed and the dict cleared in `reset_runtime`).
- `tests/test_observability_metrics.py`: added three tests mirroring the
model-cap tests — the path dict is bounded to `MAX_DISTINCT_PATHS + 1`
with accounting preserved and a tracked path still incrementing; the cap
warns exactly once; `reset_runtime` clears the dict and re-arms the
warning.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_observability_metrics.py -> 16 passed
uvx ruff@0.16.2 check headroom/proxy/prometheus_metrics.py tests/test_observability_metrics.py -> All checks passed!
uvx mypy@1.20.2 headroom/proxy/prometheus_metrics.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: called `record_inbound_request` with 50,000
distinct ID-bearing paths; before the fix `len(inbound_requests_by_path)
== 50000` (unbounded), after the fix `== 257` (256 real + `"other"`)
with the overflow (49,744) counted in `"other"` and a known path like
`/v1/messages` keeping its own key. Ran the full
`tests/test_observability_metrics.py` suite.
- Observed result: the counter and its exported Prometheus series are
bounded regardless of client path cardinality; real routes are
unaffected below the cap.
- Not tested: a live long-running proxy heap-growth measurement (the
unbounded growth is demonstrated directly by the key-count reproduction
above).
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only past 256 distinct inbound paths,
where the exported per-path series switches from unbounded distinct
labels to an `"other"` bucket. Below the cap, behavior is identical.
- Kill switch / disable path: N/A (raise `MAX_DISTINCT_PATHS` to relax).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; the path counter goes back to
unbounded.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal metric-cardinality bound; matches the existing model/stack
caps)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Mirrors the existing `MAX_DISTINCT_MODELS` / `MAX_DISTINCT_STACKS`
"other"-bucketing already in this class, so the cardinality-bounding
approach and its observability trade-off are consistent across the
counters.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
746038991c
commit
88a1f4e52b
@@ -30,6 +30,15 @@ logger = logging.getLogger("headroom.proxy")
|
||||
# client-supplied model cardinality stays bounded (see record_request).
|
||||
_OTHER_MODEL = "other"
|
||||
|
||||
# Same idea for inbound request paths (see record_inbound_request). ``path`` is
|
||||
# client-controlled and effectively unbounded — ID-bearing passthrough routes
|
||||
# such as ``/v1/files/{id}`` or ``/v1/responses/{id}`` mint a distinct key per
|
||||
# request. Past this many distinct paths, further paths collapse into
|
||||
# ``_OTHER_PATH`` so the counter (and its exported Prometheus series) can't grow
|
||||
# without bound. The cap sits well above a proxy's real route count.
|
||||
MAX_DISTINCT_PATHS = 256
|
||||
_OTHER_PATH = "other"
|
||||
|
||||
# Closed label set for headroom_requests_rate_limited_total{source}. Two 429s
|
||||
# mean opposite things to an operator: "headroom" is OUR limiter refusing the
|
||||
# request (raise the cap), "upstream" is the provider refusing it (back off or
|
||||
@@ -152,6 +161,7 @@ class PrometheusMetrics:
|
||||
self.inbound_requests_active = 0
|
||||
self.inbound_requests_by_method: dict[str, int] = defaultdict(int)
|
||||
self.inbound_requests_by_path: dict[str, int] = defaultdict(int)
|
||||
self._path_cardinality_warned = False
|
||||
self.inbound_responses_by_status: dict[str, int] = defaultdict(int)
|
||||
|
||||
self.tokens_input_total = 0
|
||||
@@ -413,6 +423,7 @@ class PrometheusMetrics:
|
||||
self.inbound_requests_active = 0
|
||||
self.inbound_requests_by_method.clear()
|
||||
self.inbound_requests_by_path.clear()
|
||||
self._path_cardinality_warned = False
|
||||
self.inbound_responses_by_status.clear()
|
||||
|
||||
self.tokens_input_total = 0
|
||||
@@ -773,7 +784,23 @@ class PrometheusMetrics:
|
||||
self.inbound_requests_total += 1
|
||||
self.inbound_requests_active += 1
|
||||
self.inbound_requests_by_method[method.upper()] += 1
|
||||
self.inbound_requests_by_path[path] += 1
|
||||
# Cap client-controlled path cardinality, mirroring the model cap in
|
||||
# record_request. A membership test (never a defaultdict index) keeps an
|
||||
# over-cap path from materializing a new key and defeating the bound.
|
||||
if path in self.inbound_requests_by_path or (
|
||||
len(self.inbound_requests_by_path) < MAX_DISTINCT_PATHS
|
||||
):
|
||||
bounded_path = path
|
||||
else:
|
||||
bounded_path = _OTHER_PATH
|
||||
if not self._path_cardinality_warned:
|
||||
self._path_cardinality_warned = True
|
||||
logger.warning(
|
||||
"metrics.record: inbound path cardinality cap (%d) reached; "
|
||||
'bucketing further paths into "other"',
|
||||
MAX_DISTINCT_PATHS,
|
||||
)
|
||||
self.inbound_requests_by_path[bounded_path] += 1
|
||||
|
||||
def record_inbound_response(self, *, status_code: int | str) -> None:
|
||||
self.inbound_requests_completed += 1
|
||||
|
||||
@@ -18,7 +18,7 @@ from headroom.observability import (
|
||||
set_otel_metrics,
|
||||
unregister_otel_metric_attribute_provider,
|
||||
)
|
||||
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
||||
from headroom.proxy.prometheus_metrics import MAX_DISTINCT_PATHS, PrometheusMetrics
|
||||
from headroom.telemetry.context import MAX_DISTINCT_MODELS
|
||||
from headroom.transforms.pipeline import TransformPipeline
|
||||
|
||||
@@ -552,6 +552,61 @@ async def test_prometheus_metrics_reset_rearms_cardinality_warning() -> None:
|
||||
assert len(metrics._cache_requests_by_model) == 0
|
||||
|
||||
|
||||
def test_prometheus_metrics_caps_inbound_path_cardinality() -> None:
|
||||
"""A client hitting unbounded distinct paths (ID-bearing passthrough routes)
|
||||
cannot grow inbound_requests_by_path past MAX_DISTINCT_PATHS + the "other"
|
||||
sentinel, while the total request count stays exact."""
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
|
||||
# Fill exactly to the cap with distinct paths: no bucketing yet.
|
||||
for i in range(MAX_DISTINCT_PATHS):
|
||||
metrics.record_inbound_request(method="GET", path=f"/v1/files/file_{i}")
|
||||
assert len(metrics.inbound_requests_by_path) == MAX_DISTINCT_PATHS
|
||||
assert "other" not in metrics.inbound_requests_by_path
|
||||
|
||||
# New distinct paths past the cap bucket into "other", never their own key.
|
||||
for i in range(5):
|
||||
metrics.record_inbound_request(method="GET", path=f"/v1/responses/resp_{i}")
|
||||
assert "/v1/responses/resp_0" not in metrics.inbound_requests_by_path
|
||||
assert metrics.inbound_requests_by_path["other"] == 5
|
||||
assert len(metrics.inbound_requests_by_path) == MAX_DISTINCT_PATHS + 1
|
||||
|
||||
# An already-tracked path keeps incrementing after the cap is reached.
|
||||
metrics.record_inbound_request(method="GET", path="/v1/files/file_0")
|
||||
assert metrics.inbound_requests_by_path["/v1/files/file_0"] == 2
|
||||
|
||||
# Accounting is preserved: every inbound request is counted somewhere.
|
||||
total_calls = MAX_DISTINCT_PATHS + 5 + 1
|
||||
assert metrics.inbound_requests_total == total_calls
|
||||
assert sum(metrics.inbound_requests_by_path.values()) == total_calls
|
||||
|
||||
|
||||
def test_prometheus_metrics_path_cardinality_warns_once(
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Bucketing inbound paths into "other" logs exactly one warning."""
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
with caplog.at_level(logging.WARNING, logger="headroom.proxy"):
|
||||
for i in range(MAX_DISTINCT_PATHS + 10):
|
||||
metrics.record_inbound_request(method="GET", path=f"/v1/files/file_{i}")
|
||||
cap_warnings = [r for r in caplog.records if "path cardinality cap" in r.getMessage()]
|
||||
assert len(cap_warnings) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prometheus_metrics_reset_rearms_path_cardinality_warning() -> None:
|
||||
"""reset_runtime clears the path dict and re-arms the one-shot cap warning."""
|
||||
metrics = PrometheusMetrics(stateless=True)
|
||||
for i in range(MAX_DISTINCT_PATHS + 5):
|
||||
metrics.record_inbound_request(method="GET", path=f"/v1/files/file_{i}")
|
||||
assert metrics._path_cardinality_warned is True
|
||||
|
||||
await metrics.reset_runtime()
|
||||
|
||||
assert metrics._path_cardinality_warned is False
|
||||
assert len(metrics.inbound_requests_by_path) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prometheus_metrics_export_bounds_model_series() -> None:
|
||||
"""export() emits at most MAX_DISTINCT_MODELS model series plus the 'other' bucket."""
|
||||
|
||||
Reference in New Issue
Block a user