fix(proxy): bound per-model state keyed by the client model string (#4407)

## Description

The request body's `model` string was used, unbounded in count and
length, as a key in `CostTracker`'s per-model maps, in the
estimated-basis warning set, and in `SavingsTracker.by_model` (written
whole to `proxy_savings.json` and returned by `/stats`). Keys are now
clipped to `MAX_LABEL_LENGTH` (128) and names past `MAX_TRACKED_MODELS`
(200) distinct keys are bucketed into `"other"`, the limits
`persistent_metrics.py` already uses. Pricing still uses the real model
name when a request is recorded (the budget ledger is unchanged). Token
totals stay exact because over-cap requests are bucketed, not dropped.
Dollar totals stay exact too: `totals()` and `stats()` price each bucket
by its key at read time and `other` has no price, so a request whose
label is folded into `other` is priced once at record time (with the
real model name) into five running accumulators (input, output,
list-priced compression savings, cache-aware compression savings,
tool-schema savings) that are added to the read-time sums. A
`passthrough:` name that overflows folds into `passthrough:other`, so
`stats()` keeps excluding it. History checkpoints in
`proxy_savings.json` clip the model name to 128 characters. The
estimated-basis warning set stops growing at 200 names for the whole
process.

Fixes #4402

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/persistent_metrics.py`: add `OTHER_MODEL`,
`PASSTHROUGH_PREFIX`, `OTHER_PASSTHROUGH_MODEL` and
`bounded_model_key(model, tracked)` (overflow of a `passthrough:` name
goes to `passthrough:other`).
- `headroom/proxy/cost.py`: `record_tokens` computes `key =
bounded_model_key(model, self._requests_by_model)` and uses it for the
per-model and `(model, long_context)` maps and for the estimated-basis
warning; `estimate_cost` still receives `model`.
`_warn_estimated_basis_once` adds nothing once the process-wide set
holds `MAX_TRACKED_MODELS` names.
- `headroom/proxy/cost.py`: new `_folded_input_usd` /
`_folded_output_usd` / `_folded_savings_usd` /
`_folded_cache_aware_savings_usd` / `_folded_tool_savings_usd` (cleared
by `reset_runtime`) and `_book_folded_usd`, called from `record_tokens`
only when the label was folded into `other` (a clipped label keeps its
read-time pricing, and an overflowing `passthrough:` name folds into
`passthrough:other`, so nothing is priced twice); it uses the same
formulas as the read-time pass (for the last two, the same
`_bucket_by_cache_mix` regions and the same `_get_cache_prices(...,
for_billing=False)` lookups). `totals()` and `stats()` add the
accumulators to the input, output, savings, `cache_aware_savings_usd`
and `tool_savings_usd` sums.
- `headroom/proxy/savings_tracker.py`: `_record_by_model_locked` uses
the same helper; both history checkpoint writers clip `model` to
`MAX_LABEL_LENGTH`.
- `tests/test_bounded_model_keys.py`: 10 tests: the 6 from the first
round (CostTracker maps, warning set, persisted `by_model`, overflowing
passthrough names, warning set across `reset_runtime()`, persisted
history model length) plus
`test_priced_request_after_saturation_keeps_aggregate_dollars` (the
reviewer's scenario: `totals()` equals `(100000, 0.25)` and equals
`period_cost_breakdown()['total_usd']`),
`test_overflow_conserves_every_priced_stats_dollar_figure` (a saturated
tracker and an unsaturated control agree on input, output, total and
savings dollars and on `totals()`, with cache read/write and output
tokens) `test_overflowing_passthrough_names_add_no_dollars`, and (round
3) `test_overflow_keeps_cache_aware_savings_for_reviewer_probe` (the
review's exact probe, plus `reset_runtime()` clearing the accumulators).
`test_overflow_conserves_every_priced_stats_dollar_figure` now also
records `tool_schema_saved` and compares `cache_aware_savings_usd` and
`tool_savings_usd`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Round 3 (review 5636282042), with only the `cost.py` change of the new
commit reverted and the new tests kept:

```text
E   AssertionError: cache_aware_savings_usd
E   assert 0.0 == 0.1
FAILED tests/test_bounded_model_keys.py::test_overflow_conserves_every_priced_stats_dollar_figure
FAILED tests/test_bounded_model_keys.py::test_overflow_keeps_cache_aware_savings_for_reviewer_probe
========================= 2 failed, 8 passed in 5.30s ==========================
```

With the fix:

```text
============================== 10 passed in 6.46s ==============================
```

Round 2 (review 5636023862), with only the `cost.py` change of the new
commit reverted and the new tests kept:

```text
E   assert (100000, 0.0) == (100000, 0.25)
E   AssertionError: total_input_cost_usd
E   assert 0.0 == 0.125
FAILED tests/test_bounded_model_keys.py::test_priced_request_after_saturation_keeps_aggregate_dollars
FAILED tests/test_bounded_model_keys.py::test_overflow_conserves_every_priced_stats_dollar_figure
========================= 2 failed, 7 passed in 4.41s ==========================
```

With the fix:

```text
============================== 9 passed in 4.13s ===============================
```

Round 1 (kept for reference): with the source changes reverted, the 3
tests added in the follow-up commit failed
(`test_overflowing_passthrough_names_stay_out_of_processed_totals`,
`test_estimated_basis_warning_set_is_capped_across_resets`,
`test_savings_history_checkpoints_clip_model_names`).

Round 3 neighbours: `test_*cost*.py`, `test_*savings*.py` (except
`test_savings_items.py`, which needs `respx`, not installed in my
environment), `test_*persistent_metrics*.py`, `test_per_model*.py`,
`test_stats_*.py` and the new file: 505 passed, 4 skipped, 2 failed
(`test_bundled_tools_savings.py::test_ensure_tools_installs_every_tool`
and `::test_ast_grep_slice_saves_tokens`, bundled-tool installs, files
this change does not touch; not re-run on a clean checkout). Round 2
neighbours (`test_persistent_metrics*.py`,
`test_cost_pricing_warning_dedup.py`, `test_cost_budget_*.py`,
`test_cost_tracker_totals.py`, `test_cost_card_attribution.py`,
`test_proxy_savings_history.py`, `test_savings_tracker_zero_price.py`)
plus `test_cost_cache_prices.py`, `test_cost_tracker_counterfactual.py`
and `test_savings_tracker_output_cost.py`, together with the new file:
166 passed, 0 failed. `tests/test_proxy_budget.py` and
`tests/test_proxy_budget_provider_enforcement.py` have 2
websocket-preflight failures
(`test_zero_budget_websocket_preflight_rejected`,
`test_exhausted_budget_rejects_responses_websocket_before_upstream`)
that also fail on a clean checkout of `e995eb9f` in my environment and
are unrelated to this change.

`ruff check` and `ruff format --check` on `headroom/proxy/cost.py` and
`tests/test_bounded_model_keys.py` after the round-3 commit: All checks
passed / 2 files already formatted. `mypy` was not run.

## Real Behavior Proof

- Environment: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn
0.54.0, fastapi 0.158.1, starlette 1.9.0; base `e995eb9f`;
`headroom/_core.abi3.so` copied (untracked, not committed) from the
official `headroom_ai-0.41.0` wheel.
- Exact command / steps: stub upstream on 127.0.0.1:18811 (returns a
message with a `usage` block including `cache_read_input_tokens: 500`,
so each request writes a history point); `HEADROOM_SAVINGS_PATH=<file>
ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18811 python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18810 --no-cache
--no-rate-limit`; 400 `POST /v1/messages` with distinct 1500-character
model names; then `GET /stats`, SIGINT, read the savings file. Before =
base source (the change reverted with `git apply -R`), after = this
change. Rounds 2 and 3: `PYTHONPATH=. python -P -m pytest -q -p
no:cacheprovider tests/test_bounded_model_keys.py`, run with the
`cost.py` change of the new commit reverted (`git apply -R`) and with it
applied.
- Observed result: Rounds 2 and 3: no live-proxy run was done. The money
path is `CostTracker`, driven directly in unit tests. Before the fix,
`CostTracker.totals()` after 200 distinct unpriced zero-token names and
then `gpt-4o` with 100,000 uncached input tokens returned `(100000,
0.0)` while `period_cost_breakdown()['total_usd']` was 0.25; after the
fix it returns `(100000, 0.25)`. In the second test, a saturated tracker
and an unsaturated control now agree on `total_input_cost_usd`,
`output_cost_usd`, `total_cost_usd` and `savings_usd` (before: input 0.0
against 0.125). Round 3, the review's probe (200 unpriced zero-token
labels, then `gpt-4o` with tokens_saved=40,000, tokens_sent=60,000,
cache_read=20,000, cache_write=10,000, uncached=30,000, output=5,000):
`cache_aware_savings_usd` was 0.0 on the saturated tracker against 0.1
on the control; it is now equal, and `tool_savings_usd` (with
`tool_schema_saved=8,000` added) is equal too. Round 1 (live proxy): on
the base commit `cost.per_model`, `persistent_savings.by_model` and the
savings-file `by_model` and `history` carry 400 distinct model names of
up to 1508 characters; `/stats` is 2,706,948 bytes and
`proxy_savings.json` is 1,565,090 bytes. With this change the three
per-model blocks have 201 keys (200 plus `other`) of at most 128
characters, the 400 history points carry model names of at most 128
characters (51,200 characters in total instead of 603,090), `/stats` is
1,444,793 bytes and `proxy_savings.json` is 395,719 bytes. With only the
first commit (before the history clip) the file was 947,609 bytes with
1508-character history names. All 400 requests returned 200 in every
run, and each response carried the stub's model echo, so the requests
reached the stub. `/stats requests.by_model` (Prometheus metrics, not
changed here) still shows 400 long keys.
- Not tested: a priced model through the real proxy after cap saturation
(round 2 is unit-level); the dashboard summary itself (only `stats()`
was compared); a real Anthropic upstream; the passthrough overflow and
the warning-set cap through the real proxy (unit tests only; the batch
routes were not driven); streaming and OpenAI/Gemini routes (they reach
the same `record_tokens` and `record_request`); Windows/macOS; the Rust
proxy.

Raw output:

```text
##### BEFORE (base e995eb9f)
client: requests 400 status counts {200: 400} responses carrying the stub's model echo: 400
/stats .requests.by_model: 400 keys, longest key 1508 chars
/stats .persistent_savings.by_model: 400 keys, longest key 1508 chars
/stats .cost.per_model: 400 keys, longest key 1508 chars
/stats response bytes: 2706948
proxy_savings.json: bytes 1565090 by_model keys 400 longest key 1508 'other' present False
proxy_savings.json history: points 400 longest model 1508 total model chars 603090
##### AFTER (this change)
client: requests 400 status counts {200: 400} responses carrying the stub's model echo: 400
/stats .requests.by_model: 400 keys, longest key 1508 chars
/stats .persistent_savings.by_model: 201 keys, longest key 128 chars
/stats .cost.per_model: 201 keys, longest key 128 chars
/stats response bytes: 1444793
proxy_savings.json: bytes 395719 by_model keys 201 longest key 128 'other' present True
proxy_savings.json history: points 400 longest model 128 total model chars 51200
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for more than 200 distinct model
names or names longer than 128 characters; those are bucketed or clipped
in the per-model stats
- Kill switch / disable path: none
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation checklist item: N/A, no user-facing doc changes.
- Per-model attribution is coarsened into `other` when labels fold; the
aggregate dollar fields (`total_input_cost_usd`, `output_cost_usd`,
`total_cost_usd`, `savings_usd`, `cache_aware_savings_usd`,
`tool_savings_usd`) and `totals()` are conserved.
- Not changed: `context_guard._learned_limits` (only written after a
plausible "prompt is too long" upstream error), the `/stats`
`requests.by_model` block (`PrometheusMetrics`, capped in count but not
key length), and loading an already-oversized `by_model` from an
existing `proxy_savings.json`.
- Produced with AI assistance (Claude Code) operated by me (HetCreep);
the reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
HetCreep
2026-10-24 08:38:49 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 421d453c4c
commit 33e5f67464
4 changed files with 329 additions and 44 deletions
+132 -40
View File
@@ -27,6 +27,11 @@ from headroom.proxy.budget_basis_policy import (
resolve_estimated_basis_policy,
)
from headroom.proxy.modes import PROXY_MODE_CACHE
from headroom.proxy.persistent_metrics import (
MAX_TRACKED_MODELS,
OTHER_MODEL,
bounded_model_key,
)
if TYPE_CHECKING:
from headroom.proxy.prometheus_metrics import PrometheusMetrics
@@ -89,7 +94,11 @@ _warned_estimated_basis_models: set[str] = set()
def _warn_estimated_basis_once(model: str) -> None:
"""Warn the first time ``model`` books a cost against Headroom's estimate."""
if model in _warned_estimated_basis_models:
# Process-wide cap: the set outlives any one tracker (reset_runtime), so the
# per-tracker key bound alone does not limit it.
if model in _warned_estimated_basis_models or (
len(_warned_estimated_basis_models) >= MAX_TRACKED_MODELS
):
return
_warned_estimated_basis_models.add(model)
logger.warning(
@@ -932,6 +941,14 @@ class CostTracker:
self._api_cache_write_5m_by_model: dict[str, int] = {}
self._api_cache_write_1h_by_model: dict[str, int] = {}
self._api_uncached_by_model: dict[str, int] = {}
# Priced dollars of requests whose model label was folded into ``other``:
# the read-time pass prices by label, and ``other`` has no price, so
# those dollars are priced once here, at record time.
self._folded_input_usd = 0.0
self._folded_output_usd = 0.0
self._folded_savings_usd = 0.0
self._folded_cache_aware_savings_usd = 0.0
self._folded_tool_savings_usd = 0.0
def reset_runtime(self) -> None:
"""Reset in-memory cost/token counters for local test/debug use."""
@@ -960,6 +977,11 @@ class CostTracker:
self._api_cache_write_5m_by_model.clear()
self._api_cache_write_1h_by_model.clear()
self._api_uncached_by_model.clear()
self._folded_input_usd = 0.0
self._folded_output_usd = 0.0
self._folded_savings_usd = 0.0
self._folded_cache_aware_savings_usd = 0.0
self._folded_tool_savings_usd = 0.0
def estimate_cost(
self,
@@ -1087,10 +1109,9 @@ class CostTracker:
model,
)
tokens_saved = 0
self._tokens_saved_by_model[model] = (
self._tokens_saved_by_model.get(model, 0) + tokens_saved
)
self._tool_saved_by_model[model] = self._tool_saved_by_model.get(model, 0) + max(
key = bounded_model_key(model, self._requests_by_model)
self._tokens_saved_by_model[key] = self._tokens_saved_by_model.get(key, 0) + tokens_saved
self._tool_saved_by_model[key] = self._tool_saved_by_model.get(key, 0) + max(
0, tool_schema_saved
)
# A request the provider reported no cache data for (or whose write
@@ -1128,7 +1149,7 @@ class CostTracker:
uncached_tokens=uncached_tokens,
region="live_zone",
)
wkey = (model, long_context)
wkey = (key, long_context)
self._saved_write_5m_by_tier[wkey] = self._saved_write_5m_by_tier.get(wkey, 0.0) + c_w5m
self._saved_write_1h_by_tier[wkey] = self._saved_write_1h_by_tier.get(wkey, 0.0) + c_w1h
self._saved_list_by_tier[wkey] = self._saved_list_by_tier.get(wkey, 0.0) + c_list
@@ -1147,46 +1168,54 @@ class CostTracker:
uncached_tokens=uncached_tokens,
region="prefix",
)
self._tool_saved_read_by_model[model] = (
self._tool_saved_read_by_model.get(model, 0.0) + read_part
self._tool_saved_read_by_model[key] = (
self._tool_saved_read_by_model.get(key, 0.0) + read_part
)
self._tool_saved_write_5m_by_model[model] = (
self._tool_saved_write_5m_by_model.get(model, 0.0) + w5m_part
self._tool_saved_write_5m_by_model[key] = (
self._tool_saved_write_5m_by_model.get(key, 0.0) + w5m_part
)
self._tool_saved_write_1h_by_model[model] = (
self._tool_saved_write_1h_by_model.get(model, 0.0) + w1h_part
self._tool_saved_write_1h_by_model[key] = (
self._tool_saved_write_1h_by_model.get(key, 0.0) + w1h_part
)
self._tool_saved_list_by_model[model] = (
self._tool_saved_list_by_model.get(model, 0.0) + list_part
self._tool_saved_list_by_model[key] = (
self._tool_saved_list_by_model.get(key, 0.0) + list_part
)
self._tokens_sent_by_model[model] = self._tokens_sent_by_model.get(model, 0) + tokens_sent
self._tokens_sent_by_model[key] = self._tokens_sent_by_model.get(key, 0) + tokens_sent
if provider_reported:
self._provider_tokens_sent_by_model[model] = (
self._provider_tokens_sent_by_model.get(model, 0) + tokens_sent
self._provider_tokens_sent_by_model[key] = (
self._provider_tokens_sent_by_model.get(key, 0) + tokens_sent
)
self._provider_requests_by_model[model] = (
self._provider_requests_by_model.get(model, 0) + 1
)
okey = (model, long_context)
self._provider_requests_by_model[key] = self._provider_requests_by_model.get(key, 0) + 1
okey = (key, long_context)
self._output_tokens_by_tier[okey] = self._output_tokens_by_tier.get(okey, 0) + max(
0, output_tokens
)
self._requests_by_model[model] = self._requests_by_model.get(model, 0) + 1
self._api_cache_read_by_model[model] = (
self._api_cache_read_by_model.get(model, 0) + cache_read_tokens
self._requests_by_model[key] = self._requests_by_model.get(key, 0) + 1
self._api_cache_read_by_model[key] = (
self._api_cache_read_by_model.get(key, 0) + cache_read_tokens
)
self._api_cache_write_by_model[model] = (
self._api_cache_write_by_model.get(model, 0) + write_eff
self._api_cache_write_by_model[key] = self._api_cache_write_by_model.get(key, 0) + write_eff
self._api_cache_write_5m_by_model[key] = (
self._api_cache_write_5m_by_model.get(key, 0) + write_5m_eff
)
self._api_cache_write_5m_by_model[model] = (
self._api_cache_write_5m_by_model.get(model, 0) + write_5m_eff
)
self._api_cache_write_1h_by_model[model] = (
self._api_cache_write_1h_by_model.get(model, 0) + write_1h_eff
)
self._api_uncached_by_model[model] = (
self._api_uncached_by_model.get(model, 0) + uncached_tokens
self._api_cache_write_1h_by_model[key] = (
self._api_cache_write_1h_by_model.get(key, 0) + write_1h_eff
)
self._api_uncached_by_model[key] = self._api_uncached_by_model.get(key, 0) + uncached_tokens
if key == OTHER_MODEL and model != OTHER_MODEL:
self._book_folded_usd(
model,
long_context=long_context,
tokens_saved=tokens_saved,
tokens_sent=tokens_sent,
cache_read=cache_read_tokens,
write=write_eff,
write_5m=write_5m_eff,
write_1h=write_1h_eff,
uncached=uncached_tokens,
output=output_tokens,
tool_schema_saved=tool_schema_saved,
)
# Populate _costs so check_budget() has real data to enforce against.
# When the call site had no API usage breakdown (all cache/uncached
@@ -1221,7 +1250,7 @@ class CostTracker:
if not (uncached_tokens or cache_read_tokens or cache_write_tokens):
input_tokens = tokens_sent
basis = COST_BASIS_ESTIMATED
_warn_estimated_basis_once(model)
_warn_estimated_basis_once(key)
cost = self.estimate_cost(
model=model,
input_tokens=input_tokens,
@@ -1426,6 +1455,68 @@ class CostTracker:
w5m = total - w1h
return w5m * w5m_price + w1h * w1h_price
def _book_folded_usd(
self,
model: str,
*,
long_context: bool,
tokens_saved: int,
tokens_sent: int,
cache_read: int,
write: int,
write_5m: int,
write_1h: int,
uncached: int,
output: int,
tool_schema_saved: int,
) -> None:
"""Price one request whose label was folded, with the read-time formulas."""
prices = self._get_cache_prices(model)
if prices:
cr_price, w5_price, w1h_price, uncached_price = prices
if cache_read + write + uncached > 0:
w1h = min(max(0, write_1h), write)
self._folded_input_usd += (
cache_read * cr_price
+ (write - w1h) * w5_price
+ w1h * w1h_price
+ uncached * uncached_price
)
else:
self._folded_input_usd += tokens_sent * uncached_price
self._folded_savings_usd += tokens_saved * uncached_price
out_price = self._get_output_price(model, long_context=long_context)
if out_price:
self._folded_output_usd += max(0, output) * out_price
# Cache-aware compression and tool-schema savings: same buckets and
# same price lookups as the record-time maps and stats() use.
if tokens_saved > 0:
_r, c5, c1, cl = _bucket_by_cache_mix(
tokens_saved,
cache_read_tokens=0,
cache_write_tokens=write,
cache_write_5m_tokens=write_5m,
cache_write_1h_tokens=write_1h,
uncached_tokens=uncached,
region="live_zone",
)
ca = self._get_cache_prices(model, long_context=long_context, for_billing=False)
if ca:
self._folded_cache_aware_savings_usd += c5 * ca[1] + c1 * ca[2] + cl * ca[3]
if tool_schema_saved > 0:
rd, t5, t1, tl = _bucket_by_cache_mix(
tool_schema_saved,
cache_read_tokens=cache_read,
cache_write_tokens=write,
cache_write_5m_tokens=write_5m,
cache_write_1h_tokens=write_1h,
uncached_tokens=uncached,
region="prefix",
)
tp = self._get_cache_prices(model, for_billing=False)
if tp:
self._folded_tool_savings_usd += rd * tp[0] + t5 * tp[1] + t1 * tp[2] + tl * tp[3]
def _get_cache_prices(
self, model: str, *, long_context: bool = False, for_billing: bool = True
) -> tuple[float, float, float, float] | None:
@@ -1500,7 +1591,7 @@ class CostTracker:
)
else:
cost_with_headroom += sent * uncached_price
return total_input_tokens, round(cost_with_headroom, 4)
return total_input_tokens, round(cost_with_headroom + self._folded_input_usd, 4)
def stats(self) -> dict:
"""Get token statistics per model."""
@@ -1577,12 +1668,13 @@ class CostTracker:
cost_with_headroom += model_cost
total_billed_input_tokens += billed_tokens
cost_with_headroom += self._folded_input_usd
# Compression savings: price saved tokens at the model's list input price.
# This is simple, monotonic, and transparent — each saved token is valued
# at the published $/token rate for its model. Not affected by cache mix.
# Budget enforcement reads this figure, so it stays as-is; the
# cache-aware valuation below is reported alongside it.
savings_usd = 0.0
savings_usd = self._folded_savings_usd
for model in self._tokens_saved_by_model:
if model.startswith("passthrough:"):
continue
@@ -1597,7 +1689,7 @@ class CostTracker:
# Completion spend. The input-only figure above is what a budget and the
# per-model table want; a card that says "$X spent" has to include the
# tokens the model emitted, or the spend it shows isn't the bill.
output_cost_usd = 0.0
output_cost_usd = self._folded_output_usd
for (model, long_context), out_tokens in self._output_tokens_by_tier.items():
price = self._get_output_price(model, long_context=long_context)
if price:
@@ -1610,7 +1702,7 @@ class CostTracker:
# pricing (``savings_usd``) ignores both. Reported separately so budget
# enforcement keeps its monotonic list-priced basis; the dashboard's
# cost card prefers this one.
cache_aware_savings_usd = 0.0
cache_aware_savings_usd = self._folded_cache_aware_savings_usd
for key in (
set(self._saved_write_5m_by_tier)
| set(self._saved_write_1h_by_tier)
@@ -1645,7 +1737,7 @@ class CostTracker:
# multiplier. Reported as its OWN key rather than widening
# ``savings_usd``: that figure feeds budget enforcement, and the
# session summary widens the cost card from this key instead.
tool_savings_usd = 0.0
tool_savings_usd = self._folded_tool_savings_usd
tool_models = (
set(self._tool_saved_read_by_model)
| set(self._tool_saved_write_5m_by_model)
+20
View File
@@ -14,6 +14,26 @@ MAX_STACK_VALUES = 64
MAX_TRACKED_MODELS = 200
MAX_EXPOSED_MODELS = 100
MAX_LABEL_LENGTH = 128
OTHER_MODEL = "other"
# ``CostTracker.stats()`` skips keys with this prefix; an overflowing
# passthrough name must keep it or it would enter the processed-traffic totals.
PASSTHROUGH_PREFIX = "passthrough:"
OTHER_PASSTHROUGH_MODEL = PASSTHROUGH_PREFIX + OTHER_MODEL
def bounded_model_key(model: str, tracked: Any) -> str:
"""Key under which a client-supplied model name may enter a per-model map.
``model`` comes from the request body, so it is clipped to
``MAX_LABEL_LENGTH`` and a new name past ``MAX_TRACKED_MODELS`` distinct
keys collapses into ``OTHER_MODEL`` (``OTHER_PASSTHROUGH_MODEL`` for a
``passthrough:`` name). ``tracked`` is the map being written.
"""
key = model[:MAX_LABEL_LENGTH]
if key in tracked or len(tracked) < MAX_TRACKED_MODELS:
return key
return OTHER_PASSTHROUGH_MODEL if key.startswith(PASSTHROUGH_PREFIX) else OTHER_MODEL
KNOWN_MISS_REASONS = frozenset({"ttl_expiry", "prefix_change", "unknown"})
# Mirrors RATE_LIMIT_SOURCES in prometheus_metrics: "headroom" is our own
+8 -4
View File
@@ -26,7 +26,11 @@ from typing import Any
from headroom import paths as _paths
from headroom.log_safety import WarnOnce, describe_exception, safe_id
from headroom.proxy import project_name_policy
from headroom.proxy.persistent_metrics import PersistentMetricsState
from headroom.proxy.persistent_metrics import (
MAX_LABEL_LENGTH,
PersistentMetricsState,
bounded_model_key,
)
PROJECT_NAME_MAX_LENGTH = project_name_policy.PROJECT_NAME_MAX_LENGTH
sanitize_project_name = project_name_policy.sanitize_project_name
@@ -1097,7 +1101,7 @@ class SavingsTracker:
"timestamp": _to_utc_iso(timestamp_dt),
"provider": _normalize_provider(provider),
"agent": _normalize_agent(agent),
"model": _normalize_model(model),
"model": _normalize_model(model)[:MAX_LABEL_LENGTH],
"total_tokens_saved": lifetime["tokens_saved"],
"compression_savings_usd": lifetime["compression_savings_usd"],
"total_input_tokens": lifetime["total_input_tokens"],
@@ -1401,7 +1405,7 @@ class SavingsTracker:
"timestamp": _to_utc_iso(timestamp_dt),
"provider": _normalize_provider(provider),
"agent": _normalize_agent(agent),
"model": _normalize_model(model),
"model": _normalize_model(model)[:MAX_LABEL_LENGTH],
"total_tokens_saved": lifetime["tokens_saved"],
"compression_savings_usd": lifetime["compression_savings_usd"],
"cache_read_tokens": lifetime["cache_read_tokens"],
@@ -1578,7 +1582,7 @@ class SavingsTracker:
without migration.
"""
by_model: dict[str, dict[str, Any]] = self._state.setdefault("by_model", {})
key = _normalize_model(model)
key = bounded_model_key(_normalize_model(model), by_model)
entry = by_model.setdefault(key, _empty_by_model_entry())
entry["requests"] += max(requests_delta, 0)
entry["tokens_saved"] += max(tokens_saved_delta, 0)
+169
View File
@@ -0,0 +1,169 @@
"""A client-supplied ``model`` string must not grow per-model state without bound."""
from __future__ import annotations
import json
from headroom.proxy.cost import CostTracker
from headroom.proxy.persistent_metrics import (
MAX_LABEL_LENGTH,
MAX_TRACKED_MODELS,
)
from headroom.proxy.savings_tracker import SavingsTracker
_NAMES = MAX_TRACKED_MODELS + 50
def test_cost_tracker_per_model_maps_are_bounded():
tracker = CostTracker()
for i in range(_NAMES):
tracker.record_tokens(f"m-{i}-" + "x" * 2000, tokens_saved=1, tokens_sent=10)
assert len(tracker._requests_by_model) == MAX_TRACKED_MODELS + 1
assert len(tracker._tokens_sent_by_model) == MAX_TRACKED_MODELS + 1
assert tracker._requests_by_model["other"] == _NAMES - MAX_TRACKED_MODELS
assert all(len(k) <= MAX_LABEL_LENGTH for k in tracker._requests_by_model)
# Totals stay exact: over-cap requests are bucketed, not dropped.
assert sum(tracker._tokens_sent_by_model.values()) == 10 * _NAMES
def test_estimated_basis_warning_set_is_bounded():
import headroom.proxy.cost as cost_mod
cost_mod._warned_estimated_basis_models.clear()
tracker = CostTracker()
for i in range(_NAMES):
tracker.record_tokens(f"w-{i}", tokens_saved=0, tokens_sent=10)
assert len(cost_mod._warned_estimated_basis_models) <= MAX_TRACKED_MODELS + 1
cost_mod._warned_estimated_basis_models.clear()
def test_savings_tracker_by_model_is_bounded_and_persisted_small(tmp_path):
path = tmp_path / "proxy_savings.json"
tracker = SavingsTracker(path=str(path), save_flush_every=1)
for i in range(_NAMES):
tracker.record_request(model=f"s-{i}-" + "y" * 2000, input_tokens=100, tokens_saved=10)
by_model = json.loads(path.read_text(encoding="utf-8"))["by_model"]
assert len(by_model) == MAX_TRACKED_MODELS + 1
assert all(len(k) <= MAX_LABEL_LENGTH for k in by_model)
assert by_model["other"]["requests"] == _NAMES - MAX_TRACKED_MODELS
assert sum(e["tokens_saved"] for e in by_model.values()) == 10 * _NAMES
def test_overflowing_passthrough_names_stay_out_of_processed_totals():
tracker = CostTracker()
for i in range(MAX_TRACKED_MODELS):
tracker.record_tokens(f"passthrough:batch:m{i}", tokens_saved=1, tokens_sent=1)
tracker.record_tokens("passthrough:batch:next", tokens_saved=10, tokens_sent=90)
stats = tracker.stats()
assert stats["total_tokens_saved"] == 0
assert stats["total_input_tokens"] == 0
assert "other" not in tracker._requests_by_model
assert tracker._requests_by_model["passthrough:other"] == 1
def test_estimated_basis_warning_set_is_capped_across_resets():
import headroom.proxy.cost as cost_mod
cost_mod._warned_estimated_basis_models.clear()
tracker = CostTracker()
for i in range(3):
for j in range(MAX_TRACKED_MODELS):
tracker.record_tokens(f"r{i}-{j}", tokens_saved=0, tokens_sent=10)
tracker.reset_runtime()
assert len(cost_mod._warned_estimated_basis_models) <= MAX_TRACKED_MODELS
cost_mod._warned_estimated_basis_models.clear()
def test_savings_history_checkpoints_clip_model_names(tmp_path):
tracker = SavingsTracker(path=str(tmp_path / "proxy_savings.json"), save_flush_every=1)
tracker.record_request(model="h-" + "z" * 2000, input_tokens=100, tokens_saved=10)
history = json.loads((tmp_path / "proxy_savings.json").read_text(encoding="utf-8"))["history"]
assert history
assert all(len(point["model"]) <= MAX_LABEL_LENGTH for point in history)
def _saturate_with_unpriced_names(tracker: CostTracker) -> None:
for i in range(MAX_TRACKED_MODELS):
tracker.record_tokens(f"unpriced-{i}", tokens_saved=0, tokens_sent=0)
def test_priced_request_after_saturation_keeps_aggregate_dollars():
control = CostTracker()
control.record_tokens("gpt-4o", tokens_saved=0, tokens_sent=100_000, uncached_tokens=100_000)
assert control.totals() == (100_000, 0.25)
tracker = CostTracker()
_saturate_with_unpriced_names(tracker)
tracker.record_tokens("gpt-4o", tokens_saved=0, tokens_sent=100_000, uncached_tokens=100_000)
assert tracker._requests_by_model["other"] == 1
assert tracker.totals() == (100_000, 0.25)
assert tracker.totals()[1] == tracker.period_cost_breakdown()["total_usd"]
assert tracker.stats()["total_input_cost_usd"] == 0.25
def test_overflow_conserves_every_priced_stats_dollar_figure():
kwargs = {
"tokens_saved": 40_000,
"tokens_sent": 60_000,
"cache_read_tokens": 20_000,
"cache_write_tokens": 10_000,
"uncached_tokens": 30_000,
"output_tokens": 5_000,
"tool_schema_saved": 8_000,
}
control = CostTracker()
control.record_tokens("gpt-4o", **kwargs)
tracker = CostTracker()
_saturate_with_unpriced_names(tracker)
tracker.record_tokens("gpt-4o", **kwargs)
want, got = control.stats(), tracker.stats()
assert want["total_cost_usd"] > 0
assert want["cache_aware_savings_usd"] > 0
assert want["tool_savings_usd"] > 0
for field in (
"total_input_cost_usd",
"output_cost_usd",
"total_cost_usd",
"savings_usd",
"cache_aware_savings_usd",
"tool_savings_usd",
):
assert got[field] == want[field], field
assert tracker.totals() == control.totals()
def test_overflow_keeps_cache_aware_savings_for_reviewer_probe():
kwargs = {
"tokens_saved": 40_000,
"tokens_sent": 60_000,
"cache_read_tokens": 20_000,
"cache_write_tokens": 10_000,
"uncached_tokens": 30_000,
"output_tokens": 5_000,
}
control = CostTracker()
control.record_tokens("gpt-4o", **kwargs)
tracker = CostTracker()
_saturate_with_unpriced_names(tracker)
tracker.record_tokens("gpt-4o", **kwargs)
assert control.stats()["cache_aware_savings_usd"] > 0
assert tracker.stats()["cache_aware_savings_usd"] == control.stats()["cache_aware_savings_usd"]
tracker.reset_runtime()
assert tracker.stats()["cache_aware_savings_usd"] == 0.0
assert tracker.stats()["tool_savings_usd"] == 0.0
def test_overflowing_passthrough_names_add_no_dollars():
tracker = CostTracker()
for i in range(MAX_TRACKED_MODELS):
tracker.record_tokens(f"passthrough:batch:m{i}", tokens_saved=0, tokens_sent=1)
tracker.record_tokens(
"passthrough:gpt-4o", tokens_saved=0, tokens_sent=100_000, uncached_tokens=100_000
)
assert tracker.stats()["total_input_cost_usd"] == 0.0