mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
The request body's `model` string was used, unbounded in count and
length, as a key in `CostTracker`'s per-model maps, in the
estimated-basis warning set, and in `SavingsTracker.by_model` (written
whole to `proxy_savings.json` and returned by `/stats`). Keys are now
clipped to `MAX_LABEL_LENGTH` (128) and names past `MAX_TRACKED_MODELS`
(200) distinct keys are bucketed into `"other"`, the limits
`persistent_metrics.py` already uses. Pricing still uses the real model
name when a request is recorded (the budget ledger is unchanged). Token
totals stay exact because over-cap requests are bucketed, not dropped.
Dollar totals stay exact too: `totals()` and `stats()` price each bucket
by its key at read time and `other` has no price, so a request whose
label is folded into `other` is priced once at record time (with the
real model name) into five running accumulators (input, output,
list-priced compression savings, cache-aware compression savings,
tool-schema savings) that are added to the read-time sums. A
`passthrough:` name that overflows folds into `passthrough:other`, so
`stats()` keeps excluding it. History checkpoints in
`proxy_savings.json` clip the model name to 128 characters. The
estimated-basis warning set stops growing at 200 names for the whole
process.
Fixes #4402
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/persistent_metrics.py`: add `OTHER_MODEL`,
`PASSTHROUGH_PREFIX`, `OTHER_PASSTHROUGH_MODEL` and
`bounded_model_key(model, tracked)` (overflow of a `passthrough:` name
goes to `passthrough:other`).
- `headroom/proxy/cost.py`: `record_tokens` computes `key =
bounded_model_key(model, self._requests_by_model)` and uses it for the
per-model and `(model, long_context)` maps and for the estimated-basis
warning; `estimate_cost` still receives `model`.
`_warn_estimated_basis_once` adds nothing once the process-wide set
holds `MAX_TRACKED_MODELS` names.
- `headroom/proxy/cost.py`: new `_folded_input_usd` /
`_folded_output_usd` / `_folded_savings_usd` /
`_folded_cache_aware_savings_usd` / `_folded_tool_savings_usd` (cleared
by `reset_runtime`) and `_book_folded_usd`, called from `record_tokens`
only when the label was folded into `other` (a clipped label keeps its
read-time pricing, and an overflowing `passthrough:` name folds into
`passthrough:other`, so nothing is priced twice); it uses the same
formulas as the read-time pass (for the last two, the same
`_bucket_by_cache_mix` regions and the same `_get_cache_prices(...,
for_billing=False)` lookups). `totals()` and `stats()` add the
accumulators to the input, output, savings, `cache_aware_savings_usd`
and `tool_savings_usd` sums.
- `headroom/proxy/savings_tracker.py`: `_record_by_model_locked` uses
the same helper; both history checkpoint writers clip `model` to
`MAX_LABEL_LENGTH`.
- `tests/test_bounded_model_keys.py`: 10 tests: the 6 from the first
round (CostTracker maps, warning set, persisted `by_model`, overflowing
passthrough names, warning set across `reset_runtime()`, persisted
history model length) plus
`test_priced_request_after_saturation_keeps_aggregate_dollars` (the
reviewer's scenario: `totals()` equals `(100000, 0.25)` and equals
`period_cost_breakdown()['total_usd']`),
`test_overflow_conserves_every_priced_stats_dollar_figure` (a saturated
tracker and an unsaturated control agree on input, output, total and
savings dollars and on `totals()`, with cache read/write and output
tokens) `test_overflowing_passthrough_names_add_no_dollars`, and (round
3) `test_overflow_keeps_cache_aware_savings_for_reviewer_probe` (the
review's exact probe, plus `reset_runtime()` clearing the accumulators).
`test_overflow_conserves_every_priced_stats_dollar_figure` now also
records `tool_schema_saved` and compares `cache_aware_savings_usd` and
`tool_savings_usd`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Round 3 (review 5636282042), with only the `cost.py` change of the new
commit reverted and the new tests kept:
```text
E AssertionError: cache_aware_savings_usd
E assert 0.0 == 0.1
FAILED tests/test_bounded_model_keys.py::test_overflow_conserves_every_priced_stats_dollar_figure
FAILED tests/test_bounded_model_keys.py::test_overflow_keeps_cache_aware_savings_for_reviewer_probe
========================= 2 failed, 8 passed in 5.30s ==========================
```
With the fix:
```text
============================== 10 passed in 6.46s ==============================
```
Round 2 (review 5636023862), with only the `cost.py` change of the new
commit reverted and the new tests kept:
```text
E assert (100000, 0.0) == (100000, 0.25)
E AssertionError: total_input_cost_usd
E assert 0.0 == 0.125
FAILED tests/test_bounded_model_keys.py::test_priced_request_after_saturation_keeps_aggregate_dollars
FAILED tests/test_bounded_model_keys.py::test_overflow_conserves_every_priced_stats_dollar_figure
========================= 2 failed, 7 passed in 4.41s ==========================
```
With the fix:
```text
============================== 9 passed in 4.13s ===============================
```
Round 1 (kept for reference): with the source changes reverted, the 3
tests added in the follow-up commit failed
(`test_overflowing_passthrough_names_stay_out_of_processed_totals`,
`test_estimated_basis_warning_set_is_capped_across_resets`,
`test_savings_history_checkpoints_clip_model_names`).
Round 3 neighbours: `test_*cost*.py`, `test_*savings*.py` (except
`test_savings_items.py`, which needs `respx`, not installed in my
environment), `test_*persistent_metrics*.py`, `test_per_model*.py`,
`test_stats_*.py` and the new file: 505 passed, 4 skipped, 2 failed
(`test_bundled_tools_savings.py::test_ensure_tools_installs_every_tool`
and `::test_ast_grep_slice_saves_tokens`, bundled-tool installs, files
this change does not touch; not re-run on a clean checkout). Round 2
neighbours (`test_persistent_metrics*.py`,
`test_cost_pricing_warning_dedup.py`, `test_cost_budget_*.py`,
`test_cost_tracker_totals.py`, `test_cost_card_attribution.py`,
`test_proxy_savings_history.py`, `test_savings_tracker_zero_price.py`)
plus `test_cost_cache_prices.py`, `test_cost_tracker_counterfactual.py`
and `test_savings_tracker_output_cost.py`, together with the new file:
166 passed, 0 failed. `tests/test_proxy_budget.py` and
`tests/test_proxy_budget_provider_enforcement.py` have 2
websocket-preflight failures
(`test_zero_budget_websocket_preflight_rejected`,
`test_exhausted_budget_rejects_responses_websocket_before_upstream`)
that also fail on a clean checkout of `e995eb9f` in my environment and
are unrelated to this change.
`ruff check` and `ruff format --check` on `headroom/proxy/cost.py` and
`tests/test_bounded_model_keys.py` after the round-3 commit: All checks
passed / 2 files already formatted. `mypy` was not run.
## Real Behavior Proof
- Environment: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn
0.54.0, fastapi 0.158.1, starlette 1.9.0; base `e995eb9f`;
`headroom/_core.abi3.so` copied (untracked, not committed) from the
official `headroom_ai-0.41.0` wheel.
- Exact command / steps: stub upstream on 127.0.0.1:18811 (returns a
message with a `usage` block including `cache_read_input_tokens: 500`,
so each request writes a history point); `HEADROOM_SAVINGS_PATH=<file>
ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18811 python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18810 --no-cache
--no-rate-limit`; 400 `POST /v1/messages` with distinct 1500-character
model names; then `GET /stats`, SIGINT, read the savings file. Before =
base source (the change reverted with `git apply -R`), after = this
change. Rounds 2 and 3: `PYTHONPATH=. python -P -m pytest -q -p
no:cacheprovider tests/test_bounded_model_keys.py`, run with the
`cost.py` change of the new commit reverted (`git apply -R`) and with it
applied.
- Observed result: Rounds 2 and 3: no live-proxy run was done. The money
path is `CostTracker`, driven directly in unit tests. Before the fix,
`CostTracker.totals()` after 200 distinct unpriced zero-token names and
then `gpt-4o` with 100,000 uncached input tokens returned `(100000,
0.0)` while `period_cost_breakdown()['total_usd']` was 0.25; after the
fix it returns `(100000, 0.25)`. In the second test, a saturated tracker
and an unsaturated control now agree on `total_input_cost_usd`,
`output_cost_usd`, `total_cost_usd` and `savings_usd` (before: input 0.0
against 0.125). Round 3, the review's probe (200 unpriced zero-token
labels, then `gpt-4o` with tokens_saved=40,000, tokens_sent=60,000,
cache_read=20,000, cache_write=10,000, uncached=30,000, output=5,000):
`cache_aware_savings_usd` was 0.0 on the saturated tracker against 0.1
on the control; it is now equal, and `tool_savings_usd` (with
`tool_schema_saved=8,000` added) is equal too. Round 1 (live proxy): on
the base commit `cost.per_model`, `persistent_savings.by_model` and the
savings-file `by_model` and `history` carry 400 distinct model names of
up to 1508 characters; `/stats` is 2,706,948 bytes and
`proxy_savings.json` is 1,565,090 bytes. With this change the three
per-model blocks have 201 keys (200 plus `other`) of at most 128
characters, the 400 history points carry model names of at most 128
characters (51,200 characters in total instead of 603,090), `/stats` is
1,444,793 bytes and `proxy_savings.json` is 395,719 bytes. With only the
first commit (before the history clip) the file was 947,609 bytes with
1508-character history names. All 400 requests returned 200 in every
run, and each response carried the stub's model echo, so the requests
reached the stub. `/stats requests.by_model` (Prometheus metrics, not
changed here) still shows 400 long keys.
- Not tested: a priced model through the real proxy after cap saturation
(round 2 is unit-level); the dashboard summary itself (only `stats()`
was compared); a real Anthropic upstream; the passthrough overflow and
the warning-set cap through the real proxy (unit tests only; the batch
routes were not driven); streaming and OpenAI/Gemini routes (they reach
the same `record_tokens` and `record_request`); Windows/macOS; the Rust
proxy.
Raw output:
```text
##### BEFORE (base e995eb9f)
client: requests 400 status counts {200: 400} responses carrying the stub's model echo: 400
/stats .requests.by_model: 400 keys, longest key 1508 chars
/stats .persistent_savings.by_model: 400 keys, longest key 1508 chars
/stats .cost.per_model: 400 keys, longest key 1508 chars
/stats response bytes: 2706948
proxy_savings.json: bytes 1565090 by_model keys 400 longest key 1508 'other' present False
proxy_savings.json history: points 400 longest model 1508 total model chars 603090
##### AFTER (this change)
client: requests 400 status counts {200: 400} responses carrying the stub's model echo: 400
/stats .requests.by_model: 400 keys, longest key 1508 chars
/stats .persistent_savings.by_model: 201 keys, longest key 128 chars
/stats .cost.per_model: 201 keys, longest key 128 chars
/stats response bytes: 1444793
proxy_savings.json: bytes 395719 by_model keys 201 longest key 128 'other' present True
proxy_savings.json history: points 400 longest model 128 total model chars 51200
```
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for more than 200 distinct model
names or names longer than 128 characters; those are bucketed or clipped
in the per-model stats
- Kill switch / disable path: none
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation checklist item: N/A, no user-facing doc changes.
- Per-model attribution is coarsened into `other` when labels fold; the
aggregate dollar fields (`total_input_cost_usd`, `output_cost_usd`,
`total_cost_usd`, `savings_usd`, `cache_aware_savings_usd`,
`tool_savings_usd`) and `totals()` are conserved.
- Not changed: `context_guard._learned_limits` (only written after a
plausible "prompt is too long" upstream error), the `/stats`
`requests.by_model` block (`PrometheusMetrics`, capped in count but not
key length), and loading an already-oversized `by_model` from an
existing `proxy_savings.json`.
- Produced with AI assistance (Claude Code) operated by me (HetCreep);
the reproduction was run on my machine on a clone of the repository.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
170 lines
6.4 KiB
Python
170 lines
6.4 KiB
Python
"""A client-supplied ``model`` string must not grow per-model state without bound."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
from headroom.proxy.cost import CostTracker
|
|
from headroom.proxy.persistent_metrics import (
|
|
MAX_LABEL_LENGTH,
|
|
MAX_TRACKED_MODELS,
|
|
)
|
|
from headroom.proxy.savings_tracker import SavingsTracker
|
|
|
|
_NAMES = MAX_TRACKED_MODELS + 50
|
|
|
|
|
|
def test_cost_tracker_per_model_maps_are_bounded():
|
|
tracker = CostTracker()
|
|
for i in range(_NAMES):
|
|
tracker.record_tokens(f"m-{i}-" + "x" * 2000, tokens_saved=1, tokens_sent=10)
|
|
|
|
assert len(tracker._requests_by_model) == MAX_TRACKED_MODELS + 1
|
|
assert len(tracker._tokens_sent_by_model) == MAX_TRACKED_MODELS + 1
|
|
assert tracker._requests_by_model["other"] == _NAMES - MAX_TRACKED_MODELS
|
|
assert all(len(k) <= MAX_LABEL_LENGTH for k in tracker._requests_by_model)
|
|
# Totals stay exact: over-cap requests are bucketed, not dropped.
|
|
assert sum(tracker._tokens_sent_by_model.values()) == 10 * _NAMES
|
|
|
|
|
|
def test_estimated_basis_warning_set_is_bounded():
|
|
import headroom.proxy.cost as cost_mod
|
|
|
|
cost_mod._warned_estimated_basis_models.clear()
|
|
tracker = CostTracker()
|
|
for i in range(_NAMES):
|
|
tracker.record_tokens(f"w-{i}", tokens_saved=0, tokens_sent=10)
|
|
assert len(cost_mod._warned_estimated_basis_models) <= MAX_TRACKED_MODELS + 1
|
|
cost_mod._warned_estimated_basis_models.clear()
|
|
|
|
|
|
def test_savings_tracker_by_model_is_bounded_and_persisted_small(tmp_path):
|
|
path = tmp_path / "proxy_savings.json"
|
|
tracker = SavingsTracker(path=str(path), save_flush_every=1)
|
|
for i in range(_NAMES):
|
|
tracker.record_request(model=f"s-{i}-" + "y" * 2000, input_tokens=100, tokens_saved=10)
|
|
|
|
by_model = json.loads(path.read_text(encoding="utf-8"))["by_model"]
|
|
assert len(by_model) == MAX_TRACKED_MODELS + 1
|
|
assert all(len(k) <= MAX_LABEL_LENGTH for k in by_model)
|
|
assert by_model["other"]["requests"] == _NAMES - MAX_TRACKED_MODELS
|
|
assert sum(e["tokens_saved"] for e in by_model.values()) == 10 * _NAMES
|
|
|
|
|
|
def test_overflowing_passthrough_names_stay_out_of_processed_totals():
|
|
tracker = CostTracker()
|
|
for i in range(MAX_TRACKED_MODELS):
|
|
tracker.record_tokens(f"passthrough:batch:m{i}", tokens_saved=1, tokens_sent=1)
|
|
tracker.record_tokens("passthrough:batch:next", tokens_saved=10, tokens_sent=90)
|
|
|
|
stats = tracker.stats()
|
|
assert stats["total_tokens_saved"] == 0
|
|
assert stats["total_input_tokens"] == 0
|
|
assert "other" not in tracker._requests_by_model
|
|
assert tracker._requests_by_model["passthrough:other"] == 1
|
|
|
|
|
|
def test_estimated_basis_warning_set_is_capped_across_resets():
|
|
import headroom.proxy.cost as cost_mod
|
|
|
|
cost_mod._warned_estimated_basis_models.clear()
|
|
tracker = CostTracker()
|
|
for i in range(3):
|
|
for j in range(MAX_TRACKED_MODELS):
|
|
tracker.record_tokens(f"r{i}-{j}", tokens_saved=0, tokens_sent=10)
|
|
tracker.reset_runtime()
|
|
assert len(cost_mod._warned_estimated_basis_models) <= MAX_TRACKED_MODELS
|
|
cost_mod._warned_estimated_basis_models.clear()
|
|
|
|
|
|
def test_savings_history_checkpoints_clip_model_names(tmp_path):
|
|
tracker = SavingsTracker(path=str(tmp_path / "proxy_savings.json"), save_flush_every=1)
|
|
tracker.record_request(model="h-" + "z" * 2000, input_tokens=100, tokens_saved=10)
|
|
|
|
history = json.loads((tmp_path / "proxy_savings.json").read_text(encoding="utf-8"))["history"]
|
|
assert history
|
|
assert all(len(point["model"]) <= MAX_LABEL_LENGTH for point in history)
|
|
|
|
|
|
def _saturate_with_unpriced_names(tracker: CostTracker) -> None:
|
|
for i in range(MAX_TRACKED_MODELS):
|
|
tracker.record_tokens(f"unpriced-{i}", tokens_saved=0, tokens_sent=0)
|
|
|
|
|
|
def test_priced_request_after_saturation_keeps_aggregate_dollars():
|
|
control = CostTracker()
|
|
control.record_tokens("gpt-4o", tokens_saved=0, tokens_sent=100_000, uncached_tokens=100_000)
|
|
assert control.totals() == (100_000, 0.25)
|
|
|
|
tracker = CostTracker()
|
|
_saturate_with_unpriced_names(tracker)
|
|
tracker.record_tokens("gpt-4o", tokens_saved=0, tokens_sent=100_000, uncached_tokens=100_000)
|
|
|
|
assert tracker._requests_by_model["other"] == 1
|
|
assert tracker.totals() == (100_000, 0.25)
|
|
assert tracker.totals()[1] == tracker.period_cost_breakdown()["total_usd"]
|
|
assert tracker.stats()["total_input_cost_usd"] == 0.25
|
|
|
|
|
|
def test_overflow_conserves_every_priced_stats_dollar_figure():
|
|
kwargs = {
|
|
"tokens_saved": 40_000,
|
|
"tokens_sent": 60_000,
|
|
"cache_read_tokens": 20_000,
|
|
"cache_write_tokens": 10_000,
|
|
"uncached_tokens": 30_000,
|
|
"output_tokens": 5_000,
|
|
"tool_schema_saved": 8_000,
|
|
}
|
|
control = CostTracker()
|
|
control.record_tokens("gpt-4o", **kwargs)
|
|
tracker = CostTracker()
|
|
_saturate_with_unpriced_names(tracker)
|
|
tracker.record_tokens("gpt-4o", **kwargs)
|
|
|
|
want, got = control.stats(), tracker.stats()
|
|
assert want["total_cost_usd"] > 0
|
|
assert want["cache_aware_savings_usd"] > 0
|
|
assert want["tool_savings_usd"] > 0
|
|
for field in (
|
|
"total_input_cost_usd",
|
|
"output_cost_usd",
|
|
"total_cost_usd",
|
|
"savings_usd",
|
|
"cache_aware_savings_usd",
|
|
"tool_savings_usd",
|
|
):
|
|
assert got[field] == want[field], field
|
|
assert tracker.totals() == control.totals()
|
|
|
|
|
|
def test_overflow_keeps_cache_aware_savings_for_reviewer_probe():
|
|
kwargs = {
|
|
"tokens_saved": 40_000,
|
|
"tokens_sent": 60_000,
|
|
"cache_read_tokens": 20_000,
|
|
"cache_write_tokens": 10_000,
|
|
"uncached_tokens": 30_000,
|
|
"output_tokens": 5_000,
|
|
}
|
|
control = CostTracker()
|
|
control.record_tokens("gpt-4o", **kwargs)
|
|
tracker = CostTracker()
|
|
_saturate_with_unpriced_names(tracker)
|
|
tracker.record_tokens("gpt-4o", **kwargs)
|
|
assert control.stats()["cache_aware_savings_usd"] > 0
|
|
assert tracker.stats()["cache_aware_savings_usd"] == control.stats()["cache_aware_savings_usd"]
|
|
tracker.reset_runtime()
|
|
assert tracker.stats()["cache_aware_savings_usd"] == 0.0
|
|
assert tracker.stats()["tool_savings_usd"] == 0.0
|
|
|
|
|
|
def test_overflowing_passthrough_names_add_no_dollars():
|
|
tracker = CostTracker()
|
|
for i in range(MAX_TRACKED_MODELS):
|
|
tracker.record_tokens(f"passthrough:batch:m{i}", tokens_saved=0, tokens_sent=1)
|
|
tracker.record_tokens(
|
|
"passthrough:gpt-4o", tokens_saved=0, tokens_sent=100_000, uncached_tokens=100_000
|
|
)
|
|
assert tracker.stats()["total_input_cost_usd"] == 0.0
|