Files
headroom/tests/test_cost_budget_performance.py
danieldutuandJerrettDavis 41c15b311c fix(proxy): keep in-period costs in the budget ledger when the entry cap is reached (#4448)
## Description

With `CostTracker.MAX_COST_ENTRIES` at 5, a $50.00 monthly budget and
$3.00 per request, 25 of 25 requests returned HTTP 200 and the stub
upstream received 25 requests. The period total stopped at $15.00, the
sum of the newest 5 records, so `check_budget()` never reached the
limit.

Cause: `_record_budget_cost` (`headroom/proxy/cost.py:1269-1270` at
e995eb9f) popped the oldest ledger record and subtracted its cost from
the period totals when `_budget_costs` held `MAX_COST_ENTRIES` records.
It did this for records still inside the budget period. The same cap
applies at 100,000 records in production; the issue did not measure a
run that long.

Fix: I removed the count-based eviction. `_refresh_budget_window`
already evicts each expired record once, so the ledger holds the current
period only. The #3368 amortized O(1) aggregate is unchanged. The
reporting history `_costs` keeps its 100,000-entry cap.

Trade-off: ledger memory grows with the number of requests in one
period. I chose this over folding the oldest in-period records into one
aggregate record because it needs no new record type and the in-period
total stays exact. Question for maintainers: is unbounded growth within
one period acceptable, or do you want the fold-into-aggregate option? I
can change the PR to that.

@HetCreep reported the defect in #4416 and described this option. I
started from the report and wrote the change independently.

Closes #4416

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/cost.py`: `_record_budget_cost` no longer evicts the
oldest ledger record when `MAX_COST_ENTRIES` is reached. Period-window
eviction in `_refresh_budget_window` is unchanged, so the ledger is
bounded by the period.
- `tests/test_cost_budget_performance.py`:
`test_budget_aggregate_keeps_existing_100k_record_cap` renamed to
`test_budget_ledger_keeps_in_period_entries_past_record_cap`; it now
asserts the in-period total 7.0 over 3 records instead of the
under-count 6.0 over 2.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Scope of what ran: `uv run pytest tests/test_cost_budget_*.py` (31
tests), not the full suite. `mypy headroom` was not run.

### Test Output

```text
$ uv run pytest tests/test_cost_budget_*.py
31 passed

$ uv run ruff check .
All checks passed!

$ uv run ruff format --check .
1812 files already formatted

# before the fix, same test file at e995eb9f with the updated assertion:
FAILED tests/test_cost_budget_performance.py::test_budget_ledger_keeps_in_period_entries_past_record_cap - assert 6.0 == 7.0
```

## Real Behavior Proof

- Environment: macOS (Darwin 27.0.0), Python 3.12, `uv run`, Anthropic
handler, model `claude-sonnet-4-5`, proxy at `0fd3f8f5`.
- Exact command / steps: `stub.py` listens on `127.0.0.1:18999` and
answers every POST with `usage.input_tokens=1000000` ($3.00 at
claude-sonnet-4-5 pricing). `proxy.py` sets
`CostTracker.MAX_COST_ENTRIES = 5` and runs `headroom proxy --port 18787
--budget 50 --budget-period monthly` against the stub. Then 25 requests:
`for i in $(seq 25); do curl -s -o /dev/null -w '%{http_code}' -X POST
http://127.0.0.1:18787/v1/messages -H 'content-type: application/json'
-d
"{\"model\":\"claude-sonnet-4-5\",\"max_tokens\":16,\"messages\":[{\"role\":\"user\",\"content\":\"hi
$i\"}]}"; done`
- Observed result: before the fix (`main` at e995eb9f) requests 1-25
returned 200 and the stub counted 25 upstream requests, with the period
total stuck at $15.00. After the fix, requests 1-17 returned 200,
requests 18-25 returned 429, the stub counted 17 upstream requests, and
17 requests booked $51.00 against the $50.00 limit.
- Not tested: a run with 100,000 records, other provider handlers,
hourly and daily periods end to end, streaming responses, Linux and
Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. Budget enforcement is not behind a
rollout channel.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes. With `--budget` set and more
than `MAX_COST_ENTRIES` (default 100,000) requests in one period, the
proxy now returns 429 once the period total passes the budget. Before,
the total under-counted and the budget never tripped. Without
`--budget`, nothing changes.
- Kill switch / disable path: run without `--budget`; the ledger path is
not used.
- Unsafe override required: no.
- Qualification impact: none known. No qualification suite covers budget
enforcement.
- Rollback path: revert this commit. The ledger returns to count-based
eviction and the under-count.

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

n/a

## Additional Notes

Comments and documentation: the change removes 2 lines and adds 1
comment; no user-facing documentation describes `MAX_COST_ENTRIES`, so
none changed. `mypy headroom` was not run; the diff touches no
annotations. Memory trade-off: the ledger now grows with the number of
requests in one period. I chose this over folding the oldest in-period
records into one aggregate record because it needs no new record type
and the in-period total stays exact. If you prefer the aggregate record,
I can send that as a follow-up.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:37:31 -07:00

103 lines
3.5 KiB
Python

"""Incremental budget aggregation regression coverage (#3367)."""
from __future__ import annotations
from datetime import datetime
import pytest
from headroom.proxy.budget_basis_policy import COST_BASIS_ESTIMATED, COST_BASIS_MEASURED
from headroom.proxy.cost import CostEntry, CostTracker
def test_budget_breakdown_does_not_scan_reporting_history() -> None:
tracker = CostTracker(budget_limit_usd=100.0)
now = datetime.now()
tracker._costs.extend(
CostEntry(now, 1.0, COST_BASIS_MEASURED) for _ in range(tracker.MAX_COST_ENTRIES)
)
tracker._record_budget_cost(CostEntry(now, 2.0, COST_BASIS_MEASURED))
class NonIterableHistory:
def __iter__(self):
raise AssertionError("budget check rescanned reporting history")
tracker._costs = NonIterableHistory() # type: ignore[assignment]
assert tracker.check_budget() == (True, 98.0)
def test_budget_aggregate_preserves_basis_totals_and_policy() -> None:
tracker = CostTracker(
budget_limit_usd=10.0,
estimated_basis_policy="ignore",
)
now = datetime.now()
tracker._record_budget_cost(CostEntry(now, 3.0, COST_BASIS_MEASURED))
tracker._record_budget_cost(CostEntry(now, 20.0, COST_BASIS_ESTIMATED))
assert tracker.period_cost_breakdown() == {
"period": "daily",
"policy": "ignore",
"total_usd": 23.0,
"measured_usd": 3.0,
"estimated_usd": 20.0,
"estimated_pct": pytest.approx(87.0),
"records": 2,
"estimated_records": 1,
}
assert tracker.check_budget() == (True, 7.0)
@pytest.mark.parametrize(
("period", "now", "expired_at", "current_at"),
[
(
"hourly",
datetime(2026, 8, 31, 12),
datetime(2026, 8, 31, 10),
datetime(2026, 8, 31, 11, 30),
),
(
"daily",
datetime(2026, 8, 31, 12),
datetime(2026, 8, 30, 23, 59),
datetime(2026, 8, 31, 1),
),
(
"monthly",
datetime(2026, 8, 31, 12),
datetime(2026, 7, 31, 23, 59),
datetime(2026, 8, 1),
),
],
)
def test_budget_window_evicts_expired_entries_once(period, now, expired_at, current_at) -> None:
tracker = CostTracker(budget_limit_usd=100.0, budget_period=period)
tracker._record_budget_cost(CostEntry(expired_at, 40.0, COST_BASIS_MEASURED))
tracker._record_budget_cost(CostEntry(current_at, 5.0, COST_BASIS_ESTIMATED))
tracker._refresh_budget_window(now)
assert tracker._budget_measured_usd == 0.0
assert tracker._budget_estimated_usd == 5.0
assert tracker._budget_estimated_records == 1
assert len(tracker._budget_costs) == 1
# Re-reading the same window is constant-time and must not subtract twice.
tracker._refresh_budget_window(now)
assert tracker._budget_estimated_usd == 5.0
assert len(tracker._budget_costs) == 1
def test_budget_ledger_keeps_in_period_entries_past_record_cap() -> None:
tracker = CostTracker(budget_limit_usd=1_000_000.0)
tracker.MAX_COST_ENTRIES = 2
now = datetime.now()
tracker._record_budget_cost(CostEntry(now, 1.0, COST_BASIS_MEASURED))
tracker._record_budget_cost(CostEntry(now, 2.0, COST_BASIS_ESTIMATED))
tracker._record_budget_cost(CostEntry(now, 4.0, COST_BASIS_MEASURED))
assert tracker.period_cost_breakdown()["total_usd"] == 7.0
assert tracker.period_cost_breakdown()["records"] == 3
assert tracker.period_cost_breakdown()["estimated_records"] == 1