mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): keep in-period costs in the budget ledger when the entry cap is reached (#4448)
## Description With `CostTracker.MAX_COST_ENTRIES` at 5, a $50.00 monthly budget and $3.00 per request, 25 of 25 requests returned HTTP 200 and the stub upstream received 25 requests. The period total stopped at $15.00, the sum of the newest 5 records, so `check_budget()` never reached the limit. Cause: `_record_budget_cost` (`headroom/proxy/cost.py:1269-1270` ate995eb9f) popped the oldest ledger record and subtracted its cost from the period totals when `_budget_costs` held `MAX_COST_ENTRIES` records. It did this for records still inside the budget period. The same cap applies at 100,000 records in production; the issue did not measure a run that long. Fix: I removed the count-based eviction. `_refresh_budget_window` already evicts each expired record once, so the ledger holds the current period only. The #3368 amortized O(1) aggregate is unchanged. The reporting history `_costs` keeps its 100,000-entry cap. Trade-off: ledger memory grows with the number of requests in one period. I chose this over folding the oldest in-period records into one aggregate record because it needs no new record type and the in-period total stays exact. Question for maintainers: is unbounded growth within one period acceptable, or do you want the fold-into-aggregate option? I can change the PR to that. @HetCreep reported the defect in #4416 and described this option. I started from the report and wrote the change independently. Closes #4416 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/cost.py`: `_record_budget_cost` no longer evicts the oldest ledger record when `MAX_COST_ENTRIES` is reached. Period-window eviction in `_refresh_budget_window` is unchanged, so the ledger is bounded by the period. - `tests/test_cost_budget_performance.py`: `test_budget_aggregate_keeps_existing_100k_record_cap` renamed to `test_budget_ledger_keeps_in_period_entries_past_record_cap`; it now asserts the in-period total 7.0 over 3 records instead of the under-count 6.0 over 2. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Scope of what ran: `uv run pytest tests/test_cost_budget_*.py` (31 tests), not the full suite. `mypy headroom` was not run. ### Test Output ```text $ uv run pytest tests/test_cost_budget_*.py 31 passed $ uv run ruff check . All checks passed! $ uv run ruff format --check . 1812 files already formatted # before the fix, same test file ate995eb9fwith the updated assertion: FAILED tests/test_cost_budget_performance.py::test_budget_ledger_keeps_in_period_entries_past_record_cap - assert 6.0 == 7.0 ``` ## Real Behavior Proof - Environment: macOS (Darwin 27.0.0), Python 3.12, `uv run`, Anthropic handler, model `claude-sonnet-4-5`, proxy at `0fd3f8f5`. - Exact command / steps: `stub.py` listens on `127.0.0.1:18999` and answers every POST with `usage.input_tokens=1000000` ($3.00 at claude-sonnet-4-5 pricing). `proxy.py` sets `CostTracker.MAX_COST_ENTRIES = 5` and runs `headroom proxy --port 18787 --budget 50 --budget-period monthly` against the stub. Then 25 requests: `for i in $(seq 25); do curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1:18787/v1/messages -H 'content-type: application/json' -d "{\"model\":\"claude-sonnet-4-5\",\"max_tokens\":16,\"messages\":[{\"role\":\"user\",\"content\":\"hi $i\"}]}"; done` - Observed result: before the fix (`main` ate995eb9f) requests 1-25 returned 200 and the stub counted 25 upstream requests, with the period total stuck at $15.00. After the fix, requests 1-17 returned 200, requests 18-25 returned 429, the stub counted 17 upstream requests, and 17 requests booked $51.00 against the $50.00 limit. - Not tested: a run with 100,000 records, other provider handlers, hourly and daily periods end to end, streaming responses, Linux and Windows. ## Runtime Rollout Safety - Rollout-managed feature(s): none. Budget enforcement is not behind a rollout channel. - Minimum rollout channel: n/a. - Stable/default behavior changed: yes. With `--budget` set and more than `MAX_COST_ENTRIES` (default 100,000) requests in one period, the proxy now returns 429 once the period total passes the budget. Before, the total under-counted and the budget never tripped. Without `--budget`, nothing changes. - Kill switch / disable path: run without `--budget`; the ledger path is not used. - Unsafe override required: no. - Qualification impact: none known. No qualification suite covers budget enforcement. - Rollback path: revert this commit. The ledger returns to count-based eviction and the under-count. ## Review Readiness - [x] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) n/a ## Additional Notes Comments and documentation: the change removes 2 lines and adds 1 comment; no user-facing documentation describes `MAX_COST_ENTRIES`, so none changed. `mypy headroom` was not run; the diff touches no annotations. Memory trade-off: the ledger now grows with the number of requests in one period. I chose this over folding the oldest in-period records into one aggregate record because it needs no new record type and the in-period total stays exact. If you prefer the aggregate record, I can send that as a follow-up. Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
a616232d75
commit
41c15b311c
@@ -873,7 +873,9 @@ class CostTracker:
|
||||
# the reporting history, while this deque evicts entries as soon as the
|
||||
# configured hourly/daily/monthly window advances. Each entry is added
|
||||
# and removed once, making expiry amortized O(1) instead of rescanning
|
||||
# up to MAX_COST_ENTRIES before every request (#3367).
|
||||
# up to MAX_COST_ENTRIES before every request (#3367). It is bounded by
|
||||
# the period, not by MAX_COST_ENTRIES: evicting an in-period entry would
|
||||
# drop spend that still counts against the budget (#4416).
|
||||
self._budget_costs: deque[CostEntry] = deque()
|
||||
self._budget_measured_usd = 0.0
|
||||
self._budget_estimated_usd = 0.0
|
||||
@@ -1266,8 +1268,6 @@ class CostTracker:
|
||||
def _record_budget_cost(self, entry: CostEntry) -> None:
|
||||
"""Add one entry to the current-period aggregate in amortized O(1)."""
|
||||
self._refresh_budget_window(entry.timestamp)
|
||||
if len(self._budget_costs) >= self.MAX_COST_ENTRIES:
|
||||
self._subtract_budget_cost(self._budget_costs.popleft())
|
||||
self._budget_costs.append(entry)
|
||||
if entry.basis == COST_BASIS_ESTIMATED:
|
||||
self._budget_estimated_usd += entry.cost_usd
|
||||
|
||||
@@ -89,7 +89,7 @@ def test_budget_window_evicts_expired_entries_once(period, now, expired_at, curr
|
||||
assert len(tracker._budget_costs) == 1
|
||||
|
||||
|
||||
def test_budget_aggregate_keeps_existing_100k_record_cap() -> None:
|
||||
def test_budget_ledger_keeps_in_period_entries_past_record_cap() -> None:
|
||||
tracker = CostTracker(budget_limit_usd=1_000_000.0)
|
||||
tracker.MAX_COST_ENTRIES = 2
|
||||
now = datetime.now()
|
||||
@@ -97,6 +97,6 @@ def test_budget_aggregate_keeps_existing_100k_record_cap() -> None:
|
||||
tracker._record_budget_cost(CostEntry(now, 2.0, COST_BASIS_ESTIMATED))
|
||||
tracker._record_budget_cost(CostEntry(now, 4.0, COST_BASIS_MEASURED))
|
||||
|
||||
assert tracker.period_cost_breakdown()["total_usd"] == 6.0
|
||||
assert tracker.period_cost_breakdown()["records"] == 2
|
||||
assert tracker.period_cost_breakdown()["total_usd"] == 7.0
|
||||
assert tracker.period_cost_breakdown()["records"] == 3
|
||||
assert tracker.period_cost_breakdown()["estimated_records"] == 1
|
||||
|
||||
Reference in New Issue
Block a user