mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(reporting): price the savings tile instead of fabricating $0.00 (#3821)
## Description `generate_report()` priced nothing and reported the absence as a number. It called `headroom.utils.estimate_cost()` without a `provider`; that function returns `None` before doing any work (`headroom/utils.py:238-239`), and the call site swallowed the `None` with `or 0.0`: ```python cost_before = estimate_cost(stats["total_tokens_before"], 0, "gpt-4o") or 0.0 cost_after = estimate_cost(stats["total_tokens_after"], 0, "gpt-4o") or 0.0 estimated_savings = format_cost(cost_before - cost_after) ``` Both sides are therefore always `0.0`, so the **Est. Cost Savings** tile read `$0.0000` on every report the function ever produced — verified on the commit that shipped as 0.39.1. This is a fabricated fact, not a missing value: a report saying "you saved $0" while its neighbouring tile says "245,000 tokens saved" is the same failure #3744 just fixed for negative savings. The fix prices through `ModelRegistry.estimate_cost()` — the repo's own LiteLLM-database pricer, which needs no provider instance and no credentials — and when the rate card genuinely has no entry it renders `unavailable` instead of `$0.0000`. Closes #3820 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/reporting/generator.py`: price the two totals through `ModelRegistry.estimate_cost("gpt-4o", tokens, 0)`; report `unavailable` when a price is unobtainable instead of coercing it to `0.0`. The pricer is imported inside `generate_report()` — imported at module level it costs **1.19 s** and pulls `litellm` into a module that currently imports in **0.071 s** with `litellm` unloaded (both numbers measured below), which would tax every importer of `headroom.reporting` for a value only the HTML report needs. - `tests/test_reporting.py`: two new tests that leave pricing unpatched (the report is priced against the real bundled rate card), one for the unpriceable path, plus removal of the two stubs that hid the bug. `test_generate_report_writes_output_and_closes_storage` patched `generator.estimate_cost` with `lambda tokens, output_tokens, model: {100: 2.0, 0: None}[tokens]` — a stub that answers the exact 3-positional call shape the real function rejects, and whose `$2.00` the test then asserted. That is why 11 green tests coexisted with a permanently wrong tile. ## Testing - [x] Unit tests pass (`pytest`) — for the suites below; see *Not tested* for the local limit - [x] Linting passes (`ruff check .`) - [ ] Type checking: `mypy headroom` is already red on `main` (see *Gates*) — my file is clean and I add no errors - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Red before the fix — the two new tests, everything else untouched, run against `main` at `d13e196` with only the test file changed: ```text tests/test_reporting.py ..........FF. [100%] _________________ test_generate_report_prices_the_savings_tile _________________ assert stats["estimated_savings"] == format_cost(before - after) E AssertionError: assert '$0.0000' == '$0.61' __________ test_generate_report_says_unavailable_when_it_cannot_price __________ assert stats["estimated_savings"] == "unavailable" E AssertionError: assert '$0.0000' == 'unavailable' ========================= 2 failed, 11 passed in 1.51s ========================= ``` Green after: ```text tests/test_reporting.py ............. [100%] ============================== 13 passed in 1.40s ============================== ``` Control suites around the changed seams (`test_reporting`, `test_utils`, `test_pricing`, `test_pricing_litellm`, `test_models`), same command on the base worktree for the pre-existing subset: ```text ============================== 69 passed in 6.21s ============================== ``` Mutation checks — each regression is caught by the test that owns it: ```text # `or 0.0` swallow put back: FAILED tests/test_reporting.py::test_generate_report_says_unavailable_when_it_cannot_price E AssertionError: assert '$0.0000' == 'unavailable' ========================= 1 failed, 12 passed in 6.84s ========================= # None branch dropped: FAILED tests/test_reporting.py::test_generate_report_says_unavailable_when_it_cannot_price E TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType' ========================= 1 failed, 12 passed in 6.50s ========================= ``` Gates: ```text $ .venv/bin/python -m ruff check . All checks passed! $ .venv/bin/python -m ruff format --check . 1658 files already formatted $ .venv/bin/python -m mypy headroom headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context [truthy-function] Found 1 error in 1 file (checked 543 source files) $ .venv/bin/python -m mypy headroom/reporting/generator.py Success: no issues found in 1 source file ``` The repo-wide `mypy headroom` line is **not** caused by this change: the same command on an untouched `main` worktree at `d13e196` produces the identical single error in `headroom/integrations/langchain/chat_model.py`, a file this PR does not touch. I did not fix it here — it belongs to whichever change owns that adapter, and folding it in would widen a two-file diff. Import-cost numbers behind keeping the pricer inside the function: ```text import headroom.reporting: 0.071s litellm_loaded=False # before, and after this patch from headroom.models.registry import ModelRegistry: 1.192s litellm_loaded=True ``` **Not tested locally:** the full `pytest tests`. 211 test modules import the compiled `headroom._core`, which I did not build (`uv sync --extra dev --no-install-project`), and the collectible remainder is slow enough to need CI's sharding — `tests/test_bundled_tools_savings.py` alone takes 73.46 s on the base commit `d13e196`. I checked that file separately and it passes; nothing in it touches `headroom.reporting`. ## Real Behavior Proof - **Environment:** macOS 27 arm64, Python 3.14.4, source tree at `main` = `d13e196` (`chore: release 0.39.1 (#3807)`, `_version` reports `0.39.2-dev`), deps from `uv sync --extra dev --no-install-project`, no provider credentials, no network calls in the path under test. - **Exact command / steps:** write two real requests into a real SQLite metrics DB through `create_storage()`, call the documented `headroom.generate_report()` (it is exported from `headroom/__init__.py` and in `docs/content/docs/api-reference.mdx`), then read the tile text out of the generated HTML with a regex rather than looking at the page: ```python storage = create_storage("sqlite:///tmp/r.db") for i, (before, after) in enumerate([(200_000, 60_000), (150_000, 45_000)]): storage.save(RequestMetrics( request_id=f"req-{i}", timestamp=datetime(2026, 9, 27, 10, i), model="gpt-4o", stream=False, mode="optimize", tokens_input_before=before, tokens_input_after=after, tokens_output=1000)) html = open(generate_report("sqlite:///tmp/r.db", "/tmp/report.html")).read() ``` - **Observed result — before the patch** (this is the issue's reproduction, run on `d13e196`): ```text stored summary: {'total_requests': 2, 'total_tokens_before': 350000, 'total_tokens_after': 105000, 'total_tokens_saved': 245000} report written: /tmp/hr-repro2/report.html 'Tokens Saved' tile -> 245,000 'Est. Cost Savings' tile -> $0.0000 ``` - **Observed result — after the patch**, same script, same DB contents, only `headroom/reporting/generator.py` changed: ```text stored summary: {'total_requests': 2, 'total_tokens_before': 350000, 'total_tokens_after': 105000, 'total_tokens_saved': 245000} report written: /tmp/hr-repro2b/report.html 'Tokens Saved' tile -> 245,000 'Est. Cost Savings' tile -> $0.61 registry gpt-4o pricing: before=$0.8750 after=$0.2625 savings=$0.6125 ``` `$0.61` is `format_cost(0.8750 - 0.2625)`, i.e. the tile now agrees with the rate card the report already names, and the number moves when the stored tokens move. - **Not tested:** Windows; the `headroom-ai[proxy]` wheel's Rust path (`headroom/reporting/` is pure Python with no Rust mirror — I ran it from the source tree, not the wheel); opening the report in a browser (the tile text was read from the generated HTML, not screenshotted); what happens to this tile when a deployment's traffic is mostly a model LiteLLM has no price for — that case is now labelled `unavailable` and I have not seen it in real traffic. ## Runtime Rollout Safety - Rollout-managed feature(s): none — `generate_report()` is a plain SDK call, not behind a rollout flag. - Minimum rollout channel: n/a. - Stable/default behavior changed: yes, deliberately. Every report's **Est. Cost Savings** tile changes from the constant `$0.0000` to a computed price, or to `unavailable` when the rate card has no entry. No other tile, no stored field, no on-disk format changes. - Kill switch / disable path: none needed; reverting the single commit restores the previous (incorrect) behaviour. - Unsafe override required: none. - Qualification impact: none known — no qualification asserts on this string, and the two tests that did assert on it asserted on their own stub's return value. - Rollback path: `git revert` the commit; it touches two files and adds no state. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - **Docs left alone deliberately.** `docs/content/docs/api-reference.mdx` documents `generate_report()` without promising a dollar figure, so nothing becomes wrong here. I did not add a line about the `unavailable` string — one word in a tile that needs no legend felt like noise, but I'll document it if you'd rather it be explicit. - **The hardcoded `"gpt-4o"` rate card is untouched on purpose.** A model-agnostic report pricing all savings at one reference model is a real question, and it is a different change from "the tile always said zero". Widening this PR would make the before/after harder to read. - **`utils.estimate_cost()`'s `None` contract is left alone.** The silent-swallow hazard exists for any future caller too, but `headroom/utils.py` is being edited in the open by #3116, so I stayed out of that file rather than create a conflict for them. - Diff is 2 files, +72/−13. No dependency, lockfile, or generated file is touched. Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
d90b32008d
commit
f519fa8b97
@@ -7,7 +7,7 @@ from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from ..storage import create_storage
|
||||
from ..utils import estimate_cost, format_cost
|
||||
from ..utils import format_cost
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
@@ -337,9 +337,16 @@ def generate_report(
|
||||
tpm_multiplier = 1.0
|
||||
|
||||
# Estimate cost savings (using gpt-4o pricing)
|
||||
cost_before = estimate_cost(stats["total_tokens_before"], 0, "gpt-4o") or 0.0
|
||||
cost_after = estimate_cost(stats["total_tokens_after"], 0, "gpt-4o") or 0.0
|
||||
estimated_savings = format_cost(cost_before - cost_after)
|
||||
# Imported here: the LiteLLM-backed pricer costs ~1.2s of import time.
|
||||
from ..models.registry import ModelRegistry
|
||||
|
||||
cost_before = ModelRegistry.estimate_cost("gpt-4o", stats["total_tokens_before"], 0)
|
||||
cost_after = ModelRegistry.estimate_cost("gpt-4o", stats["total_tokens_after"], 0)
|
||||
# "Cannot price this" is not "$0 saved", so an unpriced report says so.
|
||||
if cost_before is None or cost_after is None:
|
||||
estimated_savings = "unavailable"
|
||||
else:
|
||||
estimated_savings = format_cost(cost_before - cost_after)
|
||||
|
||||
stats["tpm_multiplier"] = tpm_multiplier
|
||||
stats["estimated_savings"] = estimated_savings
|
||||
|
||||
+61
-9
@@ -9,7 +9,9 @@ from types import SimpleNamespace
|
||||
import pytest
|
||||
|
||||
import headroom.reporting as reporting
|
||||
from headroom.models.registry import ModelRegistry
|
||||
from headroom.reporting import generator
|
||||
from headroom.utils import format_cost
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -255,12 +257,6 @@ def test_generate_report_writes_output_and_closes_storage(
|
||||
generator, "_generate_recommendations", lambda *args: [{"title": "Keep going"}]
|
||||
)
|
||||
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
|
||||
monkeypatch.setattr(
|
||||
generator,
|
||||
"estimate_cost",
|
||||
lambda tokens, output_tokens, model: {100: 2.0, 0: None}[tokens],
|
||||
)
|
||||
monkeypatch.setattr(generator, "format_cost", lambda cost: f"${cost:.2f}")
|
||||
|
||||
output_path = tmp_path / "report.html"
|
||||
result = generator.generate_report(
|
||||
@@ -274,10 +270,68 @@ def test_generate_report_writes_output_and_closes_storage(
|
||||
assert output_path.read_text() == "<html>report</html>"
|
||||
assert render_calls[0]["period"] == expected_period
|
||||
assert render_calls[0]["stats"]["tpm_multiplier"] == 100.0
|
||||
assert render_calls[0]["stats"]["estimated_savings"] == "$2.00"
|
||||
assert storage.closed is True
|
||||
|
||||
|
||||
def _capture_stats(monkeypatch, tmp_path, stats: dict) -> dict:
|
||||
"""Run generate_report() over *stats* and hand back the rendered stats dict.
|
||||
|
||||
Only the storage and template seams are patched: pricing stays the product's
|
||||
own, so a report that cannot price itself shows up here as a wrong number.
|
||||
"""
|
||||
render_calls: list[dict] = []
|
||||
|
||||
class FakeTemplate:
|
||||
def render(self, **kwargs) -> str:
|
||||
render_calls.append(kwargs)
|
||||
return "<html>report</html>"
|
||||
|
||||
monkeypatch.setattr(generator, "create_storage", lambda store_url: FakeStorage(stats, []))
|
||||
monkeypatch.setattr(generator, "_build_waste_histogram", lambda *args: [])
|
||||
monkeypatch.setattr(generator, "_get_top_waste_requests", lambda *args, **kwargs: [])
|
||||
monkeypatch.setattr(generator, "_generate_recommendations", lambda *args: [])
|
||||
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
|
||||
|
||||
generator.generate_report("sqlite:///demo.db", output_path=str(tmp_path / "report.html"))
|
||||
return render_calls[0]["stats"]
|
||||
|
||||
|
||||
SAVINGS_STATS = {
|
||||
"total_requests": 2,
|
||||
"total_tokens_saved": 245_000,
|
||||
"avg_tokens_saved": 122_500,
|
||||
"total_tokens_before": 350_000,
|
||||
"total_tokens_after": 105_000,
|
||||
"avg_cache_alignment": 80,
|
||||
"audit_count": 0,
|
||||
"optimize_count": 2,
|
||||
}
|
||||
|
||||
|
||||
def test_generate_report_prices_the_savings_tile(monkeypatch, tmp_path) -> None:
|
||||
"""245k tokens saved must cost something, not read as a $0.0000 claim."""
|
||||
stats = _capture_stats(monkeypatch, tmp_path, dict(SAVINGS_STATS))
|
||||
|
||||
before = ModelRegistry.estimate_cost("gpt-4o", 350_000, 0)
|
||||
after = ModelRegistry.estimate_cost("gpt-4o", 105_000, 0)
|
||||
assert before is not None and after is not None, "gpt-4o left the bundled rate card"
|
||||
assert stats["estimated_savings"] == format_cost(before - after)
|
||||
assert stats["estimated_savings"] != "$0.00"
|
||||
|
||||
|
||||
def test_generate_report_says_unavailable_when_it_cannot_price(monkeypatch, tmp_path) -> None:
|
||||
"""An unknown price is a missing number, not zero saved."""
|
||||
monkeypatch.setattr(
|
||||
ModelRegistry,
|
||||
"estimate_cost",
|
||||
classmethod(lambda cls, model, *args, **kwargs: None),
|
||||
)
|
||||
|
||||
stats = _capture_stats(monkeypatch, tmp_path, dict(SAVINGS_STATS))
|
||||
|
||||
assert stats["estimated_savings"] == "unavailable"
|
||||
|
||||
|
||||
def test_generate_report_closes_storage_when_render_fails(monkeypatch, tmp_path) -> None:
|
||||
storage = FakeStorage(
|
||||
{
|
||||
@@ -302,8 +356,6 @@ def test_generate_report_closes_storage_when_render_fails(monkeypatch, tmp_path)
|
||||
monkeypatch.setattr(generator, "_get_top_waste_requests", lambda *args, **kwargs: [])
|
||||
monkeypatch.setattr(generator, "_generate_recommendations", lambda *args: [])
|
||||
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
|
||||
monkeypatch.setattr(generator, "estimate_cost", lambda *args: 0.0)
|
||||
monkeypatch.setattr(generator, "format_cost", lambda cost: "$0.00")
|
||||
|
||||
with pytest.raises(RuntimeError, match="boom"):
|
||||
generator.generate_report("sqlite:///demo.db", output_path=str(tmp_path / "report.html"))
|
||||
|
||||
Reference in New Issue
Block a user