fix(reporting): price the savings tile instead of fabricating $0.00 (#3821)

## Description

`generate_report()` priced nothing and reported the absence as a number.
It called `headroom.utils.estimate_cost()` without a `provider`; that
function returns `None` before doing any work
(`headroom/utils.py:238-239`), and the call site swallowed the `None`
with `or 0.0`:

```python
cost_before = estimate_cost(stats["total_tokens_before"], 0, "gpt-4o") or 0.0
cost_after = estimate_cost(stats["total_tokens_after"], 0, "gpt-4o") or 0.0
estimated_savings = format_cost(cost_before - cost_after)
```

Both sides are therefore always `0.0`, so the **Est. Cost Savings** tile
read `$0.0000` on every report the function ever produced — verified on
the commit that shipped as 0.39.1. This is a fabricated fact, not a
missing value: a report saying "you saved $0" while its neighbouring
tile says "245,000 tokens saved" is the same failure #3744 just fixed
for negative savings.

The fix prices through `ModelRegistry.estimate_cost()` — the repo's own
LiteLLM-database pricer, which needs no provider instance and no
credentials — and when the rate card genuinely has no entry it renders
`unavailable` instead of `$0.0000`.

Closes #3820

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/reporting/generator.py`: price the two totals through
`ModelRegistry.estimate_cost("gpt-4o", tokens, 0)`; report `unavailable`
when a price is unobtainable instead of coercing it to `0.0`. The pricer
is imported inside `generate_report()` — imported at module level it
costs **1.19 s** and pulls `litellm` into a module that currently
imports in **0.071 s** with `litellm` unloaded (both numbers measured
below), which would tax every importer of `headroom.reporting` for a
value only the HTML report needs.
- `tests/test_reporting.py`: two new tests that leave pricing unpatched
(the report is priced against the real bundled rate card), one for the
unpriceable path, plus removal of the two stubs that hid the bug.
`test_generate_report_writes_output_and_closes_storage` patched
`generator.estimate_cost` with `lambda tokens, output_tokens, model:
{100: 2.0, 0: None}[tokens]` — a stub that answers the exact
3-positional call shape the real function rejects, and whose `$2.00` the
test then asserted. That is why 11 green tests coexisted with a
permanently wrong tile.

## Testing

- [x] Unit tests pass (`pytest`) — for the suites below; see *Not
tested* for the local limit
- [x] Linting passes (`ruff check .`)
- [ ] Type checking: `mypy headroom` is already red on `main` (see
*Gates*) — my file is clean and I add no errors
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Red before the fix — the two new tests, everything else untouched, run
against `main` at `d13e196` with only the test file changed:

```text
tests/test_reporting.py ..........FF.                                    [100%]

_________________ test_generate_report_prices_the_savings_tile _________________
    assert stats["estimated_savings"] == format_cost(before - after)
E   AssertionError: assert '$0.0000' == '$0.61'
__________ test_generate_report_says_unavailable_when_it_cannot_price __________
    assert stats["estimated_savings"] == "unavailable"
E   AssertionError: assert '$0.0000' == 'unavailable'
========================= 2 failed, 11 passed in 1.51s =========================
```

Green after:

```text
tests/test_reporting.py .............                                    [100%]
============================== 13 passed in 1.40s ==============================
```

Control suites around the changed seams (`test_reporting`, `test_utils`,
`test_pricing`, `test_pricing_litellm`, `test_models`), same command on
the base worktree for the pre-existing subset:

```text
============================== 69 passed in 6.21s ==============================
```

Mutation checks — each regression is caught by the test that owns it:

```text
# `or 0.0` swallow put back:
FAILED tests/test_reporting.py::test_generate_report_says_unavailable_when_it_cannot_price
E   AssertionError: assert '$0.0000' == 'unavailable'
========================= 1 failed, 12 passed in 6.84s =========================

# None branch dropped:
FAILED tests/test_reporting.py::test_generate_report_says_unavailable_when_it_cannot_price
E   TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType'
========================= 1 failed, 12 passed in 6.50s =========================
```

Gates:

```text
$ .venv/bin/python -m ruff check .
All checks passed!
$ .venv/bin/python -m ruff format --check .
1658 files already formatted
$ .venv/bin/python -m mypy headroom
headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 543 source files)
$ .venv/bin/python -m mypy headroom/reporting/generator.py
Success: no issues found in 1 source file
```

The repo-wide `mypy headroom` line is **not** caused by this change: the
same command on an untouched `main` worktree at `d13e196` produces the
identical single error in
`headroom/integrations/langchain/chat_model.py`, a file this PR does not
touch. I did not fix it here — it belongs to whichever change owns that
adapter, and folding it in would widen a two-file diff.

Import-cost numbers behind keeping the pricer inside the function:

```text
import headroom.reporting: 0.071s litellm_loaded=False   # before, and after this patch
from headroom.models.registry import ModelRegistry: 1.192s litellm_loaded=True
```

**Not tested locally:** the full `pytest tests`. 211 test modules import
the compiled `headroom._core`, which I did not build (`uv sync --extra
dev --no-install-project`), and the collectible remainder is slow enough
to need CI's sharding — `tests/test_bundled_tools_savings.py` alone
takes 73.46 s on the base commit `d13e196`. I checked that file
separately and it passes; nothing in it touches `headroom.reporting`.

## Real Behavior Proof

- **Environment:** macOS 27 arm64, Python 3.14.4, source tree at `main`
= `d13e196` (`chore: release 0.39.1 (#3807)`, `_version` reports
`0.39.2-dev`), deps from `uv sync --extra dev --no-install-project`, no
provider credentials, no network calls in the path under test.
- **Exact command / steps:** write two real requests into a real SQLite
metrics DB through `create_storage()`, call the documented
`headroom.generate_report()` (it is exported from `headroom/__init__.py`
and in `docs/content/docs/api-reference.mdx`), then read the tile text
out of the generated HTML with a regex rather than looking at the page:

```python
storage = create_storage("sqlite:///tmp/r.db")
for i, (before, after) in enumerate([(200_000, 60_000), (150_000, 45_000)]):
    storage.save(RequestMetrics(
        request_id=f"req-{i}", timestamp=datetime(2026, 9, 27, 10, i),
        model="gpt-4o", stream=False, mode="optimize",
        tokens_input_before=before, tokens_input_after=after, tokens_output=1000))

html = open(generate_report("sqlite:///tmp/r.db", "/tmp/report.html")).read()
```

- **Observed result — before the patch** (this is the issue's
reproduction, run on `d13e196`):

```text
stored summary: {'total_requests': 2, 'total_tokens_before': 350000,
                 'total_tokens_after': 105000, 'total_tokens_saved': 245000}
report written: /tmp/hr-repro2/report.html
  'Tokens Saved'    tile -> 245,000
  'Est. Cost Savings' tile -> $0.0000
```

- **Observed result — after the patch**, same script, same DB contents,
only `headroom/reporting/generator.py` changed:

```text
stored summary: {'total_requests': 2, 'total_tokens_before': 350000,
                 'total_tokens_after': 105000, 'total_tokens_saved': 245000}
report written: /tmp/hr-repro2b/report.html
  'Tokens Saved'    tile -> 245,000
  'Est. Cost Savings' tile -> $0.61

  registry gpt-4o pricing: before=$0.8750 after=$0.2625 savings=$0.6125
```

`$0.61` is `format_cost(0.8750 - 0.2625)`, i.e. the tile now agrees with
the rate card the report already names, and the number moves when the
stored tokens move.
- **Not tested:** Windows; the `headroom-ai[proxy]` wheel's Rust path
(`headroom/reporting/` is pure Python with no Rust mirror — I ran it
from the source tree, not the wheel); opening the report in a browser
(the tile text was read from the generated HTML, not screenshotted);
what happens to this tile when a deployment's traffic is mostly a model
LiteLLM has no price for — that case is now labelled `unavailable` and I
have not seen it in real traffic.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — `generate_report()` is a plain SDK
call, not behind a rollout flag.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, deliberately. Every report's
**Est. Cost Savings** tile changes from the constant `$0.0000` to a
computed price, or to `unavailable` when the rate card has no entry. No
other tile, no stored field, no on-disk format changes.
- Kill switch / disable path: none needed; reverting the single commit
restores the previous (incorrect) behaviour.
- Unsafe override required: none.
- Qualification impact: none known — no qualification asserts on this
string, and the two tests that did assert on it asserted on their own
stub's return value.
- Rollback path: `git revert` the commit; it touches two files and adds
no state.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- **Docs left alone deliberately.**
`docs/content/docs/api-reference.mdx` documents `generate_report()`
without promising a dollar figure, so nothing becomes wrong here. I did
not add a line about the `unavailable` string — one word in a tile that
needs no legend felt like noise, but I'll document it if you'd rather it
be explicit.
- **The hardcoded `"gpt-4o"` rate card is untouched on purpose.** A
model-agnostic report pricing all savings at one reference model is a
real question, and it is a different change from "the tile always said
zero". Widening this PR would make the before/after harder to read.
- **`utils.estimate_cost()`'s `None` contract is left alone.** The
silent-swallow hazard exists for any future caller too, but
`headroom/utils.py` is being edited in the open by #3116, so I stayed
out of that file rather than create a conflict for them.
- Diff is 2 files, +72/−13. No dependency, lockfile, or generated file
is touched.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Aashish Mahato
2026-09-28 23:12:53 -05:00
committed by GitHub
co-authored by JerrettDavis
parent d90b32008d
commit f519fa8b97
2 changed files with 72 additions and 13 deletions
+11 -4
View File
@@ -7,7 +7,7 @@ from pathlib import Path
from typing import TYPE_CHECKING, Any
from ..storage import create_storage
from ..utils import estimate_cost, format_cost
from ..utils import format_cost
if TYPE_CHECKING:
pass
@@ -337,9 +337,16 @@ def generate_report(
tpm_multiplier = 1.0
# Estimate cost savings (using gpt-4o pricing)
cost_before = estimate_cost(stats["total_tokens_before"], 0, "gpt-4o") or 0.0
cost_after = estimate_cost(stats["total_tokens_after"], 0, "gpt-4o") or 0.0
estimated_savings = format_cost(cost_before - cost_after)
# Imported here: the LiteLLM-backed pricer costs ~1.2s of import time.
from ..models.registry import ModelRegistry
cost_before = ModelRegistry.estimate_cost("gpt-4o", stats["total_tokens_before"], 0)
cost_after = ModelRegistry.estimate_cost("gpt-4o", stats["total_tokens_after"], 0)
# "Cannot price this" is not "$0 saved", so an unpriced report says so.
if cost_before is None or cost_after is None:
estimated_savings = "unavailable"
else:
estimated_savings = format_cost(cost_before - cost_after)
stats["tpm_multiplier"] = tpm_multiplier
stats["estimated_savings"] = estimated_savings
+61 -9
View File
@@ -9,7 +9,9 @@ from types import SimpleNamespace
import pytest
import headroom.reporting as reporting
from headroom.models.registry import ModelRegistry
from headroom.reporting import generator
from headroom.utils import format_cost
@dataclass
@@ -255,12 +257,6 @@ def test_generate_report_writes_output_and_closes_storage(
generator, "_generate_recommendations", lambda *args: [{"title": "Keep going"}]
)
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
monkeypatch.setattr(
generator,
"estimate_cost",
lambda tokens, output_tokens, model: {100: 2.0, 0: None}[tokens],
)
monkeypatch.setattr(generator, "format_cost", lambda cost: f"${cost:.2f}")
output_path = tmp_path / "report.html"
result = generator.generate_report(
@@ -274,10 +270,68 @@ def test_generate_report_writes_output_and_closes_storage(
assert output_path.read_text() == "<html>report</html>"
assert render_calls[0]["period"] == expected_period
assert render_calls[0]["stats"]["tpm_multiplier"] == 100.0
assert render_calls[0]["stats"]["estimated_savings"] == "$2.00"
assert storage.closed is True
def _capture_stats(monkeypatch, tmp_path, stats: dict) -> dict:
"""Run generate_report() over *stats* and hand back the rendered stats dict.
Only the storage and template seams are patched: pricing stays the product's
own, so a report that cannot price itself shows up here as a wrong number.
"""
render_calls: list[dict] = []
class FakeTemplate:
def render(self, **kwargs) -> str:
render_calls.append(kwargs)
return "<html>report</html>"
monkeypatch.setattr(generator, "create_storage", lambda store_url: FakeStorage(stats, []))
monkeypatch.setattr(generator, "_build_waste_histogram", lambda *args: [])
monkeypatch.setattr(generator, "_get_top_waste_requests", lambda *args, **kwargs: [])
monkeypatch.setattr(generator, "_generate_recommendations", lambda *args: [])
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
generator.generate_report("sqlite:///demo.db", output_path=str(tmp_path / "report.html"))
return render_calls[0]["stats"]
SAVINGS_STATS = {
"total_requests": 2,
"total_tokens_saved": 245_000,
"avg_tokens_saved": 122_500,
"total_tokens_before": 350_000,
"total_tokens_after": 105_000,
"avg_cache_alignment": 80,
"audit_count": 0,
"optimize_count": 2,
}
def test_generate_report_prices_the_savings_tile(monkeypatch, tmp_path) -> None:
"""245k tokens saved must cost something, not read as a $0.0000 claim."""
stats = _capture_stats(monkeypatch, tmp_path, dict(SAVINGS_STATS))
before = ModelRegistry.estimate_cost("gpt-4o", 350_000, 0)
after = ModelRegistry.estimate_cost("gpt-4o", 105_000, 0)
assert before is not None and after is not None, "gpt-4o left the bundled rate card"
assert stats["estimated_savings"] == format_cost(before - after)
assert stats["estimated_savings"] != "$0.00"
def test_generate_report_says_unavailable_when_it_cannot_price(monkeypatch, tmp_path) -> None:
"""An unknown price is a missing number, not zero saved."""
monkeypatch.setattr(
ModelRegistry,
"estimate_cost",
classmethod(lambda cls, model, *args, **kwargs: None),
)
stats = _capture_stats(monkeypatch, tmp_path, dict(SAVINGS_STATS))
assert stats["estimated_savings"] == "unavailable"
def test_generate_report_closes_storage_when_render_fails(monkeypatch, tmp_path) -> None:
storage = FakeStorage(
{
@@ -302,8 +356,6 @@ def test_generate_report_closes_storage_when_render_fails(monkeypatch, tmp_path)
monkeypatch.setattr(generator, "_get_top_waste_requests", lambda *args, **kwargs: [])
monkeypatch.setattr(generator, "_generate_recommendations", lambda *args: [])
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
monkeypatch.setattr(generator, "estimate_cost", lambda *args: 0.0)
monkeypatch.setattr(generator, "format_cost", lambda cost: "$0.00")
with pytest.raises(RuntimeError, match="boom"):
generator.generate_report("sqlite:///demo.db", output_path=str(tmp_path / "report.html"))