fix(pricing): suppress litellm's Provider List banner that corrupts perf JSON (#4197)

## Description

`headroom perf --format json` (and `--format csv`) can print litellm's
red "Provider List: https://docs.litellm.ai/docs/providers" banner to
**stdout** ahead of the document. The output then doesn't parse:

```text
$ headroom perf --hours 24 --format json | python -c "import json,sys; json.load(sys.stdin)"
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
```

Here is the path. `build_perf_summary` prices each model through
`headroom.pricing.litellm_pricing.resolve_litellm_model`, which probes
candidates with `litellm.cost_per_token`. For a model litellm can't map
to a provider, `get_llm_provider` calls `print()` on the banner unless
`litellm.suppress_debug_info` is set. `headroom.providers.litellm`,
`headroom.providers.anthropic` and `headroom.learn.analyzer` already set
that flag, but the perf CLI imports none of them. `litellm_pricing`
imports litellm without setting it. #2860 cached model resolution, so
the banner now prints once per model per process; a fresh CLI run is a
new process, so it still prints every time.

This PR sets `litellm.suppress_debug_info = True` right after
`litellm_pricing`'s guarded import, so every pricing caller is covered,
perf included.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/pricing/litellm_pricing.py`: set
`litellm.suppress_debug_info = True` after the import, the same flag the
provider modules already set.
- `tests/test_cli_perf_format.py`:
`test_perf_json_survives_litellm_provider_banner`. In a fresh
interpreter (litellm's default `suppress_debug_info = False`), it builds
the perf JSON summary for a report with an unknown model and asserts
that "Provider List" is absent from stdout and that the output parses
with `json.loads`. It uses a subprocess because the flag is set at
import time, and reloading the module in-process would leave stale class
references for later tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_cli_perf_format.py -k banner -v
tests/test_cli_perf_format.py::test_perf_json_survives_litellm_provider_banner PASSED [100%]
================= 1 passed, 22 deselected, 1 warning in 2.89s ==================
# fails on main: "Provider List" found in output, JSONDecodeError

$ pytest tests/test_cli_perf_format.py tests/test_pricing_litellm.py \
    tests/test_pricing_litellm_model_resolution.py tests/test_litellm_probe_fallback_logs.py \
    tests/test_savings_tracker_litellm_resolution_cache.py
======================== 46 passed, 1 warning in 3.07s =========================

$ ruff check headroom/pricing/litellm_pricing.py tests/test_cli_perf_format.py
All checks passed!
$ ruff format --check headroom/pricing/litellm_pricing.py tests/test_cli_perf_format.py
2 files already formatted
$ mypy headroom/pricing/litellm_pricing.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.13, litellm 1.88.1. A real proxy log
directory with 24 hours of traffic (1,668 requests across several model
names).
- Exact command / steps: `headroom perf --hours 24 --format json >
out.json`, then `python -c "import json; json.load(open('out.json'))"`.
Run once with the installed 0.40.0 and once with this branch on
`PYTHONPATH`, both from a cwd outside the checkout.
- Observed result: 0.40.0 output starts with `\x1b[1;31mProvider List:
https://docs.litellm.ai/docs/providers\x1b[0m` and `json.load` raises
`JSONDecodeError: Expecting value: line 2 column 1`. This branch's
output starts with `{` / `"window_hours": 24.0` and parses
(`total_requests` 1668).
- Not tested: litellm versions other than 1.88.1. `suppress_debug_info`
is a long-standing public module attribute, already relied on by
`headroom/providers/litellm.py`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only that litellm's debug banner and
"Give Feedback / Get Help" hints are no longer printed by processes that
use `headroom.pricing.litellm_pricing`. The proxy already suppressed
them through `headroom.providers.litellm`. Pricing results are
unchanged.
- Kill switch / disable path: none needed; there's no functional
behaviour to disable.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert the one-line change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- No docs change: the documented `perf --format json` contract (a JSON
document on stdout) now actually holds.
- Earlier reports #613 and #2851 covered the same banner on the proxy
path. This is the CLI/pricing path, which those fixes didn't reach.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4197"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
M Barry
2026-10-15 16:32:36 -05:00
committed by GitHub
co-authored by JD Davis
parent ba824539d6
commit e01ad350dc
2 changed files with 37 additions and 0 deletions
+4
View File
@@ -44,6 +44,10 @@ try:
del _os.environ[_leaked_key]
del _env_snapshot, _os
# Probing an unknown model makes litellm print() its "Provider List"
# banner to stdout, which corrupts `headroom perf --format json`.
litellm.suppress_debug_info = True
LITELLM_AVAILABLE = True
except ImportError:
litellm = None # type: ignore[assignment]
+33
View File
@@ -230,6 +230,39 @@ def test_perf_json_format(runner, monkeypatch):
assert data["overhead"]["optimization_ms"]["p95_ms"] == 11.8
def test_perf_json_survives_litellm_provider_banner(tmp_path):
"""Pricing an unknown model makes litellm print() its "Provider List"
banner to stdout unless suppressed. The perf CLI imports no module that
suppresses it, so the pricing module has to.
Runs in a fresh interpreter: the flag is set at import time, and
reloading the module in-process would leave stale class references
behind for later tests.
"""
import subprocess
import sys
pytest.importorskip("litellm")
script = (
"import json, litellm\n"
"assert not litellm.suppress_debug_info\n"
"from headroom.perf.analyzer import build_perf_summary, PerfReport, PerfRecord\n"
"rec = PerfRecord(timestamp='2026-06-05 10:00:00,000', request_id='hr_1',\n"
" model='no-such-model-banner-probe', num_messages=1, tokens_before=10,\n"
" tokens_after=5, tokens_saved=5, cache_read=0, cache_write=0,\n"
" cache_hit_pct=0, optimization_ms=1.0, transforms=[])\n"
"print(json.dumps(build_perf_summary(PerfReport(perf_records=[rec]))))\n"
)
env = {k: v for k, v in os.environ.items() if k != "LITELLM_SUPPRESS_DEBUG_INFO"}
env["HOME"] = str(tmp_path)
result = subprocess.run(
[sys.executable, "-c", script], capture_output=True, text=True, env=env, timeout=120
)
assert result.returncode == 0, result.stderr
assert "Provider List" not in result.stdout
assert json.loads(result.stdout)["total_requests"] == 1
def test_perf_json_raw_is_array(runner, monkeypatch):
_patch_report(monkeypatch, _sample_report())
result = runner.invoke(main, ["perf", "--format", "json", "--raw"])