mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(pricing): suppress litellm's Provider List banner that corrupts perf JSON (#4197)
## Description `headroom perf --format json` (and `--format csv`) can print litellm's red "Provider List: https://docs.litellm.ai/docs/providers" banner to **stdout** ahead of the document. The output then doesn't parse: ```text $ headroom perf --hours 24 --format json | python -c "import json,sys; json.load(sys.stdin)" json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1) ``` Here is the path. `build_perf_summary` prices each model through `headroom.pricing.litellm_pricing.resolve_litellm_model`, which probes candidates with `litellm.cost_per_token`. For a model litellm can't map to a provider, `get_llm_provider` calls `print()` on the banner unless `litellm.suppress_debug_info` is set. `headroom.providers.litellm`, `headroom.providers.anthropic` and `headroom.learn.analyzer` already set that flag, but the perf CLI imports none of them. `litellm_pricing` imports litellm without setting it. #2860 cached model resolution, so the banner now prints once per model per process; a fresh CLI run is a new process, so it still prints every time. This PR sets `litellm.suppress_debug_info = True` right after `litellm_pricing`'s guarded import, so every pricing caller is covered, perf included. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/pricing/litellm_pricing.py`: set `litellm.suppress_debug_info = True` after the import, the same flag the provider modules already set. - `tests/test_cli_perf_format.py`: `test_perf_json_survives_litellm_provider_banner`. In a fresh interpreter (litellm's default `suppress_debug_info = False`), it builds the perf JSON summary for a report with an unknown model and asserts that "Provider List" is absent from stdout and that the output parses with `json.loads`. It uses a subprocess because the flag is set at import time, and reloading the module in-process would leave stale class references for later tests. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_cli_perf_format.py -k banner -v tests/test_cli_perf_format.py::test_perf_json_survives_litellm_provider_banner PASSED [100%] ================= 1 passed, 22 deselected, 1 warning in 2.89s ================== # fails on main: "Provider List" found in output, JSONDecodeError $ pytest tests/test_cli_perf_format.py tests/test_pricing_litellm.py \ tests/test_pricing_litellm_model_resolution.py tests/test_litellm_probe_fallback_logs.py \ tests/test_savings_tracker_litellm_resolution_cache.py ======================== 46 passed, 1 warning in 3.07s ========================= $ ruff check headroom/pricing/litellm_pricing.py tests/test_cli_perf_format.py All checks passed! $ ruff format --check headroom/pricing/litellm_pricing.py tests/test_cli_perf_format.py 2 files already formatted $ mypy headroom/pricing/litellm_pricing.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.13, litellm 1.88.1. A real proxy log directory with 24 hours of traffic (1,668 requests across several model names). - Exact command / steps: `headroom perf --hours 24 --format json > out.json`, then `python -c "import json; json.load(open('out.json'))"`. Run once with the installed 0.40.0 and once with this branch on `PYTHONPATH`, both from a cwd outside the checkout. - Observed result: 0.40.0 output starts with `\x1b[1;31mProvider List: https://docs.litellm.ai/docs/providers\x1b[0m` and `json.load` raises `JSONDecodeError: Expecting value: line 2 column 1`. This branch's output starts with `{` / `"window_hours": 24.0` and parses (`total_requests` 1668). - Not tested: litellm versions other than 1.88.1. `suppress_debug_info` is a long-standing public module attribute, already relied on by `headroom/providers/litellm.py`. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: only that litellm's debug banner and "Give Feedback / Get Help" hints are no longer printed by processes that use `headroom.pricing.litellm_pricing`. The proxy already suppressed them through `headroom.providers.litellm`. Pricing results are unchanged. - Kill switch / disable path: none needed; there's no functional behaviour to disable. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert the one-line change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - No docs change: the documented `perf --format json` contract (a JSON document on stdout) now actually holds. - Earlier reports #613 and #2851 covered the same banner on the proxy path. This is the CLI/pricing path, which those fixes didn't reach. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4197" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -44,6 +44,10 @@ try:
|
||||
del _os.environ[_leaked_key]
|
||||
del _env_snapshot, _os
|
||||
|
||||
# Probing an unknown model makes litellm print() its "Provider List"
|
||||
# banner to stdout, which corrupts `headroom perf --format json`.
|
||||
litellm.suppress_debug_info = True
|
||||
|
||||
LITELLM_AVAILABLE = True
|
||||
except ImportError:
|
||||
litellm = None # type: ignore[assignment]
|
||||
|
||||
@@ -230,6 +230,39 @@ def test_perf_json_format(runner, monkeypatch):
|
||||
assert data["overhead"]["optimization_ms"]["p95_ms"] == 11.8
|
||||
|
||||
|
||||
def test_perf_json_survives_litellm_provider_banner(tmp_path):
|
||||
"""Pricing an unknown model makes litellm print() its "Provider List"
|
||||
banner to stdout unless suppressed. The perf CLI imports no module that
|
||||
suppresses it, so the pricing module has to.
|
||||
|
||||
Runs in a fresh interpreter: the flag is set at import time, and
|
||||
reloading the module in-process would leave stale class references
|
||||
behind for later tests.
|
||||
"""
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
pytest.importorskip("litellm")
|
||||
script = (
|
||||
"import json, litellm\n"
|
||||
"assert not litellm.suppress_debug_info\n"
|
||||
"from headroom.perf.analyzer import build_perf_summary, PerfReport, PerfRecord\n"
|
||||
"rec = PerfRecord(timestamp='2026-06-05 10:00:00,000', request_id='hr_1',\n"
|
||||
" model='no-such-model-banner-probe', num_messages=1, tokens_before=10,\n"
|
||||
" tokens_after=5, tokens_saved=5, cache_read=0, cache_write=0,\n"
|
||||
" cache_hit_pct=0, optimization_ms=1.0, transforms=[])\n"
|
||||
"print(json.dumps(build_perf_summary(PerfReport(perf_records=[rec]))))\n"
|
||||
)
|
||||
env = {k: v for k, v in os.environ.items() if k != "LITELLM_SUPPRESS_DEBUG_INFO"}
|
||||
env["HOME"] = str(tmp_path)
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", script], capture_output=True, text=True, env=env, timeout=120
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "Provider List" not in result.stdout
|
||||
assert json.loads(result.stdout)["total_requests"] == 1
|
||||
|
||||
|
||||
def test_perf_json_raw_is_array(runner, monkeypatch):
|
||||
_patch_report(monkeypatch, _sample_report())
|
||||
result = runner.invoke(main, ["perf", "--format", "json", "--raw"])
|
||||
|
||||
Reference in New Issue
Block a user