mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): log unknown model ids as one bounded token; skip unparseable PERF lines (#4374)
## Description Two halves of #4372: 1. `OpenAIProvider._warn_unknown_model` (`headroom/providers/openai.py`) interpolated the client-supplied model id into a WARNING verbatim, so a newline in it wrote an extra line into `proxy.log` — the file `perf/analyzer.py` reads PERF records from. It now uses the existing `safe_id()` helper, exactly as the neighbouring LiteLLM log line in the same class already does (`safe_id(model)` a few lines above). 2. `parse_log_files` (`headroom/perf/analyzer.py`) built every `PerfRecord` with unguarded `int()`/`float()` casts, so one PERF-shaped line with a non-numeric field aborted `headroom perf` and `agent-savings --check-perf`. Such a line is now skipped and counted in a new `records_malformed` field, reported as its own "Malformed PERF lines" line (never as "Records outside window", which tells the user to raise `--hours`); well-formed records parse exactly as before. Fixes #4372 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/providers/openai.py`: `'{model}'` → `{safe_id(model)}` in the unknown-model warning. - `headroom/perf/analyzer.py`: the `PerfRecord(...)` construction moves into `try/except ValueError`; on failure the line is skipped and counted in `PerfReport.records_malformed` (new, default 0), and `_track_window` runs only for records that parsed. `format_report` prints `Malformed PERF lines: N (skipped — a numeric field did not parse)` when N > 0, and `build_perf_summary` carries `records_malformed`. `records_filtered_out` keeps meaning time-window exclusions only. - `tests/test_providers_openai_unknown_model_log.py` (new): the warning for a model id containing a newline is a single line. - `tests/test_perf_per_port_logs.py`: `test_parse_log_files_skips_a_perf_line_with_non_numeric_fields` — a bad PERF line next to a good one; only the good record is reported, `records_malformed == 1`, `records_filtered_out == 0`, the text report shows the malformed line and not "Records outside window", and the JSON summary carries the count. ## Testing - [x] Unit tests pass (`pytest`) — the two touched test files + `tests/test_cli_perf_format.py`: 47 passed - [x] Linting passes (`ruff check .`) — on the four changed files, plus `ruff format --check` - [ ] Type checking passes (`mypy headroom`) — not run; no new types - [x] New tests added for new functionality - [x] Manual testing performed (real `headroom perf` CLI run, below) ### Test Output ```text # new tests, source at main4c7acba5(fail-before) FAILED tests/test_perf_per_port_logs.py::test_parse_log_files_skips_a_perf_line_with_non_numeric_fields FAILED tests/test_providers_openai_unknown_model_log.py::test_unknown_model_warning_is_one_line 2 failed, 21 deselected in 5.79s # with the fix $ pytest tests/test_perf_per_port_logs.py tests/test_providers_openai_unknown_model_log.py tests/test_cli_perf_format.py -q 47 passed in 8.35s $ ruff check <4 files> && ruff format --check <4 files> All checks passed! 4 files already formatted ``` ## Real Behavior Proof - Environment: WSL2 Ubuntu, Python 3.12.15, pytest 9.1.1, ruff 0.17.1; a clone of this fork at `4c7acba5` with the core runtime dependencies only - Exact command / steps: add the tests alone and run them (both fail: the warning spans two lines, and `parse_log_files` raises `ValueError` on `msgs=lots`), apply the source change and run the three perf/provider test files and ruff, as pasted above - Observed result: the warning is one bounded line for any model id; a malformed PERF line is skipped and counted while the valid record in the same file is still reported - Not tested: other places that log a raw model id (the issue lists the trace through `handlers/openai.py`; only this sink is changed here); the dashboard `/stats` path, which already catches the exception ### CLI run (real `headroom perf`, not a unit test) A log directory holding one valid PERF line and one PERF line whose `msgs=` field is `lots`, then `headroom perf --hours 0` against it: ```text ---- main4c7acba5Traceback (most recent call last): ... ValueError: invalid literal for int() with base 10: 'lots' ---- this branch Malformed PERF lines: 1 (skipped — a numeric field did not parse) Requests: 1 ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a (not rollout-managed) - Stable/default behavior changed: the unknown-model warning shows the id via `repr` (quoted, escapes visible); malformed PERF lines are skipped and counted instead of aborting the report; the JSON summary gains one key, `records_malformed` - Kill switch / disable path: none added - Unsafe override required: no - Qualification impact: none known; no `CHANGELOG.md` edit (release-please) - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (none needed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Produced with AI assistance (Claude Code) operated by me (HetCreep); the test runs above were executed on my machine on a clone of this fork. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
b50b3eb5ed
commit
c014bd6311
@@ -224,6 +224,7 @@ class PerfReport:
|
||||
oldest_kept_ts: str | None = None
|
||||
newest_kept_ts: str | None = None
|
||||
records_filtered_out: int = 0
|
||||
records_malformed: int = 0
|
||||
# True when no time cutoff was applied (--hours 0, or a value so large it
|
||||
# overflows datetime arithmetic). The header says "all data" instead of a
|
||||
# misleading "last 0h".
|
||||
@@ -386,9 +387,8 @@ def parse_log_files(last_n_hours: float = 168.0) -> PerfReport:
|
||||
if not _within_window(ts):
|
||||
report.records_filtered_out += 1
|
||||
continue
|
||||
_track_window(ts)
|
||||
report.perf_records.append(
|
||||
PerfRecord(
|
||||
try:
|
||||
record = PerfRecord(
|
||||
timestamp=ts,
|
||||
request_id=m.group("rid"),
|
||||
model=kv.get("model", ""),
|
||||
@@ -411,7 +411,13 @@ def parse_log_files(last_n_hours: float = 168.0) -> PerfReport:
|
||||
from_response_cache=kv.get("cached", "0") == "1",
|
||||
stages=stages_by_rid.get(m.group("rid"), {}),
|
||||
)
|
||||
)
|
||||
except ValueError:
|
||||
# A non-numeric field is not a PERF line Headroom wrote
|
||||
# (e.g. text a client put into a logged model id).
|
||||
report.records_malformed += 1
|
||||
continue
|
||||
_track_window(ts)
|
||||
report.perf_records.append(record)
|
||||
continue
|
||||
|
||||
# content_router summary lines
|
||||
@@ -544,6 +550,11 @@ def format_report(report: PerfReport) -> str:
|
||||
f"Records outside window: {report.records_filtered_out:,} "
|
||||
"(filtered out — increase --hours to include them)"
|
||||
)
|
||||
if report.records_malformed > 0:
|
||||
lines.append(
|
||||
f"Malformed PERF lines: {report.records_malformed:,} "
|
||||
"(skipped — a numeric field did not parse)"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
records = report.perf_records
|
||||
@@ -1400,6 +1411,7 @@ def build_perf_summary(report: PerfReport) -> dict:
|
||||
"newest": report.newest_kept_ts,
|
||||
},
|
||||
"records_filtered_out": report.records_filtered_out,
|
||||
"records_malformed": report.records_malformed,
|
||||
"total_requests": len(records),
|
||||
"total_tokens_before": total_before,
|
||||
"total_tokens_after": total_after,
|
||||
|
||||
@@ -646,7 +646,7 @@ class OpenAIProvider(Provider):
|
||||
if model not in _UNKNOWN_MODEL_WARNINGS:
|
||||
_UNKNOWN_MODEL_WARNINGS.add(model)
|
||||
logger.warning(
|
||||
f"Unknown OpenAI model '{model}': {reason} ({limit:,} tokens). "
|
||||
f"Unknown OpenAI model {safe_id(model)}: {reason} ({limit:,} tokens). "
|
||||
f"To configure explicitly, set HEADROOM_MODEL_LIMITS env var or "
|
||||
f"add to ~/.headroom/models.json"
|
||||
)
|
||||
|
||||
@@ -401,3 +401,22 @@ def test_router_content_dump_needs_the_content_opt_in(
|
||||
router_logger.setLevel(previous)
|
||||
|
||||
assert ("secret-tool-output" in stream.getvalue()) is logged
|
||||
|
||||
|
||||
def test_parse_log_files_skips_a_perf_line_with_non_numeric_fields(workspace: Path) -> None:
|
||||
"""A PERF-shaped line with a non-numeric count is skipped, not fatal."""
|
||||
log_dir = workspace / "logs"
|
||||
ts = "2026-08-22 10:00:00,000"
|
||||
bad = _perf_line(ts, "hr_bad", "model-BAD").replace("msgs=3", "msgs=lots")
|
||||
good = _perf_line(ts, "hr_good", "model-GOOD")
|
||||
(log_dir / "proxy.log").write_text(bad + "\n" + good + "\n")
|
||||
|
||||
report = analyzer.parse_log_files(last_n_hours=0.0)
|
||||
|
||||
assert {r.model for r in report.perf_records} == {"model-GOOD"}
|
||||
assert report.records_malformed == 1
|
||||
assert report.records_filtered_out == 0 # nothing was outside the window
|
||||
text = analyzer.format_report(report)
|
||||
assert "Malformed PERF lines:" in text
|
||||
assert "Records outside window" not in text
|
||||
assert analyzer.build_perf_summary(report)["records_malformed"] == 1
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
"""The unknown-model warning must log the client-supplied model id as one bounded token."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from headroom.providers import openai as openai_provider
|
||||
|
||||
|
||||
def test_unknown_model_warning_is_one_line(caplog) -> None:
|
||||
provider = openai_provider.OpenAIProvider()
|
||||
openai_provider._UNKNOWN_MODEL_WARNINGS.discard("odd\nmodel")
|
||||
with caplog.at_level(logging.WARNING, logger=openai_provider.logger.name):
|
||||
provider._warn_unknown_model("odd\nmodel", 128000, "pattern match")
|
||||
messages = [r.getMessage() for r in caplog.records if "Unknown OpenAI model" in r.getMessage()]
|
||||
assert messages
|
||||
assert all("\n" not in m for m in messages)
|
||||
Reference in New Issue
Block a user