fix(proxy): log unknown model ids as one bounded token; skip unparseable PERF lines (#4374)

## Description

Two halves of #4372:

1. `OpenAIProvider._warn_unknown_model` (`headroom/providers/openai.py`)
interpolated the client-supplied model id into a WARNING verbatim, so a
newline in it wrote an extra line into `proxy.log` — the file
`perf/analyzer.py` reads PERF records from. It now uses the existing
`safe_id()` helper, exactly as the neighbouring LiteLLM log line in the
same class already does (`safe_id(model)` a few lines above).
2. `parse_log_files` (`headroom/perf/analyzer.py`) built every
`PerfRecord` with unguarded `int()`/`float()` casts, so one PERF-shaped
line with a non-numeric field aborted `headroom perf` and `agent-savings
--check-perf`. Such a line is now skipped and counted in a new
`records_malformed` field, reported as its own "Malformed PERF lines"
line (never as "Records outside window", which tells the user to raise
`--hours`); well-formed records parse exactly as before.

Fixes #4372

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/openai.py`: `'{model}'` → `{safe_id(model)}` in
the unknown-model warning.
- `headroom/perf/analyzer.py`: the `PerfRecord(...)` construction moves
into `try/except ValueError`; on failure the line is skipped and counted
in `PerfReport.records_malformed` (new, default 0), and `_track_window`
runs only for records that parsed. `format_report` prints `Malformed
PERF lines: N (skipped — a numeric field did not parse)` when N > 0, and
`build_perf_summary` carries `records_malformed`. `records_filtered_out`
keeps meaning time-window exclusions only.
- `tests/test_providers_openai_unknown_model_log.py` (new): the warning
for a model id containing a newline is a single line.
- `tests/test_perf_per_port_logs.py`:
`test_parse_log_files_skips_a_perf_line_with_non_numeric_fields` — a bad
PERF line next to a good one; only the good record is reported,
`records_malformed == 1`, `records_filtered_out == 0`, the text report
shows the malformed line and not "Records outside window", and the JSON
summary carries the count.

## Testing

- [x] Unit tests pass (`pytest`) — the two touched test files +
`tests/test_cli_perf_format.py`: 47 passed
- [x] Linting passes (`ruff check .`) — on the four changed files, plus
`ruff format --check`
- [ ] Type checking passes (`mypy headroom`) — not run; no new types
- [x] New tests added for new functionality
- [x] Manual testing performed (real `headroom perf` CLI run, below)

### Test Output

```text
# new tests, source at main 4c7acba5 (fail-before)
FAILED tests/test_perf_per_port_logs.py::test_parse_log_files_skips_a_perf_line_with_non_numeric_fields
FAILED tests/test_providers_openai_unknown_model_log.py::test_unknown_model_warning_is_one_line
2 failed, 21 deselected in 5.79s

# with the fix
$ pytest tests/test_perf_per_port_logs.py tests/test_providers_openai_unknown_model_log.py tests/test_cli_perf_format.py -q
47 passed in 8.35s

$ ruff check <4 files> && ruff format --check <4 files>
All checks passed!
4 files already formatted
```

## Real Behavior Proof

- Environment: WSL2 Ubuntu, Python 3.12.15, pytest 9.1.1, ruff 0.17.1; a
clone of this fork at `4c7acba5` with the core runtime dependencies only
- Exact command / steps: add the tests alone and run them (both fail:
the warning spans two lines, and `parse_log_files` raises `ValueError`
on `msgs=lots`), apply the source change and run the three perf/provider
test files and ruff, as pasted above
- Observed result: the warning is one bounded line for any model id; a
malformed PERF line is skipped and counted while the valid record in the
same file is still reported
- Not tested: other places that log a raw model id (the issue lists the
trace through `handlers/openai.py`; only this sink is changed here); the
dashboard `/stats` path, which already catches the exception

### CLI run (real `headroom perf`, not a unit test)

A log directory holding one valid PERF line and one PERF line whose
`msgs=` field is `lots`, then `headroom perf --hours 0` against it:

```text
---- main 4c7acba5
Traceback (most recent call last):
  ...
ValueError: invalid literal for int() with base 10: 'lots'

---- this branch
Malformed PERF lines:    1 (skipped — a numeric field did not parse)
Requests:     1
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a (not rollout-managed)
- Stable/default behavior changed: the unknown-model warning shows the
id via `repr` (quoted, escapes visible); malformed PERF lines are
skipped and counted instead of aborting the report; the JSON summary
gains one key, `records_malformed`
- Kill switch / disable path: none added
- Unsafe override required: no
- Qualification impact: none known; no `CHANGELOG.md` edit
(release-please)
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
test runs above were executed on my machine on a clone of this fork.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
HetCreep
2026-10-28 21:23:28 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent b50b3eb5ed
commit c014bd6311
4 changed files with 53 additions and 5 deletions
+16 -4
View File
@@ -224,6 +224,7 @@ class PerfReport:
oldest_kept_ts: str | None = None
newest_kept_ts: str | None = None
records_filtered_out: int = 0
records_malformed: int = 0
# True when no time cutoff was applied (--hours 0, or a value so large it
# overflows datetime arithmetic). The header says "all data" instead of a
# misleading "last 0h".
@@ -386,9 +387,8 @@ def parse_log_files(last_n_hours: float = 168.0) -> PerfReport:
if not _within_window(ts):
report.records_filtered_out += 1
continue
_track_window(ts)
report.perf_records.append(
PerfRecord(
try:
record = PerfRecord(
timestamp=ts,
request_id=m.group("rid"),
model=kv.get("model", ""),
@@ -411,7 +411,13 @@ def parse_log_files(last_n_hours: float = 168.0) -> PerfReport:
from_response_cache=kv.get("cached", "0") == "1",
stages=stages_by_rid.get(m.group("rid"), {}),
)
)
except ValueError:
# A non-numeric field is not a PERF line Headroom wrote
# (e.g. text a client put into a logged model id).
report.records_malformed += 1
continue
_track_window(ts)
report.perf_records.append(record)
continue
# content_router summary lines
@@ -544,6 +550,11 @@ def format_report(report: PerfReport) -> str:
f"Records outside window: {report.records_filtered_out:,} "
"(filtered out — increase --hours to include them)"
)
if report.records_malformed > 0:
lines.append(
f"Malformed PERF lines: {report.records_malformed:,} "
"(skipped — a numeric field did not parse)"
)
lines.append("")
records = report.perf_records
@@ -1400,6 +1411,7 @@ def build_perf_summary(report: PerfReport) -> dict:
"newest": report.newest_kept_ts,
},
"records_filtered_out": report.records_filtered_out,
"records_malformed": report.records_malformed,
"total_requests": len(records),
"total_tokens_before": total_before,
"total_tokens_after": total_after,
+1 -1
View File
@@ -646,7 +646,7 @@ class OpenAIProvider(Provider):
if model not in _UNKNOWN_MODEL_WARNINGS:
_UNKNOWN_MODEL_WARNINGS.add(model)
logger.warning(
f"Unknown OpenAI model '{model}': {reason} ({limit:,} tokens). "
f"Unknown OpenAI model {safe_id(model)}: {reason} ({limit:,} tokens). "
f"To configure explicitly, set HEADROOM_MODEL_LIMITS env var or "
f"add to ~/.headroom/models.json"
)
+19
View File
@@ -401,3 +401,22 @@ def test_router_content_dump_needs_the_content_opt_in(
router_logger.setLevel(previous)
assert ("secret-tool-output" in stream.getvalue()) is logged
def test_parse_log_files_skips_a_perf_line_with_non_numeric_fields(workspace: Path) -> None:
"""A PERF-shaped line with a non-numeric count is skipped, not fatal."""
log_dir = workspace / "logs"
ts = "2026-08-22 10:00:00,000"
bad = _perf_line(ts, "hr_bad", "model-BAD").replace("msgs=3", "msgs=lots")
good = _perf_line(ts, "hr_good", "model-GOOD")
(log_dir / "proxy.log").write_text(bad + "\n" + good + "\n")
report = analyzer.parse_log_files(last_n_hours=0.0)
assert {r.model for r in report.perf_records} == {"model-GOOD"}
assert report.records_malformed == 1
assert report.records_filtered_out == 0 # nothing was outside the window
text = analyzer.format_report(report)
assert "Malformed PERF lines:" in text
assert "Records outside window" not in text
assert analyzer.build_perf_summary(report)["records_malformed"] == 1
@@ -0,0 +1,17 @@
"""The unknown-model warning must log the client-supplied model id as one bounded token."""
from __future__ import annotations
import logging
from headroom.providers import openai as openai_provider
def test_unknown_model_warning_is_one_line(caplog) -> None:
provider = openai_provider.OpenAIProvider()
openai_provider._UNKNOWN_MODEL_WARNINGS.discard("odd\nmodel")
with caplog.at_level(logging.WARNING, logger=openai_provider.logger.name):
provider._warn_unknown_model("odd\nmodel", 128000, "pattern match")
messages = [r.getMessage() for r in caplog.records if "Unknown OpenAI model" in r.getMessage()]
assert messages
assert all("\n" not in m for m in messages)