mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
### The test we ran
A Daytona sandbox running Headroom Enterprise 0.4.3 (core 0.39.1), with
Codex CLI 0.162.1 and Claude Code connected to real GitHub, Context7 and
Fetch MCP servers, working through the workshop tasks.
### What we found
- The tool-search plugin's turn hook rewrote the Codex `/v1/responses`
input and saved about **8K tokens per request**. `run_request_hooks`
booked that saving through `ctx.record_savings(hook.savings_source,
tokens=...)`.
- After many such requests, `GET /stats` still showed
**`extension_savings: {}`**.
There were two causes:
1. **`extension_savings` has no writer in core.** The only thing that
fills it is `PrometheusMetrics.record_extension_savings`, and core never
calls it. Attribution rows from `ctx.record_savings` /
`record_scope_savings` go to `savings_by_source`, which `/stats` shows
as `savings.by_source`.
2. **The Codex WebSocket path drops attribution entirely.** Both WS
compression calls (first frame and later frames) called
`_compress_openai_responses_payload_in_executor` without `savings_tags`.
So `TurnContext.tags` was a throwaway `{}`, and the hook's row (along
with the built-in tool-search deferral row) never reached
`savings.by_source` either. Codex uses the WS transport by default. The
HTTP `/v1/responses` path already passed `savings_tags`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- **`/stats.extension_savings`** now reports measured savings per
source: the **realized** attribution rows, plus
`record_extension_savings` totals. If a source also calls
`record_extension_savings` (`headroom-hdr` reports through both APIs on
every request), it shows the larger of its two totals rather than their
sum, so hdr is not double-counted. Nothing ties an attribution row to a
`record_extension_savings` call, so a plugin that used the two APIs on
different turns would be under-reported, never over-reported.
Unrealized/estimated rows (tool-search deferral, `realized=False`) stay
out and remain visible in `savings.by_source`.
- **Codex WS:** each frame compresses into its own `savings_tags` dict.
Its rows are merged into the session's pending list on the event loop,
and only when that frame was actually rewritten and sent. Each per-turn
`RequestOutcome` carries the pending rows and then clears them. A turn
whose `response.completed` has no input usage leaves its rows pending,
the same gate as its headline `saved_delta`
(`_deferrable_savings_delta`). Rows no turn claimed go out with the
session-end residual outcome, and only if the residual has input usage.
A fresh dict per frame matters because a timed-out compression thread
keeps running (`_run_compression_in_executor` can't cancel it), and it
must not book rows for a frame that went out uncompressed.
**Realized vs estimated:** `savings.by_source` already keeps one row per
`(source, realized)`, so realized and unrealized savings are shown
separately. `extension_savings` now carries only the realized ones.
**Review fix (@JerrettDavis, bounded WS pending list):** the
session-wide pending list grew one row per rewritten frame, while
`savings_attribution.from_tags` read only the first `MAX_SOURCES` (32)
rows. After 33 frames that each booked a 100-token `tool_search` row,
the handoff cleared all 33 rows but booked only 3,200 of the 3,300
tokens. `savings_attribution.accumulate(rows, items)` replaces the
`extend` at both WS frame paths.
**Bounded rows, nothing dropped (follow-up to @JerrettDavis's two
reviews):** the 32-row cap that silently dropped rows is gone from
`record_savings`, `from_tags` and `accumulate`. Storage is bounded by
coalescing plus one overflow row:
- every writer goes through the one `accumulate` helper: the per-request
ledger (`record_savings`, so every handler) and the WS pending list.
Rows with the same `(source, realized, estimated)` key are summed into
one row (`tokens`, `usd`, `events`). `details` are kept only while every
summed row has the same `details`.
- **cardinality bound:** a ledger keeps at most `MAX_NAMED_ROWS` (256)
named rows. A new source past that bound is summed into an `other` row
with the same `(realized, estimated)`, and its `details` are dropped.
Tokens, dollars and events are conserved: nothing is dropped, only the
source name. `tool_search` never folds, because
`tool_schema_savings_policy` finds the deferral's row by name to lower
it when hooks un-defer tools.
- **finite sums:** a row's `usd` saturates at ±`MAX_ROW_USD` (1e15) and
its `tokens` stay within 0..`MAX_ROW_TOKENS` (10^15). A row that was
clipped, or summed from one that was, carries `saturated: true`. Every
total stays finite, so the Prometheus exposition and
`json.dumps(allow_nan=False)` never see `inf`/`nan`.
- each row carries `events`: how many recordings it stands for (1 on a
fresh row, summed on merge). `PrometheusMetrics.record_request`
(`headroom_savings_attribution_events_total`), the OTel
`headroom.savings.attribution.events` counter and the perf analyzer's
"Savings by Source" add that count instead of 1 per row, so 33 coalesced
frames are 33 events, not 1 (Devin's finding). A row without `events`
(older PERF logs, rows built by hand) counts as 1.
- a row that a reader edited in place (`reconcile_deferred_tokens`, the
tool-search plugin) is summed defensively, so a junk value counts as 0
instead of raising inside the handler. The core deferral and the native
deferral hook both book `tool_search` as estimated, so they now share
one row, and the reconcile debits that row; totals are unchanged.
- `MAX_SOURCES` is no longer used. No importer was found outside this
repo's own test, but `savings_attribution` is imported by plugins, so
the name is kept behind a module `__getattr__` that raises a
`DeprecationWarning`.
- **Scope of the bound:** it is per ledger (one request, or one WS
session's pending list). The process-wide
`PrometheusMetrics.savings_by_source` table has no source cap yet; that
is a separate PR.
### Before / after
| | origin/main | this PR |
|---|---|---|
| A turn hook books 8,000 tokens on 2 requests →
`/stats.extension_savings` | `{}` | `{"tool_search": 16000}` |
| An hdr-style plugin calls both APIs (300 tokens × 2) | `{"hdr": 600}`
| `{"hdr": 600}` (not 1200) |
| A Codex WS turn with a folding hook → attribution rows on the turn
outcome | `[]` | `[("ws_fold", realized=True)]` |
| 33 WS frames, each booking `tool_search` 100 tokens, then one billed
turn → booked `tool_search` tokens (helper-level, the reviewer's
scenario) | 3,200 of 3,300, 33 rows stored | 3,300, 1 row stored |
| Real WS handler, 33 frames each with a folding hook (`ws_fold` row)
plus a 100-token hook row → non-`ws_fold` tokens booked | 1,600 of 3,300
(66 rows stored) | 3,300 |
| Same, repeated source → rows stored / hook events in
`headroom_savings_attribution_events_total` | aecae31b: 2 rows / 1 event
| 2 rows / 33 events |
| Same, 33 distinct sources → rows stored / hook events / `other` row |
aecae31b: 29 rows / 28 events / yes (6 sources folded) | 34 rows (33 +
`ws_fold`) / 33 events / no |
| Same, 300 distinct sources → rows stored / tokens and USD booked |
rows past 32 dropped | 257 rows (256 named + `other`), every token and
dollar booked |
| One ledger, 2,000 distinct sources plus repeats | rows past 32 dropped
| 256 named rows + 2 `other` rows (realized, estimated); tokens, USD and
events conserved |
| Same source booked 1e308 USD twice, then 1.0 | `inf` in `/stats` and
Prometheus | `usd` = 1e15, `saturated: true`, exposition finite |
| One HTTP request, 40 distinct sources + 1 repeat, 100 tokens each
(`record_savings`, every handler) → booked tokens / events | 3,200 of
4,100 / 32 (8 rows dropped) | 4,100 / 41, 40 rows stored |
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
-
`tests/test_compression_observability.py::test_turn_hook_savings_surface_in_stats_extension_savings`:
fails on origin/main, passes here.
-
`tests/test_openai_codex_ws_lifecycle.py::test_ws_turn_hook_savings_reach_the_turn_outcome[billed|usage_less]`:
drives the real WS handler with a registered hook. `billed` fails on
origin/main (no rows). `usage_less` fails if the input-usage gate is
removed.
- Related suites pass: the WS lifecycle, per-frame memory, savings
deferral and compression scheduler tests, extension attribution,
responses context compaction, tool-search repair after hooks, and the
handler outcome tag invariant (136 passed).
- ruff check, ruff format --check and mypy pass.
-
`tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked`
(review fix, renamed from `..._beyond_max_sources_...`), parametrized
over a repeated source × 33 frames, 33 distinct sources, and 300
distinct sources (folds into `other`, 257 rows, USD conserved): drives
the real WS handler through the rewritten frames. The upstream holds its
events until all 33 frames arrive, so every row is pending when the one
billed outcome claims it. The rows are booked through the real
`from_tags` → `PrometheusMetrics.record_request`. The test asserts that
every token is booked (3,300), that the hook rows count 33 events, that
there is no `other` row, and that the stored row count equals the
distinct keys (2 for a repeated source, 34 for 33 distinct sources plus
`ws_fold`).
-
`tests/test_extension_attribution.py::test_a_request_with_40_sources_books_every_token`:
one request's ledger with 40 distinct sources and one repeat books 4,100
tokens and 41 events through `PrometheusMetrics.record_request`.
-
`tests/test_extension_attribution.py::test_rows_coalesce_per_source_realized_and_estimated`:
same source with different `realized`/`estimated` stays in separate
rows; same key sums tokens and events.
-
`tests/test_observability_metrics.py::test_coalesced_savings_rows_count_every_event`:
the OTel events counter adds a row's `events` (33), and a row without
the field counts as 1.
-
`tests/test_cli_perf_format.py::test_coalesced_savings_rows_count_every_event_in_text_and_summary`:
both perf folds (`headroom perf` text report and the JSON `by_source`
summary) add a row's `events`.
-
`tests/test_extension_attribution.py::test_merging_into_a_junk_row_does_not_raise_inside_the_handler`:
a ledger row edited in place with junk (`tokens=None`) is summed as 0
instead of raising out of `record_savings`.
-
`tests/test_extension_attribution.py::test_distinct_sources_past_the_bound_fold_into_other_and_conserve_totals`:
2,000 sources plus estimated repeats give 258 rows, with tokens, USD and
events conserved.
-
`tests/test_extension_attribution.py::test_same_source_usd_saturates_instead_of_overflowing`,
`test_an_infinite_total_already_in_the_ledger_stays_saturated[±1.0]`,
`test_tokens_stay_within_zero_and_the_bound`: sums stay finite and are
flagged `saturated`.
-
`tests/test_extension_attribution.py::test_tool_search_keeps_its_own_row_past_the_bound`:
the deferral row is never folded, so the reconcile still finds it.
-
`tests/test_extension_attribution.py::test_max_sources_is_deprecated_not_removed`:
the old name still imports, with a `DeprecationWarning`.
-
`tests/test_turn_hooks.py::test_request_runner_attributes_savings_with_handler_counters`:
expected row now includes `events: 1`.
### Test Output
```text
# review fix, savings_attribution.py + openai.py reverted to the previous commit, new test:
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_max_sources_are_all_booked[repeated_source] (assert 1600 == (100 * 33))
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_max_sources_are_all_booked[distinct_sources] (assert 1600 == (100 * 33))
# helper-level, the reviewer's scenario (33 frames x tool_search 100):
append: 33 rows stored, 3200 booked
accumulate: 1 row stored, 3300 booked
40 distinct sources x 100: 29 rows stored, 4000 booked ("other" = 1200)
# no-cap follow-up, the 5 source files reverted to aecae31b, new/updated tests:
FAILED tests/test_extension_attribution.py::test_a_request_with_40_sources_books_every_token
FAILED tests/test_extension_attribution.py::test_rows_coalesce_per_source_realized_and_estimated
FAILED tests/test_observability_metrics.py::test_coalesced_savings_rows_count_every_event
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked[repeated_source]
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked[distinct_sources]
# with the no-cap follow-up: every test file that touches the savings ledger (21 files)
# plus gateway, perf format/per-port logs and /stats suites: 831 passed, 5 skipped (90d5e132)
# with the fix (aecae31b):
WS lifecycle, per-frame memory, savings deferral, compression scheduler, WS timings, pyo3 WS compression,
outcome tag invariant, extension attribution, compression observability, tool-search aggregation,
responses context compaction, proxy openai: 189 passed
ruff 0.17.0 check + format --check: clean on changed files
mypy headroom --ignore-missing-imports: no new errors (1 pre-existing local error in integrations/langchain/chat_model.py:380, untouched)
```
## Real Behavior Proof
- Environment: macOS 26.7, Python 3.12.13, this branch. The WS handler
is driven through the repo's fake-upstream harness
(`tests/test_openai_codex_ws_lifecycle.py`), which runs the real
`handle_openai_responses_ws`, `run_request_hooks` and outcome funnel
with only the upstream socket scripted; `/stats` runs on the real app
via `TestClient`; OTel uses the real `HeadroomOtelMetrics` with an
in-memory reader.
- Exact command / steps: `pytest tests/test_openai_codex_ws_lifecycle.py
tests/test_extension_attribution.py tests/test_observability_metrics.py
tests/test_compression_observability.py tests/test_turn_hooks.py` (plus
the other ledger suites listed under Testing), each run once on this
branch and once with the source files reverted.
- Observed result: with a registered folding hook, the hook really
rewrites the forwarded `input` (the upstream received only `"new"`) and
the turn outcome carries `[("ws_fold", realized=True, tokens>0)]`.
`/stats.extension_savings` reports `{"tool_search": 16000, "hdr": 600}`
for the two-request scenario above. 33 rewritten WS frames book 3,300
hook tokens and 33 events with no `other` row; 300 distinct sources fold
into 256 named rows plus `other` with every token and dollar booked; one
request with 40 distinct sources plus a repeat books 4,100 tokens and 41
events in 40 rows.
- Not tested: a real Codex CLI session against OpenAI on this branch, a
real Codex session with more than 32 rewritten frames before a billed
turn, and the real `headroom-tool-search` plugin were not re-run after
the patch. The sandbox numbers above (about 8K tokens/request,
`extension_savings: {}`) are from 0.39.1 before the fix.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: `/stats.extension_savings` now
includes realized attribution rows, and Codex WS turns now carry
attribution rows. Request bodies and headline totals are unchanged.
- Kill switch / disable path: none needed; the change only affects
reporting
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- plus the code-reviewer agent on each review fix. Its earlier SHOULD on
the collapsed `events` count is now fixed (rows carry `events`).
- [x] This PR is ready for human review
## Additional Notes
- `record_extension_savings` stays public and unchanged.
- The Prometheus exposition format is unchanged;
`headroom_savings_attribution_events_total` now counts recordings, which
is what it counted before rows were coalesced.
- Ledger rows now carry an `events` field. Readers that ignore unknown
keys are unaffected.
- A reader that finds "rows a hook added" by slicing the ledger at its
old length (`ledger[before:]`) can miss a hook's tokens when the hook
books a key that already had a row. Core has no such reader;
`headroom-kiro`'s handler does and should switch to a before/after sum.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
601 lines
23 KiB
Python
601 lines
23 KiB
Python
"""Tests for `headroom perf --format {text,json,csv}` (issue #595)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import csv
|
|
import io
|
|
import json
|
|
import os
|
|
from datetime import datetime, timedelta
|
|
|
|
import pytest
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main
|
|
from headroom.perf import analyzer
|
|
from headroom.perf.analyzer import (
|
|
PerfRecord,
|
|
PerfReport,
|
|
TransformRecord,
|
|
build_overhead_summary,
|
|
build_perf_summary,
|
|
build_savings_audit,
|
|
perf_records_as_dicts,
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def runner() -> CliRunner:
|
|
return CliRunner()
|
|
|
|
|
|
def _sample_report() -> PerfReport:
|
|
"""A small report with two models, cache numbers, and a transform."""
|
|
return PerfReport(
|
|
perf_records=[
|
|
PerfRecord(
|
|
timestamp="2026-06-05 10:00:00,000",
|
|
request_id="hr_1",
|
|
model="claude-sonnet-4.5",
|
|
num_messages=10,
|
|
tokens_before=1000,
|
|
tokens_after=400,
|
|
tokens_saved=600,
|
|
cache_read=800,
|
|
cache_write=200,
|
|
cache_hit_pct=80,
|
|
optimization_ms=12.0,
|
|
transforms=["content_router"],
|
|
),
|
|
PerfRecord(
|
|
timestamp="2026-06-05 11:00:00,000",
|
|
request_id="hr_2",
|
|
model="claude-opus-4-8",
|
|
num_messages=4,
|
|
tokens_before=1000,
|
|
tokens_after=600,
|
|
tokens_saved=400,
|
|
cache_read=200,
|
|
cache_write=0,
|
|
cache_hit_pct=100,
|
|
optimization_ms=8.0,
|
|
transforms=["content_router"],
|
|
),
|
|
],
|
|
transform_records=[
|
|
TransformRecord(
|
|
timestamp="2026-06-05 10:00:00,000",
|
|
name="content_router",
|
|
tokens_before=2000,
|
|
tokens_after=1000,
|
|
tokens_saved=1000,
|
|
),
|
|
],
|
|
log_files_read=1,
|
|
total_lines_parsed=42,
|
|
requested_hours=24.0,
|
|
oldest_kept_ts="2026-06-05 10:00:00,000",
|
|
newest_kept_ts="2026-06-05 11:00:00,000",
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Pure builders
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_build_perf_summary_totals_and_pct():
|
|
summary = build_perf_summary(_sample_report())
|
|
|
|
assert summary["total_requests"] == 2
|
|
assert summary["total_tokens_before"] == 2000
|
|
assert summary["total_tokens_after"] == 1000
|
|
assert summary["tokens_saved"] == 1000
|
|
# 1000 / 2000 == 50.0%
|
|
assert summary["savings_pct"] == 50.0
|
|
# cache: read 1000, write 200 -> 1000 / 1200 == 83.3%
|
|
assert summary["cache_read_tokens"] == 1000
|
|
assert summary["cache_write_tokens"] == 200
|
|
assert summary["cache_hit_pct"] == 83.3
|
|
assert summary["window_hours"] == 24.0
|
|
assert summary["savings_audit"]["prompt_reduction_tokens"] == 1000
|
|
assert summary["savings_audit"]["accounting_delta_tokens"] == 0
|
|
|
|
|
|
def test_build_perf_summary_by_model_and_transform():
|
|
summary = build_perf_summary(_sample_report())
|
|
|
|
models = {m["model"]: m for m in summary["by_model"]}
|
|
assert set(models) == {"claude-sonnet-4.5", "claude-opus-4-8"}
|
|
assert models["claude-sonnet-4.5"]["tokens_saved"] == 600
|
|
assert models["claude-sonnet-4.5"]["savings_pct"] == 60.0
|
|
assert models["claude-opus-4-8"]["savings_pct"] == 40.0
|
|
|
|
assert summary["by_transform"][0]["transform"] == "content_router"
|
|
assert summary["by_transform"][0]["tokens_saved"] == 1000
|
|
assert summary["by_transform"][0]["uses"] == 1
|
|
|
|
|
|
def test_build_perf_summary_empty_report_no_zero_division():
|
|
summary = build_perf_summary(PerfReport(requested_hours=168.0))
|
|
assert summary["total_requests"] == 0
|
|
assert summary["savings_pct"] == 0.0
|
|
assert summary["cache_hit_pct"] == 0.0
|
|
assert summary["by_model"] == []
|
|
assert summary["overhead"]["optimization_ms"]["count"] == 0
|
|
|
|
|
|
def test_build_overhead_summary_attributes_slow_stages():
|
|
report = PerfReport(
|
|
perf_records=[
|
|
PerfRecord(
|
|
timestamp="2026-06-05 10:00:00,000",
|
|
request_id="fast",
|
|
model="gpt-5",
|
|
tokens_before=1000,
|
|
tokens_after=500,
|
|
tokens_saved=500,
|
|
optimization_ms=100.0,
|
|
total_ms=300.0,
|
|
stages={"cache_align": 10.0, "content_router": 90.0},
|
|
),
|
|
PerfRecord(
|
|
timestamp="2026-06-05 10:01:00,000",
|
|
request_id="slow",
|
|
model="gpt-5",
|
|
tokens_before=1000,
|
|
tokens_after=500,
|
|
tokens_saved=500,
|
|
optimization_ms=700.0,
|
|
total_ms=900.0,
|
|
stages={"kompress": 650.0, "content_router": 40.0},
|
|
),
|
|
]
|
|
)
|
|
|
|
overhead = build_overhead_summary(report, slow_threshold_ms=500.0)
|
|
|
|
assert overhead["optimization_ms"]["count"] == 2
|
|
assert overhead["optimization_ms"]["average_ms"] == 400.0
|
|
assert overhead["optimization_ms"]["p50_ms"] == 400.0
|
|
assert overhead["optimization_ms"]["p95_ms"] == 670.0
|
|
assert overhead["optimization_ms"]["p99_ms"] == 694.0
|
|
assert overhead["optimization_ms"]["slow_request_count"] == 1
|
|
assert overhead["stage_breakdown"][0]["stage"] == "kompress"
|
|
assert overhead["stage_breakdown"][0]["total_ms"] == 650.0
|
|
assert overhead["top_slow_requests"][0]["request_id"] == "slow"
|
|
assert overhead["top_slow_requests"][0]["slowest_stage"] == "kompress"
|
|
|
|
|
|
def test_build_savings_audit_surfaces_accounting_deltas():
|
|
report = PerfReport(
|
|
perf_records=[
|
|
PerfRecord(
|
|
timestamp="2026-06-05 10:00:00,000",
|
|
request_id="hr_bad",
|
|
model="gpt-5",
|
|
tokens_before=100,
|
|
tokens_after=90,
|
|
tokens_saved=250,
|
|
),
|
|
PerfRecord(
|
|
timestamp="2026-06-05 10:01:00,000",
|
|
request_id="hr_growth",
|
|
model="gpt-5",
|
|
tokens_before=100,
|
|
tokens_after=120,
|
|
tokens_saved=1,
|
|
),
|
|
]
|
|
)
|
|
|
|
audit = build_savings_audit(report)
|
|
|
|
assert audit["logged_tokens_saved"] == 251
|
|
assert audit["prompt_reduction_tokens"] == 10
|
|
assert audit["accounting_delta_tokens"] == 241
|
|
assert audit["record_counts"]["with_accounting_delta"] == 2
|
|
assert audit["record_counts"]["logged_saved_gt_tokens_before"] == 1
|
|
assert audit["record_counts"]["prompt_grew_but_logged_savings_positive"] == 1
|
|
assert audit["suspicious_records"][0]["request_id"] == "hr_bad"
|
|
assert "logged_saved_gt_tokens_before" in audit["suspicious_records"][0]["reasons"]
|
|
|
|
|
|
def test_perf_records_as_dicts_roundtrips_fields():
|
|
dicts = perf_records_as_dicts(_sample_report())
|
|
assert len(dicts) == 2
|
|
assert dicts[0]["request_id"] == "hr_1"
|
|
assert dicts[0]["tokens_saved"] == 600
|
|
# transforms stays a list for JSON consumers
|
|
assert dicts[0]["transforms"] == ["content_router"]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# CLI integration
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _patch_report(monkeypatch, report: PerfReport) -> None:
|
|
monkeypatch.setattr(analyzer, "parse_log_files", lambda last_n_hours=168.0: report)
|
|
|
|
|
|
def test_perf_json_format(runner, monkeypatch):
|
|
_patch_report(monkeypatch, _sample_report())
|
|
result = runner.invoke(main, ["perf", "--format", "json"])
|
|
assert result.exit_code == 0, result.output
|
|
data = json.loads(result.output)
|
|
assert data["savings_pct"] == 50.0
|
|
assert "by_model" in data
|
|
assert data["total_requests"] == 2
|
|
assert data["overhead"]["optimization_ms"]["p95_ms"] == 11.8
|
|
|
|
|
|
def test_perf_json_survives_litellm_provider_banner(tmp_path):
|
|
"""Pricing an unknown model makes litellm print() its "Provider List"
|
|
banner to stdout unless suppressed. The perf CLI imports no module that
|
|
suppresses it, so the pricing module has to.
|
|
|
|
Runs in a fresh interpreter: the flag is set at import time, and
|
|
reloading the module in-process would leave stale class references
|
|
behind for later tests.
|
|
"""
|
|
import subprocess
|
|
import sys
|
|
|
|
pytest.importorskip("litellm")
|
|
script = (
|
|
"import json, litellm\n"
|
|
"assert not litellm.suppress_debug_info\n"
|
|
"from headroom.perf.analyzer import build_perf_summary, PerfReport, PerfRecord\n"
|
|
"rec = PerfRecord(timestamp='2026-06-05 10:00:00,000', request_id='hr_1',\n"
|
|
" model='no-such-model-banner-probe', num_messages=1, tokens_before=10,\n"
|
|
" tokens_after=5, tokens_saved=5, cache_read=0, cache_write=0,\n"
|
|
" cache_hit_pct=0, optimization_ms=1.0, transforms=[])\n"
|
|
"print(json.dumps(build_perf_summary(PerfReport(perf_records=[rec]))))\n"
|
|
)
|
|
env = {k: v for k, v in os.environ.items() if k != "LITELLM_SUPPRESS_DEBUG_INFO"}
|
|
env["HOME"] = str(tmp_path)
|
|
result = subprocess.run(
|
|
[sys.executable, "-c", script], capture_output=True, text=True, env=env, timeout=120
|
|
)
|
|
assert result.returncode == 0, result.stderr
|
|
assert "Provider List" not in result.stdout
|
|
assert json.loads(result.stdout)["total_requests"] == 1
|
|
|
|
|
|
def test_perf_json_raw_is_array(runner, monkeypatch):
|
|
_patch_report(monkeypatch, _sample_report())
|
|
result = runner.invoke(main, ["perf", "--format", "json", "--raw"])
|
|
assert result.exit_code == 0, result.output
|
|
data = json.loads(result.output)
|
|
assert isinstance(data, list)
|
|
assert len(data) == 2
|
|
assert data[0]["request_id"] == "hr_1"
|
|
|
|
|
|
def test_perf_json_raw_preserves_client_field(runner, monkeypatch):
|
|
report = _sample_report()
|
|
report.perf_records[0].client = "codex"
|
|
_patch_report(monkeypatch, report)
|
|
|
|
result = runner.invoke(main, ["perf", "--format", "json", "--raw"])
|
|
|
|
assert result.exit_code == 0, result.output
|
|
data = json.loads(result.output)
|
|
assert data[0]["client"] == "codex"
|
|
|
|
|
|
def test_parse_perf_line_preserves_client_field(monkeypatch, tmp_path):
|
|
log_dir = tmp_path / "logs"
|
|
log_dir.mkdir()
|
|
(log_dir / "proxy.log").write_text(
|
|
"2026-06-10 10:00:00,000 - headroom.proxy - INFO - "
|
|
"[hr_codex] PERF model=gpt-5 msgs=3 tok_before=1000 "
|
|
"tok_after=90 tok_saved=910 cache_read=0 cache_write=0 "
|
|
"cache_hit_pct=0 opt_ms=12 transforms=content_router client=codex\n"
|
|
)
|
|
monkeypatch.setattr(analyzer, "LOG_DIR", log_dir)
|
|
|
|
report = analyzer.parse_log_files(last_n_hours=0)
|
|
|
|
assert len(report.perf_records) == 1
|
|
assert report.perf_records[0].client == "codex"
|
|
|
|
|
|
def _perf_line(ts: datetime, client: str) -> str:
|
|
return (
|
|
f"{ts.strftime('%Y-%m-%d %H:%M:%S')},000 - headroom.proxy - INFO - "
|
|
f"[hr_x] PERF model=gpt-5 msgs=3 tok_before=1000 "
|
|
f"tok_after=90 tok_saved=910 cache_read=0 cache_write=0 "
|
|
f"cache_hit_pct=0 opt_ms=12 transforms=content_router client={client}\n"
|
|
)
|
|
|
|
|
|
def _write_log(path, text: str, mtime: datetime) -> None:
|
|
path.write_text(text)
|
|
stamp = mtime.timestamp()
|
|
os.utime(path, (stamp, stamp))
|
|
|
|
|
|
def test_windowed_parse_skips_rotated_logs_older_than_the_cutoff(monkeypatch, tmp_path):
|
|
"""A windowed query must cost O(window), not O(total log history).
|
|
|
|
`/stats` recomputes throughput over the last hour on a 10s cache TTL, so
|
|
reading every rotated log each time made the endpoint slower the longer
|
|
the proxy had been running.
|
|
"""
|
|
log_dir = tmp_path / "logs"
|
|
log_dir.mkdir()
|
|
now = datetime.now()
|
|
_write_log(
|
|
log_dir / "proxy.log.1",
|
|
_perf_line(now - timedelta(days=3), "stale"),
|
|
now - timedelta(days=3),
|
|
)
|
|
_write_log(log_dir / "proxy.log", _perf_line(now - timedelta(minutes=5), "live"), now)
|
|
monkeypatch.setattr(analyzer, "LOG_DIR", log_dir)
|
|
|
|
report = analyzer.parse_log_files(last_n_hours=1.0)
|
|
|
|
assert [r.client for r in report.perf_records] == ["live"]
|
|
# The stale file was never opened, so its lines were never even counted.
|
|
# Asserted before the counters below because a read-then-filter
|
|
# implementation also yields the right records -- only the work differs.
|
|
assert report.total_lines_parsed == 1
|
|
assert report.log_files_read == 1
|
|
assert report.log_files_skipped == 1
|
|
|
|
|
|
def test_unwindowed_parse_still_reads_every_rotated_log(monkeypatch, tmp_path):
|
|
"""`--hours 0` means "all data" and must not prune anything."""
|
|
log_dir = tmp_path / "logs"
|
|
log_dir.mkdir()
|
|
now = datetime.now()
|
|
_write_log(
|
|
log_dir / "proxy.log.1",
|
|
_perf_line(now - timedelta(days=3), "stale"),
|
|
now - timedelta(days=3),
|
|
)
|
|
_write_log(log_dir / "proxy.log", _perf_line(now - timedelta(minutes=5), "live"), now)
|
|
monkeypatch.setattr(analyzer, "LOG_DIR", log_dir)
|
|
|
|
report = analyzer.parse_log_files(last_n_hours=0)
|
|
|
|
assert {r.client for r in report.perf_records} == {"stale", "live"}
|
|
assert report.log_files_skipped == 0
|
|
assert report.log_files_read == 2
|
|
|
|
|
|
def test_perf_csv_by_model(runner, monkeypatch):
|
|
_patch_report(monkeypatch, _sample_report())
|
|
result = runner.invoke(main, ["perf", "--format", "csv"])
|
|
assert result.exit_code == 0, result.output
|
|
rows = list(csv.DictReader(io.StringIO(result.output)))
|
|
assert {r["model"] for r in rows} == {"claude-sonnet-4.5", "claude-opus-4-8"}
|
|
sonnet = next(r for r in rows if r["model"] == "claude-sonnet-4.5")
|
|
assert sonnet["tokens_saved"] == "600"
|
|
assert sonnet["prompt_reduction_tokens"] == "600"
|
|
assert sonnet["accounting_delta_tokens"] == "0"
|
|
|
|
|
|
def test_perf_csv_raw_per_record(runner, monkeypatch):
|
|
report = _sample_report()
|
|
report.perf_records[0].client = "codex"
|
|
_patch_report(monkeypatch, report)
|
|
result = runner.invoke(main, ["perf", "--format", "csv", "--raw"])
|
|
assert result.exit_code == 0, result.output
|
|
rows = list(csv.DictReader(io.StringIO(result.output)))
|
|
assert len(rows) == 2
|
|
assert rows[0]["request_id"] == "hr_1"
|
|
assert rows[0]["client"] == "codex"
|
|
# transforms flattened to a string cell
|
|
assert rows[0]["transforms"] == "content_router"
|
|
|
|
|
|
def test_perf_text_default_unchanged(runner, monkeypatch):
|
|
_patch_report(monkeypatch, _sample_report())
|
|
result = runner.invoke(main, ["perf"])
|
|
assert result.exit_code == 0, result.output
|
|
assert "Headroom Performance Report" in result.output
|
|
assert "p50/p95/p99" in result.output
|
|
|
|
|
|
def test_perf_rejects_unknown_format(runner, monkeypatch):
|
|
_patch_report(monkeypatch, _sample_report())
|
|
result = runner.invoke(main, ["perf", "--format", "xml"])
|
|
assert result.exit_code != 0
|
|
|
|
|
|
def test_parse_perf_line_preserves_blank_client_field(
|
|
tmp_path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
logs_dir = tmp_path / "logs"
|
|
logs_dir.mkdir()
|
|
monkeypatch.setattr(analyzer, "LOG_DIR", logs_dir)
|
|
(logs_dir / "proxy.log").write_text(
|
|
"2026-06-10 10:00:00,000 - headroom.proxy - INFO - [req-blank] PERF "
|
|
"model=gpt-5 msgs=1 tok_before=100 tok_after=50 tok_saved=50 "
|
|
"cache_read=0 cache_write=0 cache_hit_pct=0 opt_ms=1 transforms=test client=\n",
|
|
encoding="utf-8",
|
|
)
|
|
|
|
report = analyzer.parse_log_files(last_n_hours=0)
|
|
|
|
assert len(report.perf_records) == 1
|
|
assert report.perf_records[0].client == ""
|
|
|
|
|
|
def test_throughput_parsing_and_calculations(monkeypatch, tmp_path):
|
|
logs_dir = tmp_path / "logs"
|
|
logs_dir.mkdir()
|
|
monkeypatch.setattr(analyzer, "LOG_DIR", logs_dir)
|
|
|
|
log_content = (
|
|
'2026-06-10 10:00:00,000 - headroom.proxy - INFO - [req1] STAGE_TIMINGS {"event": "stage_timings", "stages": {"compression_first_stage": 100.0, "upstream_connect": 50.0}}\n'
|
|
"2026-06-10 10:00:01,000 - headroom.proxy - INFO - [req1] PERF model=gpt-5 msgs=1 tok_before=1000 tok_after=400 tok_saved=600 opt_ms=10 total_ms=500 tok_out=500 ttfb_ms=100 transforms=test client=codex\n"
|
|
'2026-06-10 10:00:02,000 - headroom.proxy - INFO - [req2] STAGE_TIMINGS {"event": "stage_timings", "stages": {"compression": 200.0, "upstream_connect": 50.0}}\n'
|
|
"2026-06-10 10:00:03,000 - headroom.proxy - INFO - [req2] PERF model=gpt-5 msgs=1 tok_before=2000 tok_after=1000 tok_saved=1000 opt_ms=20 total_ms=1000 tok_out=1000 ttfb_ms=200 transforms=test client=codex\n"
|
|
"2026-06-10 10:00:05,000 - headroom.proxy - INFO - [req3] PERF model=gpt-5 msgs=1 tok_before=1500 tok_after=500 tok_saved=1000 opt_ms=15 total_ms=600 tok_out=600 ttfb_ms=150 transforms=test client=codex\n"
|
|
'2026-06-10 10:00:06,000 - headroom.proxy - INFO - [req4] STAGE_TIMINGS {"event": "stage_timings", "stages": {"compression_first_stage": 150.0, "upstream_connect": 50.0}}\n'
|
|
"2026-06-10 10:00:07,000 - headroom.proxy - INFO - [req4] PERF model=gpt-5 msgs=1 tok_before=1200 tok_after=300 tok_saved=900 opt_ms=12 total_ms=400 tok_out=400 ttfb_ms=80 transforms=test client=codex\n"
|
|
'2026-06-10 10:00:08,000 - headroom.proxy - INFO - [req5] STAGE_TIMINGS {"event": "stage_timings", "stages": {"compression_first_stage": 50.0, "upstream_connect": 50.0}}\n'
|
|
"2026-06-10 10:00:09,000 - headroom.proxy - INFO - [req5] PERF model=gpt-5 msgs=1 tok_before=800 tok_after=200 tok_saved=600 opt_ms=5 total_ms=300 tok_out=300 ttfb_ms=50 transforms=test client=codex\n"
|
|
)
|
|
(logs_dir / "proxy.log").write_text(log_content, encoding="utf-8")
|
|
|
|
report = analyzer.parse_log_files(last_n_hours=0)
|
|
|
|
assert len(report.perf_records) == 5
|
|
|
|
assert report.perf_records[0].request_id == "req1"
|
|
assert report.perf_records[0].total_ms == 500.0
|
|
assert report.perf_records[0].tokens_out == 500
|
|
assert report.perf_records[0].ttfb_ms == 100.0
|
|
assert report.perf_records[0].stages == {
|
|
"compression_first_stage": 100.0,
|
|
"upstream_connect": 50.0,
|
|
}
|
|
|
|
assert report.perf_records[2].request_id == "req3"
|
|
assert report.perf_records[2].stages == {}
|
|
|
|
summary = build_perf_summary(report)
|
|
assert "throughput" in summary
|
|
tp = summary["throughput"]
|
|
|
|
rolling = tp["rolling"]
|
|
assert rolling["input_wall_clock"] > 0
|
|
assert rolling["input_active_p50"] == 2500.0
|
|
assert rolling["compression_p50"] == 10000.0
|
|
|
|
|
|
def test_throughput_empty_and_percentiles():
|
|
from headroom.perf.analyzer import (
|
|
PerfReport,
|
|
_calculate_throughput_stats,
|
|
_percentile,
|
|
calculate_throughput,
|
|
)
|
|
|
|
# Empty percentiles
|
|
assert _percentile([], 0.5) == 0.0
|
|
|
|
# Percentiles boundary checks
|
|
assert _percentile([10.0], 0.5) == 10.0
|
|
assert _percentile([10.0, 20.0], 0.5) == 15.0
|
|
assert _percentile([10.0, 20.0], 0.0) == 10.0
|
|
assert _percentile([10.0, 20.0], 1.0) == 20.0
|
|
assert _percentile([10.0, 20.0], 1.5) == 20.0
|
|
|
|
# Empty calculate_throughput
|
|
empty_report = PerfReport()
|
|
tp = calculate_throughput(empty_report)
|
|
assert tp["rolling"]["input_wall_clock"] == 0.0
|
|
assert tp["current"]["input_wall_clock"] == 0.0
|
|
|
|
# _calculate_throughput_stats with empty records
|
|
stats = _calculate_throughput_stats([], 10.0)
|
|
assert stats["input_wall_clock"] == 0.0
|
|
|
|
|
|
# ---- savings attributed to named sources -----------------------------------
|
|
# The PERF line carried `savings=` and the parser decoded it from the start, but
|
|
# nothing rendered it, so a paid extension's contribution was invisible in the
|
|
# report operators actually read.
|
|
|
|
|
|
def _savings_perf_line(encoded: str, req: int = 1) -> str:
|
|
return (
|
|
f"2026-08-31 16:00:0{req},000 - headroom.proxy - INFO - [hr_1_00000{req}] PERF "
|
|
"model=claude-haiku-4-5 msgs=12 tok_before=59343 tok_after=30613 "
|
|
"tok_saved=28730 tok_inflated=0 tool_saved=0 total_saved=28730 cache_read=0 "
|
|
"cache_write=0 cache_hit_pct=0 opt_ms=12 total_ms=900 tok_out=100 "
|
|
f"ttfb_ms=800 savings={encoded} transforms=turn_hook"
|
|
)
|
|
|
|
|
|
def _report_for(lines: list[str], tmp_path, monkeypatch) -> str:
|
|
"""Render a report over `lines`, using this file's established LOG_DIR seam.
|
|
|
|
Deliberately NOT `HEADROOM_WORKSPACE_DIR`: that env var flips which branch
|
|
resolves the log directory, which changes behaviour for the rotated-log
|
|
tests above.
|
|
"""
|
|
from headroom.perf import analyzer
|
|
|
|
logs = tmp_path / "logs"
|
|
logs.mkdir(parents=True, exist_ok=True)
|
|
(logs / "proxy.log").write_text("\n".join(lines) + "\n")
|
|
monkeypatch.setattr(analyzer, "LOG_DIR", logs)
|
|
return analyzer.format_report(analyzer.parse_log_files(last_n_hours=0))
|
|
|
|
|
|
def test_dollar_only_source_is_reported_with_zero_tokens(tmp_path, monkeypatch):
|
|
"""A router saves DOLLARS and exactly zero tokens; both must be legible.
|
|
|
|
routemegood sends the same tokens to a cheaper model, so every token-savings
|
|
channel records nothing for it. Rendering the $ beside a 0-token row is the
|
|
only honest option — folding them together would invent a saving.
|
|
"""
|
|
from headroom.proxy.savings_attribution import encode
|
|
|
|
out = _report_for(
|
|
[
|
|
_savings_perf_line(
|
|
encode([{"source": "routemegood", "tokens": 0, "usd": 0.1257, "realized": True}])
|
|
)
|
|
],
|
|
tmp_path,
|
|
monkeypatch,
|
|
)
|
|
assert "Savings by Source" in out
|
|
assert "routemegood" in out
|
|
assert "0 tokens" in out
|
|
assert "$0.13" in out
|
|
|
|
|
|
def test_token_source_and_dollar_source_coexist(tmp_path, monkeypatch):
|
|
from headroom.proxy.savings_attribution import encode
|
|
|
|
out = _report_for(
|
|
[
|
|
_savings_perf_line(
|
|
encode(
|
|
[
|
|
{"source": "routemegood", "tokens": 0, "usd": 0.0431, "realized": True},
|
|
{"source": "lossless_guard", "tokens": 2233, "usd": 0.0, "realized": True},
|
|
]
|
|
)
|
|
)
|
|
],
|
|
tmp_path,
|
|
monkeypatch,
|
|
)
|
|
assert "routemegood" in out and "lossless_guard" in out
|
|
assert "2,233 tokens" in out
|
|
|
|
|
|
def test_no_section_when_nothing_attributed(tmp_path, monkeypatch):
|
|
out = _report_for([_savings_perf_line("none")], tmp_path, monkeypatch)
|
|
assert "Savings by Source" not in out
|
|
|
|
|
|
def test_coalesced_savings_rows_count_every_event_in_text_and_summary():
|
|
"""A PERF row summed from 3 recordings is 3 events; a row without the field is 1."""
|
|
report = PerfReport(
|
|
perf_records=[
|
|
PerfRecord(
|
|
timestamp="2026-06-05 10:00:00,000",
|
|
request_id="hr_1",
|
|
savings_breakdown=[
|
|
{"source": "tool_search", "tokens": 300, "events": 3},
|
|
{"source": "router", "tokens": 5},
|
|
],
|
|
)
|
|
]
|
|
)
|
|
|
|
by_source = {row["source"]: row["events"] for row in build_perf_summary(report)["by_source"]}
|
|
assert by_source == {"tool_search": 3, "router": 1}
|
|
assert "tool_search: 3 events" in analyzer.format_report(report)
|