Commit Graph
11 Commits
Author SHA1 Message Date
1d112cd36c fix(proxy): show turn-hook savings in /stats extension_savings, incl. Codex WS (#4233)
## Description

### The test we ran

A Daytona sandbox running Headroom Enterprise 0.4.3 (core 0.39.1), with
Codex CLI 0.162.1 and Claude Code connected to real GitHub, Context7 and
Fetch MCP servers, working through the workshop tasks.

### What we found

- The tool-search plugin's turn hook rewrote the Codex `/v1/responses`
input and saved about **8K tokens per request**. `run_request_hooks`
booked that saving through `ctx.record_savings(hook.savings_source,
tokens=...)`.
- After many such requests, `GET /stats` still showed
**`extension_savings: {}`**.

There were two causes:

1. **`extension_savings` has no writer in core.** The only thing that
fills it is `PrometheusMetrics.record_extension_savings`, and core never
calls it. Attribution rows from `ctx.record_savings` /
`record_scope_savings` go to `savings_by_source`, which `/stats` shows
as `savings.by_source`.
2. **The Codex WebSocket path drops attribution entirely.** Both WS
compression calls (first frame and later frames) called
`_compress_openai_responses_payload_in_executor` without `savings_tags`.
So `TurnContext.tags` was a throwaway `{}`, and the hook's row (along
with the built-in tool-search deferral row) never reached
`savings.by_source` either. Codex uses the WS transport by default. The
HTTP `/v1/responses` path already passed `savings_tags`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- **`/stats.extension_savings`** now reports measured savings per
source: the **realized** attribution rows, plus
`record_extension_savings` totals. If a source also calls
`record_extension_savings` (`headroom-hdr` reports through both APIs on
every request), it shows the larger of its two totals rather than their
sum, so hdr is not double-counted. Nothing ties an attribution row to a
`record_extension_savings` call, so a plugin that used the two APIs on
different turns would be under-reported, never over-reported.
Unrealized/estimated rows (tool-search deferral, `realized=False`) stay
out and remain visible in `savings.by_source`.
- **Codex WS:** each frame compresses into its own `savings_tags` dict.
Its rows are merged into the session's pending list on the event loop,
and only when that frame was actually rewritten and sent. Each per-turn
`RequestOutcome` carries the pending rows and then clears them. A turn
whose `response.completed` has no input usage leaves its rows pending,
the same gate as its headline `saved_delta`
(`_deferrable_savings_delta`). Rows no turn claimed go out with the
session-end residual outcome, and only if the residual has input usage.
A fresh dict per frame matters because a timed-out compression thread
keeps running (`_run_compression_in_executor` can't cancel it), and it
must not book rows for a frame that went out uncompressed.

**Realized vs estimated:** `savings.by_source` already keeps one row per
`(source, realized)`, so realized and unrealized savings are shown
separately. `extension_savings` now carries only the realized ones.

**Review fix (@JerrettDavis, bounded WS pending list):** the
session-wide pending list grew one row per rewritten frame, while
`savings_attribution.from_tags` read only the first `MAX_SOURCES` (32)
rows. After 33 frames that each booked a 100-token `tool_search` row,
the handoff cleared all 33 rows but booked only 3,200 of the 3,300
tokens. `savings_attribution.accumulate(rows, items)` replaces the
`extend` at both WS frame paths.

**Bounded rows, nothing dropped (follow-up to @JerrettDavis's two
reviews):** the 32-row cap that silently dropped rows is gone from
`record_savings`, `from_tags` and `accumulate`. Storage is bounded by
coalescing plus one overflow row:
- every writer goes through the one `accumulate` helper: the per-request
ledger (`record_savings`, so every handler) and the WS pending list.
Rows with the same `(source, realized, estimated)` key are summed into
one row (`tokens`, `usd`, `events`). `details` are kept only while every
summed row has the same `details`.
- **cardinality bound:** a ledger keeps at most `MAX_NAMED_ROWS` (256)
named rows. A new source past that bound is summed into an `other` row
with the same `(realized, estimated)`, and its `details` are dropped.
Tokens, dollars and events are conserved: nothing is dropped, only the
source name. `tool_search` never folds, because
`tool_schema_savings_policy` finds the deferral's row by name to lower
it when hooks un-defer tools.
- **finite sums:** a row's `usd` saturates at ±`MAX_ROW_USD` (1e15) and
its `tokens` stay within 0..`MAX_ROW_TOKENS` (10^15). A row that was
clipped, or summed from one that was, carries `saturated: true`. Every
total stays finite, so the Prometheus exposition and
`json.dumps(allow_nan=False)` never see `inf`/`nan`.
- each row carries `events`: how many recordings it stands for (1 on a
fresh row, summed on merge). `PrometheusMetrics.record_request`
(`headroom_savings_attribution_events_total`), the OTel
`headroom.savings.attribution.events` counter and the perf analyzer's
"Savings by Source" add that count instead of 1 per row, so 33 coalesced
frames are 33 events, not 1 (Devin's finding). A row without `events`
(older PERF logs, rows built by hand) counts as 1.
- a row that a reader edited in place (`reconcile_deferred_tokens`, the
tool-search plugin) is summed defensively, so a junk value counts as 0
instead of raising inside the handler. The core deferral and the native
deferral hook both book `tool_search` as estimated, so they now share
one row, and the reconcile debits that row; totals are unchanged.
- `MAX_SOURCES` is no longer used. No importer was found outside this
repo's own test, but `savings_attribution` is imported by plugins, so
the name is kept behind a module `__getattr__` that raises a
`DeprecationWarning`.
- **Scope of the bound:** it is per ledger (one request, or one WS
session's pending list). The process-wide
`PrometheusMetrics.savings_by_source` table has no source cap yet; that
is a separate PR.

### Before / after

| | origin/main | this PR |
|---|---|---|
| A turn hook books 8,000 tokens on 2 requests →
`/stats.extension_savings` | `{}` | `{"tool_search": 16000}` |
| An hdr-style plugin calls both APIs (300 tokens × 2) | `{"hdr": 600}`
| `{"hdr": 600}` (not 1200) |
| A Codex WS turn with a folding hook → attribution rows on the turn
outcome | `[]` | `[("ws_fold", realized=True)]` |
| 33 WS frames, each booking `tool_search` 100 tokens, then one billed
turn → booked `tool_search` tokens (helper-level, the reviewer's
scenario) | 3,200 of 3,300, 33 rows stored | 3,300, 1 row stored |
| Real WS handler, 33 frames each with a folding hook (`ws_fold` row)
plus a 100-token hook row → non-`ws_fold` tokens booked | 1,600 of 3,300
(66 rows stored) | 3,300 |
| Same, repeated source → rows stored / hook events in
`headroom_savings_attribution_events_total` | aecae31b: 2 rows / 1 event
| 2 rows / 33 events |
| Same, 33 distinct sources → rows stored / hook events / `other` row |
aecae31b: 29 rows / 28 events / yes (6 sources folded) | 34 rows (33 +
`ws_fold`) / 33 events / no |
| Same, 300 distinct sources → rows stored / tokens and USD booked |
rows past 32 dropped | 257 rows (256 named + `other`), every token and
dollar booked |
| One ledger, 2,000 distinct sources plus repeats | rows past 32 dropped
| 256 named rows + 2 `other` rows (realized, estimated); tokens, USD and
events conserved |
| Same source booked 1e308 USD twice, then 1.0 | `inf` in `/stats` and
Prometheus | `usd` = 1e15, `saturated: true`, exposition finite |
| One HTTP request, 40 distinct sources + 1 repeat, 100 tokens each
(`record_savings`, every handler) → booked tokens / events | 3,200 of
4,100 / 32 (8 rows dropped) | 4,100 / 41, 40 rows stored |

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

-
`tests/test_compression_observability.py::test_turn_hook_savings_surface_in_stats_extension_savings`:
fails on origin/main, passes here.
-
`tests/test_openai_codex_ws_lifecycle.py::test_ws_turn_hook_savings_reach_the_turn_outcome[billed|usage_less]`:
drives the real WS handler with a registered hook. `billed` fails on
origin/main (no rows). `usage_less` fails if the input-usage gate is
removed.
- Related suites pass: the WS lifecycle, per-frame memory, savings
deferral and compression scheduler tests, extension attribution,
responses context compaction, tool-search repair after hooks, and the
handler outcome tag invariant (136 passed).
- ruff check, ruff format --check and mypy pass.
-
`tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked`
(review fix, renamed from `..._beyond_max_sources_...`), parametrized
over a repeated source × 33 frames, 33 distinct sources, and 300
distinct sources (folds into `other`, 257 rows, USD conserved): drives
the real WS handler through the rewritten frames. The upstream holds its
events until all 33 frames arrive, so every row is pending when the one
billed outcome claims it. The rows are booked through the real
`from_tags` → `PrometheusMetrics.record_request`. The test asserts that
every token is booked (3,300), that the hook rows count 33 events, that
there is no `other` row, and that the stored row count equals the
distinct keys (2 for a repeated source, 34 for 33 distinct sources plus
`ws_fold`).
-
`tests/test_extension_attribution.py::test_a_request_with_40_sources_books_every_token`:
one request's ledger with 40 distinct sources and one repeat books 4,100
tokens and 41 events through `PrometheusMetrics.record_request`.
-
`tests/test_extension_attribution.py::test_rows_coalesce_per_source_realized_and_estimated`:
same source with different `realized`/`estimated` stays in separate
rows; same key sums tokens and events.
-
`tests/test_observability_metrics.py::test_coalesced_savings_rows_count_every_event`:
the OTel events counter adds a row's `events` (33), and a row without
the field counts as 1.
-
`tests/test_cli_perf_format.py::test_coalesced_savings_rows_count_every_event_in_text_and_summary`:
both perf folds (`headroom perf` text report and the JSON `by_source`
summary) add a row's `events`.
-
`tests/test_extension_attribution.py::test_merging_into_a_junk_row_does_not_raise_inside_the_handler`:
a ledger row edited in place with junk (`tokens=None`) is summed as 0
instead of raising out of `record_savings`.
-
`tests/test_extension_attribution.py::test_distinct_sources_past_the_bound_fold_into_other_and_conserve_totals`:
2,000 sources plus estimated repeats give 258 rows, with tokens, USD and
events conserved.
-
`tests/test_extension_attribution.py::test_same_source_usd_saturates_instead_of_overflowing`,
`test_an_infinite_total_already_in_the_ledger_stays_saturated[±1.0]`,
`test_tokens_stay_within_zero_and_the_bound`: sums stay finite and are
flagged `saturated`.
-
`tests/test_extension_attribution.py::test_tool_search_keeps_its_own_row_past_the_bound`:
the deferral row is never folded, so the reconcile still finds it.
-
`tests/test_extension_attribution.py::test_max_sources_is_deprecated_not_removed`:
the old name still imports, with a `DeprecationWarning`.
-
`tests/test_turn_hooks.py::test_request_runner_attributes_savings_with_handler_counters`:
expected row now includes `events: 1`.

### Test Output

```text
# review fix, savings_attribution.py + openai.py reverted to the previous commit, new test:
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_max_sources_are_all_booked[repeated_source]   (assert 1600 == (100 * 33))
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_max_sources_are_all_booked[distinct_sources]  (assert 1600 == (100 * 33))
# helper-level, the reviewer's scenario (33 frames x tool_search 100):
append:     33 rows stored, 3200 booked
accumulate:  1 row stored,  3300 booked
40 distinct sources x 100: 29 rows stored, 4000 booked ("other" = 1200)
# no-cap follow-up, the 5 source files reverted to aecae31b, new/updated tests:
FAILED tests/test_extension_attribution.py::test_a_request_with_40_sources_books_every_token
FAILED tests/test_extension_attribution.py::test_rows_coalesce_per_source_realized_and_estimated
FAILED tests/test_observability_metrics.py::test_coalesced_savings_rows_count_every_event
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked[repeated_source]
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked[distinct_sources]
# with the no-cap follow-up: every test file that touches the savings ledger (21 files)
# plus gateway, perf format/per-port logs and /stats suites: 831 passed, 5 skipped (90d5e132)
# with the fix (aecae31b):
WS lifecycle, per-frame memory, savings deferral, compression scheduler, WS timings, pyo3 WS compression,
outcome tag invariant, extension attribution, compression observability, tool-search aggregation,
responses context compaction, proxy openai: 189 passed
ruff 0.17.0 check + format --check: clean on changed files
mypy headroom --ignore-missing-imports: no new errors (1 pre-existing local error in integrations/langchain/chat_model.py:380, untouched)
```

## Real Behavior Proof

- Environment: macOS 26.7, Python 3.12.13, this branch. The WS handler
is driven through the repo's fake-upstream harness
(`tests/test_openai_codex_ws_lifecycle.py`), which runs the real
`handle_openai_responses_ws`, `run_request_hooks` and outcome funnel
with only the upstream socket scripted; `/stats` runs on the real app
via `TestClient`; OTel uses the real `HeadroomOtelMetrics` with an
in-memory reader.
- Exact command / steps: `pytest tests/test_openai_codex_ws_lifecycle.py
tests/test_extension_attribution.py tests/test_observability_metrics.py
tests/test_compression_observability.py tests/test_turn_hooks.py` (plus
the other ledger suites listed under Testing), each run once on this
branch and once with the source files reverted.
- Observed result: with a registered folding hook, the hook really
rewrites the forwarded `input` (the upstream received only `"new"`) and
the turn outcome carries `[("ws_fold", realized=True, tokens>0)]`.
`/stats.extension_savings` reports `{"tool_search": 16000, "hdr": 600}`
for the two-request scenario above. 33 rewritten WS frames book 3,300
hook tokens and 33 events with no `other` row; 300 distinct sources fold
into 256 named rows plus `other` with every token and dollar booked; one
request with 40 distinct sources plus a repeat books 4,100 tokens and 41
events in 40 rows.
- Not tested: a real Codex CLI session against OpenAI on this branch, a
real Codex session with more than 32 rewritten frames before a billed
turn, and the real `headroom-tool-search` plugin were not re-run after
the patch. The sandbox numbers above (about 8K tokens/request,
`extension_savings: {}`) are from 0.39.1 before the fix.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: `/stats.extension_savings` now
includes realized attribution rows, and Codex WS turns now carry
attribution rows. Request bodies and headline totals are unchanged.
- Kill switch / disable path: none needed; the change only affects
reporting
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- plus the code-reviewer agent on each review fix. Its earlier SHOULD on
the collapsed `events` count is now fixed (rows carry `events`).
- [x] This PR is ready for human review

## Additional Notes

- `record_extension_savings` stays public and unchanged.
- The Prometheus exposition format is unchanged;
`headroom_savings_attribution_events_total` now counts recordings, which
is what it counted before rows were coalesced.
- Ledger rows now carry an `events` field. Readers that ignore unknown
keys are unaffected.
- A reader that finds "rows a hook added" by slicing the ledger at its
old length (`ledger[before:]`) can miss a hook's tokens when the hook
books a key that already had a row. Core has no such reader;
`headroom-kiro`'s handler does and should switch to a before/after sum.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-17 11:29:04 -07:00
M BarryandJD Davis e01ad350dc fix(pricing): suppress litellm's Provider List banner that corrupts perf JSON (#4197)
## Description

`headroom perf --format json` (and `--format csv`) can print litellm's
red "Provider List: https://docs.litellm.ai/docs/providers" banner to
**stdout** ahead of the document. The output then doesn't parse:

```text
$ headroom perf --hours 24 --format json | python -c "import json,sys; json.load(sys.stdin)"
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
```

Here is the path. `build_perf_summary` prices each model through
`headroom.pricing.litellm_pricing.resolve_litellm_model`, which probes
candidates with `litellm.cost_per_token`. For a model litellm can't map
to a provider, `get_llm_provider` calls `print()` on the banner unless
`litellm.suppress_debug_info` is set. `headroom.providers.litellm`,
`headroom.providers.anthropic` and `headroom.learn.analyzer` already set
that flag, but the perf CLI imports none of them. `litellm_pricing`
imports litellm without setting it. #2860 cached model resolution, so
the banner now prints once per model per process; a fresh CLI run is a
new process, so it still prints every time.

This PR sets `litellm.suppress_debug_info = True` right after
`litellm_pricing`'s guarded import, so every pricing caller is covered,
perf included.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/pricing/litellm_pricing.py`: set
`litellm.suppress_debug_info = True` after the import, the same flag the
provider modules already set.
- `tests/test_cli_perf_format.py`:
`test_perf_json_survives_litellm_provider_banner`. In a fresh
interpreter (litellm's default `suppress_debug_info = False`), it builds
the perf JSON summary for a report with an unknown model and asserts
that "Provider List" is absent from stdout and that the output parses
with `json.loads`. It uses a subprocess because the flag is set at
import time, and reloading the module in-process would leave stale class
references for later tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_cli_perf_format.py -k banner -v
tests/test_cli_perf_format.py::test_perf_json_survives_litellm_provider_banner PASSED [100%]
================= 1 passed, 22 deselected, 1 warning in 2.89s ==================
# fails on main: "Provider List" found in output, JSONDecodeError

$ pytest tests/test_cli_perf_format.py tests/test_pricing_litellm.py \
    tests/test_pricing_litellm_model_resolution.py tests/test_litellm_probe_fallback_logs.py \
    tests/test_savings_tracker_litellm_resolution_cache.py
======================== 46 passed, 1 warning in 3.07s =========================

$ ruff check headroom/pricing/litellm_pricing.py tests/test_cli_perf_format.py
All checks passed!
$ ruff format --check headroom/pricing/litellm_pricing.py tests/test_cli_perf_format.py
2 files already formatted
$ mypy headroom/pricing/litellm_pricing.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.13, litellm 1.88.1. A real proxy log
directory with 24 hours of traffic (1,668 requests across several model
names).
- Exact command / steps: `headroom perf --hours 24 --format json >
out.json`, then `python -c "import json; json.load(open('out.json'))"`.
Run once with the installed 0.40.0 and once with this branch on
`PYTHONPATH`, both from a cwd outside the checkout.
- Observed result: 0.40.0 output starts with `\x1b[1;31mProvider List:
https://docs.litellm.ai/docs/providers\x1b[0m` and `json.load` raises
`JSONDecodeError: Expecting value: line 2 column 1`. This branch's
output starts with `{` / `"window_hours": 24.0` and parses
(`total_requests` 1668).
- Not tested: litellm versions other than 1.88.1. `suppress_debug_info`
is a long-standing public module attribute, already relied on by
`headroom/providers/litellm.py`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only that litellm's debug banner and
"Give Feedback / Get Help" hints are no longer printed by processes that
use `headroom.pricing.litellm_pricing`. The proxy already suppressed
them through `headroom.providers.litellm`. Pricing results are
unchanged.
- Kill switch / disable path: none needed; there's no functional
behaviour to disable.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert the one-line change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- No docs change: the documented `perf --format json` contract (a JSON
document on stdout) now actually holds.
- Earlier reports #613 and #2851 covered the same banner on the proxy
path. This is the CLI/pricing path, which those fixes didn't reach.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4197"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-15 16:32:36 -05:00
9a72bc0242 perf: add savings audit output (#1211)
## Description
- add a savings_audit object to perf JSON output
- expose raw prompt reduction, logged savings, accounting delta,
suspicious records, and top saving requests
- add audit columns to aggregate CSV output and a compact text report
section

## Testing

- [x] Python compilation check passes
- [x] Current refreshed head has no failing GitHub checks

### Test Output

```text
python -m py_compile headroom/perf/analyzer.py headroom/cli/perf.py tests/test_cli_perf_format.py
Result: passed
GitHub status rollup: no failures
```
- python -m py_compile headroom/perf/analyzer.py headroom/cli/perf.py
tests/test_cli_perf_format.py
- pytest tests/test_cli_perf_format.py could not run locally because
pytest is not installed in this Python environment

## Type of Change

- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring

## Changes Made

- Add a savings_audit object to PERF JSON output.
- Expose prompt reduction, recorded savings, accounting deltas,
suspicious records, and top-saving requests.
- Add audit columns to aggregate CSV output and a compact text report.

### Test Output

```text
python -m py_compile headroom/perf/analyzer.py headroom/cli/perf.py tests/test_cli_perf_format.py: passed
Current refreshed head: no failing GitHub checks
```

## Real Behavior Proof

- Environment: current refreshed PR head and the PERF analyzer/CLI
formatting test environment.
- Exact command / steps: compiled the changed analyzer, CLI, and
formatting-test modules and inspected the generated JSON, CSV, and text
audit surfaces.
- Observed result: audit data exposes raw reduction, recorded savings,
deltas, suspicious rows, and top requests without changing existing PERF
records.
- Not tested: a production-scale PERF log corpus.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
2026-10-02 23:46:58 -05:00
Tejas Chopra 1390d89715 fix(shaper): measure output-token steering, and stop claiming savings we did not measure (#3377)
## What this does

Verbosity steering already shipped (BETA, L2). This branch measures it
properly, fixes two ways it was misbehaving, and changes what Headroom
is willing to *claim* about it.

Net effect for a user: steering saves more than it did, on more models,
and the dashboard stops showing a savings percentage that was never
measured on their traffic.

## Three changes worth reviewing separately

### 1. The L3 steering text had two defects, both measured

**"Reproduce code, error strings, numbers, units, and API or CLI names
exactly"** read as a licence to emit code. Against the same SWE-bench
turn it left gpt-5.2 unchanged but inflated gpt-5.1 by 58% (1631 → 2573
output tokens), cutting L3's saving on that model from 51% to 23%.
Replaced with a completeness floor that forbids dropping task-critical
content — negations included — without instructing the model to
reproduce anything. The whole OpenAI family went from *negative* to
+31.2%.

**"reference by path and line"** was too soft. Told to stop restating
code, models wrote "change the check in the mixin" and the reader had to
ask which file. Measured with a blind judge playing the engineer who
must apply the fix: of four blocker categories only "under-specified
location" moved between L0 and L3 (0 → 3 across 9 paired tasks). Notably
the *predicted* failure — "omit rationale" → "but why?" — scored zero at
both levels. Making the citation obligatory took that category to 0 and
turn inflation from +23.1% to 0.0%.

Suppressing content is safe. Suppressing the pointer to it is not: the
reader buys it back at the cost of a whole turn, which at agentic
context sizes is far more expensive than the tokens saved.

### 2. The savings table now ships empty

`MODELLED_REDUCTION` previously carried built-in factors. It now ships
`{}` plus a `register_modelled_factors()` seam.

A credible reduction factor is not a constant. It depends on the model
family, the shape of the turn, and the exact steering text — and
producing one means running a paired benchmark across models and paying
for both arms. A constant compiled into the source is a number measured
on somebody else's traffic.

So `estimate_from_model` returns `None` for every level and the
dashboard renders its "not measured" fallback. That is the correct
rendering of *not measured*; a number nobody measured is not. Both real
paths are unchanged and documented at the table: run a holdout (which
outranks this tier entirely), or register factors from a benchmark.

The old built-in figure is a good argument for this: it claimed `(0.336,
0.738)` from five synthetic prompts, measured against a version of the
L3 string that has since changed twice. It over-reported by roughly 2×
at the optimistic end — the direction that matters for a number on a
dashboard.

### 3. Everything else

Thinking tokens split from visible output (`thinking_tokens.py`), the
estimator that was wired but never ran now runs, and the
`OUTCOME_OBSERVED` pipeline stage.

An earlier commit in this range adds a `headroom-output` plugin and a
writable `PRE_SEND_PARAMS` body hook; a later one removes both. Live
measurement showed per-turn effort routing ~15× underwater on mechanical
turns, so the hypothesis it was built on did not survive. The net diff
has neither, and a test now pins the hook as deliberately absent — an
unused writable-body seam is a liability, not an extension point.

## How it was measured

SWE-bench Verified, 12 instances carrying the real source files each
gold patch touches (6k–48k chars), stratified by the dataset's
difficulty label, run at L0 and L3 across six models and paired on
instance. Truncated pairs excluded — a capped response's length is the
ceiling, not the model's choice.

| | pairs | reduction |
|---|---|---|
| Anthropic (opus, sonnet, haiku) | 33 | 18.2% |
| OpenAI (terra, luna, sol) | 28 | 31.2% |

61 usable pairs, median +31.8%, mean +23.0% (SE 5.3, t=4.4), all six
positive in sign, 13% of individual pairs negative.

**On power, since it is easy to over-read the per-model figures:** only
luna (t=6.0), terra (t=3.0) and sonnet (t=2.7) are individually
distinguishable from zero. sol, haiku and opus are not (t≈1.3, 1.3,
0.7). They are single samples per instance against a model whose
*unsteered* output varies by a median 33% run to run — measured
directly, by re-running the L0 arm on identical prompts. The aggregate
effect is solid; a single-model number from this run is not.

## Risk

- Steering is the one lever that mutates the provider prefix-cache key,
so the text is byte-stable and applied once per conversation;
`steering_allowed_for` suppresses it entirely in `mode="cache"`.
- The feature stays opt-in (`available_in=STABLE`, no
`default_enabled_in`). L3 is a visible change to every response and
should be a choice.
- `shaper_enabled_for` keeps byte-faithful forwarding intact: default-on
plus `optimize=False` means don't shape, because creating a `system`
field where the client sent none breaks the contract.

## Testing

612 targeted tests pass; full suite green apart from
`test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`,
which fails identically on `main`.

The final commit fixes a UP037 lint error that arrived with the
savings-rendering work on main. It is unrelated to this branch but fails
CI lint for anything opened on top of it.
2026-09-01 15:28:09 +05:30
Tejas ChopraandClaude Opus 5 56f809ce39 perf: render the savings each source actually reported
The PERF line has carried `savings=` since attribution was added, and the
parser has decoded it into `savings_breakdown` all along — but nothing ever
rendered it. An operator reading `headroom perf` could not tell that a paid
extension had contributed anything, because the only per-engine table is
built from `Transform NAME:` lines, which just the SDK pipeline emits.

Dollars are printed BESIDE tokens, never folded into them. A router that
sends the same tokens to a cheaper model saves real money and exactly zero
tokens; every token-savings channel in the proxy records nothing for it (and
three of the four reject a non-positive value outright). A `0 tokens` row
with a `$` figure is the honest rendering of that, and one combined number
would be a saving nobody made.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PbXdkNvPqxmrPJxxyvwtZS
2026-08-31 16:33:53 +05:30
gglucass 6c9f41e08c perf(perf): skip rotated logs outside the requested window (#3081)
## Description

`parse_log_files(last_n_hours=N)` reads every `proxy.log*` file in full
— line by line, applying the PERF / STAGE_TIMINGS / ROUTER regexes to
each — and only then filters records against the cutoff. The cost of a
windowed query is O(retained log history), not O(window).

`/stats` is the hot caller. `_build_stats_payload` recomputes throughput
over `last_n_hours=1.0` behind a 10s cache TTL, so anything polling the
endpoint re-reads and re-regexes the entire rotated set every 10 seconds
for an answer that lives in the tail of the newest file or two.

Rotation caps the log directory at 10 MB × 5 backups
(`proxy/helpers.py`), so this is a bounded ~60 MB rather than an
unbounded leak. But it is a fixed tax that ramps up as a user's logs
fill toward that ceiling and then stays there — on a machine that has
reached the cap it is ~0.43s of pure waste on every stats rebuild.

The fix: skip any file whose mtime predates the cutoff. The logs are
append-only, so a file untouched since before the window cannot contain
a record inside it. `--hours 0` ("all data") still reads everything.

## Type of Change

- [x] Performance improvement

## Changes Made

- `parse_log_files` prunes rotated files by mtime before opening them;
files are `stat`'d once and the value reused for the ordering
(previously `stat`'d once per file anyway, as the sort key).
- A file that rotates away between `glob` and `stat` is skipped instead
of raising `OSError`.
- New `PerfReport.log_files_skipped` so coverage reporting stays honest
— `log_files_read` on its own would silently understate how much log
exists on disk. Defaulted, so existing callers are unaffected.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Both new tests were confirmed to fail against unpatched `main`. The
windowed one fails on behavior (`total_lines_parsed`: `assert 2 == 1`),
not merely on the new field — the assertion order is deliberate, since a
read-then-filter implementation produces the same records and only
differs in work done.

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_cli_perf_format.py \
    tests/test_proxy_dashboard_stats_cache.py tests/test_agent_savings.py -q
59 passed, 1 skipped, 1 warning in 3.36s

$ uvx ruff check headroom/perf/analyzer.py tests/test_cli_perf_format.py
All checks passed!

$ uvx ruff format --check headroom/perf/analyzer.py tests/test_cli_perf_format.py
2 files already formatted

$ uv run --frozen --extra dev mypy headroom/perf/analyzer.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.10.18, headroom-ai at
6d2254df, against a real `~/.headroom/logs` holding 54 MB across six
rotations (`proxy.log` + `.1`–`.5`) from a proxy that had been running
for weeks.
- Exact command / steps: pointed `analyzer.LOG_DIR` at the live log
directory and timed `parse_log_files(last_n_hours=1.0)` three times,
taking the median; ran it once on this branch and once with
`headroom/perf/analyzer.py` stashed back to `main`.
- Observed result: main = 0.426s median, 6 files read, 246,819 lines
parsed. This branch = 0.141s median, 2 files read, 4 skipped, 48,147
lines parsed. 3.0x faster, 80% fewer lines parsed, identical throughput
figure. The two files still read are the live log and one rotation that
had been written inside the last hour, which is correct.
- Not tested: Windows and Linux (the mtime semantics used here are
POSIX-standard and `pathlib` handles both, but I ran only macOS). No
benchmark on a log directory below the rotation ceiling — the win there
is proportionally smaller by construction, since there is less stale
history to skip.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this is a pure read-path
optimization inside the perf log parser.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: no. Windowed queries return the same
records; only the work to produce them changes. `--hours 0` is
untouched.
- Kill switch / disable path: n/a — revert the commit. There is no flag
because there is no behavior to toggle.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: single-commit revert; `PerfReport.log_files_skipped` is
a defaulted field, so no persisted or serialized data depends on it.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
2026-08-17 20:20:56 -07:00
akothari-godaddy 7ddcbcb616 perf: surface optimization overhead diagnostics (#1212)
## Summary
- add overhead diagnostics to perf JSON output
- report optimization p50/p95/p99, slow request percentage, per-stage
totals/percentiles, and top slow requests
- update text report and recommendations to point at the slowest stage
and HEADROOM_COMPRESSION_TIMEOUT_SECONDS when optimization is
consistently slow

## Verification
- python -m py_compile headroom/perf/analyzer.py
tests/test_cli_perf_format.py
- pytest tests/test_cli_perf_format.py could not run locally because
pytest is not installed in this Python environment
2026-07-15 21:04:21 +00:00
Shlok TiwariandAntigravity Agent 0d89c674cd feat: measure and surface token throughput (tokens/sec) through the proxy (#983)
## Description

This PR implements measuring and surfacing token throughput
(tokens/second) through the proxy in the `headroom perf` CLI/analyzer
and the dashboard UI. It tracks multiple throughput metrics—Input
(wall-clock/active), Compression, Forward, and Generation
throughput—supporting both rolling percentiles (p50/p95) and current
(last 5 minutes) metrics.

Closes #959

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Proxy Instrumentation (`headroom/proxy/outcome.py`)**: Added
`total_ms`, `tok_out`, and `ttfb_ms` to the structured `PERF` logging
payload.
- **Log Parsing & Computations (`headroom/perf/analyzer.py`)**: Updated
log parsing to read `STAGE_TIMINGS` and correlation fields from `PERF`,
computing active/wall-clock throughputs for input, compression, forward,
and generation stages.
- **API Exposing (`headroom/proxy/server.py`)**: Exposes calculated
rolling throughput percentiles and last-5-minute averages under the
`throughput` field in `/stats`.
- **Dashboard UI Layout
(`headroom/dashboard/templates/dashboard.html`)**: Refactored the
dashboard grid layout from 3 columns to 4 columns to house the new
throughput hero card showing real-time token performance.
- **Verification Tests (`tests/test_cli_perf_format.py`)**: Added test
coverage specifically targeting token throughput log parser extraction,
stage correlation, math correctness, and edge-case handling (empty
fields, division by zero).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$env:PYTHONPATH="c:\Users\hp\Desktop\Headroom_oss"; .venv\Scripts\pytest tests/test_cli_perf_format.py
============================= test session starts =============================
platform win32 -- Python 3.11.15, pytest-9.1.0, pluggy-1.6.0 -- C:\Users\hp\Desktop\Headroom_oss\.venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\hp\Desktop\Headroom_oss
configfile: pyproject.toml
plugins: anyio-4.13.0
collecting ... collected 14 items

tests/test_cli_perf_format.py::test_build_perf_summary_totals_and_pct PASSED [  7%]
tests/test_cli_perf_format.py::test_build_perf_summary_by_model_and_transform PASSED [ 14%]
tests/test_cli_perf_format.py::test_build_perf_summary_empty_report_no_zero_division PASSED [ 21%]
tests/test_cli_perf_format.py::test_perf_records_as_dicts_roundtrips_fields PASSED [ 28%]
tests/test_cli_perf_format.py::test_perf_json_format PASSED              [ 35%]
tests/test_cli_perf_format.py::test_perf_json_raw_is_array PASSED        [ 42%]
tests/test_cli_perf_format.py::test_perf_json_raw_preserves_client_field PASSED [ 50%]
tests/test_cli_perf_format.py::test_parse_perf_line_preserves_client_field PASSED [ 57%]
tests/test_cli_perf_format.py::test_perf_csv_by_model PASSED             [ 64%]
tests/test_cli_perf_format.py::test_perf_csv_raw_per_record PASSED       [ 71%]
tests/test_cli_perf_format.py::test_perf_text_default_unchanged PASSED   [ 78%]
tests/test_cli_perf_format.py::test_perf_rejects_unknown_format PASSED   [ 85%]
tests/test_cli_perf_format.py::test_parse_perf_line_preserves_blank_client_field PASSED [ 92%]
tests/test_cli_perf_format.py::test_throughput_parsing_and_calculations PASSED [100%]

============================== warnings summary ===============================
.venv\Lib\site-packages\_pytest\config\__init__.py:1464
  C:\Users\hp\Desktop\Headroom_oss\.venv\Lib\site-packages\_pytest\config\__init__.py:1464: PytestConfigWarning: Unknown config option: asyncio_mode
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

.venv\Lib\site-packages\opentelemetry\util\_importlib_metadata.py:32
  C:\Users\hp\Desktop\Headroom_oss\.venv\Lib\site-packages\opentelemetry\util\_importlib_metadata.py:32: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
    return EntryPoints(ep for group_eps in eps.values() for ep in group_eps)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================= 14 passed, 2 warnings in 4.77s ========================
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.15
- Exact command / steps: Run the pytest suite against the newly created
token throughput parsing routines:
`$env:PYTHONPATH="c:\Users\hp\Desktop\Headroom_oss";
.venv\Scripts\pytest tests/test_cli_perf_format.py`
- Observed result: The suite executes 14 tests successfully, including
the newly added `test_throughput_parsing_and_calculations` verification
test verifying mathematical precision and fallback logic.
- Not tested: None (all metrics are fully covered by unit tests in
`test_cli_perf_format.py`)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Backwards compatibility: Older log outputs lacking `tok_out` or
`ttfb_ms` parse cleanly and fallback defaults prevent parser crashes.

---------

Co-authored-by: Antigravity Agent <agent@antigravity.local>
2026-06-17 09:42:38 -05:00
Michael Sam 6d3f39f213 feat: add dashboard agent usage stats (#814)
## Description

Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.

Fixes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### New Files

**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.

### Modified Files

- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.

## Testing

- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
2026-06-12 14:12:22 -05:00
Michael Sam d2cdab268d feat(proxy): add agent-90 savings profile (#830)
## Summary
- add an `agent-90` savings profile with cross-agent proxy env exports
- wire the profile into proxy/router runtime kwargs, including
force-Kompress routing and a smaller read-protection window
- expose effective savings-profile config in `/stats` and add focused
regression coverage

## Type of change
- [x] feat (non-breaking change which adds functionality)
- [ ] fix (non-breaking change which fixes an issue)
- [ ] docs
- [ ] test/CI-only
- [ ] refactor-only

## Testing
- [x] `python3 -m py_compile headroom/agent_savings.py
headroom/cli/agent_savings.py headroom/cli/main.py headroom/cli/proxy.py
headroom/proxy/models.py headroom/proxy/server.py
headroom/transforms/content_router.py tests/test_agent_savings.py
tests/test_proxy_healthchecks.py tests/test_cli/test_wrap_persistent.py
tests/test_transforms/test_content_router.py`
- [x] `git diff --check`
- [x] manual smoke: `agent-savings --profile agent-90 --format json`
returns `HEADROOM_TARGET_RATIO=0.10`
- [x] manual smoke:
`proxy_pipeline_kwargs(ProxyConfig(savings_profile="agent-90"))` enables
`force_kompress`, system/user compression, and
`read_protection_window=2`
- [x] manual smoke: Anthropic-style `tool_result` routes through
Kompress with `target_ratio=0.10`
- [ ] `pytest` suite not run: pytest is not installed in the available
local Python environments

## Notes
This keeps agent-90 as an opt-in profile. Existing defaults remain
unchanged unless `HEADROOM_SAVINGS_PROFILE=agent-90` or
`ProxyConfig(savings_profile="agent-90")` is set.
2026-06-11 18:58:06 -05:00
9fe4886cf6 feat(perf): add --format {text,json,csv} to headroom perf (#648)
* feat(perf): add structured summary/record builders to analyzer

parse_log_files() already returns a fully-structured PerfReport, but
the only way to read it was the colored text report. Add reusable
machine-readable views so CI guards, dashboards, and agent harnesses
can consume perf data without scraping ANSI text:

- build_perf_summary(report) -> dict with the aggregated KPIs
  (savings_pct, cache_hit_pct, by_model, by_transform, ...), mirroring
  format_report() numbers exactly.
- perf_records_as_dicts(report) -> per-record list for --raw output.
- PERF_RECORD_FIELDS: shared column order for CSV/raw consumers.

Pure additions; no behaviour change to existing callers. Part of #595.

* feat(perf): add --format {text,json,csv} to headroom perf

Adds a machine-readable output path to the perf command (issue #595):

- --format json: aggregated summary (default) or, with --raw, a JSON
  array of per-record dicts.
- --format csv: per-model breakdown (default) or, with --raw, one row
  per PERF record using the shared PERF_RECORD_FIELDS column order.
- --format text (default): unchanged human-readable report.

Enables CI guards (jq '.savings_pct < 70'), dashboards, and agent
wrappers to consume perf data without scraping ANSI text.

Closes #595.

* test(perf): cover --format json/csv and structured builders

Unit tests for build_perf_summary (totals, savings/cache pct,
by_model/by_transform, empty-report zero-division guard) and
perf_records_as_dicts, plus CliRunner integration tests for
--format json, json --raw, csv, csv --raw, the unchanged text
default, and rejection of an unknown format. Part of #595.

* fix(perf): rename transform loop var to satisfy mypy

The structured-summary builder reused `recs` for both the per-model
(list[PerfRecord]) and per-transform (list[TransformRecord]) groupings, so
mypy flagged the second assignment as an incompatible-type reuse
(analyzer.py:704). Rename the transform loop variable to `t_recs` so each
loop keeps a single element type. No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Kumario1 <ramsakal.ipec@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-06 22:43:10 -07:00