Commit Graph
13 Commits
Author SHA1 Message Date
00436a8b11 feat(savings): book modelled output savings per request, as a token and dollar range (#4878)
## Description

Benchmark factors registered with `register_modelled_factors` reached
`/stats` only as a percentage. They never reached the per-request
rollup, so modelled output savings had no tokens or dollars in the
lifetime totals that the local dashboard and the control-plane heartbeat
read. This PR books each shaped request's output saving as a low–high
range of tokens and dollars, from the same tier the dashboard reports.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue) — an unshaped
treatment request was booked a baseline saving
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `SavingsRecorder.request_savings_range(labels, output_tokens) -> (low,
high)` replaces `estimate_request_savings` in the outcome funnel.
- It counts only requests that were actually shaped, marked by the
`output_shaper:verbosity:L<n>` label. The treatment stratum label is
attached before shaping is attempted, so it proves nothing.
  - A holdout (control) request books nothing.
  - A response with no reported output usage (output 0) books nothing.
- The tier comes from `SavingsLedger.select_tier(level)`, the function
`best_estimate` now delegates to. Lifetime dollars therefore always come
from the tier that the dashboard's percentage comes from:
- **measured:** the request's own stratum's control mean minus its
output, with low = high. It applies only if that stratum qualifies at
the cluster gate the measurement used. A stratum outside the measurement
books 0, because the measured headline excludes it too.
    - **estimated:** the request's stratum baseline, with low = high.
    - **modelled:** the factor band, `observed × r/(1−r)` at each end.
- `SavingsTracker`:
- Books the low end everywhere it booked before (history, rollups,
OTel).
- Adds `lifetime.output_tokens_saved_high` and
`output_savings_usd_high`, priced per model at output rates.
  - Both state builders write the new keys, so a fresh tracker has them.
- Dashboard: the Output Tokens Saved tile adds "≈ $low–$high saved on
output · low–high tokens, all time", read through a component getter.

OSS is unchanged. It ships an empty factor table, so the modelled tier
never appears.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): clean on the touched files
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ git log --oneline -1
1600ecae Merge branch 'main' into feat/output-savings-dashboard-range
$ pytest -q $(rg -l "output_savings|output_tokens_saved|best_estimate" tests) tests/test_request_outcome.py
421 passed in 23.29s
$ ruff check headroom
All checks passed!
```

Hosted CI:
`test_upstream_guard.py::test_openai_routes_refuse_a_private_override_instead_of_using_the_default`
and
`test_grok_shared_proxy_call_sites.py::test_the_base_url_override_is_resolved_once_per_request`
also fail on `main` at `d6df127d`.

New tests:
- the factor band;
- one control observation without a baseline keeps the modelled tier,
returning (200, 800);
- a measured holdout in stratum A credits A's requests from A's control
mean, while a request in stratum B (no control) books 0;
- a response without output usage books nothing;
- a baseline outranks the factors;
- an unshaped treatment request books nothing;
- no factors registered;
- the lifetime range survives a restart;
- a fresh tracker has the keys;
- end to end: outcome funnel → `PrometheusMetrics.record_request` →
tracker lifetime.

## Real Behavior Proof

- **Environment:**
  - macOS (darwin), Python 3.12.
- The real `headroom proxy`, run from a scratch merge of this PR with
#4877 (version `0.45.0-dev`).
  - An isolated `HEADROOM_WORKSPACE_DIR` per scenario.
- A local fake Anthropic upstream that returns `usage.output_tokens=800`
and records each request body.
- With an extension: a licensed extension that registers factors for its
levels (values not shown here).
- **Exact command / steps:**
- Start the proxy: `HEADROOM_OUTPUT_SHAPER=1 headroom proxy --port 18799
--anthropic-api-url http://127.0.0.1:18798`, with and without the
extension.
- Send 3 `POST /v1/messages` requests, each with a different user
prompt. Identical prompts are served from the proxy's response cache and
never reach the upstream.
- Read `GET /stats` → `tokens.output_reduction`, and the tracker's
persisted `proxy_savings.json` → `lifetime`.
- **Observed result** (3 requests × 800 output tokens each, all 3
reached the upstream):

| Scenario | `output_reduction` | lifetime saved tokens and USD |
|---|---|---|
| Core only (no extension) | `available: false` | 0–0 |
| Extension loaded | `modelled`, with a low–high band | a low–high
range, low < high; each end equals 3 × 800 × r/(1−r) for that end of the
registered band, and USD is tokens × the model's output price |
| Extension, `HEADROOM_VERBOSITY_LEVEL=3` pinned | `modelled`, the L3
band | a smaller low–high range, by the same arithmetic |
| Extension requested, no licence | `available: false` | 0–0 |

- **Not tested:**
- The dashboard tile rendered in a browser. Its Alpine expression was
evaluated in Node and renders "≈ $1.50–$4.20 saved on output · 1000–2600
tokens, all time" for a range, and a single figure for a point estimate.
  - A real upstream provider.
- Streaming responses. The funnel is shared, but no streaming request
was sent.
- The measured and estimated tiers on live holdout or baseline traffic.
These are covered by unit tests, not a run.

## Runtime Rollout Safety

- **Rollout-managed feature(s):** `proxy_output_shaper`. Nothing is
booked unless steering is on and factors are registered.
- **Minimum rollout channel:** unchanged.
- **Stable/default behavior changed:** none in OSS, because no factors
are registered. With an extension, two lifetime keys are added, and the
per-request saving follows the dashboard's tier.
- **Kill switch / disable path:** `HEADROOM_OUTPUT_SHAPER=0`, or don't
load the extension.
- **Unsafe override required:** none.
- **Qualification impact:** each shaped request runs one `select_tier`
inside the recorder lock, off the event loop in the existing
`asyncio.to_thread` block.
- **Rollback path:** revert the PR. Older cores drop the two `_high`
keys when they load the savings file, and nothing else reads them.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (no docs
describe these lifetime keys)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Known gap: history checkpoints carry only the low end. If a reload
repairs lifetime from history, the band narrows for that window.
- Companion PR: #4877 adds the extension seams.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-17 09:49:13 +05:30
a9815bbb5f fix(proxy): record the provider's billed input count on streamed turns (#4052)
## Description

Every input-volume figure Headroom reports for streamed Claude traffic
was Headroom's own tokenizer estimate, not Anthropic's billed count.
That covers "input after Headroom", `/stats` `tokens.input`, the
dashboard's per-model table and the licence usage report's
`tokens_after`. Since Claude Code always streams, this was effectively
all Claude Code traffic, so customers comparing our numbers with the
Anthropic Console could not reconcile them.

**Root cause.** `emit_request_outcome` records `provider_input_tokens or
optimized_tokens`. The streaming finalizer never set
`provider_input_tokens` for Anthropic (#3946 fixed only the uncached
sub-count), so every streamed Claude turn fell back to
`optimized_tokens`. That value is:
- an `o200k_base` count standing in for Claude's private tokenizer,
measured against `count_tokens` on 6 real Claude Code transcripts at
15–25% under Claude 4.x and 35–38% under Claude 5.x;
- a count of the forwarded **messages only**, which never sees the
system prompt or tool definitions.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/proxy/provider_usage.py` (new).** One place that knows
each provider's usage semantics:
- Anthropic, Bedrock and Vertex: `input_tokens` is the uncached tail,
and the billed total is `input + cache_read + cache_write`.
- OpenAI `prompt_tokens`/`input_tokens` and Gemini `promptTokenCount`
are already inclusive of cached tokens.
- **Streaming finalizer.** Sets `provider_input_tokens` from the
provider's usage on the Anthropic/OpenAI/Gemini SSE path, the Bedrock
stream path and the OpenAI-via-backend stream path.
`RequestOutcome.from_stream` gains the parameter.
- **OpenAI pass-through.** On both streaming and non-streaming paths,
Anthropic-shape usage was read as if `input_tokens` were the total. It
now uses the billed total and is marked provider-reported.
- **Gemini non-streaming and OpenAI Responses WebSocket.** These already
used the provider's count as `optimized_tokens` but never declared it,
so they now also set `provider_input_tokens`, but only when the provider
actually sent a count.
- **No estimate is ever labelled as billed.** The Bedrock #1132
`message_start` backfill and Gemini's fallback for a missing
`promptTokenCount` both stay estimated. The pass-through usage dialect
is chosen by provider, not inferred from which keys are present.
- **Provenance counters.** Every input total now carries how much of it
is billed versus estimated:
- `/stats` `tokens.input_provider_reported`, `input_estimated`,
`input_provider_reported_percent`, `requests_input_*`.
- Licence usage report: `tokens_after_provider_reported`,
`tokens_after_estimated`, `requests_provider_reported`, plus the billed
`cache_read_tokens`, `cache_write_tokens` and `uncached_input_tokens`,
so the cloud side can reconcile with the console (input + cache read +
cache write). The report also states `tokens_saved_basis:
"headroom_tokenizer"`.
- A one-time `event=input_tokens_estimated` warning per provider/model
when a successful request carried no provider usage.
- **No delta math changes.** `optimized_tokens` stays on the local
scale, and `tokens_saved`, ratios and the ledger delta are untouched.
The provider count is never subtracted from a local count.

## Testing

- [ ] Unit tests pass (`pytest`): targeted suites only, listed below.
The full suite is left to CI (see Additional Notes).
- [x] Linting passes (`ruff check .`): run on all changed files.
- [x] Type checking passes (`mypy headroom`): run on the changed
modules.
- [x] New tests added for new functionality
- [x] Manual testing performed

`tests/test_provider_billed_input.py` adds 15 tests, plus regression
tests for the review fixes in
`tests/test_bedrock_streaming_input_tokens.py` (the #1132 backfill is
reported as estimated) and `tests/test_gemini_ccr_continuation_usage.py`
(a missing `promptTokenCount` is not provider-reported):
- **Recorded fixture.**
`tests/fixtures/anthropic/stream_cached_turn.sse` is a **real recorded
Anthropic stream**: a second call on a cached ~8.9K-token system prompt,
billed as 19 uncached + 8,857 cache-read. It is replayed in 97-byte
chunks through the proxy's own SSE usage parser and
`_finalize_stream_response`, then through `emit_request_outcome` into a
real `PrometheusMetrics`, `CostTracker` and `UsageReporter`. The test
asserts that `/stats` input and the licence `tokens_after` equal
Anthropic's 8,876.
- Other cases:
  - an OpenAI stream records the inclusive `prompt_tokens`;
  - a stream without usage stays estimated;
  - each provider shape is covered;
  - the pass-through normaliser returns the total.

Against the unpatched handlers, the 5 behaviour tests fail. The 8 that
pass exercise only the new helper.

### Test Output

```text
$ pytest tests/test_provider_billed_input.py tests/test_proxy_handler_helpers.py tests/test_request_outcome.py \
    tests/test_proxy_streaming_request_logger.py tests/test_proxy_cache_ttl_metrics.py \
    tests/test_usage_reporter_snapshot.py tests/test_5xx_accounting_all_providers.py tests/test_copilot_provider_label.py \
    tests/test_bedrock_streaming_input_tokens.py tests/test_gemini_ccr_continuation_usage.py -q
181 passed in 6.56s

# same new tests against the unpatched handlers
FAILED tests/test_provider_billed_input.py::test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail
FAILED tests/test_provider_billed_input.py::test_recorded_anthropic_stream_records_anthropics_billed_input
FAILED tests/test_provider_billed_input.py::test_openai_stream_records_inclusive_prompt_tokens
FAILED tests/test_provider_billed_input.py::test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed
FAILED tests/test_provider_billed_input.py::test_estimated_input_is_reported_as_estimated
5 failed, 8 passed

$ ruff check <changed files>        -> All checks passed!
$ mypy <changed proxy/telemetry modules> -> Success: no issues found in 6 source files
```

## Real Behavior Proof

- Environment: local proxy from this branch (`headroom proxy --port 8931
--stateless --no-telemetry --no-cache`) against the real Anthropic API.
For the baseline, the same command ran from a `main` worktree on port
8932.
- Exact command / steps: 3 streamed `claude-sonnet-4-6` turns through
the proxy, with a cached ~20K-token system prompt, one tool and a
300-item JSON tool result. Anthropic's `message_start` usage (input +
cache read + cache write) was summed from each stream and compared with
`GET /stats` → `tokens`.
- Observed result: the author-provided live Anthropic comparison below
reports exact billed-input agreement at 61,999 tokens; the baseline
undercounts by 37%. This external run was not independently repeated
during the current-head review.

  | | Anthropic billed input | Headroom `/stats` `tokens.input` |
  |---|---|---|
  | `main` | 62,001 | **38,835** (−37%) |
| this PR | 61,999 | **61,999** (exact),
`input_provider_reported_percent: 100.0` |

The two runs differ by 2 tokens because they are separate API calls.
`proxy_compression_saved` was 29,313 on both, so savings behaviour is
unchanged.
- Not tested: live Bedrock, Vertex and Gemini streams (unit-tested on
provider shapes only), and the licence cloud endpoint receiving the new
fields.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is accounting only, and request
and response bytes are unchanged.
- Minimum rollout channel: N/A
- Stable/default behavior changed: reported input volume now uses the
provider's billed count where the provider reports one. New fields are
added to `/stats` and the licence usage payload; no existing field is
removed or renamed.
- Kill switch / disable path: none needed. With no usage frame, the
fallback is the previous estimate, now labelled `input_estimated`.
- Unsafe override required: no
- Qualification impact: dashboards and licence reports will show larger,
correct Claude input totals once installs upgrade, so historical and new
figures are not directly comparable.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes: targeted
suites only; full suite on CI
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- **Full suite.** Locally, the full run on the pre-rebase base reached
2,147 passed before stopping on
`tests/test_cli/test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`.
That test fails identically on unmodified `main` (unrelated MCP install
args). The full local run was then abandoned because several
wrap/install tests launch a real `serena project index` (rust-analyzer,
tsserver, rustc), which should be stubbed in a separate PR.
- **Docs.** The new `/stats` and licence fields are not yet documented.
- Follow-ups not in this PR:
- **Raw Bedrock invoke pass-through (`handlers/bedrock.py`)** forwards
bytes without parsing them, so it still records an estimate. It is now
counted as `input_estimated`, so it's visible rather than silent.
- **Batch create** has no usage until results come back, so it stays
estimated as above.
- **`tokens_saved` is still on the `o200k_base` scale**, so it
under-reports Claude savings. Calibrating it to provider tokens is a
separate PR.
  - **The licence cloud endpoint** needs to read the new fields.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Current maintainer verification (October 8)

Exact head: `81527148ed9777e1b9ca921bbf6bb457d49354db`, containing main
`01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a clean merge. No
contributor logic was modified. 63 focused tests pass with the existing
compatible native extension on Windows/Python 3.12. Three
backend-boundary probes preserve zero client input and exact cache-read,
cache-write, and combined provider input totals. This is local
mocked-backend/TestClient evidence; the author-provided
external-provider evidence above was not independently rerun.
Current-head hosted snapshot: 31 successful checks, 12 skips, no failed
or live checks.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-09 14:16:49 -07:00
f78e66f7cc fix(proxy): stop headroom logger from suppressing propagation to stdout/stderr (#3096)
## Description

`_setup_file_logging()` in `headroom/proxy/helpers.py` unconditionally
set `headroom_logger.propagate = False`, on the theory that `wrap.py`
redirects stderr into the same `proxy.log` file the rotating handler
writes to, so propagation would double-write records. That premise is
wrong: `wrap.py` redirects the proxy subprocess's stdout/stderr to a
separate sibling file, `proxy-stdio.log` (see
`_get_proxy_stdio_log_path()`), never `proxy.log`. Severing propagation
bought no deduplication — it only stopped every `headroom.*` log record
from reaching root's `logging.basicConfig()` handler
(`proxy/server.py`), which is what writes to stderr/stdout. In
containerized deployments (Docker/Kubernetes), stdout/stderr is the only
channel the platform collects, so this silently dropped all application
logs.

Closes #3087

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed `headroom_logger.propagate = False` from
`_setup_file_logging()` in `headroom/proxy/helpers.py`.
- Updated the surrounding comment to explain why propagation is safe to
leave on: `proxy.log` (file handler) and stderr/stdout (root's
`basicConfig` handler) are two distinct destinations, so nothing is
duplicated in either file.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check .
All checks passed!

$ pytest tests/test_corrupt_golden_bytes_recovery.py -q
collected 9 items
tests/test_corrupt_golden_bytes_recovery.py .........                    [100%]
9 passed in 0.47s
```

`mypy headroom` was not run to a clean result: it fails before reaching
this file with a pre-existing, unrelated error in the installed `numpy`
stub (`Type statement is only supported in Python 3.12 and greater`) in
this sandbox's dependency set. Not caused by this change (a 4-line
pure-Python logging tweak).

The broader `pytest tests/` collection also hits ~170 pre-existing,
unrelated errors in this sandbox: `headroom._core`, the compiled Rust
extension, isn't built here (no `cargo`/`maturin` available), so every
module importing it fails at collection. This change doesn't touch Rust
code or `_core` and isn't implicated.

No new tests added: this restores an existing, already-tested code path
(see `tests/conftest.py`'s autouse fixture and
`tests/test_corrupt_golden_bytes_recovery.py`, both of which already
assert/require `propagate=True` for `caplog` to work) to its intended
default rather than adding new behavior.

## Real Behavior Proof

- Environment: local sandbox, Python 3.12.3, unable to build/import
`headroom.proxy.server` directly (missing compiled `headroom._core`).
Verified with an isolated script that copies `_setup_file_logging()`'s
exact logic byte-for-byte plus the same `logging.basicConfig()` call
`proxy/server.py` makes at import time.
- Exact command / steps: ran the fixed logic, then emitted
`logging.getLogger("headroom.proxy").info("compression timeout: request
abc123 exceeded 30s budget")`.
- Observed result: the `headroom.proxy` record was written to stderr AND
to `proxy.log` simultaneously (full output below); re-running the same
script with the old `headroom_logger.propagate = False` line restored
reproduces the bug — the record disappears from stderr entirely and only
lands in `proxy.log`.
  ```text
  propagate flag on 'headroom' logger: True
--- emitting a headroom.proxy log record, watch for it below on stderr
---
2026-08-17 21:17:39,545 - headroom.proxy - INFO - compression timeout:
request abc123 exceeded 30s budget
  --- also present in proxy.log (file handler): OK ---
  ```
- Not tested: end-to-end through the actual compiled proxy binary/server
process (blocked by missing `headroom._core` in this sandbox) and real
container log collection (Docker/Kubernetes) — the isolated repro
exercises the exact same logging configuration code the real process
runs, so this is considered sufficient proxy proof for a pure
logging-config change.

## Runtime Rollout Safety

- Rollout-managed feature(s): None — this is unconditional startup
logging configuration, not behind a rollout flag.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes — `headroom.*` log records now
also reach stderr/stdout (previously file-only). No change to
request/response handling, compression, or any user-facing proxy
behavior.
- Kill switch / disable path: None needed; this restores default Python
logging propagation. If a deployment truly wants file-only logging,
`HEADROOM_LOG_TO_STDOUT`-style suppression could be added later, but no
such request exists today.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this commit to restore `propagate = False`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — logging configuration change, no UI impact.

## Additional Notes

- "New and existing unit tests pass locally" is left unchecked because
the full `pytest tests/` collection can't complete in this sandbox
(missing compiled `headroom._core` Rust extension, unrelated to this
change) — see Test Output above for what was actually run and passed.
- "I have added tests" is left unchecked: existing tests
(`tests/conftest.py` autouse fixture,
`tests/test_corrupt_golden_bytes_recovery.py`) already assert the code
path this restores; happy to add a focused regression test asserting
`logging.getLogger("headroom").propagate is True` after
`_setup_file_logging()` if maintainers want one.

---------

Co-authored-by: = <=>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 00:03:53 -05:00
81fe9d5345 fix(metrics): attribute tool-schema savings per model, not just compression (#3155)
## Description

Reported against 0.36.0 (VS Code + Copilot + Claude Code): the per-model
breakdown disagreed with the headline printed four lines above it.

```
Tokens saved: 625,277
  · messages       36,071
  · tool schemas  589,206
Per-Model Breakdown
  <a>: 35,907 tokens saved
  <b>:      0 tokens saved
  <c>:    164 tokens saved
  <d>:      0 tokens saved
```

The rows sum to **36,071** — the *messages* line exactly. All 589,206
tokens of tool-schema deferral, 94% of the headline, had no row to land
in, so every tool-heavy model reported "0 tokens saved" while real
dollars were credited to it.

Deferral is disjoint from message compression by construction: deferred
schemas never enter the message token counts, so they move neither
`tokens_saved` nor `tokens_sent`. The headline, the PERF line, and the
savings ledger (#2795) all already fold the two together. Three
per-model surfaces did not.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`perf/analyzer.py`** — the per-model loop summed `tokens_saved`
while its own headline summed `tokens_saved + tool_saved`. Now uses the
same all-layers construction (`headline_before = before + tool_saved`),
and prints a `· messages / · tool schemas` split line only when there is
a split to show.
- **`proxy/savings_tracker.py`** — added a `tool_tokens_saved` bucket to
`_empty_by_model_entry()`, normalization, and
`_record_by_model_locked()`; `record_request()` gained a
`tool_search_saved` parameter. `_by_model_snapshot_locked()` ranks and
computes `savings_percent` off the combined figure and exposes
`headline_tokens_saved`.
- **`proxy/prometheus_metrics.py`** — **the seam.** `record_request`
already accepted `tool_search_saved` and already folded it into the
per-model *dollars*, but never passed it to
`savings_tracker.record_request`. Tokens and money therefore disagreed
on the same row.
- **`proxy/cost.py`** (feeds the dashboard's "Per-Model Token Savings"
table) — added `_tool_saved_by_model`, a `tool_schema_saved` kwarg, and
`compression_tokens_saved` / `tool_tokens_saved` alongside a combined
`tokens_saved`. The `stats()` loop now iterates the **union** of both
dicts: keying off compression alone dropped a deferral-only model from
the table entirely rather than merely under-reporting it.
- **`proxy/outcome.py`** — forwards the figure it already computed for
`metrics.record_request` to `cost_tracker.record_tokens`.
- **`dashboard.html`** — the "Tokens Saved" cell gains a `title` showing
the compression/deferral split.

Design notes:
- Components stay separately addressable rather than widening an
existing field's meaning in place, so persisted state remains readable
by older readers.
- Percentages use the all-layers numerator over `saved + sent` —
deferred schemas were never in `sent`, so that is still the pre-Headroom
volume.
- `CostTracker.stats()["savings_usd"]` is deliberately **not** widened:
deferral is already priced by `SavingsTracker`, and this tracker's
dollars feed budget enforcement, where counting it twice would
double-book the saving.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_per_model_tool_savings.py -q
11 passed in 0.94s

# Same file against pre-fix code (git stash), proving the tests bite:
5 failed, 1 passed
  FAILED test_per_model_rows_reconcile_with_the_headline
  FAILED test_a_tool_only_model_no_longer_reads_zero
  FAILED test_tracker_attributes_deferral_to_the_model
  FAILED test_tracker_default_is_unchanged_without_deferral
  FAILED test_state_written_before_this_field_existed_still_loads
(the one that passes pre-fix is the "compression-only model is unchanged" guard)

$ pytest tests/ -q          # this branch
3 failed, 11374 passed, 587 skipped in 343.55s

$ pytest tests/ -q          # clean origin/main, same machine
3 failed, 11364 passed, 587 skipped in 352.64s

Identical 3 failures on both — pre-existing and environmental, not regressions:
  test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline
  test_release_workflows.py::test_no_native_tls_in_wheel_build_tree   (FileNotFoundError: 'cargo')
  test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter
    (whole-suite ordering flake; tests/test_graceful_shutdown.py passes 11/11 in isolation on this branch)

$ ruff check headroom/
All checks passed!

$ mypy headroom/proxy/cost.py headroom/proxy/savings_tracker.py \
       headroom/proxy/prometheus_metrics.py headroom/proxy/outcome.py \
       headroom/perf/analyzer.py
Success: no issues found in 5 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, this branch rebased on
`origin/main` @ `1f96dabc`.
- Exact command / steps: reproduced the reported shape as a unit test —
three models with 35,907 / 0 / 164 message savings and 400,000 / 189,206
/ 0 deferral, then rendered `format_report`.
- Observed result: headline `Tokens saved: 625,277` unchanged; rows now
read `435,907` / `189,206` / `164` and sum to the headline. The seam
test drives the real `PrometheusMetrics.record_request` and asserts the
tracker's `by_model` entry ends up at `tokens_saved=400,
tool_tokens_saved=54,000, headline_tokens_saved=54,400`.
- Not tested: no live proxy run against a real Copilot/Claude Code
session; the arithmetic is pinned at the four code seams instead. The
dashboard `title` tooltip is markup-only and not covered by a rendering
test.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this is reporting arithmetic, not a
request-path behavior.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, displayed per-model token
savings and percentages increase to include tool-schema deferral. No
request is treated differently.
- Kill switch / disable path: n/a. Components remain separately readable
(`compression_tokens_saved` / `tool_tokens_saved`) if a consumer wants
the old message-only figure.
- Unsafe override required: none.
- Qualification impact: none — `savings_usd` and budget enforcement are
unchanged by design.
- Rollback path: revert the commit; `tool_tokens_saved` in persisted
state is then simply ignored by the older reader.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 11:54:02 -07:00
Suliman Abdulrazzaq 620028fa18 fix(proxy): emit request log timestamps in UTC
## Description

`RequestLog.timestamp` was serialized with `datetime.now().isoformat()`,
which omits timezone information. Browsers then interpret the value as
local time, so requests from a UTC container can display negative ages
in non-UTC dashboards.

Closes #2910

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Emit request-log timestamps from `datetime.now(timezone.utc)` so the
ISO-8601 value includes `+00:00`.
- Add a regression test that parses the emitted timestamp and requires a
UTC offset.

## Testing

- [x] New tests added for the regression
- [x] `python -m compileall -q headroom/proxy/outcome.py
tests/test_request_outcome.py`
- [x] `git diff --check`
- [ ] Unit tests pass (`pytest`) — the repository's Rust extension
cannot build in this Windows environment because `link.exe` (MSVC) is
unavailable; the focused test is included for CI.

### Test Output

```text
python -m compileall -q headroom/proxy/outcome.py tests/test_request_outcome.py
(pass)

git diff --check
(pass)

uv run pytest tests/test_request_outcome.py -q
blocked while building headroom-py: linker `link.exe` not found
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.11; proxy timestamps are generated
in `headroom/proxy/outcome.py`.
- Exact command / steps: traced the Recent Requests write path and added
a timestamp assertion in `tests/test_request_outcome.py` (CI will run
with the project's Rust toolchain).
- Observed result: the production call now emits an ISO-8601 timestamp
with `+00:00`; the regression assertion requires an offset-aware UTC
value, preventing browser timezone skew.
- Not tested: full pytest suite locally because the MSVC linker is
unavailable.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review
- [x] I have added tests that prove my fix is effective
- [x] I did not edit `CHANGELOG.md`

Signed-off-by: Suliman Abdulrazzaq <suliman9000a@gmail.com>
2026-08-11 09:53:53 -07:00
Tejas Chopra a033ac4176 fix(cost): send litellm the total prompt so --budget stops seeing $0 (#2757)
## Description

**`--budget` has been silently inert on any cache-warm request** — which
is the normal case in an agent session. This is a disabled control, not
a metrics bug.

`record_tokens` passed only the **uncached slice** as litellm's
`prompt_tokens`. Measured, `litellm.cost_per_token` charges:

```
  (prompt_tokens - cache_read - cache_creation) * input_rate
+ cache_read     * read_rate
+ cache_creation * write_rate
```

So `prompt_tokens` is the **whole** prompt and litellm removes the
cached parts itself. Handing it the uncached slice drives the input term
**negative** as soon as anything is cached. `estimate_cost` ends with
`float(total) if total > 0 else None`, so it returned `None`, no
`CostEntry` was appended, and `check_budget()` saw **$0**.

| model | 100k prompt, 80k cached — old call | booked |
| --- | --- | --- |
| `gpt-5` | **-$0.065000** | None |
| `gpt-4o-mini` | **-$0.003000** | None |
| `claude-sonnet-4-5` | **-$0.156000** | None |

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

Fixing the total alone would **over-charge OpenAI**, because two bugs
are entangled here.

OpenAI exposes no cache-write counter, so
`_infer_openai_cache_write_tokens` uses the uncached portion as a write
proxy. At two of the four inference sites (`openai.py:4304`,
`openai.py:5413`) `uncached_input_tokens` is derived by subtracting
**only** `cache_read`, so `cache_write_tokens` and `uncached_tokens` are
**the same tokens**. Summing all three would double-count the prompt and
charge a write premium OpenAI does not have. (The other two sites —
`openai.py:3969` and `streaming.py:2024` — subtract both, so their
buckets are genuinely disjoint; those are left alone.)

- `cost.py` — `record_tokens` now passes `uncached + cache_read +
cache_write` as the prompt total.
- `cost.py` — new `cache_inferred: bool = False` parameter. When set,
the inferred write is excluded from **both** the prompt total and the
write premium. The default preserves behaviour for every provider that
reports disjoint buckets.
- `outcome.py` — plumbs `outcome.cache_inferred` through. The field
already existed on `RequestOutcome` for the dashboard; it just never
reached cost.
- `handlers/openai.py` — sets `cache_inferred=True` at the two outcome
sites whose buckets genuinely duplicate.

## Testing

- [x] Unit tests pass (new file)
- [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17)
- [x] New tests added
- [x] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/proxy/handlers/openai.py
All checks passed!
$ uvx ruff@0.15.17 format --check <same three>
3 files already formatted

$ pytest tests/test_cost_budget_total_prompt.py -q
5 passed in 0.26s
```

The 5 new tests assert on the **arguments handed to `estimate_cost`**
rather than on dollar values, so they pin the contract that broke
without depending on litellm's pricing tables — or on litellm being
installed at all.

## Real Behavior Proof

- **Environment:** macOS 26.4 arm64, isolated git worktree off
`upstream/main`.

**litellm's actual formula**, established by probe rather than assumed:

```text
rates (claude-sonnet-4-5): input=3e-06 read=3e-07 write=3.75e-06
total=100k, 80k read, 5k write  ->  $0.087750
  hypothesis (p-r)*ir   + r*rr + w*wr = $0.102750   x
  hypothesis (p-r-w)*ir + r*rr + w*wr = $0.087750   <- matches
```

**Before → after:**

```text
Anthropic, disjoint: uncached=900 read=48000 write=1500
  OLD prompt=900     raw=-0.125775  booked=None   <-- BUDGET BLIND
  NEW prompt=50400   raw= 0.022725  booked=0.022725

OpenAI, inferred write == uncached: uncached=20000 read=80000 write=20000
  OLD prompt=20000                  raw=-0.090000  booked=None   <-- BUDGET BLIND
  NEW prompt=100000, write excluded raw= 0.035000  booked=0.035   truth=0.035000
```

The OpenAI "after" equals the hand-computed truth `20,000*input +
80,000*read_rate` exactly.

- **Not fully tested locally.** The cost/outcome suites show an
**identical 10-failure set** on this branch and on clean `upstream/main`
in the same throwaway env — all `ModuleNotFoundError: headroom._core`,
the compiled Rust extension this machine cannot currently build. So they
are environment-only, not regressions. One of them,
`test_funnel_passes_canonical_record_tokens_shape`, covers the
`record_tokens` call shape this PR changes, so **CI is the authoritative
check for that one**.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] I did **not** edit `CHANGELOG.md`

## Context

Found during a tokenizer-consistency audit that also turned up:
HuggingFace-routed models counting a 6,000-char message as **2 tokens**,
`gpt-5`/`o4-mini` falling to a char estimator, and the Kompress size
gate missing its own cap by 24%. Those are separate PRs — different
subsystems, different risk.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-08-03 22:41:10 -07:00
nangsontay 184146b688 fix(savings): surface request growth the tok_saved clamp swallows (#2708)
## Description

`tokens_saved` is clamped at zero, so a request the proxy forwards
**larger** than it arrived is indistinguishable in the PERF line from
one it simply could not compress. Both read `tok_saved=0`.

That ambiguity hides real regressions. Anything that appends to the body
after compression — proactive context expansion, memory injection — can
outweigh the compression it sits on top of and still look like a neutral
turn. On the session that prompted this, a request went from 55,161
tokens in to 57,845 out and reported `tok_saved=0`, for 19 consecutive
turns, with nothing in the logs distinguishing it from a turn with
nothing left to compress.

This reports the swallowed amount as `tok_inflated`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `RequestOutcome.tokens_inflated`: `max(0, optimized_tokens -
original_tokens)`, derived from two counts the outcome already carries —
**no new plumbing at any of the emit sites**.
- Added `tok_inflated=` to the PERF log line, next to `tok_saved=`.

Diagnostic only, deliberately. It does **not** feed `tokens_saved` or
`attempted_input_tokens`, for two reasons:

1. `attempted_input_tokens = optimized_tokens + tokens_saved` is a
*size*, not a signed delta. Letting the second term go negative makes it
smaller than the bytes actually forwarded, corrupting the active-savings
denominator.
2. Injection paths already book their own cost through the
retrieval-drawback channel. A negative landing in `tokens_saved` as well
would count the same loss twice.

So the clamp stays and the hidden number surfaces beside it. Worth
noting there is already a revert-on-inflation guard *before*
compression's own inflation can escape (`anthropic.py`, "Optimization
inflated tokens … reverting to original messages") — it is only growth
added *after* that point which the clamp was silently absorbing.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_request_outcome.py tests/test_cli_perf_format.py -q
============================== 54 passed in 1.56s ==============================

$ pytest tests/ -q -k "outcome or perf or savings or stats"
======= 510 passed, 33 skipped, 9674 deselected, 542 warnings in 40.37s ========

$ ruff check headroom/proxy/outcome.py tests/test_request_outcome.py
All checks passed!

$ ruff format --check headroom/proxy/outcome.py tests/test_request_outcome.py
2 files already formatted

$ mypy headroom/proxy/outcome.py --ignore-missing-imports
Success: no issues found in 1 source file
```

Four new tests pin the distinction that was missing: shrank (0), no-op
compression (0, and `tok_saved` also 0 — the two cases that used to look
identical), grew (reports 2684 while `tok_saved` stays 0), and that
`attempted_input_tokens` / `savings_pct` keep their unsigned semantics.

`tests/test_cli_perf_format.py` parses hand-written PERF fixtures by
field name, so adding a field does not disturb it — verified green
above.

## Real Behavior Proof

### The field catching a real inflating request

- Environment: macOS 15 (arm64), Python 3.13.14. A proxy booted from
this branch: `headroom proxy --mode token --backend anthropic
--anthropic-api-url http://127.0.0.1:<stub>`, isolated `HOME` so the run
could not touch a developer's live logs/store, `HF_HOME` pointed at
cached kompress weights so the lossy+CCR-marker path is exercised and
proactive expansion can actually arm.
- Exact command / steps: three requests over one conversation through
the real HTTP path (`x-headroom-cwd: /tmp/proof`, `user-agent:
claude-code/1.4.2`): a user turn carrying the real
`~/.claude/rules/*.md` text (~8.2k tokens); then `assistant` + a short
user turn so that block becomes compressible and gets tracked as a CCR
entry; then a follow-up whose leading text block shares vocabulary with
it, so proactive expansion fires and appends the original — which is how
a request ends up leaving larger than it arrived. PERF lines read from
the isolated `~/.headroom/logs/proxy.log`.
- Observed result: real PERF output from that run —

  ```text
msgs=1 tok_before=8170 tok_after=9553 tok_saved=0 tok_inflated=1383 ...
transforms=router:text_block:mixed
msgs=3 tok_before=8184 tok_after=9567 tok_saved=0 tok_inflated=1383 ...
transforms=router:text_block:mixed
msgs=5 tok_before=8245 tok_after=11880 tok_saved=0 tok_inflated=3635 ...
transforms=router:text_block:mixed
  ```

Correlated from the same run: `CCR Tracker: Proactively expanded
f0cf4efb42373ec225f57725 (1417 items)`, and the stub upstream confirms
the block reached the wire (`has_expansion_block: true`, forwarded body
54,130 B on the third request).

Every one of those turns reports `tok_saved=0`. Before this change that
is all the log said, and it is the same thing it says when there was
simply nothing left to compress. `tok_inflated=3635` is the number that
was missing.

For contrast, the same scenario run against a build where the request
genuinely shrinks reported `tok_before=8245 tok_after=7359
tok_saved=886` — that build predates this field, so it does not print
`tok_inflated`; the point is only that the inflating and shrinking cases
are the two states the field has to separate, and on `main` today both
render as `tok_saved=0` whenever the growth path is taken. The
`tok_inflated=0` case on a shrinking request is covered by unit test.

### Scale of what was hidden

From a live `--mode token` proxy on Claude Code traffic across four
rotated logs: **305 of 3,743 requests (8%)** had `tok_after >
tok_before` while every one reported `tok_saved=0` — 192,829 tokens of
growth rendered as "nothing to compress". The worst single session held
+2,643/turn for 19 consecutive turns.

- Not tested: the `headroom perf` CLI was not run against a real log
file containing the new field — it parses by field name and
`tests/test_cli_perf_format.py` is green, but that is test-level rather
than end-to-end evidence. Streaming responses were not exercised (the
stub replies non-streaming), so the streaming emit path carries the new
field on the strength of sharing `emit_request_outcome` rather than by
observation. No dashboard or Prometheus consumer was re-run.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — one added field on an existing log line; no user-facing surface.

## Additional Notes

- Independent of #2706 and #2707 — verified `conflicts=0` via `git
merge-tree`; mergeable in any order.
- Related but deliberately out of scope: `main` has no producer for
retrieval-cost accounting (`record_savings_event` takes no
`kind`/`tokens_retrieved`, and nothing writes `tokens_retrieved`
anywhere), so proactive expansion's cost is not booked into net savings
at all. Adding that channel is a cross-cutting accounting change and
belongs in its own PR; this one only makes the growth visible in the
log.
2026-08-03 20:18:01 -07:00
inix 4aac068814 fix(proxy/metrics): move the savings-ledger append off the event loop (#2439)
## Description

`PrometheusMetrics.record_request` appends one durable JSONL event per
compressed request. That append is synchronous: `open` + `fcntl.flock` +
`write`, plus a full-file rewrite once the ledger passes 1 MB. It runs
on the event loop, inside `self._lock`.

`export()` takes that same lock and holds it for the entire Prometheus
serialization, so a slow ledger write stops `/metrics` cold. In a repro
run of 200 compressed requests, `/metrics` completed zero scrapes and
the event loop never yielded once across 6.4 seconds.

The append now runs in a thread, outside the lock. `savings_ledger`
already takes its own `flock` across processes, so the metrics lock was
never what made the write safe.

Both halves are one change. Awaiting inside the lock would hold it for
the whole write rather than just the syscall, which is worse than what
is on main today.

The file already documents this hazard against itself.
`record_stage_timings` (`prometheus_metrics.py:867-874`) picks a plain
`threading.Lock` over `self._lock` specifically because "the async lock
is also held by `export()` during Prometheus scrapes." The ledger append
was the pattern that docstring warns about.

No filed issue for this one.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Move the `savings_ledger.record_savings_event` call in
`record_request` out of `async with self._lock` and run it through
`asyncio.to_thread`. The call site keeps its keyword arguments verbatim;
`to_thread` forwards `**kwargs`, so no `functools.partial` wrapper is
needed.
- Keep the `await`. Callers still see the event on disk when
`record_request` returns, which
`tests/test_savings_ledger_before_forwarded.py` asserts synchronously.
- Add `tests/test_savings_ledger_offload.py`: lock scope, event-loop
responsiveness, durability on return, and both arms of the `tokens_saved
> 0 and not stateless` gate.

`savings_ledger.py` is untouched. It stays synchronous so the MCP
`headroom_compress` caller in `ccr/mcp_server.py:789` does not have to
change.

Sizing the executor is left alone on purpose. `asyncio.to_thread` uses
the default pool, which is the documented tool for blocking I/O and
already the idiom here (`helpers.py:1297`, `server.py:1694`, `:3557`,
`:3613`, `:4244`). The compression pools are sized `max(1,
os.cpu_count())` for CPU-bound work, and `PrometheusMetrics` holds no
reference to `HeadroomProxy` anyway, so reaching them would mean a new
constructor parameter.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_savings_ledger_offload.py tests/test_savings_ledger.py tests/test_savings_ledger_before_forwarded.py -q
======================== 26 passed, 1 warning in 4.08s =========================

$ ruff check . && ruff format --check headroom/proxy/prometheus_metrics.py tests/test_savings_ledger_offload.py
All checks passed!
2 files already formatted

$ mypy headroom/proxy/prometheus_metrics.py
Success: no issues found in 1 source file
```

Broader sweep across the blast radius, 145 test files matching savings /
metrics / outcome / stats / proxy / handler / server / ledger / cost /
prometheus, each run under a per-file wall-clock watchdog:

```text
138 files pass, 1470 tests passed
7 non-green:
  HANG tests/test_agent_savings.py
  HANG tests/test_ccr_mcp_server.py
  HANG tests/test_netcost_gate.py
  HANG tests/test_proxy_compress_endpoint.py
  HANG tests/test_proxy_mode_benchmark.py
  HANG tests/test_read_maturation_handler_nobust.py
  FAIL tests/test_proxy_copilot_auth_hooks.py::test_openai_passthrough_applies_copilot_auth

Same 7 files re-run with headroom/proxy/prometheus_metrics.py reverted to c400f908:
  identical set, identical failure. diff of the two non-green lists is empty.
```

The before and after non-green sets match exactly, so nothing here is a
regression from this PR. See Additional Notes for the hang.

## Real Behavior Proof

- Environment: macOS 15.4 (Darwin 25.4.0) arm64, Python 3.13.13,
uv-managed venv, git worktree at `upstream/main` `c400f908`.
- Exact command / steps: a standalone asyncio script, not the unit
tests. It builds a real `PrometheusMetrics` (no injected tracker, so it
self-constructs with `save_flush_every=PROXY_SAVINGS_FLUSH_EVERY`
exactly as the proxy does) against a real on-disk ledger pre-seeded to
3.00 MB so `_maybe_compact`'s full-file rewrite actually fires. It then
drives 200 `record_request` calls at concurrency 16 while a `/metrics`
scraper calls `export()` every 20 ms and a canary coroutine ticks every
5 ms. Ran twice from the same script: once with
`headroom/proxy/prometheus_metrics.py` reverted to `c400f908`, once with
this change. Seeding the ledger past 1 MB is the part that matters. On a
fresh ledger the write is microseconds, compaction never fires, and the
run shows no delta at all.
- Observed result: before, `/metrics` completed 0 scrapes and the canary
ticked once in 6419 ms. After, 206 scrapes at p50 0.1 ms and max 0.2 ms,
and 418 canary ticks with a 92.3 ms worst gap. Total wall clock barely
moved, 6419 ms to 6473 ms, which is the expected result and not a null
one: the same disk work still serializes on the ledger's own `flock`,
now in a thread instead of on the loop. Unit-test view of the same
behavior, with a 500 ms stub standing in for the write: before, `event
loop stalled 0.506s during a 0.500s ledger write` and the competing lock
holder waited `+0.502s`; after, both pass.
- Not tested: Windows, where `savings_ledger` already skips locking
because `fcntl` is unavailable. Multi-process contention on one ledger
file, which this change does not alter. The residual 92.3 ms loop gap
after the fix, which traces to `SavingsTracker._save_locked`'s
`os.fsync` (`savings_tracker.py:1445`) firing every 25th request from
inside the same lock, a separate path this PR leaves alone.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, no user-visible surface.

## Additional Notes

Docs checklist item is N/A. Nothing user-facing moves; `headroom
savings` reads the same ledger, with the same contents, written from a
thread.

Two related in-lock costs on this same code path are deliberately out of
scope, one logical change per commit:
`_current_savings_tracker_totals()` at `:798` rebuilds
`cost_tracker.stats()` per request, and `_resolve_litellm_model` in
`savings_tracker.py` is uncached across roughly seven calls per request.
Happy to follow up on either.

`make ci-precheck` was not run end to end. The uv-managed worktree venv
has no `pip`, so the `ci-precheck-python` hook's `pip install -e .` step
fails on this machine for reasons unrelated to the change. Ran `pytest`,
`ruff`, and `mypy` directly instead, output above. No Rust touched.

One heads up worth passing on, since it is why the numbers above are a
sweep and not a single full-suite line. Seven test files do not complete
on this macOS box: six hang and one fails. The hangs park the main
thread in `_dispatch_semaphore_wait_slow` with CPU time frozen and never
recover, and `pytest-timeout --timeout-method=signal` cannot break them
out, so the block is native, below the interpreter.
`tests/test_adversarial_grid.py::test_grid_shape_and_schema` is the
first one a full run reaches.

All seven reproduce identically on unmodified `c400f908` with this
change reverted, so they predate the PR. I ran the reverted comparison
specifically to rule out a thread-before-fork interaction from the new
`to_thread` call, which was the plausible way this change could have
caused it. It did not. Happy to open a separate issue with the sample
output if that is useful.


---

## Follow-up: a cancellation bug this change introduced

Self-review turned up a second problem in this change, so the fix rides
along here.

Moving the append into a thread added the first suspension point in
`record_request` that can tear state. The metrics lock at
`prometheus_metrics.py:701` suspends too, but only under contention, and
it sits ahead of every mutation, so a cancellation there recorded
nothing at all. The new await is different. It sits after the Prometheus
counters commit and before OTel and the funnel's effects 2/3/4, and it
suspends on every compressed request.

Four of the funnel's call sites are `finally:` blocks inside streaming
async generators (`streaming.py:1611`, `:1859`, `:2069`,
`openai.py:8614`). A client disconnect cancels that task. The
cancellation lands on the new await, so Prometheus counts the request
while the cost tracker, the request log, and the PERF line `headroom
perf` reads never see it. `emit_request_outcome` has one try/except and
it sits before `record_request`, so nothing catches this.

`_record_request_outcome` now wraps the funnel in `asyncio.shield`. One
line at a single choke point, covering all 28 call sites. The shield
leaves the cancellation itself alone: the await still raises
`CancelledError`, so generator teardown propagates as before. Only the
bookkeeping survives.

Real stack, uvicorn 0.40.0 + starlette 1.3.1, raw-socket disconnect
mid-stream:

| effect | before | after |
|---|---|---|
| Prometheus counters | committed | committed |
| ledger write | ran | ran |
| OTel | **skipped** | ran |
| cost tracker | **skipped** | ran |
| request log | **skipped** | ran |
| PERF line | **skipped** | ran |
| caller sees `CancelledError` | yes | yes |

The new test fails on its parent commit with a `TimeoutError`.

## Test changes

Dropped `test_event_loop_keeps_running_during_the_ledger_write`. It
detected a strict subset of what the lock test already detects:

| scenario | lock test | loop test |
|---|---|---|
| correct: outside lock + `to_thread` | PASS | PASS |
| regress: INSIDE lock + `to_thread` | FAIL | **PASS** |
| regress: outside lock + sync write | FAIL | FAIL |
| pre-fix: INSIDE lock + sync write | FAIL | FAIL |

Added a concurrency test in its place, which covers what the offload
actually introduces: before the move every proxy ledger write ran on the
one event-loop thread and was serialised for free, and now N in-flight
requests append from N worker threads.

One thing left open. That same intra-process concurrency reaches
`_maybe_compact`, which rewrites the file in place. On POSIX the
ledger's own `flock` serialises it. On Windows `_HAS_FCNTL` is false and
all locking is skipped, so a single Windows proxy can now interleave
writers where the loop thread used to serialise them. The cross-process
form of that is pre-existing and called out at `savings_ledger.py:38`.
Happy to take the intra-process guard here or in a follow-up.

Two notes on the sweep above, now that the diff is three files. The
blast radius re-run at this head is 136 of 145 files green, and the nine
non-green are identical with and without the change. Two of them
(`test_gemini_function_response_waste.py`,
`test_openai_responses_context_compaction.py`) are not in the seven
listed earlier; I re-ran both against a reverted `server.py` and they
hang the same way on both sides.
2026-07-20 11:01:34 -07:00
AshishandClaude Fable 5 a14ab45cf0 fix(proxy): make budget enforcement actually work (#885)
## Description

`CostTracker._costs` was initialized but never written to, so
`get_period_cost()` always returned `0` and `check_budget()` always
returned "allowed" — the `--budget` flag was a silent no-op.
`_prune_old_costs()` was dead code with zero callers. This makes budget
enforcement actually work: requests are rejected once the configured
limit is reached.

Closes # <!-- no tracked issue; discovered during a proxy-pipeline audit
-->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/proxy/cost.py`** — `record_tokens()` now computes the
request cost via `estimate_cost()` and appends it to `_costs`,
activating `_prune_old_costs()`. When a call site has no API usage
breakdown (cache/uncached all zero), `tokens_sent` is used as the input
count so input cost is not silently dropped. `COST_RETENTION_HOURS` 24 →
744 so retention covers the longest budget period (monthly sums from the
1st; 24h retention would have under-enforced monthly budgets).
- **`headroom/proxy/outcome.py`** — the request funnel passes
`output_tokens` through to `record_tokens()` so costs include output,
for all providers.
- **`headroom/cli/proxy.py`** — added `--budget-period
[hourly|daily|monthly]` (env `HEADROOM_BUDGET_PERIOD`); it existed in
`ProxyConfig` and the server entry point but was unreachable from the
main CLI. Fixed the `--budget` help text that wrongly said "resets at
midnight UTC".
- **`headroom/cli/main.py`** — minor registration/version plumbing.
- Tests: regression coverage for the full `record_tokens →
get_period_cost → check_budget` chain, the `tokens_sent` fallback, and
the `--budget-period` flag/env wiring.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_cost_tracker_counterfactual.py tests/test_request_outcome.py -q
40 passed

$ ruff check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py
All checks passed!

$ mypy headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py --ignore-missing-imports
Success: no issues found
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, branch `fix/budget-enforcement`
at the PR head commit.
- Exact command / steps: `pytest
tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs
-v` — sets `CostTracker(budget_limit_usd=0.0001)`, records ~$1.50 of
Sonnet input, then asserts `check_budget()` returns not-allowed with
`remaining == 0`.
- Observed result: budget is now enforced — `get_period_cost()` reflects
real spend and `check_budget()` rejects once the limit is exceeded (the
proxy returns HTTP 429 on that path). On `main` the same test fails
because `_costs` is never populated and `check_budget()` always returns
allowed.
- Not tested: live end-to-end rejection against a running proxy with
real upstream traffic; the running proxy needs a restart on this version
to pick up the fix.

```text
$ pytest tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs \
         tests/test_cost_tracker_counterfactual.py::test_budget_input_cost_counted_without_usage_breakdown -v
2 passed
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — CLI/backend change with no UI surface. See **Test Output** and
**Real Behavior Proof** above for terminal evidence.

## Additional Notes

- The `ci.yml` coverage-upload change originally added here (commit
`120696e5`) was superseded by an equivalent block the maintainer added
to `main`; the merge from main resolved to main's version. Codecov now
reports all modified lines covered.
- N/A checklist items: no docs or CHANGELOG entry — this is an internal
correctness fix to an existing flag.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:22:27 -05:00
JD Davis 3c77e52ce4 feat: add Vertex AI proxy routing (#793)
## Description

Adds first-class GCP Vertex AI proxy routing for publisher REST
endpoints so Vertex requests are forwarded to a configurable regional
Vertex host instead of falling through to the generic
OpenAI/Anthropic/Gemini passthrough selection.

Fixes #792

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update

## Changes Made

- Added a `vertex` provider target with `VERTEX_TARGET_API_URL` and
`--vertex-api-url` support.
- Registered explicit Vertex publisher routes for Google
`generateContent`, `streamGenerateContent`, `countTokens` and Anthropic
publisher `rawPredict`, `streamRawPredict` passthrough.
- Added startup banner/routing output for Vertex AI.
- Added focused tests for provider target resolution, CLI/env config,
banner output, and route delegation.
- Added `wiki/vertex.md` with usage examples and Google Cloud source
links.

## Sources

- Vertex AI Gemini inference reference:
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference
- Google Cloud REST authentication:
https://docs.cloud.google.com/docs/authentication/rest
- Google Application Default Credentials:
https://docs.cloud.google.com/docs/authentication/application-default-credentials

## Testing

- [x] Linting passes (`python -m ruff check .`)
- [x] New tests added for new functionality
- [x] Focused unit tests pass
- [ ] Full unit suite completed locally
- [ ] Rust tests completed locally
- [ ] Type checking passes locally

## Test Output

```text
$ python -m ruff check .
All checks passed!

$ python -m pytest tests/test_provider_registry.py tests/test_provider_proxy_routes.py tests/test_cli_proxy_env.py tests/test_banner_upstream_targets.py -q
57 passed, 1 warning in 13.75s
```

Local limitations:

- `python -m pytest tests scripts/tests -q` timed out after 1 hour on
this Windows machine before completing.
- `cargo test -p headroom-proxy --test integration_vertex_raw_predict`
could not run because `cargo` is not installed on PATH in this
environment.
- The commit hook's `mypy` step fails locally on an existing Windows
`fcntl` typing issue in `headroom/subscription/tracker.py`; `ruff`,
`ruff-format`, and plugin-version hooks passed, and the commit was made
with only `mypy` skipped.

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I have added tests that prove the feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
2026-06-09 23:05:30 -07:00
chopratejas 694589fec4 refactor(proxy): collapse 3 stream finalizers onto RequestOutcome.from_stream
Three streaming finalizers — ``_finalize_stream_response``,
``_stream_response_bedrock``, ``_stream_openai_via_backend`` — each
duplicated the same set of body- and config-derived fields when
constructing a ``RequestOutcome``:

  * ``attempted_input_tokens = optimized_tokens + tokens_saved``
  * ``num_messages = len(body.get("messages", []))``
  * ``request_messages`` conditional on ``config.log_full_messages``
  * ``transforms_applied`` list → tuple (frozen-dataclass contract)
  * ``tags or {}`` normalization
  * ``turn_id`` via ``compute_turn_id``

The last one was a real bug. Only the Bedrock site computed
``turn_id`` — sites 1 and 3 silently dropped it, breaking the
dashboard's multi-turn-session grouping for every Anthropic-SSE and
OpenAI-via-backend request. The new ``RequestOutcome.from_stream``
classmethod computes it uniformly so the three finalizers cannot
drift apart on derivation logic again.

Each call site now hands ``from_stream`` the body + provider-specific
cache/timing fields and gets a fully-constructed outcome back. The
funnel call after it stays identical (``await
self._record_request_outcome(outcome)``).
2026-05-14 22:34:23 -07:00
chopratejas 96a674f1b9 feat(proxy): add client (harness) identification — per-harness analytics across every handler
Cashes in the RequestOutcome refactor with a typed-field surface that
gives EVERY handler per-harness visibility — Codex / Claude Code /
aider / Cursor / Zed / opencode / DROID / antigravity / etc. — for one
field-add across the contract.

The one-field-add proof

Headroom went from "what fraction of OUR requests come from which
harness?" being unanswerable (handlers logged ad-hoc User-Agent strings
in heterogeneous tag dicts at 18 sites, with 9 of 18 not even
populating them) to one structured ``client: str | None`` value on
every observation flowing through the funnel. No new bookkeeping at
call sites; every handler picks it up via a single
``classify_client(headers)`` call at request entry.

Implementation

* New ``CLIENT_UA_MAP`` + ``classify_client()`` in
  ``headroom/proxy/auth_mode.py``. Substring match against
  User-Agent; ``X-Client`` header overrides UA. Returns ``str | None``
  so ``None`` is the loud "unidentified" signal rather than a silent
  empty bucket.
* New ``RequestOutcome.client: str | None = None`` field.
* Funnel updates (in ``outcome.py``):
  - Appends ``client=X`` to the PERF log line ONLY when set, so
    ``headroom perf --client X`` parsing stays clean for
    unidentified traffic (no bogus ``client=`` token).
  - Copies ``client`` into ``RequestLog.tags["client"]`` so the
    dashboard's existing tag-based filtering surfaces per-harness
    slicing with zero new columns.
* Every handler that constructs a RequestOutcome now passes
  ``client=client`` — wired across streaming.py (3 finalizers,
  with ``_finalize_stream_response`` gaining a new optional kwarg
  since it doesn't have direct access to headers), anthropic.py
  (6 sites), openai.py (8 sites including Codex WS), gemini.py
  (2 emitting sites), batch.py (5 sites).

Harnesses recognised

  Anthropic ecosystem:  claude-code, claude-cli, claude-vscode,
                        anthropic-cli
  OpenAI ecosystem:     codex-cli
  Editors:              cursor, zed
  AI coding harnesses:  aider, droid, opencode, github-copilot
  Other:                antigravity (Google experimental)

Adding a new client is a one-line edit to ``CLIENT_UA_MAP``.

Tests

* 8 new tests in ``test_request_outcome.py`` covering:
  - ``client`` field round-trips on the value type
  - ``classify_client`` against every recognised UA prefix
  - ``X-Client`` header override beats UA match
  - ``None`` for unknown traffic (the loud signal)
  - Funnel appends ``client=X`` to PERF when set
  - Funnel OMITS ``client=`` from PERF when None (no bogus empty)
  - Funnel stamps ``client`` into ``RequestLog.tags``
* All 228 existing tests still pass (full sweep across streaming,
  cache, Codex, Anthropic, OpenAI, Gemini, batch, auth-mode).
* ruff + ruff-format + mypy clean.

What's now true that wasn't before

Once this lands, the dashboard can answer:

* "Show me cache hit rate by harness"
  → ``GROUP BY tags.client FROM request_log``
* "Which harness contributes the most cache writes?"
  → same
* "Per-harness savings ratio"
  → same
* ``headroom perf --client codex`` / ``--client claude-code``
  → analyzer filters PERF log lines on ``client=X`` token

Zero new bookkeeping in handlers. Zero changes to Prometheus label
cardinality (kept the client dimension out of Prometheus on purpose —
the tags route is the right surface). The "what's our traffic split
by harness?" question is now answerable in three places (PERF log,
RequestLog tags, dashboard widgets that already filter on tags)
without any per-provider work.
2026-05-14 19:39:35 -07:00
chopratejas e898f68b89 refactor(proxy): introduce RequestOutcome funnel; collapse 3 streaming finalizers
P0 audit (docs/superpowers/specs/P0-proxy-pipeline-audit.md) catalogued
**18 metrics.record_request call sites** across 4 handler files with **4
distinct argument shapes**: 9 of 18 omitted `cached=`, 7 of 18 omitted
`attempted_input_tokens=` (= bug #454/#455's "headline 0%"), only 4 sites
emitted a `PERF` log line (= bug #327's "msgs=0" sibling — Codex traffic
invisible to `headroom perf`), and `cache_hit` was hardcoded `False` at
9 of 18 RequestLog sites.

The cause was structural, not tactical: every site was independently
deciding what "record this completed request" meant. This PR puts a
single value type + a single function between the handlers and the
metrics layer.

Two new files:

* `headroom/proxy/outcome.py` — `RequestOutcome` frozen dataclass.
  Captures everything we ever need to record about one completed
  request: identity, tokens, cache stats (per-TTL splits + inferred
  flag for OpenAI), timing, transforms, diagnostics. Provider-specific
  fields default to neutral values so non-Anthropic handlers don't have
  to know about 5m/1h splits, non-OpenAI handlers don't have to know
  about inferred writes, etc. Computed properties (`cache_hit`,
  `cache_hit_pct`, `savings_pct`) make "forgot to compute it" mistakes
  structurally impossible.

* `HeadroomProxy._record_request_outcome` in `server.py` — the single
  funnel. Owns the four downstream effects in canonical order:
    1. `metrics.record_request(...)` with the FULL kwarg set
    2. `cost_tracker.record_tokens(...)` with `(model, tokens_saved,
       optimized_tokens)` positional + all cache kwargs
    3. `logger.log(RequestLog(...))` with `cache_hit` correctly derived
    4. structured `PERF` log line in the canonical key=value shape

Migrated three streaming finalizers in this PR:

* `_finalize_stream_response` (Anthropic native + OpenAI HTTP streaming)
* `_stream_response_bedrock` (Bedrock-native Anthropic streaming)
* `_stream_openai_via_backend` (OpenAI/Azure backend via LiteLLM/AnyLLM)

All three previously had inline, drifted versions of the four-call
sequence. Each is now ~70 fewer lines: build a `RequestOutcome` from
local context, call `self._record_request_outcome(outcome)`. The
prefix-tracker mutation (Anthropic-specific) stays outside the funnel —
different concern.

Six more migrations queued for follow-up PRs (handle_anthropic_messages
6 sites, handle_openai_chat, handle_openai_responses, handle_openai_
responses_ws 2 sites, handle_gemini_*, handle_databricks_invocations).
Each is mechanical now.

Tests
* New: `tests/test_request_outcome.py` — 14 tests covering value-type
  contract (frozen, derived properties, neutral defaults) + funnel
  contract (full record_request kwargs, canonical record_tokens shape,
  derived cache_hit in RequestLog, PERF log key=value format,
  optional cost_tracker/logger). Bind the real production method via
  descriptor binding so the test exercises the real implementation, not
  a fork.
* All 135 existing streaming/cache/Codex tests pass with zero
  regressions (`tests/test_backend_streaming_cache_metrics.py`,
  `test_proxy_streaming_request_logger.py`, `test_proxy_streaming_resilience.py`,
  `test_proxy_anthropic_cache_stability.py`, `test_openai_codex_*`,
  `test_responses_ws_pyo3_compression.py`, `test_anthropic_pre_upstream_backpressure.py`).
* `mypy headroom/proxy/{outcome,server,handlers/streaming}.py` clean.
* `ruff check` clean.

Surface impact
* −238 lines from `handlers/streaming.py` (deduplication).
* +92 lines in `server.py` (the funnel — counted ONCE, not 18×).
* +130 lines in new `outcome.py` (frozen dataclass + docstrings).
* Net production code: ~−16 lines today, ~−500 lines after the
  remaining six migrations land.

Forward design constraints (per
docs/superpowers/specs/P0-proxy-pipeline-audit.md §7)
* KISS: one value type, one function, no factory hierarchies.
* No regex in routing — handlers stay provider-specific in their
  upstream contract. Output unification only.
* No silent fallbacks — `cache_hit` is computed, not defaulted.
  `cache_inferred=True` is the loud signal when OpenAI write count
  came from `_infer_openai_cache_write_tokens`.
* PERF format frozen so `headroom/perf/analyzer.py` keeps parsing
  cleanly; P3 follow-up replaces the free-text shape with a
  structured event.
2026-05-14 19:38:54 -07:00