mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
feat/https-compose-caddy
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
00436a8b11 |
feat(savings): book modelled output savings per request, as a token and dollar range (#4878)
## Description
Benchmark factors registered with `register_modelled_factors` reached
`/stats` only as a percentage. They never reached the per-request
rollup, so modelled output savings had no tokens or dollars in the
lifetime totals that the local dashboard and the control-plane heartbeat
read. This PR books each shaped request's output saving as a low–high
range of tokens and dollars, from the same tier the dashboard reports.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue) — an unshaped
treatment request was booked a baseline saving
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `SavingsRecorder.request_savings_range(labels, output_tokens) -> (low,
high)` replaces `estimate_request_savings` in the outcome funnel.
- It counts only requests that were actually shaped, marked by the
`output_shaper:verbosity:L<n>` label. The treatment stratum label is
attached before shaping is attempted, so it proves nothing.
- A holdout (control) request books nothing.
- A response with no reported output usage (output 0) books nothing.
- The tier comes from `SavingsLedger.select_tier(level)`, the function
`best_estimate` now delegates to. Lifetime dollars therefore always come
from the tier that the dashboard's percentage comes from:
- **measured:** the request's own stratum's control mean minus its
output, with low = high. It applies only if that stratum qualifies at
the cluster gate the measurement used. A stratum outside the measurement
books 0, because the measured headline excludes it too.
- **estimated:** the request's stratum baseline, with low = high.
- **modelled:** the factor band, `observed × r/(1−r)` at each end.
- `SavingsTracker`:
- Books the low end everywhere it booked before (history, rollups,
OTel).
- Adds `lifetime.output_tokens_saved_high` and
`output_savings_usd_high`, priced per model at output rates.
- Both state builders write the new keys, so a fresh tracker has them.
- Dashboard: the Output Tokens Saved tile adds "≈ $low–$high saved on
output · low–high tokens, all time", read through a component getter.
OSS is unchanged. It ships an empty factor table, so the modelled tier
never appears.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): clean on the touched files
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ git log --oneline -1
|
||
|
|
a9815bbb5f |
fix(proxy): record the provider's billed input count on streamed turns (#4052)
## Description Every input-volume figure Headroom reports for streamed Claude traffic was Headroom's own tokenizer estimate, not Anthropic's billed count. That covers "input after Headroom", `/stats` `tokens.input`, the dashboard's per-model table and the licence usage report's `tokens_after`. Since Claude Code always streams, this was effectively all Claude Code traffic, so customers comparing our numbers with the Anthropic Console could not reconcile them. **Root cause.** `emit_request_outcome` records `provider_input_tokens or optimized_tokens`. The streaming finalizer never set `provider_input_tokens` for Anthropic (#3946 fixed only the uncached sub-count), so every streamed Claude turn fell back to `optimized_tokens`. That value is: - an `o200k_base` count standing in for Claude's private tokenizer, measured against `count_tokens` on 6 real Claude Code transcripts at 15–25% under Claude 4.x and 35–38% under Claude 5.x; - a count of the forwarded **messages only**, which never sees the system prompt or tool definitions. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`headroom/proxy/provider_usage.py` (new).** One place that knows each provider's usage semantics: - Anthropic, Bedrock and Vertex: `input_tokens` is the uncached tail, and the billed total is `input + cache_read + cache_write`. - OpenAI `prompt_tokens`/`input_tokens` and Gemini `promptTokenCount` are already inclusive of cached tokens. - **Streaming finalizer.** Sets `provider_input_tokens` from the provider's usage on the Anthropic/OpenAI/Gemini SSE path, the Bedrock stream path and the OpenAI-via-backend stream path. `RequestOutcome.from_stream` gains the parameter. - **OpenAI pass-through.** On both streaming and non-streaming paths, Anthropic-shape usage was read as if `input_tokens` were the total. It now uses the billed total and is marked provider-reported. - **Gemini non-streaming and OpenAI Responses WebSocket.** These already used the provider's count as `optimized_tokens` but never declared it, so they now also set `provider_input_tokens`, but only when the provider actually sent a count. - **No estimate is ever labelled as billed.** The Bedrock #1132 `message_start` backfill and Gemini's fallback for a missing `promptTokenCount` both stay estimated. The pass-through usage dialect is chosen by provider, not inferred from which keys are present. - **Provenance counters.** Every input total now carries how much of it is billed versus estimated: - `/stats` `tokens.input_provider_reported`, `input_estimated`, `input_provider_reported_percent`, `requests_input_*`. - Licence usage report: `tokens_after_provider_reported`, `tokens_after_estimated`, `requests_provider_reported`, plus the billed `cache_read_tokens`, `cache_write_tokens` and `uncached_input_tokens`, so the cloud side can reconcile with the console (input + cache read + cache write). The report also states `tokens_saved_basis: "headroom_tokenizer"`. - A one-time `event=input_tokens_estimated` warning per provider/model when a successful request carried no provider usage. - **No delta math changes.** `optimized_tokens` stays on the local scale, and `tokens_saved`, ratios and the ledger delta are untouched. The provider count is never subtracted from a local count. ## Testing - [ ] Unit tests pass (`pytest`): targeted suites only, listed below. The full suite is left to CI (see Additional Notes). - [x] Linting passes (`ruff check .`): run on all changed files. - [x] Type checking passes (`mypy headroom`): run on the changed modules. - [x] New tests added for new functionality - [x] Manual testing performed `tests/test_provider_billed_input.py` adds 15 tests, plus regression tests for the review fixes in `tests/test_bedrock_streaming_input_tokens.py` (the #1132 backfill is reported as estimated) and `tests/test_gemini_ccr_continuation_usage.py` (a missing `promptTokenCount` is not provider-reported): - **Recorded fixture.** `tests/fixtures/anthropic/stream_cached_turn.sse` is a **real recorded Anthropic stream**: a second call on a cached ~8.9K-token system prompt, billed as 19 uncached + 8,857 cache-read. It is replayed in 97-byte chunks through the proxy's own SSE usage parser and `_finalize_stream_response`, then through `emit_request_outcome` into a real `PrometheusMetrics`, `CostTracker` and `UsageReporter`. The test asserts that `/stats` input and the licence `tokens_after` equal Anthropic's 8,876. - Other cases: - an OpenAI stream records the inclusive `prompt_tokens`; - a stream without usage stays estimated; - each provider shape is covered; - the pass-through normaliser returns the total. Against the unpatched handlers, the 5 behaviour tests fail. The 8 that pass exercise only the new helper. ### Test Output ```text $ pytest tests/test_provider_billed_input.py tests/test_proxy_handler_helpers.py tests/test_request_outcome.py \ tests/test_proxy_streaming_request_logger.py tests/test_proxy_cache_ttl_metrics.py \ tests/test_usage_reporter_snapshot.py tests/test_5xx_accounting_all_providers.py tests/test_copilot_provider_label.py \ tests/test_bedrock_streaming_input_tokens.py tests/test_gemini_ccr_continuation_usage.py -q 181 passed in 6.56s # same new tests against the unpatched handlers FAILED tests/test_provider_billed_input.py::test_passthrough_normaliser_returns_anthropic_total_not_uncached_tail FAILED tests/test_provider_billed_input.py::test_recorded_anthropic_stream_records_anthropics_billed_input FAILED tests/test_provider_billed_input.py::test_openai_stream_records_inclusive_prompt_tokens FAILED tests/test_provider_billed_input.py::test_streamed_claude_turn_reaches_dashboard_and_licence_report_as_billed FAILED tests/test_provider_billed_input.py::test_estimated_input_is_reported_as_estimated 5 failed, 8 passed $ ruff check <changed files> -> All checks passed! $ mypy <changed proxy/telemetry modules> -> Success: no issues found in 6 source files ``` ## Real Behavior Proof - Environment: local proxy from this branch (`headroom proxy --port 8931 --stateless --no-telemetry --no-cache`) against the real Anthropic API. For the baseline, the same command ran from a `main` worktree on port 8932. - Exact command / steps: 3 streamed `claude-sonnet-4-6` turns through the proxy, with a cached ~20K-token system prompt, one tool and a 300-item JSON tool result. Anthropic's `message_start` usage (input + cache read + cache write) was summed from each stream and compared with `GET /stats` → `tokens`. - Observed result: the author-provided live Anthropic comparison below reports exact billed-input agreement at 61,999 tokens; the baseline undercounts by 37%. This external run was not independently repeated during the current-head review. | | Anthropic billed input | Headroom `/stats` `tokens.input` | |---|---|---| | `main` | 62,001 | **38,835** (−37%) | | this PR | 61,999 | **61,999** (exact), `input_provider_reported_percent: 100.0` | The two runs differ by 2 tokens because they are separate API calls. `proxy_compression_saved` was 29,313 on both, so savings behaviour is unchanged. - Not tested: live Bedrock, Vertex and Gemini streams (unit-tested on provider shapes only), and the licence cloud endpoint receiving the new fields. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is accounting only, and request and response bytes are unchanged. - Minimum rollout channel: N/A - Stable/default behavior changed: reported input volume now uses the provider's billed count where the provider reports one. New fields are added to `/stats` and the licence usage payload; no existing field is removed or renamed. - Kill switch / disable path: none needed. With no usage frame, the fallback is the previous estimate, now labelled `input_estimated`. - Unsafe override required: no - Qualification impact: dashboards and licence reports will show larger, correct Claude input totals once installs upgrade, so historical and new figures are not directly comparable. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes: targeted suites only; full suite on CI - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - **Full suite.** Locally, the full run on the pre-rebase base reached 2,147 passed before stopping on `tests/test_cli/test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`. That test fails identically on unmodified `main` (unrelated MCP install args). The full local run was then abandoned because several wrap/install tests launch a real `serena project index` (rust-analyzer, tsserver, rustc), which should be stubbed in a separate PR. - **Docs.** The new `/stats` and licence fields are not yet documented. - Follow-ups not in this PR: - **Raw Bedrock invoke pass-through (`handlers/bedrock.py`)** forwards bytes without parsing them, so it still records an estimate. It is now counted as `input_estimated`, so it's visible rather than silent. - **Batch create** has no usage until results come back, so it stays estimated as above. - **`tokens_saved` is still on the `o200k_base` scale**, so it under-reports Claude savings. Calibrating it to provider tokens is a separate PR. - **The licence cloud endpoint** needs to read the new fields. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Current maintainer verification (October 8) Exact head: `81527148ed9777e1b9ca921bbf6bb457d49354db`, containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a clean merge. No contributor logic was modified. 63 focused tests pass with the existing compatible native extension on Windows/Python 3.12. Three backend-boundary probes preserve zero client input and exact cache-read, cache-write, and combined provider input totals. This is local mocked-backend/TestClient evidence; the author-provided external-provider evidence above was not independently rerun. Current-head hosted snapshot: 31 successful checks, 12 skips, no failed or live checks. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
f78e66f7cc |
fix(proxy): stop headroom logger from suppressing propagation to stdout/stderr (#3096)
## Description `_setup_file_logging()` in `headroom/proxy/helpers.py` unconditionally set `headroom_logger.propagate = False`, on the theory that `wrap.py` redirects stderr into the same `proxy.log` file the rotating handler writes to, so propagation would double-write records. That premise is wrong: `wrap.py` redirects the proxy subprocess's stdout/stderr to a separate sibling file, `proxy-stdio.log` (see `_get_proxy_stdio_log_path()`), never `proxy.log`. Severing propagation bought no deduplication — it only stopped every `headroom.*` log record from reaching root's `logging.basicConfig()` handler (`proxy/server.py`), which is what writes to stderr/stdout. In containerized deployments (Docker/Kubernetes), stdout/stderr is the only channel the platform collects, so this silently dropped all application logs. Closes #3087 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed `headroom_logger.propagate = False` from `_setup_file_logging()` in `headroom/proxy/helpers.py`. - Updated the surrounding comment to explain why propagation is safe to leave on: `proxy.log` (file handler) and stderr/stdout (root's `basicConfig` handler) are two distinct destinations, so nothing is duplicated in either file. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check . All checks passed! $ pytest tests/test_corrupt_golden_bytes_recovery.py -q collected 9 items tests/test_corrupt_golden_bytes_recovery.py ......... [100%] 9 passed in 0.47s ``` `mypy headroom` was not run to a clean result: it fails before reaching this file with a pre-existing, unrelated error in the installed `numpy` stub (`Type statement is only supported in Python 3.12 and greater`) in this sandbox's dependency set. Not caused by this change (a 4-line pure-Python logging tweak). The broader `pytest tests/` collection also hits ~170 pre-existing, unrelated errors in this sandbox: `headroom._core`, the compiled Rust extension, isn't built here (no `cargo`/`maturin` available), so every module importing it fails at collection. This change doesn't touch Rust code or `_core` and isn't implicated. No new tests added: this restores an existing, already-tested code path (see `tests/conftest.py`'s autouse fixture and `tests/test_corrupt_golden_bytes_recovery.py`, both of which already assert/require `propagate=True` for `caplog` to work) to its intended default rather than adding new behavior. ## Real Behavior Proof - Environment: local sandbox, Python 3.12.3, unable to build/import `headroom.proxy.server` directly (missing compiled `headroom._core`). Verified with an isolated script that copies `_setup_file_logging()`'s exact logic byte-for-byte plus the same `logging.basicConfig()` call `proxy/server.py` makes at import time. - Exact command / steps: ran the fixed logic, then emitted `logging.getLogger("headroom.proxy").info("compression timeout: request abc123 exceeded 30s budget")`. - Observed result: the `headroom.proxy` record was written to stderr AND to `proxy.log` simultaneously (full output below); re-running the same script with the old `headroom_logger.propagate = False` line restored reproduces the bug — the record disappears from stderr entirely and only lands in `proxy.log`. ```text propagate flag on 'headroom' logger: True --- emitting a headroom.proxy log record, watch for it below on stderr --- 2026-08-17 21:17:39,545 - headroom.proxy - INFO - compression timeout: request abc123 exceeded 30s budget --- also present in proxy.log (file handler): OK --- ``` - Not tested: end-to-end through the actual compiled proxy binary/server process (blocked by missing `headroom._core` in this sandbox) and real container log collection (Docker/Kubernetes) — the isolated repro exercises the exact same logging configuration code the real process runs, so this is considered sufficient proxy proof for a pure logging-config change. ## Runtime Rollout Safety - Rollout-managed feature(s): None — this is unconditional startup logging configuration, not behind a rollout flag. - Minimum rollout channel: N/A - Stable/default behavior changed: Yes — `headroom.*` log records now also reach stderr/stdout (previously file-only). No change to request/response handling, compression, or any user-facing proxy behavior. - Kill switch / disable path: None needed; this restores default Python logging propagation. If a deployment truly wants file-only logging, `HEADROOM_LOG_TO_STDOUT`-style suppression could be added later, but no such request exists today. - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this commit to restore `propagate = False`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — logging configuration change, no UI impact. ## Additional Notes - "New and existing unit tests pass locally" is left unchecked because the full `pytest tests/` collection can't complete in this sandbox (missing compiled `headroom._core` Rust extension, unrelated to this change) — see Test Output above for what was actually run and passed. - "I have added tests" is left unchecked: existing tests (`tests/conftest.py` autouse fixture, `tests/test_corrupt_golden_bytes_recovery.py`) already assert the code path this restores; happy to add a focused regression test asserting `logging.getLogger("headroom").propagate is True` after `_setup_file_logging()` if maintainers want one. --------- Co-authored-by: = <=> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
81fe9d5345 |
fix(metrics): attribute tool-schema savings per model, not just compression (#3155)
## Description Reported against 0.36.0 (VS Code + Copilot + Claude Code): the per-model breakdown disagreed with the headline printed four lines above it. ``` Tokens saved: 625,277 · messages 36,071 · tool schemas 589,206 Per-Model Breakdown <a>: 35,907 tokens saved <b>: 0 tokens saved <c>: 164 tokens saved <d>: 0 tokens saved ``` The rows sum to **36,071** — the *messages* line exactly. All 589,206 tokens of tool-schema deferral, 94% of the headline, had no row to land in, so every tool-heavy model reported "0 tokens saved" while real dollars were credited to it. Deferral is disjoint from message compression by construction: deferred schemas never enter the message token counts, so they move neither `tokens_saved` nor `tokens_sent`. The headline, the PERF line, and the savings ledger (#2795) all already fold the two together. Three per-model surfaces did not. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`perf/analyzer.py`** — the per-model loop summed `tokens_saved` while its own headline summed `tokens_saved + tool_saved`. Now uses the same all-layers construction (`headline_before = before + tool_saved`), and prints a `· messages / · tool schemas` split line only when there is a split to show. - **`proxy/savings_tracker.py`** — added a `tool_tokens_saved` bucket to `_empty_by_model_entry()`, normalization, and `_record_by_model_locked()`; `record_request()` gained a `tool_search_saved` parameter. `_by_model_snapshot_locked()` ranks and computes `savings_percent` off the combined figure and exposes `headline_tokens_saved`. - **`proxy/prometheus_metrics.py`** — **the seam.** `record_request` already accepted `tool_search_saved` and already folded it into the per-model *dollars*, but never passed it to `savings_tracker.record_request`. Tokens and money therefore disagreed on the same row. - **`proxy/cost.py`** (feeds the dashboard's "Per-Model Token Savings" table) — added `_tool_saved_by_model`, a `tool_schema_saved` kwarg, and `compression_tokens_saved` / `tool_tokens_saved` alongside a combined `tokens_saved`. The `stats()` loop now iterates the **union** of both dicts: keying off compression alone dropped a deferral-only model from the table entirely rather than merely under-reporting it. - **`proxy/outcome.py`** — forwards the figure it already computed for `metrics.record_request` to `cost_tracker.record_tokens`. - **`dashboard.html`** — the "Tokens Saved" cell gains a `title` showing the compression/deferral split. Design notes: - Components stay separately addressable rather than widening an existing field's meaning in place, so persisted state remains readable by older readers. - Percentages use the all-layers numerator over `saved + sent` — deferred schemas were never in `sent`, so that is still the pre-Headroom volume. - `CostTracker.stats()["savings_usd"]` is deliberately **not** widened: deferral is already priced by `SavingsTracker`, and this tracker's dollars feed budget enforcement, where counting it twice would double-book the saving. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_per_model_tool_savings.py -q 11 passed in 0.94s # Same file against pre-fix code (git stash), proving the tests bite: 5 failed, 1 passed FAILED test_per_model_rows_reconcile_with_the_headline FAILED test_a_tool_only_model_no_longer_reads_zero FAILED test_tracker_attributes_deferral_to_the_model FAILED test_tracker_default_is_unchanged_without_deferral FAILED test_state_written_before_this_field_existed_still_loads (the one that passes pre-fix is the "compression-only model is unchanged" guard) $ pytest tests/ -q # this branch 3 failed, 11374 passed, 587 skipped in 343.55s $ pytest tests/ -q # clean origin/main, same machine 3 failed, 11364 passed, 587 skipped in 352.64s Identical 3 failures on both — pre-existing and environmental, not regressions: test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline test_release_workflows.py::test_no_native_tls_in_wheel_build_tree (FileNotFoundError: 'cargo') test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter (whole-suite ordering flake; tests/test_graceful_shutdown.py passes 11/11 in isolation on this branch) $ ruff check headroom/ All checks passed! $ mypy headroom/proxy/cost.py headroom/proxy/savings_tracker.py \ headroom/proxy/prometheus_metrics.py headroom/proxy/outcome.py \ headroom/perf/analyzer.py Success: no issues found in 5 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.13, this branch rebased on `origin/main` @ `1f96dabc`. - Exact command / steps: reproduced the reported shape as a unit test — three models with 35,907 / 0 / 164 message savings and 400,000 / 189,206 / 0 deferral, then rendered `format_report`. - Observed result: headline `Tokens saved: 625,277` unchanged; rows now read `435,907` / `189,206` / `164` and sum to the headline. The seam test drives the real `PrometheusMetrics.record_request` and asserts the tracker's `by_model` entry ends up at `tokens_saved=400, tool_tokens_saved=54,000, headline_tokens_saved=54,400`. - Not tested: no live proxy run against a real Copilot/Claude Code session; the arithmetic is pinned at the four code seams instead. The dashboard `title` tooltip is markup-only and not covered by a rendering test. ## Runtime Rollout Safety - Rollout-managed feature(s): none — this is reporting arithmetic, not a request-path behavior. - Minimum rollout channel: n/a. - Stable/default behavior changed: yes, displayed per-model token savings and percentages increase to include tool-schema deferral. No request is treated differently. - Kill switch / disable path: n/a. Components remain separately readable (`compression_tokens_saved` / `tool_tokens_saved`) if a consumer wants the old message-only figure. - Unsafe override required: none. - Qualification impact: none — `savings_usd` and budget enforcement are unchanged by design. - Rollback path: revert the commit; `tool_tokens_saved` in persisted state is then simply ignored by the older reader. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
620028fa18 |
fix(proxy): emit request log timestamps in UTC
## Description `RequestLog.timestamp` was serialized with `datetime.now().isoformat()`, which omits timezone information. Browsers then interpret the value as local time, so requests from a UTC container can display negative ages in non-UTC dashboards. Closes #2910 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Emit request-log timestamps from `datetime.now(timezone.utc)` so the ISO-8601 value includes `+00:00`. - Add a regression test that parses the emitted timestamp and requires a UTC offset. ## Testing - [x] New tests added for the regression - [x] `python -m compileall -q headroom/proxy/outcome.py tests/test_request_outcome.py` - [x] `git diff --check` - [ ] Unit tests pass (`pytest`) — the repository's Rust extension cannot build in this Windows environment because `link.exe` (MSVC) is unavailable; the focused test is included for CI. ### Test Output ```text python -m compileall -q headroom/proxy/outcome.py tests/test_request_outcome.py (pass) git diff --check (pass) uv run pytest tests/test_request_outcome.py -q blocked while building headroom-py: linker `link.exe` not found ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.11; proxy timestamps are generated in `headroom/proxy/outcome.py`. - Exact command / steps: traced the Recent Requests write path and added a timestamp assertion in `tests/test_request_outcome.py` (CI will run with the project's Rust toolchain). - Observed result: the production call now emits an ISO-8601 timestamp with `+00:00`; the regression assertion requires an offset-aware UTC value, preventing browser timezone skew. - Not tested: full pytest suite locally because the MSVC linker is unavailable. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review - [x] I have added tests that prove my fix is effective - [x] I did not edit `CHANGELOG.md` Signed-off-by: Suliman Abdulrazzaq <suliman9000a@gmail.com> |
||
|
|
a033ac4176 |
fix(cost): send litellm the total prompt so --budget stops seeing $0 (#2757)
## Description **`--budget` has been silently inert on any cache-warm request** — which is the normal case in an agent session. This is a disabled control, not a metrics bug. `record_tokens` passed only the **uncached slice** as litellm's `prompt_tokens`. Measured, `litellm.cost_per_token` charges: ``` (prompt_tokens - cache_read - cache_creation) * input_rate + cache_read * read_rate + cache_creation * write_rate ``` So `prompt_tokens` is the **whole** prompt and litellm removes the cached parts itself. Handing it the uncached slice drives the input term **negative** as soon as anything is cached. `estimate_cost` ends with `float(total) if total > 0 else None`, so it returned `None`, no `CostEntry` was appended, and `check_budget()` saw **$0**. | model | 100k prompt, 80k cached — old call | booked | | --- | --- | --- | | `gpt-5` | **-$0.065000** | None | | `gpt-4o-mini` | **-$0.003000** | None | | `claude-sonnet-4-5` | **-$0.156000** | None | Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Fixing the total alone would **over-charge OpenAI**, because two bugs are entangled here. OpenAI exposes no cache-write counter, so `_infer_openai_cache_write_tokens` uses the uncached portion as a write proxy. At two of the four inference sites (`openai.py:4304`, `openai.py:5413`) `uncached_input_tokens` is derived by subtracting **only** `cache_read`, so `cache_write_tokens` and `uncached_tokens` are **the same tokens**. Summing all three would double-count the prompt and charge a write premium OpenAI does not have. (The other two sites — `openai.py:3969` and `streaming.py:2024` — subtract both, so their buckets are genuinely disjoint; those are left alone.) - `cost.py` — `record_tokens` now passes `uncached + cache_read + cache_write` as the prompt total. - `cost.py` — new `cache_inferred: bool = False` parameter. When set, the inferred write is excluded from **both** the prompt total and the write premium. The default preserves behaviour for every provider that reports disjoint buckets. - `outcome.py` — plumbs `outcome.cache_inferred` through. The field already existed on `RequestOutcome` for the dashboard; it just never reached cost. - `handlers/openai.py` — sets `cache_inferred=True` at the two outcome sites whose buckets genuinely duplicate. ## Testing - [x] Unit tests pass (new file) - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/proxy/handlers/openai.py All checks passed! $ uvx ruff@0.15.17 format --check <same three> 3 files already formatted $ pytest tests/test_cost_budget_total_prompt.py -q 5 passed in 0.26s ``` The 5 new tests assert on the **arguments handed to `estimate_cost`** rather than on dollar values, so they pin the contract that broke without depending on litellm's pricing tables — or on litellm being installed at all. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated git worktree off `upstream/main`. **litellm's actual formula**, established by probe rather than assumed: ```text rates (claude-sonnet-4-5): input=3e-06 read=3e-07 write=3.75e-06 total=100k, 80k read, 5k write -> $0.087750 hypothesis (p-r)*ir + r*rr + w*wr = $0.102750 x hypothesis (p-r-w)*ir + r*rr + w*wr = $0.087750 <- matches ``` **Before → after:** ```text Anthropic, disjoint: uncached=900 read=48000 write=1500 OLD prompt=900 raw=-0.125775 booked=None <-- BUDGET BLIND NEW prompt=50400 raw= 0.022725 booked=0.022725 OpenAI, inferred write == uncached: uncached=20000 read=80000 write=20000 OLD prompt=20000 raw=-0.090000 booked=None <-- BUDGET BLIND NEW prompt=100000, write excluded raw= 0.035000 booked=0.035 truth=0.035000 ``` The OpenAI "after" equals the hand-computed truth `20,000*input + 80,000*read_rate` exactly. - **Not fully tested locally.** The cost/outcome suites show an **identical 10-failure set** on this branch and on clean `upstream/main` in the same throwaway env — all `ModuleNotFoundError: headroom._core`, the compiled Rust extension this machine cannot currently build. So they are environment-only, not regressions. One of them, `test_funnel_passes_canonical_record_tokens_shape`, covers the `record_tokens` call shape this PR changes, so **CI is the authoritative check for that one**. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Context Found during a tokenizer-consistency audit that also turned up: HuggingFace-routed models counting a 6,000-char message as **2 tokens**, `gpt-5`/`o4-mini` falling to a char estimator, and the Kompress size gate missing its own cap by 24%. Those are separate PRs — different subsystems, different risk. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
|
184146b688 |
fix(savings): surface request growth the tok_saved clamp swallows (#2708)
## Description `tokens_saved` is clamped at zero, so a request the proxy forwards **larger** than it arrived is indistinguishable in the PERF line from one it simply could not compress. Both read `tok_saved=0`. That ambiguity hides real regressions. Anything that appends to the body after compression — proactive context expansion, memory injection — can outweigh the compression it sits on top of and still look like a neutral turn. On the session that prompted this, a request went from 55,161 tokens in to 57,845 out and reported `tok_saved=0`, for 19 consecutive turns, with nothing in the logs distinguishing it from a turn with nothing left to compress. This reports the swallowed amount as `tok_inflated`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `RequestOutcome.tokens_inflated`: `max(0, optimized_tokens - original_tokens)`, derived from two counts the outcome already carries — **no new plumbing at any of the emit sites**. - Added `tok_inflated=` to the PERF log line, next to `tok_saved=`. Diagnostic only, deliberately. It does **not** feed `tokens_saved` or `attempted_input_tokens`, for two reasons: 1. `attempted_input_tokens = optimized_tokens + tokens_saved` is a *size*, not a signed delta. Letting the second term go negative makes it smaller than the bytes actually forwarded, corrupting the active-savings denominator. 2. Injection paths already book their own cost through the retrieval-drawback channel. A negative landing in `tokens_saved` as well would count the same loss twice. So the clamp stays and the hidden number surfaces beside it. Worth noting there is already a revert-on-inflation guard *before* compression's own inflation can escape (`anthropic.py`, "Optimization inflated tokens … reverting to original messages") — it is only growth added *after* that point which the clamp was silently absorbing. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_request_outcome.py tests/test_cli_perf_format.py -q ============================== 54 passed in 1.56s ============================== $ pytest tests/ -q -k "outcome or perf or savings or stats" ======= 510 passed, 33 skipped, 9674 deselected, 542 warnings in 40.37s ======== $ ruff check headroom/proxy/outcome.py tests/test_request_outcome.py All checks passed! $ ruff format --check headroom/proxy/outcome.py tests/test_request_outcome.py 2 files already formatted $ mypy headroom/proxy/outcome.py --ignore-missing-imports Success: no issues found in 1 source file ``` Four new tests pin the distinction that was missing: shrank (0), no-op compression (0, and `tok_saved` also 0 — the two cases that used to look identical), grew (reports 2684 while `tok_saved` stays 0), and that `attempted_input_tokens` / `savings_pct` keep their unsigned semantics. `tests/test_cli_perf_format.py` parses hand-written PERF fixtures by field name, so adding a field does not disturb it — verified green above. ## Real Behavior Proof ### The field catching a real inflating request - Environment: macOS 15 (arm64), Python 3.13.14. A proxy booted from this branch: `headroom proxy --mode token --backend anthropic --anthropic-api-url http://127.0.0.1:<stub>`, isolated `HOME` so the run could not touch a developer's live logs/store, `HF_HOME` pointed at cached kompress weights so the lossy+CCR-marker path is exercised and proactive expansion can actually arm. - Exact command / steps: three requests over one conversation through the real HTTP path (`x-headroom-cwd: /tmp/proof`, `user-agent: claude-code/1.4.2`): a user turn carrying the real `~/.claude/rules/*.md` text (~8.2k tokens); then `assistant` + a short user turn so that block becomes compressible and gets tracked as a CCR entry; then a follow-up whose leading text block shares vocabulary with it, so proactive expansion fires and appends the original — which is how a request ends up leaving larger than it arrived. PERF lines read from the isolated `~/.headroom/logs/proxy.log`. - Observed result: real PERF output from that run — ```text msgs=1 tok_before=8170 tok_after=9553 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=3 tok_before=8184 tok_after=9567 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=5 tok_before=8245 tok_after=11880 tok_saved=0 tok_inflated=3635 ... transforms=router:text_block:mixed ``` Correlated from the same run: `CCR Tracker: Proactively expanded f0cf4efb42373ec225f57725 (1417 items)`, and the stub upstream confirms the block reached the wire (`has_expansion_block: true`, forwarded body 54,130 B on the third request). Every one of those turns reports `tok_saved=0`. Before this change that is all the log said, and it is the same thing it says when there was simply nothing left to compress. `tok_inflated=3635` is the number that was missing. For contrast, the same scenario run against a build where the request genuinely shrinks reported `tok_before=8245 tok_after=7359 tok_saved=886` — that build predates this field, so it does not print `tok_inflated`; the point is only that the inflating and shrinking cases are the two states the field has to separate, and on `main` today both render as `tok_saved=0` whenever the growth path is taken. The `tok_inflated=0` case on a shrinking request is covered by unit test. ### Scale of what was hidden From a live `--mode token` proxy on Claude Code traffic across four rotated logs: **305 of 3,743 requests (8%)** had `tok_after > tok_before` while every one reported `tok_saved=0` — 192,829 tokens of growth rendered as "nothing to compress". The worst single session held +2,643/turn for 19 consecutive turns. - Not tested: the `headroom perf` CLI was not run against a real log file containing the new field — it parses by field name and `tests/test_cli_perf_format.py` is green, but that is test-level rather than end-to-end evidence. Streaming responses were not exercised (the stub replies non-streaming), so the streaming emit path carries the new field on the strength of sharing `emit_request_outcome` rather than by observation. No dashboard or Prometheus consumer was re-run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — one added field on an existing log line; no user-facing surface. ## Additional Notes - Independent of #2706 and #2707 — verified `conflicts=0` via `git merge-tree`; mergeable in any order. - Related but deliberately out of scope: `main` has no producer for retrieval-cost accounting (`record_savings_event` takes no `kind`/`tokens_retrieved`, and nothing writes `tokens_retrieved` anywhere), so proactive expansion's cost is not booked into net savings at all. Adding that channel is a cross-cutting accounting change and belongs in its own PR; this one only makes the growth visible in the log. |
||
|
|
4aac068814 |
fix(proxy/metrics): move the savings-ledger append off the event loop (#2439)
## Description
`PrometheusMetrics.record_request` appends one durable JSONL event per
compressed request. That append is synchronous: `open` + `fcntl.flock` +
`write`, plus a full-file rewrite once the ledger passes 1 MB. It runs
on the event loop, inside `self._lock`.
`export()` takes that same lock and holds it for the entire Prometheus
serialization, so a slow ledger write stops `/metrics` cold. In a repro
run of 200 compressed requests, `/metrics` completed zero scrapes and
the event loop never yielded once across 6.4 seconds.
The append now runs in a thread, outside the lock. `savings_ledger`
already takes its own `flock` across processes, so the metrics lock was
never what made the write safe.
Both halves are one change. Awaiting inside the lock would hold it for
the whole write rather than just the syscall, which is worse than what
is on main today.
The file already documents this hazard against itself.
`record_stage_timings` (`prometheus_metrics.py:867-874`) picks a plain
`threading.Lock` over `self._lock` specifically because "the async lock
is also held by `export()` during Prometheus scrapes." The ledger append
was the pattern that docstring warns about.
No filed issue for this one.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Move the `savings_ledger.record_savings_event` call in
`record_request` out of `async with self._lock` and run it through
`asyncio.to_thread`. The call site keeps its keyword arguments verbatim;
`to_thread` forwards `**kwargs`, so no `functools.partial` wrapper is
needed.
- Keep the `await`. Callers still see the event on disk when
`record_request` returns, which
`tests/test_savings_ledger_before_forwarded.py` asserts synchronously.
- Add `tests/test_savings_ledger_offload.py`: lock scope, event-loop
responsiveness, durability on return, and both arms of the `tokens_saved
> 0 and not stateless` gate.
`savings_ledger.py` is untouched. It stays synchronous so the MCP
`headroom_compress` caller in `ccr/mcp_server.py:789` does not have to
change.
Sizing the executor is left alone on purpose. `asyncio.to_thread` uses
the default pool, which is the documented tool for blocking I/O and
already the idiom here (`helpers.py:1297`, `server.py:1694`, `:3557`,
`:3613`, `:4244`). The compression pools are sized `max(1,
os.cpu_count())` for CPU-bound work, and `PrometheusMetrics` holds no
reference to `HeadroomProxy` anyway, so reaching them would mean a new
constructor parameter.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_savings_ledger_offload.py tests/test_savings_ledger.py tests/test_savings_ledger_before_forwarded.py -q
======================== 26 passed, 1 warning in 4.08s =========================
$ ruff check . && ruff format --check headroom/proxy/prometheus_metrics.py tests/test_savings_ledger_offload.py
All checks passed!
2 files already formatted
$ mypy headroom/proxy/prometheus_metrics.py
Success: no issues found in 1 source file
```
Broader sweep across the blast radius, 145 test files matching savings /
metrics / outcome / stats / proxy / handler / server / ledger / cost /
prometheus, each run under a per-file wall-clock watchdog:
```text
138 files pass, 1470 tests passed
7 non-green:
HANG tests/test_agent_savings.py
HANG tests/test_ccr_mcp_server.py
HANG tests/test_netcost_gate.py
HANG tests/test_proxy_compress_endpoint.py
HANG tests/test_proxy_mode_benchmark.py
HANG tests/test_read_maturation_handler_nobust.py
FAIL tests/test_proxy_copilot_auth_hooks.py::test_openai_passthrough_applies_copilot_auth
Same 7 files re-run with headroom/proxy/prometheus_metrics.py reverted to
|
||
|
|
a14ab45cf0 |
fix(proxy): make budget enforcement actually work (#885)
## Description
`CostTracker._costs` was initialized but never written to, so
`get_period_cost()` always returned `0` and `check_budget()` always
returned "allowed" — the `--budget` flag was a silent no-op.
`_prune_old_costs()` was dead code with zero callers. This makes budget
enforcement actually work: requests are rejected once the configured
limit is reached.
Closes # <!-- no tracked issue; discovered during a proxy-pipeline audit
-->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`headroom/proxy/cost.py`** — `record_tokens()` now computes the
request cost via `estimate_cost()` and appends it to `_costs`,
activating `_prune_old_costs()`. When a call site has no API usage
breakdown (cache/uncached all zero), `tokens_sent` is used as the input
count so input cost is not silently dropped. `COST_RETENTION_HOURS` 24 →
744 so retention covers the longest budget period (monthly sums from the
1st; 24h retention would have under-enforced monthly budgets).
- **`headroom/proxy/outcome.py`** — the request funnel passes
`output_tokens` through to `record_tokens()` so costs include output,
for all providers.
- **`headroom/cli/proxy.py`** — added `--budget-period
[hourly|daily|monthly]` (env `HEADROOM_BUDGET_PERIOD`); it existed in
`ProxyConfig` and the server entry point but was unreachable from the
main CLI. Fixed the `--budget` help text that wrongly said "resets at
midnight UTC".
- **`headroom/cli/main.py`** — minor registration/version plumbing.
- Tests: regression coverage for the full `record_tokens →
get_period_cost → check_budget` chain, the `tokens_sent` fallback, and
the `--budget-period` flag/env wiring.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_cost_tracker_counterfactual.py tests/test_request_outcome.py -q
40 passed
$ ruff check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py
All checks passed!
$ mypy headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py --ignore-missing-imports
Success: no issues found
```
## Real Behavior Proof
- Environment: local macOS, Python 3.11, branch `fix/budget-enforcement`
at the PR head commit.
- Exact command / steps: `pytest
tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs
-v` — sets `CostTracker(budget_limit_usd=0.0001)`, records ~$1.50 of
Sonnet input, then asserts `check_budget()` returns not-allowed with
`remaining == 0`.
- Observed result: budget is now enforced — `get_period_cost()` reflects
real spend and `check_budget()` rejects once the limit is exceeded (the
proxy returns HTTP 429 on that path). On `main` the same test fails
because `_costs` is never populated and `check_budget()` always returns
allowed.
- Not tested: live end-to-end rejection against a running proxy with
real upstream traffic; the running proxy needs a restart on this version
to pick up the fix.
```text
$ pytest tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs \
tests/test_cost_tracker_counterfactual.py::test_budget_input_cost_counted_without_usage_breakdown -v
2 passed
```
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — CLI/backend change with no UI surface. See **Test Output** and
**Real Behavior Proof** above for terminal evidence.
## Additional Notes
- The `ci.yml` coverage-upload change originally added here (commit
`120696e5`) was superseded by an equivalent block the maintainer added
to `main`; the merge from main resolved to main's version. Codecov now
reports all modified lines covered.
- N/A checklist items: no docs or CHANGELOG entry — this is an internal
correctness fix to an existing flag.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
3c77e52ce4 |
feat: add Vertex AI proxy routing (#793)
## Description Adds first-class GCP Vertex AI proxy routing for publisher REST endpoints so Vertex requests are forwarded to a configurable regional Vertex host instead of falling through to the generic OpenAI/Anthropic/Gemini passthrough selection. Fixes #792 ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update ## Changes Made - Added a `vertex` provider target with `VERTEX_TARGET_API_URL` and `--vertex-api-url` support. - Registered explicit Vertex publisher routes for Google `generateContent`, `streamGenerateContent`, `countTokens` and Anthropic publisher `rawPredict`, `streamRawPredict` passthrough. - Added startup banner/routing output for Vertex AI. - Added focused tests for provider target resolution, CLI/env config, banner output, and route delegation. - Added `wiki/vertex.md` with usage examples and Google Cloud source links. ## Sources - Vertex AI Gemini inference reference: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference - Google Cloud REST authentication: https://docs.cloud.google.com/docs/authentication/rest - Google Application Default Credentials: https://docs.cloud.google.com/docs/authentication/application-default-credentials ## Testing - [x] Linting passes (`python -m ruff check .`) - [x] New tests added for new functionality - [x] Focused unit tests pass - [ ] Full unit suite completed locally - [ ] Rust tests completed locally - [ ] Type checking passes locally ## Test Output ```text $ python -m ruff check . All checks passed! $ python -m pytest tests/test_provider_registry.py tests/test_provider_proxy_routes.py tests/test_cli_proxy_env.py tests/test_banner_upstream_targets.py -q 57 passed, 1 warning in 13.75s ``` Local limitations: - `python -m pytest tests scripts/tests -q` timed out after 1 hour on this Windows machine before completing. - `cargo test -p headroom-proxy --test integration_vertex_raw_predict` could not run because `cargo` is not installed on PATH in this environment. - The commit hook's `mypy` step fails locally on an existing Windows `fcntl` typing issue in `headroom/subscription/tracker.py`; `ruff`, `ruff-format`, and plugin-version hooks passed, and the commit was made with only `mypy` skipped. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I have added tests that prove the feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
|
694589fec4 |
refactor(proxy): collapse 3 stream finalizers onto RequestOutcome.from_stream
Three streaming finalizers — ``_finalize_stream_response``,
``_stream_response_bedrock``, ``_stream_openai_via_backend`` — each
duplicated the same set of body- and config-derived fields when
constructing a ``RequestOutcome``:
* ``attempted_input_tokens = optimized_tokens + tokens_saved``
* ``num_messages = len(body.get("messages", []))``
* ``request_messages`` conditional on ``config.log_full_messages``
* ``transforms_applied`` list → tuple (frozen-dataclass contract)
* ``tags or {}`` normalization
* ``turn_id`` via ``compute_turn_id``
The last one was a real bug. Only the Bedrock site computed
``turn_id`` — sites 1 and 3 silently dropped it, breaking the
dashboard's multi-turn-session grouping for every Anthropic-SSE and
OpenAI-via-backend request. The new ``RequestOutcome.from_stream``
classmethod computes it uniformly so the three finalizers cannot
drift apart on derivation logic again.
Each call site now hands ``from_stream`` the body + provider-specific
cache/timing fields and gets a fully-constructed outcome back. The
funnel call after it stays identical (``await
self._record_request_outcome(outcome)``).
|
||
|
|
96a674f1b9 |
feat(proxy): add client (harness) identification — per-harness analytics across every handler
Cashes in the RequestOutcome refactor with a typed-field surface that
gives EVERY handler per-harness visibility — Codex / Claude Code /
aider / Cursor / Zed / opencode / DROID / antigravity / etc. — for one
field-add across the contract.
The one-field-add proof
Headroom went from "what fraction of OUR requests come from which
harness?" being unanswerable (handlers logged ad-hoc User-Agent strings
in heterogeneous tag dicts at 18 sites, with 9 of 18 not even
populating them) to one structured ``client: str | None`` value on
every observation flowing through the funnel. No new bookkeeping at
call sites; every handler picks it up via a single
``classify_client(headers)`` call at request entry.
Implementation
* New ``CLIENT_UA_MAP`` + ``classify_client()`` in
``headroom/proxy/auth_mode.py``. Substring match against
User-Agent; ``X-Client`` header overrides UA. Returns ``str | None``
so ``None`` is the loud "unidentified" signal rather than a silent
empty bucket.
* New ``RequestOutcome.client: str | None = None`` field.
* Funnel updates (in ``outcome.py``):
- Appends ``client=X`` to the PERF log line ONLY when set, so
``headroom perf --client X`` parsing stays clean for
unidentified traffic (no bogus ``client=`` token).
- Copies ``client`` into ``RequestLog.tags["client"]`` so the
dashboard's existing tag-based filtering surfaces per-harness
slicing with zero new columns.
* Every handler that constructs a RequestOutcome now passes
``client=client`` — wired across streaming.py (3 finalizers,
with ``_finalize_stream_response`` gaining a new optional kwarg
since it doesn't have direct access to headers), anthropic.py
(6 sites), openai.py (8 sites including Codex WS), gemini.py
(2 emitting sites), batch.py (5 sites).
Harnesses recognised
Anthropic ecosystem: claude-code, claude-cli, claude-vscode,
anthropic-cli
OpenAI ecosystem: codex-cli
Editors: cursor, zed
AI coding harnesses: aider, droid, opencode, github-copilot
Other: antigravity (Google experimental)
Adding a new client is a one-line edit to ``CLIENT_UA_MAP``.
Tests
* 8 new tests in ``test_request_outcome.py`` covering:
- ``client`` field round-trips on the value type
- ``classify_client`` against every recognised UA prefix
- ``X-Client`` header override beats UA match
- ``None`` for unknown traffic (the loud signal)
- Funnel appends ``client=X`` to PERF when set
- Funnel OMITS ``client=`` from PERF when None (no bogus empty)
- Funnel stamps ``client`` into ``RequestLog.tags``
* All 228 existing tests still pass (full sweep across streaming,
cache, Codex, Anthropic, OpenAI, Gemini, batch, auth-mode).
* ruff + ruff-format + mypy clean.
What's now true that wasn't before
Once this lands, the dashboard can answer:
* "Show me cache hit rate by harness"
→ ``GROUP BY tags.client FROM request_log``
* "Which harness contributes the most cache writes?"
→ same
* "Per-harness savings ratio"
→ same
* ``headroom perf --client codex`` / ``--client claude-code``
→ analyzer filters PERF log lines on ``client=X`` token
Zero new bookkeeping in handlers. Zero changes to Prometheus label
cardinality (kept the client dimension out of Prometheus on purpose —
the tags route is the right surface). The "what's our traffic split
by harness?" question is now answerable in three places (PERF log,
RequestLog tags, dashboard widgets that already filter on tags)
without any per-provider work.
|
||
|
|
e898f68b89 |
refactor(proxy): introduce RequestOutcome funnel; collapse 3 streaming finalizers
P0 audit (docs/superpowers/specs/P0-proxy-pipeline-audit.md) catalogued **18 metrics.record_request call sites** across 4 handler files with **4 distinct argument shapes**: 9 of 18 omitted `cached=`, 7 of 18 omitted `attempted_input_tokens=` (= bug #454/#455's "headline 0%"), only 4 sites emitted a `PERF` log line (= bug #327's "msgs=0" sibling — Codex traffic invisible to `headroom perf`), and `cache_hit` was hardcoded `False` at 9 of 18 RequestLog sites. The cause was structural, not tactical: every site was independently deciding what "record this completed request" meant. This PR puts a single value type + a single function between the handlers and the metrics layer. Two new files: * `headroom/proxy/outcome.py` — `RequestOutcome` frozen dataclass. Captures everything we ever need to record about one completed request: identity, tokens, cache stats (per-TTL splits + inferred flag for OpenAI), timing, transforms, diagnostics. Provider-specific fields default to neutral values so non-Anthropic handlers don't have to know about 5m/1h splits, non-OpenAI handlers don't have to know about inferred writes, etc. Computed properties (`cache_hit`, `cache_hit_pct`, `savings_pct`) make "forgot to compute it" mistakes structurally impossible. * `HeadroomProxy._record_request_outcome` in `server.py` — the single funnel. Owns the four downstream effects in canonical order: 1. `metrics.record_request(...)` with the FULL kwarg set 2. `cost_tracker.record_tokens(...)` with `(model, tokens_saved, optimized_tokens)` positional + all cache kwargs 3. `logger.log(RequestLog(...))` with `cache_hit` correctly derived 4. structured `PERF` log line in the canonical key=value shape Migrated three streaming finalizers in this PR: * `_finalize_stream_response` (Anthropic native + OpenAI HTTP streaming) * `_stream_response_bedrock` (Bedrock-native Anthropic streaming) * `_stream_openai_via_backend` (OpenAI/Azure backend via LiteLLM/AnyLLM) All three previously had inline, drifted versions of the four-call sequence. Each is now ~70 fewer lines: build a `RequestOutcome` from local context, call `self._record_request_outcome(outcome)`. The prefix-tracker mutation (Anthropic-specific) stays outside the funnel — different concern. Six more migrations queued for follow-up PRs (handle_anthropic_messages 6 sites, handle_openai_chat, handle_openai_responses, handle_openai_ responses_ws 2 sites, handle_gemini_*, handle_databricks_invocations). Each is mechanical now. Tests * New: `tests/test_request_outcome.py` — 14 tests covering value-type contract (frozen, derived properties, neutral defaults) + funnel contract (full record_request kwargs, canonical record_tokens shape, derived cache_hit in RequestLog, PERF log key=value format, optional cost_tracker/logger). Bind the real production method via descriptor binding so the test exercises the real implementation, not a fork. * All 135 existing streaming/cache/Codex tests pass with zero regressions (`tests/test_backend_streaming_cache_metrics.py`, `test_proxy_streaming_request_logger.py`, `test_proxy_streaming_resilience.py`, `test_proxy_anthropic_cache_stability.py`, `test_openai_codex_*`, `test_responses_ws_pyo3_compression.py`, `test_anthropic_pre_upstream_backpressure.py`). * `mypy headroom/proxy/{outcome,server,handlers/streaming}.py` clean. * `ruff check` clean. Surface impact * −238 lines from `handlers/streaming.py` (deduplication). * +92 lines in `server.py` (the funnel — counted ONCE, not 18×). * +130 lines in new `outcome.py` (frozen dataclass + docstrings). * Net production code: ~−16 lines today, ~−500 lines after the remaining six migrations land. Forward design constraints (per docs/superpowers/specs/P0-proxy-pipeline-audit.md §7) * KISS: one value type, one function, no factory hierarchies. * No regex in routing — handlers stay provider-specific in their upstream contract. Output unification only. * No silent fallbacks — `cache_hit` is computed, not defaulted. `cache_inferred=True` is the loud signal when OpenAI write count came from `_infer_openai_cache_write_tokens`. * PERF format frozen so `headroom/perf/analyzer.py` keeps parsing cleanly; P3 follow-up replaces the free-text shape with a structured event. |