fix(proxy): show turn-hook savings in /stats extension_savings, incl. Codex WS (#4233)

## Description

### The test we ran

A Daytona sandbox running Headroom Enterprise 0.4.3 (core 0.39.1), with
Codex CLI 0.162.1 and Claude Code connected to real GitHub, Context7 and
Fetch MCP servers, working through the workshop tasks.

### What we found

- The tool-search plugin's turn hook rewrote the Codex `/v1/responses`
input and saved about **8K tokens per request**. `run_request_hooks`
booked that saving through `ctx.record_savings(hook.savings_source,
tokens=...)`.
- After many such requests, `GET /stats` still showed
**`extension_savings: {}`**.

There were two causes:

1. **`extension_savings` has no writer in core.** The only thing that
fills it is `PrometheusMetrics.record_extension_savings`, and core never
calls it. Attribution rows from `ctx.record_savings` /
`record_scope_savings` go to `savings_by_source`, which `/stats` shows
as `savings.by_source`.
2. **The Codex WebSocket path drops attribution entirely.** Both WS
compression calls (first frame and later frames) called
`_compress_openai_responses_payload_in_executor` without `savings_tags`.
So `TurnContext.tags` was a throwaway `{}`, and the hook's row (along
with the built-in tool-search deferral row) never reached
`savings.by_source` either. Codex uses the WS transport by default. The
HTTP `/v1/responses` path already passed `savings_tags`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- **`/stats.extension_savings`** now reports measured savings per
source: the **realized** attribution rows, plus
`record_extension_savings` totals. If a source also calls
`record_extension_savings` (`headroom-hdr` reports through both APIs on
every request), it shows the larger of its two totals rather than their
sum, so hdr is not double-counted. Nothing ties an attribution row to a
`record_extension_savings` call, so a plugin that used the two APIs on
different turns would be under-reported, never over-reported.
Unrealized/estimated rows (tool-search deferral, `realized=False`) stay
out and remain visible in `savings.by_source`.
- **Codex WS:** each frame compresses into its own `savings_tags` dict.
Its rows are merged into the session's pending list on the event loop,
and only when that frame was actually rewritten and sent. Each per-turn
`RequestOutcome` carries the pending rows and then clears them. A turn
whose `response.completed` has no input usage leaves its rows pending,
the same gate as its headline `saved_delta`
(`_deferrable_savings_delta`). Rows no turn claimed go out with the
session-end residual outcome, and only if the residual has input usage.
A fresh dict per frame matters because a timed-out compression thread
keeps running (`_run_compression_in_executor` can't cancel it), and it
must not book rows for a frame that went out uncompressed.

**Realized vs estimated:** `savings.by_source` already keeps one row per
`(source, realized)`, so realized and unrealized savings are shown
separately. `extension_savings` now carries only the realized ones.

**Review fix (@JerrettDavis, bounded WS pending list):** the
session-wide pending list grew one row per rewritten frame, while
`savings_attribution.from_tags` read only the first `MAX_SOURCES` (32)
rows. After 33 frames that each booked a 100-token `tool_search` row,
the handoff cleared all 33 rows but booked only 3,200 of the 3,300
tokens. `savings_attribution.accumulate(rows, items)` replaces the
`extend` at both WS frame paths.

**Bounded rows, nothing dropped (follow-up to @JerrettDavis's two
reviews):** the 32-row cap that silently dropped rows is gone from
`record_savings`, `from_tags` and `accumulate`. Storage is bounded by
coalescing plus one overflow row:
- every writer goes through the one `accumulate` helper: the per-request
ledger (`record_savings`, so every handler) and the WS pending list.
Rows with the same `(source, realized, estimated)` key are summed into
one row (`tokens`, `usd`, `events`). `details` are kept only while every
summed row has the same `details`.
- **cardinality bound:** a ledger keeps at most `MAX_NAMED_ROWS` (256)
named rows. A new source past that bound is summed into an `other` row
with the same `(realized, estimated)`, and its `details` are dropped.
Tokens, dollars and events are conserved: nothing is dropped, only the
source name. `tool_search` never folds, because
`tool_schema_savings_policy` finds the deferral's row by name to lower
it when hooks un-defer tools.
- **finite sums:** a row's `usd` saturates at ±`MAX_ROW_USD` (1e15) and
its `tokens` stay within 0..`MAX_ROW_TOKENS` (10^15). A row that was
clipped, or summed from one that was, carries `saturated: true`. Every
total stays finite, so the Prometheus exposition and
`json.dumps(allow_nan=False)` never see `inf`/`nan`.
- each row carries `events`: how many recordings it stands for (1 on a
fresh row, summed on merge). `PrometheusMetrics.record_request`
(`headroom_savings_attribution_events_total`), the OTel
`headroom.savings.attribution.events` counter and the perf analyzer's
"Savings by Source" add that count instead of 1 per row, so 33 coalesced
frames are 33 events, not 1 (Devin's finding). A row without `events`
(older PERF logs, rows built by hand) counts as 1.
- a row that a reader edited in place (`reconcile_deferred_tokens`, the
tool-search plugin) is summed defensively, so a junk value counts as 0
instead of raising inside the handler. The core deferral and the native
deferral hook both book `tool_search` as estimated, so they now share
one row, and the reconcile debits that row; totals are unchanged.
- `MAX_SOURCES` is no longer used. No importer was found outside this
repo's own test, but `savings_attribution` is imported by plugins, so
the name is kept behind a module `__getattr__` that raises a
`DeprecationWarning`.
- **Scope of the bound:** it is per ledger (one request, or one WS
session's pending list). The process-wide
`PrometheusMetrics.savings_by_source` table has no source cap yet; that
is a separate PR.

### Before / after

| | origin/main | this PR |
|---|---|---|
| A turn hook books 8,000 tokens on 2 requests →
`/stats.extension_savings` | `{}` | `{"tool_search": 16000}` |
| An hdr-style plugin calls both APIs (300 tokens × 2) | `{"hdr": 600}`
| `{"hdr": 600}` (not 1200) |
| A Codex WS turn with a folding hook → attribution rows on the turn
outcome | `[]` | `[("ws_fold", realized=True)]` |
| 33 WS frames, each booking `tool_search` 100 tokens, then one billed
turn → booked `tool_search` tokens (helper-level, the reviewer's
scenario) | 3,200 of 3,300, 33 rows stored | 3,300, 1 row stored |
| Real WS handler, 33 frames each with a folding hook (`ws_fold` row)
plus a 100-token hook row → non-`ws_fold` tokens booked | 1,600 of 3,300
(66 rows stored) | 3,300 |
| Same, repeated source → rows stored / hook events in
`headroom_savings_attribution_events_total` | aecae31b: 2 rows / 1 event
| 2 rows / 33 events |
| Same, 33 distinct sources → rows stored / hook events / `other` row |
aecae31b: 29 rows / 28 events / yes (6 sources folded) | 34 rows (33 +
`ws_fold`) / 33 events / no |
| Same, 300 distinct sources → rows stored / tokens and USD booked |
rows past 32 dropped | 257 rows (256 named + `other`), every token and
dollar booked |
| One ledger, 2,000 distinct sources plus repeats | rows past 32 dropped
| 256 named rows + 2 `other` rows (realized, estimated); tokens, USD and
events conserved |
| Same source booked 1e308 USD twice, then 1.0 | `inf` in `/stats` and
Prometheus | `usd` = 1e15, `saturated: true`, exposition finite |
| One HTTP request, 40 distinct sources + 1 repeat, 100 tokens each
(`record_savings`, every handler) → booked tokens / events | 3,200 of
4,100 / 32 (8 rows dropped) | 4,100 / 41, 40 rows stored |

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

-
`tests/test_compression_observability.py::test_turn_hook_savings_surface_in_stats_extension_savings`:
fails on origin/main, passes here.
-
`tests/test_openai_codex_ws_lifecycle.py::test_ws_turn_hook_savings_reach_the_turn_outcome[billed|usage_less]`:
drives the real WS handler with a registered hook. `billed` fails on
origin/main (no rows). `usage_less` fails if the input-usage gate is
removed.
- Related suites pass: the WS lifecycle, per-frame memory, savings
deferral and compression scheduler tests, extension attribution,
responses context compaction, tool-search repair after hooks, and the
handler outcome tag invariant (136 passed).
- ruff check, ruff format --check and mypy pass.
-
`tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked`
(review fix, renamed from `..._beyond_max_sources_...`), parametrized
over a repeated source × 33 frames, 33 distinct sources, and 300
distinct sources (folds into `other`, 257 rows, USD conserved): drives
the real WS handler through the rewritten frames. The upstream holds its
events until all 33 frames arrive, so every row is pending when the one
billed outcome claims it. The rows are booked through the real
`from_tags` → `PrometheusMetrics.record_request`. The test asserts that
every token is booked (3,300), that the hook rows count 33 events, that
there is no `other` row, and that the stored row count equals the
distinct keys (2 for a repeated source, 34 for 33 distinct sources plus
`ws_fold`).
-
`tests/test_extension_attribution.py::test_a_request_with_40_sources_books_every_token`:
one request's ledger with 40 distinct sources and one repeat books 4,100
tokens and 41 events through `PrometheusMetrics.record_request`.
-
`tests/test_extension_attribution.py::test_rows_coalesce_per_source_realized_and_estimated`:
same source with different `realized`/`estimated` stays in separate
rows; same key sums tokens and events.
-
`tests/test_observability_metrics.py::test_coalesced_savings_rows_count_every_event`:
the OTel events counter adds a row's `events` (33), and a row without
the field counts as 1.
-
`tests/test_cli_perf_format.py::test_coalesced_savings_rows_count_every_event_in_text_and_summary`:
both perf folds (`headroom perf` text report and the JSON `by_source`
summary) add a row's `events`.
-
`tests/test_extension_attribution.py::test_merging_into_a_junk_row_does_not_raise_inside_the_handler`:
a ledger row edited in place with junk (`tokens=None`) is summed as 0
instead of raising out of `record_savings`.
-
`tests/test_extension_attribution.py::test_distinct_sources_past_the_bound_fold_into_other_and_conserve_totals`:
2,000 sources plus estimated repeats give 258 rows, with tokens, USD and
events conserved.
-
`tests/test_extension_attribution.py::test_same_source_usd_saturates_instead_of_overflowing`,
`test_an_infinite_total_already_in_the_ledger_stays_saturated[±1.0]`,
`test_tokens_stay_within_zero_and_the_bound`: sums stay finite and are
flagged `saturated`.
-
`tests/test_extension_attribution.py::test_tool_search_keeps_its_own_row_past_the_bound`:
the deferral row is never folded, so the reconcile still finds it.
-
`tests/test_extension_attribution.py::test_max_sources_is_deprecated_not_removed`:
the old name still imports, with a `DeprecationWarning`.
-
`tests/test_turn_hooks.py::test_request_runner_attributes_savings_with_handler_counters`:
expected row now includes `events: 1`.

### Test Output

```text
# review fix, savings_attribution.py + openai.py reverted to the previous commit, new test:
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_max_sources_are_all_booked[repeated_source]   (assert 1600 == (100 * 33))
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_max_sources_are_all_booked[distinct_sources]  (assert 1600 == (100 * 33))
# helper-level, the reviewer's scenario (33 frames x tool_search 100):
append:     33 rows stored, 3200 booked
accumulate:  1 row stored,  3300 booked
40 distinct sources x 100: 29 rows stored, 4000 booked ("other" = 1200)
# no-cap follow-up, the 5 source files reverted to aecae31b, new/updated tests:
FAILED tests/test_extension_attribution.py::test_a_request_with_40_sources_books_every_token
FAILED tests/test_extension_attribution.py::test_rows_coalesce_per_source_realized_and_estimated
FAILED tests/test_observability_metrics.py::test_coalesced_savings_rows_count_every_event
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked[repeated_source]
FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_pending_savings_beyond_32_rows_are_all_booked[distinct_sources]
# with the no-cap follow-up: every test file that touches the savings ledger (21 files)
# plus gateway, perf format/per-port logs and /stats suites: 831 passed, 5 skipped (90d5e132)
# with the fix (aecae31b):
WS lifecycle, per-frame memory, savings deferral, compression scheduler, WS timings, pyo3 WS compression,
outcome tag invariant, extension attribution, compression observability, tool-search aggregation,
responses context compaction, proxy openai: 189 passed
ruff 0.17.0 check + format --check: clean on changed files
mypy headroom --ignore-missing-imports: no new errors (1 pre-existing local error in integrations/langchain/chat_model.py:380, untouched)
```

## Real Behavior Proof

- Environment: macOS 26.7, Python 3.12.13, this branch. The WS handler
is driven through the repo's fake-upstream harness
(`tests/test_openai_codex_ws_lifecycle.py`), which runs the real
`handle_openai_responses_ws`, `run_request_hooks` and outcome funnel
with only the upstream socket scripted; `/stats` runs on the real app
via `TestClient`; OTel uses the real `HeadroomOtelMetrics` with an
in-memory reader.
- Exact command / steps: `pytest tests/test_openai_codex_ws_lifecycle.py
tests/test_extension_attribution.py tests/test_observability_metrics.py
tests/test_compression_observability.py tests/test_turn_hooks.py` (plus
the other ledger suites listed under Testing), each run once on this
branch and once with the source files reverted.
- Observed result: with a registered folding hook, the hook really
rewrites the forwarded `input` (the upstream received only `"new"`) and
the turn outcome carries `[("ws_fold", realized=True, tokens>0)]`.
`/stats.extension_savings` reports `{"tool_search": 16000, "hdr": 600}`
for the two-request scenario above. 33 rewritten WS frames book 3,300
hook tokens and 33 events with no `other` row; 300 distinct sources fold
into 256 named rows plus `other` with every token and dollar booked; one
request with 40 distinct sources plus a repeat books 4,100 tokens and 41
events in 40 rows.
- Not tested: a real Codex CLI session against OpenAI on this branch, a
real Codex session with more than 32 rewritten frames before a billed
turn, and the real `headroom-tool-search` plugin were not re-run after
the patch. The sandbox numbers above (about 8K tokens/request,
`extension_savings: {}`) are from 0.39.1 before the fix.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: `/stats.extension_savings` now
includes realized attribution rows, and Codex WS turns now carry
attribution rows. Request bodies and headline totals are unchanged.
- Kill switch / disable path: none needed; the change only affects
reporting
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- plus the code-reviewer agent on each review fix. Its earlier SHOULD on
the collapsed `events` count is now fixed (rows carry `events`).
- [x] This PR is ready for human review

## Additional Notes

- `record_extension_savings` stays public and unchanged.
- The Prometheus exposition format is unchanged;
`headroom_savings_attribution_events_total` now counts recordings, which
is what it counted before rows were coalesced.
- Ledger rows now carry an `events` field. Readers that ignore unknown
keys are unaffected.
- A reader that finds "rows a hook added" by slicing the ledger at its
old length (`ledger[before:]`) can miss a hook's tokens when the hook
books a key that already had a row. Core has no such reader;
`headroom-kiro`'s handler does and should switch to a before/after sum.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-17 11:29:04 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 2912b7b620
commit 1d112cd36c
14 changed files with 625 additions and 17 deletions
+2 -1
View File
@@ -14,6 +14,7 @@ from opentelemetry.metrics import CallbackOptions, Observation
from headroom._version import get_version
from headroom.offline import guard_egress
from headroom.proxy.savings_attribution import event_count
logger = logging.getLogger(__name__)
@@ -558,7 +559,7 @@ class HeadroomOtelMetrics:
realized=bool(item.get("realized", True)),
estimated=bool(item.get("estimated", False)),
)
self._savings_attribution_events.add(1, attrs)
self._savings_attribution_events.add(event_count(item), attrs)
saved = max(0, int(item.get("tokens", 0) or 0))
if saved:
self._savings_attributed_tokens.add(saved, attrs)
+6 -2
View File
@@ -813,13 +813,15 @@ def format_report(report: PerfReport) -> str:
# exactly zero tokens; every token-savings channel in the proxy would record
# it as nothing. Showing `$` beside a `0 tokens` row is the honest rendering
# of that, and collapsing them into one number would be an invented saving.
from headroom.proxy.savings_attribution import event_count
by_source: dict[tuple[str, bool], dict[str, float]] = {}
for record in report.perf_records:
for item in getattr(record, "savings_breakdown", ()) or ():
source = str(item.get("source") or "other")
realized = bool(item.get("realized", True))
row = by_source.setdefault((source, realized), {"events": 0, "tokens": 0, "usd": 0.0})
row["events"] += 1
row["events"] += event_count(item)
row["tokens"] += max(0, _as_number(item.get("tokens"), int))
row["usd"] += _as_number(item.get("usd"), float)
if by_source:
@@ -1356,6 +1358,8 @@ def build_perf_summary(report: PerfReport) -> dict:
}
)
from headroom.proxy.savings_attribution import event_count
by_source_groups: dict[tuple[str, bool], dict[str, int | float | str | bool]] = {}
for record in records:
for item in record.savings_breakdown:
@@ -1372,7 +1376,7 @@ def build_perf_summary(report: PerfReport) -> dict:
"usd": 0.0,
},
)
row["events"] = int(row["events"]) + 1
row["events"] = int(row["events"]) + event_count(item)
raw_tokens = item.get("tokens", 0)
raw_usd = item.get("usd", 0.0)
tokens = int(raw_tokens) if isinstance(raw_tokens, (str, int, float)) else 0
+31 -1
View File
@@ -46,6 +46,9 @@ from headroom.proxy.provider_usage import (
)
from headroom.proxy.rate_limit_identity import rate_limit_identity
from headroom.proxy.savings import ItemMeasure, get_item_ledger, prefix_key, responses_items
from headroom.proxy.savings_attribution import SAVINGS_ATTRIBUTION_TAG
from headroom.proxy.savings_attribution import accumulate as accumulate_savings
from headroom.proxy.savings_attribution import from_tags as savings_from_tags
from headroom.proxy.semantic_cache_key_policy import compute_request_cache_partition
from headroom.proxy.stage_timer import StageTimer, emit_stage_timings_log
from headroom.proxy.tool_schema_savings_policy import (
@@ -7182,6 +7185,12 @@ class OpenAIHandlerMixin:
# handshake. Returns `{}` when no tags are present.
_extract_ws_tags = getattr(self, "_extract_tags", None)
ws_tags = _extract_ws_tags(ws_headers) if callable(_extract_ws_tags) else {}
# Savings attribution rows (turn hooks, tool-search deferral) from frames
# that were rewritten and sent, waiting for the next turn's outcome. Each
# frame compresses into its own dict: a timed-out compression thread
# keeps running and must not book rows for a frame sent uncompressed.
# Coalesced per source by ``accumulate_savings``, so it stays small.
ws_turn_savings: list[dict[str, Any]] = []
# Extract subprotocol from client — this is an application-level negotiation
# that MUST be forwarded end-to-end (unlike sec-websocket-key which is per-connection).
@@ -8193,6 +8202,7 @@ class OpenAIHandlerMixin:
)
_record_ws_compression_overhead(_preflight_ms)
_compression_started = time.perf_counter()
_frame_savings: dict[str, Any] = {}
try:
(
_new_inner,
@@ -8212,6 +8222,7 @@ class OpenAIHandlerMixin:
if client == "codex"
else COMPRESSION_TIMEOUT_SECONDS,
client=client,
savings_tags=_frame_savings,
)
for _timing_name, _timing_ms in _ws_compression_timing.items():
_record_ws_compression_timing(_timing_name, _timing_ms)
@@ -8260,6 +8271,9 @@ class OpenAIHandlerMixin:
)
_record_ws_compression_overhead(_rewrite_ms)
tokens_saved += int(_ws_saved)
accumulate_savings(
ws_turn_savings, savings_from_tags(_frame_savings)
)
ws_conversation_tokens_saved = int(_ws_saved)
ws_conversation_key = savings_conversation_key(
_send_body, session_id=f"ws:{session_id}"
@@ -8586,6 +8600,7 @@ class OpenAIHandlerMixin:
)
_record_ws_compression_overhead(_preflight_ms)
_compression_started = time.perf_counter()
frame_savings: dict[str, Any] = {}
try:
(
new_inner,
@@ -8605,6 +8620,7 @@ class OpenAIHandlerMixin:
if client == "codex"
else COMPRESSION_TIMEOUT_SECONDS,
client=client,
savings_tags=frame_savings,
)
for _timing_name, _timing_ms in frame_compression_timing.items():
_record_ws_compression_timing(_timing_name, _timing_ms)
@@ -8754,6 +8770,7 @@ class OpenAIHandlerMixin:
)
_record_ws_compression_overhead(_rewrite_ms)
tokens_saved += int(frame_saved)
accumulate_savings(ws_turn_savings, savings_from_tags(frame_savings))
ws_conversation_tokens_saved = int(frame_saved)
ws_conversation_key = savings_conversation_key(
new_inner, session_id=f"ws:{session_id}"
@@ -9110,6 +9127,15 @@ class OpenAIHandlerMixin:
# at the WS upgrade) so dashboards can
# slice WS turns by tag — same surface
# as HTTP turns.
# Same gate as ``saved_delta``: a usage-less turn
# leaves its rows pending for a usage-carrying one.
turn_tags = ws_tags
if ws_turn_savings and input_delta > 0:
turn_tags = {
**ws_tags,
SAVINGS_ATTRIBUTION_TAG: list(ws_turn_savings),
}
ws_turn_savings.clear()
await self._record_request_outcome(
RequestOutcome(
# Per-emission ids keep dashboard request-log keys unique.
@@ -9140,7 +9166,7 @@ class OpenAIHandlerMixin:
)
if isinstance(body, dict)
else 0,
tags=ws_tags,
tags=turn_tags,
conversation_key=ws_conversation_key,
conversation_tokens_saved=ws_conversation_tokens_saved,
client=client,
@@ -9755,6 +9781,10 @@ class OpenAIHandlerMixin:
"cache_write_tokens": str(ws_cache_write_tokens_total),
"uncached_input_tokens": str(ws_uncached_input_tokens_total),
}
if ws_turn_savings and residual_input_tokens > 0:
# Rows no per-turn record claimed ride with the residual, under
# the same input-usage gate as ``residual_tokens_saved``.
ws_session_tags[SAVINGS_ATTRIBUTION_TAG] = list(ws_turn_savings)
if (
residual_input_tokens > 0
or residual_output_tokens > 0
+20 -1
View File
@@ -22,6 +22,7 @@ if TYPE_CHECKING:
from headroom import savings_ledger
from headroom.observability import get_otel_metrics
from headroom.proxy.savings_attribution import event_count
from headroom.proxy.savings_tracker import SavingsTracker, estimate_request_savings_usd
logger = logging.getLogger("headroom.proxy")
@@ -654,6 +655,24 @@ class PrometheusMetrics:
if saved > 0:
self.extension_savings[key] += saved
def extension_savings_by_source(self) -> dict[str, int]:
"""Measured savings per source, as ``/stats`` reports ``extension_savings``.
Realized savings attribution rows (``ctx.record_savings`` from a turn
hook, ``record_scope_savings``) plus ``record_extension_savings`` totals.
A source reported through both APIs (``headroom-hdr`` does, per request)
shows the larger of the two totals rather than their sum: nothing ties an
attribution row to a ``record_extension_savings`` call, so summing would
double-count it. Estimated rows (``realized=False``) stay in
``savings.by_source`` only.
"""
out = dict(self.extension_savings)
for row in list(self.savings_by_source.values()):
source, tokens = str(row["source"]), int(row["tokens"])
if row["realized"] and tokens > out.get(source, 0):
out[source] = tokens
return out
def record_compression_failed(self, reason: str) -> None:
"""Record one fail-open compression failure, bucketed by ``reason``.
@@ -970,7 +989,7 @@ class PrometheusMetrics:
"usd": 0.0,
},
)
row["events"] = int(row["events"]) + 1
row["events"] = int(row["events"]) + event_count(item)
row["tokens"] = int(row["tokens"]) + max(0, int(item.get("tokens", 0) or 0))
row["usd"] = round(
float(row["usd"]) + float(item.get("usd", 0.0) or 0.0),
+127 -7
View File
@@ -6,12 +6,12 @@ import base64
import json
import math
import re
from collections.abc import MutableMapping
import warnings
from collections.abc import Iterable, MutableMapping
from typing import Any
SAVINGS_ATTRIBUTION_TAG = "_headroom_savings_attribution"
_NAME_RE = re.compile(r"[^a-z0-9_.-]+")
MAX_SOURCES = 32
_SCOPE_KEY = "headroom_savings_attribution"
# Per-request stage timings contributed from outside the handler, merged into
@@ -51,6 +51,22 @@ STAGE_PREFIX = "ext:"
# (16 stages x 1h is nowhere near the float ceiling).
MAX_STAGE_MS = 3_600_000.0
# Source names are extension-supplied, so a ledger keeps at most this many
# rows by name. A source past it folds into ``other`` -- one row per
# (realized, estimated) -- so storage and the per-insert scan stay bounded
# while every token, dollar and event is still booked.
MAX_NAMED_ROWS = 256
OVERFLOW_SOURCE = "other"
# Core-booked sources never fold: ``tool_schema_savings_policy`` finds the
# deferral's row by name to lower it when hooks un-defer tools.
_UNFOLDED_SOURCES = frozenset({"tool_search"})
# A row's sums saturate here instead of overflowing to ``inf`` (see
# MAX_STAGE_MS for what a non-finite value does downstream). Both are far
# beyond any real saving; a saturated row is flagged ``saturated``.
MAX_ROW_USD = 1e15
MAX_ROW_TOKENS = 10**15
def _source_name(value: object) -> str:
name = _NAME_RE.sub("_", str(value or "other").strip().lower()).strip("_.-")
@@ -164,9 +180,6 @@ def record_savings(
details: dict[str, Any] | None = None,
) -> None:
"""Attribute savings to a source; this never changes headline totals."""
ledger = _ledger(tags)
if len(ledger) >= MAX_SOURCES:
return
# Same hazard as MAX_STAGE_MS, on the amounts rather than the durations:
# ``usd=inf`` reaches ``/stats`` and raises out of the JSON encoder, and
# ``int(inf)`` raises OverflowError right here, inside the handler, on a
@@ -193,14 +206,106 @@ def record_savings(
for key, value in list(details.items())[:12]
if isinstance(value, (str, int, float, bool)) or value is None
}
ledger.append(item)
accumulate(_ledger(tags), [item])
def from_tags(tags: MutableMapping[str, Any] | None) -> list[dict[str, Any]]:
raw = (tags or {}).get(SAVINGS_ATTRIBUTION_TAG)
if not isinstance(raw, list):
return []
return [dict(item) for item in raw[:MAX_SOURCES] if isinstance(item, dict)]
return [dict(item) for item in raw if isinstance(item, dict)]
def event_count(row: dict[str, Any]) -> int:
"""How many recordings a row stands for; a row without ``events`` is one."""
try:
return max(1, int(row.get("events", 1)))
except (TypeError, ValueError, OverflowError):
return 1
def accumulate(rows: list[dict[str, Any]], items: Iterable[dict[str, Any]]) -> None:
"""Add ``items`` to ``rows``, one row per (source, realized, estimated).
Every writer goes through here -- the per-request ledger and a WS session's
pending rows alike -- and no amount is ever dropped. Same-key rows sum
``tokens``, ``usd`` and ``events``; ``details`` stay only while every
summed row agrees. Once ``rows`` holds ``MAX_NAMED_ROWS`` rows, a new
source folds into ``OVERFLOW_SOURCE`` (which also takes unnamed savings),
so the list grows by at most the overflow and ``_UNFOLDED_SOURCES`` keys.
"""
for item in items:
key = _key(item)
target = _row_for(rows, key)
if (
target is None
and key[0] not in _UNFOLDED_SOURCES
and _row_count(rows) >= MAX_NAMED_ROWS
):
if key[0] != OVERFLOW_SOURCE:
# Details describe one source; the overflow row mixes several.
item = {k: v for k, v in item.items() if k != "details"}
key = (OVERFLOW_SOURCE, key[1], key[2])
target = _row_for(rows, key)
if target is None:
target = {**item, "source": key[0], "tokens": 0, "usd": 0.0, "events": 0}
rows.append(target)
else:
if target.get("details") != item.get("details"):
target.pop("details", None)
target["events"] = event_count(target)
_add_into(target, item)
def _add_into(target: dict[str, Any], item: dict[str, Any]) -> None:
"""Sum ``item`` into ``target``; a total that hit a bound stays flagged."""
tokens, tokens_clipped = _bounded(target.get("tokens"), item.get("tokens"), 0, MAX_ROW_TOKENS)
usd, usd_clipped = _bounded(target.get("usd"), item.get("usd"), -MAX_ROW_USD, MAX_ROW_USD)
target["tokens"] = int(tokens)
target["usd"] = round(usd, 12)
target["events"] += event_count(item)
if tokens_clipped or usd_clipped or item.get("saturated"):
target["saturated"] = True
def _bounded(current: Any, added: Any, low: float, high: float) -> tuple[float, bool]:
"""``current + added`` held inside ``[low, high]``, and whether it was clipped."""
a, a_clipped = _operand(current, low, high)
b, b_clipped = _operand(added, low, high)
total = max(low, min(a + b, high))
return total, a_clipped or b_clipped or total != a + b
def _operand(value: Any, low: float, high: float) -> tuple[float, bool]:
# Rows are plain dicts a reader may edit (deferral reconcile, plugins), so
# an operand can be junk; summing must not raise inside the handler. A
# non-finite or oversized operand reads as the bound it passed, never as
# zero, so a saturated total is not reset by the next update.
try:
number = float(value or 0)
except OverflowError: # an int beyond float range
number = math.inf if value > 0 else -math.inf
except (TypeError, ValueError):
return 0.0, False
if math.isnan(number):
return 0.0, False
bounded = max(low, min(number, high))
return bounded, bounded != number
def _row_count(rows: list[Any]) -> int:
return sum(1 for row in rows if isinstance(row, dict))
def _key(row: dict[str, Any]) -> tuple[str, bool, bool]:
# ``realized`` defaults to True, as in every consumer of these rows.
return (str(row.get("source")), bool(row.get("realized", True)), bool(row.get("estimated")))
def _row_for(rows: list[dict[str, Any]], key: tuple[str, bool, bool]) -> dict[str, Any] | None:
# The ledger is a plain list reachable through ``tags``; skip what this
# module did not write rather than raise inside the handler.
return next((row for row in rows if isinstance(row, dict) and _key(row) == key), None)
_INTERNAL_TAGS = frozenset({SAVINGS_ATTRIBUTION_TAG, STAGE_TIMING_TAG})
@@ -246,3 +351,18 @@ def decode(value: str) -> list[dict[str, Any]]:
if isinstance(decoded, list)
else []
)
def __getattr__(name: str) -> Any:
# MAX_SOURCES used to cap a request's ledger at 32 rows and silently drop
# the rest; rows now coalesce per source, and sources past MAX_NAMED_ROWS
# fold into OVERFLOW_SOURCE. Kept for importers.
if name == "MAX_SOURCES":
warnings.warn(
"headroom.proxy.savings_attribution.MAX_SOURCES is unused and will be "
"removed; rows coalesce per source, past MAX_NAMED_ROWS into 'other'.",
DeprecationWarning,
stacklevel=2,
)
return 32
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+1 -1
View File
@@ -5276,7 +5276,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
else {},
"compressions_by_strategy": dict(m.compressions_by_strategy),
"tokens_saved_by_strategy": dict(m.tokens_saved_by_strategy),
"extension_savings": dict(m.extension_savings),
"extension_savings": m.extension_savings_by_source(),
"codex_ws": {
"units_total": m.codex_ws_units_total,
"units_modified_total": m.codex_ws_units_modified_total,
+4 -3
View File
@@ -84,9 +84,10 @@ def deferred_booking(tags: object, tools: object) -> DeferredBooking | None:
``None`` when nothing is booked. The schemas are copied as they were when
booked, so a later in-place edit cannot change what the credit stands for.
The ledger entry is the latest ``tool_search`` one marked ``estimated``:
both the core deferral and the native deferral hook book that way, and a
hook's own measured saving never is, so it is never the one debited.
The ledger entry is the ``tool_search`` row marked ``estimated``: both the
core deferral and the native deferral hook book that way (rows coalesce, so
they share it), and a hook's own measured saving never is, so it is never
the one debited.
"""
if not isinstance(tags, dict) or "tool_search_deferred_tokens" not in tags:
return None
+20
View File
@@ -578,3 +578,23 @@ def test_token_source_and_dollar_source_coexist(tmp_path, monkeypatch):
def test_no_section_when_nothing_attributed(tmp_path, monkeypatch):
out = _report_for([_savings_perf_line("none")], tmp_path, monkeypatch)
assert "Savings by Source" not in out
def test_coalesced_savings_rows_count_every_event_in_text_and_summary():
"""A PERF row summed from 3 recordings is 3 events; a row without the field is 1."""
report = PerfReport(
perf_records=[
PerfRecord(
timestamp="2026-06-05 10:00:00,000",
request_id="hr_1",
savings_breakdown=[
{"source": "tool_search", "tokens": 300, "events": 3},
{"source": "router", "tokens": 5},
],
)
]
)
by_source = {row["source"]: row["events"] for row in build_perf_summary(report)["by_source"]}
assert by_source == {"tool_search": 3, "router": 1}
assert "tool_search: 3 events" in analyzer.format_report(report)
+44
View File
@@ -318,6 +318,50 @@ def test_extension_savings_surface_in_stats(
}
@pytest.mark.asyncio
async def test_turn_hook_savings_surface_in_stats_extension_savings(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``ctx.record_savings`` from a turn hook must reach ``/stats.extension_savings``.
Before, only ``record_extension_savings`` filled the field and core never
calls it, so ``/stats`` showed ``{}`` while a hook saved ~8K tokens a request.
Estimated rows stay out, and a plugin reporting through both APIs (hdr) is
not counted twice.
"""
from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "proxy_savings.json"))
app = create_app(ProxyConfig(cache_enabled=False, rate_limit_enabled=False, log_requests=False))
with TestClient(app) as client:
metrics = app.state.proxy.metrics
for _ in range(2):
await metrics.record_request(
provider="openai",
model="gpt-5.4",
input_tokens=1000,
output_tokens=10,
tokens_saved=0,
latency_ms=1.0,
savings_attribution=[
{"source": "tool_search", "realized": True, "tokens": 8000},
{"source": "tool_search", "realized": False, "estimated": True, "tokens": 500},
{"source": "hdr", "realized": True, "tokens": 300},
],
)
metrics.record_extension_savings("hdr", 300)
stats = client.get("/stats").json()
assert stats["extension_savings"] == {"tool_search": 16000, "hdr": 600}
by_source = {
(row["source"], row["realized"]): row["tokens"] for row in stats["savings"]["by_source"]
}
assert by_source[("tool_search", False)] == 1000
def test_prometheus_metrics_accumulates_codex_ws_unit_and_frame_counters():
from headroom.proxy.prometheus_metrics import PrometheusMetrics
+154
View File
@@ -396,3 +396,157 @@ async def test_extension_savings_reach_the_metrics_call() -> None:
attribution = h.metrics.record_request.await_args.kwargs["savings_attribution"]
assert [(row["source"], row["usd"]) for row in attribution] == [("routemegood", 0.42)]
# --- no row cap: every recording is booked -------------------------------------
@pytest.mark.asyncio
async def test_a_request_with_40_sources_books_every_token() -> None:
"""The ledger used to stop at 32 rows and drop the 33rd recording on."""
from headroom.proxy.prometheus_metrics import PrometheusMetrics
from headroom.proxy.savings_attribution import record_savings
tags: dict = {}
for i in range(40):
record_savings(tags, f"plugin{i}", tokens=100, usd=0.01)
record_savings(tags, "plugin0", tokens=100, usd=0.01)
rows = from_tags(tags)
metrics = PrometheusMetrics()
await metrics.record_request(
provider="openai",
model="gpt-5.4",
input_tokens=10,
output_tokens=1,
tokens_saved=0,
latency_ms=1.0,
savings_attribution=rows,
)
assert len(rows) == 40
booked = metrics.savings_by_source.values()
assert sum(int(row["tokens"]) for row in booked) == 4_100
assert sum(int(row["events"]) for row in booked) == 41
assert metrics.savings_by_source["plugin0:1"]["events"] == 2
def test_rows_coalesce_per_source_realized_and_estimated() -> None:
from headroom.proxy.savings_attribution import record_savings
tags: dict = {}
for _ in range(3):
record_savings(tags, "router", tokens=1, realized=True)
record_savings(tags, "router", tokens=1, realized=False)
record_savings(tags, "router", tokens=1, estimated=True)
rows = from_tags(tags)
assert [(r["realized"], r["estimated"], r["tokens"], r["events"]) for r in rows] == [
(True, False, 3, 3),
(False, False, 3, 3),
(True, True, 3, 3),
]
def test_merging_into_a_junk_row_does_not_raise_inside_the_handler() -> None:
from headroom.proxy.savings_attribution import record_savings
tags: dict = {SAVINGS_ATTRIBUTION_TAG: [{"source": "x", "tokens": None, "usd": "junk"}]}
record_savings(tags, "x", tokens=5, usd=0.5)
assert [(r["tokens"], r["usd"], r["events"]) for r in from_tags(tags)] == [(5, 0.5, 2)]
def test_distinct_sources_past_the_bound_fold_into_other_and_conserve_totals() -> None:
"""Source names are plugin-supplied; their count must not grow the ledger."""
from headroom.proxy.savings_attribution import MAX_NAMED_ROWS, record_savings
tags: dict = {}
for i in range(2_000):
record_savings(tags, f"extension_{i}", tokens=100, usd=0.01)
record_savings(tags, f"extension_{i}", tokens=1, estimated=True)
record_savings(tags, "extension_0", tokens=100, usd=0.01) # a named row still merges
rows = from_tags(tags)
assert len(rows) == MAX_NAMED_ROWS + 2 # plus one ``other`` row per estimated flag
assert sum(r["tokens"] for r in rows) == 2_000 * 101 + 100
assert sum(r["events"] for r in rows) == 4_001
assert math.isclose(sum(r["usd"] for r in rows), 2_001 * 0.01)
named = {(r["source"], r["estimated"]): r for r in rows}
assert named[("extension_0", False)]["events"] == 2
assert {(r["source"], r["estimated"]) for r in rows if r["source"] == "other"} == {
("other", False),
("other", True),
}
@pytest.mark.asyncio
async def test_same_source_usd_saturates_instead_of_overflowing() -> None:
"""Two finite amounts must not sum to ``inf`` on the way to the metrics."""
import json
from headroom.proxy.prometheus_metrics import PrometheusMetrics
from headroom.proxy.savings_attribution import MAX_ROW_USD, record_savings
tags: dict = {}
record_savings(tags, "same", tokens=1, usd=1e308)
record_savings(tags, "same", tokens=1, usd=1e308)
record_savings(tags, "same", tokens=1, usd=1.0) # must not reset the total
rows = from_tags(tags)
assert [(r["tokens"], r["events"], r["usd"], r["saturated"]) for r in rows] == [
(3, 3, MAX_ROW_USD, True)
]
metrics = PrometheusMetrics()
await metrics.record_request(
provider="openai",
model="gpt-5.4",
input_tokens=10,
output_tokens=1,
tokens_saved=0,
latency_ms=1.0,
savings_attribution=rows,
)
booked = metrics.savings_by_source["same:1"]["usd"]
assert math.isfinite(booked)
json.dumps(metrics.savings_by_source, allow_nan=False) # what /stats does
@pytest.mark.parametrize("then_usd", [1.0, -1.0])
def test_an_infinite_total_already_in_the_ledger_stays_saturated(then_usd) -> None:
from headroom.proxy.savings_attribution import MAX_ROW_USD, record_savings
tags: dict = {SAVINGS_ATTRIBUTION_TAG: [{"source": "x", "usd": math.inf}]}
record_savings(tags, "x", usd=then_usd)
[row] = from_tags(tags)
assert row["saturated"] is True
assert row["usd"] >= MAX_ROW_USD - 1.0
def test_tokens_stay_within_zero_and_the_bound() -> None:
from headroom.proxy.savings_attribution import MAX_ROW_TOKENS, accumulate
rows: list = [{"source": "x", "tokens": -(10**400)}, {"source": "y", "tokens": 10**400}]
accumulate(rows, [{"source": "x", "tokens": 5}, {"source": "y", "tokens": 5}])
assert [(r["tokens"], r["saturated"]) for r in rows] == [(5, True), (MAX_ROW_TOKENS, True)]
def test_tool_search_keeps_its_own_row_past_the_bound() -> None:
"""Deferral reconcile finds the ``tool_search`` row by name to lower it."""
from headroom.proxy.savings_attribution import MAX_NAMED_ROWS, record_savings
tags: dict = {}
for i in range(MAX_NAMED_ROWS):
record_savings(tags, f"extension_{i}", tokens=1)
record_savings(tags, "tool_search", tokens=900, realized=False, estimated=True)
assert [r["tokens"] for r in from_tags(tags) if r["source"] == "tool_search"] == [900]
def test_max_sources_is_deprecated_not_removed() -> None:
from headroom.proxy import savings_attribution
with pytest.warns(DeprecationWarning, match="MAX_SOURCES"):
assert savings_attribution.MAX_SOURCES == 32
+17
View File
@@ -262,6 +262,23 @@ def test_request_attribute_provider_enriches_core_and_savings_metrics() -> None:
unregister_otel_metric_attribute_provider(identity_attributes)
def test_coalesced_savings_rows_count_every_event() -> None:
"""A row summed from 33 recordings is 33 events, not 1; a bare row is 1."""
reader = InMemoryMetricReader()
otel_metrics = HeadroomOtelMetrics(meter_provider=MeterProvider(metric_readers=[reader]))
otel_metrics.record_savings_attribution(
[
{"source": "tool_search", "tokens": 3_300, "events": 33},
{"source": "router", "tokens": 5},
]
)
events = _collect_metrics(reader)["headroom.savings.attribution.events"]
assert _find_point(events, source="tool_search").value == 33
assert _find_point(events, source="router").value == 1
def test_failing_request_attribute_provider_is_fail_open() -> None:
reader = InMemoryMetricReader()
provider = MeterProvider(metric_readers=[reader])
+195
View File
@@ -11,6 +11,7 @@ import asyncio
import contextlib
import json
import logging
import math
import sys
from pathlib import Path
from types import SimpleNamespace
@@ -2372,3 +2373,197 @@ async def test_ws_session_metrics_track_model_per_response_create():
"model-a",
"model-b",
]
@pytest.mark.asyncio
@pytest.mark.parametrize(
("usage", "expected"),
[({"input_tokens": 10, "output_tokens": 1}, [("ws_fold", True)]), (None, [])],
ids=["billed", "usage_less"],
)
async def test_ws_turn_hook_savings_reach_the_turn_outcome(usage, expected):
"""A turn hook's savings on a Codex WS turn must reach that turn's outcome.
The WS path compressed without ``savings_tags``, so ``ctx.record_savings``
wrote into a throwaway dict and ``/stats`` never saw the attribution. A turn
with no input usage books nothing, the same gate as its headline savings.
"""
from headroom.proxy.savings_attribution import from_tags
from headroom.proxy.turn_hooks import clear_turn_hooks, register_turn_hook
class _DropOldInput:
name = "drop_old_input"
savings_source = "ws_fold"
stream_safe = True
def on_request(self, ctx: SimpleNamespace) -> None:
del ctx.messages[:-1]
first = json.dumps(
{
"type": "response.create",
"response": {
"model": "gpt-5.4",
"input": [
{"type": "message", "role": "user", "content": "old " * 50},
{"type": "message", "role": "user", "content": "new"},
],
},
}
)
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps(
{
"type": "response.completed",
"response": {"id": "r_1", **({"usage": usage} if usage else {})},
}
),
]
)
client_ws = _FakeWebSocket(frames=[first])
handler = _DummyOpenAIHandler()
handler.config.optimize = True
outcomes = []
async def _record_request_outcome(outcome):
outcomes.append(outcome)
handler._record_request_outcome = _record_request_outcome
clear_turn_hooks()
register_turn_hook(_DropOldInput())
try:
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
finally:
clear_turn_hooks()
sent = json.loads(upstream.sent[0])["response"]["input"]
assert [item["content"] for item in sent] == ["new"] # the hook really ran
rows = [row for outcome in outcomes for row in from_tags(outcome.tags)]
assert [(row["source"], row["realized"]) for row in rows] == expected
assert all(row["tokens"] > 0 for row in rows)
class _UpstreamAfterSends(_FakeUpstream):
"""Streams its events only once ``sends`` frames reached it, so every frame's
savings are pending before the one usage-carrying outcome claims them."""
def __init__(self, events: list[str], *, sends: int) -> None:
super().__init__(events)
self._sends = sends
async def _iter(self):
while len(self.sent) < self._sends:
await asyncio.sleep(0)
for ev in self._events:
yield ev
@pytest.mark.asyncio
@pytest.mark.parametrize(
("sources_per_frame", "frames_n", "expected_rows", "overflowed"),
[
(lambda i: "tool_search", 33, 2, False),
(lambda i: f"src{i}", 33, 34, False),
# ws_fold + 255 named sources fill MAX_NAMED_ROWS; the other 45 share one row.
(lambda i: f"src{i}", 300, 257, True),
],
ids=["repeated_source", "distinct_sources", "distinct_sources_past_the_bound"],
)
async def test_ws_pending_savings_beyond_32_rows_are_all_booked(
sources_per_frame, frames_n, expected_rows, overflowed
):
"""Pending frames hand every token and every event to the billed turn.
Rows coalesce to one per (source, realized, estimated) key; distinct
sources past ``MAX_NAMED_ROWS`` fold into ``other`` rather than growing
the session's pending list without bound.
"""
from headroom.proxy.prometheus_metrics import PrometheusMetrics
from headroom.proxy.savings_attribution import SAVINGS_ATTRIBUTION_TAG, from_tags
from headroom.proxy.turn_hooks import clear_turn_hooks, register_turn_hook
seen = {"frame": 0}
class _DeferTools:
name = "defer_tools"
savings_source = "ws_fold"
stream_safe = True
def on_request(self, ctx: SimpleNamespace) -> None:
del ctx.messages[:-1]
ctx.record_savings(sources_per_frame(seen["frame"]), tokens=100, usd=0.01)
seen["frame"] += 1
def _frame(i: int) -> str:
return json.dumps(
{
"type": "response.create",
"response": {
"model": "gpt-5.4",
"input": [
{"type": "message", "role": "user", "content": f"old {i} " * 50},
{"type": "message", "role": "user", "content": f"new {i}"},
],
},
}
)
upstream = _UpstreamAfterSends(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps(
{
"type": "response.completed",
"response": {
"id": "r_1",
"usage": {"input_tokens": 10, "output_tokens": 1},
},
}
),
],
sends=frames_n,
)
client_ws = _FakeWebSocket(frames=[_frame(i) for i in range(frames_n)])
handler = _DummyOpenAIHandler()
handler.config.optimize = True
outcomes = []
async def _record_request_outcome(outcome):
outcomes.append(outcome)
handler._record_request_outcome = _record_request_outcome
clear_turn_hooks()
register_turn_hook(_DeferTools())
try:
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(client_ws)
finally:
clear_turn_hooks()
assert len(upstream.sent) == frames_n
assert seen["frame"] == frames_n
metrics = PrometheusMetrics()
stored = []
for outcome in outcomes:
rows = from_tags(outcome.tags)
stored.append(len(outcome.tags.get(SAVINGS_ATTRIBUTION_TAG, [])))
await metrics.record_request(
provider="openai",
model="gpt-5.4",
input_tokens=10,
output_tokens=1,
tokens_saved=0,
latency_ms=1.0,
savings_attribution=rows,
)
hook_rows = [row for row in metrics.savings_by_source.values() if row["source"] != "ws_fold"]
assert sum(int(row["tokens"]) for row in hook_rows) == 100 * frames_n
assert sum(int(row["events"]) for row in hook_rows) == frames_n
assert math.isclose(sum(float(row["usd"]) for row in hook_rows), 0.01 * frames_n)
assert ("other" in {row["source"] for row in hook_rows}) is overflowed
assert stored == [expected_rows]
+3 -1
View File
@@ -449,5 +449,7 @@ def test_websocket_route_carries_the_client_decision(
assert seen, "the Responses compressor was never reached on the WS path"
assert {
k: v for k, v in seen[0].items() if k not in {"timing", "deadline_started_at"}
k: v
for k, v in seen[0].items()
if k not in {"timing", "deadline_started_at", "savings_tags"}
} == expected
+1
View File
@@ -133,6 +133,7 @@ def test_request_runner_attributes_savings_with_handler_counters():
"tokens": 15,
"usd": 0.0,
"details": {"message_tokens_saved": 14, "tool_tokens_saved": 1},
"events": 1,
}
]