fix(stats): stop rejected upstream turns from feeding the savings funnel (#3615)

## Description

Carries #3010 by @nangsontay across the line. That PR moves the
`emit_request_outcome` funnel short-circuit from `status_code >= 500` to
`>= 400`, so a turn the provider rejected cannot reach
`metrics.record_request`, `cost_tracker.record_tokens`, the `RequestLog`
feed, or the PERF line. It was approved on 2026-09-13 and then blocked
on one test seam (`test_codex_rate_limit_captured_on_streaming_429`
awaits a `MagicMock`); no push since. This PR keeps the original commit
verbatim (author preserved) and adds one commit that fixes that seam and
updates the PR's new buffered-CCR test for #3092, which landed on main
afterwards and gates the buffered path on a redeemable marker rather
than the resident tool alone.

Second real-world case, independent of the 429 fast-mode one in #3010: a
Claude Code session on 2026-09-16 grew past the 1M window and entered a
compaction loop. Every compaction request came back `400
invalid_request_error: prompt is too long: 2183557 tokens > 1000000
maximum`. Anthropic processed and billed none of them (the subscription
window stayed at 2% / 4%), yet the proxy ran the full success funnel on
each. 13 rejected turns of 1.6M to 1.9M tokens added 23.7M "input
tokens" and $242 of "input cost" to the session display; 96% of that
session's reported usage was turns the provider never served.

Supersedes #3010.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/outcome.py` (from #3010, unchanged): funnel
short-circuit at `status_code >= 400`; 429 books via
`record_rate_limited`, every other 4xx/5xx via `record_failed`. Neither
counter feeds savings.
- `tests/test_outcome_records_5xx_as_failed.py` (from #3010): 429
rate-limited case, parametrized 400/401/403/404/413/422 sweep, and a 200
control that still reaches the funnel with `tokens_saved` intact.
- `tests/test_anthropic_buffered_ccr_usage.py` (from #3010, adapted):
the two buffered-CCR cases now plant an owned `<<ccr:...>>` marker via
the same fixture `tests/test_ccr_buffered_stream_signed_thinking.py`
uses, because #3092 no longer buffers on the resident tool alone.
Without that the assertion `upstream saw stream:false` fails on current
main.
- `tests/test_proxy_streaming_ratelimit_headers.py`: the mock proxy's
`metrics.record_rate_limited` is an `AsyncMock`, matching
`record_request` / `record_failed` on the same mock. This is the seam
the #3010 review flagged.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy_streaming_ratelimit_headers.py tests/test_outcome_records_5xx_as_failed.py tests/test_anthropic_buffered_ccr_usage.py tests/test_ccr_buffered_stream_signed_thinking.py -q
39 passed, 1 warning in 10.02s

$ uv run --frozen --extra dev pytest tests -q -x -k "outcome or ratelimit or rate_limit or 429 or 4xx or upstream_error or record_failed or savings_tracker or persistent_metrics or buffered"
320 passed, 10 skipped, 12765 deselected, 2 warnings in 77.61s

$ uv run --frozen --extra dev pytest tests -q -x
12172 passed, 7 failed, 593 skipped in 810s. The 7 (tests/test_proxy_health.py readyz-kompress cases and tests/test_read_maturation_handler_nobust.py) pass when run as their own files on this branch (22 passed) and on unmodified upstream/main; they are order-dependent and untouched by this change.

$ uvx ruff check headroom/proxy/outcome.py tests/test_proxy_streaming_ratelimit_headers.py tests/test_outcome_records_5xx_as_failed.py tests/test_anthropic_buffered_ccr_usage.py
All checks passed!
$ uv run --frozen --extra dev mypy headroom/proxy/outcome.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.11, this branch at
its head, against a `create_app(ProxyConfig(optimize=False, ...))` proxy
with the real `PrometheusMetrics`.
- Exact command / steps: Replayed the live loop through
`proxy._record_request_outcome`: 13 outcomes with `status_code=400`,
`original_tokens=1640331`, `optimized_tokens=1638180`,
`tokens_saved=2151` (the PERF values of request `hr_1789579601_001966`
from the 2026-09-16 session), then one identical outcome with
`status_code=200` as control. Printed `metrics.tokens_input_total`,
`requests_failed`, `requests_total` after each phase. Then swapped in
`upstream/main`'s `outcome.py` and re-ran.
- Observed result: This branch: `tokens_input_total: start=0 after
13x400=0 after 1x200=1638180`, `requests_failed=13 requests_total=1`.
`upstream/main`: the 13 rejected turns raised `tokens_input_total` to
21,296,340 before the control ran (assertion `400s still feed input
totals` fired).
- Not tested: A live upstream 400 through the whole HTTP handler on this
branch; the handler sites already pass
`status_code=response.status_code` (added in #1571 / #2682), so the
funnel is the only place the status is consulted. Not re-measured on
Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, accounting only. A 4xx upstream
response no longer increments `requests_total`, `tokens_input_total`,
the savings/cost ledgers, or the request log; it increments
`requests_failed` (or `requests_rate_limited` for 429). The response
returned to the client is unchanged.
- Kill switch / disable path: None. Revert the commit.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the squash commit; no persisted state changes
shape.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation: N/A, no user-facing surface changes beyond the counters
already documented. Credit for the fix and its analysis belongs to
@nangsontay in #3010; happy for maintainers to land whichever of the two
is more convenient and close the other.

---------

Co-authored-by: nangsontay <github@nangsontay.com>
This commit is contained in:
gglucass
2026-09-21 10:23:06 -07:00
committed by GitHub
co-authored by nangsontay
parent 6f404de54e
commit 85fac8c3a9
4 changed files with 257 additions and 18 deletions
+27 -11
View File
@@ -456,9 +456,10 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
(skipped when logger is None, i.e. ``--no-request-logging``)
4. structured PERF log line — consumed by ``headroom perf``
A failure outcome (``status_code >= 500``, e.g. a 529 surfaced after retry
exhaustion) short-circuits before effects 1-4: it records a failed request
and returns, so an upstream failure cannot feed the success stats.
A rejected outcome (``status_code >= 400``, e.g. a 429 rate limit or a 529
surfaced after retry exhaustion) short-circuits before effects 1-4: it
records the request under the counter that names what happened and returns,
so a turn the provider never billed cannot feed the success stats.
Takes the handler as a free argument rather than ``self`` so this
function is callable from:
@@ -512,14 +513,29 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
# beacon must never add latency to, or take down, the request path.
record_outcome(outcome)
# Upstream failure (>= 500, e.g. a 529 Overloaded surfaced after retry
# exhaustion) must not feed the savings/cost/log success stats; that would
# let a failed request inflate the save-rate. Record it as failed and stop,
# mirroring the pre-passthrough behaviour where an exhausted 5xx raised and
# was counted via record_failed. 4xx stay on the normal funnel: they are
# client errors the proxy still served.
if outcome.status_code >= 500:
await handler.metrics.record_failed(provider=outcome.provider)
# A rejected turn must not feed the savings/cost/log success stats; that
# would let a request the provider never billed inflate the save-rate.
# Record it under the counter that names what happened, and stop.
#
# This covers 4xx as well as 5xx. A 4xx is an error the PROXY served but the
# PROVIDER did not: nothing was generated, so nothing was billed, so
# compression on that turn saved exactly nothing. Counting it anyway is not a
# rounding error — measured on a real session where 143 of 300 turns came
# back 429 ("Usage credits are required for fast mode"), 46.5% of the
# headline `total_saved` was compression on turns Anthropic rejected, and
# `requests.rate_limited` still read 0 because only Headroom's own limiter
# ever incremented it. The 60M `tokens.input` and the $8.90 compression
# savings shown against a $2.26 measured spend came from the same place.
#
# 429 goes to record_rate_limited rather than record_failed: an upstream rate
# limit is the one 4xx a user is expected to act on (back off, raise a cap),
# and folding it into a generic failure count hides exactly that. Both
# counters are already exported and neither feeds savings.
if outcome.status_code >= 400:
if outcome.status_code == 429:
await handler.metrics.record_rate_limited(provider=outcome.provider)
else:
await handler.metrics.record_failed(provider=outcome.provider)
return
# Outcome stage: hand the meter's reading to any pipeline extension that
+181
View File
@@ -0,0 +1,181 @@
"""A buffered-CCR turn is a billed call and its usage must reach accounting.
When a ``stream:true`` request carries the ``headroom_retrieve`` tool, the
Anthropic handler rewrites it to ``stream:false`` upstream so it can resolve
retrievals server-side, then re-synthesizes SSE for the client. That buffered
response carries the same ``usage`` block any non-stream reply does, so the
provider's cache-read / cache-write / output counts must land on the outcome.
If they don't, every cached token on the dominant Claude Code path is invisible:
``metrics.cache_by_provider`` only records a provider row when cache read or
write is non-zero, so the whole prefix-cache card (hit rate, savings, TTL mix)
is computed from whatever small fraction of traffic took another path, while
compression savings on the buffered turns still count in full.
"""
from __future__ import annotations
from typing import Any
import httpx
import pytest
import respx
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.loopback_guard import require_loopback # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
# Provider-reported usage for one warm turn: most of the prompt served from
# cache, a slice newly written, a little uncached, and a real completion.
_USAGE = {
"input_tokens": 300,
"output_tokens": 120,
"cache_read_input_tokens": 46893,
"cache_creation_input_tokens": 8197,
}
_RESPONSE = {
"id": "msg_buffered",
"type": "message",
"role": "assistant",
"model": "claude-opus-5",
"content": [{"type": "text", "text": "done"}],
"stop_reason": "end_turn",
"usage": _USAGE,
}
_RETRIEVE_TOOL = {
"name": "headroom_retrieve",
"description": "Retrieve compressed content",
"input_schema": {"type": "object", "properties": {"ref": {"type": "string"}}},
}
def _app_and_outcomes(monkeypatch, **overrides):
"""App with a spy on the outcome record — where billed counts land."""
kwargs: dict[str, Any] = {
"optimize": False,
"cache_enabled": False,
"rate_limit_enabled": False,
"cost_tracking_enabled": False,
"log_requests": False,
}
kwargs.update(overrides)
app = create_app(ProxyConfig(**kwargs))
app.dependency_overrides[require_loopback] = lambda: None
outcomes: list[Any] = []
async def _spy(_self, outcome, *a, **kw):
outcomes.append(outcome)
monkeypatch.setattr(type(app.state.proxy), "_record_request_outcome", _spy, raising=True)
return app, outcomes
@pytest.fixture
def ccr_marker() -> str:
"""A marker this proxy actually owns, so retrieval could really fire.
The buffered path is only taken when the outgoing body carries a redeemable
marker (#3071); ``headroom_retrieve`` has nothing to expand otherwise. The
buffered tests are about what happens *on* that path, so they have to earn it.
"""
from headroom.cache.backends import InMemoryBackend
from headroom.cache.compression_store import get_compression_store, reset_compression_store
reset_compression_store()
store = get_compression_store(backend=InMemoryBackend())
hash_key = store.store(
"the original, uncompressed tool output",
"<<ccr:placeholder>>",
original_tokens=100,
compressed_tokens=5,
tool_name="Read",
)
try:
yield hash_key
finally:
reset_compression_store()
def _post(app, *, stream: bool, tools: list[dict] | None, marker: str | None = None):
text = "hi" if marker is None else f"hi, earlier output is at <<ccr:{marker}>>"
body: dict[str, Any] = {
"model": "claude-opus-5",
"max_tokens": 256,
"messages": [{"role": "user", "content": text}],
}
if stream:
body["stream"] = True
if tools is not None:
body["tools"] = tools
with TestClient(app) as client:
return client.post("/v1/messages", json=body, headers={"x-api-key": "sk-ant-test"})
@respx.mock
def test_non_stream_turn_records_provider_usage(monkeypatch) -> None:
"""Control: the plain non-stream path already books the usage block."""
app, outcomes = _app_and_outcomes(monkeypatch)
respx.post("https://api.anthropic.com/v1/messages").mock(
return_value=httpx.Response(200, json=_RESPONSE)
)
r = _post(app, stream=False, tools=None)
assert r.status_code == 200
assert outcomes, "an outcome must be recorded"
o = outcomes[-1]
assert o.cache_read_tokens == 46893
assert o.cache_write_tokens == 8197
assert o.output_tokens == 120
@respx.mock
def test_buffered_ccr_turn_records_provider_usage(monkeypatch, ccr_marker: str) -> None:
"""A stream:true + headroom_retrieve turn must book the same usage block."""
app, outcomes = _app_and_outcomes(monkeypatch)
route = respx.post("https://api.anthropic.com/v1/messages").mock(
return_value=httpx.Response(200, json=_RESPONSE)
)
r = _post(app, stream=True, tools=[_RETRIEVE_TOOL], marker=ccr_marker)
assert r.status_code == 200
# The handler must have buffered it: upstream saw stream:false.
sent = route.calls.last.request
assert b'"stream": false' in sent.content or b'"stream":false' in sent.content
assert outcomes, "an outcome must be recorded"
o = outcomes[-1]
assert o.cache_read_tokens == 46893, (
f"buffered-CCR turn dropped the provider cache-read count: {o.cache_read_tokens}"
)
assert o.cache_write_tokens == 8197, (
f"buffered-CCR turn dropped the provider cache-write count: {o.cache_write_tokens}"
)
assert o.output_tokens == 120, (
f"buffered-CCR turn dropped the provider output count: {o.output_tokens}"
)
@respx.mock
def test_buffered_ccr_records_usage_with_compression_on(monkeypatch, ccr_marker: str) -> None:
"""Same turn with the live token-mode pipeline running, as in production."""
app, outcomes = _app_and_outcomes(monkeypatch, optimize=True, mode="token")
respx.post("https://api.anthropic.com/v1/messages").mock(
return_value=httpx.Response(200, json=_RESPONSE)
)
r = _post(app, stream=True, tools=[_RETRIEVE_TOOL], marker=ccr_marker)
assert r.status_code == 200
assert outcomes, "an outcome must be recorded"
o = outcomes[-1]
assert (o.cache_read_tokens, o.cache_write_tokens, o.output_tokens) == (46893, 8197, 120), (
f"buffered-CCR turn dropped usage under compression: "
f"cr={o.cache_read_tokens} cw={o.cache_write_tokens} out={o.output_tokens}"
)
+48 -7
View File
@@ -1,6 +1,14 @@
"""An exhausted-5xx outcome (e.g. a 529 Overloaded surfaced after retry
exhaustion) must be counted as a failed request, not fed into the
savings/cost success funnel where it would inflate the save-rate.
"""A rejected outcome must never reach the savings/cost success funnel.
An exhausted 5xx (e.g. a 529 Overloaded surfaced after retry exhaustion) is the
original case. A 4xx belongs there too: the proxy served an error, but the
provider generated and billed nothing, so compression on that turn saved
nothing. Feeding it in let rejected turns inflate the save-rate — on a real
session with 143 of 300 turns returning 429, 46.5% of the headline
`total_saved` was compression on turns Anthropic refused.
429 is counted as rate-limited rather than generically failed, because that is
the one 4xx a user acts on.
Companion to the retry-exhaustion change that returns the real upstream 5xx
instead of collapsing it to a 502.
@@ -8,17 +16,23 @@ instead of collapsing it to a 502.
import asyncio
import pytest
from headroom.proxy.outcome import RequestOutcome, emit_request_outcome
class _Metrics:
def __init__(self):
self.failed = []
self.rate_limited = []
self.requested = []
async def record_failed(self, provider):
self.failed.append(provider)
async def record_rate_limited(self, provider):
self.rate_limited.append(provider)
async def record_request(self, **kwargs):
self.requested.append(kwargs)
@@ -31,16 +45,16 @@ class _Handler:
self.metrics = _Metrics()
def _outcome(status_code):
def _outcome(status_code, tokens_saved=0):
return RequestOutcome(
request_id="req-1",
provider="anthropic",
model="claude-opus-4-8",
original_tokens=0,
original_tokens=tokens_saved,
optimized_tokens=0,
output_tokens=0,
tokens_saved=0,
attempted_input_tokens=0,
tokens_saved=tokens_saved,
attempted_input_tokens=tokens_saved,
status_code=status_code,
)
@@ -59,6 +73,33 @@ def test_503_recorded_as_failed():
assert handler.metrics.requested == []
def test_429_recorded_as_rate_limited_and_skips_success_funnel():
"""The production case: an upstream rate limit saved nothing."""
handler = _Handler()
asyncio.run(emit_request_outcome(handler, _outcome(429, tokens_saved=6380)))
assert handler.metrics.rate_limited == ["anthropic"]
assert handler.metrics.failed == [] # a rate limit is not a generic failure
assert handler.metrics.requested == [] # its 6,380 "saved" tokens are not savings
@pytest.mark.parametrize("status", [400, 401, 403, 404, 413, 422])
def test_other_4xx_recorded_as_failed_and_skips_success_funnel(status):
handler = _Handler()
asyncio.run(emit_request_outcome(handler, _outcome(status, tokens_saved=1000)))
assert handler.metrics.failed == ["anthropic"]
assert handler.metrics.requested == []
def test_200_still_reaches_the_success_funnel():
"""Guard against over-correcting: a served turn must still be counted."""
handler = _Handler()
asyncio.run(emit_request_outcome(handler, _outcome(200, tokens_saved=1000)))
assert handler.metrics.failed == []
assert handler.metrics.rate_limited == []
assert len(handler.metrics.requested) == 1
assert handler.metrics.requested[0]["tokens_saved"] == 1000
def test_stream_outcome_529_recorded_as_failed_and_skips_success_funnel():
handler = _Handler()
outcome = RequestOutcome.from_stream(
@@ -46,6 +46,7 @@ class TestStreamingRatelimitHeaderForwarding:
proxy.metrics = MagicMock()
proxy.metrics.record_request = AsyncMock(return_value=None)
proxy.metrics.record_failed = AsyncMock(return_value=None)
proxy.metrics.record_rate_limited = AsyncMock(return_value=None)
proxy.cost_tracker = MagicMock()
proxy.cost_tracker.estimate_cost.return_value = 0.001
proxy.cost_tracker.record_request.return_value = None