mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(stats): stop rejected upstream turns from feeding the savings funnel (#3615)
## Description Carries #3010 by @nangsontay across the line. That PR moves the `emit_request_outcome` funnel short-circuit from `status_code >= 500` to `>= 400`, so a turn the provider rejected cannot reach `metrics.record_request`, `cost_tracker.record_tokens`, the `RequestLog` feed, or the PERF line. It was approved on 2026-09-13 and then blocked on one test seam (`test_codex_rate_limit_captured_on_streaming_429` awaits a `MagicMock`); no push since. This PR keeps the original commit verbatim (author preserved) and adds one commit that fixes that seam and updates the PR's new buffered-CCR test for #3092, which landed on main afterwards and gates the buffered path on a redeemable marker rather than the resident tool alone. Second real-world case, independent of the 429 fast-mode one in #3010: a Claude Code session on 2026-09-16 grew past the 1M window and entered a compaction loop. Every compaction request came back `400 invalid_request_error: prompt is too long: 2183557 tokens > 1000000 maximum`. Anthropic processed and billed none of them (the subscription window stayed at 2% / 4%), yet the proxy ran the full success funnel on each. 13 rejected turns of 1.6M to 1.9M tokens added 23.7M "input tokens" and $242 of "input cost" to the session display; 96% of that session's reported usage was turns the provider never served. Supersedes #3010. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/outcome.py` (from #3010, unchanged): funnel short-circuit at `status_code >= 400`; 429 books via `record_rate_limited`, every other 4xx/5xx via `record_failed`. Neither counter feeds savings. - `tests/test_outcome_records_5xx_as_failed.py` (from #3010): 429 rate-limited case, parametrized 400/401/403/404/413/422 sweep, and a 200 control that still reaches the funnel with `tokens_saved` intact. - `tests/test_anthropic_buffered_ccr_usage.py` (from #3010, adapted): the two buffered-CCR cases now plant an owned `<<ccr:...>>` marker via the same fixture `tests/test_ccr_buffered_stream_signed_thinking.py` uses, because #3092 no longer buffers on the resident tool alone. Without that the assertion `upstream saw stream:false` fails on current main. - `tests/test_proxy_streaming_ratelimit_headers.py`: the mock proxy's `metrics.record_rate_limited` is an `AsyncMock`, matching `record_request` / `record_failed` on the same mock. This is the seam the #3010 review flagged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_proxy_streaming_ratelimit_headers.py tests/test_outcome_records_5xx_as_failed.py tests/test_anthropic_buffered_ccr_usage.py tests/test_ccr_buffered_stream_signed_thinking.py -q 39 passed, 1 warning in 10.02s $ uv run --frozen --extra dev pytest tests -q -x -k "outcome or ratelimit or rate_limit or 429 or 4xx or upstream_error or record_failed or savings_tracker or persistent_metrics or buffered" 320 passed, 10 skipped, 12765 deselected, 2 warnings in 77.61s $ uv run --frozen --extra dev pytest tests -q -x 12172 passed, 7 failed, 593 skipped in 810s. The 7 (tests/test_proxy_health.py readyz-kompress cases and tests/test_read_maturation_handler_nobust.py) pass when run as their own files on this branch (22 passed) and on unmodified upstream/main; they are order-dependent and untouched by this change. $ uvx ruff check headroom/proxy/outcome.py tests/test_proxy_streaming_ratelimit_headers.py tests/test_outcome_records_5xx_as_failed.py tests/test_anthropic_buffered_ccr_usage.py All checks passed! $ uv run --frozen --extra dev mypy headroom/proxy/outcome.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 15.6 (Darwin 24.6.0), Python 3.11, this branch at its head, against a `create_app(ProxyConfig(optimize=False, ...))` proxy with the real `PrometheusMetrics`. - Exact command / steps: Replayed the live loop through `proxy._record_request_outcome`: 13 outcomes with `status_code=400`, `original_tokens=1640331`, `optimized_tokens=1638180`, `tokens_saved=2151` (the PERF values of request `hr_1789579601_001966` from the 2026-09-16 session), then one identical outcome with `status_code=200` as control. Printed `metrics.tokens_input_total`, `requests_failed`, `requests_total` after each phase. Then swapped in `upstream/main`'s `outcome.py` and re-ran. - Observed result: This branch: `tokens_input_total: start=0 after 13x400=0 after 1x200=1638180`, `requests_failed=13 requests_total=1`. `upstream/main`: the 13 rejected turns raised `tokens_input_total` to 21,296,340 before the control ran (assertion `400s still feed input totals` fired). - Not tested: A live upstream 400 through the whole HTTP handler on this branch; the handler sites already pass `status_code=response.status_code` (added in #1571 / #2682), so the funnel is the only place the status is consulted. Not re-measured on Windows. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes, accounting only. A 4xx upstream response no longer increments `requests_total`, `tokens_input_total`, the savings/cost ledgers, or the request log; it increments `requests_failed` (or `requests_rate_limited` for 429). The response returned to the client is unchanged. - Kill switch / disable path: None. Revert the commit. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert the squash commit; no persisted state changes shape. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Documentation: N/A, no user-facing surface changes beyond the counters already documented. Credit for the fix and its analysis belongs to @nangsontay in #3010; happy for maintainers to land whichever of the two is more convenient and close the other. --------- Co-authored-by: nangsontay <github@nangsontay.com>
This commit is contained in:
+27
-11
@@ -456,9 +456,10 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
(skipped when logger is None, i.e. ``--no-request-logging``)
|
||||
4. structured PERF log line — consumed by ``headroom perf``
|
||||
|
||||
A failure outcome (``status_code >= 500``, e.g. a 529 surfaced after retry
|
||||
exhaustion) short-circuits before effects 1-4: it records a failed request
|
||||
and returns, so an upstream failure cannot feed the success stats.
|
||||
A rejected outcome (``status_code >= 400``, e.g. a 429 rate limit or a 529
|
||||
surfaced after retry exhaustion) short-circuits before effects 1-4: it
|
||||
records the request under the counter that names what happened and returns,
|
||||
so a turn the provider never billed cannot feed the success stats.
|
||||
|
||||
Takes the handler as a free argument rather than ``self`` so this
|
||||
function is callable from:
|
||||
@@ -512,14 +513,29 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
# beacon must never add latency to, or take down, the request path.
|
||||
record_outcome(outcome)
|
||||
|
||||
# Upstream failure (>= 500, e.g. a 529 Overloaded surfaced after retry
|
||||
# exhaustion) must not feed the savings/cost/log success stats; that would
|
||||
# let a failed request inflate the save-rate. Record it as failed and stop,
|
||||
# mirroring the pre-passthrough behaviour where an exhausted 5xx raised and
|
||||
# was counted via record_failed. 4xx stay on the normal funnel: they are
|
||||
# client errors the proxy still served.
|
||||
if outcome.status_code >= 500:
|
||||
await handler.metrics.record_failed(provider=outcome.provider)
|
||||
# A rejected turn must not feed the savings/cost/log success stats; that
|
||||
# would let a request the provider never billed inflate the save-rate.
|
||||
# Record it under the counter that names what happened, and stop.
|
||||
#
|
||||
# This covers 4xx as well as 5xx. A 4xx is an error the PROXY served but the
|
||||
# PROVIDER did not: nothing was generated, so nothing was billed, so
|
||||
# compression on that turn saved exactly nothing. Counting it anyway is not a
|
||||
# rounding error — measured on a real session where 143 of 300 turns came
|
||||
# back 429 ("Usage credits are required for fast mode"), 46.5% of the
|
||||
# headline `total_saved` was compression on turns Anthropic rejected, and
|
||||
# `requests.rate_limited` still read 0 because only Headroom's own limiter
|
||||
# ever incremented it. The 60M `tokens.input` and the $8.90 compression
|
||||
# savings shown against a $2.26 measured spend came from the same place.
|
||||
#
|
||||
# 429 goes to record_rate_limited rather than record_failed: an upstream rate
|
||||
# limit is the one 4xx a user is expected to act on (back off, raise a cap),
|
||||
# and folding it into a generic failure count hides exactly that. Both
|
||||
# counters are already exported and neither feeds savings.
|
||||
if outcome.status_code >= 400:
|
||||
if outcome.status_code == 429:
|
||||
await handler.metrics.record_rate_limited(provider=outcome.provider)
|
||||
else:
|
||||
await handler.metrics.record_failed(provider=outcome.provider)
|
||||
return
|
||||
|
||||
# Outcome stage: hand the meter's reading to any pipeline extension that
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
"""A buffered-CCR turn is a billed call and its usage must reach accounting.
|
||||
|
||||
When a ``stream:true`` request carries the ``headroom_retrieve`` tool, the
|
||||
Anthropic handler rewrites it to ``stream:false`` upstream so it can resolve
|
||||
retrievals server-side, then re-synthesizes SSE for the client. That buffered
|
||||
response carries the same ``usage`` block any non-stream reply does, so the
|
||||
provider's cache-read / cache-write / output counts must land on the outcome.
|
||||
|
||||
If they don't, every cached token on the dominant Claude Code path is invisible:
|
||||
``metrics.cache_by_provider`` only records a provider row when cache read or
|
||||
write is non-zero, so the whole prefix-cache card (hit rate, savings, TTL mix)
|
||||
is computed from whatever small fraction of traffic took another path, while
|
||||
compression savings on the buffered turns still count in full.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
import respx
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from headroom.proxy.loopback_guard import require_loopback # noqa: E402
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
|
||||
# Provider-reported usage for one warm turn: most of the prompt served from
|
||||
# cache, a slice newly written, a little uncached, and a real completion.
|
||||
_USAGE = {
|
||||
"input_tokens": 300,
|
||||
"output_tokens": 120,
|
||||
"cache_read_input_tokens": 46893,
|
||||
"cache_creation_input_tokens": 8197,
|
||||
}
|
||||
|
||||
_RESPONSE = {
|
||||
"id": "msg_buffered",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "claude-opus-5",
|
||||
"content": [{"type": "text", "text": "done"}],
|
||||
"stop_reason": "end_turn",
|
||||
"usage": _USAGE,
|
||||
}
|
||||
|
||||
_RETRIEVE_TOOL = {
|
||||
"name": "headroom_retrieve",
|
||||
"description": "Retrieve compressed content",
|
||||
"input_schema": {"type": "object", "properties": {"ref": {"type": "string"}}},
|
||||
}
|
||||
|
||||
|
||||
def _app_and_outcomes(monkeypatch, **overrides):
|
||||
"""App with a spy on the outcome record — where billed counts land."""
|
||||
kwargs: dict[str, Any] = {
|
||||
"optimize": False,
|
||||
"cache_enabled": False,
|
||||
"rate_limit_enabled": False,
|
||||
"cost_tracking_enabled": False,
|
||||
"log_requests": False,
|
||||
}
|
||||
kwargs.update(overrides)
|
||||
app = create_app(ProxyConfig(**kwargs))
|
||||
app.dependency_overrides[require_loopback] = lambda: None
|
||||
outcomes: list[Any] = []
|
||||
|
||||
async def _spy(_self, outcome, *a, **kw):
|
||||
outcomes.append(outcome)
|
||||
|
||||
monkeypatch.setattr(type(app.state.proxy), "_record_request_outcome", _spy, raising=True)
|
||||
return app, outcomes
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def ccr_marker() -> str:
|
||||
"""A marker this proxy actually owns, so retrieval could really fire.
|
||||
|
||||
The buffered path is only taken when the outgoing body carries a redeemable
|
||||
marker (#3071); ``headroom_retrieve`` has nothing to expand otherwise. The
|
||||
buffered tests are about what happens *on* that path, so they have to earn it.
|
||||
"""
|
||||
from headroom.cache.backends import InMemoryBackend
|
||||
from headroom.cache.compression_store import get_compression_store, reset_compression_store
|
||||
|
||||
reset_compression_store()
|
||||
store = get_compression_store(backend=InMemoryBackend())
|
||||
hash_key = store.store(
|
||||
"the original, uncompressed tool output",
|
||||
"<<ccr:placeholder>>",
|
||||
original_tokens=100,
|
||||
compressed_tokens=5,
|
||||
tool_name="Read",
|
||||
)
|
||||
try:
|
||||
yield hash_key
|
||||
finally:
|
||||
reset_compression_store()
|
||||
|
||||
|
||||
def _post(app, *, stream: bool, tools: list[dict] | None, marker: str | None = None):
|
||||
text = "hi" if marker is None else f"hi, earlier output is at <<ccr:{marker}>>"
|
||||
body: dict[str, Any] = {
|
||||
"model": "claude-opus-5",
|
||||
"max_tokens": 256,
|
||||
"messages": [{"role": "user", "content": text}],
|
||||
}
|
||||
if stream:
|
||||
body["stream"] = True
|
||||
if tools is not None:
|
||||
body["tools"] = tools
|
||||
with TestClient(app) as client:
|
||||
return client.post("/v1/messages", json=body, headers={"x-api-key": "sk-ant-test"})
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_non_stream_turn_records_provider_usage(monkeypatch) -> None:
|
||||
"""Control: the plain non-stream path already books the usage block."""
|
||||
app, outcomes = _app_and_outcomes(monkeypatch)
|
||||
respx.post("https://api.anthropic.com/v1/messages").mock(
|
||||
return_value=httpx.Response(200, json=_RESPONSE)
|
||||
)
|
||||
|
||||
r = _post(app, stream=False, tools=None)
|
||||
|
||||
assert r.status_code == 200
|
||||
assert outcomes, "an outcome must be recorded"
|
||||
o = outcomes[-1]
|
||||
assert o.cache_read_tokens == 46893
|
||||
assert o.cache_write_tokens == 8197
|
||||
assert o.output_tokens == 120
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_buffered_ccr_turn_records_provider_usage(monkeypatch, ccr_marker: str) -> None:
|
||||
"""A stream:true + headroom_retrieve turn must book the same usage block."""
|
||||
app, outcomes = _app_and_outcomes(monkeypatch)
|
||||
route = respx.post("https://api.anthropic.com/v1/messages").mock(
|
||||
return_value=httpx.Response(200, json=_RESPONSE)
|
||||
)
|
||||
|
||||
r = _post(app, stream=True, tools=[_RETRIEVE_TOOL], marker=ccr_marker)
|
||||
|
||||
assert r.status_code == 200
|
||||
# The handler must have buffered it: upstream saw stream:false.
|
||||
sent = route.calls.last.request
|
||||
assert b'"stream": false' in sent.content or b'"stream":false' in sent.content
|
||||
|
||||
assert outcomes, "an outcome must be recorded"
|
||||
o = outcomes[-1]
|
||||
assert o.cache_read_tokens == 46893, (
|
||||
f"buffered-CCR turn dropped the provider cache-read count: {o.cache_read_tokens}"
|
||||
)
|
||||
assert o.cache_write_tokens == 8197, (
|
||||
f"buffered-CCR turn dropped the provider cache-write count: {o.cache_write_tokens}"
|
||||
)
|
||||
assert o.output_tokens == 120, (
|
||||
f"buffered-CCR turn dropped the provider output count: {o.output_tokens}"
|
||||
)
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_buffered_ccr_records_usage_with_compression_on(monkeypatch, ccr_marker: str) -> None:
|
||||
"""Same turn with the live token-mode pipeline running, as in production."""
|
||||
app, outcomes = _app_and_outcomes(monkeypatch, optimize=True, mode="token")
|
||||
respx.post("https://api.anthropic.com/v1/messages").mock(
|
||||
return_value=httpx.Response(200, json=_RESPONSE)
|
||||
)
|
||||
|
||||
r = _post(app, stream=True, tools=[_RETRIEVE_TOOL], marker=ccr_marker)
|
||||
|
||||
assert r.status_code == 200
|
||||
assert outcomes, "an outcome must be recorded"
|
||||
o = outcomes[-1]
|
||||
assert (o.cache_read_tokens, o.cache_write_tokens, o.output_tokens) == (46893, 8197, 120), (
|
||||
f"buffered-CCR turn dropped usage under compression: "
|
||||
f"cr={o.cache_read_tokens} cw={o.cache_write_tokens} out={o.output_tokens}"
|
||||
)
|
||||
@@ -1,6 +1,14 @@
|
||||
"""An exhausted-5xx outcome (e.g. a 529 Overloaded surfaced after retry
|
||||
exhaustion) must be counted as a failed request, not fed into the
|
||||
savings/cost success funnel where it would inflate the save-rate.
|
||||
"""A rejected outcome must never reach the savings/cost success funnel.
|
||||
|
||||
An exhausted 5xx (e.g. a 529 Overloaded surfaced after retry exhaustion) is the
|
||||
original case. A 4xx belongs there too: the proxy served an error, but the
|
||||
provider generated and billed nothing, so compression on that turn saved
|
||||
nothing. Feeding it in let rejected turns inflate the save-rate — on a real
|
||||
session with 143 of 300 turns returning 429, 46.5% of the headline
|
||||
`total_saved` was compression on turns Anthropic refused.
|
||||
|
||||
429 is counted as rate-limited rather than generically failed, because that is
|
||||
the one 4xx a user acts on.
|
||||
|
||||
Companion to the retry-exhaustion change that returns the real upstream 5xx
|
||||
instead of collapsing it to a 502.
|
||||
@@ -8,17 +16,23 @@ instead of collapsing it to a 502.
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.outcome import RequestOutcome, emit_request_outcome
|
||||
|
||||
|
||||
class _Metrics:
|
||||
def __init__(self):
|
||||
self.failed = []
|
||||
self.rate_limited = []
|
||||
self.requested = []
|
||||
|
||||
async def record_failed(self, provider):
|
||||
self.failed.append(provider)
|
||||
|
||||
async def record_rate_limited(self, provider):
|
||||
self.rate_limited.append(provider)
|
||||
|
||||
async def record_request(self, **kwargs):
|
||||
self.requested.append(kwargs)
|
||||
|
||||
@@ -31,16 +45,16 @@ class _Handler:
|
||||
self.metrics = _Metrics()
|
||||
|
||||
|
||||
def _outcome(status_code):
|
||||
def _outcome(status_code, tokens_saved=0):
|
||||
return RequestOutcome(
|
||||
request_id="req-1",
|
||||
provider="anthropic",
|
||||
model="claude-opus-4-8",
|
||||
original_tokens=0,
|
||||
original_tokens=tokens_saved,
|
||||
optimized_tokens=0,
|
||||
output_tokens=0,
|
||||
tokens_saved=0,
|
||||
attempted_input_tokens=0,
|
||||
tokens_saved=tokens_saved,
|
||||
attempted_input_tokens=tokens_saved,
|
||||
status_code=status_code,
|
||||
)
|
||||
|
||||
@@ -59,6 +73,33 @@ def test_503_recorded_as_failed():
|
||||
assert handler.metrics.requested == []
|
||||
|
||||
|
||||
def test_429_recorded_as_rate_limited_and_skips_success_funnel():
|
||||
"""The production case: an upstream rate limit saved nothing."""
|
||||
handler = _Handler()
|
||||
asyncio.run(emit_request_outcome(handler, _outcome(429, tokens_saved=6380)))
|
||||
assert handler.metrics.rate_limited == ["anthropic"]
|
||||
assert handler.metrics.failed == [] # a rate limit is not a generic failure
|
||||
assert handler.metrics.requested == [] # its 6,380 "saved" tokens are not savings
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [400, 401, 403, 404, 413, 422])
|
||||
def test_other_4xx_recorded_as_failed_and_skips_success_funnel(status):
|
||||
handler = _Handler()
|
||||
asyncio.run(emit_request_outcome(handler, _outcome(status, tokens_saved=1000)))
|
||||
assert handler.metrics.failed == ["anthropic"]
|
||||
assert handler.metrics.requested == []
|
||||
|
||||
|
||||
def test_200_still_reaches_the_success_funnel():
|
||||
"""Guard against over-correcting: a served turn must still be counted."""
|
||||
handler = _Handler()
|
||||
asyncio.run(emit_request_outcome(handler, _outcome(200, tokens_saved=1000)))
|
||||
assert handler.metrics.failed == []
|
||||
assert handler.metrics.rate_limited == []
|
||||
assert len(handler.metrics.requested) == 1
|
||||
assert handler.metrics.requested[0]["tokens_saved"] == 1000
|
||||
|
||||
|
||||
def test_stream_outcome_529_recorded_as_failed_and_skips_success_funnel():
|
||||
handler = _Handler()
|
||||
outcome = RequestOutcome.from_stream(
|
||||
|
||||
@@ -46,6 +46,7 @@ class TestStreamingRatelimitHeaderForwarding:
|
||||
proxy.metrics = MagicMock()
|
||||
proxy.metrics.record_request = AsyncMock(return_value=None)
|
||||
proxy.metrics.record_failed = AsyncMock(return_value=None)
|
||||
proxy.metrics.record_rate_limited = AsyncMock(return_value=None)
|
||||
proxy.cost_tracker = MagicMock()
|
||||
proxy.cost_tracker.estimate_cost.return_value = 0.001
|
||||
proxy.cost_tracker.record_request.return_value = None
|
||||
|
||||
Reference in New Issue
Block a user