mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy/bedrock): record the real upstream status on InvokeModel outcomes (#4025)
## Description `handle_bedrock_invoke` (the Bedrock `InvokeModel` REST passthrough handler) builds a `RequestOutcome` without ever passing `status_code`, so every outcome it records keeps the dataclass default of `200` regardless of what the Bedrock upstream actually returned. A throttled (429), unauthorized (403), bad-request (400), or 5xx response from the configured `--bedrock-api-url` gateway is forwarded to the client with its real status, but is recorded in Headroom's own savings/cost/request-log funnel as a successful `200` request -- so `emit_request_outcome`'s `status_code >= 400` short-circuit (added in #2682 / #3010) never fires for this path, and a rejected Bedrock turn still feeds `metrics.record_request`, the cost tracker, and the PERF log exactly like a served one. This is a different gap from the one #2682 documented and deliberately left alone: that PR's scope note says `_stream_response_bedrock` (the SDK/boto3-based direct-to-AWS streaming path in `streaming.py`) "never holds an HTTP response object" so it can't pass a real status. `handle_bedrock_invoke` is a separate code path -- the REST gateway passthrough described in `bedrock.py`'s own module docstring -- and it does forward via raw `httpx`, so `response.status_code` (the real upstream status, already used to build the `StreamingResponse`/`JSONResponse` returned to the client) was sitting right there and simply never got threaded into the outcome. Every other `RequestOutcome` call site in the raw-httpx handlers (`anthropic.py`, `batch.py`, `gemini.py`, most of `openai.py`) passes `status_code=response.status_code` (or equivalent); `bedrock.py` was the one exception. Closes: none filed -- found by direct source review, not from an existing issue. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/bedrock.py`: pass `status_code=response.status_code` when constructing the `RequestOutcome` in `handle_bedrock_invoke`, with a comment distinguishing this path (real httpx response available) from the SDK-based streaming path that genuinely cannot. - `tests/test_proxy/test_bedrock_passthrough.py`: added `test_outcome_carries_real_upstream_status_on_error`, parametrized over `[400, 403, 429, 500, 503]`, asserting the recorded `RequestOutcome.status_code` matches the real upstream status rather than the `200` default. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) -- see "Not tested" below - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_bedrock_passthrough.py -q collected 23 items tests\test_proxy\test_bedrock_passthrough.py ....................... [100%] 23 passed in 23.15s $ uv run ruff check headroom/proxy/handlers/bedrock.py tests/test_proxy/test_bedrock_passthrough.py All checks passed! $ uv run ruff format --check headroom/proxy/handlers/bedrock.py tests/test_proxy/test_bedrock_passthrough.py 2 files already formatted ``` RED proof -- new test fails against the pre-fix handler (stashed only `bedrock.py`, kept the new test): ```text $ uv run pytest tests/test_proxy/test_bedrock_passthrough.py::test_outcome_carries_real_upstream_status_on_error -q ... assert outcome.status_code == upstream_status E AssertionError: assert 200 == 400 E + where 200 = RequestOutcome(request_id='hr_...', provider='bedrock', ...).status_code 7 failed (one per parametrized status), 0 passed ``` GREEN -- same command with the fix restored: all 5 parametrized cases (400/403/429/500/503) pass, full file 23/23 passed. ## Real Behavior Proof - Environment: Windows 11, Python 3.12.4, headroom `main` @ `3c418f6f`, `uv run` with the `proxy` extra installed. - Exact command / steps: A standalone script (not committed, not a unit test) started a real `http.server.ThreadingHTTPServer` bound to `127.0.0.1:<ephemeral-port>` -- an actual TCP socket, a genuine stand-in for a Bedrock re-signing gateway -- that answers `POST /model/.../invoke` with a crafted upstream `429 ThrottlingException`. `ProxyConfig.bedrock_api_url` was pointed at that server; nothing else was touched. The script then booted the real FastAPI app via `TestClient`, which runs the real ASGI lifespan, so `proxy.http_client` is the genuine, **unmodified** `httpx.AsyncClient` created by `proxy.startup()` -- it was never replaced or mocked, unlike the unit test and the original proof script. It POSTed a real request through the live route to `/model/{model_id}/invoke`, so the request traveled over an actual socket through the real `_forward_bedrock`/httpx code path to the real local server and back. `proxy._record_request_outcome` was wrapped only to capture the `RequestOutcome` object for printing -- it still called through to the real implementation, so the real funnel (`emit_request_outcome`, Prometheus counters, the proxy's own end-of-run session-summary log) executed for real. Ran once with `headroom/proxy/handlers/bedrock.py` checked out from `main` (pre-fix) and once with the fix restored (post-fix); the fix was never left in the reverted state (verified with `git status` / `git diff` after restoring). - Observed result: Pre-fix: `HTTP response returned to client: 429` / `RequestOutcome.status_code recorded for metrics/savings funnel: 200` / `Would emit_request_outcome short-circuit (status_code >= 400)? False`. The proxy's own session-summary log for that run printed `Total requests: 1`, `Rate limited: 0`, `Failed: 0` -- the throttled request was silently counted as an ordinary served request. Post-fix, identical request/server: `HTTP response returned to client: 429` / `RequestOutcome.status_code recorded for metrics/savings funnel: 429` / `Would emit_request_outcome short-circuit (status_code >= 400)? True`, and the session-summary log changed to `Total requests: 0`, `Rate limited: 1`, ` upstream 429s: 1`. In both runs the client-facing response was correctly 429 -- only the internal accounting silently diverged from it pre-fix, which is exactly the bug this PR fixes. - Not tested: `mypy headroom` -- this Windows dev box's `uv sync --extra dev` fails building the `hnswlib` native extension (missing MSVC toolchain, unrelated to this change); only the `proxy` extra was synced. No live AWS Bedrock or a real LiteLLM/LocalStack gateway was exercised -- the upstream in this proof is a local `http.server` instance standing in for that gateway. To be explicit about what is and isn't real here: the real parts are the TCP socket, the HTTP round trip, headroom's own `httpx.AsyncClient`, and the full `handle_bedrock_invoke`/`_forward_bedrock`/`emit_request_outcome` code path, all unmodified and unmocked; the one stand-in is the endpoint itself (a local server playing the role of the AWS/gateway upstream, not AWS or a real gateway). The streaming route (`invoke-with-response-stream`) was not re-run under this real-socket proof; it shares the identical `_forward_bedrock` call and outcome-recording code with `invoke`, and is already covered by the existing mocked `test_invoke_with_response_stream_is_byte_faithful` test. ## Runtime Rollout Safety - Rollout-managed feature(s): none -- Bedrock passthrough is already gated behind `--bedrock-api-url`/`BEDROCK_TARGET_API_URL` being set; this change doesn't add or touch a rollout flag. - Minimum rollout channel: N/A, ships on stable with no gate. - Stable/default behavior changed: yes, but only for already-failing requests -- a Bedrock `InvokeModel` call that upstream rejects (4xx/5xx) will now correctly show up under `requests_rate_limited`/`requests_failed` instead of inflating `requests_total`/`tokens_saved_total`/cost. Successful (2xx) requests are unaffected (`status_code=200` either way). - Kill switch / disable path: none added; revert is a one-line removal of the `status_code=response.status_code` kwarg. - Unsafe override required: no. - Qualification impact: none. - Rollback path: `git revert` this commit; no persisted state to migrate. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Found by direct source review of `headroom/proxy/handlers/bedrock.py` while auditing provider-handler divergence (comparing how each provider path populates `RequestOutcome`), not from the issue queue. Cross-checked against #2682 and #3010 (which both touch `RequestOutcome.status_code` plumbing on other paths) to confirm this specific gap -- the REST gateway passthrough, as opposed to the SDK-based streaming path #2682 explicitly left alone -- was not already covered. **2026-10-08 update:** addressed Devin's review comment (mocked-upstream proof) and the governance bot's template-field check: - The `Real Behavior Proof` section above now documents a genuine local-socket proof: a real `http.server` TCP listener stands in for the Bedrock gateway, and headroom's own `httpx.AsyncClient` / `handle_bedrock_invoke` / `_forward_bedrock` / `emit_request_outcome` code is exercised completely unmocked against it. The existing mocked unit test (`test_outcome_carries_real_upstream_status_on_error`) is unchanged and still serves as the fast, deterministic correctness check; the new evidence is additional runtime proof, not a replacement. - `Real Behavior Proof` and `Runtime Rollout Safety` now use the exact plain-text field labels (`Environment:`, `Exact command / steps:`, `Observed result:`, `Not tested:`) the governance script matches on, instead of the differently-worded bold labels the previous revision used. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4025" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
20ce054f8d
commit
159daa25a0
@@ -243,6 +243,14 @@ class BedrockHandlerMixin:
|
||||
# contract treats 0 as "not measured") — Bedrock responses are forwarded
|
||||
# byte-faithfully and never parsed. The valuable figure, request-side
|
||||
# compression, is recorded in full.
|
||||
#
|
||||
# status_code=response.status_code: unlike streaming.py's SDK-based
|
||||
# _stream_response_bedrock (which never holds an HTTP response object
|
||||
# and so keeps the dataclass default of 200 — see PR #2682), this
|
||||
# handler forwards via raw httpx and always has the real upstream
|
||||
# status on the returned Response/StreamingResponse. Leaving it out
|
||||
# here made every upstream 4xx/5xx (throttling, auth failure, model
|
||||
# not found) record as a 200 success in the savings/cost funnel.
|
||||
try:
|
||||
from headroom.proxy.outcome import RequestOutcome
|
||||
|
||||
@@ -261,6 +269,7 @@ class BedrockHandlerMixin:
|
||||
pipeline_timing=pipeline_timing,
|
||||
tags=tags,
|
||||
client=client,
|
||||
status_code=response.status_code,
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
|
||||
@@ -450,6 +450,34 @@ def test_outcome_recorded_with_bedrock_provider():
|
||||
assert outcome.tokens_saved == 750
|
||||
|
||||
|
||||
@pytest.mark.parametrize("upstream_status", [400, 403, 429, 500, 503])
|
||||
def test_outcome_carries_real_upstream_status_on_error(upstream_status):
|
||||
"""Regression: handle_bedrock_invoke built RequestOutcome without a
|
||||
status_code kwarg, so an upstream throttle/auth/model error recorded as
|
||||
the dataclass default of 200 — the failed turn then fed the success
|
||||
funnel (savings/cost/request-log) in emit_request_outcome instead of
|
||||
being short-circuited by its `status_code >= 400` guard."""
|
||||
app = create_app(_make_config())
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
_install_fake_client(
|
||||
proxy, _FakeUpstream(status_code=upstream_status, chunks=(b'{"error":true}',))
|
||||
)
|
||||
proxy._record_request_outcome = AsyncMock()
|
||||
proxy.anthropic_pipeline.apply = MagicMock(
|
||||
return_value=_FakeResult([{"role": "user", "content": "c"}], 1000, 250)
|
||||
)
|
||||
resp = client.post(
|
||||
INVOKE,
|
||||
json={"messages": [{"role": "user", "content": "q" * 3000}], "max_tokens": 8},
|
||||
)
|
||||
|
||||
assert resp.status_code == upstream_status
|
||||
assert proxy._record_request_outcome.await_count == 1
|
||||
outcome = proxy._record_request_outcome.await_args.args[0]
|
||||
assert outcome.status_code == upstream_status
|
||||
|
||||
|
||||
# ── env config path ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user