Files
headroom/tests/gateway/test_contract_response_half.py
Tejas Chopra 5cac7ab693 fix(gateway): report why /v1/compress/response rejected a body (#4081)
## Description

`POST /v1/compress/response` (the response half of the gateway contract)
caught every body-read failure and answered `"Invalid JSON in request
body."`. That covered malformed JSON, but also an oversize body, a
failed decompression, invalid UTF-8 and a JSON array. A Kong or LiteLLM
integrator debugging a 400 could not tell which one it was.

The other handlers (anthropic, openai, gemini, batch; about 10 call
sites) already use `helpers.invalid_request_body_message(exc)`. It picks
a fixed string by exception type, so it never echoes the exception text
that CodeQL's `py/stack-trace-exposure` flags, and it logs the exception
type on the proxy side. This handler now uses it too. That gives
specific messages for malformed JSON, a non-object, invalid UTF-8 and an
oversize body. A failed decompression (for example a bad gzip stream)
still gets the generic `Invalid request body`, as it does on every other
handler. A separate decompression message would mean changing the shared
helper for all handlers, so that is out of scope here.

The `if not isinstance(body, dict)` branch that followed is removed.
`_read_request_json` raises `RequestBodyNotObject` for a non-object
before it can return one, so the branch could not be reached. Its case
is now reported as `Invalid request body: must be a JSON object`.

The status code (400) and the error `type` (`invalid_request`) are
unchanged. Only the `message` text differs. No test, doc or consumer
repo matched on the old string: I grepped `tests`, `docs`, `headroom`
and the gateway integrations, which branch on the status code.

Refs #4136

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/gateway_turn.py`: `handle_compress_response` answers
400 with `invalid_request_body_message(exc)`, and the unreachable
non-dict check is gone.
- `tests/gateway/test_contract_response_half.py`: add
`test_invalid_body_400_says_why`, parametrized over three bodies. Each
case asserts a 400, `invalid_request`, and a message that names the
reason:
  - malformed JSON;
  - a JSON array;
  - invalid UTF-8.

  All three cases fail on main.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/gateway/test_contract_response_half.py -k invalid   # before the fix
FAILED tests/gateway/test_contract_response_half.py::test_invalid_body_400_says_why[malformed]
FAILED tests/gateway/test_contract_response_half.py::test_invalid_body_400_says_why[not-object]
FAILED tests/gateway/test_contract_response_half.py::test_invalid_body_400_says_why[not-utf8]
================== 3 failed, 6 passed, 28 deselected in 7.18s ==================
$ pytest -q -p no:cacheprovider tests/gateway/ tests/test_gateway_turn_unit.py tests/test_invalid_request_body_message.py   # after
================== 323 passed, 4 skipped in 171.41s (0:02:51) ==================
$ uvx ruff@0.16.8 check headroom/proxy/gateway_turn.py tests/gateway/test_contract_response_half.py
All checks passed!
$ mypy headroom/proxy/gateway_turn.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, branch checkout at ae6a419a, using
the repo .venv. A temp HOME, workspace and CODEX_HOME were used, with
offline mode on.
- Exact command / steps:
1. Ran the shared smoke harness, which starts a real proxy from this
checkout against a local fake upstream and sends one Anthropic and one
OpenAI request.
2. Started a real proxy with `python -m headroom.cli proxy --port <p>`
and used curl to POST five bodies to
`http://127.0.0.1:<p>/v1/compress/response`: four bad ones, and one
valid body with an unknown turn as a control.
- Observed result:
```text
branch: cleanup/proxy-core-a-compress-response-errors @ ae6a419a
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>):
(smoke dir: <tmp>)
```
```text
malformed JSON         -> {"error":{"type":"invalid_request","message":"Invalid request body: malformed JSON"}} HTTP400
JSON array             -> {"error":{"type":"invalid_request","message":"Invalid request body: must be a JSON object"}} HTTP400
invalid UTF-8          -> {"error":{"type":"invalid_request","message":"Invalid request body: not valid UTF-8 (possibly compressed?)"}} HTTP400
bad gzip               -> {"error":{"type":"invalid_request","message":"Invalid request body"}} HTTP400
valid, unknown turn    -> {"error":{"type":"unknown_turn","message":"No pending turn 't-x' (never registered, already finished, or expired)."}} HTTP404
```
Every bad body is a 400 with type `invalid_request`, and the message
names the reason. The exception is bad gzip, which gets the generic
message, as on every other handler. On main, all four said `Invalid JSON
in request body.`. The control request still reaches turn lookup (404
`unknown_turn`).
- Not tested: A live Kong or LiteLLM gateway. Both key on the status and
`type`, which are unchanged.

## Runtime Rollout Safety

- Rollout-managed feature(s): None (the gateway contract is controlled
by `HEADROOM_GATEWAY_CONTRACT`, unchanged).
- Minimum rollout channel: N/A; error-message text only.
- Stable/default behavior changed: Only the `error.message` text on a
400 for a bad body. The status, error type and every success path are
unchanged.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of a maintainer-requested codebase cleanup (dead code, duplication,
debuggability), split into small PRs for review. It is part of
maintainer-requested cleanup #4136.

No docs quote the old message, so none needed changing. The change is
small and self-explanatory, so no new comments were added.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-10-10 09:05:53 -07:00

477 lines
18 KiB
Python

"""Response half of the gateway turn contract: ``POST /v1/compress/response`` (spec section 2).
Invariants protected here:
* R-LOOKUP - unknown/expired turn -> 404 ``unknown_turn``; a completed turn is
gone (second post is 404).
* R-VALID - 400 on missing/invalid ``turn_id``, non-object ``response``,
non-object/negative/bool usage, and a missing ``response`` on a
turn that carries ``redrive``.
* R-DONE - shape of the ``done`` answer; ``response`` is null unless a hook
replaced it.
* R-USAGE - Anthropic / OpenAI / Kong usage shapes feed the session tracker
exactly like ``/v1/usage`` (same ``frozen_message_count``); a
usage with no cache signal is accepted but not applied.
* R-OUTCOME - with ``relay_usage`` the RequestOutcome is deferred and recorded
exactly once, completed with the provider's numbers; without it
the outcome is recorded at the request half as today; registry
expiry records the draft.
"""
from __future__ import annotations
import time
from typing import Any
import pytest
from tests.gateway.conftest import compress, count_loop_tasks, registry_of
from tests.gateway.samples import big_tool_history
RESPONSE_HALF = "/v1/compress/response"
DONE_KEYS = {
"action",
"turn_id",
"response",
"frozen_message_count",
"usage_applied",
"rounds",
"billed_usage",
}
def _open_turn(client, session_id: str | None = "resp-s", **gateway: Any) -> dict[str, Any]:
gateway.setdefault("can_relay_response", True)
body: dict[str, Any] = {"model": "gpt-4o", "messages": big_tool_history(), "gateway": gateway}
if session_id is not None:
body["config"] = {"session_id": session_id}
resp = compress(client, body)
assert resp.status_code == 200, resp.text
data = resp.json()
assert "relay_usage" in data["obligations"]
return data
def _post(client, payload: dict[str, Any]):
return client.post(RESPONSE_HALF, json=payload)
# --------------------------------------------------------------------------- #
# R-LOOKUP / R-VALID #
# --------------------------------------------------------------------------- #
def test_unknown_turn_is_404(headroom_client) -> None:
"""R-LOOKUP"""
resp = _post(headroom_client, {"turn_id": "0" * 32, "usage": {"prompt_tokens": 1}})
assert resp.status_code == 404, resp.text
assert resp.json()["error"]["type"] == "unknown_turn"
def test_unknown_turn_leaves_no_footprint(headroom_client) -> None:
"""R-LOOKUP: a flood of bogus ids must not grow the registry."""
registry = registry_of(headroom_client)
assert registry is not None
before = len(registry)
for i in range(10):
assert _post(headroom_client, {"turn_id": f"{i:032x}"}).status_code == 404
assert len(registry) == before
@pytest.mark.parametrize(
"payload",
[
{},
{"turn_id": None},
{"turn_id": 42},
{"turn_id": ""},
{"turn_id": "x" * 300},
],
ids=["missing", "null", "int", "empty", "too-long"],
)
def test_invalid_turn_id_is_400(headroom_client, payload: dict[str, Any]) -> None:
"""R-VALID"""
resp = _post(headroom_client, payload)
assert resp.status_code == 400, resp.text
assert resp.json()["error"]["type"] == "invalid_request"
def test_invalid_json_is_400(headroom_client) -> None:
"""R-VALID"""
resp = headroom_client.post(
RESPONSE_HALF, content=b"nope", headers={"content-type": "application/json"}
)
assert resp.status_code == 400
@pytest.mark.parametrize(
("content", "reason"),
[
(b"nope", "malformed JSON"),
(b"[1, 2]", "must be a JSON object"),
(b"\xff\xfe{}", "not valid UTF-8"),
],
ids=["malformed", "not-object", "not-utf8"],
)
def test_invalid_body_400_says_why(headroom_client, content: bytes, reason: str) -> None:
"""The 400 names the reason, like every other handler's body errors."""
resp = headroom_client.post(
RESPONSE_HALF, content=content, headers={"content-type": "application/json"}
)
assert resp.status_code == 400
error = resp.json()["error"]
assert error["type"] == "invalid_request"
assert reason in error["message"]
@pytest.mark.parametrize(
"extra",
[
{"response": "not-an-object"},
{"response": [1, 2]},
{"usage": "nope"},
{"usage": {"prompt_tokens": -1}},
{"usage": {"cache_read_input_tokens": True}},
{"usage": {"prompt_tokens_details": {"cached_tokens": -5}}},
],
ids=[
"response-str",
"response-list",
"usage-str",
"usage-negative",
"usage-bool",
"cached-negative",
],
)
def test_malformed_response_or_usage_is_400_and_keeps_turn(
headroom_client, extra: dict[str, Any]
) -> None:
"""R-VALID: a rejected relay must not consume the turn."""
turn = _open_turn(headroom_client)
resp = _post(headroom_client, {"turn_id": turn["turn_id"], **extra})
assert resp.status_code == 400, resp.text
assert resp.json()["error"]["type"] == "invalid_request"
assert registry_of(headroom_client).get(turn["turn_id"]) is not None
# --------------------------------------------------------------------------- #
# R-DONE #
# --------------------------------------------------------------------------- #
def test_done_with_null_response_and_turn_removed(headroom_client) -> None:
"""R-DONE + R-LOOKUP: no hook ran, so ``response`` is null; the turn is consumed."""
turn = _open_turn(headroom_client)
resp = _post(
headroom_client,
{
"turn_id": turn["turn_id"],
"status": 200,
"latency_ms": 12.5,
"model": "gpt-4o-2024-08-06",
"usage": {"prompt_tokens": 100, "completion_tokens": 7},
},
)
assert resp.status_code == 200, resp.text
data = resp.json()
assert set(data) == DONE_KEYS
assert data["action"] == "done"
assert data["turn_id"] == turn["turn_id"]
assert data["response"] is None
assert data["rounds"] == 0
assert data["usage_applied"] is False # no cache signal in that usage
assert data["billed_usage"] == {"input_tokens": 100, "output_tokens": 7}
assert isinstance(data["frozen_message_count"], int)
assert registry_of(headroom_client).get(turn["turn_id"]) is None
again = _post(headroom_client, {"turn_id": turn["turn_id"]})
assert again.status_code == 404
assert again.json()["error"]["type"] == "unknown_turn"
def test_done_without_session_has_null_frozen_count(headroom_client) -> None:
"""R-DONE: stateless turns carry no tracker, so ``frozen_message_count`` is null
and usage is never applied."""
turn = _open_turn(headroom_client, session_id=None)
resp = _post(
headroom_client,
{
"turn_id": turn["turn_id"],
"usage": {"input_tokens": 10, "output_tokens": 2, "cache_read_input_tokens": 5000},
},
)
assert resp.status_code == 200, resp.text
data = resp.json()
assert data["action"] == "done"
assert data["frozen_message_count"] is None
assert data["usage_applied"] is False
def test_done_with_minimal_payload(headroom_client) -> None:
"""R-DONE: ``turn_id`` alone is a valid relay (status defaults to 200)."""
turn = _open_turn(headroom_client)
resp = _post(headroom_client, {"turn_id": turn["turn_id"]})
assert resp.status_code == 200, resp.text
assert resp.json()["action"] == "done"
assert resp.json()["usage_applied"] is False
def test_done_leaves_no_tasks_behind(headroom_client) -> None:
"""R-DONE: a hook-less turn never starts a suspended runner, and after ``done``
the loop has exactly the tasks it had before."""
baseline = count_loop_tasks(headroom_client)
turn = _open_turn(headroom_client)
_post(headroom_client, {"turn_id": turn["turn_id"], "usage": {"prompt_tokens": 1}})
assert count_loop_tasks(headroom_client) <= baseline
# --------------------------------------------------------------------------- #
# R-USAGE #
# --------------------------------------------------------------------------- #
ANTHROPIC_WARM = {
"input_tokens": 200,
"output_tokens": 9,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 50_000,
}
OPENAI_WARM = {
"prompt_tokens": 50_200,
"completion_tokens": 9,
"prompt_tokens_details": {"cached_tokens": 50_000},
}
OPENAI_TOPLEVEL_CACHED = {"prompt_tokens": 50_200, "completion_tokens": 9, "cached_tokens": 50_000}
KONG_NESTED = {"usage": OPENAI_WARM} # Kong ai.proxy log statistics keep it nested
def _frozen_after_legacy_usage(client, history, usage: dict[str, Any]) -> int:
"""The yardstick: what ``/v1/usage`` does with the Anthropic-shaped equivalent."""
sid = f"legacy-{time.time_ns()}"
assert (
compress(
client, {"model": "gpt-4o", "messages": history, "config": {"session_id": sid}}
).status_code
== 200
)
resp = client.post("/v1/usage", json={"session_id": sid, "usage": usage})
assert resp.status_code == 200, resp.text
assert resp.json()["applied"] is True
return resp.json()["frozen_message_count"]
@pytest.mark.parametrize(
"usage",
[ANTHROPIC_WARM, OPENAI_WARM, OPENAI_TOPLEVEL_CACHED, KONG_NESTED],
ids=["anthropic", "openai", "openai-toplevel-cached", "kong-nested"],
)
def test_usage_shapes_apply_to_tracker_like_v1_usage(
headroom_client, usage: dict[str, Any]
) -> None:
"""R-USAGE: every accepted shape lands on the tracker and advances the frozen
count exactly as the equivalent ``/v1/usage`` relay does."""
history = big_tool_history()
expected = _frozen_after_legacy_usage(
headroom_client,
history,
{"cache_read_input_tokens": 0, "cache_creation_input_tokens": 50_000},
)
turn = _open_turn(headroom_client, session_id="usage-shape")
resp = _post(headroom_client, {"turn_id": turn["turn_id"], "usage": usage})
assert resp.status_code == 200, resp.text
data = resp.json()
assert data["usage_applied"] is True
assert data["frozen_message_count"] == expected
assert data["frozen_message_count"] >= 1
@pytest.mark.parametrize(
"usage",
[
{"prompt_tokens": 120, "completion_tokens": 3},
{"input_tokens": 120, "output_tokens": 3},
{"prompt_tokens": 120, "prompt_tokens_details": {"cached_tokens": 0}},
{"input_tokens": 120, "output_tokens": 3, "cache_read_input_tokens": 0},
],
ids=["openai-bare", "anthropic-bare", "openai-cached-zero", "anthropic-read-zero-only"],
)
def test_usage_without_cache_signal_is_accepted_but_not_applied(headroom_client, usage) -> None:
"""R-USAGE: unlike ``/v1/usage`` a signal-free relay is not a 400 - the response
half still completes the outcome - but it must not touch the tracker (applying
it would assert a fully-cold prefix the provider never reported)."""
turn = _open_turn(headroom_client, session_id="usage-nosignal")
before = turn["session"]["frozen_message_count"]
resp = _post(headroom_client, {"turn_id": turn["turn_id"], "usage": usage})
assert resp.status_code == 200, resp.text
data = resp.json()
assert data["action"] == "done"
assert data["usage_applied"] is False
tracker = headroom_client.app.state.proxy.session_tracker_store.peek(
"compress\x00usage-nosignal"
)
assert tracker is not None
assert tracker.get_frozen_message_count() == data["frozen_message_count"]
assert data["frozen_message_count"] <= max(before, len(big_tool_history()))
def test_both_cache_fields_zero_is_a_confirmed_cold_turn(headroom_client) -> None:
"""R-USAGE: both fields present and 0 is a genuine fully-cold signal (the same
rule ``/v1/usage`` applies) and IS applied."""
turn = _open_turn(headroom_client, session_id="usage-cold")
resp = _post(
headroom_client,
{
"turn_id": turn["turn_id"],
"usage": {
"input_tokens": 1,
"output_tokens": 1,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
assert resp.status_code == 200, resp.text
assert resp.json()["usage_applied"] is True
# --------------------------------------------------------------------------- #
# R-OUTCOME #
# --------------------------------------------------------------------------- #
def test_outcome_deferred_then_recorded_once_with_provider_numbers(
headroom_client, outcome_spy
) -> None:
"""R-OUTCOME: with ``relay_usage`` nothing is recorded at the request half; the
response half records exactly one outcome carrying output/input/cache tokens
and the gateway-measured latency."""
outcomes = outcome_spy(headroom_client)
turn = _open_turn(headroom_client, session_id="outcome-defer")
assert outcomes == [], "outcome must be deferred while relay_usage is pending"
resp = _post(
headroom_client,
{
"turn_id": turn["turn_id"],
"status": 200,
"latency_ms": 250.0,
"usage": {
"prompt_tokens": 1234,
"completion_tokens": 56,
"prompt_tokens_details": {"cached_tokens": 1000},
},
},
)
assert resp.status_code == 200, resp.text
assert len(outcomes) == 1
o = outcomes[0]
assert o.provider == "compress"
assert o.model == "gpt-4o"
assert o.output_tokens == 56
assert o.provider_input_tokens == 1234
assert o.cache_read_tokens == 1000
assert o.total_latency_ms >= 250.0
assert o.original_tokens == turn["tokens_before"]
assert o.optimized_tokens == turn["tokens_after"]
assert o.tokens_saved == turn["tokens_saved"]
assert o.status_code == 200
def test_outcome_anthropic_cache_fields(headroom_client, outcome_spy) -> None:
"""R-OUTCOME: Anthropic read + write buckets land on the outcome's cache fields."""
outcomes = outcome_spy(headroom_client)
turn = _open_turn(headroom_client, session_id="outcome-anthropic")
_post(
headroom_client,
{
"turn_id": turn["turn_id"],
"usage": {
"input_tokens": 300,
"output_tokens": 12,
"cache_read_input_tokens": 2000,
"cache_creation_input_tokens": 500,
},
},
)
assert len(outcomes) == 1
o = outcomes[0]
assert o.output_tokens == 12
assert o.cache_read_tokens == 2000
assert o.cache_write_tokens == 500
# Anthropic's input_tokens is the uncached remainder; the billed prompt is
# that plus both cache buckets, as the direct handler has always recorded.
assert o.provider_input_tokens == 2800
assert o.uncached_input_tokens == 300
def test_outcome_recorded_immediately_without_relay(headroom_client, outcome_spy) -> None:
"""R-OUTCOME: no ``relay_usage`` -> recorded at the request half, as today."""
outcomes = outcome_spy(headroom_client)
body = {
"model": "gpt-4o",
"messages": big_tool_history(),
"gateway": {"can_relay_response": False},
}
data = compress(headroom_client, body).json()
assert data["obligations"] == []
assert len(outcomes) == 1
assert outcomes[0].output_tokens == 0
def test_outcome_status_from_provider(headroom_client, outcome_spy) -> None:
"""R-OUTCOME: a provider 5xx relayed through the response half is recorded with
that status so it cannot inflate save-rate (same rule as the proxy path)."""
outcomes = outcome_spy(headroom_client)
turn = _open_turn(headroom_client, session_id="outcome-5xx")
resp = _post(headroom_client, {"turn_id": turn["turn_id"], "status": 529, "latency_ms": 5})
assert resp.status_code == 200, resp.text
assert resp.json()["action"] == "done"
assert len(outcomes) == 1
assert outcomes[0].status_code == 529
def test_expiry_records_draft_and_then_404(headroom_client, outcome_spy) -> None:
"""R-OUTCOME: a turn the gateway never completes is recorded as-is (output 0)
when the registry sweeps it, and a late relay is a 404."""
outcomes = outcome_spy(headroom_client)
turn = _open_turn(headroom_client, session_id="outcome-expire")
registry = registry_of(headroom_client)
assert registry is not None and registry.get(turn["turn_id"]) is not None
assert outcomes == []
# In production the sweep runs inside an HTTP handler, i.e. on the server
# loop, where the draft can be handed to the async outcome funnel. Run it
# there (the TestClient portal) rather than from the test thread, which has
# no loop and would only park the draft as an orphan.
deadline = time.time() + 10_000
headroom_client.portal.call(lambda: registry.sweep(deadline))
assert registry.get(turn["turn_id"]) is None
late = _post(headroom_client, {"turn_id": turn["turn_id"], "usage": {"prompt_tokens": 1}})
assert late.status_code == 404
assert late.json()["error"]["type"] == "unknown_turn"
# The record is scheduled as a task; the request above let the loop run it.
assert len(outcomes) == 1
assert outcomes[0].output_tokens == 0
assert outcomes[0].original_tokens == turn["tokens_before"]
def test_ttl_env_is_honoured(make_headroom_client, monkeypatch, outcome_spy) -> None:
"""R-OUTCOME: ``HEADROOM_GATEWAY_TURN_TTL_SECONDS`` bounds how long a pending
turn lives; a lazy sweep on the next lookup reclaims it."""
monkeypatch.setenv("HEADROOM_GATEWAY_TURN_TTL_SECONDS", "0")
client = make_headroom_client()
outcomes = outcome_spy(client)
turn = _open_turn(client, session_id="ttl-zero")
time.sleep(0.05)
late = _post(client, {"turn_id": turn["turn_id"], "usage": {"prompt_tokens": 1}})
assert late.status_code == 404, late.text
assert len(outcomes) == 1
def test_registry_stats_shape(headroom_client) -> None:
"""R-LOOKUP: ``stats()`` exposes at least the pending count for /stats and debugging."""
registry = registry_of(headroom_client)
assert registry is not None
turn = _open_turn(headroom_client, session_id="stats")
stats = registry.stats()
assert isinstance(stats, dict)
assert stats.get("pending") == len(registry) >= 1
_post(headroom_client, {"turn_id": turn["turn_id"]})
assert registry.stats().get("pending") == len(registry)