mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(ccr): answer headroom_retrieve on the direct chat path instead of forwarding it (#3816)
## Description `ccr_inject_tool` defaults to `True` and injection is permitted on any non-streaming chat turn, but the direct HTTP branch of `handle_openai_chat` had **no CCR tool-call handling at all** — the code said so itself, in the comment sitting where that handling belonged. Every other path resolves the call: | path | resolves | |---|---| | `headroom/gateway_turn.py:1003` | yes | | `headroom/proxy/handlers/gemini.py:837` | yes | | `headroom/proxy/handlers/anthropic.py:4286` | yes | | `handlers/openai.py` custom backend (`:4875`) | yes | | `handlers/openai.py` Responses API (`:6627`) | yes | | `handlers/openai.py` **direct chat** | **no** | So the proxy advertised a tool it would not answer. A model that called it sent the client a `tool_calls` entry naming a function the client never declared and cannot implement, with `finish_reason: "tool_calls"` — which stalls any agent loop driven off that field. This is the same failure the chat-streaming guard exists to prevent, on the side of the fork where that guard does not apply. Prerequisite for #3815. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/handlers/openai.py`: on the direct chat branch, detect CCR tool calls with `has_ccr_tool_calls` and resolve them via `CCRResponseHandler.handle_response` before the response is forwarded, driven by a continuation closure that re-POSTs through `_retry_request` to the same URL with non-streaming headers. - Accounting: fold the pre-continuation response into the existing `TurnHookUsage` (`record` + `settle`) so a retrieval turn's two upstream calls both reach `total_input_tokens`, instead of only the last one. - Forwarded body keeps the provider's own `usage` untouched — byte-faithful forwarding preserved; the extra call lands only in Headroom's internal accounting. - Resolution fails open: on any exception the model's original reply is forwarded rather than turning a usable turn into a 502. - `tests/test_direct_chat_ccr_resolution.py`: new, 4 tests. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_direct_chat_ccr_resolution.py -q 4 passed # 44-file sweep over CCR, the OpenAI handlers and forwarded tools arrays 831 passed, 7 skipped, 0 failed $ ruff check . && ruff format --check . All checks passed! 1659 files already formatted $ mypy headroom --ignore-missing-imports Success: no issues found in 543 source files ``` CI on `095aeaaa`: 27 checks pass (all four pytest shards, lint, commitlint, both CodeQL suites, wheel build, every audit), 12 skipped as not-applicable, 0 failures. ## Real Behavior Proof - Environment: local macOS, Python 3.12; FastAPI `TestClient` against the real `create_app` / `ProxyConfig` with `_retry_request` stubbed to serve a scripted upstream, plus the real `CompressionStore`. No live provider call. - Exact command / steps: wrote all four tests first and ran them against unmodified `origin/main` (`d13e1966`), then against the fix. Separately deleted only the `_hook_usage.record(_pre_ccr_json, ...)` fold to confirm the accounting test fails for the right reason. - Observed result: on unmodified main the client received the injected tool back — `assert 'headroom_retrieve' not in ['headroom_retrieve']` — and only one upstream call was made where a resolved turn makes two. After the fix: the client gets `content='there are 2 rows'`, two upstream calls, the continuation carries the retrieved original, the forwarded `usage.prompt_tokens` is still the provider's `400`, and `RequestOutcome.provider_input_tokens >= 100 + 400`. With only the fold deleted the accounting test failed with "cost tracking saw 400 input tokens for a turn that made two upstream calls costing 100 + 400; the first call was dropped". - Not tested: no run against a live OpenAI-compatible provider, and no end-to-end run with a real agent client — the evidence here is the test harness plus CI, not production traffic. Chat streaming is untouched and out of scope. ## Runtime Rollout Safety - Rollout-managed feature(s): None new. Reuses the existing `ccr_handle_responses` / `ccr_inject_tool` config already gating CCR everywhere else. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes — on the direct non-streaming chat path a `headroom_retrieve` call is now resolved in-band instead of being forwarded to the client. That is the bug being fixed; no other response shape changes, and a turn with no CCR tool call takes a byte-identical path. - Kill switch / disable path: `--no-ccr` (env `HEADROOM_NO_CCR`) sets `ccr_handle_responses=False`, which leaves `ccr_response_handler` as `None` (`headroom/proxy/server.py:1292`) and makes the new block inert — it is guarded on that handler being present. - Unsafe override required: No. - Qualification impact: None. No new config surface, no new env var, no change to compression behavior or savings accounting semantics — only that a previously-dropped upstream call is now counted. - Rollback path: Revert this commit. The change is additive and confined to one branch of `handle_openai_chat` plus one new test file; nothing persisted and no schema or config change to unwind. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` Documentation: no corresponding change — this fixes one path to match the documented behavior of every other path rather than altering the documented contract. ## Additional Notes Non-streaming only, by design. Chat streaming still refuses to inject and stays untouched; extending buffered CCR to it is step 2 of #3815, and this is its prerequisite. Independent of #3810 — that one controls *when* `headroom_retrieve` enters the tools array, this one controls *whether the proxy answers it* on one path. Zero file overlap, so the two can land in either order. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
66258c40a4
commit
126e144876
@@ -289,6 +289,35 @@ RESPONSES_USAGE_KEYS = {
|
||||
}
|
||||
|
||||
|
||||
class CCRContinuationHTTPError(Exception):
|
||||
"""A CCR continuation call came back non-2xx.
|
||||
|
||||
``_retry_request`` deliberately returns rather than raises for the statuses
|
||||
it will not retry — any 4xx, an exhausted 429/529, an exhausted 5xx — so a
|
||||
continuation failure is indistinguishable from success unless the status is
|
||||
checked. Parsing such a body and returning it as the model's answer would
|
||||
serialize an upstream *error* under the original call's ``200``.
|
||||
|
||||
Raising instead routes the turn into
|
||||
``CCRResponseHandler.handle_response``'s existing continuation-failure
|
||||
path, which forwards the response it already had. That keeps one behaviour
|
||||
for every way a continuation can fail: transport error, timeout, and now
|
||||
an error status.
|
||||
|
||||
Carries the status and nothing else. An upstream error body is untrusted
|
||||
content that routinely contains credential fragments, tenant identifiers
|
||||
and excerpts of the request, and the status alone identifies the failure.
|
||||
The message matters as much as any explicit log call: ``handle_response``
|
||||
logs ``repr(e)`` when a continuation raises
|
||||
(``ccr/response_handler.py:541``), so anything placed here reaches the log
|
||||
whether or not this module logs it too.
|
||||
"""
|
||||
|
||||
def __init__(self, status_code: int) -> None:
|
||||
super().__init__(f"CCR continuation returned HTTP {status_code}")
|
||||
self.status_code = status_code
|
||||
|
||||
|
||||
class TurnHookUsage:
|
||||
"""Upstream calls a turn hook caused that nothing else will account for.
|
||||
|
||||
@@ -5259,12 +5288,130 @@ class OpenAIHandlerMixin:
|
||||
_thinking = ThinkingTokens()
|
||||
total_latency = (time.time() - start_time) * 1000
|
||||
|
||||
# ── CCR retrieval resolution (direct chat path) ──────────
|
||||
# Every other path answers a `headroom_retrieve` call:
|
||||
# gateway_turn.py, the Gemini and Anthropic handlers, the custom
|
||||
# backend branch above, and Responses. This branch did not, so a
|
||||
# tool the proxy itself injects (`ccr_inject_tool` defaults True)
|
||||
# reached the client as a `tool_calls` entry for a function it
|
||||
# never declared and cannot implement — stalling any agent loop
|
||||
# driven off `finish_reason`.
|
||||
#
|
||||
# The continuation is a real, billed upstream call, so it is
|
||||
# recorded into `_hook_usage` exactly like a turn-hook re-drive:
|
||||
# `settle(final)` drops the one response the usage block below
|
||||
# reads and totals the rest. Counting only the last call would
|
||||
# let retrieval hide its own cost, which is the mistake the
|
||||
# matching comment on the re-drive block below warns about.
|
||||
_ccr_final_json: dict[str, Any] | None = None
|
||||
_ccr_cont_error: int | None = None
|
||||
if self.ccr_response_handler is not None and response.status_code == 200:
|
||||
try:
|
||||
_pre_ccr_json = response.json()
|
||||
except Exception: # pragma: no cover - non-JSON upstream
|
||||
_pre_ccr_json = None
|
||||
if _pre_ccr_json and self.ccr_response_handler.has_ccr_tool_calls(
|
||||
_pre_ccr_json, "openai"
|
||||
):
|
||||
logger.info(
|
||||
f"[{request_id}] CCR: retrieval tool call on the direct "
|
||||
"chat path, resolving before replying"
|
||||
)
|
||||
|
||||
async def _ccr_api_call_fn(
|
||||
msgs: list[dict[str, Any]],
|
||||
tls: list[dict[str, Any]] | None,
|
||||
) -> dict[str, Any]:
|
||||
continuation_body = {**body, "messages": msgs}
|
||||
if tls is not None:
|
||||
continuation_body["tools"] = tls
|
||||
# A continuation is always non-streaming, whatever
|
||||
# the client asked for.
|
||||
continuation_body.pop("stream", None)
|
||||
continuation_headers = {
|
||||
k: v
|
||||
for k, v in headers.items()
|
||||
if k.lower()
|
||||
not in (
|
||||
"accept",
|
||||
"content-type",
|
||||
"content-length",
|
||||
"content-encoding",
|
||||
)
|
||||
}
|
||||
continuation_headers["content-type"] = "application/json"
|
||||
continuation_headers["accept"] = "application/json"
|
||||
nonlocal _ccr_cont_error
|
||||
cont = await self._retry_request(
|
||||
"POST", url, continuation_headers, continuation_body
|
||||
)
|
||||
# `_retry_request` returns non-2xx verbatim for
|
||||
# everything it will not retry, so the status has to
|
||||
# be checked before the body is treated as an answer.
|
||||
# Without this an upstream 401, or an exhausted 429,
|
||||
# reaches the client as HTTP 200 carrying an error
|
||||
# object.
|
||||
if not 200 <= cont.status_code < 300:
|
||||
# Status only. The body is untrusted upstream
|
||||
# content and never leaves this branch.
|
||||
_ccr_cont_error = cont.status_code
|
||||
raise CCRContinuationHTTPError(cont.status_code)
|
||||
cont_json: dict[str, Any] = cont.json()
|
||||
_hook_usage.record(cont_json, **CHAT_USAGE_KEYS)
|
||||
return cont_json
|
||||
|
||||
try:
|
||||
_hook_usage.record(_pre_ccr_json, **CHAT_USAGE_KEYS)
|
||||
_ccr_final_json = await self.ccr_response_handler.handle_response(
|
||||
_pre_ccr_json,
|
||||
optimized_messages,
|
||||
tools,
|
||||
_ccr_api_call_fn,
|
||||
provider="openai",
|
||||
)
|
||||
# `handle_response` swallows a continuation failure
|
||||
# and hands back the response it already had, so a
|
||||
# failed turn is only detectable here. Identity —
|
||||
# not equality — is the signal that nothing was
|
||||
# resolved, and it also covers max-rounds
|
||||
# exhaustion. Dropping back to `None` makes the
|
||||
# return below forward the upstream bytes verbatim
|
||||
# instead of re-serializing a dict that came from
|
||||
# them, which is what byte-faithful forwarding
|
||||
# means on a path that changed nothing.
|
||||
if _ccr_cont_error is not None or _ccr_final_json is _pre_ccr_json:
|
||||
if _ccr_cont_error is not None:
|
||||
logger.warning(
|
||||
f"[{request_id}] CCR: continuation returned HTTP "
|
||||
f"{_ccr_cont_error}; forwarding the upstream reply "
|
||||
"unchanged"
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
f"[{request_id}] CCR: retrieval did not resolve; "
|
||||
"forwarding the upstream reply unchanged"
|
||||
)
|
||||
_ccr_final_json = None
|
||||
_hook_usage.settle(_pre_ccr_json)
|
||||
else:
|
||||
_hook_usage.settle(_ccr_final_json)
|
||||
except Exception as ccr_err:
|
||||
# Fail open to the model's own reply rather than 502 a
|
||||
# turn the client could still act on.
|
||||
logger.warning(
|
||||
f"[{request_id}] CCR: direct-path resolution failed "
|
||||
f"({type(ccr_err).__name__}: {ccr_err}); forwarding "
|
||||
"the upstream reply unchanged"
|
||||
)
|
||||
_ccr_final_json = None
|
||||
_hook_usage.settle(_pre_ccr_json)
|
||||
|
||||
total_input_tokens = optimized_tokens # fallback
|
||||
output_tokens = 0
|
||||
cache_read_tokens = 0
|
||||
resp_json = None
|
||||
try:
|
||||
resp_json = response.json()
|
||||
resp_json = _ccr_final_json if _ccr_final_json is not None else response.json()
|
||||
usage = resp_json.get("usage", {})
|
||||
# Coerce present-but-null counts: the arithmetic below
|
||||
# (`_infer_openai_cache_write_tokens`, `max(...)`) runs
|
||||
@@ -5499,7 +5646,11 @@ class OpenAIHandlerMixin:
|
||||
)
|
||||
|
||||
return Response(
|
||||
content=response.content,
|
||||
content=(
|
||||
json.dumps(_ccr_final_json).encode()
|
||||
if _ccr_final_json is not None
|
||||
else response.content
|
||||
),
|
||||
status_code=response.status_code,
|
||||
headers=response_headers,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,420 @@
|
||||
"""The direct chat-completions path must answer the retrieval tool it injects.
|
||||
|
||||
`headroom_retrieve` is injected into a non-streaming chat request by default
|
||||
(`ccr_inject_tool=True`), but `handlers/openai.py`'s direct HTTP branch had no
|
||||
CCR tool-call handling at all — the code said so itself. Every other path
|
||||
resolves the call: `gateway_turn.py:1003`, `handlers/gemini.py:837`,
|
||||
`handlers/anthropic.py:4286`, `handlers/openai.py` for the custom backend and
|
||||
for Responses. Only the direct chat branch did not.
|
||||
|
||||
So the proxy advertised a tool it would not answer. A model that called it sent
|
||||
the client a `tool_calls` entry for a tool the client never declared and cannot
|
||||
implement, which stalls any agent loop driven off `finish_reason`.
|
||||
|
||||
The accounting pair is the part worth keeping honest. A resolved turn makes two
|
||||
billed upstream calls, and the continuation resends the whole conversation, so
|
||||
counting only the last one hides roughly half the turn. The split matters too:
|
||||
the forwarded body keeps the provider's own usage untouched (byte-faithful
|
||||
forwarding), while Headroom's cost tracking folds in the extra call — the same
|
||||
division the turn-hook re-drive path already uses via `TurnHookUsage`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
|
||||
fastapi = pytest.importorskip("fastapi")
|
||||
httpx = pytest.importorskip("httpx")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from headroom.cache.compression_store import ( # noqa: E402
|
||||
get_compression_store,
|
||||
reset_compression_store,
|
||||
)
|
||||
from headroom.ccr.tool_injection import CCR_TOOL_NAME # noqa: E402
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
|
||||
ORIGINAL = "row 1: the original uncompressed rows\nrow 2: more of them"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def reset_store():
|
||||
reset_compression_store()
|
||||
yield
|
||||
reset_compression_store()
|
||||
|
||||
|
||||
def _stored_hash() -> str:
|
||||
return get_compression_store().store(
|
||||
original=ORIGINAL,
|
||||
compressed="[2 items compressed to 0]",
|
||||
original_item_count=2,
|
||||
compressed_item_count=0,
|
||||
)
|
||||
|
||||
|
||||
def _config() -> ProxyConfig:
|
||||
# No backend configured -> the direct OpenAI HTTP path.
|
||||
return ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
ccr_handle_responses=True,
|
||||
ccr_inject_tool=True,
|
||||
)
|
||||
|
||||
|
||||
def _retrieve_call_response(hash_key: str) -> dict:
|
||||
"""An upstream reply in which the model asks to retrieve."""
|
||||
return {
|
||||
"id": "chatcmpl-1",
|
||||
"object": "chat.completion",
|
||||
"model": "gpt-4o",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": CCR_TOOL_NAME,
|
||||
"arguments": json.dumps({"hash": hash_key}),
|
||||
},
|
||||
}
|
||||
],
|
||||
},
|
||||
"finish_reason": "tool_calls",
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 100, "completion_tokens": 20, "total_tokens": 120},
|
||||
}
|
||||
|
||||
|
||||
def _final_response() -> dict:
|
||||
"""The continuation reply, after the tool result was supplied."""
|
||||
return {
|
||||
"id": "chatcmpl-2",
|
||||
"object": "chat.completion",
|
||||
"model": "gpt-4o",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {"role": "assistant", "content": "there are 2 rows"},
|
||||
"finish_reason": "stop",
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 400, "completion_tokens": 10, "total_tokens": 410},
|
||||
}
|
||||
|
||||
|
||||
class _CollectHandler(logging.Handler):
|
||||
"""Collect `headroom.*` records regardless of propagation.
|
||||
|
||||
`caplog` attaches to the root logger, but the proxy's own startup
|
||||
(`helpers._setup_file_logging`) sets `propagate = False` on `headroom`, so
|
||||
records never reach root — and it does that *during* `create_app`, after
|
||||
conftest's autouse reset has already run. A handler on the `headroom`
|
||||
logger itself still fires, because `propagate` only governs whether records
|
||||
are passed further up. Anything asserting on log contents here has to go
|
||||
through this, or it silently asserts against an empty string.
|
||||
"""
|
||||
|
||||
def __init__(self, sink: list[str]) -> None:
|
||||
super().__init__(level=logging.DEBUG)
|
||||
self._sink = sink
|
||||
self.setFormatter(logging.Formatter("%(name)s %(levelname)s %(message)s"))
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
try:
|
||||
self._sink.append(self.format(record))
|
||||
except Exception: # pragma: no cover - a test handler must not raise
|
||||
pass
|
||||
|
||||
|
||||
def _run(
|
||||
upstream_sequence: list, log_capture: list[str] | None = None
|
||||
) -> tuple[httpx.Response, list[dict], list]:
|
||||
"""Post one chat turn, serving `upstream_sequence` to successive calls.
|
||||
|
||||
Each entry is either a payload dict (served as `200`) or an explicit
|
||||
`(status, payload)` pair — non-2xx continuations are the whole point of the
|
||||
status-handling tests, and `_retry_request` hands those back without
|
||||
raising.
|
||||
|
||||
Returns the response, the request bodies that went upstream, and the
|
||||
RequestOutcome objects the proxy recorded.
|
||||
"""
|
||||
sent: list[dict] = []
|
||||
outcomes: list = []
|
||||
remaining = list(upstream_sequence)
|
||||
|
||||
def _next() -> tuple[int, dict]:
|
||||
entry = remaining.pop(0) if remaining else upstream_sequence[-1]
|
||||
if isinstance(entry, tuple):
|
||||
return entry
|
||||
return 200, entry
|
||||
|
||||
async def fake_retry(method, url, headers, req_body, *args, **kwargs):
|
||||
sent.append(req_body)
|
||||
status, payload = _next()
|
||||
return httpx.Response(status, json=payload, headers={"content-type": "application/json"})
|
||||
|
||||
app = create_app(_config())
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy._retry_request = fake_retry
|
||||
|
||||
# The continuation on this path may go out through the shared client
|
||||
# rather than _retry_request; capture both so the test does not depend
|
||||
# on which transport the implementation picks.
|
||||
class _FakeHTTPClient:
|
||||
async def post(self, url, content=None, headers=None, **kwargs): # noqa: ANN001
|
||||
import json as _json
|
||||
|
||||
sent.append(_json.loads(content) if content else {})
|
||||
status, payload = _next()
|
||||
return httpx.Response(
|
||||
status, json=payload, headers={"content-type": "application/json"}
|
||||
)
|
||||
|
||||
proxy.http_client = _FakeHTTPClient()
|
||||
|
||||
_real_record = proxy._record_request_outcome
|
||||
|
||||
async def _capture(outcome, *args, **kwargs): # noqa: ANN001
|
||||
outcomes.append(outcome)
|
||||
return await _real_record(outcome, *args, **kwargs)
|
||||
|
||||
proxy._record_request_outcome = _capture
|
||||
|
||||
# After startup, so the proxy's own logging setup cannot undo it.
|
||||
_handler: logging.Handler | None = None
|
||||
if log_capture is not None:
|
||||
_headroom_logger = logging.getLogger("headroom")
|
||||
_headroom_logger.disabled = False
|
||||
_headroom_logger.setLevel(logging.DEBUG)
|
||||
_handler = _CollectHandler(log_capture)
|
||||
_headroom_logger.addHandler(_handler)
|
||||
|
||||
try:
|
||||
resp = client.post(
|
||||
"/v1/chat/completions",
|
||||
json={
|
||||
"model": "gpt-4o",
|
||||
"messages": [{"role": "user", "content": "how many rows?"}],
|
||||
"stream": False,
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {"name": "Read", "description": "read a file"},
|
||||
}
|
||||
],
|
||||
},
|
||||
headers={"Authorization": "Bearer test-key", "x-api-key": "test-key"},
|
||||
)
|
||||
finally:
|
||||
if _handler is not None:
|
||||
logging.getLogger("headroom").removeHandler(_handler)
|
||||
return resp, sent, outcomes
|
||||
|
||||
|
||||
def test_a_retrieval_call_is_resolved_before_the_client_sees_it():
|
||||
"""The client must never receive a headroom_retrieve tool call."""
|
||||
hash_key = _stored_hash()
|
||||
resp, sent, _outcomes = _run([_retrieve_call_response(hash_key), _final_response()])
|
||||
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
message = body["choices"][0]["message"]
|
||||
|
||||
tool_calls = message.get("tool_calls") or []
|
||||
names = [tc.get("function", {}).get("name") for tc in tool_calls]
|
||||
assert CCR_TOOL_NAME not in names, (
|
||||
"the proxy injected headroom_retrieve and then handed the model's call "
|
||||
"straight to the client, which cannot implement it"
|
||||
)
|
||||
assert message.get("content") == "there are 2 rows"
|
||||
assert len(sent) == 2, f"expected an original call plus a continuation, got {len(sent)}"
|
||||
|
||||
|
||||
def test_the_continuation_carries_the_retrieved_content():
|
||||
"""The retrieved original has to reach the model, not just be looked up."""
|
||||
hash_key = _stored_hash()
|
||||
_resp, sent, _outcomes = _run([_retrieve_call_response(hash_key), _final_response()])
|
||||
|
||||
assert len(sent) == 2
|
||||
continuation = sent[1]
|
||||
serialized = str(continuation)
|
||||
assert ORIGINAL.split("\n")[0] in serialized, (
|
||||
"the continuation request did not carry the retrieved original content"
|
||||
)
|
||||
|
||||
|
||||
def test_the_forwarded_body_keeps_the_provider_usage_untouched():
|
||||
"""Byte-faithful forwarding: the client sees the provider's own final usage.
|
||||
|
||||
Rewriting `usage` in the forwarded body would misreport what the upstream
|
||||
call actually returned. Extra calls belong in Headroom's own accounting,
|
||||
which the next test covers — this is the same split the turn-hook re-drive
|
||||
path already uses.
|
||||
"""
|
||||
hash_key = _stored_hash()
|
||||
resp, _sent, _outcomes = _run([_retrieve_call_response(hash_key), _final_response()])
|
||||
assert (resp.json().get("usage") or {}).get("prompt_tokens") == 400
|
||||
|
||||
|
||||
ERROR_BODY = {"error": {"message": "invalid api key", "type": "invalid_request_error"}}
|
||||
|
||||
# A value that must never be logged. Upstream error bodies are untrusted and
|
||||
# routinely carry credential fragments, tenant identifiers and excerpts of the
|
||||
# request that produced them.
|
||||
SENTINEL = "sk-live-SENTINEL-must-never-be-logged-8f3a1c"
|
||||
ERROR_BODY_WITH_SECRET = {
|
||||
"error": {
|
||||
"message": f"authentication failed for key {SENTINEL} (tenant acme-prod-42)",
|
||||
"type": "invalid_request_error",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [401, 429, 500])
|
||||
def test_a_failed_continuation_body_never_reaches_the_logs(status):
|
||||
"""The status is diagnostic enough; the body is untrusted content.
|
||||
|
||||
Two paths would leak it, and both have to stay closed:
|
||||
|
||||
* anything this module logs itself, and
|
||||
* the exception's own message — `handle_response` logs `repr(e)` when a
|
||||
continuation raises (`ccr/response_handler.py:541`), so a preview placed
|
||||
on the exception reaches the log even if this module never logs it.
|
||||
|
||||
The positive assertion at the end is load-bearing: it is what proves the
|
||||
capture is actually wired to the `headroom` logger. Without it the negative
|
||||
assertions pass against an empty string and the test is worthless.
|
||||
"""
|
||||
hash_key = _stored_hash()
|
||||
logs: list[str] = []
|
||||
resp, sent, _outcomes = _run(
|
||||
[_retrieve_call_response(hash_key), (status, ERROR_BODY_WITH_SECRET)],
|
||||
log_capture=logs,
|
||||
)
|
||||
text = "\n".join(logs)
|
||||
|
||||
assert len(sent) == 2, "the continuation should still have been attempted"
|
||||
# Proves the capture works, so the assertions below mean something.
|
||||
assert f"HTTP {status}" in text, (
|
||||
f"the {status} status was not logged, so either the failure is invisible "
|
||||
"or this test is not capturing headroom's logs at all"
|
||||
)
|
||||
assert SENTINEL not in text, (
|
||||
"an upstream error body reached the logs; even a bounded prefix is an "
|
||||
"exfiltration path for credentials and tenant identifiers"
|
||||
)
|
||||
assert "acme-prod-42" not in text, "a tenant identifier reached the logs"
|
||||
assert SENTINEL not in resp.text, "the upstream error body reached the client"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"status",
|
||||
[
|
||||
401, # any 4xx: `_retry_request` returns it without raising
|
||||
429, # retries exhausted, returned verbatim
|
||||
500, # retries exhausted, returned via the HTTPStatusError branch
|
||||
],
|
||||
)
|
||||
def test_a_non_2xx_continuation_never_reaches_the_client_as_200(status):
|
||||
"""An upstream error must not be served under the first call's `200`.
|
||||
|
||||
`_retry_request` deliberately does not raise for anything it will not retry,
|
||||
so a continuation failure looks exactly like a success unless the status is
|
||||
checked. Parsing that body and returning it as the model's answer produced
|
||||
an HTTP 200 whose payload was an error object with no `choices` at all — a
|
||||
shape no client expects, and one that hides an auth or quota failure.
|
||||
"""
|
||||
hash_key = _stored_hash()
|
||||
resp, sent, _outcomes = _run([_retrieve_call_response(hash_key), (status, ERROR_BODY)])
|
||||
|
||||
assert len(sent) == 2, "the continuation should still have been attempted"
|
||||
body = resp.json()
|
||||
assert "error" not in body, (
|
||||
f"a {status} continuation body was forwarded as the model's reply "
|
||||
f"under HTTP {resp.status_code}"
|
||||
)
|
||||
assert body.get("choices"), "the forwarded body lost its choices"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [401, 429, 500])
|
||||
def test_a_failed_continuation_forwards_the_original_reply_unchanged(status):
|
||||
"""Fail open, and byte-for-byte.
|
||||
|
||||
`handle_response` swallows a continuation failure and returns the response
|
||||
it already had, which is the same contract a transport error or timeout on
|
||||
that call already gets. The client therefore sees the model's original
|
||||
reply — including the unresolved tool call it cannot run, which is strictly
|
||||
better than an error object dressed as a completion, and is the documented
|
||||
behaviour for every other way a continuation can fail.
|
||||
|
||||
Byte fidelity matters here too: nothing was resolved, so the upstream bytes
|
||||
must be forwarded as they arrived rather than re-serialized from a parsed
|
||||
dict.
|
||||
"""
|
||||
hash_key = _stored_hash()
|
||||
original = _retrieve_call_response(hash_key)
|
||||
resp, _sent, _outcomes = _run([original, (status, ERROR_BODY)])
|
||||
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == original, "the original upstream reply was not forwarded unchanged"
|
||||
assert resp.content == httpx.Response(200, json=original).content, (
|
||||
"the body was re-serialized instead of forwarded verbatim"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [401, 429, 500])
|
||||
def test_a_failed_continuation_is_not_billed(status):
|
||||
"""A non-2xx continuation returns no usage, so it must not be counted.
|
||||
|
||||
Only the first call was billed. Recording the error response — or settling
|
||||
against the wrong object — would inflate the turn with tokens that were
|
||||
never charged, which is the same class of bug in the opposite direction as
|
||||
the dropped first call.
|
||||
"""
|
||||
hash_key = _stored_hash()
|
||||
_resp, sent, outcomes = _run([_retrieve_call_response(hash_key), (status, ERROR_BODY)])
|
||||
|
||||
assert len(sent) == 2
|
||||
assert outcomes, "no RequestOutcome was recorded"
|
||||
recorded = outcomes[-1].provider_input_tokens
|
||||
assert recorded == 100, (
|
||||
f"cost tracking saw {recorded} input tokens; only the first call "
|
||||
"(100) was billed, the continuation returned an error with no usage"
|
||||
)
|
||||
|
||||
|
||||
def test_headroom_accounting_counts_both_upstream_calls():
|
||||
"""A retrieval turn makes two billed calls; cost tracking must see both.
|
||||
|
||||
`handle_response` replaces the response rather than merging usage, so the
|
||||
pre-continuation call's tokens are invisible unless they are folded in
|
||||
explicitly. The continuation resends the whole conversation, so counting
|
||||
only the last call hides roughly half the turn — the same mistake the
|
||||
re-drive block warns about ("counting only the last one lets the feature
|
||||
hide its own overhead behind the saving it is claiming").
|
||||
"""
|
||||
hash_key = _stored_hash()
|
||||
_resp, sent, outcomes = _run([_retrieve_call_response(hash_key), _final_response()])
|
||||
assert len(sent) == 2
|
||||
assert outcomes, "no RequestOutcome was recorded"
|
||||
|
||||
recorded = outcomes[-1].provider_input_tokens
|
||||
assert recorded >= 100 + 400, (
|
||||
f"cost tracking saw {recorded} input tokens for a turn that made two "
|
||||
"upstream calls costing 100 + 400; the first call was dropped"
|
||||
)
|
||||
Reference in New Issue
Block a user