mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description A corrupt TLS record after the handshake surfaces as a raw `ssl.SSLError` (for example `[SSL: RECORD_LAYER_FAILURE] record layer failure`, or the `SSLV3_ALERT_BAD_RECORD_MAC` bursts in #1615). httpcore maps `ssl.SSLError` to `ConnectError` only in `start_tls`; its `read`/`write` maps cover only anyio's resource errors and timeouts, and anyio's `TLSStream` re-raises a non-EOF `SSLError` as it is. So the error is not an `httpx.TransportError`, and the #1639 retry in `_retry_request` and `_stream_response` lets it through. That error alone does not say whether upstream received the request. This PR retries it only when httpcore reports that writing the request failed: its documented `trace` request extension emits `send_request_headers.failed` or `send_request_body.failed`. Upstream then never had a complete request to run. A TLS error raised after the request was written propagates, as on main, so a POST upstream may have accepted is never sent twice. In the streaming path it still ends as the SSE 502, now with `upstream_tls_error` instead of a generic `api_error`. What this no longer fixes: the case the first version of this PR targeted, a corrupt record in the reply on a reused HTTP/1.1 keep-alive connection, happens after the request was written. It is ambiguous, so it still fails the request (502), as on main. What it does fix: a TLS failure while the request is still being written, for example on a shared HTTP/2 connection where a large body waits for flow-control window and the read for that window hits the corrupt record. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `helpers.py`: `RequestWriteTrace`, an httpcore `trace` callback that records whether `send_request_headers` or `send_request_body` failed. `forbids_retry(exc)` is true for a raw `ssl.SSLError` raised after a complete write. - `server.py` (`_retry_request`) and `handlers/streaming.py` (`_stream_response`): each attempt attaches a fresh trace, and a raw `ssl.SSLError` is retried only when that trace shows the write failed. Every other retry is unchanged. The streaming exhaustion handler still catches `ssl.SSLError`, for the 502 `upstream_tls_error` SSE body. - `tests/test_h2_stream_reset_retry.py`: the SSL cases now run through real httpcore over a fake socket. An accepted POST sends one POST and propagates, in the buffered and the streaming path. A failed write is retried, in both paths. The review's harness (an `httpx.MockTransport` that records the POST, then raises `ssl.SSLError`) sends one POST. - Three test fakes of `http_client.post` accept the new `extensions` kwarg. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_h2_stream_reset_retry.py -q 11 passed in 0.39s # source reverted to the previous head's retry (240680df), tests kept: FAILED tests/test_h2_stream_reset_retry.py::test_retry_request_does_not_replay_a_post_upstream_received (assert 2 == 1) FAILED tests/test_h2_stream_reset_retry.py::test_stream_does_not_replay_a_post_upstream_received (assert 2 == 1) FAILED tests/test_h2_stream_reset_retry.py::test_ssl_error_without_write_evidence_is_not_retried (assert 2 == 1) 3 failed, 8 passed # source as on upstream/main (no SSL retry), tests kept: FAILED tests/test_h2_stream_reset_retry.py::test_retry_request_retries_ssl_error_while_writing_the_request FAILED tests/test_h2_stream_reset_retry.py::test_stream_does_not_replay_a_post_upstream_received (raw SSLError escapes; no upstream_tls_error 502) FAILED tests/test_h2_stream_reset_retry.py::test_stream_retries_ssl_error_while_writing_the_request 3 failed, 8 passed $ uv run --frozen --extra dev pytest <134 files: the tests this PR and main changed since200c838c, every test importing handlers.streaming, and every test calling _retry_request, _stream_response or a fake http_client> -q 2457 passed, 2 failed, 22 skipped in 178s # both failures are timing assertions in test_read_maturation_handler_nobust.py, which replaces # _retry_request with a mock. Run alone 3 times, that file fails 1 test in 3 of 3 runs on this head # and in 2 of 3 runs with server.py, streaming.py and helpers.py as on upstream/main. $ uvx ruff@0.17.3 check <touched files> All checks passed! $ uvx ruff@0.17.3 format --check <touched files> already formatted $ mypy headroom/proxy/helpers.py headroom/proxy/server.py headroom/proxy/handlers/streaming.py --ignore-missing-imports Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS 27.2 (Apple Silicon), Python 3.13, httpx 0.28.1, httpcore 1.0.9, anyio 4.14.2. `headroom proxy --port 6811 --no-rate-limit --anthropic-api-url https://127.0.0.1:8445` (plus `--no-http2` in run 1), a clean temp HOME per run, `HEADROOM_CA_BUNDLE` pointing at a local test CA. Each run starts its own stub and proxy and stops both. - Exact command / steps: Controlled upstream: a local TLS stub, not a real provider. Run 1 (HTTP/1.1, accepted POST): the stub reads the whole request and counts it, then writes a corrupt TLS application-data record instead of a reply and closes; a byte-identical replay of a body it has seen gets a normal 200. One buffered and one streaming `POST /v1/messages` through the proxy, on upstream/main9f2cc017, on the previous head240680dfand on this head. Run 2 (HTTP/2, write failure): on the first connection the stub grants no flow-control window beyond the default 65,535 bytes, so a 418,702-byte streaming POST is still being written when the stub writes the corrupt record; later connections are served normally. On main and on this head. - Observed result: Run 1: main and this head send each POST once and return 502;240680dfsends each POST twice, byte-identical, and returns the second reply as 200. Run 2: main returns 502 `api_error`; this head retries on a fresh connection and returns 200, and the stub received one complete POST. - Not tested: A real provider sending a corrupt record, and a real HTTP/2 connection shared by several streams when one of them hits it. ```text Run 1: controlled upstream, HTTP/1.1, TLS broken after the whole POST was received version request POSTs the stub received client got upstream/main buffered 1 HTTP 502 api_error upstream/main streaming 1 HTTP 502 api_error240680dfbuffered 2 (byte-identical) HTTP 200 (reply to the replay)240680dfstreaming 2 (byte-identical) HTTP 200 (reply to the replay) this head buffered 1 HTTP 502 api_error this head streaming 1 HTTP 502, SSE error event with "code": "upstream_tls_error" Run 2: controlled upstream, HTTP/2, TLS broken at 65,535 of 418,702 body bytes upstream/main streaming 1 incomplete HTTP 502 api_error this head streaming 1 incomplete, 1 complete HTTP 200 (message_start ... message_stop) WARNING [hr_1794561952_000001] Connection error to upstream API (attempt 1/3): SSLError(1, '[SSL: RECORD_LAYER_FAILURE] record layer failure (_ssl.c:2660)'); retrying in 899ms ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes. A raw TLS error raised while the request is being written is retried on a fresh connection, like other transport errors. A raw TLS error after the request was written is not retried, as on main; in the streaming path it now ends as a 502 `upstream_tls_error` SSE error instead of a 502 `api_error`. - Kill switch / disable path: `--retry-max-attempts 1` (`HEADROOM_RETRY_MAX_ATTEMPTS=1`) disables the retry, as for every transport error. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes - Changed after review: the first version retried any raw `ssl.SSLError` before the first byte reached the client. JerrettDavis and Devin showed that replays a POST upstream may already have accepted. This version retries only a failed write, and the title now says so. - No idempotency key is added. The retry rests only on httpcore reporting that the request write failed. - Not changed here: the existing #1639 retries of `httpx.TransportError` (for example a `ReadError` or `ReadTimeout` after the request was written) carry the same after-the-write ambiguity. This PR leaves them as they are. - Companion PR #4775 is independent of this change. The cancellation-stress numbers quoted in the earlier version of this body were measured with the broader retry, so I removed them. - Documentation: no doc describes the retry classes, so there is nothing to update. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
1933 lines
68 KiB
Python
1933 lines
68 KiB
Python
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import base64
|
|
import builtins
|
|
import json
|
|
from types import SimpleNamespace
|
|
from unittest.mock import patch
|
|
|
|
import httpx
|
|
import pytest
|
|
from fastapi.responses import StreamingResponse
|
|
|
|
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin, _is_googleapis_endpoint
|
|
from headroom.proxy.handlers.openai import (
|
|
OpenAIHandlerMixin,
|
|
_decode_openai_bearer_payload,
|
|
_passthrough_usage_from_json,
|
|
_prefers_http1_passthrough,
|
|
prefers_http1_passthrough,
|
|
)
|
|
from headroom.proxy.handlers.streaming import StreamingMixin
|
|
from headroom.proxy.helpers import (
|
|
_headroom_bypass_enabled,
|
|
is_third_party_model,
|
|
relocate_system_messages_to_top_level,
|
|
)
|
|
from headroom.proxy.outcome import emit_request_outcome
|
|
from headroom.proxy.server import HeadroomProxy
|
|
|
|
|
|
def _jwt(payload: object) -> str:
|
|
header = {"alg": "none", "typ": "JWT"}
|
|
|
|
def encode(part: object) -> str:
|
|
raw = json.dumps(part, separators=(",", ":")).encode("utf-8")
|
|
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
|
|
|
|
return f"{encode(header)}.{encode(payload)}."
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("url", "expected"),
|
|
[
|
|
("https://us-central1-aiplatform.googleapis.com/v1", True),
|
|
("https://googleapis.com/v1", True),
|
|
("https://AIPLATFORM.GOOGLEAPIS.COM./v1", True),
|
|
("https://googleapis.com.example.test/v1", False),
|
|
("https://notgoogleapis.com/v1", False),
|
|
("https://googleapis.com@attacker.test/v1", False),
|
|
("not a url", False),
|
|
("", False),
|
|
],
|
|
)
|
|
def test_googleapis_endpoint_gate_uses_hostname_boundary(url: str, expected: bool) -> None:
|
|
assert _is_googleapis_endpoint(url) is expected
|
|
|
|
|
|
class _ImageCompressor:
|
|
def __init__(self, compressed_message):
|
|
self._compressed_message = compressed_message
|
|
|
|
def compress(self, messages, provider): # noqa: ANN001, ANN201
|
|
assert provider == "anthropic"
|
|
return [self._compressed_message]
|
|
|
|
|
|
class _FreshCompressor:
|
|
instances = 0
|
|
|
|
def __init__(self):
|
|
type(self).instances += 1
|
|
|
|
|
|
class _TimeoutHttpClient:
|
|
async def request(self, **kwargs): # noqa: ANN001, ANN201
|
|
raise httpx.ConnectTimeout("connect timed out")
|
|
|
|
|
|
class _RecordingHttpClient:
|
|
def __init__(self, label: str) -> None:
|
|
self.label = label
|
|
self.calls = 0
|
|
|
|
async def request(self, **kwargs): # noqa: ANN001, ANN201
|
|
self.calls += 1
|
|
request = httpx.Request(kwargs["method"], kwargs["url"])
|
|
return httpx.Response(
|
|
200,
|
|
request=request,
|
|
headers={"content-type": "application/json"},
|
|
json={"client": self.label},
|
|
)
|
|
|
|
|
|
class _ChatGPTAccountRequest:
|
|
method = "GET"
|
|
headers = {}
|
|
url = SimpleNamespace(path="/backend-api/me", query="")
|
|
|
|
async def body(self) -> bytes:
|
|
return b""
|
|
|
|
|
|
class _PassthroughRequest:
|
|
method = "GET"
|
|
headers = {}
|
|
url = SimpleNamespace(path="/some/other/path", query="")
|
|
|
|
async def body(self) -> bytes:
|
|
return b""
|
|
|
|
|
|
class _VertexPassthroughRequest:
|
|
method = "POST"
|
|
headers = {}
|
|
url = SimpleNamespace(
|
|
path="/v1/projects/p/locations/us-central1/publishers/google/models/gemini-2.0-flash:generateContent",
|
|
query="",
|
|
)
|
|
|
|
async def body(self) -> bytes:
|
|
return b'{"contents":[]}'
|
|
|
|
|
|
class _VertexStreamPassthroughRequest:
|
|
method = "POST"
|
|
headers = {}
|
|
url = SimpleNamespace(
|
|
path="/v1/projects/p/locations/us-central1/publishers/google/models/gemini-2.0-flash:streamGenerateContent",
|
|
query="alt=sse",
|
|
)
|
|
|
|
async def body(self) -> bytes:
|
|
return b'{"contents":[]}'
|
|
|
|
|
|
class _VertexGeminiImageRequest:
|
|
method = "POST"
|
|
headers = {}
|
|
query_params = {}
|
|
scope: dict = {"type": "http", "method": "POST"}
|
|
url = SimpleNamespace(
|
|
path="/v1/projects/p/locations/us-central1/publishers/google/models/gemini-2.0-flash:generateContent",
|
|
query="",
|
|
)
|
|
|
|
async def stream(self):
|
|
yield await self.body()
|
|
|
|
async def body(self) -> bytes:
|
|
return json.dumps(
|
|
{
|
|
"contents": [
|
|
{
|
|
"role": "user",
|
|
"parts": [
|
|
{
|
|
"inlineData": {
|
|
"mimeType": "image/png",
|
|
"data": "aW1hZ2U=",
|
|
}
|
|
}
|
|
],
|
|
}
|
|
]
|
|
}
|
|
).encode("utf-8")
|
|
|
|
|
|
class _VertexUsageClient:
|
|
async def request(self, **kwargs): # noqa: ANN001, ANN201
|
|
request = httpx.Request(kwargs["method"], kwargs["url"], content=kwargs["content"])
|
|
return httpx.Response(
|
|
200,
|
|
request=request,
|
|
headers={"content-type": "application/json"},
|
|
json={
|
|
"candidates": [{"content": {"parts": [{"text": "ok"}]}}],
|
|
"usageMetadata": {
|
|
"promptTokenCount": 11,
|
|
"candidatesTokenCount": 7,
|
|
"cachedContentTokenCount": 3,
|
|
},
|
|
},
|
|
)
|
|
|
|
|
|
class _AsyncChunks(httpx.AsyncByteStream):
|
|
def __init__(self, chunks: list[bytes]) -> None:
|
|
self._chunks = chunks
|
|
|
|
async def __aiter__(self): # noqa: ANN204
|
|
for chunk in self._chunks:
|
|
yield chunk
|
|
|
|
|
|
class _VertexStreamClient:
|
|
def __init__(self) -> None:
|
|
self.sent_url = ""
|
|
|
|
def build_request(self, method, url, headers, content): # noqa: ANN001, ANN201
|
|
self.sent_url = str(url)
|
|
return httpx.Request(method, url, headers=headers, content=content)
|
|
|
|
async def send(self, request, stream=False): # noqa: ANN001, ANN201
|
|
assert stream is True
|
|
return httpx.Response(
|
|
200,
|
|
request=request,
|
|
headers={"content-type": "text/event-stream"},
|
|
stream=_AsyncChunks(
|
|
[
|
|
b'data: {"candidates":[{"content":{"parts":[{"text":"hello"}]}}]}\n\n',
|
|
b'data: {"usageMetadata":{"promptTokenCount":13,'
|
|
b'"candidatesTokenCount":5,"cachedContentTokenCount":2}}\n\n',
|
|
]
|
|
),
|
|
)
|
|
|
|
|
|
class _RetryThenSuccessClient:
|
|
def __init__(self) -> None:
|
|
self.attempts = 0
|
|
|
|
async def post(self, url, content, headers, timeout=None, extensions=None): # noqa: ANN001, ANN201
|
|
self.attempts += 1
|
|
if self.attempts == 1:
|
|
raise httpx.ConnectTimeout("connect timed out")
|
|
del timeout
|
|
request = httpx.Request("POST", url, headers=headers, content=content)
|
|
return httpx.Response(200, request=request, content=b"{}")
|
|
|
|
|
|
def test_decode_openai_bearer_payload_handles_missing_and_non_mapping_payloads() -> None:
|
|
assert _decode_openai_bearer_payload({}) is None
|
|
assert _decode_openai_bearer_payload({"authorization": "Basic abc"}) is None
|
|
assert (
|
|
_decode_openai_bearer_payload({"authorization": f"Bearer {_jwt(['not', 'a', 'dict'])}"})
|
|
is None
|
|
)
|
|
|
|
|
|
def test_openai_handler_prefix_helpers_cover_edge_cases() -> None:
|
|
assert OpenAIHandlerMixin._strict_previous_turn_frozen_count([], 2) == 2
|
|
assert (
|
|
OpenAIHandlerMixin._strict_previous_turn_frozen_count(
|
|
[{"role": "assistant"}, {"role": "user"}],
|
|
0,
|
|
)
|
|
== 1
|
|
)
|
|
assert (
|
|
OpenAIHandlerMixin._strict_previous_turn_frozen_count(
|
|
[{"role": "assistant"}, {"role": "tool", "content": "observation"}],
|
|
0,
|
|
)
|
|
== 1
|
|
)
|
|
assert (
|
|
OpenAIHandlerMixin._strict_previous_turn_frozen_count(
|
|
[{"role": "user"}, {"role": "assistant"}, {"role": "tool", "content": "obs"}],
|
|
3,
|
|
)
|
|
== 2
|
|
)
|
|
assert (
|
|
OpenAIHandlerMixin._strict_previous_turn_frozen_count(
|
|
[{"role": "assistant"}, {"role": "function", "content": "legacy observation"}],
|
|
0,
|
|
)
|
|
== 1
|
|
)
|
|
assert (
|
|
OpenAIHandlerMixin._strict_previous_turn_frozen_count(
|
|
[{"role": "user"}, {"role": "assistant"}],
|
|
0,
|
|
)
|
|
== 2
|
|
)
|
|
|
|
original = [{"role": "system", "content": "keep"}, {"role": "user", "content": "hello"}]
|
|
restored, changed = OpenAIHandlerMixin._restore_frozen_prefix(
|
|
original,
|
|
[],
|
|
frozen_message_count=1,
|
|
)
|
|
assert restored == [{"role": "system", "content": "keep"}]
|
|
assert changed == 1
|
|
|
|
restored, changed = OpenAIHandlerMixin._restore_frozen_prefix(
|
|
original,
|
|
[{"role": "system", "content": "changed"}, {"role": "user", "content": "hello"}],
|
|
frozen_message_count=1,
|
|
)
|
|
assert restored == original
|
|
assert changed == 1
|
|
|
|
|
|
def test_relocate_system_messages_moves_stray_system_into_top_level() -> None:
|
|
# Issue #765: compression relocated the harness system block into
|
|
# messages[0] as a role="system" entry, which Anthropic rejects with a 400.
|
|
# The forwarder guard must move it back to the top-level `system` parameter.
|
|
messages = [
|
|
{"role": "system", "content": "You are a harness."},
|
|
{"role": "user", "content": "hi"},
|
|
]
|
|
clean, system, changed = relocate_system_messages_to_top_level(messages, None)
|
|
|
|
assert changed is True
|
|
# No role="system" entry may survive in messages[] — that is the wire-contract violation.
|
|
assert all(m.get("role") != "system" for m in clean)
|
|
assert clean == [{"role": "user", "content": "hi"}]
|
|
# The relocated content lands in the top-level system parameter.
|
|
assert system == [{"type": "text", "text": "You are a harness."}]
|
|
|
|
|
|
def test_relocate_system_messages_appends_to_existing_system() -> None:
|
|
messages = [
|
|
{"role": "system", "content": [{"type": "text", "text": "B"}]},
|
|
{"role": "user", "content": "hi"},
|
|
]
|
|
clean, system, changed = relocate_system_messages_to_top_level(messages, "A")
|
|
|
|
assert changed is True
|
|
assert clean == [{"role": "user", "content": "hi"}]
|
|
# Existing system first, relocated content after — wire order preserved.
|
|
assert system == [{"type": "text", "text": "A"}, {"type": "text", "text": "B"}]
|
|
|
|
|
|
def test_relocate_system_messages_noop_without_system_entry() -> None:
|
|
messages = [{"role": "user", "content": "hi"}]
|
|
clean, system, changed = relocate_system_messages_to_top_level(messages, "A")
|
|
|
|
assert changed is False
|
|
assert clean is messages
|
|
assert system == "A"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"model",
|
|
[
|
|
"claude-opus-5",
|
|
# Claude Haiku 5.5 supports mid-conversation system messages too; it was
|
|
# missing here, so Claude Code's Haiku subagent traffic had every valid
|
|
# section hoisted into the top-level system prompt.
|
|
"claude-haiku-5-5",
|
|
],
|
|
)
|
|
def test_relocate_system_messages_preserves_valid_mid_conversation_section(model: str) -> None:
|
|
messages = [
|
|
{"role": "user", "content": "Run the tests."},
|
|
{
|
|
"role": "system",
|
|
"content": "The user added: update the changelog too.",
|
|
},
|
|
{"role": "assistant", "content": "I will do both."},
|
|
]
|
|
|
|
clean, system, changed = relocate_system_messages_to_top_level(messages, "base", model)
|
|
|
|
assert changed is False
|
|
assert clean is messages
|
|
assert system == "base"
|
|
|
|
|
|
def test_relocate_system_messages_preserves_consecutive_valid_section_at_end() -> None:
|
|
messages = [
|
|
{"role": "user", "content": [{"type": "tool_result", "content": "ok"}]},
|
|
{"role": "system", "content": "First update."},
|
|
{"role": "system", "content": "Second update."},
|
|
]
|
|
|
|
clean, system, changed = relocate_system_messages_to_top_level(
|
|
messages, None, "global.anthropic.claude-sonnet-5-v1:0"
|
|
)
|
|
|
|
assert changed is False
|
|
assert clean is messages
|
|
assert system is None
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"messages",
|
|
[
|
|
[
|
|
{"role": "assistant", "content": "answer"},
|
|
{"role": "system", "content": "bad predecessor"},
|
|
],
|
|
[
|
|
{"role": "user", "content": "question"},
|
|
{"role": "system", "content": "bad successor"},
|
|
{"role": "user", "content": "another question"},
|
|
],
|
|
],
|
|
)
|
|
def test_relocate_system_messages_still_moves_invalid_mid_conversation_placement(
|
|
messages: list[dict],
|
|
) -> None:
|
|
clean, system, changed = relocate_system_messages_to_top_level(messages, None, "claude-fable-5")
|
|
|
|
assert changed is True
|
|
assert all(message.get("role") != "system" for message in clean)
|
|
assert system == [{"type": "text", "text": messages[1]["content"]}]
|
|
|
|
|
|
@pytest.mark.parametrize("model", ["claude-sonnet-4-6", "claude-haiku-4-5"])
|
|
def test_relocate_system_messages_moves_valid_shape_for_unsupported_model(model: str) -> None:
|
|
messages = [
|
|
{"role": "user", "content": "question"},
|
|
{"role": "system", "content": "mid-turn instruction"},
|
|
]
|
|
|
|
clean, system, changed = relocate_system_messages_to_top_level(messages, None, model)
|
|
|
|
assert changed is True
|
|
assert clean == [{"role": "user", "content": "question"}]
|
|
assert system == [{"type": "text", "text": "mid-turn instruction"}]
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("model", "third_party_upstream", "expected"),
|
|
[
|
|
("qwen3.8-flash-next", True, True),
|
|
("glm-5.3", True, True),
|
|
# Claude-named models behind a gateway keep the conservative hoist.
|
|
("claude-opus-5", True, False),
|
|
("anthropic/claude-sonnet-4-6", True, False),
|
|
("global.anthropic.claude-sonnet-5-v1:0", True, False),
|
|
# Anthropic itself (or Vertex / a backend) keeps the model allow-list.
|
|
("qwen3.8-flash-next", False, False),
|
|
],
|
|
)
|
|
def test_is_third_party_model(model: str, third_party_upstream: bool, expected: bool) -> None:
|
|
assert is_third_party_model(model, third_party_upstream=third_party_upstream) is expected
|
|
|
|
|
|
def test_relocate_system_messages_keeps_valid_sections_for_third_party_model() -> None:
|
|
# #3552 kept non-text blocks in place for third-party upstreams, but text
|
|
# sections were still hoisted for every model there. A client that appends a
|
|
# section most turns changed the top-level system prompt on each request, and
|
|
# the upstream's prefix cache missed the whole conversation every turn.
|
|
messages = [
|
|
{"role": "user", "content": "Run the tests."},
|
|
{"role": "system", "content": "The user added: update the changelog too."},
|
|
{"role": "assistant", "content": "I will do both."},
|
|
{"role": "user", "content": [{"type": "tool_result", "content": "ok"}]},
|
|
{"role": "system", "content": "<system-reminder>todo list changed</system-reminder>"},
|
|
]
|
|
|
|
clean, system, changed = relocate_system_messages_to_top_level(
|
|
messages, "base", "qwen3.8-flash-next", third_party_model=True
|
|
)
|
|
|
|
assert changed is False
|
|
assert clean is messages
|
|
assert system == "base"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"messages",
|
|
[
|
|
# The #765 case: a system block left at messages[0] is invalid anywhere.
|
|
[
|
|
{"role": "system", "content": "You are a harness."},
|
|
{"role": "user", "content": "hi"},
|
|
],
|
|
[
|
|
{"role": "assistant", "content": "answer"},
|
|
{"role": "system", "content": "You are a harness."},
|
|
],
|
|
],
|
|
)
|
|
def test_relocate_system_messages_still_moves_invalid_placement_for_third_party_model(
|
|
messages: list[dict],
|
|
) -> None:
|
|
clean, system, changed = relocate_system_messages_to_top_level(
|
|
messages, None, "qwen3.8-flash-next", third_party_model=True
|
|
)
|
|
|
|
assert changed is True
|
|
assert all(message.get("role") != "system" for message in clean)
|
|
assert system == [{"type": "text", "text": "You are a harness."}]
|
|
|
|
|
|
def test_relocate_system_messages_keeps_image_blocks_out_of_top_level_system() -> None:
|
|
image_block = {
|
|
"type": "image",
|
|
"source": {"type": "base64", "media_type": "image/png", "data": "aGk="},
|
|
}
|
|
messages = [
|
|
{"role": "user", "content": [{"type": "text", "text": "look at this"}]},
|
|
{
|
|
"role": "system",
|
|
"content": [
|
|
{"type": "text", "text": "<system-reminder>image attached</system-reminder>"},
|
|
image_block,
|
|
],
|
|
},
|
|
{"role": "assistant", "content": [{"type": "text", "text": "ok"}]},
|
|
{"role": "user", "content": [{"type": "text", "text": "hi"}]},
|
|
]
|
|
system = [{"type": "text", "text": "You are Claude Code."}]
|
|
|
|
clean, new_system, changed = relocate_system_messages_to_top_level(messages, system, None)
|
|
|
|
assert changed is True
|
|
assert isinstance(new_system, list)
|
|
assert all(not isinstance(block, dict) or block.get("type") == "text" for block in new_system)
|
|
retained = [
|
|
message.get("content")
|
|
for message in clean
|
|
if isinstance(message, dict) and message.get("role") == "system"
|
|
]
|
|
assert any(image_block in (content or []) for content in retained)
|
|
|
|
|
|
def test_relocate_system_messages_hoists_only_text_from_mixed_sections() -> None:
|
|
messages = [
|
|
{"role": "user", "content": "question"},
|
|
{
|
|
"role": "system",
|
|
"content": [
|
|
"plain string section",
|
|
{"type": "text", "text": "structured note"},
|
|
{"type": "image", "source": {"type": "url", "url": "https://x/y.png"}},
|
|
],
|
|
},
|
|
{"role": "assistant", "content": "ok"},
|
|
]
|
|
|
|
clean, new_system, changed = relocate_system_messages_to_top_level(messages, None, None)
|
|
|
|
assert changed is True
|
|
assert new_system == [
|
|
{"type": "text", "text": "plain string section"},
|
|
{"type": "text", "text": "structured note"},
|
|
]
|
|
retained = [
|
|
message
|
|
for message in clean
|
|
if isinstance(message, dict) and message.get("role") == "system"
|
|
]
|
|
assert retained == [
|
|
{
|
|
"role": "system",
|
|
"content": [{"type": "image", "source": {"type": "url", "url": "https://x/y.png"}}],
|
|
}
|
|
]
|
|
|
|
|
|
def test_relocate_system_messages_image_only_sections_pass_through_unchanged() -> None:
|
|
image_block = {
|
|
"type": "image",
|
|
"source": {"type": "base64", "media_type": "image/png", "data": "aGk="},
|
|
}
|
|
messages = [
|
|
{"role": "user", "content": "question"},
|
|
{"role": "system", "content": [image_block]},
|
|
{"role": "assistant", "content": "ok"},
|
|
]
|
|
system = "base"
|
|
|
|
clean, new_system, changed = relocate_system_messages_to_top_level(messages, system, None)
|
|
|
|
assert changed is False
|
|
assert clean == messages
|
|
assert new_system == system
|
|
|
|
|
|
def test_relocate_system_messages_drops_non_text_from_leading_section(
|
|
caplog: pytest.LogCaptureFixture,
|
|
) -> None:
|
|
image_block = {
|
|
"type": "image",
|
|
"source": {"type": "base64", "media_type": "image/png", "data": "aGk="},
|
|
}
|
|
cached_text = {
|
|
"type": "text",
|
|
"text": "leading instruction",
|
|
"cache_control": {"type": "ephemeral"},
|
|
}
|
|
messages: list[dict] = [
|
|
{"role": "system", "content": [cached_text, image_block]},
|
|
{"role": "user", "content": "hi"},
|
|
]
|
|
|
|
caplog.set_level("WARNING", logger="headroom.proxy")
|
|
clean, new_system, changed = relocate_system_messages_to_top_level(
|
|
messages, [{"type": "text", "text": "base"}], "claude-opus-5"
|
|
)
|
|
|
|
assert changed is True
|
|
assert clean == [{"role": "user", "content": "hi"}]
|
|
assert new_system == [
|
|
{"type": "text", "text": "base"},
|
|
cached_text,
|
|
]
|
|
assert "event=system_relocation_block_dropped" in caplog.text
|
|
assert "block_type=image" in caplog.text
|
|
|
|
|
|
def test_relocate_system_messages_drops_all_non_text_leading_section() -> None:
|
|
image_block = {
|
|
"type": "image",
|
|
"source": {"type": "base64", "media_type": "image/png", "data": "aGk="},
|
|
}
|
|
messages: list[dict] = [
|
|
{"role": "system", "content": [image_block]},
|
|
{"role": "user", "content": "hi"},
|
|
]
|
|
|
|
clean, new_system, changed = relocate_system_messages_to_top_level(
|
|
messages, [{"type": "text", "text": "base"}], "claude-opus-5"
|
|
)
|
|
|
|
assert changed is True
|
|
assert clean == [{"role": "user", "content": "hi"}]
|
|
assert new_system == [{"type": "text", "text": "base"}]
|
|
|
|
|
|
@pytest.mark.parametrize("block_type", ["text", "image", "tool_use", "tool_result", None])
|
|
def test_leading_system_text_uses_only_anthropic_text_block_fields(block_type) -> None:
|
|
from copy import deepcopy
|
|
|
|
cache_control = {"type": "ephemeral", "ttl": "1h"}
|
|
citations = [
|
|
{
|
|
"type": "char_location",
|
|
"cited_text": "instruction",
|
|
"document_index": 0,
|
|
"document_title": "source",
|
|
"start_char_index": 0,
|
|
"end_char_index": 11,
|
|
}
|
|
]
|
|
block = {
|
|
"type": block_type,
|
|
"text": "instruction",
|
|
"cache_control": cache_control,
|
|
"citations": citations,
|
|
"source": {"type": "url", "url": "https://example.invalid/image.png"},
|
|
"id": "tool1",
|
|
"name": "read",
|
|
"input": {"path": "file.txt"},
|
|
"tool_use_id": "tool1",
|
|
"is_error": False,
|
|
}
|
|
messages = [
|
|
{"role": "system", "content": [block]},
|
|
{"role": "user", "content": "hi"},
|
|
]
|
|
original = deepcopy(messages)
|
|
clean, system, changed = relocate_system_messages_to_top_level(messages, None, None)
|
|
assert changed
|
|
assert clean == [{"role": "user", "content": "hi"}]
|
|
assert system == [
|
|
{
|
|
"type": "text",
|
|
"text": "instruction",
|
|
"cache_control": cache_control,
|
|
"citations": citations,
|
|
}
|
|
]
|
|
assert messages == original
|
|
|
|
|
|
def test_all_contiguous_leading_system_sections_are_normalized() -> None:
|
|
image = {"type": "image", "source": {"type": "url", "url": "https://example.invalid/a"}}
|
|
clean, system, changed = relocate_system_messages_to_top_level(
|
|
[
|
|
{"role": "system", "content": "first instruction"},
|
|
{"role": "system", "content": [image]},
|
|
{"role": "system", "content": [{"type": "text", "text": "second instruction"}]},
|
|
{"role": "user", "content": "hi"},
|
|
],
|
|
None,
|
|
None,
|
|
)
|
|
assert changed
|
|
assert clean == [{"role": "user", "content": "hi"}]
|
|
assert system == [
|
|
{"type": "text", "text": "first instruction"},
|
|
{"type": "text", "text": "second instruction"},
|
|
]
|
|
|
|
|
|
def test_headroom_bypass_helper_is_transport_neutral() -> None:
|
|
assert _headroom_bypass_enabled({"x-headroom-bypass": "true"}) is True
|
|
assert _headroom_bypass_enabled({"x-headroom-bypass": " TRUE "}) is True
|
|
assert _headroom_bypass_enabled({"x-headroom-mode": "passthrough"}) is True
|
|
assert _headroom_bypass_enabled({"x-headroom-mode": " PASSTHROUGH "}) is True
|
|
assert _headroom_bypass_enabled({"x-headroom-bypass": "false"}) is False
|
|
assert _headroom_bypass_enabled({}) is False
|
|
assert _headroom_bypass_enabled(None) is False
|
|
assert OpenAIHandlerMixin._headroom_bypass_enabled({"x-headroom-bypass": "true"}) is True
|
|
|
|
|
|
def test_openai_passthrough_without_config_preserves_generic_request() -> None:
|
|
handler = object.__new__(OpenAIHandlerMixin)
|
|
handler.http_client = _RecordingHttpClient("h2")
|
|
request = _PassthroughRequest()
|
|
|
|
response = asyncio.run(handler.handle_passthrough(request, "https://api.openai.com"))
|
|
|
|
assert response.status_code == 200
|
|
assert json.loads(response.body)["client"] == "h2"
|
|
|
|
|
|
def test_openai_passthrough_connect_timeout_returns_502() -> None:
|
|
handler = object.__new__(OpenAIHandlerMixin)
|
|
handler.http_client = _TimeoutHttpClient()
|
|
|
|
async def run():
|
|
return await handler.handle_passthrough(
|
|
_PassthroughRequest(),
|
|
"https://api.openai.com",
|
|
)
|
|
|
|
response = asyncio.run(run())
|
|
|
|
assert response.status_code == 502
|
|
payload = json.loads(response.body)
|
|
assert payload["error"]["type"] == "connection_error"
|
|
assert "Failed to connect to upstream API" in payload["error"]["message"]
|
|
|
|
|
|
def test_prefers_http1_passthrough_matches_chatgpt_hosts_only() -> None:
|
|
assert _prefers_http1_passthrough("https://chatgpt.com") is True
|
|
assert _prefers_http1_passthrough("https://chatgpt.com/backend-api/me") is True
|
|
assert _prefers_http1_passthrough("https://api.chatgpt.com") is True
|
|
assert _prefers_http1_passthrough("https://CHATGPT.COM/backend-api/me") is True
|
|
assert _prefers_http1_passthrough("https://api.openai.com") is False
|
|
assert _prefers_http1_passthrough("https://notchatgpt.com") is False
|
|
assert _prefers_http1_passthrough("https://chatgpt.com.evil.com") is False
|
|
assert _prefers_http1_passthrough("") is False
|
|
|
|
|
|
def test_chatgpt_passthrough_uses_http1_client() -> None:
|
|
handler = object.__new__(OpenAIHandlerMixin)
|
|
handler.http_client = _RecordingHttpClient("h2")
|
|
handler.http_client_h1 = _RecordingHttpClient("h1")
|
|
|
|
response = asyncio.run(
|
|
handler.handle_passthrough(_ChatGPTAccountRequest(), "https://chatgpt.com")
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert json.loads(response.body)["client"] == "h1"
|
|
assert handler.http_client.calls == 0
|
|
assert handler.http_client_h1.calls == 1
|
|
|
|
|
|
def test_non_chatgpt_passthrough_uses_default_client() -> None:
|
|
handler = object.__new__(OpenAIHandlerMixin)
|
|
handler.http_client = _RecordingHttpClient("h2")
|
|
handler.http_client_h1 = _RecordingHttpClient("h1")
|
|
|
|
response = asyncio.run(
|
|
handler.handle_passthrough(_PassthroughRequest(), "https://api.openai.com")
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert json.loads(response.body)["client"] == "h2"
|
|
assert handler.http_client.calls == 1
|
|
assert handler.http_client_h1.calls == 0
|
|
|
|
|
|
def test_chatgpt_passthrough_falls_back_when_h1_client_missing() -> None:
|
|
handler = object.__new__(OpenAIHandlerMixin)
|
|
handler.http_client = _RecordingHttpClient("h2")
|
|
handler.http_client_h1 = None
|
|
|
|
response = asyncio.run(
|
|
handler.handle_passthrough(_ChatGPTAccountRequest(), "https://chatgpt.com")
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert json.loads(response.body)["client"] == "h2"
|
|
assert handler.http_client.calls == 1
|
|
|
|
|
|
def test_streaming_client_selection() -> None:
|
|
assert prefers_http1_passthrough("https://chatgpt.com/backend-api/codex/responses") is True
|
|
assert prefers_http1_passthrough("https://api.openai.com/v1/chat/completions") is False
|
|
|
|
mixin = object.__new__(StreamingMixin)
|
|
mixin.http_client = "h2"
|
|
mixin.http_client_h1 = "h1"
|
|
|
|
assert mixin._select_streaming_client("https://chatgpt.com/backend-api/codex/responses") == "h1"
|
|
assert mixin._select_streaming_client("https://api.openai.com/v1/chat/completions") == "h2"
|
|
|
|
mixin.http_client_h1 = None
|
|
assert mixin._select_streaming_client("https://chatgpt.com/backend-api/codex/responses") == "h2"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_retry_request_selects_http1_for_chatgpt(monkeypatch) -> None:
|
|
calls = []
|
|
|
|
class DummyClient:
|
|
def __init__(self, name: str):
|
|
self.name = name
|
|
|
|
async def post(self, url, **kwargs):
|
|
calls.append((self.name, url))
|
|
return httpx.Response(200, json={"ok": True})
|
|
|
|
proxy = object.__new__(HeadroomProxy)
|
|
proxy.config = type("Config", (), {"retry_max_attempts": 1, "retry_enabled": False})()
|
|
proxy.http_client = DummyClient("h2")
|
|
proxy.http_client_h1 = DummyClient("h1")
|
|
|
|
await proxy._retry_request("POST", "https://chatgpt.com/backend-api/codex/responses", {}, {})
|
|
assert calls == [("h1", "https://chatgpt.com/backend-api/codex/responses")]
|
|
|
|
calls.clear()
|
|
await proxy._retry_request("POST", "https://api.openai.com/v1/chat/completions", {}, {})
|
|
assert calls == [("h2", "https://api.openai.com/v1/chat/completions")]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_proxy_routes_openai_responses_subpath_selects_http1(monkeypatch) -> None:
|
|
from fastapi import FastAPI
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.providers.openai_responses import OpenAIResponsesSubpathRoute
|
|
from headroom.providers.proxy_routes import _register_openai_responses_subpath_route
|
|
|
|
received_clients = []
|
|
|
|
async def fake_subpath(http_client, request, sub_path):
|
|
received_clients.append(http_client)
|
|
return httpx.Response(200, json={"ok": True})
|
|
|
|
monkeypatch.setattr(
|
|
"headroom.providers.proxy_routes.handle_chatgpt_codex_responses_subpath",
|
|
fake_subpath,
|
|
)
|
|
|
|
app = FastAPI()
|
|
proxy = type("Proxy", (), {"http_client": "h2", "http_client_h1": "h1"})()
|
|
spec = OpenAIResponsesSubpathRoute(
|
|
path="/v1/responses/{sub_path:path}",
|
|
methods=("GET", "POST"),
|
|
)
|
|
_register_openai_responses_subpath_route(app, proxy, spec)
|
|
|
|
client = TestClient(app)
|
|
client.get("/v1/responses/items/resp_1")
|
|
assert received_clients == ["h1"]
|
|
|
|
|
|
def test_passthrough_usage_normalizes_vertex_usage_metadata() -> None:
|
|
usage = _passthrough_usage_from_json(
|
|
{
|
|
"usageMetadata": {
|
|
"promptTokenCount": 11,
|
|
"candidatesTokenCount": 7,
|
|
"cachedContentTokenCount": 3,
|
|
}
|
|
}
|
|
)
|
|
|
|
assert usage == {
|
|
"input_tokens": 11,
|
|
"output_tokens": 7,
|
|
"cache_read_input_tokens": 3,
|
|
}
|
|
|
|
|
|
def test_gemini_output_tokens_includes_thinking_when_exclusive() -> None:
|
|
"""Gemini 2.5 thinking: when prompt + candidates != total, thoughtsTokenCount
|
|
is a separate output bucket and must be added, or output cost undercounts."""
|
|
from headroom.proxy.token_counting import gemini_output_tokens
|
|
|
|
exclusive = {
|
|
"promptTokenCount": 1000,
|
|
"candidatesTokenCount": 200,
|
|
"thoughtsTokenCount": 500,
|
|
"totalTokenCount": 1700,
|
|
}
|
|
assert gemini_output_tokens(exclusive) == 700 # 200 visible + 500 thinking
|
|
|
|
# Inclusive: candidatesTokenCount already covers thoughts (prompt+cand==total).
|
|
inclusive = {
|
|
"promptTokenCount": 1000,
|
|
"candidatesTokenCount": 700,
|
|
"thoughtsTokenCount": 500,
|
|
"totalTokenCount": 1700,
|
|
}
|
|
assert gemini_output_tokens(inclusive) == 700
|
|
|
|
# No thinking tokens: just the candidates count (common non-2.5 case).
|
|
assert gemini_output_tokens({"candidatesTokenCount": 42, "totalTokenCount": 100}) == 42
|
|
# Robust to empty / missing fields.
|
|
assert gemini_output_tokens({}) == 0
|
|
|
|
|
|
def test_passthrough_usage_counts_gemini_thinking_tokens() -> None:
|
|
"""_passthrough_usage_from_json must include thinking tokens in output_tokens."""
|
|
usage = _passthrough_usage_from_json(
|
|
{
|
|
"usageMetadata": {
|
|
"promptTokenCount": 1000,
|
|
"candidatesTokenCount": 200,
|
|
"thoughtsTokenCount": 500,
|
|
"totalTokenCount": 1700,
|
|
"cachedContentTokenCount": 100,
|
|
}
|
|
}
|
|
)
|
|
assert usage["output_tokens"] == 700
|
|
assert usage["input_tokens"] == 1000
|
|
|
|
|
|
def test_vertex_passthrough_records_usage_metadata_for_dashboard() -> None:
|
|
handler = object.__new__(HeadroomProxy)
|
|
handler.http_client = _VertexUsageClient()
|
|
outcomes = []
|
|
|
|
async def next_request_id(): # noqa: ANN202
|
|
return "req_vertex"
|
|
|
|
async def record(outcome): # noqa: ANN001, ANN202
|
|
outcomes.append(outcome)
|
|
|
|
handler._next_request_id = next_request_id
|
|
handler._record_request_outcome = record
|
|
|
|
response = asyncio.run(
|
|
handler.handle_passthrough(
|
|
_VertexPassthroughRequest(),
|
|
"https://vertex.test",
|
|
"generateContent",
|
|
"vertex:google",
|
|
)
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert len(outcomes) == 1
|
|
outcome = outcomes[0]
|
|
assert outcome.provider == "vertex:google"
|
|
assert outcome.model == "gemini-2.0-flash"
|
|
assert outcome.optimized_tokens == 11
|
|
assert outcome.output_tokens == 7
|
|
assert outcome.cache_read_tokens == 3
|
|
|
|
|
|
def test_vertex_stream_passthrough_preserves_chunks_and_records_usage() -> None:
|
|
handler = object.__new__(HeadroomProxy)
|
|
handler.http_client = _VertexStreamClient()
|
|
outcomes = []
|
|
|
|
async def next_request_id(): # noqa: ANN202
|
|
return "req_vertex_stream"
|
|
|
|
async def record(outcome): # noqa: ANN001, ANN202
|
|
outcomes.append(outcome)
|
|
|
|
handler._next_request_id = next_request_id
|
|
handler._record_request_outcome = record
|
|
|
|
response = asyncio.run(
|
|
handler.handle_passthrough(
|
|
_VertexStreamPassthroughRequest(),
|
|
"https://vertex.test",
|
|
"streamGenerateContent",
|
|
"vertex:google",
|
|
)
|
|
)
|
|
|
|
assert isinstance(response, StreamingResponse)
|
|
|
|
async def collect(): # noqa: ANN202
|
|
return [chunk async for chunk in response.body_iterator]
|
|
|
|
chunks = asyncio.run(collect())
|
|
|
|
assert len(chunks) == 2
|
|
assert chunks[0].startswith(b'data: {"candidates"')
|
|
assert b'"usageMetadata"' in chunks[1]
|
|
assert len(outcomes) == 1
|
|
outcome = outcomes[0]
|
|
assert outcome.provider == "vertex:google"
|
|
assert outcome.model == "gemini-2.0-flash"
|
|
assert outcome.optimized_tokens == 13
|
|
assert outcome.output_tokens == 5
|
|
assert outcome.cache_read_tokens == 2
|
|
|
|
|
|
def test_vertex_stream_passthrough_records_outcome_when_stream_ends_mid_character() -> None:
|
|
closed: list[bool] = []
|
|
|
|
class _CountingChunks(_AsyncChunks):
|
|
async def aclose(self) -> None:
|
|
closed.append(True)
|
|
|
|
class _TruncatedClient(_VertexStreamClient):
|
|
async def send(self, request, stream=False): # noqa: ANN001, ANN201
|
|
return httpx.Response(
|
|
200,
|
|
request=request,
|
|
headers={"content-type": "text/event-stream"},
|
|
stream=_CountingChunks(
|
|
[
|
|
b'data: {"usageMetadata":{"promptTokenCount":13,'
|
|
b'"candidatesTokenCount":5}}\n\n',
|
|
b'data: {"t":"\xe2\x82', # first 2 bytes of a 3-byte character
|
|
]
|
|
),
|
|
)
|
|
|
|
handler = object.__new__(HeadroomProxy)
|
|
handler.http_client = _TruncatedClient()
|
|
outcomes = []
|
|
|
|
async def next_request_id(): # noqa: ANN202
|
|
return "req_vertex_truncated"
|
|
|
|
async def record(outcome): # noqa: ANN001, ANN202
|
|
outcomes.append(outcome)
|
|
|
|
handler._next_request_id = next_request_id
|
|
handler._record_request_outcome = record
|
|
|
|
response = asyncio.run(
|
|
handler.handle_passthrough(
|
|
_VertexStreamPassthroughRequest(),
|
|
"https://vertex.test",
|
|
"streamGenerateContent",
|
|
"vertex:google",
|
|
)
|
|
)
|
|
|
|
async def collect(): # noqa: ANN202
|
|
return [chunk async for chunk in response.body_iterator]
|
|
|
|
assert len(asyncio.run(collect())) == 2
|
|
assert len(outcomes) == 1
|
|
assert closed == [True]
|
|
# Usage already received is kept, but a stream cut mid-character did not
|
|
# complete: it takes the same failure status as other post-start stream
|
|
# failures, so the outcome funnel books it as failed, not as a success.
|
|
assert outcomes[0].optimized_tokens == 13
|
|
assert outcomes[0].output_tokens == 5
|
|
assert outcomes[0].status_code == 502
|
|
|
|
failed: list[str] = []
|
|
requested: list[dict] = []
|
|
|
|
class _Metrics:
|
|
async def record_failed(self, provider): # noqa: ANN001, ANN202
|
|
failed.append(provider)
|
|
|
|
async def record_request(self, **kwargs): # noqa: ANN003, ANN202
|
|
requested.append(kwargs)
|
|
|
|
asyncio.run(emit_request_outcome(SimpleNamespace(metrics=_Metrics()), outcomes[0]))
|
|
assert failed == [outcomes[0].provider]
|
|
assert requested == []
|
|
|
|
|
|
def test_stream_finalizer_records_vertex_provider_for_dashboard() -> None:
|
|
handler = object.__new__(HeadroomProxy)
|
|
handler.config = SimpleNamespace(log_full_messages=False)
|
|
outcomes = []
|
|
|
|
async def record(outcome): # noqa: ANN001, ANN202
|
|
outcomes.append(outcome)
|
|
|
|
handler._record_request_outcome = record
|
|
|
|
asyncio.run(
|
|
handler._finalize_stream_response(
|
|
body={"contents": [{"role": "user", "parts": [{"text": "hello"}]}]},
|
|
provider="gemini",
|
|
outcome_provider="vertex:google",
|
|
model="gemini-2.0-flash",
|
|
request_id="req_vertex_stream_final",
|
|
original_tokens=20,
|
|
optimized_tokens=12,
|
|
tokens_saved=8,
|
|
transforms_applied=["test-transform"],
|
|
optimization_latency=3.0,
|
|
stream_state={
|
|
"input_tokens": 12,
|
|
"output_tokens": 5,
|
|
"cache_read_input_tokens": 2,
|
|
"cache_creation_input_tokens": 0,
|
|
"cache_creation_ephemeral_5m_input_tokens": 0,
|
|
"cache_creation_ephemeral_1h_input_tokens": 0,
|
|
"total_bytes": 100,
|
|
"sse_buffer": bytearray(),
|
|
"ttfb_ms": 4.0,
|
|
},
|
|
start_time=0.0,
|
|
tags={"route": "vertex"},
|
|
)
|
|
)
|
|
|
|
assert len(outcomes) == 1
|
|
outcome = outcomes[0]
|
|
assert outcome.provider == "vertex:google"
|
|
assert outcome.model == "gemini-2.0-flash"
|
|
assert outcome.optimized_tokens == 12
|
|
assert outcome.output_tokens == 5
|
|
assert outcome.tokens_saved == 8
|
|
assert outcome.cache_read_tokens == 2
|
|
|
|
|
|
def _finalize_anthropic_stream(optimized_tokens, input_tokens): # noqa: ANN001, ANN202
|
|
handler = object.__new__(HeadroomProxy)
|
|
handler.config = SimpleNamespace(log_full_messages=False)
|
|
outcomes = []
|
|
|
|
async def record(outcome): # noqa: ANN001, ANN202
|
|
outcomes.append(outcome)
|
|
|
|
handler._record_request_outcome = record
|
|
|
|
asyncio.run(
|
|
handler._finalize_stream_response(
|
|
body={"messages": [{"role": "user", "content": "hello"}]},
|
|
provider="anthropic",
|
|
model="claude-opus-4-1",
|
|
request_id="req_anthropic_stream_final",
|
|
original_tokens=optimized_tokens + 5,
|
|
optimized_tokens=optimized_tokens,
|
|
tokens_saved=5,
|
|
transforms_applied=[],
|
|
optimization_latency=1.0,
|
|
stream_state={
|
|
"input_tokens": input_tokens,
|
|
"output_tokens": 7,
|
|
"cache_read_input_tokens": 360_949,
|
|
"cache_creation_input_tokens": 840,
|
|
"cache_creation_ephemeral_5m_input_tokens": 0,
|
|
"cache_creation_ephemeral_1h_input_tokens": 840,
|
|
"total_bytes": 100,
|
|
"sse_buffer": bytearray(),
|
|
"ttfb_ms": 4.0,
|
|
},
|
|
start_time=0.0,
|
|
)
|
|
)
|
|
(outcome,) = outcomes
|
|
return outcome
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"optimized_tokens",
|
|
[
|
|
# Local count 50k over the provider's (a production turn reported
|
|
# input_tokens=2 while the old derivation logged 50,663 uncached).
|
|
412_452,
|
|
# Local count under the provider's: the old derivation clamped to 0.
|
|
300_000,
|
|
],
|
|
)
|
|
def test_stream_finalizer_takes_anthropic_uncached_input_from_usage(optimized_tokens) -> None: # noqa: ANN001
|
|
outcome = _finalize_anthropic_stream(optimized_tokens, input_tokens=2)
|
|
|
|
assert outcome.uncached_input_tokens == 2
|
|
assert outcome.cache_read_tokens == 360_949
|
|
assert outcome.cache_write_tokens == 840
|
|
|
|
|
|
def test_stream_finalizer_derives_anthropic_uncached_input_without_usage() -> None:
|
|
# No message_start usage (an error before the stream began): keep the
|
|
# tokenizer-based derivation.
|
|
outcome = _finalize_anthropic_stream(400_000, input_tokens=None)
|
|
|
|
assert outcome.uncached_input_tokens == 400_000 - 360_949 - 840
|
|
|
|
|
|
def test_sse_parser_leaves_absent_anthropic_input_tokens_absent() -> None:
|
|
# A message_start usage without input_tokens must not read as a provider
|
|
# count of 0, or the finalizer books 0 uncached instead of deriving it.
|
|
proxy = object.__new__(HeadroomProxy)
|
|
event = {"type": "message_start", "message": {"usage": {"cache_read_input_tokens": 9}}}
|
|
state = {"sse_buffer": bytearray(f"data: {json.dumps(event)}\n\n".encode())}
|
|
|
|
usage = proxy._parse_sse_usage_from_buffer(state, "anthropic")
|
|
|
|
assert usage is not None
|
|
assert "input_tokens" not in usage
|
|
assert usage["cache_read_input_tokens"] == 9
|
|
|
|
|
|
def test_vertex_gemini_non_text_generate_records_dashboard_outcome() -> None:
|
|
handler = object.__new__(HeadroomProxy)
|
|
handler.memory_handler = None
|
|
handler.rate_limiter = None
|
|
outcomes = []
|
|
upstream_urls = []
|
|
|
|
async def next_request_id(): # noqa: ANN202
|
|
return "req_vertex_image"
|
|
|
|
async def record(outcome): # noqa: ANN001, ANN202
|
|
outcomes.append(outcome)
|
|
|
|
async def retry_request(method, url, headers, body): # noqa: ANN001, ANN202
|
|
upstream_urls.append(url)
|
|
request = httpx.Request(method, url, headers=headers)
|
|
return httpx.Response(
|
|
200,
|
|
request=request,
|
|
headers={"content-type": "application/json"},
|
|
json={
|
|
"usageMetadata": {
|
|
"promptTokenCount": 31,
|
|
"candidatesTokenCount": 4,
|
|
"cachedContentTokenCount": 6,
|
|
}
|
|
},
|
|
)
|
|
|
|
handler._next_request_id = next_request_id
|
|
handler._record_request_outcome = record
|
|
handler._retry_request = retry_request
|
|
|
|
response = asyncio.run(
|
|
handler.handle_gemini_generate_content(
|
|
_VertexGeminiImageRequest(),
|
|
"gemini-2.0-flash",
|
|
"https://vertex.test",
|
|
"vertex:google",
|
|
)
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert upstream_urls == [
|
|
"https://vertex.test/v1/projects/p/locations/us-central1/publishers/google/models/gemini-2.0-flash:generateContent"
|
|
]
|
|
assert response.headers["x-headroom-tokens-before"] == "31"
|
|
assert response.headers["x-headroom-tokens-after"] == "31"
|
|
assert response.headers["x-headroom-tokens-saved"] == "0"
|
|
assert len(outcomes) == 1
|
|
outcome = outcomes[0]
|
|
assert outcome.provider == "vertex:google"
|
|
assert outcome.model == "gemini-2.0-flash"
|
|
assert outcome.original_tokens == 31
|
|
assert outcome.optimized_tokens == 31
|
|
assert outcome.output_tokens == 4
|
|
assert outcome.cache_read_tokens == 6
|
|
assert outcome.num_messages == 1
|
|
|
|
|
|
def test_retry_request_retries_connect_timeout() -> None:
|
|
proxy = object.__new__(HeadroomProxy)
|
|
proxy.http_client = _RetryThenSuccessClient()
|
|
proxy.config = SimpleNamespace(
|
|
retry_enabled=True,
|
|
retry_max_attempts=2,
|
|
retry_base_delay_ms=0,
|
|
retry_max_delay_ms=0,
|
|
)
|
|
|
|
response = asyncio.run(
|
|
proxy._retry_request(
|
|
"POST",
|
|
"https://api.openai.com/v1/responses",
|
|
{},
|
|
{"model": "gpt-5"},
|
|
)
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert proxy.http_client.attempts == 2
|
|
|
|
|
|
def test_retry_request_returns_503_when_shutdown_interrupts_retry_sleep() -> None:
|
|
class _Always429Client:
|
|
def __init__(self) -> None:
|
|
self.attempts = 0
|
|
|
|
async def post(self, url, **kwargs): # type: ignore[no-untyped-def]
|
|
self.attempts += 1
|
|
return httpx.Response(
|
|
429,
|
|
request=httpx.Request("POST", url),
|
|
json={"error": {"message": "slow down"}},
|
|
headers={"retry-after": "30"},
|
|
)
|
|
|
|
proxy = object.__new__(HeadroomProxy)
|
|
proxy.http_client = _Always429Client()
|
|
proxy.config = SimpleNamespace(
|
|
retry_enabled=True,
|
|
retry_max_attempts=3,
|
|
retry_base_delay_ms=30000,
|
|
retry_max_delay_ms=30000,
|
|
)
|
|
proxy._shutdown_event = asyncio.Event()
|
|
proxy._shutdown_event.set()
|
|
|
|
response = asyncio.run(
|
|
proxy._retry_request(
|
|
"POST",
|
|
"https://api.anthropic.test/v1/messages",
|
|
{},
|
|
{"model": "claude-3-5-sonnet"},
|
|
)
|
|
)
|
|
|
|
assert response.status_code == 503
|
|
assert response.json() == {
|
|
"error": {
|
|
"type": "shutdown",
|
|
"message": "Proxy is shutting down; retry backoff cancelled.",
|
|
}
|
|
}
|
|
assert response.headers["retry-after"] == "0"
|
|
assert proxy.http_client.attempts == 1
|
|
|
|
|
|
def test_anthropic_tool_sort_and_context_append_helpers() -> None:
|
|
tools = [
|
|
{"type": "function", "function": {"name": "beta"}},
|
|
{"name": "alpha"},
|
|
{"type": "tool"},
|
|
]
|
|
|
|
sorted_tools = AnthropicHandlerMixin._sort_tools_deterministically(tools)
|
|
|
|
assert [AnthropicHandlerMixin._tool_sort_key(tool)[0] for tool in sorted_tools] == [
|
|
"alpha",
|
|
"beta",
|
|
"tool",
|
|
]
|
|
assert AnthropicHandlerMixin._sort_tools_deterministically(None) is None
|
|
assert AnthropicHandlerMixin._tools_for_forwarding(tools, preserve_order=True) == tools
|
|
assert [
|
|
AnthropicHandlerMixin._tool_sort_key(tool)[0]
|
|
for tool in AnthropicHandlerMixin._tools_for_forwarding(tools, preserve_order=False) or []
|
|
] == [
|
|
"alpha",
|
|
"beta",
|
|
"tool",
|
|
]
|
|
assert (
|
|
AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn(
|
|
[], "ctx", frozen_message_count=0
|
|
)
|
|
== []
|
|
)
|
|
assert AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn(
|
|
[{"role": "user", "content": "hello"}],
|
|
"ctx",
|
|
frozen_message_count=0,
|
|
) == [{"role": "user", "content": "hello\n\nctx"}]
|
|
# PR-A2 semantics: list-content user messages get the context appended
|
|
# to the first text block (live-zone-tail injection).
|
|
assert AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn(
|
|
[{"role": "user", "content": [{"type": "text", "text": "hello"}]}],
|
|
"ctx",
|
|
frozen_message_count=0,
|
|
) == [{"role": "user", "content": [{"type": "text", "text": "hello\n\nctx"}]}]
|
|
|
|
|
|
def test_append_context_skips_trailing_system_message() -> None:
|
|
# Claude Code 2.1.x request shape: the user turn is followed by a
|
|
# role="system" message carrying the environment and the cache breakpoint.
|
|
trailing_system = {
|
|
"role": "system",
|
|
"content": [
|
|
{"type": "text", "text": "# Environment", "cache_control": {"type": "ephemeral"}}
|
|
],
|
|
}
|
|
user_turn = {
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "text", "text": "<system-reminder>ctx</system-reminder>"},
|
|
{"type": "text", "text": "question"},
|
|
],
|
|
}
|
|
inject = AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn
|
|
|
|
assert inject([user_turn, trailing_system], "memory", frozen_message_count=0) == [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "text", "text": "<system-reminder>ctx</system-reminder>\n\nmemory"},
|
|
{"type": "text", "text": "question"},
|
|
],
|
|
},
|
|
trailing_system,
|
|
]
|
|
# The user turn is still subject to the frozen prefix.
|
|
messages = [user_turn, trailing_system]
|
|
assert inject(messages, "memory", frozen_message_count=1) is messages
|
|
# Skipping system messages never reaches past a non-user turn.
|
|
messages = [user_turn, {"role": "assistant", "content": "ok"}, trailing_system]
|
|
assert inject(messages, "memory", frozen_message_count=0) is messages
|
|
# Tool-result-only turns have no text block to extend.
|
|
messages = [
|
|
{"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t1", "content": "x"}]},
|
|
trailing_system,
|
|
]
|
|
assert inject(messages, "memory", frozen_message_count=0) is messages
|
|
messages = [trailing_system]
|
|
assert inject(messages, "memory", frozen_message_count=0) is messages
|
|
|
|
|
|
def test_proactive_expansion_appends_after_cached_prefix_without_mutating_it() -> None:
|
|
"""CCR expansion must not rewrite a message the provider has already cached (#4287)."""
|
|
from headroom.ccr.context_tracker import (
|
|
PROACTIVE_EXPANSION_CLOSE_TAG,
|
|
PROACTIVE_EXPANSION_OPEN_TAG,
|
|
)
|
|
|
|
env_block = {"type": "text", "text": "# Environment", "cache_control": {"type": "ephemeral"}}
|
|
trailing_system = {"role": "system", "content": [env_block]}
|
|
first_user = {
|
|
"role": "user",
|
|
"content": [{"type": "text", "text": "hello from turn 0"}],
|
|
}
|
|
messages = [first_user, trailing_system]
|
|
expansion = f"{PROACTIVE_EXPANSION_OPEN_TAG}\nexpanded\n{PROACTIVE_EXPANSION_CLOSE_TAG}"
|
|
|
|
forwarded = AnthropicHandlerMixin._append_proactive_expansion_after_cached_prefix(
|
|
messages, expansion
|
|
)
|
|
|
|
assert forwarded[0] == first_user
|
|
assert forwarded[1]["role"] == "system"
|
|
assert forwarded[1]["content"][0] == env_block
|
|
assert forwarded[1]["content"][1] == {"type": "text", "text": expansion}
|
|
assert "cache_control" not in forwarded[1]["content"][1]
|
|
assert AnthropicHandlerMixin._without_proactive_expansion_messages(forwarded) == messages
|
|
assert messages[0]["content"][0]["text"] == "hello from turn 0"
|
|
assert messages[1]["content"] == [env_block]
|
|
|
|
|
|
def test_proactive_expansion_without_cache_control_stays_on_the_live_user_turn() -> None:
|
|
from headroom.ccr.context_tracker import (
|
|
PROACTIVE_EXPANSION_CLOSE_TAG,
|
|
PROACTIVE_EXPANSION_OPEN_TAG,
|
|
)
|
|
|
|
messages = [{"role": "user", "content": "hello"}]
|
|
expansion = f"{PROACTIVE_EXPANSION_OPEN_TAG}\nexpanded\n{PROACTIVE_EXPANSION_CLOSE_TAG}"
|
|
forwarded = AnthropicHandlerMixin._append_proactive_expansion_after_cached_prefix(
|
|
messages, expansion
|
|
)
|
|
assert forwarded[0]["content"][0] == {"type": "text", "text": "hello"}
|
|
assert forwarded[0]["content"][1] == {"type": "text", "text": expansion}
|
|
assert AnthropicHandlerMixin._without_proactive_expansion_messages(forwarded) == [
|
|
{"role": "user", "content": [{"type": "text", "text": "hello"}]}
|
|
]
|
|
assert messages[0]["content"] == "hello"
|
|
|
|
|
|
def test_proactive_expansion_keeps_tool_results_together_and_client_text() -> None:
|
|
from headroom.ccr.context_tracker import (
|
|
PROACTIVE_EXPANSION_CLOSE_TAG,
|
|
PROACTIVE_EXPANSION_OPEN_TAG,
|
|
)
|
|
|
|
expansion = f"{PROACTIVE_EXPANSION_OPEN_TAG}\nexample\n{PROACTIVE_EXPANSION_CLOSE_TAG}"
|
|
tool_a = {
|
|
"type": "tool_result",
|
|
"tool_use_id": "A",
|
|
"content": "a",
|
|
"cache_control": {"type": "ephemeral"},
|
|
}
|
|
tool_b = {"type": "tool_result", "tool_use_id": "B", "content": "b"}
|
|
messages = [{"role": "user", "content": [tool_a, tool_b]}]
|
|
forwarded = AnthropicHandlerMixin._append_proactive_expansion_after_cached_prefix(
|
|
messages, expansion
|
|
)
|
|
content = forwarded[0]["content"]
|
|
assert [block.get("type") for block in content] == ["tool_result", "tool_result", "text"]
|
|
assert content[0] is tool_a and content[1] is tool_b and "cache_control" not in content[2]
|
|
assert AnthropicHandlerMixin._without_proactive_expansion_messages(forwarded) == messages
|
|
client = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "text", "text": "Explain this:"},
|
|
{"type": "text", "text": expansion},
|
|
],
|
|
}
|
|
]
|
|
assert AnthropicHandlerMixin._without_proactive_expansion_messages(client) == client
|
|
|
|
|
|
def test_anthropic_image_compression_helper_only_rewrites_latest_eligible_turn() -> None:
|
|
image_message = {
|
|
"role": "user",
|
|
"content": [{"type": "image", "source": {"type": "base64", "data": "abc"}}],
|
|
}
|
|
compressed = {
|
|
"role": "user",
|
|
"content": [{"type": "image", "source": {"type": "base64", "data": "xyz"}}],
|
|
}
|
|
|
|
assert (
|
|
AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
|
|
[],
|
|
frozen_message_count=0,
|
|
compressor=_ImageCompressor(compressed),
|
|
)
|
|
== []
|
|
)
|
|
assert AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
|
|
[image_message],
|
|
frozen_message_count=1,
|
|
compressor=_ImageCompressor(compressed),
|
|
) == [image_message]
|
|
assert AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
|
|
[{"role": "assistant", "content": image_message["content"]}],
|
|
frozen_message_count=0,
|
|
compressor=_ImageCompressor(compressed),
|
|
) == [{"role": "assistant", "content": image_message["content"]}]
|
|
assert AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
|
|
[{"role": "user", "content": "no-image"}],
|
|
frozen_message_count=0,
|
|
compressor=_ImageCompressor(compressed),
|
|
) == [{"role": "user", "content": "no-image"}]
|
|
assert AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
|
|
[image_message],
|
|
frozen_message_count=0,
|
|
compressor=_ImageCompressor(image_message),
|
|
) == [image_message]
|
|
assert AnthropicHandlerMixin._compress_latest_user_turn_images_cache_safe(
|
|
[image_message],
|
|
frozen_message_count=0,
|
|
compressor=_ImageCompressor(compressed),
|
|
) == [compressed]
|
|
|
|
|
|
def test_proxy_helper_reuses_a_singleton_image_compressor(monkeypatch) -> None:
|
|
# #2513: the compressor caches heavyweight models, so it must be a
|
|
# process-wide singleton rather than a fresh instance per request.
|
|
from headroom.proxy import helpers
|
|
|
|
monkeypatch.setattr(helpers, "_image_compressor_available", None)
|
|
monkeypatch.setattr(helpers, "_image_compressor_instance", None)
|
|
_FreshCompressor.instances = 0
|
|
|
|
with patch("headroom.image.ImageCompressor", _FreshCompressor):
|
|
first = helpers._get_image_compressor()
|
|
second = helpers._get_image_compressor()
|
|
|
|
assert isinstance(first, _FreshCompressor)
|
|
assert first is second
|
|
assert first._is_singleton is True
|
|
assert _FreshCompressor.instances == 1
|
|
|
|
|
|
def test_proxy_helper_caches_image_stack_import_failure(monkeypatch) -> None:
|
|
from headroom.proxy import helpers
|
|
|
|
real_import = builtins.__import__
|
|
calls = 0
|
|
|
|
def fake_import(name, *args, **kwargs): # noqa: ANN001, ANN202
|
|
nonlocal calls
|
|
if name == "headroom.image":
|
|
calls += 1
|
|
raise ImportError("image extras unavailable")
|
|
return real_import(name, *args, **kwargs)
|
|
|
|
monkeypatch.setattr(helpers, "_image_compressor_available", None)
|
|
monkeypatch.setattr(helpers, "_image_compressor_instance", None)
|
|
monkeypatch.setattr(builtins, "__import__", fake_import)
|
|
|
|
assert helpers._get_image_compressor() is None
|
|
assert helpers._get_image_compressor() is None
|
|
assert calls == 1
|
|
assert helpers._image_compressor_available is False
|
|
|
|
|
|
def test_anthropic_cache_delta_helpers_cover_string_list_and_role_mismatch() -> None:
|
|
previous_original = [{"role": "user", "content": "hello"}]
|
|
previous_forwarded = [{"role": "user", "content": "HELLO"}]
|
|
|
|
assert AnthropicHandlerMixin._extract_cache_stable_delta(
|
|
[{"role": "user", "content": "hello"}, {"role": "assistant", "content": "next"}],
|
|
previous_original,
|
|
previous_forwarded,
|
|
) == (previous_forwarded, [{"role": "assistant", "content": "next"}])
|
|
assert (
|
|
AnthropicHandlerMixin._extract_cache_stable_delta(
|
|
[{"role": "assistant", "content": "hello"}],
|
|
previous_original,
|
|
previous_forwarded,
|
|
)
|
|
is None
|
|
)
|
|
|
|
string_suffix = AnthropicHandlerMixin._extract_cache_stable_last_message_suffix(
|
|
[{"role": "user", "content": "hello world"}],
|
|
previous_original,
|
|
previous_forwarded,
|
|
)
|
|
assert string_suffix == ([], previous_forwarded[0], [{"role": "user", "content": " world"}])
|
|
|
|
list_suffix = AnthropicHandlerMixin._extract_cache_stable_last_message_suffix(
|
|
[
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "text", "text": "a"}, {"type": "text", "text": "b"}],
|
|
}
|
|
],
|
|
[{"role": "user", "content": [{"type": "text", "text": "a"}]}],
|
|
[{"role": "user", "content": [{"type": "text", "text": "A"}]}],
|
|
)
|
|
assert list_suffix == (
|
|
[],
|
|
{"role": "user", "content": [{"type": "text", "text": "A"}]},
|
|
[{"role": "user", "content": [{"type": "text", "text": "b"}]}],
|
|
)
|
|
|
|
assert AnthropicHandlerMixin._merge_appended_message_delta(
|
|
{"role": "user", "content": "HELLO"},
|
|
{"role": "user", "content": " world"},
|
|
) == {"role": "user", "content": "HELLO world"}
|
|
assert AnthropicHandlerMixin._merge_appended_message_delta(
|
|
{"role": "user", "content": [{"type": "text", "text": "A"}]},
|
|
{"role": "user", "content": [{"type": "text", "text": "b"}]},
|
|
) == {"role": "user", "content": [{"type": "text", "text": "A"}, {"type": "text", "text": "b"}]}
|
|
assert (
|
|
AnthropicHandlerMixin._merge_appended_message_delta(
|
|
{"role": "user", "content": "A"},
|
|
{"role": "assistant", "content": "B"},
|
|
)
|
|
is None
|
|
)
|
|
|
|
|
|
def test_anthropic_assistant_message_helper_requires_assistant_role() -> None:
|
|
assert AnthropicHandlerMixin._assistant_message_from_response_json(None) is None
|
|
assert AnthropicHandlerMixin._assistant_message_from_response_json({"role": "user"}) is None
|
|
assert AnthropicHandlerMixin._assistant_message_from_response_json(
|
|
{"role": "assistant", "content": [{"type": "text", "text": "ok"}]}
|
|
) == {"role": "assistant", "content": [{"type": "text", "text": "ok"}]}
|
|
|
|
|
|
# ============================================================================
|
|
# CCR workspace resolution (cross-project leak fix, 2026-05-26).
|
|
#
|
|
# These tests pin the `_resolve_ccr_workspace` static helper that the
|
|
# anthropic handler uses to scope the proactive-expansion cache by
|
|
# project identity. The resolver shares its tier order with the memory
|
|
# subsystem's ProjectResolver: x-headroom-project-id → x-headroom-cwd →
|
|
# CLI override → system-prompt `cwd:` line. Returns `("", None)` on
|
|
# no signal — the fail-closed signal that callers gate on.
|
|
# ============================================================================
|
|
|
|
|
|
def _fake_request(headers: dict[str, str]) -> SimpleNamespace:
|
|
"""Minimal Starlette/FastAPI-shaped request object for resolver tests."""
|
|
return SimpleNamespace(headers=headers)
|
|
|
|
|
|
def test_resolve_ccr_workspace_explicit_project_id_wins() -> None:
|
|
"""x-headroom-project-id is the highest-priority signal."""
|
|
request = _fake_request({"x-headroom-project-id": "my-cool-project"})
|
|
body = {}
|
|
key, label = AnthropicHandlerMixin()._resolve_ccr_workspace(request, body)
|
|
assert key.startswith("my-cool-project-")
|
|
assert len(key.split("-")[-1]) == 16
|
|
assert label == "my-cool-project"
|
|
|
|
|
|
def test_resolve_ccr_workspace_cwd_header() -> None:
|
|
"""x-headroom-cwd produces a stable per-cwd key + basename label."""
|
|
request = _fake_request({"x-headroom-cwd": "/home/user/code/daphni-rails"})
|
|
body = {}
|
|
key, label = AnthropicHandlerMixin()._resolve_ccr_workspace(request, body)
|
|
# Key format: "{basename}-{sha256[:16]}" — stable per absolute cwd.
|
|
assert key.startswith("daphni-rails-")
|
|
assert len(key) >= len("daphni-rails-") + 16
|
|
assert label == "daphni-rails"
|
|
|
|
|
|
def test_resolve_ccr_workspace_uses_configured_project_root_override() -> None:
|
|
"""The CLI project-root override reaches CCR workspace resolution."""
|
|
handler = AnthropicHandlerMixin()
|
|
handler.config = SimpleNamespace(memory_project_root_override="/home/user/code/project-c")
|
|
|
|
key, label = handler._resolve_ccr_workspace(_fake_request({}), {})
|
|
|
|
assert key.startswith("project-c-")
|
|
assert label == "project-c"
|
|
|
|
|
|
def test_resolve_ccr_workspace_two_cwds_get_distinct_keys() -> None:
|
|
"""Two different cwds produce different workspace keys (cross-leak prevention)."""
|
|
handler = AnthropicHandlerMixin()
|
|
key_a, _ = handler._resolve_ccr_workspace(
|
|
_fake_request({"x-headroom-cwd": "/home/user/code/daphni-rails"}), {}
|
|
)
|
|
key_b, _ = handler._resolve_ccr_workspace(
|
|
_fake_request({"x-headroom-cwd": "/home/user/code/tamag0"}), {}
|
|
)
|
|
assert key_a != key_b, "different cwds must yield different workspace keys"
|
|
|
|
|
|
def test_resolve_ccr_workspace_project_label_alone_fails_closed() -> None:
|
|
"""The savings label must not become a memory/CCR identity."""
|
|
key, label = AnthropicHandlerMixin()._resolve_ccr_workspace(
|
|
_fake_request({"x-headroom-project": "api"}), {}
|
|
)
|
|
assert key == ""
|
|
assert label is None
|
|
|
|
|
|
def test_resolve_ccr_workspace_project_label_does_not_collapse_cwds() -> None:
|
|
"""A user-supplied label cannot merge two distinct cwd identities."""
|
|
key_a, _ = AnthropicHandlerMixin()._resolve_ccr_workspace(
|
|
_fake_request(
|
|
{
|
|
"x-headroom-project": "api",
|
|
"x-headroom-cwd": "/work/acme/api",
|
|
}
|
|
),
|
|
{},
|
|
)
|
|
key_b, _ = AnthropicHandlerMixin()._resolve_ccr_workspace(
|
|
_fake_request(
|
|
{
|
|
"x-headroom-project": "api",
|
|
"x-headroom-cwd": "/work/other/api",
|
|
}
|
|
),
|
|
{},
|
|
)
|
|
assert key_a != key_b
|
|
|
|
|
|
def test_resolve_ccr_workspace_no_signal_returns_empty() -> None:
|
|
"""No project-id, no cwd header, no system prompt → fail-closed signal."""
|
|
request = _fake_request({})
|
|
body = {}
|
|
key, label = AnthropicHandlerMixin()._resolve_ccr_workspace(request, body)
|
|
assert key == ""
|
|
assert label is None
|
|
|
|
|
|
def test_resolve_ccr_workspace_system_prompt_cwd_fallback() -> None:
|
|
"""System prompt with `cwd:` line is the lowest-tier fallback."""
|
|
request = _fake_request({})
|
|
body = {
|
|
"system": [{"type": "text", "text": "You are helpful.\ncwd: /home/u/code/my-project\nGo."}]
|
|
}
|
|
key, label = AnthropicHandlerMixin()._resolve_ccr_workspace(request, body)
|
|
# The label is the basename of the cwd extracted from the prompt.
|
|
assert label == "my-project"
|
|
assert key.startswith("my-project-")
|
|
|
|
|
|
def test_resolve_ccr_workspace_malformed_request_returns_empty() -> None:
|
|
"""A request whose headers attribute can't be dict()-ed fails closed, not crashes."""
|
|
|
|
class _BrokenHeaders:
|
|
def __iter__(self):
|
|
raise RuntimeError("boom")
|
|
|
|
request = SimpleNamespace(headers=_BrokenHeaders())
|
|
body = {}
|
|
# The helper catches the exception, logs it, and returns the fail-
|
|
# closed sentinel ("", None). Critically, it does NOT raise — the
|
|
# proxy must continue serving the request even if CCR scoping fails.
|
|
key, label = AnthropicHandlerMixin()._resolve_ccr_workspace(request, body)
|
|
assert key == ""
|
|
assert label is None
|
|
|
|
|
|
class TestHasNewCcrMarkers:
|
|
"""#1850: replayed (overlay) markers must not count as new-this-turn.
|
|
|
|
``overlay_cached_prefix`` replays the previously-forwarded compressed prefix
|
|
byte-identical to keep the messages cache warm — which reintroduces its old
|
|
``hash=…`` markers. If those replayed markers counted as "new", the handler
|
|
would re-inject the retrieve tool every frozen turn and bust the *tools*
|
|
cache. ``has_new_ccr_markers`` filters them out.
|
|
"""
|
|
|
|
@staticmethod
|
|
def _hashes(*contents: str) -> list[str]:
|
|
from headroom.ccr.tool_injection import CCRToolInjector
|
|
|
|
inj = CCRToolInjector(
|
|
provider="anthropic", inject_tool=False, inject_system_instructions=False
|
|
)
|
|
inj.scan_for_markers([{"role": "user", "content": c} for c in contents])
|
|
return inj.detected_hashes
|
|
|
|
def test_replayed_markers_are_not_new(self):
|
|
from headroom.proxy.helpers import has_new_ccr_markers
|
|
|
|
marker = "[100 items compressed to 10. Retrieve more: hash=abc123def456abc123def456]"
|
|
current = self._hashes(marker)
|
|
assert current, "sanity: the marker must be detected"
|
|
# Every marker was already in what we forwarded last turn → nothing new.
|
|
assert (
|
|
has_new_ccr_markers(
|
|
current_detected_hashes=current,
|
|
previous_forwarded_messages=[{"role": "user", "content": marker}],
|
|
provider="anthropic",
|
|
)
|
|
is False
|
|
)
|
|
|
|
def test_genuinely_new_marker_is_detected(self):
|
|
from headroom.proxy.helpers import has_new_ccr_markers
|
|
|
|
old = "[100 items compressed to 10. Retrieve more: hash=abc123def456abc123def456]"
|
|
new = "[50 items compressed to 5. Retrieve more: hash=deadbeefdeadbeefdeadbeef]"
|
|
current = self._hashes(old, new)
|
|
# Only `old` was forwarded before; `new` is fresh → override must fire.
|
|
assert (
|
|
has_new_ccr_markers(
|
|
current_detected_hashes=current,
|
|
previous_forwarded_messages=[{"role": "user", "content": old}],
|
|
provider="anthropic",
|
|
)
|
|
is True
|
|
)
|
|
|
|
def test_no_previous_forward_means_all_new(self):
|
|
from headroom.proxy.helpers import has_new_ccr_markers
|
|
|
|
marker = "[100 items compressed to 10. Retrieve more: hash=abc123def456abc123def456]"
|
|
assert (
|
|
has_new_ccr_markers(
|
|
current_detected_hashes=self._hashes(marker),
|
|
previous_forwarded_messages=None,
|
|
provider="anthropic",
|
|
)
|
|
is True
|
|
)
|
|
|
|
def test_no_markers_means_nothing_new(self):
|
|
from headroom.proxy.helpers import has_new_ccr_markers
|
|
|
|
assert (
|
|
has_new_ccr_markers(
|
|
current_detected_hashes=[],
|
|
previous_forwarded_messages=None,
|
|
provider="anthropic",
|
|
)
|
|
is False
|
|
)
|
|
|
|
|
|
def test_strict_frozen_count_tool_and_function_tail_are_mutable():
|
|
# OpenAI function-calling harnesses (Kimi / fireworks) end each turn with a
|
|
# role:"tool" (or legacy role:"function") observation — NOT role:"user".
|
|
# Gating the mutable tail on role=="user" froze the whole conversation on
|
|
# every such turn => zero compression. Tool/function observations must be
|
|
# treated as the mutable delta (freeze all-but-last), like a user obs.
|
|
from headroom.proxy.handlers.openai import OpenAIHandlerMixin as M
|
|
|
|
# role:tool tail -> only the last message is mutable (frozen = final_idx)
|
|
assert (
|
|
M._strict_previous_turn_frozen_count(
|
|
[{"role": "user"}, {"role": "assistant"}, {"role": "tool"}], 0
|
|
)
|
|
== 2
|
|
)
|
|
assert (
|
|
M._strict_previous_turn_frozen_count(
|
|
[{"role": "user"}, {"role": "assistant"}, {"role": "function"}], 0
|
|
)
|
|
== 2
|
|
)
|
|
# assistant/system tail is NOT an observation -> freeze everything
|
|
assert (
|
|
M._strict_previous_turn_frozen_count(
|
|
[{"role": "user"}, {"role": "tool"}, {"role": "assistant"}], 0
|
|
)
|
|
== 3
|
|
)
|
|
|
|
|
|
class _ClientDisconnectRequest:
|
|
"""Mock request whose body() raises ClientDisconnect to simulate mid-stream cancel."""
|
|
|
|
method = "POST"
|
|
headers = {"content-type": "application/json"}
|
|
url = SimpleNamespace(path="/v1/chat/completions", query="")
|
|
|
|
async def body(self) -> bytes:
|
|
from starlette.requests import ClientDisconnect
|
|
|
|
raise ClientDisconnect()
|
|
|
|
|
|
class _ClientDisconnectStreamRequest:
|
|
"""Mock request for streaming passthrough with ClientDisconnect."""
|
|
|
|
method = "POST"
|
|
headers = {"content-type": "application/json"}
|
|
url = SimpleNamespace(
|
|
path="/v1/projects/p/locations/us-central1/publishers/google/models/gemini-2.0-flash:streamGenerateContent",
|
|
query="alt=sse",
|
|
)
|
|
|
|
async def body(self) -> bytes:
|
|
from starlette.requests import ClientDisconnect
|
|
|
|
raise ClientDisconnect()
|
|
|
|
|
|
def test_handle_passthrough_client_disconnect():
|
|
"""ClientDisconnect during body read returns 204 instead of crashing TaskGroup."""
|
|
handler = object.__new__(OpenAIHandlerMixin)
|
|
response = asyncio.run(
|
|
handler.handle_passthrough(_ClientDisconnectRequest(), "https://api.openai.com")
|
|
)
|
|
assert response.status_code == 204
|
|
|
|
|
|
def test_handle_streaming_passthrough_client_disconnect():
|
|
"""ClientDisconnect during streaming body read returns 204."""
|
|
handler = object.__new__(OpenAIHandlerMixin)
|
|
response = asyncio.run(
|
|
handler.handle_passthrough(
|
|
_ClientDisconnectStreamRequest(),
|
|
"https://us-central1-aiplatform.googleapis.com",
|
|
endpoint_name="streamRawPredict",
|
|
provider="vertex:google",
|
|
)
|
|
)
|
|
assert response.status_code == 204
|