mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Vertex rejects onboarding requests for a handful of recurring reasons — expired ADC, a partner model never enabled in Model Garden, a model not served in the requested location, a missing optional SDK — and its own error text names none of the fixes. Users see a bare `404` or an opaque `500` through their SDK and cannot tell a proxy bug from a project-config gap. This makes the proxy explain those failures where it happens, adds a `[vertex]` extra so the LiteLLM Vertex backend can actually start, and corrects an example, a test matrix, and three documentation claims that could not have worked as written. Closes #2632 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Explain Vertex failures in-band** (`headroom/providers/vertex/diagnostics.py`, new) - On `401/403/404/429`, attach an actionable hint three ways: a WARNING in the proxy log, an `x-headroom-hint` response header, and a note appended to `error.message`. The last is load-bearing — SDKs raise with the message string and discard headers. - Successful responses are never modified; annotation is idempotent; statuses we cannot explain get no hint rather than a plausible-sounding wrong one. - Routes opt in via a one-line `@with_vertex_diagnostics` decorator reading path params FastAPI already injects (**+8 lines** in `proxy_routes.py`, no re-indentation). - Same treatment for backend setup failures, where `--backend vertex` surfaced LiteLLM's raw `No module named 'vertexai'` as an opaque 500. **Make `--backend vertex` able to start** (`pyproject.toml`, `registry.py`, `cli/proxy.py`) - New `[vertex]` extra carrying `google-cloud-aiplatform>=1.38`. Deliberately **not** folded into `[proxy]`: measured at ~175 MB installed, and the native passthrough routes never needed it. Follows the existing `proxy-prod` split. - Startup preflight keyed on the **resolved provider**, so every alias normalized at `registry.py:229` (`vertex`, `vertex_ai`, `litellm-vertex`, `google-vertex`, `googlevertex`) is covered. Skipped when a backend class is injected, so existing fake-based tests are unaffected. - New `BackendUnavailableError` so the CLI prints the remedy and exits `2` instead of dumping a traceback through startup internals. **Correct things that could not have worked** | | was | is | |---|---|---| | SDK client | `http_options={"api_endpoint": ...}` | `{"base_url": "http://..."}` — `api_endpoint` is not a field on `HttpOptions`; pydantic rejects it before any request is sent | | Thinking config | `ThinkingConfig(thinking_budget_tokens=)` | `thinking_budget=` | | Request payload | `contents: [{parts: [...]}]` | adds `role: "user"` — the `global` endpoint enforces it, regional was silently lenient | | Auth guidance | `gcloud auth print-access-token` | `application-default` — the plain user token gets `401 ACCESS_TOKEN_TYPE_UNSUPPORTED` | | Test matrix | model × location cross-product | explicit servable pairs | - Test matrix is now four pairs that can actually serve: `global`/`gemini-flash-latest`, `global`/`claude-sonnet-4-6`, `europe-west2`/`gemini-3.5-flash`, `us-east5`/`claude-sonnet-4-6`. Gemini 3.x has no US regional endpoint and `-latest` aliases are global-only, so a pinned `us-central1` Gemini case can never pass. No Gemini 2.5 — it retires 2026-10-16. - `test_thinking_levels` asserted only `status_code < 500` while sending an ignored `thinkingBudgetTokens` key: it passed on 400s and proved nothing. Now asserts 200 plus scored usage metadata with the correct `thinkingBudget` spelling. - Cases the caller's project genuinely cannot serve now **skip** with an actionable reason instead of reporting a red proxy. **Docs** (`proxy.mdx`, `installation.mdx`, `wiki/vertex.md`) - `proxy.mdx` claimed `publisher=anthropic` passthrough "routes through the same LiteLLM-Vertex path as `--backend vertex_ai`". It does not — without the flag it is a direct forward, which is why it works on a bare `[proxy]` install where the LiteLLM path cannot even import. - Documented the trap of combining `--backend vertex` with native Vertex URLs: LiteLLM takes project/region/model from its own config and ignores the URL. - `installation.mdx` and `wiki/vertex.md` said the SDK is "not included in any extra". - Cleared 15 pre-existing `ruff` findings in the two files this PR rewrites (4 remain in `plugins/headroom-oauth2`, untouched by this work). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ mypy headroom Success: no issues found in 511 source files $ ruff check . Found 4 errors. # all in plugins/headroom-oauth2, pre-existing, untouched by this PR $ pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q 44 passed, 1 warning in 69.86s # 8 live integration + 36 offline diagnostics, 0 skipped $ pytest tests/ -k "anthropic_messages or handler_anthropic or backend or registry or cli_proxy or vertex or proxy_routes" 853 passed, 49 skipped, 9345 deselected, 2 warnings in 28.41s ``` New tests in `tests/test_vertex_diagnostics.py` (36, no credentials required): status coverage, per-publisher remedy divergence, idempotency, `content-length` consistency after body rewrite, non-JSON and body-less/streaming responses, every backend alias preflighted, other providers unaffected, injected backend bypasses the preflight. ## Real Behavior Proof - Environment: macOS 14, Python 3.11.15, live GCP project (`alanblount-sandbox`) with `aiplatform.googleapis.com` enabled and Claude enabled in Model Garden; ADC via `gcloud auth application-default login` - Exact command / steps: `export GCP_PROJECT_ID=alanblount-sandbox && export GCP_ACCESS_TOKEN=$(gcloud auth application-default print-access-token) && uv run --extra proxy --with google-genai python3 examples/vertex_genai_sdk_demo.py && uv run --extra proxy --with pytest-asyncio pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q && headroom proxy --backend vertex` (exits 2, then retried with `[proxy,vertex]` extra installed) - Observed result: demo printed `✓ Standard response received successfully! > 1, 2, 3, 4, 5` and `✓ Thinking response received successfully!`; 44 tests passed (8 live integration, 36 diagnostics), 0 skipped; `--backend vertex` exited 2 with the remedy message; same request with the extra installed returned 200; invalid region produced `[headroom] hint:` in the response body and `x-headroom-hint` header; success responses were byte-clean - Not tested: Windows; `us`/`eu` multi-region endpoints and Claude 4.7+ (project cannot serve them); Provisioned Throughput; Rust proxy path; `--backend vertex` dropping URL project/model on native Vertex routes (documented as known gap in Additional Notes) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy and CLI behavior; terminal output is quoted above. ## Additional Notes - `uv.lock` intentionally not included: the new extra is opt-in and does not change default resolution. - Error-body rewriting is confined to already-failing responses and wrapped so a diagnostic can never turn a clean upstream error into a proxy 500. - The preflight raises where the provider is resolved (`registry.py`) rather than at the CLI, so programmatic embedders get the same guarantee; the CLI only formats it. - Three test files fail to collect on this branch (`test_hermes_passthrough_compression`, `test_observability_metrics`, `test_observability_tracing`). They predate this work and are unrelated, so they were excluded rather than fixed — happy to open a separate issue. - `[vertex]` was chosen over adding the dependency to `[proxy]` after measuring the footprint at ~175 MB. If maintainers prefer it in `[proxy]`, that is a one-line change. ## Runtime Rollout Safety - Rollout-managed feature(s): none; diagnostics annotate already-failing Vertex responses and the backend dependency is opt-in through the vertex extra. - Minimum rollout channel: normal stable release after Vertex diagnostics, route, and backend-path tests pass. - Stable/default behavior changed: successful responses and native passthrough remain unchanged; supported Vertex failures gain an actionable header/message, and selecting the LiteLLM Vertex backend without its SDK now fails once at startup. - Kill switch / disable path: use native Vertex passthrough without the LiteLLM backend, or human-revert the diagnostic/preflight change. - Unsafe override required: none. - Qualification impact: offline diagnostics, native integration, and mocked LiteLLM Vertex request-shaping tests must remain green. - Rollback path: human-revert the PR; no persisted state or migration is involved. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2640" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `5d7071f348d2462f666fd7ac1ceb8ac18050169a`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration headc70c1a497and current author responses, including fixed public error hints, log escaping, body/header consistency, all four backend error sites, and native-route versus LiteLLM SDK dependency behavior. Clean integration with upstream/main2d3ee51846. No new PR-specific source blocker found in reviewed scope; no contributor diagnostic/routing logic rewritten. 232 Vertex diagnostics/native-route/backend/CLI/release/version/MCP tests pass with11 skips. Offline lock validation passes (268 packages); all12 changed Python files pass Ruff check/format, and whitespace passes. These checks cover synthetic error/transport paths and do not establish live cloud regional availability or billing behavior. Earlier author live evidence remains scoped to its stated environment; no fresh cloud request or native build claimed here. Historical shared cached-read fixture failure was main-reproduced and is not a contributor-code blocker. Maintainers handle current exact-head hosted qualification before approval; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `486a716adbf642ae5e43276df4c2dc1e63b901d7`, upstream main `c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`. Clean mechanical sync with upstream mainc01cf07d94. Re-reviewed current author/Devin discussion and public diagnostic/error boundaries, native-versus-SDK route separation and startup preflight. This sync does not alter the PR Vertex diagnostics, backend or route-registry modules.158 focused diagnostics/native-route/backend/streaming/CLI tests pass,11live-environment skips,1deprecation warning. Offline uv lock --check passes;all12 changed Python files pass Ruff lint/format and main-relative whitespace passes. Fixed hints remain separate from raw provider error text; multi-worker preflight and non-Google partner guidance retain resolved regressions. No contributor diagnostic/routing correctness logic rewritten. Tests use existing current-main DLL and synthetic errors/transports; no fresh native build, live regional availability/provider inference or billing qualification claimed. Formal approval awaits exact-head hosted qualification; maintainers handle CI. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `0b75c609f24562c8b6ed2aaeaf58239984b4810c`, upstream main `ce4275491299231bc9d2907833f72f72ea22b8a9`. Requalified after human merges #4667,#4669,#4670 against upstream maince42754912. Earlier approval and publication guards stopped stale candidates before either was posted. Current clean merge retains Vertex diagnostics/native-versus-SDK routing and main enable_kompress checks, lossless cold-prefix disable, and Python token-window boundary behavior.190 combined Vertex/backend/CLI/Kompress-disable/must-keep tests pass,11live-environment skips,1warning. Offline lock validation,12-file Ruff lint/format and main-relative whitespace pass. No contributor diagnostic/routing logic rewritten. Native Rust/Cargo sources are identical to c01 across these merges; existing matching-core DLL supports tests, not a fresh native build. No new live Vertex regional/provider/billing qualification claimed. All prior-main hosted results are superseded for readiness; approval awaits exact checks on this published merge head. Maintainers handle CI; no agent merge. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `1fe62702ce614315679d68d9d126e31c57208c67`, upstream main `00d4f441d3f1d48c463fea57e6c574c2f3671302`. Requalified after human merges #4667,#4669,#4670 against upstream maince42754912. Earlier approval and publication guards stopped stale candidates before either was posted. Current clean merge retains Vertex diagnostics/native-versus-SDK routing and main enable_kompress checks, lossless cold-prefix disable, and Python token-window boundary behavior.190 combined Vertex/backend/CLI/Kompress-disable/must-keep tests pass,11live-environment skips,1warning. Offline lock validation,12-file Ruff lint/format and main-relative whitespace pass. No contributor diagnostic/routing logic rewritten. Native Rust/Cargo sources are identical to c01 across these merges; existing matching-core DLL supports tests, not a fresh native build. No new live Vertex regional/provider/billing qualification claimed. All prior-main hosted results are superseded for readiness; approval awaits exact checks on this published merge head. Maintainers handle CI; no agent merge. Final current-main requalification:07d5add5ba, including human merges #4671 action pinning and #4648 multimodal memory preservation. Clean mechanical merge; Vertex/backend/CLI inputs retain the preceding 190-test qualification. Fresh workflow/release suite: 62 passed. Fresh memory wrapper suite: 9 passed. New wrapper retains list payload parts and extracts textual search inputs; no contributor Vertex logic rewritten. Existing native input provenance remains unchanged. Hosted qualification must run on this final head before approval. Newer human merge #4646 advanced authoritative main to8cdce172eabefore publication. Cleanly integrated quote-aware read detection; reviewed new masking/segmentation overlap. Fresh read-command/content-router suite: 188 passed, 1 optional SOCKS warning. Vertex code remains unchanged by this merge. Final additional human merge #4067 packaging changes advanced main toa8905214d: cleanly integrated explicit torch-extras split, retaining Vertex optional dependencies. Fresh ONNX/optional-dependency contracts: 12 passed. Offline lock check and whitespace pass. No live clean-install or Torch backend claim. Earlier publication candidates were stopped before push by fresh-main guard. Latest human merge #4042 advances authoritative main tocde584d7fe. Clean merge preserves optional LangChain availability handling and CI dependency setup. Fresh binding/compatibility/startup-log/xAI regression suite: 29 passed. New main integration re-reviewed; no contributor Vertex behavior rewritten. Subsequent #4049 main merge00d4f441dadds graceful context lookup/init failure handling. Cleanly integrated with Vertex handler diagnostics intact. Fresh Anthropic pre-upstream/memory native ops/OpenAI Codex routing/WebSocket lifecycle suite: 192 passed, 1 deprecation warning. Reviewed exception handling and transport overlap; no live provider qualification claim. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `6c63811d6de7491ebce9a16f89e197f5f5dd86dd`, upstream main `2b0315b2d4b0d54947050bbdad2de9e89df06378`. Requalified against authoritative main2b0315b2d4b0d54947050bbdad2de9e89df06378 after further human merges. All10 integration patches exactly match00d4f441-to-current-main changes after diff-location normalization; contributor Vertex diagnostics/routing policy unchanged.539combined Vertex/backend/CLI/Kompress-disable/must-keep/binary/doctor/Windows-supervisor/provider/search/text tests pass;13platform/live-environment skips,2warnings. Nine changed Python files pass Ruff lint/format; offline lock and whitespace pass. Existing pr3982 native DLL verified against every tracked native build-input hash and artifact checksum; no fresh native build claimed. Current author/Devin discussion checked; prior fixes remain intact. No live Vertex regional/provider/billing,IPv6 listener or unsupported cloud qualification claimed. No new finding in reviewed scope. Exact-head hosted checks must qualify this published head before formal approval; maintainers handle them. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
442 lines
16 KiB
Python
442 lines
16 KiB
Python
"""Tests for the OpenAI chat-completions backend (LiteLLM/Bedrock) path.
|
|
|
|
Covers Fix #1 (PrefixCacheTracker.update_from_response on backend path)
|
|
and Fix #2 (CCR response intercept for the OpenAI provider shape) on the
|
|
non-streaming backend path of ``handle_openai_chat``.
|
|
|
|
All three scenarios mock ``anthropic_backend.send_openai_message`` so we
|
|
don't need a real provider:
|
|
|
|
1. Backend response with cache_read_input_tokens > 0 → tracker.update_from_response
|
|
is called with the right cache_read_tokens and cache_write_tokens.
|
|
2. Backend response with headroom_retrieve tool call → ccr_response_handler.handle_response
|
|
is awaited with provider="openai", and the final body returned.
|
|
3. CCR intercept exception path → re-raises (NOT swallowed).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from types import SimpleNamespace
|
|
from unittest.mock import AsyncMock, MagicMock, patch
|
|
|
|
import pytest
|
|
|
|
fastapi = pytest.importorskip("fastapi")
|
|
httpx = pytest.importorskip("httpx")
|
|
|
|
from fastapi.testclient import TestClient # noqa: E402
|
|
|
|
from headroom.backends.base import BackendResponse # noqa: E402
|
|
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
|
|
|
|
|
class _RecordingTracker:
|
|
"""Stub PrefixCacheTracker that records ``update_from_response`` calls."""
|
|
|
|
def __init__(self, provider: str = "openai") -> None:
|
|
self.provider = provider
|
|
self.calls: list[dict] = []
|
|
self._frozen = 0
|
|
self._last_original: list[dict] = []
|
|
self._last_forwarded: list[dict] = []
|
|
|
|
def update_from_response(
|
|
self,
|
|
cache_read_tokens: int,
|
|
cache_write_tokens: int,
|
|
messages: list[dict],
|
|
message_token_counts: list[int] | None = None,
|
|
original_messages: list[dict] | None = None,
|
|
) -> None:
|
|
self.calls.append(
|
|
{
|
|
"cache_read_tokens": cache_read_tokens,
|
|
"cache_write_tokens": cache_write_tokens,
|
|
"messages": messages,
|
|
}
|
|
)
|
|
self._last_original = list(original_messages or messages)
|
|
self._last_forwarded = list(messages)
|
|
|
|
# Minimal surface used by handle_openai_chat — return 0 so we never freeze.
|
|
def get_frozen_message_count(self) -> int:
|
|
return self._frozen
|
|
|
|
def get_last_original_messages(self) -> list[dict]:
|
|
return list(self._last_original)
|
|
|
|
def get_last_forwarded_messages(self) -> list[dict]:
|
|
return list(self._last_forwarded)
|
|
|
|
|
|
def _make_config() -> ProxyConfig:
|
|
return ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
backend="anyllm",
|
|
anyllm_provider="openai",
|
|
)
|
|
|
|
|
|
def _make_mock_backend(response_body: dict, status_code: int = 200) -> MagicMock:
|
|
backend = MagicMock()
|
|
backend.name = "anyllm-openai"
|
|
backend.send_openai_message = AsyncMock(
|
|
return_value=BackendResponse(
|
|
body=response_body,
|
|
status_code=status_code,
|
|
headers={"content-type": "application/json"},
|
|
)
|
|
)
|
|
return backend
|
|
|
|
|
|
def _make_litellm_response() -> SimpleNamespace:
|
|
return SimpleNamespace(
|
|
id="resp_2392",
|
|
created=123456,
|
|
choices=[
|
|
SimpleNamespace(
|
|
index=0,
|
|
finish_reason="stop",
|
|
message=SimpleNamespace(role="assistant", content="ok", tool_calls=None),
|
|
)
|
|
],
|
|
usage=SimpleNamespace(prompt_tokens=2, completion_tokens=3, total_tokens=5),
|
|
)
|
|
|
|
|
|
def _install_tracker_stub(client: TestClient) -> _RecordingTracker:
|
|
"""Force the session_tracker_store to hand out our recording tracker."""
|
|
tracker = _RecordingTracker(provider="openai")
|
|
# Find the proxy instance behind the app — it's stored as app.state.proxy.
|
|
proxy = client.app.state.proxy
|
|
proxy.session_tracker_store.get_or_create = MagicMock(return_value=tracker)
|
|
return tracker
|
|
|
|
|
|
def test_backend_response_updates_prefix_tracker_with_bedrock_cache_fields():
|
|
"""Bedrock/Anthropic-shape cache fields → tracker sees authoritative read/write counts."""
|
|
config = _make_config()
|
|
response_body = {
|
|
"id": "chatcmpl-bedrock-1",
|
|
"object": "chat.completion",
|
|
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": "Hi!"},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {
|
|
"prompt_tokens": 1000,
|
|
"completion_tokens": 20,
|
|
"total_tokens": 1020,
|
|
# Bedrock/Anthropic top-level keys
|
|
"cache_read_input_tokens": 700,
|
|
"cache_creation_input_tokens": 100,
|
|
# OpenAI shape (always populated by the LiteLLM normalizer)
|
|
"prompt_tokens_details": {"cached_tokens": 700},
|
|
},
|
|
}
|
|
|
|
mock_backend = _make_mock_backend(response_body)
|
|
with patch("headroom.proxy.server.AnyLLMBackend", return_value=mock_backend):
|
|
app = create_app(config)
|
|
with TestClient(app) as client:
|
|
tracker = _install_tracker_stub(client)
|
|
resp = client.post(
|
|
"/v1/chat/completions",
|
|
json={
|
|
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
"stream": False,
|
|
},
|
|
headers={"Authorization": "Bearer test-key"},
|
|
)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
assert mock_backend.send_openai_message.await_count == 1
|
|
assert len(tracker.calls) == 1, tracker.calls
|
|
call = tracker.calls[0]
|
|
# Prefer the Bedrock authoritative top-level read/write counts.
|
|
assert call["cache_read_tokens"] == 700
|
|
assert call["cache_write_tokens"] == 100
|
|
|
|
|
|
def test_backend_response_falls_back_to_openai_cached_tokens_when_bedrock_keys_absent():
|
|
"""Pure OpenAI shape (no top-level Anthropic keys) → fall back to prompt_tokens_details + infer write."""
|
|
config = _make_config()
|
|
response_body = {
|
|
"id": "chatcmpl-openai-1",
|
|
"object": "chat.completion",
|
|
"model": "gpt-4o-mini",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": "Hi!"},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {
|
|
"prompt_tokens": 500,
|
|
"completion_tokens": 10,
|
|
"total_tokens": 510,
|
|
# No top-level Anthropic keys, only OpenAI shape
|
|
"prompt_tokens_details": {"cached_tokens": 200},
|
|
},
|
|
}
|
|
|
|
mock_backend = _make_mock_backend(response_body)
|
|
with patch("headroom.proxy.server.AnyLLMBackend", return_value=mock_backend):
|
|
app = create_app(config)
|
|
with TestClient(app) as client:
|
|
tracker = _install_tracker_stub(client)
|
|
resp = client.post(
|
|
"/v1/chat/completions",
|
|
json={
|
|
"model": "gpt-4o-mini",
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
"stream": False,
|
|
},
|
|
headers={"Authorization": "Bearer test-key"},
|
|
)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
assert len(tracker.calls) == 1
|
|
call = tracker.calls[0]
|
|
assert call["cache_read_tokens"] == 200
|
|
# No cache_creation_input_tokens → inferred = prompt_tokens - cache_read = 500 - 200 = 300
|
|
assert call["cache_write_tokens"] == 300
|
|
|
|
|
|
def test_litellm_vertex_backend_path_preserves_max_tokens_and_vendor_fields():
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
backend="litellm-vertex",
|
|
)
|
|
|
|
with (
|
|
patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}),
|
|
patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp,
|
|
# Suppress the SDK preflight: this test proves LiteLLM request-shaping,
|
|
# not whether google-cloud-aiplatform is installed in the CI environment.
|
|
patch("headroom.providers.vertex.diagnostics.vertex_sdk_available", return_value=True),
|
|
):
|
|
mock_acomp.return_value = _make_litellm_response()
|
|
app = create_app(config)
|
|
with TestClient(app) as client:
|
|
response = client.post(
|
|
"/v1/chat/completions",
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
"max_tokens": 32,
|
|
"chat_template_kwargs": {"enable_thinking": False},
|
|
"stream": False,
|
|
},
|
|
headers={"Authorization": "Bearer test-key"},
|
|
)
|
|
|
|
assert response.status_code == 200, response.text
|
|
kwargs = mock_acomp.await_args.kwargs
|
|
assert kwargs["max_tokens"] == 32
|
|
assert kwargs["extra_body"] == {"chat_template_kwargs": {"enable_thinking": False}}
|
|
assert "max_completion_tokens" not in kwargs["extra_body"]
|
|
|
|
|
|
def test_backend_response_with_ccr_tool_call_is_intercepted_and_resolved():
|
|
"""OpenAI-shape response carrying headroom_retrieve → CCR handler resolves it."""
|
|
config = _make_config()
|
|
# First response: tool_call for headroom_retrieve
|
|
tool_call_response = {
|
|
"id": "chatcmpl-ccr-1",
|
|
"object": "chat.completion",
|
|
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_abc",
|
|
"type": "function",
|
|
"function": {
|
|
"name": "headroom_retrieve",
|
|
"arguments": '{"hash": "deadbeef"}',
|
|
},
|
|
}
|
|
],
|
|
},
|
|
"finish_reason": "tool_calls",
|
|
}
|
|
],
|
|
"usage": {
|
|
"prompt_tokens": 100,
|
|
"completion_tokens": 10,
|
|
"total_tokens": 110,
|
|
},
|
|
}
|
|
final_resp_json = {
|
|
"id": "chatcmpl-ccr-final",
|
|
"object": "chat.completion",
|
|
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": "Resolved!"},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {
|
|
"prompt_tokens": 100,
|
|
"completion_tokens": 5,
|
|
"total_tokens": 105,
|
|
},
|
|
}
|
|
|
|
mock_backend = _make_mock_backend(tool_call_response)
|
|
with patch("headroom.proxy.server.AnyLLMBackend", return_value=mock_backend):
|
|
app = create_app(config)
|
|
with TestClient(app) as client:
|
|
_install_tracker_stub(client)
|
|
proxy = client.app.state.proxy
|
|
# Replace the response handler with a recording mock.
|
|
recording_handler = MagicMock()
|
|
recording_handler.has_ccr_tool_calls = MagicMock(return_value=True)
|
|
recording_handler.handle_response = AsyncMock(return_value=final_resp_json)
|
|
proxy.ccr_response_handler = recording_handler
|
|
|
|
resp = client.post(
|
|
"/v1/chat/completions",
|
|
json={
|
|
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
"stream": False,
|
|
},
|
|
headers={"Authorization": "Bearer test-key"},
|
|
)
|
|
|
|
assert resp.status_code == 200, resp.text
|
|
# handle_response was awaited with provider="openai"
|
|
recording_handler.handle_response.assert_awaited_once()
|
|
_args, kwargs = recording_handler.handle_response.call_args
|
|
assert kwargs.get("provider") == "openai"
|
|
# Resolved body propagated back to the client
|
|
assert resp.json()["choices"][0]["message"]["content"] == "Resolved!"
|
|
|
|
|
|
def test_backend_ccr_intercept_exception_is_reraised_not_swallowed():
|
|
"""CCR resolution failure on the backend path → 500, NOT silent fallback to original body."""
|
|
config = _make_config()
|
|
tool_call_response = {
|
|
"id": "chatcmpl-ccr-fail",
|
|
"object": "chat.completion",
|
|
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_bad",
|
|
"type": "function",
|
|
"function": {
|
|
"name": "headroom_retrieve",
|
|
"arguments": '{"hash": "badhash"}',
|
|
},
|
|
}
|
|
],
|
|
},
|
|
"finish_reason": "tool_calls",
|
|
}
|
|
],
|
|
"usage": {"prompt_tokens": 50, "completion_tokens": 5, "total_tokens": 55},
|
|
}
|
|
|
|
records: list[logging.LogRecord] = []
|
|
|
|
class _Collect(logging.Handler):
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
records.append(record)
|
|
|
|
handler = _Collect(level=logging.DEBUG)
|
|
proxy_logger = logging.getLogger("headroom.proxy")
|
|
|
|
mock_backend = _make_mock_backend(tool_call_response)
|
|
with patch("headroom.proxy.server.AnyLLMBackend", return_value=mock_backend):
|
|
app = create_app(config)
|
|
with TestClient(app) as client:
|
|
# Attach after startup: proxy logging setup would otherwise
|
|
# detach a handler added earlier (caplog has the same problem).
|
|
proxy_logger.addHandler(handler)
|
|
_install_tracker_stub(client)
|
|
proxy = client.app.state.proxy
|
|
failing_handler = MagicMock()
|
|
failing_handler.has_ccr_tool_calls = MagicMock(return_value=True)
|
|
failing_handler.handle_response = AsyncMock(
|
|
side_effect=RuntimeError("ccr-store-blew-up")
|
|
)
|
|
proxy.ccr_response_handler = failing_handler
|
|
|
|
resp = client.post(
|
|
"/v1/chat/completions",
|
|
json={
|
|
"model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
"stream": False,
|
|
},
|
|
headers={"Authorization": "Bearer test-key"},
|
|
)
|
|
proxy_logger.removeHandler(handler)
|
|
|
|
# The outer `try/except Exception` on the backend block converts the
|
|
# re-raise into a 500 response. The critical assertion is that the
|
|
# original tool_call body is NOT returned to the client — which is
|
|
# what a silent fallback would do.
|
|
failing_handler.handle_response.assert_awaited_once()
|
|
assert resp.status_code == 500, (
|
|
f"expected 500 (CCR error re-raised), got {resp.status_code}: {resp.text[:200]}"
|
|
)
|
|
body = resp.json()
|
|
# Confirm we didn't propagate the original tool_call body.
|
|
assert (
|
|
"choices" not in body
|
|
or body.get("choices", [{}])[0].get("message", {}).get("tool_calls") is None
|
|
)
|
|
assert "error" in body
|
|
# The client gets the fixed vocabulary and a correlation id, not the
|
|
# exception text; the operator log keeps the detail under that id.
|
|
assert body["error"]["code"] == "internal_error"
|
|
request_id = body["error"]["request_id"]
|
|
assert request_id and f"request_id={request_id}" in body["error"]["message"]
|
|
assert "ccr-store-blew-up" not in resp.text
|
|
logged = [r.getMessage() for r in records]
|
|
assert any(f"[{request_id}]" in m and "ccr-store-blew-up" in m for m in logged), logged
|
|
|
|
|
|
def test_backend_streaming_passes_prefix_tracker_through():
|
|
"""Streaming backend path should accept and use prefix_tracker — non-regression smoke."""
|
|
# The wiring contract is structural — just confirm the parameter exists.
|
|
import inspect
|
|
|
|
from headroom.proxy.handlers.streaming import StreamingMixin
|
|
|
|
sig = inspect.signature(StreamingMixin._stream_openai_via_backend)
|
|
assert "prefix_tracker" in sig.parameters, (
|
|
"_stream_openai_via_backend must accept prefix_tracker to match the direct path"
|
|
)
|
|
assert "optimized_messages" in sig.parameters, (
|
|
"_stream_openai_via_backend must accept optimized_messages so the "
|
|
"tracker can record the messages that were sent"
|
|
)
|