Files
headroom/tests/test_vertex_diagnostics.py
alan blountandJD Davis 0ddaa56c21 fix(vertex): explain Vertex failures in-band, correct the model/region matrix (#2640)
## Description

Vertex rejects onboarding requests for a handful of recurring reasons —
expired ADC, a partner model never enabled in Model Garden, a model not
served in the requested location, a missing optional SDK — and its own
error text names none of the fixes. Users see a bare `404` or an opaque
`500` through their SDK and cannot tell a proxy bug from a
project-config gap.

This makes the proxy explain those failures where it happens, adds a
`[vertex]` extra so the LiteLLM Vertex backend can actually start, and
corrects an example, a test matrix, and three documentation claims that
could not have worked as written.

Closes #2632

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

**Explain Vertex failures in-band**
(`headroom/providers/vertex/diagnostics.py`, new)

- On `401/403/404/429`, attach an actionable hint three ways: a WARNING
in the proxy log, an `x-headroom-hint` response header, and a note
appended to `error.message`. The last is load-bearing — SDKs raise with
the message string and discard headers.
- Successful responses are never modified; annotation is idempotent;
statuses we cannot explain get no hint rather than a plausible-sounding
wrong one.
- Routes opt in via a one-line `@with_vertex_diagnostics` decorator
reading path params FastAPI already injects (**+8 lines** in
`proxy_routes.py`, no re-indentation).
- Same treatment for backend setup failures, where `--backend vertex`
surfaced LiteLLM's raw `No module named 'vertexai'` as an opaque 500.

**Make `--backend vertex` able to start** (`pyproject.toml`,
`registry.py`, `cli/proxy.py`)

- New `[vertex]` extra carrying `google-cloud-aiplatform>=1.38`.
Deliberately **not** folded into `[proxy]`: measured at ~175 MB
installed, and the native passthrough routes never needed it. Follows
the existing `proxy-prod` split.
- Startup preflight keyed on the **resolved provider**, so every alias
normalized at `registry.py:229` (`vertex`, `vertex_ai`,
`litellm-vertex`, `google-vertex`, `googlevertex`) is covered. Skipped
when a backend class is injected, so existing fake-based tests are
unaffected.
- New `BackendUnavailableError` so the CLI prints the remedy and exits
`2` instead of dumping a traceback through startup internals.

**Correct things that could not have worked**

| | was | is |
|---|---|---|
| SDK client | `http_options={"api_endpoint": ...}` | `{"base_url":
"http://..."}` — `api_endpoint` is not a field on `HttpOptions`;
pydantic rejects it before any request is sent |
| Thinking config | `ThinkingConfig(thinking_budget_tokens=)` |
`thinking_budget=` |
| Request payload | `contents: [{parts: [...]}]` | adds `role: "user"` —
the `global` endpoint enforces it, regional was silently lenient |
| Auth guidance | `gcloud auth print-access-token` |
`application-default` — the plain user token gets `401
ACCESS_TOKEN_TYPE_UNSUPPORTED` |
| Test matrix | model × location cross-product | explicit servable pairs
|

- Test matrix is now four pairs that can actually serve:
`global`/`gemini-flash-latest`, `global`/`claude-sonnet-4-6`,
`europe-west2`/`gemini-3.5-flash`, `us-east5`/`claude-sonnet-4-6`.
Gemini 3.x has no US regional endpoint and `-latest` aliases are
global-only, so a pinned `us-central1` Gemini case can never pass. No
Gemini 2.5 — it retires 2026-10-16.
- `test_thinking_levels` asserted only `status_code < 500` while sending
an ignored `thinkingBudgetTokens` key: it passed on 400s and proved
nothing. Now asserts 200 plus scored usage metadata with the correct
`thinkingBudget` spelling.
- Cases the caller's project genuinely cannot serve now **skip** with an
actionable reason instead of reporting a red proxy.

**Docs** (`proxy.mdx`, `installation.mdx`, `wiki/vertex.md`)

- `proxy.mdx` claimed `publisher=anthropic` passthrough "routes through
the same LiteLLM-Vertex path as `--backend vertex_ai`". It does not —
without the flag it is a direct forward, which is why it works on a bare
`[proxy]` install where the LiteLLM path cannot even import.
- Documented the trap of combining `--backend vertex` with native Vertex
URLs: LiteLLM takes project/region/model from its own config and ignores
the URL.
- `installation.mdx` and `wiki/vertex.md` said the SDK is "not included
in any extra".
- Cleared 15 pre-existing `ruff` findings in the two files this PR
rewrites (4 remain in `plugins/headroom-oauth2`, untouched by this
work).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ mypy headroom
Success: no issues found in 511 source files

$ ruff check .
Found 4 errors.        # all in plugins/headroom-oauth2, pre-existing, untouched by this PR

$ pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q
44 passed, 1 warning in 69.86s          # 8 live integration + 36 offline diagnostics, 0 skipped

$ pytest tests/ -k "anthropic_messages or handler_anthropic or backend or registry or cli_proxy or vertex or proxy_routes"
853 passed, 49 skipped, 9345 deselected, 2 warnings in 28.41s
```

New tests in `tests/test_vertex_diagnostics.py` (36, no credentials
required): status coverage, per-publisher remedy divergence,
idempotency, `content-length` consistency after body rewrite, non-JSON
and body-less/streaming responses, every backend alias preflighted,
other providers unaffected, injected backend bypasses the preflight.

## Real Behavior Proof

- Environment: macOS 14, Python 3.11.15, live GCP project
(`alanblount-sandbox`) with `aiplatform.googleapis.com` enabled and
Claude enabled in Model Garden; ADC via `gcloud auth application-default
login`
- Exact command / steps: `export GCP_PROJECT_ID=alanblount-sandbox &&
export GCP_ACCESS_TOKEN=$(gcloud auth application-default
print-access-token) && uv run --extra proxy --with google-genai python3
examples/vertex_genai_sdk_demo.py && uv run --extra proxy --with
pytest-asyncio pytest tests/test_proxy_vertex_native_integration.py
tests/test_vertex_diagnostics.py -q && headroom proxy --backend vertex`
(exits 2, then retried with `[proxy,vertex]` extra installed)
- Observed result: demo printed `✓ Standard response received
successfully! > 1, 2, 3, 4, 5` and `✓ Thinking response received
successfully!`; 44 tests passed (8 live integration, 36 diagnostics), 0
skipped; `--backend vertex` exited 2 with the remedy message; same
request with the extra installed returned 200; invalid region produced
`[headroom] hint:` in the response body and `x-headroom-hint` header;
success responses were byte-clean
- Not tested: Windows; `us`/`eu` multi-region endpoints and Claude 4.7+
(project cannot serve them); Provisioned Throughput; Rust proxy path;
`--backend vertex` dropping URL project/model on native Vertex routes
(documented as known gap in Additional Notes)


## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — proxy and CLI behavior; terminal output is quoted above.

## Additional Notes

- `uv.lock` intentionally not included: the new extra is opt-in and does
not change default resolution.
- Error-body rewriting is confined to already-failing responses and
wrapped so a diagnostic can never turn a clean upstream error into a
proxy 500.
- The preflight raises where the provider is resolved (`registry.py`)
rather than at the CLI, so programmatic embedders get the same
guarantee; the CLI only formats it.
- Three test files fail to collect on this branch
(`test_hermes_passthrough_compression`, `test_observability_metrics`,
`test_observability_tracing`). They predate this work and are unrelated,
so they were excluded rather than fixed — happy to open a separate
issue.
- `[vertex]` was chosen over adding the dependency to `[proxy]` after
measuring the footprint at ~175 MB. If maintainers prefer it in
`[proxy]`, that is a one-line change.
## Runtime Rollout Safety

- Rollout-managed feature(s): none; diagnostics annotate already-failing
Vertex responses and the backend dependency is opt-in through the vertex
extra.
- Minimum rollout channel: normal stable release after Vertex
diagnostics, route, and backend-path tests pass.
- Stable/default behavior changed: successful responses and native
passthrough remain unchanged; supported Vertex failures gain an
actionable header/message, and selecting the LiteLLM Vertex backend
without its SDK now fails once at startup.
- Kill switch / disable path: use native Vertex passthrough without the
LiteLLM backend, or human-revert the diagnostic/preflight change.
- Unsafe override required: none.
- Qualification impact: offline diagnostics, native integration, and
mocked LiteLLM Vertex request-shaping tests must remain green.
- Rollback path: human-revert the PR; no persisted state or migration is
involved.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2640"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `5d7071f348d2462f666fd7ac1ceb8ac18050169a`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Re-reviewed latest integration head c70c1a497 and current author
responses, including fixed public error hints, log escaping, body/header
consistency, all four backend error sites, and native-route versus
LiteLLM SDK dependency behavior. Clean integration with upstream/main
2d3ee51846. No new PR-specific source
blocker found in reviewed scope; no contributor diagnostic/routing logic
rewritten.

232 Vertex diagnostics/native-route/backend/CLI/release/version/MCP
tests pass with11 skips. Offline lock validation passes (268 packages);
all12 changed Python files pass Ruff check/format, and whitespace
passes. These checks cover synthetic error/transport paths and do not
establish live cloud regional availability or billing behavior. Earlier
author live evidence remains scoped to its stated environment; no fresh
cloud request or native build claimed here. Historical shared
cached-read fixture failure was main-reproduced and is not a
contributor-code blocker. Maintainers handle current exact-head hosted
qualification before approval; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `486a716adbf642ae5e43276df4c2dc1e63b901d7`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical sync with upstream main
c01cf07d94. Re-reviewed current
author/Devin discussion and public diagnostic/error boundaries,
native-versus-SDK route separation and startup preflight. This sync does
not alter the PR Vertex diagnostics, backend or route-registry
modules.158 focused diagnostics/native-route/backend/streaming/CLI tests
pass,11live-environment skips,1deprecation warning. Offline uv lock
--check passes;all12 changed Python files pass Ruff lint/format and
main-relative whitespace passes. Fixed hints remain separate from raw
provider error text; multi-worker preflight and non-Google partner
guidance retain resolved regressions. No contributor diagnostic/routing
correctness logic rewritten. Tests use existing current-main DLL and
synthetic errors/transports; no fresh native build, live regional
availability/provider inference or billing qualification claimed. Formal
approval awaits exact-head hosted qualification; maintainers handle CI.
Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `0b75c609f24562c8b6ed2aaeaf58239984b4810c`, upstream main
`ce4275491299231bc9d2907833f72f72ea22b8a9`.

Requalified after human merges #4667,#4669,#4670 against upstream main
ce42754912. Earlier approval and
publication guards stopped stale candidates before either was posted.
Current clean merge retains Vertex diagnostics/native-versus-SDK routing
and main enable_kompress checks, lossless cold-prefix disable, and
Python token-window boundary behavior.190 combined
Vertex/backend/CLI/Kompress-disable/must-keep tests
pass,11live-environment skips,1warning. Offline lock validation,12-file
Ruff lint/format and main-relative whitespace pass. No contributor
diagnostic/routing logic rewritten. Native Rust/Cargo sources are
identical to c01 across these merges; existing matching-core DLL
supports tests, not a fresh native build. No new live Vertex
regional/provider/billing qualification claimed. All prior-main hosted
results are superseded for readiness; approval awaits exact checks on
this published merge head. Maintainers handle CI; no agent merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `1fe62702ce614315679d68d9d126e31c57208c67`, upstream main
`00d4f441d3f1d48c463fea57e6c574c2f3671302`.

Requalified after human merges #4667,#4669,#4670 against upstream main
ce42754912. Earlier approval and
publication guards stopped stale candidates before either was posted.
Current clean merge retains Vertex diagnostics/native-versus-SDK routing
and main enable_kompress checks, lossless cold-prefix disable, and
Python token-window boundary behavior.190 combined
Vertex/backend/CLI/Kompress-disable/must-keep tests
pass,11live-environment skips,1warning. Offline lock validation,12-file
Ruff lint/format and main-relative whitespace pass. No contributor
diagnostic/routing logic rewritten. Native Rust/Cargo sources are
identical to c01 across these merges; existing matching-core DLL
supports tests, not a fresh native build. No new live Vertex
regional/provider/billing qualification claimed. All prior-main hosted
results are superseded for readiness; approval awaits exact checks on
this published merge head. Maintainers handle CI; no agent merge.

Final current-main requalification:
07d5add5ba, including human merges #4671
action pinning and #4648 multimodal memory preservation. Clean
mechanical merge; Vertex/backend/CLI inputs retain the preceding
190-test qualification. Fresh workflow/release suite: 62 passed. Fresh
memory wrapper suite: 9 passed. New wrapper retains list payload parts
and extracts textual search inputs; no contributor Vertex logic
rewritten. Existing native input provenance remains unchanged. Hosted
qualification must run on this final head before approval.

Newer human merge #4646 advanced authoritative main to
8cdce172ea before publication. Cleanly
integrated quote-aware read detection; reviewed new masking/segmentation
overlap. Fresh read-command/content-router suite: 188 passed, 1 optional
SOCKS warning. Vertex code remains unchanged by this merge.

Final additional human merge #4067 packaging changes advanced main to
a8905214d: cleanly integrated explicit torch-extras split, retaining
Vertex optional dependencies. Fresh ONNX/optional-dependency contracts:
12 passed. Offline lock check and whitespace pass. No live clean-install
or Torch backend claim. Earlier publication candidates were stopped
before push by fresh-main guard.

Latest human merge #4042 advances authoritative main to
cde584d7fe. Clean merge preserves optional
LangChain availability handling and CI dependency setup. Fresh
binding/compatibility/startup-log/xAI regression suite: 29 passed. New
main integration re-reviewed; no contributor Vertex behavior rewritten.

Subsequent #4049 main merge 00d4f441d adds graceful context lookup/init
failure handling. Cleanly integrated with Vertex handler diagnostics
intact. Fresh Anthropic pre-upstream/memory native ops/OpenAI Codex
routing/WebSocket lifecycle suite: 192 passed, 1 deprecation warning.
Reviewed exception handling and transport overlap; no live provider
qualification claim.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `6c63811d6de7491ebce9a16f89e197f5f5dd86dd`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Requalified against authoritative
main2b0315b2d4b0d54947050bbdad2de9e89df06378 after further human merges.
All10 integration patches exactly match00d4f441-to-current-main changes
after diff-location normalization; contributor Vertex
diagnostics/routing policy unchanged.539combined
Vertex/backend/CLI/Kompress-disable/must-keep/binary/doctor/Windows-supervisor/provider/search/text
tests pass;13platform/live-environment skips,2warnings. Nine changed
Python files pass Ruff lint/format; offline lock and whitespace pass.
Existing pr3982 native DLL verified against every tracked native
build-input hash and artifact checksum; no fresh native build claimed.
Current author/Devin discussion checked; prior fixes remain intact. No
live Vertex regional/provider/billing,IPv6 listener or unsupported cloud
qualification claimed. No new finding in reviewed scope. Exact-head
hosted checks must qualify this published head before formal approval;
maintainers handle them. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-06 12:27:47 -06:00

330 lines
14 KiB
Python

"""Vertex onboarding diagnostics: hints a user sees when Vertex rejects a request.
Every hint must be fixed text (public_errors contract): request data and raw
exception text may *select* a hint but must never reach the client.
"""
import json
import logging
from unittest.mock import patch
import pytest
from headroom.providers.registry import BackendUnavailableError, create_proxy_backend
from headroom.providers.vertex import (
HINT_HEADER,
annotate_vertex_error,
public_backend_error_message,
vertex_error_hint,
)
from headroom.proxy.public_errors import INTERNAL_ERROR, public_message
_HOSTILE = "m\r\nX-Injected: 1<script>caf\u00e9-secret-proj-123"
_SECRET_RAW = (
"Your default credentials were not found. project=secret-proj-123 "
"GOOGLE_APPLICATION_CREDENTIALS=/etc/keys/sa.json host=10.1.2.3"
)
_LEAKS = ("secret-proj-123", "/etc/keys/sa.json", "10.1.2.3")
class _FakeResponse:
"""Minimal stand-in for a Starlette Response."""
def __init__(self, status_code: int, body: bytes | None = None):
self.status_code = status_code
self.headers: dict[str, str] = {}
if body is not None:
self.body = body
def _error_body(status: int, message: str | None = "upstream said no") -> bytes:
error: dict = {"code": status}
if message is not None:
error["message"] = message
return json.dumps({"error": error}).encode()
class TestVertexErrorHint:
@pytest.mark.parametrize("status", [200, 400, 500, 503])
def test_no_hint_for_unexplainable_statuses(self, status):
"""Inventing a hint for an unknown status would mislead, not help."""
assert vertex_error_hint(status, publisher="google") is None
def test_401_points_at_adc_not_user_token(self):
hint = vertex_error_hint(401)
assert "gcloud auth application-default login" in hint
assert "gcloud auth print-access-token` user token is NOT accepted" in hint
def test_429_names_quota(self):
assert vertex_error_hint(429, publisher="anthropic").startswith("Quota exhausted")
@pytest.mark.parametrize("status", [403, 404])
def test_partner_vs_gemini_remedies(self, status):
claude = vertex_error_hint(status, publisher="anthropic")
gemini = vertex_error_hint(status, publisher="google")
mistral = vertex_error_hint(status, publisher="mistralai")
assert "Model Garden" in claude and "Claude 4.7+" in claude
assert "no US regional endpoint" in gemini and "Model Garden" not in gemini
assert "Model Garden" in mistral and "Gemini" not in mistral and "Claude 4" not in mistral
@pytest.mark.parametrize("status", [401, 403, 404, 429])
@pytest.mark.parametrize("publisher", ["google", "anthropic", "", _HOSTILE])
def test_every_hint_is_a_legal_header_value(self, status, publisher):
"""h11 rejects control / non-latin-1 bytes at send time, after we return."""
hint = vertex_error_hint(status, publisher=publisher)
hint.encode("latin-1")
assert all(" " <= ch <= "~" for ch in hint)
class TestAnnotateVertexError:
def test_success_untouched(self):
body = json.dumps({"candidates": []}).encode()
resp = annotate_vertex_error(_FakeResponse(200, body))
assert resp.body == body and HINT_HEADER not in resp.headers
def test_hint_in_header_and_error_message_with_consistent_length(self):
resp = annotate_vertex_error(_FakeResponse(404, _error_body(404)), publisher="google")
message = json.loads(resp.body)["error"]["message"]
assert message == f"upstream said no\n[headroom] hint: {resp.headers[HINT_HEADER]}"
assert resp.headers["content-length"] == str(len(resp.body))
def test_annotation_is_idempotent(self):
resp = annotate_vertex_error(_FakeResponse(401, _error_body(401)))
once = resp.body
assert annotate_vertex_error(resp).body == once
def test_error_without_message_gets_namespaced_key(self):
resp = annotate_vertex_error(_FakeResponse(404, _error_body(404, message=None)))
assert json.loads(resp.body)["headroom_hint"] == resp.headers[HINT_HEADER]
@pytest.mark.parametrize("body", [b"<html>nope</html>", b"[1, 2]", None])
def test_unrewritable_bodies_survive_with_header(self, body):
"""Non-JSON, non-object JSON and streaming (no body) keep the header only."""
resp = annotate_vertex_error(_FakeResponse(403, body), publisher="anthropic")
assert getattr(resp, "body", None) == body
assert HINT_HEADER in resp.headers
def test_hostile_path_params_reach_log_escaped_but_never_client(self, caplog):
with caplog.at_level(logging.WARNING):
resp = annotate_vertex_error(
_FakeResponse(404, _error_body(404)),
location=_HOSTILE,
publisher=_HOSTILE,
model=_HOSTILE,
)
client_visible = resp.headers[HINT_HEADER] + resp.body.decode()
assert "X-Injected" not in client_visible and "secret-proj" not in client_visible
# Operators keep the context, with CR/LF escaped by %r.
assert "secret-proj-123" in caplog.text and "\r\n" not in caplog.text
class TestNativeRoutesWiring:
"""The decorator must actually be applied on the real FastAPI routes."""
@pytest.fixture
def client(self, monkeypatch):
pytest.importorskip("fastapi")
from fastapi.responses import JSONResponse
from fastapi.testclient import TestClient
from headroom.proxy.server import HeadroomProxy, ProxyConfig, create_app
async def upstream_404(self, request, *args):
return JSONResponse({"error": {"code": 404, "message": "not found"}}, 404)
for name in ("handle_gemini_generate_content", "handle_passthrough"):
monkeypatch.setattr(HeadroomProxy, name, upstream_404)
app = create_app(ProxyConfig(optimize=False, cache_enabled=False, rate_limit_enabled=False))
with TestClient(app) as client:
yield client
@pytest.mark.parametrize(
"publisher,action,expect",
[
("google", "generateContent", "no US regional endpoint"),
("anthropic", "generateContent", "Model Garden"),
],
)
def test_upstream_404_carries_hint(self, client, publisher, action, expect):
resp = client.post(
f"/v1/projects/p/locations/us-central1/publishers/{publisher}/models/m:{action}"
)
assert resp.status_code == 404
assert expect in resp.headers[HINT_HEADER]
assert "[headroom] hint:" in resp.json()["error"]["message"]
def test_encoded_crlf_in_path_is_not_reflected(self, client):
"""%0d%0a decodes into the `model` path param; the response must stay sane."""
resp = client.post(
"/v1/projects/p/locations/us-central1/publishers/google/models/"
"m%0d%0aX-Injected:%201:generateContent"
)
assert resp.status_code == 404
assert "no US regional endpoint" in resp.headers[HINT_HEADER]
assert "x-injected" not in resp.headers
assert "X-Injected" not in resp.text
class TestPublicBackendErrorMessage:
def test_internal_error_gets_fixed_sentence_plus_hint_only(self):
out = public_backend_error_message(RuntimeError(_SECRET_RAW), _SECRET_RAW)
prefix = f"{public_message(INTERNAL_ERROR)}\n[headroom] hint: "
assert out.startswith(prefix)
assert "gcloud auth application-default login" in out
assert not any(leak in out for leak in _LEAKS)
def test_missing_sdk_hint(self):
raw = "vertexai import failed. Got error: No module named 'vertexai' at /opt/venv"
out = public_backend_error_message(ImportError(raw), raw)
assert "headroom-ai[proxy,vertex]" in out
assert "do NOT need `--backend vertex`" in out
assert "/opt/venv" not in out
def test_unrecognized_error_is_plain_public_message(self):
raw = "upstream timed out talking to 10.9.9.9"
assert public_backend_error_message(RuntimeError(raw), raw) == public_message(
INTERNAL_ERROR
)
def test_chained_proxy_hint_is_not_repeated(self):
"""A provider HTTP error passes its text through; it may already hold our hint."""
httpx = pytest.importorskip("httpx")
openai = pytest.importorskip("openai")
raw = (
"vertexai import failed: No module named 'vertexai'\n"
"[headroom] hint: install headroom-ai[proxy,vertex]"
)
resp = httpx.Response(500, request=httpx.Request("POST", "http://x"))
exc = openai.InternalServerError(raw, response=resp, body=None)
assert public_backend_error_message(exc, raw).count("[headroom] hint:") == 1
class TestLiteLLMVertexBackendErrors:
"""Through the real backend: raw text stays in the log, hint reaches client."""
@pytest.fixture
def litellm_mod(self, monkeypatch):
mod = pytest.importorskip("headroom.backends.litellm")
if not mod.LITELLM_AVAILABLE:
pytest.skip("litellm not installed")
async def boom(**_kw):
raise RuntimeError(_SECRET_RAW)
monkeypatch.setattr(mod, "acompletion", boom)
return mod
_BODY = {"model": "claude-sonnet-4-6", "messages": [{"role": "user", "content": "hi"}]}
_OPENAI_BODY = {"model": "gemini-3.5-flash", "messages": [{"role": "user", "content": "hi"}]}
@pytest.mark.asyncio
async def test_send_message(self, litellm_mod, caplog):
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
with caplog.at_level(logging.ERROR):
resp = await backend.send_message(dict(self._BODY), {})
message = resp.body["error"]["message"]
assert "[headroom] hint:" in message
assert not any(leak in message for leak in _LEAKS)
assert "secret-proj-123" in caplog.text
@pytest.mark.asyncio
async def test_send_openai_message(self, litellm_mod):
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
resp = await backend.send_openai_message(dict(self._OPENAI_BODY), {})
message = resp.body["error"]["message"]
assert "[headroom] hint:" in message
assert not any(leak in message for leak in _LEAKS)
@pytest.mark.asyncio
async def test_stream_message_error_event(self, litellm_mod):
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
events = [e async for e in backend.stream_message(dict(self._BODY, stream=True), {})]
errors = [e.data["error"]["message"] for e in events if e.event_type == "error"]
assert len(errors) == 1
assert "[headroom] hint:" in errors[0]
assert not any(leak in errors[0] for leak in _LEAKS)
@pytest.mark.asyncio
async def test_stream_openai_message_error_chunk(self, litellm_mod):
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
chunks = [c async for c in backend.stream_openai_message(dict(self._OPENAI_BODY), {})]
assert chunks[-1] == "data: [DONE]\n\n"
message = json.loads(chunks[-2].removeprefix("data: "))["error"]["message"]
assert "[headroom] hint:" in message
assert not any(leak in message for leak in _LEAKS)
@pytest.mark.asyncio
async def test_non_vertex_backend_gets_no_vertex_hint(self, litellm_mod):
backend = litellm_mod.LiteLLMBackend(provider="openrouter")
resp = await backend.send_message(dict(self._BODY), {})
assert "[headroom] hint:" not in resp.body["error"]["message"]
class TestVertexSdkPreflight:
"""`--backend vertex` without the SDK must fail at startup, not per-request."""
@pytest.fixture
def no_sdk(self, monkeypatch):
monkeypatch.setattr(
"headroom.providers.vertex.diagnostics.importlib.util.find_spec",
lambda name: None if name == "vertexai" else object(),
)
def _create(self, backend, **kwargs):
return create_proxy_backend(
backend=backend,
anyllm_provider="",
bedrock_region=None,
logger=logging.getLogger("test"),
**kwargs,
)
@pytest.mark.parametrize(
"backend", ["vertex", "vertex_ai", "litellm-vertex", "google-vertex", "googlevertex"]
)
def test_every_vertex_alias_is_preflighted(self, no_sdk, backend):
with pytest.raises(BackendUnavailableError) as excinfo:
self._create(backend)
assert "headroom-ai[proxy,vertex]" in str(excinfo.value)
def test_other_providers_are_unaffected(self, no_sdk):
"""The preflight must not become a general-purpose backend gate."""
self._create("litellm-openrouter")
def test_injected_backend_class_bypasses_preflight(self, no_sdk):
sentinel = object()
assert self._create("vertex", litellm_backend_cls=lambda **_: sentinel) is sentinel
def test_present_sdk_passes(self, monkeypatch):
monkeypatch.setattr(
"headroom.providers.vertex.diagnostics.importlib.util.find_spec",
lambda name: object(),
)
assert self._create("vertex") is not None
def test_cli_prints_remedy_and_exits_2(self):
click_testing = pytest.importorskip("click.testing")
from headroom.cli.main import main
def unavailable(config, **kwargs):
raise BackendUnavailableError("Vertex SDK is missing. remedy-text")
with patch("headroom.proxy.server.run_server", unavailable):
result = click_testing.CliRunner().invoke(main, ["proxy"])
assert result.exit_code == 2
assert "Cannot start proxy: Vertex SDK is missing. remedy-text" in result.output
assert "Traceback" not in result.output
def test_cli_multi_worker_fails_in_parent_before_serving(self, no_sdk):
"""--workers >1 builds the app inside each worker, past the CLI's handler."""
click_testing = pytest.importorskip("click.testing")
from headroom.cli.main import main
with patch("headroom.proxy.server.run_server") as run_server:
result = click_testing.CliRunner().invoke(
main, ["proxy", "--backend", "vertex", "--workers", "2"]
)
assert result.exit_code == 2
assert "headroom-ai[proxy,vertex]" in result.output
run_server.assert_not_called()