## Description Vertex rejects onboarding requests for a handful of recurring reasons — expired ADC, a partner model never enabled in Model Garden, a model not served in the requested location, a missing optional SDK — and its own error text names none of the fixes. Users see a bare `404` or an opaque `500` through their SDK and cannot tell a proxy bug from a project-config gap. This makes the proxy explain those failures where it happens, adds a `[vertex]` extra so the LiteLLM Vertex backend can actually start, and corrects an example, a test matrix, and three documentation claims that could not have worked as written. Closes #2632 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Explain Vertex failures in-band** (`headroom/providers/vertex/diagnostics.py`, new) - On `401/403/404/429`, attach an actionable hint three ways: a WARNING in the proxy log, an `x-headroom-hint` response header, and a note appended to `error.message`. The last is load-bearing — SDKs raise with the message string and discard headers. - Successful responses are never modified; annotation is idempotent; statuses we cannot explain get no hint rather than a plausible-sounding wrong one. - Routes opt in via a one-line `@with_vertex_diagnostics` decorator reading path params FastAPI already injects (**+8 lines** in `proxy_routes.py`, no re-indentation). - Same treatment for backend setup failures, where `--backend vertex` surfaced LiteLLM's raw `No module named 'vertexai'` as an opaque 500. **Make `--backend vertex` able to start** (`pyproject.toml`, `registry.py`, `cli/proxy.py`) - New `[vertex]` extra carrying `google-cloud-aiplatform>=1.38`. Deliberately **not** folded into `[proxy]`: measured at ~175 MB installed, and the native passthrough routes never needed it. Follows the existing `proxy-prod` split. - Startup preflight keyed on the **resolved provider**, so every alias normalized at `registry.py:229` (`vertex`, `vertex_ai`, `litellm-vertex`, `google-vertex`, `googlevertex`) is covered. Skipped when a backend class is injected, so existing fake-based tests are unaffected. - New `BackendUnavailableError` so the CLI prints the remedy and exits `2` instead of dumping a traceback through startup internals. **Correct things that could not have worked** | | was | is | |---|---|---| | SDK client | `http_options={"api_endpoint": ...}` | `{"base_url": "http://..."}` — `api_endpoint` is not a field on `HttpOptions`; pydantic rejects it before any request is sent | | Thinking config | `ThinkingConfig(thinking_budget_tokens=)` | `thinking_budget=` | | Request payload | `contents: [{parts: [...]}]` | adds `role: "user"` — the `global` endpoint enforces it, regional was silently lenient | | Auth guidance | `gcloud auth print-access-token` | `application-default` — the plain user token gets `401 ACCESS_TOKEN_TYPE_UNSUPPORTED` | | Test matrix | model × location cross-product | explicit servable pairs | - Test matrix is now four pairs that can actually serve: `global`/`gemini-flash-latest`, `global`/`claude-sonnet-4-6`, `europe-west2`/`gemini-3.5-flash`, `us-east5`/`claude-sonnet-4-6`. Gemini 3.x has no US regional endpoint and `-latest` aliases are global-only, so a pinned `us-central1` Gemini case can never pass. No Gemini 2.5 — it retires 2026-10-16. - `test_thinking_levels` asserted only `status_code < 500` while sending an ignored `thinkingBudgetTokens` key: it passed on 400s and proved nothing. Now asserts 200 plus scored usage metadata with the correct `thinkingBudget` spelling. - Cases the caller's project genuinely cannot serve now **skip** with an actionable reason instead of reporting a red proxy. **Docs** (`proxy.mdx`, `installation.mdx`, `wiki/vertex.md`) - `proxy.mdx` claimed `publisher=anthropic` passthrough "routes through the same LiteLLM-Vertex path as `--backend vertex_ai`". It does not — without the flag it is a direct forward, which is why it works on a bare `[proxy]` install where the LiteLLM path cannot even import. - Documented the trap of combining `--backend vertex` with native Vertex URLs: LiteLLM takes project/region/model from its own config and ignores the URL. - `installation.mdx` and `wiki/vertex.md` said the SDK is "not included in any extra". - Cleared 15 pre-existing `ruff` findings in the two files this PR rewrites (4 remain in `plugins/headroom-oauth2`, untouched by this work). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ mypy headroom Success: no issues found in 511 source files $ ruff check . Found 4 errors. # all in plugins/headroom-oauth2, pre-existing, untouched by this PR $ pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q 44 passed, 1 warning in 69.86s # 8 live integration + 36 offline diagnostics, 0 skipped $ pytest tests/ -k "anthropic_messages or handler_anthropic or backend or registry or cli_proxy or vertex or proxy_routes" 853 passed, 49 skipped, 9345 deselected, 2 warnings in 28.41s ``` New tests in `tests/test_vertex_diagnostics.py` (36, no credentials required): status coverage, per-publisher remedy divergence, idempotency, `content-length` consistency after body rewrite, non-JSON and body-less/streaming responses, every backend alias preflighted, other providers unaffected, injected backend bypasses the preflight. ## Real Behavior Proof - Environment: macOS 14, Python 3.11.15, live GCP project (`alanblount-sandbox`) with `aiplatform.googleapis.com` enabled and Claude enabled in Model Garden; ADC via `gcloud auth application-default login` - Exact command / steps: `export GCP_PROJECT_ID=alanblount-sandbox && export GCP_ACCESS_TOKEN=$(gcloud auth application-default print-access-token) && uv run --extra proxy --with google-genai python3 examples/vertex_genai_sdk_demo.py && uv run --extra proxy --with pytest-asyncio pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q && headroom proxy --backend vertex` (exits 2, then retried with `[proxy,vertex]` extra installed) - Observed result: demo printed `✓ Standard response received successfully! > 1, 2, 3, 4, 5` and `✓ Thinking response received successfully!`; 44 tests passed (8 live integration, 36 diagnostics), 0 skipped; `--backend vertex` exited 2 with the remedy message; same request with the extra installed returned 200; invalid region produced `[headroom] hint:` in the response body and `x-headroom-hint` header; success responses were byte-clean - Not tested: Windows; `us`/`eu` multi-region endpoints and Claude 4.7+ (project cannot serve them); Provisioned Throughput; Rust proxy path; `--backend vertex` dropping URL project/model on native Vertex routes (documented as known gap in Additional Notes) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy and CLI behavior; terminal output is quoted above. ## Additional Notes - `uv.lock` intentionally not included: the new extra is opt-in and does not change default resolution. - Error-body rewriting is confined to already-failing responses and wrapped so a diagnostic can never turn a clean upstream error into a proxy 500. - The preflight raises where the provider is resolved (`registry.py`) rather than at the CLI, so programmatic embedders get the same guarantee; the CLI only formats it. - Three test files fail to collect on this branch (`test_hermes_passthrough_compression`, `test_observability_metrics`, `test_observability_tracing`). They predate this work and are unrelated, so they were excluded rather than fixed — happy to open a separate issue. - `[vertex]` was chosen over adding the dependency to `[proxy]` after measuring the footprint at ~175 MB. If maintainers prefer it in `[proxy]`, that is a one-line change. ## Runtime Rollout Safety - Rollout-managed feature(s): none; diagnostics annotate already-failing Vertex responses and the backend dependency is opt-in through the vertex extra. - Minimum rollout channel: normal stable release after Vertex diagnostics, route, and backend-path tests pass. - Stable/default behavior changed: successful responses and native passthrough remain unchanged; supported Vertex failures gain an actionable header/message, and selecting the LiteLLM Vertex backend without its SDK now fails once at startup. - Kill switch / disable path: use native Vertex passthrough without the LiteLLM backend, or human-revert the diagnostic/preflight change. - Unsafe override required: none. - Qualification impact: offline diagnostics, native integration, and mocked LiteLLM Vertex request-shaping tests must remain green. - Rollback path: human-revert the PR; no persisted state or migration is involved. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2640" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `5d7071f348d2462f666fd7ac1ceb8ac18050169a`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration headc70c1a497and current author responses, including fixed public error hints, log escaping, body/header consistency, all four backend error sites, and native-route versus LiteLLM SDK dependency behavior. Clean integration with upstream/main2d3ee51846. No new PR-specific source blocker found in reviewed scope; no contributor diagnostic/routing logic rewritten. 232 Vertex diagnostics/native-route/backend/CLI/release/version/MCP tests pass with11 skips. Offline lock validation passes (268 packages); all12 changed Python files pass Ruff check/format, and whitespace passes. These checks cover synthetic error/transport paths and do not establish live cloud regional availability or billing behavior. Earlier author live evidence remains scoped to its stated environment; no fresh cloud request or native build claimed here. Historical shared cached-read fixture failure was main-reproduced and is not a contributor-code blocker. Maintainers handle current exact-head hosted qualification before approval; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `486a716adbf642ae5e43276df4c2dc1e63b901d7`, upstream main `c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`. Clean mechanical sync with upstream mainc01cf07d94. Re-reviewed current author/Devin discussion and public diagnostic/error boundaries, native-versus-SDK route separation and startup preflight. This sync does not alter the PR Vertex diagnostics, backend or route-registry modules.158 focused diagnostics/native-route/backend/streaming/CLI tests pass,11live-environment skips,1deprecation warning. Offline uv lock --check passes;all12 changed Python files pass Ruff lint/format and main-relative whitespace passes. Fixed hints remain separate from raw provider error text; multi-worker preflight and non-Google partner guidance retain resolved regressions. No contributor diagnostic/routing correctness logic rewritten. Tests use existing current-main DLL and synthetic errors/transports; no fresh native build, live regional availability/provider inference or billing qualification claimed. Formal approval awaits exact-head hosted qualification; maintainers handle CI. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `0b75c609f24562c8b6ed2aaeaf58239984b4810c`, upstream main `ce4275491299231bc9d2907833f72f72ea22b8a9`. Requalified after human merges #4667,#4669,#4670 against upstream maince42754912. Earlier approval and publication guards stopped stale candidates before either was posted. Current clean merge retains Vertex diagnostics/native-versus-SDK routing and main enable_kompress checks, lossless cold-prefix disable, and Python token-window boundary behavior.190 combined Vertex/backend/CLI/Kompress-disable/must-keep tests pass,11live-environment skips,1warning. Offline lock validation,12-file Ruff lint/format and main-relative whitespace pass. No contributor diagnostic/routing logic rewritten. Native Rust/Cargo sources are identical to c01 across these merges; existing matching-core DLL supports tests, not a fresh native build. No new live Vertex regional/provider/billing qualification claimed. All prior-main hosted results are superseded for readiness; approval awaits exact checks on this published merge head. Maintainers handle CI; no agent merge. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `1fe62702ce614315679d68d9d126e31c57208c67`, upstream main `00d4f441d3f1d48c463fea57e6c574c2f3671302`. Requalified after human merges #4667,#4669,#4670 against upstream maince42754912. Earlier approval and publication guards stopped stale candidates before either was posted. Current clean merge retains Vertex diagnostics/native-versus-SDK routing and main enable_kompress checks, lossless cold-prefix disable, and Python token-window boundary behavior.190 combined Vertex/backend/CLI/Kompress-disable/must-keep tests pass,11live-environment skips,1warning. Offline lock validation,12-file Ruff lint/format and main-relative whitespace pass. No contributor diagnostic/routing logic rewritten. Native Rust/Cargo sources are identical to c01 across these merges; existing matching-core DLL supports tests, not a fresh native build. No new live Vertex regional/provider/billing qualification claimed. All prior-main hosted results are superseded for readiness; approval awaits exact checks on this published merge head. Maintainers handle CI; no agent merge. Final current-main requalification:07d5add5ba, including human merges #4671 action pinning and #4648 multimodal memory preservation. Clean mechanical merge; Vertex/backend/CLI inputs retain the preceding 190-test qualification. Fresh workflow/release suite: 62 passed. Fresh memory wrapper suite: 9 passed. New wrapper retains list payload parts and extracts textual search inputs; no contributor Vertex logic rewritten. Existing native input provenance remains unchanged. Hosted qualification must run on this final head before approval. Newer human merge #4646 advanced authoritative main to8cdce172eabefore publication. Cleanly integrated quote-aware read detection; reviewed new masking/segmentation overlap. Fresh read-command/content-router suite: 188 passed, 1 optional SOCKS warning. Vertex code remains unchanged by this merge. Final additional human merge #4067 packaging changes advanced main toa8905214d: cleanly integrated explicit torch-extras split, retaining Vertex optional dependencies. Fresh ONNX/optional-dependency contracts: 12 passed. Offline lock check and whitespace pass. No live clean-install or Torch backend claim. Earlier publication candidates were stopped before push by fresh-main guard. Latest human merge #4042 advances authoritative main tocde584d7fe. Clean merge preserves optional LangChain availability handling and CI dependency setup. Fresh binding/compatibility/startup-log/xAI regression suite: 29 passed. New main integration re-reviewed; no contributor Vertex behavior rewritten. Subsequent #4049 main merge00d4f441dadds graceful context lookup/init failure handling. Cleanly integrated with Vertex handler diagnostics intact. Fresh Anthropic pre-upstream/memory native ops/OpenAI Codex routing/WebSocket lifecycle suite: 192 passed, 1 deprecation warning. Reviewed exception handling and transport overlap; no live provider qualification claim. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `6c63811d6de7491ebce9a16f89e197f5f5dd86dd`, upstream main `2b0315b2d4b0d54947050bbdad2de9e89df06378`. Requalified against authoritative main2b0315b2d4b0d54947050bbdad2de9e89df06378 after further human merges. All10 integration patches exactly match00d4f441-to-current-main changes after diff-location normalization; contributor Vertex diagnostics/routing policy unchanged.539combined Vertex/backend/CLI/Kompress-disable/must-keep/binary/doctor/Windows-supervisor/provider/search/text tests pass;13platform/live-environment skips,2warnings. Nine changed Python files pass Ruff lint/format; offline lock and whitespace pass. Existing pr3982 native DLL verified against every tracked native build-input hash and artifact checksum; no fresh native build claimed. Current author/Devin discussion checked; prior fixes remain intact. No live Vertex regional/provider/billing,IPv6 listener or unsupported cloud qualification claimed. No new finding in reviewed scope. Exact-head hosted checks must qualify this published head before formal approval; maintainers handle them. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
6.2 KiB
Vertex AI
Headroom supports Google Cloud Vertex AI publisher endpoints through the proxy passthrough surface. Configure the proxy with a regional Vertex base URL, then send normal Vertex REST requests through Headroom.
Google documents Gemini generation on Vertex with generateContent and
streamGenerateContent, and the request body uses the Vertex/Gemini contents
shape. See Google Cloud's model inference reference:
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference
Google Cloud REST calls authenticate with a bearer access token. Use
gcloud auth application-default print-access-token: a plain
gcloud auth print-access-token user token is rejected by
aiplatform.googleapis.com with 401 ACCESS_TOKEN_TYPE_UNSUPPORTED for many
identities. Application Default Credentials search
GOOGLE_APPLICATION_CREDENTIALS, local ADC files, and attached service accounts
in that order. Tokens expire after ~1 hour. See:
- https://docs.cloud.google.com/docs/authentication/rest
- https://docs.cloud.google.com/docs/authentication/application-default-credentials
Configure
Set the Vertex regional host explicitly:
headroom proxy --vertex-api-url https://us-central1-aiplatform.googleapis.com
The same setting is available through VERTEX_TARGET_API_URL. Left unset, the
proxy derives the host from each request's locations/{location} segment, so
one proxy serves every region plus global.
Picking a location
Vertex serves each publisher model in only a subset of locations, and this is
the most common source of a confusing 404 during onboarding:
| Model | Where it serves |
|---|---|
gemini-flash-latest, gemini-flash-lite-latest |
global only |
gemini-3.5-flash |
global, europe-west2, asia-northeast1, asia-south1, asia-southeast1 |
claude-sonnet-4-6 (and 4.6-or-older Claude) |
global, us-east5, europe-west1, asia-southeast1 |
| Claude 4.7+ | global or the us/eu multi-region only -- no named regions |
Gemini 3.x has no US regional endpoint. Start with global and only pin a
region when you need Provisioned Throughput or data residency. Partner models
(Claude, Llama, Mistral) also need a one-time per-project enable in Model
Garden before they serve.
When Vertex rejects a request with 401, 403, 404 or 429, Headroom appends a [headroom] hint: ... note to
error.message (also emitted as an x-headroom-hint header and a proxy WARNING)
naming the likely fix. Hints are fixed text and never echo the request path,
project, credentials or exception detail; raw provider errors stay in the proxy
log.
Gemini On Vertex
Send Vertex publisher paths through the proxy unchanged:
LOCATION="global"
MODEL="gemini-flash-latest"
ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
curl -sS \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Content-Type: application/json" \
http://127.0.0.1:8787/v1/projects/PROJECT_ID/locations/${LOCATION}/publishers/google/models/${MODEL}:generateContent \
-d '{
"contents": [
{
"role": "user",
"parts": [{"text": "Summarize this repository in one paragraph."}]
}
]
}'
Google Gen AI SDK (Proxy)
Point the google-genai SDK at Headroom with
genai.Client(vertexai=True, project=..., location="global", http_options={"base_url": "http://127.0.0.1:8787"}).
examples/vertex_genai_sdk_demo.py runs this end to end: it starts the proxy,
probes it (including a thinking config) and tears it down.
Note the google-genai SDK only builds publishers/google/... paths, so it
cannot reach Claude on Vertex; use the publishers/anthropic/...:rawPredict
route below for that.
Supported passthrough actions:
generateContentstreamGenerateContentcountTokens
Anthropic Publisher On Vertex
Headroom also forwards Anthropic publisher calls on Vertex:
rawPredictstreamRawPredict
The Python proxy preserves caller-supplied Google bearer auth. The native Rust proxy path additionally resolves GCP ADC and injects the bearer token for the Anthropic publisher route.
These native routes are a straight passthrough: they need no --backend
flag and no extra beyond [proxy].
--backend vertex (aliases: vertex_ai, litellm-vertex, google-vertex) is a
different mode. It routes Anthropic Messages traffic — /v1/messages — through
LiteLLM, and needs the Vertex SDK:
pip install "headroom-ai[proxy,vertex]"
That extra is kept out of [proxy] because google-cloud-aiplatform and its
transitive tree add roughly 175 MB, and only this backend uses it. Select the
backend without it and the proxy refuses to start rather than failing on every
request:
Cannot start proxy: Vertex backend selected but the Vertex SDK is missing. ...
Do not combine the two. With --backend vertex set, Claude (publishers/anthropic) requests
are re-routed through LiteLLM, which takes the project, region and model from
its own configuration (VERTEXAI_PROJECT / VERTEXAI_LOCATION) and ignores the
ones in your URL — so a request that works without the flag can come back 404
for a model you never named.
Claude Code with Headroom compression (validated)
To run Claude Code against Claude-on-Vertex with Headroom compressing the context, use the dedicated, tested runbook:
➡️ Claude Code + Vertex + Headroom
Short version: run Claude Code in normal Anthropic mode (ANTHROPIC_BASE_URL
→ the proxy) and start the proxy with --backend litellm-vertex_ai --region <loc> --code-aware; Headroom holds the GCP ADC creds and calls Vertex.
⚠️ Do not put Claude Code into Vertex mode and point
ANTHROPIC_VERTEX_BASE_URLat the proxy. Claude Code's client-side model probe rejects any non-Google Vertex URL before sending a request ("model … not available on your vertex deployment"), so the proxy is never reached. Use the Anthropic-mode runbook above instead.⚠️ Two easy-to-miss requirements:
pip install "headroom-ai[proxy,vertex]"(the LiteLLMvertex_aiprovider needs the Vertex SDK) and the--code-awareflag (code compression is off by default). Without the first the proxy refuses to start; without the second you gettokens_saved: 0.