fix(vertex): explain Vertex failures in-band, correct the model/region matrix (#2640)

## Description

Vertex rejects onboarding requests for a handful of recurring reasons —
expired ADC, a partner model never enabled in Model Garden, a model not
served in the requested location, a missing optional SDK — and its own
error text names none of the fixes. Users see a bare `404` or an opaque
`500` through their SDK and cannot tell a proxy bug from a
project-config gap.

This makes the proxy explain those failures where it happens, adds a
`[vertex]` extra so the LiteLLM Vertex backend can actually start, and
corrects an example, a test matrix, and three documentation claims that
could not have worked as written.

Closes #2632

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

**Explain Vertex failures in-band**
(`headroom/providers/vertex/diagnostics.py`, new)

- On `401/403/404/429`, attach an actionable hint three ways: a WARNING
in the proxy log, an `x-headroom-hint` response header, and a note
appended to `error.message`. The last is load-bearing — SDKs raise with
the message string and discard headers.
- Successful responses are never modified; annotation is idempotent;
statuses we cannot explain get no hint rather than a plausible-sounding
wrong one.
- Routes opt in via a one-line `@with_vertex_diagnostics` decorator
reading path params FastAPI already injects (**+8 lines** in
`proxy_routes.py`, no re-indentation).
- Same treatment for backend setup failures, where `--backend vertex`
surfaced LiteLLM's raw `No module named 'vertexai'` as an opaque 500.

**Make `--backend vertex` able to start** (`pyproject.toml`,
`registry.py`, `cli/proxy.py`)

- New `[vertex]` extra carrying `google-cloud-aiplatform>=1.38`.
Deliberately **not** folded into `[proxy]`: measured at ~175 MB
installed, and the native passthrough routes never needed it. Follows
the existing `proxy-prod` split.
- Startup preflight keyed on the **resolved provider**, so every alias
normalized at `registry.py:229` (`vertex`, `vertex_ai`,
`litellm-vertex`, `google-vertex`, `googlevertex`) is covered. Skipped
when a backend class is injected, so existing fake-based tests are
unaffected.
- New `BackendUnavailableError` so the CLI prints the remedy and exits
`2` instead of dumping a traceback through startup internals.

**Correct things that could not have worked**

| | was | is |
|---|---|---|
| SDK client | `http_options={"api_endpoint": ...}` | `{"base_url":
"http://..."}` — `api_endpoint` is not a field on `HttpOptions`;
pydantic rejects it before any request is sent |
| Thinking config | `ThinkingConfig(thinking_budget_tokens=)` |
`thinking_budget=` |
| Request payload | `contents: [{parts: [...]}]` | adds `role: "user"` —
the `global` endpoint enforces it, regional was silently lenient |
| Auth guidance | `gcloud auth print-access-token` |
`application-default` — the plain user token gets `401
ACCESS_TOKEN_TYPE_UNSUPPORTED` |
| Test matrix | model × location cross-product | explicit servable pairs
|

- Test matrix is now four pairs that can actually serve:
`global`/`gemini-flash-latest`, `global`/`claude-sonnet-4-6`,
`europe-west2`/`gemini-3.5-flash`, `us-east5`/`claude-sonnet-4-6`.
Gemini 3.x has no US regional endpoint and `-latest` aliases are
global-only, so a pinned `us-central1` Gemini case can never pass. No
Gemini 2.5 — it retires 2026-10-16.
- `test_thinking_levels` asserted only `status_code < 500` while sending
an ignored `thinkingBudgetTokens` key: it passed on 400s and proved
nothing. Now asserts 200 plus scored usage metadata with the correct
`thinkingBudget` spelling.
- Cases the caller's project genuinely cannot serve now **skip** with an
actionable reason instead of reporting a red proxy.

**Docs** (`proxy.mdx`, `installation.mdx`, `wiki/vertex.md`)

- `proxy.mdx` claimed `publisher=anthropic` passthrough "routes through
the same LiteLLM-Vertex path as `--backend vertex_ai`". It does not —
without the flag it is a direct forward, which is why it works on a bare
`[proxy]` install where the LiteLLM path cannot even import.
- Documented the trap of combining `--backend vertex` with native Vertex
URLs: LiteLLM takes project/region/model from its own config and ignores
the URL.
- `installation.mdx` and `wiki/vertex.md` said the SDK is "not included
in any extra".
- Cleared 15 pre-existing `ruff` findings in the two files this PR
rewrites (4 remain in `plugins/headroom-oauth2`, untouched by this
work).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ mypy headroom
Success: no issues found in 511 source files

$ ruff check .
Found 4 errors.        # all in plugins/headroom-oauth2, pre-existing, untouched by this PR

$ pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q
44 passed, 1 warning in 69.86s          # 8 live integration + 36 offline diagnostics, 0 skipped

$ pytest tests/ -k "anthropic_messages or handler_anthropic or backend or registry or cli_proxy or vertex or proxy_routes"
853 passed, 49 skipped, 9345 deselected, 2 warnings in 28.41s
```

New tests in `tests/test_vertex_diagnostics.py` (36, no credentials
required): status coverage, per-publisher remedy divergence,
idempotency, `content-length` consistency after body rewrite, non-JSON
and body-less/streaming responses, every backend alias preflighted,
other providers unaffected, injected backend bypasses the preflight.

## Real Behavior Proof

- Environment: macOS 14, Python 3.11.15, live GCP project
(`alanblount-sandbox`) with `aiplatform.googleapis.com` enabled and
Claude enabled in Model Garden; ADC via `gcloud auth application-default
login`
- Exact command / steps: `export GCP_PROJECT_ID=alanblount-sandbox &&
export GCP_ACCESS_TOKEN=$(gcloud auth application-default
print-access-token) && uv run --extra proxy --with google-genai python3
examples/vertex_genai_sdk_demo.py && uv run --extra proxy --with
pytest-asyncio pytest tests/test_proxy_vertex_native_integration.py
tests/test_vertex_diagnostics.py -q && headroom proxy --backend vertex`
(exits 2, then retried with `[proxy,vertex]` extra installed)
- Observed result: demo printed `✓ Standard response received
successfully! > 1, 2, 3, 4, 5` and `✓ Thinking response received
successfully!`; 44 tests passed (8 live integration, 36 diagnostics), 0
skipped; `--backend vertex` exited 2 with the remedy message; same
request with the extra installed returned 200; invalid region produced
`[headroom] hint:` in the response body and `x-headroom-hint` header;
success responses were byte-clean
- Not tested: Windows; `us`/`eu` multi-region endpoints and Claude 4.7+
(project cannot serve them); Provisioned Throughput; Rust proxy path;
`--backend vertex` dropping URL project/model on native Vertex routes
(documented as known gap in Additional Notes)


## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — proxy and CLI behavior; terminal output is quoted above.

## Additional Notes

- `uv.lock` intentionally not included: the new extra is opt-in and does
not change default resolution.
- Error-body rewriting is confined to already-failing responses and
wrapped so a diagnostic can never turn a clean upstream error into a
proxy 500.
- The preflight raises where the provider is resolved (`registry.py`)
rather than at the CLI, so programmatic embedders get the same
guarantee; the CLI only formats it.
- Three test files fail to collect on this branch
(`test_hermes_passthrough_compression`, `test_observability_metrics`,
`test_observability_tracing`). They predate this work and are unrelated,
so they were excluded rather than fixed — happy to open a separate
issue.
- `[vertex]` was chosen over adding the dependency to `[proxy]` after
measuring the footprint at ~175 MB. If maintainers prefer it in
`[proxy]`, that is a one-line change.
## Runtime Rollout Safety

- Rollout-managed feature(s): none; diagnostics annotate already-failing
Vertex responses and the backend dependency is opt-in through the vertex
extra.
- Minimum rollout channel: normal stable release after Vertex
diagnostics, route, and backend-path tests pass.
- Stable/default behavior changed: successful responses and native
passthrough remain unchanged; supported Vertex failures gain an
actionable header/message, and selecting the LiteLLM Vertex backend
without its SDK now fails once at startup.
- Kill switch / disable path: use native Vertex passthrough without the
LiteLLM backend, or human-revert the diagnostic/preflight change.
- Unsafe override required: none.
- Qualification impact: offline diagnostics, native integration, and
mocked LiteLLM Vertex request-shaping tests must remain green.
- Rollback path: human-revert the PR; no persisted state or migration is
involved.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2640"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `5d7071f348d2462f666fd7ac1ceb8ac18050169a`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Re-reviewed latest integration head c70c1a497 and current author
responses, including fixed public error hints, log escaping, body/header
consistency, all four backend error sites, and native-route versus
LiteLLM SDK dependency behavior. Clean integration with upstream/main
2d3ee51846. No new PR-specific source
blocker found in reviewed scope; no contributor diagnostic/routing logic
rewritten.

232 Vertex diagnostics/native-route/backend/CLI/release/version/MCP
tests pass with11 skips. Offline lock validation passes (268 packages);
all12 changed Python files pass Ruff check/format, and whitespace
passes. These checks cover synthetic error/transport paths and do not
establish live cloud regional availability or billing behavior. Earlier
author live evidence remains scoped to its stated environment; no fresh
cloud request or native build claimed here. Historical shared
cached-read fixture failure was main-reproduced and is not a
contributor-code blocker. Maintainers handle current exact-head hosted
qualification before approval; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `486a716adbf642ae5e43276df4c2dc1e63b901d7`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical sync with upstream main
c01cf07d94. Re-reviewed current
author/Devin discussion and public diagnostic/error boundaries,
native-versus-SDK route separation and startup preflight. This sync does
not alter the PR Vertex diagnostics, backend or route-registry
modules.158 focused diagnostics/native-route/backend/streaming/CLI tests
pass,11live-environment skips,1deprecation warning. Offline uv lock
--check passes;all12 changed Python files pass Ruff lint/format and
main-relative whitespace passes. Fixed hints remain separate from raw
provider error text; multi-worker preflight and non-Google partner
guidance retain resolved regressions. No contributor diagnostic/routing
correctness logic rewritten. Tests use existing current-main DLL and
synthetic errors/transports; no fresh native build, live regional
availability/provider inference or billing qualification claimed. Formal
approval awaits exact-head hosted qualification; maintainers handle CI.
Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `0b75c609f24562c8b6ed2aaeaf58239984b4810c`, upstream main
`ce4275491299231bc9d2907833f72f72ea22b8a9`.

Requalified after human merges #4667,#4669,#4670 against upstream main
ce42754912. Earlier approval and
publication guards stopped stale candidates before either was posted.
Current clean merge retains Vertex diagnostics/native-versus-SDK routing
and main enable_kompress checks, lossless cold-prefix disable, and
Python token-window boundary behavior.190 combined
Vertex/backend/CLI/Kompress-disable/must-keep tests
pass,11live-environment skips,1warning. Offline lock validation,12-file
Ruff lint/format and main-relative whitespace pass. No contributor
diagnostic/routing logic rewritten. Native Rust/Cargo sources are
identical to c01 across these merges; existing matching-core DLL
supports tests, not a fresh native build. No new live Vertex
regional/provider/billing qualification claimed. All prior-main hosted
results are superseded for readiness; approval awaits exact checks on
this published merge head. Maintainers handle CI; no agent merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `1fe62702ce614315679d68d9d126e31c57208c67`, upstream main
`00d4f441d3f1d48c463fea57e6c574c2f3671302`.

Requalified after human merges #4667,#4669,#4670 against upstream main
ce42754912. Earlier approval and
publication guards stopped stale candidates before either was posted.
Current clean merge retains Vertex diagnostics/native-versus-SDK routing
and main enable_kompress checks, lossless cold-prefix disable, and
Python token-window boundary behavior.190 combined
Vertex/backend/CLI/Kompress-disable/must-keep tests
pass,11live-environment skips,1warning. Offline lock validation,12-file
Ruff lint/format and main-relative whitespace pass. No contributor
diagnostic/routing logic rewritten. Native Rust/Cargo sources are
identical to c01 across these merges; existing matching-core DLL
supports tests, not a fresh native build. No new live Vertex
regional/provider/billing qualification claimed. All prior-main hosted
results are superseded for readiness; approval awaits exact checks on
this published merge head. Maintainers handle CI; no agent merge.

Final current-main requalification:
07d5add5ba, including human merges #4671
action pinning and #4648 multimodal memory preservation. Clean
mechanical merge; Vertex/backend/CLI inputs retain the preceding
190-test qualification. Fresh workflow/release suite: 62 passed. Fresh
memory wrapper suite: 9 passed. New wrapper retains list payload parts
and extracts textual search inputs; no contributor Vertex logic
rewritten. Existing native input provenance remains unchanged. Hosted
qualification must run on this final head before approval.

Newer human merge #4646 advanced authoritative main to
8cdce172ea before publication. Cleanly
integrated quote-aware read detection; reviewed new masking/segmentation
overlap. Fresh read-command/content-router suite: 188 passed, 1 optional
SOCKS warning. Vertex code remains unchanged by this merge.

Final additional human merge #4067 packaging changes advanced main to
a8905214d: cleanly integrated explicit torch-extras split, retaining
Vertex optional dependencies. Fresh ONNX/optional-dependency contracts:
12 passed. Offline lock check and whitespace pass. No live clean-install
or Torch backend claim. Earlier publication candidates were stopped
before push by fresh-main guard.

Latest human merge #4042 advances authoritative main to
cde584d7fe. Clean merge preserves optional
LangChain availability handling and CI dependency setup. Fresh
binding/compatibility/startup-log/xAI regression suite: 29 passed. New
main integration re-reviewed; no contributor Vertex behavior rewritten.

Subsequent #4049 main merge 00d4f441d adds graceful context lookup/init
failure handling. Cleanly integrated with Vertex handler diagnostics
intact. Fresh Anthropic pre-upstream/memory native ops/OpenAI Codex
routing/WebSocket lifecycle suite: 192 passed, 1 deprecation warning.
Reviewed exception handling and transport overlap; no live provider
qualification claim.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `6c63811d6de7491ebce9a16f89e197f5f5dd86dd`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Requalified against authoritative
main2b0315b2d4b0d54947050bbdad2de9e89df06378 after further human merges.
All10 integration patches exactly match00d4f441-to-current-main changes
after diff-location normalization; contributor Vertex
diagnostics/routing policy unchanged.539combined
Vertex/backend/CLI/Kompress-disable/must-keep/binary/doctor/Windows-supervisor/provider/search/text
tests pass;13platform/live-environment skips,2warnings. Nine changed
Python files pass Ruff lint/format; offline lock and whitespace pass.
Existing pr3982 native DLL verified against every tracked native
build-input hash and artifact checksum; no fresh native build claimed.
Current author/Devin discussion checked; prior fixes remain intact. No
live Vertex regional/provider/billing,IPv6 listener or unsupported cloud
qualification claimed. No new finding in reviewed scope. Exact-head
hosted checks must qualify this published head before formal approval;
maintainers handle them. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
alan blount
2026-11-06 12:27:47 -06:00
committed by GitHub
co-authored by JD Davis
parent 2b0315b2d4
commit 0ddaa56c21
17 changed files with 1496 additions and 34 deletions
+5 -2
View File
@@ -298,8 +298,11 @@ These variables configure Headroom at runtime. Set them in your shell, `.env` fi
| `VERTEXAI_PROJECT` | GCP project id for the Vertex AI backend (LiteLLM-specific — distinct from `GOOGLE_CLOUD_PROJECT`; set it explicitly to avoid silently billing your ADC default quota project) |
| `VERTEXAI_LOCATION` | GCP region for the Vertex AI backend (LiteLLM-specific — distinct from `GOOGLE_CLOUD_LOCATION`) |
The Vertex AI backend also requires `google-cloud-aiplatform>=1.38`, which is not
included in any extra or Docker image — see
The Vertex AI backend also requires `google-cloud-aiplatform>=1.38`, which ships
in the `[vertex]` extra (`pip install "headroom-ai[proxy,vertex]"`) rather than
`[proxy]`, since it adds roughly 175 MB and only that backend needs it. The
proxy refuses to start if the backend is selected without it. Vertex's native
passthrough routes need neither the extra nor the backend flag — see
[Google Vertex AI](/docs/proxy#google-vertex-ai) for setup details.
### Proxy configuration
+37 -8
View File
@@ -1032,16 +1032,23 @@ OPENROUTER_API_KEY=sk-or-... headroom proxy --backend openrouter
`--backend vertex_ai` delegates to [LiteLLM](https://docs.litellm.ai/docs/providers/vertex),
which brings two requirements that are easy to miss:
**1. Install the Vertex SDK.** `google-cloud-aiplatform` is not included in any
Headroom extra (`[proxy]`, `[all]`, …) or Docker image variant, so install it
alongside Headroom:
**1. Install the Vertex SDK.** `google-cloud-aiplatform` ships in the dedicated
`[vertex]` extra, kept out of `[proxy]` because it and its transitive tree add
roughly 175 MB and only this backend needs it:
```bash
pip install "headroom-ai[proxy]" "google-cloud-aiplatform>=1.38"
pip install "headroom-ai[proxy,vertex]"
```
Without it, the first Vertex request fails with
`litellm.BadRequestError: … vertexai import failed … No module named 'vertexai'`.
Without it the proxy **refuses to start**, naming the fix and exiting `2`:
```text
Cannot start proxy: Vertex backend selected but the Vertex SDK is missing. ...
```
This is a startup check rather than a per-request failure, because the backend
would otherwise construct cleanly and then fail on every call with an opaque
`litellm.BadRequestError: … No module named 'vertexai'`.
**2. Set the LiteLLM project/location variables.** LiteLLM reads the GCP project
and region from `VERTEXAI_PROJECT` / `VERTEXAI_LOCATION` — these are **not** the
@@ -1093,12 +1100,34 @@ mirror Vertex's native REST shape verbatim — no backend flag needed:
```
Requests with `publisher=google` (Gemini models) go through Headroom's full
Gemini optimization handler; `publisher=anthropic` (Claude on Vertex) routes
through the same LiteLLM-Vertex path as `--backend vertex_ai`. Any client that
Gemini optimization handler; `publisher=anthropic` (Claude on Vertex) is
forwarded to Vertex directly, preserving your bearer token. Any client that
already speaks the native Vertex REST API can simply point its endpoint at the
proxy — this is the mechanism `headroom wrap claude` uses in
[Vertex mode](/docs/claude-code-vertex).
<Callout type="warn">
These routes need **neither** `--backend vertex_ai` nor the `[vertex]` extra. Do
not combine the two: with the backend flag set, Claude publisher requests are
re-routed through LiteLLM, which derives the project, region and model from its
own configuration and ignores the ones in your URL — a request that works
without the flag can return a `404` for a model you never asked for.
</Callout>
### Reading Vertex errors
When Vertex rejects a request with `401`, `403`, `404` or `429`, Headroom appends
a `[headroom] hint: …` note to `error.message`, sets an `x-headroom-hint`
response header, and logs a warning. The hint names the likely cause — expired
ADC, a partner model not yet enabled in Model Garden, or a model that is not
served in the requested location. Successful responses are never modified.
Hints are fixed text: they never echo your request path, project, credentials or
the underlying exception. With `--backend vertex`, a backend that cannot serve
(missing Vertex SDK, no ADC) returns the standard sanitized proxy error plus the
same kind of fixed hint; the raw provider detail is only in the proxy log, under
the response's `request_id`.
### Bedrock via a local gateway
`--backend bedrock` accepts **Anthropic** input (`/v1/messages`) and re-signs to AWS. Some setups are the other way around: the client already speaks **Bedrock** (e.g. Claude Code with `CLAUDE_CODE_USE_BEDROCK=1`, or any AWS SDK pointed at a custom endpoint), sending `POST /model/{id}/invoke` to a local gateway that re-signs and forwards to AWS (LiteLLM, LocalStack, a corporate Bedrock proxy).
+213
View File
@@ -0,0 +1,213 @@
#!/usr/bin/env python3
"""End-to-end demo: google-genai SDK -> Headroom proxy -> Vertex AI (Gemini Enterprise).
Proves that standard Vertex native configurations route properly through the proxy.
What this script does
---------------------
1. Spawns the Headroom proxy as a subprocess. No ``--backend vertex``: the native
Vertex routes are a passthrough and need neither that flag nor ``[vertex]``.
2. Waits for /readyz.
3. Configures standard google-genai SDK with vertexai=True hitting the proxy.
4. Sends an inference probe to validate native proxy connectivity.
5. Sends an inference probe with thinking configs to validate extensions.
6. Tears the proxy back down.
Requirements
------------
- GCP credentials with Vertex AI access (run `gcloud auth application-default login`)
- ``pip install "headroom-ai[proxy]" google-genai``
Run
---
export GCP_PROJECT_ID=your-project
python examples/vertex_genai_sdk_demo.py
Note the proxy needs the ``[proxy]`` extra (fastapi, uvicorn, httpx[http2]); with
only bare ``httpx`` installed it fails at startup on the missing ``h2`` package.
"""
from __future__ import annotations
import argparse
import os
import subprocess
import sys
import time
import urllib.error
import urllib.request
from contextlib import suppress
from pathlib import Path
# Defer imports until runtime to guarantee proxy has a chance to start
try:
from google import genai
from google.genai import types
except ImportError:
print("Error: google-genai is not installed.")
print("Run `pip install google-genai` and try again.")
sys.exit(1)
# ----------------------------------------------------------------------------
# Constants
# ----------------------------------------------------------------------------
DEFAULT_PORT = 8787
# The google-genai SDK always builds `publishers/google/...` paths, so this demo
# can only exercise Gemini. Claude on Vertex lives under `publishers/anthropic`
# and uses :rawPredict -- see tests/test_proxy_vertex_native_integration.py.
#
# `global` is the default because the evergreen "-latest" aliases are global-only;
# Gemini 3.x currently has no US regional endpoint.
DEFAULT_REGION = "global"
DEFAULT_MODEL = "gemini-flash-latest"
# Other options: gemini-3.5-flash (also servable in europe-west2, asia-northeast1)
# ----------------------------------------------------------------------------
# Proxy lifecycle
# ----------------------------------------------------------------------------
def start_proxy(port: int) -> subprocess.Popen[bytes]:
"""Spawn `headroom proxy` as a subprocess."""
env = os.environ.copy()
env.setdefault("HEADROOM_LOG", "INFO")
cmd = [sys.executable, "-m", "headroom.cli", "proxy", "--port", str(port)]
print(f" $ {' '.join(cmd)}", file=sys.stderr)
log_path = Path("/tmp") / f"vertex_genai_sdk_demo_{port}.log"
log_file = log_path.open("wb")
proc = subprocess.Popen(
cmd,
env=env,
stdout=log_file,
stderr=subprocess.STDOUT,
)
print(f" proxy logs -> {log_path}", file=sys.stderr)
return proc
def wait_for_proxy_ready(port: int, timeout_s: float = 30.0) -> None:
"""Poll /readyz until the proxy answers or timeout."""
url = f"http://127.0.0.1:{port}/readyz"
deadline = time.time() + timeout_s
last_err: Exception | None = None
while time.time() < deadline:
try:
with urllib.request.urlopen(url, timeout=1) as resp:
if resp.status == 200:
return
except (urllib.error.URLError, ConnectionError, TimeoutError) as e:
last_err = e
time.sleep(0.5)
raise RuntimeError(
f"Proxy on port {port} did not become ready within {timeout_s}s; last error: {last_err!r}"
)
def stop_proxy(proc: subprocess.Popen[bytes]) -> None:
"""Politely shut the proxy down."""
with suppress(ProcessLookupError):
proc.terminate()
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait(timeout=5)
# ----------------------------------------------------------------------------
# Main demo
# ----------------------------------------------------------------------------
def run_demo(port: int, region: str, model_id: str) -> int:
print("=" * 76)
print(" Headroom E2E: google-genai SDK -> Headroom proxy -> Vertex")
print("=" * 76)
print(f" port={port} region={region} model={model_id}")
print()
print("[1/3] Spawning Headroom proxy ...")
proxy = start_proxy(port=port)
try:
try:
wait_for_proxy_ready(port=port, timeout_s=45.0)
except Exception as e:
print(f" ! Proxy failed to start: {e}", file=sys.stderr)
return 2
print(" proxy ready.")
print("\n[2/3] Configuring google-genai SDK for Vertex via Proxy")
project_id = os.environ.get("GCP_PROJECT_ID")
if not project_id:
print(
" ! GCP_PROJECT_ID is not set. Every probe below will 404.\n"
" export GCP_PROJECT_ID=$(gcloud config get-value project)",
file=sys.stderr,
)
return 2
# We enforce vertexai=True to hit standard Vertex boundaries.
# Ensure your GCP ADC variables are valid / authorized if routing through real GCP APIs.
client = genai.Client(
vertexai=True,
project=project_id,
location=region,
http_options={"base_url": f"http://127.0.0.1:{port}"},
)
print("\n[3/3] Probes")
print("\n a. Standard Inference:")
content = "Count to 5, listing each number separated by commas."
try:
response = client.models.generate_content(
model=model_id,
contents=content,
)
print(" ✓ Standard response received successfully!")
print(f" > {response.text.strip()}")
except Exception as e:
# Vertex 401/403/404/429 errors carry a `[headroom] hint:` from the proxy.
print(f" ! Standard inference failed: {e}")
return 1
print("\n b. Inference with Thinking Config:")
# Configure thinking config (where supported).
# For now, we just pass parameters and see if Headroom properly parses/forwards them.
try:
config = types.GenerateContentConfig(
thinking_config=types.ThinkingConfig(thinking_budget=128)
)
response = client.models.generate_content(
model=model_id,
contents="Think briefly. What is heavier: a kg of feathers or a kg of steel?",
config=config,
)
print(" ✓ Thinking response received successfully!")
print(f" > {response.text.strip()}")
except Exception as e:
print(f" ! Thinking inference failed: {e}")
return 1
return 0
finally:
print("\n shutting down proxy ...")
stop_proxy(proxy)
def main() -> int:
ap = argparse.ArgumentParser(description="google-genai SDK -> Headroom proxy -> Vertex")
ap.add_argument("--port", type=int, default=DEFAULT_PORT)
ap.add_argument("--region", default=DEFAULT_REGION)
ap.add_argument("--model", default=DEFAULT_MODEL)
args = ap.parse_args()
return run_demo(port=args.port, region=args.region, model_id=args.model)
if __name__ == "__main__":
sys.exit(main())
+19 -4
View File
@@ -826,6 +826,18 @@ class LiteLLMBackend(Backend):
To add a new provider, just add an entry to PROVIDER_REGISTRY above.
"""
def _public_error_message(self, exc: BaseException, raw_text: str) -> str:
"""Client-safe error text (public_errors contract).
For Vertex, a fixed setup hint is appended when the raw text shows a
missing SDK or ADC; the raw text itself only ever reaches the log.
"""
if self.provider in ("vertex_ai", "vertex_ai_beta"):
from headroom.providers.vertex import public_backend_error_message
return public_backend_error_message(exc, raw_text)
return client_message(exc, raw_text)
def __init__(
self,
provider: str = "bedrock",
@@ -1353,7 +1365,7 @@ class LiteLLMBackend(Backend):
logger.error(f"LiteLLM error: {error_message}")
# Provider API errors keep their text; transport failures are
# reduced to the public vocabulary (see proxy/public_errors).
error_message = client_message(e, error_message)
error_message = self._public_error_message(e, error_message)
# Map to Anthropic error format
error_type = "api_error"
@@ -1790,7 +1802,10 @@ class LiteLLMBackend(Backend):
event_type="error",
data={
"type": "error",
"error": {"type": "api_error", "message": client_message(e, error_message)},
"error": {
"type": "api_error",
"message": self._public_error_message(e, error_message),
},
},
)
@@ -1982,7 +1997,7 @@ class LiteLLMBackend(Backend):
except Exception as e:
error_message = format_exception_message(e)
logger.error(f"LiteLLM OpenAI error: {error_message}")
error_message = client_message(e, error_message)
error_message = self._public_error_message(e, error_message)
# Map to OpenAI error format
error_type = "api_error"
@@ -2092,7 +2107,7 @@ class LiteLLMBackend(Backend):
logger.error(f"LiteLLM OpenAI streaming error: {error_message}")
error_data = {
"error": {
"message": client_message(e, error_message),
"message": self._public_error_message(e, error_message),
"type": "api_error",
"code": "backend_error",
}
+14
View File
@@ -16,6 +16,8 @@ import click
from headroom import paths as _paths
from headroom.providers.registry import (
BackendUnavailableError,
preflight_backend,
resolve_api_overrides,
resolve_api_targets,
resolve_extra_headers,
@@ -1944,6 +1946,13 @@ Press Ctrl+C to stop.
# Option E: start embedding server sidecar if requested
# -----------------------------------------------------------------------
_embed_watchdog = None
try:
# Here, not only in create_app: with --workers >1 each worker builds its
# own app and uvicorn would keep restarting them instead of exiting.
preflight_backend(backend)
except BackendUnavailableError as exc:
click.echo(f"\nCannot start proxy: {exc}", err=True)
raise SystemExit(2) from None
if embedding_server:
_embed_socket = embedding_server_socket or f"/tmp/headroom-embed-{config.port}.sock"
# Pass socket path to all worker processes via environment variable
@@ -1996,6 +2005,11 @@ Press Ctrl+C to stop.
except KeyboardInterrupt:
click.echo("\nShutting down...")
raise SystemExit(130) from None
except BackendUnavailableError as exc:
# Operator misconfiguration, not a crash to debug: show the remedy, not
# a traceback through our startup internals.
click.echo(f"\nCannot start proxy: {exc}", err=True)
raise SystemExit(2) from None
finally:
if _embed_watchdog is not None:
import asyncio as _asyncio2
+8
View File
@@ -67,6 +67,7 @@ from headroom.providers.vertex import (
is_vertex_google_publisher,
vertex_anthropic_target,
vertex_publisher_provider_name,
with_vertex_diagnostics,
)
from headroom.proxy.passthrough import (
custom_base_passthrough_telemetry as _custom_base_passthrough_telemetry,
@@ -359,6 +360,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
@app.post(
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:generateContent"
)
@with_vertex_diagnostics
async def vertex_generate_content(
request: Request,
api_version: str,
@@ -380,6 +382,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
@app.post(
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamGenerateContent"
)
@with_vertex_diagnostics
async def vertex_stream_generate_content(
request: Request,
api_version: str,
@@ -405,6 +408,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
@app.post(
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:countTokens"
)
@with_vertex_diagnostics
async def vertex_count_tokens(
request: Request,
api_version: str,
@@ -426,6 +430,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
@app.post(
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:rawPredict"
)
@with_vertex_diagnostics
async def vertex_raw_predict(
request: Request,
api_version: str,
@@ -447,6 +452,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
@app.post(
"/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:rawPredict"
)
@with_vertex_diagnostics
async def vertex_raw_predict_no_version(
request: Request,
project: str,
@@ -471,6 +477,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
@app.post(
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamRawPredict"
)
@with_vertex_diagnostics
async def vertex_stream_raw_predict(
request: Request,
api_version: str,
@@ -497,6 +504,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
@app.post(
"/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamRawPredict"
)
@with_vertex_diagnostics
async def vertex_stream_raw_predict_no_version(
request: Request,
project: str,
+42 -7
View File
@@ -25,6 +25,14 @@ AnyLLMBackendType: Any = None
LiteLLMBackendType: Any = None
class BackendUnavailableError(RuntimeError):
"""A selected backend cannot serve and no amount of retrying will help.
Raised at startup for operator misconfiguration (a missing optional
dependency, say) so callers can print the remedy instead of a traceback.
"""
@dataclass(frozen=True)
class ProviderApiOverrides:
"""Optional upstream API URL overrides configured for the proxy."""
@@ -213,6 +221,36 @@ def build_proxy_provider_runtime(config: Any) -> ProxyProviderRuntime:
)
def _litellm_provider(backend: str) -> str:
provider = (backend if backend.startswith("litellm-") else f"litellm-{backend}").replace(
"litellm-", ""
)
# `litellm-vertex` is the name in our docs/help, but LiteLLM (and our
# provider registry) keys Google Vertex on `vertex_ai`. Without this alias
# the provider falls through to a generic pass-through: wrong model prefix
# (`vertex/…` instead of `vertex_ai/…`), region dropped, auth mishandled.
if provider in ("vertex", "google-vertex", "googlevertex"):
return "vertex_ai"
return provider
def preflight_backend(backend: str) -> None:
"""Raise ``BackendUnavailableError`` if ``backend``'s optional SDK is missing.
Without the Vertex SDK the LiteLLM backend still constructs cleanly and then
fails on every call with a provider string that names neither the cause nor
the cure. The CLI calls this in the parent process too: with ``--workers >1``
each worker builds its own app and uvicorn would just keep restarting them.
"""
if backend == "anthropic" or backend == "anyllm" or backend.startswith("anyllm-"):
return
if _litellm_provider(backend) in ("vertex_ai", "vertex_ai_beta"):
# Imported locally: headroom.providers.vertex imports this module.
from headroom.providers.vertex import ensure_vertex_sdk_available
ensure_vertex_sdk_available()
def create_proxy_backend(
*,
backend: str,
@@ -249,13 +287,10 @@ def create_proxy_backend(
return None
normalized_backend = backend if backend.startswith("litellm-") else f"litellm-{backend}"
provider = normalized_backend.replace("litellm-", "")
# `litellm-vertex` is the name in our docs/help, but LiteLLM (and our
# provider registry) keys Google Vertex on `vertex_ai`. Without this alias
# the provider falls through to a generic pass-through: wrong model prefix
# (`vertex/…` instead of `vertex_ai/…`), region dropped, auth mishandled.
if provider in ("vertex", "google-vertex", "googlevertex"):
provider = "vertex_ai"
provider = _litellm_provider(backend)
if litellm_backend_cls is None:
# Skipped when a backend class is injected, so tests can use fakes.
preflight_backend(backend)
try:
backend_cls = litellm_backend_cls or _load_litellm_backend()
instance = cast(
+14
View File
@@ -1,5 +1,13 @@
"""Google Vertex provider helpers."""
from .diagnostics import (
HINT_HEADER,
annotate_vertex_error,
ensure_vertex_sdk_available,
public_backend_error_message,
vertex_error_hint,
with_vertex_diagnostics,
)
from .runtime import (
VERTEX_ANTHROPIC_PROVIDER_NAME,
VERTEX_COUNT_TOKENS,
@@ -17,6 +25,7 @@ from .runtime import (
)
__all__ = [
"HINT_HEADER",
"VERTEX_ANTHROPIC_PROVIDER_NAME",
"VERTEX_COUNT_TOKENS",
"VERTEX_GENERATE_CONTENT",
@@ -25,9 +34,14 @@ __all__ = [
"VERTEX_STREAM_GENERATE_CONTENT",
"VERTEX_STREAM_RAW_PREDICT",
"VertexPublisherAction",
"annotate_vertex_error",
"ensure_vertex_sdk_available",
"is_vertex_anthropic_publisher",
"is_vertex_google_publisher",
"public_backend_error_message",
"vertex_anthropic_target",
"vertex_error_hint",
"vertex_publisher_provider_name",
"vertex_target_for_location",
"with_vertex_diagnostics",
]
+209
View File
@@ -0,0 +1,209 @@
"""Actionable diagnostics for Vertex upstream failures.
Vertex rejects requests for a handful of boring, recurring reasons -- expired
ADC, an un-enabled partner model, a model that simply is not served in the
requested location -- and its own error text says nothing about how to fix any
of them. Users hit these while onboarding, see a bare 404 through whatever SDK
they are using, and have no way to tell a proxy bug from a project-config gap.
So annotate the failure at the proxy and surface a hint three ways: a WARNING in
the proxy log, an ``x-headroom-hint`` response header, and (because most SDKs
only ever show the message string) an appended note on ``error.message``.
Every hint is one of the fixed constants below: request data and exception text
select a hint but are never interpolated into it (public_errors contract).
Only already-failing responses are touched; success bodies are never modified.
"""
from __future__ import annotations
import functools
import importlib.util
import json
import logging
from collections.abc import Awaitable, Callable
from typing import Any
from headroom.providers.registry import BackendUnavailableError
from headroom.proxy.public_errors import client_message
from .runtime import is_vertex_anthropic_publisher
logger = logging.getLogger(__name__)
HINT_HEADER = "x-headroom-hint"
_ADC_REFRESH = (
"Credentials are missing or expired (Vertex access tokens last ~1h). "
"Re-auth with `gcloud auth application-default login`, and note that a bare "
"`gcloud auth print-access-token` user token is NOT accepted -- use "
"`gcloud auth application-default print-access-token`."
)
_ENABLE_API = (
"Confirm `aiplatform.googleapis.com` is enabled on the project and that the "
"caller holds roles/aiplatform.user."
)
_ENABLE_PARTNER = (
"Partner models (Anthropic Claude, Llama, Mistral) need a one-time per-project "
"enable in Model Garden before they will serve -- an otherwise healthy project "
"returns 404/403 until someone clicks through."
)
_LOCATION_GEMINI = (
"Gemini 3.x has no US regional endpoint: the evergreen `-latest` aliases are "
"global-only, and e.g. gemini-3.5-flash serves from `global`, europe-west2 and "
"asia-northeast1. Retry against `global` before assuming the proxy is at fault."
)
_LOCATION_CLAUDE = (
"Claude 4.6 and older serve from `us-east5`/europe-west1/asia-southeast1 or "
"`global`; Claude 4.7+ dropped named regions and is reachable only via the "
"`us`/`eu` multi-region or `global` endpoints."
)
_QUOTA = (
"Quota exhausted for this model in this location. Back off and retry, request a "
"quota increase, or send the request to another supported region."
)
_MISSING_VERTEX_SDK = (
"`--backend vertex` routes Anthropic Messages traffic through LiteLLM's vertex_ai "
'provider, which needs the Vertex SDK: `pip install "headroom-ai[proxy,vertex]"` '
'(or `pip install "google-cloud-aiplatform>=1.38"`). '
"Note the native Vertex routes (/v1/projects/.../publishers/anthropic/models/...:rawPredict) "
"are a straight passthrough and do NOT need `--backend vertex` -- drop the flag if you are "
"calling Vertex paths directly."
)
_MISSING_ADC = (
"No usable Google credentials were found. Run `gcloud auth application-default login`, or "
"point GOOGLE_APPLICATION_CREDENTIALS at a service-account key."
)
def vertex_sdk_available() -> bool:
"""Whether the LiteLLM vertex_ai provider's `vertexai` import will succeed."""
return importlib.util.find_spec("vertexai") is not None
def ensure_vertex_sdk_available() -> None:
"""Refuse to start a Vertex LiteLLM backend that cannot possibly serve.
Without the SDK the backend still constructs cleanly and then fails on
*every* request with an opaque provider string. Failing here instead turns a
per-request mystery into one startup error at the moment of misconfiguration.
"""
if not vertex_sdk_available():
raise BackendUnavailableError(
f"Vertex backend selected but the Vertex SDK is missing. {_MISSING_VERTEX_SDK}"
)
def backend_error_hint(message: str) -> str | None:
"""Fixed setup hint for a raw LiteLLM Vertex error, or None if unrecognized."""
lowered = message.lower()
if "google-cloud-aiplatform" in lowered or (
"vertexai" in lowered and ("no module named" in lowered or "import failed" in lowered)
):
return _MISSING_VERTEX_SDK
if "default credentials" in lowered or "could not automatically determine" in lowered:
return _MISSING_ADC
return None
def public_backend_error_message(exc: BaseException, raw_text: str) -> str:
"""``client_message`` plus a fixed setup hint selected by ``raw_text``.
``raw_text`` itself never reaches the client; the caller logs it.
"""
message = client_message(exc, raw_text)
hint = backend_error_hint(raw_text)
# A chained Headroom upstream may already have added one.
if not hint or "[headroom] hint:" in message:
return message
return f"{message}\n[headroom] hint: {hint}"
def vertex_error_hint(status_code: int, *, publisher: str = "") -> str | None:
"""Fixed hint for a failing Vertex status, or None if we have none."""
if status_code == 401:
return _ADC_REFRESH
if status_code == 429:
return _QUOTA
if status_code not in (403, 404):
return None
# 403 and 404 are the same problem for the user: the project cannot serve
# this model here. Which remedy applies depends on the publisher.
if is_vertex_anthropic_publisher(publisher):
return f"{_ENABLE_API} {_ENABLE_PARTNER} {_LOCATION_CLAUDE}"
if publisher != "google":
return f"{_ENABLE_API} {_ENABLE_PARTNER}"
return f"{_ENABLE_API} {_LOCATION_GEMINI}"
def annotate_vertex_error(
response: Any,
*,
location: str = "",
publisher: str = "",
model: str = "",
) -> Any:
"""Attach a Headroom hint to a failing Vertex response, in place.
Returns the same response. Any problem while annotating is swallowed: a
diagnostic must never turn a clean upstream error into a proxy 500.
"""
status = getattr(response, "status_code", None)
hint = vertex_error_hint(status, publisher=publisher) if isinstance(status, int) else None
if hint is None:
return response
try:
# Idempotent: nested route helpers can annotate the same response twice.
if HINT_HEADER in response.headers:
return response
# %r: path params are client-supplied; keep CR/LF from forging log lines.
logger.warning(
"vertex upstream %s for %r/%r @ %r -- %s", status, publisher, model, location, hint
)
response.headers[HINT_HEADER] = hint
# Streaming responses have no materialized body; header + log suffice.
body = getattr(response, "body", None)
if not isinstance(body, (bytes, bytearray)):
return response
payload = json.loads(body)
if not isinstance(payload, dict):
return response
error = payload.get("error")
if isinstance(error, dict) and isinstance(error.get("message"), str):
# Most SDKs only show `error.message`, so the hint must live there.
error["message"] = f"{error['message']}\n[headroom] hint: {hint}"
else:
payload["headroom_hint"] = hint
response.body = json.dumps(payload).encode("utf-8")
response.headers["content-length"] = str(len(response.body))
except Exception: # noqa: BLE001 - see docstring
pass
return response
def with_vertex_diagnostics(
handler: Callable[..., Awaitable[Any]],
) -> Callable[..., Awaitable[Any]]:
"""Decorate a Vertex route so its failures carry an actionable hint.
Reads `location`/`publisher`/`model` from the path params FastAPI injects.
"""
@functools.wraps(handler)
async def wrapper(*args: Any, **kwargs: Any) -> Any:
return annotate_vertex_error(
await handler(*args, **kwargs),
location=kwargs.get("location", ""),
publisher=kwargs.get("publisher", ""),
model=kwargs.get("model", ""),
)
return wrapper
+11
View File
@@ -137,6 +137,17 @@ proxy-prod = [
"headroom-ai[proxy]",
"gunicorn>=21.0.0; sys_platform != 'win32'",
]
# Google Vertex through the LiteLLM translation backend
# (`--backend vertex` / `vertex_ai` / `litellm-vertex`), which imports `vertexai`.
# Kept separate from [proxy] for the same reason as proxy-prod: this pulls
# google-cloud-aiplatform and its transitive tree (~175MB installed), and the
# proxy's *native* Vertex passthrough routes
# (/v1/projects/.../locations/.../publishers/...) do not need it at all. Only
# the LiteLLM backend does. Install with:
# pip install headroom-ai[proxy,vertex]
vertex = [
"google-cloud-aiplatform>=1.38",
]
# AST-based code compression (tree-sitter)
# tree-sitter-language-pack >=1.0 removed the bundled tree-sitter package and
# switched to an incompatible internal node API (.kind vs .type, callable
+4 -1
View File
@@ -1016,7 +1016,10 @@ class TestCLIProxyBackend:
def mock_run_server(config, **kwargs):
captured_config["config"] = config
with patch("headroom.proxy.server.run_server", mock_run_server):
with (
patch("headroom.proxy.server.run_server", mock_run_server),
patch("headroom.providers.vertex.diagnostics.vertex_sdk_available", return_value=True),
):
result = runner.invoke(
main,
["proxy", "--backend", "litellm-vertex"],
+3
View File
@@ -281,6 +281,9 @@ class TestOpenAIStreamingMock:
with (
patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}),
patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp,
# Suppress the SDK preflight: this test proves LiteLLM request-shaping,
# not whether google-cloud-aiplatform is installed in the CI environment.
patch("headroom.providers.vertex.diagnostics.vertex_sdk_available", return_value=True),
):
mock_acomp.return_value = fake_stream()
app = create_app(config)
@@ -224,6 +224,9 @@ def test_litellm_vertex_backend_path_preserves_max_tokens_and_vendor_fields():
with (
patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}),
patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp,
# Suppress the SDK preflight: this test proves LiteLLM request-shaping,
# not whether google-cloud-aiplatform is installed in the CI environment.
patch("headroom.providers.vertex.diagnostics.vertex_sdk_available", return_value=True),
):
mock_acomp.return_value = _make_litellm_response()
app = create_app(config)
@@ -0,0 +1,239 @@
"""Integration tests for Vertex native API endpoint with real API calls through the proxy.
Run with:
export GCP_PROJECT_ID="your-project"
export GCP_ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
pytest tests/test_proxy_vertex_native_integration.py -rs -v
Use the *application-default* token, not `gcloud auth print-access-token`: a bare
user token is rejected by aiplatform.googleapis.com with 401 UNAUTHENTICATED.
Tokens expire hourly, so re-export before each run.
Partner (Anthropic) models must be enabled per-project in Model Garden. Cases
that the project cannot serve SKIP with an actionable reason instead of failing;
run with `-rs` so those reasons are printed.
"""
import json
import os
import pytest
pytestmark = pytest.mark.skipif(
not os.environ.get("GCP_ACCESS_TOKEN") or not os.environ.get("GCP_PROJECT_ID"),
reason="GCP_ACCESS_TOKEN or GCP_PROJECT_ID not set",
)
pytest.importorskip("fastapi")
pytest.importorskip("httpx")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
@pytest.fixture
def vertex_client():
"""Create test client for Vertex AI native API with optimization enabled."""
config = ProxyConfig(
optimize=True, # Enable compression
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
)
app = create_app(config)
with TestClient(app) as client:
yield client
@pytest.fixture
def api_key():
"""Get GCP access token from environment."""
return os.environ.get("GCP_ACCESS_TOKEN")
@pytest.fixture
def project_id():
"""Get GCP project ID from environment."""
return os.environ.get("GCP_PROJECT_ID")
def _upstream_message(response) -> str:
"""Best-effort extraction of the Vertex error message."""
try:
return str(response.json().get("error", {}).get("message", response.text))
except Exception:
return response.text
def _require_upstream_ok(response, url, location, model, publisher) -> None:
"""Fail loudly on proxy defects; skip on project-provisioning gaps.
A non-200 here is one of two very different things, and conflating them
makes this suite useless:
* the proxy mangled/misrouted the request -> a real failure, fail hard
* the caller's GCP project cannot serve this model here -> not our bug
Partner (Anthropic) models in particular require a per-project enable step
in Model Garden, so a project that is otherwise healthy will 403/404 on
Claude until someone clicks through. Skip those with an actionable reason
rather than reporting a red proxy.
"""
if response.status_code == 200:
return
detail = _upstream_message(response)
ctx = f"{publisher}/{model} @ {location}\n URL: {url}\n Vertex said: {detail}"
if response.status_code == 401:
pytest.fail(
f"401 UNAUTHENTICATED for {ctx}\n"
" GCP_ACCESS_TOKEN is missing, stale (they expire hourly), or is a user\n"
" token where ADC is required. Refresh with:\n"
" export GCP_ACCESS_TOKEN=$(gcloud auth application-default print-access-token)"
)
if response.status_code == 403:
pytest.skip(
f"403 PERMISSION_DENIED for {ctx}\n"
" The project cannot serve this model. Either aiplatform.googleapis.com is\n"
" disabled, the principal lacks roles/aiplatform.user, or (for Anthropic)\n"
" the model has not been enabled in Model Garden for this project."
)
if response.status_code == 404:
pytest.skip(
f"404 NOT_FOUND for {ctx}\n"
" This model is not available to the project at this location.\n"
f" For publisher '{publisher}', enable the model in Model Garden, or confirm\n"
" the location is in its supported-regions list. Note Gemini 3.x has no US\n"
" regional endpoint, and Claude 4.7+ dropped named regions for us/eu/global."
)
if response.status_code == 429:
pytest.skip(f"429 RESOURCE_EXHAUSTED (quota) for {ctx}")
if response.status_code == 400 and "not servable in region" in detail:
pytest.skip(
f"400 FAILED_PRECONDITION for {ctx}\n"
" The parametrized location/model pair is wrong -- fix the test matrix."
)
pytest.fail(
f"Expected 200, got {response.status_code} for {ctx}\n"
" This status is not a known provisioning condition, so treat it as a proxy\n"
" defect (bad path rewrite, mangled body, dropped auth header) until proven\n"
" otherwise by reproducing the same call directly against Vertex with curl."
)
# Location/model pairs are explicit rather than a cross-product: Vertex serves a
# given publisher model in only a subset of locations, so a cross-product
# generates combinations that can never return 200.
#
# global -> evergreen "-latest" aliases (global-only) + Claude
# europe-west2 -> Gemini 3.x has no US regional endpoint; EU/APAC regions only
# us-east5 -> Claude 4.6 and older have real regional serving here
#
# Deliberately no Gemini 2.5: it retires 2026-10-16 and should not be the thing
# this suite proves the proxy against.
@pytest.mark.parametrize(
"location,model,publisher",
[
("global", "gemini-flash-latest", "google"),
("global", "claude-sonnet-4-6", "anthropic"),
("europe-west2", "gemini-3.5-flash", "google"),
("us-east5", "claude-sonnet-4-6", "anthropic"),
],
)
class TestVertexNativeGenerateContent:
"""Test Vertex model endpoints."""
def test_basic_generation(self, vertex_client, api_key, project_id, location, model, publisher):
"""Basic text generation works."""
# Determine the action based on publisher
action = "generateContent" if publisher == "google" else "rawPredict"
url = f"/v1/projects/{project_id}/locations/{location}/publishers/{publisher}/models/{model}:{action}"
# Prepare the payload (Anthropic Vertex requires Claude messages format for rawPredict,
# Google Vertex requires contents format for generateContent)
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
if publisher == "google":
payload = {
"contents": [
{
"role": "user",
"parts": [{"text": "What is 2+2? Reply with just the number."}],
}
]
}
else:
payload = {
"anthropic_version": "vertex-2023-10-16",
"messages": [
{"role": "user", "content": "What is 2+2? Reply with just the number."}
],
"max_tokens": 100,
}
response = vertex_client.post(url, headers=headers, json=payload)
_require_upstream_ok(response, url, location, model, publisher)
data = response.json()
if publisher == "google":
assert "candidates" in data
assert len(data["candidates"]) > 0
text = data["candidates"][0]["content"]["parts"][0]["text"]
assert "4" in text
else:
assert "content" in data
assert len(data["content"]) > 0
text = data["content"][0]["text"]
assert "4" in text
def test_thinking_levels(self, vertex_client, api_key, project_id, location, model, publisher):
"""Test thinking extensions where supported."""
action = "generateContent" if publisher == "google" else "rawPredict"
url = f"/v1/projects/{project_id}/locations/{location}/publishers/{publisher}/models/{model}:{action}"
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
if publisher == "google":
payload = {
"contents": [
{"role": "user", "parts": [{"text": "Think deeply about the number 42."}]}
],
# REST spelling is thinkingBudget (camelCase); thinkingBudgetTokens
# is silently ignored, which makes the assertion below meaningless.
"generationConfig": {
"thinkingConfig": {"thinkingBudget": 512, "includeThoughts": True}
},
}
else:
payload = {
"anthropic_version": "vertex-2023-10-16",
"messages": [{"role": "user", "content": "Think deeply about the number 42."}],
"max_tokens": 8192,
"thinking": {"type": "enabled", "budget_tokens": 1024},
}
response = vertex_client.post(url, headers=headers, json=payload)
_require_upstream_ok(response, url, location, model, publisher)
# A 200 alone is weak evidence: prove the thinking-configured request
# actually produced a scored generation rather than an empty envelope.
data = response.json()
if publisher == "google":
assert data.get("candidates"), (
f"No candidates in thinking response: {json.dumps(data)[:500]}"
)
assert "usageMetadata" in data, f"No usageMetadata: {json.dumps(data)[:500]}"
else:
assert data.get("content"), f"No content in thinking response: {json.dumps(data)[:500]}"
assert data.get("usage"), f"No usage in thinking response: {json.dumps(data)[:500]}"
# Rest of the standard testing for compression could be added, similar to Gemini tests.
+329
View File
@@ -0,0 +1,329 @@
"""Vertex onboarding diagnostics: hints a user sees when Vertex rejects a request.
Every hint must be fixed text (public_errors contract): request data and raw
exception text may *select* a hint but must never reach the client.
"""
import json
import logging
from unittest.mock import patch
import pytest
from headroom.providers.registry import BackendUnavailableError, create_proxy_backend
from headroom.providers.vertex import (
HINT_HEADER,
annotate_vertex_error,
public_backend_error_message,
vertex_error_hint,
)
from headroom.proxy.public_errors import INTERNAL_ERROR, public_message
_HOSTILE = "m\r\nX-Injected: 1<script>caf\u00e9-secret-proj-123"
_SECRET_RAW = (
"Your default credentials were not found. project=secret-proj-123 "
"GOOGLE_APPLICATION_CREDENTIALS=/etc/keys/sa.json host=10.1.2.3"
)
_LEAKS = ("secret-proj-123", "/etc/keys/sa.json", "10.1.2.3")
class _FakeResponse:
"""Minimal stand-in for a Starlette Response."""
def __init__(self, status_code: int, body: bytes | None = None):
self.status_code = status_code
self.headers: dict[str, str] = {}
if body is not None:
self.body = body
def _error_body(status: int, message: str | None = "upstream said no") -> bytes:
error: dict = {"code": status}
if message is not None:
error["message"] = message
return json.dumps({"error": error}).encode()
class TestVertexErrorHint:
@pytest.mark.parametrize("status", [200, 400, 500, 503])
def test_no_hint_for_unexplainable_statuses(self, status):
"""Inventing a hint for an unknown status would mislead, not help."""
assert vertex_error_hint(status, publisher="google") is None
def test_401_points_at_adc_not_user_token(self):
hint = vertex_error_hint(401)
assert "gcloud auth application-default login" in hint
assert "gcloud auth print-access-token` user token is NOT accepted" in hint
def test_429_names_quota(self):
assert vertex_error_hint(429, publisher="anthropic").startswith("Quota exhausted")
@pytest.mark.parametrize("status", [403, 404])
def test_partner_vs_gemini_remedies(self, status):
claude = vertex_error_hint(status, publisher="anthropic")
gemini = vertex_error_hint(status, publisher="google")
mistral = vertex_error_hint(status, publisher="mistralai")
assert "Model Garden" in claude and "Claude 4.7+" in claude
assert "no US regional endpoint" in gemini and "Model Garden" not in gemini
assert "Model Garden" in mistral and "Gemini" not in mistral and "Claude 4" not in mistral
@pytest.mark.parametrize("status", [401, 403, 404, 429])
@pytest.mark.parametrize("publisher", ["google", "anthropic", "", _HOSTILE])
def test_every_hint_is_a_legal_header_value(self, status, publisher):
"""h11 rejects control / non-latin-1 bytes at send time, after we return."""
hint = vertex_error_hint(status, publisher=publisher)
hint.encode("latin-1")
assert all(" " <= ch <= "~" for ch in hint)
class TestAnnotateVertexError:
def test_success_untouched(self):
body = json.dumps({"candidates": []}).encode()
resp = annotate_vertex_error(_FakeResponse(200, body))
assert resp.body == body and HINT_HEADER not in resp.headers
def test_hint_in_header_and_error_message_with_consistent_length(self):
resp = annotate_vertex_error(_FakeResponse(404, _error_body(404)), publisher="google")
message = json.loads(resp.body)["error"]["message"]
assert message == f"upstream said no\n[headroom] hint: {resp.headers[HINT_HEADER]}"
assert resp.headers["content-length"] == str(len(resp.body))
def test_annotation_is_idempotent(self):
resp = annotate_vertex_error(_FakeResponse(401, _error_body(401)))
once = resp.body
assert annotate_vertex_error(resp).body == once
def test_error_without_message_gets_namespaced_key(self):
resp = annotate_vertex_error(_FakeResponse(404, _error_body(404, message=None)))
assert json.loads(resp.body)["headroom_hint"] == resp.headers[HINT_HEADER]
@pytest.mark.parametrize("body", [b"<html>nope</html>", b"[1, 2]", None])
def test_unrewritable_bodies_survive_with_header(self, body):
"""Non-JSON, non-object JSON and streaming (no body) keep the header only."""
resp = annotate_vertex_error(_FakeResponse(403, body), publisher="anthropic")
assert getattr(resp, "body", None) == body
assert HINT_HEADER in resp.headers
def test_hostile_path_params_reach_log_escaped_but_never_client(self, caplog):
with caplog.at_level(logging.WARNING):
resp = annotate_vertex_error(
_FakeResponse(404, _error_body(404)),
location=_HOSTILE,
publisher=_HOSTILE,
model=_HOSTILE,
)
client_visible = resp.headers[HINT_HEADER] + resp.body.decode()
assert "X-Injected" not in client_visible and "secret-proj" not in client_visible
# Operators keep the context, with CR/LF escaped by %r.
assert "secret-proj-123" in caplog.text and "\r\n" not in caplog.text
class TestNativeRoutesWiring:
"""The decorator must actually be applied on the real FastAPI routes."""
@pytest.fixture
def client(self, monkeypatch):
pytest.importorskip("fastapi")
from fastapi.responses import JSONResponse
from fastapi.testclient import TestClient
from headroom.proxy.server import HeadroomProxy, ProxyConfig, create_app
async def upstream_404(self, request, *args):
return JSONResponse({"error": {"code": 404, "message": "not found"}}, 404)
for name in ("handle_gemini_generate_content", "handle_passthrough"):
monkeypatch.setattr(HeadroomProxy, name, upstream_404)
app = create_app(ProxyConfig(optimize=False, cache_enabled=False, rate_limit_enabled=False))
with TestClient(app) as client:
yield client
@pytest.mark.parametrize(
"publisher,action,expect",
[
("google", "generateContent", "no US regional endpoint"),
("anthropic", "generateContent", "Model Garden"),
],
)
def test_upstream_404_carries_hint(self, client, publisher, action, expect):
resp = client.post(
f"/v1/projects/p/locations/us-central1/publishers/{publisher}/models/m:{action}"
)
assert resp.status_code == 404
assert expect in resp.headers[HINT_HEADER]
assert "[headroom] hint:" in resp.json()["error"]["message"]
def test_encoded_crlf_in_path_is_not_reflected(self, client):
"""%0d%0a decodes into the `model` path param; the response must stay sane."""
resp = client.post(
"/v1/projects/p/locations/us-central1/publishers/google/models/"
"m%0d%0aX-Injected:%201:generateContent"
)
assert resp.status_code == 404
assert "no US regional endpoint" in resp.headers[HINT_HEADER]
assert "x-injected" not in resp.headers
assert "X-Injected" not in resp.text
class TestPublicBackendErrorMessage:
def test_internal_error_gets_fixed_sentence_plus_hint_only(self):
out = public_backend_error_message(RuntimeError(_SECRET_RAW), _SECRET_RAW)
prefix = f"{public_message(INTERNAL_ERROR)}\n[headroom] hint: "
assert out.startswith(prefix)
assert "gcloud auth application-default login" in out
assert not any(leak in out for leak in _LEAKS)
def test_missing_sdk_hint(self):
raw = "vertexai import failed. Got error: No module named 'vertexai' at /opt/venv"
out = public_backend_error_message(ImportError(raw), raw)
assert "headroom-ai[proxy,vertex]" in out
assert "do NOT need `--backend vertex`" in out
assert "/opt/venv" not in out
def test_unrecognized_error_is_plain_public_message(self):
raw = "upstream timed out talking to 10.9.9.9"
assert public_backend_error_message(RuntimeError(raw), raw) == public_message(
INTERNAL_ERROR
)
def test_chained_proxy_hint_is_not_repeated(self):
"""A provider HTTP error passes its text through; it may already hold our hint."""
httpx = pytest.importorskip("httpx")
openai = pytest.importorskip("openai")
raw = (
"vertexai import failed: No module named 'vertexai'\n"
"[headroom] hint: install headroom-ai[proxy,vertex]"
)
resp = httpx.Response(500, request=httpx.Request("POST", "http://x"))
exc = openai.InternalServerError(raw, response=resp, body=None)
assert public_backend_error_message(exc, raw).count("[headroom] hint:") == 1
class TestLiteLLMVertexBackendErrors:
"""Through the real backend: raw text stays in the log, hint reaches client."""
@pytest.fixture
def litellm_mod(self, monkeypatch):
mod = pytest.importorskip("headroom.backends.litellm")
if not mod.LITELLM_AVAILABLE:
pytest.skip("litellm not installed")
async def boom(**_kw):
raise RuntimeError(_SECRET_RAW)
monkeypatch.setattr(mod, "acompletion", boom)
return mod
_BODY = {"model": "claude-sonnet-4-6", "messages": [{"role": "user", "content": "hi"}]}
_OPENAI_BODY = {"model": "gemini-3.5-flash", "messages": [{"role": "user", "content": "hi"}]}
@pytest.mark.asyncio
async def test_send_message(self, litellm_mod, caplog):
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
with caplog.at_level(logging.ERROR):
resp = await backend.send_message(dict(self._BODY), {})
message = resp.body["error"]["message"]
assert "[headroom] hint:" in message
assert not any(leak in message for leak in _LEAKS)
assert "secret-proj-123" in caplog.text
@pytest.mark.asyncio
async def test_send_openai_message(self, litellm_mod):
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
resp = await backend.send_openai_message(dict(self._OPENAI_BODY), {})
message = resp.body["error"]["message"]
assert "[headroom] hint:" in message
assert not any(leak in message for leak in _LEAKS)
@pytest.mark.asyncio
async def test_stream_message_error_event(self, litellm_mod):
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
events = [e async for e in backend.stream_message(dict(self._BODY, stream=True), {})]
errors = [e.data["error"]["message"] for e in events if e.event_type == "error"]
assert len(errors) == 1
assert "[headroom] hint:" in errors[0]
assert not any(leak in errors[0] for leak in _LEAKS)
@pytest.mark.asyncio
async def test_stream_openai_message_error_chunk(self, litellm_mod):
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
chunks = [c async for c in backend.stream_openai_message(dict(self._OPENAI_BODY), {})]
assert chunks[-1] == "data: [DONE]\n\n"
message = json.loads(chunks[-2].removeprefix("data: "))["error"]["message"]
assert "[headroom] hint:" in message
assert not any(leak in message for leak in _LEAKS)
@pytest.mark.asyncio
async def test_non_vertex_backend_gets_no_vertex_hint(self, litellm_mod):
backend = litellm_mod.LiteLLMBackend(provider="openrouter")
resp = await backend.send_message(dict(self._BODY), {})
assert "[headroom] hint:" not in resp.body["error"]["message"]
class TestVertexSdkPreflight:
"""`--backend vertex` without the SDK must fail at startup, not per-request."""
@pytest.fixture
def no_sdk(self, monkeypatch):
monkeypatch.setattr(
"headroom.providers.vertex.diagnostics.importlib.util.find_spec",
lambda name: None if name == "vertexai" else object(),
)
def _create(self, backend, **kwargs):
return create_proxy_backend(
backend=backend,
anyllm_provider="",
bedrock_region=None,
logger=logging.getLogger("test"),
**kwargs,
)
@pytest.mark.parametrize(
"backend", ["vertex", "vertex_ai", "litellm-vertex", "google-vertex", "googlevertex"]
)
def test_every_vertex_alias_is_preflighted(self, no_sdk, backend):
with pytest.raises(BackendUnavailableError) as excinfo:
self._create(backend)
assert "headroom-ai[proxy,vertex]" in str(excinfo.value)
def test_other_providers_are_unaffected(self, no_sdk):
"""The preflight must not become a general-purpose backend gate."""
self._create("litellm-openrouter")
def test_injected_backend_class_bypasses_preflight(self, no_sdk):
sentinel = object()
assert self._create("vertex", litellm_backend_cls=lambda **_: sentinel) is sentinel
def test_present_sdk_passes(self, monkeypatch):
monkeypatch.setattr(
"headroom.providers.vertex.diagnostics.importlib.util.find_spec",
lambda name: object(),
)
assert self._create("vertex") is not None
def test_cli_prints_remedy_and_exits_2(self):
click_testing = pytest.importorskip("click.testing")
from headroom.cli.main import main
def unavailable(config, **kwargs):
raise BackendUnavailableError("Vertex SDK is missing. remedy-text")
with patch("headroom.proxy.server.run_server", unavailable):
result = click_testing.CliRunner().invoke(main, ["proxy"])
assert result.exit_code == 2
assert "Cannot start proxy: Vertex SDK is missing. remedy-text" in result.output
assert "Traceback" not in result.output
def test_cli_multi_worker_fails_in_parent_before_serving(self, no_sdk):
"""--workers >1 builds the app inside each worker, past the CLI's handler."""
click_testing = pytest.importorskip("click.testing")
from headroom.cli.main import main
with patch("headroom.proxy.server.run_server") as run_server:
result = click_testing.CliRunner().invoke(
main, ["proxy", "--backend", "vertex", "--workers", "2"]
)
assert result.exit_code == 2
assert "headroom-ai[proxy,vertex]" in result.output
run_server.assert_not_called()
Generated
+268 -1
View File
@@ -1273,6 +1273,203 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/6b/50/2a923c4dc02522ff9567f07c67ed1deca6d5fa9474f4708168b8088dfa37/gitpython-3.2.1-py3-none-any.whl", hash = "sha256:d66b0832aa8755c1a84892937282df0befb16c97e5a64aeebd68be9c5e323f53", size = 231548, upload-time = "2026-10-14T12:20:18.844Z" },
]
[[package]]
name = "google-api-core"
version = "2.25.2"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "google-auth" },
{ name = "googleapis-common-protos" },
{ name = "proto-plus" },
{ name = "protobuf" },
{ name = "requests" },
]
sdist = { url = "https://files.pythonhosted.org/packages/09/cd/63f1557235c2440fe0577acdbc32577c5c002684c58c7f4d770a92366a24/google_api_core-2.25.2.tar.gz", hash = "sha256:1c63aa6af0d0d5e37966f157a77f9396d820fba59f9e43e9415bc3dc5baff300", size = 166266, upload-time = "2025-10-03T00:07:34.778Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c8/d8/894716a5423933f5c8d2d5f04b16f052a515f78e815dab0c2c6f1fd105dc/google_api_core-2.25.2-py3-none-any.whl", hash = "sha256:e9a8f62d363dc8424a8497f4c2a47d6bcda6c16514c935629c257ab5d10210e7", size = 162489, upload-time = "2025-10-03T00:07:32.924Z" },
]
[package.optional-dependencies]
grpc = [
{ name = "grpcio" },
{ name = "grpcio-status" },
]
[[package]]
name = "google-auth"
version = "2.62.0"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "cryptography" },
{ name = "pyasn1-modules" },
]
sdist = { url = "https://files.pythonhosted.org/packages/f5/b9/4b2528f30114b106e3c3a7298ba38d85e682fd1f83fe7c4a5b77c7677082/google_auth-2.62.0.tar.gz", hash = "sha256:0bef0ce54bdf9ce226c5d66e4264413bd918141c31bbe49fb52eac882f513d69", size = 398180, upload-time = "2026-10-12T19:20:48.328Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/eb/a3/87fe9b7203ec2e56ea6a57c75e6d632eaf9487c7f7978cdad1d801bee0af/google_auth-2.62.0-py3-none-any.whl", hash = "sha256:4ff4319aeb4ad128409759d397a9fcafad126d0031d241cc0dd6b9a00b43e3f3", size = 267911, upload-time = "2026-10-12T19:20:46.355Z" },
]
[package.optional-dependencies]
pyopenssl = [
{ name = "cryptography" },
]
requests = [
{ name = "requests" },
]
[[package]]
name = "google-cloud-aiplatform"
version = "1.148.1"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "docstring-parser" },
{ name = "google-api-core", extra = ["grpc"] },
{ name = "google-auth" },
{ name = "google-cloud-bigquery" },
{ name = "google-cloud-resource-manager" },
{ name = "google-cloud-storage" },
{ name = "google-genai" },
{ name = "packaging" },
{ name = "proto-plus" },
{ name = "protobuf" },
{ name = "pydantic" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/c9/f3/b2a9417014c93858a2e3266134f931eefd972c2d410b25d7b8782fc6f143/google_cloud_aiplatform-1.148.1.tar.gz", hash = "sha256:75d605fba34e68714bd08e1e482755d0a6e3ae972805f809d088e686c30879e7", size = 10278758, upload-time = "2026-04-17T23:45:26.738Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/56/5b/e3515d7bbba602c2b0f6a0da5431785e897252443682e4735d0e6873dc8f/google_cloud_aiplatform-1.148.1-py2.py3-none-any.whl", hash = "sha256:035101e2d8e65c6a706cc3930b2452de7ddcbde50dd130320fcea0d8b03b0c5a", size = 8434481, upload-time = "2026-04-17T23:45:22.919Z" },
]
[[package]]
name = "google-cloud-bigquery"
version = "3.42.3"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "google-api-core", extra = ["grpc"] },
{ name = "google-auth", extra = ["pyopenssl"] },
{ name = "google-cloud-core" },
{ name = "google-resumable-media" },
{ name = "packaging" },
{ name = "python-dateutil" },
{ name = "requests" },
]
sdist = { url = "https://files.pythonhosted.org/packages/0a/53/6a9c19cde15ffe3f218653e4f11d08b2ef97dad78c07c473bc95f8ce7aa9/google_cloud_bigquery-3.42.3.tar.gz", hash = "sha256:d03f8da5ed94aeae5457f3127216cb385392ba266bede25ea257aeec94512900", size = 518359, upload-time = "2026-07-30T18:15:19.314Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/0a/fe/a862130426b56c062dcbc2fcb5a9e4bec9fea9193821718d718f9f10be61/google_cloud_bigquery-3.42.3-py3-none-any.whl", hash = "sha256:81b9bfa3a5fa098a04351c1a12579d16f28b93b836e4556ef6410a01deebf418", size = 264652, upload-time = "2026-07-30T18:15:17.549Z" },
]
[[package]]
name = "google-cloud-core"
version = "2.6.0"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "google-api-core" },
{ name = "google-auth" },
]
sdist = { url = "https://files.pythonhosted.org/packages/a8/dd/1eef226e470369b26824a505c34482c0b493bc35fe8e0c6b003b5feca21a/google_cloud_core-2.6.0.tar.gz", hash = "sha256:e76149739f90fac1fc6757c09f47eaccb3145b54adbd7759b0f7c4b235f46c83", size = 36001, upload-time = "2026-05-07T08:04:04.124Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/84/4a/98da8930ab109c73d9a5d13782a9ebb81ea8c111f6d534a567b71d23e52b/google_cloud_core-2.6.0-py3-none-any.whl", hash = "sha256:6d63ac8e5eca6d9e4319d0a1e2265fadcd7f1049904378caecfa01cf52dd869e", size = 29390, upload-time = "2026-05-07T08:02:34.672Z" },
]
[[package]]
name = "google-cloud-resource-manager"
version = "1.18.0"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "google-api-core", extra = ["grpc"] },
{ name = "google-auth" },
{ name = "grpc-google-iam-v1" },
{ name = "grpcio" },
{ name = "proto-plus" },
{ name = "protobuf" },
]
sdist = { url = "https://files.pythonhosted.org/packages/12/70/d467d93528905700826d26a48ad16ac39d4b36dd6efb5a86fb338efa81aa/google_cloud_resource_manager-1.18.0.tar.gz", hash = "sha256:db689f800a14c66d041196a7fbb8bb8aae8dc87f28c2929e101a5ec766b15512", size = 463674, upload-time = "2026-06-22T23:22:27.09Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/3f/62/2dbecda08a32aa9d4e6909eba3e31731c2d4452687bb6364a8d7950aed74/google_cloud_resource_manager-1.18.0-py3-none-any.whl", hash = "sha256:69bab144acd75878ebe44b720903dc3d140cb7d3be3261eaa8bc81e48afaff33", size = 404211, upload-time = "2026-06-22T23:20:25.88Z" },
]
[[package]]
name = "google-cloud-storage"
version = "3.4.1"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "google-api-core" },
{ name = "google-auth" },
{ name = "google-cloud-core" },
{ name = "google-crc32c" },
{ name = "google-resumable-media" },
{ name = "requests" },
]
sdist = { url = "https://files.pythonhosted.org/packages/bd/ef/7cefdca67a6c8b3af0ec38612f9e78e5a9f6179dd91352772ae1a9849246/google_cloud_storage-3.4.1.tar.gz", hash = "sha256:6f041a297e23a4b485fad8c305a7a6e6831855c208bcbe74d00332a909f82268", size = 17238203, upload-time = "2025-10-08T18:43:39.665Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/83/6e/b47d83d3a35231c6232566341b0355cce78fd4e6988a7343725408547b2c/google_cloud_storage-3.4.1-py3-none-any.whl", hash = "sha256:972764cc0392aa097be8f49a5354e22eb47c3f62370067fb1571ffff4a1c1189", size = 290142, upload-time = "2025-10-08T18:43:37.524Z" },
]
[[package]]
name = "google-crc32c"
version = "1.9.0"
source = { registry = "https://pypi.org/simple/" }
sdist = { url = "https://files.pythonhosted.org/packages/fa/25/9cb0c1c31c45b893eb8f11ae70b3f4309432d59b5acaebca5dbe791729a4/google_crc32c-1.9.0.tar.gz", hash = "sha256:7b8c84c3d159ab6817fe3f74e6e6cef099c3f95dcec3abc0d8afb1404642efbe", size = 14857, upload-time = "2026-09-24T21:39:32.067Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/0e/55/a2f07f15e624f0de79359b1a6c1deb59ec5061bd3b38744b3b2849400662/google_crc32c-1.9.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:457d0d9a4718fd52b1494eac5c200ad25beeadbdc91843d550a003910838589f", size = 31958, upload-time = "2026-09-24T21:19:00.994Z" },
{ url = "https://files.pythonhosted.org/packages/f8/b3/923743597b774bbcf12a7c3e00e48d745e15fd616ad7489a40a63fff8f2f/google_crc32c-1.9.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:ccfe40021fd6afe23361175cf7551e3cef5fd34dc1ebe319f14993a83579e0eb", size = 31805, upload-time = "2026-09-24T21:22:25.019Z" },
{ url = "https://files.pythonhosted.org/packages/df/a6/4d0352fe889663e0d81cea7fc664ec9158727384de4a44ab10e9967a7682/google_crc32c-1.9.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fbef61a3794e011c65fb4396a196cf123a7f474fe5a443db8e5dd7d751b9e6d4", size = 38119, upload-time = "2026-09-24T21:38:06.634Z" },
{ url = "https://files.pythonhosted.org/packages/aa/e3/26685384e4b66ff0928d9566ef6110a7df76029175a1842329d7e3515f10/google_crc32c-1.9.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:86764b99e7a607830d93cb5b75e0ec3ff6cb06d3c274624418473cee701900d4", size = 36677, upload-time = "2026-09-24T21:38:08.082Z" },
{ url = "https://files.pythonhosted.org/packages/cb/ce/4e90102e84880e97d3cf935f2672ecd29191bdeacf57f01740f92debda00/google_crc32c-1.9.0-cp311-cp311-win_amd64.whl", hash = "sha256:43a2dc26f9be213fbe0b4fc4a1088c5d45cbfcb3247420ccc820f0fc3edeea86", size = 35096, upload-time = "2026-09-24T21:39:28.201Z" },
{ url = "https://files.pythonhosted.org/packages/e4/5d/0730e1b3a14d054d1466f2fec88dadf978509c749a3d96d8b069cc56d38a/google_crc32c-1.9.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:53fdafef58e230d0c946ab5f8446d123d9f548230a73b29c8b41c9546f268bc1", size = 31961, upload-time = "2026-09-24T21:19:01.724Z" },
{ url = "https://files.pythonhosted.org/packages/dd/32/d085abaf2fd907121975b92245bb3480fb8be40c37d03f9d6c41857f84c3/google_crc32c-1.9.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:8b91f41645b15a720357183fa5716682ada441873e3c462c15f9714be36f146b", size = 31804, upload-time = "2026-09-24T21:22:25.81Z" },
{ url = "https://files.pythonhosted.org/packages/94/78/dd1935432337e5da7af391a6fc9f161c1c8e9b9002a402b9190135fe1b59/google_crc32c-1.9.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:16865b477d7941712cb0e0aad8ad4815e984fb5fc16d3fdaef7d986e26e53c95", size = 37775, upload-time = "2026-09-24T21:38:09.249Z" },
{ url = "https://files.pythonhosted.org/packages/9e/43/9db03635bb10188d93dcbab9baa2a8670a0da4e868b4370cdbd98d65fed8/google_crc32c-1.9.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3abb18297d9ef0ab120531838be0e6d68c9fa876570e11c229c48f2edac23ce7", size = 36347, upload-time = "2026-09-24T21:38:10.141Z" },
{ url = "https://files.pythonhosted.org/packages/cf/eb/94dee516c846bd9382c3f566d8f8e5fb9e90599e45afeb697f9fc2533528/google_crc32c-1.9.0-cp312-cp312-win_amd64.whl", hash = "sha256:fb63a8d7fa2e95dcff1ca16af2f4d88b526fa5ff72d1696285884ac2d49b6963", size = 35097, upload-time = "2026-09-24T21:39:28.934Z" },
{ url = "https://files.pythonhosted.org/packages/3f/34/cb484e8b6174f130f8c6dc79c733a9dd8869b410ad6511fb6104c46b973a/google_crc32c-1.9.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:f1dc17d987ddcc5eba12a7ce48f0eb93141dea236b170c1101151396edf2f0cf", size = 31960, upload-time = "2026-09-24T21:19:02.454Z" },
{ url = "https://files.pythonhosted.org/packages/af/25/3e8e567bd48448e225ea27318ccf2b94e05124e7b8b97b13eaec9e127199/google_crc32c-1.9.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:f894a2877650b56201d26a012a257b76d54a68834dc3913a93830ca8a047b075", size = 31803, upload-time = "2026-09-24T21:22:27.008Z" },
{ url = "https://files.pythonhosted.org/packages/f0/18/bee0dd59ae622482dc6463636c79e4bde7c954d061c859c9256362c9931a/google_crc32c-1.9.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:4488f1553a9ab7e86cdedc833374a7e904031803b995dc0bd0be48c271fa6556", size = 37776, upload-time = "2026-09-24T21:38:11.056Z" },
{ url = "https://files.pythonhosted.org/packages/fd/b6/e76e80fed5f2558273c7839e622f98095c9b36c719c7147e38e3c055cb70/google_crc32c-1.9.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0568b17ed90ac596f29400d99e243fd0cc6276766183def888d1bf8d1dc13827", size = 36350, upload-time = "2026-09-24T21:38:12.138Z" },
{ url = "https://files.pythonhosted.org/packages/87/34/165542bfa99dfef91a76471cc48cce74b8ff4e295722896087ab2b8e8611/google_crc32c-1.9.0-cp313-cp313-win_amd64.whl", hash = "sha256:8583ec21d56b565d68ab2963cc7e21b3b271247c29b04286068255ef65f221bd", size = 35090, upload-time = "2026-09-24T21:39:29.764Z" },
{ url = "https://files.pythonhosted.org/packages/8f/eb/43ea41f4061a1cad87b2b6559c98e960e45bf551fe66f83d833b98aaf0c9/google_crc32c-1.9.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:6a3b2c8a343c570ed8100a7627c20badfd92c6caa2067093a86be45af27f5b1b", size = 31963, upload-time = "2026-09-24T21:19:03.208Z" },
{ url = "https://files.pythonhosted.org/packages/45/d2/a968c0c29ccd2b0c980ff4f9e3f7035cee28c23a1c57541825cc8221858c/google_crc32c-1.9.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:13179f7e3282617923e957b8e54b8f9c3968030f48640a9f47fd7c5c38c4a215", size = 31805, upload-time = "2026-09-24T21:22:27.917Z" },
{ url = "https://files.pythonhosted.org/packages/03/73/388e493d6c3e252e37165d22efe5a1361f872a24425391b999822861b23a/google_crc32c-1.9.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:265233aff33d835f5b909584fe36ab29647b598c271b661a300001099109e53e", size = 37925, upload-time = "2026-09-24T21:38:13.32Z" },
{ url = "https://files.pythonhosted.org/packages/98/36/190d32caa363ef25d685f422ed1bbf93ff1140fb22fd4d90f24cec209977/google_crc32c-1.9.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:dee799544cae42a42b17a88e38b59cf2c271051dc001da2117a8ff240ffa0548", size = 36495, upload-time = "2026-09-24T21:38:14.211Z" },
{ url = "https://files.pythonhosted.org/packages/d3/fd/81cefea6adae7bd92abb23d4567d199f6485a20ec0a305ca5fa04c52b9c5/google_crc32c-1.9.0-cp314-cp314-win_amd64.whl", hash = "sha256:af73200fa9791ccd380f3598235dba8d82b8af0905df045b3dc60b59836e8ddd", size = 35535, upload-time = "2026-09-24T21:39:30.52Z" },
{ url = "https://files.pythonhosted.org/packages/c5/18/19d4f17f3f33f8fdffcb3e1e69219d6f7ec2c359c160867b04dac1d0a64d/google_crc32c-1.9.0-cp315-cp315-macosx_12_0_arm64.whl", hash = "sha256:e6e8be8a94436079cb5340f6d495d9d7ba30124d8b952703994c739c7c06e236", size = 31977, upload-time = "2026-09-24T21:19:03.976Z" },
{ url = "https://files.pythonhosted.org/packages/81/b4/8010372c4b46f2ee2352dfdb630c397570cd85522a315df024ad2f9459aa/google_crc32c-1.9.0-cp315-cp315-macosx_12_0_x86_64.whl", hash = "sha256:f2b64641bca27497b986b9d87883014035aa904cb4fa333407c6752b3afee9ba", size = 31809, upload-time = "2026-09-24T21:22:29.1Z" },
{ url = "https://files.pythonhosted.org/packages/c5/f8/7e33845d6b90ce1cf37cfabf25cb859277c7d3533ef1b6b1e1ca58581549/google_crc32c-1.9.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f97c3806dcea41c29c04965347b0e12481561b75e0045dc7a4f69d75dec5d9b1", size = 38359, upload-time = "2026-09-24T21:38:14.983Z" },
{ url = "https://files.pythonhosted.org/packages/36/ff/556b2423f449a7515af6b8222a4d7833cbe09ff3e8d2f0b80471f5f6d02e/google_crc32c-1.9.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0abe7e202c25909869c35672ab0f2fe748a7acf276eb78577332a7c38999740f", size = 36858, upload-time = "2026-09-24T21:38:15.799Z" },
{ url = "https://files.pythonhosted.org/packages/40/71/4733f1b7c921d04a2bb9b9916cf66498bf7ad0860a06289413830da83192/google_crc32c-1.9.0-cp315-cp315-win_amd64.whl", hash = "sha256:5695c8b9327e040b2aba12c6659b0acb5995314ef0af0192da66e662e011103b", size = 35534, upload-time = "2026-09-24T21:39:31.337Z" },
]
[[package]]
name = "google-genai"
version = "1.75.0"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "anyio" },
{ name = "distro" },
{ name = "google-auth", extra = ["requests"] },
{ name = "httpx" },
{ name = "pydantic" },
{ name = "requests" },
{ name = "sniffio" },
{ name = "tenacity" },
{ name = "typing-extensions" },
{ name = "websockets" },
]
sdist = { url = "https://files.pythonhosted.org/packages/9d/59/3ed61240ef20b3ae6ed54e82c6f8b6d1f194947bc6679679dd6cdb037594/google_genai-1.75.0.tar.gz", hash = "sha256:56bac3991b311c93f980c0a2abcd287b672146905df1fbd71c92ed633d5a07cf", size = 539039, upload-time = "2026-05-04T22:48:54.857Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/2d/b6/552d40e96da22921eb1fead7c14b00b5b5473a20e45959488660fab35ee2/google_genai-1.75.0-py3-none-any.whl", hash = "sha256:8dc4c096e7d6288c3087f6893f582fe52468932464781edb8193bd92b9fefb2c", size = 793726, upload-time = "2026-05-04T22:48:53.033Z" },
]
[[package]]
name = "google-resumable-media"
version = "2.11.0"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "google-crc32c" },
]
sdist = { url = "https://files.pythonhosted.org/packages/cf/64/df6a482d5aa39d7f7be186d892377d605cae6c88525fd851d456e8bbe9c9/google_resumable_media-2.11.0.tar.gz", hash = "sha256:febd83686752799661b4de575f0b993c5c25c349a5362556fc4d7be164056a37", size = 2164914, upload-time = "2026-09-29T19:26:13.546Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/5e/2e/4f0a152f2e576e496f31ba1c3c62ed174a8878d06008916a7edc58b1bb28/google_resumable_media-2.11.0-py3-none-any.whl", hash = "sha256:f43d15e6a7f818f762eaead0f369c551f8275a4179c9d6225d0d259f49b87b5d", size = 81515, upload-time = "2026-09-29T19:25:47.31Z" },
]
[[package]]
name = "googleapis-common-protos"
version = "1.74.0"
@@ -1285,6 +1482,11 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/b6/b0/be5d3329badb9230b765de6eea66b73abd5944bdeb5afb3562ddcd80ae84/googleapis_common_protos-1.74.0-py3-none-any.whl", hash = "sha256:702216f78610bb510e3f12ac3cafd281b7ac45cc5d86e90ad87e4d301a3426b5", size = 300743, upload-time = "2026-04-02T21:22:49.108Z" },
]
[package.optional-dependencies]
grpc = [
{ name = "grpcio" },
]
[[package]]
name = "greenlet"
version = "3.4.0"
@@ -1332,6 +1534,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/db/72/85ae954d734703ab48e622c59d4ce35d77ce840c265814af9c078cacc7aa/greenlet-3.4.0-cp314-cp314t-win_amd64.whl", hash = "sha256:1a4a48f24681300c640f143ba7c404270e1ebbbcf34331d7104a4ff40f8ea705", size = 245554, upload-time = "2026-04-08T17:03:50.044Z" },
]
[[package]]
name = "grpc-google-iam-v1"
version = "0.14.4"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "googleapis-common-protos", extra = ["grpc"] },
{ name = "grpcio" },
{ name = "protobuf" },
]
sdist = { url = "https://files.pythonhosted.org/packages/44/4f/d098419ad0bfc06c9ce440575f05aa22d8973b6c276e86ac7890093d3c37/grpc_google_iam_v1-0.14.4.tar.gz", hash = "sha256:392b3796947ed6334e61171d9ab06bf7eb357f554e5fc7556ad7aab6d0e17038", size = 23706, upload-time = "2026-04-01T01:57:49.813Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/89/22/c2dd50c09bf679bd38173656cd4402d2511e563b33bc88f90009cf50613c/grpc_google_iam_v1-0.14.4-py3-none-any.whl", hash = "sha256:412facc320fcbd94034b4df3d557662051d4d8adfa86e0ddb4dca70a3f739964", size = 32675, upload-time = "2026-04-01T01:57:47.69Z" },
]
[[package]]
name = "grpcio"
version = "1.80.0"
@@ -1383,6 +1599,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/e5/8c/bbe6baf2557262834f2070cf668515fa308b2d38a4bbf771f8f7872a7036/grpcio-1.80.0-cp314-cp314-win_amd64.whl", hash = "sha256:3b01e1f5464c583d2f567b2e46ff0d516ef979978f72091fd81f5ab7fa6e2e7f", size = 5019457, upload-time = "2026-03-30T08:48:37.308Z" },
]
[[package]]
name = "grpcio-status"
version = "1.71.2"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "googleapis-common-protos" },
{ name = "grpcio" },
{ name = "protobuf" },
]
sdist = { url = "https://files.pythonhosted.org/packages/fd/d1/b6e9877fedae3add1afdeae1f89d1927d296da9cf977eca0eb08fb8a460e/grpcio_status-1.71.2.tar.gz", hash = "sha256:c7a97e176df71cdc2c179cd1847d7fc86cca5832ad12e9798d7fed6b7a1aab50", size = 13677, upload-time = "2025-06-28T04:24:05.426Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/67/58/317b0134129b556a93a3b0afe00ee675b5657f0155509e22fcb853bafe2d/grpcio_status-1.71.2-py3-none-any.whl", hash = "sha256:803c98cb6a8b7dc6dbb785b1111aed739f241ab5e9da0bba96888aa74704cfd3", size = 14424, upload-time = "2025-06-28T04:23:42.136Z" },
]
[[package]]
name = "gunicorn"
version = "26.0.0"
@@ -1682,6 +1912,9 @@ strands = [
vector = [
{ name = "hnswlib" },
]
vertex = [
{ name = "google-cloud-aiplatform" },
]
voice = [
{ name = "onnxruntime", version = "1.23.2", source = { registry = "https://pypi.org/simple/" }, marker = "python_full_version < '3.14' and platform_machine == 'x86_64' and sys_platform == 'darwin'" },
{ name = "onnxruntime", version = "1.26.0", source = { registry = "https://pypi.org/simple/" }, marker = "platform_machine != 'x86_64' or sys_platform != 'darwin'" },
@@ -1717,6 +1950,7 @@ requires-dist = [
{ name = "fastapi", marker = "extra == 'proxy'", specifier = ">=0.100.0" },
{ name = "fastembed", marker = "(python_full_version < '3.14' and extra == 'relevance') or (platform_machine != 'x86_64' and extra == 'relevance') or (sys_platform != 'darwin' and extra == 'relevance')", specifier = ">=0.6.0" },
{ name = "filelock", specifier = ">=4.2.1" },
{ name = "google-cloud-aiplatform", marker = "extra == 'vertex'", specifier = ">=1.38" },
{ name = "gunicorn", marker = "sys_platform != 'win32' and extra == 'proxy-prod'", specifier = ">=21.0.0" },
{ name = "headroom-ai", extras = ["lean"], marker = "extra == 'sandbox'" },
{ name = "headroom-ai", extras = ["proxy"], marker = "extra == 'proxy-prod'" },
@@ -1814,7 +2048,7 @@ requires-dist = [
{ name = "xlwt", marker = "extra == 'dev'", specifier = ">=1.3.0" },
{ name = "zstandard", marker = "extra == 'proxy'", specifier = ">=0.20.0" },
]
provides-extras = ["proxy", "proxy-prod", "code", "ml", "memory", "vector", "memory-stack", "pytorch-mps", "relevance", "image", "reports", "spreadsheet", "otel", "anyllm", "langchain", "langgraph", "agno", "strands", "autogen", "mcp", "voice", "voice-train", "evals", "bedrock", "redis", "html", "dev", "all", "lean", "sandbox"]
provides-extras = ["proxy", "proxy-prod", "vertex", "code", "ml", "memory", "vector", "memory-stack", "pytorch-mps", "relevance", "image", "reports", "spreadsheet", "otel", "anyllm", "langchain", "langgraph", "agno", "strands", "autogen", "mcp", "voice", "voice-train", "evals", "bedrock", "redis", "html", "dev", "all", "lean", "sandbox"]
[[package]]
name = "hf-xet"
@@ -4150,6 +4384,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/5b/5a/bc7b4a4ef808fa59a816c17b20c4bef6884daebbdf627ff2a161da67da19/propcache-0.4.1-py3-none-any.whl", hash = "sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237", size = 13305, upload-time = "2025-10-08T19:49:00.792Z" },
]
[[package]]
name = "proto-plus"
version = "1.28.2"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "protobuf" },
]
sdist = { url = "https://files.pythonhosted.org/packages/73/3e/29e0d6a2c5adde6ab5772253fd16ab346324026b89a66e354689c86d0584/proto_plus-1.28.2.tar.gz", hash = "sha256:26d843eb99c1e32fdf1d20ff0faae56607f7748fe774acf9ecd5cfe6c6472501", size = 58063, upload-time = "2026-07-22T16:28:29.119Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/9d/84/4e9a53a062d4073c74897a6bd20fff74d55307341b3e85c081002462b3ef/proto_plus-1.28.2-py3-none-any.whl", hash = "sha256:b874236fcac2358f601e4330bcb76cb8b89c851303ccf4078408b3d4774d1c52", size = 50693, upload-time = "2026-07-22T16:28:24.059Z" },
]
[[package]]
name = "protobuf"
version = "5.29.6"
@@ -4280,6 +4526,27 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/50/f2/c0e76a0b451ffdf0cf788932e182758eb7558953f4f27f1aff8e2518b653/pyarrow-23.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:527e8d899f14bd15b740cd5a54ad56b7f98044955373a17179d5956ddb93d9ce", size = 28365807, upload-time = "2026-02-16T10:14:03.892Z" },
]
[[package]]
name = "pyasn1"
version = "0.6.5"
source = { registry = "https://pypi.org/simple/" }
sdist = { url = "https://files.pythonhosted.org/packages/88/7e/3c0da41061a2021cac7cd524fd57092adbe5b515e8bf9005702e366802e2/pyasn1-0.6.5.tar.gz", hash = "sha256:6b11616a6bcf3a40e635f082bb232fa57b5e0dd7f517bd7ddd69ec3794508a47", size = 155152, upload-time = "2026-10-27T06:32:49.125Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/19/1f/3eefd5c84c35cae44549ce1c7f7e58f7945b34f5a6bce07a135438522073/pyasn1-0.6.5-py3-none-any.whl", hash = "sha256:4017b0fce125096b8573b39c989fdb1a4b19cf761e683398184550fea0e66587", size = 84807, upload-time = "2026-10-27T06:32:48.045Z" },
]
[[package]]
name = "pyasn1-modules"
version = "0.4.2"
source = { registry = "https://pypi.org/simple/" }
dependencies = [
{ name = "pyasn1" },
]
sdist = { url = "https://files.pythonhosted.org/packages/e9/e6/78ebbb10a8c8e4b61a59249394a4a594c1a7af95593dc933a349c8d00964/pyasn1_modules-0.4.2.tar.gz", hash = "sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6", size = 307892, upload-time = "2025-03-28T02:41:22.17Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/47/8d/d529b5d697919ba8c11ad626e835d4039be708a35b0d22de83a269a6682c/pyasn1_modules-0.4.2-py3-none-any.whl", hash = "sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a", size = 181259, upload-time = "2025-03-28T02:41:19.028Z" },
]
[[package]]
name = "pyclipper"
version = "1.4.0"
+78 -11
View File
@@ -9,11 +9,13 @@ Google documents Gemini generation on Vertex with `generateContent` and
shape. See Google Cloud's model inference reference:
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference
Google Cloud REST calls authenticate with a bearer access token. For local
development, Google documents both `gcloud auth print-access-token` and
`gcloud auth application-default print-access-token`; Application Default
Credentials search `GOOGLE_APPLICATION_CREDENTIALS`, local ADC files, and
attached service accounts in that order. See:
Google Cloud REST calls authenticate with a bearer access token. Use
**`gcloud auth application-default print-access-token`**: a plain
`gcloud auth print-access-token` user token is rejected by
`aiplatform.googleapis.com` with `401 ACCESS_TOKEN_TYPE_UNSUPPORTED` for many
identities. Application Default Credentials search
`GOOGLE_APPLICATION_CREDENTIALS`, local ADC files, and attached service accounts
in that order. Tokens expire after ~1 hour. See:
- https://docs.cloud.google.com/docs/authentication/rest
- https://docs.cloud.google.com/docs/authentication/application-default-credentials
@@ -26,19 +28,46 @@ Set the Vertex regional host explicitly:
headroom proxy --vertex-api-url https://us-central1-aiplatform.googleapis.com
```
The same setting is available through `VERTEX_TARGET_API_URL`.
The same setting is available through `VERTEX_TARGET_API_URL`. Left unset, the
proxy derives the host from each request's `locations/{location}` segment, so
one proxy serves every region plus `global`.
### Picking a location
Vertex serves each publisher model in only a subset of locations, and this is
the most common source of a confusing `404` during onboarding:
| Model | Where it serves |
| --- | --- |
| `gemini-flash-latest`, `gemini-flash-lite-latest` | `global` only |
| `gemini-3.5-flash` | `global`, `europe-west2`, `asia-northeast1`, `asia-south1`, `asia-southeast1` |
| `claude-sonnet-4-6` (and 4.6-or-older Claude) | `global`, `us-east5`, `europe-west1`, `asia-southeast1` |
| Claude 4.7+ | `global` or the `us`/`eu` multi-region only -- no named regions |
**Gemini 3.x has no US regional endpoint.** Start with `global` and only pin a
region when you need Provisioned Throughput or data residency. Partner models
(Claude, Llama, Mistral) also need a one-time per-project enable in Model
Garden before they serve.
When Vertex rejects a request with `401`, `403`, `404` or `429`, Headroom appends a `[headroom] hint: ...` note to
`error.message` (also emitted as an `x-headroom-hint` header and a proxy WARNING)
naming the likely fix. Hints are fixed text and never echo the request path,
project, credentials or exception detail; raw provider errors stay in the proxy
log.
## Gemini On Vertex
Send Vertex publisher paths through the proxy unchanged:
```bash
ACCESS_TOKEN="$(gcloud auth print-access-token)"
LOCATION="global"
MODEL="gemini-flash-latest"
ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
curl -sS \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Content-Type: application/json" \
http://127.0.0.1:8787/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-2.0-flash:generateContent \
http://127.0.0.1:8787/v1/projects/PROJECT_ID/locations/${LOCATION}/publishers/google/models/${MODEL}:generateContent \
-d '{
"contents": [
{
@@ -49,6 +78,17 @@ curl -sS \
}'
```
## Google Gen AI SDK (Proxy)
Point the `google-genai` SDK at Headroom with
`genai.Client(vertexai=True, project=..., location="global", http_options={"base_url": "http://127.0.0.1:8787"})`.
`examples/vertex_genai_sdk_demo.py` runs this end to end: it starts the proxy,
probes it (including a thinking config) and tears it down.
Note the `google-genai` SDK only builds `publishers/google/...` paths, so it
cannot reach Claude on Vertex; use the `publishers/anthropic/...:rawPredict`
route below for that.
Supported passthrough actions:
- `generateContent`
@@ -66,6 +106,32 @@ The Python proxy preserves caller-supplied Google bearer auth. The native Rust
proxy path additionally resolves GCP ADC and injects the bearer token for the
Anthropic publisher route.
These native routes are a straight passthrough: they need **no** `--backend`
flag and no extra beyond `[proxy]`.
`--backend vertex` (aliases: `vertex_ai`, `litellm-vertex`, `google-vertex`) is a
different mode. It routes Anthropic *Messages* traffic — `/v1/messages` — through
LiteLLM, and needs the Vertex SDK:
```bash
pip install "headroom-ai[proxy,vertex]"
```
That extra is kept out of `[proxy]` because `google-cloud-aiplatform` and its
transitive tree add roughly 175 MB, and only this backend uses it. Select the
backend without it and the proxy refuses to start rather than failing on every
request:
```text
Cannot start proxy: Vertex backend selected but the Vertex SDK is missing. ...
```
Do not combine the two. With `--backend vertex` set, Claude (`publishers/anthropic`) requests
are re-routed through LiteLLM, which takes the project, region and model from
its own configuration (`VERTEXAI_PROJECT` / `VERTEXAI_LOCATION`) and ignores the
ones in your URL — so a request that works without the flag can come back `404`
for a model you never named.
## Claude Code with Headroom compression (validated)
To run **Claude Code** against Claude-on-Vertex **with Headroom compressing the
@@ -82,6 +148,7 @@ Short version: run Claude Code in **normal Anthropic mode** (`ANTHROPIC_BASE_URL
> URL before sending a request ("model … not available on your vertex deployment"),
> so the proxy is never reached. Use the Anthropic-mode runbook above instead.
>
> ⚠️ Two easy-to-miss requirements: `pip install "google-cloud-aiplatform>=1.38"`
> (LiteLLM `vertex_ai` provider) and the `--code-aware` flag (code compression is
> off by default). Without them you get a 500 or `tokens_saved: 0`.
> ⚠️ Two easy-to-miss requirements: `pip install "headroom-ai[proxy,vertex]"`
> (the LiteLLM `vertex_ai` provider needs the Vertex SDK) and the `--code-aware`
> flag (code compression is off by default). Without the first the proxy refuses
> to start; without the second you get `tokens_saved: 0`.