mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(vertex): explain Vertex failures in-band, correct the model/region matrix (#2640)
## Description Vertex rejects onboarding requests for a handful of recurring reasons — expired ADC, a partner model never enabled in Model Garden, a model not served in the requested location, a missing optional SDK — and its own error text names none of the fixes. Users see a bare `404` or an opaque `500` through their SDK and cannot tell a proxy bug from a project-config gap. This makes the proxy explain those failures where it happens, adds a `[vertex]` extra so the LiteLLM Vertex backend can actually start, and corrects an example, a test matrix, and three documentation claims that could not have worked as written. Closes #2632 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made **Explain Vertex failures in-band** (`headroom/providers/vertex/diagnostics.py`, new) - On `401/403/404/429`, attach an actionable hint three ways: a WARNING in the proxy log, an `x-headroom-hint` response header, and a note appended to `error.message`. The last is load-bearing — SDKs raise with the message string and discard headers. - Successful responses are never modified; annotation is idempotent; statuses we cannot explain get no hint rather than a plausible-sounding wrong one. - Routes opt in via a one-line `@with_vertex_diagnostics` decorator reading path params FastAPI already injects (**+8 lines** in `proxy_routes.py`, no re-indentation). - Same treatment for backend setup failures, where `--backend vertex` surfaced LiteLLM's raw `No module named 'vertexai'` as an opaque 500. **Make `--backend vertex` able to start** (`pyproject.toml`, `registry.py`, `cli/proxy.py`) - New `[vertex]` extra carrying `google-cloud-aiplatform>=1.38`. Deliberately **not** folded into `[proxy]`: measured at ~175 MB installed, and the native passthrough routes never needed it. Follows the existing `proxy-prod` split. - Startup preflight keyed on the **resolved provider**, so every alias normalized at `registry.py:229` (`vertex`, `vertex_ai`, `litellm-vertex`, `google-vertex`, `googlevertex`) is covered. Skipped when a backend class is injected, so existing fake-based tests are unaffected. - New `BackendUnavailableError` so the CLI prints the remedy and exits `2` instead of dumping a traceback through startup internals. **Correct things that could not have worked** | | was | is | |---|---|---| | SDK client | `http_options={"api_endpoint": ...}` | `{"base_url": "http://..."}` — `api_endpoint` is not a field on `HttpOptions`; pydantic rejects it before any request is sent | | Thinking config | `ThinkingConfig(thinking_budget_tokens=)` | `thinking_budget=` | | Request payload | `contents: [{parts: [...]}]` | adds `role: "user"` — the `global` endpoint enforces it, regional was silently lenient | | Auth guidance | `gcloud auth print-access-token` | `application-default` — the plain user token gets `401 ACCESS_TOKEN_TYPE_UNSUPPORTED` | | Test matrix | model × location cross-product | explicit servable pairs | - Test matrix is now four pairs that can actually serve: `global`/`gemini-flash-latest`, `global`/`claude-sonnet-4-6`, `europe-west2`/`gemini-3.5-flash`, `us-east5`/`claude-sonnet-4-6`. Gemini 3.x has no US regional endpoint and `-latest` aliases are global-only, so a pinned `us-central1` Gemini case can never pass. No Gemini 2.5 — it retires 2026-10-16. - `test_thinking_levels` asserted only `status_code < 500` while sending an ignored `thinkingBudgetTokens` key: it passed on 400s and proved nothing. Now asserts 200 plus scored usage metadata with the correct `thinkingBudget` spelling. - Cases the caller's project genuinely cannot serve now **skip** with an actionable reason instead of reporting a red proxy. **Docs** (`proxy.mdx`, `installation.mdx`, `wiki/vertex.md`) - `proxy.mdx` claimed `publisher=anthropic` passthrough "routes through the same LiteLLM-Vertex path as `--backend vertex_ai`". It does not — without the flag it is a direct forward, which is why it works on a bare `[proxy]` install where the LiteLLM path cannot even import. - Documented the trap of combining `--backend vertex` with native Vertex URLs: LiteLLM takes project/region/model from its own config and ignores the URL. - `installation.mdx` and `wiki/vertex.md` said the SDK is "not included in any extra". - Cleared 15 pre-existing `ruff` findings in the two files this PR rewrites (4 remain in `plugins/headroom-oauth2`, untouched by this work). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ mypy headroom Success: no issues found in 511 source files $ ruff check . Found 4 errors. # all in plugins/headroom-oauth2, pre-existing, untouched by this PR $ pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q 44 passed, 1 warning in 69.86s # 8 live integration + 36 offline diagnostics, 0 skipped $ pytest tests/ -k "anthropic_messages or handler_anthropic or backend or registry or cli_proxy or vertex or proxy_routes" 853 passed, 49 skipped, 9345 deselected, 2 warnings in 28.41s ``` New tests in `tests/test_vertex_diagnostics.py` (36, no credentials required): status coverage, per-publisher remedy divergence, idempotency, `content-length` consistency after body rewrite, non-JSON and body-less/streaming responses, every backend alias preflighted, other providers unaffected, injected backend bypasses the preflight. ## Real Behavior Proof - Environment: macOS 14, Python 3.11.15, live GCP project (`alanblount-sandbox`) with `aiplatform.googleapis.com` enabled and Claude enabled in Model Garden; ADC via `gcloud auth application-default login` - Exact command / steps: `export GCP_PROJECT_ID=alanblount-sandbox && export GCP_ACCESS_TOKEN=$(gcloud auth application-default print-access-token) && uv run --extra proxy --with google-genai python3 examples/vertex_genai_sdk_demo.py && uv run --extra proxy --with pytest-asyncio pytest tests/test_proxy_vertex_native_integration.py tests/test_vertex_diagnostics.py -q && headroom proxy --backend vertex` (exits 2, then retried with `[proxy,vertex]` extra installed) - Observed result: demo printed `✓ Standard response received successfully! > 1, 2, 3, 4, 5` and `✓ Thinking response received successfully!`; 44 tests passed (8 live integration, 36 diagnostics), 0 skipped; `--backend vertex` exited 2 with the remedy message; same request with the extra installed returned 200; invalid region produced `[headroom] hint:` in the response body and `x-headroom-hint` header; success responses were byte-clean - Not tested: Windows; `us`/`eu` multi-region endpoints and Claude 4.7+ (project cannot serve them); Provisioned Throughput; Rust proxy path; `--backend vertex` dropping URL project/model on native Vertex routes (documented as known gap in Additional Notes) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy and CLI behavior; terminal output is quoted above. ## Additional Notes - `uv.lock` intentionally not included: the new extra is opt-in and does not change default resolution. - Error-body rewriting is confined to already-failing responses and wrapped so a diagnostic can never turn a clean upstream error into a proxy 500. - The preflight raises where the provider is resolved (`registry.py`) rather than at the CLI, so programmatic embedders get the same guarantee; the CLI only formats it. - Three test files fail to collect on this branch (`test_hermes_passthrough_compression`, `test_observability_metrics`, `test_observability_tracing`). They predate this work and are unrelated, so they were excluded rather than fixed — happy to open a separate issue. - `[vertex]` was chosen over adding the dependency to `[proxy]` after measuring the footprint at ~175 MB. If maintainers prefer it in `[proxy]`, that is a one-line change. ## Runtime Rollout Safety - Rollout-managed feature(s): none; diagnostics annotate already-failing Vertex responses and the backend dependency is opt-in through the vertex extra. - Minimum rollout channel: normal stable release after Vertex diagnostics, route, and backend-path tests pass. - Stable/default behavior changed: successful responses and native passthrough remain unchanged; supported Vertex failures gain an actionable header/message, and selecting the LiteLLM Vertex backend without its SDK now fails once at startup. - Kill switch / disable path: use native Vertex passthrough without the LiteLLM backend, or human-revert the diagnostic/preflight change. - Unsafe override required: none. - Qualification impact: offline diagnostics, native integration, and mocked LiteLLM Vertex request-shaping tests must remain green. - Rollback path: human-revert the PR; no persisted state or migration is involved. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2640" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `5d7071f348d2462f666fd7ac1ceb8ac18050169a`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration headc70c1a497and current author responses, including fixed public error hints, log escaping, body/header consistency, all four backend error sites, and native-route versus LiteLLM SDK dependency behavior. Clean integration with upstream/main2d3ee51846. No new PR-specific source blocker found in reviewed scope; no contributor diagnostic/routing logic rewritten. 232 Vertex diagnostics/native-route/backend/CLI/release/version/MCP tests pass with11 skips. Offline lock validation passes (268 packages); all12 changed Python files pass Ruff check/format, and whitespace passes. These checks cover synthetic error/transport paths and do not establish live cloud regional availability or billing behavior. Earlier author live evidence remains scoped to its stated environment; no fresh cloud request or native build claimed here. Historical shared cached-read fixture failure was main-reproduced and is not a contributor-code blocker. Maintainers handle current exact-head hosted qualification before approval; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `486a716adbf642ae5e43276df4c2dc1e63b901d7`, upstream main `c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`. Clean mechanical sync with upstream mainc01cf07d94. Re-reviewed current author/Devin discussion and public diagnostic/error boundaries, native-versus-SDK route separation and startup preflight. This sync does not alter the PR Vertex diagnostics, backend or route-registry modules.158 focused diagnostics/native-route/backend/streaming/CLI tests pass,11live-environment skips,1deprecation warning. Offline uv lock --check passes;all12 changed Python files pass Ruff lint/format and main-relative whitespace passes. Fixed hints remain separate from raw provider error text; multi-worker preflight and non-Google partner guidance retain resolved regressions. No contributor diagnostic/routing correctness logic rewritten. Tests use existing current-main DLL and synthetic errors/transports; no fresh native build, live regional availability/provider inference or billing qualification claimed. Formal approval awaits exact-head hosted qualification; maintainers handle CI. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `0b75c609f24562c8b6ed2aaeaf58239984b4810c`, upstream main `ce4275491299231bc9d2907833f72f72ea22b8a9`. Requalified after human merges #4667,#4669,#4670 against upstream maince42754912. Earlier approval and publication guards stopped stale candidates before either was posted. Current clean merge retains Vertex diagnostics/native-versus-SDK routing and main enable_kompress checks, lossless cold-prefix disable, and Python token-window boundary behavior.190 combined Vertex/backend/CLI/Kompress-disable/must-keep tests pass,11live-environment skips,1warning. Offline lock validation,12-file Ruff lint/format and main-relative whitespace pass. No contributor diagnostic/routing logic rewritten. Native Rust/Cargo sources are identical to c01 across these merges; existing matching-core DLL supports tests, not a fresh native build. No new live Vertex regional/provider/billing qualification claimed. All prior-main hosted results are superseded for readiness; approval awaits exact checks on this published merge head. Maintainers handle CI; no agent merge. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `1fe62702ce614315679d68d9d126e31c57208c67`, upstream main `00d4f441d3f1d48c463fea57e6c574c2f3671302`. Requalified after human merges #4667,#4669,#4670 against upstream maince42754912. Earlier approval and publication guards stopped stale candidates before either was posted. Current clean merge retains Vertex diagnostics/native-versus-SDK routing and main enable_kompress checks, lossless cold-prefix disable, and Python token-window boundary behavior.190 combined Vertex/backend/CLI/Kompress-disable/must-keep tests pass,11live-environment skips,1warning. Offline lock validation,12-file Ruff lint/format and main-relative whitespace pass. No contributor diagnostic/routing logic rewritten. Native Rust/Cargo sources are identical to c01 across these merges; existing matching-core DLL supports tests, not a fresh native build. No new live Vertex regional/provider/billing qualification claimed. All prior-main hosted results are superseded for readiness; approval awaits exact checks on this published merge head. Maintainers handle CI; no agent merge. Final current-main requalification:07d5add5ba, including human merges #4671 action pinning and #4648 multimodal memory preservation. Clean mechanical merge; Vertex/backend/CLI inputs retain the preceding 190-test qualification. Fresh workflow/release suite: 62 passed. Fresh memory wrapper suite: 9 passed. New wrapper retains list payload parts and extracts textual search inputs; no contributor Vertex logic rewritten. Existing native input provenance remains unchanged. Hosted qualification must run on this final head before approval. Newer human merge #4646 advanced authoritative main to8cdce172eabefore publication. Cleanly integrated quote-aware read detection; reviewed new masking/segmentation overlap. Fresh read-command/content-router suite: 188 passed, 1 optional SOCKS warning. Vertex code remains unchanged by this merge. Final additional human merge #4067 packaging changes advanced main toa8905214d: cleanly integrated explicit torch-extras split, retaining Vertex optional dependencies. Fresh ONNX/optional-dependency contracts: 12 passed. Offline lock check and whitespace pass. No live clean-install or Torch backend claim. Earlier publication candidates were stopped before push by fresh-main guard. Latest human merge #4042 advances authoritative main tocde584d7fe. Clean merge preserves optional LangChain availability handling and CI dependency setup. Fresh binding/compatibility/startup-log/xAI regression suite: 29 passed. New main integration re-reviewed; no contributor Vertex behavior rewritten. Subsequent #4049 main merge00d4f441dadds graceful context lookup/init failure handling. Cleanly integrated with Vertex handler diagnostics intact. Fresh Anthropic pre-upstream/memory native ops/OpenAI Codex routing/WebSocket lifecycle suite: 192 passed, 1 deprecation warning. Reviewed exception handling and transport overlap; no live provider qualification claim. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `6c63811d6de7491ebce9a16f89e197f5f5dd86dd`, upstream main `2b0315b2d4b0d54947050bbdad2de9e89df06378`. Requalified against authoritative main2b0315b2d4b0d54947050bbdad2de9e89df06378 after further human merges. All10 integration patches exactly match00d4f441-to-current-main changes after diff-location normalization; contributor Vertex diagnostics/routing policy unchanged.539combined Vertex/backend/CLI/Kompress-disable/must-keep/binary/doctor/Windows-supervisor/provider/search/text tests pass;13platform/live-environment skips,2warnings. Nine changed Python files pass Ruff lint/format; offline lock and whitespace pass. Existing pr3982 native DLL verified against every tracked native build-input hash and artifact checksum; no fresh native build claimed. Current author/Devin discussion checked; prior fixes remain intact. No live Vertex regional/provider/billing,IPv6 listener or unsupported cloud qualification claimed. No new finding in reviewed scope. Exact-head hosted checks must qualify this published head before formal approval; maintainers handle them. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -298,8 +298,11 @@ These variables configure Headroom at runtime. Set them in your shell, `.env` fi
|
||||
| `VERTEXAI_PROJECT` | GCP project id for the Vertex AI backend (LiteLLM-specific — distinct from `GOOGLE_CLOUD_PROJECT`; set it explicitly to avoid silently billing your ADC default quota project) |
|
||||
| `VERTEXAI_LOCATION` | GCP region for the Vertex AI backend (LiteLLM-specific — distinct from `GOOGLE_CLOUD_LOCATION`) |
|
||||
|
||||
The Vertex AI backend also requires `google-cloud-aiplatform>=1.38`, which is not
|
||||
included in any extra or Docker image — see
|
||||
The Vertex AI backend also requires `google-cloud-aiplatform>=1.38`, which ships
|
||||
in the `[vertex]` extra (`pip install "headroom-ai[proxy,vertex]"`) rather than
|
||||
`[proxy]`, since it adds roughly 175 MB and only that backend needs it. The
|
||||
proxy refuses to start if the backend is selected without it. Vertex's native
|
||||
passthrough routes need neither the extra nor the backend flag — see
|
||||
[Google Vertex AI](/docs/proxy#google-vertex-ai) for setup details.
|
||||
|
||||
### Proxy configuration
|
||||
|
||||
@@ -1032,16 +1032,23 @@ OPENROUTER_API_KEY=sk-or-... headroom proxy --backend openrouter
|
||||
`--backend vertex_ai` delegates to [LiteLLM](https://docs.litellm.ai/docs/providers/vertex),
|
||||
which brings two requirements that are easy to miss:
|
||||
|
||||
**1. Install the Vertex SDK.** `google-cloud-aiplatform` is not included in any
|
||||
Headroom extra (`[proxy]`, `[all]`, …) or Docker image variant, so install it
|
||||
alongside Headroom:
|
||||
**1. Install the Vertex SDK.** `google-cloud-aiplatform` ships in the dedicated
|
||||
`[vertex]` extra, kept out of `[proxy]` because it and its transitive tree add
|
||||
roughly 175 MB and only this backend needs it:
|
||||
|
||||
```bash
|
||||
pip install "headroom-ai[proxy]" "google-cloud-aiplatform>=1.38"
|
||||
pip install "headroom-ai[proxy,vertex]"
|
||||
```
|
||||
|
||||
Without it, the first Vertex request fails with
|
||||
`litellm.BadRequestError: … vertexai import failed … No module named 'vertexai'`.
|
||||
Without it the proxy **refuses to start**, naming the fix and exiting `2`:
|
||||
|
||||
```text
|
||||
Cannot start proxy: Vertex backend selected but the Vertex SDK is missing. ...
|
||||
```
|
||||
|
||||
This is a startup check rather than a per-request failure, because the backend
|
||||
would otherwise construct cleanly and then fail on every call with an opaque
|
||||
`litellm.BadRequestError: … No module named 'vertexai'`.
|
||||
|
||||
**2. Set the LiteLLM project/location variables.** LiteLLM reads the GCP project
|
||||
and region from `VERTEXAI_PROJECT` / `VERTEXAI_LOCATION` — these are **not** the
|
||||
@@ -1093,12 +1100,34 @@ mirror Vertex's native REST shape verbatim — no backend flag needed:
|
||||
```
|
||||
|
||||
Requests with `publisher=google` (Gemini models) go through Headroom's full
|
||||
Gemini optimization handler; `publisher=anthropic` (Claude on Vertex) routes
|
||||
through the same LiteLLM-Vertex path as `--backend vertex_ai`. Any client that
|
||||
Gemini optimization handler; `publisher=anthropic` (Claude on Vertex) is
|
||||
forwarded to Vertex directly, preserving your bearer token. Any client that
|
||||
already speaks the native Vertex REST API can simply point its endpoint at the
|
||||
proxy — this is the mechanism `headroom wrap claude` uses in
|
||||
[Vertex mode](/docs/claude-code-vertex).
|
||||
|
||||
<Callout type="warn">
|
||||
These routes need **neither** `--backend vertex_ai` nor the `[vertex]` extra. Do
|
||||
not combine the two: with the backend flag set, Claude publisher requests are
|
||||
re-routed through LiteLLM, which derives the project, region and model from its
|
||||
own configuration and ignores the ones in your URL — a request that works
|
||||
without the flag can return a `404` for a model you never asked for.
|
||||
</Callout>
|
||||
|
||||
### Reading Vertex errors
|
||||
|
||||
When Vertex rejects a request with `401`, `403`, `404` or `429`, Headroom appends
|
||||
a `[headroom] hint: …` note to `error.message`, sets an `x-headroom-hint`
|
||||
response header, and logs a warning. The hint names the likely cause — expired
|
||||
ADC, a partner model not yet enabled in Model Garden, or a model that is not
|
||||
served in the requested location. Successful responses are never modified.
|
||||
|
||||
Hints are fixed text: they never echo your request path, project, credentials or
|
||||
the underlying exception. With `--backend vertex`, a backend that cannot serve
|
||||
(missing Vertex SDK, no ADC) returns the standard sanitized proxy error plus the
|
||||
same kind of fixed hint; the raw provider detail is only in the proxy log, under
|
||||
the response's `request_id`.
|
||||
|
||||
### Bedrock via a local gateway
|
||||
|
||||
`--backend bedrock` accepts **Anthropic** input (`/v1/messages`) and re-signs to AWS. Some setups are the other way around: the client already speaks **Bedrock** (e.g. Claude Code with `CLAUDE_CODE_USE_BEDROCK=1`, or any AWS SDK pointed at a custom endpoint), sending `POST /model/{id}/invoke` to a local gateway that re-signs and forwards to AWS (LiteLLM, LocalStack, a corporate Bedrock proxy).
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
#!/usr/bin/env python3
|
||||
"""End-to-end demo: google-genai SDK -> Headroom proxy -> Vertex AI (Gemini Enterprise).
|
||||
|
||||
Proves that standard Vertex native configurations route properly through the proxy.
|
||||
|
||||
What this script does
|
||||
---------------------
|
||||
1. Spawns the Headroom proxy as a subprocess. No ``--backend vertex``: the native
|
||||
Vertex routes are a passthrough and need neither that flag nor ``[vertex]``.
|
||||
2. Waits for /readyz.
|
||||
3. Configures standard google-genai SDK with vertexai=True hitting the proxy.
|
||||
4. Sends an inference probe to validate native proxy connectivity.
|
||||
5. Sends an inference probe with thinking configs to validate extensions.
|
||||
6. Tears the proxy back down.
|
||||
|
||||
Requirements
|
||||
------------
|
||||
- GCP credentials with Vertex AI access (run `gcloud auth application-default login`)
|
||||
- ``pip install "headroom-ai[proxy]" google-genai``
|
||||
|
||||
Run
|
||||
---
|
||||
export GCP_PROJECT_ID=your-project
|
||||
python examples/vertex_genai_sdk_demo.py
|
||||
|
||||
Note the proxy needs the ``[proxy]`` extra (fastapi, uvicorn, httpx[http2]); with
|
||||
only bare ``httpx`` installed it fails at startup on the missing ``h2`` package.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from contextlib import suppress
|
||||
from pathlib import Path
|
||||
|
||||
# Defer imports until runtime to guarantee proxy has a chance to start
|
||||
try:
|
||||
from google import genai
|
||||
from google.genai import types
|
||||
except ImportError:
|
||||
print("Error: google-genai is not installed.")
|
||||
print("Run `pip install google-genai` and try again.")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
DEFAULT_PORT = 8787
|
||||
# The google-genai SDK always builds `publishers/google/...` paths, so this demo
|
||||
# can only exercise Gemini. Claude on Vertex lives under `publishers/anthropic`
|
||||
# and uses :rawPredict -- see tests/test_proxy_vertex_native_integration.py.
|
||||
#
|
||||
# `global` is the default because the evergreen "-latest" aliases are global-only;
|
||||
# Gemini 3.x currently has no US regional endpoint.
|
||||
DEFAULT_REGION = "global"
|
||||
DEFAULT_MODEL = "gemini-flash-latest"
|
||||
# Other options: gemini-3.5-flash (also servable in europe-west2, asia-northeast1)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Proxy lifecycle
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def start_proxy(port: int) -> subprocess.Popen[bytes]:
|
||||
"""Spawn `headroom proxy` as a subprocess."""
|
||||
env = os.environ.copy()
|
||||
env.setdefault("HEADROOM_LOG", "INFO")
|
||||
|
||||
cmd = [sys.executable, "-m", "headroom.cli", "proxy", "--port", str(port)]
|
||||
print(f" $ {' '.join(cmd)}", file=sys.stderr)
|
||||
log_path = Path("/tmp") / f"vertex_genai_sdk_demo_{port}.log"
|
||||
log_file = log_path.open("wb")
|
||||
proc = subprocess.Popen(
|
||||
cmd,
|
||||
env=env,
|
||||
stdout=log_file,
|
||||
stderr=subprocess.STDOUT,
|
||||
)
|
||||
print(f" proxy logs -> {log_path}", file=sys.stderr)
|
||||
return proc
|
||||
|
||||
|
||||
def wait_for_proxy_ready(port: int, timeout_s: float = 30.0) -> None:
|
||||
"""Poll /readyz until the proxy answers or timeout."""
|
||||
url = f"http://127.0.0.1:{port}/readyz"
|
||||
deadline = time.time() + timeout_s
|
||||
last_err: Exception | None = None
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=1) as resp:
|
||||
if resp.status == 200:
|
||||
return
|
||||
except (urllib.error.URLError, ConnectionError, TimeoutError) as e:
|
||||
last_err = e
|
||||
time.sleep(0.5)
|
||||
raise RuntimeError(
|
||||
f"Proxy on port {port} did not become ready within {timeout_s}s; last error: {last_err!r}"
|
||||
)
|
||||
|
||||
|
||||
def stop_proxy(proc: subprocess.Popen[bytes]) -> None:
|
||||
"""Politely shut the proxy down."""
|
||||
with suppress(ProcessLookupError):
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait(timeout=5)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Main demo
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_demo(port: int, region: str, model_id: str) -> int:
|
||||
print("=" * 76)
|
||||
print(" Headroom E2E: google-genai SDK -> Headroom proxy -> Vertex")
|
||||
print("=" * 76)
|
||||
print(f" port={port} region={region} model={model_id}")
|
||||
print()
|
||||
|
||||
print("[1/3] Spawning Headroom proxy ...")
|
||||
proxy = start_proxy(port=port)
|
||||
try:
|
||||
try:
|
||||
wait_for_proxy_ready(port=port, timeout_s=45.0)
|
||||
except Exception as e:
|
||||
print(f" ! Proxy failed to start: {e}", file=sys.stderr)
|
||||
return 2
|
||||
print(" proxy ready.")
|
||||
|
||||
print("\n[2/3] Configuring google-genai SDK for Vertex via Proxy")
|
||||
project_id = os.environ.get("GCP_PROJECT_ID")
|
||||
if not project_id:
|
||||
print(
|
||||
" ! GCP_PROJECT_ID is not set. Every probe below will 404.\n"
|
||||
" export GCP_PROJECT_ID=$(gcloud config get-value project)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
# We enforce vertexai=True to hit standard Vertex boundaries.
|
||||
# Ensure your GCP ADC variables are valid / authorized if routing through real GCP APIs.
|
||||
client = genai.Client(
|
||||
vertexai=True,
|
||||
project=project_id,
|
||||
location=region,
|
||||
http_options={"base_url": f"http://127.0.0.1:{port}"},
|
||||
)
|
||||
|
||||
print("\n[3/3] Probes")
|
||||
print("\n a. Standard Inference:")
|
||||
content = "Count to 5, listing each number separated by commas."
|
||||
|
||||
try:
|
||||
response = client.models.generate_content(
|
||||
model=model_id,
|
||||
contents=content,
|
||||
)
|
||||
print(" ✓ Standard response received successfully!")
|
||||
print(f" > {response.text.strip()}")
|
||||
except Exception as e:
|
||||
# Vertex 401/403/404/429 errors carry a `[headroom] hint:` from the proxy.
|
||||
print(f" ! Standard inference failed: {e}")
|
||||
return 1
|
||||
|
||||
print("\n b. Inference with Thinking Config:")
|
||||
# Configure thinking config (where supported).
|
||||
# For now, we just pass parameters and see if Headroom properly parses/forwards them.
|
||||
try:
|
||||
config = types.GenerateContentConfig(
|
||||
thinking_config=types.ThinkingConfig(thinking_budget=128)
|
||||
)
|
||||
response = client.models.generate_content(
|
||||
model=model_id,
|
||||
contents="Think briefly. What is heavier: a kg of feathers or a kg of steel?",
|
||||
config=config,
|
||||
)
|
||||
print(" ✓ Thinking response received successfully!")
|
||||
print(f" > {response.text.strip()}")
|
||||
except Exception as e:
|
||||
print(f" ! Thinking inference failed: {e}")
|
||||
return 1
|
||||
|
||||
return 0
|
||||
|
||||
finally:
|
||||
print("\n shutting down proxy ...")
|
||||
stop_proxy(proxy)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="google-genai SDK -> Headroom proxy -> Vertex")
|
||||
ap.add_argument("--port", type=int, default=DEFAULT_PORT)
|
||||
ap.add_argument("--region", default=DEFAULT_REGION)
|
||||
ap.add_argument("--model", default=DEFAULT_MODEL)
|
||||
args = ap.parse_args()
|
||||
return run_demo(port=args.port, region=args.region, model_id=args.model)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -826,6 +826,18 @@ class LiteLLMBackend(Backend):
|
||||
To add a new provider, just add an entry to PROVIDER_REGISTRY above.
|
||||
"""
|
||||
|
||||
def _public_error_message(self, exc: BaseException, raw_text: str) -> str:
|
||||
"""Client-safe error text (public_errors contract).
|
||||
|
||||
For Vertex, a fixed setup hint is appended when the raw text shows a
|
||||
missing SDK or ADC; the raw text itself only ever reaches the log.
|
||||
"""
|
||||
if self.provider in ("vertex_ai", "vertex_ai_beta"):
|
||||
from headroom.providers.vertex import public_backend_error_message
|
||||
|
||||
return public_backend_error_message(exc, raw_text)
|
||||
return client_message(exc, raw_text)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
provider: str = "bedrock",
|
||||
@@ -1353,7 +1365,7 @@ class LiteLLMBackend(Backend):
|
||||
logger.error(f"LiteLLM error: {error_message}")
|
||||
# Provider API errors keep their text; transport failures are
|
||||
# reduced to the public vocabulary (see proxy/public_errors).
|
||||
error_message = client_message(e, error_message)
|
||||
error_message = self._public_error_message(e, error_message)
|
||||
|
||||
# Map to Anthropic error format
|
||||
error_type = "api_error"
|
||||
@@ -1790,7 +1802,10 @@ class LiteLLMBackend(Backend):
|
||||
event_type="error",
|
||||
data={
|
||||
"type": "error",
|
||||
"error": {"type": "api_error", "message": client_message(e, error_message)},
|
||||
"error": {
|
||||
"type": "api_error",
|
||||
"message": self._public_error_message(e, error_message),
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
@@ -1982,7 +1997,7 @@ class LiteLLMBackend(Backend):
|
||||
except Exception as e:
|
||||
error_message = format_exception_message(e)
|
||||
logger.error(f"LiteLLM OpenAI error: {error_message}")
|
||||
error_message = client_message(e, error_message)
|
||||
error_message = self._public_error_message(e, error_message)
|
||||
|
||||
# Map to OpenAI error format
|
||||
error_type = "api_error"
|
||||
@@ -2092,7 +2107,7 @@ class LiteLLMBackend(Backend):
|
||||
logger.error(f"LiteLLM OpenAI streaming error: {error_message}")
|
||||
error_data = {
|
||||
"error": {
|
||||
"message": client_message(e, error_message),
|
||||
"message": self._public_error_message(e, error_message),
|
||||
"type": "api_error",
|
||||
"code": "backend_error",
|
||||
}
|
||||
|
||||
@@ -16,6 +16,8 @@ import click
|
||||
|
||||
from headroom import paths as _paths
|
||||
from headroom.providers.registry import (
|
||||
BackendUnavailableError,
|
||||
preflight_backend,
|
||||
resolve_api_overrides,
|
||||
resolve_api_targets,
|
||||
resolve_extra_headers,
|
||||
@@ -1944,6 +1946,13 @@ Press Ctrl+C to stop.
|
||||
# Option E: start embedding server sidecar if requested
|
||||
# -----------------------------------------------------------------------
|
||||
_embed_watchdog = None
|
||||
try:
|
||||
# Here, not only in create_app: with --workers >1 each worker builds its
|
||||
# own app and uvicorn would keep restarting them instead of exiting.
|
||||
preflight_backend(backend)
|
||||
except BackendUnavailableError as exc:
|
||||
click.echo(f"\nCannot start proxy: {exc}", err=True)
|
||||
raise SystemExit(2) from None
|
||||
if embedding_server:
|
||||
_embed_socket = embedding_server_socket or f"/tmp/headroom-embed-{config.port}.sock"
|
||||
# Pass socket path to all worker processes via environment variable
|
||||
@@ -1996,6 +2005,11 @@ Press Ctrl+C to stop.
|
||||
except KeyboardInterrupt:
|
||||
click.echo("\nShutting down...")
|
||||
raise SystemExit(130) from None
|
||||
except BackendUnavailableError as exc:
|
||||
# Operator misconfiguration, not a crash to debug: show the remedy, not
|
||||
# a traceback through our startup internals.
|
||||
click.echo(f"\nCannot start proxy: {exc}", err=True)
|
||||
raise SystemExit(2) from None
|
||||
finally:
|
||||
if _embed_watchdog is not None:
|
||||
import asyncio as _asyncio2
|
||||
|
||||
@@ -67,6 +67,7 @@ from headroom.providers.vertex import (
|
||||
is_vertex_google_publisher,
|
||||
vertex_anthropic_target,
|
||||
vertex_publisher_provider_name,
|
||||
with_vertex_diagnostics,
|
||||
)
|
||||
from headroom.proxy.passthrough import (
|
||||
custom_base_passthrough_telemetry as _custom_base_passthrough_telemetry,
|
||||
@@ -359,6 +360,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
|
||||
@app.post(
|
||||
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:generateContent"
|
||||
)
|
||||
@with_vertex_diagnostics
|
||||
async def vertex_generate_content(
|
||||
request: Request,
|
||||
api_version: str,
|
||||
@@ -380,6 +382,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
|
||||
@app.post(
|
||||
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamGenerateContent"
|
||||
)
|
||||
@with_vertex_diagnostics
|
||||
async def vertex_stream_generate_content(
|
||||
request: Request,
|
||||
api_version: str,
|
||||
@@ -405,6 +408,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
|
||||
@app.post(
|
||||
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:countTokens"
|
||||
)
|
||||
@with_vertex_diagnostics
|
||||
async def vertex_count_tokens(
|
||||
request: Request,
|
||||
api_version: str,
|
||||
@@ -426,6 +430,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
|
||||
@app.post(
|
||||
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:rawPredict"
|
||||
)
|
||||
@with_vertex_diagnostics
|
||||
async def vertex_raw_predict(
|
||||
request: Request,
|
||||
api_version: str,
|
||||
@@ -447,6 +452,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
|
||||
@app.post(
|
||||
"/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:rawPredict"
|
||||
)
|
||||
@with_vertex_diagnostics
|
||||
async def vertex_raw_predict_no_version(
|
||||
request: Request,
|
||||
project: str,
|
||||
@@ -471,6 +477,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
|
||||
@app.post(
|
||||
"/{api_version}/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamRawPredict"
|
||||
)
|
||||
@with_vertex_diagnostics
|
||||
async def vertex_stream_raw_predict(
|
||||
request: Request,
|
||||
api_version: str,
|
||||
@@ -497,6 +504,7 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
|
||||
@app.post(
|
||||
"/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:streamRawPredict"
|
||||
)
|
||||
@with_vertex_diagnostics
|
||||
async def vertex_stream_raw_predict_no_version(
|
||||
request: Request,
|
||||
project: str,
|
||||
|
||||
@@ -25,6 +25,14 @@ AnyLLMBackendType: Any = None
|
||||
LiteLLMBackendType: Any = None
|
||||
|
||||
|
||||
class BackendUnavailableError(RuntimeError):
|
||||
"""A selected backend cannot serve and no amount of retrying will help.
|
||||
|
||||
Raised at startup for operator misconfiguration (a missing optional
|
||||
dependency, say) so callers can print the remedy instead of a traceback.
|
||||
"""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProviderApiOverrides:
|
||||
"""Optional upstream API URL overrides configured for the proxy."""
|
||||
@@ -213,6 +221,36 @@ def build_proxy_provider_runtime(config: Any) -> ProxyProviderRuntime:
|
||||
)
|
||||
|
||||
|
||||
def _litellm_provider(backend: str) -> str:
|
||||
provider = (backend if backend.startswith("litellm-") else f"litellm-{backend}").replace(
|
||||
"litellm-", ""
|
||||
)
|
||||
# `litellm-vertex` is the name in our docs/help, but LiteLLM (and our
|
||||
# provider registry) keys Google Vertex on `vertex_ai`. Without this alias
|
||||
# the provider falls through to a generic pass-through: wrong model prefix
|
||||
# (`vertex/…` instead of `vertex_ai/…`), region dropped, auth mishandled.
|
||||
if provider in ("vertex", "google-vertex", "googlevertex"):
|
||||
return "vertex_ai"
|
||||
return provider
|
||||
|
||||
|
||||
def preflight_backend(backend: str) -> None:
|
||||
"""Raise ``BackendUnavailableError`` if ``backend``'s optional SDK is missing.
|
||||
|
||||
Without the Vertex SDK the LiteLLM backend still constructs cleanly and then
|
||||
fails on every call with a provider string that names neither the cause nor
|
||||
the cure. The CLI calls this in the parent process too: with ``--workers >1``
|
||||
each worker builds its own app and uvicorn would just keep restarting them.
|
||||
"""
|
||||
if backend == "anthropic" or backend == "anyllm" or backend.startswith("anyllm-"):
|
||||
return
|
||||
if _litellm_provider(backend) in ("vertex_ai", "vertex_ai_beta"):
|
||||
# Imported locally: headroom.providers.vertex imports this module.
|
||||
from headroom.providers.vertex import ensure_vertex_sdk_available
|
||||
|
||||
ensure_vertex_sdk_available()
|
||||
|
||||
|
||||
def create_proxy_backend(
|
||||
*,
|
||||
backend: str,
|
||||
@@ -249,13 +287,10 @@ def create_proxy_backend(
|
||||
return None
|
||||
|
||||
normalized_backend = backend if backend.startswith("litellm-") else f"litellm-{backend}"
|
||||
provider = normalized_backend.replace("litellm-", "")
|
||||
# `litellm-vertex` is the name in our docs/help, but LiteLLM (and our
|
||||
# provider registry) keys Google Vertex on `vertex_ai`. Without this alias
|
||||
# the provider falls through to a generic pass-through: wrong model prefix
|
||||
# (`vertex/…` instead of `vertex_ai/…`), region dropped, auth mishandled.
|
||||
if provider in ("vertex", "google-vertex", "googlevertex"):
|
||||
provider = "vertex_ai"
|
||||
provider = _litellm_provider(backend)
|
||||
if litellm_backend_cls is None:
|
||||
# Skipped when a backend class is injected, so tests can use fakes.
|
||||
preflight_backend(backend)
|
||||
try:
|
||||
backend_cls = litellm_backend_cls or _load_litellm_backend()
|
||||
instance = cast(
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
"""Google Vertex provider helpers."""
|
||||
|
||||
from .diagnostics import (
|
||||
HINT_HEADER,
|
||||
annotate_vertex_error,
|
||||
ensure_vertex_sdk_available,
|
||||
public_backend_error_message,
|
||||
vertex_error_hint,
|
||||
with_vertex_diagnostics,
|
||||
)
|
||||
from .runtime import (
|
||||
VERTEX_ANTHROPIC_PROVIDER_NAME,
|
||||
VERTEX_COUNT_TOKENS,
|
||||
@@ -17,6 +25,7 @@ from .runtime import (
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"HINT_HEADER",
|
||||
"VERTEX_ANTHROPIC_PROVIDER_NAME",
|
||||
"VERTEX_COUNT_TOKENS",
|
||||
"VERTEX_GENERATE_CONTENT",
|
||||
@@ -25,9 +34,14 @@ __all__ = [
|
||||
"VERTEX_STREAM_GENERATE_CONTENT",
|
||||
"VERTEX_STREAM_RAW_PREDICT",
|
||||
"VertexPublisherAction",
|
||||
"annotate_vertex_error",
|
||||
"ensure_vertex_sdk_available",
|
||||
"is_vertex_anthropic_publisher",
|
||||
"is_vertex_google_publisher",
|
||||
"public_backend_error_message",
|
||||
"vertex_anthropic_target",
|
||||
"vertex_error_hint",
|
||||
"vertex_publisher_provider_name",
|
||||
"vertex_target_for_location",
|
||||
"with_vertex_diagnostics",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
"""Actionable diagnostics for Vertex upstream failures.
|
||||
|
||||
Vertex rejects requests for a handful of boring, recurring reasons -- expired
|
||||
ADC, an un-enabled partner model, a model that simply is not served in the
|
||||
requested location -- and its own error text says nothing about how to fix any
|
||||
of them. Users hit these while onboarding, see a bare 404 through whatever SDK
|
||||
they are using, and have no way to tell a proxy bug from a project-config gap.
|
||||
|
||||
So annotate the failure at the proxy and surface a hint three ways: a WARNING in
|
||||
the proxy log, an ``x-headroom-hint`` response header, and (because most SDKs
|
||||
only ever show the message string) an appended note on ``error.message``.
|
||||
|
||||
Every hint is one of the fixed constants below: request data and exception text
|
||||
select a hint but are never interpolated into it (public_errors contract).
|
||||
Only already-failing responses are touched; success bodies are never modified.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import importlib.util
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from typing import Any
|
||||
|
||||
from headroom.providers.registry import BackendUnavailableError
|
||||
from headroom.proxy.public_errors import client_message
|
||||
|
||||
from .runtime import is_vertex_anthropic_publisher
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
HINT_HEADER = "x-headroom-hint"
|
||||
|
||||
_ADC_REFRESH = (
|
||||
"Credentials are missing or expired (Vertex access tokens last ~1h). "
|
||||
"Re-auth with `gcloud auth application-default login`, and note that a bare "
|
||||
"`gcloud auth print-access-token` user token is NOT accepted -- use "
|
||||
"`gcloud auth application-default print-access-token`."
|
||||
)
|
||||
|
||||
_ENABLE_API = (
|
||||
"Confirm `aiplatform.googleapis.com` is enabled on the project and that the "
|
||||
"caller holds roles/aiplatform.user."
|
||||
)
|
||||
|
||||
_ENABLE_PARTNER = (
|
||||
"Partner models (Anthropic Claude, Llama, Mistral) need a one-time per-project "
|
||||
"enable in Model Garden before they will serve -- an otherwise healthy project "
|
||||
"returns 404/403 until someone clicks through."
|
||||
)
|
||||
|
||||
_LOCATION_GEMINI = (
|
||||
"Gemini 3.x has no US regional endpoint: the evergreen `-latest` aliases are "
|
||||
"global-only, and e.g. gemini-3.5-flash serves from `global`, europe-west2 and "
|
||||
"asia-northeast1. Retry against `global` before assuming the proxy is at fault."
|
||||
)
|
||||
|
||||
_LOCATION_CLAUDE = (
|
||||
"Claude 4.6 and older serve from `us-east5`/europe-west1/asia-southeast1 or "
|
||||
"`global`; Claude 4.7+ dropped named regions and is reachable only via the "
|
||||
"`us`/`eu` multi-region or `global` endpoints."
|
||||
)
|
||||
|
||||
_QUOTA = (
|
||||
"Quota exhausted for this model in this location. Back off and retry, request a "
|
||||
"quota increase, or send the request to another supported region."
|
||||
)
|
||||
|
||||
_MISSING_VERTEX_SDK = (
|
||||
"`--backend vertex` routes Anthropic Messages traffic through LiteLLM's vertex_ai "
|
||||
'provider, which needs the Vertex SDK: `pip install "headroom-ai[proxy,vertex]"` '
|
||||
'(or `pip install "google-cloud-aiplatform>=1.38"`). '
|
||||
"Note the native Vertex routes (/v1/projects/.../publishers/anthropic/models/...:rawPredict) "
|
||||
"are a straight passthrough and do NOT need `--backend vertex` -- drop the flag if you are "
|
||||
"calling Vertex paths directly."
|
||||
)
|
||||
|
||||
_MISSING_ADC = (
|
||||
"No usable Google credentials were found. Run `gcloud auth application-default login`, or "
|
||||
"point GOOGLE_APPLICATION_CREDENTIALS at a service-account key."
|
||||
)
|
||||
|
||||
|
||||
def vertex_sdk_available() -> bool:
|
||||
"""Whether the LiteLLM vertex_ai provider's `vertexai` import will succeed."""
|
||||
return importlib.util.find_spec("vertexai") is not None
|
||||
|
||||
|
||||
def ensure_vertex_sdk_available() -> None:
|
||||
"""Refuse to start a Vertex LiteLLM backend that cannot possibly serve.
|
||||
|
||||
Without the SDK the backend still constructs cleanly and then fails on
|
||||
*every* request with an opaque provider string. Failing here instead turns a
|
||||
per-request mystery into one startup error at the moment of misconfiguration.
|
||||
"""
|
||||
if not vertex_sdk_available():
|
||||
raise BackendUnavailableError(
|
||||
f"Vertex backend selected but the Vertex SDK is missing. {_MISSING_VERTEX_SDK}"
|
||||
)
|
||||
|
||||
|
||||
def backend_error_hint(message: str) -> str | None:
|
||||
"""Fixed setup hint for a raw LiteLLM Vertex error, or None if unrecognized."""
|
||||
lowered = message.lower()
|
||||
if "google-cloud-aiplatform" in lowered or (
|
||||
"vertexai" in lowered and ("no module named" in lowered or "import failed" in lowered)
|
||||
):
|
||||
return _MISSING_VERTEX_SDK
|
||||
if "default credentials" in lowered or "could not automatically determine" in lowered:
|
||||
return _MISSING_ADC
|
||||
return None
|
||||
|
||||
|
||||
def public_backend_error_message(exc: BaseException, raw_text: str) -> str:
|
||||
"""``client_message`` plus a fixed setup hint selected by ``raw_text``.
|
||||
|
||||
``raw_text`` itself never reaches the client; the caller logs it.
|
||||
"""
|
||||
message = client_message(exc, raw_text)
|
||||
hint = backend_error_hint(raw_text)
|
||||
# A chained Headroom upstream may already have added one.
|
||||
if not hint or "[headroom] hint:" in message:
|
||||
return message
|
||||
return f"{message}\n[headroom] hint: {hint}"
|
||||
|
||||
|
||||
def vertex_error_hint(status_code: int, *, publisher: str = "") -> str | None:
|
||||
"""Fixed hint for a failing Vertex status, or None if we have none."""
|
||||
if status_code == 401:
|
||||
return _ADC_REFRESH
|
||||
if status_code == 429:
|
||||
return _QUOTA
|
||||
if status_code not in (403, 404):
|
||||
return None
|
||||
# 403 and 404 are the same problem for the user: the project cannot serve
|
||||
# this model here. Which remedy applies depends on the publisher.
|
||||
if is_vertex_anthropic_publisher(publisher):
|
||||
return f"{_ENABLE_API} {_ENABLE_PARTNER} {_LOCATION_CLAUDE}"
|
||||
if publisher != "google":
|
||||
return f"{_ENABLE_API} {_ENABLE_PARTNER}"
|
||||
return f"{_ENABLE_API} {_LOCATION_GEMINI}"
|
||||
|
||||
|
||||
def annotate_vertex_error(
|
||||
response: Any,
|
||||
*,
|
||||
location: str = "",
|
||||
publisher: str = "",
|
||||
model: str = "",
|
||||
) -> Any:
|
||||
"""Attach a Headroom hint to a failing Vertex response, in place.
|
||||
|
||||
Returns the same response. Any problem while annotating is swallowed: a
|
||||
diagnostic must never turn a clean upstream error into a proxy 500.
|
||||
"""
|
||||
status = getattr(response, "status_code", None)
|
||||
hint = vertex_error_hint(status, publisher=publisher) if isinstance(status, int) else None
|
||||
if hint is None:
|
||||
return response
|
||||
try:
|
||||
# Idempotent: nested route helpers can annotate the same response twice.
|
||||
if HINT_HEADER in response.headers:
|
||||
return response
|
||||
# %r: path params are client-supplied; keep CR/LF from forging log lines.
|
||||
logger.warning(
|
||||
"vertex upstream %s for %r/%r @ %r -- %s", status, publisher, model, location, hint
|
||||
)
|
||||
response.headers[HINT_HEADER] = hint
|
||||
|
||||
# Streaming responses have no materialized body; header + log suffice.
|
||||
body = getattr(response, "body", None)
|
||||
if not isinstance(body, (bytes, bytearray)):
|
||||
return response
|
||||
payload = json.loads(body)
|
||||
if not isinstance(payload, dict):
|
||||
return response
|
||||
error = payload.get("error")
|
||||
if isinstance(error, dict) and isinstance(error.get("message"), str):
|
||||
# Most SDKs only show `error.message`, so the hint must live there.
|
||||
error["message"] = f"{error['message']}\n[headroom] hint: {hint}"
|
||||
else:
|
||||
payload["headroom_hint"] = hint
|
||||
response.body = json.dumps(payload).encode("utf-8")
|
||||
response.headers["content-length"] = str(len(response.body))
|
||||
except Exception: # noqa: BLE001 - see docstring
|
||||
pass
|
||||
return response
|
||||
|
||||
|
||||
def with_vertex_diagnostics(
|
||||
handler: Callable[..., Awaitable[Any]],
|
||||
) -> Callable[..., Awaitable[Any]]:
|
||||
"""Decorate a Vertex route so its failures carry an actionable hint.
|
||||
|
||||
Reads `location`/`publisher`/`model` from the path params FastAPI injects.
|
||||
"""
|
||||
|
||||
@functools.wraps(handler)
|
||||
async def wrapper(*args: Any, **kwargs: Any) -> Any:
|
||||
return annotate_vertex_error(
|
||||
await handler(*args, **kwargs),
|
||||
location=kwargs.get("location", ""),
|
||||
publisher=kwargs.get("publisher", ""),
|
||||
model=kwargs.get("model", ""),
|
||||
)
|
||||
|
||||
return wrapper
|
||||
@@ -137,6 +137,17 @@ proxy-prod = [
|
||||
"headroom-ai[proxy]",
|
||||
"gunicorn>=21.0.0; sys_platform != 'win32'",
|
||||
]
|
||||
# Google Vertex through the LiteLLM translation backend
|
||||
# (`--backend vertex` / `vertex_ai` / `litellm-vertex`), which imports `vertexai`.
|
||||
# Kept separate from [proxy] for the same reason as proxy-prod: this pulls
|
||||
# google-cloud-aiplatform and its transitive tree (~175MB installed), and the
|
||||
# proxy's *native* Vertex passthrough routes
|
||||
# (/v1/projects/.../locations/.../publishers/...) do not need it at all. Only
|
||||
# the LiteLLM backend does. Install with:
|
||||
# pip install headroom-ai[proxy,vertex]
|
||||
vertex = [
|
||||
"google-cloud-aiplatform>=1.38",
|
||||
]
|
||||
# AST-based code compression (tree-sitter)
|
||||
# tree-sitter-language-pack >=1.0 removed the bundled tree-sitter package and
|
||||
# switched to an incompatible internal node API (.kind vs .type, callable
|
||||
|
||||
@@ -1016,7 +1016,10 @@ class TestCLIProxyBackend:
|
||||
def mock_run_server(config, **kwargs):
|
||||
captured_config["config"] = config
|
||||
|
||||
with patch("headroom.proxy.server.run_server", mock_run_server):
|
||||
with (
|
||||
patch("headroom.proxy.server.run_server", mock_run_server),
|
||||
patch("headroom.providers.vertex.diagnostics.vertex_sdk_available", return_value=True),
|
||||
):
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["proxy", "--backend", "litellm-vertex"],
|
||||
|
||||
@@ -281,6 +281,9 @@ class TestOpenAIStreamingMock:
|
||||
with (
|
||||
patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}),
|
||||
patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp,
|
||||
# Suppress the SDK preflight: this test proves LiteLLM request-shaping,
|
||||
# not whether google-cloud-aiplatform is installed in the CI environment.
|
||||
patch("headroom.providers.vertex.diagnostics.vertex_sdk_available", return_value=True),
|
||||
):
|
||||
mock_acomp.return_value = fake_stream()
|
||||
app = create_app(config)
|
||||
|
||||
@@ -224,6 +224,9 @@ def test_litellm_vertex_backend_path_preserves_max_tokens_and_vendor_fields():
|
||||
with (
|
||||
patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}),
|
||||
patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp,
|
||||
# Suppress the SDK preflight: this test proves LiteLLM request-shaping,
|
||||
# not whether google-cloud-aiplatform is installed in the CI environment.
|
||||
patch("headroom.providers.vertex.diagnostics.vertex_sdk_available", return_value=True),
|
||||
):
|
||||
mock_acomp.return_value = _make_litellm_response()
|
||||
app = create_app(config)
|
||||
|
||||
@@ -0,0 +1,239 @@
|
||||
"""Integration tests for Vertex native API endpoint with real API calls through the proxy.
|
||||
|
||||
Run with:
|
||||
export GCP_PROJECT_ID="your-project"
|
||||
export GCP_ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
|
||||
pytest tests/test_proxy_vertex_native_integration.py -rs -v
|
||||
|
||||
Use the *application-default* token, not `gcloud auth print-access-token`: a bare
|
||||
user token is rejected by aiplatform.googleapis.com with 401 UNAUTHENTICATED.
|
||||
Tokens expire hourly, so re-export before each run.
|
||||
|
||||
Partner (Anthropic) models must be enabled per-project in Model Garden. Cases
|
||||
that the project cannot serve SKIP with an actionable reason instead of failing;
|
||||
run with `-rs` so those reasons are printed.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not os.environ.get("GCP_ACCESS_TOKEN") or not os.environ.get("GCP_PROJECT_ID"),
|
||||
reason="GCP_ACCESS_TOKEN or GCP_PROJECT_ID not set",
|
||||
)
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
pytest.importorskip("httpx")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def vertex_client():
|
||||
"""Create test client for Vertex AI native API with optimization enabled."""
|
||||
config = ProxyConfig(
|
||||
optimize=True, # Enable compression
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
)
|
||||
app = create_app(config)
|
||||
with TestClient(app) as client:
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def api_key():
|
||||
"""Get GCP access token from environment."""
|
||||
return os.environ.get("GCP_ACCESS_TOKEN")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def project_id():
|
||||
"""Get GCP project ID from environment."""
|
||||
return os.environ.get("GCP_PROJECT_ID")
|
||||
|
||||
|
||||
def _upstream_message(response) -> str:
|
||||
"""Best-effort extraction of the Vertex error message."""
|
||||
try:
|
||||
return str(response.json().get("error", {}).get("message", response.text))
|
||||
except Exception:
|
||||
return response.text
|
||||
|
||||
|
||||
def _require_upstream_ok(response, url, location, model, publisher) -> None:
|
||||
"""Fail loudly on proxy defects; skip on project-provisioning gaps.
|
||||
|
||||
A non-200 here is one of two very different things, and conflating them
|
||||
makes this suite useless:
|
||||
|
||||
* the proxy mangled/misrouted the request -> a real failure, fail hard
|
||||
* the caller's GCP project cannot serve this model here -> not our bug
|
||||
|
||||
Partner (Anthropic) models in particular require a per-project enable step
|
||||
in Model Garden, so a project that is otherwise healthy will 403/404 on
|
||||
Claude until someone clicks through. Skip those with an actionable reason
|
||||
rather than reporting a red proxy.
|
||||
"""
|
||||
if response.status_code == 200:
|
||||
return
|
||||
|
||||
detail = _upstream_message(response)
|
||||
ctx = f"{publisher}/{model} @ {location}\n URL: {url}\n Vertex said: {detail}"
|
||||
|
||||
if response.status_code == 401:
|
||||
pytest.fail(
|
||||
f"401 UNAUTHENTICATED for {ctx}\n"
|
||||
" GCP_ACCESS_TOKEN is missing, stale (they expire hourly), or is a user\n"
|
||||
" token where ADC is required. Refresh with:\n"
|
||||
" export GCP_ACCESS_TOKEN=$(gcloud auth application-default print-access-token)"
|
||||
)
|
||||
|
||||
if response.status_code == 403:
|
||||
pytest.skip(
|
||||
f"403 PERMISSION_DENIED for {ctx}\n"
|
||||
" The project cannot serve this model. Either aiplatform.googleapis.com is\n"
|
||||
" disabled, the principal lacks roles/aiplatform.user, or (for Anthropic)\n"
|
||||
" the model has not been enabled in Model Garden for this project."
|
||||
)
|
||||
|
||||
if response.status_code == 404:
|
||||
pytest.skip(
|
||||
f"404 NOT_FOUND for {ctx}\n"
|
||||
" This model is not available to the project at this location.\n"
|
||||
f" For publisher '{publisher}', enable the model in Model Garden, or confirm\n"
|
||||
" the location is in its supported-regions list. Note Gemini 3.x has no US\n"
|
||||
" regional endpoint, and Claude 4.7+ dropped named regions for us/eu/global."
|
||||
)
|
||||
|
||||
if response.status_code == 429:
|
||||
pytest.skip(f"429 RESOURCE_EXHAUSTED (quota) for {ctx}")
|
||||
|
||||
if response.status_code == 400 and "not servable in region" in detail:
|
||||
pytest.skip(
|
||||
f"400 FAILED_PRECONDITION for {ctx}\n"
|
||||
" The parametrized location/model pair is wrong -- fix the test matrix."
|
||||
)
|
||||
|
||||
pytest.fail(
|
||||
f"Expected 200, got {response.status_code} for {ctx}\n"
|
||||
" This status is not a known provisioning condition, so treat it as a proxy\n"
|
||||
" defect (bad path rewrite, mangled body, dropped auth header) until proven\n"
|
||||
" otherwise by reproducing the same call directly against Vertex with curl."
|
||||
)
|
||||
|
||||
|
||||
# Location/model pairs are explicit rather than a cross-product: Vertex serves a
|
||||
# given publisher model in only a subset of locations, so a cross-product
|
||||
# generates combinations that can never return 200.
|
||||
#
|
||||
# global -> evergreen "-latest" aliases (global-only) + Claude
|
||||
# europe-west2 -> Gemini 3.x has no US regional endpoint; EU/APAC regions only
|
||||
# us-east5 -> Claude 4.6 and older have real regional serving here
|
||||
#
|
||||
# Deliberately no Gemini 2.5: it retires 2026-10-16 and should not be the thing
|
||||
# this suite proves the proxy against.
|
||||
@pytest.mark.parametrize(
|
||||
"location,model,publisher",
|
||||
[
|
||||
("global", "gemini-flash-latest", "google"),
|
||||
("global", "claude-sonnet-4-6", "anthropic"),
|
||||
("europe-west2", "gemini-3.5-flash", "google"),
|
||||
("us-east5", "claude-sonnet-4-6", "anthropic"),
|
||||
],
|
||||
)
|
||||
class TestVertexNativeGenerateContent:
|
||||
"""Test Vertex model endpoints."""
|
||||
|
||||
def test_basic_generation(self, vertex_client, api_key, project_id, location, model, publisher):
|
||||
"""Basic text generation works."""
|
||||
|
||||
# Determine the action based on publisher
|
||||
action = "generateContent" if publisher == "google" else "rawPredict"
|
||||
url = f"/v1/projects/{project_id}/locations/{location}/publishers/{publisher}/models/{model}:{action}"
|
||||
|
||||
# Prepare the payload (Anthropic Vertex requires Claude messages format for rawPredict,
|
||||
# Google Vertex requires contents format for generateContent)
|
||||
|
||||
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
|
||||
|
||||
if publisher == "google":
|
||||
payload = {
|
||||
"contents": [
|
||||
{
|
||||
"role": "user",
|
||||
"parts": [{"text": "What is 2+2? Reply with just the number."}],
|
||||
}
|
||||
]
|
||||
}
|
||||
else:
|
||||
payload = {
|
||||
"anthropic_version": "vertex-2023-10-16",
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is 2+2? Reply with just the number."}
|
||||
],
|
||||
"max_tokens": 100,
|
||||
}
|
||||
|
||||
response = vertex_client.post(url, headers=headers, json=payload)
|
||||
_require_upstream_ok(response, url, location, model, publisher)
|
||||
data = response.json()
|
||||
|
||||
if publisher == "google":
|
||||
assert "candidates" in data
|
||||
assert len(data["candidates"]) > 0
|
||||
text = data["candidates"][0]["content"]["parts"][0]["text"]
|
||||
assert "4" in text
|
||||
else:
|
||||
assert "content" in data
|
||||
assert len(data["content"]) > 0
|
||||
text = data["content"][0]["text"]
|
||||
assert "4" in text
|
||||
|
||||
def test_thinking_levels(self, vertex_client, api_key, project_id, location, model, publisher):
|
||||
"""Test thinking extensions where supported."""
|
||||
action = "generateContent" if publisher == "google" else "rawPredict"
|
||||
url = f"/v1/projects/{project_id}/locations/{location}/publishers/{publisher}/models/{model}:{action}"
|
||||
|
||||
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
|
||||
|
||||
if publisher == "google":
|
||||
payload = {
|
||||
"contents": [
|
||||
{"role": "user", "parts": [{"text": "Think deeply about the number 42."}]}
|
||||
],
|
||||
# REST spelling is thinkingBudget (camelCase); thinkingBudgetTokens
|
||||
# is silently ignored, which makes the assertion below meaningless.
|
||||
"generationConfig": {
|
||||
"thinkingConfig": {"thinkingBudget": 512, "includeThoughts": True}
|
||||
},
|
||||
}
|
||||
else:
|
||||
payload = {
|
||||
"anthropic_version": "vertex-2023-10-16",
|
||||
"messages": [{"role": "user", "content": "Think deeply about the number 42."}],
|
||||
"max_tokens": 8192,
|
||||
"thinking": {"type": "enabled", "budget_tokens": 1024},
|
||||
}
|
||||
|
||||
response = vertex_client.post(url, headers=headers, json=payload)
|
||||
_require_upstream_ok(response, url, location, model, publisher)
|
||||
|
||||
# A 200 alone is weak evidence: prove the thinking-configured request
|
||||
# actually produced a scored generation rather than an empty envelope.
|
||||
data = response.json()
|
||||
if publisher == "google":
|
||||
assert data.get("candidates"), (
|
||||
f"No candidates in thinking response: {json.dumps(data)[:500]}"
|
||||
)
|
||||
assert "usageMetadata" in data, f"No usageMetadata: {json.dumps(data)[:500]}"
|
||||
else:
|
||||
assert data.get("content"), f"No content in thinking response: {json.dumps(data)[:500]}"
|
||||
assert data.get("usage"), f"No usage in thinking response: {json.dumps(data)[:500]}"
|
||||
|
||||
# Rest of the standard testing for compression could be added, similar to Gemini tests.
|
||||
@@ -0,0 +1,329 @@
|
||||
"""Vertex onboarding diagnostics: hints a user sees when Vertex rejects a request.
|
||||
|
||||
Every hint must be fixed text (public_errors contract): request data and raw
|
||||
exception text may *select* a hint but must never reach the client.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.providers.registry import BackendUnavailableError, create_proxy_backend
|
||||
from headroom.providers.vertex import (
|
||||
HINT_HEADER,
|
||||
annotate_vertex_error,
|
||||
public_backend_error_message,
|
||||
vertex_error_hint,
|
||||
)
|
||||
from headroom.proxy.public_errors import INTERNAL_ERROR, public_message
|
||||
|
||||
_HOSTILE = "m\r\nX-Injected: 1<script>caf\u00e9-secret-proj-123"
|
||||
_SECRET_RAW = (
|
||||
"Your default credentials were not found. project=secret-proj-123 "
|
||||
"GOOGLE_APPLICATION_CREDENTIALS=/etc/keys/sa.json host=10.1.2.3"
|
||||
)
|
||||
_LEAKS = ("secret-proj-123", "/etc/keys/sa.json", "10.1.2.3")
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
"""Minimal stand-in for a Starlette Response."""
|
||||
|
||||
def __init__(self, status_code: int, body: bytes | None = None):
|
||||
self.status_code = status_code
|
||||
self.headers: dict[str, str] = {}
|
||||
if body is not None:
|
||||
self.body = body
|
||||
|
||||
|
||||
def _error_body(status: int, message: str | None = "upstream said no") -> bytes:
|
||||
error: dict = {"code": status}
|
||||
if message is not None:
|
||||
error["message"] = message
|
||||
return json.dumps({"error": error}).encode()
|
||||
|
||||
|
||||
class TestVertexErrorHint:
|
||||
@pytest.mark.parametrize("status", [200, 400, 500, 503])
|
||||
def test_no_hint_for_unexplainable_statuses(self, status):
|
||||
"""Inventing a hint for an unknown status would mislead, not help."""
|
||||
assert vertex_error_hint(status, publisher="google") is None
|
||||
|
||||
def test_401_points_at_adc_not_user_token(self):
|
||||
hint = vertex_error_hint(401)
|
||||
assert "gcloud auth application-default login" in hint
|
||||
assert "gcloud auth print-access-token` user token is NOT accepted" in hint
|
||||
|
||||
def test_429_names_quota(self):
|
||||
assert vertex_error_hint(429, publisher="anthropic").startswith("Quota exhausted")
|
||||
|
||||
@pytest.mark.parametrize("status", [403, 404])
|
||||
def test_partner_vs_gemini_remedies(self, status):
|
||||
claude = vertex_error_hint(status, publisher="anthropic")
|
||||
gemini = vertex_error_hint(status, publisher="google")
|
||||
mistral = vertex_error_hint(status, publisher="mistralai")
|
||||
assert "Model Garden" in claude and "Claude 4.7+" in claude
|
||||
assert "no US regional endpoint" in gemini and "Model Garden" not in gemini
|
||||
assert "Model Garden" in mistral and "Gemini" not in mistral and "Claude 4" not in mistral
|
||||
|
||||
@pytest.mark.parametrize("status", [401, 403, 404, 429])
|
||||
@pytest.mark.parametrize("publisher", ["google", "anthropic", "", _HOSTILE])
|
||||
def test_every_hint_is_a_legal_header_value(self, status, publisher):
|
||||
"""h11 rejects control / non-latin-1 bytes at send time, after we return."""
|
||||
hint = vertex_error_hint(status, publisher=publisher)
|
||||
hint.encode("latin-1")
|
||||
assert all(" " <= ch <= "~" for ch in hint)
|
||||
|
||||
|
||||
class TestAnnotateVertexError:
|
||||
def test_success_untouched(self):
|
||||
body = json.dumps({"candidates": []}).encode()
|
||||
resp = annotate_vertex_error(_FakeResponse(200, body))
|
||||
assert resp.body == body and HINT_HEADER not in resp.headers
|
||||
|
||||
def test_hint_in_header_and_error_message_with_consistent_length(self):
|
||||
resp = annotate_vertex_error(_FakeResponse(404, _error_body(404)), publisher="google")
|
||||
message = json.loads(resp.body)["error"]["message"]
|
||||
assert message == f"upstream said no\n[headroom] hint: {resp.headers[HINT_HEADER]}"
|
||||
assert resp.headers["content-length"] == str(len(resp.body))
|
||||
|
||||
def test_annotation_is_idempotent(self):
|
||||
resp = annotate_vertex_error(_FakeResponse(401, _error_body(401)))
|
||||
once = resp.body
|
||||
assert annotate_vertex_error(resp).body == once
|
||||
|
||||
def test_error_without_message_gets_namespaced_key(self):
|
||||
resp = annotate_vertex_error(_FakeResponse(404, _error_body(404, message=None)))
|
||||
assert json.loads(resp.body)["headroom_hint"] == resp.headers[HINT_HEADER]
|
||||
|
||||
@pytest.mark.parametrize("body", [b"<html>nope</html>", b"[1, 2]", None])
|
||||
def test_unrewritable_bodies_survive_with_header(self, body):
|
||||
"""Non-JSON, non-object JSON and streaming (no body) keep the header only."""
|
||||
resp = annotate_vertex_error(_FakeResponse(403, body), publisher="anthropic")
|
||||
assert getattr(resp, "body", None) == body
|
||||
assert HINT_HEADER in resp.headers
|
||||
|
||||
def test_hostile_path_params_reach_log_escaped_but_never_client(self, caplog):
|
||||
with caplog.at_level(logging.WARNING):
|
||||
resp = annotate_vertex_error(
|
||||
_FakeResponse(404, _error_body(404)),
|
||||
location=_HOSTILE,
|
||||
publisher=_HOSTILE,
|
||||
model=_HOSTILE,
|
||||
)
|
||||
client_visible = resp.headers[HINT_HEADER] + resp.body.decode()
|
||||
assert "X-Injected" not in client_visible and "secret-proj" not in client_visible
|
||||
# Operators keep the context, with CR/LF escaped by %r.
|
||||
assert "secret-proj-123" in caplog.text and "\r\n" not in caplog.text
|
||||
|
||||
|
||||
class TestNativeRoutesWiring:
|
||||
"""The decorator must actually be applied on the real FastAPI routes."""
|
||||
|
||||
@pytest.fixture
|
||||
def client(self, monkeypatch):
|
||||
pytest.importorskip("fastapi")
|
||||
from fastapi.responses import JSONResponse
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.proxy.server import HeadroomProxy, ProxyConfig, create_app
|
||||
|
||||
async def upstream_404(self, request, *args):
|
||||
return JSONResponse({"error": {"code": 404, "message": "not found"}}, 404)
|
||||
|
||||
for name in ("handle_gemini_generate_content", "handle_passthrough"):
|
||||
monkeypatch.setattr(HeadroomProxy, name, upstream_404)
|
||||
app = create_app(ProxyConfig(optimize=False, cache_enabled=False, rate_limit_enabled=False))
|
||||
with TestClient(app) as client:
|
||||
yield client
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"publisher,action,expect",
|
||||
[
|
||||
("google", "generateContent", "no US regional endpoint"),
|
||||
("anthropic", "generateContent", "Model Garden"),
|
||||
],
|
||||
)
|
||||
def test_upstream_404_carries_hint(self, client, publisher, action, expect):
|
||||
resp = client.post(
|
||||
f"/v1/projects/p/locations/us-central1/publishers/{publisher}/models/m:{action}"
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
assert expect in resp.headers[HINT_HEADER]
|
||||
assert "[headroom] hint:" in resp.json()["error"]["message"]
|
||||
|
||||
def test_encoded_crlf_in_path_is_not_reflected(self, client):
|
||||
"""%0d%0a decodes into the `model` path param; the response must stay sane."""
|
||||
resp = client.post(
|
||||
"/v1/projects/p/locations/us-central1/publishers/google/models/"
|
||||
"m%0d%0aX-Injected:%201:generateContent"
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
assert "no US regional endpoint" in resp.headers[HINT_HEADER]
|
||||
assert "x-injected" not in resp.headers
|
||||
assert "X-Injected" not in resp.text
|
||||
|
||||
|
||||
class TestPublicBackendErrorMessage:
|
||||
def test_internal_error_gets_fixed_sentence_plus_hint_only(self):
|
||||
out = public_backend_error_message(RuntimeError(_SECRET_RAW), _SECRET_RAW)
|
||||
prefix = f"{public_message(INTERNAL_ERROR)}\n[headroom] hint: "
|
||||
assert out.startswith(prefix)
|
||||
assert "gcloud auth application-default login" in out
|
||||
assert not any(leak in out for leak in _LEAKS)
|
||||
|
||||
def test_missing_sdk_hint(self):
|
||||
raw = "vertexai import failed. Got error: No module named 'vertexai' at /opt/venv"
|
||||
out = public_backend_error_message(ImportError(raw), raw)
|
||||
assert "headroom-ai[proxy,vertex]" in out
|
||||
assert "do NOT need `--backend vertex`" in out
|
||||
assert "/opt/venv" not in out
|
||||
|
||||
def test_unrecognized_error_is_plain_public_message(self):
|
||||
raw = "upstream timed out talking to 10.9.9.9"
|
||||
assert public_backend_error_message(RuntimeError(raw), raw) == public_message(
|
||||
INTERNAL_ERROR
|
||||
)
|
||||
|
||||
def test_chained_proxy_hint_is_not_repeated(self):
|
||||
"""A provider HTTP error passes its text through; it may already hold our hint."""
|
||||
httpx = pytest.importorskip("httpx")
|
||||
openai = pytest.importorskip("openai")
|
||||
raw = (
|
||||
"vertexai import failed: No module named 'vertexai'\n"
|
||||
"[headroom] hint: install headroom-ai[proxy,vertex]"
|
||||
)
|
||||
resp = httpx.Response(500, request=httpx.Request("POST", "http://x"))
|
||||
exc = openai.InternalServerError(raw, response=resp, body=None)
|
||||
assert public_backend_error_message(exc, raw).count("[headroom] hint:") == 1
|
||||
|
||||
|
||||
class TestLiteLLMVertexBackendErrors:
|
||||
"""Through the real backend: raw text stays in the log, hint reaches client."""
|
||||
|
||||
@pytest.fixture
|
||||
def litellm_mod(self, monkeypatch):
|
||||
mod = pytest.importorskip("headroom.backends.litellm")
|
||||
if not mod.LITELLM_AVAILABLE:
|
||||
pytest.skip("litellm not installed")
|
||||
|
||||
async def boom(**_kw):
|
||||
raise RuntimeError(_SECRET_RAW)
|
||||
|
||||
monkeypatch.setattr(mod, "acompletion", boom)
|
||||
return mod
|
||||
|
||||
_BODY = {"model": "claude-sonnet-4-6", "messages": [{"role": "user", "content": "hi"}]}
|
||||
_OPENAI_BODY = {"model": "gemini-3.5-flash", "messages": [{"role": "user", "content": "hi"}]}
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_message(self, litellm_mod, caplog):
|
||||
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
|
||||
with caplog.at_level(logging.ERROR):
|
||||
resp = await backend.send_message(dict(self._BODY), {})
|
||||
message = resp.body["error"]["message"]
|
||||
assert "[headroom] hint:" in message
|
||||
assert not any(leak in message for leak in _LEAKS)
|
||||
assert "secret-proj-123" in caplog.text
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_openai_message(self, litellm_mod):
|
||||
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
|
||||
resp = await backend.send_openai_message(dict(self._OPENAI_BODY), {})
|
||||
message = resp.body["error"]["message"]
|
||||
assert "[headroom] hint:" in message
|
||||
assert not any(leak in message for leak in _LEAKS)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stream_message_error_event(self, litellm_mod):
|
||||
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
|
||||
events = [e async for e in backend.stream_message(dict(self._BODY, stream=True), {})]
|
||||
errors = [e.data["error"]["message"] for e in events if e.event_type == "error"]
|
||||
assert len(errors) == 1
|
||||
assert "[headroom] hint:" in errors[0]
|
||||
assert not any(leak in errors[0] for leak in _LEAKS)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stream_openai_message_error_chunk(self, litellm_mod):
|
||||
backend = litellm_mod.LiteLLMBackend(provider="vertex_ai")
|
||||
chunks = [c async for c in backend.stream_openai_message(dict(self._OPENAI_BODY), {})]
|
||||
assert chunks[-1] == "data: [DONE]\n\n"
|
||||
message = json.loads(chunks[-2].removeprefix("data: "))["error"]["message"]
|
||||
assert "[headroom] hint:" in message
|
||||
assert not any(leak in message for leak in _LEAKS)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_vertex_backend_gets_no_vertex_hint(self, litellm_mod):
|
||||
backend = litellm_mod.LiteLLMBackend(provider="openrouter")
|
||||
resp = await backend.send_message(dict(self._BODY), {})
|
||||
assert "[headroom] hint:" not in resp.body["error"]["message"]
|
||||
|
||||
|
||||
class TestVertexSdkPreflight:
|
||||
"""`--backend vertex` without the SDK must fail at startup, not per-request."""
|
||||
|
||||
@pytest.fixture
|
||||
def no_sdk(self, monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"headroom.providers.vertex.diagnostics.importlib.util.find_spec",
|
||||
lambda name: None if name == "vertexai" else object(),
|
||||
)
|
||||
|
||||
def _create(self, backend, **kwargs):
|
||||
return create_proxy_backend(
|
||||
backend=backend,
|
||||
anyllm_provider="",
|
||||
bedrock_region=None,
|
||||
logger=logging.getLogger("test"),
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"backend", ["vertex", "vertex_ai", "litellm-vertex", "google-vertex", "googlevertex"]
|
||||
)
|
||||
def test_every_vertex_alias_is_preflighted(self, no_sdk, backend):
|
||||
with pytest.raises(BackendUnavailableError) as excinfo:
|
||||
self._create(backend)
|
||||
assert "headroom-ai[proxy,vertex]" in str(excinfo.value)
|
||||
|
||||
def test_other_providers_are_unaffected(self, no_sdk):
|
||||
"""The preflight must not become a general-purpose backend gate."""
|
||||
self._create("litellm-openrouter")
|
||||
|
||||
def test_injected_backend_class_bypasses_preflight(self, no_sdk):
|
||||
sentinel = object()
|
||||
assert self._create("vertex", litellm_backend_cls=lambda **_: sentinel) is sentinel
|
||||
|
||||
def test_present_sdk_passes(self, monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"headroom.providers.vertex.diagnostics.importlib.util.find_spec",
|
||||
lambda name: object(),
|
||||
)
|
||||
assert self._create("vertex") is not None
|
||||
|
||||
def test_cli_prints_remedy_and_exits_2(self):
|
||||
click_testing = pytest.importorskip("click.testing")
|
||||
from headroom.cli.main import main
|
||||
|
||||
def unavailable(config, **kwargs):
|
||||
raise BackendUnavailableError("Vertex SDK is missing. remedy-text")
|
||||
|
||||
with patch("headroom.proxy.server.run_server", unavailable):
|
||||
result = click_testing.CliRunner().invoke(main, ["proxy"])
|
||||
assert result.exit_code == 2
|
||||
assert "Cannot start proxy: Vertex SDK is missing. remedy-text" in result.output
|
||||
assert "Traceback" not in result.output
|
||||
|
||||
def test_cli_multi_worker_fails_in_parent_before_serving(self, no_sdk):
|
||||
"""--workers >1 builds the app inside each worker, past the CLI's handler."""
|
||||
click_testing = pytest.importorskip("click.testing")
|
||||
from headroom.cli.main import main
|
||||
|
||||
with patch("headroom.proxy.server.run_server") as run_server:
|
||||
result = click_testing.CliRunner().invoke(
|
||||
main, ["proxy", "--backend", "vertex", "--workers", "2"]
|
||||
)
|
||||
assert result.exit_code == 2
|
||||
assert "headroom-ai[proxy,vertex]" in result.output
|
||||
run_server.assert_not_called()
|
||||
@@ -1273,6 +1273,203 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/6b/50/2a923c4dc02522ff9567f07c67ed1deca6d5fa9474f4708168b8088dfa37/gitpython-3.2.1-py3-none-any.whl", hash = "sha256:d66b0832aa8755c1a84892937282df0befb16c97e5a64aeebd68be9c5e323f53", size = 231548, upload-time = "2026-10-14T12:20:18.844Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-api-core"
|
||||
version = "2.25.2"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "google-auth" },
|
||||
{ name = "googleapis-common-protos" },
|
||||
{ name = "proto-plus" },
|
||||
{ name = "protobuf" },
|
||||
{ name = "requests" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/09/cd/63f1557235c2440fe0577acdbc32577c5c002684c58c7f4d770a92366a24/google_api_core-2.25.2.tar.gz", hash = "sha256:1c63aa6af0d0d5e37966f157a77f9396d820fba59f9e43e9415bc3dc5baff300", size = 166266, upload-time = "2025-10-03T00:07:34.778Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/c8/d8/894716a5423933f5c8d2d5f04b16f052a515f78e815dab0c2c6f1fd105dc/google_api_core-2.25.2-py3-none-any.whl", hash = "sha256:e9a8f62d363dc8424a8497f4c2a47d6bcda6c16514c935629c257ab5d10210e7", size = 162489, upload-time = "2025-10-03T00:07:32.924Z" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
grpc = [
|
||||
{ name = "grpcio" },
|
||||
{ name = "grpcio-status" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-auth"
|
||||
version = "2.62.0"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "cryptography" },
|
||||
{ name = "pyasn1-modules" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f5/b9/4b2528f30114b106e3c3a7298ba38d85e682fd1f83fe7c4a5b77c7677082/google_auth-2.62.0.tar.gz", hash = "sha256:0bef0ce54bdf9ce226c5d66e4264413bd918141c31bbe49fb52eac882f513d69", size = 398180, upload-time = "2026-10-12T19:20:48.328Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/eb/a3/87fe9b7203ec2e56ea6a57c75e6d632eaf9487c7f7978cdad1d801bee0af/google_auth-2.62.0-py3-none-any.whl", hash = "sha256:4ff4319aeb4ad128409759d397a9fcafad126d0031d241cc0dd6b9a00b43e3f3", size = 267911, upload-time = "2026-10-12T19:20:46.355Z" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
pyopenssl = [
|
||||
{ name = "cryptography" },
|
||||
]
|
||||
requests = [
|
||||
{ name = "requests" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-cloud-aiplatform"
|
||||
version = "1.148.1"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "docstring-parser" },
|
||||
{ name = "google-api-core", extra = ["grpc"] },
|
||||
{ name = "google-auth" },
|
||||
{ name = "google-cloud-bigquery" },
|
||||
{ name = "google-cloud-resource-manager" },
|
||||
{ name = "google-cloud-storage" },
|
||||
{ name = "google-genai" },
|
||||
{ name = "packaging" },
|
||||
{ name = "proto-plus" },
|
||||
{ name = "protobuf" },
|
||||
{ name = "pydantic" },
|
||||
{ name = "typing-extensions" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c9/f3/b2a9417014c93858a2e3266134f931eefd972c2d410b25d7b8782fc6f143/google_cloud_aiplatform-1.148.1.tar.gz", hash = "sha256:75d605fba34e68714bd08e1e482755d0a6e3ae972805f809d088e686c30879e7", size = 10278758, upload-time = "2026-04-17T23:45:26.738Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/56/5b/e3515d7bbba602c2b0f6a0da5431785e897252443682e4735d0e6873dc8f/google_cloud_aiplatform-1.148.1-py2.py3-none-any.whl", hash = "sha256:035101e2d8e65c6a706cc3930b2452de7ddcbde50dd130320fcea0d8b03b0c5a", size = 8434481, upload-time = "2026-04-17T23:45:22.919Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-cloud-bigquery"
|
||||
version = "3.42.3"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "google-api-core", extra = ["grpc"] },
|
||||
{ name = "google-auth", extra = ["pyopenssl"] },
|
||||
{ name = "google-cloud-core" },
|
||||
{ name = "google-resumable-media" },
|
||||
{ name = "packaging" },
|
||||
{ name = "python-dateutil" },
|
||||
{ name = "requests" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/0a/53/6a9c19cde15ffe3f218653e4f11d08b2ef97dad78c07c473bc95f8ce7aa9/google_cloud_bigquery-3.42.3.tar.gz", hash = "sha256:d03f8da5ed94aeae5457f3127216cb385392ba266bede25ea257aeec94512900", size = 518359, upload-time = "2026-07-30T18:15:19.314Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/0a/fe/a862130426b56c062dcbc2fcb5a9e4bec9fea9193821718d718f9f10be61/google_cloud_bigquery-3.42.3-py3-none-any.whl", hash = "sha256:81b9bfa3a5fa098a04351c1a12579d16f28b93b836e4556ef6410a01deebf418", size = 264652, upload-time = "2026-07-30T18:15:17.549Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-cloud-core"
|
||||
version = "2.6.0"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "google-api-core" },
|
||||
{ name = "google-auth" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/a8/dd/1eef226e470369b26824a505c34482c0b493bc35fe8e0c6b003b5feca21a/google_cloud_core-2.6.0.tar.gz", hash = "sha256:e76149739f90fac1fc6757c09f47eaccb3145b54adbd7759b0f7c4b235f46c83", size = 36001, upload-time = "2026-05-07T08:04:04.124Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/84/4a/98da8930ab109c73d9a5d13782a9ebb81ea8c111f6d534a567b71d23e52b/google_cloud_core-2.6.0-py3-none-any.whl", hash = "sha256:6d63ac8e5eca6d9e4319d0a1e2265fadcd7f1049904378caecfa01cf52dd869e", size = 29390, upload-time = "2026-05-07T08:02:34.672Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-cloud-resource-manager"
|
||||
version = "1.18.0"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "google-api-core", extra = ["grpc"] },
|
||||
{ name = "google-auth" },
|
||||
{ name = "grpc-google-iam-v1" },
|
||||
{ name = "grpcio" },
|
||||
{ name = "proto-plus" },
|
||||
{ name = "protobuf" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/12/70/d467d93528905700826d26a48ad16ac39d4b36dd6efb5a86fb338efa81aa/google_cloud_resource_manager-1.18.0.tar.gz", hash = "sha256:db689f800a14c66d041196a7fbb8bb8aae8dc87f28c2929e101a5ec766b15512", size = 463674, upload-time = "2026-06-22T23:22:27.09Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/3f/62/2dbecda08a32aa9d4e6909eba3e31731c2d4452687bb6364a8d7950aed74/google_cloud_resource_manager-1.18.0-py3-none-any.whl", hash = "sha256:69bab144acd75878ebe44b720903dc3d140cb7d3be3261eaa8bc81e48afaff33", size = 404211, upload-time = "2026-06-22T23:20:25.88Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-cloud-storage"
|
||||
version = "3.4.1"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "google-api-core" },
|
||||
{ name = "google-auth" },
|
||||
{ name = "google-cloud-core" },
|
||||
{ name = "google-crc32c" },
|
||||
{ name = "google-resumable-media" },
|
||||
{ name = "requests" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/bd/ef/7cefdca67a6c8b3af0ec38612f9e78e5a9f6179dd91352772ae1a9849246/google_cloud_storage-3.4.1.tar.gz", hash = "sha256:6f041a297e23a4b485fad8c305a7a6e6831855c208bcbe74d00332a909f82268", size = 17238203, upload-time = "2025-10-08T18:43:39.665Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/83/6e/b47d83d3a35231c6232566341b0355cce78fd4e6988a7343725408547b2c/google_cloud_storage-3.4.1-py3-none-any.whl", hash = "sha256:972764cc0392aa097be8f49a5354e22eb47c3f62370067fb1571ffff4a1c1189", size = 290142, upload-time = "2025-10-08T18:43:37.524Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-crc32c"
|
||||
version = "1.9.0"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/fa/25/9cb0c1c31c45b893eb8f11ae70b3f4309432d59b5acaebca5dbe791729a4/google_crc32c-1.9.0.tar.gz", hash = "sha256:7b8c84c3d159ab6817fe3f74e6e6cef099c3f95dcec3abc0d8afb1404642efbe", size = 14857, upload-time = "2026-09-24T21:39:32.067Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/0e/55/a2f07f15e624f0de79359b1a6c1deb59ec5061bd3b38744b3b2849400662/google_crc32c-1.9.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:457d0d9a4718fd52b1494eac5c200ad25beeadbdc91843d550a003910838589f", size = 31958, upload-time = "2026-09-24T21:19:00.994Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f8/b3/923743597b774bbcf12a7c3e00e48d745e15fd616ad7489a40a63fff8f2f/google_crc32c-1.9.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:ccfe40021fd6afe23361175cf7551e3cef5fd34dc1ebe319f14993a83579e0eb", size = 31805, upload-time = "2026-09-24T21:22:25.019Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/df/a6/4d0352fe889663e0d81cea7fc664ec9158727384de4a44ab10e9967a7682/google_crc32c-1.9.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fbef61a3794e011c65fb4396a196cf123a7f474fe5a443db8e5dd7d751b9e6d4", size = 38119, upload-time = "2026-09-24T21:38:06.634Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/aa/e3/26685384e4b66ff0928d9566ef6110a7df76029175a1842329d7e3515f10/google_crc32c-1.9.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:86764b99e7a607830d93cb5b75e0ec3ff6cb06d3c274624418473cee701900d4", size = 36677, upload-time = "2026-09-24T21:38:08.082Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/cb/ce/4e90102e84880e97d3cf935f2672ecd29191bdeacf57f01740f92debda00/google_crc32c-1.9.0-cp311-cp311-win_amd64.whl", hash = "sha256:43a2dc26f9be213fbe0b4fc4a1088c5d45cbfcb3247420ccc820f0fc3edeea86", size = 35096, upload-time = "2026-09-24T21:39:28.201Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e4/5d/0730e1b3a14d054d1466f2fec88dadf978509c749a3d96d8b069cc56d38a/google_crc32c-1.9.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:53fdafef58e230d0c946ab5f8446d123d9f548230a73b29c8b41c9546f268bc1", size = 31961, upload-time = "2026-09-24T21:19:01.724Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/dd/32/d085abaf2fd907121975b92245bb3480fb8be40c37d03f9d6c41857f84c3/google_crc32c-1.9.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:8b91f41645b15a720357183fa5716682ada441873e3c462c15f9714be36f146b", size = 31804, upload-time = "2026-09-24T21:22:25.81Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/94/78/dd1935432337e5da7af391a6fc9f161c1c8e9b9002a402b9190135fe1b59/google_crc32c-1.9.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:16865b477d7941712cb0e0aad8ad4815e984fb5fc16d3fdaef7d986e26e53c95", size = 37775, upload-time = "2026-09-24T21:38:09.249Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/9e/43/9db03635bb10188d93dcbab9baa2a8670a0da4e868b4370cdbd98d65fed8/google_crc32c-1.9.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3abb18297d9ef0ab120531838be0e6d68c9fa876570e11c229c48f2edac23ce7", size = 36347, upload-time = "2026-09-24T21:38:10.141Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/cf/eb/94dee516c846bd9382c3f566d8f8e5fb9e90599e45afeb697f9fc2533528/google_crc32c-1.9.0-cp312-cp312-win_amd64.whl", hash = "sha256:fb63a8d7fa2e95dcff1ca16af2f4d88b526fa5ff72d1696285884ac2d49b6963", size = 35097, upload-time = "2026-09-24T21:39:28.934Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/3f/34/cb484e8b6174f130f8c6dc79c733a9dd8869b410ad6511fb6104c46b973a/google_crc32c-1.9.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:f1dc17d987ddcc5eba12a7ce48f0eb93141dea236b170c1101151396edf2f0cf", size = 31960, upload-time = "2026-09-24T21:19:02.454Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/af/25/3e8e567bd48448e225ea27318ccf2b94e05124e7b8b97b13eaec9e127199/google_crc32c-1.9.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:f894a2877650b56201d26a012a257b76d54a68834dc3913a93830ca8a047b075", size = 31803, upload-time = "2026-09-24T21:22:27.008Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f0/18/bee0dd59ae622482dc6463636c79e4bde7c954d061c859c9256362c9931a/google_crc32c-1.9.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:4488f1553a9ab7e86cdedc833374a7e904031803b995dc0bd0be48c271fa6556", size = 37776, upload-time = "2026-09-24T21:38:11.056Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/fd/b6/e76e80fed5f2558273c7839e622f98095c9b36c719c7147e38e3c055cb70/google_crc32c-1.9.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0568b17ed90ac596f29400d99e243fd0cc6276766183def888d1bf8d1dc13827", size = 36350, upload-time = "2026-09-24T21:38:12.138Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/87/34/165542bfa99dfef91a76471cc48cce74b8ff4e295722896087ab2b8e8611/google_crc32c-1.9.0-cp313-cp313-win_amd64.whl", hash = "sha256:8583ec21d56b565d68ab2963cc7e21b3b271247c29b04286068255ef65f221bd", size = 35090, upload-time = "2026-09-24T21:39:29.764Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8f/eb/43ea41f4061a1cad87b2b6559c98e960e45bf551fe66f83d833b98aaf0c9/google_crc32c-1.9.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:6a3b2c8a343c570ed8100a7627c20badfd92c6caa2067093a86be45af27f5b1b", size = 31963, upload-time = "2026-09-24T21:19:03.208Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/45/d2/a968c0c29ccd2b0c980ff4f9e3f7035cee28c23a1c57541825cc8221858c/google_crc32c-1.9.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:13179f7e3282617923e957b8e54b8f9c3968030f48640a9f47fd7c5c38c4a215", size = 31805, upload-time = "2026-09-24T21:22:27.917Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/03/73/388e493d6c3e252e37165d22efe5a1361f872a24425391b999822861b23a/google_crc32c-1.9.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:265233aff33d835f5b909584fe36ab29647b598c271b661a300001099109e53e", size = 37925, upload-time = "2026-09-24T21:38:13.32Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/98/36/190d32caa363ef25d685f422ed1bbf93ff1140fb22fd4d90f24cec209977/google_crc32c-1.9.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:dee799544cae42a42b17a88e38b59cf2c271051dc001da2117a8ff240ffa0548", size = 36495, upload-time = "2026-09-24T21:38:14.211Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d3/fd/81cefea6adae7bd92abb23d4567d199f6485a20ec0a305ca5fa04c52b9c5/google_crc32c-1.9.0-cp314-cp314-win_amd64.whl", hash = "sha256:af73200fa9791ccd380f3598235dba8d82b8af0905df045b3dc60b59836e8ddd", size = 35535, upload-time = "2026-09-24T21:39:30.52Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c5/18/19d4f17f3f33f8fdffcb3e1e69219d6f7ec2c359c160867b04dac1d0a64d/google_crc32c-1.9.0-cp315-cp315-macosx_12_0_arm64.whl", hash = "sha256:e6e8be8a94436079cb5340f6d495d9d7ba30124d8b952703994c739c7c06e236", size = 31977, upload-time = "2026-09-24T21:19:03.976Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/81/b4/8010372c4b46f2ee2352dfdb630c397570cd85522a315df024ad2f9459aa/google_crc32c-1.9.0-cp315-cp315-macosx_12_0_x86_64.whl", hash = "sha256:f2b64641bca27497b986b9d87883014035aa904cb4fa333407c6752b3afee9ba", size = 31809, upload-time = "2026-09-24T21:22:29.1Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c5/f8/7e33845d6b90ce1cf37cfabf25cb859277c7d3533ef1b6b1e1ca58581549/google_crc32c-1.9.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f97c3806dcea41c29c04965347b0e12481561b75e0045dc7a4f69d75dec5d9b1", size = 38359, upload-time = "2026-09-24T21:38:14.983Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/36/ff/556b2423f449a7515af6b8222a4d7833cbe09ff3e8d2f0b80471f5f6d02e/google_crc32c-1.9.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0abe7e202c25909869c35672ab0f2fe748a7acf276eb78577332a7c38999740f", size = 36858, upload-time = "2026-09-24T21:38:15.799Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/40/71/4733f1b7c921d04a2bb9b9916cf66498bf7ad0860a06289413830da83192/google_crc32c-1.9.0-cp315-cp315-win_amd64.whl", hash = "sha256:5695c8b9327e040b2aba12c6659b0acb5995314ef0af0192da66e662e011103b", size = 35534, upload-time = "2026-09-24T21:39:31.337Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-genai"
|
||||
version = "1.75.0"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "anyio" },
|
||||
{ name = "distro" },
|
||||
{ name = "google-auth", extra = ["requests"] },
|
||||
{ name = "httpx" },
|
||||
{ name = "pydantic" },
|
||||
{ name = "requests" },
|
||||
{ name = "sniffio" },
|
||||
{ name = "tenacity" },
|
||||
{ name = "typing-extensions" },
|
||||
{ name = "websockets" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/9d/59/3ed61240ef20b3ae6ed54e82c6f8b6d1f194947bc6679679dd6cdb037594/google_genai-1.75.0.tar.gz", hash = "sha256:56bac3991b311c93f980c0a2abcd287b672146905df1fbd71c92ed633d5a07cf", size = 539039, upload-time = "2026-05-04T22:48:54.857Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/2d/b6/552d40e96da22921eb1fead7c14b00b5b5473a20e45959488660fab35ee2/google_genai-1.75.0-py3-none-any.whl", hash = "sha256:8dc4c096e7d6288c3087f6893f582fe52468932464781edb8193bd92b9fefb2c", size = 793726, upload-time = "2026-05-04T22:48:53.033Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "google-resumable-media"
|
||||
version = "2.11.0"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "google-crc32c" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/cf/64/df6a482d5aa39d7f7be186d892377d605cae6c88525fd851d456e8bbe9c9/google_resumable_media-2.11.0.tar.gz", hash = "sha256:febd83686752799661b4de575f0b993c5c25c349a5362556fc4d7be164056a37", size = 2164914, upload-time = "2026-09-29T19:26:13.546Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/5e/2e/4f0a152f2e576e496f31ba1c3c62ed174a8878d06008916a7edc58b1bb28/google_resumable_media-2.11.0-py3-none-any.whl", hash = "sha256:f43d15e6a7f818f762eaead0f369c551f8275a4179c9d6225d0d259f49b87b5d", size = 81515, upload-time = "2026-09-29T19:25:47.31Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "googleapis-common-protos"
|
||||
version = "1.74.0"
|
||||
@@ -1285,6 +1482,11 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b6/b0/be5d3329badb9230b765de6eea66b73abd5944bdeb5afb3562ddcd80ae84/googleapis_common_protos-1.74.0-py3-none-any.whl", hash = "sha256:702216f78610bb510e3f12ac3cafd281b7ac45cc5d86e90ad87e4d301a3426b5", size = 300743, upload-time = "2026-04-02T21:22:49.108Z" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
grpc = [
|
||||
{ name = "grpcio" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "greenlet"
|
||||
version = "3.4.0"
|
||||
@@ -1332,6 +1534,20 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/db/72/85ae954d734703ab48e622c59d4ce35d77ce840c265814af9c078cacc7aa/greenlet-3.4.0-cp314-cp314t-win_amd64.whl", hash = "sha256:1a4a48f24681300c640f143ba7c404270e1ebbbcf34331d7104a4ff40f8ea705", size = 245554, upload-time = "2026-04-08T17:03:50.044Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "grpc-google-iam-v1"
|
||||
version = "0.14.4"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "googleapis-common-protos", extra = ["grpc"] },
|
||||
{ name = "grpcio" },
|
||||
{ name = "protobuf" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/44/4f/d098419ad0bfc06c9ce440575f05aa22d8973b6c276e86ac7890093d3c37/grpc_google_iam_v1-0.14.4.tar.gz", hash = "sha256:392b3796947ed6334e61171d9ab06bf7eb357f554e5fc7556ad7aab6d0e17038", size = 23706, upload-time = "2026-04-01T01:57:49.813Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/89/22/c2dd50c09bf679bd38173656cd4402d2511e563b33bc88f90009cf50613c/grpc_google_iam_v1-0.14.4-py3-none-any.whl", hash = "sha256:412facc320fcbd94034b4df3d557662051d4d8adfa86e0ddb4dca70a3f739964", size = 32675, upload-time = "2026-04-01T01:57:47.69Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "grpcio"
|
||||
version = "1.80.0"
|
||||
@@ -1383,6 +1599,20 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e5/8c/bbe6baf2557262834f2070cf668515fa308b2d38a4bbf771f8f7872a7036/grpcio-1.80.0-cp314-cp314-win_amd64.whl", hash = "sha256:3b01e1f5464c583d2f567b2e46ff0d516ef979978f72091fd81f5ab7fa6e2e7f", size = 5019457, upload-time = "2026-03-30T08:48:37.308Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "grpcio-status"
|
||||
version = "1.71.2"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "googleapis-common-protos" },
|
||||
{ name = "grpcio" },
|
||||
{ name = "protobuf" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/fd/d1/b6e9877fedae3add1afdeae1f89d1927d296da9cf977eca0eb08fb8a460e/grpcio_status-1.71.2.tar.gz", hash = "sha256:c7a97e176df71cdc2c179cd1847d7fc86cca5832ad12e9798d7fed6b7a1aab50", size = 13677, upload-time = "2025-06-28T04:24:05.426Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/67/58/317b0134129b556a93a3b0afe00ee675b5657f0155509e22fcb853bafe2d/grpcio_status-1.71.2-py3-none-any.whl", hash = "sha256:803c98cb6a8b7dc6dbb785b1111aed739f241ab5e9da0bba96888aa74704cfd3", size = 14424, upload-time = "2025-06-28T04:23:42.136Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "gunicorn"
|
||||
version = "26.0.0"
|
||||
@@ -1682,6 +1912,9 @@ strands = [
|
||||
vector = [
|
||||
{ name = "hnswlib" },
|
||||
]
|
||||
vertex = [
|
||||
{ name = "google-cloud-aiplatform" },
|
||||
]
|
||||
voice = [
|
||||
{ name = "onnxruntime", version = "1.23.2", source = { registry = "https://pypi.org/simple/" }, marker = "python_full_version < '3.14' and platform_machine == 'x86_64' and sys_platform == 'darwin'" },
|
||||
{ name = "onnxruntime", version = "1.26.0", source = { registry = "https://pypi.org/simple/" }, marker = "platform_machine != 'x86_64' or sys_platform != 'darwin'" },
|
||||
@@ -1717,6 +1950,7 @@ requires-dist = [
|
||||
{ name = "fastapi", marker = "extra == 'proxy'", specifier = ">=0.100.0" },
|
||||
{ name = "fastembed", marker = "(python_full_version < '3.14' and extra == 'relevance') or (platform_machine != 'x86_64' and extra == 'relevance') or (sys_platform != 'darwin' and extra == 'relevance')", specifier = ">=0.6.0" },
|
||||
{ name = "filelock", specifier = ">=4.2.1" },
|
||||
{ name = "google-cloud-aiplatform", marker = "extra == 'vertex'", specifier = ">=1.38" },
|
||||
{ name = "gunicorn", marker = "sys_platform != 'win32' and extra == 'proxy-prod'", specifier = ">=21.0.0" },
|
||||
{ name = "headroom-ai", extras = ["lean"], marker = "extra == 'sandbox'" },
|
||||
{ name = "headroom-ai", extras = ["proxy"], marker = "extra == 'proxy-prod'" },
|
||||
@@ -1814,7 +2048,7 @@ requires-dist = [
|
||||
{ name = "xlwt", marker = "extra == 'dev'", specifier = ">=1.3.0" },
|
||||
{ name = "zstandard", marker = "extra == 'proxy'", specifier = ">=0.20.0" },
|
||||
]
|
||||
provides-extras = ["proxy", "proxy-prod", "code", "ml", "memory", "vector", "memory-stack", "pytorch-mps", "relevance", "image", "reports", "spreadsheet", "otel", "anyllm", "langchain", "langgraph", "agno", "strands", "autogen", "mcp", "voice", "voice-train", "evals", "bedrock", "redis", "html", "dev", "all", "lean", "sandbox"]
|
||||
provides-extras = ["proxy", "proxy-prod", "vertex", "code", "ml", "memory", "vector", "memory-stack", "pytorch-mps", "relevance", "image", "reports", "spreadsheet", "otel", "anyllm", "langchain", "langgraph", "agno", "strands", "autogen", "mcp", "voice", "voice-train", "evals", "bedrock", "redis", "html", "dev", "all", "lean", "sandbox"]
|
||||
|
||||
[[package]]
|
||||
name = "hf-xet"
|
||||
@@ -4150,6 +4384,18 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/5b/5a/bc7b4a4ef808fa59a816c17b20c4bef6884daebbdf627ff2a161da67da19/propcache-0.4.1-py3-none-any.whl", hash = "sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237", size = 13305, upload-time = "2025-10-08T19:49:00.792Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "proto-plus"
|
||||
version = "1.28.2"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "protobuf" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/73/3e/29e0d6a2c5adde6ab5772253fd16ab346324026b89a66e354689c86d0584/proto_plus-1.28.2.tar.gz", hash = "sha256:26d843eb99c1e32fdf1d20ff0faae56607f7748fe774acf9ecd5cfe6c6472501", size = 58063, upload-time = "2026-07-22T16:28:29.119Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/9d/84/4e9a53a062d4073c74897a6bd20fff74d55307341b3e85c081002462b3ef/proto_plus-1.28.2-py3-none-any.whl", hash = "sha256:b874236fcac2358f601e4330bcb76cb8b89c851303ccf4078408b3d4774d1c52", size = 50693, upload-time = "2026-07-22T16:28:24.059Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "protobuf"
|
||||
version = "5.29.6"
|
||||
@@ -4280,6 +4526,27 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/50/f2/c0e76a0b451ffdf0cf788932e182758eb7558953f4f27f1aff8e2518b653/pyarrow-23.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:527e8d899f14bd15b740cd5a54ad56b7f98044955373a17179d5956ddb93d9ce", size = 28365807, upload-time = "2026-02-16T10:14:03.892Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyasn1"
|
||||
version = "0.6.5"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/88/7e/3c0da41061a2021cac7cd524fd57092adbe5b515e8bf9005702e366802e2/pyasn1-0.6.5.tar.gz", hash = "sha256:6b11616a6bcf3a40e635f082bb232fa57b5e0dd7f517bd7ddd69ec3794508a47", size = 155152, upload-time = "2026-10-27T06:32:49.125Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/19/1f/3eefd5c84c35cae44549ce1c7f7e58f7945b34f5a6bce07a135438522073/pyasn1-0.6.5-py3-none-any.whl", hash = "sha256:4017b0fce125096b8573b39c989fdb1a4b19cf761e683398184550fea0e66587", size = 84807, upload-time = "2026-10-27T06:32:48.045Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyasn1-modules"
|
||||
version = "0.4.2"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "pyasn1" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/e9/e6/78ebbb10a8c8e4b61a59249394a4a594c1a7af95593dc933a349c8d00964/pyasn1_modules-0.4.2.tar.gz", hash = "sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6", size = 307892, upload-time = "2025-03-28T02:41:22.17Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/47/8d/d529b5d697919ba8c11ad626e835d4039be708a35b0d22de83a269a6682c/pyasn1_modules-0.4.2-py3-none-any.whl", hash = "sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a", size = 181259, upload-time = "2025-03-28T02:41:19.028Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyclipper"
|
||||
version = "1.4.0"
|
||||
|
||||
+78
-11
@@ -9,11 +9,13 @@ Google documents Gemini generation on Vertex with `generateContent` and
|
||||
shape. See Google Cloud's model inference reference:
|
||||
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference
|
||||
|
||||
Google Cloud REST calls authenticate with a bearer access token. For local
|
||||
development, Google documents both `gcloud auth print-access-token` and
|
||||
`gcloud auth application-default print-access-token`; Application Default
|
||||
Credentials search `GOOGLE_APPLICATION_CREDENTIALS`, local ADC files, and
|
||||
attached service accounts in that order. See:
|
||||
Google Cloud REST calls authenticate with a bearer access token. Use
|
||||
**`gcloud auth application-default print-access-token`**: a plain
|
||||
`gcloud auth print-access-token` user token is rejected by
|
||||
`aiplatform.googleapis.com` with `401 ACCESS_TOKEN_TYPE_UNSUPPORTED` for many
|
||||
identities. Application Default Credentials search
|
||||
`GOOGLE_APPLICATION_CREDENTIALS`, local ADC files, and attached service accounts
|
||||
in that order. Tokens expire after ~1 hour. See:
|
||||
|
||||
- https://docs.cloud.google.com/docs/authentication/rest
|
||||
- https://docs.cloud.google.com/docs/authentication/application-default-credentials
|
||||
@@ -26,19 +28,46 @@ Set the Vertex regional host explicitly:
|
||||
headroom proxy --vertex-api-url https://us-central1-aiplatform.googleapis.com
|
||||
```
|
||||
|
||||
The same setting is available through `VERTEX_TARGET_API_URL`.
|
||||
The same setting is available through `VERTEX_TARGET_API_URL`. Left unset, the
|
||||
proxy derives the host from each request's `locations/{location}` segment, so
|
||||
one proxy serves every region plus `global`.
|
||||
|
||||
### Picking a location
|
||||
|
||||
Vertex serves each publisher model in only a subset of locations, and this is
|
||||
the most common source of a confusing `404` during onboarding:
|
||||
|
||||
| Model | Where it serves |
|
||||
| --- | --- |
|
||||
| `gemini-flash-latest`, `gemini-flash-lite-latest` | `global` only |
|
||||
| `gemini-3.5-flash` | `global`, `europe-west2`, `asia-northeast1`, `asia-south1`, `asia-southeast1` |
|
||||
| `claude-sonnet-4-6` (and 4.6-or-older Claude) | `global`, `us-east5`, `europe-west1`, `asia-southeast1` |
|
||||
| Claude 4.7+ | `global` or the `us`/`eu` multi-region only -- no named regions |
|
||||
|
||||
**Gemini 3.x has no US regional endpoint.** Start with `global` and only pin a
|
||||
region when you need Provisioned Throughput or data residency. Partner models
|
||||
(Claude, Llama, Mistral) also need a one-time per-project enable in Model
|
||||
Garden before they serve.
|
||||
|
||||
When Vertex rejects a request with `401`, `403`, `404` or `429`, Headroom appends a `[headroom] hint: ...` note to
|
||||
`error.message` (also emitted as an `x-headroom-hint` header and a proxy WARNING)
|
||||
naming the likely fix. Hints are fixed text and never echo the request path,
|
||||
project, credentials or exception detail; raw provider errors stay in the proxy
|
||||
log.
|
||||
|
||||
## Gemini On Vertex
|
||||
|
||||
Send Vertex publisher paths through the proxy unchanged:
|
||||
|
||||
```bash
|
||||
ACCESS_TOKEN="$(gcloud auth print-access-token)"
|
||||
LOCATION="global"
|
||||
MODEL="gemini-flash-latest"
|
||||
ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
|
||||
|
||||
curl -sS \
|
||||
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
|
||||
-H "Content-Type: application/json" \
|
||||
http://127.0.0.1:8787/v1/projects/PROJECT_ID/locations/us-central1/publishers/google/models/gemini-2.0-flash:generateContent \
|
||||
http://127.0.0.1:8787/v1/projects/PROJECT_ID/locations/${LOCATION}/publishers/google/models/${MODEL}:generateContent \
|
||||
-d '{
|
||||
"contents": [
|
||||
{
|
||||
@@ -49,6 +78,17 @@ curl -sS \
|
||||
}'
|
||||
```
|
||||
|
||||
## Google Gen AI SDK (Proxy)
|
||||
|
||||
Point the `google-genai` SDK at Headroom with
|
||||
`genai.Client(vertexai=True, project=..., location="global", http_options={"base_url": "http://127.0.0.1:8787"})`.
|
||||
`examples/vertex_genai_sdk_demo.py` runs this end to end: it starts the proxy,
|
||||
probes it (including a thinking config) and tears it down.
|
||||
|
||||
Note the `google-genai` SDK only builds `publishers/google/...` paths, so it
|
||||
cannot reach Claude on Vertex; use the `publishers/anthropic/...:rawPredict`
|
||||
route below for that.
|
||||
|
||||
Supported passthrough actions:
|
||||
|
||||
- `generateContent`
|
||||
@@ -66,6 +106,32 @@ The Python proxy preserves caller-supplied Google bearer auth. The native Rust
|
||||
proxy path additionally resolves GCP ADC and injects the bearer token for the
|
||||
Anthropic publisher route.
|
||||
|
||||
These native routes are a straight passthrough: they need **no** `--backend`
|
||||
flag and no extra beyond `[proxy]`.
|
||||
|
||||
`--backend vertex` (aliases: `vertex_ai`, `litellm-vertex`, `google-vertex`) is a
|
||||
different mode. It routes Anthropic *Messages* traffic — `/v1/messages` — through
|
||||
LiteLLM, and needs the Vertex SDK:
|
||||
|
||||
```bash
|
||||
pip install "headroom-ai[proxy,vertex]"
|
||||
```
|
||||
|
||||
That extra is kept out of `[proxy]` because `google-cloud-aiplatform` and its
|
||||
transitive tree add roughly 175 MB, and only this backend uses it. Select the
|
||||
backend without it and the proxy refuses to start rather than failing on every
|
||||
request:
|
||||
|
||||
```text
|
||||
Cannot start proxy: Vertex backend selected but the Vertex SDK is missing. ...
|
||||
```
|
||||
|
||||
Do not combine the two. With `--backend vertex` set, Claude (`publishers/anthropic`) requests
|
||||
are re-routed through LiteLLM, which takes the project, region and model from
|
||||
its own configuration (`VERTEXAI_PROJECT` / `VERTEXAI_LOCATION`) and ignores the
|
||||
ones in your URL — so a request that works without the flag can come back `404`
|
||||
for a model you never named.
|
||||
|
||||
## Claude Code with Headroom compression (validated)
|
||||
|
||||
To run **Claude Code** against Claude-on-Vertex **with Headroom compressing the
|
||||
@@ -82,6 +148,7 @@ Short version: run Claude Code in **normal Anthropic mode** (`ANTHROPIC_BASE_URL
|
||||
> URL before sending a request ("model … not available on your vertex deployment"),
|
||||
> so the proxy is never reached. Use the Anthropic-mode runbook above instead.
|
||||
>
|
||||
> ⚠️ Two easy-to-miss requirements: `pip install "google-cloud-aiplatform>=1.38"`
|
||||
> (LiteLLM `vertex_ai` provider) and the `--code-aware` flag (code compression is
|
||||
> off by default). Without them you get a 500 or `tokens_saved: 0`.
|
||||
> ⚠️ Two easy-to-miss requirements: `pip install "headroom-ai[proxy,vertex]"`
|
||||
> (the LiteLLM `vertex_ai` provider needs the Vertex SDK) and the `--code-aware`
|
||||
> flag (code compression is off by default). Without the first the proxy refuses
|
||||
> to start; without the second you get `tokens_saved: 0`.
|
||||
|
||||
Reference in New Issue
Block a user