mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(proxy): unified multi-upstream routing for Anthropic Cloud and local oMLX (#4597)
## Description Adds unified multi-upstream routing for Anthropic Cloud and local Apple Silicon MLX (`oMLX`) inference on a single Headroom proxy instance, with strict allowlist-based local model selection, post-rewrite destination resolution, cloud credential isolation, and trusted upstream extra-header support. ## Why (Rationale & Problem) Developers running hybrid cloud + local Apple Silicon MLX workflows (`oMLX` / `jundot/omlx` serving `/v1/messages` alongside Anthropic Cloud) previously had to run separate Headroom proxy instances or reconfigure upstream URLs when switching between Claude and local `oMLX` models. ## What For (Use Cases & Value) Allows a single Headroom proxy instance to dynamically route explicit local `omlx/*` / `local/*` / MLX model requests (or requests to `/omlx/v1/messages`) to a local `oMLX` server (`HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL`) while keeping all Claude models (`claude-haiku-4-5-20251001`, `claude-haiku-5-5`, `claude-fable-5`, `claude-sonnet-4-5`, etc.) and unmatched models on `api.anthropic.com` with full context compression and telemetry. ## How (Implementation, Invariants & Verification) - **`headroom/proxy/handlers/anthropic.py`**: - `is_omlx_local_model`: Only routes to `omlx_target` when the model explicitly matches local model prefixes (`omlx/`, `mlx/`, `local/`, `mlx-community/`, `gemma`, `qwen`, `llama`, `mistral`, `deepseek`, `phi`, `lfm`, `smollm`, `starcoder`, `codellama`, `yi-`, `nemotron`) or the operator-configured `HEADROOM_OMLX_MODELS` allowlist. Never routes `claude*` models (unless explicitly prefixed with `omlx/` or `local/`) to oMLX. - Evaluates `_maybe_route_model` **before** local/cloud target selection so cloud-to-local and local-to-cloud model rewrites select the matching destination URL and credential policy. - Strips leading `omlx/` or `local/` routing prefixes from `body["model"]` (recording `omlx_strip_model_prefix` on `body_mutation_tracker`) before forwarding to `oMLX`'s native `/v1/messages` endpoint. - Strips inbound Anthropic cloud credentials (`x-api-key` and `authorization`) when forwarding to local oMLX so cloud tokens never leak to the local process, optionally substituting `HEADROOM_OMLX_API_KEY` / `OMLX_API_KEY` when configured. - **`headroom/config.py`, `headroom/proxy/models.py`, `headroom/cli/proxy.py`, `headroom/proxy/server.py`, `headroom/providers/proxy_routes.py`**: - Consistent resolution of `OMLX_TARGET_API_URL`, `HEADROOM_OMLX_BASE_URL`, and `--omlx-api-url` across SDK config, `ProxyConfig`, Click CLI, and server factory startup. - **`headroom/proxy/upstream_trust.py`**: - Includes `omlx_target_api_url` in `trusted_upstream_hosts` so operator-configured `anthropic_extra_headers` are permitted for the designated oMLX gateway. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `resolve_omlx_target_api_url` in `headroom/config.py` and wired `--omlx-api-url` / `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL` across `headroom/cli/proxy.py`, `headroom/proxy/models.py`, `headroom/proxy/server.py`, and `headroom/providers/proxy_routes.py`. - Replaced open-ended `cloud_prefixes` fallback in `headroom/proxy/handlers/anthropic.py` with explicit local-prefix + `HEADROOM_OMLX_MODELS` allowlist matching (`is_omlx_local_model`) and `strip_omlx_model_prefix`. - Moved local/cloud target resolution after `_maybe_route_model` and stripped Anthropic cloud credentials (`x-api-key`, `authorization`) on local oMLX routes. - Added `omlx_target_api_url` to `_API_URL_ATTRS` and `trusted_upstream_hosts` in `headroom/proxy/upstream_trust.py`. - Added 42 unit tests in `tests/test_proxy/test_omlx_routing.py`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/test_proxy/test_omlx_routing.py tests/test_proxy/test_model_router_wiring.py tests/test_proxy/test_anthropic_upstream_header.py tests/test_upstream_guard.py -v 121 passed in 91.28s $ .venv/bin/ruff check headroom/config.py headroom/proxy/models.py headroom/proxy/upstream_trust.py headroom/providers/proxy_routes.py headroom/cli/proxy.py headroom/proxy/server.py headroom/proxy/handlers/anthropic.py tests/test_proxy/test_omlx_routing.py All checks passed! $ .venv/bin/mypy headroom/config.py headroom/proxy/models.py headroom/proxy/upstream_trust.py headroom/providers/proxy_routes.py headroom/cli/proxy.py Success: no issues found in 5 source files ``` ## Live Verification & Proof (CONTRIBUTING.md) - **Setup**: - macOS Apple Silicon (M4 Max), Python 3.14.10 in `.venv`, Headroom proxy configured with `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888` and `ANTHROPIC_API_KEY=sk-ant-api03-...`. - **Commands**: 1. `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888 .venv/bin/pytest tests/test_proxy/test_omlx_routing.py -v` 2. POST `/v1/messages` with `model="claude-haiku-4-5-20251001"`, `model="claude-haiku-5-5"`, `model="claude-fable-5"`, and `model="claude-sonnet-4-5"` while `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888` is active. 3. POST `/v1/messages` with `model="omlx/qwen3-8b"` and `headers={"x-api-key": "sk-ant-cloud-secret"}`. - **Observed results**: - All 33 supported Claude model IDs (including `claude-haiku-4-5-20251001`, `claude-haiku-5-5`, and `claude-fable-5`) routed to `https://api.anthropic.com/v1/messages` with cloud credentials preserved. - `model="omlx/qwen3-8b"` routed to `http://127.0.0.1:8888/v1/messages` with `body["model"]` normalized to `"qwen3-8b"` and inbound Anthropic `x-api-key` / `Authorization` headers stripped (or replaced by `HEADROOM_OMLX_API_KEY` when set). - **Untested cases**: - Multi-node remote oMLX cluster failover (single target URL per proxy instance is supported). ## Real Behavior Proof - Environment: macOS Darwin arm64 (Apple Silicon), Python 3.14.10, `HEADROOM_OMLX_BASE_URL=http://127.0.0.1:8888` - Exact command / steps: `.venv/bin/pytest tests/test_proxy/test_omlx_routing.py -v` - Observed result: 42 passed in 15.28s — verified cloud routing for all Claude models, local routing and prefix stripping for `omlx/*` / `local/*` / `gemma*` / `qwen*`, pre-target evaluation of `_maybe_route_model`, cloud credential stripping, and `trusted_upstream_hosts` inclusion. - Not tested: Windows named-pipe transport with local oMLX. ## Runtime Rollout Safety - Rollout-managed feature(s): None (opt-in via `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL` / `--omlx-api-url`). - Minimum rollout channel: stable (no-op unless operator sets `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL`). - Stable/default behavior changed: No — when `omlx_target_api_url` is unset, `/v1/messages` routing is 100% unchanged. - Kill switch / disable path: Unset `HEADROOM_OMLX_BASE_URL` and `OMLX_TARGET_API_URL`. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Unset `HEADROOM_OMLX_BASE_URL` / `OMLX_TARGET_API_URL` or revert commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ### Maintainer re-review of the credential-isolation update Reviewed updated author head5c45c8f964, already based on upstream mainc01cf07d94. The previous semantic merge hold no longer applies: the contributor rebased and integrated destination-aware header construction with main's earlier enforcement flow. Configured x-api-key/Authorization are filtered from local routes without overriding the local key; cloud routes retain configured headers, and cloud subscription notification is suppressed for local targets. Current discussion and prior Devin findings considered; scope remains Anthropic Messages-compatible oMLX. Fresh routing/backpressure/rate-limit/subscription verification: 134 passed, one skip, deprecation warnings only. All eight changed Python files pass Ruff lint/format and main-relative whitespace. A diagnostic-only external control restoring unfiltered extra-header selection makes all eight requested credential-isolation regressions fail; this does not modify the PR or claim an old-head test execution. Independent actual Uvicorn/TCP forwarding with controlled local Anthropic-compatible upstreams passed 12 cases: eight local credential-isolation combinations plus four cloud-retention controls. Dummy credentials only; subscription polling disabled for this header-only probe. No real oMLX model/server, cloud inference, or branch-specific native build claimed; current-main DLL supports local startup. The prior configured-header and current-main integration findings are addressed. No new source blocker found; formal approval awaits exact-head hosted qualification. No contributor code changed and nothing merged. ### Maintainer current-main requalification Exact head `1b9082fb4baf4cb2cf8984e241fdae77a4494e83`, upstream main `2b0315b2d4b0d54947050bbdad2de9e89df06378`. Requalified against upstream main2b0315b2d4. All 71 integration patches match the main delta after diff-location normalization; the contributor's destination-aware header policy is unchanged. 266 routing, backpressure, rate-limit, subscription and doctor tests pass, with one skip and three deprecation warnings. Twelve fresh Uvicorn/TCP cases pass: eight local credential-isolation combinations and four cloud-retention controls. These use dummy credentials and local Anthropic-compatible upstream fixtures, with subscription polling disabled. They do not qualify a real oMLX model/server or cloud inference. Ruff lint/format passes for all 33 changed Python files; offline lock and whitespace checks pass. The existing native DLL matches the recorded build-input hashes and artifact checksum; no fresh build is claimed. Current author and Devin discussions were considered, and the prior credential-isolation finding remains addressed. A direct fork push was denied with HTTP 403. GitHub's PR branch-update API applied the clean main merge instead. Its published tree,116abaccb5, exactly matches the tested candidate's complete tree. The test evidence therefore applies to this head. Approval awaits exact-head hosted checks; maintainers handle CI. No PR was merged. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JD Davis
parent
2568cbedaa
commit
f896161340
@@ -1123,6 +1123,14 @@ def dashboard(port: int | None, no_open: bool) -> None:
|
||||
default=None,
|
||||
help=("Custom Vertex AI regional API URL for publisher endpoints (env: VERTEX_TARGET_API_URL)"),
|
||||
)
|
||||
@click.option(
|
||||
"--omlx-api-url",
|
||||
default=None,
|
||||
help=(
|
||||
"Custom local oMLX API URL for hybrid local/cloud routing "
|
||||
"(env: HEADROOM_OMLX_BASE_URL or OMLX_TARGET_API_URL)"
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--region",
|
||||
default="us-west-2",
|
||||
@@ -1283,6 +1291,7 @@ def proxy(
|
||||
gemini_api_url: str | None,
|
||||
cloudcode_api_url: str | None,
|
||||
vertex_api_url: str | None,
|
||||
omlx_api_url: str | None,
|
||||
region: str,
|
||||
bedrock_region: str | None,
|
||||
bedrock_profile: str | None,
|
||||
@@ -1508,6 +1517,8 @@ def proxy(
|
||||
if memory_qdrant_api_key is not None:
|
||||
qdrant_overrides["memory_qdrant_api_key"] = memory_qdrant_api_key
|
||||
|
||||
from headroom.config import resolve_omlx_target_api_url
|
||||
|
||||
config = ProxyConfig(
|
||||
host=host,
|
||||
port=port,
|
||||
@@ -1521,6 +1532,7 @@ def proxy(
|
||||
gemini_api_url=provider_api_overrides.gemini,
|
||||
cloudcode_api_url=provider_api_overrides.cloudcode,
|
||||
vertex_api_url=provider_api_overrides.vertex,
|
||||
omlx_target_api_url=resolve_omlx_target_api_url(omlx_api_url, os.environ),
|
||||
mode=effective_mode,
|
||||
optimize=not no_optimize,
|
||||
cache_enabled=not no_cache,
|
||||
|
||||
+24
-1
@@ -4,8 +4,9 @@ from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import InitVar, dataclass, field
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
@@ -14,6 +15,25 @@ from typing import Any, Literal
|
||||
from headroom.models.config import ML_MODEL_DEFAULTS
|
||||
from headroom.rollout import RolloutSnapshot, resolve_rollout
|
||||
|
||||
OMLX_TARGET_API_URL_ENV = "OMLX_TARGET_API_URL"
|
||||
HEADROOM_OMLX_BASE_URL_ENV = "HEADROOM_OMLX_BASE_URL"
|
||||
HEADROOM_OMLX_URL_ENV = "HEADROOM_OMLX_URL"
|
||||
|
||||
|
||||
def resolve_omlx_target_api_url(
|
||||
explicit: str | None = None,
|
||||
environ: Mapping[str, str] | None = None,
|
||||
) -> str | None:
|
||||
"""Resolve the local oMLX target API URL from an explicit override or environment."""
|
||||
if explicit and explicit.strip():
|
||||
return explicit.strip().rstrip("/")
|
||||
env = environ if environ is not None else os.environ
|
||||
for key in (OMLX_TARGET_API_URL_ENV, HEADROOM_OMLX_BASE_URL_ENV, HEADROOM_OMLX_URL_ENV):
|
||||
value = env.get(key)
|
||||
if value and value.strip():
|
||||
return value.strip().rstrip("/")
|
||||
return None
|
||||
|
||||
|
||||
class HeadroomMode(str, Enum):
|
||||
"""Operating modes for Headroom."""
|
||||
@@ -963,11 +983,14 @@ class HeadroomConfig:
|
||||
# Canonical pipeline lifecycle extensions
|
||||
pipeline_extensions: list[Any] = field(default_factory=list)
|
||||
discover_pipeline_extensions: bool = True
|
||||
omlx_target_api_url: str | None = None
|
||||
|
||||
def __post_init__(self, content_router_enabled: bool | None = None) -> None:
|
||||
if self.rollout is None:
|
||||
requested = ("tool_result_interceptors",) if self.intercept_tool_results else ()
|
||||
self.rollout = resolve_rollout(requested=requested)
|
||||
if self.omlx_target_api_url is None:
|
||||
self.omlx_target_api_url = resolve_omlx_target_api_url()
|
||||
|
||||
def get_context_limit(self, model: str) -> int | None:
|
||||
"""
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request, WebSocket
|
||||
@@ -331,6 +332,19 @@ def register_provider_routes(app: FastAPI, proxy: Any) -> None:
|
||||
)
|
||||
return await proxy.handle_anthropic_messages(request)
|
||||
|
||||
@app.post("/omlx/v1/messages")
|
||||
async def omlx_messages(request: Request):
|
||||
from headroom.config import resolve_omlx_target_api_url
|
||||
|
||||
omlx_target = (
|
||||
getattr(proxy.config, "omlx_target_api_url", None)
|
||||
or resolve_omlx_target_api_url(environ=os.environ)
|
||||
or "http://127.0.0.1:8888"
|
||||
)
|
||||
return await proxy.handle_anthropic_messages(
|
||||
request, upstream_base_url=omlx_target.rstrip("/")
|
||||
)
|
||||
|
||||
@app.post("/anthropic/v1/messages")
|
||||
async def foundry_anthropic_messages(request: Request):
|
||||
normalize_request_path(request, "/v1/messages")
|
||||
|
||||
@@ -38,6 +38,7 @@ from headroom.agent_savings import proxy_pipeline_kwargs
|
||||
from headroom.ccr.context_tracker import looks_like_claude_code_compact_summary
|
||||
from headroom.ccr.marker_resolution import resolve_markers_in_response
|
||||
from headroom.ccr.tool_calls import has_ccr_retrieve_tool
|
||||
from headroom.config import resolve_omlx_target_api_url
|
||||
from headroom.copilot_auth import apply_copilot_api_auth, is_copilot_upstream_url
|
||||
from headroom.log_safety import describe_exception
|
||||
from headroom.pipeline import PipelineStage, summarize_routing_markers
|
||||
@@ -87,6 +88,144 @@ from headroom.utils import format_exception_message
|
||||
|
||||
logger = logging.getLogger("headroom.proxy")
|
||||
|
||||
_EXPLICIT_OMLX_ROUTING_PREFIXES: tuple[str, ...] = ("omlx/", "mlx/", "local/")
|
||||
_LOCAL_OMLX_MODEL_PREFIXES: tuple[str, ...] = (
|
||||
"omlx/",
|
||||
"mlx/",
|
||||
"local/",
|
||||
"mlx-community/",
|
||||
"gemma",
|
||||
"qwen",
|
||||
"llama",
|
||||
"mistral",
|
||||
"deepseek",
|
||||
"phi",
|
||||
"lfm",
|
||||
"smollm",
|
||||
"starcoder",
|
||||
"codellama",
|
||||
"yi-",
|
||||
"nemotron",
|
||||
)
|
||||
|
||||
|
||||
def is_omlx_local_model(model: Any) -> bool:
|
||||
"""Return True only when ``model`` explicitly targets local oMLX inference.
|
||||
|
||||
Unmatched models and all ``claude*`` models (unless explicitly prefixed with
|
||||
``omlx/``, ``mlx/``, or ``local/``) stay on the Anthropic cloud target so
|
||||
supported Claude models (e.g. ``claude-haiku-4-5-20251001``,
|
||||
``claude-haiku-5-5``, ``claude-fable-5``) are never misrouted to localhost.
|
||||
"""
|
||||
model_str = str(model or "").strip().lower()
|
||||
if not model_str:
|
||||
return False
|
||||
if model_str.startswith(_EXPLICIT_OMLX_ROUTING_PREFIXES):
|
||||
return True
|
||||
if model_str.startswith("claude") or "claude-" in model_str:
|
||||
return False
|
||||
allowlist_raw = os.environ.get("HEADROOM_OMLX_MODELS", "")
|
||||
if allowlist_raw:
|
||||
allowlist = {entry.strip().lower() for entry in allowlist_raw.split(",") if entry.strip()}
|
||||
if model_str in allowlist:
|
||||
return True
|
||||
return model_str.startswith(_LOCAL_OMLX_MODEL_PREFIXES)
|
||||
|
||||
|
||||
def strip_omlx_model_prefix(model: str) -> str:
|
||||
"""Strip a leading ``omlx/`` or ``local/`` routing prefix from ``model``."""
|
||||
lowered = model.lower()
|
||||
for prefix in ("omlx/", "local/"):
|
||||
if lowered.startswith(prefix):
|
||||
stripped = model[len(prefix) :].strip()
|
||||
if stripped:
|
||||
return stripped
|
||||
break
|
||||
return model
|
||||
|
||||
|
||||
_ANTHROPIC_CLOUD_AUTH_HEADERS: frozenset[str] = frozenset({"x-api-key", "authorization"})
|
||||
|
||||
|
||||
def _select_extra_headers_for_destination(
|
||||
extra: dict[str, str] | None,
|
||||
*,
|
||||
routed_to_omlx: bool,
|
||||
) -> dict[str, str] | None:
|
||||
"""Select configured extra headers appropriate for the target destination.
|
||||
|
||||
Operator-configured ``anthropic_extra_headers`` cloud credentials
|
||||
(``x-api-key``, ``Authorization``) belong exclusively to Anthropic Cloud and
|
||||
must never be merged into local oMLX requests or override
|
||||
``HEADROOM_OMLX_API_KEY``. Non-credential gateway headers still flow through
|
||||
``merge_extra_headers`` subject to ``is_trusted_upstream``.
|
||||
"""
|
||||
if not extra or not routed_to_omlx:
|
||||
return extra
|
||||
filtered = {k: v for k, v in extra.items() if k.lower() not in _ANTHROPIC_CLOUD_AUTH_HEADERS}
|
||||
return filtered or None
|
||||
|
||||
|
||||
def _build_outbound_anthropic_headers(
|
||||
stripped_inbound_headers: dict[str, str],
|
||||
extra_headers: dict[str, str] | None,
|
||||
*,
|
||||
routed_to_omlx: bool,
|
||||
upstream_url: str | None,
|
||||
config: Any,
|
||||
) -> dict[str, str]:
|
||||
"""Build destination-aware outbound headers for Anthropic Cloud or local oMLX."""
|
||||
from headroom.proxy.helpers import merge_extra_headers
|
||||
|
||||
base_headers = dict(stripped_inbound_headers)
|
||||
if routed_to_omlx:
|
||||
for cred_key in [k for k in base_headers if k.lower() in _ANTHROPIC_CLOUD_AUTH_HEADERS]:
|
||||
base_headers.pop(cred_key, None)
|
||||
omlx_api_key = (
|
||||
os.environ.get("HEADROOM_OMLX_API_KEY") or os.environ.get("OMLX_API_KEY") or ""
|
||||
).strip()
|
||||
if omlx_api_key:
|
||||
base_headers["x-api-key"] = omlx_api_key
|
||||
base_headers["authorization"] = f"Bearer {omlx_api_key}"
|
||||
return merge_extra_headers(
|
||||
base_headers,
|
||||
_select_extra_headers_for_destination(extra_headers, routed_to_omlx=routed_to_omlx),
|
||||
upstream_url=upstream_url,
|
||||
config=config,
|
||||
)
|
||||
|
||||
|
||||
def _normalize_omlx_upstream_path(path: str) -> str:
|
||||
"""Normalize ``/omlx/...`` proxy route prefixes to the upstream Anthropic path."""
|
||||
if path.startswith("/omlx"):
|
||||
return path[len("/omlx") :] or "/v1/messages"
|
||||
return path
|
||||
|
||||
|
||||
def _notify_subscription_tracker_if_cloud(
|
||||
request: Request,
|
||||
headers: dict[str, str],
|
||||
*,
|
||||
routed_to_omlx: bool,
|
||||
) -> None:
|
||||
"""Notify the Anthropic OAuth subscription tracker only on cloud-bound requests."""
|
||||
if routed_to_omlx:
|
||||
return
|
||||
auth_header = next((v for k, v in headers.items() if k.lower() == "authorization"), "")
|
||||
if auth_header.startswith("Bearer ") and not auth_header.startswith("Bearer sk-ant-api"):
|
||||
from headroom.subscription.tracker import get_subscription_tracker as _get_sub_tracker
|
||||
|
||||
sub_tracker = _get_sub_tracker()
|
||||
if sub_tracker is not None:
|
||||
from headroom.subscription.credential_policy import is_local_operator_connection
|
||||
|
||||
# Only the local operator's bearer may become the polled
|
||||
# account; a network caller only marks activity (01-F16).
|
||||
sub_tracker.notify_active(
|
||||
auth_header,
|
||||
from_local_operator=is_local_operator_connection(request),
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _thinking_estimator() -> Callable[[str], int] | None:
|
||||
@@ -1380,17 +1519,27 @@ class AnthropicHandlerMixin:
|
||||
_strip_internal_headers,
|
||||
apply_keep_last_turns,
|
||||
log_outbound_headers,
|
||||
merge_extra_headers,
|
||||
)
|
||||
|
||||
omlx_target = getattr(
|
||||
self.config, "omlx_target_api_url", None
|
||||
) or resolve_omlx_target_api_url(environ=os.environ)
|
||||
routed_to_omlx = request.url.path.startswith("/omlx") or bool(
|
||||
upstream_base_url
|
||||
and omlx_target
|
||||
and upstream_base_url.rstrip("/") == omlx_target.rstrip("/")
|
||||
)
|
||||
|
||||
_pre_strip_count = sum(1 for k in headers if k.lower().startswith("x-headroom-"))
|
||||
headers = _strip_internal_headers(headers)
|
||||
_stripped_inbound_headers = _strip_internal_headers(headers)
|
||||
# `upstream_base_url` is the per-request `x-headroom-base-url`
|
||||
# override when the client sent one. These headers are secrets, so
|
||||
# they only travel to a host the operator designated.
|
||||
headers = merge_extra_headers(
|
||||
headers,
|
||||
# they only travel to a host the operator designated, and cloud
|
||||
# credentials in `anthropic_extra_headers` are withheld on local oMLX routes.
|
||||
headers = _build_outbound_anthropic_headers(
|
||||
_stripped_inbound_headers,
|
||||
self.config.anthropic_extra_headers,
|
||||
routed_to_omlx=routed_to_omlx,
|
||||
upstream_url=upstream_base_url,
|
||||
config=self.config,
|
||||
)
|
||||
@@ -1401,28 +1550,6 @@ class AnthropicHandlerMixin:
|
||||
request_id=request_id,
|
||||
)
|
||||
|
||||
# Subscription tracker: notify on OAuth requests (not API-key requests)
|
||||
_auth_header = headers.get("authorization", "")
|
||||
if _auth_header.startswith("Bearer ") and not _auth_header.startswith(
|
||||
"Bearer sk-ant-api"
|
||||
):
|
||||
from headroom.subscription.tracker import (
|
||||
get_subscription_tracker as _get_sub_tracker,
|
||||
)
|
||||
|
||||
_sub_tracker = _get_sub_tracker()
|
||||
if _sub_tracker is not None:
|
||||
from headroom.subscription.credential_policy import (
|
||||
is_local_operator_connection,
|
||||
)
|
||||
|
||||
# Only the local operator's bearer may become the polled
|
||||
# account; a network caller only marks activity (01-F16).
|
||||
_sub_tracker.notify_active(
|
||||
_auth_header,
|
||||
from_local_operator=is_local_operator_connection(request),
|
||||
)
|
||||
|
||||
# Rate limiting
|
||||
if self.rate_limiter:
|
||||
# One identity rule for every provider: peer-owned, and
|
||||
@@ -1499,10 +1626,34 @@ class AnthropicHandlerMixin:
|
||||
)
|
||||
from headroom.proxy.route_advice import resolver_for
|
||||
|
||||
backend = resolver_for(self).for_request(request, body=body)
|
||||
passthrough_model = body.get("model") or model_override or "unknown"
|
||||
if not isinstance(passthrough_model, str):
|
||||
passthrough_model = "unknown"
|
||||
passthrough_bytes = original_body_bytes
|
||||
if not upstream_base_url and omlx_target and is_omlx_local_model(passthrough_model):
|
||||
upstream_base_url = omlx_target.rstrip("/")
|
||||
routed_to_omlx = True
|
||||
headers = _build_outbound_anthropic_headers(
|
||||
_stripped_inbound_headers,
|
||||
self.config.anthropic_extra_headers,
|
||||
routed_to_omlx=True,
|
||||
upstream_url=upstream_base_url,
|
||||
config=self.config,
|
||||
)
|
||||
if routed_to_omlx and isinstance(body.get("model"), str):
|
||||
stripped_passthrough_model = strip_omlx_model_prefix(passthrough_model)
|
||||
if stripped_passthrough_model != passthrough_model:
|
||||
passthrough_model = stripped_passthrough_model
|
||||
body["model"] = stripped_passthrough_model
|
||||
from headroom.proxy.body_forwarding import serialize_body_canonical
|
||||
|
||||
passthrough_bytes = serialize_body_canonical(body)
|
||||
_notify_subscription_tracker_if_cloud(
|
||||
request, headers, routed_to_omlx=routed_to_omlx
|
||||
)
|
||||
backend = (
|
||||
None if routed_to_omlx else resolver_for(self).for_request(request, body=body)
|
||||
)
|
||||
if backend is not None:
|
||||
if model_override and not body.get("model"):
|
||||
body["model"] = model_override
|
||||
@@ -1520,14 +1671,16 @@ class AnthropicHandlerMixin:
|
||||
)
|
||||
passthrough_base = upstream_base_url or self.ANTHROPIC_API_URL
|
||||
passthrough_url = build_anthropic_upstream_url(
|
||||
passthrough_base, request.url.path, request.url.query
|
||||
passthrough_base,
|
||||
_normalize_omlx_upstream_path(request.url.path),
|
||||
request.url.query,
|
||||
)
|
||||
assert self.http_client is not None, "http_client must be initialized"
|
||||
upstream_request = self.http_client.build_request(
|
||||
request.method,
|
||||
passthrough_url,
|
||||
headers=headers,
|
||||
content=original_body_bytes,
|
||||
content=passthrough_bytes,
|
||||
)
|
||||
await _finalize_pre_upstream()
|
||||
return await self.handle_passthrough(
|
||||
@@ -1559,6 +1712,7 @@ class AnthropicHandlerMixin:
|
||||
):
|
||||
body["model"] = model
|
||||
body_mutation_tracker.mark_mutated("sanitize_model_id")
|
||||
|
||||
messages = body.get("messages", [])
|
||||
# Strip streaming-only "index" keys from request content blocks BEFORE any
|
||||
# prefix-cache tracking or compression. The proxy's streaming reconstruction
|
||||
@@ -1668,6 +1822,45 @@ class AnthropicHandlerMixin:
|
||||
model, messages, body, body_mutation_tracker, _bypass
|
||||
)
|
||||
|
||||
_prev_routed_to_omlx = routed_to_omlx
|
||||
routed_to_omlx = request.url.path.startswith("/omlx") or bool(
|
||||
upstream_base_url
|
||||
and omlx_target
|
||||
and upstream_base_url.rstrip("/") == omlx_target.rstrip("/")
|
||||
)
|
||||
if not upstream_base_url and omlx_target:
|
||||
is_local = is_omlx_local_model(model)
|
||||
if is_local:
|
||||
upstream_base_url = omlx_target.rstrip("/")
|
||||
routed_to_omlx = True
|
||||
logger.info(
|
||||
"[%s] Unified proxy: routing local model '%s' to oMLX: %s",
|
||||
request_id,
|
||||
model,
|
||||
upstream_base_url,
|
||||
)
|
||||
if routed_to_omlx != _prev_routed_to_omlx:
|
||||
headers = _build_outbound_anthropic_headers(
|
||||
_stripped_inbound_headers,
|
||||
self.config.anthropic_extra_headers,
|
||||
routed_to_omlx=routed_to_omlx,
|
||||
upstream_url=upstream_base_url,
|
||||
config=self.config,
|
||||
)
|
||||
if routed_to_omlx and isinstance(model, str):
|
||||
stripped_model = strip_omlx_model_prefix(model)
|
||||
if stripped_model != model:
|
||||
model = stripped_model
|
||||
body["model"] = stripped_model
|
||||
body_mutation_tracker.mark_mutated("omlx_strip_model_prefix")
|
||||
_notify_subscription_tracker_if_cloud(request, headers, routed_to_omlx=routed_to_omlx)
|
||||
_auth_header = (
|
||||
headers.get("authorization")
|
||||
or headers.get("Authorization")
|
||||
or request.headers.get("authorization")
|
||||
or ""
|
||||
)
|
||||
|
||||
# NOTE: Upstream temporarily disabled broad image compression due to
|
||||
# token-counting inaccuracies. We only compress the latest non-frozen
|
||||
# user turn later in this handler to preserve Anthropic prefix caching.
|
||||
@@ -4323,9 +4516,12 @@ class AnthropicHandlerMixin:
|
||||
# every one of those turns to "anthropic" on the dashboard.
|
||||
# For a non-Copilot base the builder only joins base + path, so the
|
||||
# URL itself is unchanged.
|
||||
request_path = request.url.path
|
||||
if request_path.startswith("/omlx"):
|
||||
request_path = request_path[len("/omlx") :] or "/v1/messages"
|
||||
url = build_anthropic_upstream_url(
|
||||
upstream_base_url or self.ANTHROPIC_API_URL,
|
||||
request.url.path,
|
||||
request_path,
|
||||
request.url.query,
|
||||
)
|
||||
# Keep a conversation on one replica of providers that route on a
|
||||
@@ -5444,8 +5640,10 @@ class AnthropicHandlerMixin:
|
||||
# counters. Provider-specific OAuth/subscription
|
||||
# accounting — stays outside the funnel (different
|
||||
# concern, only fires for Bearer-not-sk-ant tokens).
|
||||
if _auth_header.startswith("Bearer ") and not _auth_header.startswith(
|
||||
"Bearer sk-ant-api"
|
||||
if (
|
||||
not routed_to_omlx
|
||||
and _auth_header.startswith("Bearer ")
|
||||
and not _auth_header.startswith("Bearer sk-ant-api")
|
||||
):
|
||||
from headroom.subscription.tracker import (
|
||||
get_subscription_tracker as _get_sub_tracker,
|
||||
|
||||
@@ -12,6 +12,7 @@ from dataclasses import InitVar, dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any, Literal
|
||||
|
||||
from headroom.config import resolve_omlx_target_api_url
|
||||
from headroom.memory import qdrant_env
|
||||
from headroom.providers.registry import ProviderApiOverrides
|
||||
from headroom.proxy.buffered_ccr_response import DEFAULT_BUFFERED_CCR_GRACE_SECONDS
|
||||
@@ -164,6 +165,9 @@ class ProxyConfig:
|
||||
gemini_api_url: str | None = None # Custom Gemini API URL override
|
||||
cloudcode_api_url: str | None = None # Custom Cloud Code Assist API URL override
|
||||
vertex_api_url: str | None = None # Custom Vertex AI regional API URL override
|
||||
omlx_target_api_url: str | None = (
|
||||
None # Custom oMLX local inference API URL (e.g. http://127.0.0.1:8888)
|
||||
)
|
||||
# Extra headers merged into (and overriding) forwarded Anthropic/OpenAI requests.
|
||||
# JSON-object config knobs; see settings_store's anthropic_extra_headers/
|
||||
# openai_extra_headers and providers.registry.resolve_extra_headers.
|
||||
@@ -587,6 +591,8 @@ class ProxyConfig:
|
||||
def __post_init__(self, smart_routing: bool | None = None) -> None:
|
||||
if self.rollout is None:
|
||||
self.rollout = resolve_rollout()
|
||||
if self.omlx_target_api_url is None:
|
||||
self.omlx_target_api_url = resolve_omlx_target_api_url()
|
||||
# ``read_maturation`` remains a concrete, already-resolved runtime
|
||||
# setting for programmatic/config-file callers. The CLI composition
|
||||
# root derives it from this same snapshot before constructing the
|
||||
|
||||
@@ -84,6 +84,7 @@ from headroom.config import (
|
||||
DEFAULT_EXCLUDE_TOOLS,
|
||||
CacheAlignerConfig,
|
||||
ReadLifecycleConfig,
|
||||
resolve_omlx_target_api_url,
|
||||
)
|
||||
from headroom.dashboard import get_dashboard_html
|
||||
from headroom.log_safety import describe_exception, safe_id
|
||||
@@ -6608,6 +6609,7 @@ def _proxy_config_from_env() -> ProxyConfig:
|
||||
),
|
||||
ccr_buffered_chat_streaming=_get_env_bool("HEADROOM_CCR_BUFFERED_CHAT_STREAMING", False),
|
||||
vertex_api_url=os.environ.get("VERTEX_TARGET_API_URL"),
|
||||
omlx_target_api_url=resolve_omlx_target_api_url(environ=os.environ),
|
||||
backend=_get_env_str("HEADROOM_BACKEND", "anthropic"),
|
||||
bedrock_region=_get_env_str("HEADROOM_BEDROCK_REGION", "us-west-2"),
|
||||
bedrock_profile=os.environ.get("AWS_PROFILE"),
|
||||
@@ -7112,6 +7114,10 @@ if __name__ == "__main__":
|
||||
"--vertex-api-url",
|
||||
help=f"Custom Vertex AI regional API URL (default: {DEFAULT_VERTEX_API_URL})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--omlx-api-url",
|
||||
help="Custom local oMLX API URL (env: HEADROOM_OMLX_BASE_URL or OMLX_TARGET_API_URL)",
|
||||
)
|
||||
|
||||
# Backend (anthropic direct, bedrock, openrouter, anyllm, or litellm-<provider>)
|
||||
parser.add_argument(
|
||||
@@ -7433,6 +7439,9 @@ if __name__ == "__main__":
|
||||
min_value=0,
|
||||
),
|
||||
vertex_api_url=_get_env_str("VERTEX_TARGET_API_URL", args.vertex_api_url),
|
||||
omlx_target_api_url=resolve_omlx_target_api_url(
|
||||
getattr(args, "omlx_api_url", None), os.environ
|
||||
),
|
||||
# Backend settings
|
||||
backend=_get_env_str("HEADROOM_BACKEND", args.backend), # type: ignore[arg-type]
|
||||
bedrock_region=_get_env_str("HEADROOM_BEDROCK_REGION", args.bedrock_region),
|
||||
|
||||
@@ -38,6 +38,8 @@ import os
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from headroom.config import resolve_omlx_target_api_url
|
||||
|
||||
logger = logging.getLogger("headroom.proxy")
|
||||
|
||||
ALLOWED_HOSTS_ENV = "HEADROOM_UPSTREAM_ALLOWED_HOSTS"
|
||||
@@ -50,6 +52,7 @@ _API_URL_ATTRS = (
|
||||
"cloudcode_api_url",
|
||||
"vertex_api_url",
|
||||
"bedrock_api_url",
|
||||
"omlx_target_api_url",
|
||||
)
|
||||
|
||||
# Hosts that are always operator-designated: they are what the provider targets
|
||||
@@ -106,6 +109,9 @@ def trusted_upstream_hosts(config: Any = None) -> frozenset[str]:
|
||||
host = url_host(getattr(config, attr, None))
|
||||
if host:
|
||||
hosts.add(host)
|
||||
omlx_env_host = url_host(resolve_omlx_target_api_url(environ=os.environ))
|
||||
if omlx_env_host:
|
||||
hosts.add(omlx_env_host)
|
||||
hosts |= _env_allowed_hosts()
|
||||
return frozenset(hosts)
|
||||
|
||||
|
||||
@@ -0,0 +1,461 @@
|
||||
"""Unit tests for hybrid Anthropic Cloud + local oMLX proxy routing (PR #4597)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.config import HeadroomConfig, resolve_omlx_target_api_url
|
||||
from headroom.providers.anthropic import ANTHROPIC_CONTEXT_LIMITS, ANTHROPIC_PRICING
|
||||
from headroom.proxy.handlers.anthropic import is_omlx_local_model, strip_omlx_model_prefix
|
||||
from headroom.proxy.model_router import ModelRoute, ModelRouterConfig
|
||||
from headroom.proxy.server import ProxyConfig, _proxy_config_from_env, create_app
|
||||
from headroom.proxy.upstream_trust import is_trusted_upstream, trusted_upstream_hosts
|
||||
|
||||
MESSAGES = "/v1/messages"
|
||||
OMLX_MESSAGES = "/omlx/v1/messages"
|
||||
OMLX_URL = "http://127.0.0.1:8888"
|
||||
|
||||
|
||||
def _install_fake_client(proxy) -> MagicMock:
|
||||
"""Replace proxy.http_client so forwarding never touches the network."""
|
||||
response = httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "msg_test",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": "ok"}],
|
||||
"model": "test",
|
||||
"stop_reason": "end_turn",
|
||||
"usage": {"input_tokens": 10, "output_tokens": 2},
|
||||
},
|
||||
request=httpx.Request("POST", "http://upstream/v1/messages"),
|
||||
)
|
||||
client = MagicMock()
|
||||
client.post = AsyncMock(return_value=response)
|
||||
client.request = AsyncMock(return_value=response)
|
||||
client.send = AsyncMock(return_value=response)
|
||||
client.build_request = MagicMock(
|
||||
return_value=httpx.Request("POST", "http://upstream/v1/messages", content=b"{}")
|
||||
)
|
||||
client.aclose = AsyncMock()
|
||||
proxy.http_client = client
|
||||
return client
|
||||
|
||||
|
||||
def _make_config(**overrides) -> ProxyConfig:
|
||||
base = {
|
||||
"optimize": False,
|
||||
"image_optimize": False,
|
||||
"cache_enabled": False,
|
||||
"rate_limit_enabled": False,
|
||||
"cost_tracking_enabled": False,
|
||||
"ccr_inject_tool": False,
|
||||
"ccr_handle_responses": False,
|
||||
"ccr_context_tracking": False,
|
||||
"mode": "token",
|
||||
}
|
||||
base.update(overrides)
|
||||
return ProxyConfig(**base)
|
||||
|
||||
|
||||
def _forwarded_url(http: MagicMock) -> str:
|
||||
return str(http.post.call_args.args[0])
|
||||
|
||||
|
||||
def _forwarded_headers(http: MagicMock) -> dict[str, str]:
|
||||
raw = http.post.call_args.kwargs.get("headers") or {}
|
||||
return {k.lower(): v for k, v in raw.items()}
|
||||
|
||||
|
||||
def _forwarded_body(http: MagicMock) -> dict:
|
||||
content = http.post.call_args.kwargs["content"]
|
||||
return json.loads(content)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"claude_model",
|
||||
[
|
||||
"claude-haiku-4-5-20251001",
|
||||
"claude-haiku-5-5",
|
||||
"claude-fable-5",
|
||||
"claude-fable-5-1",
|
||||
"claude-sonnet-4-5",
|
||||
"claude-sonnet-5",
|
||||
"claude-opus-4-8",
|
||||
*sorted(set(ANTHROPIC_CONTEXT_LIMITS) | set(ANTHROPIC_PRICING)),
|
||||
],
|
||||
)
|
||||
def test_claude_models_route_to_anthropic_cloud_even_with_omlx_configured(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
claude_model: str,
|
||||
) -> None:
|
||||
monkeypatch.setenv("HEADROOM_OMLX_BASE_URL", OMLX_URL)
|
||||
monkeypatch.setenv("OMLX_TARGET_API_URL", OMLX_URL)
|
||||
assert not is_omlx_local_model(claude_model)
|
||||
|
||||
app = create_app(_make_config(omlx_target_api_url=OMLX_URL))
|
||||
with TestClient(app) as client:
|
||||
http = _install_fake_client(client.app.state.proxy)
|
||||
resp = client.post(
|
||||
MESSAGES,
|
||||
json={
|
||||
"model": claude_model,
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
},
|
||||
headers={"x-api-key": "sk-ant-cloud-secret"},
|
||||
)
|
||||
|
||||
assert resp.status_code == 200
|
||||
assert _forwarded_url(http) == "https://api.anthropic.com/v1/messages"
|
||||
assert _forwarded_headers(http).get("x-api-key") == "sk-ant-cloud-secret"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("local_model", "expected_wire_model"),
|
||||
[
|
||||
("omlx/qwen3-8b", "qwen3-8b"),
|
||||
("local/gemma-4-9b", "gemma-4-9b"),
|
||||
("gemma-4-9b", "gemma-4-9b"),
|
||||
("qwen2.5-coder", "qwen2.5-coder"),
|
||||
],
|
||||
)
|
||||
def test_explicit_local_models_route_to_omlx_target(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
local_model: str,
|
||||
expected_wire_model: str,
|
||||
) -> None:
|
||||
monkeypatch.delenv("HEADROOM_OMLX_API_KEY", raising=False)
|
||||
monkeypatch.delenv("OMLX_API_KEY", raising=False)
|
||||
assert is_omlx_local_model(local_model)
|
||||
assert strip_omlx_model_prefix(local_model) == expected_wire_model
|
||||
|
||||
app = create_app(_make_config(omlx_target_api_url=OMLX_URL))
|
||||
with TestClient(app) as client:
|
||||
http = _install_fake_client(client.app.state.proxy)
|
||||
resp = client.post(
|
||||
MESSAGES,
|
||||
json={
|
||||
"model": local_model,
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
},
|
||||
)
|
||||
|
||||
assert resp.status_code == 200
|
||||
assert _forwarded_url(http) == f"{OMLX_URL}/v1/messages"
|
||||
assert _forwarded_body(http)["model"] == expected_wire_model
|
||||
|
||||
|
||||
def test_allowlist_env_routes_custom_model_to_omlx(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_OMLX_MODELS", "custom-apple-silicon-7b, another-local")
|
||||
assert is_omlx_local_model("custom-apple-silicon-7b")
|
||||
assert not is_omlx_local_model("unlisted-cloud-model")
|
||||
|
||||
|
||||
def test_maybe_route_model_runs_before_local_or_cloud_target_selection() -> None:
|
||||
# 1. Cloud -> local rewrite must route to omlx_target and strip omlx/ prefix.
|
||||
cloud_to_local_cfg = _make_config(
|
||||
omlx_target_api_url=OMLX_URL,
|
||||
model_router=ModelRouterConfig(
|
||||
enabled=True,
|
||||
routes=(
|
||||
ModelRoute(
|
||||
to_model="omlx/qwen3-8b",
|
||||
max_input_tokens=100_000,
|
||||
require_no_tools=True,
|
||||
name="offload-local",
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
app = create_app(cloud_to_local_cfg)
|
||||
with TestClient(app) as client:
|
||||
http = _install_fake_client(client.app.state.proxy)
|
||||
resp = client.post(
|
||||
MESSAGES,
|
||||
json={
|
||||
"model": "claude-sonnet-4-5",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
headers={"x-api-key": "sk-ant-cloud-secret"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert _forwarded_url(http) == f"{OMLX_URL}/v1/messages"
|
||||
assert _forwarded_body(http)["model"] == "qwen3-8b"
|
||||
assert "x-api-key" not in _forwarded_headers(http)
|
||||
|
||||
# 2. Local -> cloud rewrite must route to Anthropic cloud and keep cloud credentials.
|
||||
local_to_cloud_cfg = _make_config(
|
||||
omlx_target_api_url=OMLX_URL,
|
||||
model_router=ModelRouterConfig(
|
||||
enabled=True,
|
||||
routes=(
|
||||
ModelRoute(
|
||||
to_model="claude-haiku-4-5-20251001",
|
||||
max_input_tokens=100_000,
|
||||
require_no_tools=True,
|
||||
name="escalate-cloud",
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
app2 = create_app(local_to_cloud_cfg)
|
||||
with TestClient(app2) as client:
|
||||
http2 = _install_fake_client(client.app.state.proxy)
|
||||
resp2 = client.post(
|
||||
MESSAGES,
|
||||
json={
|
||||
"model": "omlx/qwen3-8b",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
headers={"x-api-key": "sk-ant-cloud-secret"},
|
||||
)
|
||||
assert resp2.status_code == 200
|
||||
assert _forwarded_url(http2) == "https://api.anthropic.com/v1/messages"
|
||||
assert _forwarded_body(http2)["model"] == "claude-haiku-4-5-20251001"
|
||||
assert _forwarded_headers(http2).get("x-api-key") == "sk-ant-cloud-secret"
|
||||
|
||||
|
||||
def test_cloud_credentials_stripped_or_replaced_when_forwarding_to_omlx(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.delenv("HEADROOM_OMLX_API_KEY", raising=False)
|
||||
monkeypatch.delenv("OMLX_API_KEY", raising=False)
|
||||
|
||||
app = create_app(_make_config(omlx_target_api_url=OMLX_URL))
|
||||
with TestClient(app) as client:
|
||||
http = _install_fake_client(client.app.state.proxy)
|
||||
resp = client.post(
|
||||
MESSAGES,
|
||||
json={
|
||||
"model": "omlx/qwen3-8b",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
headers={
|
||||
"x-api-key": "sk-ant-api03-secret",
|
||||
"Authorization": "Bearer sk-ant-oat01-secret",
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
forwarded_headers = _forwarded_headers(http)
|
||||
assert "x-api-key" not in forwarded_headers
|
||||
assert "authorization" not in forwarded_headers
|
||||
|
||||
# When HEADROOM_OMLX_API_KEY is configured, replace with the local key.
|
||||
monkeypatch.setenv("HEADROOM_OMLX_API_KEY", "local-omlx-token")
|
||||
with TestClient(app) as client:
|
||||
http2 = _install_fake_client(client.app.state.proxy)
|
||||
resp2 = client.post(
|
||||
OMLX_MESSAGES,
|
||||
json={
|
||||
"model": "omlx/qwen3-8b",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
headers={
|
||||
"x-api-key": "sk-ant-api03-secret",
|
||||
"Authorization": "Bearer sk-ant-oat01-secret",
|
||||
},
|
||||
)
|
||||
assert resp2.status_code == 200
|
||||
forwarded_headers2 = _forwarded_headers(http2)
|
||||
assert forwarded_headers2.get("x-api-key") == "local-omlx-token"
|
||||
assert forwarded_headers2.get("authorization") == "Bearer local-omlx-token"
|
||||
|
||||
|
||||
def test_headroom_omlx_base_url_and_omlx_target_api_url_env_consistency(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.delenv("OMLX_TARGET_API_URL", raising=False)
|
||||
monkeypatch.delenv("HEADROOM_OMLX_URL", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_OMLX_BASE_URL", "http://127.0.0.1:8000/")
|
||||
|
||||
assert resolve_omlx_target_api_url() == "http://127.0.0.1:8000"
|
||||
assert HeadroomConfig().omlx_target_api_url == "http://127.0.0.1:8000"
|
||||
assert ProxyConfig().omlx_target_api_url == "http://127.0.0.1:8000"
|
||||
assert _proxy_config_from_env().omlx_target_api_url == "http://127.0.0.1:8000"
|
||||
|
||||
from headroom.cli.proxy import proxy as proxy_cli_cmd
|
||||
|
||||
runner = CliRunner()
|
||||
help_res = runner.invoke(proxy_cli_cmd, ["--help"])
|
||||
assert help_res.exit_code == 0
|
||||
assert "--omlx-api-url" in help_res.output
|
||||
|
||||
|
||||
def test_omlx_target_api_url_included_in_trusted_upstream_hosts(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.delenv("OMLX_TARGET_API_URL", raising=False)
|
||||
monkeypatch.delenv("HEADROOM_OMLX_BASE_URL", raising=False)
|
||||
monkeypatch.delenv("HEADROOM_OMLX_URL", raising=False)
|
||||
|
||||
cfg = _make_config(
|
||||
omlx_target_api_url="http://omlx.internal:8888",
|
||||
anthropic_extra_headers={"X-Custom-Gateway-Auth": "gateway-secret"},
|
||||
)
|
||||
assert "omlx.internal" in trusted_upstream_hosts(cfg)
|
||||
assert is_trusted_upstream("http://omlx.internal:8888/v1/messages", cfg)
|
||||
|
||||
app = create_app(cfg)
|
||||
with TestClient(app) as client:
|
||||
http = _install_fake_client(client.app.state.proxy)
|
||||
resp = client.post(
|
||||
MESSAGES,
|
||||
json={
|
||||
"model": "omlx/qwen3-8b",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert _forwarded_headers(http).get("x-custom-gateway-auth") == "gateway-secret"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("endpoint", [MESSAGES, OMLX_MESSAGES])
|
||||
@pytest.mark.parametrize(
|
||||
("extra_header_name", "extra_header_value"),
|
||||
[
|
||||
("x-api-key", "dummy-cloud-secret"),
|
||||
("Authorization", "Bearer dummy-cloud-oauth"),
|
||||
],
|
||||
)
|
||||
@pytest.mark.parametrize("local_omlx_key", [None, "local-omlx-token"])
|
||||
def test_configured_anthropic_extra_headers_never_leak_or_override_on_omlx_routes(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
endpoint: str,
|
||||
extra_header_name: str,
|
||||
extra_header_value: str,
|
||||
local_omlx_key: str | None,
|
||||
) -> None:
|
||||
"""Cover all 8 destination-aware credential isolation combinations (PR #4597 review)."""
|
||||
monkeypatch.delenv("OMLX_API_KEY", raising=False)
|
||||
if local_omlx_key is None:
|
||||
monkeypatch.delenv("HEADROOM_OMLX_API_KEY", raising=False)
|
||||
else:
|
||||
monkeypatch.setenv("HEADROOM_OMLX_API_KEY", local_omlx_key)
|
||||
|
||||
cfg = _make_config(
|
||||
omlx_target_api_url=OMLX_URL,
|
||||
anthropic_extra_headers={
|
||||
extra_header_name: extra_header_value,
|
||||
"X-Custom-Gateway-Auth": "gateway-secret",
|
||||
},
|
||||
)
|
||||
app = create_app(cfg)
|
||||
with TestClient(app) as client:
|
||||
http = _install_fake_client(client.app.state.proxy)
|
||||
resp = client.post(
|
||||
endpoint,
|
||||
json={
|
||||
"model": "omlx/qwen3-8b",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
headers={
|
||||
"x-api-key": "inbound-cloud-secret",
|
||||
"Authorization": "Bearer inbound-cloud-oauth",
|
||||
},
|
||||
)
|
||||
|
||||
assert resp.status_code == 200
|
||||
assert _forwarded_url(http) == f"{OMLX_URL}/v1/messages"
|
||||
assert _forwarded_body(http)["model"] == "qwen3-8b"
|
||||
forwarded_headers = _forwarded_headers(http)
|
||||
assert forwarded_headers.get("x-custom-gateway-auth") == "gateway-secret"
|
||||
if local_omlx_key is None:
|
||||
assert "x-api-key" not in forwarded_headers
|
||||
assert "authorization" not in forwarded_headers
|
||||
else:
|
||||
assert forwarded_headers.get("x-api-key") == local_omlx_key
|
||||
assert forwarded_headers.get("authorization") == f"Bearer {local_omlx_key}"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("extra_header_name", "extra_header_value", "expected_lower_key"),
|
||||
[
|
||||
("x-api-key", "configured-cloud-api-key", "x-api-key"),
|
||||
("Authorization", "Bearer configured-cloud-oauth", "authorization"),
|
||||
],
|
||||
)
|
||||
def test_cloud_requests_retain_configured_anthropic_extra_headers_with_omlx_enabled(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
extra_header_name: str,
|
||||
extra_header_value: str,
|
||||
expected_lower_key: str,
|
||||
) -> None:
|
||||
monkeypatch.setenv("HEADROOM_OMLX_API_KEY", "local-omlx-token")
|
||||
cfg = _make_config(
|
||||
omlx_target_api_url=OMLX_URL,
|
||||
anthropic_extra_headers={extra_header_name: extra_header_value},
|
||||
)
|
||||
app = create_app(cfg)
|
||||
with TestClient(app) as client:
|
||||
http = _install_fake_client(client.app.state.proxy)
|
||||
resp = client.post(
|
||||
MESSAGES,
|
||||
json={
|
||||
"model": "claude-haiku-4-5-20251001",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
},
|
||||
headers={extra_header_name: "inbound-overridden-secret"},
|
||||
)
|
||||
|
||||
assert resp.status_code == 200
|
||||
assert _forwarded_url(http) == "https://api.anthropic.com/v1/messages"
|
||||
assert _forwarded_headers(http).get(expected_lower_key) == extra_header_value
|
||||
|
||||
|
||||
def test_subscription_tracker_skips_omlx_routes_and_notifies_cloud_routes(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
import headroom.subscription.tracker as sub_tracker_mod
|
||||
|
||||
fake_tracker = MagicMock()
|
||||
monkeypatch.setattr(sub_tracker_mod, "get_subscription_tracker", lambda: fake_tracker)
|
||||
monkeypatch.delenv("HEADROOM_OMLX_API_KEY", raising=False)
|
||||
monkeypatch.delenv("OMLX_API_KEY", raising=False)
|
||||
|
||||
cfg = _make_config(
|
||||
omlx_target_api_url=OMLX_URL,
|
||||
anthropic_extra_headers={"Authorization": "Bearer configured-cloud-oauth"},
|
||||
)
|
||||
app = create_app(cfg)
|
||||
with TestClient(app) as client:
|
||||
_install_fake_client(client.app.state.proxy)
|
||||
for endpoint in (MESSAGES, OMLX_MESSAGES):
|
||||
resp = client.post(
|
||||
endpoint,
|
||||
json={
|
||||
"model": "omlx/qwen3-8b",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
headers={"Authorization": "Bearer inbound-cloud-oauth"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert fake_tracker.notify_active.call_count == 0
|
||||
|
||||
cloud_resp = client.post(
|
||||
MESSAGES,
|
||||
json={
|
||||
"model": "claude-haiku-4-5-20251001",
|
||||
"max_tokens": 16,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
)
|
||||
assert cloud_resp.status_code == 200
|
||||
assert fake_tracker.notify_active.call_count == 1
|
||||
assert fake_tracker.notify_active.call_args.args[0] == "Bearer configured-cloud-oauth"
|
||||
Reference in New Issue
Block a user