mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description Claude Code (and other Anthropic SDK clients) uses `HEAD /api/hello` or `GET /api/hello` as an unauthenticated connectivity probe to verify upstream reachability. Because this probe carries no Anthropic authentication headers, the unrouted passthrough in `select_passthrough_base_url` fell through to the default OpenAI target, opening an unexpected outbound connection to `api.openai.com` (which returns 404) on an Anthropic-only setup. This mirrors the Copilot completion routing fix (#3076): `/api/hello` is an Anthropic-specific endpoint and is now explicitly routed to the Anthropic target even when no authentication headers are present. Closes #3336 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/providers/route_specs.py`: add `GET` and `HEAD` `/api/hello` to `ANTHROPIC_PASSTHROUGH_ROUTES`. - `headroom/providers/proxy_targets.py`: add `is_anthropic_hello_path()` and route `/api/hello` to `api_target(proxy, "anthropic")` in `select_passthrough_base_url()`. - `headroom/providers/registry.py`: accept optional `path` in `select_passthrough_base_url()` to route `/api/hello` to Anthropic target. - `tests/test_provider_route_specs.py`, `tests/test_provider_proxy_targets.py`, `tests/test_provider_proxy_routes.py`, `tests/test_provider_registry.py`: update existing route and target assertions. - `tests/test_anthropic_hello_probe_routing.py`: new targeted regression tests for path normalization, base URL resolution, and unauthenticated end-to-end `HEAD` and `GET` routing. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_provider_route_specs.py tests/test_provider_proxy_targets.py tests/test_provider_proxy_routes.py tests/test_provider_registry.py tests/test_anthropic_hello_probe_routing.py -q .................................................................. [100%] 66 passed, 1 warning in 56.04s $ ruff check headroom/providers/route_specs.py headroom/providers/proxy_targets.py headroom/providers/registry.py tests/test_anthropic_hello_probe_routing.py All checks passed! $ ruff format --check headroom/providers/route_specs.py headroom/providers/proxy_targets.py headroom/providers/registry.py tests/test_anthropic_hello_probe_routing.py 4 files already formatted $ mypy headroom/providers/route_specs.py headroom/providers/proxy_targets.py headroom/providers/registry.py tests/test_anthropic_hello_probe_routing.py --follow-imports=silent --ignore-missing-imports Success: no issues found in 4 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.14.6, local test server with Anthropic and OpenAI target stubs. - Exact command / steps: Sent unauthenticated `HEAD /api/hello` (carrying `User-Agent: Bun/1.1.20`) and `GET /api/hello` requests to the proxy using `fastapi.testclient.TestClient`. - Observed result: Both requests routed directly to `https://api.anthropic.test` with HTTP 200; `fake_passthrough` recorded `base_url == "https://api.anthropic.test"` and `provider == "anthropic"`. Other unauthenticated endpoints continue falling through to `https://api.openai.test`. - Not tested: Live Anthropic egress against production Claude API. ### Maintainer live qualification (2026-10-08) - Environment: actual Windows 11, Python 3.12.13, uvicorn proxy over loopback TCP. Provider API-key environment variables were empty; the client sent no authentication headers. - Exact steps: start the unchanged proxy from this head with `anthropic_api_url=https://api.anthropic.com` and the OpenAI target deliberately set to unreachable `http://127.0.0.1:1`; send keyless `HEAD /api/hello` and `GET /api/hello` through the proxy with httpx. - Observed: public Anthropic returned **HEAD 200 with zero body bytes** and **GET 200 with `{"message": "hello"}`**. No handler or provider call was mocked. A second real TCP probe against a loopback Anthropic test server captured exactly HEAD and GET `/api/hello`, both 200, while the OpenAI target remained unreachable. **66 focused provider-routing tests passed**. - Scope: connectivity canaries only; no inference request or credentialed API action was made. Hosted CI was held by GitHub's fork workflow-approval gate, which the maintainer approved after reviewing this exact head; CI is now running and PR approval remains pending until it qualifies. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A. - Stable/default behavior changed: `/api/hello` probes now forward to Anthropic rather than OpenAI. Other unauthenticated routes remain unchanged. - Kill switch / disable path: N/A. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4059" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end -->
507 lines
17 KiB
Python
507 lines
17 KiB
Python
from __future__ import annotations
|
|
|
|
import logging
|
|
|
|
import pytest
|
|
|
|
from headroom.providers.registry import (
|
|
ProviderApiOverrides,
|
|
build_proxy_provider_runtime,
|
|
create_proxy_backend,
|
|
format_backend_status,
|
|
resolve_api_overrides,
|
|
resolve_api_targets,
|
|
resolve_extra_headers,
|
|
)
|
|
from headroom.proxy.models import ProxyConfig
|
|
|
|
|
|
def test_resolve_api_overrides_prefers_explicit_values_over_environment(monkeypatch) -> None:
|
|
monkeypatch.setenv("ANTHROPIC_TARGET_API_URL", "https://env.anthropic.example/v1")
|
|
monkeypatch.setenv("OPENAI_TARGET_API_URL", "https://env.openai.example/v1")
|
|
monkeypatch.setenv("VERTEX_TARGET_API_URL", "https://env-vertex-aiplatform.example/v1")
|
|
|
|
overrides = resolve_api_overrides(
|
|
anthropic_api_url="https://cli.anthropic.example/v1",
|
|
openai_api_url=None,
|
|
gemini_api_url=None,
|
|
cloudcode_api_url=None,
|
|
vertex_api_url="https://cli-vertex-aiplatform.example/v1",
|
|
)
|
|
|
|
assert overrides == ProviderApiOverrides(
|
|
anthropic="https://cli.anthropic.example/v1",
|
|
openai="https://env.openai.example/v1",
|
|
gemini=None,
|
|
cloudcode=None,
|
|
vertex="https://cli-vertex-aiplatform.example/v1",
|
|
)
|
|
|
|
|
|
def test_resolve_api_targets_normalizes_trailing_v1() -> None:
|
|
targets = resolve_api_targets(
|
|
ProviderApiOverrides(
|
|
anthropic="https://anthropic.example/v1/",
|
|
openai="https://openai.example/v1",
|
|
gemini="https://gemini.example/v1",
|
|
cloudcode="https://cloudcode.example/v1/",
|
|
vertex="https://vertex.example/v1/",
|
|
)
|
|
)
|
|
|
|
assert targets.anthropic == "https://anthropic.example"
|
|
assert targets.openai == "https://openai.example"
|
|
assert targets.gemini == "https://gemini.example"
|
|
assert targets.cloudcode == "https://cloudcode.example"
|
|
assert targets.vertex == "https://vertex.example"
|
|
|
|
|
|
def test_copilot_openai_target_routes_anthropic_to_copilot() -> None:
|
|
"""When the OpenAI target is a Copilot host and no Anthropic override is set,
|
|
the Anthropic target must default to the same Copilot host.
|
|
|
|
Copilot serves Claude models via its Anthropic surface (``/v1/messages``) on
|
|
the same host. Without this, Claude requests fell back to api.anthropic.com
|
|
and 401'd with the Copilot bearer ("Invalid bearer token", #3247).
|
|
"""
|
|
targets = resolve_api_targets(
|
|
ProviderApiOverrides(
|
|
anthropic=None,
|
|
openai="https://api.githubcopilot.com",
|
|
gemini=None,
|
|
cloudcode=None,
|
|
vertex=None,
|
|
)
|
|
)
|
|
assert targets.openai == "https://api.githubcopilot.com"
|
|
assert targets.anthropic == "https://api.githubcopilot.com"
|
|
|
|
|
|
def test_explicit_anthropic_override_wins_over_copilot_default() -> None:
|
|
"""An explicit Anthropic target is never overridden by the Copilot default."""
|
|
targets = resolve_api_targets(
|
|
ProviderApiOverrides(
|
|
anthropic="https://api.anthropic.com",
|
|
openai="https://api.githubcopilot.com",
|
|
gemini=None,
|
|
cloudcode=None,
|
|
vertex=None,
|
|
)
|
|
)
|
|
assert targets.anthropic == "https://api.anthropic.com"
|
|
|
|
|
|
def test_non_copilot_openai_target_leaves_anthropic_default() -> None:
|
|
"""A non-Copilot OpenAI target must not touch the Anthropic default."""
|
|
targets = resolve_api_targets(
|
|
ProviderApiOverrides(
|
|
anthropic=None,
|
|
openai="https://api.openai.com",
|
|
gemini=None,
|
|
cloudcode=None,
|
|
vertex=None,
|
|
)
|
|
)
|
|
assert targets.anthropic == "https://api.anthropic.com"
|
|
|
|
|
|
def test_proxy_config_exposes_provider_api_overrides() -> None:
|
|
config = ProxyConfig(
|
|
anthropic_api_url="https://anthropic.example",
|
|
openai_api_url="https://openai.example",
|
|
gemini_api_url=None,
|
|
cloudcode_api_url="https://cloudcode.example",
|
|
vertex_api_url="https://vertex.example",
|
|
)
|
|
|
|
assert config.provider_api_overrides == ProviderApiOverrides(
|
|
anthropic="https://anthropic.example",
|
|
openai="https://openai.example",
|
|
gemini=None,
|
|
cloudcode="https://cloudcode.example",
|
|
vertex="https://vertex.example",
|
|
)
|
|
|
|
|
|
def test_format_backend_status_for_anyllm() -> None:
|
|
assert (
|
|
format_backend_status(
|
|
backend="anyllm",
|
|
anyllm_provider="groq",
|
|
bedrock_region="us-central1",
|
|
)
|
|
== "Groq via any-llm"
|
|
)
|
|
|
|
|
|
def test_format_backend_status_for_anthropic_direct() -> None:
|
|
assert (
|
|
format_backend_status(
|
|
backend="anthropic",
|
|
anyllm_provider="ignored",
|
|
bedrock_region=None,
|
|
)
|
|
== "ANTHROPIC (direct API)"
|
|
)
|
|
|
|
|
|
def test_proxy_provider_runtime_routes_model_metadata_and_passthrough() -> None:
|
|
runtime = build_proxy_provider_runtime(ProxyConfig())
|
|
|
|
assert runtime.model_metadata_provider({"x-api-key": "test"}) == "anthropic"
|
|
assert runtime.model_metadata_provider({}) == "openai"
|
|
assert (
|
|
runtime.select_passthrough_base_url({"x-api-key": "test"}) == runtime.api_targets.anthropic
|
|
)
|
|
assert (
|
|
runtime.select_passthrough_base_url({"x-goog-api-key": "test"})
|
|
== runtime.api_targets.gemini
|
|
)
|
|
assert runtime.select_passthrough_base_url({"api-key": "azure", "x-headroom-base-url": ""}) == (
|
|
runtime.api_targets.openai
|
|
)
|
|
assert runtime.select_passthrough_base_url({}, "/api/hello") == runtime.api_targets.anthropic
|
|
assert runtime.select_passthrough_base_url({}, "/other/path") == runtime.api_targets.openai
|
|
|
|
|
|
def test_create_proxy_backend_handles_missing_litellm_backend(caplog) -> None:
|
|
logger = logging.getLogger("test")
|
|
|
|
with caplog.at_level(logging.WARNING):
|
|
missing = create_proxy_backend(
|
|
backend="bedrock",
|
|
anyllm_provider="ignored",
|
|
bedrock_region="us-east-1",
|
|
logger=logger,
|
|
litellm_backend_cls=lambda provider, region, profile_name=None: (_ for _ in ()).throw(
|
|
ImportError("missing")
|
|
),
|
|
)
|
|
|
|
assert missing is None
|
|
assert "LiteLLM backend not available" in caplog.text
|
|
|
|
|
|
def test_create_proxy_backend_logs_structured_failure_details(caplog) -> None:
|
|
logger = logging.getLogger("test")
|
|
|
|
with caplog.at_level(logging.ERROR):
|
|
missing = create_proxy_backend(
|
|
backend="bedrock",
|
|
anyllm_provider="ignored",
|
|
bedrock_region="us-east-1",
|
|
logger=logger,
|
|
litellm_backend_cls=lambda provider, region, profile_name=None: (_ for _ in ()).throw(
|
|
RuntimeError("boom")
|
|
),
|
|
)
|
|
|
|
assert missing is None
|
|
assert "backend initialization failed: backend=litellm-bedrock provider=bedrock error=boom" in (
|
|
caplog.text
|
|
)
|
|
|
|
|
|
def test_proxy_provider_runtime_loaders_cache_backend_types(monkeypatch) -> None:
|
|
import headroom.providers.registry as registry
|
|
|
|
anyllm_loads = 0
|
|
litellm_loads = 0
|
|
|
|
class FakeAnyLLMBackend:
|
|
pass
|
|
|
|
class FakeLiteLLMBackend:
|
|
pass
|
|
|
|
def fake_import(name, globals=None, locals=None, fromlist=(), level=0):
|
|
nonlocal anyllm_loads, litellm_loads
|
|
if name == "headroom.backends.anyllm":
|
|
anyllm_loads += 1
|
|
return type("Module", (), {"AnyLLMBackend": FakeAnyLLMBackend})()
|
|
if name == "headroom.backends.litellm":
|
|
litellm_loads += 1
|
|
return type("Module", (), {"LiteLLMBackend": FakeLiteLLMBackend})()
|
|
raise AssertionError(name)
|
|
|
|
monkeypatch.setattr(registry, "AnyLLMBackendType", None)
|
|
monkeypatch.setattr(registry, "LiteLLMBackendType", None)
|
|
monkeypatch.setattr("builtins.__import__", fake_import)
|
|
|
|
assert registry._load_anyllm_backend() is FakeAnyLLMBackend
|
|
assert registry._load_anyllm_backend() is FakeAnyLLMBackend
|
|
assert registry._load_litellm_backend() is FakeLiteLLMBackend
|
|
assert registry._load_litellm_backend() is FakeLiteLLMBackend
|
|
assert anyllm_loads == 1
|
|
assert litellm_loads == 1
|
|
|
|
|
|
def test_proxy_provider_runtime_transport_helpers_handle_missing_usage() -> None:
|
|
import headroom.providers.registry as registry
|
|
|
|
class Storage:
|
|
def __init__(self) -> None:
|
|
self.saved = []
|
|
|
|
def save(self, metrics) -> None:
|
|
self.saved.append(metrics)
|
|
|
|
client = type(
|
|
"Client",
|
|
(),
|
|
{
|
|
"_storage": Storage(),
|
|
"_original": type(
|
|
"Original",
|
|
(),
|
|
{
|
|
"chat": type(
|
|
"Chat",
|
|
(),
|
|
{
|
|
"completions": type(
|
|
"Completions",
|
|
(),
|
|
{
|
|
"create": staticmethod(
|
|
lambda **kwargs: type("Resp", (), {"usage": None})()
|
|
)
|
|
},
|
|
)()
|
|
},
|
|
)(),
|
|
"messages": type(
|
|
"Messages",
|
|
(),
|
|
{
|
|
"create": staticmethod(
|
|
lambda **kwargs: type("Resp", (), {"usage": None})()
|
|
)
|
|
},
|
|
)(),
|
|
},
|
|
)(),
|
|
},
|
|
)()
|
|
openai_metrics = type("Metrics", (), {"tokens_output": 0, "cached_tokens": 0})()
|
|
anthropic_metrics = type("Metrics", (), {"tokens_output": 0, "cached_tokens": 0})()
|
|
|
|
registry._call_openai_transport(
|
|
client,
|
|
model="gpt-4o",
|
|
messages=[],
|
|
stream=False,
|
|
metrics=openai_metrics,
|
|
)
|
|
registry._call_anthropic_transport(
|
|
client,
|
|
model="claude",
|
|
messages=[],
|
|
stream=False,
|
|
metrics=anthropic_metrics,
|
|
)
|
|
|
|
assert openai_metrics.tokens_output == 0
|
|
assert openai_metrics.cached_tokens == 0
|
|
assert anthropic_metrics.tokens_output == 0
|
|
assert anthropic_metrics.cached_tokens == 0
|
|
assert len(client._storage.saved) == 2
|
|
|
|
|
|
def test_proxy_provider_runtime_transport_helpers_handle_usage_without_optional_cache_fields() -> (
|
|
None
|
|
):
|
|
import headroom.providers.registry as registry
|
|
|
|
class Storage:
|
|
def __init__(self) -> None:
|
|
self.saved = []
|
|
|
|
def save(self, metrics) -> None:
|
|
self.saved.append(metrics)
|
|
|
|
client = type(
|
|
"Client",
|
|
(),
|
|
{
|
|
"_storage": Storage(),
|
|
"_original": type(
|
|
"Original",
|
|
(),
|
|
{
|
|
"chat": type(
|
|
"Chat",
|
|
(),
|
|
{
|
|
"completions": type(
|
|
"Completions",
|
|
(),
|
|
{
|
|
"create": staticmethod(
|
|
lambda **kwargs: type(
|
|
"Resp",
|
|
(),
|
|
{
|
|
"usage": type(
|
|
"Usage",
|
|
(),
|
|
{"completion_tokens": 7},
|
|
)()
|
|
},
|
|
)()
|
|
)
|
|
},
|
|
)()
|
|
},
|
|
)(),
|
|
"messages": type(
|
|
"Messages",
|
|
(),
|
|
{
|
|
"create": staticmethod(
|
|
lambda **kwargs: type(
|
|
"Resp",
|
|
(),
|
|
{
|
|
"usage": type(
|
|
"Usage",
|
|
(),
|
|
{"output_tokens": 5},
|
|
)()
|
|
},
|
|
)()
|
|
)
|
|
},
|
|
)(),
|
|
},
|
|
)(),
|
|
},
|
|
)()
|
|
openai_metrics = type("Metrics", (), {"tokens_output": 0, "cached_tokens": 0})()
|
|
anthropic_metrics = type("Metrics", (), {"tokens_output": 0, "cached_tokens": 0})()
|
|
|
|
registry._call_openai_transport(
|
|
client,
|
|
model="gpt-4o",
|
|
messages=[],
|
|
stream=False,
|
|
metrics=openai_metrics,
|
|
)
|
|
registry._call_anthropic_transport(
|
|
client,
|
|
model="claude",
|
|
messages=[],
|
|
stream=False,
|
|
metrics=anthropic_metrics,
|
|
)
|
|
|
|
assert openai_metrics.tokens_output == 7
|
|
assert openai_metrics.cached_tokens == 0
|
|
assert anthropic_metrics.tokens_output == 5
|
|
assert anthropic_metrics.cached_tokens == 0
|
|
assert len(client._storage.saved) == 2
|
|
|
|
|
|
def test_proxy_provider_runtime_openai_transport_handles_prompt_details_without_cached_tokens() -> (
|
|
None
|
|
):
|
|
import headroom.providers.registry as registry
|
|
|
|
class Storage:
|
|
def __init__(self) -> None:
|
|
self.saved = []
|
|
|
|
def save(self, metrics) -> None:
|
|
self.saved.append(metrics)
|
|
|
|
client = type(
|
|
"Client",
|
|
(),
|
|
{
|
|
"_storage": Storage(),
|
|
"_original": type(
|
|
"Original",
|
|
(),
|
|
{
|
|
"chat": type(
|
|
"Chat",
|
|
(),
|
|
{
|
|
"completions": type(
|
|
"Completions",
|
|
(),
|
|
{
|
|
"create": staticmethod(
|
|
lambda **kwargs: type(
|
|
"Resp",
|
|
(),
|
|
{
|
|
"usage": type(
|
|
"Usage",
|
|
(),
|
|
{
|
|
"completion_tokens": 9,
|
|
"prompt_tokens_details": type(
|
|
"Details",
|
|
(),
|
|
{},
|
|
)(),
|
|
},
|
|
)()
|
|
},
|
|
)()
|
|
)
|
|
},
|
|
)()
|
|
},
|
|
)()
|
|
},
|
|
)(),
|
|
},
|
|
)()
|
|
metrics = type("Metrics", (), {"tokens_output": 0, "cached_tokens": 0})()
|
|
|
|
registry._call_openai_transport(
|
|
client,
|
|
model="gpt-4o",
|
|
messages=[],
|
|
stream=False,
|
|
metrics=metrics,
|
|
)
|
|
|
|
assert metrics.tokens_output == 9
|
|
assert metrics.cached_tokens == 0
|
|
assert len(client._storage.saved) == 1
|
|
|
|
|
|
def test_resolve_extra_headers_cli_wins_over_env(monkeypatch) -> None:
|
|
monkeypatch.setenv("ANTHROPIC_TARGET_API_HEADERS", '{"Env-Header": "env-value"}')
|
|
result = resolve_extra_headers('{"Cli-Header": "cli-value"}', "ANTHROPIC_TARGET_API_HEADERS")
|
|
assert result == {"Cli-Header": "cli-value"}
|
|
|
|
|
|
def test_resolve_extra_headers_falls_back_to_env(monkeypatch) -> None:
|
|
monkeypatch.setenv("OPENAI_TARGET_API_HEADERS", '{"Env-Header": "env-value"}')
|
|
result = resolve_extra_headers(None, "OPENAI_TARGET_API_HEADERS")
|
|
assert result == {"Env-Header": "env-value"}
|
|
|
|
|
|
def test_resolve_extra_headers_unset_returns_none(monkeypatch) -> None:
|
|
monkeypatch.delenv("ANTHROPIC_TARGET_API_HEADERS", raising=False)
|
|
assert resolve_extra_headers(None, "ANTHROPIC_TARGET_API_HEADERS") is None
|
|
|
|
|
|
def test_resolve_extra_headers_invalid_json_raises(monkeypatch) -> None:
|
|
with pytest.raises(ValueError):
|
|
resolve_extra_headers("not json", "ANTHROPIC_TARGET_API_HEADERS")
|
|
|
|
|
|
def test_resolve_extra_headers_non_object_raises(monkeypatch) -> None:
|
|
with pytest.raises(ValueError):
|
|
resolve_extra_headers('["a", "b"]', "ANTHROPIC_TARGET_API_HEADERS")
|
|
|
|
|
|
def test_resolve_extra_headers_non_string_value_raises(monkeypatch) -> None:
|
|
with pytest.raises(ValueError):
|
|
resolve_extra_headers('{"Key": 123}', "ANTHROPIC_TARGET_API_HEADERS")
|