mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description `upstream_trust.is_trusted_upstream` decides whether operator-configured extra headers (`OPENAI_TARGET_API_HEADERS` / `ANTHROPIC_TARGET_API_HEADERS`, stored as secrets) may be merged for a client-chosen `x-headroom-base-url`. It compared hostnames only, so `http://<designated host>` or `https://<designated host>:<other port>` also received those headers, over cleartext in the http case. It also used one trusted set for every provider, so a chat/responses request with `x-headroom-base-url: https://api.anthropic.com` received the OpenAI extras, and `/v1/messages` pointed at `https://api.openai.com` or the configured OpenAI gateway received the Anthropic extras. Trust is now decided per provider, on the origin (scheme, host, port). The OpenAI extras go only to the OpenAI target / `api.openai.com`; the Anthropic extras only to the Anthropic target, the oMLX target or `api.anthropic.com`; `HEADROOM_UPSTREAM_ALLOWED_HOSTS` entries count for both. Origin rules: - a bare designation (`api.openai.com`, or a scheme-less configured URL) means `https` on port 443; - configured provider API URLs (`OPENAI_TARGET_API_URL`, `ANTHROPIC_TARGET_API_URL`, oMLX, Gemini/Vertex/Cloud Code/Bedrock) designate their own origin, e.g. `http://omlx.internal:8888`; - `HEADROOM_UPSTREAM_ALLOWED_HOSTS` entries may be a bare host, `host:port` (https) or a full URL (`http://gateway.internal:4000`). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update ## Changes Made - `headroom/proxy/upstream_trust.py`: `_origin()`, `trusted_upstream_origins()`, `is_trusted_upstream` matches on origin; malformed ports and bracketed hosts are refused instead of raising; the withheld-headers warning is keyed and logged by origin (`host=http://gateway.internal:4000`) so a listed bare host reached over http still warns; the now-unused host-only helpers `trusted_upstream_hosts` / `_env_allowed_hosts` are removed (only one test used them). - `merge_extra_headers` / `is_trusted_upstream` take `provider`; the OpenAI chat and responses handlers pass `"openai"`, the Anthropic handler (`_build_outbound_anthropic_headers`) passes `"anthropic"`. Without `provider` the union of all targets is used, as before. - Docs: `configuration.mdx` (designation rules, example, upgrade note), `security-model.mdx`, `pipeline-extensions.mdx`. - Tests: new `tests/test_upstream_trust_origin.py` (8 cases, through `merge_extra_headers` and `is_trusted_upstream`); `tests/test_proxy/test_omlx_routing.py` asserts through `is_trusted_upstream` instead of the removed helper. **Sibling PRs:** `fix/upstream-idna-pin-once`, `fix/upstream-guard-off-loop` and `fix/openai-base-url-reject` change how `handlers/openai.py` resolves the URL passed to `merge_extra_headers`; this PR only changes `upstream_trust.py`, so there is no textual overlap and it can merge in any order. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New tests on origin/main (c21b0813) FAILED tests/test_upstream_trust_origin.py::test_secret_extra_headers_are_not_merged_for_another_scheme_or_port FAILED tests/test_upstream_trust_origin.py::test_a_configured_api_url_designates_its_own_origin FAILED tests/test_upstream_trust_origin.py::test_a_scheme_less_configured_api_url_means_https FAILED tests/test_upstream_trust_origin.py::test_allowed_hosts_entries_designate_one_origin[llm.corp-...] FAILED tests/test_upstream_trust_origin.py::test_allowed_hosts_entries_designate_one_origin[llm.corp:8443-...] FAILED tests/test_upstream_trust_origin.py::test_allowed_hosts_entries_designate_one_origin[http://llm.corp:8080-...] FAILED tests/test_upstream_trust_origin.py::test_malformed_destinations_are_not_trusted[https://[api.openai.com]/v1] FAILED tests/test_upstream_trust_origin.py::test_malformed_destinations_are_not_trusted[https://api.openai.com:notaport/v1] ============================== 8 failed in 1.13s =============================== # This branch ============================== 8 passed in 1.67s =============================== # Every test file that mentions extra_headers / UPSTREAM_ALLOWED_HOSTS / upstream_trust (24 files) =========== 571 passed, 12 skipped, 8 warnings in 166.82s (0:02:46) ============ # after review follow-ups (origin-keyed warning, helper removal) tests/test_upstream_trust_origin.py tests/test_proxy/test_omlx_routing.py tests/test_upstream_credential_scoping.py tests/test_proxy/test_factory_passthrough.py tests/test_grok_shared_proxy_call_sites.py tests/test_header_isolation.py ======================= 145 passed, 1 warning in 37.22s ======================== $ ruff check / ruff format --check (0.17.3) on changed files All checks passed! $ mypy headroom/proxy/upstream_trust.py --ignore-missing-imports Success: no issues found in 1 source file ``` ### Per-provider scoping (follow-up commit37ff98d0) ```text # 3 new tests in tests/test_upstream_trust_origin.py, before the commit FAILED tests/test_upstream_trust_origin.py::test_each_provider_extras_stay_with_that_provider FAILED tests/test_upstream_trust_origin.py::test_an_allowed_host_is_designated_for_both_providers FAILED tests/test_upstream_trust_origin.py::test_the_handlers_scope_extras_to_their_own_provider # after ============================== 11 passed in 0.71s ============================== # every test file mentioning extra_headers / UPSTREAM_ALLOWED_HOSTS / upstream_trust ============ 574 passed, 12 skipped, 8 warnings in 86.12s (0:01:26) ============ ``` Full app (`create_app`, upstream replaced by an httpx MockTransport that records headers; `openai_api_url=https://gateway.example.invalid`, OpenAI extra `x-operator-oai`, Anthropic extra `x-operator-ant`): ```text before: chat-xprov base=https://api.anthropic.com -> 200 oai=OAI-MARKER ant=- resp-xprov base=https://api.anthropic.com -> 200 oai=OAI-MARKER ant=- msgs-xprov base=https://api.openai.com -> 200 oai=- ant=ANT-MARKER msgs-xprov base=https://gateway.example.invalid -> 200 oai=- ant=ANT-MARKER after: chat-xprov base=https://api.anthropic.com -> 200 oai=- ant=- resp-xprov base=https://api.anthropic.com -> 200 oai=- ant=- msgs-xprov base=https://api.openai.com -> 200 oai=- ant=- msgs-xprov base=https://gateway.example.invalid -> 200 oai=- ant=- ``` Same-provider cases are unchanged (chat to the OpenAI gateway and messages to `https://api.anthropic.com` still carry their own extras). ## Real Behavior Proof - Environment: macOS, Python 3.12, `python -m headroom.cli proxy --no-optimize` from origin/main and from this branch with `HEADROOM_UPSTREAM_ALLOWED_HOSTS=localtest.me`, `HEADROOM_ALLOWED_BASE_URLS=localtest.me`, `OPENAI_TARGET_API_HEADERS='{"X-Operator-Key": "operator-secret"}'`; `localtest.me` resolves to 127.0.0.1 where a local server records request headers. - Exact command / steps: `curl http://127.0.0.1:$PORT/v1/chat/completions -H 'x-headroom-base-url: http://localtest.me:$REC/v1' ...` (plain http, non-443 port of the designated bare host). - Observed result: ```text == origin/main HTTP 200 recorder: /v1/v1/chat/completions auth= Bearer client-key x-operator-key= operator-secret == this branch HTTP 200 recorder: /v1/v1/chat/completions auth= Bearer client-key x-operator-key= None log: upstream_extra_headers_withheld host=http://localtest.me:21154 ``` The request is still proxied; only the operator's header is withheld. - Not tested: Copilot / Bedrock live upstreams (they pass `upstream_url=None` or do not merge extras, so they are unaffected). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: **yes, for one setup.** An operator who listed a bare host in `HEADROOM_UPSTREAM_ALLOWED_HOSTS` and has clients reach it over plain `http://` or a non-443 port (via `x-headroom-base-url`) stops sending the configured extra headers there after upgrading. The request is still proxied, and the proxy logs `upstream_extra_headers_withheld host=<scheme>://<host>:<port>` once per origin. Also, a deployment that relied on one provider's extras reaching the other provider's configured target (for example the Anthropic extras sent to `OPENAI_TARGET_API_URL` through `/v1/messages`) stops sending them; list that origin in `HEADROOM_UPSTREAM_ALLOWED_HOSTS` if that is intended. Remedy for the scheme/port case: list the exact origin, e.g. `HEADROOM_UPSTREAM_ALLOWED_HOSTS=http://gateway.internal:4000` (or `gateway.internal:8443` for https on another port). Configured provider targets keep working because they designate their own origin. - Kill switch / disable path: the env entry above. - Unsafe override required: no - Qualification impact: none - Rollback path: revert ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Fix adapted from a proposal by HetCreep. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4858" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
125 lines
5.1 KiB
Python
125 lines
5.1 KiB
Python
"""Operator secret headers go to a designated origin, not just a designated host.
|
|
|
|
``x-headroom-base-url`` is chosen by the client. When trust compared only the
|
|
hostname, ``http://`` or another port of a designated host also received the
|
|
operator's extra headers.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.helpers import merge_extra_headers
|
|
from headroom.proxy.upstream_trust import ALLOWED_HOSTS_ENV, is_trusted_upstream
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _no_env_allowlist(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.delenv(ALLOWED_HOSTS_ENV, raising=False)
|
|
|
|
|
|
def test_secret_extra_headers_are_not_merged_for_another_scheme_or_port() -> None:
|
|
extra = {"api-key": "corp-gateway-secret"}
|
|
|
|
def sent(url: str) -> dict[str, str]:
|
|
return merge_extra_headers({}, extra, upstream_url=url)
|
|
|
|
assert sent("https://api.openai.com/v1/chat/completions") == extra
|
|
assert sent("https://API.OpenAI.com:443/v1/chat/completions") == extra
|
|
assert sent("http://api.openai.com/v1/chat/completions") == {}
|
|
assert sent("https://api.openai.com:8443/v1/chat/completions") == {}
|
|
|
|
|
|
def test_a_configured_api_url_designates_its_own_origin() -> None:
|
|
cfg = SimpleNamespace(openai_api_url="http://llm.internal:8080/v1")
|
|
|
|
assert is_trusted_upstream("http://llm.internal:8080/v1/chat/completions", cfg)
|
|
assert not is_trusted_upstream("https://llm.internal:8080/v1/chat/completions", cfg)
|
|
assert not is_trusted_upstream("http://llm.internal:9090/v1/chat/completions", cfg)
|
|
assert not is_trusted_upstream("http://llm.internal/v1/chat/completions", cfg)
|
|
|
|
|
|
def test_a_scheme_less_configured_api_url_means_https() -> None:
|
|
cfg = SimpleNamespace(anthropic_api_url="gateway.example.com/v1")
|
|
|
|
assert is_trusted_upstream("https://gateway.example.com/v1/messages", cfg)
|
|
assert not is_trusted_upstream("http://gateway.example.com/v1/messages", cfg)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("entry", "trusted", "untrusted"),
|
|
[
|
|
("llm.corp", "https://llm.corp/v1", ["http://llm.corp/v1", "https://llm.corp:8443/v1"]),
|
|
("llm.corp:8443", "https://llm.corp:8443/v1", ["https://llm.corp/v1"]),
|
|
("http://llm.corp:8080", "http://llm.corp:8080/v1", ["https://llm.corp:8080/v1"]),
|
|
],
|
|
)
|
|
def test_allowed_hosts_entries_designate_one_origin(
|
|
monkeypatch: pytest.MonkeyPatch, entry: str, trusted: str, untrusted: list[str]
|
|
) -> None:
|
|
monkeypatch.setenv(ALLOWED_HOSTS_ENV, f"other.example, {entry}")
|
|
|
|
assert is_trusted_upstream(trusted)
|
|
for url in untrusted:
|
|
assert not is_trusted_upstream(url), url
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"url",
|
|
[
|
|
"https://[api.openai.com]/v1",
|
|
"https://api.openai.com:notaport/v1",
|
|
],
|
|
)
|
|
def test_malformed_destinations_are_not_trusted(url: str) -> None:
|
|
assert not is_trusted_upstream(url)
|
|
|
|
|
|
def test_each_provider_extras_stay_with_that_provider() -> None:
|
|
"""OpenAI extras never go to Anthropic's API, and Anthropic extras never go to OpenAI's."""
|
|
openai_extra = {"x-operator-oai": "oai-secret"}
|
|
anthropic_extra = {"x-operator-ant": "ant-secret"}
|
|
cfg = SimpleNamespace(openai_api_url="https://gateway.example/v1")
|
|
|
|
def sent(extra: dict[str, str], url: str, provider: str) -> dict[str, str]:
|
|
return merge_extra_headers({}, extra, upstream_url=url, config=cfg, provider=provider)
|
|
|
|
assert sent(openai_extra, "https://api.openai.com/v1", "openai") == openai_extra
|
|
assert sent(openai_extra, "https://gateway.example/v1", "openai") == openai_extra
|
|
assert sent(openai_extra, "https://api.anthropic.com/v1", "openai") == {}
|
|
assert sent(anthropic_extra, "https://api.anthropic.com/v1", "anthropic") == anthropic_extra
|
|
assert sent(anthropic_extra, "https://api.openai.com/v1", "anthropic") == {}
|
|
assert sent(anthropic_extra, "https://gateway.example/v1", "anthropic") == {}
|
|
|
|
|
|
def test_an_allowed_host_is_designated_for_both_providers(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setenv(ALLOWED_HOSTS_ENV, "shared-gateway.example")
|
|
for provider in ("openai", "anthropic"):
|
|
assert is_trusted_upstream("https://shared-gateway.example/v1", None, provider)
|
|
|
|
|
|
def test_the_handlers_scope_extras_to_their_own_provider(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Wiring check: chat sends only OpenAI extras, messages only Anthropic extras."""
|
|
from headroom.proxy.handlers.anthropic import _build_outbound_anthropic_headers
|
|
from tests.test_grok_shared_proxy_call_sites import _OPENAI_CLIENT_HEADERS, _run_chat
|
|
|
|
monkeypatch.setenv("HEADROOM_ALLOWED_BASE_URLS", "api.anthropic.com")
|
|
outbound = _run_chat(
|
|
{**_OPENAI_CLIENT_HEADERS, "x-headroom-base-url": "https://api.anthropic.com"},
|
|
model="gpt-4o-mini",
|
|
extras={"X-Operator-Sentinel": "must-not-leak"},
|
|
)
|
|
assert outbound.url.startswith("https://api.anthropic.com")
|
|
assert "x-operator-sentinel" not in outbound.headers
|
|
|
|
headers = _build_outbound_anthropic_headers(
|
|
{},
|
|
{"x-operator-ant": "ant-secret"},
|
|
routed_to_omlx=False,
|
|
upstream_url="https://api.openai.com",
|
|
config=SimpleNamespace(),
|
|
)
|
|
assert "x-operator-ant" not in headers
|