Files
headroom/tests/test_upstream_trust_origin.py
6d8121cfd4 fix(proxy): send operator extra headers only to a designated origin (#4858)
## Description

`upstream_trust.is_trusted_upstream` decides whether operator-configured
extra headers (`OPENAI_TARGET_API_HEADERS` /
`ANTHROPIC_TARGET_API_HEADERS`, stored as secrets) may be merged for a
client-chosen `x-headroom-base-url`. It compared hostnames only, so
`http://<designated host>` or `https://<designated host>:<other port>`
also received those headers, over cleartext in the http case.

It also used one trusted set for every provider, so a chat/responses
request with `x-headroom-base-url: https://api.anthropic.com` received
the OpenAI extras, and `/v1/messages` pointed at
`https://api.openai.com` or the configured OpenAI gateway received the
Anthropic extras.

Trust is now decided per provider, on the origin (scheme, host, port).
The OpenAI extras go only to the OpenAI target / `api.openai.com`; the
Anthropic extras only to the Anthropic target, the oMLX target or
`api.anthropic.com`; `HEADROOM_UPSTREAM_ALLOWED_HOSTS` entries count for
both.

Origin rules:
- a bare designation (`api.openai.com`, or a scheme-less configured URL)
means `https` on port 443;
- configured provider API URLs (`OPENAI_TARGET_API_URL`,
`ANTHROPIC_TARGET_API_URL`, oMLX, Gemini/Vertex/Cloud Code/Bedrock)
designate their own origin, e.g. `http://omlx.internal:8888`;
- `HEADROOM_UPSTREAM_ALLOWED_HOSTS` entries may be a bare host,
`host:port` (https) or a full URL (`http://gateway.internal:4000`).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update

## Changes Made

- `headroom/proxy/upstream_trust.py`: `_origin()`,
`trusted_upstream_origins()`, `is_trusted_upstream` matches on origin;
malformed ports and bracketed hosts are refused instead of raising; the
withheld-headers warning is keyed and logged by origin
(`host=http://gateway.internal:4000`) so a listed bare host reached over
http still warns; the now-unused host-only helpers
`trusted_upstream_hosts` / `_env_allowed_hosts` are removed (only one
test used them).
- `merge_extra_headers` / `is_trusted_upstream` take `provider`; the
OpenAI chat and responses handlers pass `"openai"`, the Anthropic
handler (`_build_outbound_anthropic_headers`) passes `"anthropic"`.
Without `provider` the union of all targets is used, as before.
- Docs: `configuration.mdx` (designation rules, example, upgrade note),
`security-model.mdx`, `pipeline-extensions.mdx`.
- Tests: new `tests/test_upstream_trust_origin.py` (8 cases, through
`merge_extra_headers` and `is_trusted_upstream`);
`tests/test_proxy/test_omlx_routing.py` asserts through
`is_trusted_upstream` instead of the removed helper.

**Sibling PRs:** `fix/upstream-idna-pin-once`,
`fix/upstream-guard-off-loop` and `fix/openai-base-url-reject` change
how `handlers/openai.py` resolves the URL passed to
`merge_extra_headers`; this PR only changes `upstream_trust.py`, so
there is no textual overlap and it can merge in any order.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New tests on origin/main (c21b0813)
FAILED tests/test_upstream_trust_origin.py::test_secret_extra_headers_are_not_merged_for_another_scheme_or_port
FAILED tests/test_upstream_trust_origin.py::test_a_configured_api_url_designates_its_own_origin
FAILED tests/test_upstream_trust_origin.py::test_a_scheme_less_configured_api_url_means_https
FAILED tests/test_upstream_trust_origin.py::test_allowed_hosts_entries_designate_one_origin[llm.corp-...]
FAILED tests/test_upstream_trust_origin.py::test_allowed_hosts_entries_designate_one_origin[llm.corp:8443-...]
FAILED tests/test_upstream_trust_origin.py::test_allowed_hosts_entries_designate_one_origin[http://llm.corp:8080-...]
FAILED tests/test_upstream_trust_origin.py::test_malformed_destinations_are_not_trusted[https://[api.openai.com]/v1]
FAILED tests/test_upstream_trust_origin.py::test_malformed_destinations_are_not_trusted[https://api.openai.com:notaport/v1]
============================== 8 failed in 1.13s ===============================

# This branch
============================== 8 passed in 1.67s ===============================

# Every test file that mentions extra_headers / UPSTREAM_ALLOWED_HOSTS / upstream_trust (24 files)
=========== 571 passed, 12 skipped, 8 warnings in 166.82s (0:02:46) ============
# after review follow-ups (origin-keyed warning, helper removal)
tests/test_upstream_trust_origin.py tests/test_proxy/test_omlx_routing.py tests/test_upstream_credential_scoping.py
tests/test_proxy/test_factory_passthrough.py tests/test_grok_shared_proxy_call_sites.py tests/test_header_isolation.py
======================= 145 passed, 1 warning in 37.22s ========================

$ ruff check / ruff format --check (0.17.3) on changed files
All checks passed!
$ mypy headroom/proxy/upstream_trust.py --ignore-missing-imports
Success: no issues found in 1 source file
```

### Per-provider scoping (follow-up commit 37ff98d0)

```text
# 3 new tests in tests/test_upstream_trust_origin.py, before the commit
FAILED tests/test_upstream_trust_origin.py::test_each_provider_extras_stay_with_that_provider
FAILED tests/test_upstream_trust_origin.py::test_an_allowed_host_is_designated_for_both_providers
FAILED tests/test_upstream_trust_origin.py::test_the_handlers_scope_extras_to_their_own_provider
# after
============================== 11 passed in 0.71s ==============================
# every test file mentioning extra_headers / UPSTREAM_ALLOWED_HOSTS / upstream_trust
============ 574 passed, 12 skipped, 8 warnings in 86.12s (0:01:26) ============
```

Full app (`create_app`, upstream replaced by an httpx MockTransport that
records headers; `openai_api_url=https://gateway.example.invalid`,
OpenAI extra `x-operator-oai`, Anthropic extra `x-operator-ant`):

```text
before:
chat-xprov base=https://api.anthropic.com        -> 200 oai=OAI-MARKER ant=-
resp-xprov base=https://api.anthropic.com        -> 200 oai=OAI-MARKER ant=-
msgs-xprov base=https://api.openai.com           -> 200 oai=-          ant=ANT-MARKER
msgs-xprov base=https://gateway.example.invalid  -> 200 oai=-          ant=ANT-MARKER
after:
chat-xprov base=https://api.anthropic.com        -> 200 oai=-          ant=-
resp-xprov base=https://api.anthropic.com        -> 200 oai=-          ant=-
msgs-xprov base=https://api.openai.com           -> 200 oai=-          ant=-
msgs-xprov base=https://gateway.example.invalid  -> 200 oai=-          ant=-
```
Same-provider cases are unchanged (chat to the OpenAI gateway and
messages to `https://api.anthropic.com` still carry their own extras).

## Real Behavior Proof

- Environment: macOS, Python 3.12, `python -m headroom.cli proxy
--no-optimize` from origin/main and from this branch with
`HEADROOM_UPSTREAM_ALLOWED_HOSTS=localtest.me`,
`HEADROOM_ALLOWED_BASE_URLS=localtest.me`,
`OPENAI_TARGET_API_HEADERS='{"X-Operator-Key": "operator-secret"}'`;
`localtest.me` resolves to 127.0.0.1 where a local server records
request headers.
- Exact command / steps: `curl
http://127.0.0.1:$PORT/v1/chat/completions -H 'x-headroom-base-url:
http://localtest.me:$REC/v1' ...` (plain http, non-443 port of the
designated bare host).
- Observed result:
  ```text
  == origin/main
  HTTP 200
recorder: /v1/v1/chat/completions auth= Bearer client-key
x-operator-key= operator-secret
  == this branch
  HTTP 200
recorder: /v1/v1/chat/completions auth= Bearer client-key
x-operator-key= None
  log: upstream_extra_headers_withheld host=http://localtest.me:21154
  ```
  The request is still proxied; only the operator's header is withheld.
- Not tested: Copilot / Bedrock live upstreams (they pass
`upstream_url=None` or do not merge extras, so they are unaffected).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: **yes, for one setup.** An operator
who listed a bare host in `HEADROOM_UPSTREAM_ALLOWED_HOSTS` and has
clients reach it over plain `http://` or a non-443 port (via
`x-headroom-base-url`) stops sending the configured extra headers there
after upgrading. The request is still proxied, and the proxy logs
`upstream_extra_headers_withheld host=<scheme>://<host>:<port>` once per
origin. Also, a deployment that relied on one provider's extras reaching
the other provider's configured target (for example the Anthropic extras
sent to `OPENAI_TARGET_API_URL` through `/v1/messages`) stops sending
them; list that origin in `HEADROOM_UPSTREAM_ALLOWED_HOSTS` if that is
intended. Remedy for the scheme/port case: list the exact origin, e.g.
`HEADROOM_UPSTREAM_ALLOWED_HOSTS=http://gateway.internal:4000` (or
`gateway.internal:8443` for https on another port). Configured provider
targets keep working because they designate their own origin.
- Kill switch / disable path: the env entry above.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4858"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-14 10:46:28 +05:30

125 lines
5.1 KiB
Python

"""Operator secret headers go to a designated origin, not just a designated host.
``x-headroom-base-url`` is chosen by the client. When trust compared only the
hostname, ``http://`` or another port of a designated host also received the
operator's extra headers.
"""
from __future__ import annotations
from types import SimpleNamespace
import pytest
from headroom.proxy.helpers import merge_extra_headers
from headroom.proxy.upstream_trust import ALLOWED_HOSTS_ENV, is_trusted_upstream
@pytest.fixture(autouse=True)
def _no_env_allowlist(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv(ALLOWED_HOSTS_ENV, raising=False)
def test_secret_extra_headers_are_not_merged_for_another_scheme_or_port() -> None:
extra = {"api-key": "corp-gateway-secret"}
def sent(url: str) -> dict[str, str]:
return merge_extra_headers({}, extra, upstream_url=url)
assert sent("https://api.openai.com/v1/chat/completions") == extra
assert sent("https://API.OpenAI.com:443/v1/chat/completions") == extra
assert sent("http://api.openai.com/v1/chat/completions") == {}
assert sent("https://api.openai.com:8443/v1/chat/completions") == {}
def test_a_configured_api_url_designates_its_own_origin() -> None:
cfg = SimpleNamespace(openai_api_url="http://llm.internal:8080/v1")
assert is_trusted_upstream("http://llm.internal:8080/v1/chat/completions", cfg)
assert not is_trusted_upstream("https://llm.internal:8080/v1/chat/completions", cfg)
assert not is_trusted_upstream("http://llm.internal:9090/v1/chat/completions", cfg)
assert not is_trusted_upstream("http://llm.internal/v1/chat/completions", cfg)
def test_a_scheme_less_configured_api_url_means_https() -> None:
cfg = SimpleNamespace(anthropic_api_url="gateway.example.com/v1")
assert is_trusted_upstream("https://gateway.example.com/v1/messages", cfg)
assert not is_trusted_upstream("http://gateway.example.com/v1/messages", cfg)
@pytest.mark.parametrize(
("entry", "trusted", "untrusted"),
[
("llm.corp", "https://llm.corp/v1", ["http://llm.corp/v1", "https://llm.corp:8443/v1"]),
("llm.corp:8443", "https://llm.corp:8443/v1", ["https://llm.corp/v1"]),
("http://llm.corp:8080", "http://llm.corp:8080/v1", ["https://llm.corp:8080/v1"]),
],
)
def test_allowed_hosts_entries_designate_one_origin(
monkeypatch: pytest.MonkeyPatch, entry: str, trusted: str, untrusted: list[str]
) -> None:
monkeypatch.setenv(ALLOWED_HOSTS_ENV, f"other.example, {entry}")
assert is_trusted_upstream(trusted)
for url in untrusted:
assert not is_trusted_upstream(url), url
@pytest.mark.parametrize(
"url",
[
"https://[api.openai.com]/v1",
"https://api.openai.com:notaport/v1",
],
)
def test_malformed_destinations_are_not_trusted(url: str) -> None:
assert not is_trusted_upstream(url)
def test_each_provider_extras_stay_with_that_provider() -> None:
"""OpenAI extras never go to Anthropic's API, and Anthropic extras never go to OpenAI's."""
openai_extra = {"x-operator-oai": "oai-secret"}
anthropic_extra = {"x-operator-ant": "ant-secret"}
cfg = SimpleNamespace(openai_api_url="https://gateway.example/v1")
def sent(extra: dict[str, str], url: str, provider: str) -> dict[str, str]:
return merge_extra_headers({}, extra, upstream_url=url, config=cfg, provider=provider)
assert sent(openai_extra, "https://api.openai.com/v1", "openai") == openai_extra
assert sent(openai_extra, "https://gateway.example/v1", "openai") == openai_extra
assert sent(openai_extra, "https://api.anthropic.com/v1", "openai") == {}
assert sent(anthropic_extra, "https://api.anthropic.com/v1", "anthropic") == anthropic_extra
assert sent(anthropic_extra, "https://api.openai.com/v1", "anthropic") == {}
assert sent(anthropic_extra, "https://gateway.example/v1", "anthropic") == {}
def test_an_allowed_host_is_designated_for_both_providers(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv(ALLOWED_HOSTS_ENV, "shared-gateway.example")
for provider in ("openai", "anthropic"):
assert is_trusted_upstream("https://shared-gateway.example/v1", None, provider)
def test_the_handlers_scope_extras_to_their_own_provider(monkeypatch: pytest.MonkeyPatch) -> None:
"""Wiring check: chat sends only OpenAI extras, messages only Anthropic extras."""
from headroom.proxy.handlers.anthropic import _build_outbound_anthropic_headers
from tests.test_grok_shared_proxy_call_sites import _OPENAI_CLIENT_HEADERS, _run_chat
monkeypatch.setenv("HEADROOM_ALLOWED_BASE_URLS", "api.anthropic.com")
outbound = _run_chat(
{**_OPENAI_CLIENT_HEADERS, "x-headroom-base-url": "https://api.anthropic.com"},
model="gpt-4o-mini",
extras={"X-Operator-Sentinel": "must-not-leak"},
)
assert outbound.url.startswith("https://api.anthropic.com")
assert "x-operator-sentinel" not in outbound.headers
headers = _build_outbound_anthropic_headers(
{},
{"x-operator-ant": "ant-secret"},
routed_to_omlx=False,
upstream_url="https://api.openai.com",
config=SimpleNamespace(),
)
assert "x-operator-ant" not in headers