Files
headroom/tests/test_grok_shared_proxy_call_sites.py
491afe98b1 fix(proxy): reject a private x-headroom-base-url on chat and responses with 400 (#4857)
## Description

When the upstream guard rejected an `x-headroom-base-url` override (an
address on a private, loopback, link-local or otherwise non-public
network, or a host outside `HEADROOM_ALLOWED_BASE_URLS` in allowlist
mode), `_resolve_openai_upstream_base` logged a warning and returned
`None`. `/v1/chat/completions` and `/v1/responses` then sent the request
— the prompt and the client's `Authorization`, meant for the gateway the
client named — to the configured `OPENAI_API_URL` instead. Every other
route that reads the header (`headroom/providers/proxy_routes.py`:
Anthropic messages, Gemini, models, embeddings, audio, passthrough)
refuses with `400 Rejected unsafe upstream base URL`.

The OpenAI chat and responses handlers now refuse the same way.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update

## Changes Made

- `headroom/proxy/handlers/openai.py`: `_resolve_openai_upstream_base`
raises `HTTPException(400, "Rejected unsafe upstream base URL")` instead
of returning `None` when the guard rejects the override (malformed /
non-http(s) values are unchanged: still ignored).
- `docs/content/docs/security-model.mdx`: one behaviour for all routes
(the page previously documented the two different outcomes).
-
`tests/test_upstream_guard.py::test_openai_routes_refuse_a_private_override_instead_of_using_the_default[chat|responses]`:
real app, configured OpenAI target is a local recording server; asserts
400 and that the configured target is never contacted.

**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-guard-off-loop` adds `_resolve_openai_upstream_base_async`
next to this function; whichever merges second must make the async
resolver raise the same 400 (that PR's rejection branch is the one place
to change). `fix/upstream-idna-pin-once` touches the call sites only.
Suggested order: idna-pin-once, off-loop, then this PR rebased.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New test on origin/main (c21b0813)
E   AssertionError: {"secret":"internal-only"}
E   assert 200 == 400                (chat: answered by the configured target)
E   assert 200 == 400                (responses: answered by the configured target)
======================= 2 failed, 61 deselected in 5.64s =======================

# This branch
======================= 2 passed, 61 deselected in 5.81s =======================

# Area suites: test_upstream_guard, test_grok_shared_proxy_call_sites, test_proxy/test_openai_upstream_header,
# test_proxy_openai_cache_key_integration, test_upstream_credential_scoping, test_header_isolation,
# test_proxy_hardening, test_openai_codex_routing, test_proxy_loopback_gating
================= 429 passed, 10 warnings in 87.64s (0:01:27) ==================

$ ruff check / ruff format --check (0.17.3) on changed files
All checks passed!
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, `python -m headroom.cli proxy
--no-optimize` from this branch and from origin/main,
`OPENAI_TARGET_API_URL` pointing at a local server that records every
request it receives.
- Exact command / steps:
`curl http://127.0.0.1:$PORT/v1/chat/completions -H 'authorization:
Bearer client-gateway-key' -H 'x-headroom-base-url:
http://192.168.1.50:8000/v1' -d '{"model":"gpt-4o","messages":[...]}'`,
and the same for `/v1/responses`.
- Observed result:
  ```text
  == origin/main
  POST /v1/chat/completions -> HTTP 200 {"...":"from-configured-target"}
  POST /v1/responses        -> HTTP 200 {"...":"from-configured-target"}
  configured target received:
    /v1/chat/completions auth= Bearer client-gateway-key
    /v1/responses        auth= Bearer client-gateway-key
  == this branch
POST /v1/chat/completions -> HTTP 400 {"detail":"Rejected unsafe
upstream base URL"}
POST /v1/responses -> HTTP 400 {"detail":"Rejected unsafe upstream base
URL"}
  configured target received: (nothing)
  ```
- Not tested: Codex WebSocket responses path (it does not read this
header).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: **yes.** A client that sends
`x-headroom-base-url` pointing at a private/LAN/loopback address (or, in
allowlist mode, an unlisted host) to `/v1/chat/completions` or
`/v1/responses` now gets a 400 instead of having its request answered by
the configured OpenAI target. Such a request never reached the host the
client named, so the only working setups affected are ones that relied
on the silent fallback. A DNS lookup that fails or times out for a
public gateway is also a rejection, so it now surfaces as a 400 instead
of quietly sending the request to the default target.
- Kill switch / disable path: operators who route to an internal gateway
add it to `HEADROOM_ALLOWED_BASE_URLS` (existing setting); clients can
drop the header to use the configured target.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

Follow-up, not in this PR: the OpenCode plugin
(`plugins/opencode/src/transport.ts`, `isRoutableUpstream`) only keeps
loopback upstreams out of the proxy, so an OpenCode provider on a LAN
address is still routed through Headroom, which now answers 400 for it
(previously the request went to the default OpenAI target). Sending
private-network upstreams direct needs a source change plus regenerating
the committed bundles (`headroom/providers/opencode/_dist/index.js`,
`hook-shim/handler.js`), and hostnames that resolve to LAN addresses
cannot be recognised without DNS, so it is left for a separate PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4857"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-15 01:52:42 +05:30

456 lines
16 KiB
Python

"""Call-site regressions for Grok routing and OpenAI-header isolation.
The tests assert the outbound URL and headers together. A routing-only unit test
would miss operator credentials leaking after a Grok request selects xAI, while a
header-only test could pass if a handler simply stopped merging extras everywhere.
"""
from __future__ import annotations
from collections.abc import Callable
from types import SimpleNamespace
from typing import Any, NamedTuple, cast
import anyio
import pytest
from headroom.cache.prefix_tracker import SessionTrackerStore
from tests.test_openai_codex_routing import (
_build_request,
_DummyOpenAIHandler,
_DummyTokenizer,
_jwt,
)
@pytest.fixture(autouse=True)
def _allow_client_chosen_test_upstreams(monkeypatch: pytest.MonkeyPatch) -> None:
"""Allowlist the client-chosen upstreams these tests name.
``x-headroom-base-url`` is validated by the SSRF guard before it is honored,
and in allowlist mode only these hosts pass. Naming them keeps the header
policy under test here rather than DNS: ``litellm.internal`` is deliberately
unresolvable, and the ``api.x.ai`` cases would otherwise depend on a live
lookup. The trailing-dot form is a distinct host to the guard, which matches
on the parsed hostname without normalizing it.
This is the SSRF allowlist, not the designated-upstream list that governs
which hosts may receive operator secrets, so the header assertions below
still exercise the real policy.
"""
monkeypatch.setenv("HEADROOM_ALLOWED_BASE_URLS", "litellm.internal,api.x.ai,api.x.ai.")
class _MinimalConfig(SimpleNamespace):
"""Treat unset optional chat features as disabled."""
def __getattr__(self, name: str) -> None:
return None
class _ChatHandler(_DummyOpenAIHandler):
"""Add the state read by the full chat-completions path."""
anthropic_backend: Any
ccr_response_handler: Any
def __init__(self) -> None:
super().__init__()
self.config = _MinimalConfig(**vars(self.config))
self.cache = None
self.session_tracker_store = cast(SimpleNamespace, SessionTrackerStore())
_GROK_HEADERS = {
"Authorization": "Bearer xai-redacted",
"x-xai-token-auth": "xai-grok-cli",
"User-Agent": "grok-pager/0.2.117 grok-shell/0.2.117 (macos; aarch64)",
}
_OPENAI_CLIENT_HEADERS = {
"Authorization": "Bearer sk-test",
"User-Agent": "codex-tui/0.146.0",
}
_OPERATOR_EXTRAS = {
"X-Operator-Sentinel": "must-not-leak",
"Authorization": "Bearer operator-openai-key",
}
_SENTINEL_ONLY_EXTRAS = {"X-Operator-Sentinel": "must-not-leak"}
_SENTINEL_VALUE = "must-not-leak"
_OPERATOR_KEY = "Bearer operator-openai-key"
_CLIENT_XAI_KEY = "Bearer xai-redacted"
@pytest.fixture(autouse=True)
def _stub_tokenizer(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _DummyTokenizer())
class _Outbound(NamedTuple):
url: str
headers: dict[str, str]
def _lower_unique(headers: dict[str, str]) -> dict[str, str]:
"""Normalize names and fail if duplicate case variants would go on wire."""
lowered = {key.lower(): value for key, value in headers.items()}
assert len(lowered) == len(headers), f"duplicate header names on wire: {sorted(headers)}"
return lowered
def _outbound(captured: tuple[str, str, dict, dict] | None) -> _Outbound:
assert captured is not None
_method, url, headers, _body = captured
return _Outbound(url=url, headers=_lower_unique(headers))
def _run_chat(
request_headers: dict[str, str],
*,
model: str = "grok-4",
extras: dict[str, str] = _OPERATOR_EXTRAS,
openai_api_url: str | None = None,
) -> _Outbound:
handler = _ChatHandler()
handler.config.openai_extra_headers = dict(extras)
if openai_api_url is not None:
handler.OPENAI_API_URL = openai_api_url
request = _build_request(
{"model": model, "messages": [{"role": "user", "content": "hello"}]},
request_headers,
path="/v1/chat/completions",
)
anyio.run(handler.handle_openai_chat, request)
return _outbound(handler.captured_request)
def _run_responses(
request_headers: dict[str, str],
*,
model: str = "grok-4",
extras: dict[str, str] = _OPERATOR_EXTRAS,
openai_api_url: str | None = None,
) -> _Outbound:
handler = _DummyOpenAIHandler()
handler.config.openai_extra_headers = dict(extras)
if openai_api_url is not None:
handler.OPENAI_API_URL = openai_api_url
request = _build_request({"model": model, "input": "hello"}, request_headers)
anyio.run(handler.handle_openai_responses, request)
return _outbound(handler.captured_request)
_Runner = Callable[..., _Outbound]
@pytest.mark.parametrize(
("run", "expected_url"),
[
(_run_chat, "https://api.x.ai/v1/chat/completions"),
(_run_responses, "https://api.x.ai/v1/responses"),
],
ids=("chat", "responses"),
)
def test_grok_call_sites_withhold_operator_extras_from_xai(run: _Runner, expected_url: str) -> None:
outbound = run(_GROK_HEADERS)
assert outbound.url == expected_url
assert "x-operator-sentinel" not in outbound.headers
assert outbound.headers["authorization"] == _CLIENT_XAI_KEY
@pytest.mark.parametrize(
("run", "expected_url"),
[
(_run_chat, "https://api.openai.com/v1/chat/completions"),
(_run_responses, "https://api.openai.com/v1/responses"),
],
ids=("chat", "responses"),
)
def test_ordinary_openai_call_sites_still_merge_operator_extras(
run: _Runner, expected_url: str
) -> None:
outbound = run(_OPENAI_CLIENT_HEADERS, model="gpt-4o-mini")
assert outbound.url == expected_url
assert outbound.headers["x-operator-sentinel"] == _SENTINEL_VALUE
assert outbound.headers["authorization"] == _OPERATOR_KEY
@pytest.mark.parametrize(
("run", "expected_url"),
[
(_run_chat, "https://litellm.internal/v1/chat/completions"),
(_run_responses, "https://litellm.internal/v1/responses"),
],
ids=("chat", "responses"),
)
def test_configured_gateway_outranks_grok_and_keeps_operator_extras(
run: _Runner, expected_url: str
) -> None:
outbound = run(_GROK_HEADERS, openai_api_url="https://litellm.internal")
assert outbound.url == expected_url
assert outbound.headers["x-operator-sentinel"] == _SENTINEL_VALUE
assert outbound.headers["authorization"] == _OPERATOR_KEY
@pytest.mark.parametrize(
("run", "base_url", "expected_url", "withhold"),
[
(
_run_chat,
"https://litellm.internal",
"https://litellm.internal/v1/chat/completions",
True,
),
(
_run_chat,
"https://api.x.ai/v1",
"https://api.x.ai/v1/v1/chat/completions",
True,
),
(
_run_responses,
"https://api.x.ai:8443",
"https://api.x.ai:8443/v1/responses",
True,
),
(
_run_responses,
"https://api.x.ai.",
"https://api.x.ai./v1/responses",
True,
),
],
ids=("untrusted-non-xai", "xai-subpath", "xai-port", "xai-trailing-dot"),
)
def test_custom_base_url_applies_hostname_scoped_header_policy(
run: _Runner,
base_url: str,
expected_url: str,
withhold: bool,
) -> None:
outbound = run(
{"Authorization": _CLIENT_XAI_KEY, "x-headroom-base-url": base_url},
)
assert outbound.url == expected_url
if withhold:
assert "x-operator-sentinel" not in outbound.headers
assert outbound.headers["authorization"] == _CLIENT_XAI_KEY
else:
assert outbound.headers["x-operator-sentinel"] == _SENTINEL_VALUE
assert outbound.headers["authorization"] == _OPERATOR_KEY
def test_configured_xai_target_withholds_ambient_openai_extras() -> None:
outbound = _run_responses(_GROK_HEADERS, openai_api_url="https://api.x.ai")
assert outbound.url == "https://api.x.ai/v1/responses"
assert "x-operator-sentinel" not in outbound.headers
assert outbound.headers["authorization"] == _CLIENT_XAI_KEY
def test_chatgpt_auth_wins_routing_while_xai_candidate_withholds_extras() -> None:
token = _jwt({"https://api.openai.com/auth": {"chatgpt_account_id": "acct-from-jwt"}})
outbound = _run_responses(
{**_GROK_HEADERS, "Authorization": f"Bearer {token}"},
model="gpt-5.4",
extras=_SENTINEL_ONLY_EXTRAS,
)
assert outbound.url == "https://chatgpt.com/backend-api/codex/responses"
assert outbound.headers["chatgpt-account-id"] == "acct-from-jwt"
assert "x-operator-sentinel" not in outbound.headers
class _RecordingBackend:
name = "litellm"
def __init__(self) -> None:
self.captured_headers: dict[str, str] | None = None
async def send_openai_message(self, body: dict, headers: dict) -> SimpleNamespace:
self.captured_headers = dict(headers)
return SimpleNamespace(
status_code=200,
error=None,
body={
"id": "chatcmpl-backend-1",
"choices": [{"index": 0, "message": {"role": "assistant", "content": "ok"}}],
"usage": {"prompt_tokens": 2, "completion_tokens": 1},
},
)
def test_configured_chat_backend_retains_its_operator_extras() -> None:
handler = _ChatHandler()
handler.config.openai_extra_headers = dict(_OPERATOR_EXTRAS)
backend = _RecordingBackend()
handler.anthropic_backend = backend
handler.ccr_response_handler = None
request = _build_request(
{"model": "grok-4", "messages": [{"role": "user", "content": "hello"}]},
_GROK_HEADERS,
path="/v1/chat/completions",
)
response = anyio.run(handler.handle_openai_chat, request)
assert response.status_code == 200
assert backend.captured_headers is not None
headers = _lower_unique(backend.captured_headers)
assert headers["x-operator-sentinel"] == _SENTINEL_VALUE
assert headers["authorization"] == _OPERATOR_KEY
assert handler.captured_request is None
@pytest.mark.parametrize("handle", ["handle_openai_chat", "handle_openai_responses"])
def test_the_base_url_override_is_resolved_once_per_request(
handle: str, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The destination and the extra-header trust check must come from one lookup.
The first lookup of the client's host answers with a private address and
every later one with a public address. When the handler resolved the header
more than once, the trust check saw "no override" while the destination was
the client's host, so operator extras were sent there.
"""
import socket
from headroom.proxy import upstream_guard
monkeypatch.delenv("HEADROOM_ALLOWED_BASE_URLS", raising=False)
calls: list[str] = []
def changing_answer(host: str, *args: object, **kwargs: object) -> list[object]:
calls.append(host)
address = "10.0.0.5" if len(calls) == 1 else "93.184.216.34"
return [(None, None, None, None, (address, 443))]
monkeypatch.setattr(socket, "getaddrinfo", changing_answer)
upstream_guard.clear_validated_addresses()
handler = _ChatHandler()
handler.config.openai_extra_headers = dict(_SENTINEL_ONLY_EXTRAS)
body: dict[str, Any] = {"model": "gpt-4o-mini"}
if handle == "handle_openai_chat":
body["messages"] = [{"role": "user", "content": "hello"}]
else:
body["input"] = "hello"
request = _build_request(
body,
{**_OPENAI_CLIENT_HEADERS, "x-headroom-base-url": "https://changing.example/v1"},
path="/v1/chat/completions" if handle == "handle_openai_chat" else "/v1/responses",
)
from fastapi import HTTPException
# The single lookup answers with a private address, so the override is refused.
with pytest.raises(HTTPException) as refused:
anyio.run(getattr(handler, handle), request)
assert refused.value.status_code == 400
assert calls == ["changing.example"], f"the override was resolved {len(calls)} times"
assert handler.captured_request is None, "the request was sent somewhere"
def _run_image_chat(monkeypatch: pytest.MonkeyPatch, *, expire_pins: bool) -> dict[str, list[Any]]:
"""Send an image chat request to a client-chosen gateway; record guard activity.
The stub image compression optionally ages every pin in the request's scope
past its deadline, standing in for a compression that outlasts the pin.
"""
import asyncio
import socket
from headroom.proxy import upstream_guard
monkeypatch.delenv("HEADROOM_ALLOWED_BASE_URLS", raising=False)
seen: dict[str, list[Any]] = {"lookups": [], "guard_on_loop": [], "pin_at_send": []}
def _public(host: str, *a: object, **k: object) -> list[object]:
seen["lookups"].append(host)
return [(None, None, None, None, ("93.184.216.34", 443))]
real_guard = upstream_guard.is_safe_upstream_url
def _recording_guard(url: str, **kwargs: Any) -> bool:
try:
asyncio.get_running_loop()
except RuntimeError:
seen["guard_on_loop"].append(False)
else:
seen["guard_on_loop"].append(True)
return real_guard(url, **kwargs)
monkeypatch.setattr(socket, "getaddrinfo", _public)
monkeypatch.setattr(upstream_guard, "is_safe_upstream_url", _recording_guard)
upstream_guard.clear_validated_addresses()
class _Compressor:
def has_images(self, messages: object) -> bool:
return True
def close(self) -> None:
return None
async def _compression(messages: Any, **kwargs: Any) -> tuple[Any, None]:
if expire_pins:
scope = upstream_guard._PIN_SCOPE.get() or {}
for host, pin in list(scope.items()):
scope[host] = upstream_guard.UpstreamPin(pin.addresses, 0.0)
return messages, None
monkeypatch.setattr("headroom.proxy.helpers._get_image_compressor", lambda: _Compressor())
monkeypatch.setattr(
"headroom.proxy.handlers.openai.run_image_compression_isolated", _compression
)
class _Handler(_ChatHandler):
async def _retry_request(self, method: str, url: str, headers: dict, body: dict, **kwargs): # noqa: ANN201
seen["pin_at_send"].append(upstream_guard.guarded_pin("gateway.example"))
return await super()._retry_request(method, url, headers, body, **kwargs)
handler = _Handler()
handler.config.image_optimize = True
request = _build_request(
{
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "what is this"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}},
],
}
],
},
{**_OPENAI_CLIENT_HEADERS, "x-headroom-base-url": "https://gateway.example/v1"},
path="/v1/chat/completions",
)
anyio.run(handler.handle_openai_chat, request)
return seen
def test_a_long_image_compression_does_not_leave_an_expired_pin(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A pin that lapsed during image compression is renewed off the event loop."""
seen = _run_image_chat(monkeypatch, expire_pins=True)
assert seen["pin_at_send"], "the request was not sent"
pin = seen["pin_at_send"][0]
assert pin is not None and not pin.is_expired(), "the connection would find an expired pin"
assert seen["guard_on_loop"][-1] is False, "the renewal ran on the event-loop thread"
def test_a_still_valid_pin_is_not_checked_again_after_image_compression(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A second lookup is only needed once the pin lapsed; it could fail a valid request."""
seen = _run_image_chat(monkeypatch, expire_pins=False)
assert seen["pin_at_send"], "the request was not sent"
assert seen["lookups"] == ["gateway.example"], seen["lookups"]