mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
`handle_openai_chat` and `handle_openai_responses` validated
`x-headroom-base-url` with the synchronous `is_safe_upstream_url`, which
waits for the DNS lookup (up to `HEADROOM_UPSTREAM_RESOLVE_TIMEOUT_S`, 3
s by default) on the event-loop thread. Chat did this three times per
request and responses twice, so one slow-resolving host held up every
other in-flight request for several seconds. The other routes in
`headroom/providers/proxy_routes.py` already use
`is_safe_upstream_url_async`.
Both handlers now resolve the header once, through a new
`_resolve_openai_upstream_base_async` that uses the async guard (lookup
on a worker thread, pin scope opened on the request's task), and reuse
that result for the destination and the extra-header decision.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
## Changes Made
- `headroom/proxy/handlers/openai.py`: header parsing split into
`_openai_base_url_candidate`; sync `_resolve_openai_upstream_base` (kept
for `_resolve_openai_upstream`) and new
`_resolve_openai_upstream_base_async` share it;
`_openai_upstream_from(request, custom)` picks the upstream from an
already-resolved override; chat and responses call the async resolver
once.
- `tests/test_proxy/test_openai_upstream_guard_off_loop.py`: asserts the
guard runs once and not on the event-loop thread, for chat and
responses.
- `tests/test_proxy_openai_cache_key_integration.py`: its stub now
patches the async resolver the handler calls.
**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-idna-pin-once` adds the identical `_openai_upstream_from`
and resolve-once hunks (this PR is a superset of that part);
`fix/openai-base-url-reject` changes the rejection branch of
`_resolve_openai_upstream_base`, which after this PR must be mirrored in
`_resolve_openai_upstream_base_async`. Suggested order: idna-pin-once,
then this PR, then reject (rebasing reject means raising the 400 from
the shared rejection path in both resolvers).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# New test on origin/main (c21b0813)
E AssertionError: the guard ran on the event-loop thread
E assert [True, True, True] == [False] (chat)
E assert [True, True] == [False] (responses)
============================== 2 failed in 0.72s ===============================
# This branch: new test + cache-key integration
============================== 7 passed in 7.30s ===============================
# Area suites: test_upstream_guard, test_grok_shared_proxy_call_sites, test_proxy/test_openai_upstream_header,
# test_proxy_openai_cache_key_integration, test_upstream_credential_scoping, test_header_isolation,
# test_proxy_hardening, test_openai_codex_routing, test_proxy/test_openai_upstream_guard_off_loop
======================= 282 passed, 5 warnings in 56.32s =======================
$ ruff check / ruff format --check (0.17.3) on changed files
All checks passed!
3 files already formatted
$ mypy headroom/proxy/handlers/openai.py --ignore-missing-imports
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, full app from `create_app` driven
through `httpx.ASGITransport` on one event loop; `socket.getaddrinfo`
replaced so `slow.example` takes 2 s to answer (everything else real).
- Exact command / steps: start a `/v1/chat/completions` request with
`x-headroom-base-url: https://slow.example/v1`, then 0.2 s later send
`GET /livez` and time it.
- Observed result:
```text
== origin/main
/livez sent 0.2s after a chat request naming a 2s-to-resolve host: HTTP
200 after 6.03s (0.20s if the loop is free)
== this branch
/livez sent 0.2s after a chat request naming a 2s-to-resolve host: HTTP
200 after 0.20s (0.20s if the loop is free)
```
- Not tested: under real slow DNS.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no change in which upstream is
chosen; the lookup moves to a worker thread and happens once per
request.
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Fix adapted from a proposal by HetCreep.
Follow-up, not in this PR: `select_passthrough_base_url`
(`headroom/providers/proxy_targets.py`, Azure `api-key` branch) and
`headroom/providers/registry.py` still call the sync guard; the
passthrough route already checks the same header with the async guard
first, so the sync call there is a second lookup on the loop.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4856"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Reported by @JoaoMarcos44. Thank you for your contribution! 🩵
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
175 lines
6.2 KiB
Python
175 lines
6.2 KiB
Python
"""Integration RBP for the OpenAI handler's SemanticCache key threading.
|
|
|
|
Companion to ``test_proxy_semantic_cache_key_integration.py`` (Anthropic). Drives
|
|
the real ``/v1/chat/completions`` handler with the cache enabled and a stubbed
|
|
upstream, proving the OpenAI handler actually threads each newly-added
|
|
response-shaping field into the cache get/set calls: two requests with identical
|
|
``messages`` but a different ``response_format`` / ``tool_choice`` / ``seed`` must
|
|
NOT collide, while a repeat of the first IS served from cache.
|
|
|
|
A cache-key unit test cannot catch this — it exercises ``_compute_key`` directly.
|
|
The failure mode this guards is the handler's ``cache_key_fields`` snapshot
|
|
omitting a ``body.get(...)`` for a field: ``_compute_key`` would distinguish the
|
|
field fine, but the handler never passes it. Before the OpenAI snapshot widening
|
|
a request differing only in ``response_format`` collided and was served the
|
|
first request's response.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
|
|
def _make_cached_proxy_client() -> TestClient:
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=True,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
)
|
|
return TestClient(create_app(config))
|
|
|
|
|
|
def _body(**extra: object) -> dict:
|
|
body: dict = {
|
|
"model": "gpt-4o-mini",
|
|
"max_tokens": 64,
|
|
"messages": [{"role": "user", "content": "Say hi."}],
|
|
"stream": False,
|
|
}
|
|
body.update(extra)
|
|
return body
|
|
|
|
|
|
def _content(response: httpx.Response) -> str:
|
|
return response.json()["choices"][0]["message"]["content"]
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"field,a,b",
|
|
[
|
|
("response_format", {"type": "json_object"}, {"type": "text"}),
|
|
("tool_choice", "auto", "none"),
|
|
("seed", 1, 2),
|
|
("reasoning_effort", "low", "high"),
|
|
],
|
|
)
|
|
def test_openai_differing_field_not_served_from_cache(field, a, b) -> None:
|
|
"""A and B share messages and differ only in ``field``; B must not be served
|
|
A's cached response, and a repeat of A must hit the cache."""
|
|
calls = {"n": 0}
|
|
|
|
with _make_cached_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
calls["n"] += 1
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "chatcmpl_1",
|
|
"object": "chat.completion",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": f"resp-{calls['n']}"},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {"prompt_tokens": 10, "completion_tokens": 3, "total_tokens": 13},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
headers = {"authorization": "Bearer test-key"}
|
|
|
|
# A: field=a -> upstream call 1, cached under A's key.
|
|
ra = client.post("/v1/chat/completions", headers=headers, json=_body(**{field: a}))
|
|
assert ra.status_code == 200
|
|
assert _content(ra) == "resp-1"
|
|
assert calls["n"] == 1
|
|
|
|
# B: field=b, SAME messages -> must reach the upstream again, not be
|
|
# served A's cached response. With the field missing from the key, B
|
|
# collided with A and calls stayed 1 (the bug this guards).
|
|
rb = client.post("/v1/chat/completions", headers=headers, json=_body(**{field: b}))
|
|
assert rb.status_code == 200
|
|
assert _content(rb) == "resp-2"
|
|
assert calls["n"] == 2
|
|
|
|
# A again -> served from cache, upstream NOT called.
|
|
ra2 = client.post("/v1/chat/completions", headers=headers, json=_body(**{field: a}))
|
|
assert ra2.status_code == 200
|
|
assert _content(ra2) == "resp-1"
|
|
assert calls["n"] == 2
|
|
|
|
|
|
def test_openai_different_custom_upstream_not_served_from_cache(monkeypatch) -> None:
|
|
"""Two OpenAI-compatible gateways must not share a response entry (#3346)."""
|
|
calls: list[str] = []
|
|
|
|
async def _custom_base(headers): # noqa: ANN001
|
|
return headers.get("x-headroom-base-url")
|
|
|
|
monkeypatch.setattr(
|
|
"headroom.proxy.handlers.openai._resolve_openai_upstream_base_async", _custom_base
|
|
)
|
|
|
|
with _make_cached_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
calls.append(str(url))
|
|
content = "gateway-A" if "gateway-a" in str(url) else "gateway-B"
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": f"chatcmpl_{len(calls)}",
|
|
"object": "chat.completion",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": content},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {"prompt_tokens": 10, "completion_tokens": 3, "total_tokens": 13},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
headers = {"authorization": "Bearer test-key"}
|
|
body = _body()
|
|
|
|
first = client.post(
|
|
"/v1/chat/completions",
|
|
headers={**headers, "x-headroom-base-url": "https://gateway-a.example"},
|
|
json=body,
|
|
)
|
|
second = client.post(
|
|
"/v1/chat/completions",
|
|
headers={**headers, "x-headroom-base-url": "https://gateway-b.example"},
|
|
json=body,
|
|
)
|
|
|
|
assert first.status_code == 200
|
|
assert second.status_code == 200
|
|
assert _content(first) == "gateway-A"
|
|
assert _content(second) == "gateway-B"
|
|
assert calls == [
|
|
"https://gateway-a.example/v1/chat/completions",
|
|
"https://gateway-b.example/v1/chat/completions",
|
|
]
|