Files
headroom/tests/test_public_errors.py
Tejas ChopraandClaude Fable 5.1 861e94d8c1 fix(proxy): keep exception text and the proxy token out of client output (#3851)
## Why

Two leaks of internal detail to clients and logs (findings 01-F4 and
10-F4).

- When an upstream connection failed, several response paths returned
`str(e)` from `httpx`, `ssl` or the OS in 500/502/503 bodies and SSE
`error` events. That text names the resolved upstream host, its address
and the failing library. A client that can steer the upstream
(`x-headroom-base-url`) could learn internal hostnames one error at a
time.
- `extract_tags` copied every `x-headroom-*` header into
`RequestOutcome.tags`, including `x-headroom-proxy-token`, which
gateways send on every turn. The token then appeared in the request log
(`--log-file`), dashboard facets and any export that carries tags.

## What changes

- New module `headroom/proxy/public_errors.py`. `classify_exception()`
maps transport and OS failures to fixed codes: `upstream_unreachable`,
`upstream_timeout`, `upstream_tls_error`, `upstream_protocol_error`;
anything else is `internal_error`. Every client- or model-visible path
fails closed: unrecognised exception text (`RuntimeError`, `ValueError`,
parser, database or library errors) never leaves the proxy. Provider
error text is forwarded only when the exception carries a provider HTTP
error response (`openai.APIStatusError`, which LiteLLM's provider errors
subclass, or `anthropic.APIStatusError`). Helpers build the existing
wire shapes with the same `error.type`, plus `code` and `request_id`.
The full exception stays in the server log under that request id.
- Every site that built a client-visible message from an exception now
uses it: `handlers/streaming.py` (direct, Bedrock and OpenAI-via-backend
SSE), `handlers/openai.py` (backend 500, `/v1/compress` 503, both
passthrough connect paths), `handlers/anthropic.py` (Bedrock backend
500), `handlers/bedrock.py` (connect 502), the memory tool handlers
(tool results returned to the model), and the `litellm` / `anyllm`
backends. The existing corporate-TLS hint is kept.
- `internal_header_policy.py` gains `is_credential_header()` and
`is_credential_tag_key()`. They match whole name components (`token`,
`secret`, `password`, `authorization`, `cookie`, `apikey`, `key`, and
similar), so `X-Headroom-Key` and `X-Headroom-Api-Key` are covered, so
numeric tags such as `tool-search-deferred-tokens` are not caught.
- `extract_tags` skips credential headers.
`savings_attribution.public_tags`, the last step before the request log,
also drops credential-shaped keys.

## Behaviour change

- Transport-layer and internal failures now return a fixed sentence plus
`code` and `request_id`. Clients that parsed the old free-text messages
should read `error.code`. Provider HTTP error responses (and their
status codes) are unchanged. In the LiteLLM / any-llm backends, an
exception that is not a provider response now returns `The proxy could
not complete the request.`; the HTTP status mapping is unchanged.
- Memory tool results for any failure return `{"status": "error",
"error": "<code>", "message": ...}` instead of the exception text.
- `x-headroom-*` headers whose name contains a credential component are
no longer recorded as tags. A benign label that happens to use such a
name should be renamed.

## Tests

- `tests/test_public_errors.py`: classification of transport exceptions
(including TLS wrapped in a connect error), non-transport exceptions
left alone, wire shapes carry `code` and `request_id`; the passthrough,
direct-streaming and `/v1/compress` error paths contain no upstream
detail; memory tool results hide transport and internal text;
`client_message` / `tool_result_error` / the LiteLLM backend return
`internal_error` for `RuntimeError`, `ValueError`, `KeyError` and
`JSONDecodeError` carrying a path, host and token (10 tests in this file
fail without the fail-closed change), while an
`openai.AuthenticationError` keeps its text; `extract_tags` and
`public_tags` never carry credentials. Six of these fail on `main`.
- Existing tests that asserted the old exception text in client output
now assert the new contract: the client gets the fixed `code` and
`request_id`, the exception text is absent, and the `headroom.proxy` log
has the full text under the same `request_id`
(`test_litellm_transport_errors.py`, `test_openai_backend_path.py`,
`test_proxy_compress_endpoint.py`,
`test_proxy_streaming_ratelimit_headers.py`).
- Manual run with every upstream connect failing: streaming
`/v1/messages`, streaming `/v1/chat/completions` and passthrough
`/v1/models` return `code: upstream_unreachable` with no address. A
request with `x-headroom-proxy-token` and `x-headroom-license-token`
leaves neither value in the `--log-file` output.

Not tested live: the Bedrock and LiteLLM / any-llm backend paths (unit
tests only) and OTEL/Langfuse export (tags are filtered before they
reach it).

## Open review points

- Resolved: unclassified exception text (JerrettDavis, 2026-09-29) now
fails closed in `client_message` and `tool_result_error`.
- Resolved: `X-Headroom-Key` is matched (`key` is a credential
component; the `api`+`key` pair rule is gone). No internal or test tag
key has a `key` component.
- `is_credential_header` special-cases `authorization`, `cookie`,
`api-key` and `x-api-key`, but its only caller passes only
`x-headroom-*` names, so those branches never run.
- `_is_tls_failure` also matches on message text. Provider HTTP error
responses are now checked first in the backends, so this only affects
non-provider exceptions.
- any-llm: only provider errors raised as `openai`/`anthropic`
`APIStatusError` keep their text; any other any-llm exception type
returns `internal_error`.

Revised 2026-10-01 after review: unrecognised exceptions now return a
fixed `internal_error` message and only provider HTTP error responses
keep their text.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-01 19:28:34 -05:00

301 lines
11 KiB
Python

"""Transport and internal exception text must never reach a client (01-F4).
``httpx``/``ssl``/OS exception text names the resolved upstream host, its IP
and the failing library. Every response path maps such failures to the fixed
vocabulary in :mod:`headroom.proxy.public_errors` and keeps the detail in the
server log under the request id.
Companion: the shared proxy token must never land in request tags (10-F4).
"""
from __future__ import annotations
import asyncio
import json
import ssl
from types import SimpleNamespace
from unittest.mock import AsyncMock
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy import public_errors # noqa: E402
from headroom.proxy.handlers.openai import OpenAIHandlerMixin # noqa: E402
from headroom.proxy.helpers import extract_tags # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
# A string that only ever appears in exception text; every assertion below is
# "this never reaches the wire".
SECRET = "gateway-int.corp.example:443 (10.42.0.7)"
# --- Vocabulary --------------------------------------------------------------
@pytest.mark.parametrize(
("exc", "code"),
[
(httpx.ConnectError(SECRET), public_errors.UPSTREAM_UNREACHABLE),
(httpx.ConnectTimeout(SECRET), public_errors.UPSTREAM_UNREACHABLE),
(httpx.PoolTimeout(SECRET), public_errors.UPSTREAM_UNREACHABLE),
(httpx.ReadTimeout(SECRET), public_errors.UPSTREAM_TIMEOUT),
(httpx.WriteTimeout(SECRET), public_errors.UPSTREAM_TIMEOUT),
(httpx.RemoteProtocolError(SECRET), public_errors.UPSTREAM_PROTOCOL_ERROR),
(httpx.ReadError(SECRET), public_errors.UPSTREAM_PROTOCOL_ERROR),
(httpx.DecodingError(SECRET), public_errors.UPSTREAM_PROTOCOL_ERROR),
(ConnectionResetError(104, SECRET), public_errors.UPSTREAM_UNREACHABLE),
(OSError(113, SECRET), public_errors.UPSTREAM_UNREACHABLE),
(asyncio.TimeoutError(), public_errors.UPSTREAM_TIMEOUT),
(
ssl.SSLCertVerificationError(1, "CERTIFICATE_VERIFY_FAILED"),
public_errors.UPSTREAM_TLS_ERROR,
),
],
)
def test_transport_exceptions_classify_to_fixed_codes(exc, code) -> None:
assert public_errors.classify_exception(exc) == code
assert SECRET not in public_errors.public_message(code, request_id="req-1")
def test_tls_failure_wrapped_in_connect_error_is_tls() -> None:
inner = ssl.SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] " + SECRET)
outer = httpx.ConnectError(SECRET)
outer.__cause__ = inner
assert public_errors.classify_exception(outer) == public_errors.UPSTREAM_TLS_ERROR
def test_non_transport_exceptions_are_not_classified() -> None:
"""Provider API errors (litellm/anyllm exceptions) keep their own message."""
assert public_errors.classify_exception(RuntimeError("model not found: gpt-x")) is None
assert public_errors.classify_exception(ValueError("bad")) is None
assert public_errors.classify_or_internal(ValueError("bad")) == public_errors.INTERNAL_ERROR
def test_public_message_rejects_unknown_codes() -> None:
with pytest.raises(ValueError):
public_errors.public_message("stack trace here")
def test_wire_shapes_carry_code_and_request_id() -> None:
openai = public_errors.openai_error_body(
public_errors.UPSTREAM_UNREACHABLE, request_id="req-9", error_type="connection_error"
)
assert openai["error"]["type"] == "connection_error"
assert openai["error"]["code"] == "upstream_unreachable"
assert openai["error"]["request_id"] == "req-9"
assert "req-9" in openai["error"]["message"]
anthropic = public_errors.anthropic_error_body(public_errors.INTERNAL_ERROR, request_id="req-9")
assert anthropic["type"] == "error"
assert anthropic["error"]["type"] == "api_error"
assert anthropic["error"]["code"] == "internal_error"
assert anthropic["request_id"] == "req-9"
# --- Handler paths -------------------------------------------------------------
def _config(**overrides) -> ProxyConfig:
base: dict[str, object] = {
"optimize": False,
"cache_enabled": False,
"rate_limit_enabled": False,
"cost_tracking_enabled": False,
"log_requests": False,
"ccr_inject_tool": False,
"ccr_handle_responses": False,
"ccr_context_tracking": False,
"image_optimize": False,
"retry_enabled": False,
}
base.update(overrides)
return ProxyConfig(**base)
class _SecretConnectTimeoutClient:
async def request(self, **kwargs): # noqa: ANN001, ANN201
raise httpx.ConnectTimeout(f"connect to {SECRET} timed out")
class _PassthroughRequest:
method = "GET"
headers = {}
url = SimpleNamespace(path="/some/other/path", query="")
async def body(self) -> bytes:
return b""
def test_passthrough_connect_error_body_has_no_upstream_detail() -> None:
handler = object.__new__(OpenAIHandlerMixin)
handler.http_client = _SecretConnectTimeoutClient()
response = asyncio.run(
handler.handle_passthrough(_PassthroughRequest(), "https://api.openai.com")
)
assert response.status_code == 502
payload = json.loads(response.body)
assert payload["error"]["type"] == "connection_error"
assert payload["error"]["code"] == public_errors.UPSTREAM_UNREACHABLE
assert SECRET not in response.body.decode()
def test_direct_streaming_connect_error_event_has_no_upstream_detail(monkeypatch) -> None:
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
with TestClient(create_app(_config()), raise_server_exceptions=False) as client:
proxy = client.app.state.proxy
proxy.http_client.send = AsyncMock(side_effect=httpx.ConnectError(f"refused by {SECRET}"))
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-haiku-4-5",
"max_tokens": 16,
"stream": True,
"messages": [{"role": "user", "content": "hi"}],
},
)
assert response.status_code == 502
text = response.text
assert SECRET not in text
assert "event: error" in text
assert public_errors.UPSTREAM_UNREACHABLE in text
def test_compress_endpoint_error_body_has_no_exception_detail(monkeypatch) -> None:
app = create_app(_config(optimize=True))
with TestClient(
app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345), raise_server_exceptions=False
) as client:
proxy = client.app.state.proxy
monkeypatch.setattr(
proxy,
"_run_compression_in_executor",
AsyncMock(side_effect=RuntimeError(f"compressor exploded reading {SECRET}")),
)
response = client.post(
"/v1/compress",
json={"messages": [{"role": "user", "content": "compress me"}], "model": "gpt-4"},
)
assert response.status_code == 503
body = response.json()
assert body["error"]["type"] == "compression_error"
assert body["error"]["code"] == public_errors.INTERNAL_ERROR
assert SECRET not in response.text
@pytest.mark.asyncio
async def test_memory_tool_result_hides_transport_and_internal_text(tmp_path) -> None:
from headroom.proxy.memory_handler import MemoryConfig, MemoryHandler
handler = MemoryHandler(
MemoryConfig(enabled=False, use_native_tool=True, native_memory_dir=str(tmp_path / "n")),
agent_type="codex",
)
class _Backend:
exc: BaseException = OSError(5, f"sqlite at {SECRET} is locked")
async def save_memory(self, **kwargs): # noqa: ANN003
raise self.exc
backend = _Backend()
handler._backend = backend
transport = json.loads(
await handler._execute_memory_tool("memory_save", {"content": "x"}, "u1")
)
assert transport["status"] == "error"
assert SECRET not in json.dumps(transport)
assert transport["error"] == public_errors.UPSTREAM_UNREACHABLE
backend.exc = RuntimeError(f"save failed at {LEAKY}")
internal = json.loads(await handler._execute_memory_tool("memory_save", {"content": "x"}, "u1"))
assert internal["error"] == public_errors.INTERNAL_ERROR
assert LEAKY not in json.dumps(internal)
# --- Unclassified exceptions fail closed ---------------------------------------
# Exception text that names a local path, an internal host and a token.
LEAKY = r"sqlite at C:\service\tenant-a.db via db-int.corp.example failed with sk-live-secret"
UNCLASSIFIED = [
RuntimeError(LEAKY),
ValueError(LEAKY),
KeyError(LEAKY),
json.JSONDecodeError(LEAKY, "{}", 0),
]
@pytest.mark.parametrize("exc", UNCLASSIFIED, ids=lambda e: type(e).__name__)
def test_client_message_hides_unclassified_exception_text(exc) -> None:
message = public_errors.client_message(exc, LEAKY)
assert message == public_errors.public_message(public_errors.INTERNAL_ERROR)
assert "sk-live-secret" not in message
@pytest.mark.parametrize("exc", UNCLASSIFIED, ids=lambda e: type(e).__name__)
def test_tool_result_error_hides_unclassified_exception_text(exc) -> None:
result = public_errors.tool_result_error(exc)
assert result["error"] == public_errors.INTERNAL_ERROR
assert "sk-live-secret" not in json.dumps(result)
def test_client_message_forwards_provider_http_error_response() -> None:
import openai
response = httpx.Response(401, request=httpx.Request("POST", "https://api.example/v1"))
exc = openai.AuthenticationError("invalid x-api-key", response=response, body=None)
assert public_errors.client_message(exc, "invalid x-api-key") == "invalid x-api-key"
@pytest.mark.asyncio
async def test_litellm_backend_hides_unclassified_exception_text(monkeypatch) -> None:
from headroom.backends import litellm as litellm_backend
monkeypatch.setattr(litellm_backend, "acompletion", AsyncMock(side_effect=RuntimeError(LEAKY)))
backend = litellm_backend.LiteLLMBackend(provider="openai")
result = await backend.send_openai_message(
{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}]}, {}
)
assert "sk-live-secret" not in json.dumps(result.body)
assert result.body["error"]["message"] == public_errors.public_message(
public_errors.INTERNAL_ERROR
)
# --- Tags (10-F4) -------------------------------------------------------------
def test_extract_tags_never_carries_credentials() -> None:
tags = extract_tags(
{
"X-Headroom-Proxy-Token": "hlk_secret_token_value",
"x-headroom-project": "alpha",
"x-headroom-session-id": "s-1",
"x-headroom-license-token": "lic",
"x-headroom-api-key": "k",
"X-Headroom-Key": "hr_cloud_key",
"authorization": "Bearer sk-live",
}
)
assert tags == {"project": "alpha", "session-id": "s-1"}
def test_public_tags_drop_credential_like_keys_defensively() -> None:
from headroom.proxy.savings_attribution import public_tags
assert public_tags({"proxy-token": "x", "project": "alpha", "some_secret": "y"}) == {
"project": "alpha"
}