fix(proxy): keep exception text and the proxy token out of client output (#3851)

## Why

Two leaks of internal detail to clients and logs (findings 01-F4 and
10-F4).

- When an upstream connection failed, several response paths returned
`str(e)` from `httpx`, `ssl` or the OS in 500/502/503 bodies and SSE
`error` events. That text names the resolved upstream host, its address
and the failing library. A client that can steer the upstream
(`x-headroom-base-url`) could learn internal hostnames one error at a
time.
- `extract_tags` copied every `x-headroom-*` header into
`RequestOutcome.tags`, including `x-headroom-proxy-token`, which
gateways send on every turn. The token then appeared in the request log
(`--log-file`), dashboard facets and any export that carries tags.

## What changes

- New module `headroom/proxy/public_errors.py`. `classify_exception()`
maps transport and OS failures to fixed codes: `upstream_unreachable`,
`upstream_timeout`, `upstream_tls_error`, `upstream_protocol_error`;
anything else is `internal_error`. Every client- or model-visible path
fails closed: unrecognised exception text (`RuntimeError`, `ValueError`,
parser, database or library errors) never leaves the proxy. Provider
error text is forwarded only when the exception carries a provider HTTP
error response (`openai.APIStatusError`, which LiteLLM's provider errors
subclass, or `anthropic.APIStatusError`). Helpers build the existing
wire shapes with the same `error.type`, plus `code` and `request_id`.
The full exception stays in the server log under that request id.
- Every site that built a client-visible message from an exception now
uses it: `handlers/streaming.py` (direct, Bedrock and OpenAI-via-backend
SSE), `handlers/openai.py` (backend 500, `/v1/compress` 503, both
passthrough connect paths), `handlers/anthropic.py` (Bedrock backend
500), `handlers/bedrock.py` (connect 502), the memory tool handlers
(tool results returned to the model), and the `litellm` / `anyllm`
backends. The existing corporate-TLS hint is kept.
- `internal_header_policy.py` gains `is_credential_header()` and
`is_credential_tag_key()`. They match whole name components (`token`,
`secret`, `password`, `authorization`, `cookie`, `apikey`, `key`, and
similar), so `X-Headroom-Key` and `X-Headroom-Api-Key` are covered, so
numeric tags such as `tool-search-deferred-tokens` are not caught.
- `extract_tags` skips credential headers.
`savings_attribution.public_tags`, the last step before the request log,
also drops credential-shaped keys.

## Behaviour change

- Transport-layer and internal failures now return a fixed sentence plus
`code` and `request_id`. Clients that parsed the old free-text messages
should read `error.code`. Provider HTTP error responses (and their
status codes) are unchanged. In the LiteLLM / any-llm backends, an
exception that is not a provider response now returns `The proxy could
not complete the request.`; the HTTP status mapping is unchanged.
- Memory tool results for any failure return `{"status": "error",
"error": "<code>", "message": ...}` instead of the exception text.
- `x-headroom-*` headers whose name contains a credential component are
no longer recorded as tags. A benign label that happens to use such a
name should be renamed.

## Tests

- `tests/test_public_errors.py`: classification of transport exceptions
(including TLS wrapped in a connect error), non-transport exceptions
left alone, wire shapes carry `code` and `request_id`; the passthrough,
direct-streaming and `/v1/compress` error paths contain no upstream
detail; memory tool results hide transport and internal text;
`client_message` / `tool_result_error` / the LiteLLM backend return
`internal_error` for `RuntimeError`, `ValueError`, `KeyError` and
`JSONDecodeError` carrying a path, host and token (10 tests in this file
fail without the fail-closed change), while an
`openai.AuthenticationError` keeps its text; `extract_tags` and
`public_tags` never carry credentials. Six of these fail on `main`.
- Existing tests that asserted the old exception text in client output
now assert the new contract: the client gets the fixed `code` and
`request_id`, the exception text is absent, and the `headroom.proxy` log
has the full text under the same `request_id`
(`test_litellm_transport_errors.py`, `test_openai_backend_path.py`,
`test_proxy_compress_endpoint.py`,
`test_proxy_streaming_ratelimit_headers.py`).
- Manual run with every upstream connect failing: streaming
`/v1/messages`, streaming `/v1/chat/completions` and passthrough
`/v1/models` return `code: upstream_unreachable` with no address. A
request with `x-headroom-proxy-token` and `x-headroom-license-token`
leaves neither value in the `--log-file` output.

Not tested live: the Bedrock and LiteLLM / any-llm backend paths (unit
tests only) and OTEL/Langfuse export (tags are filtered before they
reach it).

## Open review points

- Resolved: unclassified exception text (JerrettDavis, 2026-09-29) now
fails closed in `client_message` and `tool_result_error`.
- Resolved: `X-Headroom-Key` is matched (`key` is a credential
component; the `api`+`key` pair rule is gone). No internal or test tag
key has a `key` component.
- `is_credential_header` special-cases `authorization`, `cookie`,
`api-key` and `x-api-key`, but its only caller passes only
`x-headroom-*` names, so those branches never run.
- `_is_tls_failure` also matches on message text. Provider HTTP error
responses are now checked first in the backends, so this only affects
non-provider exceptions.
- any-llm: only provider errors raised as `openai`/`anthropic`
`APIStatusError` keep their text; any other any-llm exception type
returns `internal_error`.

Revised 2026-10-01 after review: unrecognised exceptions now return a
fixed `internal_error` message and only provider HTTP error responses
keep their text.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-10-01 19:28:34 -05:00
committed by GitHub
co-authored by Claude Fable 5.1
parent 3c9ed0101a
commit 861e94d8c1
19 changed files with 835 additions and 128 deletions
+6 -3
View File
@@ -12,6 +12,7 @@ import uuid
from collections.abc import AsyncIterator
from typing import Any, cast
from headroom.proxy.public_errors import client_message
from headroom.utils import format_exception_message
from .base import Backend, BackendResponse, StreamEvent
@@ -505,7 +506,7 @@ class AnyLLMBackend(Backend):
event_type="error",
data={
"type": "error",
"error": {"type": "api_error", "message": error_message},
"error": {"type": "api_error", "message": client_message(e, error_message)},
},
)
@@ -601,7 +602,9 @@ class AnyLLMBackend(Backend):
error_type = "api_error"
status_code = 500
error_message = format_exception_message(e)
# Provider API errors keep their text; transport failures are reduced
# to the public vocabulary (see proxy/public_errors).
error_message = client_message(e, format_exception_message(e))
error_str = str(e).lower()
if "authentication" in error_str or "api_key" in error_str or "api key" in error_str:
error_type = "invalid_api_key" if openai_format else "authentication_error"
@@ -669,7 +672,7 @@ class AnyLLMBackend(Backend):
logger.error(f"any-llm OpenAI streaming error: {error_message}")
error_data = {
"error": {
"message": error_message,
"message": client_message(e, error_message),
"type": "api_error",
"code": "backend_error",
}
+7 -2
View File
@@ -21,6 +21,7 @@ from collections.abc import AsyncIterator
from dataclasses import dataclass, field
from typing import Any
from headroom.proxy.public_errors import client_message
from headroom.utils import format_exception_message
from .base import Backend, BackendResponse, StreamEvent
@@ -1274,6 +1275,9 @@ class LiteLLMBackend(Backend):
except Exception as e:
error_message = format_exception_message(e)
logger.error(f"LiteLLM error: {error_message}")
# Provider API errors keep their text; transport failures are
# reduced to the public vocabulary (see proxy/public_errors).
error_message = client_message(e, error_message)
# Map to Anthropic error format
error_type = "api_error"
@@ -1689,7 +1693,7 @@ class LiteLLMBackend(Backend):
event_type="error",
data={
"type": "error",
"error": {"type": "api_error", "message": error_message},
"error": {"type": "api_error", "message": client_message(e, error_message)},
},
)
@@ -1886,6 +1890,7 @@ class LiteLLMBackend(Backend):
except Exception as e:
error_message = format_exception_message(e)
logger.error(f"LiteLLM OpenAI error: {error_message}")
error_message = client_message(e, error_message)
# Map to OpenAI error format
error_type = "api_error"
@@ -1997,7 +2002,7 @@ class LiteLLMBackend(Backend):
logger.error(f"LiteLLM OpenAI streaming error: {error_message}")
error_data = {
"error": {
"message": error_message,
"message": client_message(e, error_message),
"type": "api_error",
"code": "backend_error",
}
+6 -4
View File
@@ -33,6 +33,7 @@ from headroom.ccr.context_tracker import looks_like_claude_code_compact_summary
from headroom.ccr.marker_resolution import resolve_markers_in_response
from headroom.copilot_auth import apply_copilot_api_auth, is_copilot_upstream_url
from headroom.pipeline import PipelineStage, summarize_routing_markers
from headroom.proxy import public_errors
from headroom.proxy.anthropic_wire import (
build_anthropic_upstream_url,
is_safeguard_capable_request,
@@ -3782,10 +3783,11 @@ class AnthropicHandlerMixin:
await _finalize_pre_upstream()
return JSONResponse(
status_code=500,
content={
"type": "error",
"error": {"type": "api_error", "message": error_message},
},
content=public_errors.anthropic_error_body(
public_errors.classify_or_internal(e),
request_id=str(request_id),
error_type="api_error",
),
)
# Direct Anthropic API, or a provider-compatible Anthropic
+7 -6
View File
@@ -301,14 +301,15 @@ class BedrockHandlerMixin:
upstream = await self.http_client.send(upstream_request, stream=True) # type: ignore[attr-defined]
except (httpx.ConnectError, httpx.TimeoutException) as err:
logger.warning("[%s] %s upstream connect failed: %s", request_id, LOG_TAG, err)
from headroom.proxy import public_errors
return JSONResponse(
status_code=502,
content={
"error": {
"type": "connection_error",
"message": f"Failed to connect to Bedrock upstream: {err}",
}
},
content=public_errors.openai_error_body(
public_errors.classify_or_internal(err),
request_id=str(request_id),
error_type="connection_error",
),
)
# Forward raw (still-encoded) bytes, so strip hop-by-hop headers that
+20 -25
View File
@@ -77,6 +77,7 @@ from headroom.providers.codex.runtime import (
from headroom.providers.copilot import model_prefers_responses_api
from headroom.providers.grok.runtime import DEFAULT_API_URL as XAI_API_URL
from headroom.providers.proxy_targets import route_grok_to_xai
from headroom.proxy import public_errors
from headroom.proxy.auth_mode import (
classify_auth_mode,
classify_client,
@@ -5131,13 +5132,11 @@ class OpenAIHandlerMixin:
logger.error(f"[{request_id}] Backend error: {e}")
return JSONResponse(
status_code=500,
content={
"error": {
"message": str(e),
"type": "api_error",
"code": "backend_error",
}
},
content=public_errors.openai_error_body(
public_errors.classify_or_internal(e),
request_id=str(request_id),
error_type="api_error",
),
)
# Direct OpenAI API (no backend configured). Reuse the upstream resolved
@@ -10875,12 +10874,10 @@ class OpenAIHandlerMixin:
await self.metrics.record_failed(provider="compress")
return JSONResponse(
status_code=503,
content={
"error": {
"type": "compression_error",
"message": str(e),
}
},
content=public_errors.openai_error_body(
public_errors.classify_or_internal(e),
error_type="compression_error",
),
)
async def handle_compress_usage(self, request: Request) -> JSONResponse:
@@ -11267,12 +11264,11 @@ class OpenAIHandlerMixin:
tls_hint = await describe_upstream_failure_async(e, url)
return Response(
content=json.dumps(
{
"error": {
"type": "connection_error",
"message": tls_hint or f"Failed to connect to upstream API: {e}",
}
}
public_errors.openai_error_body(
public_errors.classify_or_internal(e),
error_type="connection_error",
hint=tls_hint,
)
),
status_code=502,
media_type="application/json",
@@ -11458,12 +11454,11 @@ class OpenAIHandlerMixin:
tls_hint = await describe_upstream_failure_async(e, url)
return Response(
content=json.dumps(
{
"error": {
"type": "connection_error",
"message": tls_hint or f"Failed to connect to upstream API: {e}",
}
}
public_errors.openai_error_body(
public_errors.classify_or_internal(e),
error_type="connection_error",
hint=tls_hint,
)
),
status_code=502,
media_type="application/json",
+30 -31
View File
@@ -28,6 +28,7 @@ if TYPE_CHECKING:
import httpx
from headroom.copilot_auth import apply_copilot_api_auth
from headroom.proxy import public_errors
from headroom.proxy.stream_output_tokens import estimate_output_tokens
from headroom.proxy.thinking_tokens import ThinkingTokens, extract_thinking_tokens
from headroom.utils import format_exception_message
@@ -1063,17 +1064,17 @@ class StreamingMixin:
from headroom.proxy.tls_diagnostics import describe_upstream_failure_async
tls_hint = await describe_upstream_failure_async(e, url)
client_message = tls_hint or f"Failed to connect to upstream API: {error_msg}"
# The exception text itself stays in the log above: it names the
# resolved upstream host and address (public_errors module doc).
error_body = public_errors.anthropic_error_body(
public_errors.classify_or_internal(e),
request_id=request_id,
error_type="connection_error",
hint=tls_hint,
)
async def _error_gen():
error_event = {
"type": "error",
"error": {
"type": "connection_error",
"message": client_message,
},
}
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
yield f"event: error\ndata: {json.dumps(error_body)}\n\n".encode()
self._cleanup_mid_turn_stream(session_key)
return StreamingResponse(
@@ -1428,13 +1429,11 @@ class StreamingMixin:
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.PoolTimeout) as e:
logger.error(f"[{request_id}] Connection error to upstream API: {e}")
error_event = {
"type": "error",
"error": {
"type": "connection_error",
"message": f"Failed to connect to upstream API: {e}",
},
}
error_event = public_errors.anthropic_error_body(
public_errors.classify_or_internal(e),
request_id=request_id,
error_type="connection_error",
)
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
except httpx.HTTPStatusError as e:
logger.error(f"[{request_id}] HTTP error from upstream API: {e}")
@@ -1442,10 +1441,11 @@ class StreamingMixin:
yield e.response.content
except Exception as e:
logger.error(f"[{request_id}] Unexpected streaming error: {e}")
error_event = {
"type": "error",
"error": {"type": "api_error", "message": str(e)},
}
error_event = public_errors.anthropic_error_body(
public_errors.classify_or_internal(e),
request_id=request_id,
error_type="api_error",
)
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
finally:
pending_messages = self._cleanup_mid_turn_stream(
@@ -1673,10 +1673,11 @@ class StreamingMixin:
except Exception as e:
error_message = format_exception_message(e)
logger.error(f"[{request_id}] Bedrock streaming error: {error_message}")
error_event = {
"type": "error",
"error": {"type": "api_error", "message": error_message},
}
error_event = public_errors.anthropic_error_body(
public_errors.classify_or_internal(e),
request_id=request_id,
error_type="api_error",
)
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
finally:
@@ -1868,13 +1869,11 @@ class StreamingMixin:
yield chunk_bytes
except Exception as e:
logger.error(f"[{request_id}] Backend streaming error: {e}")
error_data = {
"error": {
"message": str(e),
"type": "api_error",
"code": "backend_error",
}
}
error_data = public_errors.openai_error_body(
public_errors.classify_or_internal(e),
request_id=request_id,
error_type="api_error",
)
yield f"data: {json.dumps(error_data)}\n\n".encode()
yield b"data: [DONE]\n\n"
finally:
+10 -1
View File
@@ -301,11 +301,20 @@ def extract_tags(headers: Any) -> dict[str, str]:
Header name match is case-insensitive; the returned key has the
``x-headroom-`` prefix stripped.
Credential-bearing headers are never tags. ``x-headroom-proxy-token`` is
the proxy's own bearer and gateways (Kong, Envoy, LiteLLM) send it on every
turn; before this filter it landed in ``RequestOutcome.tags`` and from
there in the request log, dashboard facets and every export that carries
tags. The rule lives in :mod:`internal_header_policy` so producers and
consumers of tags agree on it.
"""
from headroom.proxy.internal_header_policy import is_credential_header
return {
k.lower().replace("x-headroom-", ""): v
for k, v in headers.items()
if k.lower().startswith("x-headroom-")
if k.lower().startswith("x-headroom-") and not is_credential_header(k)
}
+58
View File
@@ -10,6 +10,64 @@ STRIP_INTERNAL_HEADERS_ENV = "HEADROOM_STRIP_INTERNAL_HEADERS"
StripInternalHeadersMode = Literal["enabled", "disabled"]
STRIP_INTERNAL_HEADERS_DEFAULT: StripInternalHeadersMode = "enabled"
# The proxy's own credential header. Read by the security gate and the
# WebSocket auth middleware; it must never be copied into tags, logs, metrics
# labels or exports, which is what ``is_credential_header`` guards.
PROXY_TOKEN_HEADER = "x-headroom-proxy-token"
# Name components that mark an ``x-headroom-*`` header (or the tag key derived
# from it by stripping the prefix) as credential material rather than a label.
# Names are lower-cased, ``_`` is folded to ``-`` and the name is split on
# ``-``, so ``X-Headroom-License_Token`` -> ``license-token`` -> {license, token}.
# Whole components only: ``proxy-token`` is a credential, the numeric
# ``tool-search-deferred-tokens`` tag is not.
CREDENTIAL_NAME_COMPONENTS: frozenset[str] = frozenset(
{
"token",
"secret",
"password",
"passwd",
"credential",
"credentials",
"authorization",
"apikey",
"key", # x-headroom-key (Headroom Cloud), x-headroom-api-key
"bearer",
"cookie",
"signature",
}
)
def _name_components(name: str) -> list[str]:
return [part for part in name.strip().lower().replace("_", "-").split("-") if part]
def is_credential_tag_key(key: str) -> bool:
"""True if a tag key (header name minus the ``x-headroom-`` prefix) is a credential.
Tag keys become request-log fields, dashboard facets and export labels,
so this is the single rule every tag producer and consumer applies.
"""
parts = _name_components(key)
if parts[:2] == ["x", "headroom"]:
parts = parts[2:]
return any(part in CREDENTIAL_NAME_COMPONENTS for part in parts)
def _normalise_name(name: str) -> str:
return "-".join(_name_components(name))
def is_credential_header(name: str) -> bool:
"""True if an inbound header carries a credential and must not be tagged."""
normalised = _normalise_name(name)
if normalised == PROXY_TOKEN_HEADER:
return True
if normalised in ("authorization", "proxy-authorization", "cookie", "api-key", "x-api-key"):
return True
return normalised.startswith(INTERNAL_HEADER_PREFIX) and is_credential_tag_key(normalised)
def resolve_strip_internal_headers_mode(raw: str | None) -> StripInternalHeadersMode:
"""Resolve the configured internal-header strip mode."""
+4 -3
View File
@@ -43,6 +43,7 @@ from headroom.memory.storage_router import (
RequestContext,
ResolvedScope,
)
from headroom.proxy.public_errors import tool_result_error
if TYPE_CHECKING:
from headroom.memory.backends.local import LocalBackend
@@ -1232,7 +1233,7 @@ your responses, not to drive new actions."""
except Exception as e:
logger.error(f"Memory: Tool {tool_name} failed: {e}")
return json.dumps({"status": "error", "error": str(e)})
return json.dumps(tool_result_error(e))
async def _execute_save(
self,
@@ -1521,7 +1522,7 @@ your responses, not to drive new actions."""
results = await list_fn(user_id=effective_user_id, limit=limit)
except Exception as e:
logger.warning(f"Memory: list_memories failed for user {effective_user_id}: {e}")
return json.dumps({"status": "error", "error": str(e)})
return json.dumps(tool_result_error(e))
else:
try:
results = await backend.search_memories(
@@ -1531,7 +1532,7 @@ your responses, not to drive new actions."""
)
except Exception as e:
logger.warning(f"Memory: list fallback search failed: {e}")
return json.dumps({"status": "error", "error": str(e)})
return json.dumps(tool_result_error(e))
entries: list[dict[str, Any]] = []
for r in results:
+3 -1
View File
@@ -32,6 +32,8 @@ import logging
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Literal
from headroom.proxy.public_errors import tool_result_error
if TYPE_CHECKING:
from headroom.memory.backends.local import LocalBackend
@@ -1145,7 +1147,7 @@ To SAVE: create /memories/<topic>.txt "content"
return json.dumps({"error": f"Unknown tool: {tool_name}"})
except Exception as e:
logger.error(f"MemoryToolAdapter: Tool {tool_name} failed: {e}")
return json.dumps({"status": "error", "error": str(e)})
return json.dumps(tool_result_error(e))
async def _execute_save(self, input_data: dict[str, Any], user_id: str) -> str:
"""Execute memory_save tool."""
+256
View File
@@ -0,0 +1,256 @@
"""Fixed-vocabulary client-facing errors for transport and internal failures.
Why this exists
---------------
When an upstream connection fails, the exception text produced by ``httpx``,
``ssl`` or the OS names the resolved upstream host, its IP address, the
library that failed and sometimes the local interface. Returning that text to
the caller (``"message": str(e)``) turns every transport failure into an
information-disclosure finding: a client that can steer the upstream (for
example through ``x-headroom-base-url``) learns internal hostnames and
addresses one error at a time.
The rule enforced here is simple and applies to every response path:
* **Transport-layer and internal exception text never reaches a client.** It is
logged server-side with the request id, and the client receives one of the
fixed codes below plus that request id so the two can be correlated.
* **Only a provider's own HTTP error response is forwarded.** A model
provider's ``{"error": {"message": ...}}`` (authentication failed, model not
found, rate limited) is addressed to the caller. It is recognised by
exception type (:func:`is_provider_response_error`), never by guessing from
text; every other exception (``RuntimeError``, ``ValueError``, parser,
database or library errors) fails closed to :data:`INTERNAL_ERROR`.
The module is pure policy: no I/O, no logging, no FastAPI types. Handlers and
backends build their wire shape (OpenAI ``{"error": {...}}`` or Anthropic
``{"type": "error", "error": {...}}``) with the helpers at the bottom so the
vocabulary cannot drift between transports.
"""
from __future__ import annotations
import asyncio
import ssl
from typing import Any, Final
try: # httpx is a proxy-extra dependency; the SDK imports this module without it.
import httpx
except ImportError: # pragma: no cover - exercised only in SDK-only installs
httpx = None # type: ignore[assignment]
# --- The vocabulary --------------------------------------------------------
UPSTREAM_UNREACHABLE: Final = "upstream_unreachable"
UPSTREAM_TIMEOUT: Final = "upstream_timeout"
UPSTREAM_TLS_ERROR: Final = "upstream_tls_error"
UPSTREAM_PROTOCOL_ERROR: Final = "upstream_protocol_error"
INTERNAL_ERROR: Final = "internal_error"
PUBLIC_ERROR_CODES: Final[frozenset[str]] = frozenset(
{
UPSTREAM_UNREACHABLE,
UPSTREAM_TIMEOUT,
UPSTREAM_TLS_ERROR,
UPSTREAM_PROTOCOL_ERROR,
INTERNAL_ERROR,
}
)
# One sentence per code. These are the only strings a client sees for a
# transport or internal failure; they carry no host, address, path or
# library detail by construction.
_MESSAGES: Final[dict[str, str]] = {
UPSTREAM_UNREACHABLE: "Failed to connect to upstream API.",
UPSTREAM_TIMEOUT: "Upstream API did not respond in time.",
UPSTREAM_TLS_ERROR: "Could not establish a trusted TLS connection to the upstream API.",
UPSTREAM_PROTOCOL_ERROR: ("Upstream API closed the connection or sent a malformed response."),
INTERNAL_ERROR: "The proxy could not complete the request.",
}
# Anthropic's error envelope uses ``error.type``; keep the values it already
# emits today so SDKs keep classifying them the same way.
_ANTHROPIC_TYPES: Final[dict[str, str]] = {
UPSTREAM_UNREACHABLE: "connection_error",
UPSTREAM_TIMEOUT: "connection_error",
UPSTREAM_TLS_ERROR: "connection_error",
UPSTREAM_PROTOCOL_ERROR: "api_error",
INTERNAL_ERROR: "api_error",
}
# --- Classification --------------------------------------------------------
def _iter_causes(exc: BaseException) -> list[BaseException]:
"""The exception and its ``__cause__``/``__context__`` chain, outermost first."""
seen: list[BaseException] = []
current: BaseException | None = exc
while current is not None and current not in seen and len(seen) < 16:
seen.append(current)
current = current.__cause__ or current.__context__
return seen
def _is_tls_failure(exc: BaseException) -> bool:
for item in _iter_causes(exc):
if isinstance(item, ssl.SSLError):
return True
text = str(item).upper()
if "CERTIFICATE_VERIFY_FAILED" in text or "SSL:" in text or "TLS" in text.split():
return True
return False
def classify_exception(exc: BaseException) -> str | None:
"""Return the public code for a transport/runtime failure, or ``None``.
``None`` means "this is not a transport-layer failure". Client-facing code
uses :func:`classify_or_internal` so that anything unrecognised becomes
:data:`INTERNAL_ERROR`: it was never meant for a client.
"""
if _is_tls_failure(exc):
return UPSTREAM_TLS_ERROR
if httpx is not None:
if isinstance(exc, (httpx.ConnectTimeout, httpx.PoolTimeout)):
return UPSTREAM_UNREACHABLE
if isinstance(exc, httpx.TimeoutException):
return UPSTREAM_TIMEOUT
if isinstance(exc, httpx.ConnectError):
return UPSTREAM_UNREACHABLE
if isinstance(exc, httpx.ProtocolError):
return UPSTREAM_PROTOCOL_ERROR
if isinstance(exc, httpx.DecodingError):
return UPSTREAM_PROTOCOL_ERROR
if isinstance(exc, httpx.NetworkError):
# ReadError / WriteError / CloseError: the connection existed and
# then broke mid-exchange.
return UPSTREAM_PROTOCOL_ERROR
if isinstance(exc, httpx.TransportError):
return UPSTREAM_UNREACHABLE
if isinstance(exc, (asyncio.TimeoutError, TimeoutError)):
return UPSTREAM_TIMEOUT
if isinstance(exc, ConnectionError):
# ConnectionResetError / BrokenPipeError / ConnectionRefusedError.
return UPSTREAM_UNREACHABLE
if isinstance(exc, OSError):
# Anything else from the socket layer (EHOSTUNREACH, ENETDOWN, ...).
return UPSTREAM_UNREACHABLE
return None
def classify_or_internal(exc: BaseException) -> str:
"""Classify, treating anything unrecognised as :data:`INTERNAL_ERROR`."""
return classify_exception(exc) or INTERNAL_ERROR
# --- Wire shapes -------------------------------------------------------------
def public_message(code: str, *, request_id: str | None = None, hint: str | None = None) -> str:
"""The client-facing sentence for ``code``.
``hint`` is an *already public* operator-facing string (for example the
corporate TLS-inspection explanation from ``tls_diagnostics``) and is
appended verbatim; callers must not pass exception text through it.
"""
if code not in PUBLIC_ERROR_CODES:
raise ValueError(f"not a public error code: {code!r}")
text = _MESSAGES[code]
if hint:
text = f"{text} {hint.strip()}"
if request_id:
text = f"{text} (request_id={request_id})"
return text
def openai_error_body(
code: str,
*,
request_id: str | None = None,
error_type: str | None = None,
hint: str | None = None,
) -> dict[str, Any]:
"""``{"error": {...}}`` in the OpenAI shape.
``error_type`` defaults to the value each call site emitted before this
module existed (``connection_error`` for connection failures, otherwise
``api_error``) so clients that switch on it keep working; ``code`` always
carries the fixed vocabulary.
"""
error: dict[str, Any] = {
"message": public_message(code, request_id=request_id, hint=hint),
"type": error_type or _ANTHROPIC_TYPES[code],
"code": code,
}
if request_id:
error["request_id"] = request_id
return {"error": error}
def anthropic_error_body(
code: str,
*,
request_id: str | None = None,
error_type: str | None = None,
hint: str | None = None,
) -> dict[str, Any]:
"""``{"type": "error", "error": {...}}`` in the Anthropic shape."""
error: dict[str, Any] = {
"type": error_type or _ANTHROPIC_TYPES[code],
"message": public_message(code, request_id=request_id, hint=hint),
"code": code,
}
body: dict[str, Any] = {"type": "error", "error": error}
if request_id:
body["request_id"] = request_id
return body
def is_provider_response_error(exc: BaseException) -> bool:
"""True when ``exc`` carries a model provider's HTTP error response.
LiteLLM's provider errors (AuthenticationError, NotFoundError,
RateLimitError, ...) subclass ``openai.APIStatusError``. Imported lazily:
neither SDK is needed to import this module.
"""
try:
import openai
if isinstance(exc, openai.APIStatusError):
return True
except ImportError: # pragma: no cover - openai is a proxy-extra dependency
pass
try:
import anthropic
if isinstance(exc, anthropic.APIStatusError):
return True
except ImportError: # pragma: no cover - optional dependency
pass
return False
def client_message(exc: BaseException, provider_text: str) -> str:
"""The message a client may see for ``exc``.
``provider_text`` is returned only when ``exc`` is a provider's HTTP error
response (authentication, model not found, rate limited): that text is
addressed to the caller. Transport failures map to their fixed code and
anything else to :data:`INTERNAL_ERROR`; the caller logs the real text.
"""
if is_provider_response_error(exc):
return provider_text
return public_message(classify_or_internal(exc))
def tool_result_error(exc: BaseException) -> dict[str, Any]:
"""``{"status": "error", ...}`` for a JSON tool result returned to a model.
The model only ever sees a fixed code: a storage, network or internal
failure names no path, host or library. The caller logs the real text.
"""
code = classify_or_internal(exc)
return {"status": "error", "error": code, "message": public_message(code)}
+11 -1
View File
@@ -212,8 +212,18 @@ def public_tags(tags: MutableMapping[str, Any] | None) -> dict[str, Any]:
They are carried on ``tags`` because that is the one dict that reaches the
outcome funnel from every handler; letting them through to ``RequestLog``
would put a list and a dict into a string-keyed label store.
Credential-shaped keys are dropped here as well. :func:`helpers.extract_tags`
already refuses them at the source; this is the last gate before the
request log for tags a hook or extension added by hand.
"""
return {key: value for key, value in (tags or {}).items() if key not in _INTERNAL_TAGS}
from headroom.proxy.internal_header_policy import is_credential_tag_key
return {
key: value
for key, value in (tags or {}).items()
if key not in _INTERNAL_TAGS and not is_credential_tag_key(str(key))
}
def encode(items: list[dict[str, Any]]) -> str:
+3 -2
View File
@@ -377,7 +377,8 @@ async def test_send_message_returns_error_response(monkeypatch: pytest.MonkeyPat
assert result.status_code == 401
assert result.body["error"]["type"] == "authentication_error"
assert result.error == "authentication api_key missing"
# Unclassified exception text stays in the server log, never the client.
assert result.error == "The proxy could not complete the request."
@pytest.mark.asyncio
@@ -416,7 +417,7 @@ async def test_stream_message_yields_events_and_error(monkeypatch: pytest.Monkey
instance_error.raise_error = RuntimeError("stream broke")
error_events = [event async for event in backend_error.stream_message({"messages": []}, {})]
assert error_events[-1].event_type == "error"
assert error_events[-1].data["error"]["message"] == "stream broke"
assert error_events[-1].data["error"]["message"] == "The proxy could not complete the request."
def _tool_call_delta(*, index, tc_id=None, name=None, arguments=None): # noqa: ANN001, ANN202
+28 -14
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import json
from collections.abc import AsyncIterator
from unittest.mock import AsyncMock, MagicMock, patch
@@ -17,8 +18,18 @@ from fastapi.testclient import TestClient # noqa: E402
from headroom.backends.anyllm import AnyLLMBackend # noqa: E402
from headroom.backends.base import BackendResponse # noqa: E402
from headroom.backends.litellm import LiteLLMBackend # noqa: E402
from headroom.proxy.public_errors import ( # noqa: E402
UPSTREAM_PROTOCOL_ERROR,
UPSTREAM_TIMEOUT,
public_message,
)
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
# Transport exceptions never surface their own text (it names the upstream host);
# an empty-message ReadError/ReadTimeout maps to the fixed vocabulary instead.
_PROTOCOL = public_message(UPSTREAM_PROTOCOL_ERROR)
_TIMEOUT = public_message(UPSTREAM_TIMEOUT)
_BODY = {
"model": "claude-sonnet-4-20250514",
"messages": [{"role": "user", "content": "hello"}],
@@ -39,8 +50,8 @@ async def test_send_message_names_transport_error_without_message() -> None:
result = await backend.send_message(_BODY, {})
assert result.status_code == 500
assert result.error == "ReadError (no message)"
assert result.body["error"]["message"] == "ReadError (no message)"
assert result.error == _PROTOCOL
assert result.body["error"]["message"] == _PROTOCOL
@pytest.mark.asyncio
@@ -57,7 +68,7 @@ async def test_stream_message_names_transport_error_without_message() -> None:
events = [event async for event in backend.stream_message(_BODY, {})]
error_event = next(event for event in events if event.event_type == "error")
assert error_event.data["error"]["message"] == "ReadTimeout (no message)"
assert error_event.data["error"]["message"] == _TIMEOUT
@pytest.mark.asyncio
@@ -71,8 +82,8 @@ async def test_anyllm_backend_names_transport_error_without_message() -> None:
result = await backend.send_message(_BODY, {})
assert result.status_code == 500
assert result.error == "ReadError (no message)"
assert result.body["error"]["message"] == "ReadError (no message)"
assert result.error == _PROTOCOL
assert result.body["error"]["message"] == _PROTOCOL
@pytest.mark.asyncio
@@ -86,7 +97,7 @@ async def test_anyllm_stream_backend_names_transport_error_without_message() ->
events = [event async for event in backend.stream_message(_BODY, {})]
error_event = next(event for event in events if event.event_type == "error")
assert error_event.data["error"]["message"] == "ReadTimeout (no message)"
assert error_event.data["error"]["message"] == _TIMEOUT
@pytest.mark.asyncio
@@ -108,10 +119,10 @@ async def test_openai_backend_boundaries_name_transport_errors_without_message()
litellm_backend = LiteLLMBackend(provider="openrouter")
litellm_result = await litellm_backend.send_openai_message(_BODY, {})
assert anyllm_result.body["error"]["message"] == "ReadError (no message)"
assert anyllm_result.error == "ReadError (no message)"
assert litellm_result.body["error"]["message"] == "ReadTimeout (no message)"
assert litellm_result.error == "ReadTimeout (no message)"
assert anyllm_result.body["error"]["message"] == _PROTOCOL
assert anyllm_result.error == _PROTOCOL
assert litellm_result.body["error"]["message"] == _TIMEOUT
assert litellm_result.error == _TIMEOUT
@pytest.mark.asyncio
@@ -133,8 +144,8 @@ async def test_openai_stream_boundaries_name_transport_errors_without_message()
litellm_backend = LiteLLMBackend(provider="openrouter")
litellm_chunks = [chunk async for chunk in litellm_backend.stream_openai_message(_BODY, {})]
assert '"message": "ReadError (no message)"' in anyllm_chunks[0]
assert '"message": "ReadTimeout (no message)"' in litellm_chunks[0]
assert f'"message": {json.dumps(_PROTOCOL)}' in anyllm_chunks[0]
assert f'"message": {json.dumps(_TIMEOUT)}' in litellm_chunks[0]
def _erroring_anthropic_backend() -> MagicMock:
@@ -187,7 +198,9 @@ def test_anthropic_proxy_names_nonstream_transport_error_without_message() -> No
)
assert response.status_code == 500
assert response.json()["error"]["message"] == "ReadError (no message)"
# Proxy-level replies append the request id for log correlation.
assert response.json()["error"]["message"].startswith(_PROTOCOL)
assert response.json()["error"]["code"] == UPSTREAM_PROTOCOL_ERROR
def test_bedrock_stream_names_transport_error_without_message() -> None:
@@ -202,4 +215,5 @@ def test_bedrock_stream_names_transport_error_without_message() -> None:
)
assert response.status_code == 200
assert '"message": "ReadTimeout (no message)"' in response.text
assert _TIMEOUT in response.text
assert '"code": "upstream_timeout"' in response.text
+4 -1
View File
@@ -751,7 +751,10 @@ async def test_execute_memory_tool_save_returns_dedup_hint(handler: MemoryHandle
backend.raise_on = "save"
errored = json.loads(await handler._execute_memory_tool("memory_save", {"content": "x"}, "u1"))
assert errored == {"status": "error", "error": "save failed"}
# Backend exception text is logged, never returned to the model.
assert errored["status"] == "error"
assert errored["error"] == "internal_error"
assert "save failed" not in json.dumps(errored)
@pytest.mark.asyncio
+22 -1
View File
@@ -16,6 +16,7 @@ don't need a real provider:
from __future__ import annotations
import logging
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
@@ -359,10 +360,22 @@ def test_backend_ccr_intercept_exception_is_reraised_not_swallowed():
"usage": {"prompt_tokens": 50, "completion_tokens": 5, "total_tokens": 55},
}
records: list[logging.LogRecord] = []
class _Collect(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
records.append(record)
handler = _Collect(level=logging.DEBUG)
proxy_logger = logging.getLogger("headroom.proxy")
mock_backend = _make_mock_backend(tool_call_response)
with patch("headroom.proxy.server.AnyLLMBackend", return_value=mock_backend):
app = create_app(config)
with TestClient(app) as client:
# Attach after startup: proxy logging setup would otherwise
# detach a handler added earlier (caplog has the same problem).
proxy_logger.addHandler(handler)
_install_tracker_stub(client)
proxy = client.app.state.proxy
failing_handler = MagicMock()
@@ -381,6 +394,7 @@ def test_backend_ccr_intercept_exception_is_reraised_not_swallowed():
},
headers={"Authorization": "Bearer test-key"},
)
proxy_logger.removeHandler(handler)
# The outer `try/except Exception` on the backend block converts the
# re-raise into a 500 response. The critical assertion is that the
@@ -397,7 +411,14 @@ def test_backend_ccr_intercept_exception_is_reraised_not_swallowed():
or body.get("choices", [{}])[0].get("message", {}).get("tool_calls") is None
)
assert "error" in body
assert "ccr-store-blew-up" in body["error"]["message"]
# The client gets the fixed vocabulary and a correlation id, not the
# exception text; the operator log keeps the detail under that id.
assert body["error"]["code"] == "internal_error"
request_id = body["error"]["request_id"]
assert request_id and f"request_id={request_id}" in body["error"]["message"]
assert "ccr-store-blew-up" not in resp.text
logged = [r.getMessage() for r in records]
assert any(f"[{request_id}]" in m and "ccr-store-blew-up" in m for m in logged), logged
def test_backend_streaming_passes_prefix_tracker_through():
+5 -6
View File
@@ -392,12 +392,11 @@ class TestCompressEndpointCompression:
)
assert response.status_code == 503
assert response.json() == {
"error": {
"type": "compression_error",
"message": "compression broke",
}
}
body = response.json()
# Exception text stays in the server log; the client sees the code.
assert body["error"]["type"] == "compression_error"
assert body["error"]["code"] == "internal_error"
assert "compression broke" not in response.text
record_failed.assert_awaited_once_with(provider="compress")
def test_compression_timeout_records_fail_open_outcome(self, client, monkeypatch):
+55 -27
View File
@@ -9,6 +9,7 @@ without passing any upstream headers — silently dropping ratelimit info.
"""
import json
import logging
from unittest.mock import AsyncMock, MagicMock
import httpx
@@ -496,42 +497,69 @@ class TestStreamingRatelimitHeaderForwarding:
mock_request = MagicMock()
proxy.http_client.build_request = MagicMock(return_value=mock_request)
proxy.http_client.send = AsyncMock(side_effect=httpx.ConnectError("Connection refused"))
result = await proxy._stream_response(
url="https://api.anthropic.com/v1/messages",
headers={"x-api-key": "sk-test"},
body={
"model": "claude-sonnet-4-20250514",
"max_tokens": 100,
"stream": True,
"messages": [{"role": "user", "content": "hi"}],
},
provider="anthropic",
model="claude-sonnet-4-20250514",
request_id="test-error",
original_tokens=10,
optimized_tokens=10,
tokens_saved=0,
transforms_applied=[],
tags={},
optimization_latency=0.0,
proxy.http_client.send = AsyncMock(
side_effect=httpx.ConnectError("Connection refused to 10.0.0.7:443")
)
# Should return a StreamingResponse with error SSE event
assert result.media_type == "text/event-stream"
# Attach directly to the proxy logger: caplog does not reliably see
# ``headroom.*`` records once proxy logging is configured.
records: list[logging.LogRecord] = []
# Consume the generator to get the error event
chunks = []
async for chunk in result.body_iterator:
chunks.append(chunk)
class _Collect(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
records.append(record)
handler = _Collect(level=logging.DEBUG)
proxy_logger = logging.getLogger("headroom.proxy")
proxy_logger.addHandler(handler)
try:
result = await proxy._stream_response(
url="https://api.anthropic.com/v1/messages",
headers={"x-api-key": "sk-test"},
body={
"model": "claude-sonnet-4-20250514",
"max_tokens": 100,
"stream": True,
"messages": [{"role": "user", "content": "hi"}],
},
provider="anthropic",
model="claude-sonnet-4-20250514",
request_id="test-error",
original_tokens=10,
optimized_tokens=10,
tokens_saved=0,
transforms_applied=[],
tags={},
optimization_latency=0.0,
)
# Should return a StreamingResponse with error SSE event
assert result.media_type == "text/event-stream"
# Consume the generator to get the error event
chunks = []
async for chunk in result.body_iterator:
chunks.append(chunk)
finally:
proxy_logger.removeHandler(handler)
assert len(chunks) == 1
raw = chunks[0].decode("utf-8")
assert "event: error" in raw
error_data = json.loads(raw.split("data: ")[1].strip())
assert error_data["error"]["type"] == "connection_error"
assert "Connection refused" in error_data["error"]["message"]
# Client sees the fixed vocabulary plus a correlation id, never the
# transport text (which can carry upstream hosts and addresses).
assert error_data["error"]["code"] == "upstream_unreachable"
assert error_data["request_id"] == "test-error"
assert "request_id=test-error" in error_data["error"]["message"]
assert "Connection refused" not in raw
assert "10.0.0.7" not in raw
# The operator still gets the full detail, keyed by the same id.
logged = [r.getMessage() for r in records]
assert any(
"[test-error]" in m and "Connection refused to 10.0.0.7:443" in m for m in logged
), logged
@pytest.mark.asyncio
async def test_connect_timeout_retries_before_returning_stream(self):
+300
View File
@@ -0,0 +1,300 @@
"""Transport and internal exception text must never reach a client (01-F4).
``httpx``/``ssl``/OS exception text names the resolved upstream host, its IP
and the failing library. Every response path maps such failures to the fixed
vocabulary in :mod:`headroom.proxy.public_errors` and keeps the detail in the
server log under the request id.
Companion: the shared proxy token must never land in request tags (10-F4).
"""
from __future__ import annotations
import asyncio
import json
import ssl
from types import SimpleNamespace
from unittest.mock import AsyncMock
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy import public_errors # noqa: E402
from headroom.proxy.handlers.openai import OpenAIHandlerMixin # noqa: E402
from headroom.proxy.helpers import extract_tags # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
# A string that only ever appears in exception text; every assertion below is
# "this never reaches the wire".
SECRET = "gateway-int.corp.example:443 (10.42.0.7)"
# --- Vocabulary --------------------------------------------------------------
@pytest.mark.parametrize(
("exc", "code"),
[
(httpx.ConnectError(SECRET), public_errors.UPSTREAM_UNREACHABLE),
(httpx.ConnectTimeout(SECRET), public_errors.UPSTREAM_UNREACHABLE),
(httpx.PoolTimeout(SECRET), public_errors.UPSTREAM_UNREACHABLE),
(httpx.ReadTimeout(SECRET), public_errors.UPSTREAM_TIMEOUT),
(httpx.WriteTimeout(SECRET), public_errors.UPSTREAM_TIMEOUT),
(httpx.RemoteProtocolError(SECRET), public_errors.UPSTREAM_PROTOCOL_ERROR),
(httpx.ReadError(SECRET), public_errors.UPSTREAM_PROTOCOL_ERROR),
(httpx.DecodingError(SECRET), public_errors.UPSTREAM_PROTOCOL_ERROR),
(ConnectionResetError(104, SECRET), public_errors.UPSTREAM_UNREACHABLE),
(OSError(113, SECRET), public_errors.UPSTREAM_UNREACHABLE),
(asyncio.TimeoutError(), public_errors.UPSTREAM_TIMEOUT),
(
ssl.SSLCertVerificationError(1, "CERTIFICATE_VERIFY_FAILED"),
public_errors.UPSTREAM_TLS_ERROR,
),
],
)
def test_transport_exceptions_classify_to_fixed_codes(exc, code) -> None:
assert public_errors.classify_exception(exc) == code
assert SECRET not in public_errors.public_message(code, request_id="req-1")
def test_tls_failure_wrapped_in_connect_error_is_tls() -> None:
inner = ssl.SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] " + SECRET)
outer = httpx.ConnectError(SECRET)
outer.__cause__ = inner
assert public_errors.classify_exception(outer) == public_errors.UPSTREAM_TLS_ERROR
def test_non_transport_exceptions_are_not_classified() -> None:
"""Provider API errors (litellm/anyllm exceptions) keep their own message."""
assert public_errors.classify_exception(RuntimeError("model not found: gpt-x")) is None
assert public_errors.classify_exception(ValueError("bad")) is None
assert public_errors.classify_or_internal(ValueError("bad")) == public_errors.INTERNAL_ERROR
def test_public_message_rejects_unknown_codes() -> None:
with pytest.raises(ValueError):
public_errors.public_message("stack trace here")
def test_wire_shapes_carry_code_and_request_id() -> None:
openai = public_errors.openai_error_body(
public_errors.UPSTREAM_UNREACHABLE, request_id="req-9", error_type="connection_error"
)
assert openai["error"]["type"] == "connection_error"
assert openai["error"]["code"] == "upstream_unreachable"
assert openai["error"]["request_id"] == "req-9"
assert "req-9" in openai["error"]["message"]
anthropic = public_errors.anthropic_error_body(public_errors.INTERNAL_ERROR, request_id="req-9")
assert anthropic["type"] == "error"
assert anthropic["error"]["type"] == "api_error"
assert anthropic["error"]["code"] == "internal_error"
assert anthropic["request_id"] == "req-9"
# --- Handler paths -------------------------------------------------------------
def _config(**overrides) -> ProxyConfig:
base: dict[str, object] = {
"optimize": False,
"cache_enabled": False,
"rate_limit_enabled": False,
"cost_tracking_enabled": False,
"log_requests": False,
"ccr_inject_tool": False,
"ccr_handle_responses": False,
"ccr_context_tracking": False,
"image_optimize": False,
"retry_enabled": False,
}
base.update(overrides)
return ProxyConfig(**base)
class _SecretConnectTimeoutClient:
async def request(self, **kwargs): # noqa: ANN001, ANN201
raise httpx.ConnectTimeout(f"connect to {SECRET} timed out")
class _PassthroughRequest:
method = "GET"
headers = {}
url = SimpleNamespace(path="/some/other/path", query="")
async def body(self) -> bytes:
return b""
def test_passthrough_connect_error_body_has_no_upstream_detail() -> None:
handler = object.__new__(OpenAIHandlerMixin)
handler.http_client = _SecretConnectTimeoutClient()
response = asyncio.run(
handler.handle_passthrough(_PassthroughRequest(), "https://api.openai.com")
)
assert response.status_code == 502
payload = json.loads(response.body)
assert payload["error"]["type"] == "connection_error"
assert payload["error"]["code"] == public_errors.UPSTREAM_UNREACHABLE
assert SECRET not in response.body.decode()
def test_direct_streaming_connect_error_event_has_no_upstream_detail(monkeypatch) -> None:
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
with TestClient(create_app(_config()), raise_server_exceptions=False) as client:
proxy = client.app.state.proxy
proxy.http_client.send = AsyncMock(side_effect=httpx.ConnectError(f"refused by {SECRET}"))
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-haiku-4-5",
"max_tokens": 16,
"stream": True,
"messages": [{"role": "user", "content": "hi"}],
},
)
assert response.status_code == 502
text = response.text
assert SECRET not in text
assert "event: error" in text
assert public_errors.UPSTREAM_UNREACHABLE in text
def test_compress_endpoint_error_body_has_no_exception_detail(monkeypatch) -> None:
app = create_app(_config(optimize=True))
with TestClient(
app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345), raise_server_exceptions=False
) as client:
proxy = client.app.state.proxy
monkeypatch.setattr(
proxy,
"_run_compression_in_executor",
AsyncMock(side_effect=RuntimeError(f"compressor exploded reading {SECRET}")),
)
response = client.post(
"/v1/compress",
json={"messages": [{"role": "user", "content": "compress me"}], "model": "gpt-4"},
)
assert response.status_code == 503
body = response.json()
assert body["error"]["type"] == "compression_error"
assert body["error"]["code"] == public_errors.INTERNAL_ERROR
assert SECRET not in response.text
@pytest.mark.asyncio
async def test_memory_tool_result_hides_transport_and_internal_text(tmp_path) -> None:
from headroom.proxy.memory_handler import MemoryConfig, MemoryHandler
handler = MemoryHandler(
MemoryConfig(enabled=False, use_native_tool=True, native_memory_dir=str(tmp_path / "n")),
agent_type="codex",
)
class _Backend:
exc: BaseException = OSError(5, f"sqlite at {SECRET} is locked")
async def save_memory(self, **kwargs): # noqa: ANN003
raise self.exc
backend = _Backend()
handler._backend = backend
transport = json.loads(
await handler._execute_memory_tool("memory_save", {"content": "x"}, "u1")
)
assert transport["status"] == "error"
assert SECRET not in json.dumps(transport)
assert transport["error"] == public_errors.UPSTREAM_UNREACHABLE
backend.exc = RuntimeError(f"save failed at {LEAKY}")
internal = json.loads(await handler._execute_memory_tool("memory_save", {"content": "x"}, "u1"))
assert internal["error"] == public_errors.INTERNAL_ERROR
assert LEAKY not in json.dumps(internal)
# --- Unclassified exceptions fail closed ---------------------------------------
# Exception text that names a local path, an internal host and a token.
LEAKY = r"sqlite at C:\service\tenant-a.db via db-int.corp.example failed with sk-live-secret"
UNCLASSIFIED = [
RuntimeError(LEAKY),
ValueError(LEAKY),
KeyError(LEAKY),
json.JSONDecodeError(LEAKY, "{}", 0),
]
@pytest.mark.parametrize("exc", UNCLASSIFIED, ids=lambda e: type(e).__name__)
def test_client_message_hides_unclassified_exception_text(exc) -> None:
message = public_errors.client_message(exc, LEAKY)
assert message == public_errors.public_message(public_errors.INTERNAL_ERROR)
assert "sk-live-secret" not in message
@pytest.mark.parametrize("exc", UNCLASSIFIED, ids=lambda e: type(e).__name__)
def test_tool_result_error_hides_unclassified_exception_text(exc) -> None:
result = public_errors.tool_result_error(exc)
assert result["error"] == public_errors.INTERNAL_ERROR
assert "sk-live-secret" not in json.dumps(result)
def test_client_message_forwards_provider_http_error_response() -> None:
import openai
response = httpx.Response(401, request=httpx.Request("POST", "https://api.example/v1"))
exc = openai.AuthenticationError("invalid x-api-key", response=response, body=None)
assert public_errors.client_message(exc, "invalid x-api-key") == "invalid x-api-key"
@pytest.mark.asyncio
async def test_litellm_backend_hides_unclassified_exception_text(monkeypatch) -> None:
from headroom.backends import litellm as litellm_backend
monkeypatch.setattr(litellm_backend, "acompletion", AsyncMock(side_effect=RuntimeError(LEAKY)))
backend = litellm_backend.LiteLLMBackend(provider="openai")
result = await backend.send_openai_message(
{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}]}, {}
)
assert "sk-live-secret" not in json.dumps(result.body)
assert result.body["error"]["message"] == public_errors.public_message(
public_errors.INTERNAL_ERROR
)
# --- Tags (10-F4) -------------------------------------------------------------
def test_extract_tags_never_carries_credentials() -> None:
tags = extract_tags(
{
"X-Headroom-Proxy-Token": "hlk_secret_token_value",
"x-headroom-project": "alpha",
"x-headroom-session-id": "s-1",
"x-headroom-license-token": "lic",
"x-headroom-api-key": "k",
"X-Headroom-Key": "hr_cloud_key",
"authorization": "Bearer sk-live",
}
)
assert tags == {"project": "alpha", "session-id": "s-1"}
def test_public_tags_drop_credential_like_keys_defensively() -> None:
from headroom.proxy.savings_attribution import public_tags
assert public_tags({"proxy-token": "x", "project": "alpha", "some_secret": "y"}) == {
"project": "alpha"
}