mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): keep exception text and the proxy token out of client output (#3851)
## Why
Two leaks of internal detail to clients and logs (findings 01-F4 and
10-F4).
- When an upstream connection failed, several response paths returned
`str(e)` from `httpx`, `ssl` or the OS in 500/502/503 bodies and SSE
`error` events. That text names the resolved upstream host, its address
and the failing library. A client that can steer the upstream
(`x-headroom-base-url`) could learn internal hostnames one error at a
time.
- `extract_tags` copied every `x-headroom-*` header into
`RequestOutcome.tags`, including `x-headroom-proxy-token`, which
gateways send on every turn. The token then appeared in the request log
(`--log-file`), dashboard facets and any export that carries tags.
## What changes
- New module `headroom/proxy/public_errors.py`. `classify_exception()`
maps transport and OS failures to fixed codes: `upstream_unreachable`,
`upstream_timeout`, `upstream_tls_error`, `upstream_protocol_error`;
anything else is `internal_error`. Every client- or model-visible path
fails closed: unrecognised exception text (`RuntimeError`, `ValueError`,
parser, database or library errors) never leaves the proxy. Provider
error text is forwarded only when the exception carries a provider HTTP
error response (`openai.APIStatusError`, which LiteLLM's provider errors
subclass, or `anthropic.APIStatusError`). Helpers build the existing
wire shapes with the same `error.type`, plus `code` and `request_id`.
The full exception stays in the server log under that request id.
- Every site that built a client-visible message from an exception now
uses it: `handlers/streaming.py` (direct, Bedrock and OpenAI-via-backend
SSE), `handlers/openai.py` (backend 500, `/v1/compress` 503, both
passthrough connect paths), `handlers/anthropic.py` (Bedrock backend
500), `handlers/bedrock.py` (connect 502), the memory tool handlers
(tool results returned to the model), and the `litellm` / `anyllm`
backends. The existing corporate-TLS hint is kept.
- `internal_header_policy.py` gains `is_credential_header()` and
`is_credential_tag_key()`. They match whole name components (`token`,
`secret`, `password`, `authorization`, `cookie`, `apikey`, `key`, and
similar), so `X-Headroom-Key` and `X-Headroom-Api-Key` are covered, so
numeric tags such as `tool-search-deferred-tokens` are not caught.
- `extract_tags` skips credential headers.
`savings_attribution.public_tags`, the last step before the request log,
also drops credential-shaped keys.
## Behaviour change
- Transport-layer and internal failures now return a fixed sentence plus
`code` and `request_id`. Clients that parsed the old free-text messages
should read `error.code`. Provider HTTP error responses (and their
status codes) are unchanged. In the LiteLLM / any-llm backends, an
exception that is not a provider response now returns `The proxy could
not complete the request.`; the HTTP status mapping is unchanged.
- Memory tool results for any failure return `{"status": "error",
"error": "<code>", "message": ...}` instead of the exception text.
- `x-headroom-*` headers whose name contains a credential component are
no longer recorded as tags. A benign label that happens to use such a
name should be renamed.
## Tests
- `tests/test_public_errors.py`: classification of transport exceptions
(including TLS wrapped in a connect error), non-transport exceptions
left alone, wire shapes carry `code` and `request_id`; the passthrough,
direct-streaming and `/v1/compress` error paths contain no upstream
detail; memory tool results hide transport and internal text;
`client_message` / `tool_result_error` / the LiteLLM backend return
`internal_error` for `RuntimeError`, `ValueError`, `KeyError` and
`JSONDecodeError` carrying a path, host and token (10 tests in this file
fail without the fail-closed change), while an
`openai.AuthenticationError` keeps its text; `extract_tags` and
`public_tags` never carry credentials. Six of these fail on `main`.
- Existing tests that asserted the old exception text in client output
now assert the new contract: the client gets the fixed `code` and
`request_id`, the exception text is absent, and the `headroom.proxy` log
has the full text under the same `request_id`
(`test_litellm_transport_errors.py`, `test_openai_backend_path.py`,
`test_proxy_compress_endpoint.py`,
`test_proxy_streaming_ratelimit_headers.py`).
- Manual run with every upstream connect failing: streaming
`/v1/messages`, streaming `/v1/chat/completions` and passthrough
`/v1/models` return `code: upstream_unreachable` with no address. A
request with `x-headroom-proxy-token` and `x-headroom-license-token`
leaves neither value in the `--log-file` output.
Not tested live: the Bedrock and LiteLLM / any-llm backend paths (unit
tests only) and OTEL/Langfuse export (tags are filtered before they
reach it).
## Open review points
- Resolved: unclassified exception text (JerrettDavis, 2026-09-29) now
fails closed in `client_message` and `tool_result_error`.
- Resolved: `X-Headroom-Key` is matched (`key` is a credential
component; the `api`+`key` pair rule is gone). No internal or test tag
key has a `key` component.
- `is_credential_header` special-cases `authorization`, `cookie`,
`api-key` and `x-api-key`, but its only caller passes only
`x-headroom-*` names, so those branches never run.
- `_is_tls_failure` also matches on message text. Provider HTTP error
responses are now checked first in the backends, so this only affects
non-provider exceptions.
- any-llm: only provider errors raised as `openai`/`anthropic`
`APIStatusError` keep their text; any other any-llm exception type
returns `internal_error`.
Revised 2026-10-01 after review: unrecognised exceptions now return a
fixed `internal_error` message and only provider HTTP error responses
keep their text.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
3c9ed0101a
commit
861e94d8c1
@@ -12,6 +12,7 @@ import uuid
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any, cast
|
||||
|
||||
from headroom.proxy.public_errors import client_message
|
||||
from headroom.utils import format_exception_message
|
||||
|
||||
from .base import Backend, BackendResponse, StreamEvent
|
||||
@@ -505,7 +506,7 @@ class AnyLLMBackend(Backend):
|
||||
event_type="error",
|
||||
data={
|
||||
"type": "error",
|
||||
"error": {"type": "api_error", "message": error_message},
|
||||
"error": {"type": "api_error", "message": client_message(e, error_message)},
|
||||
},
|
||||
)
|
||||
|
||||
@@ -601,7 +602,9 @@ class AnyLLMBackend(Backend):
|
||||
error_type = "api_error"
|
||||
status_code = 500
|
||||
|
||||
error_message = format_exception_message(e)
|
||||
# Provider API errors keep their text; transport failures are reduced
|
||||
# to the public vocabulary (see proxy/public_errors).
|
||||
error_message = client_message(e, format_exception_message(e))
|
||||
error_str = str(e).lower()
|
||||
if "authentication" in error_str or "api_key" in error_str or "api key" in error_str:
|
||||
error_type = "invalid_api_key" if openai_format else "authentication_error"
|
||||
@@ -669,7 +672,7 @@ class AnyLLMBackend(Backend):
|
||||
logger.error(f"any-llm OpenAI streaming error: {error_message}")
|
||||
error_data = {
|
||||
"error": {
|
||||
"message": error_message,
|
||||
"message": client_message(e, error_message),
|
||||
"type": "api_error",
|
||||
"code": "backend_error",
|
||||
}
|
||||
|
||||
@@ -21,6 +21,7 @@ from collections.abc import AsyncIterator
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from headroom.proxy.public_errors import client_message
|
||||
from headroom.utils import format_exception_message
|
||||
|
||||
from .base import Backend, BackendResponse, StreamEvent
|
||||
@@ -1274,6 +1275,9 @@ class LiteLLMBackend(Backend):
|
||||
except Exception as e:
|
||||
error_message = format_exception_message(e)
|
||||
logger.error(f"LiteLLM error: {error_message}")
|
||||
# Provider API errors keep their text; transport failures are
|
||||
# reduced to the public vocabulary (see proxy/public_errors).
|
||||
error_message = client_message(e, error_message)
|
||||
|
||||
# Map to Anthropic error format
|
||||
error_type = "api_error"
|
||||
@@ -1689,7 +1693,7 @@ class LiteLLMBackend(Backend):
|
||||
event_type="error",
|
||||
data={
|
||||
"type": "error",
|
||||
"error": {"type": "api_error", "message": error_message},
|
||||
"error": {"type": "api_error", "message": client_message(e, error_message)},
|
||||
},
|
||||
)
|
||||
|
||||
@@ -1886,6 +1890,7 @@ class LiteLLMBackend(Backend):
|
||||
except Exception as e:
|
||||
error_message = format_exception_message(e)
|
||||
logger.error(f"LiteLLM OpenAI error: {error_message}")
|
||||
error_message = client_message(e, error_message)
|
||||
|
||||
# Map to OpenAI error format
|
||||
error_type = "api_error"
|
||||
@@ -1997,7 +2002,7 @@ class LiteLLMBackend(Backend):
|
||||
logger.error(f"LiteLLM OpenAI streaming error: {error_message}")
|
||||
error_data = {
|
||||
"error": {
|
||||
"message": error_message,
|
||||
"message": client_message(e, error_message),
|
||||
"type": "api_error",
|
||||
"code": "backend_error",
|
||||
}
|
||||
|
||||
@@ -33,6 +33,7 @@ from headroom.ccr.context_tracker import looks_like_claude_code_compact_summary
|
||||
from headroom.ccr.marker_resolution import resolve_markers_in_response
|
||||
from headroom.copilot_auth import apply_copilot_api_auth, is_copilot_upstream_url
|
||||
from headroom.pipeline import PipelineStage, summarize_routing_markers
|
||||
from headroom.proxy import public_errors
|
||||
from headroom.proxy.anthropic_wire import (
|
||||
build_anthropic_upstream_url,
|
||||
is_safeguard_capable_request,
|
||||
@@ -3782,10 +3783,11 @@ class AnthropicHandlerMixin:
|
||||
await _finalize_pre_upstream()
|
||||
return JSONResponse(
|
||||
status_code=500,
|
||||
content={
|
||||
"type": "error",
|
||||
"error": {"type": "api_error", "message": error_message},
|
||||
},
|
||||
content=public_errors.anthropic_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
request_id=str(request_id),
|
||||
error_type="api_error",
|
||||
),
|
||||
)
|
||||
|
||||
# Direct Anthropic API, or a provider-compatible Anthropic
|
||||
|
||||
@@ -301,14 +301,15 @@ class BedrockHandlerMixin:
|
||||
upstream = await self.http_client.send(upstream_request, stream=True) # type: ignore[attr-defined]
|
||||
except (httpx.ConnectError, httpx.TimeoutException) as err:
|
||||
logger.warning("[%s] %s upstream connect failed: %s", request_id, LOG_TAG, err)
|
||||
from headroom.proxy import public_errors
|
||||
|
||||
return JSONResponse(
|
||||
status_code=502,
|
||||
content={
|
||||
"error": {
|
||||
"type": "connection_error",
|
||||
"message": f"Failed to connect to Bedrock upstream: {err}",
|
||||
}
|
||||
},
|
||||
content=public_errors.openai_error_body(
|
||||
public_errors.classify_or_internal(err),
|
||||
request_id=str(request_id),
|
||||
error_type="connection_error",
|
||||
),
|
||||
)
|
||||
|
||||
# Forward raw (still-encoded) bytes, so strip hop-by-hop headers that
|
||||
|
||||
@@ -77,6 +77,7 @@ from headroom.providers.codex.runtime import (
|
||||
from headroom.providers.copilot import model_prefers_responses_api
|
||||
from headroom.providers.grok.runtime import DEFAULT_API_URL as XAI_API_URL
|
||||
from headroom.providers.proxy_targets import route_grok_to_xai
|
||||
from headroom.proxy import public_errors
|
||||
from headroom.proxy.auth_mode import (
|
||||
classify_auth_mode,
|
||||
classify_client,
|
||||
@@ -5131,13 +5132,11 @@ class OpenAIHandlerMixin:
|
||||
logger.error(f"[{request_id}] Backend error: {e}")
|
||||
return JSONResponse(
|
||||
status_code=500,
|
||||
content={
|
||||
"error": {
|
||||
"message": str(e),
|
||||
"type": "api_error",
|
||||
"code": "backend_error",
|
||||
}
|
||||
},
|
||||
content=public_errors.openai_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
request_id=str(request_id),
|
||||
error_type="api_error",
|
||||
),
|
||||
)
|
||||
|
||||
# Direct OpenAI API (no backend configured). Reuse the upstream resolved
|
||||
@@ -10875,12 +10874,10 @@ class OpenAIHandlerMixin:
|
||||
await self.metrics.record_failed(provider="compress")
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
"error": {
|
||||
"type": "compression_error",
|
||||
"message": str(e),
|
||||
}
|
||||
},
|
||||
content=public_errors.openai_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
error_type="compression_error",
|
||||
),
|
||||
)
|
||||
|
||||
async def handle_compress_usage(self, request: Request) -> JSONResponse:
|
||||
@@ -11267,12 +11264,11 @@ class OpenAIHandlerMixin:
|
||||
tls_hint = await describe_upstream_failure_async(e, url)
|
||||
return Response(
|
||||
content=json.dumps(
|
||||
{
|
||||
"error": {
|
||||
"type": "connection_error",
|
||||
"message": tls_hint or f"Failed to connect to upstream API: {e}",
|
||||
}
|
||||
}
|
||||
public_errors.openai_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
error_type="connection_error",
|
||||
hint=tls_hint,
|
||||
)
|
||||
),
|
||||
status_code=502,
|
||||
media_type="application/json",
|
||||
@@ -11458,12 +11454,11 @@ class OpenAIHandlerMixin:
|
||||
tls_hint = await describe_upstream_failure_async(e, url)
|
||||
return Response(
|
||||
content=json.dumps(
|
||||
{
|
||||
"error": {
|
||||
"type": "connection_error",
|
||||
"message": tls_hint or f"Failed to connect to upstream API: {e}",
|
||||
}
|
||||
}
|
||||
public_errors.openai_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
error_type="connection_error",
|
||||
hint=tls_hint,
|
||||
)
|
||||
),
|
||||
status_code=502,
|
||||
media_type="application/json",
|
||||
|
||||
@@ -28,6 +28,7 @@ if TYPE_CHECKING:
|
||||
import httpx
|
||||
|
||||
from headroom.copilot_auth import apply_copilot_api_auth
|
||||
from headroom.proxy import public_errors
|
||||
from headroom.proxy.stream_output_tokens import estimate_output_tokens
|
||||
from headroom.proxy.thinking_tokens import ThinkingTokens, extract_thinking_tokens
|
||||
from headroom.utils import format_exception_message
|
||||
@@ -1063,17 +1064,17 @@ class StreamingMixin:
|
||||
from headroom.proxy.tls_diagnostics import describe_upstream_failure_async
|
||||
|
||||
tls_hint = await describe_upstream_failure_async(e, url)
|
||||
client_message = tls_hint or f"Failed to connect to upstream API: {error_msg}"
|
||||
# The exception text itself stays in the log above: it names the
|
||||
# resolved upstream host and address (public_errors module doc).
|
||||
error_body = public_errors.anthropic_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
request_id=request_id,
|
||||
error_type="connection_error",
|
||||
hint=tls_hint,
|
||||
)
|
||||
|
||||
async def _error_gen():
|
||||
error_event = {
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "connection_error",
|
||||
"message": client_message,
|
||||
},
|
||||
}
|
||||
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
|
||||
yield f"event: error\ndata: {json.dumps(error_body)}\n\n".encode()
|
||||
|
||||
self._cleanup_mid_turn_stream(session_key)
|
||||
return StreamingResponse(
|
||||
@@ -1428,13 +1429,11 @@ class StreamingMixin:
|
||||
|
||||
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.PoolTimeout) as e:
|
||||
logger.error(f"[{request_id}] Connection error to upstream API: {e}")
|
||||
error_event = {
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "connection_error",
|
||||
"message": f"Failed to connect to upstream API: {e}",
|
||||
},
|
||||
}
|
||||
error_event = public_errors.anthropic_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
request_id=request_id,
|
||||
error_type="connection_error",
|
||||
)
|
||||
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
|
||||
except httpx.HTTPStatusError as e:
|
||||
logger.error(f"[{request_id}] HTTP error from upstream API: {e}")
|
||||
@@ -1442,10 +1441,11 @@ class StreamingMixin:
|
||||
yield e.response.content
|
||||
except Exception as e:
|
||||
logger.error(f"[{request_id}] Unexpected streaming error: {e}")
|
||||
error_event = {
|
||||
"type": "error",
|
||||
"error": {"type": "api_error", "message": str(e)},
|
||||
}
|
||||
error_event = public_errors.anthropic_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
request_id=request_id,
|
||||
error_type="api_error",
|
||||
)
|
||||
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
|
||||
finally:
|
||||
pending_messages = self._cleanup_mid_turn_stream(
|
||||
@@ -1673,10 +1673,11 @@ class StreamingMixin:
|
||||
except Exception as e:
|
||||
error_message = format_exception_message(e)
|
||||
logger.error(f"[{request_id}] Bedrock streaming error: {error_message}")
|
||||
error_event = {
|
||||
"type": "error",
|
||||
"error": {"type": "api_error", "message": error_message},
|
||||
}
|
||||
error_event = public_errors.anthropic_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
request_id=request_id,
|
||||
error_type="api_error",
|
||||
)
|
||||
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
|
||||
|
||||
finally:
|
||||
@@ -1868,13 +1869,11 @@ class StreamingMixin:
|
||||
yield chunk_bytes
|
||||
except Exception as e:
|
||||
logger.error(f"[{request_id}] Backend streaming error: {e}")
|
||||
error_data = {
|
||||
"error": {
|
||||
"message": str(e),
|
||||
"type": "api_error",
|
||||
"code": "backend_error",
|
||||
}
|
||||
}
|
||||
error_data = public_errors.openai_error_body(
|
||||
public_errors.classify_or_internal(e),
|
||||
request_id=request_id,
|
||||
error_type="api_error",
|
||||
)
|
||||
yield f"data: {json.dumps(error_data)}\n\n".encode()
|
||||
yield b"data: [DONE]\n\n"
|
||||
finally:
|
||||
|
||||
@@ -301,11 +301,20 @@ def extract_tags(headers: Any) -> dict[str, str]:
|
||||
|
||||
Header name match is case-insensitive; the returned key has the
|
||||
``x-headroom-`` prefix stripped.
|
||||
|
||||
Credential-bearing headers are never tags. ``x-headroom-proxy-token`` is
|
||||
the proxy's own bearer and gateways (Kong, Envoy, LiteLLM) send it on every
|
||||
turn; before this filter it landed in ``RequestOutcome.tags`` and from
|
||||
there in the request log, dashboard facets and every export that carries
|
||||
tags. The rule lives in :mod:`internal_header_policy` so producers and
|
||||
consumers of tags agree on it.
|
||||
"""
|
||||
from headroom.proxy.internal_header_policy import is_credential_header
|
||||
|
||||
return {
|
||||
k.lower().replace("x-headroom-", ""): v
|
||||
for k, v in headers.items()
|
||||
if k.lower().startswith("x-headroom-")
|
||||
if k.lower().startswith("x-headroom-") and not is_credential_header(k)
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -10,6 +10,64 @@ STRIP_INTERNAL_HEADERS_ENV = "HEADROOM_STRIP_INTERNAL_HEADERS"
|
||||
StripInternalHeadersMode = Literal["enabled", "disabled"]
|
||||
STRIP_INTERNAL_HEADERS_DEFAULT: StripInternalHeadersMode = "enabled"
|
||||
|
||||
# The proxy's own credential header. Read by the security gate and the
|
||||
# WebSocket auth middleware; it must never be copied into tags, logs, metrics
|
||||
# labels or exports, which is what ``is_credential_header`` guards.
|
||||
PROXY_TOKEN_HEADER = "x-headroom-proxy-token"
|
||||
|
||||
# Name components that mark an ``x-headroom-*`` header (or the tag key derived
|
||||
# from it by stripping the prefix) as credential material rather than a label.
|
||||
# Names are lower-cased, ``_`` is folded to ``-`` and the name is split on
|
||||
# ``-``, so ``X-Headroom-License_Token`` -> ``license-token`` -> {license, token}.
|
||||
# Whole components only: ``proxy-token`` is a credential, the numeric
|
||||
# ``tool-search-deferred-tokens`` tag is not.
|
||||
CREDENTIAL_NAME_COMPONENTS: frozenset[str] = frozenset(
|
||||
{
|
||||
"token",
|
||||
"secret",
|
||||
"password",
|
||||
"passwd",
|
||||
"credential",
|
||||
"credentials",
|
||||
"authorization",
|
||||
"apikey",
|
||||
"key", # x-headroom-key (Headroom Cloud), x-headroom-api-key
|
||||
"bearer",
|
||||
"cookie",
|
||||
"signature",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _name_components(name: str) -> list[str]:
|
||||
return [part for part in name.strip().lower().replace("_", "-").split("-") if part]
|
||||
|
||||
|
||||
def is_credential_tag_key(key: str) -> bool:
|
||||
"""True if a tag key (header name minus the ``x-headroom-`` prefix) is a credential.
|
||||
|
||||
Tag keys become request-log fields, dashboard facets and export labels,
|
||||
so this is the single rule every tag producer and consumer applies.
|
||||
"""
|
||||
parts = _name_components(key)
|
||||
if parts[:2] == ["x", "headroom"]:
|
||||
parts = parts[2:]
|
||||
return any(part in CREDENTIAL_NAME_COMPONENTS for part in parts)
|
||||
|
||||
|
||||
def _normalise_name(name: str) -> str:
|
||||
return "-".join(_name_components(name))
|
||||
|
||||
|
||||
def is_credential_header(name: str) -> bool:
|
||||
"""True if an inbound header carries a credential and must not be tagged."""
|
||||
normalised = _normalise_name(name)
|
||||
if normalised == PROXY_TOKEN_HEADER:
|
||||
return True
|
||||
if normalised in ("authorization", "proxy-authorization", "cookie", "api-key", "x-api-key"):
|
||||
return True
|
||||
return normalised.startswith(INTERNAL_HEADER_PREFIX) and is_credential_tag_key(normalised)
|
||||
|
||||
|
||||
def resolve_strip_internal_headers_mode(raw: str | None) -> StripInternalHeadersMode:
|
||||
"""Resolve the configured internal-header strip mode."""
|
||||
|
||||
@@ -43,6 +43,7 @@ from headroom.memory.storage_router import (
|
||||
RequestContext,
|
||||
ResolvedScope,
|
||||
)
|
||||
from headroom.proxy.public_errors import tool_result_error
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from headroom.memory.backends.local import LocalBackend
|
||||
@@ -1232,7 +1233,7 @@ your responses, not to drive new actions."""
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Memory: Tool {tool_name} failed: {e}")
|
||||
return json.dumps({"status": "error", "error": str(e)})
|
||||
return json.dumps(tool_result_error(e))
|
||||
|
||||
async def _execute_save(
|
||||
self,
|
||||
@@ -1521,7 +1522,7 @@ your responses, not to drive new actions."""
|
||||
results = await list_fn(user_id=effective_user_id, limit=limit)
|
||||
except Exception as e:
|
||||
logger.warning(f"Memory: list_memories failed for user {effective_user_id}: {e}")
|
||||
return json.dumps({"status": "error", "error": str(e)})
|
||||
return json.dumps(tool_result_error(e))
|
||||
else:
|
||||
try:
|
||||
results = await backend.search_memories(
|
||||
@@ -1531,7 +1532,7 @@ your responses, not to drive new actions."""
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Memory: list fallback search failed: {e}")
|
||||
return json.dumps({"status": "error", "error": str(e)})
|
||||
return json.dumps(tool_result_error(e))
|
||||
|
||||
entries: list[dict[str, Any]] = []
|
||||
for r in results:
|
||||
|
||||
@@ -32,6 +32,8 @@ import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
from headroom.proxy.public_errors import tool_result_error
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from headroom.memory.backends.local import LocalBackend
|
||||
|
||||
@@ -1145,7 +1147,7 @@ To SAVE: create /memories/<topic>.txt "content"
|
||||
return json.dumps({"error": f"Unknown tool: {tool_name}"})
|
||||
except Exception as e:
|
||||
logger.error(f"MemoryToolAdapter: Tool {tool_name} failed: {e}")
|
||||
return json.dumps({"status": "error", "error": str(e)})
|
||||
return json.dumps(tool_result_error(e))
|
||||
|
||||
async def _execute_save(self, input_data: dict[str, Any], user_id: str) -> str:
|
||||
"""Execute memory_save tool."""
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
"""Fixed-vocabulary client-facing errors for transport and internal failures.
|
||||
|
||||
Why this exists
|
||||
---------------
|
||||
When an upstream connection fails, the exception text produced by ``httpx``,
|
||||
``ssl`` or the OS names the resolved upstream host, its IP address, the
|
||||
library that failed and sometimes the local interface. Returning that text to
|
||||
the caller (``"message": str(e)``) turns every transport failure into an
|
||||
information-disclosure finding: a client that can steer the upstream (for
|
||||
example through ``x-headroom-base-url``) learns internal hostnames and
|
||||
addresses one error at a time.
|
||||
|
||||
The rule enforced here is simple and applies to every response path:
|
||||
|
||||
* **Transport-layer and internal exception text never reaches a client.** It is
|
||||
logged server-side with the request id, and the client receives one of the
|
||||
fixed codes below plus that request id so the two can be correlated.
|
||||
* **Only a provider's own HTTP error response is forwarded.** A model
|
||||
provider's ``{"error": {"message": ...}}`` (authentication failed, model not
|
||||
found, rate limited) is addressed to the caller. It is recognised by
|
||||
exception type (:func:`is_provider_response_error`), never by guessing from
|
||||
text; every other exception (``RuntimeError``, ``ValueError``, parser,
|
||||
database or library errors) fails closed to :data:`INTERNAL_ERROR`.
|
||||
|
||||
The module is pure policy: no I/O, no logging, no FastAPI types. Handlers and
|
||||
backends build their wire shape (OpenAI ``{"error": {...}}`` or Anthropic
|
||||
``{"type": "error", "error": {...}}``) with the helpers at the bottom so the
|
||||
vocabulary cannot drift between transports.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import ssl
|
||||
from typing import Any, Final
|
||||
|
||||
try: # httpx is a proxy-extra dependency; the SDK imports this module without it.
|
||||
import httpx
|
||||
except ImportError: # pragma: no cover - exercised only in SDK-only installs
|
||||
httpx = None # type: ignore[assignment]
|
||||
|
||||
# --- The vocabulary --------------------------------------------------------
|
||||
|
||||
UPSTREAM_UNREACHABLE: Final = "upstream_unreachable"
|
||||
UPSTREAM_TIMEOUT: Final = "upstream_timeout"
|
||||
UPSTREAM_TLS_ERROR: Final = "upstream_tls_error"
|
||||
UPSTREAM_PROTOCOL_ERROR: Final = "upstream_protocol_error"
|
||||
INTERNAL_ERROR: Final = "internal_error"
|
||||
|
||||
PUBLIC_ERROR_CODES: Final[frozenset[str]] = frozenset(
|
||||
{
|
||||
UPSTREAM_UNREACHABLE,
|
||||
UPSTREAM_TIMEOUT,
|
||||
UPSTREAM_TLS_ERROR,
|
||||
UPSTREAM_PROTOCOL_ERROR,
|
||||
INTERNAL_ERROR,
|
||||
}
|
||||
)
|
||||
|
||||
# One sentence per code. These are the only strings a client sees for a
|
||||
# transport or internal failure; they carry no host, address, path or
|
||||
# library detail by construction.
|
||||
_MESSAGES: Final[dict[str, str]] = {
|
||||
UPSTREAM_UNREACHABLE: "Failed to connect to upstream API.",
|
||||
UPSTREAM_TIMEOUT: "Upstream API did not respond in time.",
|
||||
UPSTREAM_TLS_ERROR: "Could not establish a trusted TLS connection to the upstream API.",
|
||||
UPSTREAM_PROTOCOL_ERROR: ("Upstream API closed the connection or sent a malformed response."),
|
||||
INTERNAL_ERROR: "The proxy could not complete the request.",
|
||||
}
|
||||
|
||||
# Anthropic's error envelope uses ``error.type``; keep the values it already
|
||||
# emits today so SDKs keep classifying them the same way.
|
||||
_ANTHROPIC_TYPES: Final[dict[str, str]] = {
|
||||
UPSTREAM_UNREACHABLE: "connection_error",
|
||||
UPSTREAM_TIMEOUT: "connection_error",
|
||||
UPSTREAM_TLS_ERROR: "connection_error",
|
||||
UPSTREAM_PROTOCOL_ERROR: "api_error",
|
||||
INTERNAL_ERROR: "api_error",
|
||||
}
|
||||
|
||||
|
||||
# --- Classification --------------------------------------------------------
|
||||
|
||||
|
||||
def _iter_causes(exc: BaseException) -> list[BaseException]:
|
||||
"""The exception and its ``__cause__``/``__context__`` chain, outermost first."""
|
||||
seen: list[BaseException] = []
|
||||
current: BaseException | None = exc
|
||||
while current is not None and current not in seen and len(seen) < 16:
|
||||
seen.append(current)
|
||||
current = current.__cause__ or current.__context__
|
||||
return seen
|
||||
|
||||
|
||||
def _is_tls_failure(exc: BaseException) -> bool:
|
||||
for item in _iter_causes(exc):
|
||||
if isinstance(item, ssl.SSLError):
|
||||
return True
|
||||
text = str(item).upper()
|
||||
if "CERTIFICATE_VERIFY_FAILED" in text or "SSL:" in text or "TLS" in text.split():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def classify_exception(exc: BaseException) -> str | None:
|
||||
"""Return the public code for a transport/runtime failure, or ``None``.
|
||||
|
||||
``None`` means "this is not a transport-layer failure". Client-facing code
|
||||
uses :func:`classify_or_internal` so that anything unrecognised becomes
|
||||
:data:`INTERNAL_ERROR`: it was never meant for a client.
|
||||
"""
|
||||
if _is_tls_failure(exc):
|
||||
return UPSTREAM_TLS_ERROR
|
||||
|
||||
if httpx is not None:
|
||||
if isinstance(exc, (httpx.ConnectTimeout, httpx.PoolTimeout)):
|
||||
return UPSTREAM_UNREACHABLE
|
||||
if isinstance(exc, httpx.TimeoutException):
|
||||
return UPSTREAM_TIMEOUT
|
||||
if isinstance(exc, httpx.ConnectError):
|
||||
return UPSTREAM_UNREACHABLE
|
||||
if isinstance(exc, httpx.ProtocolError):
|
||||
return UPSTREAM_PROTOCOL_ERROR
|
||||
if isinstance(exc, httpx.DecodingError):
|
||||
return UPSTREAM_PROTOCOL_ERROR
|
||||
if isinstance(exc, httpx.NetworkError):
|
||||
# ReadError / WriteError / CloseError: the connection existed and
|
||||
# then broke mid-exchange.
|
||||
return UPSTREAM_PROTOCOL_ERROR
|
||||
if isinstance(exc, httpx.TransportError):
|
||||
return UPSTREAM_UNREACHABLE
|
||||
|
||||
if isinstance(exc, (asyncio.TimeoutError, TimeoutError)):
|
||||
return UPSTREAM_TIMEOUT
|
||||
if isinstance(exc, ConnectionError):
|
||||
# ConnectionResetError / BrokenPipeError / ConnectionRefusedError.
|
||||
return UPSTREAM_UNREACHABLE
|
||||
if isinstance(exc, OSError):
|
||||
# Anything else from the socket layer (EHOSTUNREACH, ENETDOWN, ...).
|
||||
return UPSTREAM_UNREACHABLE
|
||||
return None
|
||||
|
||||
|
||||
def classify_or_internal(exc: BaseException) -> str:
|
||||
"""Classify, treating anything unrecognised as :data:`INTERNAL_ERROR`."""
|
||||
return classify_exception(exc) or INTERNAL_ERROR
|
||||
|
||||
|
||||
# --- Wire shapes -------------------------------------------------------------
|
||||
|
||||
|
||||
def public_message(code: str, *, request_id: str | None = None, hint: str | None = None) -> str:
|
||||
"""The client-facing sentence for ``code``.
|
||||
|
||||
``hint`` is an *already public* operator-facing string (for example the
|
||||
corporate TLS-inspection explanation from ``tls_diagnostics``) and is
|
||||
appended verbatim; callers must not pass exception text through it.
|
||||
"""
|
||||
if code not in PUBLIC_ERROR_CODES:
|
||||
raise ValueError(f"not a public error code: {code!r}")
|
||||
text = _MESSAGES[code]
|
||||
if hint:
|
||||
text = f"{text} {hint.strip()}"
|
||||
if request_id:
|
||||
text = f"{text} (request_id={request_id})"
|
||||
return text
|
||||
|
||||
|
||||
def openai_error_body(
|
||||
code: str,
|
||||
*,
|
||||
request_id: str | None = None,
|
||||
error_type: str | None = None,
|
||||
hint: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""``{"error": {...}}`` in the OpenAI shape.
|
||||
|
||||
``error_type`` defaults to the value each call site emitted before this
|
||||
module existed (``connection_error`` for connection failures, otherwise
|
||||
``api_error``) so clients that switch on it keep working; ``code`` always
|
||||
carries the fixed vocabulary.
|
||||
"""
|
||||
error: dict[str, Any] = {
|
||||
"message": public_message(code, request_id=request_id, hint=hint),
|
||||
"type": error_type or _ANTHROPIC_TYPES[code],
|
||||
"code": code,
|
||||
}
|
||||
if request_id:
|
||||
error["request_id"] = request_id
|
||||
return {"error": error}
|
||||
|
||||
|
||||
def anthropic_error_body(
|
||||
code: str,
|
||||
*,
|
||||
request_id: str | None = None,
|
||||
error_type: str | None = None,
|
||||
hint: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""``{"type": "error", "error": {...}}`` in the Anthropic shape."""
|
||||
error: dict[str, Any] = {
|
||||
"type": error_type or _ANTHROPIC_TYPES[code],
|
||||
"message": public_message(code, request_id=request_id, hint=hint),
|
||||
"code": code,
|
||||
}
|
||||
body: dict[str, Any] = {"type": "error", "error": error}
|
||||
if request_id:
|
||||
body["request_id"] = request_id
|
||||
return body
|
||||
|
||||
|
||||
def is_provider_response_error(exc: BaseException) -> bool:
|
||||
"""True when ``exc`` carries a model provider's HTTP error response.
|
||||
|
||||
LiteLLM's provider errors (AuthenticationError, NotFoundError,
|
||||
RateLimitError, ...) subclass ``openai.APIStatusError``. Imported lazily:
|
||||
neither SDK is needed to import this module.
|
||||
"""
|
||||
try:
|
||||
import openai
|
||||
|
||||
if isinstance(exc, openai.APIStatusError):
|
||||
return True
|
||||
except ImportError: # pragma: no cover - openai is a proxy-extra dependency
|
||||
pass
|
||||
try:
|
||||
import anthropic
|
||||
|
||||
if isinstance(exc, anthropic.APIStatusError):
|
||||
return True
|
||||
except ImportError: # pragma: no cover - optional dependency
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def client_message(exc: BaseException, provider_text: str) -> str:
|
||||
"""The message a client may see for ``exc``.
|
||||
|
||||
``provider_text`` is returned only when ``exc`` is a provider's HTTP error
|
||||
response (authentication, model not found, rate limited): that text is
|
||||
addressed to the caller. Transport failures map to their fixed code and
|
||||
anything else to :data:`INTERNAL_ERROR`; the caller logs the real text.
|
||||
"""
|
||||
if is_provider_response_error(exc):
|
||||
return provider_text
|
||||
return public_message(classify_or_internal(exc))
|
||||
|
||||
|
||||
def tool_result_error(exc: BaseException) -> dict[str, Any]:
|
||||
"""``{"status": "error", ...}`` for a JSON tool result returned to a model.
|
||||
|
||||
The model only ever sees a fixed code: a storage, network or internal
|
||||
failure names no path, host or library. The caller logs the real text.
|
||||
"""
|
||||
code = classify_or_internal(exc)
|
||||
return {"status": "error", "error": code, "message": public_message(code)}
|
||||
@@ -212,8 +212,18 @@ def public_tags(tags: MutableMapping[str, Any] | None) -> dict[str, Any]:
|
||||
They are carried on ``tags`` because that is the one dict that reaches the
|
||||
outcome funnel from every handler; letting them through to ``RequestLog``
|
||||
would put a list and a dict into a string-keyed label store.
|
||||
|
||||
Credential-shaped keys are dropped here as well. :func:`helpers.extract_tags`
|
||||
already refuses them at the source; this is the last gate before the
|
||||
request log for tags a hook or extension added by hand.
|
||||
"""
|
||||
return {key: value for key, value in (tags or {}).items() if key not in _INTERNAL_TAGS}
|
||||
from headroom.proxy.internal_header_policy import is_credential_tag_key
|
||||
|
||||
return {
|
||||
key: value
|
||||
for key, value in (tags or {}).items()
|
||||
if key not in _INTERNAL_TAGS and not is_credential_tag_key(str(key))
|
||||
}
|
||||
|
||||
|
||||
def encode(items: list[dict[str, Any]]) -> str:
|
||||
|
||||
@@ -377,7 +377,8 @@ async def test_send_message_returns_error_response(monkeypatch: pytest.MonkeyPat
|
||||
|
||||
assert result.status_code == 401
|
||||
assert result.body["error"]["type"] == "authentication_error"
|
||||
assert result.error == "authentication api_key missing"
|
||||
# Unclassified exception text stays in the server log, never the client.
|
||||
assert result.error == "The proxy could not complete the request."
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -416,7 +417,7 @@ async def test_stream_message_yields_events_and_error(monkeypatch: pytest.Monkey
|
||||
instance_error.raise_error = RuntimeError("stream broke")
|
||||
error_events = [event async for event in backend_error.stream_message({"messages": []}, {})]
|
||||
assert error_events[-1].event_type == "error"
|
||||
assert error_events[-1].data["error"]["message"] == "stream broke"
|
||||
assert error_events[-1].data["error"]["message"] == "The proxy could not complete the request."
|
||||
|
||||
|
||||
def _tool_call_delta(*, index, tc_id=None, name=None, arguments=None): # noqa: ANN001, ANN202
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import AsyncIterator
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
@@ -17,8 +18,18 @@ from fastapi.testclient import TestClient # noqa: E402
|
||||
from headroom.backends.anyllm import AnyLLMBackend # noqa: E402
|
||||
from headroom.backends.base import BackendResponse # noqa: E402
|
||||
from headroom.backends.litellm import LiteLLMBackend # noqa: E402
|
||||
from headroom.proxy.public_errors import ( # noqa: E402
|
||||
UPSTREAM_PROTOCOL_ERROR,
|
||||
UPSTREAM_TIMEOUT,
|
||||
public_message,
|
||||
)
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
|
||||
# Transport exceptions never surface their own text (it names the upstream host);
|
||||
# an empty-message ReadError/ReadTimeout maps to the fixed vocabulary instead.
|
||||
_PROTOCOL = public_message(UPSTREAM_PROTOCOL_ERROR)
|
||||
_TIMEOUT = public_message(UPSTREAM_TIMEOUT)
|
||||
|
||||
_BODY = {
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
@@ -39,8 +50,8 @@ async def test_send_message_names_transport_error_without_message() -> None:
|
||||
result = await backend.send_message(_BODY, {})
|
||||
|
||||
assert result.status_code == 500
|
||||
assert result.error == "ReadError (no message)"
|
||||
assert result.body["error"]["message"] == "ReadError (no message)"
|
||||
assert result.error == _PROTOCOL
|
||||
assert result.body["error"]["message"] == _PROTOCOL
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -57,7 +68,7 @@ async def test_stream_message_names_transport_error_without_message() -> None:
|
||||
events = [event async for event in backend.stream_message(_BODY, {})]
|
||||
|
||||
error_event = next(event for event in events if event.event_type == "error")
|
||||
assert error_event.data["error"]["message"] == "ReadTimeout (no message)"
|
||||
assert error_event.data["error"]["message"] == _TIMEOUT
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -71,8 +82,8 @@ async def test_anyllm_backend_names_transport_error_without_message() -> None:
|
||||
result = await backend.send_message(_BODY, {})
|
||||
|
||||
assert result.status_code == 500
|
||||
assert result.error == "ReadError (no message)"
|
||||
assert result.body["error"]["message"] == "ReadError (no message)"
|
||||
assert result.error == _PROTOCOL
|
||||
assert result.body["error"]["message"] == _PROTOCOL
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -86,7 +97,7 @@ async def test_anyllm_stream_backend_names_transport_error_without_message() ->
|
||||
events = [event async for event in backend.stream_message(_BODY, {})]
|
||||
|
||||
error_event = next(event for event in events if event.event_type == "error")
|
||||
assert error_event.data["error"]["message"] == "ReadTimeout (no message)"
|
||||
assert error_event.data["error"]["message"] == _TIMEOUT
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -108,10 +119,10 @@ async def test_openai_backend_boundaries_name_transport_errors_without_message()
|
||||
litellm_backend = LiteLLMBackend(provider="openrouter")
|
||||
litellm_result = await litellm_backend.send_openai_message(_BODY, {})
|
||||
|
||||
assert anyllm_result.body["error"]["message"] == "ReadError (no message)"
|
||||
assert anyllm_result.error == "ReadError (no message)"
|
||||
assert litellm_result.body["error"]["message"] == "ReadTimeout (no message)"
|
||||
assert litellm_result.error == "ReadTimeout (no message)"
|
||||
assert anyllm_result.body["error"]["message"] == _PROTOCOL
|
||||
assert anyllm_result.error == _PROTOCOL
|
||||
assert litellm_result.body["error"]["message"] == _TIMEOUT
|
||||
assert litellm_result.error == _TIMEOUT
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -133,8 +144,8 @@ async def test_openai_stream_boundaries_name_transport_errors_without_message()
|
||||
litellm_backend = LiteLLMBackend(provider="openrouter")
|
||||
litellm_chunks = [chunk async for chunk in litellm_backend.stream_openai_message(_BODY, {})]
|
||||
|
||||
assert '"message": "ReadError (no message)"' in anyllm_chunks[0]
|
||||
assert '"message": "ReadTimeout (no message)"' in litellm_chunks[0]
|
||||
assert f'"message": {json.dumps(_PROTOCOL)}' in anyllm_chunks[0]
|
||||
assert f'"message": {json.dumps(_TIMEOUT)}' in litellm_chunks[0]
|
||||
|
||||
|
||||
def _erroring_anthropic_backend() -> MagicMock:
|
||||
@@ -187,7 +198,9 @@ def test_anthropic_proxy_names_nonstream_transport_error_without_message() -> No
|
||||
)
|
||||
|
||||
assert response.status_code == 500
|
||||
assert response.json()["error"]["message"] == "ReadError (no message)"
|
||||
# Proxy-level replies append the request id for log correlation.
|
||||
assert response.json()["error"]["message"].startswith(_PROTOCOL)
|
||||
assert response.json()["error"]["code"] == UPSTREAM_PROTOCOL_ERROR
|
||||
|
||||
|
||||
def test_bedrock_stream_names_transport_error_without_message() -> None:
|
||||
@@ -202,4 +215,5 @@ def test_bedrock_stream_names_transport_error_without_message() -> None:
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert '"message": "ReadTimeout (no message)"' in response.text
|
||||
assert _TIMEOUT in response.text
|
||||
assert '"code": "upstream_timeout"' in response.text
|
||||
|
||||
@@ -751,7 +751,10 @@ async def test_execute_memory_tool_save_returns_dedup_hint(handler: MemoryHandle
|
||||
|
||||
backend.raise_on = "save"
|
||||
errored = json.loads(await handler._execute_memory_tool("memory_save", {"content": "x"}, "u1"))
|
||||
assert errored == {"status": "error", "error": "save failed"}
|
||||
# Backend exception text is logged, never returned to the model.
|
||||
assert errored["status"] == "error"
|
||||
assert errored["error"] == "internal_error"
|
||||
assert "save failed" not in json.dumps(errored)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -16,6 +16,7 @@ don't need a real provider:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
@@ -359,10 +360,22 @@ def test_backend_ccr_intercept_exception_is_reraised_not_swallowed():
|
||||
"usage": {"prompt_tokens": 50, "completion_tokens": 5, "total_tokens": 55},
|
||||
}
|
||||
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
proxy_logger = logging.getLogger("headroom.proxy")
|
||||
|
||||
mock_backend = _make_mock_backend(tool_call_response)
|
||||
with patch("headroom.proxy.server.AnyLLMBackend", return_value=mock_backend):
|
||||
app = create_app(config)
|
||||
with TestClient(app) as client:
|
||||
# Attach after startup: proxy logging setup would otherwise
|
||||
# detach a handler added earlier (caplog has the same problem).
|
||||
proxy_logger.addHandler(handler)
|
||||
_install_tracker_stub(client)
|
||||
proxy = client.app.state.proxy
|
||||
failing_handler = MagicMock()
|
||||
@@ -381,6 +394,7 @@ def test_backend_ccr_intercept_exception_is_reraised_not_swallowed():
|
||||
},
|
||||
headers={"Authorization": "Bearer test-key"},
|
||||
)
|
||||
proxy_logger.removeHandler(handler)
|
||||
|
||||
# The outer `try/except Exception` on the backend block converts the
|
||||
# re-raise into a 500 response. The critical assertion is that the
|
||||
@@ -397,7 +411,14 @@ def test_backend_ccr_intercept_exception_is_reraised_not_swallowed():
|
||||
or body.get("choices", [{}])[0].get("message", {}).get("tool_calls") is None
|
||||
)
|
||||
assert "error" in body
|
||||
assert "ccr-store-blew-up" in body["error"]["message"]
|
||||
# The client gets the fixed vocabulary and a correlation id, not the
|
||||
# exception text; the operator log keeps the detail under that id.
|
||||
assert body["error"]["code"] == "internal_error"
|
||||
request_id = body["error"]["request_id"]
|
||||
assert request_id and f"request_id={request_id}" in body["error"]["message"]
|
||||
assert "ccr-store-blew-up" not in resp.text
|
||||
logged = [r.getMessage() for r in records]
|
||||
assert any(f"[{request_id}]" in m and "ccr-store-blew-up" in m for m in logged), logged
|
||||
|
||||
|
||||
def test_backend_streaming_passes_prefix_tracker_through():
|
||||
|
||||
@@ -392,12 +392,11 @@ class TestCompressEndpointCompression:
|
||||
)
|
||||
|
||||
assert response.status_code == 503
|
||||
assert response.json() == {
|
||||
"error": {
|
||||
"type": "compression_error",
|
||||
"message": "compression broke",
|
||||
}
|
||||
}
|
||||
body = response.json()
|
||||
# Exception text stays in the server log; the client sees the code.
|
||||
assert body["error"]["type"] == "compression_error"
|
||||
assert body["error"]["code"] == "internal_error"
|
||||
assert "compression broke" not in response.text
|
||||
record_failed.assert_awaited_once_with(provider="compress")
|
||||
|
||||
def test_compression_timeout_records_fail_open_outcome(self, client, monkeypatch):
|
||||
|
||||
@@ -9,6 +9,7 @@ without passing any upstream headers — silently dropping ratelimit info.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
@@ -496,42 +497,69 @@ class TestStreamingRatelimitHeaderForwarding:
|
||||
|
||||
mock_request = MagicMock()
|
||||
proxy.http_client.build_request = MagicMock(return_value=mock_request)
|
||||
proxy.http_client.send = AsyncMock(side_effect=httpx.ConnectError("Connection refused"))
|
||||
|
||||
result = await proxy._stream_response(
|
||||
url="https://api.anthropic.com/v1/messages",
|
||||
headers={"x-api-key": "sk-test"},
|
||||
body={
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"max_tokens": 100,
|
||||
"stream": True,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
provider="anthropic",
|
||||
model="claude-sonnet-4-20250514",
|
||||
request_id="test-error",
|
||||
original_tokens=10,
|
||||
optimized_tokens=10,
|
||||
tokens_saved=0,
|
||||
transforms_applied=[],
|
||||
tags={},
|
||||
optimization_latency=0.0,
|
||||
proxy.http_client.send = AsyncMock(
|
||||
side_effect=httpx.ConnectError("Connection refused to 10.0.0.7:443")
|
||||
)
|
||||
|
||||
# Should return a StreamingResponse with error SSE event
|
||||
assert result.media_type == "text/event-stream"
|
||||
# Attach directly to the proxy logger: caplog does not reliably see
|
||||
# ``headroom.*`` records once proxy logging is configured.
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
# Consume the generator to get the error event
|
||||
chunks = []
|
||||
async for chunk in result.body_iterator:
|
||||
chunks.append(chunk)
|
||||
class _Collect(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Collect(level=logging.DEBUG)
|
||||
proxy_logger = logging.getLogger("headroom.proxy")
|
||||
proxy_logger.addHandler(handler)
|
||||
try:
|
||||
result = await proxy._stream_response(
|
||||
url="https://api.anthropic.com/v1/messages",
|
||||
headers={"x-api-key": "sk-test"},
|
||||
body={
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"max_tokens": 100,
|
||||
"stream": True,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
provider="anthropic",
|
||||
model="claude-sonnet-4-20250514",
|
||||
request_id="test-error",
|
||||
original_tokens=10,
|
||||
optimized_tokens=10,
|
||||
tokens_saved=0,
|
||||
transforms_applied=[],
|
||||
tags={},
|
||||
optimization_latency=0.0,
|
||||
)
|
||||
|
||||
# Should return a StreamingResponse with error SSE event
|
||||
assert result.media_type == "text/event-stream"
|
||||
|
||||
# Consume the generator to get the error event
|
||||
chunks = []
|
||||
async for chunk in result.body_iterator:
|
||||
chunks.append(chunk)
|
||||
finally:
|
||||
proxy_logger.removeHandler(handler)
|
||||
|
||||
assert len(chunks) == 1
|
||||
raw = chunks[0].decode("utf-8")
|
||||
assert "event: error" in raw
|
||||
error_data = json.loads(raw.split("data: ")[1].strip())
|
||||
assert error_data["error"]["type"] == "connection_error"
|
||||
assert "Connection refused" in error_data["error"]["message"]
|
||||
# Client sees the fixed vocabulary plus a correlation id, never the
|
||||
# transport text (which can carry upstream hosts and addresses).
|
||||
assert error_data["error"]["code"] == "upstream_unreachable"
|
||||
assert error_data["request_id"] == "test-error"
|
||||
assert "request_id=test-error" in error_data["error"]["message"]
|
||||
assert "Connection refused" not in raw
|
||||
assert "10.0.0.7" not in raw
|
||||
# The operator still gets the full detail, keyed by the same id.
|
||||
logged = [r.getMessage() for r in records]
|
||||
assert any(
|
||||
"[test-error]" in m and "Connection refused to 10.0.0.7:443" in m for m in logged
|
||||
), logged
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_connect_timeout_retries_before_returning_stream(self):
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
"""Transport and internal exception text must never reach a client (01-F4).
|
||||
|
||||
``httpx``/``ssl``/OS exception text names the resolved upstream host, its IP
|
||||
and the failing library. Every response path maps such failures to the fixed
|
||||
vocabulary in :mod:`headroom.proxy.public_errors` and keeps the detail in the
|
||||
server log under the request id.
|
||||
|
||||
Companion: the shared proxy token must never land in request tags (10-F4).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import ssl
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from headroom.proxy import public_errors # noqa: E402
|
||||
from headroom.proxy.handlers.openai import OpenAIHandlerMixin # noqa: E402
|
||||
from headroom.proxy.helpers import extract_tags # noqa: E402
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
|
||||
# A string that only ever appears in exception text; every assertion below is
|
||||
# "this never reaches the wire".
|
||||
SECRET = "gateway-int.corp.example:443 (10.42.0.7)"
|
||||
|
||||
|
||||
# --- Vocabulary --------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("exc", "code"),
|
||||
[
|
||||
(httpx.ConnectError(SECRET), public_errors.UPSTREAM_UNREACHABLE),
|
||||
(httpx.ConnectTimeout(SECRET), public_errors.UPSTREAM_UNREACHABLE),
|
||||
(httpx.PoolTimeout(SECRET), public_errors.UPSTREAM_UNREACHABLE),
|
||||
(httpx.ReadTimeout(SECRET), public_errors.UPSTREAM_TIMEOUT),
|
||||
(httpx.WriteTimeout(SECRET), public_errors.UPSTREAM_TIMEOUT),
|
||||
(httpx.RemoteProtocolError(SECRET), public_errors.UPSTREAM_PROTOCOL_ERROR),
|
||||
(httpx.ReadError(SECRET), public_errors.UPSTREAM_PROTOCOL_ERROR),
|
||||
(httpx.DecodingError(SECRET), public_errors.UPSTREAM_PROTOCOL_ERROR),
|
||||
(ConnectionResetError(104, SECRET), public_errors.UPSTREAM_UNREACHABLE),
|
||||
(OSError(113, SECRET), public_errors.UPSTREAM_UNREACHABLE),
|
||||
(asyncio.TimeoutError(), public_errors.UPSTREAM_TIMEOUT),
|
||||
(
|
||||
ssl.SSLCertVerificationError(1, "CERTIFICATE_VERIFY_FAILED"),
|
||||
public_errors.UPSTREAM_TLS_ERROR,
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_transport_exceptions_classify_to_fixed_codes(exc, code) -> None:
|
||||
assert public_errors.classify_exception(exc) == code
|
||||
assert SECRET not in public_errors.public_message(code, request_id="req-1")
|
||||
|
||||
|
||||
def test_tls_failure_wrapped_in_connect_error_is_tls() -> None:
|
||||
inner = ssl.SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] " + SECRET)
|
||||
outer = httpx.ConnectError(SECRET)
|
||||
outer.__cause__ = inner
|
||||
assert public_errors.classify_exception(outer) == public_errors.UPSTREAM_TLS_ERROR
|
||||
|
||||
|
||||
def test_non_transport_exceptions_are_not_classified() -> None:
|
||||
"""Provider API errors (litellm/anyllm exceptions) keep their own message."""
|
||||
assert public_errors.classify_exception(RuntimeError("model not found: gpt-x")) is None
|
||||
assert public_errors.classify_exception(ValueError("bad")) is None
|
||||
assert public_errors.classify_or_internal(ValueError("bad")) == public_errors.INTERNAL_ERROR
|
||||
|
||||
|
||||
def test_public_message_rejects_unknown_codes() -> None:
|
||||
with pytest.raises(ValueError):
|
||||
public_errors.public_message("stack trace here")
|
||||
|
||||
|
||||
def test_wire_shapes_carry_code_and_request_id() -> None:
|
||||
openai = public_errors.openai_error_body(
|
||||
public_errors.UPSTREAM_UNREACHABLE, request_id="req-9", error_type="connection_error"
|
||||
)
|
||||
assert openai["error"]["type"] == "connection_error"
|
||||
assert openai["error"]["code"] == "upstream_unreachable"
|
||||
assert openai["error"]["request_id"] == "req-9"
|
||||
assert "req-9" in openai["error"]["message"]
|
||||
|
||||
anthropic = public_errors.anthropic_error_body(public_errors.INTERNAL_ERROR, request_id="req-9")
|
||||
assert anthropic["type"] == "error"
|
||||
assert anthropic["error"]["type"] == "api_error"
|
||||
assert anthropic["error"]["code"] == "internal_error"
|
||||
assert anthropic["request_id"] == "req-9"
|
||||
|
||||
|
||||
# --- Handler paths -------------------------------------------------------------
|
||||
|
||||
|
||||
def _config(**overrides) -> ProxyConfig:
|
||||
base: dict[str, object] = {
|
||||
"optimize": False,
|
||||
"cache_enabled": False,
|
||||
"rate_limit_enabled": False,
|
||||
"cost_tracking_enabled": False,
|
||||
"log_requests": False,
|
||||
"ccr_inject_tool": False,
|
||||
"ccr_handle_responses": False,
|
||||
"ccr_context_tracking": False,
|
||||
"image_optimize": False,
|
||||
"retry_enabled": False,
|
||||
}
|
||||
base.update(overrides)
|
||||
return ProxyConfig(**base)
|
||||
|
||||
|
||||
class _SecretConnectTimeoutClient:
|
||||
async def request(self, **kwargs): # noqa: ANN001, ANN201
|
||||
raise httpx.ConnectTimeout(f"connect to {SECRET} timed out")
|
||||
|
||||
|
||||
class _PassthroughRequest:
|
||||
method = "GET"
|
||||
headers = {}
|
||||
url = SimpleNamespace(path="/some/other/path", query="")
|
||||
|
||||
async def body(self) -> bytes:
|
||||
return b""
|
||||
|
||||
|
||||
def test_passthrough_connect_error_body_has_no_upstream_detail() -> None:
|
||||
handler = object.__new__(OpenAIHandlerMixin)
|
||||
handler.http_client = _SecretConnectTimeoutClient()
|
||||
|
||||
response = asyncio.run(
|
||||
handler.handle_passthrough(_PassthroughRequest(), "https://api.openai.com")
|
||||
)
|
||||
|
||||
assert response.status_code == 502
|
||||
payload = json.loads(response.body)
|
||||
assert payload["error"]["type"] == "connection_error"
|
||||
assert payload["error"]["code"] == public_errors.UPSTREAM_UNREACHABLE
|
||||
assert SECRET not in response.body.decode()
|
||||
|
||||
|
||||
def test_direct_streaming_connect_error_event_has_no_upstream_detail(monkeypatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
||||
with TestClient(create_app(_config()), raise_server_exceptions=False) as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.http_client.send = AsyncMock(side_effect=httpx.ConnectError(f"refused by {SECRET}"))
|
||||
|
||||
response = client.post(
|
||||
"/v1/messages",
|
||||
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
||||
json={
|
||||
"model": "claude-haiku-4-5",
|
||||
"max_tokens": 16,
|
||||
"stream": True,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 502
|
||||
text = response.text
|
||||
assert SECRET not in text
|
||||
assert "event: error" in text
|
||||
assert public_errors.UPSTREAM_UNREACHABLE in text
|
||||
|
||||
|
||||
def test_compress_endpoint_error_body_has_no_exception_detail(monkeypatch) -> None:
|
||||
app = create_app(_config(optimize=True))
|
||||
with TestClient(
|
||||
app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345), raise_server_exceptions=False
|
||||
) as client:
|
||||
proxy = client.app.state.proxy
|
||||
monkeypatch.setattr(
|
||||
proxy,
|
||||
"_run_compression_in_executor",
|
||||
AsyncMock(side_effect=RuntimeError(f"compressor exploded reading {SECRET}")),
|
||||
)
|
||||
|
||||
response = client.post(
|
||||
"/v1/compress",
|
||||
json={"messages": [{"role": "user", "content": "compress me"}], "model": "gpt-4"},
|
||||
)
|
||||
|
||||
assert response.status_code == 503
|
||||
body = response.json()
|
||||
assert body["error"]["type"] == "compression_error"
|
||||
assert body["error"]["code"] == public_errors.INTERNAL_ERROR
|
||||
assert SECRET not in response.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_memory_tool_result_hides_transport_and_internal_text(tmp_path) -> None:
|
||||
from headroom.proxy.memory_handler import MemoryConfig, MemoryHandler
|
||||
|
||||
handler = MemoryHandler(
|
||||
MemoryConfig(enabled=False, use_native_tool=True, native_memory_dir=str(tmp_path / "n")),
|
||||
agent_type="codex",
|
||||
)
|
||||
|
||||
class _Backend:
|
||||
exc: BaseException = OSError(5, f"sqlite at {SECRET} is locked")
|
||||
|
||||
async def save_memory(self, **kwargs): # noqa: ANN003
|
||||
raise self.exc
|
||||
|
||||
backend = _Backend()
|
||||
handler._backend = backend
|
||||
|
||||
transport = json.loads(
|
||||
await handler._execute_memory_tool("memory_save", {"content": "x"}, "u1")
|
||||
)
|
||||
assert transport["status"] == "error"
|
||||
assert SECRET not in json.dumps(transport)
|
||||
assert transport["error"] == public_errors.UPSTREAM_UNREACHABLE
|
||||
|
||||
backend.exc = RuntimeError(f"save failed at {LEAKY}")
|
||||
internal = json.loads(await handler._execute_memory_tool("memory_save", {"content": "x"}, "u1"))
|
||||
assert internal["error"] == public_errors.INTERNAL_ERROR
|
||||
assert LEAKY not in json.dumps(internal)
|
||||
|
||||
|
||||
# --- Unclassified exceptions fail closed ---------------------------------------
|
||||
|
||||
# Exception text that names a local path, an internal host and a token.
|
||||
LEAKY = r"sqlite at C:\service\tenant-a.db via db-int.corp.example failed with sk-live-secret"
|
||||
|
||||
UNCLASSIFIED = [
|
||||
RuntimeError(LEAKY),
|
||||
ValueError(LEAKY),
|
||||
KeyError(LEAKY),
|
||||
json.JSONDecodeError(LEAKY, "{}", 0),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc", UNCLASSIFIED, ids=lambda e: type(e).__name__)
|
||||
def test_client_message_hides_unclassified_exception_text(exc) -> None:
|
||||
message = public_errors.client_message(exc, LEAKY)
|
||||
assert message == public_errors.public_message(public_errors.INTERNAL_ERROR)
|
||||
assert "sk-live-secret" not in message
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc", UNCLASSIFIED, ids=lambda e: type(e).__name__)
|
||||
def test_tool_result_error_hides_unclassified_exception_text(exc) -> None:
|
||||
result = public_errors.tool_result_error(exc)
|
||||
assert result["error"] == public_errors.INTERNAL_ERROR
|
||||
assert "sk-live-secret" not in json.dumps(result)
|
||||
|
||||
|
||||
def test_client_message_forwards_provider_http_error_response() -> None:
|
||||
import openai
|
||||
|
||||
response = httpx.Response(401, request=httpx.Request("POST", "https://api.example/v1"))
|
||||
exc = openai.AuthenticationError("invalid x-api-key", response=response, body=None)
|
||||
assert public_errors.client_message(exc, "invalid x-api-key") == "invalid x-api-key"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_litellm_backend_hides_unclassified_exception_text(monkeypatch) -> None:
|
||||
from headroom.backends import litellm as litellm_backend
|
||||
|
||||
monkeypatch.setattr(litellm_backend, "acompletion", AsyncMock(side_effect=RuntimeError(LEAKY)))
|
||||
backend = litellm_backend.LiteLLMBackend(provider="openai")
|
||||
result = await backend.send_openai_message(
|
||||
{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}]}, {}
|
||||
)
|
||||
assert "sk-live-secret" not in json.dumps(result.body)
|
||||
assert result.body["error"]["message"] == public_errors.public_message(
|
||||
public_errors.INTERNAL_ERROR
|
||||
)
|
||||
|
||||
|
||||
# --- Tags (10-F4) -------------------------------------------------------------
|
||||
|
||||
|
||||
def test_extract_tags_never_carries_credentials() -> None:
|
||||
tags = extract_tags(
|
||||
{
|
||||
"X-Headroom-Proxy-Token": "hlk_secret_token_value",
|
||||
"x-headroom-project": "alpha",
|
||||
"x-headroom-session-id": "s-1",
|
||||
"x-headroom-license-token": "lic",
|
||||
"x-headroom-api-key": "k",
|
||||
"X-Headroom-Key": "hr_cloud_key",
|
||||
"authorization": "Bearer sk-live",
|
||||
}
|
||||
)
|
||||
assert tags == {"project": "alpha", "session-id": "s-1"}
|
||||
|
||||
|
||||
def test_public_tags_drop_credential_like_keys_defensively() -> None:
|
||||
from headroom.proxy.savings_attribution import public_tags
|
||||
|
||||
assert public_tags({"proxy-token": "x", "project": "alpha", "some_secret": "y"}) == {
|
||||
"project": "alpha"
|
||||
}
|
||||
Reference in New Issue
Block a user