mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
refactor(integrations): share the local/cloud compress backend of asgi and litellm_callback (#4124)
## Description `CompressionMiddleware` (`headroom/integrations/asgi.py`) and `HeadroomCallback` (`headroom/integrations/litellm_callback.py`) carried identical copies of `_local_compress` and `_cloud_compress`; `diff` of the two method blocks reported no difference. The cloud URL constant, the `HEADROOM_API_KEY` / `HEADROOM_API_URL` reads and `cloud_mode` were copied too. This PR moves them into a `CompressBackend` mixin in `headroom/integrations/_compress_backend.py`. That module imports neither starlette nor litellm, so each integration still works without the other's dependency. Refs #4136 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring ## Changes Made - New `headroom/integrations/_compress_backend.py` with `CompressBackend`, which provides: - `_init_compress_backend(model_limit, hooks, api_key, api_url)`; - `cloud_mode`; - `_local_compress`; - `_cloud_compress`, which keeps the `guard_egress` offline check. - `CompressionMiddleware(CompressBackend)` and `HeadroomCallback(CompressBackend, CustomLogger)` call `_init_compress_backend` and drop their copies. - The method names stay, because `tests/test_offline_egress_chokepoint.py` calls `_cloud_compress` directly. - Each class sets `_log` to its own module logger, so the "Headroom Cloud API error" warning keeps its logger name (`headroom.integrations.asgi` / `headroom.integrations.litellm_callback`). - One log change: that WARNING used to include the first 200 characters of the cloud response body, which can echo request content. It now reads `Headroom Cloud API error: HTTP <status> from <api_url>; request sent uncompressed`, with the URL passed through `redact_url` from `headroom/log_safety.py` (#4138): scheme, host and port only, the path shown as `/<path>` and any query as `?<redacted>`. The body is not logged at any level: a separate DEBUG line gives only the bare media type (`type/subtype`, header parameters dropped) and the body length. - `_DEFAULT_CLOUD_URL` and `guard_egress` stay importable from both modules. - Left alone on purpose: - each class's `aclose`: they differ slightly in when `_client` is cleared; - `min_tokens`: it is stored and never read in either class, as on main. The maintainer should decide separately whether to wire it up. - No `CustomLogger` attribute clashes with the mixin's names. I checked `dir(CustomLogger)` and the instance `vars()`. - Tests: - New `tests/test_integrations/test_compress_backend.py`, parametrized over `CompressionMiddleware` and `HeadroomCallback` with an `httpx.MockTransport` client: a 200 returns the service result (URL, `X-Headroom-Key` header and JSON body checked), and a 503 whose body echoes the prompt returns `None`. `test_cloud_error_logs_no_echoed_content_at_any_level` runs both integrations with the logger at DEBUG and at WARNING, formats every record with `logging.Formatter().format`, and asserts that prompt, API-key, URL-query and URL-path canaries (plain and `%`-encoded) are absent, including a prompt canary echoed in the `Content-Type` header. It fails on 95c608ee; the path cases fail before #4138'sb566a758, and the header case fails on4fa6bb34. `test_direct_load_does_not_import_the_integrations_package` now also checks that `headroom._core` is not loaded, and the fixture comment names what the loaded files import. - `tests/test_integrations/test_litellm_callback.py`: the direct-file loader registers a file-loaded `_compress_backend` before loading `litellm_callback.py`, so the test again avoids importing `headroom.integrations` and LangChain. New `test_direct_load_does_not_import_the_integrations_package` checks that in a fresh interpreter. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compress_api.py tests/test_litellm_callback.py tests/test_integrations/test_litellm_callback.py origin/main: 37 passed this branch: 37 passed $ LITELLM_LOCAL_MODEL_COST_MAP=True pytest tests/test_offline_egress_chokepoint.py -k "asgi or litellm" origin/main: 2 passed this branch: 2 passed $ pytest tests/test_offline_egress_chokepoint.py origin/main: 2 failed, 141 passed this branch: 2 failed, 141 passed (pre-existing and environmental: the same two asgi/litellm tests fail on main, because importing litellm fetches its model cost map from raw.githubusercontent.com and the test's no-socket fixture refuses it; with LITELLM_LOCAL_MODEL_COST_MAP=True both pass on main and on this branch) $ pytest tests/test_integrations/test_compress_backend.py tests/test_integrations/test_litellm_callback.py tests/test_compress_api.py tests/test_litellm_callback.py tests/test_offline_egress_chokepoint.py this branch: 185 passed # test_compress_backend.py against the previous commit's backend (04d38103): 2 failed (warning carried the body), 2 passed $ uvx ruff@0.16.8 check / format <touched files> -> All checks passed! / unchanged $ mypy headroom/integrations/_compress_backend.py asgi.py litellm_callback.py --ignore-missing-imports headroom/integrations/langchain/chat_model.py:394: error: ... [truthy-function] (pre-existing, followed import, untouched file) ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, repo `.venv` (litellm, starlette and httpx installed), this branch's worktree. Temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`. - Exact command / steps: two runs, listed below. 1. Shared smoke harness: a real proxy from this checkout against a local fake upstream (`smoke.sh <worktree>`). 2. The shared cloud path over real HTTP, with both module loggers at DEBUG: a local HTTP server answers 503 (echoing the request, which carries a prompt canary) and then 200, for `HeadroomCallback` and then `CompressionMiddleware` (`_cloud_compress`). The API key, a `/bot…` path segment and a `?token=` query on the URL are canaries too. - Observed result: the output of both runs is in the two blocks below. ```text branch: cleanup/cli-integrations-shared-compress-backend @fad5714chealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-58395.log): 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes. 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. (smoke dir: <tmp>) ``` ```text HeadroomCallback 503 -> None | 200 -> 3 CompressionMiddleware 503 -> None | 200 -> 3 WARNING headroom.integrations.litellm_callback: Headroom Cloud API error: HTTP 503 from http://127.0.0.1:PORT/<path>?<redacted>; request sent uncompressed DEBUG headroom.integrations.litellm_callback: Headroom Cloud API error response: content-type=text/plain, 138 bytes WARNING headroom.integrations.asgi: Headroom Cloud API error: HTTP 503 from http://127.0.0.1:PORT/<path>?<redacted>; request sent uncompressed DEBUG headroom.integrations.asgi: Headroom Cloud API error response: content-type=text/plain, 138 bytes canaries in log: 0 ``` - Not tested: the real Headroom Cloud service. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a, library refactor. - Stable/default behavior changed: no. The env vars, default URL, payload, timeout, offline refusal and log text are all the same. - Kill switch / disable path: n/a. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. Depends on #4138; its two files show in this diff until it merges. - `tests/test_integrations/test_litellm_callback.py` loads `litellm_callback.py` from its file path, to avoid importing the `headroom.integrations` package. Because the module now imports `headroom.integrations._compress_backend`, the loader registers a file-loaded copy of that module first; a module already in `sys.modules` is returned without importing its parent package. Normal imports of `headroom.integrations.litellm_callback` load the package anyway, as before. - A test that patched `headroom.integrations.asgi.guard_egress` would no longer intercept the call, because the call now lives in `_compress_backend`. No such test exists. - Docs impact (the changed warning text) is recorded in the docs ledger; there are no doc edits in this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
4e6e955cad
commit
58caafb59b
@@ -0,0 +1,152 @@
|
||||
"""Local or Headroom Cloud compression, shared by the ASGI and LiteLLM integrations.
|
||||
|
||||
``CompressionMiddleware`` (asgi.py) and ``HeadroomCallback`` (litellm_callback.py)
|
||||
compress the same way: in-process with ``headroom.compress()`` by default, or
|
||||
through the Headroom Cloud API when an API key is set.
|
||||
|
||||
This module must not import starlette or litellm, so either integration can be
|
||||
used without the other's dependency.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from headroom.log_safety import redact_url
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
|
||||
_DEFAULT_MODEL = "claude-sonnet-4-5-20250929"
|
||||
|
||||
|
||||
_KNOWN_MEDIA_TYPES = frozenset(
|
||||
{
|
||||
"application/json",
|
||||
"application/problem+json",
|
||||
"application/octet-stream",
|
||||
"text/plain",
|
||||
"text/html",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _media_type(content_type: str) -> str:
|
||||
"""Return just the ``type/subtype`` of a Content-Type header, or ``<other>``.
|
||||
|
||||
Parameters and anything that is not a plain media type are dropped, so an
|
||||
error response cannot smuggle echoed content into the log through a header.
|
||||
"""
|
||||
# Valid type/subtype syntax can still contain echoed private content.
|
||||
media = content_type.split(";", 1)[0].strip().lower()
|
||||
return media if media in _KNOWN_MEDIA_TYPES else "<other>"
|
||||
|
||||
|
||||
class CompressBackend:
|
||||
"""Mixin that picks local or cloud compression and holds the cloud client.
|
||||
|
||||
Subclasses call ``_init_compress_backend`` from ``__init__`` and set
|
||||
``_log`` to their module logger, so log lines keep that logger's name.
|
||||
"""
|
||||
|
||||
_log: logging.Logger
|
||||
|
||||
def _init_compress_backend(
|
||||
self,
|
||||
model_limit: int,
|
||||
hooks: Any,
|
||||
api_key: str | None,
|
||||
api_url: str | None,
|
||||
) -> None:
|
||||
self._model_limit = model_limit
|
||||
self._hooks = hooks
|
||||
|
||||
# Cloud mode: if api_key is set, compress via Headroom Cloud API
|
||||
# Falls back to HEADROOM_API_KEY env var
|
||||
self._api_key = api_key or os.environ.get("HEADROOM_API_KEY", "").strip() or None
|
||||
self._api_url = (
|
||||
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
|
||||
).rstrip("/")
|
||||
self._client: Any = None # Lazy-initialized httpx.AsyncClient
|
||||
|
||||
@property
|
||||
def cloud_mode(self) -> bool:
|
||||
"""Whether cloud compression is enabled."""
|
||||
return self._api_key is not None
|
||||
|
||||
def _local_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
|
||||
"""Compress locally using headroom.compress()."""
|
||||
from headroom.compress import compress
|
||||
|
||||
result = compress(
|
||||
messages=messages,
|
||||
model=model or _DEFAULT_MODEL,
|
||||
model_limit=self._model_limit,
|
||||
hooks=self._hooks,
|
||||
)
|
||||
return {
|
||||
"messages": result.messages,
|
||||
"tokens_before": result.tokens_before,
|
||||
"tokens_after": result.tokens_after,
|
||||
"tokens_saved": result.tokens_saved,
|
||||
"compression_ratio": result.compression_ratio,
|
||||
}
|
||||
|
||||
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
|
||||
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
|
||||
|
||||
This is the one path in these integrations that puts the caller's
|
||||
prompt content on the wire to a Headroom-operated host, so it is
|
||||
exactly what HEADROOM_OFFLINE exists to stop. "Opt-in by configuration"
|
||||
was the old reason for leaving it open, and it is not good enough: an
|
||||
operator who sets an air-gap switch is overriding earlier configuration
|
||||
on purpose. The refusal is loud rather than a silent fall-through to
|
||||
local compression, because silently compressing locally would hide the
|
||||
fact that the deployment is no longer doing what it was configured to do.
|
||||
"""
|
||||
guard_egress("Headroom Cloud compression", self._api_url)
|
||||
if self._client is None:
|
||||
try:
|
||||
import httpx
|
||||
except ImportError as e:
|
||||
raise ImportError(
|
||||
"httpx is required for Headroom Cloud mode: pip install httpx"
|
||||
) from e
|
||||
self._client = httpx.AsyncClient(timeout=30.0)
|
||||
|
||||
client = self._client
|
||||
assert client is not None
|
||||
resp = await client.post(
|
||||
f"{self._api_url}/v1/saas/compress",
|
||||
headers={
|
||||
"X-Headroom-Key": self._api_key,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
content=json.dumps(
|
||||
{
|
||||
"messages": messages,
|
||||
"model": model or _DEFAULT_MODEL,
|
||||
"model_limit": self._model_limit,
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
if resp.status_code != 200:
|
||||
# The response body can echo request content: log metadata only, at
|
||||
# every level.
|
||||
self._log.warning(
|
||||
"Headroom Cloud API error: HTTP %d from %s; request sent uncompressed",
|
||||
resp.status_code,
|
||||
redact_url(self._api_url),
|
||||
)
|
||||
self._log.debug(
|
||||
"Headroom Cloud API error response: content-type=%s, %d bytes",
|
||||
_media_type(resp.headers.get("content-type", "")),
|
||||
len(resp.content),
|
||||
)
|
||||
return None
|
||||
|
||||
result: dict[str, Any] = resp.json()
|
||||
return result
|
||||
@@ -28,18 +28,20 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import MutableMapping
|
||||
from typing import Any
|
||||
|
||||
from starlette.types import ASGIApp, Receive, Scope, Send
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
# _DEFAULT_CLOUD_URL and guard_egress stay importable from this module.
|
||||
from headroom.integrations._compress_backend import ( # noqa: F401
|
||||
_DEFAULT_CLOUD_URL,
|
||||
CompressBackend,
|
||||
)
|
||||
from headroom.offline import guard_egress # noqa: F401
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
|
||||
|
||||
# Paths that contain LLM messages to compress
|
||||
_LLM_PATHS = (
|
||||
"/v1/messages", # Anthropic
|
||||
@@ -49,7 +51,7 @@ _LLM_PATHS = (
|
||||
)
|
||||
|
||||
|
||||
class CompressionMiddleware:
|
||||
class CompressionMiddleware(CompressBackend):
|
||||
"""ASGI middleware that compresses LLM request messages.
|
||||
|
||||
Two modes:
|
||||
@@ -64,6 +66,8 @@ class CompressionMiddleware:
|
||||
- x-headroom-compressed: "true" if compression occurred
|
||||
"""
|
||||
|
||||
_log = logger
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
app: ASGIApp,
|
||||
@@ -75,20 +79,7 @@ class CompressionMiddleware:
|
||||
) -> None:
|
||||
self.app = app
|
||||
self._min_tokens = min_tokens
|
||||
self._model_limit = model_limit
|
||||
self._hooks = hooks
|
||||
|
||||
# Cloud mode: if api_key is set, compress via Headroom Cloud API
|
||||
self._api_key = api_key or os.environ.get("HEADROOM_API_KEY", "").strip() or None
|
||||
self._api_url = (
|
||||
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
|
||||
).rstrip("/")
|
||||
self._client: Any = None # Lazy-initialized httpx.AsyncClient
|
||||
|
||||
@property
|
||||
def cloud_mode(self) -> bool:
|
||||
"""Whether cloud compression is enabled."""
|
||||
return self._api_key is not None
|
||||
self._init_compress_backend(model_limit, hooks, api_key, api_url)
|
||||
|
||||
async def aclose(self) -> None:
|
||||
"""Close the underlying httpx.AsyncClient, if one was created."""
|
||||
@@ -186,67 +177,3 @@ class CompressionMiddleware:
|
||||
await send(message)
|
||||
|
||||
await self.app(scope, modified_receive, metrics_send)
|
||||
|
||||
def _local_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
|
||||
"""Compress locally using headroom.compress()."""
|
||||
from headroom.compress import compress
|
||||
|
||||
result = compress(
|
||||
messages=messages,
|
||||
model=model or "claude-sonnet-4-5-20250929",
|
||||
model_limit=self._model_limit,
|
||||
hooks=self._hooks,
|
||||
)
|
||||
return {
|
||||
"messages": result.messages,
|
||||
"tokens_before": result.tokens_before,
|
||||
"tokens_after": result.tokens_after,
|
||||
"tokens_saved": result.tokens_saved,
|
||||
"compression_ratio": result.compression_ratio,
|
||||
}
|
||||
|
||||
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
|
||||
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
|
||||
|
||||
This is the one path in this file that puts the caller's prompt
|
||||
content on the wire to a Headroom-operated host, so it is exactly what
|
||||
HEADROOM_OFFLINE exists to stop. "Opt-in by configuration" was the old
|
||||
reason for leaving it open, and it is not good enough: an operator who
|
||||
sets an air-gap switch is overriding earlier configuration on purpose.
|
||||
The refusal is loud rather than a silent fall-through to local
|
||||
compression, because silently compressing locally would hide the fact
|
||||
that the deployment is no longer doing what it was configured to do.
|
||||
"""
|
||||
guard_egress("Headroom Cloud compression", self._api_url)
|
||||
if self._client is None:
|
||||
try:
|
||||
import httpx
|
||||
except ImportError as e:
|
||||
raise ImportError(
|
||||
"httpx is required for Headroom Cloud mode: pip install httpx"
|
||||
) from e
|
||||
self._client = httpx.AsyncClient(timeout=30.0)
|
||||
|
||||
client = self._client
|
||||
assert client is not None
|
||||
resp = await client.post(
|
||||
f"{self._api_url}/v1/saas/compress",
|
||||
headers={
|
||||
"X-Headroom-Key": self._api_key,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
content=json.dumps(
|
||||
{
|
||||
"messages": messages,
|
||||
"model": model or "claude-sonnet-4-5-20250929",
|
||||
"model_limit": self._model_limit,
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
if resp.status_code != 200:
|
||||
logger.warning("Headroom Cloud API error: %d %s", resp.status_code, resp.text[:200])
|
||||
return None
|
||||
|
||||
result: dict[str, Any] = resp.json()
|
||||
return result
|
||||
|
||||
@@ -16,16 +16,18 @@ Cloud mode requires httpx: pip install httpx
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
# _DEFAULT_CLOUD_URL and guard_egress stay importable from this module.
|
||||
from headroom.integrations._compress_backend import ( # noqa: F401
|
||||
_DEFAULT_CLOUD_URL,
|
||||
CompressBackend,
|
||||
)
|
||||
from headroom.offline import guard_egress # noqa: F401
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
|
||||
|
||||
|
||||
try:
|
||||
from litellm.integrations.custom_logger import CustomLogger as _CustomLogger
|
||||
@@ -33,7 +35,7 @@ except ImportError: # litellm not installed — fall back to plain object
|
||||
_CustomLogger = object # type: ignore[assignment,misc]
|
||||
|
||||
|
||||
class HeadroomCallback(_CustomLogger):
|
||||
class HeadroomCallback(CompressBackend, _CustomLogger):
|
||||
"""LiteLLM callback that compresses messages before each API call.
|
||||
|
||||
Implements the LiteLLM callback hooks looked up by name:
|
||||
@@ -63,6 +65,8 @@ class HeadroomCallback(_CustomLogger):
|
||||
HEADROOM_API_KEY: "hdr_xxx"
|
||||
"""
|
||||
|
||||
_log = logger
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
min_tokens: int = 500,
|
||||
@@ -73,30 +77,14 @@ class HeadroomCallback(_CustomLogger):
|
||||
) -> None:
|
||||
super().__init__()
|
||||
self._min_tokens = min_tokens
|
||||
self._model_limit = model_limit
|
||||
self._hooks = hooks
|
||||
self._total_saved = 0
|
||||
|
||||
# Cloud mode: if api_key is set, compress via Headroom Cloud API
|
||||
# Falls back to HEADROOM_API_KEY env var
|
||||
import os
|
||||
|
||||
self._api_key = api_key or os.environ.get("HEADROOM_API_KEY", "").strip() or None
|
||||
self._api_url = (
|
||||
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
|
||||
).rstrip("/")
|
||||
self._client: Any = None # Lazy-initialized httpx.AsyncClient
|
||||
self._init_compress_backend(model_limit, hooks, api_key, api_url)
|
||||
|
||||
@property
|
||||
def total_tokens_saved(self) -> int:
|
||||
"""Total tokens saved across all calls."""
|
||||
return self._total_saved
|
||||
|
||||
@property
|
||||
def cloud_mode(self) -> bool:
|
||||
"""Whether cloud compression is enabled."""
|
||||
return self._api_key is not None
|
||||
|
||||
async def aclose(self) -> None:
|
||||
"""Close the shared cloud HTTP client, if it was initialized.
|
||||
|
||||
@@ -167,70 +155,6 @@ class HeadroomCallback(_CustomLogger):
|
||||
|
||||
return data
|
||||
|
||||
def _local_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
|
||||
"""Compress locally using headroom.compress()."""
|
||||
from headroom.compress import compress
|
||||
|
||||
result = compress(
|
||||
messages=messages,
|
||||
model=model or "claude-sonnet-4-5-20250929",
|
||||
model_limit=self._model_limit,
|
||||
hooks=self._hooks,
|
||||
)
|
||||
return {
|
||||
"messages": result.messages,
|
||||
"tokens_before": result.tokens_before,
|
||||
"tokens_after": result.tokens_after,
|
||||
"tokens_saved": result.tokens_saved,
|
||||
"compression_ratio": result.compression_ratio,
|
||||
}
|
||||
|
||||
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
|
||||
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
|
||||
|
||||
This is the one path in this file that puts the caller's prompt
|
||||
content on the wire to a Headroom-operated host, so it is exactly what
|
||||
HEADROOM_OFFLINE exists to stop. "Opt-in by configuration" was the old
|
||||
reason for leaving it open, and it is not good enough: an operator who
|
||||
sets an air-gap switch is overriding earlier configuration on purpose.
|
||||
The refusal is loud rather than a silent fall-through to local
|
||||
compression, because silently compressing locally would hide the fact
|
||||
that the deployment is no longer doing what it was configured to do.
|
||||
"""
|
||||
guard_egress("Headroom Cloud compression", self._api_url)
|
||||
if self._client is None:
|
||||
try:
|
||||
import httpx
|
||||
except ImportError as e:
|
||||
raise ImportError(
|
||||
"httpx is required for Headroom Cloud mode: pip install httpx"
|
||||
) from e
|
||||
self._client = httpx.AsyncClient(timeout=30.0)
|
||||
|
||||
client = self._client
|
||||
assert client is not None
|
||||
resp = await client.post(
|
||||
f"{self._api_url}/v1/saas/compress",
|
||||
headers={
|
||||
"X-Headroom-Key": self._api_key,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
content=json.dumps(
|
||||
{
|
||||
"messages": messages,
|
||||
"model": model or "claude-sonnet-4-5-20250929",
|
||||
"model_limit": self._model_limit,
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
if resp.status_code != 200:
|
||||
logger.warning("Headroom Cloud API error: %d %s", resp.status_code, resp.text[:200])
|
||||
return None
|
||||
|
||||
result: dict[str, Any] = resp.json()
|
||||
return result
|
||||
|
||||
async def async_post_call_success_hook(
|
||||
self,
|
||||
data: dict[str, Any],
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
"""Cloud path of the compression backend shared by the ASGI and LiteLLM integrations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from headroom.integrations.asgi import CompressionMiddleware
|
||||
from headroom.integrations.litellm_callback import HeadroomCallback
|
||||
|
||||
_MESSAGES = [{"role": "user", "content": "secret prompt text"}]
|
||||
|
||||
|
||||
@pytest.fixture(params=["asgi", "litellm"])
|
||||
def backend(request, monkeypatch: pytest.MonkeyPatch):
|
||||
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
||||
if request.param == "asgi":
|
||||
return CompressionMiddleware(app=None, api_key="hdr_test", api_url="https://cloud.test/")
|
||||
return HeadroomCallback(api_key="hdr_test", api_url="https://cloud.test/")
|
||||
|
||||
|
||||
def _serve(backend, handler) -> list[httpx.Request]:
|
||||
seen: list[httpx.Request] = []
|
||||
|
||||
def record(request: httpx.Request) -> httpx.Response:
|
||||
seen.append(request)
|
||||
return handler(request)
|
||||
|
||||
backend._client = httpx.AsyncClient(transport=httpx.MockTransport(record))
|
||||
return seen
|
||||
|
||||
|
||||
def test_cloud_success_returns_the_service_result(backend) -> None:
|
||||
payload = {"messages": [{"role": "user", "content": "short"}], "tokens_saved": 3}
|
||||
seen = _serve(backend, lambda request: httpx.Response(200, json=payload))
|
||||
|
||||
result = asyncio.run(backend._cloud_compress(_MESSAGES, "gpt-4o"))
|
||||
|
||||
assert result == payload
|
||||
(request,) = seen
|
||||
assert str(request.url) == "https://cloud.test/v1/saas/compress"
|
||||
assert request.headers["X-Headroom-Key"] == "hdr_test"
|
||||
assert json.loads(request.content) == {
|
||||
"messages": _MESSAGES,
|
||||
"model": "gpt-4o",
|
||||
"model_limit": backend._model_limit,
|
||||
}
|
||||
|
||||
|
||||
class _Records(logging.Handler):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(logging.DEBUG)
|
||||
self.records: list[logging.LogRecord] = []
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
self.records.append(record)
|
||||
|
||||
|
||||
_PROMPT_CANARY = "PROMPT-CANARY-7f3e"
|
||||
_KEY_CANARY = "hdr_KEY-CANARY-91ab"
|
||||
_URL_CANARY = "URL-CANARY-c0de"
|
||||
_PATH_CANARY = "PRIVATE_PATH_CANARY"
|
||||
|
||||
# (api_url, how the WARNING shows it); every canary must stay out of every record.
|
||||
_API_URLS = {
|
||||
"query": (f"https://cloud.test/?token={_URL_CANARY}", "https://cloud.test/?<redacted>"),
|
||||
"path": (f"https://cloud.test/bot{_PATH_CANARY}", "https://cloud.test/<path>"),
|
||||
# %50 is "P": the encoded form must not slip through either.
|
||||
"encoded-path": ("https://cloud.test/bot%50RIVATE_PATH_CANARY", "https://cloud.test/<path>"),
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind", ["asgi", "litellm"])
|
||||
def test_real_cloud_error_media_type_cannot_echo_credentials(kind, monkeypatch) -> None:
|
||||
canary = "PRIVATE_CREDENTIAL_CANARY"
|
||||
body = b"service error"
|
||||
received = []
|
||||
|
||||
class ErrorServer(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
received.append(json.loads(self.rfile.read(int(self.headers["Content-Length"]))))
|
||||
self.send_response(503)
|
||||
self.send_header("Content-Type", f"text/{canary}")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
with HTTPServer(("127.0.0.1", 0), ErrorServer) as server:
|
||||
worker = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
worker.start()
|
||||
api_url = f"http://127.0.0.1:{server.server_port}"
|
||||
integration = CompressionMiddleware if kind == "asgi" else HeadroomCallback
|
||||
kwargs = {"app": None} if kind == "asgi" else {}
|
||||
instance = integration(api_key=_KEY_CANARY, api_url=api_url, **kwargs)
|
||||
handler = _Records()
|
||||
monkeypatch.setattr(instance._log, "level", logging.DEBUG)
|
||||
monkeypatch.setattr(instance._log, "disabled", False)
|
||||
instance._log.addHandler(handler)
|
||||
|
||||
async def request():
|
||||
try:
|
||||
return await instance._cloud_compress(_MESSAGES, "gpt-4o")
|
||||
finally:
|
||||
if instance._client is not None:
|
||||
await instance._client.aclose()
|
||||
|
||||
try:
|
||||
assert asyncio.run(request()) is None
|
||||
finally:
|
||||
instance._log.removeHandler(handler)
|
||||
server.shutdown()
|
||||
worker.join(timeout=5)
|
||||
assert received[0]["messages"] == _MESSAGES
|
||||
records = [logging.Formatter().format(record) for record in handler.records]
|
||||
assert any("HTTP 503" in record for record in records)
|
||||
assert any(f"{len(body)} bytes" in record for record in records)
|
||||
assert all(canary.lower() not in record.lower() for record in records)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind", ["asgi", "litellm"])
|
||||
@pytest.mark.parametrize("level", [logging.DEBUG, logging.WARNING])
|
||||
@pytest.mark.parametrize("url_case", sorted(_API_URLS))
|
||||
def test_cloud_error_logs_no_echoed_content_at_any_level(
|
||||
kind: str, level: int, url_case: str, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""An error response echoing the prompt must not put it, the key, or URL
|
||||
credentials (query or path) in any formatted record, with DEBUG on or off."""
|
||||
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
api_url, shown_url = _API_URLS[url_case]
|
||||
if kind == "asgi":
|
||||
backend = CompressionMiddleware(app=None, api_key=_KEY_CANARY, api_url=api_url)
|
||||
else:
|
||||
backend = HeadroomCallback(api_key=_KEY_CANARY, api_url=api_url)
|
||||
messages = [{"role": "user", "content": f"my secret is {_PROMPT_CANARY}"}]
|
||||
echo = f"bad request: {json.dumps(messages)} key={_KEY_CANARY}"
|
||||
_serve(
|
||||
backend,
|
||||
lambda request: httpx.Response(
|
||||
503,
|
||||
text=echo,
|
||||
# A header can echo content too: only the bare media type may be logged.
|
||||
headers={"content-type": f"text/plain; note={_PROMPT_CANARY}"},
|
||||
),
|
||||
)
|
||||
# Attach to the module logger itself: proxy startup elsewhere in the suite can
|
||||
# stop headroom.* records from reaching pytest's root-level caplog handler.
|
||||
handler = _Records()
|
||||
monkeypatch.setattr(backend._log, "level", level)
|
||||
monkeypatch.setattr(backend._log, "disabled", False)
|
||||
backend._log.addHandler(handler)
|
||||
try:
|
||||
assert asyncio.run(backend._cloud_compress(messages, "gpt-4o")) is None
|
||||
finally:
|
||||
backend._log.removeHandler(handler)
|
||||
|
||||
formatted = [logging.Formatter().format(r) for r in handler.records]
|
||||
warnings = [r.getMessage() for r in handler.records if r.levelno == logging.WARNING]
|
||||
assert warnings == [
|
||||
f"Headroom Cloud API error: HTTP 503 from {shown_url}; request sent uncompressed"
|
||||
]
|
||||
if level == logging.DEBUG:
|
||||
assert any("content-type=text/plain," in line for line in formatted)
|
||||
for line in formatted:
|
||||
for canary in (_PROMPT_CANARY, _KEY_CANARY, _URL_CANARY, _PATH_CANARY, "%50RIVATE"):
|
||||
assert canary not in line
|
||||
@@ -3,26 +3,44 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
_INTEGRATIONS = Path(__file__).resolve().parents[2] / "headroom" / "integrations"
|
||||
_BACKEND = "headroom.integrations._compress_backend"
|
||||
|
||||
def _import_callback() -> type:
|
||||
# Import the module directly to avoid triggering headroom/integrations/__init__.py
|
||||
# which pulls in langchain and the native .so extension.
|
||||
module_path = (
|
||||
Path(__file__).resolve().parents[2] / "headroom" / "integrations" / "litellm_callback.py"
|
||||
)
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"headroom.integrations.litellm_callback",
|
||||
module_path,
|
||||
)
|
||||
|
||||
def _load_file(name: str, filename: str) -> Any:
|
||||
spec = importlib.util.spec_from_file_location(name, _INTEGRATIONS / filename)
|
||||
assert spec is not None and spec.loader is not None
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod) # type: ignore[union-attr]
|
||||
return mod.HeadroomCallback # type: ignore[attr-defined]
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def _import_callback() -> type:
|
||||
# Load the module files directly to avoid triggering headroom/integrations/__init__.py,
|
||||
# which pulls in langchain and the native .so extension. litellm_callback imports
|
||||
# _compress_backend by its dotted name, so register a file-loaded copy for that one
|
||||
# import; with the name already in sys.modules, the integrations package is not
|
||||
# imported. Both files still import the top-level headroom package and its
|
||||
# stdlib-only helpers (headroom.offline, headroom.log_safety); the subprocess test
|
||||
# below checks that neither langchain nor headroom._core is loaded.
|
||||
saved = sys.modules.get(_BACKEND)
|
||||
sys.modules[_BACKEND] = _load_file(_BACKEND, "_compress_backend.py")
|
||||
try:
|
||||
mod = _load_file("headroom.integrations.litellm_callback", "litellm_callback.py")
|
||||
finally:
|
||||
if saved is None:
|
||||
del sys.modules[_BACKEND]
|
||||
else:
|
||||
sys.modules[_BACKEND] = saved
|
||||
return mod.HeadroomCallback # type: ignore[no-any-return]
|
||||
|
||||
|
||||
HeadroomCallback = _import_callback()
|
||||
@@ -68,3 +86,20 @@ class TestHeadroomCallbackClientLifecycle:
|
||||
await cb.aclose()
|
||||
|
||||
assert cb._client is None
|
||||
|
||||
|
||||
def test_direct_load_does_not_import_the_integrations_package() -> None:
|
||||
"""The loader above must keep this file isolated from LangChain, the native
|
||||
extension and the integrations package; check it in a fresh interpreter."""
|
||||
code = (
|
||||
"import importlib.util, sys\n"
|
||||
f"spec = importlib.util.spec_from_file_location('t', {str(Path(__file__))!r})\n"
|
||||
"mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)\n"
|
||||
"bad = [m for m in sys.modules if m in ('headroom.integrations', 'headroom._core')"
|
||||
" or m.startswith('langchain')]\n"
|
||||
"print(bad)\n"
|
||||
)
|
||||
out = subprocess.run(
|
||||
[sys.executable, "-c", code], capture_output=True, text=True, check=True, timeout=120
|
||||
)
|
||||
assert out.stdout.strip() == "[]", out.stdout + out.stderr
|
||||
|
||||
Reference in New Issue
Block a user