refactor(integrations): share the local/cloud compress backend of asgi and litellm_callback (#4124)

## Description

`CompressionMiddleware` (`headroom/integrations/asgi.py`) and
`HeadroomCallback` (`headroom/integrations/litellm_callback.py`) carried
identical copies of `_local_compress` and `_cloud_compress`; `diff` of
the two method blocks reported no difference. The cloud URL constant,
the `HEADROOM_API_KEY` / `HEADROOM_API_URL` reads and `cloud_mode` were
copied too.

This PR moves them into a `CompressBackend` mixin in
`headroom/integrations/_compress_backend.py`. That module imports
neither starlette nor litellm, so each integration still works without
the other's dependency.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring

## Changes Made

- New `headroom/integrations/_compress_backend.py` with
`CompressBackend`, which provides:
  - `_init_compress_backend(model_limit, hooks, api_key, api_url)`;
  - `cloud_mode`;
  - `_local_compress`;
  - `_cloud_compress`, which keeps the `guard_egress` offline check.
- `CompressionMiddleware(CompressBackend)` and
`HeadroomCallback(CompressBackend, CustomLogger)` call
`_init_compress_backend` and drop their copies.
- The method names stay, because
`tests/test_offline_egress_chokepoint.py` calls `_cloud_compress`
directly.
- Each class sets `_log` to its own module logger, so the "Headroom
Cloud API error" warning keeps its logger name
(`headroom.integrations.asgi` /
`headroom.integrations.litellm_callback`).
- One log change: that WARNING used to include the first 200 characters
of the cloud response body, which can echo request content. It now reads
`Headroom Cloud API error: HTTP <status> from <api_url>; request sent
uncompressed`, with the URL passed through `redact_url` from
`headroom/log_safety.py` (#4138): scheme, host and port only, the path
shown as `/<path>` and any query as `?<redacted>`. The body is not
logged at any level: a separate DEBUG line gives only the bare media
type (`type/subtype`, header parameters dropped) and the body length.
- `_DEFAULT_CLOUD_URL` and `guard_egress` stay importable from both
modules.
- Left alone on purpose:
- each class's `aclose`: they differ slightly in when `_client` is
cleared;
- `min_tokens`: it is stored and never read in either class, as on main.
The maintainer should decide separately whether to wire it up.
- No `CustomLogger` attribute clashes with the mixin's names. I checked
`dir(CustomLogger)` and the instance `vars()`.
- Tests:
- New `tests/test_integrations/test_compress_backend.py`, parametrized
over `CompressionMiddleware` and `HeadroomCallback` with an
`httpx.MockTransport` client: a 200 returns the service result (URL,
`X-Headroom-Key` header and JSON body checked), and a 503 whose body
echoes the prompt returns `None`.
`test_cloud_error_logs_no_echoed_content_at_any_level` runs both
integrations with the logger at DEBUG and at WARNING, formats every
record with `logging.Formatter().format`, and asserts that prompt,
API-key, URL-query and URL-path canaries (plain and `%`-encoded) are
absent, including a prompt canary echoed in the `Content-Type` header.
It fails on 95c608ee; the path cases fail before #4138's b566a758, and
the header case fails on 4fa6bb34.
`test_direct_load_does_not_import_the_integrations_package` now also
checks that `headroom._core` is not loaded, and the fixture comment
names what the loaded files import.
- `tests/test_integrations/test_litellm_callback.py`: the direct-file
loader registers a file-loaded `_compress_backend` before loading
`litellm_callback.py`, so the test again avoids importing
`headroom.integrations` and LangChain. New
`test_direct_load_does_not_import_the_integrations_package` checks that
in a fresh interpreter.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compress_api.py tests/test_litellm_callback.py tests/test_integrations/test_litellm_callback.py
origin/main: 37 passed     this branch: 37 passed

$ LITELLM_LOCAL_MODEL_COST_MAP=True pytest tests/test_offline_egress_chokepoint.py -k "asgi or litellm"
origin/main: 2 passed      this branch: 2 passed

$ pytest tests/test_offline_egress_chokepoint.py
origin/main: 2 failed, 141 passed     this branch: 2 failed, 141 passed
(pre-existing and environmental: the same two asgi/litellm tests fail on main, because importing litellm
 fetches its model cost map from raw.githubusercontent.com and the test's no-socket fixture refuses it;
 with LITELLM_LOCAL_MODEL_COST_MAP=True both pass on main and on this branch)

$ pytest tests/test_integrations/test_compress_backend.py tests/test_integrations/test_litellm_callback.py tests/test_compress_api.py tests/test_litellm_callback.py tests/test_offline_egress_chokepoint.py
this branch: 185 passed
# test_compress_backend.py against the previous commit's backend (04d38103): 2 failed (warning carried the body), 2 passed

$ uvx ruff@0.16.8 check / format <touched files>  -> All checks passed! / unchanged
$ mypy headroom/integrations/_compress_backend.py asgi.py litellm_callback.py --ignore-missing-imports
headroom/integrations/langchain/chat_model.py:394: error: ... [truthy-function]   (pre-existing, followed import, untouched file)
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv` (litellm, starlette and
httpx installed), this branch's worktree. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`.
- Exact command / steps: two runs, listed below.
1. Shared smoke harness: a real proxy from this checkout against a local
fake upstream (`smoke.sh <worktree>`).
2. The shared cloud path over real HTTP, with both module loggers at
DEBUG: a local HTTP server answers 503 (echoing the request, which
carries a prompt canary) and then 200, for `HeadroomCallback` and then
`CompressionMiddleware` (`_cloud_compress`). The API key, a `/bot…` path
segment and a `?token=` query on the URL are canaries too.
- Observed result: the output of both runs is in the two blocks below.

```text
branch: cleanup/cli-integrations-shared-compress-backend @ fad5714c
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-58395.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: <tmp>)
```

```text
HeadroomCallback 503 -> None | 200 -> 3
CompressionMiddleware 503 -> None | 200 -> 3
WARNING headroom.integrations.litellm_callback: Headroom Cloud API error: HTTP 503 from http://127.0.0.1:PORT/<path>?<redacted>; request sent uncompressed
DEBUG headroom.integrations.litellm_callback: Headroom Cloud API error response: content-type=text/plain, 138 bytes
WARNING headroom.integrations.asgi: Headroom Cloud API error: HTTP 503 from http://127.0.0.1:PORT/<path>?<redacted>; request sent uncompressed
DEBUG headroom.integrations.asgi: Headroom Cloud API error response: content-type=text/plain, 138 bytes
canaries in log: 0
```

- Not tested: the real Headroom Cloud service.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, library refactor.
- Stable/default behavior changed: no. The env vars, default URL,
payload, timeout, offline refusal and log text are all the same.
- Kill switch / disable path: n/a.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Depends on #4138; its two files show in this diff until it merges.

- `tests/test_integrations/test_litellm_callback.py` loads
`litellm_callback.py` from its file path, to avoid importing the
`headroom.integrations` package. Because the module now imports
`headroom.integrations._compress_backend`, the loader registers a
file-loaded copy of that module first; a module already in `sys.modules`
is returned without importing its parent package. Normal imports of
`headroom.integrations.litellm_callback` load the package anyway, as
before.
- A test that patched `headroom.integrations.asgi.guard_egress` would no
longer intercept the call, because the call now lives in
`_compress_backend`. No such test exists.
- Docs impact (the changed warning text) is recorded in the docs ledger;
there are no doc edits in this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-16 14:30:57 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 4e6e955cad
commit 58caafb59b
5 changed files with 396 additions and 181 deletions
+152
View File
@@ -0,0 +1,152 @@
"""Local or Headroom Cloud compression, shared by the ASGI and LiteLLM integrations.
``CompressionMiddleware`` (asgi.py) and ``HeadroomCallback`` (litellm_callback.py)
compress the same way: in-process with ``headroom.compress()`` by default, or
through the Headroom Cloud API when an API key is set.
This module must not import starlette or litellm, so either integration can be
used without the other's dependency.
"""
from __future__ import annotations
import json
import logging
import os
from typing import Any
from headroom.log_safety import redact_url
from headroom.offline import guard_egress
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
_DEFAULT_MODEL = "claude-sonnet-4-5-20250929"
_KNOWN_MEDIA_TYPES = frozenset(
{
"application/json",
"application/problem+json",
"application/octet-stream",
"text/plain",
"text/html",
}
)
def _media_type(content_type: str) -> str:
"""Return just the ``type/subtype`` of a Content-Type header, or ``<other>``.
Parameters and anything that is not a plain media type are dropped, so an
error response cannot smuggle echoed content into the log through a header.
"""
# Valid type/subtype syntax can still contain echoed private content.
media = content_type.split(";", 1)[0].strip().lower()
return media if media in _KNOWN_MEDIA_TYPES else "<other>"
class CompressBackend:
"""Mixin that picks local or cloud compression and holds the cloud client.
Subclasses call ``_init_compress_backend`` from ``__init__`` and set
``_log`` to their module logger, so log lines keep that logger's name.
"""
_log: logging.Logger
def _init_compress_backend(
self,
model_limit: int,
hooks: Any,
api_key: str | None,
api_url: str | None,
) -> None:
self._model_limit = model_limit
self._hooks = hooks
# Cloud mode: if api_key is set, compress via Headroom Cloud API
# Falls back to HEADROOM_API_KEY env var
self._api_key = api_key or os.environ.get("HEADROOM_API_KEY", "").strip() or None
self._api_url = (
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
).rstrip("/")
self._client: Any = None # Lazy-initialized httpx.AsyncClient
@property
def cloud_mode(self) -> bool:
"""Whether cloud compression is enabled."""
return self._api_key is not None
def _local_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
"""Compress locally using headroom.compress()."""
from headroom.compress import compress
result = compress(
messages=messages,
model=model or _DEFAULT_MODEL,
model_limit=self._model_limit,
hooks=self._hooks,
)
return {
"messages": result.messages,
"tokens_before": result.tokens_before,
"tokens_after": result.tokens_after,
"tokens_saved": result.tokens_saved,
"compression_ratio": result.compression_ratio,
}
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
This is the one path in these integrations that puts the caller's
prompt content on the wire to a Headroom-operated host, so it is
exactly what HEADROOM_OFFLINE exists to stop. "Opt-in by configuration"
was the old reason for leaving it open, and it is not good enough: an
operator who sets an air-gap switch is overriding earlier configuration
on purpose. The refusal is loud rather than a silent fall-through to
local compression, because silently compressing locally would hide the
fact that the deployment is no longer doing what it was configured to do.
"""
guard_egress("Headroom Cloud compression", self._api_url)
if self._client is None:
try:
import httpx
except ImportError as e:
raise ImportError(
"httpx is required for Headroom Cloud mode: pip install httpx"
) from e
self._client = httpx.AsyncClient(timeout=30.0)
client = self._client
assert client is not None
resp = await client.post(
f"{self._api_url}/v1/saas/compress",
headers={
"X-Headroom-Key": self._api_key,
"Content-Type": "application/json",
},
content=json.dumps(
{
"messages": messages,
"model": model or _DEFAULT_MODEL,
"model_limit": self._model_limit,
}
),
)
if resp.status_code != 200:
# The response body can echo request content: log metadata only, at
# every level.
self._log.warning(
"Headroom Cloud API error: HTTP %d from %s; request sent uncompressed",
resp.status_code,
redact_url(self._api_url),
)
self._log.debug(
"Headroom Cloud API error response: content-type=%s, %d bytes",
_media_type(resp.headers.get("content-type", "")),
len(resp.content),
)
return None
result: dict[str, Any] = resp.json()
return result
+10 -83
View File
@@ -28,18 +28,20 @@ from __future__ import annotations
import json
import logging
import os
from collections.abc import MutableMapping
from typing import Any
from starlette.types import ASGIApp, Receive, Scope, Send
from headroom.offline import guard_egress
# _DEFAULT_CLOUD_URL and guard_egress stay importable from this module.
from headroom.integrations._compress_backend import ( # noqa: F401
_DEFAULT_CLOUD_URL,
CompressBackend,
)
from headroom.offline import guard_egress # noqa: F401
logger = logging.getLogger(__name__)
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
# Paths that contain LLM messages to compress
_LLM_PATHS = (
"/v1/messages", # Anthropic
@@ -49,7 +51,7 @@ _LLM_PATHS = (
)
class CompressionMiddleware:
class CompressionMiddleware(CompressBackend):
"""ASGI middleware that compresses LLM request messages.
Two modes:
@@ -64,6 +66,8 @@ class CompressionMiddleware:
- x-headroom-compressed: "true" if compression occurred
"""
_log = logger
def __init__(
self,
app: ASGIApp,
@@ -75,20 +79,7 @@ class CompressionMiddleware:
) -> None:
self.app = app
self._min_tokens = min_tokens
self._model_limit = model_limit
self._hooks = hooks
# Cloud mode: if api_key is set, compress via Headroom Cloud API
self._api_key = api_key or os.environ.get("HEADROOM_API_KEY", "").strip() or None
self._api_url = (
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
).rstrip("/")
self._client: Any = None # Lazy-initialized httpx.AsyncClient
@property
def cloud_mode(self) -> bool:
"""Whether cloud compression is enabled."""
return self._api_key is not None
self._init_compress_backend(model_limit, hooks, api_key, api_url)
async def aclose(self) -> None:
"""Close the underlying httpx.AsyncClient, if one was created."""
@@ -186,67 +177,3 @@ class CompressionMiddleware:
await send(message)
await self.app(scope, modified_receive, metrics_send)
def _local_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
"""Compress locally using headroom.compress()."""
from headroom.compress import compress
result = compress(
messages=messages,
model=model or "claude-sonnet-4-5-20250929",
model_limit=self._model_limit,
hooks=self._hooks,
)
return {
"messages": result.messages,
"tokens_before": result.tokens_before,
"tokens_after": result.tokens_after,
"tokens_saved": result.tokens_saved,
"compression_ratio": result.compression_ratio,
}
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
This is the one path in this file that puts the caller's prompt
content on the wire to a Headroom-operated host, so it is exactly what
HEADROOM_OFFLINE exists to stop. "Opt-in by configuration" was the old
reason for leaving it open, and it is not good enough: an operator who
sets an air-gap switch is overriding earlier configuration on purpose.
The refusal is loud rather than a silent fall-through to local
compression, because silently compressing locally would hide the fact
that the deployment is no longer doing what it was configured to do.
"""
guard_egress("Headroom Cloud compression", self._api_url)
if self._client is None:
try:
import httpx
except ImportError as e:
raise ImportError(
"httpx is required for Headroom Cloud mode: pip install httpx"
) from e
self._client = httpx.AsyncClient(timeout=30.0)
client = self._client
assert client is not None
resp = await client.post(
f"{self._api_url}/v1/saas/compress",
headers={
"X-Headroom-Key": self._api_key,
"Content-Type": "application/json",
},
content=json.dumps(
{
"messages": messages,
"model": model or "claude-sonnet-4-5-20250929",
"model_limit": self._model_limit,
}
),
)
if resp.status_code != 200:
logger.warning("Headroom Cloud API error: %d %s", resp.status_code, resp.text[:200])
return None
result: dict[str, Any] = resp.json()
return result
+10 -86
View File
@@ -16,16 +16,18 @@ Cloud mode requires httpx: pip install httpx
from __future__ import annotations
import json
import logging
from typing import Any
from headroom.offline import guard_egress
# _DEFAULT_CLOUD_URL and guard_egress stay importable from this module.
from headroom.integrations._compress_backend import ( # noqa: F401
_DEFAULT_CLOUD_URL,
CompressBackend,
)
from headroom.offline import guard_egress # noqa: F401
logger = logging.getLogger(__name__)
_DEFAULT_CLOUD_URL = "https://api.headroomlabs.ai"
try:
from litellm.integrations.custom_logger import CustomLogger as _CustomLogger
@@ -33,7 +35,7 @@ except ImportError: # litellm not installed — fall back to plain object
_CustomLogger = object # type: ignore[assignment,misc]
class HeadroomCallback(_CustomLogger):
class HeadroomCallback(CompressBackend, _CustomLogger):
"""LiteLLM callback that compresses messages before each API call.
Implements the LiteLLM callback hooks looked up by name:
@@ -63,6 +65,8 @@ class HeadroomCallback(_CustomLogger):
HEADROOM_API_KEY: "hdr_xxx"
"""
_log = logger
def __init__(
self,
min_tokens: int = 500,
@@ -73,30 +77,14 @@ class HeadroomCallback(_CustomLogger):
) -> None:
super().__init__()
self._min_tokens = min_tokens
self._model_limit = model_limit
self._hooks = hooks
self._total_saved = 0
# Cloud mode: if api_key is set, compress via Headroom Cloud API
# Falls back to HEADROOM_API_KEY env var
import os
self._api_key = api_key or os.environ.get("HEADROOM_API_KEY", "").strip() or None
self._api_url = (
api_url or os.environ.get("HEADROOM_API_URL", "").strip() or _DEFAULT_CLOUD_URL
).rstrip("/")
self._client: Any = None # Lazy-initialized httpx.AsyncClient
self._init_compress_backend(model_limit, hooks, api_key, api_url)
@property
def total_tokens_saved(self) -> int:
"""Total tokens saved across all calls."""
return self._total_saved
@property
def cloud_mode(self) -> bool:
"""Whether cloud compression is enabled."""
return self._api_key is not None
async def aclose(self) -> None:
"""Close the shared cloud HTTP client, if it was initialized.
@@ -167,70 +155,6 @@ class HeadroomCallback(_CustomLogger):
return data
def _local_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
"""Compress locally using headroom.compress()."""
from headroom.compress import compress
result = compress(
messages=messages,
model=model or "claude-sonnet-4-5-20250929",
model_limit=self._model_limit,
hooks=self._hooks,
)
return {
"messages": result.messages,
"tokens_before": result.tokens_before,
"tokens_after": result.tokens_after,
"tokens_saved": result.tokens_saved,
"compression_ratio": result.compression_ratio,
}
async def _cloud_compress(self, messages: list[dict], model: str) -> dict[str, Any] | None:
"""Compress via Headroom Cloud API (managed CCR, TOIN, analytics).
This is the one path in this file that puts the caller's prompt
content on the wire to a Headroom-operated host, so it is exactly what
HEADROOM_OFFLINE exists to stop. "Opt-in by configuration" was the old
reason for leaving it open, and it is not good enough: an operator who
sets an air-gap switch is overriding earlier configuration on purpose.
The refusal is loud rather than a silent fall-through to local
compression, because silently compressing locally would hide the fact
that the deployment is no longer doing what it was configured to do.
"""
guard_egress("Headroom Cloud compression", self._api_url)
if self._client is None:
try:
import httpx
except ImportError as e:
raise ImportError(
"httpx is required for Headroom Cloud mode: pip install httpx"
) from e
self._client = httpx.AsyncClient(timeout=30.0)
client = self._client
assert client is not None
resp = await client.post(
f"{self._api_url}/v1/saas/compress",
headers={
"X-Headroom-Key": self._api_key,
"Content-Type": "application/json",
},
content=json.dumps(
{
"messages": messages,
"model": model or "claude-sonnet-4-5-20250929",
"model_limit": self._model_limit,
}
),
)
if resp.status_code != 200:
logger.warning("Headroom Cloud API error: %d %s", resp.status_code, resp.text[:200])
return None
result: dict[str, Any] = resp.json()
return result
async def async_post_call_success_hook(
self,
data: dict[str, Any],
@@ -0,0 +1,177 @@
"""Cloud path of the compression backend shared by the ASGI and LiteLLM integrations."""
from __future__ import annotations
import asyncio
import json
import logging
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
import httpx
import pytest
from headroom.integrations.asgi import CompressionMiddleware
from headroom.integrations.litellm_callback import HeadroomCallback
_MESSAGES = [{"role": "user", "content": "secret prompt text"}]
@pytest.fixture(params=["asgi", "litellm"])
def backend(request, monkeypatch: pytest.MonkeyPatch):
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
if request.param == "asgi":
return CompressionMiddleware(app=None, api_key="hdr_test", api_url="https://cloud.test/")
return HeadroomCallback(api_key="hdr_test", api_url="https://cloud.test/")
def _serve(backend, handler) -> list[httpx.Request]:
seen: list[httpx.Request] = []
def record(request: httpx.Request) -> httpx.Response:
seen.append(request)
return handler(request)
backend._client = httpx.AsyncClient(transport=httpx.MockTransport(record))
return seen
def test_cloud_success_returns_the_service_result(backend) -> None:
payload = {"messages": [{"role": "user", "content": "short"}], "tokens_saved": 3}
seen = _serve(backend, lambda request: httpx.Response(200, json=payload))
result = asyncio.run(backend._cloud_compress(_MESSAGES, "gpt-4o"))
assert result == payload
(request,) = seen
assert str(request.url) == "https://cloud.test/v1/saas/compress"
assert request.headers["X-Headroom-Key"] == "hdr_test"
assert json.loads(request.content) == {
"messages": _MESSAGES,
"model": "gpt-4o",
"model_limit": backend._model_limit,
}
class _Records(logging.Handler):
def __init__(self) -> None:
super().__init__(logging.DEBUG)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self.records.append(record)
_PROMPT_CANARY = "PROMPT-CANARY-7f3e"
_KEY_CANARY = "hdr_KEY-CANARY-91ab"
_URL_CANARY = "URL-CANARY-c0de"
_PATH_CANARY = "PRIVATE_PATH_CANARY"
# (api_url, how the WARNING shows it); every canary must stay out of every record.
_API_URLS = {
"query": (f"https://cloud.test/?token={_URL_CANARY}", "https://cloud.test/?<redacted>"),
"path": (f"https://cloud.test/bot{_PATH_CANARY}", "https://cloud.test/<path>"),
# %50 is "P": the encoded form must not slip through either.
"encoded-path": ("https://cloud.test/bot%50RIVATE_PATH_CANARY", "https://cloud.test/<path>"),
}
@pytest.mark.parametrize("kind", ["asgi", "litellm"])
def test_real_cloud_error_media_type_cannot_echo_credentials(kind, monkeypatch) -> None:
canary = "PRIVATE_CREDENTIAL_CANARY"
body = b"service error"
received = []
class ErrorServer(BaseHTTPRequestHandler):
def do_POST(self):
received.append(json.loads(self.rfile.read(int(self.headers["Content-Length"]))))
self.send_response(503)
self.send_header("Content-Type", f"text/{canary}")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
with HTTPServer(("127.0.0.1", 0), ErrorServer) as server:
worker = threading.Thread(target=server.serve_forever, daemon=True)
worker.start()
api_url = f"http://127.0.0.1:{server.server_port}"
integration = CompressionMiddleware if kind == "asgi" else HeadroomCallback
kwargs = {"app": None} if kind == "asgi" else {}
instance = integration(api_key=_KEY_CANARY, api_url=api_url, **kwargs)
handler = _Records()
monkeypatch.setattr(instance._log, "level", logging.DEBUG)
monkeypatch.setattr(instance._log, "disabled", False)
instance._log.addHandler(handler)
async def request():
try:
return await instance._cloud_compress(_MESSAGES, "gpt-4o")
finally:
if instance._client is not None:
await instance._client.aclose()
try:
assert asyncio.run(request()) is None
finally:
instance._log.removeHandler(handler)
server.shutdown()
worker.join(timeout=5)
assert received[0]["messages"] == _MESSAGES
records = [logging.Formatter().format(record) for record in handler.records]
assert any("HTTP 503" in record for record in records)
assert any(f"{len(body)} bytes" in record for record in records)
assert all(canary.lower() not in record.lower() for record in records)
@pytest.mark.parametrize("kind", ["asgi", "litellm"])
@pytest.mark.parametrize("level", [logging.DEBUG, logging.WARNING])
@pytest.mark.parametrize("url_case", sorted(_API_URLS))
def test_cloud_error_logs_no_echoed_content_at_any_level(
kind: str, level: int, url_case: str, monkeypatch: pytest.MonkeyPatch
) -> None:
"""An error response echoing the prompt must not put it, the key, or URL
credentials (query or path) in any formatted record, with DEBUG on or off."""
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
api_url, shown_url = _API_URLS[url_case]
if kind == "asgi":
backend = CompressionMiddleware(app=None, api_key=_KEY_CANARY, api_url=api_url)
else:
backend = HeadroomCallback(api_key=_KEY_CANARY, api_url=api_url)
messages = [{"role": "user", "content": f"my secret is {_PROMPT_CANARY}"}]
echo = f"bad request: {json.dumps(messages)} key={_KEY_CANARY}"
_serve(
backend,
lambda request: httpx.Response(
503,
text=echo,
# A header can echo content too: only the bare media type may be logged.
headers={"content-type": f"text/plain; note={_PROMPT_CANARY}"},
),
)
# Attach to the module logger itself: proxy startup elsewhere in the suite can
# stop headroom.* records from reaching pytest's root-level caplog handler.
handler = _Records()
monkeypatch.setattr(backend._log, "level", level)
monkeypatch.setattr(backend._log, "disabled", False)
backend._log.addHandler(handler)
try:
assert asyncio.run(backend._cloud_compress(messages, "gpt-4o")) is None
finally:
backend._log.removeHandler(handler)
formatted = [logging.Formatter().format(r) for r in handler.records]
warnings = [r.getMessage() for r in handler.records if r.levelno == logging.WARNING]
assert warnings == [
f"Headroom Cloud API error: HTTP 503 from {shown_url}; request sent uncompressed"
]
if level == logging.DEBUG:
assert any("content-type=text/plain," in line for line in formatted)
for line in formatted:
for canary in (_PROMPT_CANARY, _KEY_CANARY, _URL_CANARY, _PATH_CANARY, "%50RIVATE"):
assert canary not in line
@@ -3,26 +3,44 @@
from __future__ import annotations
import importlib
import subprocess
import sys
from pathlib import Path
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
_INTEGRATIONS = Path(__file__).resolve().parents[2] / "headroom" / "integrations"
_BACKEND = "headroom.integrations._compress_backend"
def _import_callback() -> type:
# Import the module directly to avoid triggering headroom/integrations/__init__.py
# which pulls in langchain and the native .so extension.
module_path = (
Path(__file__).resolve().parents[2] / "headroom" / "integrations" / "litellm_callback.py"
)
spec = importlib.util.spec_from_file_location(
"headroom.integrations.litellm_callback",
module_path,
)
def _load_file(name: str, filename: str) -> Any:
spec = importlib.util.spec_from_file_location(name, _INTEGRATIONS / filename)
assert spec is not None and spec.loader is not None
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod) # type: ignore[union-attr]
return mod.HeadroomCallback # type: ignore[attr-defined]
spec.loader.exec_module(mod)
return mod
def _import_callback() -> type:
# Load the module files directly to avoid triggering headroom/integrations/__init__.py,
# which pulls in langchain and the native .so extension. litellm_callback imports
# _compress_backend by its dotted name, so register a file-loaded copy for that one
# import; with the name already in sys.modules, the integrations package is not
# imported. Both files still import the top-level headroom package and its
# stdlib-only helpers (headroom.offline, headroom.log_safety); the subprocess test
# below checks that neither langchain nor headroom._core is loaded.
saved = sys.modules.get(_BACKEND)
sys.modules[_BACKEND] = _load_file(_BACKEND, "_compress_backend.py")
try:
mod = _load_file("headroom.integrations.litellm_callback", "litellm_callback.py")
finally:
if saved is None:
del sys.modules[_BACKEND]
else:
sys.modules[_BACKEND] = saved
return mod.HeadroomCallback # type: ignore[no-any-return]
HeadroomCallback = _import_callback()
@@ -68,3 +86,20 @@ class TestHeadroomCallbackClientLifecycle:
await cb.aclose()
assert cb._client is None
def test_direct_load_does_not_import_the_integrations_package() -> None:
"""The loader above must keep this file isolated from LangChain, the native
extension and the integrations package; check it in a fresh interpreter."""
code = (
"import importlib.util, sys\n"
f"spec = importlib.util.spec_from_file_location('t', {str(Path(__file__))!r})\n"
"mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)\n"
"bad = [m for m in sys.modules if m in ('headroom.integrations', 'headroom._core')"
" or m.startswith('langchain')]\n"
"print(bad)\n"
)
out = subprocess.run(
[sys.executable, "-c", code], capture_output=True, text=True, check=True, timeout=120
)
assert out.stdout.strip() == "[]", out.stdout + out.stderr