mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 16:45:34 +08:00
## Description `CompressionMiddleware` (`headroom/integrations/asgi.py`) and `HeadroomCallback` (`headroom/integrations/litellm_callback.py`) carried identical copies of `_local_compress` and `_cloud_compress`; `diff` of the two method blocks reported no difference. The cloud URL constant, the `HEADROOM_API_KEY` / `HEADROOM_API_URL` reads and `cloud_mode` were copied too. This PR moves them into a `CompressBackend` mixin in `headroom/integrations/_compress_backend.py`. That module imports neither starlette nor litellm, so each integration still works without the other's dependency. Refs #4136 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring ## Changes Made - New `headroom/integrations/_compress_backend.py` with `CompressBackend`, which provides: - `_init_compress_backend(model_limit, hooks, api_key, api_url)`; - `cloud_mode`; - `_local_compress`; - `_cloud_compress`, which keeps the `guard_egress` offline check. - `CompressionMiddleware(CompressBackend)` and `HeadroomCallback(CompressBackend, CustomLogger)` call `_init_compress_backend` and drop their copies. - The method names stay, because `tests/test_offline_egress_chokepoint.py` calls `_cloud_compress` directly. - Each class sets `_log` to its own module logger, so the "Headroom Cloud API error" warning keeps its logger name (`headroom.integrations.asgi` / `headroom.integrations.litellm_callback`). - One log change: that WARNING used to include the first 200 characters of the cloud response body, which can echo request content. It now reads `Headroom Cloud API error: HTTP <status> from <api_url>; request sent uncompressed`, with the URL passed through `redact_url` from `headroom/log_safety.py` (#4138): scheme, host and port only, the path shown as `/<path>` and any query as `?<redacted>`. The body is not logged at any level: a separate DEBUG line gives only the bare media type (`type/subtype`, header parameters dropped) and the body length. - `_DEFAULT_CLOUD_URL` and `guard_egress` stay importable from both modules. - Left alone on purpose: - each class's `aclose`: they differ slightly in when `_client` is cleared; - `min_tokens`: it is stored and never read in either class, as on main. The maintainer should decide separately whether to wire it up. - No `CustomLogger` attribute clashes with the mixin's names. I checked `dir(CustomLogger)` and the instance `vars()`. - Tests: - New `tests/test_integrations/test_compress_backend.py`, parametrized over `CompressionMiddleware` and `HeadroomCallback` with an `httpx.MockTransport` client: a 200 returns the service result (URL, `X-Headroom-Key` header and JSON body checked), and a 503 whose body echoes the prompt returns `None`. `test_cloud_error_logs_no_echoed_content_at_any_level` runs both integrations with the logger at DEBUG and at WARNING, formats every record with `logging.Formatter().format`, and asserts that prompt, API-key, URL-query and URL-path canaries (plain and `%`-encoded) are absent, including a prompt canary echoed in the `Content-Type` header. It fails on 95c608ee; the path cases fail before #4138'sb566a758, and the header case fails on4fa6bb34. `test_direct_load_does_not_import_the_integrations_package` now also checks that `headroom._core` is not loaded, and the fixture comment names what the loaded files import. - `tests/test_integrations/test_litellm_callback.py`: the direct-file loader registers a file-loaded `_compress_backend` before loading `litellm_callback.py`, so the test again avoids importing `headroom.integrations` and LangChain. New `test_direct_load_does_not_import_the_integrations_package` checks that in a fresh interpreter. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compress_api.py tests/test_litellm_callback.py tests/test_integrations/test_litellm_callback.py origin/main: 37 passed this branch: 37 passed $ LITELLM_LOCAL_MODEL_COST_MAP=True pytest tests/test_offline_egress_chokepoint.py -k "asgi or litellm" origin/main: 2 passed this branch: 2 passed $ pytest tests/test_offline_egress_chokepoint.py origin/main: 2 failed, 141 passed this branch: 2 failed, 141 passed (pre-existing and environmental: the same two asgi/litellm tests fail on main, because importing litellm fetches its model cost map from raw.githubusercontent.com and the test's no-socket fixture refuses it; with LITELLM_LOCAL_MODEL_COST_MAP=True both pass on main and on this branch) $ pytest tests/test_integrations/test_compress_backend.py tests/test_integrations/test_litellm_callback.py tests/test_compress_api.py tests/test_litellm_callback.py tests/test_offline_egress_chokepoint.py this branch: 185 passed # test_compress_backend.py against the previous commit's backend (04d38103): 2 failed (warning carried the body), 2 passed $ uvx ruff@0.16.8 check / format <touched files> -> All checks passed! / unchanged $ mypy headroom/integrations/_compress_backend.py asgi.py litellm_callback.py --ignore-missing-imports headroom/integrations/langchain/chat_model.py:394: error: ... [truthy-function] (pre-existing, followed import, untouched file) ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, repo `.venv` (litellm, starlette and httpx installed), this branch's worktree. Temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`. - Exact command / steps: two runs, listed below. 1. Shared smoke harness: a real proxy from this checkout against a local fake upstream (`smoke.sh <worktree>`). 2. The shared cloud path over real HTTP, with both module loggers at DEBUG: a local HTTP server answers 503 (echoing the request, which carries a prompt canary) and then 200, for `HeadroomCallback` and then `CompressionMiddleware` (`_cloud_compress`). The API key, a `/bot…` path segment and a `?token=` query on the URL are canaries too. - Observed result: the output of both runs is in the two blocks below. ```text branch: cleanup/cli-integrations-shared-compress-backend @fad5714chealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-58395.log): 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes. 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. (smoke dir: <tmp>) ``` ```text HeadroomCallback 503 -> None | 200 -> 3 CompressionMiddleware 503 -> None | 200 -> 3 WARNING headroom.integrations.litellm_callback: Headroom Cloud API error: HTTP 503 from http://127.0.0.1:PORT/<path>?<redacted>; request sent uncompressed DEBUG headroom.integrations.litellm_callback: Headroom Cloud API error response: content-type=text/plain, 138 bytes WARNING headroom.integrations.asgi: Headroom Cloud API error: HTTP 503 from http://127.0.0.1:PORT/<path>?<redacted>; request sent uncompressed DEBUG headroom.integrations.asgi: Headroom Cloud API error response: content-type=text/plain, 138 bytes canaries in log: 0 ``` - Not tested: the real Headroom Cloud service. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a, library refactor. - Stable/default behavior changed: no. The env vars, default URL, payload, timeout, offline refusal and log text are all the same. - Kill switch / disable path: n/a. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. Depends on #4138; its two files show in this diff until it merges. - `tests/test_integrations/test_litellm_callback.py` loads `litellm_callback.py` from its file path, to avoid importing the `headroom.integrations` package. Because the module now imports `headroom.integrations._compress_backend`, the loader registers a file-loaded copy of that module first; a module already in `sys.modules` is returned without importing its parent package. Normal imports of `headroom.integrations.litellm_callback` load the package anyway, as before. - A test that patched `headroom.integrations.asgi.guard_egress` would no longer intercept the call, because the call now lives in `_compress_backend`. No such test exists. - Docs impact (the changed warning text) is recorded in the docs ledger; there are no doc edits in this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
178 lines
7.0 KiB
Python
178 lines
7.0 KiB
Python
"""Cloud path of the compression backend shared by the ASGI and LiteLLM integrations."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import json
|
|
import logging
|
|
import threading
|
|
from http.server import BaseHTTPRequestHandler, HTTPServer
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
from headroom.integrations.asgi import CompressionMiddleware
|
|
from headroom.integrations.litellm_callback import HeadroomCallback
|
|
|
|
_MESSAGES = [{"role": "user", "content": "secret prompt text"}]
|
|
|
|
|
|
@pytest.fixture(params=["asgi", "litellm"])
|
|
def backend(request, monkeypatch: pytest.MonkeyPatch):
|
|
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
|
if request.param == "asgi":
|
|
return CompressionMiddleware(app=None, api_key="hdr_test", api_url="https://cloud.test/")
|
|
return HeadroomCallback(api_key="hdr_test", api_url="https://cloud.test/")
|
|
|
|
|
|
def _serve(backend, handler) -> list[httpx.Request]:
|
|
seen: list[httpx.Request] = []
|
|
|
|
def record(request: httpx.Request) -> httpx.Response:
|
|
seen.append(request)
|
|
return handler(request)
|
|
|
|
backend._client = httpx.AsyncClient(transport=httpx.MockTransport(record))
|
|
return seen
|
|
|
|
|
|
def test_cloud_success_returns_the_service_result(backend) -> None:
|
|
payload = {"messages": [{"role": "user", "content": "short"}], "tokens_saved": 3}
|
|
seen = _serve(backend, lambda request: httpx.Response(200, json=payload))
|
|
|
|
result = asyncio.run(backend._cloud_compress(_MESSAGES, "gpt-4o"))
|
|
|
|
assert result == payload
|
|
(request,) = seen
|
|
assert str(request.url) == "https://cloud.test/v1/saas/compress"
|
|
assert request.headers["X-Headroom-Key"] == "hdr_test"
|
|
assert json.loads(request.content) == {
|
|
"messages": _MESSAGES,
|
|
"model": "gpt-4o",
|
|
"model_limit": backend._model_limit,
|
|
}
|
|
|
|
|
|
class _Records(logging.Handler):
|
|
def __init__(self) -> None:
|
|
super().__init__(logging.DEBUG)
|
|
self.records: list[logging.LogRecord] = []
|
|
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
self.records.append(record)
|
|
|
|
|
|
_PROMPT_CANARY = "PROMPT-CANARY-7f3e"
|
|
_KEY_CANARY = "hdr_KEY-CANARY-91ab"
|
|
_URL_CANARY = "URL-CANARY-c0de"
|
|
_PATH_CANARY = "PRIVATE_PATH_CANARY"
|
|
|
|
# (api_url, how the WARNING shows it); every canary must stay out of every record.
|
|
_API_URLS = {
|
|
"query": (f"https://cloud.test/?token={_URL_CANARY}", "https://cloud.test/?<redacted>"),
|
|
"path": (f"https://cloud.test/bot{_PATH_CANARY}", "https://cloud.test/<path>"),
|
|
# %50 is "P": the encoded form must not slip through either.
|
|
"encoded-path": ("https://cloud.test/bot%50RIVATE_PATH_CANARY", "https://cloud.test/<path>"),
|
|
}
|
|
|
|
|
|
@pytest.mark.parametrize("kind", ["asgi", "litellm"])
|
|
def test_real_cloud_error_media_type_cannot_echo_credentials(kind, monkeypatch) -> None:
|
|
canary = "PRIVATE_CREDENTIAL_CANARY"
|
|
body = b"service error"
|
|
received = []
|
|
|
|
class ErrorServer(BaseHTTPRequestHandler):
|
|
def do_POST(self):
|
|
received.append(json.loads(self.rfile.read(int(self.headers["Content-Length"]))))
|
|
self.send_response(503)
|
|
self.send_header("Content-Type", f"text/{canary}")
|
|
self.send_header("Content-Length", str(len(body)))
|
|
self.end_headers()
|
|
self.wfile.write(body)
|
|
|
|
def log_message(self, *args):
|
|
pass
|
|
|
|
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
with HTTPServer(("127.0.0.1", 0), ErrorServer) as server:
|
|
worker = threading.Thread(target=server.serve_forever, daemon=True)
|
|
worker.start()
|
|
api_url = f"http://127.0.0.1:{server.server_port}"
|
|
integration = CompressionMiddleware if kind == "asgi" else HeadroomCallback
|
|
kwargs = {"app": None} if kind == "asgi" else {}
|
|
instance = integration(api_key=_KEY_CANARY, api_url=api_url, **kwargs)
|
|
handler = _Records()
|
|
monkeypatch.setattr(instance._log, "level", logging.DEBUG)
|
|
monkeypatch.setattr(instance._log, "disabled", False)
|
|
instance._log.addHandler(handler)
|
|
|
|
async def request():
|
|
try:
|
|
return await instance._cloud_compress(_MESSAGES, "gpt-4o")
|
|
finally:
|
|
if instance._client is not None:
|
|
await instance._client.aclose()
|
|
|
|
try:
|
|
assert asyncio.run(request()) is None
|
|
finally:
|
|
instance._log.removeHandler(handler)
|
|
server.shutdown()
|
|
worker.join(timeout=5)
|
|
assert received[0]["messages"] == _MESSAGES
|
|
records = [logging.Formatter().format(record) for record in handler.records]
|
|
assert any("HTTP 503" in record for record in records)
|
|
assert any(f"{len(body)} bytes" in record for record in records)
|
|
assert all(canary.lower() not in record.lower() for record in records)
|
|
|
|
|
|
@pytest.mark.parametrize("kind", ["asgi", "litellm"])
|
|
@pytest.mark.parametrize("level", [logging.DEBUG, logging.WARNING])
|
|
@pytest.mark.parametrize("url_case", sorted(_API_URLS))
|
|
def test_cloud_error_logs_no_echoed_content_at_any_level(
|
|
kind: str, level: int, url_case: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""An error response echoing the prompt must not put it, the key, or URL
|
|
credentials (query or path) in any formatted record, with DEBUG on or off."""
|
|
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
api_url, shown_url = _API_URLS[url_case]
|
|
if kind == "asgi":
|
|
backend = CompressionMiddleware(app=None, api_key=_KEY_CANARY, api_url=api_url)
|
|
else:
|
|
backend = HeadroomCallback(api_key=_KEY_CANARY, api_url=api_url)
|
|
messages = [{"role": "user", "content": f"my secret is {_PROMPT_CANARY}"}]
|
|
echo = f"bad request: {json.dumps(messages)} key={_KEY_CANARY}"
|
|
_serve(
|
|
backend,
|
|
lambda request: httpx.Response(
|
|
503,
|
|
text=echo,
|
|
# A header can echo content too: only the bare media type may be logged.
|
|
headers={"content-type": f"text/plain; note={_PROMPT_CANARY}"},
|
|
),
|
|
)
|
|
# Attach to the module logger itself: proxy startup elsewhere in the suite can
|
|
# stop headroom.* records from reaching pytest's root-level caplog handler.
|
|
handler = _Records()
|
|
monkeypatch.setattr(backend._log, "level", level)
|
|
monkeypatch.setattr(backend._log, "disabled", False)
|
|
backend._log.addHandler(handler)
|
|
try:
|
|
assert asyncio.run(backend._cloud_compress(messages, "gpt-4o")) is None
|
|
finally:
|
|
backend._log.removeHandler(handler)
|
|
|
|
formatted = [logging.Formatter().format(r) for r in handler.records]
|
|
warnings = [r.getMessage() for r in handler.records if r.levelno == logging.WARNING]
|
|
assert warnings == [
|
|
f"Headroom Cloud API error: HTTP 503 from {shown_url}; request sent uncompressed"
|
|
]
|
|
if level == logging.DEBUG:
|
|
assert any("content-type=text/plain," in line for line in formatted)
|
|
for line in formatted:
|
|
for canary in (_PROMPT_CANARY, _KEY_CANARY, _URL_CANARY, _PATH_CANARY, "%50RIVATE"):
|
|
assert canary not in line
|