mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Closes #4139. A history that is too large to compress should still reach the provider. This keeps the 100 MiB hard admission limit and bypasses request-side transforms above 16 MiB decoded size. The hard limit still applies before forwarding. Current head: `62ffe3defef1684f70d96f1640818d99445a43bb`, merging current main `a2d4fe72` into `1902d8ef`. The 42-test/native/manual/pre-push results below qualify `1902d8ef`, not this new merge. New-head shard 4 has a cached-prefix fixture failure under investigation; coverage uploads are still incomplete. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement - [x] Documentation update ## Changes Made - Keep the startup override and both wire/decompressed hard bounds. Large admitted histories skip normalization, image processing, memory injection, tokenization and compression on Messages, Chat/Responses, Gemini and Bedrock HTTP routes. - Keep provider routing and backend translation, including Copilot Chat, Codex `store:false` and Vertex route-forced streaming. Native forwarding retains decoded bytes unless the provider's wire contract requires a rewrite; translation retains the full message text. - Report actual provider usage, with zero compression savings or attempted tokens for the bypass. Keep `/v1/compress` and WebSocket contracts unchanged. - Add real-provider coverage for large Gemini JSON/SSE requests with query-key auth, and OpenAI JSON/SSE through the installed LiteLLM adapter. No admission-policy or CI changes in this follow-up. ## Testing - [x] Unit tests pass (focused suite and normal critical pre-push suite) - [x] Linting passes (normal hooks) - [x] New tests added for admission, recovery and passthrough accounting - [x] Manual testing performed against the real native build ### Test Output ```sh python -m pytest -q tests/test_proxy_compression_admission.py tests/test_proxy/test_gemini_plugin_routed_streaming.py --cov=headroom --cov-branch # 42 passed, 41 warnings ``` Normal pre-push `make ci-precheck` passed: 1668 Rust tests (3 ignored), 181 Python tests, lint/type checks and commitlint. No hook bypass. The initial deep-module coverage command hit a Pydantic import-isolation error; using the repo's `--cov=headroom` convention passed both complete modules without disabling plugins or tests. The earlier `96c5fe7e` qualification covered the other native protocols, configured/per-request backends, hard-ceiling recovery and complete local shards 2/3. Those runs are historical, not a new whole-suite claim. New-head hosted success is not claimed. ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14, freshly built real `headroom._core`, public Headroom ASGI routes, a real localhost HTTP provider and installed LiteLLM. Synthetic credentials, offline mode and `DO_NOT_TRACK=1`; no live provider calls. - Exact command / steps: Start `create_app(ProxyConfig(backend="litellm-openai", ...))` with all upstream URLs set to the localhost peer and `litellm.api_base=peer+"/v1"`. Leave the production 16 MiB soft/100 MiB hard limits unchanged. POST `"history " * (16777216 // 8 + 1024) + "LAST TURN"` as a user message to Chat Completions, then as a Gemini content part to generateContent/streamGenerateContent, with streaming off/on. The peer rejects a changed history, requires Gemini query-key auth and one `alt=sse`, and returns protocol-valid JSON/SSE with 101 input/1 output tokens. The maintained four small-ceiling cases are in `tests/test_proxy_compression_admission.py` and run with the command above. - Observed result: All four >16 MiB requests returned 200 with the full history intact. JSON/SSE and Gemini auth survived; streamGenerateContent forced SSE even with incoming `alt=json`. Each request recorded exactly 101 provider input/1 output tokens, zero compression attempts/savings and no failures; `/livez` stayed 200. Earlier recorded qualification also covered all five native HTTP protocols, real translating backends and 413 recovery. - Not tested: Live cloud accounts, the reporter's deployed 4 GB VPS, screenshot/image quality or a zero-JSON-parse path. One envelope/model JSON parse remains; no proxy-only RSS guarantee is claimed. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is the admission/compatibility fix. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Keep the 100 MiB hard ceiling; bypass request-side transforms above 16 MiB decoded size on the named HTTP routes. - Kill switch / disable path: Set `HEADROOM_REQUEST_BODY_MAX_BYTES=16777216` (or lower) and restart, accepting local 413s instead of the large-body path. - Unsafe override required: No. - Qualification impact: Local native protocols, backend translation, hard bounds and recovery exercised; live-provider/VPS qualification remains external. - Rollback path: Revert the admission revision and restart; no storage migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I did not edit `CHANGELOG.md` --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
419 lines
16 KiB
Python
419 lines
16 KiB
Python
"""Whole-history admission and compression passthrough behavior."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import gzip
|
|
import json
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
import textwrap
|
|
import threading
|
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|
from urllib.parse import parse_qs, urlsplit
|
|
|
|
import pytest
|
|
|
|
|
|
@pytest.mark.parametrize("raw", ["0", "-1", "not-int", "1.5"])
|
|
def test_invalid_request_body_ceiling_is_rejected(raw: str) -> None:
|
|
from headroom.proxy.request_limit_policy import resolve_request_body_max_bytes
|
|
|
|
with pytest.raises(ValueError):
|
|
resolve_request_body_max_bytes(raw)
|
|
|
|
|
|
def test_startup_override_changes_http_admission_after_restart() -> None:
|
|
program = textwrap.dedent(
|
|
"""
|
|
import json
|
|
from starlette.applications import Starlette
|
|
from starlette.responses import JSONResponse
|
|
from starlette.routing import Route
|
|
from starlette.testclient import TestClient
|
|
from headroom.proxy.request_body_limit import RequestBodyLimitMiddleware
|
|
|
|
async def preprocess(request):
|
|
await request.json()
|
|
return JSONResponse({"preprocessed": True})
|
|
|
|
app = Starlette(routes=[Route("/v1/compress", preprocess, methods=["POST"])])
|
|
app.add_middleware(RequestBodyLimitMiddleware)
|
|
with TestClient(app) as client:
|
|
response = client.post(
|
|
"/v1/compress",
|
|
json={"model": "gpt-4o", "messages": [{"role": "user", "content": "a" * 1500}]},
|
|
)
|
|
print(json.dumps({"status": response.status_code, "body": response.json()}))
|
|
"""
|
|
)
|
|
results = []
|
|
for ceiling in (1024, 2048):
|
|
process = subprocess.run(
|
|
[sys.executable, "-c", program],
|
|
env={
|
|
**os.environ,
|
|
"HEADROOM_REQUEST_BODY_MAX_BYTES": str(ceiling),
|
|
"DO_NOT_TRACK": "1",
|
|
"HEADROOM_TELEMETRY": "off",
|
|
},
|
|
capture_output=True,
|
|
text=True,
|
|
check=True,
|
|
timeout=120,
|
|
)
|
|
results.append(json.loads(process.stdout))
|
|
|
|
assert results[0]["status"] == 413
|
|
assert results[0]["body"]["error"]["code"] == "request_too_large"
|
|
assert results[1] == {"status": 200, "body": {"preprocessed": True}}
|
|
|
|
|
|
@pytest.fixture
|
|
def admitted_proxy(monkeypatch, request):
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy import request_limit_policy
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
monkeypatch.setattr(request_limit_policy, "COMPRESSION_BODY_MAX_BYTES", 1024)
|
|
backend = getattr(request, "param", "anthropic")
|
|
sdk = pytest.importorskip("litellm") if backend != "anthropic" else None
|
|
|
|
class Provider(BaseHTTPRequestHandler):
|
|
def log_message(self, *_args):
|
|
pass
|
|
|
|
def do_POST(self):
|
|
body = json.loads(self.rfile.read(int(self.headers["content-length"])))
|
|
url = urlsplit(self.path)
|
|
path = url.path
|
|
if path.endswith("/responses"):
|
|
if "input" not in body or body.get("store") is not False:
|
|
self.send_error(400, "Unsupported Responses wire request")
|
|
return
|
|
response = json.dumps(
|
|
{
|
|
"id": "resp_admission",
|
|
"object": "response",
|
|
"output": [],
|
|
"usage": {"input_tokens": 101, "output_tokens": 1},
|
|
}
|
|
).encode()
|
|
content_type = "application/json"
|
|
elif path.endswith("/chat/completions"):
|
|
payload = {
|
|
"id": "chat_admission",
|
|
"object": "chat.completion",
|
|
"created": 0,
|
|
"model": "gpt-4o",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": "ok"},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {"prompt_tokens": 101, "completion_tokens": 1, "total_tokens": 102},
|
|
}
|
|
if body.get("stream", False):
|
|
payload["object"] = "chat.completion.chunk"
|
|
payload["choices"] = [
|
|
{
|
|
"index": 0,
|
|
"delta": {"role": "assistant", "content": "ok"},
|
|
"finish_reason": "stop",
|
|
}
|
|
]
|
|
usage = payload.pop("usage")
|
|
usage_chunk = {**payload, "choices": [], "usage": usage}
|
|
response = (
|
|
"".join(
|
|
"data: " + json.dumps(chunk) + "\n\n"
|
|
for chunk in (payload, usage_chunk)
|
|
)
|
|
+ "data: [DONE]\n\n"
|
|
).encode()
|
|
content_type = "text/event-stream"
|
|
else:
|
|
response = json.dumps(payload).encode()
|
|
content_type = "application/json"
|
|
elif path.endswith((":generateContent", ":streamGenerateContent")):
|
|
query = parse_qs(url.query)
|
|
if query.get("key") != ["synthetic-only"]:
|
|
self.send_error(401, "Gemini API key required")
|
|
return
|
|
streaming = path.endswith(":streamGenerateContent")
|
|
if streaming and query.get("alt") != ["sse"]:
|
|
self.send_error(400, "Streaming endpoint requires one alt=sse")
|
|
return
|
|
payload = {
|
|
"candidates": [
|
|
{
|
|
"content": {"role": "model", "parts": [{"text": "ok"}]},
|
|
"finishReason": "STOP",
|
|
}
|
|
],
|
|
"usageMetadata": {
|
|
"promptTokenCount": 101,
|
|
"candidatesTokenCount": 1,
|
|
"totalTokenCount": 102,
|
|
},
|
|
}
|
|
if streaming:
|
|
response = ("data: " + json.dumps(payload) + "\n\n").encode()
|
|
content_type = "text/event-stream"
|
|
else:
|
|
response = json.dumps(payload).encode()
|
|
content_type = "application/json"
|
|
else:
|
|
response = json.dumps(
|
|
{
|
|
"id": "msg_admission",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"model": "claude-sonnet-4-6",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"stop_reason": "end_turn",
|
|
"usage": {"input_tokens": 101, "output_tokens": 1},
|
|
}
|
|
).encode()
|
|
content_type = "application/json"
|
|
self.send_response(200)
|
|
self.send_header("content-type", content_type)
|
|
self.send_header("content-length", str(len(response)))
|
|
self.end_headers()
|
|
self.wfile.write(response)
|
|
|
|
provider = ThreadingHTTPServer(("127.0.0.1", 0), Provider)
|
|
thread = threading.Thread(target=provider.serve_forever, daemon=True)
|
|
thread.start()
|
|
if sdk is not None:
|
|
monkeypatch.setattr(sdk, "api_base", f"http://127.0.0.1:{provider.server_port}/v1")
|
|
monkeypatch.setattr(sdk, "api_key", "synthetic-only")
|
|
monkeypatch.setenv("OPENAI_API_KEY", "synthetic-only")
|
|
config = ProxyConfig(
|
|
anthropic_api_url=f"http://127.0.0.1:{provider.server_port}",
|
|
openai_api_url=f"http://127.0.0.1:{provider.server_port}",
|
|
gemini_api_url=f"http://127.0.0.1:{provider.server_port}",
|
|
bedrock_api_url=f"http://127.0.0.1:{provider.server_port}",
|
|
vertex_api_url=f"http://127.0.0.1:{provider.server_port}",
|
|
backend=backend,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
stateless=True,
|
|
disable_kompress=True,
|
|
)
|
|
try:
|
|
with TestClient(
|
|
create_app(config),
|
|
base_url="http://127.0.0.1",
|
|
client=("127.0.0.1", 50000),
|
|
) as client:
|
|
yield client
|
|
finally:
|
|
provider.shutdown()
|
|
provider.server_close()
|
|
thread.join()
|
|
|
|
|
|
def test_gzip_history_above_soft_threshold_stays_live_without_claiming_savings(admitted_proxy):
|
|
raw = json.dumps(
|
|
{
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 32,
|
|
"messages": [{"role": "user", "content": "history " * 3000}],
|
|
}
|
|
).encode()
|
|
encoded = gzip.compress(raw)
|
|
assert len(encoded) < 1024 < len(raw)
|
|
response = admitted_proxy.post(
|
|
"/v1/messages",
|
|
content=encoded,
|
|
headers={"content-type": "application/json", "content-encoding": "gzip"},
|
|
)
|
|
assert response.status_code == 200
|
|
assert response.json()["content"] == [{"type": "text", "text": "ok"}]
|
|
stats = admitted_proxy.get("/stats").json()
|
|
assert stats["tokens"]["input_provider_reported"] == 101
|
|
assert stats["tokens"]["output"] == 1
|
|
assert stats["tokens"]["proxy_attempted_tokens"] == 0
|
|
assert stats["tokens"]["saved"] == 0
|
|
assert admitted_proxy.get("/livez").status_code == 200
|
|
following = admitted_proxy.post(
|
|
"/v1/messages",
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 32,
|
|
"messages": [{"role": "user", "content": "next turn"}],
|
|
},
|
|
)
|
|
assert following.status_code == 200
|
|
assert admitted_proxy.get("/stats").json()["requests"]["failed"] == 0
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"path",
|
|
[
|
|
"/v1/messages",
|
|
"/v1/chat/completions",
|
|
"/v1/responses",
|
|
"/v1beta/models/gemini-2.5-flash:generateContent",
|
|
"/model/anthropic.claude-v2/invoke",
|
|
],
|
|
)
|
|
def test_soft_passthrough_still_obeys_decompressed_hard_ceiling(admitted_proxy, monkeypatch, path):
|
|
from headroom.proxy import helpers
|
|
|
|
monkeypatch.setattr(helpers, "MAX_DECOMPRESSED_BODY_SIZE", 8192)
|
|
monkeypatch.setattr(helpers, "MAX_REQUEST_BODY_SIZE", 8192)
|
|
content = "history " * 3000
|
|
body = {
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 32,
|
|
"messages": [{"role": "user", "content": content}],
|
|
}
|
|
if path == "/v1/responses":
|
|
body = {"model": "gpt-4o", "input": content}
|
|
elif "/models/" in path:
|
|
body = {"contents": [{"role": "user", "parts": [{"text": content}]}]}
|
|
elif "/model/" in path:
|
|
body["anthropic_version"] = "bedrock-2023-05-31"
|
|
raw = json.dumps(body).encode()
|
|
response = admitted_proxy.post(
|
|
path,
|
|
content=gzip.compress(raw),
|
|
headers={"content-type": "application/json", "content-encoding": "gzip"},
|
|
)
|
|
assert response.status_code == 413
|
|
assert admitted_proxy.get("/livez").status_code == 200
|
|
|
|
|
|
def test_large_copilot_chat_remains_a_chat_completions_request(admitted_proxy, monkeypatch):
|
|
base = admitted_proxy.app.state.proxy.config.openai_api_url
|
|
monkeypatch.setenv("GITHUB_COPILOT_API_URL", base)
|
|
response = admitted_proxy.post(
|
|
"/v1/chat/completions",
|
|
json={
|
|
"model": "gpt-5",
|
|
"messages": [{"role": "user", "content": "history " * 3000}],
|
|
},
|
|
headers={"authorization": "Bearer tid_synthetic_client"},
|
|
)
|
|
assert response.status_code == 200
|
|
assert response.json()["choices"][0]["message"]["content"] == "ok"
|
|
|
|
|
|
def test_large_codex_request_receives_the_required_store_setting(admitted_proxy, monkeypatch):
|
|
from headroom.providers.codex import endpoints
|
|
|
|
monkeypatch.setattr(
|
|
endpoints,
|
|
"CHATGPT_BACKEND_API_URL",
|
|
admitted_proxy.app.state.proxy.config.openai_api_url,
|
|
)
|
|
response = admitted_proxy.post(
|
|
"/v1/responses",
|
|
json={"model": "gpt-5-codex", "input": "history " * 3000},
|
|
headers={
|
|
"authorization": "Bearer synthetic_subscription",
|
|
"chatgpt-account-id": "synthetic_account",
|
|
},
|
|
)
|
|
assert response.status_code == 200
|
|
assert response.json()["object"] == "response"
|
|
|
|
|
|
@pytest.mark.parametrize("admitted_proxy", ["litellm-openai"], indirect=True)
|
|
def test_large_vertex_backend_retains_route_forced_streaming(admitted_proxy):
|
|
response = admitted_proxy.post(
|
|
"/v1/projects/synthetic/locations/us-central1/publishers/anthropic/models/gpt-4o:streamRawPredict",
|
|
json={
|
|
"max_tokens": 32,
|
|
"messages": [{"role": "user", "content": "history " * 3000}],
|
|
},
|
|
)
|
|
assert response.status_code == 200
|
|
assert response.headers["content-type"].startswith("text/event-stream")
|
|
events = [
|
|
json.loads(line.removeprefix("data: "))
|
|
for line in response.text.splitlines()
|
|
if line.startswith("data: ") and line != "data: [DONE]"
|
|
]
|
|
assert events[-1]["type"] == "message_stop"
|
|
|
|
|
|
@pytest.mark.parametrize("streaming", [False, True])
|
|
def test_large_gemini_history_preserves_auth_and_streaming_contract(admitted_proxy, streaming):
|
|
endpoint = "streamGenerateContent" if streaming else "generateContent"
|
|
response = admitted_proxy.post(
|
|
f"/v1beta/models/gemini-2.5-flash:{endpoint}?key=synthetic-only&alt=json",
|
|
json={"contents": [{"role": "user", "parts": [{"text": "history " * 3000}]}]},
|
|
)
|
|
assert response.status_code == 200
|
|
if streaming:
|
|
assert response.headers["content-type"].startswith("text/event-stream")
|
|
events = [
|
|
json.loads(line.removeprefix("data: "))
|
|
for line in response.text.splitlines()
|
|
if line.startswith("data: ")
|
|
]
|
|
assert (
|
|
"".join(
|
|
part["text"]
|
|
for event in events
|
|
for candidate in event.get("candidates", [])
|
|
for part in candidate["content"]["parts"]
|
|
)
|
|
== "ok"
|
|
)
|
|
assert events[-1]["candidates"][0]["finishReason"] == "STOP"
|
|
else:
|
|
assert response.json()["candidates"][0]["content"]["parts"][0]["text"] == "ok"
|
|
stats = admitted_proxy.get("/stats").json()
|
|
assert stats["tokens"]["input_provider_reported"] == 101
|
|
assert stats["tokens"]["output"] == 1
|
|
assert stats["tokens"]["proxy_attempted_tokens"] == 0
|
|
assert stats["tokens"]["saved"] == 0
|
|
assert stats["requests"]["failed"] == 0
|
|
|
|
|
|
@pytest.mark.parametrize("admitted_proxy", ["litellm-openai"], indirect=True)
|
|
@pytest.mark.parametrize("streaming", [False, True])
|
|
def test_large_openai_backend_reports_usage_without_compression(admitted_proxy, streaming):
|
|
body = {
|
|
"model": "gpt-4o",
|
|
"messages": [{"role": "user", "content": "history " * 3000}],
|
|
"stream": streaming,
|
|
}
|
|
if streaming:
|
|
body["stream_options"] = {"include_usage": True}
|
|
response = admitted_proxy.post("/v1/chat/completions", json=body)
|
|
assert response.status_code == 200
|
|
if streaming:
|
|
assert response.headers["content-type"].startswith("text/event-stream")
|
|
events = [
|
|
json.loads(line.removeprefix("data: "))
|
|
for line in response.text.splitlines()
|
|
if line.startswith("data: ") and line != "data: [DONE]"
|
|
]
|
|
assert (
|
|
"".join(
|
|
choice["delta"].get("content", "")
|
|
for event in events
|
|
for choice in event.get("choices", [])
|
|
)
|
|
== "ok"
|
|
)
|
|
assert response.text.endswith("data: [DONE]\n\n")
|
|
else:
|
|
assert response.json()["choices"][0]["message"]["content"] == "ok"
|
|
stats = admitted_proxy.get("/stats").json()
|
|
assert stats["tokens"]["input_provider_reported"] == 101
|
|
assert stats["tokens"]["output"] == 1
|
|
assert stats["tokens"]["proxy_attempted_tokens"] == 0
|
|
assert stats["tokens"]["saved"] == 0
|
|
assert stats["requests"]["failed"] == 0
|