mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Closes #4139. A history that is too large to compress should still reach the provider. This keeps the 100 MiB hard admission limit and bypasses request-side transforms above 16 MiB decoded size. The hard limit still applies before forwarding. Current head: `62ffe3defef1684f70d96f1640818d99445a43bb`, merging current main `a2d4fe72` into `1902d8ef`. The 42-test/native/manual/pre-push results below qualify `1902d8ef`, not this new merge. New-head shard 4 has a cached-prefix fixture failure under investigation; coverage uploads are still incomplete. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement - [x] Documentation update ## Changes Made - Keep the startup override and both wire/decompressed hard bounds. Large admitted histories skip normalization, image processing, memory injection, tokenization and compression on Messages, Chat/Responses, Gemini and Bedrock HTTP routes. - Keep provider routing and backend translation, including Copilot Chat, Codex `store:false` and Vertex route-forced streaming. Native forwarding retains decoded bytes unless the provider's wire contract requires a rewrite; translation retains the full message text. - Report actual provider usage, with zero compression savings or attempted tokens for the bypass. Keep `/v1/compress` and WebSocket contracts unchanged. - Add real-provider coverage for large Gemini JSON/SSE requests with query-key auth, and OpenAI JSON/SSE through the installed LiteLLM adapter. No admission-policy or CI changes in this follow-up. ## Testing - [x] Unit tests pass (focused suite and normal critical pre-push suite) - [x] Linting passes (normal hooks) - [x] New tests added for admission, recovery and passthrough accounting - [x] Manual testing performed against the real native build ### Test Output ```sh python -m pytest -q tests/test_proxy_compression_admission.py tests/test_proxy/test_gemini_plugin_routed_streaming.py --cov=headroom --cov-branch # 42 passed, 41 warnings ``` Normal pre-push `make ci-precheck` passed: 1668 Rust tests (3 ignored), 181 Python tests, lint/type checks and commitlint. No hook bypass. The initial deep-module coverage command hit a Pydantic import-isolation error; using the repo's `--cov=headroom` convention passed both complete modules without disabling plugins or tests. The earlier `96c5fe7e` qualification covered the other native protocols, configured/per-request backends, hard-ceiling recovery and complete local shards 2/3. Those runs are historical, not a new whole-suite claim. New-head hosted success is not claimed. ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14, freshly built real `headroom._core`, public Headroom ASGI routes, a real localhost HTTP provider and installed LiteLLM. Synthetic credentials, offline mode and `DO_NOT_TRACK=1`; no live provider calls. - Exact command / steps: Start `create_app(ProxyConfig(backend="litellm-openai", ...))` with all upstream URLs set to the localhost peer and `litellm.api_base=peer+"/v1"`. Leave the production 16 MiB soft/100 MiB hard limits unchanged. POST `"history " * (16777216 // 8 + 1024) + "LAST TURN"` as a user message to Chat Completions, then as a Gemini content part to generateContent/streamGenerateContent, with streaming off/on. The peer rejects a changed history, requires Gemini query-key auth and one `alt=sse`, and returns protocol-valid JSON/SSE with 101 input/1 output tokens. The maintained four small-ceiling cases are in `tests/test_proxy_compression_admission.py` and run with the command above. - Observed result: All four >16 MiB requests returned 200 with the full history intact. JSON/SSE and Gemini auth survived; streamGenerateContent forced SSE even with incoming `alt=json`. Each request recorded exactly 101 provider input/1 output tokens, zero compression attempts/savings and no failures; `/livez` stayed 200. Earlier recorded qualification also covered all five native HTTP protocols, real translating backends and 413 recovery. - Not tested: Live cloud accounts, the reporter's deployed 4 GB VPS, screenshot/image quality or a zero-JSON-parse path. One envelope/model JSON parse remains; no proxy-only RSS guarantee is claimed. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is the admission/compatibility fix. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Keep the 100 MiB hard ceiling; bypass request-side transforms above 16 MiB decoded size on the named HTTP routes. - Kill switch / disable path: Set `HEADROOM_REQUEST_BODY_MAX_BYTES=16777216` (or lower) and restart, accepting local 413s instead of the large-body path. - Unsafe override required: No. - Qualification impact: Local native protocols, backend translation, hard bounds and recovery exercised; live-provider/VPS qualification remains external. - Rollback path: Revert the admission revision and restart; no storage migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I did not edit `CHANGELOG.md` --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
138 lines
4.9 KiB
Python
138 lines
4.9 KiB
Python
"""Org-scale sizing knobs: shared-process stores must be tunable and safe.
|
|
|
|
One Headroom process shared by many users (gateway sidecar/pool) stresses
|
|
stores that were sized for a single user's workload:
|
|
|
|
* the per-session compression-cache entry cap
|
|
(``HEADROOM_COMPRESSION_CACHE_MAX_ENTRIES``),
|
|
* the process-wide frozen-verdicts store
|
|
(``HEADROOM_FROZEN_VERDICTS_MAX``), and
|
|
* the session registry under churn (active sessions must survive a flood
|
|
of transient ones — the LRU property at scale).
|
|
|
|
Registry TTL/LRU mechanics live in ``test_compression_cache_registry.py``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
import textwrap
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
|
|
def _make_proxy():
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
)
|
|
app = create_app(config)
|
|
return app.state.proxy
|
|
|
|
|
|
@pytest.mark.parametrize("raw", ["nan", "inf", "-inf"])
|
|
def test_nonfinite_cache_ttl_still_reclaims_idle_sessions(raw, tmp_path) -> None:
|
|
program = textwrap.dedent(
|
|
"""
|
|
import json
|
|
import time
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
proxy = create_app(ProxyConfig(
|
|
optimize=False, cache_enabled=False, rate_limit_enabled=False,
|
|
cost_tracking_enabled=False, log_requests=False,
|
|
ccr_inject_tool=False, ccr_handle_responses=False,
|
|
ccr_context_tracking=False, image_optimize=False,
|
|
)).state.proxy
|
|
proxy._get_compression_cache("stale")
|
|
proxy._get_compression_cache("fresh")
|
|
now = time.time()
|
|
proxy._compression_cache_last_seen["stale"] = now - 1_000_000
|
|
proxy._compression_caches_last_cleanup = (
|
|
now - proxy._COMPRESSION_CACHE_CLEANUP_INTERVAL_SECONDS - 1
|
|
)
|
|
proxy._get_compression_cache("trigger")
|
|
print(json.dumps({
|
|
"stale": "stale" in proxy._compression_caches,
|
|
"fresh": "fresh" in proxy._compression_caches,
|
|
}))
|
|
"""
|
|
)
|
|
result = subprocess.run(
|
|
[sys.executable, "-c", program],
|
|
env={
|
|
**os.environ,
|
|
"HEADROOM_COMPRESSION_CACHE_TTL_SECONDS": raw,
|
|
"HEADROOM_WORKSPACE_DIR": str(tmp_path),
|
|
},
|
|
capture_output=True,
|
|
text=True,
|
|
check=True,
|
|
timeout=120,
|
|
)
|
|
assert json.loads(result.stdout) == {"stale": False, "fresh": True}
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# Frozen-verdicts store: process-wide, so it must be sizeable per deployment. #
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
def test_frozen_verdicts_eviction_honors_configured_cap(monkeypatch) -> None:
|
|
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
|
|
|
monkeypatch.setenv("HEADROOM_FROZEN_VERDICTS_MAX", "256")
|
|
router = ContentRouter(ContentRouterConfig())
|
|
for key in range(300):
|
|
router._record_frozen_verdict(key, True)
|
|
assert len(router._frozen_verdicts) == 256
|
|
# FIFO: the oldest keys were evicted, the newest survive.
|
|
assert 0 not in router._frozen_verdicts
|
|
assert 299 in router._frozen_verdicts
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# Session registry under org-scale churn: active sessions always survive a #
|
|
# flood of transient ones (the property that keeps busts away at capacity). #
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
def test_active_sessions_survive_transient_flood(monkeypatch) -> None:
|
|
import headroom.proxy.server as server_mod
|
|
|
|
monkeypatch.setattr(server_mod, "MAX_COMPRESSION_CACHE_SESSIONS", 100)
|
|
proxy = _make_proxy()
|
|
|
|
active = [f"active-{i}" for i in range(40)]
|
|
active_caches = {sid: proxy._get_compression_cache(sid) for sid in active}
|
|
|
|
# 400 transient sessions arrive interleaved with active-session traffic —
|
|
# 4x the cap, forcing repeated capacity evictions along the way.
|
|
for i in range(400):
|
|
proxy._get_compression_cache(f"transient-{i}")
|
|
if i % 5 == 0: # active sessions keep making requests
|
|
for sid in active:
|
|
proxy._get_compression_cache(sid)
|
|
|
|
# Every active session survived with its instance (and therefore its
|
|
# byte-replay state) intact; evictions only ever hit transient sessions.
|
|
for sid in active:
|
|
assert proxy._get_compression_cache(sid) is active_caches[sid], (
|
|
f"active session {sid} lost its cache to transient churn"
|
|
)
|
|
assert len(proxy._compression_caches) <= 100 + len(active)
|