mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Fixes an issue where setting `HEADROOM_DISABLE_KOMPRESS=1` (or configuring `ContentRouterConfig(enable_kompress=False)`) still loaded the Kompress ONNX model when `_get_kompress()` was accessed (such as during `/v1/compress` pipelines or reasoning compaction handlers). When Kompress is disabled, `_get_kompress()` now returns `None` immediately without downloading or loading ONNX model weights into memory, saving ~1 GB of resident memory. Closes #4666 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `if not self.config.enable_kompress: return None` at the entry of `ContentRouter._get_kompress()` in `headroom/transforms/content_router.py`. - Added check `and _rc_router.config.enable_kompress` before querying `_rc_kompress` in `headroom/proxy/handlers/openai.py`. - Explicitly passed `enable_kompress=False` in `ContentRouterConfig` during `cold_recompact_messages()` in `headroom/transforms/cold_prefix.py`. - Added test cases in `tests/test_proxy_disable_kompress.py` asserting that when `enable_kompress=False`, neither `is_kompress_available()`, `KompressCompressor()`, nor `_get_remote_kompress()` are ever called or instantiated. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text ============================= test session starts ============================= platform win32 -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 rootdir: C:\Users\dorni\OneDrive\Desktop\New folder\scratch\headroom configfile: pyproject.toml plugins: anyio-4.13.0, hypothesis-6.165.9, cov-7.1.0, schemathesis-4.24.3 collected 8 items tests\test_proxy_disable_kompress.py ........ [100%] ======================== 8 passed, 1 warning in 4.91s ========================= ``` ## Real Behavior Proof - Environment: Windows 11 x64, Python 3.14.5, pytest 9.0.3 - Exact command / steps: python -c "from headroom.transforms.content_router import ContentRouter, ContentRouterConfig; r = ContentRouter(ContentRouterConfig(enable_kompress=False)); print('Kompress instance with enable_kompress=False:', r._get_kompress())" - Observed result: Kompress instance with enable_kompress=False: None. Verified via mock spied call sites that is_kompress_available(), KompressCompressor(), and _get_remote_kompress() are never invoked when enable_kompress=False. - Not tested: Live Kubernetes cluster deployment, GPU ONNX execution provider. ## Runtime Rollout Safety - Rollout-managed feature(s): Kompress ONNX model loader bypass on ContentRouter - Minimum rollout channel: dev - Stable/default behavior changed: No (when enable_kompress is True or default, existing Kompress instantiation logic executes unchanged) - Kill switch / disable path: HEADROOM_DISABLE_KOMPRESS=1 / --disable-kompress - Unsafe override required: None - Qualification impact: Low (purely safeguards the disabled codepath from unintended initialization) - Rollback path: Revert commit cleanly ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Closes #4666. All tests verified locally and CI governance checks conform to repository standards. ### Maintainer current-main requalification Exact head `be97f120eb9926e99d4008471d09bd93972564c1`, upstream main `c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`. Maintainer current-main review againstc01cf07d94. Clean mechanical synchronization preserves all contributor production changes. Added only a real reasoning-request regression test: actual /v1/chat/completions routing reaches a controlled upstream, forwards reasoning unchanged when disabled, and never queries the remote loader. Isolating the old handler guard makes that test fail with one unwanted loader call; restoring exact contributor bytes passes. All55 disable/cold-prefix/sticky-reasoning/remote-Kompress tests pass with the fresh main native DLL; touched test Ruff/format and whitespace pass. No production logic rewritten. ### Maintainer independent live HTTP proof Environment: Windows x64, Python3.12.13, fresh native binding whose Rust input tree matches current main; real Uvicorn HTTP listener plus two disposable localhost HTTP fixtures (upstream and remote Kompress). No ML model weights or cloud service are needed. This is separate from the pytest assertions. Observed results: - Current main control, disable_kompress=True: HTTP200,1upstream request,1remote inference request,remote client constructed,plaintext reasoning changed. The disabled-state assertion fails as expected. - Reviewed PR, disable_kompress=True: HTTP200,1upstream request,0remote inference requests,no local or remote Kompress instance,original reasoning preserved. - Reviewed PR, disable_kompress=False positive control: HTTP200,1upstream request,1remote inference request,remote client constructed,reasoning compacted. The untested scope is local ONNX/GPU inference, quantitative RSS savings and deployed Kubernetes; this proves disabled request routing/initialization and preserves enabled remote inference. Reproduction: in an isolated checkout install the proxy extra (`python -m pip install -e ".[proxy]"`), save the code below as `kompress_probe.py`, then run `python kompress_probe.py disabled` and `python kompress_probe.py enabled`. The same disabled command on mainc01cf07dreproduces the baseline failure. The script binds only ephemeral loopback ports and shuts all servers down. <details> <summary>Standalone live HTTP probe</summary> ```python import json import os from pathlib import Path import socket import sys import threading import time from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer disabled = sys.argv[1] == 'disabled' for key in ['HTTP_PROXY', 'HTTPS_PROXY', 'ALL_PROXY', 'HEADROOM_KOMPRESS_ENDPOINT_TOKEN', 'HEADROOM_KOMPRESS_ENDPOINT_HEADERS_JSON']: os.environ.pop(key, None) os.environ.update(HEADROOM_OFFLINE='0', HEADROOM_NO_TELEMETRY='1', HEADROOM_SKIP_UPSTREAM_CHECK='1', HEADROOM_REQUIRE_RUST_CORE='false', HEADROOM_HTTP2='false', HEADROOM_DETECT_BACKEND='python', HEADROOM_THINKING_COMPACT='1', HEADROOM_THINKING_COMPACT_KEEP_LAST='0', HEADROOM_KOMPRESS_ENDPOINT_PATH='/compress', NO_PROXY='127.0.0.1,localhost') sys.path.insert(0, str(Path.cwd())) import httpx import uvicorn from headroom.proxy.server import ProxyConfig, create_app from headroom.transforms.compression_units import find_content_router seen = {'remote_calls': 0, 'upstream': []} class Fixture(BaseHTTPRequestHandler): def log_message(self, *args): pass def do_POST(self): body=json.loads(self.rfile.read(int(self.headers['content-length']))) if self.server.remote: seen['remote_calls']+=1 result={'compressed':'concise reasoning', 'original_tokens':300, 'compressed_tokens':2} else: seen['upstream'].append(body) result={'id':'fixture','object':'chat.completion','model':body['model'],'choices':[{'index':0,'message':{'role':'assistant','content':'ok'},'finish_reason':'stop'}],'usage':{'prompt_tokens':300,'completion_tokens':1,'total_tokens':301}} data=json.dumps(result).encode();self.send_response(200);self.send_header('content-type','application/json');self.send_header('content-length',str(len(data)));self.end_headers();self.wfile.write(data) upstream=ThreadingHTTPServer(('127.0.0.1',0),Fixture);upstream.remote=False remote=ThreadingHTTPServer(('127.0.0.1',0),Fixture);remote.remote=True for fixture in [upstream,remote]:threading.Thread(target=fixture.serve_forever,daemon=True).start() os.environ['HEADROOM_KOMPRESS_ENDPOINT']='http://127.0.0.1:'+str(remote.server_port) app=create_app(ProxyConfig(optimize=True, disable_kompress=disabled, openai_api_url='http://127.0.0.1:'+str(upstream.server_port), cache_enabled=False,rate_limit_enabled=False,cost_tracking_enabled=False,log_requests=False,ccr_inject_tool=False,ccr_handle_responses=False,ccr_context_tracking=False,image_optimize=False)) listener=socket.socket();listener.bind(('127.0.0.1',0));listener.listen(128) port=listener.getsockname()[1] server=uvicorn.Server(uvicorn.Config(app,log_level='error',lifespan='on')) thread=threading.Thread(target=server.run,kwargs={'sockets':[listener]},daemon=True);thread.start() reasoning=' '.join('reason'+str(i) for i in range(100)) try: deadline=time.monotonic()+30 while not server.started and thread.is_alive() and time.monotonic()<deadline:time.sleep(.02) assert server.started,'proxy failed to start' with httpx.Client(trust_env=False,timeout=30) as client: response=client.post('http://127.0.0.1:'+str(port)+'/v1/chat/completions',headers={'authorization':'Bearer local-fixture'},json={'model':'kimi-k2.7','messages':[{'role':'user','content':'question'},{'role':'assistant','content':'answer','reasoning_content':reasoning},{'role':'user','content':'continue'}]}) assert response.status_code==200,(response.status_code,response.text) assert len(seen['upstream'])==1 forwarded=seen['upstream'][0]['messages'][1].get('reasoning_content') router=find_content_router(app.state.proxy.openai_pipeline) result={'disabled':disabled,'status':response.status_code,'upstream_calls':len(seen['upstream']),'remote_calls':seen['remote_calls'],'remote_constructed':getattr(router,'_kompress_remote',None) is not None,'local_constructed':router._kompress is not None,'reasoning_preserved':forwarded==reasoning} print('RUNTIME_RESULT '+json.dumps(result),flush=True) if disabled: assert seen['remote_calls']==0 and not result['remote_constructed'] and not result['local_constructed'] assert result['reasoning_preserved'] else: assert seen['remote_calls']>0 and result['remote_constructed'] and not result['reasoning_preserved'] finally: server.should_exit=True;thread.join(10) upstream.shutdown();remote.shutdown();upstream.server_close();remote.server_close() ``` </details> Exact-head hosted qualification is handled by maintainers; approval remains pending that result. No contributor CI action or merge requested. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
265 lines
8.7 KiB
Python
265 lines
8.7 KiB
Python
"""Proxy configuration for disabling Kompress while keeping optimization on."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
from headroom.transforms import CompressionStrategy, ContentRouter
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _stub_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
|
|
monkeypatch.setenv("HEADROOM_HTTP2", "false")
|
|
|
|
|
|
def _proxy_router(config: ProxyConfig):
|
|
app = create_app(config)
|
|
proxy = app.state.proxy
|
|
return next(
|
|
transform
|
|
for transform in proxy.anthropic_pipeline.transforms
|
|
if isinstance(transform, ContentRouter)
|
|
)
|
|
|
|
|
|
def test_disable_kompress_config_keeps_optimization_but_disables_ml_fallback() -> None:
|
|
router = _proxy_router(
|
|
ProxyConfig(
|
|
optimize=True,
|
|
disable_kompress=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
)
|
|
)
|
|
|
|
assert router.config.enable_kompress is False
|
|
assert router.config.fallback_strategy == CompressionStrategy.KOMPRESS
|
|
|
|
|
|
def test_disable_kompress_defaults_to_existing_kompress_behavior() -> None:
|
|
router = _proxy_router(
|
|
ProxyConfig(
|
|
optimize=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
)
|
|
)
|
|
|
|
assert router.config.enable_kompress is True
|
|
assert router.config.fallback_strategy == CompressionStrategy.KOMPRESS
|
|
|
|
|
|
def test_health_config_reports_disable_kompress_fallback(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
from fastapi.testclient import TestClient
|
|
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=True,
|
|
disable_kompress=True,
|
|
disable_kompress_fallback=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
)
|
|
)
|
|
with TestClient(app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345)) as client:
|
|
config = client.get("/health").json()["config"]
|
|
|
|
assert config["disable_kompress"] is True
|
|
assert config["disable_kompress_fallback"] is True
|
|
|
|
|
|
def test_disable_kompress_fallback_restores_passthrough() -> None:
|
|
router = _proxy_router(
|
|
ProxyConfig(
|
|
optimize=True,
|
|
disable_kompress=True,
|
|
disable_kompress_fallback=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
)
|
|
)
|
|
|
|
assert router.config.enable_kompress is False
|
|
assert router.config.fallback_strategy == CompressionStrategy.PASSTHROUGH
|
|
|
|
|
|
def test_disable_kompress_fallback_without_disable_kompress_is_noop() -> None:
|
|
router = _proxy_router(
|
|
ProxyConfig(
|
|
optimize=True,
|
|
disable_kompress_fallback=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
)
|
|
)
|
|
|
|
assert router.config.enable_kompress is True
|
|
assert router.config.fallback_strategy == CompressionStrategy.KOMPRESS
|
|
|
|
|
|
def test_disable_kompress_prevents_kompress_instantiation_in_content_router(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
from unittest.mock import MagicMock
|
|
|
|
import headroom.transforms.kompress_compressor as kc
|
|
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
|
|
|
mock_avail = MagicMock(return_value=True)
|
|
mock_compressor = MagicMock()
|
|
monkeypatch.setattr(kc, "is_kompress_available", mock_avail)
|
|
monkeypatch.setattr(kc, "KompressCompressor", mock_compressor)
|
|
|
|
router = ContentRouter(ContentRouterConfig(enable_kompress=False))
|
|
result = router._get_kompress()
|
|
assert result is None
|
|
mock_avail.assert_not_called()
|
|
mock_compressor.assert_not_called()
|
|
|
|
|
|
def test_disable_kompress_bypasses_remote_kompress(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
from unittest.mock import MagicMock
|
|
|
|
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
|
|
|
mock_remote = MagicMock()
|
|
router = ContentRouter(ContentRouterConfig(enable_kompress=False))
|
|
monkeypatch.setattr(router, "_get_remote_kompress", mock_remote)
|
|
|
|
result = router._get_kompress()
|
|
assert result is None
|
|
mock_remote.assert_not_called()
|
|
|
|
|
|
def test_disable_kompress_reasoning_compaction_skips_kompress_loader(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
from unittest.mock import MagicMock
|
|
|
|
import headroom.transforms.kompress_compressor as kc
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
from headroom.transforms.compression_units import find_content_router
|
|
|
|
mock_avail = MagicMock(return_value=True)
|
|
mock_compressor = MagicMock()
|
|
monkeypatch.setattr(kc, "is_kompress_available", mock_avail)
|
|
monkeypatch.setattr(kc, "KompressCompressor", mock_compressor)
|
|
|
|
config = ProxyConfig(
|
|
optimize=True,
|
|
disable_kompress=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
)
|
|
app = create_app(config)
|
|
proxy = app.state.proxy
|
|
router = find_content_router(proxy.openai_pipeline)
|
|
assert router is not None
|
|
assert router.config.enable_kompress is False
|
|
assert router._get_kompress() is None
|
|
mock_avail.assert_not_called()
|
|
mock_compressor.assert_not_called()
|
|
|
|
|
|
def test_disabled_kompress_reasoning_request_never_queries_remote_loader(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
import json
|
|
from unittest.mock import MagicMock
|
|
|
|
import httpx
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.transforms.compression_units import find_content_router
|
|
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
monkeypatch.setenv("HEADROOM_THINKING_COMPACT", "1")
|
|
monkeypatch.setenv("HEADROOM_THINKING_COMPACT_KEEP_LAST", "0")
|
|
monkeypatch.setenv("HEADROOM_KOMPRESS_ENDPOINT", "http://127.0.0.1:9")
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=True,
|
|
disable_kompress=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
)
|
|
)
|
|
forwarded = []
|
|
|
|
def upstream(request: httpx.Request) -> httpx.Response:
|
|
forwarded.append(json.loads(request.content))
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "fixture",
|
|
"object": "chat.completion",
|
|
"model": "kimi-k2.7",
|
|
"choices": [
|
|
{
|
|
"index": 0,
|
|
"message": {"role": "assistant", "content": "ok"},
|
|
"finish_reason": "stop",
|
|
}
|
|
],
|
|
"usage": {"prompt_tokens": 100, "completion_tokens": 1, "total_tokens": 101},
|
|
},
|
|
)
|
|
|
|
reasoning = " ".join(f"reason{i}" for i in range(100))
|
|
with TestClient(app, client=("127.0.0.1", 50000)) as client:
|
|
proxy = app.state.proxy
|
|
|
|
async def install_transport() -> None:
|
|
await proxy.http_client.aclose()
|
|
if proxy.http_client_h1 is not None:
|
|
await proxy.http_client_h1.aclose()
|
|
proxy.http_client = httpx.AsyncClient(transport=httpx.MockTransport(upstream))
|
|
proxy.http_client_h1 = None
|
|
|
|
client.portal.call(install_transport)
|
|
router = find_content_router(proxy.openai_pipeline)
|
|
assert router is not None
|
|
remote_loader = MagicMock(return_value=None)
|
|
monkeypatch.setattr(router, "_get_remote_kompress", remote_loader)
|
|
response = client.post(
|
|
"/v1/chat/completions",
|
|
headers={"authorization": "Bearer local-fixture"},
|
|
json={
|
|
"model": "kimi-k2.7",
|
|
"messages": [
|
|
{"role": "user", "content": "question"},
|
|
{"role": "assistant", "content": "answer", "reasoning_content": reasoning},
|
|
{"role": "user", "content": "continue"},
|
|
],
|
|
},
|
|
)
|
|
assert response.status_code == 200
|
|
remote_loader.assert_not_called()
|
|
assert len(forwarded) == 1
|
|
assert forwarded[0]["messages"][1]["reasoning_content"] == reasoning
|