mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Removes private CLI helpers that no product code calls any more. Each one lost its last caller in an earlier rewrite, so only its own unit tests still called it. The tests that pinned the dead code go with it. Refs #4136 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/wrap.py`: removed six dead helpers and constants. - `_wrap_marker_proxy_is_dead`: the self-heal path now calls `_wrap_proxy_alive` inline. - `_agent_savings_config_mismatches` and `_AGENT_SAVINGS_TARGET_AGENTS`: the live restart check only compares `savings_profile`. - The `_normalize_openclaw_gateway_provider_ids` forwarder and its import alias. - A dead copy of `_opencode_home_dir`; the real one lives in `providers/opencode/config.py`. - The unused `_CONTEXT_1M_SUFFIX` and `_OPENCLAUDE_INSTRUCTIONS_FILE` constants. - `headroom/cli/proxy.py`: removed `_get_env_float_optional`, which had no callers. - `headroom/cli/mcp.py`: removed `get_headroom_command`, `load_mcp_config`, `save_mcp_config`, `find_headroom_registration` and the `CLAUDE_CONFIG_DIR` / `MCP_CONFIG_PATH` / `CLAUDE_JSON_PATH` constants. - `mcp install`, `mcp status` and `mcp uninstall` all go through `headroom.mcp_registry` registrars now. - `load_mcp_config` returned `{}` on corrupt JSON, and `save_mcp_config` would then overwrite the file. That data-loss path goes away with them. - Tests removed, because they only exercised the deleted code: - `tests/test_cli/test_mcp_status.py`, all of which tested `find_headroom_registration`; - `TestMCPConfigFunctions` and its path fixtures in `tests/test_cli/test_mcp.py`; - 3 `_wrap_marker_proxy_is_dead` tests; - 5 `_agent_savings_config_mismatches` tests; - 1 forwarder test. - The remaining `mcp uninstall` / `mcp status` tests already patch the registrars. They no longer take the path fixture. Evidence: `rg -n -w '<name>' headroom crates plugins sdk examples benchmarks scripts e2e docs wiki .github pyproject.toml tests <consumer repos>` for each name finds only the definition and the removed tests. Live behaviour stays covered: - `test_wrap_dead_marker_selfheal.py`: `test_dead_port_*`, `test_live_port_is_never_cleared` and the `_wrap_proxy_alive` tests; - `tests/test_provider_openclaw_wrap.py`; - `test_mcp.py::TestMCPStatusCommand` / `TestMCPUninstallCommand`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest -q -p no:cacheprovider tests/test_cli/test_wrap_dead_marker_selfheal.py tests/test_agent_savings.py tests/test_cli/test_wrap_openclaw.py tests/test_cli/test_wrap_opencode.py tests/test_cli/test_mcp.py tests/test_cli/test_wrap_helpers.py tests/test_provider_openclaw_wrap.py tests/test_cli/test_main_help_version.py 253 passed in 33.08s $ uvx ruff@0.16.8 check headroom/cli tests/test_cli tests/test_agent_savings.py All checks passed! $ mypy headroom/cli/wrap.py headroom/cli/mcp.py headroom/cli/proxy.py --ignore-missing-imports Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, repo `.venv`, branch cut from `origin/main` ata9815bbb. - Exact command / steps: ran the pytest files above, which cover `wrap`, `unwrap`, `mcp status` and `mcp uninstall` via CliRunner, plus `headroom --help` / `--version`. - Observed result: all 253 passed; ruff and mypy are clean. - Smoke harness (real proxy started from this checkout against a local fake upstream; temp HOME and workspace, offline): ```text branch: cleanup/cli-integrations-dead-cli-helpers @5f450556health: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-63492.log): 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes. 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. (smoke dir: <tmp>) ``` - Direct runtime command on this PR's own path (temp HOME, HEADROOM_WORKSPACE_DIR and CODEX_HOME): ```text $ HOME=<tmp>/home HEADROOM_WORKSPACE_DIR=<tmp>/ws CODEX_HOME=<tmp>/codex headroom mcp install --agent claude exit=0 --- stdout Installing Headroom MCP server... claude: registered (restart the agent if it was already running) Next steps: 1. Start the Headroom proxy (if not running): headroom proxy 2. Start your agent (e.g.) ANTHROPIC_BASE_URL=http://127.0.0.1:8787 claude 3. Restart any agent that was already running so it picks up the new MCP server. --- stderr $ HOME=<tmp>/home HEADROOM_WORKSPACE_DIR=<tmp>/ws CODEX_HOME=<tmp>/codex headroom mcp status exit=0 --- stdout Headroom MCP Status ======================================== MCP SDK: ✓ Installed Agent Config: Antigravity IDE: ✗ Not detected Claude Code: ✓ Configured $ HOME=<tmp>/home HEADROOM_WORKSPACE_DIR=<tmp>/ws CODEX_HOME=<tmp>/codex headroom mcp uninstall exit=0 --- stdout ✓ Claude Code: removed headroom MCP server(s) --- stderr ``` - Not tested: a live `headroom wrap` session against a real agent binary. Only unreferenced private code was deleted, so no runtime path changes. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A, no rollout-managed behaviour. - Stable/default behavior changed: No. Only code with no callers is removed. - Kill switch / disable path: N/A. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. - No new tests or docs are needed, because this is a pure removal and none of the removed names were documented. - `_DEFAULT_1M_MODEL` stays, because `tests/test_cli/test_wrap_helpers.py` pins it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4118" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
811 lines
28 KiB
Python
811 lines
28 KiB
Python
from __future__ import annotations
|
|
|
|
import json
|
|
import logging
|
|
from importlib import import_module
|
|
from types import SimpleNamespace
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.agent_savings import (
|
|
AGENT_90_PROFILE,
|
|
DEFAULT_PROFILE,
|
|
apply_agent_savings_env_defaults,
|
|
apply_agent_savings_profile,
|
|
get_agent_savings_profile,
|
|
proxy_pipeline_kwargs,
|
|
seed_proxy_env_defaults,
|
|
)
|
|
from headroom.cli import wrap as wrap_module
|
|
from headroom.cli.main import main
|
|
from headroom.compress import CompressConfig, compress
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.transforms.compression_units import (
|
|
CompressionUnit,
|
|
compress_unit_with_router,
|
|
)
|
|
from headroom.transforms.content_router import (
|
|
CompressionStrategy,
|
|
ContentRouter,
|
|
ContentRouterConfig,
|
|
RouterCompressionResult,
|
|
)
|
|
|
|
compress_module = import_module("headroom.compress")
|
|
|
|
|
|
def test_agent_90_profile_sets_accuracy_preserving_compress_config() -> None:
|
|
cfg = CompressConfig()
|
|
|
|
apply_agent_savings_profile(cfg, AGENT_90_PROFILE)
|
|
|
|
assert cfg.compress_user_messages is True
|
|
assert cfg.compress_system_messages is True
|
|
assert cfg.protect_recent == 2
|
|
assert cfg.protect_analysis_context is True
|
|
assert cfg.target_ratio == 0.10
|
|
assert cfg.min_tokens_to_compress == 120
|
|
|
|
|
|
def test_agent_90_profile_exports_cross_agent_proxy_env() -> None:
|
|
profile = get_agent_savings_profile(AGENT_90_PROFILE)
|
|
|
|
env = profile.proxy_env()
|
|
|
|
assert env["HEADROOM_MODE"] == "token"
|
|
assert env["HEADROOM_SAVINGS_PROFILE"] == "agent-90"
|
|
assert env["HEADROOM_SAVINGS_TARGET"] == "0.90"
|
|
assert env["HEADROOM_TARGET_RATIO"] == "0.10"
|
|
assert env["HEADROOM_COMPRESS_USER_MESSAGES"] == "1"
|
|
assert env["HEADROOM_COMPRESS_SYSTEM_MESSAGES"] == "1"
|
|
assert env["HEADROOM_MAX_ITEMS"] == "8"
|
|
assert env["HEADROOM_SMART_CRUSHER_COMPACTION"] == "0"
|
|
assert env["HEADROOM_FORCE_KOMPRESS"] == "1"
|
|
assert env["HEADROOM_ACCURACY_GUARD"] == "strict"
|
|
|
|
|
|
def test_coding_persona_compresses_recent_delta_and_stays_visible() -> None:
|
|
profile = get_agent_savings_profile("coding")
|
|
|
|
env = profile.proxy_env()
|
|
|
|
assert env["HEADROOM_SAVINGS_PROFILE"] == "coding"
|
|
assert env["HEADROOM_MODE"] == "cache" # delta-only compression at ~0 prefix-cache busts
|
|
assert env["HEADROOM_PROTECT_RECENT"] == "0" # reads guarded by type, not position
|
|
assert env["HEADROOM_MIN_TOKENS"] == "10" # low → even modest deltas are eligible
|
|
# Cache mode compresses the newest observation delta → compress_user must be ON.
|
|
assert env["HEADROOM_COMPRESS_USER_MESSAGES"] == "1"
|
|
assert env["HEADROOM_COMPRESS_SYSTEM_MESSAGES"] == "0" # system prompt is the hottest cache
|
|
assert env["HEADROOM_ACCURACY_GUARD"] == "strict"
|
|
assert "HEADROOM_TARGET_RATIO" not in env # unset → Kompress / ambient default decides
|
|
# Coding posture toggles seeded through the profile.
|
|
assert env["HEADROOM_TOOL_SEARCH"] == "1"
|
|
assert env["HEADROOM_DEDUPE"] == "1"
|
|
assert env["HEADROOM_LOSSLESS_THEN_LOSSY"] == "1"
|
|
assert env["HEADROOM_PROTECT_READS"] == "1"
|
|
assert env["HEADROOM_CODE_AWARE_ENABLED"] == "1"
|
|
assert env["HEADROOM_LOSSLESS"] == "0" # lossy enabled (CCR keeps it recoverable)
|
|
assert env["HEADROOM_MIN_CHARS_FOR_BLOCK"] == "25"
|
|
|
|
|
|
def test_coding_profile_couples_zero_protect_recent_with_type_read_guard() -> None:
|
|
"""Fidelity invariant behind #2145's protect_recent 2->0.
|
|
|
|
Dropping positional protection (protect_recent=0) is only safe because the
|
|
code working set stays byte-exact via the TYPE-based read guard
|
|
(protect_reads=True), and any *other* recent delta that does get compressed
|
|
stays losslessly recoverable via CCR (lossless=0 means lossy-with-CCR, not
|
|
silent loss) while the frozen prefix is left untouched (cache mode). This is
|
|
the honest boundary: recent file reads are verbatim, recent non-read deltas
|
|
are recoverable — not "nothing is ever touched". If a future edit drops the
|
|
read guard while keeping protect_recent=0, recent reads would silently
|
|
degrade, so this test fails closed on that pairing.
|
|
"""
|
|
profile = get_agent_savings_profile("coding")
|
|
|
|
# Positional protection is off ...
|
|
assert profile.protect_recent == 0
|
|
# ... so the byte-exact guarantee for reads MUST come from the type guard.
|
|
assert profile.protect_reads is True
|
|
|
|
env = profile.proxy_env()
|
|
assert env["HEADROOM_PROTECT_RECENT"] == "0"
|
|
assert env["HEADROOM_PROTECT_READS"] == "1"
|
|
# Lossy compression is on, but CCR keeps compressed deltas recoverable, and
|
|
# cache mode compresses only the newest delta (frozen prefix stays byte-stable).
|
|
assert env["HEADROOM_LOSSLESS"] == "0"
|
|
assert env["HEADROOM_MODE"] == "cache"
|
|
|
|
|
|
def test_general_persona_has_no_positional_code_protection() -> None:
|
|
profile = get_agent_savings_profile("general")
|
|
|
|
env = profile.proxy_env()
|
|
|
|
assert env["HEADROOM_PROTECT_RECENT"] == "0"
|
|
assert env["HEADROOM_MIN_TOKENS"] == "25"
|
|
assert "HEADROOM_TARGET_RATIO" not in env
|
|
|
|
|
|
def test_personas_omit_target_ratio_in_pipeline_kwargs() -> None:
|
|
# coding compresses the delta observation (cache mode) → compress_user True;
|
|
# general has no positional code working set and leaves user turns intact.
|
|
for name, expected_protect, expected_compress_user, expected_min_tokens in (
|
|
("coding", 0, True, 10),
|
|
("general", 0, False, 25),
|
|
):
|
|
kwargs = proxy_pipeline_kwargs(ProxyConfig(savings_profile=name))
|
|
|
|
assert kwargs["protect_recent"] == expected_protect
|
|
assert kwargs["read_protection_window"] == expected_protect
|
|
assert kwargs["min_tokens_to_compress"] == expected_min_tokens
|
|
assert kwargs["compress_user_messages"] is expected_compress_user
|
|
assert kwargs["compress_system_messages"] is False
|
|
assert kwargs["force_kompress"] is False
|
|
assert "target_ratio" not in kwargs # persona never pins a keep-ratio
|
|
|
|
|
|
def test_persona_apply_profile_leaves_target_ratio_untouched() -> None:
|
|
cfg = CompressConfig(target_ratio=0.42)
|
|
|
|
apply_agent_savings_profile(cfg, "coding")
|
|
|
|
assert cfg.protect_recent == 0
|
|
assert cfg.min_tokens_to_compress == 10
|
|
assert cfg.target_ratio == 0.42 # persona did not override an explicit ratio
|
|
|
|
|
|
def test_agent_savings_env_defaults_preserve_user_overrides() -> None:
|
|
env = {
|
|
"HEADROOM_TARGET_RATIO": "0.25",
|
|
"HEADROOM_MAX_ITEMS": "12",
|
|
}
|
|
|
|
apply_agent_savings_env_defaults(env, AGENT_90_PROFILE)
|
|
|
|
assert env["HEADROOM_SAVINGS_PROFILE"] == "agent-90"
|
|
assert env["HEADROOM_TARGET_RATIO"] == "0.25"
|
|
assert env["HEADROOM_MAX_ITEMS"] == "12"
|
|
assert env["HEADROOM_SMART_CRUSHER_COMPACTION"] == "0"
|
|
|
|
|
|
def test_seed_proxy_env_defaults_reports_only_generated_keys() -> None:
|
|
env = {"HEADROOM_MODE": "token", "HEADROOM_LOSSLESS": "1"}
|
|
|
|
seeded = seed_proxy_env_defaults(env)
|
|
|
|
assert "HEADROOM_MODE" not in seeded
|
|
assert "HEADROOM_LOSSLESS" not in seeded
|
|
assert "HEADROOM_SAVINGS_PROFILE" in seeded
|
|
assert env["HEADROOM_MODE"] == "token"
|
|
assert env["HEADROOM_LOSSLESS"] == "1"
|
|
|
|
|
|
def test_unknown_agent_savings_profile_falls_back_to_default(
|
|
caplog: pytest.LogCaptureFixture,
|
|
) -> None:
|
|
# An unknown profile must NOT raise: it's resolved during proxy startup, so
|
|
# raising takes the whole proxy down before it opens its port (desktop asked
|
|
# for a profile a fallback runtime predates).
|
|
#
|
|
# It must degrade to the DEFAULT profile, not to "balanced". The two are not
|
|
# interchangeable: balanced flips cache->token mode, turns cross-turn dedup
|
|
# and tool-search off, stops compressing user messages, and raises the
|
|
# message floor 25x (250 vs 10) and the block floor 20x (500 vs 25). A typo
|
|
# in HEADROOM_SAVINGS_PROFILE used to silently reconfigure the entire proxy
|
|
# into that posture, which is strictly worse than behaving as if the
|
|
# variable were unset.
|
|
with caplog.at_level(logging.WARNING):
|
|
profile = get_agent_savings_profile("missing")
|
|
assert profile is get_agent_savings_profile(None)
|
|
assert profile.name == DEFAULT_PROFILE
|
|
assert profile is not get_agent_savings_profile("balanced")
|
|
assert "unknown savings profile" in caplog.text
|
|
assert "missing" in caplog.text
|
|
# The warning has to name the resolved profile, so an operator reading it
|
|
# knows what they actually got rather than only what they asked for.
|
|
assert DEFAULT_PROFILE in caplog.text
|
|
|
|
|
|
def test_agent_savings_cli_renders_shell_exports() -> None:
|
|
result = CliRunner().invoke(main, ["agent-savings", "--profile", "agent-90"])
|
|
|
|
assert result.exit_code == 0
|
|
assert 'export HEADROOM_SAVINGS_PROFILE="agent-90"' in result.output
|
|
assert 'export HEADROOM_SAVINGS_TARGET="0.90"' in result.output
|
|
assert 'export HEADROOM_ACCURACY_GUARD="strict"' in result.output
|
|
|
|
|
|
def test_agent_savings_cli_renders_json() -> None:
|
|
result = CliRunner().invoke(
|
|
main,
|
|
["agent-savings", "--profile", "agent-90", "--format", "json"],
|
|
)
|
|
|
|
assert result.exit_code == 0
|
|
assert '"HEADROOM_TARGET_RATIO": "0.10"' in result.output
|
|
|
|
|
|
def test_compress_applies_agent_savings_profile_to_pipeline(monkeypatch) -> None:
|
|
captured: dict[str, object] = {}
|
|
messages = [{"role": "user", "content": "x" * 500}]
|
|
|
|
class Pipeline:
|
|
def apply(self, **kwargs):
|
|
captured.update(kwargs)
|
|
return SimpleNamespace(
|
|
messages=messages,
|
|
tokens_before=1000,
|
|
tokens_after=100,
|
|
transforms_applied=["test"],
|
|
)
|
|
|
|
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: Pipeline())
|
|
|
|
config = CompressConfig()
|
|
apply_agent_savings_profile(config, AGENT_90_PROFILE)
|
|
|
|
result = compress(messages, config=config)
|
|
|
|
assert result.compression_ratio == 0.9
|
|
assert captured["compress_user_messages"] is True
|
|
assert captured["compress_system_messages"] is True
|
|
assert captured["protect_recent"] == 2
|
|
assert captured["protect_analysis_context"] is True
|
|
assert captured["target_ratio"] == 0.10
|
|
assert captured["min_tokens_to_compress"] == 120
|
|
|
|
|
|
def test_compress_savings_profile_does_not_mutate_supplied_config(monkeypatch) -> None:
|
|
captured: dict[str, object] = {}
|
|
messages = [{"role": "user", "content": "x" * 500}]
|
|
config = CompressConfig(
|
|
compress_user_messages=False,
|
|
compress_system_messages=False,
|
|
protect_recent=9,
|
|
protect_analysis_context=False,
|
|
target_ratio=None,
|
|
min_tokens_to_compress=999,
|
|
)
|
|
|
|
class Pipeline:
|
|
def apply(self, **kwargs):
|
|
captured.update(kwargs)
|
|
return SimpleNamespace(
|
|
messages=messages,
|
|
tokens_before=1000,
|
|
tokens_after=100,
|
|
transforms_applied=["test"],
|
|
)
|
|
|
|
monkeypatch.setattr(compress_module, "_get_pipeline", lambda: Pipeline())
|
|
|
|
compress(messages, config=config, savings_profile=AGENT_90_PROFILE)
|
|
|
|
assert captured["target_ratio"] == 0.10
|
|
assert captured["min_tokens_to_compress"] == 120
|
|
assert config.compress_user_messages is False
|
|
assert config.compress_system_messages is False
|
|
assert config.protect_recent == 9
|
|
assert config.protect_analysis_context is False
|
|
assert config.target_ratio is None
|
|
assert config.min_tokens_to_compress == 999
|
|
|
|
|
|
def test_force_kompress_falls_back_to_structural_compression_when_cold(monkeypatch):
|
|
"""A cold forced-ML profile must not hide safe content routing."""
|
|
import headroom.transforms.content_router as router_mod
|
|
|
|
router = ContentRouter(ContentRouterConfig())
|
|
compressor = SimpleNamespace(is_ready=lambda: False, ensure_background_load=MagicMock())
|
|
log_compressor = SimpleNamespace(
|
|
compress=lambda content, bias=1.0: SimpleNamespace(compressed="ERROR: compacted")
|
|
)
|
|
router._get_kompress = lambda: compressor
|
|
router._get_log_compressor = lambda: log_compressor
|
|
router._runtime_force_kompress = True
|
|
router._runtime_skip_kompress = False
|
|
router._runtime_target_ratio = None
|
|
router.config.enable_log_compressor = True
|
|
monkeypatch.setattr(
|
|
router_mod,
|
|
"_detect_content",
|
|
lambda content: router_mod.DetectionResult(router_mod.ContentType.BUILD_OUTPUT, 1.0, {}),
|
|
)
|
|
monkeypatch.setattr(router_mod, "is_mixed_content", lambda content: False)
|
|
|
|
result = router.compress("ERROR: repeated output\\n" * 40)
|
|
|
|
assert result.compressed == "ERROR: compacted"
|
|
assert result.strategy_used == CompressionStrategy.LOG
|
|
compressor.ensure_background_load.assert_called_once()
|
|
|
|
|
|
def test_wrap_agent_savings_profile_is_opt_in(monkeypatch) -> None:
|
|
monkeypatch.delenv("HEADROOM_SAVINGS_PROFILE", raising=False)
|
|
|
|
assert wrap_module._wrap_agent_savings_profile("codex") is None
|
|
|
|
monkeypatch.setenv("HEADROOM_SAVINGS_PROFILE", AGENT_90_PROFILE)
|
|
|
|
assert wrap_module._wrap_agent_savings_profile("codex") == AGENT_90_PROFILE
|
|
|
|
|
|
def test_start_proxy_does_not_inject_agent_savings_by_default(monkeypatch, tmp_path) -> None:
|
|
captured_env: dict[str, str] = {}
|
|
|
|
class Proc:
|
|
returncode = None
|
|
|
|
def poll(self) -> None:
|
|
return None
|
|
|
|
def popen(cmd, **kwargs): # noqa: ANN001
|
|
captured_env.update(kwargs["env"])
|
|
return Proc()
|
|
|
|
monkeypatch.delenv("HEADROOM_SAVINGS_PROFILE", raising=False)
|
|
monkeypatch.setattr(wrap_module.subprocess, "Popen", popen)
|
|
monkeypatch.setattr(wrap_module.time, "sleep", lambda seconds: None)
|
|
monkeypatch.setattr(wrap_module, "_check_proxy", lambda port: True)
|
|
monkeypatch.setattr(wrap_module, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
|
|
|
|
wrap_module._start_proxy(8787, agent_type="codex")
|
|
|
|
assert "HEADROOM_SAVINGS_PROFILE" not in captured_env
|
|
assert "HEADROOM_TARGET_RATIO" not in captured_env
|
|
|
|
|
|
def test_start_proxy_injects_explicit_agent_savings_profile(monkeypatch, tmp_path) -> None:
|
|
captured_env: dict[str, str] = {}
|
|
|
|
class Proc:
|
|
returncode = None
|
|
|
|
def poll(self) -> None:
|
|
return None
|
|
|
|
def popen(cmd, **kwargs): # noqa: ANN001
|
|
captured_env.update(kwargs["env"])
|
|
return Proc()
|
|
|
|
monkeypatch.setenv("HEADROOM_SAVINGS_PROFILE", AGENT_90_PROFILE)
|
|
monkeypatch.setattr(wrap_module.subprocess, "Popen", popen)
|
|
monkeypatch.setattr(wrap_module.time, "sleep", lambda seconds: None)
|
|
monkeypatch.setattr(wrap_module, "_check_proxy", lambda port: True)
|
|
monkeypatch.setattr(wrap_module, "_get_log_path", lambda port=None: tmp_path / "proxy.log")
|
|
|
|
wrap_module._start_proxy(8787, agent_type="codex")
|
|
|
|
assert captured_env["HEADROOM_SAVINGS_PROFILE"] == AGENT_90_PROFILE
|
|
assert captured_env["HEADROOM_TARGET_RATIO"] == "0.10"
|
|
|
|
|
|
def test_agent_90_profile_applies_to_proxy_config_runtime_kwargs() -> None:
|
|
config = ProxyConfig(savings_profile="agent-90")
|
|
|
|
kwargs = proxy_pipeline_kwargs(config)
|
|
|
|
assert kwargs["compress_user_messages"] is True
|
|
assert kwargs["compress_system_messages"] is True
|
|
assert kwargs["protect_recent"] == 2
|
|
assert kwargs["protect_analysis_context"] is True
|
|
assert kwargs["target_ratio"] == 0.10
|
|
assert kwargs["min_tokens_to_compress"] == 120
|
|
assert kwargs["max_items_after_crush"] == 8
|
|
assert kwargs["smart_crusher_with_compaction"] is False
|
|
assert kwargs["force_kompress"] is True
|
|
assert kwargs["read_protection_window"] == 2
|
|
|
|
|
|
def test_proxy_explicit_config_overrides_agent_90_profile() -> None:
|
|
config = ProxyConfig(
|
|
savings_profile="agent-90",
|
|
target_ratio=0.25,
|
|
protect_recent=5,
|
|
min_tokens_to_crush=300,
|
|
)
|
|
|
|
kwargs = proxy_pipeline_kwargs(config)
|
|
|
|
assert kwargs["target_ratio"] == 0.25
|
|
assert kwargs["protect_recent"] == 5
|
|
assert kwargs["min_tokens_to_compress"] == 300
|
|
|
|
|
|
def test_proxy_explicit_compress_user_messages_off_overrides_coding_profile() -> None:
|
|
# coding turns user-message compression on; an explicit off
|
|
# (HEADROOM_COMPRESS_USER_MESSAGES=0) must win like every other override,
|
|
# and leaving it unset must keep the profile default.
|
|
explicit_off = ProxyConfig(savings_profile="coding", compress_user_messages=False)
|
|
unset = ProxyConfig(savings_profile="coding")
|
|
|
|
assert proxy_pipeline_kwargs(explicit_off)["compress_user_messages"] is False
|
|
assert proxy_pipeline_kwargs(unset)["compress_user_messages"] is True
|
|
|
|
|
|
def test_agent_90_router_uses_ccr_sampling_not_lossless_table() -> None:
|
|
router = ContentRouter(
|
|
ContentRouterConfig(
|
|
smart_crusher_max_items_after_crush=8,
|
|
smart_crusher_with_compaction=False,
|
|
)
|
|
)
|
|
|
|
crusher = router._get_smart_crusher()
|
|
|
|
assert crusher is not None
|
|
assert crusher.config.max_items_after_crush == 8
|
|
assert crusher._with_compaction is False
|
|
|
|
|
|
def test_router_lossless_only_flag_reaches_crusher() -> None:
|
|
# HEADROOM_LOSSLESS_ONLY=1 sets this field on the proxy router; it
|
|
# must flow through to the SmartCrusher so a real proxy session runs
|
|
# strict marker-free mode.
|
|
router = ContentRouter(ContentRouterConfig(smart_crusher_lossless_only=True))
|
|
|
|
crusher = router._get_smart_crusher()
|
|
|
|
assert crusher is not None
|
|
assert crusher._lossless_only is True
|
|
|
|
|
|
def test_router_lossless_only_defaults_off() -> None:
|
|
# Unset (None) must not force the flag — default crushers stay in
|
|
# the marker-emitting mode.
|
|
router = ContentRouter(ContentRouterConfig())
|
|
|
|
crusher = router._get_smart_crusher()
|
|
|
|
assert crusher is not None
|
|
assert crusher._lossless_only is False
|
|
|
|
|
|
def test_agent_90_router_json_tool_output_reaches_target_with_needle() -> None:
|
|
needle = "CRITICAL_NEEDLE_42"
|
|
rows = [
|
|
{
|
|
"id": i,
|
|
"status": "ok",
|
|
"message": "normal repeated telemetry payload",
|
|
"value": i % 7,
|
|
}
|
|
for i in range(1000)
|
|
]
|
|
rows.append(
|
|
{
|
|
"id": 99999,
|
|
"status": "error",
|
|
"message": f"{needle} root cause disk full",
|
|
"value": 999.99,
|
|
}
|
|
)
|
|
router = ContentRouter(
|
|
ContentRouterConfig(
|
|
smart_crusher_max_items_after_crush=8,
|
|
smart_crusher_with_compaction=False,
|
|
)
|
|
)
|
|
|
|
result = router.compress(json.dumps(rows), question=f"Find {needle}")
|
|
before = len(result.original.split())
|
|
after = len(result.compressed.split())
|
|
|
|
assert 1 - after / before >= 0.90
|
|
assert needle in result.compressed
|
|
assert "<<ccr:" in result.compressed
|
|
|
|
|
|
def test_proxy_cli_reads_agent_90_profile_env() -> None:
|
|
captured_config: dict[str, ProxyConfig] = {}
|
|
|
|
def mock_run_server(config: ProxyConfig, **kwargs: object) -> None:
|
|
captured_config["config"] = config
|
|
|
|
runner = CliRunner()
|
|
with pytest.MonkeyPatch.context() as mp:
|
|
mp.setattr("headroom.proxy.server.run_server", mock_run_server)
|
|
result = runner.invoke(
|
|
main,
|
|
["proxy"],
|
|
env={"HEADROOM_SAVINGS_PROFILE": "agent-90"},
|
|
catch_exceptions=False,
|
|
)
|
|
|
|
assert result.exit_code == 0, result.output
|
|
config = captured_config["config"]
|
|
assert config.savings_profile == "agent-90"
|
|
assert proxy_pipeline_kwargs(config)["target_ratio"] == 0.10
|
|
|
|
|
|
def test_unit_router_receives_agent_target_ratio() -> None:
|
|
seen: dict[str, object] = {}
|
|
|
|
class Tokenizer:
|
|
def count_text(self, text: str) -> int:
|
|
return len(text.split())
|
|
|
|
class Router:
|
|
_runtime_target_ratio = None
|
|
|
|
def compress(self, text: str, **kwargs: object) -> RouterCompressionResult:
|
|
seen["target_ratio"] = self._runtime_target_ratio
|
|
return RouterCompressionResult(
|
|
compressed="short text",
|
|
original=text,
|
|
strategy_used=CompressionStrategy.KOMPRESS,
|
|
strategy_chain=["kompress"],
|
|
)
|
|
|
|
unit = CompressionUnit(
|
|
text=("long text " * 40) + "\nRetrieve more: hash=abc123\n",
|
|
provider="openai",
|
|
endpoint="responses",
|
|
role="assistant",
|
|
item_type="message",
|
|
cache_zone="live",
|
|
mutable=True,
|
|
min_bytes=1,
|
|
metadata={"compress_assistant": "true"},
|
|
)
|
|
|
|
result = compress_unit_with_router(
|
|
unit,
|
|
router=Router(),
|
|
tokenizer=Tokenizer(),
|
|
target_ratio=0.10,
|
|
)
|
|
|
|
assert result.modified is True
|
|
assert seen["target_ratio"] == 0.10
|
|
|
|
|
|
def test_agent_savings_check_perf_and_accuracy_report_passes(
|
|
monkeypatch,
|
|
tmp_path,
|
|
) -> None:
|
|
from headroom.perf import analyzer
|
|
|
|
monkeypatch.setattr(analyzer, "parse_log_files", lambda last_n_hours: object())
|
|
monkeypatch.setattr(
|
|
analyzer,
|
|
"build_perf_summary",
|
|
lambda report: {"savings_pct": 92.0},
|
|
)
|
|
report = tmp_path / "eval.json"
|
|
report.write_text(json.dumps({"totals": {"accuracy_rate": 1.0}}))
|
|
|
|
result = CliRunner().invoke(
|
|
main,
|
|
[
|
|
"agent-savings",
|
|
"--profile",
|
|
"agent-90",
|
|
"--check-perf",
|
|
"--accuracy-report",
|
|
str(report),
|
|
],
|
|
)
|
|
|
|
assert result.exit_code == 0, result.output
|
|
assert "92.0% savings meets 90.0%" in result.output
|
|
assert "100.0% accuracy meets 90.0%" in result.output
|
|
|
|
|
|
def test_agent_savings_accuracy_report_below_threshold_fails(
|
|
monkeypatch,
|
|
tmp_path,
|
|
) -> None:
|
|
from headroom.perf import analyzer
|
|
|
|
monkeypatch.setattr(analyzer, "parse_log_files", lambda last_n_hours: object())
|
|
monkeypatch.setattr(
|
|
analyzer,
|
|
"build_perf_summary",
|
|
lambda report: {"savings_pct": 92.0},
|
|
)
|
|
report = tmp_path / "eval.json"
|
|
report.write_text(json.dumps({"totals": {"accuracy_rate": 0.89}}))
|
|
|
|
result = CliRunner().invoke(
|
|
main,
|
|
[
|
|
"agent-savings",
|
|
"--profile",
|
|
"agent-90",
|
|
"--check-perf",
|
|
"--accuracy-report",
|
|
str(report),
|
|
],
|
|
)
|
|
|
|
assert result.exit_code != 0
|
|
assert "89.0% accuracy below 90.0%" in result.output
|
|
|
|
|
|
def test_agent_savings_requires_each_agent_to_meet_target(monkeypatch) -> None:
|
|
from headroom.perf import analyzer
|
|
from headroom.perf.analyzer import PerfRecord, PerfReport
|
|
|
|
report = PerfReport(
|
|
perf_records=[
|
|
PerfRecord(
|
|
timestamp="2026-06-10 10:00:00,000",
|
|
request_id="claude-1",
|
|
model="claude-sonnet",
|
|
client="claude",
|
|
tokens_before=1000,
|
|
tokens_after=80,
|
|
tokens_saved=920,
|
|
),
|
|
PerfRecord(
|
|
timestamp="2026-06-10 10:01:00,000",
|
|
request_id="codex-1",
|
|
model="gpt-5",
|
|
client="codex",
|
|
tokens_before=1000,
|
|
tokens_after=90,
|
|
tokens_saved=910,
|
|
),
|
|
PerfRecord(
|
|
timestamp="2026-06-10 10:02:00,000",
|
|
request_id="cursor-1",
|
|
model="gpt-5",
|
|
client="cursor",
|
|
tokens_before=1000,
|
|
tokens_after=70,
|
|
tokens_saved=930,
|
|
),
|
|
]
|
|
)
|
|
monkeypatch.setattr(analyzer, "parse_log_files", lambda last_n_hours: report)
|
|
|
|
result = CliRunner().invoke(
|
|
main,
|
|
[
|
|
"agent-savings",
|
|
"--check-perf",
|
|
"--require-agents",
|
|
"claude,codex,cursor",
|
|
],
|
|
)
|
|
|
|
assert result.exit_code == 0, result.output
|
|
assert "claude: 92.0% savings meets 90.0%" in result.output
|
|
assert "codex: 91.0% savings meets 90.0%" in result.output
|
|
assert "cursor: 93.0% savings meets 90.0%" in result.output
|
|
|
|
|
|
def test_agent_savings_required_agent_missing_fails(monkeypatch) -> None:
|
|
from headroom.perf import analyzer
|
|
from headroom.perf.analyzer import PerfRecord, PerfReport
|
|
|
|
report = PerfReport(
|
|
perf_records=[
|
|
PerfRecord(
|
|
timestamp="2026-06-10 10:00:00,000",
|
|
request_id="claude-1",
|
|
model="claude-sonnet",
|
|
client="claude",
|
|
tokens_before=1000,
|
|
tokens_after=80,
|
|
tokens_saved=920,
|
|
),
|
|
PerfRecord(
|
|
timestamp="2026-06-10 10:01:00,000",
|
|
request_id="codex-1",
|
|
model="gpt-5",
|
|
client="codex",
|
|
tokens_before=1000,
|
|
tokens_after=90,
|
|
tokens_saved=910,
|
|
),
|
|
]
|
|
)
|
|
monkeypatch.setattr(analyzer, "parse_log_files", lambda last_n_hours: report)
|
|
|
|
result = CliRunner().invoke(
|
|
main,
|
|
[
|
|
"agent-savings",
|
|
"--check-perf",
|
|
"--require-agents",
|
|
"claude,codex,cursor",
|
|
],
|
|
)
|
|
|
|
assert result.exit_code != 0
|
|
assert "missing required agent traffic: cursor" in result.output
|
|
|
|
|
|
def test_agent_savings_writes_three_agent_smoke_fixture(tmp_path) -> None:
|
|
workspace = tmp_path / "workspace"
|
|
|
|
result = CliRunner().invoke(
|
|
main,
|
|
["agent-savings", "--write-smoke-fixture", str(workspace)],
|
|
)
|
|
|
|
assert result.exit_code == 0, result.output
|
|
assert (workspace / "logs" / "proxy.log").exists()
|
|
eval_report = workspace / "agent-90-eval.json"
|
|
assert eval_report.exists()
|
|
assert "--require-agents claude,codex,cursor" in result.output
|
|
|
|
|
|
def test_agent_savings_smoke_fixture_passes_real_gate(tmp_path) -> None:
|
|
workspace = tmp_path / "workspace"
|
|
runner = CliRunner()
|
|
|
|
write_result = runner.invoke(
|
|
main,
|
|
["agent-savings", "--write-smoke-fixture", str(workspace)],
|
|
)
|
|
assert write_result.exit_code == 0, write_result.output
|
|
|
|
gate_result = runner.invoke(
|
|
main,
|
|
[
|
|
"agent-savings",
|
|
"--check-perf",
|
|
"--hours",
|
|
"0",
|
|
"--require-agents",
|
|
"claude,codex,cursor",
|
|
"--accuracy-report",
|
|
str(workspace / "agent-90-eval.json"),
|
|
],
|
|
env={"HEADROOM_WORKSPACE_DIR": str(workspace)},
|
|
)
|
|
|
|
assert gate_result.exit_code == 0, gate_result.output
|
|
assert "claude: 92.0% savings meets 90.0%" in gate_result.output
|
|
assert "codex: 91.0% savings meets 90.0%" in gate_result.output
|
|
assert "cursor: 93.0% savings meets 90.0%" in gate_result.output
|
|
assert "100.0% accuracy meets 90.0%" in gate_result.output
|
|
|
|
|
|
def test_coding_profile_min_chars_block_reaches_router_without_env_seeding(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""The block-char floor must travel on the config object, not env only.
|
|
|
|
Every other router pipeline kwarg this function builds travels on the config
|
|
object; ``min_chars_for_block`` alone was populated only from
|
|
``HEADROOM_MIN_CHARS_FOR_BLOCK`` (emitted by ``proxy_env()``). A proxy whose
|
|
config carried ``savings_profile="coding"`` but whose process env was never
|
|
seeded applied every sibling coding knob while this floor silently stayed at
|
|
``ContentRouterConfig.min_chars_for_block_compression`` (500) instead of the
|
|
profile's 25 — a 20x gap on the gate that governs tool_result blocks.
|
|
|
|
Note this does not make the profile fully config-deliverable: fields whose
|
|
consumers read ``os.environ`` directly (``cross_turn_dedup`` via
|
|
ContentRouter, ``tool_search`` via the Anthropic handler) never pass through
|
|
this function and remain seed-dependent.
|
|
"""
|
|
monkeypatch.delenv("HEADROOM_MIN_CHARS_FOR_BLOCK", raising=False)
|
|
|
|
class _Config:
|
|
savings_profile = "coding"
|
|
min_tokens_to_crush = 500
|
|
|
|
kwargs = proxy_pipeline_kwargs(_Config())
|
|
assert kwargs["min_chars_for_block_compression"] == 25
|
|
assert kwargs["min_tokens_to_compress"] == 10
|
|
|
|
|
|
def test_explicit_min_chars_block_env_overrides_the_profile(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""An explicit operator override still wins over the profile value."""
|
|
monkeypatch.setenv("HEADROOM_MIN_CHARS_FOR_BLOCK", "120")
|
|
|
|
class _Config:
|
|
savings_profile = "coding"
|
|
min_tokens_to_crush = 500
|
|
|
|
assert proxy_pipeline_kwargs(_Config())["min_chars_for_block_compression"] == 120
|