mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(benchmarks): drop the duplicate protect_recent=0 table from the proof harness (#4754)
## Description `benchmarks/index_proof_table.py` printed two tables, "default (protect_recent=4)" and "full corpus (protect_recent=0)". The committed `benchmarks/results/index_proof_table.txt` showed them identical in every row and in the total, so the second run and its label carried no information — and the comment above them claimed the default "shields almost every tool result", which the output contradicts. They cannot ever differ. `ContentRouter` applies `protect_recent` only to content detected as `SOURCE_CODE`: ```python is_code = detection.content_type == ContentType.SOURCE_CODE if protect_recent > 0 and messages_from_end <= protect_recent and is_code: ``` Every scenario in this harness is a JSON tool payload. All ten payloads detect as `JSON_ARRAY` under **both** backends (pure-Python and `HEADROOM_DETECT_BACKEND=rust`), so the guard never fires at any `protect_recent` value. This is not a stale committed artifact that a rerun would fix. Of the three options on the issue, this takes the first: one table with the limitation stated, rather than a duplicate or a behavior change to the compressor. Closes #4732 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update ## Changes Made - `benchmarks/index_proof_table.py`: print one table (the default config) and replace the misleading comment with the actual reason the `protect_recent` comparison is a no-op for JSON payloads. - `benchmarks/results/index_proof_table.txt`: regenerated. The table body is unchanged; the duplicate second table is gone. - `tests/test_index_proof_table.py`: new output-level tests asserting the harness prints exactly one table and that the docs Markdown block states the config it was measured under. Both fail on the base revision and pass with this change. - `tests/test_transforms_content_router.py`: new `test_protect_recent_does_not_shield_non_code_tool_payloads` pins the router reason the two tables could never differ. - `headroom/compress.py`: `CompressConfig.protect_recent` docstring said "Don't compress the last N messages"; it guards recent **code** only. - `docs/content/docs/context-management.mdx`, `docs/content/docs/proxy.mdx`, `docs/content/docs/index.mdx`, `README.md`: same overstatement corrected. The published table now names the configuration it was measured under. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_index_proof_table.py tests/test_transforms_content_router.py -q 109 passed, 1 warning in 14.83s $ uv run ruff check benchmarks/index_proof_table.py headroom/compress.py \ tests/test_index_proof_table.py tests/test_transforms_content_router.py All checks passed! $ uv run ruff format --check tests/test_index_proof_table.py 1 file already formatted ``` Base-revision check for the new output tests (harness restored to `318e8e08`): ```text $ uv run pytest tests/test_index_proof_table.py -q FAILED tests/test_index_proof_table.py::test_proof_table_prints_exactly_one_table FAILED tests/test_index_proof_table.py::test_proof_markdown_block_is_the_default_config 2 failed in 5.15s ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.11, `uv` venv at the branch head `9b7e2f95` (rebased onto `acb58a0c`); content detection ran on the pure-Python backend (no ONNX Runtime 1.24+), and every payload was re-checked under `HEADROOM_DETECT_BACKEND=rust`. - Exact command / steps: `uv run python benchmarks/index_proof_table.py` on the base revision, then the same command on this branch, then `git diff` against the committed `benchmarks/results/index_proof_table.txt`. - Observed result: base revision printed two tables identical in all four rows and in the total (178,024 → 104,261, 41%). This branch prints one table with the same numbers, and its output matches the committed result file byte for byte once the first line is excluded (that line names whichever detector backend the host has). - Not tested: whether the committed numbers would change under the native detector — also left open on the issue. They do not: both backends classify all ten payloads as `JSON_ARRAY`, so detection routes them identically. ## Runtime Rollout Safety - Rollout-managed feature(s): none. Only a benchmark script, its committed output, tests, and docs change. - Minimum rollout channel: stable, no gating. - Stable/default behavior changed: no. `ContentRouter`, `CompressConfig`, and `compress()` behavior are untouched; only a docstring and documentation prose changed. The published table's numbers are identical to before. - Kill switch / disable path: not applicable. `benchmarks/index_proof_table.py` is a manual, offline script with no runtime surface. - Unsafe override required: no. - Qualification impact: none. Serialization, caching, and compression output are unchanged; the new tests are offline and stubbed. - Rollback path: revert this commit. The harness returns to printing the duplicate table; no state or migration is involved. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes `mypy headroom` left unchecked: the touched Python files add no type surface beyond a benchmark script and a test module, and mypy was not run locally. Review feedback addressed: the initial version of this PR added only a router-level test, which passes on the base revision and so could not catch the duplicate table returning. The two output-level tests in `tests/test_index_proof_table.py` replace that gap, and the base-revision failure above is their proof. The `context-management.mdx` file-read claim was also too broad — read protection releases confidently non-code data (JSON, CSV/tabular, build/test logs, git diffs, HTML, search output), so the wording now says so. `CHANGELOG.md` is untouched, per CONTRIBUTING. The branch was rebased onto `acb58a0c` (4 upstream commits: #4727, #4728, #4735, #4743) with no file overlap. Re-verified after the rebase: 109 focused tests pass, `ruff check`/`format --check` clean. `benchmarks/index_proof_table.py` is byte-identical between `318e8e08` and `acb58a0c`, so the base-revision failure above still applies unchanged.
This commit is contained in:
@@ -180,8 +180,8 @@ is the TypeScript SDK — a library you import
|
||||
## Proof: measured token savings
|
||||
|
||||
Four scenarios built from real MCP server output formats, measured with the
|
||||
provider tokenizer and the shipped `compress()`. Seeded and offline, so you get
|
||||
the same numbers we did:
|
||||
provider tokenizer and the shipped `compress()` at its default configuration.
|
||||
Seeded and offline, so you get the same numbers we did:
|
||||
|
||||
```bash
|
||||
uv run python benchmarks/index_proof_table.py --seed 20260902
|
||||
|
||||
@@ -82,61 +82,51 @@ def main() -> int:
|
||||
|
||||
tok = OpenAICompatibleTokenCounter(model=MODEL)
|
||||
|
||||
# Two configurations, because the default protects the tail of the
|
||||
# conversation and these scenarios are only 3-5 messages long. Under the
|
||||
# default, protect_recent=4 shields almost every tool result and the
|
||||
# measurement says more about the guard than about the compressor.
|
||||
# One configuration: the default, i.e. what a coding agent actually gets.
|
||||
#
|
||||
# "default" - what a coding agent actually gets out of the box.
|
||||
# "full" - protect_recent=0, every tool result eligible. This is the
|
||||
# honest number for "how far can this payload compress",
|
||||
# and it is the one a benchmark table should quote, LABELLED.
|
||||
configs = {
|
||||
"default (protect_recent=4)": CompressConfig(),
|
||||
"full corpus (protect_recent=0)": CompressConfig(protect_recent=0),
|
||||
}
|
||||
# A second `protect_recent=0` run used to be printed here, and came out
|
||||
# identical in every row and in the total (#4732). Not a coincidence: the
|
||||
# router applies `protect_recent` only to content detected as SOURCE_CODE
|
||||
# ("Don't compress recent CODE"). These scenarios are JSON tool payloads --
|
||||
# JSON_ARRAY even under the native detector -- so the guard never fires and
|
||||
# the second table carried no information. Printing it again would only
|
||||
# relabel a duplicate, so it is stated here once instead.
|
||||
cfg = CompressConfig()
|
||||
|
||||
# Built in a fixed order: every generator draws from the same global RNG,
|
||||
# so re-ordering these lines changes every number below.
|
||||
print(f"seed={args.seed} model={MODEL} tokenizer={type(tok._tokenizer).__name__}")
|
||||
|
||||
out: dict[str, list[dict]] = {}
|
||||
for cname, cfg in configs.items():
|
||||
# Reseed per configuration so both see a byte-identical corpus.
|
||||
seed_everything(args.seed)
|
||||
rows = [
|
||||
measure(
|
||||
"Code search (100 results)",
|
||||
[generate_github_code_search("JWT authentication middleware", num_results=100)],
|
||||
tok,
|
||||
cfg,
|
||||
),
|
||||
measure("SRE incident debugging", create_sre_debugging_scenario().tools, tok, cfg),
|
||||
measure("Codebase exploration", create_codebase_exploration_scenario().tools, tok, cfg),
|
||||
measure("GitHub issue triage", create_issue_triage_scenario().tools, tok, cfg),
|
||||
]
|
||||
out[cname] = rows
|
||||
seed_everything(args.seed)
|
||||
rows = [
|
||||
measure(
|
||||
"Code search (100 results)",
|
||||
[generate_github_code_search("JWT authentication middleware", num_results=100)],
|
||||
tok,
|
||||
cfg,
|
||||
),
|
||||
measure("SRE incident debugging", create_sre_debugging_scenario().tools, tok, cfg),
|
||||
measure("Codebase exploration", create_codebase_exploration_scenario().tools, tok, cfg),
|
||||
measure("GitHub issue triage", create_issue_triage_scenario().tools, tok, cfg),
|
||||
]
|
||||
|
||||
print(f"\n=== {cname} ===")
|
||||
print(f"{'Scenario':<30} {'Before':>10} {'After':>10} {'Savings':>9}")
|
||||
print("-" * 62)
|
||||
for r in rows:
|
||||
print(
|
||||
f"{r['scenario']:<30} {r['before']:>10,} {r['after']:>10,} "
|
||||
f"{r['savings_pct']:>8.0f}%"
|
||||
)
|
||||
tb = sum(r["before"] for r in rows)
|
||||
ta = sum(r["after"] for r in rows)
|
||||
print("-" * 62)
|
||||
print(f"{'TOTAL':<30} {tb:>10,} {ta:>10,} {(tb - ta) / tb * 100:>8.0f}%")
|
||||
print("\n=== default (protect_recent=4) ===")
|
||||
print(f"{'Scenario':<30} {'Before':>10} {'After':>10} {'Savings':>9}")
|
||||
print("-" * 62)
|
||||
for r in rows:
|
||||
print(f"{r['scenario']:<30} {r['before']:>10,} {r['after']:>10,} {r['savings_pct']:>8.0f}%")
|
||||
tb = sum(r["before"] for r in rows)
|
||||
ta = sum(r["after"] for r in rows)
|
||||
print("-" * 62)
|
||||
print(f"{'TOTAL':<30} {tb:>10,} {ta:>10,} {(tb - ta) / tb * 100:>8.0f}%")
|
||||
|
||||
print(
|
||||
"\n\nMarkdown for docs/content/docs/index.mdx "
|
||||
"(full-corpus config, which must be stated on the page):\n"
|
||||
"(default config, which must be stated on the page):\n"
|
||||
)
|
||||
print("| Scenario | Before | After | Savings |")
|
||||
print("|---|---|---|---|")
|
||||
for r in out["full corpus (protect_recent=0)"]:
|
||||
for r in rows:
|
||||
print(
|
||||
f"| {r['scenario']} | {r['before']:,} | {r['after']:,} | **{r['savings_pct']:.0f}%** |"
|
||||
)
|
||||
|
||||
@@ -11,18 +11,8 @@ GitHub issue triage 46,067 32,429 30%
|
||||
--------------------------------------------------------------
|
||||
TOTAL 178,024 104,261 41%
|
||||
|
||||
=== full corpus (protect_recent=0) ===
|
||||
Scenario Before After Savings
|
||||
--------------------------------------------------------------
|
||||
Code search (100 results) 17,199 13,597 21%
|
||||
SRE incident debugging 55,957 24,340 57%
|
||||
Codebase exploration 58,801 33,895 42%
|
||||
GitHub issue triage 46,067 32,429 30%
|
||||
--------------------------------------------------------------
|
||||
TOTAL 178,024 104,261 41%
|
||||
|
||||
|
||||
Markdown for docs/content/docs/index.mdx (full-corpus config, which must be stated on the page):
|
||||
Markdown for docs/content/docs/index.mdx (default config, which must be stated on the page):
|
||||
|
||||
| Scenario | Before | After | Savings |
|
||||
|---|---|---|---|
|
||||
|
||||
@@ -39,7 +39,7 @@ System messages are never dropped. They contain critical instructions, persona d
|
||||
|
||||
### Turn protection
|
||||
|
||||
The last N messages are always preserved, ensuring the model has immediate conversational context. The one-function `compress()` API exposes this as `CompressConfig.protect_recent`, which defaults to `4` messages (roughly the last 2 user/assistant turns) -- see `headroom/compress.py:110-112`.
|
||||
The last N messages are always preserved, ensuring the model has immediate conversational context. The one-function `compress()` API exposes this as `CompressConfig.protect_recent`, which defaults to `4` messages -- see `headroom/compress.py`. It guards recent **code** only: tool payloads detected as JSON, logs or text are not shielded by it. File reads are held back by read protection instead, which protects by default and releases only confidently non-code data (JSON, CSV/tabular, build/test logs, git diffs, HTML, search output) -- see `_read_output_should_be_protected`.
|
||||
|
||||
## Configuration
|
||||
|
||||
|
||||
@@ -82,7 +82,9 @@ signal survives even when the surrounding noise is compressed away
|
||||
aggressively.
|
||||
|
||||
Measured on local, seeded test data built from real MCP server output
|
||||
formats. These are not production telemetry, and they are reproducible:
|
||||
formats, run through the default `compress()` configuration
|
||||
(`protect_recent=4`). These are not production telemetry, and they are
|
||||
reproducible:
|
||||
|
||||
| Scenario | Before | After | Savings |
|
||||
|---|---|---|---|
|
||||
|
||||
@@ -645,7 +645,7 @@ Keep carrying both yourself and send them upstream unchanged. Two consequences w
|
||||
| `session_id` | string | unset | Opt into [session mode](#session-mode): Headroom keeps this conversation's replay state itself, so you can resend the raw conversation every turn and get a byte-identical prefix back. Non-empty, at most 256 characters; anything else is a `400`. Cannot be combined with `compress_user_messages` (`400`). |
|
||||
| `compress_user_messages` | boolean | `false` | Also compress user-role messages. |
|
||||
| `target_ratio` | number | unset | Target compression ratio. |
|
||||
| `protect_recent` | integer | unset | Leave the last N messages uncompressed. |
|
||||
| `protect_recent` | integer | unset | Leave recent code uncompressed in the last N messages (non-code tool payloads are not shielded). |
|
||||
| `protect_analysis_context` | boolean | unset | Preserve analysis context blocks. |
|
||||
|
||||
**`config.mode` values:**
|
||||
|
||||
@@ -114,8 +114,10 @@ class CompressConfig:
|
||||
agents where tool definitions and instructions must not be altered."""
|
||||
|
||||
protect_recent: int = 4
|
||||
"""Don't compress the last N messages (they're the active conversation).
|
||||
Set 0 to compress everything."""
|
||||
"""Don't compress CODE in the last N messages (they're the active
|
||||
conversation). Applies only to content detected as source code; JSON,
|
||||
log and text tool payloads are not shielded by it. Set 0 to compress
|
||||
everything."""
|
||||
|
||||
protect_analysis_context: bool = True
|
||||
"""Detect 'analyze'/'review' intent and protect code from compression."""
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
"""The proof table printed for the docs must stay a single, labelled table (#4732).
|
||||
|
||||
``benchmarks/index_proof_table.py`` used to print "default (protect_recent=4)"
|
||||
and "full corpus (protect_recent=0)". Both tables were identical in every row,
|
||||
because ``ContentRouter`` applies ``protect_recent`` only to content detected as
|
||||
SOURCE_CODE and every scenario here is a JSON payload
|
||||
(``tests/test_transforms_content_router.py::
|
||||
test_protect_recent_does_not_shield_non_code_tool_payloads`` pins that reason).
|
||||
The second table could therefore never differ, so it was removed.
|
||||
|
||||
These tests pin the OUTPUT, not the compression: everything expensive is
|
||||
monkeypatched away, so they stay fast and offline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import types
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
BENCHMARKS_DIR = Path(__file__).resolve().parents[1] / "benchmarks"
|
||||
|
||||
SCENARIOS = (
|
||||
"Code search (100 results)",
|
||||
"SRE incident debugging",
|
||||
"Codebase exploration",
|
||||
"GitHub issue triage",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def harness(monkeypatch: pytest.MonkeyPatch) -> types.ModuleType:
|
||||
"""Import the harness with its corpus generators and compression stubbed out."""
|
||||
# The harness imports `real_world_agent_benchmark` as a sibling module, which
|
||||
# only resolves when it is run as a script with `benchmarks/` on sys.path.
|
||||
monkeypatch.syspath_prepend(str(BENCHMARKS_DIR))
|
||||
sys.modules.pop("benchmarks.index_proof_table", None)
|
||||
|
||||
module = pytest.importorskip("benchmarks.index_proof_table")
|
||||
|
||||
class _FakeTokenizer:
|
||||
_tokenizer = object()
|
||||
|
||||
def __init__(self, model: str | None = None) -> None:
|
||||
self.model = model
|
||||
|
||||
numbers = {"before": 1_000, "after": 500}
|
||||
|
||||
def _fake_measure(label: str, tools: list[dict], tok: Any, config: Any) -> dict:
|
||||
return {
|
||||
"scenario": label,
|
||||
"before": numbers["before"],
|
||||
"after": numbers["after"],
|
||||
"saved": numbers["before"] - numbers["after"],
|
||||
"savings_pct": 50.0,
|
||||
"transforms": [],
|
||||
}
|
||||
|
||||
tool = {"result": [{"path": "src/a.py", "score": 1}]}
|
||||
monkeypatch.setattr(module, "seed_everything", lambda seed: seed)
|
||||
monkeypatch.setattr(module, "measure", _fake_measure)
|
||||
monkeypatch.setattr(module, "OpenAICompatibleTokenCounter", _FakeTokenizer)
|
||||
monkeypatch.setattr(module, "generate_github_code_search", lambda *a, **k: tool)
|
||||
monkeypatch.setattr(
|
||||
module,
|
||||
"create_sre_debugging_scenario",
|
||||
lambda: types.SimpleNamespace(tools=[tool]),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
module,
|
||||
"create_codebase_exploration_scenario",
|
||||
lambda: types.SimpleNamespace(tools=[tool]),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
module,
|
||||
"create_issue_triage_scenario",
|
||||
lambda: types.SimpleNamespace(tools=[tool]),
|
||||
)
|
||||
return module
|
||||
|
||||
|
||||
def test_proof_table_prints_exactly_one_table(
|
||||
harness: types.ModuleType, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
"""A second configuration re-introduces a table that can only duplicate the first."""
|
||||
monkeypatch.setattr(sys, "argv", ["index_proof_table.py", "--seed", "20260902"])
|
||||
assert harness.main() == 0
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert out.count("=== default (protect_recent=4) ===") == 1, out
|
||||
assert "protect_recent=0" not in out
|
||||
|
||||
|
||||
def test_proof_markdown_block_is_the_default_config(
|
||||
harness: types.ModuleType, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
"""The block pasted into the docs must state the config it was measured under."""
|
||||
monkeypatch.setattr(sys, "argv", ["index_proof_table.py", "--seed", "20260902"])
|
||||
assert harness.main() == 0
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert "Markdown for docs/content/docs/index.mdx (default config" in out
|
||||
markdown = out.split("Markdown for docs/content/docs/index.mdx", 1)[1]
|
||||
for scenario in SCENARIOS:
|
||||
assert f"| {scenario} | 1,000 | 500 | **50%** |" in markdown
|
||||
@@ -512,6 +512,59 @@ def test_force_kompress_apply_lightweight_detection_protects_recent_code(
|
||||
assert result.transforms_applied == ["router:protected:recent_code"]
|
||||
|
||||
|
||||
def test_protect_recent_does_not_shield_non_code_tool_payloads(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""``protect_recent`` guards recent CODE only (#4732).
|
||||
|
||||
JSON/log/text tool payloads stay eligible at any ``protect_recent`` value,
|
||||
which is why the index-proof harness cannot show a difference between
|
||||
``protect_recent=4`` and ``protect_recent=0`` on its JSON scenarios.
|
||||
"""
|
||||
|
||||
class FakeTokenizer:
|
||||
def count_text(self, text: str) -> int:
|
||||
return len(text.split())
|
||||
|
||||
router = ContentRouter(ContentRouterConfig(protect_recent_code=4))
|
||||
content = json.dumps([{"path": f"src/module_{i}.py", "score": i} for i in range(200)])
|
||||
|
||||
monkeypatch.setattr(
|
||||
content_router_module,
|
||||
"_detect_content",
|
||||
lambda content: DetectionResult(ContentType.JSON_ARRAY, 1.0, {}),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
router,
|
||||
"compress",
|
||||
lambda content, context="", bias=1.0, precomputed_detection=None: RouterCompressionResult(
|
||||
# CCR marker -> the original was stored and is retrievable, so the
|
||||
# #1307 reversibility gate accepts this lossy KOMPRESS tool result.
|
||||
compressed="compressed <<ccr:tool>>",
|
||||
original=content,
|
||||
strategy_used=CompressionStrategy.KOMPRESS,
|
||||
routing_log=[
|
||||
RoutingDecision(
|
||||
content_type=ContentType.JSON_ARRAY,
|
||||
strategy=CompressionStrategy.KOMPRESS,
|
||||
original_tokens=len(content.split()),
|
||||
compressed_tokens=1,
|
||||
)
|
||||
],
|
||||
),
|
||||
)
|
||||
|
||||
result = router.apply(
|
||||
[{"role": "tool", "content": content}],
|
||||
FakeTokenizer(),
|
||||
min_tokens_to_compress=10,
|
||||
protect_recent=4,
|
||||
)
|
||||
|
||||
assert result.messages[0]["content"] == "compressed <<ccr:tool>>"
|
||||
assert "router:protected:recent_code" not in result.transforms_applied
|
||||
|
||||
|
||||
def test_diff_strategy_does_not_fallback_to_kompress_when_diff_is_noop(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
|
||||
Reference in New Issue
Block a user