Files
headroom/benchmarks/index_proof_table.py
Parideboy a385ea8e60 fix(benchmarks): drop the duplicate protect_recent=0 table from the proof harness (#4754)
## Description

`benchmarks/index_proof_table.py` printed two tables, "default
(protect_recent=4)" and "full corpus (protect_recent=0)". The committed
`benchmarks/results/index_proof_table.txt` showed them identical in
every row and in the total, so the second run and its label carried no
information — and the comment above them claimed the default "shields
almost every tool result", which the output contradicts.

They cannot ever differ. `ContentRouter` applies `protect_recent` only
to content detected as `SOURCE_CODE`:

```python
is_code = detection.content_type == ContentType.SOURCE_CODE
if protect_recent > 0 and messages_from_end <= protect_recent and is_code:
```

Every scenario in this harness is a JSON tool payload. All ten payloads
detect as `JSON_ARRAY` under **both** backends (pure-Python and
`HEADROOM_DETECT_BACKEND=rust`), so the guard never fires at any
`protect_recent` value. This is not a stale committed artifact that a
rerun would fix.

Of the three options on the issue, this takes the first: one table with
the limitation stated, rather than a duplicate or a behavior change to
the compressor.

Closes #4732

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- `benchmarks/index_proof_table.py`: print one table (the default
config) and replace the misleading comment with the actual reason the
`protect_recent` comparison is a no-op for JSON payloads.
- `benchmarks/results/index_proof_table.txt`: regenerated. The table
body is unchanged; the duplicate second table is gone.
- `tests/test_index_proof_table.py`: new output-level tests asserting
the harness prints exactly one table and that the docs Markdown block
states the config it was measured under. Both fail on the base revision
and pass with this change.
- `tests/test_transforms_content_router.py`: new
`test_protect_recent_does_not_shield_non_code_tool_payloads` pins the
router reason the two tables could never differ.
- `headroom/compress.py`: `CompressConfig.protect_recent` docstring said
"Don't compress the last N messages"; it guards recent **code** only.
- `docs/content/docs/context-management.mdx`,
`docs/content/docs/proxy.mdx`, `docs/content/docs/index.mdx`,
`README.md`: same overstatement corrected. The published table now names
the configuration it was measured under.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_index_proof_table.py tests/test_transforms_content_router.py -q
109 passed, 1 warning in 14.83s

$ uv run ruff check benchmarks/index_proof_table.py headroom/compress.py \
    tests/test_index_proof_table.py tests/test_transforms_content_router.py
All checks passed!

$ uv run ruff format --check tests/test_index_proof_table.py
1 file already formatted
```

Base-revision check for the new output tests (harness restored to
`318e8e08`):

```text
$ uv run pytest tests/test_index_proof_table.py -q
FAILED tests/test_index_proof_table.py::test_proof_table_prints_exactly_one_table
FAILED tests/test_index_proof_table.py::test_proof_markdown_block_is_the_default_config
2 failed in 5.15s
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, `uv` venv at the branch head
`9b7e2f95` (rebased onto `acb58a0c`); content detection ran on the
pure-Python backend (no ONNX Runtime 1.24+), and every payload was
re-checked under `HEADROOM_DETECT_BACKEND=rust`.
- Exact command / steps: `uv run python benchmarks/index_proof_table.py`
on the base revision, then the same command on this branch, then `git
diff` against the committed `benchmarks/results/index_proof_table.txt`.
- Observed result: base revision printed two tables identical in all
four rows and in the total (178,024 → 104,261, 41%). This branch prints
one table with the same numbers, and its output matches the committed
result file byte for byte once the first line is excluded (that line
names whichever detector backend the host has).
- Not tested: whether the committed numbers would change under the
native detector — also left open on the issue. They do not: both
backends classify all ten payloads as `JSON_ARRAY`, so detection routes
them identically.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. Only a benchmark script, its
committed output, tests, and docs change.
- Minimum rollout channel: stable, no gating.
- Stable/default behavior changed: no. `ContentRouter`,
`CompressConfig`, and `compress()` behavior are untouched; only a
docstring and documentation prose changed. The published table's numbers
are identical to before.
- Kill switch / disable path: not applicable.
`benchmarks/index_proof_table.py` is a manual, offline script with no
runtime surface.
- Unsafe override required: no.
- Qualification impact: none. Serialization, caching, and compression
output are unchanged; the new tests are offline and stubbed.
- Rollback path: revert this commit. The harness returns to printing the
duplicate table; no state or migration is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

`mypy headroom` left unchecked: the touched Python files add no type
surface beyond a benchmark script and a test module, and mypy was not
run locally.

Review feedback addressed: the initial version of this PR added only a
router-level test, which passes on the base revision and so could not
catch the duplicate table returning. The two output-level tests in
`tests/test_index_proof_table.py` replace that gap, and the
base-revision failure above is their proof. The `context-management.mdx`
file-read claim was also too broad — read protection releases
confidently non-code data (JSON, CSV/tabular, build/test logs, git
diffs, HTML, search output), so the wording now says so.

`CHANGELOG.md` is untouched, per CONTRIBUTING.


The branch was rebased onto `acb58a0c` (4 upstream commits: #4727,
#4728, #4735, #4743) with no file overlap. Re-verified after the rebase:
109 focused tests pass, `ruff check`/`format --check` clean.
`benchmarks/index_proof_table.py` is byte-identical between `318e8e08`
and `acb58a0c`, so the base-revision failure above still applies
unchanged.
2026-11-10 12:35:38 -06:00

138 lines
4.9 KiB
Python

"""Regenerate the "Proof" savings table published on the docs landing page.
WHY THIS EXISTS
---------------
docs/content/docs/index.mdx published four precise before/after token counts
with no reproducible source. The nearest harness,
``real_world_agent_benchmark.py``, seeded nothing, so its corpus differed on
every run and the published figures could not be reproduced by anyone,
including us. A number on the front page of the docs that nobody can
regenerate is a liability, not evidence.
This script fixes the reproducibility half. It seeds the generators, builds the
same scenarios, and measures tokens through the real tokenizer and the real
``compress()`` path. No network, no API key, no model call: the table is a
statement about token counts, and token counts are computable locally.
uv run python benchmarks/index_proof_table.py
Deterministic: same seed in, same numbers out, on any machine.
"""
from __future__ import annotations
import argparse
import json
from real_world_agent_benchmark import ( # noqa: E402
DEFAULT_SEED,
create_codebase_exploration_scenario,
create_issue_triage_scenario,
create_sre_debugging_scenario,
generate_github_code_search,
seed_everything,
)
from headroom import CompressConfig, compress
from headroom.providers.openai_compatible import OpenAICompatibleTokenCounter
MODEL = "gpt-5.6"
def _tool_messages(tools: list[dict]) -> list[dict]:
"""The tool payloads as the proxy would actually see them on the wire."""
return [
{
"role": "tool",
"tool_call_id": f"call_{i}",
"content": json.dumps(t["result"]),
}
for i, t in enumerate(tools)
]
def measure(label: str, tools: list[dict], tok, config: CompressConfig) -> dict:
msgs = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Analyse the tool output and answer."},
*_tool_messages(tools),
]
before = sum(tok.count_text(m["content"]) for m in msgs if m["role"] == "tool")
result = compress(msgs, model=MODEL, config=config)
after = sum(
tok.count_text(m["content"])
for m in result.messages
if m.get("role") == "tool" and isinstance(m.get("content"), str)
)
saved = before - after
return {
"scenario": label,
"before": before,
"after": after,
"saved": saved,
"savings_pct": (saved / before * 100) if before else 0.0,
"transforms": sorted(set(result.transforms_applied)),
}
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--seed", type=int, default=DEFAULT_SEED)
args = ap.parse_args()
tok = OpenAICompatibleTokenCounter(model=MODEL)
# One configuration: the default, i.e. what a coding agent actually gets.
#
# A second `protect_recent=0` run used to be printed here, and came out
# identical in every row and in the total (#4732). Not a coincidence: the
# router applies `protect_recent` only to content detected as SOURCE_CODE
# ("Don't compress recent CODE"). These scenarios are JSON tool payloads --
# JSON_ARRAY even under the native detector -- so the guard never fires and
# the second table carried no information. Printing it again would only
# relabel a duplicate, so it is stated here once instead.
cfg = CompressConfig()
# Built in a fixed order: every generator draws from the same global RNG,
# so re-ordering these lines changes every number below.
print(f"seed={args.seed} model={MODEL} tokenizer={type(tok._tokenizer).__name__}")
seed_everything(args.seed)
rows = [
measure(
"Code search (100 results)",
[generate_github_code_search("JWT authentication middleware", num_results=100)],
tok,
cfg,
),
measure("SRE incident debugging", create_sre_debugging_scenario().tools, tok, cfg),
measure("Codebase exploration", create_codebase_exploration_scenario().tools, tok, cfg),
measure("GitHub issue triage", create_issue_triage_scenario().tools, tok, cfg),
]
print("\n=== default (protect_recent=4) ===")
print(f"{'Scenario':<30} {'Before':>10} {'After':>10} {'Savings':>9}")
print("-" * 62)
for r in rows:
print(f"{r['scenario']:<30} {r['before']:>10,} {r['after']:>10,} {r['savings_pct']:>8.0f}%")
tb = sum(r["before"] for r in rows)
ta = sum(r["after"] for r in rows)
print("-" * 62)
print(f"{'TOTAL':<30} {tb:>10,} {ta:>10,} {(tb - ta) / tb * 100:>8.0f}%")
print(
"\n\nMarkdown for docs/content/docs/index.mdx "
"(default config, which must be stated on the page):\n"
)
print("| Scenario | Before | After | Savings |")
print("|---|---|---|---|")
for r in rows:
print(
f"| {r['scenario']} | {r['before']:,} | {r['after']:,} | **{r['savings_pct']:.0f}%** |"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())