Files
headroom/tests/test_content_router_exclude_tools.py
gglucassandClaude Opus 5.5 1512c06235 fix(transforms): keep file-read protection in token mode under the coding profile (#3885)
## Description

In token mode, the `coding` and `general` savings profiles strip
excluded-tool output of all protection, so a Claude Code `Read` is
lossy-compressed even as the newest message.

The chain:

- `proxy/server.py` sets `router_config.protect_recent_reads_fraction =
0.3` in token mode: excluded-tool output in the last `max(4, 0.3 * n)`
messages is protected, older output may compress.
- `proxy_pipeline_kwargs()` passes `read_protection_window =
profile.protect_recent` on every request. `coding` and `general` set
`protect_recent=0`. That 0 is a positional guard meant for cache mode
(#2145); the coding profile's own comment says the code working set
"stays byte-exact via protect_reads below - a TYPE guard on file reads",
and "NEVER lossy-compress a file read (the agent patches exact bytes)".
- `ContentRouter.apply` lets the runtime window replace the fraction
window whenever the fraction is > 0 (#2105). The window becomes 0,
`messages_from_end <= 0` is never true (the newest message is 1), and
every excluded-tool result falls through to compression at any position.

Effect in token mode with `coding` or `general`:

- `Read`, `Skill` and `read_file` (`DEFAULT_BYTE_EXACT_EXCLUDE_TOOLS`
minus the verbatim set) lose byte-exact protection entirely.
- `Grep`, `Glob`, `Edit` and `Write` lose recent-output protection (they
were meant to get lossless compaction only while recent).
- `WebFetch` / `WebSearch` are unaffected
(`DEFAULT_VERBATIM_EXCLUDE_TOOLS`, protected at any age).
- `HEADROOM_PROTECT_READS` does not cover this: it only protects SHELL
reads (`_is_read_command` on the Bash command), so `cat NOTES.md` via
Bash stays protected while `Read NOTES.md` does not.

`general` seeds `HEADROOM_MODE=token`, so every `general` deployment
hits this by default. `coding` defaults to cache mode and hits it with
`--mode token`. `agent-90` (2) and `balanced` (4) pass a positive window
and are unchanged.

Fix: a runtime `read_protection_window` of 0 means "no override", not
"protect nothing". Only a positive window narrows the fraction window; 0
falls back to `max(4, 0.3 * n)`. This keeps #2105's precedence (a
fraction of 0.0 still ignores the kwarg) and leaves `protect_recent=0`
alone, since it still governs the positional guard for all tool output
in cache mode.

**This is a compression behaviour change that LOWERS savings** on recent
`Read`/`Skill`/`read_file` output (now byte-exact) and recent
`Grep`/`Glob`/`Edit`/`Write` output (now lossless compaction or
verbatim) in token mode with the `coding` or `general` profile. That is
the documented intent of both token mode's fraction window and the
coding profile. Excluded-tool output beyond the window still ages out to
compression, as token mode intends.

This is a prerequisite for #3881 (judge the `code_aware` Kompress
fallback in tokens). On main, the unprotected code `Read`s in token mode
are only left intact by accident: the word-vs-token bug #3881 fixes
makes their Kompress fallback lose. With #3881 alone they get flattened
(see proof below). #3881's measured gains were on newest-message `Read`s
that this PR protects.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_router.py` (`ContentRouter.apply`): the
runtime `read_protection_window` only replaces the fraction window when
it is > 0. A window of 0 (or less) is ignored.
- `tests/test_content_router_exclude_tools.py`:
`test_token_mode_coding_profile_keeps_recent_read_byte_exact`,
parametrized. It builds a real `HeadroomProxy(mode="token",
savings_profile="coding")`, takes kwargs from
`proxy_pipeline_kwargs(config)` (asserts `read_protection_window == 0`),
and mocks Kompress (no ONNX). A newest-message `Read` must stay
byte-exact with `router:excluded:tool`; the same `Read` with 40 messages
after it must still compress.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_content_router_exclude_tools.py \
    tests/test_transforms/test_content_router.py tests/test_transforms_content_router.py \
    tests/test_agent_savings.py tests/test_content_router_concurrency.py \
    tests/test_force_kompress_all.py tests/test_lossless_excluded_compaction.py \
    tests/test_observed_wire_shapes.py tests/test_openai_responses_read_protection.py \
    tests/test_proxy/test_header_safe_transforms.py tests/test_websearch_tool_result_protection.py -q
274 passed, 2 skipped

# new test with headroom/ change reverted (main ffc6edb4):
FAILED tests/test_content_router_exclude_tools.py::test_token_mode_coding_profile_keeps_recent_read_byte_exact[0]
E   AssertionError: newest Read must stay byte-exact in token mode
1 failed, 7 passed

$ uvx ruff@0.16.8 check headroom/transforms/content_router.py tests/test_content_router_exclude_tools.py
All checks passed!
$ uvx ruff@0.16.8 format --check headroom/transforms/content_router.py tests/test_content_router_exclude_tools.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/transforms/content_router.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 15 (16 GB), Python 3.11, `uv run --frozen --extra
dev --extra proxy --extra code`, real ONNX Kompress
(`chopratejas/kompress-v2-base` from the local HF cache), tree-sitter
installed. No mocks.
- Exact command / steps: `seed_proxy_env_defaults()`, then
`HeadroomProxy(ProxyConfig(mode=<mode>, savings_profile="coding",
code_aware_enabled=True))` and `proxy.anthropic_pipeline.apply(messages,
model="claude-sonnet-4-5", model_limit=200000, frozen_message_count=0,
prefix_replay_guaranteed=True, **proxy_pipeline_kwargs(cfg))`, the
handler's call shape. Messages are `[user, assistant tool_use Read, user
tool_result]`, the file in Claude Code's `N\t` line-number format,
optionally followed by 40 plain messages ("aged"). Arms: main
`ffc6edb4`, this PR, and each with #3881's `content_router.py` diff
applied. Tokens are cl100k counts of the `Read` block.
- Observed result: on main the newest token-mode `Read` takes the
compression route and is rewritten to one line with a CCR marker (CSV
below, and the code file once #3881 lands); with this PR it routes
`excluded_tool` and stays byte-exact, while the aged copy still
compresses. Cache mode is identical across arms.

`Read sbom/headroom-python-licenses.csv` (9,166 tokens, 331 lines):

| arm | placement | route | block after | byte-exact |
|---|---|---|---|---|
| main | newest | compression | one line + `Retrieve more` marker (9,769
tokens) | no |
| this PR | newest | `excluded_tool` | 9,166 | yes |
| this PR | aged (40 msgs after) | compression | one line + marker
(9,769 tokens) | no |
| cache mode, both arms | newest | `excluded_tool` | 9,166 | yes |

The same newest read under the `general` profile (token mode is its
default): main rewrites it the same way, this PR keeps it byte-exact via
`excluded_tool`.

`Read headroom/learn/analyzer.py` (13,461 tokens, 1,083 lines):

| arm | placement | route | block after | byte-exact |
|---|---|---|---|---|
| main | newest | compression (not excluded) | 13,461 (Kompress fallback
loses on the #3881 unit bug) | yes, by accident |
| main + #3881 | newest | compression | 11,139, newlines 1,082 -> 1, CCR
marker | no |
| this PR + #3881 | newest | `excluded_tool` | 13,461 | yes |
| this PR + #3881 | aged (40 msgs after) | compression | 11,139, CCR
marker | no |
| cache mode, all arms | newest | `excluded_tool` | 13,461 | yes |

The flattened output drops every newline and line-number column, which
is what the agent copies into `Edit(old_string=...)`.

- Not tested: live provider calls and a full Claude Code session; the
OpenAI `role: "tool"` shape (it reads the same `read_protection_window`
variable from the same `apply` scope, so the fix covers it, but no
separate proof was run); fleet-level savings impact.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, in token mode with a profile
whose `protect_recent` is 0 (`coding` with `--mode token`, and
`general`, which defaults to token mode): excluded-tool output in the
last `max(4, 0.3 * n)` messages is protected again
(`Read`/`Skill`/`read_file` byte-exact, `Grep`/`Glob`/`Edit`/`Write`
lossless compaction or verbatim). Savings on those recent blocks drop.
Cache mode, `--protect-tool-results`, and profiles with a positive
`protect_recent` are unchanged.
- Kill switch / disable path: none dedicated. `--mode cache` was never
affected; there is no config that restores "compress the newest Read"
short of reverting.
- Unsafe override required: None.
- Qualification impact: Lower token-mode savings on recent file reads
and recent Grep/Glob/Edit/Write output for `coding`/`general`; fewer CCR
retrievals and re-reads of lossy-compressed reads expected. Worth
watching token-mode savings rate on the release that carries it.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, backend routing change.

## Additional Notes

- Documentation: no doc describes `read_protection_window` or its
interaction with `protect_recent`, so there was nothing to update.
- Placement: the alternative was to stop `proxy_pipeline_kwargs()` from
emitting `read_protection_window` when `protect_recent` is 0. That
leaves any other caller that passes 0 still exposed and changes the
kwargs contract `tests/test_agent_savings.py` pins, so the guard sits in
`ContentRouter.apply`, where every caller routes through.
- A negative window used to clamp to 0 (protect nothing); it now also
falls back to the fraction window.
- Found while measuring #3881: its gains were on newest-message code
`Read`s in token mode, which should not have been compressible at all.
Merging this first keeps #3881 from turning an accidental passthrough
into a lossy rewrite of the newest file read.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 08:44:48 -05:00

318 lines
12 KiB
Python

"""Tests for --protect-tool-results / HEADROOM_PROTECT_TOOL_RESULTS.
Three behavioral tests:
1. protect_tool_results merges into the exclude set so named tools are never
lossy-compressed.
2. _parse_csv_tools parses CSV strings without merging HEADROOM_EXCLUDE_TOOLS.
3. ContentRouter with Bash in exclude_tools passes Bash tool_result verbatim.
"""
from __future__ import annotations
import pytest
from headroom.config import DEFAULT_EXCLUDE_TOOLS
from headroom.proxy.server import (
HeadroomProxy,
ProxyConfig,
_parse_csv_tools,
)
def _build(**overrides: object) -> HeadroomProxy:
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
code_aware_enabled=False,
**overrides,
)
return HeadroomProxy(config)
def _router(proxy: HeadroomProxy):
# ContentRouter is the last transform in the Anthropic pipeline.
return proxy.anthropic_pipeline.transforms[-1]
# ---------------------------------------------------------------------------
# Test 1: protect_tool_results merges into exclude set
# ---------------------------------------------------------------------------
def test_protect_tool_results_merges_into_exclude_set() -> None:
"""Bash added via protect_tool_results must appear in exclude_tools alongside
the built-in defaults (e.g. Read), base-fails / head-passes.
The frozenset is merged as-is; lowercase normalization is handled by
_parse_exclude_tools on the CLI/env path (tested separately).
"""
proxy = _build(protect_tool_results=frozenset({"Bash", "bash"}))
exclude = _router(proxy).config.exclude_tools
assert exclude is not None, "exclude_tools must be set when protect_tool_results is non-empty"
assert "Bash" in exclude, "Bash must be in exclude_tools after protect_tool_results merges"
assert "bash" in exclude, (
"lowercase bash must be in exclude_tools after protect_tool_results merges"
)
assert "Read" in exclude, "Read (built-in default) must still be in exclude_tools"
def test_protect_tool_results_disables_age_decay_in_token_mode() -> None:
"""In token mode, protect_tool_results forces protect_recent_reads_fraction to 0.0
so protected tools are never compressed by age-decay."""
proxy = _build(protect_tool_results=frozenset({"Bash", "bash"}), mode="token")
assert _router(proxy).config.protect_recent_reads_fraction == 0.0
# ---------------------------------------------------------------------------
# Test 2: CSV env var / CLI string parsing
# ---------------------------------------------------------------------------
def test_protect_tool_results_env_var_csv() -> None:
"""_parse_csv_tools parses a comma-separated value into both original-case
and lowercase entries without merging HEADROOM_EXCLUDE_TOOLS."""
result = _parse_csv_tools("Bash,WebFetch")
assert "Bash" in result
assert "bash" in result
assert "WebFetch" in result
assert "webfetch" in result
# ---------------------------------------------------------------------------
# Test 3: Bash tool_result passthrough when protected
# ---------------------------------------------------------------------------
def test_bash_tool_result_passthrough_when_protected() -> None:
"""When Bash is in exclude_tools (via protect_tool_results), its tool_result
content passes through the ContentRouter verbatim without lossy compression.
Base-fails / head-passes."""
pytest.importorskip("tiktoken") # needed for OpenAI tokenizer
from headroom.providers import OpenAIProvider
from headroom.tokenizer import Tokenizer
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
provider = OpenAIProvider()
token_counter = provider.get_token_counter("gpt-4o")
tokenizer = Tokenizer(token_counter, "gpt-4o")
# Build router with Bash explicitly in exclude_tools
config = ContentRouterConfig(
min_section_tokens=10,
exclude_tools=set(DEFAULT_EXCLUDE_TOOLS) | {"Bash", "bash"},
)
router = ContentRouter(config)
bash_output = "\n".join(
f"line {i}: some output from a bash command that is long enough to compress"
for i in range(80)
)
messages = [
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call_bash_1",
"type": "function",
"function": {"name": "Bash", "arguments": "{}"},
}
],
},
{
"role": "tool",
"tool_call_id": "call_bash_1",
"content": bash_output,
},
]
result = router.apply(messages, tokenizer)
# Bash tool_result must pass through unchanged
tool_msg = next(m for m in result.messages if m.get("tool_call_id") == "call_bash_1")
assert tool_msg["content"] == bash_output, (
"Bash tool_result must be verbatim when Bash is in exclude_tools"
)
assert "router:excluded:tool" in result.transforms_applied
# ---------------------------------------------------------------------------
# Test 4: protect_tool_results sentinel survives a profile-derived
# read_protection_window kwarg, even when the protected output is old
# ---------------------------------------------------------------------------
def test_protect_tool_results_survives_runtime_read_protection_window_kwarg() -> None:
"""A profile-derived `read_protection_window` kwarg (e.g. from
AgentSavingsProfile.protect_recent=2, threaded in via
proxy_pipeline_kwargs()) must not shrink protection below what
protect_recent_reads_fraction == 0.0 (the --protect-tool-results
sentinel) already guarantees for the whole conversation.
Regression test for the precedence bug: content_router.py used to apply
the runtime kwarg unconditionally, so a Bash tool_result more than
`read_protection_window` messages old fell through to lossy compression
even though --protect-tool-results promised it would never compress
"regardless of conversation depth" (see PR #1374)."""
pytest.importorskip("tiktoken") # needed for OpenAI tokenizer
from headroom.providers import OpenAIProvider
from headroom.tokenizer import Tokenizer
provider = OpenAIProvider()
token_counter = provider.get_token_counter("gpt-4o")
tokenizer = Tokenizer(token_counter, "gpt-4o")
proxy = _build(protect_tool_results=frozenset({"Bash", "bash"}), mode="token")
router = _router(proxy)
bash_output = "\n".join(
f"line {i}: some output from a bash command that is long enough to compress"
for i in range(80)
)
messages: list[dict[str, object]] = [
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call_bash_1",
"type": "function",
"function": {"name": "Bash", "arguments": "{}"},
}
],
},
{
"role": "tool",
"tool_call_id": "call_bash_1",
"content": bash_output,
},
]
# Pad with enough intervening turns that the Bash tool_result above
# falls outside a read_protection_window=2 (it's ~9-10 messages from
# the end once padding is added).
for i in range(8):
messages.append({"role": "user", "content": f"follow-up turn {i}"})
messages.append({"role": "assistant", "content": f"reply {i}"})
# Simulate the profile-derived kwarg the proxy threads into every
# request via proxy_pipeline_kwargs() (AgentSavingsProfile("coding")
# sets protect_recent=2).
result = router.apply(messages, tokenizer, read_protection_window=2)
tool_msg = next(m for m in result.messages if m.get("tool_call_id") == "call_bash_1")
assert tool_msg["content"] == bash_output, (
"Bash tool_result must stay verbatim: protect_recent_reads_fraction == 0.0 "
"(set by --protect-tool-results) must not be weakened by a profile-derived "
"read_protection_window kwarg"
)
assert "router:excluded:tool" in result.transforms_applied
# ---------------------------------------------------------------------------
# Test 5: token mode + coding profile keeps recent file reads byte-exact
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("trailing_turns", "window", "protected"),
[
(0, None, True), # newest Read, the profile's window of 0
(1, None, True), # 3 from the end, inside max(4, 0.3*n)
(1, 2, False), # a positive profile window still narrows it
(20, None, False), # past the 0.3 window: ages out
],
)
def test_token_mode_coding_profile_keeps_recent_read_byte_exact(
trailing_turns: int, window: int | None, protected: bool, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The coding profile's protect_recent=0 reaches the router as
read_protection_window=0. In token mode (fraction 0.3) that used to replace
the max(4, 0.3*n) window with 0, so a `Read` result lossy-compressed even
as the newest message. Recent reads must stay byte-exact; reads past the
fraction window still age out to compression, and a positive profile
window still narrows the fraction window."""
pytest.importorskip("tiktoken")
from types import SimpleNamespace
from headroom.agent_savings import proxy_pipeline_kwargs
from headroom.providers import OpenAIProvider
from headroom.tokenizer import Tokenizer
tokenizer = Tokenizer(OpenAIProvider().get_token_counter("gpt-4o"), "gpt-4o")
proxy = _build(mode="token", savings_profile="coding")
router = _router(proxy)
assert router.config.protect_recent_reads_fraction == 0.3
class FakeKompress:
def is_ready(self) -> bool:
return True
def ensure_background_load(self) -> None:
pass
def compress(self, content, **kwargs):
compressed = " ".join(content.split()[:20]) + " Retrieve more: hash=deadbeef"
return SimpleNamespace(compressed=compressed, compressed_tokens=len(compressed.split()))
monkeypatch.setattr(router, "_get_kompress", lambda: FakeKompress())
read_output = "\n".join(
f"{i:>6}\tThe release notes describe change number {i} and its rollout plan."
for i in range(1, 121)
)
messages: list[dict[str, object]] = [
{"role": "user", "content": "read the notes"},
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"id": "toolu_read_1",
"name": "Read",
"input": {"file_path": "/repo/NOTES.md"},
}
],
},
{
"role": "user",
"content": [
{"type": "tool_result", "tool_use_id": "toolu_read_1", "content": read_output}
],
},
]
for i in range(trailing_turns):
messages.append({"role": "assistant", "content": f"reply {i}"})
messages.append({"role": "user", "content": f"follow-up {i}"})
kwargs = proxy_pipeline_kwargs(proxy.config)
assert kwargs["read_protection_window"] == 0
if window is not None:
kwargs["read_protection_window"] = window
result = router.apply(messages, tokenizer, **kwargs)
content = result.messages[2]["content"][0]["content"]
if protected:
assert content == read_output, "a recent Read must stay byte-exact in token mode"
assert "router:excluded:tool" in result.transforms_applied
else:
assert content != read_output, "a Read outside the window still ages out"
# ---------------------------------------------------------------------------
# Baseline: Bash NOT in DEFAULT_EXCLUDE_TOOLS (unchanged by this PR)
# ---------------------------------------------------------------------------
def test_bash_not_in_default_exclude_tools() -> None:
"""Bash must remain absent from DEFAULT_EXCLUDE_TOOLS; protect_tool_results
is the opt-in path."""
assert "Bash" not in DEFAULT_EXCLUDE_TOOLS
assert "bash" not in DEFAULT_EXCLUDE_TOOLS