Files
headroom/tests/test_tool_schema_compaction.py
a25623109d fix(proxy): copy cached tool compaction results (hardening) (#4410)
## Description

> **Hardening, preventive.** No request sequence was found where this
changes what reaches the upstream or the client; the fix removes shared
mutable state before a future caller can depend on it. Please treat it
as low priority.

`compact_tools` and `compact_tool_descriptions` cache the compacted
`tools` list at module level and then put that same list into the
payload they return. Callers edit the tools in place afterwards
(`inject_tool_search_deferral` sets `cache_control` on a resident tool,
`helpers.py:4235`), so the edit stays in the cache and shows up in later
requests with the same digest. The returned payload now gets a deep copy
of the cached list.

Fixes #4403

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/tool_schema_compaction.py`: `updated["tools"] =
copy.deepcopy(compacted_tools)` at the four places in `compact_tools`
and `compact_tool_descriptions` (cache hit and cache miss in each). The
cache still stores the original list.
- `tests/test_tool_schema_compaction.py`:
`TestCompactedToolsAreNotShared`, one test per function: edit the
returned tool, then call again twice (a hit) and assert the edit is not
visible.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new tests, source change reverted
FAILED tests/test_tool_schema_compaction.py::TestCompactedToolsAreNotShared::test_compact_tools_edit_does_not_leak_into_later_requests
FAILED tests/test_tool_schema_compaction.py::TestCompactedToolsAreNotShared::test_compact_tool_descriptions_edit_does_not_leak
2 failed, 27 passed

# with the fix
$ pytest tests/test_tool_schema_compaction.py
29 passed

$ pytest tests/test_tool_schema_compaction.py tests/test_openai_chat_tool_desc_compaction.py tests/test_anthropic_compaction_transforms.py
44 passed, 1 warning

$ ruff check headroom/proxy/tool_schema_compaction.py tests/test_tool_schema_compaction.py
All checks passed!
$ ruff format --check headroom/proxy/tool_schema_compaction.py tests/test_tool_schema_compaction.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Ubuntu on WSL2, Python 3.12.15, upstream/main e995eb9f
plus this commit. `headroom/_core.abi3.so` copied (untracked, not
committed) from the official `headroom_ai-0.41.0` wheel because the
checkout has no built Rust core. "Before" = the source file with this
commit's change reverted (`git apply -R`).
- Exact command / steps: (1) start a stub upstream on 127.0.0.1:18870
and the real proxy `HEADROOM_TOOL_DESC_MAX_CHARS=60
ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18870 python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18871 --no-cache
--no-rate-limit`, send the same 14-tool request twice, record what the
stub receives; (2) in the same checkout, call the shipped functions in
the handler's order (`compact_tools`, then
`inject_tool_search_deferral`) and compact the same client request
again.
- Observed result: over HTTP the stub receives the same bodies before
and after (compaction still applies, nothing else changed), because the
proxy does not run tool-search deferral against a custom upstream URL.
In-process, before the fix the second identical request gets a
`cache_control` on `Bash` that the client never sent and shares tool
dicts with the first; after the fix it does not.
- Not tested: the leak through a real Anthropic upstream (tool-search
deferral is skipped for a non-Anthropic base URL, so the HTTP path
cannot show it); concurrent requests; the OpenAI and gateway callers;
`mypy`.

HTTP, before and after (stub log, identical):

```text
client request 1 HTTP 200
client request 2 HTTP 200
STUB tools=15 first_desc='Retrieve original uncompressed content that was compressed t…' cache_control_on=[]
STUB tools=15 first_desc='Retrieve original uncompressed content that was compressed t…' cache_control_on=[]
```

In-process, before (change reverted):

```text
request 1: compacted tools carry cache_control on: ['tool_3']
request 1: after tool-search deferral, tools carry cache_control on: ['Bash']
request 2 (fresh, identical client input): compacted tools carry cache_control on: ['tool_3', 'Bash']
request 1 and 2 share tool dicts: True
```

In-process, after (this commit):

```text
request 1: compacted tools carry cache_control on: ['tool_3']
request 1: after tool-search deferral, tools carry cache_control on: ['Bash']
request 2 (fresh, identical client input): compacted tools carry cache_control on: ['tool_3']
request 1 and 2 share tool dicts: False
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no; returned tool content is equal,
only the object identity changes
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none known
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Only the listed test files were run, not the whole suite. The deep copy
costs one extra copy of the tools list per call on both the hit and miss
path. I did not find a request sequence where the leak changes the body
sent upstream (same digest means the same client markers); the fix
removes the shared mutable state. Produced with AI assistance (Claude
Code) operated by me (HetCreep); the reproduction was run on my machine
on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4410"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:44:16 -07:00

511 lines
21 KiB
Python

"""Tests for headroom.proxy.tool_schema_compaction — shared tool-schema compaction.
Verifies that the compaction logic (shared by OpenAI and Anthropic handlers):
- strips JSON Schema annotation keys ($schema, title, examples, …)
- preserves property names that collide with DROP_KEYS (e.g. a field named "title")
- normalises description whitespace
- never inflates payload size
"""
from __future__ import annotations
from headroom.proxy.tool_schema_compaction import (
compact_tool_schema_value,
compact_tools,
)
# ---------------------------------------------------------------------------
# compact_tool_schema_value
# ---------------------------------------------------------------------------
class TestCompactToolSchemaValue:
"""Unit tests for compact_tool_schema_value."""
def test_drops_schema_annotations(self) -> None:
schema = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "MyToolParams",
"type": "object",
"properties": {"x": {"type": "integer"}},
"required": ["x"],
}
result = compact_tool_schema_value(schema)
assert "$schema" not in result
assert "title" not in result
assert "type" in result
assert "properties" in result
assert "required" in result
def test_preserves_property_named_title(self) -> None:
"""A field literally named 'title' must survive (not a schema annotation)."""
schema = {
"type": "object",
"properties": {
"title": {"type": "string"},
"code": {"type": "string"},
},
"required": ["title", "code"],
}
result = compact_tool_schema_value(schema)
props = result["properties"]
assert "title" in props, "property named 'title' must survive"
assert "code" in props
def test_normalises_description_whitespace(self) -> None:
schema = {
"name": "my_tool",
"description": " This is a description \n with extra spaces ",
"input_schema": {"type": "object", "properties": {}},
}
result = compact_tool_schema_value(schema)
assert result["description"] == "This is a description with extra spaces"
def test_drops_examples_and_deprecated(self) -> None:
schema = {
"type": "object",
"properties": {
"x": {
"type": "integer",
"examples": [1, 2, 3],
"deprecated": True,
},
},
}
result = compact_tool_schema_value(schema)
prop_x = result["properties"]["x"]
assert "examples" not in prop_x
assert "deprecated" not in prop_x
assert prop_x["type"] == "integer"
def test_preserves_property_named_deprecated(self) -> None:
schema = {
"type": "object",
"properties": {
"deprecated": {"type": "boolean", "description": "Is it deprecated?"},
},
}
result = compact_tool_schema_value(schema)
assert "deprecated" in result["properties"]
def test_handles_list_of_tools(self) -> None:
tools = [
{
"name": "tool_a",
"description": " First tool ",
"input_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "ToolAParams",
"type": "object",
"properties": {"a": {"type": "string"}},
},
},
{
"name": "tool_b",
"description": " Second tool ",
"input_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "ToolBParams",
"type": "object",
"properties": {"b": {"type": "integer"}},
},
},
]
result = compact_tool_schema_value(tools)
assert len(result) == 2
for tool in result:
assert "$schema" not in tool["input_schema"]
assert "title" not in tool["input_schema"]
assert " " not in tool["description"]
def test_nested_properties_preserved(self) -> None:
schema = {
"type": "object",
"properties": {
"config": {
"type": "object",
"title": "ConfigObject", # annotation — should be dropped
"properties": {
"title": {"type": "string"}, # property name — must survive
"value": {"type": "integer"},
},
},
},
}
result = compact_tool_schema_value(schema)
# Top-level config annotation dropped
assert "title" not in result["properties"]["config"]
# But nested property named "title" preserved
assert "title" in result["properties"]["config"]["properties"]
# ---------------------------------------------------------------------------
# compact_tools
# ---------------------------------------------------------------------------
class TestCompactTools:
"""Unit tests for compact_tools (full payload compaction)."""
def test_compacts_anthropic_style_payload(self) -> None:
"""Anthropic Messages API format uses 'input_schema'."""
payload = {
"model": "claude-sonnet-4-20250514",
"messages": [{"role": "user", "content": "hello"}],
"tools": [
{
"name": "get_weather",
"description": " Get the current weather ",
"input_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "GetWeatherParams",
"type": "object",
"properties": {
"location": {"type": "string"},
},
"required": ["location"],
},
},
],
}
result, modified, before, after = compact_tools(payload)
assert modified is True
assert after < before
tool = result["tools"][0]
assert " " not in tool["description"]
assert "$schema" not in tool["input_schema"]
assert "title" not in tool["input_schema"]
assert "properties" in tool["input_schema"]
def test_compacts_openai_style_payload(self) -> None:
"""OpenAI format uses 'parameters' instead of 'input_schema'."""
payload = {
"tools": [
{
"type": "function",
"name": "read_file",
"description": "Read a file from disk.",
"parameters": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "ReadFileParams",
"type": "object",
"properties": {
"path": {"type": "string", "examples": ["/tmp/test"]},
},
"required": ["path"],
},
},
],
}
result, modified, before, after = compact_tools(payload)
assert modified is True
assert after < before
params = result["tools"][0]["parameters"]
assert "$schema" not in params
assert "title" not in params
assert "examples" not in params["properties"]["path"]
def test_returns_unchanged_when_no_tools(self) -> None:
payload = {"model": "claude-sonnet-4-20250514", "messages": []}
result, modified, _, _ = compact_tools(payload)
assert modified is False
assert result is payload # same object, not copied
def test_returns_unchanged_when_empty_tools(self) -> None:
payload = {"tools": []}
result, modified, _, _ = compact_tools(payload)
assert modified is False
def test_returns_unchanged_when_already_compact(self) -> None:
payload = {
"tools": [
{
"name": "simple",
"description": "A simple tool",
"input_schema": {
"type": "object",
"properties": {"x": {"type": "integer"}},
},
},
],
}
result, modified, before, after = compact_tools(payload)
# May or may not be modified depending on description whitespace
# but should never inflate
assert after <= before
def test_preserves_non_tool_fields(self) -> None:
payload = {
"model": "claude-sonnet-4-20250514",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "hi"}],
"tools": [
{
"name": "t",
"description": "test",
"input_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
},
},
],
}
result, _, _, _ = compact_tools(payload)
assert result["model"] == "claude-sonnet-4-20250514"
assert result["max_tokens"] == 1024
assert len(result["messages"]) == 1
def test_large_github_like_tool_set(self) -> None:
"""Simulate a large tool set (like GitHub MCP with 44 tools)."""
tools = []
for i in range(44):
tools.append(
{
"name": f"github_tool_{i}",
"description": f" Perform operation {i} on GitHub repositories ",
"input_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": f"GithubTool{i}Params",
"type": "object",
"properties": {
"owner": {"type": "string", "description": " Repo owner "},
"repo": {"type": "string", "examples": ["my-repo"]},
},
"required": ["owner", "repo"],
},
}
)
payload = {"model": "claude-sonnet-4-20250514", "tools": tools}
result, modified, before, after = compact_tools(payload)
assert modified is True
savings_pct = (1 - after / before) * 100
# Expect meaningful savings (at least 15% with annotation keys + whitespace)
assert savings_pct >= 10, f"Expected ≥10% savings, got {savings_pct:.1f}%"
# ---------------------------------------------------------------------------
# Layer 2: compact_tool_descriptions
# ---------------------------------------------------------------------------
class TestTruncateDescription:
"""Unit tests for _truncate_description."""
def test_short_description_unchanged(self) -> None:
from headroom.proxy.tool_schema_compaction import _truncate_description
assert _truncate_description("Read a file.", 120) == "Read a file."
def test_first_sentence_preserved(self) -> None:
from headroom.proxy.tool_schema_compaction import _truncate_description
desc = "Fast and precise code search across ALL GitHub repositories. Best for finding exact symbols."
result = _truncate_description(desc, 60)
assert result == "Fast and precise code search across ALL GitHub repositories."
def test_first_sentence_plus_second(self) -> None:
from headroom.proxy.tool_schema_compaction import _truncate_description
desc = "Read a file. Returns the contents as text."
result = _truncate_description(desc, 60)
# First sentence is short enough, second fits within 1.5x budget
assert "Read a file." in result
assert "Returns the contents as text." in result
def test_long_first_sentence_truncated(self) -> None:
from headroom.proxy.tool_schema_compaction import _truncate_description
desc = "This is an extremely long description that goes on and on without any sentence boundary"
result = _truncate_description(desc, 40)
assert len(result) <= 45 # 40 + "…"
assert result.endswith("…")
def test_whitespace_normalised_before_truncation(self) -> None:
from headroom.proxy.tool_schema_compaction import _truncate_description
desc = " Search code. Very useful. "
result = _truncate_description(desc, 60)
# Whitespace normalised, both sentences fit within 1.5x budget
assert result == "Search code. Very useful."
def test_max_chars_zero_returns_original(self) -> None:
from headroom.proxy.tool_schema_compaction import _truncate_description
desc = "Any long description that would normally be truncated."
result = _truncate_description(desc, 0)
# max_chars=0 means disabled, return unchanged
assert result == "Any long description that would normally be truncated."
def test_chinese_description(self) -> None:
from headroom.proxy.tool_schema_compaction import _truncate_description
desc = "搜索代码仓库中的函数和类。支持正则表达式匹配。"
result = _truncate_description(desc, 30)
# First Chinese sentence fits
assert "搜索代码仓库中的函数和类。" in result
class TestCompactToolDescriptions:
"""Unit tests for compact_tool_descriptions (full payload)."""
def test_truncates_long_tool_description(self) -> None:
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions
payload = {
"tools": [
{
"name": "search_code",
"description": "Fast and precise code search across ALL GitHub repositories using GitHub's native search engine. Best for finding exact symbols, functions, classes, or specific code patterns. Returns ranked results.",
"input_schema": {"type": "object", "properties": {}},
},
],
}
result, modified, before, after = compact_tool_descriptions(payload, max_chars=60)
assert modified is True
assert after < before
tool = result["tools"][0]
# First sentence ends at "repositories." (abbrev regex match)
# With max_chars=60, first sentence "Fast...repositories." is 60 chars exactly
# so it should be preserved. Second sentence may or may not fit in 1.5x budget.
assert tool["description"].startswith("Fast and precise code search")
assert len(tool["description"]) < len(payload["tools"][0]["description"])
def test_truncates_nested_param_descriptions(self) -> None:
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions
payload = {
"tools": [
{
"name": "t",
"description": "Short.",
"input_schema": {
"type": "object",
"properties": {
"q": {
"type": "string",
"description": "The search query string to find matching code. Supports advanced syntax like OR, NOT, and quoted phrases for exact match.",
},
},
},
},
],
}
result, modified, before, after = compact_tool_descriptions(payload, max_chars=60)
assert modified is True
param_desc = result["tools"][0]["input_schema"]["properties"]["q"]["description"]
assert "The search query string to find matching code." == param_desc
def test_disabled_when_max_chars_zero(self) -> None:
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions
payload = {
"tools": [
{"name": "t", "description": "A" * 500, "input_schema": {"type": "object"}},
],
}
result, modified, _, _ = compact_tool_descriptions(payload, max_chars=0)
assert modified is False
assert result is payload
def test_no_tools_returns_unchanged(self) -> None:
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions
result, modified, _, _ = compact_tool_descriptions({"model": "x"}, max_chars=120)
assert modified is False
def test_preserves_non_description_fields(self) -> None:
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions
payload = {
"model": "claude-sonnet-4-20250514",
"tools": [
{
"name": "get_weather",
"description": "Get the current weather for a location. Supports any city worldwide.",
"input_schema": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The city name to get weather for.",
},
},
},
},
],
}
result, _, _, _ = compact_tool_descriptions(payload, max_chars=50)
assert result["model"] == "claude-sonnet-4-20250514"
assert result["tools"][0]["name"] == "get_weather"
assert result["tools"][0]["input_schema"]["properties"]["city"]["type"] == "string"
def test_large_tool_set_savings(self) -> None:
"""44 GitHub-like tools with verbose descriptions should see significant savings."""
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions
tools = []
for i in range(44):
tools.append(
{
"name": f"github_tool_{i}",
"description": f"Perform operation {i} on GitHub repositories. This tool supports advanced filtering and pagination for large result sets. Use it for code search, issue management, and PR operations.",
"input_schema": {
"type": "object",
"properties": {
"owner": {
"type": "string",
"description": "Repository owner username or organization name.",
},
"repo": {
"type": "string",
"description": "The name of the repository to operate on.",
},
},
},
}
)
payload = {"tools": tools}
result, modified, before, after = compact_tool_descriptions(payload, max_chars=80)
assert modified is True
savings_pct = (1 - after / before) * 100
assert savings_pct >= 10, f"Expected ≥10% savings, got {savings_pct:.1f}%"
class TestCompactedToolsAreNotShared:
"""Results must not alias the module-level cache (callers mutate tools in place)."""
@staticmethod
def _payload() -> dict:
return {
"tools": [
{
"name": "alias_probe",
"description": "Probe. " + "Extra words. " * 40,
"input_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Params",
"type": "object",
"properties": {"x": {"type": "integer", "title": "X"}},
},
}
]
}
def test_compact_tools_edit_does_not_leak_into_later_requests(self) -> None:
first, modified, _, _ = compact_tools(self._payload())
assert modified is True
first["tools"][-1]["cache_control"] = {"type": "ephemeral"}
for _ in range(2): # second call after the edit, and a cache-hit call
later, _, _, _ = compact_tools(self._payload())
assert "cache_control" not in later["tools"][-1]
def test_compact_tool_descriptions_edit_does_not_leak(self) -> None:
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions
first, modified, _, _ = compact_tool_descriptions(self._payload(), max_chars=40)
assert modified is True
first["tools"][-1]["cache_control"] = {"type": "ephemeral"}
for _ in range(2):
later, _, _, _ = compact_tool_descriptions(self._payload(), max_chars=40)
assert "cache_control" not in later["tools"][-1]