fix(cache): keep cache mode when Claude Code re-serializes its last reply (#4003)

## Description

This PR fixes two problems that cost Headroom users money on Claude
Code. A live benchmark (`benchmarks/tool_search_vs_native/`) found both.

**1. Cache mode broke when Claude Code re-serialized its last reply.**

Headroom records the model's raw reply as the last message of the
previous turn. Claude Code sends that reply back edited:

- it strips a leading `cd <cwd> &&` from Bash commands;
- it drops `caller`.

The history then read as `relation=diverged`
(`cache_mode_prefix_mismatch`). Cache mode forwarded the raw originals
over every earlier compressed message, and the whole conversation was
written to cache again: 66.8K tokens on every long-task run.

The fix: when only the recorded reply differs, and it is the same turn,
Headroom replays the forwarded prefix and takes the client's version of
the reply. That reply was never part of a cached prefix. "Same turn"
means the same block kinds and the same `tool_use` ids.

**2. Small MCP results were compressed.**

The Jira ticket (a few hundred tokens) lost its description. The model
called `headroom_retrieve` and then re-fetched the ticket, which cost
two extra turns and about +15% on the long task. MCP results under
`HEADROOM_MCP_RESULT_MIN_CHARS` (default 4000) are now forwarded
verbatim. This covers both Anthropic `tool_result` blocks and OpenAI
`role: "tool"` messages.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/cache/prefix_tracker.py`:** adds
`_client_rewrote_reply_delta`, tried on `RELATION_DIVERGED` after
`_replaced_system_tail_delta`. A change earlier in the history still
diverges.
- **`headroom/transforms/content_router.py`:** adds
`_is_small_mcp_result`, applied in `_process_content_blocks` (Anthropic)
and on the `role: "tool"` branch (OpenAI). It recognises the `mcp__`,
`mcp_` and `mcp--` name prefixes. `HEADROOM_MCP_RESULT_MIN_CHARS=0`
restores the old behaviour.
- **`headroom/proxy/handlers/anthropic.py`:** adds an opt-in
`HEADROOM_DEBUG_PREFIX_MISMATCH=<dir>`. It logs the first diverging
message and dumps both histories, with the directory created `0700` and
the file `0600`, because the dump holds the full conversation.
- **`tests/test_cache_mode_delta_marker.py`:**
`test_tool_use_different_input_still_falls_back` becomes
`..._forwards_the_clients_version`, because the client's copy of the
last reply is what gets forwarded and never the recorded one. A new test
keeps the divergence for an edit earlier in the history.
- **New tests:**
  - `tests/test_prefix_tracker_client_rewrote_reply.py`;
- `tests/test_content_router_mcp_small_result.py`, which covers both
wire formats and three MCP prefixes.
- **`benchmarks/tool_search_vs_native/`:** a live A/B harness. It runs
isolated `claude -p` sessions against six fake MCP servers and a fixture
repo, and reads billed usage from Claude Code's transcripts. A run
passes only if Claude Code exits 0, the tests pass, and the four MCP
actions happened (`get_issue` counts as reading the ticket; a search
does not).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_prefix_tracker_client_rewrote_reply.py tests/test_content_router_mcp_small_result.py tests/test_cache_mode_delta_marker.py
============================== 20 passed in 3.42s ==============================

$ pytest -q tests -k "content_router or router or prefix or cache_mode or delta"
========= 1 failed, 983 passed, 15147 deselected in 222.61s (0:03:42) ==========
# the 1 failure is test_content_router_single_item_deadline.py::test_single_cache_miss_fails_open_at_deadline,
# a wall-clock assertion (0.126s vs < 0.12s). It fails identically on origin/main on this machine.

$ ruff check .            (ruff 0.16.8, CI's pinned version)
All checks passed!
$ ruff format --check .
1757 files already formatted
$ mypy headroom/proxy/handlers/anthropic.py headroom/transforms/content_router.py headroom/cache/prefix_tracker.py --ignore-missing-imports
Success: no issues found
```

## Real Behavior Proof

- Environment: macOS 26.6 (arm64), Python 3.12.13 (the repo's `.venv`),
Claude Code 2.1.290, `claude-sonnet-5-5`, real Anthropic API calls, with
the proxy started from this branch.
- Exact command / steps: `python3
benchmarks/tool_search_vs_native/run.py --arms native,hr_hot,hr_client
--reps 3 --warmup`, then the same with `--context-kb 200` for the long
task.
- Observed result: median of 3 runs after a warmup; every run passed
(table below). Before the fix, `HEADROOM_DEBUG_PREFIX_MISMATCH` showed
`relation=diverged first_diff_index=8`: the Bash `tool_use` with `cd …
&&` stripped and `caller` dropped.
- Not tested: harnesses other than Claude Code (the OpenAI-format MCP
path is unit-tested only), Bedrock and Vertex upstreams, real (larger)
MCP server catalogs, models other than Sonnet 5.5.

| | native Claude Code | Headroom + CC ToolSearch | Headroom, routine
tools resident |
|---|---|---|---|
| long task, before | $0.283 | $0.323 | $0.426 (66.8K-token cache
re-write every run) |
| long task, after | $0.284 | **$0.284** | **$0.255** (6 API calls vs
native's 7) |
| short task, after | $0.061 | $0.062 | **$0.046** |

## Runtime Rollout Safety

- Rollout-managed feature(s): none; both changes are in the always-on
cache-mode and compression paths.
- Minimum rollout channel: stable.
- Stable/default behavior changed: yes. A re-serialized last reply no
longer drops cache mode to raw forwarding, and MCP results under 4000
characters are no longer compressed, giving up a few tokens of savings
on those results.
- Kill switch / disable path: `HEADROOM_MCP_RESULT_MIN_CHARS=0` for the
MCP floor. The prefix fix has no flag, because it only applies where the
old code forwarded raw.
- Unsafe override required: no.
- Qualification impact: less compression on small MCP results; fewer
cache writes and extra turns.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- **Devin review follow-ups are in `d043bd5e`:**
  - the OpenAI-format MCP floor;
  - owner-only debug dumps;
  - exit-code and `get_issue` scoring;
  - the environment and commands in the README.
- **The two `npm audit` failures** (docs, e2e/wrap/tools) also fail on
`main`.
- **`tests/test_cli_proxy_improvements.py` hangs** when the whole file
runs locally, also on `main`. That is pre-existing and not touched here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-07 10:58:23 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent dea9092e19
commit aeb09f8d18
13 changed files with 1699 additions and 2 deletions
@@ -0,0 +1,85 @@
# Claude Code tool search: native vs Headroom
A live A/B benchmark. Each run is a fresh, isolated Claude Code session with its own
throwaway `HOME`, a new copy of `fixture/`, and six fake enterprise MCP servers
(`mcp_server.py`: GitHub, Jira, Slack, Datadog, PagerDuty and Confluence, 63 tools).
Every session does the same task:
1. Read Jira BENCH-42.
2. Fix the pricing bug.
3. Run the tests.
4. Search the Datadog logs.
5. Comment on the ticket.
6. Post to Slack.
Usage is read from Claude Code's own transcript, which records what the API billed,
subagents included.
```bash
python3 benchmarks/tool_search_vs_native/run.py --arms native,hr_client,hr_server,hr_hot --reps 3 --warmup
python3 benchmarks/tool_search_vs_native/run.py ... --context-kb 200 # lookups deep in a long session
```
The script needs `ANTHROPIC_API_KEY` (or `~/env.txt`) and the repo's `.venv`. A full run
of four arms with three reps and a warmup costs about $1 short and $5 long, on Sonnet 5.5.
| Arm | Setup |
|---|---|
| `native` | Claude Code straight to api.anthropic.com, with its own ToolSearch (on by default). |
| `eager` | Claude Code straight to Anthropic with `ENABLE_TOOL_SEARCH=false`, so nothing is deferred. |
| `hr_client` | Claude Code through Headroom with `ENABLE_TOOL_SEARCH=true` (what `headroom wrap` sets today). |
| `hr_server` | Claude Code through Headroom with `ENABLE_TOOL_SEARCH=false`. Headroom defers tools on Anthropic's side. |
| `hr_hot` | As `hr_server`, but the MCP tools a developer uses routinely stay loaded (`HEADROOM_TOOL_SEARCH_CORE_TOOLS`). |
## Results (2026-10-06, Claude Code 2.1.290, Sonnet 5.5, n=3 after warmup, median)
Run on macOS 26.6 (arm64) with Python 3.12.13 (the repo's `.venv`) and real Anthropic API
calls. Commands:
```bash
python3 benchmarks/tool_search_vs_native/run.py --arms native,hr_hot,hr_client --reps 3 --warmup
python3 benchmarks/tool_search_vs_native/run.py --arms native,hr_hot,hr_client --reps 3 --warmup --context-kb 200
```
A run passes only if Claude Code exits 0, the fixture tests pass, and the session called
Jira `get_issue`, Datadog `search_logs`, Jira `add_comment` and Slack `post_message`.
All 73 recorded runs pass under that rule.
| | native | hr_client | hr_server | hr_hot |
|---|---|---|---|---|
| short task, before fixes | $0.059 | $0.062 | $0.081 | $0.049 |
| long task, before fixes | $0.283 | $0.323 | $0.350 | $0.426 |
| long task, **after fixes** | $0.284 | **$0.284** | — | **$0.255 (−10%)** |
| short task, **after fixes** | $0.061 | $0.062 | — | **$0.046 (−25%)** |
The "after" rows held in every run: on the long task `hr_hot` cost $0.254–0.256 and
`native` $0.282–0.284.
### What the runs showed
- **A tool lookup costs a re-read of the whole conversation.** That holds for Claude
Code's ToolSearch, which takes an extra turn, and for Anthropic's server-side search,
which takes an extra sampling pass inside the same call (usage sums the iterations). A
schema that stays loaded costs about 0.1× its size per turn. So keeping a routinely
used tool loaded is far cheaper than looking it up.
- **Server-side search without names searches blindly.** Claude Code tells the model the
deferred tool names, so it loads everything it needs in one `select:`. Headroom's
server-side deferral gives no names, and the model ran 2–5 regex searches. Each search
pulled in about 4.7K tokens of schemas, uncached.
- **Bug: cache mode broke when Claude Code re-serialized its last reply.** Headroom
records the model's raw reply as the end of the previous turn. Claude Code sends that
reply back edited: it strips a leading `cd <cwd> &&` from Bash commands and drops
`caller`. The history then read as diverged, Headroom forwarded the raw originals over
its earlier compressed messages, and the whole conversation was re-written to cache:
66.8K tokens on every long `hr_hot` run. Fixed in `headroom/cache/prefix_tracker.py`
(`_client_rewrote_reply_delta`). Set `HEADROOM_DEBUG_PREFIX_MISMATCH=<dir>` to log and
dump where a history diverges.
- **Bug: compressing small MCP results cost turns.** The Jira ticket (a few hundred
tokens) was compressed and lost its description, so the model called
`headroom_retrieve` and then re-fetched the ticket: two extra turns, about +15% on the
long task. MCP results under `HEADROOM_MCP_RESULT_MIN_CHARS` (default 4000) are now
forwarded verbatim, in both Anthropic `tool_result` blocks and OpenAI `role: "tool"`
messages.
The fake MCP schemas are terse compared with real servers such as GitHub's official one,
so absolute savings here understate a real catalog.
@@ -0,0 +1,16 @@
"""Checkout pricing."""
from __future__ import annotations
def apply_discount(price: float, percent: float) -> float:
"""Return ``price`` after a ``percent`` discount, rounded to cents."""
discounted = price * (1 - percent / 100)
discounted = discounted * (1 - percent / 100)
return round(discounted, 2)
def order_total(items: list[tuple[float, int]], coupon_percent: float = 0) -> float:
"""Sum ``(unit_price, quantity)`` lines and apply an optional coupon."""
subtotal = sum(price * qty for price, qty in items)
return apply_discount(subtotal, coupon_percent) if coupon_percent else round(subtotal, 2)
@@ -0,0 +1,18 @@
import unittest
from shop.pricing import apply_discount, order_total
class PricingTest(unittest.TestCase):
def test_ten_percent(self):
self.assertEqual(apply_discount(100.0, 10), 90.0)
def test_zero(self):
self.assertEqual(apply_discount(59.99, 0), 59.99)
def test_order_total_with_coupon(self):
self.assertEqual(order_total([(20.0, 2), (10.0, 1)], coupon_percent=10), 45.0)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,697 @@
#!/usr/bin/env python3
"""Deterministic stdio MCP server standing in for one enterprise service.
python3 mcp_server.py <service> <calls.jsonl>
Exposes realistic, full-size tool schemas (GitHub, Jira, Slack, Datadog, PagerDuty,
Confluence) so a coding agent sees a catalog the size real enterprise setups have.
Every tools/call is appended to calls.jsonl so the benchmark can check the task was done.
Standard library only; speaks newline-delimited JSON-RPC 2.0 (MCP stdio transport).
"""
from __future__ import annotations
import json
import sys
SERVICE = sys.argv[1]
CALL_LOG = sys.argv[2]
# (name, description, {param: (type, description)}, required)
CATALOG = {
"github": [
(
"search_repositories",
"Search GitHub repositories by keyword, language, topic, stars or owner. Returns repository metadata including default branch, visibility, open issue count and last push time.",
{
"query": (
"string",
"GitHub search syntax, e.g. 'billing language:python org:acme'",
),
"sort": ("string", "stars, forks, updated or best-match"),
"per_page": ("integer", "Results per page, 1-100"),
"page": ("integer", "Page number, starting at 1"),
},
["query"],
),
(
"get_repository",
"Get full metadata for one repository: description, topics, default branch, branch protection summary, languages, license and permissions of the caller.",
{
"owner": ("string", "Repository owner, user or organization"),
"repo": ("string", "Repository name"),
},
["owner", "repo"],
),
(
"list_pull_requests",
"List pull requests in a repository with filters for state, base branch, head branch and author. Includes review state, mergeability and CI status summary.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"state": ("string", "open, closed or all"),
"base": ("string", "Filter by base branch"),
"author": ("string", "Filter by author login"),
},
["owner", "repo"],
),
(
"get_pull_request",
"Get a pull request including body, commits, changed files with patch hunks, review comments, requested reviewers and status checks.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"number": ("integer", "Pull request number"),
},
["owner", "repo", "number"],
),
(
"create_pull_request",
"Open a pull request from a head branch into a base branch. Supports draft pull requests, maintainer edits and an initial set of reviewers and labels.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"title": ("string", "Pull request title"),
"head": ("string", "Branch with your changes"),
"base": ("string", "Branch to merge into"),
"body": ("string", "Markdown description"),
"draft": ("boolean", "Open as a draft"),
},
["owner", "repo", "title", "head", "base"],
),
(
"merge_pull_request",
"Merge a pull request using merge, squash or rebase. Fails if required checks or reviews are missing unless the caller can bypass branch protection.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"number": ("integer", "Pull request number"),
"method": ("string", "merge, squash or rebase"),
"commit_title": ("string", "Title for the merge commit"),
},
["owner", "repo", "number"],
),
(
"list_issues",
"List issues with filters for labels, assignee, milestone, state and creation date. Pull requests are excluded unless requested.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"labels": ("string", "Comma-separated label names"),
"assignee": ("string", "Assignee login, 'none' or '*'"),
"state": ("string", "open, closed or all"),
},
["owner", "repo"],
),
(
"create_issue",
"Create an issue with title, body, labels, assignees and milestone.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"title": ("string", "Issue title"),
"body": ("string", "Markdown body"),
"labels": ("array", "Label names"),
"assignees": ("array", "Assignee logins"),
},
["owner", "repo", "title"],
),
(
"add_issue_comment",
"Add a comment to an issue or pull request conversation.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"number": ("integer", "Issue or pull request number"),
"body": ("string", "Markdown comment"),
},
["owner", "repo", "number", "body"],
),
(
"get_file_contents",
"Read a file or list a directory at a ref. Returns decoded content for text files and metadata for binaries.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"path": ("string", "Path within the repository"),
"ref": ("string", "Branch, tag or commit SHA"),
},
["owner", "repo", "path"],
),
(
"list_commits",
"List commits on a branch or touching a path, newest first, with author, message and verification status.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"sha": ("string", "Branch or commit to start from"),
"path": ("string", "Only commits touching this path"),
"since": ("string", "ISO 8601 timestamp"),
},
["owner", "repo"],
),
(
"get_workflow_runs",
"List GitHub Actions workflow runs with status, conclusion, triggering event, branch and timing.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"workflow": ("string", "Workflow file name or ID"),
"branch": ("string", "Filter by branch"),
"status": ("string", "queued, in_progress, completed, failure, success"),
},
["owner", "repo"],
),
(
"get_job_logs",
"Download the log of one workflow job, optionally only the failing step.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"job_id": ("integer", "Workflow job ID"),
"failed_only": ("boolean", "Return only the failing step"),
},
["owner", "repo", "job_id"],
),
(
"create_branch",
"Create a branch from an existing ref.",
{
"owner": ("string", "Repository owner"),
"repo": ("string", "Repository name"),
"branch": ("string", "New branch name"),
"from_ref": ("string", "Branch, tag or SHA to branch from"),
},
["owner", "repo", "branch"],
),
(
"search_code",
"Search code across repositories with GitHub code search syntax. Returns file paths, repository and matching fragments.",
{"query": ("string", "Code search query"), "per_page": ("integer", "Results per page")},
["query"],
),
],
"jira": [
(
"search_issues",
"Search Jira issues with JQL. Returns key, summary, status, assignee, priority, labels and updated time for each match.",
{
"jql": ("string", "JQL query, e.g. 'project = BENCH AND status = Open'"),
"max_results": ("integer", "Maximum issues to return"),
"fields": ("array", "Fields to include"),
},
["jql"],
),
(
"get_issue",
"Get one Jira issue with description, comments, status history, linked issues, components, fix versions and custom fields.",
{
"issue_key": ("string", "Issue key, e.g. BENCH-42"),
"expand": ("string", "Extra sections such as changelog or renderedFields"),
},
["issue_key"],
),
(
"create_issue",
"Create a Jira issue in a project with type, summary, description, priority, labels and assignee.",
{
"project": ("string", "Project key"),
"issue_type": ("string", "Bug, Task, Story or Epic"),
"summary": ("string", "One-line summary"),
"description": ("string", "Description in Jira markdown"),
"priority": ("string", "Highest, High, Medium, Low"),
"labels": ("array", "Labels"),
},
["project", "issue_type", "summary"],
),
(
"update_issue",
"Update fields on an issue: summary, description, labels, priority, assignee or custom fields.",
{"issue_key": ("string", "Issue key"), "fields": ("object", "Field name to new value")},
["issue_key", "fields"],
),
(
"add_comment",
"Add a comment to a Jira issue. Supports Jira markdown and @mentions.",
{
"issue_key": ("string", "Issue key, e.g. BENCH-42"),
"body": ("string", "Comment text"),
},
["issue_key", "body"],
),
(
"transition_issue",
"Move an issue through its workflow, for example To Do -> In Progress -> Done, with an optional resolution and comment.",
{
"issue_key": ("string", "Issue key"),
"transition": ("string", "Transition name or ID"),
"resolution": ("string", "Resolution when closing"),
"comment": ("string", "Comment to add with the transition"),
},
["issue_key", "transition"],
),
(
"assign_issue",
"Assign an issue to a user, or unassign it.",
{
"issue_key": ("string", "Issue key"),
"assignee": ("string", "Account ID, or null to unassign"),
},
["issue_key"],
),
(
"list_sprints",
"List sprints on a board with state and dates.",
{
"board_id": ("integer", "Agile board ID"),
"state": ("string", "active, future or closed"),
},
["board_id"],
),
(
"get_sprint_issues",
"List issues in a sprint with status and story points.",
{"sprint_id": ("integer", "Sprint ID"), "jql": ("string", "Extra JQL filter")},
["sprint_id"],
),
(
"link_issues",
"Link two issues with a relationship such as blocks, relates to or duplicates.",
{
"inward_issue": ("string", "Inward issue key"),
"outward_issue": ("string", "Outward issue key"),
"link_type": ("string", "Link type name"),
},
["inward_issue", "outward_issue", "link_type"],
),
(
"add_worklog",
"Log time spent on an issue.",
{
"issue_key": ("string", "Issue key"),
"time_spent": ("string", "Duration such as 2h 30m"),
"comment": ("string", "Work description"),
},
["issue_key", "time_spent"],
),
(
"get_project",
"Get a project's issue types, components, versions and workflow scheme.",
{"project": ("string", "Project key")},
["project"],
),
],
"slack": [
(
"post_message",
"Post a message to a Slack channel or thread. Supports mrkdwn formatting, blocks and unfurl settings.",
{
"channel": ("string", "Channel name like #eng-alerts or channel ID"),
"text": ("string", "Message text in mrkdwn"),
"thread_ts": ("string", "Reply in this thread"),
},
["channel", "text"],
),
(
"list_channels",
"List public and private channels the bot can see, with member counts and topics.",
{
"types": ("string", "public_channel, private_channel, mpim, im"),
"limit": ("integer", "Maximum channels"),
},
[],
),
(
"get_channel_history",
"Read recent messages from a channel, newest first.",
{
"channel": ("string", "Channel name or ID"),
"limit": ("integer", "Maximum messages"),
"oldest": ("string", "Only messages after this timestamp"),
},
["channel"],
),
(
"get_thread_replies",
"Read every reply in a message thread.",
{
"channel": ("string", "Channel ID"),
"thread_ts": ("string", "Parent message timestamp"),
},
["channel", "thread_ts"],
),
(
"search_messages",
"Search messages across the workspace with Slack search modifiers such as in:, from: and before:.",
{"query": ("string", "Search query"), "count": ("integer", "Results to return")},
["query"],
),
(
"add_reaction",
"Add an emoji reaction to a message.",
{
"channel": ("string", "Channel ID"),
"timestamp": ("string", "Message timestamp"),
"name": ("string", "Emoji name without colons"),
},
["channel", "timestamp", "name"],
),
(
"get_user_profile",
"Get a user's profile, title, time zone and status.",
{"user": ("string", "User ID or email")},
["user"],
),
(
"set_channel_topic",
"Set a channel's topic.",
{"channel": ("string", "Channel ID"), "topic": ("string", "New topic")},
["channel", "topic"],
),
(
"upload_file",
"Upload a file or snippet to one or more channels.",
{
"channels": ("array", "Channel IDs"),
"content": ("string", "File content"),
"filename": ("string", "File name"),
"title": ("string", "Title"),
},
["channels", "content"],
),
(
"schedule_message",
"Schedule a message for later delivery.",
{
"channel": ("string", "Channel ID"),
"text": ("string", "Message text"),
"post_at": ("integer", "Unix timestamp"),
},
["channel", "text", "post_at"],
),
],
"datadog": [
(
"query_metrics",
"Query a metric time series with Datadog query syntax over a time range. Returns points, unit and aggregation.",
{
"query": ("string", "e.g. avg:billing.api.latency{env:prod} by {endpoint}"),
"from_ts": ("integer", "Start, Unix seconds"),
"to_ts": ("integer", "End, Unix seconds"),
},
["query", "from_ts", "to_ts"],
),
(
"search_logs",
"Search logs with Datadog log query syntax, newest first, including attributes and tags.",
{
"query": ("string", "Log query, e.g. service:billing status:error"),
"from_time": ("string", "Relative or absolute start, e.g. now-1h"),
"limit": ("integer", "Maximum log events"),
},
["query"],
),
(
"list_monitors",
"List monitors with state, query, thresholds, tags and notification targets.",
{
"name": ("string", "Filter by name"),
"tags": ("string", "Comma-separated tags"),
"group_states": ("string", "alert, warn, no data, ok"),
},
[],
),
(
"get_monitor",
"Get one monitor's definition, current state per group and recent state changes.",
{"monitor_id": ("integer", "Monitor ID")},
["monitor_id"],
),
(
"mute_monitor",
"Mute a monitor, optionally for one scope and until a time.",
{
"monitor_id": ("integer", "Monitor ID"),
"scope": ("string", "Scope to mute"),
"end": ("integer", "Unix time to unmute"),
},
["monitor_id"],
),
(
"list_dashboards",
"List dashboards with title, author and URL.",
{"filter": ("string", "Title filter")},
[],
),
(
"get_apm_traces",
"Search APM traces by service, resource and duration, with span breakdown.",
{
"service": ("string", "Service name"),
"resource": ("string", "Resource name"),
"min_duration_ms": ("integer", "Minimum duration"),
},
["service"],
),
(
"list_incidents",
"List incidents with severity, state, commander and timeline summary.",
{
"state": ("string", "active, stable or resolved"),
"severity": ("string", "SEV-1 to SEV-5"),
},
[],
),
(
"create_event",
"Post an event to the event stream, for example a deploy marker.",
{
"title": ("string", "Event title"),
"text": ("string", "Event body"),
"tags": ("array", "Tags"),
},
["title", "text"],
),
(
"get_service_dependencies",
"Get upstream and downstream dependencies of a service from APM.",
{"service": ("string", "Service name"), "env": ("string", "Environment")},
["service"],
),
],
"pagerduty": [
(
"list_incidents",
"List PagerDuty incidents with urgency, status, service and assignments.",
{
"statuses": ("array", "triggered, acknowledged, resolved"),
"service_ids": ("array", "Service IDs"),
"since": ("string", "ISO 8601 start"),
},
[],
),
(
"get_incident",
"Get one incident with timeline, alerts, notes and responders.",
{"incident_id": ("string", "Incident ID")},
["incident_id"],
),
(
"acknowledge_incident",
"Acknowledge an incident on behalf of the caller.",
{"incident_id": ("string", "Incident ID")},
["incident_id"],
),
(
"resolve_incident",
"Resolve an incident with an optional resolution note.",
{"incident_id": ("string", "Incident ID"), "note": ("string", "Resolution note")},
["incident_id"],
),
(
"list_oncalls",
"List who is on call for schedules and escalation policies.",
{
"schedule_ids": ("array", "Schedule IDs"),
"since": ("string", "Start time"),
"until": ("string", "End time"),
},
[],
),
(
"add_incident_note",
"Add a note to an incident timeline.",
{"incident_id": ("string", "Incident ID"), "content": ("string", "Note text")},
["incident_id", "content"],
),
(
"list_services",
"List services with escalation policy and integrations.",
{"query": ("string", "Name filter")},
[],
),
(
"create_incident",
"Trigger a new incident on a service.",
{
"service_id": ("string", "Service ID"),
"title": ("string", "Incident title"),
"urgency": ("string", "high or low"),
"body": ("string", "Details"),
},
["service_id", "title"],
),
],
"confluence": [
(
"search_pages",
"Search Confluence pages with CQL, returning title, space, excerpt and last editor.",
{
"cql": ("string", "CQL query, e.g. space = ENG and text ~ 'discount'"),
"limit": ("integer", "Maximum results"),
},
["cql"],
),
(
"get_page",
"Get a page body in storage format with version, ancestors and labels.",
{
"page_id": ("string", "Page ID"),
"expand": ("string", "body.storage, version, ancestors"),
},
["page_id"],
),
(
"create_page",
"Create a page in a space, optionally under a parent page.",
{
"space": ("string", "Space key"),
"title": ("string", "Page title"),
"body": ("string", "Body in storage format"),
"parent_id": ("string", "Parent page ID"),
},
["space", "title", "body"],
),
(
"update_page",
"Update a page body; requires the next version number.",
{
"page_id": ("string", "Page ID"),
"title": ("string", "Title"),
"body": ("string", "New body"),
"version": ("integer", "Next version number"),
},
["page_id", "body", "version"],
),
(
"add_page_comment",
"Add a footer comment to a page.",
{"page_id": ("string", "Page ID"), "body": ("string", "Comment body")},
["page_id", "body"],
),
(
"list_spaces",
"List spaces with key, name, type and homepage.",
{"type": ("string", "global or personal"), "limit": ("integer", "Maximum spaces")},
[],
),
(
"get_page_children",
"List child pages of a page.",
{"page_id": ("string", "Page ID")},
["page_id"],
),
(
"add_label",
"Add labels to a page.",
{"page_id": ("string", "Page ID"), "labels": ("array", "Labels")},
["page_id", "labels"],
),
],
}
ISSUE = {
"key": "BENCH-42",
"summary": "Discount applied twice at checkout",
"status": "Open",
"priority": "High",
"description": (
"Customers with a 10% coupon are charged 81% of the price instead of 90%. "
"The bug is in shop/pricing.py, function apply_discount. "
"tests/test_pricing.py reproduces it. When fixed, comment on this issue with a one-line "
"summary of the fix and post a short note to the #eng-alerts Slack channel."
),
"comments": [],
}
def schema(params: dict, required: list) -> dict:
props = {}
for name, (typ, desc) in params.items():
p = {"type": typ, "description": desc}
if typ == "array":
p["items"] = {"type": "string"}
props[name] = p
return {"type": "object", "properties": props, "required": required}
TOOLS = [
{"name": n, "description": d, "inputSchema": schema(p, r)} for n, d, p, r in CATALOG[SERVICE]
]
def call(name: str, args: dict) -> str:
with open(CALL_LOG, "a") as fh:
fh.write(json.dumps({"service": SERVICE, "tool": name, "args": args}) + "\n")
if SERVICE == "jira" and name == "get_issue":
return json.dumps(ISSUE if args.get("issue_key") == "BENCH-42" else {"error": "not found"})
if SERVICE == "jira" and name == "search_issues":
return json.dumps(
{"issues": [{k: ISSUE[k] for k in ("key", "summary", "status", "priority")}]}
)
return json.dumps({"ok": True, "service": SERVICE, "tool": name})
def main() -> None:
for line in sys.stdin:
try:
msg = json.loads(line)
except ValueError:
continue
mid, method = msg.get("id"), msg.get("method")
if mid is None: # notification
continue
if method == "initialize":
result = {
"protocolVersion": msg.get("params", {}).get("protocolVersion", "2025-06-18"),
"capabilities": {"tools": {}},
"serverInfo": {"name": SERVICE, "version": "1.0.0"},
}
elif method == "tools/list":
result = {"tools": TOOLS}
elif method == "tools/call":
p = msg.get("params", {})
result = {
"content": [{"type": "text", "text": call(p.get("name"), p.get("arguments") or {})}]
}
elif method == "ping":
result = {}
else:
sys.stdout.write(
json.dumps(
{
"jsonrpc": "2.0",
"id": mid,
"error": {"code": -32601, "message": f"unknown method {method}"},
}
)
+ "\n"
)
sys.stdout.flush()
continue
sys.stdout.write(json.dumps({"jsonrpc": "2.0", "id": mid, "result": result}) + "\n")
sys.stdout.flush()
if __name__ == "__main__":
main()
+402
View File
@@ -0,0 +1,402 @@
#!/usr/bin/env python3
"""Claude Code native tool search vs Headroom server-side tool search.
python3 run.py --arms native,hr_server --reps 3 --model claude-sonnet-5-5
Every run is a fresh, isolated Claude Code (throwaway HOME, fresh copy of the fixture
repo, six fake enterprise MCP servers) doing the same task: read a Jira ticket, fix the
bug, run the tests, comment on the ticket, post to Slack. Usage is read from Claude
Code's own transcript (what the API billed), so the numbers include subagents.
Arms:
native Claude Code straight to api.anthropic.com, its own ToolSearch (default on)
eager Claude Code straight to Anthropic with ENABLE_TOOL_SEARCH=false (no deferral)
hr_client Claude Code -> Headroom, ENABLE_TOOL_SEARCH=true (today's `headroom wrap`)
hr_server Claude Code -> Headroom, ENABLE_TOOL_SEARCH=false; Headroom defers server-side
hr_hot as hr_server, but the MCP tools this developer used before stay loaded
(HEADROOM_TOOL_SEARCH_CORE_TOOLS): no lookup at all for routine tools
hr_ext as hr_server, plus the proxy env given with --proxy-env KEY=VALUE
(repeatable), e.g. to measure a proxy extension
Needs ANTHROPIC_API_KEY (or ~/env.txt). Results go to results/<timestamp>/.
"""
from __future__ import annotations
import argparse
import glob
import json
import os
import shutil
import socket
import statistics
import subprocess
import sys
import time
from pathlib import Path
HERE = Path(__file__).resolve().parent
REPO = HERE.parents[1]
SERVERS = ["github", "jira", "slack", "datadog", "pagerduty", "confluence"]
TASK = (
"Jira ticket BENCH-42 reports a pricing bug in this repository. Look the ticket up in Jira, "
"fix the bug, and run the tests with `python3 -m unittest`. Then search Datadog logs for "
"errors from service billing in the last hour, add a comment on BENCH-42 with a one-line "
"summary of the fix and the Datadog result, and post a short note about the fix to the "
"#eng-alerts Slack channel. Do not ask questions; just do it."
)
# Headroom's default resident set (headroom/proxy/helpers.py _TOOL_SEARCH_CORE_TOOLS) plus the
# MCP tools a developer doing this kind of work has used before: a learned working set.
CORE = [
"bash",
"bash_background",
"bash_background_output",
"bash_background_wait",
"bash_background_kill",
"read",
"write",
"edit",
"multiedit",
"apply_patch",
"glob",
"grep",
"task",
"todowrite",
"todoread",
"webfetch",
"question",
"skill",
"toolsearch",
]
HOT = [
"mcp__jira__get_issue",
"mcp__jira__add_comment",
"mcp__slack__post_message",
"mcp__datadog__search_logs",
]
PROXY_ENV = {
"hr_client": {},
"hr_server": {},
"hr_hot": {"HEADROOM_TOOL_SEARCH_CORE_TOOLS": ",".join(CORE + HOT)},
# As hr_server plus whatever --proxy-env adds, e.g. a proxy extension under test.
"hr_ext": {},
}
ALLOWED = ["Bash", "Read", "Edit", "Write", "Glob", "Grep", "ToolSearch"] + [
f"mcp__{s}" for s in SERVERS
]
# Billed input-equivalent weights (Anthropic): cache write 5m 1.25x, 1h 2x, read 0.1x.
W_5M, W_1H, W_READ = 1.25, 2.0, 0.1
def api_key() -> str:
if os.environ.get("ANTHROPIC_API_KEY"):
return os.environ["ANTHROPIC_API_KEY"]
for line in Path.home().joinpath("env.txt").read_text().splitlines():
line = line.strip().removeprefix("export ").strip()
if line.startswith("ANTHROPIC_API_KEY="):
return line.split("=", 1)[1].strip().strip("'\"")
sys.exit("ANTHROPIC_API_KEY not found")
def free_port() -> int:
with socket.socket() as s:
s.bind(("127.0.0.1", 0))
return s.getsockname()[1]
def start_proxy(out: Path, key: str, name: str, extra: dict) -> tuple[subprocess.Popen, int]:
port = free_port()
ws = out / f"headroom-workspace-{name}"
ws.mkdir(parents=True, exist_ok=True)
env = {
**os.environ,
"HEADROOM_WORKSPACE_DIR": str(ws),
"HEADROOM_TOOL_SEARCH": "1",
"HEADROOM_TELEMETRY": "off",
"ANTHROPIC_API_KEY": key,
**extra,
}
env.pop("ANTHROPIC_BASE_URL", None)
log = open(out / f"proxy-{name}.log", "w")
proc = subprocess.Popen(
[str(REPO / ".venv/bin/headroom"), "proxy", "--port", str(port)],
env=env,
stdout=log,
stderr=subprocess.STDOUT,
cwd=ws,
)
for _ in range(120):
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return proc, port
except OSError:
time.sleep(0.5)
proc.kill()
sys.exit(f"proxy did not start; see {out / f'proxy-{name}.log'}")
def usage_from_transcripts(home: Path) -> dict:
seen, calls = set(), []
for f in glob.glob(str(home / ".claude/projects/*/**/*.jsonl"), recursive=True):
for line in open(f):
try:
r = json.loads(line)
except ValueError:
continue
m = r.get("message") or {}
if m.get("role") != "assistant" or not m.get("usage") or m.get("id") in seen:
continue
seen.add(m.get("id"))
u = m["usage"]
cc = u.get("cache_creation") or {}
w1h = cc.get("ephemeral_1h_input_tokens", 0)
w5m = cc.get("ephemeral_5m_input_tokens", u.get("cache_creation_input_tokens", 0) - w1h)
tool_names = [
b.get("name")
for b in m.get("content") or []
if isinstance(b, dict) and b.get("type") in ("tool_use", "server_tool_use")
]
calls.append(
{
"input": u.get("input_tokens", 0),
"w5m": w5m,
"w1h": w1h,
"read": u.get("cache_read_input_tokens", 0),
"output": u.get("output_tokens", 0),
"sidechain": bool(r.get("isSidechain")),
"tools": tool_names,
}
)
t = {k: sum(c[k] for c in calls) for k in ("input", "w5m", "w1h", "read", "output")}
t["api_calls"] = len(calls)
t["toolsearch_calls"] = sum(c["tools"].count("ToolSearch") for c in calls)
t["toolsearch_only_turns"] = sum(
1 for c in calls if c["tools"] and set(c["tools"]) == {"ToolSearch"}
)
t["server_searches"] = sum(
1 for c in calls for n in c["tools"] if n and n.startswith("tool_search_tool")
)
t["billed_input_equiv"] = round(
t["input"] + W_5M * t["w5m"] + W_1H * t["w1h"] + W_READ * t["read"]
)
return t
def check_task(work: Path, calls_log: Path) -> dict:
tests = subprocess.run(
[sys.executable, "-m", "unittest", "-q"], cwd=work, capture_output=True, text=True
)
calls = (
[json.loads(line) for line in calls_log.read_text().splitlines()]
if calls_log.exists()
else []
)
did = lambda svc, tool: any(c["service"] == svc and c["tool"] == tool for c in calls) # noqa: E731
return {
"tests_pass": tests.returncode == 0,
"read_ticket": did("jira", "get_issue"), # a search result has no description
"searched_logs": did("datadog", "search_logs"),
"commented": did("jira", "add_comment"),
"posted_slack": did("slack", "post_message"),
"mcp_calls": len(calls),
}
def run_one(
arm: str, rep: int, out: Path, key: str, model: str, port: int | None, context_kb: int
) -> dict:
d = out / f"{arm}-{rep}"
home, work = d / "home", d / "work"
home.mkdir(parents=True)
shutil.copytree(HERE / "fixture", work)
task = TASK
if context_kb: # put the MCP lookups deep in the session, where real ones happen
(work / "docs").mkdir()
(work / "docs/ARCHITECTURE.md").write_text(architecture_doc(context_kb))
task = "First read docs/ARCHITECTURE.md in full (all of it, in chunks if needed). " + TASK
subprocess.run(["git", "init", "-q"], cwd=work)
subprocess.run(["git", "add", "-A"], cwd=work)
subprocess.run(
["git", "-c", "user.email=b@b", "-c", "user.name=b", "commit", "-qm", "init"], cwd=work
)
calls_log = d / "mcp_calls.jsonl"
cfg = {
"mcpServers": {
s: {"command": sys.executable, "args": [str(HERE / "mcp_server.py"), s, str(calls_log)]}
for s in SERVERS
}
}
(d / "mcp.json").write_text(json.dumps(cfg))
env = {
k: v
for k, v in os.environ.items()
if not k.startswith(("ANTHROPIC_", "CLAUDE_", "ENABLE_TOOL_SEARCH", "HEADROOM_"))
}
env.update(
{
"HOME": str(home),
"ANTHROPIC_API_KEY": key,
"DISABLE_AUTOUPDATER": "1",
"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
"DISABLE_TELEMETRY": "1",
}
)
if arm == "eager":
env["ENABLE_TOOL_SEARCH"] = "false"
elif arm in PROXY_ENV:
env["ANTHROPIC_BASE_URL"] = f"http://127.0.0.1:{port}"
env["ENABLE_TOOL_SEARCH"] = "true" if arm == "hr_client" else "false"
cmd = [
"claude",
"-p",
task,
"--model",
model,
"--mcp-config",
str(d / "mcp.json"),
"--strict-mcp-config",
"--output-format",
"json",
"--allowedTools",
*ALLOWED,
]
t0 = time.time()
p = subprocess.run(cmd, cwd=work, env=env, capture_output=True, text=True, timeout=900)
wall = time.time() - t0
(d / "stdout.json").write_text(p.stdout)
(d / "stderr.txt").write_text(p.stderr)
try:
cc = json.loads(p.stdout)
except ValueError:
cc = {}
row = {
"arm": arm,
"rep": rep,
"wall_s": round(wall, 1),
"exit": p.returncode,
"cc_cost_usd": cc.get("total_cost_usd"),
"cc_turns": cc.get("num_turns"),
**usage_from_transcripts(home),
**check_task(work, calls_log),
}
# A session that errored out does not pass, even if it got the actions done.
row["task_ok"] = row["exit"] == 0 and all(
row[k] for k in ("tests_pass", "read_ticket", "searched_logs", "commented", "posted_slack")
)
return row
def architecture_doc(kb: int) -> str:
"""Deterministic, non-repetitive prose so compression cannot trivially erase it."""
import random
rng = random.Random(42)
words = (
"ledger invoice coupon tenant region shard replica queue retry idempotency webhook settlement "
"refund currency rounding audit partition latency throughput cache eviction migration rollout "
"flag canary checkout cart session token gateway schema index backfill reconciliation"
).split()
out, size, n = ["# Billing platform architecture\n"], 0, 0
while size < kb * 1024:
n += 1
para = f"\n## Section {n}\n" + " ".join(rng.choice(words) for _ in range(120)) + ".\n"
out.append(para)
size += len(para)
return "".join(out)
def summarize(rows: list[dict]) -> str:
keys = [
"task_ok",
"cc_cost_usd",
"billed_input_equiv",
"output",
"api_calls",
"toolsearch_only_turns",
"server_searches",
"read",
"w5m",
"w1h",
"input",
"wall_s",
]
lines = ["| arm | n | " + " | ".join(keys) + " |", "|" + "---|" * (len(keys) + 2)]
for arm in dict.fromkeys(r["arm"] for r in rows):
rs = [r for r in rows if r["arm"] == arm]
cells = []
for k in keys:
vals = [r[k] for r in rs if r.get(k) is not None]
if k == "task_ok":
cells.append(f"{sum(vals)}/{len(rs)}")
elif not vals:
cells.append("-")
else:
med = statistics.median(vals)
cells.append(
f"{med:.3f}"
if k == "cc_cost_usd"
else f"{med:,.0f}"
if k != "wall_s"
else f"{med:.0f}"
)
lines.append(f"| {arm} | {len(rs)} | " + " | ".join(cells) + " |")
return "\n".join(lines) + "\n\n(median per run)"
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--arms", default="native,hr_server,hr_client,eager")
ap.add_argument("--reps", type=int, default=3)
ap.add_argument("--model", default="claude-sonnet-5-5")
ap.add_argument(
"--context-kb", type=int, default=0, help="add a doc of this size to read before the task"
)
ap.add_argument(
"--warmup",
action="store_true",
help="run each arm once first and discard it (warms caches)",
)
ap.add_argument(
"--proxy-env",
action="append",
default=[],
metavar="KEY=VALUE",
help="extra proxy environment for the hr_ext arm (repeatable)",
)
ap.add_argument("--out", default=str(HERE / "results" / time.strftime("%Y%m%d-%H%M%S")))
a = ap.parse_args()
out = Path(a.out)
out.mkdir(parents=True, exist_ok=True)
key = api_key()
arms = a.arms.split(",")
PROXY_ENV["hr_ext"] = dict(kv.split("=", 1) for kv in a.proxy_env)
proxies = {arm: start_proxy(out, key, arm, PROXY_ENV[arm]) for arm in arms if arm in PROXY_ENV}
rows = []
try:
if a.warmup:
for arm in arms:
run_one(arm, 0, out, key, a.model, proxies.get(arm, (None, None))[1], a.context_kb)
for rep in range(1, a.reps + 1): # interleave arms so drift hits every arm alike
for arm in arms:
row = run_one(
arm, rep, out, key, a.model, proxies.get(arm, (None, None))[1], a.context_kb
)
rows.append(row)
print(json.dumps(row), flush=True)
(out / "rows.jsonl").open("a").write(json.dumps(row) + "\n")
finally:
for proc, _ in proxies.values():
proc.terminate()
s = summarize(rows)
(out / "summary.md").write_text(s + "\n")
print(s)
if __name__ == "__main__":
main()
+59
View File
@@ -442,6 +442,63 @@ def _replaced_system_tail_delta(
return None
def _block_identity(block: Any) -> tuple[Any, ...]:
"""What identifies a reply block across a client's own re-serialization."""
if not isinstance(block, dict):
return ("raw", block)
kind = block.get("type")
if kind in ("tool_use", "server_tool_use"):
return (kind, block.get("id"), block.get("name"))
return (kind,)
def _client_rewrote_reply_delta(
current_messages: list[dict[str, Any]],
previous_original_messages: list[dict[str, Any]],
previous_forwarded_messages: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]] | None:
"""Cache-stable split when the client re-serialized the model's last reply.
The handlers record the model's raw reply as the last message of the previous
turn, but a client may send that reply back edited: Claude Code strips a leading
``cd <cwd> &&`` from Bash commands and drops the ``caller`` annotation. Every
earlier message still matches, and the reply itself was model output, never part
of a cached prefix, so the forwarded history up to the reply is still exactly what
the provider cached. Replay it, take the client's version of the reply, and
treat what follows as the delta. Falling back to raw forwarding here instead
replaces every earlier compressed message with its original and re-writes the
whole conversation to cache.
The reply must be the same turn: an assistant message with the same block kinds
in the same order and the same tool_use ids.
"""
n = len(previous_original_messages)
if n < 2 or len(previous_forwarded_messages) != n or len(current_messages) < n:
return None
prev_reply, cur_reply = previous_original_messages[-1], current_messages[n - 1]
if not (
isinstance(prev_reply, dict)
and isinstance(cur_reply, dict)
and prev_reply.get("role") == "assistant"
and cur_reply.get("role") == "assistant"
):
return None
if _canonicalize_for_prefix_compare(
current_messages[: n - 1]
) != _canonicalize_for_prefix_compare(previous_original_messages[: n - 1]):
return None
prev_blocks = _canonicalize_for_prefix_compare(prev_reply).get("content")
cur_blocks = _canonicalize_for_prefix_compare(cur_reply).get("content")
if not isinstance(prev_blocks, list) or not isinstance(cur_blocks, list):
return None
if [_block_identity(b) for b in prev_blocks] != [_block_identity(b) for b in cur_blocks]:
return None
return (
copy.deepcopy(previous_forwarded_messages[: n - 1] + [cur_reply]),
copy.deepcopy(current_messages[n:]),
)
def classify_history_relation(
current_messages: list[dict[str, Any]],
previous_messages: list[dict[str, Any]],
@@ -513,6 +570,8 @@ def extract_cache_stable_delta(
if relation.kind == RELATION_DIVERGED:
return _replaced_system_tail_delta(
current_messages, previous_original_messages, previous_forwarded_messages
) or _client_rewrote_reply_delta(
current_messages, previous_original_messages, previous_forwarded_messages
)
if relation.kind not in (RELATION_EXACT, RELATION_MESSAGE_APPEND):
# A same-message block append needs a block-level splice in
+56
View File
@@ -15,6 +15,7 @@ import uuid
from collections.abc import Callable
from datetime import datetime
from functools import lru_cache
from pathlib import Path
from typing import TYPE_CHECKING, Any
from urllib.parse import urlsplit
@@ -287,6 +288,57 @@ def _looks_like_sse_response(response: httpx.Response) -> bool:
return head.startswith(b"event:") or head.startswith(b"data:")
def _dump_prefix_mismatch(request_id: str, current: list, previous: list) -> None:
"""Debug aid (HEADROOM_DEBUG_PREFIX_MISMATCH=<dir>): log where the client's
history stopped matching the last turn, and dump both sides for diffing.
The dump holds the full conversation, user content included. It is for local
debugging only: never set this on a shared or production proxy.
"""
from headroom.cache.prefix_tracker import (
_canonicalize_for_prefix_compare,
classify_history_relation,
)
rel = classify_history_relation(current, previous)
first = next(
(
i
for i in range(min(len(current), len(previous)))
if _canonicalize_for_prefix_compare(current[i])
!= _canonicalize_for_prefix_compare(previous[i])
),
None,
)
logger.info(
"[%s] prefix_mismatch_debug relation=%s first_diff_index=%s current_len=%d previous_len=%d",
request_id,
rel.kind,
first,
len(current),
len(previous),
)
try:
out = Path(os.environ["HEADROOM_DEBUG_PREFIX_MISMATCH"]) / f"{request_id}.json"
out.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
# Owner-only: the dump holds the full conversation. The mode passed to
# os.open applies only when the file is created, so tighten an existing
# file too before writing into it.
fd = os.open(out, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
# The context manager owns the descriptor from here, so it is closed
# even if tightening the mode fails.
with os.fdopen(fd, "w") as fh:
if hasattr(os, "fchmod"): # POSIX; Windows has no POSIX modes to fix
os.fchmod(fh.fileno(), 0o600)
json.dump(
{"first_diff_index": first, "current": current, "previous": previous},
fh,
default=str,
)
except Exception: # noqa: BLE001 - debug aid only
pass
class AnthropicHandlerMixin:
"""Mixin providing Anthropic API handler methods for HeadroomProxy."""
@@ -2276,6 +2328,10 @@ class AnthropicHandlerMixin:
"[%s] Compression skipped: reason=cache_mode_prefix_mismatch",
request_id,
)
if os.environ.get("HEADROOM_DEBUG_PREFIX_MISMATCH"):
_dump_prefix_mismatch(
request_id, original_client_messages, previous_original_messages
)
optimized_messages = messages
optimized_tokens = original_tokens
elif tracker_frozen_count > 0:
+51
View File
@@ -1832,6 +1832,37 @@ class RouterCompressionResult:
)
def _mcp_result_min_chars() -> int:
"""Below this many characters an MCP tool result is never compressed."""
try:
return max(0, int(os.environ.get("HEADROOM_MCP_RESULT_MIN_CHARS", "4000")))
except ValueError:
return 4000
#: MCP tool-name prefixes seen on the wire: ``mcp__server__tool`` (Claude Code,
#: Codex, Droid), ``mcp_server_tool`` (Gemini, VS Code Copilot), ``mcp--server--tool``
#: (Roo). Harnesses that send bare MCP names cannot be told apart from built-ins.
_MCP_NAME_PREFIXES = ("mcp_", "mcp--")
def _is_small_mcp_result(tool_name: Any, text: Any) -> bool:
"""True when ``text`` is an MCP tool's output short enough to keep verbatim.
A Jira ticket or a Slack thread is a few hundred tokens of high-value fields.
Compressing it saves tens of tokens, and when the model misses a field it
re-calls the tool or headroom_retrieve: a whole extra turn that re-reads the
entire cached conversation (measured: +2 turns, +15% cost on
benchmarks/tool_search_vs_native). ``HEADROOM_MCP_RESULT_MIN_CHARS=0``
restores the old behaviour.
"""
return (
isinstance(text, str)
and str(tool_name or "").lower().startswith(_MCP_NAME_PREFIXES)
and len(text) < _mcp_result_min_chars()
)
@dataclass
class ContentRouterConfig:
"""Configuration for intelligent content routing.
@@ -6225,6 +6256,16 @@ class ContentRouter(Transform):
tool_name = tool_name_map.get(tool_call_id, "")
bias = self._get_tool_bias(tool_name) if tool_name else 1.0
# Small MCP results stay verbatim: twin of the Anthropic
# tool_result branch in _process_content_blocks.
if _is_small_mcp_result(tool_name, content):
result_slots[i] = message
transforms_applied.append("router:mcp_small_result_verbatim")
route_counts["mcp_small_result"] = route_counts.get("mcp_small_result", 0) + 1
if collect_diagnostics:
_diag[i] = "protected:mcp_small_result"
continue
# Bash-search lossless pre-empt: a read-only search (grep/rg/git
# grep) run via a shell tool yields byte-losslessly foldable
# output. Fold it instead of the lossy strategy path.
@@ -7375,6 +7416,16 @@ class ContentRouter(Transform):
else tool_content
)
# Small MCP results stay verbatim (see _is_small_mcp_result).
if _is_small_mcp_result(tool_name, tool_text):
new_blocks.append(block)
transforms_applied.append("router:mcp_small_result_verbatim")
if route_counts is not None:
route_counts["mcp_small_result"] = (
route_counts.get("mcp_small_result", 0) + 1
)
continue
# Bash-search lossless pre-empt (twin of the string-form path):
# fold read-only search output (grep/rg/git grep) byte-losslessly
# instead of taking the lossy strategy path.
+35 -2
View File
@@ -151,8 +151,11 @@ def test_tool_use_caller_annotation_ignored():
assert len(out[1]) == 1
def test_tool_use_different_input_still_falls_back():
# Safety: a real change to the tool command must still bail.
def test_tool_use_different_input_forwards_the_clients_version():
# The last recorded message is the model's raw reply, which clients re-serialize
# (Claude Code strips "cd <cwd> &&" from Bash). The client's version is what gets
# forwarded, never the recorded one, so a changed command reaches the provider
# exactly as the client sent it.
def _asst(cmd):
return {
"role": "assistant",
@@ -164,6 +167,36 @@ def test_tool_use_different_input_still_falls_back():
prev_orig = [B("user", "task"), _asst("ls")]
prev_fwd = copy.deepcopy(prev_orig)
cur = [B("user", "task"), _asst("rm -rf /"), _tr("t1", "out", as_string=True)]
out = delta(cur, prev_orig, prev_fwd)
assert out is not None
assert out[0][-1] == cur[1]
assert out[1] == cur[2:]
def test_tool_use_changed_earlier_in_history_still_falls_back():
# A change BEFORE the last recorded reply is a real divergence.
def _asst(cmd):
return {
"role": "assistant",
"content": [
{"type": "tool_use", "id": "tu1", "name": "bash", "input": {"command": cmd}}
],
}
prev_orig = [
B("user", "task"),
_asst("ls"),
_tr("tu1", "out", as_string=True),
B("assistant", "done"),
]
prev_fwd = copy.deepcopy(prev_orig)
cur = [
B("user", "task"),
_asst("rm -rf /"),
_tr("tu1", "out", as_string=True),
B("assistant", "done"),
B("user", "next"),
]
assert delta(cur, prev_orig, prev_fwd) is None
@@ -0,0 +1,132 @@
"""Small MCP tool results are forwarded verbatim (HEADROOM_MCP_RESULT_MIN_CHARS)."""
import json
from headroom.tokenizers import get_tokenizer
from headroom.transforms.content_router import ContentRouter
TICKET = json.dumps(
{
"key": "BENCH-42",
"summary": "Discount applied twice at checkout",
"status": "Open",
"description": "Customers with a 10% coupon are charged 81% of the price instead of 90%. "
"The bug is in shop/pricing.py, function apply_discount.",
"comments": [{"author": f"user{i}", "body": "seen this too " * 3} for i in range(8)],
}
)
def _messages(tool_name):
return [
{"role": "user", "content": "fix BENCH-42"},
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"id": "t1",
"name": tool_name,
"input": {"issue_key": "BENCH-42"},
}
],
},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "t1",
"content": [{"type": "text", "text": TICKET}],
}
],
},
{"role": "assistant", "content": "Reading the ticket."},
{"role": "user", "content": "go on"},
]
def _tool_result(out):
return out.messages[2]["content"][0]
def _apply(tool_name):
return ContentRouter().apply(
_messages(tool_name),
get_tokenizer("claude-sonnet-5-5"),
min_tokens_to_compress=10,
min_chars_for_block_compression=25,
)
def test_small_mcp_result_is_verbatim(monkeypatch):
monkeypatch.delenv("HEADROOM_MCP_RESULT_MIN_CHARS", raising=False)
out = _apply("mcp__jira__get_issue")
assert _tool_result(out)["content"][0]["text"] == TICKET
assert "router:mcp_small_result_verbatim" in out.transforms_applied
def test_disabled_floor_lets_the_router_see_it(monkeypatch):
monkeypatch.setenv("HEADROOM_MCP_RESULT_MIN_CHARS", "0")
out = _apply("mcp__jira__get_issue")
assert "router:mcp_small_result_verbatim" not in out.transforms_applied
def test_floor_can_be_disabled(monkeypatch):
from headroom.transforms import content_router as cr
monkeypatch.setenv("HEADROOM_MCP_RESULT_MIN_CHARS", "0")
assert cr._mcp_result_min_chars() == 0
monkeypatch.setenv("HEADROOM_MCP_RESULT_MIN_CHARS", "junk")
assert cr._mcp_result_min_chars() == 4000
def _openai_messages(tool_name):
return [
{"role": "user", "content": "fix BENCH-42"},
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call_1",
"type": "function",
"function": {"name": tool_name, "arguments": '{"issue_key": "BENCH-42"}'},
}
],
},
{"role": "tool", "tool_call_id": "call_1", "content": TICKET},
{"role": "assistant", "content": "Reading the ticket."},
{"role": "user", "content": "go on"},
]
def _apply_openai(tool_name):
return ContentRouter().apply(
_openai_messages(tool_name),
get_tokenizer("claude-sonnet-5-5"),
min_tokens_to_compress=10,
min_chars_for_block_compression=25,
)
def test_small_mcp_result_is_verbatim_openai_format(monkeypatch):
monkeypatch.delenv("HEADROOM_MCP_RESULT_MIN_CHARS", raising=False)
for name in ("mcp__jira__get_issue", "mcp_jira_get_issue", "mcp--jira--get_issue"):
out = _apply_openai(name)
assert out.messages[2]["content"] == TICKET, name
assert "router:mcp_small_result_verbatim" in out.transforms_applied, name
def test_openai_floor_can_be_disabled(monkeypatch):
monkeypatch.setenv("HEADROOM_MCP_RESULT_MIN_CHARS", "0")
out = _apply_openai("mcp__jira__get_issue")
assert "router:mcp_small_result_verbatim" not in out.transforms_applied
def test_non_mcp_tool_is_not_covered():
from headroom.transforms.content_router import _is_small_mcp_result
assert _is_small_mcp_result("mcp__jira__get_issue", "x" * 100)
assert not _is_small_mcp_result("Bash", "x" * 100)
assert not _is_small_mcp_result("mcp__jira__get_issue", "x" * 5000)
@@ -0,0 +1,148 @@
"""Cache mode must survive a client re-serializing the model's last reply.
Claude Code sends the model's Bash call back with the leading ``cd <cwd> &&``
stripped and without ``caller``. Before the fix the whole history counted as
diverged, cache mode fell back to raw forwarding, and every earlier compressed
message was replaced by its original: a full cache re-write (observed 66.8K
tokens per occurrence in benchmarks/tool_search_vs_native).
"""
import os
import stat
import pytest
from headroom.cache.prefix_tracker import extract_cache_stable_delta
USER = {"role": "user", "content": "fix the bug"}
RESULT_ORIGINAL = {
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": "x" * 5000}],
}
RESULT_COMPRESSED = {
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": "x…"}],
}
CALL = {
"role": "assistant",
"content": [{"type": "tool_use", "id": "t1", "name": "Read", "input": {"file_path": "a"}}],
}
RAW_REPLY = {
"role": "assistant",
"content": [
{"type": "text", "text": "Listing files."},
{
"type": "tool_use",
"id": "t2",
"name": "Bash",
"input": {"command": "cd /work && git ls-files"},
"caller": {"type": "direct"},
},
],
}
CLIENT_REPLY = {
"role": "assistant",
"content": [
{"type": "text", "text": "Listing files."},
{"type": "tool_use", "id": "t2", "name": "Bash", "input": {"command": "git ls-files"}},
],
}
NEXT = {
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "t2", "content": "a.py"}],
}
def test_rewritten_reply_keeps_compressed_prefix():
previous_original = [USER, CALL, RESULT_ORIGINAL, RAW_REPLY]
previous_forwarded = [USER, CALL, RESULT_COMPRESSED, RAW_REPLY]
current = [USER, CALL, RESULT_ORIGINAL, CLIENT_REPLY, NEXT]
delta = extract_cache_stable_delta(current, previous_original, previous_forwarded)
assert delta is not None
prefix, appended = delta
assert prefix[2] == RESULT_COMPRESSED # earlier compression replayed, cache intact
assert prefix[3] == CLIENT_REPLY # the client's own version of the reply
assert appended == [NEXT]
def test_different_turn_is_still_a_divergence():
other = {
"role": "assistant",
"content": [
{"type": "text", "text": "Listing files."},
{"type": "tool_use", "id": "t9", "name": "Bash", "input": {"command": "ls"}},
],
}
previous_original = [USER, CALL, RESULT_ORIGINAL, RAW_REPLY]
previous_forwarded = [USER, CALL, RESULT_COMPRESSED, RAW_REPLY]
assert (
extract_cache_stable_delta(
[USER, CALL, RESULT_ORIGINAL, other, NEXT], previous_original, previous_forwarded
)
is None
)
def test_earlier_history_edit_is_still_a_divergence():
edited_user = {"role": "user", "content": "something else"}
previous_original = [USER, CALL, RESULT_ORIGINAL, RAW_REPLY]
previous_forwarded = [USER, CALL, RESULT_COMPRESSED, RAW_REPLY]
assert (
extract_cache_stable_delta(
[edited_user, CALL, RESULT_ORIGINAL, CLIENT_REPLY, NEXT],
previous_original,
previous_forwarded,
)
is None
)
@pytest.mark.skipif(not hasattr(os, "fchmod"), reason="POSIX file modes only")
def test_debug_dump_is_owner_only_even_if_it_exists(tmp_path, monkeypatch):
from headroom.proxy.handlers.anthropic import _dump_prefix_mismatch
monkeypatch.setenv("HEADROOM_DEBUG_PREFIX_MISMATCH", str(tmp_path))
# A dump left world-readable by an earlier run: created under the common
# 022 umask, so it gets the default 0644 without any explicit chmod.
existing = tmp_path / "req1.json"
previous_umask = os.umask(0o022)
try:
existing.write_text("{}")
finally:
os.umask(previous_umask)
assert stat.S_IMODE(existing.stat().st_mode) == 0o644
_dump_prefix_mismatch("req1", [USER, CLIENT_REPLY], [USER, RAW_REPLY])
_dump_prefix_mismatch("req2", [USER, CLIENT_REPLY], [USER, RAW_REPLY])
for name in ("req1.json", "req2.json"):
assert stat.S_IMODE((tmp_path / name).stat().st_mode) == 0o600, name
def test_debug_dump_closes_its_descriptor_when_chmod_fails(tmp_path, monkeypatch):
# A long-running proxy must not leak a descriptor per failed debug dump.
from headroom.proxy.handlers import anthropic as handler
monkeypatch.setenv("HEADROOM_DEBUG_PREFIX_MISMATCH", str(tmp_path))
opened = []
real_fdopen = os.fdopen
def tracking_fdopen(fd, *args, **kwargs):
fh = real_fdopen(fd, *args, **kwargs)
opened.append(fh)
return fh
def failing_fchmod(fd, mode):
raise PermissionError("not permitted")
monkeypatch.setattr(handler.os, "fdopen", tracking_fdopen)
monkeypatch.setattr(handler.os, "fchmod", failing_fchmod, raising=False)
handler._dump_prefix_mismatch("req1", [USER, CLIENT_REPLY], [USER, RAW_REPLY])
assert len(opened) == 1
assert opened[0].closed