Commit Graph
2 Commits
Author SHA1 Message Date
a25623109d fix(proxy): copy cached tool compaction results (hardening) (#4410)
## Description

> **Hardening, preventive.** No request sequence was found where this
changes what reaches the upstream or the client; the fix removes shared
mutable state before a future caller can depend on it. Please treat it
as low priority.

`compact_tools` and `compact_tool_descriptions` cache the compacted
`tools` list at module level and then put that same list into the
payload they return. Callers edit the tools in place afterwards
(`inject_tool_search_deferral` sets `cache_control` on a resident tool,
`helpers.py:4235`), so the edit stays in the cache and shows up in later
requests with the same digest. The returned payload now gets a deep copy
of the cached list.

Fixes #4403

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/tool_schema_compaction.py`: `updated["tools"] =
copy.deepcopy(compacted_tools)` at the four places in `compact_tools`
and `compact_tool_descriptions` (cache hit and cache miss in each). The
cache still stores the original list.
- `tests/test_tool_schema_compaction.py`:
`TestCompactedToolsAreNotShared`, one test per function: edit the
returned tool, then call again twice (a hit) and assert the edit is not
visible.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new tests, source change reverted
FAILED tests/test_tool_schema_compaction.py::TestCompactedToolsAreNotShared::test_compact_tools_edit_does_not_leak_into_later_requests
FAILED tests/test_tool_schema_compaction.py::TestCompactedToolsAreNotShared::test_compact_tool_descriptions_edit_does_not_leak
2 failed, 27 passed

# with the fix
$ pytest tests/test_tool_schema_compaction.py
29 passed

$ pytest tests/test_tool_schema_compaction.py tests/test_openai_chat_tool_desc_compaction.py tests/test_anthropic_compaction_transforms.py
44 passed, 1 warning

$ ruff check headroom/proxy/tool_schema_compaction.py tests/test_tool_schema_compaction.py
All checks passed!
$ ruff format --check headroom/proxy/tool_schema_compaction.py tests/test_tool_schema_compaction.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Ubuntu on WSL2, Python 3.12.15, upstream/main e995eb9f
plus this commit. `headroom/_core.abi3.so` copied (untracked, not
committed) from the official `headroom_ai-0.41.0` wheel because the
checkout has no built Rust core. "Before" = the source file with this
commit's change reverted (`git apply -R`).
- Exact command / steps: (1) start a stub upstream on 127.0.0.1:18870
and the real proxy `HEADROOM_TOOL_DESC_MAX_CHARS=60
ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18870 python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18871 --no-cache
--no-rate-limit`, send the same 14-tool request twice, record what the
stub receives; (2) in the same checkout, call the shipped functions in
the handler's order (`compact_tools`, then
`inject_tool_search_deferral`) and compact the same client request
again.
- Observed result: over HTTP the stub receives the same bodies before
and after (compaction still applies, nothing else changed), because the
proxy does not run tool-search deferral against a custom upstream URL.
In-process, before the fix the second identical request gets a
`cache_control` on `Bash` that the client never sent and shares tool
dicts with the first; after the fix it does not.
- Not tested: the leak through a real Anthropic upstream (tool-search
deferral is skipped for a non-Anthropic base URL, so the HTTP path
cannot show it); concurrent requests; the OpenAI and gateway callers;
`mypy`.

HTTP, before and after (stub log, identical):

```text
client request 1 HTTP 200
client request 2 HTTP 200
STUB tools=15 first_desc='Retrieve original uncompressed content that was compressed t…' cache_control_on=[]
STUB tools=15 first_desc='Retrieve original uncompressed content that was compressed t…' cache_control_on=[]
```

In-process, before (change reverted):

```text
request 1: compacted tools carry cache_control on: ['tool_3']
request 1: after tool-search deferral, tools carry cache_control on: ['Bash']
request 2 (fresh, identical client input): compacted tools carry cache_control on: ['tool_3', 'Bash']
request 1 and 2 share tool dicts: True
```

In-process, after (this commit):

```text
request 1: compacted tools carry cache_control on: ['tool_3']
request 1: after tool-search deferral, tools carry cache_control on: ['Bash']
request 2 (fresh, identical client input): compacted tools carry cache_control on: ['tool_3']
request 1 and 2 share tool dicts: False
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no; returned tool content is equal,
only the object identity changes
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none known
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Only the listed test files were run, not the whole suite. The deep copy
costs one extra copy of the tools list per call on both the hit and miss
path. I did not find a request sequence where the leak changes the body
sent upstream (same digest means the same client markers); the fix
removes the shared mutable state. Produced with AI assistance (Claude
Code) operated by me (HetCreep); the reproduction was run on my machine
on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4410"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:44:16 -07:00
3dd9660d91 feat: 3-layer context compression pipeline (L1+L2+L3) (#1405)
## Description

> **Default behavior is unchanged:** only L1 (annotation-key stripping)
is on by default. L2 (description truncation) and L3 (system-prompt
compression) are **opt-in** via `HEADROOM_TOOL_DESC_MAX_CHARS` and
`HEADROOM_SYSTEM_COMPACT=1` respectively — instruction-level compression
never runs unless an operator explicitly enables it. Verified in
`system_compact.py`: `system_compact_enabled()` returns `False` when the
env var is unset.

Reduces MCP-injected context overhead (~40K tokens / 20% of a 200K
window) through a progressive 3-layer compression pipeline. Each layer
is independently controlled, fail-safe, and additive — operators can
enable L1 only (default) or opt into L2/L3 for deeper savings.

### Layer 1: Tool Schema Annotation Key Stripping (default on)
- Strip JSON Schema annotation keys (`$schema`, `title`, `examples`,
`deprecated`, `default`, `readOnly`, `writeOnly`) from tool definitions
- Normalise whitespace in `description` fields
- Zero risk — removes only non-constraint metadata that models ignore
- ~8% savings on tool schema size

### Layer 2: Tool Description Truncation (opt-in:
`HEADROOM_TOOL_DESC_MAX_CHARS`)
- Truncate verbose tool/parameter descriptions to configurable length
- Preserves first complete sentence (critical for model tool selection)
- Optionally appends second sentence within 1.5× budget
- Hard-truncates with `...` if a single sentence exceeds limit
- Recursively processes nested `description` fields in
`input_schema`/`parameters`
- ~43% savings on description text (estimated ~17K tokens)

### Layer 3: System Prompt CCR Compression (opt-in:
`HEADROOM_SYSTEM_COMPACT`)
- Compress `system[]` content blocks using existing
`ContentRouter.compress()`
- Only compresses blocks exceeding `HEADROOM_SYSTEM_COMPACT_MIN_CHARS`
(default 500)
- Preserves `cache_control` markers and non-text blocks
- Fail-safe: leaves block unchanged if compression fails or doesn't save
size
- ~14.5% savings on system prompt (estimated ~3.5K tokens)

**Combined savings (all 3 layers enabled): ~40K → ~17K tokens (~58%
reduction)**

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/tool_schema_compaction.py` — New shared module: L1
annotation stripping + L2 description truncation with
`strip_annotation_keys()` and `truncate_descriptions()`
- `headroom/proxy/system_compaction.py` — New module: L3 system prompt
CCR compression with `compact_system_blocks()`
- `headroom/proxy/handlers/anthropic.py` — Add L1+L2+L3 call sites
(after tool assembly, before PRE_SEND)
- `headroom/proxy/handlers/openai.py` — Add L1+L2+L3 call sites
(parallel to Anthropic handler)
- `tests/test_tool_schema_compaction.py` — 42 unit tests covering edge
cases, nested schemas, fail-safe behavior
- `tests/test_system_compaction.py` — Tests for L3 compression,
cache_control preservation, min-chars gating

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_tool_schema_compaction.py tests/test_system_compaction.py tests/test_anthropic_compaction_transforms.py -v
===== 49 passed in 4.96s =====

$ uv run ruff check <changed files>
All checks passed!

$ uv run mypy headroom/proxy/tool_schema_compaction.py headroom/proxy/system_compaction.py headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py
Success: no issues found in 4 source files

# Manual verification with HEADROOM_TOOL_DESC_MAX_CHARS=120
# Single tool schema: 548→434 bytes (L1, 20.8% saved) → 315 bytes (L2, 27.4% saved)
# Combined: 548→315, 42.5% saved
# Full request with proxy: orig=39179 opt=31988 saved=7191 (18.4% compression)
```

## Real Behavior Proof

- Environment: macOS, Python 3.13, headroom proxy v0.28.0, Claude Code
CLI
- Exact command / steps:
1. Start proxy with `HEADROOM_TOOL_DESC_MAX_CHARS=120
HEADROOM_SYSTEM_COMPACT=1 headroom proxy`
  2. Route Claude Code traffic through proxy
  3. Check `/stats` endpoint for `transforms_applied` and byte savings
- Observed result: L1/L2/L3 transforms applied correctly, ~58% token
reduction on MCP-heavy context
- Not tested: Windows, production deployment

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

- L2 and L3 are **opt-in** via env vars. Default behavior is unchanged
(only L1 active).
- All layers have fail-safe fallbacks — if compaction fails or doesn't
reduce size, the original payload passes through unchanged.
- The Anthropic handler now appends `anthropic:tool_schema_compaction`
(L1), `anthropic:tool_desc_compaction` (L2), and
`anthropic:system_compact` (L3) to `transforms_applied`, so `/stats` and
transformation accounting are no longer blind to compression that
changed the request. Covered by handler-level e2e regression in
`tests/test_anthropic_compaction_transforms.py` (positive + negative
cases). The earlier follow-up #1423 is superseded — no longer needed.

---------

Signed-off-by: lg320531124 <lg320531124@users.noreply.github.com>
Co-authored-by: lg320531124 <lg320531124@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 19:58:51 +00:00