fix(proxy): keep the newest user message verbatim in cache-mode delta compression (#3923)

## Description

**What users see:** with the default settings (`cache` mode plus the
`coding` savings profile), the proxy can rewrite the newest user message
of a conversation from the second request onward. #3619 made the rule
that the user's own prompt reaches the model verbatim, but one code path
missed it. For a Claude Code agent team this is the message a teammate
receives from the lead (`<teammate-message>…`), which is one way a
teammate can end up with a message it calls garbage (#1174).

**Why it happens:** in cache mode the first request of a conversation is
compressed in full. After that, only the new messages (the "stable
delta") are compressed and spliced onto the prefix that was already
forwarded. Every Anthropic `pipeline.apply(...)` call passes
`prefix_replay_guaranteed=True` except this delta call. Without that
flag the router does not protect the newest user message, and the coding
profile turns user-message compression on.

**Fix:** pass `prefix_replay_guaranteed=True` on the delta call as well.
The flag means "the caller replays last turn's forwarded bytes over this
turn's output", which is exactly what the delta path does. Tool results
in the newest message still compress; only the user's own text is kept
as written.

Refs #1174. This is one of two independent causes found while
investigating that issue; the other is in a separate PR (linked in the
issue). I am not using `Closes` because the reporter has not confirmed
which cause they hit.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: the cache-mode stable-delta
`anthropic_pipeline.apply(...)` call now passes
`prefix_replay_guaranteed=True`, matching the other five Anthropic
calls.
- `tests/test_proxy_anthropic_cache_stability.py`: new
`test_cache_mode_delta_keeps_newest_user_prompt_verbatim` drives a
second-turn request through the real handler and asserts that the delta
call carries the flag. It fails on `main` (`assert None is True`). What
the router does with the flag is already covered by
`test_follow_up_prompt_string_after_assistant_turn_is_verbatim_under_replay`
and its siblings in `tests/test_transforms_content_router.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New test on unmodified main
$ pytest tests/test_proxy_anthropic_cache_stability.py -k delta_keeps_newest -q
E   AssertionError: assert None is True
1 failed

# With the fix, related suites
$ pytest tests/test_proxy_anthropic_cache_stability.py tests/test_transforms_content_router.py \
    tests/test_cache_mode_cold_start.py tests/test_cold_start_fast_pass.py \
    tests/test_anthropic_stage_timings.py tests/test_anthropic_compaction_transforms.py \
    tests/test_claude_session_mode_benchmark.py tests/test_proxy_handler_helpers.py -q -n 6
207 passed, 2 warnings in 24.80s

$ uvx ruff@0.16.8 check . && uvx ruff@0.16.8 format --check .
All checks passed!
1705 files already formatted

$ uvx mypy@1.20.2 headroom --ignore-missing-imports --no-site-packages
headroom\fileperms.py:71: error: Module has no attribute "fchmod"  [attr-defined]
Found 1 error in 1 file (checked 554 source files)
# Windows-only: os.fchmod does not exist on Windows. The file is not touched
# by this PR; CI runs mypy on Linux.
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, this branch vs `upstream/main`
at d5318ac2. Real proxy app (`create_app`) with `mode="cache"`,
`savings_profile="coding"`, the real compression pipeline and a warm
Kompress ONNX model; only the upstream HTTP call is stubbed.
- Exact command / steps: two requests of one conversation. Request 1: a
short spawn prompt. Request 2: the same history plus an assistant reply
plus a new user message, "here is the current task list, pick the first
pending task owned by nobody and claim it", followed by a 60-item
task-list JSON (one item pending). Compare the newest user message the
proxy forwards with the one the client sent.
- Observed result: on `main` the second request goes through the delta
path with `prefix_replay_guaranteed=None`, the router applies
`router:text_block:mixed`, and the user message is forwarded as 2,920
characters instead of 6,563. The JSON is re-encoded into a compact table
(`[60]{blockedBy:json,id:string,...}` followed by CSV-like rows). On
this branch the same request shows `prefix_replay_guaranteed=True`, no
transform on that message, and the message is forwarded byte-for-byte
(6,563 → 6,563). With plain-prose messages `main` also hands the text to
a compressor (`route_counts` shows `cache_miss` + `ratio_too_high`) and
only keeps the original because the result was not small enough; on this
branch the text is never handed to a compressor.
- Not tested: a live Claude Code agent-team run against the real
Anthropic API. Effect on savings for sessions whose follow-up messages
are large pasted documents (those are now kept verbatim on turn 2+, the
same as on turn 1 and in token mode).

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This path is not gated in
`headroom/rollout.py`.
- Minimum rollout channel: Not applicable.
- Stable/default behavior changed: Yes, in cache mode with user-message
compression on (the default coding profile): from the second request on,
the text of the newest user message is no longer compressed. This
matches what the first request and token mode already do.
- Kill switch / disable path: None needed; reverting the one-line change
restores the old behaviour.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR. No state or persisted format changes.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- No documentation change: #3619 already documents the "prompt stays
verbatim on replaying callers" behaviour; this applies it to the path it
missed.
- The newest user message is still compressed when it answers a fenced
shell command in the previous assistant turn (a text harness's tool
output). That exception comes from #3619 and is unchanged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3923"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Parideboy
2026-10-05 10:02:30 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent d99779dae6
commit 613ae92150
2 changed files with 79 additions and 0 deletions
+5
View File
@@ -2207,6 +2207,11 @@ class AnthropicHandlerMixin:
model_limit=context_limit,
context=extract_user_query(compression_input),
frozen_message_count=prefix_n,
# The compressed delta is replayed
# verbatim next turn, so the router
# keeps the newest user prompt intact
# here as on every other path (#1174).
prefix_replay_guaranteed=True,
idle_seconds=idle_seconds,
biases=biases,
protect=protect,
@@ -1287,6 +1287,80 @@ def test_cache_mode_reuses_prior_forwarded_prefix_and_compresses_only_new_suffix
]
def test_cache_mode_delta_keeps_newest_user_prompt_verbatim() -> None:
"""The cache-mode delta call must tell the router the prefix is replayed.
Without ``prefix_replay_guaranteed`` the router treats the newest user
message as compressible text, so under the coding profile (which turns on
user-message compression) a follow-up prompt -- e.g. a Claude Code
teammate's ``<teammate-message>`` -- reached the model lossily rewritten
on every turn after the first (#1174). Every other Anthropic
``pipeline.apply`` call already passes the flag.
"""
captured: dict = {}
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "cache"
proxy.config.image_optimize = False
tracker = _FakePrefixTracker(frozen_count=0)
tracker._last_original_messages = [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
]
tracker._last_forwarded_messages = list(tracker._last_original_messages)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
def _fake_apply(**kwargs):
captured.update(kwargs)
return SimpleNamespace(
messages=list(kwargs["messages"]),
transforms_applied=[],
timing={},
tokens_before=10,
tokens_after=10,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "msg_delta_prompt",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {"input_tokens": 10, "output_tokens": 1},
},
)
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [
{"role": "user", "content": "turn1"},
{"role": "assistant", "content": "turn1-assistant"},
{"role": "user", "content": "<teammate-message>next task</teammate-message>"},
],
},
)
assert response.status_code == 200
assert captured["frozen_message_count"] == 2 # the delta path ran
assert captured.get("prefix_replay_guaranteed") is True
def test_anthropic_handler_splits_prefix_trackers_when_tool_profiles_differ() -> None:
"""The handler must pass its non-message cache affinity into resolution.