Files
headroom/tests/test_read_maturation.py
c21b0813bd fix(compress): stop adding markers the caller did not ask for (#4678)
`/v1/compress` returns a body that someone else's client forwards
unchanged, so anything Headroom adds there reaches the provider. Two
things got added that the caller never asked for.

## Stale-Read retrieval hashes in marker-free modes

Read lifecycle replaces a Read that a later edit made stale with a short
notice. That notice always ended in `Retrieve original: hash=…` and
wrote the original to the CCR store, even in the default and
`lossy_inline` modes. The docs call those modes marker-free, and their
callers have no retrieval tool, so the hash pointed nowhere.

- Marker-free modes keep the notice ("re-read the file for current
content") but drop the hash and skip the store write.
- Lossless routers (proxy `--lossless`,
`ContentRouterConfig(lossless=True)`) skip read lifecycle entirely,
because the notice drops the file's bytes.
- Read maturation now recognises the hashless notice. Otherwise it would
replace the notice with a marker whose hash points at the notice itself.

## Cache breakpoints the client didn't send

On the session path, `normalize_message_cache_control` fell back to
placing `{"type": "ephemeral"}` on the last block message when the
client marked nothing. A caller that chose not to cache got billed cache
writes. `/v1/compress` now passes `place_unmarked=False`: if the client
marked no message, markers are stripped and none are placed. Proxy
handlers keep today's behaviour, and client markers in a misaligned list
still go through the old path.

## Tests

Each new test fails on main.

- `test_marker_free_modes_mark_a_stale_read_without_a_retrieval_hash`,
for the default mode and `lossy_inline`
- `test_lossless_router_keeps_a_stale_read`
- `test_session_turns_add_no_marker_the_caller_did_not_send`
- `test_respects_lifecycle_markers`, extended to the hashless notice

The related suites (gateway, transforms, read maturation, prefix
tracker, cache_control, compress endpoint and session) pass: 1371 passed
and 9 skipped before the review fixes, then 222 of 222 on the touched
files after them. Lint is clean on CI's ruff 0.17.0, and mypy is clean
on the touched modules.

## Not in this PR

- Read maturation (`handlers/anthropic.py`) is not gated on `--lossless`
or `--no-ccr` either. It's the same kind of bug on the proxy path, so it
belongs in a separate fix.
- The "lossless densify only" pipeline in `compress.py` sets
`ccr_enabled=False` without `lossless=True`, so read lifecycle still
runs there.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4678"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Real Behavior Proof

Maintainer qualification on Windows, Python 3.12.13, using an actual
Uvicorn server bound to an owned loopback socket and HTTPX over real
HTTP. The normal `/v1/compress` endpoint and compression pipeline ran
with offline mode and Kompress disabled; no handler, compression stage,
HTTP response or store was mocked. Workspace, configuration, savings and
TOIN paths were isolated. All 243 native inputs and the reused artifact
SHA-256
`5be63a4f31d3e38e7427d28380ecb977616e3f6a3dd6a1c59af5ff6361a05978` were
verified; this was not a fresh native build.

- Exact command / steps: the committed probe is `python -m pytest -q -s
tests/test_compress_runtime_http.py`. The qualifying run used the
maintainer loopback-only launcher with an explicit verified native path
and SHA-256, followed by the four focused
session/API/maturation/lifecycle suites. All 138 tests passed. The probe
starts the actual server, GETs `/health`, and sends nine POSTs to
`/v1/compress`: a Read followed by Edit in default, lossy-inline and
lossless modes, then three session turns without caller cache markers
and three turns with a caller marker. It stops and joins the server in
`finally`.
- Observed result: health and all nine compression requests returned
HTTP 200. Default and lossy-inline emitted stale notices with no
retrieval hash or CCR hashes, saving 3,576 tokens per request. Lossless
preserved the original Read bytes. All three unmarked session turns
returned zero cache markers; all three marked controls retained exactly
one. Session compression saved 7,350 / 15,901 / 25,958 tokens. Server
shutdown joined successfully.
- Before-fix control: the same real-server requests with all six
affected production modules replaced by current main
`200c838c7393d20ae56a0c338f832ed27133c398` failed five assertions:
default and lossy-inline included retrieval hashes, and each unmarked
session turn added a cache marker. Lossless and the three marked
controls passed on main and are not claimed as reproduced regressions.
Candidate source was restored byte-for-byte.
- Not tested: provider forwarding or billing, a live model, deployment
behavior, or the separate proxy read-maturation scope already listed
above. External startup connection attempts were blocked by the
qualifying launcher; this is not a zero-attempt network claim. No author
readiness or self-review declaration is added by this maintainer
evidence.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-13 09:39:10 -06:00

399 lines
15 KiB
Python

"""Tests for Mechanism B: activity-based hold-back Read maturation.
The invariants under test, beyond decision behavior:
1. The provider-cached (frozen) prefix is never mutated, and a client
cache_control breakpoint is never moved or dropped.
2. Replay is deterministic — once matured, the same marker is applied on
every subsequent request, byte-identical.
3. Holding is derived from the conversation (file activity), so the
decision survives state loss; only matured markers are stateful.
"""
from __future__ import annotations
import pytest
from headroom.config import ReadMaturationConfig
from headroom.transforms.read_maturation import ReadMaturationManager
CONTENT = " 1\tdef foo():\n 2\t return 42\n" * 60 # > 2048B
SMALL = " 1\tok\n"
def anthropic_read(tc_id: str, file_path: str, content: str) -> list[dict]:
return [
{
"role": "assistant",
"content": [
{"type": "tool_use", "id": tc_id, "name": "Read", "input": {"file_path": file_path}}
],
},
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": tc_id, "content": content}],
},
]
def anthropic_edit(tc_id: str, file_path: str) -> list[dict]:
return [
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"id": tc_id,
"name": "Edit",
"input": {"file_path": file_path, "old_string": "a", "new_string": "b"},
}
],
},
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": tc_id, "content": "ok"}],
},
]
def openai_read(tc_id: str, file_path: str, content: str) -> list[dict]:
return [
{
"role": "assistant",
"tool_calls": [
{
"id": tc_id,
"function": {"name": "Read", "arguments": f'{{"file_path": "{file_path}"}}'},
}
],
},
{"role": "tool", "tool_call_id": tc_id, "content": content},
]
def quiet(n: int) -> list[dict]:
"""n assistant turns with no file activity (advances the quiet clock)."""
return [
{"role": "assistant", "content": [{"type": "text", "text": f"thinking {i}"}]}
for i in range(n)
]
def base_conv() -> list[dict]:
return [{"role": "user", "content": "look"}, *anthropic_read("r1", "/x/foo.py", CONTENT)]
def manager(**overrides) -> ReadMaturationManager:
cfg = ReadMaturationConfig(enabled=True, **overrides)
return ReadMaturationManager(cfg)
def read_content(res, idx=2):
return res.messages[idx]["content"][0]["content"]
class TestActivityDecision:
def test_disabled_is_noop(self):
m = ReadMaturationManager(ReadMaturationConfig(enabled=False))
res = m.apply(base_conv())
assert res.messages == base_conv()
assert res.holding_msg_indices == []
def test_fresh_read_holds_verbatim(self):
res = manager().apply(base_conv())
assert read_content(res) == CONTENT
assert res.holding_msg_indices == [2]
assert res.holding_reads == 1
assert res.newly_matured == 0
def test_holds_while_file_quiet_below_quiesce(self):
msgs = [*base_conv(), *quiet(4)] # quiet = 4 < 5
res = manager(quiesce_turns=5).apply(msgs)
assert res.holding_msg_indices == [2]
assert read_content(res) == CONTENT
def test_matures_after_quiesce(self):
msgs = [*base_conv(), *quiet(5)] # quiet = 5 >= 5
res = manager(quiesce_turns=5).apply(msgs)
assert res.newly_matured == 1
assert res.holding_msg_indices == []
marker = read_content(res)
assert "compressed after use" in marker
assert "/x/foo.py" in marker
assert "Retrieve original: hash=" in marker
assert res.bytes_saved > 0
def test_file_activity_resets_quiet_clock(self):
# read, 4 quiet, edit same file, 4 quiet: quiet=4 < 5 → still held
msgs = [*base_conv(), *quiet(4), *anthropic_edit("e1", "/x/foo.py"), *quiet(4)]
res = manager(quiesce_turns=5).apply(msgs)
assert res.holding_msg_indices == [2]
# one more quiet turn → file quiesced → matures
res = manager(quiesce_turns=5).apply([*msgs, *quiet(1)])
assert res.newly_matured == 1
def test_activity_on_other_file_does_not_reset(self):
msgs = [*base_conv(), *quiet(3), *anthropic_edit("e1", "/x/OTHER.py"), *quiet(1)]
# foo.py quiet for 5 assistant turns (3 quiet + edit-turn + 1 quiet)
res = manager(quiesce_turns=5).apply(msgs)
assert res.newly_matured == 1
def test_max_hold_caps_busy_files(self):
# File touched every turn — never quiesces — but the hold cap fires.
msgs = base_conv()
for i in range(6):
msgs += anthropic_edit(f"e{i}", "/x/foo.py")
res = manager(quiesce_turns=100, max_hold_turns=6).apply(msgs)
assert res.newly_matured == 1
assert res.holding_msg_indices == []
def test_replay_is_deterministic_and_stateful(self):
m = manager(quiesce_turns=5)
matured_msgs = [*base_conv(), *quiet(5)]
a = read_content(m.apply(matured_msgs))
# Replay applies even when the conversation grows and the file is
# touched again later (matured is final).
later = [*matured_msgs, *anthropic_edit("e1", "/x/foo.py")]
b = read_content(m.apply(later))
c = read_content(m.apply([*later, *quiet(3)]))
assert a == b == c
def test_small_reads_ignored(self):
msgs = [
{"role": "user", "content": "look"},
*anthropic_read("r1", "/x/a.py", SMALL),
*quiet(10),
]
res = manager().apply(msgs)
assert res.holding_msg_indices == []
assert read_content(res) == SMALL
def test_frozen_prefix_untouched(self):
msgs = [*base_conv(), *quiet(10)]
m = manager(quiesce_turns=5)
res = m.apply(msgs, frozen_message_count=len(msgs))
assert res.holding_msg_indices == []
assert read_content(res) == CONTENT
def test_held_read_matures_only_outside_frozen_prefix(self):
"""A held Read is cached with its turn. At quiesce it matures if it
is still outside the frozen prefix (e.g. a cold prefix); once the
frozen prefix covers it, it stays verbatim — maturing it would
re-write everything after it at the cache-write rate."""
held = base_conv()
msgs = [*held, *quiet(5)]
m = manager(quiesce_turns=5)
assert m.apply(held).holding_msg_indices == [2]
res = m.apply(msgs, frozen_message_count=2) # Read in the live zone
assert res.newly_matured == 1
assert "compressed after use" in read_content(res)
m = manager(quiesce_turns=5)
assert m.apply(held).holding_msg_indices == [2]
res = m.apply(msgs, frozen_message_count=3) # frozen prefix covers it
assert res.newly_matured == 0
assert res.messages is msgs
@pytest.mark.parametrize(
"notice",
[
"[Read content stale: /x/foo.py ... Retrieve original: hash=abc123]",
"[Read content stale: /x/foo.py was modified after this read]",
],
)
def test_respects_lifecycle_markers(self, notice):
marker = notice + " " * 2048
msgs = [
{"role": "user", "content": "look"},
*anthropic_read("r1", "/x/foo.py", marker),
*quiet(10),
]
res = manager().apply(msgs)
assert res.holding_msg_indices == []
assert res.newly_matured == 0
def test_client_breakpoint_on_fresh_read_is_held_and_kept(self):
"""Claude Code parks its tail breakpoint on the newest block —
right after a Read, that's the Read result itself. The read must
still be held (verbatim) and the breakpoint must stay on it, so
the turn is cached; it also survives the block's maturation."""
msgs = base_conv()
msgs[2]["content"][0]["cache_control"] = {"type": "ephemeral"}
m = manager(quiesce_turns=5)
res = m.apply(msgs)
assert res.holding_msg_indices == [2]
assert res.messages[2]["content"][0] == msgs[2]["content"][0]
res = m.apply([*msgs, *quiet(5)])
assert res.newly_matured == 1
assert res.messages[2]["content"][0]["cache_control"] == {"type": "ephemeral"}
def test_openai_format(self):
msgs = [{"role": "user", "content": "look"}, *openai_read("r1", "/x/foo.py", CONTENT)]
m = manager(quiesce_turns=5)
res = m.apply(msgs)
assert res.holding_msg_indices == [2]
res = m.apply([*msgs, *quiet(5)])
assert "compressed after use" in res.messages[2]["content"]
def test_files_mature_independently(self):
# foo.py quiet for ages; bar.py just read → foo matures, bar holds.
msgs = [
*base_conv(),
*quiet(6),
*anthropic_read("r2", "/x/bar.py", CONTENT),
]
res = manager(quiesce_turns=5).apply(msgs)
assert res.newly_matured == 1 # foo.py
assert "compressed after use" in read_content(res, 2)
assert res.holding_msg_indices == [10] # bar.py result message
assert read_content(res, 10) == CONTENT
def test_decision_survives_state_loss(self):
# A fresh manager (proxy restart) makes the same holding/matured
# decision because holding is derived from the conversation.
msgs = [*base_conv(), *quiet(5)]
first = read_content(manager(quiesce_turns=5).apply(msgs))
second = read_content(manager(quiesce_turns=5).apply(msgs))
assert "compressed after use" in first and "compressed after use" in second
# Markers differ only if CCR hashing differed — it must not.
assert first == second
class TestCcrIntegration:
def test_original_stored_and_retrievable(self):
from headroom.cache.backends.memory import InMemoryBackend
from headroom.cache.compression_store import CompressionStore
store = CompressionStore(backend=InMemoryBackend())
m = ReadMaturationManager(
ReadMaturationConfig(enabled=True, quiesce_turns=5), compression_store=store
)
res = m.apply([*base_conv(), *quiet(5)])
marker = read_content(res)
ccr_hash = marker.split("hash=")[1].rstrip("]")
entry = store.retrieve(ccr_hash)
assert entry is not None
assert entry.original_content == CONTENT
assert entry.compression_strategy == "read_maturation"
class TestProxyWiring:
def test_proxy_config_flag_default_off(self):
from headroom.proxy.models import ProxyConfig
assert ProxyConfig().read_maturation is False
assert ProxyConfig(read_maturation=True).read_maturation is True
def test_session_state_rides_on_prefix_tracker(self):
"""The handler's session-state flow: a manager attached to the
tracker carries matured markers across requests."""
from headroom.cache.prefix_tracker import PrefixCacheTracker
from headroom.config import ReadMaturationConfig
from headroom.transforms.read_maturation import ReadMaturationManager
tracker = PrefixCacheTracker("anthropic")
assert tracker.read_maturation_manager is None
# Request 1: fresh read — held.
tracker.read_maturation_manager = ReadMaturationManager(
ReadMaturationConfig(enabled=True, quiesce_turns=5)
)
msgs = base_conv()
msgs[2]["content"][0] = {
**msgs[2]["content"][0],
}
res = tracker.read_maturation_manager.apply(msgs)
assert res.holding_msg_indices == [2]
# Request N (file quiet): same manager matures and replays.
later = [*base_conv(), *quiet(5)]
res = tracker.read_maturation_manager.apply(later)
assert res.newly_matured == 1
replay = tracker.read_maturation_manager.apply(later)
assert replay.replacements_applied == 1
assert replay.newly_matured == 0
if __name__ == "__main__":
pytest.main([__file__, "-v"])
class TestFirstAppearanceAccounting:
"""A matured Read's savings must book once, on the turn it matures.
The client re-sends the raw conversation every turn, so a plain
original-vs-optimized token diff re-counts the same removal on every
later request. ``replayed_token_debt`` measures that from the
request's own endpoints — raw client snapshot vs forwarded messages —
rather than from what ``apply`` replaced, because after maturation
the marker normally reaches the wire through the cached-prefix replay
and ``apply`` replaces nothing at all.
"""
def _matured(self):
"""Returns (manager, raw client messages, forwarded messages)."""
m = manager(quiesce_turns=5)
raw = [*base_conv(), *quiet(5)]
res = m.apply(raw)
assert res.newly_matured == 1
return m, raw, res.messages
@staticmethod
def _marker(sent):
return sent[2]["content"][0]["content"]
def test_first_appearance_books_in_full(self):
m, raw, sent = self._matured()
assert m.replayed_token_debt(raw, sent, len) == 0
def test_later_request_is_charged_even_without_a_replacement(self):
# The second call passes the same forwarded form WITHOUT running
# apply again: that is the cached-prefix replay, the path that
# actually re-books the removal in production.
m, raw, sent = self._matured()
m.replayed_token_debt(raw, sent, len)
expected = len(CONTENT) - len(self._marker(sent))
assert m.replayed_token_debt(raw, sent, len) == expected
def test_marker_echo_is_not_a_replay(self):
# The client echoes the marker form back: nothing was removed this
# request, so there is nothing to un-book.
m, _, sent = self._matured()
assert m.replayed_token_debt(sent, sent, len) == 0
def test_unchanged_wire_form_is_not_a_replay(self):
# Forwarded verbatim (maturation state lost, hold re-established):
# no removal on the wire, no debt.
m, raw, _ = self._matured()
assert m.replayed_token_debt(raw, raw, len) == 0
def test_delta_is_tokenized_once_per_tool_call(self):
m, raw, sent = self._matured()
m.replayed_token_debt(raw, sent, len)
calls = []
def count(text: str) -> int:
calls.append(text)
return len(text)
expected = len(CONTENT) - len(self._marker(sent))
assert m.replayed_token_debt(raw, sent, count) == expected
assert len(calls) == 2 # content + marker, once
assert m.replayed_token_debt(raw, sent, count) == expected
assert len(calls) == 2 # cached for the session
def test_replay_request_nets_zero_savings(self):
# With a compositional counter the diff on a replay request equals
# exactly the debt, so first-appearance accounting books nothing new.
m, raw, sent = self._matured()
m.replayed_token_debt(raw, sent, len)
diff = len(CONTENT) - len(self._marker(sent))
assert max(0, diff - m.replayed_token_debt(raw, sent, len)) == 0
def test_nothing_matured_has_zero_debt(self):
m = manager()
res = m.apply(base_conv())
assert m.replayed_token_debt(base_conv(), res.messages, len) == 0