Commit Graph
3 Commits
Author SHA1 Message Date
205668d953 fix(read-maturation): keep the client's tail cache breakpoint on held Reads (#3977)
## Description

Closes #3997

With `read_maturation` on, every held Read drops Claude Code's tail
cache breakpoint, so the rest of the conversation is re-sent uncached on
every held turn. On one machine on 2026-10-05 (wheel 0.39.0), 37 of
1,353 requests were in 8 holds. One 13-request hold kept `cache_read`
stuck at 176,693 with `cache_write` 0, while the uncached tail grew from
3,879 to 14,639 tokens per request. Over its last 10 requests that hold
sent 83.8k tokens uncached. With the tail cached, the same requests cost
about 36k token-equivalents.

Cause: when `ReadMaturationManager.apply` returns `holding_msg_indices`,
the handler calls `relocate_cache_breakpoint`. That strips every
`cache_control` from the earliest held Read to the end of the
conversation and puts one marker on the message before the Read. Claude
Code marks the last two messages, so its tail marker is always dropped.
The proxy already logs this on every held turn as
`event=cache_breakpoints ... dropped=true tail_grew=true`. Nothing after
the held Read is cache-written until the Read matures, which takes at
least `quiesce_turns` (5) turns and up to `max_hold_turns` (25). The
hold saves one cache write of the Read (1.25x for a 5m TTL, 2x for 1h,
on a few thousand tokens). It pays for that with a full-price resend of
the Read and everything after it on every held turn.

This PR removes the relocation and forwards the client's breakpoints
unchanged, so the held Read is cached with its turn. On the next
request, the provider-confirmed prefix (`frozen_message_count`) covers
the Read, and the existing `i < frozen_message_count` gate in
`ReadMaturationManager.apply` leaves it verbatim. That is intended.
Maturing a Read that is already cached rewrites everything after it at
the cache-write rate, and a week of this machine's traffic, replayed,
shows that costs far more than it saves:

- Median rewrite: 19.2k tokens at the 1h write price (2x).
- What maturing saves: a Read of about 2.1k tokens, read at 0.1x over a
median of 36 remaining turns.
- Net: maturing every Read that is already cached comes to about -23.7M
base-input-token equivalents per week. Not maturing them comes to 0.

The result: maturing a Read that is already cached becomes rare, and
tokens saved from `read_maturation` drop by design. A Read still matures
when it is outside the confirmed prefix once its file goes quiet, for
example on a turn where the frozen count is reset to 0 or where the
client put no breakpoint after the Read.

Trade-offs:

- A session that ends right after the Read pays the cache-write premium
on the tail once, where the old code sent it uncached. Break-even is
about 2-3 held requests, and holds last at least 5 turns.
- The frozen count the gate uses is the handler's clamped one:
`prepare_turn` takes min(provider-confirmed, locally replayable). If no
tool output in the session has been compressed since the Read, the clamp
can hold the count at the Read even though the provider has cached it. A
Read can then still mature inside the provider-cached prefix, which is a
rewrite. Sessions with compressible tool output release the clamp on the
next request (see the probe below). Using the unclamped count for this
gate would be a one-line follow-up. I left it out to keep this PR to the
breakpoint fix.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/read_maturation.py`: delete
`relocate_cache_breakpoint`. The handler was its only production caller.
Update the module docstring: the timeline, invariant 1 (now "the frozen
prefix is never mutated"), and why a cached Read stays verbatim. The
manager logic is unchanged.
- `headroom/proxy/handlers/anthropic.py`: forward `maturation.messages`
as returned, with no relocation.
- `headroom/config.py`, `headroom/proxy/models.py`,
`headroom/cli/proxy.py`, `headroom/rollout.py`,
`docs/content/docs/proxy.mdx`: these descriptions said held Reads are
kept out of the prefix cache, which is no longer true. They now say
Reads stay verbatim and compress once quiet unless the provider cache
already holds them.
- `tests/test_read_maturation_handler_nobust.py`:
`test_verbatim_read_never_cache_written_before_maturation` asserted the
old invariant: the verbatim Read is never forwarded inside the cached
prefix. That invariant is exactly the bug, so the test is now
`test_held_read_keeps_client_breakpoint_then_matures`. It asserts that
the client's breakpoint is forwarded where the client put it on every
turn. It still checks that the Read is held and then matures in order:
in this fixture nothing compresses, so the frozen count stays at the
Read. A comment in the test explains the change.
- `tests/test_read_maturation_handler_nobust.py`: new
`test_read_cached_with_its_turn_stays_verbatim`. It drives the real
handler for 7 turns with compressible Bash output around the Read, and
asserts three things: the breakpoints are kept on every turn, the frozen
count passes the held Read from the second request on, and the Read is
never rewritten.
- `tests/test_read_maturation.py`: new
`test_held_read_matures_only_outside_frozen_prefix`. A held Read matures
at quiesce when it is in the live zone and stays verbatim when the
frozen prefix covers it. The test for a client breakpoint on a fresh
Read now asserts the breakpoint stays put and survives maturation.
`TestBreakpointRelocation` is removed along with the helper.
- `tests/test_cache_ttl_preserved.py`: drop the two relocation TTL
tests. The tool-search deferral TTL tests stay.
- `tests/test_live/test_live_maturation.py`: request A asserts the held
Read is cache-written with the client's tail marker. Request B, with the
Read inside the confirmed prefix, asserts it is not matured and that A's
cached prefix is read back.

## Testing

- [x] Unit tests pass (`pytest`) - the read-maturation, breakpoint and
rollout files below, not the whole suite
- [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff
format --check`
- [x] Type checking passes (`mypy headroom`) - on the six changed
modules
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New and updated tests with `headroom/` at `main` (befdb52f):

```text
$ uv run --frozen --extra dev pytest tests/test_read_maturation.py tests/test_read_maturation_handler_nobust.py tests/test_cache_ttl_preserved.py -q
E   AssertionError: client cache breakpoint moved or dropped on turn(s) [0, 1, 2, 3, 4]
E   AssertionError: client cache breakpoint moved or dropped on turn(s) [0, 1, 2, 3, 4]
FAILED tests/test_read_maturation_handler_nobust.py::test_held_read_keeps_client_breakpoint_then_matures
FAILED tests/test_read_maturation_handler_nobust.py::test_read_cached_with_its_turn_stays_verbatim
========================= 2 failed, 33 passed in 9.68s =========================
```

With this change:

```text
$ uv run --frozen --extra dev pytest tests/gateway/test_cache_breakpoint_budget.py tests/test_cache_breakpoint_diagnostics.py \
    tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_rollout.py \
    tests/test_testing_harness.py tests/test_read_maturation.py tests/test_read_maturation_handler_nobust.py \
    tests/test_cache_ttl_preserved.py tests/test_live/test_live_maturation.py -q
======================== 170 passed, 2 skipped in 9.11s ========================

$ uv run --frozen ruff check <10 changed .py files>
All checks passed!
$ uv run --frozen ruff format --check <10 changed .py files>
10 files already formatted
$ uv run --frozen --extra dev mypy headroom/transforms/read_maturation.py headroom/proxy/handlers/anthropic.py \
    headroom/config.py headroom/cli/proxy.py headroom/proxy/models.py headroom/rollout.py --ignore-missing-imports
Success: no issues found in 6 source files
```

## Real Behavior Proof

- Environment: macOS 27 (arm64), Python 3.11.13, `uv run --frozen
--extra dev`, this branch and `main` (befdb52f). Live evidence comes
from a Claude Code session behind a 0.39.0 proxy with
`HEADROOM_READ_MATURATION=1` on the beta channel.
- Exact command / steps: a probe script drives
`create_app(ProxyConfig(read_maturation=True, mode="token",
cache_enabled=True))` through `TestClient` for 8 turns of one Claude
Code-shaped conversation. The conversation has two 1h system
breakpoints, client breakpoints on the last block of the last two
messages, compressible Bash JSON output, a 3 KB line-numbered Read at
message 10, and then quiet turns. The upstream mock reports cache reads
up to the last forwarded breakpoint, using the prefix tracker's own
estimator. The probe prints `count_cache_breakpoints` for each request
in and out, the frozen count maturation saw, and the Read's forwarded
form.
- Observed result: on `main`, every held turn forwards 3 of the client's
4 breakpoints, with the tail breakpoint 1, 3, 5, 7, 9 messages from the
end. The frozen count never passes the Read (10), and the Read matures
on turn 5. On this branch, all 4 breakpoints reach the tail on every
turn, the frozen count passes the Read from the second request on (11,
13, 15, ...), and the Read stays verbatim. In production, the same hold
logged `event=cache_breakpoints ... dropped=true tail_grew=true` on
every held request, with the cost figures in the Description.

```text
main (befdb52f):
INFO turn 0: in_total=4 out_total=3 in_last_tail=0 out_last_tail=1 maturation_frozen=0 read=verbatim
INFO turn 1: in_total=4 out_total=3 in_last_tail=0 out_last_tail=3 maturation_frozen=10 read=verbatim
INFO turn 2: in_total=4 out_total=3 in_last_tail=0 out_last_tail=5 maturation_frozen=10 read=verbatim
INFO turn 3: in_total=4 out_total=3 in_last_tail=0 out_last_tail=7 maturation_frozen=10 read=verbatim
INFO turn 4: in_total=4 out_total=3 in_last_tail=0 out_last_tail=9 maturation_frozen=10 read=verbatim
INFO turn 5: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=10 read=matured
INFO turn 6: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=21 read=matured
INFO turn 7: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=23 read=matured

this branch:
INFO turn 0: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=0 read=verbatim
INFO turn 1: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=11 read=verbatim
INFO turn 2: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=13 read=verbatim
INFO turn 3: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=15 read=verbatim
INFO turn 4: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=17 read=verbatim
INFO turn 5: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=19 read=verbatim
INFO turn 6: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=21 read=verbatim
INFO turn 7: in_total=4 out_total=4 in_last_tail=0 out_last_tail=0 maturation_frozen=23 read=verbatim
```

- Not tested: `tests/test_live/test_live_maturation.py` against the real
API (no `ANTHROPIC_API_KEY` here; it was updated and skips). This exact
branch has not run against live traffic. The live figures above come
from the 0.39.0 wheel. The full `pytest` suite was not run, only the
files listed under Testing.

## Runtime Rollout Safety

- Rollout-managed feature(s): read_maturation (behavior change inside
the existing feature; no new feature).
- Minimum rollout channel: Beta (unchanged - inherits the feature's
existing gate).
- Stable/default behavior changed: No. read_maturation is beta-gated and
off by default, and none of this code runs when it is off. With it on,
held Reads now keep the client's cache breakpoints and are cached with
their turn, so a Read the provider cache already covers is no longer
matured.
- Kill switch / disable path: disabling read_maturation
(`HEADROOM_READ_MATURATION=0`, or drop `--read-maturation`) disables
this with it; the old breakpoint relocation has no separate switch
because it only added cost.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: revert the commit; no persisted format or config
surface changes, and matured-marker session state is unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- `relocate_cache_breakpoint` was public in
`headroom.transforms.read_maturation` but not re-exported. A repo-wide
grep found no caller outside the handler and tests.
- `count_cache_breakpoints` and the `event=cache_breakpoints` log line
already flagged every held turn as `dropped=true`. With this change
those turns log `dropped=false`.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3977"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 20:57:44 -05:00
Tejas Chopra 723b80c091 feat(read-maturation): activity-based hold-back Read maturation (Mechanism B) (#1068)
## What

Splits **read maturation (Mechanism B)** out of #818 into its own PR, so
#818 can stay focused on the other compression knobs + the SQLite CCR
store. Audit-reads (traffic audits) stays in #818.

Read maturation holds fresh large `Read` outputs **out of the provider
prefix cache** while their file is still active, keeps them verbatim the
whole time the model is working with them, and **matures** them into a
CCR-backed marker once the file has been quiet for `quiesce_turns`. Only
the final compressed form ever enters the cache, so **no cached byte is
ever mutated — there is nothing to bust.**

Activity-based rather than a fixed hold window: the `audit-reads`
simulation showed next-touch gaps are fat-tailed (p50 = 4 turns, p90 =
81), so no fixed window covers the tail while a quiesce rule covers the
activity cluster and lets the tail self-heal via partial-range re-reads.

**Default OFF** — experimental, flag/env gated, validated in pilots
first.

## Changes

- `config.ReadMaturationConfig` (`enabled=False`, `quiesce_turns=5`,
`max_hold_turns=25`, `min_size_bytes=2048`)
- `ProxyConfig` fields mirroring the above
- `ReadMaturationManager` transform + `relocate_cache_breakpoint`
(`headroom/transforms/read_maturation.py`)
- Session-scoped manager rides on `PrefixCacheTracker` — shares the
session's cache affinity and TTL cleanup
- Handler wiring in `anthropic.py`: runs **after** compression (so
`read_lifecycle` markers are respected) and **before** body assembly;
advisory — never fails the request
- CLI flags + env vars (`--read-maturation*` /
`HEADROOM_READ_MATURATION*`)
- `_proxy_config_from_env` wiring for both the multi-worker and CLI
server paths

## Bug fix included

The `--read-maturation` flag was missing
`envvar="HEADROOM_READ_MATURATION"`, so the env var was silently ignored
on the CLI path (only the multi-worker `_proxy_config_from_env` path
read it). Fixed here.

## Tests

- `tests/test_read_maturation.py` — unit
- `tests/test_read_maturation_handler_nobust.py` — handler never busts
cache
- `tests/test_live/test_live_maturation.py` — live harness

`25 passed` locally; ruff check/format clean; mypy clean (only
pre-existing `annotation-unchecked` notes).
2026-06-22 22:52:42 -05:00
Tejas Chopra b7be3814f1 feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description

A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### 1. Rust compressor extraction

- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.

### 2. CCR store hardening

- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).

### 3. Traffic audit tooling (measure before tuning)

- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.

### 4. Read maturation (Mechanism B) — experimental, default OFF

- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.

### 5. Rebase / CI fixups (this update)

- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s

$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed

$ mypy headroom/
Success: no issues found in 365 source files

$ python -m compileall headroom/ -q
COMPILE-OK

# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
#   "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```

## Real Behavior Proof

- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
7cb0f43b); GitHub Actions CI run 27488990477 for the test shards
- Exact command / steps: rebased onto latest main (clean, 13 commits
replayed, 0 conflicts); ran the pytest suites and mypy above locally;
inspected CI shard logs to confirm the failure was the codecov upload,
not the test phase
- Observed result: 253 targeted tests pass locally; mypy clean on 365
files; CI test phase reports `1528 passed, 120 skipped`; the only red
step (codecov `upload-coverage` → "Token required because branch is
protected") is resolved by the rebased-in #968 CODECOV_TOKEN fix
- Not tested: the read-maturation live-API no-bust validation
(`tests/test_live/`) was not re-run in this rebase pass (requires
provider keys); it was validated when the feature first landed, and no
maturation code changed in the rebase — only CCR-default test assertions
and the duplicate-field resolution

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

CHANGELOG is generated by release-please from the conventional commits,
so the CHANGELOG box is intentionally left unchecked. "Manual testing
performed" is unchecked deliberately — see `Real Behavior Proof` → `Not
tested` for the exact boundary (the live-API maturation validation was
not re-run in this rebase pass).

### Follow-ups (tracked, not in this PR)

- Mechanism B provider extensions: OpenAI-family wiring (no breakpoint
hold — bounded near-tail bust) and the Codex runtime read-detector (the
audit classifier is the prototype).
- Pilot enablement playbook: run `audit-reads --simulate-maturation` on
target traffic → pick `quiesce_turns` → enable via env → watch cache hit
rate + `read_maturation:N` transform tags.
2026-06-16 20:21:13 -07:00