mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
205668d953 |
fix(read-maturation): keep the client's tail cache breakpoint on held Reads (#3977)
## Description Closes #3997 With `read_maturation` on, every held Read drops Claude Code's tail cache breakpoint, so the rest of the conversation is re-sent uncached on every held turn. On one machine on 2026-10-05 (wheel 0.39.0), 37 of 1,353 requests were in 8 holds. One 13-request hold kept `cache_read` stuck at 176,693 with `cache_write` 0, while the uncached tail grew from 3,879 to 14,639 tokens per request. Over its last 10 requests that hold sent 83.8k tokens uncached. With the tail cached, the same requests cost about 36k token-equivalents. Cause: when `ReadMaturationManager.apply` returns `holding_msg_indices`, the handler calls `relocate_cache_breakpoint`. That strips every `cache_control` from the earliest held Read to the end of the conversation and puts one marker on the message before the Read. Claude Code marks the last two messages, so its tail marker is always dropped. The proxy already logs this on every held turn as `event=cache_breakpoints ... dropped=true tail_grew=true`. Nothing after the held Read is cache-written until the Read matures, which takes at least `quiesce_turns` (5) turns and up to `max_hold_turns` (25). The hold saves one cache write of the Read (1.25x for a 5m TTL, 2x for 1h, on a few thousand tokens). It pays for that with a full-price resend of the Read and everything after it on every held turn. This PR removes the relocation and forwards the client's breakpoints unchanged, so the held Read is cached with its turn. On the next request, the provider-confirmed prefix (`frozen_message_count`) covers the Read, and the existing `i < frozen_message_count` gate in `ReadMaturationManager.apply` leaves it verbatim. That is intended. Maturing a Read that is already cached rewrites everything after it at the cache-write rate, and a week of this machine's traffic, replayed, shows that costs far more than it saves: - Median rewrite: 19.2k tokens at the 1h write price (2x). - What maturing saves: a Read of about 2.1k tokens, read at 0.1x over a median of 36 remaining turns. - Net: maturing every Read that is already cached comes to about -23.7M base-input-token equivalents per week. Not maturing them comes to 0. The result: maturing a Read that is already cached becomes rare, and tokens saved from `read_maturation` drop by design. A Read still matures when it is outside the confirmed prefix once its file goes quiet, for example on a turn where the frozen count is reset to 0 or where the client put no breakpoint after the Read. Trade-offs: - A session that ends right after the Read pays the cache-write premium on the tail once, where the old code sent it uncached. Break-even is about 2-3 held requests, and holds last at least 5 turns. - The frozen count the gate uses is the handler's clamped one: `prepare_turn` takes min(provider-confirmed, locally replayable). If no tool output in the session has been compressed since the Read, the clamp can hold the count at the Read even though the provider has cached it. A Read can then still mature inside the provider-cached prefix, which is a rewrite. Sessions with compressible tool output release the clamp on the next request (see the probe below). Using the unclamped count for this gate would be a one-line follow-up. I left it out to keep this PR to the breakpoint fix. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/read_maturation.py`: delete `relocate_cache_breakpoint`. The handler was its only production caller. Update the module docstring: the timeline, invariant 1 (now "the frozen prefix is never mutated"), and why a cached Read stays verbatim. The manager logic is unchanged. - `headroom/proxy/handlers/anthropic.py`: forward `maturation.messages` as returned, with no relocation. - `headroom/config.py`, `headroom/proxy/models.py`, `headroom/cli/proxy.py`, `headroom/rollout.py`, `docs/content/docs/proxy.mdx`: these descriptions said held Reads are kept out of the prefix cache, which is no longer true. They now say Reads stay verbatim and compress once quiet unless the provider cache already holds them. - `tests/test_read_maturation_handler_nobust.py`: `test_verbatim_read_never_cache_written_before_maturation` asserted the old invariant: the verbatim Read is never forwarded inside the cached prefix. That invariant is exactly the bug, so the test is now `test_held_read_keeps_client_breakpoint_then_matures`. It asserts that the client's breakpoint is forwarded where the client put it on every turn. It still checks that the Read is held and then matures in order: in this fixture nothing compresses, so the frozen count stays at the Read. A comment in the test explains the change. - `tests/test_read_maturation_handler_nobust.py`: new `test_read_cached_with_its_turn_stays_verbatim`. It drives the real handler for 7 turns with compressible Bash output around the Read, and asserts three things: the breakpoints are kept on every turn, the frozen count passes the held Read from the second request on, and the Read is never rewritten. - `tests/test_read_maturation.py`: new `test_held_read_matures_only_outside_frozen_prefix`. A held Read matures at quiesce when it is in the live zone and stays verbatim when the frozen prefix covers it. The test for a client breakpoint on a fresh Read now asserts the breakpoint stays put and survives maturation. `TestBreakpointRelocation` is removed along with the helper. - `tests/test_cache_ttl_preserved.py`: drop the two relocation TTL tests. The tool-search deferral TTL tests stay. - `tests/test_live/test_live_maturation.py`: request A asserts the held Read is cache-written with the client's tail marker. Request B, with the Read inside the confirmed prefix, asserts it is not matured and that A's cached prefix is read back. ## Testing - [x] Unit tests pass (`pytest`) - the read-maturation, breakpoint and rollout files below, not the whole suite - [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff format --check` - [x] Type checking passes (`mypy headroom`) - on the six changed modules - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New and updated tests with `headroom/` at `main` ( |
||
|
|
723b80c091 |
feat(read-maturation): activity-based hold-back Read maturation (Mechanism B) (#1068)
## What Splits **read maturation (Mechanism B)** out of #818 into its own PR, so #818 can stay focused on the other compression knobs + the SQLite CCR store. Audit-reads (traffic audits) stays in #818. Read maturation holds fresh large `Read` outputs **out of the provider prefix cache** while their file is still active, keeps them verbatim the whole time the model is working with them, and **matures** them into a CCR-backed marker once the file has been quiet for `quiesce_turns`. Only the final compressed form ever enters the cache, so **no cached byte is ever mutated — there is nothing to bust.** Activity-based rather than a fixed hold window: the `audit-reads` simulation showed next-touch gaps are fat-tailed (p50 = 4 turns, p90 = 81), so no fixed window covers the tail while a quiesce rule covers the activity cluster and lets the tail self-heal via partial-range re-reads. **Default OFF** — experimental, flag/env gated, validated in pilots first. ## Changes - `config.ReadMaturationConfig` (`enabled=False`, `quiesce_turns=5`, `max_hold_turns=25`, `min_size_bytes=2048`) - `ProxyConfig` fields mirroring the above - `ReadMaturationManager` transform + `relocate_cache_breakpoint` (`headroom/transforms/read_maturation.py`) - Session-scoped manager rides on `PrefixCacheTracker` — shares the session's cache affinity and TTL cleanup - Handler wiring in `anthropic.py`: runs **after** compression (so `read_lifecycle` markers are respected) and **before** body assembly; advisory — never fails the request - CLI flags + env vars (`--read-maturation*` / `HEADROOM_READ_MATURATION*`) - `_proxy_config_from_env` wiring for both the multi-worker and CLI server paths ## Bug fix included The `--read-maturation` flag was missing `envvar="HEADROOM_READ_MATURATION"`, so the env var was silently ignored on the CLI path (only the multi-worker `_proxy_config_from_env` path read it). Fixed here. ## Tests - `tests/test_read_maturation.py` — unit - `tests/test_read_maturation_handler_nobust.py` — handler never busts cache - `tests/test_live/test_live_maturation.py` — live harness `25 passed` locally; ruff check/format clean; mypy clean (only pre-existing `annotation-unchecked` notes). |
||
|
|
b7be3814f1 |
feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description
A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### 1. Rust compressor extraction
- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.
### 2. CCR store hardening
- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).
### 3. Traffic audit tooling (measure before tuning)
- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.
### 4. Read maturation (Mechanism B) — experimental, default OFF
- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.
### 5. Rebase / CI fixups (this update)
- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s
$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed
$ mypy headroom/
Success: no issues found in 365 source files
$ python -m compileall headroom/ -q
COMPILE-OK
# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
# "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```
## Real Behavior Proof
- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
|