Commit Graph
40 Commits
Author SHA1 Message Date
6b7b6358d5 fix(benchmarks): refuse a dirty or non-git --worktree in codemode_ab (#4736)
## Description

`benchmarks/codemode_ab.py` runs `git reset --hard -q` and `git clean
-qfd` in `--worktree` after any task that leaves `git status
--porcelain` non-empty (`main()`, lines 319-324 on `1da8930a`). The
module docstring assumes a clean detached worktree, but nothing checks
that `--worktree` is a git work tree or was clean before the first run.
If it holds uncommitted work, that work is deleted after the first task.

With this PR `main()` checks once before the loop: if `--worktree` is
not inside a git work tree, or `git status --porcelain` is not empty, it
prints a message on stderr and returns exit code 2. The per-run reset is
unchanged.

Fixes #4731

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `benchmarks/codemode_ab.py`: after `wt =
Path(args.worktree).resolve()`, run `git status --porcelain` in `wt`. A
non-zero exit gives `error: --worktree <path> is not inside a git work
tree`; non-empty output gives an error naming the reset/clean commands
and suggesting commit, stash or a clean detached worktree. Both return 2
before any `claude` call.
- `tests/test_codemode_ab_worktree_guard.py`: new regression tests for
the dirty, non-git and clean cases (follow-up commit).

## Testing

- [x] Unit tests pass (`pytest`, new file only)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`tests/test_codemode_ab_worktree_guard.py` (new) calls `main()` on a tmp
git repo with `run_one` stubbed, so the `claude` CLI is never reached:
dirty tree (modified tracked file plus untracked file) must return 2
with the files intact; a plain directory must return 2 with the
not-a-git-work-tree message; a clean repo must return 0. Only that file
was run, not the full suite; `ruff format` and `ruff check` on it are
clean; `mypy` was not run. The script was also checked end to end
against a stub `claude` (below).

pytest, Ubuntu on WSL2, Python 3.12, `-p no:cacheprovider`.

Pass after (this branch):

```text
collected 3 items

tests/test_codemode_ab_worktree_guard.py ...                             [100%]

============================== 3 passed in 21.33s ==============================
```

Fail before (same test file, `benchmarks/codemode_ab.py` from
`origin/main` `1da8930a`):

```text
E   assert 0 == 2
E   assert 0 == 2
FAILED tests/test_codemode_ab_worktree_guard.py::test_dirty_worktree_is_refused_and_left_intact
FAILED tests/test_codemode_ab_worktree_guard.py::test_non_git_directory_is_refused
========================= 2 failed, 1 passed in 17.63s =========================
```

On `origin/main` the dirty and non-git cases return 0 instead of 2
because there is no guard (the test stops at that first assertion; the
reset of the dirty tree is shown by the stub run above). The clean case
passes on both, as it should.

### Test Output

Same stub `claude` (prints a fixed JSON result, touches nothing) and
same fixture on both versions of the script: a git repo with one commit,
then ` M tracked.txt` and an untracked `notes.md`. Ubuntu on WSL2,
Python 3, `python3 -I`, `HOME` pointed at an empty dir.

Fail before (`benchmarks/codemode_ab.py` from `origin/main` `1da8930a`):

```text
before: [ M tracked.txt ?? notes.md ]
  ccr_py_count           steered  rep0  $0.0000  turns=1  calls=None  ok=True
    !! worktree dirtied, resetting: M tracked.txt
?? notes.md
after: status=[] tracked=[committed ] notes.md=NO
```

Pass after (this branch):

```text
before: [ M tracked.txt ?? notes.md ]
error: --worktree /var/tmp/cb-u16-ab/wt-branch has uncommitted changes; the script runs `git reset --hard` and `git clean -fd` there after each run. Commit or stash first, or use a clean detached worktree.
exit=2
after: status=[ M tracked.txt ?? notes.md ] tracked=[committed MY EDIT ] notes.md=yes
```

Not a git directory (this branch): `error: --worktree
/var/tmp/cb-u16-ab/nogit is not inside a git work tree`, exit 2.

Clean fixture (this branch), full run with the stub: `exit=0`, last line
`wrote .../out/r.json`, 16 records written.

## Real Behavior Proof

- Environment: Ubuntu on WSL2, Python 3, git, stub `claude` first on
`PATH`, `HOME` redirected to an empty directory. No real `claude` run
and no model call.
- Exact command / steps: `python3 -I -u benchmarks/codemode_ab.py
--worktree <fixture> --plugin-dir <empty dir> --out <tmp>/r.json`, once
per script version and fixture.
- Observed result: on `1da8930a` the dirty fixture ended with an empty
status, `tracked.txt` back to `committed` and `notes.md` deleted; on
this branch the script exited 2 with the message above and both files
were untouched. On a clean fixture this branch still completed all runs
with exit 0.
- Not tested: a real `claude -p` run; a `--worktree` path that does not
exist (the `git` call would raise `FileNotFoundError`); worktrees made
with `git worktree add`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; benchmark script only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no, for the library or proxy. The
script now exits 2 instead of resetting a dirty `--worktree`.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (only the
new test file was run; see Testing)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Known limit: a worktree edited while a benchmark is running is still
reset by the existing per-run cleanup. The check runs once, before the
first run.
- A user who wants to reuse a dirty tree now has to commit or stash
first. I kept the check to the smallest form; no `--force` flag was
added.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
stub-claude runs above were done on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-11 09:58:13 -06:00
Parideboy a385ea8e60 fix(benchmarks): drop the duplicate protect_recent=0 table from the proof harness (#4754)
## Description

`benchmarks/index_proof_table.py` printed two tables, "default
(protect_recent=4)" and "full corpus (protect_recent=0)". The committed
`benchmarks/results/index_proof_table.txt` showed them identical in
every row and in the total, so the second run and its label carried no
information — and the comment above them claimed the default "shields
almost every tool result", which the output contradicts.

They cannot ever differ. `ContentRouter` applies `protect_recent` only
to content detected as `SOURCE_CODE`:

```python
is_code = detection.content_type == ContentType.SOURCE_CODE
if protect_recent > 0 and messages_from_end <= protect_recent and is_code:
```

Every scenario in this harness is a JSON tool payload. All ten payloads
detect as `JSON_ARRAY` under **both** backends (pure-Python and
`HEADROOM_DETECT_BACKEND=rust`), so the guard never fires at any
`protect_recent` value. This is not a stale committed artifact that a
rerun would fix.

Of the three options on the issue, this takes the first: one table with
the limitation stated, rather than a duplicate or a behavior change to
the compressor.

Closes #4732

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- `benchmarks/index_proof_table.py`: print one table (the default
config) and replace the misleading comment with the actual reason the
`protect_recent` comparison is a no-op for JSON payloads.
- `benchmarks/results/index_proof_table.txt`: regenerated. The table
body is unchanged; the duplicate second table is gone.
- `tests/test_index_proof_table.py`: new output-level tests asserting
the harness prints exactly one table and that the docs Markdown block
states the config it was measured under. Both fail on the base revision
and pass with this change.
- `tests/test_transforms_content_router.py`: new
`test_protect_recent_does_not_shield_non_code_tool_payloads` pins the
router reason the two tables could never differ.
- `headroom/compress.py`: `CompressConfig.protect_recent` docstring said
"Don't compress the last N messages"; it guards recent **code** only.
- `docs/content/docs/context-management.mdx`,
`docs/content/docs/proxy.mdx`, `docs/content/docs/index.mdx`,
`README.md`: same overstatement corrected. The published table now names
the configuration it was measured under.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_index_proof_table.py tests/test_transforms_content_router.py -q
109 passed, 1 warning in 14.83s

$ uv run ruff check benchmarks/index_proof_table.py headroom/compress.py \
    tests/test_index_proof_table.py tests/test_transforms_content_router.py
All checks passed!

$ uv run ruff format --check tests/test_index_proof_table.py
1 file already formatted
```

Base-revision check for the new output tests (harness restored to
`318e8e08`):

```text
$ uv run pytest tests/test_index_proof_table.py -q
FAILED tests/test_index_proof_table.py::test_proof_table_prints_exactly_one_table
FAILED tests/test_index_proof_table.py::test_proof_markdown_block_is_the_default_config
2 failed in 5.15s
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, `uv` venv at the branch head
`9b7e2f95` (rebased onto `acb58a0c`); content detection ran on the
pure-Python backend (no ONNX Runtime 1.24+), and every payload was
re-checked under `HEADROOM_DETECT_BACKEND=rust`.
- Exact command / steps: `uv run python benchmarks/index_proof_table.py`
on the base revision, then the same command on this branch, then `git
diff` against the committed `benchmarks/results/index_proof_table.txt`.
- Observed result: base revision printed two tables identical in all
four rows and in the total (178,024 → 104,261, 41%). This branch prints
one table with the same numbers, and its output matches the committed
result file byte for byte once the first line is excluded (that line
names whichever detector backend the host has).
- Not tested: whether the committed numbers would change under the
native detector — also left open on the issue. They do not: both
backends classify all ten payloads as `JSON_ARRAY`, so detection routes
them identically.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. Only a benchmark script, its
committed output, tests, and docs change.
- Minimum rollout channel: stable, no gating.
- Stable/default behavior changed: no. `ContentRouter`,
`CompressConfig`, and `compress()` behavior are untouched; only a
docstring and documentation prose changed. The published table's numbers
are identical to before.
- Kill switch / disable path: not applicable.
`benchmarks/index_proof_table.py` is a manual, offline script with no
runtime surface.
- Unsafe override required: no.
- Qualification impact: none. Serialization, caching, and compression
output are unchanged; the new tests are offline and stubbed.
- Rollback path: revert this commit. The harness returns to printing the
duplicate table; no state or migration is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

`mypy headroom` left unchecked: the touched Python files add no type
surface beyond a benchmark script and a test module, and mypy was not
run locally.

Review feedback addressed: the initial version of this PR added only a
router-level test, which passes on the base revision and so could not
catch the duplicate table returning. The two output-level tests in
`tests/test_index_proof_table.py` replace that gap, and the
base-revision failure above is their proof. The `context-management.mdx`
file-read claim was also too broad — read protection releases
confidently non-code data (JSON, CSV/tabular, build/test logs, git
diffs, HTML, search output), so the wording now says so.

`CHANGELOG.md` is untouched, per CONTRIBUTING.


The branch was rebased onto `acb58a0c` (4 upstream commits: #4727,
#4728, #4735, #4743) with no file overlap. Re-verified after the rebase:
109 focused tests pass, `ruff check`/`format --check` clean.
`benchmarks/index_proof_table.py` is byte-identical between `318e8e08`
and `acb58a0c`, so the base-revision failure above still applies
unchanged.
2026-11-10 12:35:38 -06:00
aeb09f8d18 fix(cache): keep cache mode when Claude Code re-serializes its last reply (#4003)
## Description

This PR fixes two problems that cost Headroom users money on Claude
Code. A live benchmark (`benchmarks/tool_search_vs_native/`) found both.

**1. Cache mode broke when Claude Code re-serialized its last reply.**

Headroom records the model's raw reply as the last message of the
previous turn. Claude Code sends that reply back edited:

- it strips a leading `cd <cwd> &&` from Bash commands;
- it drops `caller`.

The history then read as `relation=diverged`
(`cache_mode_prefix_mismatch`). Cache mode forwarded the raw originals
over every earlier compressed message, and the whole conversation was
written to cache again: 66.8K tokens on every long-task run.

The fix: when only the recorded reply differs, and it is the same turn,
Headroom replays the forwarded prefix and takes the client's version of
the reply. That reply was never part of a cached prefix. "Same turn"
means the same block kinds and the same `tool_use` ids.

**2. Small MCP results were compressed.**

The Jira ticket (a few hundred tokens) lost its description. The model
called `headroom_retrieve` and then re-fetched the ticket, which cost
two extra turns and about +15% on the long task. MCP results under
`HEADROOM_MCP_RESULT_MIN_CHARS` (default 4000) are now forwarded
verbatim. This covers both Anthropic `tool_result` blocks and OpenAI
`role: "tool"` messages.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/cache/prefix_tracker.py`:** adds
`_client_rewrote_reply_delta`, tried on `RELATION_DIVERGED` after
`_replaced_system_tail_delta`. A change earlier in the history still
diverges.
- **`headroom/transforms/content_router.py`:** adds
`_is_small_mcp_result`, applied in `_process_content_blocks` (Anthropic)
and on the `role: "tool"` branch (OpenAI). It recognises the `mcp__`,
`mcp_` and `mcp--` name prefixes. `HEADROOM_MCP_RESULT_MIN_CHARS=0`
restores the old behaviour.
- **`headroom/proxy/handlers/anthropic.py`:** adds an opt-in
`HEADROOM_DEBUG_PREFIX_MISMATCH=<dir>`. It logs the first diverging
message and dumps both histories, with the directory created `0700` and
the file `0600`, because the dump holds the full conversation.
- **`tests/test_cache_mode_delta_marker.py`:**
`test_tool_use_different_input_still_falls_back` becomes
`..._forwards_the_clients_version`, because the client's copy of the
last reply is what gets forwarded and never the recorded one. A new test
keeps the divergence for an edit earlier in the history.
- **New tests:**
  - `tests/test_prefix_tracker_client_rewrote_reply.py`;
- `tests/test_content_router_mcp_small_result.py`, which covers both
wire formats and three MCP prefixes.
- **`benchmarks/tool_search_vs_native/`:** a live A/B harness. It runs
isolated `claude -p` sessions against six fake MCP servers and a fixture
repo, and reads billed usage from Claude Code's transcripts. A run
passes only if Claude Code exits 0, the tests pass, and the four MCP
actions happened (`get_issue` counts as reading the ticket; a search
does not).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_prefix_tracker_client_rewrote_reply.py tests/test_content_router_mcp_small_result.py tests/test_cache_mode_delta_marker.py
============================== 20 passed in 3.42s ==============================

$ pytest -q tests -k "content_router or router or prefix or cache_mode or delta"
========= 1 failed, 983 passed, 15147 deselected in 222.61s (0:03:42) ==========
# the 1 failure is test_content_router_single_item_deadline.py::test_single_cache_miss_fails_open_at_deadline,
# a wall-clock assertion (0.126s vs < 0.12s). It fails identically on origin/main on this machine.

$ ruff check .            (ruff 0.16.8, CI's pinned version)
All checks passed!
$ ruff format --check .
1757 files already formatted
$ mypy headroom/proxy/handlers/anthropic.py headroom/transforms/content_router.py headroom/cache/prefix_tracker.py --ignore-missing-imports
Success: no issues found
```

## Real Behavior Proof

- Environment: macOS 26.6 (arm64), Python 3.12.13 (the repo's `.venv`),
Claude Code 2.1.290, `claude-sonnet-5-5`, real Anthropic API calls, with
the proxy started from this branch.
- Exact command / steps: `python3
benchmarks/tool_search_vs_native/run.py --arms native,hr_hot,hr_client
--reps 3 --warmup`, then the same with `--context-kb 200` for the long
task.
- Observed result: median of 3 runs after a warmup; every run passed
(table below). Before the fix, `HEADROOM_DEBUG_PREFIX_MISMATCH` showed
`relation=diverged first_diff_index=8`: the Bash `tool_use` with `cd …
&&` stripped and `caller` dropped.
- Not tested: harnesses other than Claude Code (the OpenAI-format MCP
path is unit-tested only), Bedrock and Vertex upstreams, real (larger)
MCP server catalogs, models other than Sonnet 5.5.

| | native Claude Code | Headroom + CC ToolSearch | Headroom, routine
tools resident |
|---|---|---|---|
| long task, before | $0.283 | $0.323 | $0.426 (66.8K-token cache
re-write every run) |
| long task, after | $0.284 | **$0.284** | **$0.255** (6 API calls vs
native's 7) |
| short task, after | $0.061 | $0.062 | **$0.046** |

## Runtime Rollout Safety

- Rollout-managed feature(s): none; both changes are in the always-on
cache-mode and compression paths.
- Minimum rollout channel: stable.
- Stable/default behavior changed: yes. A re-serialized last reply no
longer drops cache mode to raw forwarding, and MCP results under 4000
characters are no longer compressed, giving up a few tokens of savings
on those results.
- Kill switch / disable path: `HEADROOM_MCP_RESULT_MIN_CHARS=0` for the
MCP floor. The prefix fix has no flag, because it only applies where the
old code forwarded raw.
- Unsafe override required: no.
- Qualification impact: less compression on small MCP results; fewer
cache writes and extra turns.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- **Devin review follow-ups are in `d043bd5e`:**
  - the OpenAI-format MCP floor;
  - owner-only debug dumps;
  - exit-code and `get_issue` scoring;
  - the environment and commands in the README.
- **The two `npm audit` failures** (docs, e2e/wrap/tools) also fail on
`main`.
- **`tests/test_cli_proxy_improvements.py` hangs** when the whole file
runs locally, also on `main`. That is pre-existing and not touched here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 10:58:23 -05:00
trs-80andClaude Opus 5 3e36aefb43 perf(memory): reindex FTS5 a page at a time instead of a record at a time (#3466)
## Description

`headroom memory reindex` clears `memory_fts` and rebuilds it with one
`index_memory()` call per record. `FTS5TextIndex.index_raw()` commits
per call, so rebuilding a 1,000-memory store costs 1,000 transactions
and 1,000 fsyncs. The adapter already ships `index_batch_memories()`;
this uses it.

Closes # N/A

## Type of Change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `reindex_memories()` now calls `index_batch_memories()` once per page,
reusing the existing `_REINDEX_PAGE_SIZE` constant rather than
introducing a new knob. No adapter changes.
- On a failed page, the previous per-record loop runs **for that page
only**, so valid records still land and the failing record is still
named in the warning.
- A page that fails and then fully recovers does not change the exit
status. `ok` stays tied to whether records actually got indexed, exactly
as before.
- Added `benchmarks/memory_fts_reindex_benchmark.py` for reproduction.

**Why per page rather than one batch over the whole store.** A single
transaction would be marginally faster, but the reindex deliberately
clears the index before rebuilding (`DELETE FROM memory_fts`,
`headroom/cli/memory.py:1208-1210`). One failed statement would then
roll the whole rebuild back to an *empty* index — silently destroying
search coverage while reporting a clean-looking failure. A page bounds
both journal growth and blast radius, and keeps essentially all of the
speedup.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Six new tests in `tests/test_cli_memory_index_sync.py`, reusing that
file's existing helpers and its `_REINDEX_PAGE_SIZE` monkeypatch
pattern:

1. `..._batches_pages_instead_of_indexing_record_by_record` — asserts
the observed page sizes are `[2, 2, 1]` and fails outright if the
per-record path is touched.
2. `..._batched_output_matches_per_record_output` — every stored column
(`memory_id`, `content`, `user_id`, `session_id`, `category`) and the
FTS query result ordering are identical either way.
3. `..._empty_store_indexes_nothing_and_succeeds`.
4. `..._indexes_more_records_than_one_page`.
5. `..._failed_batch_still_indexes_valid_records_in_that_page` — the
failure mode the design is built around.
6. `..._failed_record_is_named_and_exits_nonzero` — per-record
diagnosability and exit status survive batching.

Tests 1 and 5 fail on the base commit and pass with this change.

### Test Output

```text
$ pytest tests/test_cli_memory_index_sync.py tests/test_memory/ -q
477 passed, 145 skipped in 10.38s

$ pytest tests/test_memory/ tests/test_cli_memory_index_sync.py tests/test_cli/ -q
1212 passed, 146 skipped, 2 failed
# Both failures (test_unwrap_claude_stops_claude_owned_persistent_deployment,
# test_wrap_codex_prepare_only_updates_config) also fail on the base commit in
# this environment: they read ambient CODEX_HOME / ANTHROPIC_BASE_URL from the
# developer's shell rather than isolating it. Unrelated to this change.

# The two behavioral tests against the base commit:
$ git stash push headroom/cli/memory.py
$ pytest tests/test_cli_memory_index_sync.py -q -k "reindex_batches or failed_batch"
2 failed

$ pre-commit run   # via commit hooks
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.13.11, stdlib `sqlite3` with
FTS5, worktree off `upstream/main` @ `e67b3c8a`.
- Exact command / steps: `python
benchmarks/memory_fts_reindex_benchmark.py --records 1000 --runs 3` for
the indexing operation; separately, a `CliRunner` harness timing the
whole `memory reindex` command over a 1,000-memory store, run on this
branch and then with `headroom/cli/memory.py` stashed. Each run uses a
fresh temporary database.
- Observed result: the indexing operation alone goes from **759.7 ms to
8.1 ms** (median of 3, ~94x). The **whole `memory reindex` command**
goes from **733.5 ms to 15.8 ms** (median of 3, ~46x) — reported
separately because the command also pages the primary store and does
vector-orphan cleanup, which this change does not touch. The audit that
motivated this measured 778.69 ms vs 5.60 ms for the operation;
reproduced within noise.
- Not tested: real-world stores larger than 1,000 records, concurrent
access during reindex, and behavior on network filesystems where fsync
costs differ. Synthetic records (~40 tokens each) rather than production
memory content. No end-to-end proxy or agent-latency claim is made.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; no flag or channel gates this.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Same records indexed, same
fields, same FTS results, same output lines, same exit status. Only the
transaction granularity changes.
- Kill switch / disable path: none needed. If the batch path raises for
any reason, the command falls back to the previous per-record behavior
automatically, per page.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the commit. `reindex` is idempotent — it clears
and rebuilds the index from the primary store — so re-running it on any
build produces the same result regardless of which version wrote the
index.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- No user-facing docs describe this internal rebuild path, so
"documentation update" is satisfied by the comment explaining why the
batch is bounded to a page rather than the whole store — that constraint
is the non-obvious part of this diff.
- Follow-up, deliberately not in this PR: the two failing tests noted
above depend on the developer's ambient `CODEX_HOME` and
`ANTHROPIC_BASE_URL`, so they pass or fail depending on whose shell runs
them. That is a test-isolation bug worth its own change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 16:45:52 -05:00
Tejas ChopraandClaude Opus 5 34da21e89e fix(tokenizers): disable trust_remote_code and allowlist HF tokenizer repos (#3723)
## A-1 (S1, remote code execution): HF tokenizer loader ran caller-named
repositories

`headroom/tokenizers/huggingface.py` called
`AutoTokenizer.from_pretrained(...,
trust_remote_code=True)` on an identifier that traces back to a proxied
request body.
`get_tokenizer_name()` ended in *"assume the model name is the tokenizer
name"*, so an
unrecognised `model` field was handed straight to the loader, which
resolves it against the
HuggingFace Hub and — with `trust_remote_code=True` — executes whatever
Python that
repository ships. Publish a repo, name it in a request body, get code
execution in the proxy
process. Nothing constrained the identifier.

### The fix

1. **`trust_remote_code=False` on every path**, passed explicitly rather
than leaning on a
   library default upstream can change.
2. **`ALLOWED_TOKENIZER_REPOS`**, derived from the mappings the product
actually ships
(`MODEL_TO_TOKENIZER.values()`, so adding a family allows it and nothing
drifts in).
`_load_tokenizer` checks it *before* importing transformers: an
unallowlisted name is
refused outright and never becomes a Hub lookup. It fails closed to
character estimation
rather than silently substituting a different vocabulary under the
caller's name.
3. **`get_tokenizer_name()` fails closed.** The fallthrough now returns
`DEFAULT_TOKENIZER`
instead of echoing the caller's string. A full repo id is accepted only
when it is on the
allowlist, and then the value returned is *our* canonical spelling,
never the caller's.

Two layers on purpose: the resolver is the normal path, the loader is
the chokepoint —
`_load_tokenizer` is importable and callable with an arbitrary string.

`benchmarks/comprehensive_eval.py` also passed `trust_remote_code=True`
to
`datasets.load_dataset`. The brief only named the two tokenizer call
sites; that third one is
fixed here too, and the tree-wide test below now holds the line for all
of them.

### Operator escape hatch

`HEADROOM_HF_TOKENIZER_ALLOWLIST` (comma-separated repo ids) lets an
operator admit private
or self-hosted tokenizer repositories. It is deliberately
environment-only — server-side
configuration, never anything a client can reach through a request body
— and repos added
there are still loaded with `trust_remote_code=False`.

### Default change a customer could depend on

An unrecognised model previously resolved to *itself* as a tokenizer
name. In practice a
made-up id failed to resolve on the Hub and counting fell back to
estimation, so most
deployments see no change. Two cases do change:

- A caller who relied on passing an **arbitrary HF repo id** not in
`MODEL_TO_TOKENIZER`
(e.g. a private fine-tune) and getting exact counts now gets the default
tokenizer
  instead. Remedy: add the repo to `HEADROOM_HF_TOKENIZER_ALLOWLIST`.
- Unknown models now resolve to `DEFAULT_TOKENIZER` (Llama 3), so a
deployment with that
vocabulary in its HF cache gets a real BPE count where it previously got
estimation.

Repo ids that already resolved correctly are unchanged. Alias and prefix
matching run first
and are untouched, so existing precedence — including
`mistralai/Mixtral-8x7B-v0.1` hitting
the `mistral` prefix — resolves exactly as before; there is an explicit
test pinning that so
a future reorder has to acknowledge it.

### Tests

- `tests/test_huggingface_tokenizer_allowlist.py` — a crafted `model`
never reaches
`from_pretrained`; every resolution lands inside the shipped set; the
loader refuses an
unallowlisted repo without touching the Hub; `trust_remote_code` is
`False` on both the
cache-only and bounded-network calls; every shipped alias still resolves
to its mapping;
the operator env hatch admits only what it names; and, since matching is
case-insensitive, the string handed to the Hub is the allowlist's own
spelling
  rather than the caller's variant.
- `tests/test_no_trust_remote_code.py` — tree-wide scan over every
git-tracked file
  asserting every `trust_remote_code` assignment is `False`.
- `tests/test_huggingface_tokenizer_timeout.py` — the existing GH #1701
timeout tests used
placeholder repo ids (`some/model`, `slow/model`, `offline/model`) which
the allowlist now
rejects before the timeout logic runs, short-circuiting what those tests
exist to prove.
Switched to real shipped repo ids so they still exercise the
bounded-load path.

### Fail-before / pass-after evidence

Method: commit, `git checkout origin/main --
headroom/tokenizers/huggingface.py
benchmarks/comprehensive_eval.py` (source reverted, new tests kept),
run, then
`git checkout HEAD -- ...` to restore.

**Before (source reverted, tests kept) — 10 failed, 4 passed:**

```
tests/test_huggingface_tokenizer_allowlist.py FFFF..F..FFF               [ 85%]
tests/test_no_trust_remote_code.py FF                                    [100%]

=================================== FAILURES ===================================
E   AssertionError: attacker-named repository was handed to from_pretrained
    assert 'attacker-controlled/headroom-rce-poc' not in ['attacker-controlled/headroom-rce-poc']

E   AssertionError: 'attacker-controlled/headroom-rce-poc' escaped the allowlist
    assert 'attacker-controlled/headroom-rce-poc' in frozenset({'01-ai/Yi-1.5-6B', ...})
     +  where 'attacker-controlled/headroom-rce-poc' = get_tokenizer_name('attacker-controlled/headroom-rce-poc')

E   AttributeError: module 'headroom.tokenizers.huggingface' has no attribute 'DEFAULT_TOKENIZER'

E   AssertionError: trust_remote_code must be False everywhere — it executes repository-supplied Python in this process:
      benchmarks/comprehensive_eval.py:44: trust_remote_code=True,
      headroom/tokenizers/huggingface.py:162: trust_remote_code=True,
      headroom/tokenizers/huggingface.py:184: trust_remote_code=True,

E   AssertionError: assert {'True'} == {'False'}

=========================== short test summary info ============================
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_never_reaches_the_loader
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_a_shipped_tokenizer
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_the_declared_default
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_loader_fetches_the_allowlists_own_spelling
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_remote_code_disabled_on_cache_and_network_paths
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_repo_id_matching_is_case_insensitive_but_returns_our_spelling
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_operator_env_allowlist_admits_a_private_repo
FAILED tests/test_huggingface_tokenizer_allowlist.py::test_empty_operator_env_admits_nothing_extra
FAILED tests/test_no_trust_remote_code.py::test_remote_code_is_never_enabled_anywhere_in_the_tree
FAILED tests/test_no_trust_remote_code.py::test_the_tokenizer_loader_sets_the_flag_explicitly
========================= 10 failed, 4 passed in 1.47s =========================
```

**After (source restored):**

```
tests/test_huggingface_tokenizer_allowlist.py ............               [ 54%]
tests/test_no_trust_remote_code.py ..                                    [ 63%]
tests/test_huggingface_tokenizer_timeout.py ........                     [100%]

============================== 22 passed in 2.73s ==============================
```

Wider tokenizer suites (`test_tokenizers.py`, `test_tokenizer.py`,
`test_tokenizers/`,
`test_tokenizer_encoding_resolution.py`,
`test_tokenizer_selection_coverage.py`,
`test_provider_tokenizer_one_ruler.py`,
`test_tokenizer_count_offload.py`,
`test_strands_tokenizer.py`,
`test_compress_route_tokenizer_by_model.py`,
`test_evals_cjk_tokenization.py`): **166 passed, 14 skipped**.

Lint: `ruff 0.16.7 check .` and `format --check .` both clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 20:20:56 +02:00
Tejas ChopraandClaude Opus 5 89a58fd152 fix(pricing): price savings against the cache mix, not flat list price (#3699)
## The problem

Every savings figure Headroom reports is a counterfactual: *what would
this request have cost had we not removed those tokens?* We answered it
with `tokens_saved * input_cost_per_token` — flat list price — which is
true **exactly once**, for the first request of a cache window. After
that, the tokens we removed would have been served from the provider's
prompt cache at a fraction of list price (Anthropic 0.10x, OpenAI 0.50x,
Gemini 0.10x).

Measured on a real 238M-input-token corpus that was **87.8% cache
reads**:

| | |
|---|---|
| effective rate actually **paid** | $1.592/M |
| rate savings were **priced at** | $4.339/M — **2.73x** |
| reported compression savings | **$330.89** |
| same tokens at the observed mix | ~$121 |

For **tool-schema deferral** the gap is ~10x: tool definitions sit at
the very front of the prompt and are the most cacheable tokens in it.

## The fix

The correct rate depends on **where in the request** the tokens sat:

| Region | What | Pricing |
|---|---|---|
| `PREFIX` | tool schemas — ahead of every cache breakpoint | warm turn
= **entirely a cache read** (read-first, not pro-rata) |
| `LIVE_ZONE` | the appended delta handlers compress | **never** a cache
read; cold turn saves 1.25x list, not 1.0x |

`headroom/pricing/counterfactual.py` is now the single place cache rates
are resolved and counterfactual tokens are priced — for the live cost
card, the persisted tracker and the durable ledger alike, so the three
cannot drift apart again. It replaces three separate implementations,
two of which were list-only.

Verified, warm Anthropic turn, 8k deferred tool-schema tokens: `$0.0240
list → $0.0057` (4.2x correction). Cold turn: `$0.0240 → $0.0300` —
correctly **above** list, the case list pricing *under*-reported.

## Also fixed

- **1h-TTL writes were priced at the 5m rate.** The proxy counted and
displayed the 5m/1h split but never priced it. LiteLLM publishes the 1h
rate per model (Sonnet: 2.00x base); `pricing/cache_ttl.py` — written
for exactly this and never imported by the cost path — derives it where
a row omits one.
- **Prefix tokens were bucketed pro-rata**, handing them a share of the
1.25x write premium they never paid (~2x high on a warm turn).
- **OTel exporter crash risk**: it called `float()` over every value in
the savings breakdown, so a non-numeric field would take the whole
request-metrics path down.
- **Ledger compaction thrash** (pre-existing, amplified here):
`_maybe_compact` rewrote the entire file under `flock` on *every append*
once it passed 1 MB with nothing out of retention to drop. Added a
per-process back-off; retention is still enforced on read.

## Schema changes

**Ledger v1 → v2**, **savings state v5 → v6.**

v1 stored one list-priced dollar and no mix, so its history was not
merely overstated — it was **unfixable**, because nothing on disk said
what fraction of the removed tokens would have been cache reads. v2
stores the savings split and the observed mix, so a future basis change
can re-derive the whole ledger.

Pre-v6 state seeds both columns from the single figure it has and is
labelled `list` for good; `savings_basis_migrated_at` marks where the
honest numbers start. v1 ledger events still read and aggregate
alongside v2.

## Honesty over precision

Every figure now ships with its list-price ceiling and a `basis` label
saying how soundly it was derived (`catalog`, `catalog+ttl-ratio`,
`provider-ratio`, `list`, `no-mix`, `unpriced`). Totals report their
**weakest** contributing basis, so one unpriceable request can't launder
a total into looking catalog-grade. A request with no cache breakdown —
the MCP tool path, a non-reporting gateway, a streaming turn with no
usage frame — still prices at list and **says so**.

## Harness- and provider-agnostic

The fix sits at the single `emit_request_outcome` → `record_request`
chokepoint that all **32** handler emit sites route through (anthropic
6, openai 11, batch 5, gemini 3, streaming 3, bedrock 1, gateway_turn 2,
server 1) — no per-handler work.

`tests/test_savings_provider_matrix.py` asserts the matrix directly:

| Provider | Reports | Handled |
|---|---|---|
| Anthropic / Bedrock / Vertex | read + write + 5m/1h + uncached | full
|
| OpenAI | read + uncached, write **inferred** | inferred write dropped
— no premium OpenAI doesn't charge |
| Gemini | read only | prices from reads alone |
| gateway / MCP | nothing | list + labelled `no-mix` |

## Tests

- **New:** 21 pricing + 13 provider-matrix + 3 ledger (compaction
back-off, v1/v2 mixing)
- **Updated:** 6 files for the deliberate schema change
- Local sweep across everything touching savings/cost/pricing/metrics:
**904 passed**; targeted re-run after review fixes: **125 passed**
- `ruff check` + `format` clean under the **CI-pinned 0.16.7** (local
0.16.3 missed 5 C408s)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 14:41:30 -07:00
trs-80andClaude Opus 5 389e45cdde perf(cache): purge expired CCR rows by deadline instead of decoding every payload (#3465)
## Description

`CompressionStore._clean_expired()` runs on every CCR insertion and
calls `backend.items()`, which deserializes every live original just to
read the `created_at`/`ttl` that SQLite already stores as columns. With
1,000 unexpired 16 KiB originals, one new insertion decoded all 1,000
payloads.

This replaces that scan with a metadata-only delete on the persistent
backend.

Closes # N/A

## Type of Change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `SQLiteBackend.purge_expired()` deletes expired rows with `DELETE ...
WHERE created_at + ttl < ?`, backed by a new expression index
`idx_ccr_expiry_deadline`. `_maybe_purge()` now shares the same helper
instead of duplicating the statement.
- `CompressionStore._clean_expired()` calls the backend method when the
backend offers one, and folds its deleted-row count into
`_stale_heap_entries` so eviction accounting and the heap-rebuild
threshold behave exactly as before. Backends without `purge_expired` —
`InMemoryBackend` and any third-party implementation — keep the existing
item-scan path, so the `CompressionStoreBackend` protocol is unchanged.
- Dropped the superseded `idx_ccr_expiry (created_at)`. Nothing in the
tree searches or orders by bare `created_at`, so it only cost writes;
the schema now drops it, migrating existing database files on open.
- Added `benchmarks/ccr_sqlite_expiry_benchmark.py` for reproduction.

Behavior deliberately preserved: expired entries are still cleaned
before any live entry is evicted, TTL semantics and the expiry boundary
are unchanged (`created_at + ttl == now` is not expired), and
originals/markers/retrieval, persistence and eviction feedback are
untouched.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Three new tests in `tests/test_ccr_sqlite_backend.py`:

1. `test_purge_expired_deletes_by_deadline_and_keeps_the_boundary_row` —
mixed TTLs and the exact expiry boundary.
2.
`test_new_insert_purges_expired_sqlite_entry_without_decoding_live_payloads`
— fails the test if a new insertion decodes any existing payload; this
is the regression guard for the issue.
3. `test_opening_a_db_with_the_old_index_drops_it_and_keeps_the_rows` —
a database file written with the old index migrates on open with its
rows intact.

Tests 1 and 2 both fail on the base commit and pass with this change.

### Test Output

```text
$ pytest tests/test_ccr_sqlite_backend.py tests/test_compression_store.py -q
98 passed in 7.35s

$ pytest tests/ -q -k "ccr or compression_store or cache"
1839 passed, 8 failed
# All 8 failures (7 Playwright dashboard, 1 live-API) also fail on the base
# commit in this environment; they are unrelated to this change.

# The two new behavioral tests against the base commit:
$ git stash push headroom/cache/backends/sqlite.py headroom/cache/compression_store.py
$ pytest tests/test_ccr_sqlite_backend.py -q -k "purge_expired or purges_expired_sqlite_entry"
E   Failed: new insertion decoded existing CCR payloads
2 failed, 21 deselected

$ pre-commit run --all-files   # via commit hooks
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.13.11, SQLite via stdlib
`sqlite3`, worktree off `upstream/main` @ `e67b3c8a`.
- Exact command / steps: `python
benchmarks/ccr_sqlite_expiry_benchmark.py --entries 1000 --payload-bytes
16384 --runs 7`, run on this branch and then with the two source files
stashed, sequentially in the same process environment with identical
inputs.

  ```text
candidate:
{"preexisting_entries":1000,"original_bytes_each":16384,"runs":7,"median_insert_ms":0.04654214717447758}
baseline:
{"preexisting_entries":1000,"original_bytes_each":16384,"runs":7,"median_insert_ms":17.63250003568828}
  ```

- Observed result: median insertion **17.63 ms → 0.047 ms**; payload
deserializations during cleanup **1,000+ → 0** (counted by instrumenting
`_entry_from_json`, and asserted permanently by test 2). Separately, on
1,000 rows × 16 KiB (n=200), dropping the old `created_at` index leaves
the purge delete unchanged at ~0.0012 ms and makes each insert ~14%
cheaper (0.0064 ms → 0.0056 ms); the new expression index is what takes
the purge from 0.171 ms to 0.0012 ms.
- Not tested: end-to-end proxy latency. This is an insertion-path
micro-measurement on synthetic entries and should not be read as an
end-to-end speedup. No measurement on very large databases (>1,000
entries) or under concurrent writers. `InMemoryBackend` is unchanged and
unmeasured.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; no flag or channel gates this.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no observable behavior change — the
same entries expire at the same times. Only the persistent backend's
schema (one index added, one dropped) and the cleanup mechanism change.
- Kill switch / disable path: none needed; a backend without
`purge_expired` transparently uses the previous code path.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the commit. The index changes are idempotent DDL
applied on open, so a reverted build re-creates `idx_ccr_expiry` and
simply stops using `idx_ccr_expiry_deadline`; cached rows are never
rewritten or dropped by the migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- "Documentation update" is checked as done in the sense that no
user-facing docs describe this internal cleanup path; the new behavior
is documented in docstrings and in the schema comment explaining why the
old index is dropped.
- The `_purge_expired`/`purge_expired` split is deliberate rather than
redundant: `SQLiteBackend._lock` is a plain `threading.Lock`, and
`_maybe_purge()` already runs inside `set()`'s critical section, so it
cannot call the public lock-acquiring method.
- `purge_expired(now=None)` takes an explicit time only so the boundary
test can assert deterministically that `created_at + ttl == now` is not
expired.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 20:42:04 -07:00
trs-80 3169935858 perf(ccr): bound retrieval history with a deque instead of reslicing (#3471)
## Description

Once the CCR retrieval history reaches its 1,000-event cap, every
further retrieval appended to the list and then rebuilt it from a
1,000-element slice, under the store lock.
`collections.deque(maxlen=1000)` drops the oldest event as the new one
lands, so the trim is deleted rather than moved.

This is a maintenance change that also removes an O(cap) copy. It is
**not** an end-to-end speedup, and the measurements below say so: the
history append itself gets ~3.9× cheaper, while the full `retrieve()`
path is unchanged within noise.

`_pending_feedback_events` deliberately stays an unbounded list. It is a
separate drain-by-swap queue that still owes a notification for every
event, including ones the capped history has already discarded, so both
containers now carry a comment saying which is which.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/compression_store.py`: `_retrieval_events` becomes
`deque(maxlen=self._max_events)`; the append-then-reslice in
`_log_retrieval` is removed. `_max_events` stays as the single readable
name for the cap — it is read at construction, by the new tests and by
the benchmark, and is never assigned after `__init__`.
- `tests/test_compression_store.py`: two tests — retention and ordering
at the cap, and one feedback notification per retrieval past the cap.
- `benchmarks/ccr_retrieval_history_benchmark.py`: standalone script
measuring the history append at steady state, following the convention
the other benchmarks in that directory use.

Consumer audit before changing anything: the container is only ever hit
with `len`, `append`, `clear`, iteration, `list()` and `sys.getsizeof` —
all deque-safe. The one slice, `events_copy[-limit:]` in
`get_retrieval_events`, operates on the list copy taken under the lock,
so deque's lack of slice support is never reached. Repo-wide there is no
non-Python consumer, no subclass, no pickling or deepcopy path and no
`isinstance(list)` check, and the deleted line was the only assignment
to the attribute — so nothing can rebind an unbounded list back onto it.
One `threading.Lock` guards every access.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Both new tests pass before and after this change, deliberately: for a
maintenance change, parity is the claim. They exist because the
pre-existing `test_retrieval_events_list_bounded` asserts only `len(...)
<= 1000`, which cannot tell which end was dropped. Mutation-checked: an
unbounded `deque()` fails the length assertion, and a head-trim keeping
the oldest 1,000 passes the length check but fails on order.

### Test Output

```text
$ pytest tests/test_compression_store.py tests/test_ccr.py tests/test_ccr_feedback.py \
         tests/test_critical_gaps.py tests/test_critical_fixes.py tests/test_proxy_ccr.py \
         tests/test_ccr_sqlite_backend.py tests/test_ccr_batch_store.py \
         tests/test_memory_tracker_integration.py -q
209 passed, 3 skipped in 16.66s

$ pytest tests/ -q -k "ccr or compression_store or compression_feedback or telemetry \
                       or critical_gaps or critical_fixes or memory_tracker" \
         --ignore=tests/test_integrations
1 failed, 921 passed, 11 skipped in 77.38s
# the one failure is test_ccr_marker_round_trip_live, a live-API test that fails
# identically on the unchanged base

$ python benchmarks/ccr_retrieval_history_benchmark.py
container: deque
cap: 1000 events
appends per run: 20000, runs: 7
median: 0.481 us/append

$ pre-commit run --files headroom/cache/compression_store.py \
      tests/test_compression_store.py benchmarks/ccr_retrieval_history_benchmark.py
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: macOS 15.6, Python 3.13.11, branch based on `main` @
`e67b3c8a`. Single process, single thread, no contention.
- Exact command / steps: filled a store's history to its cap so every
subsequent append must evict, then timed 20,000 `_log_retrieval` calls,
7 runs, median reported. Ran the unchanged tree and this change in both
orderings to rule out ordering effects. Separately timed the full public
`retrieve()` path, 2,000 calls × 5 runs, also in both orderings.
- Observed result: history append **1.81 to 0.47 us** and, with the
order reversed, **1.85 to 0.48 us** per append. Full `retrieve()`: 7.67
vs 7.89 us, and reversed 7.79 vs 7.82 us — unchanged within noise in
both directions. Behavior parity checked value by value:
`get_stats()["event_count"]`, `get_retrieval_events()`
contents/order/length across limits from negative through 1e9, and
`clear()` are all identical; `deque.clear()` preserves `maxlen`, so the
container stays bounded afterwards.
- Not tested: no multi-threaded or contended measurement, no
multi-worker or production workload, and no memory-over-time profile.
The benchmark times a private method (`_log_retrieval`) because the
public `retrieve()` buries the difference in backend I/O and the
feedback drain — that is stated in the benchmark's own docstring. The
result is a microbenchmark of one operation and must not be read as a
proxy for retrieval latency.

One value does change numerically, and it is not hidden:
`get_memory_stats().size_bytes` shifts by a few hundred bytes, because
`sys.getsizeof` reports a deque's 64-slot blocks rather than a list's
contiguous array (56 to 760 bytes empty; 8,056 to 8,680 at 1,000
entries). It is a shallow size that already excluded the events
themselves, `budget_bytes` is `None`, and no test or dashboard compares
it against anything.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. CCR retrieval-event history is not
rollout-gated.
- Minimum rollout channel: N/A — no rollout-managed feature is touched.
- Stable/default behavior changed: No. The cap, its size, retention
order, retrieval results, lock guarantees and feedback delivery are all
unchanged; only the container enforcing the cap differs, plus the
`size_bytes` gauge noted above.
- Kill switch / disable path: N/A — no flag. Reverting the commit
restores the list and its trim.
- Unsafe override required: No.
- Qualification impact: None. No compression decision, retrieval result,
CCR marker, TTL or persistence path is touched — this is the in-memory
display history only.
- Rollback path: `git revert` of this commit, single step. No schema,
data or config migration is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable — no user-visible surface changes.

## Additional Notes

- No separate documentation file needed updating; the two comments in
`__init__` are the documentation this needs, because the one thing a
future reader can get wrong here is bounding `_pending_feedback_events`
to match.
- Overlaps nothing in flight: PR #3465 touches this same module, but its
entire footprint is inside `_clean_expired`, which never reads the
retrieval history, and this change never touches `_stale_heap_entries`.
- Unrelated, left alone, noticed while running the suites:
`tests/test_realignment_live_multi_turn.py::test_ccr_marker_round_trip_live`
and
`tests/test_memory_usage_integration.py::TestProxyMemoryIntegration::test_real_api_calls_memory_tracking`
both need live API access and fail on a clean checkout without it.
2026-09-11 20:42:01 -07:00
Tejas ChopraandClaude Opus 5 96dfc7005f chore: finish the org and docs-domain migration (#3389)
Two stale identities left over from moves that were only half applied.

## GitHub org

`install/state.py` already migrated Docker images to
`ghcr.io/headroomlabs-ai/headroom` and carries a
`_DEPRECATED_IMAGE_REPO` constant that rewrites the old path on load —
so this move was decided long ago. Everything else still said
`chopratejas/headroom`.

Rewritten across packaging (`pyproject.toml`, `Cargo.toml`), both plugin
marketplaces, the agent-hooks manifests, install scripts, CI templates,
crate docs and tests.

**Two deliberate exclusions:**
- **`ghcr.io/chopratejas/headroom` is untouched everywhere.** It is the
*input* to `_migrate_deprecated_image()` and to the tests covering it.
Rewriting it would silently disable the migration.
- **`CHANGELOG.md` keeps its 238 historical references.** GitHub's
rename redirect resolves them, and rewriting would bury this diff in
noise.

**HuggingFace model ids are NOT affected and were never in scope.**
`chopratejas/kompress-base`, `kompress-v2-base`, `kompress-finance`,
`technique-router`, `technique-router-onnx` and
`siglip-image-encoder-onnx` are account-scoped identifiers on a
different service. All six were validated against the HF API before this
change:

| Model | HTTP |
|---|---|
| `kompress-base` | 200 |
| `kompress-v2-base` | 200 |
| `technique-router` | 200 |
| `technique-router-onnx` | 200 |
| `siglip-image-encoder-onnx` | 200 |
| `kompress-finance` | **401 (private)** |

The 401 is unrelated to this migration, but worth knowing:
`kompress_compressor.py:1255` references it, so that path is not
reachable by an OSS user without access.

**One functional fix, not cosmetic:** `_marketplace_source()`
(`headroom/cli/init.py`) returned the old org as its fallback, so plugin
installs were pointed at the retired repo.

## Docs domain

The site moved to `docs.headroomlabs.ai`, but eleven references still
named the vercel.app host — including **`llms.txt` (27 links)**, which
is served at `docs.headroomlabs.ai/llms.txt` and read by LLM crawlers,
the **dashboard's own Documentation link**, and a `headroom wrap` help
message. The docs app's canonical-URL fallbacks were fixed separately in
#3388.

## Also folded in

`RUST_DEV.md` corrections found while verifying the above:
- `make build-wheel` points at `Cargo.toml`, not `pyproject.toml`
- the Rust toolchain is pinned, not tracking `stable`
- `headroom-simulators` was missing from the workspace listing
- the Python `CompressionStore` defaults to **SQLite**, not in-memory,
when `HEADROOM_CCR_BACKEND` is unset (verified by execution) — this one
matters for multi-worker reasoning

And the install scripts pointed at `docs/docker-install.md`, a path that
no longer exists; they now point at the live docs page.

## Verification

297 tests pass, including the plugin-manifest, install-state and
release-workflow suites that assert on these exact strings. `ruff check`
clean. Unrelated lockfile churn (`uv.lock`, both `package-lock.json`) is
deliberately excluded.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 15:53:35 +05:30
Tejas ChopraandClaude Opus 5 4bca514acd docs: correct the published site against the code (#3388)
Audited all 52 pages of docs.headroomlabs.ai/docs against source,
executing code rather than reading it. **36 pages changed.** Every
figure that could not be traced to a benchmark, a test, or a committed
artifact was measured or removed — none was replaced with a different
guess.

## The site was telling users things that were not true

**Telemetry.** `proxy.mdx` stated *"nothing is sent to Headroom Labs"*
and that the anonymous beacon *"has been removed from the code."*
Neither is true — `BEACON_DEFAULT_ON = True`
(`headroom/telemetry/beacon.py:56`), `is_beacon_enabled()` is fail-open,
and there is a live POST at `headroom/telemetry/session.py:75`.

`proxy.mdx` and `limitations.mdx` now describe both switches accurately,
say why the beacon exists (compression quality is the product; it shows
when a release regresses a ratio across real workloads), and list the 12
fields the collector allowlists instead of asserting "content-free". The
install id is disclosed for what it is: a random UUID, not a machine
fingerprint, resettable by deleting the file.

**Numbers.** The landing page's "Proof" table claimed 92/92/47/73%
savings from a harness that seeded nothing, so it could not reproduce a
figure even in principle.

| Scenario | Claimed | Measured |
|---|---|---|
| Code search (100 results) | 92% | **21%** |
| SRE incident debugging | 92% | **57%** |
| Codebase exploration | 47% | **42%** |
| GitHub issue triage | 73% | **30%** |

`real_world_agent_benchmark.py` now seeds, and
`benchmarks/index_proof_table.py` measures the same four scenarios
through the real tokenizer and the real `compress()` path with no
network call. Stable across five seeds; the page cites the command.

Latency was wrong in the *other* direction — 189 ms and 943 ms published
against **0.21 ms and 0.80 ms** measured. The docs made the product look
~1000× slower than it is.

**Examples.** Every `agno` example was broken at line one, including
Quick Start: `HeadroomAgnoModel`'s first dataclass field is `id`, so a
positional wrapped model bound to the wrong field and raised
`ValueError`. `api-reference.mdx` — the canonical signature reference —
had 13 broken examples, including three `AttributeError`s in the first
three lines of its `simulate()` block. Four integration pages omitted
the `[proxy]` extra and then told the reader to run `headroom proxy`,
which fails immediately. `ccr.mdx` showed SDK-level retrieval that only
exists in the proxy handlers.

Also: six Prometheus metric names that are never emitted, a Grafana
dashboard filtering on a label no metric carries, a documented deploy
workflow that does not exist, a 365-day retention default the code caps
at 30, and CCR TTL stated as 5 minutes against an actual 1800s.

## Canonical URL bug

`robots.ts`, `sitemap.ts` and `layout.tsx` fell back to a hardcoded
`headroom-docs.vercel.app` when `NEXT_PUBLIC_SITE_URL` was unset — so
the live site served a sitemap and `Host:` on the wrong domain and
pointed crawlers at the wrong canonical. Verified in the build output:
robots and all 53 sitemap URLs now carry `docs.headroomlabs.ai`.

Setting `NEXT_PUBLIC_SITE_URL=https://docs.headroomlabs.ai` in Vercel is
still worth doing so previews match, but the code no longer depends on
it.

## Verification

Four adversarial judges re-checked this work. They confirmed the
landing-page table reproduces digit-for-digit across five seeds, found
**zero cross-page contradictions**, independently regenerated all nine
surviving compression percentages, and proved `headroom_keep_turns` dead
at runtime with a mocked-transport test. They also caught two links this
audit had reversed to the deprecated org, and 18 examples it had missed
— both fixed here.

`next build` passes with all 52 pages prerendered, matching the
pre-change baseline.

## Follow-up, not in this PR

- GitHub org migration (`chopratejas` → `headroomlabs-ai`) across
source, manifests and packaging, plus stale-domain fixes in `llms.txt`,
the dashboard UI and `pyproject.toml`.
- `wiki/` disposition: it is unpublished and unbuilt, and
`wiki/benchmarks.md` carried an invented production-telemetry section
(250+ instances, 1.4 billion tokens saved) that exists nowhere in the
repo.
- `langchain.mdx` imports are dead against LangChain 1.x
(`langchain.memory`, `langchain.retrievers`,
`create_openai_tools_agent`). Needs a pin-vs-rewrite decision.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 15:15:37 +05:30
Tejas Chopra e0ce4b1d48 fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description

Removes both third-party CLI context tools — **rtk** and **lean-ctx** —
and with them the context-tool selector itself. Headroom no longer
downloads, installs or configures either one, and there is no
replacement.

The previous pass (#2344) gated only three entry points inside
`headroom/cli/wrap.py`. That left the feature reachable in practice:

| Gap | Effect |
|---|---|
| `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global
--auto-patch` from bash/PowerShell, **bypassing the Python gate
entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook
regardless of `HEADROOM_RTK` |
| `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was
broken by default**: `rtk_required=True` met a gate returning `None` →
`SystemExit(1)`. Invisible because all 8 openhands tests patched
`_ensure_rtk_binary` to a fake path |
| `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to
`rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker
polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) |
| No cleanup path | Nothing removed artifacts an earlier default had
installed, so a machine that once ran the old default kept rtk in the
loop forever (#1669, #1955) |

Also worth noting: the rtk binary download had **no SHA or signature
verification** — only `rtk --version` as a smoke test.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

**Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages,
`headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` /
`_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` /
`--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap
subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the
dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine
getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers,
`benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path
filters.

**Fails loudly, not silently** — `--context-tool` / `--no-context-tool`
/ `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in
shell profiles, aliases and CI jobs, and accepting them as a no-op would
read as Headroom having quietly stopped working. The installers reject
them too, which matters more than it looks: their arg parsers forward
the first unknown flag **and everything after it** to the wrapped tool,
so a leftover `--no-rtk` would have silently swallowed a following
`--port` and then been ignored downstream.

**New `headroom/context_tool_cleanup.py`** — deleting the code cannot
help a machine that already ran the old default, since the hooks,
binaries and injected guidance are durable on disk.
`purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and
removes the registered hook entries, the generated hook scripts, the
Headroom-managed `~/.local/bin` symlinks, the vendored
`~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server
entry and the marker-fenced instruction blocks. Deliberately
conservative: idempotent, **skips** a malformed config rather than
overwriting it, and only unlinks a symlink resolving inside Headroom's
own bin dir so a user's own build is untouched. It reports on
**stderr**, because `wrap/unwrap openclaw --prepare-only` emit
machine-readable JSON on stdout as their entire contract. Skipped for
`wrap selfheal` (runs from a SessionStart hook; must not race Claude
Code's writer for `~/.claude.json`) and for `--help`, which must stay
read-only.

**Client-config hardening** (discovered while investigating a "corrupted
Serena settings file" report) — `wrap.py` reset a settings file to `{}`
when an existing file would not parse, then wrote that back. One
hand-edited typo or a transient `EACCES`/`EINTR` on a valid file
destroyed the user's `permissions`, `env` and `hooks`, on **every
`headroom wrap claude`**. It now refuses to write. Separately,
`fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`),
fixing all 14 non-atomic client-config writes at once; it follows
symlinks rather than replacing them (dotfile managers) and preserves an
existing file's mode.

**Deliberately kept** — `rtk` stays in the wrapper-peel list in
`transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as
shell-command grammar, so `rtk cat f` is still classified as a file read
for anyone running their own rtk install, which the purge intentionally
leaves alone.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
All checks passed!

$ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
1255 files already formatted

$ mypy headroom/
Success: no issues found in 508 source files

$ pytest tests/test_context_tool_cleanup.py -q
11 passed

$ pytest tests/test_fsutil.py -q
12 passed

$ pytest tests/test_cli/test_wrap_codex.py -q            # 89 tests
89 passed in 431.68s
$ pytest tests/test_cli/test_wrap_opencode.py -q
39 passed in 257.46s
$ pytest tests/test_cli/test_wrap_helpers.py -q
45 passed
$ pytest tests/test_paths.py -q
75 passed
$ pytest tests/test_cli/test_unwrap_claude.py -q
14 passed
$ pytest tests/test_proxy_savings_history.py -q
39 passed
$ pytest tests/test_cli/test_wrap_copilot.py -q
27 passed
$ pytest tests/test_cli/test_wrap_zcode.py -q
20 passed
$ pytest tests/test_subscription_tracker.py -q
9 passed
$ pytest tests/test_proxy_dashboard_stats_cache.py -q
5 passed, 1 skipped
```

Repo-wide grep for 14 removed symbols (`headroom.rtk`,
`headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`,
`_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`,
`wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`,
`tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`,
`*.html`: **zero hits**.

Notable test changes: `test_wrap_openhands.py` no longer patches
`_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0
unpatched — the regression that was previously masked.
`test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed
(every test drove RTK instruction injection). A new
`test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed`
proves a pre-removal `subscription_state.json` still loads.

## Real Behavior Proof

- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @
this branch, real `~/.headroom` and `~/.claude` on the dev machine.
- **Exact command / steps and observed result:**

```text
# 1. Retired flag fails loudly instead of silently no-op'ing
$ headroom wrap codex --prepare-only --context-tool rtk
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they
rewrote shell commands through a third-party binary Headroom no longer manages.
Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL;
`headroom wrap` uninstalls what they left behind on first run.

$ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ...

# 2. install.sh rejects the retired flags (extracted parse_wrap_args harness)
['--no-rtk', '--port', '9999']   rc=1  ERROR: CLI context tools ... Drop --no-rtk
['--context-tool=rtk']           rc=1  ERROR: CLI context tools ... Drop --context-tool
$ bash -n scripts/install.sh   # syntax OK

# 3. Purge ran against the real machine, which had all the orphaned artifacts
$ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..."
  removed ~/.headroom/bin/lean-ctx        (51 MB)
  removed ~/.headroom/bin/rtk             (7.7 MB)
  removed ~/.local/bin/rtk                (symlink into ~/.headroom/bin)
  removed ~/.claude/hooks/rtk-rewrite.sh
  removed 8 lean-ctx-* hook scripts
# ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged
# → ~59 MB reclaimed, no unrelated key touched

# 4. stdout stays machine-readable while the purge reports (planted a fake artifact)
$ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err
$ cat out
{"enabled":true,"config":{"proxyPort":8787,...}}     # parses as JSON
$ cat err
Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk

# 5. --help is inert (planted artifact survives), a real run purges
$ headroom wrap codex --help   → artifact survived: CORRECT
$ headroom wrap openclaw --prepare-only → purged: CORRECT

# 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json
top-level keys 90 -> 90;  projects 19 -> 19;  LOST keys: none
all content outside mcpServers byte-identical: True
```

Dashboard rendered via the Playwright test after the panel removal:
"Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`,
and "Token Usage" reads Before Compression → Proxy Removed → After
Compression with no "Filtered (this session)" row. Nothing below the
removed panel broke.

- **Not tested:** Windows and Linux (macOS only) — `install.ps1` is
verified by brace-balance and inspection, not executed, since no `pwsh`
is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated
but not run; it needs the Docker e2e image. `serena project index`
interaction is exercised in the stacked base PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

**Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge
that first; this PR's base should then be retargeted to `main`, or it
will read as containing that fix too.

**Breaking-change migration for users:**
- Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`,
`--context-tool`, `--no-context-tool` from any alias, script or CI job,
and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error
rather than being ignored, so the failure is immediate and
self-explaining.
- Previously-installed artifacts are purged automatically on the next
`wrap`/`unwrap`; no manual cleanup needed.
- `headroom perf --json` no longer carries a `cli_filtering` key, and
`/stats` no longer returns a `context_tool` section.

**Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from
`README.md`,
`docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`,
`docs/observability.md` and the matching `wiki/` pages.
`REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED
rather than deleted, to keep the planning record.

**Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as
dead code — its only caller was the `except KeyboardInterrupt` guarding
the binary download, so with no download there is nothing slow left to
interrupt.
2026-07-30 22:59:41 -07:00
Zhenjia ZHOUandClaude Opus 4.8 4035c04187 feat(text-crusher): CJK-aware segmentation + relevance via ICU (#1504)
## Description

`TextCrusher` (the native extractive prose compressor added in #1171)
only handled ASCII: `split_segments` split on `.!?`+whitespace and
`tokens` split on whitespace/alphanumeric runs. CJK
(Chinese/Japanese/Korean) has neither spaces nor ASCII terminators, so a
whole CJK paragraph collapsed into **one segment / one token** — it
passed through at ~0% compression, and BM25 relevance + salience scored
zero terms.

This makes `TextCrusher` CJK-aware. CJK-bearing content takes an ICU
(`icu_segmenter`, UAX#29 sentence + dictionary word) segmentation path,
with a length fallback for terminator-sparse runs, a local BM25 over the
ICU word tokens, and ICU-token salience. Dispatch is on **content
only**, so pure-ASCII text is byte-identical to before — the shared
`BM25Scorer` and the ASCII path are untouched.

It also adds a committed, reproducible answer-retention eval
(`benchmarks/i18n_compression_eval.py`) with a deterministic zh/ja/ko CI
regression gate, so the improvement below is permanently verifiable
rather than a one-off measurement.

Extends #1171.

## Type of Change

- [x] Bug fix (CJK passed through near-uncompressed)
- [x] New feature (CJK segmentation / relevance support)
- [x] Performance improvement (CJK now compresses; ICU segmenters
cached, not rebuilt per call)

## Changes Made

- `is_cjk` predicate gates a CJK path (ideographs, kana, Hangul, CJK
punctuation, full/half-width forms).
- `split_segments` → ICU `SentenceSegmenter` for CJK + a mandatory
length fallback (whitespace / CJK punctuation / hard cap) for
terminator-sparse runs; ASCII path unchanged.
- `tokens` → ICU `WordSegmenter` (dictionary) for CJK; ASCII path
unchanged.
- `relevance_cjk`: a local BM25 over ICU word tokens — the shared ASCII
`BM25Scorer` scores zero terms for CJK and is parity-locked, so this is
an intentional separate scorer (documented in code).
- CJK salience uses ICU tokens (whitespace-split gave one giant "word" →
zero salience).
- `count_tokens`: CJK-aware so `compression_ratio` isn't nonsense for
space-free text.
- ICU segmenters resolved once in `static LazyLock` (compiled_data is
static) instead of rebuilt per call.
- New dep `icu_segmenter` 2.2, `compiled_data` only (see Dependency
below).
- `benchmarks/i18n_compression_eval.py` +
`tests/test_transforms/test_text_crusher_cjk_eval.py`: a zh/ja/ko
answer-retention eval — a deterministic needle CI gate (always-runs, no
external data), real-transcript fidelity with CJK-aware salient, and
optional `multi-wiki-qa` natural-data retention (loaded via the
`[evals]` `datasets` extra, skipped if absent; data never vendored —
CC-BY-NC-SA).

## Testing

- [x] Unit tests pass (`pytest` + `cargo test`)
- [x] Linting passes (`ruff check`/`format` on the new eval + test —
clean)
- [ ] Type checking passes (`mypy headroom`) — N/A, the only Python
added is a benchmark + test, not `headroom/` source
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

```text
$ cargo test -p headroom-core --lib text_crusher
running 12 tests
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 841 filtered out

$ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher*.py
15 passed

$ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher_cjk_eval.py
6 passed   # deterministic zh/ja/ko needle CI gate

$ cargo clippy -p headroom-core && ruff check benchmarks/i18n_compression_eval.py   # both clean
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.3.0), Python in a uv venv,
`headroom-core` built via `uv pip install -e .` (maturin), branch
`feat/cjk-text-compression`.
- Exact command / steps: built `_core`, then ran a mixed
Chinese+Japanese doc (no spaces, `。` terminators) through
`TextCrusher().compress(doc, "认证令牌缓存策略", 0.3)`; separately evaluated
answer-retention on the public CMRC2018 Chinese QA dev set (bury the
gold-answer paragraph among 25 distractors, query = the question,
compress to 30%, check the gold answer survives), and end-to-end through
`ContentRouter`.
- Observed result: a mixed Chinese+Japanese doc compressed 189 → 78
tokens (ratio 0.41, kept 3/8 segments) with the query-relevant sentence
surviving — before this change the same doc was a single segment → 100%
passthrough. On the public CMRC2018 Chinese QA dev set, answer-retention
under 30% compression rose 34% → 93% (multiple seeds). End-to-end
through `ContentRouter` on real CJK content, aggregate savings rose 16%
→ 40%. Pure-ASCII (English) output stayed byte-identical (the English
parity fixtures did not move). Demo terminal output:

    ```text
    ORIGINAL  tokens= 189  chars=189
    COMPRESS  tokens=  78  ratio=0.41  segments kept 3/8
    QUERY-RELEVANT sentence survived: True
    --- compressed output (verbatim kept CJK sentences) ---
    认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期条目。
    请求重试使用指数退避并设置最大次数上限。
    数据备份每天凌晨执行并保留最近三十天的快照。
    ```
The committed eval now demonstrates this across all three CJK languages.
The deterministic needle gate (in CI via
`tests/test_transforms/test_text_crusher_cjk_eval.py`, 6 passed) has
TextCrusher keep the query-relevant needle while truncate/random drop it
in zh, ja, and ko. On real `multi-wiki-qa` natural data (n=80/lang),
query-aware answer-retention is **zh 74% / ja 70% / ko 50%** vs
**25–41%** for the truncate/random baselines:

    ```text
=== Part A: multi-wiki-qa answer-retention (n=80/lang, target_ratio=0.3)
===
      lang    text_crusher  truncate  random
      zh-cn           74%       25%     38%
      ja              70%       31%     39%
      ko              50%       26%     41%
    ```
Korean is measurably weaker (ICU has no Korean dictionary and falls back
to UAX#29 word-breaking) — still well above baselines, and scoped as a
follow-up.
- Not tested: the live proxy HTTP path (validated at the `ContentRouter`
/ `TextCrusher` layer, not via a running proxy); no-space Korean
(standard Korean is space-delimited and is covered); non-CJK SE-Asian
scripts (out of scope).

## Dependency (per CONTRIBUTING supply-chain policy)

`icu_segmenter` 2.2 (ICU4X), `features = ["compiled_data"]`:

- **Why this package (vs. ourselves / existing deps):** CJK needs
dictionary/UAX#29 segmentation. A hand-rolled char-bigram scored
slightly worse on real data (CMRC2018 answer-retention: 92.5% ICU vs 91%
bigram, 4 seeds); jieba/lindera are ZH-only or 13–207 MB dicts. ICU4X
covers zh/ja/ko in one crate. The existing `unicode-segmentation` does
UAX#29 only (no CJK dictionary), so it can't word-segment space-free
CJK.
- **Who maintains it:** the official `unicode-org` ICU4X project; active
release cadence (2.2 in 2025); no known CVEs.
- **Install surface:** ~13 new pure-Rust crates, no build scripts, no
native code, no build/runtime network. `compiled_data` bundles locale
data at compile time (hermetic). `auto`/`lstm` deliberately NOT enabled
— LSTM covers SE-Asian scripts (Thai/Lao), not CJK, and would pull in
`libm` for nothing.
- **Why this version:** 2.x is the stabilized ICU4X API (1.x used a
different data-provider model); floored at 2.2 (Cargo.lock pins the
patch) since segmenter boundaries are observable in output and bumps
should be deliberate.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (CHANGELOG)
- [x] My changes generate no new warnings (clippy + fmt clean)
- [x] I have added tests that prove my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md

## Additional Notes

- **Parity:** the shared `BM25Scorer` (byte-exact parity-locked with
`headroom/relevance/bm25.py`) is untouched. `relevance_cjk` is a
separate local scorer because the shared one's tokenizer is ASCII-only.
The whole CJK path lives in Rust (`text_crusher.py` is a thin wrapper
over `_core`), so there is no Python mirror to keep in sync; the parity
fixtures stay green (only the CJK `unicode` fixture was re-recorded,
intentionally; English fixtures unchanged).
- **Known by-design gap (not a bug):** CJK content + a pure-ASCII query
yields no token overlap, so relevance falls back to recency + salience
(cross-script query matching is unsupported).
- The Python added is a benchmark
(`benchmarks/i18n_compression_eval.py`) plus its test, not `headroom/`
runtime source — both are `ruff`-clean; `mypy headroom` is unaffected.
- **License:** the optional Part A pulls `alexandrainst/multi-wiki-qa`
(CC-BY-NC-SA-4.0) at run time via the `[evals]` extra and is skipped if
absent — the dataset is never vendored into the repo, and the always-run
CI gate (Part C) uses only our own deterministic data.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 19:58:48 +00:00
Tejas Chopra 10251b65ca docs: sync README + benchmarks with code (drop retired IntelligentContext/RollingWindow) (#1545)
## Description

Sync the docs with the code after the live-zone realignment. The
`IntelligentContextManager` (ICM), `RollingWindow`, and scoring modules
were deleted in PR #350 (May 2026), but the README and benchmark
docstrings still advertised them as live, and an example still imported
the deleted module (broken on run). This fixes the README + benchmarks
and removes the dead example.

I validated the README against the code with three parallel
static-analysis sub-agents (features/architecture,
CLI/extras/wrap-matrix, public API/integrations). Most of the README
checked out accurate; only the items below were stale/wrong.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- README: removed the `IntelligentContext` bullet and
`IntelligentContext / RollingWindow` from the transforms list (both
deleted in PR #350).
- README: standardized `Kompress-base` -> `Kompress-v2-base` to match
the HF model id `chopratejas/kompress-v2-base` and the existing badges
(diagram re-aligned).
- README: corrected the CodeCompressor language list to match the
`CodeLanguage` enum (added TS, C, Perl).
- README: softened the unanchored "6 algorithms" tagline to
"content-aware compressors".
- README: Cortex Code is library-mode only — there is no `headroom wrap
cortex`, so the compatibility-matrix row no longer shows a wrap
checkmark.
- Deleted `examples/test_intelligent_context_toin_ccr.py` — it imported
the deleted `IntelligentContextManager` (ImportError on run) and is
unreferenced.
- Removed stale `RollingWindow` mentions from benchmark
docstrings/comments (`benchmarks/__init__.py`, `bench_transforms.py`,
`bench_latency.py`, `scenarios/conversations.py`); the accurate PR-B1
retirement comment is kept.

## Testing

- [ ] Unit tests pass (`pytest`) — N/A, docs/docstring + example
deletion only
- [x] Linting passes — `ruff check` clean on all changed benchmark files
- [ ] Type checking passes — N/A (no type-relevant changes)
- [ ] New tests added — N/A
- [x] Manual testing performed — see Real Behavior Proof

### Test Output

```text
$ ruff check benchmarks/__init__.py benchmarks/bench_transforms.py benchmarks/bench_latency.py benchmarks/scenarios/conversations.py
All checks passed!

# stale refs remaining in README/benchmarks (excluding accurate retirement notes):
$ grep -rn "IntelligentContext|RollingWindow|Kompress-base" README.md benchmarks/ | grep -v retire
(only benchmarks/bench_transforms.py:362 — the accurate PR-B1 retirement comment)

# deleted example is unreferenced anywhere:
$ grep -rn "test_intelligent_context_toin_ccr" --include=*.md --include=*.yml --include=*.py .
(no hits)
```

## Real Behavior Proof

- Environment: macOS (darwin, arm64), Python 3.12 `.venv`, ruff 0.14.x,
repo at branch `docs/sync-readme-with-code` off latest `main`.
- Exact command / steps: (1) three parallel sub-agents
grep/Read-validated README claims vs `headroom/`, `pyproject.toml`,
`sdk/typescript/`; (2) directly verified each flagged mismatch
(`CodeLanguage` enum, `HF_MODEL_ID`, absence of
`IntelligentContext`/`RollingWindow` classes); (3) confirmed the example
imports a deleted module and is unreferenced; (4) `ruff check` on
changed benchmark files; (5) re-grepped README + benchmarks for any
remaining stale refs.
- Observed result: README and benchmark docstrings now match the code;
the only surviving `RollingWindow` string is the accurate retirement
comment; the broken example is removed; ruff passes; the ASCII
architecture diagram still aligns after the `Kompress-v2-base` rename.
- Not tested: rendering of the README on GitHub/PyPI (text-only change);
the separate `docs/content/` and `wiki/` doc sets (see Additional Notes
— out of scope for this PR).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(docs/example cleanup)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)

## Additional Notes

**Larger related finding (NOT in this PR):** the published docs site
(`docs/content/docs/*.mdx`) and the `wiki/*.md` set still document
`IntelligentContextManager`, `RollingWindow`, `RollingWindowConfig`,
`IntelligentContextConfig`, and `ScoringWeights` as live API — with
`from headroom import RollingWindow` / `from headroom.transforms import
IntelligentContextManager` code examples that would `ImportError`. It is
half-migrated (a couple of `.mdx` files already note "removed in 0.9.x"
while neighbors still teach it as current). This is ~15 files and the
fixes require rewriting examples to the live-zone model, not just
deletions — recommended as a focused follow-up PR rather than bundling
it here.
2026-06-28 22:36:41 -07:00
Tejas Chopra c2fc4d3753 fix(ccr): make headroom_retrieve a hash-only full-content lookup (#1532)
The optional `query` parameter on headroom_retrieve routed retrieval
through CompressionStore.search(), which BM25-scored the items inside a
single cached blob and dropped everything below a 0.3 relevance floor.
On small per-blob corpora with conversational queries this returned an
empty result the large majority of the time, so the LLM saw "nothing
found" for content that was actually present — pushing users to turn
compression off entirely.

Retrieval is fundamentally a hash lookup (this already matches the Rust
proxy's CCR store, which is put/get only — "no BM25 search"). Remove the
query/search path end to end and always return the full original
content:

Core (Python proxy):
- tool schemas (anthropic/openai/google) drop the `query` property
- parse_tool_call returns the hash (str | None) instead of (hash, query)
- response handler, proxy POST/GET/tool-call handlers, the MCP retrieve
tool, and the streaming feedback recorders retrieve by hash only
- proactive context-tracker expansion always restores full content
- delete CompressionStore.search() and its BM25 machinery (the bm25
module stays — it is still used by relevance/)
- CCRToolCall.query, CCRToolResult.was_search, and
ExpansionRecommendation.expand_full/search_query are removed

Plugins (advertised a now-defunct query param to the LLM):
- hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop
`query` from their schemas, signatures, request URLs, and tests

Benchmarks/docs:
- ccr_regression + adversarial benchmarks switch from store.search() to
full hash retrieval (search input-injection tests repurposed to the
hash, the only remaining input surface)
- wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx,
config.py and store docstrings updated to describe hash-only retrieval

Tests updated to assert full-content retrieval and guard the removed
surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean.

## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
2026-06-28 10:32:43 -07:00
Zhenjia ZHOUandClaude Opus 4.8 6c68ff4e9f perf(compression): take large cold-start contexts off the synchronous kompress path (#1171) (#1298)
## Description

On a cold-start large context, kompress (ModernBERT ONNX) runs
**synchronously on the request thread** — ~200–300s for ~1M tokens. It
blows the 30s compression budget, leaks a non-preemptible worker, and
cascades (executor saturation → queue timeouts on healthy requests); on
timeout the request is forwarded **uncompressed** after eating 30s. This
adds four layered, **default-off, fail-open** mitigations so the request
path is never blocked on ML compression.

Closes #1171

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Phase 0 — size gate** (`HEADROOM_KOMPRESS_MAX_TOKENS`, default
50000): route oversized text away from ModernBERT (→ LogCompressor /
TextCrusher / passthrough) at the single `_try_ml_compressor` boundary.
- **Phase 1 — cooperative deadline**
(`HEADROOM_COMPRESSION_DEADLINE_MS`, default 20000): any kompress run
self-terminates at the next chunk boundary past the budget, keeping the
unprocessed tail verbatim.
- **Phase 2 — TextCrusher** (`HEADROOM_TEXT_CRUSHER`): a new **native
Rust** extractive prose compressor in
`crates/headroom-core/src/transforms/text_crusher/`, exposed via PyO3 as
`headroom._core.TextCrusher` with a thin Python wrapper. It **reuses the
shared `crate::relevance::BM25Scorer`** rather than reimplementing BM25,
and ships record/replay parity fixtures (mirroring the SmartCrusher
Rust-core + Python-shim pattern).
- **Phase 3 — off-path compression**
(`HEADROOM_BACKGROUND_COMPRESSION`): forward uncompressed immediately
and compress in a per-process background drain; a byte-identical cache
hit on a later turn means the request never blocks on ML.
- Benchmark (`benchmarks/text_crusher_quality_eval.py`), CHANGELOG
entry, and docstrings documenting the fail-open limits.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`, new modules)
- [x] New tests added for new functionality
- [ ] Manual testing performed (Phase 0/1 gate-fire + deadline observed
on real traffic in earlier iterations; Phase 3 off-path is unit- +
byte-identity-tested, not yet live-validated)

### Test Output

```text
$ pytest tests/test_transforms/ tests/test_cache/ \
    tests/test_proxy/test_background_compression.py tests/test_proxy/test_phase3_byte_identity.py -q
501 passed, 37 skipped in 40.33s

$ cargo test -p headroom-core --lib text_crusher
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 834 filtered out

$ ruff check <changed files>
All checks passed!

$ mypy headroom/proxy/background_compression.py headroom/transforms/text_crusher.py
Success: no issues found in 2 source files
```

New coverage: size-gate incl. the strategy-dispatch funnel (KOMPRESS +
TEXT); partial-run deadline (chunk-0 compressed + chunk-1 verbatim
tail); BackgroundCompressor (dedup / queue-full / fail-open); Phase 3
byte-identity round-trip; TextCrusher unit + parity.

## Real Behavior Proof

- Environment: macOS, local dev — `uv` venv, Rust `_core` built via `uv
pip install -e .`.
- Exact command / steps: the `pytest` / `cargo test` / `ruff` / `mypy`
commands shown under Test Output; quality eval `python
benchmarks/text_crusher_quality_eval.py /tmp/squad_dev.json`.
- Observed result: 501 Python + 3 Rust tests pass; ruff + mypy clean on
changed/new modules. Quality eval: TextCrusher keeps ~94% of buried
SQuAD answers at 30% size vs ~36% truncate/random; self-contained speed
run ~333k words in ~76ms (one O(n) pass) — sub-second where ModernBERT
takes minutes (fast-vs-slow contrast, not a same-input run).
- Not tested: Phase 3 off-path on live traffic; multi-worker
(per-process by design — see Additional Notes).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- **All four features are off by default and fail-open** — with the env
flags unset the paths are no-ops for realistic inputs; on any error the
request is forwarded (compressed if possible, else verbatim), never
dropped. A full background queue / duplicate key surfaces as
`deferred:dropped`.
- **Known limits (documented in `background_compression.py`):** Phase 3
is per-process, in-memory, and token-mode-only — these are
**lost-savings, never lost-correctness**, and consistent with the
project's existing per-process compression cache + sticky-session
multi-worker model. The startup multi-worker warning now names off-path
background compression.
- Phase 2 reuses the existing BM25 scorer; reuse did not improve
answer-retention over a Python prototype (query-awareness dominates) —
its value is the Rust speed + repo-conventional Rust-core/Python-shim
shape.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 10:48:06 -05:00
14e8dc4c84 feat(learn): weight loops in Headroom Learn + RTK-loop eval (#1160)
## Description

`headroom learn` ranked recommendations by a single LLM-guessed
`estimated_tokens_saved` with a flat hardcoded `confidence`, and had
**no notion of a loop**. So (1) RTK re-fetch loops were invisible - RTK
truncates a command's output, the agent re-runs larger-limit variants,
those calls *succeed* (`is_error=False`), and `analyze()` even
early-returned when a session had no failures and no events - and (2)
even when surfaced, a loop ranked no higher than a one-off mistake. This
adds loop-aware weighting plus the eval that reproduces an RTK loop,
runs it through Learn, and checks the guardrail prevents re-triggering.

Closes #1159

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `headroom/learn/loops.py`: `detect_loops()` (canonical signature
collapses RTK pagination/limit variants; classifies error vs rtk-refetch
loops; **measured** wasted tokens), `format_loops_for_digest()`,
`apply_loop_weighting()`.
- `analyzer.py`: detect loops up front (fixes the no-failure
early-return), lead the digest with them, prioritize loops in the system
prompt, re-sort after weighting.
- `models.py`: `Recommendation.is_loop_guardrail` / `loop_occurrences`.
- `benchmarks/rtk_loop_learn_eval.py` + `headroom/learn/fixtures.py`:
the two-phase RTK-loop eval and its session fixtures.
- Tests, `docs/rtk-loop-weighting.md`, CHANGELOG entry.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) - not run (mypy not in my
minimal env; see Not tested)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_learn/ -q
190 passed, 3 skipped, 1 warning in 5.85s
$ ruff check <changed files>
All checks passed!
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.0), Python 3.10.18, fresh venv (`pip
install -e` minus the optional `hnswlib`/proxy extras, which are
unrelated to `learn`); real LLM via the analyzer's claude CLI backend
(`HEADROOM_LEARN_CLI=claude`, claude-cli 2.1.158) — no API key used.
- Exact command / steps: `HEADROOM_LEARN_CLI=claude python -c "from
benchmarks.rtk_loop_learn_eval import run_eval;
c=run_eval(use_real_llm=True); print(c.render())"`
- Observed result: the analyzer shelled out to a real model and produced
the "Commands" guardrail quoted below, naming the looping command. The
digest reports the measured 5,005-token waste and asks the model to rank
loops first, so the model emitted that figure; in this run the guardrail
ranked **#1** and the scorecard was all-PASS (below). Caveat — real-mode
is run-dependent: the rule's wording, and whether the post-hoc
`apply_loop_weighting` fuzzy match fires, vary across runs (in one run
it did not tag the rule). The **deterministic CI eval** (stub LLM) is
the stable, reproducible artifact; this real run corroborates it.
- Not tested: the analyzer's API-key path (ANTHROPIC/OPENAI/GEMINI) —
exercised the equivalent claude CLI backend instead; `mypy`; a live
agent *obeying* the written rule end-to-end (Phase 2 is a non-recurrence
check, not a live agent — called out in the doc).

Real model output from this run, ranked #1 at the measured 5,005-token
weight:

> **Commands** — When grepping logs (or any large file), never loop with
increasing `| head -N` limits — tool output is capped at ~4 KB
regardless of N, so repeated attempts return identical bytes. Instead:
redirect to a temp file (`grep ... > /tmp/out.txt`) then read it, or use
`grep -c` first…

```text
[PASS] loop_detected          (1 loop(s), ~5,005 tok wasted)
[PASS] guardrail_produced
[PASS] ranked_first
[PASS] names_command
[PASS] prescribes_fix
[PASS] weight_reflects_waste
[PASS] guardrail_holds
RESULT: PASS
```

(One real-mode run via the claude CLI backend. The deterministic
`pytest` eval above is the stable artifact; see the run-dependence
caveat under Observed result.)

The real run also caught an over-brittle check: an earlier
`names_command` required the literal "TimeoutError"; the real model
wrote a *more general* rule (grep + `head -N`) without it, so I fixed
the check to verify the looping **command** is named, not an incidental
literal.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- No new dependencies. No network, no user/assistant content dropped —
operates on already-captured session digests.
- Kept as one logical change. mypy not run locally (minimal env); happy
to address anything CI's mypy flags.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-06-22 18:49:08 -05:00
chopratejas e190544c77 fix: B2 — live-zone block dispatcher skeleton
Phase B step 2 of the live-zone-only realignment. Replaces PR-A1's
unconditional "passthrough" stub with a real dispatcher that
inspects the Anthropic /v1/messages body, identifies the live zone
(latest user message at index >= frozen_message_count), and routes
each block to a per-type compressor. PR-B2 wires every per-type
compressor to a no-op, so the dispatcher returns
LiveZoneOutcome::NoChange on every call — bytes-in == bytes-out.
PR-B3+ replaces the no-ops with SmartCrusher, Log, Search, Diff,
and Code compressors.

Adds:
- crates/headroom-core/src/transforms/live_zone.rs — public API:
  - `compress_live_zone(body, frozen_message_count, AuthMode)`
  - `LiveZoneOutcome::{NoChange, Modified}`
  - `CompressionManifest` with per-block outcomes (message_index,
    block_index, block_type, BlockAction).
  - `BlockAction::{NoOpSkeleton, Excluded { reason }}`. The
    HOT_ZONE_BLOCK_TYPES list (`tool_use`, `thinking`,
    `redacted_thinking`, `compaction`) excludes blocks even when
    they appear in the latest user message.
  - `AuthMode::{Payg, OAuth, Subscription}` — accepted but unused
    in B2; PR-F2 wires the auth-mode gate.
  - 12 unit tests pin: empty messages, no messages field, invalid
    JSON, latest user message selection, frozen_count respect,
    hot-zone block exclusion, string-shaped content, no user msg
    in live zone, AuthMode no-op, NoChange contract, manifest
    counters, frozen-count clamping.

- crates/headroom-proxy/src/compression/live_zone_anthropic.rs —
  new entry point. `compress_anthropic_request` parses the body,
  resolves frozen_count via `resolve_frozen_count` (PR-A4 helper),
  dispatches via `compress_live_zone`, and returns
  `Outcome::NoCompression` on PR-B2 success / `Outcome::Passthrough
  { reason: NotJson | NoMessages | ModeOff }` on body-shape /
  policy issues. Six unit tests pin: mode_off short-circuit, no
  messages field, invalid JSON, valid body NoCompression,
  empty body, cache_control disabled.

Modifies:
- compression/mod.rs — re-exports `compress_anthropic_request` from
  `live_zone_anthropic` instead of `anthropic`. The old anthropic
  module is reduced to the `resolve_frozen_count` helper only
  (not deleted, because its CacheControlAutoFrozen-policy gate is
  reused).
- proxy.rs — passes `state.config.cache_control_auto_frozen` into
  the dispatcher. Drops the obsolete "live_zone reserved for
  Phase B" warning that PR-A1 emitted on every request.
- compression/anthropic.rs — pruned to the resolve_frozen_count
  helper plus its tests. The PR-A1 passthrough stub
  `compress_anthropic_request` is gone (live_zone_anthropic owns
  the name now).
- config.rs — `compression_mode` doc updated to reflect the wired
  dispatcher (no longer "reserved for Phase B").
- tests/integration_compression.rs — `compression_decision_logged`
  pins the new log contract (`decision="no_change"`,
  `reason="no_op_skeleton_pr_b2"`, plus manifest fields
  `frozen_message_count`, `messages_total`, `live_zone_blocks`).
  Asserts the obsolete Phase A warning is NOT emitted.
- proxy.rs no longer imports CompressionMode (only used inside the
  retired warning).

Benchmark cleanup (B1 leftovers that surfaced now):
- benchmarks/proxy_mode_benchmark.py + claude_session_mode_benchmark.py:
  drop `intelligent_context=False` arg from ProxyConfig (the field
  was retired in B1; tests/test_proxy_mode_benchmark.py and
  tests/test_claude_session_mode_benchmark.py imported these
  factories and started failing).
- benchmarks/bench_transforms.py: delete TestRollingWindowBenchmarks
  class; rewire TestTransformPipelineBenchmarks fixture without
  RollingWindow.
- benchmarks/conftest.py: drop rolling_window_config fixture.
- benchmarks/run_benchmarks.py: drop the `window` suite + table
  rows referencing RollingWindow.

Cache-safety invariant:
- PR-B2 dispatcher never mutates body bytes (no-op skeleton). The
  proxy forwards the original buffered bytes byte-equal. Phase A's
  SHA-256 fixtures pin this.
- `passthrough_mode_live_zone_currently_passthrough_byte_equal_sha256`
  retitled comment to reflect the dispatcher being live but
  no-op.

Acceptance:
- cargo build --workspace + clippy + fmt: green.
- cargo test --workspace --exclude headroom-py: all green
  (777 + 12 new live_zone + 6 new live_zone_anthropic tests).
- pytest: 4678 passed, 240 skipped, 0 failed.
- Anthropic decision log includes manifest fields per the
  observability contract documented in
  REALIGNMENT/02-architecture.md.

Per-PR-B2 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 12:45:43 -07:00
chopratejas 0161cdb386 chore(transforms): retire dead text_compressor module (Phase 3e.3)
`headroom/transforms/text_compressor.py` was a regex-line-sampling
fallback that nothing in the runtime called. ContentRouter routes
`CompressionStrategy.TEXT` straight to the Kompress ML compressor at
`content_router.py:1046` — the comment there literally says 'Prefer
Kompress ML compressor for text'. The Python file was orphaned but
still imported by its own test class, making it look live in the 3e
queue.

Drops the 3e.3 port from the queue: there's nothing to port.

# Removed
* `headroom/transforms/text_compressor.py` (255 LOC, unused)
* `tests/test_text_compressors.py::TestTextCompressor` (3 tests)
* `text_compressor` mention in `error_detection.py` shim docstring
* `text_compressor` mention in `test_signals_keyword_parity.py` docstring
* `TextCompressor` mention in `bench_latency.py` scenario comment

# Kept (defensive)
The legacy marker regex in `ccr/tool_injection.py:213` stays — it
parses an even older TextCompressor output format (pre-2026), is
purely defensive, and removal buys nothing. Test references to that
format in `test_ccr_tool_injection.py` document the regex contract
and stay too.

# Test plan
* `make ci-precheck` clean
* `tests/test_text_compressors.py` 19 passes (was 22, dropped 3)
2026-04-29 21:14:36 -07:00
rrubayet321 2ae71fe44d chore: add nosec B324 annotations to non-cryptographic MD5 usages and update temporary database path to use system temp directory 2026-04-07 13:07:26 +06:00
JerrettDavisandCopilot 6f701033a1 test: align anthropic cache stability fixtures
Sync the Anthropic cache stability test double with the prefix tracker contract used by the handler.

Format the benchmark scripts that were failing ruff format --check in CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-06 23:26:51 -05:00
JerrettDavis 8e4d7759de Harden cache validation reporting and TTL analysis 2026-04-06 20:11:21 -05:00
JerrettDavis ca7384402b Fix review regressions in compare, wrap, and Docker 2026-04-05 17:12:48 -05:00
JerrettDavis 5ffa77f4a3 Harden anthropic cache-mode replay stability 2026-04-05 16:11:39 -05:00
JerrettDavis d00c6739e1 Fix CI regressions for cache benchmark work 2026-04-04 22:33:44 -05:00
JerrettDavis 96fd3d9652 Add Anthropic cache-mode delta replay 2026-04-04 22:23:10 -05:00
JerrettDavis f2a32f9721 Harden Anthropic cache mode stability 2026-04-04 22:06:52 -05:00
JerrettDavis 1206c0ede2 Improve Claude cache simulation accounting 2026-04-04 21:54:45 -05:00
JerrettDavis 09829be0f4 Add memory-conscious Claude session benchmark harness 2026-04-04 21:19:40 -05:00
JerrettDavis 83b730f2b9 Fix CI lint/format failures after proxy mode hardening 2026-04-04 14:42:32 -05:00
JerrettDavis 54419ad8b8 Rebrand proxy modes to token/cache and harden cache-mode stability 2026-04-04 14:32:07 -05:00
chopratejas 3290a3d582 Remove LLMLingua: Kompress is the sole text compressor
LLMLingua was the original ML text compressor (BERT-based). Kompress
(ModernBERT, trained on 330K structured tool outputs) replaced it with
better compression quality and simpler architecture.

Removed across 35 files:
- Deleted headroom/transforms/llmlingua_compressor.py
- Deleted tests/test_transforms/test_llmlingua_compressor.py
- Deleted tests/test_proxy_llmlingua.py
- Removed all enable_llmlingua config, _get_llmlingua methods,
  LLMLingua fallback paths, LLMLINGUA strategy enum values
- Removed CLI flags, model configs, compression handler references
- Simplified ContentRouter: Kompress is primary and only text compressor
2026-03-26 11:11:00 -07:00
chopratejas 1474355f1c Format prefix_cache_benchmark.py for ruff 0.15 compatibility 2026-03-12 22:03:38 -07:00
chopratejas 8f438b0674 Introducing headroom wrap
- headroom wrap claude is the simplest way to start up claude
- It will also install rtk-ai locally
- rtk-ai is a cli wrapper that can save ~90% tokens for CLI calls made by Claude Code
2026-03-11 23:01:42 -07:00
chopratejas a2b2ec5463 Add OSS evaluation suite, universal JSON crush, latency benchmarks
Evaluation Suite:
- Tiered eval framework (Tier 1 ~$3/15min, Tier 2/3 for extended coverage)
- 16 benchmarks across 3 tiers: GSM8K, TruthfulQA, MMLU, ARC, HumanEval,
  SQuAD v2, BFCL, Tool Outputs, CCR needle retention, HotpotQA, and more
- Before/After runner with full proxy support (compression + CCR retrieval)
- LLM-as-judge for ground-truth comparison (BFCL function calling)
- Zero-cost compression-only runner (CCR needle retention, info retention)
- Cost tracker with per-model pricing and budget enforcement
- Report card generator (Markdown, JSON, HTML)
- Suite CLI: python -m headroom.evals suite --tier 1
- Fix BFCL dataset loader for current HuggingFace schema
- CI workflow: PR smoke test + weekly full Tier 1

Results: SQuAD 97%, BFCL 97%, Tool Outputs 100%, CCR 100%

SmartCrusher:
- Universal JSON crush for heterogeneous arrays
- Fix mypy redefinition warning in _crush_string_array

Other:
- Latency benchmark suite with docs
- Known limitations doc
- Prompt comparison evaluator
- Config updates for new features
2026-02-23 19:08:54 -08:00
chopratejas bd2d447c26 Add quality retention eval and fix linting for Python 3.12
- Add quality_retention_eval.py for needle-in-haystack testing to verify
  intelligent compression retains critical information (100% retention achieved)
- Add intelligent_context_integration_test.py for comprehensive pipeline testing
- Add test_progressive_summarizer.py with 36 tests for ProgressiveSummarizer
- Add HeadroomConfig parameter to HeadroomClient for direct config injection
- Update pipeline.py with IntelligentContextManager wiring and logging
- Fix all ruff linting issues and format for Python 3.12 compatibility
- Add comprehensive_eval.py benchmark for multi-scenario evaluation
- Add real_data_demo.py for production-scale volume testing
- Add reasoning agent test examples (groq, debug)
2026-01-19 21:52:18 -08:00
chopratejas 77248cd1a3 Enhance /stats endpoint and add comprehensive benchmarks
- Add detailed breakdowns by provider and model to /stats
- Include compression, telemetry, and feedback loop statistics
- Add latency.average_ms metric
- Add real-world agent benchmark with MCP tool patterns
- Add worst-case and adversarial benchmarks for edge cases
- Bump version to 0.2.12
2026-01-17 22:49:04 -08:00
chopratejas e4a41faa33 Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file
- Fix F401: Add noqa for availability check imports
- Fix F402: Rename loop variables shadowing imports
- Fix E722: Replace bare except with except Exception
- Fix B904: Add exception chaining (from e)
- Fix F811: Remove duplicate imports
- Fix B027: Add noqa for empty close() method
- Fix E741: Rename ambiguous variable l -> label
- Fix I001: Import sorting issues
- Apply ruff format to all 106 files

All 902 tests pass.
2026-01-10 15:33:44 -08:00
chopratejas 7a05808e0f Add cache optimization module with scalable dynamic content detection
Implements a comprehensive cache optimization layer for LLM providers:

- Provider-specific optimizers (Anthropic, OpenAI, Google) with distinct
  caching strategies: explicit breakpoints, prefix stabilization, and
  CachedContent API respectively

- Scalable dynamic content detector using three strategies:
  1. Structural detection: "Label: value" patterns (language-agnostic)
  2. Entropy-based detection: high-entropy strings (IDs, tokens, hashes)
  3. Universal patterns: ISO 8601, UUIDs, JWTs, hex hashes

- NO hardcoded locale-specific patterns (no month names, etc.)

- Semantic caching layer with LRU eviction and TTL support

- Plugin registry for provider selection and custom optimizers

- 131 tests, real-world benchmarks showing 20-55% compression at <0.3ms
2026-01-07 14:07:49 -08:00
chopratejas 175746cc26 Prepare for OSS release v0.2.0
This commit prepares Headroom for public open source release with
comprehensive documentation, licensing, and community infrastructure.

License & Legal:
- Add Apache 2.0 LICENSE file
- Add NOTICE file with third-party attributions
- Add SECURITY.md for vulnerability reporting

Community:
- Add CONTRIBUTING.md with contribution guidelines
- Add CODE_OF_CONDUCT.md (Contributor Covenant)
- Add GitHub issue templates (bug report, feature request)
- Add pull request template

Documentation:
- Update README.md with compelling value proposition
- Add docs/getting-started.md
- Add docs/proxy.md for proxy server documentation
- Add docs/transforms.md for transform reference
- Add docs/api.md for API reference
- Add examples/README.md

Package Infrastructure:
- Add headroom/py.typed for PEP 561 compliance
- Add headroom/cli.py for CLI entry point
- Add .github/workflows/ci.yml for CI pipeline
- Add .github/workflows/publish.yml for PyPI publishing
- Update pyproject.toml with proper metadata

New Features:
- Add multi-provider support (Google, Cohere, LiteLLM, OpenAI-compatible)
- Add universal tokenizer registry with multiple backends
- Add model registry with pricing and context limits
- Add production proxy server with caching and rate limiting

Code Quality:
- Fix 83 lint issues via ruff auto-fix
- Fix version consistency (benchmarks 0.1.0 → 0.2.0)
- Add skip decorators for optional dependency tests
2026-01-07 11:36:44 -08:00
chopratejas 9c7d4512d6 Initial commit: Headroom SDK - LLM context optimization toolkit
A comprehensive SDK for optimizing LLM context windows, reducing token
usage while preserving critical information for AI agents.

Core Features:
- SmartCrusher: Statistical compression of tool outputs (70-85% reduction)
- CacheAligner: Prefix optimization for prompt cache hits
- RollingWindow: Intelligent context window management
- BM25/Hybrid relevance scoring for smart item selection

Integrations:
- OpenAI and Anthropic provider support
- LangChain integration (ChatModel, Callbacks, Runnable)
- MCP (Model Context Protocol) integration for tool compression

Test Coverage:
- 372 tests passing across all modules
- 35 performance benchmarks
- Real-world agent evaluations with 88% token savings

Key Components:
- headroom/transforms/: Core compression transforms
- headroom/providers/: OpenAI and Anthropic support
- headroom/integrations/: LangChain and MCP integrations
- headroom/relevance/: BM25 and hybrid scoring
- headroom/pricing/: Model pricing registry
- benchmarks/: Performance benchmark suite
- examples/: Usage examples and demos
2026-01-06 23:16:58 -08:00