Commit Graph
14 Commits
Author SHA1 Message Date
Rahul selvarajandJD Davis 398d7cdcf4 fix(ccr): only refuse a bare marker, not originals that start and end with markers (#4626)
## Description

The bare-marker guard added for #2694 in `CompressionStore.store()`
checks
`startswith("<<ccr:") and endswith(">>")` on a single line. That also
matches
an original that starts with one marker and ends with a different one,
e.g.
`<<ccr:abcd>> Keep this original instruction and the next reference
<<ccr:ef01>>`.
Such an original is refused, an error is logged, and `retrieve(hash)`
returns
`None`.

The guard is now a full match of exactly one marker (`<<ccr:[^>\n]+>>`
on the
stripped value). A bare marker, with or without surrounding whitespace,
is
still refused and still cannot overwrite an existing entry. Originals
that
merely contain markers are stored, as the existing comment already
intends.

Closes #4266

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/compression_store.py`: new module-level
`_BARE_CCR_MARKER_RE`;
the guard in `store()` uses `fullmatch` on it. The character class
excludes
  `>` and newline, so two adjacent markers (`<<ccr:a>><<ccr:b>>`) or a
multi-line value are not a bare marker, matching the old single-line
intent.
  Marker shapes with commas or spaces (`<<ccr:HASH,KIND,SIZE>>`,
  `<<ccr:HASH N_rows_offloaded>>`) are still recognised as bare. Comment
  updated to mention #4266.
- `tests/test_compression_store.py`: two parametrized regression tests
in
  `TestCompressionStoreRetrieve`:
- three originals that start and end with separate markers are stored
and
    retrieved exactly (all three fail on `main`);
  - three bare markers (plain, whitespace-padded `,base64,` form, and
`N_rows_offloaded` form) are refused and do not overwrite an existing
    entry under the same `explicit_hash`.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

I ran the store tests and the CCR-related suites, not the full `pytest`
run
(see Not tested). The one failure below is the same on `main` without
this
change: the source checkout has no compiled `headroom._core`.

### Test Output

```text
# Before the fix (main fe69f381), new tests only
$ pytest tests/test_compression_store.py -k "separate_markers or bare_marker"
3 failed, 3 passed, 88 deselected

# After the fix
$ pytest tests/test_compression_store.py -k "separate_markers or bare_marker"
6 passed, 88 deselected

$ pytest tests/test_compression_store.py
94 passed

$ pytest tests/test_compression_store.py tests/test_ccr.py tests/test_ccr_batch_store.py \
    tests/test_ccr_marker_resolution.py tests/test_ccr_row_drop_store_bridge.py \
    tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms/test_smart_crusher_ccr_roundtrip.py
1 failed, 127 passed, 2 skipped
FAILED tests/test_ccr.py::TestCCREdgeCases::test_ccr_disabled_no_caching - ModuleNotFoundError: No module named 'headroom._core'
(same failure on main: 1 failed, 121 passed, 2 skipped)

$ ruff check .
All checks passed!
$ ruff format --check .
1864 files already formatted
$ mypy headroom
Success: no issues found in 579 source files
```

## Real Behavior Proof

- Environment: Linux, Python 3.12.3, venv with `headroom-ai==0.41.0`
  dependencies, repo source on `PYTHONPATH`, ruff 0.17.2, mypy 2.4.1.
- Exact command / steps: with the real in-memory `CompressionStore`:

  ```python
  from headroom.cache.compression_store import CompressionStore
  s = CompressionStore()
for o in ["<<ccr:abcd>> Keep this original instruction and the next
reference <<ccr:ef01>>", "<<ccr:abcd>><<ccr:ef01>>"]:
      h = s.store(original=o, compressed="x"); e = s.retrieve(h)
print(repr(o), "->", None if e is None else repr(e.original_content))
h = s.store(original="real bytes", compressed="x", explicit_hash="abcd")
s.store(original=" <<ccr:abcd>> ", compressed="x", explicit_hash="abcd")
print("bare marker over existing entry ->",
repr(s.retrieve(h).original_content))
  ```

- Observed result:

  ```text
  ## main fe69f381
CCR store: refusing to persist a bare retrieval marker as
original_content (hash=4d9d07471e6dab57de91d06a tool=None strategy=None
len=79) — ...
CCR store: refusing to persist a bare retrieval marker as
original_content (hash=7c324546fd0901418cd74797 tool=None strategy=None
len=24) — ...
CCR store: refusing to persist a bare retrieval marker as
original_content (hash=abcd tool=None strategy=None len=12) — ...
'<<ccr:abcd>> Keep this original instruction and the next reference
<<ccr:ef01>>' -> None
  '<<ccr:abcd>><<ccr:ef01>>' -> None
  bare marker over existing entry -> 'real bytes'

  ## this branch
CCR store: refusing to persist a bare retrieval marker as
original_content (hash=abcd tool=None strategy=None len=12) — ...
'<<ccr:abcd>> Keep this original instruction and the next reference
<<ccr:ef01>>' -> '<<ccr:abcd>> Keep this original instruction and the
next reference <<ccr:ef01>>'
  '<<ccr:abcd>><<ccr:ef01>>' -> '<<ccr:abcd>><<ccr:ef01>>'
  bare marker over existing entry -> 'real bytes'
  ```

- Not tested: the full `pytest` suite, the SQLite/Redis backends, and a
  running proxy. I could not build the Rust extension here (no
cargo/maturin), so tests that need `headroom._core` from source were not
  run against this change.

## Runtime Rollout Safety

- Rollout-managed feature(s): N/A — input guard in
`CompressionStore.store()` only, no rollout-managed feature.
- Minimum rollout channel: N/A
- Stable/default behavior changed: only that originals starting and
ending with two different markers are now stored instead of refused.
- Kill switch / disable path: N/A
- Unsafe override required: N/A
- Qualification impact: N/A
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, no UI change.

## Additional Notes

- Documentation: N/A, no user-facing docs describe this guard.
- "New and existing unit tests pass locally" is unticked because of the
one
pre-existing `headroom._core` failure above, which also fails on `main`
in
  this environment.

## AI Assistance

AI assistance: this change was prepared with Claude (AI coding
assistant) and tested as described above.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4626"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `8b32b57aea436c7cc8138bfa7b5cab7319901d99`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Clean mechanical merge of current upstream main. The production delta
remains the single full-match guard for exactly one bare CCR marker.
Originals containing separate/adjacent markers are retained; single bare
markers cannot overwrite an existing original. The source and test
integration has no new finding. All 157 focused
compression-store/buffered-OpenAI/marker-resolution tests pass, plus ten
shared Gemini request-fixture cases after the test-only repair for
independently verified main failures. Changed-file Ruff/format and
main-relative whitespace checks pass.

Independent Windows/Python 3.12 filesystem proof uses real
CompressionStore and SQLiteBackend, a fresh backend instance over the
same database, three compound originals and three bare-marker forms.
This PR persists and retrieves all compound originals exactly; actual
main e4a rejects all three. On both trees, all bare-marker forms
preserve the previously stored original. No mocks, live provider
requests or production service restart are used. This does not qualify
concurrent writers or every marker shape. The existing native binary was
supplied only for endpoint-test startup, not as a fresh native-build
claim.

To reproduce, save the program below and run `python marker_runtime.py
ABSOLUTE_PR_CHECKOUT pr`, then `python marker_runtime.py
ABSOLUTE_MAIN_CHECKOUT main` against main
e4a60ed72f. Replace its owned scratch-root
path with an existing local scratch directory on another machine. It
closes its real database connections and cleans the temporary databases.


<details><summary>Reproduction program</summary>

```python
from contextlib import ExitStack
import json
from pathlib import Path
import sys
import tempfile

sys.path.insert(0,sys.argv[1])
from headroom.cache.compression_store import CompressionStore
from headroom.cache.backends.sqlite import SQLiteBackend

owned=Path('G:/git/headroom/.codex').resolve()
compound=['<<ccr:abcd>> Keep this instruction <<ccr:ef01>>','<<ccr:abcd>><<ccr:ef01>>','<<ccr:abcd,base64,2.0KB>> then <<ccr:ef01 5_rows_offloaded>>']
bare=['<<ccr:abcd>>','  <<ccr:abcd,base64,2.0KB>>\n','<<ccr:abcd 5_rows_offloaded>>']
results=[]
with tempfile.TemporaryDirectory(prefix='pr4626-runtime-',dir=owned) as directory:
    root=Path(directory).resolve();assert root.is_relative_to(owned)
    with ExitStack() as close:
        def store_at(path):
            backend=SQLiteBackend(path)
            if backend._conn is not None:close.callback(backend._conn.close)
            return CompressionStore(backend=backend,enable_feedback=False)
        for i,original in enumerate(compound):
            path=root/f'compound-{i}.db'
            store=store_at(path)
            key=store.store(original,'x')
            reopened=store_at(path)
            entry=reopened.retrieve(key)
            results.append({'kind':'compound','input':original,'persisted_exact':entry is not None and entry.original_content==original})
        for i,marker in enumerate(bare):
            path=root/f'bare-{i}.db'
            store=store_at(path)
            key=store.store('real original bytes','x',explicit_hash='abcd')
            store.store(marker,'x',explicit_hash='abcd')
            entry=store_at(path).retrieve(key)
            results.append({'kind':'bare','input':marker,'original_preserved':entry is not None and entry.original_content=='real original bytes'})
        assert all(r.get('original_preserved',True) for r in results)
        if sys.argv[2]=='pr':assert all(r.get('persisted_exact',True) for r in results)
        else:assert not any(r.get('persisted_exact',False) for r in results)
        print(json.dumps({'source':sys.argv[1],'results':results},indent=2))

```
</details>

<details><summary>Observed output: pr4626-e4a-runtime.log</summary>

```json
{
  "source": "G:/git/headroom-review4626",
  "results": [
    {
      "kind": "compound",
      "input": "<<ccr:abcd>> Keep this instruction <<ccr:ef01>>",
      "persisted_exact": true
    },
    {
      "kind": "compound",
      "input": "<<ccr:abcd>><<ccr:ef01>>",
      "persisted_exact": true
    },
    {
      "kind": "compound",
      "input": "<<ccr:abcd,base64,2.0KB>> then <<ccr:ef01 5_rows_offloaded>>",
      "persisted_exact": true
    },
    {
      "kind": "bare",
      "input": "<<ccr:abcd>>",
      "original_preserved": true
    },
    {
      "kind": "bare",
      "input": "  <<ccr:abcd,base64,2.0KB>>\n",
      "original_preserved": true
    },
    {
      "kind": "bare",
      "input": "<<ccr:abcd 5_rows_offloaded>>",
      "original_preserved": true
    }
  ]
}

```
</details>

<details><summary>Observed output: pr4626-e4a-main-control.log</summary>

```json
{
  "source": "G:/git/headroom-review-main4626",
  "results": [
    {
      "kind": "compound",
      "input": "<<ccr:abcd>> Keep this instruction <<ccr:ef01>>",
      "persisted_exact": false
    },
    {
      "kind": "compound",
      "input": "<<ccr:abcd>><<ccr:ef01>>",
      "persisted_exact": false
    },
    {
      "kind": "compound",
      "input": "<<ccr:abcd,base64,2.0KB>> then <<ccr:ef01 5_rows_offloaded>>",
      "persisted_exact": false
    },
    {
      "kind": "bare",
      "input": "<<ccr:abcd>>",
      "original_preserved": true
    },
    {
      "kind": "bare",
      "input": "  <<ccr:abcd,base64,2.0KB>>\n",
      "original_preserved": true
    },
    {
      "kind": "bare",
      "input": "<<ccr:abcd 5_rows_offloaded>>",
      "original_preserved": true
    }
  ]
}

```
</details>


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-05 10:47:52 -06:00
Omar Gerardo 2fed685e77 perf(cache): drop eager full-backend TTL sweep on every new-key store() (#3256)
## Description

Every new-key `CompressionStore.store()` previously called
`_clean_expired()` before
checking capacity. The default `SQLiteBackend.items()` implementation
selects and
deserializes every stored entry, so each recoverable compression store
paid for an
unthrottled O(n) scan.

This removes that eager scan while preserving mixed-TTL capacity
correctness. A lazy
expiration heap removes expired entries first, and the existing
creation-time heap still
evicts the oldest live entry when capacity remains full.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed the eager full-backend TTL sweep from the new-key store path.
- Added a lazy expiration heap alongside the creation-time eviction
heap.
- Preferred expired entries under capacity pressure without recording
success feedback.
- Kept both heaps bounded across replacement, read cleanup, backend-side
SQLite purges,
  and explicit clearing.
- Saturated oversized positive TTL expiration times to infinity.
- Rebuilt both heaps from one backend snapshot when stale-entry
thresholds are reached.
- Added regressions for no-scan stores, mixed TTLs, replacement, read
cleanup, oversized
  TTLs, SQLite purges, feedback behavior, and oldest-live eviction.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HF_HUB_OFFLINE=1 LITELLM_LOCAL_MODEL_COST_MAP=true \
  .venv/bin/python -m pytest tests/test_compression_store.py
80 passed

$ .venv/bin/ruff check headroom/cache/compression_store.py tests/test_compression_store.py
All checks passed!

$ .venv/bin/ruff format --check headroom/cache/compression_store.py tests/test_compression_store.py
2 files already formatted

$ .venv/bin/mypy headroom
Success: no issues found

$ make ci-precheck
ci-precheck PASSED - safe to push
```

The no-scan regression was also run against the parent behavior. It
failed with five
backend scans for five stores and passes with zero scans after this
change.

## Real Behavior Proof

- Environment: macOS 26.6, Python 3.13.13, SQLite 3.51.0, branch
  `fix/ccr-store-ttl-scan` at `4cf29b81`.
- Exact command / steps: Seeded a fresh SQLite CCR database with 2,000
entries containing
512-byte payloads, then timed 50 new-key `store()` calls before and
after the eager-scan
removal. Separately ran the mixed-TTL regression with capacity two:
store an older
long-lived entry, store a newer short-lived entry, advance beyond the
short TTL, then
  store a third entry.
- Observed result: Parent behavior took 0.684 seconds total (13.67
ms/store); the patched
path took 0.001 seconds total (0.03 ms/store). Under mixed-TTL pressure,
the expired
newer entry is removed and the older live entry remains. The targeted
cache suite has
  80 passing tests.
- Not tested: Windows or Linux runtime benchmarks, multi-process write
contention,
  third-party backends, or database sizes above 2,000 rows.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Yes. New-key CCR stores no longer
perform a full TTL
sweep; expired entries are instead reconciled lazily through the
expiration heap.
- Kill switch / disable path: None specific to this optimization.
- Unsafe override required: No.
- Qualification impact: No public API, configuration, schema, or
feature-qualification
  changes.
- Rollback path: Revert commits `4cf29b81` and `506a8972`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my
  Conventional Commit PR title (a CI guard enforces this)

## Screenshots (if applicable)

N/A.

## Additional Notes

Documentation is unchanged because this is an internal cache
implementation fix with no
public API or configuration changes.
2026-09-25 10:28:06 -05:00
5cb87bc351 fix(cache): default CCR payload previews off; harden the log when they are on (#3728)
## A-6 — tool-result content was written to the runtime log by default
(S2)

### RELEASE NOTE — a default changes, and operators lose log content
they may rely on

`HEADROOM_LOG_PAYLOAD_PREVIEW` **now defaults to off.** Operator-visible
impact:

- Each `event=headroom_retrieve` line in
`~/.headroom/logs/proxy-<port>.log` now carries
`"payload_preview":""` and `"payload_preview_chars":0` instead of up to
4096 characters
of the retrieved payload. `payload_chars` (the byte count) is unchanged,
so anything
measuring retrieval size still works; anything grepping the log for
retrieved *content*
  will find nothing until the operator opts in.
- The opt-in is now **explicit and positive**:
`HEADROOM_LOG_PAYLOAD_PREVIEW=1|true|yes|on`.
Previously *any* value other than `0/false/no/off` — including a typo —
meant "on".
- **On Linux and macOS, every Headroom log file is now created `0600`**
(owner-only) — the
runtime log `~/.headroom/logs/proxy-<port>.log`, each of its rotated
backups
(`proxy-8000.log.1`, …), and the `--log-file` JSONL request log. This is
**unconditional**:
it does not depend on `HEADROOM_LOG_PAYLOAD_PREVIEW`, `--log-messages`,
or any other
switch. **A log shipper, sidecar, or collector that reads these files as
a different user
or group will stop being able to read them.** Run the collector as the
same user, or copy
the file out with a privileged job. On Windows nothing changes — see the
Windows note
  below.
- If the runtime log path already exists as a **symlink**, Headroom now
refuses to write the
log at all (with a warning naming the path) instead of writing through
it. An operator who
deliberately symlinks `~/.headroom/logs/proxy-<port>.log` onto another
volume must replace
  it with a real path or point `HEADROOM_WORKSPACE_DIR` at that volume.

### The bug

`_payload_preview_enabled()` returned `True` for an unset variable, so a
stock proxy wrote
up to 4 KB of the retrieved tool-result payload into the always-on
runtime log on every CCR
retrieval. That payload is verbatim tool output: source code, customer
data, and any
credential the three redaction regexes (`*_KEY|TOKEN|SECRET|...`,
`Bearer`/`Basic`, `sk-…`)
do not happen to match. The file was created at the process umask —
`0644` on a normal
developer machine.

**The claim held**, at `headroom/cache/compression_store.py:59`/`:116`
on current `main`
(line numbers unchanged since `89a58fd1`). Demonstrated by the
fail-before run below, where
the new test found the string `ssn=123-45-6789` sitting in a real log
file, mode `0644`.

### The fix

- `headroom/cache/compression_store.py` — `_payload_preview_enabled()`
defaults to `False`
and now requires an affirmative `1/true/yes/on`. The redaction pass is
untouched; it is
now a second line of defence behind an explicit opt-in rather than the
only one.
- `headroom/fileperms.py` (new) — owner-only file creation, and an
explicit
`OWNER_ONLY_SUPPORTED` flag recording *where the mode bits actually
decide read access*.
- `headroom/proxy/helpers.py` — `_OwnerOnlyRotatingFileHandler`, now
used by
`_setup_file_logging` **unconditionally**. `logging` opens the stream
itself at
construction *and on every rollover* with no mode hook, so `_open` is
replaced outright:
the file is opened `O_CREAT|O_APPEND|O_WRONLY|O_NOFOLLOW` at `0600` and
tightened with
`fchmod` on that same descriptor (so an existing `0644` log is fixed,
and the mode cannot
land on a different inode). `rotate()` re-applies the mode to each
backup, and backups
  left `0644` by an older build are tightened when the handler opens.
- `headroom/proxy/request_logger.py` — the `--log-file` JSONL, which
carries whole request
  and response bodies under `--log-messages`, is created the same way.
- `docs/content/docs/proxy.mdx` — the variable was entirely
undocumented; it now has a row
in **Observability**, plus a **Who can read the logs** subsection
stating the POSIX
  guarantee and the Windows non-guarantee in as many words.

### Fail-before / pass-after

Method: commit the work, `git checkout origin/main --
headroom/cache/compression_store.py
headroom/proxy/helpers.py` keeping the new tests, run, then `git
checkout HEAD -- <those
files>`.

**Before (sources reverted to `origin/main`, new tests kept):**

```
__________ test_runtime_log_holds_no_payload_text_at_default_settings __________
tests/test_cache_breakpoint_diagnostics.py:246: in test_runtime_log_holds_no_payload_text_at_default_settings
    assert secret not in text
E   assert 'BEGIN-CUSTO...USTOMER-DATA' not in '2026-09-23 ...MER-DATA"}\n'
E     'BEGIN-CUSTOMER-DAT...s END-CUSTOMER-DATA' is contained here:
E       preview":"BEGIN-CUSTOMER-DATA ssn=123-45-6789 def sekrit(): pass END-CUSTOMER-DATA"}

_____________ test_runtime_log_is_owner_only_when_preview_enabled ______________
tests/test_cache_breakpoint_diagnostics.py:257: in test_runtime_log_is_owner_only_when_preview_enabled
    assert stat.S_IMODE(log_path.stat().st_mode) == 0o600
E   AssertionError: assert 420 == 384        # 0o644 vs 0o600

____ test_runtime_log_hardening_survives_a_pre_existing_world_readable_log _____
E   AssertionError: assert 420 == 384

FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_disabled_by_default
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_stays_off_for_anything_else[]
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_stays_off_for_anything_else[maybe]
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_stays_off_for_anything_else[  ]
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_holds_no_payload_text_at_default_settings
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_is_owner_only_when_preview_enabled
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_hardening_survives_a_pre_existing_world_readable_log
========================= 7 failed, 17 passed in 1.59s =========================
```

That first failure is the finding itself: on `main`, customer data lands
in a real log file
under default settings.

**After (sources restored):**

```
tests/test_cache_breakpoint_diagnostics.py ........................      [ 24%]
tests/test_compression_store.py ............................................ [100%]
============================= 100 passed in 9.36s ==============================
```

`tests/test_perf_per_port_logs.py` (10 passed) covers the handler swap —
dedup, per-worker
paths, and sequential port changes still behave.

### Tests

New, in `tests/test_cache_breakpoint_diagnostics.py`:

- `test_payload_preview_disabled_by_default` — unset ⇒ byte counts only.
- `test_payload_preview_opt_in_values` /
`test_payload_preview_stays_off_for_anything_else`
— only an affirmative value enables it; a typo (`maybe`, `""`,
whitespace) does not.
- `test_runtime_log_holds_no_payload_text_at_default_settings` —
end-to-end through the
real `_setup_file_logging` handler: a store+retrieve leaves
`event=headroom_retrieve` in
  the file with the right `payload_chars` and **no** payload text.
- `test_runtime_log_is_owner_only_when_preview_enabled` — `0600` on the
created file.
-
`test_runtime_log_hardening_survives_a_pre_existing_world_readable_log`
— a `0644` log
from a pre-hardening run is chmodded, since `O_CREAT`'s mode does not
apply to it.

Updated: the two tests in `tests/test_compression_store.py` that
exercise the preview and
its redaction now opt in explicitly, and
`test_payload_preview_enabled_by_default` is
inverted to `test_payload_preview_disabled_by_default`.

### Notes

- Independent of #3726 (A-5) — both are based on `origin/main`, and they
touch disjoint
  parts of `compression_store.py`.
- A-5's audit found that the runtime log is one of the files
`--stateless` does *not*
suppress, so before this PR a stateless proxy was getting payload text
on disk too.
- `CHANGELOG.md` not touched (CI guard).


---

## Update after review (@JerrettDavis) — both blockers addressed

### 1. Windows: the guarantee is now scoped in code, not just in prose

You were right that this was a control that silently did nothing on a
supported platform.
On NTFS, read access is decided by the file's ACL; `os.chmod` only
toggles the read-only
attribute, so `chmod(0o600)` returns successfully while `stat.S_IMODE`
still reports
`0666` — which is exactly what your run showed.

I did **not** implement a Windows ACL. Python ships no ACL API; doing it
means ~150 lines
of `ctypes` building a `SECURITY_ATTRIBUTES` DACL, which I cannot
execute or test on this
machine, and I would rather not ship unverifiable security code than
ship an untested one.

So the guarantee is scoped honestly, and made loud rather than silent:

- `headroom/fileperms.OWNER_ONLY_SUPPORTED` is `os.name == "posix"`.
`restrict_fd_to_owner`
/ `restrict_path_to_owner` return `False` off POSIX rather than
reporting a protection
  they did not establish.
- `_setup_file_logging` emits **one warning per process** on such a
platform: *"Headroom
cannot create &lt;path&gt; owner-only on this platform: file modes do
not control read
access here, and Headroom does not set an ACL. The runtime log can
contain request and
response content (`--log-messages`, wire debug,
`HEADROOM_LOG_PAYLOAD_PREVIEW`) — protect
the log directory itself."* Logging still works; it just no longer
implies a protection.
- The docs say it in as many words, under **Who can read the logs**: on
Windows the log
inherits the ACL of its parent directory, and the directory is the
access-control
  boundary.
- The tests assert the real platform-conditional behaviour. The mode
assertions carry
`@_posix_only` with that reason;
`test_owner_only_support_matches_what_the_platform_can_enforce`
  asserts the flag equals `os.name == "posix"` **on every platform**;
`test_no_owner_only_claim_is_made_off_posix` runs *only* on Windows and
asserts the
  absence of the claim (the handler still creates the file;
  `restrict_path_to_owner` returns `False`); and
`test_unsupported_platform_says_so_rather_than_silently_not_protecting`
simulates the
unsupported platform on any host so the warning itself is covered by
Linux CI.

**In one line: on Linux and macOS the log is `0600`, enforced. On
Windows Headroom does not
restrict the log, says so at startup and in the docs, and no test claims
otherwise.**

### 2. The permissions no longer key off `HEADROOM_LOG_PAYLOAD_PREVIEW`

Agreed, and taken as stated: **every** Headroom runtime log is created
owner-only, rollover
files included, regardless of which content switch is on. Previews were
never the only
source of request content in that file — `--log-messages` bodies,
wire-debug dumps and
query logging land there too, each behind its own switch, so gating on
any one of them left
the rest writing a sensitive file at the umask. `_setup_file_logging`
now always installs
`_OwnerOnlyRotatingFileHandler` and no longer imports
`_payload_preview_enabled` at all.
I extended it to the `--log-file` JSONL as well, since under
`--log-messages` that is the
file with the whole bodies in it.

Previews still default off — that part of the PR is unchanged.

This **does** break log collection for an operator reading the file as
another user, which
is the trade-off the original PR body ducked. It is now called out
explicitly in the release
note at the top, with the two workarounds.

### Also fixed while in here (found in an internal pass)

- **Rotated backups.** `rotate()` re-applies `0600` to the destination.
`os.rename` happens
to carry the mode across, but a configured `rotator` need not, so the
invariant is
asserted on the result rather than assumed from how it got there.
Backups left `0644` by
  an older build are tightened when the handler opens.
- **`_open` followed symlinks.** The old code pre-created the file and
then delegated to
`super()._open()`, so both the `chmod` and the write followed a symlink
planted at the log
path — an attacker chose the destination *and* the permissions. `_open`
is now replaced
outright and opens with `O_NOFOLLOW` at `0600`, tightening with `fchmod`
on that same
descriptor (no TOCTOU window, and no way for the mode to land on another
inode). A log
path that is *already* a symlink is refused up front with a warning,
which is what carries
the refusal on platforms without `O_NOFOLLOW`. Deliberate decision:
**fail closed** — a
redirected sensitive log is worse than no log, and logging is already
best-effort here.

### Fail-before / pass-after for this round

Baseline is `ec014d64` (the commit you reviewed), not `origin/main`,
since these are new
findings on top of it. Method: commit, `git checkout ec014d64 --
headroom/proxy/helpers.py
headroom/proxy/request_logger.py headroom/cache/compression_store.py`
keeping the new
tests, run, restore.

```
FAILED test_runtime_log_is_owner_only_with_previews_off                       assert 420 == 384
FAILED test_runtime_log_hardening_survives_a_pre_existing_world_readable_log   assert 420 == 384
FAILED test_rotated_backups_are_owner_only                                     assert 420 == 384
FAILED test_pre_existing_backups_are_tightened_when_the_handler_opens          assert 420 == 384
FAILED test_runtime_log_refuses_a_symlinked_path       AssertionError: logging was wired up through the symlink
FAILED test_jsonl_request_log_is_owner_only                                    assert 420 == 384
FAILED test_unsupported_platform_says_so_rather_than_silently_not_protecting
    AttributeError: module 'headroom.proxy.helpers' has no attribute '_owner_only_warning_emitted'
=================== 7 failed, 25 passed, 1 skipped in 1.23s ====================
```

(`420` is `0o644`, `384` is `0o600`. A `predictable_umask` fixture pins
the umask to `0o022`
so "would have been world-readable" is not down to the runner's
environment.)

After, sources restored:

```
tests/test_cache_breakpoint_diagnostics.py tests/test_compression_store.py
tests/test_perf_per_port_logs.py tests/test_proxy/test_request_logger.py
tests/test_proxy_streaming_request_logger.py
======================= 132 passed, 1 skipped in 10.90s ========================
```

The 1 skip is `test_no_owner_only_claim_is_made_off_posix` — it is the
Windows-only half,
and it is the one I cannot run here.

Ruff 0.16.7 `check` and `format --check` are clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-24 13:09:06 +02:00
trs-80 3169935858 perf(ccr): bound retrieval history with a deque instead of reslicing (#3471)
## Description

Once the CCR retrieval history reaches its 1,000-event cap, every
further retrieval appended to the list and then rebuilt it from a
1,000-element slice, under the store lock.
`collections.deque(maxlen=1000)` drops the oldest event as the new one
lands, so the trim is deleted rather than moved.

This is a maintenance change that also removes an O(cap) copy. It is
**not** an end-to-end speedup, and the measurements below say so: the
history append itself gets ~3.9× cheaper, while the full `retrieve()`
path is unchanged within noise.

`_pending_feedback_events` deliberately stays an unbounded list. It is a
separate drain-by-swap queue that still owes a notification for every
event, including ones the capped history has already discarded, so both
containers now carry a comment saying which is which.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/compression_store.py`: `_retrieval_events` becomes
`deque(maxlen=self._max_events)`; the append-then-reslice in
`_log_retrieval` is removed. `_max_events` stays as the single readable
name for the cap — it is read at construction, by the new tests and by
the benchmark, and is never assigned after `__init__`.
- `tests/test_compression_store.py`: two tests — retention and ordering
at the cap, and one feedback notification per retrieval past the cap.
- `benchmarks/ccr_retrieval_history_benchmark.py`: standalone script
measuring the history append at steady state, following the convention
the other benchmarks in that directory use.

Consumer audit before changing anything: the container is only ever hit
with `len`, `append`, `clear`, iteration, `list()` and `sys.getsizeof` —
all deque-safe. The one slice, `events_copy[-limit:]` in
`get_retrieval_events`, operates on the list copy taken under the lock,
so deque's lack of slice support is never reached. Repo-wide there is no
non-Python consumer, no subclass, no pickling or deepcopy path and no
`isinstance(list)` check, and the deleted line was the only assignment
to the attribute — so nothing can rebind an unbounded list back onto it.
One `threading.Lock` guards every access.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Both new tests pass before and after this change, deliberately: for a
maintenance change, parity is the claim. They exist because the
pre-existing `test_retrieval_events_list_bounded` asserts only `len(...)
<= 1000`, which cannot tell which end was dropped. Mutation-checked: an
unbounded `deque()` fails the length assertion, and a head-trim keeping
the oldest 1,000 passes the length check but fails on order.

### Test Output

```text
$ pytest tests/test_compression_store.py tests/test_ccr.py tests/test_ccr_feedback.py \
         tests/test_critical_gaps.py tests/test_critical_fixes.py tests/test_proxy_ccr.py \
         tests/test_ccr_sqlite_backend.py tests/test_ccr_batch_store.py \
         tests/test_memory_tracker_integration.py -q
209 passed, 3 skipped in 16.66s

$ pytest tests/ -q -k "ccr or compression_store or compression_feedback or telemetry \
                       or critical_gaps or critical_fixes or memory_tracker" \
         --ignore=tests/test_integrations
1 failed, 921 passed, 11 skipped in 77.38s
# the one failure is test_ccr_marker_round_trip_live, a live-API test that fails
# identically on the unchanged base

$ python benchmarks/ccr_retrieval_history_benchmark.py
container: deque
cap: 1000 events
appends per run: 20000, runs: 7
median: 0.481 us/append

$ pre-commit run --files headroom/cache/compression_store.py \
      tests/test_compression_store.py benchmarks/ccr_retrieval_history_benchmark.py
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: macOS 15.6, Python 3.13.11, branch based on `main` @
`e67b3c8a`. Single process, single thread, no contention.
- Exact command / steps: filled a store's history to its cap so every
subsequent append must evict, then timed 20,000 `_log_retrieval` calls,
7 runs, median reported. Ran the unchanged tree and this change in both
orderings to rule out ordering effects. Separately timed the full public
`retrieve()` path, 2,000 calls × 5 runs, also in both orderings.
- Observed result: history append **1.81 to 0.47 us** and, with the
order reversed, **1.85 to 0.48 us** per append. Full `retrieve()`: 7.67
vs 7.89 us, and reversed 7.79 vs 7.82 us — unchanged within noise in
both directions. Behavior parity checked value by value:
`get_stats()["event_count"]`, `get_retrieval_events()`
contents/order/length across limits from negative through 1e9, and
`clear()` are all identical; `deque.clear()` preserves `maxlen`, so the
container stays bounded afterwards.
- Not tested: no multi-threaded or contended measurement, no
multi-worker or production workload, and no memory-over-time profile.
The benchmark times a private method (`_log_retrieval`) because the
public `retrieve()` buries the difference in backend I/O and the
feedback drain — that is stated in the benchmark's own docstring. The
result is a microbenchmark of one operation and must not be read as a
proxy for retrieval latency.

One value does change numerically, and it is not hidden:
`get_memory_stats().size_bytes` shifts by a few hundred bytes, because
`sys.getsizeof` reports a deque's 64-slot blocks rather than a list's
contiguous array (56 to 760 bytes empty; 8,056 to 8,680 at 1,000
entries). It is a shallow size that already excluded the events
themselves, `budget_bytes` is `None`, and no test or dashboard compares
it against anything.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. CCR retrieval-event history is not
rollout-gated.
- Minimum rollout channel: N/A — no rollout-managed feature is touched.
- Stable/default behavior changed: No. The cap, its size, retention
order, retrieval results, lock guarantees and feedback delivery are all
unchanged; only the container enforcing the cap differs, plus the
`size_bytes` gauge noted above.
- Kill switch / disable path: N/A — no flag. Reverting the commit
restores the list and its trim.
- Unsafe override required: No.
- Qualification impact: None. No compression decision, retrieval result,
CCR marker, TTL or persistence path is touched — this is the in-memory
display history only.
- Rollback path: `git revert` of this commit, single step. No schema,
data or config migration is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable — no user-visible surface changes.

## Additional Notes

- No separate documentation file needed updating; the two comments in
`__init__` are the documentation this needs, because the one thing a
future reader can get wrong here is bounding `_pending_feedback_events`
to match.
- Overlaps nothing in flight: PR #3465 touches this same module, but its
entire footprint is inside `_clean_expired`, which never reads the
retrieval history, and this change never touches `_stale_heap_entries`.
- Unrelated, left alone, noticed while running the suites:
`tests/test_realignment_live_multi_turn.py::test_ccr_marker_round_trip_live`
and
`tests/test_memory_usage_integration.py::TestProxyMemoryIntegration::test_real_api_calls_memory_tracking`
both need live API access and fail on a clean checkout without it.
2026-09-11 20:42:01 -07:00
Abhay SinghandTejas Chopra 113894600c fix(cache/ccr): don't evict a live entry on a duplicate store at capacity (#2082)
## Description

`CompressionStore.store` (`headroom/cache/compression_store.py`) runs
eviction **before** it knows
whether the incoming `hash_key` is new or a re-store of an
already-present key:

```python
with self._lock:
    self._evict_if_needed()          # <-- runs first

    existing = self._backend.get(hash_key)
    if existing is not None:
        ...                          # duplicate / collision: overwrite in place
        self._stale_heap_entries += 1
    self._backend.set(hash_key, entry)
```

When the store is full and the incoming key **already exists** (a
duplicate re-store),
`_evict_if_needed()` removes the oldest *distinct* entry to "make room"
— but then `set()` merely
overwrites the existing key in place, so no room was ever needed. Net
effect: `count` drops to
`max_entries - 1` and a **live, never-retrieved entry is destroyed**.
That entry's `<<ccr:...>>`
marker, still sitting in the conversation history, then resolves to a
404 on `/v1/retrieve`.

This is not a corner case: the CCR mirror bridge
(`_mirror_single_hash_to_python_store` in
`smart_crusher.py`) re-`store()`s the same `explicit_hash` every turn a
`<<ccr:…>>` marker is
re-encountered, and markers persist across turns — so a full store
silently deletes a live sibling
entry on each duplicate.

Concrete (with the repo's `max_entries=3` fixture): store c0, c1, c2
(full), then re-store c1
(same content ⇒ same hash). Eviction pops the oldest (c0), deletes it,
then c1 is overwritten in
place. Final state: {c1, c2}, count 2, and **c0 is gone** — its marker
is now unredeemable.

Closes: no issue filed — found while auditing the compression store.

## Fix

Decide novelty before evicting: only `_evict_if_needed()` for a
genuinely new key; a
duplicate/replace overwrites in place (no eviction). The
collision/duplicate logging and
stale-heap accounting are unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/cache/compression_store.py`: `store()` reads `existing`
first and only evicts when the key is new.
- `tests/test_compression_store.py`: add
`test_duplicate_store_at_capacity_does_not_evict` (re-store an existing
hash at capacity keeps all entries and count at `max_entries`).

## Testing

- [x] New regression test added (`tests/test_compression_store.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).

```text
$ uvx ruff@0.15.17 check headroom/cache/compression_store.py tests/test_compression_store.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the store/evict
logic with a dependency-free script (replicating the backend + eviction
heap) and left the full pytest to CI.
- Exact command / steps: filled a `max_entries=3` store with c0/c1/c2,
then re-stored c1 (duplicate), through the old (evict-first) and new
(check-first) logic; also confirmed a genuinely new key still evicts the
oldest.
- Observed result: the old logic drops c0 (count 2); the new keeps all
three; and a new key at capacity still evicts the oldest:

```text
OLD: after duplicate re-store of h1 -> keys=['h1', 'h2'] count=2
NEW: after duplicate re-store of h1 -> keys=['h0', 'h1', 'h2'] count=3
NEW still evicts oldest for a genuinely new key at capacity
DUPLICATE-STORE EVICTION FIX VERIFIED (old drops a live entry; new keeps it)
```

- Not tested: a full proxy CCR round-trip (needs the heavy stack). The
fix is confined to `store()` and the new test drives it directly with
the `max_entries=3` fixture. Existing eviction tests use distinct keys
and stay green. Full local `pytest` deferred to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Internal reordering only, no signature change; no call sites or
backend mocks break.
- @JerrettDavis tagging you — this silently drops a live CCR entry
(making its marker 404) whenever a duplicate hash is re-stored at
capacity, which the mirror bridge does routinely. Thanks.

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-13 09:46:24 -04:00
Tejas Chopra c2fc4d3753 fix(ccr): make headroom_retrieve a hash-only full-content lookup (#1532)
The optional `query` parameter on headroom_retrieve routed retrieval
through CompressionStore.search(), which BM25-scored the items inside a
single cached blob and dropped everything below a 0.3 relevance floor.
On small per-blob corpora with conversational queries this returned an
empty result the large majority of the time, so the LLM saw "nothing
found" for content that was actually present — pushing users to turn
compression off entirely.

Retrieval is fundamentally a hash lookup (this already matches the Rust
proxy's CCR store, which is put/get only — "no BM25 search"). Remove the
query/search path end to end and always return the full original
content:

Core (Python proxy):
- tool schemas (anthropic/openai/google) drop the `query` property
- parse_tool_call returns the hash (str | None) instead of (hash, query)
- response handler, proxy POST/GET/tool-call handlers, the MCP retrieve
tool, and the streaming feedback recorders retrieve by hash only
- proactive context-tracker expansion always restores full content
- delete CompressionStore.search() and its BM25 machinery (the bm25
module stays — it is still used by relevance/)
- CCRToolCall.query, CCRToolResult.was_search, and
ExpansionRecommendation.expand_full/search_query are removed

Plugins (advertised a now-defunct query param to the LLM):
- hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop
`query` from their schemas, signatures, request URLs, and tests

Benchmarks/docs:
- ccr_regression + adversarial benchmarks switch from store.search() to
full hash retrieval (search input-injection tests repurposed to the
hash, the only remaining input surface)
- wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx,
config.py and store docstrings updated to describe hash-only retrieval

Tests updated to assert full-content retrieval and guard the removed
surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean.

## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
2026-06-28 10:32:43 -07:00
Tejas Chopra b7be3814f1 feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description

A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### 1. Rust compressor extraction

- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.

### 2. CCR store hardening

- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).

### 3. Traffic audit tooling (measure before tuning)

- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.

### 4. Read maturation (Mechanism B) — experimental, default OFF

- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.

### 5. Rebase / CI fixups (this update)

- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s

$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed

$ mypy headroom/
Success: no issues found in 365 source files

$ python -m compileall headroom/ -q
COMPILE-OK

# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
#   "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```

## Real Behavior Proof

- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
7cb0f43b); GitHub Actions CI run 27488990477 for the test shards
- Exact command / steps: rebased onto latest main (clean, 13 commits
replayed, 0 conflicts); ran the pytest suites and mypy above locally;
inspected CI shard logs to confirm the failure was the codecov upload,
not the test phase
- Observed result: 253 targeted tests pass locally; mypy clean on 365
files; CI test phase reports `1528 passed, 120 skipped`; the only red
step (codecov `upload-coverage` → "Token required because branch is
protected") is resolved by the rebased-in #968 CODECOV_TOKEN fix
- Not tested: the read-maturation live-API no-bust validation
(`tests/test_live/`) was not re-run in this rebase pass (requires
provider keys); it was validated when the feature first landed, and no
maturation code changed in the rebase — only CCR-default test assertions
and the duplicate-field resolution

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

CHANGELOG is generated by release-please from the conventional commits,
so the CHANGELOG box is intentionally left unchecked. "Manual testing
performed" is unchecked deliberately — see `Real Behavior Proof` → `Not
tested` for the exact boundary (the live-API maturation validation was
not re-run in this rebase pass).

### Follow-ups (tracked, not in this PR)

- Mechanism B provider extensions: OpenAI-family wiring (no breakpoint
hold — bounded near-tail bust) and the Codex runtime read-detector (the
audit classifier is the prototype).
- Pilot enablement playbook: run `audit-reads --simulate-maturation` on
target traffic → pick `quiesce_turns` → enable via env → watch cache hit
rate + `read_maturation:N` transform tags.
2026-06-16 20:21:13 -07:00
Hc 2533f7703e fix(ccr): make retrieval TTL configurable (#715)
## Description

Make the CCR retrieval store TTL configurable so long-running agent jobs
can keep `headroom_retrieve` markers resolvable beyond the previous
hard-coded 300-second window.

Fixes #714

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `HEADROOM_CCR_TTL_SECONDS` for the global CCR `CompressionStore`
default TTL, with validation and fallback to 300 seconds.
- Expose the effective CCR TTL as `store.default_ttl_seconds` from
`/v1/retrieve/stats`.
- Distinguish missing vs expired CCR retrieval failures in
`/v1/retrieve`, `/v1/retrieve/{hash}`, tool-call handling, and CCR
response handling.
- Update CCR docs, README wording, and CHANGELOG for the user-facing TTL
behavior.
- Add regression tests for env-configured TTL, invalid env fallback,
explicit TTL precedence, stats exposure, and expired retrieval detail.

## Reproduction

Before this change, the proxy-global CCR store always used the default
300-second TTL when callers used `get_compression_store()` without an
explicit `default_ttl`. A long-running agent job could receive a
`<<ccr:...>>` marker and fail later with a generic not-found/expired
response after the fixed 5-minute retention window.

The new tests cover this by setting `HEADROOM_CCR_TTL_SECONDS`, creating
CCR entries through the same global store used by the proxy, and
asserting that stored entries and `/v1/retrieve/stats` use the
configured retention.

## Real behavior proof

Setup tested:

- macOS 15.7.2
- Python 3.13.9 via `uv run --frozen --extra dev --extra proxy`
- Local Headroom proxy subprocess: `python -m headroom.proxy.server`
- Proxy config: `HEADROOM_TELEMETRY=off`,
`HEADROOM_CACHE_ENABLED=false`, `HEADROOM_RATE_LIMIT_ENABLED=false`
- Provider/model: no live upstream provider needed; exercised local
`/v1/compress` -> `/v1/retrieve` CCR HTTP flow with model `gpt-4o`

Exact steps run after the patch:

1. Start a real Headroom proxy process with
`HEADROOM_CCR_TTL_SECONDS=7200`.
2. POST a 200-item tool-result payload to `/v1/compress`.
3. Extract the emitted `<<ccr:...>>` marker hash from the compressed
messages.
4. GET `/v1/retrieve/stats`.
5. POST `/v1/retrieve` with the marker hash.
6. Repeat with `HEADROOM_CCR_TTL_SECONDS=1`, wait 1.5 seconds, and
retrieve the same way to verify explicit expiration reporting.

Observed result:

```json
{
  "long_ttl": {
    "ccr_hash": "b473e632aa47",
    "retrieve_status": 200,
    "retrieved_content_has_result_199": true,
    "stats_default_ttl_seconds": 7200,
    "stats_entry_count": 1,
    "ttl_seconds": 7200
  },
  "short_ttl_expired": {
    "ccr_hash": "b473e632aa47",
    "retrieve_detail": "Entry expired (CCR TTL: 1 seconds; age: 2 seconds)",
    "retrieve_status": 404,
    "stats_default_ttl_seconds": 1,
    "stats_entry_count": 1,
    "ttl_seconds": 1
  }
}
```

What I did not test:

- A live OpenRouter/OpenAI/Anthropic upstream request.
- A durable/non-memory CCR backend.
- A literal 5+ minute wall-clock wait; the process E2E used `7200` to
prove configured retention and `1` second to prove expiration behavior
quickly.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

## Test Output

```bash
UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_compression_store.py tests/test_proxy_ccr.py tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py
# 152 passed, 2 warnings in 12.87s

UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_adapter_hooks.py tests/test_toin_feedback.py tests/test_toin_fixes.py
# 55 passed, 13 skipped, 1 warning in 0.53s

UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff check .
# All checks passed!

UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff format --check .
# 776 files already formatted

UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy mypy headroom
# Success: no issues found in 346 source files
```

Existing warnings observed in the targeted tests were unrelated to this
change:

- AnthropicProvider tiktoken approximation warning in proxy CCR tests.
- Deprecated `ToolIntelligenceNetwork.get_recommendation()` warning in
existing TOIN assertions.

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Not applicable.

## Additional Notes

No new dependencies. Default behavior remains 300 seconds unless
`HEADROOM_CCR_TTL_SECONDS` is set.
2026-06-10 23:20:46 -05:00
Tejas Chopra 62a1f23b88 fix: log Codex ws cancellations safely 2026-05-10 09:06:07 -07:00
Tejas Chopra eaf5980b4a fix: stabilize codex compression, stats, and proxy lifecycle 2026-05-09 13:47:53 -07:00
chopratejas 914a34fbd8 test(crusher): update compression_store hash test to expect SHA-256[:24]
Companion to the MD5→SHA-256 switch in 98d458f. The hash-pinning test
asserted `hashlib.md5(content.encode()).hexdigest()[:24]`; flip it to
the new function. Also expanded the failure message so the next
person debugging this knows why this gate exists and what they need
to verify if they're tempted to change the hash function again.

CI test (3.10/3.11/3.12/3.13) failed on this single assertion after
the SHA-256 switch; with the test updated, the rest of the
compression_store regression (76 tests) stays green.
2026-05-05 12:59:13 -07:00
rrubayet321 2ae71fe44d chore: add nosec B324 annotations to non-cryptographic MD5 usages and update temporary database path to use system temp directory 2026-04-07 13:07:26 +06:00
chopratejasandClaude Opus 4.6 cfd44b3f6a Fix test failures: update hash test for MD5, handle unicode surrogates
- Update test_hash_uses_sha256_truncated → test_hash_uses_md5_truncated
  to match the SHA256→MD5 change in compression_store.py
- Use errors="surrogatepass" in compute_hash to handle lone surrogates
  in unicode content (fixes pre-existing UnicodeEncodeError)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 11:32:01 -07:00
chopratejasandClaude Opus 4.5 d3298368bf fix: improve error handling and add comprehensive test coverage
Bug fixes:
- Replace bare except handlers with specific exception types and logging
  in proxy/server.py (6 instances for CCR, SSE parsing, cost tracking)
- Fix session_id filtering security bug in memory/backends/local.py
  (sessions were not properly isolated in vector search)

New tests (344 total):
- test_ccr_batch_processor.py: 51 tests for batch result processing
- test_compression_store.py: 76 tests for compression cache
- test_log_compressor.py: 47 tests for log format detection/compression
- test_search_compressor.py: 48 tests for grep output compression
- test_integrations/langchain/: 122 tests for LangChain integration
  (agents, memory, retriever, streaming)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 16:08:36 -08:00