mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
398d7cdcf4 |
fix(ccr): only refuse a bare marker, not originals that start and end with markers (#4626)
## Description The bare-marker guard added for #2694 in `CompressionStore.store()` checks `startswith("<<ccr:") and endswith(">>")` on a single line. That also matches an original that starts with one marker and ends with a different one, e.g. `<<ccr:abcd>> Keep this original instruction and the next reference <<ccr:ef01>>`. Such an original is refused, an error is logged, and `retrieve(hash)` returns `None`. The guard is now a full match of exactly one marker (`<<ccr:[^>\n]+>>` on the stripped value). A bare marker, with or without surrounding whitespace, is still refused and still cannot overwrite an existing entry. Originals that merely contain markers are stored, as the existing comment already intends. Closes #4266 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/compression_store.py`: new module-level `_BARE_CCR_MARKER_RE`; the guard in `store()` uses `fullmatch` on it. The character class excludes `>` and newline, so two adjacent markers (`<<ccr:a>><<ccr:b>>`) or a multi-line value are not a bare marker, matching the old single-line intent. Marker shapes with commas or spaces (`<<ccr:HASH,KIND,SIZE>>`, `<<ccr:HASH N_rows_offloaded>>`) are still recognised as bare. Comment updated to mention #4266. - `tests/test_compression_store.py`: two parametrized regression tests in `TestCompressionStoreRetrieve`: - three originals that start and end with separate markers are stored and retrieved exactly (all three fail on `main`); - three bare markers (plain, whitespace-padded `,base64,` form, and `N_rows_offloaded` form) are refused and do not overwrite an existing entry under the same `explicit_hash`. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed I ran the store tests and the CCR-related suites, not the full `pytest` run (see Not tested). The one failure below is the same on `main` without this change: the source checkout has no compiled `headroom._core`. ### Test Output ```text # Before the fix (main |
||
|
|
2fed685e77 |
perf(cache): drop eager full-backend TTL sweep on every new-key store() (#3256)
## Description Every new-key `CompressionStore.store()` previously called `_clean_expired()` before checking capacity. The default `SQLiteBackend.items()` implementation selects and deserializes every stored entry, so each recoverable compression store paid for an unthrottled O(n) scan. This removes that eager scan while preserving mixed-TTL capacity correctness. A lazy expiration heap removes expired entries first, and the existing creation-time heap still evicts the oldest live entry when capacity remains full. No linked issue. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed the eager full-backend TTL sweep from the new-key store path. - Added a lazy expiration heap alongside the creation-time eviction heap. - Preferred expired entries under capacity pressure without recording success feedback. - Kept both heaps bounded across replacement, read cleanup, backend-side SQLite purges, and explicit clearing. - Saturated oversized positive TTL expiration times to infinity. - Rebuilt both heaps from one backend snapshot when stale-entry thresholds are reached. - Added regressions for no-scan stores, mixed TTLs, replacement, read cleanup, oversized TTLs, SQLite purges, feedback behavior, and oldest-live eviction. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ HF_HUB_OFFLINE=1 LITELLM_LOCAL_MODEL_COST_MAP=true \ .venv/bin/python -m pytest tests/test_compression_store.py 80 passed $ .venv/bin/ruff check headroom/cache/compression_store.py tests/test_compression_store.py All checks passed! $ .venv/bin/ruff format --check headroom/cache/compression_store.py tests/test_compression_store.py 2 files already formatted $ .venv/bin/mypy headroom Success: no issues found $ make ci-precheck ci-precheck PASSED - safe to push ``` The no-scan regression was also run against the parent behavior. It failed with five backend scans for five stores and passes with zero scans after this change. ## Real Behavior Proof - Environment: macOS 26.6, Python 3.13.13, SQLite 3.51.0, branch `fix/ccr-store-ttl-scan` at `4cf29b81`. - Exact command / steps: Seeded a fresh SQLite CCR database with 2,000 entries containing 512-byte payloads, then timed 50 new-key `store()` calls before and after the eager-scan removal. Separately ran the mixed-TTL regression with capacity two: store an older long-lived entry, store a newer short-lived entry, advance beyond the short TTL, then store a third entry. - Observed result: Parent behavior took 0.684 seconds total (13.67 ms/store); the patched path took 0.001 seconds total (0.03 ms/store). Under mixed-TTL pressure, the expired newer entry is removed and the older live entry remains. The targeted cache suite has 80 passing tests. - Not tested: Windows or Linux runtime benchmarks, multi-process write contention, third-party backends, or database sizes above 2,000 rows. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A. - Stable/default behavior changed: Yes. New-key CCR stores no longer perform a full TTL sweep; expired entries are instead reconciled lazily through the expiration heap. - Kill switch / disable path: None specific to this optimization. - Unsafe override required: No. - Qualification impact: No public API, configuration, schema, or feature-qualification changes. - Rollback path: Revert commits `4cf29b81` and `506a8972`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes Documentation is unchanged because this is an internal cache implementation fix with no public API or configuration changes. |
||
|
|
5cb87bc351 |
fix(cache): default CCR payload previews off; harden the log when they are on (#3728)
## A-6 — tool-result content was written to the runtime log by default
(S2)
### RELEASE NOTE — a default changes, and operators lose log content
they may rely on
`HEADROOM_LOG_PAYLOAD_PREVIEW` **now defaults to off.** Operator-visible
impact:
- Each `event=headroom_retrieve` line in
`~/.headroom/logs/proxy-<port>.log` now carries
`"payload_preview":""` and `"payload_preview_chars":0` instead of up to
4096 characters
of the retrieved payload. `payload_chars` (the byte count) is unchanged,
so anything
measuring retrieval size still works; anything grepping the log for
retrieved *content*
will find nothing until the operator opts in.
- The opt-in is now **explicit and positive**:
`HEADROOM_LOG_PAYLOAD_PREVIEW=1|true|yes|on`.
Previously *any* value other than `0/false/no/off` — including a typo —
meant "on".
- **On Linux and macOS, every Headroom log file is now created `0600`**
(owner-only) — the
runtime log `~/.headroom/logs/proxy-<port>.log`, each of its rotated
backups
(`proxy-8000.log.1`, …), and the `--log-file` JSONL request log. This is
**unconditional**:
it does not depend on `HEADROOM_LOG_PAYLOAD_PREVIEW`, `--log-messages`,
or any other
switch. **A log shipper, sidecar, or collector that reads these files as
a different user
or group will stop being able to read them.** Run the collector as the
same user, or copy
the file out with a privileged job. On Windows nothing changes — see the
Windows note
below.
- If the runtime log path already exists as a **symlink**, Headroom now
refuses to write the
log at all (with a warning naming the path) instead of writing through
it. An operator who
deliberately symlinks `~/.headroom/logs/proxy-<port>.log` onto another
volume must replace
it with a real path or point `HEADROOM_WORKSPACE_DIR` at that volume.
### The bug
`_payload_preview_enabled()` returned `True` for an unset variable, so a
stock proxy wrote
up to 4 KB of the retrieved tool-result payload into the always-on
runtime log on every CCR
retrieval. That payload is verbatim tool output: source code, customer
data, and any
credential the three redaction regexes (`*_KEY|TOKEN|SECRET|...`,
`Bearer`/`Basic`, `sk-…`)
do not happen to match. The file was created at the process umask —
`0644` on a normal
developer machine.
**The claim held**, at `headroom/cache/compression_store.py:59`/`:116`
on current `main`
(line numbers unchanged since `89a58fd1`). Demonstrated by the
fail-before run below, where
the new test found the string `ssn=123-45-6789` sitting in a real log
file, mode `0644`.
### The fix
- `headroom/cache/compression_store.py` — `_payload_preview_enabled()`
defaults to `False`
and now requires an affirmative `1/true/yes/on`. The redaction pass is
untouched; it is
now a second line of defence behind an explicit opt-in rather than the
only one.
- `headroom/fileperms.py` (new) — owner-only file creation, and an
explicit
`OWNER_ONLY_SUPPORTED` flag recording *where the mode bits actually
decide read access*.
- `headroom/proxy/helpers.py` — `_OwnerOnlyRotatingFileHandler`, now
used by
`_setup_file_logging` **unconditionally**. `logging` opens the stream
itself at
construction *and on every rollover* with no mode hook, so `_open` is
replaced outright:
the file is opened `O_CREAT|O_APPEND|O_WRONLY|O_NOFOLLOW` at `0600` and
tightened with
`fchmod` on that same descriptor (so an existing `0644` log is fixed,
and the mode cannot
land on a different inode). `rotate()` re-applies the mode to each
backup, and backups
left `0644` by an older build are tightened when the handler opens.
- `headroom/proxy/request_logger.py` — the `--log-file` JSONL, which
carries whole request
and response bodies under `--log-messages`, is created the same way.
- `docs/content/docs/proxy.mdx` — the variable was entirely
undocumented; it now has a row
in **Observability**, plus a **Who can read the logs** subsection
stating the POSIX
guarantee and the Windows non-guarantee in as many words.
### Fail-before / pass-after
Method: commit the work, `git checkout origin/main --
headroom/cache/compression_store.py
headroom/proxy/helpers.py` keeping the new tests, run, then `git
checkout HEAD -- <those
files>`.
**Before (sources reverted to `origin/main`, new tests kept):**
```
__________ test_runtime_log_holds_no_payload_text_at_default_settings __________
tests/test_cache_breakpoint_diagnostics.py:246: in test_runtime_log_holds_no_payload_text_at_default_settings
assert secret not in text
E assert 'BEGIN-CUSTO...USTOMER-DATA' not in '2026-09-23 ...MER-DATA"}\n'
E 'BEGIN-CUSTOMER-DAT...s END-CUSTOMER-DATA' is contained here:
E preview":"BEGIN-CUSTOMER-DATA ssn=123-45-6789 def sekrit(): pass END-CUSTOMER-DATA"}
_____________ test_runtime_log_is_owner_only_when_preview_enabled ______________
tests/test_cache_breakpoint_diagnostics.py:257: in test_runtime_log_is_owner_only_when_preview_enabled
assert stat.S_IMODE(log_path.stat().st_mode) == 0o600
E AssertionError: assert 420 == 384 # 0o644 vs 0o600
____ test_runtime_log_hardening_survives_a_pre_existing_world_readable_log _____
E AssertionError: assert 420 == 384
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_disabled_by_default
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_stays_off_for_anything_else[]
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_stays_off_for_anything_else[maybe]
FAILED tests/test_cache_breakpoint_diagnostics.py::test_payload_preview_stays_off_for_anything_else[ ]
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_holds_no_payload_text_at_default_settings
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_is_owner_only_when_preview_enabled
FAILED tests/test_cache_breakpoint_diagnostics.py::test_runtime_log_hardening_survives_a_pre_existing_world_readable_log
========================= 7 failed, 17 passed in 1.59s =========================
```
That first failure is the finding itself: on `main`, customer data lands
in a real log file
under default settings.
**After (sources restored):**
```
tests/test_cache_breakpoint_diagnostics.py ........................ [ 24%]
tests/test_compression_store.py ............................................ [100%]
============================= 100 passed in 9.36s ==============================
```
`tests/test_perf_per_port_logs.py` (10 passed) covers the handler swap —
dedup, per-worker
paths, and sequential port changes still behave.
### Tests
New, in `tests/test_cache_breakpoint_diagnostics.py`:
- `test_payload_preview_disabled_by_default` — unset ⇒ byte counts only.
- `test_payload_preview_opt_in_values` /
`test_payload_preview_stays_off_for_anything_else`
— only an affirmative value enables it; a typo (`maybe`, `""`,
whitespace) does not.
- `test_runtime_log_holds_no_payload_text_at_default_settings` —
end-to-end through the
real `_setup_file_logging` handler: a store+retrieve leaves
`event=headroom_retrieve` in
the file with the right `payload_chars` and **no** payload text.
- `test_runtime_log_is_owner_only_when_preview_enabled` — `0600` on the
created file.
-
`test_runtime_log_hardening_survives_a_pre_existing_world_readable_log`
— a `0644` log
from a pre-hardening run is chmodded, since `O_CREAT`'s mode does not
apply to it.
Updated: the two tests in `tests/test_compression_store.py` that
exercise the preview and
its redaction now opt in explicitly, and
`test_payload_preview_enabled_by_default` is
inverted to `test_payload_preview_disabled_by_default`.
### Notes
- Independent of #3726 (A-5) — both are based on `origin/main`, and they
touch disjoint
parts of `compression_store.py`.
- A-5's audit found that the runtime log is one of the files
`--stateless` does *not*
suppress, so before this PR a stateless proxy was getting payload text
on disk too.
- `CHANGELOG.md` not touched (CI guard).
---
## Update after review (@JerrettDavis) — both blockers addressed
### 1. Windows: the guarantee is now scoped in code, not just in prose
You were right that this was a control that silently did nothing on a
supported platform.
On NTFS, read access is decided by the file's ACL; `os.chmod` only
toggles the read-only
attribute, so `chmod(0o600)` returns successfully while `stat.S_IMODE`
still reports
`0666` — which is exactly what your run showed.
I did **not** implement a Windows ACL. Python ships no ACL API; doing it
means ~150 lines
of `ctypes` building a `SECURITY_ATTRIBUTES` DACL, which I cannot
execute or test on this
machine, and I would rather not ship unverifiable security code than
ship an untested one.
So the guarantee is scoped honestly, and made loud rather than silent:
- `headroom/fileperms.OWNER_ONLY_SUPPORTED` is `os.name == "posix"`.
`restrict_fd_to_owner`
/ `restrict_path_to_owner` return `False` off POSIX rather than
reporting a protection
they did not establish.
- `_setup_file_logging` emits **one warning per process** on such a
platform: *"Headroom
cannot create <path> owner-only on this platform: file modes do
not control read
access here, and Headroom does not set an ACL. The runtime log can
contain request and
response content (`--log-messages`, wire debug,
`HEADROOM_LOG_PAYLOAD_PREVIEW`) — protect
the log directory itself."* Logging still works; it just no longer
implies a protection.
- The docs say it in as many words, under **Who can read the logs**: on
Windows the log
inherits the ACL of its parent directory, and the directory is the
access-control
boundary.
- The tests assert the real platform-conditional behaviour. The mode
assertions carry
`@_posix_only` with that reason;
`test_owner_only_support_matches_what_the_platform_can_enforce`
asserts the flag equals `os.name == "posix"` **on every platform**;
`test_no_owner_only_claim_is_made_off_posix` runs *only* on Windows and
asserts the
absence of the claim (the handler still creates the file;
`restrict_path_to_owner` returns `False`); and
`test_unsupported_platform_says_so_rather_than_silently_not_protecting`
simulates the
unsupported platform on any host so the warning itself is covered by
Linux CI.
**In one line: on Linux and macOS the log is `0600`, enforced. On
Windows Headroom does not
restrict the log, says so at startup and in the docs, and no test claims
otherwise.**
### 2. The permissions no longer key off `HEADROOM_LOG_PAYLOAD_PREVIEW`
Agreed, and taken as stated: **every** Headroom runtime log is created
owner-only, rollover
files included, regardless of which content switch is on. Previews were
never the only
source of request content in that file — `--log-messages` bodies,
wire-debug dumps and
query logging land there too, each behind its own switch, so gating on
any one of them left
the rest writing a sensitive file at the umask. `_setup_file_logging`
now always installs
`_OwnerOnlyRotatingFileHandler` and no longer imports
`_payload_preview_enabled` at all.
I extended it to the `--log-file` JSONL as well, since under
`--log-messages` that is the
file with the whole bodies in it.
Previews still default off — that part of the PR is unchanged.
This **does** break log collection for an operator reading the file as
another user, which
is the trade-off the original PR body ducked. It is now called out
explicitly in the release
note at the top, with the two workarounds.
### Also fixed while in here (found in an internal pass)
- **Rotated backups.** `rotate()` re-applies `0600` to the destination.
`os.rename` happens
to carry the mode across, but a configured `rotator` need not, so the
invariant is
asserted on the result rather than assumed from how it got there.
Backups left `0644` by
an older build are tightened when the handler opens.
- **`_open` followed symlinks.** The old code pre-created the file and
then delegated to
`super()._open()`, so both the `chmod` and the write followed a symlink
planted at the log
path — an attacker chose the destination *and* the permissions. `_open`
is now replaced
outright and opens with `O_NOFOLLOW` at `0600`, tightening with `fchmod`
on that same
descriptor (no TOCTOU window, and no way for the mode to land on another
inode). A log
path that is *already* a symlink is refused up front with a warning,
which is what carries
the refusal on platforms without `O_NOFOLLOW`. Deliberate decision:
**fail closed** — a
redirected sensitive log is worse than no log, and logging is already
best-effort here.
### Fail-before / pass-after for this round
Baseline is `ec014d64` (the commit you reviewed), not `origin/main`,
since these are new
findings on top of it. Method: commit, `git checkout
|
||
|
|
3169935858 |
perf(ccr): bound retrieval history with a deque instead of reslicing (#3471)
## Description
Once the CCR retrieval history reaches its 1,000-event cap, every
further retrieval appended to the list and then rebuilt it from a
1,000-element slice, under the store lock.
`collections.deque(maxlen=1000)` drops the oldest event as the new one
lands, so the trim is deleted rather than moved.
This is a maintenance change that also removes an O(cap) copy. It is
**not** an end-to-end speedup, and the measurements below say so: the
history append itself gets ~3.9× cheaper, while the full `retrieve()`
path is unchanged within noise.
`_pending_feedback_events` deliberately stays an unbounded list. It is a
separate drain-by-swap queue that still owes a notification for every
event, including ones the capped history has already discarded, so both
containers now carry a comment saying which is which.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/compression_store.py`: `_retrieval_events` becomes
`deque(maxlen=self._max_events)`; the append-then-reslice in
`_log_retrieval` is removed. `_max_events` stays as the single readable
name for the cap — it is read at construction, by the new tests and by
the benchmark, and is never assigned after `__init__`.
- `tests/test_compression_store.py`: two tests — retention and ordering
at the cap, and one feedback notification per retrieval past the cap.
- `benchmarks/ccr_retrieval_history_benchmark.py`: standalone script
measuring the history append at steady state, following the convention
the other benchmarks in that directory use.
Consumer audit before changing anything: the container is only ever hit
with `len`, `append`, `clear`, iteration, `list()` and `sys.getsizeof` —
all deque-safe. The one slice, `events_copy[-limit:]` in
`get_retrieval_events`, operates on the list copy taken under the lock,
so deque's lack of slice support is never reached. Repo-wide there is no
non-Python consumer, no subclass, no pickling or deepcopy path and no
`isinstance(list)` check, and the deleted line was the only assignment
to the attribute — so nothing can rebind an unbounded list back onto it.
One `threading.Lock` guards every access.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Both new tests pass before and after this change, deliberately: for a
maintenance change, parity is the claim. They exist because the
pre-existing `test_retrieval_events_list_bounded` asserts only `len(...)
<= 1000`, which cannot tell which end was dropped. Mutation-checked: an
unbounded `deque()` fails the length assertion, and a head-trim keeping
the oldest 1,000 passes the length check but fails on order.
### Test Output
```text
$ pytest tests/test_compression_store.py tests/test_ccr.py tests/test_ccr_feedback.py \
tests/test_critical_gaps.py tests/test_critical_fixes.py tests/test_proxy_ccr.py \
tests/test_ccr_sqlite_backend.py tests/test_ccr_batch_store.py \
tests/test_memory_tracker_integration.py -q
209 passed, 3 skipped in 16.66s
$ pytest tests/ -q -k "ccr or compression_store or compression_feedback or telemetry \
or critical_gaps or critical_fixes or memory_tracker" \
--ignore=tests/test_integrations
1 failed, 921 passed, 11 skipped in 77.38s
# the one failure is test_ccr_marker_round_trip_live, a live-API test that fails
# identically on the unchanged base
$ python benchmarks/ccr_retrieval_history_benchmark.py
container: deque
cap: 1000 events
appends per run: 20000, runs: 7
median: 0.481 us/append
$ pre-commit run --files headroom/cache/compression_store.py \
tests/test_compression_store.py benchmarks/ccr_retrieval_history_benchmark.py
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
mypy.....................................................................Passed
```
## Real Behavior Proof
- Environment: macOS 15.6, Python 3.13.11, branch based on `main` @
`e67b3c8a`. Single process, single thread, no contention.
- Exact command / steps: filled a store's history to its cap so every
subsequent append must evict, then timed 20,000 `_log_retrieval` calls,
7 runs, median reported. Ran the unchanged tree and this change in both
orderings to rule out ordering effects. Separately timed the full public
`retrieve()` path, 2,000 calls × 5 runs, also in both orderings.
- Observed result: history append **1.81 to 0.47 us** and, with the
order reversed, **1.85 to 0.48 us** per append. Full `retrieve()`: 7.67
vs 7.89 us, and reversed 7.79 vs 7.82 us — unchanged within noise in
both directions. Behavior parity checked value by value:
`get_stats()["event_count"]`, `get_retrieval_events()`
contents/order/length across limits from negative through 1e9, and
`clear()` are all identical; `deque.clear()` preserves `maxlen`, so the
container stays bounded afterwards.
- Not tested: no multi-threaded or contended measurement, no
multi-worker or production workload, and no memory-over-time profile.
The benchmark times a private method (`_log_retrieval`) because the
public `retrieve()` buries the difference in backend I/O and the
feedback drain — that is stated in the benchmark's own docstring. The
result is a microbenchmark of one operation and must not be read as a
proxy for retrieval latency.
One value does change numerically, and it is not hidden:
`get_memory_stats().size_bytes` shifts by a few hundred bytes, because
`sys.getsizeof` reports a deque's 64-slot blocks rather than a list's
contiguous array (56 to 760 bytes empty; 8,056 to 8,680 at 1,000
entries). It is a shallow size that already excluded the events
themselves, `budget_bytes` is `None`, and no test or dashboard compares
it against anything.
## Runtime Rollout Safety
- Rollout-managed feature(s): None. CCR retrieval-event history is not
rollout-gated.
- Minimum rollout channel: N/A — no rollout-managed feature is touched.
- Stable/default behavior changed: No. The cap, its size, retention
order, retrieval results, lock guarantees and feedback delivery are all
unchanged; only the container enforcing the cap differs, plus the
`size_bytes` gauge noted above.
- Kill switch / disable path: N/A — no flag. Reverting the commit
restores the list and its trim.
- Unsafe override required: No.
- Qualification impact: None. No compression decision, retrieval result,
CCR marker, TTL or persistence path is touched — this is the in-memory
display history only.
- Rollback path: `git revert` of this commit, single step. No schema,
data or config migration is involved.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
Not applicable — no user-visible surface changes.
## Additional Notes
- No separate documentation file needed updating; the two comments in
`__init__` are the documentation this needs, because the one thing a
future reader can get wrong here is bounding `_pending_feedback_events`
to match.
- Overlaps nothing in flight: PR #3465 touches this same module, but its
entire footprint is inside `_clean_expired`, which never reads the
retrieval history, and this change never touches `_stale_heap_entries`.
- Unrelated, left alone, noticed while running the suites:
`tests/test_realignment_live_multi_turn.py::test_ccr_marker_round_trip_live`
and
`tests/test_memory_usage_integration.py::TestProxyMemoryIntegration::test_real_api_calls_memory_tracking`
both need live API access and fail on a clean checkout without it.
|
||
|
|
113894600c |
fix(cache/ccr): don't evict a live entry on a duplicate store at capacity (#2082)
## Description
`CompressionStore.store` (`headroom/cache/compression_store.py`) runs
eviction **before** it knows
whether the incoming `hash_key` is new or a re-store of an
already-present key:
```python
with self._lock:
self._evict_if_needed() # <-- runs first
existing = self._backend.get(hash_key)
if existing is not None:
... # duplicate / collision: overwrite in place
self._stale_heap_entries += 1
self._backend.set(hash_key, entry)
```
When the store is full and the incoming key **already exists** (a
duplicate re-store),
`_evict_if_needed()` removes the oldest *distinct* entry to "make room"
— but then `set()` merely
overwrites the existing key in place, so no room was ever needed. Net
effect: `count` drops to
`max_entries - 1` and a **live, never-retrieved entry is destroyed**.
That entry's `<<ccr:...>>`
marker, still sitting in the conversation history, then resolves to a
404 on `/v1/retrieve`.
This is not a corner case: the CCR mirror bridge
(`_mirror_single_hash_to_python_store` in
`smart_crusher.py`) re-`store()`s the same `explicit_hash` every turn a
`<<ccr:…>>` marker is
re-encountered, and markers persist across turns — so a full store
silently deletes a live sibling
entry on each duplicate.
Concrete (with the repo's `max_entries=3` fixture): store c0, c1, c2
(full), then re-store c1
(same content ⇒ same hash). Eviction pops the oldest (c0), deletes it,
then c1 is overwritten in
place. Final state: {c1, c2}, count 2, and **c0 is gone** — its marker
is now unredeemable.
Closes: no issue filed — found while auditing the compression store.
## Fix
Decide novelty before evicting: only `_evict_if_needed()` for a
genuinely new key; a
duplicate/replace overwrites in place (no eviction). The
collision/duplicate logging and
stale-heap accounting are unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cache/compression_store.py`: `store()` reads `existing`
first and only evicts when the key is new.
- `tests/test_compression_store.py`: add
`test_duplicate_store_at_capacity_does_not_evict` (re-store an existing
hash at capacity keeps all entries and count at `max_entries`).
## Testing
- [x] New regression test added (`tests/test_compression_store.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/cache/compression_store.py tests/test_compression_store.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the store/evict
logic with a dependency-free script (replicating the backend + eviction
heap) and left the full pytest to CI.
- Exact command / steps: filled a `max_entries=3` store with c0/c1/c2,
then re-stored c1 (duplicate), through the old (evict-first) and new
(check-first) logic; also confirmed a genuinely new key still evicts the
oldest.
- Observed result: the old logic drops c0 (count 2); the new keeps all
three; and a new key at capacity still evicts the oldest:
```text
OLD: after duplicate re-store of h1 -> keys=['h1', 'h2'] count=2
NEW: after duplicate re-store of h1 -> keys=['h0', 'h1', 'h2'] count=3
NEW still evicts oldest for a genuinely new key at capacity
DUPLICATE-STORE EVICTION FIX VERIFIED (old drops a live entry; new keeps it)
```
- Not tested: a full proxy CCR round-trip (needs the heavy stack). The
fix is confined to `store()` and the new test drives it directly with
the `max_entries=3` fixture. Existing eviction tests use distinct keys
and stay green. Full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Internal reordering only, no signature change; no call sites or
backend mocks break.
- @JerrettDavis tagging you — this silently drops a live CCR entry
(making its marker 404) whenever a duplicate hash is re-stored at
capacity, which the mirror bridge does routinely. Thanks.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
c2fc4d3753 |
fix(ccr): make headroom_retrieve a hash-only full-content lookup (#1532)
The optional `query` parameter on headroom_retrieve routed retrieval through CompressionStore.search(), which BM25-scored the items inside a single cached blob and dropped everything below a 0.3 relevance floor. On small per-blob corpora with conversational queries this returned an empty result the large majority of the time, so the LLM saw "nothing found" for content that was actually present — pushing users to turn compression off entirely. Retrieval is fundamentally a hash lookup (this already matches the Rust proxy's CCR store, which is put/get only — "no BM25 search"). Remove the query/search path end to end and always return the full original content: Core (Python proxy): - tool schemas (anthropic/openai/google) drop the `query` property - parse_tool_call returns the hash (str | None) instead of (hash, query) - response handler, proxy POST/GET/tool-call handlers, the MCP retrieve tool, and the streaming feedback recorders retrieve by hash only - proactive context-tracker expansion always restores full content - delete CompressionStore.search() and its BM25 machinery (the bm25 module stays — it is still used by relevance/) - CCRToolCall.query, CCRToolResult.was_search, and ExpansionRecommendation.expand_full/search_query are removed Plugins (advertised a now-defunct query param to the LLM): - hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop `query` from their schemas, signatures, request URLs, and tests Benchmarks/docs: - ccr_regression + adversarial benchmarks switch from store.search() to full hash retrieval (search input-injection tests repurposed to the hash, the only remaining input surface) - wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx, config.py and store docstrings updated to describe hash-only retrieval Tests updated to assert full-content retrieval and guard the removed surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean. ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
|
b7be3814f1 |
feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description
A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### 1. Rust compressor extraction
- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.
### 2. CCR store hardening
- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).
### 3. Traffic audit tooling (measure before tuning)
- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.
### 4. Read maturation (Mechanism B) — experimental, default OFF
- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.
### 5. Rebase / CI fixups (this update)
- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s
$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed
$ mypy headroom/
Success: no issues found in 365 source files
$ python -m compileall headroom/ -q
COMPILE-OK
# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
# "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```
## Real Behavior Proof
- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
|
||
|
|
2533f7703e |
fix(ccr): make retrieval TTL configurable (#715)
## Description Make the CCR retrieval store TTL configurable so long-running agent jobs can keep `headroom_retrieve` markers resolvable beyond the previous hard-coded 300-second window. Fixes #714 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `HEADROOM_CCR_TTL_SECONDS` for the global CCR `CompressionStore` default TTL, with validation and fallback to 300 seconds. - Expose the effective CCR TTL as `store.default_ttl_seconds` from `/v1/retrieve/stats`. - Distinguish missing vs expired CCR retrieval failures in `/v1/retrieve`, `/v1/retrieve/{hash}`, tool-call handling, and CCR response handling. - Update CCR docs, README wording, and CHANGELOG for the user-facing TTL behavior. - Add regression tests for env-configured TTL, invalid env fallback, explicit TTL precedence, stats exposure, and expired retrieval detail. ## Reproduction Before this change, the proxy-global CCR store always used the default 300-second TTL when callers used `get_compression_store()` without an explicit `default_ttl`. A long-running agent job could receive a `<<ccr:...>>` marker and fail later with a generic not-found/expired response after the fixed 5-minute retention window. The new tests cover this by setting `HEADROOM_CCR_TTL_SECONDS`, creating CCR entries through the same global store used by the proxy, and asserting that stored entries and `/v1/retrieve/stats` use the configured retention. ## Real behavior proof Setup tested: - macOS 15.7.2 - Python 3.13.9 via `uv run --frozen --extra dev --extra proxy` - Local Headroom proxy subprocess: `python -m headroom.proxy.server` - Proxy config: `HEADROOM_TELEMETRY=off`, `HEADROOM_CACHE_ENABLED=false`, `HEADROOM_RATE_LIMIT_ENABLED=false` - Provider/model: no live upstream provider needed; exercised local `/v1/compress` -> `/v1/retrieve` CCR HTTP flow with model `gpt-4o` Exact steps run after the patch: 1. Start a real Headroom proxy process with `HEADROOM_CCR_TTL_SECONDS=7200`. 2. POST a 200-item tool-result payload to `/v1/compress`. 3. Extract the emitted `<<ccr:...>>` marker hash from the compressed messages. 4. GET `/v1/retrieve/stats`. 5. POST `/v1/retrieve` with the marker hash. 6. Repeat with `HEADROOM_CCR_TTL_SECONDS=1`, wait 1.5 seconds, and retrieve the same way to verify explicit expiration reporting. Observed result: ```json { "long_ttl": { "ccr_hash": "b473e632aa47", "retrieve_status": 200, "retrieved_content_has_result_199": true, "stats_default_ttl_seconds": 7200, "stats_entry_count": 1, "ttl_seconds": 7200 }, "short_ttl_expired": { "ccr_hash": "b473e632aa47", "retrieve_detail": "Entry expired (CCR TTL: 1 seconds; age: 2 seconds)", "retrieve_status": 404, "stats_default_ttl_seconds": 1, "stats_entry_count": 1, "ttl_seconds": 1 } } ``` What I did not test: - A live OpenRouter/OpenAI/Anthropic upstream request. - A durable/non-memory CCR backend. - A literal 5+ minute wall-clock wait; the process E2E used `7200` to prove configured retention and `1` second to prove expiration behavior quickly. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ## Test Output ```bash UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_compression_store.py tests/test_proxy_ccr.py tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py # 152 passed, 2 warnings in 12.87s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_adapter_hooks.py tests/test_toin_feedback.py tests/test_toin_fixes.py # 55 passed, 13 skipped, 1 warning in 0.53s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff check . # All checks passed! UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff format --check . # 776 files already formatted UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy mypy headroom # Success: no issues found in 346 source files ``` Existing warnings observed in the targeted tests were unrelated to this change: - AnthropicProvider tiktoken approximation warning in proxy CCR tests. - Deprecated `ToolIntelligenceNetwork.get_recommendation()` warning in existing TOIN assertions. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Not applicable. ## Additional Notes No new dependencies. Default behavior remains 300 seconds unless `HEADROOM_CCR_TTL_SECONDS` is set. |
||
|
|
62a1f23b88 | fix: log Codex ws cancellations safely | ||
|
|
eaf5980b4a | fix: stabilize codex compression, stats, and proxy lifecycle | ||
|
|
914a34fbd8 |
test(crusher): update compression_store hash test to expect SHA-256[:24]
Companion to the MD5→SHA-256 switch in 98d458f. The hash-pinning test asserted `hashlib.md5(content.encode()).hexdigest()[:24]`; flip it to the new function. Also expanded the failure message so the next person debugging this knows why this gate exists and what they need to verify if they're tempted to change the hash function again. CI test (3.10/3.11/3.12/3.13) failed on this single assertion after the SHA-256 switch; with the test updated, the rest of the compression_store regression (76 tests) stays green. |
||
|
|
2ae71fe44d | chore: add nosec B324 annotations to non-cryptographic MD5 usages and update temporary database path to use system temp directory | ||
|
|
cfd44b3f6a |
Fix test failures: update hash test for MD5, handle unicode surrogates
- Update test_hash_uses_sha256_truncated → test_hash_uses_md5_truncated to match the SHA256→MD5 change in compression_store.py - Use errors="surrogatepass" in compute_hash to handle lone surrogates in unicode content (fixes pre-existing UnicodeEncodeError) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
|
d3298368bf |
fix: improve error handling and add comprehensive test coverage
Bug fixes: - Replace bare except handlers with specific exception types and logging in proxy/server.py (6 instances for CCR, SSE parsing, cost tracking) - Fix session_id filtering security bug in memory/backends/local.py (sessions were not properly isolated in vector search) New tests (344 total): - test_ccr_batch_processor.py: 51 tests for batch result processing - test_compression_store.py: 76 tests for compression cache - test_log_compressor.py: 47 tests for log format detection/compression - test_search_compressor.py: 48 tests for grep output compression - test_integrations/langchain/: 122 tests for LangChain integration (agents, memory, retriever, streaming) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |