mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
98018333f9 |
fix(cache): fail closed on SQLite corruption instead of deleting the store online (#4361)
## Description
Closes #4335
`SQLiteBackend._handle_db_error` used to delete the database file and
recreate it when it saw corruption. With several workers sharing the
file, that destroys data online: on Windows the unlink raised an
unhandled `PermissionError`, and on POSIX siblings kept using old
handles while this worker opened a replacement, and removing live
`-wal`/`-shm` state could discard committed originals. This PR makes
recovery fail closed instead:
1. **No online deletion.** On corruption the instance closes only its
own connection and goes offline. The database and its `-wal`/`-shm`
sidecars are left in place for offline repair, so sibling workers are
unaffected.
2. **Bounded reconnect.** While offline, operations raise
`StoreUnavailableError` internally (a miss for reads). The connection is
retried with backoff from 1s doubling to 60s, and comes back once the
file has been repaired or removed.
3. **Writes report failure.** Following `main`'s contract,
`SQLiteBackend.set()` returns `False` whenever the row was not committed
(offline, transient busy/locked, or corruption during the post-write
purge), so `CompressionStore.store` raises `StorageError` and producers
keep the original.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cache/backends/sqlite.py`: fail-closed `_handle_db_error`,
`_connection()` with bounded reopen backoff, `set()` returns
`True`/`False`.
- `tests/test_ccr_sqlite_backend.py`: multi-connection test that
originals survive the recovery boundary, a guard that recovery never
unlinks the store or sidecars, backoff and recovery tests, and
producer-level outage tests (`TestProducerDuringOutage`).
- `base.py`, `compression_store.py` and `backends/__init__.py` are
unchanged relative to `main`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_ccr_sqlite_backend.py
38 passed in 1.00s
$ ruff check headroom/cache tests/test_ccr_sqlite_backend.py
All checks passed!
```
## Real Behavior Proof
- Environment: Linux, Python 3.13.5, at
|
||
|
|
4e6e955cad |
fix(memory): log storage and SQLite failures that currently make retrieval miss silently (#4111)
## Description Several storage paths turn a real failure into a quiet miss. They either log nothing, or log a line that does not say which operation or file failed. When CCR or memory retrieval misses, an operator cannot tell corruption or a broken plugin from a genuine "not found". This PR adds the missing log lines. Return values and control flow do not change. Refs #4136 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/storage/__init__.py` (`create_storage`): - A `headroom.storage_backend` plugin that raised on load was swallowed by `except Exception: pass`. So was a scheme with no plugin. Both then fell back to `SQLiteStorage(store_url)`, which opens a SQLite file named after the URL. - Both cases now log a warning that names the scheme. The URL itself is not logged, because it can carry credentials. - A plugin that raises is logged with the exception type only, because its text can echo the URL. A separate DEBUG line gives `log_safety.describe_exception` (exception types and code locations, never the message), so no level carries the URL or its password. - The module gets a `logger`. - `headroom/cache/backends/sqlite.py` (CCR store): - `_entry_from_json` returned `None` with no log for a row that fails to decode or rebuild. So `get()` reported a plain miss for an entry that exists but is unreadable. - It now logs a warning with the hash, the db path and the exception type. `items()` decodes every row on each eviction pass, so the warning fires once per hash. - The once-per-hash guard is `log_safety.WarnOnce`. Within a one-hour window it warns for at most 1,000 hashes, then logs one overflow warning, and later rows log at DEBUG only, so repeated eviction scans over many bad rows cannot flood the log. When the window ends, a row that is still broken warns again, about once an hour. A hash is used up only while WARNING is enabled. - CCR keys come from model-issued retrieve calls, so they are logged through `log_safety.safe_id`: a repr (newlines escaped) capped at 80 characters. - Failed `delete`/`exists`/`clear`/`count`/`keys`/`items`/`get_stats` all logged as `"op"`. Each now logs its real name, with the key for `get`/`set`/`delete`/`exists`. - The transient-error warning now includes the db path. - `headroom/memory/traffic_learner.py`: the two read-only SQLite reads returned `[]` on `OperationalError`/`DatabaseError` with no log: the hydrate path and `_load_persisted_patterns_from_sqlite`. Both now warn with the db path. Also: - dropped a redundant `conn.close()` in an `except` whose `finally` already closes; - the two best-effort `conn.close()` calls now catch `sqlite3.Error` and log at debug, instead of `except Exception: pass`. - `headroom/memory/backends/direct_mem0.py`: `get_memory`/`delete_memory` turned any Qdrant/Neo4j exception into "not found". They now warn with the memory id and the exception type first. - The exception text is not logged at any level, because backend errors can carry stored memory or connection details. A DEBUG line gives `describe_exception`. - Memory ids come from the model, so they are logged through `log_safety.safe_id`, which escapes newlines and caps the length. - mem0 2.x `Memory.delete` raises exactly `ValueError(f"Memory with id {memory_id} not found")` for an unknown id. Only that message, for the requested id, logs at DEBUG; the delete is model-driven, so it is expected. Any other `ValueError`, such as a missing collection, still warns. - When a corrupt CCR store is recreated, the unreadable-row guard starts fresh, because the new file has no rows. - `headroom/learn/registry.py`: a built-in learn plugin that fails to import was logged at debug, so `headroom learn --agent <name>` only said "Unknown agent". It is now a warning with the traceback. Built-in plugins use only the stdlib, so this never fires on a healthy install. - `headroom/cache/compression_store.py`: the "Failed to load CCR backend" warning now says it falls back to the in-process store, and what that costs. This matches the SQLite branch next to it. It logs the exception type only, because the backend's text can echo `HEADROOM_REDIS_URL`. DEBUG gets `describe_exception`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q -p no:cacheprovider tests/test_storage_backends.py tests/test_storage tests/test_ccr_sqlite_backend.py tests/test_compression_store.py tests/test_learn/test_registry.py tests/test_memory/test_traffic_learner.py tests/test_memory/test_direct_mem0.py tests/test_private_state_files.py tests/test_adapter_hooks.py tests/test_cache/test_backends.py tests/test_learn/test_opencode_scanner.py tests/test_openai_responses_traffic_learner.py tests/test_learn/test_section_lifecycle.py tests/test_learn/test_worktree_merge.py tests/test_memory_handler_native_ops.py ================= 537 passed, 1 skipped, 5 warnings in 34.37s ================== # The three new tests against origin/main (product code reverted): FAILED tests/test_storage_backends.py::test_create_storage_warns_when_a_backend_plugin_fails_to_load FAILED tests/test_ccr_sqlite_backend.py::TestSQLiteBackend::test_unreadable_row_is_logged_once_and_healthy_rows_are_not FAILED tests/test_learn/test_registry.py::TestBuiltinLoadFailure::test_broken_builtin_plugin_is_logged_as_a_warning $ uvx ruff@0.16.8 check <changed files> All checks passed! # Added in review: the hydration-failure warning; mem0 delete/get paths (only mem0's exact missing-memory message is quiet; a missing collection warns); newline and 100,000-char model ids; two scans over more bad rows than the cap; and storage-plugin, CCR-backend and mem0 failures whose records are rendered with logging.Formatter at DEBUG and WARNING and checked for credential and content canaries. # Those tests against the previous head |
||
|
|
728ff7e22b |
fix(cache/ccr): tolerate a malformed CCR entry row instead of crashing the store (#3156)
## Description
`SQLiteBackend._entry_from_json` (the CCR compression store's persistent
backend) deserializes each row's JSON blob into a `CompressionEntry`:
```python
try:
data = json.loads(raw)
except (json.JSONDecodeError, TypeError):
return None
known = {f.name for f in fields(CompressionEntry)}
return CompressionEntry(**{k: v for k, v in data.items() if k in known})
```
The `try` guards only the `json.loads`. The `CompressionEntry(**...)`
construction is unguarded, and `CompressionEntry` has eleven required
fields (no default): `hash`, `original_content`, `compressed_content`,
`original_tokens`, `compressed_tokens`, `original_item_count`,
`compressed_item_count`, `tool_name`, `tool_call_id`, `query_context`,
`created_at`. A row that **parses as JSON but is missing a required
field** — schema drift after an upgrade, or a partially written row —
therefore raises `TypeError` (and a non-`dict` blob such as `null`
raises too).
That exception propagates out of `get()` **and** `items()`. `items()` is
the real problem: `CompressionStore._clean_expired()` calls it on
**every** store's eviction pass, so a single malformed row does not just
fail one retrieval — it crashes all reads, all evictions, and therefore
all subsequent stores, until the row is somehow removed (which the
crashing store can no longer do).
The field-filtering and the `Optional` return type of `_entry_from_json`
already document the intent: unknown/malformed rows should degrade to a
miss, not raise (the sibling `test_unknown_json_fields_tolerated` covers
the forward-compat direction). This PR extends that resilience to the
missing-required-field / non-dict cases by guarding the construction.
Reproduction:
```python
import json, time
from headroom.cache.backends.sqlite import SQLiteBackend
b = SQLiteBackend("ccr.db")
b._conn.execute(
"INSERT OR REPLACE INTO ccr_entries (hash, entry_json, created_at, ttl) VALUES (?,?,?,?)",
("poison", '{"hash": "poison", "original_content": "x"}', time.time(), 1800),
)
b._conn.commit()
b.get("poison") # before: TypeError (missing 9 required arguments)
b.items() # before: TypeError -> breaks eviction on every store
```
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/backends/sqlite.py` (`_entry_from_json`): return
`None` for a non-`dict` blob, and wrap the `CompressionEntry(**...)`
construction in `except (TypeError, ValueError)` so a row missing a
required field (or carrying a wrong-typed value) degrades to a miss
instead of raising.
- `tests/test_ccr_sqlite_backend.py`: added
`test_missing_required_field_degrades_to_miss` — a poison row and a
`null` blob make `get()` return `None`, `items()` skip them (the good
row survives), and a full `CompressionStore.store()` cycle (which evicts
via `items()`) still works with the poison row present.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_ccr_sqlite_backend.py -> 21 passed, 1 skipped
uvx ruff@0.16.2 check headroom/cache/backends/sqlite.py tests/test_ccr_sqlite_backend.py -> All checks passed!
uvx mypy@1.20.2 headroom/cache/backends/sqlite.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: inserted a valid-JSON row missing required
fields directly into `ccr_entries` and called the backend
(`get('poison')` and `items()` both raised `TypeError:
CompressionEntry.__init__() missing 9 required positional arguments`);
applied the guard and re-ran the new regression against the un-patched
method (`python -m pytest tests/test_ccr_sqlite_backend.py -k
missing_required_field` -> failed on the raise), then with the fix -> 21
passed, 1 skipped for the file.
- Observed result: before, one malformed row raised out of
`get()`/`items()` (and `items()` runs during every store's eviction);
after, `get()` returns `None`, `items()` skips it, and stores/evictions
continue normally.
- Not tested: an actual on-disk schema-drift upgrade path; the malformed
row is injected directly, which is the exact shape `_entry_from_json`
consumes.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is defensive deserialization in
the CCR SQLite backend, not a rollout-channel-gated runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: a malformed/partial CCR row now
degrades to a miss instead of raising. Well-formed rows are loaded
exactly as before.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: the CCR store survives a corrupt/old-schema row
instead of crashing reads, evictions, and stores; the affected marker
simply misses until re-compressed (the same outcome the store already
documents for an evicted entry).
- Rollback path: revert this PR; `_entry_from_json` returns to the
unguarded construction.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal backend behavior; the class docstring already notes the prior
raise)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
The class docstring currently states that a blob missing a required
field "raises `TypeError` on construction." That was accurate but is
exactly the failure this PR removes; the field-filtering right above it
exists to make the backend forward-compatible, and crashing on a missing
field defeats that intent — especially because `items()` fans the crash
out to eviction on every store. I left the docstring's forward-compat
description intact since the observable contract (`_entry_from_json`
returns `Optional`) is now honored for both directions.
|
||
|
|
e64b9f58a2 |
test(cache): make purge insertion eligible after setup (#3790)
## Summary Fix the release-blocking SQLite purge test fixture. Its setup performs several stores (which advances the backend's intentional 60-second purge throttle), then manually backdates one row and expects the immediately following insert to purge it. Explicitly reset the purge eligibility after backdating so the assertion tests metadata-only purge behavior rather than suite timing. ## Validation - Focused regression: 1 passed - Full SQLite backend file: 23 passed, 4 skipped - Ruff: clean - git diff --check: clean |
||
|
|
9b8cae8448 |
fix(security): create credential files private instead of narrowing after write (#3215)
## Description Three code paths persist sensitive local data by writing the file first and narrowing its permissions afterward: ```python path.write_text(...) # created at the umask default — typically 0o644 path.chmod(0o600) # narrowed only now ``` Between those two calls the file is group/world readable, and if the process dies in the gap (or the `chmod` raises and is swallowed) it stays that way permanently. On a shared host a local unprivileged user can read the contents in that window. Affected files: - `headroom/copilot_auth.py` — the GitHub Copilot **OAuth refresh token** (the most sensitive of the three). - `headroom/providers/codex/recovery.py` — the backup manifest describing the recovered account's file layout. - `headroom/cache/backends/sqlite.py` — the CCR originals database, which holds raw tool output (file contents, command output). Here it is worse: `sqlite3.connect` creates the db file itself at the umask default, and the `chmod` runs only after the first `DELETE`/`commit`, so the window spans real writes. The repo already knows the correct primitive — `headroom/telemetry/session.py` writes its install-id with `os.open(path, O_WRONLY|O_CREAT|O_TRUNC, 0o600)`, which creates the file private from birth (umask can only clear permission bits, never add them, and `0o600` has none set for group/other). This is **low severity**: local-only, POSIX-only (Windows does not use these bits), and not remotely exploitable. It is a defense-in-depth hardening of files that are meant to be private and are already `chmod`ed — this just closes the creation window. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/copilot_auth.py` / `headroom/providers/codex/recovery.py`: added a module-local `_write_private_text(path, text)` that narrows any pre-existing file to `0o600` first (so a legacy world-readable file is not left wide while new secret content is written into it), then writes through `os.open(..., O_WRONLY|O_CREAT|O_TRUNC, 0o600)` so new files are private from birth, and `chmod`s once more after. The two token/manifest writers now call it instead of `write_text` + `chmod`. - `headroom/cache/backends/sqlite.py`: added `_ensure_private(path)` and call it in `_open()` **before** `sqlite3.connect`, so the db file is created (or an existing one narrowed) at `0o600` before sqlite opens it. The existing post-commit `chmod` loop over `-wal`/`-shm` is kept: those are created by sqlite during writes, and narrowing them after commit is unchanged behavior. - `tests/test_copilot_auth.py`, `tests/test_ccr_sqlite_backend.py`, `tests/test_cli/test_recover_codex.py`: added **born-private** regressions for all three writers. Asserting the *final* mode would pass on the old code too (its trailing `chmod` always ends at `0o600`), so each test neutralizes every `chmod` and forces `umask(0o022)`, leaving the mode to reflect only how the file was *created* — `0o600` for `os.open(..., 0o600)`, `0o644` for the old create-then-narrow. Plus a content-roundtrip test confirming the token still writes and reads back. The permission assertions are POSIX-gated (`skipif os.name == 'nt'`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text tests/test_copilot_auth.py -> 128/131 passed (born-private POSIX test skipped on the Windows dev box) tests/test_ccr_sqlite_backend.py -> born-private + existing tests pass (POSIX test skipped on Windows) tests/test_cli/test_recover_codex.py -> born-private manifest test passes on POSIX (skipped on Windows) uvx ruff@0.16.2 check <changed files> -> All checks passed! uvx mypy@1.20.2 headroom/copilot_auth.py headroom/providers/codex/recovery.py headroom/cache/backends/sqlite.py -> Success: no issues found in 3 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.16.2 and mypy 1.20.2 via uvx. - Exact command / steps: verified all three writers round-trip content correctly. The permission tests are genuine red->green: with every `chmod` neutralized and `umask(0o022)` forced, the new `os.open(..., 0o600)` path yields a `0o600` file while the old create-then-`chmod` path would leave `0o644` (the forced umask makes this independent of the runner's ambient umask). These assertions are POSIX-only; on this Windows dev box they are skipped (Windows does not apply POSIX permission bits) and run on Linux CI. - Observed result: content is written and read back identically; no behavior change beyond the file being private from creation. - Not tested: the POSIX permission-bit assertions did not execute on this Windows host (they are `skipif os.name == 'nt'` and run on Linux/macOS CI); no live GitHub OAuth login or real codex recovery run — the writers/helpers are exercised directly. ## Runtime Rollout Safety - Rollout-managed feature(s): none. These are local file writes (auth token, backup manifest, CCR cache db), not rollout-channel-gated runtime features. - Minimum rollout channel: N/A. - Stable/default behavior changed: no. Files end up at the same `0o600` they were `chmod`ed to before; only the creation window is closed. Content and read paths are unchanged. - Kill switch / disable path: N/A. - Unsafe override required: no. - Qualification impact: none. On Windows the `chmod`/`os.open` mode is a no-op exactly as the prior `chmod` was. - Rollback path: revert this PR; the writers return to `write_text` + `chmod`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: internal behavior, unchanged semantics) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes The `-wal`/`-shm` sidecar files remain narrowed after the first commit rather than pre-created private, deliberately: pre-creating them risks sqlite WAL-format edge cases, and the durable store (the main db) — which is what persists between runs and holds the originals — is now private before sqlite ever opens it. |
||
|
|
389e45cdde |
perf(cache): purge expired CCR rows by deadline instead of decoding every payload (#3465)
## Description
`CompressionStore._clean_expired()` runs on every CCR insertion and
calls `backend.items()`, which deserializes every live original just to
read the `created_at`/`ttl` that SQLite already stores as columns. With
1,000 unexpired 16 KiB originals, one new insertion decoded all 1,000
payloads.
This replaces that scan with a metadata-only delete on the persistent
backend.
Closes # N/A
## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `SQLiteBackend.purge_expired()` deletes expired rows with `DELETE ...
WHERE created_at + ttl < ?`, backed by a new expression index
`idx_ccr_expiry_deadline`. `_maybe_purge()` now shares the same helper
instead of duplicating the statement.
- `CompressionStore._clean_expired()` calls the backend method when the
backend offers one, and folds its deleted-row count into
`_stale_heap_entries` so eviction accounting and the heap-rebuild
threshold behave exactly as before. Backends without `purge_expired` —
`InMemoryBackend` and any third-party implementation — keep the existing
item-scan path, so the `CompressionStoreBackend` protocol is unchanged.
- Dropped the superseded `idx_ccr_expiry (created_at)`. Nothing in the
tree searches or orders by bare `created_at`, so it only cost writes;
the schema now drops it, migrating existing database files on open.
- Added `benchmarks/ccr_sqlite_expiry_benchmark.py` for reproduction.
Behavior deliberately preserved: expired entries are still cleaned
before any live entry is evicted, TTL semantics and the expiry boundary
are unchanged (`created_at + ttl == now` is not expired), and
originals/markers/retrieval, persistence and eviction feedback are
untouched.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Three new tests in `tests/test_ccr_sqlite_backend.py`:
1. `test_purge_expired_deletes_by_deadline_and_keeps_the_boundary_row` —
mixed TTLs and the exact expiry boundary.
2.
`test_new_insert_purges_expired_sqlite_entry_without_decoding_live_payloads`
— fails the test if a new insertion decodes any existing payload; this
is the regression guard for the issue.
3. `test_opening_a_db_with_the_old_index_drops_it_and_keeps_the_rows` —
a database file written with the old index migrates on open with its
rows intact.
Tests 1 and 2 both fail on the base commit and pass with this change.
### Test Output
```text
$ pytest tests/test_ccr_sqlite_backend.py tests/test_compression_store.py -q
98 passed in 7.35s
$ pytest tests/ -q -k "ccr or compression_store or cache"
1839 passed, 8 failed
# All 8 failures (7 Playwright dashboard, 1 live-API) also fail on the base
# commit in this environment; they are unrelated to this change.
# The two new behavioral tests against the base commit:
$ git stash push headroom/cache/backends/sqlite.py headroom/cache/compression_store.py
$ pytest tests/test_ccr_sqlite_backend.py -q -k "purge_expired or purges_expired_sqlite_entry"
E Failed: new insertion decoded existing CCR payloads
2 failed, 21 deselected
$ pre-commit run --all-files # via commit hooks
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
mypy.....................................................................Passed
```
## Real Behavior Proof
- Environment: macOS 15 (arm64), Python 3.13.11, SQLite via stdlib
`sqlite3`, worktree off `upstream/main` @ `e67b3c8a`.
- Exact command / steps: `python
benchmarks/ccr_sqlite_expiry_benchmark.py --entries 1000 --payload-bytes
16384 --runs 7`, run on this branch and then with the two source files
stashed, sequentially in the same process environment with identical
inputs.
```text
candidate:
{"preexisting_entries":1000,"original_bytes_each":16384,"runs":7,"median_insert_ms":0.04654214717447758}
baseline:
{"preexisting_entries":1000,"original_bytes_each":16384,"runs":7,"median_insert_ms":17.63250003568828}
```
- Observed result: median insertion **17.63 ms → 0.047 ms**; payload
deserializations during cleanup **1,000+ → 0** (counted by instrumenting
`_entry_from_json`, and asserted permanently by test 2). Separately, on
1,000 rows × 16 KiB (n=200), dropping the old `created_at` index leaves
the purge delete unchanged at ~0.0012 ms and makes each insert ~14%
cheaper (0.0064 ms → 0.0056 ms); the new expression index is what takes
the purge from 0.171 ms to 0.0012 ms.
- Not tested: end-to-end proxy latency. This is an insertion-path
micro-measurement on synthetic entries and should not be read as an
end-to-end speedup. No measurement on very large databases (>1,000
entries) or under concurrent writers. `InMemoryBackend` is unchanged and
unmeasured.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; no flag or channel gates this.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no observable behavior change — the
same entries expire at the same times. Only the persistent backend's
schema (one index added, one dropped) and the cleanup mechanism change.
- Kill switch / disable path: none needed; a backend without
`purge_expired` transparently uses the previous code path.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the commit. The index changes are idempotent DDL
applied on open, so a reverted build re-creates `idx_ccr_expiry` and
simply stops using `idx_ccr_expiry_deadline`; cached rows are never
rewritten or dropped by the migration.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
- "Documentation update" is checked as done in the sense that no
user-facing docs describe this internal cleanup path; the new behavior
is documented in docstrings and in the schema comment explaining why the
old index is dropped.
- The `_purge_expired`/`purge_expired` split is deliberate rather than
redundant: `SQLiteBackend._lock` is a plain `threading.Lock`, and
`_maybe_purge()` already runs inside `set()`'s critical section, so it
cannot call the public lock-acquiring method.
- `purge_expired(now=None)` takes an explicit time only so the boundary
test can assert deterministically that `created_at + ttl == now` is not
expired.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
96e1dfe395 |
fix(ccr): honor workspace dir for sqlite store (#1564)
## Description CCR's default SQLite backend ignores `HEADROOM_WORKSPACE_DIR`. When users relocate Headroom's read-write state with the canonical workspace env var, the CCR store still wrote `ccr_store.db` under `~/.headroom` unless they also set `HEADROOM_CCR_SQLITE_PATH`. This change keeps `HEADROOM_CCR_SQLITE_PATH` as the strongest per-store override, then resolves the default SQLite database as `workspace_dir() / "ccr_store.db"` from `headroom.paths.workspace_dir()`. With no env vars set, `workspace_dir()` still falls back to `~/.headroom`, so the effective default remains unchanged. The default backend stays SQLite, preserving restart survival and multi-worker sharing. Closes #1558 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Route `headroom.cache.backends.sqlite.default_db_path()` through `headroom.paths.workspace_dir()` when `HEADROOM_CCR_SQLITE_PATH` is unset. - Preserve `HEADROOM_CCR_SQLITE_PATH` as the strongest override. - Keep the no-env effective fallback at `~/.headroom/ccr_store.db` through `workspace_dir()` resolution. - Update default-path wording in SQLite/compression-store/backends docs to remove unconditional fallback claims. - Add focused regression and preservation tests in `tests/test_ccr_sqlite_backend.py` for: - workspace override when `HEADROOM_WORKSPACE_DIR` is set and `HEADROOM_CCR_SQLITE_PATH` is unset. - env path override still winning. - no-env fallback to `~/.headroom`. - explicit `SQLiteBackend(db_path=...)` authority. - existing restart and two-connection behavior. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_ccr_sqlite_backend.py -k "workspace_dir or sqlite_path_env_wins or home_fallback or explicit_db_path or default_backend_is_sqlite or survives_reopen or two_connections_share_data" -v`) - [x] Linting passes (`uv run ruff check headroom/cache/backends/sqlite.py headroom/cache/compression_store.py headroom/cache/backends/__init__.py tests/test_ccr_sqlite_backend.py`) - [ ] Type checking not run (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text Base proof before the production fix: uv run pytest tests/test_ccr_sqlite_backend.py -k "workspace_dir" -v FAILED tests/test_ccr_sqlite_backend.py::TestDefaults::test_workspace_dir - AssertionError: assert 'C:\Users\Rod\AppData\Local\Temp\pytest-of-Rod\pytest-484\test_workspace_dir0\fake_home\.headroom\ccr_store.db' == 'C:\Users\Rod\AppData\Local\Temp\pytest-of-Rod\pytest-484\test_workspace_dir0\workspace\ccr_store.db' 1 failed, 20 deselected Focused validation after the fix: uv run pytest tests/test_ccr_sqlite_backend.py -k "workspace_dir or sqlite_path_env_wins or home_fallback or explicit_db_path or default_backend_is_sqlite or survives_reopen or two_connections_share_data" -v 7 passed, 14 deselected, 1 warning in 0.20s uv run ruff check headroom/cache/backends/sqlite.py headroom/cache/compression_store.py headroom/cache/backends/__init__.py tests/test_ccr_sqlite_backend.py All checks passed! ``` ## Real Behavior Proof - Environment: local pytest filesystem-path regression tests with temporary home and workspace directories. - Exact command / steps: run `uv run pytest tests/test_ccr_sqlite_backend.py -k "workspace_dir" -v` on base with the new regression test present, then run `uv run pytest tests/test_ccr_sqlite_backend.py -k "workspace_dir or sqlite_path_env_wins or home_fallback or explicit_db_path or default_backend_is_sqlite or survives_reopen or two_connections_share_data" -v` and `uv run ruff check headroom/cache/backends/sqlite.py headroom/cache/compression_store.py headroom/cache/backends/__init__.py tests/test_ccr_sqlite_backend.py` on the patched branch. - Observed result: the base proof fails because the default backend path resolves to `fake_home\\.headroom\\ccr_store.db` instead of `workspace\\ccr_store.db`; after the fix, the focused pytest selection passes, `HEADROOM_CCR_SQLITE_PATH` still wins, the no-env fallback still resolves through `~/.headroom`, explicit `db_path` remains authoritative, and `ruff check` passes. - Not tested: live proxy traffic with real CCR compression/retrieve requests, because the changed surface is the deterministic default path resolver and default backend construction. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` should remain unchanged because the repo's release automation derives changelog entries from conventional commits. |
||
|
|
b7be3814f1 |
feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description
A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### 1. Rust compressor extraction
- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.
### 2. CCR store hardening
- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).
### 3. Traffic audit tooling (measure before tuning)
- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.
### 4. Read maturation (Mechanism B) — experimental, default OFF
- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.
### 5. Rebase / CI fixups (this update)
- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s
$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed
$ mypy headroom/
Success: no issues found in 365 source files
$ python -m compileall headroom/ -q
COMPILE-OK
# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
# "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```
## Real Behavior Proof
- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
|