fix(cache): fail closed on SQLite corruption instead of deleting the store online (#4361)

## Description

Closes #4335

`SQLiteBackend._handle_db_error` used to delete the database file and
recreate it when it saw corruption. With several workers sharing the
file, that destroys data online: on Windows the unlink raised an
unhandled `PermissionError`, and on POSIX siblings kept using old
handles while this worker opened a replacement, and removing live
`-wal`/`-shm` state could discard committed originals. This PR makes
recovery fail closed instead:

1. **No online deletion.** On corruption the instance closes only its
own connection and goes offline. The database and its `-wal`/`-shm`
sidecars are left in place for offline repair, so sibling workers are
unaffected.
2. **Bounded reconnect.** While offline, operations raise
`StoreUnavailableError` internally (a miss for reads). The connection is
retried with backoff from 1s doubling to 60s, and comes back once the
file has been repaired or removed.
3. **Writes report failure.** Following `main`'s contract,
`SQLiteBackend.set()` returns `False` whenever the row was not committed
(offline, transient busy/locked, or corruption during the post-write
purge), so `CompressionStore.store` raises `StorageError` and producers
keep the original.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/cache/backends/sqlite.py`: fail-closed `_handle_db_error`,
`_connection()` with bounded reopen backoff, `set()` returns
`True`/`False`.
- `tests/test_ccr_sqlite_backend.py`: multi-connection test that
originals survive the recovery boundary, a guard that recovery never
unlinks the store or sidecars, backoff and recovery tests, and
producer-level outage tests (`TestProducerDuringOutage`).
- `base.py`, `compression_store.py` and `backends/__init__.py` are
unchanged relative to `main`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_ccr_sqlite_backend.py
38 passed in 1.00s
$ ruff check headroom/cache tests/test_ccr_sqlite_backend.py
All checks passed!
```

## Real Behavior Proof

- Environment: Linux, Python 3.13.5, at 568fe40 (current `main` merged
in).
- Exact command / steps: `python3 -m pytest
tests/test_ccr_sqlite_backend.py -q`
- Observed result: `38 passed`. Covered there: two real `SQLiteBackend`
instances on one WAL database; after corruption recovery on one, the
main file and the live `-wal`/`-shm` sidecars still exist, the
recovering instance misses reads and `set()` returns `False`, the
sibling still reads both originals, and an unlink guard on `Path` proves
recovery never deletes the store or a sidecar. With every handle closed
and the files removed offline, the recovering instance reconnects after
the backoff and writes again. Backoff, reopen-failure and
producer-outage (`StorageError`) paths are covered by their own tests.
- Not tested here: Windows. The two multi-worker tests now close every
connection before the simulated offline repair and no longer write into
the memory-mapped `-shm`, which were the two Windows failures reported
in review, but I have no Windows machine to run them on. Real corruption
of an on-disk file is simulated by calling `_handle_db_error` with the
SQLite corruption message rather than by damaging the file.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only the previously-crashing error
path — healthy-path behavior is unchanged
- Kill switch / disable path: n/a
- Unsafe override required: none
- Qualification impact: none (no new dependencies or runtime
requirements)
- Rollback path: revert the PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Sent by an AI coding agent (OpenCode) on behalf of Yi-111-a.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4361"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Yi-111-a <yi-111-a@example.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
yi111
2026-11-03 22:56:31 -06:00
committed by GitHub
co-authored by Yi-111-a JD Davis
parent d232d097cf
commit 98018333f9
2 changed files with 419 additions and 31 deletions
+124 -27
View File
@@ -57,6 +57,20 @@ _PURGE_INTERVAL = 60.0
# hour; further ones log at debug only. A row that stays broken warns again
# about once an hour, which keeps it visible without flooding.
_MAX_REPORTED_UNREADABLE = 1000
# After a failed reopen, retry on the next operation once this many seconds
# have passed, doubling per consecutive failure up to the cap. Bounded so a
# store whose filesystem problem has cleared comes back on its own, without
# hammering a still-broken path on every call.
_REOPEN_BACKOFF_INITIAL = 1.0
_REOPEN_BACKOFF_MAX = 60.0
class StoreUnavailableError(sqlite3.OperationalError):
"""The connection is down after a failed reopen and the retry is not due.
Subclasses ``sqlite3.OperationalError`` so every existing
``except sqlite3.DatabaseError`` path degrades it to a miss.
"""
def default_db_path() -> Path:
@@ -92,7 +106,9 @@ class SQLiteBackend:
self._unreadable_warned = WarnOnce(
_MAX_REPORTED_UNREADABLE, "unreadable CCR SQLite entries"
)
self._conn = self._open()
self._conn: sqlite3.Connection | None = self._open()
self._reopen_backoff = _REOPEN_BACKOFF_INITIAL
self._next_reopen_at = 0.0
@staticmethod
def _ensure_private(path: Path) -> None:
@@ -138,6 +154,32 @@ class SQLiteBackend:
pass
return conn
def _connection(self) -> sqlite3.Connection:
"""Return the live connection, reopening it if a retry is due.
Caller holds ``_lock``. Raises :class:`StoreUnavailableError` while the
store is down: reads degrade to a miss and ``set()`` returns ``False``
to the producer.
"""
if self._conn is not None:
return self._conn
now = time.monotonic()
if now < self._next_reopen_at:
raise StoreUnavailableError(f"CCR SQLite store {self._path} is unavailable")
try:
self._conn = self._open()
except Exception as e: # noqa: BLE001 - any open failure keeps the store down
self._schedule_reopen(now)
raise StoreUnavailableError(f"CCR SQLite store {self._path} reopen failed: {e}") from e
self._reopen_backoff = _REOPEN_BACKOFF_INITIAL
self._next_reopen_at = 0.0
logger.warning("CCR SQLite store %s reopened; storage is available again.", self._path)
return self._conn
def _schedule_reopen(self, now: float) -> None:
self._next_reopen_at = now + self._reopen_backoff
self._reopen_backoff = min(self._reopen_backoff * 2, _REOPEN_BACKOFF_MAX)
@staticmethod
def _is_corruption(error: Exception) -> bool:
"""Only genuine file corruption justifies recreating the database.
@@ -152,30 +194,47 @@ class SQLiteBackend:
return "malformed" in msg or "not a database" in msg
def _handle_db_error(self, error: sqlite3.DatabaseError, op: str) -> None:
"""Corruption → recreate (loud). Anything else (busy/locked/io) →
log and treat the operation as a miss; never destroy data over a
transient error."""
"""Corruption → fail closed (loud). Anything else (busy/locked/io) →
log and treat the operation as a miss; never destroy data."""
if isinstance(error, StoreUnavailableError):
logger.warning(
"CCR SQLite %s skipped: %s (nothing was read or persisted; "
"will retry the connection)",
op,
error,
)
return
if not self._is_corruption(error):
logger.warning(
"CCR SQLite %s failed on %s (transient, no reset): %s", op, self._path, error
)
return
logger.warning(
"CCR SQLite store at %s is corrupt (%s); recreating. "
"Previously stored originals are lost — affected retrieval "
"markers will miss until their content is re-compressed.",
"CCR SQLite store at %s is corrupt (%s); going offline without "
"touching the file. The database and its -wal/-shm sidecars are "
"preserved for offline recovery: sibling workers may still hold "
"live handles, so an online recreate would split-brain their "
"reads/writes or discard committed originals from old WAL state. "
"A later operation retries the connection after the backoff; the "
"retry succeeds once the file has been repaired or removed offline. "
"Until then nothing is read or persisted through this instance.",
self._path,
error,
)
try:
self._conn.close()
if self._conn is not None:
self._conn.close()
except Exception: # noqa: BLE001 - best-effort close on corrupt handle
pass
self._path.unlink(missing_ok=True)
self._conn = None
self._unreadable_warned = WarnOnce(
_MAX_REPORTED_UNREADABLE, "unreadable CCR SQLite entries"
)
self._conn = self._open()
# Reopen on the next operation once the backoff allows it. Reopening a
# still-corrupt file raises again inside ``_open()`` (schema run hits
# the corruption), which re-enters the fail-closed state; recovery is
# always offline, by the operator.
self._schedule_reopen(time.monotonic())
def _report_unreadable(self, hash_key: str, reason: object) -> None:
first = self._unreadable_warned.first(hash_key, logger)
@@ -212,11 +271,12 @@ class SQLiteBackend:
def _purge_expired(self, now: float) -> int:
"""Delete expired rows using metadata; caller holds ``_lock``."""
cursor = self._conn.execute(
conn = self._connection()
cursor = conn.execute(
"DELETE FROM ccr_entries WHERE created_at + ttl < ?",
(now,),
)
self._conn.commit()
conn.commit()
self._last_purge = now
return cursor.rowcount
@@ -239,7 +299,8 @@ class SQLiteBackend:
def get(self, hash_key: str) -> CompressionEntry | None:
with self._lock:
try:
row = self._conn.execute(
conn = self._connection()
row = conn.execute(
"SELECT entry_json FROM ccr_entries WHERE hash = ?",
(hash_key,),
).fetchone()
@@ -250,28 +311,57 @@ class SQLiteBackend:
return None
return self._entry_from_json(row[0], hash_key)
def set(self, hash_key: str, entry: CompressionEntry) -> None:
def set(self, hash_key: str, entry: CompressionEntry) -> bool | None:
"""Persist ``entry``; return ``False`` if it was not.
Unlike reads, a failed write must not look like success: the caller
is about to hand out ``hash_key`` in place of the original, so it has
to learn that the original is not retrievable and keep it instead.
``CompressionStore.store`` turns the ``False`` into ``StorageError``.
"""
payload = json.dumps(asdict(entry), ensure_ascii=False)
with self._lock:
try:
self._conn.execute(
conn = self._connection()
conn.execute(
"INSERT OR REPLACE INTO ccr_entries "
"(hash, entry_json, created_at, ttl) VALUES (?, ?, ?, ?)",
(hash_key, payload, entry.created_at, entry.ttl),
)
self._conn.commit()
self._maybe_purge()
conn.commit()
except sqlite3.DatabaseError as e:
self._handle_db_error(e, f"set {safe_id(hash_key)}")
logger.warning(
"CCR SQLite write of %s was not persisted; the producer "
"must keep the original content",
safe_id(hash_key),
)
return False
try:
self._maybe_purge()
except sqlite3.DatabaseError as e:
self._handle_db_error(e, "purge")
if self._is_corruption(e):
# The store went offline mid-write; whether the row just
# committed is still readable is unknown, so report the
# write as lost rather than hand out its hash.
logger.warning(
"CCR SQLite write of %s was lost when the store went "
"offline during the post-write purge",
safe_id(hash_key),
)
return False
return True
def delete(self, hash_key: str) -> bool:
with self._lock:
try:
cur = self._conn.execute(
conn = self._connection()
cur = conn.execute(
"DELETE FROM ccr_entries WHERE hash = ?",
(hash_key,),
)
self._conn.commit()
conn.commit()
return cur.rowcount > 0
except sqlite3.DatabaseError as e:
self._handle_db_error(e, f"delete {safe_id(hash_key)}")
@@ -280,7 +370,8 @@ class SQLiteBackend:
def exists(self, hash_key: str) -> bool:
with self._lock:
try:
row = self._conn.execute(
conn = self._connection()
row = conn.execute(
"SELECT 1 FROM ccr_entries WHERE hash = ?",
(hash_key,),
).fetchone()
@@ -292,15 +383,17 @@ class SQLiteBackend:
def clear(self) -> None:
with self._lock:
try:
self._conn.execute("DELETE FROM ccr_entries")
self._conn.commit()
conn = self._connection()
conn.execute("DELETE FROM ccr_entries")
conn.commit()
except sqlite3.DatabaseError as e:
self._handle_db_error(e, "clear")
def count(self) -> int:
with self._lock:
try:
row = self._conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
conn = self._connection()
row = conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
except sqlite3.DatabaseError as e:
self._handle_db_error(e, "count")
return 0
@@ -310,7 +403,8 @@ class SQLiteBackend:
"""Return a token that changes when another connection commits."""
with self._lock:
try:
row = self._conn.execute("PRAGMA data_version").fetchone()
conn = self._connection()
row = conn.execute("PRAGMA data_version").fetchone()
except sqlite3.DatabaseError as e:
self._handle_db_error(e, "revision")
return None
@@ -319,7 +413,8 @@ class SQLiteBackend:
def keys(self) -> list[str]:
with self._lock:
try:
rows = self._conn.execute("SELECT hash FROM ccr_entries").fetchall()
conn = self._connection()
rows = conn.execute("SELECT hash FROM ccr_entries").fetchall()
except sqlite3.DatabaseError as e:
self._handle_db_error(e, "keys")
return []
@@ -328,7 +423,8 @@ class SQLiteBackend:
def items(self) -> list[tuple[str, CompressionEntry]]:
with self._lock:
try:
rows = self._conn.execute("SELECT hash, entry_json FROM ccr_entries").fetchall()
conn = self._connection()
rows = conn.execute("SELECT hash, entry_json FROM ccr_entries").fetchall()
except sqlite3.DatabaseError as e:
self._handle_db_error(e, "items")
return []
@@ -342,7 +438,8 @@ class SQLiteBackend:
def get_stats(self) -> dict[str, Any]:
with self._lock:
try:
count_row = self._conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
conn = self._connection()
count_row = conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
except sqlite3.DatabaseError as e:
self._handle_db_error(e, "stats")
count_row = (0,)
+295 -4
View File
@@ -16,6 +16,7 @@ import pytest
from headroom.cache.backends.sqlite import SQLiteBackend
from headroom.cache.compression_store import CompressionEntry, CompressionStore
from headroom.exceptions import StorageError
def make_entry(hash_key: str = "h1", content: str = "x" * 600, ttl: int = 1800) -> CompressionEntry:
@@ -383,6 +384,14 @@ class TestSQLiteBackend:
assert got.retrieval_count == 1
def _close(backend: SQLiteBackend) -> None:
"""Close a backend's connection, if any (tests only; there is no public close)."""
conn = backend._conn
backend._conn = None
if conn is not None:
conn.close()
class TestMultiWorkerSafety:
def test_busy_error_does_not_delete_database(self, db_path):
"""SQLITE_BUSY (OperationalError, a DatabaseError subclass) under
@@ -416,15 +425,207 @@ class TestMultiWorkerSafety:
assert db_path.exists()
assert b.get("h1") is not None
def test_corruption_message_triggers_reset(self, db_path):
def test_corruption_fails_closed_and_siblings_keep_the_data(self, db_path):
"""Corruption recovery must not destroy anything online: a sibling
worker may still hold live handles, so deleting the main file or its
-wal/-shm sidecars would split-brain the sibling or discard committed
originals. Fail closed instead — this instance goes offline, the file
survives for offline repair, and siblings are unaffected."""
b = SQLiteBackend(db_path)
assert b.set("h1", make_entry()) is True
sibling = SQLiteBackend(db_path)
assert sibling.set("h2", make_entry("h2")) is True
import headroom.cache.backends.sqlite as sqlite_mod
# Same monotonic clock monkey as the recover tests below use; the
# reconnect retry follows the shared bounded backoff.
clock = [1000.0]
monkeypatch = pytest.MonkeyPatch()
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
try:
b._handle_db_error(sqlite3.DatabaseError("database disk image is malformed"), "get")
# Nothing was deleted: main file and every sidecar survive.
assert db_path.exists()
for suffix in ("-wal", "-shm"):
sidecar = db_path.parent / f"{db_path.name}{suffix}"
assert sidecar.exists()
# This instance is down: reads miss, writes are refused loudly.
assert b.get("h1") is None
assert b.set("h3", make_entry("h3")) is False
# The sibling, still connected to the preserved file, keeps serving
# the originals that were stored before the recovery boundary.
assert sibling.get("h1") is not None
assert sibling.get("h2") is not None
# Offline repair happens with every worker stopped: close the
# sibling's handle first (Windows refuses to unlink an open file),
# then the operator removes the corrupt file and its sidecars.
_close(sibling)
for suffix in ("", "-wal", "-shm"):
p = db_path if not suffix else db_path.parent / f"{db_path.name}{suffix}"
p.unlink(missing_ok=True)
# Once the retry is due, this instance reconnects to a fresh database.
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
assert b.set("h4", make_entry("h4")) is True
assert b.get("h4") is not None
finally:
monkeypatch.undo()
_close(sibling)
_close(b)
def test_corruption_recovery_never_unlinks_the_store(self, db_path, monkeypatch):
"""The recovery path must not delete the main file or any sidecar:
sibling workers may still hold live handles, and the survivors are
needed for offline repair."""
b = SQLiteBackend(db_path)
# A live sibling keeps the real WAL-mode -wal/-shm sidecars in place;
# they are checked as-is rather than overwritten, because the -shm file
# is memory-mapped while a connection is open (Windows rejects writes).
sibling = SQLiteBackend(db_path)
try:
assert b.set("h1", make_entry()) is True
sidecars = [db_path.parent / f"{db_path.name}{suffix}" for suffix in ("-wal", "-shm")]
for sidecar in sidecars:
assert sidecar.exists()
real_unlink = type(db_path).unlink
def refuse(self, *args, **kwargs):
if self.parent == db_path.parent and self.name.startswith(db_path.name):
raise AssertionError(
"corruption recovery must not unlink the store or its sidecars"
)
return real_unlink(self, *args, **kwargs)
monkeypatch.setattr(type(db_path), "unlink", refuse)
b._handle_db_error(sqlite3.DatabaseError("file is not a database"), "get")
monkeypatch.undo()
assert db_path.exists()
for sidecar in sidecars:
assert sidecar.exists()
assert sibling.get("h1") is not None
finally:
_close(sibling)
_close(b)
def test_failed_reopen_after_recovery_recovers_later(self, db_path, monkeypatch, caplog):
"""A reopen that fails while the store is offline must not leave the
store dead for the process lifetime: once the retry is due and the
open succeeds, writes persist and reads hit again."""
import headroom.cache.backends.sqlite as sqlite_mod
b = SQLiteBackend(db_path)
b.set("h1", make_entry())
real_open = b._open
clock = [1000.0]
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
def broken_open():
raise sqlite3.OperationalError("unable to open database file")
monkeypatch.setattr(b, "_open", broken_open)
with caplog.at_level(logging.WARNING):
b._handle_db_error(sqlite3.DatabaseError("database disk image is malformed"), "get")
# Down and the retry is not due yet: a miss, and the refused
# write reports False rather than succeeding silently.
assert b.get("h1") is None
assert b.set("h2", make_entry("h2")) is False
assert "nothing was read or persisted" in caplog.text
# The retry is due but the open still fails: still a miss, and
# the next retry is pushed further out.
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
assert b.get("h1") is None
# The filesystem problem clears and the backoff elapses.
monkeypatch.setattr(b, "_open", real_open)
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
assert b.set("h3", make_entry("h3")) is True
got = b.get("h3")
assert got is not None
assert got.hash == "h3"
def test_reopen_retry_backs_off_while_open_keeps_failing(self, db_path, monkeypatch):
import headroom.cache.backends.sqlite as sqlite_mod
b = SQLiteBackend(db_path)
clock = [1000.0]
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
attempts = []
def broken_open():
attempts.append(clock[0])
raise sqlite3.OperationalError("unable to open database file")
monkeypatch.setattr(b, "_open", broken_open)
b._handle_db_error(sqlite3.DatabaseError("file is not a database"), "get")
assert len(attempts) == 0 # recovery only schedules; no open yet
clock[0] += sqlite_mod._REOPEN_BACKOFF_INITIAL
assert b.get("h1") is None
assert len(attempts) == 1
for _ in range(5):
assert b.get("h1") is None # not due: no extra open attempts
assert len(attempts) == 1
clock[0] += sqlite_mod._REOPEN_BACKOFF_INITIAL
assert b.get("h1") is None # the wait doubled: not due yet
assert len(attempts) == 1
clock[0] += sqlite_mod._REOPEN_BACKOFF_INITIAL
assert b.get("h1") is None
assert len(attempts) == 2
# The wait doubled again, so one more initial interval is not enough.
clock[0] += sqlite_mod._REOPEN_BACKOFF_INITIAL
assert b.get("h1") is None
assert len(attempts) == 2
def test_reopen_failure_recovery_ends_the_offline_state(self, db_path, monkeypatch):
import headroom.cache.backends.sqlite as sqlite_mod
b = SQLiteBackend(db_path)
b.set("h1", make_entry())
real_open = b._open
clock = [1000.0]
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
def broken_open():
raise sqlite3.OperationalError("unable to open database file")
monkeypatch.setattr(b, "_open", broken_open)
b._handle_db_error(sqlite3.DatabaseError("database disk image is malformed"), "get")
# Database recreated: empty but functional.
assert b.count() == 0
b.set("h2", make_entry("h2"))
assert b.get("h1") is None
monkeypatch.setattr(b, "_open", real_open)
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
assert b.set("h2", make_entry("h2")) is True
assert b.exists("h2")
def test_transient_write_failure_returns_false_and_keeps_the_database(self, db_path):
"""A write lost to a transient error (busy/locked) is still lost:
set() must report it, and must not reset the database."""
b = SQLiteBackend(db_path)
assert b.set("h1", make_entry()) is True
class LockedInsertConn:
def __init__(self, real):
self._real = real
def execute(self, *args, **kwargs):
if args and "INSERT" in args[0]:
raise sqlite3.OperationalError("database is locked")
return self._real.execute(*args, **kwargs)
def __getattr__(self, name):
return getattr(self._real, name)
real = b._conn
b._conn = LockedInsertConn(real) # type: ignore[assignment]
assert b.set("h2", make_entry("h2")) is False
b._conn = real
assert db_path.exists()
assert b.get("h1") is not None
assert b.get("h2") is None
def test_busy_timeout_configured(self, db_path):
b = SQLiteBackend(db_path)
timeout = b._conn.execute("PRAGMA busy_timeout").fetchone()[0]
@@ -529,5 +730,95 @@ class TestDefaults:
assert "re-run" in CCR_MISS_MESSAGE
COMMENTED_TOML = (
"".join(f"# note {i}: keep this setting in sync with the deploy runbook\n" for i in range(30))
+ '[package]\nname = "demo"\nversion = "0.1.0"\n\n[dependencies]\nserde = "1.0"\n'
)
class TestProducerDuringOutage:
"""The producer-facing contract while the backend is down after a failed
corruption-recovery reopen: writes fail loudly so producers keep the
original, reads stay fail-open, and both work again after recovery."""
@pytest.fixture
def outage(self, db_path, monkeypatch):
import headroom.cache.backends.sqlite as sqlite_mod
backend = SQLiteBackend(db_path)
real_open = backend._open
clock = [1000.0]
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
def broken_open():
raise sqlite3.OperationalError("unable to open database file")
monkeypatch.setattr(backend, "_open", broken_open)
backend._handle_db_error(sqlite3.DatabaseError("database disk image is malformed"), "get")
def recover():
monkeypatch.setattr(backend, "_open", real_open)
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
return backend, recover
def test_compression_store_raises_and_recovers(self, outage):
backend, recover = outage
store = CompressionStore(backend=backend, enable_feedback=False)
with pytest.raises(StorageError):
store.store("original " * 100, "compressed")
# Reads stay fail-open: a miss, not an exception.
assert store.retrieve("0" * 24) is None
recover()
hash_key = store.store("original " * 100, "compressed")
entry = store.retrieve(hash_key)
assert entry is not None
assert entry.original_content == "original " * 100
def test_retrieve_survives_failed_access_count_write(self, db_path):
backend = SQLiteBackend(db_path)
store = CompressionStore(backend=backend, enable_feedback=False)
hash_key = store.store("original " * 100, "compressed")
def failing_set(k, entry):
return False
backend.set = failing_set # type: ignore[method-assign]
entry = store.retrieve(hash_key)
assert entry is not None
assert entry.original_content == "original " * 100
def test_config_compressor_keeps_original_during_outage(self, outage, monkeypatch):
from headroom.parser import CCR_RETRIEVAL_MARKER_RE
from headroom.transforms.config_compressor import (
ConfigCompressor,
ConfigCompressorConfig,
)
backend, recover = outage
store = CompressionStore(backend=backend, enable_feedback=False)
monkeypatch.setattr("headroom.cache.compression_store.get_compression_store", lambda: store)
comp = ConfigCompressor(ConfigCompressorConfig(enable_ccr=True))
result = comp.compress(COMMENTED_TOML)
# No hash was issued for an original that was never stored, so all
# 30 comments stay in the output.
assert result.ccr_hash is None
assert result.lines_elided == 0
assert not CCR_RETRIEVAL_MARKER_RE.search(result.compressed)
for i in range(30):
assert f"# note {i}:" in result.compressed
recover()
result = comp.compress(COMMENTED_TOML)
assert result.ccr_hash is not None
assert "# note 0:" not in result.compressed
entry = store.retrieve(result.ccr_hash)
assert entry is not None
assert entry.original_content == COMMENTED_TOML
if __name__ == "__main__":
pytest.main([__file__, "-v"])