mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(cache): fail closed on SQLite corruption instead of deleting the store online (#4361)
## Description
Closes #4335
`SQLiteBackend._handle_db_error` used to delete the database file and
recreate it when it saw corruption. With several workers sharing the
file, that destroys data online: on Windows the unlink raised an
unhandled `PermissionError`, and on POSIX siblings kept using old
handles while this worker opened a replacement, and removing live
`-wal`/`-shm` state could discard committed originals. This PR makes
recovery fail closed instead:
1. **No online deletion.** On corruption the instance closes only its
own connection and goes offline. The database and its `-wal`/`-shm`
sidecars are left in place for offline repair, so sibling workers are
unaffected.
2. **Bounded reconnect.** While offline, operations raise
`StoreUnavailableError` internally (a miss for reads). The connection is
retried with backoff from 1s doubling to 60s, and comes back once the
file has been repaired or removed.
3. **Writes report failure.** Following `main`'s contract,
`SQLiteBackend.set()` returns `False` whenever the row was not committed
(offline, transient busy/locked, or corruption during the post-write
purge), so `CompressionStore.store` raises `StorageError` and producers
keep the original.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cache/backends/sqlite.py`: fail-closed `_handle_db_error`,
`_connection()` with bounded reopen backoff, `set()` returns
`True`/`False`.
- `tests/test_ccr_sqlite_backend.py`: multi-connection test that
originals survive the recovery boundary, a guard that recovery never
unlinks the store or sidecars, backoff and recovery tests, and
producer-level outage tests (`TestProducerDuringOutage`).
- `base.py`, `compression_store.py` and `backends/__init__.py` are
unchanged relative to `main`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_ccr_sqlite_backend.py
38 passed in 1.00s
$ ruff check headroom/cache tests/test_ccr_sqlite_backend.py
All checks passed!
```
## Real Behavior Proof
- Environment: Linux, Python 3.13.5, at 568fe40 (current `main` merged
in).
- Exact command / steps: `python3 -m pytest
tests/test_ccr_sqlite_backend.py -q`
- Observed result: `38 passed`. Covered there: two real `SQLiteBackend`
instances on one WAL database; after corruption recovery on one, the
main file and the live `-wal`/`-shm` sidecars still exist, the
recovering instance misses reads and `set()` returns `False`, the
sibling still reads both originals, and an unlink guard on `Path` proves
recovery never deletes the store or a sidecar. With every handle closed
and the files removed offline, the recovering instance reconnects after
the backoff and writes again. Backoff, reopen-failure and
producer-outage (`StorageError`) paths are covered by their own tests.
- Not tested here: Windows. The two multi-worker tests now close every
connection before the simulated offline repair and no longer write into
the memory-mapped `-shm`, which were the two Windows failures reported
in review, but I have no Windows machine to run them on. Real corruption
of an on-disk file is simulated by calling `_handle_db_error` with the
SQLite corruption message rather than by damaging the file.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only the previously-crashing error
path — healthy-path behavior is unchanged
- Kill switch / disable path: n/a
- Unsafe override required: none
- Qualification impact: none (no new dependencies or runtime
requirements)
- Rollback path: revert the PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Sent by an AI coding agent (OpenCode) on behalf of Yi-111-a.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4361"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Yi-111-a <yi-111-a@example.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Yi-111-a
JD Davis
parent
d232d097cf
commit
98018333f9
Vendored
+124
-27
@@ -57,6 +57,20 @@ _PURGE_INTERVAL = 60.0
|
||||
# hour; further ones log at debug only. A row that stays broken warns again
|
||||
# about once an hour, which keeps it visible without flooding.
|
||||
_MAX_REPORTED_UNREADABLE = 1000
|
||||
# After a failed reopen, retry on the next operation once this many seconds
|
||||
# have passed, doubling per consecutive failure up to the cap. Bounded so a
|
||||
# store whose filesystem problem has cleared comes back on its own, without
|
||||
# hammering a still-broken path on every call.
|
||||
_REOPEN_BACKOFF_INITIAL = 1.0
|
||||
_REOPEN_BACKOFF_MAX = 60.0
|
||||
|
||||
|
||||
class StoreUnavailableError(sqlite3.OperationalError):
|
||||
"""The connection is down after a failed reopen and the retry is not due.
|
||||
|
||||
Subclasses ``sqlite3.OperationalError`` so every existing
|
||||
``except sqlite3.DatabaseError`` path degrades it to a miss.
|
||||
"""
|
||||
|
||||
|
||||
def default_db_path() -> Path:
|
||||
@@ -92,7 +106,9 @@ class SQLiteBackend:
|
||||
self._unreadable_warned = WarnOnce(
|
||||
_MAX_REPORTED_UNREADABLE, "unreadable CCR SQLite entries"
|
||||
)
|
||||
self._conn = self._open()
|
||||
self._conn: sqlite3.Connection | None = self._open()
|
||||
self._reopen_backoff = _REOPEN_BACKOFF_INITIAL
|
||||
self._next_reopen_at = 0.0
|
||||
|
||||
@staticmethod
|
||||
def _ensure_private(path: Path) -> None:
|
||||
@@ -138,6 +154,32 @@ class SQLiteBackend:
|
||||
pass
|
||||
return conn
|
||||
|
||||
def _connection(self) -> sqlite3.Connection:
|
||||
"""Return the live connection, reopening it if a retry is due.
|
||||
|
||||
Caller holds ``_lock``. Raises :class:`StoreUnavailableError` while the
|
||||
store is down: reads degrade to a miss and ``set()`` returns ``False``
|
||||
to the producer.
|
||||
"""
|
||||
if self._conn is not None:
|
||||
return self._conn
|
||||
now = time.monotonic()
|
||||
if now < self._next_reopen_at:
|
||||
raise StoreUnavailableError(f"CCR SQLite store {self._path} is unavailable")
|
||||
try:
|
||||
self._conn = self._open()
|
||||
except Exception as e: # noqa: BLE001 - any open failure keeps the store down
|
||||
self._schedule_reopen(now)
|
||||
raise StoreUnavailableError(f"CCR SQLite store {self._path} reopen failed: {e}") from e
|
||||
self._reopen_backoff = _REOPEN_BACKOFF_INITIAL
|
||||
self._next_reopen_at = 0.0
|
||||
logger.warning("CCR SQLite store %s reopened; storage is available again.", self._path)
|
||||
return self._conn
|
||||
|
||||
def _schedule_reopen(self, now: float) -> None:
|
||||
self._next_reopen_at = now + self._reopen_backoff
|
||||
self._reopen_backoff = min(self._reopen_backoff * 2, _REOPEN_BACKOFF_MAX)
|
||||
|
||||
@staticmethod
|
||||
def _is_corruption(error: Exception) -> bool:
|
||||
"""Only genuine file corruption justifies recreating the database.
|
||||
@@ -152,30 +194,47 @@ class SQLiteBackend:
|
||||
return "malformed" in msg or "not a database" in msg
|
||||
|
||||
def _handle_db_error(self, error: sqlite3.DatabaseError, op: str) -> None:
|
||||
"""Corruption → recreate (loud). Anything else (busy/locked/io) →
|
||||
log and treat the operation as a miss; never destroy data over a
|
||||
transient error."""
|
||||
"""Corruption → fail closed (loud). Anything else (busy/locked/io) →
|
||||
log and treat the operation as a miss; never destroy data."""
|
||||
if isinstance(error, StoreUnavailableError):
|
||||
logger.warning(
|
||||
"CCR SQLite %s skipped: %s (nothing was read or persisted; "
|
||||
"will retry the connection)",
|
||||
op,
|
||||
error,
|
||||
)
|
||||
return
|
||||
if not self._is_corruption(error):
|
||||
logger.warning(
|
||||
"CCR SQLite %s failed on %s (transient, no reset): %s", op, self._path, error
|
||||
)
|
||||
return
|
||||
logger.warning(
|
||||
"CCR SQLite store at %s is corrupt (%s); recreating. "
|
||||
"Previously stored originals are lost — affected retrieval "
|
||||
"markers will miss until their content is re-compressed.",
|
||||
"CCR SQLite store at %s is corrupt (%s); going offline without "
|
||||
"touching the file. The database and its -wal/-shm sidecars are "
|
||||
"preserved for offline recovery: sibling workers may still hold "
|
||||
"live handles, so an online recreate would split-brain their "
|
||||
"reads/writes or discard committed originals from old WAL state. "
|
||||
"A later operation retries the connection after the backoff; the "
|
||||
"retry succeeds once the file has been repaired or removed offline. "
|
||||
"Until then nothing is read or persisted through this instance.",
|
||||
self._path,
|
||||
error,
|
||||
)
|
||||
try:
|
||||
self._conn.close()
|
||||
if self._conn is not None:
|
||||
self._conn.close()
|
||||
except Exception: # noqa: BLE001 - best-effort close on corrupt handle
|
||||
pass
|
||||
self._path.unlink(missing_ok=True)
|
||||
self._conn = None
|
||||
self._unreadable_warned = WarnOnce(
|
||||
_MAX_REPORTED_UNREADABLE, "unreadable CCR SQLite entries"
|
||||
)
|
||||
self._conn = self._open()
|
||||
# Reopen on the next operation once the backoff allows it. Reopening a
|
||||
# still-corrupt file raises again inside ``_open()`` (schema run hits
|
||||
# the corruption), which re-enters the fail-closed state; recovery is
|
||||
# always offline, by the operator.
|
||||
self._schedule_reopen(time.monotonic())
|
||||
|
||||
def _report_unreadable(self, hash_key: str, reason: object) -> None:
|
||||
first = self._unreadable_warned.first(hash_key, logger)
|
||||
@@ -212,11 +271,12 @@ class SQLiteBackend:
|
||||
|
||||
def _purge_expired(self, now: float) -> int:
|
||||
"""Delete expired rows using metadata; caller holds ``_lock``."""
|
||||
cursor = self._conn.execute(
|
||||
conn = self._connection()
|
||||
cursor = conn.execute(
|
||||
"DELETE FROM ccr_entries WHERE created_at + ttl < ?",
|
||||
(now,),
|
||||
)
|
||||
self._conn.commit()
|
||||
conn.commit()
|
||||
self._last_purge = now
|
||||
return cursor.rowcount
|
||||
|
||||
@@ -239,7 +299,8 @@ class SQLiteBackend:
|
||||
def get(self, hash_key: str) -> CompressionEntry | None:
|
||||
with self._lock:
|
||||
try:
|
||||
row = self._conn.execute(
|
||||
conn = self._connection()
|
||||
row = conn.execute(
|
||||
"SELECT entry_json FROM ccr_entries WHERE hash = ?",
|
||||
(hash_key,),
|
||||
).fetchone()
|
||||
@@ -250,28 +311,57 @@ class SQLiteBackend:
|
||||
return None
|
||||
return self._entry_from_json(row[0], hash_key)
|
||||
|
||||
def set(self, hash_key: str, entry: CompressionEntry) -> None:
|
||||
def set(self, hash_key: str, entry: CompressionEntry) -> bool | None:
|
||||
"""Persist ``entry``; return ``False`` if it was not.
|
||||
|
||||
Unlike reads, a failed write must not look like success: the caller
|
||||
is about to hand out ``hash_key`` in place of the original, so it has
|
||||
to learn that the original is not retrievable and keep it instead.
|
||||
``CompressionStore.store`` turns the ``False`` into ``StorageError``.
|
||||
"""
|
||||
payload = json.dumps(asdict(entry), ensure_ascii=False)
|
||||
with self._lock:
|
||||
try:
|
||||
self._conn.execute(
|
||||
conn = self._connection()
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO ccr_entries "
|
||||
"(hash, entry_json, created_at, ttl) VALUES (?, ?, ?, ?)",
|
||||
(hash_key, payload, entry.created_at, entry.ttl),
|
||||
)
|
||||
self._conn.commit()
|
||||
self._maybe_purge()
|
||||
conn.commit()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, f"set {safe_id(hash_key)}")
|
||||
logger.warning(
|
||||
"CCR SQLite write of %s was not persisted; the producer "
|
||||
"must keep the original content",
|
||||
safe_id(hash_key),
|
||||
)
|
||||
return False
|
||||
try:
|
||||
self._maybe_purge()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "purge")
|
||||
if self._is_corruption(e):
|
||||
# The store went offline mid-write; whether the row just
|
||||
# committed is still readable is unknown, so report the
|
||||
# write as lost rather than hand out its hash.
|
||||
logger.warning(
|
||||
"CCR SQLite write of %s was lost when the store went "
|
||||
"offline during the post-write purge",
|
||||
safe_id(hash_key),
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
def delete(self, hash_key: str) -> bool:
|
||||
with self._lock:
|
||||
try:
|
||||
cur = self._conn.execute(
|
||||
conn = self._connection()
|
||||
cur = conn.execute(
|
||||
"DELETE FROM ccr_entries WHERE hash = ?",
|
||||
(hash_key,),
|
||||
)
|
||||
self._conn.commit()
|
||||
conn.commit()
|
||||
return cur.rowcount > 0
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, f"delete {safe_id(hash_key)}")
|
||||
@@ -280,7 +370,8 @@ class SQLiteBackend:
|
||||
def exists(self, hash_key: str) -> bool:
|
||||
with self._lock:
|
||||
try:
|
||||
row = self._conn.execute(
|
||||
conn = self._connection()
|
||||
row = conn.execute(
|
||||
"SELECT 1 FROM ccr_entries WHERE hash = ?",
|
||||
(hash_key,),
|
||||
).fetchone()
|
||||
@@ -292,15 +383,17 @@ class SQLiteBackend:
|
||||
def clear(self) -> None:
|
||||
with self._lock:
|
||||
try:
|
||||
self._conn.execute("DELETE FROM ccr_entries")
|
||||
self._conn.commit()
|
||||
conn = self._connection()
|
||||
conn.execute("DELETE FROM ccr_entries")
|
||||
conn.commit()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "clear")
|
||||
|
||||
def count(self) -> int:
|
||||
with self._lock:
|
||||
try:
|
||||
row = self._conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
|
||||
conn = self._connection()
|
||||
row = conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "count")
|
||||
return 0
|
||||
@@ -310,7 +403,8 @@ class SQLiteBackend:
|
||||
"""Return a token that changes when another connection commits."""
|
||||
with self._lock:
|
||||
try:
|
||||
row = self._conn.execute("PRAGMA data_version").fetchone()
|
||||
conn = self._connection()
|
||||
row = conn.execute("PRAGMA data_version").fetchone()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "revision")
|
||||
return None
|
||||
@@ -319,7 +413,8 @@ class SQLiteBackend:
|
||||
def keys(self) -> list[str]:
|
||||
with self._lock:
|
||||
try:
|
||||
rows = self._conn.execute("SELECT hash FROM ccr_entries").fetchall()
|
||||
conn = self._connection()
|
||||
rows = conn.execute("SELECT hash FROM ccr_entries").fetchall()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "keys")
|
||||
return []
|
||||
@@ -328,7 +423,8 @@ class SQLiteBackend:
|
||||
def items(self) -> list[tuple[str, CompressionEntry]]:
|
||||
with self._lock:
|
||||
try:
|
||||
rows = self._conn.execute("SELECT hash, entry_json FROM ccr_entries").fetchall()
|
||||
conn = self._connection()
|
||||
rows = conn.execute("SELECT hash, entry_json FROM ccr_entries").fetchall()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "items")
|
||||
return []
|
||||
@@ -342,7 +438,8 @@ class SQLiteBackend:
|
||||
def get_stats(self) -> dict[str, Any]:
|
||||
with self._lock:
|
||||
try:
|
||||
count_row = self._conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
|
||||
conn = self._connection()
|
||||
count_row = conn.execute("SELECT COUNT(*) FROM ccr_entries").fetchone()
|
||||
except sqlite3.DatabaseError as e:
|
||||
self._handle_db_error(e, "stats")
|
||||
count_row = (0,)
|
||||
|
||||
@@ -16,6 +16,7 @@ import pytest
|
||||
|
||||
from headroom.cache.backends.sqlite import SQLiteBackend
|
||||
from headroom.cache.compression_store import CompressionEntry, CompressionStore
|
||||
from headroom.exceptions import StorageError
|
||||
|
||||
|
||||
def make_entry(hash_key: str = "h1", content: str = "x" * 600, ttl: int = 1800) -> CompressionEntry:
|
||||
@@ -383,6 +384,14 @@ class TestSQLiteBackend:
|
||||
assert got.retrieval_count == 1
|
||||
|
||||
|
||||
def _close(backend: SQLiteBackend) -> None:
|
||||
"""Close a backend's connection, if any (tests only; there is no public close)."""
|
||||
conn = backend._conn
|
||||
backend._conn = None
|
||||
if conn is not None:
|
||||
conn.close()
|
||||
|
||||
|
||||
class TestMultiWorkerSafety:
|
||||
def test_busy_error_does_not_delete_database(self, db_path):
|
||||
"""SQLITE_BUSY (OperationalError, a DatabaseError subclass) under
|
||||
@@ -416,15 +425,207 @@ class TestMultiWorkerSafety:
|
||||
assert db_path.exists()
|
||||
assert b.get("h1") is not None
|
||||
|
||||
def test_corruption_message_triggers_reset(self, db_path):
|
||||
def test_corruption_fails_closed_and_siblings_keep_the_data(self, db_path):
|
||||
"""Corruption recovery must not destroy anything online: a sibling
|
||||
worker may still hold live handles, so deleting the main file or its
|
||||
-wal/-shm sidecars would split-brain the sibling or discard committed
|
||||
originals. Fail closed instead — this instance goes offline, the file
|
||||
survives for offline repair, and siblings are unaffected."""
|
||||
b = SQLiteBackend(db_path)
|
||||
assert b.set("h1", make_entry()) is True
|
||||
sibling = SQLiteBackend(db_path)
|
||||
assert sibling.set("h2", make_entry("h2")) is True
|
||||
|
||||
import headroom.cache.backends.sqlite as sqlite_mod
|
||||
|
||||
# Same monotonic clock monkey as the recover tests below use; the
|
||||
# reconnect retry follows the shared bounded backoff.
|
||||
clock = [1000.0]
|
||||
monkeypatch = pytest.MonkeyPatch()
|
||||
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
|
||||
try:
|
||||
b._handle_db_error(sqlite3.DatabaseError("database disk image is malformed"), "get")
|
||||
# Nothing was deleted: main file and every sidecar survive.
|
||||
assert db_path.exists()
|
||||
for suffix in ("-wal", "-shm"):
|
||||
sidecar = db_path.parent / f"{db_path.name}{suffix}"
|
||||
assert sidecar.exists()
|
||||
# This instance is down: reads miss, writes are refused loudly.
|
||||
assert b.get("h1") is None
|
||||
assert b.set("h3", make_entry("h3")) is False
|
||||
# The sibling, still connected to the preserved file, keeps serving
|
||||
# the originals that were stored before the recovery boundary.
|
||||
assert sibling.get("h1") is not None
|
||||
assert sibling.get("h2") is not None
|
||||
|
||||
# Offline repair happens with every worker stopped: close the
|
||||
# sibling's handle first (Windows refuses to unlink an open file),
|
||||
# then the operator removes the corrupt file and its sidecars.
|
||||
_close(sibling)
|
||||
for suffix in ("", "-wal", "-shm"):
|
||||
p = db_path if not suffix else db_path.parent / f"{db_path.name}{suffix}"
|
||||
p.unlink(missing_ok=True)
|
||||
# Once the retry is due, this instance reconnects to a fresh database.
|
||||
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
|
||||
assert b.set("h4", make_entry("h4")) is True
|
||||
assert b.get("h4") is not None
|
||||
finally:
|
||||
monkeypatch.undo()
|
||||
_close(sibling)
|
||||
_close(b)
|
||||
|
||||
def test_corruption_recovery_never_unlinks_the_store(self, db_path, monkeypatch):
|
||||
"""The recovery path must not delete the main file or any sidecar:
|
||||
sibling workers may still hold live handles, and the survivors are
|
||||
needed for offline repair."""
|
||||
b = SQLiteBackend(db_path)
|
||||
# A live sibling keeps the real WAL-mode -wal/-shm sidecars in place;
|
||||
# they are checked as-is rather than overwritten, because the -shm file
|
||||
# is memory-mapped while a connection is open (Windows rejects writes).
|
||||
sibling = SQLiteBackend(db_path)
|
||||
try:
|
||||
assert b.set("h1", make_entry()) is True
|
||||
sidecars = [db_path.parent / f"{db_path.name}{suffix}" for suffix in ("-wal", "-shm")]
|
||||
for sidecar in sidecars:
|
||||
assert sidecar.exists()
|
||||
|
||||
real_unlink = type(db_path).unlink
|
||||
|
||||
def refuse(self, *args, **kwargs):
|
||||
if self.parent == db_path.parent and self.name.startswith(db_path.name):
|
||||
raise AssertionError(
|
||||
"corruption recovery must not unlink the store or its sidecars"
|
||||
)
|
||||
return real_unlink(self, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(type(db_path), "unlink", refuse)
|
||||
b._handle_db_error(sqlite3.DatabaseError("file is not a database"), "get")
|
||||
monkeypatch.undo()
|
||||
|
||||
assert db_path.exists()
|
||||
for sidecar in sidecars:
|
||||
assert sidecar.exists()
|
||||
assert sibling.get("h1") is not None
|
||||
finally:
|
||||
_close(sibling)
|
||||
_close(b)
|
||||
|
||||
def test_failed_reopen_after_recovery_recovers_later(self, db_path, monkeypatch, caplog):
|
||||
"""A reopen that fails while the store is offline must not leave the
|
||||
store dead for the process lifetime: once the retry is due and the
|
||||
open succeeds, writes persist and reads hit again."""
|
||||
import headroom.cache.backends.sqlite as sqlite_mod
|
||||
|
||||
b = SQLiteBackend(db_path)
|
||||
b.set("h1", make_entry())
|
||||
real_open = b._open
|
||||
clock = [1000.0]
|
||||
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
|
||||
|
||||
def broken_open():
|
||||
raise sqlite3.OperationalError("unable to open database file")
|
||||
|
||||
monkeypatch.setattr(b, "_open", broken_open)
|
||||
with caplog.at_level(logging.WARNING):
|
||||
b._handle_db_error(sqlite3.DatabaseError("database disk image is malformed"), "get")
|
||||
# Down and the retry is not due yet: a miss, and the refused
|
||||
# write reports False rather than succeeding silently.
|
||||
assert b.get("h1") is None
|
||||
assert b.set("h2", make_entry("h2")) is False
|
||||
assert "nothing was read or persisted" in caplog.text
|
||||
|
||||
# The retry is due but the open still fails: still a miss, and
|
||||
# the next retry is pushed further out.
|
||||
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
|
||||
assert b.get("h1") is None
|
||||
|
||||
# The filesystem problem clears and the backoff elapses.
|
||||
monkeypatch.setattr(b, "_open", real_open)
|
||||
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
|
||||
assert b.set("h3", make_entry("h3")) is True
|
||||
got = b.get("h3")
|
||||
assert got is not None
|
||||
assert got.hash == "h3"
|
||||
|
||||
def test_reopen_retry_backs_off_while_open_keeps_failing(self, db_path, monkeypatch):
|
||||
import headroom.cache.backends.sqlite as sqlite_mod
|
||||
|
||||
b = SQLiteBackend(db_path)
|
||||
clock = [1000.0]
|
||||
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
|
||||
attempts = []
|
||||
|
||||
def broken_open():
|
||||
attempts.append(clock[0])
|
||||
raise sqlite3.OperationalError("unable to open database file")
|
||||
|
||||
monkeypatch.setattr(b, "_open", broken_open)
|
||||
b._handle_db_error(sqlite3.DatabaseError("file is not a database"), "get")
|
||||
assert len(attempts) == 0 # recovery only schedules; no open yet
|
||||
clock[0] += sqlite_mod._REOPEN_BACKOFF_INITIAL
|
||||
assert b.get("h1") is None
|
||||
assert len(attempts) == 1
|
||||
for _ in range(5):
|
||||
assert b.get("h1") is None # not due: no extra open attempts
|
||||
assert len(attempts) == 1
|
||||
clock[0] += sqlite_mod._REOPEN_BACKOFF_INITIAL
|
||||
assert b.get("h1") is None # the wait doubled: not due yet
|
||||
assert len(attempts) == 1
|
||||
clock[0] += sqlite_mod._REOPEN_BACKOFF_INITIAL
|
||||
assert b.get("h1") is None
|
||||
assert len(attempts) == 2
|
||||
# The wait doubled again, so one more initial interval is not enough.
|
||||
clock[0] += sqlite_mod._REOPEN_BACKOFF_INITIAL
|
||||
assert b.get("h1") is None
|
||||
assert len(attempts) == 2
|
||||
|
||||
def test_reopen_failure_recovery_ends_the_offline_state(self, db_path, monkeypatch):
|
||||
import headroom.cache.backends.sqlite as sqlite_mod
|
||||
|
||||
b = SQLiteBackend(db_path)
|
||||
b.set("h1", make_entry())
|
||||
real_open = b._open
|
||||
clock = [1000.0]
|
||||
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
|
||||
|
||||
def broken_open():
|
||||
raise sqlite3.OperationalError("unable to open database file")
|
||||
|
||||
monkeypatch.setattr(b, "_open", broken_open)
|
||||
b._handle_db_error(sqlite3.DatabaseError("database disk image is malformed"), "get")
|
||||
# Database recreated: empty but functional.
|
||||
assert b.count() == 0
|
||||
b.set("h2", make_entry("h2"))
|
||||
assert b.get("h1") is None
|
||||
|
||||
monkeypatch.setattr(b, "_open", real_open)
|
||||
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
|
||||
assert b.set("h2", make_entry("h2")) is True
|
||||
assert b.exists("h2")
|
||||
|
||||
def test_transient_write_failure_returns_false_and_keeps_the_database(self, db_path):
|
||||
"""A write lost to a transient error (busy/locked) is still lost:
|
||||
set() must report it, and must not reset the database."""
|
||||
b = SQLiteBackend(db_path)
|
||||
assert b.set("h1", make_entry()) is True
|
||||
|
||||
class LockedInsertConn:
|
||||
def __init__(self, real):
|
||||
self._real = real
|
||||
|
||||
def execute(self, *args, **kwargs):
|
||||
if args and "INSERT" in args[0]:
|
||||
raise sqlite3.OperationalError("database is locked")
|
||||
return self._real.execute(*args, **kwargs)
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self._real, name)
|
||||
|
||||
real = b._conn
|
||||
b._conn = LockedInsertConn(real) # type: ignore[assignment]
|
||||
assert b.set("h2", make_entry("h2")) is False
|
||||
b._conn = real
|
||||
assert db_path.exists()
|
||||
assert b.get("h1") is not None
|
||||
assert b.get("h2") is None
|
||||
|
||||
def test_busy_timeout_configured(self, db_path):
|
||||
b = SQLiteBackend(db_path)
|
||||
timeout = b._conn.execute("PRAGMA busy_timeout").fetchone()[0]
|
||||
@@ -529,5 +730,95 @@ class TestDefaults:
|
||||
assert "re-run" in CCR_MISS_MESSAGE
|
||||
|
||||
|
||||
COMMENTED_TOML = (
|
||||
"".join(f"# note {i}: keep this setting in sync with the deploy runbook\n" for i in range(30))
|
||||
+ '[package]\nname = "demo"\nversion = "0.1.0"\n\n[dependencies]\nserde = "1.0"\n'
|
||||
)
|
||||
|
||||
|
||||
class TestProducerDuringOutage:
|
||||
"""The producer-facing contract while the backend is down after a failed
|
||||
corruption-recovery reopen: writes fail loudly so producers keep the
|
||||
original, reads stay fail-open, and both work again after recovery."""
|
||||
|
||||
@pytest.fixture
|
||||
def outage(self, db_path, monkeypatch):
|
||||
import headroom.cache.backends.sqlite as sqlite_mod
|
||||
|
||||
backend = SQLiteBackend(db_path)
|
||||
real_open = backend._open
|
||||
clock = [1000.0]
|
||||
monkeypatch.setattr(sqlite_mod.time, "monotonic", lambda: clock[0])
|
||||
|
||||
def broken_open():
|
||||
raise sqlite3.OperationalError("unable to open database file")
|
||||
|
||||
monkeypatch.setattr(backend, "_open", broken_open)
|
||||
backend._handle_db_error(sqlite3.DatabaseError("database disk image is malformed"), "get")
|
||||
|
||||
def recover():
|
||||
monkeypatch.setattr(backend, "_open", real_open)
|
||||
clock[0] += sqlite_mod._REOPEN_BACKOFF_MAX
|
||||
|
||||
return backend, recover
|
||||
|
||||
def test_compression_store_raises_and_recovers(self, outage):
|
||||
backend, recover = outage
|
||||
store = CompressionStore(backend=backend, enable_feedback=False)
|
||||
|
||||
with pytest.raises(StorageError):
|
||||
store.store("original " * 100, "compressed")
|
||||
# Reads stay fail-open: a miss, not an exception.
|
||||
assert store.retrieve("0" * 24) is None
|
||||
|
||||
recover()
|
||||
hash_key = store.store("original " * 100, "compressed")
|
||||
entry = store.retrieve(hash_key)
|
||||
assert entry is not None
|
||||
assert entry.original_content == "original " * 100
|
||||
|
||||
def test_retrieve_survives_failed_access_count_write(self, db_path):
|
||||
backend = SQLiteBackend(db_path)
|
||||
store = CompressionStore(backend=backend, enable_feedback=False)
|
||||
hash_key = store.store("original " * 100, "compressed")
|
||||
|
||||
def failing_set(k, entry):
|
||||
return False
|
||||
|
||||
backend.set = failing_set # type: ignore[method-assign]
|
||||
entry = store.retrieve(hash_key)
|
||||
assert entry is not None
|
||||
assert entry.original_content == "original " * 100
|
||||
|
||||
def test_config_compressor_keeps_original_during_outage(self, outage, monkeypatch):
|
||||
from headroom.parser import CCR_RETRIEVAL_MARKER_RE
|
||||
from headroom.transforms.config_compressor import (
|
||||
ConfigCompressor,
|
||||
ConfigCompressorConfig,
|
||||
)
|
||||
|
||||
backend, recover = outage
|
||||
store = CompressionStore(backend=backend, enable_feedback=False)
|
||||
monkeypatch.setattr("headroom.cache.compression_store.get_compression_store", lambda: store)
|
||||
comp = ConfigCompressor(ConfigCompressorConfig(enable_ccr=True))
|
||||
|
||||
result = comp.compress(COMMENTED_TOML)
|
||||
# No hash was issued for an original that was never stored, so all
|
||||
# 30 comments stay in the output.
|
||||
assert result.ccr_hash is None
|
||||
assert result.lines_elided == 0
|
||||
assert not CCR_RETRIEVAL_MARKER_RE.search(result.compressed)
|
||||
for i in range(30):
|
||||
assert f"# note {i}:" in result.compressed
|
||||
|
||||
recover()
|
||||
result = comp.compress(COMMENTED_TOML)
|
||||
assert result.ccr_hash is not None
|
||||
assert "# note 0:" not in result.compressed
|
||||
entry = store.retrieve(result.ccr_hash)
|
||||
assert entry is not None
|
||||
assert entry.original_content == COMMENTED_TOML
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
||||
Reference in New Issue
Block a user