fix(memory): close SQLite connections in memory/fts5/graph adapters (#3153)

## Description

`_get_conn()` in
`headroom/memory/adapters/{sqlite,fts5,sqlite_graph}.py` opens a fresh
`sqlite3.Connection` on every call, and every call site uses `with
self._get_conn() as conn:`. In Python's stdlib,
`Connection.__enter__`/`__exit__` only commit/rollback the transaction
on exit — they never close the connection. None of these three files
ever call `.close()`, so every memory save/query/search/delete leaks one
OS file descriptor. In a long-lived proxy process this exhausts the FD
ulimit and eventually fails with `OSError: [Errno 24] Too many open
files`. The sibling `adapters/sqlite_vector.py` gets this right
(explicit per-thread cached connections + `close()`), which is what made
the omission here stand out.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/memory/adapters/sqlite.py`: `_get_conn()` is now a
`@contextlib.contextmanager` that commits/rolls back via `with conn:`
and closes the connection in a `finally` block.
- `headroom/memory/adapters/fts5.py`: same fix.
- `headroom/memory/adapters/sqlite_graph.py`: same fix, preserving the
existing page-cache/foreign-key PRAGMA setup before the connection is
yielded.
- No call sites changed — `with self._get_conn() as conn:` still works
identically against a context-manager-returning method.
- Added `tests/test_memory/test_adapters_connection_leak.py` covering
all three adapters.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
$ uv run pytest tests/test_memory/test_adapters_connection_leak.py -v
tests/test_memory/test_adapters_connection_leak.py::test_sqlite_memory_store_closes_connections PASSED
tests/test_memory/test_adapters_connection_leak.py::test_fts5_text_index_closes_connections PASSED
tests/test_memory/test_adapters_connection_leak.py::test_sqlite_graph_store_closes_connections PASSED
3 passed in 0.34s

$ uv run pytest tests/test_sqlite_graph_store.py tests/test_memory/ tests/cli/test_memory.py \
    tests/test_security_validations.py tests/test_cli_memory_index_sync.py -q
689 passed, 6 skipped, 79 warnings in 24.41s

$ uv run ruff check headroom/memory/adapters/sqlite.py headroom/memory/adapters/fts5.py \
    headroom/memory/adapters/sqlite_graph.py tests/test_memory/test_adapters_connection_leak.py
All checks passed!

$ uv run mypy headroom/memory/adapters/sqlite.py headroom/memory/adapters/fts5.py \
    headroom/memory/adapters/sqlite_graph.py
Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.11 (uv-managed venv), headroom repo at
`main` (post-#3094)
- Exact command / steps: added
`tests/test_memory/test_adapters_connection_leak.py`, which
monkeypatches `sqlite3.connect` to record every connection created by
`SQLiteMemoryStore`, `FTS5TextIndex`, and `SQLiteGraphStore` during
construction + one real operation (`save`/`index_raw`/`add_entity`),
then asserts each recorded connection raises `sqlite3.ProgrammingError`
on use (i.e. is closed). Ran it against the pre-fix code, then against
the fix.
- Observed result: against pre-fix code all 3 tests failed with `Failed:
DID NOT RAISE <class 'sqlite3.ProgrammingError'>` (connections stayed
open). Against the fix all 3 pass, and the full `test_memory/` +
`test_sqlite_graph_store.py` suites (689 tests) still pass unchanged.
- Not tested: a real FD-exhaustion soak test (thousands of connections
against a live proxy until `ulimit -n` is hit) — the monkeypatched
close-assertion is an equivalent, much faster proxy for the same fact,
so I didn't run a soak test.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this is unconditional bug-fix
behavior, not gated by any rollout flag.
- Minimum rollout channel: N/A
- Stable/default behavior changed: connections now close after each
operation (previously leaked); transaction commit/rollback semantics are
unchanged.
- Kill switch / disable path: N/A (no flag; revert the commit if needed)
- Unsafe override required: N/A
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Found while auditing the codebase for unreported bugs (not an existing
GitHub issue). Two more candidates from the same audit (unbounded
`by_model` dict growth in `savings_tracker.py`, and a `CompressionCache`
leak contradicting its own docstring in `content_router.py`) will follow
as separate PRs.
This commit is contained in:
Ashish
2026-09-30 23:54:43 -05:00
committed by GitHub
parent 728ff7e22b
commit 231a627702
4 changed files with 109 additions and 13 deletions
+11 -5
View File
@@ -7,6 +7,7 @@ and Unicode tokenization for high-quality search results.
from __future__ import annotations
import contextlib
import re
import sqlite3
from dataclasses import dataclass, field
@@ -17,7 +18,7 @@ from ..models import Memory
from ..ports import TextFilter, TextSearchResult
if TYPE_CHECKING:
pass
from collections.abc import Iterator
@dataclass
@@ -67,15 +68,20 @@ class FTS5TextIndex:
self.db_path = Path(db_path)
self._init_db()
def _get_conn(self) -> sqlite3.Connection:
@contextlib.contextmanager
def _get_conn(self) -> Iterator[sqlite3.Connection]:
"""Get a new database connection (thread-safe pattern).
Returns:
A new SQLite connection with row factory configured.
Commits on clean exit, rolls back on exception, and always closes
the connection -- callers use ``with self._get_conn() as conn:``.
"""
conn = sqlite3.connect(str(self.db_path))
conn.row_factory = sqlite3.Row
return conn
try:
with conn:
yield conn
finally:
conn.close()
def _init_db(self) -> None:
"""Initialize the FTS5 virtual table schema."""
+12 -4
View File
@@ -9,6 +9,7 @@ Provides persistent storage for Memory objects with full support for:
from __future__ import annotations
import contextlib
import json
import re
import sqlite3
@@ -20,6 +21,8 @@ from ..models import Memory, ScopeLevel, normalize_entity_refs
from ..ports import MemoryFilter
if TYPE_CHECKING:
from collections.abc import Iterator
import numpy as np
# Regex pattern for safe metadata keys: alphanumeric, underscores, hyphens only
@@ -73,15 +76,20 @@ class SQLiteMemoryStore:
self.db_path = Path(db_path)
self._init_db()
def _get_conn(self) -> sqlite3.Connection:
@contextlib.contextmanager
def _get_conn(self) -> Iterator[sqlite3.Connection]:
"""Get a new database connection (thread-safe pattern).
Returns:
A new SQLite connection with row factory configured.
Commits on clean exit, rolls back on exception, and always closes
the connection -- callers use ``with self._get_conn() as conn:``.
"""
conn = sqlite3.connect(str(self.db_path))
conn.row_factory = sqlite3.Row
return conn
try:
with conn:
yield conn
finally:
conn.close()
def _init_db(self) -> None:
"""Initialize the database schema with indexes."""
+12 -4
View File
@@ -13,6 +13,7 @@ This is a drop-in replacement for InMemoryGraphStore that:
from __future__ import annotations
import contextlib
import json
import logging
import sqlite3
@@ -25,6 +26,8 @@ from typing import TYPE_CHECKING, Any
from .graph_models import Entity, Relationship, RelationshipDirection, Subgraph
if TYPE_CHECKING:
from collections.abc import Iterator
from ..tracker import ComponentStats
logger = logging.getLogger(__name__)
@@ -74,11 +77,12 @@ class SQLiteGraphStore:
self._lock = RLock()
self._init_db()
def _get_conn(self) -> sqlite3.Connection:
@contextlib.contextmanager
def _get_conn(self) -> Iterator[sqlite3.Connection]:
"""Get a new database connection (thread-safe pattern).
Returns:
A new SQLite connection with row factory configured.
Commits on clean exit, rolls back on exception, and always closes
the connection -- callers use ``with self._get_conn() as conn:``.
"""
conn = sqlite3.connect(str(self.db_path))
conn.row_factory = sqlite3.Row
@@ -90,7 +94,11 @@ class SQLiteGraphStore:
# Enable foreign keys
conn.execute("PRAGMA foreign_keys = ON")
return conn
try:
with conn:
yield conn
finally:
conn.close()
def _init_db(self) -> None:
"""Initialize the database schema with indexes."""
@@ -0,0 +1,74 @@
"""Regression tests: SQLite adapters must close every connection they open.
``_get_conn()`` in each adapter opens a brand-new ``sqlite3.Connection`` per
call. ``with conn:`` only commits/rolls back on exit -- it does not close the
connection -- so without an explicit ``.close()`` every operation leaks a
file descriptor.
"""
from __future__ import annotations
import sqlite3
from pathlib import Path
import pytest
from headroom.memory.adapters.fts5 import FTS5TextIndex
from headroom.memory.adapters.graph_models import Entity
from headroom.memory.adapters.sqlite import SQLiteMemoryStore
from headroom.memory.adapters.sqlite_graph import SQLiteGraphStore
from headroom.memory.models import Memory
def _track_connections(monkeypatch: pytest.MonkeyPatch) -> list[sqlite3.Connection]:
"""Wrap ``sqlite3.connect`` to record every connection it creates."""
created: list[sqlite3.Connection] = []
real_connect = sqlite3.connect
def tracking_connect(*args: object, **kwargs: object) -> sqlite3.Connection:
conn = real_connect(*args, **kwargs) # type: ignore[arg-type]
created.append(conn)
return conn
monkeypatch.setattr(sqlite3, "connect", tracking_connect)
return created
def _assert_all_closed(connections: list[sqlite3.Connection]) -> None:
assert connections, "expected at least one sqlite3.Connection to be created"
for conn in connections:
with pytest.raises(sqlite3.ProgrammingError, match="closed database"):
conn.execute("SELECT 1")
async def test_sqlite_memory_store_closes_connections(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
created = _track_connections(monkeypatch)
store = SQLiteMemoryStore(str(tmp_path / "mem.db"))
await store.save(Memory(content="hello", user_id="alice"))
_assert_all_closed(created)
def test_fts5_text_index_closes_connections(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
created = _track_connections(monkeypatch)
index = FTS5TextIndex(str(tmp_path / "fts.db"))
index.index_raw("mem-1", "hello world", {"user_id": "alice"})
_assert_all_closed(created)
async def test_sqlite_graph_store_closes_connections(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
created = _track_connections(monkeypatch)
store = SQLiteGraphStore(str(tmp_path / "graph.db"))
await store.add_entity(Entity(user_id="alice", name="Project X", entity_type="project"))
_assert_all_closed(created)