mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
perf(memory): reindex FTS5 a page at a time instead of a record at a time (#3466)
## Description `headroom memory reindex` clears `memory_fts` and rebuilds it with one `index_memory()` call per record. `FTS5TextIndex.index_raw()` commits per call, so rebuilding a 1,000-memory store costs 1,000 transactions and 1,000 fsyncs. The adapter already ships `index_batch_memories()`; this uses it. Closes # N/A ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `reindex_memories()` now calls `index_batch_memories()` once per page, reusing the existing `_REINDEX_PAGE_SIZE` constant rather than introducing a new knob. No adapter changes. - On a failed page, the previous per-record loop runs **for that page only**, so valid records still land and the failing record is still named in the warning. - A page that fails and then fully recovers does not change the exit status. `ok` stays tied to whether records actually got indexed, exactly as before. - Added `benchmarks/memory_fts_reindex_benchmark.py` for reproduction. **Why per page rather than one batch over the whole store.** A single transaction would be marginally faster, but the reindex deliberately clears the index before rebuilding (`DELETE FROM memory_fts`, `headroom/cli/memory.py:1208-1210`). One failed statement would then roll the whole rebuild back to an *empty* index — silently destroying search coverage while reporting a clean-looking failure. A page bounds both journal growth and blast radius, and keeps essentially all of the speedup. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Six new tests in `tests/test_cli_memory_index_sync.py`, reusing that file's existing helpers and its `_REINDEX_PAGE_SIZE` monkeypatch pattern: 1. `..._batches_pages_instead_of_indexing_record_by_record` — asserts the observed page sizes are `[2, 2, 1]` and fails outright if the per-record path is touched. 2. `..._batched_output_matches_per_record_output` — every stored column (`memory_id`, `content`, `user_id`, `session_id`, `category`) and the FTS query result ordering are identical either way. 3. `..._empty_store_indexes_nothing_and_succeeds`. 4. `..._indexes_more_records_than_one_page`. 5. `..._failed_batch_still_indexes_valid_records_in_that_page` — the failure mode the design is built around. 6. `..._failed_record_is_named_and_exits_nonzero` — per-record diagnosability and exit status survive batching. Tests 1 and 5 fail on the base commit and pass with this change. ### Test Output ```text $ pytest tests/test_cli_memory_index_sync.py tests/test_memory/ -q 477 passed, 145 skipped in 10.38s $ pytest tests/test_memory/ tests/test_cli_memory_index_sync.py tests/test_cli/ -q 1212 passed, 146 skipped, 2 failed # Both failures (test_unwrap_claude_stops_claude_owned_persistent_deployment, # test_wrap_codex_prepare_only_updates_config) also fail on the base commit in # this environment: they read ambient CODEX_HOME / ANTHROPIC_BASE_URL from the # developer's shell rather than isolating it. Unrelated to this change. # The two behavioral tests against the base commit: $ git stash push headroom/cli/memory.py $ pytest tests/test_cli_memory_index_sync.py -q -k "reindex_batches or failed_batch" 2 failed $ pre-commit run # via commit hooks ruff (legacy alias)......................................................Passed ruff format..............................................................Passed mypy.....................................................................Passed ``` ## Real Behavior Proof - Environment: macOS 15 (arm64), Python 3.13.11, stdlib `sqlite3` with FTS5, worktree off `upstream/main` @ `e67b3c8a`. - Exact command / steps: `python benchmarks/memory_fts_reindex_benchmark.py --records 1000 --runs 3` for the indexing operation; separately, a `CliRunner` harness timing the whole `memory reindex` command over a 1,000-memory store, run on this branch and then with `headroom/cli/memory.py` stashed. Each run uses a fresh temporary database. - Observed result: the indexing operation alone goes from **759.7 ms to 8.1 ms** (median of 3, ~94x). The **whole `memory reindex` command** goes from **733.5 ms to 15.8 ms** (median of 3, ~46x) — reported separately because the command also pages the primary store and does vector-orphan cleanup, which this change does not touch. The audit that motivated this measured 778.69 ms vs 5.60 ms for the operation; reproduced within noise. - Not tested: real-world stores larger than 1,000 records, concurrent access during reindex, and behavior on network filesystems where fsync costs differ. Synthetic records (~40 tokens each) rather than production memory content. No end-to-end proxy or agent-latency claim is made. ## Runtime Rollout Safety - Rollout-managed feature(s): none; no flag or channel gates this. - Minimum rollout channel: N/A. - Stable/default behavior changed: no. Same records indexed, same fields, same FTS results, same output lines, same exit status. Only the transaction granularity changes. - Kill switch / disable path: none needed. If the batch path raises for any reason, the command falls back to the previous per-record behavior automatically, per page. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert the commit. `reindex` is idempotent — it clears and rebuilds the index from the primary store — so re-running it on any build produces the same result regardless of which version wrote the index. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - No user-facing docs describe this internal rebuild path, so "documentation update" is satisfied by the comment explaining why the batch is bounded to a page rather than the whole store — that constraint is the non-obvious part of this diff. - Follow-up, deliberately not in this PR: the two failing tests noted above depend on the developer's ambient `CODEX_HOME` and `ANTHROPIC_BASE_URL`, so they pass or fail depending on whose shell runs them. That is a test-isolation bug worth its own change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Measure FTS5 reindex throughput: per-record commits vs batched transactions."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
# Run as a script sys.path[0] is benchmarks/, so an editable install of another
|
||||
# checkout would win and we would silently measure the wrong tree.
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
|
||||
from headroom.cli.memory import _REINDEX_PAGE_SIZE
|
||||
from headroom.memory.adapters.fts5 import FTS5TextIndex
|
||||
from headroom.memory.models import Memory
|
||||
|
||||
|
||||
def make_memories(count: int, content_words: int) -> list[Memory]:
|
||||
body = " ".join(f"token{n}" for n in range(content_words))
|
||||
return [
|
||||
Memory(
|
||||
id=f"mem-{index:06d}",
|
||||
content=f"{body} unique{index}",
|
||||
user_id="bench-user",
|
||||
session_id=f"session-{index % 7}",
|
||||
)
|
||||
for index in range(count)
|
||||
]
|
||||
|
||||
|
||||
def benchmark(count: int, content_words: int, page_size: int, runs: int) -> dict[str, object]:
|
||||
memories = make_memories(count, content_words)
|
||||
per_record: list[float] = []
|
||||
batched: list[float] = []
|
||||
|
||||
for _ in range(runs):
|
||||
# Fresh temporary database per mode per run so neither inherits the
|
||||
# other's page cache or index state.
|
||||
with tempfile.TemporaryDirectory(prefix="headroom-fts-bench-") as tmp:
|
||||
fts = FTS5TextIndex(db_path=str(Path(tmp) / "memory.db"))
|
||||
start = time.perf_counter()
|
||||
for memory in memories:
|
||||
asyncio.run(fts.index_memory(memory))
|
||||
per_record.append((time.perf_counter() - start) * 1000)
|
||||
|
||||
with tempfile.TemporaryDirectory(prefix="headroom-fts-bench-") as tmp:
|
||||
fts = FTS5TextIndex(db_path=str(Path(tmp) / "memory.db"))
|
||||
start = time.perf_counter()
|
||||
for offset in range(0, len(memories), page_size):
|
||||
asyncio.run(fts.index_batch_memories(memories[offset : offset + page_size]))
|
||||
batched.append((time.perf_counter() - start) * 1000)
|
||||
|
||||
return {
|
||||
"records": count,
|
||||
"content_words": content_words,
|
||||
"page_size": page_size,
|
||||
"runs": runs,
|
||||
"per_record_ms": statistics.median(per_record),
|
||||
"batched_ms": statistics.median(batched),
|
||||
"speedup": round(statistics.median(per_record) / statistics.median(batched), 1),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--records", type=int, nargs="+", default=[1000])
|
||||
parser.add_argument("--content-words", type=int, default=40)
|
||||
parser.add_argument("--page-size", type=int, default=_REINDEX_PAGE_SIZE)
|
||||
parser.add_argument("--runs", type=int, default=3)
|
||||
args = parser.parse_args()
|
||||
|
||||
results = [
|
||||
benchmark(count, args.content_words, args.page_size, args.runs) for count in args.records
|
||||
]
|
||||
print(json.dumps(results, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+22
-5
@@ -1212,14 +1212,31 @@ def reindex_memories(ctx: click.Context, db_path: str) -> None:
|
||||
print_error(f"Failed to clear FTS5 index: {exc}")
|
||||
sys.exit(1)
|
||||
|
||||
# Index a page at a time. One transaction per page instead of per record
|
||||
# is ~100x faster, while still bounding how much the journal grows and
|
||||
# how much a single failure can take with it. On a failed page, retry
|
||||
# that page record by record so valid records still land and the broken
|
||||
# one is still named -- the index was already cleared above, so a page
|
||||
# that only rolled back would leave a silent hole in search coverage.
|
||||
fts_indexed = 0
|
||||
for mem in memories:
|
||||
for start in range(0, len(memories), _REINDEX_PAGE_SIZE):
|
||||
page = memories[start : start + _REINDEX_PAGE_SIZE]
|
||||
try:
|
||||
asyncio.run(fts.index_memory(mem))
|
||||
fts_indexed += 1
|
||||
fts_indexed += asyncio.run(fts.index_batch_memories(page))
|
||||
continue
|
||||
except Exception as exc:
|
||||
print_warning(f"FTS5: failed to index {mem.id[:8]}: {exc}")
|
||||
ok = False
|
||||
# Not a failure on its own: nothing from this page committed,
|
||||
# and the per-record retry below still gets to index it. Exit
|
||||
# status stays tied to whether records actually got indexed.
|
||||
print_warning(f"FTS5: page rolled back ({exc}); retrying record by record")
|
||||
|
||||
for mem in page:
|
||||
try:
|
||||
asyncio.run(fts.index_memory(mem))
|
||||
fts_indexed += 1
|
||||
except Exception as exc:
|
||||
print_warning(f"FTS5: failed to index {mem.id[:8]}: {exc}")
|
||||
ok = False
|
||||
|
||||
# --- Vector: remove orphaned entries (requires sqlite-vec) ---
|
||||
vector_db = db.parent / f"{db.stem}_vectors.db"
|
||||
|
||||
@@ -506,3 +506,154 @@ def test_purge_command_exits_nonzero_when_index_sync_fails(tmp_path, monkeypatch
|
||||
assert result.exit_code == 1
|
||||
assert asyncio.run(store.get(memory.id)) is None
|
||||
assert "index sync incomplete" in result.output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# reindex batching
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _always_raises(message: str):
|
||||
async def _raise(*_args, **_kwargs):
|
||||
raise RuntimeError(message)
|
||||
|
||||
return _raise
|
||||
|
||||
|
||||
def _fts_rows(db_path: Path) -> dict[str, tuple[str, str, str, str]]:
|
||||
with sqlite3.connect(str(db_path)) as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT memory_id, content, user_id, session_id, category FROM memory_fts"
|
||||
).fetchall()
|
||||
return {r[0]: (r[1], r[2], r[3], r[4]) for r in rows}
|
||||
|
||||
|
||||
def _seeded_store(db_path: Path, count: int) -> list[Memory]:
|
||||
store = SQLiteMemoryStore(str(db_path))
|
||||
memories = []
|
||||
for i in range(count):
|
||||
mem = _make_memory(f"id-{i}", f"content {i}")
|
||||
mem.session_id = f"session-{i % 3}"
|
||||
memories.append(mem)
|
||||
asyncio.run(store.save(mem))
|
||||
return memories
|
||||
|
||||
|
||||
def test_reindex_batches_pages_instead_of_indexing_record_by_record(tmp_path, monkeypatch):
|
||||
"""The batch API does the work; the per-record path is only a fallback."""
|
||||
db_path = tmp_path / "memory.db"
|
||||
_seeded_store(db_path, 5)
|
||||
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 2)
|
||||
|
||||
batch_sizes: list[int] = []
|
||||
real_batch = FTS5TextIndex.index_batch_memories
|
||||
|
||||
async def spy_batch(self, memories):
|
||||
batch_sizes.append(len(memories))
|
||||
return await real_batch(self, memories)
|
||||
|
||||
monkeypatch.setattr(FTS5TextIndex, "index_batch_memories", spy_batch)
|
||||
monkeypatch.setattr(
|
||||
FTS5TextIndex,
|
||||
"index_memory",
|
||||
lambda *_a, **_k: pytest.fail("reindex fell back to per-record indexing"),
|
||||
)
|
||||
|
||||
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert batch_sizes == [2, 2, 1]
|
||||
assert "Re-indexed 5/5 memories" in result.output
|
||||
|
||||
|
||||
def test_reindex_batched_output_matches_per_record_output(tmp_path, monkeypatch):
|
||||
"""Every indexed field and FTS query result is identical either way."""
|
||||
batched_db = tmp_path / "batched.db"
|
||||
per_record_db = tmp_path / "per_record.db"
|
||||
for db_path in (batched_db, per_record_db):
|
||||
_seeded_store(db_path, 7)
|
||||
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 3)
|
||||
|
||||
batched = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(batched_db)])
|
||||
assert batched.exit_code == 0, batched.output
|
||||
|
||||
# Force the per-record path by making every batch fail.
|
||||
monkeypatch.setattr(
|
||||
FTS5TextIndex,
|
||||
"index_batch_memories",
|
||||
_always_raises("batching disabled for parity check"),
|
||||
)
|
||||
per_record = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(per_record_db)])
|
||||
assert per_record.exit_code == 0, per_record.output
|
||||
|
||||
assert _fts_rows(batched_db) == _fts_rows(per_record_db)
|
||||
batched_hits = FTS5TextIndex(db_path=str(batched_db)).search("content", k=10)
|
||||
per_record_hits = FTS5TextIndex(db_path=str(per_record_db)).search("content", k=10)
|
||||
assert [h.memory_id for h in batched_hits] == [h.memory_id for h in per_record_hits]
|
||||
|
||||
|
||||
def test_reindex_empty_store_indexes_nothing_and_succeeds(tmp_path):
|
||||
db_path = tmp_path / "memory.db"
|
||||
SQLiteMemoryStore(str(db_path))
|
||||
|
||||
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert _fts_ids(db_path) == set()
|
||||
assert "Re-indexed 0/0 memories" in result.output
|
||||
|
||||
|
||||
def test_reindex_indexes_more_records_than_one_page(tmp_path, monkeypatch):
|
||||
db_path = tmp_path / "memory.db"
|
||||
memories = _seeded_store(db_path, 7)
|
||||
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 3)
|
||||
|
||||
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert _fts_ids(db_path) == {m.id for m in memories}
|
||||
assert "Re-indexed 7/7 memories" in result.output
|
||||
|
||||
|
||||
def test_reindex_failed_batch_still_indexes_valid_records_in_that_page(tmp_path, monkeypatch):
|
||||
"""A rolled-back page must not leave a hole: valid records still land."""
|
||||
db_path = tmp_path / "memory.db"
|
||||
memories = _seeded_store(db_path, 4)
|
||||
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 4)
|
||||
monkeypatch.setattr(
|
||||
FTS5TextIndex, "index_batch_memories", _always_raises("simulated batch failure")
|
||||
)
|
||||
|
||||
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert _fts_ids(db_path) == {m.id for m in memories}
|
||||
assert "page rolled back" in result.output
|
||||
assert "Re-indexed 4/4 memories" in result.output
|
||||
|
||||
|
||||
def test_reindex_failed_record_is_named_and_exits_nonzero(tmp_path, monkeypatch):
|
||||
"""Per-record diagnosability survives batching."""
|
||||
db_path = tmp_path / "memory.db"
|
||||
memories = _seeded_store(db_path, 3)
|
||||
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 3)
|
||||
monkeypatch.setattr(
|
||||
FTS5TextIndex, "index_batch_memories", _always_raises("simulated batch failure")
|
||||
)
|
||||
|
||||
broken = memories[1].id
|
||||
real_index_memory = FTS5TextIndex.index_memory
|
||||
|
||||
async def index_or_fail(self, memory):
|
||||
if memory.id == broken:
|
||||
raise RuntimeError("simulated record failure")
|
||||
await real_index_memory(self, memory)
|
||||
|
||||
monkeypatch.setattr(FTS5TextIndex, "index_memory", index_or_fail)
|
||||
|
||||
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
|
||||
|
||||
assert result.exit_code == 1
|
||||
assert f"failed to index {broken[:8]}" in result.output
|
||||
assert _fts_ids(db_path) == {memories[0].id, memories[2].id}
|
||||
assert "Re-indexed 2/3 memories" in result.output
|
||||
|
||||
Reference in New Issue
Block a user