perf(memory): reindex FTS5 a page at a time instead of a record at a time (#3466)

## Description

`headroom memory reindex` clears `memory_fts` and rebuilds it with one
`index_memory()` call per record. `FTS5TextIndex.index_raw()` commits
per call, so rebuilding a 1,000-memory store costs 1,000 transactions
and 1,000 fsyncs. The adapter already ships `index_batch_memories()`;
this uses it.

Closes # N/A

## Type of Change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `reindex_memories()` now calls `index_batch_memories()` once per page,
reusing the existing `_REINDEX_PAGE_SIZE` constant rather than
introducing a new knob. No adapter changes.
- On a failed page, the previous per-record loop runs **for that page
only**, so valid records still land and the failing record is still
named in the warning.
- A page that fails and then fully recovers does not change the exit
status. `ok` stays tied to whether records actually got indexed, exactly
as before.
- Added `benchmarks/memory_fts_reindex_benchmark.py` for reproduction.

**Why per page rather than one batch over the whole store.** A single
transaction would be marginally faster, but the reindex deliberately
clears the index before rebuilding (`DELETE FROM memory_fts`,
`headroom/cli/memory.py:1208-1210`). One failed statement would then
roll the whole rebuild back to an *empty* index — silently destroying
search coverage while reporting a clean-looking failure. A page bounds
both journal growth and blast radius, and keeps essentially all of the
speedup.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Six new tests in `tests/test_cli_memory_index_sync.py`, reusing that
file's existing helpers and its `_REINDEX_PAGE_SIZE` monkeypatch
pattern:

1. `..._batches_pages_instead_of_indexing_record_by_record` — asserts
the observed page sizes are `[2, 2, 1]` and fails outright if the
per-record path is touched.
2. `..._batched_output_matches_per_record_output` — every stored column
(`memory_id`, `content`, `user_id`, `session_id`, `category`) and the
FTS query result ordering are identical either way.
3. `..._empty_store_indexes_nothing_and_succeeds`.
4. `..._indexes_more_records_than_one_page`.
5. `..._failed_batch_still_indexes_valid_records_in_that_page` — the
failure mode the design is built around.
6. `..._failed_record_is_named_and_exits_nonzero` — per-record
diagnosability and exit status survive batching.

Tests 1 and 5 fail on the base commit and pass with this change.

### Test Output

```text
$ pytest tests/test_cli_memory_index_sync.py tests/test_memory/ -q
477 passed, 145 skipped in 10.38s

$ pytest tests/test_memory/ tests/test_cli_memory_index_sync.py tests/test_cli/ -q
1212 passed, 146 skipped, 2 failed
# Both failures (test_unwrap_claude_stops_claude_owned_persistent_deployment,
# test_wrap_codex_prepare_only_updates_config) also fail on the base commit in
# this environment: they read ambient CODEX_HOME / ANTHROPIC_BASE_URL from the
# developer's shell rather than isolating it. Unrelated to this change.

# The two behavioral tests against the base commit:
$ git stash push headroom/cli/memory.py
$ pytest tests/test_cli_memory_index_sync.py -q -k "reindex_batches or failed_batch"
2 failed

$ pre-commit run   # via commit hooks
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.13.11, stdlib `sqlite3` with
FTS5, worktree off `upstream/main` @ `e67b3c8a`.
- Exact command / steps: `python
benchmarks/memory_fts_reindex_benchmark.py --records 1000 --runs 3` for
the indexing operation; separately, a `CliRunner` harness timing the
whole `memory reindex` command over a 1,000-memory store, run on this
branch and then with `headroom/cli/memory.py` stashed. Each run uses a
fresh temporary database.
- Observed result: the indexing operation alone goes from **759.7 ms to
8.1 ms** (median of 3, ~94x). The **whole `memory reindex` command**
goes from **733.5 ms to 15.8 ms** (median of 3, ~46x) — reported
separately because the command also pages the primary store and does
vector-orphan cleanup, which this change does not touch. The audit that
motivated this measured 778.69 ms vs 5.60 ms for the operation;
reproduced within noise.
- Not tested: real-world stores larger than 1,000 records, concurrent
access during reindex, and behavior on network filesystems where fsync
costs differ. Synthetic records (~40 tokens each) rather than production
memory content. No end-to-end proxy or agent-latency claim is made.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; no flag or channel gates this.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Same records indexed, same
fields, same FTS results, same output lines, same exit status. Only the
transaction granularity changes.
- Kill switch / disable path: none needed. If the batch path raises for
any reason, the command falls back to the previous per-record behavior
automatically, per page.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the commit. `reindex` is idempotent — it clears
and rebuilds the index from the primary store — so re-running it on any
build produces the same result regardless of which version wrote the
index.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- No user-facing docs describe this internal rebuild path, so
"documentation update" is satisfied by the comment explaining why the
batch is bounded to a page rather than the whole store — that constraint
is the non-obvious part of this diff.
- Follow-up, deliberately not in this PR: the two failing tests noted
above depend on the developer's ambient `CODEX_HOME` and
`ANTHROPIC_BASE_URL`, so they pass or fail depending on whose shell runs
them. That is a test-isolation bug worth its own change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
trs-80
2026-09-24 16:45:52 -05:00
committed by GitHub
co-authored by Claude Opus 5
parent 6880984b4a
commit 3e36aefb43
3 changed files with 258 additions and 5 deletions
@@ -0,0 +1,85 @@
#!/usr/bin/env python3
"""Measure FTS5 reindex throughput: per-record commits vs batched transactions."""
from __future__ import annotations
import argparse
import asyncio
import json
import statistics
import sys
import tempfile
import time
from pathlib import Path
# Run as a script sys.path[0] is benchmarks/, so an editable install of another
# checkout would win and we would silently measure the wrong tree.
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from headroom.cli.memory import _REINDEX_PAGE_SIZE
from headroom.memory.adapters.fts5 import FTS5TextIndex
from headroom.memory.models import Memory
def make_memories(count: int, content_words: int) -> list[Memory]:
body = " ".join(f"token{n}" for n in range(content_words))
return [
Memory(
id=f"mem-{index:06d}",
content=f"{body} unique{index}",
user_id="bench-user",
session_id=f"session-{index % 7}",
)
for index in range(count)
]
def benchmark(count: int, content_words: int, page_size: int, runs: int) -> dict[str, object]:
memories = make_memories(count, content_words)
per_record: list[float] = []
batched: list[float] = []
for _ in range(runs):
# Fresh temporary database per mode per run so neither inherits the
# other's page cache or index state.
with tempfile.TemporaryDirectory(prefix="headroom-fts-bench-") as tmp:
fts = FTS5TextIndex(db_path=str(Path(tmp) / "memory.db"))
start = time.perf_counter()
for memory in memories:
asyncio.run(fts.index_memory(memory))
per_record.append((time.perf_counter() - start) * 1000)
with tempfile.TemporaryDirectory(prefix="headroom-fts-bench-") as tmp:
fts = FTS5TextIndex(db_path=str(Path(tmp) / "memory.db"))
start = time.perf_counter()
for offset in range(0, len(memories), page_size):
asyncio.run(fts.index_batch_memories(memories[offset : offset + page_size]))
batched.append((time.perf_counter() - start) * 1000)
return {
"records": count,
"content_words": content_words,
"page_size": page_size,
"runs": runs,
"per_record_ms": statistics.median(per_record),
"batched_ms": statistics.median(batched),
"speedup": round(statistics.median(per_record) / statistics.median(batched), 1),
}
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--records", type=int, nargs="+", default=[1000])
parser.add_argument("--content-words", type=int, default=40)
parser.add_argument("--page-size", type=int, default=_REINDEX_PAGE_SIZE)
parser.add_argument("--runs", type=int, default=3)
args = parser.parse_args()
results = [
benchmark(count, args.content_words, args.page_size, args.runs) for count in args.records
]
print(json.dumps(results, indent=2))
if __name__ == "__main__":
main()
+22 -5
View File
@@ -1212,14 +1212,31 @@ def reindex_memories(ctx: click.Context, db_path: str) -> None:
print_error(f"Failed to clear FTS5 index: {exc}")
sys.exit(1)
# Index a page at a time. One transaction per page instead of per record
# is ~100x faster, while still bounding how much the journal grows and
# how much a single failure can take with it. On a failed page, retry
# that page record by record so valid records still land and the broken
# one is still named -- the index was already cleared above, so a page
# that only rolled back would leave a silent hole in search coverage.
fts_indexed = 0
for mem in memories:
for start in range(0, len(memories), _REINDEX_PAGE_SIZE):
page = memories[start : start + _REINDEX_PAGE_SIZE]
try:
asyncio.run(fts.index_memory(mem))
fts_indexed += 1
fts_indexed += asyncio.run(fts.index_batch_memories(page))
continue
except Exception as exc:
print_warning(f"FTS5: failed to index {mem.id[:8]}: {exc}")
ok = False
# Not a failure on its own: nothing from this page committed,
# and the per-record retry below still gets to index it. Exit
# status stays tied to whether records actually got indexed.
print_warning(f"FTS5: page rolled back ({exc}); retrying record by record")
for mem in page:
try:
asyncio.run(fts.index_memory(mem))
fts_indexed += 1
except Exception as exc:
print_warning(f"FTS5: failed to index {mem.id[:8]}: {exc}")
ok = False
# --- Vector: remove orphaned entries (requires sqlite-vec) ---
vector_db = db.parent / f"{db.stem}_vectors.db"
+151
View File
@@ -506,3 +506,154 @@ def test_purge_command_exits_nonzero_when_index_sync_fails(tmp_path, monkeypatch
assert result.exit_code == 1
assert asyncio.run(store.get(memory.id)) is None
assert "index sync incomplete" in result.output
# ---------------------------------------------------------------------------
# reindex batching
# ---------------------------------------------------------------------------
def _always_raises(message: str):
async def _raise(*_args, **_kwargs):
raise RuntimeError(message)
return _raise
def _fts_rows(db_path: Path) -> dict[str, tuple[str, str, str, str]]:
with sqlite3.connect(str(db_path)) as conn:
rows = conn.execute(
"SELECT memory_id, content, user_id, session_id, category FROM memory_fts"
).fetchall()
return {r[0]: (r[1], r[2], r[3], r[4]) for r in rows}
def _seeded_store(db_path: Path, count: int) -> list[Memory]:
store = SQLiteMemoryStore(str(db_path))
memories = []
for i in range(count):
mem = _make_memory(f"id-{i}", f"content {i}")
mem.session_id = f"session-{i % 3}"
memories.append(mem)
asyncio.run(store.save(mem))
return memories
def test_reindex_batches_pages_instead_of_indexing_record_by_record(tmp_path, monkeypatch):
"""The batch API does the work; the per-record path is only a fallback."""
db_path = tmp_path / "memory.db"
_seeded_store(db_path, 5)
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 2)
batch_sizes: list[int] = []
real_batch = FTS5TextIndex.index_batch_memories
async def spy_batch(self, memories):
batch_sizes.append(len(memories))
return await real_batch(self, memories)
monkeypatch.setattr(FTS5TextIndex, "index_batch_memories", spy_batch)
monkeypatch.setattr(
FTS5TextIndex,
"index_memory",
lambda *_a, **_k: pytest.fail("reindex fell back to per-record indexing"),
)
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
assert result.exit_code == 0, result.output
assert batch_sizes == [2, 2, 1]
assert "Re-indexed 5/5 memories" in result.output
def test_reindex_batched_output_matches_per_record_output(tmp_path, monkeypatch):
"""Every indexed field and FTS query result is identical either way."""
batched_db = tmp_path / "batched.db"
per_record_db = tmp_path / "per_record.db"
for db_path in (batched_db, per_record_db):
_seeded_store(db_path, 7)
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 3)
batched = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(batched_db)])
assert batched.exit_code == 0, batched.output
# Force the per-record path by making every batch fail.
monkeypatch.setattr(
FTS5TextIndex,
"index_batch_memories",
_always_raises("batching disabled for parity check"),
)
per_record = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(per_record_db)])
assert per_record.exit_code == 0, per_record.output
assert _fts_rows(batched_db) == _fts_rows(per_record_db)
batched_hits = FTS5TextIndex(db_path=str(batched_db)).search("content", k=10)
per_record_hits = FTS5TextIndex(db_path=str(per_record_db)).search("content", k=10)
assert [h.memory_id for h in batched_hits] == [h.memory_id for h in per_record_hits]
def test_reindex_empty_store_indexes_nothing_and_succeeds(tmp_path):
db_path = tmp_path / "memory.db"
SQLiteMemoryStore(str(db_path))
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
assert result.exit_code == 0, result.output
assert _fts_ids(db_path) == set()
assert "Re-indexed 0/0 memories" in result.output
def test_reindex_indexes_more_records_than_one_page(tmp_path, monkeypatch):
db_path = tmp_path / "memory.db"
memories = _seeded_store(db_path, 7)
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 3)
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
assert result.exit_code == 0, result.output
assert _fts_ids(db_path) == {m.id for m in memories}
assert "Re-indexed 7/7 memories" in result.output
def test_reindex_failed_batch_still_indexes_valid_records_in_that_page(tmp_path, monkeypatch):
"""A rolled-back page must not leave a hole: valid records still land."""
db_path = tmp_path / "memory.db"
memories = _seeded_store(db_path, 4)
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 4)
monkeypatch.setattr(
FTS5TextIndex, "index_batch_memories", _always_raises("simulated batch failure")
)
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
assert result.exit_code == 0, result.output
assert _fts_ids(db_path) == {m.id for m in memories}
assert "page rolled back" in result.output
assert "Re-indexed 4/4 memories" in result.output
def test_reindex_failed_record_is_named_and_exits_nonzero(tmp_path, monkeypatch):
"""Per-record diagnosability survives batching."""
db_path = tmp_path / "memory.db"
memories = _seeded_store(db_path, 3)
monkeypatch.setattr(memory_cli, "_REINDEX_PAGE_SIZE", 3)
monkeypatch.setattr(
FTS5TextIndex, "index_batch_memories", _always_raises("simulated batch failure")
)
broken = memories[1].id
real_index_memory = FTS5TextIndex.index_memory
async def index_or_fail(self, memory):
if memory.id == broken:
raise RuntimeError("simulated record failure")
await real_index_memory(self, memory)
monkeypatch.setattr(FTS5TextIndex, "index_memory", index_or_fail)
result = CliRunner().invoke(main, ["memory", "reindex", "--db-path", str(db_path)])
assert result.exit_code == 1
assert f"failed to index {broken[:8]}" in result.output
assert _fts_ids(db_path) == {memories[0].id, memories[2].id}
assert "Re-indexed 2/3 memories" in result.output