Commit Graph
2 Commits
Author SHA1 Message Date
trs-80andClaude Opus 5 3e36aefb43 perf(memory): reindex FTS5 a page at a time instead of a record at a time (#3466)
## Description

`headroom memory reindex` clears `memory_fts` and rebuilds it with one
`index_memory()` call per record. `FTS5TextIndex.index_raw()` commits
per call, so rebuilding a 1,000-memory store costs 1,000 transactions
and 1,000 fsyncs. The adapter already ships `index_batch_memories()`;
this uses it.

Closes # N/A

## Type of Change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `reindex_memories()` now calls `index_batch_memories()` once per page,
reusing the existing `_REINDEX_PAGE_SIZE` constant rather than
introducing a new knob. No adapter changes.
- On a failed page, the previous per-record loop runs **for that page
only**, so valid records still land and the failing record is still
named in the warning.
- A page that fails and then fully recovers does not change the exit
status. `ok` stays tied to whether records actually got indexed, exactly
as before.
- Added `benchmarks/memory_fts_reindex_benchmark.py` for reproduction.

**Why per page rather than one batch over the whole store.** A single
transaction would be marginally faster, but the reindex deliberately
clears the index before rebuilding (`DELETE FROM memory_fts`,
`headroom/cli/memory.py:1208-1210`). One failed statement would then
roll the whole rebuild back to an *empty* index — silently destroying
search coverage while reporting a clean-looking failure. A page bounds
both journal growth and blast radius, and keeps essentially all of the
speedup.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Six new tests in `tests/test_cli_memory_index_sync.py`, reusing that
file's existing helpers and its `_REINDEX_PAGE_SIZE` monkeypatch
pattern:

1. `..._batches_pages_instead_of_indexing_record_by_record` — asserts
the observed page sizes are `[2, 2, 1]` and fails outright if the
per-record path is touched.
2. `..._batched_output_matches_per_record_output` — every stored column
(`memory_id`, `content`, `user_id`, `session_id`, `category`) and the
FTS query result ordering are identical either way.
3. `..._empty_store_indexes_nothing_and_succeeds`.
4. `..._indexes_more_records_than_one_page`.
5. `..._failed_batch_still_indexes_valid_records_in_that_page` — the
failure mode the design is built around.
6. `..._failed_record_is_named_and_exits_nonzero` — per-record
diagnosability and exit status survive batching.

Tests 1 and 5 fail on the base commit and pass with this change.

### Test Output

```text
$ pytest tests/test_cli_memory_index_sync.py tests/test_memory/ -q
477 passed, 145 skipped in 10.38s

$ pytest tests/test_memory/ tests/test_cli_memory_index_sync.py tests/test_cli/ -q
1212 passed, 146 skipped, 2 failed
# Both failures (test_unwrap_claude_stops_claude_owned_persistent_deployment,
# test_wrap_codex_prepare_only_updates_config) also fail on the base commit in
# this environment: they read ambient CODEX_HOME / ANTHROPIC_BASE_URL from the
# developer's shell rather than isolating it. Unrelated to this change.

# The two behavioral tests against the base commit:
$ git stash push headroom/cli/memory.py
$ pytest tests/test_cli_memory_index_sync.py -q -k "reindex_batches or failed_batch"
2 failed

$ pre-commit run   # via commit hooks
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.13.11, stdlib `sqlite3` with
FTS5, worktree off `upstream/main` @ `e67b3c8a`.
- Exact command / steps: `python
benchmarks/memory_fts_reindex_benchmark.py --records 1000 --runs 3` for
the indexing operation; separately, a `CliRunner` harness timing the
whole `memory reindex` command over a 1,000-memory store, run on this
branch and then with `headroom/cli/memory.py` stashed. Each run uses a
fresh temporary database.
- Observed result: the indexing operation alone goes from **759.7 ms to
8.1 ms** (median of 3, ~94x). The **whole `memory reindex` command**
goes from **733.5 ms to 15.8 ms** (median of 3, ~46x) — reported
separately because the command also pages the primary store and does
vector-orphan cleanup, which this change does not touch. The audit that
motivated this measured 778.69 ms vs 5.60 ms for the operation;
reproduced within noise.
- Not tested: real-world stores larger than 1,000 records, concurrent
access during reindex, and behavior on network filesystems where fsync
costs differ. Synthetic records (~40 tokens each) rather than production
memory content. No end-to-end proxy or agent-latency claim is made.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; no flag or channel gates this.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Same records indexed, same
fields, same FTS results, same output lines, same exit status. Only the
transaction granularity changes.
- Kill switch / disable path: none needed. If the batch path raises for
any reason, the command falls back to the previous per-record behavior
automatically, per page.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the commit. `reindex` is idempotent — it clears
and rebuilds the index from the primary store — so re-running it on any
build produces the same result regardless of which version wrote the
index.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- No user-facing docs describe this internal rebuild path, so
"documentation update" is satisfied by the comment explaining why the
batch is bounded to a page rather than the whole store — that constraint
is the non-obvious part of this diff.
- Follow-up, deliberately not in this PR: the two failing tests noted
above depend on the developer's ambient `CODEX_HOME` and
`ANTHROPIC_BASE_URL`, so they pass or fail depending on whose shell runs
them. That is a test-isolation bug worth its own change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 16:45:52 -05:00
Radhakrishnan PachyappanandTejas Chopra fd4628d821 fix(memory): sync FTS5 and vector indexes on CLI delete/edit/prune/purge
## Problem

`headroom memory delete`, `prune`, `edit`, and `purge` all operate on
the bare `SQLiteMemoryStore` — they update the primary `memories` table
but never touch the FTS5 full-text index (`memory_fts` in `memory.db`)
or the vector index (`vec_metadata` / `vec_embeddings` in
`memory_vectors.db`). The index maintenance path lives in
`HierarchicalMemory.delete()` / `.update()`, which the CLI never
instantiates.

**Symptoms (from #2856):**
```sql
-- After deleting 16 of 46 memories via CLI:
SELECT COUNT(*) FROM memories;    -- 30
SELECT COUNT(*) FROM memory_fts;  -- 46  ← orphans
-- memory_vectors.db
SELECT COUNT(*) FROM vec_metadata;  -- 46  ← orphans
```
Deleted memories keep surfacing in `memory_search` results even after a
full server restart, because server startup only re-embeds memories
whose `embedding IS NULL` — it never removes orphaned index entries.

Fixes #2856.

## Solution

Add two best-effort helpers to `headroom/cli/memory.py` that use
**direct SQLite** (no `sqlite-vec` extension, no embedder):

- **`_remove_from_search_indexes(db_path, memory_ids)`**: removes
specific IDs from `memory_fts` and from `vec_metadata` /
`vec_embeddings`. Skips silently if an index doesn't exist.
- **`_clear_all_search_indexes(db_path)`**: truncates both indexes
completely (for purge).

Wire these up in four commands:
| Command | Change |
|---|---|
| `delete` | `_remove_from_search_indexes` after `store.delete_batch()`
|
| `prune` | `_remove_from_search_indexes` after `store.delete_batch()` |
| `purge` | `_clear_all_search_indexes` after `store.clear_all()` |
| `edit` | If content changed: remove stale entries, clear `embedding`
(server re-embeds on next startup), re-add FTS5 entry with new content
immediately |

The edit path re-adds the FTS5 entry right away so keyword search
reflects the new content without requiring a server restart. Vector
search is deferred to the next startup re-embed cycle (same as what the
server already does for missing embeddings).

## Changes

- `headroom/cli/memory.py` — two new helpers; four command call sites
- `tests/test_cli_memory_index_sync.py` (new) — 9 unit tests covering
both helpers with FTS5 and a stub vector DB. No `sqlite-vec` or embedder
required; tests run locally.

## Testing

```
$ python -m pytest tests/test_cli_memory_index_sync.py -v
...
9 passed in 2.38s
```

---------

Signed-off-by: Radhakrishnan Pachyappan <radhakrishnan.p@op.tech>
Signed-off-by: Radhakrishnan Pachyappan <gingeekrishna@gmail.com>
Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
2026-08-11 14:25:36 -07:00