mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3e36aefb43 |
perf(memory): reindex FTS5 a page at a time instead of a record at a time (#3466)
## Description `headroom memory reindex` clears `memory_fts` and rebuilds it with one `index_memory()` call per record. `FTS5TextIndex.index_raw()` commits per call, so rebuilding a 1,000-memory store costs 1,000 transactions and 1,000 fsyncs. The adapter already ships `index_batch_memories()`; this uses it. Closes # N/A ## Type of Change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `reindex_memories()` now calls `index_batch_memories()` once per page, reusing the existing `_REINDEX_PAGE_SIZE` constant rather than introducing a new knob. No adapter changes. - On a failed page, the previous per-record loop runs **for that page only**, so valid records still land and the failing record is still named in the warning. - A page that fails and then fully recovers does not change the exit status. `ok` stays tied to whether records actually got indexed, exactly as before. - Added `benchmarks/memory_fts_reindex_benchmark.py` for reproduction. **Why per page rather than one batch over the whole store.** A single transaction would be marginally faster, but the reindex deliberately clears the index before rebuilding (`DELETE FROM memory_fts`, `headroom/cli/memory.py:1208-1210`). One failed statement would then roll the whole rebuild back to an *empty* index — silently destroying search coverage while reporting a clean-looking failure. A page bounds both journal growth and blast radius, and keeps essentially all of the speedup. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Six new tests in `tests/test_cli_memory_index_sync.py`, reusing that file's existing helpers and its `_REINDEX_PAGE_SIZE` monkeypatch pattern: 1. `..._batches_pages_instead_of_indexing_record_by_record` — asserts the observed page sizes are `[2, 2, 1]` and fails outright if the per-record path is touched. 2. `..._batched_output_matches_per_record_output` — every stored column (`memory_id`, `content`, `user_id`, `session_id`, `category`) and the FTS query result ordering are identical either way. 3. `..._empty_store_indexes_nothing_and_succeeds`. 4. `..._indexes_more_records_than_one_page`. 5. `..._failed_batch_still_indexes_valid_records_in_that_page` — the failure mode the design is built around. 6. `..._failed_record_is_named_and_exits_nonzero` — per-record diagnosability and exit status survive batching. Tests 1 and 5 fail on the base commit and pass with this change. ### Test Output ```text $ pytest tests/test_cli_memory_index_sync.py tests/test_memory/ -q 477 passed, 145 skipped in 10.38s $ pytest tests/test_memory/ tests/test_cli_memory_index_sync.py tests/test_cli/ -q 1212 passed, 146 skipped, 2 failed # Both failures (test_unwrap_claude_stops_claude_owned_persistent_deployment, # test_wrap_codex_prepare_only_updates_config) also fail on the base commit in # this environment: they read ambient CODEX_HOME / ANTHROPIC_BASE_URL from the # developer's shell rather than isolating it. Unrelated to this change. # The two behavioral tests against the base commit: $ git stash push headroom/cli/memory.py $ pytest tests/test_cli_memory_index_sync.py -q -k "reindex_batches or failed_batch" 2 failed $ pre-commit run # via commit hooks ruff (legacy alias)......................................................Passed ruff format..............................................................Passed mypy.....................................................................Passed ``` ## Real Behavior Proof - Environment: macOS 15 (arm64), Python 3.13.11, stdlib `sqlite3` with FTS5, worktree off `upstream/main` @ `e67b3c8a`. - Exact command / steps: `python benchmarks/memory_fts_reindex_benchmark.py --records 1000 --runs 3` for the indexing operation; separately, a `CliRunner` harness timing the whole `memory reindex` command over a 1,000-memory store, run on this branch and then with `headroom/cli/memory.py` stashed. Each run uses a fresh temporary database. - Observed result: the indexing operation alone goes from **759.7 ms to 8.1 ms** (median of 3, ~94x). The **whole `memory reindex` command** goes from **733.5 ms to 15.8 ms** (median of 3, ~46x) — reported separately because the command also pages the primary store and does vector-orphan cleanup, which this change does not touch. The audit that motivated this measured 778.69 ms vs 5.60 ms for the operation; reproduced within noise. - Not tested: real-world stores larger than 1,000 records, concurrent access during reindex, and behavior on network filesystems where fsync costs differ. Synthetic records (~40 tokens each) rather than production memory content. No end-to-end proxy or agent-latency claim is made. ## Runtime Rollout Safety - Rollout-managed feature(s): none; no flag or channel gates this. - Minimum rollout channel: N/A. - Stable/default behavior changed: no. Same records indexed, same fields, same FTS results, same output lines, same exit status. Only the transaction granularity changes. - Kill switch / disable path: none needed. If the batch path raises for any reason, the command falls back to the previous per-record behavior automatically, per page. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert the commit. `reindex` is idempotent — it clears and rebuilds the index from the primary store — so re-running it on any build produces the same result regardless of which version wrote the index. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - No user-facing docs describe this internal rebuild path, so "documentation update" is satisfied by the comment explaining why the batch is bounded to a page rather than the whole store — that constraint is the non-obvious part of this diff. - Follow-up, deliberately not in this PR: the two failing tests noted above depend on the developer's ambient `CODEX_HOME` and `ANTHROPIC_BASE_URL`, so they pass or fail depending on whose shell runs them. That is a test-isolation bug worth its own change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
fd4628d821 |
fix(memory): sync FTS5 and vector indexes on CLI delete/edit/prune/purge
## Problem `headroom memory delete`, `prune`, `edit`, and `purge` all operate on the bare `SQLiteMemoryStore` — they update the primary `memories` table but never touch the FTS5 full-text index (`memory_fts` in `memory.db`) or the vector index (`vec_metadata` / `vec_embeddings` in `memory_vectors.db`). The index maintenance path lives in `HierarchicalMemory.delete()` / `.update()`, which the CLI never instantiates. **Symptoms (from #2856):** ```sql -- After deleting 16 of 46 memories via CLI: SELECT COUNT(*) FROM memories; -- 30 SELECT COUNT(*) FROM memory_fts; -- 46 ← orphans -- memory_vectors.db SELECT COUNT(*) FROM vec_metadata; -- 46 ← orphans ``` Deleted memories keep surfacing in `memory_search` results even after a full server restart, because server startup only re-embeds memories whose `embedding IS NULL` — it never removes orphaned index entries. Fixes #2856. ## Solution Add two best-effort helpers to `headroom/cli/memory.py` that use **direct SQLite** (no `sqlite-vec` extension, no embedder): - **`_remove_from_search_indexes(db_path, memory_ids)`**: removes specific IDs from `memory_fts` and from `vec_metadata` / `vec_embeddings`. Skips silently if an index doesn't exist. - **`_clear_all_search_indexes(db_path)`**: truncates both indexes completely (for purge). Wire these up in four commands: | Command | Change | |---|---| | `delete` | `_remove_from_search_indexes` after `store.delete_batch()` | | `prune` | `_remove_from_search_indexes` after `store.delete_batch()` | | `purge` | `_clear_all_search_indexes` after `store.clear_all()` | | `edit` | If content changed: remove stale entries, clear `embedding` (server re-embeds on next startup), re-add FTS5 entry with new content immediately | The edit path re-adds the FTS5 entry right away so keyword search reflects the new content without requiring a server restart. Vector search is deferred to the next startup re-embed cycle (same as what the server already does for missing embeddings). ## Changes - `headroom/cli/memory.py` — two new helpers; four command call sites - `tests/test_cli_memory_index_sync.py` (new) — 9 unit tests covering both helpers with FTS5 and a stub vector DB. No `sqlite-vec` or embedder required; tests run locally. ## Testing ``` $ python -m pytest tests/test_cli_memory_index_sync.py -v ... 9 passed in 2.38s ``` --------- Signed-off-by: Radhakrishnan Pachyappan <radhakrishnan.p@op.tech> Signed-off-by: Radhakrishnan Pachyappan <gingeekrishna@gmail.com> Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |