trending/codegraph
1
0
Fork 0
mirror of https://github.com/colbymchenry/codegraph.git synced 2026-11-14 22:43:17 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix(cli): read commands and the prompt hook open the index read-only (#2936)

Browse Source 
Running query, files, node, callers, callees, impact, context, explore or
affected while `codegraph index` rebuilt a project could make the rebuild
fail with "database is locked", and so could the hidden prompt-hook, which
Claude Code runs on its own for structural prompts. Each opened the index
read-write, so its open ran the writer's repairs: during the rebuild's bulk
load they recreated the secondary indexes the indexer had dropped, and the
indexer's next write, whose transaction had already read, failed with
SQLITE_BUSY_SNAPSHOT (517). PR #2930 fixed the same thing for status.

These commands now open the index read-only, through one helper, and leave
migrations, index recovery and WAL maintenance to the writer:
- query, files, node, callers, callees, impact, context, explore, affected
  and prompt-hook are read-only. explore and node run the MCP tool handler
  in process without a lifecycle, so they never sync or open another
  project.
- While another live process holds rebuild.pid they say so in one line on
  stderr (a --json answer stays parseable), as status does.
- The prompt hook stays out of a running rebuild altogether: the index is
  half-built until it ends, the MCP tools answer "rebuild in progress"
  meanwhile, and not opening it keeps the hook from holding the file the
  rebuild replaces. Its gate records the new outcome noop-rebuild.
- The hook's one write, filling an empty segment vocabulary (issue #1142),
  goes through a writer connection of its own, opened only once it holds
  the index lock, which every index and sync holds while it writes.
- status (read-only since PR #2930), sync, index, init and uninit are
  unchanged; the last four write.

A read-only open migrates nothing. QueryBuilder.hasSchema lets each read
that needs a later table or column answer as that migration's defaults
would: files.generated (v9), unresolved_refs.status and name_tail (v8), and
name_segment_vocab (v7). The status pending count from PR #2930 uses it too.

Validation (Windows 11, Node 22.22):
- The issue's interleaving, each command in another process inside the
  writer's edge transaction: on main all 14 command forms make the insert
  fail with errcode 517 and bring the 5 dropped edge indexes back; with
  this change all 14 inserts succeed and the indexes stay dropped.
- Real rebuilds of excalidraw 95307fa1 (736 files) with query and the
  prompt hook looping beside them, default settings and with the
  resolution window forced: main failed 6 of 14 ("store worker: database
  is locked" with an empty index, or "database is locked" partway through
  resolution), and 12 of its 109 query runs failed at open with
  SQLITE_BUSY. This change failed 0 of 14, with 0 failures in 299 query and
  284 hook runs and the same graph every time (14,236 nodes, 61,210 edges).
- Old indexes: 24 command forms on indexes at schemas 4 to 16 (11 schemas
  rebuilt from a current index, plus indexes built by 0.9.9, 1.4.0, 1.5.0,
  1.6.0 and 1.6.2): output identical to main's, which migrates first, 384 of
  384. The schema stays as it was, except where the hook fills the
  vocabulary of a pre-v7 index; that writer's open migrates it, as every
  read-write open did.
- Timing on excalidraw at rest, 22 interleaved runs per command: medians
  within -5.8% to +3.3%, -0.9% in total.
- read-commands-during-rebuild.test.ts: 38/38 pass. With main's sources 35
  fail; with only the CLI opens reverted 33, only the schema fallbacks 19,
  only the read-only vocabulary fill 2.

Follow-up to PR #2930.
Fixes #2913

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-10-27 22:26:03 +00:00
committed by GitHub
co-authored by Claude Opus 5.5
parent cbd9817f7c
commit 475e56b209
6 changed files with 517 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/bin/codegraph.ts
+46 -20
View File
@@ -378,6 +378,32 @@ function warn(message: string): void {
console.log(chalk.yellow(getGlyphs().warn) + ' ' + message);
}
/**
* Open a project's index for a command that only reads it: `query`,
* `explore`, `node`, the prompt hook and the rest (#2913).
*
* Opened read-write, the index runs the writer's repairs: migrations, the
* rebuild of the indexes and search triggers an interrupted bulk load left
* out, and WAL maintenance. While `codegraph index` or a sync writes in
* another process, those repairs recreate the indexes its bulk load dropped,
* and its next write fails with SQLITE_BUSY_SNAPSHOT, "database is locked".
* Read-only, a command leaves all of that to the writer, as `status` does
* since #2930; an index an older release built is read as it is (see
* QueryBuilder.hasSchema).
*
* During a rebuild the answer comes from a graph still being built, so the
* command says so, on stderr to keep a `--json` answer parseable.
*/
async function openIndexToRead(projectPath: string): Promise<import('../index').CodeGraph> {
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(projectPath, { readOnly: true });
const { isRebuildInProgress } = await import('../mcp/writer-lock');
if (isRebuildInProgress(projectPath)) {
console.error(`${chalk.yellow(getGlyphs().warn)} An index rebuild ("codegraph index") is running in another process — this answer comes from the part of the index it has rebuilt so far.`);
}
return cg;
}
/** "not found" (+ optional did-you-mean) when no exact symbol matches. */
function formatSymbolNotFound(symbol: string, fuzzyNames: string[]): string {
const suggestions = [...new Set(fuzzyNames.filter((n) => n !== symbol))].slice(0, 3);
@@ -1344,8 +1370,7 @@ program
process.exit(1);
}
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(projectPath);
const cg = await openIndexToRead(projectPath);
const limit = parseInt(options.limit || '10', 10);
const rawResults = cg.searchNodes(search, {
@@ -1430,8 +1455,7 @@ program
process.exit(1);
}
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(projectPath);
const cg = await openIndexToRead(projectPath);
const { ToolHandler } = await import('../mcp/tools');
const handler = new ToolHandler(cg);
@@ -1487,8 +1511,7 @@ program
process.exit(1);
}
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(projectPath);
const cg = await openIndexToRead(projectPath);
const result = await cg.buildContext(taskParts.join(' '), {
format,
@@ -1592,8 +1615,15 @@ program
`${lead}\n${projects.map((p) => ` - projectPath: "${p}"`).join('\n')}\n`;
if (plan.exploreRoot) {
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(plan.exploreRoot);
// A `codegraph index` in another process is rebuilding this index: it
// holds part of the graph until the rebuild ends, and the rebuild
// replaces the file, which an open handle blocks on Windows. Inject
// nothing, as the MCP tools answer "rebuild in progress" (#2913).
const { isRebuildInProgress } = await import('../mcp/writer-lock');
if (isRebuildInProgress(plan.exploreRoot)) { gate('noop-rebuild'); return; }
// Claude Code runs this hook on its own, on every prompt: read-only, so
// it never repairs what an index or sync is in the middle of writing.
const cg = await openIndexToRead(plan.exploreRoot);
try {
const others = plan.nudgeProjects.length
? `\n${nudge(plan.nudgeProjects, 'Other indexed projects in this workspace — pass projectPath to query them:')}`
@@ -1644,8 +1674,9 @@ program
// (#1142). Heal it here: on a populated vocab this is one SELECT;
// the actual backfill is a one-time batched pass whose cost the MCP
// server's own catch-up sync usually pays first (it runs at every
// session start). A distinct noop outcome keeps a dormant vocab
// from polluting the noop-unverified recall signal.
// session start). It writes through a connection of its own, under
// the index lock (#2913). A distinct noop outcome keeps a dormant
// vocab from polluting the noop-unverified recall signal.
const vocabReady = await cg.healSegmentVocabIfEmpty().catch(() => false);
if (!vocabReady) { gate('noop-vocab-empty'); return; }
const related = cg.getSegmentMatches(proseWords);
@@ -1716,8 +1747,7 @@ program
process.exit(1);
}
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(projectPath);
const cg = await openIndexToRead(projectPath);
const { ToolHandler } = await import('../mcp/tools');
const handler = new ToolHandler(cg);
@@ -1788,8 +1818,7 @@ program
process.exit(1);
}
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(projectPath);
const cg = await openIndexToRead(projectPath);
let files = cg.getFiles();
if (files.length === 0) {
@@ -2361,8 +2390,7 @@ for (const direction of ['callers', 'callees'] as const) {
process.exit(1);
}
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(projectPath);
const cg = await openIndexToRead(projectPath);
try {
const limit = parseInt(options.limit || '20', 10);
const { nodes: targets } = lookupSymbolNodes(cg, symbol, goPackageReaderFor(cg));
@@ -2485,8 +2513,7 @@ program
process.exit(1);
}
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(projectPath);
const cg = await openIndexToRead(projectPath);
try {
const depth = Math.min(Math.max(parseInt(options.depth || '2', 10), 1), 10);
const { nodes: targets } = lookupSymbolNodes(cg, symbol, goPackageReaderFor(cg));
@@ -2626,8 +2653,7 @@ program
process.exit(0);
}
const { default: CodeGraph } = await loadCodeGraph();
const cg = await CodeGraph.open(projectPath);
const cg = await openIndexToRead(projectPath);
const maxDepth = parseInt(options.depth || '5', 10);
// Custom filter pattern
src/db/queries.ts
+62 -16
View File
@@ -466,6 +466,33 @@ export class QueryBuilder {
// connection, or a worker following a rebuilt index keeps its answer (#1864).
this.dominantFileMemo = undefined;
this.readMemos.clear();
this.presentSchema.clear();
}
/**
* Tables (`name`) and columns (`table.column`) that later migrations added,
* found in this connection's database. See {@link hasSchema}.
*/
private presentSchema = new Set<string>();
/**
* Whether the database holds `table`, or its `column`. An index an older
* release built lacks what later migrations add until a writer's open
* migrates it, and a read-only open migrates nothing: the CLI's read
* commands and the prompt hook open read-only so that a running rebuild is
* left alone (#2913). A read that needs a newer table or column checks for
* it here and, without it, answers as the migration's defaults would.
* Found once and remembered, since a schema only grows; a missing one is
* looked for again, so a writer's later migration is seen.
*/
private hasSchema(table: string, column?: string): boolean {
const key = column ? `${table}.${column}` : table;
if (this.presentSchema.has(key)) return true;
const found = column
? this.db.prepare('SELECT 1 FROM pragma_table_info(?) WHERE name = ?').get(table, column) !== undefined
: this.db.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?").get(table) !== undefined;
if (found) this.presentSchema.add(key);
return found;
}
/**
@@ -885,6 +912,8 @@ export class QueryBuilder {
/** True when the vocab has no rows — an index built before the table existed.
* `sync` uses this to heal such databases (see rebuildNameSegmentVocabFrom). */
isNameSegmentVocabEmpty(): boolean {
// Before schema v7 there is no table, which its migration creates empty (#2913).
if (!this.hasSchema('name_segment_vocab')) return true;
const row = this.db.prepare('SELECT 1 FROM name_segment_vocab LIMIT 1').get();
return row === undefined;
}
@@ -931,7 +960,7 @@ export class QueryBuilder {
minWords: number,
limit: number,
): Array<{ name: string; matches: number }> {
if (variants.length === 0) return [];
if (variants.length === 0 || !this.hasSchema('name_segment_vocab')) return [];
const placeholders = variants.map(() => '?').join(', ');
const whens = variants.map(() => 'WHEN ? THEN ?').join(' ');
const rows = this.db
@@ -956,7 +985,7 @@ export class QueryBuilder {
/** How many distinct names each segment appears in — the rarity signal that
* separates a discriminative word ("checkout") from a ubiquitous one ("state"). */
getSegmentNameCounts(segments: string[]): Map<string, number> {
if (segments.length === 0) return new Map();
if (segments.length === 0 || !this.hasSchema('name_segment_vocab')) return new Map();
const placeholders = segments.map(() => '?').join(', ');
const rows = this.db
.prepare(
@@ -969,6 +998,7 @@ export class QueryBuilder {
/** Names containing the given segment (rare-single-word tier). */
getNamesForSegment(segment: string, limit: number): string[] {
if (!this.hasSchema('name_segment_vocab')) return [];
const rows = this.db
.prepare('SELECT name FROM name_segment_vocab WHERE segment = ? ORDER BY length(name) ASC LIMIT ?')
.all(segment, limit) as Array<{ name: string }>;
@@ -1916,32 +1946,34 @@ export class QueryBuilder {
// EXISTS uses FTS postings and the segment primary key, never source scans.
// Vocab rows can outlive deleted definitions, so verify them against nodes.
// An index from before schema v7 has no vocab: FTS alone answers (#2913).
const vocab = this.hasSchema('name_segment_vocab');
const rows = this.db.prepare(`
WITH words(word, pattern) AS (VALUES ${checked.map(() => '(?, ?)').join(', ')})
SELECT word, (
EXISTS (SELECT 1 FROM nodes_fts WHERE nodes_fts MATCH pattern)
EXISTS (SELECT 1 FROM nodes_fts WHERE nodes_fts MATCH pattern)${vocab ? `
OR EXISTS (
SELECT 1 FROM name_segment_vocab v WHERE v.segment = word
AND EXISTS (SELECT 1 FROM nodes n WHERE n.name = v.name AND n.kind NOT IN ('file', 'import'))
)
)` : ''}
) AS matched FROM words
`).all(...checked.flatMap(w => [w, `{name qualified_name signature docstring} : "${w}"*`])) as
Array<{ word: string; matched: number }>;
const names = this.db.prepare(`
const names = this.db.prepare(`${vocab ? `
SELECT name FROM (
SELECT v.name FROM name_segment_vocab v
WHERE v.segment IN (${checked.map(() => '?').join(', ')})
AND EXISTS (SELECT 1 FROM nodes n WHERE n.name = v.name AND n.kind NOT IN ('file', 'import'))
LIMIT 12
)
UNION ALL
UNION ALL` : ''}
SELECT name FROM (
SELECT n.name FROM nodes_fts JOIN nodes n ON n.rowid = nodes_fts.rowid
WHERE nodes_fts MATCH ? AND n.kind NOT IN ('file', 'import')
LIMIT 12
)
`).all(...checked, `name : (${checked.map(w => `"${w}"*`).join(' OR ')})`) as Array<{ name: string }>;
`).all(...(vocab ? checked : []), `name : (${checked.map(w => `"${w}"*`).join(' OR ')})`) as Array<{ name: string }>;
return {
matched: rows.filter(r => r.matched).map(r => r.word),
unmatched: rows.filter(r => !r.matched).map(r => r.word),
@@ -2647,9 +2679,12 @@ export class QueryBuilder {
): Array<{ node: Node; generated: boolean }> {
if (kinds.length === 0 || limit <= 0) return [];
const placeholders = kinds.map(() => '?').join(',');
// Before schema v9 no file carries the flag, as its migration's default
// leaves them (#2913).
const generated = this.hasSchema('files', 'generated') ? 'COALESCE(f.generated, 0)' : '0';
const rows = this.db
.prepare(
`SELECT n.*, COALESCE(f.generated, 0) AS file_generated
`SELECT n.*, ${generated} AS file_generated
FROM nodes n
LEFT JOIN files f ON f.path = n.file_path
WHERE n.kind IN (${placeholders})
@@ -2687,22 +2722,29 @@ export class QueryBuilder {
const unique = [...new Set(names)].filter((name) => name.length > 0);
const found = new Set<string>();
if (unique.length === 0) return found;
// Before schema v8 a reference had no tail and was never parked, and the
// migration leaves every row pending with an empty tail: only the name
// can match (#2913).
const parked = this.hasSchema('unresolved_refs', 'status');
for (let i = 0; i < unique.length; i += SQLITE_PARAM_CHUNK_SIZE) {
const chunk = unique.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
const placeholders = chunk.map(() => '?').join(',');
const rows = this.db
.prepare(
`SELECT DISTINCT reference_name AS name FROM unresolved_refs
WHERE reference_name IN (${placeholders})
const byName = `SELECT DISTINCT reference_name AS name FROM unresolved_refs
WHERE reference_name IN (${placeholders})`;
const rows = (parked
? this.db
.prepare(
`${byName}
UNION
SELECT DISTINCT name_tail AS name FROM unresolved_refs
WHERE name_tail IN (${placeholders})
UNION
SELECT DISTINCT ${MEMBER_TAIL_SQL} AS name FROM unresolved_refs
WHERE ${MODULE_PARKED_SQL} AND ${MEMBER_TAIL_SQL} IN (${placeholders})`
)
.all(...chunk, ...chunk, ...chunk) as Array<{ name: string }>;
)
.all(...chunk, ...chunk, ...chunk)
: this.db.prepare(byName).all(...chunk)) as Array<{ name: string }>;
for (const row of rows) found.add(row.name);
}
return found;
@@ -3745,6 +3787,9 @@ export class QueryBuilder {
const unique = [...new Set(filePaths)];
const found = new Set<string>();
if (unique.length === 0) return found;
// An index from before schema v9 has no flag, which its migration adds
// as 0 everywhere: the path convention alone decides (#2913).
if (!this.hasSchema('files', 'generated')) return found;
for (let i = 0; i < unique.length; i += SQLITE_PARAM_CHUNK_SIZE) {
const chunk = unique.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
@@ -3893,6 +3938,8 @@ export class QueryBuilder {
/** How many indexed files carry the generated flag. Surfaced by `status`. */
countGeneratedFiles(): number {
// Before schema v9 no file carries the flag (#2913).
if (!this.hasSchema('files', 'generated')) return 0;
const row = this.db
.prepare('SELECT COUNT(*) AS n FROM files WHERE generated = 1')
.get() as { n: number } | undefined;
@@ -4138,8 +4185,7 @@ export class QueryBuilder {
// and the v8 migration marks every one of its rows pending. A read-only
// open migrates nothing (`codegraph status`, #2913), so count them all
// until a writer's open has migrated it.
const columns = this.db.prepare('PRAGMA table_info(unresolved_refs)').all() as Array<{ name: string }>;
if (!columns.some((c) => c.name === 'status')) {
if (!this.hasSchema('unresolved_refs', 'status')) {
return (this.db.prepare('SELECT COUNT(*) as count FROM unresolved_refs').get() as { count: number }).count;
}
this.stmts.getUnresolvedCount = this.db.prepare(
src/index.ts
+20 -5
View File
@@ -2360,6 +2360,12 @@ export class CodeGraph {
* usable (already populated — the overwhelmingly common one-SELECT case —
* or healed here); false when it isn't (empty graph, or another process
* holds the index lock — that process's own sync heals it).
*
* A read-only instance, as the hook opens since #2913, writes through a
* connection of its own, opened only once it holds the index lock. Every
* index and sync holds that lock while it writes, so the repairs a writer's
* open runs, and the migration an index from before the table needs, cannot
* disturb a bulk load in progress.
*/
async healSegmentVocabIfEmpty(): Promise<boolean> {
const empty = (() => {
@@ -2375,7 +2381,16 @@ export class CodeGraph {
}
try {
if (!this.queries.isNameSegmentVocabEmpty()) return true; // raced: healed meanwhile
await this.rebuildNameSegmentVocab();
if (!this.db.readOnly) {
await this.rebuildNameSegmentVocab();
return true;
}
const writer = DatabaseConnection.open(this.db.getPath());
try {
await this.rebuildNameSegmentVocab(new QueryBuilder(writer.getDb()));
} finally {
writer.close();
}
return true;
} finally {
this.fileLock.release();
@@ -2387,15 +2402,15 @@ export class CodeGraph {
* Rebuild the segment vocabulary from the current graph, batched and
* yielding — the upgrade-heal path for indexes built before the vocab table
* existed. Runs inside the index mutex/lock (sync and
* healSegmentVocabIfEmpty hold them).
* healSegmentVocabIfEmpty hold them), through `queries`.
*/
private async rebuildNameSegmentVocab(): Promise<void> {
private async rebuildNameSegmentVocab(queries: QueryBuilder = this.queries): Promise<void> {
const maybeYield = createYielder();
const BATCH = 2000;
for (let offset = 0; ; offset += BATCH) {
const names = this.queries.getDistinctNodeNames(BATCH, offset);
const names = queries.getDistinctNodeNames(BATCH, offset);
if (names.length === 0) break;
this.queries.insertNameSegmentsBatch(names);
queries.insertNameSegmentsBatch(names);
await maybeYield();
}
}