fix(explore): a long query, like a pasted log, answers in seconds on a large index (#2804)

The prompt hook hands codegraph_explore the whole prompt, so a pasted
log, document or source file arrives as a query of hundreds of words.
findRelevantContext searched every word several times, and the reads
behind those searches scanned the index each time:

- findNodesByNameSubstring, four times per symbol word: `name LIKE
  '%word%'` with a leading wildcard reads every node of those kinds,
  about 80 ms each on vscode (562,636 nodes);
- the LIKE fallback after an FTS miss (`name LIKE` or `qualified_name
  LIKE '%word%'`), about 120 ms per miss, and most words of a log miss;
- the fuzzy fallback after that re-read `SELECT DISTINCT name` each time
  (about 150 ms), and getAllNodeNames, documented as cached, was not;
- FTS joined nodes on the TEXT id instead of the rowid;
- scorePathRelevance re-split every query word for every scored path,
  and the deprioritize predicate stat'ed codegraph.json per candidate.

On vscode a 1.7k-character log took 80 s in explore, a 10k-character
skill document 258 s, and the hook gave up on both. A per-statement
profile put 65-80% of that in those scans.

The fix:
- A name-substring index answers the substring lookups from memory with
  exactly the rows SQLite returns, in its order: ASCII-only case folding
  and `_` as one character, like LIKE, and ties in length in the order
  the plan reads rows, the kind index's or rowid's. The plan is asked of
  SQLite once per statement shape, since a small index is scanned whole;
  a `%`, a language filter or any other plan goes to SQLite as before.
- A word of letters and digits that no name and no letters-and-digits
  run of a qualified name holds can match nothing, so the LIKE fallback
  returns empty without scanning for it.
- The fuzzy fallback reads the name list once and skips names outside
  its edit-distance length window.
- These reads are memoized per database change stamp, like the dominant
  file (PR #1864): rebuilt after any write from any connection, never
  kept inside a transaction, dropped on rebind.
- FTS joins on rowid, with CROSS JOIN keeping nodes_fts the outer loop
  as before.
- Path relevance prepares each query once and each path is scored once
  per search; the deprioritize patterns are read once per search.
- findRelevantContext searches at most 64 symbol names and 128 search
  terms of a query. Both lists put code-shaped words first, so a long
  paste is searched by its identifiers and its opening words. A
  16-symbol bag (75 terms) stays whole. This is the one intended change
  in output, for long queries only.

Validation (vscode ff0a86dc, median of 3 interleaved, fresh process):
- explore, ordinary queries: 6 of 6 outputs byte-identical, 5.5-7.5 s
  fresh (the dominant-file aggregation is about 4 s of that) and
  0.9-2.1 s warm vs 2.7-3.9 s on main;
- explore, long queries: log 1.7k 80.1 to 10.7 s, markdown 3k 73.3 to
  9.0 s, Java 3k 108.8 to 12.2 s, TS 3k 63.4 to 11.9 s, markdown 10k
  258.3 to 10.8 s, Java 10k 144.7 to 11.1 s, TS 10k 146.7 to 9.6 s;
- the real prompt hook on the issue's prompt shapes: short 7.2 to
  7.0 s, log 84.2 to 12.9 s, markdown 3k 78.1 to 10.2 s, Java 3k 113.1
  to 13.3 s, markdown 10k 261.7 to 11.3 s, Java 10k 146.3 to 11.1 s;
- without the word caps, all 7 long queries gave byte-identical output
  4.3-9.0x faster;
- django 79502d08 and prometheus ed26d727: 11 of 11 queries
  byte-identical, tracebacks included, 1.0-2.9x faster;
- the substring index against the LIKE scan on vscode: 4,830 of 4,830
  lookups identical (805 words, snake_case included);
- explore-long-query.test.ts: 13 tests; 8 fail on main's source.

Issue #2620. Closed contributor PR #2621 (retsamekohC) was read for
guidance; open PR #2260 (danusha2345) overlaps on the name-list memo
and path relevance.

Fixes #2620

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -22,6 +22,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - In Python, a call through a module-level instance, like `_svc.delete(obj)` after `_svc = SceneService()`, now links to the method its class has or inherits, even when other classes have a method of that name. (#2582)
 - In Python, a call on a module imported from outside your project, like `subprocess.run(…)` after `import subprocess`, no longer links to a project method that happens to share its name. Thanks @Egrn for the report. (#2599)
 - `codegraph_explore`, the CLI `codegraph explore` and the prompt hook no longer stall for a minute or more on a large project after a full index; the step that finds the project's busiest file no longer slows to a crawl when a few files are huge, like a generated Android `R.java` (#2571). Thanks @yangfengfan2011.
+- `codegraph_explore`, the CLI `codegraph explore` and the Claude Code prompt hook now answer a long query, like a pasted log, document or source file, in seconds on a large project instead of minutes. Every word of such a query used to search the whole index several times over, so the prompt hook ran out of time on long prompts and added nothing. Those searches now reuse what they read, and a very long query is searched by its code names and its opening words rather than by every word in it. Answers to ordinary questions and lists of symbol names are unchanged. Thanks @retsamekohC for the report and the first fix. (#2620)
 - The Claude Code prompt hook now goes by what you typed. Claude Code's desktop app puts status notes, like which other sessions are linked, in front of your prompt, and the hook read them as part of it, so every turn, even a one-word "yes", could get code context about the note's words, and so could a task notification, a subagent's report or a message from another session that came after one. The notes are now skipped, and a message from another session gets no context, as task notifications and subagent reports already didn't. Thanks @WonderMr for the report and @yang0228 for the first fix. (#2635)
 - `codegraph_explore` and `codegraph context` now find your project's busiest file, and favor the code around it, even when twenty or more test or generated files are busier still, such as a package of protobuf stubs. Before, the search looked only at the 20 busiest files and then set the tests and generated ones aside, so it could find no file at all, or judge one far ahead of the next when it was not. `getTopRouteFile()` in the library counts every file's routes the same way. Thanks @inth3shadows for the report. (#2578)
 - In JavaScript and TypeScript, a call written as the body of an `if`, `while`, `for` or `do` without braces, like `if (e.key === "Escape") setOpen(false);`, no longer links to a same-named property or method in another file when the name is the file's own, such as a `setOpen` from `useState` (#2585).

__tests__/explore-long-query.test.ts

@@ -0,0 +1,390 @@
+/**
+ * Long `codegraph_explore` queries on a large index (#2620).
+ *
+ * The prompt hook hands explore the whole prompt, so a pasted log, document
+ * or source file reaches it as a query of hundreds of words, and every word
+ * cost several reads that each scanned the index: on a 560k-node index such
+ * a query took one to four minutes and the hook gave up. These tests pin the
+ * two halves of the fix:
+ *
+ * - the per-word reads give exactly the rows they gave before, now from
+ *   memory (the name-substring lookups, the fuzzy fallback's name list, the
+ *   LIKE fallback's "nothing to find" check), read once per database state;
+ * - a query searches at most MAX_QUERY_SYMBOLS names and MAX_QUERY_TERMS
+ *   terms, code-shaped words first, so its cost no longer grows with its
+ *   length.
+ */
+import { describe, it, expect, afterEach, vi } from 'vitest';
+import * as fs from 'fs';
+import * as path from 'path';
+import * as os from 'os';
+import CodeGraph from '../src';
+import { QueryBuilder } from '../src/db/queries';
+import { MAX_QUERY_SYMBOLS, MAX_QUERY_TERMS } from '../src/context';
+import { scorePathRelevance, extractSearchTerms, normalizeNameToken, isTestFile } from '../src/search/query-utils';
+import type { SqliteDatabase } from '../src/db/sqlite-adapter';
+
+function queriesOf(cg: CodeGraph): QueryBuilder {
+  return (cg as unknown as { queries: QueryBuilder }).queries;
+}
+function rawDb(cg: CodeGraph): SqliteDatabase {
+  return (cg as unknown as { db: { getDb(): SqliteDatabase } }).db.getDb();
+}
+
+/** Count the statements run whose SQL matches `pattern`, through the adapter. */
+function countStatements(db: SqliteDatabase, pattern: RegExp): { count: number } {
+  const counter = { count: 0 };
+  const prepare = db.prepare.bind(db);
+  vi.spyOn(db, 'prepare').mockImplementation((sql: string) => {
+    const stmt = prepare(sql);
+    if (!pattern.test(sql)) return stmt;
+    const wrap = <T extends (...a: any[]) => any>(fn: T): T => ((...args: any[]) => { counter.count++; return fn(...args); }) as T;
+    return { ...stmt, all: wrap(stmt.all.bind(stmt)), get: wrap(stmt.get.bind(stmt)), iterate: wrap(stmt.iterate.bind(stmt)), run: stmt.run.bind(stmt) };
+  });
+  return counter;
+}
+
+const DEF = ['class', 'interface', 'struct', 'union', 'trait', 'protocol', 'enum', 'type_alias'];
+const CALL = ['function', 'method', 'component'];
+const SEED = ['function', 'method', 'component', 'variable', 'constant'];
+
+/** The lookup as it was before the index: a LIKE scan, ties left to the plan. */
+function likeScan(db: SqliteDatabase, substring: string, kinds: string[], excludePrefix: boolean, limit: number): string[] {
+  let sql = 'SELECT nodes.*, 1.0 as score FROM nodes WHERE name LIKE ?';
+  const params: (string | number)[] = [`%${substring}%`];
+  if (excludePrefix) { sql += ' AND name NOT LIKE ?'; params.push(`${substring}%`); }
+  sql += ` AND kind IN (${kinds.map(() => '?').join(',')}) ORDER BY length(name) ASC LIMIT ?`;
+  return (db.prepare(sql).all(...params, ...kinds, limit) as Array<{ id: string }>).map((row) => row.id);
+}
+
+/** Many names of equal length across kinds and files, so ties decide the order. */
+function writeTieProject(dir: string): void {
+  const files: Record<string, string> = {};
+  for (let f = 0; f < 6; f++) {
+    const lines: string[] = [];
+    for (let i = 0; i < 12; i++) {
+      const n = (f * 7 + i * 5) % 12;
+      lines.push(`export function load${n}Item${f}(): void {}`);
+      lines.push(`export class Item${n}Loader${f} { loadItem${n}(): void {} }`);
+      lines.push(`export interface Item${n}Shape${f} { loadItem${n}Shape(): void; }`);
+      lines.push(`export const ITEM_${n}_LOADER_${f} = ${i};`);
+    }
+    lines.push('export function load_item_snake(): void {}');
+    lines.push('export function loadXitemXsnake(): void {}');
+    lines.push('export class LOADITEMLOUD {}');
+    lines.push('export function größeLaden(): void {}');
+    lines.push('export function GrößeLaden(): void {}');
+    lines.push('export function ÉtatLoader(): void {}');
+    lines.push('export function étatLoader(): void {}');
+    lines.push('export function 𝑥ItemLoader(): void {}');
+    files[`src/part${(f * 5) % 6}/items${f}.ts`] = lines.join('\n') + '\n';
+  }
+  for (const [rel, text] of Object.entries(files)) {
+    fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true });
+    fs.writeFileSync(path.join(dir, rel), text);
+  }
+}
+
+const PATTERNS = [
+  'Item', 'item', 'ITEM', 'Loader', 'Load', 'load', 'Item1', 'Item1loader', 'Loaditem', 'Shape', 'Oader', 'O',
+  'Load_item', 'Item_', '_', 'I_em', 'Load_i', 'Item_1', 'Größe', 'größe', 'GRÖSSE', 'État', 'état', 'Etat',
+  '𝑥', '𝑥item', 'Xitem', 'Loud', 'Missingword', 'Item.', 'Item(',
+];
+
+describe('name-substring lookups give the rows a LIKE scan gives (#2620)', () => {
+  let dir: string;
+  const open: CodeGraph[] = [];
+  afterEach(() => {
+    for (const cg of open.splice(0)) cg.close();
+    vi.restoreAllMocks();
+    if (dir) fs.rmSync(dir, { recursive: true, force: true });
+  });
+
+  async function setup(): Promise<CodeGraph> {
+    dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-substr-'));
+    writeTieProject(dir);
+    const cg = await CodeGraph.init(dir, { index: true });
+    open.push(cg);
+    return cg;
+  }
+
+  function expectSameRows(cg: CodeGraph, plan: 'rowid' | 'index'): void {
+    const q = queriesOf(cg);
+    const db = rawDb(cg);
+    let compared = 0;
+    for (const substring of PATTERNS) {
+      for (const kinds of [DEF, CALL, SEED, ['method'], ['constant', 'function']]) {
+        for (const excludePrefix of [false, true]) {
+          for (const limit of [200, 7, 1]) {
+            const expected = likeScan(db, substring, kinds, excludePrefix, limit);
+            const actual = q.findNodesByNameSubstring(substring, { kinds: kinds as never, excludePrefix, limit }).map((r) => r.node.id);
+            expect(actual, `${substring} ${kinds.join(',')} excludePrefix=${excludePrefix} limit=${limit}`).toEqual(expected);
+            compared++;
+          }
+        }
+      }
+    }
+    expect(compared).toBeGreaterThan(900);
+    // The answers came from the index, under the plan SQLite chose.
+    const memo = (q as unknown as { readMemos: Map<string, { value: { plans: Map<string, string | null> } | null }> })
+      .readMemos.get('nameSubstringIndex');
+    expect(memo?.value).toBeTruthy();
+    const plans = [...memo!.value!.plans.values()];
+    expect(plans).toContain(plan);
+    expect(plans).not.toContain(null);
+  }
+
+  it('matches when SQLite scans the table (a small index)', async () => {
+    const cg = await setup();
+    const plan = rawDb(cg).prepare(
+      `EXPLAIN QUERY PLAN SELECT * FROM nodes WHERE name LIKE '%x%' AND kind IN (${DEF.map(() => '?').join(',')}) ORDER BY length(name) LIMIT 200`,
+    ).all(...DEF) as Array<{ detail: string }>;
+    expect(plan.map((r) => r.detail)).toContain('SCAN nodes');
+    expectSameRows(cg, 'rowid');
+  });
+
+  it('matches when SQLite reads through the kind index (a large index)', async () => {
+    const cg = await setup();
+    const db = rawDb(cg);
+    // The statistics of a large index (vscode's), so the planner reads by kind.
+    db.exec('ANALYZE');
+    db.exec("UPDATE sqlite_stat1 SET stat = '562636' || substr(stat, instr(stat, ' ')) WHERE tbl = 'nodes' AND instr(stat, ' ') > 0");
+    db.exec("UPDATE sqlite_stat1 SET stat = '562636 1001 3 1 1' WHERE idx = 'idx_nodes_kind'");
+    db.exec('ANALYZE sqlite_schema');
+    const plan = db.prepare(
+      `EXPLAIN QUERY PLAN SELECT * FROM nodes WHERE name LIKE '%x%' AND kind IN (${DEF.map(() => '?').join(',')}) ORDER BY length(name) LIMIT 200`,
+    ).all(...DEF) as Array<{ detail: string }>;
+    expect(plan.map((r) => r.detail).some((d) => d.startsWith('SEARCH nodes USING INDEX idx_nodes_kind'))).toBe(true);
+    expectSameRows(cg, 'index');
+  });
+
+  it('asks SQLite inside a transaction, and for a lookup without a kind filter', async () => {
+    const cg = await setup();
+    const q = queriesOf(cg);
+    const db = rawDb(cg);
+    const scans = countStatements(db, /^\s*SELECT nodes\.\*, 1\.0 as score\s+FROM nodes\s+WHERE name LIKE \?/);
+    db.exec('BEGIN');
+    try {
+      expect(q.findNodesByNameSubstring('Item', { kinds: DEF as never, limit: 50 }).map((r) => r.node.id))
+        .toEqual(likeScan(db, 'Item', DEF, false, 50));
+    } finally {
+      db.exec('ROLLBACK');
+    }
+    expect(scans.count).toBe(2); // the lookup itself, and the comparison
+    q.findNodesByNameSubstring('Item', { limit: 50 });
+    expect(scans.count).toBe(3);
+    q.findNodesByNameSubstring('Item', { kinds: DEF as never, limit: 50 });
+    expect(scans.count).toBe(3);
+  });
+});
+
+describe('per-word reads happen once per database state (#2620)', () => {
+  let dir: string;
+  const open: CodeGraph[] = [];
+  afterEach(() => {
+    for (const cg of open.splice(0)) cg.close();
+    vi.restoreAllMocks();
+    if (dir) fs.rmSync(dir, { recursive: true, force: true });
+  });
+
+  async function setup(): Promise<CodeGraph> {
+    dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-perword-'));
+    fs.mkdirSync(path.join(dir, 'src', 'zorbtown'), { recursive: true });
+    fs.writeFileSync(path.join(dir, 'src', 'zorbtown', 'orders.ts'), [
+      'export class OrderStateMachine {',
+      '  submitOrder(): boolean { return this.transition("submitted"); }',
+      '  transition(state: string): boolean { return state.length > 0; }',
+      '}',
+      'export function linkSessions(ids: string[]): string[] { return ids; }',
+    ].join('\n') + '\n');
+    const cg = await CodeGraph.init(dir, { index: true });
+    open.push(cg);
+    return cg;
+  }
+
+  it('reads the name list once for every word the fuzzy fallback tries', async () => {
+    const cg = await setup();
+    const reads = vi.spyOn(queriesOf(cg), 'getAllNodeNames');
+    for (const word of ['submitordr', 'transitoin', 'linksesions', 'qwertyuiop', 'zxcvbnmasd']) cg.searchNodes(word);
+    expect(reads).toHaveBeenCalledTimes(1);
+    // A typo still finds its symbol.
+    expect(cg.searchNodes('transitoin').map((r) => r.node.name)).toContain('transition');
+  });
+
+  it('reads the names again after a sync or another connection\'s write', async () => {
+    const cg = await setup();
+    expect(cg.searchNodes('ordermachne').map((r) => r.node.name)).not.toContain('OrderMachine');
+    fs.writeFileSync(path.join(dir, 'src', 'zorbtown', 'more.ts'), 'export class OrderMachine {}\n');
+    await cg.sync();
+    expect(cg.searchNodes('ordermachne').map((r) => r.node.name)).toContain('OrderMachine');
+
+    const reader = await CodeGraph.open(dir);
+    open.push(reader);
+    expect(reader.searchNodes('invoicemachne').map((r) => r.node.name)).not.toContain('InvoiceMachine');
+    fs.writeFileSync(path.join(dir, 'src', 'zorbtown', 'invoice.ts'), 'export class InvoiceMachine {}\n');
+    await cg.sync();
+    expect(reader.searchNodes('invoicemachne').map((r) => r.node.name)).toContain('InvoiceMachine');
+  });
+
+  it('reads the names fresh inside a transaction, and does not keep them', async () => {
+    const cg = await setup();
+    const db = rawDb(cg);
+    expect(cg.searchNodes('linksesions').map((r) => r.node.name)).toContain('linkSessions');
+    db.exec('BEGIN');
+    try {
+      db.exec("DELETE FROM nodes WHERE name = 'linkSessions'");
+      expect(cg.searchNodes('linksesions').map((r) => r.node.name)).not.toContain('linkSessions');
+    } finally {
+      db.exec('ROLLBACK');
+    }
+    expect(cg.searchNodes('linksesions').map((r) => r.node.name)).toContain('linkSessions');
+  });
+
+  it('skips the LIKE scan for a word no name or qualified name holds, and still finds one inside a qualified name', async () => {
+    const cg = await setup();
+    const scans = countStatements(rawDb(cg), /qualified_name LIKE \?/);
+    // No token starts with these, so FTS misses and the LIKE fallback runs.
+    expect(cg.searchNodes('qwzxv')).toEqual([]);
+    expect(cg.searchNodes('plmokn')).toEqual([]);
+    expect(scans.count).toBe(0);
+    // "rbtow" is inside the folder name in a file's qualified name (its path),
+    // and inside no symbol name.
+    const inPath = cg.searchNodes('rbtow');
+    expect(inPath.map((r) => r.node.filePath)).toContain('src/zorbtown/orders.ts');
+    // "tateMach" is inside a name.
+    expect(cg.searchNodes('tatemach').map((r) => r.node.name)).toContain('OrderStateMachine');
+    expect(scans.count).toBe(2);
+  });
+});
+
+describe('a long query searches a bounded number of words (#2620)', () => {
+  let dir: string;
+  const open: CodeGraph[] = [];
+  afterEach(() => {
+    for (const cg of open.splice(0)) cg.close();
+    vi.restoreAllMocks();
+    if (dir) fs.rmSync(dir, { recursive: true, force: true });
+  });
+
+  async function setup(): Promise<CodeGraph> {
+    dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-longquery-'));
+    fs.writeFileSync(path.join(dir, 'orders.ts'), [
+      'export class OrderStateMachine {',
+      '  submitOrder(): boolean { return this.transition("submitted"); }',
+      '  transition(state: string): boolean { return state.length > 0; }',
+      '}',
+      'export function reconcileLedger(): void {}',
+      'export function zebrafinchquota(): void {}',
+    ].join('\n') + '\n');
+    const cg = await CodeGraph.init(dir, { index: true });
+    open.push(cg);
+    return cg;
+  }
+
+  /** Distinct lowercase filler words that name nothing in the project. */
+  function filler(n: number): string {
+    const words: string[] = [];
+    for (let i = 0; i < n; i++) words.push(`word${String.fromCharCode(97 + (i % 26))}${String.fromCharCode(97 + Math.floor(i / 26) % 26)}x`);
+    return words.join(' ');
+  }
+
+  it('runs a bounded number of searches however long the query is', async () => {
+    const cg = await setup();
+    const q = queriesOf(cg);
+    const search = vi.spyOn(q, 'searchNodes');
+    const substring = vi.spyOn(q, 'findNodesByNameSubstring');
+    await cg.findRelevantContext(`how does submitOrder work? ${filler(400)}`);
+    // A name is searched as a definition prefix (with its stems) and by four
+    // substring lookups; a term once.
+    expect(search.mock.calls.length).toBeLessThanOrEqual(MAX_QUERY_SYMBOLS * 4 + MAX_QUERY_TERMS);
+    expect(substring.mock.calls.length).toBeLessThanOrEqual(MAX_QUERY_SYMBOLS * 4);
+  });
+
+  it('searches a bag of symbol names whole', async () => {
+    const cg = await setup();
+    const search = vi.spyOn(queriesOf(cg), 'searchNodes');
+    const bag = 'AuthService AuthController TokenStore SessionManager UserRepository PasswordHasher JwtStrategy '
+      + 'RefreshTokenService LoginHandler LogoutHandler AuthGuard RolesGuard PermissionService AuditLogger '
+      + 'RateLimiter CsrfMiddleware reconcileLedger';
+    expect(extractSearchTerms(bag).length).toBeGreaterThan(64);
+    expect(extractSearchTerms(bag).length).toBeLessThanOrEqual(MAX_QUERY_TERMS);
+    await cg.findRelevantContext(bag);
+    const searched = new Set(search.mock.calls.map((c) => String(c[0]).toLowerCase()));
+    for (const term of extractSearchTerms(bag)) expect(searched).toContain(term);
+  });
+
+  it('still searches every word of a query within the bound', async () => {
+    const cg = await setup();
+    const search = vi.spyOn(queriesOf(cg), 'searchNodes');
+    await cg.findRelevantContext(`${filler(20)} zebrafinchquota`);
+    const searched = new Set(search.mock.calls.map((c) => String(c[0]).toLowerCase()));
+    expect(searched).toContain('zebrafinchquota');
+    expect(searched).toContain('wordaax');
+  });
+
+  it('keeps the identifiers of a long paste, wherever they are in it', async () => {
+    const cg = await setup();
+    const result = await cg.findRelevantContext(`${filler(300)} where does reconcileLedger run?`);
+    expect([...result.nodes.values()].map((n) => n.name)).toContain('reconcileLedger');
+  });
+
+  it('drops the plain words past the bound', async () => {
+    const cg = await setup();
+    const search = vi.spyOn(queriesOf(cg), 'searchNodes');
+    await cg.findRelevantContext(`${filler(300)} zebrafinchquota`);
+    const searched = new Set(search.mock.calls.map((c) => String(c[0]).toLowerCase()));
+    expect(searched).not.toContain('zebrafinchquota');
+  });
+});
+
+describe('path relevance scores are unchanged by the per-query cache (#2620)', () => {
+  /** scorePathRelevance as it was, re-splitting the query for every path. */
+  function reference(filePath: string, query: string, projectNameTokens?: Set<string>, isDeprioritized?: boolean): number {
+    const pathLower = filePath.toLowerCase();
+    const fileName = path.basename(filePath).toLowerCase();
+    const dirName = path.dirname(filePath).toLowerCase();
+    let score = 0;
+    const allWords = query.split(/\s+/).filter((w) => w.length > 0);
+    if (allWords.length === 0) return 0;
+    const words = projectNameTokens && projectNameTokens.size > 0
+      ? allWords.filter((w) => !projectNameTokens.has(normalizeNameToken(w)))
+      : allWords;
+    const scored = words.length > 0 ? words : allWords;
+    for (const word of scored) {
+      const subtokens = extractSearchTerms(word, { stems: false });
+      if (subtokens.length === 0) continue;
+      if (subtokens.some((t) => fileName.includes(t))) score += 10;
+      if (subtokens.some((t) => dirName.includes(t))) score += 5;
+      else if (subtokens.some((t) => pathLower.includes(t))) score += 3;
+    }
+    const queryLower = query.toLowerCase();
+    const isTestQuery = queryLower.includes('test') || queryLower.includes('spec');
+    if ((!isTestQuery && isTestFile(filePath)) || isDeprioritized === true) score -= 15;
+    return score;
+  }
+
+  it('scores every path as before, repeated words and project names included', () => {
+    const queries = [
+      'how does the order service submit an order',
+      'OrderService.submit order ORDER order_service submitOrder',
+      'SuperBizAgent agent routes for the agent',
+      'test the order service', '   ', '', 'spec', 'the of and',
+      Array.from({ length: 200 }, (_, i) => ['order', 'service', 'ledger', 'Submit', 'superbizagent'][i % 5]).join(' '),
+    ];
+    const paths = [
+      'src/orders/order-service.ts', 'src/orders/__tests__/order-service.test.ts', 'SuperBizAgent/agent/routes.ts',
+      'lib/ledger.rb', 'packages/submit/index.js', 'examples/order/main.go', 'README.md', 'a',
+    ];
+    const tokens = [undefined, new Set<string>(), new Set(['superbizagent'])];
+    for (const query of queries) {
+      for (const filePath of paths) {
+        for (const projectNameTokens of tokens) {
+          for (const dep of [undefined, true, false]) {
+            expect(scorePathRelevance(filePath, query, projectNameTokens, dep), `${query} | ${filePath}`)
+              .toBe(reference(filePath, query, projectNameTokens, dep));
+          }
+        }
+      }
+    }
+  });
+});

src/context/index.ts

@@ -37,6 +37,21 @@ import { describeSynthesizedHop } from '../graph/synthesized-hop';
  */
 const MAX_COPYBOOK_DECLARATION_ENTRIES = 3;
 
+/**
+ * The most symbol names, and search terms, of one query that
+ * `findRelevantContext` searches the index for. A question or a bag of symbol
+ * names stays well below both: a name like `RefreshTokenService` is one
+ * symbol but several terms (`refreshtokenservice`, `refresh`, `token`,
+ * `service`, stems). A pasted log, document or source file runs to hundreds
+ * of words, and every word costs several searches of the index, so such a
+ * query took minutes on a large project and the prompt hook gave up on it
+ * (#2620). Both lists put code-shaped words (camelCase, snake_case, dotted
+ * names) first and keep the order they appear in, so a long query is searched
+ * by its identifiers and its opening words.
+ */
+export const MAX_QUERY_SYMBOLS = 64;
+export const MAX_QUERY_TERMS = 128;
+
 /**
  * Extract likely symbol names from a natural language query
  *
@@ -220,9 +235,14 @@ export class ContextBuilder {
    * and its non-production budget cap are deliberately NOT joined: those REMOVE
    * content, and `deprioritize` is a ranking lever by definition — `exclude` is
    * the lever for taking things out of reach.
+   *
+   * Returned as a predicate with the patterns read once, for the candidate
+   * loops of one search: asking the live matcher per candidate re-checked
+   * `codegraph.json` hundreds of thousands of times on a long query (#2620).
    */
-  private isDeprioritized(filePath: string): boolean {
+  private deprioritizedPaths(): (filePath: string) => boolean {
-    return this.queries.getDeprioritizedPathMatcher()?.(filePath) ?? false;
+    const matcher = this.queries.getDeprioritizedPathSnapshot?.() ?? this.queries.getDeprioritizedPathMatcher?.();
+    return (filePath) => matcher?.(filePath) ?? false;
   }
 
   /**
@@ -473,7 +493,7 @@ export class ContextBuilder {
     // === HYBRID SEARCH ===
 
     // Step 1: Extract potential symbol names from query
-    const symbolsFromQuery = extractSymbolsFromQuery(query);
+    const symbolsFromQuery = extractSymbolsFromQuery(query).slice(0, MAX_QUERY_SYMBOLS);
     logDebug('Extracted symbols from query', { query, symbols: symbolsFromQuery });
 
     // Step 2: Look up exact matches for extracted symbols
@@ -639,7 +659,8 @@ export class ContextBuilder {
       // fell through to the fuzzy fallback and returned whatever program name
       // sat within edit distance (`CVACT01Y` → `CBACT01C`).
       const searchTerms = extractSearchTerms(query)
-        .filter((term) => !copybooks.members.has(term.toUpperCase()));
+        .filter((term) => !copybooks.members.has(term.toUpperCase()))
+        .slice(0, MAX_QUERY_TERMS);
       if (searchTerms.length > 0) {
         // Search each term individually to get broader coverage,
         // then boost results that match multiple terms
@@ -759,7 +780,7 @@ export class ContextBuilder {
     // than nodes matching just one generic term. Without this, "ExecutionUtils"
     // (matches only "execution") fills budget slots meant for "ShardSearchRequest"
     // (matches "shard" + "search" + "request").
-    const queryTermsForBoost = extractSearchTerms(query);
+    const queryTermsForBoost = extractSearchTerms(query).slice(0, MAX_QUERY_TERMS);
     if (queryTermsForBoost.length >= 2) {
       // Group terms that are substrings of each other (stem variants of the same
       // root word). "indexed", "indexe", "index" should count as ONE concept match,
@@ -853,6 +874,18 @@ export class ContextBuilder {
       // SEPARATE LIKE batch so one hot single-word term can't crowd classes
       // out of the length-ordered 200-row batch.
       const camelCallableKinds: NodeKind[] = ['function', 'method', 'component'];
+      // A path's relevance to the query depends on nothing else, and the
+      // candidates of every term share their files: score each path once.
+      const isDeprioritized = this.deprioritizedPaths();
+      const pathScores = new Map<string, number>();
+      const pathScoreOf = (filePath: string): number => {
+        let score = pathScores.get(filePath);
+        if (score === undefined) {
+          score = scorePathRelevance(filePath, query, undefined, isDeprioritized(filePath));
+          pathScores.set(filePath, score);
+        }
+        return score;
+      };
       const camelSearchedTerms = new Set<string>();
       const searchIdSet = new Set(searchResults.map(r => r.node.id));
       // Track per-node term hits for multi-term boosting
@@ -902,12 +935,7 @@ export class ContextBuilder {
           if (searchIdSet.has(r.node.id)) continue;
           if (isTestFile(r.node.filePath) && !isTestQuery) continue;
 
-          const pathScore = scorePathRelevance(
+          const pathScore = pathScoreOf(r.node.filePath);
-            r.node.filePath,
-            query,
-            undefined,
-            this.isDeprioritized(r.node.filePath),
-          );
           const brevityBonus = Math.max(0, 6 - (name.length - titleCased.length) / 4);
           termCandidates.push({ node: r.node, score: 8 + brevityBonus + pathScore });
         }
@@ -994,12 +1022,7 @@ export class ContextBuilder {
         const compoundResults: SearchResult[] = [];
         for (const [, entry] of compoundTermMap) {
           if (entry.terms.size >= 2) {
-            const pathScore = scorePathRelevance(
+            const pathScore = pathScoreOf(entry.node.filePath);
-              entry.node.filePath,
-              query,
-              undefined,
-              this.isDeprioritized(entry.node.filePath),
-            );
             const brevityBonus = Math.max(0, 6 - entry.node.name.length / 8);
             compoundResults.push({
               node: entry.node,

src/db/name-substring-index.ts

@@ -0,0 +1,222 @@
+/**
+ * An in-memory answer to the substring lookup `codegraph_explore` runs for
+ * every word of its query (#2620):
+ *
+ *   SELECT … FROM nodes WHERE name LIKE '%word%' [AND name NOT LIKE 'word%']
+ *     AND kind IN (…) ORDER BY length(name) LIMIT n
+ *
+ * A leading `%` can't use an index, so SQLite reads every node of those kinds
+ * for each call: about 80 ms on a 560k-node index, four calls a word, and a
+ * pasted log or document has hundreds of words. This index reads the names
+ * once and answers each call with one string search.
+ *
+ * It returns exactly what SQLite would. LIKE folds case for ASCII letters only,
+ * `_` matches any one character, and `length()` counts characters, not UTF-16
+ * units, so all three are done that way here. SQLite leaves rows of equal
+ * length in the order its plan read them: through `idx_nodes_kind` (kind,
+ * file_path, start_line, id) on a large index, by rowid when it scans a small
+ * one. The index keeps both orders, and the caller says which plan SQLite
+ * chose. A pattern with a `%` is left to SQLite.
+ */
+
+/**
+ * The kinds the index holds: the definitions and callables explore's
+ * camel-case and compound steps ask about, and the variables and constants its
+ * named-symbol seeding adds.
+ */
+export const NAME_SUBSTRING_KINDS: readonly string[] = [
+  'class', 'component', 'constant', 'enum', 'function', 'interface', 'method',
+  'protocol', 'struct', 'trait', 'type_alias', 'union', 'variable',
+];
+
+/** Separates the names in {@link NameSubstringIndex.hay}; no name may contain it. */
+const SEPARATOR = '\u0000';
+
+export interface NameSubstringIndex {
+  /** Kind codes: `kinds[entryKind[e]]` is entry `e`'s kind. */
+  readonly kinds: readonly string[];
+  /** Every distinct ASCII-lowercased name, each followed by {@link SEPARATOR}. */
+  readonly hay: string;
+  /** Where each distinct name starts in `hay`, plus one past the end. */
+  readonly starts: Int32Array;
+  /** The entries of distinct name `i` are `nameEntries[nameEntryStart[i] … nameEntryStart[i+1])`, ascending. */
+  readonly nameEntryStart: Int32Array;
+  readonly nameEntries: Int32Array;
+  /** Per entry, in index order. */
+  readonly entryRowid: Float64Array;
+  readonly entryKind: Uint8Array;
+  readonly entryLength: Int32Array;
+}
+
+/** Lowercase ASCII letters only, as SQLite's LIKE compares them. */
+function asciiLower(text: string): string {
+  return /[A-Z]/.test(text) ? text.replace(/[A-Z]+/g, (run) => run.toLowerCase()) : text;
+}
+
+/** `length(text)` as SQLite counts it: characters, so a surrogate pair is one. */
+function sqliteLength(text: string): number {
+  let length = text.length;
+  for (let i = 0; i < text.length; i++) {
+    const code = text.charCodeAt(i);
+    if (code >= 0xd800 && code <= 0xdbff && i + 1 < text.length) {
+      const next = text.charCodeAt(i + 1);
+      if (next >= 0xdc00 && next <= 0xdfff) { length--; i++; }
+    }
+  }
+  return length;
+}
+
+/**
+ * Build the index from rows of {@link NAME_SUBSTRING_KINDS} read in
+ * `idx_nodes_kind` order, which becomes the entries' order. Returns null when
+ * a name holds the separator, so the caller keeps asking SQLite.
+ */
+export function buildNameSubstringIndex(
+  rows: Iterable<{ rowid: number; kind: string; name: string }>,
+): NameSubstringIndex | null {
+  const kindCodes = new Map(NAME_SUBSTRING_KINDS.map((kind, i) => [kind, i]));
+  const nameIds = new Map<string, number>();
+  const names: string[] = [];
+  const rowids: number[] = [];
+  const kinds: number[] = [];
+  const lengths: number[] = [];
+  const nameOf: number[] = [];
+  for (const row of rows) {
+    const code = kindCodes.get(row.kind);
+    if (code === undefined) continue;
+    const lowered = asciiLower(row.name);
+    if (lowered.includes(SEPARATOR)) return null;
+    let id = nameIds.get(lowered);
+    if (id === undefined) {
+      id = names.length;
+      nameIds.set(lowered, id);
+      names.push(lowered);
+    }
+    rowids.push(row.rowid);
+    kinds.push(code);
+    lengths.push(sqliteLength(row.name));
+    nameOf.push(id);
+  }
+
+  const starts = new Int32Array(names.length + 1);
+  let offset = 0;
+  for (let i = 0; i < names.length; i++) {
+    starts[i] = offset;
+    offset += names[i]!.length + SEPARATOR.length;
+  }
+  starts[names.length] = offset;
+
+  // Group entries by name, keeping index order inside each group.
+  const nameEntryStart = new Int32Array(names.length + 1);
+  for (const id of nameOf) nameEntryStart[id + 1]!++;
+  for (let i = 0; i < names.length; i++) nameEntryStart[i + 1]! += nameEntryStart[i]!;
+  const fill = nameEntryStart.slice(0, names.length);
+  const nameEntries = new Int32Array(nameOf.length);
+  for (let entry = 0; entry < nameOf.length; entry++) nameEntries[fill[nameOf[entry]!]!++] = entry;
+
+  return {
+    kinds: NAME_SUBSTRING_KINDS,
+    hay: names.join(SEPARATOR) + SEPARATOR,
+    starts,
+    nameEntryStart,
+    nameEntries,
+    entryRowid: Float64Array.from(rowids),
+    entryKind: Uint8Array.from(kinds),
+    entryLength: Int32Array.from(lengths),
+  };
+}
+
+/** The distinct name whose span in `hay` holds `offset`. */
+function nameAt(starts: Int32Array, offset: number): number {
+  let lo = 0;
+  let hi = starts.length - 2;
+  while (lo < hi) {
+    const mid = (lo + hi + 1) >> 1;
+    if (starts[mid]! <= offset) lo = mid; else hi = mid - 1;
+  }
+  return lo;
+}
+
+/**
+ * A matcher for a LIKE pattern without `%`: where the next match at or after
+ * `from` starts, and whether one starts exactly at `at`. Without `_` this is a
+ * plain string search; each `_` matches any one character but the separator.
+ */
+function patternMatcher(needle: string): { next(hay: string, from: number): number; startsAt(hay: string, at: number): boolean } {
+  if (!needle.includes('_')) {
+    return {
+      next: (hay, from) => hay.indexOf(needle, from),
+      startsAt: (hay, at) => hay.startsWith(needle, at),
+    };
+  }
+  // `u`, so a wildcard takes a whole character, as SQLite's `_` does.
+  const source = needle
+    .split('_')
+    .map((part) => part.replace(/[\^$\\.*+?()[\]{}|/]/g, '\\$&'))
+    .join('[^\\u0000]');
+  const anywhere = new RegExp(source, 'gu');
+  const anchored = new RegExp(source, 'uy');
+  return {
+    next(hay, from) {
+      anywhere.lastIndex = from;
+      const match = anywhere.exec(hay);
+      return match ? match.index : -1;
+    },
+    startsAt(hay, at) {
+      anchored.lastIndex = at;
+      return anchored.test(hay);
+    },
+  };
+}
+
+/**
+ * The order SQLite's plan reads candidate rows in, which its ORDER BY keeps
+ * for rows of equal length: the kind index's, or the table's by rowid.
+ */
+export type ScanOrder = 'index' | 'rowid';
+
+/**
+ * The rowids SQLite's substring lookup returns, in its order, or null when the
+ * index can't answer exactly (a `%` in the pattern, an empty pattern, or a
+ * kind it doesn't hold). `order` is the order the plan reads rows in.
+ */
+export function searchNameSubstring(
+  index: NameSubstringIndex,
+  substring: string,
+  kinds: readonly string[],
+  excludePrefix: boolean,
+  limit: number,
+  order: ScanOrder,
+): number[] | null {
+  const needle = asciiLower(substring);
+  if (needle.length === 0 || needle.includes('%') || needle.includes(SEPARATOR)) return null;
+  const wanted = new Uint8Array(index.kinds.length);
+  for (const kind of kinds) {
+    const code = index.kinds.indexOf(kind);
+    if (code < 0) return null;
+    wanted[code] = 1;
+  }
+
+  const { hay, starts, nameEntryStart, nameEntries, entryKind, entryLength } = index;
+  const matcher = patternMatcher(needle);
+  const hits: number[] = [];
+  for (let at = matcher.next(hay, 0); at >= 0; ) {
+    const name = nameAt(starts, at);
+    if (!(excludePrefix && matcher.startsAt(hay, starts[name]!))) {
+      for (let k = nameEntryStart[name]!; k < nameEntryStart[name + 1]!; k++) {
+        const entry = nameEntries[k]!;
+        if (wanted[entryKind[entry]!]) hits.push(entry);
+      }
+    }
+    at = matcher.next(hay, starts[name + 1]!);
+  }
+  // Entry numbers follow the kind index, rowids the table: either breaks
+  // length ties the way SQLite's plan does.
+  const rowid = index.entryRowid;
+  hits.sort(order === 'index'
+    ? (a, b) => (entryLength[a]! - entryLength[b]!) || (a - b)
+    : (a, b) => (entryLength[a]! - entryLength[b]!) || (rowid[a]! - rowid[b]!));
+  // A negative LIMIT is no limit in SQLite.
+  if (limit >= 0 && hits.length > limit) hits.length = limit;
+  return hits.map((entry) => index.entryRowid[entry]!);
+}

src/db/queries.ts

@@ -23,6 +23,7 @@ import { parseQuery, boundedEditDistance } from '../search/query-parser';
 import { isGeneratedFile } from '../extraction/generated-detection';
 import { splitIdentifierSegments } from '../search/identifier-segments';
 import { isModuleKey, referenceNameTail, MEMBER_TAIL_SQL, MODULE_PARKED_SQL } from './reference-tail';
+import { NAME_SUBSTRING_KINDS, buildNameSubstringIndex, searchNameSubstring, type NameSubstringIndex, type ScanOrder } from './name-substring-index';
 
 /**
  * Files that should not be candidates for "dominant file" detection: tests,
@@ -107,6 +108,15 @@ const THROUGH_NAMES = (column: string): string => `(
  */
 export const DEPRIORITIZED_NAME_BONUS_SCALE = 0.75;
 
+/**
+ * A `deprioritize` path predicate (#982) that can also hand out a copy with
+ * the project's patterns read once, for loops over many paths (#2620).
+ */
+export interface DeprioritizedPathMatcher {
+  (filePath: string): boolean;
+  snapshot?: () => (filePath: string) => boolean;
+}
+
 /**
  * The fields that tell one node of a file from another across a re-index of
  * that file, whose node ids change with every line shift (#2276).
@@ -308,6 +318,12 @@ export class QueryBuilder {
     | { stamp: string; value: { filePath: string; edgeCount: number; nextEdgeCount: number } | null }
     | undefined;
 
+  // Query-independent reads that a search repeats for every word of its query
+  // — the name list the fuzzy fallback scans, the name-substring index — each
+  // tagged with the change stamp it was read under, like dominantFileMemo
+  // (#2620). See memoizedRead().
+  private readMemos = new Map<string, { stamp: string; value: unknown }>();
+
   // Prepared statements (lazily initialized)
   private stmts: {
     insertNode?: SqliteStatement;
@@ -448,6 +464,26 @@ export class QueryBuilder {
     // different databases report the same one — the memo goes with the old
     // connection, or a worker following a rebuilt index keeps its answer (#1864).
     this.dominantFileMemo = undefined;
+    this.readMemos.clear();
+  }
+
+  /**
+   * `compute()`'s answer, reused until the database changes (see
+   * getChangeStamp), or null inside a transaction, where a rollback could
+   * leave a stale answer under an unchanged stamp: the caller then reads the
+   * database directly, as it did before the memo.
+   */
+  private memoizedRead<T>(key: string, compute: () => T): T | null {
+    if (this.db.inTransaction !== false) {
+      this.readMemos.delete(key);
+      return null;
+    }
+    const stamp = this.getChangeStamp();
+    const memo = this.readMemos.get(key);
+    if (memo?.stamp === stamp) return memo.value as T;
+    const value = compute();
+    this.readMemos.set(key, { stamp, value });
+    return value;
   }
 
   private edgeKindStmt(sql: string): SqliteStatement {
@@ -486,6 +522,17 @@ export class QueryBuilder {
     return this.isDeprioritizedPath;
   }
 
+  /**
+   * The `deprioritize` predicate with the project's patterns read once, for a
+   * loop that asks about many paths. The plain predicate re-checks
+   * `codegraph.json` on every call, which a long query's candidate loops made
+   * hundreds of thousands of times (#2620).
+   */
+  getDeprioritizedPathSnapshot(): ((filePath: string) => boolean) | undefined {
+    const matcher = this.isDeprioritizedPath as DeprioritizedPathMatcher | undefined;
+    return matcher?.snapshot ? matcher.snapshot() : matcher;
+  }
+
   // ===========================================================================
   // Node Operations
   // ===========================================================================
@@ -1698,6 +1745,7 @@ export class QueryBuilder {
     // Apply multi-signal scoring
     if (results.length > 0 && (text || query)) {
       const scoringQuery = text || query;
+      const isDeprioritizedPath = this.getDeprioritizedPathSnapshot();
       results = results.map(r => {
         // A path the project de-prioritized is saying its symbol NAMES are not
         // the answer, so the exact-name bonus has to be damped too. The -15 path
@@ -1705,8 +1753,9 @@ export class QueryBuilder {
         // on #982's repro, a `usage()` helper sat at 74.8 vs 51.2 for the top
         // product symbol — -15 lands at 59.8, still ahead). Damped, not zeroed,
         // so the tree stays findable when it genuinely is what you asked for.
-        // Evaluated once and reused: the predicate stats the config file.
+        // Evaluated once per result and reused, from a snapshot that read the
-        const deprioritized = this.isDeprioritizedPath?.(r.node.filePath) ?? false;
+        // config once for the whole loop.
+        const deprioritized = isDeprioritizedPath?.(r.node.filePath) ?? false;
         const nameBonus = nameMatchBonus(r.node.name, scoringQuery);
         return {
           ...r,
@@ -1789,15 +1838,20 @@ export class QueryBuilder {
     const lowered = text.toLowerCase();
     const maxDist = lowered.length <= 4 ? 1 : 2;
 
-    // Pull the distinct name list once. The set is cached on QueryBuilder
+    // The distinct name list, lowercased, read once until the database
-    // by getAllNodeNames(); even on a 200k-node project the distinct
+    // changes: this fallback runs for every query word that FTS and LIKE both
-    // name set is typically O(10k) because most names repeat. The
+    // miss, which is most words of a pasted log, and the DISTINCT scan alone
-    // candidate-cap below bounds memory regardless.
+    // took ~150 ms a call on a 560k-node index (#2620). Inside a transaction
-    const allNames = this.getAllNodeNames();
+    // it is read fresh, as before.
+    const { names, lowered: loweredNames, starts } = this.fuzzyNames();
     const candidates: Array<{ name: string; dist: number }> = [];
-    for (const name of allNames) {
+    for (let i = 0; i < names.length; i++) {
-      const dist = boundedEditDistance(name.toLowerCase(), lowered, maxDist);
+      // An edit distance is at least the length difference, so a name outside
-      if (dist <= maxDist) candidates.push({ name, dist });
+      // this window could never pass the bound: skipping it changes nothing.
+      const end = starts[i + 1]! - 1;
+      if (Math.abs(end - starts[i]! - lowered.length) > maxDist) continue;
+      const dist = boundedEditDistance(loweredNames.slice(starts[i]!, end), lowered, maxDist);
+      if (dist <= maxDist) candidates.push({ name: names[i]!, dist });
     }
     candidates.sort((a, b) => a.dist - b.dist);
 
@@ -1952,10 +2006,18 @@ export class QueryBuilder {
     // nameMatchBonus) can promote results that BM25 alone undervalues.
     const ftsLimit = Math.max(limit * 5, 100);
 
+    // Joined on rowid: nodes_fts is external-content over nodes with
+    // content_rowid='rowid', so its `id` column is read back from the very row
+    // the rowid names. The integer key gives the same rows without a lookup
+    // through the TEXT id index for each match, which roughly halves this
+    // query on a large index, where explore runs it once or twice for every
+    // query word (#2620). CROSS JOIN keeps nodes_fts the outer loop, as it was:
+    // on a small index the planner would otherwise scan nodes and probe FTS
+    // once per node.
     let sql = `
       SELECT nodes.*, bm25(nodes_fts, 0, 20, 5, 1, 2) as score
       FROM nodes_fts
-      JOIN nodes ON nodes_fts.id = nodes.id
+      CROSS JOIN nodes ON nodes.rowid = nodes_fts.rowid
       WHERE nodes_fts MATCH ?
     `;
 
@@ -1993,6 +2055,21 @@ export class QueryBuilder {
   private searchNodesLike(query: string, options: SearchOptions): SearchResult[] {
     const { kinds, languages, limit = 100, offset = 0 } = options;
 
+    // This runs when FTS found nothing, and a word FTS can't find is usually
+    // in no name at all: most words of a pasted log or document. Each scan
+    // read every node, about 120 ms on a 560k-node index (#2620). A word of
+    // ASCII letters and digits can only match inside a name, or inside one
+    // run of such characters in a qualified name, and lowercasing keeps those
+    // runs whole, so when neither lowercased list holds it there is nothing
+    // to find. Any other word is scanned as before.
+    if (/^[A-Za-z0-9]+$/.test(query)) {
+      const runs = this.memoizedRead('qualifiedNameRuns', () => this.readQualifiedNameRuns());
+      if (runs !== null) {
+        const word = query.toLowerCase();
+        if (!this.fuzzyNames().lowered.includes(word) && !runs.includes(word)) return [];
+      }
+    }
+
     let sql = `
       SELECT nodes.*,
         CASE
@@ -2183,6 +2260,22 @@ export class QueryBuilder {
     sql += ' ORDER BY length(name) ASC LIMIT ?';
     params.push(limit);
 
+    // Explore asks this four times for every word of its query, and the
+    // leading `%` makes each one a scan of every node of those kinds (#2620).
+    // The in-memory index answers with the same rows in the same order: ties
+    // in length come out in the order SQLite's plan for this very statement
+    // reads rows, the kind index's or the table's. Anything it can't answer
+    // exactly (no kind filter, a language filter, a `%` in the word, another
+    // plan, a transaction) is asked of SQLite.
+    if (kinds && kinds.length > 0 && !(languages && languages.length > 0)) {
+      const index = this.memoizedRead('nameSubstringIndex', () => this.readNameSubstringIndex());
+      const order = index ? this.substringScanOrder(index, sql, params, excludePrefix ? 2 : 1) : null;
+      const rowids = order ? searchNameSubstring(index!.index, substring, kinds, excludePrefix === true, limit, order) : null;
+      if (rowids) {
+        return this.getNodeRowsByRowid(rowids).map((row) => ({ node: rowToNode(row), score: 1.0 }));
+      }
+    }
+
     const rows = this.db.prepare(sql).all(...params) as (NodeRow & { score: number })[];
     return rows.map((row) => ({
       node: rowToNode(row),
@@ -2190,6 +2283,76 @@ export class QueryBuilder {
     }));
   }
 
+  /**
+   * The order SQLite's plan for a substring lookup reads candidate rows in,
+   * which is the order its ORDER BY leaves ties of equal length in: the kind
+   * index's (kind, file_path, start_line, id), or the table's (rowid). Which
+   * one depends on the statistics — a small index is scanned whole — so it is
+   * asked of SQLite once per statement shape. Null for any other plan.
+   */
+  private substringScanOrder(
+    memo: { index: NameSubstringIndex; plans: Map<string, ScanOrder | null> },
+    sql: string,
+    params: readonly (string | number)[],
+    patternCount: number,
+  ): ScanOrder | null {
+    // The patterns (the first `patternCount` parameters) can't change the
+    // plan, since a leading `%` uses no index; the kinds and the limit can.
+    const key = `${sql}\u0000${params.slice(patternCount).join('\u0000')}`;
+    let order = memo.plans.get(key);
+    if (order === undefined) {
+      order = null;
+      try {
+        const details = (this.db.prepare(`EXPLAIN QUERY PLAN ${sql}`).all(...params) as Array<{ detail: string }>)
+          .map((row) => row.detail)
+          .filter((detail) => /^(SCAN|SEARCH) (TABLE )?nodes\b/.test(detail));
+        if (details.length === 1) {
+          if (/^SEARCH (TABLE )?nodes USING (COVERING )?INDEX idx_nodes_kind \(/.test(details[0]!)) order = 'index';
+          else if (/^SCAN (TABLE )?nodes$/.test(details[0]!)) order = 'rowid';
+        }
+      } catch {
+        order = null;
+      }
+      memo.plans.set(key, order);
+    }
+    return order;
+  }
+
+  /**
+   * Every node of the kinds the name-substring index holds, read in
+   * idx_nodes_kind order (kind, file_path, start_line, id): the order its
+   * results break length ties by.
+   */
+  private readNameSubstringIndex(): { index: NameSubstringIndex; plans: Map<string, ScanOrder | null> } | null {
+    const rows = this.db
+      .prepare(
+        `SELECT rowid AS rowid, kind, name FROM nodes WHERE kind IN (${NAME_SUBSTRING_KINDS.map(() => '?').join(',')})`
+        + ' ORDER BY kind, file_path, start_line, id'
+      )
+      .all(...NAME_SUBSTRING_KINDS) as Array<{ rowid: number; kind: string; name: string }>;
+    const index = buildNameSubstringIndex(rows);
+    return index ? { index, plans: new Map() } : null;
+  }
+
+  /** Whole node rows for `rowids`, in that order; a rowid no longer present is skipped. */
+  private getNodeRowsByRowid(rowids: readonly number[]): NodeRow[] {
+    if (rowids.length === 0) return [];
+    const byRowid = new Map<number, NodeRow>();
+    for (let i = 0; i < rowids.length; i += SQLITE_PARAM_CHUNK_SIZE) {
+      const chunk = rowids.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
+      const rows = this.db
+        .prepare(`SELECT rowid AS rowid_, nodes.* FROM nodes WHERE rowid IN (${chunk.map(() => '?').join(',')})`)
+        .all(...chunk) as Array<NodeRow & { rowid_: number }>;
+      for (const row of rows) byRowid.set(row.rowid_, row);
+    }
+    const out: NodeRow[] = [];
+    for (const rowid of rowids) {
+      const row = byRowid.get(rowid);
+      if (row) out.push(row);
+    }
+    return out;
+  }
+
   // ===========================================================================
   // Edge Operations
   // ===========================================================================
@@ -4054,6 +4217,40 @@ export class QueryBuilder {
     return rows.map((r) => r.name);
   }
 
+  /**
+   * The distinct node names, in {@link getAllNodeNames} order, and the same
+   * names lowercased and newline-joined: name `i` lowercased is
+   * `lowered.slice(starts[i], starts[i + 1] - 1)`. Read once until the
+   * database changes (fresh inside a transaction). The fuzzy fallback compares
+   * against it, and the LIKE fallback asks it whether any name holds a word.
+   */
+  private fuzzyNames(): { names: string[]; lowered: string; starts: Int32Array } {
+    const read = (): { names: string[]; lowered: string; starts: Int32Array } => {
+      const names = this.getAllNodeNames();
+      const lowered = names.map((name) => name.toLowerCase());
+      const starts = new Int32Array(names.length + 1);
+      for (let i = 0; i < lowered.length; i++) starts[i + 1] = starts[i]! + lowered[i]!.length + 1;
+      return { names, lowered: lowered.join('\n') + '\n', starts };
+    };
+    return this.memoizedRead('fuzzyNames', read) ?? read();
+  }
+
+  /**
+   * The runs of ASCII letters and digits in the distinct qualified names,
+   * lowercased and newline-joined: where a word LIKE could find in a
+   * qualified name has to be.
+   */
+  private readQualifiedNameRuns(): string {
+    const runs = new Set<string>();
+    const rows = this.db.prepare('SELECT DISTINCT qualified_name AS name FROM nodes').all() as Array<{ name: string }>;
+    for (const row of rows) {
+      for (const run of row.name.toLowerCase().split(/[^a-z0-9]+/)) {
+        if (run) runs.add(run);
+      }
+    }
+    return [...runs].join('\n');
+  }
+
   /** The names of a language's nodes of the given kinds (repeats included), decoding nothing else. */
   getNodeNamesOfKinds(language: Language, kinds: readonly NodeKind[]): string[] {
     // `+language`: the kind index narrows further than the language one.

src/index.ts

@@ -30,7 +30,7 @@ import {
 } from './types';
 import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db';
 import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve';
-import { QueryBuilder } from './db/queries';
+import { QueryBuilder, type DeprioritizedPathMatcher } from './db/queries';
 import { importPathKeys, moduleReferenceKeys } from './db/reference-tail';
 import {
   isInitialized,
@@ -218,24 +218,47 @@ export class CodeGraph {
     // on the pattern array's identity, which the cache keeps stable.
     let cachedPatterns: string[] | undefined;
     let cachedMatcher: ReturnType<typeof ignore> | undefined;
-    this.queries.setDeprioritizedPathMatcher((filePath: string): boolean => {
+    // The compiled matcher for the patterns `codegraph.json` holds right now,
+    // or null when there are none.
+    const currentMatcher = (): ReturnType<typeof ignore> | null => {
+      const patterns = loadDeprioritizePatterns(this.projectRoot);
+      if (patterns.length === 0) return null;
+      if (patterns !== cachedPatterns) {
+        cachedPatterns = patterns;
+        cachedMatcher = ignore().add(patterns);
+      }
+      return cachedMatcher!;
+    };
+    const matches = (matcher: ReturnType<typeof ignore> | null, filePath: string): boolean => {
+      if (!matcher) return false;
+      const rel = path.isAbsolute(filePath)
+        ? path.relative(this.projectRoot, filePath)
+        : filePath;
+      if (!rel || rel.startsWith('..')) return false;
+      return matcher.ignores(rel.split(path.sep).join('/'));
+    };
+    const deprioritized: DeprioritizedPathMatcher = (filePath: string): boolean => {
       try {
-        const patterns = loadDeprioritizePatterns(this.projectRoot);
+        return matches(currentMatcher(), filePath);
-        if (patterns.length === 0) return false;
-        if (patterns !== cachedPatterns) {
-          cachedPatterns = patterns;
-          cachedMatcher = ignore().add(patterns);
-        }
-        const rel = path.isAbsolute(filePath)
-          ? path.relative(this.projectRoot, filePath)
-          : filePath;
-        if (!rel || rel.startsWith('..')) return false;
-        return cachedMatcher!.ignores(rel.split(path.sep).join('/'));
       } catch {
         // Ranking must never take the search down with it.
         return false;
       }
-    });
+    };
+    // One `stat` for a whole loop of candidates instead of one per candidate
+    // (#2620): a search reads the config once, then asks the frozen matcher.
+    deprioritized.snapshot = () => {
+      let matcher: ReturnType<typeof ignore> | null = null;
+      try { matcher = currentMatcher(); } catch { matcher = null; }
+      return (filePath: string): boolean => {
+        try {
+          return matches(matcher, filePath);
+        } catch {
+          return false;
+        }
+      };
+    };
+    this.queries.setDeprioritizedPathMatcher(deprioritized);
 
     this.orchestrator = new ExtractionOrchestrator(this.projectRoot, this.queries);
     this.resolver = createResolver(this.projectRoot, this.queries);

src/search/query-utils.ts

@@ -215,19 +215,30 @@ export function extractSearchTerms(query: string, options?: { stems?: boolean })
 }
 
 /**
- * Score path relevance to a query
+ * The query half of {@link scorePathRelevance}: the sub-tokens of each query
- * Higher score = more relevant path
+ * word that is scored, with how many times the word occurs, and whether the
+ * query is about tests.
  */
-export function scorePathRelevance(
+interface PathRelevanceQuery {
-  filePath: string,
+  projectNameTokens: Set<string> | undefined;
-  query: string,
+  /** No words at all: every path scores 0. */
-  projectNameTokens?: Set<string>,
+  empty: boolean;
-  isDeprioritized?: boolean,
+  words: Array<{ subtokens: string[]; count: number }>;
-): number {
+  isTestQuery: boolean;
-  const pathLower = filePath.toLowerCase();
+}
-  const fileName = path.basename(filePath).toLowerCase();
+
-  const dirName = path.dirname(filePath).toLowerCase();
+/**
-  let score = 0;
+ * The few most recent queries' halves. A search scores every candidate path
+ * against one query, and re-splitting every word of it for each path was
+ * most of the scoring time on a long query: a pasted page is hundreds of
+ * words, and explore scores thousands of paths (#2620).
+ */
+const pathRelevanceQueries = new Map<string, PathRelevanceQuery>();
+const PATH_RELEVANCE_QUERY_CACHE = 8;
+
+function pathRelevanceQuery(query: string, projectNameTokens: Set<string> | undefined): PathRelevanceQuery {
+  const cached = pathRelevanceQueries.get(query);
+  if (cached && cached.projectNameTokens === projectNameTokens) return cached;
 
   // Score per original query WORD, not per sub-token. A single PascalCase word
   // splits into many sub-tokens (a project name "SuperBizAgent" →
@@ -239,30 +250,72 @@ export function scorePathRelevance(
   // camelCase/snake split per word (so `getUserName` still matches a
   // `get_user_name` path) — we just attribute each word's matches once.
   const allWords = query.split(/\s+/).filter((w) => w.length > 0);
-  if (allWords.length === 0) return 0;
 
   // A query word that just names the PROJECT (its go.mod / package.json / repo
   // name) carries no discriminative path signal — drop it so the rest of the
   // query decides the ranking, instead of every file under a `<ProjectName>…/`
   // tree winning on the project name alone (#720). Only when OTHER words remain,
   // so a bare project-name query still scores on its path.
-  const words =
+  const kept =
     projectNameTokens && projectNameTokens.size > 0
       ? allWords.filter((w) => !projectNameTokens.has(normalizeNameToken(w)))
       : allWords;
-  const scored = words.length > 0 ? words : allWords;
+  const scored = kept.length > 0 ? kept : allWords;
 
+  // A word that occurs twice adds its score twice; words with the same
+  // sub-tokens score alike, so each such group is matched once and counted.
+  const groups = new Map<string, { subtokens: string[]; count: number }>();
   for (const word of scored) {
     // Use base terms only — stem variants inflate path scores by generating
     // many near-duplicate terms that all match the same path segments.
     const subtokens = extractSearchTerms(word, { stems: false });
     if (subtokens.length === 0) continue;
+    const key = subtokens.join('\u0000');
+    const group = groups.get(key);
+    if (group) group.count++;
+    else groups.set(key, { subtokens, count: 1 });
+  }
+
+  const queryLower = query.toLowerCase();
+  const prepared: PathRelevanceQuery = {
+    projectNameTokens,
+    empty: allWords.length === 0,
+    words: [...groups.values()],
+    isTestQuery: queryLower.includes('test') || queryLower.includes('spec'),
+  };
+  pathRelevanceQueries.delete(query);
+  pathRelevanceQueries.set(query, prepared);
+  if (pathRelevanceQueries.size > PATH_RELEVANCE_QUERY_CACHE) {
+    pathRelevanceQueries.delete(pathRelevanceQueries.keys().next().value!);
+  }
+  return prepared;
+}
+
+/**
+ * Score path relevance to a query
+ * Higher score = more relevant path
+ */
+export function scorePathRelevance(
+  filePath: string,
+  query: string,
+  projectNameTokens?: Set<string>,
+  isDeprioritized?: boolean,
+): number {
+  const prepared = pathRelevanceQuery(query, projectNameTokens);
+  if (prepared.empty) return 0;
+
+  const pathLower = filePath.toLowerCase();
+  const fileName = path.basename(filePath).toLowerCase();
+  const dirName = path.dirname(filePath).toLowerCase();
+  let score = 0;
+
+  for (const { subtokens, count } of prepared.words) {
     // Exact filename match (strongest)
-    if (subtokens.some((t) => fileName.includes(t))) score += 10;
+    if (subtokens.some((t) => fileName.includes(t))) score += 10 * count;
     // Directory match
-    if (subtokens.some((t) => dirName.includes(t))) score += 5;
+    if (subtokens.some((t) => dirName.includes(t))) score += 5 * count;
     // General path match
-    else if (subtokens.some((t) => pathLower.includes(t))) score += 3;
+    else if (subtokens.some((t) => pathLower.includes(t))) score += 3 * count;
   }
 
   // Deprioritize test files unless the query is explicitly about tests, and
@@ -274,9 +327,7 @@ export function scorePathRelevance(
   //    standing statement by the project, so it is NOT waived. The name-bonus
   //    damping at the call site is what keeps such a tree findable.
   //  - a path that is both is docked ONCE, not twice.
-  const queryLower = query.toLowerCase();
+  const offTarget = (!prepared.isTestQuery && isTestFile(filePath)) || isDeprioritized === true;
-  const isTestQuery = queryLower.includes('test') || queryLower.includes('spec');
-  const offTarget = (!isTestQuery && isTestFile(filePath)) || isDeprioritized === true;
   if (offTarget) {
     score -= 15;
   }