fix(explore): a long query, like a pasted log, answers in seconds on a large index (#2804)

The prompt hook hands codegraph_explore the whole prompt, so a pasted
log, document or source file arrives as a query of hundreds of words.
findRelevantContext searched every word several times, and the reads
behind those searches scanned the index each time:

- findNodesByNameSubstring, four times per symbol word: `name LIKE
  '%word%'` with a leading wildcard reads every node of those kinds,
  about 80 ms each on vscode (562,636 nodes);
- the LIKE fallback after an FTS miss (`name LIKE` or `qualified_name
  LIKE '%word%'`), about 120 ms per miss, and most words of a log miss;
- the fuzzy fallback after that re-read `SELECT DISTINCT name` each time
  (about 150 ms), and getAllNodeNames, documented as cached, was not;
- FTS joined nodes on the TEXT id instead of the rowid;
- scorePathRelevance re-split every query word for every scored path,
  and the deprioritize predicate stat'ed codegraph.json per candidate.

On vscode a 1.7k-character log took 80 s in explore, a 10k-character
skill document 258 s, and the hook gave up on both. A per-statement
profile put 65-80% of that in those scans.

The fix:
- A name-substring index answers the substring lookups from memory with
  exactly the rows SQLite returns, in its order: ASCII-only case folding
  and `_` as one character, like LIKE, and ties in length in the order
  the plan reads rows, the kind index's or rowid's. The plan is asked of
  SQLite once per statement shape, since a small index is scanned whole;
  a `%`, a language filter or any other plan goes to SQLite as before.
- A word of letters and digits that no name and no letters-and-digits
  run of a qualified name holds can match nothing, so the LIKE fallback
  returns empty without scanning for it.
- The fuzzy fallback reads the name list once and skips names outside
  its edit-distance length window.
- These reads are memoized per database change stamp, like the dominant
  file (PR #1864): rebuilt after any write from any connection, never
  kept inside a transaction, dropped on rebind.
- FTS joins on rowid, with CROSS JOIN keeping nodes_fts the outer loop
  as before.
- Path relevance prepares each query once and each path is scored once
  per search; the deprioritize patterns are read once per search.
- findRelevantContext searches at most 64 symbol names and 128 search
  terms of a query. Both lists put code-shaped words first, so a long
  paste is searched by its identifiers and its opening words. A
  16-symbol bag (75 terms) stays whole. This is the one intended change
  in output, for long queries only.

Validation (vscode ff0a86dc, median of 3 interleaved, fresh process):
- explore, ordinary queries: 6 of 6 outputs byte-identical, 5.5-7.5 s
  fresh (the dominant-file aggregation is about 4 s of that) and
  0.9-2.1 s warm vs 2.7-3.9 s on main;
- explore, long queries: log 1.7k 80.1 to 10.7 s, markdown 3k 73.3 to
  9.0 s, Java 3k 108.8 to 12.2 s, TS 3k 63.4 to 11.9 s, markdown 10k
  258.3 to 10.8 s, Java 10k 144.7 to 11.1 s, TS 10k 146.7 to 9.6 s;
- the real prompt hook on the issue's prompt shapes: short 7.2 to
  7.0 s, log 84.2 to 12.9 s, markdown 3k 78.1 to 10.2 s, Java 3k 113.1
  to 13.3 s, markdown 10k 261.7 to 11.3 s, Java 10k 146.3 to 11.1 s;
- without the word caps, all 7 long queries gave byte-identical output
  4.3-9.0x faster;
- django 79502d08 and prometheus ed26d727: 11 of 11 queries
  byte-identical, tracebacks included, 1.0-2.9x faster;
- the substring index against the LIKE scan on vscode: 4,830 of 4,830
  lookups identical (805 words, snake_case included);
- explore-long-query.test.ts: 13 tests; 8 fail on main's source.

Issue #2620. Closed contributor PR #2621 (retsamekohC) was read for
guidance; open PR #2260 (danusha2345) overlaps on the name-list memo
and path relevance.

Fixes #2620

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/context/index.ts

@@ -37,6 +37,21 @@ import { describeSynthesizedHop } from '../graph/synthesized-hop';
  */
 const MAX_COPYBOOK_DECLARATION_ENTRIES = 3;
 
+/**
+ * The most symbol names, and search terms, of one query that
+ * `findRelevantContext` searches the index for. A question or a bag of symbol
+ * names stays well below both: a name like `RefreshTokenService` is one
+ * symbol but several terms (`refreshtokenservice`, `refresh`, `token`,
+ * `service`, stems). A pasted log, document or source file runs to hundreds
+ * of words, and every word costs several searches of the index, so such a
+ * query took minutes on a large project and the prompt hook gave up on it
+ * (#2620). Both lists put code-shaped words (camelCase, snake_case, dotted
+ * names) first and keep the order they appear in, so a long query is searched
+ * by its identifiers and its opening words.
+ */
+export const MAX_QUERY_SYMBOLS = 64;
+export const MAX_QUERY_TERMS = 128;
+
 /**
  * Extract likely symbol names from a natural language query
  *
@@ -220,9 +235,14 @@ export class ContextBuilder {
    * and its non-production budget cap are deliberately NOT joined: those REMOVE
    * content, and `deprioritize` is a ranking lever by definition — `exclude` is
    * the lever for taking things out of reach.
+   *
+   * Returned as a predicate with the patterns read once, for the candidate
+   * loops of one search: asking the live matcher per candidate re-checked
+   * `codegraph.json` hundreds of thousands of times on a long query (#2620).
    */
-  private isDeprioritized(filePath: string): boolean {
-    return this.queries.getDeprioritizedPathMatcher()?.(filePath) ?? false;
+  private deprioritizedPaths(): (filePath: string) => boolean {
+    const matcher = this.queries.getDeprioritizedPathSnapshot?.() ?? this.queries.getDeprioritizedPathMatcher?.();
+    return (filePath) => matcher?.(filePath) ?? false;
   }
 
   /**
@@ -473,7 +493,7 @@ export class ContextBuilder {
     // === HYBRID SEARCH ===
 
     // Step 1: Extract potential symbol names from query
-    const symbolsFromQuery = extractSymbolsFromQuery(query);
+    const symbolsFromQuery = extractSymbolsFromQuery(query).slice(0, MAX_QUERY_SYMBOLS);
     logDebug('Extracted symbols from query', { query, symbols: symbolsFromQuery });
 
     // Step 2: Look up exact matches for extracted symbols
@@ -639,7 +659,8 @@ export class ContextBuilder {
       // fell through to the fuzzy fallback and returned whatever program name
       // sat within edit distance (`CVACT01Y` → `CBACT01C`).
       const searchTerms = extractSearchTerms(query)
-        .filter((term) => !copybooks.members.has(term.toUpperCase()));
+        .filter((term) => !copybooks.members.has(term.toUpperCase()))
+        .slice(0, MAX_QUERY_TERMS);
       if (searchTerms.length > 0) {
         // Search each term individually to get broader coverage,
         // then boost results that match multiple terms
@@ -759,7 +780,7 @@ export class ContextBuilder {
     // than nodes matching just one generic term. Without this, "ExecutionUtils"
     // (matches only "execution") fills budget slots meant for "ShardSearchRequest"
     // (matches "shard" + "search" + "request").
-    const queryTermsForBoost = extractSearchTerms(query);
+    const queryTermsForBoost = extractSearchTerms(query).slice(0, MAX_QUERY_TERMS);
     if (queryTermsForBoost.length >= 2) {
       // Group terms that are substrings of each other (stem variants of the same
       // root word). "indexed", "indexe", "index" should count as ONE concept match,
@@ -853,6 +874,18 @@ export class ContextBuilder {
       // SEPARATE LIKE batch so one hot single-word term can't crowd classes
       // out of the length-ordered 200-row batch.
       const camelCallableKinds: NodeKind[] = ['function', 'method', 'component'];
+      // A path's relevance to the query depends on nothing else, and the
+      // candidates of every term share their files: score each path once.
+      const isDeprioritized = this.deprioritizedPaths();
+      const pathScores = new Map<string, number>();
+      const pathScoreOf = (filePath: string): number => {
+        let score = pathScores.get(filePath);
+        if (score === undefined) {
+          score = scorePathRelevance(filePath, query, undefined, isDeprioritized(filePath));
+          pathScores.set(filePath, score);
+        }
+        return score;
+      };
       const camelSearchedTerms = new Set<string>();
       const searchIdSet = new Set(searchResults.map(r => r.node.id));
       // Track per-node term hits for multi-term boosting
@@ -902,12 +935,7 @@ export class ContextBuilder {
           if (searchIdSet.has(r.node.id)) continue;
           if (isTestFile(r.node.filePath) && !isTestQuery) continue;
 
-          const pathScore = scorePathRelevance(
-            r.node.filePath,
-            query,
-            undefined,
-            this.isDeprioritized(r.node.filePath),
-          );
+          const pathScore = pathScoreOf(r.node.filePath);
           const brevityBonus = Math.max(0, 6 - (name.length - titleCased.length) / 4);
           termCandidates.push({ node: r.node, score: 8 + brevityBonus + pathScore });
         }
@@ -994,12 +1022,7 @@ export class ContextBuilder {
         const compoundResults: SearchResult[] = [];
         for (const [, entry] of compoundTermMap) {
           if (entry.terms.size >= 2) {
-            const pathScore = scorePathRelevance(
-              entry.node.filePath,
-              query,
-              undefined,
-              this.isDeprioritized(entry.node.filePath),
-            );
+            const pathScore = pathScoreOf(entry.node.filePath);
             const brevityBonus = Math.max(0, 6 - entry.node.name.length / 8);
             compoundResults.push({
               node: entry.node,