The index stores a name in any script as one full-text token, and
`codegraph query 로그인` found it, but every tokenizer between a question
and the index knew only ASCII: extractSearchTerms split on [^a-zA-Z0-9]
and kept 3+ characters, context's extractSymbolsFromQuery matched ASCII
patterns only, and explore's exploreQueryTokens and the Flow's flowTokens
required a leading [A-Za-z_$] and 3+ characters. So `explore 寻路服务`
answered "No relevant code found", `context 로그인` printed only its
header and `explore 入口 末端` had no Flow, while their ASCII twins worked.
Shared helpers in search/query-utils now read words in any script whole:
letters, digits and combining marks, cut where Han, kana or Hangul meets
another script (NavMeshAgent类型 is NavMeshAgent and 类型, never
agent类型), at least 2 characters in Han, kana or Hangul and 3 elsewhere,
so Russian на and из stay out. extractSearchTerms and the explore and
Flow tokens use them. Context adds names with a non-ASCII letter as whole
runs, and a mixed name like get用户信息 as written, with no n-grams, so
a partial Chinese word still never becomes a search of its own (PR #2005).
The ASCII-only patterns see a word with accented or other non-ASCII
letters blanked out, so état is no longer read as tat. A Han, kana or
Hangul token counts as a deliberately named identifier, as such a name
has no case or underscore to show it, so a query of such names is
neither damped as prose nor flagged low-confidence.
Validation: new cjk-query-tokens tests (12) cover CJK, mixed, Cyrillic
and accented words, reference copies of the old tokenizers over an ASCII
corpus, and explore and context end to end on Korean, Chinese and
Japanese fixtures; 9 fail on main's source. Explore and context A/B with
214 ASCII queries (gin 76, Alamofire 79, express 59; 49 lead with a
Flow): all byte-identical. Of 12 non-ASCII queries, 6 changed on gin and
4 on express, as intended: état, Señal and naïve are read whole, and
handleHTTPRequest处理 now names handleHTTPRequest. Tokenizing a
10,000-character query takes at most 1.8 ms, against 0.1-0.3 ms before.
Reimplements PR #516 by @andy-sg and PR #1262 by @yimingll.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prompt hook hands codegraph_explore the whole prompt, so a pasted
log, document or source file arrives as a query of hundreds of words.
findRelevantContext searched every word several times, and the reads
behind those searches scanned the index each time:
- findNodesByNameSubstring, four times per symbol word: `name LIKE
'%word%'` with a leading wildcard reads every node of those kinds,
about 80 ms each on vscode (562,636 nodes);
- the LIKE fallback after an FTS miss (`name LIKE` or `qualified_name
LIKE '%word%'`), about 120 ms per miss, and most words of a log miss;
- the fuzzy fallback after that re-read `SELECT DISTINCT name` each time
(about 150 ms), and getAllNodeNames, documented as cached, was not;
- FTS joined nodes on the TEXT id instead of the rowid;
- scorePathRelevance re-split every query word for every scored path,
and the deprioritize predicate stat'ed codegraph.json per candidate.
On vscode a 1.7k-character log took 80 s in explore, a 10k-character
skill document 258 s, and the hook gave up on both. A per-statement
profile put 65-80% of that in those scans.
The fix:
- A name-substring index answers the substring lookups from memory with
exactly the rows SQLite returns, in its order: ASCII-only case folding
and `_` as one character, like LIKE, and ties in length in the order
the plan reads rows, the kind index's or rowid's. The plan is asked of
SQLite once per statement shape, since a small index is scanned whole;
a `%`, a language filter or any other plan goes to SQLite as before.
- A word of letters and digits that no name and no letters-and-digits
run of a qualified name holds can match nothing, so the LIKE fallback
returns empty without scanning for it.
- The fuzzy fallback reads the name list once and skips names outside
its edit-distance length window.
- These reads are memoized per database change stamp, like the dominant
file (PR #1864): rebuilt after any write from any connection, never
kept inside a transaction, dropped on rebind.
- FTS joins on rowid, with CROSS JOIN keeping nodes_fts the outer loop
as before.
- Path relevance prepares each query once and each path is scored once
per search; the deprioritize patterns are read once per search.
- findRelevantContext searches at most 64 symbol names and 128 search
terms of a query. Both lists put code-shaped words first, so a long
paste is searched by its identifiers and its opening words. A
16-symbol bag (75 terms) stays whole. This is the one intended change
in output, for long queries only.
Validation (vscode ff0a86dc, median of 3 interleaved, fresh process):
- explore, ordinary queries: 6 of 6 outputs byte-identical, 5.5-7.5 s
fresh (the dominant-file aggregation is about 4 s of that) and
0.9-2.1 s warm vs 2.7-3.9 s on main;
- explore, long queries: log 1.7k 80.1 to 10.7 s, markdown 3k 73.3 to
9.0 s, Java 3k 108.8 to 12.2 s, TS 3k 63.4 to 11.9 s, markdown 10k
258.3 to 10.8 s, Java 10k 144.7 to 11.1 s, TS 10k 146.7 to 9.6 s;
- the real prompt hook on the issue's prompt shapes: short 7.2 to
7.0 s, log 84.2 to 12.9 s, markdown 3k 78.1 to 10.2 s, Java 3k 113.1
to 13.3 s, markdown 10k 261.7 to 11.3 s, Java 10k 146.3 to 11.1 s;
- without the word caps, all 7 long queries gave byte-identical output
4.3-9.0x faster;
- django 79502d08 and prometheus ed26d727: 11 of 11 queries
byte-identical, tracebacks included, 1.0-2.9x faster;
- the substring index against the LIKE scan on vscode: 4,830 of 4,830
lookups identical (805 words, snake_case included);
- explore-long-query.test.ts: 13 tests; 8 fail on main's source.
Issue #2620. Closed contributor PR #2621 (retsamekohC) was read for
guidance; open PR #2260 (danusha2345) overlaps on the name-list memo
and path relevance.
Fixes#2620
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
isTestPath knew `tests/`, `test/`, `__tests__/`, CamelCase source sets and
`foo-test/` modules, but not the folders LLVM, Clang, clangd, lldb, offload
and Breakpad keep their unit tests in (`unittests/`), nor glog's
`dcheck_unittest/`-style test programs: `/unittests/` holds `tests/` only
after a `t`, and the `foo_test/` form wants a separator right before `test`.
So a helper there with no test-like name (LLVM's CountCopyAndMove.h,
SelectionDAGTestBase.h, Breakpad's dump_analysis.cc) read as production code.
isTestPath now also counts a `unittests/` folder and a `foo_unittest/` or
`foo-unittests/` one. A bare `unittest/` stays production: CPython's
`Lib/unittest/` is the framework, and LLVM vendors googletest under
`third-party/unittest/`.
isTestSuitePath counts the same folders as suites, as it counts `tests/`:
their files build into test binaries and never link into the program, and
glog builds each `foo_unittest/` folder as a test program of its own (a
`foo-test/` module stays out, as shipped test support). That matters for
C/C++ because the suite rule's CamelCase file names skip `.cpp`, so on main
none of LLVM's `*Test.cpp` files were suites and production locals named
`Ctx`, `KV`, `V` or `OpInfo` linked to what a unit test declares.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Google names C++ tests `foo_unittest.cc` (protobuf, Breakpad, glog,
Chromium), not `foo_test.cc`. The test-file check wanted a separator right
before `test`, so each one read as production code: search and explore
ranked it alongside the code it tests, `codegraph affected` never listed
it, and the resolver's test rules ran backwards for it. Production calls
landed on a unittest's own declarations (`stack_frame_entries_.size()` on
a unittest's `StackHelper`, a template's `AddressType()` on a unittest's
typedef), and a unittest's calls into test suites outside what it
includes (a Python `*_unittest.py` using a `tests/` fake) were treated as
production calls. #2421 already links a C/C++ unittest to the test
helpers it includes.
`unittest` and `unittests` join the separator-delimited test suffixes
(`foo_unittest.cc`, `foo-unittest.cpp`, `foo.unittest.js`), and a file
named so is a test suite to the resolver, as `foo_test.cc` is. A bare
`unittest.go` stays production code: promtool's runs rule tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bare basename several directories share (`editorOptions.ts`, `worker.rs`,
`compiler.py`) pinned every match, and the matches split the pinned
reservation: on vscode `editorOptions.ts clampedInt cursorStyleToString
cursorStyleFromString` gave the file holding all three 8,626 chars, against
15,181 without the path, and spent the rest on a workbench helper defining
none of them.
Path extraction now takes an injected `symbolFiles` lookup (the module stays
DB-free, like `existsOnDisk`). When a span matches several files, the matches
defining a precise symbol token the query names (camelCase, PascalCase,
snake_case, qualified — the seeder's NL-stopword shape test) are pinned and the
rest set aside; the summary line names them. When no match defines one, or
every match does, the span resolves exactly as before. The same test resolves
a basename shared past the ambiguity budget (django's `models.py`) when it
narrows into budget, and a line anchor binds once the span narrows to one file.
Replay of 196 queries over 8 indexed repos, main vs this branch: every
unambiguous pin is byte-identical (38 unique basenames, 13 dir/basename, 8
file:line, 38 earlier-sweep path queries), as are 26 path-free queries, 13
where every match defines a named symbol and 8 naming no symbol. All 52
changed responses are shared basenames — named FULL 207 -> 231 (11 up, 0 down), named lines 6,045 -> 7,577,
none over 25K.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`SQLCompiler.as_sql pre_sql_setup get_select` on django returned get_select and
pre_sql_setup in full and SQLCompiler.as_sql -- qualified, step 1 of the Flow,
226 lines -- as one signature line. compiler.py renders as a per-symbol
focused view, which picked bodies in source order within a tier, so the
unnamed bridge get_qualify_sql and get_select (both above as_sql in the file)
took the cap. The follow-ups agents issue (`compiler.py:776`,
`compiler.py lines 900-1003`) pinned the file and dropped the line numbers.
Every agent run ended in a Read of compiler.py.
- query-paths: keep `:776` / `:12-40` / `#L88-L120` suffixes as line anchors
and bind prose ranges (`lines 900-1003`, `L900-L1003`, `lines 900 to 1003`)
to the nearest single-file path.
- EXACT targets: a qualified name resolving to <=3 callables, or the callable
enclosing a single-line anchor. Multi-line anchors render as that span.
- Focused view: tiers exact > named spine step > unnamed bridge > unique named >
family co-named, spine in call order. A tier 0-1 body that does not fit is
windowed (head + calls into the question's other symbols) with the explore
range that returns each hole, never reduced to a signature. Priced in
rendered chars net of the signature lines, so the section keeps its promise.
- Cluster path: exact clusters rank first with the spine's bounded overshoot,
shrink by exact rendered size, and hold back what lower-ranked exact
clusters still owe; files holding one are funded like spine files.
- Blast radius leads with exact targets.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A query span like `scripts/deploy` was neither pinned nor reported.
`isClearlyPathShaped` demands a dot-extension on the last segment, so a
dotless slashed span that resolved to no indexed file fell through every
arm: it was not consumed, never entered `unresolvedPathSpans`, and its
fragments (`scripts`, `deploy`) went on to feed FTS. The agent got a pile
of unrelated source with no hint that the file it named was never read.
Both caveat sinks — the empty-subgraph note and the summary line — were
already in place; they were simply never handed the span.
Shape alone cannot decide this: `and/or`, `input/output` and
`client/server` have exactly the same shape, and loosening the shape test
to "has a slash" strips that prose out of every query and mints a false
"No indexed file uniquely matches `and/or`" caveat (the existing
`leaves slash-bearing non-paths alone` test pins this). The file's
existence on disk is the discriminator instead — and since query-paths.ts
is deliberately pure (no DB, no fs, safe in the query-pool workers), the
check is injected by the caller as an `existsOnDisk` predicate. The call
site in tools.ts owns the project root and runs it through
`validatePathWithinRoot`, so a `../` span cannot probe outside the
project, and only a regular file counts — a directory span keeps flowing
to the normal matching pipeline.
Retrieval effect (AGENTS.md "do not regress"), measured as a deterministic
A/B on the built dist over an indexed repro, baseline = upstream/main:
- query that is only the dotless path — before: 998 chars of unrelated
source, no caveat; after: `No relevant code found ... (no indexed file
uniquely matches `scripts/deploy`)`.
- same path inside a real symbol query — output byte-identical except the
added caveat sentence (+51 chars); the same 4 symbols across 2 files.
- query with no path span — byte-identical between arms.
Claude-Session: https://claude.ai/code/session_01XDR1wm73oH8J8cRWnKyv9m
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
- resolution/tier-synthesizer.ts: http-client (literal fetch/axios/ky/got/$fetch paths, axios.create baseURL instances, template holes as :params, base-URL holes by a two-segment tail; unique match only), queue-job (BullMQ/Bull add ↔ @Process/@Processor, WorkerHost process, new Worker, queue.process), event-bus (EventEmitter2 emit ↔ @OnEvent with globs; socket emit ↔ @SubscribeMessage / socket.on both ways with tier); channel, tier, callee, registeredAt on every edge; generic transport events never pair; test and generated files never sources; registered before the emitter pass
- steps.ts: crossing() reads tier/channel before languages; an endpoint reached across a tier is a bridge box and a boundary like a screen (through=1 enters it); a channel's call is not also an effect; sites read as written; a Next 'use server' action is a crossing by its directive (when.ts directive); a function-valued constant handler (asyncHandler(...)) is a route root and borrows the file-scope calls and refs within its lines
- express.ts: app.use('/prefix', router) mounts composed onto route names in postExtract (nested, by import or require); chained router.route('/x').get(h).put(h2) extracted, across lines
- frameworks/package-deps.ts: dependencies read from workspace package.json files too (Express, React, Expo Router, NestJS detect)
- routing manifest names constant handlers; e2e/ is a test directory; explore's Flow section labels the new channels
- tests: ui-steps-cross-tier (monorepo fixture: Next client + Express/Nest API), servers test updated for the queue landing
- docs: CHANGELOG, spec §3.13 cross-tier paragraph, CLAUDE.md, callback-edge-synthesis.md, plan P3 built
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01REFyW9hmNrxhwN5wxRoAkC
- api/route-roots.ts: the symbol a route runs (references-edge handler, exported page component, or the route itself for an inline handler), shared by steps and screens; the bare Steps tab lists an API's endpoints by router file
- api/effects.ts: database / response / queue / email / payments / cache / auth / process / network / storage / device / telemetry, matched on the call as written per language family, with model + read/write and the literal status on a response site
- graph/branch-guards.ts: callSitesForFile (the whole member chain), memberTypesInTree, decoratorsForFile, request/decorator triggers with the middleware/guard chain; guard + argument rules for Python, Java, Kotlin, C#, Go and C
- steps.ts: classify on the chain before trusting a name match, retarget this.x.y() by declared type, skip test doubles after the effect pre-check, project kind on the wire
- viewer: kindWord/kindWords per project kind, endpoint chooser, response boxes labelled by status codes
- python.ts: FastAPI detected from a monorepo sub-directory; is-test-file: samples/examples package paths are not tests
- tests: ui-steps-api-servers, ui-effects, branch-guards-languages; spec §3.13 Servers paragraph, CHANGELOG, plan doc
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01REFyW9hmNrxhwN5wxRoAkC
`#/entry` answers "where does anything start" at full length, and turns any row
that names a symbol into a flow.
Server. `/api/entrypoints` gains `frameworks` (from `getDetectedFrameworks`), a
`tests` list, a `routes` limit of its own, and a cache keyed on the index build
— nothing here is read from disk, so unlike `/api/source` a cached answer cannot
be stale about drift. `routes.items` is now a `WireList` like every other list on
the payload.
Routes carry where the URL is REGISTERED as well as where it is served:
`getRoutingManifest` selects the route node's id, file and line, and
`buildRoutes` splits the verb off the name against a fixed list (never "the
first word", which would take the head off a file-routed `/blog/[slug]`). All
four payroll-go routes register in one router file and three are served from
another — group by the handler file and one router becomes two groups plus an
orphan.
`isTestFile` is split into `isTestPath` (test filename and directory
conventions) + the non-production catch-all, byte-identical at every existing
call site. The Tests list uses the narrow half: an example, a benchmark or a
fixture is off-target for ranking but is not a test, and a heading that says
"Tests" must not quietly count them. Tests rank by REACH — distinct other files
touched — because Go, Rust and Java put test work inside functions where a
module-level-calls ranking sees nothing. Two read-only engine queries make that
affordable: `getFileReachCounts` (the mirror of `getFileDependentCounts`, driven
from `nodes` by path so the cost follows the files asked about rather than the
edge table) and `getFileNodes`.
Viewer. `ui/src/lib/entry-model.ts` folds the four lists into file groups —
pure, and `panel.rows` stays exactly the sections it draws. `EntryView` +
`EntrySection` render them with the caller rail's `.filegroup` / `.row` shapes
rather than a second visual language for the same idea. A row that names a
callable symbol carries a `Flow ›` chip; the other end is typed or picked with
`→ here` on another row. File and test rows carry none: `/api/flow` searches by
name, and a file has none the path finder can look up.
A project with fewer than three resolvable routes gets no Routes heading at all,
not an empty one. Typing into the search box now also returns matching entry
points under their own heading below the symbol matches, so a URL comes back
with its handler attached; rows already in the results are dropped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(config): add codegraph.json "deprioritize" for ranking-only path down-weighting
matchesNonProductionDir hardcodes example/sample/fixture/benchmark/demo,
so a peripheral tree only the project knows about — optional-skills/,
scripts/ — gets no de-prioritization. When helpers there carry generic
symbol names, an exact name match hands them a large bonus and they crowd
out the product code that answers the query (#982).
deprioritize is the RANKING counterpart to exclude: those paths stay
indexed and findable, they just stop outranking first-party code. It is
deliberately distinct from the corpus-frequency discount, which keys on a
name being common and is near-inert on #982's own repro where only two
symbols are named usage.
The -15 path penalty alone is not enough, and measuring showed why: on
that repro a usage() helper sits at 74.8 against 51.2 for the top product
symbol, so -15 lands at 59.8 and still leads. The path penalty is additive
and the name bonus it must counter is additive and larger. A de-prioritized
path is saying its symbol NAMES are not the answer, so the exact-name bonus
is damped to 0.25x there as well — damped, not zeroed, so the tree still
ranks when it genuinely is what you asked for.
Refs #982
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SKXAJMrVrdHS5Uco6ABtky
* fix(config): read deprioritize lazily and apply it in explore too
Review of the first cut found two real defects.
The matcher was built once in wireLayers(), which runs only from the
constructor and from reopenIfReplaced(). The MCP server keeps one
CodeGraph per project root alive for its whole lifetime, so editing
codegraph.json appeared to do nothing until the process restarted --
exclude and include do not behave that way. The predicate now reads
loadDeprioritizePatterns() per call (mtime-cached, one stat) and memoizes
the compiled matcher on the pattern array's identity. A regression test
writes the config after opening the project and fails on the old code.
Explore passed no matcher to scorePathRelevance at either of its two call
sites, so the setting only half-applied -- and #982's reproduction rows
B, C and D are all codegraph explore, which made this the surface the
issue actually reports on. Both sites now pass it.
Explore's hard early-continue filters and its non-production budget cap
are deliberately NOT joined: those REMOVE content, and deprioritize is a
ranking lever by definition. README narrowed accordingly -- it previously
claimed this extends the built-in list, which overstated it.
Also from review: scorePathRelevance takes a boolean rather than a
predicate (the caller already evaluated it, and it was being invoked
twice per result), the predicate body is exception-guarded so a bad path
can never take a search down, the misplaced const moved out from between
imports, two vacuous test assertions tightened, and tests added for the
single-penalty invariant, the deliberate isTestQuery asymmetry, and a
query that genuinely targets the de-prioritized tree.
Refs #982
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SKXAJMrVrdHS5Uco6ABtky
* fix(search): derive the deprioritize name-bonus damping instead of picking it (#982)
The 0.25 scale was a guess. On a 62k-node django index it measurably breaks
the "discount, don't erase" rule the lever is built on: exact-name queries for
symbols that live only in the de-prioritized tree (child, parent, method) fall
behind mere prefix matches (children, all_parents, method_decorator).
The prefix arm of nameMatchBonus tops out below 40, and a de-prioritized node
also takes the -15 path penalty, so 80 * SCALE - 15 > 40 is the bound that
keeps a damped exact match ahead of a prefix match at any corpus shape. 0.75
clears it; crowd-out removal is nearly identical to 0.5 (39 vs 40 of 88
peripheral top-10 slots cleared on django), so the deeper discount bought
almost nothing and cost the invariant.
Two tests pin the bound, including one that fails at the old 0.25.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Previously, naming a kebab-case file without its extension (e.g., `background-image-table` vs. `background-image-table.tsx`) in a `codegraph_explore` query would shred the name into fragments (`background`, `image`, `table`), admitting irrelevant sibling files and crowding out the intended target.
This change introduces a new resolution pass in `extractQueryPaths` specifically for extension-less kebab basenames. Queries now accurately identify and pin these files. Unresolved hyphenated prose (e.g., `cross-call`) is left in the query for FTS without being flagged as an unknown path. Resolution prioritizes explicit slashed/dotted paths and respects an ambiguity budget for common stems to prevent over-pinning.
Previously, `codegraph_explore` queries explicitly naming files by path (e.g., `src/routes/m/projects/[id]/runs/[runId]/+page.svelte`) were shredded. Bracketed path segments exploded into "named symbol" seeds, and FTS on fragments like `page` or `runs` admitted every sibling file, starving the user's intended target.
This change introduces:
- **Query path pinning:** File paths named in a query are now resolved against the index, "pinned," and stripped from the query. Pinned files are guaranteed inclusion, top ranking, and fair allocation. Unresolvable path-like spans are reported.
- **Segment vocabulary supplement:** Natural language query terms (e.g., "auto-scroll to bottom") can now reach camelCase identifiers (e.g., `pinFeedIfNearBottom`, `feedAtBottom`) by matching against their constituent segments.
- **Variable seeding:** `variable` and `constant` node kinds are now included in identifier seeding, improving recall for `$state`-style variables common in frameworks like Svelte.
Five hardening fixes to the #1136 MEDIUM (graph-derived) tier:
- #1141: updateNode() now writes the segment vocabulary like insertNode()
does — framework post-extract renames (NestJS route prefixing) left the
new name permanently unsearchable (the old rows orphaned, the backfill
gated on an EMPTY vocab, so even a full re-index re-created the drift).
- #1142: new CodeGraph.healSegmentVocabIfEmpty() — the hook opens the
graph without sync, so a database migrated from pre-vocab schema kept
the MEDIUM tier dormant until some unrelated sync ran. The hook heals
on first use (one SELECT when populated; lock-aware, defers to a
running sync) and records noop-vocab-empty when it can't.
- #1144: a name whose only nodes are file/import kind is skipped instead
of falling back to surfacing an import statement as a matched symbol;
import specifiers no longer enter the vocab at all (shared
isSegmentableKind gate across insertNode/updateNode/rebuild page query)
since they can never be surfaced and only inflate rarity statistics.
- #1145: plural variant folding is keyed on English plural spelling —
bare-s plurals no longer mint a bogus -es sibling (services→servic),
unambiguous sibilant-es plurals no longer mint a bogus -s sibling
(classes→classe), trailing -ss singulars no longer strip (class→clas);
genuinely ambiguous endings (caches/databases) still emit both keys.
- #1146: getSegmentCoOccurrence folds variants to their original word
inside the SQL (CASE mapping + COUNT(DISTINCT word)) so a plural pair
of ONE word can't tie with a genuine two-word match and crowd it past
the pre-fold ORDER BY/LIMIT; the JS re-check stays as the honesty layer.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The keyword gate (#1126) can never know a repo's domain nouns. This adds
the graph-derived tier the design discussion converged on: symbol names
are split into prose segments at index time (name_segment_vocab, riding
the insertNode write path), and the hook verifies a prompt's plain words
against them — "the state machine des commandes" → OrderStateMachine, in
any language whose technical nouns are Latin script.
Confidence now decides HOW MUCH to inject, not just whether:
- HIGH (keyword, or index-verified code token): full explore injection,
unchanged — the validated adoption lever.
- MEDIUM (segment matches only): a ~500-byte pointer naming the matching
symbols; the AGENT writes the explore query. Never runs explore, so a
fuzzy match can't inject 16KB of wrong-feature context.
- Silent otherwise, as before.
Precision is derived from the repo's own naming statistics plus measured
FP fixes: co-occurrence (≥2 words on one name) always qualifies; a single
word must be ≥5 chars, cluster across 2–25 names (singletons are prose
coincidence: "deploy to production" → matchesNonProductionDir), match a
multi-segment name, and not be an English function/filler word (the one
place a word list is honest: identifiers are English, so only English
prose collides). Every candidate is re-verified against nodes before
being surfaced — vocab rows are proposals, deletions leave orphans by
design, a full index rebuilds from scratch, and sync heals pre-upgrade
databases (batched + yielding; emptiness captured at sync ENTRY so the
sync's own writes can't mask the backfill).
Schema v7 migration is DDL-only (instant; none of the #1067 row-churn
hazards). Gate outcomes roll up as anonymous usage counters
(prompt-hook-gate-<outcome>, names only, never content) through the
existing telemetry pipeline — recall becomes measurable, and the counters
are the agreed kill-criterion data for ever revisiting a local classifier.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The per-word path fix (#745) brought the backend to parity but not above:
the project name still gave the lexically-matching stack a residual dir
match + an FTS class-name match, so a backend query that included the
project name still ranked the frontend at/above the backend.
Derive the project name from go.mod module / package.json name / repo dir,
and treat a query word matching it as non-discriminative: drop it from path
relevance and from codegraph_explore's PascalCase type-disambiguation bias
(reporter's suggestions #1/#2) — unless it's the only query word, so a bare
project-name search still scores.
Narrow by construction: the down-weighting fires ONLY when a query word
matches the derived project name (≥5 chars), so every query that doesn't
name the project is byte-identical. On the reporter's repro the backend
controllers now top a backend question that includes the project name;
queries without it, bare project-name queries, and normal symbol queries
are unchanged. Query-time only (no re-index).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A multi-word PascalCase query token — typically a project name a user
includes (`SuperBizAgent backend routes`) — splits into sub-tokens
(superbizagent / super / biz / agent) that ALL match the same path segment,
so path relevance summed +5 four times for one concept. In a mixed-stack
repo that ~doubled every score of the lexically-matching stack's file,
burying the stack the query was about.
Score path relevance per original query WORD instead: a word matches a path
level if any of its sub-tokens do, and counts once — while still splitting
the word (via extractSearchTerms on the original case) so it matches across
naming conventions (`getUserName` → `get_user_name`). Distinct words each
still contribute.
Partial fix: this removes the dominant path over-counting (backend rises
from absent-in-top-6 to parity on the reporter's repro). The residual lexical
edge from the project name in the FTS class-name match + dir match is a deeper
down-weighting change, tracked separately. No re-index needed (query-time).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Completes the explore-overhaul arc: `codegraph_explore` becomes the single primary tool an agent reaches for, and its coverage + output shape are tuned so flow/architecture questions resolve with near-zero Read/Grep.
### What changed
- **explore is the sole primary tool** — removed `codegraph_context` (the fuzzy-input Read-trigger) and `codegraph_trace` (under-picked by agents); explore already surfaces the call flow among the symbols you name. A plain natural-language question now works as the query.
- **Store/handler coverage** — functions defined inside object literals (Zustand `create((set, get) => ({ … }))`, Redux/Pinia/MobX, exported handler/route maps) are indexed as real symbols, including calls through `useStore.getState().fn()` and destructured `const { fn } = useStore.getState()`. A general AST rule, not a per-lib hack.
- **Overload disambiguation** — explore leads with the *right* definition when a method name is overloaded across types (a PascalCase type token in the query biases to that type's own def); `codegraph_node` returns *every* overload's body in one call, with an optional `file`/`line` selector to pin one.
- **Method-atomic render** — explore never returns half a method; at the size budget it drops whole methods/files (and lists what it dropped) instead of truncating a body mid-method.
- **Native-read-shaped output** — per-call output is capped to ~24K with a 25K hard ceiling and concentrated into ~150–250-line flow windows, mirroring how the agent natively reads; repo size scales the *call* budget, not the per-call size (a larger response just gets externalized to a file the host Reads back).
- **Blast radius** folded into explore (dependents + covering tests, locations only).
### Benchmark (refreshed on this build)
Re-validated the 7-repo A/B on 2026-06-02 (Opus 4.8, effort=high, median of 4). WITH arm re-measured on this build, WITHOUT reused:
**~16% cheaper · 47% fewer tokens · 22% faster · 58% fewer tool calls** — 0 file reads on 6 of 7 repos (Gin ~1).
The arc trades larger, cache-heavy explore responses for guaranteed near-zero reads, so cost/token margins soften vs the prior build (Excalidraw and Tokio land at cost break-even) while time and tool-calls stay clear wins everywhere — consistent with the project's stated optimization target (latency + tool-calls, not token cost).
### Validation
- Full suite green: **1112 passed, 2 skipped**.
- 28/28 plain WITH runs across the 7 README repos completed clean; reads median 0 on 6/7.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* feat(search): field-qualified queries (kind:/lang:/path:/name:) + fuzzy typo fallback
Two UX improvements that turn a free-text search into something a
real user can drive precisely.
1) Field-qualified queries.
A new query parser (src/search/query-parser.ts) splits the raw query
into structured filters and a free-text remainder:
kind:function name:auth path:src/api authenticate
becomes
{ kinds: ['function'], nameFilters: ['auth'],
pathFilters: ['src/api'], text: 'authenticate' }
Filters compose with the SearchOptions arg (intersection). Unknown
prefixes pass through as plain text so `query "TODO:"` keeps working.
Quoted values (`path:"my dir"`) handle whitespace. When the user
specifies only filters with no text, the search uses a filter-only
candidate scan instead of bailing out.
Recognised today:
kind: any NodeKind value
lang: any Language value (alias: language:)
path: case-insensitive substring of file_path
name: case-insensitive substring of node.name
2) Fuzzy fallback.
When BOTH FTS and LIKE return nothing AND the text is at least 3
chars, the resolver scans the distinct-name set with a bounded
Damerau-Levenshtein-style edit distance (≤2 for ≥5 chars, ≤1 for
4-char queries, off for shorter). Bounded edit-distance early-exits
once the row min exceeds maxDist, so this stays O(distinct-names *
avg-name-length) with a very low constant.
Verified live against ollama/ollama@v0.22.0:
query "kind:function auth" → only function-kind hits
query "lang:go path:server route" → Go files under server/
query "getUssr" (typo) → finds getUser, SetUser
query "confg" (typo) → finds Config
Full test suite: 380 passed.
* fix(search): address reviewer findings — tokenizer mid-token quotes, fuzzy fan-out cap, larger filter-only over-fetch, unit tests
Five fixes from independent review:
- parseQuery tokenizer: quotes that appear MID-token (path:"my dir/
file") were not being recognised — only quotes at the start of a
token were treated as quoted spans. The fixture path:"my dir"
parsed as ['path:"my', 'dir"'] instead of ['path:"my dir"'].
Tokeniser is now a single state machine that scans into a token
until whitespace OR a quote, and recognises quotes anywhere within
the token (skips to the matching close quote).
- searchNodesFuzzy: cap the per-name follow-up SQL queries at
Math.max(limit*2, 50) AFTER edit-distance filtering. Without
this, a project with many similar names (getUser1, getUser2...)
could fan out far beyond limit queries before the inner-loop
break kicks in.
- searchAllByFilters (filter-only no-text path): bumped over-fetch
multiplier from 2× to 5× so a selective post-filter (e.g.
path:src/very/specific/file.ts) doesn't return fewer than limit
results despite the DB having matches.
- 23 new unit tests in __tests__/search-query-parser.test.ts:
parseQuery covers known-field filter, lang/language alias,
multiple kind: ORs, quoted spans (incl. mid-token), URL
passthrough, empty-value passthrough, unknown prefix passthrough,
unknown value passthrough, all-filters-no-text, empty input,
20k-char input. boundedEditDistance covers identity, single
insertion/deletion/substitution, length-difference shortcut,
empty inputs, case-sensitivity, early-exit correctness.
Full test suite: 853 passed (up from 830).
* refactor(search): derive parser kind/lang sets from types.ts as const
Convert NodeKind and Language to runtime-iterable as const arrays
(NODE_KINDS, LANGUAGES) so the query parser imports the canonical
list instead of duplicating it. Also fix the path: JSDoc to say
substring (matches the .includes() impl).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses single files monopolizing the node budget when BFS traverses from multiple entry points in the same class. Caps each file to ~20% of maxNodes and limits test/sample/integration files to 15% to ensure cross-file diversity in context results. Expands isTestFile detection to include integration, sample, example, and other non-production directories.
Addresses path relevance scoring inflation where stem variants created many near-duplicate terms that all matched the same path segments. Adds stems option to extractSearchTerms (default true) and disables stems for path scoring while keeping them for FTS matching. Also improves name match bonus scoring with length-based prefix matching and higher exact match scores.
Uses max FTS score as baseline for exact name matches to ensure nameMatchBonus differentiation during rescoring, increases exact match limit from 5 to 20 candidates, and adds common conversational terms to stop words to reduce query noise.
Expands symbol lookup with morphological variants (e.g., "caching"→"cache", "eviction"→"evict") to find related class definitions that FTS prefix matching would otherwise miss. Includes comprehensive stemming rules for common English suffixes (-ing, -tion, -ed, -er, etc.) and integrates stem expansion into definition prefix search for improved symbol discovery.
Adds nameMatchBonus scoring to prioritize results where node names exactly or partially match query terms. Implements dedicated field extraction for Java/C# to properly categorize class fields vs variables. Optimizes BM25 search with column weights favoring name matches and increased result fetching before post-processing. Refines stop words list to preserve common programming terms like "get", "find", "list".
Improves search accuracy by boosting results when multiple query symbols appear in the same file, addressing cases where common names like "run" return too many results. Adds Go method receiver type extraction to qualified names for better searchability (e.g., "scrapeLoop.run"). Optimizes database queries with two-pass approach to handle distinctive vs common symbol names efficiently.
Eliminate cross-language false positives in name resolution and deprioritize
test files in context building. Benchmarked on a Python+Rust codebase where
37% of edges were false positives from Python built-in methods resolving to
Rust functions (e.g., list.extend → Rust extend).
Resolution fixes (index-time):
- Filter Python built-in type method calls (list.extend, dict.update, etc.)
- Filter bare Python built-in method names (append, extend, pop, keys, etc.)
- Add language boundary checks to matchMethodCall strategies 1, 2, and 3
- Penalize cross-language matches: -80 points in findBestMatch (was 0)
- Reduce confidence for single cross-language exact matches (0.5 vs 0.9)
- Prefer same-language candidates in matchFuzzy
Context relevance fixes (query-time):
- Add isTestFile() utility detecting test files across Python/JS/TS/Go/Rust/Java
- Deprioritize test files in scorePathRelevance (-15 penalty)
- Reduce test file scores to 30% in context builder result merging
- Both skip deprioritization when query mentions "test" or "spec"
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
extractSearchTerms now splits camelCase, PascalCase, snake_case, and
dot.notation into individual tokens (e.g. "getUserName" → ["user", "name"]).
Stop words expanded with code-specific noise words (code, file, function,
method, class, type, etc.) to improve search precision.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Both functions have zero callers — dead code on arrival. Remove them
and their tests (9 tests) to keep the module focused on what's
actually used: search term extraction, path relevance scoring, and
kind bonuses.
