* feat(config): add codegraph.json "deprioritize" for ranking-only path down-weighting
matchesNonProductionDir hardcodes example/sample/fixture/benchmark/demo,
so a peripheral tree only the project knows about — optional-skills/,
scripts/ — gets no de-prioritization. When helpers there carry generic
symbol names, an exact name match hands them a large bonus and they crowd
out the product code that answers the query (#982).
deprioritize is the RANKING counterpart to exclude: those paths stay
indexed and findable, they just stop outranking first-party code. It is
deliberately distinct from the corpus-frequency discount, which keys on a
name being common and is near-inert on #982's own repro where only two
symbols are named usage.
The -15 path penalty alone is not enough, and measuring showed why: on
that repro a usage() helper sits at 74.8 against 51.2 for the top product
symbol, so -15 lands at 59.8 and still leads. The path penalty is additive
and the name bonus it must counter is additive and larger. A de-prioritized
path is saying its symbol NAMES are not the answer, so the exact-name bonus
is damped to 0.25x there as well — damped, not zeroed, so the tree still
ranks when it genuinely is what you asked for.
Refs #982
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SKXAJMrVrdHS5Uco6ABtky
* fix(config): read deprioritize lazily and apply it in explore too
Review of the first cut found two real defects.
The matcher was built once in wireLayers(), which runs only from the
constructor and from reopenIfReplaced(). The MCP server keeps one
CodeGraph per project root alive for its whole lifetime, so editing
codegraph.json appeared to do nothing until the process restarted --
exclude and include do not behave that way. The predicate now reads
loadDeprioritizePatterns() per call (mtime-cached, one stat) and memoizes
the compiled matcher on the pattern array's identity. A regression test
writes the config after opening the project and fails on the old code.
Explore passed no matcher to scorePathRelevance at either of its two call
sites, so the setting only half-applied -- and #982's reproduction rows
B, C and D are all codegraph explore, which made this the surface the
issue actually reports on. Both sites now pass it.
Explore's hard early-continue filters and its non-production budget cap
are deliberately NOT joined: those REMOVE content, and deprioritize is a
ranking lever by definition. README narrowed accordingly -- it previously
claimed this extends the built-in list, which overstated it.
Also from review: scorePathRelevance takes a boolean rather than a
predicate (the caller already evaluated it, and it was being invoked
twice per result), the predicate body is exception-guarded so a bad path
can never take a search down, the misplaced const moved out from between
imports, two vacuous test assertions tightened, and tests added for the
single-penalty invariant, the deliberate isTestQuery asymmetry, and a
query that genuinely targets the de-prioritized tree.
Refs #982
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SKXAJMrVrdHS5Uco6ABtky
* fix(search): derive the deprioritize name-bonus damping instead of picking it (#982)
The 0.25 scale was a guess. On a 62k-node django index it measurably breaks
the "discount, don't erase" rule the lever is built on: exact-name queries for
symbols that live only in the de-prioritized tree (child, parent, method) fall
behind mere prefix matches (children, all_parents, method_decorator).
The prefix arm of nameMatchBonus tops out below 40, and a de-prioritized node
also takes the -15 path penalty, so 80 * SCALE - 15 > 40 is the bound that
keeps a damped exact match ahead of a prefix match at any corpus shape. 0.75
clears it; crowd-out removal is nearly identical to 0.5 (39 vs 40 of 88
peripheral top-10 slots cleared on django), so the deeper discount bought
almost nothing and cost the invariant.
Two tests pin the bound, including one that fails at the old 0.25.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A Git super-repo whose `.gitignore` excludes its child repositories indexed
~nothing at the parent: CodeGraph respects `.gitignore` by default (#970,
#1065), so the excluded children were skipped and `codegraph init` printed
"Done" with 0 nodes — even though `init` inside each child worked fine. The
empty index was silent and unexplained.
`init`/`index` now detect the gitignored child repos they skipped when an
index comes up empty of symbols, name them, and — in an interactive terminal
— offer to index them (writing an `includeIgnored` entry to codegraph.json and
re-indexing on the spot); non-interactive runs print the exact codegraph.json
snippet to add. Gated on nodesCreated === 0, so a project that deliberately
keeps gitignored reference clones out of a working index is never nagged.
- extraction: findUnindexedIgnoredRepos — the inverse of discoverEmbeddedRepoRoots
(bounded, skips default-ignored dirs, respects existing includeIgnored)
- project-config: addIncludeIgnoredPatterns — create/merge codegraph.json,
idempotent, refuses to clobber malformed JSON
- cli: wire the detect-name-offer flow into both `init` and `index`
- tests: +13 covering detection, config writing, and the no-nag gate
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an `include` list to the root `codegraph.json` that forces gitignored first-party source (second-VCS / SVN / Perforce dual-tracked repos) into the index — discovered directly off disk on the full index, incremental sync, and file-watching, on both git and non-git projects. Gitignore-style patterns, root-relative; explicit `exclude` still wins and built-in skips (node_modules, dist, .git) are never re-included. Complements `exclude` and `includeIgnored`.
Closes#1163.
Thanks @luoyxy for the contribution.
Three fixes for a repo that commits a large JS/TS theme/SDK (Metronic under
static/, ~1,600 tracked files):
1. A SECOND "Resolving refs" quadratic that #915 didn't cover. #915 capped
import-name collisions; this caps method-name collisions (init/update/render
re-declared on every widget), which flow through matchMethodCall Strategy 3
and findBestMatch instead. New AMBIGUOUS_NAME_CEILING (default 500, env
CODEGRAPH_AMBIGUOUS_NAME_CEILING): above it the fuzzy strategies decline
rather than score K candidates — no proximity score can pick the one true
target among thousands anyway. Resolving drops from O(K^2) to linear in refs
(e.g. 900-file synthetic: 28.7s -> 3.4s), edge counts unchanged, and the cap
never fires on normal repos (max real method-collision ~40).
2. A new `exclude` array in codegraph.json keeps git-TRACKED paths out of the
index, which .gitignore can't do (enumeration is `git ls-files`). Mirrors the
existing includeIgnored plumbing across the git, sync, and non-git-walk
paths.
3. `index`/`init` now install the #850 liveness + #277 ppid watchdogs (which
were serve-only), so a wedged or orphaned indexer self-terminates instead of
pinning a core. The --liftoff-only relaunch's spawnSync can't forward
signals, so killing the parent shim used to orphan the worker.
Tests: ubiquitous-name ceiling, exclude (incl. tracked-file exclusion on git +
non-git), orphan self-termination (POSIX), and ppid-parser units. Shared the
ppid parsers out of mcp/index.ts into mcp/ppid-watchdog.ts.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#514 (v1.0.0) began walking into gitignored directories to discover and
index the git repos nested inside them. That broke users who rely on
.gitignore to exclude a directory: a gitignored folder of cloned
reference repos blew graphs up (one report went 10k to 500k edges, #976)
and stalled indexing on multi-gigabyte trees of clones (#970).
Respect .gitignore by default again. Discovering embedded repos inside a
gitignored directory is now opt-in via codegraph.json:
{ "includeIgnored": ["packages/", "services/"] }
The single choke point findIgnoredEmbeddedRepos now returns nothing
unless a gitignored dir matches the project's includeIgnored patterns,
and the matcher is threaded from the scan root through the full-index,
incremental-sync, and watcher-scope paths. Downstream ScopeIgnore and the
watcher are unchanged: they key off the discovered embedded roots, so
gating discovery fixes the indexer, sync, and watcher together. Untracked
embedded repos (#193) stay indexed by default.
This restores the super-repo-of-clones behavior (#622, #699) for the
people who want it, while making the default match what every other tool
(and CodeGraph's own git ls-files foundation) does: .gitignore excludes.
project-config.ts now parses codegraph.json once (loadParsedConfig) and
exposes loadIncludeIgnoredPatterns alongside the existing extension map.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The extension → language table was hardcoded, so a codebase using a
non-standard extension for a supported language (e.g. `.dota_lua` for Lua)
had those files silently skipped — no way to opt them in short of patching
the source.
Add an opt-in, project-scoped `codegraph.json` at the repo root:
{ "extensions": { ".dota_lua": "lua", ".tpl": "php" } }
Mappings merge on top of the built-in defaults and take precedence (so a
built-in can be re-pointed, e.g. `.h` → `cpp`). Absent or malformed config
is the zero-config default — byte-identical to prior behavior; an invalid
target language or unparseable file is warned-and-skipped, never fatal.
Implementation:
- New `src/project-config.ts` — `loadExtensionOverrides(rootDir)`, validated
against `isLanguageSupported`, mtime-cached per root.
- `detectLanguage` / `isSourceFile` gain an optional `overrides` arg
(omitting it is the existing behavior).
- Overrides threaded per-operation through every extraction call site
(scan/walk gates, git change-detection, grammar selection, extraction,
the file watcher), resolved from the project root — no process-global
state, so the multi-project daemon stays isolated. The parse worker
receives the resolved language in its message.
Tests: 13 new cases (unit, loader validation/normalization/caching, and a
full-index integration proving a custom-extension file is extracted while
the zero-config path indexes nothing). Worker path smoke-tested via the
built CLI.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
