A .cbmignore negation pattern (e.g. "!obj/") whose rule is the last
match for a directory now un-skips built-in skip-list dirs, so projects
can index sources that live under names like obj/, target/, or docs/
(issue #500). Distills the design and test intents of PR #543 onto the
current 3-state cbm_gitignore_match_result API: instead of a second
inverted-polarity matcher, should_skip_directory consults the existing
last-match-wins result (<0 = negated) inside the built-in branch and
falls through so .gitignore/global/local rules still apply.
Non-negatable safety core: .git, node_modules, and the worktree-internal
dirs (.worktrees, .claude-worktrees) can never be un-skipped — a
repo-committed .cbmignore must not defeat OOM/safety skips (#489, #802).
Un-skipped dirs are walked normally and therefore no longer appear in
the excluded-subtree list (#411 coherence).
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to #683 (@Naam). The worktree info/exclude fix read the `.git` gitlink
file and its `commondir` with raw fopen(); on Windows raw fopen uses the ANSI
code page and fails to open non-ASCII (e.g. CJK) repo paths, so a worktree
indexed under such a path would silently not honor info/exclude /
core.excludesFile. Switch both reads to cbm_fopen() (the UTF-8 -> _wfopen
wrapper), matching the rest of discover.c (e.g. read_core_excludes_file at 248).
No behavior change on POSIX (raw fopen already opens UTF-8 byte paths there);
this is a Windows-only correctness/consistency fix, verified by the full test
suite staying green and cross-platform CI. discover.c now has no raw fopen.
Refs #682, #683
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Standard fopen on Windows uses the ANSI/system codepage and silently
fails to open files whose absolute path contains non-ASCII characters
(accented, CJK, etc.). This causes the pipeline to skip parsing those
files entirely — File nodes are created by directory traversal (which
already uses wide Win32 APIs) but tree-sitter never receives the source,
so no Function/Class/Method nodes are extracted.
Add cbm_fopen(path, mode) to compat_fs:
- Windows: converts the UTF-8 path to UTF-16LE via cbm_utf8_to_wide
(existing helper in win_utf8.h) and calls _wfopen.
- POSIX: delegates to fopen (no-op change, paths are already UTF-8).
Replace every fopen call that operates on user project file paths:
- src/pipeline/pass_parallel.c (primary indexing path)
- src/pipeline/pass_definitions.c
- src/pipeline/pass_calls.c
- src/pipeline/pass_semantic.c
- src/pipeline/pass_usages.c
- src/pipeline/pass_lsp_cross.c
- src/pipeline/pass_pkgmap.c
- src/pipeline/pass_k8s.c
- src/pipeline/pass_envscan.c
- src/pipeline/path_alias.c (tsconfig detection)
- src/pipeline/artifact.c
- src/discover/language.c (content-sniff for language detection)
- src/discover/gitignore.c
- src/discover/discover.c
- src/discover/userconfig.c
- src/mcp/mcp.c (read_file_lines, adr.md reader)
Fixes#700
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
In a linked worktree, <repo>/.git is a regular file ("gitdir: <path>")
rather than a directory. cbm_discover_ex gated all per-clone ignore
sources on .git being a directory, so inside a worktree it skipped
.git/info/exclude and the global core.excludesfile entirely and walked
into directories the user had excluded there (e.g. build/, twister-out/).
Committed .gitignore still applied via the issue #510 path, which is why
the gap was easy to miss.
Resolve the git common dir up front: when .git is a gitlink file, parse
'gitdir:', then follow <gitdir>/commondir to the shared dir where git
keeps info/exclude and config (falling back to the gitdir when no
commondir file is present). info/exclude and config are then read from
that common dir, so a worktree behaves like an ordinary checkout.
Adds discover_worktree_info_exclude (fails before this change) and
discover_worktree_committed_gitignore.
Signed-off-by: Nahim El Atmani <2959826+Naam@users.noreply.github.com>
QUALITY_ANALYSIS gap #1: .claude-worktrees was indexed (nearly tripling this
repo's indexed surface + polluting queries with stale duplicate symbols).
Discovery skipped .claude and .worktrees but not .claude-worktrees. Add it to
ALWAYS_SKIP_DIRS. Flips the discovery-hygiene invariant green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_gitignore_merge copied src patterns into dst one strdup at a time and
returned void on failure, leaving dst with a partial set of patterns and no
signal to the caller. If the dropped pattern was the critical exclude (the
.git/info/exclude entry that prevents an OOM walk), the bug could silently
recur with some patterns applied and others dropped.
Make the merge atomic: on any allocation failure roll back the partial copies
so dst is left exactly as it was, and return bool so callers can tell. The
.gitignore caller now degrades gracefully (exclude patterns skipped, as if the
file were absent) instead of ending up with a half-merged matcher.
Adds a reproduce-first test that injects a strdup failure mid-merge via a test
seam and asserts dst is unchanged; it is red without the rollback.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Drop the repo-local .git/config pass in resolve_global_excludes_path() so
an untrusted repo cannot point core.excludesFile at an arbitrary file (e.g.
~/.ssh/id_rsa) and have cbm open and read it merely by being indexed. git is
safe here because it only trusts a repo's .git/config after a deliberate
clone/init; the indexer has no such trust step. Global excludes now resolve
only from the user-owned ~/.gitconfig and $XDG_CONFIG_HOME/git/config. A
repo's own ignores remain handled by .gitignore / .cbmignore.
Add a guard test asserting a repo-local .git/config whose excludesFile points
outside the repo is ignored, not read.
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(discover): honor .git/info/exclude during file discovery
index_repository only loaded .gitignore when building the exclusion
matcher. .git/info/exclude — the per-clone exclude file that Git treats
as authoritative — was never read, so paths excluded only there were
walked in full. On repos with Sandcastle worktrees this caused the
indexer to traverse 661K files / 170 GB and OOM (issue #489).
Add cbm_gitignore_merge() to append patterns from one matcher into
another. In cbm_discover_ex, after loading .gitignore, also load
.git/info/exclude and merge its patterns in. If only the exclude file
exists, use it directly. No downstream call paths change.
Add three unit tests for cbm_gitignore_merge and two integration tests
that reproduce the issue scenario (exclude-only and exclude-stacked-
with-gitignore).
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
* style(discover): fix clang-format alignment in cbm_gitignore_merge
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
---------
Signed-off-by: ShauryaaSharma <shauryasofficial27@gmail.com>
deploy/ and deployed/ build artifact directories were in FAST_SKIP_DIRS,
which only applies in fast/moderate indexing modes. In the default full
mode they were indexed alongside source files, creating duplicate
Class/Method rows with stale data. Any MATCH (c:Class {name:'Foo'})
without a file_path filter returned multiple rows — one from source,
one from the build copy.
Move deploy and deployed to ALWAYS_SKIP_DIRS so they are excluded in all
indexing modes.
Regression test: discover_deploy_excluded_full_mode in test_discover.c.
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
index_repository silently dropped entire subtrees (ALWAYS_SKIP dirs in all modes; FAST_SKIP dirs like tools/scripts/bin in moderate/fast) with no record — users couldn't tell what wasn't indexed. The discover walk now records skipped directory rel-paths; cbm_discover_ex returns them; the pipeline holds + exposes them; and the index response carries a compact summary: "excluded":{"dirs":[up to 25],"count":N,"truncated":bool}. Behavior-neutral for indexing. Adds a reproduction in test_integration.c.
Helm chart helpers (.tpl) and Go templates produced no semantic graph — just a
flat key flood. Wire the existing gotemplate grammar for real structure:
- .tpl files map to the gotemplate language (Helm _helpers.tpl).
- {{ define "chart.fullname" }} becomes a Function node named after the
template (the name is a string literal, not an identifier, so it's resolved
in a dedicated walker branch).
- {{ include "x" . }} and {{ template "x" . }} resolve to the referenced
template name instead of the bare `include` builtin, so they form CALLS
edges to the define'd Function. template_action is added to the call set.
Lets you trace which charts include which named templates. Tests cover define
-> Function and include/template -> CALLS, plus the .tpl mapping.
Refs #338
Vendor the MIT-licensed cfmleditor/tree-sitter-cfml grammars and wire both
CFML dialects through the pipeline:
- .cfc components -> cfscript grammar (CBM_LANG_CFSCRIPT): a JS-like grammar, so
function/method declarations, calls and imports extract with the shared JS/TS
arrays. Components' functions become Function nodes.
- .cfm templates -> cfml tag grammar (CBM_LANG_CFML): tag-based (HTML-derived).
Embedded <cfscript> functions extract via function_declaration; tag-based
<cffunction name="..."> nodes are handled by a dedicated walker branch that
reads the name attribute (cf_function_tag has no name field). Shared scanner
headers are vendored at vendored/common/ so the grammar's ../../common/
includes resolve without editing vendored sources.
Adds an ASCII case-insensitive compare for CFML's case-insensitive attributes.
Tests cover both dialects and the extension mapping. LICENSEs vendored.
Closes#38
Vendor the MIT-licensed tree-sitter-qmljs grammar (a TypeScript superset with
declarative ui_* nodes) and wire .qml files through the full pipeline: language
enum, extension + display-name mapping, grammar shim, and a language spec that
reuses the JS/TS function/call/branch arrays and adds QML class/field/import
types (ui_inline_component, ui_property, ui_signal, ui_import).
QML files now produce Function nodes (incl. JS functions inside QML objects),
class/interface/enum nodes, imports and CALLS edges, so call chains through a
Qt project's .qml files can be traced. LICENSE vendored alongside the grammar.
Closes#42
Laravel Blade templates (*.blade.php) fell through to the single-extension
lookup and were mis-classified as PHP, so the PHP extractor tried to parse
Blade syntax. Add a built-in compound-extension table (checked before the
single-extension fallback and alongside user config) mapping .blade.php to
CBM_LANG_BLADE, so Blade files are discovered and parsed with the Blade
grammar. Combined with the now-non-blocking discovery gate (c29e6d5), this
resolves the "agent stuck on Blade files" report. Plain .php is unaffected.
.mjs (ES modules) and .cjs (CommonJS) were absent from the extension
table, so those files were never assigned a language — they weren't
indexed (no graph nodes) and search_code returned 0 for any pattern in
them. Add .mjs/.cjs → JavaScript and .mts/.cts → TypeScript.
Verified end-to-end: a function in a .mjs file is now extracted and found
by search_code. Closes#197.
- cbm_arena_alloc returns NULL on a NULL arena (both arena.c copies)
instead of dereferencing it — defense-in-depth against the NULL-arena
type-allocation path that could crash the LSP type layer.
- discover: add "vendored" to the always-skip directory list.
Distilled from #374, taking the parts not already on main. The PR's
ts_lsp tuple-arena allocation, the C/C++ template formal-count clamp,
and the type_rep unbound-param preservation it also carried all landed
independently in v0.7.0 / via #322 / #360, so only these two
defensive improvements remain. Relates to #390.
Replace ANSI Windows APIs (FindFirstFileA, CreateFileA, etc.) with
wide-char (W) counterparts. Add UTF-8/UTF-16 conversion helpers.
Fix stat() calls in discover pipeline to use _wstat64.
Fixes#357
The per-file and cross-file LSP type-aware call/usage resolution now runs in every index mode (full/moderate/fast) instead of being gated behind a separate CBM_MODE_ADVANCED. Remove the mode from both index-mode enums, the g_lsp_enabled toggle and its save/restore dance, and the per-mode gating in discover, pipeline and the incremental path.
Now that the LSP walkers are O(n), the callee-attribution accuracy (e.g. ~91k call overrides on roslyn, ~60k on kubernetes) is worth the latency in every mode. Fast mode still filters files and skips similarity/semantic edges, so it stays faster than full while now also benefiting from LSP resolution.
A 'mode':'advanced' request falls back to the full default — identical behaviour to the old advanced mode, so existing callers are unaffected. This also fixes the test_incremental.c failures, which were asserting LSP-resolved results that the gated incremental path never produced.
New CBM_MODE_ADVANCED=3. Both per-file LSP (type-aware resolution
during extract) and the cross-file LSP pass (during resolve) are now
OPT-IN: FULL and below run tree-sitter extraction + registry textual
resolution only; ADVANCED adds the full LSP layer.
Rationale: profiling showed LSP changes the kubernetes graph by only
~0.01% of edges (it mostly re-confirms textual resolution) while
costing significant memory + time. Making it opt-in keeps the default
FULL path lean (3.0 GB vs 5.6 GB peak RSS on K8s) without meaningfully
worse output, and lets users pay for LSP precision only when wanted.
Mechanism:
- cbm.c: g_lsp_enabled atomic toggle wraps the entire per-file LSP
block (Go/C/C++/PHP/Python/TS/JS/C#) + the C/C++ preprocessor-pass
LSP call. Default true so ad-hoc extraction / tests keep historical
behaviour. cbm_set_lsp_enabled / cbm_get_lsp_enabled in cbm.h.
- pipeline.c: sets the toggle (save→set→restore so it never leaks to
other callers) from p->mode before extraction; gates the cross-LSP
all_defs collection + registry build on advanced; gates the
sequential lsp_cross dispatch. Fixed the moderate_only comparison
(== FAST, not > MODERATE) so ADVANCED is not treated as lighter.
- pipeline_incremental.c: same toggle save/set/restore.
- discover.h/.c: added CBM_MODE_ADVANCED to discover's enum + all 4
filter comparisons treat ADVANCED like FULL (full discovery, no
fast-filtering). Without this, ADVANCED (numerically 3) was
mis-detected as fast-mode and dropped ~half the files.
- mcp.c: parse mode "advanced" + tool schema enum/description.
Dual-mode K8s bench: FULL 46.7s (lsp_overrides=0, 3.0 GB) vs
ADVANCED 48.8s (lsp_overrides=80,818, 5.6 GB). Graphs within 0.01%.
All 3,615 tests pass.
The .m extension was missing from EXT_TABLE, causing detect_file_language
to return CBM_LANG_COUNT (Unknown) before the content-based disambiguation
logic (cbm_disambiguate_m) could ever run. This made .m files invisible
to the indexer.
Add {".m", CBM_LANG_MATLAB} to EXT_TABLE so the initial lookup succeeds,
allowing the subsequent cbm_disambiguate_m call to correctly determine
whether the file is MATLAB, Objective-C, or Magma.
Fixes#296
Adds Pine Script (TradingView's indicator/strategy DSL) as a first-class
language using the project's existing language-add pipeline:
- scripts/new-languages.json: register pine (display "PineScript",
ts_func tree_sitter_pine, repo kvarenzn/tree-sitter-pine, extension
.pine, has_scanner=true, module_root source_file)
- internal/cbm/vendored/grammars/pine/: vendored from
https://github.com/kvarenzn/tree-sitter-pine via scripts/vendor-grammar.sh
— parser.c, scanner.c, tree_sitter/ headers. License is ISC (declared
in upstream package.json; written into vendored LICENSE for clarity
since upstream has no LICENSE file)
- internal/cbm/grammar_pine.c: 4-line wrapper generated by
scripts/generate-lang-code.py wrappers
- internal/cbm/cbm.h: CBM_LANG_PINE before CBM_LANG_COUNT
- internal/cbm/lang_specs.c: 7 node-type arrays (func / class /
module / call / var / branch / assign) cross-checked against
kvarenzn's grammar.js, plus the spec table entry
- src/discover/language.c: .pine -> CBM_LANG_PINE extension dispatch
and "PineScript" name
scripts/audit-grammar-security.sh on the vendored grammar: no
dangerous includes (only string/stdint/stdbool/stdlib), no system /
popen / exec / socket / mmap / dlopen calls, no constructor/destructor
attributes, no inline asm. The 148-line scanner.c only handles
INDENT/DEDENT/NEWLINE tokens via the standard tree-sitter lexer API.
Distilled from #273 rather than vendored verbatim because the
original PR's parser.c was a plain 46k-line drop without provenance,
no LICENSE attribution, and bundled an unrelated Makefile change
(scripts/embed-frontend.sh -> sh scripts/embed-frontend.sh) that
didn't belong to a Pine Script feature. The node-type choices in
the lang spec match the original PR's selections, all of which were
verified to be real grammar.js rules.
Full suite: 2840 passed, 0 failed under ASan + UBSan.
Closes#273.
Co-authored-by: vinay-veerappa <vinay.veerappa@gmail.com>
Add automated extension-to-language mapping tests for all new grammars
that have file extensions. Removes conflicting extensions:
- .blade.php: compound extension, handled by userconfig system
- .env*: filenames, not extensions (no auto-discovery for dotfiles)
- .sv/.svh: already mapped to Verilog, SystemVerilog uses same grammar
Nested .gitignore support (fixes#178):
- Load per-subdirectory .gitignore during walk, match paths relative to
the gitignore's directory via local_rel_path()
- Root and nested gitignores stack independently
- Owned gitignores collected and freed at walk end (avoids use-after-free
from borrowed pointers on the iterative stack)
Security:
- Run security-strings/install/network + ClamAV + Windows Defender on
ALL binary variants (standard + UI), not just standard
- Whitelist UI bundle URLs (React, Three.js, Google Fonts, Tailwind, W3C)
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
- platform.c: move nlen into if(env) block (variableScope)
- userconfig.c: move home into else block (variableScope)
- pipeline.c: remove dead rc!=0 check after httplinks removal
- cli.c: rmdir_recursive propagates cbm_rmdir errors, remove redundant local
- clang-tidy: whitelist cosmetic-only checks, enable all safety checks
- Thread-safe getenv via environ iteration (concurrency-mt-unsafe fix)
- Multiple files: add constants.h includes for named constants
Lower thresholds to best-practice values:
cognitive-complexity: 250 -> 25 (industry default)
statements: 400 -> 200
lines: 800 -> 400
Add NOLINTNEXTLINE to 168 functions that currently exceed these thresholds.
Each suppression will be removed as the function is split in subsequent
commits. Zero net behavioral changes.
Also: extract execute_with_clause from cypher execute_single.
- clang-format: fix alignment in userconfig.c and userconfig.h
- Magic number 1280 → PATH_BUF_SZ named constant
- Remove dead g-- before break in dedup loop (cppcheck)
- Add "Custom File Extensions" section to README documenting
.codebase-memory.json and global config
- Add CBM_LANG_KUSTOMIZE and CBM_LANG_K8S to CBMLanguage enum (before CBM_LANG_COUNT)
- Add kustomization.yaml/yml to FILENAME_TABLE mapped to CBM_LANG_KUSTOMIZE
- Add Kustomize and Kubernetes entries to LANG_NAMES
- Implement cbm_is_kustomize_file() with to_lower+strcmp pattern
- Implement cbm_is_k8s_manifest() scanning first 4KB for apiVersion: via ci_strstr()
- Declare both helpers in pipeline_internal.h Infrascan helpers section
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- pipeline.c: inline userconfig cleanup on early-exit paths (discover
failure and cancel check) so userconfig is always freed, fixing the
resource leak when pipeline exits before reaching the cleanup label
- userconfig.c: free partially-built entries array on global config
parse failure to fix alloc-failure memory leak
- language.c: probe user config for compound extensions (e.g.
".blade.php") by scanning from the first dot to the last, falling back
to built-in single-extension lookup — preserves built-in behavior
while correctly resolving user-defined compound extensions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Read extra_extensions from ~/.config/codebase-memory-mcp/config.json
(global) and {repo}/.codebase-memory.json (project), merging them with
project config taking precedence. Unknown language values warn and skip.
Missing files are silently ignored (fail-open).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Enforce WarningsAsErrors: '*' with zero suppressions of entire check categories.
All non-FP warnings fixed at source; true false positives get per-instance NOLINT
with reasoning. Fixes include: braces around statements (1679), isolate declarations
(75), explicit casts for multilevel pointer conversions, missing includes, function
parameter documentation via NOLINT, and proper NOLINTBEGIN/NOLINTEND nesting.
