The previous PreToolUse hook gated Grep/Glob/Read/Search with 'exit 2'
on the first call per session, which broke Claude Code's
read-before-edit invariant (issue #362) and could deny tool calls under
upgrade/missing-binary failure modes.
Replace it with a structurally non-blocking augmenter:
- New 'codebase-memory-mcp hook-augment' subcommand reads the hook JSON
from stdin and, for Grep/Glob, queries search_graph (in-process, no
shell) and emits hookSpecificOutput.additionalContext. Every failure
path (no project, short token, missing binary, slow query, timeout)
exits 0 with no stdout — the hook physically cannot block a tool call.
- 300 ms SIGALRM/_exit(0) in-process deadline; 5 s settings.json timeout
backstop. Output is written exactly once at the very end, so a
mid-work timeout yields a clean no-op (never partial JSON).
- Matcher narrowed to 'Grep|Glob' (Read explicitly excluded) for Claude;
Gemini matcher narrowed to 'google_search|grep_search' (excludes
read_file) for the same reason.
- The installed shim is a thin wrapper that delegates to the binary;
legacy filename 'cbm-code-discovery-gate' is kept so existing
settings.json entries upgrade with zero migration. Installer refuses
to embed binary paths containing a double quote (shim injection
defense).
- Per-agent 'old matchers' lists let upsert/remove clean up historical
matcher strings during upgrade.
- Smoke tests (8d/8e/8l) updated to assert the new behavior and
regress-test against re-introducing Read in the matcher or 'exit 2'
in the shim.
- Session reminder text updated: 'always Read a file before editing it'
replaces the prior 'fall back to Read only for text content'.
(cherry picked from commit f72c8e68c4d91e52911a569a967ad782ce5472b2)
Add a C# language LSP resolver mirroring the existing Go/C/PHP/Python
modules: type evaluation, using-directive resolution, method lookup,
and a corpus-seeded .NET BCL stdlib data table.
- internal/cbm/lsp/cs_lsp.{c,h}: resolver implementation
- internal/cbm/lsp/generated/cs_stdlib_data.c: BCL type/method tables
- wire CBM_LANG_CSHARP dispatch in cbm.c and lsp_all.c
- register cs_lsp + cs_lsp_bench suites in the test harness
- tests/test_cs_lsp.c, tests/test_cs_lsp_bench.c
(cherry picked from commit 8f677c67a0c9d7829cd6ac27393de2d0aaaac534)
cbm_store_checkpoint() invoked SQLITE_CHECKPOINT_TRUNCATE, the most
aggressive mode. When two cbm-mcp processes share a cache dir, one
process's TRUNCATE can shrink files while another has them mmap'd,
raising SIGBUS on macOS. PASSIVE never blocks readers and never
ftruncate()s either file; SQLite still autocheckpoints in PASSIVE
mode at 1000-page boundaries, so reclamation is unaffected for
single-process users.
Recommended by SQLite docs for shared databases:
https://www.sqlite.org/pragma.html#pragma_wal_checkpoint
Hard-coded `PRAGMA mmap_size = 67108864` in configure_pragmas() left
no path for users running multiple cbm-mcp instances against the
same cache to opt out of memory-mapped I/O. On macOS, when one
instance's checkpoint or reindex truncates the DB file under
another instance's live mmap, accessing the now-missing pages
raises SIGBUS, taking the process down.
Setting CBM_SQLITE_MMAP_SIZE=0 reverts to read()/pread() I/O,
which returns recoverable SQLITE_IOERR instead of crashing the
process.
- Default unchanged (67108864 / 64 MB). No behavior change for
single-instance users.
- Malformed values (non-numeric, partial-numeric) fall back to
the default rather than failing the store open.
- Negative values clamp to 0.
- New tests: tests/test_store_pragmas.c covers all five resolver
paths plus an integration smoke that opens a file-backed store
with mmap disabled.
Empirical evidence: 9 SIGBUS crash reports collected on macOS
arm64 v0.6.0 in a 14-hour window, all signature 'cluster_pagein
past EOF' with stacks bottoming in SQLite btree code under the
watcher thread's incremental-index pipeline.
Per-file LSP (run from cbm_extract_file) only sees one file's defs, so
callees whose receiver type comes from an imported module stayed
unresolved — even though Go/C/Py/TS already had cross-file entry points
(cbm_run_X_lsp_cross). Those entry points existed but no pipeline pass
ever invoked them. PHP didn't even have a cross-file entry point.
This change lands the missing pieces:
1. cbm_run_php_lsp_cross + cbm_batch_php_lsp_cross — PHP cross-file
resolver mirroring py_lsp_cross / ts_lsp_cross. Reuses PHPLSPContext,
php_lsp_init, php_lsp_add_use, php_lsp_process_file. Handles PHP-
specific labels (Class / Interface / Trait / Enum / Type) and uses
def_module_qn so cross-file return types qualify against the def's
own module rather than the importing module's.
2. pass_lsp_cross.c — new pipeline pass. For every cached file result,
builds a project-wide CBMLSPDef[] (CBMDefinition → CBMLSPDef
converter), per-file import map (from gbuf IMPORTS edges), then
dispatches to the matching cbm_run_X_lsp_cross. Output strings get
copied into the per-file arena (with O(N²) memory bug avoided by
running each file inside a fresh scratch arena that gets destroyed
after the LSP call — the LSP allocates its registry + stdlib +
project defs in scratch, not in the caller's persistent arena).
3. Wired into both pipelines:
- sequential: definitions → k8s → lsp_cross → calls → ...
- parallel: parallel_extract → registry_build → lsp_cross →
parallel_resolve → ...
4. Updated tests/test_parallel.c run_parallel helper to mirror the new
parallel ordering so existing cross-file Python/PHP tests now exercise
pass_lsp_cross.
Languages covered: Go, C/C++/CUDA, Python, TS/JS/JSX/TSX, PHP. PHP cross
file is new. The other four had functions but no pipeline integration
until this change.
Verified on a 1670-file FastAPI index: pass_lsp_cross processes 1050
Python files (525 non-LSP files skipped), collects 4070 project defs in
9.2s, and parallel_resolve subsequently reports lsp_overrides=568 — the
LSP-strategy attribution gain over per-file alone.
Per-LSP emit functions dedup by (caller_qn, callee_qn) within their own
run; pxc_append_results dedups again at append time so cross-file
entries duplicating per-file output get dropped instead of inflating
result->resolved_calls.
Resolved 5 conflicts where the php-lsp branch and main both added LSP
infrastructure but with different APIs:
- Makefile.cbm: kept both TEST_PHP_LSP_SRCS and main's py_lsp/scope/
type_rep/stack_overflow test source variables in ALL_TEST_SRCS.
- internal/cbm/cbm.c: kept both #include lsp/php_lsp.h and lsp/py_lsp.h;
the per-language dispatch already supports stacking PHP after PYTHON.
- internal/cbm/lsp_all.c: kept both lsp/php_lsp.c + php_stdlib_data.c
AND lsp/py_lsp.c + python_stdlib_data.c in the compilation unit.
- src/pipeline/pass_calls.c: took main's version entirely.
find_lsp_resolution + resolve_single_call(ctx, call, lsp_calls, ...)
is language-agnostic — it matches any CBMResolvedCall whose caller_qn
hits the textual call's enclosing function and whose callee_qn
short-name hits the textual callee. PHP LSP populates the same
result->resolved_calls array via cbm_run_php_lsp, so the unified
resolver handles PHP for free. Dropped the php-branch's
qn_short_name + lsp_override_resolution helpers as superseded.
Locked confidence floor at main's CBM_LSP_CONFIDENCE_FLOOR = 0.6f
(was 0.5f on the php branch).
- tests/test_main.c: kept all suite externs and RUN_SUITE invocations
for php_lsp + py_lsp/py_lsp_bench/py_lsp_stress/py_lsp_scale +
scope + type_rep.
Auto-merged: src/pipeline/pass_parallel.c.
Brings in 52 commits from main since the branch was forked at c768b61
(#207 watcher follow-up): #208 through #281, dependabot bumps, Pine
Script, AUR docs, INHERITS edges, get_architecture wiring, codeql
revert, the entire python-lsp subsystem.
Suite: 3249 passed, 0 failed (was 2971 on main; +278 new tests from
php-lsp matches the branch's claimed test count).
Audit for O(n^2) hot paths after Round 10. The dedup loop in
py_emit_resolved_call scanned ALL prior entries linearly, making
emission O(N) and total file work O(N^2). On a synthetic 2000-class
fixture with 8000 calls, that dominated time at 5083 ms.
Cap the scan window at the last 256 entries. Most duplicate
emissions come from the same expression being walked twice (resolver
+ emitter passes), so they're temporally adjacent — a 256-entry
window catches every common case while keeping the dedup O(1) per
emission.
Scale measurements (test_py_lsp_scale.c added):
| Classes | Calls | Resolved | Before | After |
|--------:|------:|---------:|-------:|------:|
| 100 | 400 | 300 | 55ms | 57ms |
| 500 | 2000 | 1500 | 333ms | 323ms |
| 2000 | 8000 | 6000 | 5083ms | 1804ms|
Ratio for 20x input: 91.8x -> 31.4x (linear would be 20x; quadratic
would be 400x). No remaining quadratic hot path in code added by
this branch.
Other hot paths surveyed:
- py_register_def -> cbm_registry_lookup_type: O(types) per def at
registration, O(D x types) for D defs. Pre-existing linear scan in
type_registry.c; hash-indexing it would be a cross-language
optimization (Go and C++ LSPs benefit too).
- py_lookup_attribute / py_lookup_field: O(types) per call site.
Same pre-existing linear scan. With ~900 stdlib types + per-file
defs, accounts for ~75µs/call at 2000-class scale.
- MODULE submodule prefix scan: O(funcs) per module-attribute access,
early-exits on first match. Bounded by registry size.
- lambda / dict-literal registries: linear, but bounded by per-file
N which is small.
Bench unchanged at 100% / 12ms (52/52 calls, 178 LOC).
Round 8. Stress test suite (test_py_lsp_stress.c) added with 22 advanced
patterns; bench fixture grew to 178 lines covering the full surface.
with-as binding via __enter__:
- Look up __enter__ on the context manager's NAMED type. Use its
return type when known; fall back to the manager type itself
(matches the practical case where __enter__ returns self). Handles
tree-sitter Python's wrapping of `X as y` in an as_pattern node
inside with_item, plus the flat field-only shape.
except-as binding:
- Bind exception alias to the caught type. Resolves through
except_clause's as_pattern wrapper and the as_pattern_target
identifier wrapper.
Tuple unpacking:
- Both `a, b = f()` and `for k, v in d.items()` destructure the RHS
element-by-element when the rhs is TUPLE or TEMPLATE("tuple", ...).
*rest binds the trailing slice as list[elem]. Shared helper
py_bind_for_target used by for-statement, comprehension for_in_clause,
and assignment LHS.
Slice subscript:
- `lst[1:3]` returns the same TEMPLATE container — list[T][1:3] is
list[T], dict[K, V][1:3] is dict[K, V] (rare but consistent).
Container TypeVar substitution:
- Without a constraint solver we can't generally substitute K, V into
registered method return types. Hand-rolled the most-impactful
cases: dict.items/keys/values/get/pop/copy, list.copy/__iter__/pop,
set.copy. These return properly parameterized ItemsView[K, V] /
KeysView[K] / ValuesView[V] / Optional[V] / etc. so subsequent
iteration and unpacking yield typed elements.
ItemsView / KeysView / ValuesView / dict_items / dict_keys / dict_values
added to py_iterable_element_type. ItemsView[K, V] iterates as
tuple[K, V], unpacked correctly by py_bind_for_target.
Bench: 178 LOC, 52 calls, **52 resolved (100%)**, 11.59 ms (ASan + UBSan).
22 new stress tests, 8 of 10 documented gaps closed in this round.
Remaining gaps documented inline (typeddict_subscript, post-early-return
narrowing, function-as-dict-value, match sequence pattern, lambda
inference). All 2898 prior tests stay green; total 2920 / 0.
Final phase of Python LSP integration. Adds:
- docs/BENCHMARK_PYTHON.md — methodology document covering target
metrics (per-file LSP overhead < 10 ms, resolution ratio ≥ 40% on
application code, ≥ 70% with stdlib, < 5% spot-check false positive
rate), the declared accepted-failure-class ceiling (metaclass
__call__, __getattr__/__getattribute__, dynamic setattr, runtime
importlib, monkey-patching, eval/exec, descriptors beyond the
standard property/classmethod/staticmethod, dynamic type() classes),
and instructions for running scripts/benchmark-index.sh against the
largest Python repos in datadice (falkemedia: 232 files, api-hub:
188 files).
- tests/test_py_lsp_bench.c — in-process benchmark wired into
scripts/test.sh. Runs cbm_extract_file on a 65-line Python fixture
covering imports, dataclasses, classmethods, single-inheritance with
super(), typed parameters, attribute chains, and stdlib calls.
Asserts soft floor (resolved >= calls/4) and hard time budget
(< 50 ms under sanitizers). Initial run reports:
bench: 65 lines, 16 calls, 13 resolved (81%), 2.45 ms
81% beats the 40% application-code target and the 70%
application-plus-stdlib target. 2.45 ms under ASan + UBSan is well
inside the 10 ms (no-sanitizer) production budget.
This wraps the 11-phase plan documented in PYTHON_LSP_PLAN.md. Total
test count: 2874 / 0 (100% pass). Repository-level benchmarks via
scripts/benchmark-index.sh remain a manual operation against a fresh
production binary; methodology is now in place for when that runs.
CALLS edges for cross-file function calls were mis-wired or missing in
TypeScript projects that use path aliases (e.g. @/lib/auth → src/lib/auth):
1. resolve_relative_import only handled "./relative" paths, so
"@/lib/auth"-style imports never reached IMPORTS edges. The import
map stayed empty, Strategy 1 in cbm_registry_resolve failed, and
resolution fell through to name-based lookups that pick the wrong
target when functions share names across files.
2. resolve_import_map built the candidate QN as the bare module QN
when the callee had no dot ("requireAdmin" vs "pkg.Func"), instead
of module_qn.requireAdmin. Even with a populated import map, bare
calls still mis-resolved.
Fix:
- New src/pipeline/path_alias.{h,c}: language-agnostic build-tool
path-alias subsystem. Currently parses tsconfig.json and
jsconfig.json compilerOptions.paths + baseUrl via the existing
yyjson dependency (with ALLOW_COMMENTS|ALLOW_TRAILING_COMMAS for
JSONC). The data model (cbm_path_alias_t / map / scope /
collection) is generic; adding Vite/Webpack/Python loaders means
writing a new file-parsing helper and registering it in
find_alias_files — the resolver, collection, and pipeline
integration do not change.
- O(n log n) ordering: scope-by-dir-specificity and entry-by-
prefix-specificity are sorted with qsort, replacing the O(n²)
bubble sorts that the original draft used. With caps of 256
entries per config and 256 configs per repo, the sort is
effectively flat.
- Resource ceilings with logging: directory recursion capped at
32 levels; total config files capped at 256; per-config size
capped at 64 KB. Cap hits emit cbm_log_warn so silent truncation
on pathological monorepos shows up in the index log.
- Walk hygiene: skips node_modules, .git, dist, build, .next,
coverage, target.
- Single integration point: cbm_pipeline_resolve_module gets a
Strategy 1b between relative-import resolution and pkgmap so both
the sequential and parallel pipelines pick up alias resolution
automatically. Non-TS projects pay nothing — the loader returns
NULL when no usable configs are found.
- Layer 2 fix in registry.c: resolve_import_map now builds the
candidate as resolved.prefix when suffix is NULL (one-line fix,
matching comment).
- Tests: 10 path_alias tests (basic wildcard, specificity, exact,
extension stripping, baseUrl, NULL safety, nearest-ancestor scope
selection, end-to-end loader against a tmp monorepo, no-configs
case) plus a registry test for the bare-call fix. Full suite:
2835 passed, 0 failed.
Cherry-picked and generalized from #243 (originally TypeScript-only
under tsconfig-named types). Renamed types (cbm_path_alias_scope_t /
collection_t in place of cbm_tsconfig_*) and centralised the load
into cbm_load_path_aliases so future loaders for other build tools
slot in without touching extraction or pipeline code.
Closes#243.
Co-authored-by: Peter Cox <info@petercox.ie>
Covers parameter binding (typed param, arrow function, $this, catch),
method dispatch (typed receiver, chain, self/parent), static calls,
namespace + use resolution, PHPDoc @var, magic methods, and the
ConfiguresPrompts $prompt->value() regression from PHP_LSP_PRE_FLIGHT.md.
Wired into ALL_TEST_SRCS and suite_php_lsp invocation in test_main.c.
All 16 tests pass; total suite is now 2829/0 passing.
Phase 2 of Python LSP integration. Adds the py_lsp module mirroring
the go_lsp / c_lsp pattern: PyLSPContext, single-file entry
(cbm_run_py_lsp), cross-file + batch shells, stdlib registration stub.
Bodies are no-ops at this phase. Subsequent phases fill in:
Phase 3 — imports
Phase 4 — scope binding
Phase 5 — expression typing
Phase 6 — attribute resolution + method dispatch
Phase 7 — decorators
Phase 8 — class hierarchy + generics
Phase 9 — cross-file resolution + batch
Phase 10 — cbm_python_stdlib_register from typeshed
Wired into cbm_extract_file alongside the existing Go and C/C++ LSP
calls. lsp_all.c #includes py_lsp.c so the CGo umbrella picks it up
without a separate translation unit.
tests/test_py_lsp.c adds 5 smoke cases: empty source, single function,
single class, syntax error recovery, imports pass-through. The helper
shape (extract_py / find_resolved / require_resolved) matches
test_go_lsp.c so future phases can grow the suite by category.
All 2839 existing tests stay green; 5 new py_lsp tests pass.
Phase 1 of Python LSP integration. Five new CBMTypeKind variants needed
for Python type modeling, plus structural-equality helper used by union
deduplication and protocol matching:
- UNION: A | B / Union[A, B] / Optional[T]. cbm_type_union normalizes
by flattening nested UNIONs, deduplicating members via cbm_type_equal,
and collapsing single-member unions to their member. Empty unions
reduce to UNKNOWN.
- LITERAL: Literal["foo", 3] — wraps a base type with literal text;
text-distinct literals compare unequal.
- PROTOCOL: typing.Protocol with method-name set. Structural matching
via cbm_type_protocol_satisfied_by — succeeds when the candidate's
method set is a superset of the protocol's.
- MODULE: bound name from `import X`, distinguishing module-attribute
lookup from class-attribute lookup. Equality is by module_qn.
- CALLABLE: Callable[[A, B], R] including the elliptic Callable[..., R]
encoded as param_count = -1.
cbm_type_equal added as a recursive structural-equality helper covering
all kinds; FUNC/INTERFACE/STRUCT fall back to pointer identity since
their structural members are rarely compared and the cost would be
substantial.
Adds tests/test_type_rep.c with 16 cases covering normalization,
equality, and protocol satisfaction.
All 2823 existing tests stay green; 16 new type-rep tests pass.
AST traversal functions used fixed-size TSNode stack[] arrays. When the
DFS stack filled up, the child-push loop exited silently, dropping
entire subtrees without warning (e.g. 600 TS imports → 511, 130 Express
routes capped at 512). Reported in #199; also addresses #213 (large
TypeScript files producing zero nodes from stack exhaustion) and #215
(SEGV in template calls from stack overflow).
Adds ts_node_stack.h — a growable stack backed by the existing arena
allocator. Initial capacity matches previous fixed caps so small files
allocate no extra memory; doubles on overflow instead of truncating.
Old blocks are abandoned in the arena and freed on arena_destroy at
the end of file extraction (no realloc, no mixed lifetimes).
Applied to all 14 TSNode fixed stacks across 9 extraction files
(extract_calls, extract_channels, extract_defs, extract_env_accesses,
extract_imports, extract_semantic, extract_type_assigns, extract_type_refs,
extract_usages). walk_defs (uses walk_defs_frame_t, different struct
type) deliberately left as-is.
7 regression tests in tests/test_stack_overflow.c cover TS imports >512,
JS/Python calls >512, Go calls >1024, Express routes >150, deeply
nested calls, and YAML vars >256.
Cherry-picked from #217 with merge-conflict resolution against post-#206/#207
main:
- Makefile.cbm: keep TEST_STACK_OVERFLOW_SRCS plus main's new
TEST_ZSTD_SRCS / TEST_ARTIFACT_SRCS.
- extract_channels.c: keep main's scan_string_consts_js name (main also
added scan_string_consts_python after the PR was opened); declare
CHAN_STACK_CAP in the enum block.
- test_stack_overflow.c: replace sprintf with snprintf using the actual
remaining buffer size, since macOS clang rejects sprintf and naive
snprintf with sizeof(p) on a char* fails -Wformat-truncation.
Closes#199, #213, #215. Closes#217.
Phase 0 of Python LSP integration. Two shared-infrastructure fixes that
silently bug Python without affecting Go/C++ regression tests:
- CBMScope replaces the fixed 64-binding array with a growable linked
list of 16-binding chunks. The arena pointer is captured at scope_push
and reused for chunk allocation, so existing callers benefit from
dynamic growth without an API change. Python class bodies and large
dataclass scopes routinely exceed 64 locals; this prevents silent
drops past the legacy cap.
- CBM_LSP_MAX_LOOKUP_DEPTH=16 wraps go_lookup_field_or_method and
c_lookup_member via depth-aware internal helpers. Pathological
multi-inheritance hierarchies (SQLAlchemy / Django / Zope) can exceed
20 levels and previously stack-overflowed; now they degrade to NULL
past depth 16 instead.
Adds tests/test_scope.c with 10 cases covering basic API, parent-chain
walks, child-shadowing, and dynamic growth at 300 bindings + chunk
boundaries. test_main.c and Makefile.cbm wire the new suite.
PYTHON_LSP_PLAN.md added at the worktree root: full phased plan plus
the verbatim challenger review.
All 2823 existing tests stay green; 10 new scope tests pass.
Bare import specifiers (@myorg/pkg, github.com/foo/bar, use my_crate::foo)
previously produced zero IMPORTS edges. This adds manifest-aware resolution
during parallel extraction for JS/TS (package.json), Go (go.mod),
Rust (Cargo.toml), Python (pyproject.toml), PHP (composer.json),
Java (pom.xml), Gradle (build.gradle), Dart (pubspec.yaml),
Elixir (mix.exs), and Ruby (*.gemspec).
Workers parse manifest files during extraction with zero extra I/O,
entries are merged into a hash table before registry build, and
cbm_pipeline_resolve_module() provides unified resolution with
prefix matching for Go/Java/PHP path-based specifiers.
Closes#180.
Based on #184 by dLo999 (JS/TS package map approach).
Co-Authored-By: Dustin Obrecht <dustin@kurtnoble.com>
Add .codebase-memory/graph.db.zst — a zstd-compressed knowledge graph
artifact that can be committed to the repo. Teammates bootstrap from
the artifact instead of running a full reindex from scratch.
- Vendor zstd 1.5.7 (amalgamated build) for 8-13:1 compression
- Two-tier export: zstd -9 + index stripping for explicit index,
zstd -3 for watcher/incremental auto-updates
- Import: decompress → integrity check → auto-recreate indexes
- Bootstrap in handle_index_repository: when no local DB exists but
artifact is present, import first then run incremental
- Auto-create .gitattributes with merge=ours to prevent conflicts
- Fix: add missing idx_edges_url_path to create_user_indexes and
url_path_gen generated column to init_schema
- 13 new tests (5 zstd wrapper + 8 artifact round-trip/edge cases)
New index_repository mode="cross-repo-intelligence" that matches Routes,
Channels, and async topics across indexed projects:
- CROSS_HTTP_CALLS: matches HTTP_CALLS edges against Route nodes in
other projects via deterministic QN (__route__METHOD__/path)
- CROSS_ASYNC_CALLS: matches ASYNC_CALLS edges by broker + topic
- CROSS_CHANNEL: matches Channel nodes (EMITS/LISTENS_ON) by name +
transport across projects
Edges are written bidirectionally: both source and target project DBs
get a CROSS_* edge so the link is visible from either side.
Usage: index_repository(repo_path="/path", mode="cross-repo-intelligence",
target_projects=["*"])
New files: src/pipeline/pass_cross_repo.c/.h
Modified: src/mcp/mcp.c (handler + tool schema), Makefile.cbm
- code_vectors_blob.S: preprocessor conditionals for macOS (Mach-O
__DATA,__const + underscore prefix) vs Linux (ELF .rodata, no prefix)
- Makefile: use $(CC) -c instead of $(AS) to enable preprocessor on .S
- Add vendored/nomic to KNOWN_VENDORED security allowlist (pure int8
vector data, zero executable code)
- Update vendored checksums
Ports four targeted improvements from #162 (Koolerx):
- FTS5 BM25 ranked search via a new `query` parameter on search_graph.
Uses a contentless nodes_fts virtual table populated at dump time with
a cbm_camel_split() SQL function that splits camelCase identifiers
(updateCloudClient -> update Cloud Client) so tokenized word queries
match identifier internals. BM25 rank is boosted by node label
(Function/Method +10, Route +8, Class/Interface/Type/Enum +5) and
noise labels (File/Folder/Module/Variable) are filtered out. The new
path is opt-in via the query parameter; regex name_pattern and vector
semantic_query paths are untouched.
- Enable SQLITE_ENABLE_FTS5 in the vendored sqlite3 build flags.
- Rebuild nodes_fts after incremental reindex. The btree dump path
bypasses any FTS5 triggers, so the index is wiped and repopulated via
delete-all + INSERT .. SELECT after each incremental merge.
- Add Interface to the label filter on cbm_registry_add in both the
parallel and sequential definition passes, so C#/Java class
declarations like Foo : IBar can resolve IBar to an INHERITS edge
target during enrichment.
- Add an explicit C# base_list handler to extract_base_classes() that
iterates named children directly instead of returning the raw node
text. Without this the fallback path returned ": IExamService" (with
the leading separator) and registry lookup always failed, so C#
INHERITS edges to interfaces never resolved.
Co-Authored-By: Koolerx <Koolerx@users.noreply.github.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
Local test infra now covers 4 platforms:
- Linux arm64 (Ubuntu, ASan+LeakSan)
- Linux portable (Alpine musl, static binary)
- Windows (mingw cross-compile)
- macOS (native)
Fixes:
- Remove sys/unistd.h and sys/poll.h (glibc-only, musl lacks them)
- Add STATIC=1 support to Makefile (appends -static to LDFLAGS)
- CI portable build uses docker run alpine instead of container directive
(GitHub Actions JS actions break in Alpine containers on ARM64)
tre-compile.c has misaligned struct access on Windows (UBSan kills
the test runner). The misalignment is in vendored code we don't
control. Add -fno-sanitize=alignment to tre test build only.
Comprehensive test suite using real repo (FastAPI 0.99.1, ~1100 files):
- Full index baseline with perf metrics (nodes, edges, RSS, timing)
- Incremental: modify/add/delete/rename files, noop, formatter run
- Adversarial: empty file, syntax errors, 5000-line function, binary
- Stress: rapid reindex, batch 20 files, delete-all + re-add
- Accuracy: incremental vs full within ±2 nodes/edges
- MCP tools: all 14 tools, every parameter, content validation
- Error handling: bad project, missing params for each tool
- Performance: 5s timeout on every tool call
Fix use-after-free in detect_changes and manage_adr:
yyjson_mut_arr_add_str stored pointer to stack buffer without copy.
Changed to yyjson_mut_arr_add_strcpy / yyjson_mut_obj_add_strcpy.
- lang_specs.c: add LangSpec entries for CBM_LANG_KUSTOMIZE and CBM_LANG_K8S
(both reuse tree_sitter_yaml()); add cbm_ts_language() switch cases
- extract_k8s.c: new file implementing cbm_extract_k8s(); kustomize path
walks block_sequence items under resources/bases/patches/components/
patchesStrategicMerge and emits CBMImport per scalar; k8s path extracts
apiVersion/kind/metadata.name and emits CBMDefinition with label "Resource"
and name "Kind/metadata-name"; malformed manifests (missing kind or name)
produce zero definitions
- cbm.h: declare cbm_extract_k8s() alongside other sub-extractor entry points
- cbm.c: call cbm_extract_k8s() after unified extraction for the two new langs
- Makefile.cbm: add extract_k8s.c to EXTRACTION_SRCS
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Read extra_extensions from ~/.config/codebase-memory-mcp/config.json
(global) and {repo}/.codebase-memory.json (project), merging them with
project config taking precedence. Unknown language values warn and skip.
Missing files are silently ignored (fail-open).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Incremental pipeline (pipeline_incremental.c):
- Disk-based delta re-indexing: classifies files by mtime+size vs stored
hashes, deletes changed nodes (edges cascade), re-parses only changed
files, merges into existing DB via cbm_gbuf_merge_into_store
- pipeline.c auto-routes: existing DB with hashes -> incremental, else full
- Full pipeline persists file hashes after dump for next incremental run
- No-op when nothing changed (watcher polls every 5-60s, 99% skip)
- 4 tests: noop, changed file, deleted file, new file added
ADR hints (ported from Go):
- index_repository: adr_present + adr_hint when no ADR exists
- get_graph_schema: adr_present + adr_hint per project
- manage_adr GET: adr_hint with creation instructions when no ADR
cbm_store_begin_bulk() was switching the SQLite journal mode from WAL to
MEMORY for write throughput. If the process crashed mid-bulk-write the
in-memory rollback journal was lost, leaving the database file in a
partially-written, unrecoverable state.
WAL mode is inherently crash-safe: uncommitted WAL entries are discarded
on the next open. The performance benefit of bulk mode is preserved via
synchronous=OFF and an enlarged cache_size, both of which are safe under
WAL.
Remove the PRAGMA journal_mode = MEMORY from cbm_store_begin_bulk and
the matching PRAGMA journal_mode = WAL from cbm_store_end_bulk. Update
the header comments to reflect the new invariant.
Add tests/test_store_bulk.c with three tests:
- bulk_pragma_wal_invariant: asserts journal_mode remains "wal" after
cbm_store_begin_bulk via an independent read-only connection
- bulk_pragma_end_wal_invariant: asserts journal_mode remains "wal" after
cbm_store_end_bulk
- bulk_crash_recovery: forks a child that enters bulk mode, opens an
explicit transaction, writes data, then calls _exit() without committing;
the parent verifies the database opens cleanly and baseline data survives
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Call cbm_mem_collect() (mi_collect(true)) after pipeline_free in both
index_repository and auto-index, returning mimalloc pages to OS
- Add -static to Windows linker flags so the binary is standalone
(no libc++.dll / libunwind.dll dependency)
1. cbm-with-ui target: add $(OBJS_VENDORED_PROD) as dependency so
grammar .o files are compiled before linking in parallel make.
Without this, make -jN fires the link step as soon as embed
finishes, racing against grammar compilations.
2. test_ui.c: replace system("mkdir -p") with cbm_mkdir_p() for
Windows compatibility. Add compat_fs.h include.
Martin Vogel