Apply the platform.h memory wrappers across the largest free / realloc
hot spots so cleanup paths are uniform and pointers are NULLed on free
to prevent accidental use-after-free / double-free.
- src/cypher/cypher.c (~85 sites): replace free((void *)x) with
safe_str_free(&x) for owned const char * fields; the array-pointer
cleanup in parse_in_list error paths uses safe_free(vals) since vals
is const char **.
- src/store/store.c (~81 sites): same const-string substitution across
cbm_search_result_t cleanup, project list, schema info, etc.
- src/mcp/mcp.c (~21 sites): same, plus collect_grep_matches growth
loop converted to safe_grow(gm, gm_count, gm_cap, PAIR_LEN) — safe
here because grep_match_t holds fixed-size char arrays only.
- src/pipeline/pass_githistory.c: free_counter callback uses
safe_str_free for the hash key.
Refactor work cherry-picked from #210; #210 itself bundled .claude/,
.github/workflows/, and a stray .claude/worktrees/ submodule that did
not belong in the repo, and most of its other changes had landed via
#206 / #207 / #208 already. The cypher / store growth-path safe_realloc
loops are intentionally NOT converted to safe_grow because their
elements own heap pointers — a failed realloc would leak them; the
non-destructive realloc + cleanup pattern from #206 is preserved.
Co-authored-by: map588 <map588@pitt.edu>
Centralize repetitive heap-management patterns into four wrappers
alongside the existing safe_realloc:
- safe_free(ptr): frees and NULLs a pointer to prevent
double-free / use-after-free
- safe_str_free(&str): frees a const char * and NULLs it without
forcing the caller to cast away const at every site
- safe_buf_free(buf, &count): frees a heap array and zeros the
paired count, so the dangling-array / nonzero-count footgun
cannot survive
- safe_grow(arr, n, cap, factor): one-line capacity-doubling
realloc for arrays whose elements do not own additional heap
memory; documents the safe_realloc free-on-failure semantic so
arrays of owned pointers keep using a manual realloc+cleanup
pattern
Cherry-picked from #209's platform.h additions; the rest of #209
duplicated work that landed via #206 and #207, so closing that PR
in favor of this targeted commit.
Co-authored-by: map588 <map588@pitt.edu>
Follow-up to #208. The decorator pass now emits USAGE edges, but the
degree-filter subqueries in cbm_store_search counted only CALLS edges,
so a function used only via @decorator was still reported as dead by
search_graph(max_degree=0). Extend the in_deg/out_deg subqueries to
count CALLS and USAGE so both pass_usages output and the new decorator
edges feed into dead-code detection as intended.
Decorators previously only created DECORATES edges. References like @login_required were invisible to find-references queries that walk CALLS/USAGE relationships. Both sequential (resolve_decorator) and parallel (resolve_def_decorators) paths now emit a USAGE edge alongside DECORATES. Empty {} properties avoid clobbering richer metadata from pass_calls when a function both has @decorator and calls decorator() directly. Using USAGE instead of CALLS keeps pass_tests' CALLS-scan from inferring false TESTS edges (decorated test function → decorator).
Follow-up to #207:
- cbm_watcher_new: NULL-check cbm_ht_create() and free w on OOM so
later watch/unwatch/touch don't dereference a NULL hash table.
- cbm_watcher_watch_count: drop const from the signature and remove
the const_cast around projects_lock. The function necessarily
touches synchronization primitives so a const receiver was
misleading; all current callers pass a non-const pointer.
- http_server: replace lazy log mutex init with 3-state UNINIT/INITING/INITED CAS pattern; cbm_ui_log_init() called from main before thread creation, also still invoked lazily from cbm_ui_log_append() so early single-threaded logs are not dropped
- watcher: protect projects hash table with cbm_mutex_t (watch/unwatch/touch/watch_count/free); poll_once snapshots project pointers under lock then iterates without it so git I/O and indexing don't block watch/unwatch
- watcher: handle state_new() OOM in watch path
- compat_thread: add cbm_thread_detach() for POSIX and Windows; both join() and detach() clear the handle on success for consistent lifecycle tracking
- http_server: detach index job threads to prevent handle leaks
Follow-up to #206. Adds the allocation-check coverage that the original
PR did not reach:
- store: NULL-check malloc in cbm_store_get_schema node-labels and
edge-types blocks; replace safe_realloc with non-destructive realloc +
cleanup so a partial result is freed (and node-labels block freed via
cbm_store_schema_free) instead of leaked.
- cypher: NULL-check heap_strdup return in parse_props (key/value),
parse_rel_types (first and subsequent types), and parse_in_list (op
and list values). Each failure path frees prior heap_strdup'd entries
before returning.
- cypher: parse_case_expr realloc-failure cleanup now also frees
then_val for the just-parsed branch and for each previously stored
branch, not just the when_expr trees.
strndup is a POSIX/GNU extension not available in mingw clang on
Windows. Replace 6 bare strndup() calls with cbm_strndup() and
include foundation/compat.h, matching the pattern already used in
pass_calls.c.
- pass_parallel.c: detect gRPC calls when resolved QN contains
ServiceClient/ServiceGrpc/Servicer — emits GRPC_CALLS edges for
calls resolved to generated protobuf client interfaces
- pass_parallel.c: emit_grpc_edge falls back to resolved QN when
callee_name alone doesn't contain service.method pattern
- extract_calls.c: add resolve_chained_selector() for iterative
extraction of chained method calls (NewClient(conn).GetBar →
"NewClient.GetBar") without recursion
Known limitation: Go receiver-method chained calls like
pb.NewCartServiceClient(conn).GetCart() in rpc.go are not yet
extracted as CALLS edges — the Go grammar's AST structure for
these patterns needs dedicated investigation.
Bare import specifiers (@myorg/pkg, github.com/foo/bar, use my_crate::foo)
previously produced zero IMPORTS edges. This adds manifest-aware resolution
during parallel extraction for JS/TS (package.json), Go (go.mod),
Rust (Cargo.toml), Python (pyproject.toml), PHP (composer.json),
Java (pom.xml), Gradle (build.gradle), Dart (pubspec.yaml),
Elixir (mix.exs), and Ruby (*.gemspec).
Workers parse manifest files during extraction with zero extra I/O,
entries are merged into a hash table before registry build, and
cbm_pipeline_resolve_module() provides unified resolution with
prefix matching for Go/Java/PHP path-based specifiers.
Closes#180.
Based on #184 by dLo999 (JS/TS package map approach).
Co-Authored-By: Dustin Obrecht <dustin@kurtnoble.com>
Adds per-label/type property key discovery via json_each() so users can
see which properties are available for Cypher queries without trial and
error. Base columns listed first, JSON keys appended (capped at 50).
Closes#179.
Based on #181 by dLo999.
Co-Authored-By: Dustin Obrecht <dustin@kurtnoble.com>
Detect ~/.kiro/ directory, auto-configure MCP at ~/.kiro/settings/mcp.json
using standard mcpServers format. Adds install, uninstall, detection, and test.
Based on #174 by dLo999, rebased onto refactored install infrastructure.
Co-Authored-By: Dustin Obrecht <dustin@kurtnoble.com>
Add .codebase-memory/graph.db.zst — a zstd-compressed knowledge graph
artifact that can be committed to the repo. Teammates bootstrap from
the artifact instead of running a full reindex from scratch.
- Vendor zstd 1.5.7 (amalgamated build) for 8-13:1 compression
- Two-tier export: zstd -9 + index stripping for explicit index,
zstd -3 for watcher/incremental auto-updates
- Import: decompress → integrity check → auto-recreate indexes
- Bootstrap in handle_index_repository: when no local DB exists but
artifact is present, import first then run incremental
- Auto-create .gitattributes with merge=ours to prevent conflicts
- Fix: add missing idx_edges_url_path to create_user_indexes and
url_path_gen generated column to init_schema
- 13 new tests (5 zstd wrapper + 8 artifact round-trip/edge cases)
Pipeline: gRPC/GraphQL/tRPC client libs in service_patterns, edge emitters
in pass_parallel, __grpc__ Route nodes from .proto service defs in
pass_route_nodes, cross-repo matchers in pass_cross_repo. Also adds MQTT,
NATS, Dapr async patterns.
Protobuf extraction: service as Class, rpc as Function with rpc_name
fallback in func_name_node. Route nodes created by line-range matching
rpc Functions to enclosing service Classes.
Tested: Online Boutique protos produce 16 gRPC Route nodes.
New index_repository mode="cross-repo-intelligence" that matches Routes,
Channels, and async topics across indexed projects:
- CROSS_HTTP_CALLS: matches HTTP_CALLS edges against Route nodes in
other projects via deterministic QN (__route__METHOD__/path)
- CROSS_ASYNC_CALLS: matches ASYNC_CALLS edges by broker + topic
- CROSS_CHANNEL: matches Channel nodes (EMITS/LISTENS_ON) by name +
transport across projects
Edges are written bidirectionally: both source and target project DBs
get a CROSS_* edge so the link is visible from either side.
Usage: index_repository(repo_path="/path", mode="cross-repo-intelligence",
target_projects=["*"])
New files: src/pipeline/pass_cross_repo.c/.h
Modified: src/mcp/mcp.c (handler + tool schema), Makefile.cbm
- Tool handlers: set isError:true on actual failures (delete_project, manage_adr,
detect_changes), include errno context in system errors, add structured hints
for empty results (search_graph, query_graph, trace_path, list_projects, index_status)
- verify_project_indexed: include available projects list in error response
- CLI: extract text from MCP envelope for human-readable output, errors to stderr,
exit code 1 on tool errors, --json flag for raw output
- Cancellation: handle notifications/cancelled, track active pipeline in server
struct, signal handler cancels in-progress pipeline and releases lock, watcher
checks g_shutdown before starting new indexing
- Enum member extraction: C#/Java/TS/C++ enum values as Variable nodes
- JS/TS destructured variable splitting: const { A, B, C } = require()
emits individual Variable nodes instead of one blob
- Cypher -- SQL-style single-line comments
- Cypher in_degree/out_degree virtual node properties (thread-safe
via binding store pointer, enables dead code detection via Cypher)
- Cypher regex matching in inline property filters
- Resolution tiebreaking: test/mock QN deprioritization and C# field
type hint resolution for obj.Method() disambiguation
Co-Authored-By: Tommy <Koolerx@users.noreply.github.com>
Add automated extension-to-language mapping tests for all new grammars
that have file extensions. Removes conflicting extensions:
- .blade.php: compound extension, handled by userconfig system
- .env*: filenames, not extensions (no auto-discovery for dotfiles)
- .sv/.svh: already mapped to Verilog, SystemVerilog uses same grammar
- delete_project now calls cbm_mem_collect() after closing the database
to return freed pages to OS immediately
- Set purge_delay=0 (was 1000ms) so mimalloc purges freed pages without delay
- Set purge_decommits=1 explicitly so macOS uses MADV_FREE_REUSABLE
(immediate RSS accounting) instead of MADV_FREE (lazy, RSS stays high)
Before: deleting Linux kernel index left 8.1G RSS (2.1G dirty pages)
After: dirty pages purged immediately, RSS reflects actual usage
Nested .gitignore support (fixes#178):
- Load per-subdirectory .gitignore during walk, match paths relative to
the gitignore's directory via local_rel_path()
- Root and nested gitignores stack independently
- Owned gitignores collected and freed at walk end (avoids use-after-free
from borrowed pointers on the iterative stack)
Security:
- Run security-strings/install/network + ClamAV + Windows Defender on
ALL binary variants (standard + UI), not just standard
- Whitelist UI bundle URLs (React, Three.js, Google Fonts, Tailwind, W3C)
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
check_already_latest() exits early before printing dry-run info when
the running version matches the latest release. Skip this check in
dry-run mode so the user sees what would happen.
- semantic.c: free strdup'd keys+values in token_map via cbm_ht_foreach
before cbm_ht_free (44KB leak in 7920 allocations)
- code_vectors_blob.S: add Windows COFF branch (.rdata,"dr") alongside
macOS Mach-O and Linux ELF sections
Ports four targeted improvements from #162 (Koolerx):
- FTS5 BM25 ranked search via a new `query` parameter on search_graph.
Uses a contentless nodes_fts virtual table populated at dump time with
a cbm_camel_split() SQL function that splits camelCase identifiers
(updateCloudClient -> update Cloud Client) so tokenized word queries
match identifier internals. BM25 rank is boosted by node label
(Function/Method +10, Route +8, Class/Interface/Type/Enum +5) and
noise labels (File/Folder/Module/Variable) are filtered out. The new
path is opt-in via the query parameter; regex name_pattern and vector
semantic_query paths are untouched.
- Enable SQLITE_ENABLE_FTS5 in the vendored sqlite3 build flags.
- Rebuild nodes_fts after incremental reindex. The btree dump path
bypasses any FTS5 triggers, so the index is wiped and repopulated via
delete-all + INSERT .. SELECT after each incremental merge.
- Add Interface to the label filter on cbm_registry_add in both the
parallel and sequential definition passes, so C#/Java class
declarations like Foo : IBar can resolve IBar to an INHERITS edge
target during enrichment.
- Add an explicit C# base_list handler to extract_base_classes() that
iterates named children directly instead of returning the raw node
text. Without this the fallback path returned ": IExamService" (with
the leading separator) and registry lookup always failed, so C#
INHERITS edges to interfaces never resolved.
Co-Authored-By: Koolerx <Koolerx@users.noreply.github.com>
- Parallelize pass_similarity and pass_semantic_edges via worker pool with
thread-local edge buffers; sequential final merge since gbuf is not
thread-safe. Adds cbm_lsh_query_into() as a thread-safe variant with
caller-provided candidate buffer.
- Add activatable profiling subsystem (CBM_PROFILE=1 env or --profile flag)
for step-level timing of extract, resolve, corpus build, vector phases,
and sqlite dump. Zero overhead when disabled.
- Fix cbm_index_mode_t enum mismatch between pipeline.h (FULL=0, MODERATE=1,
FAST=2) and discover.h (FULL=0, FAST=1). mode=fast silently no-op'd
fast-discovery filtering because discover.c compared against the wrong
value. Linux kernel fast mode went 1:40 -> 3:11 as a result; now back to
1:40. Broaden the filter guard to mode != CBM_MODE_FULL so MODERATE and
FAST both get aggressive discovery.
- Clamp cbm_sem_combined_score output to [0, 1]. The proximity multiplier
returns up to 1.10 as a same-file boost which could push the final
cosine score above 1.0.
- Short-circuit semantic scoring when MinHash jaccard >= 0.95. Exact
near-clones are already emitted as SIMILAR_TO edges; returning 0 here
avoids flooding SEMANTICALLY_RELATED with cross-service copy-paste
boilerplate and frees the edge budget for genuine vocabulary-bridged
relations.
- Validate search_graph semantic_query as an array of strings and return
a clear error for a single-string input. Update the tool description
to spell out the requirement explicitly with an example.
- JSON-escape user-controlled strings (callee names, call arguments,
URL paths, import local_name) in call/argument properties. Introduces
cbm_json_escape() in foundation/str_util.
- Skip SQLite pending_byte_page (file offset 0x40000000) during raw page
writes in sqlite_writer to avoid corrupting databases that cross the
1 GiB boundary.
- Migrate pretrained vector blob from UniXcoder (51K tokens) to
nomic-embed-code (40856 tokens x 768d int8). Includes the extraction
script under scripts/extract_nomic_vectors.py.
Major quality improvement: replace random index vectors with pre-trained
Nomic nomic-embed-code token embeddings. Tokens like 'error' and 'exception' now start
with similar vectors (learned from millions of code repos) instead of
arbitrary random projections. Co-occurrence enrichment adds project-specific
context on top.
Architecture:
- vendored/nomic/code_vectors.bin: 37.7MB raw int8 vectors
- vendored/nomic/code_vectors_blob.S: assembler .incbin (instant build)
- vendored/nomic/code_vectors.h: extern declarations + pretrained_vec_at()
- vendored/nomic/code_tokens.h: 40856 token strings (575KB)
- semantic.c: cbm_sem_random_index() now looks up pretrained vectors first,
falls back to sparse random for unknown tokens
- CBM_SEM_DIM raised from 256 to 768 to match Nomic nomic-embed-code
Also: RRI (Reflective Random Indexing), code pattern vocabulary injection,
120+ abbreviation expansions, callee/caller/body token enrichment,
label filter (Function/Method/Class only) in vector search SQL.
Binary size: 136MB → 169MB (+33MB from embedded vectors).
Search quality: keyword queries return relevant error-handling functions.
Domain-specific keyword queries return the expected functions.
Vector search pipeline (end-to-end):
- store.c: cbm_cosine_i8 custom SQLite function (int8 dot product with
sqrt magnitude), registered at store open time
- store.c: cbm_store_vector_search() builds merged query vector from
keywords via base RI vectors (xxHash), int8-quantizes, runs SQL query
with JOIN nodes ON node_id = id for metadata
- mcp.c: search_graph gains semantic_query param (array of strings),
returns semantic_results alongside regular results
- sqlite_writer.c: fixed build_vector_record to include node_id in
record body (matches build_node_record pattern)
- mcp.c: use yyjson_mut_obj_add_strcpy (not add_str) to avoid
use-after-free when vector results are freed before JSON serialization
Quality note: base RI vectors at query time don't capture co-occurrence
enrichment — results are moderate quality. Storing enriched token vectors
would improve this significantly (future work).
Four changes to reduce false positives from 382K to 58K SIMILAR_TO edges
on the Linux kernel while maintaining ~98% precision:
1. MIN_NODES raised to 30 (leaf tokens) — aligns with BigCloneBench
standard of 50 raw source tokens / 6 lines minimum clone size
2. collect_ast_tokens now records LEAF nodes only, not internal grammar
nodes. Leaf-only counting is language-agnostic: leaves correspond to
actual source tokens regardless of grammar verbosity (C tree-sitter
produces 2x more internal nodes than Python for the same logic)
3. hash_trigrams tracks unique structural trigram count via hash set.
Functions with < 32 unique structural trigrams are skipped — too few
distinct features for K=64 MinHash to produce reliable estimates
4. Structural-weighted MinHash via repetition: each trigram gets weight
0-3 based on count of non-normalised tokens (I/S/N/T = normalised).
Weight-0 trigrams (all-normalised, e.g. I|I|I) are pure noise and
skipped entirely. Weight 1-3 trigrams use repetition-based weighted
MinHash — higher weight = more hash attempts per seed = more likely
to dominate the signature. Achieves IDF-like effect without needing
corpus-wide statistics.
Also adds O(1) hash-set dedup in LSH query (replacing O(n) linear scan)
and bucket size cap of 200 to skip oversized buckets.
Linux kernel: 1:41, similarity 7.7s, 395K fp, 58K edges, ~98% precision.
Tests updated with larger Go functions to meet new thresholds.
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
All hardcoded ~/.cache/codebase-memory-mcp paths now route through
cbm_resolve_cache_dir() in platform.c. Priority: CBM_CACHE_DIR env
var > default ~/.cache/codebase-memory-mcp.
Added db_path_for_project() helper in http_server.c to eliminate 5
copies of the same path construction pattern.
Fixes#154
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
Check GitHub releases/latest redirect header before downloading.
Saves bandwidth and avoids unnecessary index rebuilds.
- "Already up to date" when version matches or is ahead
- --force flag to bypass the check
- Graceful degradation when network is unavailable
- Uses same curl dependency as the download itself
Fixes#142
Co-Authored-By: dLo999 <dLo999@users.noreply.github.com>
One non-UTF-8 node name crashed the entire /api/layout endpoint
(HTTP 500). Force layout can also produce NaN/Inf positions.
- Use YYJSON_WRITE_ALLOW_INVALID_UNICODE (matches MCP serializer)
- Sanitize NaN/Inf: positions default to 0.0, size to 1.0
- Log yyjson error code on serialization failure
Co-Authored-By: Selene29 <Selene29@users.noreply.github.com>
PATH uses ; on Windows, not : — strtok with : splits C:\Users at
the drive letter colon, producing garbage paths. Also skip S_IXUSR
check on Windows since stat() doesn't set Unix permission bits.
Fixes#159
Co-Authored-By: slvnlrt <slvnlrt@users.noreply.github.com>
Replaces hardcoded ~/.local/bin/ with OS-native runtime detection:
- Windows: GetModuleFileNameA()
- macOS: _NSGetExecutablePath()
- Linux: readlink("/proc/self/exe")
Falls back to ~/.local/bin/ if detection fails.
Fixes install.ps1 placing binary in $LOCALAPPDATA/Programs/ while
configs point to non-existent ~/.local/bin/ path.
Fixes#158
Co-Authored-By: slvnlrt <slvnlrt@users.noreply.github.com>
KiloCode is a VS Code extension — uses the same base config dir.
Apply same platform-aware paths as Zed/VS Code: macOS uses
$HOME/Library/Application Support/Code/, Windows uses %APPDATA%/Code/.
cbm_store_bfs doesn't support direction="both" — it silently falls
through to outbound only. Run two BFS passes (inbound + outbound)
and merge results, matching how handle_trace_call_path works.
Also fix use-after-free: use strcpy for BFS node names that get
freed before JSON serialization.
