Deep --no-ff merge of upstream DeusData/main (597 commits, merge-base 988d975)
into fork api-consolidation (HEAD 3018b22), producing a superset that combines
upstream breadth with the fork's surviving depth.
Upstream additions integrated: semantic/embedding search (vendored/nomic),
minhash/SimHash, cross-repo + cross-file LSP + route + pkgmap passes, artifact
import/export, path_alias, git_context, hook_augment, Windows support
(compat_fs/compat_thread), the UI layer, and ~140 expanded tree-sitter grammars.
Fork depth retained: PageRank/LinkRank ranking (src/pagerank/; sort_by=
relevance/linkrank/degree/calls), dependency indexing (src/depindex/;
CBM_INDEX_MODE=DEP), token reduction, and HTTP-endpoint discovery
(src/pipeline/httplink.c, pass_httplinks.c).
Conflict resolution (27 regions across the 39-file intersection):
- src/mcp/mcp.c: kept the fork superset handler pattern (resolve_project_store +
expand_project_param + inject_context_once) as the single path; UNIONed the
tool dispatch table; emit status:"indexed", excluded[], cross_service trace.
- src/store/store.c: upstream base + fork pagerank/linkrank/node_degree tables
and sort_by plumbing; cbm_store_node_degree now excludes INHERITS
(WHERE type != 'INHERITS'); removed the duplicate cbm_store_get_db.
- src/pipeline/pipeline.c: took upstream run_extraction_phase; grafted the fork
passes (httplinks, normalize, flush_store dump branch); call
cbm_pipeline_set_committed_counts before cbm_gbuf_dump_to_sqlite.
- internal/cbm/lang_specs.c: 158 entries combining the fork section_node_types
with upstream ts_factory/embedded_imports; wired 13 orphaned type-arrays.
- src/depindex/depindex.h: renamed CBM_PATH_MAX -> CBM_DEP_PATH_MAX (it collided
with constants.h enum CBM_PATH_MAX=1024) + 11 call sites in depindex.c.
- Makefile.cbm: upstream grammar/nomic units + fork CGO targets; added
httplink.c + pass_httplinks.c to PIPELINE_SRCS; removed mongoose NOSAN
machinery (mongoose was deleted upstream).
- httplinks restored (the merge silently deleted them because the fork had left
them unchanged since the base): httplink.c/.h, pass_httplinks.c,
tests/test_httplink.c; registered RUN_SUITE(httplink) in tests/test_main.c.
- CBM_INDEX_MODE enum unified to FULL=0, MODERATE=1, FAST=2, DEP=3.
Completeness (dual-parent check): 0 upstream files dropped from the merge tree;
6 fork files dropped, all intentional (upstream-superseded): CHANGELOG.md,
vendored/mimalloc/src/{arena-abandon,segment-map,segment}.c,
vendored/mongoose/{mongoose.c,mongoose.h}. Fork features (pagerank, depindex,
token_reduction, httplink) verified present and registered in tests/test_main.c.
src/pipeline/pass_parallel.c has an identical function set to upstream (49=49,
none dropped); the large line-deltas vs upstream are within-function divergence
of the fork extractor, not dropped functions.
Gate (run manually; hooks skip on merge commits): build green
(make -f Makefile.cbm cbm, exit 0); full C test suite 5980 passed / 2 failed /
0 crashes. The fork's pre-existing ASan crash in build_project_list_error_srv is
fixed by upstream's mcp.c rewrite, so the merge is strictly better than the fork
baseline on the crash axis.
Known issue, tracked separately (NOT a merge defect): 2 nondeterministic
failures in tests/test_incremental.c — incr_db_deleted_recovery:772 (diff_pct)
and incr_accuracy_vs_full:803 (full_nodes vs incr_nodes). Cause: the fork's
extractor (internal/cbm/extract_*.c) and parallel-resolve Route-node creation
are not thread-safe under the concurrent use of upstream's parallel pipeline
(run_parallel_pipeline), producing nondeterministic route/infra HANDLES edges
(~234k edge swing between runs). Sequential extraction (CBM_WORKERS=1) is
deterministic and correct. Root cause, bisect, and repro are documented in
notes/2026-06-24-fork-vs-upstream-divergence-assessment.md (section 8f); the fix
is deferred to a dedicated thread-safety audit (the make test-tsan target is
currently a stub and needs wiring).
10 KiB
Contributing to codebase-memory-mcp
Contributions are welcome. This guide covers setup, testing, and PR guidelines.
Important
: This project is a pure C binary (rewritten from Go in v0.5.0). Please submit C code, not Go. Go PRs may be ported but cannot be merged directly.
Build from Source
Prerequisites: C compiler (gcc or clang), make, zlib, Git. Optional: Node.js 22+ (for graph UI).
git clone https://github.com/DeusData/codebase-memory-mcp.git
cd codebase-memory-mcp
git config core.hooksPath scripts/hooks # activates pre-commit security checks
scripts/build.sh
macOS: xcode-select --install provides clang.
Linux: sudo apt install build-essential zlib1g-dev (Debian/Ubuntu) or sudo dnf install gcc zlib-devel (Fedora).
The binary is output to build/c/codebase-memory-mcp.
Run Tests
scripts/test.sh
This builds with ASan + UBSan and runs all tests (~2040 cases). Key test files:
tests/test_pipeline.c— pipeline integration teststests/test_httplink.c— HTTP route extraction and linkingtests/test_mcp.c— MCP protocol and tool handler teststests/test_store_*.c— SQLite graph store tests
Run C Server Tests
The MCP server core is written in C and has its own test suite under tests/:
make -f Makefile.cbm test # full suite with ASan + UBSan
make -f Makefile.cbm test-leak # heap leak check (see below)
make -f Makefile.cbm test-analyze # Clang static analyzer (requires clang, not gcc)
Memory leak detection:
On macOS, test-leak builds a sanitizer-free binary (test-runner-nosan) and runs Apple's
leaks --atExit on it. ASan replaces malloc, so the standard test-runner cannot be inspected
by leaks — the separate nosan build is required.
On Linux, test-leak runs the regular test-runner with ASAN_OPTIONS=detect_leaks=1 to
activate LSan.
In both cases the full report is written to build/c/leak-report.txt. A clean run ends with:
Process NNNNN: 0 leaks for 0 total leaked bytes.
Run Linter
scripts/lint.sh
Runs clang-tidy, cppcheck, and clang-format. All must pass before committing (also enforced by pre-commit hook).
Run Security Audit
make -f Makefile.cbm security
Runs 8 security layers: static allow-list audit, binary string scan, UI audit, install audit, network egress test, MCP robustness (fuzz), vendored dependency integrity, and frontend integrity.
Project Structure
src/
foundation/ Arena allocator, hash table, string utils, platform compat
store/ SQLite graph storage (WAL mode, FTS5)
cypher/ Cypher query → SQL translation
mcp/ MCP server (JSON-RPC 2.0 over stdio, 14 tools)
pipeline/ Multi-pass indexing pipeline
pass_*.c Individual pipeline passes (definitions, calls, usages, etc.)
httplink.c HTTP route extraction (Go/Express/Laravel/Ktor/Python)
discover/ File discovery with gitignore support
watcher/ Git-based background auto-sync
cli/ CLI subcommands (install, update, uninstall, config)
ui/ Graph visualization HTTP server (first-party httpd)
internal/cbm/ Tree-sitter AST extraction (64 languages, vendored C grammars)
vendored/ sqlite3, yyjson, mimalloc, xxhash, tre, nomic
graph-ui/ React/Three.js frontend for graph visualization
scripts/ Build, test, lint, security audit scripts
tests/ All C test files
Adding or Fixing Language Support
Language support is split between two layers:
- Tree-sitter extraction (
internal/cbm/): Grammar loading, AST node type configuration inlang_specs.c, function/call/import extraction inextract_*.c - Pipeline passes (
src/pipeline/): Call resolution, usage tracking, HTTP route linking
Workflow for language fixes:
- Check the language spec in
internal/cbm/lang_specs.c - Use regression tests to verify extraction:
tests/test_extraction.c - Check parity tests:
internal/cbm/regression_test.go(legacy, being migrated) - Add a test case in
tests/test_pipeline.cfor integration-level fixes - Verify with a real open-source repo
Infrastructure Languages (Infra-Pass Pattern)
Languages like Dockerfile, docker-compose, Kubernetes manifests, and Kustomize do not require a new tree-sitter grammar. Instead they follow an infra-pass pattern, reusing the existing tree-sitter YAML grammar where applicable:
- Detection helpers in
src/pipeline/pass_infrascan.c— functions likecbm_is_dockerfile(),cbm_is_k8s_manifest(),cbm_is_kustomize_file()identify files by name and/or content heuristics (e.g., presence ofapiVersion:). - Custom extractors in
internal/cbm/extract_k8s.c— tree-sitter-based parsers that walk the YAML AST (using the tree-sitter YAML grammar) and populateCBMFileResultwith imports and definitions. - Pipeline pass (
pass_k8s.c,pass_infrascan.c) — calls the extractor and emits graph nodes/edges. K8s manifests emitResourcenodes; Kustomize files emitModulenodes withIMPORTSedges to referenced resource files.
When adding a new infrastructure language:
- Add a detection helper (
cbm_is_<lang>_file()) inpass_infrascan.cor a newpass_<lang>.c. - Add the
CBM_LANG_<LANG>enum value ininternal/cbm/cbm.hand a row in the language table inlang_specs.c. - Write a custom extractor that returns
CBMFileResult*— do not add a tree-sitter grammar. - Register the pass in
pipeline.c. - Add tests in
tests/test_pipeline.cfollowing theTEST(infra_is_dockerfile)andTEST(k8s_extract_manifest)patterns.
Commit Format
Use conventional commits: type(scope): description
| Type | When to use |
|---|---|
feat |
New feature or capability |
fix |
Bug fix |
test |
Adding or updating tests |
refactor |
Code change that neither fixes a bug nor adds a feature |
perf |
Performance improvement |
docs |
Documentation only |
chore |
Build scripts, CI, dependency updates |
Examples: fix(store): set busy_timeout before WAL, feat(cli): add --progress flag
Pull Request Guidelines
Before You Write Code
- Open an issue first — always. Every PR must reference a tracking issue (
Fixes #NorCloses #N). Describe what you want to change and why. Wait for maintainer feedback before implementing. PRs without a prior issue discussion will be closed. - Bug fixes and test additions are the exception — these are welcome without prior discussion, as long as they're focused.
What Requires Explicit Maintainer Approval
The following changes will not be merged without prior design discussion in an issue:
- API surface changes — adding, removing, renaming, or changing defaults of MCP tools
- New pipeline passes or indexing algorithms — anything that changes what gets extracted or how
- Build system / Makefile changes — beyond trivial fixes
- Project configuration — CLAUDE.md, skill files, .mcp.json, CI workflows
- New dependencies — vendored or otherwise
- Breaking changes of any kind
If in doubt, open an issue and ask.
PR Scope and Size
- One issue per PR. Each PR must address exactly one bug, one feature, or one refactor. Do not bundle multiple fixes or feature additions into a single PR. Kitchen-sink PRs will be closed with a request to split.
- Keep PRs small. A good PR is under 500 lines. If your change is larger, split it into reviewable increments that each stand on their own.
- Don't mix features with fixes. If you find a bug while implementing a feature, submit the bug fix as a separate PR.
Code Requirements
- C code only — this project was rewritten from Go to pure C in v0.5.0. Go PRs will be acknowledged and potentially ported, but cannot be merged directly.
- Include tests for new functionality
- Run
scripts/test.shandscripts/lint.shbefore submitting - Keep PRs focused — avoid unrelated reformatting or refactoring
Security
We take security seriously. All PRs go through:
- Manual security review (dangerous calls, network access, file writes, prompt injection)
- Automated 8-layer security audit in CI
- Vendored dependency integrity checks
If you add a new system(), popen(), fork(), or network call, it must be justified and added to scripts/security-allowlist.txt.
Good First Issues
Check issues labeled good first issue for beginner-friendly tasks with clear scope and guidance.
License and sign-off (DCO) — required on every commit
All contributions are licensed under the project's MIT License
(inbound = outbound). To make that explicit and permanent, this project
uses the Developer Certificate of Origin 1.1 — the same mechanism
as the Linux kernel: every commit must carry a Signed-off-by trailer
matching the commit author.
git commit -s # adds: Signed-off-by: Your Name <you@example.com>
Adding a Signed-off-by line to a commit constitutes your certification
of the Developer Certificate of Origin 1.1 — in full, all four
clauses — for that contribution. The sign-off must match the commit's
author name and email (enforced by CI). In short: you certify that you
wrote the change or otherwise have the right to submit it under the MIT
license, and that you understand the contribution and your sign-off are
public and permanent.
(Independently of the DCO, submitting a contribution to this repository is also subject to GitHub's Terms of Service §D.6, under which contributions are licensed inbound = outbound — i.e., under this repository's MIT license.)
Enforcement is strict and automated:
- CI rejects every push and pull request containing an unsigned commit
(
scripts/check-dco.sh). - Install the local hook so unsigned commits are rejected at commit time:
scripts/install-git-hooks.sh
Forgot to sign? git commit --amend -s fixes the last commit;
git rebase --signoff <base> fixes a whole branch.