Commit Graph
60 Commits
Author SHA1 Message Date
Andrew Hundt 3fd21f8a23 mcp: fix 3 dep search bugs found by binary dogfooding
Bug 1: resolve_store opens wrong DB for dep projects
  Root cause: resolve_store("myapp.dep.pandas") opens myapp.dep.pandas.db
  instead of myapp.db where deps actually live (same-db design).
  Fix: parent_project_for_db() strips .dep.* suffix to find parent DB.

Bug 2: store.c prefix match was actually exact match
  Root cause: cbm_store_search with params->project set (non-exact mode)
  used "n.project = ?" — exact match, not prefix. Deps invisible.
  Fix: SQL now uses "(n.project = ? OR n.project LIKE ?||'.%')" for
  prefix mode, so search_graph(project="myapp") returns project + deps.

Bug 3: cbm_is_dep_project fails for cross-project deps
  Root cause: Early return when session_project doesn't match prefix.
  "otherapp.dep.pandas" with session "myapp" → false (should be true).
  Fix: Fall through to generic .dep. strstr check when session prefix
  doesn't match. Any project containing ".dep." is a dependency.

Bug 4: Package name extraction used wrong offset
  Root cause: Used strlen(session_project) as offset into project name,
  but session_project is CWD-detected, not the indexed project.
  Fix: Use strstr(project, ".dep.") to find separator position directly.

Binary verification (all confirmed working):
- search_graph(project="myapp") → 18 results (9 project + 9 dep)
- source:"project" vs source:"dependency" correctly tagged
- package:"testlib" correctly extracted
- Multiple deps in one index_dependencies call works

Tests: 2173 passing (updated test_depindex cross-project assertion).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 23:51:34 -04:00
Andrew Hundt f6f0767748 mcp: fix resource error double-wrapping + add 6 JSON-RPC structure e2e tests
Root cause: handle_resources_read returned a pre-formatted JSON-RPC error
(via cbm_jsonrpc_format_error), but cbm_mcp_server_handle wrapped it again
in cbm_jsonrpc_format_response. Result: {result: {jsonrpc, id:0, error: {...}}}
instead of the correct {error: {...}}.

Fix: handle_resources_read now takes req_id + err_out params. On error, sets
*err_out to a properly-formatted JSON-RPC error with the correct request id.
The dispatch code returns err_out directly, bypassing the result wrapper.
On success, returns raw result JSON for normal wrapping.

Also: resolve_resource_store() opens the session project DB on demand so
resources work even before any tool call.

New tests (6):
- resource_error_is_top_level_not_nested_in_result: verifies error at top
  level with correct request id (the exact bug that was found)
- resource_error_missing_uri_is_top_level: same for missing uri
- resource_error_no_params_is_top_level: same for no params
- resource_success_has_result_not_error: complement — success has "result"
- resource_schema_returns_real_data_when_indexed: schema has node_labels
- resource_status_returns_not_indexed_when_no_store: fresh server status

Total: 2173 tests passing.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 23:18:35 -04:00
Andrew Hundt 462f064e72 mcp: fix resources/read returning empty {} — orphan content_obj never attached to doc root
Bug: handle_resources_read created content_obj = yyjson_mut_obj(doc) and passed
it to build_resource_{schema,architecture,status}, but content_obj was never
added to the document root. yy_doc_to_str(doc) serialized the empty root → "{}".

Fix: pass root directly to builders instead of the orphan content_obj. Also add
resolve_resource_store() helper that opens the session project DB on demand so
resources return data even before any tool call (resources/read can be the first
call after initialize).

Verified with real indexed codebase (22,828 nodes):
- codebase://status → {"project":"...","status":"ready","nodes":22828,"edges":50639}
- codebase://schema → {"node_labels":[{"label":"Function","count":12695},...]}
- codebase://architecture → {"total_nodes":22828,"total_edges":50639,...}

Tests: 2167 passing.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 23:02:08 -04:00
Andrew Hundt 819ab6a890 mcp: cross-reference tools ↔ resources in descriptions for AI discoverability
Per MCP best practices (modelcontextprotocol.io/specification/2025-06-18/server/tools):
tool descriptions are the primary way LLMs discover capabilities. Descriptions must
be self-sufficient — an AI reading just the tool list should know the full workflow.

Tool description changes:
- search_code_graph: now says "Read codebase://schema for available node labels
  and edge types before writing Cypher queries"
- trace_call_path: now says "Read codebase://architecture for key functions
  to start tracing from"
- get_code: now says "Get qualified_name values from search_code_graph results"
- _hidden_tools hint: now lists all 3 resource URIs with usage guidance

Resource description changes (more actionable):
- codebase://schema: lists example labels, says "Read this before writing Cypher"
- codebase://architecture: mentions PageRank, says "Read this first to understand
  codebase structure and find important entry points"
- codebase://status: lists all fields, says "Read this to check if project is indexed"

Tests: 2 new tests verify tool descriptions reference resources and _hidden_tools
hint mentions all 3 resource URIs. Total: 2167 tests passing.

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 22:45:33 -04:00
Andrew Hundt d87d408447 mcp: fix 18 vague error messages + add 16 behavioral/spec compliance tests
Error messages: Every error now includes:
- WHAT failed (the specific input that caused the error)
- HOW to fix it (actionable "hint" field with next step)
- WHERE to look (tool names, param examples, valid options)

Fixed errors (18):
- "no project loaded" (3x) → + hint:"Run index_repository..."
- "function not found" → includes searched function name + hint
- "symbol not found" → includes searched qualified_name + hint
- "query is required" → + hint with Cypher syntax example
- "function_name is required" → + hint with param example
- "qualified_name is required" → + hint with format + "Use search_code_graph"
- "pattern is required" → + hint about regex vs literal
- "repo_path is required" → + hint about absolute path
- "project_name is required" → + hint:"Use list_projects"
- "project not found" (2x) → + hint:"Run index_repository or list_projects"
- "project not found or not indexed" → + hint with both options
- "failed to create pipeline" → + hint about path/permissions
- "search failed: temp file" → + hint about /tmp disk space
- "search failed" → + hint about grep installation
- "git diff failed" → + hint about git installation
- "missing tool name" → + lists available tools + "Use tools/list"
- "unknown tool: X" → + lists available tools + "Use tools/list"

New tests (16):
- MCP spec compliance: protocol version, subscribe:false, listChanged:true,
  resources/list fields, resources/read contents array, missing uri, no params
- Client behavioral differences: resource client never gets _context (3 calls),
  legacy client gets _context only first call, empty resources:{} counts as
  support, no-initialize defaults to legacy
- Error message quality: hint field present on no-project, function-not-found
  includes name, symbol-not-found includes qn, all required-param errors have
  hints, unknown-tool lists valid options, resource -32002 is actionable

Total: 2165 tests passing.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 22:29:06 -04:00
Andrew Hundt 7adaf00851 mcp: fix 3 MCP resources spec compliance issues
Audited against https://modelcontextprotocol.io/docs/concepts/resources

F1: notifications/resources/updated → notifications/resources/list_changed
    We declared listChanged:true in server capabilities, not subscribe:true.
    list_changed is for data changes; updated is for per-resource subscriptions.

F2: Error code -32602 → -32002 for unknown resource URI
    MCP spec Error Handling section specifies -32002 for "Resource not found".
    -32602 is "Invalid params" which is wrong — the URI param is valid, the
    resource just doesn't exist.

F3: Error message now actionable — includes the bad URI and lists all 3
    valid resource URIs (codebase://schema, codebase://architecture,
    codebase://status) with hint to use resources/list.

Tests: 2149 passing (assertions updated for new error code and message).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 22:11:23 -04:00
Andrew Hundt 8b2f2a75cc mcp: add MCP resources (resources/list + resources/read) with fallback context injection
Phase 10: Replace one-shot static context injection with persistent MCP resources
for clients that support them (Claude Code, VS Code Copilot, OpenCode).

Resources exposed:
- codebase://schema — node labels and edge types with counts
- codebase://architecture — graph size, key functions by PageRank, relationship patterns
- codebase://status — index status, PageRank stats, ecosystem, dependencies

Implementation:
- resources/list returns 3 resource URIs with descriptions
- resources/read dispatches by URI to build_resource_{schema,architecture,status}
- Server advertises resources capability in initialize response (listChanged:true)
- Client capabilities.resources parsed from initialize params (client_has_resources flag)
- inject_context_once skipped when client supports resources (0 token overhead)
- notifications/resources/updated sent after index_repository, index_dependencies, autoindex
- Fallback: legacy clients without resources support still get _context injection

Tests: 8 new tests (resources_list, resources_read x4, initialize_advertises,
client_capability_parsing, fallback_injection). Total: 2149 tests passing.

Also: add .claude/ to .gitignore for local project memory.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 21:26:54 -04:00
Andrew Hundt 4834ea90b7 mcp: auto-index on first use + auto-context injection + use-after-free fix
Auto-index on first use (REQUIRE_STORE + search_graph):
  When store is NULL and session_root is a valid directory:
  1. If autoindex_active: join background thread, re-resolve store
  2. If still NULL: run cbm_pipeline_run() synchronously, then
     cbm_dep_auto_index() + cbm_pagerank_compute_with_config()
  Handles all 3 paths: CWD (detect_session), explicit path (Rule 0),
  MCP roots (future). access(session_root, F_OK) guard prevents
  triggering on non-existent paths in tests.

inject_context_once(): auto-provide architecture/schema on first response.
  First tool response gets _context header with: status, nodes, edges,
  node_labels, edge_types, ranked_nodes, pagerank_computed_at,
  detected_ecosystem. Subsequent responses only get session_project.

Fix: use-after-free in inject_context_once (ASAN crash at mcp.c:937).
  cbm_store_schema_free() freed label/type strings while yyjson still
  held borrowed pointers. Fix: yyjson_mut_obj_add_strcpy() copies strings
  into yyjson's allocator before schema is freed.

Fix: _hidden_tools count corrected from "12" to "14" (14 tools hidden).

Tests: 2 new (first_response_has_context_header, context_has_schema_info).
Total: 2141 tests passing.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 05:33:25 -04:00
Andrew Hundt c7aaef853c mcp: progressive disclosure + env var override + session_project in all handlers
Progressive disclosure: _hidden_tools entry in streamlined tool list tells AI
  which 12 tools are hidden and how to enable them (CBM_TOOL_MODE=classic env
  var or config set tool_mode classic or per-tool config set tool_<name> true).
  Hidden tools still dispatch normally — AI can call them after discovery.

Env var override: CBM_TOOL_MODE env var takes precedence over config for
  tool_mode. Enables backwards compat without needing a config store.

Session context: add session_project to trace_call_path and get_architecture
  responses. Now all major tool responses include session_project so AI always
  knows which project it's working with.

Tests: 4 new in test_tool_consolidation.c:
  - streamlined_mode_has_hidden_tools_hint
  - hidden_tools_still_dispatch
  - search_graph_has_session_project
  - index_status_has_session_project

Total: 2139 tests passing (13 new Phase 9 tests total)
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 03:54:55 -04:00
Andrew Hundt 46e15c4593 mcp: fix gaps — config-backed PageRank callers, Phase 9 test suite (9 tests)
G1: Wire cbm_pagerank_compute_with_config(store, project, srv->config) into
    handle_index_repository and autoindex_thread (2 callers in mcp.c).
    Edge weight config keys now actually used at runtime.
G5: Create tests/test_tool_consolidation.c with 9 tests covering:
    - streamlined_mode_shows_3_tools (NULL srv → 3 consolidated tools)
    - classic_mode_shows_all_15_tools (via server_handle)
    - search_code_graph_structured_dispatch (name_pattern → search_graph)
    - search_code_graph_cypher_dispatch (cypher → query_graph)
    - get_code_dispatch (→ get_code_snippet)
    - old_tool_names_still_dispatch (backwards compat)
    - project_param_path_detection (expand_project_param Rule 0)
    - unknown_tool_returns_error
    - null_tool_name_returns_error
Register suite in test_main.c + Makefile.cbm.

Total: 2135 tests passing (9 new)
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 03:50:06 -04:00
Andrew Hundt ff833362f6 mcp: Phase 9 API consolidation — 15 tools to 3 streamlined + config-based visibility
STREAMLINED_TOOLS[]: search_code_graph (merges search_graph + query_graph via cypher param),
  trace_call_path (unchanged), get_code (alias for get_code_snippet)
cbm_mcp_tools_list(srv): filters by tool_mode config (streamlined=3, classic=all 15)
  Per-tool re-enable via config set tool_<name> true
Dispatch: search_code_graph routes to handle_query_graph when cypher param present,
  otherwise handle_search_graph. get_code routes to handle_get_code_snippet.
expand_project_param Rule 0: detects paths (/, ~, ./) and converts via
  cbm_project_name_from_path(). Enables project="/path/to/repo".
Server struct: add context_injected field for Phase 9 auto-context (future).
mcp.h: forward-declare cbm_mcp_server_t at top, cbm_mcp_tools_list takes srv param.
Tests: updated for streamlined mode (3 tools default, old names hidden).

Total: 2126 tests passing
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 03:26:13 -04:00
Andrew Hundt cecf6278cb mcp: apply Phase 8.5 refinements to merged branch
mcp.c: key_functions (top 10 by PageRank) in get_architecture response
mcp.c: pagerank stats (ranked_nodes, computed_at) in index_status response
mcp.c: conditional in_degree/out_degree (only when PageRank not computed)
pagerank.c: cbm_pagerank_compute_with_config() for config-backed edge weights
pagerank.h: 9 CBM_CONFIG_EDGE_WEIGHT_* config key constants + forward decl
cli.c: cbm_config_get_double() for double config values
test_pagerank.c: 7 Phase 8.5 tests (key_functions, config, stats, streamlining)

Total: 2126 tests passing (7 new over merged baseline of 2119)
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-22 03:09:11 -04:00
Andrew Hundt 1885d13802 merge pagerank-ranking: add PageRank node + LinkRank edge ranking
Merge pagerank-ranking branch into combined token-reduction +
reference-api-indexing branch. Resolves conflicts in mcp.c
(sort_by coexists with compact/summary/token-reduction params),
Makefile.cbm (restore TEST_TOKEN_REDUCTION_SRCS lost in merge).

All 2119 tests pass (35 PageRank + 22 token-reduction + 2062 existing).
2026-03-21 23:29:31 -04:00
Andrew Hundt ef76a7c125 pagerank: add PageRank node ranking + LinkRank edge ranking (Phase 8)
Implement PageRank (power iteration, d=0.85, weighted edges) and
LinkRank (Kim et al. 2010) to rank nodes by structural importance
and edges by traversal probability. References: aider repomap,
NetworkX, RepoGraph (peer-reviewed).

New files:
  src/pagerank/pagerank.{h,c} — algorithm + API (~380 lines)
  tests/test_pagerank.c — 35 tests (core, edge cases, LinkRank)

Schema: pagerank + linkrank tables with indexes (store.c)
Store: conditional LEFT JOIN pagerank on search, sort_by dispatch
  (relevance/name/degree), pr_rank in BFS visited, lr_rank in edges
MCP: sort_by param on search_graph, pagerank in response JSON,
  pagerank scores in trace_call_path callees/callers
Index: compute after pipeline in all 3 paths (handler, watcher,
  autoindex) + index_dependencies

Edge weights: CALLS=1.0 DEFINES_METHOD=0.8 DEFINES=0.5 IMPORTS=0.3
  USAGE=0.2 CONFIGURES=0.1 HTTP_CALLS=0.5 ASYNC_CALLS=0.8

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-21 23:22:07 -04:00
Andrew Hundt 61394569a3 mcp: close remaining gaps — trace/snippet source tagging, cross-edges
Gap 3 (trace boundary tagging): trace_call_path now tags each caller
and callee with source:"project"|"dependency" and read_only:true for
dep nodes. Uses cbm_is_dep_project() for consistent tagging.

Gap 4 (snippet provenance): build_snippet_response adds source and
read_only fields so get_code_snippet results indicate whether code
is from the project or a dependency.

Cross-edges: cbm_dep_link_cross_edges implemented — searches project
Variable nodes, looks for matching Module nodes in dep projects
(project.dep.%), creates IMPORTS edges to link them. Enables
trace_call_path to follow imports across project/dep boundary.

Gap 1 (watcher dep re-index) was already done in prior commit.

Files: src/mcp/mcp.c (trace + snippet tagging),
src/depindex/depindex.c (cross-edge implementation)

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-21 20:17:09 -04:00
Andrew Hundt b3f209370e mcp: close remaining gaps — trace/snippet source tagging, cross-edges
Gap 3 (trace boundary tagging): trace_call_path now tags each caller
and callee with source:"project"|"dependency" and read_only:true for
dep nodes. Uses cbm_is_dep_project() for consistent tagging.

Gap 4 (snippet provenance): build_snippet_response adds source and
read_only fields so get_code_snippet results indicate whether code
is from the project or a dependency.

Cross-edges: cbm_dep_link_cross_edges implemented — searches project
Variable nodes, looks for matching Module nodes in dep projects
(project.dep.%), creates IMPORTS edges to link them. Enables
trace_call_path to follow imports across project/dep boundary.

Gap 1 (watcher dep re-index) was already done in prior commit.

Files: src/mcp/mcp.c (trace + snippet tagging),
src/depindex/depindex.c (cross-edge implementation)

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-21 20:07:57 -04:00
Andrew Hundt 719d90841e mcp: fix MCP connection hang from stale/corrupt .db files in cache
Root cause: handle_list_projects opens every .db file in
~/.cache/codebase-memory-mcp/ via cbm_store_open_path (which runs
CREATE TABLE IF NOT EXISTS, modifying foreign databases). With 62
stale .db files (1.3GB) including a corrupt 223MB "..db" (empty
project name), the server hung during Claude Code health checks.

Fixes:
- Add validate_cbm_db(): read-only SQLite validation with magic byte
  check + 'nodes' table schema check + 1s busy_timeout. Never modifies
  foreign databases. Logs actionable warnings on skip.
- Guard detect_session() against empty/dot project names that produce
  the corrupt "..db" filename
- Skip "..db" and ".db" filenames in handle_list_projects
- Skip empty/dot project names after filename-to-name extraction
- Force unbuffered stdin/stdout via setvbuf for MCP stdio protocol
- Add #include <sqlite3.h> for read-only validation queries

Files: src/main.c (setvbuf), src/mcp/mcp.c (validate_cbm_db,
detect_session guard, list_projects guards, sqlite3.h include)

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-21 20:01:17 -04:00
Andrew Hundt d5511d7e88 mcp: fix MCP connection hang from stale/corrupt .db files in cache
Root cause: handle_list_projects opens every .db file in
~/.cache/codebase-memory-mcp/ via cbm_store_open_path (which runs
CREATE TABLE IF NOT EXISTS, modifying foreign databases). With 62
stale .db files (1.3GB) including a corrupt 223MB "..db" (empty
project name), the server hung during Claude Code health checks.

Fixes:
- Add validate_cbm_db(): read-only SQLite validation with magic byte
  check + 'nodes' table schema check + 1s busy_timeout. Never modifies
  foreign databases. Logs actionable warnings on skip.
- Guard detect_session() against empty/dot project names that produce
  the corrupt "..db" filename
- Skip "..db" and ".db" filenames in handle_list_projects
- Skip empty/dot project names after filename-to-name extraction
- Force unbuffered stdin/stdout via setvbuf for MCP stdio protocol
- Add #include <sqlite3.h> for read-only validation queries

Files: src/main.c (setvbuf), src/mcp/mcp.c (validate_cbm_db,
detect_session guard, list_projects guards, sqlite3.h include)

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-21 19:26:10 -04:00
Andrew Hundt e462901f9a merge reference-api-indexing: expand_project_param, result tagging, dep re-index
Merge latest reference-api-indexing (bd09623) with gap implementations:

Conflicts resolved (2 in mcp.c search_graph handler):
1. Param parsing: keep reduce-token-usage compact/summary/mode params,
   use fill_project_params() from reference-api-indexing for smart project
2. Result loop: keep reduce-token-usage compact mode + pagination_hint,
   add unconditional source/package/read_only tagging from ref-api-indexing

Superset features in merged handler:
- expand_project_param (self/dep/deps/glob/prefix resolution)
- compact mode (omit redundant name when suffix of QN)
- summary mode (aggregate counts by label/file)
- pagination_hint with offset/limit guidance
- session_project in all responses
- source:"project"/"dependency" on every result
- package + read_only:true on dep results
- dep auto-reindex in all 3 paths (handler, watcher, autoindex)
- index_status reports dependencies[] + detected_ecosystem

All 2081 tests pass (2042 base + 22 token-reduction + 17 dep-indexing).
2026-03-21 00:04:48 -04:00
Andrew Hundt 7530a30575 mcp: expand_project_param, result tagging, dep auto-reindex in all 3 paths
expand_project_param() (mcp.c:764-840):
- "self" → session project exact match
- "dep"/"deps" → session.dep prefix match
- "dep.pandas" → session.dep.pandas prefix
- "myapp.pandas" → myapp.dep.pandas (auto-insert .dep.)
- Glob "*" → SQL LIKE with % substitution
- fill_project_params() helper sets cbm_search_params_t fields

search_graph result tagging (mcp.c:930-960):
- Every result tagged source:"project" or source:"dependency"
- Dep results get package name + read_only:true
- session_project added to response for AI project name awareness
- Uses cbm_is_dep_project() with session context for precision

handle_index_status (mcp.c:1046-1100):
- Reports dependencies[] array with package names and node counts
- Reports detected_ecosystem from project root marker files
- session_project in response

Dep auto-reindex in all 3 re-index paths:
- handle_index_repository (mcp.c:1472): cbm_dep_auto_index after dump
- watcher_index_fn (main.c:86-96): cbm_dep_auto_index after dump
- autoindex_thread (mcp.c:2496-2501): cbm_dep_auto_index after dump
All use DRY cbm_dep_auto_index() with CBM_DEFAULT_AUTO_DEP_LIMIT

cbm_mcp_server_set_session_project() added (mcp.h:128, mcp.c:526)

Fix: yyjson_mut_obj_add_strcpy for dep package names from search results
(heap-use-after-free when cbm_store_search_free frees borrowed strings)

Fix: db_project selection when session_project is empty (integration test
integ_mcp_delete_project was failing — resolve_store got NULL instead of
project name after expand_project_param)

Tests: 29 depindex tests (2059 total, all passing)
- test_search_results_have_source_field: project results tagged
- test_search_dep_results_tagged_dependency: dep results have package+read_only
- test_search_response_has_session_project: session_project in response
- test_index_status_shows_deps: dependencies[] in index_status response

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-21 00:00:24 -04:00
Andrew Hundt 432d06f5ba merge reference-api-indexing: dep indexing + token reduction combined
Merge reference-api-indexing into token-reduction-and-reference-indexing.
Three conflicts resolved with superset approach:

1. search_graph schema (mcp.c:272): Keep reduce-token-usage's mode/compact/
   pagination_hint params AND reference-api-indexing's include_dependencies
2. search_code schema (mcp.c:349): Keep reduce-token-usage's configurable
   limit description AND reference-api-indexing's case_sensitive param
3. store.c ORDER BY (1866): Keep reference-api-indexing's project-first
   sort for mixed project+dep results (superset of stable name,id sort)

SKILL.md: took reduce-token-usage version (complete token reduction docs).

All 2077 tests pass (2042 base + 22 token-reduction + 13 dep-indexing).
2026-03-20 23:35:55 -04:00
Andrew Hundt 54d68382a5 depindex: implement dep indexing pipeline, smart project param, search_code fix
New module src/depindex/ with package resolution (uv/cargo/npm/bun),
ecosystem detection, dep discovery from indexed graph, auto-index helper,
and cross-boundary edge creation stub. Dependencies stored in same db
with {project}.dep.{package} naming convention.

Pipeline changes:
- Add CBM_MODE_DEP index mode (keeps vendor/, .d.ts for dep source)
- Add cbm_pipeline_set_project_name() to override auto-derived name
- Add cbm_pipeline_set_flush_store() for upsert vs fresh dump
- Conditional dump/flush at pipeline.c:646

Store changes:
- Add project_pattern (LIKE) and project_exact fields to cbm_search_params_t
- Support LIKE queries for glob-style project filtering
- Add project-first ORDER BY for mixed project+dep results
- Stable pagination via ORDER BY name, id

MCP changes:
- Replace index_dependencies stub with full implementation
  (source_paths[] primary interface, package_manager optional shortcut)
- Fix detect_session() to use cbm_project_name_from_path (Bug #12)
- REQUIRE_STORE error now includes actionable hint field
- search_code: fix -m limit exhaustion (limit*50 min 500 vs limit*3)
- search_code: add case_sensitive param (default false = case-insensitive)

DRY improvements:
- CBM_MANIFEST_FILES shared list in depindex.h used by pass_configlink.c
  and dep discovery (adds pyproject.toml, setup.py, Pipfile)
- Remove package.json and composer.json from IGNORED_JSON_FILES
  (needed by pass_configlink and dep auto-discovery)

Tests: 25 depindex tests (2055 total, all passing)
- Package manager parse/str roundtrip, dep naming, is_dep detection
- Ecosystem detection (python/rust/none), manifest path matching
- npm resolution with fixture, pipeline set_project_name
- MCP tool validation, AI grounding, dep reindex replaces

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 23:27:32 -04:00
Andrew Hundt 7626a5b14f mcp.c: OOM-safe guards + notes/ documentation with mermaid diagrams
OOM fixes (applied to both feature branches):
1. trace_call_path: calloc for seen_out/seen_in dedup arrays gracefully
   degrades on OOM — skips dedup instead of NULL-dereference crash
2. build_snippet_response: head_tail combined buffer malloc falls back
   to head-only output on OOM instead of NULL snprintf

Documentation (notes/ folder):
- notes/token-reduction-changes.md: 8 RTK-inspired strategies, config
  system, real-world results, mermaid architecture diagram
- notes/reference-api-indexing-changes.md: 7-layer AI grounding defense,
  QN prefix format, deferred work, mermaid flow diagram
- notes/merged-branch-changes.md: branch lineage gitGraph, combined
  architecture diagram, snippet mode decision flow, token reduction
  pipeline per tool, test coverage, merge conflict resolution

Tests: 2064 passed, 0 failed
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 14:30:46 -04:00
Andrew Hundt 0e0b941811 mcp.c: add include_dependencies to search_graph tool schema
The include_dependencies parameter was parsed in the handler (line 776)
but not declared in the TOOLS[] schema JSON. This meant LLMs could not
discover the parameter from tool descriptions — it was silently accepted
but undiscoverable.

Added include_dependencies boolean property with description to the
search_graph tool schema, matching the merged branch's schema.

Tests: 2042 passed, 0 failed
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 14:30:35 -04:00
Andrew Hundt 40387a0fcc mcp.c: clarify code comments for token metadata, pagination, head_tail
- Token metadata comment: explain _result_bytes (byte length of inner
  JSON text) and _est_tokens (bytes/4, same heuristic as RTK's
  estimate_tokens function in tracking.rs)

- Pagination hint: add comment explaining the pagination_hint field
  purpose (tells caller how to get next page)

- Head/tail mode: document the 60/40 split rationale (60% head captures
  signature/setup, 40% tail captures return/cleanup; middle
  implementation detail is what gets omitted)

Tests: 2064 passed, 0 failed
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 13:09:10 -04:00
Andrew Hundt e8f2453104 mcp.c: fix 6 issues found in code review
1. Remove misleading "Set limit=0 for no cap" from search_graph schema
   description — store.c maps limit=0 to 500K, not truly unlimited

2. Eliminate redundant is_summary_early variable — merge into single
   is_summary bool computed once before the search query

3. Add bounds-check comment for summary mode labels[64] array explaining
   the cap matches CBM's ~12 label types with margin

4. Replace %zu with %lu + (unsigned long) cast in query_graph truncation
   snprintf for portability (existing codebase avoids %zu)

5. Add include_dependencies parameter to search_graph tool schema so
   LLMs can discover the opt-in dependency inclusion feature

6. Remove hardcoded "default":50 from search_code JSON schema — actual
   default comes from config key search_limit at runtime

Tests: 2064 passed, 0 failed
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 01:35:10 -04:00
Andrew Hundt 94a59e718d Merge branch 'reduce-token-usage' into token-reduction-and-reference-indexing 2026-03-20 00:54:21 -04:00
Andrew Hundt d3020fdb13 mcp: config-backed defaults + magic-number-free tool descriptions
All token reduction defaults are now configurable at runtime via the
config system (cbm_config_get_int). Config keys:
- search_limit: default result limit for search_graph/search_code
- snippet_max_lines: default max source lines for get_code_snippet
- trace_max_results: default max BFS nodes for trace_call_path
- query_max_output_bytes: default output cap for query_graph

Tool schema descriptions no longer contain hardcoded numbers — they
reference config keys instead, so changing a default won't make the
description misleading.

Tool descriptions now include comprehensive AI guidance:
- search_graph: how to paginate (offset+limit), mode=summary for overview
- query_graph: max_output_bytes=0 for unlimited, LIMIT in Cypher
- get_code_snippet: mode=signature for API lookup, mode=head_tail for
  preserving return/cleanup, max_lines=0 for full source
- trace_call_path: max_results for exhaustive traces, callees_total
  for truncation awareness
- All tools: config key names documented for runtime override

Tests: 2052 passed, 0 failed
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 00:54:00 -04:00
Andrew Hundt 97e0d0d231 mcp: fix summary mode aggregation limit + add pagination hint
Summary mode bug: by_label only counted 50 results (the default limit)
instead of all symbols. Fix: override effective_limit to 10000 when
mode=summary so aggregation covers representative sample.

Pagination: when has_more=true, add pagination_hint field:
"Use offset:50 and limit:50 for next page (13818 total)"
This guides LLMs to use offset/limit for progressive exploration.

Verified on RTK codebase (45,388 symbols):
- Summary mode: 1,317 bytes with accurate label counts
- Default search: pagination_hint present when has_more=true
- All 2064 tests pass

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 00:35:56 -04:00
Andrew Hundt 70490ec09d mcp: fix summary mode aggregation limit + add pagination hint
Summary mode bug: by_label only counted 50 results (the default limit)
instead of all symbols. Fix: override effective_limit to 10000 when
mode=summary so aggregation covers representative sample.

Pagination: when has_more=true, add pagination_hint field:
"Use offset:50 and limit:50 for next page (13818 total)"
This guides LLMs to use offset/limit for progressive exploration.

Verified on RTK codebase (45,388 symbols):
- Summary mode: 1,317 bytes with accurate label counts
- Default search: pagination_hint present when has_more=true
- All 2064 tests pass

Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 00:35:44 -04:00
Andrew Hundt c1b0ab0602 merge: combine token reduction (reduce-token-usage) + reference API indexing (reference-api-indexing)
Resolve 5 conflicts across 3 files:
- Makefile.cbm: combine TEST_TOKEN_REDUCTION_SRCS + TEST_DEPINDEX_SRCS in ALL_TEST_SRCS
- src/mcp/mcp.c: merge compact/summary/search_mode params with include_dependencies param
  in handle_search_graph; add source:"project" field in full-mode results loop at line ~912
- tests/test_main.c: register both suite_token_reduction and suite_depindex

Combined capabilities on merged branch:
- 8 RTK-inspired token reduction strategies (CBM_DEFAULT_* constants, 3 snippet modes,
  compact/summary search, BFS dedup, query output cap, _result_bytes/_est_tokens metadata)
- index_dependencies MCP tool with AI grounding (7-layer defense: separate _deps.db,
  include_dependencies=false default, dep.{mgr}.{pkg} QN prefix, source field, external property)

Tests: 2064 passed, 0 failed (22 token_reduction + 12 depindex + 2030 existing)
2026-03-20 00:13:20 -04:00
Andrew Hundt 0693d08edd mcp: add index_dependencies tool + AI grounding infrastructure
Register index_dependencies MCP tool for indexing dependency/library
source code into a separate dependency graph. Dependencies are stored
in {project}_deps.db (separate from project.db) and are NOT included
in queries unless include_dependencies=true is passed.

AI grounding safeguards (7-layer defense):
1. Storage: separate _deps.db not touched by index_repository
2. Query default: include_dependencies=false (deps excluded by default)
3. QN prefix: dep.{mgr}.{package}.{symbol} convention documented
4. Response field: "source":"project" / "source":"dependency" labels
5. Properties: "external":true on dependency nodes
6. Tool description: explicitly states "SEPARATE dependency graph"
7. Boundary markers: trace_call_path shows project→dep edges

Current state:
- Tool registered with full parameter validation (project, package_manager required)
- include_dependencies param added to search_graph with source field
- Handler returns structured "not_yet_implemented" status
- Full dep resolution pipeline (depindex module) designed but deferred

Tests: 12 new tests in test_depindex.c, all passing.
       All 2042 existing tests pass with zero regressions.

Next: implement src/depindex/ module for actual package resolution
      (uv/cargo/npm/bun), dependency file discovery, and pipeline
      integration per the plan in plans/serialized-pondering-puppy.md.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 00:08:16 -04:00
Andrew Hundt f4dc4c163f mcp: reduce token consumption via RTK-inspired filtering strategies
Apply 8 token reduction techniques inspired by RTK (Rust Token Killer):

1. Default search limits: search_graph/search_code default limit 500K→50
   (CBM_DEFAULT_SEARCH_LIMIT constant). Callers can override explicitly.

2. Smart truncation for get_code_snippet: 3 modes (full/signature/head_tail)
   with max_lines=200 default (CBM_DEFAULT_SNIPPET_MAX_LINES). head_tail
   preserves function signature + return/cleanup code. Signature mode
   returns only API surface without reading source files.

3. Compact mode for search_graph/trace_call_path: omits redundant name
   field when it's the last segment of qualified_name.

4. Summary mode for search_graph: returns aggregated counts by label and
   file (top 20) instead of individual results. 95% token reduction.

5. Trace edge case fixes: max_results param (default 25), BFS cycle
   deduplication by node ID, candidates array for ambiguous function names,
   callees_total/callers_total counts.

6. query_graph output truncation: max_output_bytes (default 32KB) caps
   worst-case output. Does NOT change max_rows (which is a scan-limit
   that would break aggregation queries).

7. Token metadata: _result_bytes and _est_tokens in all MCP tool responses
   for LLM token awareness.

8. Stable pagination: ORDER BY name, id for deterministic pagination.

All defaults use named constants (CBM_DEFAULT_*) — no magic numbers.
CYPHER_RESULT_CEILING reduced 100K→10K as safety net.

Tests: 22 new tests in test_token_reduction.c, all passing.
       All 2060+ existing tests pass with zero regressions.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
2026-03-20 00:02:43 -04:00
Martin Vogel 0ef1b7b8c4 Release RAM after indexing, static-link Windows binary
- Call cbm_mem_collect() (mi_collect(true)) after pipeline_free in both
  index_repository and auto-index, returning mimalloc pages to OS
- Add -static to Windows linker flags so the binary is standalone
  (no libc++.dll / libunwind.dll dependency)
2026-03-19 22:17:38 +01:00
Martin Vogel 3744c6d419 Fix MCP protocolVersion: return plain string, not nested object
protocolVersion was {"version":"2024-11-05"} — MCP spec requires "2024-11-05".
Claude Code rejected the malformed response and marked the server as failed.
2026-03-19 21:20:06 +01:00
Martin Vogel 7629940dca Restore session auto-detect + auto-index + config CLI from Go
- Config store: persistent SQLite key-value in _config.db (get/set/delete/bool/int)
- Config CLI: codebase-memory-mcp config list|get|set|reset
  Keys: auto_index (default false), auto_index_limit (default 50000)
- Session detection: detect project root from CWD on MCP initialize
- Auto-index: if auto_index=true and project not yet indexed, background pipeline
- Watcher hookup: after index completes, register project for ongoing git polling
- 6 new config store tests (open/close, get/set, bool, int, delete, persistence)

Enable auto-indexing:
  codebase-memory-mcp config set auto_index true
2026-03-19 20:13:17 +01:00
Martin Vogel 0b8a456b52 Restore startup update check: notify on first tool call if newer release exists
Background thread checks api.github.com/repos/.../releases/latest on initialize,
stores one-shot notice prepended to the first tools/call response content array.
Ported from Go internal/tools/tools.go checkForUpdate + addUpdateNotice pattern.
2026-03-19 19:30:21 +01:00
442cb958f0 Add KiloCode support, fix Zed JSONC parsing (#24, #53)
- KiloCode: detect globalStorage dir, upsert mcp_settings.json, install
  global rules at ~/.kilocode/rules/codebase-memory-mcp.md
- Zed: parse JSONC (comments + trailing commas) in existing settings.json
  — fixes "Invalid JSON" error on install with existing Zed config
- 2 new tests: KiloCode detection, Zed JSONC settings parsing
- Now supports 8 agents: Claude Code, Codex, Gemini, Zed, OpenCode,
  Antigravity, Aider, KiloCode

Co-Authored-By: David34920 <179368816+David34920@users.noreply.github.com>
Co-Authored-By: Paul Cohen <200370+sonicviz@users.noreply.github.com>
Co-Authored-By: Chitral Verma <11135032+chitralverma@users.noreply.github.com>
Co-Authored-By: Noel Kurian <1851449+noelkurian@users.noreply.github.com>
Co-Authored-By: José Almeida <57680069+zeval@users.noreply.github.com>
Co-Authored-By: Vasani Harshil <147682127+harshil480@users.noreply.github.com>
2026-03-19 18:56:16 +01:00
Martin Vogel 159378ad8e Multi-agent install: auto-detect 7 coding agents, upsert configs + hooks + instructions
- Detect Claude Code, Codex CLI, Gemini CLI, Zed, OpenCode, Antigravity, Aider
- Per-agent MCP config upsert: Codex TOML, OpenCode JSON, Antigravity JSON
- Instruction file upsert with marker-based sections (no duplicates on re-install)
- Claude Code PreToolUse hook (Grep|Glob|Read -> MCP reminder)
- Gemini CLI BeforeTool hook (grep_search|read_file -> MCP reminder)
- Fix Zed MCP config: args:[""] instead of broken source:"custom" (#24)
- Rewrite skill descriptions to directive pattern (100% auto-activation)
- Keyword-rich MCP tool descriptions for Tool Search discovery
- 35 new tests (agent detection, config upsert, instructions, hooks)
- All platforms green: macOS 2022, Linux 2004+18skip, Windows cross-compile OK
2026-03-19 16:08:43 +01:00
Martin Vogel dc919b2231 Create cache directory before SQLite dump in pipeline
The dump phase writes to ~/.cache/codebase-memory-mcp/<project>.db
but the directory may not exist on fresh CI runners. Add cbm_mkdir_p()
to ensure the parent directory is created before writing. This fixes
smoke test failures where index_repository returned status "error"
because the dump phase couldn't create the database file.
2026-03-19 12:11:02 +01:00
Martin Vogel 18c23d438f Fix 12 more Windows test failures: mkstemp, temp paths, TRE config
- cbm_mkstemp: Windows implementation using _mktemp + _open
- cbm_tmpdir(): portable temp directory function (replaces /tmp/)
- Fix hardcoded /tmp/ paths in test_language, test_gitignore,
  test_store_search, test_sqlite_writer
- Enlarge remaining small path buffers for Windows %TEMP% paths
2026-03-19 02:58:03 +01:00
Martin Vogel d6ec954b9c Fix cbm_mkdtemp: enlarge all test buffers to 256 bytes
The /tmp/ → %TEMP%\ translation produces longer paths (~50 chars)
than the original templates (~25 chars). Changed all 68 test
declarations from char tmpdir[] = "..." (sized to literal) to
char tmpdir[256] with snprintf initialization.
2026-03-19 02:38:02 +01:00
Martin Vogel 6f5b77aeba Fix TRE wchar: undefine TRE_WCHAR/TRE_MULTIBYTE (ifdef checks presence) 2026-03-19 02:21:56 +01:00
Martin Vogel 5f7cd226cb Use system regex on MSYS2, TRE only for pure MinGW (CBM_USE_TRE) 2026-03-19 02:05:20 +01:00
Martin Vogel 07f82b9c54 Vendor TRE regex for Windows, fix cbm_mkdtemp buffer overflow
- Vendor TRE regex library (BSD-licensed, ~9K LOC) for Windows —
  eliminates the TODO stub that caused all regex tests to fail
- Fix cbm_mkdtemp stack-buffer-overflow: use static buffer for
  /tmp/ → %TEMP%\ path translation (Windows paths are longer)
- Wine test results: 1861 passed, 50 failed (Wine filesystem limits),
  75 skipped — remaining failures are Wine-specific, not Windows bugs
2026-03-19 01:47:37 +01:00
Martin Vogel d1632ec7e1 Fix cbm_mkdtemp for Windows, add Wine test execution to Docker
- cbm_mkdtemp: translate /tmp/ to %TEMP%\ on Windows
- Docker test-windows: cross-compile + run under Wine (without ASan,
  which doesn't work under Wine)
- Wine on PATH fix for Ubuntu noble (/usr/lib/wine/)
- Tool preflight check in Dockerfile
- 1887/1992 tests pass under Wine (52 fail: regex not available in
  pure MinGW — works on real MSYS2 which provides <regex.h>)
2026-03-19 00:59:54 +01:00
Martin Vogel fdb83f83f7 Fix Windows test compilation: mkdtemp, mkdir, setenv, pipe compat
- cbm_mkdtemp: _mktemp + _mkdir on Windows
- cbm_setenv/cbm_unsetenv: _putenv_s on Windows
- cbm_pipe: _pipe on Windows
- Replace bare POSIX calls in 12 test files with compat wrappers
- All test files now include compat.h
2026-03-18 19:19:42 +01:00
Martin Vogel 85fc74fc31 Full Windows portability: proper implementations, not stubs
- handle_processes: GetProcessMemoryInfo + GetProcessTimes (Windows),
  getrusage (POSIX) — real process metrics on both platforms
- handle_process_kill: TerminateProcess via OpenProcess (Windows),
  kill(SIGTERM) (POSIX)
- index_thread_fn: CreateProcess + WaitForSingleObject (Windows),
  fork+exec+waitpid (POSIX)
- cbm_clock_gettime: QueryPerformanceCounter (Windows) — replaces
  all bare clock_gettime calls across 6 files
- cbm_nanosleep, cbm_strcasestr, cbm_mkdir: compat shims
- CBM_TLS: _Thread_local (C11 standard, works on all compilers)
- Replace bare strndup with cbm_strndup, guard POSIX headers
- Add mingw-w64 cross-compile to Docker test infrastructure
2026-03-18 18:47:43 +01:00
Martin Vogel 26a78142d6 Fix Windows/MinGW build: TLS, strndup, sys/wait.h
- CBM_TLS: use _Thread_local (C11 standard) instead of __declspec(thread)
  which only works on MSVC, not MinGW GCC
- Replace bare strndup() with cbm_strndup() in 6 files (strndup is POSIX,
  not available on Windows)
- Guard sys/wait.h with #ifndef _WIN32 in http_server.c
2026-03-18 17:42:44 +01:00
Martin Vogel c8bdf924e5 Fix production build errors, add build step to Docker test infra
- Fix GCC -O2 warnings: calloc for cf (maybe-uninitialized),
  suppress stringop-truncation/alloc-size-larger-than (false positives),
  guard louvain against negative node_count
- Default compiler: cc on macOS, gcc on Linux/Windows (fixes gcc-14
  not found on MSYS2)
- Docker test infra now runs BOTH test (ASan) and build (-O2 -Werror)
2026-03-18 17:15:46 +01:00