index_repository rendered every non-zero pipeline code as "Pipeline
failed. Check repo_path exists and contains source files", which sent
people debugging a path that was fine (#2020). Three outcomes now say
what actually happened:
- CBM_PIPELINE_ABORT_OVER_BUDGET (#1997, #832): status "error" with
reason "over_memory_budget", previous_index "preserved", the budget and
peak RSS in MB, and a hint naming CBM_MEM_BUDGET_MB / CBM_WORKERS /
excludes. The handler logs index.abort reason=over_memory_budget. The
CLI prints the same JSON and exits 1; a supervised worker still exits 0
with the response written, so the daemon passes it through verbatim
instead of entering skip-and-continue recovery.
- CBM_PIPELINE_ABORT_PRESERVE_DB: status "aborted_previous_preserved"
(retry; the previous index still serves).
- CBM_PIPELINE_PERSIST_FAILED: status "persist_failed" (disk/permissions).
The incremental test helper goes fail-closed on error responses so a
transient pre-publication abort surfaces once at the source instead of
cascading into dozens of downstream count failures.
Test: an in-process index over 64 files at a 1 MiB budget seam returns
the named error, leaves the project DB the same size on disk, and
index_status still reports the previous node count.
Refs #2020#1997#832
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The x86-64 leg runs this lane without exclusions on purpose, to settle
which limits are architectural. It has now run, and it disproves part of
what the previous block asserted. That block claimed all seven excluded
suites "abort with stack-overflow". Five do. Two do not, and lumping
them together hid two different problems behind one rationale.
(A) stack-overflow, five suites: grammar_regression grammar_labels
pipeline lang_contract grammar_probe_e. Confirmed on BOTH arm64 and
x86-64 (CI logged 5), so it is not the aarch64 artifact an earlier
note claimed. The recursion guards bound DEPTH while the resource
exhausted is BYTES; that follow-up stands unchanged.
(B) cli: no overflow at all. On x86-64 it runs to completion, 253
passed / 5 failed, every failure in the install or activation path,
with "agent_config agent=OpenClaw op=mcp_install" above them. Green
on every other venue. MSan reported zero use-of-uninitialized-value
in it, so the exclusion costs no uninit coverage. Recorded as
undiagnosed rather than guessed at: the local lane is arm64 where
these suites hit (A) before reaching this code, so there is no
faithful venue to iterate in and each attempt is a ~30min round
trip. That is a follow-up with an owner, not a dismissal.
(C) incremental: an RSS BUDGET failure, 3054MB against a 2304MB limit
-- not an overflow either. MSan maps shadow (and origin) memory for
every allocation, so the budget cannot separate a leak from shadow.
FIXED rather than excluded: the assertion is now skipped under
__has_feature(memory_sanitizer) only, so the guard keeps its teeth
on every other platform, where inflating the budget would have
blinded it. The suite stays IN the lane.
Verified: with (C) fixed, incremental is 163 passed / 0 failed and ZERO
stack-overflows under the local arm64 MSan container -- so it never
belonged in the overflow list on either architecture.
msan-lane.sh no longer forces MSAN_EXCLUDE empty. That override existed
to ask the architectural question; it is answered, and keeping it would
re-red the gate for causes already recorded. Both venues now read the one
authoritative list in scripts/msan.sh, which still warns loudly that the
lane is partial.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The incremental suite cloned fastapi from the network on every run and
indexed a DIFFERENT corpus per environment: CI used a sparse checkout
without docs/ and tests/ (the assertion thresholds are sized for it)
while local runs indexed the full tree — twice the files for the same
assertions, a per-run network dependency, and the suite's dominant cost
on the sanitized container leg.
The fixture is now cloned from the network at most once per machine
into a persistent cache (committed with an atomic rename so a torn
download can never pose as a valid cache; CBM_TEST_FIXTURE_CACHE
overrides the location), every run local-clones from it offline, and
docs/ plus tests/ are trimmed everywhere — the portable equivalent of
CI's historical sparse profile, so all environments now index the same
corpus the assertions were written against.
Same 163 tests, same assertions: 74s -> 31s native, with the container
leg's cut scaling with the halved indexing volume, and the "network
offline" flake source reduced to first-run-per-machine.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 3 of the graph-analysis overhaul. Two new capabilities on a shared
multi-source traversal substrate, plus a context-bomb guard.
detect_changes — rebuilt into real blast-radius analysis:
The old handler parsed and echoed depth but never traversed: impacted was
just the symbols DEFINED in changed files. It now resolves the git diff to
those symbols, then runs ONE multi-source BFS to the transitive impact set.
- default direction inbound = the blast radius (transitive callers that may
need review); outbound/both available.
- seeds are excluded from the result, and a changed file reached from
another changed file is not counted as extra impact (that is not
downstream fallout).
- merge_base SHA pins the exact diff base for reproducibility.
- impacted_modules: a 2-segment (src/mcp, tests/...) rollup — a quotient
view of where the radius lands, always complete (overflow bucketed, never
silently dropped). Per-symbol rows cap at limit (default 200, nearest
hops first) with impacted_total exact and impacted_omitted surfaced.
On a 7-file core diff: 1469 impacted symbols, 64KB unclipped -> 10.7KB with
the rollup + cap. format:json returns the same model as structured JSON.
cbm_store_bfs_multi (store): one recursive CTE anchored on ALL seeds via a
temp table — never seed_count separate walks over overlapping hubs.
Seed-exclusion NOT IN, MIN(hop) across the set, canonical (hop,id) order,
a memory-safety ceiling that reports *truncated instead of silently
capping. Reproduce-first: seed-exclusion + min-hop, and truncation.
get_architecture aspects:[cycles] — SCC condensation:
Iterative Tarjan (heap stacks, no recursion overflow) over the CALLS graph
reports circular dependencies (strongly-connected components of size > 1).
cbm_store_fetch_call_edges feeds it with a 400k-edge scan budget that
reports partial coverage rather than truncating silently. Opt-in ONLY —
never via a default or 'all' call, since it scans the whole call graph.
On this repo it surfaces 26 real cycles (e.g. the C-LSP type-evaluation
recursion) in ~5KB. Reproduce-first: a 3-node cycle is found, an acyclic
chain is not, and a default call does not run the scan.
get_code_snippet: cap the returned line span at 500 (a structural Module/File
node spans its whole file — an unclipped read returned ~400KB in one call).
source_clipped flags it; start_line/end_line still allow a targeted re-read.
Full suite 6,372 passed / 0 failed; lint clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The incremental path passed changed_files[i].rel_path as the File node NAME,
while the full pipeline names File nodes by basename (pipeline.c "Create File
node"). Node upserts match by qualified name, so an incremental re-index of a
touched file RENAMED its File node in place ("applications.py" ->
"fastapi/applications.py"). File nodes are USAGE-edge sources, so:
- the same tree yields different graphs depending on whether it was indexed
fresh or incrementally, and the divergence grows as files are edited
(add AND modify both rename);
- name-based traversal dead-ends at any touched file's node (e.g. walking
inbound USAGE transitively misses everything that reaches a target through
a recently added/modified class).
Mirror the full pipeline's node shape at the incremental call site: basename
as the name, and the same {"extension": ...} props (previously "{}").
Adds incr_file_node_name_stays_basename to the incremental suite: modify a
file, incremental re-index, then assert no File node name contains '/'.
Red without the fix, green with it; full incremental suite 162/162.
Signed-off-by: Tommy Corbett <spde89@gmail.com>
Completes the compact-output pass across the remaining query tools
(consumers are LLM agents; every response byte is context tokens):
- trace_path: callees[N]{qn,hop}/callers[N]{qn,hop} tables (risk/test/
args columns per flags); the per-hop JSON key envelope was 84% of the
payload. 3,589B -> 1,634B on a representative trace.
- query_graph: rows[N]{cols}: table (columns double as the header);
format:"json" restores columns/rows arrays.
- search_code compact mode: results/raw/dirs tables; drops the
field (duplicate of the qn tail). 2,488B -> 1,135B for 17 hits.
- search_code full mode: per-hit source capped at a 60-line window
anchored on the first match, with source_start/source_truncated
markers — uncapped whole-symbol dumps ran to 142KB per response; the
complete symbol stays one get_code_snippet call away.
- get_architecture: default (no aspects) is now a summary (languages,
packages, entry_points + always-on totals/label/type counts) with an
aspects_hint — the old default rendered EVERYTHING including the full
file_tree (94KB -> 4.5KB -> 2.8KB with TOON tables); all sections
emit as TOON tables, aspects:[...] and ["all"] keep full access
(52KB vs 94KB even for the explicit full dump).
- get_code_snippet: drops the property-blob enrichment (41% of the
response; signature/docstring are literally in the source; fp/sp/bt
never belonged there). Metrics remain reachable via search_graph
fields=[...].
- list_projects: per-project 12-field git block (mostly null) replaced
by the branch name; name/root/nodes/edges/size kept.
- search_graph description: in/out documented as TOTAL all-edge degree,
NOT caller/callee counts (agent evals mis-read it as callers).
- format:"json" escape hatch on every converted tool.
Guards: tool_output_byte_budgets pins absolute byte ceilings on default
search/trace outputs (a property blob sneaking back into row emission
blows them immediately); snippet_enriched_properties inverted to pin
the no-spill contract. Suite 5,984 green; smoke updated to the TOON
contract end-to-end.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Tool responses land in an LLM agent's context window, where the legacy
per-node JSON objects were dominated by data no agent can use: the
similarity/semantic pipeline intermediates fp (~450B minhash hex), bt
(body-token bag) and sp (structural profile) plus 14 complexity metrics
were grafted onto every result by enrich_node_properties — ~64% of every
node emission, ~370 tokens per search hit.
search_graph now defaults to TOON (Token-Oriented Object Notation,
toonformat.dev): scalars as 'key: value' lines and results as a
'results[N]{qn,label,file,lines,in,out}:' header + one row per hit.
Published benchmarks show equal-or-better LLM retrieval accuracy at
40-60% fewer tokens than JSON; rows now also carry line ranges, which
the legacy output lacked.
- fields:[...] opts into per-node property columns (complexity,
signature, docstring, ...); fp/sp/bt are blocklisted even there
- format:"json" restores the legacy verbose objects byte-identically;
include_connected forces it (nested neighbor lists)
- BM25 and semantic modes emit the same table shape (rank/score column)
- semantic-only calls no longer run the unfiltered regex search that
silently prepended up to 200 enriched nodes (102.8KB -> 4.2KB)
- default limit 200 -> 50 (BM25 100 -> 50): cheap first page, page via
has_more/offset as before
- new emitter module src/mcp/compact_out.{c,h} (string builder + TOON
scalar/table emission with spec quoting rules)
Measured on a real index (14-hit regex search): 20,793B -> 1,327B
(-94%); BM25 19,667B -> 5,081B; semantic 102,837B -> 4,181B. A typical
discovery session drops ~50.9K -> ~4.7K tokens combined with the
follow-up slices.
Guards: tool_search_graph_toon_never_leaks_internal_fields (sentinel
fp/sp/bt never emitted, even when requested), rewritten
tool_search_graph_includes_node_properties pins default-lean/fields-
opt-in/json-escape-hatch; suite 5983 passed, smoke green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The incr_full_index guard measured 2050-2072MB peak for the 1100-file
Python index across CI runs after the retention/source-text-cap and
RAM-tiering work shifted the x86 baseline up. The old 2048MB limit sat
right on the line and flaked marginally (2050 vs 2048). Raise the x86
base budget to 2304MB to restore headroom; a genuine leak (GBs over)
still trips it, and the ARM/large-page bumps to 2816 are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
incr_full_index bumps the full-index RSS limit from 2048MB to 2816MB to absorb
ARM's documented ~2385MB peak (vs x86's ~1870MB on the same index), but gated
the bump on a 16KB page size. ARM Linux uses 4KB pages, so the bump never fired
there even though glibc's per-CPU malloc arenas + allocation rounding still
inflate RSS to ~2385MB — CI's ubuntu-22.04-arm measured 2386MB against the
un-bumped 2048 limit and failed (the value is the documented ARM peak, i.e. not
a regression: the page-size heuristic just missed 4KB-page ARM).
Also apply the higher ARM budget on any ARM target (__aarch64__/__arm__/
_M_ARM64). The guard still catches a real leak (GBs over 2816) while no longer
false-failing on 4KB-page ARM. x86 stays at the tight 2048 budget.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The incremental full-index RSS-delta assertion used a fixed 2048 MB
ceiling that 16 KB-page ARM hosts exceed -- larger page granularity
inflates resident set. Raise the ceiling to 2816 MB when the page size
is >= 16 KB, keeping the tighter bound on 4 KB-page hosts.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
extract_class_def() sets start_line and end_line from the tree-sitter
node span, but left def.lines = 0 (zeroed by memset at function entry).
The equivalent function push_method_def() correctly computes
def.lines = end_line - start_line + 1
The same one-liner was simply missing from the class extraction path.
This affects all languages — Python, TypeScript, Go, and any other
language where Class nodes are extracted via extract_class_def().
Regression test: after indexing a project, at least one Class node must
have c.lines > 0.
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
detect_changes advertised a `since` parameter in its inputSchema but the
handler never read it — it always diffed against base_branch (default
"main"), so detect_changes(since="HEAD~10") silently returned the wrong or
empty result when HEAD was on the default branch.
Fix: read `since` and, when present, route it through base_branch so the
existing shell-arg validation (cbm_validate_shell_arg) and the
`<base>...HEAD` diff apply unchanged; `since` takes precedence over
base_branch. Also narrows the schema description — the prior "date" form
(e.g. 2026-01-01) is not a revision and never worked through this path — and
documents the inherited three-dot semantics. Adds regression tests
tool_detect_changes_since and tool_detect_changes_since_precedence.
Refs #371
Signed-off-by: Kris Kersey <kris@kerseyfabrications.com>
MATCH (c:Class)-[:DEFINES_METHOD]->(m:Method) returned at most 10 results
for any class, regardless of how many methods it actually has.
Root cause: bind_cap was set to scan_count (the number of nodes matched in
the initial pattern — typically 1 when querying a single class by name).
max_new = bind_cap * 10 = 10, so the edge expansion loop exited after
collecting 10 results. No error, no warning, no truncation indicator.
This is language-agnostic: any class with more than 10 methods in any
language was silently truncated. The fix is two characters:
bind_cap = scan_count > max_rows ? scan_count : max_rows
Regression test: a Python class with 15 methods must return all 15 via
MATCH (c:Class)-[:DEFINES_METHOD]->(m:Method) with label filtering.
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
Add scripts/check-no-test-skips.sh (run from lint) which fails the lint phase on any plain SKIP() or direct tf_skip_count manipulation; only SKIP_PLATFORM() (for genuinely platform-specific tests) is tolerated. Add FAIL() and SKIP_PLATFORM() helpers to the test framework and convert the remaining SKIP()/perf-gated skips across the suite into pass-or-fail assertions, so a suite that cannot meet its preconditions reports a red failure instead of a silent skip.
When include_tests=false (default), test files are filtered from trace
results. When true, test nodes appear with is_test=true marker. Helps
answer "which tests cover this function's blast radius?"
Also adds tests for include_tests and risk_labels params.
Co-Authored-By: maplenk <maplenk@users.noreply.github.com>
Inject incr_test_injected/helper in suite setup so tool tests work
with CBM_SKIP_PERF=1. Use different function names in incr_modify_file
to avoid duplicate injection. Update baseline counts after injection.
Lower graph thresholds for sparse checkout on CI.
CBM_SKIP_PERF=1 now skips only phases 2-7 (incremental delta, stress,
perf regression) inside the suite. Phase 1 (initial index) and phases
8-30 (135 tool integration tests) always run. This keeps ~2672 tests
on CI vs the full ~2692 locally.
Also fix Windows build: guard symlink() with #ifndef _WIN32.
- Add CBM_SKIP_PERF=1 env var to skip incremental/perf test suite
- CI and Docker test targets skip perf by default (run.sh perf for manual)
- Convert all perf assertions to warnings (log timing, never block)
- Fix store.h anonymous enum in struct (GCC rejects, clang accepts)
- Fix test_store_search.c mkstemp on non-template path
- Add ca-certificates to Docker test image for git HTTPS
- Add cbm_gmtime_r shim in compat.h (Windows gmtime_s wrapper)
- Fix compat.c missing constants.h include (Windows build)
- Fix platform.c _environ redeclaration on mingw
- Rename trace_call_path -> trace_path in smoke/soak/fuzz scripts
On CI runners, skip docs/ and tests/ directories when cloning
FastAPI (62% of 18K files). The fastapi/ package source remains
intact for all test assertions. Locally, full clone is preserved.
Shared CI runners are too variable for hard timing thresholds.
Performance is still logged and warns above 15s, but no longer
blocks the test suite. Added 30min timeout on test-unix jobs.
Prefix-to-decorator bridge now matches by file path segments: prefix
"/bq/bulk/ranked" only connects to handlers in paths containing
"bq/bulk/ranked", not all handlers in the service.
HCL infra binding extraction: scan resource blocks for attribute pairs
with source keys (topic, queue_name) and target keys (uri, push_endpoint).
Handles nested blocks (push_config { push_endpoint = "..." }).
Extract route_path and route_method directly from decorator AST nodes
during tree-sitter walk. Route + HANDLES edges created during
definition node creation. ensure_decorator_routes pass recreates
Route+HANDLES for existing nodes in incremental mode.
Prefix Routes bridged to handler Functions by service directory matching.
sqlite_writer.c pb_build_interior: final root-level sep_cell
allocations were not freed. Each cbm_write_db call leaked 84 bytes.
Detected by ASan on Linux CI with 155 incremental test iterations.
Also fix CI timing: use MAX_TOOL_MS (15s) for perf assertions,
skip timing on full re-index operations (CI runners are slower).
- Serialize return_type, parent_class, base_classes on definition nodes
- DATA_FLOWS edges link callers through Route nodes to handlers
- YAML nested field extraction with dotted key paths (key_path property)
- Rename trace_call_path → trace_path with mode parameter:
calls (default), data_flow, cross_service
- Backward compatible: trace_call_path alias still accepted
- Updated CLI help text and all tests
Comprehensive test suite using real repo (FastAPI 0.99.1, ~1100 files):
- Full index baseline with perf metrics (nodes, edges, RSS, timing)
- Incremental: modify/add/delete/rename files, noop, formatter run
- Adversarial: empty file, syntax errors, 5000-line function, binary
- Stress: rapid reindex, batch 20 files, delete-all + re-add
- Accuracy: incremental vs full within ±2 nodes/edges
- MCP tools: all 14 tools, every parameter, content validation
- Error handling: bad project, missing params for each tool
- Performance: 5s timeout on every tool call
Fix use-after-free in detect_changes and manage_adr:
yyjson_mut_arr_add_str stored pointer to stack buffer without copy.
Changed to yyjson_mut_arr_add_strcpy / yyjson_mut_obj_add_strcpy.
