The parallel runner's isolated-process execution exposed three cli tests
that were passing for the wrong reason on Linux: cbm_zed_config_dir (and
the VS Code/vendor path resolvers) prefer $XDG_CONFIG_HOME over
$HOME/.config, CI runners export it, and an isolated `test-runner cli`
process therefore resolved config paths OUTSIDE the test fixture — the
tests only stayed green in the all-suites single process because an
earlier suite's env mutations leaked into them (hidden cross-suite
coupling, deterministic failure in isolation, macOS unaffected since its
resolvers do not consult XDG).
- cli_detect_agents_finds_zed: save/set XDG_CONFIG_HOME to the fixture's
.config for the detection call, restore after.
- cli_vscode_profile_mcp_uninstall: same pin alongside its existing
HOME/PATH/APPDATA handling.
- cli_durable_profiles_follow_current_vendor_paths: XDG_CONFIG_HOME joins
its unset-and-restore env list.
All 222 cli tests pass locally with a hostile XDG_CONFIG_HOME exported.
No production code changed; no assertion weakened.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Flake audit: the whole corpus has exactly two wall-clock test deadlines —
the EOF-shutdown alarm(5) and the issue-832 supervisor-spawn alarm(60),
both in the mcp suite (the latter already flaked once under heavy local
load). Serializing mcp costs ~6s of tail and closes the starvable-deadline
class completely; every remaining parallel-wave suite is deadline-free.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up correcting d8a78e8, whose message claimed a broken merge cap
crashes "within the first second" so the 10s alarm could not mask it.
Falsification disproved that claim AND found the guard was vacuous on
POSIX from the start:
- With CBM_TS_STACK_MERGE_MAX_DEPTH deleted outright, the ORIGINAL
unbounded test stayed GREEN (220s full grind, no crash) on macOS ASan.
- Small thread stacks (2 MB, then 512 KB) did not change that.
- Instrumenting the vendored recursion showed why: the recursive merge
path is NEVER ENTERED for this input in the observable window — max
depth stays 0. There is nothing for a stack bound to catch here.
So the cap-regression DETECTION lives in the Windows in-process branch
(real ~1 MB native stack, where #913 actually manifested — exercised by
the windows CI leg). The POSIX branch is, and always was, a bounded
no-crash smoke of the pathological parse; the 10s alarm makes it
deterministic (12-15s across three runs, was 2-218s machine-dependent)
without any detection loss, because there was no POSIX detection to
lose. The experimental pthread machinery is dropped; the test comment
now carries the falsification data so no future reader mistakes the
POSIX branch for a cap guard.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
perl_glr_deep_parse_recursion_capped wandered between 2s and 218s for the
IDENTICAL input: past the merge cap the ambiguity-exploded GLR parse grinds
at an environment-dependent rate (allocator/scheduler state in the forked
child), and the test paid for however long that grind ran. It was the
critical path of every parallel run and the dominant chunk of the
sequential one.
The guard's actual signal is the CRASH, not the grind: when
CBM_TS_STACK_MERGE_MAX_DEPTH regresses, the recursive stack-merge
overflows within the first second at this depth (same profile as the
sibling pre-guard probe, rc=139 in under a second). The forked child now
arms a 10s alarm whose handler exits cleanly — a real SIGSEGV/SIGBUS
still terminates the child by signal first and stays visible to
WIFSIGNALED, so the RED path is untouched; only the pointless remainder
of the grind is skipped. Windows (in-process branch) is unchanged.
Suite b: 13s across three consecutive runs (was 2-218s). Full parallel
run: 97s wall, 6,374 passed / 0 failed / 1 skipped, union guard clean —
vs 351s sequential on the same machine (3.6x).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
First CI validation round of the parallel runner, three findings:
- Windows failed ALL 104 suites with "0 passed, 1 failed" each: the CRT
emits --list-suites lines as CRLF, so every dispatched name carried a
trailing CR ("arena\r" is an unknown suite). The union guard caught it
loudly, exactly as designed. The driver now strips CR when writing the
suite list.
- The ubuntu-gcc legs had 3 real failures in the cli suite: it spawns
subprocesses with fixed deadlines, and a fully saturated 4-core runner
starves those deadlines into flakes. Deadline-sensitive suites (cli,
subprocess, watcher, incremental, httpd, ui, index_resilience, and the
stack_overflow family) now run SEQUENTIALLY after the parallel wave on a
quiet machine — same suites, same tests, same gates, only the schedule
differs; the union guard checks the combined result set.
- Failing suites now print every FAIL site with context (the tail-30 of a
long suite log hid which tests actually failed on CI).
Local: totals unchanged (6,361 passed / 0 failed / 1 skipped), union guard
clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The runner executed all 104 suites sequentially (~10 min locally, the bulk
of every CI test leg). The suites are process-isolated already (per-process
mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite
failures), so the serialization was pure convention.
- test-runner --list-suites: prints every registered suite, one per line,
emitted by the SAME macro table that executes suites — the list cannot
drift from the run set by construction.
- scripts/run-tests-parallel.sh: runs each suite as its own process
(jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS
CONTRACT: a union guard fails the gate if the set of suites that
produced a result differs from --list-suites (nothing can be silently
dropped, a newly added suite is picked up automatically); per-suite
pass/fail/skip are summed into the sequential runner's exact summary
format; any suite crash, failure, or omission exits nonzero. Per-suite
wall times are printed for balance tracking.
- make test-par: the parallel target. make test (sequential) is unchanged
and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh).
- scripts/test.sh: builds, then routes through test-par — every CI test
leg gets the speedup with zero workflow-topology change (same jobs,
same gates, same billing; the legs just finish sooner).
- The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests,
pure re-registration): as one suite it was the wall-clock critical path
of any parallel run — every other suite finished underneath its ~4
minutes.
Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs
parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped),
union guard clean. The TSan leg keeps its dedicated subset runner
(sequential) — out of scope here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The PR CI caught two consumers the format migration missed — both parsed
tool output by its old shape and failed silently or with a wrong count:
- hook-augment built its PreToolUse additionalContext by reading the
legacy 'results' object-array from search_graph format:json; after the
json-tree reshape that key no longer exists, so the hook emitted nothing
(caught end-to-end by tests/windows/test_hook_augment.py). The parser
now walks groups -> qn_prefix/file + column-ordered rows. A new local
test feeds the parser the LIVE server envelope, so any future drift
between response shape and hook parser fails in the normal suite on
every platform — not only in the Windows CI guard. Reproduced RED
(ctx NULL, the exact CI failure) before the fix, GREEN after.
- smoke-test.sh carried six old-shape parsers (callers[N], rows[N]{,
clusters[N], semantic[N]{, and the assert_toon_table helper); the tree
headers never matched, so trace verification read 0 callers. All six
updated to the tree contract; the full smoke script passes end-to-end
locally.
cli suite 222 passed; lint clean; scripted smoke ALL PASSED.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fresh review pass over the whole branch; three classes of fix:
Cursor safety — the 'legacy' generation gap:
A pre-migration DB (indexed by an older binary, opened read-only) has no
store_meta, so its generation reads 'legacy' — and 'legacy' == 'legacy'
across rebuilds, which silently defeats stale-cursor detection for exactly
those DBs: a stale (hop,id) watermark would resume on wrong node ids.
Cursors are now neither minted (truncated + raise-limit hint instead) nor
accepted (cursor_unsupported teaching error) under a legacy generation.
The standard flow was verified safe empirically: every index run seeds
store_meta via cbm_store_upsert_project in the flush path, and a rebuilt
DB file gets a fresh random db_uid, so aliasing is impossible there. The
guard is unreachable through the public API (upsert always seeds) — it
protects only genuinely old DBs, hence no synthetic test.
format:'json' now emits the tree model on EVERY tool (the last five):
- BM25 query mode: {total, cols, rows[[qn,label,file,lines,rank]]} —
rank order preserved, ranked output is never regrouped.
- semantic: {semantic: {cols, rows[[qn,label,file,score]]}}.
- search_code compact/full: {cols, rows} with nested match-line arrays;
full appends a per-row source object (same guarded, windowed reader);
raw_matches likewise {cols, rows}. files mode stays a plain list.
- get_architecture: all twelve sections emit {cols, rows} via one shared
section helper; cluster row cells carry their nested lists as arrays.
- include_connected became a real column in BOTH encodings
(';'-joined cell in the text tree, nested name array in json) — the
include-connected-forces-JSON coupling is gone, and the last legacy
per-node-object emitters (emit_search_results, enrich_node_properties)
are deleted.
Teaching-error parity: detect_changes now rejects an unknown
with the valid values (it silently fell back to inbound; trace_path already
errored — the caller would have misread the result's semantics).
Full suite 6,373 passed / 0 failed; lint clean. Live-verified on the
self-index: bm25/search_code/architecture json shapes, the connected
column, and cursor pagination.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Fresh-eyes review of the phase-2/3 work; five defects, each against our own
stated principles (no silent caps, exact counts, same model in both
encodings):
- trace callees_total/callers_total counted test-file rows even when
include_tests=false hides them from the table — the scalar disagreed with
the enumerable set (a field-eval agent read callers_total=175 against a
handful of visible rows and stopped trusting the tool). Totals now apply
the same filter as the emitters; description documents the transitive
(within-depth) semantics. Guard test: filtered=1 vs include_tests=2.
- cycles clamped the SCC list at 100 silently, losing the true count.
cycles_total is now always exact and cycles_omitted is emitted when the
display clamp engages (both encodings).
- detect_changes accepted a 'limit' param (referenced by its own omitted-
hint) that the schema never declared — undiscoverable. Declared.
- detect_changes format:json lacked the impacted_modules rollup the tree
encoding carries — model parity restored via a shared rollup helper.
- tree search extra-field cells were appended raw: a signature with spaces
shifted every following column. Cells now route through the shared
emitters (space-triggered quoting, '-' placeholder); the properties test
pins the quoted form.
Full suite 6,373 passed / 0 failed; lint clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 3 of the graph-analysis overhaul. Two new capabilities on a shared
multi-source traversal substrate, plus a context-bomb guard.
detect_changes — rebuilt into real blast-radius analysis:
The old handler parsed and echoed depth but never traversed: impacted was
just the symbols DEFINED in changed files. It now resolves the git diff to
those symbols, then runs ONE multi-source BFS to the transitive impact set.
- default direction inbound = the blast radius (transitive callers that may
need review); outbound/both available.
- seeds are excluded from the result, and a changed file reached from
another changed file is not counted as extra impact (that is not
downstream fallout).
- merge_base SHA pins the exact diff base for reproducibility.
- impacted_modules: a 2-segment (src/mcp, tests/...) rollup — a quotient
view of where the radius lands, always complete (overflow bucketed, never
silently dropped). Per-symbol rows cap at limit (default 200, nearest
hops first) with impacted_total exact and impacted_omitted surfaced.
On a 7-file core diff: 1469 impacted symbols, 64KB unclipped -> 10.7KB with
the rollup + cap. format:json returns the same model as structured JSON.
cbm_store_bfs_multi (store): one recursive CTE anchored on ALL seeds via a
temp table — never seed_count separate walks over overlapping hubs.
Seed-exclusion NOT IN, MIN(hop) across the set, canonical (hop,id) order,
a memory-safety ceiling that reports *truncated instead of silently
capping. Reproduce-first: seed-exclusion + min-hop, and truncation.
get_architecture aspects:[cycles] — SCC condensation:
Iterative Tarjan (heap stacks, no recursion overflow) over the CALLS graph
reports circular dependencies (strongly-connected components of size > 1).
cbm_store_fetch_call_edges feeds it with a 400k-edge scan budget that
reports partial coverage rather than truncating silently. Opt-in ONLY —
never via a default or 'all' call, since it scans the whole call graph.
On this repo it surfaces 26 real cycles (e.g. the C-LSP type-evaluation
recursion) in ~5KB. Reproduce-first: a 3-node cycle is found, an acyclic
chain is not, and a default call does not run the scan.
get_code_snippet: cap the returned line span at 500 (a structural Module/File
node spans its whole file — an unclipped read returned ~400KB in one call).
source_clipped flags it; start_line/end_line still allow a targeted re-read.
Full suite 6,372 passed / 0 failed; lint clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
- cbm_toon_* renamed to cbm_tree_* (the emitters have emitted tree syntax
since the format flip; the name now matches).
- format enums reduced to [tree, json], default tree; dead tree/toon flags
removed; tool RESPONSE descriptions rewritten for the grouped-tree
contract (group prefix printed once, full qn = prefix + '.' + name).
- format:'json' is now the json-stringified tree: the SAME grouped model
serialized as JSON — {total, cols, groups:[{qn_prefix, file,
rows:[[column-ordered cells]]}], has_more} for search, per-leg
{cols, groups} for trace (risk/args columns follow the flags),
query_graph's columns+rows shape already complied. Per-row key envelopes
and the legacy property blob are gone (blob-never-spills pinned by test);
requested add columns in both encodings.
- include_connected keeps the per-node-object shape (nested neighbors) —
the one remaining legacy consumer, conversion tracked.
- legacy verbose emitters deleted (bfs_to_json_array).
Full suite 6,367 passed / 0 failed; lint clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Round-2 overhaul of the tool-output surface, built eval-first (byte harness +
no-context field-eval agents, always vs the v0.9.0 release binary):
Format — tree syntax everywhere:
- compact_out emits tree tables: 'key: N (cols: a b c)' headers, indented
space-delimited rows, '-' placeholders, space-aware quoting. Every table
in every tool converts mechanically.
- Grouped prefix-factored output is the DEFAULT for search_graph and
trace_path: rows grouped under their shared (qn-prefix, file) printed
once, row qn = group prefix + '.' + name (stated in-band). Measured:
192-hit search 2,929B (was 4,811 TOON / 238,466 v0.9.0), 82-row trace
2,256B (was 4,234 / 8,094). Field eval: 5/5 answers correct, QN
reconstruction verified, no comprehension loss.
- detail:'ids' tier: bare-qn enumeration for wide sweeps (-44%).
Pagination — exactly-once, stateless:
- store_meta generation (per-DB random uid + mutation counter bumped in
cbm_store_upsert_project — every index run's choke point): cursors can
never silently validate against a rebuilt DB.
- trace cursors: opaque token (leg, generation, params-hash, (hop,id)
watermark); pages drain callees then callers with exact
callees_total/callers_total on every page; teaching errors for
params-mismatch and staleness. Guard: 12 callees paged 5+5+2, every row
on exactly one page.
- deterministic ordering: BFS ORDER BY hop,id; canonical (hop,id) sort +
min-hop dedup across same-name seeds (hop fed risk_labels wrong before);
search (name,id)/(rank,id) tie-breaks make offset pages stable.
Lean defaults + teaching errors:
- get_graph_schema no longer advertises blocked internal fields (fp/sp/bt).
- index_repository coverage lists: counts + 5 examples + one-line pointer
notes (full record was always in the logfile); 6.0KB -> 1.5KB here,
53KB observed on a large repo. index_status gates the git context block
behind verbose:true.
- trace limit param (<=5000) + truncated:true — never a silent truncation;
invalid direction now errors with the valid values; requesting core
columns via fields hints instead of emitting empty cells; cypher
resolves f.file/f.qn/f.path aliases.
Trace-correctness fixes (reproduce-first, RED verified):
- bfs_collect_edges 4KB id-string overflow: past ~1000 visited nodes the
edge SQL failed to prepare and EVERY trace edge silently vanished; now a
per-connection temp-table join (star guard: 1200/1200 edges).
- bfs_union_same_name: min-hop across seeds + edge dedup.
Guards: gating tool-output regression suite (context-explosion detector:
blocked-field leaks, repeated-key envelopes, oversized notes + calibrated
byte ceilings + semantic floors). Full suite 6,367 passed / 0 failed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Add a checkpoint-starvation warning to cbm_store_checkpoint: read the
wal_checkpoint_v2 out-params (frames in the WAL vs frames reclaimed) and, when
the WAL is large (>= ~1 GiB of frames) but the PASSIVE checkpoint couldn't reset
it because readers hold marks, log store.wal.starving. This is the operator
visibility the reclaim limit can't provide on its own — it surfaces the
starvation (and its scale) before the -wal file fills the disk. Portable
(no file stat / no path handling), fires only under genuine starvation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On-disk write connections set no journal_size_limit, so the -wal file is never
physically shrunk: all our checkpoints are PASSIVE (TRUNCATE is deliberately
avoided because its ftruncate(fd,0) can SIGBUS a sibling mmap'd process on
macOS), and PASSIVE never ftruncates. Under checkpoint starvation the WAL then
grows without bound (reported: 115 GB, filled the drive) and even a later
successful checkpoint leaves the file at its peak size (#1083).
Set journal_size_limit = 256 MiB on write connections so SQLite truncates the
-wal back to that bound whenever a checkpoint can finally reset the log —
reclaiming disk opportunistically during a session, not only on clean close.
256 MiB is far above the healthy WAL (~4 MiB under the default 1000-page
autocheckpoint), so normal indexing never triggers truncate/regrow churn; the
limit only fires after abnormal growth. No TRUNCATE checkpoint is introduced.
Scope: this is the *reclaim* half of #1083. It does not prevent the peak under
continuous reader contention (a checkpoint can't reset the log while a reader
holds a mark) — the write-amplification driver (concurrent full re-indexes of
one repo) is serialized by the cross-process pipeline lock in #892.
Test: a file-backed write store now reports journal_size_limit = 256 MiB
(RED on main: -1 / unlimited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
CodeQL cpp/comparison-with-wider-type (4 high-severity alerts, #69-#72) flagged
two loops in the vendored ObjectScript scanner (objectscript_common.h, in both
the _routine and _udl grammar copies) where a uint8_t counter is compared
against an int length: the reverse_marker scan and the html_marker_buffer
reversal. Both lengths are hard-bounded by MARKER_BUFFER_MAX_LEN (30), so the
uint8_t counter can never wrap and the flagged infinite loop is unreachable in
practice — but widening the counters to int removes the pattern and clears the
gate. Provably behavior-neutral (int holds every uint8_t value; length <= 30).
Recorded as a local modification in the vendored grammar MANIFEST so it is
re-applied on the next re-vendor.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A function-like macro invocation whose argument is a type token — e.g.
ALLOC(int, n) — makes tree-sitter's C/C++ grammar emit an ERROR node (it parses
the type in expression position), which cbm_collect_error_regions recorded as a
parse_partial coverage gap. But the macro is #defined in the same file and the
call sits inside an already-extracted function body, so nothing is actually
missing from the graph — it's a benign call the grammar can't parse without the
preprocessor (#1071, systematic across allocation-macro-style codebases).
Subtract an error region only when it both (a) contains an invocation of a
file-defined function-like macro and (b) is fully enclosed by an extracted
Function/Method body. Condition (b) keeps a TOP-LEVEL macro invocation that
expands to a definition still flagged (#949: the generated def isn't in the
original span), and fails safe — a benign top-level call stays flagged rather
than a real gap being hidden.
Tests: a type-arg macro call inside a function no longer reports parse_partial;
a real in-body syntax error still does; #949/#946 preserved.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An unbounded whole-graph OPTIONAL MATCH or a GROUP BY that produces roughly one
group per node does O(bindings x groups) work in execute_return_agg and can run
for minutes on a large graph before emitting a single row. The 100k row ceiling
never fires (no rows are produced), so query_graph just hangs with no partial
result and no error (#601).
Arm a monotonic wall-clock budget (default 30s) at query entry and check it
(throttled, every 1024 iterations) in the scan, relationship-expansion and
aggregation hot loops. On expiry the query aborts with a clear, actionable error
instead of hanging. A thread-local test hook forces the budget so the guard is
covered by a deterministic reproduce-first test; a companion test confirms the
default budget does not false-positive on a normal query.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_pipeline_fqn_compute stripped the file extension for EVERY qualified
name, including File-node QNs (name=="__file__"). Two different files
whose names share a stem therefore collided on the File QN and the
graph kept only one node per stem:
- .env / .env.local / .env.production all strip to ".env" — only one
survives per directory (#1077).
- a C/C++ header and its same-stem source (NodeController.h vs .cpp)
strip to the same stem, so the header got no File node of its own,
leaving it disconnected (#964's remaining half after #983 fixed the
include→Class resolution).
File-node QNs now keep the full filename (tokenize_path splits on '/'
only, so the extension rides in the last segment and stays unique).
Extension stripping is retained for MODULE/symbol QNs (name==NULL or a
symbol name) — that stem unification is load-bearing for C/C++
declaration↔definition cross-file resolution, and is left untouched.
All 26 __file__ QN sites route through this one function, so node
creation and every lookup stay consistent.
Reproduce-first:
- fqn_file_qn_preserves_dotfile_variants_issue1077 and
fqn_file_qn_distinguishes_same_stem_header_source_issue964 assert the
distinct File QNs (RED before: both collide to the stripped stem).
- fqn_module_qn_still_strips_extension guards that module QNs are
unaffected.
- repro_issue964 now asserts the header keeps its own File node AND is
connected (CONTAINS_FILE + DEFINES the class it declares), matching
the reporter's zero-inbound disconnection metric — GREEN.
Closes#1077Closes#964
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #913 GLR recursion cap patches vendored ts_runtime/src/stack.c;
document it in THIRD_PARTY.md so the vendored modification is on record
and re-applied on the next ts_runtime re-vendor (Stage 4: 0.26.x). The
patch is marked inline; this adds the provenance-level note the policy
requires for any change to vendored code.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parallel resolver dropped a CALLS edge whenever the LSP produced a
resolution whose callee_qn was not a node in the graph buffer: the
registry fallback lived in an else that ran only when NO LSP resolution
existed, so an LSP-with-unresolvable-target left res empty and the edge
was discarded outright. The sequential pass falls THROUGH to the
registry resolver in the same situation.
This diverged the two pipelines on the exact intersection the reporter
identified (#1085): a JSX component imported through a tsconfig paths
alias. The TS LSP resolves the element ref to an alias-path QN that
never matches a def node, so lsp_target is NULL and the parallel path
dropped it — while sequential resolved it via the import_map /
unique_name registry path. On a Next.js repo this silently removed
~21% of the call graph (every alias-imported JSX composition edge), and
the parallel path is the default above 50 files, so the default index
was the broken one.
The fix runs the registry fallback whenever the LSP did not yield a
gbuf-resolvable target, matching the sequential fall-through and
restoring seq/parallel parity.
Reproduce-first: calls_jsx_component_via_tsconfig_alias_parallel_issue1085
indexes the alias+JSX shape through the parallel path (et_index_parallel
pads past the 50-file threshold) and asserts the CALLS edges land on the
component. RED before (0 edges), GREEN after.
Closes#1085
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SEPURI-SAI-KRISHNA