An 81k-file TypeScript corpus previously never finished indexing. Three
stacked defects, found by stack-sampling the live grind and reading the
supervisor's quarantine records:
1) walk_defs pushed children via index-based ts_node_child(i), which is
O(i) per call in tree-sitter — O(n^2) per wide node. A 3.5 MB file
with ~580k flat comment siblings needed ~1.7e11 iterator steps and
hung extraction past the supervisor's quiet-timeout. All six descent
sites (including the default loop) now collect children linearly with
a TSTreeCursor (wd_collect_children); small nodes keep the direct
index path. Guard: extract_wide_flat_file_is_linear (400k comment
siblings — the monster's exact shape — bounded at 30s).
2) The sequential cross-LSP driver handed the FULL def list to a full
per-file registry build+finalize — O(files x defs); 74% of samples
sat in build_qn_index. The parallel resolve worker's dispatch
(module-def filter -> shared prebuilt registries -> filtered
fallback) is now extracted into cbm_pxc_dispatch_file and drives
BOTH pipelines — one path, one semantics. The shared registries
live in a caller-owned arena that outlives the calls pass
(resolved_calls borrow registry strings; freeing at pass end was a
use-after-free ASan caught in pass_calls). Rust is exempt from the
def filter: Cargo-manifest cross-crate resolution needs defs from
OTHER workspace crates that the own+imports filter drops (this
starvation predated the unification on the parallel path). Guard:
pipeline_seq_ts_cross_uses_shared_registry (full builds 40 -> 0,
cross-file edge preserved).
3) The crash supervisor's recovery re-ran the worker SINGLE-THREADED to
keep one exact marker. At scale that fell into the sequential crawl,
was killed as a hang mid-pass, and the stale extraction marker got
four innocent files quarantined, one 15-minute retry at a time.
Recovery (and the terminal partial run) now re-run PARALLEL; the
marker is an append journal ("S <rel>" / "D <rel>", written by
extraction and both resolve drivers), the suspect set is the open-S
in-flight set, and a file is quarantined only when it recurs across
two consecutive failed runs — one per round, oldest open S first;
disjoint consecutive sets stop the loop rather than blame an
innocent. Guard: index_recovery_parallel_quarantines_crasher
(single-threaded spawn count must stay zero; verified RED against
the old loop).
Also: parse_ts_type_text gains a dynamic work budget proportional to
the input (1M + 64x source bytes, CBM_TS_TYPE_BUDGET override, warn
once then degrade to unknown) replacing unbounded recursion cost.
Measured on microsoft/TypeScript (81,398 files): infinite -> 33s cold
full index, 295k nodes / 779k edges, zero dangling edges, zero files
skipped — the monster file itself now indexes cleanly. Full 9-language
sweep green with no quality flags; kernel unchanged at 390s / 8.5M
nodes / 0 dangling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_mem_init is init-once (guarded by g_initialized), so the
backpressure test's setenv(CBM_MEM_BUDGET_MB=1) + re-init 'restore'
was a silent no-op: whichever init ran first fixed the budget for the
whole process. In the full runner this test's init won, locking a 1 MB
budget that leaked into every later budget consumer —
mem_over_budget_low_rss red on every platform, and all subsequent
pipeline tests silently running with backpressure permanently engaged.
Add cbm_mem_set_budget_for_tests(): writes the budget directly without
touching the init guard. The test now saves the caller-visible budget,
sets 1 MB via the hook, and restores before asserting. Env dance
removed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The deterministic same-QN collision rule picked the lexicographically
smallest (label, file_path, start_line, name), which silently inverted
the documented upsert/merge semantics: re-upserting a QN with updated
fields kept the OLD content (gbuf_upsert_updates,
gbuf_upsert_same_qn_updates_all_fields, gbuf_merge_overlapping_qns all
red on every platform).
Replace it with a single mixed-direction total order applied in both
cbm_gbuf_upsert_node and merge_update_existing: smallest file_path,
then LARGEST start_line, then largest name/label. Still a pure content
function of the two candidates (commutative, associative,
scheduling-independent — multi-threaded runs stay byte-identical,
verified by the xfs MT x4 fingerprint guard), and line-descending
within a file restores the classic contract: a later definition in the
same file replaces the earlier one, and a full tie is the same entity
refreshed in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The header referenced the license identifier of a third-party library's
bundled dependency while explaining that the implementation deliberately
does not use that library. License scanners match the bare identifier
string and misclassify the file. Reword to state provenance directly:
written from the published papers, nothing vendored or derived.
Comment-only change; also picks up the formatter pass on this file.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This reverts commit 0475a880e5735ef5cb0b0d99532e552096e5a2d5.
Measured on the linux kernel: peak RSS unchanged (23.95 vs 24.4 GB, within
run noise) at +22 s wall -- the extraction peak turned out to be monotonic
retained-tree/source accumulation, which is schedule-independent, not the
concurrent giant transients this queue bounded. Net negative as measured;
the retention-side fix is parked for post-release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase-aligned RSS attribution located the global peak (24.4 GB at t=47s,
~14 GB transient above the retained floor) INSIDE parallel extraction: the
size-descending queue put the kernel's 12 largest files into 12 workers
simultaneously, each holding a multi-MB source + its 10-50x TSTree + arena.
Every post-extraction optimization sat below this waterline.
Replace the single biggest-first cursor with a dual-ended claim queue over
the same size-sorted list: big_workers = clamp(workers/6, 1, 3) claim from
the BIG end while the rest rush the small end, so at most big_workers giant
working-sets coexist -- a structural bound, not a scheduling heuristic (a
static interleave cannot bound concurrency under dynamic pulls: smalls
finish in ms and workers converge back onto the giants). Giants still start
at t=0, preserving the LPT no-straggler property; workers never idle while
either end has files. Per-file claim flags (atomic exchange) make the
crossover race-free; end-of-queue rescans are O(n) bytes once.
Intra-file parallel parsing was researched and ruled out: tree-sitter has
no such concept (incremental != parallel; one parser per thread is the
model), and data-parallel parsing theory (PAPAGENO through SLE 2025 cyclic
OPG work) requires operator-precedence grammars with local parsability --
none of the 159 vendored GLR grammars qualify, and C's preprocessor makes
byte-chunking unsound.
Output is schedule-independent since the determinism fix and verified so:
xfs sorted edge set byte-identical to the previous commit. pipeline 216
passed. Worst case (all-giant repo) unchanged by design.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two borrow-don't-copy wins from the layer-duplication analysis (a hot string
was resident 4-5x across extraction arenas, gbuf, registries, defs and
tokens). Graph output verified byte-identical.
1. Resolution registry (registry.c): every registered definition strdup'd its
qualified name TWICE (exact-map key + by_name array item) and its label
once (~8.5M label strdups). The by_name items now borrow the exact map's
single owned key (same lifetime, freed in one place) and labels intern
into a small per-registry pool (<= ~30 distinct). ~280 MB + ~17M
allocations on the kernel.
2. Semantic tokens (pass_semantic_edges.c): all_tokens held one strdup per
token INSTANCE — identical tokens ("xfs", "error", ...) duplicated across
hundreds of thousands of functions. Each tokenize worker now interns into
a per-worker pool (key == value == the single owned copy; no contention,
content-identical strings so downstream corpus/tfidf output is unchanged);
all_tokens slots borrow. Ownership moves from per-func frees to pool
frees at pass end.
Suites: semantic 33, simhash 24, pipeline 216, registry 54, incremental 161.
xfs: two MT runs byte-identical, and byte-identical to the pre-change graph.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The semantic pass kept four dense 768-float vectors resident per function
(ri/api/type/deco = 12.3 KB per func, ~9.4 GB transient on the linux kernel)
solely so candidate pairs could be cosine-scored after LSH. Replace the
resident floats with 4-bit quantized codes (~0.5 KB each, ~6x): new
src/semantic/rotsq.{h,c} — rotated scalar quantization from the RaBitQ
family of algorithms (Gao et al., SIGMOD 2024/2025), implemented from the
papers in plain C11 (no Eigen/library dependency): a deterministic
XXH3-seeded +-1 diagonal followed by a Fast Walsh-Hadamard rotation
(768 -> 1024 padded) spreads vector mass so per-vector scalar quantization
is near-optimal; the inner product is recovered from the codes with the
exact SQ expansion (one u8 integer dot + four multiplies) — a deterministic
pure function of the codes. Named rotsq, not rabitq: this is the family
core, not the papers' exact codebook construction.
Producers build each dense vector in a worker-local buffer, encode, and
discard; the int8 qvec export is unchanged. LSH signatures are computed
from DEQUANTIZED codes against hyperplanes in the rotated basis — random
hyperplanes are basis-agnostic, so no dense originals are retained and the
phase structure is untouched.
Estimator quality gated by sem_rotsq_ip_error_bounds (seeded vectors:
mean error < 1%, max < 4% of unit scale, ASan-green). Effect on output:
multi-threaded determinism preserved (xfs: two runs byte-identical);
SEMANTICALLY_RELATED edges shift once as scores move within ~1% of the
0.75 threshold (xfs: ~130 -> 203; sampled new pairs are high quality —
alloc/free, read-verify and collapse/insert families).
Suites: semantic 33 (incl. the new estimator gate), simhash 24,
pipeline 216 — zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three memory quick-wins from the gbuf compaction research (graph output
verified identical; all delete/incremental paths suite-covered):
1. Release ALL lookup indexes right after build_dump_nodes instead of after
edge building: nothing later in the dump reads them, and at kernel scale
they are ~3.8 GB (hash buckets + key strings + pointer arrays) that
previously coexisted with every dump-side transient — the dump-phase RSS
peak. At 2x-kernel scale this is ~7.5 GB at exactly the OOM point.
2. node_by_id was a hash table keyed on STRDUP'D DECIMAL STRINGS of the
int64 id — ~0.44 GB of buckets and key strings, plus a snprintf + strdup +
hash on every hot find_by_id call. Ids are dense and sequential; a plain
grow-on-demand pointer array replaces all of it with one bounds-checked
load.
3. The per-node and per-edge `project` field was write-only (every reader
uses the store structs or gb->project directly; -Werror confirms) —
8 bytes x 24M instances ~= 192 MB of dead pointers, deleted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing produced a different graph on every run (xfs: node
labels flipping Class/Function/Macro, ~2k edge lines differing across USAGE/
CALLS/IMPORTS/SIMILAR_TO/SEMANTICALLY_RELATED) and lost edges vs repeated
runs. Five root causes, all order- or scheduling-dependence:
1. gbuf QN collisions resolved last-wins in MERGE ORDER: in C a struct, a
function and a macro can share one name, and the QN scheme does not encode
entity kind, so whichever entity a worker merged last overwrote the node's
label/name/lines/props (the flickering node set that fed every pass). Both
cbm_gbuf_upsert_node and the parallel-merge path now keep the SAME entity's
refresh semantics but resolve cross-entity collisions by a canonical
content rule -- lexicographically smallest (label, file_path, start_line,
name) -- and keep the label/name secondary indexes consistent when the
survivor changes (the old overwrite left nodes listed under their original
label, which is how the flicker reached find_by_label consumers).
2. Semantic-edges admission raced per-node max_edges budgets on shared atomic
counters under dynamic scheduling: scoring stays parallel and pure; pairs
carry (func index, candidate rank); one sequential pass admits in canonical
order. funcs[] itself is sorted by qualified name up front (the label-index
scan order is merge order).
3. Import resolution's symbol-name fallback returned the FIRST same-named node
in registration order; now the lexicographically-smallest-QN candidate.
4. Similarity entries inherited merge order into LSH chains + per-node cap
truncation, and pair ownership keyed on run-varying node ids; entries are
sorted by QN and ownership compares QNs (lsh entries carry the QN).
5. Call-neighbor context (semantic vectors + tokens) truncated MAX_CALLEES
subsets in edge-array order; neighbor names are sorted first.
Verified: four consecutive multi-threaded kernel-xfs runs produce byte-
identical sorted node AND edge sets (13341/58598, zero diff). The repro guard
repro_parallel_edge_determinism (MT x K mutual equality) is GREEN; a new
deliberately-RED repro_seq_parallel_equivalence tracks the remaining OPEN bug
that the sequential and parallel pipelines systematically disagree (~3459
USAGE / ~1666 WRITES lines on xfs) -- separate code paths, fix deferred.
Suites: pipeline 216, c_lsp 750, rust_lsp 506, py_lsp 63, java 94, kotlin 78,
php 278, registry 54, store_nodes 56, incremental 161, semantic 32, simhash 24
(one test updated: SIMILAR_TO pair direction is now canonical by QN), mcp 140,
subprocess 17 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to the conditional shared registry: the eager pipeline.c build paid
the O(all_defs) construction and a multi-million-entry registry RSS on every
run that contains rust files, even when every rust file filters to a proper
import subset and the shared registry is never consulted. Build it lazily
instead: the first worker that hits a NULL-filter rust file takes a mutex,
double-checks, builds into a dedicated arena, seals and atomically publishes;
later null-filter files reuse it and subset files never trigger it. Verified
the build fires exactly once (scale log: types=2320 funcs=8860 on the kernel
rust subtree) and the shared-path unit test still resolves identically to the
per-file build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Profiling the kernel showed 4 tiny trait-heavy rust files at ~90 s EACH in
cross-file resolve. Live call trees put the time not in lookups but in
REGISTRY CONSTRUCTION: cbm_pxc_filter_defs_for_file returns NULL for these
files (their path-derived module QNs never match the def_module_qn index
keys), so each one built its own registry over the ENTIRE project def set
(millions of entries at kernel scale).
Original premise corrected during implementation: only 4/87 rust files (on
the kernel rust subtree) hit the NULL-filter path; 83/87 resolve against
proper import-filtered subsets. A registry shared across ALL rust files
would hand the subset files a superset and could add edges. The fix is
therefore CONDITIONAL, byte-identical for every file by construction:
- filter NULL -> ONE shared all_defs registry (built once in pipeline.c,
finalized, read_only-sealed; module_qn=NULL is safe: an instrumented probe
found 0 NULL def_module_qn across all_defs, so the per-file def_mod
fallback never fired anyway). Kills the amplifier.
- filter SUBSET -> today's per-file subset build, unchanged.
The cross-crate Cargo manifest reaches the shared path through
cbm_pxc_get_rust_manifest() (same worker-thread thread-local the fallback
read), preserving cross-crate resolution exactly.
Byte-identity verified empirically: single-threaded sorted edge-set diff on
the kernel rust subtree, fix off vs on -- zero difference outside
SEMANTICALLY_RELATED edges, which self-differ between two runs of the SAME
unfixed binary (the pre-existing semantic non-determinism guarded by the
repro_parallel_edge_determinism RED test). New suite test
rustlsp_shared_registry_resolves_like_per_file exercises the shared path.
Suites: rust_lsp 506, c_lsp 750, registry 54, pipeline 216, java_lsp 94,
kotlin_lsp 78, php_lsp 278, store_nodes 56 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
dump_and_persist_hashes upserted file hashes one statement per file --
~89k autocommit transactions (one fsync each) on the linux kernel. Switch to
the existing cbm_store_upsert_file_hash_batch (same cached upsert inside one
begin/commit, rollback on error): identical tuples and replace semantics, one
fsync. OOM fallback to the per-file path retained.
Also instrument the whole post-dump tail with CBM_PROFILE spans
(persist/1_reopen .. 6_artifact_export) -- this stretch was ~60 s unattributed
on kernel-scale runs; the FTS5 backfill over all node names is now measurable
before any optimization decision.
New deterministic guard: store_file_hash_batch_equals_loop asserts the batched
path produces byte-identical file_hashes rows to the per-file loop. Suites:
pipeline 216, store_nodes 56, store_edges 25, store_search 65, store_arch 52,
store_bulk 3, store_pragmas 7, store_checkpoint 1, incremental 161 -- zero
failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The worker's msg=prof pass/sub-phase report is written only to its supervisor
logfile, which was deleted on every clean exit -- profiling a successful run was
impossible (the log survived only on failure). Keep it when profiling is active
and echo the path via index.supervisor.profile_log. Default behavior unchanged:
clean-run logs are still pruned when CBM_PROFILE is off.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two wall-clock fixes from a linux-kernel (8.5M nodes) profiling run:
1. Extraction back-pressure futility latch: when a full collect+nap cycle ends
still over the RSS budget, the resident floor (accumulated graph + retained
sources) -- not in-flight transients -- holds the memory, so napping again on
every subsequent pull reclaims nothing and only idles workers (measured: one
full cycle per pull, ~390 s at 79% avg CPU across 63k pulls). Latch bp_futile
after the first futile cycle, proceed with the designed soft overshoot, and
re-arm via the cheap per-pull probe once RSS drains under budget. WARN
mem.backpressure.futile exactly once per latch event (0->1 transition).
2. Supervised-worker fast exit: the index worker piecemeal-freed a multi-GB
graph (and store) right before process death (measured: minutes of free()
in the tail). In worker role, skip cbm_pipeline_free and _Exit after the
response is written and flushed; the OS reclaims memory wholesale. In-process
paths (tests, kill switch, degrade) free normally, keeping ASan/LSan
meaningful.
Reproduce-first: pipeline_backpressure_futile_nap_disengages (64-file fixture >
MIN_FILES_FOR_PARALLEL so the parallel gate actually runs, CBM_MEM_BUDGET_MB=1,
deterministic nap-cycle counter; engagement guard cycles>=1 prevents a vacuous
pass). RED on the old gate: 64 cycles (one per pull) > bound cores+2. GREEN with
the latch. Suites: pipeline 216, mcp 140, subprocess 17 -- all pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Follow-up to #881. cbm_build_win_cmdline() returned false from inside its loop on
buffer overflow without NUL-terminating buf, although its header contract states the
buffer holds a string. No live bug — both call sites (cbm_run_win and the UI
http_server index spawn) ignore buf when the function returns false — but a future
caller trusting the documented contract could read an unterminated (possibly
uninitialized) buffer. Set buf[0]='\0' on the overflow path so it is always a valid
string, and correct the header comment to match.
Reproduce-first: the new win_cmdline_overflow_leaves_empty_string test feeds an argv
that overflows a small cap and asserts buf[0]=='\0'. It is RED on the pre-fix code
(buf[0] holds the first quoted byte '"' = 0x22) and GREEN with the fix.
Also tidies two review nits in the same files:
- fix the stale cbm_cmdline_put comment (it returns pos UNCHANGED on overflow;
the overflow is signalled via *ovf, not an advancing pos),
- bounds-guard the reference CommandLineToArgvW parser in the round-trip test.
Refs: #881
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Explicit variable-length Cypher bounds ([:CALLS*1..N]) and the client
`depth` on trace_call_path / detect_changes were passed straight into
cbm_store_bfs, so an arbitrarily large value drove an unbounded traversal
(a DoS on cyclic graphs). The MCP_MAX_DEPTH ceiling was defined but never
applied, and CYP_MAX_DEPTH capped only the unbounded var-length forms.
Add env-configurable ceilings cbm_cypher_max_depth() (CBM_CYPHER_MAX_DEPTH,
default 10) and cbm_mcp_max_depth() (CBM_MCP_MAX_DEPTH, default 15), clamp
both entry points to them, and WARN on clamp -- never a silent truncation.
Remove the now-dead CYP_MAX_DEPTH / MCP_MAX_DEPTH constants.
Reproduce-first: cypher_exec_var_length_explicit_bound_capped (12 -> 10 rows)
and tool_trace_call_path_depth_clamped (depth 1000 -> hop 15).
Closes#887.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
artifact.c shells out to git twice via cbm_popen with the repo path interpolated
straight into the command:
git -C '%s' rev-parse HEAD 2>/dev/null (git_head_hash)
git -C '%s' config merge.ours.driver true 2>/dev/null (ensure_gitattributes)
Both used SINGLE quotes with NO validation of repo_path. cmd.exe does not honor
single quotes, so on Windows a repo path containing a space broke argument grouping
(the artifact commit hash and merge-driver config silently failed for any spaced repo
path), and an embedded quote or shell metacharacter could break out of the intended
argument. This was the only unvalidated git shell interpolation left in the tree.
Different bug class from the CreateProcess argv-quoting fix (which re-parses a single
command STRING back into argv); this is popen/shell command-string quoting — hence a
separate, atomic PR per review.
Fix: validate repo_path with cbm_artifact_repo_path_is_shell_safe() and switch to
double quotes + a per-platform null device, matching git_context.c (the best-hardened
git shell-out): cbm_validate_shell_arg rejects quote / backslash / substitution
metacharacters, and on Windows we also reject the cmd.exe expansion metacharacters
% ! ^. Double quotes are honored by both POSIX sh and cmd.exe, so a legitimate spaced
repo path now works — the concrete regression the single-quote form caused.
The validator is exposed (artifact.h) and guarded by a new test_artifact.c suite:
plain + spaced paths accepted; quote / ; / $() / backtick / pipe injection rejected;
the cmd.exe metacharacters % ! ^ rejected on Windows (allowed on POSIX, where they are
literal inside double quotes).
Refs: #881
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Fixes the Windows-only smoke-windows (standard + ui) failure in the release dry
run, where index_repository reported outcome=exit_nonzero and "Indexing worker
crashed on a file" on big_templated.hpp. It is not the #424 allocator class and
not big_templated.hpp — the file parses fine; the worker never reaches it.
This is the Windows residual of #838. That issue fixed the CLI parser to accept
the supervisor's raw-JSON-positional + --response-out flag layout, verified with a
PowerShell repro where the JSON argument arrives intact. But the real supervisor
spawns the worker via CreateProcessA, whose command-line re-parse strips the JSON's
quotes, so on Windows the now-fixed parser still receives corrupted JSON.
On Windows the index supervisor spawns the worker as
`<self> cli --index-worker index_repository <args_json> ...`, but cbm_run_win
built the CreateProcess command line by wrapping each argv element in bare quotes
with no escaping. Windows re-parses that single string back into argv and strips
the inner quotes, so the JSON argument {"repo_path":"..."} arrived at the child as
{repo_path:...} — invalid JSON. The worker exited non-zero at arg parse
("repo_path is required"); the supervisor classified exit_nonzero and blamed the
last-marked file (big_templated.hpp), producing the misleading crash message.
POSIX is unaffected: cbm_run_posix passes the argv array straight to execv with no
re-parse, which is why only smoke-windows was red while Linux/macOS indexed the
same fixture fine.
Fix: cbm_build_win_cmdline() applies the Microsoft C runtime quoting rules
(quote-wrap + escape embedded quotes and their preceding backslash runs) so the
child re-parses byte-identical argv. It is defined unconditionally (pure string
logic, no Windows headers) and unit-tested on every platform via a round-trip
against a reference CommandLineToArgvW parser.
Also in this change:
- ui/http_server.c: the UI index spawn had a byte-for-byte duplicate of the same
bug (its own "%s" cmdline). Route it through cbm_build_win_cmdline too.
- mcp/index_supervisor.c: log the worker's raw exit_code and KEEP its log on any
failure (delete only on a clean run). Previously the log was always deleted and
only outcome+signal were logged, so a non-zero worker left nothing to diagnose —
the CI blind spot that hid this bug behind a generic message.
A separate hardening of pipeline/artifact.c's single-quoted git shell-outs (a
different bug class — popen command-string quoting, not CreateProcess argv) is
split into a follow-up PR with its own guard test, per review.
Verified on a CLANG64 build: the supervised big_templated.hpp fixture now indexes
clean (reap outcome=clean exit_code=0); subprocess suite 11/11, affected suites
(git_context, index_resilience, ui, httpd, mcp, security, str_util) green.
Refs: #838, #423
Signed-off-by: Flipper <jacobphilipp@ymail.com>
cbm_mem_peak_rss() returned mimalloc's peak_rss (from getrusage ru_maxrss,
KB-granular) while on Linux cbm_mem_rss() reads the live /proc/self/statm
value (page-granular, the RSS-undercount fallback). The two sources can
disagree by a few pages, so a live current read could momentarily exceed the
reported peak and break the definitional peak >= current invariant — flaking
the mem_peak_rss_gte_rss test on the Linux/ARM CI leg (observed as peak 172KB
under current). Reconcile the sources so the reported peak is at least the
current RSS. Not observable on macOS, where both come from mimalloc.
The mem_peak_rss_gte_rss guard now faults in a 32MB buffer before asserting so
the invariant is checked against a non-trivial live current read.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
libgit2 is licensed GPLv2-with-linking-exception, and this project is
deliberately GPL-free (see scripts/license-policy.json). libgit2 was only
an OPTIONAL, faster git-history code path guarded by HAVE_LIBGIT2 and
auto-detected via pkg-config; it always shipped with a popen("git log ...")
fallback that release binaries already used (they were built without
libgit2). Making that popen fallback the SOLE git-history implementation
means there is NO change to shipped behavior.
Changes:
- pass_githistory.c: collapse the HAVE_LIBGIT2 #ifdef so only the popen
parse_git_log remains; drop the now-unused <git2.h>/<time.h> includes.
- cbm.c / cbm.h / main.c / subprocess.c / index_supervisor.h: remove the
libgit2 mimalloc allocator bind and its >=1.7.0 version guards; keep the
tree-sitter + sqlite3 binds; correct the comments.
- Makefile.cbm: remove the optional-libgit2 pkg-config detection block,
the REQUIRE_LIBGIT2 error gate, and all LIBGIT2 CFLAGS/LIBS/FLAGS.
- .github/workflows/_test.yml: drop the REQUIRE_LIBGIT2 test-matrix leg
plus its libgit2-dev/pkg-config apt install (lower CI cost, no new
gating, no trigger change).
- flake.nix: drop libgit2/pkg-config from the dev shell (now-dead dep).
HAVE_LIBGIT2 no longer exists anywhere in the tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #789 (safe features only). Adds backend dead-code classification
(status + in_calls in the layout JSON), a GET /api/repo-info endpoint for GitHub
deep-links, and the frontend dead-code filters, node code preview, and deep-links.
The render-cap revert was dropped (kept 2000 for DEFAULT_MAX_NODES/HARD_MAX_NODES
and GRAPH_RENDER_NODE_LIMIT); the sidebar regex-search refactor was omitted.
Security fixes over the original: repo-info strips credentials from any returned
remote_url; the legitimate https blob-URL construction is allow-listed so the
static gate passes; libgit2 is not re-initialized/shutdown per request (reuses the
process-wide init from cbm_alloc_init); deep-link path segments are URL-encoded.
Co-authored-by: Andy Zehady <azehady@ciroos.ai>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #776's 132460f5. This project sets arena_eager_commit=0 +
purge_decommits=1 + purge_delay=0 in cbm_mem_init to reduce upfront memory, so
mimalloc's committed-page counter reads low. On Linux mi_process_info() never
sets current_rss (vendored/mimalloc/src/prim/unix/prim.c only fills peak_rss),
so current_rss defaults to that low committed counter — cbm_mem_rss() returned a
few MB while true RSS was multiple GB, leaving cbm_mem_over_budget() backpressure,
the memory ceiling, and the host RAM tier blind on Linux.
Prefer os_rss() (/proc/self/statm) as the primary source on Linux; macOS and
Windows are unchanged (their mi_process_info current_rss is accurate via
task_info / GetProcessMemoryInfo). cbm_mem_peak_rss() is untouched. Complements
#752 by making the RAM tiers actually bite on Linux.
Reproduce-first: mem_rss_reflects_external_resident_memory pins mimalloc's
committed counter low with a live mi_malloc, then grows true RSS via a raw 256MB
mmap (invisible to mimalloc); unfixed Linux returns the ~few-MB committed counter
(RED), fixed Linux returns /proc RSS (GREEN). macOS/Windows pass either way, so
the RED manifests on the Linux CI leg.
Co-authored-by: petercoxphoto <info@petercox.ie>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #782 keeping its retire-on-live_count correctness, replacing the
global-lock+linear-scan with aligned-page O(1) lookup + per-page lock-free
remote-free queue (no hot-path contention). Each 64KB slab page is allocated
aligned to its own size, so the owning page of any chunk is recovered by masking
the pointer; a lock-free 3-level radix page map (cold-path writes only) tells
slab_free whether a masked base is a real slab page or a plain heap pointer
without dereferencing an unrelated address. Owner frees hit the thread-local
free list; foreign frees CAS onto that page's lock-free MPSC remote-free stack.
Pages carry refcount = handed-out chunks + owner guard; reclaim/destroy retire a
page that still holds foreign-live chunks and free it when the final chunk
returns, so a page holding a live tree-sitter lexer chunk is never freed under
it. Keeps #782's resolve-worker terminal slab teardown.
Closes the #852 use-after-free and the cross-thread invalid free. Reproduce-first
guards in test_mem.c are RED on main (heap-use-after-free / bad-free under ASan)
and GREEN with this fix.
Closes#852
Supersedes #782
Co-authored-by: SS-42 <47749027+SS-42@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #832 supervisor keystone added a second watcher-registration site in
autoindex_thread's supervised-success branch, but wired it as a direct
cbm_watcher_watch() guarded only by `if (srv->watcher)`. srv->watcher is set
unconditionally (cbm_mcp_server_set_watcher), so that guard does NOT honour the
auto_watch config. PR #849 had routed ALL watcher registration through
register_watcher_if_enabled() (which checks auto_watch_enabled() and logs
watcher.register.skipped reason=auto_watch_off). On a supervisor-marked prod
host with a fresh (unindexed) project and `config set auto_watch false`, the
supervised path therefore registered a background watcher against the user's
explicit opt-out — defeating #849's multi-project containment. The in-process
twin ~20 lines below already used register_watcher_if_enabled() correctly.
Fix: replace the ungated block in the supervised-success branch with
register_watcher_if_enabled(srv), matching the in-process branch. Surrounding
supervised-success logging is unchanged.
Reproduce-first: mcp_auto_watch_false_skips_supervised_autoindex_issue853 drives
the real public entry initialize -> maybe_auto_index -> autoindex_thread on a
supervisor-marked host (kill switch off) with a FRESH project and
auto_watch=false, in a fork+alarm-bounded child (like the #832/#845 gate tests).
cbm_mcp_server_free() joins the autoindex thread so the registration decision has
run before the watch count is read. It asserts spawn_count advanced (so the probe
cannot pass vacuously by skipping the supervised branch) and watch_count == 0.
RED on the unfixed code (watch_count == 1, exit 61); GREEN after the fix
(register_watcher_if_enabled skips → watcher.register.skipped reason=auto_watch_off).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #685 (nguyentamdat) rebased onto current main, plus two
research-driven refinements and a genuine reproduce-first guard.
The parallel extract retains each file's source text so the fused cross-file
LSP resolve can re-parse it. That retention is transient but a peak-RSS driver.
On main the caps are flat (100 MiB/file, 2 GiB total) and a file over the cap is
silently unretained AND its cross-file resolution is skipped -- a graph-quality
gap. This change bounds retention AND keeps every cross-file edge.
- Source-text cap as a FLOOR, not just a ceiling: retention total defaults to
min(cbm_mem_budget()/8, 1 GiB), per-file min(32 MiB, total). Following the
rust-analyzer memory model, the RAM-derived default is clamped to a small
absolute ceiling so a huge-RAM host does not hold tens of GB it would re-read
cheaply. Both caps env-overridable via CBM_RETAIN_TOTAL_MB /
CBM_RETAIN_PER_FILE_MB (limits.c convention); ceilings bound only the
auto-derived default, never an explicit operator/caller choice. A dropped file
emits one index.retain_capped WARN per run.
- Bounded re-read fallback (the correctness guarantee): resolve_worker re-reads
an unretained file's source on demand (bounded, freed immediately) instead of
skipping resolution, wired at every cross-LSP site that consumes source. The
cap now only trades retained RAM for a bounded re-read, never a lost edge.
- cbm_parallel_extract_ex + opts struct (cbm_parallel_extract is now a wrapper
passing NULL -> env-derived defaults); malloc/calloc NULL-check hardening.
Reproduce-first: parallel_cross_file_reread_preserves_unretained_edges uses a
Java<->Kotlin pair whose cross-file lsp edges are genuinely source-dependent;
the edges are lost when the caller is unretained and the fallback is absent
(RED), present with it (GREEN), with a retained CONTROL scenario proving
non-vacuity. #685's original Python red test was a false guard (per-file py_lsp
already resolves those calls) and is replaced. Peak-bound guards
(retained_bytes <= total_cap; retain_sources=false retains nothing) in test_mem.c.
Verify: make -f Makefile.cbm cbm && make -f Makefile.cbm lint-ci; test-runner
parallel pipeline incremental py_lsp ts_lsp java_lsp kotlin_lsp c_lsp cs_lsp
go_lsp rust_lsp mem -> 2323 passed.
Co-authored-by: nguyentamdat <nguyentamdat@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The long-lived MCP server ran the full index pipeline in-process on two
background paths -- the session auto-index (autoindex_thread, mcp.c) and the
watcher re-index (watcher_index_fn, main.c). The parallel pipeline's worker
threads abandon their mimalloc pages at thread exit, and mimalloc v3 defaults
page_reclaim_on_free=0, so those pages are never reclaimed when the main thread
later frees their blocks (mi_collect cannot touch abandoned pages either). RSS
then ratchets across re-index cycles and never comes back down (#832).
The index_repository tool path already avoided this by running the index in a
supervised worker subprocess: on child exit the kernel reclaims 100% of the
child's RSS. Route the two background paths through that same supervised runner:
- supervisor_invalidate_store() is now NULL-safe (the watcher path has no MCP
server / cached store).
- New static index_run_supervised_path() builds {"repo_path": root} and reuses
the existing spawn + skip-and-continue recovery loop (no duplication). New
public cbm_mcp_index_run_supervised_path() is the srv-less entry main.c calls.
- autoindex_thread and watcher_index_fn gate on cbm_index_supervisor_should_wrap():
supervised on a marked host with the kill switch off, else fall through to the
unchanged in-process pipeline (kill switch / spawn-failure degrade). The
watcher keeps its non-blocking try_lock gate + g_shutdown check exactly; the
lock still serialises re-indexes. cbm_mem_collect() stays on the in-process
branch only.
Also fold in the in-process allocator fallback: cbm_mem_init sets
mi_option_page_reclaim_on_free=1, restoring v2 behaviour so any path that stays
in-process (kill switch, degrade, embedders) also reclaims abandoned worker
pages. The subprocess is the primary cure (child exit returns RSS every cycle);
this is the fallback.
Tests: a deterministic gating guard (test_mcp.c) proves the background paths now
spawn a worker child and index the fixture (RED when routed in-process:
spawn_count unchanged; GREEN after). The test-runner acts as a real in-process
worker when spawned as `cli --index-worker` (public APIs only, no production
seam). A non-gating prod-binary RSS reproduction (tests/repro/issue832_rss.py)
shows the supervised server flat at ~10 MB vs ~131 MB in-process.
Scope: JobObject, the #841 spurious-trigger fix, retention caps, budget tiering
and the backpressure busy-spin are follow-ups.
Refs #832, #841.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
LA-10