The no-skips lint gate forbids bare SKIP(); the parallel-recovery guard
is genuinely platform-specific (POSIX fork harness — Windows coverage
comes from the smoke invariants), so it uses SKIP_PLATFORM like the
neighboring supervisor guards.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An 81k-file TypeScript corpus previously never finished indexing. Three
stacked defects, found by stack-sampling the live grind and reading the
supervisor's quarantine records:
1) walk_defs pushed children via index-based ts_node_child(i), which is
O(i) per call in tree-sitter — O(n^2) per wide node. A 3.5 MB file
with ~580k flat comment siblings needed ~1.7e11 iterator steps and
hung extraction past the supervisor's quiet-timeout. All six descent
sites (including the default loop) now collect children linearly with
a TSTreeCursor (wd_collect_children); small nodes keep the direct
index path. Guard: extract_wide_flat_file_is_linear (400k comment
siblings — the monster's exact shape — bounded at 30s).
2) The sequential cross-LSP driver handed the FULL def list to a full
per-file registry build+finalize — O(files x defs); 74% of samples
sat in build_qn_index. The parallel resolve worker's dispatch
(module-def filter -> shared prebuilt registries -> filtered
fallback) is now extracted into cbm_pxc_dispatch_file and drives
BOTH pipelines — one path, one semantics. The shared registries
live in a caller-owned arena that outlives the calls pass
(resolved_calls borrow registry strings; freeing at pass end was a
use-after-free ASan caught in pass_calls). Rust is exempt from the
def filter: Cargo-manifest cross-crate resolution needs defs from
OTHER workspace crates that the own+imports filter drops (this
starvation predated the unification on the parallel path). Guard:
pipeline_seq_ts_cross_uses_shared_registry (full builds 40 -> 0,
cross-file edge preserved).
3) The crash supervisor's recovery re-ran the worker SINGLE-THREADED to
keep one exact marker. At scale that fell into the sequential crawl,
was killed as a hang mid-pass, and the stale extraction marker got
four innocent files quarantined, one 15-minute retry at a time.
Recovery (and the terminal partial run) now re-run PARALLEL; the
marker is an append journal ("S <rel>" / "D <rel>", written by
extraction and both resolve drivers), the suspect set is the open-S
in-flight set, and a file is quarantined only when it recurs across
two consecutive failed runs — one per round, oldest open S first;
disjoint consecutive sets stop the loop rather than blame an
innocent. Guard: index_recovery_parallel_quarantines_crasher
(single-threaded spawn count must stay zero; verified RED against
the old loop).
Also: parse_ts_type_text gains a dynamic work budget proportional to
the input (1M + 64x source bytes, CBM_TS_TYPE_BUDGET override, warn
once then degrade to unknown) replacing unbounded recursion cost.
Measured on microsoft/TypeScript (81,398 files): infinite -> 33s cold
full index, 295k nodes / 779k edges, zero dangling edges, zero files
skipped — the monster file itself now indexes cleanly. Full 9-language
sweep green with no quality flags; kernel unchanged at 390s / 8.5M
nodes / 0 dangling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The supervised-worker fast path deliberately skips the pipeline
teardown (the OS reclaims the process wholesale) and pairs that with
_Exit, which bypasses atexit/LeakSanitizer. The production worker in
run_cli has both halves; the test binary's worker emulation
(tf_maybe_run_index_worker) only had the first: it returned normally
through main(), LeakSanitizer ran at exit, reported the intentionally
unfreed pipeline and forced exit code 1 — so the supervisor read a
healthy index as worker_failed and IDX832 went red on exactly the
platforms whose ASan builds run a leak pass (all four Linux legs;
macOS/Windows ASan have none). Reproduced and verified fixed in an
ubuntu:22.04 arm64 container: reap outcome=clean, mcp suite 140/0.
Mirror the production exit: deliver the response file, fflush, _Exit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_mem_init is init-once (guarded by g_initialized), so the
backpressure test's setenv(CBM_MEM_BUDGET_MB=1) + re-init 'restore'
was a silent no-op: whichever init ran first fixed the budget for the
whole process. In the full runner this test's init won, locking a 1 MB
budget that leaked into every later budget consumer —
mem_over_budget_low_rss red on every platform, and all subsequent
pipeline tests silently running with backpressure permanently engaged.
Add cbm_mem_set_budget_for_tests(): writes the budget directly without
touching the init guard. The test now saves the caller-visible budget,
sets 1 MB via the hook, and restores before asserting. Env dance
removed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The semantic pass kept four dense 768-float vectors resident per function
(ri/api/type/deco = 12.3 KB per func, ~9.4 GB transient on the linux kernel)
solely so candidate pairs could be cosine-scored after LSH. Replace the
resident floats with 4-bit quantized codes (~0.5 KB each, ~6x): new
src/semantic/rotsq.{h,c} — rotated scalar quantization from the RaBitQ
family of algorithms (Gao et al., SIGMOD 2024/2025), implemented from the
papers in plain C11 (no Eigen/library dependency): a deterministic
XXH3-seeded +-1 diagonal followed by a Fast Walsh-Hadamard rotation
(768 -> 1024 padded) spreads vector mass so per-vector scalar quantization
is near-optimal; the inner product is recovered from the codes with the
exact SQ expansion (one u8 integer dot + four multiplies) — a deterministic
pure function of the codes. Named rotsq, not rabitq: this is the family
core, not the papers' exact codebook construction.
Producers build each dense vector in a worker-local buffer, encode, and
discard; the int8 qvec export is unchanged. LSH signatures are computed
from DEQUANTIZED codes against hyperplanes in the rotated basis — random
hyperplanes are basis-agnostic, so no dense originals are retained and the
phase structure is untouched.
Estimator quality gated by sem_rotsq_ip_error_bounds (seeded vectors:
mean error < 1%, max < 4% of unit scale, ASan-green). Effect on output:
multi-threaded determinism preserved (xfs: two runs byte-identical);
SEMANTICALLY_RELATED edges shift once as scores move within ~1% of the
0.75 threshold (xfs: ~130 -> 203; sampled new pairs are high quality —
alloc/free, read-verify and collapse/insert families).
Suites: semantic 33 (incl. the new estimator gate), simhash 24,
pipeline 216 — zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing produced a different graph on every run (xfs: node
labels flipping Class/Function/Macro, ~2k edge lines differing across USAGE/
CALLS/IMPORTS/SIMILAR_TO/SEMANTICALLY_RELATED) and lost edges vs repeated
runs. Five root causes, all order- or scheduling-dependence:
1. gbuf QN collisions resolved last-wins in MERGE ORDER: in C a struct, a
function and a macro can share one name, and the QN scheme does not encode
entity kind, so whichever entity a worker merged last overwrote the node's
label/name/lines/props (the flickering node set that fed every pass). Both
cbm_gbuf_upsert_node and the parallel-merge path now keep the SAME entity's
refresh semantics but resolve cross-entity collisions by a canonical
content rule -- lexicographically smallest (label, file_path, start_line,
name) -- and keep the label/name secondary indexes consistent when the
survivor changes (the old overwrite left nodes listed under their original
label, which is how the flicker reached find_by_label consumers).
2. Semantic-edges admission raced per-node max_edges budgets on shared atomic
counters under dynamic scheduling: scoring stays parallel and pure; pairs
carry (func index, candidate rank); one sequential pass admits in canonical
order. funcs[] itself is sorted by qualified name up front (the label-index
scan order is merge order).
3. Import resolution's symbol-name fallback returned the FIRST same-named node
in registration order; now the lexicographically-smallest-QN candidate.
4. Similarity entries inherited merge order into LSH chains + per-node cap
truncation, and pair ownership keyed on run-varying node ids; entries are
sorted by QN and ownership compares QNs (lsh entries carry the QN).
5. Call-neighbor context (semantic vectors + tokens) truncated MAX_CALLEES
subsets in edge-array order; neighbor names are sorted first.
Verified: four consecutive multi-threaded kernel-xfs runs produce byte-
identical sorted node AND edge sets (13341/58598, zero diff). The repro guard
repro_parallel_edge_determinism (MT x K mutual equality) is GREEN; a new
deliberately-RED repro_seq_parallel_equivalence tracks the remaining OPEN bug
that the sequential and parallel pipelines systematically disagree (~3459
USAGE / ~1666 WRITES lines on xfs) -- separate code paths, fix deferred.
Suites: pipeline 216, c_lsp 750, rust_lsp 506, py_lsp 63, java 94, kotlin 78,
php 278, registry 54, store_nodes 56, incremental 161, semantic 32, simhash 24
(one test updated: SIMILAR_TO pair direction is now canonical by QN), mcp 140,
subprocess 17 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Profiling the kernel showed 4 tiny trait-heavy rust files at ~90 s EACH in
cross-file resolve. Live call trees put the time not in lookups but in
REGISTRY CONSTRUCTION: cbm_pxc_filter_defs_for_file returns NULL for these
files (their path-derived module QNs never match the def_module_qn index
keys), so each one built its own registry over the ENTIRE project def set
(millions of entries at kernel scale).
Original premise corrected during implementation: only 4/87 rust files (on
the kernel rust subtree) hit the NULL-filter path; 83/87 resolve against
proper import-filtered subsets. A registry shared across ALL rust files
would hand the subset files a superset and could add edges. The fix is
therefore CONDITIONAL, byte-identical for every file by construction:
- filter NULL -> ONE shared all_defs registry (built once in pipeline.c,
finalized, read_only-sealed; module_qn=NULL is safe: an instrumented probe
found 0 NULL def_module_qn across all_defs, so the per-file def_mod
fallback never fired anyway). Kills the amplifier.
- filter SUBSET -> today's per-file subset build, unchanged.
The cross-crate Cargo manifest reaches the shared path through
cbm_pxc_get_rust_manifest() (same worker-thread thread-local the fallback
read), preserving cross-crate resolution exactly.
Byte-identity verified empirically: single-threaded sorted edge-set diff on
the kernel rust subtree, fix off vs on -- zero difference outside
SEMANTICALLY_RELATED edges, which self-differ between two runs of the SAME
unfixed binary (the pre-existing semantic non-determinism guarded by the
repro_parallel_edge_determinism RED test). New suite test
rustlsp_shared_registry_resolves_like_per_file exercises the shared path.
Suites: rust_lsp 506, c_lsp 750, registry 54, pipeline 216, java_lsp 94,
kotlin_lsp 78, php_lsp 278, store_nodes 56 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing loses edges vs single-threaded and is
run-to-run non-deterministic (fs/xfs: ST 58573 edges vs MT 58561 on the
first divergent run; rust subtree flickers 44445-44857 across runs).
Discovered while hardening perf-change equivalence checks; also the
reason raw node/edge counts are not a valid equivalence gate.
The guard indexes a corpus once single-threaded (reference) and K=4x
multi-threaded (fresh store each run) and asserts the SORTED
(source_qn|type|target_qn) edge sets are all identical. RED today;
GREEN if and only if MT output is deterministic and equals ST.
Calibration note (false-guard avoidance): three escalating synthetic C
corpora (300/500/600 files, dense cross-file calls, token-identical
bodies) stayed fully deterministic -- the similarity pass emits zero
edges on synthetic C, and the race needs real-code edge diversity. A
synthetic guard would pass on buggy code, so this repro uses the
smallest real corpus that reproduces (kernel fs/xfs) and SKIPs when the
corpus is absent. Suspected root causes documented in the test header
(per-worker edge-buffer merge, gbuf edge dedup check-then-insert,
symmetric SIMILAR_TO emission, resolve ordering). Fix deferred; this
test is its guard.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
dump_and_persist_hashes upserted file hashes one statement per file --
~89k autocommit transactions (one fsync each) on the linux kernel. Switch to
the existing cbm_store_upsert_file_hash_batch (same cached upsert inside one
begin/commit, rollback on error): identical tuples and replace semantics, one
fsync. OOM fallback to the per-file path retained.
Also instrument the whole post-dump tail with CBM_PROFILE spans
(persist/1_reopen .. 6_artifact_export) -- this stretch was ~60 s unattributed
on kernel-scale runs; the FTS5 backfill over all node names is now measurable
before any optimization decision.
New deterministic guard: store_file_hash_batch_equals_loop asserts the batched
path produces byte-identical file_hashes rows to the per-file loop. Suites:
pipeline 216, store_nodes 56, store_edges 25, store_search 65, store_arch 52,
store_bulk 3, store_pragmas 7, store_checkpoint 1, incremental 161 -- zero
failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Rust cross-file resolver's trait-method, sole-trait-impl, and free-func
fallback loops scanned the ENTIRE type/func registry per call site on a miss.
On kernel-scale registries (~10^6 entries, macro-inflated) four tiny trait-heavy
rust files burned ~96 s EACH (~200k eval-capped call sites x full-registry
strcmp scans).
Fix: two hash indexes built at registry finalize, mirroring the existing
bucket+entries idiom -- fnv1a(bare(embedded_type)) -> chain of declaring type
indices (trait -> implementers), and fnv1a(short_name) -> chain of free-func
indices. Chains preserve ascending registry order (first-match tie-break
unchanged); allocation-free iterators fall back to the post-finalize tail and
to a full scan on unfinalized registries, so the candidate set is identical in
every mode. The four rust loops now walk the iterator and re-apply their exact
original predicates -- byte-identical results, loop bodies untouched.
Verified: graph equivalence on the kernel rust/ subtree (nodes 9654, edges
44798 identical with and without the change); registry_short_name_indexes unit
test pins iterator contents/order/dedup; suites c_lsp 750, rust_lsp 505,
registry 54, pipeline 216 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_register_instance_field mutated the shared Tier-2 registry through direct
struct writes that bypass the read_only seal: it overwrote field-type pointers
in place and reassigned the sealed type entry's field arrays to memory owned by
the per-file arena. In the fused parallel resolve path (one sealed registry
shared by all resolve workers) that is a cross-thread data race, and it leaves
the shared registry pointing into arenas that die with their file -- a
cross-file use-after-free. The count-based seal guard could not see it because
type/func counts never change.
Python extraction emits no instance fields, so resolve-time discovery is the
ONLY source of self.x / PEP-526 fields; a plain read_only gate would silently
drop real attribute-chain CALLS edges. Fix instead routes discovery into a
per-file field overlay on PyLSPContext (arena-backed, dies with the file) and
has py_lookup_field_depth consult shared-base + overlay at the exact precedence
point of the direct field scan. Mutable per-file registries keep the original
in-place write, byte-identical.
Reproduce-first, both halves pinned:
- seal_py_shared_registry_readonly_fields: RED on the bug (field count 0->1 on
a sealed registry), GREEN with the overlay.
- pylsp_fused_self_attr_chain_via_overlay: GREEN with the overlay, RED under a
simulated plain gate (self.b.work() CALLS edge lost) -- proven both ways.
Suites: c_lsp 749, py_lsp 63+45 stress/scale/bench, pipeline 216, registry 54,
scope 10, type_rep 19 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two wall-clock fixes from a linux-kernel (8.5M nodes) profiling run:
1. Extraction back-pressure futility latch: when a full collect+nap cycle ends
still over the RSS budget, the resident floor (accumulated graph + retained
sources) -- not in-flight transients -- holds the memory, so napping again on
every subsequent pull reclaims nothing and only idles workers (measured: one
full cycle per pull, ~390 s at 79% avg CPU across 63k pulls). Latch bp_futile
after the first futile cycle, proceed with the designed soft overshoot, and
re-arm via the cheap per-pull probe once RSS drains under budget. WARN
mem.backpressure.futile exactly once per latch event (0->1 transition).
2. Supervised-worker fast exit: the index worker piecemeal-freed a multi-GB
graph (and store) right before process death (measured: minutes of free()
in the tail). In worker role, skip cbm_pipeline_free and _Exit after the
response is written and flushed; the OS reclaims memory wholesale. In-process
paths (tests, kill switch, degrade) free normally, keeping ASan/LSan
meaningful.
Reproduce-first: pipeline_backpressure_futile_nap_disengages (64-file fixture >
MIN_FILES_FOR_PARALLEL so the parallel gate actually runs, CBM_MEM_BUDGET_MB=1,
deterministic nap-cycle counter; engagement guard cycles>=1 prevents a vacuous
pass). RED on the old gate: 64 cycles (one per pull) > bound cores+2. GREEN with
the latch. Suites: pipeline 216, mcp 140, subprocess 17 -- all pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Follow-up to #881. cbm_build_win_cmdline() returned false from inside its loop on
buffer overflow without NUL-terminating buf, although its header contract states the
buffer holds a string. No live bug — both call sites (cbm_run_win and the UI
http_server index spawn) ignore buf when the function returns false — but a future
caller trusting the documented contract could read an unterminated (possibly
uninitialized) buffer. Set buf[0]='\0' on the overflow path so it is always a valid
string, and correct the header comment to match.
Reproduce-first: the new win_cmdline_overflow_leaves_empty_string test feeds an argv
that overflows a small cap and asserts buf[0]=='\0'. It is RED on the pre-fix code
(buf[0] holds the first quoted byte '"' = 0x22) and GREEN with the fix.
Also tidies two review nits in the same files:
- fix the stale cbm_cmdline_put comment (it returns pos UNCHANGED on overflow;
the overflow is signalled via *ovf, not an advancing pos),
- bounds-guard the reference CommandLineToArgvW parser in the round-trip test.
Refs: #881
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Explicit variable-length Cypher bounds ([:CALLS*1..N]) and the client
`depth` on trace_call_path / detect_changes were passed straight into
cbm_store_bfs, so an arbitrarily large value drove an unbounded traversal
(a DoS on cyclic graphs). The MCP_MAX_DEPTH ceiling was defined but never
applied, and CYP_MAX_DEPTH capped only the unbounded var-length forms.
Add env-configurable ceilings cbm_cypher_max_depth() (CBM_CYPHER_MAX_DEPTH,
default 10) and cbm_mcp_max_depth() (CBM_MCP_MAX_DEPTH, default 15), clamp
both entry points to them, and WARN on clamp -- never a silent truncation.
Remove the now-dead CYP_MAX_DEPTH / MCP_MAX_DEPTH constants.
Reproduce-first: cypher_exec_var_length_explicit_bound_capped (12 -> 10 rows)
and tool_trace_call_path_depth_clamped (depth 1000 -> hop 15).
Closes#887.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
artifact.c shells out to git twice via cbm_popen with the repo path interpolated
straight into the command:
git -C '%s' rev-parse HEAD 2>/dev/null (git_head_hash)
git -C '%s' config merge.ours.driver true 2>/dev/null (ensure_gitattributes)
Both used SINGLE quotes with NO validation of repo_path. cmd.exe does not honor
single quotes, so on Windows a repo path containing a space broke argument grouping
(the artifact commit hash and merge-driver config silently failed for any spaced repo
path), and an embedded quote or shell metacharacter could break out of the intended
argument. This was the only unvalidated git shell interpolation left in the tree.
Different bug class from the CreateProcess argv-quoting fix (which re-parses a single
command STRING back into argv); this is popen/shell command-string quoting — hence a
separate, atomic PR per review.
Fix: validate repo_path with cbm_artifact_repo_path_is_shell_safe() and switch to
double quotes + a per-platform null device, matching git_context.c (the best-hardened
git shell-out): cbm_validate_shell_arg rejects quote / backslash / substitution
metacharacters, and on Windows we also reject the cmd.exe expansion metacharacters
% ! ^. Double quotes are honored by both POSIX sh and cmd.exe, so a legitimate spaced
repo path now works — the concrete regression the single-quote form caused.
The validator is exposed (artifact.h) and guarded by a new test_artifact.c suite:
plain + spaced paths accepted; quote / ; / $() / backtick / pipe injection rejected;
the cmd.exe metacharacters % ! ^ rejected on Windows (allowed on POSIX, where they are
literal inside double quotes).
Refs: #881
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Fixes the Windows-only smoke-windows (standard + ui) failure in the release dry
run, where index_repository reported outcome=exit_nonzero and "Indexing worker
crashed on a file" on big_templated.hpp. It is not the #424 allocator class and
not big_templated.hpp — the file parses fine; the worker never reaches it.
This is the Windows residual of #838. That issue fixed the CLI parser to accept
the supervisor's raw-JSON-positional + --response-out flag layout, verified with a
PowerShell repro where the JSON argument arrives intact. But the real supervisor
spawns the worker via CreateProcessA, whose command-line re-parse strips the JSON's
quotes, so on Windows the now-fixed parser still receives corrupted JSON.
On Windows the index supervisor spawns the worker as
`<self> cli --index-worker index_repository <args_json> ...`, but cbm_run_win
built the CreateProcess command line by wrapping each argv element in bare quotes
with no escaping. Windows re-parses that single string back into argv and strips
the inner quotes, so the JSON argument {"repo_path":"..."} arrived at the child as
{repo_path:...} — invalid JSON. The worker exited non-zero at arg parse
("repo_path is required"); the supervisor classified exit_nonzero and blamed the
last-marked file (big_templated.hpp), producing the misleading crash message.
POSIX is unaffected: cbm_run_posix passes the argv array straight to execv with no
re-parse, which is why only smoke-windows was red while Linux/macOS indexed the
same fixture fine.
Fix: cbm_build_win_cmdline() applies the Microsoft C runtime quoting rules
(quote-wrap + escape embedded quotes and their preceding backslash runs) so the
child re-parses byte-identical argv. It is defined unconditionally (pure string
logic, no Windows headers) and unit-tested on every platform via a round-trip
against a reference CommandLineToArgvW parser.
Also in this change:
- ui/http_server.c: the UI index spawn had a byte-for-byte duplicate of the same
bug (its own "%s" cmdline). Route it through cbm_build_win_cmdline too.
- mcp/index_supervisor.c: log the worker's raw exit_code and KEEP its log on any
failure (delete only on a clean run). Previously the log was always deleted and
only outcome+signal were logged, so a non-zero worker left nothing to diagnose —
the CI blind spot that hid this bug behind a generic message.
A separate hardening of pipeline/artifact.c's single-quoted git shell-outs (a
different bug class — popen command-string quoting, not CreateProcess argv) is
split into a follow-up PR with its own guard test, per review.
Verified on a CLANG64 build: the supervised big_templated.hpp fixture now indexes
clean (reap outcome=clean exit_code=0); subprocess suite 11/11, affected suites
(git_context, index_resilience, ui, httpd, mcp, security, str_util) green.
Refs: #838, #423
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Closes the gap flagged in the distill review (test_security.c: "NOT proven
here: the full UI repro — listening socket + MSYS2 handle walk under a
single-threaded server"). Test-only; no production code changes.
Guard 1 — deterministic socket-isolation probe (any Windows, RED-able):
popen_isolates_listening_socket (tests/test_security.c) opens a real
inheritable AFD/listening socket — the exact handle class that deadlocked
git in #798 — then spawns THIS binary through cbm_popen (a cmd.exe
grandchild, git's spawn shape) in a new __cbm_sockprobe re-exec mode
(tests/test_main.c). The child reports via exit code whether that socket
handle is live in its address space: isolated spawn -> getsockopt fails ->
exit 0 (GREEN); raw-_popen regression leaks it transitively through
cmd.exe -> getsockopt succeeds -> exit 42 (RED). Verified RED with a local
spawn-inherits-all revert, GREEN with the isolated spawn.
Guard 2 — end-to-end liveness under live UI sockets (tests/test_httpd.c):
- ui_server_list_projects_responds_under_watchdog: POST /rpc list_projects
to the running single-threaded UI server with a client SO_RCVTIMEO
watchdog; a wedge -> no 200 -> hard FAIL, never an infinite CI hang.
- git_context_resolve_no_hang_under_live_ui_sockets: while the UI server
holds live listening/AFD handles in-process, run cbm_git_context_resolve
(the exact path list_projects takes) on a worker thread under a
WaitForSingleObject(30s) watchdog; a hang -> FAIL. SKIPs where git cannot
init a repo via system().
All three run in the existing security/httpd suites on the test-windows CI
leg; no new build wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018g9f9jr5S46BfKaDvDykBp
Signed-off-by: Flipper <jacobphilipp@ymail.com>
mem_rss_reflects_external_resident_memory asserted cbm_mem_rss() >= 128MB
after touching a 256MB region, but on Windows cbm_mem_rss() reads
WorkingSetSize (GetProcessMemoryInfo), which the OS trims under memory
pressure — a stressed windows-11-arm runner kept only ~97MB resident and
the leg (a required gate) flaked. Re-touch the region immediately before
measuring and assert a 32MB Windows threshold that survives aggressive
trimming while staying far above the ~1MB mimalloc warm buffer, so it still
guards the real regression (a broken small-counter RSS). The Linux #else
branch keeps its 128MB assertion (the actual undercount it was written for).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_mem_peak_rss() returned mimalloc's peak_rss (from getrusage ru_maxrss,
KB-granular) while on Linux cbm_mem_rss() reads the live /proc/self/statm
value (page-granular, the RSS-undercount fallback). The two sources can
disagree by a few pages, so a live current read could momentarily exceed the
reported peak and break the definitional peak >= current invariant — flaking
the mem_peak_rss_gte_rss test on the Linux/ARM CI leg (observed as peak 172KB
under current). Reconcile the sources so the reported peak is at least the
current RSS. Not observable on macOS, where both come from mimalloc.
The mem_peak_rss_gte_rss guard now faults in a 32MB buffer before asserting so
the invariant is checked against a non-trivial live current read.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The incr_full_index guard measured 2050-2072MB peak for the 1100-file
Python index across CI runs after the retention/source-text-cap and
RAM-tiering work shifted the x86 baseline up. The old 2048MB limit sat
right on the line and flaked marginally (2050 vs 2048). Raise the x86
base budget to 2304MB to restore headroom; a genuine leak (GBs over)
still trips it, and the ARM/large-page bumps to 2816 are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #789 (safe features only). Adds backend dead-code classification
(status + in_calls in the layout JSON), a GET /api/repo-info endpoint for GitHub
deep-links, and the frontend dead-code filters, node code preview, and deep-links.
The render-cap revert was dropped (kept 2000 for DEFAULT_MAX_NODES/HARD_MAX_NODES
and GRAPH_RENDER_NODE_LIMIT); the sidebar regex-search refactor was omitted.
Security fixes over the original: repo-info strips credentials from any returned
remote_url; the legitimate https blob-URL construction is allow-listed so the
static gate passes; libgit2 is not re-initialized/shutdown per request (reuses the
process-wide init from cbm_alloc_init); deep-link path segments are URL-encoded.
Co-authored-by: Andy Zehady <azehady@ciroos.ai>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #776's 132460f5. This project sets arena_eager_commit=0 +
purge_decommits=1 + purge_delay=0 in cbm_mem_init to reduce upfront memory, so
mimalloc's committed-page counter reads low. On Linux mi_process_info() never
sets current_rss (vendored/mimalloc/src/prim/unix/prim.c only fills peak_rss),
so current_rss defaults to that low committed counter — cbm_mem_rss() returned a
few MB while true RSS was multiple GB, leaving cbm_mem_over_budget() backpressure,
the memory ceiling, and the host RAM tier blind on Linux.
Prefer os_rss() (/proc/self/statm) as the primary source on Linux; macOS and
Windows are unchanged (their mi_process_info current_rss is accurate via
task_info / GetProcessMemoryInfo). cbm_mem_peak_rss() is untouched. Complements
#752 by making the RAM tiers actually bite on Linux.
Reproduce-first: mem_rss_reflects_external_resident_memory pins mimalloc's
committed counter low with a live mi_malloc, then grows true RSS via a raw 256MB
mmap (invisible to mimalloc); unfixed Linux returns the ~few-MB committed counter
(RED), fixed Linux returns /proc RSS (GREEN). macOS/Windows pass either way, so
the RED manifests on the Linux CI leg.
Co-authored-by: petercoxphoto <info@petercox.ie>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #782 keeping its retire-on-live_count correctness, replacing the
global-lock+linear-scan with aligned-page O(1) lookup + per-page lock-free
remote-free queue (no hot-path contention). Each 64KB slab page is allocated
aligned to its own size, so the owning page of any chunk is recovered by masking
the pointer; a lock-free 3-level radix page map (cold-path writes only) tells
slab_free whether a masked base is a real slab page or a plain heap pointer
without dereferencing an unrelated address. Owner frees hit the thread-local
free list; foreign frees CAS onto that page's lock-free MPSC remote-free stack.
Pages carry refcount = handed-out chunks + owner guard; reclaim/destroy retire a
page that still holds foreign-live chunks and free it when the final chunk
returns, so a page holding a live tree-sitter lexer chunk is never freed under
it. Keeps #782's resolve-worker terminal slab teardown.
Closes the #852 use-after-free and the cross-thread invalid free. Reproduce-first
guards in test_mem.c are RED on main (heap-use-after-free / bad-free under ASan)
and GREEN with this fix.
Closes#852
Supersedes #782
Co-authored-by: SS-42 <47749027+SS-42@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The idx832/idx853 fork-harness helpers are used only in the POSIX (#else)
branch of their tests; on Windows (SKIP_PLATFORM) they are unused →
-Werror,-Wunused-function on the CLANG64 leg. Guard the definitions to
match their call sites.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Inline Rust test fns (#[test], #[tokio::test], #[async_std::test],
#[actix_rt::test], #[test_case::case]) inside a regular .rs file were indexed
as ordinary Functions (is_test=false) and leaked past the store.c is_test!=1
filter into graph/agent context — test detection was file-path-based only
(cbm_is_test_file: *_test.rs / test_*). Add rust_def_is_test(decorators)
(mirroring rust_cfg_qualified_name's scan) and set def.is_test in
extract_func_def's Rust branch.
Closes#855.
Signed-off-by: Greg Tiller <tiller@dal.ca>
The #832 supervisor keystone added a second watcher-registration site in
autoindex_thread's supervised-success branch, but wired it as a direct
cbm_watcher_watch() guarded only by `if (srv->watcher)`. srv->watcher is set
unconditionally (cbm_mcp_server_set_watcher), so that guard does NOT honour the
auto_watch config. PR #849 had routed ALL watcher registration through
register_watcher_if_enabled() (which checks auto_watch_enabled() and logs
watcher.register.skipped reason=auto_watch_off). On a supervisor-marked prod
host with a fresh (unindexed) project and `config set auto_watch false`, the
supervised path therefore registered a background watcher against the user's
explicit opt-out — defeating #849's multi-project containment. The in-process
twin ~20 lines below already used register_watcher_if_enabled() correctly.
Fix: replace the ungated block in the supervised-success branch with
register_watcher_if_enabled(srv), matching the in-process branch. Surrounding
supervised-success logging is unchanged.
Reproduce-first: mcp_auto_watch_false_skips_supervised_autoindex_issue853 drives
the real public entry initialize -> maybe_auto_index -> autoindex_thread on a
supervisor-marked host (kill switch off) with a FRESH project and
auto_watch=false, in a fork+alarm-bounded child (like the #832/#845 gate tests).
cbm_mcp_server_free() joins the autoindex thread so the registration decision has
run before the watch count is read. It asserts spawn_count advanced (so the probe
cannot pass vacuously by skipping the supervised branch) and watch_count == 0.
RED on the unfixed code (watch_count == 1, exit 61); GREEN after the fix
(register_watcher_if_enabled skips → watcher.register.skipped reason=auto_watch_off).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #685 (nguyentamdat) rebased onto current main, plus two
research-driven refinements and a genuine reproduce-first guard.
The parallel extract retains each file's source text so the fused cross-file
LSP resolve can re-parse it. That retention is transient but a peak-RSS driver.
On main the caps are flat (100 MiB/file, 2 GiB total) and a file over the cap is
silently unretained AND its cross-file resolution is skipped -- a graph-quality
gap. This change bounds retention AND keeps every cross-file edge.
- Source-text cap as a FLOOR, not just a ceiling: retention total defaults to
min(cbm_mem_budget()/8, 1 GiB), per-file min(32 MiB, total). Following the
rust-analyzer memory model, the RAM-derived default is clamped to a small
absolute ceiling so a huge-RAM host does not hold tens of GB it would re-read
cheaply. Both caps env-overridable via CBM_RETAIN_TOTAL_MB /
CBM_RETAIN_PER_FILE_MB (limits.c convention); ceilings bound only the
auto-derived default, never an explicit operator/caller choice. A dropped file
emits one index.retain_capped WARN per run.
- Bounded re-read fallback (the correctness guarantee): resolve_worker re-reads
an unretained file's source on demand (bounded, freed immediately) instead of
skipping resolution, wired at every cross-LSP site that consumes source. The
cap now only trades retained RAM for a bounded re-read, never a lost edge.
- cbm_parallel_extract_ex + opts struct (cbm_parallel_extract is now a wrapper
passing NULL -> env-derived defaults); malloc/calloc NULL-check hardening.
Reproduce-first: parallel_cross_file_reread_preserves_unretained_edges uses a
Java<->Kotlin pair whose cross-file lsp edges are genuinely source-dependent;
the edges are lost when the caller is unretained and the fallback is absent
(RED), present with it (GREEN), with a retained CONTROL scenario proving
non-vacuity. #685's original Python red test was a false guard (per-file py_lsp
already resolves those calls) and is replaced. Peak-bound guards
(retained_bytes <= total_cap; retain_sources=false retains nothing) in test_mem.c.
Verify: make -f Makefile.cbm cbm && make -f Makefile.cbm lint-ci; test-runner
parallel pipeline incremental py_lsp ts_lsp java_lsp kotlin_lsp c_lsp cs_lsp
go_lsp rust_lsp mem -> 2323 passed.
Co-authored-by: nguyentamdat <nguyentamdat@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The long-lived MCP server ran the full index pipeline in-process on two
background paths -- the session auto-index (autoindex_thread, mcp.c) and the
watcher re-index (watcher_index_fn, main.c). The parallel pipeline's worker
threads abandon their mimalloc pages at thread exit, and mimalloc v3 defaults
page_reclaim_on_free=0, so those pages are never reclaimed when the main thread
later frees their blocks (mi_collect cannot touch abandoned pages either). RSS
then ratchets across re-index cycles and never comes back down (#832).
The index_repository tool path already avoided this by running the index in a
supervised worker subprocess: on child exit the kernel reclaims 100% of the
child's RSS. Route the two background paths through that same supervised runner:
- supervisor_invalidate_store() is now NULL-safe (the watcher path has no MCP
server / cached store).
- New static index_run_supervised_path() builds {"repo_path": root} and reuses
the existing spawn + skip-and-continue recovery loop (no duplication). New
public cbm_mcp_index_run_supervised_path() is the srv-less entry main.c calls.
- autoindex_thread and watcher_index_fn gate on cbm_index_supervisor_should_wrap():
supervised on a marked host with the kill switch off, else fall through to the
unchanged in-process pipeline (kill switch / spawn-failure degrade). The
watcher keeps its non-blocking try_lock gate + g_shutdown check exactly; the
lock still serialises re-indexes. cbm_mem_collect() stays on the in-process
branch only.
Also fold in the in-process allocator fallback: cbm_mem_init sets
mi_option_page_reclaim_on_free=1, restoring v2 behaviour so any path that stays
in-process (kill switch, degrade, embedders) also reclaims abandoned worker
pages. The subprocess is the primary cure (child exit returns RSS every cycle);
this is the fallback.
Tests: a deterministic gating guard (test_mcp.c) proves the background paths now
spawn a worker child and index the fixture (RED when routed in-process:
spawn_count unchanged; GREEN after). The test-runner acts as a real in-process
worker when spawned as `cli --index-worker` (public APIs only, no production
seam). A non-gating prod-binary RSS reproduction (tests/repro/issue832_rss.py)
shows the supervised server flat at ~10 MB vs ~131 MB in-process.
Scope: JobObject, the #841 spurious-trigger fix, retention caps, budget tiering
and the backpressure busy-spin are follow-ups.
Refs #832, #841.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel