The no-skips lint gate forbids bare SKIP(); the parallel-recovery guard
is genuinely platform-specific (POSIX fork harness — Windows coverage
comes from the smoke invariants), so it uses SKIP_PLATFORM like the
neighboring supervisor guards.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An 81k-file TypeScript corpus previously never finished indexing. Three
stacked defects, found by stack-sampling the live grind and reading the
supervisor's quarantine records:
1) walk_defs pushed children via index-based ts_node_child(i), which is
O(i) per call in tree-sitter — O(n^2) per wide node. A 3.5 MB file
with ~580k flat comment siblings needed ~1.7e11 iterator steps and
hung extraction past the supervisor's quiet-timeout. All six descent
sites (including the default loop) now collect children linearly with
a TSTreeCursor (wd_collect_children); small nodes keep the direct
index path. Guard: extract_wide_flat_file_is_linear (400k comment
siblings — the monster's exact shape — bounded at 30s).
2) The sequential cross-LSP driver handed the FULL def list to a full
per-file registry build+finalize — O(files x defs); 74% of samples
sat in build_qn_index. The parallel resolve worker's dispatch
(module-def filter -> shared prebuilt registries -> filtered
fallback) is now extracted into cbm_pxc_dispatch_file and drives
BOTH pipelines — one path, one semantics. The shared registries
live in a caller-owned arena that outlives the calls pass
(resolved_calls borrow registry strings; freeing at pass end was a
use-after-free ASan caught in pass_calls). Rust is exempt from the
def filter: Cargo-manifest cross-crate resolution needs defs from
OTHER workspace crates that the own+imports filter drops (this
starvation predated the unification on the parallel path). Guard:
pipeline_seq_ts_cross_uses_shared_registry (full builds 40 -> 0,
cross-file edge preserved).
3) The crash supervisor's recovery re-ran the worker SINGLE-THREADED to
keep one exact marker. At scale that fell into the sequential crawl,
was killed as a hang mid-pass, and the stale extraction marker got
four innocent files quarantined, one 15-minute retry at a time.
Recovery (and the terminal partial run) now re-run PARALLEL; the
marker is an append journal ("S <rel>" / "D <rel>", written by
extraction and both resolve drivers), the suspect set is the open-S
in-flight set, and a file is quarantined only when it recurs across
two consecutive failed runs — one per round, oldest open S first;
disjoint consecutive sets stop the loop rather than blame an
innocent. Guard: index_recovery_parallel_quarantines_crasher
(single-threaded spawn count must stay zero; verified RED against
the old loop).
Also: parse_ts_type_text gains a dynamic work budget proportional to
the input (1M + 64x source bytes, CBM_TS_TYPE_BUDGET override, warn
once then degrade to unknown) replacing unbounded recursion cost.
Measured on microsoft/TypeScript (81,398 files): infinite -> 33s cold
full index, 295k nodes / 779k edges, zero dangling edges, zero files
skipped — the monster file itself now indexes cleanly. Full 9-language
sweep green with no quality flags; kernel unchanged at 390s / 8.5M
nodes / 0 dangling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The supervised-worker fast path deliberately skips the pipeline
teardown (the OS reclaims the process wholesale) and pairs that with
_Exit, which bypasses atexit/LeakSanitizer. The production worker in
run_cli has both halves; the test binary's worker emulation
(tf_maybe_run_index_worker) only had the first: it returned normally
through main(), LeakSanitizer ran at exit, reported the intentionally
unfreed pipeline and forced exit code 1 — so the supervisor read a
healthy index as worker_failed and IDX832 went red on exactly the
platforms whose ASan builds run a leak pass (all four Linux legs;
macOS/Windows ASan have none). Reproduced and verified fixed in an
ubuntu:22.04 arm64 container: reap outcome=clean, mcp suite 140/0.
Mirror the production exit: deliver the response file, fflush, _Exit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_mem_init is init-once (guarded by g_initialized), so the
backpressure test's setenv(CBM_MEM_BUDGET_MB=1) + re-init 'restore'
was a silent no-op: whichever init ran first fixed the budget for the
whole process. In the full runner this test's init won, locking a 1 MB
budget that leaked into every later budget consumer —
mem_over_budget_low_rss red on every platform, and all subsequent
pipeline tests silently running with backpressure permanently engaged.
Add cbm_mem_set_budget_for_tests(): writes the budget directly without
touching the init guard. The test now saves the caller-visible budget,
sets 1 MB via the hook, and restores before asserting. Env dance
removed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The deterministic same-QN collision rule picked the lexicographically
smallest (label, file_path, start_line, name), which silently inverted
the documented upsert/merge semantics: re-upserting a QN with updated
fields kept the OLD content (gbuf_upsert_updates,
gbuf_upsert_same_qn_updates_all_fields, gbuf_merge_overlapping_qns all
red on every platform).
Replace it with a single mixed-direction total order applied in both
cbm_gbuf_upsert_node and merge_update_existing: smallest file_path,
then LARGEST start_line, then largest name/label. Still a pure content
function of the two candidates (commutative, associative,
scheduling-independent — multi-threaded runs stay byte-identical,
verified by the xfs MT x4 fingerprint guard), and line-descending
within a file restores the classic contract: a later definition in the
same file replaces the earlier one, and a full tie is the same entity
refreshed in place.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The header referenced the license identifier of a third-party library's
bundled dependency while explaining that the implementation deliberately
does not use that library. License scanners match the bare identifier
string and misclassify the file. Reword to state provenance directly:
written from the published papers, nothing vendored or derived.
Comment-only change; also picks up the formatter pass on this file.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This reverts commit 0475a880e5735ef5cb0b0d99532e552096e5a2d5.
Measured on the linux kernel: peak RSS unchanged (23.95 vs 24.4 GB, within
run noise) at +22 s wall -- the extraction peak turned out to be monotonic
retained-tree/source accumulation, which is schedule-independent, not the
concurrent giant transients this queue bounded. Net negative as measured;
the retention-side fix is parked for post-release.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase-aligned RSS attribution located the global peak (24.4 GB at t=47s,
~14 GB transient above the retained floor) INSIDE parallel extraction: the
size-descending queue put the kernel's 12 largest files into 12 workers
simultaneously, each holding a multi-MB source + its 10-50x TSTree + arena.
Every post-extraction optimization sat below this waterline.
Replace the single biggest-first cursor with a dual-ended claim queue over
the same size-sorted list: big_workers = clamp(workers/6, 1, 3) claim from
the BIG end while the rest rush the small end, so at most big_workers giant
working-sets coexist -- a structural bound, not a scheduling heuristic (a
static interleave cannot bound concurrency under dynamic pulls: smalls
finish in ms and workers converge back onto the giants). Giants still start
at t=0, preserving the LPT no-straggler property; workers never idle while
either end has files. Per-file claim flags (atomic exchange) make the
crossover race-free; end-of-queue rescans are O(n) bytes once.
Intra-file parallel parsing was researched and ruled out: tree-sitter has
no such concept (incremental != parallel; one parser per thread is the
model), and data-parallel parsing theory (PAPAGENO through SLE 2025 cyclic
OPG work) requires operator-precedence grammars with local parsability --
none of the 159 vendored GLR grammars qualify, and C's preprocessor makes
byte-chunking unsound.
Output is schedule-independent since the determinism fix and verified so:
xfs sorted edge set byte-identical to the previous commit. pipeline 216
passed. Worst case (all-giant repo) unchanged by design.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two borrow-don't-copy wins from the layer-duplication analysis (a hot string
was resident 4-5x across extraction arenas, gbuf, registries, defs and
tokens). Graph output verified byte-identical.
1. Resolution registry (registry.c): every registered definition strdup'd its
qualified name TWICE (exact-map key + by_name array item) and its label
once (~8.5M label strdups). The by_name items now borrow the exact map's
single owned key (same lifetime, freed in one place) and labels intern
into a small per-registry pool (<= ~30 distinct). ~280 MB + ~17M
allocations on the kernel.
2. Semantic tokens (pass_semantic_edges.c): all_tokens held one strdup per
token INSTANCE — identical tokens ("xfs", "error", ...) duplicated across
hundreds of thousands of functions. Each tokenize worker now interns into
a per-worker pool (key == value == the single owned copy; no contention,
content-identical strings so downstream corpus/tfidf output is unchanged);
all_tokens slots borrow. Ownership moves from per-func frees to pool
frees at pass end.
Suites: semantic 33, simhash 24, pipeline 216, registry 54, incremental 161.
xfs: two MT runs byte-identical, and byte-identical to the pre-change graph.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The semantic pass kept four dense 768-float vectors resident per function
(ri/api/type/deco = 12.3 KB per func, ~9.4 GB transient on the linux kernel)
solely so candidate pairs could be cosine-scored after LSH. Replace the
resident floats with 4-bit quantized codes (~0.5 KB each, ~6x): new
src/semantic/rotsq.{h,c} — rotated scalar quantization from the RaBitQ
family of algorithms (Gao et al., SIGMOD 2024/2025), implemented from the
papers in plain C11 (no Eigen/library dependency): a deterministic
XXH3-seeded +-1 diagonal followed by a Fast Walsh-Hadamard rotation
(768 -> 1024 padded) spreads vector mass so per-vector scalar quantization
is near-optimal; the inner product is recovered from the codes with the
exact SQ expansion (one u8 integer dot + four multiplies) — a deterministic
pure function of the codes. Named rotsq, not rabitq: this is the family
core, not the papers' exact codebook construction.
Producers build each dense vector in a worker-local buffer, encode, and
discard; the int8 qvec export is unchanged. LSH signatures are computed
from DEQUANTIZED codes against hyperplanes in the rotated basis — random
hyperplanes are basis-agnostic, so no dense originals are retained and the
phase structure is untouched.
Estimator quality gated by sem_rotsq_ip_error_bounds (seeded vectors:
mean error < 1%, max < 4% of unit scale, ASan-green). Effect on output:
multi-threaded determinism preserved (xfs: two runs byte-identical);
SEMANTICALLY_RELATED edges shift once as scores move within ~1% of the
0.75 threshold (xfs: ~130 -> 203; sampled new pairs are high quality —
alloc/free, read-verify and collapse/insert families).
Suites: semantic 33 (incl. the new estimator gate), simhash 24,
pipeline 216 — zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three memory quick-wins from the gbuf compaction research (graph output
verified identical; all delete/incremental paths suite-covered):
1. Release ALL lookup indexes right after build_dump_nodes instead of after
edge building: nothing later in the dump reads them, and at kernel scale
they are ~3.8 GB (hash buckets + key strings + pointer arrays) that
previously coexisted with every dump-side transient — the dump-phase RSS
peak. At 2x-kernel scale this is ~7.5 GB at exactly the OOM point.
2. node_by_id was a hash table keyed on STRDUP'D DECIMAL STRINGS of the
int64 id — ~0.44 GB of buckets and key strings, plus a snprintf + strdup +
hash on every hot find_by_id call. Ids are dense and sequential; a plain
grow-on-demand pointer array replaces all of it with one bounds-checked
load.
3. The per-node and per-edge `project` field was write-only (every reader
uses the store structs or gb->project directly; -Werror confirms) —
8 bytes x 24M instances ~= 192 MB of dead pointers, deleted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The four kernel rust files' wall time was a CONSTANT ~63.5 s that no
resolution-side fix moved -- because it was not resolution. The shared
all_defs registry build had two lookups INSIDE its registration loop
(receiver auto-registration probe via cbm_registry_lookup_type, and the
trait-linkage scan), and before finalize the registry has no buckets: both
probes were full linear scans over the types registered so far. On the
1.4M-def shared build that is ~10^10 strcmps ~= 63 s, paid once by the
building file while the sibling null-filter files waited on the once-guard
for exactly the same wall time. This is the checklist's own
lookup-in-registration-loop pattern -- invisible on small per-file builds.
Fix: a CBMIdxMemo (new in lsp_neg_memo.h: exact-match, strcmp-verified
string-key -> index map, arena-backed, first-put-wins to mirror the linear
scans' first-match semantics) maintained alongside every add_type in
rust_populate_cross_registry; both probes become O(1).
Also rides along: a per-root-invocation macro-expansion memo. Kernel
macro_rules are self-recursive (define_sizes! re-invokes itself) and the
expansion fallback ("no pattern matched -- still expand the first rule")
made recursion non-convergent: unbounded BREADTH under the depth-8 guard
turned 2-44 source invocations into ~200k expansions (each a fresh
tree-sitter parse of the substituted body). Identical (macro, substituted
text) now expands once per chain; the memo resets at each top-level source
site so per-site attribution is untouched, and the arg re-parse dedup only
applies at recursion depth > 0. Side effect: the eval-step cap no longer
exhausts, so a few previously-truncated real calls now resolve (kernel:
+25 edges).
Verified: kernel rust subtree byte-identical before/after (sorted node and
edge sets, exact); rust_lsp suite 506 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Macro-expanded kernel rust asks the same failing (receiver, method) /
(trait, method) questions thousands of times per file; each repeat re-ran the
full registry-pure cascade (embedded-impl walk + trait-default tail) against
the shared 1.4M-entry registry -- ~63 s per trait-heavy file even after the
build-once registry and the trait indexes.
New internal/cbm/lsp/lsp_neg_memo.h: a generic, arena-backed negative-lookup
memo for LSP resolve cascades (open-addressing u64 set, site-tagged FNV pair
keys, lazy alloc, grow-by-rehash). Gated on a SEALED registry (read_only), so
a miss is a pure fact of (registry, query); the cheap direct lookup stays
before the memo check so a hash collision can never hide a real hit. Any
language wires it in a few lines -- rust first (rust_resolve_trait_method and
rust_find_sole_trait_impl, both proven to read nothing but the registry and
their query strings); the C resolver's bespoke memo can migrate, and the
audit's php/c#/java/kotlin cascades are wiring candidates.
Only the full miss is memoized (zero impls and no trait default), preserving
out_impl_count/out_n fidelity for the ambiguous >=2-impl cases.
Verified: kernel rust subtree indexed before/after -- sorted node AND edge
sets byte-identical (0 diff; exact comparison, now that indexing is
deterministic). rust_lsp suite 506 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing produced a different graph on every run (xfs: node
labels flipping Class/Function/Macro, ~2k edge lines differing across USAGE/
CALLS/IMPORTS/SIMILAR_TO/SEMANTICALLY_RELATED) and lost edges vs repeated
runs. Five root causes, all order- or scheduling-dependence:
1. gbuf QN collisions resolved last-wins in MERGE ORDER: in C a struct, a
function and a macro can share one name, and the QN scheme does not encode
entity kind, so whichever entity a worker merged last overwrote the node's
label/name/lines/props (the flickering node set that fed every pass). Both
cbm_gbuf_upsert_node and the parallel-merge path now keep the SAME entity's
refresh semantics but resolve cross-entity collisions by a canonical
content rule -- lexicographically smallest (label, file_path, start_line,
name) -- and keep the label/name secondary indexes consistent when the
survivor changes (the old overwrite left nodes listed under their original
label, which is how the flicker reached find_by_label consumers).
2. Semantic-edges admission raced per-node max_edges budgets on shared atomic
counters under dynamic scheduling: scoring stays parallel and pure; pairs
carry (func index, candidate rank); one sequential pass admits in canonical
order. funcs[] itself is sorted by qualified name up front (the label-index
scan order is merge order).
3. Import resolution's symbol-name fallback returned the FIRST same-named node
in registration order; now the lexicographically-smallest-QN candidate.
4. Similarity entries inherited merge order into LSH chains + per-node cap
truncation, and pair ownership keyed on run-varying node ids; entries are
sorted by QN and ownership compares QNs (lsh entries carry the QN).
5. Call-neighbor context (semantic vectors + tokens) truncated MAX_CALLEES
subsets in edge-array order; neighbor names are sorted first.
Verified: four consecutive multi-threaded kernel-xfs runs produce byte-
identical sorted node AND edge sets (13341/58598, zero diff). The repro guard
repro_parallel_edge_determinism (MT x K mutual equality) is GREEN; a new
deliberately-RED repro_seq_parallel_equivalence tracks the remaining OPEN bug
that the sequential and parallel pipelines systematically disagree (~3459
USAGE / ~1666 WRITES lines on xfs) -- separate code paths, fix deferred.
Suites: pipeline 216, c_lsp 750, rust_lsp 506, py_lsp 63, java 94, kotlin 78,
php 278, registry 54, store_nodes 56, incremental 161, semantic 32, simhash 24
(one test updated: SIMILAR_TO pair direction is now canonical by QN), mcp 140,
subprocess 17 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Follow-up to the conditional shared registry: the eager pipeline.c build paid
the O(all_defs) construction and a multi-million-entry registry RSS on every
run that contains rust files, even when every rust file filters to a proper
import subset and the shared registry is never consulted. Build it lazily
instead: the first worker that hits a NULL-filter rust file takes a mutex,
double-checks, builds into a dedicated arena, seals and atomically publishes;
later null-filter files reuse it and subset files never trigger it. Verified
the build fires exactly once (scale log: types=2320 funcs=8860 on the kernel
rust subtree) and the shared-path unit test still resolves identically to the
per-file build.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Profiling the kernel showed 4 tiny trait-heavy rust files at ~90 s EACH in
cross-file resolve. Live call trees put the time not in lookups but in
REGISTRY CONSTRUCTION: cbm_pxc_filter_defs_for_file returns NULL for these
files (their path-derived module QNs never match the def_module_qn index
keys), so each one built its own registry over the ENTIRE project def set
(millions of entries at kernel scale).
Original premise corrected during implementation: only 4/87 rust files (on
the kernel rust subtree) hit the NULL-filter path; 83/87 resolve against
proper import-filtered subsets. A registry shared across ALL rust files
would hand the subset files a superset and could add edges. The fix is
therefore CONDITIONAL, byte-identical for every file by construction:
- filter NULL -> ONE shared all_defs registry (built once in pipeline.c,
finalized, read_only-sealed; module_qn=NULL is safe: an instrumented probe
found 0 NULL def_module_qn across all_defs, so the per-file def_mod
fallback never fired anyway). Kills the amplifier.
- filter SUBSET -> today's per-file subset build, unchanged.
The cross-crate Cargo manifest reaches the shared path through
cbm_pxc_get_rust_manifest() (same worker-thread thread-local the fallback
read), preserving cross-crate resolution exactly.
Byte-identity verified empirically: single-threaded sorted edge-set diff on
the kernel rust subtree, fix off vs on -- zero difference outside
SEMANTICALLY_RELATED edges, which self-differ between two runs of the SAME
unfixed binary (the pre-existing semantic non-determinism guarded by the
repro_parallel_edge_determinism RED test). New suite test
rustlsp_shared_registry_resolves_like_per_file exercises the shared path.
Suites: rust_lsp 506, c_lsp 750, registry 54, pipeline 216, java_lsp 94,
kotlin_lsp 78, php_lsp 278, store_nodes 56 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing loses edges vs single-threaded and is
run-to-run non-deterministic (fs/xfs: ST 58573 edges vs MT 58561 on the
first divergent run; rust subtree flickers 44445-44857 across runs).
Discovered while hardening perf-change equivalence checks; also the
reason raw node/edge counts are not a valid equivalence gate.
The guard indexes a corpus once single-threaded (reference) and K=4x
multi-threaded (fresh store each run) and asserts the SORTED
(source_qn|type|target_qn) edge sets are all identical. RED today;
GREEN if and only if MT output is deterministic and equals ST.
Calibration note (false-guard avoidance): three escalating synthetic C
corpora (300/500/600 files, dense cross-file calls, token-identical
bodies) stayed fully deterministic -- the similarity pass emits zero
edges on synthetic C, and the race needs real-code edge diversity. A
synthetic guard would pass on buggy code, so this repro uses the
smallest real corpus that reproduces (kernel fs/xfs) and SKIPs when the
corpus is absent. Suspected root causes documented in the test header
(per-worker edge-buffer merge, gbuf edge dedup check-then-insert,
symmetric SIMILAR_TO emission, resolve ordering). Fix deferred; this
test is its guard.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
dump_and_persist_hashes upserted file hashes one statement per file --
~89k autocommit transactions (one fsync each) on the linux kernel. Switch to
the existing cbm_store_upsert_file_hash_batch (same cached upsert inside one
begin/commit, rollback on error): identical tuples and replace semantics, one
fsync. OOM fallback to the per-file path retained.
Also instrument the whole post-dump tail with CBM_PROFILE spans
(persist/1_reopen .. 6_artifact_export) -- this stretch was ~60 s unattributed
on kernel-scale runs; the FTS5 backfill over all node names is now measurable
before any optimization decision.
New deterministic guard: store_file_hash_batch_equals_loop asserts the batched
path produces byte-identical file_hashes rows to the per-file loop. Suites:
pipeline 216, store_nodes 56, store_edges 25, store_search 65, store_arch 52,
store_bulk 3, store_pragmas 7, store_checkpoint 1, incremental 161 -- zero
failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Rust cross-file resolver's trait-method, sole-trait-impl, and free-func
fallback loops scanned the ENTIRE type/func registry per call site on a miss.
On kernel-scale registries (~10^6 entries, macro-inflated) four tiny trait-heavy
rust files burned ~96 s EACH (~200k eval-capped call sites x full-registry
strcmp scans).
Fix: two hash indexes built at registry finalize, mirroring the existing
bucket+entries idiom -- fnv1a(bare(embedded_type)) -> chain of declaring type
indices (trait -> implementers), and fnv1a(short_name) -> chain of free-func
indices. Chains preserve ascending registry order (first-match tie-break
unchanged); allocation-free iterators fall back to the post-finalize tail and
to a full scan on unfinalized registries, so the candidate set is identical in
every mode. The four rust loops now walk the iterator and re-apply their exact
original predicates -- byte-identical results, loop bodies untouched.
Verified: graph equivalence on the kernel rust/ subtree (nodes 9654, edges
44798 identical with and without the change); registry_short_name_indexes unit
test pins iterator contents/order/dedup; suites c_lsp 750, rust_lsp 505,
registry 54, pipeline 216 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_register_instance_field mutated the shared Tier-2 registry through direct
struct writes that bypass the read_only seal: it overwrote field-type pointers
in place and reassigned the sealed type entry's field arrays to memory owned by
the per-file arena. In the fused parallel resolve path (one sealed registry
shared by all resolve workers) that is a cross-thread data race, and it leaves
the shared registry pointing into arenas that die with their file -- a
cross-file use-after-free. The count-based seal guard could not see it because
type/func counts never change.
Python extraction emits no instance fields, so resolve-time discovery is the
ONLY source of self.x / PEP-526 fields; a plain read_only gate would silently
drop real attribute-chain CALLS edges. Fix instead routes discovery into a
per-file field overlay on PyLSPContext (arena-backed, dies with the file) and
has py_lookup_field_depth consult shared-base + overlay at the exact precedence
point of the direct field scan. Mutable per-file registries keep the original
in-place write, byte-identical.
Reproduce-first, both halves pinned:
- seal_py_shared_registry_readonly_fields: RED on the bug (field count 0->1 on
a sealed registry), GREEN with the overlay.
- pylsp_fused_self_attr_chain_via_overlay: GREEN with the overlay, RED under a
simulated plain gate (self.b.work() CALLS edge lost) -- proven both ways.
Suites: c_lsp 749, py_lsp 63+45 stress/scale/bench, pipeline 216, registry 54,
scope 10, type_rep 19 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Kernel-scale profiling put ~87% of the 361 s cross-file resolve phase inside
c_lookup_member_depth's miss cascade: in macro-heavy C nearly every member
lookup misses, identical (type_qn, member) pairs miss millions of times, and
each miss re-paid an arena_sprintf("%s.%s") (the strlen storm) plus the
O(type_count) short-name registry scan.
Fix, semantics preserved exactly:
- Per-file negative memo (open-addressing uint64 set) on (type_qn, member),
active only at depth 0 under the shared READ-ONLY Tier-2 registry, where the
registry-only cascades are pure functions of (type_qn, registry). The direct
method lookup always runs first (collision/staleness guard) and the
context-dependent scope-alias path is always evaluated; only the provably-
NULL module-prefix/base-class/short-name cascades are skipped on a memo hit.
- Module-prefixed QN retries build into a stack buffer via cached
module_qn_len (arena_sprintf only as truncation fallback).
Soundness: cbm_c_build_cross_registry finalizes then sets read_only=true;
cbm_registry_add_func/add_type hard-return when read_only, so no miss can
flip to a hit during resolve. Suites: c_lsp 748/0 (incl. the tier2 read-only
guards), pipeline+registry+scope+semantic 656/0. Graph equivalence verified
on fs/xfs: nodes 13341 / edges 58559 byte-identical with and without the fix.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The worker's msg=prof pass/sub-phase report is written only to its supervisor
logfile, which was deleted on every clean exit -- profiling a successful run was
impossible (the log survived only on failure). Keep it when profiling is active
and echo the path via index.supervisor.profile_log. Default behavior unchanged:
clean-run logs are still pruned when CBM_PROFILE is off.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two wall-clock fixes from a linux-kernel (8.5M nodes) profiling run:
1. Extraction back-pressure futility latch: when a full collect+nap cycle ends
still over the RSS budget, the resident floor (accumulated graph + retained
sources) -- not in-flight transients -- holds the memory, so napping again on
every subsequent pull reclaims nothing and only idles workers (measured: one
full cycle per pull, ~390 s at 79% avg CPU across 63k pulls). Latch bp_futile
after the first futile cycle, proceed with the designed soft overshoot, and
re-arm via the cheap per-pull probe once RSS drains under budget. WARN
mem.backpressure.futile exactly once per latch event (0->1 transition).
2. Supervised-worker fast exit: the index worker piecemeal-freed a multi-GB
graph (and store) right before process death (measured: minutes of free()
in the tail). In worker role, skip cbm_pipeline_free and _Exit after the
response is written and flushed; the OS reclaims memory wholesale. In-process
paths (tests, kill switch, degrade) free normally, keeping ASan/LSan
meaningful.
Reproduce-first: pipeline_backpressure_futile_nap_disengages (64-file fixture >
MIN_FILES_FOR_PARALLEL so the parallel gate actually runs, CBM_MEM_BUDGET_MB=1,
deterministic nap-cycle counter; engagement guard cycles>=1 prevents a vacuous
pass). RED on the old gate: 64 cycles (one per pull) > bound cores+2. GREEN with
the latch. Suites: pipeline 216, mcp 140, subprocess 17 -- all pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
install.ps1 kept detecting arch=amd64 on windows-11-arm because it runs under x64
emulation, where neither $env:PROCESSOR_ARCHITECTURE nor .NET OSArchitecture (on
Windows PowerShell 5.1) reports the real Arm64. Add a CBM_ARCH env override to
install.ps1 (wins over auto-detect; also a genuine escape hatch for emulated
invocations) and have smoke Phase 13 pass the authoritative DL_ARCH. Deterministic
-- no reliance on in-process detection under emulation. Same emulated-arch class
as #907 (uname) and #908.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
install.ps1 reported "arch: amd64" on windows-11-arm and 404'd on the amd64 zip:
$env:PROCESSOR_ARCHITECTURE reports the emulated "AMD64" for an x64 process under
ARM64 emulation, and PROCESSOR_ARCHITEW6432 is unset for 64-bit emulated procs.
RuntimeInformation.OSArchitecture reports the real OS arch (Arm64) regardless of
process emulation. Same emulated-arch class as the smoke uname fix (#907); this
one is a genuine product improvement -- real ARM users invoking install.ps1 from
an x64 context now get the arm64 binary.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The persistent windows-11-arm smoke Phase 12a failure was a 404, not a network
error: it requested codebase-memory-mcp-windows-amd64.zip on the arm64 leg. Cause:
DL_ARCH came from `uname -m`, which on windows-11-arm is an emulated x86_64 MSYS2
uname reporting "x86_64" -> wrong (amd64) archive -> 404 (server has arm64). Phase
14 worked because the binary's own detect_arch() is native. Prefer SMOKE_ARCH
(passed from the smoke workflow's matrix.arch) over uname; fall back to uname for
local runs. This is the real cause the earlier curl/proxy/ipv4 attempts masked --
the 404 was swallowed by 2>/dev/null until #905 surfaced it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 12a swallowed curl's stderr (2>/dev/null), so the persistent windows-arm64
"curl download failed" was undiagnosable. Two changes: (1) add --noproxy '*' so
curl never routes the local 127.0.0.1 test server through a proxy env var -- a
strong candidate since the app's own WinHTTP downloader reaches the server in
Phase 14 while only msys2 curl fails, instantly; (2) print curl's stderr on
failure so the actual reason is visible if it persists. Harmless on all
platforms (the smoke server is always local).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Root cause of the windows-11-arm smoke Phase 12a failure: the smoke `curl` was
curl-8.20.0 from the base MSYS2 (msys) repo, which on an ARM64 host runs as an
x86_64 binary under emulation, and its networking could not reach the local
http.server -- while every NATIVE ARM64 process did (python's http.server bound
and served, and the product binary's own downloader completed Phase 14's update
from the same 127.0.0.1:18080). Install the native mingw-w64-clang-<arch>-curl so
the smoke curl is native (aarch64 on windows-11-arm), matching python and the
product binary. It takes PATH precedence over the base msys curl in the CLANG*
environments. amd64 gets the native x86_64 curl (already worked, now explicit).
Last red legs in the release dry run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
curl (already installed) failed instantly in smoke Phase 12a on windows-11-arm
while the server was up and the binary's own downloader reached it -- the classic
localhost->::1 (IPv6) resolution: msys2 curl tried IPv6 but python's http.server
was IPv4-only. Bind the server to 127.0.0.1 and use a 127.0.0.1 SMOKE_DOWNLOAD_URL
so both ends are explicit IPv4. amd64 windows is unaffected (already reached via
IPv4). These were the last red legs in the release dry run.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The smoke-windows job's msys2 install list omitted curl, relying on an ambient
curl that resolves on CLANG64 (amd64) but not on CLANGARM64 (windows-11-arm).
Phase 12a's `curl` download failed there even though the server was up -- the
binary's own downloader (Phase 14) succeeded against the same server. Install
curl explicitly so both arches use the msys2 curl in the same network namespace
as the local http.server. These were the last two red legs in the release dry
run (everything else -- test, build incl build-windows-arm64, all smoke, and
soak incl soak-quick-windows-arm64 -- is green).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two release-blockers surfaced once #882 made every smoke leg a required gate
(continue-on-error previously masked them):
- smoke-unix ran the *dynamic* linux binary on ubuntu-22.04 / 22.04-arm, where
it cannot start (it links glibc 2.38+) -> Phase 1 died. Older-glibc coverage
is the -portable (static) binary's job and is green via the smoke-linux-portable
broad legs, so drop the impossible dynamic broad legs (keep macOS).
- the ClamAV scan died on every linux leg: apt auto-starts the clamav-freshclam
daemon which holds the freshclam log lock, so the manual freshclam failed with
"Failed to lock the log file ... Resource temporarily unavailable". Stop the
daemon first.
Also add native ARM64 Windows as a first-class release target -- the one platform
we test (windows-11-arm) but never shipped:
- build-windows-arm64 job (CLANGARM64) producing codebase-memory-mcp[-ui]-windows-arm64.zip
- smoke-windows is now arch-aware: windows-11-arm smokes the NATIVE arm64 binary
instead of the x86_64 binary under emulation
- a native soak-quick-windows-arm64 leg (no sanitizer -- ASan is unavailable on
native ARM64 Windows)
- arm64 detection in install.ps1
release.yml already publishes windows-arm64 (asset filter + merge-multiple) and
build_update_url/detect_arch already resolve it; npm/pip/install.sh compute arm64
already. scoop/winget/chocolatey per-version manifests are updated at release time.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The dry-run smoke Phase 14 runs the binary's real `update` command, which on
linux downloads the fully-static "-portable" asset (build_update_url in
src/cli/cli.c appends -portable on linux; _build.yml's build-linux-portable job
ships codebase-memory-mcp[-ui]-linux-<arch>-portable.tar.gz). The smoke artifact
server only served the non-portable name, so `update` 404'd and the config-
refresh assertion failed on every linux smoke leg. The mismatch (the smoke half
of the -portable delivery change) was masked until every smoke leg became a
required gate.
Mirror the tarball under the -portable name on linux (standard and ui) so the
served names match build_update_url; checksums.txt globs *.tar.gz and picks up
the new files. Verified the fixed logic serves the exact name the binary
requests for both variants; the gating smoke Phase 14 is the regression guard.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel