Follow-up to #787/#828, disclosed during #828's review: the same clobber
class also affects two call sites #828 deliberately left out of scope.
create_env_configures_for_file (pass_definitions.c) and resolve_file_throws
(pass_parallel.c) both resolved their edge source via a direct
cbm_gbuf_find_by_qn(enclosing_func_qn) lookup with no guard against the
lookup landing on the shared Folder/Project node for directory-module
languages (Java/Go) -- the exact #787 root cause, just unguarded at two
more sites. A class-level construct (no enclosing function) has its
enclosing_func_qn fall back to the MODULE QN, which for a directory-based
module collides with the package's Folder node, conflating every
same-package file's CONFIGURES/THROWS edges onto one arbitrary source.
- create_env_configures_for_file: added the same
cbm_pipeline_node_is_dir_container guard already used by the three
#828-fixed finders, falling back to the per-file File node on a
directory-container hit.
- resolve_file_throws: was missing the `rel` parameter its sibling
resolve_file_calls/usages/rw already carry, so it inlined a raw lookup
instead of calling the shared, already-guarded find_source_node. Added
`rel` and switched to find_source_node, matching its siblings exactly.
Regression guard: tests/repro/repro_issue842.c, reproduce-first per the
issue's request for "an #828-style multi-order red fixture":
- CONFIGURES reproduces on a small 5-file sequential fixture (Go, since
Java's System.getenv detection is separately broken -- extract_env_key_from_call
reads a "function" field Java's method_invocation node doesn't have; noted
in the test, out of scope here).
- THROWS only reproduces past MIN_FILES_FOR_PARALLEL=50 (the worker-merge
path) -- confirmed empirically that the small sequential fixture already
attributes correctly even on unfixed HEAD, mirroring #787's own two-faced
(sequential/parallel) bug. Padded with 60 filler classes, following
repro_issue787.c's build_parallel_fixture precedent.
RED on unfixed HEAD (both tests): 0 passed, 2 failed. GREEN with this fix:
2 passed, 0 failed. Full scripts/test.sh (5958 cases) passes clean with
this change; scripts/lint.sh's two pre-existing failures (pass_lsp_cross.h
parameter-name mismatch, extract_defs.c cppcheck AST parse) are unrelated
and present identically on unfixed main.
Fixes#842
Signed-off-by: Dhruvy0804 <dhruvgarg553@gmail.com>
Completes the #963 coverage taxonomy: alongside the best-effort FAILURE
signals (parse_partial, skipped), the report now lists what was
deliberately NOT indexed — a deterministic, by-design class, never mixed
into the failure surfaces.
- discover: file skips are classified by reason (gitignore / cbmignore /
skip-list / ignored-suffix / fast-pattern / size-cap) and collected via
the new cbm_discover_ex2 (per-file entries capped at 2000 with the
uncapped total reported — truncation is explicit, never silent; whole
excluded subtrees stay exhaustive as before). Unsupported-extension
files are deliberately NOT recorded: no grammar for an extension is
not an ignore decision, and listing every README/asset would drown
the signal
- pipeline: ignored files captured per run (full + incremental) and
persisted into the index_coverage table as not_indexed_file rows,
excluded dirs as not_indexed_dir; incremental runs rewrite the
by-design set fresh from discovery instead of carrying old rows
- store: the deleted-file prune exempts not_indexed_* (deliberately
unindexed paths never have hash rows); the missed shadow graph keeps
showing FAILURES only — the UI's report-an-edge-case callout can
never fire for gitignored paths, and a project whose only coverage
rows are by-design gets no miss graph at all
- mcp: index_repository response gains not_indexed_files {files, count,
truncated, note} (dirs stay under excluded); index_status's coverage
report gains a not_indexed section (dirs + files + note) and no longer
lumps by-design rows into skipped; tool descriptions updated — the
by-design class is presented as deliberate and deterministic (change
the ignore rules and re-index), with no best-effort hedging
- tests: e2e guard (gitignored file + dir reported with reasons on both
surfaces, failure surfaces stay clean, missed graph excludes them,
rows survive an incremental re-index, ignored constructs genuinely
absent from the graph); store unit test covers the prune exemption
and shadow-graph exclusion
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Per review: no dedicated coverage tool.
- index_status now carries the coverage report (parse_partial + skipped
lists from the separate index_coverage table, plus the best-effort
coverage_note); get_index_coverage is removed. Tool description
updated so agents learn the coverage semantics from index_status;
index_repository and the response note now point there
- the read hook resolves coverage through index_status (same envelope
fields, unchanged note wording); verified end-to-end against a real
index (flagged file -> note with exact ranges, clean file -> silence)
- get_code_snippet responses from a parse_partial file now carry a
correlated coverage_note ("line range(s) X could not be parsed ...
the source above is ground truth") — the result names its file, so
the warning is precisely anchored. Entirely-skipped files cannot
appear here (no nodes), so no uncorrelated noise is possible; clean
files carry no note (asserted both ways in the e2e test)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four refinements to the coverage signal:
- extraction: subtract DEFINITE recovery before flagging. Tree-sitter
error recovery plus the ERROR-descending def walker often re-extract
constructs inside a failed region (verified: `def broken(:` comes
back as a def); a region whose every line is covered by definitions
that START inside it is not a miss, and a fully recovered file is not
flagged at all. Container defs (Module/Package) don't count as
evidence, and partially covered regions stay flagged — the
#ifdef-split case keeps its flag because the first branch's function
is genuinely lost
- naming: the query_graph option is graph="missed" (the graph shows
ONLY misses — "coverage" was misleading); shadow project renamed to
"<name>::missed"; tool descriptions updated so agents discover both
the option and its semantics
- hook: the CLI-installed PreToolUse augmenter now also matches Read
and injects a coverage note when the file being read is listed as not
fully indexed ("line ranges X-Y could not be parsed — the file
content you are reading is ground truth"). Safe against the old
issue-362 hazard: the augmenter is structurally non-blocking (always
exit 0, additionalContext only), mirroring the Gemini matcher that
already includes read_file; matcher upgrade bookkeeping updated
- ui: "Missed files" toggle in the graph sidebar renders the miss
graph as a second graph option — /api/layout gains graph=missed
(same db file, shadow-project scoping; base project name validated
as before)
Tests: recovery-subtraction cases (recovered def unflagged, garbage
region flagged, trailing recovered defs keep the flag), CLI matcher
tests updated, e2e resilience fixtures switched to an unrecovered
miss; verified end-to-end: hook emits the note for a flagged file and
stays silent otherwise, /api/layout?graph=missed serves the miss graph
and the code graph is unchanged, frontend build + tests green.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Persist the parse-coverage signal outside the graph and make the misses
queryable both as a list and as a graph.
- store: new index_coverage table (project, rel_path, kind, detail) —
coverage is metadata ABOUT the graph and never mixes into the graph
tables; replace/get/free APIs with a deleted-file prune keyed off
file_hashes (the live-file set)
- store: materialize a derived miss graph under the shadow project
"<project>::coverage" (Project -> Folder chain -> File{kind,detail},
CONTAINS_FOLDER/CONTAINS_FILE edges), rebuilt from the table inside
the same transaction on every replace; invisible to list_projects
(which scans the cache directory, not the projects table)
- pipeline: coverage is written in the same persist step as every other
post-dump SQL write (hashes, ADR, FTS). Full runs replace the whole
set; incremental runs merge rows surviving from files not re-extracted
with this run's fresh entries, so a fixed file's flag clears and
deleted files prune automatically
- mcp: new get_index_coverage tool (per-project parse_partial + skipped
lists with the best-effort note); query_graph gains graph="coverage"
to run the same cypher against the miss graph; the File-node property
marker from the previous commit is dropped in favor of the separate
table — the code graph gains no coverage rows
- wording: constructs in flagged ranges MAY be missing — tree-sitter
error recovery still salvages some (verified: a function inside a
flagged #ifdef region and even a broken def were still extracted)
- tests: store round-trip/prune/shadow-graph unit test; e2e asserts the
response fields, the advertised coverage-graph query, code-graph
purity, and that the flag clears after the file is fixed (incremental
route)
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Files whose parse tree contains tree-sitter ERROR/MISSING regions were
silently indexed as if complete — constructs inside those regions are
absent from the graph with no signal anywhere (classic trigger: the
preprocessor-blind #ifdef-split-brace pattern in C).
- cbm_extract_file: set parse_incomplete and record the 1-based line
ranges of the top-most error regions (bounded at 64), serialized as
"start-end,start-end" into the result arena
- pipeline: record such files under the new phase "parse_partial"
(distinct from skips — the file IS indexed) in both the parallel and
sequential extraction paths; stamp the File node with
{parse_incomplete:true, error_ranges} on full and incremental runs
via a new cbm_gbuf_set_node_props (upsert survivor rules bypass)
- mcp: report parse_partial {files, count, truncated, note} separately
from skipped[] (skipped_count semantics unchanged); the per-run
logfile lists parse_partial rows; the index_repository tool
description documents the signal and the query_graph pattern to find
flagged files
- tests: new parse_coverage suite (RED-first, 7 cases incl. false-
positive guards and the region cap) plus an end-to-end resilience
guard asserting response fields, logfile, File-node marker, and the
advertised query_graph query
The signal is explicitly best-effort: a flag means constructs in the
listed ranges were dropped (prefer grep there); the absence of a flag
is NOT a completeness guarantee.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #363 memory axis (CBM_MEM_BUDGET_MB) has shipped, so this reproduce-first
case is now a permanent GREEN guard, not an open bug. Its header claimed the
override "does not exist" and asserted the budget is honoured "regardless of
host RAM" — both stale: the resolver now clamps the override to effective
(cgroup/host) RAM. On a <4 GiB host the old 4096 MiB assertion would clamp and
falsely flip the bug board RED.
Rewrite the header to describe the shipped + clamped behaviour and drop the
override to 128 MiB, which is at/below any realistic host's RAM so it is
honoured exactly and stays stable on every runner. The clamp path itself is
unit-tested in tests/test_mem.c (resolve_budget_override_clamped_to_total).
Signed-off-by: Sam Li <yangsec888@gmail.com>
Fold the inline CBM_MEM_BUDGET_MB override logic in cbm_mem_init into a pure,
testable cbm_mem_resolve_budget() that returns a result struct
(budget/source/clamped/invalid), so the parse + clamp lives in exactly one
place and cbm_mem_init only surfaces the outcome as log lines.
Parsing now matches the strict src/foundation/limits.c convention (errno/ERANGE,
reject trailing garbage, positive-only): a fat-fingered value like "8GB" or a
20-digit typo becomes a warning + safe ram_fraction fallback instead of a
silently wrong budget. A valid-but-huge value clamps to detected total RAM
(logged mem.budget.clamped) rather than overflowing the MiB->bytes multiply and
wrapping to a near-zero budget (which would pin cbm_mem_over_budget() true and
stall indexing). mem.init now logs source= on both the override and fraction
paths, and the README gains the CBM_MEM_BUDGET_MB row it lacked.
Hardening of the existing #363 knob.
Signed-off-by: Sam Li <yangsec888@gmail.com>
cbm_cli_sha256_file shelled out to sha256sum/shasum, which is unportable:
the tool may be absent, and under cmd.exe the single-quoted path is not
dequoted so the hash of a downloaded release could never be computed on
Windows. Add a small in-process SHA-256 (FIPS 180-4, src/foundation/sha256.c)
and use it directly — no external tool, no per-OS branching, no popen. The
checksum test now covers three NIST vectors (empty, "abc", and a 56-byte
input that forces padding into a second block).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The layout endpoint treated the default render cap (2000) as a ceiling:
an explicit max_nodes above it was silently clamped back down, so the UI
could never show more than 2000 nodes regardless of the request. Make the
default a default — an explicit request is now honored up to a hard
ceiling of 10M nodes; CBM_UI_MAX_RENDER_NODES still adjusts the ceiling
for constrained deployments. The default budget rises to 5000, and the
layout refinement pass scales its iteration count down past 100k/500k
bodies to keep very large budgets responsive. Edges need no changes: the
layout already returns every edge between loaded nodes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
index_repository canonicalizes repo_path but imposes no workspace boundary,
so a caller-supplied path can resolve anywhere on disk. Add an opt-in
CBM_ALLOWED_ROOT: when set, a repo_path that resolves outside it is refused
(reusing the same containment check the read sinks use). Unset by default, so
the standard index-the-given-path behaviour is unchanged; deployments where
repo_path may be influenced by an untrusted caller can set a boundary.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The UI handlers accumulate JSON with `pos += snprintf(buf + pos, size - pos,
...)`. snprintf returns the length it WOULD have written, so on truncation
pos ran past the end of the buffer and the next call's `size - pos` wrapped
to a huge value and wrote out of bounds — reachable through the directory
browser, whose readdir entries can fill the buffer before the trailing
parent/roots appends. Route every accumulator through a clamping
http_appendf() helper that pins pos at the buffer size on truncation. Adds a
forked-child regression test that browses a directory wide enough to fill the
buffer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The macOS digest path invoked `shasum -a CBM_SZ_256` — a literal string
(not a macro; string literals do not expand) naming an invalid algorithm, so
the digest always failed on macOS, and the self-update path then treated a
could-not-verify result as non-fatal and installed the download regardless.
Use the correct algorithm, and install only a positively-verified download:
a mismatch, a missing checksum entry, or an unavailable hash tool now all
abort. Adds a known-vector digest regression test.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
base_branch is spliced into `git diff --name-only "<base>"...HEAD`. The
existing check rejected shell metacharacters but not a leading '-', so a
value like --output=<path> was passed to git as an option and wrote the diff
to an arbitrary file. A real git ref never starts with '-'; reject it
alongside the metacharacter check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The graph-UI server binds to loopback only and reflected a localhost Origin
for CORS, but did not inspect the Host header — so a request routed to the
local port under a foreign name (a rebinding DNS record, a proxy) was served
normally, reaching the state-changing endpoints. Parse the Host header and
refuse any request whose Host is not a loopback address before dispatch; a
request with no Host (local HTTP/1.0 tooling) still proceeds. Adds a
live-socket regression test.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
get_code_snippet resolved a result path and confirmed it stayed under the
project root before reading, but the search_code source-attach path built
the same kind of path and read it with no such check — so a result whose
indexed path resolves outside the root (a `..` segment, or a symlink /
junction followed during discovery) could be read back into the response.
Extract the existing containment check into a shared cbm_path_within_root()
and route both read sinks through it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The import path allocated the destination from the plaintext original_size
field in artifact.json and passed that value to the decoder truncated
through int. A large real database already wraps that cast (a full-kernel
index is ~14 GB), and a doctored field could additionally desync the decoder
capacity from the actual buffer. Read the decompressed size from the zstd
frame's own content-size header, require the metadata field to match it, cap
the total, and run the decoder with a size_t capacity equal to the
allocation. cbm_zstd_decompress now takes size_t lengths and returns int64_t.
Adds a regression test that a mismatched original_size is rejected.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The result projection is materialized per row into fixed-width stack
arrays (CBM_SZ_32 columns) in execute_return_simple and its siblings,
indexed by the parsed RETURN item count — which the parser left
unbounded. A RETURN/WITH clause with more than 32 projection items
wrote past those arrays. Reject an over-wide projection at parse time,
with a fork-isolated regression test driving a 48-column RETURN.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The guard's sparse defs scaled WITH the input (one per 2000 lines), so
the fixture inherited a slice of the separate per-def sibling-scan cost
(O(defs x siblings) — its own tracked finding): an n^2/2000 term that
is negligible on clang-macOS but dominant enough on windows-CLANG64
ASan to push LINEAR walk code to a 43x measured ratio, over the 40x
bound. Ten defs at fixed positions regardless of size keep the
anti-vacuous breadth check while the sibling-scan term stays 10 x n =
linear. Measured after the change: fixed code 22x, pre-merge quadratic
walk 339x — the discriminator now has ~2x headroom on the green side
and ~8x on the red side across all three toolchains measured.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An absolute wall-clock bound conflates machine speed with complexity
class: the gcc-13-ARM ASan CI leg runs this extraction ~200x slower
than clang at measured-perfectly-linear complexity (27.1/54.2/108.3/
214.8s for 50k/100k/200k/400k siblings) and flunked a 30s bound on
LINEAR code — the leg has always been the slow one (71min in the last
fully green run, unchanged since).
Assert the complexity class instead: extract the same comment-sibling
fixture at 20k and 400k lines and bound the growth ratio at 40x.
Calibrated from measurement, not models — the ASan builds carry a large
linear per-line baseline that dilutes small-growth ratios (at 8x growth
the known-quadratic pre-merge walk measured 22.4x and slipped under a
24x bound), while at 20x growth linear code measures ~20x on both
toolchains and the quadratic walk measures 64x. Verified RED (64.4x,
fails) against the pre-merge walk_defs and GREEN (19x) on the fix; the
test now prints both timings unconditionally for future triage.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The no-skips lint gate forbids bare SKIP(); the parallel-recovery guard
is genuinely platform-specific (POSIX fork harness — Windows coverage
comes from the smoke invariants), so it uses SKIP_PLATFORM like the
neighboring supervisor guards.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An 81k-file TypeScript corpus previously never finished indexing. Three
stacked defects, found by stack-sampling the live grind and reading the
supervisor's quarantine records:
1) walk_defs pushed children via index-based ts_node_child(i), which is
O(i) per call in tree-sitter — O(n^2) per wide node. A 3.5 MB file
with ~580k flat comment siblings needed ~1.7e11 iterator steps and
hung extraction past the supervisor's quiet-timeout. All six descent
sites (including the default loop) now collect children linearly with
a TSTreeCursor (wd_collect_children); small nodes keep the direct
index path. Guard: extract_wide_flat_file_is_linear (400k comment
siblings — the monster's exact shape — bounded at 30s).
2) The sequential cross-LSP driver handed the FULL def list to a full
per-file registry build+finalize — O(files x defs); 74% of samples
sat in build_qn_index. The parallel resolve worker's dispatch
(module-def filter -> shared prebuilt registries -> filtered
fallback) is now extracted into cbm_pxc_dispatch_file and drives
BOTH pipelines — one path, one semantics. The shared registries
live in a caller-owned arena that outlives the calls pass
(resolved_calls borrow registry strings; freeing at pass end was a
use-after-free ASan caught in pass_calls). Rust is exempt from the
def filter: Cargo-manifest cross-crate resolution needs defs from
OTHER workspace crates that the own+imports filter drops (this
starvation predated the unification on the parallel path). Guard:
pipeline_seq_ts_cross_uses_shared_registry (full builds 40 -> 0,
cross-file edge preserved).
3) The crash supervisor's recovery re-ran the worker SINGLE-THREADED to
keep one exact marker. At scale that fell into the sequential crawl,
was killed as a hang mid-pass, and the stale extraction marker got
four innocent files quarantined, one 15-minute retry at a time.
Recovery (and the terminal partial run) now re-run PARALLEL; the
marker is an append journal ("S <rel>" / "D <rel>", written by
extraction and both resolve drivers), the suspect set is the open-S
in-flight set, and a file is quarantined only when it recurs across
two consecutive failed runs — one per round, oldest open S first;
disjoint consecutive sets stop the loop rather than blame an
innocent. Guard: index_recovery_parallel_quarantines_crasher
(single-threaded spawn count must stay zero; verified RED against
the old loop).
Also: parse_ts_type_text gains a dynamic work budget proportional to
the input (1M + 64x source bytes, CBM_TS_TYPE_BUDGET override, warn
once then degrade to unknown) replacing unbounded recursion cost.
Measured on microsoft/TypeScript (81,398 files): infinite -> 33s cold
full index, 295k nodes / 779k edges, zero dangling edges, zero files
skipped — the monster file itself now indexes cleanly. Full 9-language
sweep green with no quality flags; kernel unchanged at 390s / 8.5M
nodes / 0 dangling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#856: native `fetch()` calls (Node 18+ built-in, and the identical
browser/WHATWG API — no static way to tell them apart, and no need to:
both make a real outbound HTTP request) were invisible to cross-repo
intelligence. A bare `fetch(url)` has no import and typically no local
definition, so registry resolution comes back empty and the call was
silently dropped instead of becoming an HTTP_CALLS edge.
The obvious fix — add "fetch" to the http_libraries substring table in
service_patterns.c — is wrong: that table is consulted by an early,
unconditional check in both pass_calls.c and pass_parallel.c that runs
before/regardless of whether the callee resolves to a real local
definition (by design, so `axios.get(url)` still classifies even when the
registry mis-binds bare `get` to an unrelated local method). That's safe
for "axios"/"requests" because nobody names their own function that; it
is not safe for "fetch", which collides with a plausible local identifier
(`function fetch(){}` or `const fetch = () => {}`).
Instead, cbm_service_pattern_is_global_fetch() is a new, narrow, exact-
match check consulted only in the *empty-resolution* fallback in both
files — the existing #523 path for unindexed external libraries. By the
time that branch runs, the registry has already had its chance to
resolve "fetch" to a local/imported definition; only a genuine miss
reaches the new check. This also means a member call like `repo.fetch()`
is naturally excluded (its callee_name is "repo.fetch", not the bare
"fetch" this checks for).
pass_parallel.c's empty-resolution branch calls the low-level
emit_http_async_service_edge() directly rather than going through
emit_service_edge(), which re-derives its own classification from
res->qualified_name via cbm_service_pattern_match() — a call with
"fetch" would come back CBM_SVC_NONE there and silently fall through to
a plain CALLS edge.
Three new tests in test_pipeline.c:
- bare fetch() -> HTTP_CALLS, sequential path (< 50 files)
- bare fetch() -> HTTP_CALLS, forced through the parallel resolver
(>= 50 files)
- a local `function fetch(){}` shadowing the global -> plain CALLS to
the local definition, zero HTTP_CALLS (the false-positive this PR
must not introduce), bundled with a `repo.fetch()` member-call check
in the first test
Verified end-to-end, not just by inspection: prod build clean under
-Wall -Wextra -Werror; test build clean under ASan+UBSan; full suite
5936 passed, 1 skipped (pre-existing, unrelated), 0 failed; clang-format
clean on all touched files; lint-cppcheck's pre-existing failures
(extract_defs.c, compat_regex.c) reproduced identically on unmodified
main via stash/pop, confirming they predate this change.
Refs #592.
Signed-off-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com>
The supervised-worker fast path deliberately skips the pipeline
teardown (the OS reclaims the process wholesale) and pairs that with
_Exit, which bypasses atexit/LeakSanitizer. The production worker in
run_cli has both halves; the test binary's worker emulation
(tf_maybe_run_index_worker) only had the first: it returned normally
through main(), LeakSanitizer ran at exit, reported the intentionally
unfreed pipeline and forced exit code 1 — so the supervisor read a
healthy index as worker_failed and IDX832 went red on exactly the
platforms whose ASan builds run a leak pass (all four Linux legs;
macOS/Windows ASan have none). Reproduced and verified fixed in an
ubuntu:22.04 arm64 container: reap outcome=clean, mcp suite 140/0.
Mirror the production exit: deliver the response file, fflush, _Exit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_mem_init is init-once (guarded by g_initialized), so the
backpressure test's setenv(CBM_MEM_BUDGET_MB=1) + re-init 'restore'
was a silent no-op: whichever init ran first fixed the budget for the
whole process. In the full runner this test's init won, locking a 1 MB
budget that leaked into every later budget consumer —
mem_over_budget_low_rss red on every platform, and all subsequent
pipeline tests silently running with backpressure permanently engaged.
Add cbm_mem_set_budget_for_tests(): writes the budget directly without
touching the init guard. The test now saves the caller-visible budget,
sets 1 MB via the hook, and restores before asserting. Env dance
removed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The semantic pass kept four dense 768-float vectors resident per function
(ri/api/type/deco = 12.3 KB per func, ~9.4 GB transient on the linux kernel)
solely so candidate pairs could be cosine-scored after LSH. Replace the
resident floats with 4-bit quantized codes (~0.5 KB each, ~6x): new
src/semantic/rotsq.{h,c} — rotated scalar quantization from the RaBitQ
family of algorithms (Gao et al., SIGMOD 2024/2025), implemented from the
papers in plain C11 (no Eigen/library dependency): a deterministic
XXH3-seeded +-1 diagonal followed by a Fast Walsh-Hadamard rotation
(768 -> 1024 padded) spreads vector mass so per-vector scalar quantization
is near-optimal; the inner product is recovered from the codes with the
exact SQ expansion (one u8 integer dot + four multiplies) — a deterministic
pure function of the codes. Named rotsq, not rabitq: this is the family
core, not the papers' exact codebook construction.
Producers build each dense vector in a worker-local buffer, encode, and
discard; the int8 qvec export is unchanged. LSH signatures are computed
from DEQUANTIZED codes against hyperplanes in the rotated basis — random
hyperplanes are basis-agnostic, so no dense originals are retained and the
phase structure is untouched.
Estimator quality gated by sem_rotsq_ip_error_bounds (seeded vectors:
mean error < 1%, max < 4% of unit scale, ASan-green). Effect on output:
multi-threaded determinism preserved (xfs: two runs byte-identical);
SEMANTICALLY_RELATED edges shift once as scores move within ~1% of the
0.75 threshold (xfs: ~130 -> 203; sampled new pairs are high quality —
alloc/free, read-verify and collapse/insert families).
Suites: semantic 33 (incl. the new estimator gate), simhash 24,
pipeline 216 — zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Multi-threaded indexing produced a different graph on every run (xfs: node
labels flipping Class/Function/Macro, ~2k edge lines differing across USAGE/
CALLS/IMPORTS/SIMILAR_TO/SEMANTICALLY_RELATED) and lost edges vs repeated
runs. Five root causes, all order- or scheduling-dependence:
1. gbuf QN collisions resolved last-wins in MERGE ORDER: in C a struct, a
function and a macro can share one name, and the QN scheme does not encode
entity kind, so whichever entity a worker merged last overwrote the node's
label/name/lines/props (the flickering node set that fed every pass). Both
cbm_gbuf_upsert_node and the parallel-merge path now keep the SAME entity's
refresh semantics but resolve cross-entity collisions by a canonical
content rule -- lexicographically smallest (label, file_path, start_line,
name) -- and keep the label/name secondary indexes consistent when the
survivor changes (the old overwrite left nodes listed under their original
label, which is how the flicker reached find_by_label consumers).
2. Semantic-edges admission raced per-node max_edges budgets on shared atomic
counters under dynamic scheduling: scoring stays parallel and pure; pairs
carry (func index, candidate rank); one sequential pass admits in canonical
order. funcs[] itself is sorted by qualified name up front (the label-index
scan order is merge order).
3. Import resolution's symbol-name fallback returned the FIRST same-named node
in registration order; now the lexicographically-smallest-QN candidate.
4. Similarity entries inherited merge order into LSH chains + per-node cap
truncation, and pair ownership keyed on run-varying node ids; entries are
sorted by QN and ownership compares QNs (lsh entries carry the QN).
5. Call-neighbor context (semantic vectors + tokens) truncated MAX_CALLEES
subsets in edge-array order; neighbor names are sorted first.
Verified: four consecutive multi-threaded kernel-xfs runs produce byte-
identical sorted node AND edge sets (13341/58598, zero diff). The repro guard
repro_parallel_edge_determinism (MT x K mutual equality) is GREEN; a new
deliberately-RED repro_seq_parallel_equivalence tracks the remaining OPEN bug
that the sequential and parallel pipelines systematically disagree (~3459
USAGE / ~1666 WRITES lines on xfs) -- separate code paths, fix deferred.
Suites: pipeline 216, c_lsp 750, rust_lsp 506, py_lsp 63, java 94, kotlin 78,
php 278, registry 54, store_nodes 56, incremental 161, semantic 32, simhash 24
(one test updated: SIMILAR_TO pair direction is now canonical by QN), mcp 140,
subprocess 17 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Profiling the kernel showed 4 tiny trait-heavy rust files at ~90 s EACH in
cross-file resolve. Live call trees put the time not in lookups but in
REGISTRY CONSTRUCTION: cbm_pxc_filter_defs_for_file returns NULL for these
files (their path-derived module QNs never match the def_module_qn index
keys), so each one built its own registry over the ENTIRE project def set
(millions of entries at kernel scale).
Original premise corrected during implementation: only 4/87 rust files (on
the kernel rust subtree) hit the NULL-filter path; 83/87 resolve against
proper import-filtered subsets. A registry shared across ALL rust files
would hand the subset files a superset and could add edges. The fix is
therefore CONDITIONAL, byte-identical for every file by construction:
- filter NULL -> ONE shared all_defs registry (built once in pipeline.c,
finalized, read_only-sealed; module_qn=NULL is safe: an instrumented probe
found 0 NULL def_module_qn across all_defs, so the per-file def_mod
fallback never fired anyway). Kills the amplifier.
- filter SUBSET -> today's per-file subset build, unchanged.
The cross-crate Cargo manifest reaches the shared path through
cbm_pxc_get_rust_manifest() (same worker-thread thread-local the fallback
read), preserving cross-crate resolution exactly.
Byte-identity verified empirically: single-threaded sorted edge-set diff on
the kernel rust subtree, fix off vs on -- zero difference outside
SEMANTICALLY_RELATED edges, which self-differ between two runs of the SAME
unfixed binary (the pre-existing semantic non-determinism guarded by the
repro_parallel_edge_determinism RED test). New suite test
rustlsp_shared_registry_resolves_like_per_file exercises the shared path.
Suites: rust_lsp 506, c_lsp 750, registry 54, pipeline 216, java_lsp 94,
kotlin_lsp 78, php_lsp 278, store_nodes 56 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel