Score every Function/Method/Class node during indexing and persist the
result as a numeric "importance" key inside the node's existing
properties_json:
importance = sqrt(num_refs) * priv * generic * distinct * test_penalty
num_refs is the incoming CALLS + USAGE degree. The multipliers demote
private (leading-underscore) and generic names (defined in >= 5 distinct
files), promote distinctive identifiers (snake_case or camelCase, len >=
8), and demote test scaffolding -- symbols in a test file per the graph's
own cbm_is_test_path() classifier, or targets of an incoming TESTS edge.
Weighted degree only; a transitive (PageRank) refinement is deliberately
not built here until it can be shown to beat this on a fixed judgment
set.
No schema change and no CBM_INDEX_FORMAT_VERSION bump: the score lives in
the existing properties_json TEXT column, so no index is forced to
rebuild and indexes written by older builds simply lack the key.
Three failure modes this shape invites, all of them silent, are addressed
directly:
* Cost. The generic-name multiplier needs |{files a name is defined in}|.
Computing that per NODE with a pairwise file-path scan is O(k^3) in a
same-name group of size k -- ruinous on real corpora, where a single
Java name group reaches k > 4000. Here the count is computed once per
distinct NAME and memoized, and file paths are deduplicated through a
hash set, so total distinct-file work is linear in the graph. Measured
cold full index of a 666 MB Java corpus, same -O2 binary: 85 s with the
pass disabled, 80 s with this implementation, 481 s with the per-node
shape.
* Registration. PREDUMP_PASS_COUNT is now derived from the pass table
with sizeof instead of being hand-written. A hand-written count that
lags an appended entry silently skips the last-registered pass with
every test still green.
* Incremental. The incremental post-pass sequence runs the pass too, over
a graph rehydrated from the store whose nodes already carry the key.
The write-back overwrites an existing key in place instead of appending
a second one, which would otherwise corrupt properties_json into
{"importance":1.0,...,"importance":2.0}.
tests/test_importance.c covers the formula and binds all three failure
modes; the complexity suite gains a GATED counter asserting the pass's
same-name-group work stays linear (~2x, not ~4x) under a corpus doubling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Co-authored-by: petercoxphoto <20230727+petercoxphoto@users.noreply.github.com>
The no-skips lint gate (scripts/check-no-test-skips.sh) rightly rejected the
throughput-report test's two SKIP() calls:
- CBM_SKIP_PERF=1 is deliberate operator configuration, not a hidden
environment failure: reporting is off by request, so the test PASSes with a
stderr note instead of skipping.
- an uncreatable report dir IS an environment failure and now FAILs with the
remedy in the message (set CBM_COMPLEXITY_REPORT_DIR), per the policy text.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Finding #1669 took an 11-corpus A/B across two release binaries. This
suite makes that bug class fail a unit test in seconds, on every
platform, from tiny corpora.
Method: build k and 2k REPLICATED module copies, run the full in-process
pipeline on both, assert counter RATIOS. Independent copies mean every
extensive quantity — nodes, edges, Σ per-file registry defs — must grow
linearly (ratio ~2). A files x corpus coupling makes per-file work itself
grow with k and lands at ratio ~4. Ratios expose the exponent regardless
of absolute scale, so 60-120 files suffice.
Verdicts are pure functions of (code, input): gates ride ONLY on
deterministic work counters and data-product counts, never on wall time.
Throughput (nodes/s, edges/s) is information-only, written to
private/benchmarks/complexity-<ts>.json (local, gitignored; skipped
under CBM_SKIP_PERF where rates are meaningless).
Two corpus shapes, both needed:
- Independent modules (java/py/go/ts templates): catches cross-module
contamination and dedup breakage. The #1669 bug is GREEN here — fully
closed modules filter perfectly, which is exactly why it survived.
- The growing shared package (bigpkg): one Java package whose file count
scales with k — the real-repo shape (files concentrate in large
packages). The JVM namespace filter branch makes per-file work track
package size, so this corpus is the honest #1669 reproducer:
ratio 4.00 RED on the pre-fix tree, 4.00 RED for a module-scoped
overlay, 2.00 GREEN for the own-file overlay. It discriminated the
correct fix design before the fix was written.
Both legs of every pair exceed MIN_FILES_FOR_PARALLEL(50): below it the
sequential path runs, which builds no shared registries and would be the
wrong code path to gate (its per-file cost is bounded by the 50-file
ceiling).
Recorded but deliberately NOT gated, with reasons at the case:
tail_candidates and fallback_rows are legitimately superlinear under
replication until those scans are bounded, and measured ~1 ns/unit.
Every ratio gate carries a non-vacuousness floor on the base counter so
broken counter wiring fails loudly instead of green-washing
(cbm_pxc_count_perfile_defs feeds the overlay path into the same
counter the fallback path already used; wired for Java, extend to the
TS overlay when touching ts_lsp).
Dynamic coverage: languages iterate CBM_LANG_COUNT; embedded templates
cover the LSP-hybrid languages, and tests/fixtures/complexity/<lang>/
dirs are auto-discovered so a new language joins the guard by dropping
fixtures. Uncovered languages are listed in the report with the reason.
The local report additionally carries per-language node/edge counts with
ratios and a per-pass elapsed_ms table per run (captured via a TEE log
sink during the in-process pipeline runs) — trend data for humans, still
never a gate.
Refs #1669.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
