Add scripts/check-no-test-skips.sh (run from lint) which fails the lint phase on any plain SKIP() or direct tf_skip_count manipulation; only SKIP_PLATFORM() (for genuinely platform-specific tests) is tolerated. Add FAIL() and SKIP_PLATFORM() helpers to the test framework and convert the remaining SKIP()/perf-gated skips across the suite into pass-or-fail assertions, so a suite that cannot meet its preconditions reports a red failure instead of a silent skip.
Four changes to reduce false positives from 382K to 58K SIMILAR_TO edges
on the Linux kernel while maintaining ~98% precision:
1. MIN_NODES raised to 30 (leaf tokens) — aligns with BigCloneBench
standard of 50 raw source tokens / 6 lines minimum clone size
2. collect_ast_tokens now records LEAF nodes only, not internal grammar
nodes. Leaf-only counting is language-agnostic: leaves correspond to
actual source tokens regardless of grammar verbosity (C tree-sitter
produces 2x more internal nodes than Python for the same logic)
3. hash_trigrams tracks unique structural trigram count via hash set.
Functions with < 32 unique structural trigrams are skipped — too few
distinct features for K=64 MinHash to produce reliable estimates
4. Structural-weighted MinHash via repetition: each trigram gets weight
0-3 based on count of non-normalised tokens (I/S/N/T = normalised).
Weight-0 trigrams (all-normalised, e.g. I|I|I) are pure noise and
skipped entirely. Weight 1-3 trigrams use repetition-based weighted
MinHash — higher weight = more hash attempts per seed = more likely
to dominate the signature. Achieves IDF-like effect without needing
corpus-wide statistics.
Also adds O(1) hash-set dedup in LSH query (replacing O(n) linear scan)
and bucket size cap of 200 to skip oversized buckets.
Linux kernel: 1:41, similarity 7.7s, 395K fp, 58K edges, ~98% precision.
Tests updated with larger Go functions to meet new thresholds.
Compute K=64 MinHash signatures from normalized AST node-type trigrams
during function extraction, then generate SIMILAR_TO edges via LSH
(b=32, r=2) for function pairs with Jaccard >= 0.95.
- src/simhash/minhash.{h,c}: MinHash compute, Jaccard, hex encode/decode,
LSH index with band hashing for O(n) candidate generation
- src/pipeline/pass_similarity.c: post-pass reads fingerprints from node
properties, builds LSH index, emits SIMILAR_TO edges with jaccard and
same_file metadata. Same-language only, max 10 edges per node.
- internal/cbm/cbm.h: fingerprint fields on CBMDefinition
- internal/cbm/extract_defs.c: compute_fingerprint() hook at 3 extraction
sites after complexity, skip functions with < 10 AST body nodes
- pass_definitions.c + pass_parallel.c: serialize fingerprint to "fp" hex
in properties_json for both sequential and parallel pipeline paths
- pipeline.c + pipeline_incremental.c: register pass_similarity in both
full and incremental post-pass lists
- tests/test_simhash.c: 28 tests across 4 suites (core, LSH, edge gen,
pipeline integration with generated Go project + incremental)
Martin Vogel