mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-29 07:13:44 +08:00
Martin Vogel
09ce20afc8
Four changes to reduce false positives from 382K to 58K SIMILAR_TO edges on the Linux kernel while maintaining ~98% precision: 1. MIN_NODES raised to 30 (leaf tokens) — aligns with BigCloneBench standard of 50 raw source tokens / 6 lines minimum clone size 2. collect_ast_tokens now records LEAF nodes only, not internal grammar nodes. Leaf-only counting is language-agnostic: leaves correspond to actual source tokens regardless of grammar verbosity (C tree-sitter produces 2x more internal nodes than Python for the same logic) 3. hash_trigrams tracks unique structural trigram count via hash set. Functions with < 32 unique structural trigrams are skipped — too few distinct features for K=64 MinHash to produce reliable estimates 4. Structural-weighted MinHash via repetition: each trigram gets weight 0-3 based on count of non-normalised tokens (I/S/N/T = normalised). Weight-0 trigrams (all-normalised, e.g. I|I|I) are pure noise and skipped entirely. Weight 1-3 trigrams use repetition-based weighted MinHash — higher weight = more hash attempts per seed = more likely to dominate the signature. Achieves IDF-like effect without needing corpus-wide statistics. Also adds O(1) hash-set dedup in LSH query (replacing O(n) linear scan) and bucket size cap of 200 to skip oversized buckets. Linux kernel: 1:41, similarity 7.7s, 395K fp, 58K edges, ~98% precision. Tests updated with larger Go functions to meet new thresholds.