inject_calls_pattern_tokens read a node's CALLS edges straight out of the
graph buffer, which holds them in insertion order — and insertion order is the
order parallel workers happened to merge. It then injected the callee names in
that order and truncated at max_tokens, so which callees survived depended on
worker scheduling. Co-occurrence enrichment reads a window around each token
position, so the surviving vectors differed between runs while the document
frequencies (and therefore idf) stayed identical, which is why this showed up
as a handful of SEMANTICALLY_RELATED edges flickering and nothing else.
It now collects the neighbours through collect_sorted_call_neighbors(), the
same canonical ordering the rest of the pass already uses.
Three runs of the Go corpus now agree exactly — 334535 nodes / 1795409 edges
with vec=0e45744ff1042b51 idf=7e994a1d864a7b07 toks=2518a08a0b6af0c7 — and so
do php, kotlin and rust. The edge counts move slightly against the previous
build (php 57182 -> 57198, kotlin +17, rust +3): sorting changes which tokens
survive the truncation, so the semantic edges change with it. That is a
deliberate output change, not a regression.
Also adds the checksums that found it, all off unless their env var is set:
CBM_SEM_STATE_DUMP writes one hash per scoring input (codes, tf-idf, struct
profile, minhash) and one per corpus (ri vectors, idf, and the token SEQUENCE,
which is what differed here), and CBM_SEM_PAIR_DUMP writes every candidate pair
that passed the threshold with its score, in canonical order. This subsystem
has now produced three order-dependence bugs; a checksum at the seam is what
turns "the graph differs" into a localised answer in minutes, and it costs
nothing when the variable is unset.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
11cb476dff