perf(relevance): order BM25 query terms once, not once per document (#3468)

## Description

`bm25_score` collected `query_freq.keys()` into a `Vec` and sorted it on
every call. That sort exists only to make matched-term output
deterministic, and the query is fixed for a whole batch — so
`score_batch` repeated the same allocation and sort once per document.

Closes # N/A

## Type of Change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `query_freq` is now a `BTreeMap<String, usize>` instead of a
`HashMap`. This **removes** the work rather than hoisting it: the map
already iterates in key order, so the collect-and-sort disappears
entirely and both `score` and `score_batch` benefit. Net effect on
`bm25_score` is fewer lines, not more.
- Added `benches/bm25_scoring.rs` (criterion, following the existing
three benches) varying document count and query length.
- Added `examples/bm25_query_prep_parity.rs`, the harness backing the
bit-identical claim below.
- New test
`matched_terms_are_lexicographic_regardless_of_context_order`. To be
clear, this does **not** fail on the base commit — the base sorts too.
It is a forward guard: swapping back to a `HashMap` without restoring
the sort would make matched-term order nondeterministic, and this
catches that.

**Why `BTreeMap` rather than a hoisted sorted `Vec`.** A `Vec` passed
down from `score_batch` would keep iteration contiguous, but it needs a
signature change, a second construction site in `score`, and leaves the
sort in the code. The `BTreeMap` deletes the sort outright, and it
measured faster in every shape tested, so the extra machinery was not
worth it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo test --release --lib
925 passed; 0 failed; 1 ignored

$ cargo test --release --lib relevance
49 passed; 0 failed

$ cargo clippy --release --all-targets     # no errors
$ cargo fmt --check                        # clean

# Python side, against a maturin build of THIS branch (see note below):
$ pytest tests/test_relevance.py tests/test_relevance_extra.py \
         tests/test_relevance_split.py tests/parity/ \
         tests/test_acceptance.py tests/test_issue_746_tool_search.py -q
124 passed, 4 skipped
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Rust release profile, criterion 0.8,
Python 3.13.11, worktree off `upstream/main` @ `e67b3c8a`.
- Exact command / steps: `cargo run --release --example
bm25_query_prep_parity > out.txt` on this branch and with
`src/relevance/bm25.rs` stashed, then `diff`. For timings, `cargo bench
--bench bm25_scoring` on each side, run twice in opposite order to
control for thermal drift. Python tests were run against a `maturin
build --release` wheel from this branch, not a copied `.so`.
- Observed result: **bit-identical output.** The parity harness prints
raw IEEE-754 score bits (not decimals), rankings, reasons and matched
terms for empty batches/contexts/documents, no-overlap, repeated query
terms, exact ties, the long-token bonus, and 3 configs x 5 document
counts x 3 query lengths. Baseline and candidate produce the same 3,739
lines, SHA `4e48098593e36aa1937e1629c4048e1280cfab47`. Timings,
candidate faster in all nine shapes:

  | shape | baseline | candidate | delta |
  |---|---|---|---|
  | 10 docs, 4-word query | 51.55 us | 50.22 us | -2.6% |
  | 10 docs, 16-word query | 60.72 us | 56.49 us | -7.0% |
  | 10 docs, 64-word query | 65.94 us | 61.39 us | -6.9% |
  | 100 docs, 4-word query | 515.4 us | 504.7 us | -2.1% |
  | 100 docs, 16-word query | 590.6 us | 553.2 us | -6.3% |
  | 100 docs, 64-word query | 597.6 us | 562.4 us | -5.9% |
  | 1000 docs, 4-word query | 5.184 ms | 5.133 ms | -1.0% |
  | 1000 docs, 16-word query | 5.926 ms | 5.639 ms | -4.8% |
  | 1000 docs, 64-word query | 5.975 ms | 5.631 ms | -5.8% |

The reverse-order confirmation run reproduced the 64-word column at
-5.5%, -4.4% and -5.9%.
- Not tested: real proxy or agent workloads — these are synthetic
documents from a fixed 16-word vocabulary, and no end-to-end latency
claim is made. Only x86-free Apple-silicon measurements. CJK content
takes a separate scorer in `text_crusher` and is unaffected.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; no flag or channel gates this.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Scores are bit-identical, and
rankings, reasons and matched-term ordering are unchanged.
- Kill switch / disable path: none needed; there is no behavioral switch
to disable.
- Unsafe override required: none.
- Qualification impact: none. Scoring output is unchanged, so no
re-qualification of compression decisions is required.
- Rollback path: revert the commit. `bm25_score` is a pure function over
its inputs with no persisted state, so nothing needs migrating in either
direction.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- The comment above the loop was rewritten rather than deleted: it
explains why terms are visited alphabetically at all, which is still the
load-bearing reason the ordering is not an implementation detail.
- Documentation: no user-facing docs describe this internal scoring
path; the behavioral contract lives in that comment and in the new
ordering test.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
trs-80
2026-09-24 16:45:35 -05:00
committed by GitHub
co-authored by Claude Opus 5
parent c766bdb2b3
commit 6880984b4a
4 changed files with 288 additions and 15 deletions
+4
View File
@@ -210,3 +210,7 @@ harness = false
[[bench]]
name = "auth_mode"
harness = false
[[bench]]
name = "bm25_scoring"
harness = false
@@ -0,0 +1,63 @@
//! Criterion benchmark for `BM25Scorer::score_batch`.
//!
//! Query preparation used to happen once per document: every `bm25_score`
//! call collected and sorted the same query keys. Cost therefore scales with
//! documents x query length, so both are varied here.
use std::hint::black_box;
use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion, Throughput};
use headroom_core::relevance::{BM25Scorer, RelevanceScorer};
const VOCAB: [&str; 16] = [
"handler",
"parse",
"timeout",
"cache",
"retry",
"index",
"error",
"550e8400-e29b-41d4-a716-446655440000",
"9f8e7d6c-1234-4321-abcd-0123456789ab",
"12345",
"987654",
"resolve",
"connection",
"serialize",
"upstream",
"budget",
];
fn text(seed: usize, words: usize) -> String {
let mut parts: Vec<&str> = Vec::with_capacity(words);
for i in 0..words {
parts.push(VOCAB[(seed * 7 + i * 3) % VOCAB.len()]);
}
parts.join(" ")
}
fn bench_score_batch(c: &mut Criterion) {
let scorer = BM25Scorer::default();
let mut group = c.benchmark_group("bm25/score_batch");
for docs in [10usize, 100, 1000] {
for query_words in [4usize, 16, 64] {
let items: Vec<String> = (0..docs).map(|d| text(d, 60)).collect();
let refs: Vec<&str> = items.iter().map(|s| s.as_str()).collect();
let context = text(9_999, query_words);
group.throughput(Throughput::Elements(docs as u64));
group.bench_with_input(
BenchmarkId::from_parameter(format!("{docs}d_q{query_words}w")),
&(refs, context),
|b, (refs, context)| {
b.iter(|| black_box(scorer.score_batch(black_box(refs), black_box(context))))
},
);
}
}
group.finish();
}
criterion_group!(benches, bench_score_batch);
criterion_main!(benches);
@@ -0,0 +1,188 @@
//! Dump BM25 scores, rankings and matched terms so the query-preparation
//! change can be diffed byte-for-byte against its baseline.
//!
//! Scores print as raw IEEE-754 bits, not decimals: the change alters when
//! query terms are ordered, not the order itself, so accumulation must stay
//! bit-identical rather than merely close.
//!
//! `cargo run --release --example bm25_query_prep_parity > out.txt` on each
//! side, then `diff`.
use headroom_core::relevance::{BM25Scorer, RelevanceScorer};
/// xorshift64*, so both sides generate the identical corpus without
/// depending on a rand version.
struct Rng(u64);
impl Rng {
fn next(&mut self) -> u64 {
let mut x = self.0;
x ^= x >> 12;
x ^= x << 25;
x ^= x >> 27;
self.0 = x;
x.wrapping_mul(0x2545_F491_4F6C_DD1D)
}
fn below(&mut self, n: u64) -> u64 {
if n == 0 {
0
} else {
self.next() % n
}
}
}
const VOCAB: [&str; 16] = [
"handler",
"parse",
"timeout",
"cache",
"retry",
"index",
"error",
"550e8400-e29b-41d4-a716-446655440000",
"9f8e7d6c-1234-4321-abcd-0123456789ab",
"12345",
"987654",
"a",
"zz",
"Mixed_Case_Token",
"UPPER",
"repeated",
];
fn text(rng: &mut Rng, words: usize) -> String {
let mut parts: Vec<&str> = Vec::with_capacity(words);
for _ in 0..words {
parts.push(VOCAB[rng.below(VOCAB.len() as u64) as usize]);
}
parts.join(" ")
}
fn dump(name: &str, scorer: &BM25Scorer, items: &[String], context: &str) {
let refs: Vec<&str> = items.iter().map(|s| s.as_str()).collect();
let batch = scorer.score_batch(&refs, context);
println!(
"===== {name} (n={}, context={:?}) =====",
items.len(),
context
);
for (i, s) in batch.iter().enumerate() {
// Raw bits: a reordered summation would change these even when the
// rounded decimal looks identical.
println!(
" batch[{i}] bits={:016x} reason={:?} matched={:?}",
s.score.to_bits(),
s.reason,
s.matched_terms
);
}
// Ranking is what callers actually consume, so compare it explicitly.
let mut ranked: Vec<usize> = (0..batch.len()).collect();
ranked.sort_by(|a, b| {
batch[*b]
.score
.partial_cmp(&batch[*a].score)
.unwrap_or(std::cmp::Ordering::Equal)
.then_with(|| a.cmp(b))
});
println!(" ranking={ranked:?}");
// The single-item path shares bm25_score, so verify it agrees with the
// batch path item for item (reasons and caps differ by design).
for (i, item) in items.iter().enumerate().take(20) {
let single = scorer.score(item, context);
println!(
" single[{i}] bits={:016x} reason={:?} matched={:?}",
single.score.to_bits(),
single.reason,
single.matched_terms
);
}
println!();
}
fn main() {
let scorer = BM25Scorer::default();
let unnormalized = BM25Scorer::new(1.5, 0.75, false, 10.0);
let tuned = BM25Scorer::new(0.5, 0.0, true, 3.0);
// Degenerate shapes first.
dump("empty-batch", &scorer, &[], "error handler");
dump("empty-context", &scorer, &["error here".into()], "");
dump(
"empty-docs",
&scorer,
&[String::new(), String::new()],
"error",
);
dump("single-doc", &scorer, &["error handler".into()], "error");
dump(
"no-overlap",
&scorer,
&["aaa bbb ccc".into(), "ddd eee".into()],
"zzz yyy",
);
// Repeated query terms: query frequency > 1 multiplies each term score,
// so a dropped or double-counted key shows up immediately.
dump(
"repeated-query-terms",
&scorer,
&[
"error error handler".into(),
"handler parse".into(),
"error".into(),
],
"error error error handler",
);
// Exact ties: identical documents must keep identical scores and a
// stable ranking.
dump(
"exact-ties",
&scorer,
&vec!["error handler parse".to_string(); 6],
"error parse",
);
// Long tokens trip the >=8 char bonus in finalize_score.
dump(
"long-token-bonus",
&scorer,
&[
"550e8400-e29b-41d4-a716-446655440000 tail".into(),
"short a b".into(),
],
"550e8400-e29b-41d4-a716-446655440000 short",
);
// Vary document count and query length across configs.
for (label, s) in [
("default", &scorer),
("unnormalized", &unnormalized),
("tuned", &tuned),
] {
for docs in [1usize, 2, 17, 64, 250] {
for query_words in [1usize, 4, 32] {
let mut rng = Rng(0x9E37_79B9_7F4A_7C15 ^ (docs as u64) << 8 ^ query_words as u64);
let items: Vec<String> = (0..docs)
.map(|_| {
let words = 1 + rng.below(40) as usize;
text(&mut rng, words)
})
.collect();
let context = text(&mut rng, query_words);
dump(
&format!("{label}-d{docs}-q{query_words}"),
s,
&items,
&context,
);
}
}
}
}
+33 -15
View File
@@ -13,7 +13,7 @@
//! high-signal matches). Final score is clamped to `[0, 1]` via
//! `RelevanceScore::new`.
use std::collections::HashMap;
use std::collections::{BTreeMap, HashMap};
use std::sync::LazyLock;
use regex::Regex;
@@ -87,7 +87,7 @@ impl BM25Scorer {
fn bm25_score(
&self,
doc_tokens: &[String],
query_freq: &HashMap<String, usize>,
query_freq: &BTreeMap<String, usize>,
avg_doc_len: f64,
) -> (f64, Vec<String>) {
if doc_tokens.is_empty() || query_freq.is_empty() {
@@ -113,16 +113,13 @@ impl BM25Scorer {
let mut matched: Vec<String> = Vec::new();
let idf = 2.0_f64.ln();
// Iterate query_freq in HashMap order — Python iterates dict
// order (insertion order in 3.7+). For matched_terms we only
// care about MEMBERSHIP not ordering downstream, but we sort
// tokens alphabetically here for deterministic test output
// when multiple terms match.
let mut keys: Vec<&String> = query_freq.keys().collect();
keys.sort();
for term in keys {
let qf = query_freq[term];
// Python iterates dict order (insertion order in 3.7+). For
// matched_terms we only care about MEMBERSHIP not ordering
// downstream, but terms are visited alphabetically here for
// deterministic test output when multiple terms match. A
// `BTreeMap` already iterates in key order, so batch scoring no
// longer re-sorts the same query keys once per document.
for (term, qf) in query_freq {
let Some(&f) = doc_freq.get(term.as_str()) else {
continue;
};
@@ -132,7 +129,7 @@ impl BM25Scorer {
let numerator = f * (self.k1 + 1.0);
let denominator = f + self.k1 * (1.0 - self.b + self.b * doc_len / avgdl);
let term_score = idf * numerator / denominator;
score += term_score * qf as f64;
score += term_score * *qf as f64;
}
(score, matched)
@@ -158,7 +155,7 @@ impl RelevanceScorer for BM25Scorer {
let context_tokens = self.tokenize(context);
// Build query frequency map from context.
let mut query_freq: HashMap<String, usize> = HashMap::new();
let mut query_freq: BTreeMap<String, usize> = BTreeMap::new();
for t in &context_tokens {
*query_freq.entry(t.clone()).or_insert(0) += 1;
}
@@ -197,7 +194,7 @@ impl RelevanceScorer for BM25Scorer {
.collect();
}
let mut query_freq: HashMap<String, usize> = HashMap::new();
let mut query_freq: BTreeMap<String, usize> = BTreeMap::new();
for t in &context_tokens {
*query_freq.entry(t.clone()).or_insert(0) += 1;
}
@@ -351,6 +348,27 @@ mod tests {
assert_eq!(scores.len(), 50);
}
#[test]
fn matched_terms_are_lexicographic_regardless_of_context_order() {
// Query preparation is what makes matched_terms deterministic. The
// context here is deliberately in reverse lexicographic order, so a
// map that iterates in insertion or hash order would fail this.
let doc = "zulu yankee mike delta alpha";
let context = "zulu yankee mike delta alpha";
let expected = ["alpha", "delta", "mike", "yankee", "zulu"];
let single = scorer().score(doc, context);
assert_eq!(single.matched_terms, expected);
// Every document in a batch must get the same ordering, not just
// the first one: preparation is shared across the whole batch.
let items = vec![doc; 4];
for score in scorer().score_batch(&items, context) {
// The batch path caps matched_terms at 5.
assert_eq!(score.matched_terms, expected);
}
}
// ---------- BM25 formula ----------
#[test]