mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
perf(relevance): order BM25 query terms once, not once per document (#3468)
## Description
`bm25_score` collected `query_freq.keys()` into a `Vec` and sorted it on
every call. That sort exists only to make matched-term output
deterministic, and the query is fixed for a whole batch — so
`score_batch` repeated the same allocation and sort once per document.
Closes # N/A
## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `query_freq` is now a `BTreeMap<String, usize>` instead of a
`HashMap`. This **removes** the work rather than hoisting it: the map
already iterates in key order, so the collect-and-sort disappears
entirely and both `score` and `score_batch` benefit. Net effect on
`bm25_score` is fewer lines, not more.
- Added `benches/bm25_scoring.rs` (criterion, following the existing
three benches) varying document count and query length.
- Added `examples/bm25_query_prep_parity.rs`, the harness backing the
bit-identical claim below.
- New test
`matched_terms_are_lexicographic_regardless_of_context_order`. To be
clear, this does **not** fail on the base commit — the base sorts too.
It is a forward guard: swapping back to a `HashMap` without restoring
the sort would make matched-term order nondeterministic, and this
catches that.
**Why `BTreeMap` rather than a hoisted sorted `Vec`.** A `Vec` passed
down from `score_batch` would keep iteration contiguous, but it needs a
signature change, a second construction site in `score`, and leaves the
sort in the code. The `BTreeMap` deletes the sort outright, and it
measured faster in every shape tested, so the extra machinery was not
worth it.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test --release --lib
925 passed; 0 failed; 1 ignored
$ cargo test --release --lib relevance
49 passed; 0 failed
$ cargo clippy --release --all-targets # no errors
$ cargo fmt --check # clean
# Python side, against a maturin build of THIS branch (see note below):
$ pytest tests/test_relevance.py tests/test_relevance_extra.py \
tests/test_relevance_split.py tests/parity/ \
tests/test_acceptance.py tests/test_issue_746_tool_search.py -q
124 passed, 4 skipped
```
## Real Behavior Proof
- Environment: macOS 15 (arm64), Rust release profile, criterion 0.8,
Python 3.13.11, worktree off `upstream/main` @ `e67b3c8a`.
- Exact command / steps: `cargo run --release --example
bm25_query_prep_parity > out.txt` on this branch and with
`src/relevance/bm25.rs` stashed, then `diff`. For timings, `cargo bench
--bench bm25_scoring` on each side, run twice in opposite order to
control for thermal drift. Python tests were run against a `maturin
build --release` wheel from this branch, not a copied `.so`.
- Observed result: **bit-identical output.** The parity harness prints
raw IEEE-754 score bits (not decimals), rankings, reasons and matched
terms for empty batches/contexts/documents, no-overlap, repeated query
terms, exact ties, the long-token bonus, and 3 configs x 5 document
counts x 3 query lengths. Baseline and candidate produce the same 3,739
lines, SHA `4e48098593e36aa1937e1629c4048e1280cfab47`. Timings,
candidate faster in all nine shapes:
| shape | baseline | candidate | delta |
|---|---|---|---|
| 10 docs, 4-word query | 51.55 us | 50.22 us | -2.6% |
| 10 docs, 16-word query | 60.72 us | 56.49 us | -7.0% |
| 10 docs, 64-word query | 65.94 us | 61.39 us | -6.9% |
| 100 docs, 4-word query | 515.4 us | 504.7 us | -2.1% |
| 100 docs, 16-word query | 590.6 us | 553.2 us | -6.3% |
| 100 docs, 64-word query | 597.6 us | 562.4 us | -5.9% |
| 1000 docs, 4-word query | 5.184 ms | 5.133 ms | -1.0% |
| 1000 docs, 16-word query | 5.926 ms | 5.639 ms | -4.8% |
| 1000 docs, 64-word query | 5.975 ms | 5.631 ms | -5.8% |
The reverse-order confirmation run reproduced the 64-word column at
-5.5%, -4.4% and -5.9%.
- Not tested: real proxy or agent workloads — these are synthetic
documents from a fixed 16-word vocabulary, and no end-to-end latency
claim is made. Only x86-free Apple-silicon measurements. CJK content
takes a separate scorer in `text_crusher` and is unaffected.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; no flag or channel gates this.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Scores are bit-identical, and
rankings, reasons and matched-term ordering are unchanged.
- Kill switch / disable path: none needed; there is no behavioral switch
to disable.
- Unsafe override required: none.
- Qualification impact: none. Scoring output is unchanged, so no
re-qualification of compression decisions is required.
- Rollback path: revert the commit. `bm25_score` is a pure function over
its inputs with no persisted state, so nothing needs migrating in either
direction.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
- The comment above the loop was rewritten rather than deleted: it
explains why terms are visited alphabetically at all, which is still the
load-bearing reason the ordering is not an implementation detail.
- Documentation: no user-facing docs describe this internal scoring
path; the behavioral contract lives in that comment and in the new
ordering test.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -210,3 +210,7 @@ harness = false
|
||||
[[bench]]
|
||||
name = "auth_mode"
|
||||
harness = false
|
||||
|
||||
[[bench]]
|
||||
name = "bm25_scoring"
|
||||
harness = false
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
//! Criterion benchmark for `BM25Scorer::score_batch`.
|
||||
//!
|
||||
//! Query preparation used to happen once per document: every `bm25_score`
|
||||
//! call collected and sorted the same query keys. Cost therefore scales with
|
||||
//! documents x query length, so both are varied here.
|
||||
|
||||
use std::hint::black_box;
|
||||
|
||||
use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion, Throughput};
|
||||
use headroom_core::relevance::{BM25Scorer, RelevanceScorer};
|
||||
|
||||
const VOCAB: [&str; 16] = [
|
||||
"handler",
|
||||
"parse",
|
||||
"timeout",
|
||||
"cache",
|
||||
"retry",
|
||||
"index",
|
||||
"error",
|
||||
"550e8400-e29b-41d4-a716-446655440000",
|
||||
"9f8e7d6c-1234-4321-abcd-0123456789ab",
|
||||
"12345",
|
||||
"987654",
|
||||
"resolve",
|
||||
"connection",
|
||||
"serialize",
|
||||
"upstream",
|
||||
"budget",
|
||||
];
|
||||
|
||||
fn text(seed: usize, words: usize) -> String {
|
||||
let mut parts: Vec<&str> = Vec::with_capacity(words);
|
||||
for i in 0..words {
|
||||
parts.push(VOCAB[(seed * 7 + i * 3) % VOCAB.len()]);
|
||||
}
|
||||
parts.join(" ")
|
||||
}
|
||||
|
||||
fn bench_score_batch(c: &mut Criterion) {
|
||||
let scorer = BM25Scorer::default();
|
||||
let mut group = c.benchmark_group("bm25/score_batch");
|
||||
|
||||
for docs in [10usize, 100, 1000] {
|
||||
for query_words in [4usize, 16, 64] {
|
||||
let items: Vec<String> = (0..docs).map(|d| text(d, 60)).collect();
|
||||
let refs: Vec<&str> = items.iter().map(|s| s.as_str()).collect();
|
||||
let context = text(9_999, query_words);
|
||||
|
||||
group.throughput(Throughput::Elements(docs as u64));
|
||||
group.bench_with_input(
|
||||
BenchmarkId::from_parameter(format!("{docs}d_q{query_words}w")),
|
||||
&(refs, context),
|
||||
|b, (refs, context)| {
|
||||
b.iter(|| black_box(scorer.score_batch(black_box(refs), black_box(context))))
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
group.finish();
|
||||
}
|
||||
|
||||
criterion_group!(benches, bench_score_batch);
|
||||
criterion_main!(benches);
|
||||
@@ -0,0 +1,188 @@
|
||||
//! Dump BM25 scores, rankings and matched terms so the query-preparation
|
||||
//! change can be diffed byte-for-byte against its baseline.
|
||||
//!
|
||||
//! Scores print as raw IEEE-754 bits, not decimals: the change alters when
|
||||
//! query terms are ordered, not the order itself, so accumulation must stay
|
||||
//! bit-identical rather than merely close.
|
||||
//!
|
||||
//! `cargo run --release --example bm25_query_prep_parity > out.txt` on each
|
||||
//! side, then `diff`.
|
||||
|
||||
use headroom_core::relevance::{BM25Scorer, RelevanceScorer};
|
||||
|
||||
/// xorshift64*, so both sides generate the identical corpus without
|
||||
/// depending on a rand version.
|
||||
struct Rng(u64);
|
||||
|
||||
impl Rng {
|
||||
fn next(&mut self) -> u64 {
|
||||
let mut x = self.0;
|
||||
x ^= x >> 12;
|
||||
x ^= x << 25;
|
||||
x ^= x >> 27;
|
||||
self.0 = x;
|
||||
x.wrapping_mul(0x2545_F491_4F6C_DD1D)
|
||||
}
|
||||
|
||||
fn below(&mut self, n: u64) -> u64 {
|
||||
if n == 0 {
|
||||
0
|
||||
} else {
|
||||
self.next() % n
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const VOCAB: [&str; 16] = [
|
||||
"handler",
|
||||
"parse",
|
||||
"timeout",
|
||||
"cache",
|
||||
"retry",
|
||||
"index",
|
||||
"error",
|
||||
"550e8400-e29b-41d4-a716-446655440000",
|
||||
"9f8e7d6c-1234-4321-abcd-0123456789ab",
|
||||
"12345",
|
||||
"987654",
|
||||
"a",
|
||||
"zz",
|
||||
"Mixed_Case_Token",
|
||||
"UPPER",
|
||||
"repeated",
|
||||
];
|
||||
|
||||
fn text(rng: &mut Rng, words: usize) -> String {
|
||||
let mut parts: Vec<&str> = Vec::with_capacity(words);
|
||||
for _ in 0..words {
|
||||
parts.push(VOCAB[rng.below(VOCAB.len() as u64) as usize]);
|
||||
}
|
||||
parts.join(" ")
|
||||
}
|
||||
|
||||
fn dump(name: &str, scorer: &BM25Scorer, items: &[String], context: &str) {
|
||||
let refs: Vec<&str> = items.iter().map(|s| s.as_str()).collect();
|
||||
let batch = scorer.score_batch(&refs, context);
|
||||
|
||||
println!(
|
||||
"===== {name} (n={}, context={:?}) =====",
|
||||
items.len(),
|
||||
context
|
||||
);
|
||||
for (i, s) in batch.iter().enumerate() {
|
||||
// Raw bits: a reordered summation would change these even when the
|
||||
// rounded decimal looks identical.
|
||||
println!(
|
||||
" batch[{i}] bits={:016x} reason={:?} matched={:?}",
|
||||
s.score.to_bits(),
|
||||
s.reason,
|
||||
s.matched_terms
|
||||
);
|
||||
}
|
||||
|
||||
// Ranking is what callers actually consume, so compare it explicitly.
|
||||
let mut ranked: Vec<usize> = (0..batch.len()).collect();
|
||||
ranked.sort_by(|a, b| {
|
||||
batch[*b]
|
||||
.score
|
||||
.partial_cmp(&batch[*a].score)
|
||||
.unwrap_or(std::cmp::Ordering::Equal)
|
||||
.then_with(|| a.cmp(b))
|
||||
});
|
||||
println!(" ranking={ranked:?}");
|
||||
|
||||
// The single-item path shares bm25_score, so verify it agrees with the
|
||||
// batch path item for item (reasons and caps differ by design).
|
||||
for (i, item) in items.iter().enumerate().take(20) {
|
||||
let single = scorer.score(item, context);
|
||||
println!(
|
||||
" single[{i}] bits={:016x} reason={:?} matched={:?}",
|
||||
single.score.to_bits(),
|
||||
single.reason,
|
||||
single.matched_terms
|
||||
);
|
||||
}
|
||||
println!();
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let scorer = BM25Scorer::default();
|
||||
let unnormalized = BM25Scorer::new(1.5, 0.75, false, 10.0);
|
||||
let tuned = BM25Scorer::new(0.5, 0.0, true, 3.0);
|
||||
|
||||
// Degenerate shapes first.
|
||||
dump("empty-batch", &scorer, &[], "error handler");
|
||||
dump("empty-context", &scorer, &["error here".into()], "");
|
||||
dump(
|
||||
"empty-docs",
|
||||
&scorer,
|
||||
&[String::new(), String::new()],
|
||||
"error",
|
||||
);
|
||||
dump("single-doc", &scorer, &["error handler".into()], "error");
|
||||
dump(
|
||||
"no-overlap",
|
||||
&scorer,
|
||||
&["aaa bbb ccc".into(), "ddd eee".into()],
|
||||
"zzz yyy",
|
||||
);
|
||||
|
||||
// Repeated query terms: query frequency > 1 multiplies each term score,
|
||||
// so a dropped or double-counted key shows up immediately.
|
||||
dump(
|
||||
"repeated-query-terms",
|
||||
&scorer,
|
||||
&[
|
||||
"error error handler".into(),
|
||||
"handler parse".into(),
|
||||
"error".into(),
|
||||
],
|
||||
"error error error handler",
|
||||
);
|
||||
|
||||
// Exact ties: identical documents must keep identical scores and a
|
||||
// stable ranking.
|
||||
dump(
|
||||
"exact-ties",
|
||||
&scorer,
|
||||
&vec!["error handler parse".to_string(); 6],
|
||||
"error parse",
|
||||
);
|
||||
|
||||
// Long tokens trip the >=8 char bonus in finalize_score.
|
||||
dump(
|
||||
"long-token-bonus",
|
||||
&scorer,
|
||||
&[
|
||||
"550e8400-e29b-41d4-a716-446655440000 tail".into(),
|
||||
"short a b".into(),
|
||||
],
|
||||
"550e8400-e29b-41d4-a716-446655440000 short",
|
||||
);
|
||||
|
||||
// Vary document count and query length across configs.
|
||||
for (label, s) in [
|
||||
("default", &scorer),
|
||||
("unnormalized", &unnormalized),
|
||||
("tuned", &tuned),
|
||||
] {
|
||||
for docs in [1usize, 2, 17, 64, 250] {
|
||||
for query_words in [1usize, 4, 32] {
|
||||
let mut rng = Rng(0x9E37_79B9_7F4A_7C15 ^ (docs as u64) << 8 ^ query_words as u64);
|
||||
let items: Vec<String> = (0..docs)
|
||||
.map(|_| {
|
||||
let words = 1 + rng.below(40) as usize;
|
||||
text(&mut rng, words)
|
||||
})
|
||||
.collect();
|
||||
let context = text(&mut rng, query_words);
|
||||
dump(
|
||||
&format!("{label}-d{docs}-q{query_words}"),
|
||||
s,
|
||||
&items,
|
||||
&context,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -13,7 +13,7 @@
|
||||
//! high-signal matches). Final score is clamped to `[0, 1]` via
|
||||
//! `RelevanceScore::new`.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::sync::LazyLock;
|
||||
|
||||
use regex::Regex;
|
||||
@@ -87,7 +87,7 @@ impl BM25Scorer {
|
||||
fn bm25_score(
|
||||
&self,
|
||||
doc_tokens: &[String],
|
||||
query_freq: &HashMap<String, usize>,
|
||||
query_freq: &BTreeMap<String, usize>,
|
||||
avg_doc_len: f64,
|
||||
) -> (f64, Vec<String>) {
|
||||
if doc_tokens.is_empty() || query_freq.is_empty() {
|
||||
@@ -113,16 +113,13 @@ impl BM25Scorer {
|
||||
let mut matched: Vec<String> = Vec::new();
|
||||
let idf = 2.0_f64.ln();
|
||||
|
||||
// Iterate query_freq in HashMap order — Python iterates dict
|
||||
// order (insertion order in 3.7+). For matched_terms we only
|
||||
// care about MEMBERSHIP not ordering downstream, but we sort
|
||||
// tokens alphabetically here for deterministic test output
|
||||
// when multiple terms match.
|
||||
let mut keys: Vec<&String> = query_freq.keys().collect();
|
||||
keys.sort();
|
||||
|
||||
for term in keys {
|
||||
let qf = query_freq[term];
|
||||
// Python iterates dict order (insertion order in 3.7+). For
|
||||
// matched_terms we only care about MEMBERSHIP not ordering
|
||||
// downstream, but terms are visited alphabetically here for
|
||||
// deterministic test output when multiple terms match. A
|
||||
// `BTreeMap` already iterates in key order, so batch scoring no
|
||||
// longer re-sorts the same query keys once per document.
|
||||
for (term, qf) in query_freq {
|
||||
let Some(&f) = doc_freq.get(term.as_str()) else {
|
||||
continue;
|
||||
};
|
||||
@@ -132,7 +129,7 @@ impl BM25Scorer {
|
||||
let numerator = f * (self.k1 + 1.0);
|
||||
let denominator = f + self.k1 * (1.0 - self.b + self.b * doc_len / avgdl);
|
||||
let term_score = idf * numerator / denominator;
|
||||
score += term_score * qf as f64;
|
||||
score += term_score * *qf as f64;
|
||||
}
|
||||
|
||||
(score, matched)
|
||||
@@ -158,7 +155,7 @@ impl RelevanceScorer for BM25Scorer {
|
||||
let context_tokens = self.tokenize(context);
|
||||
|
||||
// Build query frequency map from context.
|
||||
let mut query_freq: HashMap<String, usize> = HashMap::new();
|
||||
let mut query_freq: BTreeMap<String, usize> = BTreeMap::new();
|
||||
for t in &context_tokens {
|
||||
*query_freq.entry(t.clone()).or_insert(0) += 1;
|
||||
}
|
||||
@@ -197,7 +194,7 @@ impl RelevanceScorer for BM25Scorer {
|
||||
.collect();
|
||||
}
|
||||
|
||||
let mut query_freq: HashMap<String, usize> = HashMap::new();
|
||||
let mut query_freq: BTreeMap<String, usize> = BTreeMap::new();
|
||||
for t in &context_tokens {
|
||||
*query_freq.entry(t.clone()).or_insert(0) += 1;
|
||||
}
|
||||
@@ -351,6 +348,27 @@ mod tests {
|
||||
assert_eq!(scores.len(), 50);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn matched_terms_are_lexicographic_regardless_of_context_order() {
|
||||
// Query preparation is what makes matched_terms deterministic. The
|
||||
// context here is deliberately in reverse lexicographic order, so a
|
||||
// map that iterates in insertion or hash order would fail this.
|
||||
let doc = "zulu yankee mike delta alpha";
|
||||
let context = "zulu yankee mike delta alpha";
|
||||
let expected = ["alpha", "delta", "mike", "yankee", "zulu"];
|
||||
|
||||
let single = scorer().score(doc, context);
|
||||
assert_eq!(single.matched_terms, expected);
|
||||
|
||||
// Every document in a batch must get the same ordering, not just
|
||||
// the first one: preparation is shared across the whole batch.
|
||||
let items = vec![doc; 4];
|
||||
for score in scorer().score_batch(&items, context) {
|
||||
// The batch path caps matched_terms at 5.
|
||||
assert_eq!(score.matched_terms, expected);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- BM25 formula ----------
|
||||
|
||||
#[test]
|
||||
|
||||
Reference in New Issue
Block a user