mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(smart-crusher): keep the row the question names (#4713)
## Description Ask "Who opened issue #4475?" over a 100-issue `gh issue list` result, and SmartCrusher cuts row #4475. `prioritize_indices` fills the budget with errors, outliers and first/last rows, then fills by position, so rows matching a query anchor (ID, UUID, email, hostname, quoted string) come last. `plan_top_n` already keeps anchor matches unconditionally, and the docs list them as preserved. This makes `smart_sample`, `cluster_sample` and `time_series` do the same. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `apply_query_signals` returns the anchor matches; `prioritize_indices` keeps them with the critical set. - If an anchor matches more rows than the whole budget, it isn't selective, so those rows compete as before. Like errors, matches can push the result over `max_items`, by at most `max_items` rows. - BM25 matches are unchanged. Anchor matching is the existing `item_matches_anchors`, unchanged here. - The `planning.rs` call sites only pass the new set; the extra lines are rustfmt. ## Testing - [x] New tests: errors fill the budget, no error rows, anchor matches more rows than the budget, end to end through `crush_array`. Each fails without the fix; `cargo mutants --in-diff`: 10/10 caught. - [x] `cargo test --workspace`, `cargo fmt --check`, `cargo clippy --workspace -D warnings` (1.95.0), `make test-parity` (smart_crusher 17/17). Fork CI: green. ### Test Output ``` $ cargo test -p headroom-core --lib smart_crusher test transforms::smart_crusher::orchestration::tests::prioritize_keeps_query_match_when_errors_fill_budget ... ok test transforms::smart_crusher::orchestration::tests::prioritize_keeps_query_match_without_error_rows ... ok test transforms::smart_crusher::orchestration::tests::prioritize_ignores_anchor_matching_more_than_budget ... ok test transforms::smart_crusher::crusher::tests::query_named_row_survives_lossy_crush ... ok test result: ok. 354 passed; 0 failed; 0 ignored; 0 measured; 689 filtered out ``` ## Real Behavior Proof - Environment: macOS arm64, `compress()` from the README (0.43.1 vs this branch built from source), gpt-oss-20b local, temperature 0. - Exact command / steps: 12 random issues, "Who opened issue #N, and is it open or closed?" Scripts and answers: https://github.com/cherylaurelia/headroom-demo/tree/main/bench - Observed result: 0.43.1 kept 2 of the 12 asked rows (2/12 correct); this branch keeps 12/12 (10/12 correct; the 2 misses had the row in context). About 90 more prompt tokens. - Not tested: rows named only by title words can still be cut. Through `headroom proxy`, Claude Code also needs `fix/toolsearch-query-note` (its relevance query is "Tool loaded."): this fix alone 2/12, both 12/12. ## Runtime Rollout Safety - Rollout-managed feature(s): n/a - Minimum rollout channel: n/a - Stable/default behavior changed: yes, one more kept row when the question names it - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I did **not** edit `CHANGELOG.md` Written with Claude Code, which also ran the tests and measurements above. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4713" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -2206,6 +2206,35 @@ mod tests {
|
||||
assert_eq!(recovered, Value::Array(rows));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn query_named_row_survives_lossy_crush() {
|
||||
// An issue list where every 4th title is a bug report: the error
|
||||
// rows alone fill the lossy budget, and the row the question names
|
||||
// by number (index 81, past the first/last anchors) must be kept
|
||||
// rather than cut by position.
|
||||
let c = SmartCrusher::without_compaction(SmartCrusherConfig::default());
|
||||
let items: Vec<Value> = (0..100)
|
||||
.map(|i| {
|
||||
let title = if i % 4 == 0 {
|
||||
format!("[BUG] request failed in module {}", i % 7)
|
||||
} else {
|
||||
format!("Add option for module {}", i % 7)
|
||||
};
|
||||
json!({
|
||||
"number": if i == 81 { 4475 } else { 4300 + 2 * i },
|
||||
"title": title,
|
||||
"state": if i % 3 == 0 { "OPEN" } else { "CLOSED" },
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
let result = c.crush_array(&items, "Who opened issue #4475?", 1.0);
|
||||
assert!(
|
||||
result.items.iter().any(|it| it["number"] == json!(4475)),
|
||||
"issue 4475 was dropped: {}",
|
||||
result.strategy_info
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn crusher_with_custom_scorer() {
|
||||
use crate::relevance::BM25Scorer;
|
||||
|
||||
@@ -143,7 +143,10 @@ pub fn fill_remaining_slots(
|
||||
/// 3. **Already under budget?** Return as-is.
|
||||
/// 4. **Otherwise**: keep ALL critical items (errors + structural
|
||||
/// outliers + numeric anomalies — non-negotiable per Python's
|
||||
/// "quality guarantee"). Then add first-3 + last-2 if room. Then
|
||||
/// "quality guarantee") and the items that exactly match a query
|
||||
/// anchor (`query_matches`: an ID, UUID, email, hostname or quoted
|
||||
/// string from the user's question), unless the anchors match more
|
||||
/// than `effective_max` items. Then add first-3 + last-2 if room. Then
|
||||
/// fill remaining with non-critical kept indices in ascending order.
|
||||
///
|
||||
/// May return MORE than `effective_max` items when critical items
|
||||
@@ -156,6 +159,7 @@ pub fn prioritize_indices(
|
||||
n: usize,
|
||||
analysis: Option<&ArrayAnalysis>,
|
||||
effective_max: usize,
|
||||
query_matches: &BTreeSet<usize>,
|
||||
) -> BTreeSet<usize> {
|
||||
// Dedup pass.
|
||||
let mut current = if config.dedup_identical_items {
|
||||
@@ -195,6 +199,15 @@ pub fn prioritize_indices(
|
||||
prioritized.extend(&anomaly_indices);
|
||||
prioritized.extend(&learned_indices);
|
||||
|
||||
// The rows the question names, kept unconditionally as `plan_top_n`
|
||||
// already does. An anchor that matches more rows than the whole budget
|
||||
// is not selective (a shared hostname, a common number) and pinning all
|
||||
// of them would defeat the crush, so those rows compete as ordinary
|
||||
// kept indices instead.
|
||||
if query_matches.len() <= effective_max {
|
||||
prioritized.extend(query_matches.intersection(¤t));
|
||||
}
|
||||
|
||||
// First 3 / last 2 anchors if we have room.
|
||||
let mut remaining = effective_max.saturating_sub(prioritized.len());
|
||||
if remaining > 0 {
|
||||
@@ -411,11 +424,23 @@ mod tests {
|
||||
|
||||
// ---------- prioritize_indices ----------
|
||||
|
||||
fn prioritize(kept: &BTreeSet<usize>, items: &[Value], max: usize) -> BTreeSet<usize> {
|
||||
prioritize_indices(
|
||||
&cfg(),
|
||||
kept,
|
||||
items,
|
||||
items.len(),
|
||||
None,
|
||||
max,
|
||||
&BTreeSet::new(),
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prioritize_under_budget_passthrough_after_dedup() {
|
||||
let items: Vec<Value> = (0..5).map(|i| json!({"id": i})).collect();
|
||||
let kept = idx_set(&[0, 1, 2]);
|
||||
let result = prioritize_indices(&cfg(), &kept, &items, items.len(), None, 10);
|
||||
let result = prioritize(&kept, &items, 10);
|
||||
// 3 items < max 10 → fill kicks in; we get 5 (all items).
|
||||
assert_eq!(result.len(), 5);
|
||||
}
|
||||
@@ -428,7 +453,7 @@ mod tests {
|
||||
json!({"name": "bob"}),
|
||||
];
|
||||
let kept = idx_set(&[0, 1, 2]);
|
||||
let result = prioritize_indices(&cfg(), &kept, &items, items.len(), None, 10);
|
||||
let result = prioritize(&kept, &items, 10);
|
||||
// Dedup collapses 0+1 to 0; fill stays put because n=3 already covered.
|
||||
assert_eq!(result, idx_set(&[0, 2]));
|
||||
}
|
||||
@@ -441,19 +466,77 @@ mod tests {
|
||||
.collect();
|
||||
items.push(json!({"id": 30, "msg": "FATAL: out of memory"}));
|
||||
let kept: BTreeSet<usize> = (0..items.len()).collect();
|
||||
let result = prioritize_indices(&cfg(), &kept, &items, items.len(), None, 10);
|
||||
let result = prioritize(&kept, &items, 10);
|
||||
assert!(
|
||||
result.contains(&30),
|
||||
"error item must survive prioritization"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prioritize_keeps_query_match_when_errors_fill_budget() {
|
||||
// 39 error-keyword rows fill the budget of 10; the row the question
|
||||
// names (index 25) has no error keyword and must still survive.
|
||||
let items: Vec<Value> = (0..40)
|
||||
.map(|i| {
|
||||
let outcome = if i == 25 { "ok" } else { "failed" };
|
||||
json!({"number": 4400 + i, "title": format!("parser {outcome} on case {i}")})
|
||||
})
|
||||
.collect();
|
||||
let kept: BTreeSet<usize> = (0..items.len()).collect();
|
||||
let result = prioritize_indices(
|
||||
&cfg(),
|
||||
&kept,
|
||||
&items,
|
||||
items.len(),
|
||||
None,
|
||||
10,
|
||||
&idx_set(&[25]),
|
||||
);
|
||||
assert!(
|
||||
result.contains(&25),
|
||||
"query-matched row must survive prioritization"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prioritize_keeps_query_match_without_error_rows() {
|
||||
// No critical rows at all: the budget fills by position, and the
|
||||
// matched row in the middle must still be kept.
|
||||
let items: Vec<Value> = (0..40)
|
||||
.map(|i| json!({"id": 9000 + i, "v": "ok"}))
|
||||
.collect();
|
||||
let kept: BTreeSet<usize> = (0..items.len()).collect();
|
||||
let result = prioritize_indices(
|
||||
&cfg(),
|
||||
&kept,
|
||||
&items,
|
||||
items.len(),
|
||||
None,
|
||||
10,
|
||||
&idx_set(&[20]),
|
||||
);
|
||||
assert!(result.contains(&20), "got {result:?}");
|
||||
assert!(result.len() <= 10, "got {} items", result.len());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prioritize_ignores_anchor_matching_more_than_budget() {
|
||||
// An anchor that matches 20 rows against a budget of 10 isn't
|
||||
// selective: it must not force all 20 into the result.
|
||||
let items: Vec<Value> = (0..30).map(|i| json!({"id": i, "v": i})).collect();
|
||||
let kept: BTreeSet<usize> = (0..items.len()).collect();
|
||||
let matches: BTreeSet<usize> = (5..25).collect();
|
||||
let result = prioritize_indices(&cfg(), &kept, &items, items.len(), None, 10, &matches);
|
||||
assert!(result.len() <= 10, "got {} items", result.len());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prioritize_includes_first_3_and_last_2_when_room() {
|
||||
// No errors / outliers / anomalies → first 3 + last 2 anchors fill.
|
||||
let items: Vec<Value> = (0..30).map(|i| json!({"id": i, "v": i})).collect();
|
||||
let kept: BTreeSet<usize> = (5..15).collect();
|
||||
let result = prioritize_indices(&cfg(), &kept, &items, items.len(), None, 10);
|
||||
let result = prioritize(&kept, &items, 10);
|
||||
// With no critical items and budget 10, dedup is a no-op (all
|
||||
// distinct) and fill keeps us at <= 10. We should see at least
|
||||
// some of items 0..3 OR 28..30 covered through fill.
|
||||
|
||||
@@ -217,13 +217,21 @@ impl<'a> SmartCrusherPlanner<'a> {
|
||||
}
|
||||
|
||||
// 5/6. Query-anchor matches + relevance scores.
|
||||
self.apply_query_signals(items, query_context, item_strings, &mut keep, false);
|
||||
let query_matches =
|
||||
self.apply_query_signals(items, query_context, item_strings, &mut keep, false);
|
||||
|
||||
// TOIN preserve_fields.
|
||||
self.apply_preserve_field_matches(items, query_context, preserve_fields, &mut keep);
|
||||
|
||||
let final_keep =
|
||||
prioritize_indices(self.config, &keep, items, n, Some(analysis), max_items);
|
||||
let final_keep = prioritize_indices(
|
||||
self.config,
|
||||
&keep,
|
||||
items,
|
||||
n,
|
||||
Some(analysis),
|
||||
max_items,
|
||||
&query_matches,
|
||||
);
|
||||
plan.keep_indices = final_keep.into_iter().collect();
|
||||
plan
|
||||
}
|
||||
@@ -406,13 +414,21 @@ impl<'a> SmartCrusherPlanner<'a> {
|
||||
}
|
||||
|
||||
// 4/5. Query signals.
|
||||
self.apply_query_signals(items, query_context, item_strings, &mut keep, false);
|
||||
let query_matches =
|
||||
self.apply_query_signals(items, query_context, item_strings, &mut keep, false);
|
||||
|
||||
// TOIN preserve_fields.
|
||||
self.apply_preserve_field_matches(items, query_context, preserve_fields, &mut keep);
|
||||
|
||||
let final_keep =
|
||||
prioritize_indices(self.config, &keep, items, n, Some(analysis), max_items);
|
||||
let final_keep = prioritize_indices(
|
||||
self.config,
|
||||
&keep,
|
||||
items,
|
||||
n,
|
||||
Some(analysis),
|
||||
max_items,
|
||||
&query_matches,
|
||||
);
|
||||
plan.keep_indices = final_keep.into_iter().collect();
|
||||
plan
|
||||
}
|
||||
@@ -458,13 +474,21 @@ impl<'a> SmartCrusherPlanner<'a> {
|
||||
self.apply_constraints(items, item_strings, &mut keep);
|
||||
|
||||
// 4/5. Query signals.
|
||||
self.apply_query_signals(items, query_context, item_strings, &mut keep, false);
|
||||
let query_matches =
|
||||
self.apply_query_signals(items, query_context, item_strings, &mut keep, false);
|
||||
|
||||
// TOIN preserve_fields.
|
||||
self.apply_preserve_field_matches(items, query_context, preserve_fields, &mut keep);
|
||||
|
||||
let final_keep =
|
||||
prioritize_indices(self.config, &keep, items, n, Some(analysis), max_items);
|
||||
let final_keep = prioritize_indices(
|
||||
self.config,
|
||||
&keep,
|
||||
items,
|
||||
n,
|
||||
Some(analysis),
|
||||
max_items,
|
||||
&query_matches,
|
||||
);
|
||||
plan.keep_indices = final_keep.into_iter().collect();
|
||||
plan
|
||||
}
|
||||
@@ -474,7 +498,8 @@ impl<'a> SmartCrusherPlanner<'a> {
|
||||
/// Apply query-anchor matches (deterministic) + relevance scoring
|
||||
/// (probabilistic). When `keep_existing_only` is true (top_n's
|
||||
/// "additive only" mode), only items not already in keep are added.
|
||||
/// When false, all matches are added.
|
||||
/// When false, all matches are added. Returns the indices added by the
|
||||
/// deterministic anchor match, which `prioritize_indices` keeps.
|
||||
fn apply_query_signals(
|
||||
&self,
|
||||
items: &[Value],
|
||||
@@ -482,9 +507,10 @@ impl<'a> SmartCrusherPlanner<'a> {
|
||||
item_strings: Option<&[String]>,
|
||||
keep: &mut BTreeSet<usize>,
|
||||
keep_existing_only: bool,
|
||||
) {
|
||||
) -> BTreeSet<usize> {
|
||||
let mut anchor_matches = BTreeSet::new();
|
||||
if query_context.is_empty() {
|
||||
return;
|
||||
return anchor_matches;
|
||||
}
|
||||
|
||||
// Deterministic anchor match.
|
||||
@@ -495,6 +521,7 @@ impl<'a> SmartCrusherPlanner<'a> {
|
||||
}
|
||||
if item_matches_anchors(item, &anchors) {
|
||||
keep.insert(i);
|
||||
anchor_matches.insert(i);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -519,6 +546,7 @@ impl<'a> SmartCrusherPlanner<'a> {
|
||||
keep.insert(i);
|
||||
}
|
||||
}
|
||||
anchor_matches
|
||||
}
|
||||
|
||||
fn apply_preserve_field_matches(
|
||||
|
||||
Reference in New Issue
Block a user