fix(relevance): cap embedding batch size so ORT's arena stays small (#4507)

## Description

The embedding relevance scorer calls fastembed's `embed()` at its
default batch of 256. fastembed pads each batch to its longest text. Its
ONNX session keeps ONNX Runtime's CPU arena on, and the arena is never
returned to the OS. A relevance split over a large tool output therefore
sizes the arena to the biggest batch it has ever scored, and the memory
stays pinned for the life of the proxy.

Found on an Extra Headroom user's macOS proxy (wheel 0.41.0, 4 days
uptime): phys_footprint was 3.94 GB, and 2.5 GB of it was arena regions
in the BFC doubling pattern (32, 64, 128 and 256 MB, then 1 GB and 1
GB), all dirty and swapped out. The tracked caches (`/debug/memory`)
totalled 10 MB. The periodic `malloc_zone_pressure_relief` trim cannot
release arena memory.

This PR caps both scorers at 16 texts per ONNX run: Python
`EmbeddingScorer._encode` and the Rust `EmbeddingScorer::score_batch`.
That retains about 10x less memory, runs faster because each batch
carries less padding, and leaves keep/drop decisions unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/relevance/embedding.py`: new `EMBED_BATCH_SIZE = 16`, passed
as `batch_size` to `model.embed()`.
- `crates/headroom-core/src/relevance/embedding.rs`: `score_batch`
passes the scorer's `batch_size` instead of `None`. It is chosen once in
`try_new_with_model`: `Some(EMBED_BATCH_SIZE)` (16), or `None` (one
batch, as before) for models fastembed marks `QuantizationMode::Dynamic`
(e.g. `AllMiniLML6V2Q`), whose `embed` rejects a batch size smaller than
the input because each run calibrates its own int8 range.
- `tests/test_relevance_extra.py`: asserts that `_encode` passes the
cap. The test fails without the fix (`assert [{}] == [{'batch_size':
16}]`).
- Real-model tests in both languages
(`fastembed_batches_past_the_cap_keep_each_score_with_its_item` and
`test_batches_past_the_cap_keep_each_score_with_its_item`): 40 texts of
mixed lengths span three batches with the one related item in the last
batch. That item must still rank first, and every batched score must
match the item scored on its own.
- Rust dynamic-quantized coverage:
`dynamic_quantized_models_embed_in_one_batch` (runs in CI, no model)
pins the per-model choice;
`fastembed_dynamic_quantized_model_scores_on_both_sides_of_the_cap`
(real `AllMiniLML6V2Q`) scores 15, 16 and 40 items plus the context and
requires no inference-error fallback and the related item ranked first.
It fails on 92707844 at 16 items with fastembed's "Dynamic quantization
cannot be used with batching" error.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_relevance_extra.py tests/test_relevance.py tests/test_relevance_split.py -q
63 passed, 4 skipped in 1.75s
$ uvx ruff check headroom/relevance/embedding.py tests/test_relevance_extra.py
All checks passed!
$ uvx ruff format --check headroom/relevance/embedding.py tests/test_relevance_extra.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/relevance/embedding.py
Success: no issues found in 1 source file
$ cargo fmt --all -- --check   (clean)
$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.27 dylib> cargo test -p headroom-core --lib relevance::embedding -- --test-threads=1
test result: ok. 17 passed; 0 failed   (real models, incl. the 40-text multi-batch test and AllMiniLML6V2Q at 15/16/40 items)
$ cargo test -p headroom-core --lib relevance::embedding
test result: ok. 17 passed; 0 failed   (model tests skip without RUN_FASTEMBED_TESTS)
$ cargo test -p headroom-core --no-default-features --lib relevance::embedding
test result: ok. 8 passed; 0 failed
$ cargo clippy -p headroom-core --all-targets -- -D warnings
Finished (no warnings)
$ FASTEMBED_CACHE_PATH=<cache> uv run --frozen --extra dev --with fastembed==0.9.1 pytest tests/test_relevance.py -k TestEmbeddingScorer
4 passed   (real model, incl. the 40-text multi-batch test)
$ uv run --frozen --extra dev pytest tests/test_*content_router*.py tests/test_relevance*.py -q
236 passed, 7 skipped
```

## Real Behavior Proof

- Environment: macOS 27 on an M2 (16 GB), the installed headroom-ai
0.41.0 runtime (fastembed 0.8.0, onnxruntime 1.27),
BAAI/bge-small-en-v1.5 from the local cache; for the end-to-end run, a
scratch proxy started with Extra Headroom's flags (`--no-http2
--no-rate-limit --log-messages`, isolated HOME, telemetry off) in front
of a local mock Anthropic upstream.
- Exact command / steps: (1) bench: 120 records cut from a 7k-line Rust
source file by the real `segment()`, `EmbeddingScorer.score_batch` 3
times per batch size in a fresh process each, measured with `footprint`;
(2) equivalence: `plan_relevance_split` with `create_scorer("hybrid")`
over 24 cases (8 real corpora of 5 to 411 records: Rust source, proxy
log, grep output, git log, transcript JSONL, long-line markdown, cargo
output, a tiny log; 3 queries each), batch 16 and 256 in separate
processes; (3) end to end: 13 Claude Code shaped requests (question,
Bash `tool_use`, a 2k to 145k token `tool_result`) through the scratch
proxy, batch 16 and 256.
- Observed result: identical splits in 24 of 24 cases at 1.7x less time;
end to end, one relevance split grew the batch-256 proxy by 10.4 GB (to
an 11 GB footprint) against 0.99 GB at batch 16, with 12 of 13 forwarded
bodies byte-identical.

  | Check | Batch 256 (before) | Batch 16 (this PR) |
  |---|---|---|
| Bench, 120 records: time / retained | 16.8 s / +3769 MB | 8.7 s / +339
MB (max score diff 7.8e-5, 0 flips) |
| Equivalence, 24 cases: total split time | 1295 s | 746 s (24/24
identical runs, worst score diff 4.8e-5) |
  | End to end: proxy footprint growth | +10.4 GB | +0.99 GB |
| End to end: 16k-token search output | hit the 30 s compression
timeout, forwarded uncompressed | compressed in 20.4 s, 5,338 tokens
saved |
- Not tested: Linux and Windows; memory and timing of the Rust scorer
(its real-model tests pass, but no proxy path loads its model in
0.41.0); turning the arena off instead, which on macOS retained between
+76 and +1277 MB across runs at batch 256 (libmalloc keeps the freed
blocks), so the batch size is the dependable lever.

Rust, dynamic-quantized model (added for review, 135c2b66). This is a
real-model cargo test (real ONNX model through fastembed-rs 5.17.3,
onnxruntime 1.27, macOS 27 on M2), not a proxy run: in the 0.41.0 wheel
the proxy builds the Rust `HybridScorer::default()`, which never loads
an embedding model, so no proxy request reaches this code.

```text
$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.27 dylib> cargo test -p headroom-core --lib relevance::embedding::tests::fastembed_dynamic_quantized -- --test-threads=1
# on 92707844 (test added, source unchanged):
panicked at crates/headroom-core/src/relevance/embedding.rs:569:17:
16 items: Embedding: inference failed: Dynamic quantization cannot be used with batching. [...]
test result: FAILED. 0 passed; 1 failed
# on 135c2b66:
test relevance::embedding::tests::fastembed_dynamic_quantized_model_scores_on_both_sides_of_the_cap ... ok
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, not rollout-managed
- Stable/default behavior changed: yes; embedding inference now runs in
batches of 16 texts instead of 256 (scores match within 1e-4, no
keep/drop changes measured). Rust models fastembed marks
dynamic-quantized (e.g. `AllMiniLML6V2Q` via `try_new_with_model`) still
run as one batch, as before.
- Kill switch / disable path: none needed for a batch-size constant;
revert the commit
- Unsafe override required: no
- Qualification impact: none expected; relevance-split decisions were
identical on the measured input
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation: N/A, since no user-facing setting changed. The arena is
left on: the batch cap bounds it, and turning it off alone did not
reliably lower retained memory on macOS. Extra Headroom will carry this
as a temporary patch against 0.41.0 until a release includes it.

Python has no equivalent constraint: fastembed 0.9.1 has no quantization
check on `embed`; `_embed_documents`
(`fastembed/text/onnx_text_model.py`) splits every model's input with
`iter_batch`, and the only batch-size check is `>= 1`
(`fastembed/common/utils.py`). Its previous default of 256 already split
every model's input the same way.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-10-28 23:45:34 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 526e06b7e0
commit a2d4fe7238
4 changed files with 136 additions and 5 deletions
@@ -30,10 +30,23 @@
use std::sync::Mutex;
#[cfg(feature = "ml")]
use fastembed::{EmbeddingModel, InitOptions, TextEmbedding};
use fastembed::{EmbeddingModel, InitOptions, QuantizationMode, TextEmbedding};
use super::base::{RelevanceScore, RelevanceScorer};
/// Texts per ONNX Runtime run, matching Python's `EMBED_BATCH_SIZE`: fastembed's
/// default of 256 grows ORT's never-shrinking CPU arena to the biggest batch scored.
#[cfg(feature = "ml")]
const EMBED_BATCH_SIZE: usize = 16;
/// fastembed refuses to split a dynamic-quantized model's input across runs (each
/// run calibrates its own int8 range, so the embeddings would not compare): one batch.
#[cfg(feature = "ml")]
fn embed_batch_size(model_kind: &EmbeddingModel) -> Option<usize> {
(TextEmbedding::get_quantization_mode(model_kind) != QuantizationMode::Dynamic)
.then_some(EMBED_BATCH_SIZE)
}
/// fastembed-backed semantic relevance scorer.
///
/// Construct via `EmbeddingScorer::try_new()` to handle the model-load
@@ -56,6 +69,8 @@ pub struct EmbeddingScorer {
/// dominant cost so contention is bounded by inference latency,
/// not lock latency.
model: Option<Mutex<TextEmbedding>>,
/// `embed` batch size for this model, from `embed_batch_size`.
batch_size: Option<usize>,
}
#[cfg(feature = "ml")]
@@ -75,6 +90,7 @@ impl Default for EmbeddingScorer {
EmbeddingScorer {
model_name: "BAAI/bge-small-en-v1.5".to_string(),
model: None,
batch_size: Some(EMBED_BATCH_SIZE),
}
}
}
@@ -129,11 +145,13 @@ impl EmbeddingScorer {
// `dynamic_ort_loader_ready`).
crate::transforms::magika_detector::dynamic_ort_loader_ready()
.map_err(|e| format!("EmbeddingScorer: ONNX Runtime unavailable: {e}"))?;
let batch_size = embed_batch_size(&model_kind);
let model = TextEmbedding::try_new(InitOptions::new(model_kind))
.map_err(|e| format!("EmbeddingScorer model load failed: {}", e))?;
Ok(EmbeddingScorer {
model_name: name,
model: Some(Mutex::new(model)),
batch_size,
})
}
}
@@ -192,11 +210,11 @@ impl RelevanceScorer for EmbeddingScorer {
}
};
// Encode items + context in one batch — saves model dispatch
// overhead. Mirrors Python fastembed batch encoding.
// Encode items + context in one `embed` call, which runs them in
// `self.batch_size` chunks. Mirrors Python fastembed batch encoding.
let mut all_texts: Vec<String> = items.iter().map(|s| s.to_string()).collect();
all_texts.push(context.to_string());
let embeddings = match guard.embed(all_texts, None) {
let embeddings = match guard.embed(all_texts, self.batch_size) {
Ok(e) => e,
Err(e) => {
return items
@@ -352,6 +370,7 @@ mod tests {
EmbeddingScorer {
model_name: "test".to_string(),
model: None,
batch_size: Some(EMBED_BATCH_SIZE),
}
}
@@ -462,6 +481,15 @@ mod tests {
}
}
#[cfg(feature = "ml")]
#[test]
fn dynamic_quantized_models_embed_in_one_batch() {
let capped = Some(EMBED_BATCH_SIZE);
assert_eq!(embed_batch_size(&EmbeddingModel::BGESmallENV15), capped);
assert_eq!(embed_batch_size(&EmbeddingModel::BGESmallENV15Q), capped);
assert_eq!(embed_batch_size(&EmbeddingModel::AllMiniLML6V2Q), None);
}
// ---------- model-backed tests (gated on RUN_FASTEMBED_TESTS) ----------
#[cfg(feature = "ml")]
@@ -506,4 +534,71 @@ mod tests {
assert!((0.0..=1.0).contains(&sc.score));
}
}
#[cfg(feature = "ml")]
#[test]
fn fastembed_batches_past_the_cap_keep_each_score_with_its_item() {
if !fastembed_enabled() {
return;
}
// Three EMBED_BATCH_SIZE batches of mixed lengths, the one related item
// in the last: a misaligned batch boundary would hand its score away.
let s = EmbeddingScorer::try_new().expect("model loads");
let mut items: Vec<String> = (0..40)
.map(|i| {
format!(
"inventory row {i}: {}",
"pallets of garden furniture ".repeat(i % 5 + 1)
)
})
.collect();
items[37] = "authentication failed for user admin: invalid password".to_string();
let refs: Vec<&str> = items.iter().map(String::as_str).collect();
let batch = s.score_batch(&refs, "login error");
assert_eq!(batch.len(), 40);
let top = (0..40)
.max_by(|&a, &b| batch[a].score.total_cmp(&batch[b].score))
.unwrap();
assert_eq!(top, 37);
for (item, sc) in refs.iter().zip(&batch) {
let single = s.score(item, "login error").score;
assert!(
(sc.score - single).abs() < 1e-3,
"{item}: batched {} vs single {single}",
sc.score
);
}
}
#[cfg(feature = "ml")]
#[test]
fn fastembed_dynamic_quantized_model_scores_on_both_sides_of_the_cap() {
if !fastembed_enabled() {
return;
}
// 15 items plus the context fill exactly one EMBED_BATCH_SIZE run; 16
// and 40 go past it, where fastembed refuses to split this model.
let s = EmbeddingScorer::try_new_with_model(EmbeddingModel::AllMiniLML6V2Q)
.expect("model loads");
for n in [15, 16, 40] {
let mut items: Vec<String> = (0..n)
.map(|i| format!("inventory row {i}: pallets of garden furniture"))
.collect();
items[n - 3] = "authentication failed for user admin: invalid password".to_string();
let refs: Vec<&str> = items.iter().map(String::as_str).collect();
let batch = s.score_batch(&refs, "login error");
assert_eq!(batch.len(), n);
for sc in &batch {
assert!(
!sc.reason.contains("inference failed"),
"{n} items: {}",
sc.reason
);
}
let top = (0..n)
.max_by(|&a, &b| batch[a].score.total_cmp(&batch[b].score))
.unwrap();
assert_eq!(top, n - 3, "{n} items");
}
}
}
+7 -1
View File
@@ -37,6 +37,12 @@ from headroom.onnx_runtime import _resolve_revision, hf_hub_download_local_first
from .base import RelevanceScore, RelevanceScorer
# Texts per ONNX Runtime run. fastembed pads each batch to its longest text, and
# its default batch of 256 grows ORT's CPU arena (which never shrinks) to fit
# the biggest batch a proxy ever scores: 2.5 GB retained in a long-lived macOS
# proxy. 16 holds ~10x less and runs faster, with less padding per batch.
EMBED_BATCH_SIZE = 16
# numpy is an optional dependency - import lazily
_numpy = None
@@ -329,7 +335,7 @@ class EmbeddingScorer(RelevanceScorer):
"""
np = _get_numpy()
model = self._get_model()
embeddings = list(model.embed(texts))
embeddings = list(model.embed(texts, batch_size=EMBED_BATCH_SIZE))
return np.array(embeddings)
def score(self, item: str, context: str) -> RelevanceScore:
+14
View File
@@ -186,6 +186,20 @@ class TestEmbeddingScorer:
# "failed"/"error" semantically relates to "errors"
assert scores[0].score > scores[1].score
def test_batches_past_the_cap_keep_each_score_with_its_item(self, scorer):
"""Three EMBED_BATCH_SIZE batches of mixed lengths, the related item in
the last: a misaligned batch boundary would hand its score away."""
items = [
f"inventory row {i}: " + "pallets of garden furniture " * (i % 5 + 1) for i in range(40)
]
items[37] = "authentication failed for user admin: invalid password"
scores = [s.score for s in scorer.score_batch(items, "login error")]
assert len(scores) == 40
assert max(range(40), key=scores.__getitem__) == 37
for item, batched in zip(items, scores):
assert batched == pytest.approx(scorer.score(item, "login error").score, abs=1e-3)
def test_paraphrase_match(self, scorer):
"""Embeddings match paraphrases."""
items = [
+16
View File
@@ -124,6 +124,22 @@ def test_embedding_score_and_batch_with_fake_model(monkeypatch) -> None:
assert [item.reason for item in batch] == ["Embedding: 0.75", "Embedding: 0.25"]
def test_embedding_encode_caps_the_onnx_batch(monkeypatch) -> None:
# The default fastembed batch (256) sizes ORT's never-shrinking arena to the
# largest relevance split ever scored; every encode must pass the cap.
calls = []
class FakeModel:
def embed(self, texts, **kwargs):
calls.append(kwargs)
return [[1.0, 0.0] for _ in texts]
scorer = EmbeddingScorer()
monkeypatch.setattr(scorer, "_get_model", lambda: FakeModel())
assert len(scorer._encode([f"record {i}" for i in range(300)])) == 300
assert calls == [{"batch_size": embedding.EMBED_BATCH_SIZE}]
def test_hybrid_constructor_alpha_variants_and_single_score_paths(monkeypatch) -> None:
bm25_result = RelevanceScore(score=0.1, reason="bm25", matched_terms=["term"])
emb_result = RelevanceScore(score=0.9, reason="emb", matched_terms=[])