Files
headroom/tests/test_relevance_extra.py
gglucassandClaude Opus 5.5 2363f83309 fix(relevance): hand the embedding arena back after each run on macOS (#4567)
## Description

Follow-up to #4507. The batch cap bounds one ONNX run, but concurrent
relevance splits share the scorer's ONNX session, and ONNX Runtime's CPU
arena keeps its high-water mark: the memory it grows to for 3 or 4
splits at once stays pinned for the life of the proxy.

Found on an Extra Headroom user's macOS proxy after #4507's batch size
was in place (wheel 0.41.0 + the cap as a patch, 13.5 h uptime):
phys_footprint 3.1 GB, of which 2.1 GB was arena regions in the BFC
doubling pattern (16, 32, 32, 64, 128, 256, 512 MB and 1 GB). The
periodic `malloc_zone_pressure_relief` trim releases none of it, and
neither does dropping the whole model.

This PR has each embedding run pass ORT's
`memory.enable_memory_arena_shrinkage` run option, so the arena hands
its free regions back after every run. It does this only on macOS under
`MallocLargeCache=0`, which the proxy CLI's malloc re-exec (#2879)
already sets. With libmalloc's large cache on, shrinking made it worse:
the freed blocks stay dirty in the cache and the arena grows again on
top of them (+3.8 GB against +2.5 GB).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/relevance/embedding.py`: new
`_shrink_arena_after_each_run(model)`, called once per model from
`EmbeddingScorer._encode`. On macOS with `MallocLargeCache=0` it wraps
the fastembed model's `InferenceSession.run` so every call gets
`RunOptions` with `memory.enable_memory_arena_shrinkage=cpu:0` (a
caller's own `run_options` still win). Elsewhere it does nothing. Any
failure (onnxruntime missing, fastembed internals moved) is logged at
debug and leaves today's behaviour.
- `tests/test_relevance_extra.py`:
`test_embedding_encode_shrinks_the_arena_only_without_the_large_cache`,
parametrized over darwin + `MallocLargeCache=0` (shrinks), darwin
without it, and linux (both unchanged). It fails without the fix
(`assert [None, None] == ['cpu:0', 'cpu:0']`).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev --extra relevance pytest tests/test_relevance_extra.py tests/test_relevance.py tests/test_relevance_split.py tests/test_*content_router*.py -q
286 passed, 2 skipped in 10.68s
$ uv run --frozen --extra dev --extra relevance pytest tests/test_relevance_extra.py -q -k shrinks_the_arena   # embedding.py at upstream/main
E   AssertionError: assert [None, None] == ['cpu:0', 'cpu:0']
1 failed, 2 passed
$ uvx ruff check headroom/relevance/embedding.py tests/test_relevance_extra.py
All checks passed!
$ uvx ruff format --check headroom/relevance/embedding.py tests/test_relevance_extra.py
2 files already formatted
$ uv run --frozen --extra dev --extra relevance --with mypy==1.20.2 mypy headroom/relevance/embedding.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 27.2 on an Apple M6, this branch's `uv` environment
(`--extra relevance`: onnxruntime 1.26.0, fastembed 0.8.0),
BAAI/bge-small-en-v1.5 from the local cache, run with
`MallocLargeCache=0 MallocAggressiveMadvise=1` (what the CLI re-exec
sets).
- Exact command / steps: a script imports
`headroom.relevance.embedding`, warms an `EmbeddingScorer`, scores 128
mixed-length records once, then twice runs 4 threads that each score the
same 128 records (4 concurrent relevance splits), waits 0.5 s and reads
`phys_footprint` with `footprint`; then times twenty 5-record calls.
"Before" swaps `_shrink_arena_after_each_run` for a no-op; each run is a
fresh process.
- Observed result: memory left after the parallel splits dropped from
+2.1 to 2.3 GB to +83 to 595 MB at the same speed, with bit-identical
embeddings (same sum over the 128-record output in every run).

  | | Before (main) | This PR |
  |---|---|---|
| Retained after 4 parallel splits, x2 | +2136, +2245, +2274 MB | +595,
+83, +591, +319, +83, +331 MB |
  | 4 parallel splits, x2 | 12.5 to 13.5 s | 12.4 to 14.0 s |
  | One 128-record split | 4.3 to 4.7 s | 4.3 to 5.2 s |
  | One 5-record call | 21 to 22 ms | 22 to 27 ms |

The same change against the 0.41.0 wheel's runtime (onnxruntime 1.27.0,
fastembed 0.9.1) settled at +80 to 85 MB in every run. Without
`MallocLargeCache=0`, shrinkage retained +3.6 to 3.9 GB against +2.5 to
2.6 GB without it, which is why it is gated.
- Not tested: Linux and Windows, so this PR leaves them unchanged; the
Rust scorer (no proxy path loads its model in 0.41.0); macOS releases
before 27.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, not rollout-managed
- Stable/default behavior changed: yes, on macOS when
`MallocLargeCache=0` is set (the default for the proxy CLI): the
embedding ONNX session now shrinks its CPU arena after each run.
Embeddings are unchanged; other platforms are unchanged.
- Kill switch / disable path: `HEADROOM_MALLOC_TUNING=0` (no re-exec, so
`MallocLargeCache` is unset and shrinkage stays off), or revert the
commit
- Unsafe override required: no
- Qualification impact: none expected; embeddings were bit-identical in
every measured run
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation: N/A, no user-facing setting changed. fastembed exposes no
run options, so the helper wraps the session's `run` on the instance
(`TextEmbedding.model.model`); if a fastembed release moves that
attribute, the wrap fails, is logged at debug, and the arena behaves as
on main. Extra Headroom carries this as a temporary patch against 0.41.0
until a release includes it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4567"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 19:26:42 -06:00

238 lines
8.7 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
from __future__ import annotations
import builtins
import sys
from dataclasses import dataclass
from types import SimpleNamespace
import pytest
import headroom.relevance as relevance_mod
from headroom.relevance import (
BM25Scorer,
EmbeddingScorer,
HybridScorer,
create_scorer,
embedding,
hybrid,
)
from headroom.relevance.base import RelevanceScore, RelevanceScorer, default_batch_score
@dataclass
class DummyRelevanceScorer(RelevanceScorer):
def score(self, item: str, context: str) -> RelevanceScore:
return RelevanceScore(score=0.4, reason=f"{item}:{context}")
def score_batch(self, items: list[str], context: str) -> list[RelevanceScore]:
return [RelevanceScore(score=0.2, reason=context) for _ in items]
def test_base_default_batch_and_abstract_methods() -> None:
scorer = DummyRelevanceScorer()
batch = default_batch_score(scorer, ["a", "b"], "ctx")
assert [item.reason for item in batch] == ["a:ctx", "b:ctx"]
assert RelevanceScorer.score(scorer, "a", "ctx") is None
assert RelevanceScorer.score_batch(scorer, ["a"], "ctx") is None
assert RelevanceScorer.is_available() is True
def test_create_scorer_embedding_unavailable_branch(monkeypatch) -> None:
monkeypatch.setattr(
relevance_mod.EmbeddingScorer, "is_available", classmethod(lambda cls: False)
)
with pytest.raises(RuntimeError, match="sentence-transformers"):
create_scorer("embedding")
def test_bm25_internal_paths_and_non_normalized_mode() -> None:
scorer = BM25Scorer(normalize_score=False)
assert scorer._tokenize("") == []
assert scorer._compute_idf("x", doc_count=1, doc_freq=0) == 0.0
assert scorer._compute_idf("x", doc_count=1, doc_freq=1) > 0
assert scorer._bm25_score([], ["a"]) == (0.0, [])
assert scorer._bm25_score(["a"], []) == (0.0, [])
no_match = scorer.score("hello world", "missing")
assert no_match.reason == "BM25: no term matches"
one_match = scorer.score("find alice", "alice")
assert one_match.reason == "BM25: matched 'alice'"
assert one_match.score > 0
many_match = scorer.score("alpha beta gamma delta", "alpha beta gamma delta")
assert many_match.reason.startswith("BM25: matched 4 terms")
batch = scorer.score_batch(["alpha", "alpha beta"], "alpha beta")
assert [item.reason for item in batch] == ["BM25: 1 terms", "BM25: 2 terms"]
def test_embedding_numpy_and_model_error_paths(monkeypatch) -> None:
embedding._numpy = None
real_import = builtins.__import__
def fake_import(name, globals=None, locals=None, fromlist=(), level=0):
if name == "numpy":
raise ImportError("missing")
return real_import(name, globals, locals, fromlist, level)
monkeypatch.setattr(builtins, "__import__", fake_import)
with pytest.raises(ImportError, match="numpy is required"):
embedding._get_numpy()
monkeypatch.setattr(builtins, "__import__", real_import)
fake_np = SimpleNamespace(
linalg=SimpleNamespace(norm=lambda value: 0 if value == [0, 0] else 1),
dot=lambda a, b: -1,
)
monkeypatch.setattr(embedding, "_numpy", fake_np)
assert embedding._cosine_similarity([0, 0], [1, 0]) == 0.0
assert embedding._cosine_similarity([1, 0], [0, 1]) == 0.0
monkeypatch.setattr(EmbeddingScorer, "is_available", classmethod(lambda cls: False))
with pytest.raises(RuntimeError, match="requires fastembed"):
EmbeddingScorer()._get_model()
def test_embedding_score_empty_and_batch_shortcuts() -> None:
scorer = EmbeddingScorer()
assert scorer.score("", "ctx").reason == "Embedding: empty input"
assert scorer.score("item", "").reason == "Embedding: empty input"
assert scorer.score_batch([], "ctx") == []
assert scorer.score_batch(["item"], "")[0].reason == "Embedding: empty context"
def test_embedding_score_and_batch_with_fake_model(monkeypatch) -> None:
scorer = EmbeddingScorer()
monkeypatch.setattr(
scorer,
"_encode",
lambda texts: (
[[1.0, 0.0], [0.5, 0.5]] if len(texts) == 2 else [[1.0, 0.0], [0.0, 1.0], [1.0, 0.0]]
),
)
monkeypatch.setattr(
embedding, "_cosine_similarity", lambda a, b: 0.75 if a == [1.0, 0.0] else 0.25
)
single = scorer.score("item", "ctx")
assert single.score == 0.75
assert single.reason == "Embedding: semantic similarity 0.75"
batch = scorer.score_batch(["first", "second"], "ctx")
assert [item.score for item in batch] == [0.75, 0.25]
assert [item.reason for item in batch] == ["Embedding: 0.75", "Embedding: 0.25"]
def test_embedding_encode_caps_the_onnx_batch(monkeypatch) -> None:
# The default fastembed batch (256) sizes ORT's never-shrinking arena to the
# largest relevance split ever scored; every encode must pass the cap.
calls = []
class FakeModel:
def embed(self, texts, **kwargs):
calls.append(kwargs)
return [[1.0, 0.0] for _ in texts]
scorer = EmbeddingScorer()
monkeypatch.setattr(scorer, "_get_model", lambda: FakeModel())
assert len(scorer._encode([f"record {i}" for i in range(300)])) == 300
assert calls == [{"batch_size": embedding.EMBED_BATCH_SIZE}]
@pytest.mark.parametrize(
("platform", "large_cache", "shrinks"),
[("darwin", "0", True), ("darwin", None, False), ("linux", "0", False)],
)
def test_embedding_encode_shrinks_the_arena_only_without_the_large_cache(
monkeypatch, platform, large_cache, shrinks
) -> None:
# Shrinking under libmalloc's large cache re-grows the arena, so the run
# options are passed only on macOS with MallocLargeCache=0 (the CLI re-exec).
pytest.importorskip("onnxruntime")
seen = []
class Session:
def run(self, names, feed, run_options=None):
seen.append(
run_options
and run_options.get_run_config_entry("memory.enable_memory_arena_shrinkage")
)
class FakeModel:
model = SimpleNamespace(model=Session())
def embed(self, texts, **kwargs):
self.model.model.run(["out"], {})
return [[1.0, 0.0] for _ in texts]
monkeypatch.setattr(sys, "platform", platform)
if large_cache is None:
monkeypatch.delenv("MallocLargeCache", raising=False)
else:
monkeypatch.setenv("MallocLargeCache", large_cache)
model = FakeModel()
scorer = EmbeddingScorer()
monkeypatch.setattr(scorer, "_get_model", lambda: model)
scorer._encode(["a"])
scorer._encode(["b"])
assert seen == (["cpu:0", "cpu:0"] if shrinks else [None, None])
def test_hybrid_constructor_alpha_variants_and_single_score_paths(monkeypatch) -> None:
bm25_result = RelevanceScore(score=0.1, reason="bm25", matched_terms=["term"])
emb_result = RelevanceScore(score=0.9, reason="emb", matched_terms=[])
class FakeBM25:
def score(self, item: str, context: str) -> RelevanceScore:
return bm25_result
def score_batch(self, items: list[str], context: str) -> list[RelevanceScore]:
return [bm25_result for _ in items]
class FakeEmbedding:
def score(self, item: str, context: str) -> RelevanceScore:
return emb_result
def score_batch(self, items: list[str], context: str) -> list[RelevanceScore]:
return [emb_result for _ in items]
scorer = HybridScorer(
alpha=0.4, adaptive=True, bm25_scorer=FakeBM25(), embedding_scorer=FakeEmbedding()
)
assert scorer.has_embedding_support() is True
assert scorer._compute_alpha("find id 1234") == 0.65
assert scorer._compute_alpha("find host api.example.com") == 0.6
assert scorer._compute_alpha("find email test@example.com") == 0.6
single = scorer.score("item", "show me errors")
assert single.score == pytest.approx(0.58)
assert "Hybrid (α=0.40): BM25=0.10, Semantic=0.90" == single.reason
batch = scorer.score_batch(["a", "b"], "show me errors")
assert len(batch) == 2
assert batch[0].reason == "Hybrid (α=0.40): BM25=0.10, Emb=0.90"
def test_hybrid_fallback_and_empty_batch(monkeypatch) -> None:
scorer = HybridScorer(bm25_scorer=BM25Scorer())
scorer._embedding_available = False
scorer.embedding = None
empty = scorer.score_batch([], "ctx")
assert empty == []
boosted = scorer.score('{"id":"123","name":"alice"}', "alice")
assert boosted.score >= 0.3
assert "BM25 only, boosted" in boosted.reason
boosted_batch = scorer.score_batch(['{"id":"123"}', '{"id":"456"}'], "123 456")
assert all("BM25 only, boosted" in item.reason for item in boosted_batch)
def test_hybrid_auto_fallback_when_embeddings_unavailable(monkeypatch) -> None:
monkeypatch.setattr(hybrid.EmbeddingScorer, "is_available", classmethod(lambda cls: False))
scorer = HybridScorer()
assert scorer.has_embedding_support() is False