mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description Follow-up to #4507. The batch cap bounds one ONNX run, but concurrent relevance splits share the scorer's ONNX session, and ONNX Runtime's CPU arena keeps its high-water mark: the memory it grows to for 3 or 4 splits at once stays pinned for the life of the proxy. Found on an Extra Headroom user's macOS proxy after #4507's batch size was in place (wheel 0.41.0 + the cap as a patch, 13.5 h uptime): phys_footprint 3.1 GB, of which 2.1 GB was arena regions in the BFC doubling pattern (16, 32, 32, 64, 128, 256, 512 MB and 1 GB). The periodic `malloc_zone_pressure_relief` trim releases none of it, and neither does dropping the whole model. This PR has each embedding run pass ORT's `memory.enable_memory_arena_shrinkage` run option, so the arena hands its free regions back after every run. It does this only on macOS under `MallocLargeCache=0`, which the proxy CLI's malloc re-exec (#2879) already sets. With libmalloc's large cache on, shrinking made it worse: the freed blocks stay dirty in the cache and the arena grows again on top of them (+3.8 GB against +2.5 GB). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/relevance/embedding.py`: new `_shrink_arena_after_each_run(model)`, called once per model from `EmbeddingScorer._encode`. On macOS with `MallocLargeCache=0` it wraps the fastembed model's `InferenceSession.run` so every call gets `RunOptions` with `memory.enable_memory_arena_shrinkage=cpu:0` (a caller's own `run_options` still win). Elsewhere it does nothing. Any failure (onnxruntime missing, fastembed internals moved) is logged at debug and leaves today's behaviour. - `tests/test_relevance_extra.py`: `test_embedding_encode_shrinks_the_arena_only_without_the_large_cache`, parametrized over darwin + `MallocLargeCache=0` (shrinks), darwin without it, and linux (both unchanged). It fails without the fix (`assert [None, None] == ['cpu:0', 'cpu:0']`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev --extra relevance pytest tests/test_relevance_extra.py tests/test_relevance.py tests/test_relevance_split.py tests/test_*content_router*.py -q 286 passed, 2 skipped in 10.68s $ uv run --frozen --extra dev --extra relevance pytest tests/test_relevance_extra.py -q -k shrinks_the_arena # embedding.py at upstream/main E AssertionError: assert [None, None] == ['cpu:0', 'cpu:0'] 1 failed, 2 passed $ uvx ruff check headroom/relevance/embedding.py tests/test_relevance_extra.py All checks passed! $ uvx ruff format --check headroom/relevance/embedding.py tests/test_relevance_extra.py 2 files already formatted $ uv run --frozen --extra dev --extra relevance --with mypy==1.20.2 mypy headroom/relevance/embedding.py --ignore-missing-imports Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 27.2 on an Apple M6, this branch's `uv` environment (`--extra relevance`: onnxruntime 1.26.0, fastembed 0.8.0), BAAI/bge-small-en-v1.5 from the local cache, run with `MallocLargeCache=0 MallocAggressiveMadvise=1` (what the CLI re-exec sets). - Exact command / steps: a script imports `headroom.relevance.embedding`, warms an `EmbeddingScorer`, scores 128 mixed-length records once, then twice runs 4 threads that each score the same 128 records (4 concurrent relevance splits), waits 0.5 s and reads `phys_footprint` with `footprint`; then times twenty 5-record calls. "Before" swaps `_shrink_arena_after_each_run` for a no-op; each run is a fresh process. - Observed result: memory left after the parallel splits dropped from +2.1 to 2.3 GB to +83 to 595 MB at the same speed, with bit-identical embeddings (same sum over the 128-record output in every run). | | Before (main) | This PR | |---|---|---| | Retained after 4 parallel splits, x2 | +2136, +2245, +2274 MB | +595, +83, +591, +319, +83, +331 MB | | 4 parallel splits, x2 | 12.5 to 13.5 s | 12.4 to 14.0 s | | One 128-record split | 4.3 to 4.7 s | 4.3 to 5.2 s | | One 5-record call | 21 to 22 ms | 22 to 27 ms | The same change against the 0.41.0 wheel's runtime (onnxruntime 1.27.0, fastembed 0.9.1) settled at +80 to 85 MB in every run. Without `MallocLargeCache=0`, shrinkage retained +3.6 to 3.9 GB against +2.5 to 2.6 GB without it, which is why it is gated. - Not tested: Linux and Windows, so this PR leaves them unchanged; the Rust scorer (no proxy path loads its model in 0.41.0); macOS releases before 27. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a, not rollout-managed - Stable/default behavior changed: yes, on macOS when `MallocLargeCache=0` is set (the default for the proxy CLI): the embedding ONNX session now shrinks its CPU arena after each run. Embeddings are unchanged; other platforms are unchanged. - Kill switch / disable path: `HEADROOM_MALLOC_TUNING=0` (no re-exec, so `MallocLargeCache` is unset and shrinkage stays off), or revert the commit - Unsafe override required: no - Qualification impact: none expected; embeddings were bit-identical in every measured run - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Documentation: N/A, no user-facing setting changed. fastembed exposes no run options, so the helper wraps the session's `run` on the instance (`TextEmbedding.model.model`); if a fastembed release moves that attribute, the wrap fails, is logged at debug, and the arena behaves as on main. Extra Headroom carries this as a temporary patch against 0.41.0 until a release includes it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4567" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
238 lines
8.7 KiB
Python
238 lines
8.7 KiB
Python
from __future__ import annotations
|
||
|
||
import builtins
|
||
import sys
|
||
from dataclasses import dataclass
|
||
from types import SimpleNamespace
|
||
|
||
import pytest
|
||
|
||
import headroom.relevance as relevance_mod
|
||
from headroom.relevance import (
|
||
BM25Scorer,
|
||
EmbeddingScorer,
|
||
HybridScorer,
|
||
create_scorer,
|
||
embedding,
|
||
hybrid,
|
||
)
|
||
from headroom.relevance.base import RelevanceScore, RelevanceScorer, default_batch_score
|
||
|
||
|
||
@dataclass
|
||
class DummyRelevanceScorer(RelevanceScorer):
|
||
def score(self, item: str, context: str) -> RelevanceScore:
|
||
return RelevanceScore(score=0.4, reason=f"{item}:{context}")
|
||
|
||
def score_batch(self, items: list[str], context: str) -> list[RelevanceScore]:
|
||
return [RelevanceScore(score=0.2, reason=context) for _ in items]
|
||
|
||
|
||
def test_base_default_batch_and_abstract_methods() -> None:
|
||
scorer = DummyRelevanceScorer()
|
||
batch = default_batch_score(scorer, ["a", "b"], "ctx")
|
||
assert [item.reason for item in batch] == ["a:ctx", "b:ctx"]
|
||
|
||
assert RelevanceScorer.score(scorer, "a", "ctx") is None
|
||
assert RelevanceScorer.score_batch(scorer, ["a"], "ctx") is None
|
||
assert RelevanceScorer.is_available() is True
|
||
|
||
|
||
def test_create_scorer_embedding_unavailable_branch(monkeypatch) -> None:
|
||
monkeypatch.setattr(
|
||
relevance_mod.EmbeddingScorer, "is_available", classmethod(lambda cls: False)
|
||
)
|
||
with pytest.raises(RuntimeError, match="sentence-transformers"):
|
||
create_scorer("embedding")
|
||
|
||
|
||
def test_bm25_internal_paths_and_non_normalized_mode() -> None:
|
||
scorer = BM25Scorer(normalize_score=False)
|
||
assert scorer._tokenize("") == []
|
||
assert scorer._compute_idf("x", doc_count=1, doc_freq=0) == 0.0
|
||
assert scorer._compute_idf("x", doc_count=1, doc_freq=1) > 0
|
||
assert scorer._bm25_score([], ["a"]) == (0.0, [])
|
||
assert scorer._bm25_score(["a"], []) == (0.0, [])
|
||
|
||
no_match = scorer.score("hello world", "missing")
|
||
assert no_match.reason == "BM25: no term matches"
|
||
|
||
one_match = scorer.score("find alice", "alice")
|
||
assert one_match.reason == "BM25: matched 'alice'"
|
||
assert one_match.score > 0
|
||
|
||
many_match = scorer.score("alpha beta gamma delta", "alpha beta gamma delta")
|
||
assert many_match.reason.startswith("BM25: matched 4 terms")
|
||
|
||
batch = scorer.score_batch(["alpha", "alpha beta"], "alpha beta")
|
||
assert [item.reason for item in batch] == ["BM25: 1 terms", "BM25: 2 terms"]
|
||
|
||
|
||
def test_embedding_numpy_and_model_error_paths(monkeypatch) -> None:
|
||
embedding._numpy = None
|
||
real_import = builtins.__import__
|
||
|
||
def fake_import(name, globals=None, locals=None, fromlist=(), level=0):
|
||
if name == "numpy":
|
||
raise ImportError("missing")
|
||
return real_import(name, globals, locals, fromlist, level)
|
||
|
||
monkeypatch.setattr(builtins, "__import__", fake_import)
|
||
with pytest.raises(ImportError, match="numpy is required"):
|
||
embedding._get_numpy()
|
||
|
||
monkeypatch.setattr(builtins, "__import__", real_import)
|
||
fake_np = SimpleNamespace(
|
||
linalg=SimpleNamespace(norm=lambda value: 0 if value == [0, 0] else 1),
|
||
dot=lambda a, b: -1,
|
||
)
|
||
monkeypatch.setattr(embedding, "_numpy", fake_np)
|
||
assert embedding._cosine_similarity([0, 0], [1, 0]) == 0.0
|
||
assert embedding._cosine_similarity([1, 0], [0, 1]) == 0.0
|
||
|
||
monkeypatch.setattr(EmbeddingScorer, "is_available", classmethod(lambda cls: False))
|
||
with pytest.raises(RuntimeError, match="requires fastembed"):
|
||
EmbeddingScorer()._get_model()
|
||
|
||
|
||
def test_embedding_score_empty_and_batch_shortcuts() -> None:
|
||
scorer = EmbeddingScorer()
|
||
assert scorer.score("", "ctx").reason == "Embedding: empty input"
|
||
assert scorer.score("item", "").reason == "Embedding: empty input"
|
||
assert scorer.score_batch([], "ctx") == []
|
||
assert scorer.score_batch(["item"], "")[0].reason == "Embedding: empty context"
|
||
|
||
|
||
def test_embedding_score_and_batch_with_fake_model(monkeypatch) -> None:
|
||
scorer = EmbeddingScorer()
|
||
monkeypatch.setattr(
|
||
scorer,
|
||
"_encode",
|
||
lambda texts: (
|
||
[[1.0, 0.0], [0.5, 0.5]] if len(texts) == 2 else [[1.0, 0.0], [0.0, 1.0], [1.0, 0.0]]
|
||
),
|
||
)
|
||
monkeypatch.setattr(
|
||
embedding, "_cosine_similarity", lambda a, b: 0.75 if a == [1.0, 0.0] else 0.25
|
||
)
|
||
|
||
single = scorer.score("item", "ctx")
|
||
assert single.score == 0.75
|
||
assert single.reason == "Embedding: semantic similarity 0.75"
|
||
|
||
batch = scorer.score_batch(["first", "second"], "ctx")
|
||
assert [item.score for item in batch] == [0.75, 0.25]
|
||
assert [item.reason for item in batch] == ["Embedding: 0.75", "Embedding: 0.25"]
|
||
|
||
|
||
def test_embedding_encode_caps_the_onnx_batch(monkeypatch) -> None:
|
||
# The default fastembed batch (256) sizes ORT's never-shrinking arena to the
|
||
# largest relevance split ever scored; every encode must pass the cap.
|
||
calls = []
|
||
|
||
class FakeModel:
|
||
def embed(self, texts, **kwargs):
|
||
calls.append(kwargs)
|
||
return [[1.0, 0.0] for _ in texts]
|
||
|
||
scorer = EmbeddingScorer()
|
||
monkeypatch.setattr(scorer, "_get_model", lambda: FakeModel())
|
||
assert len(scorer._encode([f"record {i}" for i in range(300)])) == 300
|
||
assert calls == [{"batch_size": embedding.EMBED_BATCH_SIZE}]
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
("platform", "large_cache", "shrinks"),
|
||
[("darwin", "0", True), ("darwin", None, False), ("linux", "0", False)],
|
||
)
|
||
def test_embedding_encode_shrinks_the_arena_only_without_the_large_cache(
|
||
monkeypatch, platform, large_cache, shrinks
|
||
) -> None:
|
||
# Shrinking under libmalloc's large cache re-grows the arena, so the run
|
||
# options are passed only on macOS with MallocLargeCache=0 (the CLI re-exec).
|
||
pytest.importorskip("onnxruntime")
|
||
seen = []
|
||
|
||
class Session:
|
||
def run(self, names, feed, run_options=None):
|
||
seen.append(
|
||
run_options
|
||
and run_options.get_run_config_entry("memory.enable_memory_arena_shrinkage")
|
||
)
|
||
|
||
class FakeModel:
|
||
model = SimpleNamespace(model=Session())
|
||
|
||
def embed(self, texts, **kwargs):
|
||
self.model.model.run(["out"], {})
|
||
return [[1.0, 0.0] for _ in texts]
|
||
|
||
monkeypatch.setattr(sys, "platform", platform)
|
||
if large_cache is None:
|
||
monkeypatch.delenv("MallocLargeCache", raising=False)
|
||
else:
|
||
monkeypatch.setenv("MallocLargeCache", large_cache)
|
||
model = FakeModel()
|
||
scorer = EmbeddingScorer()
|
||
monkeypatch.setattr(scorer, "_get_model", lambda: model)
|
||
scorer._encode(["a"])
|
||
scorer._encode(["b"])
|
||
assert seen == (["cpu:0", "cpu:0"] if shrinks else [None, None])
|
||
|
||
|
||
def test_hybrid_constructor_alpha_variants_and_single_score_paths(monkeypatch) -> None:
|
||
bm25_result = RelevanceScore(score=0.1, reason="bm25", matched_terms=["term"])
|
||
emb_result = RelevanceScore(score=0.9, reason="emb", matched_terms=[])
|
||
|
||
class FakeBM25:
|
||
def score(self, item: str, context: str) -> RelevanceScore:
|
||
return bm25_result
|
||
|
||
def score_batch(self, items: list[str], context: str) -> list[RelevanceScore]:
|
||
return [bm25_result for _ in items]
|
||
|
||
class FakeEmbedding:
|
||
def score(self, item: str, context: str) -> RelevanceScore:
|
||
return emb_result
|
||
|
||
def score_batch(self, items: list[str], context: str) -> list[RelevanceScore]:
|
||
return [emb_result for _ in items]
|
||
|
||
scorer = HybridScorer(
|
||
alpha=0.4, adaptive=True, bm25_scorer=FakeBM25(), embedding_scorer=FakeEmbedding()
|
||
)
|
||
assert scorer.has_embedding_support() is True
|
||
assert scorer._compute_alpha("find id 1234") == 0.65
|
||
assert scorer._compute_alpha("find host api.example.com") == 0.6
|
||
assert scorer._compute_alpha("find email test@example.com") == 0.6
|
||
|
||
single = scorer.score("item", "show me errors")
|
||
assert single.score == pytest.approx(0.58)
|
||
assert "Hybrid (α=0.40): BM25=0.10, Semantic=0.90" == single.reason
|
||
|
||
batch = scorer.score_batch(["a", "b"], "show me errors")
|
||
assert len(batch) == 2
|
||
assert batch[0].reason == "Hybrid (α=0.40): BM25=0.10, Emb=0.90"
|
||
|
||
|
||
def test_hybrid_fallback_and_empty_batch(monkeypatch) -> None:
|
||
scorer = HybridScorer(bm25_scorer=BM25Scorer())
|
||
scorer._embedding_available = False
|
||
scorer.embedding = None
|
||
|
||
empty = scorer.score_batch([], "ctx")
|
||
assert empty == []
|
||
|
||
boosted = scorer.score('{"id":"123","name":"alice"}', "alice")
|
||
assert boosted.score >= 0.3
|
||
assert "BM25 only, boosted" in boosted.reason
|
||
|
||
boosted_batch = scorer.score_batch(['{"id":"123"}', '{"id":"456"}'], "123 456")
|
||
assert all("BM25 only, boosted" in item.reason for item in boosted_batch)
|
||
|
||
|
||
def test_hybrid_auto_fallback_when_embeddings_unavailable(monkeypatch) -> None:
|
||
monkeypatch.setattr(hybrid.EmbeddingScorer, "is_available", classmethod(lambda cls: False))
|
||
scorer = HybridScorer()
|
||
assert scorer.has_embedding_support() is False
|