mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(relevance): hand the embedding arena back after each run on macOS (#4567)
## Description Follow-up to #4507. The batch cap bounds one ONNX run, but concurrent relevance splits share the scorer's ONNX session, and ONNX Runtime's CPU arena keeps its high-water mark: the memory it grows to for 3 or 4 splits at once stays pinned for the life of the proxy. Found on an Extra Headroom user's macOS proxy after #4507's batch size was in place (wheel 0.41.0 + the cap as a patch, 13.5 h uptime): phys_footprint 3.1 GB, of which 2.1 GB was arena regions in the BFC doubling pattern (16, 32, 32, 64, 128, 256, 512 MB and 1 GB). The periodic `malloc_zone_pressure_relief` trim releases none of it, and neither does dropping the whole model. This PR has each embedding run pass ORT's `memory.enable_memory_arena_shrinkage` run option, so the arena hands its free regions back after every run. It does this only on macOS under `MallocLargeCache=0`, which the proxy CLI's malloc re-exec (#2879) already sets. With libmalloc's large cache on, shrinking made it worse: the freed blocks stay dirty in the cache and the arena grows again on top of them (+3.8 GB against +2.5 GB). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/relevance/embedding.py`: new `_shrink_arena_after_each_run(model)`, called once per model from `EmbeddingScorer._encode`. On macOS with `MallocLargeCache=0` it wraps the fastembed model's `InferenceSession.run` so every call gets `RunOptions` with `memory.enable_memory_arena_shrinkage=cpu:0` (a caller's own `run_options` still win). Elsewhere it does nothing. Any failure (onnxruntime missing, fastembed internals moved) is logged at debug and leaves today's behaviour. - `tests/test_relevance_extra.py`: `test_embedding_encode_shrinks_the_arena_only_without_the_large_cache`, parametrized over darwin + `MallocLargeCache=0` (shrinks), darwin without it, and linux (both unchanged). It fails without the fix (`assert [None, None] == ['cpu:0', 'cpu:0']`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev --extra relevance pytest tests/test_relevance_extra.py tests/test_relevance.py tests/test_relevance_split.py tests/test_*content_router*.py -q 286 passed, 2 skipped in 10.68s $ uv run --frozen --extra dev --extra relevance pytest tests/test_relevance_extra.py -q -k shrinks_the_arena # embedding.py at upstream/main E AssertionError: assert [None, None] == ['cpu:0', 'cpu:0'] 1 failed, 2 passed $ uvx ruff check headroom/relevance/embedding.py tests/test_relevance_extra.py All checks passed! $ uvx ruff format --check headroom/relevance/embedding.py tests/test_relevance_extra.py 2 files already formatted $ uv run --frozen --extra dev --extra relevance --with mypy==1.20.2 mypy headroom/relevance/embedding.py --ignore-missing-imports Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 27.2 on an Apple M6, this branch's `uv` environment (`--extra relevance`: onnxruntime 1.26.0, fastembed 0.8.0), BAAI/bge-small-en-v1.5 from the local cache, run with `MallocLargeCache=0 MallocAggressiveMadvise=1` (what the CLI re-exec sets). - Exact command / steps: a script imports `headroom.relevance.embedding`, warms an `EmbeddingScorer`, scores 128 mixed-length records once, then twice runs 4 threads that each score the same 128 records (4 concurrent relevance splits), waits 0.5 s and reads `phys_footprint` with `footprint`; then times twenty 5-record calls. "Before" swaps `_shrink_arena_after_each_run` for a no-op; each run is a fresh process. - Observed result: memory left after the parallel splits dropped from +2.1 to 2.3 GB to +83 to 595 MB at the same speed, with bit-identical embeddings (same sum over the 128-record output in every run). | | Before (main) | This PR | |---|---|---| | Retained after 4 parallel splits, x2 | +2136, +2245, +2274 MB | +595, +83, +591, +319, +83, +331 MB | | 4 parallel splits, x2 | 12.5 to 13.5 s | 12.4 to 14.0 s | | One 128-record split | 4.3 to 4.7 s | 4.3 to 5.2 s | | One 5-record call | 21 to 22 ms | 22 to 27 ms | The same change against the 0.41.0 wheel's runtime (onnxruntime 1.27.0, fastembed 0.9.1) settled at +80 to 85 MB in every run. Without `MallocLargeCache=0`, shrinkage retained +3.6 to 3.9 GB against +2.5 to 2.6 GB without it, which is why it is gated. - Not tested: Linux and Windows, so this PR leaves them unchanged; the Rust scorer (no proxy path loads its model in 0.41.0); macOS releases before 27. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a, not rollout-managed - Stable/default behavior changed: yes, on macOS when `MallocLargeCache=0` is set (the default for the proxy CLI): the embedding ONNX session now shrinks its CPU arena after each run. Embeddings are unchanged; other platforms are unchanged. - Kill switch / disable path: `HEADROOM_MALLOC_TUNING=0` (no re-exec, so `MallocLargeCache` is unset and shrinkage stays off), or revert the commit - Unsafe override required: no - Qualification impact: none expected; embeddings were bit-identical in every measured run - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Documentation: N/A, no user-facing setting changed. fastembed exposes no run options, so the helper wraps the session's `run` on the instance (`TextEmbedding.model.model`); if a fastembed release moves that attribute, the wrap fails, is logged at debug, and the arena behaves as on main. Extra Headroom carries this as a temporary patch against 0.41.0 until a release includes it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4567" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
fe69f38116
commit
2363f83309
@@ -27,6 +27,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
@@ -43,6 +44,32 @@ from .base import RelevanceScore, RelevanceScorer
|
||||
# proxy. 16 holds ~10x less and runs faster, with less padding per batch.
|
||||
EMBED_BATCH_SIZE = 16
|
||||
|
||||
|
||||
def _shrink_arena_after_each_run(model: Any) -> None: # reaches into fastembed internals
|
||||
"""Have ORT hand its free CPU arena regions back after every run.
|
||||
|
||||
The batch cap bounds one run, but concurrent relevance splits share the
|
||||
session and still stack its arena: ~2.2 GB retained after four parallel
|
||||
128-record splits in a macOS proxy, ~80 MB with arena shrinkage, same speed.
|
||||
Only on macOS under ``MallocLargeCache=0``, which the proxy CLI's malloc
|
||||
re-exec sets: with libmalloc's large cache on, the freed regions stay dirty
|
||||
and shrinking re-grows the arena (~3.8 GB). Other platforms are unmeasured.
|
||||
"""
|
||||
model._headroom_arena_shrink_checked = True
|
||||
if sys.platform != "darwin" or os.environ.get("MallocLargeCache") != "0":
|
||||
return
|
||||
try:
|
||||
import onnxruntime as ort
|
||||
|
||||
session = model.model.model # TextEmbedding -> OnnxTextEmbedding -> InferenceSession
|
||||
options = ort.RunOptions()
|
||||
options.add_run_config_entry("memory.enable_memory_arena_shrinkage", "cpu:0")
|
||||
run = session.run
|
||||
session.run = lambda names, feed, run_options=None: run(names, feed, run_options or options)
|
||||
except Exception as exc: # noqa: BLE001 - memory only, the arena keeps its high-water mark
|
||||
logger.debug("ORT arena shrinkage unavailable: %s", exc)
|
||||
|
||||
|
||||
# numpy is an optional dependency - import lazily
|
||||
_numpy = None
|
||||
|
||||
@@ -335,6 +362,8 @@ class EmbeddingScorer(RelevanceScorer):
|
||||
"""
|
||||
np = _get_numpy()
|
||||
model = self._get_model()
|
||||
if not getattr(model, "_headroom_arena_shrink_checked", False):
|
||||
_shrink_arena_after_each_run(model)
|
||||
embeddings = list(model.embed(texts, batch_size=EMBED_BATCH_SIZE))
|
||||
return np.array(embeddings)
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import builtins
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from types import SimpleNamespace
|
||||
|
||||
@@ -140,6 +141,45 @@ def test_embedding_encode_caps_the_onnx_batch(monkeypatch) -> None:
|
||||
assert calls == [{"batch_size": embedding.EMBED_BATCH_SIZE}]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("platform", "large_cache", "shrinks"),
|
||||
[("darwin", "0", True), ("darwin", None, False), ("linux", "0", False)],
|
||||
)
|
||||
def test_embedding_encode_shrinks_the_arena_only_without_the_large_cache(
|
||||
monkeypatch, platform, large_cache, shrinks
|
||||
) -> None:
|
||||
# Shrinking under libmalloc's large cache re-grows the arena, so the run
|
||||
# options are passed only on macOS with MallocLargeCache=0 (the CLI re-exec).
|
||||
pytest.importorskip("onnxruntime")
|
||||
seen = []
|
||||
|
||||
class Session:
|
||||
def run(self, names, feed, run_options=None):
|
||||
seen.append(
|
||||
run_options
|
||||
and run_options.get_run_config_entry("memory.enable_memory_arena_shrinkage")
|
||||
)
|
||||
|
||||
class FakeModel:
|
||||
model = SimpleNamespace(model=Session())
|
||||
|
||||
def embed(self, texts, **kwargs):
|
||||
self.model.model.run(["out"], {})
|
||||
return [[1.0, 0.0] for _ in texts]
|
||||
|
||||
monkeypatch.setattr(sys, "platform", platform)
|
||||
if large_cache is None:
|
||||
monkeypatch.delenv("MallocLargeCache", raising=False)
|
||||
else:
|
||||
monkeypatch.setenv("MallocLargeCache", large_cache)
|
||||
model = FakeModel()
|
||||
scorer = EmbeddingScorer()
|
||||
monkeypatch.setattr(scorer, "_get_model", lambda: model)
|
||||
scorer._encode(["a"])
|
||||
scorer._encode(["b"])
|
||||
assert seen == (["cpu:0", "cpu:0"] if shrinks else [None, None])
|
||||
|
||||
|
||||
def test_hybrid_constructor_alpha_variants_and_single_score_paths(monkeypatch) -> None:
|
||||
bm25_result = RelevanceScore(score=0.1, reason="bm25", matched_terms=["term"])
|
||||
emb_result = RelevanceScore(score=0.9, reason="emb", matched_terms=[])
|
||||
|
||||
Reference in New Issue
Block a user