fix(relevance): hand the embedding arena back after each run on macOS (#4567)

## Description

Follow-up to #4507. The batch cap bounds one ONNX run, but concurrent
relevance splits share the scorer's ONNX session, and ONNX Runtime's CPU
arena keeps its high-water mark: the memory it grows to for 3 or 4
splits at once stays pinned for the life of the proxy.

Found on an Extra Headroom user's macOS proxy after #4507's batch size
was in place (wheel 0.41.0 + the cap as a patch, 13.5 h uptime):
phys_footprint 3.1 GB, of which 2.1 GB was arena regions in the BFC
doubling pattern (16, 32, 32, 64, 128, 256, 512 MB and 1 GB). The
periodic `malloc_zone_pressure_relief` trim releases none of it, and
neither does dropping the whole model.

This PR has each embedding run pass ORT's
`memory.enable_memory_arena_shrinkage` run option, so the arena hands
its free regions back after every run. It does this only on macOS under
`MallocLargeCache=0`, which the proxy CLI's malloc re-exec (#2879)
already sets. With libmalloc's large cache on, shrinking made it worse:
the freed blocks stay dirty in the cache and the arena grows again on
top of them (+3.8 GB against +2.5 GB).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/relevance/embedding.py`: new
`_shrink_arena_after_each_run(model)`, called once per model from
`EmbeddingScorer._encode`. On macOS with `MallocLargeCache=0` it wraps
the fastembed model's `InferenceSession.run` so every call gets
`RunOptions` with `memory.enable_memory_arena_shrinkage=cpu:0` (a
caller's own `run_options` still win). Elsewhere it does nothing. Any
failure (onnxruntime missing, fastembed internals moved) is logged at
debug and leaves today's behaviour.
- `tests/test_relevance_extra.py`:
`test_embedding_encode_shrinks_the_arena_only_without_the_large_cache`,
parametrized over darwin + `MallocLargeCache=0` (shrinks), darwin
without it, and linux (both unchanged). It fails without the fix
(`assert [None, None] == ['cpu:0', 'cpu:0']`).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev --extra relevance pytest tests/test_relevance_extra.py tests/test_relevance.py tests/test_relevance_split.py tests/test_*content_router*.py -q
286 passed, 2 skipped in 10.68s
$ uv run --frozen --extra dev --extra relevance pytest tests/test_relevance_extra.py -q -k shrinks_the_arena   # embedding.py at upstream/main
E   AssertionError: assert [None, None] == ['cpu:0', 'cpu:0']
1 failed, 2 passed
$ uvx ruff check headroom/relevance/embedding.py tests/test_relevance_extra.py
All checks passed!
$ uvx ruff format --check headroom/relevance/embedding.py tests/test_relevance_extra.py
2 files already formatted
$ uv run --frozen --extra dev --extra relevance --with mypy==1.20.2 mypy headroom/relevance/embedding.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 27.2 on an Apple M6, this branch's `uv` environment
(`--extra relevance`: onnxruntime 1.26.0, fastembed 0.8.0),
BAAI/bge-small-en-v1.5 from the local cache, run with
`MallocLargeCache=0 MallocAggressiveMadvise=1` (what the CLI re-exec
sets).
- Exact command / steps: a script imports
`headroom.relevance.embedding`, warms an `EmbeddingScorer`, scores 128
mixed-length records once, then twice runs 4 threads that each score the
same 128 records (4 concurrent relevance splits), waits 0.5 s and reads
`phys_footprint` with `footprint`; then times twenty 5-record calls.
"Before" swaps `_shrink_arena_after_each_run` for a no-op; each run is a
fresh process.
- Observed result: memory left after the parallel splits dropped from
+2.1 to 2.3 GB to +83 to 595 MB at the same speed, with bit-identical
embeddings (same sum over the 128-record output in every run).

  | | Before (main) | This PR |
  |---|---|---|
| Retained after 4 parallel splits, x2 | +2136, +2245, +2274 MB | +595,
+83, +591, +319, +83, +331 MB |
  | 4 parallel splits, x2 | 12.5 to 13.5 s | 12.4 to 14.0 s |
  | One 128-record split | 4.3 to 4.7 s | 4.3 to 5.2 s |
  | One 5-record call | 21 to 22 ms | 22 to 27 ms |

The same change against the 0.41.0 wheel's runtime (onnxruntime 1.27.0,
fastembed 0.9.1) settled at +80 to 85 MB in every run. Without
`MallocLargeCache=0`, shrinkage retained +3.6 to 3.9 GB against +2.5 to
2.6 GB without it, which is why it is gated.
- Not tested: Linux and Windows, so this PR leaves them unchanged; the
Rust scorer (no proxy path loads its model in 0.41.0); macOS releases
before 27.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, not rollout-managed
- Stable/default behavior changed: yes, on macOS when
`MallocLargeCache=0` is set (the default for the proxy CLI): the
embedding ONNX session now shrinks its CPU arena after each run.
Embeddings are unchanged; other platforms are unchanged.
- Kill switch / disable path: `HEADROOM_MALLOC_TUNING=0` (no re-exec, so
`MallocLargeCache` is unset and shrinkage stays off), or revert the
commit
- Unsafe override required: no
- Qualification impact: none expected; embeddings were bit-identical in
every measured run
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation: N/A, no user-facing setting changed. fastembed exposes no
run options, so the helper wraps the session's `run` on the instance
(`TextEmbedding.model.model`); if a fastembed release moves that
attribute, the wrap fails, is logged at debug, and the arena behaves as
on main. Extra Headroom carries this as a temporary patch against 0.41.0
until a release includes it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4567"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-11-03 19:26:42 -06:00
committed by GitHub
co-authored by Claude Opus 5.5
parent fe69f38116
commit 2363f83309
2 changed files with 69 additions and 0 deletions
+29
View File
@@ -27,6 +27,7 @@ from __future__ import annotations
import logging
import os
import sys
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
@@ -43,6 +44,32 @@ from .base import RelevanceScore, RelevanceScorer
# proxy. 16 holds ~10x less and runs faster, with less padding per batch.
EMBED_BATCH_SIZE = 16
def _shrink_arena_after_each_run(model: Any) -> None: # reaches into fastembed internals
"""Have ORT hand its free CPU arena regions back after every run.
The batch cap bounds one run, but concurrent relevance splits share the
session and still stack its arena: ~2.2 GB retained after four parallel
128-record splits in a macOS proxy, ~80 MB with arena shrinkage, same speed.
Only on macOS under ``MallocLargeCache=0``, which the proxy CLI's malloc
re-exec sets: with libmalloc's large cache on, the freed regions stay dirty
and shrinking re-grows the arena (~3.8 GB). Other platforms are unmeasured.
"""
model._headroom_arena_shrink_checked = True
if sys.platform != "darwin" or os.environ.get("MallocLargeCache") != "0":
return
try:
import onnxruntime as ort
session = model.model.model # TextEmbedding -> OnnxTextEmbedding -> InferenceSession
options = ort.RunOptions()
options.add_run_config_entry("memory.enable_memory_arena_shrinkage", "cpu:0")
run = session.run
session.run = lambda names, feed, run_options=None: run(names, feed, run_options or options)
except Exception as exc: # noqa: BLE001 - memory only, the arena keeps its high-water mark
logger.debug("ORT arena shrinkage unavailable: %s", exc)
# numpy is an optional dependency - import lazily
_numpy = None
@@ -335,6 +362,8 @@ class EmbeddingScorer(RelevanceScorer):
"""
np = _get_numpy()
model = self._get_model()
if not getattr(model, "_headroom_arena_shrink_checked", False):
_shrink_arena_after_each_run(model)
embeddings = list(model.embed(texts, batch_size=EMBED_BATCH_SIZE))
return np.array(embeddings)
+40
View File
@@ -1,6 +1,7 @@
from __future__ import annotations
import builtins
import sys
from dataclasses import dataclass
from types import SimpleNamespace
@@ -140,6 +141,45 @@ def test_embedding_encode_caps_the_onnx_batch(monkeypatch) -> None:
assert calls == [{"batch_size": embedding.EMBED_BATCH_SIZE}]
@pytest.mark.parametrize(
("platform", "large_cache", "shrinks"),
[("darwin", "0", True), ("darwin", None, False), ("linux", "0", False)],
)
def test_embedding_encode_shrinks_the_arena_only_without_the_large_cache(
monkeypatch, platform, large_cache, shrinks
) -> None:
# Shrinking under libmalloc's large cache re-grows the arena, so the run
# options are passed only on macOS with MallocLargeCache=0 (the CLI re-exec).
pytest.importorskip("onnxruntime")
seen = []
class Session:
def run(self, names, feed, run_options=None):
seen.append(
run_options
and run_options.get_run_config_entry("memory.enable_memory_arena_shrinkage")
)
class FakeModel:
model = SimpleNamespace(model=Session())
def embed(self, texts, **kwargs):
self.model.model.run(["out"], {})
return [[1.0, 0.0] for _ in texts]
monkeypatch.setattr(sys, "platform", platform)
if large_cache is None:
monkeypatch.delenv("MallocLargeCache", raising=False)
else:
monkeypatch.setenv("MallocLargeCache", large_cache)
model = FakeModel()
scorer = EmbeddingScorer()
monkeypatch.setattr(scorer, "_get_model", lambda: model)
scorer._encode(["a"])
scorer._encode(["b"])
assert seen == (["cpu:0", "cpu:0"] if shrinks else [None, None])
def test_hybrid_constructor_alpha_variants_and_single_score_paths(monkeypatch) -> None:
bm25_result = RelevanceScore(score=0.1, reason="bm25", matched_terms=["term"])
emb_result = RelevanceScore(score=0.9, reason="emb", matched_terms=[])