mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
The embedding relevance scorer calls fastembed's `embed()` at its
default batch of 256. fastembed pads each batch to its longest text. Its
ONNX session keeps ONNX Runtime's CPU arena on, and the arena is never
returned to the OS. A relevance split over a large tool output therefore
sizes the arena to the biggest batch it has ever scored, and the memory
stays pinned for the life of the proxy.
Found on an Extra Headroom user's macOS proxy (wheel 0.41.0, 4 days
uptime): phys_footprint was 3.94 GB, and 2.5 GB of it was arena regions
in the BFC doubling pattern (32, 64, 128 and 256 MB, then 1 GB and 1
GB), all dirty and swapped out. The tracked caches (`/debug/memory`)
totalled 10 MB. The periodic `malloc_zone_pressure_relief` trim cannot
release arena memory.
This PR caps both scorers at 16 texts per ONNX run: Python
`EmbeddingScorer._encode` and the Rust `EmbeddingScorer::score_batch`.
That retains about 10x less memory, runs faster because each batch
carries less padding, and leaves keep/drop decisions unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/relevance/embedding.py`: new `EMBED_BATCH_SIZE = 16`, passed
as `batch_size` to `model.embed()`.
- `crates/headroom-core/src/relevance/embedding.rs`: `score_batch`
passes the scorer's `batch_size` instead of `None`. It is chosen once in
`try_new_with_model`: `Some(EMBED_BATCH_SIZE)` (16), or `None` (one
batch, as before) for models fastembed marks `QuantizationMode::Dynamic`
(e.g. `AllMiniLML6V2Q`), whose `embed` rejects a batch size smaller than
the input because each run calibrates its own int8 range.
- `tests/test_relevance_extra.py`: asserts that `_encode` passes the
cap. The test fails without the fix (`assert [{}] == [{'batch_size':
16}]`).
- Real-model tests in both languages
(`fastembed_batches_past_the_cap_keep_each_score_with_its_item` and
`test_batches_past_the_cap_keep_each_score_with_its_item`): 40 texts of
mixed lengths span three batches with the one related item in the last
batch. That item must still rank first, and every batched score must
match the item scored on its own.
- Rust dynamic-quantized coverage:
`dynamic_quantized_models_embed_in_one_batch` (runs in CI, no model)
pins the per-model choice;
`fastembed_dynamic_quantized_model_scores_on_both_sides_of_the_cap`
(real `AllMiniLML6V2Q`) scores 15, 16 and 40 items plus the context and
requires no inference-error fallback and the related item ranked first.
It fails on 92707844 at 16 items with fastembed's "Dynamic quantization
cannot be used with batching" error.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_relevance_extra.py tests/test_relevance.py tests/test_relevance_split.py -q
63 passed, 4 skipped in 1.75s
$ uvx ruff check headroom/relevance/embedding.py tests/test_relevance_extra.py
All checks passed!
$ uvx ruff format --check headroom/relevance/embedding.py tests/test_relevance_extra.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/relevance/embedding.py
Success: no issues found in 1 source file
$ cargo fmt --all -- --check (clean)
$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.27 dylib> cargo test -p headroom-core --lib relevance::embedding -- --test-threads=1
test result: ok. 17 passed; 0 failed (real models, incl. the 40-text multi-batch test and AllMiniLML6V2Q at 15/16/40 items)
$ cargo test -p headroom-core --lib relevance::embedding
test result: ok. 17 passed; 0 failed (model tests skip without RUN_FASTEMBED_TESTS)
$ cargo test -p headroom-core --no-default-features --lib relevance::embedding
test result: ok. 8 passed; 0 failed
$ cargo clippy -p headroom-core --all-targets -- -D warnings
Finished (no warnings)
$ FASTEMBED_CACHE_PATH=<cache> uv run --frozen --extra dev --with fastembed==0.9.1 pytest tests/test_relevance.py -k TestEmbeddingScorer
4 passed (real model, incl. the 40-text multi-batch test)
$ uv run --frozen --extra dev pytest tests/test_*content_router*.py tests/test_relevance*.py -q
236 passed, 7 skipped
```
## Real Behavior Proof
- Environment: macOS 27 on an M2 (16 GB), the installed headroom-ai
0.41.0 runtime (fastembed 0.8.0, onnxruntime 1.27),
BAAI/bge-small-en-v1.5 from the local cache; for the end-to-end run, a
scratch proxy started with Extra Headroom's flags (`--no-http2
--no-rate-limit --log-messages`, isolated HOME, telemetry off) in front
of a local mock Anthropic upstream.
- Exact command / steps: (1) bench: 120 records cut from a 7k-line Rust
source file by the real `segment()`, `EmbeddingScorer.score_batch` 3
times per batch size in a fresh process each, measured with `footprint`;
(2) equivalence: `plan_relevance_split` with `create_scorer("hybrid")`
over 24 cases (8 real corpora of 5 to 411 records: Rust source, proxy
log, grep output, git log, transcript JSONL, long-line markdown, cargo
output, a tiny log; 3 queries each), batch 16 and 256 in separate
processes; (3) end to end: 13 Claude Code shaped requests (question,
Bash `tool_use`, a 2k to 145k token `tool_result`) through the scratch
proxy, batch 16 and 256.
- Observed result: identical splits in 24 of 24 cases at 1.7x less time;
end to end, one relevance split grew the batch-256 proxy by 10.4 GB (to
an 11 GB footprint) against 0.99 GB at batch 16, with 12 of 13 forwarded
bodies byte-identical.
| Check | Batch 256 (before) | Batch 16 (this PR) |
|---|---|---|
| Bench, 120 records: time / retained | 16.8 s / +3769 MB | 8.7 s / +339
MB (max score diff 7.8e-5, 0 flips) |
| Equivalence, 24 cases: total split time | 1295 s | 746 s (24/24
identical runs, worst score diff 4.8e-5) |
| End to end: proxy footprint growth | +10.4 GB | +0.99 GB |
| End to end: 16k-token search output | hit the 30 s compression
timeout, forwarded uncompressed | compressed in 20.4 s, 5,338 tokens
saved |
- Not tested: Linux and Windows; memory and timing of the Rust scorer
(its real-model tests pass, but no proxy path loads its model in
0.41.0); turning the arena off instead, which on macOS retained between
+76 and +1277 MB across runs at batch 256 (libmalloc keeps the freed
blocks), so the batch size is the dependable lever.
Rust, dynamic-quantized model (added for review, 135c2b66). This is a
real-model cargo test (real ONNX model through fastembed-rs 5.17.3,
onnxruntime 1.27, macOS 27 on M2), not a proxy run: in the 0.41.0 wheel
the proxy builds the Rust `HybridScorer::default()`, which never loads
an embedding model, so no proxy request reaches this code.
```text
$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.27 dylib> cargo test -p headroom-core --lib relevance::embedding::tests::fastembed_dynamic_quantized -- --test-threads=1
# on 92707844 (test added, source unchanged):
panicked at crates/headroom-core/src/relevance/embedding.rs:569:17:
16 items: Embedding: inference failed: Dynamic quantization cannot be used with batching. [...]
test result: FAILED. 0 passed; 1 failed
# on 135c2b66:
test relevance::embedding::tests::fastembed_dynamic_quantized_model_scores_on_both_sides_of_the_cap ... ok
```
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, not rollout-managed
- Stable/default behavior changed: yes; embedding inference now runs in
batches of 16 texts instead of 256 (scores match within 1e-4, no
keep/drop changes measured). Rust models fastembed marks
dynamic-quantized (e.g. `AllMiniLML6V2Q` via `try_new_with_model`) still
run as one batch, as before.
- Kill switch / disable path: none needed for a batch-size constant;
revert the commit
- Unsafe override required: no
- Qualification impact: none expected; relevance-split decisions were
identical on the measured input
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Documentation: N/A, since no user-facing setting changed. The arena is
left on: the batch cap bounds it, and turning it off alone did not
reliably lower retained memory on macOS. Extra Headroom will carry this
as a temporary patch against 0.41.0 until a release includes it.
Python has no equivalent constraint: fastembed 0.9.1 has no quantization
check on `embed`; `_embed_documents`
(`fastembed/text/onnx_text_model.py`) splits every model's input with
`iter_batch`, and the only batch-size check is `>= 1`
(`fastembed/common/utils.py`). Its previous default of 256 already split
every model's input the same way.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
402 lines
14 KiB
Python
402 lines
14 KiB
Python
"""Tests for the relevance scoring module.
|
|
|
|
Tests all scorer tiers:
|
|
- BM25Scorer (zero dependencies)
|
|
- EmbeddingScorer (requires sentence-transformers)
|
|
- HybridScorer (combines BM25 + embeddings)
|
|
"""
|
|
|
|
import pytest
|
|
|
|
from headroom.relevance import (
|
|
BM25Scorer,
|
|
EmbeddingScorer,
|
|
HybridScorer,
|
|
RelevanceScore,
|
|
create_scorer,
|
|
embedding_available,
|
|
)
|
|
|
|
|
|
class TestRelevanceScore:
|
|
"""Tests for RelevanceScore dataclass."""
|
|
|
|
def test_score_clamping_high(self):
|
|
"""Scores above 1.0 are clamped."""
|
|
score = RelevanceScore(score=1.5, reason="test")
|
|
assert score.score == 1.0
|
|
|
|
def test_score_clamping_low(self):
|
|
"""Scores below 0.0 are clamped."""
|
|
score = RelevanceScore(score=-0.5, reason="test")
|
|
assert score.score == 0.0
|
|
|
|
def test_score_valid_range(self):
|
|
"""Scores in valid range are preserved."""
|
|
score = RelevanceScore(score=0.75, reason="test")
|
|
assert score.score == 0.75
|
|
|
|
|
|
class TestBM25Scorer:
|
|
"""Tests for BM25 keyword relevance scorer."""
|
|
|
|
def test_exact_uuid_match(self):
|
|
"""BM25 finds exact UUID matches."""
|
|
scorer = BM25Scorer()
|
|
items = [
|
|
'{"id": "550e8400-e29b-41d4-a716-446655440000", "name": "Alice"}',
|
|
'{"id": "123e4567-e89b-12d3-a456-426614174000", "name": "Bob"}',
|
|
]
|
|
context = "find record 550e8400-e29b-41d4-a716-446655440000"
|
|
|
|
scores = scorer.score_batch(items, context)
|
|
assert scores[0].score > scores[1].score
|
|
assert "550e8400-e29b-41d4-a716-446655440000" in scores[0].matched_terms
|
|
|
|
def test_numeric_id_match(self):
|
|
"""BM25 matches numeric IDs."""
|
|
scorer = BM25Scorer()
|
|
items = [
|
|
'{"user_id": 12345, "name": "Alice"}',
|
|
'{"user_id": 67890, "name": "Bob"}',
|
|
]
|
|
context = "find user 12345"
|
|
|
|
scores = scorer.score_batch(items, context)
|
|
assert scores[0].score > scores[1].score
|
|
|
|
def test_keyword_match(self):
|
|
"""BM25 matches keywords."""
|
|
scorer = BM25Scorer()
|
|
items = [
|
|
'{"status": "error", "message": "Connection refused"}',
|
|
'{"status": "success", "data": [1, 2, 3]}',
|
|
]
|
|
context = "show me errors"
|
|
|
|
scores = scorer.score_batch(items, context)
|
|
# "error" should match "errors" via common stem
|
|
assert scores[0].score >= scores[1].score
|
|
|
|
def test_empty_context(self):
|
|
"""Empty context returns zero scores."""
|
|
scorer = BM25Scorer()
|
|
items = ['{"id": "123"}', '{"id": "456"}']
|
|
context = ""
|
|
|
|
scores = scorer.score_batch(items, context)
|
|
assert all(s.score == 0.0 for s in scores)
|
|
|
|
def test_empty_items(self):
|
|
"""Empty items list returns empty scores."""
|
|
scorer = BM25Scorer()
|
|
scores = scorer.score_batch([], "some context")
|
|
assert scores == []
|
|
|
|
def test_single_item(self):
|
|
"""Single item scoring works."""
|
|
scorer = BM25Scorer()
|
|
item = '{"name": "Alice", "role": "admin"}'
|
|
context = "find Alice"
|
|
|
|
score = scorer.score(item, context)
|
|
assert score.score > 0
|
|
assert "alice" in [t.lower() for t in score.matched_terms]
|
|
|
|
def test_is_available(self):
|
|
"""BM25Scorer is always available."""
|
|
assert BM25Scorer.is_available()
|
|
|
|
def test_compute_idf_follows_standard_formula(self):
|
|
"""IDF rewards rare terms and decays toward zero for common terms."""
|
|
scorer = BM25Scorer()
|
|
|
|
# Absent term contributes nothing.
|
|
assert scorer._compute_idf("x", doc_count=10, doc_freq=0) == 0.0
|
|
|
|
# A term in 1/10 docs is more discriminative than one in 9/10 docs.
|
|
rare = scorer._compute_idf("x", doc_count=10, doc_freq=1)
|
|
common = scorer._compute_idf("x", doc_count=10, doc_freq=9)
|
|
assert rare > common > 0
|
|
|
|
def test_batch_idf_downweights_common_terms(self):
|
|
"""A discriminative term outranks one shared across the whole corpus.
|
|
|
|
``shared`` appears in every item, so its corpus IDF approaches zero,
|
|
while ``zeta`` appears in a single item and stays discriminative. An
|
|
item matched only on the rare term must therefore outrank an item
|
|
matched only on the ubiquitous term.
|
|
"""
|
|
scorer = BM25Scorer()
|
|
items = [
|
|
"shared zeta", # matches both query terms, one of them rare
|
|
"shared alpha", # matches only the ubiquitous term
|
|
"shared beta",
|
|
"shared gamma",
|
|
]
|
|
context = "shared zeta"
|
|
|
|
scores = scorer.score_batch(items, context)
|
|
assert scores[0].score > scores[1].score
|
|
assert scores[0].score > scores[2].score
|
|
|
|
def test_batch_idf_does_not_change_matched_terms(self):
|
|
"""Corpus IDF affects ranking only, not which terms are reported."""
|
|
scorer = BM25Scorer()
|
|
items = ["alpha", "alpha beta"]
|
|
scores = scorer.score_batch(items, "alpha beta")
|
|
assert scores[0].matched_terms == ["alpha"]
|
|
assert sorted(scores[1].matched_terms) == ["alpha", "beta"]
|
|
|
|
def test_precomputed_query_freq_matches_recomputed(self):
|
|
"""Passing a precomputed Counter(query_tokens) into ``_bm25_score``
|
|
must yield identical (score, matched_terms) to letting it recompute
|
|
the Counter itself. ``score_batch`` relies on this to count the shared
|
|
context once instead of once per item."""
|
|
from collections import Counter
|
|
|
|
scorer = BM25Scorer()
|
|
doc_tokens = scorer._tokenize("alpha beta beta gamma id-9c1f2a3b4d5e")
|
|
query_tokens = scorer._tokenize("beta gamma id-9c1f2a3b4d5e delta")
|
|
|
|
recomputed = scorer._bm25_score(doc_tokens, query_tokens)
|
|
with_freq = scorer._bm25_score(doc_tokens, query_tokens, query_freq=Counter(query_tokens))
|
|
assert with_freq == recomputed
|
|
|
|
|
|
class TestEmbeddingScorer:
|
|
"""Tests for embedding-based semantic scorer."""
|
|
|
|
@pytest.fixture
|
|
def scorer(self):
|
|
"""Create embedding scorer if available."""
|
|
if not EmbeddingScorer.is_available():
|
|
pytest.skip("sentence-transformers not installed")
|
|
return EmbeddingScorer()
|
|
|
|
def test_semantic_match(self, scorer):
|
|
"""Embeddings find semantic matches."""
|
|
items = [
|
|
'{"status": "failed", "error": "connection refused"}',
|
|
'{"status": "success", "data": [1, 2, 3]}',
|
|
]
|
|
context = "show me the errors"
|
|
|
|
scores = scorer.score_batch(items, context)
|
|
# "failed"/"error" semantically relates to "errors"
|
|
assert scores[0].score > scores[1].score
|
|
|
|
def test_batches_past_the_cap_keep_each_score_with_its_item(self, scorer):
|
|
"""Three EMBED_BATCH_SIZE batches of mixed lengths, the related item in
|
|
the last: a misaligned batch boundary would hand its score away."""
|
|
items = [
|
|
f"inventory row {i}: " + "pallets of garden furniture " * (i % 5 + 1) for i in range(40)
|
|
]
|
|
items[37] = "authentication failed for user admin: invalid password"
|
|
|
|
scores = [s.score for s in scorer.score_batch(items, "login error")]
|
|
assert len(scores) == 40
|
|
assert max(range(40), key=scores.__getitem__) == 37
|
|
for item, batched in zip(items, scores):
|
|
assert batched == pytest.approx(scorer.score(item, "login error").score, abs=1e-3)
|
|
|
|
def test_paraphrase_match(self, scorer):
|
|
"""Embeddings match paraphrases."""
|
|
items = [
|
|
'{"message": "The server crashed with a fatal error"}',
|
|
'{"message": "The weather today is sunny and warm"}',
|
|
]
|
|
context = "system failure and errors"
|
|
|
|
scores = scorer.score_batch(items, context)
|
|
# "server crashed with fatal error" is much closer to "system failure and errors"
|
|
# than "weather is sunny" - this should be a clear semantic difference
|
|
assert scores[0].score > scores[1].score
|
|
|
|
def test_batch_efficiency(self, scorer):
|
|
"""Batch scoring is efficient."""
|
|
items = [f'{{"id": {i}}}' for i in range(100)]
|
|
context = "find item"
|
|
|
|
# Should not raise and should complete quickly
|
|
scores = scorer.score_batch(items, context)
|
|
assert len(scores) == 100
|
|
|
|
|
|
class TestHybridScorer:
|
|
"""Tests for hybrid BM25 + embedding scorer."""
|
|
|
|
def test_always_available(self):
|
|
"""HybridScorer is always available (falls back to BM25)."""
|
|
assert HybridScorer.is_available()
|
|
|
|
def test_uuid_query_favors_bm25(self):
|
|
"""UUID queries increase BM25 weight."""
|
|
scorer = HybridScorer(adaptive=True)
|
|
|
|
# UUID query
|
|
alpha_uuid = scorer._compute_alpha("find 550e8400-e29b-41d4-a716-446655440000")
|
|
# Semantic query
|
|
alpha_semantic = scorer._compute_alpha("show me recent errors")
|
|
|
|
assert alpha_uuid > alpha_semantic
|
|
assert alpha_uuid >= 0.8 # High BM25 weight for UUIDs
|
|
|
|
def test_numeric_id_query_increases_alpha(self):
|
|
"""Numeric ID queries increase BM25 weight."""
|
|
scorer = HybridScorer(adaptive=True)
|
|
|
|
alpha_with_ids = scorer._compute_alpha("find users 12345 and 67890")
|
|
alpha_no_ids = scorer._compute_alpha("show all users")
|
|
|
|
assert alpha_with_ids > alpha_no_ids
|
|
|
|
def test_fixed_alpha_mode(self):
|
|
"""Fixed alpha mode uses constant weight."""
|
|
scorer = HybridScorer(alpha=0.7, adaptive=False)
|
|
|
|
alpha1 = scorer._compute_alpha("find 550e8400-e29b-41d4-a716-446655440000")
|
|
alpha2 = scorer._compute_alpha("show me errors")
|
|
|
|
assert alpha1 == alpha2 == 0.7
|
|
|
|
def test_fallback_to_bm25(self):
|
|
"""Without embeddings, returns BM25 only."""
|
|
# Create scorer without embeddings
|
|
scorer = HybridScorer()
|
|
scorer._embedding_available = False
|
|
scorer.embedding = None
|
|
|
|
items = ['{"id": "123"}', '{"id": "456"}']
|
|
context = "find 123"
|
|
|
|
scores = scorer.score_batch(items, context)
|
|
assert all("BM25 only" in s.reason for s in scores)
|
|
|
|
def test_hybrid_scoring(self):
|
|
"""Hybrid scoring combines BM25 and embeddings when available."""
|
|
scorer = HybridScorer(adaptive=False, alpha=0.5)
|
|
|
|
if not scorer.has_embedding_support():
|
|
pytest.skip("sentence-transformers not installed")
|
|
|
|
items = ['{"id": "123", "name": "Alice"}']
|
|
context = "find user 123"
|
|
|
|
scores = scorer.score_batch(items, context)
|
|
assert "Hybrid" in scores[0].reason
|
|
assert "BM25=" in scores[0].reason
|
|
|
|
|
|
class TestCreateScorer:
|
|
"""Tests for the create_scorer factory function."""
|
|
|
|
def test_create_bm25(self):
|
|
"""Create BM25 scorer."""
|
|
scorer = create_scorer("bm25")
|
|
assert isinstance(scorer, BM25Scorer)
|
|
|
|
def test_create_bm25_case_insensitive(self):
|
|
"""Tier names are case insensitive."""
|
|
scorer = create_scorer("BM25")
|
|
assert isinstance(scorer, BM25Scorer)
|
|
|
|
def test_create_hybrid(self):
|
|
"""Create hybrid scorer."""
|
|
scorer = create_scorer("hybrid")
|
|
assert isinstance(scorer, HybridScorer)
|
|
|
|
def test_create_embedding_requires_deps(self):
|
|
"""Embedding scorer requires sentence-transformers."""
|
|
if EmbeddingScorer.is_available():
|
|
scorer = create_scorer("embedding")
|
|
assert isinstance(scorer, EmbeddingScorer)
|
|
else:
|
|
with pytest.raises(RuntimeError, match="sentence-transformers"):
|
|
create_scorer("embedding")
|
|
|
|
def test_invalid_tier(self):
|
|
"""Invalid tier raises ValueError."""
|
|
with pytest.raises(ValueError, match="Unknown scorer tier"):
|
|
create_scorer("invalid")
|
|
|
|
def test_pass_kwargs(self):
|
|
"""Kwargs are passed to scorer constructor."""
|
|
scorer = create_scorer("bm25", k1=2.0, b=0.5)
|
|
assert scorer.k1 == 2.0
|
|
assert scorer.b == 0.5
|
|
|
|
|
|
class TestEmbeddingAvailable:
|
|
"""Tests for embedding_available helper."""
|
|
|
|
def test_returns_bool(self):
|
|
"""Returns boolean."""
|
|
result = embedding_available()
|
|
assert isinstance(result, bool)
|
|
|
|
def test_matches_class_method(self):
|
|
"""Matches EmbeddingScorer.is_available()."""
|
|
assert embedding_available() == EmbeddingScorer.is_available()
|
|
|
|
|
|
class TestSmartCrusherIntegration:
|
|
"""Integration tests for SmartCrusher with RelevanceScorer."""
|
|
|
|
def test_context_extraction(self):
|
|
"""Context is extracted from messages."""
|
|
from headroom.transforms.smart_crusher import SmartCrusher, SmartCrusherConfig
|
|
|
|
crusher = SmartCrusher(config=SmartCrusherConfig())
|
|
|
|
messages = [
|
|
{"role": "user", "content": "Find user with ID 12345"},
|
|
{"role": "assistant", "content": "I'll search for that user."},
|
|
]
|
|
|
|
context = crusher._extract_context_from_messages(messages)
|
|
assert "12345" in context
|
|
assert "Find user" in context
|
|
|
|
def test_context_from_anthropic_style_messages(self):
|
|
"""Context extracted from Anthropic-style messages."""
|
|
from headroom.transforms.smart_crusher import SmartCrusher, SmartCrusherConfig
|
|
|
|
crusher = SmartCrusher(config=SmartCrusherConfig())
|
|
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "text", "text": "Search for Alice's records"},
|
|
],
|
|
},
|
|
]
|
|
|
|
context = crusher._extract_context_from_messages(messages)
|
|
assert "Alice" in context
|
|
|
|
def test_context_from_tool_calls(self):
|
|
"""Context extracted from tool call arguments."""
|
|
from headroom.transforms.smart_crusher import SmartCrusher, SmartCrusherConfig
|
|
|
|
crusher = SmartCrusher(config=SmartCrusherConfig())
|
|
|
|
messages = [
|
|
{"role": "user", "content": "Get user info"},
|
|
{
|
|
"role": "assistant",
|
|
"tool_calls": [
|
|
{
|
|
"function": {
|
|
"name": "get_user",
|
|
"arguments": '{"user_id": "550e8400-e29b-41d4"}',
|
|
},
|
|
},
|
|
],
|
|
},
|
|
]
|
|
|
|
context = crusher._extract_context_from_messages(messages)
|
|
assert "550e8400-e29b-41d4" in context
|