fix(memory): bound retrieval queries before embedding (#4244)

## Description

Closes #4223.

The proxy now embeds a bounded, user-first retrieval query instead of
concatenating entire tool outputs into a deadline-limited lookup. The
raw `MemoryQuery` and its default SDK rendering still retain the full
sources.

Memory failures use the exception representation, so an empty
`TimeoutError` is no longer a blank warning. The existing Anthropic
deadline is unchanged and documented; this does not promise that a cold
model or slow embedding server will finish within it.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- Bound proxy embedding input to 4096 characters, including delimiters;
spend the budget on the latest user question and recent context tails.
- Reserve question and context space inside the active local tokenizer's
window, rather than relying on a character cap to prevent truncation.
- Keep the full-fidelity SDK behavior when no budget is requested.
- Make generic memory errors identifiable in the shared handler and
provider handlers.
- Cover Unicode, newest-context precedence, zero character budget and
actual tokenizer-window retention; keep the full-fidelity SDK tests.

## Testing

- [ ] Unit tests pass (pytest) — all 67 affected query, memory-handler
and tokenizer-window regressions passed locally, not the entire Python
suite.
- [ ] Linting passes (ruff check .) — normal changed-file Ruff hooks
passed.
- [ ] Type checking passes (mypy headroom) — normal mypy hook passed.
- [x] Manual testing performed

### Test Output

```text
Affected query/native-memory/tool/auto-tail/tokenizer suites: 67 passed, 1 warning
Ordinary make ci-precheck: PASS (181 critical Python tests, 6 warnings)
```

The focused-suite warning concerns assertion rewriting of the
already-imported `asyncio` name; it was not suppressed. All five
installed pytest plugins were loaded explicitly. Normal Ruff
formatting/checking, mypy and commitlint hooks passed on `8eb10bfd`. The
ordinary full pre-push gate passed on that same commit, including Rust
workspace checks/tests, the native build and critical Python suites; the
normal fast-forward fork push completed without a bypass.

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, the actual local ONNX
model/session, SQLite and vector retrieval. The optional
sentence-transformers CPU backend was also exercised in a separate
complete CPU-wheel environment; no project dependency changes.
- Exact command / steps: save a SQLite-corruption memory and an
unrelated cooking memory through the real `LocalBackend`. Call
`MemoryHandler.search_and_format_context` under a two-second
`asyncio.wait_for` with the same long repeated question and SQLite tool
error before and after the change. Record the real ONNX session's input
IDs/attention mask without replacing inference; decode those IDs to
check the final question and tool error. Verify the original
`MemoryQuery` fields stay intact.
- Observed result: both fragments now reach the model and the relevant
memory is retrieved. Same synthetic input and default cache behavior; no
fake embedding or search result:

| Measurement | Before | After |
| --- | ---: | ---: |
| Submitted characters | 2594 | 1146 |
| Tokens before model truncation | 560 | 256 |
| Question tail and tool error reach model | No | Yes |
| Relevant memory retrieved | No | Yes |
| Warm lookup seconds | 0.076 | 0.241 |

A separately configured 96-token ONNX model also retained both
fragments. The real sentence-transformers CPU model and a previously
uninitialized local ONNX backend passed the same retention check and
generated real embeddings. The legacy messages-only path still submitted
4096 characters; the empty-timeout diagnostic still logged
`TimeoutError()` and failed open.

- Not tested: the reporter's Granite embedding server, cold-start
deadlines or network latency, arbitrary remote tokenizer windows, live
provider requests, Windows/macOS or GPU/MPS. Only the diagnostic failure
was injected; successful persistence, embeddings and retrieval used real
backends.

## Runtime Rollout Safety

- Rollout-managed feature(s): existing automatic memory lookup; no new
feature gate.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: proxy lookups use the bounded
representation; the SDK's default full rendering is unchanged.
- Kill switch / disable path: existing `--no-memory-context` or memory
configuration.
- Unsafe override required: no.
- Qualification impact: actual retrieval, raw-input preservation and
diagnostics checked locally.
- Rollback path: revert the fix; no schema migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it

## Additional Notes

AI-assisted; no subagents.
This commit is contained in:
Mansurshakh Japarov
2026-10-24 08:45:33 -07:00
committed by GitHub
parent 9ac1b52c27
commit 8c07cf4a0f
10 changed files with 354 additions and 57 deletions
+29
View File
@@ -148,6 +148,35 @@ headroom proxy --codex-wire-debug
HEADROOM_COMPRESS_PASSTHROUGH=1 headroom proxy
```
Automatic memory lookup renders a separate, bounded embedding query:
| Input | Maximum characters |
| --- | ---: |
| Entire query, including delimiters | 4096 |
| Latest user message tail | 2048 |
| Each recent tool-output tail | 512 |
| Each recent assistant-turn tail | 512 |
These are source-size limits, not token estimates. The local ONNX and
sentence-transformers embedders then use their actual tokenizer and configured
input window, including special tokens. When both sources are present, they
reserve room for the question and context separately; unused space can go to
the other source. Newest tool evidence is at the end of the context so retaining
its tail does not silently discard it behind a long question.
Remote embedders that do not expose a local tokenizer retain the character
limits; these do not guarantee an arbitrary remote model's token window.
The raw `MemoryQuery` fields stay unchanged, and SDK callers that render without
a budget still receive the full sources.
Anthropic's pre-upstream lookup keeps its existing 2.0-second deadline,
configured by `ProxyConfig.anthropic_pre_upstream_memory_context_timeout_seconds`.
Bounding the input limits embedding work; it is not a guarantee against cold
model initialization or a slow embedding server. Memory failures include the
exception representation, so an otherwise empty timeout is identifiable as
`TimeoutError()`.
<Callout type="info" title="LLMLingua removed from the proxy CLI">
The old LLMLingua proxy toggles are no longer part of the CLI. Headroom's proxy compression path uses ContentRouter plus the current built-in compressors, including Kompress where applicable.
</Callout>
+97
View File
@@ -14,7 +14,9 @@ import asyncio
import logging
import os
import warnings
from collections.abc import Callable
from concurrent.futures import ThreadPoolExecutor
from copy import deepcopy
from functools import cached_property
from typing import TYPE_CHECKING, Any, cast
@@ -26,6 +28,8 @@ from headroom.onnx_runtime import create_cpu_session_options, hf_hub_download_lo
if TYPE_CHECKING:
from sentence_transformers import SentenceTransformer
from tokenizers import Tokenizer
from transformers import PreTrainedTokenizerBase
# Suppress HuggingFace Hub warnings about missing tokens and rate limits.
# These appear whenever hf_hub_download is called without HF_TOKEN set.
@@ -172,6 +176,48 @@ def _normalize_embeddings_batch(embeddings: np.ndarray) -> np.ndarray:
return result
def _fit_query_to_window(
user_text: str,
context: str,
max_tokens: int,
count_tokens: Callable[[str], int],
) -> str:
"""Reserve token space for both sources, retaining their recent tails."""
def tail(text: str, budget: int) -> str:
if budget <= 0:
return ""
if count_tokens(text) <= budget:
return text
low, high = 0, len(text)
while low < high:
middle = (low + high + 1) // 2
if count_tokens(text[-middle:]) <= budget:
low = middle
else:
high = middle - 1
return text[-low:] if low else ""
user_tokens, context_tokens = count_tokens(user_text), count_tokens(context)
available = max(0, max_tokens - (count_tokens("\n") if user_tokens and context_tokens else 0))
user_budget = min(user_tokens, (available + 1) // 2 if context_tokens else available)
context_budget = min(context_tokens, available - user_budget)
user_budget = min(user_tokens, available - context_budget)
while True:
rendered = "\n".join(
part for part in (tail(user_text, user_budget), tail(context, context_budget)) if part
)
excess = count_tokens(rendered) - max_tokens
if excess <= 0:
return rendered
# Retokenizing a suffix can change its first subword. Enforce the final
# joined window too, rather than assuming token counts are additive.
if user_budget >= context_budget and user_budget:
user_budget = max(0, user_budget - excess)
else:
context_budget = max(0, context_budget - excess)
# =============================================================================
# LocalEmbedder - sentence-transformers
# =============================================================================
@@ -219,6 +265,7 @@ class LocalEmbedder:
self._model_name = model_name or ML_MODEL_DEFAULTS.sentence_transformer
self._requested_device = device
self._model: SentenceTransformer | None = None
self._query_tokenizer: PreTrainedTokenizerBase | None = None
self._device: str | None = None
self._dimension: int | None = None
self._lock = asyncio.Lock()
@@ -328,6 +375,30 @@ class LocalEmbedder:
return _normalize_embedding(embedding)
async def prepare_query(self, user_text: str, context: str) -> str:
"""Fit both query sources with the configured model's tokenizer."""
async with self._lock:
if self._model is None:
await asyncio.get_event_loop().run_in_executor(None, self._load_model)
assert self._model is not None
if self._query_tokenizer is None:
self._query_tokenizer = deepcopy(self._model.tokenizer)
# Configure the private copy before concurrent query encodes.
self._query_tokenizer.encode("", add_special_tokens=False, truncation=False)
tokenizer = self._query_tokenizer
available = max(0, self._model.max_seq_length - tokenizer.num_special_tokens_to_add())
return await asyncio.get_event_loop().run_in_executor(
self._executor,
lambda: _fit_query_to_window(
user_text,
context,
available,
lambda text: len(
tokenizer.encode(text, add_special_tokens=False, truncation=False)
),
),
)
async def embed_batch(self, texts: list[str]) -> list[np.ndarray]:
"""Generate embeddings for multiple texts.
@@ -407,6 +478,7 @@ class LocalEmbedder:
self._executor.shutdown(wait=False)
self._executor = None
self._model = None
self._query_tokenizer = None
# =============================================================================
@@ -437,6 +509,7 @@ class OnnxLocalEmbedder:
self._max_length = max_length
self._session: Any = None
self._tokenizer: Any = None
self._query_tokenizer: Tokenizer | None = None
self._input_names: list[str] = []
self._lock = asyncio.Lock()
@@ -543,6 +616,29 @@ class OnnxLocalEmbedder:
embedding = await loop.run_in_executor(None, self._embed_single, text)
return cast(np.ndarray, embedding)
async def prepare_query(self, user_text: str, context: str) -> str:
"""Fit both query sources without mutating the inference tokenizer."""
from tokenizers import Tokenizer
async with self._lock:
if self._session is None:
await asyncio.get_event_loop().run_in_executor(None, self._load_model)
if self._query_tokenizer is None:
self._query_tokenizer = Tokenizer.from_str(self._tokenizer.to_str())
self._query_tokenizer.no_truncation()
self._query_tokenizer.no_padding()
tokenizer = self._query_tokenizer
available = max(0, self._max_length - tokenizer.num_special_tokens_to_add(False))
return await asyncio.get_event_loop().run_in_executor(
None,
lambda: _fit_query_to_window(
user_text,
context,
available,
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
),
)
async def embed_batch(self, texts: list[str]) -> list[np.ndarray]:
"""Generate embeddings for multiple texts."""
if not texts:
@@ -591,6 +687,7 @@ class OnnxLocalEmbedder:
"""Close resources."""
self._session = None
self._tokenizer = None
self._query_tokenizer = None
# =============================================================================
+14
View File
@@ -16,6 +16,7 @@ from __future__ import annotations
import asyncio
import logging
import uuid
from collections.abc import Awaitable, Callable
from dataclasses import dataclass, field
from pathlib import Path
from typing import TYPE_CHECKING, Any
@@ -405,6 +406,19 @@ class LocalBackend:
# Return primary memory
return primary_memory
async def prepare_search_query(self, user_text: str, context: str) -> str:
"""Apply the active local model's query window, including on cold backends."""
await self._ensure_initialized()
assert self._hierarchical_memory is not None
prepare: Callable[[str, str], Awaitable[str]] | None = getattr(
self._hierarchical_memory.embedder, "prepare_query", None
)
if prepare is not None:
return await prepare(user_text, context)
# Remote embedders do not expose their tokenizer; callers still supply
# character-bounded sources, without claiming a remote token guarantee.
return "\n".join(part for part in (user_text, context) if part)
async def search_memories(
self,
query: str,
+1 -1
View File
@@ -3030,7 +3030,7 @@ class AnthropicHandlerMixin:
query=user_query,
)
except Exception as e:
logger.warning(f"[{request_id}] Memory: Context injection failed: {e}")
logger.warning(f"[{request_id}] Memory: Context injection failed: {e!r}")
# Inject memory tools — PR-A7 (P0-6) routes through
# `apply_session_sticky_memory_tools` so tool list bytes
+3 -3
View File
@@ -4416,7 +4416,7 @@ class OpenAIHandlerMixin:
memory_tools_injected = True
logger.info(f"[{request_id}] Memory: Injected memory tools (openai)")
except Exception as e:
logger.warning(f"[{request_id}] Memory injection failed: {e}")
logger.warning(f"[{request_id}] Memory injection failed: {e!r}")
if memory_context_injected or memory_tools_injected:
remembered_event = self.pipeline_extensions.emit(
@@ -6311,7 +6311,7 @@ class OpenAIHandlerMixin:
f"[{request_id}] Memory: Injected memory tools (openai/responses)"
)
except Exception as e:
logger.warning(f"[{request_id}] Memory injection failed (responses): {e}")
logger.warning(f"[{request_id}] Memory injection failed (responses): {e!r}")
elif self.memory_handler and memory_user_id and _bypass:
logger.info(
"[%s] Responses memory passthrough reason=bypass_header",
@@ -8353,7 +8353,7 @@ class OpenAIHandlerMixin:
return json.dumps(frame_body)
except Exception as e:
logger.warning(f"[{request_id}] WS Memory injection failed: {e}")
logger.warning(f"[{request_id}] WS Memory injection failed: {e!r}")
return frame_raw
if first_frame_is_create:
+19 -17
View File
@@ -800,18 +800,6 @@ class MemoryHandler:
)
return None
# Build the embedding query. When the handler provides a
# MemoryQuery, use its multi-source untruncated input; otherwise
# fall back to extracting from messages (kept for legacy callers
# / tests). Full fidelity in both paths.
if query is not None:
query_text = query.to_embedding_input()
else:
query_text = self._extract_user_query(messages)
if not query_text:
logger.debug("Memory: No query text for context search")
return None
# Compose the budget: explicit per-call wins; otherwise derive
# from self.config so existing top_k/min_similarity callers see
# no behaviour change.
@@ -825,6 +813,22 @@ class MemoryHandler:
)
try:
if query is not None:
from headroom.proxy.memory_query_policy import bounded_embedding_parts
parts = bounded_embedding_parts(
user_text=query.user_text,
recent_tool_outputs=query.recent_tool_outputs,
recent_assistant_turns=query.recent_assistant_turns,
)
prepare = getattr(backend, "prepare_search_query", None)
query_text = await prepare(*parts) if prepare is not None else "".join(parts)
else:
query_text = self._extract_user_query(messages)[-4096:]
if not query_text:
logger.debug("Memory: No query text for context search")
return None
# Search memories on the per-request resolved backend.
results = await backend.search_memories(
query=query_text,
@@ -871,7 +875,7 @@ class MemoryHandler:
memory_lines.append(f" (Related: {entities_str})")
except Exception as e:
logger.warning(f"Memory: Search failed for user {effective_user_id}: {e}")
logger.warning(f"Memory: Search failed for user {effective_user_id}: {e!r}")
return None
if not memory_lines:
@@ -904,10 +908,8 @@ pass that ID directly to memory_update or memory_delete — you do not need \
to call memory_search first to discover IDs. Use this context to inform \
your responses, not to drive new actions."""
# Apply the token-budget cap on the formatted block. Pre-this-
# PR there was no cap — up to ~4000 tokens could be injected
# per request. The budget bounds the output without touching
# the input query (which stays full-fidelity per MemoryQuery).
# The formatted output budget is separate from the embedding-input
# budget above; the original MemoryQuery remains full-fidelity.
context = effective_budget.apply_to_text(context)
# Track only memories that survived ranking, entry limits, and the
+16 -20
View File
@@ -1,10 +1,8 @@
"""``MemoryQuery``: multi-source, full-fidelity retrieval query.
Pre-this-PR, the retrieval query was "latest user message, truncated
to 500 chars" (memory_handler.py:807). The truncation was a real bug
— none of Letta / Mem0 / Cognee / Supermemory truncate the embedding
input. Tool outputs are often the strongest retrieval signal in
coding sessions, and they were ignored entirely.
Raw user messages and recent tool/assistant turns are retained for retrieval
strategies that need their full content. Time-limited proxy lookups can render
a separate bounded embedding input without mutating those sources.
This value type captures the query at full fidelity from three
sources:
@@ -13,10 +11,8 @@ sources:
* ``recent_tool_outputs`` — last N tool results
* ``recent_assistant_turns`` — last K assistant turns for intent
The embedding model handles its own context window (MiniLM 512 tok;
BGE-small 8K tok). Long inputs that exceed the model window become
the model's problem to mean-pool or chunk — they don't get
truncated upstream.
The default embedding representation preserves the full sources. A caller can
request a character budget when embedding latency must fit a request deadline.
"""
from __future__ import annotations
@@ -34,11 +30,9 @@ from .memory_query_policy import (
class MemoryQuery:
"""Frozen multi-source query for memory retrieval.
All fields preserve full input fidelity — no truncation, no
summarization. The caller assembles the sources; this type only
holds them. The retrieval backend decides how to embed
(mean-pool, chunk, model-side truncation, etc.) but cannot lose
information before it sees the data.
Raw fields preserve full input fidelity. Budgeted embedding rendering
produces a derived query without changing those fields; backends that
chunk or mean-pool full inputs can still use the original sources.
Tuples for the recent-* fields so the dataclass stays hashable
(frozen + value-equal).
@@ -49,18 +43,20 @@ class MemoryQuery:
recent_assistant_turns: tuple[str, ...]
conversation_id: str | None
def to_embedding_input(self) -> str:
"""Concatenate sources into a delimited embedding input.
def to_embedding_input(self, *, max_chars: int | None = None) -> str:
"""Render delimited embedding input, optionally within a character budget.
Order: prior assistant turns (oldest first) → tool outputs
(oldest first) → latest user text. User text last because the
embedder's positional weighting often emphasizes the tail of
the input.
By default, context is chronological and user text comes last.
With a character budget, user text comes first and recent context is
selected newest-first, then rendered chronologically so a tokenizer
can retain its newest tail. Proxy lookups separately reserve space
for both sources inside the active local model's token window.
"""
return render_embedding_input(
user_text=self.user_text,
recent_tool_outputs=self.recent_tool_outputs,
recent_assistant_turns=self.recent_assistant_turns,
max_chars=max_chars,
)
@classmethod
+50 -10
View File
@@ -17,18 +17,58 @@ def render_embedding_input(
user_text: str,
recent_tool_outputs: tuple[str, ...],
recent_assistant_turns: tuple[str, ...],
max_chars: int | None = None,
) -> str:
"""Concatenate memory query sources into a delimited embedding input."""
"""Render full sources, or a user-first query within an embedding budget."""
parts: list[str] = []
for asst in recent_assistant_turns:
if asst:
parts.append(ASSISTANT_DELIM + asst)
for tool_out in recent_tool_outputs:
if tool_out:
parts.append(TOOL_DELIM + tool_out)
if user_text:
parts.append(USER_DELIM + user_text)
return "".join(parts)
if max_chars is None:
for asst in recent_assistant_turns:
if asst:
parts.append(ASSISTANT_DELIM + asst)
for tool_out in recent_tool_outputs:
if tool_out:
parts.append(TOOL_DELIM + tool_out)
if user_text:
parts.append(USER_DELIM + user_text)
return "".join(parts)
return "".join(
bounded_embedding_parts(
user_text=user_text,
recent_tool_outputs=recent_tool_outputs,
recent_assistant_turns=recent_assistant_turns,
max_chars=max_chars,
)
)
def bounded_embedding_parts(
*,
user_text: str,
recent_tool_outputs: tuple[str, ...],
recent_assistant_turns: tuple[str, ...],
max_chars: int = 4096,
) -> tuple[str, str]:
"""Keep question and context separate so a tokenizer can budget both."""
remaining = max(0, max_chars)
user_chars = min(2048, max(0, remaining - len(USER_DELIM)))
question = USER_DELIM + user_text[-user_chars:] if user_text and user_chars else ""
remaining -= len(question) + bool(question)
parts: list[str] = []
for delimiter, sources in (
(TOOL_DELIM, recent_tool_outputs),
(ASSISTANT_DELIM, recent_assistant_turns),
):
for source in reversed(sources):
if not source:
continue
source_chars = min(512, remaining - len(delimiter))
if source_chars <= 0:
break
part = delimiter + source[-source_chars:]
parts.append(part)
remaining -= len(part)
return question, "".join(reversed(parts))
def extract_memory_query_sources(
+83
View File
@@ -0,0 +1,83 @@
"""Query sources must survive the actual tokenizer's finite input window."""
from tokenizers import Tokenizer
from tokenizers.models import WordPiece
from tokenizers.normalizers import BertNormalizer
from tokenizers.pre_tokenizers import BertPreTokenizer
from tokenizers.processors import TemplateProcessing
from headroom.memory.adapters.embedders import _fit_query_to_window
def make_tokenizer() -> Tokenizer:
words = [
"[UNK]",
"[CLS]",
"[SEP]",
"[PAD]",
"question",
"answer",
"tool",
"database",
"corruption",
"古",
"い",
"質",
"問",
"出",
"力",
"答",
"案",
"破",
"損",
"故",
"障",
]
vocab = {word: index for index, word in enumerate(words)}
tokenizer = Tokenizer(WordPiece(vocab, unk_token="[UNK]"))
tokenizer.normalizer = BertNormalizer(lowercase=True)
tokenizer.pre_tokenizer = BertPreTokenizer()
tokenizer.post_processor = TemplateProcessing(
single="[CLS] $A [SEP]",
special_tokens=[("[CLS]", vocab["[CLS]"]), ("[SEP]", vocab["[SEP]"])],
)
return tokenizer
def test_long_question_cannot_consume_the_tool_evidence_window() -> None:
tokenizer = make_tokenizer()
question = "question " * 300 + "answer"
context = "tool " * 300 + "database corruption"
rendered = _fit_query_to_window(
question,
context,
14,
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
)
inference = Tokenizer.from_str(tokenizer.to_str())
inference.enable_truncation(max_length=16)
ids = inference.encode(rendered).ids
assert len(tokenizer.encode(rendered).ids) <= 16
assert tokenizer.token_to_id("answer") in ids
assert tokenizer.token_to_id("database") in ids
assert tokenizer.token_to_id("corruption") in ids
def test_unicode_sources_are_budgeted_by_tokens_not_python_characters() -> None:
tokenizer = make_tokenizer()
question = "古い質問" * 100 + "答案"
context = "古い出力" * 100 + "破損故障"
rendered = _fit_query_to_window(
question,
context,
14,
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
)
inference = Tokenizer.from_str(tokenizer.to_str())
inference.enable_truncation(max_length=16)
ids = inference.encode(rendered).ids
assert len(tokenizer.encode(rendered).ids) <= 16
for character in "答案破損故障":
assert tokenizer.token_to_id(character) in ids
+42 -6
View File
@@ -3,20 +3,56 @@
from __future__ import annotations
from headroom.proxy.memory_query_policy import (
ASSISTANT_DELIM,
TOOL_DELIM,
USER_DELIM,
extract_memory_query_sources,
render_embedding_input,
)
def test_render_embedding_input_orders_sources_for_embedding() -> None:
def test_bounded_embedding_keeps_primary_question_and_context_tails() -> None:
user = "古い質問" * 2000 + " latest JWT question"
tool = "old tool output " * 1000 + " latest authentication failure"
assistant = "old assistant context " * 1000 + " inspect auth.py"
rendered = render_embedding_input(
user_text="latest user",
recent_tool_outputs=("tool output",),
recent_assistant_turns=("assistant context",),
user_text=user,
recent_tool_outputs=(tool,),
recent_assistant_turns=(assistant,),
max_chars=4096,
)
assert rendered.index("assistant context") < rendered.index("tool output")
assert rendered.index("tool output") < rendered.index("latest user")
assert len(rendered) <= 4096
assert rendered.startswith(USER_DELIM + user[-2048:])
assert TOOL_DELIM + tool[-512:] in rendered
assert ASSISTANT_DELIM + assistant[-512:] in rendered
def test_bounded_embedding_spends_remaining_budget_on_newest_context() -> None:
rendered = render_embedding_input(
user_text="latest question",
recent_tool_outputs=tuple("x" * 600 + f" end-{i}" for i in range(10)),
recent_assistant_turns=("old background",),
max_chars=1200,
)
assert len(rendered) <= 1200
assert rendered.startswith(USER_DELIM + "latest question")
assert "end-9" in rendered and "end-8" in rendered
assert "end-0" not in rendered
assert "old background" not in rendered
def test_zero_embedding_budget_does_not_send_context_without_the_question() -> None:
assert (
render_embedding_input(
user_text="question",
recent_tool_outputs=("tool context",),
recent_assistant_turns=("assistant context",),
max_chars=0,
)
== ""
)
def test_extract_sources_uses_latest_user_and_recent_context_in_order() -> None: