mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(memory): bound retrieval queries before embedding (#4244)
## Description Closes #4223. The proxy now embeds a bounded, user-first retrieval query instead of concatenating entire tool outputs into a deadline-limited lookup. The raw `MemoryQuery` and its default SDK rendering still retain the full sources. Memory failures use the exception representation, so an empty `TimeoutError` is no longer a blank warning. The existing Anthropic deadline is unchanged and documented; this does not promise that a cold model or slow embedding server will finish within it. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update ## Changes Made - Bound proxy embedding input to 4096 characters, including delimiters; spend the budget on the latest user question and recent context tails. - Reserve question and context space inside the active local tokenizer's window, rather than relying on a character cap to prevent truncation. - Keep the full-fidelity SDK behavior when no budget is requested. - Make generic memory errors identifiable in the shared handler and provider handlers. - Cover Unicode, newest-context precedence, zero character budget and actual tokenizer-window retention; keep the full-fidelity SDK tests. ## Testing - [ ] Unit tests pass (pytest) — all 67 affected query, memory-handler and tokenizer-window regressions passed locally, not the entire Python suite. - [ ] Linting passes (ruff check .) — normal changed-file Ruff hooks passed. - [ ] Type checking passes (mypy headroom) — normal mypy hook passed. - [x] Manual testing performed ### Test Output ```text Affected query/native-memory/tool/auto-tail/tokenizer suites: 67 passed, 1 warning Ordinary make ci-precheck: PASS (181 critical Python tests, 6 warnings) ``` The focused-suite warning concerns assertion rewriting of the already-imported `asyncio` name; it was not suppressed. All five installed pytest plugins were loaded explicitly. Normal Ruff formatting/checking, mypy and commitlint hooks passed on `8eb10bfd`. The ordinary full pre-push gate passed on that same commit, including Rust workspace checks/tests, the native build and critical Python suites; the normal fast-forward fork push completed without a bypass. ## Real Behavior Proof - Environment: Linux x86-64, Python 3.13.14, the actual local ONNX model/session, SQLite and vector retrieval. The optional sentence-transformers CPU backend was also exercised in a separate complete CPU-wheel environment; no project dependency changes. - Exact command / steps: save a SQLite-corruption memory and an unrelated cooking memory through the real `LocalBackend`. Call `MemoryHandler.search_and_format_context` under a two-second `asyncio.wait_for` with the same long repeated question and SQLite tool error before and after the change. Record the real ONNX session's input IDs/attention mask without replacing inference; decode those IDs to check the final question and tool error. Verify the original `MemoryQuery` fields stay intact. - Observed result: both fragments now reach the model and the relevant memory is retrieved. Same synthetic input and default cache behavior; no fake embedding or search result: | Measurement | Before | After | | --- | ---: | ---: | | Submitted characters | 2594 | 1146 | | Tokens before model truncation | 560 | 256 | | Question tail and tool error reach model | No | Yes | | Relevant memory retrieved | No | Yes | | Warm lookup seconds | 0.076 | 0.241 | A separately configured 96-token ONNX model also retained both fragments. The real sentence-transformers CPU model and a previously uninitialized local ONNX backend passed the same retention check and generated real embeddings. The legacy messages-only path still submitted 4096 characters; the empty-timeout diagnostic still logged `TimeoutError()` and failed open. - Not tested: the reporter's Granite embedding server, cold-start deadlines or network latency, arbitrary remote tokenizer windows, live provider requests, Windows/macOS or GPU/MPS. Only the diagnostic failure was injected; successful persistence, embeddings and retrieval used real backends. ## Runtime Rollout Safety - Rollout-managed feature(s): existing automatic memory lookup; no new feature gate. - Minimum rollout channel: unchanged. - Stable/default behavior changed: proxy lookups use the bounded representation; the SDK's default full rendering is unchanged. - Kill switch / disable path: existing `--no-memory-context` or memory configuration. - Unsafe override required: no. - Qualification impact: actual retrieval, raw-input preservation and diagnostics checked locally. - Rollback path: revert the fix; no schema migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I did not edit CHANGELOG.md — release-please owns it ## Additional Notes AI-assisted; no subagents.
This commit is contained in:
@@ -148,6 +148,35 @@ headroom proxy --codex-wire-debug
|
||||
HEADROOM_COMPRESS_PASSTHROUGH=1 headroom proxy
|
||||
```
|
||||
|
||||
Automatic memory lookup renders a separate, bounded embedding query:
|
||||
|
||||
| Input | Maximum characters |
|
||||
| --- | ---: |
|
||||
| Entire query, including delimiters | 4096 |
|
||||
| Latest user message tail | 2048 |
|
||||
| Each recent tool-output tail | 512 |
|
||||
| Each recent assistant-turn tail | 512 |
|
||||
|
||||
These are source-size limits, not token estimates. The local ONNX and
|
||||
sentence-transformers embedders then use their actual tokenizer and configured
|
||||
input window, including special tokens. When both sources are present, they
|
||||
reserve room for the question and context separately; unused space can go to
|
||||
the other source. Newest tool evidence is at the end of the context so retaining
|
||||
its tail does not silently discard it behind a long question.
|
||||
|
||||
Remote embedders that do not expose a local tokenizer retain the character
|
||||
limits; these do not guarantee an arbitrary remote model's token window.
|
||||
The raw `MemoryQuery` fields stay unchanged, and SDK callers that render without
|
||||
a budget still receive the full sources.
|
||||
|
||||
Anthropic's pre-upstream lookup keeps its existing 2.0-second deadline,
|
||||
configured by `ProxyConfig.anthropic_pre_upstream_memory_context_timeout_seconds`.
|
||||
Bounding the input limits embedding work; it is not a guarantee against cold
|
||||
model initialization or a slow embedding server. Memory failures include the
|
||||
exception representation, so an otherwise empty timeout is identifiable as
|
||||
`TimeoutError()`.
|
||||
|
||||
|
||||
<Callout type="info" title="LLMLingua removed from the proxy CLI">
|
||||
The old LLMLingua proxy toggles are no longer part of the CLI. Headroom's proxy compression path uses ContentRouter plus the current built-in compressors, including Kompress where applicable.
|
||||
</Callout>
|
||||
|
||||
@@ -14,7 +14,9 @@ import asyncio
|
||||
import logging
|
||||
import os
|
||||
import warnings
|
||||
from collections.abc import Callable
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from copy import deepcopy
|
||||
from functools import cached_property
|
||||
from typing import TYPE_CHECKING, Any, cast
|
||||
|
||||
@@ -26,6 +28,8 @@ from headroom.onnx_runtime import create_cpu_session_options, hf_hub_download_lo
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
from tokenizers import Tokenizer
|
||||
from transformers import PreTrainedTokenizerBase
|
||||
|
||||
# Suppress HuggingFace Hub warnings about missing tokens and rate limits.
|
||||
# These appear whenever hf_hub_download is called without HF_TOKEN set.
|
||||
@@ -172,6 +176,48 @@ def _normalize_embeddings_batch(embeddings: np.ndarray) -> np.ndarray:
|
||||
return result
|
||||
|
||||
|
||||
def _fit_query_to_window(
|
||||
user_text: str,
|
||||
context: str,
|
||||
max_tokens: int,
|
||||
count_tokens: Callable[[str], int],
|
||||
) -> str:
|
||||
"""Reserve token space for both sources, retaining their recent tails."""
|
||||
|
||||
def tail(text: str, budget: int) -> str:
|
||||
if budget <= 0:
|
||||
return ""
|
||||
if count_tokens(text) <= budget:
|
||||
return text
|
||||
low, high = 0, len(text)
|
||||
while low < high:
|
||||
middle = (low + high + 1) // 2
|
||||
if count_tokens(text[-middle:]) <= budget:
|
||||
low = middle
|
||||
else:
|
||||
high = middle - 1
|
||||
return text[-low:] if low else ""
|
||||
|
||||
user_tokens, context_tokens = count_tokens(user_text), count_tokens(context)
|
||||
available = max(0, max_tokens - (count_tokens("\n") if user_tokens and context_tokens else 0))
|
||||
user_budget = min(user_tokens, (available + 1) // 2 if context_tokens else available)
|
||||
context_budget = min(context_tokens, available - user_budget)
|
||||
user_budget = min(user_tokens, available - context_budget)
|
||||
while True:
|
||||
rendered = "\n".join(
|
||||
part for part in (tail(user_text, user_budget), tail(context, context_budget)) if part
|
||||
)
|
||||
excess = count_tokens(rendered) - max_tokens
|
||||
if excess <= 0:
|
||||
return rendered
|
||||
# Retokenizing a suffix can change its first subword. Enforce the final
|
||||
# joined window too, rather than assuming token counts are additive.
|
||||
if user_budget >= context_budget and user_budget:
|
||||
user_budget = max(0, user_budget - excess)
|
||||
else:
|
||||
context_budget = max(0, context_budget - excess)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# LocalEmbedder - sentence-transformers
|
||||
# =============================================================================
|
||||
@@ -219,6 +265,7 @@ class LocalEmbedder:
|
||||
self._model_name = model_name or ML_MODEL_DEFAULTS.sentence_transformer
|
||||
self._requested_device = device
|
||||
self._model: SentenceTransformer | None = None
|
||||
self._query_tokenizer: PreTrainedTokenizerBase | None = None
|
||||
self._device: str | None = None
|
||||
self._dimension: int | None = None
|
||||
self._lock = asyncio.Lock()
|
||||
@@ -328,6 +375,30 @@ class LocalEmbedder:
|
||||
|
||||
return _normalize_embedding(embedding)
|
||||
|
||||
async def prepare_query(self, user_text: str, context: str) -> str:
|
||||
"""Fit both query sources with the configured model's tokenizer."""
|
||||
async with self._lock:
|
||||
if self._model is None:
|
||||
await asyncio.get_event_loop().run_in_executor(None, self._load_model)
|
||||
assert self._model is not None
|
||||
if self._query_tokenizer is None:
|
||||
self._query_tokenizer = deepcopy(self._model.tokenizer)
|
||||
# Configure the private copy before concurrent query encodes.
|
||||
self._query_tokenizer.encode("", add_special_tokens=False, truncation=False)
|
||||
tokenizer = self._query_tokenizer
|
||||
available = max(0, self._model.max_seq_length - tokenizer.num_special_tokens_to_add())
|
||||
return await asyncio.get_event_loop().run_in_executor(
|
||||
self._executor,
|
||||
lambda: _fit_query_to_window(
|
||||
user_text,
|
||||
context,
|
||||
available,
|
||||
lambda text: len(
|
||||
tokenizer.encode(text, add_special_tokens=False, truncation=False)
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
async def embed_batch(self, texts: list[str]) -> list[np.ndarray]:
|
||||
"""Generate embeddings for multiple texts.
|
||||
|
||||
@@ -407,6 +478,7 @@ class LocalEmbedder:
|
||||
self._executor.shutdown(wait=False)
|
||||
self._executor = None
|
||||
self._model = None
|
||||
self._query_tokenizer = None
|
||||
|
||||
|
||||
# =============================================================================
|
||||
@@ -437,6 +509,7 @@ class OnnxLocalEmbedder:
|
||||
self._max_length = max_length
|
||||
self._session: Any = None
|
||||
self._tokenizer: Any = None
|
||||
self._query_tokenizer: Tokenizer | None = None
|
||||
self._input_names: list[str] = []
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
@@ -543,6 +616,29 @@ class OnnxLocalEmbedder:
|
||||
embedding = await loop.run_in_executor(None, self._embed_single, text)
|
||||
return cast(np.ndarray, embedding)
|
||||
|
||||
async def prepare_query(self, user_text: str, context: str) -> str:
|
||||
"""Fit both query sources without mutating the inference tokenizer."""
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
async with self._lock:
|
||||
if self._session is None:
|
||||
await asyncio.get_event_loop().run_in_executor(None, self._load_model)
|
||||
if self._query_tokenizer is None:
|
||||
self._query_tokenizer = Tokenizer.from_str(self._tokenizer.to_str())
|
||||
self._query_tokenizer.no_truncation()
|
||||
self._query_tokenizer.no_padding()
|
||||
tokenizer = self._query_tokenizer
|
||||
available = max(0, self._max_length - tokenizer.num_special_tokens_to_add(False))
|
||||
return await asyncio.get_event_loop().run_in_executor(
|
||||
None,
|
||||
lambda: _fit_query_to_window(
|
||||
user_text,
|
||||
context,
|
||||
available,
|
||||
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
|
||||
),
|
||||
)
|
||||
|
||||
async def embed_batch(self, texts: list[str]) -> list[np.ndarray]:
|
||||
"""Generate embeddings for multiple texts."""
|
||||
if not texts:
|
||||
@@ -591,6 +687,7 @@ class OnnxLocalEmbedder:
|
||||
"""Close resources."""
|
||||
self._session = None
|
||||
self._tokenizer = None
|
||||
self._query_tokenizer = None
|
||||
|
||||
|
||||
# =============================================================================
|
||||
|
||||
@@ -16,6 +16,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import logging
|
||||
import uuid
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
@@ -405,6 +406,19 @@ class LocalBackend:
|
||||
# Return primary memory
|
||||
return primary_memory
|
||||
|
||||
async def prepare_search_query(self, user_text: str, context: str) -> str:
|
||||
"""Apply the active local model's query window, including on cold backends."""
|
||||
await self._ensure_initialized()
|
||||
assert self._hierarchical_memory is not None
|
||||
prepare: Callable[[str, str], Awaitable[str]] | None = getattr(
|
||||
self._hierarchical_memory.embedder, "prepare_query", None
|
||||
)
|
||||
if prepare is not None:
|
||||
return await prepare(user_text, context)
|
||||
# Remote embedders do not expose their tokenizer; callers still supply
|
||||
# character-bounded sources, without claiming a remote token guarantee.
|
||||
return "\n".join(part for part in (user_text, context) if part)
|
||||
|
||||
async def search_memories(
|
||||
self,
|
||||
query: str,
|
||||
|
||||
@@ -3030,7 +3030,7 @@ class AnthropicHandlerMixin:
|
||||
query=user_query,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"[{request_id}] Memory: Context injection failed: {e}")
|
||||
logger.warning(f"[{request_id}] Memory: Context injection failed: {e!r}")
|
||||
|
||||
# Inject memory tools — PR-A7 (P0-6) routes through
|
||||
# `apply_session_sticky_memory_tools` so tool list bytes
|
||||
|
||||
@@ -4416,7 +4416,7 @@ class OpenAIHandlerMixin:
|
||||
memory_tools_injected = True
|
||||
logger.info(f"[{request_id}] Memory: Injected memory tools (openai)")
|
||||
except Exception as e:
|
||||
logger.warning(f"[{request_id}] Memory injection failed: {e}")
|
||||
logger.warning(f"[{request_id}] Memory injection failed: {e!r}")
|
||||
|
||||
if memory_context_injected or memory_tools_injected:
|
||||
remembered_event = self.pipeline_extensions.emit(
|
||||
@@ -6311,7 +6311,7 @@ class OpenAIHandlerMixin:
|
||||
f"[{request_id}] Memory: Injected memory tools (openai/responses)"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"[{request_id}] Memory injection failed (responses): {e}")
|
||||
logger.warning(f"[{request_id}] Memory injection failed (responses): {e!r}")
|
||||
elif self.memory_handler and memory_user_id and _bypass:
|
||||
logger.info(
|
||||
"[%s] Responses memory passthrough reason=bypass_header",
|
||||
@@ -8353,7 +8353,7 @@ class OpenAIHandlerMixin:
|
||||
|
||||
return json.dumps(frame_body)
|
||||
except Exception as e:
|
||||
logger.warning(f"[{request_id}] WS Memory injection failed: {e}")
|
||||
logger.warning(f"[{request_id}] WS Memory injection failed: {e!r}")
|
||||
return frame_raw
|
||||
|
||||
if first_frame_is_create:
|
||||
|
||||
@@ -800,18 +800,6 @@ class MemoryHandler:
|
||||
)
|
||||
return None
|
||||
|
||||
# Build the embedding query. When the handler provides a
|
||||
# MemoryQuery, use its multi-source untruncated input; otherwise
|
||||
# fall back to extracting from messages (kept for legacy callers
|
||||
# / tests). Full fidelity in both paths.
|
||||
if query is not None:
|
||||
query_text = query.to_embedding_input()
|
||||
else:
|
||||
query_text = self._extract_user_query(messages)
|
||||
if not query_text:
|
||||
logger.debug("Memory: No query text for context search")
|
||||
return None
|
||||
|
||||
# Compose the budget: explicit per-call wins; otherwise derive
|
||||
# from self.config so existing top_k/min_similarity callers see
|
||||
# no behaviour change.
|
||||
@@ -825,6 +813,22 @@ class MemoryHandler:
|
||||
)
|
||||
|
||||
try:
|
||||
if query is not None:
|
||||
from headroom.proxy.memory_query_policy import bounded_embedding_parts
|
||||
|
||||
parts = bounded_embedding_parts(
|
||||
user_text=query.user_text,
|
||||
recent_tool_outputs=query.recent_tool_outputs,
|
||||
recent_assistant_turns=query.recent_assistant_turns,
|
||||
)
|
||||
prepare = getattr(backend, "prepare_search_query", None)
|
||||
query_text = await prepare(*parts) if prepare is not None else "".join(parts)
|
||||
else:
|
||||
query_text = self._extract_user_query(messages)[-4096:]
|
||||
if not query_text:
|
||||
logger.debug("Memory: No query text for context search")
|
||||
return None
|
||||
|
||||
# Search memories on the per-request resolved backend.
|
||||
results = await backend.search_memories(
|
||||
query=query_text,
|
||||
@@ -871,7 +875,7 @@ class MemoryHandler:
|
||||
memory_lines.append(f" (Related: {entities_str})")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Memory: Search failed for user {effective_user_id}: {e}")
|
||||
logger.warning(f"Memory: Search failed for user {effective_user_id}: {e!r}")
|
||||
return None
|
||||
|
||||
if not memory_lines:
|
||||
@@ -904,10 +908,8 @@ pass that ID directly to memory_update or memory_delete — you do not need \
|
||||
to call memory_search first to discover IDs. Use this context to inform \
|
||||
your responses, not to drive new actions."""
|
||||
|
||||
# Apply the token-budget cap on the formatted block. Pre-this-
|
||||
# PR there was no cap — up to ~4000 tokens could be injected
|
||||
# per request. The budget bounds the output without touching
|
||||
# the input query (which stays full-fidelity per MemoryQuery).
|
||||
# The formatted output budget is separate from the embedding-input
|
||||
# budget above; the original MemoryQuery remains full-fidelity.
|
||||
context = effective_budget.apply_to_text(context)
|
||||
|
||||
# Track only memories that survived ranking, entry limits, and the
|
||||
|
||||
@@ -1,10 +1,8 @@
|
||||
"""``MemoryQuery``: multi-source, full-fidelity retrieval query.
|
||||
|
||||
Pre-this-PR, the retrieval query was "latest user message, truncated
|
||||
to 500 chars" (memory_handler.py:807). The truncation was a real bug
|
||||
— none of Letta / Mem0 / Cognee / Supermemory truncate the embedding
|
||||
input. Tool outputs are often the strongest retrieval signal in
|
||||
coding sessions, and they were ignored entirely.
|
||||
Raw user messages and recent tool/assistant turns are retained for retrieval
|
||||
strategies that need their full content. Time-limited proxy lookups can render
|
||||
a separate bounded embedding input without mutating those sources.
|
||||
|
||||
This value type captures the query at full fidelity from three
|
||||
sources:
|
||||
@@ -13,10 +11,8 @@ sources:
|
||||
* ``recent_tool_outputs`` — last N tool results
|
||||
* ``recent_assistant_turns`` — last K assistant turns for intent
|
||||
|
||||
The embedding model handles its own context window (MiniLM 512 tok;
|
||||
BGE-small 8K tok). Long inputs that exceed the model window become
|
||||
the model's problem to mean-pool or chunk — they don't get
|
||||
truncated upstream.
|
||||
The default embedding representation preserves the full sources. A caller can
|
||||
request a character budget when embedding latency must fit a request deadline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -34,11 +30,9 @@ from .memory_query_policy import (
|
||||
class MemoryQuery:
|
||||
"""Frozen multi-source query for memory retrieval.
|
||||
|
||||
All fields preserve full input fidelity — no truncation, no
|
||||
summarization. The caller assembles the sources; this type only
|
||||
holds them. The retrieval backend decides how to embed
|
||||
(mean-pool, chunk, model-side truncation, etc.) but cannot lose
|
||||
information before it sees the data.
|
||||
Raw fields preserve full input fidelity. Budgeted embedding rendering
|
||||
produces a derived query without changing those fields; backends that
|
||||
chunk or mean-pool full inputs can still use the original sources.
|
||||
|
||||
Tuples for the recent-* fields so the dataclass stays hashable
|
||||
(frozen + value-equal).
|
||||
@@ -49,18 +43,20 @@ class MemoryQuery:
|
||||
recent_assistant_turns: tuple[str, ...]
|
||||
conversation_id: str | None
|
||||
|
||||
def to_embedding_input(self) -> str:
|
||||
"""Concatenate sources into a delimited embedding input.
|
||||
def to_embedding_input(self, *, max_chars: int | None = None) -> str:
|
||||
"""Render delimited embedding input, optionally within a character budget.
|
||||
|
||||
Order: prior assistant turns (oldest first) → tool outputs
|
||||
(oldest first) → latest user text. User text last because the
|
||||
embedder's positional weighting often emphasizes the tail of
|
||||
the input.
|
||||
By default, context is chronological and user text comes last.
|
||||
With a character budget, user text comes first and recent context is
|
||||
selected newest-first, then rendered chronologically so a tokenizer
|
||||
can retain its newest tail. Proxy lookups separately reserve space
|
||||
for both sources inside the active local model's token window.
|
||||
"""
|
||||
return render_embedding_input(
|
||||
user_text=self.user_text,
|
||||
recent_tool_outputs=self.recent_tool_outputs,
|
||||
recent_assistant_turns=self.recent_assistant_turns,
|
||||
max_chars=max_chars,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
|
||||
@@ -17,18 +17,58 @@ def render_embedding_input(
|
||||
user_text: str,
|
||||
recent_tool_outputs: tuple[str, ...],
|
||||
recent_assistant_turns: tuple[str, ...],
|
||||
max_chars: int | None = None,
|
||||
) -> str:
|
||||
"""Concatenate memory query sources into a delimited embedding input."""
|
||||
"""Render full sources, or a user-first query within an embedding budget."""
|
||||
parts: list[str] = []
|
||||
for asst in recent_assistant_turns:
|
||||
if asst:
|
||||
parts.append(ASSISTANT_DELIM + asst)
|
||||
for tool_out in recent_tool_outputs:
|
||||
if tool_out:
|
||||
parts.append(TOOL_DELIM + tool_out)
|
||||
if user_text:
|
||||
parts.append(USER_DELIM + user_text)
|
||||
return "".join(parts)
|
||||
if max_chars is None:
|
||||
for asst in recent_assistant_turns:
|
||||
if asst:
|
||||
parts.append(ASSISTANT_DELIM + asst)
|
||||
for tool_out in recent_tool_outputs:
|
||||
if tool_out:
|
||||
parts.append(TOOL_DELIM + tool_out)
|
||||
if user_text:
|
||||
parts.append(USER_DELIM + user_text)
|
||||
return "".join(parts)
|
||||
|
||||
return "".join(
|
||||
bounded_embedding_parts(
|
||||
user_text=user_text,
|
||||
recent_tool_outputs=recent_tool_outputs,
|
||||
recent_assistant_turns=recent_assistant_turns,
|
||||
max_chars=max_chars,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def bounded_embedding_parts(
|
||||
*,
|
||||
user_text: str,
|
||||
recent_tool_outputs: tuple[str, ...],
|
||||
recent_assistant_turns: tuple[str, ...],
|
||||
max_chars: int = 4096,
|
||||
) -> tuple[str, str]:
|
||||
"""Keep question and context separate so a tokenizer can budget both."""
|
||||
remaining = max(0, max_chars)
|
||||
user_chars = min(2048, max(0, remaining - len(USER_DELIM)))
|
||||
question = USER_DELIM + user_text[-user_chars:] if user_text and user_chars else ""
|
||||
remaining -= len(question) + bool(question)
|
||||
parts: list[str] = []
|
||||
for delimiter, sources in (
|
||||
(TOOL_DELIM, recent_tool_outputs),
|
||||
(ASSISTANT_DELIM, recent_assistant_turns),
|
||||
):
|
||||
for source in reversed(sources):
|
||||
if not source:
|
||||
continue
|
||||
source_chars = min(512, remaining - len(delimiter))
|
||||
if source_chars <= 0:
|
||||
break
|
||||
part = delimiter + source[-source_chars:]
|
||||
parts.append(part)
|
||||
remaining -= len(part)
|
||||
return question, "".join(reversed(parts))
|
||||
|
||||
|
||||
def extract_memory_query_sources(
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
"""Query sources must survive the actual tokenizer's finite input window."""
|
||||
|
||||
from tokenizers import Tokenizer
|
||||
from tokenizers.models import WordPiece
|
||||
from tokenizers.normalizers import BertNormalizer
|
||||
from tokenizers.pre_tokenizers import BertPreTokenizer
|
||||
from tokenizers.processors import TemplateProcessing
|
||||
|
||||
from headroom.memory.adapters.embedders import _fit_query_to_window
|
||||
|
||||
|
||||
def make_tokenizer() -> Tokenizer:
|
||||
words = [
|
||||
"[UNK]",
|
||||
"[CLS]",
|
||||
"[SEP]",
|
||||
"[PAD]",
|
||||
"question",
|
||||
"answer",
|
||||
"tool",
|
||||
"database",
|
||||
"corruption",
|
||||
"古",
|
||||
"い",
|
||||
"質",
|
||||
"問",
|
||||
"出",
|
||||
"力",
|
||||
"答",
|
||||
"案",
|
||||
"破",
|
||||
"損",
|
||||
"故",
|
||||
"障",
|
||||
]
|
||||
vocab = {word: index for index, word in enumerate(words)}
|
||||
tokenizer = Tokenizer(WordPiece(vocab, unk_token="[UNK]"))
|
||||
tokenizer.normalizer = BertNormalizer(lowercase=True)
|
||||
tokenizer.pre_tokenizer = BertPreTokenizer()
|
||||
tokenizer.post_processor = TemplateProcessing(
|
||||
single="[CLS] $A [SEP]",
|
||||
special_tokens=[("[CLS]", vocab["[CLS]"]), ("[SEP]", vocab["[SEP]"])],
|
||||
)
|
||||
return tokenizer
|
||||
|
||||
|
||||
def test_long_question_cannot_consume_the_tool_evidence_window() -> None:
|
||||
tokenizer = make_tokenizer()
|
||||
question = "question " * 300 + "answer"
|
||||
context = "tool " * 300 + "database corruption"
|
||||
rendered = _fit_query_to_window(
|
||||
question,
|
||||
context,
|
||||
14,
|
||||
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
|
||||
)
|
||||
inference = Tokenizer.from_str(tokenizer.to_str())
|
||||
inference.enable_truncation(max_length=16)
|
||||
ids = inference.encode(rendered).ids
|
||||
|
||||
assert len(tokenizer.encode(rendered).ids) <= 16
|
||||
assert tokenizer.token_to_id("answer") in ids
|
||||
assert tokenizer.token_to_id("database") in ids
|
||||
assert tokenizer.token_to_id("corruption") in ids
|
||||
|
||||
|
||||
def test_unicode_sources_are_budgeted_by_tokens_not_python_characters() -> None:
|
||||
tokenizer = make_tokenizer()
|
||||
question = "古い質問" * 100 + "答案"
|
||||
context = "古い出力" * 100 + "破損故障"
|
||||
rendered = _fit_query_to_window(
|
||||
question,
|
||||
context,
|
||||
14,
|
||||
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
|
||||
)
|
||||
inference = Tokenizer.from_str(tokenizer.to_str())
|
||||
inference.enable_truncation(max_length=16)
|
||||
ids = inference.encode(rendered).ids
|
||||
|
||||
assert len(tokenizer.encode(rendered).ids) <= 16
|
||||
for character in "答案破損故障":
|
||||
assert tokenizer.token_to_id(character) in ids
|
||||
@@ -3,20 +3,56 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from headroom.proxy.memory_query_policy import (
|
||||
ASSISTANT_DELIM,
|
||||
TOOL_DELIM,
|
||||
USER_DELIM,
|
||||
extract_memory_query_sources,
|
||||
render_embedding_input,
|
||||
)
|
||||
|
||||
|
||||
def test_render_embedding_input_orders_sources_for_embedding() -> None:
|
||||
def test_bounded_embedding_keeps_primary_question_and_context_tails() -> None:
|
||||
user = "古い質問" * 2000 + " latest JWT question"
|
||||
tool = "old tool output " * 1000 + " latest authentication failure"
|
||||
assistant = "old assistant context " * 1000 + " inspect auth.py"
|
||||
rendered = render_embedding_input(
|
||||
user_text="latest user",
|
||||
recent_tool_outputs=("tool output",),
|
||||
recent_assistant_turns=("assistant context",),
|
||||
user_text=user,
|
||||
recent_tool_outputs=(tool,),
|
||||
recent_assistant_turns=(assistant,),
|
||||
max_chars=4096,
|
||||
)
|
||||
|
||||
assert rendered.index("assistant context") < rendered.index("tool output")
|
||||
assert rendered.index("tool output") < rendered.index("latest user")
|
||||
assert len(rendered) <= 4096
|
||||
assert rendered.startswith(USER_DELIM + user[-2048:])
|
||||
assert TOOL_DELIM + tool[-512:] in rendered
|
||||
assert ASSISTANT_DELIM + assistant[-512:] in rendered
|
||||
|
||||
|
||||
def test_bounded_embedding_spends_remaining_budget_on_newest_context() -> None:
|
||||
rendered = render_embedding_input(
|
||||
user_text="latest question",
|
||||
recent_tool_outputs=tuple("x" * 600 + f" end-{i}" for i in range(10)),
|
||||
recent_assistant_turns=("old background",),
|
||||
max_chars=1200,
|
||||
)
|
||||
|
||||
assert len(rendered) <= 1200
|
||||
assert rendered.startswith(USER_DELIM + "latest question")
|
||||
assert "end-9" in rendered and "end-8" in rendered
|
||||
assert "end-0" not in rendered
|
||||
assert "old background" not in rendered
|
||||
|
||||
|
||||
def test_zero_embedding_budget_does_not_send_context_without_the_question() -> None:
|
||||
assert (
|
||||
render_embedding_input(
|
||||
user_text="question",
|
||||
recent_tool_outputs=("tool context",),
|
||||
recent_assistant_turns=("assistant context",),
|
||||
max_chars=0,
|
||||
)
|
||||
== ""
|
||||
)
|
||||
|
||||
|
||||
def test_extract_sources_uses_latest_user_and_recent_context_in_order() -> None:
|
||||
|
||||
Reference in New Issue
Block a user