mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Closes #4223. The proxy now embeds a bounded, user-first retrieval query instead of concatenating entire tool outputs into a deadline-limited lookup. The raw `MemoryQuery` and its default SDK rendering still retain the full sources. Memory failures use the exception representation, so an empty `TimeoutError` is no longer a blank warning. The existing Anthropic deadline is unchanged and documented; this does not promise that a cold model or slow embedding server will finish within it. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update ## Changes Made - Bound proxy embedding input to 4096 characters, including delimiters; spend the budget on the latest user question and recent context tails. - Reserve question and context space inside the active local tokenizer's window, rather than relying on a character cap to prevent truncation. - Keep the full-fidelity SDK behavior when no budget is requested. - Make generic memory errors identifiable in the shared handler and provider handlers. - Cover Unicode, newest-context precedence, zero character budget and actual tokenizer-window retention; keep the full-fidelity SDK tests. ## Testing - [ ] Unit tests pass (pytest) — all 67 affected query, memory-handler and tokenizer-window regressions passed locally, not the entire Python suite. - [ ] Linting passes (ruff check .) — normal changed-file Ruff hooks passed. - [ ] Type checking passes (mypy headroom) — normal mypy hook passed. - [x] Manual testing performed ### Test Output ```text Affected query/native-memory/tool/auto-tail/tokenizer suites: 67 passed, 1 warning Ordinary make ci-precheck: PASS (181 critical Python tests, 6 warnings) ``` The focused-suite warning concerns assertion rewriting of the already-imported `asyncio` name; it was not suppressed. All five installed pytest plugins were loaded explicitly. Normal Ruff formatting/checking, mypy and commitlint hooks passed on `8eb10bfd`. The ordinary full pre-push gate passed on that same commit, including Rust workspace checks/tests, the native build and critical Python suites; the normal fast-forward fork push completed without a bypass. ## Real Behavior Proof - Environment: Linux x86-64, Python 3.13.14, the actual local ONNX model/session, SQLite and vector retrieval. The optional sentence-transformers CPU backend was also exercised in a separate complete CPU-wheel environment; no project dependency changes. - Exact command / steps: save a SQLite-corruption memory and an unrelated cooking memory through the real `LocalBackend`. Call `MemoryHandler.search_and_format_context` under a two-second `asyncio.wait_for` with the same long repeated question and SQLite tool error before and after the change. Record the real ONNX session's input IDs/attention mask without replacing inference; decode those IDs to check the final question and tool error. Verify the original `MemoryQuery` fields stay intact. - Observed result: both fragments now reach the model and the relevant memory is retrieved. Same synthetic input and default cache behavior; no fake embedding or search result: | Measurement | Before | After | | --- | ---: | ---: | | Submitted characters | 2594 | 1146 | | Tokens before model truncation | 560 | 256 | | Question tail and tool error reach model | No | Yes | | Relevant memory retrieved | No | Yes | | Warm lookup seconds | 0.076 | 0.241 | A separately configured 96-token ONNX model also retained both fragments. The real sentence-transformers CPU model and a previously uninitialized local ONNX backend passed the same retention check and generated real embeddings. The legacy messages-only path still submitted 4096 characters; the empty-timeout diagnostic still logged `TimeoutError()` and failed open. - Not tested: the reporter's Granite embedding server, cold-start deadlines or network latency, arbitrary remote tokenizer windows, live provider requests, Windows/macOS or GPU/MPS. Only the diagnostic failure was injected; successful persistence, embeddings and retrieval used real backends. ## Runtime Rollout Safety - Rollout-managed feature(s): existing automatic memory lookup; no new feature gate. - Minimum rollout channel: unchanged. - Stable/default behavior changed: proxy lookups use the bounded representation; the SDK's default full rendering is unchanged. - Kill switch / disable path: existing `--no-memory-context` or memory configuration. - Unsafe override required: no. - Qualification impact: actual retrieval, raw-input preservation and diagnostics checked locally. - Rollback path: revert the fix; no schema migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I did not edit CHANGELOG.md — release-please owns it ## Additional Notes AI-assisted; no subagents.
84 lines
2.5 KiB
Python
84 lines
2.5 KiB
Python
"""Query sources must survive the actual tokenizer's finite input window."""
|
|
|
|
from tokenizers import Tokenizer
|
|
from tokenizers.models import WordPiece
|
|
from tokenizers.normalizers import BertNormalizer
|
|
from tokenizers.pre_tokenizers import BertPreTokenizer
|
|
from tokenizers.processors import TemplateProcessing
|
|
|
|
from headroom.memory.adapters.embedders import _fit_query_to_window
|
|
|
|
|
|
def make_tokenizer() -> Tokenizer:
|
|
words = [
|
|
"[UNK]",
|
|
"[CLS]",
|
|
"[SEP]",
|
|
"[PAD]",
|
|
"question",
|
|
"answer",
|
|
"tool",
|
|
"database",
|
|
"corruption",
|
|
"古",
|
|
"い",
|
|
"質",
|
|
"問",
|
|
"出",
|
|
"力",
|
|
"答",
|
|
"案",
|
|
"破",
|
|
"損",
|
|
"故",
|
|
"障",
|
|
]
|
|
vocab = {word: index for index, word in enumerate(words)}
|
|
tokenizer = Tokenizer(WordPiece(vocab, unk_token="[UNK]"))
|
|
tokenizer.normalizer = BertNormalizer(lowercase=True)
|
|
tokenizer.pre_tokenizer = BertPreTokenizer()
|
|
tokenizer.post_processor = TemplateProcessing(
|
|
single="[CLS] $A [SEP]",
|
|
special_tokens=[("[CLS]", vocab["[CLS]"]), ("[SEP]", vocab["[SEP]"])],
|
|
)
|
|
return tokenizer
|
|
|
|
|
|
def test_long_question_cannot_consume_the_tool_evidence_window() -> None:
|
|
tokenizer = make_tokenizer()
|
|
question = "question " * 300 + "answer"
|
|
context = "tool " * 300 + "database corruption"
|
|
rendered = _fit_query_to_window(
|
|
question,
|
|
context,
|
|
14,
|
|
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
|
|
)
|
|
inference = Tokenizer.from_str(tokenizer.to_str())
|
|
inference.enable_truncation(max_length=16)
|
|
ids = inference.encode(rendered).ids
|
|
|
|
assert len(tokenizer.encode(rendered).ids) <= 16
|
|
assert tokenizer.token_to_id("answer") in ids
|
|
assert tokenizer.token_to_id("database") in ids
|
|
assert tokenizer.token_to_id("corruption") in ids
|
|
|
|
|
|
def test_unicode_sources_are_budgeted_by_tokens_not_python_characters() -> None:
|
|
tokenizer = make_tokenizer()
|
|
question = "古い質問" * 100 + "答案"
|
|
context = "古い出力" * 100 + "破損故障"
|
|
rendered = _fit_query_to_window(
|
|
question,
|
|
context,
|
|
14,
|
|
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
|
|
)
|
|
inference = Tokenizer.from_str(tokenizer.to_str())
|
|
inference.enable_truncation(max_length=16)
|
|
ids = inference.encode(rendered).ids
|
|
|
|
assert len(tokenizer.encode(rendered).ids) <= 16
|
|
for character in "答案破損故障":
|
|
assert tokenizer.token_to_id(character) in ids
|