Files
headroom/tests/test_memory/test_query_budget.py
Mansurshakh Japarov 8c07cf4a0f fix(memory): bound retrieval queries before embedding (#4244)
## Description

Closes #4223.

The proxy now embeds a bounded, user-first retrieval query instead of
concatenating entire tool outputs into a deadline-limited lookup. The
raw `MemoryQuery` and its default SDK rendering still retain the full
sources.

Memory failures use the exception representation, so an empty
`TimeoutError` is no longer a blank warning. The existing Anthropic
deadline is unchanged and documented; this does not promise that a cold
model or slow embedding server will finish within it.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- Bound proxy embedding input to 4096 characters, including delimiters;
spend the budget on the latest user question and recent context tails.
- Reserve question and context space inside the active local tokenizer's
window, rather than relying on a character cap to prevent truncation.
- Keep the full-fidelity SDK behavior when no budget is requested.
- Make generic memory errors identifiable in the shared handler and
provider handlers.
- Cover Unicode, newest-context precedence, zero character budget and
actual tokenizer-window retention; keep the full-fidelity SDK tests.

## Testing

- [ ] Unit tests pass (pytest) — all 67 affected query, memory-handler
and tokenizer-window regressions passed locally, not the entire Python
suite.
- [ ] Linting passes (ruff check .) — normal changed-file Ruff hooks
passed.
- [ ] Type checking passes (mypy headroom) — normal mypy hook passed.
- [x] Manual testing performed

### Test Output

```text
Affected query/native-memory/tool/auto-tail/tokenizer suites: 67 passed, 1 warning
Ordinary make ci-precheck: PASS (181 critical Python tests, 6 warnings)
```

The focused-suite warning concerns assertion rewriting of the
already-imported `asyncio` name; it was not suppressed. All five
installed pytest plugins were loaded explicitly. Normal Ruff
formatting/checking, mypy and commitlint hooks passed on `8eb10bfd`. The
ordinary full pre-push gate passed on that same commit, including Rust
workspace checks/tests, the native build and critical Python suites; the
normal fast-forward fork push completed without a bypass.

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, the actual local ONNX
model/session, SQLite and vector retrieval. The optional
sentence-transformers CPU backend was also exercised in a separate
complete CPU-wheel environment; no project dependency changes.
- Exact command / steps: save a SQLite-corruption memory and an
unrelated cooking memory through the real `LocalBackend`. Call
`MemoryHandler.search_and_format_context` under a two-second
`asyncio.wait_for` with the same long repeated question and SQLite tool
error before and after the change. Record the real ONNX session's input
IDs/attention mask without replacing inference; decode those IDs to
check the final question and tool error. Verify the original
`MemoryQuery` fields stay intact.
- Observed result: both fragments now reach the model and the relevant
memory is retrieved. Same synthetic input and default cache behavior; no
fake embedding or search result:

| Measurement | Before | After |
| --- | ---: | ---: |
| Submitted characters | 2594 | 1146 |
| Tokens before model truncation | 560 | 256 |
| Question tail and tool error reach model | No | Yes |
| Relevant memory retrieved | No | Yes |
| Warm lookup seconds | 0.076 | 0.241 |

A separately configured 96-token ONNX model also retained both
fragments. The real sentence-transformers CPU model and a previously
uninitialized local ONNX backend passed the same retention check and
generated real embeddings. The legacy messages-only path still submitted
4096 characters; the empty-timeout diagnostic still logged
`TimeoutError()` and failed open.

- Not tested: the reporter's Granite embedding server, cold-start
deadlines or network latency, arbitrary remote tokenizer windows, live
provider requests, Windows/macOS or GPU/MPS. Only the diagnostic failure
was injected; successful persistence, embeddings and retrieval used real
backends.

## Runtime Rollout Safety

- Rollout-managed feature(s): existing automatic memory lookup; no new
feature gate.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: proxy lookups use the bounded
representation; the SDK's default full rendering is unchanged.
- Kill switch / disable path: existing `--no-memory-context` or memory
configuration.
- Unsafe override required: no.
- Qualification impact: actual retrieval, raw-input preservation and
diagnostics checked locally.
- Rollback path: revert the fix; no schema migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it

## Additional Notes

AI-assisted; no subagents.
2026-10-24 08:45:33 -07:00

84 lines
2.5 KiB
Python

"""Query sources must survive the actual tokenizer's finite input window."""
from tokenizers import Tokenizer
from tokenizers.models import WordPiece
from tokenizers.normalizers import BertNormalizer
from tokenizers.pre_tokenizers import BertPreTokenizer
from tokenizers.processors import TemplateProcessing
from headroom.memory.adapters.embedders import _fit_query_to_window
def make_tokenizer() -> Tokenizer:
words = [
"[UNK]",
"[CLS]",
"[SEP]",
"[PAD]",
"question",
"answer",
"tool",
"database",
"corruption",
"古",
"い",
"質",
"問",
"出",
"力",
"答",
"案",
"破",
"損",
"故",
"障",
]
vocab = {word: index for index, word in enumerate(words)}
tokenizer = Tokenizer(WordPiece(vocab, unk_token="[UNK]"))
tokenizer.normalizer = BertNormalizer(lowercase=True)
tokenizer.pre_tokenizer = BertPreTokenizer()
tokenizer.post_processor = TemplateProcessing(
single="[CLS] $A [SEP]",
special_tokens=[("[CLS]", vocab["[CLS]"]), ("[SEP]", vocab["[SEP]"])],
)
return tokenizer
def test_long_question_cannot_consume_the_tool_evidence_window() -> None:
tokenizer = make_tokenizer()
question = "question " * 300 + "answer"
context = "tool " * 300 + "database corruption"
rendered = _fit_query_to_window(
question,
context,
14,
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
)
inference = Tokenizer.from_str(tokenizer.to_str())
inference.enable_truncation(max_length=16)
ids = inference.encode(rendered).ids
assert len(tokenizer.encode(rendered).ids) <= 16
assert tokenizer.token_to_id("answer") in ids
assert tokenizer.token_to_id("database") in ids
assert tokenizer.token_to_id("corruption") in ids
def test_unicode_sources_are_budgeted_by_tokens_not_python_characters() -> None:
tokenizer = make_tokenizer()
question = "古い質問" * 100 + "答案"
context = "古い出力" * 100 + "破損故障"
rendered = _fit_query_to_window(
question,
context,
14,
lambda text: len(tokenizer.encode(text, add_special_tokens=False).ids),
)
inference = Tokenizer.from_str(tokenizer.to_str())
inference.enable_truncation(max_length=16)
ids = inference.encode(rendered).ids
assert len(tokenizer.encode(rendered).ids) <= 16
for character in "答案破損故障":
assert tokenizer.token_to_id(character) in ids