Files
headroom/tests/test_memory_query_policy.py
Mansurshakh Japarov 8c07cf4a0f fix(memory): bound retrieval queries before embedding (#4244)
## Description

Closes #4223.

The proxy now embeds a bounded, user-first retrieval query instead of
concatenating entire tool outputs into a deadline-limited lookup. The
raw `MemoryQuery` and its default SDK rendering still retain the full
sources.

Memory failures use the exception representation, so an empty
`TimeoutError` is no longer a blank warning. The existing Anthropic
deadline is unchanged and documented; this does not promise that a cold
model or slow embedding server will finish within it.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- Bound proxy embedding input to 4096 characters, including delimiters;
spend the budget on the latest user question and recent context tails.
- Reserve question and context space inside the active local tokenizer's
window, rather than relying on a character cap to prevent truncation.
- Keep the full-fidelity SDK behavior when no budget is requested.
- Make generic memory errors identifiable in the shared handler and
provider handlers.
- Cover Unicode, newest-context precedence, zero character budget and
actual tokenizer-window retention; keep the full-fidelity SDK tests.

## Testing

- [ ] Unit tests pass (pytest) — all 67 affected query, memory-handler
and tokenizer-window regressions passed locally, not the entire Python
suite.
- [ ] Linting passes (ruff check .) — normal changed-file Ruff hooks
passed.
- [ ] Type checking passes (mypy headroom) — normal mypy hook passed.
- [x] Manual testing performed

### Test Output

```text
Affected query/native-memory/tool/auto-tail/tokenizer suites: 67 passed, 1 warning
Ordinary make ci-precheck: PASS (181 critical Python tests, 6 warnings)
```

The focused-suite warning concerns assertion rewriting of the
already-imported `asyncio` name; it was not suppressed. All five
installed pytest plugins were loaded explicitly. Normal Ruff
formatting/checking, mypy and commitlint hooks passed on `8eb10bfd`. The
ordinary full pre-push gate passed on that same commit, including Rust
workspace checks/tests, the native build and critical Python suites; the
normal fast-forward fork push completed without a bypass.

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, the actual local ONNX
model/session, SQLite and vector retrieval. The optional
sentence-transformers CPU backend was also exercised in a separate
complete CPU-wheel environment; no project dependency changes.
- Exact command / steps: save a SQLite-corruption memory and an
unrelated cooking memory through the real `LocalBackend`. Call
`MemoryHandler.search_and_format_context` under a two-second
`asyncio.wait_for` with the same long repeated question and SQLite tool
error before and after the change. Record the real ONNX session's input
IDs/attention mask without replacing inference; decode those IDs to
check the final question and tool error. Verify the original
`MemoryQuery` fields stay intact.
- Observed result: both fragments now reach the model and the relevant
memory is retrieved. Same synthetic input and default cache behavior; no
fake embedding or search result:

| Measurement | Before | After |
| --- | ---: | ---: |
| Submitted characters | 2594 | 1146 |
| Tokens before model truncation | 560 | 256 |
| Question tail and tool error reach model | No | Yes |
| Relevant memory retrieved | No | Yes |
| Warm lookup seconds | 0.076 | 0.241 |

A separately configured 96-token ONNX model also retained both
fragments. The real sentence-transformers CPU model and a previously
uninitialized local ONNX backend passed the same retention check and
generated real embeddings. The legacy messages-only path still submitted
4096 characters; the empty-timeout diagnostic still logged
`TimeoutError()` and failed open.

- Not tested: the reporter's Granite embedding server, cold-start
deadlines or network latency, arbitrary remote tokenizer windows, live
provider requests, Windows/macOS or GPU/MPS. Only the diagnostic failure
was injected; successful persistence, embeddings and retrieval used real
backends.

## Runtime Rollout Safety

- Rollout-managed feature(s): existing automatic memory lookup; no new
feature gate.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: proxy lookups use the bounded
representation; the SDK's default full rendering is unchanged.
- Kill switch / disable path: existing `--no-memory-context` or memory
configuration.
- Unsafe override required: no.
- Qualification impact: actual retrieval, raw-input preservation and
diagnostics checked locally.
- Rollback path: revert the fix; no schema migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it

## Additional Notes

AI-assisted; no subagents.
2026-10-24 08:45:33 -07:00

161 lines
5.2 KiB
Python

"""Tests for pure memory query construction policy."""
from __future__ import annotations
from headroom.proxy.memory_query_policy import (
ASSISTANT_DELIM,
TOOL_DELIM,
USER_DELIM,
extract_memory_query_sources,
render_embedding_input,
)
def test_bounded_embedding_keeps_primary_question_and_context_tails() -> None:
user = "古い質問" * 2000 + " latest JWT question"
tool = "old tool output " * 1000 + " latest authentication failure"
assistant = "old assistant context " * 1000 + " inspect auth.py"
rendered = render_embedding_input(
user_text=user,
recent_tool_outputs=(tool,),
recent_assistant_turns=(assistant,),
max_chars=4096,
)
assert len(rendered) <= 4096
assert rendered.startswith(USER_DELIM + user[-2048:])
assert TOOL_DELIM + tool[-512:] in rendered
assert ASSISTANT_DELIM + assistant[-512:] in rendered
def test_bounded_embedding_spends_remaining_budget_on_newest_context() -> None:
rendered = render_embedding_input(
user_text="latest question",
recent_tool_outputs=tuple("x" * 600 + f" end-{i}" for i in range(10)),
recent_assistant_turns=("old background",),
max_chars=1200,
)
assert len(rendered) <= 1200
assert rendered.startswith(USER_DELIM + "latest question")
assert "end-9" in rendered and "end-8" in rendered
assert "end-0" not in rendered
assert "old background" not in rendered
def test_zero_embedding_budget_does_not_send_context_without_the_question() -> None:
assert (
render_embedding_input(
user_text="question",
recent_tool_outputs=("tool context",),
recent_assistant_turns=("assistant context",),
max_chars=0,
)
== ""
)
def test_extract_sources_uses_latest_user_and_recent_context_in_order() -> None:
messages = [
{"role": "user", "content": "first"},
{"role": "assistant", "content": "a1"},
{"role": "tool", "content": "t1"},
{"role": "assistant", "content": "a2"},
{"role": "tool", "content": "t2"},
{"role": "user", "content": "second"},
]
user_text, tool_outputs, assistant_turns = extract_memory_query_sources(
messages,
lookback_assistant=2,
lookback_tools=2,
)
assert user_text == "second"
assert tool_outputs == ("t1", "t2")
assert assistant_turns == ("a1", "a2")
def test_extract_sources_handles_anthropic_tool_result_without_user_text() -> None:
messages = [
{"role": "user", "content": "real user"},
{
"role": "user",
"content": [{"type": "tool_result", "content": [{"type": "text", "text": "nested"}]}],
},
]
user_text, tool_outputs, assistant_turns = extract_memory_query_sources(messages)
assert user_text == "real user"
assert tool_outputs == ("nested",)
assert assistant_turns == ()
def test_extract_sources_captures_anthropic_user_text_blocks() -> None:
"""Anthropic user turns carry the prompt as text blocks (the standard Claude
Code shape). The user's question must be captured — not dropped — so memory
retrieval keys on it."""
messages = [
{"role": "user", "content": [{"type": "text", "text": "help me refactor auth"}]},
]
user_text, _tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
assert user_text == "help me refactor auth"
def test_extract_sources_skips_system_reminder_blocks() -> None:
"""Claude Code appends <system-reminder> harness blocks to the user turn.
Concatenated into the embedding input they dilute the real question below the
similarity floor so nothing is retrieved (#2195); they must be filtered out."""
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "how do I add caching to the auth handler?"},
{
"type": "text",
"text": "<system-reminder>\nThe user opened file x.\n</system-reminder>",
},
],
},
]
user_text, _tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
assert user_text == "how do I add caching to the auth handler?"
assert "system-reminder" not in user_text
def test_extract_sources_reminder_only_turn_yields_no_user_text() -> None:
messages = [
{
"role": "user",
"content": [{"type": "text", "text": "<system-reminder>x</system-reminder>"}],
},
]
user_text, _tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
assert user_text == ""
def test_extract_sources_captures_user_text_alongside_tool_result() -> None:
"""A user turn mixing a tool_result and a text block yields both: the text as
the user query and the tool output as context."""
messages = [
{
"role": "user",
"content": [
{"type": "tool_result", "content": "exit 0"},
{"type": "text", "text": "did the tests pass?"},
],
},
]
user_text, tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
assert user_text == "did the tests pass?"
assert tool_outputs == ("exit 0",)