mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Closes #4223. The proxy now embeds a bounded, user-first retrieval query instead of concatenating entire tool outputs into a deadline-limited lookup. The raw `MemoryQuery` and its default SDK rendering still retain the full sources. Memory failures use the exception representation, so an empty `TimeoutError` is no longer a blank warning. The existing Anthropic deadline is unchanged and documented; this does not promise that a cold model or slow embedding server will finish within it. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update ## Changes Made - Bound proxy embedding input to 4096 characters, including delimiters; spend the budget on the latest user question and recent context tails. - Reserve question and context space inside the active local tokenizer's window, rather than relying on a character cap to prevent truncation. - Keep the full-fidelity SDK behavior when no budget is requested. - Make generic memory errors identifiable in the shared handler and provider handlers. - Cover Unicode, newest-context precedence, zero character budget and actual tokenizer-window retention; keep the full-fidelity SDK tests. ## Testing - [ ] Unit tests pass (pytest) — all 67 affected query, memory-handler and tokenizer-window regressions passed locally, not the entire Python suite. - [ ] Linting passes (ruff check .) — normal changed-file Ruff hooks passed. - [ ] Type checking passes (mypy headroom) — normal mypy hook passed. - [x] Manual testing performed ### Test Output ```text Affected query/native-memory/tool/auto-tail/tokenizer suites: 67 passed, 1 warning Ordinary make ci-precheck: PASS (181 critical Python tests, 6 warnings) ``` The focused-suite warning concerns assertion rewriting of the already-imported `asyncio` name; it was not suppressed. All five installed pytest plugins were loaded explicitly. Normal Ruff formatting/checking, mypy and commitlint hooks passed on `8eb10bfd`. The ordinary full pre-push gate passed on that same commit, including Rust workspace checks/tests, the native build and critical Python suites; the normal fast-forward fork push completed without a bypass. ## Real Behavior Proof - Environment: Linux x86-64, Python 3.13.14, the actual local ONNX model/session, SQLite and vector retrieval. The optional sentence-transformers CPU backend was also exercised in a separate complete CPU-wheel environment; no project dependency changes. - Exact command / steps: save a SQLite-corruption memory and an unrelated cooking memory through the real `LocalBackend`. Call `MemoryHandler.search_and_format_context` under a two-second `asyncio.wait_for` with the same long repeated question and SQLite tool error before and after the change. Record the real ONNX session's input IDs/attention mask without replacing inference; decode those IDs to check the final question and tool error. Verify the original `MemoryQuery` fields stay intact. - Observed result: both fragments now reach the model and the relevant memory is retrieved. Same synthetic input and default cache behavior; no fake embedding or search result: | Measurement | Before | After | | --- | ---: | ---: | | Submitted characters | 2594 | 1146 | | Tokens before model truncation | 560 | 256 | | Question tail and tool error reach model | No | Yes | | Relevant memory retrieved | No | Yes | | Warm lookup seconds | 0.076 | 0.241 | A separately configured 96-token ONNX model also retained both fragments. The real sentence-transformers CPU model and a previously uninitialized local ONNX backend passed the same retention check and generated real embeddings. The legacy messages-only path still submitted 4096 characters; the empty-timeout diagnostic still logged `TimeoutError()` and failed open. - Not tested: the reporter's Granite embedding server, cold-start deadlines or network latency, arbitrary remote tokenizer windows, live provider requests, Windows/macOS or GPU/MPS. Only the diagnostic failure was injected; successful persistence, embeddings and retrieval used real backends. ## Runtime Rollout Safety - Rollout-managed feature(s): existing automatic memory lookup; no new feature gate. - Minimum rollout channel: unchanged. - Stable/default behavior changed: proxy lookups use the bounded representation; the SDK's default full rendering is unchanged. - Kill switch / disable path: existing `--no-memory-context` or memory configuration. - Unsafe override required: no. - Qualification impact: actual retrieval, raw-input preservation and diagnostics checked locally. - Rollback path: revert the fix; no schema migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I did not edit CHANGELOG.md — release-please owns it ## Additional Notes AI-assisted; no subagents.
161 lines
5.2 KiB
Python
161 lines
5.2 KiB
Python
"""Tests for pure memory query construction policy."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from headroom.proxy.memory_query_policy import (
|
|
ASSISTANT_DELIM,
|
|
TOOL_DELIM,
|
|
USER_DELIM,
|
|
extract_memory_query_sources,
|
|
render_embedding_input,
|
|
)
|
|
|
|
|
|
def test_bounded_embedding_keeps_primary_question_and_context_tails() -> None:
|
|
user = "古い質問" * 2000 + " latest JWT question"
|
|
tool = "old tool output " * 1000 + " latest authentication failure"
|
|
assistant = "old assistant context " * 1000 + " inspect auth.py"
|
|
rendered = render_embedding_input(
|
|
user_text=user,
|
|
recent_tool_outputs=(tool,),
|
|
recent_assistant_turns=(assistant,),
|
|
max_chars=4096,
|
|
)
|
|
|
|
assert len(rendered) <= 4096
|
|
assert rendered.startswith(USER_DELIM + user[-2048:])
|
|
assert TOOL_DELIM + tool[-512:] in rendered
|
|
assert ASSISTANT_DELIM + assistant[-512:] in rendered
|
|
|
|
|
|
def test_bounded_embedding_spends_remaining_budget_on_newest_context() -> None:
|
|
rendered = render_embedding_input(
|
|
user_text="latest question",
|
|
recent_tool_outputs=tuple("x" * 600 + f" end-{i}" for i in range(10)),
|
|
recent_assistant_turns=("old background",),
|
|
max_chars=1200,
|
|
)
|
|
|
|
assert len(rendered) <= 1200
|
|
assert rendered.startswith(USER_DELIM + "latest question")
|
|
assert "end-9" in rendered and "end-8" in rendered
|
|
assert "end-0" not in rendered
|
|
assert "old background" not in rendered
|
|
|
|
|
|
def test_zero_embedding_budget_does_not_send_context_without_the_question() -> None:
|
|
assert (
|
|
render_embedding_input(
|
|
user_text="question",
|
|
recent_tool_outputs=("tool context",),
|
|
recent_assistant_turns=("assistant context",),
|
|
max_chars=0,
|
|
)
|
|
== ""
|
|
)
|
|
|
|
|
|
def test_extract_sources_uses_latest_user_and_recent_context_in_order() -> None:
|
|
messages = [
|
|
{"role": "user", "content": "first"},
|
|
{"role": "assistant", "content": "a1"},
|
|
{"role": "tool", "content": "t1"},
|
|
{"role": "assistant", "content": "a2"},
|
|
{"role": "tool", "content": "t2"},
|
|
{"role": "user", "content": "second"},
|
|
]
|
|
|
|
user_text, tool_outputs, assistant_turns = extract_memory_query_sources(
|
|
messages,
|
|
lookback_assistant=2,
|
|
lookback_tools=2,
|
|
)
|
|
|
|
assert user_text == "second"
|
|
assert tool_outputs == ("t1", "t2")
|
|
assert assistant_turns == ("a1", "a2")
|
|
|
|
|
|
def test_extract_sources_handles_anthropic_tool_result_without_user_text() -> None:
|
|
messages = [
|
|
{"role": "user", "content": "real user"},
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "content": [{"type": "text", "text": "nested"}]}],
|
|
},
|
|
]
|
|
|
|
user_text, tool_outputs, assistant_turns = extract_memory_query_sources(messages)
|
|
|
|
assert user_text == "real user"
|
|
assert tool_outputs == ("nested",)
|
|
assert assistant_turns == ()
|
|
|
|
|
|
def test_extract_sources_captures_anthropic_user_text_blocks() -> None:
|
|
"""Anthropic user turns carry the prompt as text blocks (the standard Claude
|
|
Code shape). The user's question must be captured — not dropped — so memory
|
|
retrieval keys on it."""
|
|
messages = [
|
|
{"role": "user", "content": [{"type": "text", "text": "help me refactor auth"}]},
|
|
]
|
|
|
|
user_text, _tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
|
|
|
|
assert user_text == "help me refactor auth"
|
|
|
|
|
|
def test_extract_sources_skips_system_reminder_blocks() -> None:
|
|
"""Claude Code appends <system-reminder> harness blocks to the user turn.
|
|
Concatenated into the embedding input they dilute the real question below the
|
|
similarity floor so nothing is retrieved (#2195); they must be filtered out."""
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "text", "text": "how do I add caching to the auth handler?"},
|
|
{
|
|
"type": "text",
|
|
"text": "<system-reminder>\nThe user opened file x.\n</system-reminder>",
|
|
},
|
|
],
|
|
},
|
|
]
|
|
|
|
user_text, _tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
|
|
|
|
assert user_text == "how do I add caching to the auth handler?"
|
|
assert "system-reminder" not in user_text
|
|
|
|
|
|
def test_extract_sources_reminder_only_turn_yields_no_user_text() -> None:
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "text", "text": "<system-reminder>x</system-reminder>"}],
|
|
},
|
|
]
|
|
|
|
user_text, _tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
|
|
|
|
assert user_text == ""
|
|
|
|
|
|
def test_extract_sources_captures_user_text_alongside_tool_result() -> None:
|
|
"""A user turn mixing a tool_result and a text block yields both: the text as
|
|
the user query and the tool output as context."""
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "tool_result", "content": "exit 0"},
|
|
{"type": "text", "text": "did the tests pass?"},
|
|
],
|
|
},
|
|
]
|
|
|
|
user_text, tool_outputs, _assistant_turns = extract_memory_query_sources(messages)
|
|
|
|
assert user_text == "did the tests pass?"
|
|
assert tool_outputs == ("exit 0",)
|