Files
headroom/tests/test_memory_decision.py
f89c9282a8 fix(proxy): accept HEADROOM_MEMORY_INJECTION_MODE=tool (#4395)
## Description

`memory_decision.py` handles mode `tool`, but
`resolve_memory_injection_mode` rejected it with `ValueError`, so the
value could not be used. The resolver now accepts `tool`; the
configuration docs list it.

Second commit, after review (Devin, comment 4348232164): in `tool` mode
the old `MemoryDecision` returned `inject=False`, and the handlers gate
the memory tool definitions on that same flag, so the agent would have
had no `memory_search` to call. The intent of `tool` (commit 71d5a7b5:
no auto-injected context, the model calls the memory tools itself) is
now implemented: the decision stays open for `tool`, and
`MemoryHandler.search_and_format_context` skips the context lookup, as
it already did for `MemoryMode.TOOL`.

Fixes #4391

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/memory_injection_mode_policy.py`: add `"tool"` to the
literal type and the accepted values; update the error text.
- `headroom/proxy/helpers.py`: comment documents `tool`.
- `docs/content/docs/configuration.mdx`: list `tool` for
`HEADROOM_MEMORY_INJECTION_MODE`.
- `tests/test_memory_injection_mode_policy.py`: resolver accepts `tool`
(case/space insensitive); `get_memory_injection_mode()` returns `tool`
from the env var.
- `headroom/proxy/memory_decision_policy.py`, `memory_decision.py`:
`tool` no longer produces a skip (`mode_tool` removed); `inject` stays
True so the handlers register the memory tools.
- `headroom/proxy/memory_handler.py`: `search_and_format_context`
returns None when `HEADROOM_MEMORY_INJECTION_MODE=tool` (before any
backend lookup).
- `tests/test_memory_decision.py`,
`tests/test_memory_decision_policy.py`: `tool` expectations changed to
`inject=True`, no skip reason; new test in
`test_memory_injection_mode_policy.py` covers the context skip and the
open decision.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# with the policy change reverted (tests/test_memory_injection_mode_policy.py)
E   ValueError: Invalid HEADROOM_MEMORY_INJECTION_MODE='tool'; expected 'live_zone_tail' or 'disabled'
FAILED tests/test_memory_injection_mode_policy.py::test_resolve_memory_injection_mode_accepts_known_values_case_insensitively
FAILED tests/test_memory_injection_mode_policy.py::test_get_memory_injection_mode_accepts_tool_from_env
2 failed, 2 passed

# second commit: new test with the src changes of that commit reverted
E   assert decision.inject is True / MemoryDecision(inject=False, skip_reason='mode_tool', ...)
1 failed, 4 passed
# second commit applied
$ pytest tests/test_memory_injection_mode_policy.py tests/test_memory_decision.py tests/test_memory_decision_policy.py tests/test_memory_handler_concurrent_init.py
37 passed, 1 skipped

# first commit, the four files listed under Real Behavior Proof
6 failed, 25 passed  (all 6 failures in test_proxy_system_prompt_immutable.py, same 6 on unmodified upstream/main)

$ ruff check <changed py files>
All checks passed!
$ ruff format --check <changed py files>
3 files already formatted
```

## Real Behavior Proof

- Environment: Linux (WSL2), Python 3.12.15, the real proxy server
(`run_server`) with persistent memory on (local backend, ONNX embedder),
Rust core from the headroom-ai 0.41.0 wheel, a local stub upstream on
127.0.0.1
- Exact command / steps: seed one memory for a test user, start the
proxy with `HEADROOM_MEMORY_INJECTION_MODE=tool`, send one non-streaming
`POST /v1/messages` that declares one client tool; repeat on
upstream/main and with `live_zone_tail` as controls (details below)
- Observed result: upstream/main returns HTTP 500 (`ValueError` from the
mode resolver); this branch returns HTTP 200, the stub receives the five
memory tools and the user message without a `## Relevant Memories` block
- Not tested: real Anthropic, streaming, the OpenAI/Gemini handlers and
the WebSocket path, `use_native_tool`, a memory tool-call round trip, a
Rust core built from this checkout

Details:

Setup: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn 0.54.0,
fastapi 0.158.1, starlette 1.9.0, numpy 2.5.4, sqlite-vec 0.1.9,
onnxruntime (memory deps installed into a side directory, not the repo).
Worktree at PR head `9744d2a1` (base `e995eb9f`).
`headroom/_core.abi3.so` was copied (untracked, not committed) from the
official `headroom_ai-0.41.0-cp310-abi3-manylinux_2_28_x86_64.whl`
because the checkout has no built Rust core.

Stub upstream (`stub_mem.py`, 127.0.0.1:18796): answers every `POST
/v1/messages` with a normal 200 message and logs the tool names and the
last user message it received.
Proxy: the real server (`run_server`) with `memory_enabled=True`,
`memory_storage_mode="global"`, local backend with the ONNX embedder,
`ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18796`,
`HEADROOM_MEMORY_INJECTION_MODE=tool`, port 18795, `--no-cache
--no-rate-limit` equivalents. (`headroom proxy --memory` was not used:
the CLI needs the optional `mcp` package, which is not installed in this
environment. `mem_proxy.py` builds the same `ProxyConfig` and calls
`run_server`.)
Memory content: before each run, `mem_seed.py` saves one memory for user
`proofuser` ("The user prefers PostgreSQL 16 with pgvector for every
database.") through the real `MemoryHandler`/`LocalBackend`; a search
for it returns 1 hit.
Client (`mem_client.py`): one non-streaming `POST /v1/messages` with
`x-headroom-user-id: proofuser`, the message "Which database do I
prefer?" and one client tool `get_weather` (the memory tools are
injected only when the client declares tools).

Before = `upstream/main` (`e995eb9f`). After = PR head (`9744d2a1`).

### Before (upstream/main)

```
git head: e995eb9f  HEADROOM_MEMORY_INJECTION_MODE=tool
client: HTTP 500 Internal Server Error
ERROR - event=proxy_inbound_request_aborted ... path=/v1/messages reason=ValueError
ValueError: Invalid HEADROOM_MEMORY_INJECTION_MODE='tool'; expected 'live_zone_tail' or 'disabled'
--- stub upstream received
(nothing received)
```

Every request fails at `get_memory_injection_mode()`
(`headroom/proxy/helpers.py:267` ->
`memory_injection_mode_policy.py:19`) and never reaches the upstream.

### After (PR head 9744d2a1)

```
git head: 9744d2a1  HEADROOM_MEMORY_INJECTION_MODE=tool
client: HTTP 200 {"id": "msg_ok", "type": "message", "role": "assistant", "model": "claude-sonnet-4-5", "content": [{"type": "text", "text": "ok"}], "stop_reason": "end_turn", "usage": {"input_tokens": 10, "output_tokens": 2}}
INFO - event=memory_mode_skip mode=tool user_id=proofuser reason=tool_mode_no_auto_injection
INFO - [hr_1792672967_000001] Memory: Injected tools: ['memory_save', 'memory_search', 'memory_update', 'memory_delete', 'memory_list']
--- stub upstream received
STUB-UPSTREAM POST /v1/messages tools=['get_weather', 'headroom_retrieve', 'memory_delete', 'memory_list', 'memory_save', 'memory_search', 'memory_update']
STUB-UPSTREAM last_user_content="Which database do I prefer?"
```

The request succeeds, the upstream receives the five memory tool
definitions next to the client's own tool, and the user message is
forwarded unchanged (no `## Relevant Memories` block; `grep -c "Relevant
Memories"` on the stub log = 0).

### Control 1: default mode on the PR head (shows the seeded memory is
retrievable)

Same setup, `HEADROOM_MEMORY_INJECTION_MODE=live_zone_tail`:

```
INFO - event=memory_inject user=proofuser scope=global count=1 chars=793 budget_tokens=1024
INFO - event=memory_injection ... decision=injected_live_zone_tail bytes_injected=793
STUB-UPSTREAM ... tools=['get_weather', 'headroom_retrieve', 'memory_delete', 'memory_list', 'memory_save', 'memory_search', 'memory_update']
STUB-UPSTREAM last_user_content="Which database do I prefer?\n\n## Relevant Memories (scope: global)\n\nThese are READ-ONLY entries recalled from prior sessions ..."
```

So with the same memory and the same request, `live_zone_tail` injects
the memory context into the user message, and `tool` does not.

### Control 2: only the first commit (62ec86b5, "accept tool")

Same setup, `HEADROOM_MEMORY_INJECTION_MODE=tool`:

```
git head: 62ec86b5  HEADROOM_MEMORY_INJECTION_MODE=tool
client: HTTP 200 {...}
STUB-UPSTREAM POST /v1/messages tools=['get_weather', 'headroom_retrieve']
STUB-UPSTREAM last_user_content="Which database do I prefer?"
```

The request no longer fails, but the memory tool definitions are
missing. This is the case the second commit (`9744d2a1`) fixes.

### Commands

```
python -P mem_seed.py <db>                     # PYTHONPATH=.:deps-mem, seeds one memory for proofuser
python -P stub_mem.py 18796 stub.log &
HEADROOM_MEMORY_INJECTION_MODE=tool ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18796 PYTHONPATH=.:deps-mem \
  python -P mem_proxy.py 18795 <db> &
python -P mem_client.py 18795
```

Scripts and raw outputs: `/var/tmp/cb-pub/proof/` (WSL): `run_mem.sh`,
`mem_proxy.py`, `mem_seed.py`, `stub_mem.py`, `mem_client.py`,
`mem_{before,after,control,commit1}.proxy.log`,
`mem_{before,after,control,commit1}.stub.log`, `before_mem.out`,
`after_mem.out`, `control_mem.out`, `commit1_mem.out`.

### Not tested

- Real Anthropic API (stub upstream only).
- Streaming requests, the OpenAI Chat/Responses handlers, Gemini, and
the WebSocket path; only Anthropic `POST /v1/messages` non-streaming was
run.
- `use_native_tool` (native `memory_20250818` tool) and
`ProxyConfig.memory_mode="tool"`; only the env var was exercised.
- Memory tool execution (a `memory_save` call round trip) in tool mode.
- The Rust core is the 0.41.0 wheel's, not one built from this checkout.
- `headroom proxy --memory` via the CLI (the `mcp` package is not
installed here).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no (default stays `live_zone_tail`;
only a previously rejected value is accepted)
- Kill switch / disable path: unset the env var
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (6
existing failures in test_proxy_system_prompt_immutable.py are identical
on unmodified upstream/main in my environment)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

If maintainers would rather drop `tool` than support it, the alternative
is removing it from the resolver and the docs.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:44:42 -07:00

310 lines
11 KiB
Python

"""Tests for :class:`headroom.proxy.memory_decision.MemoryDecision`.
The point of this file is the *contract* — every behavioural assertion
here is the canonical answer to "should this request have memory
context injected into it?" that today is computed inline across the
proxy's handlers with subtle drift.
Specifically locks (post-PR-this):
* Sites 1/2/3 (Anthropic ``/v1/messages``, Gemini
``:generateContent``, OpenAI ``/v1/chat/completions``) MUST gate on
bypass — pre-this-PR they didn't, so memory injection silently
mutated requests under ``x-headroom-bypass: true``.
* Site 4 (OpenAI ``/v1/responses``) already gated; locked here.
* Site 6 (OpenAI WS ``/v1/responses``) already gated; locked here.
* ``MemoryMode`` values (``auto_tail`` / ``tool``) and the env-driven
``HEADROOM_MEMORY_INJECTION_MODE`` (``disabled`` / ``auto_tail`` /
``tool``) all surface as explicit ``skip_reason`` values, not as
hidden conditional code.
The decision is **input-side only** — it gates whether mutation of the
request bytes happens. It does NOT gate background memory STORAGE
(traffic-learner runs on a separate path and is unaffected by this
decision).
"""
from __future__ import annotations
from dataclasses import FrozenInstanceError
from types import SimpleNamespace
from typing import Any
from headroom.proxy.memory_decision import MemoryDecision
def _memory_handler() -> Any:
"""Minimal stand-in for the proxy's ``self.memory_handler``."""
return SimpleNamespace(name="local")
# ── Value-type contract ───────────────────────────────────────────────
def test_decision_is_frozen() -> None:
"""Frozen dataclass — mutation would let a handler patch the
decision after handing it to the funnel."""
d = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="auto_tail"
)
try:
d.inject = False # type: ignore[misc]
except FrozenInstanceError:
pass
else:
raise AssertionError("MemoryDecision must be frozen")
def test_decision_is_value_equal() -> None:
"""Two decisions from identical inputs compare equal."""
a = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="auto_tail"
)
b = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="auto_tail"
)
assert a == b
# ── Precedence: bypass > no_handler > no_user_id > mode > INJECT ─────
def test_injects_when_every_gate_open() -> None:
"""Happy path: no bypass, handler wired, user_id set, mode=auto_tail."""
d = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="auto_tail"
)
assert d.inject is True
assert d.skip_reason is None
def test_bypass_header_wins_over_every_other_gate() -> None:
"""``x-headroom-bypass: true`` is the user's "do not touch my
bytes" signal — highest priority, even when memory is otherwise
fully wired. Memory injection mutates the request bytes; bypass
must skip it. (This was the 3-bug Gemini-class problem pre-PR
on Anthropic, OpenAI chat, Gemini.)"""
d = MemoryDecision.decide(
headers={"x-headroom-bypass": "true"},
memory_handler=_memory_handler(),
memory_user_id="u1",
mode_name="auto_tail",
)
assert d.inject is False
assert d.skip_reason == "bypass_header"
def test_passthrough_mode_header_also_triggers_bypass_skip() -> None:
"""``x-headroom-mode: passthrough`` is the alternate spelling of
the bypass signal — mirrors _headroom_bypass_enabled semantics."""
d = MemoryDecision.decide(
headers={"x-headroom-mode": "passthrough"},
memory_handler=_memory_handler(),
memory_user_id="u1",
mode_name="auto_tail",
)
assert d.inject is False
assert d.skip_reason == "bypass_header"
def test_no_handler_is_skip() -> None:
"""No memory backend configured → ``no_handler`` reason."""
d = MemoryDecision.decide(
headers={}, memory_handler=None, memory_user_id="u1", mode_name="auto_tail"
)
assert d.inject is False
assert d.skip_reason == "no_handler"
def test_no_user_id_is_skip() -> None:
"""Memory wired but per-request user_id missing → ``no_user_id``."""
d = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id=None, mode_name="auto_tail"
)
assert d.inject is False
assert d.skip_reason == "no_user_id"
def test_empty_user_id_string_is_skip() -> None:
"""Empty string for user_id is treated as missing."""
d = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="", mode_name="auto_tail"
)
assert d.inject is False
assert d.skip_reason == "no_user_id"
def test_mode_disabled_is_skip() -> None:
"""Operator override via HEADROOM_MEMORY_INJECTION_MODE=disabled."""
d = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="disabled"
)
assert d.inject is False
assert d.skip_reason == "mode_disabled"
def test_mode_tool_keeps_tools_registered() -> None:
"""TOOL mode: context auto-injection is off (see
MemoryHandler.search_and_format_context) but the decision stays open
so the memory tools are still registered for the agent."""
d = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="tool"
)
assert d.inject is True
assert d.skip_reason is None
# ── Precedence ordering when multiple gates close ────────────────────
def test_bypass_beats_no_handler() -> None:
"""When both bypass AND no_handler would skip, surface bypass —
user's explicit signal is the more informative dashboard slice."""
d = MemoryDecision.decide(
headers={"x-headroom-bypass": "true"},
memory_handler=None,
memory_user_id=None,
mode_name="disabled",
)
assert d.skip_reason == "bypass_header"
def test_no_handler_beats_no_user_id() -> None:
"""no_handler is the more fundamental failure (no backend wired);
no_user_id only matters when a handler exists."""
d = MemoryDecision.decide(
headers={}, memory_handler=None, memory_user_id=None, mode_name="auto_tail"
)
assert d.skip_reason == "no_handler"
def test_no_user_id_beats_mode() -> None:
"""A request with no user_id can't be served regardless of mode."""
d = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id=None, mode_name="disabled"
)
assert d.skip_reason == "no_user_id"
def test_mode_disabled_beats_mode_tool() -> None:
"""``disabled`` is operator-level kill; ``tool`` is mode pref.
Disabled is the more emphatic signal."""
# Construct two separate decisions on identical inputs but mode varying;
# check that each produces its own reason (no cross-contamination).
d_dis = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="disabled"
)
d_tool = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="tool"
)
assert d_dis.skip_reason == "mode_disabled"
assert d_tool.skip_reason is None
# ── Observability fields ─────────────────────────────────────────────
def test_observability_booleans_populated_when_injecting() -> None:
"""Even on the happy path, every constituent is exposed so debug
tooling can answer "what did the decision see?" without re-running."""
d = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="auto_tail"
)
assert d.bypass_header_set is False
assert d.memory_handler_present is True
assert d.memory_user_id_present is True
assert d.mode_name == "auto_tail"
def test_observability_booleans_populated_when_skipping() -> None:
"""Same on the skip path — every constituent must be visible."""
d = MemoryDecision.decide(
headers={"x-headroom-bypass": "true"},
memory_handler=None,
memory_user_id="u1",
mode_name="auto_tail",
)
assert d.bypass_header_set is True
assert d.memory_handler_present is False
assert d.memory_user_id_present is True
assert d.mode_name == "auto_tail"
# ── apply_to_tags — mirror CompressionDecision pattern ───────────────
def test_apply_to_tags_stamps_reason_when_skipping() -> None:
"""Skip decisions surface ``memory_skip_reason`` in tags so the
dashboard can slice memory-blind traffic by cause."""
d = MemoryDecision.decide(
headers={"x-headroom-bypass": "true"},
memory_handler=_memory_handler(),
memory_user_id="u1",
mode_name="auto_tail",
)
tags: dict[str, str] = {}
d.apply_to_tags(tags)
assert tags == {"memory_skip_reason": "bypass_header"}
def test_apply_to_tags_is_a_noop_when_injecting() -> None:
"""No tag when injecting — absence is the signal for "memory was
used". Avoids spurious ``memory_skip_reason=None`` strings."""
d = MemoryDecision.decide(
headers={}, memory_handler=_memory_handler(), memory_user_id="u1", mode_name="auto_tail"
)
tags: dict[str, str] = {"client": "codex"}
d.apply_to_tags(tags)
assert tags == {"client": "codex"}
assert "memory_skip_reason" not in tags
def test_apply_to_tags_preserves_pre_existing_entries() -> None:
"""Existing tags (client, passthrough_reason from CompressionDecision)
must survive unchanged."""
d = MemoryDecision.decide(
headers={}, memory_handler=None, memory_user_id="u1", mode_name="auto_tail"
)
tags: dict[str, str] = {"client": "claude-code", "passthrough_reason": "bypass_header"}
d.apply_to_tags(tags)
assert tags == {
"client": "claude-code",
"passthrough_reason": "bypass_header",
"memory_skip_reason": "no_handler",
}
def test_apply_to_tags_for_every_skip_reason() -> None:
"""Every skip reason name must round-trip through apply_to_tags."""
cases: dict[str, dict[str, Any]] = {
"bypass_header": {
"headers": {"x-headroom-bypass": "true"},
"memory_handler": _memory_handler(),
"memory_user_id": "u1",
"mode_name": "auto_tail",
},
"no_handler": {
"headers": {},
"memory_handler": None,
"memory_user_id": "u1",
"mode_name": "auto_tail",
},
"no_user_id": {
"headers": {},
"memory_handler": _memory_handler(),
"memory_user_id": None,
"mode_name": "auto_tail",
},
"mode_disabled": {
"headers": {},
"memory_handler": _memory_handler(),
"memory_user_id": "u1",
"mode_name": "disabled",
},
}
for expected, kwargs in cases.items():
d = MemoryDecision.decide(**kwargs)
tags: dict[str, str] = {}
d.apply_to_tags(tags)
assert tags.get("memory_skip_reason") == expected, expected