Commit Graph
2 Commits
Author SHA1 Message Date
f89c9282a8 fix(proxy): accept HEADROOM_MEMORY_INJECTION_MODE=tool (#4395)
## Description

`memory_decision.py` handles mode `tool`, but
`resolve_memory_injection_mode` rejected it with `ValueError`, so the
value could not be used. The resolver now accepts `tool`; the
configuration docs list it.

Second commit, after review (Devin, comment 4348232164): in `tool` mode
the old `MemoryDecision` returned `inject=False`, and the handlers gate
the memory tool definitions on that same flag, so the agent would have
had no `memory_search` to call. The intent of `tool` (commit 71d5a7b5:
no auto-injected context, the model calls the memory tools itself) is
now implemented: the decision stays open for `tool`, and
`MemoryHandler.search_and_format_context` skips the context lookup, as
it already did for `MemoryMode.TOOL`.

Fixes #4391

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/memory_injection_mode_policy.py`: add `"tool"` to the
literal type and the accepted values; update the error text.
- `headroom/proxy/helpers.py`: comment documents `tool`.
- `docs/content/docs/configuration.mdx`: list `tool` for
`HEADROOM_MEMORY_INJECTION_MODE`.
- `tests/test_memory_injection_mode_policy.py`: resolver accepts `tool`
(case/space insensitive); `get_memory_injection_mode()` returns `tool`
from the env var.
- `headroom/proxy/memory_decision_policy.py`, `memory_decision.py`:
`tool` no longer produces a skip (`mode_tool` removed); `inject` stays
True so the handlers register the memory tools.
- `headroom/proxy/memory_handler.py`: `search_and_format_context`
returns None when `HEADROOM_MEMORY_INJECTION_MODE=tool` (before any
backend lookup).
- `tests/test_memory_decision.py`,
`tests/test_memory_decision_policy.py`: `tool` expectations changed to
`inject=True`, no skip reason; new test in
`test_memory_injection_mode_policy.py` covers the context skip and the
open decision.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# with the policy change reverted (tests/test_memory_injection_mode_policy.py)
E   ValueError: Invalid HEADROOM_MEMORY_INJECTION_MODE='tool'; expected 'live_zone_tail' or 'disabled'
FAILED tests/test_memory_injection_mode_policy.py::test_resolve_memory_injection_mode_accepts_known_values_case_insensitively
FAILED tests/test_memory_injection_mode_policy.py::test_get_memory_injection_mode_accepts_tool_from_env
2 failed, 2 passed

# second commit: new test with the src changes of that commit reverted
E   assert decision.inject is True / MemoryDecision(inject=False, skip_reason='mode_tool', ...)
1 failed, 4 passed
# second commit applied
$ pytest tests/test_memory_injection_mode_policy.py tests/test_memory_decision.py tests/test_memory_decision_policy.py tests/test_memory_handler_concurrent_init.py
37 passed, 1 skipped

# first commit, the four files listed under Real Behavior Proof
6 failed, 25 passed  (all 6 failures in test_proxy_system_prompt_immutable.py, same 6 on unmodified upstream/main)

$ ruff check <changed py files>
All checks passed!
$ ruff format --check <changed py files>
3 files already formatted
```

## Real Behavior Proof

- Environment: Linux (WSL2), Python 3.12.15, the real proxy server
(`run_server`) with persistent memory on (local backend, ONNX embedder),
Rust core from the headroom-ai 0.41.0 wheel, a local stub upstream on
127.0.0.1
- Exact command / steps: seed one memory for a test user, start the
proxy with `HEADROOM_MEMORY_INJECTION_MODE=tool`, send one non-streaming
`POST /v1/messages` that declares one client tool; repeat on
upstream/main and with `live_zone_tail` as controls (details below)
- Observed result: upstream/main returns HTTP 500 (`ValueError` from the
mode resolver); this branch returns HTTP 200, the stub receives the five
memory tools and the user message without a `## Relevant Memories` block
- Not tested: real Anthropic, streaming, the OpenAI/Gemini handlers and
the WebSocket path, `use_native_tool`, a memory tool-call round trip, a
Rust core built from this checkout

Details:

Setup: WSL2 Ubuntu (Linux 6.18 x86_64), Python 3.12.15, uvicorn 0.54.0,
fastapi 0.158.1, starlette 1.9.0, numpy 2.5.4, sqlite-vec 0.1.9,
onnxruntime (memory deps installed into a side directory, not the repo).
Worktree at PR head `9744d2a1` (base `e995eb9f`).
`headroom/_core.abi3.so` was copied (untracked, not committed) from the
official `headroom_ai-0.41.0-cp310-abi3-manylinux_2_28_x86_64.whl`
because the checkout has no built Rust core.

Stub upstream (`stub_mem.py`, 127.0.0.1:18796): answers every `POST
/v1/messages` with a normal 200 message and logs the tool names and the
last user message it received.
Proxy: the real server (`run_server`) with `memory_enabled=True`,
`memory_storage_mode="global"`, local backend with the ONNX embedder,
`ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18796`,
`HEADROOM_MEMORY_INJECTION_MODE=tool`, port 18795, `--no-cache
--no-rate-limit` equivalents. (`headroom proxy --memory` was not used:
the CLI needs the optional `mcp` package, which is not installed in this
environment. `mem_proxy.py` builds the same `ProxyConfig` and calls
`run_server`.)
Memory content: before each run, `mem_seed.py` saves one memory for user
`proofuser` ("The user prefers PostgreSQL 16 with pgvector for every
database.") through the real `MemoryHandler`/`LocalBackend`; a search
for it returns 1 hit.
Client (`mem_client.py`): one non-streaming `POST /v1/messages` with
`x-headroom-user-id: proofuser`, the message "Which database do I
prefer?" and one client tool `get_weather` (the memory tools are
injected only when the client declares tools).

Before = `upstream/main` (`e995eb9f`). After = PR head (`9744d2a1`).

### Before (upstream/main)

```
git head: e995eb9f  HEADROOM_MEMORY_INJECTION_MODE=tool
client: HTTP 500 Internal Server Error
ERROR - event=proxy_inbound_request_aborted ... path=/v1/messages reason=ValueError
ValueError: Invalid HEADROOM_MEMORY_INJECTION_MODE='tool'; expected 'live_zone_tail' or 'disabled'
--- stub upstream received
(nothing received)
```

Every request fails at `get_memory_injection_mode()`
(`headroom/proxy/helpers.py:267` ->
`memory_injection_mode_policy.py:19`) and never reaches the upstream.

### After (PR head 9744d2a1)

```
git head: 9744d2a1  HEADROOM_MEMORY_INJECTION_MODE=tool
client: HTTP 200 {"id": "msg_ok", "type": "message", "role": "assistant", "model": "claude-sonnet-4-5", "content": [{"type": "text", "text": "ok"}], "stop_reason": "end_turn", "usage": {"input_tokens": 10, "output_tokens": 2}}
INFO - event=memory_mode_skip mode=tool user_id=proofuser reason=tool_mode_no_auto_injection
INFO - [hr_1792672967_000001] Memory: Injected tools: ['memory_save', 'memory_search', 'memory_update', 'memory_delete', 'memory_list']
--- stub upstream received
STUB-UPSTREAM POST /v1/messages tools=['get_weather', 'headroom_retrieve', 'memory_delete', 'memory_list', 'memory_save', 'memory_search', 'memory_update']
STUB-UPSTREAM last_user_content="Which database do I prefer?"
```

The request succeeds, the upstream receives the five memory tool
definitions next to the client's own tool, and the user message is
forwarded unchanged (no `## Relevant Memories` block; `grep -c "Relevant
Memories"` on the stub log = 0).

### Control 1: default mode on the PR head (shows the seeded memory is
retrievable)

Same setup, `HEADROOM_MEMORY_INJECTION_MODE=live_zone_tail`:

```
INFO - event=memory_inject user=proofuser scope=global count=1 chars=793 budget_tokens=1024
INFO - event=memory_injection ... decision=injected_live_zone_tail bytes_injected=793
STUB-UPSTREAM ... tools=['get_weather', 'headroom_retrieve', 'memory_delete', 'memory_list', 'memory_save', 'memory_search', 'memory_update']
STUB-UPSTREAM last_user_content="Which database do I prefer?\n\n## Relevant Memories (scope: global)\n\nThese are READ-ONLY entries recalled from prior sessions ..."
```

So with the same memory and the same request, `live_zone_tail` injects
the memory context into the user message, and `tool` does not.

### Control 2: only the first commit (62ec86b5, "accept tool")

Same setup, `HEADROOM_MEMORY_INJECTION_MODE=tool`:

```
git head: 62ec86b5  HEADROOM_MEMORY_INJECTION_MODE=tool
client: HTTP 200 {...}
STUB-UPSTREAM POST /v1/messages tools=['get_weather', 'headroom_retrieve']
STUB-UPSTREAM last_user_content="Which database do I prefer?"
```

The request no longer fails, but the memory tool definitions are
missing. This is the case the second commit (`9744d2a1`) fixes.

### Commands

```
python -P mem_seed.py <db>                     # PYTHONPATH=.:deps-mem, seeds one memory for proofuser
python -P stub_mem.py 18796 stub.log &
HEADROOM_MEMORY_INJECTION_MODE=tool ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18796 PYTHONPATH=.:deps-mem \
  python -P mem_proxy.py 18795 <db> &
python -P mem_client.py 18795
```

Scripts and raw outputs: `/var/tmp/cb-pub/proof/` (WSL): `run_mem.sh`,
`mem_proxy.py`, `mem_seed.py`, `stub_mem.py`, `mem_client.py`,
`mem_{before,after,control,commit1}.proxy.log`,
`mem_{before,after,control,commit1}.stub.log`, `before_mem.out`,
`after_mem.out`, `control_mem.out`, `commit1_mem.out`.

### Not tested

- Real Anthropic API (stub upstream only).
- Streaming requests, the OpenAI Chat/Responses handlers, Gemini, and
the WebSocket path; only Anthropic `POST /v1/messages` non-streaming was
run.
- `use_native_tool` (native `memory_20250818` tool) and
`ProxyConfig.memory_mode="tool"`; only the env var was exercised.
- Memory tool execution (a `memory_save` call round trip) in tool mode.
- The Rust core is the 0.41.0 wheel's, not one built from this checkout.
- `headroom proxy --memory` via the CLI (the `mcp` package is not
installed here).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no (default stays `live_zone_tail`;
only a previously rejected value is accepted)
- Kill switch / disable path: unset the env var
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (6
existing failures in test_proxy_system_prompt_immutable.py are identical
on unmodified upstream/main in my environment)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

If maintainers would rather drop `tool` than support it, the alternative
is removing it from the resolver and the docs.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:44:42 -07:00
chopratejas 71d5a7b545 fix(proxy): MemoryDecision contract + 3 bypass bugs + drop 500-char query cap
Three bug classes fixed plus three architectural extension points,
together making the memory subsystem uniform across all five sites
and ready for future Mem0/Letta/Cognee backend integration.

## Bug fixes

* **3 sites silently ignored `x-headroom-bypass: true`** —
  ``anthropic.py:1303``, ``openai.py:1620`` (chat), ``gemini.py:382``
  injected memory under bypass, mutating request bytes when the user
  explicitly asked for byte-faithful passthrough. Now gated on
  ``MemoryDecision.decide(...)`` which honours bypass uniformly.

* **500-char query truncation** — ``memory_handler._extract_user_query``
  capped at 500 chars, silently throwing away signal. None of Letta /
  Mem0 / Cognee / Supermemory truncate. Removed; the embedding model
  handles its own window.

* **Gemini had no timeout** on ``search_and_format_context`` — the
  only chat handler without one. A slow backend could stall requests.
  Added ``asyncio.wait_for`` matching Anthropic + OpenAI Chat +
  Responses.

* **WS injected into ``body["instructions"]``** — the system /
  cache-hot-zone field, violating invariant I2 (all other handlers
  inject at user-message tail). Switched to ``ws_response_body["input"]``
  for string-shaped input; list-shaped input deferred to the Rust
  handler with a clear log.

## New value types (extension points)

* ``MemoryDecision`` — frozen dataclass + factory. Five-way skip
  reason enum (``bypass_header`` / ``no_handler`` / ``no_user_id`` /
  ``mode_disabled`` / ``mode_tool``). ``apply_to_tags()`` surfaces
  the skip reason in ``RequestOutcome.tags["memory_skip_reason"]``
  — dashboards can now slice memory-blind traffic by cause.

* ``MemoryQuery`` — multi-source retrieval query. ``from_messages()``
  walks the conversation and extracts latest user text + recent tool
  outputs + recent assistant turns at FULL fidelity (no truncation).
  Handles both OpenAI-shape ``role: tool`` and Anthropic-shape
  ``tool_result`` content blocks. ``to_embedding_input()`` produces
  a delimited concatenation the embedder sees as structured context.

* ``MemoryInjectionBudget`` — uniform token / entry / similarity
  bound on the formatted injection block. Pre-this-PR no cap (~4000
  tokens could land per request). Default 1024 tokens / 10 entries /
  0.3 similarity floor. ``apply_to_text()`` truncates at line
  boundaries so dashboard renders intact bullet points.

## Migration scope — all 5 sites uniform at the GATE level

| Site | Handler | Pre-PR gate | Post-PR gate |
|---|---|---|---|
| 1 | anthropic.py | `memory_handler and memory_user_id` | `memory_decision.inject` |
| 2 | gemini.py | `memory_handler and memory_user_id` | `memory_decision.inject` |
| 3 | openai.py chat | `memory_handler and memory_user_id` | `memory_decision.inject` |
| 4 | openai.py Responses | `memory_handler and memory_user_id and not _bypass` | `responses_memory_decision.inject` |
| 6 | openai.py WS | `memory_handler and body and not _ws_bypass` | `ws_memory_decision.inject` |

Site 5 (Responses bypass-elif log-only branch) is preserved verbatim.

## Deliberately deferred (separate PRs)

* **Memory injection order inversion** — sites 4 and 6 inject
  BEFORE compression; sites 1/2/3 inject AFTER. Moving 4 + 6 to
  post-compression needs its own focused cache-stability testing.
* **Importance scoring** — recency × source × access-count.
* **Per-memory atomize-and-split** — Mem0/Supermemory pattern.
* **AST-aware code chunking for tool outputs** — Supermemory's
  code-chunk approach.

The contracts shipped here (``MemoryQuery`` + ``MemoryInjectionBudget``)
are the extension points those will plug into.

## Test coverage

* 20 new tests on ``MemoryDecision``
* 14 new tests on ``MemoryQuery`` (full-fidelity, multi-source)
* 10 new tests on ``MemoryInjectionBudget``
* 3 new AST contract tests (no raw gate; no system writes; every
  search call passes ``query=``)
* All existing memory + cache-stability tests still pass (222 passed)

## Rust portability

Every new value type ports cleanly to a frozen Rust struct. Pure
functions, no I/O, no global state. Same Python ↔ Rust parity-test
pattern that ``CompressionDecision`` already uses.

## Zero-regression contract

Existing chat/completion harnesses (Claude Code, Codex, Cursor,
Continue, Aider) see ZERO wire-byte changes when bypass is NOT set.
When bypass IS set, the 3 chat handlers now correctly skip memory
injection — that's the bug fix, not a regression.
2026-05-19 11:13:52 -05:00