mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
On main,
`test_memory_call_the_proxy_did_not_inject_reaches_the_client[not_injected-anthropic]`
and `[not_injected-anthropic_stream]` fail when
`tests/test_memory_tool_user_scope.py` runs earlier in the same process
(CI shard `test (3)`).
This is a test-isolation problem, not a regression in the proxy. Memory
tool definitions are sticky per session (`SessionToolTracker`, a
process-wide tracker in `headroom/proxy/helpers.py`). The user-scope
test sends an identical Anthropic request with memory tools injected.
When the later `not_injected` case sends the same request, the session
tracker replays those tool definitions, so the proxy really does offer
`memory_delete` upstream on that request. Running the call is then the
intended behaviour. Each PR passed alone because nothing else in its own
shard had injected tools for that request first.
The fix resets the tracker before and after each test in
`tests/test_memory_tools_only_when_injected.py`. The same reset is
already used in `test_memory_tool_stream.py` and
`test_corrupt_golden_bytes_recovery.py`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `tests/test_memory_tools_only_when_injected.py`: an autouse fixture
calls `_reset_session_tool_tracker_for_test()` around each test. No
production code changes.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) (tests-only change)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Same order as CI (`tests/test_memory_tool_user_scope.py` then
`tests/test_memory_tools_only_when_injected.py`), on origin/main
fda75de5:
```text
FAILED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[not_injected-anthropic]
FAILED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[not_injected-anthropic_stream]
2 failed, 16 passed
```
On this branch:
```text
18 passed
```
Injected controls first, then the not-injected cases, in one process: 4
failed on main, `11 passed` on this branch.
`tests/test_memory_tool_user_scope.py
tests/test_memory_tools_only_when_injected.py tests/test_memory/
tests/test_memory_handler_native_ops.py`: `780 passed, 2 skipped`. ruff
check and format are clean.
## Real Behavior Proof
- Environment: macOS, Python 3.12, in-process `create_app` with a
`MockTransport` upstream, `LITELLM_LOCAL_MODEL_COST_MAP=True`.
- Exact command / steps: run the two test files above in the CI order on
main and on this branch.
- Observed result: on main the not-injected Anthropic cases fail with
`assert ['delete'] == []` because the replayed session tools are
injected; on this branch all pass.
- Not tested: n/a (test-only change).
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: no.
- Kill switch / disable path: n/a.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Sticky replay only matters within a single process that serves the same
session under different memory settings, which a running proxy (with one
fixed config) does not do.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4873"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
422 lines
14 KiB
Python
422 lines
14 KiB
Python
"""The proxy runs only the memory tools it injected on the request.
|
|
|
|
A memory-named call the proxy did not offer (``--no-memory-tools``) or that
|
|
the client declared itself belongs to the client: it reaches the client
|
|
unchanged, the memory backend is not touched, and no continuation is sent
|
|
upstream. Calls to tools the proxy did inject still run on the proxy.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import json
|
|
import sys
|
|
from types import SimpleNamespace
|
|
from typing import Any
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
import httpx
|
|
import pytest
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.helpers import _reset_session_tool_tracker_for_test
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _fresh_tool_tracker():
|
|
# Memory tools are sticky per session in a process-wide tracker: a test
|
|
# that injected them would make a later identical request replay them,
|
|
# and the proxy then correctly runs the calls it offered.
|
|
_reset_session_tool_tracker_for_test()
|
|
yield
|
|
_reset_session_tool_tracker_for_test()
|
|
|
|
|
|
class _FakeBackend:
|
|
def __init__(self) -> None:
|
|
self.calls: list[str] = []
|
|
|
|
async def list_memories(self, *, user_id: str, limit: int) -> list[Any]:
|
|
self.calls.append("list")
|
|
return [SimpleNamespace(id="mem-1", content="stored", created_at=None)]
|
|
|
|
async def delete_memory(self, memory_id: str, *args: Any, **kwargs: Any) -> bool:
|
|
self.calls.append("delete")
|
|
return True
|
|
|
|
|
|
def _make_app(tmp_path, *, inject_tools: bool) -> tuple[Any, _FakeBackend]:
|
|
config = ProxyConfig(
|
|
openai_api_url="https://openai.invalid",
|
|
anthropic_api_url="https://anthropic.invalid",
|
|
optimize=False,
|
|
image_optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
discover_pipeline_extensions=False,
|
|
compression_max_workers=1,
|
|
memory_enabled=True,
|
|
memory_storage_mode="global",
|
|
memory_inject_tools=inject_tools,
|
|
memory_inject_context=False,
|
|
memory_db_path=str(tmp_path / "memory.db"),
|
|
)
|
|
app = create_app(config)
|
|
handler = app.state.proxy.memory_handler
|
|
backend = _FakeBackend()
|
|
handler._backend = backend
|
|
handler._initialized = True
|
|
router = handler._router
|
|
if router is not None:
|
|
router.backend_for = lambda ctx: (backend, router.scope_for(ctx))
|
|
return app, backend
|
|
|
|
|
|
# --- upstream turns ---------------------------------------------------------
|
|
|
|
|
|
def _chat_turn(name: str | None) -> dict[str, Any]:
|
|
if name is None:
|
|
message: dict[str, Any] = {"role": "assistant", "content": "done"}
|
|
else:
|
|
message = {
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_1",
|
|
"type": "function",
|
|
"function": {"name": name, "arguments": '{"memory_id": "mem-1"}'},
|
|
}
|
|
],
|
|
}
|
|
return {
|
|
"id": "chatcmpl-1",
|
|
"object": "chat.completion",
|
|
"model": "gpt-4o-mini",
|
|
"choices": [{"index": 0, "message": message, "finish_reason": "stop"}],
|
|
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
|
|
}
|
|
|
|
|
|
def _responses_item(name: str) -> dict[str, Any]:
|
|
return {
|
|
"type": "function_call",
|
|
"id": "fc_1",
|
|
"call_id": "call_1",
|
|
"name": name,
|
|
"arguments": '{"memory_id": "mem-1"}',
|
|
"status": "completed",
|
|
}
|
|
|
|
|
|
def _responses_turn(name: str | None) -> dict[str, Any]:
|
|
item = (
|
|
_responses_item(name)
|
|
if name
|
|
else {
|
|
"type": "message",
|
|
"id": "msg_1",
|
|
"role": "assistant",
|
|
"status": "completed",
|
|
"content": [{"type": "output_text", "text": "done", "annotations": []}],
|
|
}
|
|
)
|
|
return {
|
|
"id": "resp_1",
|
|
"object": "response",
|
|
"status": "completed",
|
|
"model": "gpt-4o-mini",
|
|
"output": [item],
|
|
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2},
|
|
}
|
|
|
|
|
|
def _anthropic_turn(name: str | None) -> dict[str, Any]:
|
|
if name is None:
|
|
content: list[dict[str, Any]] = [{"type": "text", "text": "done"}]
|
|
else:
|
|
content = [
|
|
{"type": "tool_use", "id": "toolu_1", "name": name, "input": {"memory_id": "mem-1"}}
|
|
]
|
|
return {
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"model": "claude-sonnet-4-5",
|
|
"content": content,
|
|
"stop_reason": "tool_use" if name else "end_turn",
|
|
"stop_sequence": None,
|
|
"usage": {"input_tokens": 1, "output_tokens": 1},
|
|
}
|
|
|
|
|
|
def _anthropic_sse(name: str) -> bytes:
|
|
turn = _anthropic_turn(name)
|
|
events = [
|
|
("message_start", {"type": "message_start", "message": {**turn, "content": []}}),
|
|
(
|
|
"content_block_start",
|
|
{
|
|
"type": "content_block_start",
|
|
"index": 0,
|
|
"content_block": {"type": "tool_use", "id": "toolu_1", "name": name, "input": {}},
|
|
},
|
|
),
|
|
(
|
|
"content_block_delta",
|
|
{
|
|
"type": "content_block_delta",
|
|
"index": 0,
|
|
"delta": {"type": "input_json_delta", "partial_json": '{"memory_id": "mem-1"}'},
|
|
},
|
|
),
|
|
("content_block_stop", {"type": "content_block_stop", "index": 0}),
|
|
(
|
|
"message_delta",
|
|
{
|
|
"type": "message_delta",
|
|
"delta": {"stop_reason": "tool_use", "stop_sequence": None},
|
|
"usage": {"output_tokens": 1},
|
|
},
|
|
),
|
|
("message_stop", {"type": "message_stop"}),
|
|
]
|
|
return "".join(f"event: {e}\ndata: {json.dumps(d)}\n\n" for e, d in events).encode()
|
|
|
|
|
|
# --- HTTP paths -------------------------------------------------------------
|
|
|
|
_OPENAI_TOOL = {
|
|
"type": "function",
|
|
"function": {"name": "shell", "parameters": {"type": "object", "properties": {}}},
|
|
}
|
|
_OPENAI_CLIENT_MEMORY_DELETE = {
|
|
"type": "function",
|
|
"function": {"name": "memory_delete", "parameters": {"type": "object", "properties": {}}},
|
|
}
|
|
_RESPONSES_TOOL = {"type": "function", "name": "shell", "parameters": {"type": "object"}}
|
|
_RESPONSES_CLIENT_MEMORY_DELETE = {
|
|
"type": "function",
|
|
"name": "memory_delete",
|
|
"parameters": {"type": "object"},
|
|
}
|
|
_ANTHROPIC_TOOL = {"name": "Bash", "input_schema": {"type": "object"}}
|
|
_ANTHROPIC_CLIENT_MEMORY_DELETE = {"name": "memory_delete", "input_schema": {"type": "object"}}
|
|
|
|
_PATHS: dict[str, dict[str, Any]] = {
|
|
"chat": {
|
|
"url": "/v1/chat/completions",
|
|
"headers": {"authorization": "Bearer FAKE-KEY"},
|
|
"body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}]},
|
|
"tool": _OPENAI_TOOL,
|
|
"client_memory_delete": _OPENAI_CLIENT_MEMORY_DELETE,
|
|
"turn": _chat_turn,
|
|
},
|
|
"responses": {
|
|
"url": "/v1/responses",
|
|
"headers": {"authorization": "Bearer FAKE-KEY"},
|
|
"body": {"model": "gpt-4o-mini", "input": "hi"},
|
|
"tool": _RESPONSES_TOOL,
|
|
"client_memory_delete": _RESPONSES_CLIENT_MEMORY_DELETE,
|
|
"turn": _responses_turn,
|
|
},
|
|
"anthropic": {
|
|
"url": "/v1/messages",
|
|
"headers": {"x-api-key": "FAKE-KEY", "anthropic-version": "2023-06-01"},
|
|
"body": {
|
|
"model": "claude-sonnet-4-5",
|
|
"max_tokens": 64,
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
},
|
|
"tool": _ANTHROPIC_TOOL,
|
|
"client_memory_delete": _ANTHROPIC_CLIENT_MEMORY_DELETE,
|
|
"turn": _anthropic_turn,
|
|
},
|
|
"anthropic_stream": {
|
|
"url": "/v1/messages",
|
|
"headers": {"x-api-key": "FAKE-KEY", "anthropic-version": "2023-06-01"},
|
|
"body": {
|
|
"model": "claude-sonnet-4-5",
|
|
"max_tokens": 64,
|
|
"stream": True,
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
},
|
|
"tool": _ANTHROPIC_TOOL,
|
|
"client_memory_delete": _ANTHROPIC_CLIENT_MEMORY_DELETE,
|
|
"turn": _anthropic_turn,
|
|
},
|
|
}
|
|
|
|
|
|
async def _post(app: Any, kind: str, tools: list[dict[str, Any]], call: str) -> tuple:
|
|
spec = _PATHS[kind]
|
|
upstream_bodies: list[dict[str, Any]] = []
|
|
|
|
def upstream(request: httpx.Request) -> httpx.Response:
|
|
upstream_bodies.append(json.loads(request.content))
|
|
first = len(upstream_bodies) == 1
|
|
if kind == "anthropic_stream" and first:
|
|
return httpx.Response(
|
|
200, content=_anthropic_sse(call), headers={"content-type": "text/event-stream"}
|
|
)
|
|
return httpx.Response(200, json=spec["turn"](call if first else None))
|
|
|
|
mock = httpx.AsyncClient(transport=httpx.MockTransport(upstream))
|
|
app.state.proxy.http_client = mock
|
|
app.state.proxy.http_client_h1 = mock
|
|
transport = httpx.ASGITransport(app=app, client=("127.0.0.1", 12345))
|
|
async with httpx.AsyncClient(transport=transport, base_url="http://testserver") as client:
|
|
response = await client.post(
|
|
spec["url"],
|
|
json={**spec["body"], "tools": tools},
|
|
headers={**spec["headers"], "x-headroom-user-id": "user-1"},
|
|
timeout=30,
|
|
)
|
|
await mock.aclose()
|
|
return response, upstream_bodies
|
|
|
|
|
|
@pytest.mark.parametrize("kind", list(_PATHS))
|
|
@pytest.mark.parametrize("case", ["not_injected", "client_declared"])
|
|
async def test_memory_call_the_proxy_did_not_inject_reaches_the_client(
|
|
tmp_path, monkeypatch, kind: str, case: str
|
|
) -> None:
|
|
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")
|
|
spec = _PATHS[kind]
|
|
app, backend = _make_app(tmp_path, inject_tools=case == "client_declared")
|
|
tools = [spec["tool"]]
|
|
if case == "client_declared":
|
|
tools.append(spec["client_memory_delete"])
|
|
|
|
response, upstream_bodies = await _post(app, kind, tools, "memory_delete")
|
|
|
|
assert response.status_code == 200
|
|
assert backend.calls == []
|
|
assert len(upstream_bodies) == 1
|
|
assert "memory_delete" in response.text
|
|
assert ("toolu_1" if kind.startswith("anthropic") else "call_1") in response.text
|
|
|
|
|
|
@pytest.mark.parametrize("kind", ["chat", "responses", "anthropic"])
|
|
async def test_memory_call_the_proxy_injected_still_runs(tmp_path, monkeypatch, kind: str) -> None:
|
|
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")
|
|
app, backend = _make_app(tmp_path, inject_tools=True)
|
|
|
|
response, upstream_bodies = await _post(app, kind, [_PATHS[kind]["tool"]], "memory_list")
|
|
|
|
assert response.status_code == 200
|
|
assert backend.calls == ["list"]
|
|
assert len(upstream_bodies) == 2
|
|
assert "memory_list" not in response.text
|
|
|
|
|
|
# --- Responses over WebSocket ---------------------------------------------
|
|
|
|
|
|
class _Upstream:
|
|
"""Upstream WS that answers the first response.create with one tool call."""
|
|
|
|
def __init__(self, call: str) -> None:
|
|
self.call = call
|
|
self.sent: list[dict[str, Any]] = []
|
|
self._events: asyncio.Queue[str] = asyncio.Queue()
|
|
|
|
async def __aenter__(self) -> _Upstream:
|
|
return self
|
|
|
|
async def __aexit__(self, *exc: object) -> None:
|
|
return None
|
|
|
|
async def send(self, payload: str) -> None:
|
|
self.sent.append(json.loads(payload))
|
|
rid = f"resp_{len(self.sent)}"
|
|
if len(self.sent) == 1:
|
|
item = _responses_item(self.call)
|
|
else:
|
|
item = _responses_turn(None)["output"][0]
|
|
for event in (
|
|
{"type": "response.created", "response": {"id": rid}},
|
|
{"type": "response.output_item.added", "item": item},
|
|
{"type": "response.output_item.done", "item": item},
|
|
{"type": "response.completed", "response": {"id": rid}},
|
|
):
|
|
await self._events.put(json.dumps(event))
|
|
|
|
async def close(self) -> None:
|
|
return None
|
|
|
|
def __aiter__(self) -> _Upstream:
|
|
return self
|
|
|
|
async def __anext__(self) -> str:
|
|
return await self._events.get()
|
|
|
|
|
|
def _ws_exchange(
|
|
app: Any, tools: list[dict[str, Any]], call: str
|
|
) -> tuple[list[dict[str, Any]], _Upstream]:
|
|
upstream = _Upstream(call)
|
|
|
|
async def connect(*_args: Any, **_kwargs: Any) -> _Upstream:
|
|
return upstream
|
|
|
|
websockets_module = MagicMock()
|
|
websockets_module.connect = connect
|
|
websockets_module.Subprotocol = str
|
|
frame = {
|
|
"type": "response.create",
|
|
"response": {"model": "gpt-4o-mini", "input": "hi", "tools": tools},
|
|
}
|
|
received: list[dict[str, Any]] = []
|
|
with patch.dict(sys.modules, {"websockets": websockets_module}):
|
|
client = TestClient(app, client=("127.0.0.1", 12345))
|
|
with client.websocket_connect(
|
|
"/v1/responses",
|
|
headers={"authorization": "Bearer FAKE-KEY", "x-headroom-user-id": "user-1"},
|
|
) as websocket:
|
|
websocket.send_text(json.dumps(frame))
|
|
while True:
|
|
event = json.loads(websocket.receive_text())
|
|
received.append(event)
|
|
if event.get("type") == "response.completed":
|
|
break
|
|
return received, upstream
|
|
|
|
|
|
@pytest.mark.parametrize("case", ["not_injected", "client_declared"])
|
|
def test_ws_memory_call_the_proxy_did_not_inject_reaches_the_client(
|
|
tmp_path, monkeypatch, case: str
|
|
) -> None:
|
|
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")
|
|
app, backend = _make_app(tmp_path, inject_tools=case == "client_declared")
|
|
tools = [_RESPONSES_TOOL]
|
|
if case == "client_declared":
|
|
tools.append(_RESPONSES_CLIENT_MEMORY_DELETE)
|
|
|
|
received, upstream = _ws_exchange(app, tools, "memory_delete")
|
|
|
|
assert backend.calls == []
|
|
assert len(upstream.sent) == 1
|
|
assert any(
|
|
event.get("type") == "response.output_item.done"
|
|
and event["item"].get("name") == "memory_delete"
|
|
for event in received
|
|
)
|
|
|
|
|
|
def test_ws_memory_call_the_proxy_injected_still_runs(tmp_path, monkeypatch) -> None:
|
|
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")
|
|
app, backend = _make_app(tmp_path, inject_tools=True)
|
|
|
|
received, upstream = _ws_exchange(app, [_RESPONSES_TOOL], "memory_list")
|
|
|
|
assert backend.calls == ["list"]
|
|
assert len(upstream.sent) == 2
|
|
assert not any(event.get("item", {}).get("name") == "memory_list" for event in received)
|