Commit Graph
16 Commits
Author SHA1 Message Date
fda75de505 fix(memory): run only the memory tools the proxy injected on the request (#4870)
## Description

With memory enabled (`--memory`, `--learn`, `wrap/init --memory`), the
proxy ran any upstream tool call named
`memory_list`/`memory_save`/`memory_search`/`memory_update`/`memory_delete`,
and hid it from the client. That held even when the proxy had not
offered those tools on the request (`--no-memory-tools`), or when the
client had declared a tool with that name itself.

The proxy now runs only the memory tools it injected on the current
request. Every other call reaches the client unchanged. The proxy does
not run it, does not withhold it, and sends no continuation upstream.
Anthropic streaming already followed this rule
(`_server_memory_tool_names`). That helper is now shared, and every
executor uses it.

**Behaviour change**
- With `--no-memory-tools`, memory tool calls from the model now reach
the client instead of running on the proxy.
- A memory-named tool that the client declares is now the client's. The
proxy no longer runs it on any path, including the streaming path that
used to run it as a side effect.
- Injected memory tools behave as before: they run on the proxy, are
hidden from the client, and the turn continues upstream.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/memory_handler.py`: `server_memory_tool_names(tools,
client_tools)` moved here from `AnthropicHandlerMixin`. It now reads
Anthropic, Chat and Responses tool shapes through `extract_tool_name`.
`handle_memory_tool_calls` takes a required `tool_names` and skips every
other call.
- `handlers/openai.py`, Chat Completions (non-streaming): records the
injected names and passes them to the executor.
- `handlers/openai.py`, Responses (HTTP): runs a `function_call` only if
its name was injected. Before, any name in `MEMORY_TOOL_NAMES` ran.
- `handlers/openai.py`, Responses over WebSocket: the injected names are
kept per `response.create`, like the user id. The relay holds back and
runs only those calls.
- `handlers/anthropic.py`, non-streaming: passes the injected names it
already computed for streaming.
- `handlers/streaming.py`: removes the branch that ran memory calls
after the stream had already reached the client. It ran only for tools
the proxy had not injected. Memory handling in the stream (SSE
buffering, the Claude Code credential warning) now turns on only when
the proxy injected memory tools.
- Tests: new `tests/test_memory_tools_only_when_injected.py`. Existing
fixtures that relied on the old behaviour now inject the tool they call:
`test_codex_project_context.py`, `test_memory_handler_native_ops.py`,
`test_memory_handler_project_isolation.py`,
`test_memory_tool_stream.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

The new tests use `create_app` with a fake memory backend and a
`MockTransport` upstream. The WebSocket test uses a scripted upstream
socket. For each path (chat, Responses, Responses over WebSocket,
Anthropic, Anthropic streaming) they check two cases. First, injection
is off and the model calls `memory_delete`. Second, injection is on and
the client declares its own `memory_delete`. Each case expects zero
backend calls, exactly one upstream request, and the call in the
client's response. The control cases inject `memory_list` and expect it
to run as before.

On `origin/main` (0368a872):
```text
PASSED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[client_declared-anthropic_stream]
PASSED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_injected_still_runs[chat]
PASSED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_injected_still_runs[responses]
PASSED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_injected_still_runs[anthropic]
PASSED tests/test_memory_tools_only_when_injected.py::test_ws_memory_call_the_proxy_injected_still_runs
FAILED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[not_injected-chat]
FAILED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[not_injected-responses]
FAILED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[not_injected-anthropic]
FAILED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[not_injected-anthropic_stream]
FAILED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[client_declared-chat]
FAILED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[client_declared-responses]
FAILED tests/test_memory_tools_only_when_injected.py::test_memory_call_the_proxy_did_not_inject_reaches_the_client[client_declared-anthropic]
FAILED tests/test_memory_tools_only_when_injected.py::test_ws_memory_call_the_proxy_did_not_inject_reaches_the_client[not_injected]
FAILED tests/test_memory_tools_only_when_injected.py::test_ws_memory_call_the_proxy_did_not_inject_reaches_the_client[client_declared]
9 failed, 5 passed in 3.05s
```
On this branch:
```text
14 passed in 3.23s
```
The existing tests for memory, anthropic, openai and streaming (165
files under `tests/`, not `test_live`) pass on both branches:
```text
origin/main: 3183 passed, 44 skipped in 230.96s
this branch: 3197 passed, 44 skipped in 229.83s
```
`uvx ruff@0.17.3 check` and `format --check` pass on the changed files.
`mypy` on the four changed modules reports no issues.

## Real Behavior Proof

- Environment: macOS, Python 3.12,
`create_app(ProxyConfig(memory_enabled=True,
memory_storage_mode="global", ...))` in-process via
`httpx.ASGITransport`. The upstream is `httpx.MockTransport`, the memory
backend is a recording fake, and `LITELLM_LOCAL_MODEL_COST_MAP=True`.
- Exact command / steps: a harness sends one request per path. The
mocked upstream answers with a tool call to `memory_delete`. The harness
records backend calls, upstream request count, and whether the client
response contains the call.
- Observed result (`origin/main` → this branch):
```text
chat       inject=False memory_delete  upstream_calls=2 backend_calls=[delete] client_saw_tool_call=False  →  upstream_calls=1 backend_calls=[] client_saw_tool_call=True
responses  inject=False memory_delete  upstream_calls=2 backend_calls=[delete] client_saw_tool_call=False  →  upstream_calls=1 backend_calls=[] client_saw_tool_call=True
anthropic  inject=False memory_delete  upstream_calls=2 backend_calls=[delete] client_saw_tool_call=False  →  upstream_calls=1 backend_calls=[] client_saw_tool_call=True
anthropic stream inject=False memory_delete  backend_calls=[delete] client_saw_tool_call=True  →  backend_calls=[] client_saw_tool_call=True
chat       inject=True, client-declared memory_delete  upstream_calls=2 backend_calls=[delete]  →  upstream_calls=1 backend_calls=[]
anthropic  inject=True, client-declared memory_delete  upstream_calls=2 backend_calls=[delete]  →  upstream_calls=1 backend_calls=[]
```
- Not tested: a live provider and a live Codex WebSocket session. The
WebSocket path is covered by the in-process test above.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only for memory-named tool calls that
the proxy did not inject. See "Behaviour change" above. The default
`--memory` setup, where the proxy injects the tools, is unchanged.
- Kill switch / disable path: none needed.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Follow-up, not in this PR: on the Chat, Responses and WebSocket paths, a
single turn can call an injected memory tool and a client tool together.
The proxy then continues upstream without an output for the client's
call. That was already true for a client tool such as `shell`. It now
also applies to a memory-named tool that the client declared, which the
proxy used to run. The Anthropic non-streaming path already handles this
by returning the turn to the client. Bringing the OpenAI paths in line
is a separate change.

The docs need no change: `--no-memory-tools` is already documented as
"Disable automatic memory tool injection", which is now what it does.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4870"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-14 10:47:05 +05:30
48df6df497 fix(memory): scope memory_update and memory_delete to the calling user (#4869)
## Description

The `memory_delete` tool called `backend.delete_memory(memory_id)`
without the user id. `LocalBackend.update_memory` and `delete_memory`
ignored `user_id`, and `DirectMem0Adapter` did the same. When users
share one store (`global` storage mode, or the qdrant-neo4j backend), a
memory id that belonged to another user could be deleted or overwritten.

The handler now passes the effective user id to `delete_memory`. Both
backends now treat an id owned by someone else as not found: delete
returns `False`, and update raises the same not-found `ValueError` it
raises for an unknown id. A caller that passes no `user_id` (the CLI,
`MemorySystem`) keeps the old unscoped behaviour.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/memory_handler.py`: `_execute_delete`, and the
delete+save fallback in `_execute_update`, pass
`user_id=effective_user_id`. The fallback now returns `not_found` when
nothing was deleted, instead of saving a replacement.
- `headroom/memory/backends/local.py`: `update_memory` and
`delete_memory` check the stored owner when `user_id` is given.
- `headroom/memory/backends/direct_mem0.py`: `update_memory` and
`delete_memory` check the owner through `get_memory` when `user_id` is
given. If the lookup fails, the id is treated as not found.
- `headroom/memory/backends/mem0_system_adapter.py`: `delete_memory`
with a `user_id` used to delete anyway when the owner lookup raised (it
caught every exception) or when the record had no owner. It now deletes
only when the stored owner matches. A `ValueError`/`OSError` from the
lookup is logged and the delete is refused; anything else propagates.
`Mem0Backend` (`backends/mem0.py`) is the raw client that this adapter
wraps; the proxy never calls it directly.
- Tests: a new `tests/test_memory_tool_user_scope.py` runs through
`create_app` with a real `LocalBackend` in `global` mode and a
`MockTransport` upstream. There is also one `DirectMem0Adapter` test.
Fake backends in `test_memory_handler_native_ops.py` now accept
`user_id`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

In the test, alice saves a memory. bob's request then gets an upstream
turn that calls `memory_delete`, or `memory_update`, with alice's id.
The test passes when alice's memory is unchanged. The controls check
that bob can still delete and update his own memory.

On `origin/main` (0368a872):
```text
PASSED tests/test_memory_tool_user_scope.py::test_user_can_still_change_own_memory[memory_delete-args0]
PASSED tests/test_memory_tool_user_scope.py::test_user_can_still_change_own_memory[memory_update-args1]
FAILED tests/test_memory_tool_user_scope.py::test_memory_owned_by_another_user_is_left_alone[memory_delete-args0]
FAILED tests/test_memory_tool_user_scope.py::test_memory_owned_by_another_user_is_left_alone[memory_update-args1]
2 failed, 2 passed
FAILED tests/test_memory/test_direct_mem0.py::test_update_and_delete_leave_another_users_memory_alone
1 failed, 23 deselected
```
On this branch:
```text
PASSED tests/test_memory_tool_user_scope.py::test_memory_owned_by_another_user_is_left_alone[memory_delete-args0]
PASSED tests/test_memory_tool_user_scope.py::test_memory_owned_by_another_user_is_left_alone[memory_update-args1]
PASSED tests/test_memory_tool_user_scope.py::test_user_can_still_change_own_memory[memory_delete-args0]
PASSED tests/test_memory_tool_user_scope.py::test_user_can_still_change_own_memory[memory_update-args1]
4 passed
PASSED tests/test_memory/test_direct_mem0.py::test_update_and_delete_leave_another_users_memory_alone
1 passed, 23 deselected
```
`Mem0SystemAdapter` (second commit),
`test_delete_is_refused_unless_the_owner_is_confirmed`: 4 failed, 1
passed on the first commit (only `other-owner` passed); 5 passed with
the fix. Memory tests after it: 875 passed, 2 skipped.

The existing memory test files (74 files) pass on this branch:
```text
1447 passed, 27 skipped in 64.67s
(after review follow-ups: test_memory_tool_user_scope, test_memory_handler_native_ops, test_direct_mem0: 69 passed, 1 skipped)
```
`uvx ruff@0.17.3 check` and `format --check` pass on the changed files.
`mypy` on the three changed modules reports no issues.

## Real Behavior Proof

- Environment: macOS, Python 3.12. The proxy is created in-process with
`create_app(ProxyConfig(memory_enabled=True,
memory_storage_mode="global", memory_inject_tools=True))` and backed by
a real `LocalBackend` (SQLite plus sqlite-vec, fixed test embedder). The
upstream is `httpx.MockTransport`, and
`LITELLM_LOCAL_MODEL_COST_MAP=True`.
- Exact command / steps: `pytest tests/test_memory_tool_user_scope.py`.
alice saves a memory. A `/v1/messages` request with `x-headroom-user-id:
bob` gets an upstream `memory_delete` or `memory_update` with alice's
id. The proxy runs the tool and continues the turn.
- Observed result: on main, alice's memory is deleted, or superseded
with bob's content. On this branch it is unchanged. The model gets
`not_found` for delete, and the usual not-found error for update.
- Not tested: a live qdrant/neo4j deployment. `DirectMem0Adapter` is
covered with a stubbed mem0 client.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only for update/delete calls on an id
the caller does not own. They now return not found.
- Kill switch / disable path: none needed.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

This PR is independent of the PR that limits which memory tool calls the
proxy runs. Either can merge first.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-14 10:47:00 +05:30
Abhinav MauryaandJerrettDavis 00d4f441d3 fix(memory): gracefully degrade on context lookup and initialization errors (#3251) (#4049)
## Description

When `--memory` is enabled, memory backend initialization or database
access failures (such as `sqlite3.OperationalError: unable to open
database file` on bind mounts or invalid SQLite stores) caused request
handlers to raise unhandled exceptions and abort client requests with
HTTP 500. This change ensures that memory lookup and backend
initialization failures degrade gracefully by logging a warning and
continuing request processing without memory context (`memory_context =
None`).

Closes #3251

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- In `headroom/proxy/memory_handler.py:search_and_format_context`:
wrapped `_ensure_initialized()` and `_resolve_for_request()` in a
`try...except Exception` block, logging a warning and returning `None`
on failure.
- In `headroom/proxy/handlers/anthropic.py:handle_anthropic_messages`:
added an `except Exception as e:` handler for memory context lookup to
prevent lookup errors from raising unhandled exceptions out of the
request handler.
- In `headroom/proxy/handlers/openai.py`: added `except Exception as e:`
handlers in Responses API (`handle_openai_responses`) and WebSocket
(`handle_realtime_websocket`) context search blocks.
- Added regression tests in
`tests/test_anthropic_pre_upstream_backpressure.py` and
`tests/test_memory_handler_native_ops.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
tests/test_anthropic_pre_upstream_backpressure.py .. [100%]
2 passed in 1.56s

tests/test_memory_handler_native_ops.py .................................. [100%]
34 passed in 1.57s
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.14
- Exact command / steps: `pytest -q
tests/test_anthropic_pre_upstream_backpressure.py -k test_memory_context
&& pytest -q tests/test_memory_handler_native_ops.py`
- Observed result: Verified
`test_memory_context_exception_fails_open_and_releases_semaphore` and
`test_search_and_format_context_degrades_gracefully_on_backend_error`
pass cleanly.
- Not tested: None

## Runtime Rollout Safety

- Rollout-managed feature(s): None
- Minimum rollout channel: stable
- Stable/default behavior changed: No (gracefully falls back when memory
store is unavailable rather than returning 500)
- Kill switch / disable path: Disable memory with `--memory false` or
omitting `--memory`
- Unsafe override required: None
- Qualification impact: None
- Rollback path: git revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit `CHANGELOG.md` (release-please handles this)

## Additional Notes

Ensures optional memory subsystem failure does not take down core proxy
traffic.


## Current maintainer verification (October 9 UTC)

Exact synchronized head: `b222647f31e0fbdfd8e33279f4c3c21e3cd4e40f`,
containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d` through a
clean merge. The updated WebSocket regression gates each response pair
on its matching upstream request and keeps the client open through both
completions; the handler call is bounded by a five-second timeout. No
contributor logic was changed during synchronization.

Both affected routing/WebSocket files pass: 96 tests on Windows/Python
3.12 with the compatible existing native extension. Additional reviewer
probes pass with deliberately yielding upstream sends at 0, 1, and 10
ms, addressing the reported scheduling race. These are local
mocked-socket tests; no live external-provider or client session is
claimed. GitHub fork-workflow approval was provided for this reviewed
head; Current-head hosted snapshot completed with 30 successful checks,
12 skips, and one failed test shard. The failed per-frame memory
recovery test is a PR regression: this file has 1 failure/10 passes
locally on the PR, versus all 11 passes on main. Failed lookup currently
adds memory instructions instead of preserving the original WebSocket
frame; the new maintainer review requests that correction. The 96
routing/lifecycle tests and three scheduling probes above remain passing
evidence with this explicit limit.


### Maintainer current-main requalification

Exact head `3ca4324e69bd46a1c07288fb1b17c12b6fbcc9a7`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Synchronized with upstream main
2d3ee51846. The memory-handler conflict
retains main's unresolved-project early return before the PR's
initialization/resolve failure boundary. The test conflict retains
main's streaming-tool regression and the PR's independent lookup-failure
regression as complete separate functions. No contributor failure policy
rewritten or main project-isolation guard dropped.

The author fix addresses the previous WebSocket finding: the inner
lookup catch no longer allows later augmentation after failure, so the
outer frame boundary forwards the original raw frame and later frames
can retry. The existing per-frame original-request/recovery regression
now passes. All 205 focused HTTP, WebSocket, native-memory-operation and
backpressure tests pass; current author/Devin discussion reviewed,
including the restored deterministic coordination and five-second test
deadline. Changed Python lint/format and main-relative whitespace pass.

Python uses the existing native extension for startup, not a fresh
native build. No live memory-cloud or model-provider claim. No remaining
finding in the reviewed failure/recovery scope; formal approval awaits
exact-head hosted qualification, handled by maintainers. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `742b5aeeb1e5c75cfa6a8b51d9b69d0a3fef36c8`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical synchronization with upstream main
c01cf07d94. Re-reviewed memory
initialization/lookup failure boundaries and current author/Devin
discussion. Main's unresolved-project early return remains before
backend initialization and resolution. Failed WebSocket lookup reaches
the outer fallback before augmentation, preserving the original raw
frame while later frames can retry. The previously fixed deterministic
two-turn coordination and five-second test deadline remain intact. No
contributor failure/isolation policy rewritten.

Fresh HTTP/WebSocket/native-memory-operation/backpressure verification:
205 passed, one test-client deprecation warning. All seven changed
Python files pass Ruff lint/format and main-relative whitespace checks;
owned tree clean. Current-main native DLL supports local startup; no
live cloud/model-provider or branch-specific native-build claim. No
remaining finding in this failure/recovery scope; formal approval awaits
exact-head hosted qualification. Maintainers handle hosted checks;
nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-06 09:25:11 -06:00
957ab1830d fix(memory): fail close unresolved project operations (#2637)
## Description

An unresolved project scope still carried the global database path.
Custom-tool, native-memory, and learner paths could therefore initialize
or reuse global state after context injection had already skipped.

This PR makes unresolved project scopes backend-less and checks request
scope before any memory or learning operation. Optional memory failures
still allow the main model request to continue.

This supersedes #502 at reviewed commit
`9d45c11233959815da651393c3b3737359497882` and preserves attribution to
@chopratejas in the commit trailer.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Remove the database path and backend from unresolved project scopes.
- Gate injection, custom tools, native operations, and learning before
backend acquisition.
- Pass request context through OpenAI and Anthropic memory-tool paths.
- Preserve explicit global/user modes and fail-open model traffic.

## Testing

### October 27 combined-tree refresh

Merged upstream `17d38e5699b7d7c69ac6addcd762f39e42dd7f00` into
`5f625dba9bccee6b38ec2e9de6c04023c29358e7`. No manual source repairs
were needed.

- Focused memory, routing, WebSocket, Anthropic, and gateway tests:
**308 passed, 2 skipped**.
- The two skips cover the macOS non-UTF8 symlink fixture and optional
native tool-search extension.
- The Python pre-push test list, invoked directly against the built
extension: **177 passed, 4 skipped**.
- The real-file proof below passed again on this combined tree.
- Independent static review found no integration blocker.
- Ruff, whitespace, and commit hooks passed, including mypy and the new
commit's commitlint check.
- Full `make ci-precheck` is not green: Rust 1.99.1 reports
`clippy::question_mark` in
`crates/headroom-core/src/ccr/backends/in_memory.rs:163`. The same
command fails on clean upstream `17d38e56`.
- The Python wrapper requires `pip`, absent from this uv environment.
Its test list was run directly after `uv sync` built the extension.
- The history-wide commitlint target rejects an older commit footer. The
new merge commit passed its commit hook.
- The push hook was bypassed for these documented limitations. Hosted
qualification for the new head remains required.

The focused command below was extended with
`tests/gateway/test_hook_tool_growth.py
tests/gateway/test_native_deferral.py
tests/gateway/test_tool_compaction_accounting.py`.

### October 25 validation

Current-main refresh: `ba3320e49f202c5880e4ef8e28efe810b78eeb07`, with
`3fef15d376d9899f77a28464e2e91f8b99514930` merged as the second parent.

- [x] Focused regression tests pass
- [x] Ruff lint and format checks pass for all ten PR files
- [x] PR diff whitespace check passes
- [ ] Type checking rerun for this refresh
- [x] Real-file behavior proof completed
- [x] Independent review found no remaining integration issue

The merge required no manual code repairs. No dependency or API changes
were added to the PR.

### Exact commands

```bash
uv sync --frozen --extra dev --extra proxy
uv run --frozen --extra dev --extra proxy python -m pytest tests/test_memory_storage_router.py tests/test_memory_handler_project_isolation.py tests/test_memory_handler_native_ops.py tests/test_memory_decision.py tests/test_memory_decision_policy.py tests/test_openai_codex_routing.py tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_per_frame_memory.py tests/test_openai_responses_traffic_learner.py tests/test_anthropic_buffered_memory_continuation.py tests/test_memory_handler_concurrent_init.py -q
uv run --frozen --extra dev --extra proxy ruff check $(git diff --name-only 3fef15d3...HEAD)
uv run --frozen --extra dev --extra proxy ruff format --check $(git diff --name-only 3fef15d3...HEAD)
git diff --check 3fef15d3...HEAD
```

### Test Output

```text
259 passed, 1 skipped, 1 warning in 27.35s
All checks passed!
10 files already formatted
```

The skip is the non-UTF8 symlink fixture, which this macOS filesystem
cannot create. The warning is an existing Starlette/httpx deprecation.

The native extension was built from the combined source. These results
replace the earlier PR-only test counts for this refresh.

## Real Behavior Proof

I ran this on macOS 27.0.1 with Python 3.14.5 at
`ba3320e49f202c5880e4ef8e28efe810b78eeb07`.

The script uses the actual router and memory handler, with no mocked
backend. It creates a temporary SQLite canary table and native-memory
file.

A request without project identity returns no backend or database path.
Context lookup, five custom tools, six native commands, and Anthropic
tool dispatch skip memory access.

The script checks that the backend remains uninitialized, no project
backend is cached, and file paths and byte hashes remain unchanged. It
also reads the SQLite canary afterward.

This proves rejection before backend access. The canary database is not
a populated Headroom memory schema, so this does not demonstrate
resolved-project retrieval.

No provider request, model-request forwarding, live proxy, service
restart, or native Windows/Linux execution was tested by this script.
Temporary canary files are removed automatically.

### Reproduce

Save the script below as `memory-behavior-proof.py` in the repository
root after running the environment setup above.

```bash
HEADROOM_WORKSPACE_DIR="$PWD/proof-workspace" HEADROOM_CONFIG_DIR="$PWD/proof-config" HEADROOM_SETTINGS_PATH="$PWD/proof-settings.json" .venv/bin/python -u memory-behavior-proof.py
```

<details>
<summary>Executable script</summary>

```python
"""Check unresolved memory scope using real files and production handlers."""
import asyncio
import hashlib
import json
import os
from pathlib import Path
import sqlite3
import tempfile

os.environ["HEADROOM_BEACON"] = "off"
os.environ["HEADROOM_TELEMETRY"] = "off"
os.environ["HEADROOM_OFFLINE"] = "1"

from headroom.memory.storage_router import MemoryStorageMode, RequestContext
from headroom.proxy.memory_handler import MemoryConfig, MemoryHandler, MemoryMode


def snapshot(root):
    return {
        str(path.relative_to(root)): hashlib.sha256(path.read_bytes()).hexdigest()
        for path in root.rglob("*") if path.is_file()
    }


async def exercise(root):
    database = root / "memory.db"
    with sqlite3.connect(database) as connection:
        connection.execute("CREATE TABLE canary (content TEXT)")
        connection.execute("INSERT INTO canary VALUES ('private-existing-memory')")
    native_dir = root / "native"
    native_dir.mkdir()
    (native_dir / "canary.txt").write_text("private-existing-memory\n", encoding="utf-8", newline="\n")
    handler = MemoryHandler(MemoryConfig(
        enabled=True, backend="local", db_path=str(database),
        storage_root=str(root / "projects"), storage_mode=MemoryStorageMode.PROJECT,
        inject_context=True, mode=MemoryMode.AUTO_TAIL, use_native_tool=True,
        native_memory_dir=str(native_dir),
    ))
    before = snapshot(root)
    ctx = RequestContext(headers={}, system_prompt="You are helpful.", base_user_id="alice")
    backend, scope = handler._router.backend_for(ctx)
    assert backend is None and scope.db_path is None and scope.project_key is None
    print(f"scope: project_key={scope.project_key}, db_path={scope.db_path}, backend={backend}")
    context = await handler.search_and_format_context(
        "alice", [{"role": "user", "content": "Find my private memory"}], ctx,
    )
    assert context is None
    print(f"context lookup: {context}")
    expected = {"status": "skipped", "reason": "project_unresolved"}
    tools = {
        "memory_save": {"content": "must not persist"},
        "memory_search": {"query": "private-existing-memory"},
        "memory_update": {"memory_id": "m1", "new_content": "must not update"},
        "memory_delete": {"memory_id": "m1"},
        "memory_list": {},
    }
    for name, arguments in tools.items():
        result = json.loads(await handler._execute_memory_tool(name, arguments, "alice", request_context=ctx))
        assert result == expected
        print(f"{name}: {json.dumps(result, sort_keys=True)}")
    for command in ("view", "create", "str_replace", "insert", "delete", "rename"):
        arguments = {"command": command, "path": "/memories/canary.txt",
                     "file_text": "must not persist", "old_str": "private", "new_str": "public",
                     "insert_line": 0, "insert_text": "must not persist",
                     "old_path": "/memories/canary.txt", "new_path": "/memories/renamed.txt"}
        result = json.loads(await handler._execute_native_memory_tool(arguments, "alice", request_context=ctx))
        assert result == expected
        print(f"native {command}: {json.dumps(result, sort_keys=True)}")
    results = await handler.handle_memory_tool_calls({"content": [
        {"type": "tool_use", "id": "native-create", "name": "memory",
         "input": {"command": "create", "path": "/memories/canary.txt", "file_text": "must not persist"}},
        {"type": "tool_use", "id": "custom-save", "name": "memory_save",
         "input": {"content": "must not persist"}},
    ]}, "alice", "anthropic", request_context=ctx)
    assert len(results) == 2 and all(json.loads(item["content"]) == expected for item in results)
    print("Anthropic tool dispatch: both calls skipped")
    assert not handler.initialized and handler._backend is None and not handler._router._backends
    assert snapshot(root) == before
    with sqlite3.connect(database) as connection:
        assert connection.execute("SELECT content FROM canary").fetchall() == [("private-existing-memory",)]
    print("backend initialization: false; cached project backends: 0")
    print("SQLite and native files: byte hashes unchanged; no files added or removed")
    print("Provider traffic and model-request forwarding: not exercised by this proof")


with tempfile.TemporaryDirectory(prefix="headroom-pr2637-proof-") as directory:
    asyncio.run(exercise(Path(directory)))
print("Cleanup: temporary canary files removed")
```

</details>

<details>
<summary>Captured output</summary>

```text
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
scope: project_key=None, db_path=None, backend=None
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
context lookup: None
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
memory_save: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
memory_search: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
memory_update: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
memory_delete: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
memory_list: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
native view: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
native create: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
native str_replace: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
native insert: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
native delete: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
native rename: {"reason": "project_unresolved", "status": "skipped"}
event=memory_project_unresolved behavior=empty user_id=alice hint='set x-headroom-project-id, x-headroom-cwd, or provide a cwd in the system prompt, or set memory.unresolved_project_fallback=global to opt-in to legacy cross-project GLOBAL pooling (cross-project leak risk).'
Anthropic tool dispatch: both calls skipped
backend initialization: false; cached project backends: 0
SQLite and native files: byte hashes unchanged; no files added or removed
Provider traffic and model-request forwarding: not exercised by this proof
Cleanup: temporary canary files removed
```

</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): Project-scoped memory injection, tools,
native operations, and traffic learning.
- Minimum rollout channel: Stable/default.
- Stable/default behavior changed: Unresolved project requests skip
memory and learning instead of acquiring the global backend.
- Kill switch / disable path: Run without `--memory`; use `--no-learn`
to disable traffic learning.
- Unsafe override required: No.
- Qualification impact: Project memory isolation, explicit global/user
modes, and fail-open model traffic.
- Rollback path: Revert this PR. No schema or data migration is
required.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] Documentation changes are not required for this internal safety
fix
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Waiting follow-up: [project-scoped
TrafficLearner](https://github.com/krzkraw/headroom/tree/krz/project-scoped-traffic-learner).
It depends on the request-scoped APIs completed here.

No schema or data migration is required.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2637"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: chopratejas <chopratejas@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-28 17:42:30 -05:00
Tejas Chopra fc5ac70b4a refactor(proxy): drop the test-only memory ranker and file-backed native memory ops (#4072)
## Description

`MemoryHandler` has two pieces of code that only tests reach. This PR
removes both, together with their tests.

1. **The memory ranker.** This is `headroom/proxy/memory_ranker.py` plus
the `ranker=` keyword on `search_and_format_context`.
- No production caller passes `ranker=`. All five handler call sites
pass `query=` only: `handlers/gemini.py:715`,
`handlers/anthropic.py:2868`, and `handlers/openai.py:4569`, `6360` and
`8332`.
- So the re-rank branch never ran. The cosine plus `min_similarity`
branch is now the only path, and its logic and output format are
unchanged.
2. **The file-backed native memory ops.** These are
`_resolve_native_path`, `_native_view`, `_native_create`,
`_native_str_replace`, `_native_insert`, `_native_delete_file` and
`_native_rename`.
- `_execute_native_memory_tool` dispatches only to the `*_semantic`
variants, so these methods had no caller outside tests.

`_init_native_memory_dir` is kept. It still creates the private native
memory directory, and `tests/test_private_state_files.py` asserts its
mode.

Evidence: I searched for `memory_ranker`, `RecencyBoostRanker`,
`MemoryCandidate`, `ranker=` and each removed method. The search covered
`headroom`, `plugins`, `docs`, `tests` and the private consumer repos.
It found only the definitions and their own tests.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Delete `headroom/proxy/memory_ranker.py` and
`tests/test_memory_ranker.py`.
- `memory_handler.py`:
- Remove the `ranker` kwarg, its docstring and the re-rank branch from
`search_and_format_context`.
  - Dedent the remaining cosine branch.
  - Delete the seven file-backed native-op methods (about 260 lines).
- `tests/test_memory_handler_native_ops.py`: remove the tests of the
deleted file-backed ops. The tests for the live dispatcher and the
semantic ops stay.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_memory_handler_native_ops.py tests/test_memory_injection_budget.py tests/test_memory_injection_logging.py tests/test_private_state_files.py tests/test_public_errors.py tests/test_memory_decision_policy.py tests/test_memory_rank_policy.py tests/test_memory_handler_null_function.py tests/test_proxy_memory_integration.py
================= 122 passed, 13 skipped, 4 warnings in 14.46s =================
$ pytest -q -p no:cacheprovider <every other test file that references search_and_format_context or _execute_native_memory_tool: 14 files>
============================= 416 passed in 27.97s =============================
$ uvx ruff@0.16.8 check headroom/proxy/memory_handler.py tests/test_memory_handler_native_ops.py
All checks passed!
$ mypy headroom/proxy/memory_handler.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, this branch at 89fa67c3, using the
repo .venv. Every run used a temp `HOME`, `HEADROOM_WORKSPACE_DIR` and
`CODEX_HOME`, plus `HEADROOM_OFFLINE=1`.
- Exact command / steps:
- (1) The shared smoke harness with memory on: `smoke.sh <worktree>
--memory`. It starts a real `headroom proxy --memory` against a local
fake upstream and sends one Anthropic and one OpenAI request.
- (2) A direct script against a stub backend that returns one result
above `min_similarity` and one below. It calls
`MemoryHandler.search_and_format_context` on this branch and on
`origin/main`, then diffs the two outputs.
- Observed result:

```text
(1) branch: cleanup/proxy-core-b-memory-test-only @ 89fa67c3
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING lines: only the expected air-gap ones (offline mode, Kompress/embedder not cached, memory project unresolved).

(2) ## Relevant Memories for This User
...
1. [m1] prefers pytest -q
   (Related: pytest)
...
diff (branch vs origin/main): only `has _native_str_replace: False` vs `True`; the injected memory block is byte-identical.
```

- Not tested: Live memory injection against a real provider with a real
embedder. The run is offline, so no embedder model is available.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; dead-code removal.
- Stable/default behavior changed: No. The removed branch required a
`ranker` argument that no caller passed, and the removed methods had no
caller.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

`memory_rank_policy.py` stays: `headroom/memory/` still imports it.

After this change nothing writes into the native memory directory.
Whether to stop creating it is a separate decision, so this PR leaves it
alone.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-10-10 08:56:00 -07:00
Tejas ChopraandClaude Fable 5.1 861e94d8c1 fix(proxy): keep exception text and the proxy token out of client output (#3851)
## Why

Two leaks of internal detail to clients and logs (findings 01-F4 and
10-F4).

- When an upstream connection failed, several response paths returned
`str(e)` from `httpx`, `ssl` or the OS in 500/502/503 bodies and SSE
`error` events. That text names the resolved upstream host, its address
and the failing library. A client that can steer the upstream
(`x-headroom-base-url`) could learn internal hostnames one error at a
time.
- `extract_tags` copied every `x-headroom-*` header into
`RequestOutcome.tags`, including `x-headroom-proxy-token`, which
gateways send on every turn. The token then appeared in the request log
(`--log-file`), dashboard facets and any export that carries tags.

## What changes

- New module `headroom/proxy/public_errors.py`. `classify_exception()`
maps transport and OS failures to fixed codes: `upstream_unreachable`,
`upstream_timeout`, `upstream_tls_error`, `upstream_protocol_error`;
anything else is `internal_error`. Every client- or model-visible path
fails closed: unrecognised exception text (`RuntimeError`, `ValueError`,
parser, database or library errors) never leaves the proxy. Provider
error text is forwarded only when the exception carries a provider HTTP
error response (`openai.APIStatusError`, which LiteLLM's provider errors
subclass, or `anthropic.APIStatusError`). Helpers build the existing
wire shapes with the same `error.type`, plus `code` and `request_id`.
The full exception stays in the server log under that request id.
- Every site that built a client-visible message from an exception now
uses it: `handlers/streaming.py` (direct, Bedrock and OpenAI-via-backend
SSE), `handlers/openai.py` (backend 500, `/v1/compress` 503, both
passthrough connect paths), `handlers/anthropic.py` (Bedrock backend
500), `handlers/bedrock.py` (connect 502), the memory tool handlers
(tool results returned to the model), and the `litellm` / `anyllm`
backends. The existing corporate-TLS hint is kept.
- `internal_header_policy.py` gains `is_credential_header()` and
`is_credential_tag_key()`. They match whole name components (`token`,
`secret`, `password`, `authorization`, `cookie`, `apikey`, `key`, and
similar), so `X-Headroom-Key` and `X-Headroom-Api-Key` are covered, so
numeric tags such as `tool-search-deferred-tokens` are not caught.
- `extract_tags` skips credential headers.
`savings_attribution.public_tags`, the last step before the request log,
also drops credential-shaped keys.

## Behaviour change

- Transport-layer and internal failures now return a fixed sentence plus
`code` and `request_id`. Clients that parsed the old free-text messages
should read `error.code`. Provider HTTP error responses (and their
status codes) are unchanged. In the LiteLLM / any-llm backends, an
exception that is not a provider response now returns `The proxy could
not complete the request.`; the HTTP status mapping is unchanged.
- Memory tool results for any failure return `{"status": "error",
"error": "<code>", "message": ...}` instead of the exception text.
- `x-headroom-*` headers whose name contains a credential component are
no longer recorded as tags. A benign label that happens to use such a
name should be renamed.

## Tests

- `tests/test_public_errors.py`: classification of transport exceptions
(including TLS wrapped in a connect error), non-transport exceptions
left alone, wire shapes carry `code` and `request_id`; the passthrough,
direct-streaming and `/v1/compress` error paths contain no upstream
detail; memory tool results hide transport and internal text;
`client_message` / `tool_result_error` / the LiteLLM backend return
`internal_error` for `RuntimeError`, `ValueError`, `KeyError` and
`JSONDecodeError` carrying a path, host and token (10 tests in this file
fail without the fail-closed change), while an
`openai.AuthenticationError` keeps its text; `extract_tags` and
`public_tags` never carry credentials. Six of these fail on `main`.
- Existing tests that asserted the old exception text in client output
now assert the new contract: the client gets the fixed `code` and
`request_id`, the exception text is absent, and the `headroom.proxy` log
has the full text under the same `request_id`
(`test_litellm_transport_errors.py`, `test_openai_backend_path.py`,
`test_proxy_compress_endpoint.py`,
`test_proxy_streaming_ratelimit_headers.py`).
- Manual run with every upstream connect failing: streaming
`/v1/messages`, streaming `/v1/chat/completions` and passthrough
`/v1/models` return `code: upstream_unreachable` with no address. A
request with `x-headroom-proxy-token` and `x-headroom-license-token`
leaves neither value in the `--log-file` output.

Not tested live: the Bedrock and LiteLLM / any-llm backend paths (unit
tests only) and OTEL/Langfuse export (tags are filtered before they
reach it).

## Open review points

- Resolved: unclassified exception text (JerrettDavis, 2026-09-29) now
fails closed in `client_message` and `tool_result_error`.
- Resolved: `X-Headroom-Key` is matched (`key` is a credential
component; the `api`+`key` pair rule is gone). No internal or test tag
key has a `key` component.
- `is_credential_header` special-cases `authorization`, `cookie`,
`api-key` and `x-api-key`, but its only caller passes only
`x-headroom-*` names, so those branches never run.
- `_is_tls_failure` also matches on message text. Provider HTTP error
responses are now checked first in the backends, so this only affects
non-provider exceptions.
- any-llm: only provider errors raised as `openai`/`anthropic`
`APIStatusError` keep their text; any other any-llm exception type
returns `internal_error`.

Revised 2026-10-01 after review: unrecognised exceptions now return a
fixed `internal_error` message and only provider HTTP error responses
keep their text.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-01 19:28:34 -05:00
Tejas Chopra 1f96dabc19 fix(security): address u9up assessment findings (WEB-01–07) (#2207)
Hardens client-selected upstreams, memory identity resolution, downloaded binary integrity, telemetry import, Docker defaults, Neo4j credentials, and archive extraction. Refreshes the branch against current main and preserves newer same-origin and loopback protections.
2026-08-20 09:02:44 -05:00
Gautam Sharma 5279c33b19 fix(memory): preserve semantically similar memories (#2303)
## Description

Prevent memory_save from automatically deleting semantically similar but
distinct memories. The previous fire-and-forget deduplication path
deleted existing memories at cosine similarity scores of 0.92 or higher
after the save had already returned success. Similarity remains
available as a consolidation hint, while supersession now requires an
explicit memory_update or memory_delete operation.

  ## Type of Change

  - [x] Bug fix (non-breaking change that fixes an issue)
  - [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
  - [ ] Documentation update
  - [ ] Performance improvement
  - [ ] Code refactoring (no functional changes)

  ## Changes Made

  - Removed the automatic background deletion scheduled by memory_save.
- Removed the automatic-dedup threshold and background coroutine that
were no longer needed.
  - Preserved the existing similarity search and consolidation hint.
  - Kept explicit memory_update and memory_delete behavior unchanged.
- Added a regression test proving that distinct memories survive even at
0.99 simulated similarity.
  - Added an Unreleased changelog entry.

  ## Testing

  - [x] Unit tests pass (pytest)
  - [x] Linting passes (ruff check .)
  - [x] Type checking passes (mypy headroom)
  - [x] New tests added for new functionality
  - [x] Manual testing performed

  ### Test Output

$ uv run --extra dev --frozen pytest
tests/test_memory_handler_native_ops.py
  33 passed

  $ uv run --extra dev --frozen ruff check .
  All checks passed!

$ uv run --extra dev --frozen ruff format --check
headroom/proxy/memory_handler.py tests/test_memory_handler_native_ops.py
  2 files already formatted

  $ uv run --extra dev --frozen mypy headroom --ignore-missing-imports
  Success: no issues found in 504 source files

  $ uv run --extra dev --frozen pytest
  9361 passed, 565 skipped, 4 failed

The four full-suite failures are unrelated to this diff: the Anthropic
compaction test passed in isolation; the Codex recovery test exceeded
the macOS AF_UNIX path limit; the dashboard test expects text absent
from the existing implementation; and the content-router test expects a
  fallback absent from the existing strategy chain.

The repository-wide format check also flags pre-existing formatting in
the untouched headroom/proxy/handlers/anthropic.py.

  ## Real Behavior Proof

- Environment: macOS on Apple Silicon, CPython 3.12.13, real
LocalBackend, temporary SQLite database, and the local
sentence-transformers
    embedding backend; no external provider or model API.

- Exact command / steps: Ran uv run --extra dev --frozen python with a
temporary database, saved User's primary backend framework at work is
FastAPI., queried its similarity to User's primary backend framework at
home is FastAPI., saved the second fact through
    MemoryHandler._execute_save, and listed the user's memories.

- Observed result: The real embedding similarity was 0.9387, above the
former 0.92 deletion threshold. The second save returned saved,
included the consolidation hint, retained the original memory, and left
both distinct facts in the database (memory_count: 2).

- Not tested: Live OpenAI or Anthropic provider calls, a deployed proxy
or MCP client session, and Qdrant or Neo4j memory backends. These
paths share the handler policy changed here; backend-specific explicit
update and delete behavior is unchanged.

  ## Review Readiness

  - [x] I have performed a self-review
  - [x] This PR is ready for human review

  ## Checklist

  - [x] My code follows the project's style guidelines
  - [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
  - [ ] New and existing unit tests pass locally with my changes
  - [x] I have updated the CHANGELOG.md if applicable

  ## Additional Notes

The documentation and code-comment checklist items are not applicable
because this change removes unsafe behavior without introducing a new
public interface or complex implementation. The full-suite checkbox
remains unchecked because four unrelated tests failed locally, as
  documented above.
2026-07-16 11:32:38 -07:00
ChesterandJerrettDavis 2de07db281 fix(memory): audit passive context injection (#2212)
## Description

Close the passive-memory observability loop by recording access for
context rows that survive the final injection budget and tagging
requests where context is actually appended.

Closes #2211

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Track only memory IDs retained after ranking, similarity filtering,
entry limits, and final text truncation.
- Call optional backend `record_access` with stable de-duplication and
fail-open error handling.
- Extend structured injection logging to stamp `memory_injected=true`
when injected bytes are positive.
- Thread request tags through successful Anthropic, OpenAI Chat, OpenAI
Responses, Gemini, and Codex WebSocket injection sites.
- Add a static contract test that all current successful handler
injection logs pass tags.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --with pytest --with pytest-asyncio --with numpy --with fastapi pytest \
    tests/test_memory_handler_native_ops.py \
    tests/test_memory_auto_tail.py \
    tests/test_memory_handler_project_isolation.py \
    tests/test_memory_injection_logging.py -q
51 passed

$ uv run --with ruff ruff check <touched Python files>
All checks passed!

$ git diff --check
(no output)
```

## Real Behavior Proof

- Environment: Python 3.13 with synthetic backend and handler fixtures
- Exact command / steps: run the focused test set above
- Observed result: only IDs present after the final text budget are
access-recorded; access-write failures remain fail-open; positive
injection logs stamp `memory_injected=true`; all six current successful
injection call sites pass tags
- Not tested: live provider requests, full repository suite, third-party
backends without `record_access`

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

Access accounting is intentionally best-effort: unsupported backends and
write failures do not delay or fail the upstream model request.
Documentation and changelog changes are not needed for this internal
observability fix.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 18:17:31 +00:00
chopratejas c62d45eea8 fix(memory): expose memory IDs in auto-tail + memory_list tool + ID-usage guidance
Pre-this-PR the auto-injected memory block rendered rows as `1. <content>`
with no addressable handle. To UPDATE or DELETE a row the model first had
to call memory_search to discover its ID — two round trips, against the
model-as-judge architecture.

This PR adds three tightly-coupled affordances so the model can act on
memory directly:

1. Auto-tail rows now carry the memory ID:
     `1. [mem_alpha_001] User prefers Python`
   The bracketed token is the canonical ID — same identifier accepted by
   memory_update and memory_delete.

2. New `memory_list` tool — chronological browse (vs `memory_search`'s
   semantic lookup). Returns recent memories with their IDs. Backend
   dispatches to `Backend.list_memories` if available, else falls back
   to an empty-query `search_memories`. Caps at 100 entries.

3. ID-usage guidance text appended to the auto-tail block. Tells the
   model that bracketed IDs can go straight to memory_update /
   memory_delete with no intervening search. The guidance lives in the
   user-message tail (never system) — preserves cache-prefix byte
   stability (invariant I2).

`memory_update` and `memory_delete` tool descriptions also point at the
[id] block as a valid ID source — keeps tool docs consistent with the
new affordance.

Verification:
- 10/10 tests pass in tests/test_memory_auto_tail.py (incl. 2 new
  guidance tests + 2 new ID-format tests)
- 31/31 tests pass in tests/test_memory_handler_native_ops.py (incl. 4
  new memory_list dispatch tests + existing assertions updated for the
  [id] format change)
- Golden fixtures regenerated for the tool-description copy changes
  (tests/fixtures/memory_tool_definitions/{anthropic,openai}.json)
- Live end-to-end test against real Anthropic API
  (tests/test_proxy_memory_integration.py::TestMemoryIdAutoTailAndUpdate):
  seeded memory → auto-tail → Claude → memory_update with exact ID.
  PASSED.
2026-05-19 22:10:42 -05:00
chopratejas 67620522f9 test(memory): update _extract_user_query test to lock no-truncation
PR-this removed the 500-char query truncation in
``memory_handler._extract_user_query``. The pre-existing assertion at
``test_memory_handler_native_ops.py:845`` was testing the old buggy
behaviour (output truncated at 500) and CI caught it. Updated to
assert the full-fidelity return.
2026-05-19 11:26:21 -05:00
chopratejas 7694f050fe fix: per-project memory storage so projects can no longer bleed memories (GH #462)
Memory retrieval was partitioned only by `x-headroom-user-id`. Claude
Code never sets that header, so every project a user worked on landed
in one global `default` bucket; the proxy then injected semantically
similar memories from that mixed bucket into every `/v1/messages`
request, regardless of which repo the session was actually about. The
injected `## Relevant Memories` block reads like a prompt-injection
payload and Claude has been seen to refuse to act on it, defeating the
feature.

This change makes leakage structurally impossible by giving each
resolved workspace its own SQLite database file. The wrong DB is
simply not open during a request.

- `headroom/memory/storage_router.py` (new) — `MemoryStorageMode`
  (project/user/global), `ProjectResolver` (x-headroom-project-id →
  x-headroom-cwd → --memory-project-root CLI override → env-block
  parse: `Primary working directory:` / `Working directory:` / `cwd:`,
  no regex), and `BackendRouter` with an LRU of open `LocalBackend`s
  keyed by db_path.
- `proxy/memory_handler.py` — `MemoryConfig.storage_mode` defaults to
  `PROJECT`. Provider handlers build a `RequestContext` once and pass
  it through; `search_and_format_context`, `handle_memory_tool_calls`,
  and the `_execute_*` methods route save/search/update/delete on the
  per-project backend. Qdrant-neo4j gets a composite
  `user::project_key` partition so external Mem0-style deployments
  also isolate per project without a parallel collection.
- Fix C — injected block carries provenance:
  `## Relevant Memories (workspace: <basename>, scope: project)`.
  CCR proactive-expansion block gets a matching workspace tag.
- `memory/factory.py` — process-wide embedder cache so opening N
  project DBs doesn't load the embedder N times. OpenAI key
  validation runs ahead of the cache.
- CLI — `--memory-storage={project,user,global}` (default `project`),
  `--memory-project-root` override, rewritten `--memory` help text,
  banner reports storage mode.
- Migration UX — if the legacy single-file DB has content while
  project mode is active, an INFO log points users at
  `--memory-storage=global`. Bridge currently only syncs the legacy
  DB; a WARN fires when bridge + project mode are combined.

Backward-compatible: legacy `~/.headroom/memory.db` untouched and
reachable via `--memory-storage=global`. `request_context` is
keyword-only on entry points so existing tests/mocks keep working.

Tests: 24 new (resolver tiers, LRU eviction, two-cwd isolation,
user-mode partition, legacy fallback, provenance headers); full
suite 5260 passing, ci-precheck green.
2026-05-13 15:27:41 -07:00
chopratejas 3957288229 test(memory): add qdrant_url/qdrant_api_key to expected config dict
The `qdrant-env-vars` change in d3c37d7 (PR #266) added `qdrant_url` and
`qdrant_api_key` keys to the kwargs that `MemoryHandler` passes into
`DirectMem0Adapter.__init__`. The corresponding assertion in
`test_ensure_initialized_fast_paths_and_qdrant_variants` was missed in that
PR and has been failing on `main` ever since. Surfacing here because it
fails on every PR's CI; not caused by the Rust tokenizer work this branch
adds.

The two new keys are both `None` when the corresponding `HEADROOM_QDRANT_*`
env vars are unset, which is the case in this test.
2026-04-25 14:56:55 -07:00
JerrettDavisandCopilot a5a4486a30 test: apply linux ruff formatting
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 08:49:50 -05:00
JerrettDavisandCopilot 6e9ea54a04 test: fix PR formatting and cover log compressor
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 07:49:27 -05:00
JerrettDavisandCopilot 38bf3e639c test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 07:39:52 -05:00