Commit Graph
8 Commits
Author SHA1 Message Date
Mansurshakh JaparovandJD Davis 0f3b0d8a21 fix(memory): resolve pi cwd before injecting project tools (#4618)
## Description

Closes #4442.

pi's system `<cwd>` block now resolves the memory project. If project
scope is still unresolved with the `empty` fallback, the Anthropic
handler no longer advertises unusable memory tools, including tools
recorded earlier in the session.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Parse pi cwd only in system text. Keep main's line parser, header/CLI
precedence and limited user-message fallback.
- Reuse `is_project_unresolved` for context and sticky-tool replay.
Client tools and global/user storage behavior stay intact.
- Add initialized-handler scope regressions, replace
source-inspection/forwarding-only checks, and document project scope in
the published memory page.

## Testing

- [x] Unit tests pass (`pytest`) — related suite and the normal critical
gate, not the entire Python suite locally.
- [x] Linting passes (`ruff check .`) — hosted lint passed on 5ace63e2;
local filesystem limitation below.
- [x] Type checking passes (`mypy headroom`) — normal full hook with its
existing `--ignore-missing-imports` setting.
- [x] New tests added for the scope regression
- [x] Manual testing performed

### Test Output

```text
# 16 related test files
329 passed, 12 skipped, 1 warning in 43.07s

# ordinary pre-push: make ci-precheck
178 passed, 5 skipped, 1 warning in 10.87s
✅ ci-precheck PASSED — safe to push.

uv run ruff format --check .
1860 files already formatted
```

Normal commit hooks (Ruff, formatter, full mypy, commitlint) and the
complete pre-push path passed: Rust fmt/clippy/workspace tests, native
build/import verification, critical Python tests and commitlint. The
existing Starlette test-client warning was not filtered.

Local whole-tree Ruff reports EXE002 on five **unchanged** OAuth plugin
files: my `fuseblk` checkout exposes them as executable despite Git
modes `100644`; `chmod` returns zero without changing the mode. The
uncached check agrees. I did not change those files, add shebangs or
ignore the rule. Hosted Ruff, formatting and full mypy passed on
5ace63e2; the local filesystem limitation remains.

The first full CI caught a missing scope predicate on the existing
lifecycle test double. Fixed in
[5ace63e2](https://github.com/headroomlabs-ai/headroom/commit/5ace63e2cbd2295f9594d1e0442a679aad99979c);
its complete file passes (`5 passed, 1 warning in 13.01s`). This
follow-up changes no production code or assertions and adds no runtime
fallback. The behavior proof below ran on 7aeb83cf with the same
production source; that qualification is historical; current-head checks
are recorded below.

### Author follow-up: 80453a03

Moved the project-scope guidance out of the unpublished wiki and into
`docs/content/docs/memory.mdx`; this commit changes no production code
or tests. Viewed the actual Next.js `/docs/memory#proxy-project-scope`
page and its table-of-contents entry; no browser errors.

The existing router/scope tests pass: 65 passed, with four SQLite
ResourceWarnings. All seven pi-parser lines and all six branch edges are
covered, so I did not add duplicate tests for the old Codecov
percentage.

Normal commit and pre-push hooks passed, including Rust checks, native
build/import, 180 critical Python tests (5 skipped, 1
tokenizer-approximation warning), and commitlint. My first push timed
out with the wrong local ONNX path; the next found missing pip in the
fresh uv environment. Both setup issues were fixed without bypassing
checks. Current-head hosted CI is not yet qualified.

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, freshly built native core,
real TCP/Uvicorn proxy and SQLite/ONNX memory backend. Only the upstream
is a local Anthropic-shaped HTTP server; no provider charges.
- Exact command / steps: after `uv sync --locked --extra dev --extra
proxy --extra memory` and `make verify-rust-core`, run this from the
checkout. It prints the actual loaded native module/hash and leaves a
disposable `.headroom-4442-*` directory for inspecting SQLite. Both HTTP
listeners shut down before exit.

<details>
<summary>Standalone HTTP/SQLite reproduction</summary>

```bash
./.venv/bin/python - <<'PY'
import hashlib,http.client,json,os,socket,tempfile,threading,time
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from pathlib import Path
import uvicorn
import headroom._core as core
assert Path(core.__file__).resolve().parent == Path.cwd().resolve()/"headroom"
native_sha=hashlib.sha256(Path(core.__file__).read_bytes()).hexdigest()
from headroom.memory.storage_router import ProjectResolver,RequestContext,extract_system_prompt
from headroom.proxy.server import ProxyConfig,create_app
from headroom.proxy.memory_handler import MEMORY_TOOL_NAMES
from headroom.proxy.helpers import get_session_tool_tracker
root=Path(tempfile.mkdtemp(prefix=".headroom-4442-",dir=Path.cwd()))
project_a=root/"pi%2Fproject-a";project_b=root/"project-b"
project_a.mkdir(exist_ok=True);project_b.mkdir(exist_ok=True)
os.environ["HEADROOM_WORKSPACE_DIR"]=str(root/"workspace")
os.environ["HEADROOM_BEACON"]="off"
os.environ["HEADROOM_TELEMETRY"]="off"
os.environ["NO_PROXY"]="127.0.0.1,localhost"
forwarded=[]
class Gateway(BaseHTTPRequestHandler):
    def do_POST(self):
        body=json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        forwarded.append({"path":self.path,"body":body})
        payload=json.dumps({"id":"msg_local_scope","type":"message","role":"assistant","model":"claude-haiku-4-5","content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn","stop_sequence":None,"usage":{"input_tokens":10,"output_tokens":1}}).encode()
        self.send_response(200);self.send_header("Content-Type","application/json");self.send_header("Content-Length",str(len(payload)));self.end_headers();self.wfile.write(payload)
    def log_message(self,*args):pass
upstream=ThreadingHTTPServer(("127.0.0.1",0),Gateway)
ut=threading.Thread(target=upstream.serve_forever,daemon=True);ut.start()
config=ProxyConfig(mode="cache",anthropic_api_url=f"http://127.0.0.1:{upstream.server_port}",memory_enabled=True,memory_db_path=str(root/"legacy.db"),traffic_learning_enabled=True,subscription_tracking_enabled=False,cost_tracking_enabled=False,cache_enabled=False,rate_limit_enabled=False,log_requests=False,image_optimize=False,ccr_inject_tool=False,ccr_handle_responses=False,ccr_context_tracking=False,anthropic_pre_upstream_memory_context_timeout_seconds=30.0)
app=create_app(config)
sock=socket.socket();sock.bind(("127.0.0.1",0));sock.listen(128)
port=sock.getsockname()[1]
server=uvicorn.Server(uvicorn.Config(app,log_level="warning",lifespan="on",timeout_graceful_shutdown=3))
st=threading.Thread(target=server.run,kwargs={"sockets":[sock]},daemon=True);st.start()
try:
    deadline=time.monotonic()+180
    while not server.started:
        if not st.is_alive() or time.monotonic()>deadline:raise RuntimeError("real proxy startup failed")
        time.sleep(.02)
    session="headroom4442-native-project-scope"
    common={"Content-Type":"application/json","x-api-key":"local-unbilled-key","anthropic-version":"2023-06-01","x-headroom-user-id":"native-scope-user","x-headroom-session-id":session}
    client_tools=[{"name":"get_weather","description":"Weather","input_schema":{"type":"object","properties":{}}}]
    rows=[]
    cases=[("fresh_unresolved","You are helpful.",{},False),("pi_cwd",f"<cwd>\n{project_a}\n</cwd>",{},True),("warm_unresolved","You are helpful.",{},False),("pi_cwd_restored",f"<cwd>\n{project_a}\n</cwd>",{},True),("explicit_header_wins",f"<cwd>\n{project_b}\n</cwd>",{"x-headroom-project-id":"explicit-project"},True),("ordinary_user_cwd_ignored","You are helpful.",{},False)]
    for label,system,extra,expected in cases:
        body={"model":"claude-haiku-4-5","max_tokens":16,"system":system,"messages":[{"role":"user","content":f"cwd:\n<cwd>\n{project_b}\n</cwd>" if label=="ordinary_user_cwd_ignored" else "hello scope"}],"tools":client_tools}
        before=len(forwarded)
        conn=http.client.HTTPConnection("127.0.0.1",port,timeout=90)
        conn.request("POST","/v1/messages",body=json.dumps(body).encode(),headers={**common,**extra})
        response=conn.getresponse();payload=response.read();conn.close()
        assert response.status==200,(label,response.status,payload.decode())
        assert len(forwarded)==before+1,(label,"unexpected upstream request count")
        actual=forwarded[-1]["body"]
        names={tool.get("name") for tool in actual.get("tools",[])}
        assert names==({"get_weather"}|MEMORY_TOOL_NAMES if expected else {"get_weather"}),(label,names)
        scope=ProjectResolver().resolve(RequestContext(headers={**common,**extra},system_prompt=extract_system_prompt(body),base_user_id="native-scope-user"))
        rows.append({"case":label,"status":response.status,"memoryTools":sorted(names & MEMORY_TOOL_NAMES),"project":scope,"gatewayPath":forwarded[-1]["path"]})
    assert forwarded[1]["body"]["tools"]==forwarded[3]["body"]["tools"],"restored definitions changed"
    tracker=get_session_tool_tracker()
    assert tracker.should_inject("anthropic",session),"probe did not exercise an actual recorded sticky session"
    from urllib.parse import quote
    key_a=ProjectResolver().resolve(RequestContext(headers={"x-headroom-cwd":quote(str(project_a),safe="/")},system_prompt="",base_user_id="native-scope-user"))[0]
    assert rows[1]["project"][0]==key_a
    assert rows[4]["project"][1]=="explicit-project"
    router=app.state.proxy.memory_handler._router
    assert router is not None,"real memory backend never initialized"
    opened=router.open_backends()
    project_db=root/"memories"/"projects"/key_a/"memory.db"
    assert project_db in opened and project_db.is_file(),("real pi project database not opened",[str(x) for x in opened])
    print(json.dumps({"mode":config.mode,"realHttpListener":f"http://127.0.0.1:{port}","localGateway":f"http://127.0.0.1:{upstream.server_port}","rows":rows,"realStickyGoldenSession":True,"realPiProjectDatabase":str(project_db),"openedDatabases":[str(x) for x in opened],"nativeModule":core.__file__,"nativeSha256":native_sha,"noLivePiOrPaidProvider":True},indent=2))
finally:
    server.should_exit=True;st.join(20)
    upstream.shutdown();upstream.server_close();ut.join(5);sock.close()
    assert not st.is_alive() and not ut.is_alive(),"native proof leaked a server"


PY
```

</details>

- Observed result: unchanged `main` fc1ba891 fails at the first
unresolved request: five proxy memory tools reach upstream. With the
fix, all six requests return HTTP 200:

| Same-session request | Proxy memory tools |
| --- | ---: |
| Fresh, no cwd | 0 |
| pi system cwd | 5 |
| Cwd absent again | 0 |
| Same pi cwd restored | 5 |
| Explicit project header, different system cwd | 5 |
| Cwd block only in ordinary user text | 0 |

The client weather tool stays present. Restored definitions are
structurally equal to the first resolved turn; the explicit header wins.
The literal-percent cwd agrees with its encoded-header form, and the
actual project SQLite database opens at the expected key.

- Not tested: live pi/Caveman, paid-provider cache accounting/billing,
Windows, or other provider protocols. This does not claim to fix Caveman
header forwarding.

## Runtime Rollout Safety

- Rollout-managed feature(s): none added; existing Anthropic memory
injection.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: pi cwd resolves; unresolved
project/empty scope no longer adds unusable proxy memory tools.
- Kill switch / disable path: existing `--no-memory`; explicit
project/cwd headers still work.
- Unsafe override required: no.
- Qualification impact: no storage migration or new dependency; scoped
HTTP/SQLite behavior checked.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — related
suite and critical gate, not a whole-suite claim
- [x] I did **not** edit `CHANGELOG.md` — release-please owns it

## Screenshots (if applicable)

N/A; the observable surface is the HTTP payload and project database.

## Additional Notes

AI-assisted; no subagents. OpenAI injection policy is unchanged. Keeping
this draft until the current hosted checks are qualified.

One earlier post-build probe completed the requests and closed both
listeners but hit its whole-process shutdown deadline. The same program
and native artifact later exited normally with faulthandler enabled; its
diagnostic abort timer did not fire. I have not identified the
intermittent cause and am not claiming a shutdown fix. No production
timeout or feature flag was changed to get that result.


The two npm audits now pass on 5ace63e2, along with the other Security
jobs. The earlier [unchanged-base
failure](https://github.com/headroomlabs-ai/headroom/actions/runs/41399323584)
is historical evidence, not a current blocker. This PR changes no
dependency manifests, lockfiles, workflow or audit exceptions.


### Maintainer current-main requalification

Exact head `60597623e00967153d947a46bf197950250f310e`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Clean mechanical update from current upstream main, preserving the pi
system-cwd resolver, request-scoped project guard, main's cache-mode
search skip/logging, and client-tool preservation. Ordinary user cwd
blocks still do not enable the pi fallback; unresolved project scope
still blocks both context injection and replayed memory tools. No
contributor production logic was rewritten and no new production finding
remains in this reviewed scope.

The initial combined tree dropped main's memory-invariant file and two
stream/timeout regression cases. Maintainer test-only repairs restore
those tests rather than reducing coverage: the invariant file is
byte-identical to main, every main assertion in the four
restored/updated test files is preserved by AST comparison, and three
fixtures now supply the handler protocol methods needed for resolved
scope and beta headers. The lifecycle main control passes without that
fixture extension; no claim is made that this particular fixture failure
independently fails main. All 222 affected
scope/invariant/stream/timeout/lifecycle/backpressure/cache-stability/system-prompt
tests pass with one skip. Ten shared Gemini request-fixture cases also
pass after the separate test-only repair for independently verified main
failures. All 12 main-relative changed Python files pass Ruff/format;
whitespace checks pass.

The author's existing real TCP/SQLite/ONNX reproduction and explicit
limits were read and remain in the body. These fresh Windows checks
qualify the combined request-scope regressions; they are not a claim of
rerunning that Linux proof, live pi/client/provider behavior, or a fresh
native build. An existing binary was supplied solely for test startup.
Draft status is preserved; readiness remains the author's decision and
nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `60e01cf7c25e90abb9f5cb66fa221da83e7c5177`, upstream main
`ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`.

Clean integration with upstream/main
ea59ac3eef. Re-reviewed project-scoping
delta against intervening memory/backend/stream changes: pi cwd remains
limited to system context; ordinary user cwd blocks do not enable
fallback; unresolved projects skip context/tool injection and sticky
tool replay while retaining client tools and main cache-mode no-search
behavior.232
scope/invariant/stream/timeout/lifecycle/backpressure/cache/system-prompt/Gemini
tests pass,1optional skip; touched-source Ruff and whitespace pass. No
contributor production policy rewritten. Earlier author TCP/SQLite/ONNX
evidence remains scoped, with no fresh native build or live-pi/provider
claim. Existing draft preserved; final approval awaits exact-head hosted
qualification. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `e93ceb35df7f78bfa2dcc0611609131cf2b08b0a`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 232 passed, 1 skipped, 1 warning in
47.62s. Selected real dependency overlays and native DLL paths are
recorded in final4618-fresh-native-tests.log; the DLL provenance/input
hashes are in final-native-artifact-proof.json. No redundant rerun is
implied after this Cargo-only merge. Whitespace passes. Prior source
review and scoped real-behavior evidence remain applicable to unchanged
contributor inputs.

No new source finding in the reviewed scope. Final formal approval
awaits exact-head hosted qualification. Project resolution and
unresolved-scope injection/replay gates remain unchanged. Existing draft
status is preserved.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `d693dd2b6e33fd47ff853f7e28d938fc1dbc30ac`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Current-main requalification
against2b0315b2d4b0d54947050bbdad2de9e89df06378. Clean source
integration preserves pi systemcwd resolution and unresolved-scope
context/tool/sticky-replay gates.70integration patches exactly matchmain
delta;Anthropic exception patch differs only byclosing-parenthesis diff
alignment and its exception AST equalsmain. Broad
memory/router/projectscope/continuation/backpressure/cache/lifecycle/doctor/nativeops
run:405passed,1skipped,one newmain fake-handler fixture failed
missingis_project_unresolved. Maintainer adds only that fake method
returningFalse,respecting resolvedfixture scope;all39backpressure tests
nowpass,including unchanged
injectedSQLiteexception,HTTP200andsemaphorerelease assertions. No
production logic changed.33changedPythonlint/format,offline
lock/whitespacepass;fixedtestlint/formatrechecked.
Existingpr3982nativeDLLverified fullinputhashes/checksum;no
freshbuildclaimed. Currentauthor/Devin discussion checked,inline
endpoint empty;prior scoped realbehavior evidence remains applicable to
unchanged contributor inputs. No new sourcefinding;no
realprovider/agentlaunch claim. Draftpreserved.
Exactnewheadhostedqualification pending;maintainers
handlechecks,nothingmerged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `0da0b5a01958eaaac1d625227cb4bdb9477c5d3b`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main delta after diff-location
normalization. Contributor pi system-cwd resolution and
unresolved-project memory context/tool/sticky-replay gates are
unchanged. Fresh
memory-router/project-scope/streaming/continuation/backpressure/cache/lifecycle
and incoming Vertex/backend/CLI tests: 381 passed, 12 skipped, 1
warning. Twelve changed Python files pass Ruff lint/format; offline lock
and whitespace pass.

Static production/test scope and security review considered user versus
system cwd inputs, malformed/empty cwd, missing scope and stale tool
replay: tests keep pi cwd tags in ordinary user content from resolving
project scope, and unresolved scope blocks replay. No unrelated
executable, workflow, dependency or setup changes. Flagged inherited
workflow-validation job is excluded by unchanged workflow paths. Author
discussion checked; collected inline endpoint has no findings. Prior
scoped behavior evidence applies to unchanged contributor inputs; no new
real agent/provider launch claimed. Native DLL reused only after
complete tracked input-set/hash and artifact checksum verification, no
fresh build claimed. Draft preserved. Formal approval awaits exact
published-head hosted qualification. Maintainers handle checks; nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-10 22:28:01 -06:00
Luis ArayaandJerrettDavis f25fcc58cd feat(proxy): demote cache-mode frozen-prefix restore log from WARNING to INFO (#4635)
## Description

In `--mode cache`, restoring the frozen prefix to preserve prompt cache
stability is the expected routine behavior across turns. However,
`headroom/proxy/handlers/openai.py` logged this occurrence at `WARNING`
level:

```python
logger.warning(
    f"[{request_id}] Restored {restored_count} frozen prefix message(s) "
    "to preserve cache stability (openai)"
)
```

On production gateways, this routine log line accounted for thousands of
warnings daily (approx. 65% of total container WARNING volume),
generating significant log noise and obscuring actual anomalies during
incident triage (as reported in #4505).

This PR demotes the log line from `WARNING` to `INFO`, matching the
convention of routine cache-mode stability logging in sibling handlers
and preserving all details (`request_id`, `restored_count`).

Closes #4505

## Type of Change

- [x] Performance improvement / log signal-to-noise improvement
- [x] Code quality / maintenance

## Changes Made

- `headroom/proxy/handlers/openai.py`:
- Changed `logger.warning` to `logger.info` when `restored_count > 0` in
`is_cache_mode` restore branch.
- `tests/test_proxy_system_prompt_immutable.py`:
- Added `test_cache_mode_frozen_prefix_restore_logs_at_info_level` to
verify that prefix restoration logs at `INFO` and produces zero
`WARNING` logs.

## Testing

- [x] Unit tests pass:
  - `tests/test_proxy_system_prompt_immutable.py`: 9 passed
  - `tests/test_proxy_handler_helpers.py`: 87 passed
- [x] Linting passes: `ruff check` on modified files
- [x] Formatting passes: `ruff format --check` on modified files
- [x] New regression test added covering log level verification

### Test Output

```text
tests/test_proxy_system_prompt_immutable.py ......... [100%]
======================= 9 passed, 3 warnings in 8.26s =======================
```

## Real Behavior Proof

- Environment: Windows, Python 3.12.16, offline environment.
- Exact command / steps:
1. Ran `TestClient` post request against `/v1/chat/completions` with
proxy configured in `mode="cache"`, `optimize=True`, `frozen_count=1`.
2. Captured logging output via pytest `caplog` fixture set to
`logging.INFO` on `headroom.proxy.handlers.openai.logger`.
3. Evaluated emitted log records when pipeline mutation triggers
`_restore_frozen_prefix`.
- Observed result:
- Exactly 1 record emitted with `levelno == logging.INFO` matching
`Restored 1 frozen prefix message(s) to preserve cache stability
(openai)`.
- Zero log records emitted at `WARNING` or higher for prefix
restoration.
- Not tested: Live production gateway log ingestion pipelines.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes; routine log level demoted from
WARNING to INFO.
- Kill switch / disable path: N/A (revert commit)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I did not edit `CHANGELOG.md`

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4635"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer post-merge runtime evidence

Reviewed combined head `8f82977fd23f6d254ea4ebe5dce60f44d94dc485`,
including main `365f6fe52`. All 11 combined immutable-prompt/cache-mode
tests pass. The logging regression independently fails on current main
because it emits the previous WARNING record.

Runtime setup: Windows, Python 3.12.13, patched worktree source and the
existing local native extension. A real Uvicorn proxy and a separate
local HTTP provider ran on ephemeral loopback ports. Production HTTP
forwarding was retained; `_retry_request` was not patched. A
deterministic instrumented pipeline deliberately changed the frozen
system text, and a one-message frozen-prefix tracker exercised the real
restoration path.

Exact command from the isolated worktree, with
`PYTHONPATH=G:/git/headroom-review4635` and `HEADROOM_OFFLINE=1` set
through PowerShell environment assignments:
`G:/git/headroom/.venv/Scripts/python.exe
G:/git/headroom/.codex/review4635-loopback.py`.

Observed output: `{"status":200,"forwarded_system":"frozen
system","restore_records":1,"restore_level":"INFO"}`. The real upstream
received the restored original system text; exactly one restoration log
was emitted at INFO, with no restoration WARNING. The proxy and local
provider were shut down after the probe.

Not tested: production log aggregation, an external provider, production
compression/model performance, or a fresh local native compilation. The
injected deterministic pipeline is a trigger for restoration, not a
claim about a production compressor. Updated-head hosted qualification
is handled by maintainers; no contributor CI action is needed. Nothing
merged.


### Maintainer current-main requalification

Exact head `c32d16e0a762748c3d21c113d42a65f9d8ef9dc3`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Clean mechanical update from current upstream main. The production delta
remains exactly the frozen-prefix restoration log level, WARNING to
INFO; the message, request ID, restored count, original prefix
restoration and unrelated warnings are preserved. Thirteen cache-prompt
tests pass, and the logging regression independently fails with the main
WARNING implementation before restoring this PR's source. Ten shared
Gemini request-fixture cases pass after the test-only repair for
independently verified main failures. Changed-file Ruff/format and
whitespace checks pass.

The real TCP Uvicorn-to-loopback-provider probe on this combined tree
returns HTTP 200, forwards the restored original system text, and
records exactly one INFO restoration message. It uses a controlled
mutating pipeline/session tracker to exercise restoration; it does not
qualify model-based compression, live external providers or production
log ingestion. The existing native binary was supplied only for startup,
not as a fresh native-build claim. No new correctness finding and no
contributor production logic was rewritten.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `5b3d67e97a7b6c7b7aabf75005784420187c51a7`, upstream main
`ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`.

Clean main synchronization with upstream
ea59ac3eef and re-review of the OpenAI
handler overlap. Production delta remains exactly one WARNING-to-INFO
change for routine frozen-prefix restoration; payload restoration,
message/request-id/count and unrelated warnings are unchanged.23
immutable-system-prompt/Gemini-memory tests pass; Ruff and whitespace
pass. Main now includes the shared streamed-request fixture, so it no
longer appears in this PR delta. Prior actual TCP restoration/logging
proof remains scoped to unchanged restoration behavior; no
external-provider/log-ingestion/fresh-native-build claim. No new
finding; final approval awaits exact-head hosted qualification. Nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `72a633892f45eb0433aa745e2125d505b3eb282d`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 23 passed, 1 warning in 15.82s.
Selected real dependency overlays and native DLL paths are recorded in
final4635-fresh-native-tests.log; the DLL provenance/input hashes are in
final-native-artifact-proof.json. No redundant rerun is implied after
this Cargo-only merge. Whitespace passes. Prior source review and scoped
real-behavior evidence remain applicable to unchanged contributor
inputs.

No new source finding in the reviewed scope. Final formal approval
awaits exact-head hosted qualification.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `2fee5c09750bda95f388b1fc22628d07c06be441`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Current-main requalification
against2b0315b2d4b0d54947050bbdad2de9e89df06378. All71 integration
patches exactly match c01cf07-to-current-main delta after diff-location
normalization; entire production PR delta remains one WARNING-to-INFO
log level change,with message/requestID/count/restoration semantics
preserved.163cache-prompt/Gemini/Kompress-disable/doctor tests
pass,1deprecationwarning. Fresh realTCP Uvicorn-to-local-provider probe
returnsHTTP200,forwards restored original system text,and records
exactlyoneINFOrestoration message. Controlled mutating
pipeline/sessiontracker intentionally exercises restoration;not live
model compression,cloud provider,or production log ingestion
proof.33changedPythonfiles pass Ruff lint/format;offline
lock/whitespacepass. Existingpr3982nativeDLLverified
fullnativeinputhashes/artifactchecksum;no freshbuildclaimed.
Currentauthor/Devin discussion considered;prior mocked-only proof
concern addressed by scoped TCPforwarding evidence. No new
correctnessfinding. Exact-head hostedqualification
pendingbeforeapproval;maintainers handlechecks,nothingmerged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `a093e815fc194c62e803740e39f1900e5383e7fc`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main delta after diff-location
normalization. Entire production PR delta is still one WARNING-to-INFO
level change; restoration, message, request ID and count semantics
preserved. Fresh cache-prompt and incoming Vertex/backend/CLI tests: 171
passed, 11 skipped, 1 warning. Twelve changed Python files pass Ruff
lint/format; offline lock and whitespace pass.

Fresh real TCP Uvicorn-to-local-provider probe returns HTTP200, forwards
restored original system text and emits exactly one INFO restoration
record. A controlled mutating pipeline/session tracker exercises the
restoration path; this is not live model compression, cloud-provider or
production ingestion proof. Author/Devin discussion checked; mocked-only
qualification concern addressed by this scoped TCP evidence. Static
production/test review found no unrelated executable, workflow or
dependency changes; flagged inherited workflow-validation job is
excluded by unchanged workflow paths. Native DLL reused after complete
tracked input-set/hash and artifact checksum verification, no fresh
build claimed. Formal approval awaits exact published-head hosted
qualification. Maintainers handle checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-07 13:33:47 -06:00
Luis ArayaandJerrettDavis 36540379e0 fix(proxy): skip memory-context search on OpenAI paths in cache mode (#4634)
## Description

Follow-up to #4606. In `--mode cache`, the proxy preserves prompt cache
stability across turns by never mutating or appending to user turns.
While #4606 addressed this for Anthropic's `/v1/messages` by skipping
`search_and_format_context`, the three OpenAI-compatible paths
(`/v1/chat/completions`, `/v1/responses`, and `/v1/responses` WebSocket)
still executed the memory search and injected memory context into the
user turn even in cache mode.

This PR aligns the OpenAI handlers with Anthropic: when in cache mode
with `inject_context` enabled, it logs `skipped_cache_mode`, stamps
`memory_skip_reason="cache_mode"` in the request/frame tags, and skips
`search_and_format_context`. Memory tool injection and user input bytes
remain unchanged.

Closes #4622

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/openai.py`:
- `handle_openai_chat`: skip `search_and_format_context` when
`is_cache_mode` is active, stamp `tags["memory_skip_reason"] =
"cache_mode"`, log `decision="skipped_cache_mode"`, and keep the latest
user message unmutated.
- `handle_openai_responses`: skip `search_and_format_context` when
`is_cache_mode` is active, stamp `tags["memory_skip_reason"] =
"cache_mode"`, and log `decision="skipped_cache_mode"`.
- `handle_openai_responses_ws`: skip `search_and_format_context` when
`is_cache_mode` is active, stamp `ws_tags["memory_skip_reason"] =
"cache_mode"`, log `decision="skipped_cache_mode"`, and avoid input
mutation.
- `tests/test_proxy_system_prompt_immutable.py`:
- Updated `test_cache_mode_skips_memory_context_search_openai_chat` to
verify `memory_skip_reason="cache_mode"` on recorded outcomes and
preservation of memory tools.
- Updated `test_cache_mode_skips_memory_context_search_openai_responses`
to verify `memory_skip_reason="cache_mode"` on recorded outcomes and
preservation of memory tools.
- `tests/test_codex_ws_per_frame_memory.py`:
- Expanded `test_cache_mode_skips_memory_context_search_ws` with
multiple response.create frames verifying input preservation, memory
tool preservation, and `memory_skip_reason="cache_mode"` in turn outcome
tags.

## Testing

- [x] Unit tests pass:
  - `tests/test_proxy_system_prompt_immutable.py`: 10 passed
  - `tests/test_codex_ws_per_frame_memory.py`: 12 passed
- [x] Linting passes: `ruff check` on modified files
- [x] Formatting passes: `ruff format --check` on modified files
- [x] New regression tests added for all 3 OpenAI paths in cache mode
- [x] Manual testing performed

### Test Output

```text
tests/test_proxy_system_prompt_immutable.py .......... [ 45%]
tests/test_codex_ws_per_frame_memory.py ............   [100%]
======================= 22 passed, 4 warnings in 18.06s =======================
```

## Real Behavior Proof

- Environment: Windows, Python 3.12.16, `HEADROOM_OFFLINE=1`.
- Exact command / steps:
1. Ran `pytest tests/test_proxy_system_prompt_immutable.py` exercising
`/v1/chat/completions` and `/v1/responses` in cache mode with mock
memory handler and outcome recorder.
2. Ran `pytest tests/test_codex_ws_per_frame_memory.py -k
test_cache_mode_skips_memory_context_search_ws` exercising two
`response.create` frames in cache mode with mock WebSocket and outcome
recorder.
3. Checked that `memory_handler.search_and_format_context` is never
awaited on any path.
4. Checked that `memory_skip_reason == "cache_mode"` reaches
`RequestOutcome.tags` across Chat, Responses, and WebSocket turns.
5. Checked that input bytes and client/memory tools (`client_tool`,
`mem_search`) remain untouched in the forwarded payloads.
- Observed result:
- `memory_handler.search_and_format_context.assert_not_awaited()` passed
on all routes.
- `outcome.tags.get("memory_skip_reason") == "cache_mode"` passed on
Chat, Responses, and WS frames.
- Forwarded inputs matched raw client inputs without injected memory
context.
- Forwarded tools retained `client_tool` and memory tools
(`mem_search`).
  - 22 tests passed across both test files.
- Not tested: Live cloud provider endpoints.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes; in cache mode, memory context is
no longer injected into OpenAI requests, ensuring next-turn prefix bytes
remain byte-identical, and request/frame tags now record
`memory_skip_reason="cache_mode"`.
- Kill switch / disable path: N/A (revert commit)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I did not edit `CHANGELOG.md`

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-04 14:38:43 -06:00
365f6fe523 fix(proxy): skip memory-context search in cache mode on OpenAI routes (#4632)
## Description

Follow-up to #4600 / #4606 as requested in #4622. PR #4606 implemented
this optimization for Anthropic `/v1/messages`. This PR brings the three
OpenAI routes (`handle_openai_chat`, `handle_openai_responses`, and
`handle_openai_responses_ws`) into alignment.

In `--mode cache`, memory context injection is disabled so prompt prefix
bytes remain byte-stable across turns. Previously, all three OpenAI
paths still executed the memory-context search
(`search_and_format_context`: embedding query, vector index lookups,
graph searches) and appended the result to the latest user turn in cache
mode, introducing latency (~1–37ms+ per turn) and altering the forwarded
user turn.

With this PR:
1. `handle_openai_chat`: checks `is_cache_mode(self.config.mode)` before
executing `search_and_format_context`, emits
`decision="skipped_cache_mode"`, and keeps the user turn intact.
2. `handle_openai_responses`: checks `is_cache_mode(self.config.mode)`
before searching, emits `decision="skipped_cache_mode"`, and keeps
`body["input"]` intact.
3. `handle_openai_responses_ws`: checks
`is_cache_mode(self.config.mode)` before measuring `memory_context`,
emits `decision="skipped_cache_mode"`, and leaves
`ws_response_body["input"]` untouched.
4. Memory tool injection remains completely unchanged across all routes.

Closes #4622

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/openai.py`: Check
`is_cache_mode(self.config.mode)` when `inject_context` is enabled
across `handle_openai_chat`, `handle_openai_responses`, and
`handle_openai_responses_ws`. Emit `decision="skipped_cache_mode"` with
`bytes_injected=0` and bypass `search_and_format_context`.
- `tests/test_proxy_system_prompt_immutable.py`: Added
`test_cache_mode_skips_memory_context_search_openai_chat` and
`test_cache_mode_skips_memory_context_search_openai_responses` to verify
that `search_and_format_context` is never awaited in cache mode and user
message / input fields remain byte-equal to client values.

## Testing

- [x] Unit tests pass (`pytest
tests/test_proxy_system_prompt_immutable.py`)
- [x] Linting passes (`ruff check headroom/proxy/handlers/openai.py
tests/test_proxy_system_prompt_immutable.py`)
- [x] Formatting passes (`ruff format --check
headroom/proxy/handlers/openai.py
tests/test_proxy_system_prompt_immutable.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_proxy_system_prompt_immutable.py -q
10 passed, 1 warning in 31.92s

$ ruff check headroom/proxy/handlers/openai.py tests/test_proxy_system_prompt_immutable.py
All checks passed!

$ ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_system_prompt_immutable.py
2 files already formatted
```

## Real Behavior Proof

- Environment: Linux / WSL2 Ubuntu, Python 3.13, local proxy test
harness (`create_app` + `TestClient`).
- Exact command / steps: Executed requests against
`/v1/chat/completions` and `/v1/responses` with proxy configured in
`--mode cache` and memory context enabled. Verified upstream payload
captures and mock call history.
- Observed result: In cache mode, the memory search is bypassed on all
OpenAI requests; search_and_format_context was never awaited (0 calls),
user message content and input fields remain byte-equal without injected
memory context, and proxy emitted decision=skipped_cache_mode with
bytes_injected=0.
- Not tested: Live WebSocket connection to OpenAI upstream with
streaming audio/realtime endpoints.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no. Cache mode requires byte
stability across turns; memory search is skipped in cache mode. Token
mode is untouched.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: git revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation: n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Aligns OpenAI Chat, Responses, and WebSocket routes with Anthropic
behavior established in #4606.
- Preserves memory tool injection without modification.

Co-authored-by: shaiknaziya20 <naziyashaik5512@gmail.com>


### Maintainer combined-head validation

Reviewed head `66a09054548423e33af47121ac7e87134d023fe4` includes a
clean main sync through `12dca26aa` plus test-only maintainer
follow-ups: the existing WebSocket dummy config now explicitly uses
token mode, and a two-frame cache-mode regression covers skipped
searches, unchanged input, retained memory tools, passthrough
cancellation frames, and skip telemetry. No contributor runtime logic
was changed.

Windows / Python 3.12.13, with the existing local Rust extension:
`.venv/Scripts/python.exe -m pytest
tests/test_proxy_system_prompt_immutable.py
tests/test_codex_ws_per_frame_memory.py
tests/test_openai_codex_ws_lifecycle.py -q --tb=short
--disable-warnings` => **89 passed**. The new WebSocket regression fails
on current main because both frames still execute memory searches. Ruff
check/format and main-relative whitespace checks pass.

Additional runtime proof used a real Uvicorn proxy and a separate local
HTTP upstream, both bound to ephemeral loopback ports, with production
HTTP forwarding intact (no `_retry_request` patch). A deterministic
recording memory backend returned `LOOPBACK_MEMORY` when searched, so
the probe can distinguish an actual skip from an unconfigured/disabled
memory path. The probe sent the same authenticated Chat and Responses
requests in token mode, then cache mode, using user text `latest user
prompt` and fixed Responses instructions.

Exact invocation from the isolated worktree:
`PYTHONPATH=G:/git/headroom-review4632 HEADROOM_OFFLINE=1
G:/git/headroom/.venv/Scripts/python.exe
G:/git/headroom/.codex/review4632-loopback.py` (environment variables
set with PowerShell `$env:` assignments). Results: **four HTTP 200
responses**; token mode performed one memory search per route and
forwarded the returned memory text; cache mode performed zero searches
per route, forwarded the original user text unchanged, and recorded
`memory_skip_reason=cache_mode`. The proxy and upstream were shut down
after the run.

Limits: the loopback provider and recording memory backend are
deterministic local fixtures; no external provider, embedding model,
production memory service, load test, or real upstream WebSocket was
exercised. WebSocket multi-frame/input/tool/telemetry behavior is
covered by the focused regression above. Updated-head hosted
qualification is inspected by maintainers; no contributor CI action is
needed. Nothing merged.

---------

Co-authored-by: shaiknaziya20 <naziyashaik5512@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-04 12:37:00 -06:00
SDandJD Davis 053925b03c fix(proxy): skip the memory-context search in cache mode (#4606)
## Description

In `--mode cache`, the Anthropic `/v1/messages` handler ran the full
memory-context search (`search_and_format_context`: embed the query,
query the vector index and graph) on every request, and only then
checked `is_cache_mode()` and discarded the result with
`decision=skipped_cache_mode`. Cache mode never injects memory context,
so the search was pure latency before the request goes upstream.

This PR checks cache mode first and skips the search entirely. The
`skipped_cache_mode` decision is still logged, and memory tools are
still injected.

Closes #4600

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/anthropic.py`: when `inject_context` is on
and the proxy is in cache mode, log `skipped_cache_mode` and skip
`search_and_format_context`. The old post-search cache-mode branch is
removed.
- Corrected the gate comment: `decide_memory_injection` only skips for
`HEADROOM_MEMORY_INJECTION_MODE=disabled`, not `tool` (as noted in the
issue).
- `tests/test_proxy_system_prompt_immutable.py`: added
`test_cache_mode_skips_memory_context_search`, which sends a cache-mode
request with memory on and asserts the search is never awaited and the
latest user turn is forwarded unchanged.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): see Additional Notes
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New test, before the fix:

```text
E   AssertionError: Expected mock to not have been awaited. Awaited 1 times.
FAILED tests/test_proxy_system_prompt_immutable.py::test_cache_mode_skips_memory_context_search
```

After the fix:

```text
$ pytest tests/test_proxy_system_prompt_immutable.py -q
8 passed

$ pytest tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_memory_integration.py \
    tests/test_memory_tool_mode.py tests/test_memory_decision.py \
    tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_pipeline_lifecycle.py \
    tests/test_proxy_byte_faithful_forwarding.py -q
170 passed, 12 skipped

$ ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy_system_prompt_immutable.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, local memory backend
(`all-MiniLM-L6-v2` ONNX embedder, hnswlib vector index),
`memory_storage_mode=global` with 50 seeded memories.
- Exact command / steps: an in-process benchmark that starts the real
proxy app (`create_app` + `TestClient`) with `--mode cache` and memory
enabled, stubs only the upstream call, and sends 100 `/v1/messages`
requests. It counts calls to `search_and_format_context` and times each
request end to end. Two alternating rounds of `main` (fc1ba891) vs this
branch.
- Observed result: in cache mode, `main` ran the memory search on 100 of
100 requests (p50 ~123 ms); this PR ran it on 0 of 100 (p50 3–5 ms),
about 120 ms saved per request. Full numbers below.

| Run | Searches in cache mode | p50 | p95 | mean |
|---|---|---|---|---|
| `main`, round 1 | 100 / 100 | 122.7 ms | 152.9 ms | 126.5 ms |
| this PR, round 1 | 0 / 100 | 2.9 ms | 3.6 ms | 3.0 ms |
| `main`, round 2 | 100 / 100 | 124.0 ms | 266.7 ms | 157.6 ms |
| this PR, round 2 | 0 / 100 | 4.8 ms | 10.1 ms | 5.6 ms |

About 120 ms p50 is removed from every cache-mode request. This matches
the issue's production numbers (`memory_context` p50 114 ms on v0.42.0,
p95 up to 1,169 ms on v0.40.0).
- Not tested: the benchmark store returned no hits on search (the
reopened HNSW index comes back empty, which looks like the reload defect
in #4362), so the `main` numbers cover embedding plus index and graph
query, but not result formatting. With real hits the saving is equal or
larger. Not run against a live upstream.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no. Cache mode already discarded the
search result; only the wasted work is removed. Token mode is unchanged.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation: n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- `skipped_cache_mode` is now logged on every cache-mode request with
`inject_context` on. Before, it was logged only when the search returned
something.
- The OpenAI and Gemini handlers also call `search_and_format_context`,
but they inject the result even in cache mode, so their search is not
wasted. Whether those paths should skip injection in cache mode is a
separate question and is left out of this PR.
- `mypy headroom/proxy/handlers/anthropic.py` stops in my local
environment on a `numpy` stub (`Type statement is only supported in
Python 3.12 and greater`) before reaching project code. That is
unrelated to this change, and CI runs mypy.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4606"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 19:35:14 -06:00
Mansurshakh JaparovandJD Davis 9553d4292f fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description

Session cleanup currently drops prefix trackers after the default
600-second session lifetime even when Anthropic's request selects a
one-hour cache TTL. Losing the tracker also loses the frozen forwarded
prefix and causes the cache-preserving overlay to replay different
bytes.

Closes #4017.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Resolve the request's existing Anthropic cache TTL before selecting
the prefix lineage and pass that TTL through tracker selection/creation.
- Retain each lineage for the greater of the configured session floor
and its observed/current provider TTL plus the existing 60-second safety
margin.
- Keep the lifetime per lineage rather than changing global
configuration: a five-minute session does not inherit another session's
one-hour lifetime.
- Preserve normal expiry beyond the provider lifetime and margin.
- Honor configured cache TTL before a confirmed cache write; discard
expired state before accepting a later request's longer TTL.
- Re-register the live lineage family after lookup/cleanup so recreated
sessions cannot share a tracker's frozen state across sibling
conversations or tool profiles.
- Update existing fake callbacks to the changed tracker contract,
preserve all consumer byte-stability assertions, and remove one
handler-to-store wiring-only test rather than re-pinning its internal
identity assertions.
- Document local tracker retention separately from provider cache
availability.

## Testing

- [x] Unit tests pass (`pytest`), including frozen-prefix byte stability
and independent TTL expiry.
- [x] Changed-file lint, configured pre-commit mypy and manual
production timestamp/overlay checks pass.

### Test Output

```text
Review correction: 273 passed, 2 dependency warnings in 83.79s
{"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true}
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate
explicitly used the installed CPU `libonnxruntime.so.1.26.0` via
`ORT_DYLIB_PATH`, without skipping cached-model assertions.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 12.87s ==================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_cache/test_prefix_tracker.py
python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py
```

**74 passed** in tracker tests; **54 passed** in request/cache tests
(one existing Starlette TestClient deprecation warning). Two new
regressions fail on baseline because the request TTL is not accepted; a
separate production runtime comparison demonstrates the actual pre-fix
loss, not just the missing parameter.

After rebasing onto current main
`1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together
with `tests/test_prefix_tracker_client_rewrote_reply.py`,
`tests/test_cache_mode_delta_marker.py`, and
`tests/test_content_router_mcp_small_result.py`: **150 passed**. The
actual production timestamp/byte-retention smoke was repeated
successfully on the rebased code, covering compatibility with main's
newly merged client-reply normalization.

Review regression and affected HTTP-consumer suite: **273 passed**,
including tracker expiry, Anthropic/OpenAI cache stability,
byte-faithful forwarding, header isolation, buffered timeouts, CCR
injection, and immutable system prompts. All affected two-argument test
callbacks now accept the TTL contract. The obsolete walker-call-order
test was removed; wire-byte assertions remain. The new
configured-TTL/expiry cases fail against the published baseline; the
post-cleanup regression also verifies that an unrelated conversation
gets a different tracker after recreation.

Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and
Commitlint hooks passed. Full-tree Ruff is locally blocked by executable
filesystem permissions on five unchanged OAuth2 plugin files (tracked
100644). A separate full dev-environment mypy run found an unchanged
optional LangChain `truthy-function` error; the configured isolated
pre-commit hook passed. No unrelated source/permissions changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real
`SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token
estimator and `overlay_cached_prefix`, no mocked store or mocked clock.
- Exact command / steps: parse `cache_control: {type: ephemeral, ttl:
1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed
frozen message with different original/forwarded prefix bytes; age
`_last_activity` to `time.time()-660` and make periodic cleanup due;
resume the same lineage and compare the tracker, frozen count, and
forwarded-prefix bytes. Also age beyond 3901 seconds and compare
independent 300/3600-second sessions.
- Before: resolved TTL 3600; after 660 seconds, tracker identity
changes, frozen count drops **1→0**, and forwarded-prefix bytes differ.
- Observed result: after the fix, after 660 seconds the same tracker
remains, frozen count stays **1**, and forwarded-prefix bytes are
identical. The one-hour session survives independently, the five-minute
session expires, and the one-hour lineage still expires beyond TTL plus
margin.
- Review runtime proof against published head `e1a107df`: actual stores
and prefix overlays confirm configured TTL retention before the first
cache write, no resurrection of expired state by a later one-hour
request, eventual expiry, and separate trackers after the maintainer's
3901-second cleanup/recreation sequence. On that exact baseline the
unrelated lookup reuses the wrong tracker, although the overlay guard
preserves the particular unrelated input bytes; after the fix both
identity isolation and byte preservation hold. Identical messages under
different tool profiles also receive a cold, separate tracker after the
expired bare-key transition.
- Not tested: Anthropic cloud cache availability, billed savings, or an
actual one-hour wait. The local production state and byte invariants
were exercised by aging actual timestamps, not by a paid upstream
request. Preserving a tracker cannot guarantee that the provider retains
its cache.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; repair of existing TTL-aware
cache-prefix behavior.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: eligible local prefix trackers
survive for their provider cache lifetime. Short-TTL/default sessions
remain bounded by their own session floor.
- Kill switch / disable path: the existing prefix-freeze disable path
remains available; no new flag.
- Unsafe override required: no.
- Qualification impact: local Anthropic prefix lineage
selection/retention, not cache pricing or subscription quota policy.
- Rollback path: revert the commit to restore previous retention. No
persistent schema change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing cache TTL parser and safety margin reused; cache docs
updated.
- [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes,
and existing request byte-stability behavior.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

Remote CI results are not inferred from local verification.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 17:46:21 -05:00
chopratejas f0dcc02775 fix: A3 — byte-faithful Python forwarders; serialize canonical only when mutated
Eliminates P0-2 universally. Every Python forwarder (server.py
`_retry_request`, handlers/streaming.py `_stream_response`,
handlers/openai.py `_ws_http_fallback`, handlers/batch.py `_batch_passthrough`
+ batch-create + Google batch passthrough, handlers/anthropic.py CCR
continuation + batch endpoint) now switches from `httpx ... json=body` to
`httpx ... content=raw_bytes`. The default httpx JSON encoder was
re-serializing every request with `, `/`: ` separators and `\\uXXXX` ASCII
escapes — collapsing Anthropic prompt-cache hit-rate.

Forwarder strategy:
  - unmutated body → forward `await request.body()` verbatim;
  - mutated body  → re-serialize once via the new
    `serialize_body_canonical(body) -> bytes` helper (compact separators,
    `ensure_ascii=False`, dict insertion order preserved).

`HEADROOM_PROXY_PYTHON_FORWARDER_MODE` env var configures the mode:
  - `byte_faithful` (default) — the new behavior;
  - `legacy_json_kwarg` — explicit operator opt-in for emergency rollback.
Documented in `docs/content/docs/configuration.mdx`. NOT a fallback —
unknown values raise loudly per build constraint #4.

`BodyMutationTracker` accompanies each request through the handler so
transform sites mark the tracker (`memory_injection`,
`image_compression`, `compression_*`, `batch_compression`,
`ccr_continuation`, etc.). At forwarder dispatch we additionally compare
the final body dict against the parsed original bytes as a structural
safety net — any silent mutation we missed still triggers canonical
re-serialization.

A2 follow-up: `handlers/openai.py:534-540` (Chat Completions memory
injection) was prepending a system message; replaced with
`append_text_to_latest_user_chat_message`, the OpenAI Chat Completions
analog of `_append_context_to_latest_non_frozen_user_turn`. The cache
hot zone (system messages) is now sacrosanct on /v1/chat/completions
too. Honors `HEADROOM_MEMORY_INJECTION_MODE=disabled`.

Structured logging: every forwarder emits an `event=outbound_request`
log line with `forwarder`, `path`, `body_bytes`, `body_mutated`,
`mutation_reasons`, `source` (passthrough|canonical|legacy),
`request_id`. Never logs Authorization or full body.

`_read_request_json` factored to share `_read_request_body_bytes` with
new `read_request_json_with_bytes` so the anthropic handler can capture
both the parsed dict and the original (decompressed) bytes.

Tests:
  - `tests/test_proxy_byte_faithful_forwarding.py` (28 tests):
    SHA-256 byte-equality on /v1/messages and streaming, unicode
    preservation, numeric precision, mutation-tracker invariants,
    canonical-serializer properties, legacy-mode rollback, OpenAI
    Chat memory routing.
  - Existing test mocks updated to accept the new `**kwargs` on
    `_retry_request` (no behavior change).
  - `tests/test_proxy_handlers_batch.py` updated to read the captured
    `content=` bytes (formerly `json=`).
  - One A2 test corrected (`test_anthropic_tool_sort_and_context_append_helpers`)
    to match the live-zone-tail semantics introduced by A2.

Constraints satisfied: configurable env var; no new regex / hardcodes;
no silent fallback (`legacy_json_kwarg` is operator opt-in);
performant (`prepare_outbound_body_bytes` is O(1) for passthrough);
elegant single-responsibility helpers; structured tracing logs.
2026-05-02 09:02:10 -07:00
chopratejas 704fb2f19d fix: A2 — system prompt immutable; memory routes to live-zone tail; cache_aligner detector-only
P0-1: Delete `_inject_system_context` from `proxy/server.py`. Memory
context now routes exclusively to the first text block of the latest
non-frozen user message via `_append_context_to_latest_non_frozen_user_turn`
(promoted to the canonical default in handlers/anthropic.py). Mirror
applied to OpenAI Responses API at handlers/openai.py: `body["instructions"]`
is no longer mutated; memory context appends to the latest user item in
`body["input"]`.

P2-23: Replace `headroom/transforms/cache_aligner.py` with a detector-only
implementation. The legacy rewrite path (~400 LOC) is removed. The volatile-
content detector uses no regex — UUIDs via `uuid.UUID`, ISO 8601 via
`datetime.fromisoformat`, JWT shape via base64url segment-count check, hex
hashes via length + `int(token, 16)` validation. Volatile findings surface
through `cache_metrics`/`warnings`/`logger.warning`; the prompt is never
mutated.

Configurability: new env var `HEADROOM_MEMORY_INJECTION_MODE` with values
`live_zone_tail` (default) and `disabled`. No `system_prompt` value — that
path is permanently retired.

Structured logs: every memory injection emits `event=memory_injection`
with `decision`, `bytes_injected`, `query_hash` (BLAKE2b, never raw query),
`session_id`, `request_id`. Auth is never logged.

Tests:
- Add `tests/test_proxy_system_prompt_immutable.py` (7 tests).
- Add `tests/test_cache_aligner_detector_only.py` (20 tests).
- Replace `tests/test_transforms/test_cache_aligner.py` (rewrite-path
  tests, 58 cases) with detector-only behavior.
- Update `tests/test_acceptance.py::TestDateTrap` to pin the new
  detector-only contract.

Acceptance:
- `git grep -n "_inject_system_context\|_inject_to_system_or_instructions" headroom/`
  returns nothing.
- `git grep -n "import re\|from re import" headroom/transforms/cache_aligner.py`
  returns nothing.
- Targeted suite (`test_proxy_system_prompt_immutable.py`,
  `test_cache_aligner_detector_only.py`, `test_proxy_anthropic_cache_stability.py`,
  `test_acceptance.py::TestDateTrap`, `test_memory*.py`, `test_cli/`) green.
2026-05-02 08:34:34 -07:00