fix(codex): resolve per-turn project context (#2636)

## Description

Codex Desktop can send turns from multiple projects through one HTTP or
WebSocket route, so a static `/p/<project>` route cannot reliably
identify each turn.

This PR resolves project context per turn from explicit overrides or
Codex thread/turn metadata. Unsafe or inconsistent resolution skips
project memory and learning while the model request continues.

Related to #2355 and overlaps #2379.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Resolve `thread_id` and `turn_id` to the matching rollout cwd.
- Use the resolver for HTTP and per-frame WebSocket traffic.
- Pin each WebSocket to its first resolved project and fail closed on
mismatch.
- Strip Codex-only metadata before forwarding upstream.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
121 passed, 1 pre-existing deprecation warning
All checks passed!
Success: no issues found in 510 source files
```

## Real Behavior Proof

- Environment: macOS; isolated Codex state and project fixtures plus the
combined local deployment.
- Exact command / steps: Exercised two same-basename projects over HTTP
and concurrent WebSockets, including resume/fork, missing, stale,
locked, malformed, and mismatched contexts.
- Observed result: Each turn resolved to the correct project; unsafe
cases skipped project memory and learning without blocking the model
request. The integrated patch also ran in the local deployment for more
than 39 hours.
- Not tested: Linux, Windows, VMs, or GitHub CI.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; this change uses stable/default
behavior.
- Stable/default behavior changed: Codex HTTP and WebSocket turns
resolve project context per turn; unsafe resolution skips project memory
and learning.
- Kill switch / disable path: Run without `--memory`; use `--no-learn`
to disable traffic learning.
- Unsafe override required: No.
- Qualification impact: Codex project-context isolation for HTTP and
WebSocket traffic.
- Rollback path: Revert this PR. No schema, data, or configuration
migration is required.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] Documentation changes are not required for this internal routing
fix
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Waiting follow-up: [project-scoped
TrafficLearner](https://github.com/krzkraw/headroom/tree/krz/project-scoped-traffic-learner).
It depends on the per-turn project API introduced here.

No database or configuration migration is required.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2636"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Krzysztof Krawczyński
2026-10-05 15:35:51 -05:00
committed by GitHub
co-authored by JD Davis
parent 0ad996a82e
commit 0396ea2fc2
6 changed files with 1668 additions and 59 deletions
+461
View File
@@ -0,0 +1,461 @@
"""Read-only Codex turn-to-project resolution for Responses traffic."""
from __future__ import annotations
import asyncio
import json
import logging
import os
import sqlite3
import time
from collections.abc import Mapping
from dataclasses import dataclass
from functools import lru_cache
from pathlib import Path
from typing import Any, Literal, cast
try:
import tomllib
except ModuleNotFoundError: # pragma: no cover - Python 3.10
import tomli as tomllib
from headroom.memory.storage_router import ProjectResolver, RequestContext
from headroom.providers.codex.threads import _codex_state_db_paths
logger = logging.getLogger(__name__)
_METADATA_HEADER = "x-codex-turn-metadata"
_MAX_METADATA_BYTES = 16 * 1024
_SQLITE_TIMEOUT_SECONDS = 0.1
_SQLITE_ATTEMPTS = 2
_ROLLOUT_CACHE_MAX_ENTRIES = 256
@dataclass(frozen=True, slots=True)
class CodexResolvedProject:
"""One optional project identity and its observable resolution result."""
cwd: Path | None
project_key: str | None
source: Literal[
"x-headroom-project-id",
"x-headroom-cwd",
"configured-project-root",
"codex-turn-metadata",
"codex-client-metadata",
"responses-body-cwd",
"unresolved",
]
reason: str
class CodexProjectContextResolver:
"""Map Codex ``thread_id`` + ``turn_id`` to an exact rollout cwd."""
def __init__(self, sqlite_home: str | Path | None = None) -> None:
self._configured_sqlite_home = Path(sqlite_home).expanduser() if sqlite_home else None
def resolve(
self,
*,
headers: Mapping[str, str],
body: Mapping[str, Any],
pinned_cwd: Path | None = None,
project_root_override: str | None = None,
) -> CodexResolvedProject:
explicit_project_id = self._header(headers, "x-headroom-project-id")
if explicit_project_id:
project_identity = ProjectResolver().resolve(
RequestContext(
headers=headers,
system_prompt="",
base_user_id="",
project_root_override=project_root_override,
)
)
return CodexResolvedProject(
cwd=None,
project_key=project_identity[0] if project_identity else None,
source="x-headroom-project-id",
reason="explicit_project_id",
)
explicit_cwd = self._header(headers, "x-headroom-cwd")
if explicit_cwd:
return self._explicit_cwd_result(
explicit_cwd,
headers=headers,
source="x-headroom-cwd",
pinned_cwd=pinned_cwd,
)
if project_root_override:
return self._explicit_cwd_result(
project_root_override,
headers=headers,
source="configured-project-root",
pinned_cwd=pinned_cwd,
)
identity, source, metadata_reason = self._turn_identity(headers, body)
if identity is not None:
thread_id, turn_id = identity
if not turn_id:
return self._fallback_or_skip(
body,
headers,
pinned_cwd,
"turn_id_missing",
)
cwd, reason = self._cwd_from_state(thread_id, turn_id)
if cwd is not None:
return self._cwd_result(
cwd,
headers,
cast(Literal["codex-turn-metadata", "codex-client-metadata"], source),
pinned_cwd,
)
return self._skip(reason)
return self._fallback_or_skip(
body,
headers,
pinned_cwd,
metadata_reason,
)
async def resolve_async(
self,
*,
headers: Mapping[str, str],
body: Mapping[str, Any],
pinned_cwd: Path | None = None,
project_root_override: str | None = None,
) -> CodexResolvedProject:
"""Resolve optional project context without blocking async model traffic."""
try:
return await asyncio.to_thread(
self.resolve,
headers=headers,
body=body,
pinned_cwd=pinned_cwd,
project_root_override=project_root_override,
)
except Exception:
logger.warning("event=codex_project_resolution_failed", exc_info=True)
return self._skip("resolver_failed")
@staticmethod
def _header(headers: Mapping[str, str], name: str) -> str | None:
lowered = name.lower()
for key, value in headers.items():
if key.lower() == lowered and value and value.strip():
return value.strip()
return None
def _turn_identity(
self,
headers: Mapping[str, str],
body: Mapping[str, Any],
) -> tuple[tuple[str, str | None] | None, str, str]:
raw_header = self._header(headers, _METADATA_HEADER)
if raw_header:
if len(raw_header.encode("utf-8")) > _MAX_METADATA_BYTES:
return None, "codex-turn-metadata", "metadata_too_large"
try:
metadata = json.loads(raw_header)
except (json.JSONDecodeError, TypeError, ValueError):
return None, "codex-turn-metadata", "metadata_invalid"
identity = self._identity_from_metadata(metadata)
if identity is not None:
return identity, "codex-turn-metadata", "resolved"
for container in self._body_containers(body):
metadata = container.get("client_metadata")
identity = self._identity_from_metadata(metadata)
if identity is not None:
if sum(len(value.encode("utf-8")) for value in identity if value) > (
_MAX_METADATA_BYTES
):
return None, "codex-client-metadata", "metadata_too_large"
return identity, "codex-client-metadata", "resolved"
return None, "unresolved", "metadata_missing"
@staticmethod
def _identity_from_metadata(metadata: Any) -> tuple[str, str | None] | None:
if not isinstance(metadata, Mapping):
return None
thread_id = metadata.get("thread_id")
turn_id = metadata.get("turn_id")
if not isinstance(thread_id, str) or not thread_id.strip():
return None
return (
thread_id.strip(),
turn_id.strip() if isinstance(turn_id, str) and turn_id.strip() else None,
)
@staticmethod
def _body_containers(body: Mapping[str, Any]) -> tuple[Mapping[str, Any], ...]:
response = body.get("response")
return (body, response) if isinstance(response, Mapping) else (body,)
def _fallback_or_skip(
self,
body: Mapping[str, Any],
headers: Mapping[str, str],
pinned_cwd: Path | None,
reason: str,
) -> CodexResolvedProject:
for container in self._body_containers(body):
metadata = container.get("client_metadata")
candidates = (metadata, container) if isinstance(metadata, Mapping) else (container,)
for candidate in candidates:
for key in ("cwd", "working_directory", "project_root"):
value = candidate.get(key)
if isinstance(value, str) and value.strip():
cwd = self._canonical_cwd(value)
if cwd is None:
return self._skip("body_cwd_invalid")
return self._cwd_result(
cwd,
headers,
"responses-body-cwd",
pinned_cwd,
)
return self._skip(reason)
def _explicit_cwd_result(
self,
raw_cwd: str,
*,
headers: Mapping[str, str],
source: Literal["x-headroom-cwd", "configured-project-root"],
pinned_cwd: Path | None,
) -> CodexResolvedProject:
cwd = self._canonical_cwd(raw_cwd)
if cwd is None:
# Preserve the existing ProjectResolver behavior for explicit
# operator overrides, including remote/non-local cwd strings.
identity = ProjectResolver().resolve(
RequestContext(
headers=headers,
system_prompt="",
base_user_id="",
project_root_override=raw_cwd if source == "configured-project-root" else None,
)
)
return CodexResolvedProject(
cwd=None,
project_key=identity[0] if identity else None,
source=source,
reason="explicit_override",
)
return self._cwd_result(cwd, headers, source, pinned_cwd)
def _cwd_result(
self,
cwd: Path,
headers: Mapping[str, str],
source: Literal[
"x-headroom-cwd",
"configured-project-root",
"codex-turn-metadata",
"codex-client-metadata",
"responses-body-cwd",
],
pinned_cwd: Path | None,
) -> CodexResolvedProject:
if pinned_cwd is not None and cwd != pinned_cwd:
return self._skip("project_mismatch")
identity = ProjectResolver().resolve(
RequestContext(
headers=headers,
system_prompt="",
base_user_id="",
project_root_override=str(cwd),
)
)
return CodexResolvedProject(
cwd=cwd,
project_key=identity[0] if identity else None,
source=source,
reason="resolved",
)
@staticmethod
def _canonical_cwd(raw_cwd: str) -> Path | None:
try:
path = Path(raw_cwd).expanduser()
if not path.is_absolute():
return None
resolved = path.resolve(strict=True)
return resolved if resolved.is_dir() else None
except (OSError, RuntimeError, ValueError):
return None
def _state_root(self) -> Path:
if self._configured_sqlite_home is not None:
return self._configured_sqlite_home
codex_home = Path(os.environ.get("CODEX_HOME", "").strip() or Path.home() / ".codex")
try:
config = tomllib.loads((codex_home / "config.toml").read_text(encoding="utf-8"))
configured_sqlite_home = config.get("sqlite_home")
if isinstance(configured_sqlite_home, str) and configured_sqlite_home.strip():
return Path(configured_sqlite_home).expanduser()
except (OSError, tomllib.TOMLDecodeError):
pass
env_sqlite_home = os.environ.get("CODEX_SQLITE_HOME", "").strip()
if env_sqlite_home:
return Path(env_sqlite_home).expanduser()
return codex_home
def _state_paths(self) -> list[Path]:
root = self._state_root()
if root.is_file():
return [root.resolve()]
return _codex_state_db_paths(root)
def _cwd_from_state(self, thread_id: str, turn_id: str) -> tuple[Path | None, str]:
state_paths = self._state_paths()
if not state_paths:
return None, "state_missing"
rollouts: set[Path] = set()
saw_supported_schema = False
saw_locked = False
for state_path in state_paths:
rollout, reason = self._rollout_for_thread(state_path, thread_id)
saw_supported_schema |= reason != "state_schema_unsupported"
saw_locked |= reason == "state_locked"
if rollout is not None:
rollouts.add(rollout)
if saw_locked:
return None, "state_locked"
if len(rollouts) > 1:
return None, "state_ambiguous"
if not rollouts:
return (
None,
"thread_missing" if saw_supported_schema else "state_schema_unsupported",
)
rollout = next(iter(rollouts))
if not rollout.is_file():
return None, "rollout_stale"
return self._cwd_from_rollout(rollout, turn_id)
@staticmethod
def _rollout_for_thread(state_path: Path, thread_id: str) -> tuple[Path | None, str]:
for attempt in range(_SQLITE_ATTEMPTS):
connection: sqlite3.Connection | None = None
try:
uri = f"{state_path.as_uri()}?mode=ro"
connection = sqlite3.connect(
uri,
uri=True,
timeout=_SQLITE_TIMEOUT_SECONDS,
)
columns = {
str(row[1])
for row in connection.execute("PRAGMA table_info(threads)").fetchall()
}
id_column = (
"id" if "id" in columns else "thread_id" if "thread_id" in columns else ""
)
if not id_column or "rollout_path" not in columns:
return None, "state_schema_unsupported"
rows = connection.execute(
f"SELECT rollout_path FROM threads WHERE {id_column} = ?", # noqa: S608
(thread_id,),
).fetchall()
paths = {Path(str(row[0])).expanduser().resolve() for row in rows if row[0]}
if len(paths) > 1:
return None, "state_ambiguous"
return (next(iter(paths)), "resolved") if paths else (None, "thread_missing")
except sqlite3.OperationalError as exc:
if "locked" not in str(exc).lower() and "busy" not in str(exc).lower():
return None, "state_schema_unsupported"
if attempt + 1 < _SQLITE_ATTEMPTS:
time.sleep(_SQLITE_TIMEOUT_SECONDS)
continue
return None, "state_locked"
except (OSError, sqlite3.Error, ValueError):
return None, "state_schema_unsupported"
finally:
if connection is not None:
connection.close()
return None, "state_locked"
def _cwd_from_rollout(self, rollout: Path, turn_id: str) -> tuple[Path | None, str]:
try:
metadata = rollout.stat()
except OSError:
return None, "rollout_stale"
fingerprint = (
metadata.st_dev,
metadata.st_ino,
metadata.st_size,
metadata.st_mtime_ns,
metadata.st_ctime_ns,
)
raw_cwds, reason = _cached_raw_cwds_from_rollout(rollout, fingerprint, turn_id)
if reason != "resolved":
return None, reason
matches = {cwd for raw_cwd in raw_cwds if (cwd := self._canonical_cwd(raw_cwd))}
if len(matches) > 1:
return None, "turn_ambiguous"
if not matches:
return None, "turn_context_missing"
return next(iter(matches)), "resolved"
@staticmethod
def _read_raw_cwds_from_rollout(rollout: Path, turn_id: str) -> tuple[tuple[str, ...], str]:
matches: set[str] = set()
truncated = False
try:
with rollout.open(encoding="utf-8") as handle:
for line in handle:
if not line.strip():
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
truncated = True
continue
if record.get("type") != "turn_context":
continue
payload = record.get("payload")
if not isinstance(payload, Mapping) or payload.get("turn_id") != turn_id:
continue
cwd = payload.get("cwd")
if isinstance(cwd, str):
matches.add(cwd)
except OSError:
return (), "rollout_stale"
if truncated:
return (), "rollout_truncated"
if matches:
return tuple(sorted(matches)), "resolved"
return (), "turn_context_missing"
@staticmethod
def _skip(reason: str) -> CodexResolvedProject:
logger.info("event=codex_project_resolution_skip reason=%s", reason)
return CodexResolvedProject(
cwd=None,
project_key=None,
source="unresolved",
reason=reason,
)
@lru_cache(maxsize=_ROLLOUT_CACHE_MAX_ENTRIES)
def _cached_raw_cwds_from_rollout(
rollout: Path,
_fingerprint: tuple[int, int, int, int, int],
turn_id: str,
) -> tuple[tuple[str, ...], str]:
return CodexProjectContextResolver._read_raw_cwds_from_rollout(rollout, turn_id)
__all__ = ["CodexProjectContextResolver", "CodexResolvedProject"]
+248 -48
View File
@@ -64,6 +64,7 @@ from headroom.copilot_auth import (
is_copilot_api_url,
)
from headroom.pipeline import PipelineStage, summarize_routing_markers
from headroom.providers.codex.project_context import CodexProjectContextResolver
from headroom.providers.codex.responses import (
codex_responses_http_url,
codex_responses_websocket_url,
@@ -6089,10 +6090,6 @@ class OpenAIHandlerMixin:
bind_scope(tags, request.scope)
client = classify_client(headers)
# Learn from the original client payload before memory context or
# compression mutates it. This mirrors the Anthropic ingestion path.
await self._observe_openai_responses_traffic(body, request_id=request_id)
# PR-A5 (P5-49): strip internal x-headroom-* from upstream-bound
# headers AFTER `_extract_tags` reads them. Memory user-id reads
# `request.headers` below.
@@ -6127,7 +6124,11 @@ class OpenAIHandlerMixin:
headers = {
key: value
for key, value in headers.items()
if key.lower() != _CODEX_RESPONSES_LITE_HEADER
if key.lower()
not in {
_CODEX_RESPONSES_LITE_HEADER,
"x-codex-turn-metadata",
}
}
log_outbound_headers(
forwarder="openai_responses",
@@ -6137,6 +6138,56 @@ class OpenAIHandlerMixin:
headers, is_chatgpt_auth = _resolve_codex_routing_headers(headers)
if is_chatgpt_auth:
client = "codex"
codex_project = None
codex_project_root_override = (
getattr(getattr(self.memory_handler, "config", None), "project_root_override", "")
or getattr(self.config, "memory_project_root_override", "")
or None
)
if client == "codex":
codex_project = await CodexProjectContextResolver().resolve_async(
headers=dict(request.headers),
body=body,
project_root_override=codex_project_root_override,
)
tags["codex_project_context"] = codex_project.reason
if codex_project.project_key and classify_project(request.headers) is None:
set_current_project(codex_project.project_key)
codex_project_scope_required = client == "codex" and (
is_chatgpt_auth
or bool(request.headers.get("x-codex-turn-metadata"))
or "codex" in str(request.headers.get("user-agent") or "").lower()
or any(
isinstance(container, dict)
and isinstance(container.get("client_metadata"), dict)
and bool(container["client_metadata"].get("thread_id"))
for container in (
body,
body.get("response") if isinstance(body.get("response"), dict) else {},
)
)
)
codex_project_features_allowed = not codex_project_scope_required or bool(
codex_project
and (
codex_project.cwd is not None
or codex_project.source
in {
"x-headroom-project-id",
"x-headroom-cwd",
"configured-project-root",
}
)
)
resolved_project_root_override = codex_project_root_override or (
str(codex_project.cwd) if codex_project and codex_project.cwd else None
)
# The shared learner cannot isolate project state. Scoped Codex turns
# skip learning until a project-scoped learner is available.
if not codex_project_scope_required and (
codex_project is None or codex_project.reason == "metadata_missing"
):
await self._observe_openai_responses_traffic(body, request_id=request_id)
memory_client = (
"codex"
if is_chatgpt_auth
@@ -6192,7 +6243,7 @@ class OpenAIHandlerMixin:
# directly because `headers` was stripped of `x-headroom-*` (PR-A5).
memory_user_id: str | None = None
memory_request_ctx = None
if self.memory_handler:
if self.memory_handler and codex_project_features_allowed:
memory_user_id = resolve_memory_identity(request)
from headroom.memory.storage_router import (
RequestContext as _MemRequestContext,
@@ -6205,9 +6256,7 @@ class OpenAIHandlerMixin:
headers=dict(request.headers),
system_prompt=_extract_sys_prompt(body),
base_user_id=memory_user_id,
project_root_override=(
getattr(self.memory_handler.config, "project_root_override", "") or None
),
project_root_override=resolved_project_root_override,
)
# Rate limiting
@@ -6276,11 +6325,13 @@ class OpenAIHandlerMixin:
responses_memory_decision = MemoryDecision.decide(
headers=request.headers,
memory_handler=self.memory_handler,
memory_handler=self.memory_handler if codex_project_features_allowed else None,
memory_user_id=memory_user_id,
mode_name=get_memory_injection_mode(),
)
responses_memory_decision.apply_to_tags(tags)
if not codex_project_features_allowed:
tags["memory_skip_reason"] = "project_unresolved"
if responses_memory_decision.inject:
try:
# Memory context now routes exclusively to the live-zone tail
@@ -6677,7 +6728,9 @@ class OpenAIHandlerMixin:
)
buffered_stream_ccr = _should_buffer_openai_responses_stream_ccr(
stream=stream,
ccr_response_handler_enabled=_ccr_response_handler_enabled,
ccr_response_handler_enabled=(
_ccr_response_handler_enabled and codex_project_features_allowed
),
tools=body.get("tools"),
is_chatgpt_auth=is_chatgpt_auth,
upstream_base_url=upstream_base_url,
@@ -7502,7 +7555,11 @@ class OpenAIHandlerMixin:
upstream_headers = {
key: value
for key, value in upstream_headers.items()
if key.lower() != _CODEX_RESPONSES_LITE_HEADER
if key.lower()
not in {
_CODEX_RESPONSES_LITE_HEADER,
"x-codex-turn-metadata",
}
}
ws_memory_tools_allowed = _allow_responses_memory_tools(is_chatgpt_auth)
_lower_headers = {k.lower(): v for k, v in upstream_headers.items()}
@@ -7950,22 +8007,6 @@ class OpenAIHandlerMixin:
# the full history; reconnect replays it wholesale) is not counted
# as new evidence. Seeded from the first frame as a baseline.
ws_learner_seen_call_ids: set[str] = set()
# Baseline the first frame's transcript into the learner: record its
# tool-call ids as seen WITHOUT learning, so a reconnect that replays
# this history adds no spurious evidence. Later frames learn only the
# results appended after this point. `body` here is the original
# client frame (parsed before memory injection / compression).
if isinstance(body, dict) and body:
_ws_first_inner = (
body["response"] if isinstance(body.get("response"), dict) else body
)
if isinstance(_ws_first_inner, dict):
await self._observe_openai_ws_response_create(
_ws_first_inner,
seen_call_ids=ws_learner_seen_call_ids,
baseline=True,
request_id=request_id,
)
ws_client_frames_total = 1
ws_upstream_frames_total = 0
ws_cancel_frames = 0
@@ -8079,23 +8120,142 @@ class OpenAIHandlerMixin:
memory_user_id: str | None = None
memory_request_ctx = None
ws_project_resolver = CodexProjectContextResolver()
ws_pinned_project_cwd = None
ws_turn_project_key = classify_project(ws_headers)
ws_project_scope_required = client == "codex" and (
is_chatgpt_auth
or "codex" in str(_header_get(ws_headers, "user-agent") or "").lower()
)
ws_turn_project_features_allowed = not ws_project_scope_required
ws_turn_learning_allowed = not ws_project_scope_required
ws_first_project_resolution = True
ws_memory_generation = 0
ws_memory_response_pending = False
ws_memory_overlap = False
ws_project_root_override = (
getattr(
getattr(self.memory_handler, "config", None),
"project_root_override",
"",
)
or getattr(self.config, "memory_project_root_override", "")
or None
)
from headroom.proxy.helpers import get_memory_injection_mode, log_memory_injection
from headroom.proxy.memory_decision import MemoryDecision
from headroom.proxy.memory_query import MemoryQuery
async def _prepare_memory_frame(frame_body: dict[str, Any], frame_raw: str) -> str:
nonlocal memory_user_id, memory_request_ctx
nonlocal ws_pinned_project_cwd, ws_turn_project_key
nonlocal ws_turn_project_features_allowed
nonlocal ws_turn_learning_allowed, ws_first_project_resolution
nonlocal ws_project_scope_required
nonlocal ws_memory_generation
nonlocal ws_memory_response_pending, ws_memory_overlap
# Without turn IDs on response events, overlapping creates are
# ambiguous. Keep forwarding, but disable tool interception.
ws_memory_overlap = ws_memory_overlap or ws_memory_response_pending
ws_memory_response_pending = True
ws_memory_generation += 1
memory_user_id = None
memory_request_ctx = None
first_project_resolution = ws_first_project_resolution
ws_first_project_resolution = False
resolved_project_root_override = ws_project_root_override
if client == "codex":
# Handshake turn metadata describes only the first frame.
# Frame metadata takes precedence, including invalid metadata.
containers = ws_project_resolver._body_containers(frame_body)
frame_has_project_metadata = any(
any(
key in container for key in ("cwd", "working_directory", "project_root")
)
or (
"client_metadata" in container
and (
not isinstance(container["client_metadata"], dict)
or any(
key in container["client_metadata"]
for key in (
"thread_id",
"turn_id",
"cwd",
"working_directory",
"project_root",
)
)
)
)
for container in containers
)
project_headers = {
key: value
for key, value in ws_headers.items()
if key.lower() != "x-codex-turn-metadata"
or (first_project_resolution and not frame_has_project_metadata)
}
resolved_project = await ws_project_resolver.resolve_async(
headers=project_headers,
body=frame_body,
pinned_cwd=ws_pinned_project_cwd,
project_root_override=ws_project_root_override,
)
ws_tags["codex_project_context"] = resolved_project.reason
frame_project_scope_required = (
ws_project_scope_required or resolved_project.reason != "metadata_missing"
)
ws_project_scope_required = frame_project_scope_required
# The shared TrafficLearner has no per-project state or routing.
# Skip scoped turns until a project-scoped learner is available.
ws_turn_learning_allowed = not frame_project_scope_required
ws_turn_project_features_allowed = (
not frame_project_scope_required
or resolved_project.cwd is not None
or resolved_project.source
in {
"x-headroom-project-id",
"x-headroom-cwd",
"configured-project-root",
}
)
if resolved_project.cwd is not None:
if ws_pinned_project_cwd is None:
ws_pinned_project_cwd = resolved_project.cwd
resolved_project_root_override = str(resolved_project.cwd)
ws_turn_project_key = classify_project(ws_headers) or (
resolved_project.project_key if ws_turn_project_features_allowed else None
)
if first_project_resolution and ws_turn_learning_allowed:
inner_payload = frame_body.get("response", frame_body)
if isinstance(inner_payload, dict):
await self._observe_openai_ws_response_create(
inner_payload,
seen_call_ids=ws_learner_seen_call_ids,
baseline=True,
request_id=request_id,
)
memory_user_id_candidate = (
resolve_memory_identity(websocket) if self.memory_handler else None
resolve_memory_identity(websocket)
if self.memory_handler and ws_turn_project_features_allowed
else None
)
memory_decision = MemoryDecision.decide(
headers=ws_headers,
memory_handler=self.memory_handler,
memory_handler=(
self.memory_handler if ws_turn_project_features_allowed else None
),
memory_user_id=memory_user_id_candidate,
mode_name=get_memory_injection_mode(),
)
memory_decision.apply_to_tags(ws_tags)
if not ws_turn_project_features_allowed:
ws_tags["memory_skip_reason"] = "project_unresolved"
if not memory_decision.inject:
return frame_raw
@@ -8117,9 +8277,7 @@ class OpenAIHandlerMixin:
headers=dict(ws_headers),
system_prompt=str(ws_response_body.get("instructions") or ""),
base_user_id=memory_user_id,
project_root_override=(
getattr(self.memory_handler.config, "project_root_override", "") or None
),
project_root_override=resolved_project_root_override,
)
# Debug: log what Codex sends so we can see the full tool list
@@ -8693,12 +8851,13 @@ class OpenAIHandlerMixin:
# Learn from this turn's newly appended tool results.
# Dedup against the per-connection baseline so the
# replayed transcript prefix is not re-counted.
await self._observe_openai_ws_response_create(
inner_payload,
seen_call_ids=ws_learner_seen_call_ids,
baseline=False,
request_id=request_id,
)
if ws_turn_learning_allowed:
await self._observe_openai_ws_response_create(
inner_payload,
seen_call_ids=ws_learner_seen_call_ids,
baseline=False,
request_id=request_id,
)
store_forced = _ensure_chatgpt_responses_store_false(
inner_payload,
is_chatgpt_auth=is_chatgpt_auth,
@@ -9158,6 +9317,7 @@ class OpenAIHandlerMixin:
# over ``upstream_disconnect``.
nonlocal response_completed_seen
nonlocal upstream_relay_error
nonlocal ws_memory_response_pending
nonlocal ws_input_tokens_total, ws_output_tokens_total
nonlocal ws_cache_read_tokens_total, ws_cache_write_tokens_total
nonlocal ws_uncached_input_tokens_total
@@ -9178,6 +9338,11 @@ class OpenAIHandlerMixin:
suppress_response = False
pending_fcs: list[dict[str, Any]] = []
response_output_items: list[dict[str, Any]] = []
response_memory_generation = ws_memory_generation
response_memory_user_id = memory_user_id
response_memory_request_ctx = memory_request_ctx
response_input = list(current_response_input)
response_template = dict(current_response_template)
def _reset() -> None:
nonlocal decided, suppress_response
@@ -9297,6 +9462,7 @@ class OpenAIHandlerMixin:
conversation_key=ws_conversation_key,
conversation_tokens_saved=ws_conversation_tokens_saved,
client=client,
project=ws_turn_project_key,
)
)
@@ -9409,6 +9575,17 @@ class OpenAIHandlerMixin:
if event_type == "response.created":
response_started_ms = time.perf_counter() * 1000.0
response_memory_generation = ws_memory_generation
response_memory_user_id = memory_user_id
response_memory_request_ctx = memory_request_ctx
response_input = list(current_response_input)
response_template = dict(current_response_template)
if event_type in {
"response.completed",
"response.failed",
"response.incomplete",
}:
ws_memory_response_pending = False
(
usage_input_tokens,
usage_output_tokens,
@@ -9423,12 +9600,26 @@ class OpenAIHandlerMixin:
ws_cache_write_tokens_total += usage_cache_write_tokens
ws_uncached_input_tokens_total += usage_uncached_tokens
if event_type in {"response.failed", "response.incomplete"}:
for buffered_event in event_buffer:
await websocket.send_text(buffered_event)
_reset()
await websocket.send_text(msg_str)
continue
memory_enabled = bool(
self.memory_handler
and memory_user_id
and response_memory_user_id
and response_memory_request_ctx
and response_memory_generation == ws_memory_generation
and not ws_memory_overlap
and ws_turn_project_features_allowed
and ws_memory_tools_allowed
)
if not memory_enabled:
for buffered_event in event_buffer:
await websocket.send_text(buffered_event)
_reset()
if event_type == "response.completed":
response_completed_seen = True
await _record_ws_response_metrics()
@@ -9445,8 +9636,9 @@ class OpenAIHandlerMixin:
):
pending_fcs.append(item)
if not decided:
if not decided or suppress_response:
event_buffer.append(msg_str)
if not decided:
if event_type == "response.output_item.added":
item = event.get("item", {})
if (
@@ -9456,7 +9648,6 @@ class OpenAIHandlerMixin:
):
suppress_response = True
decided = True
event_buffer.clear()
logger.info(
f"[{request_id}] WS Memory: Detected "
f"{item.get('name')} — suppressing response"
@@ -9512,12 +9703,15 @@ class OpenAIHandlerMixin:
fc_args = {}
await self.memory_handler._ensure_initialized()
if response_memory_generation != ws_memory_generation:
break
if self.memory_handler._backend:
result = await self.memory_handler._execute_memory_tool(
fc_name,
fc_args,
memory_user_id,
response_memory_user_id,
"openai",
request_context=response_memory_request_ctx,
)
else:
result = json.dumps({"error": "backend not ready"})
@@ -9531,12 +9725,16 @@ class OpenAIHandlerMixin:
)
logger.info(
f"[{request_id}] WS Memory: Executed "
f"{fc_name} for user {memory_user_id}"
f"{fc_name} for user {response_memory_user_id}"
)
stateless_input = _responses_stateless_input_items(
current_response_input
)
if response_memory_generation != ws_memory_generation:
for buffered_event in event_buffer:
await websocket.send_text(buffered_event)
_reset()
continue
stateless_input = _responses_stateless_input_items(response_input)
stateless_output = _responses_stateless_output_items(
response_output_items
)
@@ -9546,7 +9744,7 @@ class OpenAIHandlerMixin:
*tool_outputs,
]
continuation_response = {
**current_response_template,
**response_template,
"input": current_response_input,
}
continuation_response.pop("previous_response_id", None)
@@ -9555,6 +9753,7 @@ class OpenAIHandlerMixin:
"response": continuation_response,
}
continuation_raw = _strip_codex_lite_metadata(json.dumps(cont))
ws_memory_response_pending = True
await upstream.send(
_normalize_ws_response_create_for_upstream(continuation_raw)
)
@@ -9926,6 +10125,7 @@ class OpenAIHandlerMixin:
conversation_key=ws_conversation_key,
conversation_tokens_saved=ws_conversation_tokens_saved,
client=client,
project=ws_turn_project_key,
request_messages=ws_messages_for_log
if getattr(self.config, "log_full_messages", False)
else None,
+939
View File
@@ -0,0 +1,939 @@
from __future__ import annotations
import asyncio
import json
import sqlite3
import sys
import threading
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
import anyio
import pytest
from headroom.memory.storage_router import ProjectResolver, RequestContext
from headroom.providers.codex.project_context import CodexProjectContextResolver
from tests.test_openai_codex_routing import (
_build_request,
_DummyTokenizer,
_ResponseStub,
)
from tests.test_openai_codex_routing import (
_DummyOpenAIHandler as _HTTPHandler,
)
from tests.test_openai_codex_ws_lifecycle import (
_DummyOpenAIHandler as _WSHandler,
)
from tests.test_openai_codex_ws_lifecycle import (
_FakeUpstream,
_FakeWebSocket,
_make_fake_websockets_module,
_MemoryWsHandler,
)
def _seed_thread(codex_home: Path, thread_id: str, rollout: Path) -> None:
db = codex_home / "state_5.sqlite"
with sqlite3.connect(db) as connection:
connection.execute(
"CREATE TABLE IF NOT EXISTS threads (id TEXT PRIMARY KEY, rollout_path TEXT NOT NULL)"
)
connection.execute(
"INSERT INTO threads (id, rollout_path) VALUES (?, ?)",
(thread_id, str(rollout)),
)
def _seed_rollout(path: Path, turn_id: str, cwd: Path) -> None:
path.write_text(
json.dumps(
{
"type": "turn_context",
"payload": {"turn_id": turn_id, "cwd": str(cwd)},
}
)
+ "\n",
encoding="utf-8",
)
def _append_turn(path: Path, turn_id: str, cwd: Path) -> None:
with path.open("a", encoding="utf-8") as handle:
handle.write(
json.dumps(
{
"type": "turn_context",
"payload": {"turn_id": turn_id, "cwd": str(cwd)},
}
)
+ "\n"
)
def test_codex_http_projects_are_isolated_and_ws_mismatch_fails_closed(
monkeypatch, tmp_path: Path
) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
project_a = tmp_path / "a" / "shared"
project_b = tmp_path / "b" / "shared"
project_a.mkdir(parents=True)
project_b.mkdir(parents=True)
rollout_a = codex_home / "rollout-a.jsonl"
rollout_b = codex_home / "rollout-b.jsonl"
_seed_rollout(rollout_a, "turn-a", project_a)
_seed_rollout(rollout_b, "turn-b", project_b)
_seed_thread(codex_home, "thread-a", rollout_a)
_seed_thread(codex_home, "thread-b", rollout_b)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
resolver = CodexProjectContextResolver()
resolved_a = resolver.resolve(
headers={
"x-codex-turn-metadata": json.dumps({"thread_id": "thread-a", "turn_id": "turn-a"})
},
body={},
)
resolved_b = resolver.resolve(
headers={},
body={"client_metadata": {"thread_id": "thread-b", "turn_id": "turn-b"}},
)
assert resolved_a.cwd == project_a.resolve()
assert resolved_b.cwd == project_b.resolve()
keys = {
ProjectResolver().resolve(
RequestContext(
headers={},
system_prompt="",
base_user_id="test",
project_root_override=str(resolved.cwd),
)
)[0]
for resolved in (resolved_a, resolved_b)
}
assert len(keys) == 2
mismatch = resolver.resolve(
headers={},
body={
"type": "response.create",
"response": {
"client_metadata": {
"thread_id": "thread-b",
"turn_id": "turn-b",
}
},
},
pinned_cwd=resolved_a.cwd,
)
assert mismatch.cwd is None
assert mismatch.reason == "project_mismatch"
_append_turn(rollout_a, "turn-a", project_b)
changed_rollout = resolver.resolve(
headers={},
body={"client_metadata": {"thread_id": "thread-a", "turn_id": "turn-a"}},
)
assert changed_rollout.cwd is None
assert changed_rollout.reason == "turn_ambiguous"
def test_cached_rollout_recanonicalizes_symlink_before_pinned_check(
monkeypatch, tmp_path: Path
) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
project_a = tmp_path / "a"
project_b = tmp_path / "b"
project_a.mkdir()
project_b.mkdir()
project_link = tmp_path / "current-project"
project_link.symlink_to(project_a, target_is_directory=True)
rollout = codex_home / "rollout.jsonl"
_seed_rollout(rollout, "turn", project_link)
_seed_thread(codex_home, "thread", rollout)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
resolver = CodexProjectContextResolver()
body = {"client_metadata": {"thread_id": "thread", "turn_id": "turn"}}
first = resolver.resolve(headers={}, body=body)
project_link.unlink()
project_link.symlink_to(project_b, target_is_directory=True)
second = resolver.resolve(headers={}, body=body, pinned_cwd=first.cwd)
assert first.cwd == project_a.resolve()
assert second.cwd is None
assert second.reason == "project_mismatch"
def test_turn_specific_resume_and_fork_use_exact_context(monkeypatch, tmp_path: Path) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
project_a = tmp_path / "a"
project_b = tmp_path / "b"
project_a.mkdir()
project_b.mkdir()
rollout = codex_home / "rollout.jsonl"
_seed_rollout(rollout, "turn-a", project_a)
_append_turn(rollout, "turn-b", project_b)
_seed_thread(codex_home, "resumed-thread", rollout)
_seed_thread(codex_home, "forked-thread", rollout)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
resolver = CodexProjectContextResolver()
resumed = resolver.resolve(
headers={},
body={"client_metadata": {"thread_id": "resumed-thread", "turn_id": "turn-b"}},
)
forked = resolver.resolve(
headers={},
body={"client_metadata": {"thread_id": "forked-thread", "turn_id": "turn-a"}},
)
assert resumed.cwd == project_b.resolve()
assert forked.cwd == project_a.resolve()
@pytest.mark.parametrize(
("setup", "reason"),
[
("missing_thread", "thread_missing"),
("missing_context", "turn_context_missing"),
("truncated", "rollout_truncated"),
("unsupported", "state_schema_unsupported"),
("stale", "rollout_stale"),
],
)
def test_state_failures_skip_with_structured_reason(
monkeypatch,
tmp_path: Path,
setup: str,
reason: str,
) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
rollout = codex_home / "rollout.jsonl"
project = tmp_path / "project"
project.mkdir()
if setup == "unsupported":
with sqlite3.connect(codex_home / "state_5.sqlite") as connection:
connection.execute("CREATE TABLE other (id TEXT)")
else:
_seed_rollout(rollout, "other-turn", project)
if setup == "truncated":
with rollout.open("a", encoding="utf-8") as handle:
handle.write('{"type":"turn_context"')
if setup == "stale":
rollout.unlink()
_seed_thread(codex_home, "other-thread" if setup == "missing_thread" else "thread", rollout)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
result = CodexProjectContextResolver().resolve(
headers={},
body={"client_metadata": {"thread_id": "thread", "turn_id": "turn"}},
)
assert result.cwd is None
assert result.reason == reason
def test_truncated_tail_after_exact_context_skips_safely(monkeypatch, tmp_path: Path) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
project = tmp_path / "project"
project.mkdir()
rollout = codex_home / "rollout.jsonl"
_seed_rollout(rollout, "turn", project)
with rollout.open("a", encoding="utf-8") as handle:
handle.write('{"type":"turn_context"')
_seed_thread(codex_home, "thread", rollout)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
result = CodexProjectContextResolver().resolve(
headers={},
body={"client_metadata": {"thread_id": "thread", "turn_id": "turn"}},
)
assert result.cwd is None
assert result.reason == "rollout_truncated"
def test_conflicting_state_rows_are_ambiguous_but_identical_rows_dedupe(
monkeypatch, tmp_path: Path
) -> None:
codex_home = tmp_path / "codex"
sqlite_home = codex_home / "sqlite"
sqlite_home.mkdir(parents=True)
project = tmp_path / "project"
other = tmp_path / "other"
project.mkdir()
other.mkdir()
rollout = codex_home / "rollout.jsonl"
conflicting_rollout = codex_home / "conflicting.jsonl"
_seed_rollout(rollout, "turn", project)
_seed_rollout(conflicting_rollout, "turn", other)
_seed_thread(sqlite_home, "thread", rollout)
_seed_thread(codex_home, "thread", rollout)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
resolver = CodexProjectContextResolver()
body = {"client_metadata": {"thread_id": "thread", "turn_id": "turn"}}
assert resolver.resolve(headers={}, body=body).cwd == project.resolve()
lock = sqlite3.connect(sqlite_home / "state_5.sqlite")
lock.execute("BEGIN EXCLUSIVE")
try:
locked = resolver.resolve(headers={}, body=body)
finally:
lock.rollback()
lock.close()
assert locked.cwd is None
assert locked.reason == "state_locked"
with sqlite3.connect(sqlite_home / "state_5.sqlite") as connection:
connection.execute(
"UPDATE threads SET rollout_path = ? WHERE id = ?",
(str(conflicting_rollout), "thread"),
)
ambiguous = resolver.resolve(headers={}, body=body)
assert ambiguous.cwd is None
assert ambiguous.reason == "state_ambiguous"
def test_explicit_overrides_precede_state_and_body_cwd(monkeypatch, tmp_path: Path) -> None:
project_a = tmp_path / "a"
project_b = tmp_path / "b"
project_a.mkdir()
project_b.mkdir()
monkeypatch.setenv("CODEX_HOME", str(tmp_path / "missing"))
resolver = CodexProjectContextResolver()
body = {"client_metadata": {"cwd": str(project_b)}}
project_id = resolver.resolve(
headers={"x-headroom-project-id": "chosen"},
body=body,
)
explicit_cwd = resolver.resolve(
headers={"x-headroom-cwd": str(project_a)},
body=body,
project_root_override=str(project_b),
)
assert project_id.source == "x-headroom-project-id"
assert project_id.project_key.startswith("chosen-")
assert explicit_cwd.cwd == project_a.resolve()
assert explicit_cwd.source == "x-headroom-cwd"
def test_configured_sqlite_home_precedes_environment(monkeypatch, tmp_path: Path) -> None:
codex_home = tmp_path / "codex"
configured = tmp_path / "configured"
environment = tmp_path / "environment"
codex_home.mkdir()
configured.mkdir()
environment.mkdir()
project_a = tmp_path / "a"
project_b = tmp_path / "b"
project_a.mkdir()
project_b.mkdir()
rollout_a = configured / "rollout.jsonl"
rollout_b = environment / "rollout.jsonl"
_seed_rollout(rollout_a, "turn", project_a)
_seed_rollout(rollout_b, "turn", project_b)
_seed_thread(configured, "thread", rollout_a)
_seed_thread(environment, "thread", rollout_b)
(codex_home / "config.toml").write_text(
f"sqlite_home = {json.dumps(str(configured))}\n",
encoding="utf-8",
)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
monkeypatch.setenv("CODEX_SQLITE_HOME", str(environment))
result = CodexProjectContextResolver().resolve(
headers={},
body={"client_metadata": {"thread_id": "thread", "turn_id": "turn"}},
)
assert result.cwd == project_a.resolve()
def test_body_cwd_is_bounded_fallback_and_must_be_canonical(tmp_path: Path) -> None:
project = tmp_path / "project"
project.mkdir()
resolver = CodexProjectContextResolver(tmp_path / "missing")
result = resolver.resolve(
headers={},
body={"response": {"client_metadata": {"cwd": str(project / ".." / "project")}}},
)
invalid = resolver.resolve(headers={}, body={"cwd": "relative/project"})
assert result.cwd == project.resolve()
assert result.source == "responses-body-cwd"
assert invalid.cwd is None
assert invalid.reason == "body_cwd_invalid"
def test_sqlite_lock_is_bounded_and_visible(tmp_path: Path) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
rollout = codex_home / "rollout.jsonl"
project = tmp_path / "project"
project.mkdir()
_seed_rollout(rollout, "turn", project)
_seed_thread(codex_home, "thread", rollout)
lock = sqlite3.connect(codex_home / "state_5.sqlite")
lock.execute("BEGIN EXCLUSIVE")
try:
result = CodexProjectContextResolver(codex_home).resolve(
headers={},
body={"client_metadata": {"thread_id": "thread", "turn_id": "turn"}},
)
finally:
lock.rollback()
lock.close()
assert result.cwd is None
assert result.reason == "state_locked"
def test_http_resolution_does_not_mutate_body_or_forward_internal_metadata(
monkeypatch, tmp_path: Path
) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
project_a = tmp_path / "a" / "shared"
project_b = tmp_path / "b" / "shared"
project_a.mkdir(parents=True)
project_b.mkdir(parents=True)
rollout_a = codex_home / "rollout-a.jsonl"
rollout_b = codex_home / "rollout-b.jsonl"
_seed_rollout(rollout_a, "turn-a", project_a)
_seed_rollout(rollout_b, "turn-b", project_b)
_seed_thread(codex_home, "thread-a", rollout_a)
_seed_thread(codex_home, "thread-b", rollout_b)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
body = {
"model": "gpt-5.4",
"input": "hello",
"client_metadata": {"thread_id": "thread-a", "turn_id": "turn-a"},
}
turn_header = json.dumps({"thread_id": "thread-a", "turn_id": "turn-a"})
request = _build_request(
body,
{
"Authorization": "Bearer test",
"X-Client": "codex",
"X-Codex-Turn-Metadata": turn_header,
},
)
class Handler(_HTTPHandler):
def __init__(self) -> None:
super().__init__()
self.config.memory_storage_mode = "project"
self.retry_kwargs = {}
self.observed = []
self.projects = []
async def _retry_request(self, method, url, headers, request_body, **kwargs):
self.captured_request = (method, url, headers, request_body)
self.retry_kwargs = kwargs
return _ResponseStub()
async def _observe_openai_responses_traffic(self, request_body, *, request_id):
self.observed.append(request_body)
async def _record_request_outcome(self, _outcome):
from headroom.proxy.project_context import get_current_project
self.projects.append(get_current_project())
handler = Handler()
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _DummyTokenizer())
response = anyio.run(handler.handle_openai_responses, request)
assert response.status_code == 200
assert handler.captured_request is not None
_, _, upstream_headers, upstream_body = handler.captured_request
assert upstream_body == body
assert handler.retry_kwargs["original_body_bytes"] == json.dumps(body).encode()
assert all(key.lower() != "x-codex-turn-metadata" for key in upstream_headers)
assert all(not key.lower().startswith("x-headroom-") for key in upstream_headers)
assert handler.observed == []
body_b = {
"model": "gpt-5.4",
"input": "hello B",
"client_metadata": {"thread_id": "thread-b", "turn_id": "turn-b"},
}
handler_b = Handler()
response_b = anyio.run(
handler_b.handle_openai_responses,
_build_request(
body_b,
{
"Authorization": "Bearer test",
"X-Client": "codex",
"X-Codex-Turn-Metadata": json.dumps({"thread_id": "thread-b", "turn_id": "turn-b"}),
},
),
)
assert response_b.status_code == 200
assert handler_b.captured_request is not None
assert handler_b.captured_request[3] == body_b
assert handler_b.observed == []
assert handler.projects and handler_b.projects
assert handler.projects[0] != handler_b.projects[0]
unresolved_handler = Handler()
unresolved_request = _build_request(
{"model": "gpt-5.4", "input": "unresolved"},
{
"Authorization": "Bearer test",
"User-Agent": "codex_cli_rs",
"X-Client": "codex",
},
)
assert (
anyio.run(unresolved_handler.handle_openai_responses, unresolved_request).status_code == 200
)
assert unresolved_handler.observed == []
@pytest.mark.asyncio
async def test_http_project_resolution_does_not_block_event_loop(monkeypatch) -> None:
resolver_entered = threading.Event()
resolver_release = threading.Event()
def blocked_resolve(self, **kwargs):
resolver_entered.set()
assert resolver_release.wait(timeout=3)
raise RuntimeError("blocked resolver failed")
monkeypatch.setattr(CodexProjectContextResolver, "resolve", blocked_resolve)
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda model: _DummyTokenizer())
request = _build_request(
{"model": "gpt-5.4", "input": "hello"},
{"Authorization": "Bearer test", "X-Client": "codex"},
)
handler = _HTTPHandler()
progressed_before_release = False
async def unrelated_work() -> None:
nonlocal progressed_before_release
while not resolver_entered.is_set():
await asyncio.sleep(0)
progressed_before_release = not resolver_release.is_set()
resolver_release.set()
release_timer = threading.Timer(2, resolver_release.set)
release_timer.start()
try:
response, _ = await asyncio.gather(
handler.handle_openai_responses(request),
unrelated_work(),
)
finally:
resolver_release.set()
release_timer.cancel()
release_timer.join()
assert resolver_entered.is_set()
assert progressed_before_release
assert response.status_code == 200
@pytest.mark.asyncio
@pytest.mark.parametrize("handshake_metadata", [False, True, "transport-only", "conflicting"])
async def test_ws_mismatch_skips_project_memory_but_forwards_main_traffic(
monkeypatch, tmp_path: Path, handshake_metadata: bool | str
) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
project_a = tmp_path / "a"
project_b = tmp_path / "b"
project_a.mkdir()
project_b.mkdir()
rollout_a = codex_home / "rollout-a.jsonl"
rollout_b = codex_home / "rollout-b.jsonl"
_seed_rollout(rollout_a, "turn-a", project_a)
_seed_rollout(rollout_b, "turn-b", project_b)
_seed_thread(codex_home, "thread-a", rollout_a)
_seed_thread(codex_home, "thread-b", rollout_b)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
first = json.dumps(
{
"type": "response.create",
"response": {
"input": "first",
"client_metadata": {"thread_id": "thread-a", "turn_id": "turn-a"},
},
}
)
mismatch = json.dumps(
{
"type": "response.create",
"response": {
"input": "mismatch",
"client_metadata": {"thread_id": "thread-b", "turn_id": "turn-b"},
},
}
)
class Memory:
def __init__(self) -> None:
self.config = SimpleNamespace(
inject_context=True,
inject_tools=False,
project_root_override="",
)
self.projects = []
async def search_and_format_context(self, _user_id, _messages, **kwargs):
self.projects.append(kwargs["request_context"].project_root_override)
return None
def compute_memory_tool_definitions(self, _provider):
return []
upstream = _FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
],
hold_after_events=True,
)
if handshake_metadata == "transport-only":
first_body = json.loads(first)
first_body["response"]["client_metadata"] = {"keep": "yes"}
first = json.dumps(first_body)
websocket = _FakeWebSocket(frames=[first, mismatch])
if handshake_metadata:
header_project = "b" if handshake_metadata == "conflicting" else "a"
websocket.headers["X-Codex-Turn-Metadata"] = json.dumps(
{"thread_id": f"thread-{header_project}", "turn_id": f"turn-{header_project}"}
)
handler = _WSHandler()
handler.config.memory_storage_mode = "project"
memory = Memory()
handler.memory_handler = memory
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await handler.handle_openai_responses_ws(websocket)
assert memory.projects == [str(project_a.resolve())]
assert len(upstream.sent) == 2
assert json.loads(upstream.sent[1]) == json.loads(mismatch)
@pytest.mark.asyncio
@pytest.mark.parametrize(
"tool_name", ["memory_search", "memory_save", "memory_update", "memory_delete"]
)
@pytest.mark.parametrize(
"second_project",
["same", "mismatch", "unresolved", "missing", "overlap", "pipeline", "failed", "incomplete"],
)
@pytest.mark.parametrize("disable_injection", [False, True])
@pytest.mark.parametrize("user_agent", ["codex-cli/0.5", "python-client"])
async def test_ws_memory_dispatch_uses_current_turn_project(
monkeypatch,
tmp_path: Path,
tool_name: str,
second_project: str,
disable_injection: bool,
user_agent: str,
) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
for name in ("a", "b"):
project = tmp_path / name
project.mkdir()
rollout = codex_home / f"rollout-{name}.jsonl"
_seed_rollout(rollout, f"turn-{name}", project)
_seed_thread(codex_home, f"thread-{name}", rollout)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
def frame(project: str | None) -> str:
response = {"model": "gpt-5.4", "input": "hello"}
if project:
response["client_metadata"] = {
"thread_id": f"thread-{project}",
"turn_id": f"turn-{project}",
}
return json.dumps({"type": "response.create", "response": response})
first_completed = asyncio.Event()
init_blocked = asyncio.Event()
init_release = asyncio.Event()
events: asyncio.Queue[str] = asyncio.Queue()
function_call = {
"type": "function_call",
"id": "fc-1",
"call_id": "call-1",
"name": tool_name,
"arguments": "{}",
}
class Upstream(_FakeUpstream):
async def send(self, payload: str) -> None:
await super().send(payload)
if second_project == "pipeline" and len(self.sent) == 2:
return
if len(self.sent) == 3:
init_release.set()
response_id = (
"r-2"
if second_project == "pipeline" and len(self.sent) == 3
else f"r-{len(self.sent)}"
)
await events.put(
json.dumps({"type": "response.created", "response": {"id": response_id}})
)
failure_case = second_project in {"failed", "incomplete"}
if (
(len(self.sent) == 2 and not failure_case)
or (failure_case and len(self.sent) == 1)
or (second_project == "pipeline" and len(self.sent) == 3)
):
for event_type in ("response.output_item.added", "response.output_item.done"):
await events.put(json.dumps({"type": event_type, "item": function_call}))
terminal_type = (
f"response.{second_project}"
if failure_case and len(self.sent) == 1
else "response.completed"
)
await events.put(json.dumps({"type": terminal_type, "response": {"id": response_id}}))
async def _iter(self):
while True:
yield await events.get()
class WebSocket(_FakeWebSocket):
async def receive_text(self) -> str:
if len(self._frames) == 1:
await first_completed.wait()
if disable_injection:
monkeypatch.setenv("HEADROOM_MEMORY_INJECTION_MODE", "disabled")
if second_project == "pipeline" and not self._frames and not init_release.is_set():
return frame("a")
if (
second_project == "overlap"
and not disable_injection
and not self._frames
and not init_release.is_set()
):
await init_blocked.wait()
return frame(None)
return await super().receive_text()
async def send_text(self, text: str) -> None:
await super().send_text(text)
event = json.loads(text)
if event.get("type") in {
"response.completed",
"response.failed",
"response.incomplete",
}:
if event["response"]["id"] == "r-1":
first_completed.set()
else:
self.trigger_disconnect()
class Memory(_MemoryWsHandler):
def __init__(self) -> None:
super().__init__()
self.config.project_root_override = ""
self.executed = []
async def _execute_memory_tool(
self, name, args, user_id, provider, *, request_context=None
):
self.executed.append((name, request_context))
return '{"memories": []}'
async def _ensure_initialized(self):
if second_project == "overlap":
init_blocked.set()
await init_release.wait()
await super()._ensure_initialized()
second = {
"same": "a",
"mismatch": "b",
"unresolved": "unknown",
"missing": None,
"overlap": "a",
"pipeline": None,
"failed": "a",
"incomplete": "a",
}[second_project]
upstream = Upstream([])
websocket = WebSocket(frames=[frame("a"), frame(second)], hold_after_initial=True)
websocket.headers["X-Codex-Turn-Metadata"] = json.dumps(
{"thread_id": "thread-a", "turn_id": "turn-a"}
)
websocket.headers["x-headroom-user-id"] = "user-1"
websocket.headers["user-agent"] = user_agent
handler = _WSHandler()
handler.memory_handler = memory = Memory()
observations = []
async def observe(payload, **kwargs):
observations.append(payload)
handler._observe_openai_ws_response_create = observe
with patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}):
await asyncio.wait_for(handler.handle_openai_responses_ws(websocket), timeout=3)
allowed = second_project == "same" and not disable_injection
if allowed:
assert len(memory.executed) == 1
assert memory.executed[0][0] == tool_name
assert memory.executed[0][1].project_root_override == str((tmp_path / "a").resolve())
assert len(upstream.sent) == 3
else:
assert memory.executed == []
expected_frames = (
3
if second_project == "pipeline"
or (second_project == "overlap" and not disable_injection)
else 2
)
assert len(upstream.sent) == expected_frames
assert any(json.loads(text).get("item") == function_call for text in websocket.sent_text)
assert observations == []
@pytest.mark.asyncio
async def test_concurrent_ws_connections_keep_same_basename_projects_isolated(
monkeypatch, tmp_path: Path
) -> None:
codex_home = tmp_path / "codex"
codex_home.mkdir()
project_a = tmp_path / "a" / "shared"
project_b = tmp_path / "b" / "shared"
project_a.mkdir(parents=True)
project_b.mkdir(parents=True)
rollout_a = codex_home / "rollout-a.jsonl"
rollout_b = codex_home / "rollout-b.jsonl"
_seed_rollout(rollout_a, "turn-a", project_a)
_seed_rollout(rollout_b, "turn-b", project_b)
_seed_thread(codex_home, "thread-a", rollout_a)
_seed_thread(codex_home, "thread-b", rollout_b)
monkeypatch.setenv("CODEX_HOME", str(codex_home))
class Memory:
def __init__(self) -> None:
self.config = SimpleNamespace(
inject_context=True,
inject_tools=False,
project_root_override="",
)
self.projects = []
async def search_and_format_context(self, _user_id, _messages, **kwargs):
self.projects.append(kwargs["request_context"].project_root_override)
return None
def compute_memory_tool_definitions(self, _provider):
return []
def frame(thread_id: str, turn_id: str, text: str) -> str:
return json.dumps(
{
"type": "response.create",
"response": {
"model": "gpt-5.4",
"input": text,
"client_metadata": {"thread_id": thread_id, "turn_id": turn_id},
},
}
)
completion = json.dumps(
{
"type": "response.completed",
"response": {
"id": "response",
"usage": {"input_tokens": 10, "output_tokens": 2},
},
}
)
upstreams = [
_FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "a"}}),
completion,
]
),
_FakeUpstream(
[
json.dumps({"type": "response.created", "response": {"id": "b"}}),
completion,
]
),
]
connect_kwargs = []
async def connect(*_args, **kwargs):
connect_kwargs.append(kwargs)
return upstreams.pop(0)
handlers = [_WSHandler(), _WSHandler()]
memories = [Memory(), Memory()]
for handler, memory in zip(handlers, memories, strict=True):
handler.config.memory_storage_mode = "project"
handler.memory_handler = memory
websockets = SimpleNamespace(connect=connect)
with patch.dict(sys.modules, {"websockets": websockets}):
await asyncio.gather(
handlers[0].handle_openai_responses_ws(
_FakeWebSocket(
frames=[frame("thread-a", "turn-a", "A")],
headers={
"authorization": "Bearer test",
"x-codex-turn-metadata": json.dumps(
{"thread_id": "thread-a", "turn_id": "turn-a"}
),
},
)
),
handlers[1].handle_openai_responses_ws(
_FakeWebSocket(
frames=[frame("thread-b", "turn-b", "B")],
headers={
"authorization": "Bearer test",
"x-codex-turn-metadata": json.dumps(
{"thread_id": "thread-b", "turn_id": "turn-b"}
),
},
)
),
)
expected_keys = {
CodexProjectContextResolver()
.resolve(
headers={},
body={"client_metadata": {"thread_id": thread_id, "turn_id": turn_id}},
)
.project_key
for thread_id, turn_id in (("thread-a", "turn-a"), ("thread-b", "turn-b"))
}
assert {memory.projects[0] for memory in memories} == {
str(project_a.resolve()),
str(project_b.resolve()),
}
assert {
handler.metrics.recorded_requests[-1]["project"] for handler in handlers
} == expected_keys
assert all(
"x-codex-turn-metadata" not in {key.lower() for key in kwargs["additional_headers"]}
for kwargs in connect_kwargs
)
+11 -6
View File
@@ -23,7 +23,7 @@ class _MemoryHandler:
self.config = SimpleNamespace(
inject_context=True,
inject_tools=True,
project_root_override="",
project_root_override=str(Path(__file__).resolve().parent),
)
self.queries: list[str] = []
@@ -165,7 +165,8 @@ async def test_memory_lookup_runs_for_each_issue_artifact_frame_and_preserves_no
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
],
hold_after_events=True,
)
first_turn, later_turn = _issue_2059_turns()
first_input, later_input = _issue_2059_inputs()
@@ -225,7 +226,8 @@ async def test_memory_lookup_skips_input_bearing_non_create_first_frame():
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
],
hold_after_events=True,
)
_first_input, later_input = _issue_2059_inputs()
cancel_frame = json.dumps(
@@ -256,7 +258,8 @@ async def test_memory_lookup_skips_bypassed_frames():
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
],
hold_after_events=True,
)
first, later = _issue_2059_turns()
client_ws = _FakeWebSocket(
@@ -280,7 +283,8 @@ async def test_memory_lookup_keeps_legacy_direct_first_frame():
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
],
hold_after_events=True,
)
first_input, later_input = _issue_2059_inputs()
first = _direct_turn(first_input)
@@ -307,7 +311,8 @@ async def test_memory_lookup_skips_disabled_memory(monkeypatch):
[
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
],
hold_after_events=True,
)
first, later = _issue_2059_turns()
client_ws = _FakeWebSocket(frames=[first, later])
+2 -1
View File
@@ -3,6 +3,7 @@ import base64
import json
import sys
from copy import deepcopy
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
@@ -279,7 +280,7 @@ class _MemoryToolsOnlyHandler:
self.config = SimpleNamespace(
inject_context=False,
inject_tools=True,
project_root_override="",
project_root_override=str(Path(__file__).resolve().parent),
)
self.compute_calls = 0
+7 -4
View File
@@ -12,6 +12,7 @@ import contextlib
import json
import logging
import sys
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import MagicMock, patch
@@ -75,7 +76,7 @@ class _MemoryWsHandler:
self.config = SimpleNamespace(
inject_context=False,
inject_tools=True,
project_root_override="",
project_root_override=str(Path(__file__).resolve().parent),
)
self._backend = False
@@ -101,6 +102,8 @@ class _MemoryWsHandler:
args: dict,
user_id: str,
provider: str,
*,
request_context=None,
) -> str:
assert (name, args, user_id, provider) == (
"memory_search",
@@ -2001,7 +2004,7 @@ async def test_ws_late_memory_call_after_streamed_message_passes_through():
handler.memory_handler = _MemoryWsHandler()
executed: list[tuple[str, dict, str, str]] = []
async def _execute_memory_tool(name, args, user_id, provider):
async def _execute_memory_tool(name, args, user_id, provider, *, request_context=None):
executed.append((name, args, user_id, provider))
return '{"memories": []}'
@@ -2088,7 +2091,7 @@ async def test_ws_memory_continuation_normalizes_malformed_arguments():
handler.memory_handler = _MemoryWsHandler()
executed: list[tuple[str, dict, str, str]] = []
async def _execute_memory_tool(name, args, user_id, provider):
async def _execute_memory_tool(name, args, user_id, provider, *, request_context=None):
executed.append((name, args, user_id, provider))
return '{"memories": []}'
@@ -2211,7 +2214,7 @@ async def test_ws_memory_continuation_continues_pre_stream_and_passes_late_call(
handler.memory_handler = _MemoryWsHandler()
executed: list[tuple[str, dict, str, str]] = []
async def _execute_memory_tool(name, args, user_id, provider):
async def _execute_memory_tool(name, args, user_id, provider, *, request_context=None):
executed.append((name, args, user_id, provider))
return '{"memories": []}'