mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
feat(proxy): give turn hooks one conversation id per client conversation (#4496)
## Description Plugins that pin wording per conversation (headroom-condense, and through it skill-search and tool-search) cannot see a real conversation id. Today: - **Claude:** they hash the first typed prompt, so two sessions that open with the same words share a key. - **Codex:** they get no key at all, so every new skill or MCP description waits out the one-hour hold. This PR adds an optional `TurnContext.conversation` that core fills from the client's own conversation id, so hooks can pin per real conversation. It is item #4 of the compression-gaps plan; the plugin work for #3, #6, #8 and #9 needs it to stay cache-safe. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **New field:** `TurnContext.conversation: str | None = None`. It is an opaque key that stays the same for every turn and retry of one client conversation, or `None` when the client names none. Hooks get a digest, never the raw id. | Path | Source | | --- | --- | | `/v1/messages` | sha256 of `X-Claude-Code-Session-Id` (Claude Code sends it on every request), falling back to `X-Headroom-Session-Id`; each source has its own namespace | | `/v1/responses` (HTTP and WebSocket) | the new `turn_conversation_key`, computed from the body before compression and passed through `_compress_openai_responses_payload_in_executor` | | `/v1/chat/completions` | `X-Headroom-Session-Id` only | | `/v1/compress` (gateway) | the gateway session id; `None` without one | - **Responses key.** `turn_conversation_key` uses the same identity and digest as `savings_conversation_key` (`thread_id` before `session_id`, then `client_metadata.*`, then the session header). Unlike the savings key, it also applies on `previous_response_id` turns and on bodies that name a server-side `conversation` (a string, or an object with an `id`), so the key does not drop out after the first WebSocket frame. `savings_conversation_key` keeps its existing exclusions. - **Gateway seam.** `CompressTurn.prepare()` is a third-party seam, so its signature is unchanged. Instead, the handler sets an optional writable `conversation` attribute on the turn before calling `prepare`. This is documented in `compress_turn.py`. - **No new ids.** Nothing is generated. The per-connection WebSocket uuid is not used, because a reconnect would change it. - **One path left out.** The catch-all passthrough Responses route still passes `None`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`): 3 errors, all in files this PR does not touch (embedders, model_metadata, langchain) - [x] New tests added for new functionality - [x] Manual testing performed - **New tests:** - `tests/test_turn_context_conversation.py`: - Anthropic: the same session gives one key, a different session with the same prompt gives a different key, the fallback header gets its own key, and no header gives `None`. - Responses HTTP: the same thread gives one key, including on a `previous_response_id` turn, and a different thread gives a different key. - Responses server-side `conversation`: string and object forms give one key across turns; a different conversation gives a different key; the savings key stays `None`. - Responses WebSocket: first and later frames share one key; a different thread gets a different key. - Chat Completions: covered the same way. - `tests/gateway/test_contract_request_half.py`: per-session gateway keys, and `None` without a session. - Every new test fails on `main`. - **Updated test:** `tests/test_proxy_openai.py` now pins `conversation=None` for a body with no ids. ### Test Output ```text pytest (turn hooks, gateway, Codex WS lifecycle, proxy openai, conversation savings, passthrough, hook usage): 411 passed, 5 skipped after the server-side conversation fix, on the synced head: 101 passed (conversation, gateway, Responses suites) ruff@0.17.0 check / format --check: clean mypy headroom --ignore-missing-imports: 3 errors, none in changed files ``` `tests/test_turn_hook_usage.py` needs `LITELLM_LOCAL_MODEL_COST_MAP=True` in a fresh venv; that is an existing litellm price-map fetch, and it fails the same way on `main`. ## Real Behavior Proof - Environment: - macOS 27.0.1 (Darwin 27.0.0, arm64), Python 3.12.13. - The proxy ran from this branch in a fresh `uv` venv (`uv venv -p 3.12 && uv pip install -e ".[dev]"`) on 127.0.0.1:9100, with default `ProxyConfig` (optimize on). - Upstream was the real Anthropic API, with my own API key passed by environment variable. - Client was Claude Code 2.1.311 (`claude --version`) with `--model claude-haiku-5-5`. - Exact command / steps: 1. Start the proxy with a scratch-only launcher (not committed). It calls `register_turn_hook()` with a hook whose `on_request` appends `ctx.provider`, `ctx.model`, `ctx.conversation` and `len(ctx.messages)` to a JSONL file, then calls `run_server(ProxyConfig(host="127.0.0.1", port=9100))`. Run it with `PYTHONSAFEPATH=1 HEADROOM_TELEMETRY=off LITELLM_LOCAL_MODEL_COST_MAP=True python run_proxy.py hooks.jsonl`, from a directory outside the repo. 2. Point Claude Code at the proxy: `export ANTHROPIC_BASE_URL=http://127.0.0.1:9100 CLAUDE_CONFIG_DIR=<fresh empty dir>`. 3. Start two sessions with the same prompt: `claude -p "say hi" --model claude-haiku-5-5 --output-format json`, run twice. Note each `session_id`. 4. Add two more turns to session 1: `claude -p "now say bye" --resume <session 1 id> --model claude-haiku-5-5`, then `claude -p "one more word please" --resume <session 1 id> --model claude-haiku-5-5`. 5. Read `hooks.jsonl`, and compare each key with sha256(`claude-code\0` + session_id). - Observed result (from `hooks.jsonl`, keys truncated): | Session | Turn | Messages | `ctx.conversation` | | --- | --- | --- | --- | | `305d64a9…` | first | 2 | `306306eeb9bc164b…` | | `4e50609f…` (same prompt) | first | 2 | `4a9e51bc7f8005f2…` | | `305d64a9…` | first `--resume` | 4 | `306306eeb9bc164b…` | | `305d64a9…` | second `--resume` | 6 | `306306eeb9bc164b…` | Each key equals sha256(`claude-code\0<session id>`) of the session id that Claude Code reported. All three runs answered normally, and the proxy log had no errors. - Not tested: - Codex over Responses HTTP or WebSocket (unit and handler tests cover these). - Chat Completions with `X-Headroom-Session-Id`. - The `/v1/compress` gateway path. - The server-side `conversation` body form, live (covered by the handler test). - The `X-Headroom-Session-Id` fallback on `/v1/messages`. - Claude Code `/compact` and subagents. - Bedrock and Vertex upstreams. - Linux and Windows. - A real plugin consuming the key. That is wired in headroomlabs-ai/headroom-condense#7; skill-search and tool-search adoption comes next. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: stable. - Stable/default behavior changed: no request or response bytes change. Registered turn hooks receive one new optional field; hooks that do not read it behave as before. - Kill switch / disable path: none needed; a hook that should not use the key ignores it (headroom-condense falls back to its previous key when the field is `None` or absent). - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this PR. Plugins read the field with `getattr`, so they keep working on cores without it. Cache note: compare keys only within one endpoint, since the same header gives different keys on `/v1/messages` and `/v1/responses`. Each key comes from a client id that stays the same for the whole conversation (Claude Code session ids survive retries, `--resume` and `/compact`; Codex `thread_id` wins over `session_id`, so guardian and subagent requests do not collide), and the Responses key is taken before compression rewrites the body, so a pinning hook keeps the same pin on every turn. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A (proxy-side change). ## Additional Notes - "No new warnings" is unchecked because the run was not compared warning by warning against `main`. - Follow-ups: skill-search and tool-search switch to `conversation_for(ctx)` once headroom-condense#7 is released. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Maintainer post-merge qualification Published `819f82117a4737de621528e9c49aee6aeeb4a793` includes upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 28 conversation/Responses/per-frame-memory/disconnect tests pass; Ruff/format/diff pass. Clean main merge; hook conversation policy unchanged, no new code finding. Author explicitly reserves unchecked self-review/readiness attestations; existing hold remains. Contributor correctness logic was not rewritten. Existing Changes Requested decision retained without duplicate findings. Scoped Windows validation uses the existing development native extension, not a fresh build. Hosted qualification is handled by maintainers; nothing merged. ### Maintainer current-main requalification Exact head `965ff21ffbea9b446e5e39269812bb0d8c43c108`, upstream main `ecbaf86d78cf52d185fbffa306157b9a4cc1b906`. Mechanically updated from upstream/mainecbaf86d78with no conflicts and no contributor conversation/ownership policy changes. Re-reviewed current discussion and the integration delta. All 38 focused conversation-context, Responses, per-frame WebSocket memory, disconnect and Gemini tests pass; Ruff0.17.3 lint/format and whitespace checks pass. Existing runtime evidence remains scoped; local tests use the existing native extension, not a fresh build. No new source blocker was found. The existing readiness action remains: the author has explicitly reserved and still not checked the self-review and ready-for-human-review attestations. They are not asserted on the author's behalf. Preserve the existing Changes Requested readiness hold until the author completes that action. Maintainers handle exact-head hosted checks; no contributor CI action requested. Nothing merged. Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged. ### Maintainer current-main requalification Exact head `db69aa725d085d75b0881f0b2e1f2237480ee0c6`, upstream main `8d3e9e5ca7cd645eb73084b848edbea5fc796e1a`. Clean mechanical sync with upstream main8d3e9e5ca7cd645eb73084b848edbea5fc796e1a. Re-reviewed conversation-context integration and fresh author/Devin discussion without changing contributor identity/ownership policy.38conversation/Responses/per-frame-WebSocket/disconnect/Gemini tests pass;10changed Python files pass Ruff0.17.3check/format and whitespace checks. Existing runtime evidence remains scoped; copied extension supports Python startup, not a fresh native build. No new source blocker found. The current body still leaves author self-review and ready-for-human-review attestations unchecked and reserves those actions for the author. Preserve that existing readiness hold; no boxes were ticked or attested on their behalf. This is not an inherited-main CI defect. Maintainers handle exact-head hosted checks; nothing merged. Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged. ### Maintainer current-main requalification Exact head `7411de5607beabfcbbd63b134795b0053da0e681`, upstream main `7592e681459f14a2041eb5f524a658cb6ad1dd92`. Clean current-main sync7592e681459f14a2041eb5f524a658cb6ad1dd92,including latestmemoryfixes. Conversationidentity/ownership policyunchanged.46conversation/Responses/perframeWS/disconnect/memory-regression/Gemini tests pass;whitespace passes. PriorRuff0.17.3check/format applies to unchangedPRfiles. Freshbodystillreservesuncheckedself-review/ready-for-humanreviewboxesfortheauthor. No newsourceblockerfound;retainexistingauthor-readinessholdwithouttickingboxesonbehalf. Existingruntimeevidence scoped,no freshnativebuildclaim. Maintainershandlechecks;nothingmerged. Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged. ### Maintainer current-main requalification Exact head `cdf48ae9561dd2075625a4085594ac429a421231`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Clean integration with upstream/main2d3ee51846after the external 0.43.1 release merge. Exact headroom/crates/tests comparison against previously reviewed7411de560shows no changes. Re-reviewed generated metadata; 78 release/version/MCP tests pass on this tree and whitespace passes. Earlier 46 conversation/Responses/WebSocket/disconnect/memory/Gemini passes remain scoped to unchanged inputs. No new source blocker found in reviewed scope. Existing author-owned readiness declaration remains the hold; fresh body must be checked before publication without checking author attestations on their behalf. No contributor identity/ownership policy rewritten, no fresh native-build/provider claim. Maintainers handle checks; nothing merged. Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged. ### Maintainer current-main requalification Exact head `d343c187f73b6a9005f0268e280ec657f435f1df`, upstream main `ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`. Clean integration with upstream/mainea59ac3eefafter the Click/macOS setup merge. Re-reviewed the OpenAI response replay transport-header overlap; conversation and existing tool-name map parameters remain additive.14 conversation/context-compaction tests and3 targeted route/gateway cases pass (71 unrelated cases deselected); Ruff and whitespace pass. No new source blocker found. Fresh body still leaves author-owned self-review/readiness attestations unchecked; preserve the existing hold without attesting on their behalf. Existing wider runtime evidence remains scoped; no fresh native-build or live-provider claim. Maintainers handle hosted checks; nothing merged. Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged. ### Maintainer current-main requalification Exact head `741582654b9d0817ca8318af6aaf2dd000d1ec81`, upstream main `c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`. Final post-Dependabot requalification against upstream/mainc01cf07d94. All six Dependabot updates are now merged. Relative to this branch's freshly tested prepared tree, the final main sync changes only the three intended Cargo.lock records; Python/source/test inputs are unchanged. Re-reviewed the integration and confirmed complete native source/lock/toolchain objects match the tested combined build and exact fresh native DLL used for this branch's local tests. Focused fresh-native qualification: 18 passed, 70 deselected, 1 warning in 20.19s. Selected real dependency overlays and native DLL paths are recorded in final4496-fresh-native-tests.log; the DLL provenance/input hashes are in final-native-artifact-proof.json. No redundant rerun is implied after this Cargo-only merge. Whitespace passes. Prior source review and scoped real-behavior evidence remain applicable to unchanged contributor inputs. No new source blocker found. Fresh body still leaves author self-review and ready-for-human-review attestations unchecked. Retain the author-owned readiness hold without checking those boxes on their behalf. Maintainers handle hosted checks; no external-provider claim, contributor semantic rewrite or non-Dependabot merge. Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged. ### Maintainer current-main requalification Exact head `cb5c34b6221880a811dd94d21c5e3f3e874ee565`, upstream main `da6d9736cd82074988c7d0a2360b49d9d8b6c0a0`. Current-main requalification againstda6d9736cd. All69 integration patches exactly equal c01cf07-to-current-main delta after diff-location normalization; conversation identity/ownership policy unchanged.88 conversation,Responses context-compaction,OpenAI route,gateway request-contract tests pass;1 deprecation warning. Post-suite PyTorch cache-stat logging reports closed-stream logging errors during process exit; pytest exit0 and test results are retained without claiming full runtime qualification.31 changed Python files pass Ruff lint/format; offline lock and whitespace pass. Existing pr3982 native DLL verified full input hashes/artifact checksum, no fresh build claimed. Author/Devin discussion considered; previous substantive Devin findings resolved. No new source blocker found. Author explicitly reserves self-review/ready-for-human-review attestations and both remain unchecked; retain existing readiness hold without attesting on their behalf. Maintainers handle hosted checks; no contributor logic rewrite or merge. Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged. ### Maintainer current-main requalification Exact head `951bdf5853aaf8757ba46253763c91b6601b5c6d`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean integration with upstream/main0ddaa56c21. All 19 integration patches exactly match the prior-reviewed-mainda6d9736to current-main delta after diff-location normalization; conversation identity/ownership policy is unchanged. Fresh conversation/turn-hook/Responses/gateway request-contract, doctor and incoming Vertex/backend/CLI suite: 367 passed, 11 skipped, 1 warning. Fourteen changed Python files pass Ruff lint/format; offline lock and whitespace checks pass. Author discussion and Devin findings read: both historical Devin items are marked resolved, including server-side conversation IDs. No new source blocker found. Existing author-reserved self-review/ready-for-human-review attestations remain a hold; this maintainer does not attest on the author's behalf. Static source/test scope and security inspection found no new workflow/dependency/setup path; the inherited flagged workflow-validation job is excluded by this exact PR's unchanged workflow paths. Conversation digests are not an authentication boundary. Verified recorded native input-set/hashes and artifact checksum before DLL reuse; no fresh build or new live-client qualification claimed. Maintainers handle exact-head hosted checks; no PR merged. Exact-head hosted checks are being handled by maintainers. Existing correctness/qualification holds remain; nothing merged. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
51d9dc59b3
commit
94a99ed20f
@@ -99,6 +99,10 @@ class CompressTurn(Protocol):
|
||||
# And ``compaction_tokens_before``/``compaction_tokens_after``, the tool
|
||||
# definitions before and after the schema compaction ``transform`` ran,
|
||||
# which the handler folds into both endpoints as the chat handlers do.
|
||||
# In the other direction: a turn with a writable ``conversation: str |
|
||||
# None`` attribute has it set before ``prepare`` to an opaque key for the
|
||||
# request's session (None without one), for its hooks'
|
||||
# ``TurnContext.conversation``.
|
||||
|
||||
def finish(
|
||||
self,
|
||||
|
||||
@@ -49,6 +49,7 @@ __all__ = [
|
||||
"get_conversation_savings",
|
||||
"reset_conversation_savings",
|
||||
"savings_conversation_key",
|
||||
"turn_conversation_key",
|
||||
]
|
||||
|
||||
_ID_KEYS = ("id", "conversation_id", "session_id", "thread_id")
|
||||
@@ -103,9 +104,24 @@ def savings_conversation_key(body: Any, *, session_id: str | None = None) -> str
|
||||
return None
|
||||
if body.get("previous_response_id") or body.get("conversation"):
|
||||
return None
|
||||
return turn_conversation_key(body, session_id=session_id)
|
||||
|
||||
|
||||
def turn_conversation_key(body: Any, *, session_id: str | None = None) -> str | None:
|
||||
"""The conversation a Responses request names, or ``None`` when it names none.
|
||||
|
||||
Premise 1 of :func:`savings_conversation_key` only, with the same ids and
|
||||
the same digest: an incremental body (``previous_response_id``) still
|
||||
belongs to its conversation, so this is the key turn hooks see as
|
||||
``TurnContext.conversation`` on every turn. So does a body that names a
|
||||
server-side ``conversation`` (an id, or an object with one), which
|
||||
:func:`savings_conversation_key` excludes before calling this.
|
||||
"""
|
||||
if not isinstance(body, dict):
|
||||
return None
|
||||
body = _unwrap_response_create_body(body)
|
||||
identity = ""
|
||||
for key in ("conversation_id", "thread_id", "session_id"):
|
||||
for key in ("conversation_id", "thread_id", "session_id", "conversation"):
|
||||
value = _explicit_id(body.get(key))
|
||||
if value:
|
||||
identity = f"{key}:{value}"
|
||||
|
||||
@@ -93,6 +93,9 @@ class GatewayTurn:
|
||||
self._obligations: list[str] = []
|
||||
self._turn_id: str | None = None
|
||||
self._ccr_armed = False
|
||||
# Set by the compress handler from the session id (see CompressTurn);
|
||||
# hooks see it as ``TurnContext.conversation``.
|
||||
self.conversation: str | None = None
|
||||
|
||||
# -- fail-open ---------------------------------------------------------
|
||||
def fail_open_fields(self, messages: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
@@ -124,6 +127,7 @@ class GatewayTurn:
|
||||
config=config,
|
||||
tags=tags,
|
||||
caps=self.caps,
|
||||
conversation=self.conversation,
|
||||
)
|
||||
|
||||
# -- executor side -----------------------------------------------------
|
||||
|
||||
@@ -908,6 +908,7 @@ class RequestTransformer:
|
||||
config: Any,
|
||||
tags: dict[str, Any],
|
||||
caps: GatewayCapabilities,
|
||||
conversation: str | None = None,
|
||||
) -> None:
|
||||
self.provider = provider
|
||||
self.model_name = model_name
|
||||
@@ -915,6 +916,8 @@ class RequestTransformer:
|
||||
self.config = config
|
||||
self.tags = tags
|
||||
self.caps = caps
|
||||
# ``TurnContext.conversation``: the gateway session, when one is named.
|
||||
self.conversation = conversation
|
||||
self.result: RequestTransformResult | None = None
|
||||
# Tool-definition tokens the request hooks ADDED (e.g. a search tool).
|
||||
# The compress handler adds it to ``tokens_after``: it is sent too.
|
||||
@@ -990,6 +993,7 @@ class RequestTransformer:
|
||||
tags=self.tags,
|
||||
count_messages=count_messages,
|
||||
count_tools=count_tools,
|
||||
conversation=self.conversation,
|
||||
)
|
||||
if hooks:
|
||||
msg_before = _safe_count(count_messages, messages)
|
||||
|
||||
@@ -3882,6 +3882,7 @@ class AnthropicHandlerMixin:
|
||||
# registry check keeps this a no-op when no hook is registered.
|
||||
from headroom.proxy.turn_hooks import (
|
||||
TurnContext,
|
||||
conversation_from,
|
||||
registered_turn_hooks,
|
||||
run_request_hooks,
|
||||
)
|
||||
@@ -3903,6 +3904,14 @@ class AnthropicHandlerMixin:
|
||||
tags=tags,
|
||||
count_messages=tokenizer.count_messages,
|
||||
count_tools=_count_tool_tokens,
|
||||
# Claude Code sends one session id on every request and
|
||||
# retry of a session; other clients may send Headroom's.
|
||||
conversation=conversation_from(
|
||||
"claude-code", request.headers.get("x-claude-code-session-id")
|
||||
)
|
||||
or conversation_from(
|
||||
"headroom-session", request.headers.get("x-headroom-session-id")
|
||||
),
|
||||
)
|
||||
# Snapshot BEFORE the hook (same tokenizer) so we can tell whether the
|
||||
# hook itself folded — comparing against the pipeline's optimized_tokens
|
||||
|
||||
@@ -27,7 +27,7 @@ from typing import TYPE_CHECKING, Any
|
||||
from urllib.parse import quote, unquote, urlparse
|
||||
|
||||
from headroom.log_safety import describe_exception
|
||||
from headroom.proxy.conversation_savings import savings_conversation_key
|
||||
from headroom.proxy.conversation_savings import savings_conversation_key, turn_conversation_key
|
||||
from headroom.proxy.handlers._failure_logging import log_hook_failure, log_request_failure
|
||||
from headroom.proxy.helpers import (
|
||||
COMPRESSION_TIMEOUT_SECONDS,
|
||||
@@ -59,7 +59,7 @@ from headroom.proxy.tool_schema_savings_policy import (
|
||||
tool_schema_saved_from_tags,
|
||||
without_deferral_flags,
|
||||
)
|
||||
from headroom.proxy.turn_hooks import run_responses_create_hooks
|
||||
from headroom.proxy.turn_hooks import conversation_from, run_responses_create_hooks
|
||||
from headroom.proxy.upstream_guard import is_safe_upstream_url
|
||||
from headroom.proxy.ws_headers import WS_HOP_BY_HOP_HEADERS
|
||||
from headroom.proxy.ws_session_registry import (
|
||||
@@ -196,6 +196,16 @@ def _response_ccr_hashes(messages: list[dict[str, Any]], markers: list[str]) ->
|
||||
return hashes
|
||||
|
||||
|
||||
def _responses_session_header(headers: Any) -> str | None:
|
||||
"""The conversation id a Responses client sends as a header (Codex sends
|
||||
``session_id``), on HTTP and on the WebSocket upgrade alike."""
|
||||
return (
|
||||
headers.get("conversation_id")
|
||||
or headers.get("session_id")
|
||||
or headers.get("x-headroom-session-id")
|
||||
)
|
||||
|
||||
|
||||
def _codex_ws_compression_timeout_seconds() -> float:
|
||||
return min(COMPRESSION_TIMEOUT_SECONDS, _CODEX_WS_COMPRESSION_TIMEOUT_SECONDS)
|
||||
|
||||
@@ -3091,6 +3101,7 @@ class OpenAIHandlerMixin:
|
||||
client: str | None = None,
|
||||
savings_tags: dict[str, Any] | None = None,
|
||||
deadline_started_at: float | None = None,
|
||||
conversation: str | None = None,
|
||||
tool_names_by_call_id: dict[str, str] | None = None,
|
||||
) -> tuple[dict[str, Any], bool, int, list[str], str | None, int, int, int]:
|
||||
"""Compress an OpenAI Responses payload through the shared router.
|
||||
@@ -3273,6 +3284,7 @@ class OpenAIHandlerMixin:
|
||||
if value
|
||||
else 0
|
||||
),
|
||||
conversation=conversation,
|
||||
)
|
||||
# Streaming turns get fold-only hooks, same rule as the
|
||||
# chat-completions path. A hook that defers work to `on_response`
|
||||
@@ -3393,8 +3405,12 @@ class OpenAIHandlerMixin:
|
||||
timeout: float = COMPRESSION_TIMEOUT_SECONDS,
|
||||
client: str | None = None,
|
||||
savings_tags: dict[str, Any] | None = None,
|
||||
conversation: str | None = None,
|
||||
tool_names_by_call_id: dict[str, str] | None = None,
|
||||
) -> tuple[dict[str, Any], bool, int, list[str], str | None, int, int, int, dict[str, float]]:
|
||||
"""``conversation`` is the caller's ``turn_conversation_key``,
|
||||
derived before compression rewrites the body; turn hooks see it as
|
||||
``TurnContext.conversation``."""
|
||||
timing: dict[str, float] = {}
|
||||
|
||||
# Codex >= 0.149.0 nests tool definitions in `input` items of type
|
||||
@@ -3443,6 +3459,8 @@ class OpenAIHandlerMixin:
|
||||
}
|
||||
if savings_tags is not None:
|
||||
compression_kwargs["savings_tags"] = savings_tags
|
||||
if conversation is not None:
|
||||
compression_kwargs["conversation"] = conversation
|
||||
if tool_names_by_call_id:
|
||||
compression_kwargs["tool_names_by_call_id"] = tool_names_by_call_id
|
||||
while True:
|
||||
@@ -3461,6 +3479,7 @@ class OpenAIHandlerMixin:
|
||||
"client",
|
||||
"timing",
|
||||
"deadline_started_at",
|
||||
"conversation",
|
||||
)
|
||||
if f"unexpected keyword argument '{name}'" in str(exc)
|
||||
and name in compression_kwargs
|
||||
@@ -4959,6 +4978,9 @@ class OpenAIHandlerMixin:
|
||||
|
||||
_th_ctx: TurnContext | None = None
|
||||
_th_raw_saved = None
|
||||
_th_conversation = conversation_from(
|
||||
"headroom-session", request.headers.get("x-headroom-session-id")
|
||||
)
|
||||
if registered_turn_hooks():
|
||||
_th_tools_before = body.get("tools")
|
||||
_th_tok_before = (
|
||||
@@ -4979,6 +5001,7 @@ class OpenAIHandlerMixin:
|
||||
count_tools=lambda value: (
|
||||
tokenizer.count_text(json.dumps(value, default=str)) if value else 0
|
||||
),
|
||||
conversation=_th_conversation,
|
||||
)
|
||||
# Snapshot messages BEFORE the hook (same tokenizer) so we can tell whether
|
||||
# the hook itself folded — comparing against optimized_tokens instead
|
||||
@@ -5290,6 +5313,7 @@ class OpenAIHandlerMixin:
|
||||
messages=optimized_messages,
|
||||
tools=tools,
|
||||
config=self.config,
|
||||
conversation=_th_conversation,
|
||||
),
|
||||
final_resp_json,
|
||||
api_call_fn,
|
||||
@@ -5578,6 +5602,7 @@ class OpenAIHandlerMixin:
|
||||
tools=body.get("tools"),
|
||||
config=self.config,
|
||||
tags=tags,
|
||||
conversation=_th_conversation,
|
||||
)
|
||||
|
||||
async def _hook_call_model(_msgs):
|
||||
@@ -6862,12 +6887,11 @@ class OpenAIHandlerMixin:
|
||||
# server-side state); the funnel then books per request.
|
||||
responses_conversation_key: str | None = None
|
||||
_pre_compression_conversation_key = savings_conversation_key(
|
||||
body,
|
||||
session_id=(
|
||||
request.headers.get("conversation_id")
|
||||
or request.headers.get("session_id")
|
||||
or request.headers.get("x-headroom-session-id")
|
||||
),
|
||||
body, session_id=_responses_session_header(request.headers)
|
||||
)
|
||||
# Turn hooks get the conversation on incremental bodies too.
|
||||
_turn_conversation = turn_conversation_key(
|
||||
body, session_id=_responses_session_header(request.headers)
|
||||
)
|
||||
|
||||
# The standalone Rust proxy has native /v1/responses item handling,
|
||||
@@ -6898,6 +6922,7 @@ class OpenAIHandlerMixin:
|
||||
request_id=request_id,
|
||||
client=client,
|
||||
savings_tags=tags,
|
||||
conversation=_turn_conversation,
|
||||
)
|
||||
attempted_input_tokens = int(_attempted_tokens)
|
||||
if _transforms:
|
||||
@@ -7220,6 +7245,7 @@ class OpenAIHandlerMixin:
|
||||
tools=body.get("tools"),
|
||||
config=self.config,
|
||||
tags=tags,
|
||||
conversation=_turn_conversation,
|
||||
)
|
||||
|
||||
async def _resp_hook_call_model(
|
||||
@@ -8994,6 +9020,9 @@ class OpenAIHandlerMixin:
|
||||
else COMPRESSION_TIMEOUT_SECONDS,
|
||||
client=client,
|
||||
savings_tags=_frame_savings,
|
||||
conversation=turn_conversation_key(
|
||||
_inner, session_id=_responses_session_header(ws_headers)
|
||||
),
|
||||
)
|
||||
for _timing_name, _timing_ms in _ws_compression_timing.items():
|
||||
_record_ws_compression_timing(_timing_name, _timing_ms)
|
||||
@@ -9398,6 +9427,10 @@ class OpenAIHandlerMixin:
|
||||
else COMPRESSION_TIMEOUT_SECONDS,
|
||||
client=client,
|
||||
savings_tags=frame_savings,
|
||||
conversation=turn_conversation_key(
|
||||
inner_payload,
|
||||
session_id=_responses_session_header(ws_headers),
|
||||
),
|
||||
tool_names_by_call_id=dict(ws_tool_names_by_call_id),
|
||||
)
|
||||
for _timing_name, _timing_ms in frame_compression_timing.items():
|
||||
@@ -11475,6 +11508,14 @@ class OpenAIHandlerMixin:
|
||||
# hooks, scope binding) — CPU-bound parts run inside the closures
|
||||
# below with the rest of the turn.
|
||||
if _turn is not None:
|
||||
if hasattr(_turn, "conversation"):
|
||||
try:
|
||||
_turn.conversation = conversation_from("compress", session_id)
|
||||
except AttributeError:
|
||||
logger.debug(
|
||||
"compress turn %s has a read-only conversation; left as is",
|
||||
type(_turn).__name__,
|
||||
)
|
||||
_turn.prepare(model_name=model_name, tags=tags, config=self.config)
|
||||
|
||||
# Whether the returned ``tokens_after`` was counted over the FINAL,
|
||||
|
||||
@@ -21,6 +21,7 @@ the same deprecation policy.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import inspect
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
@@ -42,6 +43,15 @@ log = logging.getLogger(__name__)
|
||||
CallModel = Callable[[list[dict[str, Any]]], Awaitable[dict[str, Any]]]
|
||||
|
||||
|
||||
def conversation_from(namespace: str, client_id: object) -> str | None:
|
||||
"""``TurnContext.conversation`` for a client-supplied id: a sha256 of
|
||||
``namespace`` and the id, so hooks get a stable opaque key and two sources
|
||||
can never alias; ``None`` when there is no id."""
|
||||
if not isinstance(client_id, str) or not client_id:
|
||||
return None
|
||||
return hashlib.sha256(f"{namespace}\0{client_id}".encode("utf-8", "ignore")).hexdigest()
|
||||
|
||||
|
||||
@dataclass
|
||||
class TurnContext:
|
||||
"""The request side of one model turn, as the proxy forwards it upstream.
|
||||
@@ -67,6 +77,14 @@ class TurnContext:
|
||||
# a token set behind the client's own value rather than overwritten. On the
|
||||
# gateway contract the merged map is the response's ``headers`` object.
|
||||
provider_headers: dict[str, str] = field(default_factory=dict)
|
||||
# An opaque key, stable for one client conversation (every turn and retry
|
||||
# of it), or None when the client names none. A digest, never the client's
|
||||
# raw id: from Claude Code's session header, Codex's thread or session id,
|
||||
# ``x-headroom-session-id``, or the gateway session. A hook that pins per
|
||||
# conversation (e.g. wording it must not change mid-cache) keys on it.
|
||||
# Keys compare within one endpoint: the same session header gives a
|
||||
# different key on /v1/messages and /v1/responses.
|
||||
conversation: str | None = None
|
||||
|
||||
def record_savings(
|
||||
self,
|
||||
|
||||
@@ -469,6 +469,32 @@ def test_turn_ids_are_unique_per_request(headroom_client) -> None:
|
||||
assert len(ids) == 5
|
||||
|
||||
|
||||
class _ConversationHook:
|
||||
name = "conversation"
|
||||
stream_safe = True
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.seen: list[str | None] = []
|
||||
|
||||
def on_request(self, ctx: Any) -> None:
|
||||
self.seen.append(ctx.conversation)
|
||||
|
||||
|
||||
def test_hooks_see_the_gateway_session_as_the_conversation(headroom_client) -> None:
|
||||
"""Each gateway session is one conversation; no session, no conversation."""
|
||||
hook = _ConversationHook()
|
||||
register_turn_hook(hook)
|
||||
for session in ("gw-c1", "gw-c1", "gw-c2"):
|
||||
resp = compress(headroom_client, _body(config={"session_id": session}, gateway={}))
|
||||
assert resp.status_code == 200, resp.text
|
||||
resp = compress(headroom_client, _body(gateway={}))
|
||||
assert resp.status_code == 200, resp.text
|
||||
first, again, other, none = hook.seen
|
||||
assert first == again and first is not None and first != "gw-c1"
|
||||
assert other not in (None, first)
|
||||
assert none is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("label", "shape", "model", "provider", "route_provider"),
|
||||
[
|
||||
|
||||
@@ -364,8 +364,9 @@ def _responses_body() -> dict[str, object]:
|
||||
@pytest.mark.parametrize(
|
||||
("user_agent", "expected"),
|
||||
[
|
||||
("opencode/0.4.2", {"client": "opencode"}),
|
||||
("codex-cli/1.2.3", {"client": "codex"}),
|
||||
# No conversation id in the body or headers: hooks get no conversation.
|
||||
("opencode/0.4.2", {"client": "opencode", "conversation": None}),
|
||||
("codex-cli/1.2.3", {"client": "codex", "conversation": None}),
|
||||
],
|
||||
)
|
||||
def test_native_responses_route_carries_the_client_decision(
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
"""``TurnContext.conversation``: one opaque key per client conversation.
|
||||
|
||||
Hooks that pin wording per conversation (headroom-condense) key on it, so two
|
||||
sessions that open with the same prompt must get different keys, and every
|
||||
turn of one conversation, retries and incremental Responses bodies included,
|
||||
the same key.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
from typing import Any
|
||||
from unittest.mock import patch
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
import respx
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
import headroom.proxy.handlers.openai as openai_module # noqa: E402
|
||||
from headroom.proxy.conversation_savings import savings_conversation_key # noqa: E402
|
||||
from headroom.proxy.loopback_guard import require_loopback # noqa: E402
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
from headroom.proxy.turn_hooks import clear_turn_hooks, register_turn_hook # noqa: E402
|
||||
from tests.test_openai_codex_ws_lifecycle import ( # noqa: E402
|
||||
_DummyOpenAIHandler,
|
||||
_FakeUpstream,
|
||||
_FakeWebSocket,
|
||||
_make_fake_websockets_module,
|
||||
)
|
||||
|
||||
|
||||
class _Recorder:
|
||||
name = "conversation_recorder"
|
||||
stream_safe = True
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.seen: list[str | None] = []
|
||||
|
||||
def on_request(self, ctx: Any) -> None:
|
||||
self.seen.append(ctx.conversation)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def recorder(monkeypatch):
|
||||
# litellm fetches its price map on import unless told not to; respx would
|
||||
# reject that request.
|
||||
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")
|
||||
clear_turn_hooks()
|
||||
hook = _Recorder()
|
||||
register_turn_hook(hook)
|
||||
yield hook
|
||||
clear_turn_hooks()
|
||||
|
||||
|
||||
def _app(*, optimize: bool):
|
||||
app = create_app(
|
||||
ProxyConfig(
|
||||
optimize=optimize,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
)
|
||||
)
|
||||
app.dependency_overrides[require_loopback] = lambda: None
|
||||
return app
|
||||
|
||||
|
||||
PROMPT = [{"role": "user", "content": "fix the bug"}]
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_claude_code_sessions_with_the_same_prompt_get_different_conversations(recorder):
|
||||
respx.post("https://api.anthropic.com/v1/messages").mock(
|
||||
return_value=httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "msg_1",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "claude-haiku-4-5",
|
||||
"content": [{"type": "text", "text": "ok"}],
|
||||
"stop_reason": "end_turn",
|
||||
"usage": {"input_tokens": 10, "output_tokens": 1},
|
||||
},
|
||||
)
|
||||
)
|
||||
body = {"model": "claude-haiku-4-5", "max_tokens": 16, "messages": PROMPT}
|
||||
sessions = [
|
||||
{"x-claude-code-session-id": "s-a"},
|
||||
{"x-claude-code-session-id": "s-a"}, # a retry, or the next turn
|
||||
{"x-claude-code-session-id": "s-b"},
|
||||
{"x-headroom-session-id": "s-a"},
|
||||
{},
|
||||
]
|
||||
with TestClient(_app(optimize=False)) as client:
|
||||
for headers in sessions:
|
||||
r = client.post(
|
||||
"/v1/messages", json=body, headers={"x-api-key": "sk-ant-test", **headers}
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
a, a_again, b, headroom_a, none = recorder.seen
|
||||
assert a == a_again and a is not None
|
||||
assert b not in (None, a)
|
||||
# Namespaced by source, and never the raw client id.
|
||||
assert headroom_a not in (None, a)
|
||||
assert "s-a" not in (a, headroom_a)
|
||||
assert none is None
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_responses_http_turns_of_one_thread_share_a_conversation(recorder):
|
||||
respx.post("https://api.openai.com/v1/responses").mock(
|
||||
return_value=httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "resp_1",
|
||||
"output": [{"type": "message", "role": "assistant", "content": []}],
|
||||
"usage": {"input_tokens": 10, "output_tokens": 1},
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
def body(thread: str, **extra: Any) -> dict[str, Any]:
|
||||
return {
|
||||
"model": "gpt-4o",
|
||||
"input": [{"type": "message", "role": "user", "content": "fix the bug"}],
|
||||
"stream": False,
|
||||
"client_metadata": {"thread_id": thread},
|
||||
**extra,
|
||||
}
|
||||
|
||||
with TestClient(_app(optimize=True)) as client:
|
||||
for sent in (body("t-1"), body("t-1", previous_response_id="resp_0"), body("t-2")):
|
||||
r = client.post("/v1/responses", json=sent, headers={"authorization": "Bearer sk-t"})
|
||||
assert r.status_code == 200, r.text
|
||||
first, incremental, other = recorder.seen
|
||||
assert first == incremental and first is not None
|
||||
assert other not in (None, first)
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_responses_server_side_conversation_names_the_conversation(recorder):
|
||||
respx.post("https://api.openai.com/v1/responses").mock(
|
||||
return_value=httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "resp_1",
|
||||
"output": [{"type": "message", "role": "assistant", "content": []}],
|
||||
"usage": {"input_tokens": 10, "output_tokens": 1},
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
def body(conversation: Any) -> dict[str, Any]:
|
||||
return {
|
||||
"model": "gpt-4o",
|
||||
"input": [{"type": "message", "role": "user", "content": "fix the bug"}],
|
||||
"stream": False,
|
||||
"conversation": conversation,
|
||||
}
|
||||
|
||||
sent = [body("conv_1"), body("conv_1"), body({"id": "conv_1"}), body("conv_2")]
|
||||
with TestClient(_app(optimize=True)) as client:
|
||||
for one in sent:
|
||||
r = client.post("/v1/responses", json=one, headers={"authorization": "Bearer sk-t"})
|
||||
assert r.status_code == 200, r.text
|
||||
first, again, as_object, other = recorder.seen
|
||||
assert first == again == as_object and first is not None
|
||||
assert other not in (None, first)
|
||||
# The savings funnel still books these per request: the provider holds the history.
|
||||
assert all(savings_conversation_key(one) is None for one in sent)
|
||||
|
||||
|
||||
def _ws_frame(thread: str, **extra: Any) -> str:
|
||||
response = {
|
||||
"model": "gpt-5.4",
|
||||
"input": "fix the bug",
|
||||
"client_metadata": {"thread_id": thread},
|
||||
**extra,
|
||||
}
|
||||
return json.dumps({"type": "response.create", "response": response})
|
||||
|
||||
|
||||
async def _ws_conversations(frames: list[str]) -> list[str | None]:
|
||||
"""The ``conversation`` each WS frame's compression pass was handed."""
|
||||
seen: list[str | None] = []
|
||||
|
||||
def _compress(payload, *, model, request_id, conversation=None, **_kw):
|
||||
seen.append(conversation)
|
||||
return payload, False, 0, [], "router_no_compression", 10, 10, 0
|
||||
|
||||
client_ws = _FakeWebSocket(frames=frames, hold_after_initial=True)
|
||||
handler = _DummyOpenAIHandler()
|
||||
handler.config.optimize = True
|
||||
handler._compress_openai_responses_payload = _compress # type: ignore[method-assign]
|
||||
upstream = _FakeUpstream([], hold_after_events=True)
|
||||
|
||||
async def _disconnect() -> None:
|
||||
await asyncio.sleep(0.05)
|
||||
client_ws.trigger_disconnect()
|
||||
|
||||
with (
|
||||
patch.object(openai_module, "COMPRESSION_TIMEOUT_SECONDS", 30.0),
|
||||
patch.dict(sys.modules, {"websockets": _make_fake_websockets_module(upstream)}),
|
||||
):
|
||||
trigger = asyncio.create_task(_disconnect())
|
||||
try:
|
||||
await asyncio.wait_for(handler.handle_openai_responses_ws(client_ws), timeout=2.0)
|
||||
finally:
|
||||
trigger.cancel()
|
||||
return seen
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_responses_ws_frames_of_one_thread_share_a_conversation():
|
||||
first, later = await _ws_conversations(
|
||||
[_ws_frame("t-1"), _ws_frame("t-1", previous_response_id="resp_1")]
|
||||
)
|
||||
(other,) = await _ws_conversations([_ws_frame("t-2")])
|
||||
assert first == later and first is not None
|
||||
assert other not in (None, first)
|
||||
|
||||
|
||||
@respx.mock
|
||||
def test_chat_completions_take_the_headroom_session_header(recorder):
|
||||
respx.post("https://api.openai.com/v1/chat/completions").mock(
|
||||
return_value=httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "c",
|
||||
"choices": [
|
||||
{"message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}
|
||||
],
|
||||
"usage": {"prompt_tokens": 10, "completion_tokens": 1},
|
||||
},
|
||||
)
|
||||
)
|
||||
body = {"model": "gpt-4o", "messages": PROMPT}
|
||||
with TestClient(_app(optimize=False)) as client:
|
||||
for headers in ({"x-headroom-session-id": "h-1"}, {"x-headroom-session-id": "h-2"}, {}):
|
||||
r = client.post(
|
||||
"/v1/chat/completions",
|
||||
json=body,
|
||||
headers={"authorization": "Bearer sk-t", **headers},
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
one, two, none = recorder.seen
|
||||
assert one is not None and two not in (None, one)
|
||||
assert none is None
|
||||
Reference in New Issue
Block a user