mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(ccr): buffer a stream only for a marker new this turn (#4607)
## Description The Anthropic handler flips a streaming request to the buffered `stream:false` path when the outgoing body carries a redeemable CCR marker (#3092). A marker stays in the conversation history for the rest of the session, so once one tool result was compressed, every later turn carried an owned marker and was buffered. On a buffered turn the client receives nothing until the whole generation finishes. The report shows the effect: 71% of Opus turns buffered (51% in a second report on Linux), zero retrievals, the buffered share climbing from 0% to 90% with proxy uptime, and time to first byte rising from about 1 s to as much as 37 s. This PR keeps buffering only for turns that show the model a marker it has not seen yet, meaning one absent from what this session forwarded last turn. Turns whose markers were all forwarded earlier stream. The model can still ask for one of those originals, and the client never declared `headroom_retrieve`, so on those turns the stream withholds the call, the proxy serves it from the compression store, and the continuation round streams into the same client message. This reuses the filter and continuation-round machinery the proxy already uses for its memory tools (#2195). Closes #3587 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/proxy/handlers/anthropic.py`: - `_outgoing_redeemable_markers` returns `(any_owned, new_owned)`. "New" is decided against the session's last forwarded messages (via the existing `has_new_ccr_markers`), not by message position, so a marker that token mode adds to an earlier tool result on this request still counts as new. With no forwarded history, every owned marker counts as new. `_outgoing_body_has_redeemable_marker` returns the second value. - A new owned marker buffers, as on `main`. An owned marker that was already forwarded streams with `serve_ccr_retrieval=True`. No owned marker streams as before (#3071). - `headroom/proxy/memory_tool_stream.py`: `MemoryToolStreamFilter(hold_tool_use=True)` holds every block from the round's first `tool_use` until the caller decides (`release_held(drop_tool_use=...)`; `closing_frames` releases them). Text before the first tool call still streams live. Off by default, so the memory-tool behavior is unchanged. - `headroom/proxy/handlers/streaming.py`: `_stream_response(serve_ccr_retrieval=...)` adds `headroom_retrieve` to the withheld tool names and holds client tool calls. `_continue_memory_tool_stream(serve_retrieval=...)` serves withheld retrievals and continues the round. A round that retrieved alongside a client tool is continued without the unrun client call, for the model to re-issue with the content in hand. That is what `handle_response` already does on the buffered path. - `headroom/ccr/response_handler.py`: `serve_withheld_retrievals` runs withheld calls through the existing `_execute_retrieval` and counts them in `ccr_retrievals`. - Tests: - New `tests/test_ccr_stream_retrieval.py`. Two turns of one session through the real proxy app: turn 1 introduces an owned marker and buffers; on turn 2 the upstream asks for that original. - The original is served server-side, the answer reaches the client, and no `headroom_retrieve` appears in the client's SSE. - The same with a retrieval mixed with a client-owned tool, in both orders: only the model's re-issued call reaches the client. - An old-marker turn without a retrieval streams byte-identical to the upstream. - Filter unit tests for hold mode. - `tests/test_ccr_buffered_stream_signed_thinking.py`: two-turn tests for the gate (a marker already forwarded does not buffer; a marker first shown in an earlier message does). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`): see Additional Notes - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output The new streaming-retrieval tests on the previous head of this PR (64164fc2), which reproduce the review finding: ```text E assert 'headroom_retrieve' not in ['headroom_retrieve'] FAILED tests/test_ccr_stream_retrieval.py::test_later_turn_retrieval_is_served_by_the_proxy FAILED tests/test_ccr_stream_retrieval.py::test_retrieval_mixed_with_a_client_tool[True] FAILED tests/test_ccr_stream_retrieval.py::test_retrieval_mixed_with_a_client_tool[False] 3 failed, 1 passed ``` At this head (d10a7446): ```text $ pytest tests/test_ccr_stream_retrieval.py tests/test_ccr_buffered_stream_signed_thinking.py tests/test_memory_tool_stream.py -q 79 passed $ pytest <every test file with ccr, anthropic, stream, buffer, memory or proxy in its name> tests/test_proxy -m "not live" -q 17 failed, 4504 passed, 129 skipped $ ruff check headroom tests/test_ccr_stream_retrieval.py && ruff format --check headroom tests/test_ccr_stream_retrieval.py All checks passed! ``` The 17 failures are in `tests/test_proxy_uds.py`, `tests/test_upstream_guard.py` and `tests/test_ccr_batch_processor.py`. The identical set fails on `main` (fc1ba891) in my environment, so they are environment-related and not from this change. ## Real Behavior Proof - Environment: macOS (Darwin 24.6.0, arm64), Python 3.12.3. The real proxy app served by uvicorn in-process, cache mode, CCR on, `headroom_retrieve` resident. Only the upstream model is faked, at the HTTP client the proxy uses, so the proxy's own buffering, streaming filter and continuation rounds all run. A streamed answer is 100 text tokens at 15 ms each; a buffered answer arrives after 1.5 s; on a retrieval turn the model writes 5 tokens, calls `headroom_retrieve` for an earlier marker, then writes the 100-token answer once the original comes back. - Exact command / steps: replay a 30-turn agent session through a streaming HTTP client. New owned markers appear on turns 1, 8, 15 and 22, every turn carries the earlier ones, and on turns 4, 11, 18 and 26 the model retrieves an earlier original. Per turn, record buffered or streamed, time to the first text token, time to the first answer token, whether `headroom_retrieve` reached the client, and whether the answer arrived. Two alternating rounds each of `main` (fc1ba891), the previous head of this PR (64164fc2) and this head (d10a7446). - Observed result: `main` buffered 30 of 30 turns (time-to-first-token p50 1,516 ms) and served all 8 retrievals; the previous head buffered 4 turns (p50 24 ms) but leaked all 8 retrievals to the client, which never got an answer on those turns; this head buffers 4 turns (p50 24 ms), serves all 8 retrievals server-side, leaks none, and the answer on a retrieval turn starts at 105 ms. Full numbers below. | Build | Turns buffered | TTFT p50, all turns | Retrievals served | Leaked to client | Retrieval turns answered | Answer starts (retrieval turns, p50) | |---|---|---|---|---|---|---| | `main` (fc1ba891) | 30 of 30 | 1,515.8 ms | 8 of 8 | 0 | 8 of 8 | 1,591.0 ms | | previous head (64164fc2) | 4 of 30 | 24.3 ms | 0 of 8 | 8 | 0 of 8 | — | | this head (d10a7446) | 4 of 30 | 23.9 ms | 8 of 8 | 0 | 8 of 8 | 105.3 ms | The two rounds of each build matched on every count. In this setup a buffered turn's first token arrives when the whole generation is done (1.5 s); with real replies of 3–37 s, as in the report, each avoided buffered turn saves correspondingly more. - Not tested: a live Anthropic upstream. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes. Turns whose redeemable markers were all forwarded on an earlier turn now stream instead of buffering, with `headroom_retrieve` withheld from the client and served by the proxy. On those turns, client tool-call blocks reach the client when the round ends rather than as they stream; text is unchanged. Turns that show the model a new marker buffer as before; turns with no owned marker stream as before. - Kill switch / disable path: none added. Existing CCR controls are unchanged. - Unsafe override required: no - Qualification impact: none - Rollback path: revert the two commits ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation: n/a, no user-facing docs describe this gate - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Screenshots (if applicable) n/a ## Additional Notes - Review follow-ups: - The first revision decided "new" by position;64164fc2decides it from what the session forwarded. -64164fc2let a later-turn retrieval reach the client;d10a7446serves it in the stream. The earlier "tradeoff" note no longer applies. - On a round that retrieved alongside a client tool, the client call is dropped and the model decides again with the content in hand, matching the buffered path. If the continuation round itself fails, the client receives the upstream error frame, as with the memory-tool continuation. - The issue also asks for a config switch to disable the buffered CCR path entirely. That is left out to keep this change focused. - `mypy headroom/proxy/handlers/anthropic.py` stops in my local environment on a `numpy` stub (`Type statement is only supported in Python 3.12 and greater`) before reaching project code. CI runs mypy. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -132,6 +132,23 @@ class CCRResponseHandler:
|
||||
self._retrieval_count = 0
|
||||
self._retrieval_count_lock = __import__("threading").Lock()
|
||||
|
||||
def serve_withheld_retrievals(self, tool_calls: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Run Anthropic ``headroom_retrieve`` calls withheld from a stream.
|
||||
|
||||
Returns one ``tool_result`` block per call that names a hash, in call
|
||||
order, for the continuation round the streaming path sends upstream.
|
||||
A call without a parsable hash gets no result, so the caller can tell
|
||||
the round cannot be replayed.
|
||||
"""
|
||||
ccr_calls, _ = self._parse_ccr_tool_calls({"content": tool_calls}, "anthropic")
|
||||
if not ccr_calls:
|
||||
return []
|
||||
with self._retrieval_count_lock:
|
||||
self._retrieval_count += len(ccr_calls)
|
||||
results = [self._execute_retrieval(call) for call in ccr_calls]
|
||||
message = self._create_tool_result_message(results, "anthropic")
|
||||
return list(message.get("content", []))
|
||||
|
||||
def has_ccr_tool_calls(
|
||||
self,
|
||||
response: dict[str, Any],
|
||||
|
||||
@@ -630,28 +630,41 @@ class AnthropicHandlerMixin:
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _outgoing_body_has_redeemable_marker(body: Any) -> bool:
|
||||
"""Does the body about to be sent carry a marker retrieval could expand?
|
||||
def _outgoing_redeemable_markers(
|
||||
body: Any,
|
||||
previous_forwarded_messages: list[dict[str, Any]] | None = None,
|
||||
) -> tuple[bool, bool]:
|
||||
"""Which owned markers does the outgoing body carry?
|
||||
|
||||
``headroom_retrieve`` exists only to expand a ``<<ccr:...>>`` marker, so
|
||||
a request carrying none cannot benefit from the buffered path (#3071).
|
||||
Returns ``(any_owned, new_owned)``. ``headroom_retrieve`` exists only to
|
||||
expand a ``<<ccr:...>>`` marker, so a request carrying none cannot use
|
||||
it (#3071).
|
||||
|
||||
``new_owned`` is a marker the model has not seen yet: one absent from
|
||||
what this session forwarded last turn. With no forwarded history to
|
||||
compare against, every owned marker counts as new. "New" is decided
|
||||
against the forwarded history, not by message position: token mode can
|
||||
compress an earlier tool result on this request, so a marker first
|
||||
seen now may sit anywhere in the history.
|
||||
|
||||
Ownership is verified rather than shape-matched: the marker shape is not
|
||||
unique to Headroom, and adopting another context tool's hash would send
|
||||
the model to an endpoint that is guaranteed to miss (#2836). A hash that
|
||||
survives ``verify_ownership`` is redeemable right now.
|
||||
|
||||
Errors are swallowed deliberately and answered ``True``. This gates a
|
||||
wire-format decision, and the safe direction on an unexpected message
|
||||
shape is the long-standing buffered behavior, not a silent change.
|
||||
Errors are swallowed deliberately and answered ``(True, True)``. This
|
||||
gates a wire-format decision, and the safe direction on an unexpected
|
||||
message shape is the long-standing buffered behavior, not a silent
|
||||
change.
|
||||
"""
|
||||
if not isinstance(body, dict):
|
||||
return True
|
||||
return True, True
|
||||
messages = body.get("messages")
|
||||
if not isinstance(messages, list) or not messages:
|
||||
return False
|
||||
return False, False
|
||||
try:
|
||||
from headroom.ccr.tool_injection import CCRToolInjector
|
||||
from headroom.proxy.helpers import has_new_ccr_markers
|
||||
|
||||
probe = CCRToolInjector(
|
||||
provider="anthropic",
|
||||
@@ -660,12 +673,27 @@ class AnthropicHandlerMixin:
|
||||
)
|
||||
probe.scan_for_markers(messages)
|
||||
if not probe.detected_hashes:
|
||||
return False
|
||||
return False, False
|
||||
probe.verify_ownership()
|
||||
return bool(probe.detected_hashes)
|
||||
if not probe.detected_hashes:
|
||||
return False, False
|
||||
return True, has_new_ccr_markers(
|
||||
current_detected_hashes=list(probe.detected_hashes),
|
||||
previous_forwarded_messages=previous_forwarded_messages,
|
||||
provider="anthropic",
|
||||
)
|
||||
except Exception: # pragma: no cover - defensive
|
||||
logger.debug("CCR: marker probe failed; keeping the buffered path", exc_info=True)
|
||||
return True
|
||||
return True, True
|
||||
|
||||
@classmethod
|
||||
def _outgoing_body_has_redeemable_marker(
|
||||
cls,
|
||||
body: Any,
|
||||
previous_forwarded_messages: list[dict[str, Any]] | None = None,
|
||||
) -> bool:
|
||||
"""Does the outgoing body show the model an owned marker it has not seen?"""
|
||||
return cls._outgoing_redeemable_markers(body, previous_forwarded_messages)[1]
|
||||
|
||||
@staticmethod
|
||||
def _extract_anthropic_cache_ttl_metrics(usage: dict[str, Any] | None) -> tuple[int, int]:
|
||||
@@ -4340,19 +4368,34 @@ class AnthropicHandlerMixin:
|
||||
and ccr_response_handler_enabled
|
||||
and has_ccr_retrieve_tool(tools if tools is not None else body.get("tools"))
|
||||
)
|
||||
buffered_retrieval_can_help = (
|
||||
retrieve_tool_is_offered and self._outgoing_body_has_redeemable_marker(body)
|
||||
#
|
||||
# A marker the model already saw on an earlier turn can still be
|
||||
# retrieved, so it does not justify buffering every later turn
|
||||
# (#3587), but it does mean a retrieval call may come. Those
|
||||
# turns stream with ``headroom_retrieve`` withheld from the
|
||||
# client and served by the proxy in a continuation round.
|
||||
owned_marker, new_owned_marker = (
|
||||
self._outgoing_redeemable_markers(body, previous_forwarded_messages)
|
||||
if retrieve_tool_is_offered
|
||||
else (False, False)
|
||||
)
|
||||
wants_buffered_stream_ccr = bool(buffered_retrieval_can_help)
|
||||
if retrieve_tool_is_offered and not buffered_retrieval_can_help:
|
||||
wants_buffered_stream_ccr = bool(new_owned_marker)
|
||||
if retrieve_tool_is_offered and not owned_marker:
|
||||
logger.info(
|
||||
f"[{request_id}] CCR: headroom_retrieve is resident but this "
|
||||
"request carries no redeemable marker, so server-side "
|
||||
"retrieval cannot fire; keeping the streaming path (#3071)"
|
||||
)
|
||||
elif retrieve_tool_is_offered and not new_owned_marker:
|
||||
logger.info(
|
||||
f"[{request_id}] CCR: every redeemable marker was already "
|
||||
"forwarded; streaming with headroom_retrieve served by the "
|
||||
"proxy (#3587)"
|
||||
)
|
||||
buffered_stream_ccr = (
|
||||
wants_buffered_stream_ccr and not outbound_locked_to_client_bytes
|
||||
)
|
||||
serve_ccr_retrieval_in_stream = bool(owned_marker) and not buffered_stream_ccr
|
||||
if wants_buffered_stream_ccr and outbound_locked_to_client_bytes:
|
||||
logger.info(
|
||||
f"[{request_id}] CCR: signed thinking blocks force byte-faithful "
|
||||
@@ -4526,6 +4569,7 @@ class AnthropicHandlerMixin:
|
||||
session_key=session_key,
|
||||
server_memory_tool_names=server_memory_tool_names,
|
||||
client_beta=client_anthropic_beta or "",
|
||||
serve_ccr_retrieval=serve_ccr_retrieval_in_stream,
|
||||
)
|
||||
else:
|
||||
# Whatever set it — the client's own ``stream: false`` or
|
||||
|
||||
@@ -520,17 +520,26 @@ class StreamingMixin:
|
||||
server_memory_tool_names: frozenset[str],
|
||||
stream_state: dict[str, Any],
|
||||
request_id: str,
|
||||
serve_retrieval: bool = False,
|
||||
) -> AsyncIterator[bytes]:
|
||||
"""Execute withheld memory tool calls and stream continuation rounds.
|
||||
"""Execute withheld proxy tool calls and stream continuation rounds.
|
||||
|
||||
``memory_filter`` has forwarded one upstream round to the client minus
|
||||
the proxy-owned memory tool calls and the message tail. When the round
|
||||
stopped on memory tool calls alone, run them, send the results upstream
|
||||
the proxy-owned tool calls and the message tail. When the round
|
||||
stopped on proxy tool calls alone, run them, send the results upstream
|
||||
as the next turn, and stream that round's blocks into the same client
|
||||
message. Otherwise end the client message. A round that also called a
|
||||
client tool still has its memory calls executed, but the turn goes back
|
||||
to the client, which cannot carry the memory results.
|
||||
|
||||
With ``serve_retrieval``, ``headroom_retrieve`` is a proxy tool too: the
|
||||
client never declared it, so a call that reached it would fail (#3587).
|
||||
The filter then holds client tool calls until the round ends, and a
|
||||
round that retrieved alongside a client tool is continued without the
|
||||
unrun client calls, for the model to re-issue with the content in
|
||||
hand. That is what the buffered path does with the same round.
|
||||
"""
|
||||
from headroom.ccr.tool_injection import CCR_TOOL_NAME
|
||||
from headroom.proxy.helpers import MAX_SSE_BUFFER_SIZE
|
||||
|
||||
try:
|
||||
@@ -541,18 +550,47 @@ class StreamingMixin:
|
||||
if not isinstance(messages, list):
|
||||
messages = None
|
||||
headers = {k: v for k, v in outbound_headers.items() if k.lower() != "content-length"}
|
||||
proxy_tool_names = server_memory_tool_names | (
|
||||
frozenset({CCR_TOOL_NAME}) if serve_retrieval else frozenset()
|
||||
)
|
||||
|
||||
def is_memory_call(block: Any) -> bool:
|
||||
def is_proxy_call(block: Any) -> bool:
|
||||
return (
|
||||
isinstance(block, dict)
|
||||
and block.get("type") == "tool_use"
|
||||
and block.get("name") in server_memory_tool_names
|
||||
and block.get("name") in proxy_tool_names
|
||||
)
|
||||
|
||||
def is_client_call(block: Any) -> bool:
|
||||
return (
|
||||
isinstance(block, dict)
|
||||
and block.get("type") == "tool_use"
|
||||
and not is_proxy_call(block)
|
||||
)
|
||||
|
||||
rounds = 0
|
||||
# What the client received across rounds; the prefix tracker must see
|
||||
# this, not round one, because it is what the client sends back.
|
||||
visible_content: list[Any] = []
|
||||
|
||||
def record_visible(
|
||||
round_response: dict[str, Any] | None,
|
||||
round_content: list[Any],
|
||||
*,
|
||||
dropped_client_calls: bool,
|
||||
) -> None:
|
||||
if round_response is None:
|
||||
return
|
||||
visible_content.extend(
|
||||
block
|
||||
for block in round_content
|
||||
if not is_proxy_call(block) and not (dropped_client_calls and is_client_call(block))
|
||||
)
|
||||
stream_state["memory_client_message"] = {
|
||||
**round_response,
|
||||
"content": list(visible_content),
|
||||
}
|
||||
|
||||
# Usage of rounds whose message_delta the client never receives.
|
||||
prior_usage: dict[str, int] = {}
|
||||
while True:
|
||||
@@ -560,13 +598,9 @@ class StreamingMixin:
|
||||
if response is not None:
|
||||
round_content = response.get("content")
|
||||
content = round_content if isinstance(round_content, list) else []
|
||||
visible_content.extend(block for block in content if not is_memory_call(block))
|
||||
stream_state["memory_client_message"] = {
|
||||
**response,
|
||||
"content": list(visible_content),
|
||||
}
|
||||
|
||||
if not memory_filter.hid_tool_calls:
|
||||
record_visible(response, content, dropped_client_calls=False)
|
||||
for frame in memory_filter.closing_frames(prior_usage=prior_usage):
|
||||
yield frame
|
||||
return
|
||||
@@ -574,7 +608,11 @@ class StreamingMixin:
|
||||
# Run the calls the filter recorded, so they are not lost when the
|
||||
# round could not be reconstructed. Only the proxy's own calls: a
|
||||
# memory-named tool the client declared is the client's to run.
|
||||
memory_calls = memory_filter.hidden_calls()
|
||||
hidden_calls = memory_filter.hidden_calls()
|
||||
memory_calls = [c for c in hidden_calls if c.get("name") in server_memory_tool_names]
|
||||
retrieval_calls = [
|
||||
c for c in hidden_calls if serve_retrieval and c.get("name") == CCR_TOOL_NAME
|
||||
]
|
||||
tool_results: list[dict[str, Any]] = []
|
||||
if memory_calls and memory_user_id is not None and self.memory_handler is not None:
|
||||
tool_results = await self.memory_handler.handle_memory_tool_calls(
|
||||
@@ -583,24 +621,38 @@ class StreamingMixin:
|
||||
"anthropic",
|
||||
request_context=memory_request_ctx,
|
||||
)
|
||||
logger.info(
|
||||
f"[{request_id}] Memory: Executed {len(tool_results)}/"
|
||||
f"{len(memory_filter.hidden_tool_names)} proxy-handled tool call(s) "
|
||||
"withheld from the SSE stream"
|
||||
)
|
||||
logger.info(
|
||||
f"[{request_id}] Memory: Executed {len(tool_results)}/"
|
||||
f"{len(memory_calls)} proxy-handled tool call(s) "
|
||||
"withheld from the SSE stream"
|
||||
)
|
||||
ccr_handler = getattr(self, "ccr_response_handler", None)
|
||||
if retrieval_calls and ccr_handler is not None:
|
||||
retrieval_results = ccr_handler.serve_withheld_retrievals(retrieval_calls)
|
||||
tool_results = [*tool_results, *retrieval_results]
|
||||
logger.info(
|
||||
f"[{request_id}] CCR: Served {len(retrieval_results)}/"
|
||||
f"{len(retrieval_calls)} headroom_retrieve call(s) withheld from "
|
||||
"the SSE stream"
|
||||
)
|
||||
|
||||
replayable = response is not None and len(tool_results) == sum(
|
||||
1 for block in content if is_memory_call(block)
|
||||
1 for block in content if is_proxy_call(block)
|
||||
)
|
||||
# Unrun client calls can be dropped only while the client has not
|
||||
# seen them, and only for a retrieval: memory results alone do not
|
||||
# give the model a reason to decide again.
|
||||
drop_client_calls = bool(retrieval_calls) and memory_filter.holds_client_tool_use
|
||||
can_continue = (
|
||||
memory_filter.stop_reason == "tool_use"
|
||||
and not memory_filter.visible_tool_use
|
||||
and (drop_client_calls or not memory_filter.holds_client_tool_use)
|
||||
and messages is not None
|
||||
and replayable
|
||||
and rounds < self._MEMORY_CONTINUATION_MAX_ROUNDS
|
||||
)
|
||||
if not can_continue:
|
||||
if memory_filter.visible_tool_use:
|
||||
if memory_filter.visible_tool_use or memory_filter.holds_client_tool_use:
|
||||
logger.info(
|
||||
f"[{request_id}] Memory: Round also called a client tool; "
|
||||
"returning the turn to the client"
|
||||
@@ -608,18 +660,31 @@ class StreamingMixin:
|
||||
elif rounds >= self._MEMORY_CONTINUATION_MAX_ROUNDS:
|
||||
logger.warning(
|
||||
f"[{request_id}] Memory: Stopped after {rounds} continuation "
|
||||
"round(s) that only called memory tools"
|
||||
"round(s) that only called proxy tools"
|
||||
)
|
||||
elif not replayable:
|
||||
logger.warning(
|
||||
f"[{request_id}] Memory: Could not rebuild the round for a "
|
||||
"continuation; ending the turn after running its memory calls"
|
||||
"continuation; ending the turn after running its proxy calls"
|
||||
)
|
||||
record_visible(response, content, dropped_client_calls=False)
|
||||
for frame in memory_filter.closing_frames(prior_usage=prior_usage):
|
||||
yield frame
|
||||
return
|
||||
|
||||
assert response is not None and messages is not None and isinstance(base_body, dict)
|
||||
record_visible(response, content, dropped_client_calls=drop_client_calls)
|
||||
# Blocks held after the first tool call (rarely any but the calls
|
||||
# themselves) still belong to the client's message.
|
||||
for frame in memory_filter.release_held(drop_tool_use=drop_client_calls):
|
||||
yield frame
|
||||
if drop_client_calls:
|
||||
logger.info(
|
||||
f"[{request_id}] CCR: Round called headroom_retrieve alongside a "
|
||||
"client tool; serving the retrieval and dropping the unrun client "
|
||||
"call(s) for the model to re-issue"
|
||||
)
|
||||
content = [block for block in content if not is_client_call(block)]
|
||||
round_usage = response.get("usage")
|
||||
if isinstance(round_usage, dict):
|
||||
for key in _ROUND_USAGE_KEYS:
|
||||
@@ -647,9 +712,10 @@ class StreamingMixin:
|
||||
yield _upstream_error_frame(error_bytes, request_id)
|
||||
return
|
||||
memory_filter = MemoryToolStreamFilter(
|
||||
server_memory_tool_names,
|
||||
proxy_tool_names,
|
||||
index_offset=memory_filter.next_index,
|
||||
forward_message_start=False,
|
||||
hold_tool_use=serve_retrieval,
|
||||
)
|
||||
# Same cap as the first round's buffer; past it the round is
|
||||
# still streamed, just not rebuilt for a further continuation.
|
||||
@@ -1082,6 +1148,7 @@ class StreamingMixin:
|
||||
savings_measure: ItemMeasure | None = None,
|
||||
server_memory_tool_names: frozenset[str] | None = None,
|
||||
client_beta: str | None = None,
|
||||
serve_ccr_retrieval: bool = False,
|
||||
) -> Response | StreamingResponse:
|
||||
"""Stream response with metrics tracking and memory tool handling.
|
||||
|
||||
@@ -1130,6 +1197,7 @@ class StreamingMixin:
|
||||
savings_measure=savings_measure,
|
||||
server_memory_tool_names=server_memory_tool_names,
|
||||
client_beta=client_beta,
|
||||
serve_ccr_retrieval=serve_ccr_retrieval,
|
||||
)
|
||||
except (Exception, asyncio.CancelledError):
|
||||
self._cleanup_mid_turn_stream(session_key)
|
||||
@@ -1165,6 +1233,7 @@ class StreamingMixin:
|
||||
savings_measure: ItemMeasure | None = None,
|
||||
server_memory_tool_names: frozenset[str] | None = None,
|
||||
client_beta: str | None = None,
|
||||
serve_ccr_retrieval: bool = False,
|
||||
) -> Response | StreamingResponse:
|
||||
"""Actual streaming implementation, guarded by _stream_response's cleanup wrapper."""
|
||||
from fastapi.responses import Response, StreamingResponse
|
||||
@@ -1339,6 +1408,9 @@ class StreamingMixin:
|
||||
and self.memory_handler is not None
|
||||
and provider == "anthropic"
|
||||
)
|
||||
# Withholding headroom_retrieve needs the Anthropic SSE shape the
|
||||
# filter and the continuation round are written for.
|
||||
serve_ccr_retrieval = serve_ccr_retrieval and provider == "anthropic"
|
||||
|
||||
# Open connection before generator to capture upstream response headers
|
||||
# (needed to forward ratelimit headers to the client via StreamingResponse)
|
||||
@@ -1672,11 +1744,23 @@ class StreamingMixin:
|
||||
pending_messages: list[dict] = []
|
||||
# Proxy-injected memory tools are invisible to the client, so their
|
||||
# tool_use blocks must never reach it (GH #2195).
|
||||
from headroom.ccr.tool_injection import CCR_TOOL_NAME
|
||||
|
||||
# Memory tools are withheld only when this request can run them;
|
||||
# the continuation must classify calls exactly as the filter did.
|
||||
filter_memory_tool_names = (
|
||||
server_memory_tool_names or frozenset() if memory_enabled else frozenset()
|
||||
)
|
||||
proxy_tool_names = filter_memory_tool_names | (
|
||||
frozenset({CCR_TOOL_NAME}) if serve_ccr_retrieval else frozenset()
|
||||
)
|
||||
memory_filter = (
|
||||
MemoryToolStreamFilter(server_memory_tool_names)
|
||||
if memory_enabled and server_memory_tool_names
|
||||
MemoryToolStreamFilter(proxy_tool_names, hold_tool_use=serve_ccr_retrieval)
|
||||
if proxy_tool_names
|
||||
else None
|
||||
)
|
||||
# A withheld retrieval needs the round rebuilt for its continuation.
|
||||
retrieval_tracking = serve_ccr_retrieval
|
||||
|
||||
def _client_frame(frame: bytes) -> bytes:
|
||||
"""``_guard_client_bytes``, noting when the terminal event goes out."""
|
||||
@@ -1743,6 +1827,7 @@ class StreamingMixin:
|
||||
_track_sse = (
|
||||
_codex_wire_debug
|
||||
or memory_enabled
|
||||
or retrieval_tracking
|
||||
or (prefix_tracker is not None and provider == "anthropic")
|
||||
)
|
||||
if _track_sse:
|
||||
@@ -1755,6 +1840,7 @@ class StreamingMixin:
|
||||
"disabling memory detection for this request"
|
||||
)
|
||||
memory_enabled = False
|
||||
retrieval_tracking = False
|
||||
|
||||
# Parse complete SSE events from buffer. A failure (e.g.
|
||||
# invalid UTF-8) is raised once the chunk is forwarded:
|
||||
@@ -1815,7 +1901,7 @@ class StreamingMixin:
|
||||
# be an upstream protocol violation — surface loudly.
|
||||
full_sse_data: str = full_sse_bytes.decode("utf-8") if full_sse_bytes else ""
|
||||
|
||||
if memory_enabled and full_sse_data:
|
||||
if (memory_enabled or retrieval_tracking) and full_sse_data:
|
||||
# Check for Claude Code credential error
|
||||
if "only authorized for use with Claude Code" in full_sse_data:
|
||||
logger.warning(
|
||||
@@ -1845,9 +1931,10 @@ class StreamingMixin:
|
||||
outbound_bytes=outbound_bytes,
|
||||
memory_user_id=memory_user_id,
|
||||
memory_request_ctx=memory_request_ctx,
|
||||
server_memory_tool_names=server_memory_tool_names or frozenset(),
|
||||
server_memory_tool_names=filter_memory_tool_names,
|
||||
stream_state=stream_state,
|
||||
request_id=request_id,
|
||||
serve_retrieval=serve_ccr_retrieval,
|
||||
):
|
||||
guarded_frame = _client_frame(frame)
|
||||
if guarded_frame:
|
||||
@@ -1856,8 +1943,10 @@ class StreamingMixin:
|
||||
parsed_response = (
|
||||
stream_state.pop("memory_client_message", None) or parsed_response
|
||||
)
|
||||
elif parsed_response and self.memory_handler.has_memory_tool_calls(
|
||||
parsed_response, provider
|
||||
elif (
|
||||
memory_enabled
|
||||
and parsed_response
|
||||
and self.memory_handler.has_memory_tool_calls(parsed_response, provider)
|
||||
):
|
||||
logger.info(
|
||||
f"[{request_id}] Memory: Detected tool calls in streaming response"
|
||||
@@ -1890,9 +1979,10 @@ class StreamingMixin:
|
||||
outbound_bytes=outbound_bytes,
|
||||
memory_user_id=memory_user_id,
|
||||
memory_request_ctx=memory_request_ctx,
|
||||
server_memory_tool_names=server_memory_tool_names or frozenset(),
|
||||
server_memory_tool_names=filter_memory_tool_names,
|
||||
stream_state=stream_state,
|
||||
request_id=request_id,
|
||||
serve_retrieval=serve_ccr_retrieval,
|
||||
):
|
||||
guarded_frame = _client_frame(frame)
|
||||
if guarded_frame:
|
||||
|
||||
@@ -90,6 +90,12 @@ class MemoryToolStreamFilter:
|
||||
extend the message the client already opened.
|
||||
max_retained_bytes: Limit on the bytes held back at any time;
|
||||
``DEFAULT_MAX_RETAINED_BYTES`` when None.
|
||||
hold_tool_use: Hold every block from the round's first ``tool_use``
|
||||
on, until the caller decides with ``release_held``. A withheld
|
||||
``headroom_retrieve`` alongside a client tool is served by
|
||||
dropping the client call for the model to re-issue, which is
|
||||
only possible while that call has not reached the client.
|
||||
Text before the first tool call still streams live.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -99,8 +105,17 @@ class MemoryToolStreamFilter:
|
||||
index_offset: int = 0,
|
||||
forward_message_start: bool = True,
|
||||
max_retained_bytes: int | None = None,
|
||||
hold_tool_use: bool = False,
|
||||
) -> None:
|
||||
self._tool_names = frozenset(tool_names)
|
||||
self._hold_tool_use = hold_tool_use
|
||||
self._holding = False
|
||||
# Frames held from the first tool_use on, with their upstream index;
|
||||
# client indices are assigned when they are released.
|
||||
self._held: list[tuple[str, dict[str, Any], bytes, int]] = []
|
||||
self._held_bytes = 0
|
||||
# Upstream indices of held client tool_use blocks.
|
||||
self._held_tool_use: set[int] = set()
|
||||
self._max_retained_bytes = (
|
||||
DEFAULT_MAX_RETAINED_BYTES if max_retained_bytes is None else max_retained_bytes
|
||||
)
|
||||
@@ -130,6 +145,39 @@ class MemoryToolStreamFilter:
|
||||
def hidden_tool_names(self) -> list[str]:
|
||||
return [block["name"] for block, _ in self._hidden.values()]
|
||||
|
||||
@property
|
||||
def holds_client_tool_use(self) -> bool:
|
||||
"""A client ``tool_use`` block is held and has not reached the client."""
|
||||
return bool(self._held_tool_use)
|
||||
|
||||
def release_held(self, *, drop_tool_use: bool = False) -> list[bytes]:
|
||||
"""Return the held frames, re-indexed, for the client.
|
||||
|
||||
``drop_tool_use`` leaves out the held client ``tool_use`` blocks: the
|
||||
round goes back to the model with the proxy's results instead, and
|
||||
the model re-issues whatever it still needs.
|
||||
"""
|
||||
frames: list[bytes] = []
|
||||
dropped = self._held_tool_use if drop_tool_use else set()
|
||||
for event_name, payload, frame, upstream_index in self._held:
|
||||
if upstream_index in dropped:
|
||||
continue
|
||||
if event_name == "content_block_start":
|
||||
block = payload.get("content_block")
|
||||
if isinstance(block, dict) and block.get("type") == "tool_use":
|
||||
self.visible_tool_use = True
|
||||
self._index_map[upstream_index] = self.next_index
|
||||
self.next_index += 1
|
||||
client_index = self._index_map.get(upstream_index, upstream_index)
|
||||
if client_index == upstream_index:
|
||||
frames.append(frame)
|
||||
else:
|
||||
frames.append(_render_frame(event_name, {**payload, "index": client_index}))
|
||||
self._held.clear()
|
||||
self._held_bytes = 0
|
||||
self._held_tool_use.clear()
|
||||
return frames
|
||||
|
||||
def hidden_calls(self) -> list[dict[str, Any]]:
|
||||
"""The withheld ``tool_use`` blocks that finished with a complete input.
|
||||
|
||||
@@ -181,7 +229,12 @@ class MemoryToolStreamFilter:
|
||||
forwarded = self._route(frame)
|
||||
if forwarded is not None:
|
||||
out.append(forwarded)
|
||||
retained = len(self._buffer) + self._tail_bytes + sum(self._hidden_bytes.values())
|
||||
retained = (
|
||||
len(self._buffer)
|
||||
+ self._tail_bytes
|
||||
+ self._held_bytes
|
||||
+ sum(self._hidden_bytes.values())
|
||||
)
|
||||
if retained > self._max_retained_bytes:
|
||||
self._buffer.clear()
|
||||
self._hidden.clear()
|
||||
@@ -189,6 +242,9 @@ class MemoryToolStreamFilter:
|
||||
self._hidden_stopped.clear()
|
||||
self._tail.clear()
|
||||
self._tail_bytes = 0
|
||||
self._held.clear()
|
||||
self._held_bytes = 0
|
||||
self._held_tool_use.clear()
|
||||
raise MemoryToolStreamOverflowError(
|
||||
f"memory tool stream filter would retain {retained} bytes "
|
||||
f"(limit {self._max_retained_bytes})"
|
||||
@@ -232,11 +288,18 @@ class MemoryToolStreamFilter:
|
||||
self._hidden[upstream_index] = (block, [])
|
||||
self._hidden_bytes[upstream_index] = len(frame)
|
||||
self._hidden_stopped.discard(upstream_index)
|
||||
self._holding = self._hold_tool_use
|
||||
return None
|
||||
if isinstance(block, dict) and block.get("type") == "tool_use":
|
||||
self.visible_tool_use = True
|
||||
self._index_map[upstream_index] = self.next_index
|
||||
self.next_index += 1
|
||||
is_tool_use = isinstance(block, dict) and block.get("type") == "tool_use"
|
||||
if self._hold_tool_use and (self._holding or is_tool_use):
|
||||
self._holding = True
|
||||
if is_tool_use:
|
||||
self._held_tool_use.add(upstream_index)
|
||||
else:
|
||||
if is_tool_use:
|
||||
self.visible_tool_use = True
|
||||
self._index_map[upstream_index] = self.next_index
|
||||
self.next_index += 1
|
||||
|
||||
if upstream_index in self._hidden:
|
||||
delta = payload.get("delta")
|
||||
@@ -252,17 +315,26 @@ class MemoryToolStreamFilter:
|
||||
elif event_name == "content_block_stop":
|
||||
self._hidden_stopped.add(upstream_index)
|
||||
return None
|
||||
if self._holding and upstream_index not in self._index_map:
|
||||
self._held.append((event_name, payload, frame, upstream_index))
|
||||
self._held_bytes += len(frame)
|
||||
return None
|
||||
client_index = self._index_map.get(upstream_index, upstream_index)
|
||||
if client_index == upstream_index:
|
||||
return frame
|
||||
return _render_frame(event_name, {**payload, "index": client_index})
|
||||
|
||||
def closing_frames(self, *, prior_usage: dict[str, int] | None = None) -> list[bytes]:
|
||||
"""End the client message after this round without a continuation."""
|
||||
"""End the client message after this round without a continuation.
|
||||
|
||||
Held frames are released first, client tool calls included: with no
|
||||
continuation, the round is the model's answer as it stands.
|
||||
"""
|
||||
held = self.release_held()
|
||||
ends_on_hidden_call = (
|
||||
self.hid_tool_calls and not self.visible_tool_use and self.stop_reason == "tool_use"
|
||||
)
|
||||
return self.tail_frames(
|
||||
return held + self.tail_frames(
|
||||
stop_reason="end_turn" if ends_on_hidden_call else None,
|
||||
prior_usage=prior_usage,
|
||||
)
|
||||
|
||||
@@ -90,11 +90,13 @@ def ccr_marker() -> str:
|
||||
|
||||
|
||||
def _body(*, with_thinking: bool, marker: str | None = None) -> dict:
|
||||
first = "hi" if marker is None else f"hi — earlier output is at <<ccr:{marker}>>"
|
||||
messages: list[dict] = [{"role": "user", "content": first}]
|
||||
# The marker rides in the latest user turn: only content new this turn
|
||||
# gates buffering (#3587).
|
||||
latest = "hi" if marker is None else f"hi — earlier output is at <<ccr:{marker}>>"
|
||||
messages: list[dict] = [{"role": "user", "content": "hi" if with_thinking else latest}]
|
||||
if with_thinking:
|
||||
messages.append(SIGNED_THINKING_TURN)
|
||||
messages.append({"role": "user", "content": "continue"})
|
||||
messages.append({"role": "user", "content": latest})
|
||||
return {
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"max_tokens": 64,
|
||||
@@ -287,6 +289,116 @@ def test_buffering_is_gated_on_a_redeemable_marker(
|
||||
assert calls["stream_body"]["stream"] is True
|
||||
|
||||
|
||||
def _owned_marker(content: str) -> str:
|
||||
"""Store one more original in the (fixture-reset) store; return its hash."""
|
||||
from headroom.cache.compression_store import get_compression_store
|
||||
|
||||
return get_compression_store().store(
|
||||
content,
|
||||
"<<ccr:placeholder>>",
|
||||
original_tokens=100,
|
||||
compressed_tokens=5,
|
||||
tool_name="Read",
|
||||
)
|
||||
|
||||
|
||||
def _tool_turn(tool_use_id: str, output: str) -> list[dict]:
|
||||
return [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [{"type": "tool_use", "id": tool_use_id, "name": "Read", "input": {}}],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "tool_result", "tool_use_id": tool_use_id, "content": output}],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _agent_body(messages: list[dict]) -> dict:
|
||||
return {
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"max_tokens": 64,
|
||||
"stream": True,
|
||||
"tools": [RETRIEVE_TOOL],
|
||||
"messages": messages,
|
||||
}
|
||||
|
||||
|
||||
def _replay_session(turns: list[list[dict]]) -> list[str]:
|
||||
"""Send each message list as one turn of the same session; report the path taken."""
|
||||
paths: list[str] = []
|
||||
|
||||
async def fake_stream_response(url, headers, body, *args, **kwargs): # noqa: ANN001
|
||||
paths.append("streamed")
|
||||
return fastapi.responses.StreamingResponse(iter([SSE_BODY]), media_type="text/event-stream")
|
||||
|
||||
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
|
||||
paths.append("buffered")
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "msg_1",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
"content": [{"type": "text", "text": "ok"}],
|
||||
"stop_reason": "end_turn",
|
||||
"usage": {"input_tokens": 10, "output_tokens": 5},
|
||||
},
|
||||
headers={"content-type": "application/json"},
|
||||
)
|
||||
|
||||
app = create_app(_config())
|
||||
with TestClient(app) as client:
|
||||
client.app.state.proxy._stream_response = fake_stream_response
|
||||
client.app.state.proxy._retry_request = fake_retry
|
||||
for messages in turns:
|
||||
resp = client.post("/v1/messages", json=_agent_body(messages), headers=_headers())
|
||||
assert resp.status_code == 200, resp.text
|
||||
return paths
|
||||
|
||||
|
||||
def test_a_marker_already_forwarded_does_not_buffer_later_turns(ccr_marker: str) -> None:
|
||||
"""A marker carried forward in history does not make every later turn buffer.
|
||||
|
||||
The marker was in front of the model on the turn that introduced it, and
|
||||
that turn buffered. Buffering every later turn for it is #3587: the
|
||||
buffered share climbed toward 90% of turns with zero retrievals.
|
||||
"""
|
||||
first = [
|
||||
{"role": "user", "content": "read the logs"},
|
||||
*_tool_turn("tu_1", f"earlier output at <<ccr:{ccr_marker}>>"),
|
||||
]
|
||||
second = [*first, *_tool_turn("tu_2", "short")]
|
||||
|
||||
assert _replay_session([first, second]) == ["buffered", "streamed"]
|
||||
|
||||
|
||||
def test_a_marker_first_shown_in_an_earlier_message_still_buffers(ccr_marker: str) -> None:
|
||||
"""New to the model is what counts, not where the marker sits.
|
||||
|
||||
Token mode can compress an earlier tool result on a later request, so the
|
||||
model first sees that marker in the middle of the history. Retrieval has
|
||||
to be handled server-side there too, or the call reaches the client.
|
||||
"""
|
||||
newly_compressed = _owned_marker("a different, uncompressed tool output")
|
||||
first = [
|
||||
{"role": "user", "content": "read the logs"},
|
||||
*_tool_turn("tu_1", f"earlier output at <<ccr:{ccr_marker}>>"),
|
||||
]
|
||||
second = [
|
||||
{"role": "user", "content": "read the logs"},
|
||||
*_tool_turn(
|
||||
"tu_1",
|
||||
f"earlier output at <<ccr:{ccr_marker}>> and <<ccr:{newly_compressed}>>",
|
||||
),
|
||||
*_tool_turn("tu_2", "short"),
|
||||
]
|
||||
|
||||
assert _replay_session([first, second]) == ["buffered", "buffered"]
|
||||
|
||||
|
||||
def test_cache_hit_never_replays_a_foreign_content_type() -> None:
|
||||
"""A cache entry cannot hand a caller a wire format it did not ask for."""
|
||||
body = {
|
||||
|
||||
@@ -0,0 +1,320 @@
|
||||
"""``headroom_retrieve`` on a streamed turn is served by the proxy (#3587).
|
||||
|
||||
A turn whose redeemable markers were all forwarded on earlier turns streams
|
||||
instead of buffering, so the client gets the first token right away. The model
|
||||
can still ask for one of those originals. The client never declared
|
||||
``headroom_retrieve`` (a coding agent answers "No such tool available"), so
|
||||
the call must not reach it: the stream withholds it, the proxy serves it from
|
||||
the compression store, and the continuation round streams into the same client
|
||||
message.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
fastapi = pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from headroom.ccr.tool_injection import create_ccr_tool_definition # noqa: E402
|
||||
from headroom.proxy.memory_tool_stream import MemoryToolStreamFilter # noqa: E402
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
|
||||
RETRIEVE_ONLY = frozenset({"headroom_retrieve"})
|
||||
ORIGINAL = "line 1 of the original build log\nline 2: the error the model needs"
|
||||
|
||||
|
||||
def _frame(payload: dict[str, Any]) -> bytes:
|
||||
return f"event: {payload['type']}\ndata: {json.dumps(payload)}\n\n".encode()
|
||||
|
||||
|
||||
def _sse(blocks: list[dict[str, Any]], stop_reason: str) -> bytes:
|
||||
"""An Anthropic SSE response carrying ``blocks``."""
|
||||
frames = [
|
||||
_frame(
|
||||
{
|
||||
"type": "message_start",
|
||||
"message": {
|
||||
"id": "msg_1",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "sonnet-test",
|
||||
"content": [],
|
||||
"stop_reason": None,
|
||||
"usage": {"input_tokens": 10, "output_tokens": 1},
|
||||
},
|
||||
}
|
||||
)
|
||||
]
|
||||
for index, block in enumerate(blocks):
|
||||
if block["type"] == "text":
|
||||
start = {"type": "text", "text": ""}
|
||||
delta = {"type": "text_delta", "text": block["text"]}
|
||||
else:
|
||||
start = {**block, "input": {}}
|
||||
delta = {"type": "input_json_delta", "partial_json": json.dumps(block["input"])}
|
||||
frames.append(
|
||||
_frame({"type": "content_block_start", "index": index, "content_block": start})
|
||||
)
|
||||
frames.append(_frame({"type": "content_block_delta", "index": index, "delta": delta}))
|
||||
frames.append(_frame({"type": "content_block_stop", "index": index}))
|
||||
frames.append(
|
||||
_frame(
|
||||
{
|
||||
"type": "message_delta",
|
||||
"delta": {"stop_reason": stop_reason, "stop_sequence": None},
|
||||
"usage": {"output_tokens": 5},
|
||||
}
|
||||
)
|
||||
)
|
||||
frames.append(_frame({"type": "message_stop"}))
|
||||
return b"".join(frames)
|
||||
|
||||
|
||||
def _events(raw: bytes) -> list[dict[str, Any]]:
|
||||
return [
|
||||
json.loads(line[len("data: ") :])
|
||||
for line in raw.decode().splitlines()
|
||||
if line.startswith("data: ")
|
||||
]
|
||||
|
||||
|
||||
def _tool_names(events: list[dict[str, Any]]) -> list[str]:
|
||||
return [
|
||||
e["content_block"]["name"]
|
||||
for e in events
|
||||
if e["type"] == "content_block_start" and e["content_block"].get("type") == "tool_use"
|
||||
]
|
||||
|
||||
|
||||
TEXT = {"type": "text", "text": "Let me look at the earlier output."}
|
||||
BASH = {"type": "tool_use", "id": "toolu_bash", "name": "Bash", "input": {"command": "ls"}}
|
||||
|
||||
|
||||
def _retrieve(hash_key: str) -> dict[str, Any]:
|
||||
return {
|
||||
"type": "tool_use",
|
||||
"id": "toolu_ccr",
|
||||
"name": "headroom_retrieve",
|
||||
"input": {"hash": hash_key},
|
||||
}
|
||||
|
||||
|
||||
class TestHoldingFilter:
|
||||
def test_text_before_the_first_tool_call_streams_live(self) -> None:
|
||||
flt = MemoryToolStreamFilter(RETRIEVE_ONLY, hold_tool_use=True)
|
||||
forwarded = _events(b"".join(flt.feed(_sse([TEXT, BASH], "tool_use"))))
|
||||
assert [e["type"] for e in forwarded] == [
|
||||
"message_start",
|
||||
"content_block_start",
|
||||
"content_block_delta",
|
||||
"content_block_stop",
|
||||
]
|
||||
assert flt.holds_client_tool_use
|
||||
|
||||
def test_without_a_retrieval_the_round_is_byte_identical(self) -> None:
|
||||
raw = _sse([TEXT, BASH], "tool_use")
|
||||
flt = MemoryToolStreamFilter(RETRIEVE_ONLY, hold_tool_use=True)
|
||||
out = b"".join(flt.feed(raw)) + b"".join(flt.closing_frames())
|
||||
assert out == raw
|
||||
|
||||
def test_dropping_held_client_calls(self) -> None:
|
||||
flt = MemoryToolStreamFilter(RETRIEVE_ONLY, hold_tool_use=True)
|
||||
flt.feed(_sse([BASH, _retrieve("abc")], "tool_use"))
|
||||
assert flt.hidden_tool_names == ["headroom_retrieve"]
|
||||
assert flt.release_held(drop_tool_use=True) == []
|
||||
assert flt.next_index == 0
|
||||
assert not flt.visible_tool_use
|
||||
|
||||
|
||||
def _sse_response(raw: bytes) -> MagicMock:
|
||||
response = MagicMock()
|
||||
response.headers = httpx.Headers({"content-type": "text/event-stream"})
|
||||
response.status_code = 200
|
||||
|
||||
async def aiter_bytes(raw: bytes = raw):
|
||||
# Split mid-frame so the filter has to reassemble events.
|
||||
yield raw[:41]
|
||||
yield raw[41:]
|
||||
|
||||
response.aiter_bytes = aiter_bytes
|
||||
response.aclose = AsyncMock()
|
||||
return response
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def owned_marker():
|
||||
"""A marker the proxy owns, with its original in the compression store."""
|
||||
from headroom.cache.backends import InMemoryBackend
|
||||
from headroom.cache.compression_store import get_compression_store, reset_compression_store
|
||||
|
||||
reset_compression_store()
|
||||
store = get_compression_store(backend=InMemoryBackend())
|
||||
hash_key = store.store(
|
||||
ORIGINAL,
|
||||
"<<ccr:placeholder>>",
|
||||
original_tokens=100,
|
||||
compressed_tokens=5,
|
||||
tool_name="Bash",
|
||||
)
|
||||
try:
|
||||
yield hash_key
|
||||
finally:
|
||||
reset_compression_store()
|
||||
|
||||
|
||||
def _two_turn_session(
|
||||
marker: str, second_turn_rounds: list[bytes]
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], int]:
|
||||
"""Turn 1 introduces ``marker``; turn 2 only carries it in history.
|
||||
|
||||
Returns the client's SSE events for turn 2, the bodies the proxy sent
|
||||
upstream on turn 2's stream (first round, then continuations), and how
|
||||
many buffered upstream requests turn 2 made.
|
||||
"""
|
||||
first_turn = [
|
||||
{"role": "user", "content": "build it"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [{"type": "tool_use", "id": "tu_1", "name": "Bash", "input": {}}],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "tu_1",
|
||||
"content": f"build log compressed at <<ccr:{marker}>>",
|
||||
}
|
||||
],
|
||||
},
|
||||
]
|
||||
second_turn = [
|
||||
*first_turn,
|
||||
{"role": "assistant", "content": [{"type": "text", "text": "The build failed."}]},
|
||||
{"role": "user", "content": "what was the exact error?"},
|
||||
]
|
||||
buffered_calls: list[int] = []
|
||||
|
||||
async def fake_retry(method, url, headers, req_body, *args, **kwargs): # noqa: ANN001
|
||||
buffered_calls.append(1)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "msg_t1",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "sonnet-test",
|
||||
"content": [{"type": "text", "text": "The build failed."}],
|
||||
"stop_reason": "end_turn",
|
||||
"usage": {"input_tokens": 10, "output_tokens": 5},
|
||||
},
|
||||
headers={"content-type": "application/json"},
|
||||
)
|
||||
|
||||
config = ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
memory_enabled=False,
|
||||
)
|
||||
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
|
||||
tools = [
|
||||
create_ccr_tool_definition("anthropic"),
|
||||
{"name": "Bash", "description": "run", "input_schema": {"type": "object"}},
|
||||
]
|
||||
|
||||
def body(messages: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
return {
|
||||
"model": "sonnet-test",
|
||||
"max_tokens": 64,
|
||||
"stream": True,
|
||||
"tools": tools,
|
||||
"messages": messages,
|
||||
}
|
||||
|
||||
app = create_app(config)
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy._retry_request = fake_retry
|
||||
first = client.post("/v1/messages", json=body(first_turn), headers=headers)
|
||||
assert first.status_code == 200, first.text
|
||||
assert len(buffered_calls) == 1, "the turn that introduces the marker buffers"
|
||||
|
||||
upstream = MagicMock(spec=httpx.AsyncClient)
|
||||
upstream.build_request = MagicMock(return_value=MagicMock())
|
||||
upstream.send = AsyncMock(side_effect=[_sse_response(r) for r in second_turn_rounds])
|
||||
proxy.http_client = upstream
|
||||
second = client.post("/v1/messages", json=body(second_turn), headers=headers)
|
||||
assert second.status_code == 200, second.text
|
||||
|
||||
sent = [json.loads(c.kwargs["content"]) for c in upstream.build_request.call_args_list]
|
||||
return _events(second.content), sent, len(buffered_calls) - 1
|
||||
|
||||
|
||||
def test_later_turn_retrieval_is_served_by_the_proxy(owned_marker: str) -> None:
|
||||
events, sent, buffered = _two_turn_session(
|
||||
owned_marker,
|
||||
[
|
||||
_sse([TEXT, _retrieve(owned_marker)], "tool_use"),
|
||||
_sse([{"type": "text", "text": "The error is on line 2."}], "end_turn"),
|
||||
],
|
||||
)
|
||||
|
||||
assert buffered == 0, "a turn with only already-forwarded markers keeps streaming"
|
||||
assert "headroom_retrieve" not in _tool_names(events)
|
||||
texts = [e["delta"]["text"] for e in events if e["type"] == "content_block_delta"]
|
||||
assert texts == ["Let me look at the earlier output.", "The error is on line 2."]
|
||||
stops = [e["delta"]["stop_reason"] for e in events if e["type"] == "message_delta"]
|
||||
assert stops == ["end_turn"]
|
||||
assert [e["type"] for e in events].count("message_start") == 1
|
||||
|
||||
assert len(sent) == 2
|
||||
continuation = sent[1]["messages"]
|
||||
assert continuation[-2]["content"][-1]["name"] == "headroom_retrieve"
|
||||
served = continuation[-1]["content"][0]
|
||||
assert served["type"] == "tool_result" and served["tool_use_id"] == "toolu_ccr"
|
||||
assert json.loads(served["content"])["original_content"] == ORIGINAL
|
||||
|
||||
|
||||
@pytest.mark.parametrize("client_call_first", [True, False])
|
||||
def test_retrieval_mixed_with_a_client_tool(owned_marker: str, client_call_first: bool) -> None:
|
||||
calls = [BASH, _retrieve(owned_marker)]
|
||||
if not client_call_first:
|
||||
calls.reverse()
|
||||
reissued = {**BASH, "id": "toolu_bash_2"}
|
||||
events, sent, buffered = _two_turn_session(
|
||||
owned_marker,
|
||||
[
|
||||
_sse(calls, "tool_use"),
|
||||
_sse([{"type": "text", "text": "Now rerun it."}, reissued], "tool_use"),
|
||||
],
|
||||
)
|
||||
|
||||
assert buffered == 0
|
||||
# Only the model's decision made with the content in hand reaches the client.
|
||||
assert _tool_names(events) == ["Bash"]
|
||||
started = [e["content_block"] for e in events if e["type"] == "content_block_start"]
|
||||
assert [b.get("id") for b in started if b.get("type") == "tool_use"] == ["toolu_bash_2"]
|
||||
stops = [e["delta"]["stop_reason"] for e in events if e["type"] == "message_delta"]
|
||||
assert stops == ["tool_use"]
|
||||
assert [e["index"] for e in events if e["type"] == "content_block_start"] == [0, 1]
|
||||
|
||||
continuation = sent[1]["messages"]
|
||||
assert [b["name"] for b in continuation[-2]["content"]] == ["headroom_retrieve"]
|
||||
assert json.loads(continuation[-1]["content"][0]["content"])["original_content"] == ORIGINAL
|
||||
|
||||
|
||||
def test_old_marker_turn_without_retrieval_streams_unchanged(owned_marker: str) -> None:
|
||||
raw = _sse([{"type": "text", "text": "It failed on line 2."}, BASH], "tool_use")
|
||||
events, sent, buffered = _two_turn_session(owned_marker, [raw])
|
||||
|
||||
assert buffered == 0
|
||||
assert len(sent) == 1, "no continuation round without a retrieval"
|
||||
assert events == _events(raw)
|
||||
Reference in New Issue
Block a user