mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
## Description Hi! Three small costs on every proxied request, found while running headroom in front of a busy agent loop: - **Snapshot copy.** Both handlers `copy.deepcopy` the client messages before the pipeline, which already makes its own deep copy, so long conversations were copied twice per request. The snapshot now aliases the list (`HEADROOM_SNAPSHOT_DEEPCOPY=1` brings the copy back). - **Token cache wiped on re-counts.** `TokenCountCache.put` ran the admission check even for a key already cached, so a full cache cleared itself when an agent re-counted a stable prefix. A put on a known key now just updates it; a new key past the cap still clears, as before. - **Decompression on the event loop.** zstd/gzip/deflate/br bodies were decompressed inline. They now go through one bounded helper on `asyncio.to_thread`; the #3284 size caps and the 413/400 errors are unchanged. Two small things ride along: both shared body readers strip streaming-only `index` keys from content blocks (well-formed requests pass through byte-identical), and a dead `except` (ruff B025) that let a missing codec escape as a raw ImportError instead of a 400. **Happy to split this.** It is three independent changes plus two small fixes, and CONTRIBUTING asks for one logical change per PR. If you'd rather review them separately, say which and I'll open them one at a time. Opening as a draft until you've had a look. ## Type of Change - [x] Performance improvement - [x] Bug fix (non-breaking change that fixes an issue): the B025 codec path and the `index` strip ## Changes Made - `proxy/handlers/anthropic.py`, `openai.py`: snapshot alias + env switch - `tokenizers/base.py`: early return in `TokenCountCache.put` for a cached key - `proxy/helpers.py`: `_decompress_bounded` on a worker thread; readers call the `index` strip - `utils.py`: `strip_streaming_only_content_fields_in_place` - tests: 8 new (`test_output_only_request_blocks.py`, `test_token_count_cache.py`) ## Testing - [x] Unit tests pass (`pytest`): the proxy suite and the changed-area files, not the whole suite - [x] Linting passes (`ruff check`, `ruff format --check`) on the touched files - [ ] Type checking (`mypy headroom`): not run - [x] New tests added ### Test Output ```text # Linux, Python 3.12.13, source checkout + the compiled core from the published wheel (not built from source) $ pytest -q tests/test_proxy*.py tests/test_openai_message_level_cache_control.py tests/test_request_body_decompression_limits.py \ tests/test_output_only_request_blocks.py tests/test_token_count_cache.py tests/test_utils.py tests/test_tokenizer.py tests/test_tokenizers.py this branch: 1095 passed, 123 skipped, 2 failed main: 1087 passed, 123 skipped, 2 failed # the same two fail on main in this environment (probably the prebuilt core vs. source), untouched by this PR: # tests/test_proxy_openai_cache_stability.py::test_openai_chat_outcome_provider_uses_the_fixed_taxonomy # tests/test_proxy_openai_cache_stability.py::test_openai_chat_custom_base_flood_cannot_grow_the_provider_set $ ruff check <7 files> All checks passed! $ ruff format --check <7 files> 7 files already formatted ``` ## Real Behavior Proof - **Environment:** Linux, Python 3.12.13. `headroom proxy --host 127.0.0.1 --port 18787 --no-telemetry` run from this branch, `ANTHROPIC_TARGET_API_URL` pointed at a small local stand-in for `/v1/messages` that records what the proxy forwards. Same run repeated on main for comparison. - **Exact steps:** (1) after an 8 s warm-up, POST `/v1/messages` with a gzip-encoded body (228 kB on the wire, 77.4 MB decompressed, 79 messages) while polling `/livez` every 10 ms; (2) POST a message whose text block carries a stray `"index": 0`. - **Observed result (this branch):** ```text gzip body: 228 kB on the wire, 77.4 MB decompressed -> HTTP 200 in 1.29s upstream received: 77,422,858 bytes, 79 messages /livez while idle: 355 polls, max 2.2 ms, over 50 ms: 0 /livez during that request: 28 polls, max 288.3 ms, over 50 ms: 5, p50 2.3 ms content block with a stray `index`: HTTP 200; upstream saw no `index` key ``` Main, same steps: HTTP 200 in 1.35s; `/livez` during the request max 303.3 ms, 6 polls over 50 ms; no `index` forwarded either (the Anthropic handler already strips it on that path). - **Honest reading:** the decompression path works end to end off the loop, with the caps intact. At this size the loop stall is dominated by JSON parsing and the pipeline, not decompression, so this run shows **no measurable latency difference**. I'm not claiming a latency win from it. The snapshot alias and the cache change remove work, but they weren't benchmarked here. - **Not tested:** a real provider under production load; mypy; the full suite; the OpenAI and passthrough paths live (unit tests only). ## Runtime Rollout Safety - Stable/default behavior changed: snapshot aliases by default; cache keeps entries on re-count; decompression on a worker thread; `index` stripped at parse - Kill switch: `HEADROOM_SNAPSHOT_DEEPCOPY=1` for the snapshot; the rest have none - Rollback path: revert the PR ## Review Readiness - [x] I have performed a self-review - [ ] This PR is ready for human review (draft: waiting on whether you'd like it split) ## Additional Notes Worth a reviewer's eye: with the alias, the snapshot shares the live `messages` list. Nothing between the snapshot and its readers mutates that list in place today, and a code comment warns against starting to. Co-authored-by: bill3wits <8632714+bill3wits@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
226 lines
8.2 KiB
Python
226 lines
8.2 KiB
Python
"""Output-only content blocks must be stripped from request messages.
|
|
|
|
Anthropic's server-side refusal-fallback feature emits an output-only
|
|
``{"type": "fallback", ...}`` block inside an assistant response. It is valid on
|
|
the response path but rejected on the request path, so replaying that assistant
|
|
turn 400s the whole request. The shared body readers must drop it before
|
|
forwarding. See ``strip_output_only_request_blocks`` in ``headroom.proxy.helpers``.
|
|
"""
|
|
|
|
import asyncio
|
|
import json
|
|
|
|
from headroom.proxy.helpers import (
|
|
read_request_json_with_bytes,
|
|
strip_output_only_request_blocks,
|
|
)
|
|
|
|
_FALLBACK = {
|
|
"type": "fallback",
|
|
"from": {"model": "claude-fable-5"},
|
|
"to": {"model": "claude-opus-4-8"},
|
|
}
|
|
|
|
|
|
class _FakeHeaders:
|
|
def __init__(self, d=None):
|
|
self._d = {k.lower(): v for k, v in (d or {}).items()}
|
|
|
|
def get(self, k, default=None):
|
|
return self._d.get(k.lower(), default)
|
|
|
|
|
|
class _FakeRequest:
|
|
def __init__(self, raw, headers=None):
|
|
self._raw = raw
|
|
self.headers = _FakeHeaders(headers)
|
|
|
|
async def body(self):
|
|
return self._raw
|
|
|
|
async def stream(self):
|
|
yield self._raw
|
|
|
|
|
|
def _has_fallback(messages):
|
|
for msg in messages:
|
|
content = msg.get("content")
|
|
if isinstance(content, list):
|
|
for block in content:
|
|
if isinstance(block, dict) and block.get("type") == "fallback":
|
|
return True
|
|
return False
|
|
|
|
|
|
def test_strip_removes_fallback_and_backfills_emptied_turn():
|
|
messages = [
|
|
{"role": "user", "content": "hi"},
|
|
# assistant turn that is ONLY a fallback signal (the crash case)
|
|
{"role": "assistant", "content": [dict(_FALLBACK)]},
|
|
# fallback prefix + real content
|
|
{"role": "assistant", "content": [dict(_FALLBACK), {"type": "text", "text": "A."}]},
|
|
]
|
|
assert strip_output_only_request_blocks(messages) is True
|
|
assert not _has_fallback(messages)
|
|
# emptied turn is backfilled with a single benign text block
|
|
assert messages[1]["content"] == [{"type": "text", "text": "(model fallback)"}]
|
|
# mixed turn keeps only the real content
|
|
assert [b["type"] for b in messages[2]["content"]] == ["text"]
|
|
# idempotent
|
|
assert strip_output_only_request_blocks(messages) is False
|
|
|
|
|
|
def test_strip_is_noop_on_clean_or_invalid_input():
|
|
assert strip_output_only_request_blocks(None) is False
|
|
assert strip_output_only_request_blocks([{"role": "user", "content": "hi"}]) is False
|
|
assert (
|
|
strip_output_only_request_blocks(
|
|
[{"role": "user", "content": [{"type": "text", "text": "x"}]}]
|
|
)
|
|
is False
|
|
)
|
|
|
|
|
|
def test_reader_strips_and_reencodes_raw_bytes():
|
|
body = {
|
|
"model": "claude-fable-5",
|
|
"messages": [
|
|
{"role": "user", "content": "hi"},
|
|
{"role": "assistant", "content": [dict(_FALLBACK)]},
|
|
],
|
|
}
|
|
raw = json.dumps(body).encode("utf-8")
|
|
result, out_raw = asyncio.run(read_request_json_with_bytes(_FakeRequest(raw)))
|
|
assert not _has_fallback(result["messages"])
|
|
# raw bytes re-encoded so byte-faithful passthrough cannot leak the pre-strip body
|
|
assert not _has_fallback(json.loads(out_raw)["messages"])
|
|
assert json.loads(out_raw) == result
|
|
|
|
|
|
def test_reader_leaves_clean_requests_byte_identical():
|
|
raw = json.dumps({"model": "x", "messages": [{"role": "user", "content": "hi"}]}).encode(
|
|
"utf-8"
|
|
)
|
|
_, out_raw = asyncio.run(read_request_json_with_bytes(_FakeRequest(raw)))
|
|
assert out_raw == raw
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Streaming-only ``index`` keys (2026-09-27 perf audit): both shared body
|
|
# readers must canonicalize them in place so the snapshot, the forwarded body,
|
|
# and the recorded/replayed prefix are all schema-valid for EVERY provider,
|
|
# not just the Anthropic handler. ``index`` is a streaming-RESPONSE field;
|
|
# echoing it into request content 400s upstreams that reject extra inputs.
|
|
# Idempotent and a no-op for well-formed requests (byte-identical passthrough
|
|
# is preserved — the readers only re-encode raw when output-only blocks are
|
|
# removed).
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_with_bytes_reader_strips_streaming_index_keys():
|
|
body = {
|
|
"model": "claude-fable-5",
|
|
"messages": [
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{"type": "text", "text": "hi", "index": 0},
|
|
{"type": "tool_use", "id": "t1", "name": "f", "input": {}, "index": 1},
|
|
],
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "t1",
|
|
"content": [{"type": "text", "text": "ok", "index": 0}],
|
|
"index": 0,
|
|
}
|
|
],
|
|
},
|
|
],
|
|
}
|
|
raw = json.dumps(body).encode("utf-8")
|
|
result, out_raw = asyncio.run(read_request_json_with_bytes(_FakeRequest(raw)))
|
|
for msg in result["messages"]:
|
|
for block in msg["content"]:
|
|
assert "index" not in block
|
|
if block.get("type") == "tool_result":
|
|
for inner in block.get("content", []):
|
|
assert "index" not in inner
|
|
|
|
|
|
def test_with_bytes_reader_index_strip_preserves_clean_bytes():
|
|
"""No fallback blocks + index stripped: raw bytes stay the original wire bytes."""
|
|
body = {
|
|
"model": "x",
|
|
"messages": [
|
|
{"role": "assistant", "content": [{"type": "text", "text": "hi", "index": 0}]},
|
|
],
|
|
}
|
|
raw = json.dumps(body).encode("utf-8")
|
|
result, out_raw = asyncio.run(read_request_json_with_bytes(_FakeRequest(raw)))
|
|
assert result["messages"][0]["content"][0] == {"type": "text", "text": "hi"}
|
|
# Dict is stripped; raw is NOT re-encoded (index strip alone must not
|
|
# force a canonical re-serialization — byte-faithful passthrough holds).
|
|
assert out_raw == raw
|
|
|
|
|
|
def test_bytes_less_reader_strips_streaming_index_keys():
|
|
"""The bytes-less reader (Gemini + OpenAI chat paths) canonicalizes too."""
|
|
from headroom.proxy.helpers import _read_request_json
|
|
|
|
body = {
|
|
"model": "x",
|
|
"messages": [
|
|
{"role": "assistant", "content": [{"type": "text", "text": "hi", "index": 3}]},
|
|
],
|
|
}
|
|
raw = json.dumps(body).encode("utf-8")
|
|
result = asyncio.run(_read_request_json(_FakeRequest(raw)))
|
|
assert result["messages"][0]["content"][0] == {"type": "text", "text": "hi"}
|
|
|
|
|
|
def test_strip_streaming_index_is_idempotent():
|
|
"""The in-place canonicalizer must be safe to re-run (the readers apply it,
|
|
and the Anthropic handler re-runs its private copy downstream)."""
|
|
from headroom.utils import strip_streaming_only_content_fields_in_place
|
|
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{"type": "text", "text": "hi", "index": 0},
|
|
{"type": "tool_use", "id": "t1", "name": "f", "input": {}, "index": 1},
|
|
],
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "t1",
|
|
"content": [{"type": "text", "text": "ok", "index": 0}],
|
|
"index": 0,
|
|
}
|
|
],
|
|
},
|
|
{"role": "user", "content": "plain string content"},
|
|
]
|
|
first = strip_streaming_only_content_fields_in_place(messages)
|
|
second = strip_streaming_only_content_fields_in_place(messages)
|
|
# returns None (in-place); idempotency is asserted by the walk below
|
|
assert first is None and second is None
|
|
for msg in messages:
|
|
content = msg.get("content")
|
|
if isinstance(content, list):
|
|
for block in content:
|
|
assert "index" not in block
|
|
if isinstance(block.get("content"), list):
|
|
for inner in block["content"]:
|
|
assert "index" not in inner
|
|
# non-list input is a safe no-op
|
|
strip_streaming_only_content_fields_in_place(None)
|
|
strip_streaming_only_content_fields_in_place({"not": "a list"})
|