mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description Complete UTF-8 SSE frames can contain U+0085, U+2028, or U+2029 inside JSON strings. Splitting decoded text with `str.splitlines()` treats those payload characters as framing, corrupting JSON and bypassing memory-tool continuation index remapping. This change splits complete frames at byte line boundaries before strict UTF-8 decoding and adds regressions for both parser paths. Fixes #4251 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Split SSE frame bytes into lines before decoding in the shared parser and memory-tool filter. - Cover all three Unicode separators, CRLF event framing, payload preservation, and continuation index remapping. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .` on changed files) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed by this PR author ### Test Output ```text uv run --no-sync pytest tests/test_sse_byte_buffer_policy.py tests/test_sse_utf8_split.py tests/test_streaming_sse_malformed_events.py tests/test_memory_tool_stream.py -q -k 'not test_injected_memory_tools_reach_the_stream' 424 passed, 1 deselected (environment-limited server setup test) uv run --no-sync ruff check headroom/proxy/sse_byte_buffer_policy.py headroom/proxy/memory_tool_stream.py tests/test_sse_byte_buffer_policy.py tests/test_memory_tool_stream.py passed uv run --no-sync ruff format --check headroom/proxy/sse_byte_buffer_policy.py headroom/proxy/memory_tool_stream.py tests/test_sse_byte_buffer_policy.py tests/test_memory_tool_stream.py passed git diff --check passed ``` ## Real Behavior Proof - Environment: Linux x86-64, Python 3.13.14. - Exact command / steps: This PR author ran the focused regression command above. Separately, the Issue reporter reports replaying the frames through a localhost HTTP server and httpx with one-byte-at-a-time reads; the Issue does not provide an exact executable command. - Observed result: The regression suite verifies all three Unicode separators survive parsing and the continuation block start and text delta both use index 2. The reporter reports that its replay preserved all three characters and produced continuation indices `[2, 2, 2]`; that replay was not independently rerun for this PR. - Not tested: Live model/provider behavior and a deployed proxy. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Not applicable. - Stable/default behavior changed: SSE payload text containing Unicode line separators remains intact. - Kill switch / disable path: Not applicable. - Unsafe override required: No. - Qualification impact: None expected; no dependency or native-code changes. - Rollback path: Revert this change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas (no new comments were needed) - [ ] I have made corresponding changes to the documentation (not applicable to this parser fix) - [ ] My changes generate no new warnings (not independently checked) - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes (focused related suite) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes The Issue reporter has a similar implementation on a fork branch, but no upstream PR exists for it. The issue remains open; this PR submits the reviewed job candidate for the maintainer's consideration. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4337" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Maintainer runtime follow-up (2026-11-04) AI-assisted qualification on Windows x64 / Python 3.12.13, after syncing main31dcaf41f. All 427 focused SSE/parser/memory-stream tests passed; lint, formatting and whitespace checks passed. This uses the existing native binary rather than a fresh native build. I independently ran a real loopback HTTP SSE server and consumed its response with httpx at one byte per read through both parser paths. All 283 byte chunks were consumed; U+0085, U+2028 and U+2029 survived exactly, and the continuation indices were [2, 2]. No live provider or deployed proxy is claimed. Reproduce from the checkout by saving this as qualification.py and running `python qualification.py`: ```python import sys,json,threading from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer import httpx from headroom.proxy.sse_byte_buffer_policy import parse_sse_events_from_byte_buffer from headroom.proxy.memory_tool_stream import MemoryToolStreamFilter from headroom.memory.tools import get_tool_names text="before\^Emiddle\u2028next\u2029after" payloads=[{"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}},{"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":text}}] wire=b"".join(("event: "+p["type"]+"\r\ndata: "+json.dumps(p,ensure_ascii=False)+"\r\n\r\n").encode("utf-8") for p in payloads) class Handler(BaseHTTPRequestHandler): def do_GET(self): self.send_response(200);self.send_header("Content-Type","text/event-stream");self.send_header("Content-Length",str(len(wire)));self.end_headers() for byte in wire:self.wfile.write(bytes([byte]));self.wfile.flush() def log_message(self,*args):pass server=ThreadingHTTPServer(("127.0.0.1",0),Handler);thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start() try: shared=[]; filtered=[]; buf=bytearray(); outbuf=bytearray(); chunks=0 flt=MemoryToolStreamFilter(get_tool_names(),index_offset=2,forward_message_start=False) with httpx.Client(trust_env=False,timeout=5) as client: with client.stream("GET",f"http://127.0.0.1:{server.server_port}/events") as response: response.raise_for_status() for chunk in response.iter_bytes(chunk_size=1): chunks+=1;buf.extend(chunk);shared.extend(parse_sse_events_from_byte_buffer(buf)) for output in flt.feed(chunk):outbuf.extend(output) filtered.extend(parse_sse_events_from_byte_buffer(outbuf)) assert chunks==len(wire) and not buf and not outbuf objects=[json.loads(data) for _,data in shared];remapped=[json.loads(data) for _,data in filtered] assert objects==payloads assert [p["index"] for p in remapped]==[2,2] assert remapped[1]["delta"]["text"]==text print(json.dumps({"transport":"loopback HTTP SSE","read_chunk_bytes":1,"chunks":chunks,"unicode_codepoints_preserved":["U+0085","U+2028","U+2029"],"shared_events":len(shared),"continuation_indices":[p["index"] for p in remapped],"status":"PASS"})) finally:server.shutdown();server.server_close();thread.join(5) ``` Current-head hosted CI is separate from this local runtime evidence. ### Maintainer current-main requalification Exact head `8253904f1d25b6c7d42d2b2bde277aee03b57202`, upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. Updated against upstream/maine4a60ed7and re-reviewed both SSE parser paths. Resolved the test-only overlap by retaining both the PR's Unicode JSON regression and main's new one-leading-space framing regression; production parser code merged automatically. All 428 parser/UTF-8/malformed-event/memory-tool-stream tests pass. Ruff lint/format and whitespace checks pass. Independently reran the real localhost HTTP SSE replay through httpx in 283 one-byte chunks on this combined checkout: U+0085/U+2028/U+2029 payload text is preserved by both parser paths and continuation indices remain [2, 2]. The existing executable reproduction in this body remains applicable. This uses the existing development native extension and does not claim a fresh native build, live provider or deployed proxy qualification. No contributor production logic was rewritten. Applied the shared test-only Gemini streamed-request fixture repair for independently established main failures; all ten cases pass. Formal approval awaits exact-head hosted qualification. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
64 lines
2.1 KiB
Python
64 lines
2.1 KiB
Python
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.sse_byte_buffer_policy import (
|
|
find_sse_event_terminator,
|
|
parse_sse_events_from_byte_buffer,
|
|
)
|
|
|
|
|
|
def test_find_sse_event_terminator_returns_earliest_separator() -> None:
|
|
assert find_sse_event_terminator(bytearray(b"data: one\r\n\r\ndata: two\n\n")) == (9, 4)
|
|
|
|
|
|
def test_parse_sse_events_drains_complete_events_and_leaves_tail() -> None:
|
|
buf = bytearray(b": ignored\nevent: delta\ndata: one\ndata: two\n\npartial")
|
|
|
|
assert parse_sse_events_from_byte_buffer(buf) == [("delta", "one\ntwo")]
|
|
assert bytes(buf) == b"partial"
|
|
|
|
|
|
def test_parse_sse_events_preserves_split_utf8_tail() -> None:
|
|
smile = "\U0001f642".encode()
|
|
buf = bytearray(b"data: hello " + smile[:2])
|
|
|
|
assert parse_sse_events_from_byte_buffer(buf) == []
|
|
buf.extend(smile[2:] + b"\n\n")
|
|
assert parse_sse_events_from_byte_buffer(buf) == [(None, "hello \U0001f642")]
|
|
|
|
|
|
def test_parse_sse_events_splits_crlf_lines() -> None:
|
|
buf = bytearray(b"event: delta\r\ndata: one\r\ndata: two\r\n\r\n")
|
|
|
|
assert parse_sse_events_from_byte_buffer(buf) == [("delta", "one\ntwo")]
|
|
assert not buf
|
|
|
|
|
|
def test_parse_sse_events_raises_on_complete_invalid_utf8_event() -> None:
|
|
with pytest.raises(UnicodeDecodeError):
|
|
parse_sse_events_from_byte_buffer(bytearray(b"data: \xff\n\n"))
|
|
|
|
|
|
@pytest.mark.parametrize("separator", ["\u0085", "\u2028", "\u2029"])
|
|
def test_parse_sse_events_preserves_unicode_line_separators_in_json(
|
|
separator: str,
|
|
) -> None:
|
|
payload = {"text": f"before{separator}after"}
|
|
wire = f"data: {json.dumps(payload, ensure_ascii=False)}\n\n".encode()
|
|
|
|
events = parse_sse_events_from_byte_buffer(bytearray(wire))
|
|
|
|
assert len(events) == 1
|
|
assert json.loads(events[0][1]) == payload
|
|
|
|
|
|
def test_parse_sse_events_strips_only_one_leading_space() -> None:
|
|
# SSE spec: "If value starts with a U+0020 SPACE character, remove it".
|
|
# Further leading whitespace is part of the payload.
|
|
buf = bytearray(b"event: delta\ndata: indented\ndata:\ttab\ndata:none\n\n")
|
|
|
|
assert parse_sse_events_from_byte_buffer(buf) == [("delta", " indented\n\ttab\nnone")]
|