Files
headroom/tests/test_sse_byte_buffer_policy.py
lorenzozaneandJD Davis 3afaa97ffc fix(proxy): preserve Unicode separators in SSE JSON (#4337)
## Description

Complete UTF-8 SSE frames can contain U+0085, U+2028, or U+2029 inside
JSON strings. Splitting decoded text with `str.splitlines()` treats
those payload characters as framing, corrupting JSON and bypassing
memory-tool continuation index remapping. This change splits complete
frames at byte line boundaries before strict UTF-8 decoding and adds
regressions for both parser paths.

Fixes #4251

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Split SSE frame bytes into lines before decoding in the shared parser
and memory-tool filter.
- Cover all three Unicode separators, CRLF event framing, payload
preservation, and continuation index remapping.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .` on changed files)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed by this PR author

### Test Output

```text
uv run --no-sync pytest tests/test_sse_byte_buffer_policy.py tests/test_sse_utf8_split.py tests/test_streaming_sse_malformed_events.py tests/test_memory_tool_stream.py -q -k 'not test_injected_memory_tools_reach_the_stream'
424 passed, 1 deselected (environment-limited server setup test)

uv run --no-sync ruff check headroom/proxy/sse_byte_buffer_policy.py headroom/proxy/memory_tool_stream.py tests/test_sse_byte_buffer_policy.py tests/test_memory_tool_stream.py
passed

uv run --no-sync ruff format --check headroom/proxy/sse_byte_buffer_policy.py headroom/proxy/memory_tool_stream.py tests/test_sse_byte_buffer_policy.py tests/test_memory_tool_stream.py
passed

git diff --check
passed
```

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14.
- Exact command / steps: This PR author ran the focused regression
command above. Separately, the Issue reporter reports replaying the
frames through a localhost HTTP server and httpx with one-byte-at-a-time
reads; the Issue does not provide an exact executable command.
- Observed result: The regression suite verifies all three Unicode
separators survive parsing and the continuation block start and text
delta both use index 2. The reporter reports that its replay preserved
all three characters and produced continuation indices `[2, 2, 2]`; that
replay was not independently rerun for this PR.
- Not tested: Live model/provider behavior and a deployed proxy.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable.
- Stable/default behavior changed: SSE payload text containing Unicode
line separators remains intact.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Qualification impact: None expected; no dependency or native-code
changes.
- Rollback path: Revert this change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
(no new comments were needed)
- [ ] I have made corresponding changes to the documentation (not
applicable to this parser fix)
- [ ] My changes generate no new warnings (not independently checked)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (focused
related suite)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

The Issue reporter has a similar implementation on a fork branch, but no
upstream PR exists for it. The issue remains open; this PR submits the
reviewed job candidate for the maintainer's consideration.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4337"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Maintainer runtime follow-up (2026-11-04)

AI-assisted qualification on Windows x64 / Python 3.12.13, after syncing
main 31dcaf41f. All 427 focused SSE/parser/memory-stream tests passed;
lint, formatting and whitespace checks passed. This uses the existing
native binary rather than a fresh native build.

I independently ran a real loopback HTTP SSE server and consumed its
response with httpx at one byte per read through both parser paths. All
283 byte chunks were consumed; U+0085, U+2028 and U+2029 survived
exactly, and the continuation indices were [2, 2]. No live provider or
deployed proxy is claimed.

Reproduce from the checkout by saving this as qualification.py and
running `python qualification.py`:

```python
import sys,json,threading
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import httpx
from headroom.proxy.sse_byte_buffer_policy import parse_sse_events_from_byte_buffer
from headroom.proxy.memory_tool_stream import MemoryToolStreamFilter
from headroom.memory.tools import get_tool_names
text="before\^Emiddle\u2028next\u2029after"
payloads=[{"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}},{"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":text}}]
wire=b"".join(("event: "+p["type"]+"\r\ndata: "+json.dumps(p,ensure_ascii=False)+"\r\n\r\n").encode("utf-8") for p in payloads)
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
  self.send_response(200);self.send_header("Content-Type","text/event-stream");self.send_header("Content-Length",str(len(wire)));self.end_headers()
  for byte in wire:self.wfile.write(bytes([byte]));self.wfile.flush()
 def log_message(self,*args):pass
server=ThreadingHTTPServer(("127.0.0.1",0),Handler);thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
try:
 shared=[]; filtered=[]; buf=bytearray(); outbuf=bytearray(); chunks=0
 flt=MemoryToolStreamFilter(get_tool_names(),index_offset=2,forward_message_start=False)
 with httpx.Client(trust_env=False,timeout=5) as client:
  with client.stream("GET",f"http://127.0.0.1:{server.server_port}/events") as response:
   response.raise_for_status()
   for chunk in response.iter_bytes(chunk_size=1):
    chunks+=1;buf.extend(chunk);shared.extend(parse_sse_events_from_byte_buffer(buf))
    for output in flt.feed(chunk):outbuf.extend(output)
    filtered.extend(parse_sse_events_from_byte_buffer(outbuf))
 assert chunks==len(wire) and not buf and not outbuf
 objects=[json.loads(data) for _,data in shared];remapped=[json.loads(data) for _,data in filtered]
 assert objects==payloads
 assert [p["index"] for p in remapped]==[2,2]
 assert remapped[1]["delta"]["text"]==text
 print(json.dumps({"transport":"loopback HTTP SSE","read_chunk_bytes":1,"chunks":chunks,"unicode_codepoints_preserved":["U+0085","U+2028","U+2029"],"shared_events":len(shared),"continuation_indices":[p["index"] for p in remapped],"status":"PASS"}))
finally:server.shutdown();server.server_close();thread.join(5)
```

Current-head hosted CI is separate from this local runtime evidence.


### Maintainer current-main requalification

Exact head `8253904f1d25b6c7d42d2b2bde277aee03b57202`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Updated against upstream/main e4a60ed7 and re-reviewed both SSE parser
paths. Resolved the test-only overlap by retaining both the PR's Unicode
JSON regression and main's new one-leading-space framing regression;
production parser code merged automatically. All 428
parser/UTF-8/malformed-event/memory-tool-stream tests pass. Ruff
lint/format and whitespace checks pass.

Independently reran the real localhost HTTP SSE replay through httpx in
283 one-byte chunks on this combined checkout: U+0085/U+2028/U+2029
payload text is preserved by both parser paths and continuation indices
remain [2, 2]. The existing executable reproduction in this body remains
applicable. This uses the existing development native extension and does
not claim a fresh native build, live provider or deployed proxy
qualification. No contributor production logic was rewritten.

Applied the shared test-only Gemini streamed-request fixture repair for
independently established main failures; all ten cases pass. Formal
approval awaits exact-head hosted qualification.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-05 10:41:45 -06:00

64 lines
2.1 KiB
Python

from __future__ import annotations
import json
import pytest
from headroom.proxy.sse_byte_buffer_policy import (
find_sse_event_terminator,
parse_sse_events_from_byte_buffer,
)
def test_find_sse_event_terminator_returns_earliest_separator() -> None:
assert find_sse_event_terminator(bytearray(b"data: one\r\n\r\ndata: two\n\n")) == (9, 4)
def test_parse_sse_events_drains_complete_events_and_leaves_tail() -> None:
buf = bytearray(b": ignored\nevent: delta\ndata: one\ndata: two\n\npartial")
assert parse_sse_events_from_byte_buffer(buf) == [("delta", "one\ntwo")]
assert bytes(buf) == b"partial"
def test_parse_sse_events_preserves_split_utf8_tail() -> None:
smile = "\U0001f642".encode()
buf = bytearray(b"data: hello " + smile[:2])
assert parse_sse_events_from_byte_buffer(buf) == []
buf.extend(smile[2:] + b"\n\n")
assert parse_sse_events_from_byte_buffer(buf) == [(None, "hello \U0001f642")]
def test_parse_sse_events_splits_crlf_lines() -> None:
buf = bytearray(b"event: delta\r\ndata: one\r\ndata: two\r\n\r\n")
assert parse_sse_events_from_byte_buffer(buf) == [("delta", "one\ntwo")]
assert not buf
def test_parse_sse_events_raises_on_complete_invalid_utf8_event() -> None:
with pytest.raises(UnicodeDecodeError):
parse_sse_events_from_byte_buffer(bytearray(b"data: \xff\n\n"))
@pytest.mark.parametrize("separator", ["\u0085", "\u2028", "\u2029"])
def test_parse_sse_events_preserves_unicode_line_separators_in_json(
separator: str,
) -> None:
payload = {"text": f"before{separator}after"}
wire = f"data: {json.dumps(payload, ensure_ascii=False)}\n\n".encode()
events = parse_sse_events_from_byte_buffer(bytearray(wire))
assert len(events) == 1
assert json.loads(events[0][1]) == payload
def test_parse_sse_events_strips_only_one_leading_space() -> None:
# SSE spec: "If value starts with a U+0020 SPACE character, remove it".
# Further leading whitespace is part of the payload.
buf = bytearray(b"event: delta\ndata: indented\ndata:\ttab\ndata:none\n\n")
assert parse_sse_events_from_byte_buffer(buf) == [("delta", " indented\n\ttab\nnone")]