mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description
`split_into_sections()` in `headroom/transforms/mixed_content.py`
recognises an opening code fence with `^```(\w*)\s*$`. Common fence tags
with non-word characters don't match: ```` ```c++ ````, ```` ```c# ````,
```` ```objective-c ````, ```` ```js {1,3} ````, ```` ```python
title="app.py" ````. When the opener is missed:
1. the fence line and code are kept as plain text;
2. the block's **closing** ```` ``` ```` matches the pattern (empty
tag), so it is taken as an **opening** fence;
3. everything after it (prose, JSON, until the next ```` ``` ```` or end
of input) becomes one `SOURCE_CODE` section with language `unknown`,
goes to the code compressor, and is re-emitted wrapped in ````
```unknown ```` … ```` ``` ````.
So the code block loses its closing fence, and the prose and data that
follow it come back inside a fake code block. Through
`headroom.compress()` this turns a tool result's JSON into a
JSON-escaped string (`\n` sequences) inside that stray fence (see the
proof below). It only takes one C++ or C# snippet in a mixed tool result
or fetched page.
Closes: N/A (no matching issue found)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/mixed_content.py`:
- `_CODE_FENCE_PATTERN` is now `^```([^\n`]*)$`. Group 1 is the whole
info string, any text without a backtick, so ```` ```c++ ````, ````
```js {1,3} ```` and ```` ``` c++ ```` (space before the tag, valid
Markdown) all open a fence. A closing ```` ``` ```` and a plain ````
```python ```` match as before. A line with a later backtick (````
```foo``` ````) still doesn't.
- New `_fence_language()` takes the info string's first word as the
language, and returns `"unknown"` for a bare fence, as before.
- It's a single quantifier, so a 200,000-character fence line takes
about 2 ms. While working on this I measured an overlapping variant
(`([^\s`]*)[^\n`]*`) at 3.3 s on a 20,000-character line, and avoided
it.
- `tests/test_mixed_content_sections.py`:
- `test_split_accepts_fence_info_strings_beyond_word_characters`,
parametrized over `c++`, `c#`, `objective-c`, `js {1,3}`, `python
title="app.py"`, ` c++` and `python\r`, checks the sections come out
PLAIN_TEXT / SOURCE_CODE(lang) / PLAIN_TEXT / JSON_ARRAY with the code
body intact.
- `test_fence_pattern_stays_linear_on_a_long_unclosed_fence_line` keeps
a 50,000-character fence line under 0.5 s. That's far above the linear
cost (under 1 ms), so a loaded runner won't trip it, but a quadratic
pattern (seconds) would. It passes on `main` too; it's there so any
future widening stays linear, as in
`test_multiline_regex_blank_runs.py`.
## Testing
- [x] Unit tests pass (`pytest`): `tests/test_mixed_content*.py`,
`tests/test_content_router*.py`,
`tests/test_transforms_content_router.py`,
`tests/test_code_compressor*.py`,
`tests/test_multiline_regex_blank_runs.py`, `tests/test_transforms/`,
not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests on `main` (fix reverted, tests kept):
```text
$ python -m pytest tests/test_mixed_content_sections.py -q
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c++-c++]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c#-c#]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[objective-c-objective-c]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[js {1,3}-js]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[python title="app.py"-python]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[ c++-c++]
========================= 6 failed, 6 passed in 1.53s ==========================
```
With the fix:
```text
$ python -m pytest tests/test_mixed_content*.py tests/test_content_router*.py tests/test_transforms_content_router.py \
tests/test_code_compressor*.py tests/test_multiline_regex_blank_runs.py tests/test_transforms -q
====================== 1044 passed, 88 skipped in 32.30s =======================
$ ruff check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
All checks passed!
$ ruff format --check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
2 files already formatted
```
## Real Behavior Proof
- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel),
`HEADROOM_OFFLINE=1`, public `headroom.compress()` API
- Exact command / steps: called `headroom.compress(messages,
model="gpt-4o", model_limit=128000, protect_recent=0)` on an
OpenAI-format conversation whose tool result is: one intro line, a ````
```c++ ```` block of two declarations, 40 prose lines, then a 400-row
JSON array (13.5K tokens). I printed the `transforms_applied`, every
fence line in the compressed tool result and its tail, on `main` and on
this branch. (Through the proxy, the same content only reaches the
mixed-content path once it is old enough to be compressed, so I used the
library entry point to keep the repro to one call.)
- Observed result: both runs take the mixed path (`router:mixed`). On
`main` the output has three fence lines (`c++`, a stray `unknown`, and a
closing one), the C++ block lost its closing fence, and the JSON tail
comes back as a JSON-escaped string inside the stray fence. On this
branch the C++ block keeps both fences, the prose stays prose, and the
JSON is compacted to CSV rows as the structured path does elsewhere.
```text
== main
tokens 13474 -> 4289 | transforms: [..., 'router:mixed:0.35']
fence lines: ['```c++', '```unknown', '```']
| Here is the Header file that we wrote for the project today.
| ```c++
| int add(int a, int b);
| int sub(int a, int b);
|
| ```unknown
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\\n398,item398,team-a,ok\\n399,item399,team-a,ok\\n"\n```'
== this branch
tokens 13474 -> 4285 | transforms: [..., 'router:mixed:0.29']
fence lines: ['```c++', '```']
| ```c++
| int add(int a, int b);
| int sub(int a, int b);
| ```
|
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\n398,item398,team-a,ok\n399,item399,team-a,ok\n'
```
- Not tested: the same content through the live proxy's mixed-content
path (in a short session the turn is protected or below the size gates,
so nothing is compressed on either branch); Rust — there is no Rust
mirror of `split_into_sections`.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: fences tagged with non-word
characters (or a space before the tag) are now parsed as code blocks.
Content with only `\w` tags (```` ```python ````, ```` ```json ````,
bare ```` ``` ````) splits exactly as before; the existing 565
router/transform tests pass unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- An **unclosed** fence still runs to the end of the input, as an
unclosed ```` ```python ```` already does on `main`. This PR only
changes which openers are recognised. Making unclosed fences fall back
to plain text would change behaviour for every tag, so it seems better
as its own PR.
- The router re-wraps a fenced section with ```` ```{section.language}
````, so trailing info-string metadata (`{1,3}`, `title=...`) is not
re-emitted. That was already true for every fence; this PR only fixes
the opener being missed.
- `_extract_fenced_command()` in `content_router.py` has its own
`[\w.-]+` fence class. It only extracts a shell command for
read-detection, so I left it alone to keep this PR to one change.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
176 lines
5.8 KiB
Python
176 lines
5.8 KiB
Python
import pytest
|
|
|
|
from headroom.transforms.content_detector import ContentType
|
|
from headroom.transforms.mixed_content import (
|
|
_extract_json_block,
|
|
is_mixed_content,
|
|
split_into_sections,
|
|
)
|
|
|
|
|
|
def test_mixed_content_detection_requires_multiple_signals():
|
|
prose = "\n".join(
|
|
[
|
|
"First sentence has enough words to count.",
|
|
"Second sentence has enough words to count.",
|
|
"Third sentence has enough words to count.",
|
|
"Fourth sentence has enough words to count.",
|
|
"Fifth sentence has enough words to count.",
|
|
"Sixth sentence has enough words to count.",
|
|
]
|
|
)
|
|
|
|
assert is_mixed_content(prose) is False
|
|
assert is_mixed_content(f"{prose}\n```python\nprint('x')\n```") is True
|
|
|
|
|
|
def test_split_into_sections_preserves_typed_boundaries():
|
|
content = "\n".join(
|
|
[
|
|
"Intro text",
|
|
"```python",
|
|
"print('x')",
|
|
"```",
|
|
'[{"id": 1}]',
|
|
"src/app.py:10:print('x')",
|
|
]
|
|
)
|
|
|
|
sections = split_into_sections(content)
|
|
|
|
assert [section.content_type for section in sections] == [
|
|
ContentType.PLAIN_TEXT,
|
|
ContentType.SOURCE_CODE,
|
|
ContentType.JSON_ARRAY,
|
|
ContentType.SEARCH_RESULTS,
|
|
]
|
|
assert sections[1].language == "python"
|
|
assert sections[1].content == "print('x')"
|
|
assert sections[1].is_code_fence is True
|
|
assert sections[2].content == '[{"id": 1}]'
|
|
assert sections[3].start_line == 5
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("info_string", "language"),
|
|
[
|
|
("c++", "c++"),
|
|
("c#", "c#"),
|
|
("objective-c", "objective-c"),
|
|
("js {1,3}", "js"),
|
|
('python title="app.py"', "python"),
|
|
(" c++", "c++"),
|
|
("python\r", "python"),
|
|
],
|
|
)
|
|
def test_split_accepts_fence_info_strings_beyond_word_characters(info_string, language):
|
|
content = "\n".join(
|
|
[
|
|
"Here is the header file we wrote today.",
|
|
f"```{info_string}",
|
|
"int add(int a, int b);",
|
|
"```",
|
|
"Please review this carefully and tell me what you think.",
|
|
'{"name": "demo", "version": 1}',
|
|
]
|
|
)
|
|
|
|
sections = split_into_sections(content)
|
|
|
|
assert [section.content_type for section in sections] == [
|
|
ContentType.PLAIN_TEXT,
|
|
ContentType.SOURCE_CODE,
|
|
ContentType.PLAIN_TEXT,
|
|
ContentType.JSON_ARRAY,
|
|
]
|
|
assert sections[1].language == language
|
|
assert sections[1].content == "int add(int a, int b);"
|
|
assert sections[1].is_code_fence is True
|
|
assert sections[2].content == "Please review this carefully and tell me what you think."
|
|
|
|
|
|
def test_fence_pattern_stays_linear_on_a_long_unclosed_fence_line():
|
|
import time
|
|
|
|
from headroom.transforms.mixed_content import _CODE_FENCE_PATTERN
|
|
|
|
line = "```" + "a" * 50_000 + "`"
|
|
best = float("inf")
|
|
for _ in range(3):
|
|
start = time.perf_counter()
|
|
assert _CODE_FENCE_PATTERN.match(line) is None
|
|
best = min(best, time.perf_counter() - start)
|
|
# Linear: well under 1 ms here. A quadratic pattern takes seconds, so a
|
|
# generous budget still catches it on a loaded runner.
|
|
assert best < 0.5, f"{best * 1000:.1f}ms on a 50,000-character fence line"
|
|
|
|
|
|
def test_split_carves_grep_context_lines_into_search_sections():
|
|
"""Regression (#3580): prose + ``grep -A`` output keeps code out of text.
|
|
|
|
Context lines (``path-NN-content``) must join the SEARCH_RESULTS section
|
|
instead of stranding source code in a PLAIN_TEXT section, where the
|
|
word-dropping prose compressor would corrupt it. ``--`` group separators
|
|
ride along as ordinary text lines.
|
|
"""
|
|
content = "\n".join(
|
|
[
|
|
"Some intro prose explaining the search.",
|
|
"src/main.py-40- context before",
|
|
"src/main.py:42:def process_data(items):",
|
|
"src/main.py-43- context after",
|
|
"--",
|
|
"src/other.py:12: return True",
|
|
"Trailing prose after the results.",
|
|
]
|
|
)
|
|
|
|
sections = split_into_sections(content)
|
|
|
|
# The ``--`` group separator breaks the run, so the two grep groups keep
|
|
# their own SEARCH_RESULTS sections instead of merging through it.
|
|
assert [section.content_type for section in sections] == [
|
|
ContentType.PLAIN_TEXT,
|
|
ContentType.SEARCH_RESULTS,
|
|
ContentType.PLAIN_TEXT,
|
|
ContentType.SEARCH_RESULTS,
|
|
ContentType.PLAIN_TEXT,
|
|
]
|
|
assert "src/main.py-40- context before" in sections[1].content
|
|
assert "src/main.py:42:def process_data(items):" in sections[1].content
|
|
assert "src/main.py-43- context after" in sections[1].content
|
|
assert "src/other.py:12: return True" in sections[3].content
|
|
|
|
|
|
def test_extract_json_block_ignores_delimiters_inside_strings():
|
|
lines = [
|
|
"[",
|
|
' {"path": "a]b", "message": "keep {literal} braces"},',
|
|
' {"path": "c"}',
|
|
"]",
|
|
]
|
|
|
|
block, end_line = _extract_json_block(lines, 0)
|
|
|
|
assert end_line == 3
|
|
assert block == "\n".join(lines)
|
|
|
|
|
|
def test_split_does_not_carve_timestamped_log_rows_as_search():
|
|
"""Regression (#3736 residual): the detector rejects ISO-timestamped log
|
|
rows as search output; the splitter must not carve them either, while
|
|
real ``path:NN:content`` grep lines still carve.
|
|
"""
|
|
logs = [
|
|
"2026-09-23T10:05:07Z worker-1 INFO request handled",
|
|
"2026-09-23T10:05:13Z worker-2 ERROR upstream timeout",
|
|
]
|
|
content = "\n".join(["Intro text", "```python", "print('x')", "```", *logs])
|
|
|
|
assert ContentType.SEARCH_RESULTS not in [
|
|
section.content_type for section in split_into_sections(content)
|
|
]
|
|
|
|
grep = "\n".join(["Intro text", "```python", "print('x')", "```", "src/app.py:10:print('x')"])
|
|
assert split_into_sections(grep)[-1].content_type == ContentType.SEARCH_RESULTS
|