mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(transforms): recognise code fences whose info string is not a bare word (#4291)
## Description
`split_into_sections()` in `headroom/transforms/mixed_content.py`
recognises an opening code fence with `^```(\w*)\s*$`. Common fence tags
with non-word characters don't match: ```` ```c++ ````, ```` ```c# ````,
```` ```objective-c ````, ```` ```js {1,3} ````, ```` ```python
title="app.py" ````. When the opener is missed:
1. the fence line and code are kept as plain text;
2. the block's **closing** ```` ``` ```` matches the pattern (empty
tag), so it is taken as an **opening** fence;
3. everything after it (prose, JSON, until the next ```` ``` ```` or end
of input) becomes one `SOURCE_CODE` section with language `unknown`,
goes to the code compressor, and is re-emitted wrapped in ````
```unknown ```` … ```` ``` ````.
So the code block loses its closing fence, and the prose and data that
follow it come back inside a fake code block. Through
`headroom.compress()` this turns a tool result's JSON into a
JSON-escaped string (`\n` sequences) inside that stray fence (see the
proof below). It only takes one C++ or C# snippet in a mixed tool result
or fetched page.
Closes: N/A (no matching issue found)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/mixed_content.py`:
- `_CODE_FENCE_PATTERN` is now `^```([^\n`]*)$`. Group 1 is the whole
info string, any text without a backtick, so ```` ```c++ ````, ````
```js {1,3} ```` and ```` ``` c++ ```` (space before the tag, valid
Markdown) all open a fence. A closing ```` ``` ```` and a plain ````
```python ```` match as before. A line with a later backtick (````
```foo``` ````) still doesn't.
- New `_fence_language()` takes the info string's first word as the
language, and returns `"unknown"` for a bare fence, as before.
- It's a single quantifier, so a 200,000-character fence line takes
about 2 ms. While working on this I measured an overlapping variant
(`([^\s`]*)[^\n`]*`) at 3.3 s on a 20,000-character line, and avoided
it.
- `tests/test_mixed_content_sections.py`:
- `test_split_accepts_fence_info_strings_beyond_word_characters`,
parametrized over `c++`, `c#`, `objective-c`, `js {1,3}`, `python
title="app.py"`, ` c++` and `python\r`, checks the sections come out
PLAIN_TEXT / SOURCE_CODE(lang) / PLAIN_TEXT / JSON_ARRAY with the code
body intact.
- `test_fence_pattern_stays_linear_on_a_long_unclosed_fence_line` keeps
a 50,000-character fence line under 0.5 s. That's far above the linear
cost (under 1 ms), so a loaded runner won't trip it, but a quadratic
pattern (seconds) would. It passes on `main` too; it's there so any
future widening stays linear, as in
`test_multiline_regex_blank_runs.py`.
## Testing
- [x] Unit tests pass (`pytest`): `tests/test_mixed_content*.py`,
`tests/test_content_router*.py`,
`tests/test_transforms_content_router.py`,
`tests/test_code_compressor*.py`,
`tests/test_multiline_regex_blank_runs.py`, `tests/test_transforms/`,
not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests on `main` (fix reverted, tests kept):
```text
$ python -m pytest tests/test_mixed_content_sections.py -q
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c++-c++]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c#-c#]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[objective-c-objective-c]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[js {1,3}-js]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[python title="app.py"-python]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[ c++-c++]
========================= 6 failed, 6 passed in 1.53s ==========================
```
With the fix:
```text
$ python -m pytest tests/test_mixed_content*.py tests/test_content_router*.py tests/test_transforms_content_router.py \
tests/test_code_compressor*.py tests/test_multiline_regex_blank_runs.py tests/test_transforms -q
====================== 1044 passed, 88 skipped in 32.30s =======================
$ ruff check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
All checks passed!
$ ruff format --check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
2 files already formatted
```
## Real Behavior Proof
- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel),
`HEADROOM_OFFLINE=1`, public `headroom.compress()` API
- Exact command / steps: called `headroom.compress(messages,
model="gpt-4o", model_limit=128000, protect_recent=0)` on an
OpenAI-format conversation whose tool result is: one intro line, a ````
```c++ ```` block of two declarations, 40 prose lines, then a 400-row
JSON array (13.5K tokens). I printed the `transforms_applied`, every
fence line in the compressed tool result and its tail, on `main` and on
this branch. (Through the proxy, the same content only reaches the
mixed-content path once it is old enough to be compressed, so I used the
library entry point to keep the repro to one call.)
- Observed result: both runs take the mixed path (`router:mixed`). On
`main` the output has three fence lines (`c++`, a stray `unknown`, and a
closing one), the C++ block lost its closing fence, and the JSON tail
comes back as a JSON-escaped string inside the stray fence. On this
branch the C++ block keeps both fences, the prose stays prose, and the
JSON is compacted to CSV rows as the structured path does elsewhere.
```text
== main
tokens 13474 -> 4289 | transforms: [..., 'router:mixed:0.35']
fence lines: ['```c++', '```unknown', '```']
| Here is the Header file that we wrote for the project today.
| ```c++
| int add(int a, int b);
| int sub(int a, int b);
|
| ```unknown
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\\n398,item398,team-a,ok\\n399,item399,team-a,ok\\n"\n```'
== this branch
tokens 13474 -> 4285 | transforms: [..., 'router:mixed:0.29']
fence lines: ['```c++', '```']
| ```c++
| int add(int a, int b);
| int sub(int a, int b);
| ```
|
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\n398,item398,team-a,ok\n399,item399,team-a,ok\n'
```
- Not tested: the same content through the live proxy's mixed-content
path (in a short session the turn is protected or below the size gates,
so nothing is compressed on either branch); Rust — there is no Rust
mirror of `split_into_sections`.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: fences tagged with non-word
characters (or a space before the tag) are now parsed as code blocks.
Content with only `\w` tags (```` ```python ````, ```` ```json ````,
bare ```` ``` ````) splits exactly as before; the existing 565
router/transform tests pass unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- An **unclosed** fence still runs to the end of the input, as an
unclosed ```` ```python ```` already does on `main`. This PR only
changes which openers are recognised. Making unclosed fences fall back
to plain text would change behaviour for every tag, so it seems better
as its own PR.
- The router re-wraps a fenced section with ```` ```{section.language}
````, so trailing info-string metadata (`{1,3}`, `title=...`) is not
re-emitted. That was already true for every fence; this PR only fixes
the opener being missed.
- `_extract_fenced_command()` in `content_router.py` has its own
`[\w.-]+` fence class. It only extracts a shell command for
read-detection, so I left it alone to keep this PR to one change.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
603d873992
commit
a8df6f919f
@@ -29,7 +29,12 @@ class ContentSection:
|
||||
atomic: bool = False
|
||||
|
||||
|
||||
_CODE_FENCE_PATTERN = re.compile(r"^```(\w*)\s*$", re.MULTILINE)
|
||||
# Group 1 is the whole info string: any text without a backtick, so
|
||||
# ``c++``, ``c#``, ``objective-c``, ``js {1,3}`` and `` c++`` (space first) all
|
||||
# open a fence. ``\w*`` missed such openers and then took the closing fence
|
||||
# for one. A single quantifier keeps a long line linear; the language is the
|
||||
# info string's first word (see ``_fence_language``).
|
||||
_CODE_FENCE_PATTERN = re.compile(r"^```([^\n`]*)$", re.MULTILINE)
|
||||
# ``[^\S\n]*``, not ``\s*``: under MULTILINE, ``^\s*`` rescans every
|
||||
# following blank line from each line start (quadratic in a blank run).
|
||||
_JSON_BLOCK_START = re.compile(r"^[^\S\n]*[\[{]", re.MULTILINE)
|
||||
@@ -37,6 +42,12 @@ _SEARCH_RESULT_PATTERN = re.compile(r"^\S+:\d+:", re.MULTILINE)
|
||||
_PROSE_PATTERN = re.compile(r"[A-Z][a-z]+\s+\w+\s+\w+")
|
||||
|
||||
|
||||
def _fence_language(info_string: str) -> str:
|
||||
"""First word of a fence info string, or ``"unknown"`` for a bare fence."""
|
||||
words = info_string.split()
|
||||
return words[0] if words else "unknown"
|
||||
|
||||
|
||||
def _is_search_line(line: str) -> bool:
|
||||
"""True for grep match lines and ``-A``/``-B``/``-C`` context lines.
|
||||
|
||||
@@ -150,7 +161,7 @@ def split_into_sections(content: str, *, isolate: tuple[str, ...] = ()) -> list[
|
||||
continue
|
||||
|
||||
if match := _CODE_FENCE_PATTERN.match(line):
|
||||
language = match.group(1) or "unknown"
|
||||
language = _fence_language(match.group(1))
|
||||
code_lines = []
|
||||
start_line = i
|
||||
i += 1
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import pytest
|
||||
|
||||
from headroom.transforms.content_detector import ContentType
|
||||
from headroom.transforms.mixed_content import (
|
||||
_extract_json_block,
|
||||
@@ -49,6 +51,60 @@ def test_split_into_sections_preserves_typed_boundaries():
|
||||
assert sections[3].start_line == 5
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("info_string", "language"),
|
||||
[
|
||||
("c++", "c++"),
|
||||
("c#", "c#"),
|
||||
("objective-c", "objective-c"),
|
||||
("js {1,3}", "js"),
|
||||
('python title="app.py"', "python"),
|
||||
(" c++", "c++"),
|
||||
("python\r", "python"),
|
||||
],
|
||||
)
|
||||
def test_split_accepts_fence_info_strings_beyond_word_characters(info_string, language):
|
||||
content = "\n".join(
|
||||
[
|
||||
"Here is the header file we wrote today.",
|
||||
f"```{info_string}",
|
||||
"int add(int a, int b);",
|
||||
"```",
|
||||
"Please review this carefully and tell me what you think.",
|
||||
'{"name": "demo", "version": 1}',
|
||||
]
|
||||
)
|
||||
|
||||
sections = split_into_sections(content)
|
||||
|
||||
assert [section.content_type for section in sections] == [
|
||||
ContentType.PLAIN_TEXT,
|
||||
ContentType.SOURCE_CODE,
|
||||
ContentType.PLAIN_TEXT,
|
||||
ContentType.JSON_ARRAY,
|
||||
]
|
||||
assert sections[1].language == language
|
||||
assert sections[1].content == "int add(int a, int b);"
|
||||
assert sections[1].is_code_fence is True
|
||||
assert sections[2].content == "Please review this carefully and tell me what you think."
|
||||
|
||||
|
||||
def test_fence_pattern_stays_linear_on_a_long_unclosed_fence_line():
|
||||
import time
|
||||
|
||||
from headroom.transforms.mixed_content import _CODE_FENCE_PATTERN
|
||||
|
||||
line = "```" + "a" * 50_000 + "`"
|
||||
best = float("inf")
|
||||
for _ in range(3):
|
||||
start = time.perf_counter()
|
||||
assert _CODE_FENCE_PATTERN.match(line) is None
|
||||
best = min(best, time.perf_counter() - start)
|
||||
# Linear: well under 1 ms here. A quadratic pattern takes seconds, so a
|
||||
# generous budget still catches it on a loaded runner.
|
||||
assert best < 0.5, f"{best * 1000:.1f}ms on a 50,000-character fence line"
|
||||
|
||||
|
||||
def test_split_carves_grep_context_lines_into_search_sections():
|
||||
"""Regression (#3580): prose + ``grep -A`` output keeps code out of text.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user