fix(transforms): recognise code fences whose info string is not a bare word (#4291)

## Description

`split_into_sections()` in `headroom/transforms/mixed_content.py`
recognises an opening code fence with `^```(\w*)\s*$`. Common fence tags
with non-word characters don't match: ```` ```c++ ````, ```` ```c# ````,
```` ```objective-c ````, ```` ```js {1,3} ````, ```` ```python
title="app.py" ````. When the opener is missed:

1. the fence line and code are kept as plain text;
2. the block's **closing** ```` ``` ```` matches the pattern (empty
tag), so it is taken as an **opening** fence;
3. everything after it (prose, JSON, until the next ```` ``` ```` or end
of input) becomes one `SOURCE_CODE` section with language `unknown`,
goes to the code compressor, and is re-emitted wrapped in ````
```unknown ```` … ```` ``` ````.

So the code block loses its closing fence, and the prose and data that
follow it come back inside a fake code block. Through
`headroom.compress()` this turns a tool result's JSON into a
JSON-escaped string (`\n` sequences) inside that stray fence (see the
proof below). It only takes one C++ or C# snippet in a mixed tool result
or fetched page.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/mixed_content.py`:
- `_CODE_FENCE_PATTERN` is now `^```([^\n`]*)$`. Group 1 is the whole
info string, any text without a backtick, so ```` ```c++ ````, ````
```js {1,3} ```` and ```` ``` c++ ```` (space before the tag, valid
Markdown) all open a fence. A closing ```` ``` ```` and a plain ````
```python ```` match as before. A line with a later backtick (````
```foo``` ````) still doesn't.
- New `_fence_language()` takes the info string's first word as the
language, and returns `"unknown"` for a bare fence, as before.
- It's a single quantifier, so a 200,000-character fence line takes
about 2 ms. While working on this I measured an overlapping variant
(`([^\s`]*)[^\n`]*`) at 3.3 s on a 20,000-character line, and avoided
it.
- `tests/test_mixed_content_sections.py`:
- `test_split_accepts_fence_info_strings_beyond_word_characters`,
parametrized over `c++`, `c#`, `objective-c`, `js {1,3}`, `python
title="app.py"`, ` c++` and `python\r`, checks the sections come out
PLAIN_TEXT / SOURCE_CODE(lang) / PLAIN_TEXT / JSON_ARRAY with the code
body intact.
- `test_fence_pattern_stays_linear_on_a_long_unclosed_fence_line` keeps
a 50,000-character fence line under 0.5 s. That's far above the linear
cost (under 1 ms), so a loaded runner won't trip it, but a quadratic
pattern (seconds) would. It passes on `main` too; it's there so any
future widening stays linear, as in
`test_multiline_regex_blank_runs.py`.

## Testing

- [x] Unit tests pass (`pytest`): `tests/test_mixed_content*.py`,
`tests/test_content_router*.py`,
`tests/test_transforms_content_router.py`,
`tests/test_code_compressor*.py`,
`tests/test_multiline_regex_blank_runs.py`, `tests/test_transforms/`,
not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests on `main` (fix reverted, tests kept):

```text
$ python -m pytest tests/test_mixed_content_sections.py -q
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c++-c++]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c#-c#]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[objective-c-objective-c]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[js {1,3}-js]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[python title="app.py"-python]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[ c++-c++]
========================= 6 failed, 6 passed in 1.53s ==========================
```

With the fix:

```text
$ python -m pytest tests/test_mixed_content*.py tests/test_content_router*.py tests/test_transforms_content_router.py \
    tests/test_code_compressor*.py tests/test_multiline_regex_blank_runs.py tests/test_transforms -q
====================== 1044 passed, 88 skipped in 32.30s =======================

$ ruff check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
All checks passed!
$ ruff format --check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel),
`HEADROOM_OFFLINE=1`, public `headroom.compress()` API
- Exact command / steps: called `headroom.compress(messages,
model="gpt-4o", model_limit=128000, protect_recent=0)` on an
OpenAI-format conversation whose tool result is: one intro line, a ````
```c++ ```` block of two declarations, 40 prose lines, then a 400-row
JSON array (13.5K tokens). I printed the `transforms_applied`, every
fence line in the compressed tool result and its tail, on `main` and on
this branch. (Through the proxy, the same content only reaches the
mixed-content path once it is old enough to be compressed, so I used the
library entry point to keep the repro to one call.)
- Observed result: both runs take the mixed path (`router:mixed`). On
`main` the output has three fence lines (`c++`, a stray `unknown`, and a
closing one), the C++ block lost its closing fence, and the JSON tail
comes back as a JSON-escaped string inside the stray fence. On this
branch the C++ block keeps both fences, the prose stays prose, and the
JSON is compacted to CSV rows as the structured path does elsewhere.

  ```text
  == main
  tokens 13474 -> 4289 | transforms: [..., 'router:mixed:0.35']
  fence lines: ['```c++', '```unknown', '```']
    | Here is the Header file that we wrote for the project today.
    | ```c++
    | int add(int a, int b);
    | int sub(int a, int b);
    |
    | ```unknown
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\\n398,item398,team-a,ok\\n399,item399,team-a,ok\\n"\n```'
  == this branch
  tokens 13474 -> 4285 | transforms: [..., 'router:mixed:0.29']
  fence lines: ['```c++', '```']
    | ```c++
    | int add(int a, int b);
    | int sub(int a, int b);
    | ```
    |
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\n398,item398,team-a,ok\n399,item399,team-a,ok\n'
  ```
- Not tested: the same content through the live proxy's mixed-content
path (in a short session the turn is protected or below the size gates,
so nothing is compressed on either branch); Rust — there is no Rust
mirror of `split_into_sections`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: fences tagged with non-word
characters (or a space before the tag) are now parsed as code blocks.
Content with only `\w` tags (```` ```python ````, ```` ```json ````,
bare ```` ``` ````) splits exactly as before; the existing 565
router/transform tests pass unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- An **unclosed** fence still runs to the end of the input, as an
unclosed ```` ```python ```` already does on `main`. This PR only
changes which openers are recognised. Making unclosed fences fall back
to plain text would change behaviour for every tag, so it seems better
as its own PR.
- The router re-wraps a fenced section with ```` ```{section.language}
````, so trailing info-string metadata (`{1,3}`, `title=...`) is not
re-emitted. That was already true for every fence; this PR only fixes
the opener being missed.
- `_extract_fenced_command()` in `content_router.py` has its own
`[\w.-]+` fence class. It only extracts a shell command for
read-detection, so I left it alone to keep this PR to one change.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Rudy Mizrahi Celekli
2026-11-04 22:44:06 -06:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 603d873992
commit a8df6f919f
2 changed files with 69 additions and 2 deletions
+13 -2
View File
@@ -29,7 +29,12 @@ class ContentSection:
atomic: bool = False
_CODE_FENCE_PATTERN = re.compile(r"^```(\w*)\s*$", re.MULTILINE)
# Group 1 is the whole info string: any text without a backtick, so
# ``c++``, ``c#``, ``objective-c``, ``js {1,3}`` and `` c++`` (space first) all
# open a fence. ``\w*`` missed such openers and then took the closing fence
# for one. A single quantifier keeps a long line linear; the language is the
# info string's first word (see ``_fence_language``).
_CODE_FENCE_PATTERN = re.compile(r"^```([^\n`]*)$", re.MULTILINE)
# ``[^\S\n]*``, not ``\s*``: under MULTILINE, ``^\s*`` rescans every
# following blank line from each line start (quadratic in a blank run).
_JSON_BLOCK_START = re.compile(r"^[^\S\n]*[\[{]", re.MULTILINE)
@@ -37,6 +42,12 @@ _SEARCH_RESULT_PATTERN = re.compile(r"^\S+:\d+:", re.MULTILINE)
_PROSE_PATTERN = re.compile(r"[A-Z][a-z]+\s+\w+\s+\w+")
def _fence_language(info_string: str) -> str:
"""First word of a fence info string, or ``"unknown"`` for a bare fence."""
words = info_string.split()
return words[0] if words else "unknown"
def _is_search_line(line: str) -> bool:
"""True for grep match lines and ``-A``/``-B``/``-C`` context lines.
@@ -150,7 +161,7 @@ def split_into_sections(content: str, *, isolate: tuple[str, ...] = ()) -> list[
continue
if match := _CODE_FENCE_PATTERN.match(line):
language = match.group(1) or "unknown"
language = _fence_language(match.group(1))
code_lines = []
start_line = i
i += 1
+56
View File
@@ -1,3 +1,5 @@
import pytest
from headroom.transforms.content_detector import ContentType
from headroom.transforms.mixed_content import (
_extract_json_block,
@@ -49,6 +51,60 @@ def test_split_into_sections_preserves_typed_boundaries():
assert sections[3].start_line == 5
@pytest.mark.parametrize(
("info_string", "language"),
[
("c++", "c++"),
("c#", "c#"),
("objective-c", "objective-c"),
("js {1,3}", "js"),
('python title="app.py"', "python"),
(" c++", "c++"),
("python\r", "python"),
],
)
def test_split_accepts_fence_info_strings_beyond_word_characters(info_string, language):
content = "\n".join(
[
"Here is the header file we wrote today.",
f"```{info_string}",
"int add(int a, int b);",
"```",
"Please review this carefully and tell me what you think.",
'{"name": "demo", "version": 1}',
]
)
sections = split_into_sections(content)
assert [section.content_type for section in sections] == [
ContentType.PLAIN_TEXT,
ContentType.SOURCE_CODE,
ContentType.PLAIN_TEXT,
ContentType.JSON_ARRAY,
]
assert sections[1].language == language
assert sections[1].content == "int add(int a, int b);"
assert sections[1].is_code_fence is True
assert sections[2].content == "Please review this carefully and tell me what you think."
def test_fence_pattern_stays_linear_on_a_long_unclosed_fence_line():
import time
from headroom.transforms.mixed_content import _CODE_FENCE_PATTERN
line = "```" + "a" * 50_000 + "`"
best = float("inf")
for _ in range(3):
start = time.perf_counter()
assert _CODE_FENCE_PATTERN.match(line) is None
best = min(best, time.perf_counter() - start)
# Linear: well under 1 ms here. A quadratic pattern takes seconds, so a
# generous budget still catches it on a loaded runner.
assert best < 0.5, f"{best * 1000:.1f}ms on a 50,000-character fence line"
def test_split_carves_grep_context_lines_into_search_sections():
"""Regression (#3580): prose + ``grep -A`` output keeps code out of text.