Files
headroom/tests/test_mixed_content_sections.py
a8df6f919f fix(transforms): recognise code fences whose info string is not a bare word (#4291)
## Description

`split_into_sections()` in `headroom/transforms/mixed_content.py`
recognises an opening code fence with `^```(\w*)\s*$`. Common fence tags
with non-word characters don't match: ```` ```c++ ````, ```` ```c# ````,
```` ```objective-c ````, ```` ```js {1,3} ````, ```` ```python
title="app.py" ````. When the opener is missed:

1. the fence line and code are kept as plain text;
2. the block's **closing** ```` ``` ```` matches the pattern (empty
tag), so it is taken as an **opening** fence;
3. everything after it (prose, JSON, until the next ```` ``` ```` or end
of input) becomes one `SOURCE_CODE` section with language `unknown`,
goes to the code compressor, and is re-emitted wrapped in ````
```unknown ```` … ```` ``` ````.

So the code block loses its closing fence, and the prose and data that
follow it come back inside a fake code block. Through
`headroom.compress()` this turns a tool result's JSON into a
JSON-escaped string (`\n` sequences) inside that stray fence (see the
proof below). It only takes one C++ or C# snippet in a mixed tool result
or fetched page.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/mixed_content.py`:
- `_CODE_FENCE_PATTERN` is now `^```([^\n`]*)$`. Group 1 is the whole
info string, any text without a backtick, so ```` ```c++ ````, ````
```js {1,3} ```` and ```` ``` c++ ```` (space before the tag, valid
Markdown) all open a fence. A closing ```` ``` ```` and a plain ````
```python ```` match as before. A line with a later backtick (````
```foo``` ````) still doesn't.
- New `_fence_language()` takes the info string's first word as the
language, and returns `"unknown"` for a bare fence, as before.
- It's a single quantifier, so a 200,000-character fence line takes
about 2 ms. While working on this I measured an overlapping variant
(`([^\s`]*)[^\n`]*`) at 3.3 s on a 20,000-character line, and avoided
it.
- `tests/test_mixed_content_sections.py`:
- `test_split_accepts_fence_info_strings_beyond_word_characters`,
parametrized over `c++`, `c#`, `objective-c`, `js {1,3}`, `python
title="app.py"`, ` c++` and `python\r`, checks the sections come out
PLAIN_TEXT / SOURCE_CODE(lang) / PLAIN_TEXT / JSON_ARRAY with the code
body intact.
- `test_fence_pattern_stays_linear_on_a_long_unclosed_fence_line` keeps
a 50,000-character fence line under 0.5 s. That's far above the linear
cost (under 1 ms), so a loaded runner won't trip it, but a quadratic
pattern (seconds) would. It passes on `main` too; it's there so any
future widening stays linear, as in
`test_multiline_regex_blank_runs.py`.

## Testing

- [x] Unit tests pass (`pytest`): `tests/test_mixed_content*.py`,
`tests/test_content_router*.py`,
`tests/test_transforms_content_router.py`,
`tests/test_code_compressor*.py`,
`tests/test_multiline_regex_blank_runs.py`, `tests/test_transforms/`,
not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests on `main` (fix reverted, tests kept):

```text
$ python -m pytest tests/test_mixed_content_sections.py -q
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c++-c++]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[c#-c#]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[objective-c-objective-c]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[js {1,3}-js]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[python title="app.py"-python]
FAILED tests/test_mixed_content_sections.py::test_split_accepts_fence_info_strings_beyond_word_characters[ c++-c++]
========================= 6 failed, 6 passed in 1.53s ==========================
```

With the fix:

```text
$ python -m pytest tests/test_mixed_content*.py tests/test_content_router*.py tests/test_transforms_content_router.py \
    tests/test_code_compressor*.py tests/test_multiline_regex_blank_runs.py tests/test_transforms -q
====================== 1044 passed, 88 skipped in 32.30s =======================

$ ruff check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
All checks passed!
$ ruff format --check headroom/transforms/mixed_content.py tests/test_mixed_content_sections.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel),
`HEADROOM_OFFLINE=1`, public `headroom.compress()` API
- Exact command / steps: called `headroom.compress(messages,
model="gpt-4o", model_limit=128000, protect_recent=0)` on an
OpenAI-format conversation whose tool result is: one intro line, a ````
```c++ ```` block of two declarations, 40 prose lines, then a 400-row
JSON array (13.5K tokens). I printed the `transforms_applied`, every
fence line in the compressed tool result and its tail, on `main` and on
this branch. (Through the proxy, the same content only reaches the
mixed-content path once it is old enough to be compressed, so I used the
library entry point to keep the repro to one call.)
- Observed result: both runs take the mixed path (`router:mixed`). On
`main` the output has three fence lines (`c++`, a stray `unknown`, and a
closing one), the C++ block lost its closing fence, and the JSON tail
comes back as a JSON-escaped string inside the stray fence. On this
branch the C++ block keeps both fences, the prose stays prose, and the
JSON is compacted to CSV rows as the structured path does elsewhere.

  ```text
  == main
  tokens 13474 -> 4289 | transforms: [..., 'router:mixed:0.35']
  fence lines: ['```c++', '```unknown', '```']
    | Here is the Header file that we wrote for the project today.
    | ```c++
    | int add(int a, int b);
    | int sub(int a, int b);
    |
    | ```unknown
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\\n398,item398,team-a,ok\\n399,item399,team-a,ok\\n"\n```'
  == this branch
  tokens 13474 -> 4285 | transforms: [..., 'router:mixed:0.29']
  fence lines: ['```c++', '```']
    | ```c++
    | int add(int a, int b);
    | int sub(int a, int b);
    | ```
    |
| Paragraph 0 explains the Design choice that we made for this module
today.
tail:
'...397,item397,team-a,ok\n398,item398,team-a,ok\n399,item399,team-a,ok\n'
  ```
- Not tested: the same content through the live proxy's mixed-content
path (in a short session the turn is protected or below the size gates,
so nothing is compressed on either branch); Rust — there is no Rust
mirror of `split_into_sections`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: fences tagged with non-word
characters (or a space before the tag) are now parsed as code blocks.
Content with only `\w` tags (```` ```python ````, ```` ```json ````,
bare ```` ``` ````) splits exactly as before; the existing 565
router/transform tests pass unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- An **unclosed** fence still runs to the end of the input, as an
unclosed ```` ```python ```` already does on `main`. This PR only
changes which openers are recognised. Making unclosed fences fall back
to plain text would change behaviour for every tag, so it seems better
as its own PR.
- The router re-wraps a fenced section with ```` ```{section.language}
````, so trailing info-string metadata (`{1,3}`, `title=...`) is not
re-emitted. That was already true for every fence; this PR only fixes
the opener being missed.
- `_extract_fenced_command()` in `content_router.py` has its own
`[\w.-]+` fence class. It only extracts a shell command for
read-detection, so I left it alone to keep this PR to one change.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:44:06 -06:00

176 lines
5.8 KiB
Python

import pytest
from headroom.transforms.content_detector import ContentType
from headroom.transforms.mixed_content import (
_extract_json_block,
is_mixed_content,
split_into_sections,
)
def test_mixed_content_detection_requires_multiple_signals():
prose = "\n".join(
[
"First sentence has enough words to count.",
"Second sentence has enough words to count.",
"Third sentence has enough words to count.",
"Fourth sentence has enough words to count.",
"Fifth sentence has enough words to count.",
"Sixth sentence has enough words to count.",
]
)
assert is_mixed_content(prose) is False
assert is_mixed_content(f"{prose}\n```python\nprint('x')\n```") is True
def test_split_into_sections_preserves_typed_boundaries():
content = "\n".join(
[
"Intro text",
"```python",
"print('x')",
"```",
'[{"id": 1}]',
"src/app.py:10:print('x')",
]
)
sections = split_into_sections(content)
assert [section.content_type for section in sections] == [
ContentType.PLAIN_TEXT,
ContentType.SOURCE_CODE,
ContentType.JSON_ARRAY,
ContentType.SEARCH_RESULTS,
]
assert sections[1].language == "python"
assert sections[1].content == "print('x')"
assert sections[1].is_code_fence is True
assert sections[2].content == '[{"id": 1}]'
assert sections[3].start_line == 5
@pytest.mark.parametrize(
("info_string", "language"),
[
("c++", "c++"),
("c#", "c#"),
("objective-c", "objective-c"),
("js {1,3}", "js"),
('python title="app.py"', "python"),
(" c++", "c++"),
("python\r", "python"),
],
)
def test_split_accepts_fence_info_strings_beyond_word_characters(info_string, language):
content = "\n".join(
[
"Here is the header file we wrote today.",
f"```{info_string}",
"int add(int a, int b);",
"```",
"Please review this carefully and tell me what you think.",
'{"name": "demo", "version": 1}',
]
)
sections = split_into_sections(content)
assert [section.content_type for section in sections] == [
ContentType.PLAIN_TEXT,
ContentType.SOURCE_CODE,
ContentType.PLAIN_TEXT,
ContentType.JSON_ARRAY,
]
assert sections[1].language == language
assert sections[1].content == "int add(int a, int b);"
assert sections[1].is_code_fence is True
assert sections[2].content == "Please review this carefully and tell me what you think."
def test_fence_pattern_stays_linear_on_a_long_unclosed_fence_line():
import time
from headroom.transforms.mixed_content import _CODE_FENCE_PATTERN
line = "```" + "a" * 50_000 + "`"
best = float("inf")
for _ in range(3):
start = time.perf_counter()
assert _CODE_FENCE_PATTERN.match(line) is None
best = min(best, time.perf_counter() - start)
# Linear: well under 1 ms here. A quadratic pattern takes seconds, so a
# generous budget still catches it on a loaded runner.
assert best < 0.5, f"{best * 1000:.1f}ms on a 50,000-character fence line"
def test_split_carves_grep_context_lines_into_search_sections():
"""Regression (#3580): prose + ``grep -A`` output keeps code out of text.
Context lines (``path-NN-content``) must join the SEARCH_RESULTS section
instead of stranding source code in a PLAIN_TEXT section, where the
word-dropping prose compressor would corrupt it. ``--`` group separators
ride along as ordinary text lines.
"""
content = "\n".join(
[
"Some intro prose explaining the search.",
"src/main.py-40- context before",
"src/main.py:42:def process_data(items):",
"src/main.py-43- context after",
"--",
"src/other.py:12: return True",
"Trailing prose after the results.",
]
)
sections = split_into_sections(content)
# The ``--`` group separator breaks the run, so the two grep groups keep
# their own SEARCH_RESULTS sections instead of merging through it.
assert [section.content_type for section in sections] == [
ContentType.PLAIN_TEXT,
ContentType.SEARCH_RESULTS,
ContentType.PLAIN_TEXT,
ContentType.SEARCH_RESULTS,
ContentType.PLAIN_TEXT,
]
assert "src/main.py-40- context before" in sections[1].content
assert "src/main.py:42:def process_data(items):" in sections[1].content
assert "src/main.py-43- context after" in sections[1].content
assert "src/other.py:12: return True" in sections[3].content
def test_extract_json_block_ignores_delimiters_inside_strings():
lines = [
"[",
' {"path": "a]b", "message": "keep {literal} braces"},',
' {"path": "c"}',
"]",
]
block, end_line = _extract_json_block(lines, 0)
assert end_line == 3
assert block == "\n".join(lines)
def test_split_does_not_carve_timestamped_log_rows_as_search():
"""Regression (#3736 residual): the detector rejects ISO-timestamped log
rows as search output; the splitter must not carve them either, while
real ``path:NN:content`` grep lines still carve.
"""
logs = [
"2026-09-23T10:05:07Z worker-1 INFO request handled",
"2026-09-23T10:05:13Z worker-2 ERROR upstream timeout",
]
content = "\n".join(["Intro text", "```python", "print('x')", "```", *logs])
assert ContentType.SEARCH_RESULTS not in [
section.content_type for section in split_into_sections(content)
]
grep = "\n".join(["Intro text", "```python", "print('x')", "```", "src/app.py:10:print('x')"])
assert split_into_sections(grep)[-1].content_type == ContentType.SEARCH_RESULTS