fix(transforms): keep the fences of a code block passed through for a protected tag (#4292)

## Description

In mixed-content routing (`ContentRouter._compress_mixed`), custom
XML-style tags are first swapped for placeholders, and any section that
contains a placeholder is passed through verbatim so the tag can be
restored (#2887). For a fenced code section, `section.content` is only
the **body**: `split_into_sections()` strips the ```` ```lang ```` and
```` ``` ```` lines. The compression branch puts them back:

```python
if section.is_code_fence and section.language:
    compressed_content = f"```{section.language}\n{compressed_content}\n```"
```

The passthrough branch doesn't, so a fenced block whose body holds a
non-HTML tag loses both fence lines. That covers XML config snippets,
templates and `<config-block>`-style examples. The model then sees the
snippet as bare text running into the surrounding prose. It happens
whenever such a block sits in a mixed tool result, for example a docs
page or a file listing.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_router.py`: in the
placeholder-passthrough branch of `_compress_mixed`, a fenced section is
emitted from its **source lines** (`start_line`..`end_line` of the text
that was split) instead of its body. The fence lines come back exactly
as written:
  - an ```` ```xml ```` fence keeps its tag;
- a bare ```` ``` ```` stays bare rather than becoming ```` ```unknown
````;
- a fence left open at the end of the input stays open, with no closing
marker invented.

The body, including its placeholders, is unchanged byte for byte, so
`restore_tags` behaves as before. Token accounting for the section is
unchanged. Non-fenced sections and the compression branch are untouched.
- `tests/test_content_router_mixed_fence_placeholder.py` (new):
- `test_mixed_passthrough_of_protected_fenced_block_keeps_fence_lines`:
an ```` ```xml ```` block with a `<config-block>` tag inside mixed text
comes out with its opening fence, three body lines and closing fence in
order;
- `test_mixed_passthrough_keeps_an_unclosed_bare_fence_as_written`: an
unclosed bare fence at the end of the input comes out unclosed and
without `unknown`.

## Testing

- [x] Unit tests pass (`pytest`): the new tests plus
`tests/test_ccr_tag_placeholder_regression.py`,
`tests/test_tag_protector_invariant.py`, `tests/test_transforms/`,
`tests/test_transforms_content_router.py`,
`tests/test_mixed_content*.py`, `tests/test_content_router*.py`, not the
full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New test on `main` (fix reverted, test kept):

```text
$ python -m pytest tests/test_content_router_mixed_fence_placeholder.py -q
FAILED tests/test_content_router_mixed_fence_placeholder.py::test_mixed_passthrough_of_protected_fenced_block_keeps_fence_lines
FAILED tests/test_content_router_mixed_fence_placeholder.py::test_mixed_passthrough_keeps_an_unclosed_bare_fence_as_written
============================== 2 failed in 2.67s ===============================
# first test: assert '```xml' in ['Here is the Template file that we wrote for the project today.', '', '<config-block>', ...]
```

With the fix:

```text
$ python -m pytest tests/test_content_router_mixed_fence_placeholder.py tests/test_ccr_tag_placeholder_regression.py \
    tests/test_tag_protector_invariant.py tests/test_transforms tests/test_transforms_content_router.py \
    tests/test_mixed_content*.py tests/test_content_router*.py -q
======================= 943 passed, 80 skipped in 36.86s =======================

$ ruff check headroom/transforms/content_router.py tests/test_content_router_mixed_fence_placeholder.py
All checks passed!
$ ruff format --check headroom/transforms/content_router.py tests/test_content_router_mixed_fence_placeholder.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel),
`HEADROOM_OFFLINE=1`, public `headroom.compress()` API
- Exact command / steps: called `headroom.compress(messages,
model="gpt-4o", model_limit=128000, protect_recent=0)` on an
OpenAI-format conversation whose tool result is an intro line, an ````
```xml ```` block containing
`<config-block><name>demo</name></config-block>`, 40 prose lines and a
400-row JSON array (13.5K tokens). I printed every fence line in the
compressed tool result and its first lines, on `main` and on this
branch.
- Observed result: both runs take the mixed path (`router:mixed:0.29`)
and compress about equally (13474 → 4280 / 4285 tokens). On `main` the
compressed tool result has no fence lines at all. On this branch the
```` ```xml ```` … ```` ``` ```` block is intact.

  ```text
  == main
  fence lines: []
    | Here is the Template file that we wrote for the project today.
    |
    | <config-block>
    |   <name>demo</name>
    | </config-block>
    |
| Paragraph 0 explains the Design choice that we made for this module
today.
  == this branch
  fence lines: ['```xml', '```']
    | Here is the Template file that we wrote for the project today.
    |
    | ```xml
    | <config-block>
    |   <name>demo</name>
    | </config-block>
    | ```
  ```
- Not tested: the live proxy (in a short session the turn is protected
or below the size gates, so mixed routing doesn't run). The compression
branch still re-wraps compressed code sections with ```` ```{language}
```` (so a bare fence there becomes ```` ```unknown ````); that's a
separate issue I left out of this PR.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only fenced blocks that contain a
protected tag: they now come back with their original fence lines (a few
tokens). All other sections are routed exactly as before.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Updated after review: the first revision re-wrapped the body with ````
```{language} ```` like the compression branch does, which invented a
closing fence for an unclosed block. Emitting the source lines is
simpler and truly verbatim.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Rudy Mizrahi Celekli
2026-10-24 08:38:01 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent f59c8ffcd8
commit 02958c6ffe
2 changed files with 61 additions and 1 deletions
+8 -1
View File
@@ -3453,6 +3453,7 @@ class ContentRouter(Transform):
# Any section carrying a placeholder is passed through verbatim
# instead of ever entering a compressor.
placeholders = [placeholder for placeholder, _ in protected]
source_lines = sections_source.split("\n")
compressed_sections: list[str] = []
routing_log: list[RoutingDecision] = []
@@ -3460,7 +3461,13 @@ class ContentRouter(Transform):
for i, section in enumerate(sections):
if placeholders and any(ph in section.content for ph in placeholders):
section_tokens = _estimate_tokens(section.content)
compressed_sections.append(section.content)
passthrough = section.content
# A fenced section's content is only the body. Emit its source
# lines instead, so the fence lines come back exactly as they
# were (and a fence left open stays open).
if section.is_code_fence:
passthrough = "\n".join(source_lines[section.start_line : section.end_line + 1])
compressed_sections.append(passthrough)
routing_log.append(
RoutingDecision(
content_type=section.content_type,
@@ -0,0 +1,53 @@
"""A fenced block passed through verbatim (it holds a protected tag) keeps its fences."""
from __future__ import annotations
from headroom.transforms.content_router import ContentRouter
CONTENT = """Here is the Template file that we wrote for the project today.
```xml
<config-block>
<name>demo</name>
</config-block>
```
Please Review this carefully and tell me what you think about it.
The Second paragraph explains the Design choice that we made here.
{"name": "demo", "version": 1, "debug": true}
Then the Search results came back from the Second query run today.
src/main.c:10: int main() {
src/util.c:22: void helper() {
"""
def test_mixed_passthrough_of_protected_fenced_block_keeps_fence_lines():
result = ContentRouter()._compress_mixed(CONTENT, context="")
lines = result.compressed.splitlines()
assert "```xml" in lines
opening = lines.index("```xml")
assert lines[opening + 1 : opening + 5] == [
"<config-block>",
" <name>demo</name>",
"</config-block>",
"```",
]
UNCLOSED_BARE = """Here is the Template file that we wrote for the project today.
Please Review this carefully and tell me what you think about it.
{"name": "demo", "version": 1, "debug": true}
Then the Search results came back from the Second query run today.
src/main.c:10: int main() {
```
<config-block>
<name>demo</name>
</config-block>"""
def test_mixed_passthrough_keeps_an_unclosed_bare_fence_as_written():
"""Passthrough is verbatim: no closing fence is invented and a bare fence stays bare."""
result = ContentRouter()._compress_mixed(UNCLOSED_BARE, context="")
lines = result.compressed.splitlines()
assert lines[-4:] == ["```", "<config-block>", " <name>demo</name>", "</config-block>"]
assert "```unknown" not in lines