fix(tag-protector): keep the enclosing tag open after a nested close in marker mode (#4682)

## Description

With `compress_tagged_content=true` (marker mode), `tag_protector`
records each custom open and close tag as its own placeholder and keeps
the body compressible. When a close tag matched an open frame on the
stack, the code ran `stack.truncate(stack_idx)` and then `stack.pop()`
(`crates/headroom-core/src/transforms/tag_protector.rs`, about line 571
on `12864fc8`). `truncate(stack_idx)` already removes the matched frame
and everything above it, so the extra `pop` removed the frame of the
enclosing tag. That tag's close then matched nothing, was counted as an
orphan close and stayed in the text as a raw tag. Block mode already did
only the `truncate`.

This is reachable on the default batch path:
`headroom/transforms/compression_batches.py` calls
`protect_tags(envelope, compress_tagged_content=True)`, and the envelope
wraps every unit in its own `<headroom-batch-{nonce}-{id}>` tag. Any
unit that holds one custom tag pair (for example
`<tool_call>…</tool_call>`) therefore lost the placeholder for its
envelope close, which went to the router as raw text.

With this PR the extra `pop` is removed and the comment explains why.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/transforms/tag_protector.rs`: marker-mode
close matching keeps only `stack.truncate(stack_idx)`; the comment now
says that `truncate` already drops the matched frame.
- Same file, tests: `marker_mode_nested_custom_tags_keep_outer_close`
checks that `<outer><inner>body</inner></outer>` gives four
placeholders, no raw `</outer>`, zero orphan closes, and an exact round
trip through `restore_tags`.

## Testing

- [x] Unit tests pass (`pytest`) (Rust only: `cargo test -p
headroom-core --lib`; no Python changed)
- [x] Linting passes (`ruff check .`) (no Python changed; `cargo fmt
--check` and `cargo clippy -D warnings` on headroom-core instead)
- [ ] Type checking passes (`mypy headroom`) (no Python changed)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# fail before (tag_protector.rs at 12864fc8 with only the new test added)
cargo test -p headroom-core --lib tag_protector: 31 passed; 1 failed
  marker_mode_nested_custom_tags_keep_outer_close ... FAILED
  cleaned = {{HEADROOM_TAG_0}}{{HEADROOM_TAG_1}}body{{HEADROOM_TAG_2}}</outer>

# pass after
cargo test -p headroom-core --lib tag_protector: 32 passed; 0 failed
cargo test -p headroom-core --lib: 1034 passed; 0 failed; 1 ignored
cargo fmt -p headroom-core -- --check: clean
cargo clippy -p headroom-core --lib -- -D warnings: clean
```

## Real Behavior Proof

- Environment: before: Ubuntu (WSL2) on Windows 11, Python 3.12.15, the
`headroom._core` extension from a 2026-10-29 build (`tag_protector.rs`
has not changed since 2026-05-02, so it matches `12864fc8`); after:
Windows 11, Rust 1.95.0 (`rust-toolchain.toml`), headroom-core at
`12864fc8` plus this change.
- Exact command / steps: before: called the shipped binding
`_core.protect_tags(text, True)` on the shape the batch path builds,
`<headroom-batch-abc-1><tool_call>do
it</tool_call></headroom-batch-abc-1>`, offline, on a copy of the
extension; after: `cargo test -p headroom-core --lib` with the new test
on a nested pair of custom tags.
- Observed result: before, the cleaned text was
`{{HEADROOM_TAG_0}}{{HEADROOM_TAG_1}}do
it{{HEADROOM_TAG_2}}</headroom-batch-abc-1>`: three placeholders, and
the envelope close left raw; after, the nested input gives four
placeholders with no raw close tag and restores exactly, the new test
passes and the rest of the headroom-core library tests pass.
- Not tested: the Python extension rebuilt from this branch (I could not
build the PyO3 wheel on this machine), and what a real compressor then
does with the raw envelope close in `compress_batch_with_router`
(whether the batch falls back to passthrough or the envelope still
parses).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly: in marker mode, the
close tag of a tag that encloses another custom tag is now protected
instead of left raw.
- Kill switch / disable path: none added.
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

A separate PR fixes a different defect in the same file (a `/` in an
unquoted attribute value made the tag self-closing); the two changes do
not overlap.

Produced with AI assistance (Claude Code) operated by me (HetCreep). The
tests and the binding call were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4682"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
HetCreep
2026-11-07 20:38:20 -06:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 8f6366cf66
commit 385d487f41
@@ -564,12 +564,14 @@ fn identify_spans(
match matching {
Some(stack_idx) => {
if compress_tagged_content {
// Pop everything above (orphan opens
// inside the matched span — their open
// markers were already recorded as spans
// and we keep them).
// Drop the matched frame and everything
// above it (orphan opens inside the matched
// span — their open markers were already
// recorded as spans and we keep them).
// `truncate` already removes the matched
// frame; popping again would drop the
// enclosing tag's frame and orphan its close.
stack.truncate(stack_idx);
let _ = stack.pop();
spans.push(Span {
start: i,
end: tag_end,
@@ -866,6 +868,17 @@ mod tests {
assert_eq!(blocks.len(), 2);
}
#[test]
fn marker_mode_nested_custom_tags_keep_outer_close() {
let text = "<outer><inner>body</inner></outer>";
let (cleaned, blocks, stats) = protect_tags(text, true);
assert_eq!(blocks.len(), 4, "{cleaned}");
assert!(!cleaned.contains("</outer>"), "{cleaned}");
assert!(cleaned.contains("body"));
assert_eq!(stats.orphan_closes, 0);
assert_eq!(restore_tags(&cleaned, &blocks), text);
}
#[test]
fn restore_basic() {
let original = "Before <system-reminder>Rule</system-reminder> After";