fix(smart-crusher): match audit-safe protected_patterns against unescaped text (#4294)

## Description

Audit-safe mode (`SmartCrusherConfig(audit_safe=True,
protected_patterns=[...])`, #1705) guarantees that rows matching a
protected regex survive compression. The regexes run against
`SmartCrusher._canon(row)`, which is `json.dumps(row, sort_keys=True,
default=str)`. `json.dumps` defaults to `ensure_ascii=True`, so every
non-ASCII character is escaped before the pattern sees it:

```python
_canon({"city": "Kraków"})   # '{"city": "Krak\\u00f3w"}'
re.search("Kraków", ...)     # None
```

A pattern that contains any non-ASCII character can never match:
accented names, CJK error text, currency symbols such as `€`.
`_scan_protected_rows` returns `[]`, nothing is spliced back or
verified, and the row is sampled out like any other. Nothing warns,
because from the crusher's point of view there was nothing to protect.
That's the exact failure audit-safe mode exists to prevent.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/smart_crusher.py`: `_canon()` serializes with
`ensure_ascii=False`, so patterns are matched against the real
characters. `_canon` is also the identity key in
`_splice_missing_protected` / verification, and both sides of that
comparison go through it, so identity stays consistent. Rows without
non-ASCII text serialize exactly as before.
- `tests/test_transforms/test_smart_crusher_audit_safe.py`:
`test_protected_patterns_match_non_ascii_text`, parametrized over
`Kraków`, `错误` and `Zürich`, builds a 600-row array where row 437's
`city` is the pattern. It checks that `_scan_protected_rows` finds that
row and that it survives a real `_smart_crush_content` lossy sample (600
→ ~16 rows).

## Testing

- [x] Unit tests pass (`pytest`): `tests/test_transforms/`,
`tests/test_garbled_compression_fixes.py`,
`tests/test_smart_crusher.py`,
`tests/test_smart_crusher_toin_attachment.py`, not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New test on `main` (fix reverted, test kept) fails at the scan:

```text
$ python -m pytest tests/test_transforms/test_smart_crusher_audit_safe.py -q -k non_ascii
E   AssertionError: assert [] == [{'id': 437, ...', 'ms': 185}]
E     Right contains one more item: {'id': 437, 'city': 'Kraków', 'ms': 185}
```

With the scan assertion removed, so the crush outcome on `main` is
visible, all three rows are sampled out:

```text
E   AssertionError: assert {'id': 437, 'city': 'Kraków', 'ms': 185} in [{'id': 0, 'city': 'Berlin', 'ms': 0}, {'id': 1, ...}, ...]
E   AssertionError: assert {'id': 437, 'city': '错误', 'ms': 185} in [...]
E   AssertionError: assert {'id': 437, 'city': 'Zürich', 'ms': 185} in [...]
======================= 3 failed, 11 deselected in 9.52s =======================
```

With the fix:

```text
$ python -m pytest tests/test_transforms/test_smart_crusher_audit_safe.py -q
============================= 14 passed in 28.95s ==============================

$ python -m pytest tests/test_transforms tests/test_garbled_compression_fixes.py tests/test_smart_crusher.py tests/test_smart_crusher_toin_attachment.py -q
================== 519 passed, 78 skipped in 67.42s (0:01:07) ==================

$ ruff check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_audit_safe.py
All checks passed!
$ ruff format --check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_audit_safe.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel),
`HEADROOM_OFFLINE=1`
- Exact command / steps: ran the full `SmartCrusher.apply(messages,
tokenizer)` transform (the path
`test_apply_preserves_protected_rows_in_tool_message` uses for the
proxy) with `SmartCrusherConfig(audit_safe=True,
protected_patterns=["Kraków"], min_tokens_to_crush=10)` on an OpenAI
tool message holding a 600-row JSON array (UTF-8, `ensure_ascii=False`,
as tool output arrives) in which one row's `city` is `Kraków`. I parsed
the compressed tool content and checked for that row, on `main` and on
this branch.
- Observed result: on `main` the protected row is dropped: 16 of 600
rows kept, `Kraków` row kept: False. On this branch it is kept: 17 of
600 rows, the 16 sampled plus the spliced protected row.

  ```text
  == main
  rows kept: 16 of 600 | Kraków row kept: False
  == this branch
  rows kept: 17 of 600 | Kraków row kept: True
  ```
- Not tested: the proxy end to end with an audit-safe config; patterns
that deliberately match the escaped form (a regex like `Krak\\u00f3w`
with an escaped backslash). I don't know of anyone relying on that, and
such a pattern stops matching after this change. A regex escape like
`Krak\u00f3w` means the character `ó` to `re`, so it never matched on
`main` and now does.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for audit-safe users whose
`protected_patterns` or protected rows contain non-ASCII text: those
rows are now protected as documented. ASCII-only rows serialize
identically, so existing audit-safe behaviour is unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Docs unchecked: the audit-safe docs don't mention ASCII. This makes
non-ASCII patterns behave the way the docs already imply.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Rudy Mizrahi Celekli
2026-10-24 08:37:46 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent ed3195beef
commit e11afdbdf8
2 changed files with 29 additions and 2 deletions
+3 -2
View File
@@ -600,9 +600,10 @@ class SmartCrusher(Transform):
Used both for protected-pattern matching and for identity
comparison across the crush boundary — kept rows are
re-serialized by Rust, so rows are matched by content, not by
Python object identity.
Python object identity. ``ensure_ascii=False`` so a pattern such
as ``Kraków`` sees the characters, not ``\\u00f3`` escapes.
"""
return json.dumps(item, sort_keys=True, default=str)
return json.dumps(item, sort_keys=True, default=str, ensure_ascii=False)
def _row_matches_protected(self, item: Any) -> bool:
text = self._canon(item)
@@ -399,3 +399,29 @@ def test_content_protection_no_op_when_pattern_count_preserved() -> None:
assert out_text == crushed_with_marker
assert was_modified is True
assert info == "lossless:table"
@pytest.mark.parametrize("pattern", ["Kraków", "错误", "Zürich"])
def test_protected_patterns_match_non_ascii_text(pattern: str) -> None:
"""Patterns are matched against the characters, not `\\uXXXX` escapes.
Tool output keeps UTF-8, so a non-ASCII pattern must find its row, and
the row must survive a real lossy crush that samples it out otherwise."""
from headroom.transforms.smart_crusher import SmartCrusher, SmartCrusherConfig
cities = ["Berlin", "Paris", "Madrid", "Rome", "Vienna", "Oslo", "Lisbon", "Prague"]
items = [{"id": i, "city": cities[i % 8], "ms": (i * 37) % 999} for i in range(600)]
items[437]["city"] = pattern
content = json.dumps(items, ensure_ascii=False)
crusher = SmartCrusher(
SmartCrusherConfig(audit_safe=True, protected_patterns=[pattern]),
with_compaction=False,
)
assert crusher._scan_protected_rows(content) == [items[437]]
crushed, was_modified, _info = crusher._smart_crush_content(content)
assert was_modified
kept = json.loads(crushed)
assert items[437] in kept
assert len(kept) < len(items)