mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(smart-crusher): match audit-safe protected_patterns against unescaped text (#4294)
## Description Audit-safe mode (`SmartCrusherConfig(audit_safe=True, protected_patterns=[...])`, #1705) guarantees that rows matching a protected regex survive compression. The regexes run against `SmartCrusher._canon(row)`, which is `json.dumps(row, sort_keys=True, default=str)`. `json.dumps` defaults to `ensure_ascii=True`, so every non-ASCII character is escaped before the pattern sees it: ```python _canon({"city": "Kraków"}) # '{"city": "Krak\\u00f3w"}' re.search("Kraków", ...) # None ``` A pattern that contains any non-ASCII character can never match: accented names, CJK error text, currency symbols such as `€`. `_scan_protected_rows` returns `[]`, nothing is spliced back or verified, and the row is sampled out like any other. Nothing warns, because from the crusher's point of view there was nothing to protect. That's the exact failure audit-safe mode exists to prevent. Closes: N/A (no matching issue found) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/smart_crusher.py`: `_canon()` serializes with `ensure_ascii=False`, so patterns are matched against the real characters. `_canon` is also the identity key in `_splice_missing_protected` / verification, and both sides of that comparison go through it, so identity stays consistent. Rows without non-ASCII text serialize exactly as before. - `tests/test_transforms/test_smart_crusher_audit_safe.py`: `test_protected_patterns_match_non_ascii_text`, parametrized over `Kraków`, `错误` and `Zürich`, builds a 600-row array where row 437's `city` is the pattern. It checks that `_scan_protected_rows` finds that row and that it survives a real `_smart_crush_content` lossy sample (600 → ~16 rows). ## Testing - [x] Unit tests pass (`pytest`): `tests/test_transforms/`, `tests/test_garbled_compression_fixes.py`, `tests/test_smart_crusher.py`, `tests/test_smart_crusher_toin_attachment.py`, not the full suite - [x] Linting passes (`ruff check .`): on the changed files - [ ] Type checking passes (`mypy headroom`): not run - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New test on `main` (fix reverted, test kept) fails at the scan: ```text $ python -m pytest tests/test_transforms/test_smart_crusher_audit_safe.py -q -k non_ascii E AssertionError: assert [] == [{'id': 437, ...', 'ms': 185}] E Right contains one more item: {'id': 437, 'city': 'Kraków', 'ms': 185} ``` With the scan assertion removed, so the crush outcome on `main` is visible, all three rows are sampled out: ```text E AssertionError: assert {'id': 437, 'city': 'Kraków', 'ms': 185} in [{'id': 0, 'city': 'Berlin', 'ms': 0}, {'id': 1, ...}, ...] E AssertionError: assert {'id': 437, 'city': '错误', 'ms': 185} in [...] E AssertionError: assert {'id': 437, 'city': 'Zürich', 'ms': 185} in [...] ======================= 3 failed, 11 deselected in 9.52s ======================= ``` With the fix: ```text $ python -m pytest tests/test_transforms/test_smart_crusher_audit_safe.py -q ============================= 14 passed in 28.95s ============================== $ python -m pytest tests/test_transforms tests/test_garbled_compression_fixes.py tests/test_smart_crusher.py tests/test_smart_crusher_toin_attachment.py -q ================== 519 passed, 78 skipped in 67.42s (0:01:07) ================== $ ruff check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_audit_safe.py All checks passed! $ ruff format --check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_audit_safe.py 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this branch (native `headroom._core` from the 0.41.0 wheel), `HEADROOM_OFFLINE=1` - Exact command / steps: ran the full `SmartCrusher.apply(messages, tokenizer)` transform (the path `test_apply_preserves_protected_rows_in_tool_message` uses for the proxy) with `SmartCrusherConfig(audit_safe=True, protected_patterns=["Kraków"], min_tokens_to_crush=10)` on an OpenAI tool message holding a 600-row JSON array (UTF-8, `ensure_ascii=False`, as tool output arrives) in which one row's `city` is `Kraków`. I parsed the compressed tool content and checked for that row, on `main` and on this branch. - Observed result: on `main` the protected row is dropped: 16 of 600 rows kept, `Kraków` row kept: False. On this branch it is kept: 17 of 600 rows, the 16 sampled plus the spliced protected row. ```text == main rows kept: 16 of 600 | Kraków row kept: False == this branch rows kept: 17 of 600 | Kraków row kept: True ``` - Not tested: the proxy end to end with an audit-safe config; patterns that deliberately match the escaped form (a regex like `Krak\\u00f3w` with an escaped backslash). I don't know of anyone relying on that, and such a pattern stops matching after this change. A regex escape like `Krak\u00f3w` means the character `ó` to `re`, so it never matched on `main` and now does. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: only for audit-safe users whose `protected_patterns` or protected rows contain non-ASCII text: those rows are now protected as documented. ASCII-only rows serialize identically, so existing audit-safe behaviour is unchanged. - Kill switch / disable path: N/A - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Docs unchecked: the audit-safe docs don't mention ASCII. This makes non-ASCII patterns behave the way the docs already imply. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
ed3195beef
commit
e11afdbdf8
@@ -600,9 +600,10 @@ class SmartCrusher(Transform):
|
||||
Used both for protected-pattern matching and for identity
|
||||
comparison across the crush boundary — kept rows are
|
||||
re-serialized by Rust, so rows are matched by content, not by
|
||||
Python object identity.
|
||||
Python object identity. ``ensure_ascii=False`` so a pattern such
|
||||
as ``Kraków`` sees the characters, not ``\\u00f3`` escapes.
|
||||
"""
|
||||
return json.dumps(item, sort_keys=True, default=str)
|
||||
return json.dumps(item, sort_keys=True, default=str, ensure_ascii=False)
|
||||
|
||||
def _row_matches_protected(self, item: Any) -> bool:
|
||||
text = self._canon(item)
|
||||
|
||||
@@ -399,3 +399,29 @@ def test_content_protection_no_op_when_pattern_count_preserved() -> None:
|
||||
assert out_text == crushed_with_marker
|
||||
assert was_modified is True
|
||||
assert info == "lossless:table"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pattern", ["Kraków", "错误", "Zürich"])
|
||||
def test_protected_patterns_match_non_ascii_text(pattern: str) -> None:
|
||||
"""Patterns are matched against the characters, not `\\uXXXX` escapes.
|
||||
Tool output keeps UTF-8, so a non-ASCII pattern must find its row, and
|
||||
the row must survive a real lossy crush that samples it out otherwise."""
|
||||
from headroom.transforms.smart_crusher import SmartCrusher, SmartCrusherConfig
|
||||
|
||||
cities = ["Berlin", "Paris", "Madrid", "Rome", "Vienna", "Oslo", "Lisbon", "Prague"]
|
||||
items = [{"id": i, "city": cities[i % 8], "ms": (i * 37) % 999} for i in range(600)]
|
||||
items[437]["city"] = pattern
|
||||
content = json.dumps(items, ensure_ascii=False)
|
||||
|
||||
crusher = SmartCrusher(
|
||||
SmartCrusherConfig(audit_safe=True, protected_patterns=[pattern]),
|
||||
with_compaction=False,
|
||||
)
|
||||
assert crusher._scan_protected_rows(content) == [items[437]]
|
||||
|
||||
crushed, was_modified, _info = crusher._smart_crush_content(content)
|
||||
|
||||
assert was_modified
|
||||
kept = json.loads(crushed)
|
||||
assert items[437] in kept
|
||||
assert len(kept) < len(items)
|
||||
|
||||
Reference in New Issue
Block a user