fix(dedup): bound line-number digit run and never fold into a longer pointer (#4488)

## Description

Two small defects in `headroom/transforms/cross_turn_dedup.py`, both
found while reviewing cross-turn dedup:

1. `_num_and_key` calls `int()` on a leading digit run. At 4,301+ digits
Python raises `ValueError`; the blanket `except` in `dedup_blocks` then
logs a warning and returns every block unchanged, so one odd line (a
minified or hex-dump line starting with digits and `:`) disables dedup
for the whole request. The line-number regex now allows at most 18
digits; a longer run is not a real line number, so the line is treated
as non-numbered (exact-match fold only).
2. A fold could be longer than the span it replaces: a 40-character
3-line span folded into a 43-character pointer. The code now builds the
pointer first and folds only when the pointer is shorter than the span
in UTF-8 bytes (the pointer's `↑` is 3 bytes, so character counts are
not enough).

Refs #4478

This PR covers items 1 and 2 of #4478 only. Items 3 (`sed -n` spellings
not recognised as reads) and 4 (`#` CCR marker for non-`#` languages) of
that issue stay open, which is why this says `Refs` and not `Fixes`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_LINENO_RE`: `[1-9]\d*` -> `[1-9]\d{0,17}`, with a comment saying
why.
- `dedup_blocks`: compute the pointer before the fold decision and
require `len(ptr.encode("utf-8")) < len(span_text.encode("utf-8"))` in
addition to the existing `min_chars` floor.
- Two tests in `tests/test_cross_turn_dedup.py`: a 10 / 4,300 /
4,301-digit run all fold; a 40-char and a 44-char span are left alone
(the 44-char one is the UTF-8 byte boundary), a 46-char span and a long
span still fold.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Fail-before (fix source reverted, new tests kept):
FAILED tests/test_cross_turn_dedup.py::test_long_digit_run_does_not_disable_dedup
FAILED tests/test_cross_turn_dedup.py::test_fold_never_longer_than_span - assert 1 == 0
2 failed, 20 deselected

# Pass-after (fix restored):
tests/test_cross_turn_dedup.py ..   2 passed, 20 deselected

# Neighbours: test_cross_turn_dedup.py, test_responses_cross_turn_dedup.py, test_openai_chat_dedup_recoverability.py
30 passed

# ruff check + ruff format --check on the two changed files
All checks passed!
2 files already formatted
```

`mypy` was not run.

Review round 1 (Devin): (1) the length check compared characters, but
the pointer's `↑` is 3 bytes, so a 44-char span still grew to a 45-byte
pointer; verified, fixed in 6bfa823f by comparing UTF-8 byte lengths,
boundary test added (fails with the byte fix reverted, passes with it).
(2) runtime proof added below: the real proxy against a local stub
upstream.

```text
# Round 1, byte fix reverted -> test_fold_never_longer_than_span fails at the 44-char assert (1 failed, 1 passed)
# Round 1, byte fix applied  -> 2 passed
# Neighbours: test_cross_turn_dedup.py, test_responses_cross_turn_dedup.py, test_openai_chat_dedup_recoverability.py -> 30 passed
# ruff check + ruff format --check on the two changed files -> All checks passed! / 2 files already formatted
```

## Real Behavior Proof

- Environment: WSL2 Ubuntu on Windows 11, Python 3.12, real `python -m
headroom.proxy.server --lossless` with `HEADROOM_DEDUPE=1` on
127.0.0.1:18965, `ANTHROPIC_TARGET_API_URL` pointing at a local stub
upstream on 127.0.0.1:18966 (fake key, no network), source at upstream
main 3fef15d3 plus this PR.
- Exact command / steps: for each source state, start the stub and the
proxy, then POST four `/v1/messages` requests; each has one assistant
turn with two `Bash` tool calls and a user turn whose two `tool_result`
blocks carry the same text: (A) 5 lines starting with a 4,301-digit run,
(B) a 40-char 3-line span, (C) a 44-char span, (D) a 46-char span. The
stub logs the text it receives. Scripts:
`/var/tmp/cb-pub/proof/fix-dedup-digit-run-and-fold-size/run_all.sh`.
- Observed result: the second block reaches the stub as follows. A
(4,301 digits): up/main unchanged, 21,679 chars, proxy logs "Cross-turn
dedup failed (ValueError)"; 7dc7cfc0 and 6bfa823f folded to 43 chars. B
(40 chars): up/main folded into a 45-byte pointer (grows); 7dc7cfc0 and
6bfa823f left unchanged. C (44 chars): up/main and 7dc7cfc0 folded into
a 45-byte pointer (grows by 1 byte); 6bfa823f left unchanged. D (46
chars): folded in all three (45 < 46 bytes).
- Not tested: real provider, streaming, OpenAI-format routes, mypy; the
proof uses the Anthropic route only and `--lossless` so the text
compressor does not rewrite the blocks before dedup.

Raw stub output (second block, `folded` = text starts a `[↑` pointer):

```text
A  up/main : t2 21679 chars / 21679 bytes, folded=False   (+ WARNING Cross-turn dedup failed (<ValueError>))
A  7dc7cfc0: t2 43 chars / 45 bytes, folded=True  "[↑5L same as msg 2: '92378250366766294...']"
A  now     : t2 43 chars / 45 bytes, folded=True
B  up/main : t2 43 chars / 45 bytes, folded=True   (span was 40 bytes)
B  7dc7cfc0: t2 40 chars / 40 bytes, folded=False
B  now     : t2 40 chars / 40 bytes, folded=False
C  up/main : t2 43 chars / 45 bytes, folded=True   (span was 44 bytes)
C  7dc7cfc0: t2 43 chars / 45 bytes, folded=True   (span was 44 bytes: grew)
C  now     : t2 44 chars / 44 bytes, folded=False
D  all three: t2 43 chars / 45 bytes, folded=True  (span 46 bytes)
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none. Cross-turn dedup is opt-in
(`enable_cross_turn_dedup` / `HEADROOM_DEDUPE`).
- Minimum rollout channel: n/a, no rollout-managed feature touched.
- Stable/default behavior changed: no. Dedup is off by default; when on,
only the two edge cases above change (a fold that would have grown the
text is skipped; a 19+ digit leading run is no longer treated as a line
number).
- Kill switch / disable path: leave `enable_cross_turn_dedup` /
`HEADROOM_DEDUPE` unset.
- Unsafe override required: no.
- Qualification impact: none expected; the change only makes dedup skip
a fold, or fold a block it previously skipped, with the same lossless
pointer format.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A, no
user-facing docs affected)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- The 18-digit bound is a judgment call: anything longer than a sane
line-number width is treated as ordinary text. Such lines still fold on
an exact match.
- Alternative for item 2 from the issue (raising `DEFAULT_MIN_CHARS`)
was not taken; the length check at the fold site is exact and also
covers a caller-supplied `min_chars`.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests above were run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
HetCreep
2026-10-28 23:37:28 -05:00
committed by GitHub
co-authored by Claude Sonnet 5.5 JD Davis
parent 9f068a6c13
commit 21e8867504
2 changed files with 54 additions and 8 deletions
+15 -8
View File
@@ -39,9 +39,9 @@ from ..log_safety import describe_exception, safe_id
logger = logging.getLogger(__name__)
# A run must be at least this many lines AND this many chars to be worth a
# pointer. Small dups are left alone (fragmenting context is not worth it) —
# and a larger floor keeps the pointer comfortably shorter than the span it
# replaces, so a fold is always a net byte win.
# pointer. Small dups are left alone (fragmenting context is not worth it).
# The floor alone does not make a fold a net byte win (the pointer can still be
# as long as a short span); dedup_blocks checks the UTF-8 sizes before folding.
DEFAULT_MIN_LINES = 3
DEFAULT_MIN_CHARS = 40
# Cap anchor candidates examined per line so a hot line (e.g. `` return``)
@@ -54,7 +54,7 @@ MAX_ANCHOR_CANDIDATES = 16
# line must start with the digit, so ``number + delta`` re-numbers byte-exactly
# without touching alignment padding, keeping renumbered folds strictly lossless.
#
# ``[1-9]\d*`` (not ``\d+``) is load-bearing: a LEADING-ZERO run (``08:00:01`` in
# The leading ``[1-9]`` (not ``\d``) is load-bearing: a LEADING-ZERO run (``08:00:01`` in
# a timestamped log, ``007:...``) is not an unpadded grep/sed line number, and
# recovery is ``str(int(number) + delta)``, which drops the zero pad — ``int("08")
# + 1`` renders ``"9"``, not ``"09"``. Matching those would fold them under a
@@ -63,7 +63,11 @@ MAX_ANCHOR_CANDIDATES = 16
# fold only on an EXACT match (delta 0) — the "prefer false negatives" posture.
# Real grep -n / sed -n / rg -n numbers never carry a leading zero, so the
# intended renumber-fold feature is unaffected.
_LINENO_RE = re.compile(r"^([1-9]\d*)(:|\t)(.*)$", re.DOTALL)
#
# ``\d{0,17}`` bounds the run at 18 digits: Python refuses ``int()`` on a run of
# 4,301+ digits (ValueError), which would abort dedup for the whole request. A
# longer run is not a real line number, so the line stays non-numbered.
_LINENO_RE = re.compile(r"^([1-9]\d{0,17})(:|\t)(.*)$", re.DOTALL)
def _num_and_key(line: str) -> tuple[int | None, str, str]:
@@ -264,9 +268,12 @@ def dedup_blocks(
if m is not None and m[0] >= min_lines:
span = lines[i : i + m[0]]
span_text = "\n".join(span)
if len(span_text) >= min_chars:
ref_turn = blocks[m[1]].turn
ptr = _pointer(span, ref_turn, m[3])
ptr = _pointer(span, blocks[m[1]].turn, m[3])
# Never fold when the pointer is not shorter than the span, in
# UTF-8 bytes ("↑" is 3 bytes, so character counts mislead).
if len(span_text) >= min_chars and len(ptr.encode("utf-8")) < len(
span_text.encode("utf-8")
):
out.append(ptr)
# Folded span is NOT verbatim in this block's output:
# mark None so it can't seed a later contiguous match,
+39
View File
@@ -472,3 +472,42 @@ def test_apply_recoverable_default_and_true_still_fold():
true_out = _apply_with_recoverable(msgs, recoverable=True)
assert "[↑" in true_out[2]["content"] # explicit recoverable -> folds
def _five_numbered(digits):
return "\n".join("1" * digits + f":alpha line {i} of the file contents" for i in range(5))
def test_long_digit_run_does_not_disable_dedup():
# int() raises ValueError on 4,301+ digits; that must not skip dedup.
for digits in (10, 4300, 4301):
t = _five_numbered(digits)
out, stats = dedup_blocks([_blk(t, 1), _blk(t, 2)])
assert stats["spans_folded"] == 1, digits
assert out[1].text != t
def test_fold_never_longer_than_span():
# A 40-char 3-line span would fold into a 43-char pointer: leave it alone.
t = "a" * 21 + "\n" + "b" * 9 + "\n" + "c" * 8
assert len(t) == 40
out, stats = dedup_blocks([_blk(t, 1), _blk(t, 2)])
assert stats["spans_folded"] == 0
assert out[1].text == t
# Byte boundary: the pointer's "↑" is 3 UTF-8 bytes. A 44-char span has a
# 43-char pointer (shorter in characters) that is 45 bytes, so folding
# would grow the output; it must stay. A 46-byte span folds.
t44 = "a" * 25 + "\n" + "b" * 9 + "\n" + "c" * 8
assert len(t44) == 44
out, stats = dedup_blocks([_blk(t44, 1), _blk(t44, 2)])
assert stats["spans_folded"] == 0
assert out[1].text == t44
t46 = "a" * 27 + "\n" + "b" * 9 + "\n" + "c" * 8
out, stats = dedup_blocks([_blk(t46, 1), _blk(t46, 2)])
assert stats["spans_folded"] == 1
assert len(out[1].text.encode("utf-8")) < len(t46.encode("utf-8"))
# A long span still folds.
long_t = _code("", 5)
out, stats = dedup_blocks([_blk(long_t, 1), _blk(long_t, 2)])
assert stats["spans_folded"] == 1
assert len(out[1].text) < len(long_t)