mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(dedup): bound line-number digit run and never fold into a longer pointer (#4488)
## Description Two small defects in `headroom/transforms/cross_turn_dedup.py`, both found while reviewing cross-turn dedup: 1. `_num_and_key` calls `int()` on a leading digit run. At 4,301+ digits Python raises `ValueError`; the blanket `except` in `dedup_blocks` then logs a warning and returns every block unchanged, so one odd line (a minified or hex-dump line starting with digits and `:`) disables dedup for the whole request. The line-number regex now allows at most 18 digits; a longer run is not a real line number, so the line is treated as non-numbered (exact-match fold only). 2. A fold could be longer than the span it replaces: a 40-character 3-line span folded into a 43-character pointer. The code now builds the pointer first and folds only when the pointer is shorter than the span in UTF-8 bytes (the pointer's `↑` is 3 bytes, so character counts are not enough). Refs #4478 This PR covers items 1 and 2 of #4478 only. Items 3 (`sed -n` spellings not recognised as reads) and 4 (`#` CCR marker for non-`#` languages) of that issue stay open, which is why this says `Refs` and not `Fixes`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `_LINENO_RE`: `[1-9]\d*` -> `[1-9]\d{0,17}`, with a comment saying why. - `dedup_blocks`: compute the pointer before the fold decision and require `len(ptr.encode("utf-8")) < len(span_text.encode("utf-8"))` in addition to the existing `min_chars` floor. - Two tests in `tests/test_cross_turn_dedup.py`: a 10 / 4,300 / 4,301-digit run all fold; a 40-char and a 44-char span are left alone (the 44-char one is the UTF-8 byte boundary), a 46-char span and a long span still fold. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Fail-before (fix source reverted, new tests kept): FAILED tests/test_cross_turn_dedup.py::test_long_digit_run_does_not_disable_dedup FAILED tests/test_cross_turn_dedup.py::test_fold_never_longer_than_span - assert 1 == 0 2 failed, 20 deselected # Pass-after (fix restored): tests/test_cross_turn_dedup.py .. 2 passed, 20 deselected # Neighbours: test_cross_turn_dedup.py, test_responses_cross_turn_dedup.py, test_openai_chat_dedup_recoverability.py 30 passed # ruff check + ruff format --check on the two changed files All checks passed! 2 files already formatted ``` `mypy` was not run. Review round 1 (Devin): (1) the length check compared characters, but the pointer's `↑` is 3 bytes, so a 44-char span still grew to a 45-byte pointer; verified, fixed in6bfa823fby comparing UTF-8 byte lengths, boundary test added (fails with the byte fix reverted, passes with it). (2) runtime proof added below: the real proxy against a local stub upstream. ```text # Round 1, byte fix reverted -> test_fold_never_longer_than_span fails at the 44-char assert (1 failed, 1 passed) # Round 1, byte fix applied -> 2 passed # Neighbours: test_cross_turn_dedup.py, test_responses_cross_turn_dedup.py, test_openai_chat_dedup_recoverability.py -> 30 passed # ruff check + ruff format --check on the two changed files -> All checks passed! / 2 files already formatted ``` ## Real Behavior Proof - Environment: WSL2 Ubuntu on Windows 11, Python 3.12, real `python -m headroom.proxy.server --lossless` with `HEADROOM_DEDUPE=1` on 127.0.0.1:18965, `ANTHROPIC_TARGET_API_URL` pointing at a local stub upstream on 127.0.0.1:18966 (fake key, no network), source at upstream main3fef15d3plus this PR. - Exact command / steps: for each source state, start the stub and the proxy, then POST four `/v1/messages` requests; each has one assistant turn with two `Bash` tool calls and a user turn whose two `tool_result` blocks carry the same text: (A) 5 lines starting with a 4,301-digit run, (B) a 40-char 3-line span, (C) a 44-char span, (D) a 46-char span. The stub logs the text it receives. Scripts: `/var/tmp/cb-pub/proof/fix-dedup-digit-run-and-fold-size/run_all.sh`. - Observed result: the second block reaches the stub as follows. A (4,301 digits): up/main unchanged, 21,679 chars, proxy logs "Cross-turn dedup failed (ValueError)";7dc7cfc0and6bfa823ffolded to 43 chars. B (40 chars): up/main folded into a 45-byte pointer (grows);7dc7cfc0and6bfa823fleft unchanged. C (44 chars): up/main and7dc7cfc0folded into a 45-byte pointer (grows by 1 byte);6bfa823fleft unchanged. D (46 chars): folded in all three (45 < 46 bytes). - Not tested: real provider, streaming, OpenAI-format routes, mypy; the proof uses the Anthropic route only and `--lossless` so the text compressor does not rewrite the blocks before dedup. Raw stub output (second block, `folded` = text starts a `[↑` pointer): ```text A up/main : t2 21679 chars / 21679 bytes, folded=False (+ WARNING Cross-turn dedup failed (<ValueError>)) A7dc7cfc0: t2 43 chars / 45 bytes, folded=True "[↑5L same as msg 2: '92378250366766294...']" A now : t2 43 chars / 45 bytes, folded=True B up/main : t2 43 chars / 45 bytes, folded=True (span was 40 bytes) B7dc7cfc0: t2 40 chars / 40 bytes, folded=False B now : t2 40 chars / 40 bytes, folded=False C up/main : t2 43 chars / 45 bytes, folded=True (span was 44 bytes) C7dc7cfc0: t2 43 chars / 45 bytes, folded=True (span was 44 bytes: grew) C now : t2 44 chars / 44 bytes, folded=False D all three: t2 43 chars / 45 bytes, folded=True (span 46 bytes) ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none. Cross-turn dedup is opt-in (`enable_cross_turn_dedup` / `HEADROOM_DEDUPE`). - Minimum rollout channel: n/a, no rollout-managed feature touched. - Stable/default behavior changed: no. Dedup is off by default; when on, only the two edge cases above change (a fold that would have grown the text is skipped; a 19+ digit leading run is no longer treated as a line number). - Kill switch / disable path: leave `enable_cross_turn_dedup` / `HEADROOM_DEDUPE` unset. - Unsafe override required: no. - Qualification impact: none expected; the change only makes dedup skip a fold, or fold a block it previously skipped, with the same lossless pointer format. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A, no user-facing docs affected) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - The 18-digit bound is a judgment call: anything longer than a sane line-number width is treated as ordinary text. Such lines still fold on an exact match. - Alternative for item 2 from the issue (raising `DEFAULT_MIN_CHARS`) was not taken; the length check at the fold site is exact and also covers a caller-supplied `min_chars`. Produced with AI assistance (Claude Code) operated by me (HetCreep); the tests above were run on my machine against a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Sonnet 5.5
JD Davis
parent
9f068a6c13
commit
21e8867504
@@ -39,9 +39,9 @@ from ..log_safety import describe_exception, safe_id
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# A run must be at least this many lines AND this many chars to be worth a
|
||||
# pointer. Small dups are left alone (fragmenting context is not worth it) —
|
||||
# and a larger floor keeps the pointer comfortably shorter than the span it
|
||||
# replaces, so a fold is always a net byte win.
|
||||
# pointer. Small dups are left alone (fragmenting context is not worth it).
|
||||
# The floor alone does not make a fold a net byte win (the pointer can still be
|
||||
# as long as a short span); dedup_blocks checks the UTF-8 sizes before folding.
|
||||
DEFAULT_MIN_LINES = 3
|
||||
DEFAULT_MIN_CHARS = 40
|
||||
# Cap anchor candidates examined per line so a hot line (e.g. `` return``)
|
||||
@@ -54,7 +54,7 @@ MAX_ANCHOR_CANDIDATES = 16
|
||||
# line must start with the digit, so ``number + delta`` re-numbers byte-exactly
|
||||
# without touching alignment padding, keeping renumbered folds strictly lossless.
|
||||
#
|
||||
# ``[1-9]\d*`` (not ``\d+``) is load-bearing: a LEADING-ZERO run (``08:00:01`` in
|
||||
# The leading ``[1-9]`` (not ``\d``) is load-bearing: a LEADING-ZERO run (``08:00:01`` in
|
||||
# a timestamped log, ``007:...``) is not an unpadded grep/sed line number, and
|
||||
# recovery is ``str(int(number) + delta)``, which drops the zero pad — ``int("08")
|
||||
# + 1`` renders ``"9"``, not ``"09"``. Matching those would fold them under a
|
||||
@@ -63,7 +63,11 @@ MAX_ANCHOR_CANDIDATES = 16
|
||||
# fold only on an EXACT match (delta 0) — the "prefer false negatives" posture.
|
||||
# Real grep -n / sed -n / rg -n numbers never carry a leading zero, so the
|
||||
# intended renumber-fold feature is unaffected.
|
||||
_LINENO_RE = re.compile(r"^([1-9]\d*)(:|\t)(.*)$", re.DOTALL)
|
||||
#
|
||||
# ``\d{0,17}`` bounds the run at 18 digits: Python refuses ``int()`` on a run of
|
||||
# 4,301+ digits (ValueError), which would abort dedup for the whole request. A
|
||||
# longer run is not a real line number, so the line stays non-numbered.
|
||||
_LINENO_RE = re.compile(r"^([1-9]\d{0,17})(:|\t)(.*)$", re.DOTALL)
|
||||
|
||||
|
||||
def _num_and_key(line: str) -> tuple[int | None, str, str]:
|
||||
@@ -264,9 +268,12 @@ def dedup_blocks(
|
||||
if m is not None and m[0] >= min_lines:
|
||||
span = lines[i : i + m[0]]
|
||||
span_text = "\n".join(span)
|
||||
if len(span_text) >= min_chars:
|
||||
ref_turn = blocks[m[1]].turn
|
||||
ptr = _pointer(span, ref_turn, m[3])
|
||||
ptr = _pointer(span, blocks[m[1]].turn, m[3])
|
||||
# Never fold when the pointer is not shorter than the span, in
|
||||
# UTF-8 bytes ("↑" is 3 bytes, so character counts mislead).
|
||||
if len(span_text) >= min_chars and len(ptr.encode("utf-8")) < len(
|
||||
span_text.encode("utf-8")
|
||||
):
|
||||
out.append(ptr)
|
||||
# Folded span is NOT verbatim in this block's output:
|
||||
# mark None so it can't seed a later contiguous match,
|
||||
|
||||
@@ -472,3 +472,42 @@ def test_apply_recoverable_default_and_true_still_fold():
|
||||
|
||||
true_out = _apply_with_recoverable(msgs, recoverable=True)
|
||||
assert "[↑" in true_out[2]["content"] # explicit recoverable -> folds
|
||||
|
||||
|
||||
def _five_numbered(digits):
|
||||
return "\n".join("1" * digits + f":alpha line {i} of the file contents" for i in range(5))
|
||||
|
||||
|
||||
def test_long_digit_run_does_not_disable_dedup():
|
||||
# int() raises ValueError on 4,301+ digits; that must not skip dedup.
|
||||
for digits in (10, 4300, 4301):
|
||||
t = _five_numbered(digits)
|
||||
out, stats = dedup_blocks([_blk(t, 1), _blk(t, 2)])
|
||||
assert stats["spans_folded"] == 1, digits
|
||||
assert out[1].text != t
|
||||
|
||||
|
||||
def test_fold_never_longer_than_span():
|
||||
# A 40-char 3-line span would fold into a 43-char pointer: leave it alone.
|
||||
t = "a" * 21 + "\n" + "b" * 9 + "\n" + "c" * 8
|
||||
assert len(t) == 40
|
||||
out, stats = dedup_blocks([_blk(t, 1), _blk(t, 2)])
|
||||
assert stats["spans_folded"] == 0
|
||||
assert out[1].text == t
|
||||
# Byte boundary: the pointer's "↑" is 3 UTF-8 bytes. A 44-char span has a
|
||||
# 43-char pointer (shorter in characters) that is 45 bytes, so folding
|
||||
# would grow the output; it must stay. A 46-byte span folds.
|
||||
t44 = "a" * 25 + "\n" + "b" * 9 + "\n" + "c" * 8
|
||||
assert len(t44) == 44
|
||||
out, stats = dedup_blocks([_blk(t44, 1), _blk(t44, 2)])
|
||||
assert stats["spans_folded"] == 0
|
||||
assert out[1].text == t44
|
||||
t46 = "a" * 27 + "\n" + "b" * 9 + "\n" + "c" * 8
|
||||
out, stats = dedup_blocks([_blk(t46, 1), _blk(t46, 2)])
|
||||
assert stats["spans_folded"] == 1
|
||||
assert len(out[1].text.encode("utf-8")) < len(t46.encode("utf-8"))
|
||||
# A long span still folds.
|
||||
long_t = _code("", 5)
|
||||
out, stats = dedup_blocks([_blk(long_t, 1), _blk(long_t, 2)])
|
||||
assert stats["spans_folded"] == 1
|
||||
assert len(out[1].text) < len(long_t)
|
||||
|
||||
Reference in New Issue
Block a user