fix(kompress): keep words past the tokenizer's truncation point (#4328)

## Description

Kompress chunks input by **words** (`chunk_words=350`) but tokenizes
each chunk with `truncation=True, max_length=512`. When a chunk exceeds
512 tokens, the words past the cut-off never get a `word_id`, so they
are never scored and never reach `kept_ids` — they are **silently
dropped**, regardless of importance. The loss is positional, so whatever
sits at the end of the chunk disappears from the compressed output.

This happens on the default proxy path (`target_ratio=None`, no forced
ratio).

350 words fit in 512 tokens only for prose-like English (~1.2
tokens/word). Logs, IDs, hashes, paths and numbers run at 2.5–3+
tokens/word, and non-Latin scripts at 3–4, so a token-heavy chunk hits
the token limit well before its word budget.

Closes #4309

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New helper `_add_truncated_words(kept_ids, word_ids, chunk_words,
chunk_start, token_count)`: the covered word count is the highest
`word_id` the tokenizer reported, and every word of the chunk past it is
added to `kept_ids` verbatim.
- The cut can also land **inside** a word. Its visible prefix counts as
covered, so the model scores part of a word and may drop the whole of
it, taking the unseen suffix with it. The last covered word is therefore
kept as well whenever the chunk reached the token limit.
- Applied in the single path and the batch path, which share the same
shape. A chunk that is not truncated is unaffected, so the model's own
keep/drop decision still stands.
- The token limit moved into `_KOMPRESS_MAX_TOKENS` so the helper and
the tokenizer call cannot drift apart.
- **A forced ratio is a hard budget the caller asked for**, so the
unscored tail is only taken back on the model-decides path
(`target_ratio=None`); the ratio path keeps the budget it was given.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_kompress_must_keep.py -q
collected 23 items

tests\test_kompress_must_keep.py .......................                 [100%]

======================== 23 passed, 1 warning in 1.81s ========================
```

The wider kompress suites (`test_kompress_compressor.py`,
`test_kompress_deadline.py`, `test_kompress_size_gate.py`,
`test_kompress_line_integrity.py`, `test_kompress_failsafe.py`,
`test_kompress_marker_gate.py`) report `10 failed, 91 passed, 15
skipped`; the same 10 fail on the unmodified tree (`10 failed, 89
passed, 15 skipped`) — they need `torch`/`onnxruntime`/`httpx`, which
are not installed here.

## Real Behavior Proof

- Environment: headroom source checkout on Windows, Python 3.12; no
third-party deps installed
- Exact command / steps: the new tests install a tokenizer that models
the real `truncation=True, max_length=512`: every word costs a number of
tokens, so a 350-word chunk of token-heavy words runs past the limit,
and the cut can fall inside a word
- Observed result: before the fix the tail of the chunk vanished from
the output entirely; after the fix the tail is kept verbatim, the word
the cut landed in is kept too, and a chunk that fits is still left to
the model
- Not tested: the real ONNX model and a real tokenizer (no
`onnxruntime`/`torch`/`transformers` here), and the end-to-end proxy
path

The four new tests fail on the unmodified tree and pass with the fix:

```text
$ pytest tests/test_kompress_must_keep.py -q -k Truncated   # fix reverted
FAILED ...::test_words_past_the_truncation_point_survive
FAILED ...::test_the_word_the_cut_landed_in_is_kept
FAILED ...::test_batch_path_keeps_truncated_words_too
FAILED ...::test_a_forced_ratio_still_gets_its_budget
4 failed, 1 passed, 18 deselected
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for chunks whose tokenized
length exceeds 512 — the truncated tail is now kept instead of dropped,
so the compressed output is longer and no longer silently lossy
- Kill switch / disable path: n/a (the change only adds words that were
previously discarded)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- The new tests live in `tests/test_kompress_must_keep.py` next to the
existing must-keep override tests. They model the real 512-token limit
(a word costs several tokens, and the cut can fall inside a word) rather
than pretending the chunk is cut at a word boundary.
- On the live-tokenizer point: this machine has no `transformers` and no
model weights, so a live tokenizer run is not possible here, and the
repository's own kompress tests inject a fake model and tokenizer for
the same reason (`tests/test_kompress_failsafe.py`). The fake above
follows the real tokenizer's contract — `is_split_into_words` word ids,
a cut at `max_length`, and a cut that can land inside a word.
- `opentelemetry` is not installed on this machine, so `import headroom`
fails and the suite cannot be collected as-is. The test run above used a
minimal local stub for the `opentelemetry` API surface that
`headroom/observability` imports; no product code was changed to
accommodate it, and the stub is not part of this PR.
- `ruff` and `mypy` were not run: neither is installed here.

## Model Used

MiniMax M3.1 Flash Free via the opencode agent harness
(`minimax-m3.1-flash-free`)

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Vaibhav Hajare
2026-11-04 22:52:01 -06:00
committed by GitHub
co-authored by JerrettDavis
parent 9355ec7019
commit bb13384829
3 changed files with 176 additions and 0 deletions
@@ -87,6 +87,8 @@ _KOMPRESS_MUST_KEEP_RE = re.compile(
r"|(?i:\b(?:and|or|nor|xor)\b)"
)
_KOMPRESS_MUST_KEEP_ENV = "HEADROOM_KOMPRESS_MUST_KEEP"
#: Tokenizer budget per chunk. Words past it are never scored (#4309).
_KOMPRESS_MAX_TOKENS = 512
KOMPRESS_BACKEND_ENV = "HEADROOM_KOMPRESS_BACKEND"
KOMPRESS_ONNX_FILENAME_ENV = "HEADROOM_KOMPRESS_ONNX_FILENAME"
@@ -142,6 +144,39 @@ def _add_kompress_must_keep_words(
kept_ids.add(word_idx + chunk_start)
def _add_truncated_words(
kept_ids: set[int],
word_ids: list[int | None],
chunk_words: list[str],
chunk_start: int,
token_count: int,
) -> None:
"""Keep the words the tokenizer's token cut took away.
Chunks are cut by words (``chunk_words``) but tokenized with
``truncation=True, max_length=512``. Words past token 512 get no
``word_id``, so they are never scored and never make ``kept_ids`` -- they
are dropped silently, regardless of importance, and the loss is
positional: whatever sits at the end of the chunk disappears (#4309).
The cut can also land inside a word. Its visible prefix counts as
covered, so the model scores part of a word and may drop the whole of
it, taking the unseen suffix with it. The last covered word is kept as
well whenever the chunk reached the token limit.
"""
covered = max((wid for wid in word_ids if wid is not None), default=-1) + 1
first_dropped = covered
if token_count >= _KOMPRESS_MAX_TOKENS and covered > 0:
first_dropped = covered - 1
for local_idx in range(first_dropped, len(chunk_words)):
kept_ids.add(local_idx + chunk_start)
def _real_token_count(attention_row: Any) -> int:
"""Number of real (non-padding) tokens in an attention-mask row."""
return sum(1 for mask in attention_row if mask)
_TABULAR_ROW_SPLIT_RE = re.compile(r"\s{2,}")
@@ -2059,6 +2094,7 @@ class KompressCompressor(Transform):
input_ids = encoding["input_ids"]
attention_mask = encoding["attention_mask"]
word_ids = encoding.word_ids(batch_index=0)
token_count = _real_token_count(attention_mask[0])
# One row, so no padding: every token in it is real.
scored = _words_seen_whole(word_ids, len(word_ids), len(chunk_words))
if scored == 0:
@@ -2157,6 +2193,11 @@ class KompressCompressor(Transform):
continue
if bool(mask_list[idx]):
kept_ids.add(wid + chunk_start)
# The words past the tokenizer's token cut were never
# scored, so the model cannot have kept them. A forced
# ratio is a hard budget the caller asked for, so only the
# model-decides path takes them back (#4309).
_add_truncated_words(kept_ids, word_ids, chunk_words, chunk_start, token_count)
# Hard override: always keep must-keep tokens regardless of model score.
# Numbers, error names, paths, and flags carry meaning agents cannot
@@ -2591,6 +2632,17 @@ class KompressCompressor(Transform):
for wid, score in word_scores.items():
if score > self.config.score_threshold:
kept_ids_per_text[text_idx].add(wid + chunk_start)
# The words past the tokenizer's token cut were never
# scored, so the model cannot have kept them. A forced
# ratio is a hard budget the caller asked for, so only
# the model-decides path takes them back (#4309).
_add_truncated_words(
kept_ids_per_text[text_idx],
word_ids,
chunk_words,
chunk_start,
_real_token_count(attention_mask[batch_idx]),
)
_add_kompress_must_keep_words(
kept_ids_per_text[text_idx], chunk_words, chunk_start
+4
View File
@@ -51,6 +51,10 @@ def test_single_and_batch_device_copies_hold_execution_slot(monkeypatch):
def __init__(self, row_lengths):
self.row_lengths = row_lengths
def __getitem__(self, index):
# Real attention masks expose each row for padding-aware counts.
return [1] * self.row_lengths[index]
def to(self, _device):
acquired = semaphore.acquire(blocking=False)
if acquired:
+120
View File
@@ -192,3 +192,123 @@ class TestMustKeepCompression:
)
assert result.compressed.split() == ["alpha", "0x7fff2038"]
class _TruncatingTok:
"""Tokenizer that models ``truncation=True, max_length=512``.
Every word costs ``tokens_per_word`` tokens, which is how a chunk of
token-heavy words (logs, IDs, paths, numbers) runs past the token limit
before its word budget is used up. Words past the limit get no
``word_id`` at all, exactly as the real tokenizer reports them. The cut
can also land inside a word: that word keeps the tokens that fit and
the rest of it never reaches the tokenizer.
"""
def __init__(self, tokens_per_word: int = 1, max_length: int = 512):
self.tokens_per_word = tokens_per_word
self.max_length = max_length
def __call__(self, chunk_words, **kw):
batch_words = (
chunk_words if (chunk_words and isinstance(chunk_words[0], list)) else [chunk_words]
)
word_ids = []
for words in batch_words:
ids: list[int] = []
used = 0
for idx in range(len(words)):
room = self.max_length - used
if room <= 0:
break
take = min(self.tokens_per_word, room)
ids.extend([idx] * take)
used += take
word_ids.append(ids)
return _Enc(
input_ids=[[0] * len(ids) for ids in word_ids],
attention_mask=[[1] * len(ids) for ids in word_ids],
_word_ids=word_ids,
)
class TestTruncatedWordsCompression:
"""Words past the tokenizer's 512-token cut are kept, not dropped (#4309).
Chunks are cut by words (``chunk_words``) but tokenized with
``truncation=True, max_length=512``. A chunk of token-heavy words hits
the token limit well before its word budget, so the tail of the chunk had
no score and was dropped silently from the default proxy path
(``target_ratio=None``).
"""
# One chunk is `chunk_words` words long, so 350 words is exactly one.
CHUNK_WORDS = 350
def _words(self, count: int) -> list[str]:
return [f"word{i}" for i in range(count)]
def _install(self, monkeypatch, tokens_per_word):
monkeypatch.setattr(
kc,
"_load_kompress",
lambda *a, **k: (_Model(), _TruncatingTok(tokens_per_word), "onnx"),
)
monkeypatch.setattr(kc, "_model_device_type", lambda *a, **k: "cpu")
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
def _compress(self, monkeypatch, tokens_per_word):
self._install(monkeypatch, tokens_per_word)
words = self._words(self.CHUNK_WORDS)
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
kept = compressor.compress(" ".join(words)).compressed.split()
return kept, words
def test_words_past_the_truncation_point_survive(self, monkeypatch):
# Two tokens per word: the 350-word chunk needs 700 tokens, so the
# tokenizer cuts it at 512. The 256 words it covered were scored, the
# other 94 were never seen and used to vanish from the output.
kept, words = self._compress(monkeypatch, tokens_per_word=2)
# 256 words were covered; the cut landed at a word boundary here, but
# the last covered word is kept too, because the cut may have fallen
# inside it.
assert kept == [words[0]] + words[255:]
def test_the_word_the_cut_landed_in_is_kept(self, monkeypatch):
# Three tokens per word: the cut at 512 falls inside word 170, whose
# last token never reached the tokenizer. The model only saw part of
# that word, so dropping it would take its unseen suffix with it.
kept, words = self._compress(monkeypatch, tokens_per_word=3)
assert kept == [words[0]] + words[170:]
def test_fully_covered_chunk_is_unchanged(self, monkeypatch):
# One token per word: the chunk fits in 512 tokens, so nothing was
# truncated and the model's own choice stands. It kept the first word
# and dropped the rest.
kept, words = self._compress(monkeypatch, tokens_per_word=1)
assert kept == [words[0]]
def test_batch_path_keeps_truncated_words_too(self, monkeypatch):
self._install(monkeypatch, 2)
words = self._words(self.CHUNK_WORDS)
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
monkeypatch.setattr(compressor, "_should_use_sequential_fallback", lambda: False)
[result] = compressor.compress_batch([" ".join(words)], batch_size=8)
assert result.compressed.split() == [words[0]] + words[255:]
def test_a_forced_ratio_still_gets_its_budget(self, monkeypatch):
# A forced ratio selects scored words within each token window.
self._install(monkeypatch, 2)
words = self._words(self.CHUNK_WORDS)
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
kept = compressor.compress(" ".join(words), target_ratio=0.5).compressed.split()
# Main now scores the formerly unseen tail in another window.
# Windows score 255 and 95 complete words, each applying the
# requested ratio independently with rounding down (127 + 47).
assert kept == words[:127] + words[255:302]