mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(kompress): keep words past the tokenizer's truncation point (#4328)
## Description Kompress chunks input by **words** (`chunk_words=350`) but tokenizes each chunk with `truncation=True, max_length=512`. When a chunk exceeds 512 tokens, the words past the cut-off never get a `word_id`, so they are never scored and never reach `kept_ids` — they are **silently dropped**, regardless of importance. The loss is positional, so whatever sits at the end of the chunk disappears from the compressed output. This happens on the default proxy path (`target_ratio=None`, no forced ratio). 350 words fit in 512 tokens only for prose-like English (~1.2 tokens/word). Logs, IDs, hashes, paths and numbers run at 2.5–3+ tokens/word, and non-Latin scripts at 3–4, so a token-heavy chunk hits the token limit well before its word budget. Closes #4309 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - New helper `_add_truncated_words(kept_ids, word_ids, chunk_words, chunk_start, token_count)`: the covered word count is the highest `word_id` the tokenizer reported, and every word of the chunk past it is added to `kept_ids` verbatim. - The cut can also land **inside** a word. Its visible prefix counts as covered, so the model scores part of a word and may drop the whole of it, taking the unseen suffix with it. The last covered word is therefore kept as well whenever the chunk reached the token limit. - Applied in the single path and the batch path, which share the same shape. A chunk that is not truncated is unaffected, so the model's own keep/drop decision still stands. - The token limit moved into `_KOMPRESS_MAX_TOKENS` so the helper and the tokenizer call cannot drift apart. - **A forced ratio is a hard budget the caller asked for**, so the unscored tail is only taken back on the model-decides path (`target_ratio=None`); the ratio path keeps the budget it was given. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_kompress_must_keep.py -q collected 23 items tests\test_kompress_must_keep.py ....................... [100%] ======================== 23 passed, 1 warning in 1.81s ======================== ``` The wider kompress suites (`test_kompress_compressor.py`, `test_kompress_deadline.py`, `test_kompress_size_gate.py`, `test_kompress_line_integrity.py`, `test_kompress_failsafe.py`, `test_kompress_marker_gate.py`) report `10 failed, 91 passed, 15 skipped`; the same 10 fail on the unmodified tree (`10 failed, 89 passed, 15 skipped`) — they need `torch`/`onnxruntime`/`httpx`, which are not installed here. ## Real Behavior Proof - Environment: headroom source checkout on Windows, Python 3.12; no third-party deps installed - Exact command / steps: the new tests install a tokenizer that models the real `truncation=True, max_length=512`: every word costs a number of tokens, so a 350-word chunk of token-heavy words runs past the limit, and the cut can fall inside a word - Observed result: before the fix the tail of the chunk vanished from the output entirely; after the fix the tail is kept verbatim, the word the cut landed in is kept too, and a chunk that fits is still left to the model - Not tested: the real ONNX model and a real tokenizer (no `onnxruntime`/`torch`/`transformers` here), and the end-to-end proxy path The four new tests fail on the unmodified tree and pass with the fix: ```text $ pytest tests/test_kompress_must_keep.py -q -k Truncated # fix reverted FAILED ...::test_words_past_the_truncation_point_survive FAILED ...::test_the_word_the_cut_landed_in_is_kept FAILED ...::test_batch_path_keeps_truncated_words_too FAILED ...::test_a_forced_ratio_still_gets_its_budget 4 failed, 1 passed, 18 deselected ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: only for chunks whose tokenized length exceeds 512 — the truncated tail is now kept instead of dropped, so the compressed output is longer and no longer silently lossy - Kill switch / disable path: n/a (the change only adds words that were previously discarded) - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - The new tests live in `tests/test_kompress_must_keep.py` next to the existing must-keep override tests. They model the real 512-token limit (a word costs several tokens, and the cut can fall inside a word) rather than pretending the chunk is cut at a word boundary. - On the live-tokenizer point: this machine has no `transformers` and no model weights, so a live tokenizer run is not possible here, and the repository's own kompress tests inject a fake model and tokenizer for the same reason (`tests/test_kompress_failsafe.py`). The fake above follows the real tokenizer's contract — `is_split_into_words` word ids, a cut at `max_length`, and a cut that can land inside a word. - `opentelemetry` is not installed on this machine, so `import headroom` fails and the suite cannot be collected as-is. The test run above used a minimal local stub for the `opentelemetry` API surface that `headroom/observability` imports; no product code was changed to accommodate it, and the stub is not part of this PR. - `ruff` and `mypy` were not run: neither is installed here. ## Model Used MiniMax M3.1 Flash Free via the opencode agent harness (`minimax-m3.1-flash-free`) --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
9355ec7019
commit
bb13384829
@@ -87,6 +87,8 @@ _KOMPRESS_MUST_KEEP_RE = re.compile(
|
||||
r"|(?i:\b(?:and|or|nor|xor)\b)"
|
||||
)
|
||||
_KOMPRESS_MUST_KEEP_ENV = "HEADROOM_KOMPRESS_MUST_KEEP"
|
||||
#: Tokenizer budget per chunk. Words past it are never scored (#4309).
|
||||
_KOMPRESS_MAX_TOKENS = 512
|
||||
KOMPRESS_BACKEND_ENV = "HEADROOM_KOMPRESS_BACKEND"
|
||||
KOMPRESS_ONNX_FILENAME_ENV = "HEADROOM_KOMPRESS_ONNX_FILENAME"
|
||||
|
||||
@@ -142,6 +144,39 @@ def _add_kompress_must_keep_words(
|
||||
kept_ids.add(word_idx + chunk_start)
|
||||
|
||||
|
||||
def _add_truncated_words(
|
||||
kept_ids: set[int],
|
||||
word_ids: list[int | None],
|
||||
chunk_words: list[str],
|
||||
chunk_start: int,
|
||||
token_count: int,
|
||||
) -> None:
|
||||
"""Keep the words the tokenizer's token cut took away.
|
||||
|
||||
Chunks are cut by words (``chunk_words``) but tokenized with
|
||||
``truncation=True, max_length=512``. Words past token 512 get no
|
||||
``word_id``, so they are never scored and never make ``kept_ids`` -- they
|
||||
are dropped silently, regardless of importance, and the loss is
|
||||
positional: whatever sits at the end of the chunk disappears (#4309).
|
||||
|
||||
The cut can also land inside a word. Its visible prefix counts as
|
||||
covered, so the model scores part of a word and may drop the whole of
|
||||
it, taking the unseen suffix with it. The last covered word is kept as
|
||||
well whenever the chunk reached the token limit.
|
||||
"""
|
||||
covered = max((wid for wid in word_ids if wid is not None), default=-1) + 1
|
||||
first_dropped = covered
|
||||
if token_count >= _KOMPRESS_MAX_TOKENS and covered > 0:
|
||||
first_dropped = covered - 1
|
||||
for local_idx in range(first_dropped, len(chunk_words)):
|
||||
kept_ids.add(local_idx + chunk_start)
|
||||
|
||||
|
||||
def _real_token_count(attention_row: Any) -> int:
|
||||
"""Number of real (non-padding) tokens in an attention-mask row."""
|
||||
return sum(1 for mask in attention_row if mask)
|
||||
|
||||
|
||||
_TABULAR_ROW_SPLIT_RE = re.compile(r"\s{2,}")
|
||||
|
||||
|
||||
@@ -2059,6 +2094,7 @@ class KompressCompressor(Transform):
|
||||
input_ids = encoding["input_ids"]
|
||||
attention_mask = encoding["attention_mask"]
|
||||
word_ids = encoding.word_ids(batch_index=0)
|
||||
token_count = _real_token_count(attention_mask[0])
|
||||
# One row, so no padding: every token in it is real.
|
||||
scored = _words_seen_whole(word_ids, len(word_ids), len(chunk_words))
|
||||
if scored == 0:
|
||||
@@ -2157,6 +2193,11 @@ class KompressCompressor(Transform):
|
||||
continue
|
||||
if bool(mask_list[idx]):
|
||||
kept_ids.add(wid + chunk_start)
|
||||
# The words past the tokenizer's token cut were never
|
||||
# scored, so the model cannot have kept them. A forced
|
||||
# ratio is a hard budget the caller asked for, so only the
|
||||
# model-decides path takes them back (#4309).
|
||||
_add_truncated_words(kept_ids, word_ids, chunk_words, chunk_start, token_count)
|
||||
|
||||
# Hard override: always keep must-keep tokens regardless of model score.
|
||||
# Numbers, error names, paths, and flags carry meaning agents cannot
|
||||
@@ -2591,6 +2632,17 @@ class KompressCompressor(Transform):
|
||||
for wid, score in word_scores.items():
|
||||
if score > self.config.score_threshold:
|
||||
kept_ids_per_text[text_idx].add(wid + chunk_start)
|
||||
# The words past the tokenizer's token cut were never
|
||||
# scored, so the model cannot have kept them. A forced
|
||||
# ratio is a hard budget the caller asked for, so only
|
||||
# the model-decides path takes them back (#4309).
|
||||
_add_truncated_words(
|
||||
kept_ids_per_text[text_idx],
|
||||
word_ids,
|
||||
chunk_words,
|
||||
chunk_start,
|
||||
_real_token_count(attention_mask[batch_idx]),
|
||||
)
|
||||
|
||||
_add_kompress_must_keep_words(
|
||||
kept_ids_per_text[text_idx], chunk_words, chunk_start
|
||||
|
||||
@@ -51,6 +51,10 @@ def test_single_and_batch_device_copies_hold_execution_slot(monkeypatch):
|
||||
def __init__(self, row_lengths):
|
||||
self.row_lengths = row_lengths
|
||||
|
||||
def __getitem__(self, index):
|
||||
# Real attention masks expose each row for padding-aware counts.
|
||||
return [1] * self.row_lengths[index]
|
||||
|
||||
def to(self, _device):
|
||||
acquired = semaphore.acquire(blocking=False)
|
||||
if acquired:
|
||||
|
||||
@@ -192,3 +192,123 @@ class TestMustKeepCompression:
|
||||
)
|
||||
|
||||
assert result.compressed.split() == ["alpha", "0x7fff2038"]
|
||||
|
||||
|
||||
class _TruncatingTok:
|
||||
"""Tokenizer that models ``truncation=True, max_length=512``.
|
||||
|
||||
Every word costs ``tokens_per_word`` tokens, which is how a chunk of
|
||||
token-heavy words (logs, IDs, paths, numbers) runs past the token limit
|
||||
before its word budget is used up. Words past the limit get no
|
||||
``word_id`` at all, exactly as the real tokenizer reports them. The cut
|
||||
can also land inside a word: that word keeps the tokens that fit and
|
||||
the rest of it never reaches the tokenizer.
|
||||
"""
|
||||
|
||||
def __init__(self, tokens_per_word: int = 1, max_length: int = 512):
|
||||
self.tokens_per_word = tokens_per_word
|
||||
self.max_length = max_length
|
||||
|
||||
def __call__(self, chunk_words, **kw):
|
||||
batch_words = (
|
||||
chunk_words if (chunk_words and isinstance(chunk_words[0], list)) else [chunk_words]
|
||||
)
|
||||
word_ids = []
|
||||
for words in batch_words:
|
||||
ids: list[int] = []
|
||||
used = 0
|
||||
for idx in range(len(words)):
|
||||
room = self.max_length - used
|
||||
if room <= 0:
|
||||
break
|
||||
take = min(self.tokens_per_word, room)
|
||||
ids.extend([idx] * take)
|
||||
used += take
|
||||
word_ids.append(ids)
|
||||
return _Enc(
|
||||
input_ids=[[0] * len(ids) for ids in word_ids],
|
||||
attention_mask=[[1] * len(ids) for ids in word_ids],
|
||||
_word_ids=word_ids,
|
||||
)
|
||||
|
||||
|
||||
class TestTruncatedWordsCompression:
|
||||
"""Words past the tokenizer's 512-token cut are kept, not dropped (#4309).
|
||||
|
||||
Chunks are cut by words (``chunk_words``) but tokenized with
|
||||
``truncation=True, max_length=512``. A chunk of token-heavy words hits
|
||||
the token limit well before its word budget, so the tail of the chunk had
|
||||
no score and was dropped silently from the default proxy path
|
||||
(``target_ratio=None``).
|
||||
"""
|
||||
|
||||
# One chunk is `chunk_words` words long, so 350 words is exactly one.
|
||||
CHUNK_WORDS = 350
|
||||
|
||||
def _words(self, count: int) -> list[str]:
|
||||
return [f"word{i}" for i in range(count)]
|
||||
|
||||
def _install(self, monkeypatch, tokens_per_word):
|
||||
monkeypatch.setattr(
|
||||
kc,
|
||||
"_load_kompress",
|
||||
lambda *a, **k: (_Model(), _TruncatingTok(tokens_per_word), "onnx"),
|
||||
)
|
||||
monkeypatch.setattr(kc, "_model_device_type", lambda *a, **k: "cpu")
|
||||
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
|
||||
|
||||
def _compress(self, monkeypatch, tokens_per_word):
|
||||
self._install(monkeypatch, tokens_per_word)
|
||||
words = self._words(self.CHUNK_WORDS)
|
||||
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
||||
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
|
||||
kept = compressor.compress(" ".join(words)).compressed.split()
|
||||
return kept, words
|
||||
|
||||
def test_words_past_the_truncation_point_survive(self, monkeypatch):
|
||||
# Two tokens per word: the 350-word chunk needs 700 tokens, so the
|
||||
# tokenizer cuts it at 512. The 256 words it covered were scored, the
|
||||
# other 94 were never seen and used to vanish from the output.
|
||||
kept, words = self._compress(monkeypatch, tokens_per_word=2)
|
||||
# 256 words were covered; the cut landed at a word boundary here, but
|
||||
# the last covered word is kept too, because the cut may have fallen
|
||||
# inside it.
|
||||
assert kept == [words[0]] + words[255:]
|
||||
|
||||
def test_the_word_the_cut_landed_in_is_kept(self, monkeypatch):
|
||||
# Three tokens per word: the cut at 512 falls inside word 170, whose
|
||||
# last token never reached the tokenizer. The model only saw part of
|
||||
# that word, so dropping it would take its unseen suffix with it.
|
||||
kept, words = self._compress(monkeypatch, tokens_per_word=3)
|
||||
assert kept == [words[0]] + words[170:]
|
||||
|
||||
def test_fully_covered_chunk_is_unchanged(self, monkeypatch):
|
||||
# One token per word: the chunk fits in 512 tokens, so nothing was
|
||||
# truncated and the model's own choice stands. It kept the first word
|
||||
# and dropped the rest.
|
||||
kept, words = self._compress(monkeypatch, tokens_per_word=1)
|
||||
assert kept == [words[0]]
|
||||
|
||||
def test_batch_path_keeps_truncated_words_too(self, monkeypatch):
|
||||
self._install(monkeypatch, 2)
|
||||
words = self._words(self.CHUNK_WORDS)
|
||||
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
||||
monkeypatch.setattr(compressor, "_should_use_sequential_fallback", lambda: False)
|
||||
|
||||
[result] = compressor.compress_batch([" ".join(words)], batch_size=8)
|
||||
|
||||
assert result.compressed.split() == [words[0]] + words[255:]
|
||||
|
||||
def test_a_forced_ratio_still_gets_its_budget(self, monkeypatch):
|
||||
# A forced ratio selects scored words within each token window.
|
||||
self._install(monkeypatch, 2)
|
||||
words = self._words(self.CHUNK_WORDS)
|
||||
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
||||
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
|
||||
|
||||
kept = compressor.compress(" ".join(words), target_ratio=0.5).compressed.split()
|
||||
|
||||
# Main now scores the formerly unseen tail in another window.
|
||||
# Windows score 255 and 95 complete words, each applying the
|
||||
# requested ratio independently with rounding down (127 + 47).
|
||||
assert kept == words[:127] + words[255:302]
|
||||
|
||||
Reference in New Issue
Block a user