Files
headroom/tests/test_kompress_token_window.py
421d453c4c fix(kompress): never drop words the model did not see (#4175)
## Description

Kompress splits text into 350-word chunks and tokenizes each chunk with
`truncation=True, max_length=512`. URLs, code, paths and citation
markers take several tokens per word, so a dense chunk runs past 512
tokens. Words beyond the cut never reach the model, get no keep
decision, and are dropped along with whatever the model did choose to
drop. The cut can also land inside a word, so the model judges the last
word it sees from a prefix.

This is the largest source of word loss I found in a week of auditing
Kompress output. On Headroom's own README, configuration docs and
changelog, about 30% of words fall past the window, and 70 to 76% of the
words Kompress drops on `main` were never scored. On my own Claude Code
and Codex traffic, 491 of 709 Kompress blocks had truncated chunks.

Closes: no existing issue. I searched issues and PRs for `max_length`,
`truncation`, `chunk_words` and `token window` and found nothing on
this, and `main` at 976aa714 still has the same chunking.
CONTRIBUTING.md sends bug fixes straight to a PR.

This follows the architecture principle in CONTRIBUTING.md, "never drop
user/assistant content ... prefer false negatives": a word the model
never scored is kept rather than dropped.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_words_seen_whole()`: counts the leading words of a chunk the model
saw in full. Once the chunk was cut (words are missing from `word_ids`,
or the token cap was hit) the last word present may be a prefix and does
not count. The cap moves into `_KOMPRESS_MAX_TOKENS`, which both
tokenizer calls use.
- `KompressCompressor.compress()`: the chunk loop advances by the words
seen whole instead of a fixed 350, so the rest (including a split word)
start the next chunk and every word gets scored. A word that fills the
window by itself is kept. Must-keep matching runs on the scored part of
the chunk only, since the remainder is matched with the next chunk.
- `KompressCompressor.compress_batch()`: chunks are built before
tokenizing, so re-chunking is not possible there without restructuring
the batch. Words not seen whole are kept instead of dropped: an unscored
word is not one the model chose to drop. Per-row token counts come from
`word_ids` (`_row_token_counts()`): padding and special tokens both map
to no word, and the longest row has no padding. This path only runs on
CUDA and MPS.
- Both scoring loops skip words from `scored` on: a split word's prefix
is rescored or kept anyway, so its prefix score must not take a top-k
slot from a whole word under `target_ratio`.
- `tests/test_kompress_token_window.py`: a fake tokenizer with a
seven-token window. Three cases where every word must come back (cut
between words, cut inside a word with a model that rejects the trailing
token, one word longer than the window), and one forced-ratio case where
a high-scoring prefix must not displace a whole word, both paths each,
plus a direct check that `_row_token_counts()` tells padding from
special tokens. None of the nine pass on `main`; all pass here.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_kompress_*.py tests/test_transforms_content_router.py \
    tests/test_transforms_content_detection.py tests/test_force_kompress_all.py \
    tests/test_router_registry_dispatch.py tests/test_content_router_single_item_deadline.py
258 passed, 1 warning in 11.07s

# same set on main 976aa714, with only the new test file added:
FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress-cut between words]
FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress-cut inside a word]
FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress-word longer than the window]
FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress_batch-cut between words]
FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress_batch-cut inside a word]
FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress_batch-word longer than the window]
FAILED tests/test_kompress_token_window.py::test_a_split_word_takes_no_slot_under_a_forced_ratio[compress]
FAILED tests/test_kompress_token_window.py::test_a_split_word_takes_no_slot_under_a_forced_ratio[compress_batch]
FAILED tests/test_kompress_token_window.py::test_row_token_counts_tell_padding_from_special_tokens
9 failed, 249 passed, 1 warning

# the Python test list from `make ci-precheck-python`, branch and main alike:
181 passed, 5 warnings

$ ruff check .
All checks passed!
$ ruff format --check .
1778 files already formatted

# mypy 1.20.2 as the pre-commit hook runs it (`mypy --ignore-missing-imports headroom`)
# stops on numpy's stubs on this machine, on main too, so with the stubs' Python version:
$ mypy --ignore-missing-imports --python-version 3.13 headroom
Found 11 errors in 4 files (checked 567 source files)
# identical 11 errors on main; none in headroom/transforms/kompress_compressor.py

$ git log --format=%B -1 <each commit> | commitlint    # repo .commitlintrc.json
(no problems, all three commits)
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.13, headroom-ai from this branch
and from `main` at 976aa714, real `chopratejas/kompress-v2-base` ONNX
model on CPU, `KompressConfig(enable_ccr=False)` defaults otherwise.
- Exact command / steps: ran the real model over public text from this
repository with the code at `main` and at this branch, then counted,
with the real ModernBERT tokenizer, which words `main` never scores
(fixed 350-word chunks truncated at 512 tokens). Scripts below.
- Observed result: on `main`, 70 to 76% of the words Kompress drops from
these documents were never scored; with the fix every word is scored and
the share of dropped words falls from 33 to 38% to 12 to 16%. Full table
and an example stretch:

```text
document                                words  unscored  dropped main  of which unscored  dropped fix
CHANGELOG.md (first 3,000 words)         2997       896   1051 (35.1%)         734 ( 70%)   449 (15.0%)
README.md                                4445      1405   1489 (33.5%)        1127 ( 76%)   543 (12.2%)
docs/content/docs/configuration.mdx      4943      1629   1888 (38.2%)        1345 ( 71%)   802 (16.2%)

First unscored stretch, CHANGELOG.md:
  original: manifest in the wrong location and failed outright even though `install apply` itself succeeded. `--env` values are merged into `DeploymentManifest.base_env` last, so they can override auto-derived defaults, and are threaded into the generated `run-headroom.sh`/`ensure-headroom.sh` (and Windows equivalents)
  main    : manifest in wrong location and failed outright even though `install apply` `--env` `DeploymentManifest.base_env` auto-derived and `run-headroom.sh`/`ensure-headroom.sh` (and
  fix     : manifest in wrong location and failed outright even though `install apply` itself succeeded. `--env` values merged into `DeploymentManifest.base_env` last, so override auto-derived defaults, and threaded into generated `run-headroom.sh`/`ensure-headroom.sh` (and Windows equivalents)
```

Latency, median of three runs on the same documents, `main` and the
branch run back to back (24-core CPU, shared machine):

```text
                      main      fix
CHANGELOG (3,000 w)  1755 ms   2641 ms
README.md            2578 ms   3987 ms
configuration.mdx    2908 ms   4355 ms
```

The model now reads the roughly 30% of words it skipped, so inference
takes about 50% longer on dense text (35 to 55% across the runs I did on
different days). The existing per-call and per-request time budgets
still apply and are checked at each chunk boundary.

- Not tested: the batched path with a real model on CUDA or MPS (covered
by the unit test with a fake tokenizer only);
`RemoteKompressCompressor`, which delegates to a server; models other
than `kompress-v2-base`.

<details><summary>proof.py</summary>

```python
"""Real-behavior proof for "never drop words the model did not see".

Runs the real Kompress ONNX model over public text from the Headroom repository
itself, once with the code at upstream main and once with the fix, and reports
for each document:

- how many words fall past the 512-token window of their 350-word chunk on main
  (computed with the real ModernBERT tokenizer, the same call Kompress makes),
- how many words each version drops,
- how many of main's dropped words the model never scored.

Usage: python proof.py <path-to-checkout> <output.json>
The checkout path decides which Kompress code is imported.
"""

from __future__ import annotations

import json
import sys

checkout, out_path = sys.argv[1], sys.argv[2]
sys.path.insert(0, checkout)

from headroom.transforms import kompress_compressor as kc  # noqa: E402

DOCS = {
    "CHANGELOG.md (first 3,000 words)": ("CHANGELOG.md", 3000),
    "README.md": ("README.md", None),
    "docs/content/docs/configuration.mdx": ("docs/content/docs/configuration.mdx", None),
}

compressor = kc.KompressCompressor(kc.KompressConfig(enable_ccr=False))
results = {}
for label, (rel, limit) in DOCS.items():
    text = open(f"{checkout}/{rel}", encoding="utf-8").read()
    if limit:
        text = " ".join(text.split(" ")[:limit])
    compressed = compressor.compress(text, allow_download=False).compressed
    results[label] = {"original": text, "compressed": compressed}
json.dump(results, open(out_path, "w"))
print("wrote", out_path)
```
</details>

<details><summary>report.py</summary>

```python
"""Compare proof.py output from main and from the fix."""

from __future__ import annotations

import json
import sys

from transformers import AutoTokenizer

before = json.load(open(sys.argv[1]))
after = json.load(open(sys.argv[2]))
tok = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base", local_files_only=True)
CHUNK = 350


def kept_indices(words: list[str], compressed: str) -> set[int]:
    """Kompress only deletes words, in order, so a greedy match recovers which survived."""
    out = compressed.split()
    kept, j = set(), 0
    for i, w in enumerate(words):
        if j < len(out) and out[j] == w:
            kept.add(i)
            j += 1
    return kept


def unscored_on_main(words: list[str]) -> set[int]:
    """Word indices main never scores: fixed 350-word chunks truncated at 512 tokens."""
    unseen = set()
    for start in range(0, len(words), CHUNK):
        chunk = words[start : start + CHUNK]
        enc = tok(chunk, is_split_into_words=True, truncation=True, max_length=512)
        seen = {w for w in enc.word_ids() if w is not None}
        unseen.update(start + i for i in range(len(chunk)) if i not in seen)
    return unseen


print(f"{'document':38} {'words':>6} {'unscored':>9} {'dropped main':>13} {'of which unscored':>18} {'dropped fix':>12}")
example = None
for label in before:
    words = before[label]["original"].split()
    unscored = unscored_on_main(words)
    drop_main = set(range(len(words))) - kept_indices(words, before[label]["compressed"])
    drop_fix = set(range(len(words))) - kept_indices(words, after[label]["compressed"])
    print(
        f"{label:38} {len(words):6} {len(unscored):9} {len(drop_main):6} ({100 * len(drop_main) / len(words):4.1f}%)"
        f" {len(drop_main & unscored):11} ({100 * len(drop_main & unscored) / max(len(drop_main), 1):3.0f}%)"
        f" {len(drop_fix):5} ({100 * len(drop_fix) / len(words):4.1f}%)"
    )
    if example is None and unscored:
        first = min(unscored)
        lo = max(0, first - 12)
        span = range(lo, min(len(words), first + 25))
        example = (
            label,
            " ".join(words[i] for i in span),
            " ".join(words[i] for i in span if i not in drop_main),
            " ".join(words[i] for i in span if i not in drop_fix),
        )

if example:
    label, orig, main_out, fix_out = example
    print(f"\nFirst unscored stretch, {label}:")
    print(f"  original: {orig}")
    print(f"  main    : {main_out}")
    print(f"  fix     : {fix_out}")
```
</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes. Kompress keeps more words on
dense text, and takes longer on it.
- Kill switch / disable path: `HEADROOM_DISABLE_KOMPRESS=1` disables
Kompress entirely
- Unsafe override required: no
- Qualification impact: none expected; compression ratio on dense text
drops because fewer unscored words are removed
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Screenshots (if applicable)

N/A, no UI change.

## AI Disclosure

The code change, the regression test, the proof scripts and this
description were drafted with Claude Code (Claude Opus 5.5; the two
split-word follow-ups from the review with Claude Fable 5.1) during an
audit of Kompress output on my own traffic. I reviewed every line and
reran the tests and the proof before submitting. The commit carries a
`Co-authored-by` trailer.

## Additional Notes

- Documentation: no docs describe the chunking, so nothing to update.
- Type checking: unchecked because `mypy headroom` does not pass on
`main` either; the change adds no errors (see Test Output).
- Test commands ran through a plain venv rather than `uv run`: ruff
0.16.8 as pinned in `pyproject.toml`, pytest 9.1.1 (the pin is
`>=7.0.0`).
- This is the first of a few Kompress fixes from the same audit
(code-shaped text reaching Kompress, identifiers and status words
missing from the must-keep list). Each will be its own PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-24 08:38:44 -07:00

160 lines
5.9 KiB
Python

"""Words the model never saw whole must not be dropped.
A chunk is 350 words, but the tokenizer truncates at 512 tokens. URLs, code and
citation markers take several tokens per word, so the tail of a dense chunk
never reached the model and every word in it was dropped, even when the model
would have kept it. The cut can also land inside a word, leaving the model a
prefix to judge.
"""
from __future__ import annotations
import numpy as np
import pytest
from headroom.transforms import kompress_compressor as kc
from headroom.transforms.kompress_compressor import KompressCompressor, KompressConfig
WINDOW = 7 # stands in for 512 tokens
CONTENT = " ".join(f"w{i}" for i in range(25))
class _Enc(dict):
def word_ids(self, batch_index=0):
return self["_word_ids"][batch_index]
class _Tok:
"""Gives each word ``tokens_per_word(word)`` tokens and cuts each row at WINDOW."""
def __init__(self, tokens_per_word):
self.tokens_per_word = tokens_per_word
def __call__(self, chunk_words, **kw):
batch = chunk_words if chunk_words and isinstance(chunk_words[0], list) else [chunk_words]
rows = [
[i for i, w in enumerate(words) for _ in range(self.tokens_per_word(w))][:WINDOW]
for words in batch
]
width = max(len(row) for row in rows)
return _Enc(
input_ids=np.zeros((len(rows), width), dtype=np.int64),
attention_mask=np.array([[1] * len(row) + [0] * (width - len(row)) for row in rows]),
_word_ids=[row + [None] * (width - len(row)) for row in rows],
)
class _KeepAllModel:
"""Keeps every token it is shown."""
def get_keep_mask(self, input_ids, attention_mask):
return [[True] * len(row) for row in input_ids]
def get_scores(self, input_ids, attention_mask):
return [[1.0] * len(row) for row in input_ids]
class _RejectsLastTokenModel:
"""Rejects the last token of each row and keeps the rest.
With two tokens per word a whole word survives on its first token. Only a
word the cut split, present as a single trailing token, is rejected.
"""
@staticmethod
def _keep(mask):
n = int(mask.sum())
return [i < n - 1 for i in range(len(mask))]
def get_keep_mask(self, input_ids, attention_mask):
return [self._keep(mask) for mask in attention_mask]
def get_scores(self, input_ids, attention_mask):
return [[1.0 if k else 0.0 for k in self._keep(mask)] for mask in attention_mask]
class _PrefixOutscoresModel:
"""Scores a row's last token highest, its first word next, the rest low.
A split word shows up as that last token. Under a forced ratio its prefix
score would take the one slot a whole word earned.
"""
def get_keep_mask(self, input_ids, attention_mask):
return [[s > 0.5 for s in row] for row in self.get_scores(input_ids, attention_mask)]
def get_scores(self, input_ids, attention_mask):
rows = []
for mask in attention_mask:
n = int(mask.sum())
rows.append([1.0 if i == n - 1 else 0.9 if i < 2 else 0.1 for i in range(len(mask))])
return rows
def _compress(monkeypatch, path, tokenizer, model, **kwargs):
"""Run CONTENT through one path with the fake model and tokenizer."""
monkeypatch.setattr(kc, "_load_kompress", lambda *a, **k: (model, tokenizer, "onnx"))
# The compressor must agree with the fake on where the window ends.
monkeypatch.setattr(kc, "_KOMPRESS_MAX_TOKENS", WINDOW)
monkeypatch.setattr(kc, "_model_device_type", lambda *a, **k: "cpu")
monkeypatch.setenv(kc._KOMPRESS_MUST_KEEP_ENV, "0")
compressor = KompressCompressor(
KompressConfig(enable_ccr=False, min_input_words=10, chunk_words=10)
)
if path == "compress":
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
return compressor.compress(CONTENT, **kwargs)
monkeypatch.setattr(compressor, "_should_use_sequential_fallback", lambda: False)
[result] = compressor.compress_batch([CONTENT], batch_size=8, **kwargs)
return result
CASES = {
"cut between words": (_Tok(lambda w: 1), _KeepAllModel()),
"cut inside a word": (_Tok(lambda w: 2), _RejectsLastTokenModel()),
"word longer than the window": (
_Tok(lambda w: WINDOW + 2 if w == "w5" else 2),
_RejectsLastTokenModel(),
),
}
@pytest.mark.parametrize("case", CASES)
@pytest.mark.parametrize("path", ["compress", "compress_batch"])
def test_words_the_model_did_not_see_whole_are_kept(monkeypatch, path, case):
result = _compress(monkeypatch, path, *CASES[case])
# The model keeps every word it sees whole, so nothing may go missing.
assert result.compressed.split() == CONTENT.split()
# Two tokens per word and a seven-token window, ratio 0.34 keeps one word of
# each scored group. The single path advances three words at a time and keeps
# the first of each group; the batch path keeps the first of each ten-word
# chunk plus the seven it never saw whole.
RATIO_EXPECTED = {
"compress": ["w0", "w3", "w6", "w9", "w12", "w15", "w18", "w21", "w24"],
"compress_batch": [
*["w0", "w3", "w4", "w5", "w6", "w7", "w8", "w9"],
*["w10", "w13", "w14", "w15", "w16", "w17", "w18", "w19"],
*["w20", "w23", "w24"],
],
}
@pytest.mark.parametrize("path", ["compress", "compress_batch"])
def test_a_split_word_takes_no_slot_under_a_forced_ratio(monkeypatch, path):
result = _compress(
monkeypatch, path, _Tok(lambda w: 2), _PrefixOutscoresModel(), target_ratio=0.34
)
assert result.compressed.split() == RATIO_EXPECTED[path]
def test_row_token_counts_tell_padding_from_special_tokens():
# [CLS] w0 w0 w1 [SEP] and [CLS] w0 [SEP] [PAD] [PAD]: the longest row has no
# padding, so its trailing run of None is the special-token count.
rows = [[None, 0, 0, 1, None], [None, 0, None, None, None]]
assert kc._row_token_counts(rows) == [5, 3]