mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description Kompress splits text into 350-word chunks and tokenizes each chunk with `truncation=True, max_length=512`. URLs, code, paths and citation markers take several tokens per word, so a dense chunk runs past 512 tokens. Words beyond the cut never reach the model, get no keep decision, and are dropped along with whatever the model did choose to drop. The cut can also land inside a word, so the model judges the last word it sees from a prefix. This is the largest source of word loss I found in a week of auditing Kompress output. On Headroom's own README, configuration docs and changelog, about 30% of words fall past the window, and 70 to 76% of the words Kompress drops on `main` were never scored. On my own Claude Code and Codex traffic, 491 of 709 Kompress blocks had truncated chunks. Closes: no existing issue. I searched issues and PRs for `max_length`, `truncation`, `chunk_words` and `token window` and found nothing on this, and `main` at976aa714still has the same chunking. CONTRIBUTING.md sends bug fixes straight to a PR. This follows the architecture principle in CONTRIBUTING.md, "never drop user/assistant content ... prefer false negatives": a word the model never scored is kept rather than dropped. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `_words_seen_whole()`: counts the leading words of a chunk the model saw in full. Once the chunk was cut (words are missing from `word_ids`, or the token cap was hit) the last word present may be a prefix and does not count. The cap moves into `_KOMPRESS_MAX_TOKENS`, which both tokenizer calls use. - `KompressCompressor.compress()`: the chunk loop advances by the words seen whole instead of a fixed 350, so the rest (including a split word) start the next chunk and every word gets scored. A word that fills the window by itself is kept. Must-keep matching runs on the scored part of the chunk only, since the remainder is matched with the next chunk. - `KompressCompressor.compress_batch()`: chunks are built before tokenizing, so re-chunking is not possible there without restructuring the batch. Words not seen whole are kept instead of dropped: an unscored word is not one the model chose to drop. Per-row token counts come from `word_ids` (`_row_token_counts()`): padding and special tokens both map to no word, and the longest row has no padding. This path only runs on CUDA and MPS. - Both scoring loops skip words from `scored` on: a split word's prefix is rescored or kept anyway, so its prefix score must not take a top-k slot from a whole word under `target_ratio`. - `tests/test_kompress_token_window.py`: a fake tokenizer with a seven-token window. Three cases where every word must come back (cut between words, cut inside a word with a model that rejects the trailing token, one word longer than the window), and one forced-ratio case where a high-scoring prefix must not displace a whole word, both paths each, plus a direct check that `_row_token_counts()` tells padding from special tokens. None of the nine pass on `main`; all pass here. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_kompress_*.py tests/test_transforms_content_router.py \ tests/test_transforms_content_detection.py tests/test_force_kompress_all.py \ tests/test_router_registry_dispatch.py tests/test_content_router_single_item_deadline.py 258 passed, 1 warning in 11.07s # same set on main976aa714, with only the new test file added: FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress-cut between words] FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress-cut inside a word] FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress-word longer than the window] FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress_batch-cut between words] FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress_batch-cut inside a word] FAILED tests/test_kompress_token_window.py::test_words_the_model_did_not_see_whole_are_kept[compress_batch-word longer than the window] FAILED tests/test_kompress_token_window.py::test_a_split_word_takes_no_slot_under_a_forced_ratio[compress] FAILED tests/test_kompress_token_window.py::test_a_split_word_takes_no_slot_under_a_forced_ratio[compress_batch] FAILED tests/test_kompress_token_window.py::test_row_token_counts_tell_padding_from_special_tokens 9 failed, 249 passed, 1 warning # the Python test list from `make ci-precheck-python`, branch and main alike: 181 passed, 5 warnings $ ruff check . All checks passed! $ ruff format --check . 1778 files already formatted # mypy 1.20.2 as the pre-commit hook runs it (`mypy --ignore-missing-imports headroom`) # stops on numpy's stubs on this machine, on main too, so with the stubs' Python version: $ mypy --ignore-missing-imports --python-version 3.13 headroom Found 11 errors in 4 files (checked 567 source files) # identical 11 errors on main; none in headroom/transforms/kompress_compressor.py $ git log --format=%B -1 <each commit> | commitlint # repo .commitlintrc.json (no problems, all three commits) ``` ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13, headroom-ai from this branch and from `main` at976aa714, real `chopratejas/kompress-v2-base` ONNX model on CPU, `KompressConfig(enable_ccr=False)` defaults otherwise. - Exact command / steps: ran the real model over public text from this repository with the code at `main` and at this branch, then counted, with the real ModernBERT tokenizer, which words `main` never scores (fixed 350-word chunks truncated at 512 tokens). Scripts below. - Observed result: on `main`, 70 to 76% of the words Kompress drops from these documents were never scored; with the fix every word is scored and the share of dropped words falls from 33 to 38% to 12 to 16%. Full table and an example stretch: ```text document words unscored dropped main of which unscored dropped fix CHANGELOG.md (first 3,000 words) 2997 896 1051 (35.1%) 734 ( 70%) 449 (15.0%) README.md 4445 1405 1489 (33.5%) 1127 ( 76%) 543 (12.2%) docs/content/docs/configuration.mdx 4943 1629 1888 (38.2%) 1345 ( 71%) 802 (16.2%) First unscored stretch, CHANGELOG.md: original: manifest in the wrong location and failed outright even though `install apply` itself succeeded. `--env` values are merged into `DeploymentManifest.base_env` last, so they can override auto-derived defaults, and are threaded into the generated `run-headroom.sh`/`ensure-headroom.sh` (and Windows equivalents) main : manifest in wrong location and failed outright even though `install apply` `--env` `DeploymentManifest.base_env` auto-derived and `run-headroom.sh`/`ensure-headroom.sh` (and fix : manifest in wrong location and failed outright even though `install apply` itself succeeded. `--env` values merged into `DeploymentManifest.base_env` last, so override auto-derived defaults, and threaded into generated `run-headroom.sh`/`ensure-headroom.sh` (and Windows equivalents) ``` Latency, median of three runs on the same documents, `main` and the branch run back to back (24-core CPU, shared machine): ```text main fix CHANGELOG (3,000 w) 1755 ms 2641 ms README.md 2578 ms 3987 ms configuration.mdx 2908 ms 4355 ms ``` The model now reads the roughly 30% of words it skipped, so inference takes about 50% longer on dense text (35 to 55% across the runs I did on different days). The existing per-call and per-request time budgets still apply and are checked at each chunk boundary. - Not tested: the batched path with a real model on CUDA or MPS (covered by the unit test with a fake tokenizer only); `RemoteKompressCompressor`, which delegates to a server; models other than `kompress-v2-base`. <details><summary>proof.py</summary> ```python """Real-behavior proof for "never drop words the model did not see". Runs the real Kompress ONNX model over public text from the Headroom repository itself, once with the code at upstream main and once with the fix, and reports for each document: - how many words fall past the 512-token window of their 350-word chunk on main (computed with the real ModernBERT tokenizer, the same call Kompress makes), - how many words each version drops, - how many of main's dropped words the model never scored. Usage: python proof.py <path-to-checkout> <output.json> The checkout path decides which Kompress code is imported. """ from __future__ import annotations import json import sys checkout, out_path = sys.argv[1], sys.argv[2] sys.path.insert(0, checkout) from headroom.transforms import kompress_compressor as kc # noqa: E402 DOCS = { "CHANGELOG.md (first 3,000 words)": ("CHANGELOG.md", 3000), "README.md": ("README.md", None), "docs/content/docs/configuration.mdx": ("docs/content/docs/configuration.mdx", None), } compressor = kc.KompressCompressor(kc.KompressConfig(enable_ccr=False)) results = {} for label, (rel, limit) in DOCS.items(): text = open(f"{checkout}/{rel}", encoding="utf-8").read() if limit: text = " ".join(text.split(" ")[:limit]) compressed = compressor.compress(text, allow_download=False).compressed results[label] = {"original": text, "compressed": compressed} json.dump(results, open(out_path, "w")) print("wrote", out_path) ``` </details> <details><summary>report.py</summary> ```python """Compare proof.py output from main and from the fix.""" from __future__ import annotations import json import sys from transformers import AutoTokenizer before = json.load(open(sys.argv[1])) after = json.load(open(sys.argv[2])) tok = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base", local_files_only=True) CHUNK = 350 def kept_indices(words: list[str], compressed: str) -> set[int]: """Kompress only deletes words, in order, so a greedy match recovers which survived.""" out = compressed.split() kept, j = set(), 0 for i, w in enumerate(words): if j < len(out) and out[j] == w: kept.add(i) j += 1 return kept def unscored_on_main(words: list[str]) -> set[int]: """Word indices main never scores: fixed 350-word chunks truncated at 512 tokens.""" unseen = set() for start in range(0, len(words), CHUNK): chunk = words[start : start + CHUNK] enc = tok(chunk, is_split_into_words=True, truncation=True, max_length=512) seen = {w for w in enc.word_ids() if w is not None} unseen.update(start + i for i in range(len(chunk)) if i not in seen) return unseen print(f"{'document':38} {'words':>6} {'unscored':>9} {'dropped main':>13} {'of which unscored':>18} {'dropped fix':>12}") example = None for label in before: words = before[label]["original"].split() unscored = unscored_on_main(words) drop_main = set(range(len(words))) - kept_indices(words, before[label]["compressed"]) drop_fix = set(range(len(words))) - kept_indices(words, after[label]["compressed"]) print( f"{label:38} {len(words):6} {len(unscored):9} {len(drop_main):6} ({100 * len(drop_main) / len(words):4.1f}%)" f" {len(drop_main & unscored):11} ({100 * len(drop_main & unscored) / max(len(drop_main), 1):3.0f}%)" f" {len(drop_fix):5} ({100 * len(drop_fix) / len(words):4.1f}%)" ) if example is None and unscored: first = min(unscored) lo = max(0, first - 12) span = range(lo, min(len(words), first + 25)) example = ( label, " ".join(words[i] for i in span), " ".join(words[i] for i in span if i not in drop_main), " ".join(words[i] for i in span if i not in drop_fix), ) if example: label, orig, main_out, fix_out = example print(f"\nFirst unscored stretch, {label}:") print(f" original: {orig}") print(f" main : {main_out}") print(f" fix : {fix_out}") ``` </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes. Kompress keeps more words on dense text, and takes longer on it. - Kill switch / disable path: `HEADROOM_DISABLE_KOMPRESS=1` disables Kompress entirely - Unsafe override required: no - Qualification impact: none expected; compression ratio on dense text drops because fewer unscored words are removed - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Screenshots (if applicable) N/A, no UI change. ## AI Disclosure The code change, the regression test, the proof scripts and this description were drafted with Claude Code (Claude Opus 5.5; the two split-word follow-ups from the review with Claude Fable 5.1) during an audit of Kompress output on my own traffic. I reviewed every line and reran the tests and the proof before submitting. The commit carries a `Co-authored-by` trailer. ## Additional Notes - Documentation: no docs describe the chunking, so nothing to update. - Type checking: unchecked because `mypy headroom` does not pass on `main` either; the change adds no errors (see Test Output). - Test commands ran through a plain venv rather than `uv run`: ruff 0.16.8 as pinned in `pyproject.toml`, pytest 9.1.1 (the pin is `>=7.0.0`). - This is the first of a few Kompress fixes from the same audit (code-shaped text reaching Kompress, identifiers and status words missing from the must-keep list). Each will be its own PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
160 lines
5.9 KiB
Python
160 lines
5.9 KiB
Python
"""Words the model never saw whole must not be dropped.
|
|
|
|
A chunk is 350 words, but the tokenizer truncates at 512 tokens. URLs, code and
|
|
citation markers take several tokens per word, so the tail of a dense chunk
|
|
never reached the model and every word in it was dropped, even when the model
|
|
would have kept it. The cut can also land inside a word, leaving the model a
|
|
prefix to judge.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import numpy as np
|
|
import pytest
|
|
|
|
from headroom.transforms import kompress_compressor as kc
|
|
from headroom.transforms.kompress_compressor import KompressCompressor, KompressConfig
|
|
|
|
WINDOW = 7 # stands in for 512 tokens
|
|
CONTENT = " ".join(f"w{i}" for i in range(25))
|
|
|
|
|
|
class _Enc(dict):
|
|
def word_ids(self, batch_index=0):
|
|
return self["_word_ids"][batch_index]
|
|
|
|
|
|
class _Tok:
|
|
"""Gives each word ``tokens_per_word(word)`` tokens and cuts each row at WINDOW."""
|
|
|
|
def __init__(self, tokens_per_word):
|
|
self.tokens_per_word = tokens_per_word
|
|
|
|
def __call__(self, chunk_words, **kw):
|
|
batch = chunk_words if chunk_words and isinstance(chunk_words[0], list) else [chunk_words]
|
|
rows = [
|
|
[i for i, w in enumerate(words) for _ in range(self.tokens_per_word(w))][:WINDOW]
|
|
for words in batch
|
|
]
|
|
width = max(len(row) for row in rows)
|
|
return _Enc(
|
|
input_ids=np.zeros((len(rows), width), dtype=np.int64),
|
|
attention_mask=np.array([[1] * len(row) + [0] * (width - len(row)) for row in rows]),
|
|
_word_ids=[row + [None] * (width - len(row)) for row in rows],
|
|
)
|
|
|
|
|
|
class _KeepAllModel:
|
|
"""Keeps every token it is shown."""
|
|
|
|
def get_keep_mask(self, input_ids, attention_mask):
|
|
return [[True] * len(row) for row in input_ids]
|
|
|
|
def get_scores(self, input_ids, attention_mask):
|
|
return [[1.0] * len(row) for row in input_ids]
|
|
|
|
|
|
class _RejectsLastTokenModel:
|
|
"""Rejects the last token of each row and keeps the rest.
|
|
|
|
With two tokens per word a whole word survives on its first token. Only a
|
|
word the cut split, present as a single trailing token, is rejected.
|
|
"""
|
|
|
|
@staticmethod
|
|
def _keep(mask):
|
|
n = int(mask.sum())
|
|
return [i < n - 1 for i in range(len(mask))]
|
|
|
|
def get_keep_mask(self, input_ids, attention_mask):
|
|
return [self._keep(mask) for mask in attention_mask]
|
|
|
|
def get_scores(self, input_ids, attention_mask):
|
|
return [[1.0 if k else 0.0 for k in self._keep(mask)] for mask in attention_mask]
|
|
|
|
|
|
class _PrefixOutscoresModel:
|
|
"""Scores a row's last token highest, its first word next, the rest low.
|
|
|
|
A split word shows up as that last token. Under a forced ratio its prefix
|
|
score would take the one slot a whole word earned.
|
|
"""
|
|
|
|
def get_keep_mask(self, input_ids, attention_mask):
|
|
return [[s > 0.5 for s in row] for row in self.get_scores(input_ids, attention_mask)]
|
|
|
|
def get_scores(self, input_ids, attention_mask):
|
|
rows = []
|
|
for mask in attention_mask:
|
|
n = int(mask.sum())
|
|
rows.append([1.0 if i == n - 1 else 0.9 if i < 2 else 0.1 for i in range(len(mask))])
|
|
return rows
|
|
|
|
|
|
def _compress(monkeypatch, path, tokenizer, model, **kwargs):
|
|
"""Run CONTENT through one path with the fake model and tokenizer."""
|
|
monkeypatch.setattr(kc, "_load_kompress", lambda *a, **k: (model, tokenizer, "onnx"))
|
|
# The compressor must agree with the fake on where the window ends.
|
|
monkeypatch.setattr(kc, "_KOMPRESS_MAX_TOKENS", WINDOW)
|
|
monkeypatch.setattr(kc, "_model_device_type", lambda *a, **k: "cpu")
|
|
monkeypatch.setenv(kc._KOMPRESS_MUST_KEEP_ENV, "0")
|
|
compressor = KompressCompressor(
|
|
KompressConfig(enable_ccr=False, min_input_words=10, chunk_words=10)
|
|
)
|
|
if path == "compress":
|
|
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
|
|
return compressor.compress(CONTENT, **kwargs)
|
|
monkeypatch.setattr(compressor, "_should_use_sequential_fallback", lambda: False)
|
|
[result] = compressor.compress_batch([CONTENT], batch_size=8, **kwargs)
|
|
return result
|
|
|
|
|
|
CASES = {
|
|
"cut between words": (_Tok(lambda w: 1), _KeepAllModel()),
|
|
"cut inside a word": (_Tok(lambda w: 2), _RejectsLastTokenModel()),
|
|
"word longer than the window": (
|
|
_Tok(lambda w: WINDOW + 2 if w == "w5" else 2),
|
|
_RejectsLastTokenModel(),
|
|
),
|
|
}
|
|
|
|
|
|
@pytest.mark.parametrize("case", CASES)
|
|
@pytest.mark.parametrize("path", ["compress", "compress_batch"])
|
|
def test_words_the_model_did_not_see_whole_are_kept(monkeypatch, path, case):
|
|
result = _compress(monkeypatch, path, *CASES[case])
|
|
|
|
# The model keeps every word it sees whole, so nothing may go missing.
|
|
assert result.compressed.split() == CONTENT.split()
|
|
|
|
|
|
# Two tokens per word and a seven-token window, ratio 0.34 keeps one word of
|
|
# each scored group. The single path advances three words at a time and keeps
|
|
# the first of each group; the batch path keeps the first of each ten-word
|
|
# chunk plus the seven it never saw whole.
|
|
RATIO_EXPECTED = {
|
|
"compress": ["w0", "w3", "w6", "w9", "w12", "w15", "w18", "w21", "w24"],
|
|
"compress_batch": [
|
|
*["w0", "w3", "w4", "w5", "w6", "w7", "w8", "w9"],
|
|
*["w10", "w13", "w14", "w15", "w16", "w17", "w18", "w19"],
|
|
*["w20", "w23", "w24"],
|
|
],
|
|
}
|
|
|
|
|
|
@pytest.mark.parametrize("path", ["compress", "compress_batch"])
|
|
def test_a_split_word_takes_no_slot_under_a_forced_ratio(monkeypatch, path):
|
|
result = _compress(
|
|
monkeypatch, path, _Tok(lambda w: 2), _PrefixOutscoresModel(), target_ratio=0.34
|
|
)
|
|
|
|
assert result.compressed.split() == RATIO_EXPECTED[path]
|
|
|
|
|
|
def test_row_token_counts_tell_padding_from_special_tokens():
|
|
# [CLS] w0 w0 w1 [SEP] and [CLS] w0 [SEP] [PAD] [PAD]: the longest row has no
|
|
# padding, so its trailing run of None is the special-token count.
|
|
rows = [[None, 0, 0, 1, None], [None, 0, None, None, None]]
|
|
|
|
assert kc._row_token_counts(rows) == [5, 3]
|