mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
#4175 and #4328 both fix #4309 (words past a chunk's 512-token cut were
dropped without a score), and together they undo each other's intent.
Since #4175, the sequential `compress()` path advances by the words the
tokenizer saw whole, so the words past the cut open the next window and
the model scores them there. #4328 then adds those same words to
`kept_ids` before that window runs. `kept_ids` only grows, so the next
window's drop decisions never take effect: on the default path
(`target_ratio=None`), every word past every cut is kept verbatim. On
token-dense text (logs, git output, paths, IDs) the cut lands early in
each 350-word chunk: in the samples below, 31-66% of all words sat past
a cut and now skip the model.
This removes `_add_truncated_words` and its two calls. What it protected
stays protected:
- Words past the cut are scored by the next window on the sequential
path, and kept verbatim by #4175's own range on the batch path (the
helper added that same range again there).
- A word the cut split is not counted as seen (`_words_seen_whole`), so
it opens the next window and is scored whole.
Found while preparing Extra Headroom's move from 0.42.0 to 0.43.
Related #4309, #4175, #4328
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/kompress_compressor.py`: remove
`_add_truncated_words` and its calls in `compress()` and
`compress_batch()`, plus `_real_token_count` (used only to feed it) and
the second `_KOMPRESS_MAX_TOKENS = 512` that #4328 added above the
original.
- `tests/test_kompress_must_keep.py`: the two sequential-path tests now
assert the next window's decision (`[w0, w255]` and `[w0, w170, w340]`)
instead of a verbatim tail. The batch-path, forced-ratio and
fully-covered tests are unchanged and pass.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ HF_HUB_OFFLINE=1 uv run --frozen --extra dev pytest tests/test_kompress_*.py tests/test_transforms_content_router.py \
tests/test_force_kompress_all.py tests/test_router_registry_dispatch.py tests/test_content_router_single_item_deadline.py
278 passed, 1 skipped in 7.60s
# the updated tests against unmodified main (this branch's tests, main's kompress_compressor.py):
FAILED tests/test_kompress_must_keep.py::TestTruncatedWordsCompression::test_words_past_the_truncation_point_are_scored
FAILED tests/test_kompress_must_keep.py::TestTruncatedWordsCompression::test_the_word_the_cut_landed_in_is_scored_whole
2 failed, 3 passed, 18 deselected
$ uvx ruff check headroom/transforms/kompress_compressor.py tests/test_kompress_must_keep.py
All checks passed!
$ uvx ruff format --check headroom/transforms/kompress_compressor.py tests/test_kompress_must_keep.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/transforms/kompress_compressor.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 27.2 arm64, Python 3.12.15, headroom-ai 0.43.1 from
PyPI with onnxruntime 1.31.1, the real Kompress model on the ONNX CPU
backend; for the branch run the worktree is first on `PYTHONPATH`.
- Exact command / steps:
`KompressCompressor(KompressConfig(enable_ccr=False)).compress(text)` on
five 4,000-word samples of token-dense text (this repo's CHANGELOG.md,
an application debug log, `git log --stat`, a Markdown doc full of
paths, a Rust source file), on 0.42.0, 0.43.1 and this branch; the
figure is `tokens_saved / original_tokens`.
- Observed result: Kompress removes 14.1% overall on 0.42.0, 5.7% on
0.43.1 and 14.1% again on this branch, matching 0.42.0 on every sample
(table below).
- Not tested: the CUDA/MPS batch path with the real model (its output is
unchanged by construction: the removed call re-added a range #4175
already keeps); answer quality downstream of the model's own keep
decisions.
| Sample | 0.42.0 | 0.43.1 | This branch |
|---|---|---|---|
| CHANGELOG.md | 15.6% | 6.7% | 15.6% |
| Application debug log | 10.9% | 0.4% | 10.9% |
| `git log --stat` | 12.2% | 3.5% | 12.2% |
| Markdown doc | 16.4% | 8.7% | 16.4% |
| Rust source | 14.8% | 8.5% | 14.8% |
| All | 14.1% | 5.7% | 14.1% |
## Runtime Rollout Safety
- Rollout-managed feature(s): None, this is Kompress compression on the
default path.
- Minimum rollout channel: N/A, not rollout-managed.
- Stable/default behavior changed: Yes, on the sequential path the words
past a chunk's token cut now follow the model's keep decision in the
next window instead of always being kept, which restores 0.42.0's
output.
- Kill switch / disable path: No new switch; the existing Kompress
disable paths are unchanged.
- Unsafe override required: No.
- Qualification impact: Kompress removes more on token-dense text, back
to 0.42.0 levels; every word is still scored before it can be dropped.
- Rollback path: Revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation: N/A, no documented behaviour changes; the class
docstring in `tests/test_kompress_must_keep.py` now describes both
paths.
- The 0.42.0 column was measured on an installed 0.42.0 runtime. This
branch's `kompress_compressor.py` is byte-identical to 0.42.0's (`git
diff 17d38e56 HEAD -- headroom/transforms/kompress_compressor.py` is
empty), since #4328 was the only change to it in 0.43.0 and 0.43.1.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4669"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
319 lines
13 KiB
Python
319 lines
13 KiB
Python
"""Tests for the must-keep token override in kompress_compressor."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
|
|
from headroom.transforms import kompress_compressor as kc
|
|
from headroom.transforms.kompress_compressor import (
|
|
_KOMPRESS_MUST_KEEP_ENV,
|
|
_KOMPRESS_MUST_KEEP_RE,
|
|
KompressCompressor,
|
|
KompressConfig,
|
|
)
|
|
|
|
|
|
class _Enc(dict):
|
|
def word_ids(self, batch_index=0):
|
|
return self["_word_ids"][batch_index]
|
|
|
|
|
|
class _Tok:
|
|
def __call__(self, chunk_words, **kw):
|
|
if chunk_words and isinstance(chunk_words[0], list):
|
|
batch_words = chunk_words
|
|
else:
|
|
batch_words = [chunk_words]
|
|
return _Enc(
|
|
input_ids=[[0] * len(words) for words in batch_words],
|
|
attention_mask=[[1] * len(words) for words in batch_words],
|
|
_word_ids=[list(range(len(words))) for words in batch_words],
|
|
)
|
|
|
|
|
|
class _Model:
|
|
def get_keep_mask(self, input_ids, attention_mask):
|
|
return [[idx == 0 for idx, _ in enumerate(row)] for row in input_ids]
|
|
|
|
def get_scores(self, input_ids, attention_mask):
|
|
return [[1.0 if idx == 0 else 0.0 for idx, _ in enumerate(row)] for row in input_ids]
|
|
|
|
|
|
def _install_fake_kompress(monkeypatch):
|
|
monkeypatch.setattr(kc, "_load_kompress", lambda *a, **k: (_Model(), _Tok(), "onnx"))
|
|
monkeypatch.setattr(kc, "_model_device_type", lambda *a, **k: "cpu")
|
|
|
|
|
|
class TestMustKeepRegex:
|
|
def test_numbers(self):
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("42")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("3.14")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("0x7fff2038")
|
|
assert not _KOMPRESS_MUST_KEEP_RE.search("word0")
|
|
|
|
def test_allcaps(self):
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("SIGILL")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("HTTP")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("EOF")
|
|
|
|
def test_dotted_paths(self):
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("libsystem_kernel.dylib")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("torch.nn")
|
|
|
|
def test_unix_paths(self):
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("/usr/lib/python3")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("/workspace/ultrawhale")
|
|
|
|
def test_extensions(self):
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("model.py")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("weights.so")
|
|
|
|
def test_flags(self):
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("--verbose")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("-n")
|
|
|
|
def test_camelcase(self):
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("IndexError")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("EXC_BAD_INSTRUCTION")
|
|
|
|
def test_plain_words_not_matched(self):
|
|
assert not _KOMPRESS_MUST_KEEP_RE.search("the")
|
|
assert not _KOMPRESS_MUST_KEEP_RE.search("process")
|
|
assert not _KOMPRESS_MUST_KEEP_RE.search("raised")
|
|
|
|
def test_boolean_connectives(self):
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("or")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("and")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("nor")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("xor")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("or,")
|
|
assert _KOMPRESS_MUST_KEEP_RE.search("(and)")
|
|
# Tokens that merely contain a connective are not connectives.
|
|
for word in ("orange", "random", "android", "core", "sort", "concat", "node"):
|
|
assert not _KOMPRESS_MUST_KEEP_RE.search(word), word
|
|
|
|
|
|
class TestBooleanConnectivesCompression:
|
|
"""`and`/`or` hold a condition together, so dropping one changes the
|
|
predicate rather than degrading the sentence. Issue #3545 measured 12 of 40
|
|
`or` lost from a repeated Python return line on the released compressor,
|
|
while `not` from the very same line survived because negation is already
|
|
pinned -- the result still reads as a line of Python and nothing marks it
|
|
as altered."""
|
|
|
|
CODE = 'return name == "__init__" or not name.startswith("_") alpha beta gamma delta'
|
|
|
|
def _compress(self, monkeypatch, env):
|
|
_install_fake_kompress(monkeypatch)
|
|
if env:
|
|
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
|
|
else:
|
|
monkeypatch.setenv(_KOMPRESS_MUST_KEEP_ENV, "0")
|
|
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
|
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
|
|
return compressor.compress(self.CODE)
|
|
|
|
def test_keeps_both_operands_of_a_negated_disjunction(self, monkeypatch):
|
|
kept = self._compress(monkeypatch, env=True).compressed.split()
|
|
assert "not" in kept
|
|
assert "or" in kept
|
|
|
|
def test_batch_path_keeps_connectives_too(self, monkeypatch):
|
|
_install_fake_kompress(monkeypatch)
|
|
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
|
|
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
|
monkeypatch.setattr(compressor, "_should_use_sequential_fallback", lambda: False)
|
|
|
|
[result] = compressor.compress_batch(
|
|
["alpha beta gamma delta epsilon zeta eta theta iota kappa one and two or three"],
|
|
batch_size=8,
|
|
)
|
|
|
|
kept = result.compressed.split()
|
|
assert "and" in kept
|
|
assert "or" in kept
|
|
|
|
def test_disabling_must_keep_leaves_the_connective_to_the_model(self, monkeypatch):
|
|
kept = self._compress(monkeypatch, env=False).compressed.split()
|
|
assert "or" not in kept
|
|
assert "not" not in kept
|
|
|
|
|
|
class TestMustKeepEnvVar:
|
|
def test_env_var_name(self):
|
|
assert _KOMPRESS_MUST_KEEP_ENV == "HEADROOM_KOMPRESS_MUST_KEEP"
|
|
|
|
def test_env_var_default_is_enabled(self, monkeypatch):
|
|
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
|
|
assert os.environ.get(_KOMPRESS_MUST_KEEP_ENV, "1") != "0"
|
|
|
|
def test_env_var_can_disable(self, monkeypatch):
|
|
monkeypatch.setenv(_KOMPRESS_MUST_KEEP_ENV, "0")
|
|
assert os.environ.get(_KOMPRESS_MUST_KEEP_ENV, "1") == "0"
|
|
|
|
|
|
class TestMustKeepCompression:
|
|
def test_compress_keeps_must_keep_word_when_model_drops_it(self, monkeypatch):
|
|
_install_fake_kompress(monkeypatch)
|
|
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
|
|
|
|
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
|
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
|
|
|
|
result = compressor.compress(
|
|
"alpha beta gamma delta epsilon zeta eta theta iota kappa 0x7fff2038 omega"
|
|
)
|
|
|
|
assert result.compressed.split() == ["alpha", "0x7fff2038"]
|
|
|
|
def test_compress_can_disable_must_keep_override(self, monkeypatch):
|
|
_install_fake_kompress(monkeypatch)
|
|
monkeypatch.setenv(_KOMPRESS_MUST_KEEP_ENV, "0")
|
|
|
|
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
|
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
|
|
|
|
result = compressor.compress(
|
|
"alpha beta gamma delta epsilon zeta eta theta iota kappa 0x7fff2038 omega"
|
|
)
|
|
|
|
assert result.compressed.split() == ["alpha"]
|
|
|
|
def test_compress_batch_keeps_must_keep_word_when_score_is_low(self, monkeypatch):
|
|
_install_fake_kompress(monkeypatch)
|
|
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
|
|
|
|
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
|
monkeypatch.setattr(compressor, "_should_use_sequential_fallback", lambda: False)
|
|
|
|
[result] = compressor.compress_batch(
|
|
["alpha beta gamma delta epsilon zeta eta theta iota kappa 0x7fff2038 omega"],
|
|
batch_size=8,
|
|
)
|
|
|
|
assert result.compressed.split() == ["alpha", "0x7fff2038"]
|
|
|
|
|
|
class _TruncatingTok:
|
|
"""Tokenizer that models ``truncation=True, max_length=512``.
|
|
|
|
Every word costs ``tokens_per_word`` tokens, which is how a chunk of
|
|
token-heavy words (logs, IDs, paths, numbers) runs past the token limit
|
|
before its word budget is used up. Words past the limit get no
|
|
``word_id`` at all, exactly as the real tokenizer reports them. The cut
|
|
can also land inside a word: that word keeps the tokens that fit and
|
|
the rest of it never reaches the tokenizer.
|
|
"""
|
|
|
|
def __init__(self, tokens_per_word: int = 1, max_length: int = 512):
|
|
self.tokens_per_word = tokens_per_word
|
|
self.max_length = max_length
|
|
|
|
def __call__(self, chunk_words, **kw):
|
|
batch_words = (
|
|
chunk_words if (chunk_words and isinstance(chunk_words[0], list)) else [chunk_words]
|
|
)
|
|
word_ids = []
|
|
for words in batch_words:
|
|
ids: list[int] = []
|
|
used = 0
|
|
for idx in range(len(words)):
|
|
room = self.max_length - used
|
|
if room <= 0:
|
|
break
|
|
take = min(self.tokens_per_word, room)
|
|
ids.extend([idx] * take)
|
|
used += take
|
|
word_ids.append(ids)
|
|
return _Enc(
|
|
input_ids=[[0] * len(ids) for ids in word_ids],
|
|
attention_mask=[[1] * len(ids) for ids in word_ids],
|
|
_word_ids=word_ids,
|
|
)
|
|
|
|
|
|
class TestTruncatedWordsCompression:
|
|
"""Words past the tokenizer's 512-token cut are never dropped unseen (#4309).
|
|
|
|
Chunks are cut by words (``chunk_words``) but tokenized with
|
|
``truncation=True, max_length=512``. A chunk of token-heavy words hits
|
|
the token limit well before its word budget, so the tail of the chunk had
|
|
no score and was dropped silently from the default proxy path
|
|
(``target_ratio=None``). The sequential path now scores that tail as the
|
|
next window and the model's decision on it stands; the batch path builds
|
|
its chunks before tokenizing, so it keeps the tail verbatim instead.
|
|
"""
|
|
|
|
# One chunk is `chunk_words` words long, so 350 words is exactly one.
|
|
CHUNK_WORDS = 350
|
|
|
|
def _words(self, count: int) -> list[str]:
|
|
return [f"word{i}" for i in range(count)]
|
|
|
|
def _install(self, monkeypatch, tokens_per_word):
|
|
monkeypatch.setattr(
|
|
kc,
|
|
"_load_kompress",
|
|
lambda *a, **k: (_Model(), _TruncatingTok(tokens_per_word), "onnx"),
|
|
)
|
|
monkeypatch.setattr(kc, "_model_device_type", lambda *a, **k: "cpu")
|
|
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
|
|
|
|
def _compress(self, monkeypatch, tokens_per_word):
|
|
self._install(monkeypatch, tokens_per_word)
|
|
words = self._words(self.CHUNK_WORDS)
|
|
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
|
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
|
|
kept = compressor.compress(" ".join(words)).compressed.split()
|
|
return kept, words
|
|
|
|
def test_words_past_the_truncation_point_are_scored(self, monkeypatch):
|
|
# Two tokens per word: the 350-word chunk needs 700 tokens, so the
|
|
# tokenizer cuts it at 512. Words 0-254 are scored in the first
|
|
# window; word 255 (the cut may have split it) and the 94 the
|
|
# tokenizer never saw start the next window. The fake model keeps the
|
|
# first word of each window, so the second window's decision decides
|
|
# their fate, not a blanket keep.
|
|
kept, words = self._compress(monkeypatch, tokens_per_word=2)
|
|
assert kept == [words[0], words[255]]
|
|
|
|
def test_the_word_the_cut_landed_in_is_scored_whole(self, monkeypatch):
|
|
# Three tokens per word: the cut at 512 falls inside word 170, whose
|
|
# last token never reached the tokenizer. The model only saw part of
|
|
# that word, so it is not judged there: it opens the next window and
|
|
# is scored whole. That window is cut inside word 340 the same way,
|
|
# and the last ten words fit in a third.
|
|
kept, words = self._compress(monkeypatch, tokens_per_word=3)
|
|
assert kept == [words[0], words[170], words[340]]
|
|
|
|
def test_fully_covered_chunk_is_unchanged(self, monkeypatch):
|
|
# One token per word: the chunk fits in 512 tokens, so nothing was
|
|
# truncated and the model's own choice stands. It kept the first word
|
|
# and dropped the rest.
|
|
kept, words = self._compress(monkeypatch, tokens_per_word=1)
|
|
assert kept == [words[0]]
|
|
|
|
def test_batch_path_keeps_truncated_words_too(self, monkeypatch):
|
|
self._install(monkeypatch, 2)
|
|
words = self._words(self.CHUNK_WORDS)
|
|
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
|
monkeypatch.setattr(compressor, "_should_use_sequential_fallback", lambda: False)
|
|
|
|
[result] = compressor.compress_batch([" ".join(words)], batch_size=8)
|
|
|
|
assert result.compressed.split() == [words[0]] + words[255:]
|
|
|
|
def test_a_forced_ratio_still_gets_its_budget(self, monkeypatch):
|
|
# A forced ratio selects scored words within each token window.
|
|
self._install(monkeypatch, 2)
|
|
words = self._words(self.CHUNK_WORDS)
|
|
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
|
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
|
|
|
|
kept = compressor.compress(" ".join(words), target_ratio=0.5).compressed.split()
|
|
|
|
# Main now scores the formerly unseen tail in another window.
|
|
# Windows score 255 and 95 complete words, each applying the
|
|
# requested ratio independently with rounding down (127 + 47).
|
|
assert kept == words[:127] + words[255:302]
|