mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(kompress): keep the boolean connectives that hold a condition together (#3687)
## Description
`_KOMPRESS_MUST_KEEP_RE` already pins the tokens whose loss *inverts* a
statement -- `not`, `never`, `unless`, `must` -- with the rationale
written out
above the class: a directive without its negation is not a degraded
instruction, it is the opposite instruction. The connectives that decide
*which*
predicates have to hold were never added, so Kompress can delete `or`
from a
boolean expression and hand back text that still reads as an intact line
of
code.
@CaptainMittens reported exactly that on
[#3545](https://github.com/headroomlabs-ai/headroom/issues/3545#issuecomment-5630849659)
with a table: from a repeated Python `return` line 12 of 40 `or` were
lost,
while the same word survived 48/48 in a prose control. I reproduced it
on
Windows with the released wheel and the real ONNX model -- see **Real
Behavior
Proof** -- including the same 28/40 figure on the bare-code row, and the
cause
is the one the table points at: `not` from that very line is kept by the
directive-word class while `or` is not covered by any class.
Uppercase `AND`/`OR` (SQL, shell) were already held by the ALLCAPS
class; only
the lowercase forms leaked, which is why this reads as "code loses its
operators, prose does not".
Fix is one added class, `(?i:\b(?:and|or|nor|xor)\b)`, following the
precedent
of the directive-word class: pin the closed-class tokens whose removal
changes
meaning, and pay for it in ratio only where the model was already
keeping them
anyway.
Related: #3545 (the defect is the one reported in that thread's
comments; it has no issue of its own, so this PR does not close it.)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/kompress_compressor.py`: append a
boolean-connective
class to `_KOMPRESS_MUST_KEEP_RE`. No call-site change -- both the
sequential
and the batch path already run the override, so they are covered by the
same
table and the same `HEADROOM_KOMPRESS_MUST_KEEP=0` toggle.
- `tests/test_kompress_must_keep.py`: extend the existing module rather
than
add a file --
- regex assertions for `or`/`and`/`nor`/`xor` (bare and punctuated:
`or,`,
`(and)`), and negative assertions that tokens which merely *contain* a
connective (`orange`, `random`, `android`, `core`, `sort`, `concat`,
`node`) stay unmatched, so the class cannot silently widen;
- one test through `compress()` and one through `compress_batch()`,
using the
module's existing fake model that keeps only the first word, so the
assertion is about the override and not about a model download;
- one test that `HEADROOM_KOMPRESS_MUST_KEEP=0` still means "the model
decides" for these tokens too.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) -- see Test Output
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
The regression tests are offline and deterministic: they use the fake
model
already in this test module, which keeps only the first word of every
chunk, so
what they assert is the override table and not a model download.
Before the fix (red):
```text
$ python -m pytest tests/test_kompress_must_keep.py -q
tests\test_kompress_must_keep.py ........FFF....... [100%]
================================== FAILURES ===================================
_________________ TestMustKeepRegex.test_boolean_connectives __________________
tests\test_kompress_must_keep.py:85: in test_boolean_connectives
assert _KOMPRESS_MUST_KEEP_RE.search("or")
E assert None
_ TestBooleanConnectivesCompression.test_keeps_both_operands_of_a_negated_disjunction _
tests\test_kompress_must_keep.py:119: in test_keeps_both_operands_of_a_negated_disjunction
assert "or" in kept
E assert 'or' in ['return', '"__init__"', 'not', 'name.startswith("_")']
___ TestBooleanConnectivesCompression.test_batch_path_keeps_connectives_too ___
tests\test_kompress_must_keep.py:133: in test_batch_path_keeps_connectives_too
assert "and" in kept
E AssertionError: assert 'and' in ['alpha']
=========================== short test summary info ===========================
FAILED tests/test_kompress_must_keep.py::TestMustKeepRegex::test_boolean_connectives
FAILED tests/test_kompress_must_keep.py::TestBooleanConnectivesCompression::test_keeps_both_operands_of_a_negated_disjunction
FAILED tests/test_kompress_must_keep.py::TestBooleanConnectivesCompression::test_batch_path_keeps_connectives_too
======================== 3 failed, 15 passed in 44.16s ========================
```
That middle failure is the defect in one line: the fake model drops
everything
but the first word, the override puts back what it recognises, and `not`
is on
that list while `or` is not.
After the fix (green), same command:
```text
tests\test_kompress_must_keep.py .................. [100%]
============================= 18 passed in 15.50s =============================
```
The table itself, read straight off `main` (67eb910e) before the change:
```text
$ python -c "from headroom.transforms.kompress_compressor import _KOMPRESS_MUST_KEEP_RE as R; \
print('or', bool(R.search('or')), '| and', bool(R.search('and')), '| not', bool(R.search('not')))"
or False | and False | not True
```
Wider subset around the change:
```text
$ python -m pytest tests/test_kompress_must_keep.py \
tests/test_kompress_failsafe.py tests/test_kompress_marker_gate.py \
tests/test_force_kompress_all.py tests/test_transforms_content_router.py \
tests/test_content_router_detection_dedup.py \
tests/test_content_router_token_units.py \
tests/test_transforms_search_compressor.py \
tests/test_transforms_content_detection.py -q
============ 1 failed, 154 passed, 7 skipped in 159.74s (0:02:39) ============
$ python -m pytest tests/test_content_router_detection_dedup.py -q
============================= 9 passed in 50.28s ==============================
```
The one failure in the first run is `RuntimeError: tiktoken is required
for
OpenAI provider` -- a missing optional dependency in my venv, not this
change.
After `pip install tiktoken` the same file passes 9/9, and no other test
in the
subset moved.
```text
$ uvx ruff@0.16.7 check headroom/transforms/kompress_compressor.py tests/test_kompress_must_keep.py
All checks passed!
$ uvx ruff@0.16.7 format --check headroom/transforms/kompress_compressor.py tests/test_kompress_must_keep.py
2 files already formatted
$ uvx mypy@1.14.1 --ignore-missing-imports headroom/transforms/kompress_compressor.py
(no output, exit 0)
```
`mypy` was run over the changed module only, not `mypy headroom` -- the
repository's full type-check target needs the compiled `_core` extension
plus
the ML extras, which I did not build here.
## Real Behavior Proof
- Environment: Windows 11 (10.0.26200) x64, Python 3.12.13, `headroom-ai
0.37.0` installed from PyPI as `headroom-ai[proxy]`. Kompress backend
`onnx`
(`chopratejas/kompress-v2-base`, `onnx/kompress-int8-wo.onnx`, CPU,
model
fetched once into the HF cache). No LLM provider and no API key is
involved:
compression runs entirely locally, which is what this proof is about.
The
patched arm swaps only the regex table into the same installed module,
so
both arms share the model, tokenizer, session and cache state.
- Exact command / steps: two scripts run under the released 0.37.0
wheel, then a shipped -> patched -> shipped A/B in one process; both are
quoted in full below.
1. Reproduce the corruption on the shipped code:
```python
# repro.py -- run under the released wheel, no repo checkout involved
from headroom.transforms.kompress_compressor import KompressCompressor
REAL = 'return name == "__init__" or not name.startswith("_")'
code = "\n".join(f" {REAL}" for _ in range(40))
prose = " ".join(
f"Note {i:02d}. request fails when token is missing or signature does
not verify."
for i in range(1, 49)
)
comp = KompressCompressor()
for name, payload in (("bare code", code), ("prose control", prose)):
out = comp.compress(payload).compressed
print(name, payload.count(" or "), "->", out.count(" or "))
```
2. Measure the fix's cost, shipped vs patched, on the same process and
model
(the `and`/`or`/`nor`/`xor` class compiled onto the shipped pattern),
over
four payloads: the two rows from the report, a prose control, 110 real
boolean lines from `headroom/transforms/*.py`, and a 624-word prose
corpus
built from the same sentences. `_time_budget_seconds` is stubbed to
`None`
so the 20 s wall-clock budget cannot pass a tail through verbatim, and
each
payload is measured shipped -> patched -> shipped to prove the numbers
are
deterministic rather than a timing artifact.
- Observed result: on shipped code the reporter's bare-code payload
keeps 28 of 40 `or`; with this class compiled in it keeps 40 of 40,
costing 0.00 pp of ratio on prose and 4.29-5.11 pp on connective-dense
code.
Reproduction on the shipped code, before any patch, with the reporter's
payloads (`KompressCompressor()` defaults, i.e. the same CCR mode the
proxy
uses):
```text
python 3.12.13
headroom-ai 0.37.0
module ...\site-packages\headroom/transforms\kompress_compressor.py
grep-shaped code ' or ' 40 -> 40 kept 40/40 chars 0.962
bare code ' or ' 40 -> 28 kept 28/40 chars 0.866
prose control ' or ' 48 -> 48 kept 48/48 chars 0.908
```
The `bare code` row is the report's second table row reproduced
numerically
(28 of 40 kept), on a different OS and a different Python from the
reporter's
macOS run. `grep-shaped` differs from their run because that payload
reaches
a different compressor here; it is excluded below rather than reported
as a
match.
Cost of the fix, same process, same model, same tokenizer, only the
regex
table swapped, each payload measured shipped -> patched -> shipped so a
row
can only be trusted if the two shipped measurements agree:
```text
bare code (issue row 2) words 280 connectives 40 | shipped 28 (70.0%,
ratio 0.796) | patched 40 (100.0%, ratio 0.839) | cost 4.29 pp |
deterministic True
prose control (issue row 3) words 312 connectives 24 | shipped 24
(100.0%, ratio 0.862) | patched 24 (100.0%, ratio 0.862) | cost 0.00 pp
| deterministic True
mixed python boolean lines words 1056 connectives 198 | shipped 144
(72.7%, ratio 0.802) | patched 198 (100.0%, ratio 0.853) | cost 5.11 pp
| deterministic True
prose with and/or words 312 connectives 24 | shipped 24 (100.0%, ratio
0.814) | patched 24 (100.0%, ratio 0.814) | cost 0.00 pp | deterministic
True
```
`mixed python boolean lines` is 55 short Python statement shapes that
each
carry two or more connectives (an `isinstance(...) and
...startswith(...)`
guard, an `any(...) and ... or default` return, a `while ... and (not
... or
...)` loop, an `assert ... in {...} or ...`, and `(a and b) or (c and
not
d)`), repeated 11 times to 1056 words. `prose with and/or` is the same
connectives in 312 words of ordinary prose, to price the change where
over-protection is cheapest. Both are synthetic on purpose: they isolate
the
variable, which is the operator, from the content around it.
The shape is the one worth the 4-5 pp: on code the model drops roughly a
quarter of the connectives that decide which predicate has to hold, on
prose
it was already keeping every one of them, so the same table costs 0.00
pp
there. This mirrors how the directive-word class above it was priced.
- Not tested: the `pytorch`/CUDA/MPS backends and the remote
Kompress endpoint; the proxy end-to-end (no provider credentials here,
and
the transform is reached identically from `ContentRouter`); whether this
changes any published benchmark or evaluation number; whether it fixes
the
*line-fusing* symptom in the original #3545 report -- I could not
reproduce
that one either (see @TigerkidYang's comment on that issue), and this PR
does
not claim to.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. The table is a module-level constant
compiled at import; no flag service or staged rollout is involved.
- Minimum rollout channel: n/a -- ordinary release.
- Stable/default behavior changed: yes, marginally and by design.
Requests
whose content contains `and`/`or`/`nor`/`xor` keep those tokens.
Measured
cost: 0.00 pp of ratio on prose, 4.29-5.11 pp on connective-dense code.
- Kill switch / disable path: the existing
`HEADROOM_KOMPRESS_MUST_KEEP=0`
removes the whole override, including this class, and returns to
model-only decisions; `--lossless` / `lossless_only` bypasses Kompress
altogether. Verified by the added `test_disabling...` test.
- Unsafe override required: no.
- Qualification impact: compressed bytes change for affected content, so
a
deployment comparing against captured prompt bytes (or relying on a
prompt
cache keyed on them) sees one diff per distinct input after upgrading --
the
same property as when the directive-word class was added.
- Rollback path: revert the single commit; no migration, no persisted
format change, no store rewrite.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` -- it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
n/a -- text output in **Test Output** and **Real Behavior Proof**.
## Additional Notes
- **Not claimed**: `in`, `is`, `if`, `else` are unprotected too, and
they are
the ones with a real ratio question (they are common English words, not
just
operators). I left them out rather than guess at their cost; measuring
them
is a one-line change to the same table.
- **Overlap**: #3119 edits the same file and reuses the
`HEADROOM_KOMPRESS_MUST_KEEP` toggle for line integrity (#3099), but
does
not touch `_KOMPRESS_MUST_KEEP_RE`, so the two changes are independent
--
whichever lands second will need a trivial rebase.
- #3544 (markers spliced into small JSON fields) is a different decision
about
where markers may go, and is left alone here.
Co-authored-by: sxh313 <250161920+sxh313@users.noreply.github.com>
This commit is contained in:
@@ -68,6 +68,16 @@ _KOMPRESS_MUST_KEEP_RE = re.compile(
|
||||
r"|(?i:\b(?:not|never|none|cannot|can't|don't|doesn't|didn't|won't|shouldn't"
|
||||
r"|mustn't|isn't|aren't|avoid|refuse|prohibited|forbidden|disallow|unless"
|
||||
r"|except|without|must|should|shall|required|always|only|mandatory)\b)"
|
||||
# Boolean connectives, for the same reason as the directive words above:
|
||||
# they decide WHICH predicates have to hold, so dropping one does not
|
||||
# weaken the statement, it changes the condition. Negation was pinned but
|
||||
# conjunction was not, so one line could lose its `or` and keep its `not`
|
||||
# (issue #3545: 12 of 40 `or` lost from a repeated Python return line,
|
||||
# while the same word survived 48/48 in prose). The surviving text is the
|
||||
# dangerous part -- `a == b not c.startswith(d)` still reads as code, and
|
||||
# nothing marks it as altered. Uppercase `AND`/`OR` were already held by
|
||||
# the ALLCAPS class; only the lowercase forms leaked.
|
||||
r"|(?i:\b(?:and|or|nor|xor)\b)"
|
||||
)
|
||||
_KOMPRESS_MUST_KEEP_ENV = "HEADROOM_KOMPRESS_MUST_KEEP"
|
||||
KOMPRESS_BACKEND_ENV = "HEADROOM_KOMPRESS_BACKEND"
|
||||
|
||||
@@ -81,6 +81,63 @@ class TestMustKeepRegex:
|
||||
assert not _KOMPRESS_MUST_KEEP_RE.search("process")
|
||||
assert not _KOMPRESS_MUST_KEEP_RE.search("raised")
|
||||
|
||||
def test_boolean_connectives(self):
|
||||
assert _KOMPRESS_MUST_KEEP_RE.search("or")
|
||||
assert _KOMPRESS_MUST_KEEP_RE.search("and")
|
||||
assert _KOMPRESS_MUST_KEEP_RE.search("nor")
|
||||
assert _KOMPRESS_MUST_KEEP_RE.search("xor")
|
||||
assert _KOMPRESS_MUST_KEEP_RE.search("or,")
|
||||
assert _KOMPRESS_MUST_KEEP_RE.search("(and)")
|
||||
# Tokens that merely contain a connective are not connectives.
|
||||
for word in ("orange", "random", "android", "core", "sort", "concat", "node"):
|
||||
assert not _KOMPRESS_MUST_KEEP_RE.search(word), word
|
||||
|
||||
|
||||
class TestBooleanConnectivesCompression:
|
||||
"""`and`/`or` hold a condition together, so dropping one changes the
|
||||
predicate rather than degrading the sentence. Issue #3545 measured 12 of 40
|
||||
`or` lost from a repeated Python return line on the released compressor,
|
||||
while `not` from the very same line survived because negation is already
|
||||
pinned -- the result still reads as a line of Python and nothing marks it
|
||||
as altered."""
|
||||
|
||||
CODE = 'return name == "__init__" or not name.startswith("_") alpha beta gamma delta'
|
||||
|
||||
def _compress(self, monkeypatch, env):
|
||||
_install_fake_kompress(monkeypatch)
|
||||
if env:
|
||||
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
|
||||
else:
|
||||
monkeypatch.setenv(_KOMPRESS_MUST_KEEP_ENV, "0")
|
||||
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
||||
monkeypatch.setattr(compressor, "_should_batch_single_content", lambda *a, **k: False)
|
||||
return compressor.compress(self.CODE)
|
||||
|
||||
def test_keeps_both_operands_of_a_negated_disjunction(self, monkeypatch):
|
||||
kept = self._compress(monkeypatch, env=True).compressed.split()
|
||||
assert "not" in kept
|
||||
assert "or" in kept
|
||||
|
||||
def test_batch_path_keeps_connectives_too(self, monkeypatch):
|
||||
_install_fake_kompress(monkeypatch)
|
||||
monkeypatch.delenv(_KOMPRESS_MUST_KEEP_ENV, raising=False)
|
||||
compressor = KompressCompressor(KompressConfig(enable_ccr=False, min_input_words=10))
|
||||
monkeypatch.setattr(compressor, "_should_use_sequential_fallback", lambda: False)
|
||||
|
||||
[result] = compressor.compress_batch(
|
||||
["alpha beta gamma delta epsilon zeta eta theta iota kappa one and two or three"],
|
||||
batch_size=8,
|
||||
)
|
||||
|
||||
kept = result.compressed.split()
|
||||
assert "and" in kept
|
||||
assert "or" in kept
|
||||
|
||||
def test_disabling_must_keep_leaves_the_connective_to_the_model(self, monkeypatch):
|
||||
kept = self._compress(monkeypatch, env=False).compressed.split()
|
||||
assert "or" not in kept
|
||||
assert "not" not in kept
|
||||
|
||||
|
||||
class TestMustKeepEnvVar:
|
||||
def test_env_var_name(self):
|
||||
|
||||
Reference in New Issue
Block a user