mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(parser): whitespace waste signal always reported zero (#1102)
## Description `detect_waste_signals` in `headroom/parser.py` computed the `whitespace_tokens` waste signal as `0` for every input, so collapsible whitespace never contributed to `WasteSignals.total()`. The root cause was this normalization step: ```python normalized_text = " ".join(ws_matches) ``` That keeps each matched whitespace run verbatim and only inserts one extra space between runs. The normalized text is therefore never shorter than the original matched whitespace text. This PR collapses each matched run to one space instead: ```python normalized_text = " " len(ws_matches) ``` ## Type of Change - [x] Bug fix - [x] Test coverage - [x] Changelog entry ## Changes Made - Fixed the whitespace waste calculation in `headroom/parser.py`. - Tightened the existing excessive-whitespace test so it asserts real savings instead of allowing `0`. - Added a regression test proving the signal scales with long whitespace runs. ## Testing ### Test Output ```text 88 parser tests passed ruff check: passed ruff format --check: passed ``` - [x] `uv run pytest tests/test_parser.py` -> `88 passed` - [x] `ruff check` - [x] `ruff format --check` ## Real Behavior Proof - Environment: current PR head, pure-Python parser path. - Exact command / steps: ran `uv run pytest tests/test_parser.py`, Ruff, and formatting checks with long whitespace-run coverage. - Observed result: 88 parser tests pass; fixture waste is 2 tokens and a realistic padded log reports 65 rather than zero. - Not tested: compiled Rust, hnswlib, torch, or live proxy paths because the changed path is pure Python. Before this fix, the existing whitespace fixture and a padded-log case both returned `signals.whitespace_tokens == 0`, even though the matched whitespace was collapsible. After this fix: ```text fixture whitespace_tokens: 2 realistic padded log whitespace_tokens: 65 ``` The new tests fail against `main` and pass with this branch. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Scope Notes I did not retest compiled Rust, `hnswlib`, torch, or live proxy paths on this Windows machine. The changed path is pure Python and covered by `tests/test_parser.py`. Co-authored-by: nyxst4ck <289980115+nyxst4ck@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
nyxst4ck
JerrettDavis
parent
ecc496709b
commit
f19bc9a9c3
+3
-1
@@ -180,7 +180,9 @@ def detect_waste_signals(text: str, tokenizer: Tokenizer) -> WasteSignals:
|
||||
if ws_matches:
|
||||
# Count tokens that could be saved by normalizing whitespace to single spaces
|
||||
ws_text = "".join(ws_matches)
|
||||
normalized_text = " ".join(ws_matches)
|
||||
# Each matched run (4+ spaces/tabs or 3+ newlines) collapses to a
|
||||
# single space, so the normalized form is one space per run.
|
||||
normalized_text = " " * len(ws_matches)
|
||||
signals.whitespace_tokens = max(
|
||||
0, tokenizer.count_text(ws_text) - tokenizer.count_text(normalized_text)
|
||||
)
|
||||
|
||||
+21
-2
@@ -217,9 +217,28 @@ class TestDetectWasteSignals:
|
||||
assert signals.base64_tokens > 0
|
||||
|
||||
def test_detect_excessive_whitespace(self, mock_tokenizer, whitespace_waste_text):
|
||||
"""Detects excessive whitespace as waste."""
|
||||
"""Detects excessive whitespace as waste.
|
||||
|
||||
The fixture has a 4-newline run and a 6-space run. Each run collapses
|
||||
to a single space when normalized, so there are real savings to report:
|
||||
ws_text "\\n\\n\\n\\n " (10 chars -> 3 tokens) vs normalized " "
|
||||
(2 chars -> 1 token) = 2 tokens saved with the mock tokenizer.
|
||||
"""
|
||||
signals = detect_waste_signals(whitespace_waste_text, mock_tokenizer)
|
||||
assert signals.whitespace_tokens >= 0 # May be 0 if normalized tokens <= matches
|
||||
assert signals.whitespace_tokens == 2
|
||||
|
||||
def test_whitespace_savings_scale_with_run_length(self, mock_tokenizer):
|
||||
"""Long whitespace runs report proportionally large savings.
|
||||
|
||||
Regression guard: normalizing must collapse each matched run to a
|
||||
single space, not just join the runs together (which left the runs
|
||||
intact and made the signal always report 0).
|
||||
"""
|
||||
text = "ERROR" + " " * 200 + "stack" + "\n" * 60 + "end"
|
||||
signals = detect_waste_signals(text, mock_tokenizer)
|
||||
# 260 whitespace chars collapse to 2 spaces; with ~1 token/4 chars the
|
||||
# savings are large and clearly non-zero.
|
||||
assert signals.whitespace_tokens > 50
|
||||
|
||||
def test_detect_json_bloat(self, mock_tokenizer, json_bloat_text):
|
||||
"""Detects large JSON blocks as bloat."""
|
||||
|
||||
Reference in New Issue
Block a user