fix(parser): whitespace waste signal always reported zero (#1102)

## Description

`detect_waste_signals` in `headroom/parser.py` computed the
`whitespace_tokens` waste signal as `0` for every input, so collapsible
whitespace never contributed to `WasteSignals.total()`.

The root cause was this normalization step:

```python
normalized_text = " ".join(ws_matches)
```

That keeps each matched whitespace run verbatim and only inserts one
extra space between runs. The normalized text is therefore never shorter
than the original matched whitespace text. This PR collapses each
matched run to one space instead:

```python
normalized_text = " "  len(ws_matches)
```

## Type of Change

- [x] Bug fix
- [x] Test coverage
- [x] Changelog entry

## Changes Made

- Fixed the whitespace waste calculation in `headroom/parser.py`.
- Tightened the existing excessive-whitespace test so it asserts real
savings instead of allowing `0`.
- Added a regression test proving the signal scales with long whitespace
runs.

## Testing

### Test Output

```text
88 parser tests passed
ruff check: passed
ruff format --check: passed
```

- [x] `uv run pytest tests/test_parser.py` -> `88 passed`
- [x] `ruff check`
- [x] `ruff format --check`

## Real Behavior Proof

- Environment: current PR head, pure-Python parser path.
- Exact command / steps: ran `uv run pytest tests/test_parser.py`, Ruff,
and formatting checks with long whitespace-run coverage.
- Observed result: 88 parser tests pass; fixture waste is 2 tokens and a
realistic padded log reports 65 rather than zero.
- Not tested: compiled Rust, hnswlib, torch, or live proxy paths because
the changed path is pure Python.

Before this fix, the existing whitespace fixture and a padded-log case
both returned `signals.whitespace_tokens == 0`, even though the matched
whitespace was collapsible.

After this fix:

```text
fixture whitespace_tokens: 2
realistic padded log whitespace_tokens: 65
```

The new tests fail against `main` and pass with this branch.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Scope Notes

I did not retest compiled Rust, `hnswlib`, torch, or live proxy paths on
this Windows machine. The changed path is pure Python and covered by
`tests/test_parser.py`.

Co-authored-by: nyxst4ck <289980115+nyxst4ck@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
nyxst4ck
2026-10-01 22:20:44 -05:00
committed by GitHub
co-authored by nyxst4ck JerrettDavis
parent ecc496709b
commit f19bc9a9c3
2 changed files with 24 additions and 3 deletions
+3 -1
View File
@@ -180,7 +180,9 @@ def detect_waste_signals(text: str, tokenizer: Tokenizer) -> WasteSignals:
if ws_matches:
# Count tokens that could be saved by normalizing whitespace to single spaces
ws_text = "".join(ws_matches)
normalized_text = " ".join(ws_matches)
# Each matched run (4+ spaces/tabs or 3+ newlines) collapses to a
# single space, so the normalized form is one space per run.
normalized_text = " " * len(ws_matches)
signals.whitespace_tokens = max(
0, tokenizer.count_text(ws_text) - tokenizer.count_text(normalized_text)
)
+21 -2
View File
@@ -217,9 +217,28 @@ class TestDetectWasteSignals:
assert signals.base64_tokens > 0
def test_detect_excessive_whitespace(self, mock_tokenizer, whitespace_waste_text):
"""Detects excessive whitespace as waste."""
"""Detects excessive whitespace as waste.
The fixture has a 4-newline run and a 6-space run. Each run collapses
to a single space when normalized, so there are real savings to report:
ws_text "\\n\\n\\n\\n " (10 chars -> 3 tokens) vs normalized " "
(2 chars -> 1 token) = 2 tokens saved with the mock tokenizer.
"""
signals = detect_waste_signals(whitespace_waste_text, mock_tokenizer)
assert signals.whitespace_tokens >= 0 # May be 0 if normalized tokens <= matches
assert signals.whitespace_tokens == 2
def test_whitespace_savings_scale_with_run_length(self, mock_tokenizer):
"""Long whitespace runs report proportionally large savings.
Regression guard: normalizing must collapse each matched run to a
single space, not just join the runs together (which left the runs
intact and made the signal always report 0).
"""
text = "ERROR" + " " * 200 + "stack" + "\n" * 60 + "end"
signals = detect_waste_signals(text, mock_tokenizer)
# 260 whitespace chars collapse to 2 spaces; with ~1 token/4 chars the
# savings are large and clearly non-zero.
assert signals.whitespace_tokens > 50
def test_detect_json_bloat(self, mock_tokenizer, json_bloat_text):
"""Detects large JSON blocks as bloat."""