fix(transforms): stop routing timestamped log rows as grep matches (#3748)

## Description

`_is_search_result_line` now rejects any line that `_TIMESTAMP_ROW_RE`
(from `lossless_compaction.py`, #3419) recognizes as a timestamped log
row, before the grep-shape patterns are tried.

The colon branch accepts `^[^\s:]+:\d+:` whenever the prefix passes
`_prefix_looks_like_path`, and that guard only rules out `<`, `>` and
`=`. A bare ISO date-hour prefix (`2026-09-23T10`) passes, so every row
of a timestamped log counts as a grep match at ratio 1.0 and the payload
routes to `SEARCH_RESULTS`.

Since #3419 the lossless fold skips timestamp rows, so the payload falls
through to the lossy `SearchCompressor`: 2,000 rows collapse to 5, the
note misreads them as "matches in `2026-09-23T10`", and the minute is
printed back as an integer (`10:00:00Z` becomes `10:0:00Z`, a time that
does not exist in the input).

Reusing the lossless module's regex keeps the two guards in lockstep: a
row the fold refuses to treat as grep output can no longer be classified
as grep output here either. The context-line branch
(`_is_grep_context_line`) already kept dates out via its `/`-or-`.`
requirement; this closes the same hole in the colon branch.

Closes #3736

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/transforms/content_detection.py`: `_is_search_result_line`
checks `_TIMESTAMP_ROW_RE.match(line)` (imported from
`lossless_compaction`) before the colon and context branches, so
timestamped log rows are never counted as grep matches.
- `tests/test_transforms/test_content_detector_timestamp_search.py`: 6
new regression tests (2 pinning the bug, 4 controls for real grep
output, date-named files like `logs/2026-09-23.log:42:`, and
extensionless `Makefile:12:`).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Red-green on base 26a2c493 (clean worktree): the two bug-pinning tests fail
(_is_search_result_line returns True; _try_detect_search returns
SEARCH_RESULTS at confidence 1.0); after the fix all 6 pass.

Adjacent suites on this branch:
  tests/test_transforms/                     427 passed, 2 failed
    (test_dense_line_elider, test_smart_crusher_bugs reproduce identically
     on a clean base worktree: pre-existing)
  test_transforms_content_detection.py +
  test_transforms_content_router.py           95 passed
  test_lossless_timestamp_guard.py,
  test_search_compressor*.py,
  test_bash_search_lossless_fold.py           93 passed

ruff check / ruff format --check / mypy: clean on touched files.
```

## Real Behavior Proof

- Environment: WSL2, Python 3.13.15, editable install of this branch
(pure-Python detector; native ONNX path unavailable in this env, same
fallback the 0.38.0 reporter's router label shows).
- Exact command / steps: the 2,000-row ISO-8601 log from the issue,
driven through `ContentRouter.compress` (the same entry
`headroom_compress` uses).
- Observed result:

Base `26a2c493`, before the fix:

```
detect: ContentType.SEARCH_RESULTS 1.0
strategy: CompressionStrategy.SEARCH
2026-09-23T10:0:00Z INFO worker[0] processed batch 0 in 0ms
2026-09-23T10:0:01Z INFO worker[1] processed batch 1 in 13ms
2026-09-23T10:0:02Z INFO worker[2] processed batch 2 in 26ms
2026-09-23T10:0:03Z INFO worker[3] processed batch 3 in 39ms
2026-09-23T10:33:19Z INFO worker[7] processed batch 1999
```

That reproduces both reported defects: 5 of 2,000 rows survive, and the
minutes are mangled to integers (`10:0:00Z`).

This branch, after the fix:

```
detect: ContentType.BUILD_OUTPUT 0.8
strategy: CompressionStrategy.LOG
[2000 lines omitted: 2000 INFO]
[2000 lines compressed to 0. Retrieve more: hash=4803fbc9753df57f2e208836]
```

The payload now routes as a log, as the issue suggested. No fabricated
timestamps appear, and the original stays retrievable through the hash,
same as the lossless path.

- Not tested: the Rust detector in `crates/headroom-core` still lacks
the prefix guard (its `try_detect_search` runs `SEARCH_RESULT_PATTERN`
alone). I left it untouched because open PR #3712 is already porting
`_prefix_looks_like_path` into that file, and adding a parallel change
there would conflict. The Python guard runs on every native fallback
path, and `_detect_content` consults `_try_detect_search` for the
HTML-misroute override as well. The reporter's other open question
(whether `_core` applies the same rule) is answered: it does not, until
#3712-style parity work lands.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this is a pure-Python
content-detection guard.
- Minimum rollout channel: N/A (no feature flag involved).
- Stable/default behavior changed: only for payloads whose lines match
`_TIMESTAMP_ROW_RE`; those stop being classified as `SEARCH_RESULTS` and
route through the log/build strategies instead. Real grep output is
unaffected (controls above).
- Kill switch / disable path: revert the single guard call in
`_is_search_result_line`.
- Unsafe override required: no.
- Qualification impact: none; no qualification suites touched.
- Rollback path: single-commit revert, no state or schema changes.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The Rust-side parity gap is intentionally out of scope to avoid
conflicting with #3712.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Xmon Dai
2026-09-24 10:48:05 -05:00
committed by GitHub
co-authored by JD Davis
parent c02929846c
commit d971f7c398
2 changed files with 91 additions and 0 deletions
+11
View File
@@ -21,6 +21,8 @@ import re
from dataclasses import dataclass
from enum import Enum
from .lossless_compaction import _TIMESTAMP_ROW_RE
class ContentType(Enum):
"""Types of content that can be compressed."""
@@ -509,11 +511,20 @@ def _is_search_result_line(line: str) -> bool:
a file path: no angle brackets and no ``=`` (rules out markup tags and
``key=value:12:`` log lines).
A timestamped log row is never grep output: since #3419 the lossless
fold skips those rows, so a payload classified as search here falls
through to the lossy SearchCompressor, which keeps 5 rows per "file"
and prints the minute back as an integer (#3736). Reject any line the
lossless fold already knows is a timestamp row, reusing the same regex
so the two guards cannot drift apart.
``grep -A``/``-B``/``-C`` context lines — both the real GNU shape
(``path-NN-content``) and the reported ``path:NN-content`` shape — are
accepted via the context predicates so code in them routes to the search
compressor instead of the prose path (#3580).
"""
if _TIMESTAMP_ROW_RE.match(line):
return False
if _SEARCH_RESULT_PATTERN.match(line) or _GREP_COLON_DASH_PATTERN.match(line):
return _prefix_looks_like_path(line.split(":", 1)[0])
return _is_grep_context_line(line)
@@ -0,0 +1,80 @@
"""Regression for #3736: ISO-8601 timestamped logs must not route as search.
The colon branch of ``_is_search_result_line`` accepts ``^[^\\s:]+:\\d+:``
whenever the prefix passes ``_prefix_looks_like_path``. A bare ISO date-hour
prefix (``2026-09-23T10``) passes that guard, so every row of a timestamped
log counts as a grep match (ratio 1.0). Since #3419 the lossless fold skips
timestamp rows, so the payload falls through to the lossy SearchCompressor,
which keeps 5 of 2,000 lines and prints the minute back as an integer
(``10:0:00Z`` — a time that does not exist in the input).
"""
from __future__ import annotations
from headroom.transforms.content_detector import (
ContentType,
_is_search_result_line,
_try_detect_search,
detect_content_type,
)
def _timestamped_log(n: int = 200) -> str:
return "\n".join(
f"2026-09-23T10:{i // 60:02d}:{i % 60:02d}Z INFO worker[{i % 8}] "
f"processed batch {i} in {(i * 13) % 900}ms"
for i in range(n)
)
def test_timestamp_row_is_not_a_search_result_line() -> None:
line = "2026-09-23T10:00:00Z INFO worker[0] processed batch 0 in 0ms"
assert not _is_search_result_line(line)
def test_timestamped_log_does_not_route_to_search() -> None:
payload = _timestamped_log()
assert _try_detect_search(payload) is None
assert detect_content_type(payload).content_type is not ContentType.SEARCH_RESULTS
def test_space_separated_timestamp_log_does_not_route_to_search() -> None:
payload = "\n".join(
f"2026-09-23 10:{i // 60:02d}:{i % 60:02d} [INFO] worker {i} processed batch {i}"
for i in range(50)
)
assert _try_detect_search(payload) is None
assert detect_content_type(payload).content_type is not ContentType.SEARCH_RESULTS
def test_genuine_grep_output_still_routes_to_search() -> None:
payload = "\n".join(
[
"src/app.py:10:def main():",
"src/app.py:20:print('hello')",
"README.md:5:usage docs",
]
)
result = _try_detect_search(payload)
assert result is not None
assert result.content_type is ContentType.SEARCH_RESULTS
def test_grep_on_timestamped_file_names_still_routes_to_search() -> None:
"""A path prefix that embeds a date is still a path, not a timestamp row."""
payload = "\n".join(
[
"logs/2026-09-23.log:42:ERROR something failed",
"logs/2026-09-23.log:43:INFO recovered",
"app/2026-09-24.log:7:WARN retry",
]
)
assert _is_search_result_line("logs/2026-09-23.log:42:ERROR something failed")
result = _try_detect_search(payload)
assert result is not None
assert result.content_type is ContentType.SEARCH_RESULTS
def test_extensionless_paths_still_match() -> None:
assert _is_search_result_line("Makefile:12:all:")
assert _is_search_result_line("Dockerfile:3:FROM scratch")