mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(transforms): stop routing timestamped log rows as grep matches (#3748)
## Description
`_is_search_result_line` now rejects any line that `_TIMESTAMP_ROW_RE`
(from `lossless_compaction.py`, #3419) recognizes as a timestamped log
row, before the grep-shape patterns are tried.
The colon branch accepts `^[^\s:]+:\d+:` whenever the prefix passes
`_prefix_looks_like_path`, and that guard only rules out `<`, `>` and
`=`. A bare ISO date-hour prefix (`2026-09-23T10`) passes, so every row
of a timestamped log counts as a grep match at ratio 1.0 and the payload
routes to `SEARCH_RESULTS`.
Since #3419 the lossless fold skips timestamp rows, so the payload falls
through to the lossy `SearchCompressor`: 2,000 rows collapse to 5, the
note misreads them as "matches in `2026-09-23T10`", and the minute is
printed back as an integer (`10:00:00Z` becomes `10:0:00Z`, a time that
does not exist in the input).
Reusing the lossless module's regex keeps the two guards in lockstep: a
row the fold refuses to treat as grep output can no longer be classified
as grep output here either. The context-line branch
(`_is_grep_context_line`) already kept dates out via its `/`-or-`.`
requirement; this closes the same hole in the colon branch.
Closes #3736
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/content_detection.py`: `_is_search_result_line`
checks `_TIMESTAMP_ROW_RE.match(line)` (imported from
`lossless_compaction`) before the colon and context branches, so
timestamped log rows are never counted as grep matches.
- `tests/test_transforms/test_content_detector_timestamp_search.py`: 6
new regression tests (2 pinning the bug, 4 controls for real grep
output, date-named files like `logs/2026-09-23.log:42:`, and
extensionless `Makefile:12:`).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
Red-green on base 26a2c493 (clean worktree): the two bug-pinning tests fail
(_is_search_result_line returns True; _try_detect_search returns
SEARCH_RESULTS at confidence 1.0); after the fix all 6 pass.
Adjacent suites on this branch:
tests/test_transforms/ 427 passed, 2 failed
(test_dense_line_elider, test_smart_crusher_bugs reproduce identically
on a clean base worktree: pre-existing)
test_transforms_content_detection.py +
test_transforms_content_router.py 95 passed
test_lossless_timestamp_guard.py,
test_search_compressor*.py,
test_bash_search_lossless_fold.py 93 passed
ruff check / ruff format --check / mypy: clean on touched files.
```
## Real Behavior Proof
- Environment: WSL2, Python 3.13.15, editable install of this branch
(pure-Python detector; native ONNX path unavailable in this env, same
fallback the 0.38.0 reporter's router label shows).
- Exact command / steps: the 2,000-row ISO-8601 log from the issue,
driven through `ContentRouter.compress` (the same entry
`headroom_compress` uses).
- Observed result:
Base `26a2c493`, before the fix:
```
detect: ContentType.SEARCH_RESULTS 1.0
strategy: CompressionStrategy.SEARCH
2026-09-23T10:0:00Z INFO worker[0] processed batch 0 in 0ms
2026-09-23T10:0:01Z INFO worker[1] processed batch 1 in 13ms
2026-09-23T10:0:02Z INFO worker[2] processed batch 2 in 26ms
2026-09-23T10:0:03Z INFO worker[3] processed batch 3 in 39ms
2026-09-23T10:33:19Z INFO worker[7] processed batch 1999
```
That reproduces both reported defects: 5 of 2,000 rows survive, and the
minutes are mangled to integers (`10:0:00Z`).
This branch, after the fix:
```
detect: ContentType.BUILD_OUTPUT 0.8
strategy: CompressionStrategy.LOG
[2000 lines omitted: 2000 INFO]
[2000 lines compressed to 0. Retrieve more: hash=4803fbc9753df57f2e208836]
```
The payload now routes as a log, as the issue suggested. No fabricated
timestamps appear, and the original stays retrievable through the hash,
same as the lossless path.
- Not tested: the Rust detector in `crates/headroom-core` still lacks
the prefix guard (its `try_detect_search` runs `SEARCH_RESULT_PATTERN`
alone). I left it untouched because open PR #3712 is already porting
`_prefix_looks_like_path` into that file, and adding a parallel change
there would conflict. The Python guard runs on every native fallback
path, and `_detect_content` consults `_try_detect_search` for the
HTML-misroute override as well. The reporter's other open question
(whether `_core` applies the same rule) is answered: it does not, until
#3712-style parity work lands.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; this is a pure-Python
content-detection guard.
- Minimum rollout channel: N/A (no feature flag involved).
- Stable/default behavior changed: only for payloads whose lines match
`_TIMESTAMP_ROW_RE`; those stop being classified as `SEARCH_RESULTS` and
route through the log/build strategies instead. Real grep output is
unaffected (controls above).
- Kill switch / disable path: revert the single guard call in
`_is_search_result_line`.
- Unsafe override required: no.
- Qualification impact: none; no qualification suites touched.
- Rollback path: single-commit revert, no state or schema changes.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
The Rust-side parity gap is intentionally out of scope to avoid
conflicting with #3712.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -21,6 +21,8 @@ import re
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
|
||||
from .lossless_compaction import _TIMESTAMP_ROW_RE
|
||||
|
||||
|
||||
class ContentType(Enum):
|
||||
"""Types of content that can be compressed."""
|
||||
@@ -509,11 +511,20 @@ def _is_search_result_line(line: str) -> bool:
|
||||
a file path: no angle brackets and no ``=`` (rules out markup tags and
|
||||
``key=value:12:`` log lines).
|
||||
|
||||
A timestamped log row is never grep output: since #3419 the lossless
|
||||
fold skips those rows, so a payload classified as search here falls
|
||||
through to the lossy SearchCompressor, which keeps 5 rows per "file"
|
||||
and prints the minute back as an integer (#3736). Reject any line the
|
||||
lossless fold already knows is a timestamp row, reusing the same regex
|
||||
so the two guards cannot drift apart.
|
||||
|
||||
``grep -A``/``-B``/``-C`` context lines — both the real GNU shape
|
||||
(``path-NN-content``) and the reported ``path:NN-content`` shape — are
|
||||
accepted via the context predicates so code in them routes to the search
|
||||
compressor instead of the prose path (#3580).
|
||||
"""
|
||||
if _TIMESTAMP_ROW_RE.match(line):
|
||||
return False
|
||||
if _SEARCH_RESULT_PATTERN.match(line) or _GREP_COLON_DASH_PATTERN.match(line):
|
||||
return _prefix_looks_like_path(line.split(":", 1)[0])
|
||||
return _is_grep_context_line(line)
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
"""Regression for #3736: ISO-8601 timestamped logs must not route as search.
|
||||
|
||||
The colon branch of ``_is_search_result_line`` accepts ``^[^\\s:]+:\\d+:``
|
||||
whenever the prefix passes ``_prefix_looks_like_path``. A bare ISO date-hour
|
||||
prefix (``2026-09-23T10``) passes that guard, so every row of a timestamped
|
||||
log counts as a grep match (ratio 1.0). Since #3419 the lossless fold skips
|
||||
timestamp rows, so the payload falls through to the lossy SearchCompressor,
|
||||
which keeps 5 of 2,000 lines and prints the minute back as an integer
|
||||
(``10:0:00Z`` — a time that does not exist in the input).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from headroom.transforms.content_detector import (
|
||||
ContentType,
|
||||
_is_search_result_line,
|
||||
_try_detect_search,
|
||||
detect_content_type,
|
||||
)
|
||||
|
||||
|
||||
def _timestamped_log(n: int = 200) -> str:
|
||||
return "\n".join(
|
||||
f"2026-09-23T10:{i // 60:02d}:{i % 60:02d}Z INFO worker[{i % 8}] "
|
||||
f"processed batch {i} in {(i * 13) % 900}ms"
|
||||
for i in range(n)
|
||||
)
|
||||
|
||||
|
||||
def test_timestamp_row_is_not_a_search_result_line() -> None:
|
||||
line = "2026-09-23T10:00:00Z INFO worker[0] processed batch 0 in 0ms"
|
||||
assert not _is_search_result_line(line)
|
||||
|
||||
|
||||
def test_timestamped_log_does_not_route_to_search() -> None:
|
||||
payload = _timestamped_log()
|
||||
assert _try_detect_search(payload) is None
|
||||
assert detect_content_type(payload).content_type is not ContentType.SEARCH_RESULTS
|
||||
|
||||
|
||||
def test_space_separated_timestamp_log_does_not_route_to_search() -> None:
|
||||
payload = "\n".join(
|
||||
f"2026-09-23 10:{i // 60:02d}:{i % 60:02d} [INFO] worker {i} processed batch {i}"
|
||||
for i in range(50)
|
||||
)
|
||||
assert _try_detect_search(payload) is None
|
||||
assert detect_content_type(payload).content_type is not ContentType.SEARCH_RESULTS
|
||||
|
||||
|
||||
def test_genuine_grep_output_still_routes_to_search() -> None:
|
||||
payload = "\n".join(
|
||||
[
|
||||
"src/app.py:10:def main():",
|
||||
"src/app.py:20:print('hello')",
|
||||
"README.md:5:usage docs",
|
||||
]
|
||||
)
|
||||
result = _try_detect_search(payload)
|
||||
assert result is not None
|
||||
assert result.content_type is ContentType.SEARCH_RESULTS
|
||||
|
||||
|
||||
def test_grep_on_timestamped_file_names_still_routes_to_search() -> None:
|
||||
"""A path prefix that embeds a date is still a path, not a timestamp row."""
|
||||
payload = "\n".join(
|
||||
[
|
||||
"logs/2026-09-23.log:42:ERROR something failed",
|
||||
"logs/2026-09-23.log:43:INFO recovered",
|
||||
"app/2026-09-24.log:7:WARN retry",
|
||||
]
|
||||
)
|
||||
assert _is_search_result_line("logs/2026-09-23.log:42:ERROR something failed")
|
||||
result = _try_detect_search(payload)
|
||||
assert result is not None
|
||||
assert result.content_type is ContentType.SEARCH_RESULTS
|
||||
|
||||
|
||||
def test_extensionless_paths_still_match() -> None:
|
||||
assert _is_search_result_line("Makefile:12:all:")
|
||||
assert _is_search_result_line("Dockerfile:3:FROM scratch")
|
||||
Reference in New Issue
Block a user