fix(search): stop a context line's body from becoming its line marker (#3788)

## Description

Closes #3545

I could not reproduce the exact `Line 476: … 497: 535: 556: …` output in
the
report, and I think the reason is worth stating up front, because it
changes what
this PR does.

That shape cannot come out of the search compressor. `parse_match_line`
splits
**one input line at a time**, and `format_output` emits **one match per
line**, so
no code path can move line 497's number onto line 476's body. Feeding
the
shipped build a body that genuinely contains those numbers reproduces
the
reported string exactly:

```
input   app/settings.py:476:settings: settings({contact_precedence: 497: 535: 556:
        app/settings.py:497:...
        app/settings.py:535:...
        app/settings.py:556:...
output  app/settings.py:476:settings: settings({contact_precedence: 497: 535: 556:
        [... and 3 more matches in app/settings.py]
```

— i.e. the numbers were part of line 476's own content all along, and
497/535/556
were dropped as ordinary matches. `Line N:` rows are not parsed as
matches at all
and pass through untouched, which matches @TigerkidYang's finding on
`main`.

**But the class of bug the issue is about is real, and it is in the
parser.** Here
is the same failure with different numbers, on a shape that grep emits
routinely —
a `-`-separated *context* line (`path-line-content`) whose body carries
a
whitespace-free `name:N:` reference:

```
input   app/settings.py-476-a:7:b:8:c
parsed  ("app/settings.py-476-a", 7, "b:8:c")
correct ("app/settings.py",      476, "a:7:b:8:c")
```

Three separate falsehoods in one row:

1. the path swallows the real `-476-` marker and invents
`app/settings.py-476-a`,
   a file that does not exist;
2. **the reported line number is the body's `7`, not the row's `476`** —
this is
   the manufactured line↔content association the issue is about;
3. the body is truncated to `b:8:c`.

Worse, the bogus path becomes the grouping key in
`parse_search_results`, so
every match in that file is filed under a nonexistent path.

This is not speculative. On the shipped `headroom-ai==0.38.0` build,
over a
generated corpus of tricky paths/line numbers/bodies:

```
$ uv run --with headroom-ai==0.38.0 python probe_parse.py
checked 2394 colon-form rows; 0 mismatched
dash-form mismatches: 279
```

`path:line:content` (match lines — the dominant shape) is clean. Every
one of the
279 failures is a `path-line-content` context line with a
whitespace-free
`name:N:` in its body.

### Root cause

`parse_match_line` tries the **colon tier first** and it commits
unconditionally
on the first `:\d+:` whose prefix has no whitespace. Its own doc comment
(search_compressor.rs:646-651) claims:

> The no-whitespace guard keeps a `foo.rs:12:` reference *inside the
body* of a
> `-`-separated context line from hijacking the parse.

The guard only holds when the body happens to contain whitespace ahead
of the
reference. `a:7:b` has none, so the colon tier takes it, and the dash
tier —
which reads the row correctly — is never consulted.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/transforms/search_compressor.rs`
- `parse_match_line` now hands priority back to the dash tier for
exactly one
shape: the dash tier *positively confirmed* a boundary that sits before
the
colon marker, **and** nothing path-like (separator or extension dot)
lies
    between the two. The colon triplet is then body text, not a marker.
- The colon tier keeps priority everywhere else, so `path:line:content`
rows
    and rows whose **path** holds the dash triplet are untouched —
    `logs/2026-05-03/app.log:12:ERROR` and
`migrations/20240101-002-add_users.sql:12:SELECT` still resolve
correctly,
    because the dash tier never confirmed a boundary there.
- `scan_match_line` becomes `scan_marker`, returning the marker it
settled on
**plus a flag** for whether that marker was a confirmed boundary
(`chosen`)
    or merely the leftmost fallback (`first`). The caller needs that
    proven-vs-guessed distinction; it is the whole safety condition.
- Doc comment updated — it previously asserted a guarantee the code did
not
    provide.
- New unit test
`dash_context_line_wins_over_a_colon_marker_in_its_body`.
- `tests/test_search_compressor.py`: new `TestContextLineBodyReference`
class
  (4 tests) covering the repro, the extension-less path variant, the
must-not-regress colon row with a dated path, and a 12-row sweep
asserting
  every surviving coordinate and body.

**Deliberately not changed:** the `Dockerfile-3-1-` shape is genuinely
ambiguous
in text (`Dockerfile-3` line 1, or `Dockerfile` line 3) and keeps its
existing
reading. The dash tier's comments already document that trade-off, and
this PR
does not widen or narrow it.

## Testing

- [x] New tests added for new functionality
- [ ] Unit tests pass (`pytest`) — see the note below
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] Manual testing performed — against the shipped wheel and a
compiled harness

**Why `pytest` is unticked:** the Python tests need the compiled
`headroom._core`
extension, which requires a checkout of the repo. I worked remote-only,
so I
could not run them. I validated every assertion they make by running the
identical expectations as a Rust test against the patched parser (below,
case
`python_test_expectations`) — but that is not the same as having run
pytest, and
I would rather say so than imply otherwise. The Rust unit test in the
crate is
exercised by the repo's `cargo` CI.

I also could not run the 8 crate-level tests that need the compressor
types
(`select_respects_per_file_cap_and_global_cap`, the four `ccr_*` tests,
etc.).
They are untouched by this diff and CI will run them.

### Rust: the parser compiled and run for real

The parser is self-contained, so I extracted the functions I touched
plus all 20
parser-only tests **verbatim** from the patched file and compiled them
with the
real toolchain (rustc 1.93.1, `--target x86_64-pc-windows-gnu`; the repo
pins
1.95.0 in `rust-toolchain.toml`):

```
$ rustc --test --edition 2021 --target x86_64-pc-windows-gnu -o parser_harness.exe parser_harness.rs
$ ./parser_harness.exe
running 20 tests
test tests::dash_context_line_wins_over_a_colon_marker_in_its_body ... ok
test tests::body_line_reference_does_not_hijack_a_context_line ... ok
test tests::dash_tier_does_not_run_past_the_path_into_the_body ... ok
test tests::date_stamped_path_is_not_misread_as_line_number_marker ... ok
test tests::dated_path_inside_a_context_body_is_not_hijacked ... ok
test tests::parses_context_line_for_extensionless_path ... ok
test tests::fixed_in_3e2_handles_windows_path_with_backslash ... ok
test tests::ambiguous_dash_line_falls_back_to_the_leftmost_marker ... ok
...
test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
```

With **only** `parse_match_line` reverted to the pre-fix three-tier
chain and
nothing else changed, the new test fails exactly as the bug predicts:

```
thread 'tests::dash_context_line_wins_over_a_colon_marker_in_its_body' panicked:
  left: Some(("app/settings.py-476-a", 7, "b:8:c"))
 right: Some(("app/settings.py", 476, "a:7:b:8:c"))
test result: FAILED. 19 passed; 1 failed; 0 ignored
```

All 19 pre-existing parser tests still pass in both runs — dated paths,
CVE
paths, Windows drive letters, dashes in filenames, the ambiguous
fallback, the
`foo.rs:12:`-with-whitespace case, and CJK.

### Rust: blast radius over a generated corpus

I also transcribed the tier logic and diffed current-order vs
patched-order
across `19 paths × 9 line numbers × 17 bodies` in both separators:

```
colon-form (2907 rows):  current-order wrong=0     patched wrong=0
dash-form  (2907 rows):  current-order wrong=612   patched wrong=153
```

All 153 remaining are the single `Dockerfile-3` path, and they are
**identical
before and after** — the documented ambiguity, unchanged. Every
unambiguous
mis-parse is fixed and no colon-form row changes.

### Lint / format

```
$ uv run --with "ruff==0.16.8" ruff check --line-length 100 --target-version py310 \
    --select E,W,F,I,B,C4,UP --ignore E501,B008,B905 tests/test_search_compressor.py
All checks passed!
$ uv run --with "ruff==0.16.8" ruff format --check --line-length 100 \
    --target-version py310 tests/test_search_compressor.py
1 file already formatted
$ rustfmt --edition 2021 --check crates/headroom-core/src/transforms/search_compressor.rs
(no diff; exit 0)
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.15, rustc 1.93.1. No repo checkout
for this
branch (remote-only contribution), so no end-to-end proxy run and no
`cargo test`
  on the full crate.
- Exact command / steps: `uv run --with headroom-ai==0.38.0 python
probe_parse.py`
against the shipped compiled parser, then `rustc --test --edition 2021
--target
x86_64-pc-windows-gnu -o parser_harness.exe parser_harness.rs` on the
patched
parser extracted verbatim, then `./parser_harness.exe`. Both command
sets are
  quoted in full under *Testing*.
- Observed result: the defect reproduces in the **shipped** build —
  `parse_search_lines("app/settings.py-476-a:7:b:8:c")` returns
`("app/settings.py-476-a", 7, "b:8:c")`, i.e. a wrong line number (7 for
476), a
path that does not exist, and a truncated body. Over a generated corpus
it reports
`checked 2394 colon-form rows; 0 mismatched` / `dash-form mismatches:
279`. After
the change the same input returns `("app/settings.py", 476,
"a:7:b:8:c")` and the
harness reports `20 passed; 0 failed`, with the 19 pre-existing parser
tests still
  green.
- Not tested: the full `cargo test` suite for the crate; the 8
crate-level
compressor tests that need the compressor types; `pytest` (needs the
built Rust
extension); a live proxy request carrying a context line; ripgrep `-C`
output
  end-to-end through the proxy.

If you want the last row closed before merge, the check is a `grep -rn
-C2`
whose context lines contain a `name:N:` reference, run through
`headroom compress` — the parsed coordinates should be the row's own.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: the change is in the Rust core, so it ships
with the
  next wheel.
- Stable/default behavior changed: yes, for one input shape only. Rows
that
previously mis-parsed now parse correctly; every other row is
byte-identical
(0 changes across 2907 colon-form rows, and the only changed dash-form
rows
  are the 459 that were wrong).
- Kill switch / disable path: revert this commit.
- Unsafe override required: none.
- Qualification impact: none. No config, flag, or metric changes.
- Rollback path: revert; the mis-parse returns.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — see
*Testing*
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- **On the report itself:** if you can still get the original failure,
the raw
tool result as your harness delivered it (plus `headroom inspect --last
1`) is
what would settle it. My working theory is the `grep_numbers_in_content`
case
above — the numbers were in line 476's own content and 497/535/556 were
dropped as ordinary matches, which reads exactly like fusion. If the raw
  result shows a genuinely different transform (e.g. `smart_crusher` /
`text_block`), this fix will not address it and the issue should stay
open on
  that thread.
- The parser is a deliberate three-tier design with carefully documented
trade-offs, so I kept the change to a single override condition rather
than
reordering the tiers. Reordering outright is the obvious alternative and
it
  **regresses** real rows — `logs/2026-05-03/app.log:12:ERROR` and
`migrations/20240101-002-add_users.sql:12:SELECT` both break — which is
why the
  override is gated on a *confirmed* dash boundary.
- The `chosen.is_some()` flag is the safety property: the permissive
tier's
leftmost fallback never counts as a confirmed boundary, so a line the
parser
  merely guessed at keeps today's behaviour.
- The sibling defect raised in this thread (Kompress dropping ` or `
from code)
  was already fixed by #3687, so it is out of scope here.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
ANIRUDDHA ADAK
2026-09-30 11:14:05 -05:00
committed by GitHub
co-authored by JD Davis
parent afaaaa8804
commit f3f2e0077a
2 changed files with 222 additions and 10 deletions
@@ -648,7 +648,10 @@ impl SearchCompressor {
/// practically never contains one (the Windows drive colon is
/// already skipped above), so the leftmost is the right one. The
/// no-whitespace guard keeps a `foo.rs:12:` reference *inside the
/// body* of a `-`-separated context line from hijacking the parse.
/// body* of a `-`-separated context line from hijacking the parse. A
/// whitespace-free body reference (`a:7:b`) still slips past it, so
/// [`parse_match_line`] hands priority back to the dash tier for that
/// one shape; see the comment there.
/// 2. **Dash tier** — `-` is grep's *context*-line separator, and unlike
/// `:` it appears inside real paths (`2026-05-03`, `CVE-2021-44228`,
/// `20240101-002-add_users.sql`), so the leftmost triplet is often
@@ -684,9 +687,50 @@ impl SearchCompressor {
/// Only reached when neither tier matched (e.g. a path containing a
/// space), so behaviour for those lines is exactly as before.
fn parse_match_line(line: &str) -> Option<(&str, u64, &str)> {
scan_match_line(line, ScanTier::Colon)
.or_else(|| scan_match_line(line, ScanTier::Dash))
.or_else(|| scan_match_line(line, ScanTier::Permissive))
let colon = scan_marker(line, ScanTier::Colon);
let dash = scan_marker(line, ScanTier::Dash);
// The colon tier keeps priority: `path:line:content` is what grep emits for
// matches and a path practically never contains a colon. It yields in
// exactly one shape — a `-`-separated context line whose body carries a
// whitespace-free `name:N:` reference (`app.py-476-a:7:b`). The dash tier
// *positively confirmed* a boundary before the colon marker, so that colon
// triplet is body text, not a marker.
//
// Without the hand-off the body's number is read as the line number, the
// path swallows the real `-N-`, and the model is shown a file that does not
// exist holding content attributed to a line it never came from
// (issue #3545). The whitespace guard inside the colon tier cannot catch
// this, because the body ahead of the reference need not contain any.
//
// The `true` in the dash arm is load-bearing, and it is the *whole* test.
// A confirmed boundary means the dash tier saw the path end there — either
// the segment carried an extension (`app.py-476-`) or nothing after the
// marker looked like path structure (`CHANGELOG-12-`) — so the bytes from
// the closing dash onwards are body text and may look like anything. Bodies
// routinely carry a filename-style reference of their own
// (`app.py-476-foo.rs:12:ref`), and an extension dot there says nothing
// about where the path ended.
//
// The rows that must keep the colon tier are the ones where the dash tier
// could *not* confirm: `logs/2026-05-03/app.log:12:ERROR` and
// `migrations/20240101-002-add_users.sql:12:SELECT` both leave the digits
// inside the path, so the flag is false and the hand-off never applies. See
// [`path_continues`], which is what decides the flag.
let dash_overrides = match (colon, dash) {
(Some((colon_marker, _)), Some((dash_marker, true))) => colon_marker.0 > dash_marker.2,
_ => false,
};
let marker = if dash_overrides {
dash
} else {
colon
.or(dash)
.or_else(|| scan_marker(line, ScanTier::Permissive))
};
build_match(line, marker?.0)
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -752,7 +796,22 @@ fn path_continues(rest: &str) -> bool {
.is_some_and(|tok| tok.contains('/') || tok.contains('\\') || has_extension_dot(tok))
}
fn scan_match_line(line: &str, tier: ScanTier) -> Option<(&str, u64, &str)> {
/// Byte offsets of one `<sep><digits><sep>` marker: where the path ends, where
/// the digits start, and where the closing separator sits.
type Marker = (usize, usize, usize);
/// Assemble `(file, line_number, content)` from a marker's byte offsets.
fn build_match(line: &str, marker: Marker) -> Option<(&str, u64, &str)> {
let (path_end, digits_start, digits_end) = marker;
let line_no = line[digits_start..digits_end].parse::<u64>().ok()?;
Some((&line[..path_end], line_no, &line[digits_end + 1..]))
}
/// Scan `line` under `tier`, returning the marker it settled on plus whether
/// that marker was a *positively confirmed* boundary (`chosen`) rather than the
/// leftmost fallback (`first`). Callers need that flag to tell a proven boundary
/// from a guess.
fn scan_marker(line: &str, tier: ScanTier) -> Option<(Marker, bool)> {
let bytes = line.as_bytes();
// Windows drive prefix: starts with [A-Za-z]:[\\/]
let scan_start = if bytes.len() >= 3
@@ -865,11 +924,7 @@ fn scan_match_line(line: &str, tier: ScanTier) -> Option<(&str, u64, &str)> {
// is genuinely ambiguous (`Makefile-7-include-2-src/foo.mk`). Fall back
// to the leftmost marker — precisely what the leftmost-wins rule this
// tier refines would have returned.
let (path_end, digits_start, digits_end) = chosen.or(first)?;
let line_no = std::str::from_utf8(&bytes[digits_start..digits_end])
.ok()
.and_then(|s| s.parse::<u64>().ok())?;
Some((&line[..path_end], line_no, &line[digits_end + 1..]))
Some((chosen.or(first)?, chosen.is_some()))
}
// ─── Internals ──────────────────────────────────────────────────────────
@@ -918,6 +973,60 @@ mod tests {
assert!(cjk_bigrams("a认b证").is_empty()); // isolated CJK chars -> no pair
}
#[test]
fn dash_context_line_wins_over_a_colon_marker_in_its_body() {
// A whitespace-free `name:N:` reference in the body of a `-`-separated
// context line used to be read as the line-number marker: the path
// absorbed the real `-N-` and the body's number was reported as the
// line number, inventing both a file that does not exist and a
// line<->content pair that never existed (issue #3545).
assert_eq!(
parse_line("app/settings.py-476-a:7:b:8:c"),
Some(("app/settings.py".into(), 476, "a:7:b:8:c".into()))
);
// Extension-less paths reach the dash tier through its negative stop
// rather than the extension rule, and must keep resolving.
assert_eq!(
parse_line("CHANGELOG-12-a:99:b"),
Some(("CHANGELOG".into(), 12, "a:99:b".into()))
);
// A body reference that looks like a *filename* is the canonical form
// the hand-off used to miss. `app.py-476-foo.rs:12:ref` is a context
// row for `app.py` line 476 whose body is `foo.rs:12:ref`. The dash
// tier confirms the boundary (the segment carries an extension), so the
// extension dot inside the body must not veto it — before the fix the
// colon tier reclaimed the row as path `app.py-476-foo.rs`, line 12.
assert_eq!(
parse_line("app.py-476-foo.rs:12:ref"),
Some(("app.py".into(), 476, "foo.rs:12:ref".into()))
);
// Same shape behind a directory, and with a Windows separator in the
// body reference.
assert_eq!(
parse_line("pkg/server.ts-91-lib/index.js:7:import"),
Some(("pkg/server.ts".into(), 91, "lib/index.js:7:import".into()))
);
// The body may even be a bare path with no reference at all.
assert_eq!(
parse_line("app.py-476-./vendor/other.py"),
Some(("app.py".into(), 476, "./vendor/other.py".into()))
);
// A colon row whose *path* holds the dash triplet is untouched: the
// dash tier never confirmed a boundary there, so the colon tier wins.
assert_eq!(
parse_line("logs/2026-05-03/app.log:12:ERROR"),
Some(("logs/2026-05-03/app.log".into(), 12, "ERROR".into()))
);
assert_eq!(
parse_line("migrations/20240101-002-add_users.sql:12:SELECT"),
Some((
"migrations/20240101-002-add_users.sql".into(),
12,
"SELECT".into()
))
);
}
#[test]
fn dash_tier_does_not_run_past_the_path_into_the_body() {
// The dash tier keeps the *last* `-<digits>-` marker so that a
+103
View File
@@ -112,6 +112,109 @@ more random text
assert len(file_matches) == 2
class TestContextLineBodyReference:
"""A ``name:N:`` reference in a context line's body must not become the marker.
ripgrep emits ``path-line-content`` for *context* lines and
``path:line:content`` for *match* lines, and real bodies routinely carry
their own ``name:N:`` references. The colon tier used to claim such a
reference as the line-number marker whenever no whitespace preceded it,
which pushed the real ``-N-`` marker into the path and reported the body's
number as the line number -- inventing a file that does not exist and
pairing content with a line it never came from (issue #3545).
"""
def test_body_reference_does_not_become_the_line_number(self):
content = "app/settings.py-476-a:7:b:8:c"
compressor = SearchCompressor()
file_matches = compressor._parse_search_results(content)
assert list(file_matches) == ["app/settings.py"]
matches = file_matches["app/settings.py"].matches
assert len(matches) == 1
assert matches[0].line_number == 476
assert matches[0].content == "a:7:b:8:c"
def test_extensionless_context_path_keeps_its_own_line_number(self):
content = "CHANGELOG-12-a:99:b"
compressor = SearchCompressor()
file_matches = compressor._parse_search_results(content)
assert list(file_matches) == ["CHANGELOG"]
matches = file_matches["CHANGELOG"].matches
assert len(matches) == 1
assert matches[0].line_number == 12
assert matches[0].content == "a:99:b"
def test_colon_row_with_dashed_path_is_unaffected(self):
"""The dash marker sits in the *path* here, so the colon tier still wins."""
content = "logs/2026-05-03/app.log:12:ERROR"
compressor = SearchCompressor()
file_matches = compressor._parse_search_results(content)
assert list(file_matches) == ["logs/2026-05-03/app.log"]
matches = file_matches["logs/2026-05-03/app.log"].matches
assert len(matches) == 1
assert matches[0].line_number == 12
assert matches[0].content == "ERROR"
def test_filename_style_body_reference_keeps_the_context_coordinates(self):
"""A body reference that looks like a *filename* must not take the row.
``app.py-476-foo.rs:12:ref`` is a ripgrep context row for ``app.py``
line 476 whose body is ``foo.rs:12:ref``. The dash tier confirms the
boundary (the segment carries an extension), so the extension dot
inside the body says nothing about where the path ended. Before the
fix the colon tier reclaimed the row as the nonexistent path
``app.py-476-foo.rs`` at line 12.
"""
content = "app.py-476-foo.rs:12:ref"
compressor = SearchCompressor()
file_matches = compressor._parse_search_results(content)
assert list(file_matches) == ["app.py"]
matches = file_matches["app.py"].matches
assert len(matches) == 1
assert matches[0].line_number == 476
assert matches[0].content == "foo.rs:12:ref"
def test_filename_style_body_reference_behind_a_directory(self):
"""Same shape with a directory component, which also looks path-like."""
content = "pkg/server.ts-91-lib/index.js:7:import"
compressor = SearchCompressor()
file_matches = compressor._parse_search_results(content)
assert list(file_matches) == ["pkg/server.ts"]
matches = file_matches["pkg/server.ts"].matches
assert len(matches) == 1
assert matches[0].line_number == 91
assert matches[0].content == "lib/index.js:7:import"
def test_bare_path_body_keeps_the_context_coordinates(self):
"""The body needs no reference at all to be path-like."""
content = "app.py-476-./vendor/other.py"
compressor = SearchCompressor()
file_matches = compressor._parse_search_results(content)
assert list(file_matches) == ["app.py"]
matches = file_matches["app.py"].matches
assert len(matches) == 1
assert matches[0].line_number == 476
assert matches[0].content == "./vendor/other.py"
def test_many_context_rows_keep_their_own_coordinates(self):
"""Every coordinate an agent would act on survives across many rows."""
content = "\n".join(f"pkg/mod/file.py-{476 + i * 7}-hits:{i}:of:9" for i in range(12))
compressor = SearchCompressor()
file_matches = compressor._parse_search_results(content)
assert list(file_matches) == ["pkg/mod/file.py"]
matches = file_matches["pkg/mod/file.py"].matches
assert [m.line_number for m in matches] == [476 + i * 7 for i in range(12)]
for i, match in enumerate(matches):
assert match.content == f"hits:{i}:of:9"
class TestFileGrouping:
"""Tests for grouping matches by file."""