fix(router): ignore shell operators inside quoted arguments in read detection (#4646)

## Description

Item 3 of #4478, the quoted-pattern case that #4521 left open. Read
detection scanned the raw command text for shell syntax without regard
to quoting: `_segment_is_read` treated any `>` as a redirect
(`content_router.py:893` on `e4a60ed7`) and split the pipeline at the
first `|`, and `_command_segments` split on `;`, `&&` and `||`. A quiet
sed whose script holds one of these characters, such as `sed -n '/if x >
0/,/return/p' a.py` or the GNU alternation `sed -n '/def \|class /p'
a.py`, was therefore not detected as a file read and lost read
protection.

The three scans now run on a copy of the command in which quoted,
backslash-escaped and commented characters are masked, and the match
positions are mapped back onto the original text. Two shell rules keep
word boundaries intact (review round 2): a backslash-newline is a line
continuation and is masked as two spaces, so a redirect or `tee` on the
next line still counts as a write; and an unquoted `#` at the start of a
word masks the rest of its line without changing quote state, so a quote
mark inside a comment cannot hide the next line's separators. Unquoted
operators behave as before.

Refs #4478

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_router.py`: new `_mask_quoted(command)`,
which returns a same-length string with characters inside `'...'` /
`"..."`, backslash-escaped characters and comment text replaced by `_`
(the quote characters are kept; a backslash-newline becomes two spaces;
a comment's newline stays visible). `_segment_is_read` runs its
redirect/`tee` check and its pipeline split on the masked string;
`_command_segments` finds `;` / `&&` / `||` on the masked string and
slices the original.
- `tests/test_transforms/test_read_command_detection.py`:
`test_quoted_shell_operators_are_arguments`, 25 cases. Quoted `>`, `>>`,
`|`, `;`, `&&` and `tee` inside a sed script or a file name are reads; a
real redirect, pipe-to-`tee` or separator next to a quoted script still
counts; `bash -lc "cat foo.py > copy.py"` is still not a read (the inner
command is checked on its own, as before). Round 2 adds: continued `>`,
`>>` and `| tee` are writes; a comment with an unmatched `"` or `'`
followed by `ls; cat foo.py` / `true; cat foo.py` on the next line is a
read; `cat foo.py # > not a redirect` is a read; a `#` inside a word is
not a comment.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) (run on
`headroom/transforms/content_router.py` only, see below)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# fail before (source at e4a60ed7, new test kept)
tests/test_transforms/test_read_command_detection.py: 9 failed, 104 passed

# round 2: the new review cases against the round-1 head 639bfa9e
7 failed, 115 passed

# pass after (head 50798e5a)
tests/test_transforms/test_read_command_detection.py: 122 passed

# wider set, same environment:
# tests/test_transforms + tests/test_*content_router*.py + tests/test_router*.py
e4a60ed7:     27 failed, 930 passed, 96 skipped   (round 1 comparison)
639bfa9e:     25 failed, 941 passed, 96 skipped
50798e5a:     18 failed, 948 passed, 96 skipped
# the 18 are the same tests on every side (the extras are the new cases); they
# come from the local `_core` build not matching this tree (Rust parity
# fixtures, minification), not from this change

# ruff check / ruff format --check: All checks passed! / 2 files already formatted
# mypy headroom/transforms/content_router.py: Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Ubuntu (WSL2) on Windows 11, Python 3.12.15, `headroom`
Python source at `e4a60ed7` and at this branch, offline.
- Exact command / steps: `ContentRouter.apply` on a three-message
transcript (user, an assistant `Bash` tool call, a 3,890-byte Python
tool result), with `HEADROOM_PROTECT_READS=1`,
`HEADROOM_SHELL_RESULT_MIN_CHARS=2000` and the router config used by
`test_quiet_sed_output_reaches_read_protection`
(`enable_kompress=False`, `enable_code_aware=False`). Checked whether
`router:read_protected` is in `transforms_applied` for each command.
- Observed result: on `e4a60ed7` the three quoted-operator commands lose
read protection; on this branch they keep it, and both controls are
unchanged (table below). For the review-round cases, real `bash -c` in a
temporary directory agrees with `_is_read_command` on this head for all
7 (continued `>`, `>>` and `| tee` write `copy.py`; the three
multiline-comment commands and `cat foo.py # > not a redirect` print the
file); on `639bfa9e` 6 of the 7 disagreed with bash.

  | command | `e4a60ed7` | this branch |
  |---|---|---|
  | `sed -n '1,120p' foo.py` (control) | protected | protected |
| `sed -n '/if value > 0/,/return 0/p' foo.py` | not protected |
protected |
  | `sed -n '/def \|class /p' foo.py` | not protected | protected |
  | `sed -n '/x;y/p' foo.py` | not protected | protected |
| `sed -n '1,120p' foo.py > out.txt` (control, real redirect) | not
protected | not protected |

- Not tested: a run in which an unprotected read is actually compressed.
With Kompress and code-aware compression off, the tool result came back
byte-identical on both sides, so this proof shows the classification
only. Not tested: the full test suite and `mypy headroom` as a whole
(only `content_router.py`). Heredoc handling (`<<`) is unchanged and
still a whole-command bailout.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly: commands whose only
`>`, `|`, `;`, `&&` or `||` sits inside quotes, after a backslash or
inside a comment are now classified by the rest of the command, so a
quiet sed or `cat` of that kind gets read protection when
`HEADROOM_PROTECT_READS` is on.
- Kill switch / disable path: None added; read protection as a whole is
still controlled by `HEADROOM_PROTECT_READS`.
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

#4478 had four items. Items 1 and 2 were fixed by #4488, item 4 by
#4583, and the quiet-option half of item 3 by #4521, whose description
notes that quoted redirect parsing remained separate. This PR covers
that remainder. The `|` and `;` cases are the same cause as the `>` case
reported in the issue (a quote-blind scan of the same command), so they
are fixed together rather than left for a later PR.

Review round 2 (head `50798e5a`) addresses the two masking regressions
found by Devin and Copilot and confirmed by JerrettDavis:
backslash-newline continuations and quote marks inside comments.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


### Maintainer requalification

Re-reviewed the author update against upstream main
e4a60ed72f. The two earlier actual Git
Bash counterexamples now classify correctly: continued redirect writes
an exact copy and is not a read; a quote inside a comment no longer
hides the next-line source read. All 132 focused read-detection and
Gemini tests pass; touched-source Ruff/format and whitespace checks
pass. The only maintainer change is the shared Gemini ASGI request test
fixture from #4644; hosted failure was the same four independently
verified main failures, with no contributor production logic changed.
Addressed Devin/Copilot threads resolved after inspecting author fixes.
Exact-head hosted CI remains pending.


### Maintainer current-main requalification

Exact head `2461a910e31c5140d160942927c1c7b3fa203cc1`, upstream main
`ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`.

Cleanly integrated upstream/main
ea59ac3eef and re-reviewed the additive
content-router overlap with main's confident HTML classification. Actual
isolated tree-sitter0.26.0:133 read-command/HTML tests pass,94 unrelated
cases deselected,1 HTML assertion fails. A control with byte-identical
main Python/transforms/test/lock inputs reproduces that same
unrecoverable_sections assertion using the same existing development
native extension, so it is not a finding against this PR. Read-command
cases pass; prior actual Git Bash continued-redirection/comment-quote
evidence remains scoped to unchanged shell parser inputs. Ruff and
whitespace pass. No contributor behavior rewritten, fresh native build
or cloud runtime claim. No new PR correctness finding; final approval
awaits exact-head hosted checks. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `69a6bd592932467162d9433601b0c2bd1c4c4728`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 134 passed, 94 deselected in 5.40s.
Selected real dependency overlays and native DLL paths are recorded in
final4646-fresh-native-tests.log; the DLL provenance/input hashes are in
final-native-artifact-proof.json. No redundant rerun is implied after
this Cargo-only merge. Whitespace passes. Prior source review and scoped
real-behavior evidence remain applicable to unchanged contributor
inputs.

No new source finding in the reviewed scope. Final formal approval
awaits exact-head hosted qualification. The earlier HTML assertion with
the old primary extension now passes with the fresh combined DLL; the
same case also passes on a byte-identical-main Python-source control. No
production workaround or assertion suppression was required.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
HetCreep
2026-11-06 09:20:15 -06:00
committed by GitHub
co-authored by JerrettDavis Claude Opus 5.5
parent 07d5add5ba
commit 8cdce172ea
2 changed files with 117 additions and 4 deletions
+73 -4
View File
@@ -880,9 +880,72 @@ def _command_segments(command: str) -> list[str]:
Splits on ``;``, ``&&`` and ``||`` — never on a single ``|``. Pipelines are kept
whole so the segment's own writes (``cat a.py | tee b.py``) stay visible; the
reduction to the first stage happens in :func:`_segment_is_read`.
reduction to the first stage happens in :func:`_segment_is_read`. Separators inside
quotes (``sed -n '/a;b/p' foo.py``) are part of an argument, not a new command.
"""
return [seg.strip() for seg in _CMD_SEPARATOR_RE.split(command) if seg.strip()]
masked = _mask_quoted(command)
segments: list[str] = []
start = 0
for m in _CMD_SEPARATOR_RE.finditer(masked):
segments.append(command[start : m.start()])
start = m.end()
segments.append(command[start:])
return [seg.strip() for seg in segments if seg.strip()]
def _mask_quoted(command: str) -> str:
"""Return ``command`` with quoted, escaped and commented characters replaced by ``_``.
The result has the same length, so match positions map back onto ``command``.
Shell operators inside quotes (``sed -n '/x >/p' foo.py``), after a backslash, or
inside a comment are not shell syntax, and a search over the masked string does
not mistake them for a redirect, pipe or command separator. The quote characters
themselves are kept.
Two shell rules keep word boundaries intact: a backslash-newline is a line
continuation, so both characters become spaces (``cat a.py \\<newline>> b.py``
still writes ``b.py``); and a ``#`` at the start of a word starts a comment that
runs to the end of the line, so quote marks inside it do not open a quote that
would hide the next line's separators.
"""
out = list(command)
quote = ""
i = 0
n = len(command)
word_start = True
while i < n:
char = command[i]
if char == "\\" and quote != "'":
if i + 1 < n and command[i + 1] == "\n":
# Line continuation: the shell removes both characters.
out[i] = out[i + 1] = " "
word_start = True
else:
out[i] = "_"
if i + 1 < n:
out[i + 1] = "_"
word_start = False
i += 2
continue
if quote:
if char == quote:
quote = ""
else:
out[i] = "_"
elif char in {"'", '"'}:
quote = char
word_start = False
elif char == "#" and word_start:
# Comment to end of line; the newline itself stays visible.
while i < n and command[i] != "\n":
out[i] = "_"
i += 1
word_start = True
continue
else:
word_start = char.isspace() or char in ";&|()"
i += 1
return "".join(out)
def _segment_is_read(c: str) -> bool:
@@ -890,9 +953,15 @@ def _segment_is_read(c: str) -> bool:
# A redirect or `tee` in THIS segment means the segment writes a file rather than
# reading one (`cat a.py > copy.py`, `cat a.py | tee copy.py`). Checked on the whole
# segment — pipeline included — before reducing to the stage that touches the file.
if re.search(r"(^|\s)(>>?|tee\b)", c):
# Both checks run on the quote-masked segment: a `>` or `|` inside a quoted sed
# script (`sed -n '/x >/p' a.py`) is part of the argument, not shell syntax.
masked = _mask_quoted(c)
if re.search(r"(^|\s)(>>?|tee\b)", masked):
return False
c = c.split("|", 1)[0].strip()
pipe = masked.find("|")
if pipe != -1:
c = c[:pipe]
c = c.strip()
if not c:
return False
# Parse the real program with the SAME structural parser the search-fold uses
@@ -123,6 +123,50 @@ def test_non_reads_are_not_detected(command: str) -> None:
assert _is_read_command(command) is False
@pytest.mark.parametrize(
("command", "expected"),
[
# Shell operators inside a quoted sed script are part of the argument (#4478).
("sed -n '/x >/p' foo.py", True),
("sed -n '/x >> y/p' foo.py", True),
('sed -n "/a > b/p" foo.py', True),
("sed -n '/a|b/p' foo.py", True),
("sed -n '/a;b/p' foo.py", True),
("sed -n '/a && b/p' foo.py", True),
("sed -n '/ tee /p' foo.py", True),
("cat 'a > b.py'", True),
# An escaped operator is a literal argument too.
(r"cat foo.py \> out", True),
# Real operators next to a quoted script still count.
("sed -n '/x >/p' foo.py > out.txt", False),
("sed -n '/a|b/p' foo.py | tee out.txt", False),
("sed -n '/a;b/p' foo.py; echo done", True),
("echo '>' && sed 's/a/b/' foo.py", False),
# The -c argument of a shell is quoted at the outer level; the inner
# command is checked on its own.
('bash -lc "cat foo.py > copy.py"', False),
('bash -lc "cat foo.py | tee copy.py"', False),
('bash -lc "cat foo.py"', True),
# A backslash-newline is a line continuation: the write on the next line
# still counts (review of #4646).
("cat foo.py \\\n> copy.py", False),
("cat foo.py \\\n>> copy.py", False),
("cat foo.py | \\\ntee copy.py", False),
("sed -n '/x >/p' foo.py \\\n> out.txt", False),
("sed -n '1,60p' \\\nfoo.py", True),
# A comment ends at the newline; its quote marks do not open a quote that
# would hide the next line's separators (review of #4646).
('echo start # " unclosed in comment\nls; cat foo.py', True),
("echo ok # author's note\ntrue; cat foo.py", True),
("cat foo.py # > not a redirect", True),
# A hash inside a word is not a comment.
("cat foo#bar.py > out.txt", False),
],
)
def test_quoted_shell_operators_are_arguments(command: str, expected: bool) -> None:
assert _is_read_command(command) is expected
def test_heredoc_body_is_not_split_into_a_bogus_read() -> None:
"""A `;` inside a heredoc body must not produce a segment that looks like a
read — the command as a whole writes a file."""