fix(router): preserve read protection for quiet sed options (#4521)

## Description

With `HEADROOM_PROTECT_READS=1`, `sed -n '1,60p' file.py` achieves
byte-exact read protection, but equivalent `-ne`, `-En`, `--quiet`, and
`--silent` forms currently do not. The classifier previously checked for
a literal `-n` token across the raw command string, which also caused it
to mistake `-n` inside an expression argument as a quiet flag.

This PR properly parses sed options before classifying whether a command
is a read. It handles grouped flags and long options, respects script
arguments and `--` delimiters, and excludes in-place edits (`-i`). GNU
and BSD `-l` forms remain recognized.

Addresses the quiet-option parsing portion of item 3 in #4478 (quoted
redirect parsing and other items in that issue remain separate). Note
that `sed -n1,80p` is invalid GNU sed syntax and remains unhandled.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Recognize quiet sed options using standard-library shell tokenization
and robust option parsing.
- Keep script arguments separate from options, and exclude in-place
edits from protected reads.
- Add command-classification unit tests and end-to-end
`ContentRouter.apply()` regression coverage for all five quiet forms.

## Testing

- [x] Unit tests pass (`pytest`): focused and related suites below
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): native Linux pre-commit
configuration
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Focused tests against unchanged main's router:
16 failed, 44 passed

# This patch:
60 passed

# Read detection, Responses read protection, read audit, maturation, lifecycle:
147 passed

# make ci-precheck-rust (Rust 1.95.0):
# cargo fmt --all -- --check; cargo clippy --workspace -- -D warnings;
# cargo test --workspace
1659 passed, 3 ignored

# make ci-precheck-python (including native extension rebuild):
177 passed, 4 skipped

# ruff check .
All checks passed!
# ruff format --check .
1834 files already formatted
```

The native extension rebuild and import verification pass cleanly. The
repository's unmodified pre-commit configuration passes on native Linux
(WSL Ubuntu, Python 3.14.4), including mypy, Ruff, formatting, conflict
checks, version checks, and commit-message hooks.

On Windows, the default mypy hook reports seven POSIX-only `os`
attribute errors in unchanged files (`fileperms.py` and
`proxy/interceptors/astgrep.py`), matching unmodified main (#4520).
Similarly, broader Windows test failures (18 failed, 5367 passed, 190
skipped, 2 errors) reproduce identically on main and stem from known
platform differences (symlink privileges, SIGHUP handling, and Windows
environment-variable size limits).

## Real Behavior Proof

- Environment: Windows x86_64, Python 3.13.9, GNU sed 4.9, current-main
Python sources with native `headroom._core` extension. ML and code
compression are disabled to directly isolate the read-protection route.
- Exact command / steps: Run the script below with main's router versus
this patch. It executes real GNU sed against a temporary Python source
file and passes each tool result through `ContentRouter.apply()`.
- Observed result: The standard `-n` control is protected on both
versions. On main, the other four forms miss `router:read_protected`.
With this patch, all five forms correctly trigger read protection and
preserve the 7,998-character output byte-for-byte.
- Not tested: Focuses on command parsing and routing behavior. BSD
option syntax is validated via parser unit tests; live proxy traffic and
model calls were not required.

Save as `repro_sed_quiet_reads.py`, then run `python
repro_sed_quiet_reads.py`:

```python
"""Run real GNU sed output through ContentRouter, without a model or provider."""
import json
import os
from pathlib import Path
import subprocess
import tempfile

os.environ["HEADROOM_OFFLINE"] = "1"
os.environ["HEADROOM_TELEMETRY"] = "off"
os.environ["HEADROOM_PROTECT_READS"] = "1"

from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
from headroom.tokenizers.estimator import EstimatingTokenCounter

sed = Path("C:/Program Files/Git/usr/bin/sed.exe")
source = "\n\n".join(
    f"def compute_{i}(value):\n"
    '    """Compute a checked result for a caller."""\n'
    "    result = value * 2\n"
    "    if result < 0:\n"
    '        raise ValueError("negative result")\n'
    "    return result\n"
    for i in range(45)
)
with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "sample.py"
    path.write_text(source, encoding="utf-8", newline="\n")
    for flags in [["-n"], ["-ne"], ["--quiet"], ["--silent"], ["-En"]]:
        output = subprocess.check_output(
            [str(sed), *flags, "1,400p", str(path)], text=True, encoding="utf-8"
        )
        command = f"sed {' '.join(flags)} '1,400p' {path.as_posix()}"
        router = ContentRouter(ContentRouterConfig(
            enable_code_aware=False, enable_kompress=False,
            protect_analysis_context=False, protect_recent_reads_fraction=0.0,
        ))
        messages = [
            {"role": "user", "content": "Read the module."},
            {"role": "assistant", "content": None, "tool_calls": [{
                "id": "read-1", "type": "function", "function": {
                    "name": "Bash", "arguments": json.dumps({"command": command})
                }
            }]},
            {"role": "tool", "tool_call_id": "read-1", "content": output},
        ]
        result = router.apply(messages, EstimatingTokenCounter(), protect_recent=0)
        actual = result.messages[2]["content"]
        print(json.dumps({
            "flags": flags, "input_chars": len(output), "output_chars": len(actual),
            "byte_exact": output == actual, "transforms": result.transforms_applied,
        }), flush=True)
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: sed read classification; byte-exact
routing changes when `HEADROOM_PROTECT_READS` is enabled (default
remains unchanged).
- Kill switch / disable path: `HEADROOM_PROTECT_READS=0` disables read
protection.
- Unsafe override required: no.
- Qualification impact: equivalent quiet reads now retain their original
output; no savings or task-quality measurement is claimed.
- Rollback path: revert this patch; no persisted-state migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing focused unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

I implemented and verified this with Codex assistance for drafting and
local tests.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Yuchuan He
2026-11-03 17:10:38 -06:00
committed by GitHub
co-authored by JerrettDavis
parent 65bcc4c313
commit 312ffec8ea
2 changed files with 177 additions and 3 deletions
+73 -2
View File
@@ -35,6 +35,7 @@ Pipeline Usage:
from __future__ import annotations
import asyncio
import getopt
import hashlib
import itertools
import json
@@ -42,6 +43,7 @@ import logging
import math
import os
import re
import shlex
import sys
import threading
import time
@@ -907,8 +909,8 @@ def _segment_is_read(c: str) -> bool:
return _is_read_command(" ".join(rest[j + 1 :]).strip("'\""))
return False
is_read = prog in _READ_VERBS or (
# `sed -n '1,20p' file` prints a range (read); bare `sed` is a stream editor.
prog == "sed" and bool(re.search(r"(^|\s)-n(\s|$)", c))
# Quiet mode prints selected file contents; bare sed is a stream editor.
prog == "sed" and _sed_is_quiet_read(rest)
)
if not is_read:
return False
@@ -916,6 +918,75 @@ def _segment_is_read(c: str) -> bool:
return not _LOCKFILE_RE.search(c)
def _strip_shell_comment(command: str) -> str:
"""Strip a comment at an unquoted word boundary, retaining literal hashes."""
quote = ""
escaped = False
word_start = True
for i, char in enumerate(command):
if escaped:
escaped = False
continue
if char == "\\" and quote != "'":
escaped = True
word_start = False
elif quote:
if char == quote:
quote = ""
elif char in {"'", '"'}:
quote = char
word_start = False
elif char == "#" and word_start:
return command[:i]
else:
word_start = char.isspace()
return command
def _sed_is_quiet_read(rest: list[str]) -> bool:
"""Recognize quiet sed options without treating script arguments as flags."""
try:
# _bash_program retains shell quotes in its whitespace-split arguments.
# Reconstruct them before parsing options, including grouped flags (-ne).
args = shlex.split(_strip_shell_comment(" ".join(rest)))
except ValueError:
return False
# GNU -l takes a line length; BSD -l is a line-buffering flag. Recognize
# either dialect so existing BSD reads do not lose their protection.
is_quiet = False
for short_options in ("anErsuzbiIe:f:l:", "anErsuzbliIe:f:"):
try:
options, _ = getopt.gnu_getopt(
args,
short_options,
[
"quiet",
"silent",
"expression=",
"file=",
"line-length=",
"regexp-extended",
"separate",
"unbuffered",
"null-data",
"binary",
"posix",
"sandbox",
"debug",
"follow-symlinks",
"in-place",
],
)
except getopt.GetoptError:
continue
# Check both dialects: GNU -l may consume a BSD -n or in-place flag.
# Any in-place interpretation takes precedence over quiet mode.
if any(opt in {"-i", "-I", "--in-place"} for opt, _ in options):
return False
is_quiet |= any(opt in {"-n", "--quiet", "--silent"} for opt, _ in options)
return is_quiet
# Shell wrappers that prefix the real program — peeled to find it. Shell
# grammar, not tunable policy: rtk (the user's token proxy), sudo/env/timeout/…
_SHELL_WRAPPERS = frozenset(
@@ -7,9 +7,16 @@ the first program was inspected, so `wc -l a.py && sed -n '1,60p' a.py` was
classified as a `wc` run and the file content was word-dropped by Kompress.
"""
import json
import pytest
from headroom.transforms.content_router import _is_read_command
from headroom.tokenizers.estimator import EstimatingTokenCounter
from headroom.transforms.content_router import (
ContentRouter,
ContentRouterConfig,
_is_read_command,
)
@pytest.mark.parametrize(
@@ -19,6 +26,19 @@ from headroom.transforms.content_router import _is_read_command
"head -80 foo.py",
"tail -n 50 foo.py",
"sed -n '1,60p' foo.py",
"sed -ne '1,60p' foo.py",
"sed -En '1,60p' foo.py",
"sed --quiet '1,60p' foo.py",
"sed --silent '1,60p' foo.py",
"sed --quiet --expression='1,60p' foo.py",
"timeout 30 sed -ne '1,60p' foo.py",
"sed -n --follow-symlinks '1,60p' foo.py",
"sed -a -n '1,60p' foo.py", # BSD sed.
"sed -n -e '1,60p' foo.py -l", # BSD line buffering (no argument).
"sed -l -n '1,60p' foo.py",
"sed -ln '1,60p' foo.py",
"sed -l 80 -n '1,60p' foo.py", # GNU line length takes an argument.
"sed -nl80 '1,60p' foo.py",
"cd /repo && cat foo.py",
"sudo cat foo.py",
"rtk cat foo.py",
@@ -75,6 +95,19 @@ def test_chained_reads_are_detected(command: str) -> None:
"cat foo.py > copy.py && echo done",
"sed -n '1,60p' foo.py >> out.txt",
"sed 's/a/b/' foo.py",
"sed -n1,60p foo.py", # Invalid: a script cannot be attached to -n.
"sed -ni '1,60p' foo.py",
"sed -n -i '1,60p' foo.py",
"sed -n -I .bak '1,60p' foo.py", # BSD in-place editing.
"sed -n -I '' '1,60p' foo.py",
"sed -n -l -i '' '1,60p' foo.py", # BSD -l must not hide in-place flags.
"sed -n -l -I .bak '1,60p' foo.py",
"sed -in '1,60p' foo.py",
"sed --quiet --in-place '1,60p' foo.py",
"sed --quiet --in-place=.bak '1,60p' foo.py",
"sed -e 's/a/ -n /' foo.py", # Script text is not a quiet option.
"sed -e ' -n ' foo.py",
"sed -- -n foo.py", # Option parsing ends at --.
# Lockfiles are regenerated, never byte-patched.
"cat uv.lock",
# `-n` from a sibling segment must not qualify a bare `sed`.
@@ -95,3 +128,73 @@ def test_heredoc_body_is_not_split_into_a_bogus_read() -> None:
read — the command as a whole writes a file."""
command = "cat > foo.py <<EOF\nprint(1); head -5 bar.py\nEOF"
assert _is_read_command(command) is False
@pytest.mark.parametrize("flag", ["-n", "-ne", "-En", "--quiet", "--silent"])
@pytest.mark.parametrize("comment", ["", " # avoid -i here", " # --unknown", " # author's note"])
def test_quiet_sed_output_reaches_read_protection(flag: str, comment: str, monkeypatch) -> None:
monkeypatch.setenv("HEADROOM_PROTECT_READS", "1")
monkeypatch.setenv("HEADROOM_SHELL_RESULT_MIN_CHARS", "2000")
# Sixty source lines exceed the small-shell shortcut and reach read protection.
content = "".join(
f"def compute_{i}(value):\n"
" return value * 2 # Preserve this source exactly for a later patch.\n"
for i in range(30)
)
assert len(content) >= 2000
messages = [
{"role": "user", "content": "Read the module."},
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "read-1",
"type": "function",
"function": {
"name": "Bash",
"arguments": json.dumps({"command": f"sed {flag} '1,60p' foo.py{comment}"}),
},
}
],
},
{"role": "tool", "tool_call_id": "read-1", "content": content},
]
router = ContentRouter(
ContentRouterConfig(
enable_code_aware=False,
enable_kompress=False,
protect_analysis_context=False,
protect_recent_reads_fraction=0.0,
)
)
result = router.apply(messages, EstimatingTokenCounter(), protect_recent=0)
assert "router:shell_small_result_verbatim" not in result.transforms_applied
assert "router:read_protected" in result.transforms_applied
assert result.messages[2]["content"] == content
@pytest.mark.parametrize(
("command", "expected"),
[
("sed -n '1,60p' foo.py # avoid -i here", True),
("sed -n '1,60p' foo.py # --unknown", True),
("sed -n '1,60p' foo.py # author's note", True),
("sed -n '1,60p' foo.py # unclosed \"quote", True),
("sed -n '1,60p' 'foo#bar.py'", True),
('sed -n "1,60p" "foo#bar.py"', True),
("sed -n '1,60p' foo#bar.py", True),
(r"sed -n '1,60p' foo\#bar.py", True),
("sed -n '/# marker/p' foo.py", True),
# Literal hashes must not hide real in-place options that follow them.
("sed -n '1,60p' foo#bar.py -i", False),
("sed -n '1,60p' 'foo#bar.py' -i", False),
(r"sed -n '1,60p' foo\#bar.py -i", False),
(r"sed -n '1,60p' foo\ #bar.py -i", False),
("sed -n '/# marker/p' foo.py -i", False),
("sed -n '1,60p' foo.py -i # --quiet", False),
("sed -n 'unterminated # quote", False),
],
)
def test_sed_shell_comments_and_literal_hashes(command: str, expected: bool) -> None:
assert _is_read_command(command) is expected