fix: protect file reads in chained shell commands (#2668)

## Description

`_is_read_command` decides whether a shell command's output is file
content the agent will patch from. Output classified as a read is passed
verbatim; everything else is eligible for lossy Kompress compression.

It inspected only the **first program**, after peeling `cd`/`sudo`/`rtk`
wrappers. Agents routinely batch a read behind other work, so every
chained read fell through to the compression path despite
`HEADROOM_PROTECT_READS=1`:

```
wc -l a.py b.py && sed -n '1,60p' c.py     # parsed as `wc`   -> not a read
grep -n X f.py | head -40; head -80 g.py   # parsed as `grep` -> not a read
ls -la || cat f.py                         # parsed as `ls`   -> not a read
```

Kompress operates on words and rejoins the survivors, so file content
arrives with function words missing. That is precisely the failure read
protection exists to prevent: the agent either re-reads to recover
detail (turn inflation) or builds an exact-match `old_string` from text
that no longer matches the file on disk.

Reported by a Headroom desktop user who saw source content garbled while
the `Read` tool path stayed byte-exact — which is what pointed at the
shell-command classifier rather than the tool exclusion list.

No linked issue; reported directly.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_is_read_command` now scans every `;` / `&&` / `||` segment and
protects if **any** segment is a read, instead of matching only the
first program.
- Extracted the existing single-command logic into `_segment_is_read`,
unchanged in behavior for a non-chained command.
- Added `_command_segments`, which splits on `;` / `&&` / `||` and
reduces each segment to its first pipeline stage.
- Pipelines are deliberately **not** split: a downstream stage consumes
the previous stage's output rather than a file, so `grep … | head -40`
stays compressible while `cat f.py | head -40` is still caught via its
first stage.
- The whole-string write/heredoc guard is unchanged, so a heredoc body
containing `;` cannot be split into a bogus read segment.
- Two latent false positives close as a consequence of per-segment
evaluation: a `-n` flag in a sibling segment no longer qualifies a bare
`sed`, and the lockfile carve-out now applies to the segment naming the
lockfile rather than the whole command.
- New test file `tests/test_transforms/test_read_command_detection.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`) — see Additional Notes
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_transforms/ \
    tests/test_observed_wire_shapes.py tests/test_content_router_exclude_tools.py -q
======================= 408 passed, 62 skipped in 13.46s =======================

$ uv run --frozen --extra dev pytest tests/test_transforms/test_read_command_detection.py -q \
    --cov=headroom.transforms.content_router --cov-report=term-missing --cov-branch
30 passed in 0.45s
# no uncovered lines or partial branches remain in the changed range (614-654)

$ uvx ruff check headroom tests --output-format concise
All checks passed!

$ uvx ruff format --check headroom/transforms/content_router.py \
    tests/test_transforms/test_read_command_detection.py
2 files already formatted

$ uv run --frozen --extra dev mypy headroom
Success: no issues found in 512 source files
```

## Real Behavior Proof

- Environment: macOS (Darwin 24.6.0), Python 3.10.18, worktree branched
from `upstream/main` @ `28aa53dc`; plus a live Headroom desktop proxy
running `headroom-ai` 0.32.1 with Claude Code pointed at
`127.0.0.1:6767`.
- Exact command / steps: (1) pre-fix on the live proxy, ran `grep -n
"Read" transforms/compression_policy.py | head -40; echo "=== POLICY
HEAD ==="; head -80 transforms/compression_policy.py` through Claude
Code's Bash tool; (2) ran `ContentRouter().apply(...)` with
`HEADROOM_PROTECT_READS=1` over a `role: tool` bash observation produced
by `wc -l app/services/inventory.py && sed -n '1,200p'
app/services/inventory.py` (11,007 chars of plain Python source, aged
behind 20 follow-up turns so the positional "recent code" guard no
longer covers it), against both the pre-fix and post-fix
`content_router.py`.
- Observed result: step 1 returned the file's own docstring with
function words dropped and a retrieval marker appended; step 2 went from
no protection transform at all (pre-fix — the observation enters the
compression path) to `router:read_protected` (post-fix). Verbatim output
below.
- Not tested: byte-level Kompress corruption inside the repo venv (the
ML model is not downloaded there — `Kompress model not ready; requests
will not be compressed` — so step 2 shows the routing change and step 1
shows the corruption); `magika` unavailable in that venv (`no pip
onnxruntime native library was found`) so detection fell through to the
regex tier during step 2, which affects which compressor a released read
routes to but not whether a read is protected; Windows shell command
strings; fleet-scale traffic replay, so the savings impact of
newly-protected reads is unquantified.

### Proof detail

Step 1 — marker appended to the returned file content:

```
[478 items compressed to 305. Retrieve more: hash=5d959a193f8641579c741ad5]
```

On disk:

> Hand-mirrored port **of**
`headroom_core::compression_policy::CompressionPolicy` (Rust). **The**
Rust crate **is the** source **of** truth; **this** module exists **so
the** Python proxy's `TransformPipeline` (which still runs
`CacheAligner` **and other** detector-only transforms) **can** read
**the** same per-mode flags **the** Rust dispatcher does.

As received by the model:

> Hand-mirrored port
`headroom_core::compression_policy::CompressionPolicy` (Rust). Rust
crate source truth; module exists Python proxy's `TransformPipeline`
(which still runs `CacheAligner` detector-only transforms) read same
per-mode flags Rust dispatcher does.

Step 2 — routing branch taken by the tool observation:

```text
##### BEFORE (upstream/main content_router.py) #####
command:               wc -l app/services/inventory.py && sed -n '1,200p' app/services/inventory.py
input chars:           11007
file content verbatim: True
transforms:            ['router:protected:user_message', ... x10]      <- no read protection

##### AFTER (this PR) #####
command:               wc -l app/services/inventory.py && sed -n '1,200p' app/services/inventory.py
input chars:           11007
file content verbatim: True
transforms:            ['router:read_protected', 'router:protected:user_message', ... x10]
```

Before the fix the observation carries no protection transform at all
and enters the compression path. After the fix it is
`router:read_protected`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A, no
user-facing docs describe this internal classifier; its docstrings are
updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — no UI surface.

## Additional Notes

- `ruff check .` reports 4 errors, all pre-existing on `upstream/main`
and all in `plugins/headroom-oauth2/` (`RUF022`, `BLE001` x2, `S110`).
Untouched by this PR, so the box is left unchecked rather than claimed.
`ruff check headroom tests` is clean.
- Trade-off: the write/heredoc guard stays whole-string rather than
per-segment. That keeps `cat a.py && echo x > b` unprotected, which is
the pre-existing behavior — chosen so a heredoc body containing `;` can
never be split into a segment that looks like a read. Narrowing it
further is a separate change.
- Follow-up for the reporter's second issue (not in scope here): passive
traffic learning has no off switch, and the desktop starts the proxy
with `--learn` unconditionally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-10-01 16:29:01 -05:00
committed by GitHub
co-authored by Claude Opus 5 JerrettDavis
parent dcec845851
commit ffc359976b
2 changed files with 140 additions and 3 deletions
+43 -3
View File
@@ -741,14 +741,54 @@ def _is_read_command(command: str) -> bool:
Excludes writes: a redirect (``>``/``>>``), ``tee``, or heredoc (``<<``) means the
command WRITES a file (e.g. ``cat > f <<EOF``), and a bare ``sed`` (without ``-n``)
is a stream edit — neither is a read.
is a stream edit — neither is a read. Redirects and ``tee`` are judged PER SEGMENT,
so a sibling write (``cat a.py && echo done > marker``) does not strip protection
from the read segment next to it.
CHAINED commands are scanned segment by segment: agents routinely batch a read
with other work (``wc -l a.py && sed -n '1,60p' a.py``, ``grep … ; head -80 b.py``),
and matching only the FIRST program left every such read unprotected — the file
content was lossy-compressed despite read protection being enabled.
"""
if not command or not isinstance(command, str):
return False
# strip leading `cd <dir> && ` chains (agents prefix reads with a cd)
c = _strip_cd_prefix(command)
# a write / append / tee / heredoc anywhere => not a pure read
if re.search(r"(^|\s)(>>?|tee\b|<<)", c):
# A heredoc is the one WHOLE-STRING bailout: its body can contain `;`/`&&`,
# which would split into bogus segments that look like reads. Redirects and
# `tee` are per-segment (see `_segment_is_read`) — a sibling write must not
# unprotect the read next to it.
if re.search(r"(^|\s)<<", c):
return False
return any(_segment_is_read(seg) for seg in _command_segments(c))
# Separators that start a NEW command. A pipeline (`|`) deliberately does NOT:
# downstream stages consume the previous stage's output, not a file, so
# `grep -n x a.py | head -40` is derived search output (compressible), while
# `cat a.py | head -40` is a file read (caught via the first stage).
_CMD_SEPARATOR_RE = re.compile(r"\|\||&&|;")
def _command_segments(command: str) -> list[str]:
"""Split a shell string into the independently-executed commands in it.
Splits on ``;``, ``&&`` and ``||`` — never on a single ``|``. Pipelines are kept
whole so the segment's own writes (``cat a.py | tee b.py``) stay visible; the
reduction to the first stage happens in :func:`_segment_is_read`.
"""
return [seg.strip() for seg in _CMD_SEPARATOR_RE.split(command) if seg.strip()]
def _segment_is_read(c: str) -> bool:
""":func:`_is_read_command` for ONE command (no ``;``/``&&``/``||`` chaining)."""
# A redirect or `tee` in THIS segment means the segment writes a file rather than
# reading one (`cat a.py > copy.py`, `cat a.py | tee copy.py`). Checked on the whole
# segment — pipeline included — before reducing to the stage that touches the file.
if re.search(r"(^|\s)(>>?|tee\b)", c):
return False
c = c.split("|", 1)[0].strip()
if not c:
return False
# Parse the real program with the SAME structural parser the search-fold uses
# (_bash_program peels sudo/env/timeout/rtk wrappers + env assignments), so
@@ -0,0 +1,97 @@
"""Tests for `_is_read_command` — which shell commands count as file reads.
Read detection drives read protection: a command classified as a file read has
its output passed verbatim instead of lossy-compressed, because the agent needs
exact bytes to build a patch from. Chained commands used to fall through: only
the first program was inspected, so `wc -l a.py && sed -n '1,60p' a.py` was
classified as a `wc` run and the file content was word-dropped by Kompress.
"""
import pytest
from headroom.transforms.content_router import _is_read_command
@pytest.mark.parametrize(
"command",
[
"cat foo.py",
"head -80 foo.py",
"tail -n 50 foo.py",
"sed -n '1,60p' foo.py",
"cd /repo && cat foo.py",
"sudo cat foo.py",
"rtk cat foo.py",
'bash -lc "cat foo.py"',
# The `-c` argument is not the first token after the shell.
'bash --norc -lc "cat foo.py"',
],
)
def test_simple_reads_are_detected(command: str) -> None:
assert _is_read_command(command) is True
@pytest.mark.parametrize(
"command",
[
# The reported case: a read batched behind an unrelated first program.
"wc -l a.py b.py && sed -n '1,60p' c.py",
"grep -n X foo.py | head -40; head -80 bar.py",
"ls -la || cat foo.py",
"echo start; cat foo.py",
# Bare `sed` is a stream edit, but the `cat` segment is still a read.
"sed 's/a/b/' foo.py && cat -n bar.py",
# A read in the first segment survives a non-read tail.
"cat foo.py && pytest -q",
# Trailing and doubled separators produce empty segments, which are skipped.
"cat foo.py;",
"ls -la; ; cat foo.py",
# A write in a SIBLING segment does not unprotect the read segment: the
# output still carries the exact bytes of foo.py.
"cat foo.py && echo done > marker",
"echo done > marker && cat foo.py",
"sed -n '1,60p' foo.py; touch out >> log",
],
)
def test_chained_reads_are_detected(command: str) -> None:
assert _is_read_command(command) is True
@pytest.mark.parametrize(
"command",
[
# Derived output — stays compressible.
"grep -n x foo.py",
"pytest -q",
"ls -la",
# A pipeline stage consumes the previous stage's output, not a file:
# `head` here truncates grep output, it does not read bar.py.
"grep -n x foo.py | head -40",
# Writes are never reads.
"cat > foo.py",
# The write lives in the read segment itself — a pipeline is one segment,
# so `tee` is still visible after the split.
"cat foo.py | tee bar.py",
"cat foo.py > copy.py && echo done",
"sed -n '1,60p' foo.py >> out.txt",
"sed 's/a/b/' foo.py",
# Lockfiles are regenerated, never byte-patched.
"cat uv.lock",
# `-n` from a sibling segment must not qualify a bare `sed`.
"sed 's/a/b/' foo.py && ls -n",
# A segment that is only an env assignment resolves to no program.
"FOO=1",
# A shell invocation with no `-c` argument runs a script, not a read.
"bash --norc",
"bash script.sh",
],
)
def test_non_reads_are_not_detected(command: str) -> None:
assert _is_read_command(command) is False
def test_heredoc_body_is_not_split_into_a_bogus_read() -> None:
"""A `;` inside a heredoc body must not produce a segment that looks like a
read — the command as a whole writes a file."""
command = "cat > foo.py <<EOF\nprint(1); head -5 bar.py\nEOF"
assert _is_read_command(command) is False