mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8cdce172ea |
fix(router): ignore shell operators inside quoted arguments in read detection (#4646)
## Description Item 3 of #4478, the quoted-pattern case that #4521 left open. Read detection scanned the raw command text for shell syntax without regard to quoting: `_segment_is_read` treated any `>` as a redirect (`content_router.py:893` on `e4a60ed7`) and split the pipeline at the first `|`, and `_command_segments` split on `;`, `&&` and `||`. A quiet sed whose script holds one of these characters, such as `sed -n '/if x > 0/,/return/p' a.py` or the GNU alternation `sed -n '/def \|class /p' a.py`, was therefore not detected as a file read and lost read protection. The three scans now run on a copy of the command in which quoted, backslash-escaped and commented characters are masked, and the match positions are mapped back onto the original text. Two shell rules keep word boundaries intact (review round 2): a backslash-newline is a line continuation and is masked as two spaces, so a redirect or `tee` on the next line still counts as a write; and an unquoted `#` at the start of a word masks the rest of its line without changing quote state, so a quote mark inside a comment cannot hide the next line's separators. Unquoted operators behave as before. Refs #4478 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_router.py`: new `_mask_quoted(command)`, which returns a same-length string with characters inside `'...'` / `"..."`, backslash-escaped characters and comment text replaced by `_` (the quote characters are kept; a backslash-newline becomes two spaces; a comment's newline stays visible). `_segment_is_read` runs its redirect/`tee` check and its pipeline split on the masked string; `_command_segments` finds `;` / `&&` / `||` on the masked string and slices the original. - `tests/test_transforms/test_read_command_detection.py`: `test_quoted_shell_operators_are_arguments`, 25 cases. Quoted `>`, `>>`, `|`, `;`, `&&` and `tee` inside a sed script or a file name are reads; a real redirect, pipe-to-`tee` or separator next to a quoted script still counts; `bash -lc "cat foo.py > copy.py"` is still not a read (the inner command is checked on its own, as before). Round 2 adds: continued `>`, `>>` and `| tee` are writes; a comment with an unmatched `"` or `'` followed by `ls; cat foo.py` / `true; cat foo.py` on the next line is a read; `cat foo.py # > not a redirect` is a read; a `#` inside a word is not a comment. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) (run on `headroom/transforms/content_router.py` only, see below) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # fail before (source at |
||
|
|
312ffec8ea |
fix(router): preserve read protection for quiet sed options (#4521)
## Description With `HEADROOM_PROTECT_READS=1`, `sed -n '1,60p' file.py` achieves byte-exact read protection, but equivalent `-ne`, `-En`, `--quiet`, and `--silent` forms currently do not. The classifier previously checked for a literal `-n` token across the raw command string, which also caused it to mistake `-n` inside an expression argument as a quiet flag. This PR properly parses sed options before classifying whether a command is a read. It handles grouped flags and long options, respects script arguments and `--` delimiters, and excludes in-place edits (`-i`). GNU and BSD `-l` forms remain recognized. Addresses the quiet-option parsing portion of item 3 in #4478 (quoted redirect parsing and other items in that issue remain separate). Note that `sed -n1,80p` is invalid GNU sed syntax and remains unhandled. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Recognize quiet sed options using standard-library shell tokenization and robust option parsing. - Keep script arguments separate from options, and exclude in-place edits from protected reads. - Add command-classification unit tests and end-to-end `ContentRouter.apply()` regression coverage for all five quiet forms. ## Testing - [x] Unit tests pass (`pytest`): focused and related suites below - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`): native Linux pre-commit configuration - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Focused tests against unchanged main's router: 16 failed, 44 passed # This patch: 60 passed # Read detection, Responses read protection, read audit, maturation, lifecycle: 147 passed # make ci-precheck-rust (Rust 1.95.0): # cargo fmt --all -- --check; cargo clippy --workspace -- -D warnings; # cargo test --workspace 1659 passed, 3 ignored # make ci-precheck-python (including native extension rebuild): 177 passed, 4 skipped # ruff check . All checks passed! # ruff format --check . 1834 files already formatted ``` The native extension rebuild and import verification pass cleanly. The repository's unmodified pre-commit configuration passes on native Linux (WSL Ubuntu, Python 3.14.4), including mypy, Ruff, formatting, conflict checks, version checks, and commit-message hooks. On Windows, the default mypy hook reports seven POSIX-only `os` attribute errors in unchanged files (`fileperms.py` and `proxy/interceptors/astgrep.py`), matching unmodified main (#4520). Similarly, broader Windows test failures (18 failed, 5367 passed, 190 skipped, 2 errors) reproduce identically on main and stem from known platform differences (symlink privileges, SIGHUP handling, and Windows environment-variable size limits). ## Real Behavior Proof - Environment: Windows x86_64, Python 3.13.9, GNU sed 4.9, current-main Python sources with native `headroom._core` extension. ML and code compression are disabled to directly isolate the read-protection route. - Exact command / steps: Run the script below with main's router versus this patch. It executes real GNU sed against a temporary Python source file and passes each tool result through `ContentRouter.apply()`. - Observed result: The standard `-n` control is protected on both versions. On main, the other four forms miss `router:read_protected`. With this patch, all five forms correctly trigger read protection and preserve the 7,998-character output byte-for-byte. - Not tested: Focuses on command parsing and routing behavior. BSD option syntax is validated via parser unit tests; live proxy traffic and model calls were not required. Save as `repro_sed_quiet_reads.py`, then run `python repro_sed_quiet_reads.py`: ```python """Run real GNU sed output through ContentRouter, without a model or provider.""" import json import os from pathlib import Path import subprocess import tempfile os.environ["HEADROOM_OFFLINE"] = "1" os.environ["HEADROOM_TELEMETRY"] = "off" os.environ["HEADROOM_PROTECT_READS"] = "1" from headroom.transforms.content_router import ContentRouter, ContentRouterConfig from headroom.tokenizers.estimator import EstimatingTokenCounter sed = Path("C:/Program Files/Git/usr/bin/sed.exe") source = "\n\n".join( f"def compute_{i}(value):\n" ' """Compute a checked result for a caller."""\n' " result = value * 2\n" " if result < 0:\n" ' raise ValueError("negative result")\n' " return result\n" for i in range(45) ) with tempfile.TemporaryDirectory() as tmp: path = Path(tmp) / "sample.py" path.write_text(source, encoding="utf-8", newline="\n") for flags in [["-n"], ["-ne"], ["--quiet"], ["--silent"], ["-En"]]: output = subprocess.check_output( [str(sed), *flags, "1,400p", str(path)], text=True, encoding="utf-8" ) command = f"sed {' '.join(flags)} '1,400p' {path.as_posix()}" router = ContentRouter(ContentRouterConfig( enable_code_aware=False, enable_kompress=False, protect_analysis_context=False, protect_recent_reads_fraction=0.0, )) messages = [ {"role": "user", "content": "Read the module."}, {"role": "assistant", "content": None, "tool_calls": [{ "id": "read-1", "type": "function", "function": { "name": "Bash", "arguments": json.dumps({"command": command}) } }]}, {"role": "tool", "tool_call_id": "read-1", "content": output}, ] result = router.apply(messages, EstimatingTokenCounter(), protect_recent=0) actual = result.messages[2]["content"] print(json.dumps({ "flags": flags, "input_chars": len(output), "output_chars": len(actual), "byte_exact": output == actual, "transforms": result.transforms_applied, }), flush=True) ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: stable. - Stable/default behavior changed: sed read classification; byte-exact routing changes when `HEADROOM_PROTECT_READS` is enabled (default remains unchanged). - Kill switch / disable path: `HEADROOM_PROTECT_READS=0` disables read protection. - Unsafe override required: no. - Qualification impact: equivalent quiet reads now retain their original output; no savings or task-quality measurement is claimed. - Rollback path: revert this patch; no persisted-state migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing focused unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes I implemented and verified this with Codex assistance for drafting and local tests. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
ffc359976b |
fix: protect file reads in chained shell commands (#2668)
## Description
`_is_read_command` decides whether a shell command's output is file
content the agent will patch from. Output classified as a read is passed
verbatim; everything else is eligible for lossy Kompress compression.
It inspected only the **first program**, after peeling `cd`/`sudo`/`rtk`
wrappers. Agents routinely batch a read behind other work, so every
chained read fell through to the compression path despite
`HEADROOM_PROTECT_READS=1`:
```
wc -l a.py b.py && sed -n '1,60p' c.py # parsed as `wc` -> not a read
grep -n X f.py | head -40; head -80 g.py # parsed as `grep` -> not a read
ls -la || cat f.py # parsed as `ls` -> not a read
```
Kompress operates on words and rejoins the survivors, so file content
arrives with function words missing. That is precisely the failure read
protection exists to prevent: the agent either re-reads to recover
detail (turn inflation) or builds an exact-match `old_string` from text
that no longer matches the file on disk.
Reported by a Headroom desktop user who saw source content garbled while
the `Read` tool path stayed byte-exact — which is what pointed at the
shell-command classifier rather than the tool exclusion list.
No linked issue; reported directly.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `_is_read_command` now scans every `;` / `&&` / `||` segment and
protects if **any** segment is a read, instead of matching only the
first program.
- Extracted the existing single-command logic into `_segment_is_read`,
unchanged in behavior for a non-chained command.
- Added `_command_segments`, which splits on `;` / `&&` / `||` and
reduces each segment to its first pipeline stage.
- Pipelines are deliberately **not** split: a downstream stage consumes
the previous stage's output rather than a file, so `grep … | head -40`
stays compressible while `cat f.py | head -40` is still caught via its
first stage.
- The whole-string write/heredoc guard is unchanged, so a heredoc body
containing `;` cannot be split into a bogus read segment.
- Two latent false positives close as a consequence of per-segment
evaluation: a `-n` flag in a sibling segment no longer qualifies a bare
`sed`, and the lockfile carve-out now applies to the segment naming the
lockfile rather than the whole command.
- New test file `tests/test_transforms/test_read_command_detection.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`) — see Additional Notes
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_transforms/ \
tests/test_observed_wire_shapes.py tests/test_content_router_exclude_tools.py -q
======================= 408 passed, 62 skipped in 13.46s =======================
$ uv run --frozen --extra dev pytest tests/test_transforms/test_read_command_detection.py -q \
--cov=headroom.transforms.content_router --cov-report=term-missing --cov-branch
30 passed in 0.45s
# no uncovered lines or partial branches remain in the changed range (614-654)
$ uvx ruff check headroom tests --output-format concise
All checks passed!
$ uvx ruff format --check headroom/transforms/content_router.py \
tests/test_transforms/test_read_command_detection.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom
Success: no issues found in 512 source files
```
## Real Behavior Proof
- Environment: macOS (Darwin 24.6.0), Python 3.10.18, worktree branched
from `upstream/main` @ `28aa53dc`; plus a live Headroom desktop proxy
running `headroom-ai` 0.32.1 with Claude Code pointed at
`127.0.0.1:6767`.
- Exact command / steps: (1) pre-fix on the live proxy, ran `grep -n
"Read" transforms/compression_policy.py | head -40; echo "=== POLICY
HEAD ==="; head -80 transforms/compression_policy.py` through Claude
Code's Bash tool; (2) ran `ContentRouter().apply(...)` with
`HEADROOM_PROTECT_READS=1` over a `role: tool` bash observation produced
by `wc -l app/services/inventory.py && sed -n '1,200p'
app/services/inventory.py` (11,007 chars of plain Python source, aged
behind 20 follow-up turns so the positional "recent code" guard no
longer covers it), against both the pre-fix and post-fix
`content_router.py`.
- Observed result: step 1 returned the file's own docstring with
function words dropped and a retrieval marker appended; step 2 went from
no protection transform at all (pre-fix — the observation enters the
compression path) to `router:read_protected` (post-fix). Verbatim output
below.
- Not tested: byte-level Kompress corruption inside the repo venv (the
ML model is not downloaded there — `Kompress model not ready; requests
will not be compressed` — so step 2 shows the routing change and step 1
shows the corruption); `magika` unavailable in that venv (`no pip
onnxruntime native library was found`) so detection fell through to the
regex tier during step 2, which affects which compressor a released read
routes to but not whether a read is protected; Windows shell command
strings; fleet-scale traffic replay, so the savings impact of
newly-protected reads is unquantified.
### Proof detail
Step 1 — marker appended to the returned file content:
```
[478 items compressed to 305. Retrieve more: hash=5d959a193f8641579c741ad5]
```
On disk:
> Hand-mirrored port **of**
`headroom_core::compression_policy::CompressionPolicy` (Rust). **The**
Rust crate **is the** source **of** truth; **this** module exists **so
the** Python proxy's `TransformPipeline` (which still runs
`CacheAligner` **and other** detector-only transforms) **can** read
**the** same per-mode flags **the** Rust dispatcher does.
As received by the model:
> Hand-mirrored port
`headroom_core::compression_policy::CompressionPolicy` (Rust). Rust
crate source truth; module exists Python proxy's `TransformPipeline`
(which still runs `CacheAligner` detector-only transforms) read same
per-mode flags Rust dispatcher does.
Step 2 — routing branch taken by the tool observation:
```text
##### BEFORE (upstream/main content_router.py) #####
command: wc -l app/services/inventory.py && sed -n '1,200p' app/services/inventory.py
input chars: 11007
file content verbatim: True
transforms: ['router:protected:user_message', ... x10] <- no read protection
##### AFTER (this PR) #####
command: wc -l app/services/inventory.py && sed -n '1,200p' app/services/inventory.py
input chars: 11007
file content verbatim: True
transforms: ['router:read_protected', 'router:protected:user_message', ... x10]
```
Before the fix the observation carries no protection transform at all
and enters the compression path. After the fix it is
`router:read_protected`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A, no
user-facing docs describe this internal classifier; its docstrings are
updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — no UI surface.
## Additional Notes
- `ruff check .` reports 4 errors, all pre-existing on `upstream/main`
and all in `plugins/headroom-oauth2/` (`RUF022`, `BLE001` x2, `S110`).
Untouched by this PR, so the box is left unchecked rather than claimed.
`ruff check headroom tests` is clean.
- Trade-off: the write/heredoc guard stays whole-string rather than
per-segment. That keeps `cat a.py && echo x > b` unprotected, which is
the pre-existing behavior — chosen so a heredoc body containing `;` can
never be split into a segment that looks like a read. Narrowing it
further is a separate change.
- Follow-up for the reporter's second issue (not in scope here): passive
traffic learning has no off switch, and the desktop starts the proxy
with `--learn` unconditionally.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|