mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description `benchmarks/codemode_ab.py` runs `git reset --hard -q` and `git clean -qfd` in `--worktree` after any task that leaves `git status --porcelain` non-empty (`main()`, lines 319-324 on `1da8930a`). The module docstring assumes a clean detached worktree, but nothing checks that `--worktree` is a git work tree or was clean before the first run. If it holds uncommitted work, that work is deleted after the first task. With this PR `main()` checks once before the loop: if `--worktree` is not inside a git work tree, or `git status --porcelain` is not empty, it prints a message on stderr and returns exit code 2. The per-run reset is unchanged. Fixes #4731 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `benchmarks/codemode_ab.py`: after `wt = Path(args.worktree).resolve()`, run `git status --porcelain` in `wt`. A non-zero exit gives `error: --worktree <path> is not inside a git work tree`; non-empty output gives an error naming the reset/clean commands and suggesting commit, stash or a clean detached worktree. Both return 2 before any `claude` call. - `tests/test_codemode_ab_worktree_guard.py`: new regression tests for the dirty, non-git and clean cases (follow-up commit). ## Testing - [x] Unit tests pass (`pytest`, new file only) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `tests/test_codemode_ab_worktree_guard.py` (new) calls `main()` on a tmp git repo with `run_one` stubbed, so the `claude` CLI is never reached: dirty tree (modified tracked file plus untracked file) must return 2 with the files intact; a plain directory must return 2 with the not-a-git-work-tree message; a clean repo must return 0. Only that file was run, not the full suite; `ruff format` and `ruff check` on it are clean; `mypy` was not run. The script was also checked end to end against a stub `claude` (below). pytest, Ubuntu on WSL2, Python 3.12, `-p no:cacheprovider`. Pass after (this branch): ```text collected 3 items tests/test_codemode_ab_worktree_guard.py ... [100%] ============================== 3 passed in 21.33s ============================== ``` Fail before (same test file, `benchmarks/codemode_ab.py` from `origin/main` `1da8930a`): ```text E assert 0 == 2 E assert 0 == 2 FAILED tests/test_codemode_ab_worktree_guard.py::test_dirty_worktree_is_refused_and_left_intact FAILED tests/test_codemode_ab_worktree_guard.py::test_non_git_directory_is_refused ========================= 2 failed, 1 passed in 17.63s ========================= ``` On `origin/main` the dirty and non-git cases return 0 instead of 2 because there is no guard (the test stops at that first assertion; the reset of the dirty tree is shown by the stub run above). The clean case passes on both, as it should. ### Test Output Same stub `claude` (prints a fixed JSON result, touches nothing) and same fixture on both versions of the script: a git repo with one commit, then ` M tracked.txt` and an untracked `notes.md`. Ubuntu on WSL2, Python 3, `python3 -I`, `HOME` pointed at an empty dir. Fail before (`benchmarks/codemode_ab.py` from `origin/main` `1da8930a`): ```text before: [ M tracked.txt ?? notes.md ] ccr_py_count steered rep0 $0.0000 turns=1 calls=None ok=True !! worktree dirtied, resetting: M tracked.txt ?? notes.md after: status=[] tracked=[committed ] notes.md=NO ``` Pass after (this branch): ```text before: [ M tracked.txt ?? notes.md ] error: --worktree /var/tmp/cb-u16-ab/wt-branch has uncommitted changes; the script runs `git reset --hard` and `git clean -fd` there after each run. Commit or stash first, or use a clean detached worktree. exit=2 after: status=[ M tracked.txt ?? notes.md ] tracked=[committed MY EDIT ] notes.md=yes ``` Not a git directory (this branch): `error: --worktree /var/tmp/cb-u16-ab/nogit is not inside a git work tree`, exit 2. Clean fixture (this branch), full run with the stub: `exit=0`, last line `wrote .../out/r.json`, 16 records written. ## Real Behavior Proof - Environment: Ubuntu on WSL2, Python 3, git, stub `claude` first on `PATH`, `HOME` redirected to an empty directory. No real `claude` run and no model call. - Exact command / steps: `python3 -I -u benchmarks/codemode_ab.py --worktree <fixture> --plugin-dir <empty dir> --out <tmp>/r.json`, once per script version and fixture. - Observed result: on `1da8930a` the dirty fixture ended with an empty status, `tracked.txt` back to `committed` and `notes.md` deleted; on this branch the script exited 2 with the message above and both files were untouched. On a clean fixture this branch still completed all runs with exit 0. - Not tested: a real `claude -p` run; a `--worktree` path that does not exist (the `git` call would raise `FileNotFoundError`); worktrees made with `git worktree add`. ## Runtime Rollout Safety - Rollout-managed feature(s): none; benchmark script only. - Minimum rollout channel: n/a - Stable/default behavior changed: no, for the library or proxy. The script now exits 2 instead of resetting a dirty `--worktree`. - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes (only the new test file was run; see Testing) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Known limit: a worktree edited while a benchmark is running is still reset by the existing per-run cleanup. The check runs once, before the first run. - A user who wants to reuse a dirty tree now has to commit or stash first. I kept the check to the smallest form; no `--force` flag was added. Produced with AI assistance (Claude Code) operated by me (HetCreep); the stub-claude runs above were done on my machine against a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
95 lines
2.8 KiB
Python
95 lines
2.8 KiB
Python
"""codemode_ab.main() must refuse a --worktree it would wipe with reset/clean."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from benchmarks import codemode_ab
|
|
|
|
|
|
def _git(repo: Path, *args: str) -> None:
|
|
subprocess.run(
|
|
["git", "-c", "user.name=t", "-c", "user.email=t@example.com", *args],
|
|
cwd=repo,
|
|
check=True,
|
|
capture_output=True,
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def run_main(tmp_path, monkeypatch, capsys):
|
|
# Stand-in for the real `claude` run so no test can reach the CLI.
|
|
task = ("t", "prompt", None)
|
|
monkeypatch.setattr(codemode_ab, "TASKS", [task])
|
|
monkeypatch.setattr(codemode_ab, "run_one", lambda *a, **k: {"cost_usd": 0, "num_turns": 1})
|
|
|
|
def run(worktree: Path):
|
|
argv = [
|
|
"codemode_ab.py",
|
|
"--worktree",
|
|
str(worktree),
|
|
"--plugin-dir",
|
|
str(tmp_path),
|
|
"--out",
|
|
str(tmp_path / "out" / "r.json"),
|
|
]
|
|
monkeypatch.setattr(sys, "argv", argv)
|
|
rc = codemode_ab.main()
|
|
return rc, capsys.readouterr().err
|
|
|
|
return run
|
|
|
|
|
|
@pytest.fixture
|
|
def repo(tmp_path):
|
|
r = tmp_path / "repo"
|
|
r.mkdir()
|
|
_git(r, "init", "-q")
|
|
(r / "tracked.txt").write_text("committed")
|
|
_git(r, "add", ".")
|
|
_git(r, "commit", "-q", "-m", "init")
|
|
return r
|
|
|
|
|
|
def test_dirty_worktree_is_refused_and_left_intact(repo, run_main):
|
|
(repo / "tracked.txt").write_text("my edit")
|
|
(repo / "notes.md").write_text("untracked work")
|
|
rc, err = run_main(repo)
|
|
assert rc == 2
|
|
assert "uncommitted changes" in err
|
|
assert (repo / "tracked.txt").read_text() == "my edit"
|
|
assert (repo / "notes.md").read_text() == "untracked work"
|
|
|
|
|
|
def test_untracked_work_is_refused_even_when_git_hides_untracked_files(repo, run_main, monkeypatch):
|
|
# status.showUntrackedFiles=no makes a bare `git status --porcelain` blind to
|
|
# untracked files, while the cleanup's `git clean -fd` still deletes them.
|
|
_git(repo, "config", "status.showUntrackedFiles", "no")
|
|
(repo / "precious-notes.txt").write_text("untracked work")
|
|
|
|
def never_called(*a, **k):
|
|
raise AssertionError("the benchmark must not start on a worktree with untracked work")
|
|
|
|
monkeypatch.setattr(codemode_ab, "run_one", never_called)
|
|
rc, err = run_main(repo)
|
|
assert rc == 2
|
|
assert "uncommitted changes" in err
|
|
assert (repo / "precious-notes.txt").read_text() == "untracked work"
|
|
|
|
|
|
def test_non_git_directory_is_refused(tmp_path, run_main):
|
|
plain = tmp_path / "plain"
|
|
plain.mkdir()
|
|
rc, err = run_main(plain)
|
|
assert rc == 2
|
|
assert "not inside a git work tree" in err
|
|
|
|
|
|
def test_clean_worktree_passes_the_guard(repo, run_main):
|
|
rc, _ = run_main(repo)
|
|
assert rc == 0
|