mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(memory): ignore leading cd prefix when pairing Bash error recoveries (#3776)
## Description `TrafficLearner` pairs unrelated Bash commands into `Learned: error recovery` rules when both commands start with the same `cd <dir> &&` prefix. Coding agents add that prefix to almost every Bash call, so most Bash recoveries written to `MEMORY.md` are bogus. Two examples written to my `MEMORY.md`: ``` - Command `cd "C:\...\repo" && grep -rn "shopping-vintage" ... -l .` fails (timeout). Use `cd "C:\...\repo" && cat components/places/categoryIcons.ts && ...` instead. - Command `cd "C:\...\repo" && npx vitest run "...SubmitPlaceForm.test.tsx" 2>&1 | tail -40` fails (runtime_error). Use `cd "C:\...\repo" && git add -A && git commit -F - <<'EOF' ...` instead. ``` The second rule tells the agent to commit instead of running the tests. Two defects compound: 1. **Matcher false positives.** `_commands_related_as_retry` requires the same first binary, plus either ≤40% edit distance or a shared substantive token. When both commands share a `cd <dir> &&` prefix, the first binary is always `cd` and the directory path is always a shared token of ≥5 characters, so any two such commands pass. 2. **Hash collapse inflates evidence.** `_normalize_bash_for_hash` cuts at the first `&&`, so every cd-prefixed pair hashes to `error_recovery|Bash|cd <dir>|cd <dir>`. Unrelated pairs in one project merge into a single row whose evidence is the sum of all of them. That is how they get past the `min_evidence` gate. This is a follow-up to #326, which introduced the retry matcher and the evidence gate. I found no existing issue or PR for this. Closes # (no existing issue) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `_strip_leading_cd()` in `headroom/memory/traffic_learner.py`. It removes leading `cd <dir> &&` / `cd <dir>;` segments, including quoted paths, backslash-escaped spaces (`cd /home/dev/my\ project && ...`) and repeated segments. A bare `cd <dir>` with no chain is left unchanged. - Apply it in `_commands_related_as_retry` before the binary, edit-distance and token-overlap checks. - Apply it in `_normalize_bash_for_hash`, so the hash key reflects the command that actually ran. - Tests in `tests/test_memory/test_traffic_learner.py`: - `test_shared_cd_prefix_alone_is_not_retry` and `test_strips_leading_cd` fail on `main` and pass with this change. - `test_cd_prefixed_retry_still_matches` guards against rejecting real retries. - **Changed expectation:** `test_cuts_at_first_chain` previously asserted `_normalize_bash_for_hash("cd /tmp && ls") == "cd /tmp"`. That assertion encoded the bug. It now uses `make build && ls`, and the cd case moved to `test_strips_leading_cd`, which asserts `"cd /tmp && ls"` normalizes to `"ls"`. ## Testing - [ ] Unit tests pass (`pytest`) (see Additional Notes) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) (see Additional Notes) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_memory/test_traffic_learner.py -k "CommandsRelatedAsRetry or NormalizeBashForHash or RefineErrorRecovery or NormalizedHash" =============== 35 passed, 122 deselected, 39 warnings in 0.80s =============== # New tests on main (fix reverted), full file: FAILED tests/test_memory/test_traffic_learner.py::TestCommandsRelatedAsRetry::test_shared_cd_prefix_alone_is_not_retry FAILED tests/test_memory/test_traffic_learner.py::TestNormalizeBashForHash::test_strips_leading_cd $ ruff check . All checks passed! $ mypy headroom/memory/traffic_learner.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.14, headroom 0.38.0, Claude Code via `headroom proxy --learn --memory`. - Exact command / steps: Read every persisted `traffic_learner` Bash `error_recovery` row from my own `memory.db` (read-only). Run `_commands_related_as_retry` and `_normalize_bash_for_hash` on each row's `failed_cmd` / `success_cmd`, first with `main` and then with this branch. - Observed result: 72 of 96 Bash rows had a cd prefix on both sides, and `main` accepted all 72 as retries; on this branch 16 are accepted. On `main` those rows also shared hash keys (e.g. 6 unrelated pairs on one `cd <repo>|cd <repo>` key); on this branch every row gets its own key. - Not tested: A full proxy session on Linux or macOS. I also did not check how long existing polluted rows take to disappear from a live `MEMORY.md` after upgrading. ## Runtime Rollout Safety - Rollout-managed feature(s): None - Minimum rollout channel: dev - Stable/default behavior changed: Yes, narrowly. Bash error-recovery pairing and hash keys ignore a leading `cd <dir>` segment. Nothing else changes. - Kill switch / disable path: N/A (existing `--learn` opt-in is unchanged) - Unsafe override required: No - Qualification impact: None - Rollback path: Revert commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes (see Additional Notes) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - **Unit tests.** On my Windows machine, 45 tests in `test_traffic_learner.py` already fail on `main`. Examples are `TestProjectForPattern` (POSIX path fixtures) and `TestUserAuthoredPreferenceFiltering`. The set of failing tests is identical before and after this change, apart from the new tests. I have not run the suite on Linux/macOS and am relying on CI for that. - **Type checking.** `mypy headroom` reports one error on Windows, in `headroom/fileperms.py:71` (`os.fchmod` does not exist on Windows). That file is untouched here. The changed module passes mypy. - **Documentation.** N/A, this is an internal matcher fix. - **Out of scope.** 16 remaining accepted pairs include some that are still questionable, where commands share a binary and a file name, e.g. `git status --short lib/x.ts` → `git add lib/x.ts && git commit`. That is a broader weakness of the token-overlap rule for compound commands. - **Existing hash keys move.** Any Bash command with a leading `cd <dir>` segment now normalizes to a different hash key, so rows persisted under the old key are not re-keyed. Their accumulated evidence counts are orphaned and learning for those commands restarts from zero. This is intentional: the old keys merged unrelated pairs, so their evidence was inflated. - **Existing rows.** Rows already persisted with the collapsed hash are not rewritten. They split into separate hashes on load, so most should fall below `min_evidence`, and the 21-day hard floor retires the rest. I did not add render-time re-validation for Bash rows, because `test_bash_recoveries_not_revalidated` documents that as intentional. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -62,6 +62,10 @@ _BASH_VOLATILE_SUFFIX_RE = re.compile(
|
||||
r"|\s+2>&1|\s+2>/dev/null)+\s*$"
|
||||
)
|
||||
|
||||
# Leading `cd <dir> && ` / `cd <dir>; ` prefixes. Agents prepend them to most
|
||||
# commands, so they carry no signal about which operation was retried.
|
||||
_BASH_LEADING_CD_RE = re.compile(r"""^\s*cd\s+(?:"[^"]*"|'[^']*'|(?:\\.|[^\s\\])+)\s*(?:&&|;)\s*""")
|
||||
|
||||
# Agent harnesses can encode orchestration metadata as user-role messages.
|
||||
# These prefixes identify whole messages that are not authored by the user.
|
||||
_HARNESS_USER_PREFIXES = (
|
||||
@@ -194,7 +198,7 @@ def _normalize_bash_for_hash(cmd: str) -> str:
|
||||
if not cmd:
|
||||
return ""
|
||||
# Drop paging, line-context flags, and redirections that vary between runs.
|
||||
trimmed = _BASH_VOLATILE_SUFFIX_RE.sub("", cmd).strip()
|
||||
trimmed = _BASH_VOLATILE_SUFFIX_RE.sub("", _strip_leading_cd(cmd)).strip()
|
||||
# Cut at the first pipe or && so we hash the primary command, not the tail.
|
||||
for sep in (" | ", " && "):
|
||||
idx = trimmed.find(sep)
|
||||
@@ -204,6 +208,15 @@ def _normalize_bash_for_hash(cmd: str) -> str:
|
||||
return trimmed
|
||||
|
||||
|
||||
def _strip_leading_cd(cmd: str) -> str:
|
||||
"""Remove leading `cd <dir> &&` / `cd <dir>;` segments from a Bash command."""
|
||||
prev = None
|
||||
while prev != cmd:
|
||||
prev = cmd
|
||||
cmd = _BASH_LEADING_CD_RE.sub("", cmd, count=1)
|
||||
return cmd
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Error Classification (reused from learn/scanner.py patterns)
|
||||
# =============================================================================
|
||||
@@ -372,6 +385,8 @@ def _commands_related_as_retry(failed: str, success: str) -> bool:
|
||||
arguments, and gets rejected. Genuine retries (extra flag, single
|
||||
arg edit) pass via the edit-distance path.
|
||||
"""
|
||||
failed = _strip_leading_cd(failed)
|
||||
success = _strip_leading_cd(success)
|
||||
if not failed or not success or failed == success:
|
||||
return False
|
||||
bin_a = _bash_first_binary(failed)
|
||||
|
||||
@@ -116,6 +116,23 @@ class TestCommandsRelatedAsRetry:
|
||||
)
|
||||
assert not _commands_related_as_retry(failed, success)
|
||||
|
||||
def test_shared_cd_prefix_alone_is_not_retry(self):
|
||||
# Agents prefix most commands with the same `cd <repo> &&`; that shared
|
||||
# prefix used to satisfy both the binary and the token-overlap checks.
|
||||
cd = 'cd "C:\\Users\\dev\\my-project" && '
|
||||
assert not _commands_related_as_retry(
|
||||
cd + "npx vitest run src/form.test.tsx", cd + "git add -A && git commit -F -"
|
||||
)
|
||||
assert not _commands_related_as_retry(
|
||||
"cd /home/dev/my-project && sed -n '90,130p' app/actions.ts",
|
||||
"cd /home/dev/my-project && cat components/icons.ts",
|
||||
)
|
||||
|
||||
def test_cd_prefixed_retry_still_matches(self):
|
||||
assert _commands_related_as_retry(
|
||||
"cd /home/dev/proj && cargo build", "cd /home/dev/proj && cargo build --release"
|
||||
)
|
||||
|
||||
def test_empty_or_equal_commands_rejected(self):
|
||||
assert not _commands_related_as_retry("", "ls")
|
||||
assert not _commands_related_as_retry("ls", "")
|
||||
@@ -1930,7 +1947,15 @@ class TestNormalizeBashForHash:
|
||||
|
||||
def test_cuts_at_first_chain(self):
|
||||
# && boundary collapses to just the primary command
|
||||
assert _normalize_bash_for_hash("cd /tmp && ls") == "cd /tmp"
|
||||
assert _normalize_bash_for_hash("make build && ls") == "make build"
|
||||
|
||||
def test_strips_leading_cd(self):
|
||||
# A `cd` prefix is not the primary command; keeping it collapsed every
|
||||
# cd-prefixed recovery in a project onto one hash key.
|
||||
assert _normalize_bash_for_hash("cd /tmp && ls") == "ls"
|
||||
assert _normalize_bash_for_hash('cd "C:\\a b"; cd sub && cargo check') == "cargo check"
|
||||
assert _normalize_bash_for_hash(r"cd /home/dev/my\ project && cargo check") == "cargo check"
|
||||
assert _normalize_bash_for_hash("cd /tmp") == "cd /tmp"
|
||||
|
||||
|
||||
class TestParseIsoTimestamp:
|
||||
|
||||
Reference in New Issue
Block a user