fix(learn): recognize escaped persisted pattern IDs (#3951)

## Description

Fix the main CI shard-2 failure introduced by #2293. The writer saves
pattern-ID comments as escaped text, but its ID parser only recognized
raw comments. After a real write/read cycle, expired categories remained
forever, active unbatched items could disappear, and repeated IDs could
keep stale text.

Recognize both raw incoming annotations and the escaped annotations
already persisted by the writer. Content remains sanitized; no HTML
comments are restored and managed-block boundary protection is
unchanged.

Failing main job:
https://github.com/headroomlabs-ai/headroom/actions/runs/37097682536/job/111131105546

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Recognize persisted escaped pattern-ID annotations in the existing
parser.
- Add real write/read regressions for retaining active unbatched items,
removing expired items, and replacing stale text for an existing ID
without a lifecycle signal.
- Keep the existing category-removal regression and managed-block
injection defenses intact.
- Explicitly test LF and CRLF saved files and mark those cases for
Windows newline coverage.

## Testing

- [x] Focused unit tests pass
- [x] Linting passes on changed files
- [x] Type checking passes on the changed writer module
- [x] New tests added for the reproduced failures
- [x] Manual reproduction performed

### Test Output

```text
Before: test_section_lifecycle.py: 3 failed, 4 passed
  (original main failure plus both new round-trip regressions)
After: lifecycle + writer + traffic learner + managed-block injection:
  231 passed, 2 deselected
  (two confirmed pre-existing Windows path tests excluded)
Broader tests/test_learn + managed-block injection:
  354 passed, 1 skipped, 2 failed
  (both failures reproduced on unchanged main; details below)
ruff check: All checks passed!
ruff format --check: 2 files already formatted
mypy headroom/learn/writer.py --follow-imports=silent:
  Success: no issues found in 1 source file
git diff --check: clean
```

All four unrelated Windows failures were reproduced in an isolated
unchanged-main checkout at 9a72bc024:

- `TestProjectForPattern::test_matches_longest_root`
- `TestProjectForPattern::test_matches_via_entity_refs`
- `TestDigestBuilder::test_includes_project_info`
- `TestDecodeProjectPath::test_home_dir_username_stays_single_component`

## Real Behavior Proof

- Environment: Windows, Python 3.12.13, isolated worktrees based on main
9a72bc024.
- Exact steps: render recommendations with `_merge_into_file`, save the
result, then merge the next run from the actual saved file.
- Observed result: unchanged main retains the expired architecture
category, drops an active unbatched rule, and duplicates stale text.
This change removes the expired category, retains the active rule, and
replaces prior text by stable ID. Only managed-block delimiters remain
literal HTML comments.
- Not tested locally: the complete repository suite; hosted CI supplies
full-suite validation. Broader local checks have the baseline Windows
failures listed above.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable; existing learn writer correction.
- Stable/default behavior changed: persisted pattern IDs now retain
their existing lifecycle meaning across subsequent writes.
- Kill switch / disable path: do not apply learned recommendations.
- Unsafe override required: no.
- Qualification impact: no provider or compression changes;
managed-block sanitization tests pass.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project style guidelines
- [x] I have performed a self-review
- [x] I have commented the non-obvious parser boundary
- [x] My changes generate no new warnings
- [x] I have added tests that prove the fix is effective
- [x] Focused tests pass locally; broader baseline failures are
disclosed above
- [x] I did **not** edit `CHANGELOG.md`

## Hosted Verification

CI completed successfully on head `332df2e16`:
https://github.com/headroomlabs-ai/headroom/actions/runs/37098932451.
All four test shards, extras, Agno, lint and build jobs passed; Security
also completed successfully. The previously failing shard 2 passed all
nine lifecycle cases and finished with 3,537 passed / 225 skipped.
Locally, all eight marked newline tests passed on Windows. The newline
review thread is resolved. Human review and merge remain pending.

## Additional Notes

The sanitizer is intentionally unchanged. The parser reads the escaped
metadata directly instead of unescaping transcript-derived content. No
merge is requested or performed by this automation.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3951"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
This commit is contained in:
JD Davis
2026-10-03 13:35:12 -05:00
committed by GitHub
parent 2bc9419467
commit 2e74f06ece
2 changed files with 52 additions and 1 deletions
+6 -1
View File
@@ -116,7 +116,12 @@ def _build_section(recommendations: list[Recommendation]) -> str:
# Matches the "*~N tokens/session saved*" annotation emitted by _build_section.
_TOKENS_ANNOTATION_PATTERN = re.compile(r"\*~([\d,]+) tokens/session saved\*\n?")
_PATTERN_ID_PATTERN = re.compile(r"<!--\s*headroom:pattern-id:([^\s>]+)\s*-->\s*$")
# New recommendations contain raw annotations; saved blocks contain the
# escaped delimiters emitted by sanitize_block_text. Recognize both without
# unescaping transcript-derived content or weakening managed-block boundaries.
_PATTERN_ID_PATTERN = re.compile(
r"(?:<!--|&lt;!--)\s*headroom:pattern-id:([^\s>]+)\s*(?:-->|--&gt;)\s*$"
)
def _merge_markdown_items(
@@ -12,6 +12,8 @@ because it asserts on writer behaviour; it drives the real, unmodified
``_patterns_to_recommendations`` to get there.
"""
import pytest
from headroom.learn.models import Recommendation, RecommendationTarget
from headroom.learn.writer import _merge_into_file
from headroom.memory.traffic_learner import (
@@ -177,6 +179,50 @@ class TestCarriedSectionLifecycle:
class TestTrafficLearnerCategoryLifecycle:
"""End-to-end through the real learner rendering."""
@pytest.mark.windows_newline
@pytest.mark.parametrize("newline", ["\n", "\r\n"], ids=["lf", "crlf"])
def test_saved_ids_keep_unbatched_items_and_remove_expired_items(self, tmp_path, newline):
context_file = tmp_path / "AGENTS.md"
prior = _rec(
"Learned: preference",
"- Old queue <!-- headroom:pattern-id:queue -->\n"
"- Prefer ripgrep <!-- headroom:pattern-id:ripgrep -->\n"
"- Use vendored SDK <!-- headroom:pattern-id:vendored -->",
)
prior.preserve_prior_items = True
context_file.write_text(
_merge_into_file(context_file, [prior]), encoding="utf-8", newline=newline
)
current = _rec("Learned: preference", "- New queue <!-- headroom:pattern-id:queue -->")
current.preserve_prior_items = True
current.active_item_ids = frozenset({"queue", "ripgrep"})
final = _merge_into_file(context_file, [current])
assert "New queue" in final
assert "Old queue" not in final
assert "Prefer ripgrep" in final
assert "Use vendored SDK" not in final
assert final.count("<!--") == 2 # Only the managed block delimiters stay literal.
@pytest.mark.windows_newline
@pytest.mark.parametrize("newline", ["\n", "\r\n"], ids=["lf", "crlf"])
def test_saved_ids_replace_prior_text_without_a_lifecycle_signal(self, tmp_path, newline):
context_file = tmp_path / "AGENTS.md"
prior = _rec("Learned: preference", "- Old queue <!-- headroom:pattern-id:queue -->")
prior.preserve_prior_items = True
context_file.write_text(
_merge_into_file(context_file, [prior]), encoding="utf-8", newline=newline
)
current = _rec("Learned: preference", "- New queue <!-- headroom:pattern-id:queue -->")
current.preserve_prior_items = True
final = _merge_into_file(context_file, [current])
assert "New queue" in final
assert "Old queue" not in final
assert final.count("headroom:pattern-id:queue") == 1
def test_category_losing_its_last_pattern_drops_its_heading(self, tmp_path):
"""An emptied category leaves the file entirely.