fix(learn): decode sanitized pattern-id markers and keep visible-text dedup keys (#3956)

## Description

**Problem:** `test_category_losing_its_last_pattern_drops_its_heading`
fails on main (reproduced independently at 2bc9419 and noted as a merge
blocker for #3942). The lifecycle pruning added in #2293 never fires for
real files, so a category whose last pattern expired keeps its heading
forever. **Root cause:** `_build_section` runs every bullet through
`sanitize_block_text`, so pattern-id comments are persisted as `<!--
headroom:pattern-id:... -->`. `_prune_carried_section` and the
item-level merge then match ids with `_PATTERN_ID_PATTERN`, which
expects a literal `<!--` — on anything parsed back from disk every id
match misses, sections are treated as untracked, and the carry path
carries them untouched. The same-section path is unaffected only because
the current run re-emits its live items with unsanitized text. **Fix:**
decode each line with `html.unescape` before matching ids (in
`_tagged_pattern_id` and the `_items` visible-text dedup), while keeping
the original sanitized line when a bullet survives, so persisted content
stays byte-stable. **Tested:** ``` pytest
tests/test_learn/test_section_lifecycle.py -q 5 passed pytest
tests/test_learn/ -q 336 passed, 4 skipped ``` The target test fails on
main without the change and passes with it; ruff check and format pass.
<!-- devin-review-badge-begin --> --- <a
href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3956"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a> <!-- devin-review-badge-end -->

## Type of Change
- [x] Bug fix

## Changes Made
Decode sanitized pattern-id markers for lifecycle tracking while
preserving original persisted lines and original visible-text comparison
keys. This removes expired tracked advice without collapsing distinct
entity-looking literal examples.

## Testing
Maintainer validation: `python -m pytest tests/test_learn/test_writer.py
tests/test_learn/test_section_lifecycle.py -q` passed all 55 tests in an
isolated Windows worktree. A mechanical merge from current main
completed without conflicts.

## Real Behavior Proof
- Environment: Windows, Python 3.12, isolated maintainer worktree.
- Exact command / steps: writer and section-lifecycle suites listed
above.
- Observed result: 55 passing tests, including escaped pattern markers
and literal-text deduplication coverage.
- Not tested: production agent runs; final hosted qualification on the
updated branch remains pending.

## Runtime Rollout Safety
- Rollout-managed feature(s): existing learning writer; this patch
changes marker matching, not feature enablement.
- Minimum rollout channel: existing learning configuration and rollout
policy remain applicable.
- Stable/default behavior changed: persisted sanitized tracking markers
are recognized during pruning and merging.
- Kill switch / disable path: disable the existing learning
writer/injection feature.
- Unsafe override required: none.
- Qualification impact: verify tracked expired advice prunes while
distinct literal examples remain distinct.
- Rollback path: revert this fix; previously written sections remain
readable, with the earlier sanitized-marker tracking limitation.

## Review Readiness
- [x] I performed self-review
- [x] This PR is ready for human review

The self-review checkbox records the maintainer review. Exact
synchronized head b252fe5499 now has
terminal hosted checks with no failures; maintainer review approved it.
Existing contributor description is preserved above.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Zhibo Zhang (Amur)
2026-10-07 20:59:38 -05:00
committed by GitHub
co-authored by JD Davis
parent 326079517b
commit ec1ba70c97
2 changed files with 52 additions and 6 deletions
+17 -5
View File
@@ -6,6 +6,7 @@ injection mechanism for each agent system (CLAUDE.md, .cursorrules, etc.).
from __future__ import annotations
import html
import re
import subprocess
from abc import ABC, abstractmethod
@@ -124,6 +125,18 @@ _PATTERN_ID_PATTERN = re.compile(
)
def _tagged_pattern_id(line: str) -> str | None:
"""Return the id inside a bullet's pattern-id comment, if any.
``_build_section`` sanitizes block content, so on disk the comment reads
``&lt;!-- ... --&gt;``; match against the decoded line so carried-forward
sections are still recognized as lifecycle-tracked. Callers keep the
original (still-sanitized) line when the bullet survives.
"""
match = _PATTERN_ID_PATTERN.search(html.unescape(line))
return match.group(1) if match else None
def _merge_markdown_items(
new_content: str,
prior_content: str,
@@ -152,8 +165,7 @@ def _merge_markdown_items(
return None
items: list[tuple[str | None, str, str]] = []
for line in lines:
id_match = _PATTERN_ID_PATTERN.search(line)
pattern_id = id_match.group(1) if id_match else None
pattern_id = _tagged_pattern_id(line)
visible = _PATTERN_ID_PATTERN.sub("", line).strip().casefold()
items.append((pattern_id, visible, line))
return items
@@ -227,12 +239,12 @@ def _prune_carried_section(
kept: list[str] = []
saw_tracked_item = False
for line in lines:
id_match = _PATTERN_ID_PATTERN.search(line)
if id_match is None:
pattern_id = _tagged_pattern_id(line)
if pattern_id is None:
kept.append(line)
continue
saw_tracked_item = True
if id_match.group(1) in active_item_ids:
if pattern_id in active_item_ids:
kept.append(line)
if not saw_tracked_item:
return None
+35 -1
View File
@@ -15,7 +15,7 @@ because it asserts on writer behaviour; it drives the real, unmodified
import pytest
from headroom.learn.models import Recommendation, RecommendationTarget
from headroom.learn.writer import _merge_into_file
from headroom.learn.writer import _merge_into_file, _merge_markdown_items, _prune_carried_section
from headroom.memory.traffic_learner import (
ExtractedPattern,
PatternCategory,
@@ -262,3 +262,37 @@ class TestTrafficLearnerCategoryLifecycle:
assert "User prefers terse output" in final
assert "### Learned: architecture" not in final
assert "Handlers live under headroom/proxy/handlers" not in final
class TestSanitizedPatternIdHandling:
"""The persisted form of a pattern-id comment is HTML-sanitized."""
def test_prune_decodes_sanitized_pattern_ids(self):
"""A carried bullet whose id comment is sanitized still expires.
On disk the comment reads ``&lt;!-- ... --&gt;``; the prune path must
treat it as a tracked item, not as untagged content to keep forever.
"""
sanitized = (
"- Handlers live under headroom/proxy/handlers"
" &lt;!-- headroom:pattern-id:abc123 --&gt;"
)
assert _prune_carried_section(sanitized, frozenset()) == ""
assert _prune_carried_section(sanitized, frozenset({"abc123"})) == sanitized
def test_merge_keeps_entity_and_literal_ampersand_bullets_distinct(self):
"""Two live bullets differing only in ``&amp;`` vs ``&`` both survive.
The visible-text dedup key must stay on the bullet text as written:
decoding it before comparing would collapse suggestions that render
the same but are literally different advice.
"""
new_content = "- Prefer `a &amp; b` in configs <!-- headroom:pattern-id:hash2 -->"
prior_content = "- Prefer `a & b` in configs <!-- headroom:pattern-id:hash1 -->"
merged = _merge_markdown_items(new_content, prior_content, frozenset({"hash1", "hash2"}))
assert merged is not None
assert "`a &amp; b`" in merged
assert "`a & b`" in merged