fix(learn): anchor the managed block on its last start marker, append when no end marker (#4342)

## Description

`_merge_into_file` (`headroom/learn/writer.py`) located the managed
block with `block_pattern` (first occurrence of the start marker to the
nearest end marker) whenever `_MARKER_START in existing`. Two
consequences, both reported in #4339:

1. Hand-written prose that quotes the start-marker string became the
start of the match, so `--apply` replaced everything from that mention
through the real block's end marker, deleting the hand-written text in
between.
2. A file holding the start marker but no end marker matched nothing:
`sub()` returned the file unchanged and every recommendation of the run
was silently dropped while the CLI still printed `[WROTE]`.

The block is now identified by the shape `_build_section` writes: a line
that is **exactly** the start marker through the next line that is
**exactly** the end marker. A marker quoted inside prose (sharing its
line with other text, or indented) or inside a fenced code block (a
Markdown example) is not a boundary, and a start marker with no end
marker before the next start marker is skipped. One helper,
`_find_managed_block()`, is used by every reader and writer of the block
in this module, so the block that is parsed is the block that is
replaced.

The first commit (2c6853ea) used "the last start marker that has an end
marker after it"; review showed that only moved the problem to prose
after the block, and left `_parse_prior_recommendations` reading a
different span. The second commit (eba6458b) replaces it with the
line-based rule above, and the third (40880d1e, after review) skips
marker lines inside fenced code blocks.

Closes #4339

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/writer.py`: new `_find_managed_block(text) -> (start,
end) | None` (linear scan of lines; whole-line markers only; marker
lines inside ``` / ~~~ fences ignored, CommonMark-style open/close
rules; first complete block). `extract_marker_block`,
`_parse_prior_recommendations`, `_merge_into_file`,
`_strip_marker_block` and the two legacy-migration presence checks in
`_migrate_legacy_block` use it. With no block (including a dangling
start marker), `_merge_into_file` appends a fresh section as for a file
with no markers. `_MARKER_PATTERN` and the `block_pattern` import are
removed from this module; `managed_block.block_pattern` and
`sanitize_block_text` are unchanged.
- `tests/test_learn/test_writer.py` (`TestClaudeCodeWriter`), four
tests:
- `test_merge_keeps_prose_that_quotes_the_start_marker_above_the_block`
and `test_merge_appends_a_block_when_the_end_marker_is_missing` (the two
cases in #4339; both fail on `main`);
- `test_merge_keeps_prose_quoting_both_markers_after_the_block` and
`test_prior_sections_come_from_the_block_not_from_quoting_prose` (the
two layouts from the first review; both fail on the first commit);
- `test_merge_leaves_a_fenced_example_of_the_block_untouched` (the
fenced-example layout from the second review; fails on
eba6458b/35cf4a98) and
`test_fenced_code_inside_the_block_does_not_hide_its_end_marker` (a
closed code fence in a recommendation body).

Behaviour notes: a file holding two complete blocks used to have both
replaced by `sub()`; the merge now rewrites the first one and leaves the
second as text, and `_strip_marker_block` (legacy migration) still
removes every complete block. An unclosed code fence inside the managed
block would hide its end marker (the merge then appends a fresh block);
recommendation bodies with balanced fences are covered by a test.

## Testing

- [x] Unit tests pass (`pytest`) — `tests/test_learn/`,
`tests/test_managed_block_injection.py`,
`tests/test_memory/test_traffic_learner.py`: 617 passed, 10 skipped (on
40880d1e)
- [x] Linting passes (`ruff check .`) — on the two changed files, plus
`ruff format --check`
- [x] Type checking passes (`mypy headroom`) — run on
`headroom/learn/writer.py`; the single reported error is in
`headroom/providers/model_metadata.py:79` and is identical on `main`
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# the four tests against the FIRST commit 2c6853ea (fail-before for the two review layouts)
$ pytest tests/test_learn/test_writer.py -q -k "quotes_the_start_marker or end_marker_is_missing or both_markers_after or quoting_prose" --tb=line
tests/test_learn/test_writer.py:379: AssertionError: assert 2 == 1          # "### Manual policy" copied into the block
FAILED tests/test_learn/test_writer.py::TestClaudeCodeWriter::test_merge_keeps_prose_quoting_both_markers_after_the_block
FAILED tests/test_learn/test_writer.py::TestClaudeCodeWriter::test_prior_sections_come_from_the_block_not_from_quoting_prose
2 failed, 2 passed, 44 deselected in 1.82s

# with eba6458b (pass-after)
4 passed, 44 deselected in 0.99s

# fenced-example case (second review) on 35cf4a98, then on 40880d1e
FAILED tests/test_learn/test_writer.py::TestClaudeCodeWriter::test_merge_leaves_a_fenced_example_of_the_block_untouched
1 failed, 1 passed, 48 deselected
2 passed, 48 deselected in 1.14s

# the #4339 cases against main 4c7acba5 (from the first round)
2 failed (hand-written rule line deleted; file returned unchanged)

$ pytest tests/test_learn tests/test_managed_block_injection.py tests/test_memory/test_traffic_learner.py -q
617 passed, 10 skipped, 1 warning in 10.90s

$ ruff check headroom/learn/writer.py tests/test_learn/test_writer.py && ruff format --check headroom/learn/writer.py tests/test_learn/test_writer.py
All checks passed!
2 files already formatted
```

## Real Behavior Proof

- Environment: WSL2 Ubuntu, Python 3.12.15, pytest 9.1.1, ruff 0.17.1,
mypy 2.4.0; a clone of this fork with the core runtime dependencies only
- Exact command / steps: on the PR branch at the first commit, apply
only the two new tests and run the four `-k` tests above (the two review
layouts fail, the two #4339 cases pass); apply the source change and run
them again (all pass); then the learn, managed-block and traffic-learner
suites, ruff and mypy as pasted
- Observed result: prose that quotes the start marker above the block,
prose that quotes both markers below it, and headings inside such prose
all stay hand-written text: the managed block is the one rebuilt, its
prior sections are read from that same block, and a dangling start
marker gets a fresh section appended instead of losing the run's
recommendations
- Not tested: `headroom learn --apply` end to end through the CLI (only
`_merge_into_file`, the parser and the writer tests); the memory
writers, which share `block_pattern` but are not changed here

## Runtime Rollout Safety

- Rollout-managed feature(s): none (`headroom learn` is not behind a
rollout flag)
- Minimum rollout channel: n/a (not rollout-managed)
- Stable/default behavior changed: only for files with marker strings
quoted in prose, a dangling start marker, or two complete blocks (see
Behaviour notes); a file the writer produced merges exactly as before
- Kill switch / disable path: none added; `headroom learn` without
`--apply` never writes
- Unsafe override required: no
- Qualification impact: none known; no `CHANGELOG.md` edit
(release-please)
- Rollback path: revert the two commits

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
test runs above were executed on my machine on a clone of this fork.
Related: #3850 (introduced `managed_block.py` and the nearest-end-marker
rule).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
HetCreep
2026-10-24 08:45:10 -07:00
committed by GitHub
co-authored by Claude Fable 5.1 JerrettDavis
parent 5f308e1cc8
commit 45dacc1f94
2 changed files with 174 additions and 12 deletions
+51 -12
View File
@@ -16,7 +16,7 @@ from pathlib import Path
from headroom._subprocess import run
from ..managed_block import block_pattern, sanitize_block_text
from ..managed_block import sanitize_block_text
from ._shared import claude_config_dir
from .models import (
ProjectInfo,
@@ -29,7 +29,40 @@ from .models import (
# cannot close the block early (see headroom.managed_block).
_MARKER_START = "<!-- headroom:learn:start -->"
_MARKER_END = "<!-- headroom:learn:end -->"
_MARKER_PATTERN = block_pattern(_MARKER_START, _MARKER_END)
def _find_managed_block(text: str) -> tuple[int, int] | None:
"""Return the ``(start, end)`` offsets of the managed block, or None.
The block is a line that is exactly the start marker through the next line
that is exactly the end marker -- the shape ``_build_section`` writes. A
marker quoted inside prose (sharing its line with other text, or indented)
or inside a fenced code block (a Markdown example) is not a block
boundary, and a start marker with no end marker before the next start
marker is not a block. Every reader and writer of the block in
this module uses this one span, so the block that is parsed is the block
that is replaced (#4339). ``end`` stops after the end marker, before its
line break.
"""
offset = 0
start: int | None = None
fence: str | None = None # the open fence's run, e.g. "```" or "~~~~"
for line in text.splitlines(keepends=True):
bare = line.rstrip()
stripped = bare.lstrip(" ")
if len(bare) - len(stripped) <= 3 and stripped[:3] in ("```", "~~~"):
run = len(stripped) - len(stripped.lstrip(stripped[0]))
if fence is None:
fence = stripped[0] * run
elif stripped[0] == fence[0] and run >= len(fence) and not stripped[run:]:
fence = None
elif fence is None:
if bare == _MARKER_START:
start = offset
elif bare == _MARKER_END and start is not None:
return start, offset + len(bare)
offset += len(line)
return None
def _read_text_tolerant(file_path: Path) -> str:
@@ -258,8 +291,8 @@ def extract_marker_block(file_content: str) -> str | None:
(including the start/end markers) so it can be fed back to an LLM as
context without losing formatting. Returns None if no block is present.
"""
match = _MARKER_PATTERN.search(file_content)
return match.group(0) if match else None
span = _find_managed_block(file_content)
return file_content[span[0] : span[1]] if span else None
def _parse_prior_recommendations(existing: str) -> list[Recommendation]:
@@ -270,10 +303,10 @@ def _parse_prior_recommendations(existing: str) -> list[Recommendation]:
_build_section — target is set to a placeholder since the marker block
itself doesn't record it (blocks are always per-file and per-target).
"""
match = _MARKER_PATTERN.search(existing)
if not match:
span = _find_managed_block(existing)
if not span:
return []
inner = match.group(0)[len(_MARKER_START) : -len(_MARKER_END)]
inner = existing[span[0] + len(_MARKER_START) : span[1] - len(_MARKER_END)]
recs: list[Recommendation] = []
for part in re.split(r"\n### ", "\n" + inner)[1:]:
@@ -364,8 +397,12 @@ def _merge_into_file(file_path: Path, new_recommendations: list[Recommendation])
section = _build_section(merged)
if file_path.exists():
existing = _read_text_tolerant(file_path)
if _MARKER_START in existing:
return _MARKER_PATTERN.sub(lambda _match: section, existing)
# Replace only the managed block itself; marker strings quoted in
# prose on either side are hand-written text. No block (including a
# start marker without its end marker) -> append a fresh one (#4339).
span = _find_managed_block(existing)
if span:
return existing[: span[0]] + section + existing[span[1] :]
return existing.rstrip() + "\n\n" + section + "\n"
return section + "\n"
@@ -376,7 +413,9 @@ def _strip_marker_block(content: str) -> str:
Used when migrating a stale block out of the team-shared CLAUDE.md into the
personal CLAUDE.local.md. Returns "" if nothing but the block remained.
"""
cleaned = _MARKER_PATTERN.sub("", content)
cleaned = content
while span := _find_managed_block(cleaned):
cleaned = cleaned[: span[0]] + cleaned[span[1] :]
cleaned = re.sub(r"\n{3,}", "\n\n", cleaned).strip()
return cleaned + "\n" if cleaned else ""
@@ -562,11 +601,11 @@ class ClaudeCodeWriter(ContextWriter):
if target_path == legacy_path or not legacy_path.exists():
return []
legacy_text = _read_text_tolerant(legacy_path)
if _MARKER_START not in legacy_text:
if _find_managed_block(legacy_text) is None:
return []
# If the target already owns a block, it is the source of truth -- don't
# double-migrate or clobber accumulated learnings.
if target_path.exists() and _MARKER_START in _read_text_tolerant(target_path):
if target_path.exists() and _find_managed_block(_read_text_tolerant(target_path)):
return []
migrated = _parse_prior_recommendations(legacy_text)
+123
View File
@@ -299,6 +299,129 @@ class TestClaudeCodeWriter:
assert r"C:\Users\john.doe\repo" in full_content
assert "stale" not in full_content
def test_merge_keeps_prose_that_quotes_the_start_marker_above_the_block(self, tmp_path):
"""Prose quoting the start marker above the block must survive the rewrite."""
context_file = tmp_path / "CLAUDE.local.md"
prose = f"The block below starts with {_MARKER_START} and is managed."
context_file.write_text(
f"{prose}\n"
"- Hand-written rule: never force-push\n\n"
f"{_MARKER_START}\n"
"## Headroom Learned Patterns\n\n"
"### Environment\n"
"- old bullet\n\n"
f"{_MARKER_END}\n"
)
recommendation = _rec(RecommendationTarget.CONTEXT_FILE, "Tooling", "- Use uv")
final = _merge_into_file(context_file, [recommendation])
assert "- Hand-written rule: never force-push" in final
assert prose in final
assert "- Use uv" in final
assert final.count(_MARKER_START) == 2
assert final.count(_MARKER_END) == 1
def test_merge_appends_a_block_when_the_end_marker_is_missing(self, tmp_path):
"""A start marker with no end marker is not a block: the recommendations must land."""
context_file = tmp_path / "CLAUDE.local.md"
original = f"{_MARKER_START}\n- Hand-written rule: never force-push\n"
context_file.write_text(original)
recommendation = _rec(RecommendationTarget.CONTEXT_FILE, "Tooling", "- Use uv")
final = _merge_into_file(context_file, [recommendation])
assert final != original
assert "- Use uv" in final
assert "- Hand-written rule: never force-push" in final
def test_merge_keeps_prose_quoting_both_markers_after_the_block(self, tmp_path):
"""Marker quotes in prose below the block are hand-written text, not a second block."""
context_file = tmp_path / "CLAUDE.local.md"
context_file.write_text(
f"{_MARKER_START}\n"
"## Headroom Learned Patterns\n\n"
"### Environment\n"
"- old bullet\n\n"
f"{_MARKER_END}\n\n"
f"Example: {_MARKER_START}\n"
"- Hand-written rule: never force-push\n"
f"{_MARKER_END}\n"
)
recommendation = _rec(RecommendationTarget.CONTEXT_FILE, "Tooling", "- Use uv")
final = _merge_into_file(context_file, [recommendation])
assert "- Hand-written rule: never force-push" in final
assert final.startswith(f"{_MARKER_START}\n## Headroom Learned Patterns")
block = final[: final.index(_MARKER_END)]
assert "- Use uv" in block
assert "- old bullet" in block
assert final.count(f"Example: {_MARKER_START}") == 1
def test_prior_sections_come_from_the_block_not_from_quoting_prose(self, tmp_path):
"""Headings in prose that quotes the start marker must not become generated sections."""
context_file = tmp_path / "CLAUDE.local.md"
context_file.write_text(
f"Example: {_MARKER_START}\n"
"### Manual policy\n"
"- Hand-written rule: never force-push\n\n"
f"{_MARKER_START}\n"
"## Headroom Learned Patterns\n\n"
"### Environment\n"
"- old bullet\n\n"
f"{_MARKER_END}\n"
)
recommendation = _rec(RecommendationTarget.CONTEXT_FILE, "Tooling", "- Use uv")
final = _merge_into_file(context_file, [recommendation])
assert final.count("### Manual policy") == 1
block = final[final.index(f"\n{_MARKER_START}\n") :]
assert "### Manual policy" not in block
assert "### Environment" in block
assert "- Use uv" in block
def test_merge_leaves_a_fenced_example_of_the_block_untouched(self, tmp_path):
"""Markers inside a fenced Markdown example are not the managed block."""
context_file = tmp_path / "CLAUDE.local.md"
example = f"```markdown\n{_MARKER_START}\nEXAMPLE_SENTINEL\n{_MARKER_END}\n```\n"
context_file.write_text(
"How the managed block looks:\n\n" + example + "\n"
f"{_MARKER_START}\n"
"## Headroom Learned Patterns\n\n"
"### Environment\n"
"- old bullet\n\n"
f"{_MARKER_END}\n"
)
recommendation = _rec(RecommendationTarget.CONTEXT_FILE, "Tooling", "- Use uv")
final = _merge_into_file(context_file, [recommendation])
assert example in final
real_block = final[final.index(example) + len(example) :]
assert "- Use uv" in real_block
assert "- old bullet" in real_block
assert "- Use uv" not in final[: final.index(example) + len(example)]
def test_fenced_code_inside_the_block_does_not_hide_its_end_marker(self, tmp_path):
"""A closed code fence in a recommendation body keeps the block recognisable."""
context_file = tmp_path / "CLAUDE.local.md"
context_file.write_text(
f"{_MARKER_START}\n"
"## Headroom Learned Patterns\n\n"
"### Build\n"
"```bash\nmake test\n```\n\n"
f"{_MARKER_END}\n"
)
recommendation = _rec(RecommendationTarget.CONTEXT_FILE, "Tooling", "- Use uv")
final = _merge_into_file(context_file, [recommendation])
assert final.count(_MARKER_START) == 1
assert final.count(_MARKER_END) == 1
assert "- Use uv" in final
def test_memory_md_carry_forward(self, tmp_path):
"""Carry-forward also works for MEMORY.md."""
proj = _project(tmp_path)