Files
headroom/tests/test_memory_frontmatter_boundary.py
Yuchuan HeandJD Davis b560cba927 fix(memory): parse frontmatter delimiters on complete lines (#4886)
## Description

A memory beginning with `Use foo---bar as the separator` writes that
text into the generated name/description. The reader takes the embedded
`---` as the closing delimiter, mixes frontmatter into the body and
changes the content hash. The next sync imports a polluted duplicate.

Recognize opening and closing frontmatter delimiters only on complete
lines, including LF and CRLF input. Preserve the original text if the
header is malformed or has no closing delimiter.

Related to #4832's full-store sync work; this fixes a separate contract
on the current main snapshot.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Require complete delimiter lines; preserve bodies containing embedded
dashes and malformed headers.
- Add regressions against the unmodified base
`5023c84e527e28281dd73d1ba04184fe738aac0f` and real file/component
coverage.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook with httpx
installed)
- [x] New tests added
- [ ] Manual testing performed by the human author

### Test Output

7 new regressions; sync, CLI index and supersession suites: 77 passed.
The existing 35 sync cases are included. Existing datetime deprecation
warnings remain.

Original pre-commit and commit-msg hooks passed. Original full pre-push
`make ci-precheck` passed; native extension rebuilt/import-verified.
Full gate summaries:

```text
cargo fmt / clippy --workspace: passed
Rust workspace: 1755 passed, 3 ignored
Native extension: rebuilt and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing development
environment; base `5023c84e527e28281dd73d1ba04184fe738aac0f`, patch
`1fda45a711b4ce38848e8615148c74dc9c1bd6ab`. Real LocalBackend factory,
SQLite/FTS/vector store, agent files and Ollama HTTP adapter; the
loopback service returns controlled 384D vectors.
- Exact command / steps: save the following script as `proof.py`. Run
`python proof.py frontmatter <fresh-output-directory> before` from the
base checkout, then `python proof.py frontmatter
<another-fresh-directory> after` from the patch. The identical script
ran outside pytest.
- Observed result: first_exported=1 on both; round_trip_equal:
main=false, patch=true; repeat_imported/repeat_exported: main=1/1,
patch=0/0; primary_count: main=2, patch=1. The patch uses one actual
embedding HTTP request; main uses a second request to embed the polluted
duplicate.
- Not tested: trained-model inference or answer quality, paid/provider
APIs, non-local backends, production workloads, other operating systems
or the entire Python suite. Local focused, broader and original-gate
results are reported separately.

<details>
<summary>Standalone runtime reproduction</summary>

```python
"""Real LocalBackend/Ollama HTTP/SQLite/agent-file before-after replay."""
import asyncio
import json
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
import sys
import threading

from headroom.memory.backends.local import LocalBackend, LocalBackendConfig
from headroom.memory.sync import sync
from headroom.memory.sync_adapters.codex_agent import CodexAdapter
from headroom.memory.sync_adapters.claude_code import ClaudeCodeAdapter

case, folder, expected = sys.argv[1:]
out = Path(folder).resolve()
out.mkdir(parents=True, exist_ok=False)
os.environ.update(HEADROOM_STATELESS="0", HEADROOM_WORKSPACE_DIR=str(out/'workspace'), HEADROOM_OFFLINE="0")

class Endpoint(BaseHTTPRequestHandler):
    requests = []
    def log_message(self, *_): pass
    def do_POST(self):
        payload=json.loads(self.rfile.read(int(self.headers['content-length'])))
        self.requests.append({'path':self.path,'body':payload})
        assert self.path=='/api/embeddings'
        body=json.dumps({'embedding':[1.0]+[0.0]*383}).encode()
        self.send_response(200)
        self.send_header('Content-Type','application/json')
        self.send_header('Content-Length',str(len(body)))
        self.end_headers()
        self.wfile.write(body)

server=ThreadingHTTPServer(('127.0.0.1',0),Endpoint)
threading.Thread(target=server.serve_forever,daemon=True).start()

async def run():
    backend=LocalBackend(LocalBackendConfig(db_path=str(out/'memory.db'),embedder_backend='ollama',embedder_model='controlled-vector',vector_dimension=384,ollama_base_url=f'http://127.0.0.1:{server.server_port}'))
    state=out/'sync-state.json'
    try:
        if case=='fingerprint':
            saved=await backend.save_memory('Original orchard fact',user_id='alice')
            adapter=CodexAdapter(out/'AGENTS.md')
            first=await sync(backend,adapter,'alice',state)
            unchanged=await sync(backend,adapter,'alice',state)
            await backend._hierarchical_memory.update(saved.id,content='Corrected orchard fact')
            result=await sync(backend,adapter,'alice',state)
            text=adapter._path.read_text(encoding='utf-8')
            data={'first_exported':first.exported,'unchanged_exported':unchanged.exported,'edited_exported':result.exported,'corrected_in_file':'Corrected orchard fact' in text,'edited_imported':result.imported}
            assert data['first_exported']==1 and data['unchanged_exported']==0
            assert data['corrected_in_file']==(expected=='after'),data
            assert result.exported==(1 if expected=='after' else 0),data
        else:
            content='Use foo---bar as the separator\nKeep the second line intact.'
            saved=await backend.save_memory(content,user_id='alice')
            adapter=ClaudeCodeAdapter(out/'claude-memory')
            first=await sync(backend,adapter,'alice',state,force=True)
            readback=await adapter.read_memories()
            repeated=await sync(backend,adapter,'alice',state,force=True)
            rows=await backend.get_user_memories('alice')
            data={'first_exported':first.exported,'round_trip_equal':readback[0].content==content,'reread_content':readback[0].content,'repeat_imported':repeated.imported,'repeat_exported':repeated.exported,'primary_count':len(rows)}
            assert data['round_trip_equal']==(expected=='after'),data
            assert data['repeat_imported']==(0 if expected=='after' else 1),data
        data['requests']=Endpoint.requests
        data['source_file']=__import__('headroom').__file__
        (out/'result.json').write_text(json.dumps(data,indent=2),encoding='utf-8',newline='\n')
        print(json.dumps(data,ensure_ascii=False))
    finally: await backend.close()
try: asyncio.run(run())
finally: server.shutdown();server.server_close()

```
</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing memory sync.
- Minimum rollout channel: existing memory sync users.
- Stable/default behavior changed: metadata containing triple dashes
remains separate from the memory body.
- Kill switch / disable path: stop invoking memory sync; no new toggle.
- Unsafe override required: no.
- Qualification impact: memory synchronization correctness; no
dependency, native or workflow change.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented the non-obvious behavior
- [x] I have added regression tests
- [x] I did not edit CHANGELOG.md

## Additional Notes

Implementation and local test execution were completed with Codex
assistance. The author personally reviewed the two patches and the
before/after evidence; this does not claim the author personally ran the
tests.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4886"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:17:10 -06:00

47 lines
1.8 KiB
Python

"""A horizontal separator in metadata must not end YAML frontmatter."""
import hashlib
import pytest
from headroom.memory.sync_adapters.claude_code import ClaudeCodeAdapter, _parse_frontmatter
@pytest.mark.parametrize("newline", ["\n", "\r\n"])
def test_embedded_dashes_are_not_a_closing_delimiter(newline):
text = newline.join(["---", "name: foo---bar", "type: project", "---", "", "Original body"])
metadata, body = _parse_frontmatter(text)
assert metadata == {"name": "foo---bar", "type": "project"}
assert body == "Original body"
@pytest.mark.parametrize("opening", ["---not-a-header", "----"])
def test_non_delimiter_opening_preserves_the_original_body(opening):
text = f"{opening}\nname: text\n---\nbody"
assert _parse_frontmatter(text) == ({}, text)
def test_closing_delimiter_must_occupy_the_line():
text = "---\nname: value\n---suffix\ntype: project\n---\nBody\n---\nTail"
metadata, body = _parse_frontmatter(text)
assert metadata == {"name": "value", "type": "project"}
assert body == "Body\n---\nTail"
def test_missing_delimiter_preserves_input():
text = "---\nname: foo---bar\nbody without closing delimiter"
assert _parse_frontmatter(text) == ({}, text)
@pytest.mark.asyncio
async def test_real_adapter_round_trips_dashes_in_generated_metadata(tmp_path):
adapter = ClaudeCodeAdapter(tmp_path / "memory")
content = "Use foo---bar as the separator\nKeep the second line intact."
original_hash = hashlib.sha256(content.encode()).hexdigest()[:16]
assert await adapter.write_memories([{"content": content, "content_hash": original_hash}]) == 1
reread = await adapter.read_memories()
assert len(reread) == 1
assert reread[0].content == content
assert reread[0].content_hash == original_hash
assert await adapter.write_memories([{"content": content, "content_hash": original_hash}]) == 0