feat(memory): persist TrafficLearner pending evidence across restarts + expose learner stats (#3104)

## Description

Pending (sub-threshold) TrafficLearner evidence lives only in
`_pattern_counts`, so every proxy restart zeroes the counters. A user
who accumulates 4 sightings of a pattern loses all of them on the next
app update or quit, which makes `min_evidence=5` effectively unreachable
for short or restart-heavy sessions - auto-learning looks dead even
though it is working. (Surfaced by a beta tester who used Claude Code ~6
hours across several days behind the proxy and saw zero learnings.)

This PR persists pending evidence across restarts via a
`pending_patterns.json` sidecar next to `memory.db`, and exposes learner
progress in the `/stats` payload so dashboards can show that learning is
alive before the first pattern crosses the threshold.

Follow-up to #2579, which bounded the same accumulator (the LRU cap is
respected on load).

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `TrafficLearner._save_pending_state()`: snapshot `_pattern_counts` to
`pending_patterns.json` (atomic tmp+rename), called on `stop()` and on
each debounced flush cycle so a crash loses at most one debounce window.
- `TrafficLearner._load_pending_state()`: rehydrate on `start()` after
`_hydrate_persisted_state()`; skips entries whose hash is already
persisted (re-sightings bump the existing row instead), tolerates
corrupt/unreadable sidecars, and respects `max_pending_patterns`.
- `get_stats()` now includes `min_evidence` so consumers can render
progress like "seen 3 of 5 times".
- `/stats` payload gains a `traffic_learner` block
(`proxy.traffic_learner.get_stats()`, `null` when learning is disabled).
- 3 new tests: restart survival end to end, corrupt sidecar ignored,
already-saved hash not rehydrated.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_memory/test_traffic_learner.py
============================= 157 passed in 2.82s ==============================

$ uvx ruff check headroom/memory/traffic_learner.py headroom/proxy/server.py tests/test_memory/test_traffic_learner.py
All checks passed!
$ uvx ruff format --check headroom/memory/traffic_learner.py headroom/proxy/server.py tests/test_memory/test_traffic_learner.py
3 files already formatted

$ uv run --frozen --extra dev mypy headroom/memory/traffic_learner.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, branch at
8895e054
- Exact command / steps: scripted two learner "processes" against the
same memory.db in a temp dir: process 1 accumulates 4 sightings of one
ENVIRONMENT pattern (min_evidence=5) and stops; process 2 starts fresh,
then sees the pattern a 5th time.
- Observed result: process 1 stats `{'patterns_extracted': 4,
'patterns_saved': 0, 'pending_patterns': 1, 'min_evidence': 5, ...}`;
sidecar after stop contains the pattern with `"count": 4`; process 2
stats after restart show `'pending_patterns': 1` before any traffic;
after the 5th sighting the DB row is saved with `evidence_count == 5`.
- Not tested: the full proxy path over live agent traffic (learner unit
surface only); mem0/remote backends, where `_resolve_backend_db_path`
returns None and persistence is a silent no-op (pre-existing behavior
for hydrate as well).

## Runtime Rollout Safety

- Rollout-managed feature(s): none - not behind a rollout flag.
- Minimum rollout channel: n/a
- Stable/default behavior changed: pending evidence now survives
restarts (the intended change); a small `pending_patterns.json` appears
next to memory.db. No change when learning is disabled or no local
backend is configured.
- Kill switch / disable path: disabling the traffic learner (existing
toggle) stops both writes and reads; deleting the sidecar restores
per-process behavior.
- Unsafe override required: no
- Qualification impact: none - `/stats` gains one additive key.
- Rollback path: revert the commit; stale sidecars are ignored dead
weight for older builds (never read).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-10-01 16:09:01 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent b625d188c5
commit 4257ed4df3
3 changed files with 188 additions and 0 deletions
+87
View File
@@ -540,6 +540,10 @@ class TrafficLearner:
# Hydrate persisted dedup state before workers spin up so cross-session
# re-sightings bump existing rows instead of creating duplicates.
await self._hydrate_persisted_state()
# Rehydrate sub-threshold evidence from the previous process. Without
# this every restart zeroed pending counts, so users with short or
# restart-heavy sessions could never cross min_evidence.
await self._load_pending_state()
if self._save_task is None or self._save_task.done():
self._save_task = asyncio.create_task(self._save_worker())
if self._flush_task is None or self._flush_task.done():
@@ -584,6 +588,9 @@ class TrafficLearner:
except Exception:
break
# Snapshot sub-threshold evidence so it survives the restart.
await self._save_pending_state()
# Final flush on shutdown — bypass debounce.
await self.flush_to_file()
@@ -601,6 +608,9 @@ class TrafficLearner:
self._flush_dirty = False
self._last_flush_at = time.monotonic()
await self.flush_to_file()
# Same debounce cycle also snapshots pending evidence, so a
# crash (vs clean stop) loses at most one debounce window.
await self._save_pending_state()
except asyncio.CancelledError:
break
except Exception as e:
@@ -846,6 +856,7 @@ class TrafficLearner:
"patterns_extracted": self._patterns_extracted,
"patterns_saved": self._patterns_saved,
"pending_patterns": len(self._pattern_counts),
"min_evidence": self._min_evidence,
"history_size": len(self._tool_history),
}
@@ -1415,6 +1426,82 @@ class TrafficLearner:
# last-wins — we only need one id to target the bump.
self._persisted_ids[h] = memory_id
def _pending_state_path(self) -> Path | None:
"""Sidecar JSON next to memory.db holding sub-threshold evidence."""
db_path = _resolve_backend_db_path(self._backend)
if db_path is None:
return None
return db_path.with_name("pending_patterns.json")
async def _save_pending_state(self) -> None:
"""Snapshot the pending accumulator so evidence survives restarts.
Best-effort: failures are logged at debug and never propagate — the
accumulator degrades gracefully back to per-process behavior.
"""
path = self._pending_state_path()
if path is None:
return
entries = [
{
"category": pattern.category.value,
"content": pattern.content,
"importance": pattern.importance,
"count": count,
"entity_refs": pattern.entity_refs,
"metadata": pattern.metadata,
"content_hash": h,
}
for h, (pattern, count) in self._pattern_counts.items()
]
def _write() -> None:
# tmp + rename so a crash mid-write can't truncate the file.
tmp = path.with_name(path.name + ".tmp")
tmp.write_text(json.dumps({"version": 1, "patterns": entries}))
os.replace(tmp, path)
try:
await asyncio.to_thread(_write)
except Exception as e:
logger.debug("Traffic learner pending-state save failed: %s", e)
async def _load_pending_state(self) -> None:
"""Rehydrate sub-threshold evidence saved by a previous process.
Runs after _hydrate_persisted_state so entries whose pattern crossed
the threshold in the meantime (hash now in _saved_hashes) are skipped;
their re-sightings bump the persisted row instead. Unreadable or
malformed files are ignored — the next snapshot overwrites them.
"""
path = self._pending_state_path()
if path is None or not path.exists():
return
try:
data = json.loads(await asyncio.to_thread(path.read_text))
except Exception as e:
logger.debug("Traffic learner pending-state load failed: %s", e)
return
entries = data.get("patterns") if isinstance(data, dict) else None
if not isinstance(entries, list):
return
for entry in entries[: self._max_pending_patterns]:
try:
h = entry["content_hash"]
if not h or h in self._saved_hashes or h in self._pattern_counts:
continue
pattern = ExtractedPattern(
category=PatternCategory(entry["category"]),
content=entry["content"],
importance=float(entry.get("importance", 0.5)),
entity_refs=list(entry.get("entity_refs", [])),
metadata=dict(entry.get("metadata", {})),
content_hash=h,
)
self._pattern_counts[h] = (pattern, max(1, int(entry.get("count", 1))))
except Exception:
continue
async def _bump_persisted_evidence(self, memory_id: str) -> None:
"""Atomically increment a persisted row's metadata.evidence_count."""
db_path = _resolve_backend_db_path(self._backend)
+7
View File
@@ -4985,6 +4985,13 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
),
},
"toin": get_toin().get_stats(),
# Auto-learning progress. `pending_patterns` counts sub-threshold
# evidence so dashboards can show the learner is alive before the
# first pattern crosses `min_evidence` (users otherwise read the
# silence as "learning is broken").
"traffic_learner": (
proxy.traffic_learner.get_stats() if proxy.traffic_learner else None
),
"proxy_inbound": proxy.metrics.inbound_snapshot(),
"cache": await proxy.cache.stats() if proxy.cache else None,
"rate_limiter": await proxy.rate_limiter.stats() if proxy.rate_limiter else None,
+94
View File
@@ -2604,3 +2604,97 @@ class TestExtractPreferencesSentenceBoundary:
assert not out[0].content.endswith(".")
assert not out[0].content.endswith("!")
assert not out[0].content.endswith("?")
# =============================================================================
# Pending-state persistence across restarts
# =============================================================================
class TestPendingStatePersistence:
"""Sub-threshold evidence must survive process restarts (sidecar JSON)."""
_PATTERN_KWARGS = {
"category": PatternCategory.ENVIRONMENT,
"content": "Use /usr/bin/python3 for system scripts.",
"importance": 0.6,
}
@pytest.mark.asyncio
async def test_pending_evidence_survives_restart(self, tmp_path):
"""2 sightings before restart + 1 after = saved with evidence_count 3."""
db = tmp_path / "memory.db"
_init_db(db)
first = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
await first.start()
for _ in range(2):
await first._accumulate(ExtractedPattern(**self._PATTERN_KWARGS))
await first.stop()
sidecar = tmp_path / "pending_patterns.json"
assert sidecar.exists()
assert _read_traffic_rows(db) == [] # still below threshold
second = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
await second.start()
assert second.get_stats()["pending_patterns"] == 1
await second._accumulate(ExtractedPattern(**self._PATTERN_KWARGS))
await _wait_for_saved(second, 1, db)
await second.stop()
rows = _read_traffic_rows(db)
assert len(rows) == 1
assert rows[0][2]["evidence_count"] == 3
@pytest.mark.asyncio
async def test_corrupt_pending_file_is_ignored(self, tmp_path):
db = tmp_path / "memory.db"
_init_db(db)
(tmp_path / "pending_patterns.json").write_text("{not json")
learner = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
await learner.start() # must not raise
assert learner.get_stats()["pending_patterns"] == 0
await learner.stop()
@pytest.mark.asyncio
async def test_already_saved_pattern_not_rehydrated_as_pending(self, tmp_path):
"""A sidecar entry whose hash is already persisted is skipped on load."""
import json as _json
db = tmp_path / "memory.db"
_init_db(db)
first = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
await first.start()
for _ in range(3):
await first._accumulate(ExtractedPattern(**self._PATTERN_KWARGS))
await _wait_for_saved(first, 1, db)
await first.stop()
# Handcraft a stale sidecar claiming the saved pattern is still pending.
h = ExtractedPattern(**self._PATTERN_KWARGS).content_hash
(tmp_path / "pending_patterns.json").write_text(
_json.dumps(
{
"version": 1,
"patterns": [
{
"category": "environment",
"content": self._PATTERN_KWARGS["content"],
"importance": 0.6,
"count": 2,
"entity_refs": [],
"metadata": {},
"content_hash": h,
}
],
}
)
)
second = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
await second.start()
assert second.get_stats()["pending_patterns"] == 0
await second.stop()