mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(memory): persist TrafficLearner pending evidence across restarts + expose learner stats (#3104)
## Description
Pending (sub-threshold) TrafficLearner evidence lives only in
`_pattern_counts`, so every proxy restart zeroes the counters. A user
who accumulates 4 sightings of a pattern loses all of them on the next
app update or quit, which makes `min_evidence=5` effectively unreachable
for short or restart-heavy sessions - auto-learning looks dead even
though it is working. (Surfaced by a beta tester who used Claude Code ~6
hours across several days behind the proxy and saw zero learnings.)
This PR persists pending evidence across restarts via a
`pending_patterns.json` sidecar next to `memory.db`, and exposes learner
progress in the `/stats` payload so dashboards can show that learning is
alive before the first pattern crosses the threshold.
Follow-up to #2579, which bounded the same accumulator (the LRU cap is
respected on load).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `TrafficLearner._save_pending_state()`: snapshot `_pattern_counts` to
`pending_patterns.json` (atomic tmp+rename), called on `stop()` and on
each debounced flush cycle so a crash loses at most one debounce window.
- `TrafficLearner._load_pending_state()`: rehydrate on `start()` after
`_hydrate_persisted_state()`; skips entries whose hash is already
persisted (re-sightings bump the existing row instead), tolerates
corrupt/unreadable sidecars, and respects `max_pending_patterns`.
- `get_stats()` now includes `min_evidence` so consumers can render
progress like "seen 3 of 5 times".
- `/stats` payload gains a `traffic_learner` block
(`proxy.traffic_learner.get_stats()`, `null` when learning is disabled).
- 3 new tests: restart survival end to end, corrupt sidecar ignored,
already-saved hash not rehydrated.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_memory/test_traffic_learner.py
============================= 157 passed in 2.82s ==============================
$ uvx ruff check headroom/memory/traffic_learner.py headroom/proxy/server.py tests/test_memory/test_traffic_learner.py
All checks passed!
$ uvx ruff format --check headroom/memory/traffic_learner.py headroom/proxy/server.py tests/test_memory/test_traffic_learner.py
3 files already formatted
$ uv run --frozen --extra dev mypy headroom/memory/traffic_learner.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 15.6 (Darwin 24.6.0), Python 3.12, branch at
8895e054
- Exact command / steps: scripted two learner "processes" against the
same memory.db in a temp dir: process 1 accumulates 4 sightings of one
ENVIRONMENT pattern (min_evidence=5) and stops; process 2 starts fresh,
then sees the pattern a 5th time.
- Observed result: process 1 stats `{'patterns_extracted': 4,
'patterns_saved': 0, 'pending_patterns': 1, 'min_evidence': 5, ...}`;
sidecar after stop contains the pattern with `"count": 4`; process 2
stats after restart show `'pending_patterns': 1` before any traffic;
after the 5th sighting the DB row is saved with `evidence_count == 5`.
- Not tested: the full proxy path over live agent traffic (learner unit
surface only); mem0/remote backends, where `_resolve_backend_db_path`
returns None and persistence is a silent no-op (pre-existing behavior
for hydrate as well).
## Runtime Rollout Safety
- Rollout-managed feature(s): none - not behind a rollout flag.
- Minimum rollout channel: n/a
- Stable/default behavior changed: pending evidence now survives
restarts (the intended change); a small `pending_patterns.json` appears
next to memory.db. No change when learning is disabled or no local
backend is configured.
- Kill switch / disable path: disabling the traffic learner (existing
toggle) stops both writes and reads; deleting the sidecar restores
per-process behavior.
- Unsafe override required: no
- Qualification impact: none - `/stats` gains one additive key.
- Rollback path: revert the commit; stale sidecars are ignored dead
weight for older builds (never read).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
b625d188c5
commit
4257ed4df3
@@ -540,6 +540,10 @@ class TrafficLearner:
|
||||
# Hydrate persisted dedup state before workers spin up so cross-session
|
||||
# re-sightings bump existing rows instead of creating duplicates.
|
||||
await self._hydrate_persisted_state()
|
||||
# Rehydrate sub-threshold evidence from the previous process. Without
|
||||
# this every restart zeroed pending counts, so users with short or
|
||||
# restart-heavy sessions could never cross min_evidence.
|
||||
await self._load_pending_state()
|
||||
if self._save_task is None or self._save_task.done():
|
||||
self._save_task = asyncio.create_task(self._save_worker())
|
||||
if self._flush_task is None or self._flush_task.done():
|
||||
@@ -584,6 +588,9 @@ class TrafficLearner:
|
||||
except Exception:
|
||||
break
|
||||
|
||||
# Snapshot sub-threshold evidence so it survives the restart.
|
||||
await self._save_pending_state()
|
||||
|
||||
# Final flush on shutdown — bypass debounce.
|
||||
await self.flush_to_file()
|
||||
|
||||
@@ -601,6 +608,9 @@ class TrafficLearner:
|
||||
self._flush_dirty = False
|
||||
self._last_flush_at = time.monotonic()
|
||||
await self.flush_to_file()
|
||||
# Same debounce cycle also snapshots pending evidence, so a
|
||||
# crash (vs clean stop) loses at most one debounce window.
|
||||
await self._save_pending_state()
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as e:
|
||||
@@ -846,6 +856,7 @@ class TrafficLearner:
|
||||
"patterns_extracted": self._patterns_extracted,
|
||||
"patterns_saved": self._patterns_saved,
|
||||
"pending_patterns": len(self._pattern_counts),
|
||||
"min_evidence": self._min_evidence,
|
||||
"history_size": len(self._tool_history),
|
||||
}
|
||||
|
||||
@@ -1415,6 +1426,82 @@ class TrafficLearner:
|
||||
# last-wins — we only need one id to target the bump.
|
||||
self._persisted_ids[h] = memory_id
|
||||
|
||||
def _pending_state_path(self) -> Path | None:
|
||||
"""Sidecar JSON next to memory.db holding sub-threshold evidence."""
|
||||
db_path = _resolve_backend_db_path(self._backend)
|
||||
if db_path is None:
|
||||
return None
|
||||
return db_path.with_name("pending_patterns.json")
|
||||
|
||||
async def _save_pending_state(self) -> None:
|
||||
"""Snapshot the pending accumulator so evidence survives restarts.
|
||||
|
||||
Best-effort: failures are logged at debug and never propagate — the
|
||||
accumulator degrades gracefully back to per-process behavior.
|
||||
"""
|
||||
path = self._pending_state_path()
|
||||
if path is None:
|
||||
return
|
||||
entries = [
|
||||
{
|
||||
"category": pattern.category.value,
|
||||
"content": pattern.content,
|
||||
"importance": pattern.importance,
|
||||
"count": count,
|
||||
"entity_refs": pattern.entity_refs,
|
||||
"metadata": pattern.metadata,
|
||||
"content_hash": h,
|
||||
}
|
||||
for h, (pattern, count) in self._pattern_counts.items()
|
||||
]
|
||||
|
||||
def _write() -> None:
|
||||
# tmp + rename so a crash mid-write can't truncate the file.
|
||||
tmp = path.with_name(path.name + ".tmp")
|
||||
tmp.write_text(json.dumps({"version": 1, "patterns": entries}))
|
||||
os.replace(tmp, path)
|
||||
|
||||
try:
|
||||
await asyncio.to_thread(_write)
|
||||
except Exception as e:
|
||||
logger.debug("Traffic learner pending-state save failed: %s", e)
|
||||
|
||||
async def _load_pending_state(self) -> None:
|
||||
"""Rehydrate sub-threshold evidence saved by a previous process.
|
||||
|
||||
Runs after _hydrate_persisted_state so entries whose pattern crossed
|
||||
the threshold in the meantime (hash now in _saved_hashes) are skipped;
|
||||
their re-sightings bump the persisted row instead. Unreadable or
|
||||
malformed files are ignored — the next snapshot overwrites them.
|
||||
"""
|
||||
path = self._pending_state_path()
|
||||
if path is None or not path.exists():
|
||||
return
|
||||
try:
|
||||
data = json.loads(await asyncio.to_thread(path.read_text))
|
||||
except Exception as e:
|
||||
logger.debug("Traffic learner pending-state load failed: %s", e)
|
||||
return
|
||||
entries = data.get("patterns") if isinstance(data, dict) else None
|
||||
if not isinstance(entries, list):
|
||||
return
|
||||
for entry in entries[: self._max_pending_patterns]:
|
||||
try:
|
||||
h = entry["content_hash"]
|
||||
if not h or h in self._saved_hashes or h in self._pattern_counts:
|
||||
continue
|
||||
pattern = ExtractedPattern(
|
||||
category=PatternCategory(entry["category"]),
|
||||
content=entry["content"],
|
||||
importance=float(entry.get("importance", 0.5)),
|
||||
entity_refs=list(entry.get("entity_refs", [])),
|
||||
metadata=dict(entry.get("metadata", {})),
|
||||
content_hash=h,
|
||||
)
|
||||
self._pattern_counts[h] = (pattern, max(1, int(entry.get("count", 1))))
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
async def _bump_persisted_evidence(self, memory_id: str) -> None:
|
||||
"""Atomically increment a persisted row's metadata.evidence_count."""
|
||||
db_path = _resolve_backend_db_path(self._backend)
|
||||
|
||||
@@ -4985,6 +4985,13 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
),
|
||||
},
|
||||
"toin": get_toin().get_stats(),
|
||||
# Auto-learning progress. `pending_patterns` counts sub-threshold
|
||||
# evidence so dashboards can show the learner is alive before the
|
||||
# first pattern crosses `min_evidence` (users otherwise read the
|
||||
# silence as "learning is broken").
|
||||
"traffic_learner": (
|
||||
proxy.traffic_learner.get_stats() if proxy.traffic_learner else None
|
||||
),
|
||||
"proxy_inbound": proxy.metrics.inbound_snapshot(),
|
||||
"cache": await proxy.cache.stats() if proxy.cache else None,
|
||||
"rate_limiter": await proxy.rate_limiter.stats() if proxy.rate_limiter else None,
|
||||
|
||||
@@ -2604,3 +2604,97 @@ class TestExtractPreferencesSentenceBoundary:
|
||||
assert not out[0].content.endswith(".")
|
||||
assert not out[0].content.endswith("!")
|
||||
assert not out[0].content.endswith("?")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Pending-state persistence across restarts
|
||||
# =============================================================================
|
||||
|
||||
|
||||
class TestPendingStatePersistence:
|
||||
"""Sub-threshold evidence must survive process restarts (sidecar JSON)."""
|
||||
|
||||
_PATTERN_KWARGS = {
|
||||
"category": PatternCategory.ENVIRONMENT,
|
||||
"content": "Use /usr/bin/python3 for system scripts.",
|
||||
"importance": 0.6,
|
||||
}
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pending_evidence_survives_restart(self, tmp_path):
|
||||
"""2 sightings before restart + 1 after = saved with evidence_count 3."""
|
||||
db = tmp_path / "memory.db"
|
||||
_init_db(db)
|
||||
|
||||
first = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
|
||||
await first.start()
|
||||
for _ in range(2):
|
||||
await first._accumulate(ExtractedPattern(**self._PATTERN_KWARGS))
|
||||
await first.stop()
|
||||
|
||||
sidecar = tmp_path / "pending_patterns.json"
|
||||
assert sidecar.exists()
|
||||
assert _read_traffic_rows(db) == [] # still below threshold
|
||||
|
||||
second = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
|
||||
await second.start()
|
||||
assert second.get_stats()["pending_patterns"] == 1
|
||||
await second._accumulate(ExtractedPattern(**self._PATTERN_KWARGS))
|
||||
await _wait_for_saved(second, 1, db)
|
||||
await second.stop()
|
||||
|
||||
rows = _read_traffic_rows(db)
|
||||
assert len(rows) == 1
|
||||
assert rows[0][2]["evidence_count"] == 3
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_corrupt_pending_file_is_ignored(self, tmp_path):
|
||||
db = tmp_path / "memory.db"
|
||||
_init_db(db)
|
||||
(tmp_path / "pending_patterns.json").write_text("{not json")
|
||||
|
||||
learner = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
|
||||
await learner.start() # must not raise
|
||||
assert learner.get_stats()["pending_patterns"] == 0
|
||||
await learner.stop()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_already_saved_pattern_not_rehydrated_as_pending(self, tmp_path):
|
||||
"""A sidecar entry whose hash is already persisted is skipped on load."""
|
||||
import json as _json
|
||||
|
||||
db = tmp_path / "memory.db"
|
||||
_init_db(db)
|
||||
|
||||
first = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
|
||||
await first.start()
|
||||
for _ in range(3):
|
||||
await first._accumulate(ExtractedPattern(**self._PATTERN_KWARGS))
|
||||
await _wait_for_saved(first, 1, db)
|
||||
await first.stop()
|
||||
|
||||
# Handcraft a stale sidecar claiming the saved pattern is still pending.
|
||||
h = ExtractedPattern(**self._PATTERN_KWARGS).content_hash
|
||||
(tmp_path / "pending_patterns.json").write_text(
|
||||
_json.dumps(
|
||||
{
|
||||
"version": 1,
|
||||
"patterns": [
|
||||
{
|
||||
"category": "environment",
|
||||
"content": self._PATTERN_KWARGS["content"],
|
||||
"importance": 0.6,
|
||||
"count": 2,
|
||||
"entity_refs": [],
|
||||
"metadata": {},
|
||||
"content_hash": h,
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
second = TrafficLearner(backend=_FakeBackend(db), min_evidence=3)
|
||||
await second.start()
|
||||
assert second.get_stats()["pending_patterns"] == 0
|
||||
await second.stop()
|
||||
|
||||
Reference in New Issue
Block a user