perf(memory): bound traffic-learner _persisted_ids with the dedup window (#3341)

## Description

`HeadroomTrafficLearner._persisted_ids` (a `dict[content_hash,
memory_id]`) grew **one entry per distinct persisted pattern for the
whole process lifetime** and was never trimmed, while its sibling dedup
set `_saved_hashes` **is** trimmed to `dedup_window`:

```python
self._saved_hashes.add(h)
if len(self._saved_hashes) > self._dedup_window:
    self._saved_hashes.pop()          # trimmed
# ...
self._persisted_ids[pattern.content_hash] = memory_id   # never trimmed
```

`_persisted_ids` is only ever *read* behind an `if h in
self._saved_hashes:` guard (the dedup shortcut at the top of
`_accumulate`). So once a hash leaves the dedup window, its
`_persisted_ids` entry can never be read again — it's pure dead weight
on the long-lived learner (which owns background save/flush tasks). The
asymmetry is the tell: `_saved_hashes` is trimmed and `_pattern_counts`
is LRU-capped right next to this, but `_persisted_ids` was missed.

The fix keeps the two in lockstep:
- When `_saved_hashes` evicts a hash, drop its `_persisted_ids` entry in
the same step.
- The async save worker records an id only while its hash is still
tracked in `_saved_hashes` (it may have been evicted between enqueue and
persist).

Neither changes the dedup/bump semantics — a hash evicted from
`_saved_hashes` was already re-learnable and could already produce a
duplicate row; these only stop `_persisted_ids` from leaking.

Reproduction (before): 200 distinct persisted patterns with
`dedup_window=4` → `_persisted_ids` holds ~200 entries while
`_saved_hashes` holds ≤4. After: `_persisted_ids` stays a subset of
`_saved_hashes`, bounded to ≤4.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/memory/traffic_learner.py`: on `_saved_hashes` eviction in
`_accumulate`, capture the popped hash and `_persisted_ids.pop(evicted,
None)`; in `_save_worker`, guard the id record with `and
pattern.content_hash in self._saved_hashes`.
- `tests/test_memory/test_traffic_learner.py`: added
`test_persisted_ids_bounded_in_lockstep_with_saved_hashes` — feeding 200
distinct patterns through a `dedup_window=4` learner leaves
`_persisted_ids` a subset of `_saved_hashes` and bounded to ≤4 (not
200).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_memory/test_traffic_learner.py  ->  153 passed, 2 failed
  (the 2 failures are pre-existing on main in TestProjectForPattern — project-root
   path matching, unrelated to this change; confirmed they fail without this patch)
uvx ruff@0.16.2 check headroom/memory/traffic_learner.py tests/test_memory/test_traffic_learner.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/memory/traffic_learner.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: drove `_accumulate` with 200 distinct patterns
(each corroborated to `min_evidence=1`) through a `dedup_window=4`
learner, recording an id for each still-tracked hash. Without the fix
`_persisted_ids` held ~200 entries and was not a subset of the 4-element
`_saved_hashes` (test fails). With the fix it stays a subset and ≤4.
Confirmed the two `TestProjectForPattern` failures pre-date this change
by stashing the patch.
- Observed result: `_persisted_ids` no longer grows without bound; it
tracks the dedup window. Dedup/bump behavior for in-window hashes is
unchanged.
- Not tested: an end-to-end run against a live memory backend (the async
`_save_worker` path needs a backend); the eviction/record invariant is
exercised directly, and the worker guard mirrors it.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Dedup shortcut and evidence-bump
behavior are unchanged for any hash still in the dedup window; only
stale `_persisted_ids` entries (already unreadable) are dropped.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; `_persisted_ids` goes back to
unbounded growth.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal memory bound)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

This mirrors the bounding already applied to `_saved_hashes`
(dedup_window trim) and `_pattern_counts` (LRU cap) in the same class,
closing the one companion structure that was left uncapped.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Abhay Singh
2026-09-30 11:16:58 -05:00
committed by GitHub
co-authored by JerrettDavis
parent 0eba65ca4a
commit 7e73438dc5
2 changed files with 112 additions and 5 deletions
+24 -5
View File
@@ -1275,10 +1275,16 @@ class TrafficLearner:
# Ready to save
del self._pattern_counts[h]
self._saved_hashes.add(h)
# Trim saved hashes to prevent unbounded growth
# Trim saved hashes to prevent unbounded growth, and drop the evicted
# hash's persisted-id entry in lockstep. ``_persisted_ids`` is only
# ever read behind an ``h in _saved_hashes`` guard (the dedup check
# above), so an id for a hash no longer tracked is dead weight;
# without this it grew one entry per distinct persisted pattern for
# the whole process lifetime while ``_saved_hashes`` stayed bounded.
if len(self._saved_hashes) > self._dedup_window:
# Remove oldest (arbitrary, set is unordered, but prevents growth)
self._saved_hashes.pop()
evicted = self._saved_hashes.pop()
self._persisted_ids.pop(evicted, None)
# Persist the real accumulated count, not the dataclass default.
pattern.evidence_count = count
@@ -1310,9 +1316,13 @@ class TrafficLearner:
},
)
self._patterns_saved += 1
# Track id so future re-sightings bump this row.
# Track id so future re-sightings bump this row — but only while
# the hash is still within the dedup window. If it was evicted
# from ``_saved_hashes`` between enqueue and now, recording its id
# would re-leak an entry that can never be read again (the dedup
# read at the top of ``_accumulate`` is gated on ``_saved_hashes``).
memory_id = getattr(memory, "id", None)
if memory_id is not None:
if memory_id is not None and pattern.content_hash in self._saved_hashes:
self._persisted_ids[pattern.content_hash] = memory_id
logger.debug(f"Traffic learner saved pattern: {pattern.content[:80]}")
@@ -1341,7 +1351,16 @@ class TrafficLearner:
try:
rows = conn.execute(
"SELECT id, content, metadata FROM memories "
"WHERE json_extract(metadata, '$.source') = 'traffic_learner'"
"WHERE json_extract(metadata, '$.source') = 'traffic_learner' "
# Hydrate at most dedup_window rows so a large persisted
# history does not start the in-memory dedup maps oversized
# (they are trimmed to dedup_window in steady state). Keep the
# most-recently-seen patterns; rows without last_seen_at
# (legacy) sort last under DESC and are dropped first, with id
# as a deterministic tie-break.
"ORDER BY json_extract(metadata, '$.last_seen_at') DESC, id DESC "
"LIMIT ?",
(self._dedup_window,),
).fetchall()
except sqlite3.DatabaseError:
return []
+88
View File
@@ -412,6 +412,41 @@ class TestTrafficLearner:
assert pattern.content_hash not in learner._pattern_counts # removed on promotion
assert pattern.content_hash in learner._saved_hashes
def test_persisted_ids_bounded_in_lockstep_with_saved_hashes(self):
"""``_persisted_ids`` (content_hash -> memory row id) must not outgrow
``_saved_hashes``. The dedup read only consults ``_persisted_ids`` behind
an ``h in _saved_hashes`` guard, so an id whose hash has been evicted from
the dedup window is dead weight; previously it accumulated one entry per
distinct persisted pattern for the whole process lifetime while
``_saved_hashes`` stayed trimmed to ``dedup_window``.
"""
import asyncio
learner = TrafficLearner(backend=None, min_evidence=1, dedup_window=4)
async def feed() -> None:
for i in range(200):
pattern = ExtractedPattern(
category=PatternCategory.PREFERENCE,
content=f"distinct preference number {i}",
importance=0.5,
)
await learner._accumulate(pattern) # first sight -> pending
await learner._accumulate(pattern) # second -> promote to saved
# Emulate the async save worker recording the row id, using the
# same "still tracked" guard the worker now applies.
if pattern.content_hash in learner._saved_hashes:
learner._persisted_ids[pattern.content_hash] = f"mem-{i}"
asyncio.run(feed())
# _saved_hashes stays bounded (existing behavior).
assert len(learner._saved_hashes) <= 4
# _persisted_ids no longer leaks: it never holds an id for a hash that is
# no longer in the dedup window, so it stays bounded too.
assert set(learner._persisted_ids).issubset(learner._saved_hashes)
assert len(learner._persisted_ids) <= 4 # not 200
@pytest.mark.asyncio
async def test_dedup(self, learner: TrafficLearner):
"""Test that identical patterns are deduplicated."""
@@ -1446,6 +1481,59 @@ class TestHydrateEdgeCases:
assert learner._saved_hashes == set()
assert learner._persisted_ids == {}
@pytest.mark.asyncio
async def test_hydration_is_bounded_to_dedup_window(self, tmp_path):
"""A persisted history larger than dedup_window must not start the
in-memory dedup maps oversized. Hydration keeps at most dedup_window
rows (the most-recently-seen), and both maps stay bounded with matching
keys — otherwise a long-lived install boots with an unbounded leak that
only trims one entry at a time."""
import json as _json
import sqlite3 as _sql
db = tmp_path / "memory.db"
_init_db(db)
window = 5
total = 20
conn = _sql.connect(db)
try:
for i in range(total):
conn.execute(
"INSERT INTO memories (id, content, metadata, entity_refs, importance) "
"VALUES (?,?,?,?,?)",
(
f"id-{i:02d}",
f"Command `cmd{i}` fails; use `alt{i}` instead.",
_json.dumps(
{
"source": "traffic_learner",
"category": "error_recovery",
"evidence_count": 2,
# Higher i == more recently seen; hydration keeps
# the newest `window` of these.
"last_seen_at": f"2026-01-01T00:{i:02d}:00+00:00",
}
),
"[]",
0.7,
),
)
conn.commit()
finally:
conn.close()
backend = _FakeBackend(db)
learner = TrafficLearner(backend=backend, min_evidence=2, dedup_window=window)
await learner._hydrate_persisted_state()
# Both maps are bounded to the window and hold exactly the same keys.
assert len(learner._saved_hashes) <= window
assert len(learner._persisted_ids) <= window
assert set(learner._persisted_ids) == set(learner._saved_hashes)
# The retained rows are the most-recently-seen ones (ids 15..19).
assert set(learner._persisted_ids.values()) == {f"id-{i:02d}" for i in range(15, 20)}
class TestBumpEdgeCases:
@pytest.mark.asyncio