mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
perf(memory): bound traffic-learner _persisted_ids with the dedup window (#3341)
## Description
`HeadroomTrafficLearner._persisted_ids` (a `dict[content_hash,
memory_id]`) grew **one entry per distinct persisted pattern for the
whole process lifetime** and was never trimmed, while its sibling dedup
set `_saved_hashes` **is** trimmed to `dedup_window`:
```python
self._saved_hashes.add(h)
if len(self._saved_hashes) > self._dedup_window:
self._saved_hashes.pop() # trimmed
# ...
self._persisted_ids[pattern.content_hash] = memory_id # never trimmed
```
`_persisted_ids` is only ever *read* behind an `if h in
self._saved_hashes:` guard (the dedup shortcut at the top of
`_accumulate`). So once a hash leaves the dedup window, its
`_persisted_ids` entry can never be read again — it's pure dead weight
on the long-lived learner (which owns background save/flush tasks). The
asymmetry is the tell: `_saved_hashes` is trimmed and `_pattern_counts`
is LRU-capped right next to this, but `_persisted_ids` was missed.
The fix keeps the two in lockstep:
- When `_saved_hashes` evicts a hash, drop its `_persisted_ids` entry in
the same step.
- The async save worker records an id only while its hash is still
tracked in `_saved_hashes` (it may have been evicted between enqueue and
persist).
Neither changes the dedup/bump semantics — a hash evicted from
`_saved_hashes` was already re-learnable and could already produce a
duplicate row; these only stop `_persisted_ids` from leaking.
Reproduction (before): 200 distinct persisted patterns with
`dedup_window=4` → `_persisted_ids` holds ~200 entries while
`_saved_hashes` holds ≤4. After: `_persisted_ids` stays a subset of
`_saved_hashes`, bounded to ≤4.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/memory/traffic_learner.py`: on `_saved_hashes` eviction in
`_accumulate`, capture the popped hash and `_persisted_ids.pop(evicted,
None)`; in `_save_worker`, guard the id record with `and
pattern.content_hash in self._saved_hashes`.
- `tests/test_memory/test_traffic_learner.py`: added
`test_persisted_ids_bounded_in_lockstep_with_saved_hashes` — feeding 200
distinct patterns through a `dedup_window=4` learner leaves
`_persisted_ids` a subset of `_saved_hashes` and bounded to ≤4 (not
200).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_memory/test_traffic_learner.py -> 153 passed, 2 failed
(the 2 failures are pre-existing on main in TestProjectForPattern — project-root
path matching, unrelated to this change; confirmed they fail without this patch)
uvx ruff@0.16.2 check headroom/memory/traffic_learner.py tests/test_memory/test_traffic_learner.py -> All checks passed!
uvx mypy@1.20.2 headroom/memory/traffic_learner.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: drove `_accumulate` with 200 distinct patterns
(each corroborated to `min_evidence=1`) through a `dedup_window=4`
learner, recording an id for each still-tracked hash. Without the fix
`_persisted_ids` held ~200 entries and was not a subset of the 4-element
`_saved_hashes` (test fails). With the fix it stays a subset and ≤4.
Confirmed the two `TestProjectForPattern` failures pre-date this change
by stashing the patch.
- Observed result: `_persisted_ids` no longer grows without bound; it
tracks the dedup window. Dedup/bump behavior for in-window hashes is
unchanged.
- Not tested: an end-to-end run against a live memory backend (the async
`_save_worker` path needs a backend); the eviction/record invariant is
exercised directly, and the worker guard mirrors it.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Dedup shortcut and evidence-bump
behavior are unchanged for any hash still in the dedup window; only
stale `_persisted_ids` entries (already unreadable) are dropped.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; `_persisted_ids` goes back to
unbounded growth.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal memory bound)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
This mirrors the bounding already applied to `_saved_hashes`
(dedup_window trim) and `_pattern_counts` (LRU cap) in the same class,
closing the one companion structure that was left uncapped.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
0eba65ca4a
commit
7e73438dc5
@@ -1275,10 +1275,16 @@ class TrafficLearner:
|
||||
# Ready to save
|
||||
del self._pattern_counts[h]
|
||||
self._saved_hashes.add(h)
|
||||
# Trim saved hashes to prevent unbounded growth
|
||||
# Trim saved hashes to prevent unbounded growth, and drop the evicted
|
||||
# hash's persisted-id entry in lockstep. ``_persisted_ids`` is only
|
||||
# ever read behind an ``h in _saved_hashes`` guard (the dedup check
|
||||
# above), so an id for a hash no longer tracked is dead weight;
|
||||
# without this it grew one entry per distinct persisted pattern for
|
||||
# the whole process lifetime while ``_saved_hashes`` stayed bounded.
|
||||
if len(self._saved_hashes) > self._dedup_window:
|
||||
# Remove oldest (arbitrary, set is unordered, but prevents growth)
|
||||
self._saved_hashes.pop()
|
||||
evicted = self._saved_hashes.pop()
|
||||
self._persisted_ids.pop(evicted, None)
|
||||
|
||||
# Persist the real accumulated count, not the dataclass default.
|
||||
pattern.evidence_count = count
|
||||
@@ -1310,9 +1316,13 @@ class TrafficLearner:
|
||||
},
|
||||
)
|
||||
self._patterns_saved += 1
|
||||
# Track id so future re-sightings bump this row.
|
||||
# Track id so future re-sightings bump this row — but only while
|
||||
# the hash is still within the dedup window. If it was evicted
|
||||
# from ``_saved_hashes`` between enqueue and now, recording its id
|
||||
# would re-leak an entry that can never be read again (the dedup
|
||||
# read at the top of ``_accumulate`` is gated on ``_saved_hashes``).
|
||||
memory_id = getattr(memory, "id", None)
|
||||
if memory_id is not None:
|
||||
if memory_id is not None and pattern.content_hash in self._saved_hashes:
|
||||
self._persisted_ids[pattern.content_hash] = memory_id
|
||||
logger.debug(f"Traffic learner saved pattern: {pattern.content[:80]}")
|
||||
|
||||
@@ -1341,7 +1351,16 @@ class TrafficLearner:
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT id, content, metadata FROM memories "
|
||||
"WHERE json_extract(metadata, '$.source') = 'traffic_learner'"
|
||||
"WHERE json_extract(metadata, '$.source') = 'traffic_learner' "
|
||||
# Hydrate at most dedup_window rows so a large persisted
|
||||
# history does not start the in-memory dedup maps oversized
|
||||
# (they are trimmed to dedup_window in steady state). Keep the
|
||||
# most-recently-seen patterns; rows without last_seen_at
|
||||
# (legacy) sort last under DESC and are dropped first, with id
|
||||
# as a deterministic tie-break.
|
||||
"ORDER BY json_extract(metadata, '$.last_seen_at') DESC, id DESC "
|
||||
"LIMIT ?",
|
||||
(self._dedup_window,),
|
||||
).fetchall()
|
||||
except sqlite3.DatabaseError:
|
||||
return []
|
||||
|
||||
@@ -412,6 +412,41 @@ class TestTrafficLearner:
|
||||
assert pattern.content_hash not in learner._pattern_counts # removed on promotion
|
||||
assert pattern.content_hash in learner._saved_hashes
|
||||
|
||||
def test_persisted_ids_bounded_in_lockstep_with_saved_hashes(self):
|
||||
"""``_persisted_ids`` (content_hash -> memory row id) must not outgrow
|
||||
``_saved_hashes``. The dedup read only consults ``_persisted_ids`` behind
|
||||
an ``h in _saved_hashes`` guard, so an id whose hash has been evicted from
|
||||
the dedup window is dead weight; previously it accumulated one entry per
|
||||
distinct persisted pattern for the whole process lifetime while
|
||||
``_saved_hashes`` stayed trimmed to ``dedup_window``.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
learner = TrafficLearner(backend=None, min_evidence=1, dedup_window=4)
|
||||
|
||||
async def feed() -> None:
|
||||
for i in range(200):
|
||||
pattern = ExtractedPattern(
|
||||
category=PatternCategory.PREFERENCE,
|
||||
content=f"distinct preference number {i}",
|
||||
importance=0.5,
|
||||
)
|
||||
await learner._accumulate(pattern) # first sight -> pending
|
||||
await learner._accumulate(pattern) # second -> promote to saved
|
||||
# Emulate the async save worker recording the row id, using the
|
||||
# same "still tracked" guard the worker now applies.
|
||||
if pattern.content_hash in learner._saved_hashes:
|
||||
learner._persisted_ids[pattern.content_hash] = f"mem-{i}"
|
||||
|
||||
asyncio.run(feed())
|
||||
|
||||
# _saved_hashes stays bounded (existing behavior).
|
||||
assert len(learner._saved_hashes) <= 4
|
||||
# _persisted_ids no longer leaks: it never holds an id for a hash that is
|
||||
# no longer in the dedup window, so it stays bounded too.
|
||||
assert set(learner._persisted_ids).issubset(learner._saved_hashes)
|
||||
assert len(learner._persisted_ids) <= 4 # not 200
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dedup(self, learner: TrafficLearner):
|
||||
"""Test that identical patterns are deduplicated."""
|
||||
@@ -1446,6 +1481,59 @@ class TestHydrateEdgeCases:
|
||||
assert learner._saved_hashes == set()
|
||||
assert learner._persisted_ids == {}
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_hydration_is_bounded_to_dedup_window(self, tmp_path):
|
||||
"""A persisted history larger than dedup_window must not start the
|
||||
in-memory dedup maps oversized. Hydration keeps at most dedup_window
|
||||
rows (the most-recently-seen), and both maps stay bounded with matching
|
||||
keys — otherwise a long-lived install boots with an unbounded leak that
|
||||
only trims one entry at a time."""
|
||||
import json as _json
|
||||
import sqlite3 as _sql
|
||||
|
||||
db = tmp_path / "memory.db"
|
||||
_init_db(db)
|
||||
|
||||
window = 5
|
||||
total = 20
|
||||
conn = _sql.connect(db)
|
||||
try:
|
||||
for i in range(total):
|
||||
conn.execute(
|
||||
"INSERT INTO memories (id, content, metadata, entity_refs, importance) "
|
||||
"VALUES (?,?,?,?,?)",
|
||||
(
|
||||
f"id-{i:02d}",
|
||||
f"Command `cmd{i}` fails; use `alt{i}` instead.",
|
||||
_json.dumps(
|
||||
{
|
||||
"source": "traffic_learner",
|
||||
"category": "error_recovery",
|
||||
"evidence_count": 2,
|
||||
# Higher i == more recently seen; hydration keeps
|
||||
# the newest `window` of these.
|
||||
"last_seen_at": f"2026-01-01T00:{i:02d}:00+00:00",
|
||||
}
|
||||
),
|
||||
"[]",
|
||||
0.7,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
backend = _FakeBackend(db)
|
||||
learner = TrafficLearner(backend=backend, min_evidence=2, dedup_window=window)
|
||||
await learner._hydrate_persisted_state()
|
||||
|
||||
# Both maps are bounded to the window and hold exactly the same keys.
|
||||
assert len(learner._saved_hashes) <= window
|
||||
assert len(learner._persisted_ids) <= window
|
||||
assert set(learner._persisted_ids) == set(learner._saved_hashes)
|
||||
# The retained rows are the most-recently-seen ones (ids 15..19).
|
||||
assert set(learner._persisted_ids.values()) == {f"id-{i:02d}" for i in range(15, 20)}
|
||||
|
||||
|
||||
class TestBumpEdgeCases:
|
||||
@pytest.mark.asyncio
|
||||
|
||||
Reference in New Issue
Block a user