mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(batch): preserve live contexts during capacity cleanup (#4210)
## Description
Closes #4181.
Preserve live batch continuation contexts when replacing an existing
entry at capacity, and reclaim expired entries before evicting the
oldest live entry. The verified fork fix is now normally merged onto
main f1bc260 and published at
`e606a044babeed4913cf6c3ecd9ef940347a3b12`; no reset, force push or hook
bypass.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Replacing an existing batch refreshes its request context and TTL
without consuming another capacity slot or evicting an unrelated live
batch.
- Inserting a new batch removes expired contexts before applying the
existing oldest-live eviction policy.
- Keep the configured capacity bounded and retain deterministic
public-store regressions for both data-loss transitions.
## Testing
- [ ] Unit tests pass (pytest) — no entire-project Python suite claim.
- [ ] Linting passes (ruff check .) — normal changed-file hooks passed;
no whole-repository Ruff claim.
- [ ] Type checking passes (mypy headroom) — no separate whole-project
mypy claim.
- [x] Manual testing performed
### Test Output
Fresh current-main focused command: `python -m pytest
tests/test_ccr_batch_store.py tests/test_ccr_batch_processor.py
tests/test_proxy_handlers_batch.py -q`.
```text
110 passed in 25.77s
```
Normal current-main merge/push ran Rust workspace checks, rebuilt the
real native extension, and completed commitlint plus:
```text
181 critical Python tests passed
ci-precheck PASSED — safe to push.
```
Original independent real-store proof and failing-before/passing-after
regressions:
https://github.com/headroomlabs-ai/headroom/issues/4181#issuecomment-6148313526
. That older four-file suite was 119 passed, 14 skipped; it is not
relabelled as the current three-file result.
## Real Behavior Proof
- Environment: Linux x86_64, Python 3.13.14, actual public
BatchContextStore.store/get API, synthetic batch requests; no mocked
store or provider credentials.
- Exact command / steps: standalone asyncio script fills a three-entry
store with a,b,c then replaces c; separately fills a two-entry store
with an older live context and a newer expired context, then inserts a
new batch.
- Observed result: before the fix replacement lost a and insertion lost
the live context; after the fix a,b,c remain and the updated c model is
returned, while insertion retains live+new and removes expired. Original
runtime evidence is linked above; the current-main consumer suites and
full push gates also passed.
- Not tested: live provider continuation quality, cloud invoices,
production throughput, Windows/macOS, or hosted CI for this newly opened
PR.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; capacity policy correction within
the existing batch context store.
- Minimum rollout channel: existing batch-store users; no new channel.
- Stable/default behavior changed: updates no longer discard an
unrelated live context; expired entries are reclaimed before live
eviction.
- Kill switch / disable path: existing batch-processing configuration;
no new override.
- Unsafe override required: no.
- Qualification impact: public-store transitions, downstream batch
consumers and normal full push gates verified; provider execution was
not needed for this store defect.
- Rollback path: revert the capacity-policy fix; no persistence
migration or capacity increase.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I did not edit CHANGELOG.md — release-please owns it
## Additional Notes
This publishes an already-verified fix from the earlier Headroom batch
now that an author PR slot is available. It is not counted toward the
newly requested ten Bend issues. AI-assisted without subagents.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4210"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
This commit is contained in:
@@ -91,9 +91,9 @@ class BatchContextStore:
|
||||
|
||||
Features:
|
||||
- TTL-based expiration
|
||||
- Automatic cleanup of expired entries
|
||||
- Expired entries reclaimed before live contexts are evicted at capacity
|
||||
- Thread-safe operations
|
||||
- Memory limits
|
||||
- Memory limits; replacing a batch refreshes its TTL without evicting another batch
|
||||
|
||||
Usage:
|
||||
store = BatchContextStore()
|
||||
@@ -135,7 +135,7 @@ class BatchContextStore:
|
||||
"""
|
||||
async with self._lock:
|
||||
# Enforce memory limit
|
||||
if len(self._contexts) >= self._max_contexts:
|
||||
if context.batch_id not in self._contexts and len(self._contexts) >= self._max_contexts:
|
||||
# Remove oldest entries
|
||||
await self._cleanup_oldest()
|
||||
|
||||
@@ -205,10 +205,15 @@ class BatchContextStore:
|
||||
|
||||
async def _cleanup_oldest(self) -> None:
|
||||
"""Remove oldest entries to make room for new ones."""
|
||||
# Sort by creation time, remove oldest 10%
|
||||
if not self._contexts:
|
||||
now = time.time()
|
||||
expired = [batch_id for batch_id, ctx in self._contexts.items() if ctx.expires_at < now]
|
||||
for batch_id in expired:
|
||||
del self._contexts[batch_id]
|
||||
if len(self._contexts) < self._max_contexts:
|
||||
return
|
||||
|
||||
# Sort remaining live contexts by creation time, remove oldest 10%.
|
||||
|
||||
sorted_entries = sorted(
|
||||
self._contexts.items(),
|
||||
key=lambda x: x[1].created_at,
|
||||
|
||||
@@ -58,6 +58,52 @@ async def test_batch_context_store_core_operations(monkeypatch) -> None:
|
||||
assert await store.remove("b2") is False
|
||||
|
||||
|
||||
async def test_replacing_full_store_context_preserves_other_batch_and_refreshes_ttl(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
now = {"value": 100.0}
|
||||
monkeypatch.setattr("headroom.ccr.batch_store.time.time", lambda: now["value"])
|
||||
store = BatchContextStore(ttl=10, max_contexts=2)
|
||||
first = BatchContext(batch_id="first", provider="openai", created_at=0)
|
||||
second = BatchContext(batch_id="second", provider="openai", created_at=1)
|
||||
await store.store(first)
|
||||
await store.store(second)
|
||||
|
||||
now["value"] = 105.0
|
||||
replacement = BatchContext(batch_id="second", provider="openai")
|
||||
replacement.add_request(
|
||||
BatchRequestContext(custom_id="request", messages=[], model="updated-model")
|
||||
)
|
||||
await store.store(replacement)
|
||||
|
||||
assert await store.get("first") is first
|
||||
assert (await store.get("second")).get_request("request").model == "updated-model"
|
||||
now["value"] = 111.0
|
||||
assert await store.get("first") is None
|
||||
assert await store.get("second") is replacement
|
||||
|
||||
|
||||
async def test_full_store_reclaims_expired_batch_before_evicting_live_batch(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
now = {"value": 100.0}
|
||||
monkeypatch.setattr("headroom.ccr.batch_store.time.time", lambda: now["value"])
|
||||
store = BatchContextStore(ttl=10, max_contexts=2)
|
||||
expired = BatchContext(batch_id="expired", provider="openai", created_at=1)
|
||||
await store.store(expired)
|
||||
now["value"] = 105.0
|
||||
live = BatchContext(batch_id="live", provider="openai", created_at=0)
|
||||
await store.store(live)
|
||||
|
||||
now["value"] = 111.0
|
||||
new = BatchContext(batch_id="new", provider="openai", created_at=2)
|
||||
await store.store(new)
|
||||
|
||||
assert await store.get("live") is live
|
||||
assert await store.get("expired") is None
|
||||
assert await store.get("new") is new
|
||||
|
||||
|
||||
async def test_batch_context_store_cleanup_stats_and_memory_stats(monkeypatch) -> None:
|
||||
now = {"value": 200.0}
|
||||
monkeypatch.setattr("headroom.ccr.batch_store.time.time", lambda: now["value"])
|
||||
|
||||
Reference in New Issue
Block a user