mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(ccr): keep the in-memory order queue from evicting re-stored entries (#4545)
## Description `InMemoryCcrStore` left a hash in its FIFO `order` queue when `get()` removed the entry because it had expired. If the same hash was stored again, `put` pushed a second copy, and at capacity `evict_until_under_capacity` popped the stale front copy and removed the fresh entry, the newest one, while an older entry survived. The queue also grew by one item for every entry that expired through `get()`. Each entry now carries a sequence number, eviction removes an entry only when the popped queue item's sequence matches, and the queue is compacted once it passes twice the capacity. Fixes #4544 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `crates/headroom-core/src/ccr/backends/in_memory.rs`: - `Entry` gets a `seq: u64`; the store gets an `AtomicU64` counter; `order` holds `(hash, seq)`. - `evict_until_under_capacity` pops `(hash, seq)` and calls `remove_if(hash, |e| e.seq == seq)`, so a stale item removes nothing and the loop moves on. - A fresh-key `put` takes the `order` lock first and does the capacity check, the map insert and the queue append inside it, re-checking for a writer that stored the same hash while it waited. So no entry is ever in the map without its queue item while eviction runs (review round 1: two writers missing the same hash could otherwise leave a capacity-1 store holding 2). Lock order is `order` then a map shard everywhere; `get` and the overwrite fast path never take `order` while holding a shard guard. - `compact_order` drops dead queue items once the queue is longer than `2 * max(capacity, 16)`. The in-place overwrite fast path is unchanged. - Three tests: `reput_after_expiry_keeps_the_newest_entry`, `order_queue_stays_bounded_when_entries_expire_through_get`, and `concurrent_fresh_inserts_never_exceed_capacity`, which uses a `#[cfg(test)]` hook to pause writer A between its insert and its queue append while writer C stores another key. ## Testing - [x] Unit tests pass (`cargo test`) - [x] Linting passes (`cargo clippy -- -D warnings`, `cargo fmt --check`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # fail before (original in_memory.rs, the two new tests only) a=None b=Some("2") order_len=3 assertion `left == right` failed: fresh a was evicted instead of b test ccr::backends::in_memory::tests::probe_reput_after_expiry_keeps_newest_entry ... FAILED map_len=0 order_len=2000 order queue grew to 2000 while the map held 0 test ccr::backends::in_memory::tests::probe_order_queue_stays_bounded ... FAILED test result: FAILED. 0 passed; 2 failed # round 2, fail before (round-1 head9ddf68cb+ the hook and the concurrency test) capacity 1 store holds 2 entries test result: FAILED. 0 passed; 1 failed # pass after: cargo test -p headroom-core --lib (default features) test result: ok. 987 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 5.39s # cargo clippy -p headroom-core --lib --tests -- -D warnings: no warnings # cargo fmt -p headroom-core -- --check: clean ``` (The fail-before run used the tests under their draft names `probe_*`; the bodies are the same apart from the debug `eprintln!` lines.) ## Real Behavior Proof - Environment: Windows 11, rustup toolchain 1.95.0 (the `rust-toolchain.toml` pin), `main` at `a2d4fe72`; a separate scratch binary crate (`ccrprobe`, not part of this PR) that depends on `headroom-core` by path with `default-features = false` and uses the store only through the public `headroom_core::ccr::{CcrStore, InMemoryCcrStore}` API, built with `cargo run --release`. - Exact command / steps: the binary (1) stores `a` in a capacity-2 store with a 30 ms TTL, sleeps 60 ms, calls `get("a")`, stores `b`, stores `a` again, stores `c`, then reads all three; (2) runs 200 rounds of 8 threads x 50 `put`s over 40 keys into a capacity-8 store and records the largest `len()` seen. Run once against this branch and once with `in_memory.rs` from `main`. - Observed result: on `main`: `after put(c): a=None b=Some("2") c=Some("3") len=2` (the freshly re-stored `a` was evicted, the older `b` kept). On this branch: `after put(c): a=Some("fresh") b=None c=Some("3") len=2`. The concurrent run printed `max len seen = 8` on both, so the over-capacity interleaving does not show up by chance; it is pinned deterministically by `concurrent_fresh_inserts_never_exceed_capacity` instead (2 entries on the round-1 head, at most 1 now). - Not tested: the Python wheel rebuilt with this crate, the Rust proxy binary, and the SQLite/Redis backends (unchanged). ## Runtime Rollout Safety - Rollout-managed feature(s): None. Internal fix to the in-memory CCR store. - Minimum rollout channel: N/A - Stable/default behavior changed: Yes, narrowly. Eviction now removes the oldest live entry instead of possibly the newest after an expire-and-re-store; the queue no longer grows past twice the capacity. Hit, miss and TTL behaviour are unchanged. - Kill switch / disable path: None needed; revert to roll back. - Unsafe override required: No - Qualification impact: None expected. One extra `u64` per entry and per queue item. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Produced with AI assistance (Claude Code) operated by me (HetCreep); the tests were run on my machine against a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ### Maintainer current-main requalification Exact head `48ae3713b096a6fba338cca7640424766bb0754a`, upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. Mechanically merged upstream/maine4a60ed7and re-reviewed the resulting in-memory CCR implementation and existing discussion. Generation-matched eviction, stale-queue compaction, and serialized fresh insertion retain the contributor fix; no production logic was rewritten. All 12 native in-memory backend tests pass on the freshly compiled current worktree, including the synchronized concurrent-insert regression, expire/re-store preservation, and bounded-queue checks. Rust formatting and whitespace checks pass. The existing public-API executable reproduction remains applicable; no fresh Python extension or full proxy runtime qualification is claimed. Applied the shared test-only Gemini streamed-request fixture repair for the independently established main defect, preserving existing assertions. All ten affected Gemini tests pass, with Ruff lint and formatting checks clean. Formal approval awaits exact-head hosted CI. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Maintainer commit-message qualification repair: current headbdae501786. Corrected only the maintainer main-sync commit message to Conventional Commit format. The complete file tree is byte-identical to the previously tested head; exact-head hosted checks are being refreshed. Nothing merged. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
f333edf20f
commit
c6f5bf9549
@@ -10,6 +10,7 @@
|
||||
//! workers (see `RUST_DEV.md` "Multi-worker deployment").
|
||||
|
||||
use std::collections::VecDeque;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::Mutex;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
@@ -33,11 +34,18 @@ use crate::ccr::{max_lifetime_for, CcrStore, DEFAULT_CAPACITY, DEFAULT_TTL};
|
||||
/// small sweep.
|
||||
pub struct InMemoryCcrStore {
|
||||
map: DashMap<String, Entry>,
|
||||
/// FIFO insertion order. Stale entries (already removed from `map`
|
||||
/// via TTL expiry) are tolerated — `pop_front` + `map.remove` is a
|
||||
/// no-op for missing keys, and capacity-bounded sweeps loop until
|
||||
/// they actually evict a real entry.
|
||||
order: Mutex<VecDeque<String>>,
|
||||
/// FIFO insertion order as `(hash, seq)`. A queue item is live only
|
||||
/// while `map[hash].seq == seq`: an entry removed by TTL expiry and
|
||||
/// stored again later gets a new `seq`, so its old queue item can
|
||||
/// neither evict the fresh entry nor pin it. Stale items are skipped
|
||||
/// by eviction and dropped by `compact_order` once the queue grows
|
||||
/// past twice the capacity.
|
||||
order: Mutex<VecDeque<(String, u64)>>,
|
||||
next_seq: AtomicU64,
|
||||
/// Test-only pause point between inserting a fresh entry and
|
||||
/// recording it in `order`, to reproduce writer interleavings.
|
||||
#[cfg(test)]
|
||||
after_insert_hook: Mutex<Option<Box<dyn FnOnce() + Send>>>,
|
||||
ttl: Duration,
|
||||
max_lifetime: Duration,
|
||||
capacity: usize,
|
||||
@@ -46,6 +54,7 @@ pub struct InMemoryCcrStore {
|
||||
#[derive(Clone)]
|
||||
struct Entry {
|
||||
payload: String,
|
||||
seq: u64,
|
||||
inserted: Instant,
|
||||
last_accessed: Instant,
|
||||
}
|
||||
@@ -77,25 +86,37 @@ impl InMemoryCcrStore {
|
||||
Self {
|
||||
map: DashMap::with_capacity(capacity),
|
||||
order: Mutex::new(VecDeque::with_capacity(capacity)),
|
||||
next_seq: AtomicU64::new(0),
|
||||
#[cfg(test)]
|
||||
after_insert_hook: Mutex::new(None),
|
||||
ttl,
|
||||
max_lifetime,
|
||||
capacity,
|
||||
}
|
||||
}
|
||||
|
||||
/// Sweep the order queue, dropping leading entries that no longer
|
||||
/// exist in the map (already expired or evicted), then evict
|
||||
/// real entries until `map.len() < capacity`. Called only from
|
||||
/// `put` on a fresh-key insert path.
|
||||
fn evict_until_under_capacity(&self) {
|
||||
let mut guard = self.order.lock().expect("ccr order mutex poisoned");
|
||||
/// Evict real entries, oldest first, until `map.len() < capacity`.
|
||||
/// The caller holds the `order` lock, which serializes every
|
||||
/// fresh-key insert, so no entry can be in the map without its
|
||||
/// queue item while this runs.
|
||||
fn evict_until_under_capacity(&self, guard: &mut VecDeque<(String, u64)>) {
|
||||
while self.map.len() >= self.capacity {
|
||||
let Some(oldest) = guard.pop_front() else {
|
||||
let Some((oldest, seq)) = guard.pop_front() else {
|
||||
break;
|
||||
};
|
||||
// `remove` is a no-op if `oldest` was already lazy-expired.
|
||||
// Loop continues until we actually shrink the map.
|
||||
self.map.remove(&oldest);
|
||||
// Remove only the entry this queue item was recorded for. A
|
||||
// stale item (the entry expired, or was stored again since)
|
||||
// removes nothing, and the loop moves on to the next one.
|
||||
self.map.remove_if(&oldest, |_, entry| entry.seq == seq);
|
||||
}
|
||||
}
|
||||
|
||||
/// Drop queue items whose entry is gone or was re-stored, so the
|
||||
/// queue stays within a constant factor of the live entries even
|
||||
/// when entries keep expiring through `get` without any eviction.
|
||||
fn compact_order(&self, order: &mut VecDeque<(String, u64)>) {
|
||||
if order.len() > self.capacity.max(16) * 2 {
|
||||
order.retain(|(hash, seq)| self.map.get(hash).is_some_and(|e| e.seq == *seq));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -119,28 +140,39 @@ impl CcrStore for InMemoryCcrStore {
|
||||
return;
|
||||
}
|
||||
|
||||
// New entry. Cap-bound first (may sweep a few stale order
|
||||
// entries), then insert and append to the FIFO queue.
|
||||
// New entry. The capacity check, the insert and the queue append
|
||||
// happen under one `order` lock, so a concurrent fresh insert can
|
||||
// never sit in the map without its queue item (which would let
|
||||
// eviction run out of queue while the map is still full). Lock
|
||||
// order is always `order` then a map shard; no path takes `order`
|
||||
// while holding a shard guard.
|
||||
let mut order = self.order.lock().expect("ccr order mutex poisoned");
|
||||
if let Some(mut existing) = self.map.get_mut(hash) {
|
||||
// Another writer stored this hash while we waited for the lock.
|
||||
let now = Instant::now();
|
||||
existing.payload = payload.to_string();
|
||||
existing.inserted = now;
|
||||
existing.last_accessed = now;
|
||||
return;
|
||||
}
|
||||
if self.map.len() >= self.capacity {
|
||||
self.evict_until_under_capacity();
|
||||
self.evict_until_under_capacity(&mut order);
|
||||
}
|
||||
let now = Instant::now();
|
||||
let seq = self.next_seq.fetch_add(1, Ordering::Relaxed);
|
||||
let entry = Entry {
|
||||
payload: payload.to_string(),
|
||||
seq,
|
||||
inserted: now,
|
||||
last_accessed: now,
|
||||
};
|
||||
let prev = self.map.insert(hash.to_string(), entry);
|
||||
if prev.is_none() {
|
||||
// Truly new key — record in FIFO order. (If `prev.is_some()`
|
||||
// it means another thread re-inserted between our get_mut
|
||||
// miss and this insert; treat that as a fast-path overwrite
|
||||
// and skip the queue append to avoid duplicates.)
|
||||
self.order
|
||||
.lock()
|
||||
.expect("ccr order mutex poisoned")
|
||||
.push_back(hash.to_string());
|
||||
self.map.insert(hash.to_string(), entry);
|
||||
#[cfg(test)]
|
||||
if let Some(hook) = self.after_insert_hook.lock().expect("hook mutex").take() {
|
||||
hook();
|
||||
}
|
||||
order.push_back((hash.to_string(), seq));
|
||||
self.compact_order(&mut order);
|
||||
}
|
||||
|
||||
fn get(&self, hash: &str) -> Option<String> {
|
||||
@@ -238,6 +270,80 @@ mod tests {
|
||||
assert_eq!(store.len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reput_after_expiry_keeps_the_newest_entry() {
|
||||
// a expires and is dropped by get(); b is stored; a is stored again
|
||||
// (now the newest entry). At capacity, the oldest live entry is b,
|
||||
// so storing c must evict b and keep the fresh a.
|
||||
let store = InMemoryCcrStore::with_capacity_and_ttl(2, Duration::from_millis(30));
|
||||
store.put("a", "old");
|
||||
std::thread::sleep(Duration::from_millis(60));
|
||||
assert_eq!(store.get("a"), None);
|
||||
store.put("b", "2");
|
||||
store.put("a", "fresh");
|
||||
store.put("c", "3");
|
||||
let a = store.get("a");
|
||||
assert_eq!(store.get("b"), None, "b was the oldest live entry");
|
||||
assert_eq!(
|
||||
a.as_deref(),
|
||||
Some("fresh"),
|
||||
"fresh a was evicted instead of b"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn order_queue_stays_bounded_when_entries_expire_through_get() {
|
||||
let store = InMemoryCcrStore::with_capacity_and_ttl(4, Duration::from_millis(1));
|
||||
for i in 0..2000 {
|
||||
let k = format!("k{i}");
|
||||
store.put(&k, "v");
|
||||
std::thread::sleep(Duration::from_millis(2));
|
||||
assert_eq!(store.get(&k), None);
|
||||
}
|
||||
let n = store.order.lock().unwrap().len();
|
||||
assert!(
|
||||
n <= 32,
|
||||
"order queue grew to {n} while the map held {}",
|
||||
store.len()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn concurrent_fresh_inserts_never_exceed_capacity() {
|
||||
use std::sync::{mpsc, Arc};
|
||||
// Capacity 1. Writer A stores x and pauses after the map insert,
|
||||
// before x is recorded in the order queue. Writer C then stores y.
|
||||
// If C could run in that gap it would find no queue item to evict
|
||||
// and leave both x and y in a capacity-1 store.
|
||||
let store = Arc::new(InMemoryCcrStore::with_capacity_and_ttl(1, DEFAULT_TTL));
|
||||
let (paused_tx, paused_rx) = mpsc::channel::<()>();
|
||||
let (release_tx, release_rx) = mpsc::channel::<()>();
|
||||
*store.after_insert_hook.lock().unwrap() = Some(Box::new(move || {
|
||||
paused_tx.send(()).unwrap();
|
||||
release_rx.recv().unwrap();
|
||||
}));
|
||||
let a = {
|
||||
let s = Arc::clone(&store);
|
||||
std::thread::spawn(move || s.put("x", "1"))
|
||||
};
|
||||
paused_rx.recv().unwrap();
|
||||
let c = {
|
||||
let s = Arc::clone(&store);
|
||||
std::thread::spawn(move || s.put("y", "2"))
|
||||
};
|
||||
// Give C time to run into the gap if it can.
|
||||
std::thread::sleep(Duration::from_millis(200));
|
||||
release_tx.send(()).unwrap();
|
||||
a.join().unwrap();
|
||||
c.join().unwrap();
|
||||
assert!(
|
||||
store.len() <= 1,
|
||||
"capacity 1 store holds {} entries",
|
||||
store.len()
|
||||
);
|
||||
assert_eq!(store.get("y").as_deref(), Some("2"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn store_is_send_sync() {
|
||||
fn assert_send_sync<T: Send + Sync>() {}
|
||||
|
||||
Reference in New Issue
Block a user