mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-12 10:35:06 +08:00
igooch
730250dca1
Phase 2 of #463, continuing from #735 (the eviction engine): the watermark-driven loop in atelet that turns eviction on, plus the flags that govern it. After this: observability (metrics/spans), the ops-tool swap, and the e2e suite. ## What - A serialized, panic-recovered pass every `--image-cache-gc-period` (default 5m; `0` disables the loop entirely). Each tick: `statfs` the cache volume + sum the pool's recorded sizes, compute a byte target — down to `--image-cache-low-percent` when usage crossed `--image-cache-high-percent`, and/or down to `--image-cache-max-bytes` — and hand it to `Store.EvictUnused`. - **The target is capped at the pool's own size.** The cache is one tenant of a shared volume; uncapped kubelet-style watermark math on a pressured volume asks a tiny cache to fix pressure it didn't cause, evicting everything every tick. - **Gated passes are detected by contract, not inference**: the engine exports `ErrIncompleteEnumeration` (wrapped into every gate return), and `classifyGCPass` — a pure, table-tested function — maps each pass to skipped/shortfall/complete/quiet. A gated pass logs ERROR "pass skipped" and never feeds the shortfall backoff. - **Genuine shortfall warns with backoff** then decays to a periodic reminder: on a volume under foreign pressure, shortfall is the steady state and must not ERROR every tick. - atelet passes `WithActorsDir(ateompath.ActorsDir)` so the root set sees placed actors (the wiring #735's notes required); `ateompath` gains the `ActorsDir` constant. - The first pass runs immediately at boot (a node starting under disk pressure shouldn't wait a full period), and a `CacheSize` failure skips the pass explicitly rather than silently zeroing the target. - `--image-cache-gc-dry-run` computes and logs every decision while deleting nothing — the production soak mechanism. A cache dir outside `BasePath` logs a warning (its watermarks would measure a different volume than actor state). - README: the GC section's "loop lands next" intro replaced with the loop/flag documentation. ## For reviewers: the enabled-by-default question Defaults ship **enabled** (5m / 85% / 80%, mirroring kubelet so operator intuition transfers), with dry-run as the opt-out soak path. The alternative is dry-run-by-default for a release. The target cap removes the known pathological case; happy to flip the default if you'd rather soak first. ## Testing - `imagegc_test.go`: watermark/cap target-math table (including the capped-target cases), flag validation. - Loop + engine together, Kind: full gVisor e2e suite green under aggressive flags (15s period, 1-byte cap); planted orphan ignored by periodic passes and reclaimed at startup; shortfall backoff observed against an impossible cap; no enumeration-gate false positives. - GKE (6 nodes): idle pool evicted under aggressive watermarks, subsequent resumes cold-pulled cleanly with snapshot memory continuity; flags restored to defaults after.