mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-02 03:24:42 +08:00
Dmitry Berkovich
3280b02fe9
Implements milestone M1 of the node-local artifact cache proposed in #690 (design in the issue comment): a generic `cmd/atelet/internal/filecache` package that will back golden-snapshot restores (today re-downloaded per actor on every start/resume) and later the sandbox-asset fetches. **it reads well commit by commit**: 1. **`atelet: add filecache store skeleton`** — constructor-only `Key`s (`SHA256Key` content-addressed, `URIKey` for immutable sources; prefix-disjoint canonical forms, entry dir = `sha256(key)`), the `entries/` + `tmp/` + `.rm-*` layout, `SweepDebris` (startup crash-debris reaper), `TotalBytes` (GC budget measure), debug-only `meta.json`. 2. **`atelet: add filecache singleflight retrieval (GetFileTo)`** — atomic get-and-link: per-key singleflight on `context.WithoutCancel` + fetch timeout (a canceled caller never aborts the download others wait on; no negative caching); fetch into `tmp/`, validate, chmod `0444` (in-place writes fail loudly instead of poisoning shared bytes), publish by one atomic rename; hit = hard link + LRU touch under `hitMu.RLock`. Path-based `FileFetcher` so `ategcs`'s sparse zstd download plugs in unchanged; `%w` wrapping end-to-end for `ateerrors` classification. 3. **`atelet: move the sparse file copy helpers into internal/sparsefile`** — mechanical move of `copyFile`/`copySparse`/`kernelCopyRange` (and their tests) out of package main so filecache can reuse them; adds `Copy(src, dst *os.File)` for caller-owned handles (source opened before its name can vanish, destination created `O_EXCL`). 4. **`filecache: add GetFileCopyTo for consumers that mutate staged files`** — the second serving mode: a private, hole-preserving copy (mode `0600`) instead of a read-only hard link, for consumers that rewrite staged files in place (ateom-microvm rewrites `config.json` at restore and merges deltas into `memory-ranges` at suspend — a shared inode would be corrupted). The copy reads a handle opened under the hit lock, so an eviction racing the copy retires only the entry's name; a copy needs no same-mount constraint. 5. **`atelet: add filecache eviction (EvictUnused)`** — pressure-driven only: min-age gate, unlinked-first then LRU ordering, stop at target. Two-phase retire inside the key's singleflight + `hitMu` exclusive (moved last-use clock or in-flight fetch vetoes; rename to `.rm-*`), slow `RemoveAll` after all retires outside the hot-path locks. Stats distinguish `Retired` (namespace removal, irreversible at rename) from `FreedBytes` (credited only after physical removal succeeds) and `PendingBytes` (retired but consumer-linked; kernel frees later). Copied-out entries carry no links, so eviction is free to take them — existing copies are private inodes and unaffected. 6. **`atelet: document filecache contracts and stress the get/evict races`** — package-doc contracts (link-out immunity, copy-out privacy, min-age sizing rule, read-only shared bytes, key immutability) plus a race-detector stress test: concurrent getters and evictors on shared keys; every get must succeed with intact content. The core safety property throughout: **eviction can only ever cost a refetch — never break a consumer**. Hard-linked files are protected by the link itself (the consumer's inode survives eviction); copies are private inodes; the min age covers the publish-to-use window. Follow-ups per the design: M2 wires a golden store into `Restore` (`downloadExternalCheckpoint`/`downloadCombinedCheckpoint`) with a GC driver loop — gVisor restores get hard links, micro-VM restores get copies; M3 adds `GetFile`/`GetDir` + the sandbox-record root set and migrates `fetchAsset`/`fetchGVisorRelease`. Tested: `go test -race -count=3 ./cmd/atelet/internal/filecache/`; every commit builds and passes tests individually; `golangci-lint`, gofmt, and boilerplate checks clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)