Files
substrate/internal/imagecache/imagecache.go
T
igooch 596cfb8ab0 imagecache: eviction engine — root set, EvictUnused, startup orphan recovery (#735)
Phase 2 of #463 (image-cache GC + observability): this is the eviction
engine itself — the root set, the eviction pass, and startup orphan
recovery — assembled from the foundations merged in #650
(sizes/last-use), #656 (spec digest), #658 (retirement primitives), and
#717 (record-first pull). What remains after this is the watermark loop
+ flags that turn it on, then metrics and e2e.

**Inert in production**: nothing calls `EvictUnused` yet — the watermark
GC loop and its flags come in the next PR. The only behavior `New` gains
is the startup orphan scan.

## What

- **`Store.InUse()`** — the root set: scans bundle overlay specs under
the actors dir (opt-in via `WithActorsDir`) for image digests, layer
dirs, and exact layer-set signatures. The exact-set rule roots the
multi-arch twin record and digestless (pre-`imageDigest`) specs' records
without making every subset image unevictable. Returns `(RootSet,
error)`: a non-nil error means the root set may be incomplete, and
deleting callers must do nothing.
- **`Store.EvictUnused(ctx, targetBytes, dryRun)`** — LRU eviction of
unprotected images and the layers their removal leaves unreferenced.
Min-age vetoes; per-victim re-checks under `hitMu` held exclusive
against the cache-hit touch; two-phase deletion via `retireLayer`, with
the slow `RemoveAll` outside all locks. If any layer of a deleted record
must be kept (still referenced, rooted, fresh, or failed to retire), the
record is atomically restored from bytes captured at listing time — a
kept layer is never left unreachable.
- **`Store.RecoverOrphans`**, called once from `New` — reclaims layers
no record references. Startup is the one moment this scan is race-free
(no pull can be in flight); a failure never fails `New`.
- **Everything fails toward retention.** Both passes skip entirely, at
ERROR naming the culprit, when their inputs can't be fully enumerated:
an unreadable/undecodable image record, a garbled diffID inside a valid
record, or an unenumerable root set (actors dir, bundles dir, or bundle
spec). Refcounts and roots from partial data would retire layers that
unread records still reference or running actors still mount — and for a
long-running actor, the bundle spec is the only protection (its record
mtime can be arbitrarily old, so min-age is no backstop). Missing
dirs/specs remain legitimate emptiness, not errors.
- **Eviction never writes into the live pool.** Retired layers are
credited from the size file that rides along with the rename-aside
(O(1), read-only, consistent with `CacheSize`); the size-file backfill
exists only on the `CacheSize` path, where it bumps the dir mtime rather
than restoring it (restoring could rewind a concurrent reuse-touch).
- Per-item log lines are Debug behind an `Enabled` gate (level control
via #677); pass summaries, evictions, and restores are Info.
- `internal/imagecache/README.md` now documents the GC: pass structure,
protection rules, failure direction, startup recovery, and two-phase
deletion, plus the on-disk layout entries (`size`, `.tmp-*`, `.rm-*`)
that shipped with #650/#658.

## Notes for reviewers

- The gc tests reuse the helpers merged with #650 (`backdate`,
`mustEnsure`) and #717's gated-registry harness — no duplicated symbols.
The slow-pull gated test gains the eviction-veto assertion deferred from
#717.
- Dry-run mutates nothing, including size-file backfills
(`layerSizeReadOnly`).
- A `Store` built without `WithActorsDir` has an empty root set
(deliberate for tools/tests).
- Carried to the wiring PR: pass `WithActorsDir` in atelet; handle
`EvictUnused`'s error before any shortfall accounting (a gated pass
returns zero stats plus an error and must not be logged as a shortfall);
and surface skipped passes as a counter in the observability PR so a
wedged node alerts rather than scrolling ERROR lines.

## Testing

- `go test -race ./internal/imagecache/`: every veto, shared-layer
survival, both restore-on-keep shapes, orphan reclaimed at startup and
ignored by the periodic pass, all enumeration gates — unreadable record
(shared-layer scenario), undecodable record, garbled diffID, and the
three unreadable root-set shapes (actors dir / bundles dir / spec, each
against a rooted image with an arbitrarily old record) — dry-run purity,
25-iteration ensure-vs-evict race loop.
- Kind, via the integration branch that stacks the watermark loop on
this PR (the loop and flags are **not** part of this PR): full gVisor
e2e suite green; a planted backdated orphan layer (no size file) was
reclaimed at atelet startup with the expected log lines, its size
credited read-only from the retired dir.
- GKE (6 nodes), same integration branch: demo lifecycle +
snapshot-continuity suites green; under aggressive watermarks the idle
pool was evicted and subsequent actor resumes cold-pulled cleanly. No
enumeration-gate false positives on either cluster.
2026-08-10 14:46:27 -07:00

710 lines
26 KiB
Go

// Copyright 2026 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package imagecache implements the node-local OCI image cache: a
// content-addressed pool of unpacked image layers shared by every actor on
// the node, plus the per-bundle overlay spec that tells the ateom runtimes
// how to compose an actor rootfs from cached layers.
//
// The work is split along the existing atelet/ateom privilege boundary:
//
// - atelet (plain root, all capabilities dropped) pulls layers and unpacks
// them into the pool (Store.EnsureImage), and writes a rootfs-overlay.json
// next to each bundle's config.json (WriteSpec). Whiteout entries are
// recorded in per-layer metadata rather than materialized, because
// overlayfs whiteouts are char devices (CAP_MKNOD) with trusted.* xattrs
// for opaque dirs (CAP_SYS_ADMIN).
// - ateom (privileged; it already owns every mount on the node) finalizes
// layers — materializing the recorded whiteout state, once per layer —
// and mounts the overlay rootfs (SetupBundleRootfs) just before
// `runsc create` / staging the micro-VM virtio-fs lower.
//
// On-disk layout under the cache root (a directory on the BasePath hostPath,
// so the same absolute paths resolve in atelet and every ateom pod):
//
// version layout version marker
// layers/sha256/<diffid-hex>/
// fs/ the unpacked layer tree (overlay lowerdir)
// whiteouts.json whiteout state recorded at unpack time
// finalized marker written by FinalizeLayer (ateom)
// manifests/sha256/<digest-hex>.json
// image config + ordered diffID list
//
// Layers land in the pool via unpack-into-tempdir + atomic rename, so a
// layer directory that exists is always complete; startup recovery only has
// to sweep orphaned temp dirs.
package imagecache
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net"
"os"
"path/filepath"
"runtime"
"strconv"
"strings"
"sync"
"time"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/remote"
"golang.org/x/sync/errgroup"
"golang.org/x/sync/singleflight"
)
const (
layoutVersion = "1"
versionFileName = "version"
layerFSDirName = "fs"
layerWhiteoutsFileName = "whiteouts.json"
layerFinalizedMarkerName = "finalized"
// layerSizeFileName holds the layer's byte count, recorded at unpack so
// sizing the pool never walks a tree. Absent for layers unpacked by
// older atelets (backfilled lazily, see layerSize). An estimate: the
// value is the tar-stream length when written at unpack, or the summed
// file sizes when backfilled, so it may differ from disk usage and
// across nodes.
layerSizeFileName = "size"
// defaultMinAge is the default eviction minimum age (see WithMinAge).
defaultMinAge = 2 * time.Minute
// layerPullConcurrency bounds concurrent layer download+unpack streams per
// image pull. Memory use is O(stream buffers) per slot, independent of
// layer size.
layerPullConcurrency = 4
)
// Store is atelet's handle to the on-disk layer pool. It is safe for
// concurrent use; concurrent pulls of the same image or layer are collapsed.
// The store assumes it is the only writer on the node (one atelet per node).
type Store struct {
root string
// authenticator, when set, is attached to pulls from registries that use
// GCP credentials (gcr.io / pkg.dev). See remoteOpts.
authenticator authn.Authenticator
localhostRegistryReplacement string
// platform overrides the default pull platform (linux/GOARCH). Used by
// validation tooling that runs on a different architecture than the
// nodes it validates for.
platform *v1.Platform
// actorsDir is scanned by InUse for bundle overlay specs; empty disables
// the scan (the root set is then empty).
actorsDir string
// minAge vetoes eviction of any layer or image record younger than this,
// covering the window between a pull (or cache-hit stat) and the bundle
// spec write / ateom mount that roots it.
minAge time.Duration
imageSF singleflight.Group
layerSF singleflight.Group
// evictMu serializes EvictUnused passes (concurrent passes would fight
// over the same candidates for no benefit).
evictMu sync.Mutex
// hitMu closes the hit-vs-evict window: held shared by the hit path
// (cachedImageHit), exclusive by eviction's record removal
// (removeStaleRecord), so a hit's last-use touch and eviction's final
// re-check can never interleave. Uncontended except during a pass.
hitMu sync.RWMutex
}
// Option configures a Store.
type Option func(*Store)
// WithAuthenticator attaches an authenticator used for gcr.io / pkg.dev
// registries. A nil authenticator is ignored.
func WithAuthenticator(a authn.Authenticator) Option {
return func(s *Store) { s.authenticator = a }
}
// WithLocalhostRegistryReplacement rewrites localhost/loopback registry refs
// to the given endpoint, mirroring the containerd mirror config used by kind
// local registries (https://kind.sigs.k8s.io/docs/user/local-registry/).
func WithLocalhostRegistryReplacement(replacement string) Option {
return func(s *Store) { s.localhostRegistryReplacement = replacement }
}
// WithPlatform overrides the pull platform (default: linux/GOARCH).
func WithPlatform(p v1.Platform) Option {
return func(s *Store) { s.platform = &p }
}
// WithActorsDir points the eviction root-set scan at the node's actors
// directory (the per-actor state dirs under ateompath.BasePath). Each
// <actorsDir>/<actorUID>/bundles/<container>/rootfs-overlay.json roots its
// image and layers against eviction. Empty disables the scan.
func WithActorsDir(dir string) Option {
return func(s *Store) { s.actorsDir = dir }
}
// WithMinAge overrides the eviction minimum age (default 2m): layers and
// image records younger than this are never evicted.
func WithMinAge(d time.Duration) Option {
return func(s *Store) { s.minAge = d }
}
// Image describes one cached, ready-to-compose image.
type Image struct {
// Digest is the manifest digest the caller's ref resolved to (for a
// multi-arch ref, the index digest as requested, not the per-platform
// child).
Digest v1.Hash
// Config is the OCI image config (entrypoint, env, ...).
Config v1.Config
// LayerDirs are the absolute cached layer directories, bottom-most layer
// first. Each contains the unpacked tree under "fs/".
LayerDirs []string
}
// imageRecord is the persisted form of a cached image, stored under
// manifests/<algorithm>/<hex>.json.
type imageRecord struct {
Version int `json:"version"`
Config v1.Config `json:"config"`
DiffIDs []string `json:"diffIDs"`
}
// New opens (creating if needed) the layer pool rooted at root and runs
// startup recovery: verifying the layout version and sweeping temp dirs left
// by unpacks that were in flight when a previous atelet died.
func New(root string, opts ...Option) (*Store, error) {
s := &Store{root: root, minAge: defaultMinAge}
for _, o := range opts {
o(s)
}
for _, d := range []string{s.layersDir(), s.manifestsDir()} {
if err := os.MkdirAll(d, 0o700); err != nil {
return nil, fmt.Errorf("while creating image cache dir %q: %w", d, err)
}
}
versionPath := filepath.Join(root, versionFileName)
switch b, err := os.ReadFile(versionPath); {
case err == nil:
if got := strings.TrimSpace(string(b)); got != layoutVersion {
// Fail loudly instead of silently mixing layouts; an operator can
// delete the cache dir to rebuild it (it holds no unique state).
return nil, fmt.Errorf("image cache at %q has layout version %q, this atelet supports %q", root, got, layoutVersion)
}
case errors.Is(err, os.ErrNotExist):
if err := os.WriteFile(versionPath, []byte(layoutVersion+"\n"), 0o600); err != nil {
return nil, fmt.Errorf("while writing image cache version marker: %w", err)
}
default:
return nil, fmt.Errorf("while reading image cache version marker: %w", err)
}
if err := s.sweepTempDirs(); err != nil {
return nil, err
}
// Startup-only orphan recovery (see RecoverOrphans for why it must not
// run during normal operation). Failure is logged inside, never fatal:
// a corrupt record must not keep atelet from serving actors.
if _, err := s.RecoverOrphans(context.Background()); err != nil {
slog.Warn("Image cache startup orphan recovery incomplete", slog.Any("err", err))
}
return s, nil
}
func (s *Store) layersDir() string { return filepath.Join(s.root, "layers", "sha256") }
func (s *Store) manifestsDir() string { return filepath.Join(s.root, "manifests", "sha256") }
func (s *Store) layerDir(diffID v1.Hash) string {
return filepath.Join(s.root, "layers", diffID.Algorithm, diffID.Hex)
}
func (s *Store) recordPath(digest v1.Hash) string {
return filepath.Join(s.root, "manifests", digest.Algorithm, digest.Hex+".json")
}
// sweepTempDirs removes unpack temp dirs, retired layer dirs, and
// manifest-record temp files orphaned by a crash. A layer dir without the
// temp/retired prefix and a record without a leading dot are always
// complete (both are moved into place with a single rename), so this is
// the only recovery the pool needs.
func (s *Store) sweepTempDirs() error {
entries, err := os.ReadDir(s.layersDir())
if err != nil {
return fmt.Errorf("while listing layer pool: %w", err)
}
swept := 0
for _, e := range entries {
// ".tmp-": an unpack in flight at crash. ".rm-": a layer retirement
// renamed aside but not yet removed. Either way, unreferenced
// garbage.
if !strings.HasPrefix(e.Name(), ".tmp-") && !strings.HasPrefix(e.Name(), retiredPrefix) {
continue
}
p := filepath.Join(s.layersDir(), e.Name())
slog.Info("Image cache sweeping orphaned layer dir", slog.String("dir", e.Name()))
if err := RemoveAllWritable(p); err != nil {
return fmt.Errorf("while sweeping orphaned layer dir %q: %w", p, err)
}
swept++
}
if swept > 0 {
slog.Info("Image cache startup sweep removed orphaned layer dirs", slog.Int("count", swept))
}
// writeRecord's temp files are ".<hex>.json.tmp-<rand>"; finished records
// are "<hex>.json", so a leading dot alone identifies an orphan.
records, err := os.ReadDir(s.manifestsDir())
if err != nil {
return fmt.Errorf("while listing manifest records: %w", err)
}
for _, e := range records {
if !strings.HasPrefix(e.Name(), ".") {
continue
}
p := filepath.Join(s.manifestsDir(), e.Name())
if err := os.Remove(p); err != nil {
return fmt.Errorf("while sweeping orphaned manifest temp file %q: %w", p, err)
}
}
return nil
}
// EnsureImage makes ref's image available in the pool and returns its config
// and ordered layer directories. Digest refs hit the cache with no network
// I/O; tag refs cost one HEAD request to resolve the tag to a manifest
// digest (so tag refs are cacheable, and a moved tag is picked up on the
// next call).
func (s *Store) EnsureImage(ctx context.Context, ref string) (*Image, error) {
parsedRef, err := s.parseRef(ref)
if err != nil {
return nil, fmt.Errorf("while parsing reference: %w", err)
}
var digest v1.Hash
if d, ok := parsedRef.(name.Digest); ok {
digest, err = v1.NewHash(d.DigestStr())
if err != nil {
return nil, fmt.Errorf("while parsing digest of %q: %w", ref, err)
}
} else {
// Tag ref: one small HEAD request pins it to an immutable manifest
// digest, which is the only safe cache key for mutable tags.
desc, err := remote.Head(parsedRef, s.remoteOpts(ctx, parsedRef)...)
if err != nil {
return nil, fmt.Errorf("while resolving tag %q to a digest: %w", ref, err)
}
digest = desc.Digest
}
img, err := s.cachedImageHit(digest)
if err != nil {
return nil, err
}
if img != nil {
slog.InfoContext(ctx, "Image cache hit", slog.String("ref", ref), slog.String("digest", digest.String()))
return img, nil
}
slog.InfoContext(ctx, "Image cache miss", slog.String("ref", ref), slog.String("digest", digest.String()))
// Collapse concurrent pulls of the same digest (e.g. several containers of
// one actor, or several actors landing at once). The winning call's ctx
// governs the pull; if it is cancelled the waiters fail too and retry at
// the RPC level.
v, err, _ := s.imageSF.Do(digest.String(), func() (any, error) {
return s.pull(ctx, parsedRef, digest)
})
if err != nil {
return nil, err
}
return v.(*Image), nil
}
// cachedImageHit is the hit side of the hitMu contract: it verifies the
// cached image and records last-use for eviction's LRU ordering, atomic
// with respect to eviction's record removal (removeStaleRecord holds
// hitMu exclusive). Refreshing the mtime also renews the min-age veto, so
// an image in active use cannot age into eviction between this stat and
// the ateom's mount.
func (s *Store) cachedImageHit(digest v1.Hash) (*Image, error) {
s.hitMu.RLock()
defer s.hitMu.RUnlock()
img, err := s.cachedImage(digest)
if err == nil && img != nil {
s.touchRecord(digest)
}
return img, err
}
// cachedImage returns the cached image for digest, or nil if the record or
// any of its layer dirs is missing (in which case the caller re-pulls; only
// the missing layers cost anything).
func (s *Store) cachedImage(digest v1.Hash) (*Image, error) {
b, err := os.ReadFile(s.recordPath(digest))
if errors.Is(err, os.ErrNotExist) {
return nil, nil
} else if err != nil {
return nil, fmt.Errorf("while reading image record for %s: %w", digest, err)
}
var rec imageRecord
if err := json.Unmarshal(b, &rec); err != nil {
return nil, fmt.Errorf("while decoding image record for %s: %w", digest, err)
}
layerDirs := make([]string, len(rec.DiffIDs))
for i, d := range rec.DiffIDs {
diffID, err := v1.NewHash(d)
if err != nil {
return nil, fmt.Errorf("invalid diffID %q in image record for %s: %w", d, digest, err)
}
dir := s.layerDir(diffID)
if _, err := os.Stat(filepath.Join(dir, layerFSDirName)); err != nil {
return nil, nil
}
layerDirs[i] = dir
}
return &Image{Digest: digest, Config: rec.Config, LayerDirs: layerDirs}, nil
}
// pull fetches the image (by its resolved digest, so what is unpacked is
// exactly what was recorded), writes the image record, and then unpacks
// every missing layer into the pool.
//
// The record comes first so that every layer is referenced — and thereby
// safe from eviction — before it can exist on disk. A record therefore
// means "known image, possibly partially present", which readers already
// handle: cachedImage verifies every layer and re-pulls what is missing,
// so an interrupted pull's record is just resumable progress.
func (s *Store) pull(ctx context.Context, parsedRef name.Reference, digest v1.Hash) (*Image, error) {
// Re-check under the flight lock: a racing EnsureImage may have completed
// the pull between our cache miss and winning the singleflight slot.
if img, err := s.cachedImage(digest); err != nil {
return nil, err
} else if img != nil {
return img, nil
}
tStart := time.Now()
digestRef := parsedRef.Context().Digest(digest.String())
img, err := remote.Image(digestRef, s.remoteOpts(ctx, parsedRef)...)
if err != nil {
return nil, fmt.Errorf("in remote.Image: %w", err)
}
cfgFile, err := img.ConfigFile()
if err != nil {
return nil, fmt.Errorf("while reading image config: %w", err)
}
layers, err := img.Layers()
if err != nil {
return nil, fmt.Errorf("while listing image layers: %w", err)
}
if len(cfgFile.RootFS.DiffIDs) != len(layers) {
return nil, fmt.Errorf("image %s config lists %d diffIDs but manifest has %d layers", digest, len(cfgFile.RootFS.DiffIDs), len(layers))
}
diffIDs := make([]string, len(layers))
for i, d := range cfgFile.RootFS.DiffIDs {
diffIDs[i] = d.String()
}
// The full diffID list is known from the config before any unpack; make
// every layer of this pull referenced before it can exist.
rec := imageRecord{Version: 1, Config: cfgFile.Config, DiffIDs: diffIDs}
if err := s.writeRecord(digest, rec); err != nil {
return nil, err
}
// For a multi-arch ref the requested digest is the index digest, but the
// layers unpacked belong to the per-platform child manifest. Record the
// image under the child digest too, so refs pinned either way hit.
var actualDigest *v1.Hash
if actual, err := img.Digest(); err == nil && actual != digest {
actualDigest = &actual
if err := s.writeRecord(actual, rec); err != nil {
slog.WarnContext(ctx, "Failed to record image under platform manifest digest",
slog.String("digest", actual.String()), slog.Any("err", err))
}
}
layerDirs := make([]string, len(layers))
g, gctx := errgroup.WithContext(ctx)
g.SetLimit(layerPullConcurrency)
for i, layer := range layers {
g.Go(func() error {
diffID, err := layer.DiffID()
if err != nil {
return fmt.Errorf("while reading layer diffID: %w", err)
}
if diffID.String() != diffIDs[i] {
// The record must reference exactly what lands on disk.
return fmt.Errorf("layer %d diffID %s does not match config rootfs diffID %s", i, diffID, diffIDs[i])
}
dir, err := s.ensureLayer(gctx, diffID, layer)
if err != nil {
return fmt.Errorf("while unpacking layer %s: %w", diffID, err)
}
layerDirs[i] = dir
// Each completed layer refreshes the record's mtime: a pull
// making progress stays fresh indefinitely; a wedged one ages
// into ordinary LRU eviction. The twin is not touched — the
// primary keeps the layers referenced, and the final rewrite
// recreates the twin if it ages out mid-pull.
s.touchRecord(digest)
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}
// Rewrite the record: eviction may legitimately remove it mid-pull
// (no progress for min-age reads as wedged), and success must never
// leave the just-unpacked layers unreferenced.
if err := s.writeRecord(digest, rec); err != nil {
return nil, fmt.Errorf("while rewriting image record after unpack: %w", err)
}
if actualDigest != nil {
if err := s.writeRecord(*actualDigest, rec); err != nil {
slog.WarnContext(ctx, "Failed to rewrite platform manifest record after unpack",
slog.String("digest", actualDigest.String()), slog.Any("err", err))
}
}
// Never return LayerDirs that are not on disk right now: a vanished
// dir fails the pull into a clean RPC retry instead of a bundle spec
// naming a missing lowerdir. Ordered after the rewrite so even this
// failure path leaves the surviving layers referenced.
for _, dir := range layerDirs {
if _, err := os.Stat(filepath.Join(dir, layerFSDirName)); err != nil {
return nil, fmt.Errorf("layer dir vanished during pull (evicted?): %w", err)
}
}
slog.InfoContext(ctx, "Image pulled into layer cache",
slog.String("digest", digest.String()),
slog.Int("layers", len(layers)),
slog.Duration("took", time.Since(tStart)))
return &Image{Digest: digest, Config: cfgFile.Config, LayerDirs: layerDirs}, nil
}
// ensureLayer makes the unpacked tree for diffID present in the pool,
// collapsing concurrent requests for the same layer across images.
func (s *Store) ensureLayer(ctx context.Context, diffID v1.Hash, layer v1.Layer) (string, error) {
dir := s.layerDir(diffID)
_, err, _ := s.layerSF.Do(layerFlightKey(diffID.Hex), func() (any, error) {
if _, err := os.Stat(filepath.Join(dir, layerFSDirName)); err == nil {
// Refresh the dir mtime inside the flight: retireLayer re-checks
// the mtime in this same flight, so a layer reused here can
// never be renamed away between this stat and the image record
// that will re-reference it.
now := time.Now()
if err := os.Chtimes(dir, now, now); err != nil {
slog.WarnContext(ctx, "Failed to refresh layer mtime on reuse", slog.String("diffid", diffID.String()), slog.Any("err", err))
}
return nil, nil
}
return nil, s.unpackLayerToPool(ctx, diffID, layer)
})
if err != nil {
return "", err
}
return dir, nil
}
// unpackLayerToPool streams the layer (download → decompress → untar) into a
// temp dir and renames it into place, so a layer dir either exists complete
// or not at all.
func (s *Store) unpackLayerToPool(ctx context.Context, diffID v1.Hash, layer v1.Layer) (retErr error) {
tmp, err := os.MkdirTemp(filepath.Dir(s.layerDir(diffID)), ".tmp-"+diffID.Hex[:12]+"-")
if err != nil {
return fmt.Errorf("while creating layer temp dir: %w", err)
}
defer func() {
// No-op once the rename has moved tmp into place.
if _, err := os.Stat(tmp); err == nil {
if rmErr := RemoveAllWritable(tmp); rmErr != nil && retErr == nil {
retErr = fmt.Errorf("while cleaning up layer temp dir: %w", rmErr)
}
}
}()
fsDir := filepath.Join(tmp, layerFSDirName)
if err := os.Mkdir(fsDir, 0o755); err != nil {
return fmt.Errorf("while creating layer fs dir: %w", err)
}
rc, err := layer.Uncompressed()
if err != nil {
return fmt.Errorf("while opening layer stream: %w", err)
}
defer rc.Close()
root, err := os.OpenRoot(fsDir)
if err != nil {
return fmt.Errorf("while opening layer fs dir as os.Root: %w", err)
}
defer root.Close()
// The uncompressed tar stream is the recorded size: an optimistic
// estimate (tar framing vs. block rounding), cheap to capture here.
cr := &countingReader{r: rc}
wh, err := unpackLayer(ctx, cr, root)
if err != nil {
return err
}
// Non-fatal: a missing size file is recovered by layerSize's backfill,
// so a metadata write must not discard a successful unpack.
if err := os.WriteFile(filepath.Join(tmp, layerSizeFileName), []byte(strconv.FormatInt(cr.n, 10)+"\n"), 0o600); err != nil {
slog.WarnContext(ctx, "Failed to record layer size; will backfill lazily",
slog.String("diffid", diffID.String()), slog.Any("err", err))
}
whBytes, err := json.Marshal(wh)
if err != nil {
return fmt.Errorf("while encoding layer whiteouts: %w", err)
}
if err := os.WriteFile(filepath.Join(tmp, layerWhiteoutsFileName), whBytes, 0o600); err != nil {
return fmt.Errorf("while writing layer whiteouts: %w", err)
}
if err := os.Rename(tmp, s.layerDir(diffID)); err != nil {
// A concurrent unpack (another process sharing the pool) may have won;
// its layer is as good as ours.
if _, statErr := os.Stat(filepath.Join(s.layerDir(diffID), layerFSDirName)); statErr == nil {
return nil
}
return fmt.Errorf("while moving layer into pool: %w", err)
}
return nil
}
func (s *Store) writeRecord(digest v1.Hash, rec imageRecord) error {
b, err := json.MarshalIndent(rec, "", " ")
if err != nil {
return fmt.Errorf("while encoding image record: %w", err)
}
path := s.recordPath(digest)
tmp, err := os.CreateTemp(filepath.Dir(path), "."+filepath.Base(path)+".tmp-*")
if err != nil {
return fmt.Errorf("while creating image record temp file: %w", err)
}
defer os.Remove(tmp.Name()) // no-op once the rename succeeds
if _, err := tmp.Write(b); err != nil {
tmp.Close()
return fmt.Errorf("while writing image record: %w", err)
}
if err := tmp.Close(); err != nil {
return fmt.Errorf("while closing image record: %w", err)
}
if err := os.Rename(tmp.Name(), path); err != nil {
return fmt.Errorf("while moving image record into place: %w", err)
}
return nil
}
// remoteOpts assembles the go-containerregistry options for pulls from
// parsedRef's registry.
func (s *Store) remoteOpts(ctx context.Context, parsedRef name.Reference) []remote.Option {
platform := v1.Platform{
Architecture: runtime.GOARCH,
OS: "linux",
}
if s.platform != nil {
platform = *s.platform
}
opts := []remote.Option{
// Propagate caller ctx into go-containerregistry so cancellation tears
// down in-flight layer-blob HTTP requests instead of letting them run
// to completion in background goroutines.
remote.WithContext(ctx),
remote.WithPlatform(platform),
}
registry := parsedRef.Context().Registry.RegistryStr()
if s.authenticator != nil && registryUsesGCPAuth(registry) {
opts = append(opts, remote.WithAuth(s.authenticator))
}
return opts
}
func registryUsesGCPAuth(registry string) bool {
return registry == "gcr.io" || strings.HasSuffix(registry, ".gcr.io") ||
registry == "pkg.dev" || strings.HasSuffix(registry, ".pkg.dev")
}
// parseRef applies the localhost-registry rewrite (kind local registries) and
// permits plain-HTTP pulls for localhost/loopback registries, matching docker
// behavior so local development needs no TLS certs.
func (s *Store) parseRef(ref string) (name.Reference, error) {
rewritten := false
if s.localhostRegistryReplacement != "" {
if newRef := s.rewriteLocalRegistry(ref); newRef != ref {
ref = newRef
rewritten = true
}
}
var nameOpts []name.Option
if rewritten || isLocalRegistry(ref) {
nameOpts = append(nameOpts, name.Insecure)
}
return name.ParseReference(ref, nameOpts...)
}
func registryHost(ref string) string {
parts := strings.SplitN(ref, "/", 2)
reg, err := name.NewRegistry(parts[0], name.Insecure)
if err != nil {
return ""
}
hostPart := reg.Name()
if h, _, err := net.SplitHostPort(hostPart); err == nil {
return h
}
return hostPart
}
func isLocalhostOrLoopback(host string) bool {
if host == "localhost" {
return true
}
if ip := net.ParseIP(host); ip != nil && ip.IsLoopback() {
return true
}
return false
}
func isLocalRegistry(ref string) bool {
// By default docker permits localhost and 127.0.0.0/8; we also permit the
// IPv6 loopback here.
return isLocalhostOrLoopback(registryHost(ref))
}
func (s *Store) rewriteLocalRegistry(ref string) string {
if isLocalRegistry(ref) {
parts := strings.SplitN(ref, "/", 2)
return s.localhostRegistryReplacement + "/" + parts[1]
}
return ref
}