Restore timing breakdown is the only unsampled per-actor latency record
we emit, and since traces are head-sampled at 1% on the data plane, and
actor identity is not available from metric labels on purpose.
This adds this to logs:
Before:
```json
{"msg":"Restore timing breakdown",
"actor":{"type":"*ateapipb.Actor","atespace":"demo","name":"counter-1"},
"download":310000000,"oci_unpack":50000000,"ateom_restore":60000000,"total":420000000}
```
After:
```json
{"msg":"Restore timing breakdown",
"ate.atespace":"ate-demo-counter-substrate","ate.actor.name":"join-probe",
"ate.actor.uid":"3a5b4e86-1a77-4da7-84c4-c0993d9c83ec",
"ate.template.atespace":"ate-demo-counter-substrate","ate.template.name":"counter",
"ate.snapshot.scope":"full","ate.snapshot.kind":"golden","ate.sandbox.class":"gvisor",
"ate.actor.restore.duration.volume_mount":0.000004583,
"ate.actor.restore.duration.manifest_fetch":0.006111636,
"ate.actor.restore.duration.sandbox_assets":0.000098958,
"ate.actor.restore.duration.download":0.025386544,
"ate.actor.restore.duration.oci_unpack":0.000954919,
"ate.actor.restore.duration.ateom_restore":0.147448298,
"ate.actor.restore.duration.total":0.181189689,
"trace_id":"c9d7ea9a5f90a5eb4399bfa0b966d5ac","span_id":"89280ba761137c8e"}
```
The duration keys are the `ate.actor.restore.duration` instrument's name
plus an `ate.snapshot.phase` value, so a log key and the histogram it
mirrors are literally the same identifier in source, which is also why
they're seconds.
Also adds `ateattr.ActorLogAttrs`, the component-log twin of
`ActorLogLabels`, held to the same key set by a test.
Verified on kind:
- the ateom Actor restoring record for the same actor carries the
identical `ate.actor.uid` and `trace_id`, this was impossible before
- `ate_actor_restore_duration_seconds_sum{phase="total"} = 0.181189689`,
matching the log to the nanosecond
- induced a download failure by deleting a snapshot object: record sets
`ate.failure.reason:` `FAILED_GET_EXTERNAL_OBJECT` and correctly omitted
`ateom_restore`; the metric for the same restore shows the same six
phases
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#783
Adds an `image` source to `ActorTemplate`'s `VolumeSource`: a container
can mount the contents of an OCI image it does not run. This is how
tooling gets into images built by third parties without rebuilding them.
```yaml
spec:
containers:
- name: sandbox
image: docker.io/example/benchmark@sha256:...
command: ["/ate/agent"]
volumeMounts:
- name: agent
mountPath: /ate
volumes:
- name: agent
image:
reference: registry.example.com/agent@sha256:...
```
## How it works
atelet pulls the image through the existing layer cache and records the
volume's layers in the bundle's overlay spec, next to the rootfs layers.
ateom composes the volume inside the bundle — the cached layers with no
writable layer on top, so the mount is read-only — and the container
binds it at the declared path. The volume is composed per container:
containers of one actor may mount the same volume, and each gets its own
mount point inside its own bundle, all backed by the same shared layers.
On resume the volume is re-composed the same way.
References must be digest-pinned, the same rule as container images: a
snapshot is only valid against the exact bytes it was taken with.
On micro-VMs the volume rides the same read-only virtio-fs share as the
container rootfs: ateom stages each composed volume beside the rootfs on
the host, and the guest binds it into the container at the declared
path.
Phase 2 of #463, following #735 and #836: `validate-image-cache` drives
eviction through `Store.EvictUnused`, deleting its pre-engine prototype.
Also carries #836's post-merge review follow-ups.
## Tool
- **Prototype evictor deleted** (mtime-sorted tree removal, then
dropping every manifest record — no refcounts, no two-phase rename, no
restore). The tool now asks the engine to reclaim the shortfall below
`--min-free-gb`, so corpus runs exercise production semantics.
`--evict-idle` maps to `WithMinAge`.
- **`--evict-all`**: one-shot flush, no refs file or auth. Unlike `rm
-rf`, running actors' images survive via bundle-spec rooting
(`WithActorsDir`). It is not synchronized with a running atelet (the
engine's locks are per-process): the pool can't be corrupted, but a
layer reused mid-pass can be evicted from under an actor — a
not-yet-mounted start fails once and heals; an already-mounted actor can
take EIO. Documented, warned at runtime; the intended end state is an
atelet-owned flush (RPC or trigger), not a second process in the pool.
- **`--evict-idle` floors at 1m when an actors dir exists**: min-age is
the only protection that applies across processes, so on a live node it
can't be tuned away. Validation hosts (no actors dir) keep full freedom.
- Gate-aware errors in both eviction paths (`ErrIncompleteEnumeration` =
"did nothing, repair the named path" vs per-item errors = stats still
print, exit 1); a 30s cooldown after fruitless passes so workers don't
serialize full engine passes when nothing is evictable.
## Carried #836 follow-ups
- Negative `--image-cache-gc-period` rejected (was silently disabling
the loop); `noteOutcome` extracted with the backoff cadence unit-tested;
engine per-pass line demoted to DEBUG (no-target ticks are now silent);
single-line sentinel wraps; `filepath.Abs` + table test for the
outside-BasePath warning; `runPass`/`Run` covered behind a `gcStore`
seam; flag help and README made truthful (`period=0` still runs startup
recovery; watermark reads ~5 pts above `df`; the total-size cap still
evicts everything evictable under sustained foreign pressure — retention
floor named as the extension).
## Testing
- Unit: `-race` clean; new tests for backoff cadence, flag validation,
path check, `runPass` skip/panic paths, and `Run` (immediate first pass
pinned deterministically; ticking covered separately).
- Kind: full e2e green; gated-pass arc verified live (one ERROR per tick
naming the corrupt record; startup scan gates and recovers; planted
orphan reclaimed at restart); `--evict-all` on a live node evicted 4
unrooted images while all rooted images survived.
- GKE (6 nodes): zero image-cache log lines fleet-wide at default log
level across ticks.
- **Corpus sweep** (7.9 GB volume, always-low-water, `--evict-idle=10s`
on a host with no actors dir, parallel pulls): 64/64 images validated
with two eviction episodes mid-run (62 img / 103 layers / 5.2 GB, then
46 img / 65 layers / 2.7 GB) — continuous eviction under concurrent real
pulls, zero races.
Add the `ate.imagecache.requests` counter.
**`ate.imagecache.outcome`** (new key in `internal/ateattr`)
| Value | Meaning |
|---|---|
| `hit` | the node holds a complete image record: each layer directory
that the record names is present |
| `miss` | the lookup must pull |
| `error` | the lookup failed; the only outcome that carries
`error.type` |
| `cancelled`, `timeout` | the caller gave up, so the cache is not at
fault |
A failed lookup is neither a hit nor a miss, so it gets its own outcome,
as
`no_free_worker` does on `ate.scheduler.outcome`. `cancelled` and
`timeout` are
outcomes for the same reason they are on `ate.router.outcome`. The hit
ratio is
therefore `hit / (hit + miss)`, with failures and abandoned lookups out
of the
denominator.
**`error.type`** — set only on the `error` outcome.
| Value | Meaning |
|---|---|
| `404`, `401`, `429`, ... | the registry rejected the request; its own
HTTP status, reported verbatim |
| `_OTHER` | the failure carries no status of its own |
Fixes#831
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Phase 2 of #463, continuing from #735 (the eviction engine): the
watermark-driven loop in atelet that turns eviction on, plus the flags
that govern it. After this: observability (metrics/spans), the ops-tool
swap, and the e2e suite.
## What
- A serialized, panic-recovered pass every `--image-cache-gc-period`
(default 5m; `0` disables the loop entirely). Each tick: `statfs` the
cache volume + sum the pool's recorded sizes, compute a byte target —
down to `--image-cache-low-percent` when usage crossed
`--image-cache-high-percent`, and/or down to `--image-cache-max-bytes` —
and hand it to `Store.EvictUnused`.
- **The target is capped at the pool's own size.** The cache is one
tenant of a shared volume; uncapped kubelet-style watermark math on a
pressured volume asks a tiny cache to fix pressure it didn't cause,
evicting everything every tick.
- **Gated passes are detected by contract, not inference**: the engine
exports `ErrIncompleteEnumeration` (wrapped into every gate return), and
`classifyGCPass` — a pure, table-tested function — maps each pass to
skipped/shortfall/complete/quiet. A gated pass logs ERROR "pass skipped"
and never feeds the shortfall backoff.
- **Genuine shortfall warns with backoff** then decays to a periodic
reminder: on a volume under foreign pressure, shortfall is the steady
state and must not ERROR every tick.
- atelet passes `WithActorsDir(ateompath.ActorsDir)` so the root set
sees placed actors (the wiring #735's notes required); `ateompath` gains
the `ActorsDir` constant.
- The first pass runs immediately at boot (a node starting under disk
pressure shouldn't wait a full period), and a `CacheSize` failure skips
the pass explicitly rather than silently zeroing the target.
- `--image-cache-gc-dry-run` computes and logs every decision while
deleting nothing — the production soak mechanism. A cache dir outside
`BasePath` logs a warning (its watermarks would measure a different
volume than actor state).
- README: the GC section's "loop lands next" intro replaced with the
loop/flag documentation.
## For reviewers: the enabled-by-default question
Defaults ship **enabled** (5m / 85% / 80%, mirroring kubelet so operator
intuition transfers), with dry-run as the opt-out soak path. The
alternative is dry-run-by-default for a release. The target cap removes
the known pathological case; happy to flip the default if you'd rather
soak first.
## Testing
- `imagegc_test.go`: watermark/cap target-math table (including the
capped-target cases), flag validation.
- Loop + engine together, Kind: full gVisor e2e suite green under
aggressive flags (15s period, 1-byte cap); planted orphan ignored by
periodic passes and reclaimed at startup; shortfall backoff observed
against an impossible cap; no enumeration-gate false positives.
- GKE (6 nodes): idle pool evicted under aggressive watermarks,
subsequent resumes cold-pulled cleanly with snapshot memory continuity;
flags restored to defaults after.
Modify tests in imagecache and tarutil to pass in environments with
restrictive umask (like 0027).
Fixes: #830
TAG=agy
CONV=c647bc5d-fac0-49c8-96e1-2aeb3de188aa
- [ X] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Phase 2 of #463 (image-cache GC + observability): this is the eviction
engine itself — the root set, the eviction pass, and startup orphan
recovery — assembled from the foundations merged in #650
(sizes/last-use), #656 (spec digest), #658 (retirement primitives), and
#717 (record-first pull). What remains after this is the watermark loop
+ flags that turn it on, then metrics and e2e.
**Inert in production**: nothing calls `EvictUnused` yet — the watermark
GC loop and its flags come in the next PR. The only behavior `New` gains
is the startup orphan scan.
## What
- **`Store.InUse()`** — the root set: scans bundle overlay specs under
the actors dir (opt-in via `WithActorsDir`) for image digests, layer
dirs, and exact layer-set signatures. The exact-set rule roots the
multi-arch twin record and digestless (pre-`imageDigest`) specs' records
without making every subset image unevictable. Returns `(RootSet,
error)`: a non-nil error means the root set may be incomplete, and
deleting callers must do nothing.
- **`Store.EvictUnused(ctx, targetBytes, dryRun)`** — LRU eviction of
unprotected images and the layers their removal leaves unreferenced.
Min-age vetoes; per-victim re-checks under `hitMu` held exclusive
against the cache-hit touch; two-phase deletion via `retireLayer`, with
the slow `RemoveAll` outside all locks. If any layer of a deleted record
must be kept (still referenced, rooted, fresh, or failed to retire), the
record is atomically restored from bytes captured at listing time — a
kept layer is never left unreachable.
- **`Store.RecoverOrphans`**, called once from `New` — reclaims layers
no record references. Startup is the one moment this scan is race-free
(no pull can be in flight); a failure never fails `New`.
- **Everything fails toward retention.** Both passes skip entirely, at
ERROR naming the culprit, when their inputs can't be fully enumerated:
an unreadable/undecodable image record, a garbled diffID inside a valid
record, or an unenumerable root set (actors dir, bundles dir, or bundle
spec). Refcounts and roots from partial data would retire layers that
unread records still reference or running actors still mount — and for a
long-running actor, the bundle spec is the only protection (its record
mtime can be arbitrarily old, so min-age is no backstop). Missing
dirs/specs remain legitimate emptiness, not errors.
- **Eviction never writes into the live pool.** Retired layers are
credited from the size file that rides along with the rename-aside
(O(1), read-only, consistent with `CacheSize`); the size-file backfill
exists only on the `CacheSize` path, where it bumps the dir mtime rather
than restoring it (restoring could rewind a concurrent reuse-touch).
- Per-item log lines are Debug behind an `Enabled` gate (level control
via #677); pass summaries, evictions, and restores are Info.
- `internal/imagecache/README.md` now documents the GC: pass structure,
protection rules, failure direction, startup recovery, and two-phase
deletion, plus the on-disk layout entries (`size`, `.tmp-*`, `.rm-*`)
that shipped with #650/#658.
## Notes for reviewers
- The gc tests reuse the helpers merged with #650 (`backdate`,
`mustEnsure`) and #717's gated-registry harness — no duplicated symbols.
The slow-pull gated test gains the eviction-veto assertion deferred from
#717.
- Dry-run mutates nothing, including size-file backfills
(`layerSizeReadOnly`).
- A `Store` built without `WithActorsDir` has an empty root set
(deliberate for tools/tests).
- Carried to the wiring PR: pass `WithActorsDir` in atelet; handle
`EvictUnused`'s error before any shortfall accounting (a gated pass
returns zero stats plus an error and must not be logged as a shortfall);
and surface skipped passes as a counter in the observability PR so a
wedged node alerts rather than scrolling ERROR lines.
## Testing
- `go test -race ./internal/imagecache/`: every veto, shared-layer
survival, both restore-on-keep shapes, orphan reclaimed at startup and
ignored by the periodic pass, all enumeration gates — unreadable record
(shared-layer scenario), undecodable record, garbled diffID, and the
three unreadable root-set shapes (actors dir / bundles dir / spec, each
against a rooted image with an arbitrarily old record) — dry-run purity,
25-iteration ensure-vs-evict race loop.
- Kind, via the integration branch that stacks the watermark loop on
this PR (the loop and flags are **not** part of this PR): full gVisor
e2e suite green; a planted backdated orphan layer (no size file) was
reclaimed at atelet startup with the expected log lines, its size
credited read-only from the retired dir.
- GKE (6 nodes), same integration branch: demo lifecycle +
snapshot-continuity suites green; under aggressive watermarks the idle
pool was evicted and subsequent actor resumes cold-pulled cleanly. No
enumeration-gate false positives on either cluster.
Fourth slice of #463 Phase 2, and the semantic pivot of the GC design:
the pull
now writes the image record **before** unpacking, touches it as each
layer
completes, rewrites it after the loop, and re-verifies every returned
layer dir.
Why record-first: the record's diffID list (known from the config before
any
download) is what the eviction engine's refcounts will trust — writing
it first
makes every layer referenced before it can exist, so pulls need no
separate
protection mechanism, and the protection survives atelet restarts. The
per-layer touch makes liveness progress-based: a wedged pull ages out
through
ordinary LRU; the end-of-pull rewrite means success always implies a
record;
the re-verify turns the residual race into a clean RPC retry instead of
a
bundle spec naming a missing lowerdir.
Also new: the config's diffID list is cross-checked against the
manifest's
layer count up front and against each layer's actual diffID as it lands,
so a
record can never reference something different from what is on disk.
**Note for @dberkov**: this inverts Phase 1's record-last ordering,
which
appears incidental (amendment 4 in the Phase 2 comment on #463 — please
contradict if there was intent). A record now explicitly means "known
image,
possibly partially present", which is what every reader already handles:
`cachedImage` verifies each layer and re-pulls only gaps. The one
behavior
change observable without GC: an interrupted pull leaves a valid partial
record — resumable progress — instead of nothing.
**Testing**: a gated registry holds real pulls mid-flight by blocking
chosen
blob downloads — interrupted pull leaves a resumable record and a retry
completes it; a record deleted mid-pull is rewritten on success;
per-layer
completions freshen the record while later layers are still gated; a
yanked
layer dir fails the pull with a retryable error. `go test -race` green.
Second foundation slice of #463 Phase 2 (GC), independent of #650 and
inert on its
own: nothing reads the new field and nothing deletes anything.
- **`OverlaySpec.ImageDigest`** (optional, `omitempty`): the manifest
digest the
bundle's image ref resolved to, populated in `prepareOCIDirectory`. The
GC
root-set scan will use it to root the whole image record while the
bundle
exists. Compatible in both directions: ateom consumers never read the
field
(old binaries drop the unknown JSON key), and specs written by older
atelets
keep parsing with the field empty — so there is no deploy-ordering
constraint.
- **Atomic `WriteSpec`** (temp file + rename in the bundle dir): the
root-set scan
will read specs concurrently with bundle preparation, and a torn spec
would
under-report the layers an actor is using — an error in the dangerous
direction (toward deleting in-use layers). With the rename, a reader
sees the
complete old spec or the complete new one, never a fragment.
Landing early matters: every bundle written from now on carries the
digest,
shrinking the digestless-spec population the GC's compatibility fallback
has to
cover.
**Testing**: round-trip + old-spec compat and no-temp-files-left unit
tests;
`go test -race` green on both affected packages.
Third foundation slice of #463 Phase 2 (GC), independent of #650 and
#656. The
primitives are uncalled until the eviction engine PR; the only live
behavior is
the startup sweep, which only ever sees `.rm-*` dirs the engine will
create.
- **`retireLayer`**: evicts a layer with one atomic rename to a `.rm-*`
name
inside the layer singleflight; the slow `RemoveAll` is the caller's job,
outside all locks. Existence/mtime pre-flight runs *outside* the flight
(a
retire must never block behind an in-progress download) and both checks
are
re-run inside it before the rename. Returns gone/vetoed/retired so the
engine
can distinguish "nothing there" from "must keep".
- **Reuse interlock**: `ensureLayer` now refreshes the layer dir mtime
inside
the same flight, so a retirement and a reuse cannot interleave — either
the
retire wins (the pull re-unpacks) or the touch wins (the retire vetoes).
- **Startup sweep**: `.rm-*` dirs (a crash between rename and removal)
are
reclaimed alongside the existing `.tmp-*` dirs, via `RemoveAllWritable`
since
layer trees legitimately contain read-only content.
- **`isLayerDirName`** validates names read from disk or records.
Divergence from the prototype: `retireLayer` reports no freed size —
size
crediting belongs to the engine's call sites and arrives with it.
**Testing**: unit tests for the status contract, startup-sweep recovery
and
non-interference, and a retire-vs-pull race test; `go test -race` green.
First slice of #463 Phase 2 (GC): the two facts eviction will need,
persisted in the
filesystem so they survive atelet restarts. Inert on its own — no
behavior change for
any current caller, and nothing deletes anything.
- **Layer sizes**: unpack counts the uncompressed tar stream and writes
a per-layer
`size` file before the atomic rename (best-effort; a missing file is
healed by a
lazy backfill). Backfill covers layers unpacked by older atelets: one
walk, once
ever, skipping unreadable dirs (atelet has no `CAP_DAC_READ_SEARCH`) and
preserving
the dir mtime.
- **Last use**: cache hits touch the image record's mtime — the
persisted LRU
timestamp — under a new `hitMu` (shared side only; the eviction pass
will be its
exclusive holder).
- **`Store.CacheSize()`**: sums recorded sizes; the accounting behind
the future
`--image-cache-max-bytes`.
On-disk changes are purely additive, so no cache-layout version bump and
no
migration: an older atelet reads this cache dir fine (ignores `size`
files), and
this atelet reads a pre-upgrade dir fine (backfill). Rollback-safe.
**Testing**: unit tests for size recording, backfill (incl. the
unreadable-dir
contract and mtime preservation), hit-path touch, and `CacheSize`; `go
test -race`
green standalone; the demo e2e suite passes unchanged on Kind against
this build,
with `size` files and record-mtime updates verified on the node.
Fix#501
Tests that need root (overlay mounts, mknod, `trusted.*` xattrs, ...)
call `roottest.Require(t, ...)` from
[internal/roottest](internal/roottest) as their first statement. They
skip in a plain `go test ./...`; CI reruns every package whose tests
import that package under `sudo`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Layer tars omit parents that exist in lower layers; unpack fabricates them
(root:root 0755) and overlayfs takes merged dir attrs from the top-most
layer containing the dir — so a fabricated parent shadowed real lower-layer
metadata (/tmp's 1777, /root's 0700). Record implicit dirs in the layer
metadata at unpack, and at compose repair the merged view from the top-most
non-implicit provider in the image's chain; the chown/chmod copy-ups land
in the bundle's private upper, never in the shared pool. Residual gaps
(mtimes, xattrs, implicit-everywhere dirs) documented in the README.
Addresses PR #467 review feedback (phantom parent dirs shadowing lower-
layer directory metadata).
Startup recovery swept layer unpack temp dirs but not writeRecord's
".<digest>.json.tmp-*" files left by a crash between create and rename.
Addresses PR #467 review feedback.
mount(2) copies its option string through a single page, which caps
digest-derived lowerdir chains (~114 bytes per layer path) at roughly 34
layers and fails with a bare EINVAL beyond that. Appending lowerdirs one
fsconfig(2) call at a time removes the aggregate limit structurally, and
failed mounts now carry the kernel's fs-context error log instead of an
opaque errno. Adds a 64-layer regression test that asserts the joined
paths exceed one page before mounting.
Minimum supported kernel: Linux 6.5 (overlayfs "lowerdir+"). All current
GKE channels meet it: Stable runs COS 121 LTS (kernel 6.6), Regular and
Rapid run COS 125/129 (kernel 6.12).
Content-addressed pool of unpacked image layers, shared by every actor on
the node; actor rootfs becomes an overlayfs mount (cached layers as read-
only lowers, bundle-local upper) instead of a full re-untar per run.
atelet (no capabilities) pulls and unpacks; the privileged ateoms finalize
whiteouts and mount. Tag refs resolve via one HEAD and become cacheable;
pull memory is O(stream buffers); the cache survives restarts.
Phase 1 of #463. Fixes#437, #166, #228.
Validated: kind + GKE counter demos (gvisor and microvm), suspend/resume
(oci_unpack ~3ms vs ~15-20s), 411 SWE-bench-scale images pulled and
unpacked with 0 failures, root-gated unit tests for the privileged paths.
Known gaps: no GC yet (Phase 2, see internal/imagecache/README.md);
upgrade ordering — deploy new ateoms before/with the new atelet.
Krisztian F