Follow up on issue #1168 and the base PR #1215
In this PR:
**Actor read/lifecycle verbs — completes Actor end-to-end**
* `Get`/`Delete`/`Suspend`/`Pause`/`Resume` `ActorRequest`: dropped the
`+k8s:opaqueType` fence on the actor ref, added `+k8s:required` +
`+k8s:subfield(atespace)=+k8s:required`. Combined with the recursion
into `Validate_ObjectRef` this reproduces exactly what
`resources.ValidateObjectRef` enforced (ref required, atespace required,
name required, both DNS-1123 labels).
* `ListActorsRequest`: tagged to match `ListAtespacesRequest` (atespace
optional + short-name format, `page_size` minimum 1, `page_token` max
length 256).
* All six `validate*Request()` functions are now one-line calls to the
generated validators. With this, every RPC verb on `Actor` and
`Atespace` is fully declarative; `resources.ValidateObjectRef` has no
callers left in `actor.go`.
**Note:**`ListActorsRequest.page_token` now capped at 256 chars (parity
with `ListAtespaces`).
**Testing**
* Existing verb tests updated to the generated error shapes
(`field.ErrorMatcher` with origin matching).
* New coverage: `page_token` boundary cases for `ListActors`;
* `controlapi`, `functionaltest`, `actoridentity`, `store`, and `atepg`
suites all pass; `go vet` and `gofmt` clean.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
---------
Co-authored-by: Michelle Au <msau42@users.noreply.github.com>
A rolling upgrade needs two substrate versions running in one cluster,
split by node: `atelet` and `ateom` speak a node-local protocol with no
cross-version guarantees, so everything on a node has to come from one
build. Today nothing records which build a node runs, and the `atelet`
DaemonSet is a single fixed-name object, so a second version can only
replace the first in place, instead of our porposed node by node rolling
([design](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A)).
This PR is the basis for upgrade. Nodes and the `atelet` DaemonSet get
keyed by an `ate.dev/substrate-version` label whose value is the build
version stamped into the binaries.
### One derivation for the version label
`internal/versionlabel` turns the build version into its two forms in
one place:
| form | grammar | used for |
|---|---|---|
| label value | k8s label value | node labels, DaemonSet labels,
nodeSelectors |
| name suffix | DNS-1123 | the DaemonSet name `atelet-<suffix>` |
A version that is not a valid label value is rejected, because the label
has to match what the `ldflags` stamp put into the binaries through
MAKEFILE and `ko`. It's also exposed shell, so the install script can
use it.
### A DaemonSet per version
`atelet-<suffix>` as the name, the version label on metadata, selector,
and pod template, and a pod nodeSelector on the same label. Two versions
run side by side on disjoint old/new node sets.
### The install becomes version-aware
- `install-ate.sh` reads the version from the same `make ldflags` output
it stamps the binaries with, then fills the manifest placeholders.
- It labels the nodes that exist at install time, nodes come after it
carry no version label yet (so add notes in README). Re-running the
install is idempotent, this is the basis for upgrade runbook later.
- The `README` documents the invariant, how to read the installed
version off the DaemonSet, and that a node added later hosts no workers
until it carries the label. On GKE the node pool label is the birth
default for new nodes - added in `tools/setup-gcp/README.md`.
### Follow-ups
- `ate-setup` need same modification
- minimize the affect for daily developer (pin the `VERSION` as
`{USER}-dev` instead of from git
- the manual rolling-upgrade runbook.
Fix#1270
Ref [design
doc](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A)
Continues the declarative-validation (DV) migration from #1215, covering
the `ActorTemplate` resource, its full spec tree, the create path, and
read/delete verbs.
**Key Changes**
* **DV Tags & Hooks**: Added validation tags and custom hooks across the
`ActorTemplate` spec (`Metadata`, `SandboxConfig`, `SnapshotsConfig`,
`Container`, `Volume`, `Resources`).
* **Create Path**: Replaced hand-written validation with generated
validators in `CreateActorTemplate`. Custom rules (e.g., `on_commit ⊆
on_pause`) are now handled via custom hooks.
* **Read/Delete Verbs**: Converted `Get`, `List`, and `Delete` requests
to use DV.
* **State Updates**: Refactored `atepg` metadata setters to update
in-place.
* Regenerated apitool exemptions (documented 15 previously-exempt
fields).
**Deliberate Behavior Changes**
The gRPC path now strictly enforces CRD rules. Specific tightenings
include:
* Empty `worker_selector` is now rejected.
* `EnvVar.name` is strictly required.
* Negative values in optional enum fields are rejected (`minimum=1`).
* `page_token` is capped at 256 characters (matching other list
requests).
**Deliberately Not Done**
* **No update DV / status tags**: `ActorTemplates` are immutable to
clients. The reconciler updates the status directly against the store,
so tagging the status subtree or adding update validation isn't
necessary right now.
* **`config_name` matching `sandbox_class`**: Enforcing this requires
calling SandboxConfigLister (ServiceImpl), so it's left as a TODO.
**Testing**
* Added ~70 positive and negative test cases in the validator tables.
* Store-backed tests (`TestCreateActorTemplate*`,
`TestUpdateActorTemplateMetadata`, and the `atepg` suite) verified
against real Postgres via testcontainers.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This PR is large, I grouped changes to the following commits:
- 1d5d8f06: moves consumers off the CRD path: demos, ate-setup scripts
and the e2e suites address templates by the actor_template ref.
- e502cf60: Ate API changes: removes actor_template_namespace,
actor_template_name from Actor, ActorAssignment and ActorSnapshot in the
public API, drops the CRD conversion fallback.
- 8f6c1adb: atecontroller: removes the ActorTemplate CRD controller.
Once this PR is submitted, existing demos that still uses CRD will stop
working. Created https://github.com/agent-substrate/substrate/pull/1355
to update existing demos.
Follow up on issue #1168 and the base PR #1215
**Overview:**
Continues the declarative validation (DV) migration from #1215 by
converting the `Worker` resource. This moves immutable-field enforcement
from the storage layer up to the service layer. With `WorkerPoolSyncer`
now using RPCs, nearly all write paths share the exact same validation.
**Key Changes:**
* **Schema (`ateapi.proto`):** Added full DV tags (required, format,
immutable) to `Worker` pod-coordinates and metadata.
`WorkerStatus.state` is now strictly bounded to its enum range.
* **Service Layer:** Replaced ~60 lines of hand-written validation with
1-line generated calls. `CreateWorker` and `UpdateWorker` now scrub
server-owned fields and enforce immutability *before* hitting the store.
* **Storage Layer:** Removed `store.CheckWorkerMutation`.
`atepg.UpdateWorker` is optimized to only clone metadata instead of the
whole worker. Immutability contract tests were moved to the service
layer.
* **Behavior Tweaks (Tightenings):**
* `page_token` is now capped at 256 chars.
* `DeleteOptions.uid` must be a valid UUID.
* Global-ref atespace violations now correctly return `Forbidden`
instead of `Invalid`.
* Immutability checks now natively cover the `WorkerPoolSyncer` path.
* **Testing:** Added comprehensive positive/negative cases for all newly
tagged fields, boundary cases, and moved the immutability test suite.
All suites pass cleanly.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1361
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Remove `ActorTemplateStatus.sandbox_assets` along with the
`SandboxAssets` messages it referenced. Nothing ever wrote the field:
sandbox assets are resolved from the WorkerPool and SandboxConfig
objects at resume time and travel to atelet via ateletpb, so freezing
them into the template status never materialized.
This requires more thought, one option is to add it as a field of
GoldenSnapshotStatus in the future, and make GoldenSnapshotStatus a
repeated field to allow multiple goldens.
certificates.k8s.io/v1beta1 podcertificaterequests and
clustertrustbundles cannot be enabled in place on an existing cluster -
the update is accepted but the APIs never become served, and the
install hangs waiting for ClusterTrustBundles. Warn in the create
cluster docs, show the bring-your-own-cluster flag, and name the
symptom.
On a fresh GCP project nothing documented which IAM bindings atelet
needs: setup-gcp creates them silently, and anyone who cannot run the
tool with project-level IAM permissions - or needs to audit what it
did - had to read cmd/iam.go and cmd/bucket.go. Spell out the exact
members, roles, and resources, the Workload Identity prerequisites,
and the gcloud equivalents, and point the README quickstart at it.
Create/Suspend/Resume Actor workflows now uses the
`Actor.actor_template` ObjectRef when specified, otherwise falls back to
the CRD in cmd/ateapi/internal/controlapi/template_convert.go.
Allow the `apitool validate` linter to define exemptions, then add
exemptions for current validation errors, and then enable this as part
of presubmit (GH action).
The idea is to bundle automation related to the API here. For now, there
is only a single `validate` command that runs a set of lint rules to
enforce that the API adheres to the style guide. This should help us
prevent regressions / deviations when modifying it.
These are not enforced as part of the presubmit checks, although we
might want to enforce that soon (right after we close the existing gaps?
see #1163). There is no support for violation exemptions for now.
In the future, we could add another command to, for example, generate a
reference documentation for the whole API.
Phase 2 of #463, following #735 and #836: `validate-image-cache` drives
eviction through `Store.EvictUnused`, deleting its pre-engine prototype.
Also carries #836's post-merge review follow-ups.
## Tool
- **Prototype evictor deleted** (mtime-sorted tree removal, then
dropping every manifest record — no refcounts, no two-phase rename, no
restore). The tool now asks the engine to reclaim the shortfall below
`--min-free-gb`, so corpus runs exercise production semantics.
`--evict-idle` maps to `WithMinAge`.
- **`--evict-all`**: one-shot flush, no refs file or auth. Unlike `rm
-rf`, running actors' images survive via bundle-spec rooting
(`WithActorsDir`). It is not synchronized with a running atelet (the
engine's locks are per-process): the pool can't be corrupted, but a
layer reused mid-pass can be evicted from under an actor — a
not-yet-mounted start fails once and heals; an already-mounted actor can
take EIO. Documented, warned at runtime; the intended end state is an
atelet-owned flush (RPC or trigger), not a second process in the pool.
- **`--evict-idle` floors at 1m when an actors dir exists**: min-age is
the only protection that applies across processes, so on a live node it
can't be tuned away. Validation hosts (no actors dir) keep full freedom.
- Gate-aware errors in both eviction paths (`ErrIncompleteEnumeration` =
"did nothing, repair the named path" vs per-item errors = stats still
print, exit 1); a 30s cooldown after fruitless passes so workers don't
serialize full engine passes when nothing is evictable.
## Carried #836 follow-ups
- Negative `--image-cache-gc-period` rejected (was silently disabling
the loop); `noteOutcome` extracted with the backoff cadence unit-tested;
engine per-pass line demoted to DEBUG (no-target ticks are now silent);
single-line sentinel wraps; `filepath.Abs` + table test for the
outside-BasePath warning; `runPass`/`Run` covered behind a `gcStore`
seam; flag help and README made truthful (`period=0` still runs startup
recovery; watermark reads ~5 pts above `df`; the total-size cap still
evicts everything evictable under sustained foreign pressure — retention
floor named as the extension).
## Testing
- Unit: `-race` clean; new tests for backoff cadence, flag validation,
path check, `runPass` skip/panic paths, and `Run` (immediate first pass
pinned deterministically; ticking covered separately).
- Kind: full e2e green; gated-pass arc verified live (one ERROR per tick
naming the corrupt record; startup scan gates and recovers; planted
orphan reclaimed at restart); `--evict-all` on a live node evicted 4
unrooted images while all rooted images survived.
- GKE (6 nodes): zero image-cache log lines fleet-wide at default log
level across ticks.
- **Corpus sweep** (7.9 GB volume, always-low-water, `--evict-idle=10s`
on a host with no actors dir, parallel pulls): 64/64 images validated
with two eviction episodes mid-run (62 img / 103 layers / 5.2 GB, then
46 img / 65 layers / 2.7 GB) — continuous eviction under concurrent real
pulls, zero races.
This PR implements the last two metrics from #433.
Right now, we can see that a resume was slow but not where.
`ate.actor.lifecycle.operation.duration` covers the whole ateapi
operation, and `atenet.router.route.duration` covers the edge, but
everything between ateapi-atelet-actors is one block that contains
fetching the manifest, downloading the snapshot, unpack the OCI image,
call to ateom.
We have `rpc.server.call.duration` that gives us the atelet restore
total time, but template, kind, and scope labels are missing, so today
we cannot really pinpoint why/where we have a regressions in latency.
In this PR I am adding per-phase histograms, here's an example of what
we can know after these changes:
```console
ateom_restore 522 ms ###############################
download 8.9 ms #
manifest_fetch 3.2 ms
oci_unpack 2.6 ms
---------------------------------------------------------
total 535 ms
```
It also fixes a gap #683 opened where a `data_on_golden` resume was
labeled identically to a plain one on the lifecycle histogram.
Things folks might want to argue with:
- total as a phase value. Partly duplicates `rpc.server.call.duration`,
but that one has no domain labels and gRPC-specifc. We can drop it, but
it's an inferior operational UX, so I'd rather have it here.
- Phases overlap, they are not a partition of total, because download
runs concurrently with the asset fetch and unpack. I called this out in
the metric description. Do not sum across phases.
- New `ate.snapshot.scope` key rather than a new `ate.snapshot.kind`
value for `data_on_golden`. A new value would collapse local and
external into one bucket, which is the biggest latency difference there
is. This does add a label to the already shipped lifecycle histogram.
Verified on kind, and all e2e suites pass, and the emitted series cover
every kind (golden, latest, local) on both metrics with no unknown
values.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The fixed 30-minute idle guard made nothing evictable while a fast corpus
filled a small disk (32-VM hicard sweep: local SSDs filled in ~12 minutes,
then every remaining image failed ENOSPC). Expose it as --evict-idle.
Content-addressed pool of unpacked image layers, shared by every actor on
the node; actor rootfs becomes an overlayfs mount (cached layers as read-
only lowers, bundle-local upper) instead of a full re-untar per run.
atelet (no capabilities) pulls and unpacks; the privileged ateoms finalize
whiteouts and mount. Tag refs resolve via one HEAD and become cacheable;
pull memory is O(stream buffers); the cache survives restarts.
Phase 1 of #463. Fixes#437, #166, #228.
Validated: kind + GKE counter demos (gvisor and microvm), suspend/resume
(oci_unpack ~3ms vs ~15-20s), 411 SWE-bench-scale images pulled and
unpacked with 0 failures, root-gated unit tests for the privileged paths.
Known gaps: no GC yet (Phase 2, see internal/imagecache/README.md);
upgrade ordering — deploy new ateoms before/with the new atelet.
Best practices is not to put a -deployment deployment, -role role, etc;
the kind already declares the kind we don't need it in the name as well.
Additionally we did not do it consistently anyways.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR introduces a new OpenTelemetry histogram
`"atelet.snapshot.size"` from `atelet` over the existing
OTLP path. It records the uncompressed size in bytes of each gVisor
snapshot image written during checkpoint -- `checkpoint.img` ,
`pages.img` and `pages_meta.img` -- tagged by image `kind` and
`ActorTemplate`. This surfaces how large actor snapshots are and which
`ActorTemplate`s are expensive to checkpoint/restore.
Add a well-defined dashboard for snapshot metrics that contains 3
charts.
- snapshot image size p99 by `ActorTemplate`
- snapshot image size p50/p95/p99 (all kinds)
- snapshot QPS by `ActorTemplate`
<img width="996" height="717" alt="image"
src="https://github.com/user-attachments/assets/99bbab46-37dc-476e-bfd9-cdffeb174b4b"
/>
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR introduces a new OpenTelemetry histogram
`"atenet.router.route.duration"` from `atenet-router` over the existing
OTLP path. It measures substrate overhead -- from envoy receiving a
request to envoy sending the request to the the resolved worker
endpoint, excluding actor compute and the response. This is the
`Substrate E2E latency` that under our control.
Add a well-defined dashboard for E2E metrics that contains 6 charts.
- Substrate routing latency — P50/P95/P99
- Substrate routing latency P99 - by stages
- Substrate routing latency P99 — by ActorTemplate
- Substrate routing QPS - by Status
- Substrate E2E (full round-trip) Latency — P50/P95/P99
- Substrate E2E (full round-trip) QPS — by Response Code
<img width="1601" height="894" alt="image"
src="https://github.com/user-attachments/assets/be6fc572-dfd8-4ef0-a0a9-a77f85f2ecf8"
/>
Fixes b/508613998
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
Benjamin Elder