Fixes#783
Adds an `image` source to `ActorTemplate`'s `VolumeSource`: a container
can mount the contents of an OCI image it does not run. This is how
tooling gets into images built by third parties without rebuilding them.
```yaml
spec:
containers:
- name: sandbox
image: docker.io/example/benchmark@sha256:...
command: ["/ate/agent"]
volumeMounts:
- name: agent
mountPath: /ate
volumes:
- name: agent
image:
reference: registry.example.com/agent@sha256:...
```
## How it works
atelet pulls the image through the existing layer cache and records the
volume's layers in the bundle's overlay spec, next to the rootfs layers.
ateom composes the volume inside the bundle — the cached layers with no
writable layer on top, so the mount is read-only — and the container
binds it at the declared path. The volume is composed per container:
containers of one actor may mount the same volume, and each gets its own
mount point inside its own bundle, all backed by the same shared layers.
On resume the volume is re-composed the same way.
References must be digest-pinned, the same rule as container images: a
snapshot is only valid against the exact bytes it was taken with.
On micro-VMs the volume rides the same read-only virtio-fs share as the
container rootfs: ateom stages each composed volume beside the rootfs on
the host, and the guest binds it into the container at the declared
path.
Fixes#957
`ate.actor.lifecycle.operation.duration` carried the pool pair on
suspend and pause only when they failed, so per-pool dashboards saw
those two operations exclusively as failures.
Both workflows record the histogram from a defer that reads the `actor`
variable, and the happy path reassigns it to the finalized record. The
finalize step commits the new state and the cleared `WorkerAssignment`
in one update, so the defer found no assignment and dropped both keys. A
failure returns the pre-finalize record, which still names the worker.
Both now snapshot `lifecycleOpAttrs(...)` just before the finalize step
— the same snapshot-before-clear crash.go does for the crash counter.
Paths that end earlier keep the current computation.
`delete` stays without a pool: it only runs from SUSPENDED or CRASHED,
which already released the worker, so there is none to name.
TestLifecycleOpPoolAttributesOnSuccess drives a real suspend and pause
through the gRPC service; both subtests fail without the fix.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #802 (first PR: the actorIdentity data source; does not close
the issue).
## What changed and why
Adds a systemInfo volume source to ActorTemplate — a read-only volume
whose files are generated by atelet on every Run/Restore, analogous to
Kubernetes projected volumes. The initial data source, actorIdentity,
writes the actor's own name to a configurable relative path:
```
spec:
volumes:
- name: system-info
systemInfo:
dataSources:
# Part 1 (this PR): own-metadata projection, downwardAPI-style
- actorMetadata:
items:
- field: name # enum: name | atespace | uid
path: actor-name
- field: atespace
path: atespace
- field: uid
path: actor-uid
containers:
- name: main
image: app@sha256:...
volumeMounts:
- name: system-info
mountPath: /run/ate
```
Because the files are regenerated before the sandbox starts, they carry
the resumed actor's own values regardless of what checkpointed state it
boots from — the property the old hardcoded /run/ate identity mount
provided, now as an explicit, extensible API that future data sources
(identity JWTs, certificates — see #802) can slot into.
**Behavior change:** the automatic /run/ate/actor-id mount is removed;
actors must opt in by declaring the volume (the e2e identity probe in
this PR is the reference example).
### Reviewer notes:
- Over half the diff is vendored + generated code
(cmd/atelet/internal/third_party/atomicwriter/, atelet.pb.go,
zz_generated.deepcopy.go, the CRD manifest). The hand-written surface is
~700 lines.
- System-info volume roots live under a new ActorPath/system-info/ host
dir, deliberately separate from durable-dir/: the micro-VM durable
machinery snapshots everything under the durable-dir root, and generated
identity files must never be captured into snapshots.
- Supports microVM as well as gVisor.
- The e2e identity suite exercises the new API end-to-end with unchanged
probe binary and assertions. It runs in the kind-cluster CI job (not run
locally).
## Checklist
- [x] Issue is linked above
- [x] Tests pass locally (go test ./...)
- [x] Root-gated tests pass if applicable (N/A — no root-gated packages
touched)
- [x] Documentation updated if behavior changed (docs/api-guide.md:
SystemInfo Volumes section with example)
---------
Co-authored-by: Taahir Ahmed <taahm@google.com>
Make workers a gobal resource, identified by a unique name. Worker APIs
will be implemented in a follow up change.
Part of #730
- [ x ] Tests pass
- [ x ] Appropriate changes to documentation are included in the PR
Finishes up the vision from #715 to have atunnel serve CONNECT on the
ingress path. This will give us the option to hit actors on other ports
besides 80. I haven't wired up atenet router yet because it's
nontrivial; we should do that in a second step so we can have a baseline
for performance
---------
Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
functional_test.go was also split into one file per resource, to match
what we're doing for the RPC handlers too. See #891
I also needed to make some changes to move the tests to a separate
package:
* `NewAteletDialer` now takes `options`, and `WithDialCredentials` lets
a test build its own transport credentials. The fake atelet is reached
over insecure transport. Didn't change existing callers. This is the
only non-test change.
* Moved a few unit tests that are actually functional tests to the
respective file under functionaltest/
Fixes#891
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
ResumeActor returned `FailedPrecondition` both when the fleet had no
free worker and when the actor was not in a resumable state. Only the
former is fixable by suspending the actor to drop its node pinning, and
a client driving that recovery could not tell the two apart.
`assignWorkerAttempt()` now returns `ResourceExhausted` for
`scheduling.ErrNoCapacity`. `FailedPrecondition` was the de-facto
capacity signal, so three router consumers move with it:
- `retryable()`: `ResourceExhausted` is parked, so saturation behaves as
before. `FailedPrecondition` stays retryable on purpose. It no longer
carries saturation, but it does cover states a concurrent operation can
move the actor out of.
- `mapResumeError()`: `ResourceExhausted` now preserves the gRPC
description, keeping saturation at 503 as today. Its former 429 "rate
limited" branch was written speculatively as nothing in substrate
produced the code until now, and the fleet being full is not the caller
sending too many requests.
- `classifyOutcome()`: `no_capacity` moves to `ResourceExhausted`, and
`FailedPrecondition` gets its own label so the capacity bucket stops
collecting non-capacity errors.
`TestResumeActor_RelocatesAfterSuspendFromPaused` covers the recovery
flow this signal exists for: a PAUSED actor pinned to a full node cannot
resume, and after a suspend clears the pinning it schedules onto a
worker on another node.
Fixes#660
- `make test` clean (except the pre-existing macOS-only
`internal/atunnel` failures, untouched by this PR)
- `make fmt` clean
- `go vet ./cmd/...` clean
Fixes#609
The startup stored-worker scan (`enqueueStoredWorkers`) returned on the
first `ListWorkers` error, skipping orphan cleanup until the next
restart. The `#674` workqueue only retries per-key reconciles, not this
initial scan, so retry the scan with capped `wait.Backoff` until it
succeeds or the context is cancelled.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Fixes#608
`crashActor` collected the `releaseWorker` error but still marked the
actor `CRASHED` and cleared its `WorkerAssignment`, stranding the
still-assigned worker with no actor left to reclaim it. On a release
failure, return instead so the actor stays intact and the workflow's
retry reclaims the worker (`releaseWorker` is idempotent).
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
ate.actor.crashes, ate.actor.lifecycle.operation.duration and
ate.scheduler.assignment.duration named a WorkerPool by name alone. A
WorkerPool is namespaced, so same-named pools in different namespaces
merged into one series, and the three could not join the instruments
that already carry both keys.
ateattr.WorkerPoolAttributes now builds the pair, and omits both keys
when no pool is assigned, so a crash before the actor reached a worker
no longer reports an empty-string pool. This changes the series identity
of the three instruments.
Fixes#951
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This makes it clear(er) which methods of store.Interface are used in
which paths.
No functional changes
@juli4n @laoj2 as a first step towards splitting the validation logic
out of the RPC layer and out of the storage layer. This at least gives
us targets for what to pay attention to first.
Actors previously ran in sandboxes sized to the whole node; there was no
way to declare how much CPU/memory a given actor should get. This adds
an explicit, immutable sizing knob on the ActorTemplate and plumbs it
into the sandbox's OCI spec for both the gVisor and micro-VM runtimes.
**API** — `ActorTemplate.spec.resources`
(`*corev1.ResourceRequirements`). The `limits` size the sandbox and are
baked into the immutable spec; the CRD and generated code are
regenerated accordingly.
**internal/sizing** (shared by both runtimes) — new `SandboxSize` value
(`FromLimits` / `VCPUs` / `ApplyToOCISpec`). `ApplyToOCISpec` writes CPU
quota+period and the memory limit onto the OCI spec, and is a no-op when
neither dimension is set, so 0 means "unconstrained". `VCPUs` rounds
milliCPU up to whole
vCPUs.
**Plumbing** — ateapi reads the template limits (`actorResourceLimits` →
`tmpl.Spec.Resources`) and supplies `CpuMilli`/`MemoryBytes` over the
actor RPCs (ateapi → atelet → ateom); ateom applies them — gVisor via
the cgroup leaf (`runsc --cpu-num-from-quota` provisions the sentry vCPU
count), micro-VM via the guest VmConfig. The two fields are carried on
the proto messages.
**Scheduling** — worker capacity is taken from the WorkerPool's
per-worker limits and advertised on the Worker; the scheduler only
places an actor on a worker whose capacity >= the actor's declared
limits. A missing worker or actor
dimension is treated as unconstrained, so placement is never blocked by
absent data.
**Docs & demos** — document the model in `api-guide.md`; the counter,
sandbox, and micro-VM demos declare actor limits, and their WorkerPool
comments now describe the real model (worker limits size the worker pod
+ advertise scheduling capacity; the sandbox itself is sized by
`ActorTemplate.spec.resources`).
- [x] Tests pass (unit tests for sizing + scheduling; e2e suite in
`internal/e2e/suites/sizing` resumes an actor and asserts, via the probe
fixture's `/resources` endpoint, that the running sandbox observes the
declared CPU/memory from the inside)
- [x] Appropriate changes to documentation are included in the PR
RunContractTests is getting large and resource tests are not always
grouped together. So, let's start with splitting them into separate
functions. As they grow, we may consider moving them to their own files,
like we did for functional tests in
https://github.com/agent-substrate/substrate/pull/918
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
* Align ActorSnapshot/ActorSnapshotTag store methods to other CRUD
methods.
* Add common struct types for List<Type> methods to carry parameters and
returned pages.
* Sort methods in store interface to group methods for each resource.
* Remove pessimistic locking from ActorSnapshot/ActorSnapshotTag RPC
handlers.
* Align CreateActorSnapshotTag method to API guideline. It should take
just the resource, no extra top level field.
* Remove the ActorSnapshotRef message.
* Add a `GetActorSnapshotTag` method for the `ActorSnapshotTag`
resource. The request should (as all Get requests) have a single
`ObjectRef` tag field.
* Change `GetActorSnapshotRequest` to have a single `ObjectRef` snapshot
field.
* Add a new field to `Actor,` called `source_snapshot`, which the
following schema:
```
message ActorSnapshotSource {
// If specified, the actor will fork from this snapshot on the first resume.
// Immutable.
ObjectRef tag = 1;
// A reference to the snapshot the tag resolved to at creation time.
// Output only.
ObjectRef snapshot = 2;
// UID of the snapshot the tag resolved to at creation time.
// Output only.
string snapshot_uid = 3;
}
```
See #912 for a more detailed description of the rationale for this
change. Fixes#912
#891
No declaration added, removed, or renamed, import sets
unchanged, and every non-header line is identical to what it replaced.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
No declaration added, removed, or renamed, import sets unchanged, and
every non-header line identical to what it replaced.
https://github.com/agent-substrate/substrate/issues/891
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Since we only have ListWorkers so far, this is just a file rename. The
rest of the handlers should be added to this file, instead of new
handler-specific files, as per #891
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The atelet dial target concatenated the pod IP with ":8085", which
net.Dial rejects for IPv6; grpc.NewClient dials lazily, so it surfaced
as Unavailable on every snapshot and restore. Use net.JoinHostPort.
Also lifts the 8085 literal into internal/atelet.DefaultPort, shared by
atelet's --port flag and the dial target.
Adds Postgres as an alternative persistence backend for ateapi as
proposed in #731. Currently this is opt-in with flags in ateapi or use
the `--store-backend=postgres` option in the install script.
Slack discussion:
https://cloud-native.slack.com/archives/C0B6M3E2J3D/p1785183769252679
Performance benchmark results for Postgres vs Redis:
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
Benchmarks are not included in this PR, but you can reproduce the
results by following the doc above
Limitations and questions:
- This draft does not yet handle migrations, this might come as a follow
up
- It currently uses `testcontainers` for testing Postgres, but as a
result it adds a lot of dependencies
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Part of #477 .
All ATV fields are immutable. SandboxConifg will be frozen into ATV at
creation time.
Adds per-resource store methods and the ateredis package for the AT and
ATV as global resources.
This is a very similar fix to #829
We're also removing the precondition as the storage layer function
arguments and passing it as a wrapper around the closure functions.
https://github.com/agent-substrate/substrate/issues/763
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This reverts the ActorTemplate valueFrom.secretKeyRef support added in
#20 (issue #15).
We don't want actors to have any access to secrets, they will be
injected on the egress route instead. Removing this now so we don't need
to copy secrets into substrate resources.
Instead of requiring custom code for each field, let the callers (i.e
RPC handlers) declare which fields are mutable and implement a generic
`Apply` function that copies fields over, using proto reflection.
The pause image holds the sandbox's namespaces and runs no workload
code. It is an implementation detail of the sandbox, not something actor
authors pick, so it belongs with the sandbox binaries that already moved
off the ActorTemplate onto the cluster-scoped SandboxConfig.
It now travels with those binaries end to end: resolved from the pool's
SandboxConfig, carried on ateletpb.SandboxAssets rather than
WorkloadSpec, and recorded in the per-actor sandbox record so Checkpoint
pins it into the snapshot manifest and Restore rebuilds the sandbox from
the image the snapshot was taken with (the golden's on a DATA_ON_GOLDEN
restore). A record without one is rejected outright rather than pulling
an empty image.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #791 — the last planned piece of the [implementation
plan](https://github.com/agent-substrate/substrate/issues/791#issuecomment-5226674097)
(after #810, #812, #813). #791 stays open until #817 (gVisor
Full-capture → Data-commit conversion, blocked on #790) is done; this PR
covers micro-VM fully and gVisor for scope-matched suspends. Also part
of the actor state machine (#119) and a prerequisite for system upgrade
flows (#473).
`SuspendActor` now accepts a PAUSED actor: instead of checkpointing a
running workload, ateapi dials the atelet on the node holding the pause
snapshot and has it upload the node-local files to object storage, then
finalizes as usual — durable `ActorSnapshot`, `SUSPENDED` status, node
pinning cleared.
Two commits, reviewable independently:
## Commit 1 — atelet: `UploadPausedCheckpoint` RPC (dead code until
commit 2)
- New `AteomHerder` RPC: a pure disk→object-storage copy driven by the
snapshot's self-describing manifest — no ateom involved (the sandbox is
gone).
- **Scope conversion** dispatches per sandbox class
(`narrowFullCaptureToData`): a micro-VM FULL capture narrows to a DATA
upload by carving out `durable-dir.tar` (constant hoisted to
`ateompath`, shared with ateom-microvm); gVisor returns `Unimplemented`
until split checkpoints land (#790); DATA can never widen to FULL; a
scope-less manifest (older atelet) is rejected rather than guessed at.
- **Idempotent retry**: local files gone + remote manifest present ⇒ a
previous invocation committed, succeed; gone on both sides ⇒
unrecoverable (`LOCAL_SNAPSHOT_GONE`, crashes the actor). Upload
failures stay plain retryable errors; the manifest uploads last as the
commit marker, never in parallel.
- The golden atespace is rejected at validation (fully on the
`field.ErrorList` framework): golden actors are never paused.
## Commit 2 — control plane: enable suspend from PAUSED
- `FromPaused` discriminator: PAUSED status, or SUSPENDING with no
worker assignment and a `LocalSnapshotInfo` — the field alone is stale
on resumed-from-pause RUNNING actors, so the nil-assignment conjunct is
load-bearing.
- `MarkSuspendingStep` accepts PAUSED and rejects a Data-captured pause
against a Full commit *before* the actor leaves PAUSED (an upload cannot
fabricate memory), using the `content_scope` recorded at pause (#812)
with an `onPause` fallback.
- `CallAteletSuspendStep` paused branch dials by node
(`DialForAteletOnNode`, #813): missing node record ⇒ crash (the snapshot
can never be found); unreachable atelet ⇒ retryable; atelet's
`LOCAL_SNAPSHOT_GONE` ⇒ crash via `maybeCrashActor`.
`FinalizeSuspendedStep` needed no changes thanks to the #813 hoist.
- Root-cause guard: `MarkPausingStep` rejects pausing golden-atespace
actors.
- Docs: pause states + the new `PAUSED → SUSPENDING` edge in the
architecture state diagram; glossary Suspend entry covers both origins.
## Tests
- **atelet unit**: 10 upload-helper subtests (conversion matrix,
idempotency probe, data-loss crash, retryable upload failure) via a
recording object-storage fake; validation table.
- **control-plane unit**: discriminator table, scope-rejection table
(incl. onPause fallback), paused preconditions (no node ⇒ CRASHED, no
atelet ⇒ `ErrNoAteletOnNode` + still SUSPENDING), golden-pause
rejection, prerequisite matrix updated.
- **functional (envtest)**: `TestSuspendActor_FromPaused` (upload called
with the pause snapshot name, no Checkpoint RPC, SUSPENDED, pinning
cleared, ActorSnapshot at the upload destination) +
retry-after-failed-upload (same destination on retry).
- **e2e (demo suite)**: the lifecycle driver gains a suspend-from-PAUSED
mode; three durable-dir cases — Full/Full, Data/Data, and the micro-VM
Full→Data extraction — assert memory/file counters survive the full
pause→suspend→resume journey and the node pinning is gone.
`go test -race ./...` clean (except the pre-existing macOS-only
`internal/atunnel` unix-socket-path failures, untouched by this PR),
`gofmt`/`go vet` clean, protos regenerated via `hack/protoc.sh`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
UpdateActor now takes an ActorRef, an ActorPrecondition, and a mutate
callback. The store reads the stored actor inside the WATCH transaction,
checks the precondition against that value, and hands it to mutate,
which edits it in place.
The storage layer retries up to 5 times when a concurrent write
invalidates it, re-running mutate against the newer state
I still need to migrate the other callers of store.UpdateActor to avoid
calling GetActor outside of the closure function.
#763
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#565
Replaces standard DNS resolution (`dns:///`) and the forced 1-minute
`MaxConnectionAge` reconnect workaround with a native, event-driven
Kubernetes gRPC resolver (`internal/k8sresolver`). The resolver uses
client-go's `SharedInformerFactory` and `EndpointSliceLister` to watch
`discoveryv1.EndpointSlice` resources in real time.
### Verification
Created a kind cluster to test changes by scaling the deployments up and
down and verified the new servers were connected to by clients.
#### 1. Kind Cluster Scale-Out & EndpointSlice Discovery
Click to view kubectl scale and EndpointSlice output
$ kubectl scale deployment/ate-api-server -n ate-system --replicas=4
deployment.apps/ate-api-server scaled
$ kubectl get pods -n ate-system -l app=ate-api-server
NAME READY STATUS RESTARTS AGE
ate-api-server-6bcd9fddbb-gxxth 1/1 Running 0 3m8s
ate-api-server-6bcd9fddbb-jgwbw 1/1 Running 0 3m8s
ate-api-server-6bcd9fddbb-m8nbx 1/1 Running 0 7m20s
ate-api-server-6bcd9fddbb-vwm2g 1/1 Running 0 7m20s
$ kubectl get endpointslices -n ate-system -l
kubernetes.io/service-name=api
NAME ADDRESSTYPE PORTS ENDPOINTS
api-dmzfc IPv4 443 10.244.0.15,10.244.0.19,10.244.0.31,10.244.0.32
#### 2. Proof of Active gRPC Load-Balancing Across All Replicas
```
[pod/ate-api-server-6bcd9fddbb-gxxth] {"time":"2026-07-29T13:20:49Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
[pod/ate-api-server-6bcd9fddbb-jgwbw] {"time":"2026-07-29T13:20:46Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
[pod/ate-api-server-6bcd9fddbb-m8nbx] {"time":"2026-07-29T13:20:48Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
[pod/ate-api-server-6bcd9fddbb-vwm2g] {"time":"2026-07-29T13:20:47Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
```
Fixes#554
Each workflow becomes a plain method orchestrating typed, idempotent
ensureX steps, e.g.:
```go
func (w *ActorWorkflow) ResumeActor(ctx, ref, boot) (*Actor, bool, error) {
// acquire actor lock, metrics defer — unchanged
actor, tmpl, snapSrc, err := w.loadActorForResume(ctx, ref, boot)
if actor.Status == RUNNING { return actor, false, nil } // explicit no-op
actor, err = w.ensureVolumesCreated(ctx, actor, tmpl)
actor, worker, err = w.ensureWorkerAssigned(ctx, actor, tmpl)
err = w.ensureVolumesAttached(ctx, actor, worker, tmpl)
tele, err = w.ensureAteletRestored(ctx, actor, worker, tmpl, snapSrc)
actor, err = w.finalizeRunning(ctx, ref)
return actor, true, nil
}
```
Each `ensureX`
- early-returns if its postcondition already holds, which is derived
from persisted state only, replacing `IsComplete`
- validates the state machine edge internally, replacing
`CheckPrerequisite`
- takes explicit typed inputs and returns what it produced.
- persists what it changed before returning, returning the stored copy
so the optimistic lock version threads forward. No shared
`*ResumeState`.
There is one behavioral change: worker ownership/eligibility validation
for a re-entered resume now runs before volumes are attached (it used to
run in `CallAteletRestoreStep`'s prerequisite, after attach), so a
resume that is about to crash on a reassigned/ineligible worker no
longer attaches volumes to it first.
- [X] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Part of #791 (PR 3 of the [implementation
plan](https://github.com/agent-substrate/substrate/issues/791#issuecomment-5226674097)).
Two behavior-preserving refactors needed before suspending a PAUSED
actor, which has no worker assignment:
**Dialer** — extract `dialAteletOnNode` from `DialForWorker`'s tail
(byNode index lookup → single-atelet check → per-pod-UID connection
cache → pinned-mTLS dial) and expose `DialForNode(nodeName)` for actors
whose state is pinned to a node without a worker pod (a PAUSED actor's
local snapshot). A node with no atelet in the informer cache returns the
new sentinel `ErrNoAteletOnNode` — documented as retryable (atelet
restarting / informer lag), distinct from the crash-worthy
`ErrWorkerPodNotFound`. `DialForWorker` behavior is unchanged apart from
error wrapping in the zero-atelets case.
**FinalizeSuspendedStep** — hoist finalization (ActorSnapshot record,
`LatestSnapshot`, the `SUSPENDED` flip, clearing
`InProgressSnapshot`/`WorkerAssignment`/`LocalSnapshotInfo`) out of the
`if assignment != nil` worker-freeing branch so it always runs.
Previously, a SUSPENDING actor with no assignment was silently skipped —
the workflow reported success while the actor stayed SUSPENDING forever.
Unreachable today (`CallAteletSuspendStep`'s prerequisite crashes
nil-assignment actors first), but load-bearing for the paused-origin
suspend, where no worker exists by design.
Tests:
- `TestDialForNode`: no atelet on node → `ErrNoAteletOnNode`; two
atelets → error; happy path asserts pod-UID-pinned credentials and
connection caching.
- `TestFinalizeSuspendedStep_NoAssignment`: SUSPENDING actor with
`LocalSnapshotInfo` and an in-progress URI but no assignment →
SUSPENDED, ActorSnapshot created at the URI, `LatestSnapshot` set,
in-progress fields and `LocalSnapshotInfo` cleared.
`go test -race ./cmd/ateapi/...` (incl. the functional suite covering
the running-path suspend end-to-end), `go vet`, `gofmt` clean.
Independent of #810 and #812.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Fixes#773
Some improvements to the way we create, track and propagate snapshot
ids.
* Consolidate snapshot id and name into a single concept.
* Use consistent terminology across the whole stack (i.e. remove
confusion about prefix vs id vs name).
* Get rid of internal only storage to hide the snapshot URI.
Part of #791 (PR 2 of the [implementation
plan](https://github.com/agent-substrate/substrate/issues/791#issuecomment-5226674097)).
Adds `LocalSnapshotInfo.content_scope` and records the template's
`onPause` scope in `FinalizePausedStep`. A later suspend of a PAUSED
actor needs to know what the local pause snapshot actually captured, to
validate the requested scope conversion *before* the actor leaves PAUSED
— even if the template's `onPause` is edited while the actor sits
paused. The snapshot's on-node manifest (#810) remains the authoritative
record at upload time; this control-plane copy is for early validation
only.
- `ateapipb` proto: `SnapshotContentScope content_scope = 3` on
`LocalSnapshotInfo` + regen
- `FinalizePausedStep`: sets `ContentScope` from
`SnapshotsConfig.OnPause` when recording `LocalSnapshotInfo`
- Unset for actors paused before this field existed (consumers fall back
to the template's `onPause`)
Tests:
- New `TestFinalizePausedStep_RecordsContentScope` (OnPause
Data/Full/unset→Full, through a real store with a worker)
- `TestFinalizePausedStep_WorkerGone` now seeds the template the step
reads
- `TestPauseActor`: expected `ContentScope` added — which exposed a
latent test bug: the custom `snapshot_prefix` comparer was asymmetric
(`strings.HasPrefix(y, x)` only); go-cmp requires Comparers to be
symmetric and probes them with swapped arguments, and the new field
shifted the comparison order so the probe finally landed on it. Fixed by
matching the prefix in both directions.
Verified end-to-end on a GKE deployment: a paused actor's Valkey record
now carries `"contentScope": "SNAPSHOT_CONTENT_SCOPE_FULL"` in
`localSnapshotInfo`.
`go test -race ./cmd/ateapi/...`, `go vet`, `gofmt` clean. Independent
of #810; no dependencies.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
`setup-envtest use` with no version re-resolves "latest" against the
remote release index on every run, so the envtest-backed packages needed
network even with fully cached binaries.
This PR pins the control plane to `1.36.x`, and fold the three
per-package `envtest` into `internal/testenv.Start`, which also honors
`-short`: `go test -short ./...` skips the envtest-backed packages and
is the guaranteed-offline path.
Have Claude to help me fix it
Fixes#789
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Closes#706
## Summary
- broker short-lived actor certificates from atelet over a same-node
mTLS Unix socket
- keep the actor private key in atunnel and renew the certificate before
expiry
- authenticate egress CONNECT using the actor certificate instead of
bearer tokens
## Testing
- `make verify`
- `go test -race ./internal/atunnel`
This PR implements the last two metrics from #433.
Right now, we can see that a resume was slow but not where.
`ate.actor.lifecycle.operation.duration` covers the whole ateapi
operation, and `atenet.router.route.duration` covers the edge, but
everything between ateapi-atelet-actors is one block that contains
fetching the manifest, downloading the snapshot, unpack the OCI image,
call to ateom.
We have `rpc.server.call.duration` that gives us the atelet restore
total time, but template, kind, and scope labels are missing, so today
we cannot really pinpoint why/where we have a regressions in latency.
In this PR I am adding per-phase histograms, here's an example of what
we can know after these changes:
```console
ateom_restore 522 ms ###############################
download 8.9 ms #
manifest_fetch 3.2 ms
oci_unpack 2.6 ms
---------------------------------------------------------
total 535 ms
```
It also fixes a gap #683 opened where a `data_on_golden` resume was
labeled identically to a plain one on the lifecycle histogram.
Things folks might want to argue with:
- total as a phase value. Partly duplicates `rpc.server.call.duration`,
but that one has no domain labels and gRPC-specifc. We can drop it, but
it's an inferior operational UX, so I'd rather have it here.
- Phases overlap, they are not a partition of total, because download
runs concurrently with the asset fetch and unpack. I called this out in
the metric description. Do not sum across phases.
- New `ate.snapshot.scope` key rather than a new `ate.snapshot.kind`
value for `data_on_golden`. A new value would collapse local and
external into one bucket, which is the biggest latency difference there
is. This does add a label to the already shipped lifecycle histogram.
Verified on kind, and all e2e suites pass, and the emitted series cover
every kind (golden, latest, local) on both metrics with no unknown
values.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #564 (Part 3)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements Part 3 of #564 by adding telemetry histogram
instrumentation for `ate.scheduler.eligible_workers` in `ateapi`. It
measures unassigned free worker capacity remaining after all scheduling
constraint filters are applied, sampled at every scheduling decision.
## Key Changes:
* Updated `Scheduler.Schedule()` to record eligible candidate workers
per pool (`recordEligibleWorkers`).
Defined `SchedulingConstraintKey` (`ate.scheduling.constraint`) and
constraint classification values (`none`, `required_nodes`, `selector`)
in `internal/ateattr/ateattr.go`.
* Added unit tests in `scheduling_test.go` covering candidate counts,
namespaced attributes, zero-capacity fleet states, empty fleets, sandbox
class mismatches, draining workers, and constraint classifications.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
Fixes#732
* `UpdateActorSnapshot` now carries the resource itself + `update_mask`
* `scope` is now applied via the update mask
* Moved `update_mask` to a separate file, so it can be reused by other
update RPCs.
* Added `uid` and `version` as optional guards.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#564 (Part 2)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements the second part of #564 by adding telemetry counter
instrumentation for `ate.actor.crashes` in `ateapi` and establishing
bounded label validation for operations and crash failure causes.
## Key Changes:
- Updated `maybeCrashActor` and `crashActor` to reference
`ateattr.OperationName*` constants and normalize `opName`.
- Passed explicit failure reasons (`ReasonCorruptedAssignment`,
`ReasonWorkerPodGone`, `ReasonWorkerReassigned`) across
`workflow_resume.go`, `workflow_suspend.go`, and `workflow_pause.go`.
- Instrumented `WorkerPoolSyncer.releaseActorOnDeadWorker` in
`syncer.go` to record `recordActorCrash` when background pod deletion
crashes an actor.
- Instrumented `FinalizePausedStep` in `workflow_pause.go` to record
`recordActorCrash` when `nodeName` is missing during pause finalization.
- Added `ate_actor_crashes` to `PlatformMetricPrefixes` in
`collector_metrics.go`.
- Added pre-creation resource cleanup in `metrics_test.go` to prevent
`AlreadyExists` errors on test reruns.
- Updated `metrics_test.go` to trigger an actor crash via `UpdateActor`
and `ResumeActor`, asserting `ate_actor_crashes` emission and label
presence in OTel collector scrape outputs.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
#732
* `UpdateActorRequest` now carries the resource itself + `update_mask`
* `worker_selector` is now applied via the update mask (making it
possible to clear the worker selector)
* `update_mask` is validated against an allowlist of paths. Currently,
only `worker_selector` is allowed.
* Added `uid` and `version` as optional guards.
I'll update UpdateActorSnapshotTag in a follow-up PR
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Ron Lev