* Align ActorSnapshot/ActorSnapshotTag store methods to other CRUD
methods.
* Add common struct types for List<Type> methods to carry parameters and
returned pages.
* Sort methods in store interface to group methods for each resource.
* Remove pessimistic locking from ActorSnapshot/ActorSnapshotTag RPC
handlers.
* Align CreateActorSnapshotTag method to API guideline. It should take
just the resource, no extra top level field.
* Remove the ActorSnapshotRef message.
* Add a `GetActorSnapshotTag` method for the `ActorSnapshotTag`
resource. The request should (as all Get requests) have a single
`ObjectRef` tag field.
* Change `GetActorSnapshotRequest` to have a single `ObjectRef` snapshot
field.
* Add a new field to `Actor,` called `source_snapshot`, which the
following schema:
```
message ActorSnapshotSource {
// If specified, the actor will fork from this snapshot on the first resume.
// Immutable.
ObjectRef tag = 1;
// A reference to the snapshot the tag resolved to at creation time.
// Output only.
ObjectRef snapshot = 2;
// UID of the snapshot the tag resolved to at creation time.
// Output only.
string snapshot_uid = 3;
}
```
See #912 for a more detailed description of the rationale for this
change. Fixes#912
#891
No declaration added, removed, or renamed, import sets
unchanged, and every non-header line is identical to what it replaced.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
No declaration added, removed, or renamed, import sets unchanged, and
every non-header line identical to what it replaced.
https://github.com/agent-substrate/substrate/issues/891
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Since we only have ListWorkers so far, this is just a file rename. The
rest of the handlers should be added to this file, instead of new
handler-specific files, as per #891
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The atelet dial target concatenated the pod IP with ":8085", which
net.Dial rejects for IPv6; grpc.NewClient dials lazily, so it surfaced
as Unavailable on every snapshot and restore. Use net.JoinHostPort.
Also lifts the 8085 literal into internal/atelet.DefaultPort, shared by
atelet's --port flag and the dial target.
Adds Postgres as an alternative persistence backend for ateapi as
proposed in #731. Currently this is opt-in with flags in ateapi or use
the `--store-backend=postgres` option in the install script.
Slack discussion:
https://cloud-native.slack.com/archives/C0B6M3E2J3D/p1785183769252679
Performance benchmark results for Postgres vs Redis:
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
Benchmarks are not included in this PR, but you can reproduce the
results by following the doc above
Limitations and questions:
- This draft does not yet handle migrations, this might come as a follow
up
- It currently uses `testcontainers` for testing Postgres, but as a
result it adds a lot of dependencies
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Part of #477 .
All ATV fields are immutable. SandboxConifg will be frozen into ATV at
creation time.
Adds per-resource store methods and the ateredis package for the AT and
ATV as global resources.
This is a very similar fix to #829
We're also removing the precondition as the storage layer function
arguments and passing it as a wrapper around the closure functions.
https://github.com/agent-substrate/substrate/issues/763
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This reverts the ActorTemplate valueFrom.secretKeyRef support added in
#20 (issue #15).
We don't want actors to have any access to secrets, they will be
injected on the egress route instead. Removing this now so we don't need
to copy secrets into substrate resources.
Instead of requiring custom code for each field, let the callers (i.e
RPC handlers) declare which fields are mutable and implement a generic
`Apply` function that copies fields over, using proto reflection.
The pause image holds the sandbox's namespaces and runs no workload
code. It is an implementation detail of the sandbox, not something actor
authors pick, so it belongs with the sandbox binaries that already moved
off the ActorTemplate onto the cluster-scoped SandboxConfig.
It now travels with those binaries end to end: resolved from the pool's
SandboxConfig, carried on ateletpb.SandboxAssets rather than
WorkloadSpec, and recorded in the per-actor sandbox record so Checkpoint
pins it into the snapshot manifest and Restore rebuilds the sandbox from
the image the snapshot was taken with (the golden's on a DATA_ON_GOLDEN
restore). A record without one is rejected outright rather than pulling
an empty image.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #791 — the last planned piece of the [implementation
plan](https://github.com/agent-substrate/substrate/issues/791#issuecomment-5226674097)
(after #810, #812, #813). #791 stays open until #817 (gVisor
Full-capture → Data-commit conversion, blocked on #790) is done; this PR
covers micro-VM fully and gVisor for scope-matched suspends. Also part
of the actor state machine (#119) and a prerequisite for system upgrade
flows (#473).
`SuspendActor` now accepts a PAUSED actor: instead of checkpointing a
running workload, ateapi dials the atelet on the node holding the pause
snapshot and has it upload the node-local files to object storage, then
finalizes as usual — durable `ActorSnapshot`, `SUSPENDED` status, node
pinning cleared.
Two commits, reviewable independently:
## Commit 1 — atelet: `UploadPausedCheckpoint` RPC (dead code until
commit 2)
- New `AteomHerder` RPC: a pure disk→object-storage copy driven by the
snapshot's self-describing manifest — no ateom involved (the sandbox is
gone).
- **Scope conversion** dispatches per sandbox class
(`narrowFullCaptureToData`): a micro-VM FULL capture narrows to a DATA
upload by carving out `durable-dir.tar` (constant hoisted to
`ateompath`, shared with ateom-microvm); gVisor returns `Unimplemented`
until split checkpoints land (#790); DATA can never widen to FULL; a
scope-less manifest (older atelet) is rejected rather than guessed at.
- **Idempotent retry**: local files gone + remote manifest present ⇒ a
previous invocation committed, succeed; gone on both sides ⇒
unrecoverable (`LOCAL_SNAPSHOT_GONE`, crashes the actor). Upload
failures stay plain retryable errors; the manifest uploads last as the
commit marker, never in parallel.
- The golden atespace is rejected at validation (fully on the
`field.ErrorList` framework): golden actors are never paused.
## Commit 2 — control plane: enable suspend from PAUSED
- `FromPaused` discriminator: PAUSED status, or SUSPENDING with no
worker assignment and a `LocalSnapshotInfo` — the field alone is stale
on resumed-from-pause RUNNING actors, so the nil-assignment conjunct is
load-bearing.
- `MarkSuspendingStep` accepts PAUSED and rejects a Data-captured pause
against a Full commit *before* the actor leaves PAUSED (an upload cannot
fabricate memory), using the `content_scope` recorded at pause (#812)
with an `onPause` fallback.
- `CallAteletSuspendStep` paused branch dials by node
(`DialForAteletOnNode`, #813): missing node record ⇒ crash (the snapshot
can never be found); unreachable atelet ⇒ retryable; atelet's
`LOCAL_SNAPSHOT_GONE` ⇒ crash via `maybeCrashActor`.
`FinalizeSuspendedStep` needed no changes thanks to the #813 hoist.
- Root-cause guard: `MarkPausingStep` rejects pausing golden-atespace
actors.
- Docs: pause states + the new `PAUSED → SUSPENDING` edge in the
architecture state diagram; glossary Suspend entry covers both origins.
## Tests
- **atelet unit**: 10 upload-helper subtests (conversion matrix,
idempotency probe, data-loss crash, retryable upload failure) via a
recording object-storage fake; validation table.
- **control-plane unit**: discriminator table, scope-rejection table
(incl. onPause fallback), paused preconditions (no node ⇒ CRASHED, no
atelet ⇒ `ErrNoAteletOnNode` + still SUSPENDING), golden-pause
rejection, prerequisite matrix updated.
- **functional (envtest)**: `TestSuspendActor_FromPaused` (upload called
with the pause snapshot name, no Checkpoint RPC, SUSPENDED, pinning
cleared, ActorSnapshot at the upload destination) +
retry-after-failed-upload (same destination on retry).
- **e2e (demo suite)**: the lifecycle driver gains a suspend-from-PAUSED
mode; three durable-dir cases — Full/Full, Data/Data, and the micro-VM
Full→Data extraction — assert memory/file counters survive the full
pause→suspend→resume journey and the node pinning is gone.
`go test -race ./...` clean (except the pre-existing macOS-only
`internal/atunnel` unix-socket-path failures, untouched by this PR),
`gofmt`/`go vet` clean, protos regenerated via `hack/protoc.sh`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
UpdateActor now takes an ActorRef, an ActorPrecondition, and a mutate
callback. The store reads the stored actor inside the WATCH transaction,
checks the precondition against that value, and hands it to mutate,
which edits it in place.
The storage layer retries up to 5 times when a concurrent write
invalidates it, re-running mutate against the newer state
I still need to migrate the other callers of store.UpdateActor to avoid
calling GetActor outside of the closure function.
#763
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#565
Replaces standard DNS resolution (`dns:///`) and the forced 1-minute
`MaxConnectionAge` reconnect workaround with a native, event-driven
Kubernetes gRPC resolver (`internal/k8sresolver`). The resolver uses
client-go's `SharedInformerFactory` and `EndpointSliceLister` to watch
`discoveryv1.EndpointSlice` resources in real time.
### Verification
Created a kind cluster to test changes by scaling the deployments up and
down and verified the new servers were connected to by clients.
#### 1. Kind Cluster Scale-Out & EndpointSlice Discovery
Click to view kubectl scale and EndpointSlice output
$ kubectl scale deployment/ate-api-server -n ate-system --replicas=4
deployment.apps/ate-api-server scaled
$ kubectl get pods -n ate-system -l app=ate-api-server
NAME READY STATUS RESTARTS AGE
ate-api-server-6bcd9fddbb-gxxth 1/1 Running 0 3m8s
ate-api-server-6bcd9fddbb-jgwbw 1/1 Running 0 3m8s
ate-api-server-6bcd9fddbb-m8nbx 1/1 Running 0 7m20s
ate-api-server-6bcd9fddbb-vwm2g 1/1 Running 0 7m20s
$ kubectl get endpointslices -n ate-system -l
kubernetes.io/service-name=api
NAME ADDRESSTYPE PORTS ENDPOINTS
api-dmzfc IPv4 443 10.244.0.15,10.244.0.19,10.244.0.31,10.244.0.32
#### 2. Proof of Active gRPC Load-Balancing Across All Replicas
```
[pod/ate-api-server-6bcd9fddbb-gxxth] {"time":"2026-07-29T13:20:49Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
[pod/ate-api-server-6bcd9fddbb-jgwbw] {"time":"2026-07-29T13:20:46Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
[pod/ate-api-server-6bcd9fddbb-m8nbx] {"time":"2026-07-29T13:20:48Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
[pod/ate-api-server-6bcd9fddbb-vwm2g] {"time":"2026-07-29T13:20:47Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
```
Fixes#554
Each workflow becomes a plain method orchestrating typed, idempotent
ensureX steps, e.g.:
```go
func (w *ActorWorkflow) ResumeActor(ctx, ref, boot) (*Actor, bool, error) {
// acquire actor lock, metrics defer — unchanged
actor, tmpl, snapSrc, err := w.loadActorForResume(ctx, ref, boot)
if actor.Status == RUNNING { return actor, false, nil } // explicit no-op
actor, err = w.ensureVolumesCreated(ctx, actor, tmpl)
actor, worker, err = w.ensureWorkerAssigned(ctx, actor, tmpl)
err = w.ensureVolumesAttached(ctx, actor, worker, tmpl)
tele, err = w.ensureAteletRestored(ctx, actor, worker, tmpl, snapSrc)
actor, err = w.finalizeRunning(ctx, ref)
return actor, true, nil
}
```
Each `ensureX`
- early-returns if its postcondition already holds, which is derived
from persisted state only, replacing `IsComplete`
- validates the state machine edge internally, replacing
`CheckPrerequisite`
- takes explicit typed inputs and returns what it produced.
- persists what it changed before returning, returning the stored copy
so the optimistic lock version threads forward. No shared
`*ResumeState`.
There is one behavioral change: worker ownership/eligibility validation
for a re-entered resume now runs before volumes are attached (it used to
run in `CallAteletRestoreStep`'s prerequisite, after attach), so a
resume that is about to crash on a reassigned/ineligible worker no
longer attaches volumes to it first.
- [X] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Part of #791 (PR 3 of the [implementation
plan](https://github.com/agent-substrate/substrate/issues/791#issuecomment-5226674097)).
Two behavior-preserving refactors needed before suspending a PAUSED
actor, which has no worker assignment:
**Dialer** — extract `dialAteletOnNode` from `DialForWorker`'s tail
(byNode index lookup → single-atelet check → per-pod-UID connection
cache → pinned-mTLS dial) and expose `DialForNode(nodeName)` for actors
whose state is pinned to a node without a worker pod (a PAUSED actor's
local snapshot). A node with no atelet in the informer cache returns the
new sentinel `ErrNoAteletOnNode` — documented as retryable (atelet
restarting / informer lag), distinct from the crash-worthy
`ErrWorkerPodNotFound`. `DialForWorker` behavior is unchanged apart from
error wrapping in the zero-atelets case.
**FinalizeSuspendedStep** — hoist finalization (ActorSnapshot record,
`LatestSnapshot`, the `SUSPENDED` flip, clearing
`InProgressSnapshot`/`WorkerAssignment`/`LocalSnapshotInfo`) out of the
`if assignment != nil` worker-freeing branch so it always runs.
Previously, a SUSPENDING actor with no assignment was silently skipped —
the workflow reported success while the actor stayed SUSPENDING forever.
Unreachable today (`CallAteletSuspendStep`'s prerequisite crashes
nil-assignment actors first), but load-bearing for the paused-origin
suspend, where no worker exists by design.
Tests:
- `TestDialForNode`: no atelet on node → `ErrNoAteletOnNode`; two
atelets → error; happy path asserts pod-UID-pinned credentials and
connection caching.
- `TestFinalizeSuspendedStep_NoAssignment`: SUSPENDING actor with
`LocalSnapshotInfo` and an in-progress URI but no assignment →
SUSPENDED, ActorSnapshot created at the URI, `LatestSnapshot` set,
in-progress fields and `LocalSnapshotInfo` cleared.
`go test -race ./cmd/ateapi/...` (incl. the functional suite covering
the running-path suspend end-to-end), `go vet`, `gofmt` clean.
Independent of #810 and #812.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Fixes#773
Some improvements to the way we create, track and propagate snapshot
ids.
* Consolidate snapshot id and name into a single concept.
* Use consistent terminology across the whole stack (i.e. remove
confusion about prefix vs id vs name).
* Get rid of internal only storage to hide the snapshot URI.
Part of #791 (PR 2 of the [implementation
plan](https://github.com/agent-substrate/substrate/issues/791#issuecomment-5226674097)).
Adds `LocalSnapshotInfo.content_scope` and records the template's
`onPause` scope in `FinalizePausedStep`. A later suspend of a PAUSED
actor needs to know what the local pause snapshot actually captured, to
validate the requested scope conversion *before* the actor leaves PAUSED
— even if the template's `onPause` is edited while the actor sits
paused. The snapshot's on-node manifest (#810) remains the authoritative
record at upload time; this control-plane copy is for early validation
only.
- `ateapipb` proto: `SnapshotContentScope content_scope = 3` on
`LocalSnapshotInfo` + regen
- `FinalizePausedStep`: sets `ContentScope` from
`SnapshotsConfig.OnPause` when recording `LocalSnapshotInfo`
- Unset for actors paused before this field existed (consumers fall back
to the template's `onPause`)
Tests:
- New `TestFinalizePausedStep_RecordsContentScope` (OnPause
Data/Full/unset→Full, through a real store with a worker)
- `TestFinalizePausedStep_WorkerGone` now seeds the template the step
reads
- `TestPauseActor`: expected `ContentScope` added — which exposed a
latent test bug: the custom `snapshot_prefix` comparer was asymmetric
(`strings.HasPrefix(y, x)` only); go-cmp requires Comparers to be
symmetric and probes them with swapped arguments, and the new field
shifted the comparison order so the probe finally landed on it. Fixed by
matching the prefix in both directions.
Verified end-to-end on a GKE deployment: a paused actor's Valkey record
now carries `"contentScope": "SNAPSHOT_CONTENT_SCOPE_FULL"` in
`localSnapshotInfo`.
`go test -race ./cmd/ateapi/...`, `go vet`, `gofmt` clean. Independent
of #810; no dependencies.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
`setup-envtest use` with no version re-resolves "latest" against the
remote release index on every run, so the envtest-backed packages needed
network even with fully cached binaries.
This PR pins the control plane to `1.36.x`, and fold the three
per-package `envtest` into `internal/testenv.Start`, which also honors
`-short`: `go test -short ./...` skips the envtest-backed packages and
is the guaranteed-offline path.
Have Claude to help me fix it
Fixes#789
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Closes#706
## Summary
- broker short-lived actor certificates from atelet over a same-node
mTLS Unix socket
- keep the actor private key in atunnel and renew the certificate before
expiry
- authenticate egress CONNECT using the actor certificate instead of
bearer tokens
## Testing
- `make verify`
- `go test -race ./internal/atunnel`
This PR implements the last two metrics from #433.
Right now, we can see that a resume was slow but not where.
`ate.actor.lifecycle.operation.duration` covers the whole ateapi
operation, and `atenet.router.route.duration` covers the edge, but
everything between ateapi-atelet-actors is one block that contains
fetching the manifest, downloading the snapshot, unpack the OCI image,
call to ateom.
We have `rpc.server.call.duration` that gives us the atelet restore
total time, but template, kind, and scope labels are missing, so today
we cannot really pinpoint why/where we have a regressions in latency.
In this PR I am adding per-phase histograms, here's an example of what
we can know after these changes:
```console
ateom_restore 522 ms ###############################
download 8.9 ms #
manifest_fetch 3.2 ms
oci_unpack 2.6 ms
---------------------------------------------------------
total 535 ms
```
It also fixes a gap #683 opened where a `data_on_golden` resume was
labeled identically to a plain one on the lifecycle histogram.
Things folks might want to argue with:
- total as a phase value. Partly duplicates `rpc.server.call.duration`,
but that one has no domain labels and gRPC-specifc. We can drop it, but
it's an inferior operational UX, so I'd rather have it here.
- Phases overlap, they are not a partition of total, because download
runs concurrently with the asset fetch and unpack. I called this out in
the metric description. Do not sum across phases.
- New `ate.snapshot.scope` key rather than a new `ate.snapshot.kind`
value for `data_on_golden`. A new value would collapse local and
external into one bucket, which is the biggest latency difference there
is. This does add a label to the already shipped lifecycle histogram.
Verified on kind, and all e2e suites pass, and the emitted series cover
every kind (golden, latest, local) on both metrics with no unknown
values.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #564 (Part 3)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements Part 3 of #564 by adding telemetry histogram
instrumentation for `ate.scheduler.eligible_workers` in `ateapi`. It
measures unassigned free worker capacity remaining after all scheduling
constraint filters are applied, sampled at every scheduling decision.
## Key Changes:
* Updated `Scheduler.Schedule()` to record eligible candidate workers
per pool (`recordEligibleWorkers`).
Defined `SchedulingConstraintKey` (`ate.scheduling.constraint`) and
constraint classification values (`none`, `required_nodes`, `selector`)
in `internal/ateattr/ateattr.go`.
* Added unit tests in `scheduling_test.go` covering candidate counts,
namespaced attributes, zero-capacity fleet states, empty fleets, sandbox
class mismatches, draining workers, and constraint classifications.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
Fixes#732
* `UpdateActorSnapshot` now carries the resource itself + `update_mask`
* `scope` is now applied via the update mask
* Moved `update_mask` to a separate file, so it can be reused by other
update RPCs.
* Added `uid` and `version` as optional guards.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#564 (Part 2)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements the second part of #564 by adding telemetry counter
instrumentation for `ate.actor.crashes` in `ateapi` and establishing
bounded label validation for operations and crash failure causes.
## Key Changes:
- Updated `maybeCrashActor` and `crashActor` to reference
`ateattr.OperationName*` constants and normalize `opName`.
- Passed explicit failure reasons (`ReasonCorruptedAssignment`,
`ReasonWorkerPodGone`, `ReasonWorkerReassigned`) across
`workflow_resume.go`, `workflow_suspend.go`, and `workflow_pause.go`.
- Instrumented `WorkerPoolSyncer.releaseActorOnDeadWorker` in
`syncer.go` to record `recordActorCrash` when background pod deletion
crashes an actor.
- Instrumented `FinalizePausedStep` in `workflow_pause.go` to record
`recordActorCrash` when `nodeName` is missing during pause finalization.
- Added `ate_actor_crashes` to `PlatformMetricPrefixes` in
`collector_metrics.go`.
- Added pre-creation resource cleanup in `metrics_test.go` to prevent
`AlreadyExists` errors on test reruns.
- Updated `metrics_test.go` to trigger an actor crash via `UpdateActor`
and `ResumeActor`, asserting `ate_actor_crashes` emission and label
presence in OTel collector scrape outputs.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
#732
* `UpdateActorRequest` now carries the resource itself + `update_mask`
* `worker_selector` is now applied via the update mask (making it
possible to clear the worker selector)
* `update_mask` is validated against an allowlist of paths. Currently,
only `worker_selector` is allowed.
* Added `uid` and `version` as optional guards.
I'll update UpdateActorSnapshotTag in a follow-up PR
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Refactors `WorkerPoolSyncer` to process events through a rate-limiting
workqueue instead of processing them synchronously inline.
Previously, store errors during Informer event handling, such as
- optimistic locking version conflicts (`store.ErrVersionConflict`)
- create races (`store.ErrAlreadyExists`)
- `WorkerPool` not yet present in the lister cache
were logged and dropped, leaving store state out of sync until a
subsequent pod event occurred.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This change formalizes the invariant that all these fields are either
all set or not. This simplifies some client code that was just checking
random worker metadata fields to see if the actor has an assignment or
not.
It also fixes a bug in `{Pause,Suspend}Actor` where we were not cleaning
up the worker pod UID field.
This PR is the second slice of the platform-metrics split (#433).
It adds two duration histograms emitted by ateapi:
- `ate.actor.lifecycle.operation.duration`:
create/resume/suspend/pause/delete, labeled by operation, template,
pool, sandbox class, and (on resume) snapshot kind
- I'd like to use this for user-facing latency for e.g. showing when
suspended actors can serve requests again. The existing
`rpc.server.call.duration` metric covers this, but the meaningful
dimensions are missing, so it's not really actionable. Extending its
labels with extra, domain-specific labels is an OTel anti-pattern, hence
the new metric.
- `ate.scheduler.assignment.duration`: worker-assignment step, labeled
by outcome (assigned / no_free_worker / error) and pool
- I'd like to use this to alert on no free worker situation, and have
proper SLOs via various percentiles for assigning latencies. There's no
RPC around this, so it's not something existing RPC metrics cover.
Tested e2e on a local kind cluste where: both metrics reach the
otel-system collector with the expected labels (resume shows
`snapshot_kind=golden`, scheduler shows `outcome=assigned`, no
`error.type` on success).
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Today everything outside ateapi hardcodes `ParentBased(NeverSample)`,
and setting `OTEL_TRACES_SAMPLER` does nothing because an explicit
sampler silences the SDK's env handling. This makes troubleshooting
production issues impossible via traces.
This PR gives every component a sane default and makes the standard OTel
env vars work:
- Control plane (ateapi, atelet, ateom) defaults to
`parentbased_traceidratio` 0.1, the router (data plane root) to 0.01,
per the discussion on #584.
- `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` override any of this
without a rebuild. Invalid values keep the component default and log a
warning instead of inheriting the SDK's fall-open-to-100% behavior.
- Envoy's `RandomSampling` is derived from the router's resolved policy,
so the two root decisions cannot drift.
- kubectl-ate without `--trace` no longer installs a tracer provider at
all: the old `NeverSample` provider injected `sampled=0`, which pinned
every parent based sampler downstream and would have defeated the server
side ratios. `--trace` still forces a full end to end trace.
- ate-controller propagates the two env vars to the ateom worker pods it
creates, same as the metric export vars.
- kind pins ateapi to `parentbased_always_on`, so the local Jaeger flow
keeps showing every API call.
- The agentgateway integration follows what we have above. its
`randomSampling` changes from `true` to `0.01` to match the data plane
default, though as static config it does not follow
`OTEL_TRACES_SAMPLER` overrides, so the two need adjusting together.
Verified on a kind cluster e2e manually. It resolved samplers logged at
startup, a `--trace` resume produced one trace across ateapi, atelet,
and ateom, an unsampled CLI call got picked up server side, Envoy
continued a sampled traceparent while sampling 0 of 30 parentless
requests at the 1% default, and an invalid env value fell back to the
component default.
Gating who may use `--trace` stays a separate follow-up (and a
discussion), and the more sophisticaed tracing policies belongs in a
collector, not in substrate.
Fixes#584
cc. @git286
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
> **Rescoped again.** This PR previously proposed
`--golden-snapshot-warmup`, a
> tunable wall-clock delay before the golden checkpoint. Per discussion,
that
> direction is dropped: the answer for a workload that cannot report
readiness
> is a readiness endpoint — a small sidecar where the workload itself
cannot be
> changed — not a longer timer. What survives is the piece that
discussion
> agreed on, and which the previous revision already flagged as a
follow-up:
> making the readyz deadline itself configurable.
>
> The warmup work is not in this branch. It is kept locally in case a
workload
> genuinely cannot be given a readiness signal before GA, and would come
back as
> its own PR if so.
## Summary
`readyz.Wait` polls until the container returns 200 or a **hardcoded
30s**
elapses. A workload that legitimately takes longer to bind its HTTP
server
cannot be accommodated without raising the ceiling for every actor in
the
cluster, and losing that race fails the actor start.
How long a workload takes to become ready is a property of that
workload, so
this makes the deadline a per-template setting rather than a package
constant.
Adds optional `timeoutSeconds` to `ContainerReadyz`. **Unset keeps
today's 30s**,
so no existing template changes behavior.
## Changes
The value rides on the existing probe, so it follows the chain the probe
already
takes and no call site needs to know about it:
`ContainerReadyz.timeoutSeconds` → `toAteletReadyz` → `ateletpb.Readyz`
→
`toAteomReadyz` → `ateompb.Readyz` → `readyz.Wait`
- `pkg/api/v1alpha1/actortemplate_types.go` — `TimeoutSeconds *int32`,
`+optional`, `Minimum=1`, `Maximum=3600`.
- `internal/proto/ateletpb/atelet.proto`,
`internal/proto/ateompb/ateom.proto` —
`int32 timeout_seconds = 2` on both `Readyz` messages.
- `cmd/ateapi/internal/controlapi/workload_spec.go`,
`cmd/atelet/main.go` — pass
it through the two conversions.
- `internal/readyz/readyz.go` — `OverallTimeout` becomes
`DefaultOverallTimeout` (still 30s) and `Wait` resolves its deadline
through a
new `overallTimeout(probe)` helper.
- Regenerated: both `.pb.go`, `zz_generated.deepcopy.go`, and the
`actortemplates` CRD.
None of the four `readyz.WaitAll` call sites change.
**On the zero value.** Unlike a warmup delay — where zero is a real
request
meaning "checkpoint immediately" — a zero readiness deadline could never
be met,
so it is never something a template author means. A non-positive value
on the
wire is therefore read as "unset" and falls back to the default, and the
CRD
field is a pointer with `Minimum=1` so the API rejects `0` outright
rather than
silently substituting 30s behind the author's back.
**On bounding**, which was the open question left on the previous
revision:
bounded at `3600`. A template asking to wait longer than an hour for
readiness
is expressing a broken workload, not a slow one, and the bound keeps a
typo from
pinning a worker for a day.
## Verification
- `go build ./...`, `go vet ./...`, `gofmt`, `go test ./...` — all pass.
- `internal/readyz/readyz_test.go` — `overallTimeout` resolves unset and
negative to the default and honors an explicit value; `Wait` against a
port
nothing binds gives up at the probe's 1s deadline rather than the 30s
default.
- `workload_spec_test.go`, `cmd/atelet/main_test.go` — the timeout
crosses both
conversions, and a probe without one stays zero on the wire.
- `actortemplate_validation_test.go` — the bounds are enforced by a real
API
server. This suite runs under envtest against the generated CRD
directory, so
it exercises the regenerated `actortemplates` CRD rather than the Go
markers:
`300` is accepted, unset is accepted, and `0`, `-1` and `3601` are all
rejected by apiserver schema validation.
- **On a real cluster, via CI.** `internal/e2e/fixtures/probe` now
declares a
`readyz` probe with `timeoutSeconds: 60`, pointed at the `/healthz` the
probe
binary already serves on `:80`. The kind e2e that runs on every PR
therefore
exercises the value crossing ateapi → atelet → ateom on real binaries,
across
the auth matrix, on both the run and restore paths. This is also the
readyz
path's first e2e coverage — no fixture declared a probe before.
Wire compatibility degrades safely in both skew directions:
`timeout_seconds` is
a new field 2 on a `Readyz` message that has only ever had field 1, so
an old
ateom ignores it and an old ateapi leaves it zero, which reads as the
30s
default.
No GKE run. What that would add over the above is a workload whose
readiness
genuinely exceeds 30s, and that is the readiness-sidecar work rather
than this
PR.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
---------
Co-authored-by: Maya Wang <mymaya@google.com>
Phase 1 of #451, micro-VM only. gVisor support will follow in a separate
PR.
## What
Adds a `snapshotsConfig.onResume` block to the ActorTemplate, selecting
per snapshot situation what supplies the guest state at resume: each
field names what is being resumed *from*, the value names the boot
source. `onPause`/`onCommit` remain pure capture scopes (`Full | Data`),
and the `ActorSnapshot` records plain `Full`/`Data` content; the
golden-combine is strictly a restore-time behavior.
```yaml
snapshotsConfig:
onPause: Data
onCommit: Data
onResume:
fromData: Golden # ColdBoot (default) | Golden — Data snapshots resume as
location: gs://… # golden memory + fs delta with the actor's data layered on
```
Under `fromData: Golden`, a data-only restore combines the template's
**golden snapshot** (guest memory + full fs delta) with the actor's
captured durable data — the actor comes back with the golden's warm
state over its own data instead of cold-booting. `fromData` applies to
every resume of a Data snapshot — commit or pause checkpoint. Future
situations (e.g. resuming a Full snapshot across a template upgrade,
#477) get sibling fields under `onResume` rather than overloading this
one.
## How
- **CRD** (`pkg/api/v1alpha1`): new `OnResumeConfig`/`ResumeSource`
types; `fromData: Golden` is CEL-gated to `sandboxClass: microvm`.
- **Control plane** (`ateapi`): resume derives the behavior from the
template's `onResume` configuration — for a Data durable snapshot or a
Data pause checkpoint it resolves the golden snapshot's location and
sends the restore-only `DATA_ON_GOLDEN` wire scope, failing early (with
an actionable error) if the golden snapshot is missing or not Full.
Golden actors always commit `Full` regardless of `onCommit` — their
snapshot is the base the combine needs.
- **Wire APIs** (`ateletpb`/`ateompb`): restore-only
`SNAPSHOT_SCOPE_DATA_ON_GOLDEN`, rejected on checkpoints;
`golden_snapshot_uri_prefix` is a top-level `RestoreRequest` field
because the actor's snapshot may be a local pause checkpoint while the
golden snapshot is always external.
- **atelet**: stages a single combined folder — the actor's files (the
durable-dir tar) win name collisions, the golden snapshot supplies the
rest. External restores download both halves concurrently; local (pause)
restores copy the actor's files from the local checkpoint dir while the
golden's files download in parallel. The golden manifest's pinned
sandbox binaries run the restored guest and are recorded on-node for
later checkpoints.
- **ateom-microvm**: `DATA_ON_GOLDEN` restores through the existing Full
path — cloud-hypervisor relaunches from the golden's guest files while
the durable virtio-fs share serves the actor's re-materialized data.
gVisor rejects the scope (defense-in-depth behind the CRD gate).
## Testing
- Unit: envtest CEL cases, converter tables, atelet request validation
(checkpoint rejection, golden-URI rules incl. local+golden),
combined-download test against a fake object store, resume-workflow
golden-resolution cases for both the durable and pause paths.
- e2e (`suites/demo`): OnGolden lifecycle cases for the commit path, the
pause path, and a two-durable-volume variant (micro-VM lane only),
asserting counters across pause/suspend and the recorded Data content
scope.
- Manually verified on a live GKE cluster: Data commit and Data pause
both resume with the golden guest's memory over the actor's data, on
different pods.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Also Fixes#575
(1) mintCert + MintJWT proto changes
(2) validate that atelet is the one that calls actorIdentity Service.
(3) verify that this atelet is only requesting a cert for an actor on
its node.
(4) verify that the actor is still running
(5) create another substatex509 extension for actor identities
JWTs needs more love, lots as TODO there
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Adds a dynamic `--log-level` flag (debug, info, warn, error;
case-insensitive) to
the server binaries, backed by a `slog.LevelVar` in `serverboot`.
Defaults to
`info` — no behavior change unless the flag is set.
- **serverboot**: the LevelVar behind `InitLogger`'s handler;
`SetLogLevel(string)`
to parse and set it (empty string = unset, a documented no-op);
`LogLevel()`
returns a `slog.Leveler` for binaries that build their own handler.
- **atelet, ateapi**: new pflag, fatal on an invalid value.
- **ateom-gvisor, ateom-microvm**: flag accepted by the binary; note
their
container args are built by the WorkerPool controller with no
pass-through yet,
so wiring it is a follow-up — and that follow-up must roll out only
after this
change's ateom images are deployed (older images reject unknown flags
and would
crashloop).
- **atenet (router + dns)**: replaces two pre-existing hand-rolled
`--log-level`
parsers that silently fell back to `info` on a bad value; both now use
`serverboot.SetLogLevel` and `serverboot.InitLogger`, which also adds
`ate.dev/trace-id` correlation to atenet logs.
- **Kind manifests**: atelet runs at `--log-level=debug` on kind
(dev/CI), so e2e
suites can assert on per-item log lines; production installs keep
`info`.
Motivation: groundwork for #463 Phase 2 observability — per-item GC log
lines can
ship at Debug and be enabled per-node instead of landing at Info.
**Testing**: unit tests pin the untouched default at exactly `info`,
dynamic
raise/lower, case-insensitivity, invalid-value rejection, and the
empty-string
no-op; `go test -race` green across serverboot, atenet, atelet, ateapi.
Fixes#564 (Part 1)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements the first part of #564 by extending metric
`atenet.router.route.duration` with singleflight activation state
(`ate.router.resume`) and outcome status labels (`ate.router.outcome`).
All telemetry attribute keys and label value sets are centralized in
`internal/ateattr` to ensure consistency across components.
## Key Changes:
* Added `resumed` boolean to `ResumeActorResponse` in `ateapi.proto`
indicating whether a cold activation workflow occurred
(`!state.WasRunning`).
* Updated `workflow_resume_test.go` to assert `resumed == true` on cold
activations and `resumed == false` for already-running actors.
* Extended `classifyOutcome(err)` in Envoy `ext_proc` to classify errors
into metric labels: `ok`, `cancelled`, `timeout`, `no_capacity`,
`lock_conflict`, `not_found`, `unavailable`, `rate_limited`, and
`resume_error`.
* Added explicit handling for `codes.Unavailable` (`"unavailable"`) and
`codes.ResourceExhausted` / `StatusCode_TooManyRequests`
(`"rate_limited"`).
* Consolidated metric label names on `atenet.router.route.duration`
using standard keys from `internal/ateattr`:
* `ate.template.namespace` (`TemplateNamespaceKey`)
* `ate.template.name` (`TemplateNameKey`)
* `ate.router.outcome` (`RouterOutcomeKey`)
* `ate.router.resume` (`RouterResumeKey`)
* Updated `manifests/ate-install/kind/kustomization.yaml` to export
`atenet-router` OTLP metrics to `opentelemetry-collector`.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
Part of #232
1. During CreateActor, the volumes to be created get instantiated in the
Actor API in PENDING status. The Actor is still instantiated as
SUSPENDED.
However, the CreateVolume operation is moved to the beginning of
ResumeActor. This is because CreateActor is intentionally designed to
not be idempotent. The downside is that the user won't know dependent
resource creation failed until the first resume.
2. Changes volume id to use actor uid instead of atespace+actorname and
adds "substrate" prefix
3. Adds a new DELETING status that gets persisted before we start to
delete volumes.
This shifts the existing DB DELETE precondition checks to the new
DELETING status. And now you can only delete the actor in DB from
DELETING status.
4. Refactor DeleteActor to use a workflow
In the future, we will also add a workflow that will do mandatory node
cleanup of volumes before starting to delete the volumes.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
`ate.workerpool.workers` tallies by pool name, state, and sandbox class,
but a WorkerPool is namespaced, so two pools that share a name sum into
a single series. This is reachable today: the counter demo and the
autoscaled-workerpool demo both create a pool named `counter`, and with
both installed the demo HPA scales on the other pool's assigned workers.
Adds `ate.workerpool.namespace` to the tally key and attributes, and
pins it in the demo's adapter query and HPA selector.
`syncWorkerToStore` checks `pod.Status.PodIP != ""` before
`DeletionTimestamp`, so a Terminating pod reporting no IP never reaches
`markWorkerDraining` and stays `STATE_ACTIVE`, schedulable for as long
as the pod lingers. Draining resolves the stored record by name and
never reads the IP, so this swaps the two guards.
The startup reconcile runs through the same function, so a pod already
Terminating when ate-api-server restarts is now marked too.
`markWorkerDraining` already runs on every update event for a
Terminating pod, so this adds no new call pattern.
Fixes#600
Sessions are no longer a concept in Substrate; Actor is the glossary
term. This completes the "s/Session/Actor" TODO that sat at the top of
ateapi.proto, and removes the TODO.
API surface:
service SessionIdentity -> ActorIdentity
MintJWTRequest.session_id -> actor_id
MintJWTResponse.session_jwt -> actor_jwt
MintCertRequest.session_id -> actor_id
MintCertResponse.session_certificates -> actor_certificates
Go packages:
cmd/ateapi/internal/sessionidentity -> actoridentity
cmd/ateapi/internal/sessionidjwt -> actoridjwt
Flags and cluster resources:
--session-id-jwt-pool -> --actor-id-jwt-pool
--session-id-ca-pool -> --actor-id-ca-pool
Secrets, volumes and mount paths renamed to match, in both
manifests/ate-install/ate-api-server.yaml and hack/install-ate.sh
(--create-session-id-ca-pool-secret -> --create-actor-id-ca-pool-secret).
Two credential identity values change with the rename:
JWT issuer https://broker.agentic-substrate-session-id-broker.svc
-> https://broker.agentic-substrate-actor-id-broker.svc
SPIFFE ID spiffe://substrate-session.local/app/../session/..
-> spiffe://substrate-actor.local/app/../actor/..
Tokens and certificates issued before this change will not validate
against the new issuer or trust domain.
BREAKING: the gRPC wire path moves from /ateapi.SessionIdentity/* to
/ateapi.ActorIdentity/*, and the Secrets must be recreated under their
new names before the new ate-api-server rolls out.
Part of [#23](https://github.com/agent-substrate/substrate/issues/23)
The overall plan is:
1) Trap SIGTERM in the ateom, and forward SIGTERM to all application
containers.
2) Once all application containers exit, ateom will exit itself.
3) The workerpool syncer will mark workers as 'DRAINING' when a pod
deletion is triggered (pod.DeletionTimestamp) to prevent actors from
being scheduled. On the pod Deleted event, the worker and actor (if it
was assigned) will be cleaned up.
This PR implements 3), there are 2 more PRs coming which will implement
1) and 2) for each ateom (gvisor and uVM).
This PR adds missing OpenTelemetry span attributes in UpdateActor and
corresponding tests.
- [X] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Julian Gutierrez Oschmann