`RevertActor` returns a RUNNING, PAUSED, or CRASHED actor to SUSPENDED
at its last external snapshot, so CRASHED is no longer a dead end that
only `DeleteActor` can clear. Docs and code comments still described it
as terminal and told operators to delete and recreate the actor, losing
its state.
Update the api-guide, architecture, upgrade guide, and kubectl-ate
README to cover the new verb, and correct the comments that justified
keeping a partial external snapshot by naming actor deletion as the only
remaining collector -- revert collects it too.
Follow up for the PR - #1675
Issue - #1556
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
We need to draw a strong distinction between credentials that will be
wielded directly by actor logic, and credentials used by system
components on behalf of an actor. This will help us avoid attacks where
an actor pretends to be a system component.
Ateom certificates are requested by ateom/atelet and used for the
atunnel connection to the egress gateway. They have a SPIFFE URI like
`spiffe://${trustdomain}/ateom-for-actor/${atespace}/${actor}`.
Actor certificates are requested by the egress gateway and used for
opening outbound requests on behalf of the actor. They have a SPIFFE URI
like `spiffe://${trustdomain}/actor/${atespace}/${actor}`. Note, this
use case is not actually implemented yet.
Fixes https://github.com/agent-substrate/substrate/issues/292
This flips the condition to crash actors: any errors returned by atelet
will cause actors to crash by default.
If we later discover specific error classes that are safe to retry, we
can explicitly mark them as retriable.
TagStatus.source_actor_uid is unused, we can add this back if we ever
need it.
#1168
We were also missing some of the DV for TagStatus fields:
* Added maxLength=2048 requirement for ExternalSnapshot.snapshot_uri
(matching in_progress_snapshot_uri)
* Added validation for actor_template_uid
* Added validation for storage_location, matching what we have in
SnapshotsConfig.storage_location
The name `readyz` is not very descriptive. We also want to avoid
`readinessProbe` to prevent confusion with the Kubernetes concept, which
represents continuous traffic gating. Renaming this field to
`wakeupProbe` clarifies its actual behavior, and the fact that it only
operates when the actor is woken up.
Part of #1378
# Description
**Substrate assumes the canonical install layout, and every deviation
fails closed.** The namespace `ate-system`, the Services `api` /
`atenet-router`, and the ServiceAccounts `atelet` / `atenet-router` are
compiled-in constants. An install in a per-developer namespace, or under
a deployment that prefixes resource names, breaks — and no failure
points at the naming.
This series makes the namespace, the Service names, and the
ServiceAccount names configurable. Every option defaults to the
canonical value. A canonical install is byte-for-byte unaffected.
| Hardcoded assumption | Failure | What it looks like instead |
|---|---|---|
| atelet's namespace in ateapi's SPIFFE check | ateapi rejects every
atelet | an mTLS handshake failure, not a naming error |
| atelet's namespace in the worker's broker check | no actor obtains a
certificate | `credential broker is not atelet` |
| NetworkPolicy's ingress namespace | the CNI drops every request to the
pool | a silent network fault |
| ateapi Service name and namespace in the client | the client cannot
authenticate | `services "api" not found`, then `invalid bearer token` |
| Resource names in `hack/install-ate.sh` and the e2e harness |
authorities land in the wrong namespace | suites die in preflight |
**A SPIFFE ID breaks on two axes.** It names a namespace and a
ServiceAccount. Both were constants, and the failure surfaces as a
rejected peer, not a missing object.
## Design notes
- **ate-controller passes the worker-side identities.** ateom already
took `--atunnel-client-identity` as a flag but relied on its default. A
new `--atunnel-broker-identity` flag alone would be inert: correct only
where the constant was already correct. The controller knows the control
plane's namespace, so it supplies both.
- **ateapi takes the whole expected identity, not a namespace.** The
dialer and `ateletauth` used the namespace only to build one string. One
place decides how atelet's identity is spelled.
- **`ateletauth` deduplicates without taking the dependency it was
avoiding.** Its constants are duplicated rather than imported so the
package does not depend on `controlapi` for three strings. `ateletdial`
would make a third copy, so the strings move to
`internal/installdefaults` instead — a leaf package of constants that
imports only the standard library, so consuming it does not reintroduce
the coupling the duplication was there to prevent.
- **The client reads environment variables, not flags.** It runs outside
the cluster. It has no downward API and nothing to discover from.
- **Namespaces come from the downward API, not flags.** Every supported
topology co-locates these components. Only Service and ServiceAccount
names, which a deployment may legitimately rename, get flags.
- **`hack/install-ate.sh` and the e2e harness are in scope.** A
relocated install cannot be bootstrapped or exercised without them. The
script refuses `ATE_NAMESPACE` or `ATE_API_SERVICE_NAME` overrides on
the manifest-applying subcommands, which would half-install:
`manifests/ate-install/` names `ate-system` and `api` literally.
## Why this is one PR
The series is stacked, not parallel. Commit 1 creates
`internal/installdefaults` and seeds seven imports; commit 2 adds
fourteen more; every later commit builds on those constants. Split into
separate PRs, each blocks on the previous merging and none reviews
independently.
The guard test cannot land first. Applied to `main` it fails on eleven
hardcoded literals across `ateletauth`, `controlapi/informer.go`,
`networkpolicy_controller.go`, both `ateom` mains, `ateclient`,
`ateletdial` and three e2e files. It is green only because the preceding
commits removed them.
A partial series still fails closed. Each commit's audit turned up more
places assuming `ate-system`, so landing commits 1-3 without 4-5 leaves
a relocated install broken later and less legibly than before.
`CONTRIBUTING.md` covers this case: when the intermediate steps are not
useful on their own, keep the change as one PR split into commits at
logical break points. Review commit by commit, and preserve the commits
on merge. If you would still rather split, the one defensible cut is by
axis — relocation (commits 1-3) then renaming (commits 4-5) with the
guard test rebased on top.
## Rebase notes (2026-09-15)
The series is rebased onto `main` at `d0d85c39`, ninety-two commits on
from the base the PR first carried:
- `main` moved actor JWT and certificate minting out of
`cmd/ateapi/internal/actoridentity` into `controlapi`, deleting the
package and dropping its inline atelet authorization check in favor of
the unified authorizer. This series previously threaded the expected
atelet identity into that check; the check no longer exists, so that
adaptation is gone. `ateletauth` survives with one caller,
`workerservice.SetWorkerCapacity`, which still takes the configured
identity.
- The same refactor replaced `BrokerConfig.ExpectedActorUID` with an
`ActorAtespace` / `ActorName` / `ActorUID` triple. `AteletSPIFFEID` is
additive and still required by `ateletdial.TLSConfig`.
- `ateletauth` (ateapi's side) and `ateletdial` (the worker's side) each
re-declared the hardcoded SPIFFE ID. Both take the expected identity as
a parameter.
- `main` deleted the atenet DNS subsystem. The series no longer touches
it, and `installdefaults` carries no DNS Service name.
- The controller passes `--atunnel-broker-identity` only when it differs
from the canonical default. An ateom old enough to predate the flag
exits on it, and `docs/upgrade.md` keeps such a pool serving during a
rolling upgrade. A relocated install gets the flag and necessarily runs
an image that accepts it.
- `main` added `internal/e2e/collector_metrics.go` (agentgateway CI
support), which addresses the router through the package-level
`routerNamespace` and `routerService` that this series replaces. Its two
call sites become `SystemNamespace()` and
`ResourceName("atenet-router")`, matching `router_client.go` and
`statusz.go`. A reviewer diffing against the older base sees those call
sites move; nothing else in that file changes.
- `main` added `cmd/credential-provider/kubernetes-secrets` (`#1335`),
whose `injectorSPIFFEID` constant is the mTLS peer check on the only
caller permitted to read Secrets. The comparison is an exact string
match, so a renamed install rejects every fetch at TLS. It becomes
`--injector-identity`, defaulting to
`installdefaults.EgressSPIFFEID(SystemNamespace)`, alongside a new
`EgressServiceAccount` constant and helper. The flag name follows
`--atunnel-client-identity`; the manifest beside it still names
`ate-system`, so a renaming deployment configures both.
- `main` replaced the dialer's worker indexer with
`DialForAteletOnNode`, dropped `WorkerPodInformer` from `controlapi`,
and removed `kataConfig` from ateom-microvm's `NewService`. The series
adapts to each narrower signature and keeps only its own added
parameter.
- The guard test's allowlist entry for the two inert `ate-system`
literals follows the code from `actoridentity.go` to
`controlapi/actor.go`. The refactor re-introduced exactly the class of
constant this series removes, so the guard earns its place.
# Testing
- `go test -race ./...` passes. `make verify` passes boilerplate,
codegen, go-modules, gofmt, golangci-lint, kube-api-linter, licenses,
metrics and postgresql-migrations. `proto-fmt` needs `clang-format`,
which is absent locally; this series touches no `.proto` files.
- Every commit builds and vets individually, via `git rebase -x 'go
build ./... && go vet ./...'`.
- On a fresh kind cluster, all twelve e2e suites pass: `capabilities`,
`combinedvolumes`, `demo`, `egressauthz`, `egressmitm`, `example`,
`identity`, `metrics`, `networking`, `networkpolicy`, `parking`,
`sizing`. The `demo`, `metrics`, `networkpolicy`, `parking` and
`networking` suites need the `--deploy-demo-counter` and
`--deploy-demo-egress` fixtures, and the MITM path needs
`--deploy-demo-egress-mitm`; without them they fail on `actor template
not found`, which reads as a control-plane fault rather than a missing
fixture.
- With the sdsmint egress gateway (`--deploy-atenet
--experimental-use-sdsmint`, `E2E_EGRESS_MITM=1`),
`TestActorEgressMITMTrust` and `TestActorEgressHTTPSByHostnameMITM` both
pass, and the `networking` suite is green at 25 passed. The two modes
are mutually exclusive by design:
`TestActorEgressHTTPSByHostnamePassthrough` covers the plain gateway and
skips under MITM, and the two MITM tests skip without it. Both modes
were run, so every case executed in one of them.
- The new unit tests use a relocated namespace and renamed
ServiceAccounts. Reintroducing each hardcoding makes them fail.
One e2e caveat that predates this series and misleads: the `identity`
suite calls `ReplaceEgressTrustPool`, which takes over the shared
`egress-mitm-ca-pool` Secret, overwrites it and registers no cleanup.
Any MITM test running afterwards fails with `certificate signed by
unknown authority`, which reads as a broken interception path rather
than a mutated fixture. Reinstall the gateway before running
`egressmitm`.
# Additional Notes
**Credential contents stay canonical.** `controlapi/actor.go` still
mints credentials naming `api.ate-system.svc`: the JWT issuer and the
certificate's Issuer CN. Relying parties validate these, so changing
them is a compatibility decision, not a lookup fix. The code carries a
TODO to make the issuer a globally unique, OIDC-compliant name.
Closes#1802.
## What this does
Deletes the `ate.scheduler.eligible_workers` histogram. Nothing replaces
it in this PR.
## Why
**It is expensive on the resume path.** `Schedule` built a second slice
of the matching workers that scheduling never read, walked it again, and
repeated `HasRoom` for each entry. `HasRoom` parses resource quantity
strings up to three times for each worker. The metric roughly doubled
the per-worker arithmetic of a placement. The filter loop is now one
pass over the fleet and builds one slice.
**A metric cannot answer the question it was built for.** #564 wanted to
tell a full fleet apart from an empty intersection of the constraints,
such as a cordoned node behind a node requirement. That needs the
selectors, the node requirement and the actor.
`docs/metrics/substrate.yaml` bars all three from a metric label and
sends them to logs and spans, so no shape of this instrument reaches the
answer. A log record at the rejection is the replacement the issue
names, and it is not in this PR.
## Also removed
`ate.scheduling.constraint` goes with the instrument. No other signal
used the attribute.
## Scope of the change
- `cmd/ateapi/internal/scheduling/metrics.go` deleted, along with the
`WithMeter` option and the second filter pass in `Schedule`.
- The instrument and the attribute removed from
`docs/metrics/registry/metrics.yaml`. The `pool-keys-paired` exception
in `docs/metrics/substrate.yaml` dropped, because the empty pool pair
was this instrument's alone.
- `docs/observability.md`: the table row and the two label notes
removed.
- The e2e collector check and the label assertions for the instrument
removed.
- `TestSchedule_EligibleWorkersMetric` removed. The behaviors it covered
(draining workers, a sandbox class mismatch, busy workers) are already
in the `TestSchedule` table.
## Testing
`go test ./cmd/... ./internal/...` passes. `make verify` passes except
`hack/verify/metrics.sh`, which needs Weaver or Docker. Neither is
available on this machine, so CI is what checks the registry.
Add an example gRPC credential-provider plugin which reads from k8s
secrets for the egress
credential-injection path.
- It resolves `ate-secret://k8s.io/default/<namespace>/<secret>/<key>`
URIs to Kubernetes Secret values, so Substrate never stores secrets — it
only
brokers a read that the provider is authorized to perform.
- It is the only
component in the injection path with Kubernetes secerts access; the
egress gateway and
its injector never read Secrets directly.
### What's included
- **`cmd/credential-provider/kubernetes-secrets`** — the gRPC service:
- Parses and validates `ate-secret://` URIs, resolving the requested
Secret
(with single-key fallback when the URI omits a key).
- Enforces an **atespace→namespace authorization policy**
(default-deny),
derived from the caller's attested actor SPIFFE ID.
- Serves over **mutual TLS**, requiring the caller's client cert to
chain to
the trust bundle *and* carry the egress injector's SPIFFE SAN.
- **Manifests** (`manifests/egress-credential-injection/`) — Deployment,
Service, ServiceAccount + RBAC, a sample namespace-policy ConfigMap, and
a
sample Secret.
- Renames the default provider address/service from `credprovider` to
`k8s-credential-provider` across `ate-setup`, install scripts, and the
egress
injection overlay.
### atespace → namespace authorization
Beyond the mTLS check that only the egress injector may call the
provider, each
request is authorized against a **default-deny atespace→namespace
policy**. The
provider derives the requesting actor's atespace from its attested
SPIFFE ID and
resolves a Secret only if that atespace is explicitly granted access to
the URI's
namespace.
An actor lifecycle record went to stdout and to OTLP in two separate
calls. The attributes were shared, but the severity was not, so the slog
level sat at the call site and the OTel severity sat on the `Event`.
Nothing made a caller write both copies either, so a new record could
reach stdout only and no test would notice.
Since this PR `actorevent.Log` now writes both copies. The level comes
from `Event.Severity`, so it is stored once. The OTLP only `Emit` and
the exported body constants are gone, so the dual write is the only way
out of the package.
Also normalizes `ate.actor.operation.name` on the crash event, which the
state change event already did.
Fixes#1744
Testing
- Unit tests for the level mapping, the dual write, and the registry
check.
- End to end on a fresh kind cluster. Both event names arrive with the
right severity (9 and 17), the right attributes, and trace context on
the record fields. The stdout copies match record for record.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
* (Prefactor): Extract defaulting logic into
`cmd/ateapi/internal/defaults`.
* When reading resources from DB, perform a default backfill.
* Change defaults for readyz probes.
Part of #1593
The tag deletion logic is getting too complex, so let's move it to a
workflow, following what we do for other resources.
#1510#664
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
#### Summary
This PR introduces the `RevertActor` RPC for actor lifecycle management.
It includes the API definition, the corresponding workflow execution
logic, observability metrics, and updates to the authorization model to
support reverting actors.
Fixes#1556
Docs updated in #1711
#### Commit-wise Changes
**1. Add RevertActor RPC (`1145d99`)**
* Introduces the new `RevertActor` RPC to the API definitions.
* Updates the corresponding protobuf bindings (affecting files like
`ateapi.pb.go` and `ateapi_pb2.py`).
**2. Add the revertActor workflow and observability metrics
(`dcb1fae`)**
* Implements the core `revertActor` workflow logic, designed to be
idempotent and re-enterable. It progresses through the following steps:
* **Mark Reverting:** Validates that the actor is in a revertable state
(`RUNNING`, `PAUSED`, or `CRASHED`) and transitions its state to
`REVERTING`.
* **Discard Worker:** Safely tears down the execution environment by
terminating the workload, detaching volumes, and releasing the assigned
worker.
* **Collect In-Progress Snapshot:** Cleans up external object storage by
deleting any objects a previous suspend operation was partway through
writing.
* **Finalize:** Commits the actor to `SUSPENDED` and strips all
node-local and in-progress state pointers (clearing `WorkerAssignment`,
`LocalSnapshotInfo`, etc.), returning the actor to its untouched
external snapshot.
* Instruments the workflow with lifecycle operation metrics (e.g.,
updating `ate.actor.lifecycle.operation.duration` to track `revert`
operations).
* *Note/TODO:* Currently, when reverting a paused actor, the workflow
drops the pointer to the node-local state but does *not* actually prune
the local checkpoint bytes from the node (this is tracked in #641).
**3. Add `can_revert` to the authorization model (`2fc53ae`)**
* Adds the `can_revert` permission to the auth model, mirroring the
shape of `can_suspend` (editor tier of the parent atespace, plus a
direct grant so a machine identity can revert the actor it drives
without holding an atespace role).
**4. Serve RevertActor and add the CLI verb (`f04fab4`)**
* Wires the `Control.RevertActor` service method to the workflow
(replacing the generated stub that previously answered `Unimplemented`).
* Adds the `"ate revert actor"` CLI command, making the feature usable
end-to-end.
* Implements `Terminate` for the fake atelet. This was necessary because
reverting an actor from the `RUNNING` state is the first path to reach
this call in functional tests (previously, delete tests skipped this
step as they ran against actors with no worker assignment).
**5. Add a manual verify script for RevertActor (`8022db3`)**
* Adds a script to manually exercise `RevertActor` against a real
control plane, since unit and functional tests only run against a fake
atelet.
* Tests reverting from `CRASHED`, `RUNNING`, and `PAUSED` states, and
verifies that attempting to revert a `SUSPENDED` actor is properly
rejected.
* Simulates a crash by deleting the worker pod the actor runs on to
verify the workflow can handle the absence of a worker to terminate.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
## What this does
ateapi writes a record every time an actor changes state. Until now
those records only went to the pod's stdout, and nothing reads stdout.
This sends the same records to a collector as OTLP log events.
Actor name and uid cannot be metric labels (too many values), and traces
are sampled at 1%. So these records are the only way to answer "what
state is this actor in, and since when".
## Changes
- `serverboot.InitLogging` sets up a LoggerProvider, next to the
existing tracer and meter ones.
- New `internal/actorevent` package builds the log records.
- ateapi emits at the two places that already write the stdout records.
- Two event names: `ate.actor.state_changed` and `ate.actor.crashed`.
- Both names are registered in `docs/metrics/registry/events.yaml`, so
`make verify` checks them.
- kind gets a logs pipeline and a count connector. The e2e suite reads
the counts back.
- Docs updated. `otel-collector.md` said substrate has no
LoggerProvider, which is no longer true.
## Opt-in
`OTEL_LOGS_EXPORTER` defaults to `none`. Only the kind overlay sets it
to `otlp`. The base ConfigMap is untouched, so no deployed environment
changes when this merges.
## Notes on the design
- **No slog bridge.** Only two call sites emit these records, so
emitting twice costs two lines. A bridge would also send every ateapi
log over the wire, could not set the event name, and would loop, because
SDK export errors are logged through slog.
- **Batching processor, not the simple one.** These records sit on the
actor resume path. A processor that exports inside the emit call would
add a blocking gRPC call there, so a slow collector would become control
plane latency.
- **Two event names, not one per state.** `ate.actor.state` already says
which transition happened. A crash gets its own name because it carries
two extra attributes and a higher severity.
- **Both copies are kept on purpose.** No collector in this repo reads
pod stdout, so nothing is duplicated today. `kubectl logs` keeps
working. If a filelog agent is ever added, drop one of the two. The
escape hatch is written down in `docs/metrics/substrate.yaml`.
## Dependencies
Adds `otel/log`, `otel/sdk/log` and `otlploggrpc`, all pinned at
v0.20.0. That is the release that matches the pinned `otel v1.44.0`.
v0.21.0 would pull the core modules to v1.45.0, which this change does
not need. The logs API has a
v1.47.0 release candidate upstream, so it is on its way to stable.
## Testing
- Unit tests for the exporter resolver, the record builder, and both
ateapi emit sites.
- The record builder test checks the attribute set matches what the
event name declares, in both directions.
- The ateapi tests check the OTLP record carries the same attributes as
the stdout record.
- Ran end to end on kind. Records arrive with the right event name,
severity, attributes, and with trace context on the record's own fields
rather than as attributes.
- Checked the off state too. With `OTEL_LOGS_EXPORTER` removed, the
collector receives no log records and stdout is unchanged.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Dialing by resolving the worker pod is problematic, because in some
cases (e.g. during Worker Pod deletion), we need to dial to atelet (to
properly call `Terminate`), but the worker is gone.
A much simpler model is to always dial by resolving the IP of the atelet
on a given node.
Adds **egress credential injection**: on the sdsmint egress gateway's
decrypted MITM leg, when an actor's `EgressPolicy` rule matches and
carries an `inject_static_headers` effect, the gateway resolves the
referenced credential from a **credential provider** and sets it as a
request header (e.g. `Authorization: Bearer <token>`) before
re-originating upstream.
This implements the `applyEffects` in the egress handler. Injector runs
as part of the existing egress ext_proc handler that already
fetches/caches/evaluates each actor's `EgressPolicy`, so the MITM leg
gains injection without a second ext_proc hop.
### How it works
- New `CredentialProvider.FetchSecret` plugin API
(`pkg/proto/credproviderpb`): the gateway calls it with the credential
URI and the actor's attested SPIFFE identity; the provider authenticates
the gateway (mTLS) and resolves the secret.
- The egress handler dials the provider over mTLS
(`egress.DialProvider`) and injects on matched, allowed requests
(`egress.applyEffects`).
- Behavior:
- **TLS MITM leg + provider configured** → inject the credential
(overwriting any actor-set header).
- **Cleartext leg, or no provider configured** → skip injection and pass
the request through (never put a secret on a cleartext wire; don't block
allowed egress).
- **Attempted but failed** (unfetchable secret, empty/malformed secret,
credential URI of an unserved provider class) → fail closed.
## Installation
One flag stands up and wires the whole stack:
```
hack/install-ate.sh --deploy-ate-system --experimental-egress-credential-injection
```
`--experimental-egress-credential-injection` deploys credprovider and
the injector, then re-wires the sdsmint egress gateway to route through
the injector. It implies `--experimental-use-sdsmint`
### New install flags
Two flags configure which credential provider the injector targets:
| Flag | Purpose | Default |
|---|---|---|
| `--credential-provider-name` | Provider class, as a `ate-secret://`
prefix; a policy URI of any other class is refused |
`ate-secret://kubernetes.io` |
| `--credential-provider-address` | Where the injector dials the
provider | `credprovider.ate-system.svc:50051` |
To build a snapshot URI, we need the external storage location (which is
stored in the actor template), actor atespace and UID (which are stored
in the actor resource). If an actorTemplate was deleted, we'd leak the
in-progress snapshot, because the storage information was gone.
Fixes#1608
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
#1522 added a resyncInterval parameter to NewActorTemplateReconciler and
#1523 added this call site. They landed without seeing each other, so
`main` doesn't compile.
SEt the resyncInterval to 7s, to match what we have in other tests
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1507
Golden snapshots currently remain owned by the temporary golden actor,
so another resume/suspend cycle or actor deletion can collect a snapshot
still referenced by its template. The controller now copies the warmed
snapshot into a published tag, deletes the golden actor, and records the
tag reference on the template. Interrupted tag creation and cleanup
remain retryable; template deletion cleans up both resources.
`CreateActor` resolves an explicit `sourceTag` or the template's golden
tag into the actor's initial snapshot. Actors created before the golden
tag is ready retain their cold-boot behavior. The golden tag uses the
template UID as its name in `ate-golden`. The proto replaces
`golden_snapshot` with `golden_tag` at field 1, without backward
compatibility.
This PR is based directly on `main` and does not depend on #1521.
Follow-up recommendation: move the create → resume → wait → suspend →
tag → delete sequence into a golden-template workflow using the existing
workflow conventions. The reconciler now coordinates multiple
recoverable steps; it could retain scheduling and retries while
delegating that sequence to the workflow. This refactor is outside this
PR.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Validation: full `env -u NO_COLOR make verify` passed after rebasing
onto `main`. After the final proto field-number change, bindings were
regenerated and the control API unit/functional tests plus proto-format
and Go-format checks passed.
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Actor template discovery currently uses a fixed 20-second resync
interval. This PR adds `--template-resync-interval` so deployments can
tune that delay, preserving the `20s` default and rejecting nonpositive
values. The setting also controls the reconciler's existing fallback
retry delay.
This is intentionally a small change to start a discussion about how
template builds should be dispatched as the template catalog grows.
The current resync fetches and decodes every template, including
completed ones. At 100,000 templates, a 20-second interval implies
roughly 5,000 template rows read per second per replica, assuming scans
finish quickly. This is an estimate from the code, not a benchmark.
Possible follow-ups:
- **Immediate enqueue:** start work after creation, retaining a slower
recovery scan for crashes between persistence and enqueue.
- **Outbox/watch:** consume changes instead of scanning the catalog. The
existing worker outbox still polls every 50 ms and broadcasts events to
each subscriber; write overhead and recovery scans need consideration.
- **Durable pending-build queue:** atomically record work, claim due
jobs using short `FOR UPDATE SKIP LOCKED` transactions, and recover
expired leases. This still polls, but queries pending work rather than
the full catalog. `LISTEN/NOTIFY` is not the proposed default because it
serializes notifying commits.
- **Imperative build / long-running operation:** give callers explicit
build control or a progress/completion handle. Either still needs
reliable execution underneath.
The main question is whether template builds need a broadcast change
feed or a queue where replicas claim different jobs. Neither alternative
is implemented here.
[Full research: database costs, execution options, recovery
requirements, and
sources](https://gist.github.com/EItanya/0a1d6893ede34f2f0e9d9d1929ecb823).
Validation: all Go race tests and repository verifiers passed using
module mode with `NO_COLOR` unset. CLI checks confirmed the default and
rejection of zero/negative intervals; the existing reconciliation test
checks a custom interval.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Populate default fields during both Create and Update RPC calls so
that resources are stored with defaults instead of zero values.
This is a much cleaner approach as it means that a) consumers don't need to
be aware of the default value and b) default values can potentially change.
Only ActorTemplate had defaults, but added default functions for all
resources so that all RPC handlers are wired up.
The egress gateway is about to enforce EgressPolicy, which means two
components now interpret the same hostname patterns and CIDRs: ateapi
when it validates a policy, and atenet when it matches one. Putting the
parsers in internal/egresspolicy and having ateapi's custom validators
call them keeps "accepted by the API" and "enforceable by the gateway"
the same set by construction. The package also compiles a policy once
and evaluates it the way the API describes, in rule order over whatever
the caller knows of the destination, first match wins: a hostname rule
never matches a destination with no name, and an IP rule never matches
one whose address the caller does not know. The decision says which
kind of rule matched, because the gateway sends the bytes to what that
rule checked. HasHostnameRules answers the question that follows, which
is whether a decision point holding only an address should refuse
outright or leave the destination to one that will see a name.
Hostname patterns now reject any name whose last label is all digits,
per RFC 1123 section 2.1, instead of consulting an IP parser. That is
the same set as before for every real address and additionally refuses
dotted-quad spellings such as "01.2.3.4" that no parser accepts.
The actor SPIFFE ID moves to internal/resources with a parser next to
the builder: ateapi mints it into the certificate's URI SAN and the
egress gateway reads it back as the actor identity on the legs inside
the tunnel, so both ends need the one definition.
# Record actor state changes from ateapi
ateapi now writes a log record every time an actor changes state.
## Before
Nothing tells you what state an actor is in. Metrics can't carry actor
identity, and the control plane samples traces at 10%, so most
transitions leave nothing behind at all. If you want to know whether an
agent is running, paused or suspended, there's nowhere to look.
## After
One filter:
```text
ate.actor.uid="8f2a1c4e6b0d47f1" AND ate.actor.state!=""
```
Last record wins. That's the state it's in now, and the timestamp is
when it got there.
The record:
```json
{
"time": "…",
"level": "INFO",
"msg": "Actor state changed",
"ate.atespace": "ate-demo-counter",
"ate.actor.name": "counter-1",
"ate.actor.uid": "8f2a…",
"ate.template.atespace": "ate-demo-counter",
"ate.template.name": "counter",
"ate.actor.operation.name": "suspend",
"ate.actor.state": "suspended"
}
```
## The two new keys
`ate.actor.state` is the `ateapipb.ActorState` values lowercased, so the
log wording and the state machine can't drift apart.
`ate.actor.operation.name` says what caused the change. The state on its
own doesn't tell you: an actor lands in `suspended` from a suspend and
`paused` from a pause, and only one of those gives the worker back.
`Actor crashed` picks up the same state key. A crash is a state change
too, and it's the one an actor can reach without any operation
finishing.
## Notes for review
**Why ateapi.** It owns the state machine. It also sees transitions that
never reach a worker, like deleting an actor that was already suspended,
or a resume that fails in the scheduler.
**It can't log a state the store never had.** The record goes out after
the store commit, and every state commit has a version precondition, so
a losing writer in a concurrent update writes nothing.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Changing the sandbox_class requires creating a new worker, you can't
change the sandbox_class of a running worker.
- [ x ] Tests pass
- [ x ] Appropriate changes to documentation are included in the PR
Fixes a few todos in the `ateapi.proto`
* Adds custom validation for `create_time` and `update_time` fields.
* Adds better validation method for the `Container.image` field.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Fixes#1432
Map known store pagination errors in `ListActorTemplates` through
`mapListError`, matching the other list RPC handlers. Invalid page
tokens now return gRPC `InvalidArgument` with `invalid page_token`.
- [x] Tests pass: ran `make verify`
- [x] Appropriate changes to documentation are included in the PR:
single line change
Fixes https://github.com/agent-substrate/substrate/issues/1566
Today the Resume workflow resolves its restore source in the following
order
1. First check if the actor has node-local snapshot,
2. then its own durable external snapshot,
3. then the template's golden snapshot.
The boot flag was consulted at exactly one point in that chain, where it
suppressed using the golden-snapshot, which made its behavior much
narrower than "boot from scratch" suggests:
- Actor has its own external snapshot and boot=true: flag ignored,
restores the actor's snapshot.
- Actor has a local snapshot and boot=true: flag ignored, restores the
local snapshot.
- Actor has no snapshot, template has no golden snapshot: cold boot from
the spec regardless of the boot flag.
- Actor has no snapshot, template has a golden snapshot: boot=false
restores the golden, boot=true cold boots from the spec. This is the
only case where the flag is used.
The glutton benchmark was the only caller that set boot=true, on each
actor's first resume, to report true cold-start latency as a separate
ResumeActorColdStart stats row. @maxsmythe let me know if this is
required.
The proto field number and name were reserved.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
I'm cleaning up the ResumeActor workflow in
https://github.com/agent-substrate/substrate/pull/1492/changes . This PR
is to add tests first to make sure the refactor does not introduce any
behaviour changes.
Testing table:
| \# | local (OnPause) | external (scope) | tmplUID | golden (scope) |
fromData | boot | want |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| 01 | — | — | — | — | unset | f | Run RPC (cold boot), Restore not
called |
| 02 | — | — | — | — | GOLDEN | f | still Run: fromData only governs
data-only snapshots; nothing to restore ⇒ golden branch skipped |
| 03 | — | — | — | FULL | unset | f | golden fallback: Restore EXTERNAL,
uri=goldenURI, Scope=FULL, golden="" |
| 04 | — | — | — | FULL | unset | t | boot=true skips golden: Run RPC
(fromData irrelevant here too) |
| 05 | — | — | — | FULL | GOLDEN | f | same as 03 (no actor snapshot ⇒
not dataOnly ⇒ GoldenSnapshotUri stays empty) |
| 06 | — | — | — | DATA | unset | f | FailedPrecondition — golden
fallback rejects a non-Full golden |
| 07 | — | — | — | FULL, malformed URI | unset | f | DataLoss — golden
fallback, malformed golden URI |
| 08 | — | — | mismatch | FULL | unset | f | golden fallback unaffected:
Restore EXTERNAL, uri=goldenURI, Scope=FULL (TemplateReplaced needs a
durable snapshot) |
| 09 | — | FULL | — | — | unset | f | Restore EXTERNAL, uri=actor
snapshot, Scope=FULL |
| 10 | — | FULL | — | FULL | GOLDEN | f | Full snapshot ignores policy:
Scope=FULL, golden="" |
| 11 | — | FULL | match | — | unset | f | matching UID ⇒ no downgrade:
Scope=FULL |
| 12 | — | FULL | mismatch | — | unset | f | repointed ⇒ Restore
EXTERNAL, Scope=DATA, golden="" |
| 13 | — | FULL | mismatch | FULL | GOLDEN | f | TemplateReplaced beats
the golden policy: Scope=DATA, golden="" |
| 14 | — | DATA | — | — | unset | f | Restore EXTERNAL, Scope=DATA,
golden="" |
| 15 | — | DATA | — | FULL | GOLDEN | f | Restore EXTERNAL, uri=actor
snapshot, Scope=DATA\_ON\_GOLDEN, golden=goldenURI |
| 16 | — | DATA | — | DATA | GOLDEN | f | FailedPrecondition — Golden
data resume, golden not Full ("regenerate the golden snapshot") |
| 17 | — | DATA | mismatch | FULL | GOLDEN | f | TemplateReplaced beats
DATA\_ON\_GOLDEN: Scope=DATA, golden="" |
| 18 | — | UNSPEC | — | — | unset | f | Restore EXTERNAL, Scope=FULL
(conversion maps UNSPECIFIED → Full) |
| 19 | — | malformed URI (FULL) | — | — | unset | f | DataLoss — actor's
own external snapshot URI fails to parse |
| 20 | FULL | — | — | — | unset | f | Restore LOCAL,
LocalConfig.SnapshotName, Scope=FULL, golden="" |
| 21 | FULL | — | — | FULL | GOLDEN | f | Full pause ignores policy:
Restore LOCAL, Scope=FULL, golden="" |
| 22 | FULL | — | mismatch | — | unset | f | Restore LOCAL, Scope=FULL —
repoint detection requires a durable snapshot, local-only actors never
downgrade |
| 23 | DATA | — | — | — | unset | f | Restore LOCAL, Scope=DATA,
golden="" |
| 24 | DATA | — | — | — | GOLDEN | f | FailedPrecondition — Golden data
resume requires a golden snapshot, template has none |
| 25 | DATA | — | — | FULL | GOLDEN | f | Restore LOCAL,
Scope=DATA\_ON\_GOLDEN, golden=goldenURI |
| 26 | DATA | — | — | DATA | GOLDEN | f | FailedPrecondition — golden
not Full, local-snapshot path |
| 27 | DATA | — | — | UNSPEC | GOLDEN | f | UNSPECIFIED golden scope
accepted as Full: Restore LOCAL, Scope=DATA\_ON\_GOLDEN |
| 28 | DATA | — | — | FULL, malformed URI | GOLDEN | f | DataLoss —
golden URI fails `ParseSnapshotURI` |
| 29 | DATA | FULL | — | FULL | GOLDEN | f | local wins over external:
Restore LOCAL, Scope=DATA\_ON\_GOLDEN (dataOnly from OnPause, not the
external scope) |
| 30 | DATA | FULL | mismatch | FULL | GOLDEN | f | TemplateReplaced
wins in the local branch too: Restore LOCAL, Scope=DATA, golden="" |
This was originally a separate service to make it easy to apply separate
authentication and authorization interceptors.
It now seems clear that our authn/z framework will be strong enough to
support atelet and external callers in one system (based on OpenFGA).
This change moves the MintJWT and MintCert RPCs into the control API,
and removes some inline authz checks that will be handled by our unified
authorizer framework.
The container image must include the image digest. This validation was
dropped during the migration of the ActorTemplate resource from CRD to
Substrate API.
Fixes#1475
`Control.CreateWorker` does not validate `Worker.sandbox_class`, so a
client can register a worker with an empty one (and any record written
before this keeps it). `RegisterWorkerCount` tallied that raw value, so
`ate.workerpool.workers` emitted `ate_sandbox_class=""` — not a member
of the `ate.sandbox.class` registry vocabulary — alongside the pool's
seeded series at 0.
This change normalizes the worker's class with the existing
`ateattr.NormalizeSandboxClass`, so an empty class reports as `unknown`.
New `ateattr.SandboxClassAttribute`, used by every emitter. This closes
the same gap on `ate.actor.crashes` and the `ate.actor.lifecycle.*`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1539
If an actor is `CRASHED`, it can't be resumed/paused/suspended anymore,
it can only be deleted.
An actor deletion will clean up everything under the prefix of
`Actor.external_snapshot.snapshot_uri` and
`Actor.in_progress_snapshot_name` (the latter will be empty if the
underlying worker was deleted).
Before, we risked leaking a snapshot after a worker deletion because we
were clearing the `in_progress_snapshot_name` field. We don't need to
clear it up because the CRASHED state is terminal.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1508
Store tag snapshots at `<base>/atespaces/<atespace>/tags/<tag-uid>`.
Replace `in_progress_snapshot_uri` with immutable `storage_location`, so
pending and completed tags share UID-based cleanup independent of the
source actor or template.
- [x] Tests pass: race-enabled control API tests and PostgreSQL tag
contract tests.
- [x] Documentation updated.
Lint and code-generation verification also passed.
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
## Summary
Fixes the flaky test `TestActorLifecycle_WithExternalVolumes` in
`cmd/ateapi/internal/controlapi/functionaltest` (`unit` lane).
**Root cause:** the test pauses and immediately resumes against a single
1-capacity worker; PauseActor commits the assignment release to the
store, but AssignWorker reads the worker cache, which sees the release
only after the PostgreSQL watch delivers it — losing that race fails the
resume with `ResourceExhausted: no free workers available`.
**Fix:** wait for the release to reach the cache with the existing
`waitForWorkerAvailable` helper (the same pattern sibling lifecycle
tests use after suspend) before resuming.
Closes#1533
## Evidence
Flake rate over last 7 days: 5.7% (26 fail / 427 pass across 453 runs)
Failures span 14 branches incl. 9 on main; infra-failure runs excluded:
0
Failing runs:
https://github.com/agent-substrate/substrate/actions/runs/33919495958,
https://github.com/agent-substrate/substrate/actions/runs/33941175484
Passing runs:
https://github.com/agent-substrate/substrate/actions/runs/33567474203,
https://github.com/agent-substrate/substrate/actions/runs/33566785143
## Test plan
- [x] `go test -race -count=5
./cmd/ateapi/internal/controlapi/functionaltest/ -run
TestActorLifecycle_WithExternalVolumes` — passes (was intermittently
failing before at ~6%)
Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>
`main` is red since #1480 merged: its regenerated
`cmd/ateapi/internal/controlapi/zz_generated.validation.go` was produced
by a stale generator build and references `deepEqualImpl_`, which
doesn't exist — `cmd/ateapi` (and everything importing `controlapi`)
fails to compile, and every open PR inherits the failure through CI's
merge-with-main (see #1512's `run-tests` for an example, and `main`'s
own post-merge `pr-workflow` runs for #1480/#1499).
Fix is a pure regeneration: `hack/update/codegen.sh` on current `main`
changes exactly one line, `deepEqualImpl_` → `ateDeepEqual` (the helper
the current generator actually emits). After it, `go build
./cmd/ateapi/...` succeeds and `go test
./cmd/ateapi/internal/controlapi/` passes.
No hand-written changes — the diff is generator output only.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The changes we need are live on HEAD of the k8s published repos. It was
a bit of a journey, since Go fought me all the way.
---
### Drop our third_party fork of k8s deps in tools
The changes we need are released now (sort of - on HEAD anyway).
---
### Bump k8s.io/streaming to v0.37.0 (not rc) in tools
---
### Bump codegen deps to HEAD in tools
GOPROXY=direct go get \
k8s.io/apimachinery@master \
k8s.io/code-generator@master
This pins them to HEAD of master. Go is terrible here: The HEAD is not
actually tagged, so Go just uses the next "reachable" tag which is
v0.36.0-alpha. The datestamp is correct, though.
---
### Drop our third_party fork of k8s deps in root
The changes we need are released now (sort of - on HEAD anyway).
---
### Bump k8s deps to v0.37.0 (not rc) in root
---
### Bump apimachinery dep in root to HEAD in root
GOPROXY=direct go mod edit -replace
k8s.io/apimachinery=k8s.io/apimachinery@master
GOPROXY=direct go mod tidy
GOPROXY=direct go mod vendor
GOPROXY=direct go mod tidy
This approach (-replace) is needed because Go is horrible here.
The master branch of k8s.io/apimachinery is not tagged, per se, but
there is an OLDER tag which is "reachable" from HEAD. So go helpfully
decides to use that (v0.36.0-alpha.2). If we just `go get ... @master`
it works for that dep (pinned to the right date) but then it looks at
transitive deps. Because the tag seems to be 0.36 (older), it
recalculates all the OTHER dependencies and downgrades a whole tangle of
things to versions that match 0.36, but we are ACTUALLY on 0.37+.
This was the only approach that I (and Gemini) could find. Blech.
---
### Run updated codegens
---
### Use DV's new maxBytes capability for `[]byte`
Removes 1 custom.
## What
Now that the ActorTemplate CRD has been deleted and its resources moved
to the substrate gRPC API and the control-plane store (created/managed
with `kubectl-ate`, persisted in PostgreSQL), several documents still
describe ActorTemplate as a Kubernetes CRD, or describe namespace/RBAC
relationships that no longer exist. This sweeps the docs for those stale
references.
Fixes#368 (docs side).
> This change was prepared with AI assistance; I have reviewed and
tested it.
- [x] Docs and comment-only change; no functional code changed, no tests
affected
`CreateActorTemplate` accepted `../escape` and `/escaped` in
`ActorMetadataItem.path` and `TrustBundleDataSource.path`, and accepted
several data sources projecting to the same path. atelet only rejects
the traversal at actor start, so the template is created but the golden
actor sits in `RESUMING`. atelet never rejects duplicate paths at all:
it writes the files in order and the last one wins.
This PR moves both checks to template validation.
- The path rule lives in a new `internal/volumepath` package that both
ateapi and atelet call, so the two cannot drift. It splits on `/` and
rejects any segment that is empty, `.`, or `..`, caps depth at 16
segments, and rejects NUL bytes. Any other byte is allowed, so Unicode
paths pass. `ActorMetadataItem.path` and `TrustBundleDataSource.path`
apply it through a `+k8s:customValidation` hook, and atelet's
`writeSystemInfoFile` applies it again before touching the host
filesystem.
- `SystemInfoVolumeSource.data_sources` gets a hook that tracks
projected paths in a set and reports a duplicate on repeats, in the same
shape as the Kubernetes projected-volume validation.
The proto doc comment on `data_sources` also says at most one
`actor_metadata` entry may appear. Nothing enforces that, and this PR
leaves it alone.
New cases in `TestValidateSystemInfoVolumeSource` and
`TestValidateTrustBundleDataSource` cover the rejected shapes, and
`internal/volumepath` has its own table test. `hack/verify/codegen.sh`
and `hack/verify/golangci-lint.sh` pass.
#1231 moves `writeSystemInfoFile` into `systeminfovolume.go` and onto
`os.Root`. Whichever PR lands second points that copy at the shared
rule.
Fixes#1436Fixes#1438
Renames the proto message, its status message and scope enum, the five
RPCs and their request and response messages, the actor_snapshot_tag
request fields, and Actor.source_snapshot_tag to source_tag. The store
interface, its Postgres table, the object-storage prefix segment and the
kubectl-ate verbs follow, so nothing keeps the old spelling.
https://github.com/agent-substrate/substrate/issues/664
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#664
This PR implements the idea described in
https://github.com/agent-substrate/substrate/issues/664#issuecomment-5499311489
It does more than Garbage Collection of snapshots, because we also got
rid of the Snapshot resource (from the DB/API).
Now, an external snapshot is owned by a single resource:
- An Actor owns the snapshot it writes at suspend
- A tag owns a copy taken at tag creation,
- An actor cloned from a tag borrows the tag's snapshot until its own
first suspend.
Garbage Collection: whoever created/owns the snapshot is the only one
who ever deletes them:
i.e., if an actor is deleted and it owns a snapshot. The underlying
snapshot is deleted with the actor.
this PR:
- Drops table actor_snapshots
- Keeps table actor_snapshot_tags
- Adds an object copy at tag creation, and an owned versus borrowed
distinction on the Actor
- Adds synchronous external snapshot deletion at actor suspend, at actor
delete, and at tag delete
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Julian Gutierrez Oschmann