Working on https://github.com/agent-substrate/substrate/issues/1563
Fixes https://github.com/agent-substrate/substrate/issues/1697,
https://github.com/agent-substrate/substrate/issues/1701
## Summary
Adds `AccessPolicy` singleton subresources for the Global
(`global:root`) and `Atespace` scopes, stored in PostgreSQL and enforced
through OpenFGA.
### Changes
- **API (`Control` service)**
- Global: `CreateGlobalAccessPolicy`, `GetGlobalAccessPolicy`,
`UpdateGlobalAccessPolicy` (no Delete; an update with empty bindings
clears all grants).
- Atespace: `CreateAtespaceAccessPolicy`, `GetAtespaceAccessPolicy`,
`UpdateAtespaceAccessPolicy`, `DeleteAtespaceAccessPolicy`.
- `Binding.members` has set semantics.
- **Always-wired authz**: the OpenFGA server, `Authorizer` and
`PolicyManager` are always created. `--experimental-enable-authz` only
controls whether the interceptor enforces checks on regular RPCs. Policy
tuples therefore stay in sync with the store in every mode, and deleting
an atespace always removes its tuples.
- **Governance RPCs are always enforced**: AccessPolicy RPCs are marked
`alwaysEnforce` in the permission registry and are checked even when
enforcement is off, so nobody can grant themselves access before it is
turned on. A test fails if an AccessPolicy RPC is added without that
marker.
- **Bootstrap owners**: `--authz-bootstrap-owners` lists
server-configured owners (required when enforcing). They are never
written to the store: `Authorizer.Check` adds a contextual `owner
global:root` tuple for the calling user only, so access follows the
server config and is dropped on restart without them. A bootstrap owner
creates the global policy.
- **Storage**: migration `000003` adds `global_access_policy` and
`atespace_access_policies`. Deleting an atespace cascades to its policy
row and removes its OpenFGA tuples in the same transaction.
- **Consistency**: `PolicyManager` takes an explicit `pgx.Tx`, locks the
metadata row (`FOR UPDATE`) and reconciles OpenFGA tuples in the same
transaction. Optimistic concurrency via `uid` / `version`.
- **Hardening**: members with control characters are rejected, as are
blank, `*`, or control-character bootstrap owner entries; non-canonical
OpenFGA error codes are mapped to `Internal`.
### Testing
- Interceptor tests for both modes: with enforcement off, regular RPCs
skip the check while governance RPCs are still checked.
- Functional CRUD tests for both AccessPolicy scopes
(`functionaltest/access_policy_test.go`).
- PostgreSQL test that a policy update and its tuple writes commit or
roll back together, including under concurrent updates.
- Governance tests: creating the policy as a bootstrap owner, a
bootstrap owner also stored as owner, and a restart without a bootstrap
owner.
### Notes
- ateapi now needs OpenFGA available at startup even with enforcement
off, and atespace deletes always write OpenFGA tuples.
- TODO: move the authz settings into the hot-reloadable config proto
(#2021) once it lands.
Today, we don't know if RPCs to atelet fail because of a transport error
or a real atelet error + we always crash the actor on atelet errors.
So, we end up crashing actors when atelet is upgrading or temporarily
unavailable.
This PR solve this by:
* Not crashing actors when the atelet RPCs return: `codes.Unavailable`,
`codes.Canceled` or `codes.DeadlineExceeded:`
* Never crashing actors after errors from atelet.Terminate. Terminate()
is called during DELETE/REVERT and we plan to also call it during CRASH
operations.
Fixes#2027
Actor lifecycle v2 will not restore an actor by combining the golden
snapshot's guest state with the actor's own data snapshot.
`onResume.fromData: GOLDEN` was the only way into that path, so this
removes the setting and everything behind it.
#### Behavior changes
- **API:** an ActorTemplate that sets `snapshotConfig.onResume` is
rejected (unknown field). Templates already stored lose the field
silently on read.
- **Resume of a DATA snapshot** always starts the containers fresh from
the image, with the durable dirs filled from the snapshot. Templates
that used `fromData: GOLDEN` now pay app startup on resume and lose any
in-memory state that came from the golden image.
- **Unchanged:** FULL snapshots, a just-created actor (borrows the
golden tag and restores it FULL, or cold-boots via `Run` if there is
none), committing the golden actor (always FULL), and creating actors
from a tag.
- **Restore path:** a DATA_ON_GOLDEN restore used to need two manifest
GETs and downloads from two snapshots; every restore now reads one
manifest and one snapshot.
- **Metrics:** `ate.snapshot.scope` takes only `full`, `data` or
`unknown`.
- **Rollout:** atelet rejects scope 3, so upgrade ateapi and atelet
together. Per the pre-v1.0 policy there is no compatibility shim.
**This is a Breaking change**
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Part of #1756.
- Adds a required `MintActorJWTRequest.expiration_seconds`, validated to
[300, 3600]. Token exchange (#1661) wants longer-lived subject tokens.
- Adds `MintActorJWTResponse.expires_at`, equal to the `exp` claim, so
callers like the egress gateway's cache don't have to decode the token.
- Changes the subject to `actor/<atespace>/<name>`, matching the path of
the actor's SPIFFE ID. The token isn't a JWT-SVID.
- Rewrites the `MintActorJWTResponse` comment to match the settled
contract.
Field 8 skips 2, 3, 4, and 6, which the old ActorIdentity request used.
Testing: validation unit tests for the bounds. The functional test
checks the lifetime, `expires_at`, and `sub`; it needs Docker, so it
hasn't run locally.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR (docs
land later in the #1756 series)
FinalizeSuspended deleted the external snapshot a suspend replaced
before committing SUSPENDED. By then the new checkpoint is uploaded and
the worker's assignment row is released, so a transient object-store
error (429/503) on that delete aborted the workflow with the actor stuck
in SUSPENDING: ResumeActor rejects that state with FailedPrecondition,
and a retried suspend re-sends Checkpoint to the already-released
worker, which most likely crashes the actor despite a good snapshot in
storage.
Commit SUSPENDED first, then release the replaced snapshot best-effort,
logging a failure instead of returning it. A failed release now leaves
the old snapshot's objects in storage until the actor is deleted, which
removes its whole prefix, rather than stranding the actor. A commit lost
to a concurrent update releases nothing, since the record still names
the old snapshot.
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
**BREAKING** API changes prior in preparation for GA.
~This PR has proto-only change to the EgressPolicy API. No
implementation; the gateway,
store contract, and e2e helpers stop compiling until the follow-up
lands.~ Note: Impl is only to make ci green. Full implementation of the
api is a fast follow
- `EgressRule` is now a union of protocols: `http`, `https`,
`tls_passthrough`. Allow-only, deny by default.
- The `hostnames`, `cidrs`, and `all` rule kinds are removed. **No field
numbers
or names are reserved: we are pre-GA and existing policies must be
recreated.**
- `rules` is an **unordered set**. When more than one rule matches,
precedence is
decided by two criteria, in order: a pattern without a wildcard wins
over one
with a wildcard, then a port other than `"*"` wins over `"*"`. This
applies
within a protocol and across `https` and `tls_passthrough` on the same
SNI.
Two rules with the same pattern on the same port are rejected on write.
Only the winning rule's effects apply.
- `http`: cleartext HTTP, matched per request on the authority and port.
Carries `effects`.
- `https`: MITMed HTTPS that the gateway intercepts. Matched on SNI and
port at the
ClientHello, then per request on the authority. Carries `effects`. MITM
is off unless a name is
listed here.
- `tls_passthrough`: TLS forwarded without decryption, matched once per
connection on SNI and port. No effects. Implicit TLS only; STARTTLS does
not match.
- Protocols are told apart by what the Actor sends first, a ClientHello
or an HTTP
request. Anything else, or a server-first protocol, is closed.
- Name fields are `host_patterns` on `http` and `https`, `sni_patterns`
on
`tls_passthrough`. Same wildcard grammar as before, documented once on
`HTTPRule.host_patterns`.
- `ports` on all three protocols are strings: a port number, or `"*"`
alone for any
port. `http` defaults to `["80"]` and `https` to `["443"]` when empty;
`tls_passthrough`
requires at least one. The port is the destination the Actor connected
to.
A port in a request's authority is neither matched nor dialed. Format
documented
once on `HTTPRule.ports`.
- `inject_static_headers` is renamed to `replace_headers`, since the
behavior is
replacement and the name leaves room for other header operations later.
The
`CredentialHeaderInjection` message keeps its name. Replacement is
conditional:
a header is replaced only when the Actor's request already carries it,
with any
placeholder value. Requests without the header pass unchanged.
- Unsupported in v1 rules for arbitrary TCP, non-TCP protocols.
- `google/protobuf/empty.proto` import dropped; nothing uses it now.
Questions
- The handler is called `https`, not `mitmHttps`. Everything in an
`https`
block is intercepted. is that clear enough?
follow-ups
- align egress implementation
- named CONNECT, or some way to preserve what the actor dialed.
- add tcp support
- hand-written validators for the new fields (`host_patterns`,
`sni_patterns`, `ports`,
the rule conflict check) and regenerate `zz_generated.validation.go`
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes: #823 (there are probably more issues that i need to find)
Refactors the atelet Restore API so the DATA_ON_GOLDEN base snapshot
travels as a typed restore source instead of a bare string, preparing
for the node-local snapshot cache (#690, #1551): `RestoreRequest` gets
its own external snapshot *source* message, and read-side attributes of
a restore source have a home — today the URI, next (in the M2 cache
work) a sharing property set by the control plane.
**This change is not backward compatible.** `golden_snapshot_uri` is
removed from `RestoreRequest` with no transition: an ateapi from before
this change sends a DATA_ON_GOLDEN restore an updated atelet rejects (no
`base_config`), and an updated ateapi sends one an old atelet rejects
(no `golden_snapshot_uri`). ateapi and atelet must be rolled together.
Fresh starts, FULL/DATA restores of an actor's own snapshot, and local
pause restores without a golden base are unaffected.
Three commits, each buildable and tested:
1. **`ateletpb: give Restore its own external snapshot source message`**
— `ExternalRestoreConfiguration` replaces
`ExternalCheckpointConfiguration` in `RestoreRequest`'s config oneof
(same wire shape; the checkpoint-side write-destination message is
untouched), and `base_config` is added for the DATA_ON_GOLDEN base. No
behavior change: nothing sets or reads the new field yet.
2. **`ateapi, atelet: carry the restore base snapshot in base_config`**
— the cutover: ateapi sets `base_config` on both golden-data resume
paths (external data snapshot, and local pause checkpoint combining with
the golden); atelet reads and validates only `base_config`. The
characteristic test of the ateapi→atelet request and the golden-data
functional test pin the new field.
3. **`ateletpb: remove golden_snapshot_uri from RestoreRequest`** — the
old field is deleted and `base_config` takes its field number, keeping
the numbering dense.
Tested: full ateapi + atelet suites including the controlapi functional
tests; gofmt, golangci-lint, boilerplate clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Part of #1756.
- Adds `--actor-jwt-issuer`. Unset, it defaults to
`https://idp.<namespace>.svc`, the Service ate-idp-server will serve
discovery from. Before this, `iss` was hardcoded to
`https://api.ate-system.svc`, which points at ateapi's gRPC port.
- Adds `oidcdiscovery.ParseIssuer`, which requires a canonical https URL
with no query, fragment, or user info and strips trailing slashes.
ate-idp-server will use it too, so both emit the same bytes.
- Actor JWT headers now include `typ: JWT`.
- Drops the `actorIdentityJWTIssuer` plumbing into `RPCService`, unused
since #1315.
The default `iss` changes, but `MintActorJWT` has no production caller
yet.
Testing: new unit tests for the parser, the default, the header, and
flag resolution. `TestMintActorJWT_Success` now checks `typ`, `iss`, and
`sub`. It needs Docker, so it hasn't run locally.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR (docs
land later in the #1756 series)
Fixes #<issue_number_goes_here>
NA
This PR enhances the tests to validate volume lifecycle during state
transition failures
• **TestDeleteActor_VolumeDeletionFailure_RetrySuccess**- Verifies that
when volume deletion fails during DeleteActor, the actor transitions to
ACTOR_STATE_DELETING and its volumes to STATUS_DELETING. A
subsequent retry successfully finalizes volume deletion and deletes the
actor from the store.
• **TestResumeActor_VolumeAttachFailureAndRetry** - Verifies that when
volume attachment fails during ResumeActor, the actor remains in
ACTOR_STATE_RESUMING with worker assignment and provisioned volumes
intact. Retrying ResumeActor re-attempts volume attachment and
successfully transitions the actor to ACTOR_STATE_RUNNING.
• **TestResumeActor_VolumeAttachFailure_DeleteActor** - Verifies that an
actor stuck in ACTOR_STATE_RESUMING after an attach failure rejects
standard DeleteActor requests, but calling DeleteActor with
AnyState=true successfully cleans up worker assignments and provisioned
volumes.
• **TestResumeActor_MultiVolumePartialAttachFailure_Retry** - Verifies
that when attaching multiple volumes during ResumeActor and one volume
fails, previously attached volumes remain attached and the actor
stays in ACTOR_STATE_RESUMING. A subsequent retry attaches only the
remaining unattached volumes and transitions the actor to
ACTOR_STATE_RUNNING.
• **TestSuspendActor_VolumeDetachFailure_RetrySuccess** - Verifies that
volume detach failures during SuspendActor leave the actor in
ACTOR_STATE_SUSPENDING with its worker assignment held. A subsequent
retry
succeeds in detaching the volume, releasing the worker, and
transitioning the actor to ACTOR_STATE_SUSPENDED.
• **TestSuspendActor_VolumeDetachFailure_DeleteActorAnyState** -
Verifies that an actor stuck in ACTOR_STATE_SUSPENDING due to a volume
detach failure rejects standard deletion, but cleanly detaches volumes
and deletes the actor when called with AnyState=true.
• **TestPauseActor_VolumeLifecycle_DetachAndResumeAttach** - Validates
the volume lifecycle across pause and unpause, confirming external
volumes are detached when transitioning from RUNNING to PAUSED and re-
attached when resumed back to RUNNING.
• **TestPauseActor_VolumeDetachFailure_RetrySuccess** - Verifies that
volume detachment failure during PauseActor leaves the actor in
ACTOR_STATE_PAUSING. A retry of PauseActor re-attempts volume
detachment,
succeeds, and advances the actor state to ACTOR_STATE_PAUSED.
• **TestResumeActor_PausedLocalSnapshotMissing_Crashes** - Verifies that
if local snapshot files on the worker are missing when resuming a PAUSED
actor, the control plane marks the actor ACTOR_STATE_CRASHED
and frees the assigned worker pod.
• **TestDetachActorVolumes** - Tests control plane volume detachment
logic across multiple scenarios, including container mount filtering,
partial detach errors, unassigned workers, and idempotent not-found
responses from plugins.
• **TestCreateActorVolumes (Storage class parameters case)** - Verifies
that parameters configured on a StorageClass (such as disk and
filesystem types) are correctly propagated into the volume context of
newly created external volumes.
• **TestEnsureExternalSnapshotsReleased_DeletePrefixFailure** - Verifies
that transient object store errors during actor deletion preserve
un-deleted snapshot objects and return an error, which cleanly
completes deletion on a retry.
• **TestMountExternalVolumes** - Tests worker-side volume mounting,
verifying host mount directory creation, handling pre-existing paths,
plugin errors, and ensuring that partial failures during multi-volume
mounts do not unmount already mounted volumes.
• **TestVolumeHostDirectoryCleanup** - Verifies worker-side host
directory management, ensuring volume unmounting preserves mount
directories while directory reset removes empty directories and prevents
data
loss on non-empty ones.
• **TestExternalVolume_NodeMigration** - Verifies cross-node actor
migration with external volumes by suspending an actor, evicting its
worker pod, and resuming it on a new worker node while ensuring
application state is preserved
- [ x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Follow up on the comment -
https://github.com/agent-substrate/substrate/pull/1675#discussion_r4031619230
Stamps `actor_template_uid` onto `ExternalSnapshot` so template
provenance travels with the snapshot itself, rather than being inferred
from the single `ActorStatus.current_actor_template_uid`.
#### Motivation
`current_actor_template_uid` records the template the **last sprint
booted with** (`finalizeRunning` stamps it on every resume). It does
*not* record the template the **snapshot** was captured under. Those two
diverge as soon as an actor runs a sprint that does not produce a new
durable snapshot, and `loadActorForResume` was using the former to
decide whether to force a `DATA`-only restore instead of `FULL`.
Walking an actor through a repoint and a crash:
| Step | Action | `current_…_uid` | `ExternalSnapshot` |
|---|---|---|---|
| **a** | Suspend on template **A** | A | captured on **A** |
| **b** | `UpdateActor` repoints the spec to **B** | A | captured on
**A** |
| **c** | Resume → `A != B`, restores `DATA`, sprint boots on **B** |
**B** | still on **A** |
| **d** | Actor crashes (or is reverted) — no new snapshot taken | B |
still on **A** |
| **e** | Resume → `current == target == B`, so **no repoint is
detected** | B | still on **A** |
At step **e** the old check reports "template not replaced" and restores
the external snapshot in `FULL` — replaying a memory image captured
under **A**'s sandbox on **B**. That is exactly the case the guard
exists to prevent; it was silently defeated by the sprint at step **c**
advancing `current_actor_template_uid` past the snapshot.
*(Note: Paused actors do not face this divergence. Because `UpdateActor`
is only allowed in the `SUSPENDED` state, a paused actor cannot have its
template updated. Therefore, local checkpoints do not need separate
template provenance tracking).*
#### Changes
- **Data Model Updates:** Added the `actor_template_uid` field to
`ExternalSnapshot`.
- **Snapshot Stamping:** Updated the capture logic to stamp external
snapshots with the active sandbox's `ActorTemplate` UID when suspending
an actor or creating an actor from a tag.
- **Resume Evaluation Logic:** Modified the external resume logic to
compare the target template UID against the specific snapshot's UID
(`ExternalSnapshot.actor_template_uid`) rather than the actor's overall
status. Local restores bypass this check entirely since templates cannot
change during a pause.
**Tests**
- `TestResumeActor_AteletWireRequest` (unit): snapshot fixtures now
carry `actor_template_uid`, since provenance is read from the snapshot
rather than from actor status.
- Functional expectations for create/update/resume/pause updated for the
new field on the external snapshot golden files.
- `TestUpdateTemplateLifecycle` (e2e): asserts
`external_snapshot.actor_template_uid` after suspend and re-suspend.
- **e2e (Update Template)**: Added coverage for repoint detection after
a revert (verifying the `resume` → `revert` → `resume` path described in
the table above).
- **e2e (Combined Volumes)**: Added coverage to explicitly verify which
volumes a revert rewinds.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes https://github.com/agent-substrate/substrate/issues/292
This flips the condition to crash actors: any errors returned by atelet
will cause actors to crash by default.
If we later discover specific error classes that are safe to retry, we
can explicitly mark them as retriable.
TagStatus.source_actor_uid is unused, we can add this back if we ever
need it.
#1168
We were also missing some of the DV for TagStatus fields:
* Added maxLength=2048 requirement for ExternalSnapshot.snapshot_uri
(matching in_progress_snapshot_uri)
* Added validation for actor_template_uid
* Added validation for storage_location, matching what we have in
SnapshotsConfig.storage_location
# Description
**Substrate assumes the canonical install layout, and every deviation
fails closed.** The namespace `ate-system`, the Services `api` /
`atenet-router`, and the ServiceAccounts `atelet` / `atenet-router` are
compiled-in constants. An install in a per-developer namespace, or under
a deployment that prefixes resource names, breaks — and no failure
points at the naming.
This series makes the namespace, the Service names, and the
ServiceAccount names configurable. Every option defaults to the
canonical value. A canonical install is byte-for-byte unaffected.
| Hardcoded assumption | Failure | What it looks like instead |
|---|---|---|
| atelet's namespace in ateapi's SPIFFE check | ateapi rejects every
atelet | an mTLS handshake failure, not a naming error |
| atelet's namespace in the worker's broker check | no actor obtains a
certificate | `credential broker is not atelet` |
| NetworkPolicy's ingress namespace | the CNI drops every request to the
pool | a silent network fault |
| ateapi Service name and namespace in the client | the client cannot
authenticate | `services "api" not found`, then `invalid bearer token` |
| Resource names in `hack/install-ate.sh` and the e2e harness |
authorities land in the wrong namespace | suites die in preflight |
**A SPIFFE ID breaks on two axes.** It names a namespace and a
ServiceAccount. Both were constants, and the failure surfaces as a
rejected peer, not a missing object.
## Design notes
- **ate-controller passes the worker-side identities.** ateom already
took `--atunnel-client-identity` as a flag but relied on its default. A
new `--atunnel-broker-identity` flag alone would be inert: correct only
where the constant was already correct. The controller knows the control
plane's namespace, so it supplies both.
- **ateapi takes the whole expected identity, not a namespace.** The
dialer and `ateletauth` used the namespace only to build one string. One
place decides how atelet's identity is spelled.
- **`ateletauth` deduplicates without taking the dependency it was
avoiding.** Its constants are duplicated rather than imported so the
package does not depend on `controlapi` for three strings. `ateletdial`
would make a third copy, so the strings move to
`internal/installdefaults` instead — a leaf package of constants that
imports only the standard library, so consuming it does not reintroduce
the coupling the duplication was there to prevent.
- **The client reads environment variables, not flags.** It runs outside
the cluster. It has no downward API and nothing to discover from.
- **Namespaces come from the downward API, not flags.** Every supported
topology co-locates these components. Only Service and ServiceAccount
names, which a deployment may legitimately rename, get flags.
- **`hack/install-ate.sh` and the e2e harness are in scope.** A
relocated install cannot be bootstrapped or exercised without them. The
script refuses `ATE_NAMESPACE` or `ATE_API_SERVICE_NAME` overrides on
the manifest-applying subcommands, which would half-install:
`manifests/ate-install/` names `ate-system` and `api` literally.
## Why this is one PR
The series is stacked, not parallel. Commit 1 creates
`internal/installdefaults` and seeds seven imports; commit 2 adds
fourteen more; every later commit builds on those constants. Split into
separate PRs, each blocks on the previous merging and none reviews
independently.
The guard test cannot land first. Applied to `main` it fails on eleven
hardcoded literals across `ateletauth`, `controlapi/informer.go`,
`networkpolicy_controller.go`, both `ateom` mains, `ateclient`,
`ateletdial` and three e2e files. It is green only because the preceding
commits removed them.
A partial series still fails closed. Each commit's audit turned up more
places assuming `ate-system`, so landing commits 1-3 without 4-5 leaves
a relocated install broken later and less legibly than before.
`CONTRIBUTING.md` covers this case: when the intermediate steps are not
useful on their own, keep the change as one PR split into commits at
logical break points. Review commit by commit, and preserve the commits
on merge. If you would still rather split, the one defensible cut is by
axis — relocation (commits 1-3) then renaming (commits 4-5) with the
guard test rebased on top.
## Rebase notes (2026-09-15)
The series is rebased onto `main` at `d0d85c39`, ninety-two commits on
from the base the PR first carried:
- `main` moved actor JWT and certificate minting out of
`cmd/ateapi/internal/actoridentity` into `controlapi`, deleting the
package and dropping its inline atelet authorization check in favor of
the unified authorizer. This series previously threaded the expected
atelet identity into that check; the check no longer exists, so that
adaptation is gone. `ateletauth` survives with one caller,
`workerservice.SetWorkerCapacity`, which still takes the configured
identity.
- The same refactor replaced `BrokerConfig.ExpectedActorUID` with an
`ActorAtespace` / `ActorName` / `ActorUID` triple. `AteletSPIFFEID` is
additive and still required by `ateletdial.TLSConfig`.
- `ateletauth` (ateapi's side) and `ateletdial` (the worker's side) each
re-declared the hardcoded SPIFFE ID. Both take the expected identity as
a parameter.
- `main` deleted the atenet DNS subsystem. The series no longer touches
it, and `installdefaults` carries no DNS Service name.
- The controller passes `--atunnel-broker-identity` only when it differs
from the canonical default. An ateom old enough to predate the flag
exits on it, and `docs/upgrade.md` keeps such a pool serving during a
rolling upgrade. A relocated install gets the flag and necessarily runs
an image that accepts it.
- `main` added `internal/e2e/collector_metrics.go` (agentgateway CI
support), which addresses the router through the package-level
`routerNamespace` and `routerService` that this series replaces. Its two
call sites become `SystemNamespace()` and
`ResourceName("atenet-router")`, matching `router_client.go` and
`statusz.go`. A reviewer diffing against the older base sees those call
sites move; nothing else in that file changes.
- `main` added `cmd/credential-provider/kubernetes-secrets` (`#1335`),
whose `injectorSPIFFEID` constant is the mTLS peer check on the only
caller permitted to read Secrets. The comparison is an exact string
match, so a renamed install rejects every fetch at TLS. It becomes
`--injector-identity`, defaulting to
`installdefaults.EgressSPIFFEID(SystemNamespace)`, alongside a new
`EgressServiceAccount` constant and helper. The flag name follows
`--atunnel-client-identity`; the manifest beside it still names
`ate-system`, so a renaming deployment configures both.
- `main` replaced the dialer's worker indexer with
`DialForAteletOnNode`, dropped `WorkerPodInformer` from `controlapi`,
and removed `kataConfig` from ateom-microvm's `NewService`. The series
adapts to each narrower signature and keeps only its own added
parameter.
- The guard test's allowlist entry for the two inert `ate-system`
literals follows the code from `actoridentity.go` to
`controlapi/actor.go`. The refactor re-introduced exactly the class of
constant this series removes, so the guard earns its place.
# Testing
- `go test -race ./...` passes. `make verify` passes boilerplate,
codegen, go-modules, gofmt, golangci-lint, kube-api-linter, licenses,
metrics and postgresql-migrations. `proto-fmt` needs `clang-format`,
which is absent locally; this series touches no `.proto` files.
- Every commit builds and vets individually, via `git rebase -x 'go
build ./... && go vet ./...'`.
- On a fresh kind cluster, all twelve e2e suites pass: `capabilities`,
`combinedvolumes`, `demo`, `egressauthz`, `egressmitm`, `example`,
`identity`, `metrics`, `networking`, `networkpolicy`, `parking`,
`sizing`. The `demo`, `metrics`, `networkpolicy`, `parking` and
`networking` suites need the `--deploy-demo-counter` and
`--deploy-demo-egress` fixtures, and the MITM path needs
`--deploy-demo-egress-mitm`; without them they fail on `actor template
not found`, which reads as a control-plane fault rather than a missing
fixture.
- With the sdsmint egress gateway (`--deploy-atenet
--experimental-use-sdsmint`, `E2E_EGRESS_MITM=1`),
`TestActorEgressMITMTrust` and `TestActorEgressHTTPSByHostnameMITM` both
pass, and the `networking` suite is green at 25 passed. The two modes
are mutually exclusive by design:
`TestActorEgressHTTPSByHostnamePassthrough` covers the plain gateway and
skips under MITM, and the two MITM tests skip without it. Both modes
were run, so every case executed in one of them.
- The new unit tests use a relocated namespace and renamed
ServiceAccounts. Reintroducing each hardcoding makes them fail.
One e2e caveat that predates this series and misleads: the `identity`
suite calls `ReplaceEgressTrustPool`, which takes over the shared
`egress-mitm-ca-pool` Secret, overwrites it and registers no cleanup.
Any MITM test running afterwards fails with `certificate signed by
unknown authority`, which reads as a broken interception path rather
than a mutated fixture. Reinstall the gateway before running
`egressmitm`.
# Additional Notes
**Credential contents stay canonical.** `controlapi/actor.go` still
mints credentials naming `api.ate-system.svc`: the JWT issuer and the
certificate's Issuer CN. Relying parties validate these, so changing
them is a compatibility decision, not a lookup fix. The code carries a
TODO to make the issuer a globally unique, OIDC-compliant name.
#### Summary
This PR introduces the `RevertActor` RPC for actor lifecycle management.
It includes the API definition, the corresponding workflow execution
logic, observability metrics, and updates to the authorization model to
support reverting actors.
Fixes#1556
Docs updated in #1711
#### Commit-wise Changes
**1. Add RevertActor RPC (`1145d99`)**
* Introduces the new `RevertActor` RPC to the API definitions.
* Updates the corresponding protobuf bindings (affecting files like
`ateapi.pb.go` and `ateapi_pb2.py`).
**2. Add the revertActor workflow and observability metrics
(`dcb1fae`)**
* Implements the core `revertActor` workflow logic, designed to be
idempotent and re-enterable. It progresses through the following steps:
* **Mark Reverting:** Validates that the actor is in a revertable state
(`RUNNING`, `PAUSED`, or `CRASHED`) and transitions its state to
`REVERTING`.
* **Discard Worker:** Safely tears down the execution environment by
terminating the workload, detaching volumes, and releasing the assigned
worker.
* **Collect In-Progress Snapshot:** Cleans up external object storage by
deleting any objects a previous suspend operation was partway through
writing.
* **Finalize:** Commits the actor to `SUSPENDED` and strips all
node-local and in-progress state pointers (clearing `WorkerAssignment`,
`LocalSnapshotInfo`, etc.), returning the actor to its untouched
external snapshot.
* Instruments the workflow with lifecycle operation metrics (e.g.,
updating `ate.actor.lifecycle.operation.duration` to track `revert`
operations).
* *Note/TODO:* Currently, when reverting a paused actor, the workflow
drops the pointer to the node-local state but does *not* actually prune
the local checkpoint bytes from the node (this is tracked in #641).
**3. Add `can_revert` to the authorization model (`2fc53ae`)**
* Adds the `can_revert` permission to the auth model, mirroring the
shape of `can_suspend` (editor tier of the parent atespace, plus a
direct grant so a machine identity can revert the actor it drives
without holding an atespace role).
**4. Serve RevertActor and add the CLI verb (`f04fab4`)**
* Wires the `Control.RevertActor` service method to the workflow
(replacing the generated stub that previously answered `Unimplemented`).
* Adds the `"ate revert actor"` CLI command, making the feature usable
end-to-end.
* Implements `Terminate` for the fake atelet. This was necessary because
reverting an actor from the `RUNNING` state is the first path to reach
this call in functional tests (previously, delete tests skipped this
step as they ran against actors with no worker assignment).
**5. Add a manual verify script for RevertActor (`8022db3`)**
* Adds a script to manually exercise `RevertActor` against a real
control plane, since unit and functional tests only run against a fake
atelet.
* Tests reverting from `CRASHED`, `RUNNING`, and `PAUSED` states, and
verifies that attempting to revert a `SUSPENDED` actor is properly
rejected.
* Simulates a crash by deleting the worker pod the actor runs on to
verify the workflow can handle the absence of a worker to terminate.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Dialing by resolving the worker pod is problematic, because in some
cases (e.g. during Worker Pod deletion), we need to dial to atelet (to
properly call `Terminate`), but the worker is gone.
A much simpler model is to always dial by resolving the IP of the atelet
on a given node.
#1522 added a resyncInterval parameter to NewActorTemplateReconciler and
#1523 added this call site. They landed without seeing each other, so
`main` doesn't compile.
SEt the resyncInterval to 7s, to match what we have in other tests
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1507
Golden snapshots currently remain owned by the temporary golden actor,
so another resume/suspend cycle or actor deletion can collect a snapshot
still referenced by its template. The controller now copies the warmed
snapshot into a published tag, deletes the golden actor, and records the
tag reference on the template. Interrupted tag creation and cleanup
remain retryable; template deletion cleans up both resources.
`CreateActor` resolves an explicit `sourceTag` or the template's golden
tag into the actor's initial snapshot. Actors created before the golden
tag is ready retain their cold-boot behavior. The golden tag uses the
template UID as its name in `ate-golden`. The proto replaces
`golden_snapshot` with `golden_tag` at field 1, without backward
compatibility.
This PR is based directly on `main` and does not depend on #1521.
Follow-up recommendation: move the create → resume → wait → suspend →
tag → delete sequence into a golden-template workflow using the existing
workflow conventions. The reconciler now coordinates multiple
recoverable steps; it could retain scheduling and retries while
delegating that sequence to the workflow. This refactor is outside this
PR.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Validation: full `env -u NO_COLOR make verify` passed after rebasing
onto `main`. After the final proto field-number change, bindings were
regenerated and the control API unit/functional tests plus proto-format
and Go-format checks passed.
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Populate default fields during both Create and Update RPC calls so
that resources are stored with defaults instead of zero values.
This is a much cleaner approach as it means that a) consumers don't need to
be aware of the default value and b) default values can potentially change.
Only ActorTemplate had defaults, but added default functions for all
resources so that all RPC handlers are wired up.
Fixes a few todos in the `ateapi.proto`
* Adds custom validation for `create_time` and `update_time` fields.
* Adds better validation method for the `Container.image` field.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Fixes#1432
Map known store pagination errors in `ListActorTemplates` through
`mapListError`, matching the other list RPC handlers. Invalid page
tokens now return gRPC `InvalidArgument` with `invalid page_token`.
- [x] Tests pass: ran `make verify`
- [x] Appropriate changes to documentation are included in the PR:
single line change
This was originally a separate service to make it easy to apply separate
authentication and authorization interceptors.
It now seems clear that our authn/z framework will be strong enough to
support atelet and external callers in one system (based on OpenFGA).
This change moves the MintJWT and MintCert RPCs into the control API,
and removes some inline authz checks that will be handled by our unified
authorizer framework.
The container image must include the image digest. This validation was
dropped during the migration of the ActorTemplate resource from CRD to
Substrate API.
Fixes#1508
Store tag snapshots at `<base>/atespaces/<atespace>/tags/<tag-uid>`.
Replace `in_progress_snapshot_uri` with immutable `storage_location`, so
pending and completed tags share UID-based cleanup independent of the
source actor or template.
- [x] Tests pass: race-enabled control API tests and PostgreSQL tag
contract tests.
- [x] Documentation updated.
Lint and code-generation verification also passed.
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
## Summary
Fixes the flaky test `TestActorLifecycle_WithExternalVolumes` in
`cmd/ateapi/internal/controlapi/functionaltest` (`unit` lane).
**Root cause:** the test pauses and immediately resumes against a single
1-capacity worker; PauseActor commits the assignment release to the
store, but AssignWorker reads the worker cache, which sees the release
only after the PostgreSQL watch delivers it — losing that race fails the
resume with `ResourceExhausted: no free workers available`.
**Fix:** wait for the release to reach the cache with the existing
`waitForWorkerAvailable` helper (the same pattern sibling lifecycle
tests use after suspend) before resuming.
Closes#1533
## Evidence
Flake rate over last 7 days: 5.7% (26 fail / 427 pass across 453 runs)
Failures span 14 branches incl. 9 on main; infra-failure runs excluded:
0
Failing runs:
https://github.com/agent-substrate/substrate/actions/runs/33919495958,
https://github.com/agent-substrate/substrate/actions/runs/33941175484
Passing runs:
https://github.com/agent-substrate/substrate/actions/runs/33567474203,
https://github.com/agent-substrate/substrate/actions/runs/33566785143
## Test plan
- [x] `go test -race -count=5
./cmd/ateapi/internal/controlapi/functionaltest/ -run
TestActorLifecycle_WithExternalVolumes` — passes (was intermittently
failing before at ~6%)
Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>
Renames the proto message, its status message and scope enum, the five
RPCs and their request and response messages, the actor_snapshot_tag
request fields, and Actor.source_snapshot_tag to source_tag. The store
interface, its Postgres table, the object-storage prefix segment and the
kubectl-ate verbs follow, so nothing keeps the old spelling.
https://github.com/agent-substrate/substrate/issues/664
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#664
This PR implements the idea described in
https://github.com/agent-substrate/substrate/issues/664#issuecomment-5499311489
It does more than Garbage Collection of snapshots, because we also got
rid of the Snapshot resource (from the DB/API).
Now, an external snapshot is owned by a single resource:
- An Actor owns the snapshot it writes at suspend
- A tag owns a copy taken at tag creation,
- An actor cloned from a tag borrows the tag's snapshot until its own
first suspend.
Garbage Collection: whoever created/owns the snapshot is the only one
who ever deletes them:
i.e., if an actor is deleted and it owns a snapshot. The underlying
snapshot is deleted with the actor.
this PR:
- Drops table actor_snapshots
- Keeps table actor_snapshot_tags
- Adds an object copy at tag creation, and an owned versus borrowed
distinction on the Actor
- Adds synchronous external snapshot deletion at actor suspend, at actor
delete, and at tag delete
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #1266
This is a draft of the core API + data store changes.
It's still a large PR, apologies.
The "as rows" commit could be split out, but this takes it to ~all of
the breaking changes we can't hide behind updating internals.
Same for the claimlock, but in both cases it seems these are worth
understanding when considering the API shape.
They're loadbearing for performance once we actually have multi-actor
workers.
This PR moves sandbox config selection from the WorkerPool to the
ActorTemplate.
The existing behavior is preserved while we are designing the upgrade:
sandbox config still cannot be updated once set (ActorTemplates are
create-only and `sandbox_config` is immutable).
For now the ActorTemplate still *requires* `sandbox_config.config_name`
— there is no resolution of the cluster default (`spec.default`). This
is temporary while we figure out the defaulting design.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes https://github.com/agent-substrate/substrate/issues/477 Implements
the following actor template update flow:
```
UpdateActor(): set .template = templ-v2
ResumeActor()
resumes using templ-v2
wait for readyz
if fail: return failure
.template = templ-v2; .status.current_template = templ-v1
(the next resume will attempt templ-v2 again)
set .status = STATUS_RUNNING
set .status.current_template = templ-v2
return success
.template = templ-v2; .status.current_template = templ-v2
```
This PR is large, I grouped changes to the following commits:
- 1d5d8f06: moves consumers off the CRD path: demos, ate-setup scripts
and the e2e suites address templates by the actor_template ref.
- e502cf60: Ate API changes: removes actor_template_namespace,
actor_template_name from Actor, ActorAssignment and ActorSnapshot in the
public API, drops the CRD conversion fallback.
- 8f6c1adb: atecontroller: removes the ActorTemplate CRD controller.
Once this PR is submitted, existing demos that still uses CRD will stop
working. Created https://github.com/agent-substrate/substrate/pull/1355
to update existing demos.
* Renames the`ate.template.namespace` attribute to
`ate.template.atespace` and sources it from the actor's `actor_template`
ObjectRef instead of the legacy CRD namespace/name pair.
* Updated functional tests now to use substrate ActorTemplates and bind
actors through the `actor_template` ref.
Remove `ActorTemplateStatus.sandbox_assets` along with the
`SandboxAssets` messages it referenced. Nothing ever wrote the field:
sandbox assets are resolved from the WorkerPool and SandboxConfig
objects at resume time and travel to atelet via ateletpb, so freezing
them into the template status never materialized.
This requires more thought, one option is to add it as a field of
GoldenSnapshotStatus in the future, and make GoldenSnapshotStatus a
repeated field to allow multiple goldens.
I noticed my machine filling up with postgres containers ...
---
First:
storetest starts one PostgreSQL container per test binary in a sync.Once
and never stops it, so every package that uses it leaks a container per
run. Testcontainers' Ryuk reaper normally covers this, but it is not
guaranteed to be running, and nothing else stops the container.
Add Shutdown, plus a RunTests wrapper for packages whose TestMain does
nothing else, and call it from the three packages that use storetest.
atepg already terminates its own container this way.
----
Second:
testcontainers does not respect docker contexts, making it difficult to
use with [limactl](https://lima-vm.io/) etc.
See: https://github.com/testcontainers/testcontainers-go/issues/2607
We automate the workaround to set the host env to point at the socket
from the active docker context.
Rename `WorkerPoolSpec.AteomImage`to `WorkerImage` and drop the
constraints so the field can be left unset.
This is the basis for let an empty workerImage lets the controller
inject a versioned default image chosen by the pool's sandbox class.
Part of #861
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Create/Suspend/Resume Actor workflows now uses the
`Actor.actor_template` ObjectRef when specified, otherwise falls back to
the CRD in cmd/ateapi/internal/controlapi/template_convert.go.
Adds a temporary converter that allows each Actor workflow to use the
substrate resource rather than CRD, to prevent regression while we are
in the parallel state.
* Allows each workflow to use the ActorTemplate substrate proto under
the hood, in preparation for the full cut over, e2e tests start testing
the new code path.
* Changes any functions that consumes the ActorTemplate CRD to use the
substrate proto instead.
* The converter will be deleted once we fully cutover to the substrate
resource.
Fix#1213, we should also reject unknown fields at request level.
Refactor all functionality back into a gRPC interceptor to check every
incoming request, no need to handle every new request method.
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Shruti Nair