Part of #1266
This is a draft of the core API + data store changes.
It's still a large PR, apologies.
The "as rows" commit could be split out, but this takes it to ~all of
the breaking changes we can't hide behind updating internals.
Same for the claimlock, but in both cases it seems these are worth
understanding when considering the API shape.
They're loadbearing for performance once we actually have multi-actor
workers.
This PR moves sandbox config selection from the WorkerPool to the
ActorTemplate.
The existing behavior is preserved while we are designing the upgrade:
sandbox config still cannot be updated once set (ActorTemplates are
create-only and `sandbox_config` is immutable).
For now the ActorTemplate still *requires* `sandbox_config.config_name`
— there is no resolution of the cluster default (`spec.default`). This
is temporary while we figure out the defaulting design.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes https://github.com/agent-substrate/substrate/issues/477 Implements
the following actor template update flow:
```
UpdateActor(): set .template = templ-v2
ResumeActor()
resumes using templ-v2
wait for readyz
if fail: return failure
.template = templ-v2; .status.current_template = templ-v1
(the next resume will attempt templ-v2 again)
set .status = STATUS_RUNNING
set .status.current_template = templ-v2
return success
.template = templ-v2; .status.current_template = templ-v2
```
This PR is large, I grouped changes to the following commits:
- 1d5d8f06: moves consumers off the CRD path: demos, ate-setup scripts
and the e2e suites address templates by the actor_template ref.
- e502cf60: Ate API changes: removes actor_template_namespace,
actor_template_name from Actor, ActorAssignment and ActorSnapshot in the
public API, drops the CRD conversion fallback.
- 8f6c1adb: atecontroller: removes the ActorTemplate CRD controller.
Once this PR is submitted, existing demos that still uses CRD will stop
working. Created https://github.com/agent-substrate/substrate/pull/1355
to update existing demos.
* Renames the`ate.template.namespace` attribute to
`ate.template.atespace` and sources it from the actor's `actor_template`
ObjectRef instead of the legacy CRD namespace/name pair.
* Updated functional tests now to use substrate ActorTemplates and bind
actors through the `actor_template` ref.
Remove `ActorTemplateStatus.sandbox_assets` along with the
`SandboxAssets` messages it referenced. Nothing ever wrote the field:
sandbox assets are resolved from the WorkerPool and SandboxConfig
objects at resume time and travel to atelet via ateletpb, so freezing
them into the template status never materialized.
This requires more thought, one option is to add it as a field of
GoldenSnapshotStatus in the future, and make GoldenSnapshotStatus a
repeated field to allow multiple goldens.
I noticed my machine filling up with postgres containers ...
---
First:
storetest starts one PostgreSQL container per test binary in a sync.Once
and never stops it, so every package that uses it leaks a container per
run. Testcontainers' Ryuk reaper normally covers this, but it is not
guaranteed to be running, and nothing else stops the container.
Add Shutdown, plus a RunTests wrapper for packages whose TestMain does
nothing else, and call it from the three packages that use storetest.
atepg already terminates its own container this way.
----
Second:
testcontainers does not respect docker contexts, making it difficult to
use with [limactl](https://lima-vm.io/) etc.
See: https://github.com/testcontainers/testcontainers-go/issues/2607
We automate the workaround to set the host env to point at the socket
from the active docker context.
Rename `WorkerPoolSpec.AteomImage`to `WorkerImage` and drop the
constraints so the field can be left unset.
This is the basis for let an empty workerImage lets the controller
inject a versioned default image chosen by the pool's sandbox class.
Part of #861
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Create/Suspend/Resume Actor workflows now uses the
`Actor.actor_template` ObjectRef when specified, otherwise falls back to
the CRD in cmd/ateapi/internal/controlapi/template_convert.go.
Adds a temporary converter that allows each Actor workflow to use the
substrate resource rather than CRD, to prevent regression while we are
in the parallel state.
* Allows each workflow to use the ActorTemplate substrate proto under
the hood, in preparation for the full cut over, e2e tests start testing
the new code path.
* Changes any functions that consumes the ActorTemplate CRD to use the
substrate proto instead.
* The converter will be deleted once we fully cutover to the substrate
resource.
Fix#1213, we should also reject unknown fields at request level.
Refactor all functionality back into a gRPC interceptor to check every
incoming request, no need to handle every new request method.
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Part of #477.
The controller has 2 components
1. A "producer" that periodically lists ActorTemplates and add work
items to a client-go workqueue.
2. A "consumer" that gets work items from the queue, and reconcile it.
AT will have the FAILED condition if we encounter non-retriable errors
during the transition
Fixes#1106. Takes over #1107 (adopting @Stevenjin8's approach and
commits from that PR — credit to him for the design; trailer omitted
only for CLA) and incorporates the review feedback it accumulated.
Workers were registered as schedulable the moment their pod had an IP —
long before ateom's gRPC server was listening, which is the scheduling
race in #1106 (actors dispatched to a cold ateom fail at dial time).
Now:
- ateom (gvisor + microvm) serves HTTP `/readyz` on a dedicated port
(8080) once its gRPC listener is up, flipping to 503 on SIGTERM so
draining workers stop receiving work.
- The worker Deployment probes that endpoint (replaces the earlier TCP
probe against the TLS port, which spammed handshake errors).
- `isWorkerEligible` requires `PodReady=True` in addition to an IP.
Eligibility gates **registration only** — a readiness flap never
deregisters a worker with a bound actor (pinned by
`TestSyncer_ReadinessFlapDoesNotDeregister`).
- A readiness server that cannot come up exits the process rather than
leaving a pod that looks alive but can never receive work.
Beyond #1107's head: dropped the pre-migration
`controlapi/syncer_test.go` copy (didn't compile after #1203 moved the
syncer), fixed the `workersync` fixtures for the new gate, and added
`TestSyncer_PodWithIPButNotReadyIsNotRegistered` (+ the flap test
above).
## Testing
- `go test ./...`: green, zero failures.
- `-race -count=10` on `workersync` + `serverboot`, `-race -count=5` on
`controllers`, `-race` on all of `controlapi` incl. functional tests:
green.
- Green under a hostile ambient env
(`KUBECTL_CONTEXT`/`KO_DOCKER_REPO`/`PROJECT_ID` set to garbage).
## Scope notes
- The one post-#1160 failure with a *post-restore* signature (run
32891995583: restored actor's own container `/healthz` timing out, actor
stuck RESUMING) is a different layer — ateom was demonstrably serving —
and is **not** claimed as fixed here. If it recurs, it deserves its own
issue. (The other suspected post-merge failure, run 32911674788, turned
out to be that PR's own x509 regression, not this flake.)
- Mixed-version caveat: this controller pointed at pods running a
pre-readyz ateom image leaves those workers unready/unregistered. The
supported install flows build both from the same source, so this only
matters for hand-rolled skew.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>
This mostly eliminates the need to hand-write validation code.
This PR is a long series of commits which add DV for most of Actor and
all of Atespace.
Here is a map to the commits:
* The first few take a dep on a new Kubernetes tag, import the code into
third_party, and apply a single patch. Because we have different Go
modules for tools, I had to do it twice. When that patch lands, we can
revert these commits, but that won't be until the 1.38 cycle in a few
months.
* The next commits slowly add DV support, so a human can review each of
them in a reasonable amount of time. The emphasis is on great test
cases.
* I made a bad choice early on as to where to generate the code into, so
I moved it. Rebasing on that was exceedingly hard, so I left it as a
move.
* This required changing update/go-generate -> update/codegen -- we need
to get the ordering of tools right, which `go generate` does not
guarantee.
* Then I added a "middle" layer called "ServiceImpl" between the RPC and
storage layers. This allows things like workflow to call the same
business logic as the RPC layer, including validation. Lots of test
fixes.
* Then I finished the Create() and Update() paths for Actor. Those
represent the "right" (or closest to) way to implement resources, and
tests for validation.
I strongly encourage reviewers to read it commit-by-commit. Rebasing
this is VERY tedious, so the sooner it lands or dies completely, the
better. Then we can start converting the rest.
`./hack/run-tool.sh validation-gen --docs` will produce some docs on the
tool and the available tags.
@laoj2 @juli4n @EItanya @HavenXia
@lalitc375 @yongruilin @jpbetz FYI
## Summary
A follow up to #640 where we introduced PostgreSQL as an alternative
storage backend, selected conditionally in ateapi.
- Deleted ateredis, its tests, and its dependencies
- Removed Redis backend selection and configuration so ateapi always
connects to Postgres
- Replaced Valkey resources with Postgres in the standard and Kind
deployment paths and simplified install script
- Replaced miniredis fixtures with isolated Postgres testcontainers and
added centralized helpers for seeding resources
- Renamed Redis-specific debug flush command to backend-neutral
`debug-clear-store` in CLI
- Updated comments and docs where applicable
## Benchmarking
Extensive benchmarking have been performed to evaluate Redis vs
Postgres, and results can be found in these two documents:
-
https://docs.google.com/document/d/10K0wB6aTeFkJCL4HN3NbLJCdFGoLYdhIcqkFnqFHkKc/edit?usp=sharing
-
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Having been reading this code for a while, and in light of changes we
want to make, I think something like this will help clarify.
I am not particularly attached to the name or style `RpcService` - it's
not usual Go convention but it is protobuf style. If anyone feels
strongly it could be `RPCService` or `ControlService` or something else.
@juli4n @laoj2
* Removed field_mask from the API
* Added a new protoupdate package to handle replacing mutable fields.
This makes sure that unknown fields in the server are not dropped by an
update from a stale/old client.
#1011
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #731 and follow up to #640. This PR makes improvements to the
Postgres store by addressing review comments on #940 and optimizing
Postgres updates as suggested in #988.
- Replaces Postgres row-locking updates for actors, templates, and
snapshot tags with bounded optimistic concurrency control using
UID/version CAS checks.
- Strengthens relational integrity with missing FKs and indexes, and
replace snapshot-tag creation pre-check with FK
- Removes unnecessary UID uniqueness constraints
- Standardize pagination and validation across stores, map malformed
tokens and invalid sizes to appropriate gRPC errors
- Improves actor resume race handling when workers disappear or
concurrent assignments exhaust their retry budget
- Improves reliability with startup retries, expired-lease cleanup,
worker-watch reconnection after malformed msg
- Expand Postgres, Redis, Contract, and Control API tests to cover new
constraints, error semantics, concurrency
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
The test calls `setupTest`, `namespaceForTest` and `assertGrpcError`,
which are defined in the `functionaltest` package, but the file landed
in `controlapi` declaring package `controlapi`, so the `controlapi` test
binary did not build.
This code passed on the PR branch, but it logically conflicted with a
previous refactor without merge conflict.
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#643
This feature introduces the ability to delete Substrate actors from any
lifecycle state (e.g., RUNNING, PAUSED, PENDING), rather than requiring
them to be hibernated into SUSPENDED (or CRASHED) beforehand. This is
crucial for cleaning up stuck or failed actors.
#### 1. Control Plane & gRPC API (pkg/proto/ateapipb, ateapi)
• DeleteActorRequest.any_state: Added a new boolean field any_state to
DeleteActorRequest.
• When false (default): enforces the existing behavior where only actors
in ACTOR_STATE_SUSPENDED or ACTOR_STATE_CRASHED (or already
ACTOR_STATE_DELETING) can be deleted.
• When true: allows deleting an actor in any state (e.g., RUNNING,
PAUSED).
• Orchestrated Cleanup Workflow (workflow_delete.go):
1. Transitions actor state to ACTOR_STATE_DELETING.
2. Calls atelet.Terminate to stop live workloads on the node.
3. Detaches external volumes from the worker node (with fallback logic
using actor status if the ActorTemplate was deleted).
4. Releases the assigned physical worker Pod back to the WorkerPool.
5. Deletes external storage volumes through the storage plugins.
6. Finalizes removal of the actor from the persistence store.
#### 2. Node Herder Agent (atelet)
• Terminate RPC (main.go): Added a new Terminate method to AteomHerder:
• Calls ateom.TerminateWorkload on the target worker pod.
• Unmounts external volumes mounted for the actor on the node.
• Cleans up and resets actor runtime host directories (OCI bundles,
checkpoints, pid files).
#### 3. In-Pod Sandboxes (ateom-gvisor & ateom-microvm)
• TerminateWorkload RPC:
• gVisor (main.go): Stops and deletes active runsc containers, unmounts
bundle rootfs overlays, and tears down the actor's interior network
namespace.
• Micro-VM (checkpoint.go): Shuts down the Cloud Hypervisor VMM,
unmounts bundle rootfs overlays, and cleans up the network namespace.
• Emits an "Actor terminated" lifecycle log and clears the active actor
attribution so the worker can host new workloads.
#### 4. CLI (kubectl-ate)
• --any-state Flag (delete_actor.go):
kubectl-ate delete actor <actor-name> -a <atespace> --any-state
──────
## Verification
[x] Changes tested in local with
```
• go build ./...: Passed
go test ./...: Passed (all unit and functional tests pass)
make lint: Passed (no lint errors)
```
Detailed manual tests
```
### Scenario 1: Force Delete a Running Actor with --any-state
1. Create the actor:
kubectl ate create actor manual-test-1 --template=ate-demo-counter-microvm/counter-microvm -a demo
2. Resume the actor:
kubectl ate resume actor manual-test-1 -a demo
3. Force delete the running actor:
kubectl ate delete actor manual-test-1 -a demo --any-state
actor "manual-test-1" deleted
```
──────
### Scenario 2: Standard Delete a Suspended Actor
```
1. Create the actor:
kubectl ate create actor manual-test-2 --template=ate-demo-counter-microvm/counter-microvm -a demo
2. Resume the actor:
kubectl ate resume actor manual-test-2 -a demo
3. Suspend the actor:
kubectl ate suspend actor manual-test-2 -a demo
4. Standard delete the suspended actor:
kubectl ate delete actor manual-test-2 -a demo
actor "manual-test-2" deleted
```
──────
### Unit & E2E Tests
# Unit & Functional Tests
go test ./cmd/ateapi/internal/controlapi -run
"TestDeleteActorWorkflow|TestEnsureMarkedDeleting"
# E2E Tests
E2E_TEMPLATE_NAMESPACE=ate-demo-counter-microvm
E2E_TEMPLATE_NAME=counter-microvm ./hack/run-e2e-kind.sh
./internal/e2e/suites/demo -run TestActorLifecycle
./hack/run-e2e-kind.sh ./internal/e2e/suites/demo -run
TestForceDeleteActorWithExternalVolume
```
- [ x] Appropriate changes to documentation are included in the PR
Fixes#957
`ate.actor.lifecycle.operation.duration` carried the pool pair on
suspend and pause only when they failed, so per-pool dashboards saw
those two operations exclusively as failures.
Both workflows record the histogram from a defer that reads the `actor`
variable, and the happy path reassigns it to the finalized record. The
finalize step commits the new state and the cleared `WorkerAssignment`
in one update, so the defer found no assignment and dropped both keys. A
failure returns the pre-finalize record, which still names the worker.
Both now snapshot `lifecycleOpAttrs(...)` just before the finalize step
— the same snapshot-before-clear crash.go does for the crash counter.
Paths that end earlier keep the current computation.
`delete` stays without a pool: it only runs from SUSPENDED or CRASHED,
which already released the worker, so there is none to name.
TestLifecycleOpPoolAttributesOnSuccess drives a real suspend and pause
through the gRPC service; both subtests fail without the fix.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Make workers a gobal resource, identified by a unique name. Worker APIs
will be implemented in a follow up change.
Part of #730
- [ x ] Tests pass
- [ x ] Appropriate changes to documentation are included in the PR
functional_test.go was also split into one file per resource, to match
what we're doing for the RPC handlers too. See #891
I also needed to make some changes to move the tests to a separate
package:
* `NewAteletDialer` now takes `options`, and `WithDialCredentials` lets
a test build its own transport credentials. The fake atelet is reached
over insecure transport. Didn't change existing callers. This is the
only non-test change.
* Moved a few unit tests that are actually functional tests to the
respective file under functionaltest/
Fixes#891
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Taahir Ahmed