This PR adds missing OpenTelemetry span attributes in UpdateActor and
corresponding tests.
- [X] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Add unit tests that exercise all the validation cases and assert on the
k8s `ErrorMatcher` instead of comparing raw error strings.
Moved most validations from functional tests to unit tests, and kept one
validation for each gRPC method to validate that handlers properly map
Go errors to gRPC statuses.
The pattern of passing a tuple of atespace and actor name together
is across the whole codebase. This simplifies both callers and callees
by leting them pass a single field that bundles both.
The ActorRef is the actor specific, typed,
in-process version of the ObjectRef we have in our gRPC API.
The error is only returned on an optimistic-concurrency version
mismatch. Rename it for that behavior so other retryable errors don't
get folded into it (but should introduce their own error name).
Followup for post-merge comment in #535
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Refresh the actor when `AssignWorkerStep`'s write hits a version
conflict so that the workflow won't kept retry with a stale version
actor and never succeed.
AI assisted on testing.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of [#170](https://github.com/agent-substrate/substrate/issues/170)
Establishes mutual TLS between all ate system components, and updates
the certificate plumbing it depends on.
Main changes:
1. The atenet router now verifies ate apiserver' serving certificate,
and presents its client cert to ate apiserver. Previously the connection
used `InsecureSkipVerify`.
2. AteApi server verifies atelet's serving cert.
Minor bug fixes:
1. Prevent `servicednssigner` from signing a cert with no DNS SANs. Also
updated valkey cluster's cert configuration, because it was relying on
the cert with empty DNS.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Initial part of #232.
Extends the ActorTemplate API to add support for per-actor external
volumes and adds control plane and atelet hooks following the Actor
lifecycle.
Callouts to volume operations are abstracted with a Volume interface.
Right now, only a mock volume plugin for testing has been implemented,
but we will add CSI support next.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR - Not
going to update documentation until we add CSI support.
AssignWorkerStep starts a background goroutine to release a stale
worker assignment. It builds its context from the request ctx, but
that ctx is cancelled right after the workflow returns (releaseLock
runs on defer). If the goroutine has not called UpdateWorker yet, the
call fails with a cancelled context and the worker keeps the stale
assignment forever. Nothing else clears it while the pod is alive.
Fix: use context.Background() for the release call, same pattern used
in cmd/atenet/internal/router/resumer.go.
Tested with go test ./cmd/ateapi/internal/controlapi/...
Should found it when writing #474..
And also correct an error in `workflow_resume.go`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Instead of requiring callers to specify a fixed deadline for the lock,
let the store return a `Lock` object that performs auto-renewal in
background.
This fixes an issue when a workflow takes longer than the hard-coded
TTL, most commonly when atelet takes long times to initialize an actor
(e.g. pulling / untaring large OCI images).
Leases are taken for 30 seconds, but renewed every 10 seconds (TTL/3).
Removing that implicit TTL-based limit means a caller that sets no
deadline of its own (e.g. kubectl-ate) could otherwise hang a
Resume/Suspend/Pause indefinitely. MaxDeadlineUnaryInterceptor adds a
10-minute ceiling on every RPC as a hang-prevention backstop.
Part of the preferred split of #433, please check that out if you need
wider context.
This PR adds `ate.workerpool.workers` metric end to end so we have
coverage for workers by pool / state / sandbox class, emitted by ateapi.
Comes with the few attribute keys it actually uses, unit tests, and an
e2e suite that drives an actor lifecycle and checks the metric (plus the
two existing platform metrics) actually lands in the collector.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This will free up the worker to which the crashed actor was assigned, so
it can be picked up by other actors.
Minor fix:
* Added functional test to make sure a crashed actor can be deleted.
Fixed flakey integration test at head.
Broken test in
https://github.com/agent-substrate/substrate/actions/runs/29795626928/job/88526261507
was because, once a old version of worker has been picked and saved in
state.worker, retrying no longer picked a new, usable worker.
Updated the implementation to only update state.worker after having
successfully picked and persisted the worker.
Fixes#369
Add a CheckPrerequisite method to the WorkflowStep interface, called by
RunWorkflow after IsComplete returns false and before Execute, so that
each workflow validates its actor state-machine edge up front while
retried (reentrant) workflows still fast-forward past completed steps.
When the worker pod disappeared before Checkpoint was called, the
suspend workflow skipped the atelet call but still let finalize promote
the reserved InProgressSnapshot URI to LatestSnapshotInfo, overwriting
the last good snapshot with one that was never written. This could break
the next resume call of this actor.
Now the dangling-worker path crashes the actor via `crashActor` too, and
crashActor clears any reserved snapshot URI so `FinalizeSuspendedStep`
won't overwrite the last good snapshot.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
* Add `args` field in addition to `command`.
* Implement logic in atelet to resolve both argv from the actor template
fields and the OCI image config (`entrypoint` and `cmd` fields).
* Update demos to not hardcode command as now we honor the image
default, which makes it not necessary.
Fixes#456
This PR sets consistent, namespaced actor-identity set on the spans we
already emit:
- `ate.atespace`, `ate.actor.id`, `ate.actor.template.{name,namespace}`,
`ate.actor.version`
- ateapi: the RPC server span for create/resume/suspend/pause/delete
We do this, so platform traces are queryable by actor, atespace, and
template. Cross-hop correlation and per-tenant/per-template filtering,
all on traces (not TSDB labels, as discussed in #174). This is a
general, workload-neutral telemetry identity plumbing. We could add
gen_ai specific stuff on top of this later.
This is a resume:
<img width="1913" height="1010" alt="image"
src="https://github.com/user-attachments/assets/876cf689-a8ec-4f37-b035-3b9b1de87d59"
/>
**Notes:**
- Left as it was on purpose: metric attribute names and the stdout log
labels (`ate.dev/*`), but we should probably do this as well. Maybe in
this PR?
- Bikeshed is welcomed on the `ate.*` spelling before merge, renaming
span attrs later is painful.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
When the worker pod is gone from the DB during pause finalization,
nodeName stays empty.
The code still wrote NodeVmsWithLocalSnapshots: []string{""}, so on the
next ResumeActor the scheduler looked for a worker with node name "",
found none, and returned "no free workers available" permanently.
Fix: only set NodeVmsWithLocalSnapshots when nodeName is non-empty.
Aligns the internal ateletpb/ateompb services with the API style guide's
actor naming (field tags unchanged, so the wire format is compatible).
internal APIs keep dedicated identity fields (not the public ObjectRef).
Replace actor_id and atespace name with `metadata` field which
holds common fields.
This commit only changes the ate-apiserver to support this field, but
intentionally skips upstream / downstream systems (such as kubectl-ate,
atenet, atelet, etc) to keep the scope manageable. This will come in subsequent PRs.
Decouple the actor resumption scheduler from Kubernetes by avoiding
direct queries to WorkerPool CRDs. Instead, cache the worker pool's
labels and sandbox class on the Worker object in Redis during pod
synchronization, and update the scheduler to evaluate worker eligibility
using these cached fields.
Related to #292 and #119
Changes:
1. ResumeActor:
1. During `ResumeActor` we'll put the actor in CRASHED state if the
Resume failed due to any errors related to the snapshot itself, which
includes:
1. Missing or corrupted snapshot when downloading from GCS
2. Runsc command fails.
2. All other types of errors will not put the Actor in CRASHED state.
2. SuspendActor or PauseActor
1. During `SuspendActor` or `PauseActor` we'll put the actor in CRASHED
state if we cannot produce a valid snapshot, which includes:
1. cannot connect to Ateom after retries,
2. runsc command fails.
3. cannot upload the snapshot to GCS (during suspend) ; or cannot save
snapshot locally (during pause).
2. All other types of errors will not put the Actor in CRASHED state.
This fixes an issue where atelet/ateom will clash the state
of actors with same ID within different atespaces.
Once substrate resources get UIDs, we should revisit the way
atelet / ateom does the bookkeeping and check in which cases it
should use UID instead of (atespace,actor_id).
An actor name is only unique within its atespace, so the two always travel together. Add ActorRef{atespace, name} and use it across the Get/Create/Update/Suspend/Pause/Resume/Delete actor requests in place of separate actor_id + atespace fields. The stored Actor object and Redis key layout are unchanged.
Implement a layer allowing to take durableDir snapshot only, without
taking snapshot of entire memory. This is a part of the #119 feature.
The "durableDir" support is introduced via adding Volumes support to
ActorTemplate.
## Knowns issues / limitations
1. No MicroVM support for homedir yet. Will be implemented in following
PR.
2. Resume from SNAPSHOT_SCOPE_DATA temporary behavior
Current behavior of resume from durableStorage returns immediately after
gVisor process is started, it means container's bootstrap is not
completed by the time resume API returned to the caller. It causes the
issue that first curl request for suspended actor throws an HTTP
exception, since webserver listening on the port inside container is not
running yet.
PR #330 fixes the issue, for a meantime the workaround is to call `
kubectl ate resume actor` and after it call to the curl itself.
- [X ] Unit tests pass
- [X ] E2E tests are updated and pass
Extends ActorTemplate.Container spec to allow defining a readyz probe.
* ateoms (both gVisor & microVM) are updated and wait for readyz signal
if it was enabled on container when creating a new container or resume
from a snapshot.
* actorTemplates controller has been updated not to wait 20 sec during
golden snapshot creation, if readyz is enabled on actor. It assumes,
atelet call returns when all containers passed readyz.
Tested manually both counters:
* gVisor
* microvm
Fixes#316
- [x] e2e tests pass
- [X] Appropriate changes to documentation are included in the PR
If WorkerEvents are missed, the local cache can drift. To recover
from this, the cache periodically executes a full `ListWorkers`.
Set a relist period of 5 minutes for now. There is no much thought
behind this number though.
Remove the need for scanning all workers from ValKey during `ResumeActor`.
Extend store.Interface with a "watcher" interface so that clients
can keep track of workers in memory and do cache invalidation.
For ValKey, the watcher is implemented by publishing events to
a ValKey channel. Clients subscribe to changes using SUBSCRIBE
and receive them via the WorkerWatch channel.
Big UX improvement for iterating on #123
Follow-up to https://github.com/agent-substrate/substrate/pull/270
(previously worker pools were directly referenced by actors so you just
made sure to pick an appropriate workerpool, we discussed there)
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This change removes the hard-link from templates to worker pools,
allowing pools to be selected using k8s labels.
Both actors and actor templates are augmented with a label selector
field.
When an actor is resumed, we resolve the set of *eligible* worker pools
by evaluating the intersection of both actor and template selectors.
Then we find a free worker from one of them. The overall design is
described in [1].
[1]:
https://github.com/agent-substrate/substrate/issues/47#issuecomment-4712334620
Troy Chiu