Fixes#564 (Part 1)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements the first part of #564 by extending metric
`atenet.router.route.duration` with singleflight activation state
(`ate.router.resume`) and outcome status labels (`ate.router.outcome`).
All telemetry attribute keys and label value sets are centralized in
`internal/ateattr` to ensure consistency across components.
## Key Changes:
* Added `resumed` boolean to `ResumeActorResponse` in `ateapi.proto`
indicating whether a cold activation workflow occurred
(`!state.WasRunning`).
* Updated `workflow_resume_test.go` to assert `resumed == true` on cold
activations and `resumed == false` for already-running actors.
* Extended `classifyOutcome(err)` in Envoy `ext_proc` to classify errors
into metric labels: `ok`, `cancelled`, `timeout`, `no_capacity`,
`lock_conflict`, `not_found`, `unavailable`, `rate_limited`, and
`resume_error`.
* Added explicit handling for `codes.Unavailable` (`"unavailable"`) and
`codes.ResourceExhausted` / `StatusCode_TooManyRequests`
(`"rate_limited"`).
* Consolidated metric label names on `atenet.router.route.duration`
using standard keys from `internal/ateattr`:
* `ate.template.namespace` (`TemplateNamespaceKey`)
* `ate.template.name` (`TemplateNameKey`)
* `ate.router.outcome` (`RouterOutcomeKey`)
* `ate.router.resume` (`RouterResumeKey`)
* Updated `manifests/ate-install/kind/kustomization.yaml` to export
`atenet-router` OTLP metrics to `opentelemetry-collector`.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
Part of #232
1. During CreateActor, the volumes to be created get instantiated in the
Actor API in PENDING status. The Actor is still instantiated as
SUSPENDED.
However, the CreateVolume operation is moved to the beginning of
ResumeActor. This is because CreateActor is intentionally designed to
not be idempotent. The downside is that the user won't know dependent
resource creation failed until the first resume.
2. Changes volume id to use actor uid instead of atespace+actorname and
adds "substrate" prefix
3. Adds a new DELETING status that gets persisted before we start to
delete volumes.
This shifts the existing DB DELETE precondition checks to the new
DELETING status. And now you can only delete the actor in DB from
DELETING status.
4. Refactor DeleteActor to use a workflow
In the future, we will also add a workflow that will do mandatory node
cleanup of volumes before starting to delete the volumes.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The pattern of passing a tuple of atespace and actor name together
is across the whole codebase. This simplifies both callers and callees
by leting them pass a single field that bundles both.
The ActorRef is the actor specific, typed,
in-process version of the ObjectRef we have in our gRPC API.
The error is only returned on an optimistic-concurrency version
mismatch. Rename it for that behavior so other retryable errors don't
get folded into it (but should introduce their own error name).
Followup for post-merge comment in #535
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Initial part of #232.
Extends the ActorTemplate API to add support for per-actor external
volumes and adds control plane and atelet hooks following the Actor
lifecycle.
Callouts to volume operations are abstracted with a Volume interface.
Right now, only a mock volume plugin for testing has been implemented,
but we will add CSI support next.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR - Not
going to update documentation until we add CSI support.
Instead of requiring callers to specify a fixed deadline for the lock,
let the store return a `Lock` object that performs auto-renewal in
background.
This fixes an issue when a workflow takes longer than the hard-coded
TTL, most commonly when atelet takes long times to initialize an actor
(e.g. pulling / untaring large OCI images).
Leases are taken for 30 seconds, but renewed every 10 seconds (TTL/3).
Removing that implicit TTL-based limit means a caller that sets no
deadline of its own (e.g. kubectl-ate) could otherwise hang a
Resume/Suspend/Pause indefinitely. MaxDeadlineUnaryInterceptor adds a
10-minute ceiling on every RPC as a hang-prevention backstop.
Fixes#369
Add a CheckPrerequisite method to the WorkflowStep interface, called by
RunWorkflow after IsComplete returns false and before Execute, so that
each workflow validates its actor state-machine edge up front while
retried (reentrant) workflows still fast-forward past completed steps.
Replace actor_id and atespace name with `metadata` field which
holds common fields.
This commit only changes the ate-apiserver to support this field, but
intentionally skips upstream / downstream systems (such as kubectl-ate,
atenet, atelet, etc) to keep the scope manageable. This will come in subsequent PRs.
Decouple the actor resumption scheduler from Kubernetes by avoiding
direct queries to WorkerPool CRDs. Instead, cache the worker pool's
labels and sandbox class on the Worker object in Redis during pod
synchronization, and update the scheduler to evaluate worker eligibility
using these cached fields.
Related to #292 and #119
Changes:
1. ResumeActor:
1. During `ResumeActor` we'll put the actor in CRASHED state if the
Resume failed due to any errors related to the snapshot itself, which
includes:
1. Missing or corrupted snapshot when downloading from GCS
2. Runsc command fails.
2. All other types of errors will not put the Actor in CRASHED state.
2. SuspendActor or PauseActor
1. During `SuspendActor` or `PauseActor` we'll put the actor in CRASHED
state if we cannot produce a valid snapshot, which includes:
1. cannot connect to Ateom after retries,
2. runsc command fails.
3. cannot upload the snapshot to GCS (during suspend) ; or cannot save
snapshot locally (during pause).
2. All other types of errors will not put the Actor in CRASHED state.
Remove the need for scanning all workers from ValKey during `ResumeActor`.
Extend store.Interface with a "watcher" interface so that clients
can keep track of workers in memory and do cache invalidation.
For ValKey, the watcher is implemented by publishing events to
a ValKey channel. Clients subscribe to changes using SUBSCRIBE
and receive them via the WorkerWatch channel.
This change removes the hard-link from templates to worker pools,
allowing pools to be selected using k8s labels.
Both actors and actor templates are augmented with a label selector
field.
When an actor is resumed, we resolve the set of *eligible* worker pools
by evaluating the intersection of both actor and template selectors.
Then we find a free worker from one of them. The overall design is
described in [1].
[1]:
https://github.com/agent-substrate/substrate/issues/47#issuecomment-4712334620
Implements `PAUSED` state for issue #119.
The snapshot files are kept locally on node VM in a separate folder. At
resume time, scheduler uses a node VM hint and picks up a worker from
the same node where files were stored at suspend time.
The local file management solution is temporary and will be replaced
once @msau42 introduces a new component that is supposed to manage files
on the node VM.
- [X] Tests pass
- [X] Manual tests with counter demo
```
>kubectl ate create actor my-counter-1 --template ate-demo-counter/counter
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 1
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 2
>kubectl ate pause actor my-counter-1 -o json
> curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 3
```
### Breaking change
This PR introduces a breaking change in the Actor proto. All existing
actor needs to be recreated, prior testing PAUSE functionality.
A couple of TODOs/Discussions which I think are important to have based
on these changes.
1. Do we need a caching client for this to avoid calling the apiserver
every time?
2. Where/how do we want to resolve secrets generally? I think there are
good arguments to be made for the control-plane OR the "node" component.
In my mind this also related to:
https://github.com/agent-substrate/substrate/issues/18Fixes#15
> It's a good idea to open an issue first for discussion.
- [ x ] Tests pass
- [ x ] Appropriate changes to documentation are included in the PR
Adds golangci-lint v2.12.2 using the per-tool-module pattern under
hack/tools/golangci-lint/. The verifier at hack/verify/golangci-lint.sh
is auto-discovered by hack/verify-all.sh, so the existing CI verify
step picks it up with no workflow YAML change.
Ships .golangci.yaml with the standard linter set, a make lint
target, and a verify wrapper that pins GOOS=linux so ateom-gvisor's
netlink references typecheck on macOS the same way they do on CI.
Adjustments to keep the first run clean: two dead-code deletions
((*runsc).cmdList, MemoryPullCache.gcsClient), two SA9003 empty
branches replaced with slog.Error / t.Logf, one S1008 return-bool
simplification, and inline //nolint for one intentional best-effort
ReleaseLock plus four audit-tracked SA1019 deprecations.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
Angela