Best practices is not to put a -deployment deployment, -role role, etc;
the kind already declares the kind we don't need it in the name as well.
Additionally we did not do it consistently anyways.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Related to #292 and #119
Changes:
1. ResumeActor:
1. During `ResumeActor` we'll put the actor in CRASHED state if the
Resume failed due to any errors related to the snapshot itself, which
includes:
1. Missing or corrupted snapshot when downloading from GCS
2. Runsc command fails.
2. All other types of errors will not put the Actor in CRASHED state.
2. SuspendActor or PauseActor
1. During `SuspendActor` or `PauseActor` we'll put the actor in CRASHED
state if we cannot produce a valid snapshot, which includes:
1. cannot connect to Ateom after retries,
2. runsc command fails.
3. cannot upload the snapshot to GCS (during suspend) ; or cannot save
snapshot locally (during pause).
2. All other types of errors will not put the Actor in CRASHED state.
## Description
Closesagent-substrate/substrate#222.
Adds a portable JWT authentication path for ateapi clients while keeping
the
existing mTLS mode as the default.
### What changed
- Added `internal/ateapiauth` with:
- `mtls` and `jwt` auth modes
- server-side JWT gRPC interceptors
- client-side dial options for projected ServiceAccount tokens
- Wired JWT auth into:
- `ate-api-server`
- `ate-controller`
- `atenet router`
- `kubectl-ate` port-forward client path
- Updated Kubernetes JWT verification to support custom HTTP clients for
OIDC/JWKS discovery.
- Added JWT install overlays:
- `manifests/ate-install/jwt`
- `manifests/ate-install/kind-jwt`
- Updated `hack/install-ate.sh` to opt into JWT mode with:
- `ATE_API_AUTH_MODE=jwt`
- `--auth-mode=jwt`
### Notes
- Default install behavior remains `mtls`. I think this deserves a
second look as JWT will support more clusters.
- The JWT overlay projects short-lived ServiceAccount tokens with
audience
`api.ate-system.svc` and mounts the service-DNS trust bundle for ateapi
TLS
verification.
- The current discovery mechanism for JWT mode involves reading the
deployment which I really don't like, but we don't have a "config file"
concept so that bit is a massive TODO.
### Validation
```bash
KIND_CLUSTER_NAME=substrate-jwt ./hack/create-kind-cluster.sh
KO_DOCKER_REPO=localhost:5001 \
ATE_INSTALL_KIND=true \
./hack/install-ate.sh --auth-mode=jwt --deploy-ate-system
Validation results:
kubectl config current-context
# kind-substrate-jwt
kubectl get pods -n ate-system
# all runtime pods Running; init jobs Completed
go run ./cmd/kubectl-ate --context kind-substrate-jwt get actors
# succeeded, empty actor list
```
Golden actors are system actors, not user actors; don't put them in an atespace named after the template's k8s namespace (a k8s namespace is not an atespace). The "ate-" prefix is reserved for system atespaces.
An actor name is only unique within its atespace, so the two always travel together. Add ActorRef{atespace, name} and use it across the Get/Create/Update/Suspend/Pause/Resume/Delete actor requests in place of separate actor_id + atespace fields. The stored Actor object and Redis key layout are unchanged.
Extends ActorTemplate.Container spec to allow defining a readyz probe.
* ateoms (both gVisor & microVM) are updated and wait for readyz signal
if it was enabled on container when creating a new container or resume
from a snapshot.
* actorTemplates controller has been updated not to wait 20 sec during
golden snapshot creation, if readyz is enabled on actor. It assumes,
atelet call returns when all containers passed readyz.
Tested manually both counters:
* gVisor
* microvm
Fixes#316
- [x] e2e tests pass
- [X] Appropriate changes to documentation are included in the PR
- workerpool_apply: rename applyMicroVMPodShape -> maybeApplyMicroVMPodShape and
pass wp.Spec.SandboxClass instead of the whole WorkerPool.
- sandboxconfig validation test: add an arm64 micro-VM asset-set case.
- specconv: TODO to forward Seccomp/Sysctl + Apparmor/SELinux for OCI parity.
- run.go: clarify the dialAgentRetry per-attempt cap vs retry-gap comment.
- roadmap: drop the microVM line (shipped).
WorkerPool renders the micro-VM worker pod shape, and the SandboxConfig
validating admission policy enforces the micro-VM asset set (cloud-hypervisor,
guest kernel, guest image, kata config).
**Updated Original PR Description**
## Summary
This PR adds a small, controlled set of scheduling fields to WorkerPool
so WorkerPool-managed Pods can be placed onto appropriate Kubernetes
nodes.
Added fields:
- `nodeSelector`
- `tolerations`
- `priorityClassName`
- `nodeAffinity`
## Scope
This intentionally does not expose a full `PodTemplateSpec`, and does
not add support for full `affinity`, `podAffinity`, or
`podAntiAffinity`.
Resource requests/limits are intentionally left out of scope for this CL
while the WorkerPool resource model is still being discussed separately.
The goal is to keep the first version small while addressing the
WorkerPool-to-Kubernetes-Node scheduling gap discussed in
[#212](https://github.com/agent-substrate/substrate/issues/212).
`nodeAffinity` is included to support heterogeneous node pools where
`nodeSelector` is too limited, such as requiring or preferring one of
several equivalent accelerator/local-SSD/cache node pools, or expressing
soft preferences during node pool migration.
`priorityClassName` is included to help preserve warm WorkerPool
capacity under cluster pressure. Since WorkerPool pods are the slots
available for actor resume, higher-priority pools can protect
latency-sensitive or interactive actor workloads from displacement by
lower-priority batch workloads, and allow different WorkerPools to
represent different service classes, e.g. interactive vs. batch.
## Related issue
Part of #212.
Related to #47.
## Problem
`go test -v ./...` (the `run-tests` job) intermittently fails during
envtest **setup** — not in any test logic:
- `fork/exec .../kubebuilder-envtest/k8s/1.36.0-linux-amd64/etcd: text
file busy`
- `setup-envtest: ... unable to create file kubectl from archive for
version-platform pair 1.36.0-linux-amd64`
Three packages use envtest (`cmd/ateapi/internal/controlapi`,
`cmd/atecontroller/internal/controllers`, `pkg/api/v1alpha1`). `go test
./...` runs them as separate test binaries **in parallel**, and each
`TestMain` shells out to `setup-envtest use`. On a cold cache they race
to download, extract, and exec the **shared** kubebuilder binaries —
hence `text file busy` (exec'ing etcd while it's still being written)
and the archive-extraction error.
Seen on `main` (e.g. the #259 push run) and on PRs. It only reproduces
on a **cold** cache, which is why it's intermittent and doesn't show up
locally once the cache is warm.
## Fix
Add `internal/envtestbins.BinaryAssetsDir()`, which runs `setup-envtest
use` under a **cross-process file lock** (`flock` on a fixed file under
`os.TempDir()`, shared by the sibling test binaries). The first caller
populates the shared cache while the others wait, then they hit a warm
no-op path lookup. The duplicated setup in all three `TestMain`s now
calls this helper.
The lock scope (user/machine-global) matches the resource it guards —
the kubebuilder binary cache is itself user-global, shared even across
repo checkouts.
## Testing
Reproduced locally by moving the warm cache aside and running the three
packages concurrently: it fails without the lock and passes with it.
Full `go test ./...` green.
Test-only change; no production code touched.
Implements `PAUSED` state for issue #119.
The snapshot files are kept locally on node VM in a separate folder. At
resume time, scheduler uses a node VM hint and picks up a worker from
the same node where files were stored at suspend time.
The local file management solution is temporary and will be replaced
once @msau42 introduces a new component that is supposed to manage files
on the node VM.
- [X] Tests pass
- [X] Manual tests with counter demo
```
>kubectl ate create actor my-counter-1 --template ate-demo-counter/counter
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 1
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 2
>kubectl ate pause actor my-counter-1 -o json
> curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 3
```
### Breaking change
This PR introduces a breaking change in the Actor proto. All existing
actor needs to be recreated, prior testing PAUSE functionality.
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
John Howard