Part of #564 (Part 4, Fixes#564)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements Part 4 of #564 by adding OpenTelemetry gauge
instrumentation for `ate.workerpool.desired_workers` and
`ate.workerpool.ready_workers` in `atecontroller`.
## Key Changes:
* Added `ReadyReplicas int32` to `WorkerPoolStatus` with
`+kubebuilder:printcolumn:name="Ready"` annotation in
`pkg/api/v1alpha1/workerpool_types.go` and regenerated the CRD manifest.
* Updated `WorkerPoolReconciler.syncStatus` to synchronize
`dep.Status.ReadyReplicas` into `wp.Status.ReadyReplicas`.
* Implemented `InitMetrics(meter)` registering
`ate.workerpool.desired_workers` and `ate.workerpool.ready_workers` as
OpenTelemetry Observable UpDownCounters (`{worker}`) with an
asynchronous observer callback that samples controller-runtime's local
informer cache (`r.Client.List`).
* Labeled metric datapoints using centralized attributes
`ateattr.WorkerPoolNamespaceKey` and `ateattr.WorkerPoolNameKey`.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
---------
Co-authored-by: Jeff Luo <jeffluoo@google.com>
`setup-envtest use` with no version re-resolves "latest" against the
remote release index on every run, so the envtest-backed packages needed
network even with fully cached binaries.
This PR pins the control plane to `1.36.x`, and fold the three
per-package `envtest` into `internal/testenv.Start`, which also honors
`-short`: `go test -short ./...` skips the envtest-backed packages and
is the guaranteed-offline path.
Have Claude to help me fix it
Fixes#789
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Every worker pod now hosts an atunnel ingress server on :443 and an
atunnel egress listener, both long-lived, with per-activation
Activate/Deactivate bracketing every Run, Restore and Checkpoint.
The nftables rules ateom installs change accordingly:
* The pod-IP:80 -> actor-veth:80 DNAT is gone. Worker port 80 is no
longer an Actor ingress path; the only way in is the mTLS listener
on :443, which authorizes the caller and checks the Actor is the one
currently assigned to this worker.
* A prerouting REDIRECT sends Actor TCP egress to the local atunnel
egress listener, preserving SO_ORIGINAL_DST. It is only installed
when the activation carries an egress gateway address, which nothing
populates yet, so the masquerade path is unchanged and Actor egress
behaves exactly as before.
The ateom Run/Restore protos gain the two fields that arm that path:
egress_gateway_address, which decides whether the redirect is installed
at all, and actor_version, the Actor resource version ate-api observed
when assigning the worker, which atunnel asserts to the egress gateway
as a lower bound on trustworthy Actor metadata. Both are consumed here
and left unset. atelet and ate-api start populating them in the egress
gateway change, which is the point at which they mean anything, so this
change adds no new requirement to the atelet wire contract.
atecontroller gives worker pods the podidentity credential + trust
bundles (the atunnel server identity), the servicedns trust bundle, and
container port 443. podidentitysigner now issues certs with
ExtKeyUsageServerAuth as well as ClientAuth, without which the worker
cannot present its podidentity cert as a TLS server cert and the
gateway handshake fails.
- Add ate.dev/worker-pool label to NetworkPolicy metadata and remove Egress management to allow default egress behavior
- Define ateSystemNamespace and atenetRouterAppName constants for Ingress peer selectors
- Centralize deterministic NetworkPolicy naming logic (resources.NetworkPolicyName) and prevent trailing hyphens
- Add comprehensive E2E test suite for networkpolicy covering control plane reconciliation, positive data plane verification (authorized ingress via atenet-router), and negative data plane verification (unauthorized ingress blocking)
Restrict Ingress to Workers and hence Actors to be only
from the Router component.
This makes the atecontroller responsible for K8s level network policy
Note: this is a belts-and-suspenders model as this applies on the Worker
level. This will be refined further with Actor-specific policies.
WorkerPool already served /scale, so `kubectl scale` worked, but a HPA
pointed at one did not: the subresource declared no labelSelectorPath,
and HPA needs a selector to find the pods whose metrics it is averaging
or to compute ambiguous selectors.
https://github.com/agent-substrate/substrate/issues/198
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Best practices is not to put a -deployment deployment, -role role, etc;
the kind already declares the kind we don't need it in the name as well.
Additionally we did not do it consistently anyways.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Related to #292 and #119
Changes:
1. ResumeActor:
1. During `ResumeActor` we'll put the actor in CRASHED state if the
Resume failed due to any errors related to the snapshot itself, which
includes:
1. Missing or corrupted snapshot when downloading from GCS
2. Runsc command fails.
2. All other types of errors will not put the Actor in CRASHED state.
2. SuspendActor or PauseActor
1. During `SuspendActor` or `PauseActor` we'll put the actor in CRASHED
state if we cannot produce a valid snapshot, which includes:
1. cannot connect to Ateom after retries,
2. runsc command fails.
3. cannot upload the snapshot to GCS (during suspend) ; or cannot save
snapshot locally (during pause).
2. All other types of errors will not put the Actor in CRASHED state.
**Updated Original PR Description**
## Summary
This PR adds a small, controlled set of scheduling fields to WorkerPool
so WorkerPool-managed Pods can be placed onto appropriate Kubernetes
nodes.
Added fields:
- `nodeSelector`
- `tolerations`
- `priorityClassName`
- `nodeAffinity`
## Scope
This intentionally does not expose a full `PodTemplateSpec`, and does
not add support for full `affinity`, `podAffinity`, or
`podAntiAffinity`.
Resource requests/limits are intentionally left out of scope for this CL
while the WorkerPool resource model is still being discussed separately.
The goal is to keep the first version small while addressing the
WorkerPool-to-Kubernetes-Node scheduling gap discussed in
[#212](https://github.com/agent-substrate/substrate/issues/212).
`nodeAffinity` is included to support heterogeneous node pools where
`nodeSelector` is too limited, such as requiring or preferring one of
several equivalent accelerator/local-SSD/cache node pools, or expressing
soft preferences during node pool migration.
`priorityClassName` is included to help preserve warm WorkerPool
capacity under cluster pressure. Since WorkerPool pods are the slots
available for actor resume, higher-priority pools can protect
latency-sensitive or interactive actor workloads from displacement by
lower-priority batch workloads, and allow different WorkerPools to
represent different service classes, e.g. interactive vs. batch.
## Related issue
Part of #212.
Related to #47.
## Problem
`go test -v ./...` (the `run-tests` job) intermittently fails during
envtest **setup** — not in any test logic:
- `fork/exec .../kubebuilder-envtest/k8s/1.36.0-linux-amd64/etcd: text
file busy`
- `setup-envtest: ... unable to create file kubectl from archive for
version-platform pair 1.36.0-linux-amd64`
Three packages use envtest (`cmd/ateapi/internal/controlapi`,
`cmd/atecontroller/internal/controllers`, `pkg/api/v1alpha1`). `go test
./...` runs them as separate test binaries **in parallel**, and each
`TestMain` shells out to `setup-envtest use`. On a cold cache they race
to download, extract, and exec the **shared** kubebuilder binaries —
hence `text file busy` (exec'ing etcd while it's still being written)
and the archive-extraction error.
Seen on `main` (e.g. the #259 push run) and on PRs. It only reproduces
on a **cold** cache, which is why it's intermittent and doesn't show up
locally once the cache is warm.
## Fix
Add `internal/envtestbins.BinaryAssetsDir()`, which runs `setup-envtest
use` under a **cross-process file lock** (`flock` on a fixed file under
`os.TempDir()`, shared by the sibling test binaries). The first caller
populates the shared cache while the others wait, then they hit a warm
no-op path lookup. The duplicated setup in all three `TestMain`s now
calls this helper.
The lock scope (user/machine-global) matches the resource it guards —
the kubebuilder binary cache is itself user-global, shared even across
repo checkouts.
## Testing
Reproduced locally by moving the warm cache aside and running the three
packages concurrently: it fails without the lock and passes with it.
Full `go test ./...` green.
Test-only change; no production code touched.
