Pick up the latest kind release for local dev clusters.
Includes fix relevant to
https://github.com/agent-substrate/substrate/issues/1133 (kindnetd
memory)
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This commit fleshes out localca's rotation support by changing the
interface so that signing operations should always go through a pool.
All existing in-tree callers are converted over to this pattern. This
ensures that signing can properly continue as a CA pool is rotated,
without requiring any process restarts.
The pool is periodically reloaded from disk in the course of signing
operations, with the loaded data being cached for up to a minute.
Some code for storing the CA state using PEM files was removed because
it seemed to be dead.
Follow-on changes will give localjwt a similar treatment, and add
administrative commands for running rotations on pools.
The image reference is pinned by digest, so the tag is discarded on
pull. Building it from ${WEAVER_VERSION} suggested that bumping that one
variable moved the image, when a new digest is needed as well.
Write the tag literally instead, next to the digest, as
`hack/third_party/kubernetes/verify-shellcheck.sh` does.
A follow-up of
https://github.com/agent-substrate/substrate/pull/1097#discussion_r3857057767
## Summary
This change updates the boilerplate verification to accept either of the
approved copyright attribution forms:
- `Copyright YYYY Google LLC`
- `Copyright YYYY The Agent Substrate Authors`
## Why
The repo now allows either owner attribution, and the boilerplate
checker should enforce that both valid forms pass without rejecting the
other.
I retained the Google LLC so that we didn't have to re-copyright all the
existing files.
## Verification
- Ran the focused matcher assertions for both allowed strings and an
invalid holder.
- Ran `hack/verify/boilerplate.sh` successfully.
This mostly eliminates the need to hand-write validation code.
This PR is a long series of commits which add DV for most of Actor and
all of Atespace.
Here is a map to the commits:
* The first few take a dep on a new Kubernetes tag, import the code into
third_party, and apply a single patch. Because we have different Go
modules for tools, I had to do it twice. When that patch lands, we can
revert these commits, but that won't be until the 1.38 cycle in a few
months.
* The next commits slowly add DV support, so a human can review each of
them in a reasonable amount of time. The emphasis is on great test
cases.
* I made a bad choice early on as to where to generate the code into, so
I moved it. Rebasing on that was exceedingly hard, so I left it as a
move.
* This required changing update/go-generate -> update/codegen -- we need
to get the ordering of tools right, which `go generate` does not
guarantee.
* Then I added a "middle" layer called "ServiceImpl" between the RPC and
storage layers. This allows things like workflow to call the same
business logic as the RPC layer, including validation. Lots of test
fixes.
* Then I finished the Create() and Update() paths for Actor. Those
represent the "right" (or closest to) way to implement resources, and
tests for validation.
I strongly encourage reviewers to read it commit-by-commit. Rebasing
this is VERY tedious, so the sooner it lands or dies completely, the
better. Then we can start converting the rest.
`./hack/run-tool.sh validation-gen --docs` will produce some docs on the
tool and the available tags.
@laoj2 @juli4n @EItanya @HavenXia
@lalitc375 @yongruilin @jpbetz FYI
`docs/observability.md` was the only description of the metrics of
Substrate, and nothing compared that text with the code. It listed 13
instruments; the code sends 21. The request-parking and actor
resource-usage instruments were
missing entirely.
This change adds an [OpenTelemetry
Weaver](https://github.com/open-telemetry/weaver) semantic convention
registry.
### What is in it
| Path | Content |
|---|---|
| `docs/metrics/registry/manifest.yaml` | The Weaver manifest: name and
schema URL. |
| `docs/metrics/registry/metrics.yaml` | 21 instruments and 12 attribute
groups: every label, its permitted values, and its buckets. |
| `docs/metrics/substrate.yaml` | The rules Weaver cannot express. |
| `hack/verify/verify-metrics.sh` | The check. |
Fixes#1093
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
install-microvm-deps.sh decided whether to run assemble.sh by testing
only whether the five asset filenames existed in $OUT. Those names don't
change when a version pin moves, so an asset dir built before a bump
looks complete while holding the old bytes.
## Summary
A follow up to #640 where we introduced PostgreSQL as an alternative
storage backend, selected conditionally in ateapi.
- Deleted ateredis, its tests, and its dependencies
- Removed Redis backend selection and configuration so ateapi always
connects to Postgres
- Replaced Valkey resources with Postgres in the standard and Kind
deployment paths and simplified install script
- Replaced miniredis fixtures with isolated Postgres testcontainers and
added centralized helpers for seeding resources
- Renamed Redis-specific debug flush command to backend-neutral
`debug-clear-store` in CLI
- Updated comments and docs where applicable
## Benchmarking
Extensive benchmarking have been performed to evaluate Redis vs
Postgres, and results can be found in these two documents:
-
https://docs.google.com/document/d/10K0wB6aTeFkJCL4HN3NbLJCdFGoLYdhIcqkFnqFHkKc/edit?usp=sharing
-
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Secures network-based gRPC connections between the ate-api control plane
and CSI driver controller services using mutual TLS (mTLS) backed by
Substrate's Pod Identity.
Requests from ate-api to CSI driver controllers occur over the network,
as configured via the CSIDriverConfig custom resource. This PR extends
the CSIDriverConfig CRD to include a tls configuration section, which
leverages
Substrate's Pod Identity certificate infrastructure (managed by
podcertcontroller) to secure these
TCP connections using mTLS. Because standard CSI drivers typically
communicate over Unix domain sockets, a proxy sidecar (e.g., envoy) is
added to the CSI controller deployment to terminate TLS and forward
requests to the local CSI socket.
#232
> It's a good idea to open an issue first for discussion.
- [X ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Run the networking suite a second time against the sdsmint (MITM)
egress gateway, so the egress path is covered on both gateway variants
rather than only on passthrough.
A part of #823
From
https://github.com/agent-substrate/substrate/actions/runs/32786529664/job/97619491954?pr=1155,
this change adds about 70 seconds to the `e2e-test` workflow:
- Step `Deploy MITM egress demo`: 8s
- Step `Run E2E tests (networking, MITM egress)`: 33s
- Step `Run E2E tests (networking, MITM egress, micro-VM)`: 30s
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This reverts commit 57c3e1927c.
Drops --experimental-additional-egress-extproc-service and the
#ATE_MITM_EXTPROC_FILTER / #ATE_MITM_EXTPROC_CLUSTER markers, so
atenet-egress-with-sdsmint.yaml renders exactly as committed again
and hack/experimental-additional-egress-extproc.sh goes away.
`--experimental-additional-egress-extproc-service=NS/SVC:PORT` makes the
sdsmint egress gateway consult an external authorization processor on
the decrypted leg, where the request is a hostname, method, and path
rather than the IP:port the CONNECT checkpoint sees. The actor's
identity crosses into that callout as the `dev.ate.actor` filter state,
so the processor gets both halves of the decision.
The wiring is generated rather than checked in:
atenet-egress-with-sdsmint.yaml carries inert `#ATE_MITM_EXTPROC_FILTER`
and `#ATE_MITM_EXTPROC_CLUSTER` markers that the installer substitutes
only when the flag is set, so an install without it renders exactly the
manifest as committed.
The flag names any Service, so the processor itself is out of scope
here.
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This allows for faster signing of certs for clusters with large numbers
of atelets.
It also allows configurable ready wait times to compensate for larger
reconciliation waits generally.
Removes the non-functional token/JWT mode for in-cluster ateapi clients.
Clients now always use mTLS certificates; related flags, install and
benchmark plumbing, tests, and overlays are deleted.
Validated with focused Go tests, shellcheck, and Kustomize renders.
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
#### benchmarking: right-size actor memory, default 256Mi
Benchmark `ActorTemplates` declared no resources, so microvm actors fell
through to the 2 GiB kata default guest — and everything the guest
kernel caches rides along in the memory snapshot, inflating snapshot
size and suspend/resume latency, which the benchmarks then measure.
Set `spec.resources.limits.memory` on the `glutton` and `sleep`
templates, parameterized as `ACTOR_MEMORY` with a `256Mi` default (the
smallest size microvm admits: 128Mi VMM reserve + 128Mi guest floor),
and thread it through the deploy chain:
- `--actor-memory` on `workloads/deploy.sh` and `deploy_locust.sh`
- `--benchmark-actor-memory` on `install-ate.sh`
- Optional per-test `actorMemory` field in the automation's `tests.yaml`
for future `WriteRAM` stress suites.
cc @bowei yeah ... we need to get these rewritten
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Envoy now terminates the TLS session inside an actor's CONNECT tunnel
with a leaf minted on demand for the SNI the client asked for, instead
of forwarding the tunneled bytes opaquely. That puts the plaintext
request on a filter chain where policy can be applied later. A new
atenet sdsmint subcommand serves those leaves over DELTA_GRPC SDS on a
unix socket.
Opt-in: --experimental-use-sdsmint selects atenet-egress-with-sdsmint.yaml
and creates the egress-mitm-ca-pool secret. The default egress manifest
and install path are unchanged.
Also: localca gains CA.Validate, GenerateCA options and a crypto.Signer
SigningKey; kubectl-ate admin make-ca-pool gains --key-type and
--common-name; atunnel returns ErrGatewayHandshake and
ConnectRejectedError instead of formatted strings. The sdsmint e2e suite
is skipped for now.
Part of #563
Substrate had no way to measure how much telemetry it sends. This PR
adds one, and builds it from the parts the benchmark suite already has.
The managed collector sends its self-metrics to Cloud Monitoring only,
and gives no data for each source. Thus you cannot see which component
makes the volume. A second collector, the telemetry meter, counts the
spans and the datapoints by `service.name`.
## Contents
- **`benchmarking/telemetry/meter.yaml`** — the meter. A tee: it counts
the telemetry by service, then sends it on to the managed collector.
Thus one run gives the volume for each service and the cost of the load
together.
- **`benchmarking/telemetry/README.md`** — the method: install, measure,
read, remove.
- **`hack/install-ate.sh --otlp-endpoint`** — sends all control plane
telemetry to the meter. Each component reads the endpoint from the
`ate-otel-config` ConfigMap through `envFrom`, and `ate-controller`
copies the value to the ateom worker pods, thus one patch is sufficient
for all of them.
- **`benchmarking/workloads/deploy.sh --otlp-endpoint`** — the same for
actor containers, which substrate configures separately.
- **`benchmarking/monitoring.yaml`** — scrapes the meter and cAdvisor.
- **`benchmarking/observability.md`** — prerequisites and the scenario
ladder.
- **`docs/dev/best-practices/otel-collector.md`** — the volume model and
the size guidance, corrected for the `ParentBased` defaults from #711.
## Correction
The `glutton` ActorTemplate had no `env`. Substrate puts no OTLP
configuration in an actor container, thus the exporter used the SDK
default `localhost:4317`, no process listens on that address in the
sandbox, and substrate discarded each actor span with no error message.
The template now sets `OTEL_EXPORTER_OTLP_ENDPOINT` and
`OTEL_RESOURCE_ATTRIBUTES`.
`ActorTemplate.spec` is immutable, thus `deploy.sh --deploy` deletes the
templates before it applies them.
## How I verified this
On a GKE cluster with managed OTel. The sequence:
```bash
# 1. Start from a clean state.
./hack/install-ate.sh --delete-benchmarks
./hack/install-ate.sh --delete-ate-system
# 2. Install substrate from this branch.
./hack/install-ate.sh --deploy-ate-system
# 3. Install the meter and Prometheus.
kubectl apply -f benchmarking/telemetry/meter.yaml
kubectl apply -f benchmarking/monitoring.yaml
# 4. Send the control plane to the meter.
METER=http://telemetry-meter.benchmarking.svc.cluster.local:4317
./hack/install-ate.sh --deploy-ate-system --otlp-endpoint "${METER}"
# 5. Install the workloads and locust, with the actors on the meter also.
./benchmarking/deploy_locust.sh --deploy --worker-count 10 \
--otlp-endpoint "${METER}"
# 6. Read the volume.
kubectl port-forward -n benchmarking svc/prometheus 9090:9090
```
## Result
The idle floor, from one measurement on 10 workers:
| Service | datapoints/min |
|---|---|
| `ate-controller` | 234.8 |
| `ateapi` | 54.3 |
| `atelet` | 44.9 |
| `atenet-router` | 10.4 |
| `ateom-gvisor` | 5.2 |
| `glutton-actor` | 4.2 |
| **Total** | **353.7** |
The `glutton-actor` row is the correction above. Before it, actor
telemetry could not arrive at all.
Positive control for the same window:
`otelcol_receiver_accepted_metric_points` 10.5/s, and each
`otelcol_receiver_refused_*` counter at zero. Thus the zero values in
the table are true zeros, and not data that did not arrive.
Two values differ from the earlier manual measurement. `ate-controller`
is now the largest source, and `ateom-gvisor` is no longer zero. The new
imagecache counters from #831 are a probable cause. This difference is
the reason to keep results out of the repository.
### Traces
Trace volume follows the root sample decision, not the rate of each
component. `atelet` and `ateom-gvisor` sent zero spans in each run,
because they inherit the decision that `ateapi` makes and start no trace
of their own. At idle each component is near zero: a trace needs an
operation, but a metric pushes on a timer.
This run gives no span volume for a usual load. The actors crashed on
suspend, and the failed retries made an artificial rate of 13.7k
spans/s. That rate is not a measurement of substrate. It does show that
the meter accepted the spans with no refusals.
## Follow-up
A separate PR adds the ladder to `automation/tests.yaml` and writes the
results as artifacts of each run. The open items are at the end of
`observability.md`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
---------
Co-authored-by: Krisztian F <103492698+krisztianfekete@users.noreply.github.com>
Follow-up to #847
cloud-hypervisor v53 ships guest clock updates:
https://github.com/cloud-hypervisor/cloud-hypervisor/releases#:~:text=Guest%20Clock%20Updates%20Across%20Snapshot/Restore%20and%20Migration
In v52, we've been booting to systemd => {chrony, journald, kata-agent}
We don't really have much use for journald, as we get logs from
containers already.
chrony however was load-bearing to re-sync the clock on resume, and
therefore systemd was load-bearing (to run chronyd + kata-agent).
We don't need chrony anymore (tested with long suspend and probing the
reported time in the actor), so we can boot directly to the agent.
This buys us some startup time, and it drops the memory floor by ~40MB.
On golden for counter demo this is ~-26%.
Reducing memory snapshot size speeds up snapshot/restore.
IP_FAMILY selects ipv4, ipv6 or dual and becomes networking.ipFamily, leaving
kind's per-family subnet defaults alone. The script also recreates a pre-IPv6
"kind" Docker network, fails fast if the daemon has IPv6 off, sets proxy_ndp
alongside proxy_arp for gVisor pod-to-pod traffic, and repoints an ipv6
kubeconfig from [::1] at localhost so a client outside the Docker host can
still reach the apiserver.
Tested on kind with all three families: node InternalIPs, Service ClusterIPs
and pod IPs land in the requested families, pod-to-pod and CoreDNS work on
them, and pods still pull through the local registry.
The egress gateway's Envoy config lives as one inline string in the
atenet-egress ConfigMap, which Kustomize can replace but cannot patch
into, so the sdsmint variant is a whole second manifest rather than an
overlay. The flag picks between the two.
deploy_ate_system and deploy_atenet apply whichever manifest the flag
selects. delete_atenet deletes both, because teardown has to clean up an
install made under the other setting and either file may declare
resources the other does not.
atenet-egress-with-sdsmint.yaml starts as a copy of atenet-egress.yaml,
so the flag is a no-op until the two diverge.
Adds Postgres as an alternative persistence backend for ateapi as
proposed in #731. Currently this is opt-in with flags in ateapi or use
the `--store-backend=postgres` option in the install script.
Slack discussion:
https://cloud-native.slack.com/archives/C0B6M3E2J3D/p1785183769252679
Performance benchmark results for Postgres vs Redis:
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
Benchmarks are not included in this PR, but you can reproduce the
results by following the doc above
Limitations and questions:
- This draft does not yet handle migrations, this might come as a follow
up
- It currently uses `testcontainers` for testing Postgres, but as a
result it adds a lot of dependencies
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
This was breaking
`hack/install-ate.sh --delete-all` with
`--delete-demo-autoscaled-workerpool is not supported on GKE`
Also removed the custom usage message for
--deploy-demo-autoscaled-workerpool. This was duplicate with the default
message:
Before:
```
Demo: demo-autoscaled-workerpool
--deploy-demo-autoscaled-workerpool Deploy demo-autoscaled-workerpool
--delete-demo-autoscaled-workerpool Delete demo-autoscaled-workerpool
--deploy-demo-autoscaled-workerpool Deploy autoscaled-workerpool demo (HPA + prometheus-adapter + counter workload)
```
After:
```
Demo: demo-autoscaled-workerpool
--deploy-demo-autoscaled-workerpool Deploy demo-autoscaled-workerpool
--delete-demo-autoscaled-workerpool Delete demo-autoscaled-workerpool
```
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Benchmarking needed updates to fix auth bitrot and to enable readint
endpointslices. Also streamlined the ability to manually benchmark
microvm and removed Python Locust workers by default to simplify
self-serve benchmarking.
## Summary
Adds `hack/metrics/ci-failure-analysis.py`, a script that samples recent
failed `pr-workflow` CI runs, fetches their job logs, and classifies
each failure by root cause rather than treating all CI failures as
equivalent.
**Failure categories:**
- `named_test_fail` — a specific Go test emitted `--- FAIL: TestName`
- `no_free_workers` — envtest/integration tests hit worker resource
contention
- `e2e_timeout` — e2e tests timed out waiting for actor/service
responses (503s, connection resets)
- `gcs_access` — GCS/S3 bucket access denied when fetching sandbox
assets
- `license_check` — `hack/verify/licenses.sh` detected uncommitted
LICENSES changes
- `verify_fail` — other `hack/verify-all.sh` step failed
- `unknown` — could not classify from log output
**Usage:**
```bash
python hack/metrics/ci-failure-analysis.py \
--repo agent-substrate/substrate \
--fetch-limit 500 --sample 40
# Save for later comparison
python hack/metrics/ci-failure-analysis.py \
--repo agent-substrate/substrate \
--fetch-limit 500 --sample 40 \
--save ci-breakdown.json
```
**Sample output (40 sampled runs, Aug 2026):**
```
Named Go test failure 50% ████████████████
No free workers 30% ██████████
E2e timeout / 503 5% █
Unclassified 15% █████
Named test failures:
6x TestDurableDirLifecycle
4x TestActorLifecycle
2x TestMultipleDurableDirLifecycle
2x TestSyncer_UpdateWorker_RetryOnVersionConflict
2x TestLoaderConcurrentHandshakes
```
This informed the timeout fixes in #800 and is intended to feed the
planned scheduled CI remediation agent.
## Checklist
- [x] Issue is linked above (see #799)
- [x] Tests pass locally (`go test ./...`)
- [x] Documentation updated if behavior changed (script is new, no
existing docs affected)
---------
Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>
stage-to-rustfs.sh required developers to install the `aws` CLI, and
reached rustfs through a `kubectl port-forward`. The port-forward is a
footgun: when it targets the wrong cluster, or runs before the control
plane exists, the uploads fail against a dead localhost:9000 rather than
saying rustfs isn't there.
Run the S3 client in a throwaway container instead, using the same
pinned amazon/aws-cli image as the rustfs-bucket-init Job that creates
the bucket, and stream each asset in on stdin. The container joins the
kind node's network namespace so rustfs's ClusterIP is routable -- a
container merely attached to the `kind` docker network reaches the
node's own IP but has no route to the service or pod CIDRs. Dropping the
port-forward also fixes the macOS case, where a container cannot reach
it at all (the container's localhost is the Docker VM, not the host).
Also wait for the rustfs rollout and the bucket-init Job before
uploading, so running this too early reports what it's waiting on
instead of timing out on a connection.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Surfaced in https://github.com/agent-substrate/substrate/pull/743
The gateway access-log assertion read the log once, immediately after curl
returned, and failed if it saw nothing new. That is racy twice over:
* Envoy emits the CONNECT access-log entry asynchronously, so for an
external destination it can land seconds after the actor's response.
* The Actor's HTTP client keeps the tunnel alive. A repeat fetch to a host
it already reached rides the open tunnel and produces no new entry at
all, so a run against a warm actor failed even though egress was working.
Poll for the new entry, and fall back to any tunnel already open for this
actor's SAN before declaring failure. Also read -c envoy explicitly (the pod
also runs the ext-proc sidecar) and mark the script executable, as every other
directly-invoked script in hack/ is.
demos/egress is a small Actor that fetches a URL it is given and echoes the
upstream status and body back, which makes the egress path observable from
outside the sandbox. hack/install-demo-egress.sh registers it as a
--deploy-demo-egress fixture and hack/verify-egress-demo.sh drives it and
checks the atenet-egress logs for the corresponding authorized CONNECT.
TestActorEgress in the networking suite covers the same path automatically:
it creates an Actor from the demo template, POSTs a fetch request through
atenet-router, and asserts 200. The suite's actor helper is parameterised by
template so the ingress test keeps using the counter fixture.
The gateway terminates actor's CONNECT request.
It requires downstream mTLS, so only a worker's atunnel can reach it. The gateway consists of an Envoy and an `atenet router --standalone` ext_proc sidecar.
Fixes#565
Replaces standard DNS resolution (`dns:///`) and the forced 1-minute
`MaxConnectionAge` reconnect workaround with a native, event-driven
Kubernetes gRPC resolver (`internal/k8sresolver`). The resolver uses
client-go's `SharedInformerFactory` and `EndpointSliceLister` to watch
`discoveryv1.EndpointSlice` resources in real time.
### Verification
Created a kind cluster to test changes by scaling the deployments up and
down and verified the new servers were connected to by clients.
#### 1. Kind Cluster Scale-Out & EndpointSlice Discovery
Click to view kubectl scale and EndpointSlice output
$ kubectl scale deployment/ate-api-server -n ate-system --replicas=4
deployment.apps/ate-api-server scaled
$ kubectl get pods -n ate-system -l app=ate-api-server
NAME READY STATUS RESTARTS AGE
ate-api-server-6bcd9fddbb-gxxth 1/1 Running 0 3m8s
ate-api-server-6bcd9fddbb-jgwbw 1/1 Running 0 3m8s
ate-api-server-6bcd9fddbb-m8nbx 1/1 Running 0 7m20s
ate-api-server-6bcd9fddbb-vwm2g 1/1 Running 0 7m20s
$ kubectl get endpointslices -n ate-system -l
kubernetes.io/service-name=api
NAME ADDRESSTYPE PORTS ENDPOINTS
api-dmzfc IPv4 443 10.244.0.15,10.244.0.19,10.244.0.31,10.244.0.32
#### 2. Proof of Active gRPC Load-Balancing Across All Replicas
```
[pod/ate-api-server-6bcd9fddbb-gxxth] {"time":"2026-07-29T13:20:49Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
[pod/ate-api-server-6bcd9fddbb-jgwbw] {"time":"2026-07-29T13:20:46Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
[pod/ate-api-server-6bcd9fddbb-m8nbx] {"time":"2026-07-29T13:20:48Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
[pod/ate-api-server-6bcd9fddbb-vwm2g] {"time":"2026-07-29T13:20:47Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
```
### Which issue(s) this PR is related to:
Fixes#721
Required for System Upgrade flow (#473)
### What this PR does / why we need it:
This PR implements graceful termination and zero-outage rolling updates
for `atenet-router` , building on the readiness probe and graceful drain
patterns established in #719 (`atelet`).
Draining a two-container networking pod (`atenet-router` Go
control-plane + `envoy` C++ proxy dataplane) introduces complex
lifecycle interdependencies. This PR resolves the dual-container SIGTERM
race, respects Envoy's `failClosed` `ext_proc` filter dependency,
preserves parked requests riding out worker pool saturation, and
accelerates idle deployments via an event-driven file handshake.
#### 1. Event-Driven Dataplane Synchronization (`emptyDir` Marker File)
Envoy fast-exits by default on SIGTERM. To keep Envoy alive while the Go
control-plane orchestrates the drain, we configured an IPC handshake
between containers via a pod-shared `emptyDir` volume mounted at
`/var/run/atenet`:
* **Go Router Container:** Removes any stale marker at startup, then
writes `/var/run/atenet/drain-complete` when its shutdown sequence
finishes.
* **Envoy Container `preStop` Hook:** Runs `while [ ! -f
/var/run/atenet/drain-complete ]; do sleep 0.5; done`.
**Outcome:** Envoy exits as soon as — the drain is done, rather than
wasting time in a fixed worst-case sleep. If the router crashes, Kubelet
terminates the `preStop` hook at `terminationGracePeriodSeconds:
60`—slower cleanup, never a wedge.
#### 2. The Multi-Container Shutdown Sequence (`drain.go`)
Because Kubernetes issues SIGTERM to both containers at once, a
coordination state machine ensures Envoy never drops a connection and
`ext_proc` is never stopped prematurely:
| Phase / (best-case ex.) Timeline | atenet-router | envoy | K8s /
Service Status |
| :--- | :--- | :--- | :--- |
| **SIGTERM Sent**<br>($t=0\text{s}$) | Catches SIGTERM<br>• Flips
`/readyz` $\rightarrow$ `503`.<br>• Starts 13s `drain-delay`. | Enters
`lifecycle.preStop` hook:<br>`while [ ! -f .../drain-complete ]; do
sleep 0.5; done`<br>• Kubelet holds SIGTERM back. | Deployment will
recreate the pod.<br>EndpointSlice controller begins dropping Old Pod
IP. |
| **Propagation**<br>($t=0\text{s}$ – $t=13\text{s}$) | Keeps serving
normally.<br>• `ext_proc` continues unparking/routing requests. | Runs
normally inside `preStop` loop.<br>• Serves active TCP connections. |
Service endpoint removal completes.<br>No new connections arrive at Old
Pod. |
| **Envoy drain**<br>($t=13\text{s}$) | Issues Envoy admin API
calls:<br>• `/healthcheck/fail`<br>•
`/drain_listeners?graceful&skip_exit`<br>• Polls `/stats` for active
downstream connections. | Begins graceful listener drain:<br>• GOAWAY /
`Connection: close` on established connections.<br>• In-flight requests
keep running. | In-flight HTTP requests finish executing through Envoy.
|
| **ext_proc drain**<br>($t\approx18\text{s}$) | Active Envoy
connections hit 0 (the poll exits early).<br>• Calls
`extproc.GracefulStop()`.<br>• Parked request streams finish. | All
downstream connections closed.<br>• Still waiting in the `preStop` loop.
| All client HTTP responses delivered. |
| **Handshake & Exit**<br>($t=18.5\text{s}$)| `ext_proc` drain
finishes.<br>• Writes `drain-complete` marker.<br>• Hard-stops xDS
(`Stop()`) & exits. | `preStop` loop detects marker file!<br>• `preStop`
exits 0.<br>• Kubelet sends SIGTERM to Envoy.<br>• Envoy exits
immediately. | Old Pod deleted cleanly at ~18.5s (well under the 60s
budget). |
Envoy ref:
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining
#### Testing & Verification
1. Unit Tests (`drain_test.go`): Added comprehensive unit tests
covering:
- Graceful completion of in-flight `ext_proc` streams within deadline.
- Force-stopping straggler streams past `--drain-timeout`.
- Envoy admin API interaction and connection polling.
- Stale marker cleanup and marker file creation.
2. Local testing on kind cluster (Observations):
Test script: `hack/verify-atenet-drain.sh`
- Steady state: `/readyz=200`, `/healthz=200`; the instant the pod
turned Terminating: `/readyz=503` while `/healthz=200` — `NotReady` but
alive, for the whole drain.
- The parked request survived the shutdown: fired at a busy 1-worker
pool → parked → pod deleted while parked → worker freed → HTTP=200,
total=1.89s, body hello from: 169.254.17.2 | preserved memory count: 1 |
preserved file counter: 1 — park → resume → route, served by the
Terminating pod during its drain-delay window.
- Pod terminated 14s after deletion — the idle floor, confirmed across
three runs: >13s drain-delay (sequence ran), ≪60s grace (marker
handshake released Envoy's preStop; no `SIGKILL`).
- Log sequence captured verbatim, drain-delay honored to the millisecond
(17:34:41.706 → 17:34:54.707):
>Shutdown signal received; draining
>(+13.000s) Draining Envoy {window: 15s}
>Envoy drained
>Starting ext_proc drain
>ext_proc drain completed within deadline
>Drain-complete marker written {path: /var/run/atenet/drain-complete}
>Shutdown complete
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
`setup-envtest use` with no version re-resolves "latest" against the
remote release index on every run, so the envtest-backed packages needed
network even with fully cached binaries.
This PR pins the control plane to `1.36.x`, and fold the three
per-package `envtest` into `internal/testenv.Start`, which also honors
`-short`: `go test -short ./...` skips the envtest-backed packages and
is the guaranteed-offline path.
Have Claude to help me fix it
Fixes#789
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Currently we are unable to easily automate the benchmarking of micro
VMs.
This PR:
* Refactors the microVM setup scripts to allow discrete installation of
uVM sanbox configs separate from counter demo
* Pipes a MicroVM container test config through the benchmarking system
* Updates default atelet daemonset to tolerate all kinds of sandboxClass
taints
OTEL_EXPORTER_OTLP_ENDPOINT is written in 9 places: hardcoded inline in
4 base workload manifests, then re-patched in 5 spots across the kind
overlays. Changing the collector address means editing all 9 and knowing
which install path renders which.
Replace them with one checked-in ConfigMap per environment, consumed by
every component via envFrom:
manifests/ate-install/ate-otel-config.yaml (GKE)
manifests/ate-install/kind/ate-otel-config.yaml (kind, same name)
The GKE copy is listed in manifests/ate-install/base, so token-client,
agentgateway and agentgateway-token-client all inherit it; the kind
overlay lists its own copy of the same ConfigMap name. No overlay builds
on both, so the two never collide.
A kustomize-only fix does not work here. The GKE path applies the base
directory raw and hack/install-ate.sh's targeted redeploys apply single
files with no Kustomize, so the mechanism has to survive `kubectl apply
-f <one-file>` -- which rules out configMapGenerator (hash-suffixed
names) and replacements. envFrom on a stable name reaches every path.
deploy_ate_system applies the ConfigMap before the rendered bundle, the
same way it already applies the namespace. A container whose envFrom
target is missing does not start, and a raw directory apply is ordered
by filename, so ate-api-server.yaml and ate-controller.yaml would
otherwise be created first and sit in CreateContainerConfigError until
the ConfigMap caught up.
This also fixes a latent bug in the targeted redeploys.
deploy_ate_apiserver, deploy_atelet and deploy_atenet apply raw files
even in kind mode, silently reverting the endpoint to the GKE value.
They now apply the environment's ConfigMap via a new apply_otel_config
helper, which picks the file by ATE_INSTALL_KIND rather than applying
the base copy unconditionally -- applying the base one on kind would
break telemetry for every component at once.
atenet-router needs no special handling: since a98f85b1 its
--otlp-collector-address defaults to $OTEL_EXPORTER_OTLP_ENDPOINT, so
Envoy's own spans follow the ConfigMap along with everything else.
OTEL_TRACES_SAMPLER deliberately stays an inline kind patch on
ate-api-server rather than moving into the ConfigMap. The ConfigMap is
shared by every component via envFrom, so putting parentbased_always_on
there would pin the router and the rest of the control plane to 100% and
undo the per-component ratios from 15eecd02.
Note that a ConfigMap edit does not roll the consuming pods the way an
inline env change did, since the pod template is unchanged. Callers must
follow a change with `kubectl rollout restart`. This is documented in
both ConfigMaps, in docs/observability.md, and in the tracing best
practices, which previously told authors to hardcode the variable.
Fixes#745
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Fixes#697
## Summary
`deploy_ate_system` gated `manifests/ate-install/generated/` behind
`ensure_crds`, whose existence check skips the directory on any upgrade
— stranding CRD schemas and, since `role.yaml` has no other apply path,
all ClusterRoles at first-install state. A controller image needing a
new permission then deadlocks on informer start while the rollout
reports success.
`deploy_ate_system` now calls `deploy_crds` unconditionally (`kubectl
apply` is idempotent). The demo scripts and per-component deploy flags
keep `ensure_crds`, where "make sure they exist" is the intended
semantic.
## Test plan
- Reproduced on a 4-day-old kind cluster: upgrading the control plane
deadlocked ate-controller on `cannot list networkpolicies`; manually
applying `role.yaml` unblocked it.
- With this fix, the same `install-ate-kind.sh --deploy-ate-system` run
prints the `deploy_crds` step and re-applied the drifted manifests
(`actortemplates.ate.dev configured`, `workerpools.ate.dev configured`,
ClusterRoles applied); cluster reconciles normally.
- `bash -n hack/install-ate.sh`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR (none
needed)
Benjamin Elder