Add the `ate.imagecache.requests` counter.
**`ate.imagecache.outcome`** (new key in `internal/ateattr`)
| Value | Meaning |
|---|---|
| `hit` | the node holds a complete image record: each layer directory
that the record names is present |
| `miss` | the lookup must pull |
| `error` | the lookup failed; the only outcome that carries
`error.type` |
| `cancelled`, `timeout` | the caller gave up, so the cache is not at
fault |
A failed lookup is neither a hit nor a miss, so it gets its own outcome,
as
`no_free_worker` does on `ate.scheduler.outcome`. `cancelled` and
`timeout` are
outcomes for the same reason they are on `ate.router.outcome`. The hit
ratio is
therefore `hit / (hit + miss)`, with failures and abandoned lookups out
of the
denominator.
**`error.type`** — set only on the `error` outcome.
| Value | Meaning |
|---|---|
| `404`, `401`, `429`, ... | the registry rejected the request; its own
HTTP status, reported verbatim |
| `_OTHER` | the failure carries no status of its own |
Fixes#831
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
### Why
The README Quickstart covers getting a local Substrate cluster running
with
the default gVisor runtime, but there's no documentation for the
micro-VM
runtime, where the friction can't be scripted away: it requires
`/dev/kvm`
(bare metal, nested virtualization, or Lima on macOS), and on Apple
Silicon
the Lima configuration and asset assembly are non-obvious. New
contributors
have to reverse-engineer the `hack/` scripts.
### What
Adds `docs/dev/microvm-local.md` — "Running the microVM runtime locally"
— a
focused guide covering only the microVM delta, based on notes from real
onboarding runs. General setup is deferred to the README Quickstart
(listed
as the guide's prerequisite) rather than duplicated:
- **Option A: Linux host with KVM** — verifying `/dev/kvm` and CPU virt
support, the rootless-Docker caveat for the KVM probe, cluster creation,
and the one-shot `run-microvm-demo-kind.sh` bring-up.
- **Option B: Apple Silicon macOS via Lima** — nested virtualization
with the
guest image pinned to Ubuntu 25.10 (until the kernel issue in the
default
image is fixed), `vzNAT` networking, and assembling the arm64 assets
inside
the Lima guest (`assemble.sh` requires a Linux host of the target arch).
- **Trying it out** — defers to the next steps the demo script prints
and
links the counter demo's micro-VM variant, instead of duplicating those
commands.
- **Troubleshooting** — symptom → root cause → fix entries actually hit
during onboarding (rootless-Docker KVM probe failures, the `aws` CLI
staging requirement until #804 lands, arm64 `virtiofsd` build deps, M1
lacking FEAT_NV2).
This reverts the ActorTemplate valueFrom.secretKeyRef support added in
#20 (issue #15).
We don't want actors to have any access to secrets, they will be
injected on the egress route instead. Removing this now so we don't need
to copy secrets into substrate resources.
Part of #564 (Part 4, Fixes#564)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements Part 4 of #564 by adding OpenTelemetry gauge
instrumentation for `ate.workerpool.desired_workers` and
`ate.workerpool.ready_workers` in `atecontroller`.
## Key Changes:
* Added `ReadyReplicas int32` to `WorkerPoolStatus` with
`+kubebuilder:printcolumn:name="Ready"` annotation in
`pkg/api/v1alpha1/workerpool_types.go` and regenerated the CRD manifest.
* Updated `WorkerPoolReconciler.syncStatus` to synchronize
`dep.Status.ReadyReplicas` into `wp.Status.ReadyReplicas`.
* Implemented `InitMetrics(meter)` registering
`ate.workerpool.desired_workers` and `ate.workerpool.ready_workers` as
OpenTelemetry Observable UpDownCounters (`{worker}`) with an
asynchronous observer callback that samples controller-runtime's local
informer cache (`r.Client.List`).
* Labeled metric datapoints using centralized attributes
`ateattr.WorkerPoolNamespaceKey` and `ateattr.WorkerPoolNameKey`.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
---------
Co-authored-by: Jeff Luo <jeffluoo@google.com>
The pause image holds the sandbox's namespaces and runs no workload
code. It is an implementation detail of the sandbox, not something actor
authors pick, so it belongs with the sandbox binaries that already moved
off the ActorTemplate onto the cluster-scoped SandboxConfig.
It now travels with those binaries end to end: resolved from the pool's
SandboxConfig, carried on ateletpb.SandboxAssets rather than
WorkloadSpec, and recorded in the per-actor sandbox record so Checkpoint
pins it into the snapshot manifest and Restore rebuilds the sandbox from
the image the snapshot was taken with (the golden's on a DATA_ON_GOLDEN
restore). A record without one is rejected outright rather than pulling
an empty image.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #791 — the last planned piece of the [implementation
plan](https://github.com/agent-substrate/substrate/issues/791#issuecomment-5226674097)
(after #810, #812, #813). #791 stays open until #817 (gVisor
Full-capture → Data-commit conversion, blocked on #790) is done; this PR
covers micro-VM fully and gVisor for scope-matched suspends. Also part
of the actor state machine (#119) and a prerequisite for system upgrade
flows (#473).
`SuspendActor` now accepts a PAUSED actor: instead of checkpointing a
running workload, ateapi dials the atelet on the node holding the pause
snapshot and has it upload the node-local files to object storage, then
finalizes as usual — durable `ActorSnapshot`, `SUSPENDED` status, node
pinning cleared.
Two commits, reviewable independently:
## Commit 1 — atelet: `UploadPausedCheckpoint` RPC (dead code until
commit 2)
- New `AteomHerder` RPC: a pure disk→object-storage copy driven by the
snapshot's self-describing manifest — no ateom involved (the sandbox is
gone).
- **Scope conversion** dispatches per sandbox class
(`narrowFullCaptureToData`): a micro-VM FULL capture narrows to a DATA
upload by carving out `durable-dir.tar` (constant hoisted to
`ateompath`, shared with ateom-microvm); gVisor returns `Unimplemented`
until split checkpoints land (#790); DATA can never widen to FULL; a
scope-less manifest (older atelet) is rejected rather than guessed at.
- **Idempotent retry**: local files gone + remote manifest present ⇒ a
previous invocation committed, succeed; gone on both sides ⇒
unrecoverable (`LOCAL_SNAPSHOT_GONE`, crashes the actor). Upload
failures stay plain retryable errors; the manifest uploads last as the
commit marker, never in parallel.
- The golden atespace is rejected at validation (fully on the
`field.ErrorList` framework): golden actors are never paused.
## Commit 2 — control plane: enable suspend from PAUSED
- `FromPaused` discriminator: PAUSED status, or SUSPENDING with no
worker assignment and a `LocalSnapshotInfo` — the field alone is stale
on resumed-from-pause RUNNING actors, so the nil-assignment conjunct is
load-bearing.
- `MarkSuspendingStep` accepts PAUSED and rejects a Data-captured pause
against a Full commit *before* the actor leaves PAUSED (an upload cannot
fabricate memory), using the `content_scope` recorded at pause (#812)
with an `onPause` fallback.
- `CallAteletSuspendStep` paused branch dials by node
(`DialForAteletOnNode`, #813): missing node record ⇒ crash (the snapshot
can never be found); unreachable atelet ⇒ retryable; atelet's
`LOCAL_SNAPSHOT_GONE` ⇒ crash via `maybeCrashActor`.
`FinalizeSuspendedStep` needed no changes thanks to the #813 hoist.
- Root-cause guard: `MarkPausingStep` rejects pausing golden-atespace
actors.
- Docs: pause states + the new `PAUSED → SUSPENDING` edge in the
architecture state diagram; glossary Suspend entry covers both origins.
## Tests
- **atelet unit**: 10 upload-helper subtests (conversion matrix,
idempotency probe, data-loss crash, retryable upload failure) via a
recording object-storage fake; validation table.
- **control-plane unit**: discriminator table, scope-rejection table
(incl. onPause fallback), paused preconditions (no node ⇒ CRASHED, no
atelet ⇒ `ErrNoAteletOnNode` + still SUSPENDING), golden-pause
rejection, prerequisite matrix updated.
- **functional (envtest)**: `TestSuspendActor_FromPaused` (upload called
with the pause snapshot name, no Checkpoint RPC, SUSPENDED, pinning
cleared, ActorSnapshot at the upload destination) +
retry-after-failed-upload (same destination on retry).
- **e2e (demo suite)**: the lifecycle driver gains a suspend-from-PAUSED
mode; three durable-dir cases — Full/Full, Data/Data, and the micro-VM
Full→Data extraction — assert memory/file counters survive the full
pause→suspend→resume journey and the node pinning is gone.
`go test -race ./...` clean (except the pre-existing macOS-only
`internal/atunnel` unix-socket-path failures, untouched by this PR),
`gofmt`/`go vet` clean, protos regenerated via `hack/protoc.sh`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Fixes#773
Some improvements to the way we create, track and propagate snapshot
ids.
* Consolidate snapshot id and name into a single concept.
* Use consistent terminology across the whole stack (i.e. remove
confusion about prefix vs id vs name).
* Get rid of internal only storage to hide the snapshot URI.
## Summary
- `atecontroller` propagates a pool's `nvidia.com/gpu` request onto the
`ateom` container and mounts the host NVIDIA toolkit read-only (path
overridable via `ATE_NVIDIA_TOOLKIT_HOST_PATH`)
- `ateom-gvisor` generates a CDI spec with `nvidia-ctk` and injects the
device nodes, driver-library mounts, and env into each actor container's
OCI spec
- runs the CDI `createContainer` hooks except `update-ldcache` which
needs a privileged ateom, staging the SONAME symlinks it would create
from each library's ELF `DT_SONAME`
- enables `runsc --nvproxy` at sandbox creation
Requesting `nvidia.com/gpu` on the pool is the only configuration
needed; a pool that requests N GPUs makes all N usable.
Two details of the CDI spec are worth calling out, because getting
either wrong fails at runtime rather than at parse time. `nvidia-ctk`
leaves `major`/`minor` unset — CDI delegates that to the OCI runtime —
so each device node is resolved by stat-ing the host; without it the
actor gets `0,0` char devices and NVML reports it cannot communicate
with the driver. And it emits per-index, per-UUID, and `all` devices
that repeat the same nodes, so only `all` is applied. The spec is plain
JSON, so `encoding/json` suffices and no CDI library is vendored.
`update-ldcache` is the one hook that cannot run here: its `ldconfig`
unshares a mount namespace and mounts a private `/proc`, which
`mount_too_revealing()` rejects under the pod's masked `/proc`.
Permitting it would need `procMount: Unmasked`, which Kubernetes only
allows with `hostUsers: false`, and that user namespace breaks the
per-actor cgroup delegation from #496. Skipping it avoids the whole
chain, so a GPU worker keeps the same posture as any other unprivileged
gVisor worker. `create-symlinks` and `enable-cuda-compat` still run
unmodified.
`--nvproxy` must be set when the sandbox is created — the `pause`
container, which holds no GPU devices — so runsc's auto-detection never
fires on its own; without the flag the GPU subcontainer crashes the
sentry on start. GPU detection matches any device index rather than
assuming `/dev/nvidia0`, since a worker sharing a multi-GPU node can be
assigned `/dev/nvidia2` and `/dev/nvidia3`.
GPU pools must set `spec.ateomImage` to a glibc build
(`KO_DEFAULTBASEIMAGE=debian:stable-slim ko build ./cmd/ateom-gvisor`)
because the distroless default cannot exec `nvidia-ctk`; the default
base is unchanged for every other pool. `atelet` also has to run on the
GPU nodes to restore actors there, so its DaemonSet needs a toleration
for whatever taint they carry. Both are documented in the API guide
rather than defaulted.
## Testing
- `make test`
- `env -u NO_COLOR make verify`
- Real GPU, Tesla T4 / driver 580.65.06, through the full actor flow:
`nvidia-smi`, `vectorAdd`, `nbody` at 3.77 TFLOP/s, PyTorch matmul via
cuBLAS at 3.5 TFLOP/s (T4 peak FP32 is ~8.1, so no measurable sandbox
penalty)
- Actor whose entrypoint runs the CUDA sample directly exits 0,
confirming the injected env reaches the workload without help from the
test harness
- Two workers holding two GPUs each on one 4-GPU node see disjoint
device sets
Snapshot and restore work when the workload holds no CUDA context. A
live CUDA context cannot be checkpointed - gVisor fails with `can't save
with live nvproxy clients` and the failed checkpoint terminates the
sandbox so a GPU actor can only be suspended between CUDA workloads.
Documented as a known limitation in the API
guide; a follow-up issue will track lifting it via `cuda-checkpoint`.
Fixes#627
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
---------
Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
### Which issue(s) this PR is related to:
Fixes#721
Required for System Upgrade flow (#473)
### What this PR does / why we need it:
This PR implements graceful termination and zero-outage rolling updates
for `atenet-router` , building on the readiness probe and graceful drain
patterns established in #719 (`atelet`).
Draining a two-container networking pod (`atenet-router` Go
control-plane + `envoy` C++ proxy dataplane) introduces complex
lifecycle interdependencies. This PR resolves the dual-container SIGTERM
race, respects Envoy's `failClosed` `ext_proc` filter dependency,
preserves parked requests riding out worker pool saturation, and
accelerates idle deployments via an event-driven file handshake.
#### 1. Event-Driven Dataplane Synchronization (`emptyDir` Marker File)
Envoy fast-exits by default on SIGTERM. To keep Envoy alive while the Go
control-plane orchestrates the drain, we configured an IPC handshake
between containers via a pod-shared `emptyDir` volume mounted at
`/var/run/atenet`:
* **Go Router Container:** Removes any stale marker at startup, then
writes `/var/run/atenet/drain-complete` when its shutdown sequence
finishes.
* **Envoy Container `preStop` Hook:** Runs `while [ ! -f
/var/run/atenet/drain-complete ]; do sleep 0.5; done`.
**Outcome:** Envoy exits as soon as — the drain is done, rather than
wasting time in a fixed worst-case sleep. If the router crashes, Kubelet
terminates the `preStop` hook at `terminationGracePeriodSeconds:
60`—slower cleanup, never a wedge.
#### 2. The Multi-Container Shutdown Sequence (`drain.go`)
Because Kubernetes issues SIGTERM to both containers at once, a
coordination state machine ensures Envoy never drops a connection and
`ext_proc` is never stopped prematurely:
| Phase / (best-case ex.) Timeline | atenet-router | envoy | K8s /
Service Status |
| :--- | :--- | :--- | :--- |
| **SIGTERM Sent**<br>($t=0\text{s}$) | Catches SIGTERM<br>• Flips
`/readyz` $\rightarrow$ `503`.<br>• Starts 13s `drain-delay`. | Enters
`lifecycle.preStop` hook:<br>`while [ ! -f .../drain-complete ]; do
sleep 0.5; done`<br>• Kubelet holds SIGTERM back. | Deployment will
recreate the pod.<br>EndpointSlice controller begins dropping Old Pod
IP. |
| **Propagation**<br>($t=0\text{s}$ – $t=13\text{s}$) | Keeps serving
normally.<br>• `ext_proc` continues unparking/routing requests. | Runs
normally inside `preStop` loop.<br>• Serves active TCP connections. |
Service endpoint removal completes.<br>No new connections arrive at Old
Pod. |
| **Envoy drain**<br>($t=13\text{s}$) | Issues Envoy admin API
calls:<br>• `/healthcheck/fail`<br>•
`/drain_listeners?graceful&skip_exit`<br>• Polls `/stats` for active
downstream connections. | Begins graceful listener drain:<br>• GOAWAY /
`Connection: close` on established connections.<br>• In-flight requests
keep running. | In-flight HTTP requests finish executing through Envoy.
|
| **ext_proc drain**<br>($t\approx18\text{s}$) | Active Envoy
connections hit 0 (the poll exits early).<br>• Calls
`extproc.GracefulStop()`.<br>• Parked request streams finish. | All
downstream connections closed.<br>• Still waiting in the `preStop` loop.
| All client HTTP responses delivered. |
| **Handshake & Exit**<br>($t=18.5\text{s}$)| `ext_proc` drain
finishes.<br>• Writes `drain-complete` marker.<br>• Hard-stops xDS
(`Stop()`) & exits. | `preStop` loop detects marker file!<br>• `preStop`
exits 0.<br>• Kubelet sends SIGTERM to Envoy.<br>• Envoy exits
immediately. | Old Pod deleted cleanly at ~18.5s (well under the 60s
budget). |
Envoy ref:
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining
#### Testing & Verification
1. Unit Tests (`drain_test.go`): Added comprehensive unit tests
covering:
- Graceful completion of in-flight `ext_proc` streams within deadline.
- Force-stopping straggler streams past `--drain-timeout`.
- Envoy admin API interaction and connection polling.
- Stale marker cleanup and marker file creation.
2. Local testing on kind cluster (Observations):
Test script: `hack/verify-atenet-drain.sh`
- Steady state: `/readyz=200`, `/healthz=200`; the instant the pod
turned Terminating: `/readyz=503` while `/healthz=200` — `NotReady` but
alive, for the whole drain.
- The parked request survived the shutdown: fired at a busy 1-worker
pool → parked → pod deleted while parked → worker freed → HTTP=200,
total=1.89s, body hello from: 169.254.17.2 | preserved memory count: 1 |
preserved file counter: 1 — park → resume → route, served by the
Terminating pod during its drain-delay window.
- Pod terminated 14s after deletion — the idle floor, confirmed across
three runs: >13s drain-delay (sequence ran), ≪60s grace (marker
handshake released Envoy's preStop; no `SIGKILL`).
- Log sequence captured verbatim, drain-delay honored to the millisecond
(17:34:41.706 → 17:34:54.707):
>Shutdown signal received; draining
>(+13.000s) Draining Envoy {window: 15s}
>Envoy drained
>Starting ext_proc drain
>ext_proc drain completed within deadline
>Drain-complete marker written {path: /var/run/atenet/drain-complete}
>Shutdown complete
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR implements the last two metrics from #433.
Right now, we can see that a resume was slow but not where.
`ate.actor.lifecycle.operation.duration` covers the whole ateapi
operation, and `atenet.router.route.duration` covers the edge, but
everything between ateapi-atelet-actors is one block that contains
fetching the manifest, downloading the snapshot, unpack the OCI image,
call to ateom.
We have `rpc.server.call.duration` that gives us the atelet restore
total time, but template, kind, and scope labels are missing, so today
we cannot really pinpoint why/where we have a regressions in latency.
In this PR I am adding per-phase histograms, here's an example of what
we can know after these changes:
```console
ateom_restore 522 ms ###############################
download 8.9 ms #
manifest_fetch 3.2 ms
oci_unpack 2.6 ms
---------------------------------------------------------
total 535 ms
```
It also fixes a gap #683 opened where a `data_on_golden` resume was
labeled identically to a plain one on the lifecycle histogram.
Things folks might want to argue with:
- total as a phase value. Partly duplicates `rpc.server.call.duration`,
but that one has no domain labels and gRPC-specifc. We can drop it, but
it's an inferior operational UX, so I'd rather have it here.
- Phases overlap, they are not a partition of total, because download
runs concurrently with the asset fetch and unpack. I called this out in
the metric description. Do not sum across phases.
- New `ate.snapshot.scope` key rather than a new `ate.snapshot.kind`
value for `data_on_golden`. A new value would collapse local and
external into one bucket, which is the biggest latency difference there
is. This does add a label to the already shipped lifecycle histogram.
Verified on kind, and all e2e suites pass, and the emitted series cover
every kind (golden, latest, local) on both metrics with no unknown
values.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #564 (Part 3)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements Part 3 of #564 by adding telemetry histogram
instrumentation for `ate.scheduler.eligible_workers` in `ateapi`. It
measures unassigned free worker capacity remaining after all scheduling
constraint filters are applied, sampled at every scheduling decision.
## Key Changes:
* Updated `Scheduler.Schedule()` to record eligible candidate workers
per pool (`recordEligibleWorkers`).
Defined `SchedulingConstraintKey` (`ate.scheduling.constraint`) and
constraint classification values (`none`, `required_nodes`, `selector`)
in `internal/ateattr/ateattr.go`.
* Added unit tests in `scheduling_test.go` covering candidate counts,
namespaced attributes, zero-capacity fleet states, empty fleets, sandbox
class mismatches, draining workers, and constraint classifications.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
atecontroller had no OTel at all. Dev-mode zap logger, and its
controller-runtime metrics were only available on a :8080 that we don't
scrape.
After this PR, logs go through the shared slog handler (plus a
`--log-level` flag to match the other binaries), and
controller-runtime's Prometheus registry is bridged onto the OTLP reader
so the reconcile/workqueue metrics actually reach the collector.
Filtering these out is a pipeline responsibility. Also added otelgrpc to
the ateapi client, which was untraced.
This unblocks #564 the workperpool metrics, cc @Angelawork, @JeffLuoo:
there's a working `MeterProvider` to use for
`ate.workerpool.desired_workers`/`ready_workers`.
Couple of things to mention for review:
-`InitMetricsPushOnly`, not `InitMetrics`, even though we do serve
:8080. That port is controller-runtime's own private registry, not the
global one `serverboot.metricsMux` serves, so a pull reader there would
collect into something we never expose.
- Bridge is pinned to v0.68.0 to match otelgrpc. Wanted to go to
v0.70.0, but that requires otel/sdk/metric 1.45.0 and pulls the whole
SDK up with it (406 vendor files instead of 81). Happy to do that bump
separately.
- zap/zapr fall out of go.mod since atecontroller was the last importer.
- I included an OTel collector image bump from the early 2024 (!) one to
latest, which was breaking exposing native histograms
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#564 (Part 2)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements the second part of #564 by adding telemetry counter
instrumentation for `ate.actor.crashes` in `ateapi` and establishing
bounded label validation for operations and crash failure causes.
## Key Changes:
- Updated `maybeCrashActor` and `crashActor` to reference
`ateattr.OperationName*` constants and normalize `opName`.
- Passed explicit failure reasons (`ReasonCorruptedAssignment`,
`ReasonWorkerPodGone`, `ReasonWorkerReassigned`) across
`workflow_resume.go`, `workflow_suspend.go`, and `workflow_pause.go`.
- Instrumented `WorkerPoolSyncer.releaseActorOnDeadWorker` in
`syncer.go` to record `recordActorCrash` when background pod deletion
crashes an actor.
- Instrumented `FinalizePausedStep` in `workflow_pause.go` to record
`recordActorCrash` when `nodeName` is missing during pause finalization.
- Added `ate_actor_crashes` to `PlatformMetricPrefixes` in
`collector_metrics.go`.
- Added pre-creation resource cleanup in `metrics_test.go` to prevent
`AlreadyExists` errors on test reruns.
- Updated `metrics_test.go` to trigger an actor crash via `UpdateActor`
and `ResumeActor`, asserting `ate_actor_crashes` emission and label
presence in OTel collector scrape outputs.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
OTEL_EXPORTER_OTLP_ENDPOINT is written in 9 places: hardcoded inline in
4 base workload manifests, then re-patched in 5 spots across the kind
overlays. Changing the collector address means editing all 9 and knowing
which install path renders which.
Replace them with one checked-in ConfigMap per environment, consumed by
every component via envFrom:
manifests/ate-install/ate-otel-config.yaml (GKE)
manifests/ate-install/kind/ate-otel-config.yaml (kind, same name)
The GKE copy is listed in manifests/ate-install/base, so token-client,
agentgateway and agentgateway-token-client all inherit it; the kind
overlay lists its own copy of the same ConfigMap name. No overlay builds
on both, so the two never collide.
A kustomize-only fix does not work here. The GKE path applies the base
directory raw and hack/install-ate.sh's targeted redeploys apply single
files with no Kustomize, so the mechanism has to survive `kubectl apply
-f <one-file>` -- which rules out configMapGenerator (hash-suffixed
names) and replacements. envFrom on a stable name reaches every path.
deploy_ate_system applies the ConfigMap before the rendered bundle, the
same way it already applies the namespace. A container whose envFrom
target is missing does not start, and a raw directory apply is ordered
by filename, so ate-api-server.yaml and ate-controller.yaml would
otherwise be created first and sit in CreateContainerConfigError until
the ConfigMap caught up.
This also fixes a latent bug in the targeted redeploys.
deploy_ate_apiserver, deploy_atelet and deploy_atenet apply raw files
even in kind mode, silently reverting the endpoint to the GKE value.
They now apply the environment's ConfigMap via a new apply_otel_config
helper, which picks the file by ATE_INSTALL_KIND rather than applying
the base copy unconditionally -- applying the base one on kind would
break telemetry for every component at once.
atenet-router needs no special handling: since a98f85b1 its
--otlp-collector-address defaults to $OTEL_EXPORTER_OTLP_ENDPOINT, so
Envoy's own spans follow the ConfigMap along with everything else.
OTEL_TRACES_SAMPLER deliberately stays an inline kind patch on
ate-api-server rather than moving into the ConfigMap. The ConfigMap is
shared by every component via envFrom, so putting parentbased_always_on
there would pin the router and the rest of the control plane to 100% and
undo the per-component ratios from 15eecd02.
Note that a ConfigMap edit does not roll the consuming pods the way an
inline env change did, since the pod template is unchanged. Callers must
follow a change with `kubectl rollout restart`. This is documented in
both ConfigMaps, in docs/observability.md, and in the tracing best
practices, which previously told authors to hardcode the variable.
Fixes#745
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This change formalizes the invariant that all these fields are either
all set or not. This simplifies some client code that was just checking
random worker metadata fields to see if the actor has an assignment or
not.
It also fixes a bug in `{Pause,Suspend}Actor` where we were not cleaning
up the worker pod UID field.
This PR is the second slice of the platform-metrics split (#433).
It adds two duration histograms emitted by ateapi:
- `ate.actor.lifecycle.operation.duration`:
create/resume/suspend/pause/delete, labeled by operation, template,
pool, sandbox class, and (on resume) snapshot kind
- I'd like to use this for user-facing latency for e.g. showing when
suspended actors can serve requests again. The existing
`rpc.server.call.duration` metric covers this, but the meaningful
dimensions are missing, so it's not really actionable. Extending its
labels with extra, domain-specific labels is an OTel anti-pattern, hence
the new metric.
- `ate.scheduler.assignment.duration`: worker-assignment step, labeled
by outcome (assigned / no_free_worker / error) and pool
- I'd like to use this to alert on no free worker situation, and have
proper SLOs via various percentiles for assigning latencies. There's no
RPC around this, so it's not something existing RPC metrics cover.
Tested e2e on a local kind cluste where: both metrics reach the
otel-system collector with the expected labels (resume shows
`snapshot_kind=golden`, scheduler shows `outcome=assigned`, no
`error.type` on success).
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Today everything outside ateapi hardcodes `ParentBased(NeverSample)`,
and setting `OTEL_TRACES_SAMPLER` does nothing because an explicit
sampler silences the SDK's env handling. This makes troubleshooting
production issues impossible via traces.
This PR gives every component a sane default and makes the standard OTel
env vars work:
- Control plane (ateapi, atelet, ateom) defaults to
`parentbased_traceidratio` 0.1, the router (data plane root) to 0.01,
per the discussion on #584.
- `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` override any of this
without a rebuild. Invalid values keep the component default and log a
warning instead of inheriting the SDK's fall-open-to-100% behavior.
- Envoy's `RandomSampling` is derived from the router's resolved policy,
so the two root decisions cannot drift.
- kubectl-ate without `--trace` no longer installs a tracer provider at
all: the old `NeverSample` provider injected `sampled=0`, which pinned
every parent based sampler downstream and would have defeated the server
side ratios. `--trace` still forces a full end to end trace.
- ate-controller propagates the two env vars to the ateom worker pods it
creates, same as the metric export vars.
- kind pins ateapi to `parentbased_always_on`, so the local Jaeger flow
keeps showing every API call.
- The agentgateway integration follows what we have above. its
`randomSampling` changes from `true` to `0.01` to match the data plane
default, though as static config it does not follow
`OTEL_TRACES_SAMPLER` overrides, so the two need adjusting together.
Verified on a kind cluster e2e manually. It resolved samplers logged at
startup, a `--trace` resume produced one trace across ateapi, atelet,
and ateom, an unsampled CLI call got picked up server side, Envoy
continued a sampled traceparent while sampling 0 of 30 parentless
requests at the 1% default, and an invalid env value fell back to the
component default.
Gating who may use `--trace` stays a separate follow-up (and a
discussion), and the more sophisticaed tracing policies belongs in a
collector, not in substrate.
Fixes#584
cc. @git286
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Agentgateway is a popular [AAIF](https://aaif.io/) project designed for
agentic systems. It is of particular interest for substrate users as it
was designed specifically to serve the role of an egress proxy for
agents and other AI-adjacent workloads, with strong support for
credential injection, token exchanges, TLS MITM, and other
authentication and authorization schemes.
Agentgateway supports the ext_proc protocol, like Envoy, so we use that
here. There has been some debate in the community around the long term
end state of ext_proc vs other options, but for now this maintains the
status quo.
While Agentgateway can be configured dynamically over XDS, it does not
take the same configuration as Envoy. However, none of the config is
dynamic anyways, so we use just a static configuration file (configmap)
for now to keep things simple.
While we don't have specific per-component/file OWNERS in the project at
the moment, I can informally commit to myself and Eitan maintaining this
integration. Given the goals around velocity in the project we can
commit to either rapidly fixing any issues that may arise or
removing/disabling the integration if this ends up slowing things down.
Phase 1 of #451, micro-VM only. gVisor support will follow in a separate
PR.
## What
Adds a `snapshotsConfig.onResume` block to the ActorTemplate, selecting
per snapshot situation what supplies the guest state at resume: each
field names what is being resumed *from*, the value names the boot
source. `onPause`/`onCommit` remain pure capture scopes (`Full | Data`),
and the `ActorSnapshot` records plain `Full`/`Data` content; the
golden-combine is strictly a restore-time behavior.
```yaml
snapshotsConfig:
onPause: Data
onCommit: Data
onResume:
fromData: Golden # ColdBoot (default) | Golden — Data snapshots resume as
location: gs://… # golden memory + fs delta with the actor's data layered on
```
Under `fromData: Golden`, a data-only restore combines the template's
**golden snapshot** (guest memory + full fs delta) with the actor's
captured durable data — the actor comes back with the golden's warm
state over its own data instead of cold-booting. `fromData` applies to
every resume of a Data snapshot — commit or pause checkpoint. Future
situations (e.g. resuming a Full snapshot across a template upgrade,
#477) get sibling fields under `onResume` rather than overloading this
one.
## How
- **CRD** (`pkg/api/v1alpha1`): new `OnResumeConfig`/`ResumeSource`
types; `fromData: Golden` is CEL-gated to `sandboxClass: microvm`.
- **Control plane** (`ateapi`): resume derives the behavior from the
template's `onResume` configuration — for a Data durable snapshot or a
Data pause checkpoint it resolves the golden snapshot's location and
sends the restore-only `DATA_ON_GOLDEN` wire scope, failing early (with
an actionable error) if the golden snapshot is missing or not Full.
Golden actors always commit `Full` regardless of `onCommit` — their
snapshot is the base the combine needs.
- **Wire APIs** (`ateletpb`/`ateompb`): restore-only
`SNAPSHOT_SCOPE_DATA_ON_GOLDEN`, rejected on checkpoints;
`golden_snapshot_uri_prefix` is a top-level `RestoreRequest` field
because the actor's snapshot may be a local pause checkpoint while the
golden snapshot is always external.
- **atelet**: stages a single combined folder — the actor's files (the
durable-dir tar) win name collisions, the golden snapshot supplies the
rest. External restores download both halves concurrently; local (pause)
restores copy the actor's files from the local checkpoint dir while the
golden's files download in parallel. The golden manifest's pinned
sandbox binaries run the restored guest and are recorded on-node for
later checkpoints.
- **ateom-microvm**: `DATA_ON_GOLDEN` restores through the existing Full
path — cloud-hypervisor relaunches from the golden's guest files while
the durable virtio-fs share serves the actor's re-materialized data.
gVisor rejects the scope (defense-in-depth behind the CRD gate).
## Testing
- Unit: envtest CEL cases, converter tables, atelet request validation
(checkpoint rejection, golden-URI rules incl. local+golden),
combined-download test against a fake object store, resume-workflow
golden-resolution cases for both the durable and pause paths.
- e2e (`suites/demo`): OnGolden lifecycle cases for the commit path, the
pause path, and a two-durable-volume variant (micro-VM lane only),
asserting counters across pause/suspend and the recorded Data content
scope.
- Manually verified on a live GKE cluster: Data commit and Data pause
both resume with the golden guest's memory over the actor's data, on
different pods.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Also Fixes#575
(1) mintCert + MintJWT proto changes
(2) validate that atelet is the one that calls actorIdentity Service.
(3) verify that this atelet is only requesting a cert for an actor on
its node.
(4) verify that the actor is still running
(5) create another substatex509 extension for actor identities
JWTs needs more love, lots as TODO there
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Bring the architecture and threat-model docs in line with the atunnel
ingress path: the router no longer rewrites :authority to a worker pod IP
and forwards over plaintext port 80, it opens an mTLS tunnel to atunnel on
worker port 443, which forwards to the Actor over its private veth.
Drop cmd/atenet/atenet-diagram.png. It predates the ext_proc/ORIGINAL_DST
design and is now wrong in the part that matters most. The README section it
illustrated gains a short accurate note about the upstream hop instead of a
dangling image reference.
Add an e2e suite covering the change end to end: TestActorDirectAccess
asserts that the worker pod's port 80 is no longer a reachable Actor ingress
path (the DNAT rule is gone) and that the same Actor still answers /readyz
through atenet-router over the atunnel mTLS hop. It uses the counter demo as
its fixture, so it only needs --deploy-demo-counter.
internal/e2e/testmain.go picks up ParseSkippedFlags so that `go test` flags
(-run, -v, ...) survive pflag parsing and reach the suite.
Substrate documents how to instrument a service but not how to stand up
the collector it exports to, so pointing it at your own collector meant
reading six manifests.
Covers both topologies: GKE Managed OpenTelemetry as the default on GKE,
with why its gateway Deployment suits Substrate's elastic actor
workload, and a DaemonSet manifest for operators who need per-node
isolation, full config control, or a non-GKE cluster.
Part of #563.
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixed language describing project goals and relationship to Kubernetes
in readme.md, architecture.md and roadmap.md
- changed the top language to focus on project goals instead of
Kubernetes relations
- changed the language describing Kubernetes relationship to describe
value of the Agent Substrate layer and Kubernetes layer
Fixes#564 (Part 1)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements the first part of #564 by extending metric
`atenet.router.route.duration` with singleflight activation state
(`ate.router.resume`) and outcome status labels (`ate.router.outcome`).
All telemetry attribute keys and label value sets are centralized in
`internal/ateattr` to ensure consistency across components.
## Key Changes:
* Added `resumed` boolean to `ResumeActorResponse` in `ateapi.proto`
indicating whether a cold activation workflow occurred
(`!state.WasRunning`).
* Updated `workflow_resume_test.go` to assert `resumed == true` on cold
activations and `resumed == false` for already-running actors.
* Extended `classifyOutcome(err)` in Envoy `ext_proc` to classify errors
into metric labels: `ok`, `cancelled`, `timeout`, `no_capacity`,
`lock_conflict`, `not_found`, `unavailable`, `rate_limited`, and
`resume_error`.
* Added explicit handling for `codes.Unavailable` (`"unavailable"`) and
`codes.ResourceExhausted` / `StatusCode_TooManyRequests`
(`"rate_limited"`).
* Consolidated metric label names on `atenet.router.route.duration`
using standard keys from `internal/ateattr`:
* `ate.template.namespace` (`TemplateNamespaceKey`)
* `ate.template.name` (`TemplateNameKey`)
* `ate.router.outcome` (`RouterOutcomeKey`)
* `ate.router.resume` (`RouterResumeKey`)
* Updated `manifests/ate-install/kind/kustomization.yaml` to export
`atenet-router` OTLP metrics to `opentelemetry-collector`.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
An ActorTemplate could declare only one durable-dir volume, and mount it
into a container only once. That is a gVisor limit, not a general one:
atelet declares the mount to gVisor through a single hardcoded annotation
key ("dev.gvisor.spec.mount.durabledir"), so a second volume would silently
overwrite the first.
The micro-VM runtime has no such constraint — every volume is a
subdirectory of the one writable virtio-fs share, and the snapshot tar
already archives that directory whole, so N volumes cost a subdirectory
each and round-trip through checkpoint/restore untouched. Gate the two
"at most one" CEL rules on sandboxClass so micro-VM templates may declare
several while gVisor keeps its cap, and say so in the messages.
What was actually missing was the volume NAME: ateom received mount paths
alone, so it inferred the single name by listing atelet's directory. Carry
each mount's name on the wire (Container.durable_dir_volume_mounts,
replacing the paths-only field, which is reserved rather than retyped since
atelet and ateom are separate images that can skew across a rollout), and
delete the inference. A container's binds now come from its own mounts, and
the actor-wide "has a durable share" question collapses to a bool.
The counter demo grows an optional --second-file-counter-directory, and a
micro-VM-only e2e case runs the lifecycle matrix against an Actor with two
durable volumes, asserting both counters advance together. gVisor skips it:
the template would be rejected at admission.
The README lists 3 of the 6 demos and 5 of the 9 docs, and its command
tour skips `cmd/ateom-microvm` and `cmd/benchmarking`. The glossary
never defines Atespace, even though it is half of an actor's identity
and has its own API, and it omits SandboxConfig.
Sessions are no longer a concept in Substrate; Actor is the glossary
term. This completes the "s/Session/Actor" TODO that sat at the top of
ateapi.proto, and removes the TODO.
API surface:
service SessionIdentity -> ActorIdentity
MintJWTRequest.session_id -> actor_id
MintJWTResponse.session_jwt -> actor_jwt
MintCertRequest.session_id -> actor_id
MintCertResponse.session_certificates -> actor_certificates
Go packages:
cmd/ateapi/internal/sessionidentity -> actoridentity
cmd/ateapi/internal/sessionidjwt -> actoridjwt
Flags and cluster resources:
--session-id-jwt-pool -> --actor-id-jwt-pool
--session-id-ca-pool -> --actor-id-ca-pool
Secrets, volumes and mount paths renamed to match, in both
manifests/ate-install/ate-api-server.yaml and hack/install-ate.sh
(--create-session-id-ca-pool-secret -> --create-actor-id-ca-pool-secret).
Two credential identity values change with the rename:
JWT issuer https://broker.agentic-substrate-session-id-broker.svc
-> https://broker.agentic-substrate-actor-id-broker.svc
SPIFFE ID spiffe://substrate-session.local/app/../session/..
-> spiffe://substrate-actor.local/app/../actor/..
Tokens and certificates issued before this change will not validate
against the new issuer or trust domain.
BREAKING: the gRPC wire path moves from /ateapi.SessionIdentity/* to
/ateapi.ActorIdentity/*, and the Secrets must be recreated under their
new names before the new ate-api-server rolls out.
Review feedback (bowei): rather than a fully independent flag, --extproc-max-requests now defaults to 0 = derive twice --parked-request-max, floored at Envoy's own default of 1024 — the lot always fits and keeps an equal share of fast-path headroom at any size, including a small or disabled lot. An absolute-percentage derivation like 120% under-provisions the fast path at small lots, which is why the headroom equals the lot instead. Explicit values still override and keep the >= lot validation, so operators who need a specific breaker retain control.
Add --extproc-max-requests (default 2048) and set circuit_breakers.max_requests on the ext_proc cluster from it, replacing the implicit Envoy default and the hand-maintained doc coupling with a guarantee. Every request's header exchange occupies one slot briefly and every parked request holds one for its entire wait, so startup validation enforces extproc-max-requests >= parked-request-max — a breaker below the lot silently truncates it with Envoy-generated 503s that bypass parking.rejected. The default leaves the lot's worth of fast-path headroom (1024 lot / 2048 breaker), so a saturated lot cannot starve requests to already-running actors.
The ext_proc cluster sets no explicit circuit_breakers, so Envoy's default max_requests=1024 applies — and every parked request holds one ext_proc stream, i.e. one active request against that cluster. A 2048 lot was therefore half unreachable: requests 1025+ would be rejected by Envoy itself, with 503s that never reach the lot and never count in parking.rejected. Set the default to 1024 to match, and document the coupling at the constant, at buildCluster, and in the design doc, including what raising the flag beyond 1024 requires (an explicit circuit_breakers.max_requests on the cluster). Also drop Unavailable from the docs' non-retryable examples — it became retryable-while-parked in the previous commit.
The budget clock starts with a flight's first caller; requests de-duplicated onto an in-flight resume share its remaining budget and outcome, so a late joiner can see budget_exhausted after waiting far less than a full budget itself. That trade is inherent to collapsing a hot actor's requests into one control-plane RPC — state it explicitly in the design doc, the flight comment, and the flag help instead of implying a per-request guarantee. Also notes that wait-duration samples record each request's own parked time, so sub-budget budget_exhausted samples are expected under sustained saturation.
A parked request could fail on an ateapi blip despite having budget remaining: retryable() rejected Unavailable, so a control-plane rolling restart failed every in-flight parked request on the single most common transient condition — against the feature's purpose of riding out momentary conditions. Make Unavailable retryable while parking is enabled (the budget still bounds the wait); disabled mode keeps the fail-fast behavior. On budget exhaustion the wrapped Unavailable maps to 503 via the existing path.
- Adopt the ParkedRequest* vocabulary for parking flags and config (bowei's suggestion): --parked-request-budget / --parked-request-max, matching fields and default consts.
- Make the parked-retry backoff configurable: --parked-request-retry-interval/-factor/-jitter, validated at startup (factor >= 1, jitter in [0,1)); the backoff still has no cap and no attempt limit, so the budget alone bounds the wait.
- Resolve the effective parking config once in Run() so the resumer's retry loop and the Envoy ext_proc timeout always agree, even when the budget flag is set non-positive.
- Drop timeline-relative wording from docs and identifiers (failFastResumeBudget, fail-fast behavior).
- Guard the parking-lot counter against going negative, loudly.
- Document exactly when the wait-duration metric is recorded and what each outcome label means.
Review feedback: folding park-budget exhaustion into the generic 'error' outcome hid the one signal operators need from this metric -- that requests waited the full budget and the pool never freed (capacity problem), as opposed to resumes failing outright (fault). The resumer now marks the surfaced capacity error with a wrapper that unwraps to the underlying gRPC status, so the HTTP mapping is unchanged and the wait-duration histogram gains a budget_exhausted outcome label.
Review feedback (thockin, bowei): parking had three overlapping disabled states -- a nil lot, an enabled flag, and an (accidental) maxParked=0 shed-everything mode. Collapse them into one: --parking-max-parked=0 disables parking, the boolean flag and the nil-lot special case are gone, and a zero-capacity lot can no longer reject every request without attempting a resume.
An oversubscribed WorkerPool (2 workers, several actors) that exercises the router parking path: requests to a saturated pool park and retry instead of failing fast, and are served once capacity frees up. Includes load.sh and --deploy-demo-parking / --delete-demo-parking wiring in hack/install-ate.sh.
When a request targets a suspended actor, the router resumes it via the
control plane before routing. A momentarily saturated worker pool makes
ResumeActor return FailedPrecondition ("no free workers available"), which
the router previously turned straight into a 503. In an oversubscribed
system that shortage usually clears within milliseconds as another actor
suspends and frees its worker, so failing fast was wasteful.
Park such requests instead: retry the resume on FailedPrecondition (in
addition to the existing Aborted conflict) until the actor becomes routable
or a bounded wait elapses, capped by a fixed-capacity admission lot that
sheds excess load. On budget expiry the underlying capacity error is
surfaced so the HTTP boundary maps it faithfully. singleflight still
collapses concurrent waiters for the same actor into one resume RPC.
Parking can be disabled to preserve the legacy fail-fast behavior.
- parking.go: parkingLot (bounded, non-blocking admission gate) + config
- resumer.go: retryable predicate + configurable park budget
- extproc.go: admit each request to the lot around the resume call
- metrics.go: parking.active / wait.duration / rejected instruments
- errors.go: parkingFullErr (503 "router at capacity")
- router.go: --parking-enabled / --parking-max-wait / --parking-max-parked
- status.go + dashboard.html: Request Parking status card
- docs/request-parking.md: feature documentation
Doc-only fix, no issue filed — happy to open one if preferred.
`-a/--atespace` became a required flag on `kubectl ate logs actors` in
#458, but the examples in the CLI README and `docs/observability.md`
were never updated. Every documented invocation currently fails:
```
$ go run ./cmd/kubectl-ate logs actors my-actor
Error: required flag(s) "atespace" not set
```
While in those files, two smaller inaccuracies in
`cmd/kubectl-ate/README.md`:
- The logs example says it "follows by default", but `--follow` defaults
to `false`. Split into a one-shot and a `-f` example.
- The global flags table was missing `--context`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
WorkerPool already served /scale, so `kubectl scale` worked, but a HPA
pointed at one did not: the subresource declared no labelSelectorPath,
and HPA needs a selector to find the pods whose metrics it is averaging
or to compute ambiguous selectors.
https://github.com/agent-substrate/substrate/issues/198
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Give the micro-VM counter demo the same durableDir volume the gVisor one
has, and drop TestDurableDirLifecycle's micro-VM skip so the scope matrix
(Full/Full, Data/Full, Data/Data) runs against both runtimes.
TestActorLifecycle no longer special-cases micro-VM to expect a file counter
of -1. That value was the demo failing to write: it counts in
/home/counter/a.txt, which nothing created in the micro-VM rootfs. The volume
mounts there now, so both runtimes assert the same counters.
Also correct the glossary: an ActorTemplate may declare one DurableDir
volume, not several — CEL has enforced that since the feature landed — and
Full scope captures the volumes whether or not the runtime keeps them inside
rootfs (the micro-VM does not).
- `docs/observability.md`: log label keys were wrong. The code emits
`ate.dev/actor_id` and `ate.dev/actor_template_name`; the doc had
`actor_id` and `actor_template`. The Cloud Logging queries would return
no results.
- `docs/dev/best-practices/tracing.md`: container ports entry used
Docker Compose syntax (`"443:443"`). Changed to `containerPort: 443`.
- `docs/dev/valkey-direct-access.md`: the `kubectl exec` command had a
line break inside `--cert`, making it fail on copy-paste.
- `code-of-conduct.md`: linked to Contributor Covenant v1.4. Updated to
v2.1.
The tracing best practices doc points to
cmd/ateapi/ateapi.go:initTracing() as the example, but that
file/function does not exist. Servers now call
internal/serverboot.InitTracing(), used in cmd/ateapi/main.go.
Verified internal/serverboot.InitTracing exists and is called from
cmd/ateapi/main.go:80.
* Add `args` field in addition to `command`.
* Implement logic in atelet to resolve both argv from the actor template
fields and the OCI image config (`entrypoint` and `cmd` fields).
* Update demos to not hardcode command as now we honor the image
default, which makes it not necessary.
Fixes#456
Eitan Yarmush