This change removes the hard-link from templates to worker pools,
allowing pools to be selected using k8s labels.
Both actors and actor templates are augmented with a label selector
field.
When an actor is resumed, we resolve the set of *eligible* worker pools
by evaluating the intersection of both actor and template selectors.
Then we find a free worker from one of them. The overall design is
described in [1].
[1]:
https://github.com/agent-substrate/substrate/issues/47#issuecomment-4712334620
**Updated Original PR Description**
## Summary
This PR adds a small, controlled set of scheduling fields to WorkerPool
so WorkerPool-managed Pods can be placed onto appropriate Kubernetes
nodes.
Added fields:
- `nodeSelector`
- `tolerations`
- `priorityClassName`
- `nodeAffinity`
## Scope
This intentionally does not expose a full `PodTemplateSpec`, and does
not add support for full `affinity`, `podAffinity`, or
`podAntiAffinity`.
Resource requests/limits are intentionally left out of scope for this CL
while the WorkerPool resource model is still being discussed separately.
The goal is to keep the first version small while addressing the
WorkerPool-to-Kubernetes-Node scheduling gap discussed in
[#212](https://github.com/agent-substrate/substrate/issues/212).
`nodeAffinity` is included to support heterogeneous node pools where
`nodeSelector` is too limited, such as requiring or preferring one of
several equivalent accelerator/local-SSD/cache node pools, or expressing
soft preferences during node pool migration.
`priorityClassName` is included to help preserve warm WorkerPool
capacity under cluster pressure. Since WorkerPool pods are the slots
available for actor resume, higher-priority pools can protect
latency-sensitive or interactive actor workloads from displacement by
lower-priority batch workloads, and allow different WorkerPools to
represent different service classes, e.g. interactive vs. batch.
## Related issue
Part of #212.
Related to #47.
## Problem
`go test -v ./...` (the `run-tests` job) intermittently fails during
envtest **setup** — not in any test logic:
- `fork/exec .../kubebuilder-envtest/k8s/1.36.0-linux-amd64/etcd: text
file busy`
- `setup-envtest: ... unable to create file kubectl from archive for
version-platform pair 1.36.0-linux-amd64`
Three packages use envtest (`cmd/ateapi/internal/controlapi`,
`cmd/atecontroller/internal/controllers`, `pkg/api/v1alpha1`). `go test
./...` runs them as separate test binaries **in parallel**, and each
`TestMain` shells out to `setup-envtest use`. On a cold cache they race
to download, extract, and exec the **shared** kubebuilder binaries —
hence `text file busy` (exec'ing etcd while it's still being written)
and the archive-extraction error.
Seen on `main` (e.g. the #259 push run) and on PRs. It only reproduces
on a **cold** cache, which is why it's intermittent and doesn't show up
locally once the cache is warm.
## Fix
Add `internal/envtestbins.BinaryAssetsDir()`, which runs `setup-envtest
use` under a **cross-process file lock** (`flock` on a fixed file under
`os.TempDir()`, shared by the sibling test binaries). The first caller
populates the shared cache while the others wait, then they hit a warm
no-op path lookup. The duplicated setup in all three `TestMain`s now
calls this helper.
The lock scope (user/machine-global) matches the resource it guards —
the kubebuilder binary cache is itself user-global, shared even across
repo checkouts.
## Testing
Reproduced locally by moving the warm cache aside and running the three
packages concurrently: it fails without the lock and passes with it.
Full `go test ./...` green.
Test-only change; no production code touched.
Implements `PAUSED` state for issue #119.
The snapshot files are kept locally on node VM in a separate folder. At
resume time, scheduler uses a node VM hint and picks up a worker from
the same node where files were stored at suspend time.
The local file management solution is temporary and will be replaced
once @msau42 introduces a new component that is supposed to manage files
on the node VM.
- [X] Tests pass
- [X] Manual tests with counter demo
```
>kubectl ate create actor my-counter-1 --template ate-demo-counter/counter
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 1
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 2
>kubectl ate pause actor my-counter-1 -o json
> curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 3
```
### Breaking change
This PR introduces a breaking change in the Actor proto. All existing
actor needs to be recreated, prior testing PAUSE functionality.
Gives an actor a reliable way to learn its own ID without parsing the
Host header. atelet populates a per-actor identity directory (currently
the single file actor-id) and bind-mounts it read-only at /run/ate into
each app container; the pause container gets none.
- A directory rather than a single-file mount, per the discussion on
#178, so future late-bound data can land as sibling files without
changing the mount shape.
- The bundle is regenerated on every resume and external bind mounts are
re-attached from the restore-time config.json, so the value stays
correct per-actor through a golden-snapshot restore; an env var (or a
file baked into the image) would be frozen at the golden actor's ID,
since it lives in the checkpointed process memory.
- actor-id is written atomically (temp file, fsync, rename, parent
fsync): the directory is visible to a possibly-running actor, so a
reader must never observe a torn value.
- The identity dir is cleared and recreated with the other per-actor
dirs in resetActorDirs.
- The in-rootfs mount point is created via os.Root, so a symlink planted
in the image cannot redirect the write outside the extracted rootfs.
Documented for actor authors in docs/api-guide.md: read it fresh on use;
raw ID, no trailing newline.
This follows the convention documented here:
AGENTS.md
```
**Where to put new Go code — quick rules:**
| Only used by one binary | `cmd/<binary>/internal/<pkg>` |
```
See the discussion here:
https://github.com/agent-substrate/substrate/issues/49
I just noticed that the directory structure is pretty deep. I want to
make this less deep before we get too far into the project:
* cmd/atenet/internal/app << this directory level is not needed. get rid of it.
* cmd/atenet/internal/router/* move everything but the cobra flags to internal/router
* cmd/atenet/internal/* just has the cobra flags definitions
This is moving files -- no logic change.
What was happening:
1. runsc checkpoint pause succeeded.
2. After that, substrate tried to run runsc state and runsc delete for the app container.
3. But after checkpointing the root sandbox container, the runsc control server was no longer usable,
so those post-checkpoint commands failed with connection refused.
4. That made CheckpointWorkload return an error even though the actual checkpoint had already
succeeded.
5. The ActorTemplate stayed stuck in WaitGoldenActor.
The change removes the post-checkpoint state/delete calls from ateom-gvisor. After checkpoint
succeeds, it only cleans up the actor network and returns success. This matches the existing contract
in the code: atelet owns resetting the actor directories after checkpoint/upload, and it already
calls resetActorDirs(...).
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Fixes#202
atelet's gRPC server runs on an unauthenticated hostPort
(`manifests/ate-install/atelet.yaml`), so its
`Run`/`Restore`/`Checkpoint`
RPCs must treat every request field as untrusted. Several of those
fields
are turned into host filesystem paths and then deleted, created,
written,
or executed, none of them validated first. A caller that can reach the
port can smuggle a path separator or `..` through them and make atelet
operate outside the intended actor tree.
### Unvalidated inputs (each an independent escape route)
- **`actor_template_namespace` / `name` / `id`** are concatenated by
`ateompath.ActorPath` into the per-actor dir that `resetActorDirs` runs
`os.RemoveAll` / `os.MkdirAll` on.
- **`target_ateom_uid`** builds the netns path
(`/run/netns/ateom:<uid>`)
and the ateom control socket (`.../ateoms/<uid>/ateom.sock`).
- **`spec.containers[].name`** becomes an OCI bundle path component
(`os.RemoveAll` / `MkdirAll` / `config.json` write).
- **`runsc.sha256_hash`** builds `static-files/runsc-<hash>`. The
cache-hit
`os.Stat` early return hands that path to ateom to **execute** before
the
existing hex check runs, so an escaped path could execute an arbitrary
on-disk binary as the sandbox runtime.
### Where the validators live
The field rules are exported from **`internal/resources`**
(`ValidateActorRef`, `ValidateAteomUID`, `ValidateContainerNames`,
`ValidateRunscHash`, `ValidateSnapshotURIPrefix`), alongside the
existing
`ValidateActorID`. Every one of these values originates upstream in
user-supplied API/CRD data (the runsc hash and snapshot location come
from
the `ActorTemplate` spec), so the API server and controller can apply
the
same rules at admission time instead of failing at actor-activation
time.
`ValidateContainerNames` takes names rather than the atelet proto so
`resources` stays proto-free. Unifying `ValidateActorID` with
`ValidateActorRef` is deferred: it changes ate-api-server's
client-facing
error responses.
The per-RPC glue stays in `cmd/atelet`: each of
`Run`/`Checkpoint`/`Restore`
calls its own typed validator (`validateRunRequest` etc.) as the first
statement at the boundary, before any path is constructed.
### What is validated
- namespace and actor ID as DNS-1123 labels, template as a DNS-1123
subdomain, matching how these are named; additionally the combined
`<ns>:<tmpl>:<id>` path component is capped at 255 bytes, since
individually valid DNS names can join to 381 bytes, over common
filesystems' filename limit;
- `target_ateom_uid` as a DNS-1123 label (pod UIDs are UUIDs, which
qualify);
- each container name as a DNS-1123 label, additionally rejecting the
reserved name `pause` (collides with the sandbox-infra bundle and races
its concurrent writer) and duplicate names (same bundle path, mutual
corruption);
- `runsc.sha256_hash` as exactly 64 hex chars, checked in `fetchRunsc`
before the path is built and before the cache-hit early return;
- `snapshot_uri_prefix` (Checkpoint/Restore) as a parseable URI naming a
bucket, with query/fragment/userinfo rejected: object names are appended
by concatenation and the storage layer re-parses using only host and
path, so those components would silently redirect the upload/download to
a different object. The scheme is deliberately not restricted; that is
storage-backend policy.
Validation failures return `codes.InvalidArgument` with the reason.
Previously the interceptor surfaced them as `codes.Internal` with the
message masked as "internal server error".
DNS-1123 is a deliberate fit. These values already *are* Kubernetes
names,
so every legitimate value passes, while the charset forbids `/`, `.`
(for
labels), and `..`, which is exactly what enables traversal. For a
cooperating caller the change is behavior-neutral: inputs newly rejected
previously failed later or misbehaved silently.
### Testing
- Table tests for each validator in `internal/resources`, targeting the
exported functions (valid values, separators, `..`, empty, case, length,
the label-vs-subdomain distinction, the combined-length cap, the
`pause`/duplicate rules, and the URI component rejections).
- Per-request validator tests in `cmd/atelet` pin which rules each RPC
enforces, one table per validator.
- `TestRPCBoundariesReject` exercises `Run`/`Checkpoint`/`Restore`
directly
and asserts a traversal value is rejected with `InvalidArgument`, which
guards against future removal or reordering of the validation call.
- `TestFetchRunscRejectsBadHash` plants a real cache file at the path an
invalid hash resolves to and asserts `fetchRunsc` still errors. It
passes
with the current code and fails if validation is moved after the
`os.Stat` early return, so it proves the ordering rather than just that
validation runs.
- [x] Tests pass
- [x] No documentation changes needed
A couple of TODOs/Discussions which I think are important to have based
on these changes.
1. Do we need a caching client for this to avoid calling the apiserver
every time?
2. Where/how do we want to resolve secrets generally? I think there are
good arguments to be made for the control-plane OR the "node" component.
In my mind this also related to:
https://github.com/agent-substrate/substrate/issues/18Fixes#15
> It's a good idea to open an issue first for discussion.
- [ x ] Tests pass
- [ x ] Appropriate changes to documentation are included in the PR
It was using a wrong name `Actor Namespace` before, which doesn't exist
yet.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR introduces a new OpenTelemetry histogram
`"atelet.snapshot.size"` from `atelet` over the existing
OTLP path. It records the uncompressed size in bytes of each gVisor
snapshot image written during checkpoint -- `checkpoint.img` ,
`pages.img` and `pages_meta.img` -- tagged by image `kind` and
`ActorTemplate`. This surfaces how large actor snapshots are and which
`ActorTemplate`s are expensive to checkpoint/restore.
Add a well-defined dashboard for snapshot metrics that contains 3
charts.
- snapshot image size p99 by `ActorTemplate`
- snapshot image size p50/p95/p99 (all kinds)
- snapshot QPS by `ActorTemplate`
<img width="996" height="717" alt="image"
src="https://github.com/user-attachments/assets/99bbab46-37dc-476e-bfd9-cdffeb174b4b"
/>
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR introduces a new OpenTelemetry histogram
`"atenet.router.route.duration"` from `atenet-router` over the existing
OTLP path. It measures substrate overhead -- from envoy receiving a
request to envoy sending the request to the the resolved worker
endpoint, excluding actor compute and the response. This is the
`Substrate E2E latency` that under our control.
Add a well-defined dashboard for E2E metrics that contains 6 charts.
- Substrate routing latency — P50/P95/P99
- Substrate routing latency P99 - by stages
- Substrate routing latency P99 — by ActorTemplate
- Substrate routing QPS - by Status
- Substrate E2E (full round-trip) Latency — P50/P95/P99
- Substrate E2E (full round-trip) QPS — by Response Code
<img width="1601" height="894" alt="image"
src="https://github.com/user-attachments/assets/be6fc572-dfd8-4ef0-a0a9-a77f85f2ecf8"
/>
Fixes b/508613998
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
# Implement Pagination and Pipelining for ListActors
## Overview
**Fixes:** #153
This PR introduces robust, memory-safe pagination to the `ListActors`
operation and optimizes the Valkey retrieval layer.
Previously, `ListActors` performed an unbounded parallel scatter-gather
across all Valkey cluster shards, pulling all actors into memory at
once. At scale (millions/billions of actors), this would cause OOM
errors and severe latency.
This update replaces the concurrent scatter-gather with a deterministic,
sequential shard scanning approach, adhering to
[AIP-158](https://google.aip.dev/158) pagination standards.
## Changes Included
- **gRPC API (`ateapi.proto`)**: Added `page_size`, `page_token`, and
`next_page_token` fields.
- **Client (`kubectl-ate`)**: Updated to seamlessly loop through
`next_page_token` values until all actors are fetched, presenting a
continuous stream to the user without blowing up memory on a single
request.
- **API Handler (`controlapi`)**: Bounded requests with a max
`page_size` limit of `1000`.
- **Persistence (`ateredis`)**:
- Implemented `listActorsPageToken` to safely serialize and encode
cross-shard cursors.
- Replaced `ForEachMaster` concurrent execution with deterministic
sequential iteration.
- Implemented Valkey Pipeline batching (`Pipelined`) for `GET` requests
to eliminate N+1 network latency.
- **Testing**: Added `TestListActors_Pagination` at both the integration
level (`functional_test.go`) and data layer (`ateredis_test.go`).
Refactored sorting to use `cmpopts.SortSlices`.
## Design Decisions
During the design phase, several explicit trade-offs were made to
balance performance with API simplicity:
1. **Opaque Tokens (AIP-158):** The `page_token` is strictly an opaque
string at the `ateapi` layer. The client (`kubectl-ate`) is completely
unaware that it contains base64-encoded JSON mapping to `{"shard",
"cursor"}`. This keeps our API database-agnostic.
2. **Empty Pages & Client-Side Looping:** The persistence layer is kept
simple and "dumb". If a shard scan returns 0 keys but the cursor is not
`0`, we return an empty page. We rely on the client `kubectl-ate` to
`for` loop until `next_page_token` is completely exhausted. Update: I
have server side looping across shards until pagesize is met to limit
round trips due to high shard count or low actor count on distributed
shards.
3. **"Soft" Consistency (Missing/Duplicate Data):** `ListActors` is an
infrequent operations command, not on the critical path. Taking a global
cluster lock for iteration would severely degrade performance. We accept
eventual consistency here: if actors migrate shards during an active
scan, they might be duplicated or missed.
4. **Topology Flux Errors:** If a client passes a valid token pointing
to a shard address that no longer exists (due to a cluster resharding
mid-scan), the API simply returns an `Aborted` error. We do not attempt
complex recovery; the client must just restart the `ListActors` command.
5. **Bounded Page Size:** Requests are currently capped at a maximum of
`1000` to prevent single massive payloads. We can revisit/change the max
later.
NOTE: Actor filtering will be implemented later.
## Architecture Flow (Valkey)
Note: If there are future plugin-db models they will need to handle the
cusor/pages at their own implementation layer per the store.go
interface.
The system now seamlessly iterates across shards, handing off cursors
safely between nodes:
```mermaid
sequenceDiagram
participant CLI as kubectl-ate
participant API as ateapi
participant Valkey as Valkey Cluster (Multiple Shards)
CLI->>API: ListActors(page_size=1000, page_token="")
Note over API: Request 1: Empty Token
API->>API: Decode page_token (empty -> start fresh)
API->>Valkey: Fetch all Master Nodes
Valkey-->>API: [Master C, Master A, Master B]
API->>API: Sort Masters alphabetically by IP/Port
Note over API: Sorted: [Master A, Master B, Master C]
API->>Valkey: SCAN (Master A, cursor=0, count=1000)
Valkey-->>API: keys=[k1..k300], next_cursor=45
Note over API: OPTIMIZATION: Pipelining avoids N+1 problem
API->>Valkey: PIPELINE GET (k1..k300) directly on Master A
Valkey-->>API: values=[v1..v300]
API->>API: Decode Protobufs
API->>API: Encode page_token={"shard": "Master A", "cursor": 45}
API-->>CLI: 300 Actors, next_page_token="base64(eyJzaGFyZ..."
%% SECOND REQUEST
CLI->>API: ListActors(page_size=1000, page_token="base64(...)")
Note over API: Request 2: Resuming Mid-Shard
API->>API: Decode page_token -> Shard="Master A", Cursor=45
API->>API: Fast-forward directly to Master A
API->>Valkey: SCAN (Master A, cursor=45, count=1000)
Valkey-->>API: keys=[k301..k400], next_cursor=0
API->>Valkey: PIPELINE GET (k301..k400) directly on Master A
Valkey-->>API: values=[v301..v400]
Note over API: Shard A is empty. Token points to Shard B!
API->>API: Encode page_token={"shard": "Master B", "cursor": 0}
API-->>CLI: 100 Actors, next_page_token="base64(eyJzaGFyZ..."
%% THIRD REQUEST
CLI->>API: ListActors(page_size=1000, page_token="base64(...)")
Note over API: Request 3: Starting New Shard
API->>API: Decode page_token -> Shard="Master B", Cursor=0
Note over API: Continues scanning Master B...
```
## Testing:
```
> ./bin/kubectl-ate get actors
NAMESPACE TEMPLATE ID STATUS ATEOM POD ATEOM IP VERSION
ate-demo-counter counter 61b478cc-2e44-4ec7-ac6c-1af3f385de86 STATUS_SUSPENDED <none> 5
```
```
~/go/bin/grpcurl -insecure -d '{"page_size": 2}' localhost:9090 ateapi.Control/ListActors
{
"actors": [
{
"actorId": "test-counter-4",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjE5OjYzNzkiLCJjdXJzb3IiOjI1MjU1fQ=="
}
```
<details>
```
bash -c '
export NEXT_TOKEN=""
while :; do
echo "--- Fetching Page ---"
# Fetch the data and store the response
RESPONSE=$(~/go/bin/grpcurl -insecure -d "{\"page_size\": 2, \"page_token\": \"$NEXT_TOKEN\"}" localhost:9090 ateapi.Control/ListActors)
# Print the response to the terminal
echo "$RESPONSE"
# Extract the token without needing jq
NEXT_TOKEN=$(echo "$RESPONSE" | grep -o "\"nextPageToken\": \"[^\"]*\"" | cut -d"\"" -f4)
# If no token is found, we reached the end
if [ -z "$NEXT_TOKEN" ]; then
echo "--- Reached the end! ---"
break
fi
echo ""
read -p "Press Enter to fetch the next page using the new token..."
done
'
```
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-4",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjE5OjYzNzkiLCJjdXJzb3IiOjI1MjU1fQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjB9"
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-1",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjQ1NjAwfQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-5",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjIxOjYzNzkiLCJjdXJzb3IiOjYyMDg2fQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"nextPageToken": "eyJzaGFyZCI6IjEwLjI0NC4wLjI1OjYzNzkiLCJjdXJzb3IiOjB9"
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-2",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
}
],
"nextPageToken":
"eyJzaGFyZCI6IjEwLjI0NC4wLjI1OjYzNzkiLCJjdXJzb3IiOjE2OTkzfQ=="
}
Press Enter to fetch the next page using the new token...
--- Fetching Page ---
{
"actors": [
{
"actorId": "test-counter-3",
"version": "1",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED"
},
{
"actorId": "61b478cc-2e44-4ec7-ac6c-1af3f385de86",
"version": "5",
"actorTemplateNamespace": "ate-demo-counter",
"actorTemplateName": "counter",
"status": "STATUS_SUSPENDED",
"lastSnapshot":
"gs://ate-snapshots/ate-demo-counter/61b478cc-2e44-4ec7-ac6c-1af3f385de86/2026-06-03T03:17:34Z-UMYVUZOWW6LDPS5X5FSG3KC4VV"
}
]
}
--- Reached the end! ---
</details>
Pod names alone can easily exceed the max length (253 > 107).
Add a symlink with the full pod name for debugging.
Follow-up #92
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
---
TODO: I'm not sure about the symlink. It's annoying mapping pod =>
hash(pod) otherwise though. We could put a file inside the directory
with the pod name instead, but I think we should probably put the full
pod name _somewhere_ in the structure so it's easier to inspect these.
These flags allow the guest workload to begin executing sooner during
restore, because gVisor loads only the necessary memory pages. Other
pages are lazily loaded in the background or as they are referenced.
This mode slightly complicates our state management, because it's more
difficult to tell when gVisor is done with the checkpoint files passed
to `runsc restore`.
We can work around this by ensuring that we always pass different
directories to `runsc checkpoint` and `runsc restore`.
`runsc checkpoint` properly handles the case where the gVisor sentry has
not finished background loading of pages from a previous `runsc
restore`.
Adds internal/version with three vars (Version, Commit, BuildDate) set
via -ldflags -X at build time. When unset (e.g. plain `go build`
without LDFLAGS), runtime/debug.ReadBuildInfo() populates them from the
embedded vcs info so `go install`, `go run`, and ko-built images
(buildvcs is on by default) still produce something meaningful.
Wired through:
- Makefile: VERSION / COMMIT / BUILD_DATE resolved from `git describe`
/ `git rev-parse` / `date`. LDFLAGS plumbed into every go-build and
ko-build target (build-images, build-atectl, build-atenet,
build-demos).
- hack/install-ate.sh: untouched here — ko apply / resolve don't
accept --ldflags, and ko's buildvcs embedding gives Commit +
BuildDate automatically through runtime/debug.ReadBuildInfo() in
the version package's init() fallback.
All six binaries respond to `--version`:
- kubectl-ate and atenet (cobra): rootCmd.Version = version.String()
- ateapi, atelet, ateom-gvisor, podcertcontroller (flag): explicit
`-version` bool that prints and exits.
Sample output:
kubectl-ate version 3b0a0d5 commit=3b0a0d5c2... \
built=2026-05-26T14:16:53Z darwin/arm64
Closes the "kubectl-ate --version has nothing to print" gap; gives ops
a way to tell at a glance which commit a deployed substrate component
came from.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Follow-up to
https://github.com/agent-substrate/substrate/pull/101#issuecomment-4597137488
This isn't a regression from the previous PR, but I noticed this bug
while reviewing.
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Resolves the three TODO comments in `cmd/atelet/oci.go` that called for
a constrained filesystem to prevent path traversal, symlink escape, and
hardlink escape during OCI tar extraction.
Uses `os.Root` (Go 1.24+) to confine all file operations to the rootfs
directory. Adds `validateTarName()` as defence in depth.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Address review feedback from @bowei: return error interface rather than
the concrete *reqError pointer so callers cannot accidentally fall into
the typed-nil pitfall. Tests use errors.As to access struct fields.
The Envoy ext_proc handler previously wrapped resume errors with %w,
leaking the gRPC encoding ("rpc error: code = ... desc = ...") into the
response body, and routed every non-NotFound failure through 500. It
also returned a generic "not found" body that did not identify which
resource was missing.
This change introduces a small mapping layer that translates gRPC status
codes into appropriate HTTP status codes with short, client-safe bodies:
NotFound -> 404
FailedPrecondition -> 503 (desc preserved, e.g. "no free workers
available", which is actionable and not
security-sensitive)
Unavailable -> 503
DeadlineExceeded -> 504
PermissionDenied -> 403
Unauthenticated -> 401
ResourceExhausted -> 429
other -> 500 (no detail leak)
The original error is preserved via Unwrap so existing slog.ErrorContext
logging keeps full fidelity.
Layering: ActorResumer no longer translates codes.NotFound into a
router-layer sentinel; gRPC errors propagate untouched so the HTTP
boundary owns the mapping.
## Symptom
When `<namespace>:<pod-name>` exceeds 107 bytes, `ateom-gvisor` exits
with `bind: invalid argument` and the controller surfaces a generic
`DeadlineExceeded`.
## Cause
Linux limits unix-socket paths to 107 bytes (`sockaddr_un.sun_path`). A
pod's ReplicaSet hash can jiggle between 8 and 10 chars across rollouts,
so a demo that fits at 8 silently breaks at 10 with no clear error.
## Fix
Add `ValidateAteomSocketPath` (with unit tests) and call it from
`ateom-gvisor` before the bind, so the operator sees the real cause in
the pod log instead of a kernel EINVAL.
Today, if a worker pod that hosts an active actor is forcibly destroyed,
substrate does not migrate the actor. The actor's AteomPodName still
points at the dead pod and the router times out forwarding to a dead IP.
WorkerPoolSyncer's pod-informer DeleteFunc and soft-delete branch now
check whether the worker being removed is bound to an actor, and if so,
reset that actor to `STATUS_SUSPENDED` before the worker row goes away.
The new helper releaseActorOnDeadWorker reads the actor, only acts if
the actor still claims this worker (a concurrent SuspendActor or
DeleteActor that already advanced state is respected), clears the
pod-binding fields and InProgressSnapshot, preserves LastSnapshot, and
writes via version-checked UpdateActor. On ErrPersistenceRetry we drop
the attempt and let the next informer event retry — no separate lock, no
in-handler retry budget.
`STATUS_SUSPENDED` is reused rather than introducing a new state. The
post-orphan invariant is identical and reusing keeps the proto, printer,
dialer, and router contracts unchanged. The next request through atenet
triggers an implicit resume; findFreeWorker picks any free worker in the
pool; LastSnapshot is restored.
Test plan: `TestSyncer_DeleteBoundWorker_ClearsActor` covers RUNNING
actor → pod delete → SUSPENDED with cleared bind fields,
InProgressSnapshot dropped, LastSnapshot preserved. All existing tests
in `cmd/ateapi/internal/controlapi/` still pass.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Co-authored-by: Julian Gutierrez Oschmann <juliangut@google.com>
This PR refactors `ateom-gvisor` and `kubectl-ate` to adopt a
flat-merged logging architecture, eliminating server-side and
client-side field guessing:
1. **Cloud Logging**: `ateom-gvisor` parses actor JSON, injects actor
labels/time, and outputs the flat map directly. GKE Cloud Logging
built-in rules parse native severity/message fields automatically.
2. **CLI Logging**: Removed the `--raw` flag. `kubectl ate logs` strips
Substrate metadata and outputs clean, consistent JSON lines with the
`time` field prepended first.
<img width="3235" height="306" alt="image"
src="https://github.com/user-attachments/assets/4a6ca291-db8d-45dc-b840-43efa74739f0"
/>
3. **Documentation**: Updated `docs/observability.md`
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Three related cleanups in cmd/{ateapi,atelet,ateom-gvisor}/main.go:
1. atelet Run/Checkpoint/Restore handlers shared three blocks of
cut-and-paste (fetchRunsc + MkdirAll, errgroup over pause + application
containers calling prepareOCIDirectory, dial-ateom + NewAteomClient, the
ateletpb -> ateompb WorkloadSpec projection). Factored into
fetchRunscAndPrep, prepareOCIBundles, dialAteom, buildAteomWorkloadSpec,
plus a uploadIfExists for the optional pages.img / pages_meta.img upload
in Checkpoint.
2. ateapi main() was a 242-line wall of inlined startup steps;
decomposed into named helpers (loadFlagsFromEnv, logFlagValues,
connectRedis with buildRedisTLSConfig + pingRedisWithRetries,
newKubeClients, buildServerCreds) so the top-level flow reads as
orchestration.
3. ateapi, atelet, and ateom-gvisor each carried their own ~30-line
initTracing + ~35-line initMetrics + slog/metrics-server boilerplate
that differed only in ServiceName, sampler choice, and whether the OTLP
exporter was wired. Lifted to a new internal/serverboot package exposing
InitLogger, InitTracing (with TracingOptions.NoExporter for ateom-gvisor
which loses egress once eth0 moves into the gvisor netns), InitMetrics,
Fatal, ShutdownProvider, and StartMetricsServer.
No behaviour change. Verified with go vet, GOOS=linux go build, and go
test ./cmd/ateapi/internal/...
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
ateom-gvisor moves the pod's eth0 interface into the actor's interior
netns before invoking runsc run/restore. The current code assumes eth0
is in the pod netns at the start of every RunWorkload/RestoreWorkload.
That assumption breaks if a previous call failed mid-flight: eth0 is
left stranded in the interior netns, the next call errors with
"eth0: Link not found", and the worker pod is stuck until restart.
Add ensureEth0InPodNetns that moves eth0 back to the pod netns if it
is in the interior, and is a no-op otherwise. Call it at the top of
both entry points to recover from a previously-aborted call, and via
a deferred handler after the eth0-into-interior move to roll back on
errors in the current call. The success path is unchanged.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Adds golangci-lint v2.12.2 using the per-tool-module pattern under
hack/tools/golangci-lint/. The verifier at hack/verify/golangci-lint.sh
is auto-discovered by hack/verify-all.sh, so the existing CI verify
step picks it up with no workflow YAML change.
Ships .golangci.yaml with the standard linter set, a make lint
target, and a verify wrapper that pins GOOS=linux so ateom-gvisor's
netlink references typecheck on macOS the same way they do on CI.
Adjustments to keep the first run clean: two dead-code deletions
((*runsc).cmdList, MemoryPullCache.gcsClient), two SA9003 empty
branches replaced with slog.Error / t.Logf, one S1008 return-bool
simplification, and inline //nolint for one intentional best-effort
ReleaseLock plus four audit-tracked SA1019 deprecations.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
This PR adds live-streaming log support (`--follow/-f`) to `kubectl-ate
logs actors`. Unlike static container logs, this implementation is
actor-aware and resilient to virtual actor suspensions and migrations:
Suspend Lifecycle:
<img width="3700" height="1856" alt="image"
src="https://github.com/user-attachments/assets/38568108-8229-40b0-afc7-d1d5ddc817e1"
/>
Resume Lifecycle:
<img width="3696" height="1530" alt="image"
src="https://github.com/user-attachments/assets/cf934608-62fe-4fb8-bd2c-5e6e7a0af776"
/>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Julian Gutierrez Oschmann