## Symptom
When `<namespace>:<pod-name>` exceeds 107 bytes, `ateom-gvisor` exits
with `bind: invalid argument` and the controller surfaces a generic
`DeadlineExceeded`.
## Cause
Linux limits unix-socket paths to 107 bytes (`sockaddr_un.sun_path`). A
pod's ReplicaSet hash can jiggle between 8 and 10 chars across rollouts,
so a demo that fits at 8 silently breaks at 10 with no clear error.
## Fix
Add `ValidateAteomSocketPath` (with unit tests) and call it from
`ateom-gvisor` before the bind, so the operator sees the real cause in
the pod log instead of a kernel EINVAL.
Today, if a worker pod that hosts an active actor is forcibly destroyed,
substrate does not migrate the actor. The actor's AteomPodName still
points at the dead pod and the router times out forwarding to a dead IP.
WorkerPoolSyncer's pod-informer DeleteFunc and soft-delete branch now
check whether the worker being removed is bound to an actor, and if so,
reset that actor to `STATUS_SUSPENDED` before the worker row goes away.
The new helper releaseActorOnDeadWorker reads the actor, only acts if
the actor still claims this worker (a concurrent SuspendActor or
DeleteActor that already advanced state is respected), clears the
pod-binding fields and InProgressSnapshot, preserves LastSnapshot, and
writes via version-checked UpdateActor. On ErrPersistenceRetry we drop
the attempt and let the next informer event retry — no separate lock, no
in-handler retry budget.
`STATUS_SUSPENDED` is reused rather than introducing a new state. The
post-orphan invariant is identical and reusing keeps the proto, printer,
dialer, and router contracts unchanged. The next request through atenet
triggers an implicit resume; findFreeWorker picks any free worker in the
pool; LastSnapshot is restored.
Test plan: `TestSyncer_DeleteBoundWorker_ClearsActor` covers RUNNING
actor → pod delete → SUSPENDED with cleared bind fields,
InProgressSnapshot dropped, LastSnapshot preserved. All existing tests
in `cmd/ateapi/internal/controlapi/` still pass.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Co-authored-by: Julian Gutierrez Oschmann <juliangut@google.com>
This PR refactors `ateom-gvisor` and `kubectl-ate` to adopt a
flat-merged logging architecture, eliminating server-side and
client-side field guessing:
1. **Cloud Logging**: `ateom-gvisor` parses actor JSON, injects actor
labels/time, and outputs the flat map directly. GKE Cloud Logging
built-in rules parse native severity/message fields automatically.
2. **CLI Logging**: Removed the `--raw` flag. `kubectl ate logs` strips
Substrate metadata and outputs clean, consistent JSON lines with the
`time` field prepended first.
<img width="3235" height="306" alt="image"
src="https://github.com/user-attachments/assets/4a6ca291-db8d-45dc-b840-43efa74739f0"
/>
3. **Documentation**: Updated `docs/observability.md`
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Three related cleanups in cmd/{ateapi,atelet,ateom-gvisor}/main.go:
1. atelet Run/Checkpoint/Restore handlers shared three blocks of
cut-and-paste (fetchRunsc + MkdirAll, errgroup over pause + application
containers calling prepareOCIDirectory, dial-ateom + NewAteomClient, the
ateletpb -> ateompb WorkloadSpec projection). Factored into
fetchRunscAndPrep, prepareOCIBundles, dialAteom, buildAteomWorkloadSpec,
plus a uploadIfExists for the optional pages.img / pages_meta.img upload
in Checkpoint.
2. ateapi main() was a 242-line wall of inlined startup steps;
decomposed into named helpers (loadFlagsFromEnv, logFlagValues,
connectRedis with buildRedisTLSConfig + pingRedisWithRetries,
newKubeClients, buildServerCreds) so the top-level flow reads as
orchestration.
3. ateapi, atelet, and ateom-gvisor each carried their own ~30-line
initTracing + ~35-line initMetrics + slog/metrics-server boilerplate
that differed only in ServiceName, sampler choice, and whether the OTLP
exporter was wired. Lifted to a new internal/serverboot package exposing
InitLogger, InitTracing (with TracingOptions.NoExporter for ateom-gvisor
which loses egress once eth0 moves into the gvisor netns), InitMetrics,
Fatal, ShutdownProvider, and StartMetricsServer.
No behaviour change. Verified with go vet, GOOS=linux go build, and go
test ./cmd/ateapi/internal/...
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
ateom-gvisor moves the pod's eth0 interface into the actor's interior
netns before invoking runsc run/restore. The current code assumes eth0
is in the pod netns at the start of every RunWorkload/RestoreWorkload.
That assumption breaks if a previous call failed mid-flight: eth0 is
left stranded in the interior netns, the next call errors with
"eth0: Link not found", and the worker pod is stuck until restart.
Add ensureEth0InPodNetns that moves eth0 back to the pod netns if it
is in the interior, and is a no-op otherwise. Call it at the top of
both entry points to recover from a previously-aborted call, and via
a deferred handler after the eth0-into-interior move to roll back on
errors in the current call. The success path is unchanged.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Adds golangci-lint v2.12.2 using the per-tool-module pattern under
hack/tools/golangci-lint/. The verifier at hack/verify/golangci-lint.sh
is auto-discovered by hack/verify-all.sh, so the existing CI verify
step picks it up with no workflow YAML change.
Ships .golangci.yaml with the standard linter set, a make lint
target, and a verify wrapper that pins GOOS=linux so ateom-gvisor's
netlink references typecheck on macOS the same way they do on CI.
Adjustments to keep the first run clean: two dead-code deletions
((*runsc).cmdList, MemoryPullCache.gcsClient), two SA9003 empty
branches replaced with slog.Error / t.Logf, one S1008 return-bool
simplification, and inline //nolint for one intentional best-effort
ReleaseLock plus four audit-tracked SA1019 deprecations.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
This PR adds live-streaming log support (`--follow/-f`) to `kubectl-ate
logs actors`. Unlike static container logs, this implementation is
actor-aware and resilient to virtual actor suspensions and migrations:
Suspend Lifecycle:
<img width="3700" height="1856" alt="image"
src="https://github.com/user-attachments/assets/38568108-8229-40b0-afc7-d1d5ddc817e1"
/>
Resume Lifecycle:
<img width="3696" height="1530" alt="image"
src="https://github.com/user-attachments/assets/cf934608-62fe-4fb8-bd2c-5e6e7a0af776"
/>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
CNIs that assign a /32 pod IP with a link-scope route to a gateway in
the same prefix (Cilium being the headline case) save the default
route before the link-scope route in netlink.RouteList output.
restoreLink iterated in that order, causing the kernel's
fib_check_nh_v4_gw to reject the default route with ENETUNREACH
("network is unreachable") because the route to the gateway had not
been installed yet.
Stable-sort the saved routes so no-gateway routes are installed
first. No-op on CNIs where the gateway is covered by a connected
subnet route already (kindnet, classic kubenet) because the existing
order already satisfies the predicate.
This commit adds a small log handler that reads the trace id out of the
context, if present, and includes it as a log attribute. Addionally,
ateom is integrated with tracing (although it has no exporter because it
has no network connectivity).
This allows easy correlation of logs from a single parent request across
ate-api-server, atelet, and ateom.
I have also fixed the potential for log tearing from the ateom runsc log
exporter by having it reuse the same slog.Handler as the main program.
- Add a "Local (kind)" subsection alongside the existing GKE tracing
recipe.
- Add a note explaining why `kubectl get actor` and `kubectl get worker`
return nothing (they live in valkey, not as CRDs).
- Add output-column glossaries for `kubectl ate get actor` and `kubectl
ate get worker`.
- Add a "Logs" section covering the `kubectl ate logs actors <id>` form.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
- Add a trivial /readyz handler (which will only be served once we
finish initializing)
- We use the metrics handler because we already expose an HTTP port
there and there should be no conflict
- Add a readiness probe to the install manifest
- Update install script to wait for readiness on core components
Improves reliability for the demo flow, including the test flake seen in
#2
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This is the initial release of the Agent Substrate.
Agent substrate is a system built on top of Kubernetes which manages agent-like
workloads to achieve higher scale and efficiency than Kubernetes alone can
offer, with lower latency. It builds on top of Kubernetes features like
Pods and Pod autoscaling, but takes the Kubernetes control-plane out of the
critical path to achieve lower latency.
It can run on any Kubernetes cluster and does not inhibit “regular” use of
Kubernetes in any way. Kubernetes provides the infrastructure provisioning and
management for all types of workloads, while Agent Substrate provides
agent-specific scheduling and control.
At its core, Agent Substrate maps a larger set of “actors” (applications such
as agents) onto a smaller set of ready “workers” (Kubernetes Pods), relying on
the fact that agent-like applications tend to be idle most of the time to
achieve heavy multiplexing. It provides functionality to manage an actor’s
lifecycle (e.g. create/destroy, suspend/resume), to assign actors to workers in real
time, and to route incoming traffic to them.
Agent Substrate is intended to be a low-opinion system. The workloads it
manages don't have to be literal AI agents, but those are the best example of
the kind of applications it is designed for. It is not an SDK for building
agents, but rather a system for running them at scale.
Agent Substrate is currently in VERY early development. It is not ready for
production use, and the APIs are almost guaranteed to change. We are not
making any guarantees about backward compatibility at this stage, and
everything in this project may be changed.
Co-authored-by: Alex Bulankou <alexbu@google.com>
Co-authored-by: Benjamin Elder <bentheelder@google.com>
Co-authored-by: Bowei Du <bowei@google.com>
Co-authored-by: Dmitry Berkovich <dberkov@google.com>
Co-authored-by: Fabricio Voznika <fvoznika@google.com>
Co-authored-by: Francisco Cabrera <fclieutier@google.com>
Co-authored-by: Haven Xia <haoyuxia@google.com>
Co-authored-by: Julian Gutierrez Oschmann <juliangut@google.com>
Co-authored-by: Kevin Steuer <ksteuer@google.com>
Co-authored-by: Max Smythe <smythe@google.com>
Co-authored-by: Maya Wang <mymaya@google.com>
Co-authored-by: Michael Taufen <mtaufen@google.com>
Co-authored-by: Shruti Nair <shrutinair@google.com>
Co-authored-by: Taahir Ahmed <taahm@google.com>
Co-authored-by: Tim Hockin <thockin@google.com>
Co-authored-by: Zoe Zhao <zoezhao@google.com>
Tim Hockin