This change removes the hard-link from templates to worker pools,
allowing pools to be selected using k8s labels.
Both actors and actor templates are augmented with a label selector
field.
When an actor is resumed, we resolve the set of *eligible* worker pools
by evaluating the intersection of both actor and template selectors.
Then we find a free worker from one of them. The overall design is
described in [1].
[1]:
https://github.com/agent-substrate/substrate/issues/47#issuecomment-4712334620
**Updated Original PR Description**
## Summary
This PR adds a small, controlled set of scheduling fields to WorkerPool
so WorkerPool-managed Pods can be placed onto appropriate Kubernetes
nodes.
Added fields:
- `nodeSelector`
- `tolerations`
- `priorityClassName`
- `nodeAffinity`
## Scope
This intentionally does not expose a full `PodTemplateSpec`, and does
not add support for full `affinity`, `podAffinity`, or
`podAntiAffinity`.
Resource requests/limits are intentionally left out of scope for this CL
while the WorkerPool resource model is still being discussed separately.
The goal is to keep the first version small while addressing the
WorkerPool-to-Kubernetes-Node scheduling gap discussed in
[#212](https://github.com/agent-substrate/substrate/issues/212).
`nodeAffinity` is included to support heterogeneous node pools where
`nodeSelector` is too limited, such as requiring or preferring one of
several equivalent accelerator/local-SSD/cache node pools, or expressing
soft preferences during node pool migration.
`priorityClassName` is included to help preserve warm WorkerPool
capacity under cluster pressure. Since WorkerPool pods are the slots
available for actor resume, higher-priority pools can protect
latency-sensitive or interactive actor workloads from displacement by
lower-priority batch workloads, and allow different WorkerPools to
represent different service classes, e.g. interactive vs. batch.
## Related issue
Part of #212.
Related to #47.
Implements `PAUSED` state for issue #119.
The snapshot files are kept locally on node VM in a separate folder. At
resume time, scheduler uses a node VM hint and picks up a worker from
the same node where files were stored at suspend time.
The local file management solution is temporary and will be replaced
once @msau42 introduces a new component that is supposed to manage files
on the node VM.
- [X] Tests pass
- [X] Manual tests with counter demo
```
>kubectl ate create actor my-counter-1 --template ate-demo-counter/counter
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 1
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 2
>kubectl ate pause actor my-counter-1 -o json
> curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 3
```
### Breaking change
This PR introduces a breaking change in the Actor proto. All existing
actor needs to be recreated, prior testing PAUSE functionality.
Gives an actor a reliable way to learn its own ID without parsing the
Host header. atelet populates a per-actor identity directory (currently
the single file actor-id) and bind-mounts it read-only at /run/ate into
each app container; the pause container gets none.
- A directory rather than a single-file mount, per the discussion on
#178, so future late-bound data can land as sibling files without
changing the mount shape.
- The bundle is regenerated on every resume and external bind mounts are
re-attached from the restore-time config.json, so the value stays
correct per-actor through a golden-snapshot restore; an env var (or a
file baked into the image) would be frozen at the golden actor's ID,
since it lives in the checkpointed process memory.
- actor-id is written atomically (temp file, fsync, rename, parent
fsync): the directory is visible to a possibly-running actor, so a
reader must never observe a torn value.
- The identity dir is cleared and recreated with the other per-actor
dirs in resetActorDirs.
- The in-rootfs mount point is created via os.Root, so a symlink planted
in the image cannot redirect the write outside the extracted rootfs.
Documented for actor authors in docs/api-guide.md: read it fresh on use;
raw ID, no trailing newline.
Fixes#211 : Create a demo that shows any worker can run any
actortemplates. Workers and actortemplates live in different namespaces.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Tested:
Deployed the demo and tested create, suspend, resume, delete actors in
both namepsaces. https://paste.googleplex.com/4999545124683776
A couple of TODOs/Discussions which I think are important to have based
on these changes.
1. Do we need a caching client for this to avoid calling the apiserver
every time?
2. Where/how do we want to resolve secrets generally? I think there are
good arguments to be made for the control-plane OR the "node" component.
In my mind this also related to:
https://github.com/agent-substrate/substrate/issues/18Fixes#15
> It's a good idea to open an issue first for discussion.
- [ x ] Tests pass
- [ x ] Appropriate changes to documentation are included in the PR
It was using a wrong name `Actor Namespace` before, which doesn't exist
yet.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Adds a Prometheus server to the kind dev stack (in `otel-system`,
alongside the OpenTelemetry Collector and Jaeger). It scrapes component
`/metrics` via the existing `prometheus.io/scrape` annotations on
`ate-api-server` and `atelet`, scoped to the `ate-system`/`otel-system`
namespaces.
Fixes#106
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR refactors `ateom-gvisor` and `kubectl-ate` to adopt a
flat-merged logging architecture, eliminating server-side and
client-side field guessing:
1. **Cloud Logging**: `ateom-gvisor` parses actor JSON, injects actor
labels/time, and outputs the flat map directly. GKE Cloud Logging
built-in rules parse native severity/message fields automatically.
2. **CLI Logging**: Removed the `--raw` flag. `kubectl ate logs` strips
Substrate metadata and outputs clean, consistent JSON lines with the
`time` field prepended first.
<img width="3235" height="306" alt="image"
src="https://github.com/user-attachments/assets/4a6ca291-db8d-45dc-b840-43efa74739f0"
/>
3. **Documentation**: Updated `docs/observability.md`
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Document the rationale behind each directory and guidelines on how to
decide where to put new code in a dedicated markdown file.
Update AGENTS.md with a summary and a reference to the new markdown
file.
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
This PR adds live-streaming log support (`--follow/-f`) to `kubectl-ate
logs actors`. Unlike static container logs, this implementation is
actor-aware and resilient to virtual actor suspensions and migrations:
Suspend Lifecycle:
<img width="3700" height="1856" alt="image"
src="https://github.com/user-attachments/assets/38568108-8229-40b0-afc7-d1d5ddc817e1"
/>
Resume Lifecycle:
<img width="3696" height="1530" alt="image"
src="https://github.com/user-attachments/assets/cf934608-62fe-4fb8-bd2c-5e6e7a0af776"
/>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This is the initial release of the Agent Substrate.
Agent substrate is a system built on top of Kubernetes which manages agent-like
workloads to achieve higher scale and efficiency than Kubernetes alone can
offer, with lower latency. It builds on top of Kubernetes features like
Pods and Pod autoscaling, but takes the Kubernetes control-plane out of the
critical path to achieve lower latency.
It can run on any Kubernetes cluster and does not inhibit “regular” use of
Kubernetes in any way. Kubernetes provides the infrastructure provisioning and
management for all types of workloads, while Agent Substrate provides
agent-specific scheduling and control.
At its core, Agent Substrate maps a larger set of “actors” (applications such
as agents) onto a smaller set of ready “workers” (Kubernetes Pods), relying on
the fact that agent-like applications tend to be idle most of the time to
achieve heavy multiplexing. It provides functionality to manage an actor’s
lifecycle (e.g. create/destroy, suspend/resume), to assign actors to workers in real
time, and to route incoming traffic to them.
Agent Substrate is intended to be a low-opinion system. The workloads it
manages don't have to be literal AI agents, but those are the best example of
the kind of applications it is designed for. It is not an SDK for building
agents, but rather a system for running them at scale.
Agent Substrate is currently in VERY early development. It is not ready for
production use, and the APIs are almost guaranteed to change. We are not
making any guarantees about backward compatibility at this stage, and
everything in this project may be changed.
Co-authored-by: Alex Bulankou <alexbu@google.com>
Co-authored-by: Benjamin Elder <bentheelder@google.com>
Co-authored-by: Bowei Du <bowei@google.com>
Co-authored-by: Dmitry Berkovich <dberkov@google.com>
Co-authored-by: Fabricio Voznika <fvoznika@google.com>
Co-authored-by: Francisco Cabrera <fclieutier@google.com>
Co-authored-by: Haven Xia <haoyuxia@google.com>
Co-authored-by: Julian Gutierrez Oschmann <juliangut@google.com>
Co-authored-by: Kevin Steuer <ksteuer@google.com>
Co-authored-by: Max Smythe <smythe@google.com>
Co-authored-by: Maya Wang <mymaya@google.com>
Co-authored-by: Michael Taufen <mtaufen@google.com>
Co-authored-by: Shruti Nair <shrutinair@google.com>
Co-authored-by: Taahir Ahmed <taahm@google.com>
Co-authored-by: Tim Hockin <thockin@google.com>
Co-authored-by: Zoe Zhao <zoezhao@google.com>
Julian Gutierrez Oschmann