The ateoms now write the actor usage records and serve their latest
sample (#2194, #2195), so atelet stops writing its own and counts CPU
per activation.
- **atelet CPU.** Baselines are keyed by actor and epoch. A new
activation counts in full if it began after atelet started, and a worker
that does not answer keeps its baselines. This fixes the undercount on
micro-VM restores and the double charge when a silent worker's baselines
were dropped.
- **atelet records removed.** The ateoms' `ate.actor.usage_sampled`
records replace them, ending the period with two usage records per
actor.
- **Timeout.** `GetActiveWorkloadStats` answers from memory, so atelet's
`statsRPCTimeout` drops from 55s to 5s. The 50s poll floor stays.
- **Cleanup.** `UsageEmitter.EmitFinal` and its flush, atelet's stdout
`SyncedWriter`, and the exported `actorlog.LabelsKey` go; nothing uses
them now.
- **Docs.** The proto comments describe the cached discovery read. The
registry and the guide say when the final record is written: whenever
the ateom tears an activation down (checkpoint, terminate, graceful
shutdown, or a micro-VM re-host), read just before the teardown.
#### Known limitation and future work
atelet's CPU counter only sees samples of actors still hosted, so it
loses each activation's tail: the CPU from the last sample atelet read
to the teardown, up to one poll interval plus one sample interval. An
activation shorter than that, the common checkpoint-when-idle case, can
lose most or all of its CPU in the metric. The usage records keep it,
through the final record.
To fix it later, the ateoms can own the counter: each ateom adds an
activation's CPU increase at every reading, initial, periodic, and
final, and exports it through the existing OTLP relay. No gRPC change is
needed. atelet then drops the baseline map, its start time, and the
silent-worker rule; this also removes the baselines a crashed worker's
leftover directory keeps. The cost is cardinality: the series become one
per worker pod per label set instead of one per node, still bounded by
pods rather than actors, plus a short-lived series per replaced pod. A
collector can drop the pod attributes if that matters.
#### Breaking change
The stdout usage record moves from atelet to the ateoms, with flat
`ate.*` keys and the message `Actor usage sampled`. Log filters on
atelet's `Actor usage sample` record stop matching: filter on
`jsonPayload.msg="Actor usage sampled"` and the `ate.*` keys instead.
Identity is no longer in the label group, so on GKE it does not land in
`LogEntry.labels`.
Part of #1748 (steps 4 to 6).
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
**This is a rename, not a behavior change.** `SnapshotConfig.onCommit`
becomes `preferredFidelity`, and the two-valued `SnapshotContentScope`
enum becomes the ordered `SnapshotFidelity` enum, with the same
vocabulary carried through atelet, ateom, and the metric label. `MEMORY`
is exactly what `FULL` was and `VOLUMES` is exactly what `DATA` was:
what gets captured on pause and suspend, what gets restored, the
golden-actor override, and the paused-upload narrowing are all
unchanged. The point is to put the lifecycle-v2 names in place before
step III adds `required_fidelity` and `resumed_fidelity` on top of them.
The one addition is `ROOTFS`, defined in the enum so it is complete but
rejected everywhere, since no runtime can produce it yet.
Part of #2102, step II as tracked in #2351. Step I (#2309) removed
`on_pause`.
Two commits, meant to be read in order.
## 1. ateapi: `SnapshotFidelity` and `preferred_fidelity`
- New ordered enum `SnapshotFidelity { VOLUMES = 1, ROOTFS = 2, MEMORY =
3 }`, each level including the ones below it. `SnapshotContentScope` is
deleted. MEMORY is the old FULL and VOLUMES the old DATA, so behavior is
unchanged.
- `SnapshotConfig.on_commit` becomes `preferred_fidelity`, documented as
a best-effort ceiling: the system never packages more than this level
and may package less when a layer cannot be captured or is too expensive
at the time. A process resumed in place keeps its memory regardless.
Defaults to MEMORY. Field numbers rearranged to `storage_location = 1`,
`preferred_fidelity = 2`.
- `ExternalSnapshot` and `LocalSnapshot` record `fidelity` instead of
`content_scope`.
- ROOTFS is in the enum so it is complete, but no sandbox runtime
captures rootfs without memory yet, so a new field-level validator
rejects it at template admission. Actors only take their fidelity from a
template, so none can carry it.
- kubectl-ate tag output, demos, benchmarking manifests, glossary, API
guide, and architecture doc use the new vocabulary. The obsolete apitool
exemption is removed.
## 2. atelet, ateom: the same enum on the wire
- Both internal protos replace `SnapshotScope` with a `SnapshotFidelity`
mirroring the public one; request fields become `fidelity` and
`desired_fidelity`. Every hop maps level by level, ROOTFS included, and
nothing is promoted: an unexpected value goes out as UNSPECIFIED for the
next hop to reject.
- Defense in depth for ROOTFS: atelet rejects it in checkpoint, restore,
and paused-upload validation, and both ateom runtimes reject it on
`CheckpointWorkload` and `RestoreWorkload` through a shared
`resources.ValidateSnapshotFidelity`.
- A golden snapshot must record MEMORY (UNSPECIFIED is no longer
tolerated).
- The metric attribute `ate.snapshot.scope` (`full`/`data`/`unknown`)
becomes `ate.snapshot.fidelity` (`volumes`/`rootfs`/`memory`/`unknown`)
in the OTel registry, `ateattr`, the ateom phase logs, and the snapshot
manifest atelet writes and reads back when narrowing a paused upload.
Dashboards keyed on the old label need updating.
## Behavior changes
- **Breaking API change.** Templates that set `onCommit` or use
`SNAPSHOT_CONTENT_SCOPE_*` values are rejected. Per the pre-v1.0 policy
there is no compatibility shim, and previously stored templates and
snapshots are not expected to exist.
- Snapshot manifests written with the old `scope` key read back with no
fidelity, which the upload path already refuses with a clear message.
- Unchanged: what MEMORY and VOLUMES capture and restore, the
golden-actor MEMORY override, and the paused-upload narrowing from
MEMORY to VOLUMES.
## Relationship to #2289#2289 restructures the snapshot status into `Snapshot` and
`SnapshotStorage`. If it lands first, the `fidelity` field in this PR
moves onto `SnapshotStorage` (per copy), which is where it belongs
long-term; the enum, the template field, and the data-plane change are
the same either way.
## Follow-ups (step III)
`required_fidelity` with `required <= preferred` validation and the
CRASHED rule, `ResumeActorResponse.resumed_fidelity`, and ROOTFS support
in the micro-VM runtime.
## Verification
- `go test` for ateapi (unit and functional), kubectl-ate, atelet,
ateom-microvm, ateom-gvisor, ateomphaselog, ateattr, resources, and
apitool pass; new validator at 100% coverage.
- golangci-lint, kube-api-linter, python-protos, boilerplate verifiers
pass; codegen, proto-fmt, licenses, go.mod show no drift.
- Not run locally: the Weaver metrics-registry check and shellcheck,
which need Docker; relying on CI for those. e2e suites compile but were
not run against a cluster.
- [x] Breaking change
- [x] Not backward compatible
🤖 Generated with [Claude Code](https://claude.com/claude-code)
#### Breaking change
With `OTEL_LOGS_EXPORTER=otlp`, actor events (`ate.actor.state_changed`
and `ate.actor.crashed`, and `ate.actor.usage_sampled` once the ateoms
emit it) no longer go to stdout. `kubectl logs` and any agent that
collects pod stdout stop receiving them.
To keep the stdout records, set `OTEL_LOGS_EXPORTER=otlp,console` in
`ate-otel-config`, then restart ate-api-server and ate-controller. The
controller restart rolls the WorkerPool Deployments, which replaces the
workers. Deployments that leave the variable unset or `none`, the
default everywhere but kind, are not affected.
Implements the decisions on #2197, and provides the shared
`OTEL_*_EXPORTER` parser agreed on #2067.
An actor event goes to one place unless you ask for both.
| `OTEL_LOGS_EXPORTER` | Actor events go to |
|---|---|
| unset or `none` | stdout, as today |
| `otlp` | OTLP only (before: OTLP and stdout) |
| `console` | stdout |
| `otlp,console` | both |
Before, `otlp` wrote every record twice, and on a cluster whose logging
agent also collects pod stdout, as GKE's does, both copies were stored.
`console` here is the component's own JSON log line, not the OTel SDK's
console exporter format, and the docs say so.
**Routing.** `actorevent` writes the OTLP event when the logs exporter
takes it and the stdout record otherwise, or both with `console`, which
`serverboot.InitLogging` applies for every component. Both forms come
from the same call, so the fields cannot drift. This covers ateapi's
lifecycle events now and the ateoms' usage records once #1984 lands.
Kind sets `otlp,console`, and atecontroller copies the value into the
worker pods, so kind keeps the events in `kubectl logs`. One
`OTEL_LOGS_EXPORTER` in `ate-otel-config` controls both streams.
**Shared parser.** `serverboot.resolveExporters` reads any
`OTEL_*_EXPORTER` variable with the rules from #2067, and returns the
set of selected exporters:
| Input | Result |
|---|---|
| unset or empty | the signal's default |
| a list, such as ` Console , OTLP ` | trimmed, lowercased, counted once
|
| an unknown or empty item, such as `otlp,zipkin` or `otlp,` | skipped
with a warning; the rest apply |
| `none` | the empty set |
| `none` with an exporter, such as `none,otlp` | rejected with a
warning; the signal's default |
| no known name, such as `zipkin` | the signal's default, with a warning
|
Logs call it with `otlp` and `console` and a default of `none`. #2190
can call it for metrics and traces in place of `otlpExportEnabled`.
`ResolveLogsExporter` no longer takes a default,
`LoggingOptions.Exporter` is now the set, and callers check it with
`Has(serverboot.ExporterOTLP)` or `Has(serverboot.ExporterConsole)`.
**Docs.** The guide lists the parser rules and the cases in which `otlp`
alone loses a lifecycle record: a full queue, a collector that stays
unreachable, or an ungraceful exit. `otlp,console` is the fallback, and
a collector that reads both forms drops one by `event.name`. The
`substrate.yaml` lint exception that recorded the double write is
removed.
**Tests.** The parser rules, each pinned by a mutation check; the logs
names on top of them; `InitLogging` setting `console`; one record per
call on each path with the same attributes; both records with `console`;
and the caller's timestamp on each.
Fixes#2197
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#997
## Summary
Give `ateapi` separate PostgreSQL connections for schema ownership and
normal application access. Migrations and outbox partition maintenance
use the owner connection, allowing application reads and writes to run
with restricted permissions.
- Each connection assumes its configured role with `SET ROLE`, keeping
object ownership and grants separate from login credentials.
- Configure owner and read/write connections through `ate-setup` and the
installation manifests.
- Default the schema to `substrate` and document provisioning for
external PostgreSQL and Cloud SQL, including one IAM login with
membership in both roles.
- Keep bundled PostgreSQL using its existing `postgres` account and role
for both pools. Restricted runtime access requires operator-provisioned
roles.
See `docs/postgres.md` for configuration details.
## Scope
This is the base PR for two separate follow-ups: [file-based
credentials](https://github.com/agent-substrate/substrate/pull/1979) and
[automatic development database
bootstrap](https://github.com/agent-substrate/substrate/pull/2025).
Neither feature is included here.
Operators provision external database roles, login memberships, schema
ownership, and grants before startup. Table migrations still run
automatically through the owner connection.
The documented external database layout assumes a fresh database; this
PR does not migrate existing table ownership or grants. Both connections
remain inside `ateapi`, so this separates database permissions without
isolating owner credentials from the application process.
## Validation
- `env -u NO_COLOR make verify` passed.
- PostgreSQL integration tests cover restricted runtime access, owner
migrations and partition maintenance, and OpenFGA table permissions.
- Installer tests cover bundled PostgreSQL, separate external logins,
and Cloud SQL configuration.
Fixes#1877
`newMeterProvider` always installed the OTLP periodic reader, so a
deployment that scrapes the components' Prometheus endpoints could not
stop the push and received every instrument twice.
`OTEL_METRICS_EXPORTER` now accepts `otlp` (the default) and `none`, and
warns on any other value and keeps OTLP, like `OTEL_LOGS_EXPORTER`. With
`none` the OTLP reader is skipped and the Prometheus reader stays.
atecontroller moves to a new `serverboot.InitMetricsBridged`: with OTLP
it bridges controller-runtime's registry onto the push and keeps the
OTel instruments off that registry; with `none` it registers
`ate.workerpool.*` on that registry so the manager's `:8080` serves
them. Its optional producer wrapper applies on the OTLP path only, where
atecontroller keeps padding empty exponential histograms for the
Telemetry API. ateom is unchanged apart from honoring the variable.
Documented in `docs/observability.md` ("Scraping instead of pushing").
- [x] Tests pass (`internal/serverboot`: value resolution and the
warning; `InitMetrics` dials by default and not with `none`;
`InitMetricsBridged` pushes the bridged family and the OTel instrument
exactly once to a fake OTLP collector without registering the instrument
on the bridged registry, collects through the producer wrapper, and with
`none` serves both from that registry without dialing)
- [x] Appropriate changes to documentation are included in the PR
- Add two new labels: `breaking-change` and `release-note/none`.
- Add .github/release.yml so GitHub's generated release notes follow the
`v0.2.0` layout:
- Breaking Changes
- Bug Fixes
- Documentation
- per-area feature sections
- Dependencies
- Other Changes.
- `docs/dev/release-notes.md` covers the sections, PR titles, and the
release steps.
- Points to the new doc content via PR template and CONTRIBUTING
- Pushing `vX.Y.Z` or `vX.Y.Z-PRE` runs `hack/release/draft-release.sh`
which creates a draft release (a prerelease for -PRE tags) with notes
from `.github/release.yml` plus a committer list, as in `v0.1.0` and
`v0.2.0`. **A maintainer edits and publishes the draft.**
Fixes#2017
Adds a credential provider for egress credential injection backed by
Google Cloud Secret Manager.
**It lives in its own Go module under `plugins/gcp-secret-manager`,
temporally hosted here until it moves to a repository of its own.**
**What it does**
- Serves `credproviderpb.CredentialProvider` over mTLS and admits only
the egress gateway's identity (`--injector-identity`).
- Resolves global and regional secrets, optionally picking one key out
of a JSON payload:
`ate-secret://secretmanager.googleapis.com/projects/<project>[/locations/<location>]/secrets/<secret>/versions/<version>[/keys/<key>]`
- Enforces a default-deny atespace→project policy
(`--project-policy-file`), the counterpart of the Kubernetes provider's
namespace policy.
- Returns a retryable 503 only for transient Secret Manager failures,
and caps each read with `--fetch-timeout` (default 3s).
**Repository changes**
- New top-level `plugins/` directory for self-contained plugins,
documented in `docs/dev/code-layout.md` and `AGENTS.md`. The module
imports only substrate's public `pkg/` packages; a test enforces this.
- CI runs the module's tests, `make verify` and golangci-lint.
govulncheck scans the module too, with the action pinned by SHA.
- `docs/egress-credential-injection.md` describes each provider's
credential URI format. The plugin's README covers installing and using
it.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This change wires the `gotestsum` seam into the actual CI and ensures
that all tests _run_ and _pass_ using a new "gate" that verifies the
JUnit artifacts they produce. Specifically, the new gate works as
follows:
1. _registers_ that a test _should_ run and produce an artifact
2. _verifies_ that the artifact was produced and _all_ tests passed
(none were skipped).
The registration phase happens when a test is run in CI and requires
that the test be provided a `E2E_JUNIT_FILE` env variable.
Partial work for #1871
- [x] Tests pass
```
Run make build-junittool
make build-junittool
bin/junittool verify -manifest "${ARTIFACTS}/expected-junit.txt"
shell: /usr/bin/bash -e {0}
env:
E2E_ATENET_DATAPLANE: agentgateway
ARTIFACTS: /home/runner/work/substrate/substrate/_artifacts
go -C tools/junittool build -o
/home/runner/work/substrate/substrate/bin//junittool .
FILE TESTS FAILURES ERRORS SKIPPED
/home/runner/work/substrate/substrate/_artifacts/e2e-gvisor.xml 84 0 0 4
/home/runner/work/substrate/substrate/_artifacts/e2e-microvm.xml 84 0 0
5
/home/runner/work/substrate/substrate/_artifacts/e2e-mitm.xml 1 0 0 0
/home/runner/work/substrate/substrate/_artifacts/e2e-mitm-microvm.xml 1
0 0 0
/home/runner/work/substrate/substrate/_artifacts/e2e-networking-mitm.xml
11 0 0 1
/home/runner/work/substrate/substrate/_artifacts/e2e-networking-mitm-microvm.xml
11 0 0 1
TOTAL
```
- [x] Appropriate changes to documentation are included in the PR
Step 3 of #1748. Vocabulary only: nothing emits the event or sets the
epoch yet.
**Event.** `ate.actor.usage_sampled` in `actorevent`, sixteen required
keys via `ateattr`: identity, pool, sandbox class, source,
`ate.stats.kind` (periodic, initial, final), the four measurements,
observation time, epoch. Measurement names follow the
`ate.actor.stats.*` instruments; units are in the registry briefs. Body
is `Actor usage sampled`, the accepted stdout shape change.
**Scope.** The usage stream has its own instrumentation scope, the
lifecycle scope with `/usage` appended, so a consumer can sample or drop
it without touching lifecycle records. `NewEmitter` takes the scope; the
package-level `Log` keeps the lifecycle one, so ateapi is unchanged.
**Epoch.** `ate.actor.epoch` and `WorkloadStatsSample.epoch_unix_nano`:
the unix-nano time an activation began. CPU time is per activation for
every source; lifetime is the sum over epochs of each epoch's highest
value. Memory usage and working set are absolute; peak is as the source
reports it. An ateom that leaves the epoch at zero still sends the raw
guest counter. The measurement comments on fields 8 to 11 say all of
this.
**Registry.** Event group in `events.yaml`, seven attributes in
`metrics.yaml`, `ate.actor.epoch` in the no-actor-identity forbidden
list. The one-record-one-place rule and the collector guide now name
both scopes.
Left for step 4: the event's code anchor moves to the ateom emit site,
atelet's poller comments and the guide's stdout usage section get
rewritten, and the controller starts passing `OTEL_LOGS_EXPORTER` to
worker pods.
Step 1 of #1748. The relay forwards `LogsService` the way it forwards
traces and metrics: source gate, verbatim payload, atelet's own
`OTEL_EXPORTER_OTLP_LOGS_HEADERS` upstream.
No content check on the records. Identity spoofing through the relay is
handled in #1748 by the consumer rule (a lifecycle record counts only
under ateapi's resource, which the source gate already enforces) and
per-pod sockets (#741), not by a denylist here.
**Endpoint and compression resolution** (second commit, from review).
The relay read the generic, traces, and metrics OTLP variables and
ignored the logs ones, so a logs-specific collector or compression
setting was silently overridden. All three signals now go through one
resolver: each falls back to the generic, and every resolved value must
agree, since the relay has one upstream connection. The conflict error
names the variable each value came from; a signal that falls back names
the generic variable, not its own unset one.
One behavior change: `traces=A, metrics=A, generic=B` with logs unset
used to pass and now fails at atelet startup, because logs falls back to
`B` and that is a two-collector configuration.
Tests: an ateom log batch arrives with the ateom's resource; the source
gate, mixed-batch, header-isolation, and per-signal header tests cover
logs; resolver tables gained logs rows including the fallback case. Docs
say the relay carries logs, traces, and metrics.
Kata 4.1.0 bundles virtiofsd 1.14.0 (required by Substrate). This
simplifies the dev process on arm64 because it removes the need to build
virtiofsd from source.
Verified on an arm64 KVM host (Lima + kind).
Fixes#1695
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## What this does
ateapi writes a record every time an actor changes state. Until now
those records only went to the pod's stdout, and nothing reads stdout.
This sends the same records to a collector as OTLP log events.
Actor name and uid cannot be metric labels (too many values), and traces
are sampled at 1%. So these records are the only way to answer "what
state is this actor in, and since when".
## Changes
- `serverboot.InitLogging` sets up a LoggerProvider, next to the
existing tracer and meter ones.
- New `internal/actorevent` package builds the log records.
- ateapi emits at the two places that already write the stdout records.
- Two event names: `ate.actor.state_changed` and `ate.actor.crashed`.
- Both names are registered in `docs/metrics/registry/events.yaml`, so
`make verify` checks them.
- kind gets a logs pipeline and a count connector. The e2e suite reads
the counts back.
- Docs updated. `otel-collector.md` said substrate has no
LoggerProvider, which is no longer true.
## Opt-in
`OTEL_LOGS_EXPORTER` defaults to `none`. Only the kind overlay sets it
to `otlp`. The base ConfigMap is untouched, so no deployed environment
changes when this merges.
## Notes on the design
- **No slog bridge.** Only two call sites emit these records, so
emitting twice costs two lines. A bridge would also send every ateapi
log over the wire, could not set the event name, and would loop, because
SDK export errors are logged through slog.
- **Batching processor, not the simple one.** These records sit on the
actor resume path. A processor that exports inside the emit call would
add a blocking gRPC call there, so a slow collector would become control
plane latency.
- **Two event names, not one per state.** `ate.actor.state` already says
which transition happened. A crash gets its own name because it carries
two extra attributes and a higher severity.
- **Both copies are kept on purpose.** No collector in this repo reads
pod stdout, so nothing is duplicated today. `kubectl logs` keeps
working. If a filelog agent is ever added, drop one of the two. The
escape hatch is written down in `docs/metrics/substrate.yaml`.
## Dependencies
Adds `otel/log`, `otel/sdk/log` and `otlploggrpc`, all pinned at
v0.20.0. That is the release that matches the pinned `otel v1.44.0`.
v0.21.0 would pull the core modules to v1.45.0, which this change does
not need. The logs API has a
v1.47.0 release candidate upstream, so it is on its way to stable.
## Testing
- Unit tests for the exporter resolver, the record builder, and both
ateapi emit sites.
- The record builder test checks the attribute set matches what the
event name declares, in both directions.
- The ateapi tests check the OTLP record carries the same attributes as
the stdout record.
- Ran end to end on kind. Records arrive with the right event name,
severity, attributes, and with trace context on the record's own fields
rather than as attributes.
- Checked the off state too. With `OTEL_LOGS_EXPORTER` removed, the
collector receives no log records and stdout is unchanged.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Nested virtualization exposes `/dev/kvm`, which microVMs need. The new
`--enable-nested-virtualization` flag turns it on for the node pool
`setup-gcp` creates. It defaults to on, so pass
`--enable-nested-virtualization=false` for a cluster that does not need
it.
Also, rename the env var `GVISOR_NODE_MACHINE_TYPE` to
`NODE_MACHINE_TYPE` to be consistent with other env vars and to avoid
confusion for microVMs. The old env var still works with a warning.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Component authors had the metric registry and the cardinality rules but
no single place that says how to add an instrument: which kind to pick,
how to name it, which labels are allowed, how the Go is shaped, how to
test it with a ManualReader, and how to register it. Each of those
conventions existed only as a pattern in one of the metrics.go files.
The guide sits beside tracing.md and otel-collector.md. It uses the
instruments already in the tree as the examples, ends with a worked
example of a node-local snapshot cache (a counter by outcome, a fill
histogram, a size gauge and an eviction counter, and what is
deliberately not a label). The registry snippets in the guide pass
`weaver registry check` when appended to the real registry.
observability.md and AGENTS.md now point at the guide from the place
that already told contributors to update the registry.
Fixes#1552
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This adds a test that serves as a guardrail to prevent changes to the
atespace-scoped tables that would make it difficult or impossible to
partition it in the future.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
cc @bowei @BenTheElder @juli4n @shrutiyam-glitch @thockin
## What
Now that the ActorTemplate CRD has been deleted and its resources moved
to the substrate gRPC API and the control-plane store (created/managed
with `kubectl-ate`, persisted in PostgreSQL), several documents still
describe ActorTemplate as a Kubernetes CRD, or describe namespace/RBAC
relationships that no longer exist. This sweeps the docs for those stale
references.
Fixes#368 (docs side).
> This change was prepared with AI assistance; I have reviewed and
tested it.
- [x] Docs and comment-only change; no functional code changed, no tests
affected
### Description
This PR introduces native, secure support for using Cloud SQL as the
PostgreSQL store backend for `ate-api-server`.
To ensure the highest level of security and ease of use in GCP
environments, this integration leverages the Cloud SQL Auth Proxy
sidecar with automatic IAM database authentication. This means transport
security (TLS 1.3 tunnel) is handled automatically, and database
sessions are authenticated using Workload Identity via short-lived OAuth
tokens, completely eliminating the need for database passwords.
### Key Changes
* **Cloud SQL Auth Proxy Sidecar:** Added
`manifests/ate-install/cloudsql-proxy-patch.yaml` to patch the sidecar
into the `ate-api-server` deployment when a Cloud SQL instance is
configured.
* **Automated Provisioning:** Extended `tools/setup-gcp` with a new
`cloudsql` command. This handles the idempotent creation of the Cloud
SQL instance, Google Service Accounts (GSA), IAM bindings, and Workload
Identity bindings.
* **Installation Script Updates:** Updated `hack/install-ate.sh` to
parse new environment variables (e.g.,
`ATE_API_POSTGRES_CLOUDSQL_INSTANCE`, `ATE_API_POSTGRES_CLOUDSQL_GSA`)
and correctly synthesize the passwordless DSN and ConfigMaps for the
proxy.
* **Security & Documentation:**
* Added extensive documentation in `tools/setup-gcp/cloud-sql.md`
covering provisioning, schema privileges, deployment, and database
scaling.
* Updated `docs/threat-model.md` to reflect the new Cloud SQL egress
flows and Auth Proxy tunnel mechanics.
* **Dependencies:** Vendored required Google API clients (`sqladmin/v1`,
`servicenetworking/v1`, `iam/v1`) for the GCP setup tool.
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
> [!WARNING]
> Recreate PostgreSQL databases from earlier development builds.
## Summary
This change replaces startup schema setup with embedded, versioned SQL
migrations.
`ateapi` uses Goose to apply migrations before readiness. Goose stores
one ledger record for each applied migration.
Goose runs each migration and inserts its ledger record in one
PostgreSQL transaction.
Closes#901.
Based on this design:
https://docs.google.com/document/d/13ixDKRoAIFXeLxS-_1nikNcAy76ca8m0eVgobfib93E/edit?usp=sharing
## Migration behavior
`ateapi` gets a session advisory lock for the configured schema before
it applies pending migrations.
One replica applies migrations while other replicas wait. Goose reads
the ledger again after it gets the lock.
If a migration fails, PostgreSQL rolls back its SQL and ledger record.
Earlier successful migrations remain applied and recorded.
Kubernetes restarts the failed replica. The next startup resumes from
the first migration without a ledger record.
## Changes
- Add Goose and a per-migration ledger.
- Replace the initial up and down files with one transactional, up-only
migration.
- Keep migration 1 aligned with the current schema, including actor
egress policy storage.
- Remove existence guards and explicit transaction statements from the
baseline migration.
- Apply all pending migrations before `ateapi` becomes ready.
- Serialize each migration run with a PostgreSQL session advisory lock.
- Start without changes when the database schema is current or ahead.
- Reject application tables that do not have a migration ledger.
- Log the starting, current, and latest versions.
- Log the applied migration count and duration.
- Retry only initial database connection failures.
- Return schema and migration errors without a retry.
- Add `--postgres-schema` and `ATE_API_POSTGRES_SCHEMA`.
- Use `public` as the default PostgreSQL schema.
- Use the configured schema for the main and watch pools.
- Restrict outbox partition maintenance to the configured schema.
- Let the installer use an external PostgreSQL database.
- Add the migration design and recovery policy to the repository.
## Migration file policy
Migration files use sequential versions and contain exactly one Goose
`Up` section.
CI rejects down migrations, nontransactional migrations, environment
substitution, explicit transaction control, and `IF NOT EXISTS` guards.
Before the first stable v1 release, developers can change or squash
migrations. Developers must recreate databases after migration history
changes.
After that release, CI rejects changes or deletions against the latest
stable release tag that contains migrations.
Goose does not store migration checksums. The binary embeds each
migration file, and release-tag checks protect released migration
history.
## Compatibility
No release includes PostgreSQL support. The `v0.0.0` release predates
the PostgreSQL backend.
Users must recreate databases from earlier PostgreSQL development
builds.
Every committed migration prefix must work with the current and previous
`ateapi` releases. This rule supports rolling upgrades and temporary
binary rollback.
A binary rollback does not roll back the database schema.
## Testing
Tests cover:
- Fresh database migration.
- Concurrent startup.
- Advisory lock waits.
- Current and ahead database schemas.
- Rejection of application tables without a migration ledger.
- Atomic rollback of a failed migration.
- Retention of earlier successful migrations.
- Resume from the failed migration after restart.
- Configured schema isolation.
- Outbox partition isolation.
- Migration file policy checks.
- Stable release migration immutability.
This PR is very large since it updates all existing demos and benchmark
workloads to use the new ActorTemplate substrate proto.
Please use the "Commits" tab to review individual commits.
Verifications done:
* Used this script: gpaste/5143788763348992 to verify that the change
from CRD -> proto are equivalent.
* The e2e tests are using the new susbtrate resources.
* Picked the parking demo to run e2e manually: gpaste/6193361380311040
This is the companion to #496
In order to grant access to `/dev/kvm` we have to either:
- use a device plugin
- use a DRA driver
- use an NRI plugin
An NRI plugin is highly privileged in it's own right for all pods on the
host and is difficult to ship portably at the moment.
DRA is promising, but not enough functionality is GA yet at our current
1.35+ target.
Device plugin fits reasonably well. We do wind up publishing an
~arbitrarily high limit, which has some cost in kubelet memory, but
otherwise is relatively clean.
This approach is what kata uses currently. Their device plugin is not
available unbundled, and we anyhow have a per-node daemonset.
atlet is taught to sniff if /dev/kvm appears on the host at all, so we
can also stop using the manually labeled nodes for microVM class and
instead schedule to the KVM + TUN devices on nodes that advertise them.
Later we can migrate to device plugin by using
I implemented that already, but I don't think it's worth merging at the
moment. We would want consumable capacity to be on by default. We can
migrate later without changing the pod spec by using
`extendedResourceName`.
https://github.com/agent-substrate/substrate/compare/main...BenTheElder:substrate:ateom-microvm-dra
NOTE: I confirmed with upstream that device plugin is not going anywhere
despite being "v1beta1", it's GA in all but name. It won't receive new
features but we don't really need anyhow. We'll move to DRA down the
line.
---
By doing this, we can drop `privileged: true` from the uVM ateom pods.
We can also drop the `ate.dev/sandboxClass` node label hacks, reducing
friction to deploy.
## Summary
A follow up to #640 where we introduced PostgreSQL as an alternative
storage backend, selected conditionally in ateapi.
- Deleted ateredis, its tests, and its dependencies
- Removed Redis backend selection and configuration so ateapi always
connects to Postgres
- Replaced Valkey resources with Postgres in the standard and Kind
deployment paths and simplified install script
- Replaced miniredis fixtures with isolated Postgres testcontainers and
added centralized helpers for seeding resources
- Renamed Redis-specific debug flush command to backend-neutral
`debug-clear-store` in CLI
- Updated comments and docs where applicable
## Benchmarking
Extensive benchmarking have been performed to evaluate Redis vs
Postgres, and results can be found in these two documents:
-
https://docs.google.com/document/d/10K0wB6aTeFkJCL4HN3NbLJCdFGoLYdhIcqkFnqFHkKc/edit?usp=sharing
-
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Actor logs used `ate.dev/actor_*` while spans and metrics use `ate.*`
registry in internal/ateattr. This PR makes ateattr the single source of
truth for everything telemetry-related.
- Renamed the six actor log labels onto the registry:
- `ate.atespace`, `ate.actor.name`, `ate.actor.uid`,
`ate.template.namespace`, `ate.template.name`,
`ate.actor.container.name`
- Logs join traces now. Records set `trace_id`, `span_id` and
`trace_flags`, so you can go from Actor restored to the resume that
caused it. Our own lines only, not an actor's stdout: one goroutine
forwards a whole container stream and can't know which request produced
a given line. Per line correlation comes with #853.
- Actors can't fake platform labels. They already couldn't overwrite
ours, but they could invent new ones like `ate.tenant` that look
platform issued downstream. Anything under `ate.` from an actor is now
dropped.
- Fixed the asymmetry that was actually left: actor supplied label
values weren't stringified, and one non string value makes Cloud Logging
discard the labels for that whole entry.
- Note: the actor_uid bullet in the issue is stale, #841 fixed it
earlier. Lifecycle records still set five labels rather than six, on
purpose as they're about the actor, so no container produced them.
Fixes#886
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
---------
Signed-off-by: krisztianfekete <git@krisztianfekete.org>
Part of #563
Substrate had no way to measure how much telemetry it sends. This PR
adds one, and builds it from the parts the benchmark suite already has.
The managed collector sends its self-metrics to Cloud Monitoring only,
and gives no data for each source. Thus you cannot see which component
makes the volume. A second collector, the telemetry meter, counts the
spans and the datapoints by `service.name`.
## Contents
- **`benchmarking/telemetry/meter.yaml`** — the meter. A tee: it counts
the telemetry by service, then sends it on to the managed collector.
Thus one run gives the volume for each service and the cost of the load
together.
- **`benchmarking/telemetry/README.md`** — the method: install, measure,
read, remove.
- **`hack/install-ate.sh --otlp-endpoint`** — sends all control plane
telemetry to the meter. Each component reads the endpoint from the
`ate-otel-config` ConfigMap through `envFrom`, and `ate-controller`
copies the value to the ateom worker pods, thus one patch is sufficient
for all of them.
- **`benchmarking/workloads/deploy.sh --otlp-endpoint`** — the same for
actor containers, which substrate configures separately.
- **`benchmarking/monitoring.yaml`** — scrapes the meter and cAdvisor.
- **`benchmarking/observability.md`** — prerequisites and the scenario
ladder.
- **`docs/dev/best-practices/otel-collector.md`** — the volume model and
the size guidance, corrected for the `ParentBased` defaults from #711.
## Correction
The `glutton` ActorTemplate had no `env`. Substrate puts no OTLP
configuration in an actor container, thus the exporter used the SDK
default `localhost:4317`, no process listens on that address in the
sandbox, and substrate discarded each actor span with no error message.
The template now sets `OTEL_EXPORTER_OTLP_ENDPOINT` and
`OTEL_RESOURCE_ATTRIBUTES`.
`ActorTemplate.spec` is immutable, thus `deploy.sh --deploy` deletes the
templates before it applies them.
## How I verified this
On a GKE cluster with managed OTel. The sequence:
```bash
# 1. Start from a clean state.
./hack/install-ate.sh --delete-benchmarks
./hack/install-ate.sh --delete-ate-system
# 2. Install substrate from this branch.
./hack/install-ate.sh --deploy-ate-system
# 3. Install the meter and Prometheus.
kubectl apply -f benchmarking/telemetry/meter.yaml
kubectl apply -f benchmarking/monitoring.yaml
# 4. Send the control plane to the meter.
METER=http://telemetry-meter.benchmarking.svc.cluster.local:4317
./hack/install-ate.sh --deploy-ate-system --otlp-endpoint "${METER}"
# 5. Install the workloads and locust, with the actors on the meter also.
./benchmarking/deploy_locust.sh --deploy --worker-count 10 \
--otlp-endpoint "${METER}"
# 6. Read the volume.
kubectl port-forward -n benchmarking svc/prometheus 9090:9090
```
## Result
The idle floor, from one measurement on 10 workers:
| Service | datapoints/min |
|---|---|
| `ate-controller` | 234.8 |
| `ateapi` | 54.3 |
| `atelet` | 44.9 |
| `atenet-router` | 10.4 |
| `ateom-gvisor` | 5.2 |
| `glutton-actor` | 4.2 |
| **Total** | **353.7** |
The `glutton-actor` row is the correction above. Before it, actor
telemetry could not arrive at all.
Positive control for the same window:
`otelcol_receiver_accepted_metric_points` 10.5/s, and each
`otelcol_receiver_refused_*` counter at zero. Thus the zero values in
the table are true zeros, and not data that did not arrive.
Two values differ from the earlier manual measurement. `ate-controller`
is now the largest source, and `ateom-gvisor` is no longer zero. The new
imagecache counters from #831 are a probable cause. This difference is
the reason to keep results out of the repository.
### Traces
Trace volume follows the root sample decision, not the rate of each
component. `atelet` and `ateom-gvisor` sent zero spans in each run,
because they inherit the decision that `ateapi` makes and start no trace
of their own. At idle each component is near zero: a trace needs an
operation, but a metric pushes on a timer.
This run gives no span volume for a usual load. The actors crashed on
suspend, and the failed retries made an artificial rate of 13.7k
spans/s. That rate is not a measurement of substrate. It does show that
the meter accepted the spans with no refusals.
## Follow-up
A separate PR adds the ladder to `automation/tests.yaml` and writes the
results as artifacts of each run. The open items are at the end of
`observability.md`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
---------
Co-authored-by: Krisztian F <103492698+krisztianfekete@users.noreply.github.com>
### Why
The README Quickstart covers getting a local Substrate cluster running
with
the default gVisor runtime, but there's no documentation for the
micro-VM
runtime, where the friction can't be scripted away: it requires
`/dev/kvm`
(bare metal, nested virtualization, or Lima on macOS), and on Apple
Silicon
the Lima configuration and asset assembly are non-obvious. New
contributors
have to reverse-engineer the `hack/` scripts.
### What
Adds `docs/dev/microvm-local.md` — "Running the microVM runtime locally"
— a
focused guide covering only the microVM delta, based on notes from real
onboarding runs. General setup is deferred to the README Quickstart
(listed
as the guide's prerequisite) rather than duplicated:
- **Option A: Linux host with KVM** — verifying `/dev/kvm` and CPU virt
support, the rootless-Docker caveat for the KVM probe, cluster creation,
and the one-shot `run-microvm-demo-kind.sh` bring-up.
- **Option B: Apple Silicon macOS via Lima** — nested virtualization
with the
guest image pinned to Ubuntu 25.10 (until the kernel issue in the
default
image is fixed), `vzNAT` networking, and assembling the arm64 assets
inside
the Lima guest (`assemble.sh` requires a Linux host of the target arch).
- **Trying it out** — defers to the next steps the demo script prints
and
links the counter demo's micro-VM variant, instead of duplicating those
commands.
- **Troubleshooting** — symptom → root cause → fix entries actually hit
during onboarding (rootless-Docker KVM probe failures, the `aws` CLI
staging requirement until #804 lands, arm64 `virtiofsd` build deps, M1
lacking FEAT_NV2).
OTEL_EXPORTER_OTLP_ENDPOINT is written in 9 places: hardcoded inline in
4 base workload manifests, then re-patched in 5 spots across the kind
overlays. Changing the collector address means editing all 9 and knowing
which install path renders which.
Replace them with one checked-in ConfigMap per environment, consumed by
every component via envFrom:
manifests/ate-install/ate-otel-config.yaml (GKE)
manifests/ate-install/kind/ate-otel-config.yaml (kind, same name)
The GKE copy is listed in manifests/ate-install/base, so token-client,
agentgateway and agentgateway-token-client all inherit it; the kind
overlay lists its own copy of the same ConfigMap name. No overlay builds
on both, so the two never collide.
A kustomize-only fix does not work here. The GKE path applies the base
directory raw and hack/install-ate.sh's targeted redeploys apply single
files with no Kustomize, so the mechanism has to survive `kubectl apply
-f <one-file>` -- which rules out configMapGenerator (hash-suffixed
names) and replacements. envFrom on a stable name reaches every path.
deploy_ate_system applies the ConfigMap before the rendered bundle, the
same way it already applies the namespace. A container whose envFrom
target is missing does not start, and a raw directory apply is ordered
by filename, so ate-api-server.yaml and ate-controller.yaml would
otherwise be created first and sit in CreateContainerConfigError until
the ConfigMap caught up.
This also fixes a latent bug in the targeted redeploys.
deploy_ate_apiserver, deploy_atelet and deploy_atenet apply raw files
even in kind mode, silently reverting the endpoint to the GKE value.
They now apply the environment's ConfigMap via a new apply_otel_config
helper, which picks the file by ATE_INSTALL_KIND rather than applying
the base copy unconditionally -- applying the base one on kind would
break telemetry for every component at once.
atenet-router needs no special handling: since a98f85b1 its
--otlp-collector-address defaults to $OTEL_EXPORTER_OTLP_ENDPOINT, so
Envoy's own spans follow the ConfigMap along with everything else.
OTEL_TRACES_SAMPLER deliberately stays an inline kind patch on
ate-api-server rather than moving into the ConfigMap. The ConfigMap is
shared by every component via envFrom, so putting parentbased_always_on
there would pin the router and the rest of the control plane to 100% and
undo the per-component ratios from 15eecd02.
Note that a ConfigMap edit does not roll the consuming pods the way an
inline env change did, since the pod template is unchanged. Callers must
follow a change with `kubectl rollout restart`. This is documented in
both ConfigMaps, in docs/observability.md, and in the tracing best
practices, which previously told authors to hardcode the variable.
Fixes#745
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Today everything outside ateapi hardcodes `ParentBased(NeverSample)`,
and setting `OTEL_TRACES_SAMPLER` does nothing because an explicit
sampler silences the SDK's env handling. This makes troubleshooting
production issues impossible via traces.
This PR gives every component a sane default and makes the standard OTel
env vars work:
- Control plane (ateapi, atelet, ateom) defaults to
`parentbased_traceidratio` 0.1, the router (data plane root) to 0.01,
per the discussion on #584.
- `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` override any of this
without a rebuild. Invalid values keep the component default and log a
warning instead of inheriting the SDK's fall-open-to-100% behavior.
- Envoy's `RandomSampling` is derived from the router's resolved policy,
so the two root decisions cannot drift.
- kubectl-ate without `--trace` no longer installs a tracer provider at
all: the old `NeverSample` provider injected `sampled=0`, which pinned
every parent based sampler downstream and would have defeated the server
side ratios. `--trace` still forces a full end to end trace.
- ate-controller propagates the two env vars to the ateom worker pods it
creates, same as the metric export vars.
- kind pins ateapi to `parentbased_always_on`, so the local Jaeger flow
keeps showing every API call.
- The agentgateway integration follows what we have above. its
`randomSampling` changes from `true` to `0.01` to match the data plane
default, though as static config it does not follow
`OTEL_TRACES_SAMPLER` overrides, so the two need adjusting together.
Verified on a kind cluster e2e manually. It resolved samplers logged at
startup, a `--trace` resume produced one trace across ateapi, atelet,
and ateom, an unsampled CLI call got picked up server side, Envoy
continued a sampled traceparent while sampling 0 of 30 parentless
requests at the 1% default, and an invalid env value fell back to the
component default.
Gating who may use `--trace` stays a separate follow-up (and a
discussion), and the more sophisticaed tracing policies belongs in a
collector, not in substrate.
Fixes#584
cc. @git286
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Substrate documents how to instrument a service but not how to stand up
the collector it exports to, so pointing it at your own collector meant
reading six manifests.
Covers both topologies: GKE Managed OpenTelemetry as the default on GKE,
with why its gateway Deployment suits Substrate's elastic actor
workload, and a DaemonSet manifest for operators who need per-node
isolation, full config control, or a non-GKE cluster.
Part of #563.
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
- `docs/observability.md`: log label keys were wrong. The code emits
`ate.dev/actor_id` and `ate.dev/actor_template_name`; the doc had
`actor_id` and `actor_template`. The Cloud Logging queries would return
no results.
- `docs/dev/best-practices/tracing.md`: container ports entry used
Docker Compose syntax (`"443:443"`). Changed to `containerPort: 443`.
- `docs/dev/valkey-direct-access.md`: the `kubectl exec` command had a
line break inside `--cert`, making it fail on copy-paste.
- `code-of-conduct.md`: linked to Contributor Covenant v1.4. Updated to
v2.1.
The tracing best practices doc points to
cmd/ateapi/ateapi.go:initTracing() as the example, but that
file/function does not exist. Servers now call
internal/serverboot.InitTracing(), used in cmd/ateapi/main.go.
Verified internal/serverboot.InitTracing exists and is called from
cmd/ateapi/main.go:80.
Assemble + stage the micro-VM runtime assets, an ateom-base image (debian-slim +
e2fsprogs for mkfs.ext4), and run-microvm-demo.sh to build + deploy the
counter-microvm demo end to end (overriding the worker base via KO_CONFIG_PATH so
no committed file is edited). Document the micro-VM sandbox class.
Document the rationale behind each directory and guidelines on how to
decide where to put new code in a dedicated markdown file.
Update AGENTS.md with a summary and a reference to the new markdown
file.
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
This is the initial release of the Agent Substrate.
Agent substrate is a system built on top of Kubernetes which manages agent-like
workloads to achieve higher scale and efficiency than Kubernetes alone can
offer, with lower latency. It builds on top of Kubernetes features like
Pods and Pod autoscaling, but takes the Kubernetes control-plane out of the
critical path to achieve lower latency.
It can run on any Kubernetes cluster and does not inhibit “regular” use of
Kubernetes in any way. Kubernetes provides the infrastructure provisioning and
management for all types of workloads, while Agent Substrate provides
agent-specific scheduling and control.
At its core, Agent Substrate maps a larger set of “actors” (applications such
as agents) onto a smaller set of ready “workers” (Kubernetes Pods), relying on
the fact that agent-like applications tend to be idle most of the time to
achieve heavy multiplexing. It provides functionality to manage an actor’s
lifecycle (e.g. create/destroy, suspend/resume), to assign actors to workers in real
time, and to route incoming traffic to them.
Agent Substrate is intended to be a low-opinion system. The workloads it
manages don't have to be literal AI agents, but those are the best example of
the kind of applications it is designed for. It is not an SDK for building
agents, but rather a system for running them at scale.
Agent Substrate is currently in VERY early development. It is not ready for
production use, and the APIs are almost guaranteed to change. We are not
making any guarantees about backward compatibility at this stage, and
everything in this project may be changed.
Co-authored-by: Alex Bulankou <alexbu@google.com>
Co-authored-by: Benjamin Elder <bentheelder@google.com>
Co-authored-by: Bowei Du <bowei@google.com>
Co-authored-by: Dmitry Berkovich <dberkov@google.com>
Co-authored-by: Fabricio Voznika <fvoznika@google.com>
Co-authored-by: Francisco Cabrera <fclieutier@google.com>
Co-authored-by: Haven Xia <haoyuxia@google.com>
Co-authored-by: Julian Gutierrez Oschmann <juliangut@google.com>
Co-authored-by: Kevin Steuer <ksteuer@google.com>
Co-authored-by: Max Smythe <smythe@google.com>
Co-authored-by: Maya Wang <mymaya@google.com>
Co-authored-by: Michael Taufen <mtaufen@google.com>
Co-authored-by: Shruti Nair <shrutinair@google.com>
Co-authored-by: Taahir Ahmed <taahm@google.com>
Co-authored-by: Tim Hockin <thockin@google.com>
Co-authored-by: Zoe Zhao <zoezhao@google.com>
Tim Bai