Adds **egress credential injection**: on the sdsmint egress gateway's
decrypted MITM leg, when an actor's `EgressPolicy` rule matches and
carries an `inject_static_headers` effect, the gateway resolves the
referenced credential from a **credential provider** and sets it as a
request header (e.g. `Authorization: Bearer <token>`) before
re-originating upstream.
This implements the `applyEffects` in the egress handler. Injector runs
as part of the existing egress ext_proc handler that already
fetches/caches/evaluates each actor's `EgressPolicy`, so the MITM leg
gains injection without a second ext_proc hop.
### How it works
- New `CredentialProvider.FetchSecret` plugin API
(`pkg/proto/credproviderpb`): the gateway calls it with the credential
URI and the actor's attested SPIFFE identity; the provider authenticates
the gateway (mTLS) and resolves the secret.
- The egress handler dials the provider over mTLS
(`egress.DialProvider`) and injects on matched, allowed requests
(`egress.applyEffects`).
- Behavior:
- **TLS MITM leg + provider configured** → inject the credential
(overwriting any actor-set header).
- **Cleartext leg, or no provider configured** → skip injection and pass
the request through (never put a secret on a cleartext wire; don't block
allowed egress).
- **Attempted but failed** (unfetchable secret, empty/malformed secret,
credential URI of an unserved provider class) → fail closed.
## Installation
One flag stands up and wires the whole stack:
```
hack/install-ate.sh --deploy-ate-system --experimental-egress-credential-injection
```
`--experimental-egress-credential-injection` deploys credprovider and
the injector, then re-wires the sdsmint egress gateway to route through
the injector. It implies `--experimental-use-sdsmint`
### New install flags
Two flags configure which credential provider the injector targets:
| Flag | Purpose | Default |
|---|---|---|
| `--credential-provider-name` | Provider class, as a `ate-secret://`
prefix; a policy URI of any other class is refused |
`ate-secret://kubernetes.io` |
| `--credential-provider-address` | Where the injector dials the
provider | `credprovider.ate-system.svc:50051` |
To build a snapshot URI, we need the external storage location (which is
stored in the actor template), actor atespace and UID (which are stored
in the actor resource). If an actorTemplate was deleted, we'd leak the
in-progress snapshot, because the storage information was gone.
Fixes#1608
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
#1522 added a resyncInterval parameter to NewActorTemplateReconciler and
#1523 added this call site. They landed without seeing each other, so
`main` doesn't compile.
SEt the resyncInterval to 7s, to match what we have in other tests
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1507
Golden snapshots currently remain owned by the temporary golden actor,
so another resume/suspend cycle or actor deletion can collect a snapshot
still referenced by its template. The controller now copies the warmed
snapshot into a published tag, deletes the golden actor, and records the
tag reference on the template. Interrupted tag creation and cleanup
remain retryable; template deletion cleans up both resources.
`CreateActor` resolves an explicit `sourceTag` or the template's golden
tag into the actor's initial snapshot. Actors created before the golden
tag is ready retain their cold-boot behavior. The golden tag uses the
template UID as its name in `ate-golden`. The proto replaces
`golden_snapshot` with `golden_tag` at field 1, without backward
compatibility.
This PR is based directly on `main` and does not depend on #1521.
Follow-up recommendation: move the create → resume → wait → suspend →
tag → delete sequence into a golden-template workflow using the existing
workflow conventions. The reconciler now coordinates multiple
recoverable steps; it could retain scheduling and retries while
delegating that sequence to the workflow. This refactor is outside this
PR.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Validation: full `env -u NO_COLOR make verify` passed after rebasing
onto `main`. After the final proto field-number change, bindings were
regenerated and the control API unit/functional tests plus proto-format
and Go-format checks passed.
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Actor template discovery currently uses a fixed 20-second resync
interval. This PR adds `--template-resync-interval` so deployments can
tune that delay, preserving the `20s` default and rejecting nonpositive
values. The setting also controls the reconciler's existing fallback
retry delay.
This is intentionally a small change to start a discussion about how
template builds should be dispatched as the template catalog grows.
The current resync fetches and decodes every template, including
completed ones. At 100,000 templates, a 20-second interval implies
roughly 5,000 template rows read per second per replica, assuming scans
finish quickly. This is an estimate from the code, not a benchmark.
Possible follow-ups:
- **Immediate enqueue:** start work after creation, retaining a slower
recovery scan for crashes between persistence and enqueue.
- **Outbox/watch:** consume changes instead of scanning the catalog. The
existing worker outbox still polls every 50 ms and broadcasts events to
each subscriber; write overhead and recovery scans need consideration.
- **Durable pending-build queue:** atomically record work, claim due
jobs using short `FOR UPDATE SKIP LOCKED` transactions, and recover
expired leases. This still polls, but queries pending work rather than
the full catalog. `LISTEN/NOTIFY` is not the proposed default because it
serializes notifying commits.
- **Imperative build / long-running operation:** give callers explicit
build control or a progress/completion handle. Either still needs
reliable execution underneath.
The main question is whether template builds need a broadcast change
feed or a queue where replicas claim different jobs. Neither alternative
is implemented here.
[Full research: database costs, execution options, recovery
requirements, and
sources](https://gist.github.com/EItanya/0a1d6893ede34f2f0e9d9d1929ecb823).
Validation: all Go race tests and repository verifiers passed using
module mode with `NO_COLOR` unset. CLI checks confirmed the default and
rejection of zero/negative intervals; the existing reconciliation test
checks a custom interval.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Populate default fields during both Create and Update RPC calls so
that resources are stored with defaults instead of zero values.
This is a much cleaner approach as it means that a) consumers don't need to
be aware of the default value and b) default values can potentially change.
Only ActorTemplate had defaults, but added default functions for all
resources so that all RPC handlers are wired up.
The egress gateway is about to enforce EgressPolicy, which means two
components now interpret the same hostname patterns and CIDRs: ateapi
when it validates a policy, and atenet when it matches one. Putting the
parsers in internal/egresspolicy and having ateapi's custom validators
call them keeps "accepted by the API" and "enforceable by the gateway"
the same set by construction. The package also compiles a policy once
and evaluates it the way the API describes, in rule order over whatever
the caller knows of the destination, first match wins: a hostname rule
never matches a destination with no name, and an IP rule never matches
one whose address the caller does not know. The decision says which
kind of rule matched, because the gateway sends the bytes to what that
rule checked. HasHostnameRules answers the question that follows, which
is whether a decision point holding only an address should refuse
outright or leave the destination to one that will see a name.
Hostname patterns now reject any name whose last label is all digits,
per RFC 1123 section 2.1, instead of consulting an IP parser. That is
the same set as before for every real address and additionally refuses
dotted-quad spellings such as "01.2.3.4" that no parser accepts.
The actor SPIFFE ID moves to internal/resources with a parser next to
the builder: ateapi mints it into the certificate's URI SAN and the
egress gateway reads it back as the actor identity on the legs inside
the tunnel, so both ends need the one definition.
# Record actor state changes from ateapi
ateapi now writes a log record every time an actor changes state.
## Before
Nothing tells you what state an actor is in. Metrics can't carry actor
identity, and the control plane samples traces at 10%, so most
transitions leave nothing behind at all. If you want to know whether an
agent is running, paused or suspended, there's nowhere to look.
## After
One filter:
```text
ate.actor.uid="8f2a1c4e6b0d47f1" AND ate.actor.state!=""
```
Last record wins. That's the state it's in now, and the timestamp is
when it got there.
The record:
```json
{
"time": "…",
"level": "INFO",
"msg": "Actor state changed",
"ate.atespace": "ate-demo-counter",
"ate.actor.name": "counter-1",
"ate.actor.uid": "8f2a…",
"ate.template.atespace": "ate-demo-counter",
"ate.template.name": "counter",
"ate.actor.operation.name": "suspend",
"ate.actor.state": "suspended"
}
```
## The two new keys
`ate.actor.state` is the `ateapipb.ActorState` values lowercased, so the
log wording and the state machine can't drift apart.
`ate.actor.operation.name` says what caused the change. The state on its
own doesn't tell you: an actor lands in `suspended` from a suspend and
`paused` from a pause, and only one of those gives the worker back.
`Actor crashed` picks up the same state key. A crash is a state change
too, and it's the one an actor can reach without any operation
finishing.
## Notes for review
**Why ateapi.** It owns the state machine. It also sees transitions that
never reach a worker, like deleting an actor that was already suspended,
or a resume that fails in the scheduler.
**It can't log a state the store never had.** The record goes out after
the store commit, and every state commit has a version precondition, so
a losing writer in a concurrent update writes nothing.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Changing the sandbox_class requires creating a new worker, you can't
change the sandbox_class of a running worker.
- [ x ] Tests pass
- [ x ] Appropriate changes to documentation are included in the PR
Fixes a few todos in the `ateapi.proto`
* Adds custom validation for `create_time` and `update_time` fields.
* Adds better validation method for the `Container.image` field.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Fixes#1432
Map known store pagination errors in `ListActorTemplates` through
`mapListError`, matching the other list RPC handlers. Invalid page
tokens now return gRPC `InvalidArgument` with `invalid page_token`.
- [x] Tests pass: ran `make verify`
- [x] Appropriate changes to documentation are included in the PR:
single line change
Fixes https://github.com/agent-substrate/substrate/issues/1566
Today the Resume workflow resolves its restore source in the following
order
1. First check if the actor has node-local snapshot,
2. then its own durable external snapshot,
3. then the template's golden snapshot.
The boot flag was consulted at exactly one point in that chain, where it
suppressed using the golden-snapshot, which made its behavior much
narrower than "boot from scratch" suggests:
- Actor has its own external snapshot and boot=true: flag ignored,
restores the actor's snapshot.
- Actor has a local snapshot and boot=true: flag ignored, restores the
local snapshot.
- Actor has no snapshot, template has no golden snapshot: cold boot from
the spec regardless of the boot flag.
- Actor has no snapshot, template has a golden snapshot: boot=false
restores the golden, boot=true cold boots from the spec. This is the
only case where the flag is used.
The glutton benchmark was the only caller that set boot=true, on each
actor's first resume, to report true cold-start latency as a separate
ResumeActorColdStart stats row. @maxsmythe let me know if this is
required.
The proto field number and name were reserved.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
I'm cleaning up the ResumeActor workflow in
https://github.com/agent-substrate/substrate/pull/1492/changes . This PR
is to add tests first to make sure the refactor does not introduce any
behaviour changes.
Testing table:
| \# | local (OnPause) | external (scope) | tmplUID | golden (scope) |
fromData | boot | want |
| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |
| 01 | — | — | — | — | unset | f | Run RPC (cold boot), Restore not
called |
| 02 | — | — | — | — | GOLDEN | f | still Run: fromData only governs
data-only snapshots; nothing to restore ⇒ golden branch skipped |
| 03 | — | — | — | FULL | unset | f | golden fallback: Restore EXTERNAL,
uri=goldenURI, Scope=FULL, golden="" |
| 04 | — | — | — | FULL | unset | t | boot=true skips golden: Run RPC
(fromData irrelevant here too) |
| 05 | — | — | — | FULL | GOLDEN | f | same as 03 (no actor snapshot ⇒
not dataOnly ⇒ GoldenSnapshotUri stays empty) |
| 06 | — | — | — | DATA | unset | f | FailedPrecondition — golden
fallback rejects a non-Full golden |
| 07 | — | — | — | FULL, malformed URI | unset | f | DataLoss — golden
fallback, malformed golden URI |
| 08 | — | — | mismatch | FULL | unset | f | golden fallback unaffected:
Restore EXTERNAL, uri=goldenURI, Scope=FULL (TemplateReplaced needs a
durable snapshot) |
| 09 | — | FULL | — | — | unset | f | Restore EXTERNAL, uri=actor
snapshot, Scope=FULL |
| 10 | — | FULL | — | FULL | GOLDEN | f | Full snapshot ignores policy:
Scope=FULL, golden="" |
| 11 | — | FULL | match | — | unset | f | matching UID ⇒ no downgrade:
Scope=FULL |
| 12 | — | FULL | mismatch | — | unset | f | repointed ⇒ Restore
EXTERNAL, Scope=DATA, golden="" |
| 13 | — | FULL | mismatch | FULL | GOLDEN | f | TemplateReplaced beats
the golden policy: Scope=DATA, golden="" |
| 14 | — | DATA | — | — | unset | f | Restore EXTERNAL, Scope=DATA,
golden="" |
| 15 | — | DATA | — | FULL | GOLDEN | f | Restore EXTERNAL, uri=actor
snapshot, Scope=DATA\_ON\_GOLDEN, golden=goldenURI |
| 16 | — | DATA | — | DATA | GOLDEN | f | FailedPrecondition — Golden
data resume, golden not Full ("regenerate the golden snapshot") |
| 17 | — | DATA | mismatch | FULL | GOLDEN | f | TemplateReplaced beats
DATA\_ON\_GOLDEN: Scope=DATA, golden="" |
| 18 | — | UNSPEC | — | — | unset | f | Restore EXTERNAL, Scope=FULL
(conversion maps UNSPECIFIED → Full) |
| 19 | — | malformed URI (FULL) | — | — | unset | f | DataLoss — actor's
own external snapshot URI fails to parse |
| 20 | FULL | — | — | — | unset | f | Restore LOCAL,
LocalConfig.SnapshotName, Scope=FULL, golden="" |
| 21 | FULL | — | — | FULL | GOLDEN | f | Full pause ignores policy:
Restore LOCAL, Scope=FULL, golden="" |
| 22 | FULL | — | mismatch | — | unset | f | Restore LOCAL, Scope=FULL —
repoint detection requires a durable snapshot, local-only actors never
downgrade |
| 23 | DATA | — | — | — | unset | f | Restore LOCAL, Scope=DATA,
golden="" |
| 24 | DATA | — | — | — | GOLDEN | f | FailedPrecondition — Golden data
resume requires a golden snapshot, template has none |
| 25 | DATA | — | — | FULL | GOLDEN | f | Restore LOCAL,
Scope=DATA\_ON\_GOLDEN, golden=goldenURI |
| 26 | DATA | — | — | DATA | GOLDEN | f | FailedPrecondition — golden
not Full, local-snapshot path |
| 27 | DATA | — | — | UNSPEC | GOLDEN | f | UNSPECIFIED golden scope
accepted as Full: Restore LOCAL, Scope=DATA\_ON\_GOLDEN |
| 28 | DATA | — | — | FULL, malformed URI | GOLDEN | f | DataLoss —
golden URI fails `ParseSnapshotURI` |
| 29 | DATA | FULL | — | FULL | GOLDEN | f | local wins over external:
Restore LOCAL, Scope=DATA\_ON\_GOLDEN (dataOnly from OnPause, not the
external scope) |
| 30 | DATA | FULL | mismatch | FULL | GOLDEN | f | TemplateReplaced
wins in the local branch too: Restore LOCAL, Scope=DATA, golden="" |
This was originally a separate service to make it easy to apply separate
authentication and authorization interceptors.
It now seems clear that our authn/z framework will be strong enough to
support atelet and external callers in one system (based on OpenFGA).
This change moves the MintJWT and MintCert RPCs into the control API,
and removes some inline authz checks that will be handled by our unified
authorizer framework.
The container image must include the image digest. This validation was
dropped during the migration of the ActorTemplate resource from CRD to
Substrate API.
Fixes#1475
`Control.CreateWorker` does not validate `Worker.sandbox_class`, so a
client can register a worker with an empty one (and any record written
before this keeps it). `RegisterWorkerCount` tallied that raw value, so
`ate.workerpool.workers` emitted `ate_sandbox_class=""` — not a member
of the `ate.sandbox.class` registry vocabulary — alongside the pool's
seeded series at 0.
This change normalizes the worker's class with the existing
`ateattr.NormalizeSandboxClass`, so an empty class reports as `unknown`.
New `ateattr.SandboxClassAttribute`, used by every emitter. This closes
the same gap on `ate.actor.crashes` and the `ate.actor.lifecycle.*`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1539
If an actor is `CRASHED`, it can't be resumed/paused/suspended anymore,
it can only be deleted.
An actor deletion will clean up everything under the prefix of
`Actor.external_snapshot.snapshot_uri` and
`Actor.in_progress_snapshot_name` (the latter will be empty if the
underlying worker was deleted).
Before, we risked leaking a snapshot after a worker deletion because we
were clearing the `in_progress_snapshot_name` field. We don't need to
clear it up because the CRASHED state is terminal.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1508
Store tag snapshots at `<base>/atespaces/<atespace>/tags/<tag-uid>`.
Replace `in_progress_snapshot_uri` with immutable `storage_location`, so
pending and completed tags share UID-based cleanup independent of the
source actor or template.
- [x] Tests pass: race-enabled control API tests and PostgreSQL tag
contract tests.
- [x] Documentation updated.
Lint and code-generation verification also passed.
---------
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
## Summary
Fixes the flaky test `TestActorLifecycle_WithExternalVolumes` in
`cmd/ateapi/internal/controlapi/functionaltest` (`unit` lane).
**Root cause:** the test pauses and immediately resumes against a single
1-capacity worker; PauseActor commits the assignment release to the
store, but AssignWorker reads the worker cache, which sees the release
only after the PostgreSQL watch delivers it — losing that race fails the
resume with `ResourceExhausted: no free workers available`.
**Fix:** wait for the release to reach the cache with the existing
`waitForWorkerAvailable` helper (the same pattern sibling lifecycle
tests use after suspend) before resuming.
Closes#1533
## Evidence
Flake rate over last 7 days: 5.7% (26 fail / 427 pass across 453 runs)
Failures span 14 branches incl. 9 on main; infra-failure runs excluded:
0
Failing runs:
https://github.com/agent-substrate/substrate/actions/runs/33919495958,
https://github.com/agent-substrate/substrate/actions/runs/33941175484
Passing runs:
https://github.com/agent-substrate/substrate/actions/runs/33567474203,
https://github.com/agent-substrate/substrate/actions/runs/33566785143
## Test plan
- [x] `go test -race -count=5
./cmd/ateapi/internal/controlapi/functionaltest/ -run
TestActorLifecycle_WithExternalVolumes` — passes (was intermittently
failing before at ~6%)
Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>
`main` is red since #1480 merged: its regenerated
`cmd/ateapi/internal/controlapi/zz_generated.validation.go` was produced
by a stale generator build and references `deepEqualImpl_`, which
doesn't exist — `cmd/ateapi` (and everything importing `controlapi`)
fails to compile, and every open PR inherits the failure through CI's
merge-with-main (see #1512's `run-tests` for an example, and `main`'s
own post-merge `pr-workflow` runs for #1480/#1499).
Fix is a pure regeneration: `hack/update/codegen.sh` on current `main`
changes exactly one line, `deepEqualImpl_` → `ateDeepEqual` (the helper
the current generator actually emits). After it, `go build
./cmd/ateapi/...` succeeds and `go test
./cmd/ateapi/internal/controlapi/` passes.
No hand-written changes — the diff is generator output only.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The changes we need are live on HEAD of the k8s published repos. It was
a bit of a journey, since Go fought me all the way.
---
### Drop our third_party fork of k8s deps in tools
The changes we need are released now (sort of - on HEAD anyway).
---
### Bump k8s.io/streaming to v0.37.0 (not rc) in tools
---
### Bump codegen deps to HEAD in tools
GOPROXY=direct go get \
k8s.io/apimachinery@master \
k8s.io/code-generator@master
This pins them to HEAD of master. Go is terrible here: The HEAD is not
actually tagged, so Go just uses the next "reachable" tag which is
v0.36.0-alpha. The datestamp is correct, though.
---
### Drop our third_party fork of k8s deps in root
The changes we need are released now (sort of - on HEAD anyway).
---
### Bump k8s deps to v0.37.0 (not rc) in root
---
### Bump apimachinery dep in root to HEAD in root
GOPROXY=direct go mod edit -replace
k8s.io/apimachinery=k8s.io/apimachinery@master
GOPROXY=direct go mod tidy
GOPROXY=direct go mod vendor
GOPROXY=direct go mod tidy
This approach (-replace) is needed because Go is horrible here.
The master branch of k8s.io/apimachinery is not tagged, per se, but
there is an OLDER tag which is "reachable" from HEAD. So go helpfully
decides to use that (v0.36.0-alpha.2). If we just `go get ... @master`
it works for that dep (pinned to the right date) but then it looks at
transitive deps. Because the tag seems to be 0.36 (older), it
recalculates all the OTHER dependencies and downgrades a whole tangle of
things to versions that match 0.36, but we are ACTUALLY on 0.37+.
This was the only approach that I (and Gemini) could find. Blech.
---
### Run updated codegens
---
### Use DV's new maxBytes capability for `[]byte`
Removes 1 custom.
## What
Now that the ActorTemplate CRD has been deleted and its resources moved
to the substrate gRPC API and the control-plane store (created/managed
with `kubectl-ate`, persisted in PostgreSQL), several documents still
describe ActorTemplate as a Kubernetes CRD, or describe namespace/RBAC
relationships that no longer exist. This sweeps the docs for those stale
references.
Fixes#368 (docs side).
> This change was prepared with AI assistance; I have reviewed and
tested it.
- [x] Docs and comment-only change; no functional code changed, no tests
affected
`CreateActorTemplate` accepted `../escape` and `/escaped` in
`ActorMetadataItem.path` and `TrustBundleDataSource.path`, and accepted
several data sources projecting to the same path. atelet only rejects
the traversal at actor start, so the template is created but the golden
actor sits in `RESUMING`. atelet never rejects duplicate paths at all:
it writes the files in order and the last one wins.
This PR moves both checks to template validation.
- The path rule lives in a new `internal/volumepath` package that both
ateapi and atelet call, so the two cannot drift. It splits on `/` and
rejects any segment that is empty, `.`, or `..`, caps depth at 16
segments, and rejects NUL bytes. Any other byte is allowed, so Unicode
paths pass. `ActorMetadataItem.path` and `TrustBundleDataSource.path`
apply it through a `+k8s:customValidation` hook, and atelet's
`writeSystemInfoFile` applies it again before touching the host
filesystem.
- `SystemInfoVolumeSource.data_sources` gets a hook that tracks
projected paths in a set and reports a duplicate on repeats, in the same
shape as the Kubernetes projected-volume validation.
The proto doc comment on `data_sources` also says at most one
`actor_metadata` entry may appear. Nothing enforces that, and this PR
leaves it alone.
New cases in `TestValidateSystemInfoVolumeSource` and
`TestValidateTrustBundleDataSource` cover the rejected shapes, and
`internal/volumepath` has its own table test. `hack/verify/codegen.sh`
and `hack/verify/golangci-lint.sh` pass.
#1231 moves `writeSystemInfoFile` into `systeminfovolume.go` and onto
`os.Root`. Whichever PR lands second points that copy at the shared
rule.
Fixes#1436Fixes#1438
Renames the proto message, its status message and scope enum, the five
RPCs and their request and response messages, the actor_snapshot_tag
request fields, and Actor.source_snapshot_tag to source_tag. The store
interface, its Postgres table, the object-storage prefix segment and the
kubectl-ate verbs follow, so nothing keeps the old spelling.
https://github.com/agent-substrate/substrate/issues/664
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#664
This PR implements the idea described in
https://github.com/agent-substrate/substrate/issues/664#issuecomment-5499311489
It does more than Garbage Collection of snapshots, because we also got
rid of the Snapshot resource (from the DB/API).
Now, an external snapshot is owned by a single resource:
- An Actor owns the snapshot it writes at suspend
- A tag owns a copy taken at tag creation,
- An actor cloned from a tag borrows the tag's snapshot until its own
first suspend.
Garbage Collection: whoever created/owns the snapshot is the only one
who ever deletes them:
i.e., if an actor is deleted and it owns a snapshot. The underlying
snapshot is deleted with the actor.
this PR:
- Drops table actor_snapshots
- Keeps table actor_snapshot_tags
- Adds an object copy at tag creation, and an owned versus borrowed
distinction on the Actor
- Adds synchronous external snapshot deletion at actor suspend, at actor
delete, and at tag delete
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #1266
This is a draft of the core API + data store changes.
It's still a large PR, apologies.
The "as rows" commit could be split out, but this takes it to ~all of
the breaking changes we can't hide behind updating internals.
Same for the claimlock, but in both cases it seems these are worth
understanding when considering the API shape.
They're loadbearing for performance once we actually have multi-actor
workers.
This PR moves sandbox config selection from the WorkerPool to the
ActorTemplate.
The existing behavior is preserved while we are designing the upgrade:
sandbox config still cannot be updated once set (ActorTemplates are
create-only and `sandbox_config` is immutable).
For now the ActorTemplate still *requires* `sandbox_config.config_name`
— there is no resolution of the cluster default (`spec.default`). This
is temporary while we figure out the defaulting design.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
`ate.failure.reason` only ever named infrastructure faults, and the
crash log didn't say which actor crashed.
This PR adds the following:
- **`ate.failure.domain`** (`infrastructure` / `workload` / `unknown`)
is available next to every `ate.failure.reason`. It's a strict function
of the reason so it costs no series, but it's *emitted* rather than
derived downstream, so a component ahead of ateapi can report a reason
this build rejects, which collapses to `UNKNOWN`, and a consumer
matching on reason names would file every one of those as infra.
`UNKNOWN` now reports `unknown`, not `infrastructure`.
- **`Actor crashed`** record from ateapi, with full identity including
the uid, via a helper both crash sites share. It sits under the same
guard as the counter so the two can't disagree on how many crashes
happened. The counter is barred from carrying actor identity, so this
record is the only way to attribute a crash to one agent.
- **`WORKLOAD_NOT_READY`** on the readyz deadline, as the first
workload-domain reason. Deadline branch only: a cancellation is ateom
draining, not the actor failing.
Tested locally:
```
desc = WORKLOAD_NOT_READY: readyz for "counter" never returned 200 within 5s
```
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Follow the style for all other enum validations.
Fix#1441
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Fixes https://github.com/agent-substrate/substrate/issues/477 Implements
the following actor template update flow:
```
UpdateActor(): set .template = templ-v2
ResumeActor()
resumes using templ-v2
wait for readyz
if fail: return failure
.template = templ-v2; .status.current_template = templ-v1
(the next resume will attempt templ-v2 again)
set .status = STATUS_RUNNING
set .status.current_template = templ-v2
return success
.template = templ-v2; .status.current_template = templ-v2
```
## Summary
A 15s SuspendActor trace showed most of the time in three windows that
tracing cannot attribute: the FinalizeSuspended step span wraps up to
five store round-trips with no spans of their own, and the lease
acquire/release around the workflow run entirely outside the step spans
(2.1s before the first step, 1.15s after the last one in the observed
trace).
This PR logs a per-call duration breakdown in each window:
- **FinalizeSuspended** logs the durations of its store calls (get,
worker release, re-fetch, snapshot create, final update), emitted from a
`defer` so a call that stalls and then fails still reports where the
time went.
- **AcquireLease** logs the expired-lease cleanup and the acquire upsert
separately; the cleanup DELETE scans the whole leases table and contends
with concurrent acquires and releases.
- **Lease close** logs the renewal-goroutine stop and the release DELETE
separately.
## Test plan
- Logging-only change; `make test` passes and existing suspend/lease
tests cover the touched paths.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
- Use direct host bind mounts for durable directories in gVisor
containers instead of internal overlay filestores.
- Move tarutil into internal/tarutil so it can be shared across ateom
packages.
- Archive durable directory bind mounts into durable-dir.tar on full and
data checkpoints (replacing runsc fscheckpoint).
- On restore, unpack durable-dir.tar into the host bind mount before
starting/restoring containers.
- Support SNAPSHOT_SCOPE_DATA_ON_GOLDEN in ateom-gvisor, combining
golden guest state with latest actor durable data.
- Allow atelet to narrow full gVisor captures to durable-dir.tar for
data snapshots.
- Remove CEL validation disallowing onResume.fromData: Golden for
gVisor, and enable combined restore demo tests for gVisor.
Fixes#451
Fixes#368 . Deletes ActorTemplate CRD and any references to it.
Note about atenet router: It had a k8sclient controller that monitors
ActorTemplate, but the results were not used. Deleted as well.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Follow up on issue #1168 and the base PR #1215
In this PR:
**Actor read/lifecycle verbs — completes Actor end-to-end**
* `Get`/`Delete`/`Suspend`/`Pause`/`Resume` `ActorRequest`: dropped the
`+k8s:opaqueType` fence on the actor ref, added `+k8s:required` +
`+k8s:subfield(atespace)=+k8s:required`. Combined with the recursion
into `Validate_ObjectRef` this reproduces exactly what
`resources.ValidateObjectRef` enforced (ref required, atespace required,
name required, both DNS-1123 labels).
* `ListActorsRequest`: tagged to match `ListAtespacesRequest` (atespace
optional + short-name format, `page_size` minimum 1, `page_token` max
length 256).
* All six `validate*Request()` functions are now one-line calls to the
generated validators. With this, every RPC verb on `Actor` and
`Atespace` is fully declarative; `resources.ValidateObjectRef` has no
callers left in `actor.go`.
**Note:**`ListActorsRequest.page_token` now capped at 256 chars (parity
with `ListAtespaces`).
**Testing**
* Existing verb tests updated to the generated error shapes
(`field.ErrorMatcher` with origin matching).
* New coverage: `page_token` boundary cases for `ListActors`;
* `controlapi`, `functionaltest`, `actoridentity`, `store`, and `atepg`
suites all pass; `go vet` and `gofmt` clean.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
---------
Co-authored-by: Michelle Au <msau42@users.noreply.github.com>
Continues the declarative-validation (DV) migration from #1215, covering
the `ActorTemplate` resource, its full spec tree, the create path, and
read/delete verbs.
**Key Changes**
* **DV Tags & Hooks**: Added validation tags and custom hooks across the
`ActorTemplate` spec (`Metadata`, `SandboxConfig`, `SnapshotsConfig`,
`Container`, `Volume`, `Resources`).
* **Create Path**: Replaced hand-written validation with generated
validators in `CreateActorTemplate`. Custom rules (e.g., `on_commit ⊆
on_pause`) are now handled via custom hooks.
* **Read/Delete Verbs**: Converted `Get`, `List`, and `Delete` requests
to use DV.
* **State Updates**: Refactored `atepg` metadata setters to update
in-place.
* Regenerated apitool exemptions (documented 15 previously-exempt
fields).
**Deliberate Behavior Changes**
The gRPC path now strictly enforces CRD rules. Specific tightenings
include:
* Empty `worker_selector` is now rejected.
* `EnvVar.name` is strictly required.
* Negative values in optional enum fields are rejected (`minimum=1`).
* `page_token` is capped at 256 characters (matching other list
requests).
**Deliberately Not Done**
* **No update DV / status tags**: `ActorTemplates` are immutable to
clients. The reconciler updates the status directly against the store,
so tagging the status subtree or adding update validation isn't
necessary right now.
* **`config_name` matching `sandbox_class`**: Enforcing this requires
calling SandboxConfigLister (ServiceImpl), so it's left as a TODO.
**Testing**
* Added ~70 positive and negative test cases in the validator tables.
* Store-backed tests (`TestCreateActorTemplate*`,
`TestUpdateActorTemplateMetadata`, and the `atepg` suite) verified
against real Postgres via testcontainers.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This PR is large, I grouped changes to the following commits:
- 1d5d8f06: moves consumers off the CRD path: demos, ate-setup scripts
and the e2e suites address templates by the actor_template ref.
- e502cf60: Ate API changes: removes actor_template_namespace,
actor_template_name from Actor, ActorAssignment and ActorSnapshot in the
public API, drops the CRD conversion fallback.
- 8f6c1adb: atecontroller: removes the ActorTemplate CRD controller.
Once this PR is submitted, existing demos that still uses CRD will stop
working. Created https://github.com/agent-substrate/substrate/pull/1355
to update existing demos.
Follow up on issue #1168 and the base PR #1215
**Overview:**
Continues the declarative validation (DV) migration from #1215 by
converting the `Worker` resource. This moves immutable-field enforcement
from the storage layer up to the service layer. With `WorkerPoolSyncer`
now using RPCs, nearly all write paths share the exact same validation.
**Key Changes:**
* **Schema (`ateapi.proto`):** Added full DV tags (required, format,
immutable) to `Worker` pod-coordinates and metadata.
`WorkerStatus.state` is now strictly bounded to its enum range.
* **Service Layer:** Replaced ~60 lines of hand-written validation with
1-line generated calls. `CreateWorker` and `UpdateWorker` now scrub
server-owned fields and enforce immutability *before* hitting the store.
* **Storage Layer:** Removed `store.CheckWorkerMutation`.
`atepg.UpdateWorker` is optimized to only clone metadata instead of the
whole worker. Immutability contract tests were moved to the service
layer.
* **Behavior Tweaks (Tightenings):**
* `page_token` is now capped at 256 chars.
* `DeleteOptions.uid` must be a valid UUID.
* Global-ref atespace violations now correctly return `Forbidden`
instead of `Invalid`.
* Immutability checks now natively cover the `WorkerPoolSyncer` path.
* **Testing:** Added comprehensive positive/negative cases for all newly
tagged fields, boundary cases, and moved the immutability test suite.
All suites pass cleanly.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
yufan-su