With --image-repo the image tag becomes the substrate version, which
names the atelet DaemonSet and the node label. A tag written with its
digest, v1@sha256:..., is a valid image reference but not a valid label
value, so the install failed at the atelet step unless VERSION was also
set. The version is now the tag alone; the digest still pins the image.
Followup for #1391
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Two follow-ups to the pre-built image install.
Digest lookups authenticated with the docker config file alone. Someone
installing a published release onto GKE, whose Artifact Registry
credentials are
gcloud's rather than a credential helper wired into
`~/.docker/config.json`, got
a working build from source and a 401 from `--image-repo`. The lookup
keychain is
now `authn.NewMultiKeychain(authn.DefaultKeychain, google.Keychain)`,
the same
relative order ko uses. `DefaultKeychain` stays first, so
`~/.docker/config.json`
still wins wherever it has an entry, and `google.Keychain` returns
`Anonymous`
immediately for any non-Google registry, so this is a no-op for Docker
Hub, ghcr,
ECR, ACR and the Kind local registry. ECR and ACR would need
credential-helper
modules this repository does not depend on, so those still need a
`docker login`;
the doc comment no longer claims the lookup authenticates the way ko
does.
Separately, `pin` appended the looked-up digest unconditionally, so
`--image-tag v1@sha256:...` produced
`<repo>/<image>:v1@sha256:...@sha256:...`,
which no registry can parse. A tag that already carries a digest now
passes
through as written.
No dependency change: `pkg/v1/google` was already vendored, and `atelet`
already
uses the same package for its image pulls.
Verified against Artifact Registry with `DOCKER_CONFIG` pointed at an
empty
directory, which reproduces the affected setup — without this change the
lookup
fails with `DENIED: Unauthenticated request`, with it the tag resolves.
Also
installed on a Kind cluster from a local registry, where all five
control-plane
images came up pinned to the digests the registry published.
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>
> [!WARNING]
> Recreate PostgreSQL databases from earlier development builds.
## Summary
This change replaces startup schema setup with embedded, versioned SQL
migrations.
`ateapi` uses Goose to apply migrations before readiness. Goose stores
one ledger record for each applied migration.
Goose runs each migration and inserts its ledger record in one
PostgreSQL transaction.
Closes#901.
Based on this design:
https://docs.google.com/document/d/13ixDKRoAIFXeLxS-_1nikNcAy76ca8m0eVgobfib93E/edit?usp=sharing
## Migration behavior
`ateapi` gets a session advisory lock for the configured schema before
it applies pending migrations.
One replica applies migrations while other replicas wait. Goose reads
the ledger again after it gets the lock.
If a migration fails, PostgreSQL rolls back its SQL and ledger record.
Earlier successful migrations remain applied and recorded.
Kubernetes restarts the failed replica. The next startup resumes from
the first migration without a ledger record.
## Changes
- Add Goose and a per-migration ledger.
- Replace the initial up and down files with one transactional, up-only
migration.
- Keep migration 1 aligned with the current schema, including actor
egress policy storage.
- Remove existence guards and explicit transaction statements from the
baseline migration.
- Apply all pending migrations before `ateapi` becomes ready.
- Serialize each migration run with a PostgreSQL session advisory lock.
- Start without changes when the database schema is current or ahead.
- Reject application tables that do not have a migration ledger.
- Log the starting, current, and latest versions.
- Log the applied migration count and duration.
- Retry only initial database connection failures.
- Return schema and migration errors without a retry.
- Add `--postgres-schema` and `ATE_API_POSTGRES_SCHEMA`.
- Use `public` as the default PostgreSQL schema.
- Use the configured schema for the main and watch pools.
- Restrict outbox partition maintenance to the configured schema.
- Let the installer use an external PostgreSQL database.
- Add the migration design and recovery policy to the repository.
## Migration file policy
Migration files use sequential versions and contain exactly one Goose
`Up` section.
CI rejects down migrations, nontransactional migrations, environment
substitution, explicit transaction control, and `IF NOT EXISTS` guards.
Before the first stable v1 release, developers can change or squash
migrations. Developers must recreate databases after migration history
changes.
After that release, CI rejects changes or deletions against the latest
stable release tag that contains migrations.
Goose does not store migration checksums. The binary embeds each
migration file, and release-tag checks protect released migration
history.
## Compatibility
No release includes PostgreSQL support. The `v0.0.0` release predates
the PostgreSQL backend.
Users must recreate databases from earlier PostgreSQL development
builds.
Every committed migration prefix must work with the current and previous
`ateapi` releases. This rule supports rolling upgrades and temporary
binary rollback.
A binary rollback does not roll back the database schema.
## Testing
Tests cover:
- Fresh database migration.
- Concurrent startup.
- Advisory lock waits.
- Current and ahead database schemas.
- Rejection of application tables without a migration ledger.
- Atomic rollback of a failed migration.
- Retention of earlier successful migrations.
- Resume from the failed migration after restart.
- Configured schema isolation.
- Outbox partition isolation.
- Migration file policy checks.
- Stable release migration immutability.
Fixes#368 . Deletes ActorTemplate CRD and any references to it.
Note about atenet router: It had a k8sclient controller that monitors
ActorTemplate, but the results were not used. Deleted as well.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1351
`Bump Go to 1.27` (69828945) raised the root `go.mod` and every
`hack/tools/*/go.mod` to `go 1.27.0`. `benchmarking/deploy_locust.sh
--deploy`
now fails at the `boomer-glutton` build:
go: go.mod requires go >= 1.27.0 (running go 1.26.7; GOTOOLCHAIN=local)
The `goboomer` stage builds `FROM golang:1.26-bookworm`, and the
official
`golang` images set `GOTOOLCHAIN=local` in their image config. That
disables
Go's toolchain resolution, so the base image tag becomes a second place
where
the project's Go version is declared. It drifted out of sync with
`go.mod` at
the 1.27 bump and would drift again at 1.28.
Simply bumping the tag would fix today's build and leave that second
declaration in place. This makes `go.mod` the only place the version is
declared instead.
## Changes
- **`benchmarking/locust/Dockerfile`**: set `GOTOOLCHAIN=auto` in the
`goboomer` stage, undoing the base image's `local`. Go now reads the
`go`
directive from `go.mod` and fetches a matching toolchain when the base
image
falls behind, so a future minor bump needs no change here.
## Validation
No test covers this file. Verified by build.
- [x] `docker build --no-cache --platform linux/amd64 -f
benchmarking/locust/Dockerfile .`
compiles `boomer-glutton`, logging `go: downloading go1.27.0` where it
previously failed.
- [x] Same build with `GOTOOLCHAIN` left at the image default still
fails with
the error above, confirming that variable is the cause rather than the
base image version.
#### What pr does
Makes resume measurements require the actor's memory to actually work:
adds `ReadRAM` — a glutton request that walks the working set (reads one
byte per 4KiB page across the requested size) before responding, plus a
`--mem-read` knob so the benchmark cycle performs that walk right after
every resume.
Today's cycle proves an actor is *reachable* after resume, not that its
memory is *usable*: the ping answers without touching the working set. A
real application must read its memory to serve requests. This matters
for where restore optimization is headed — a lazy/on-demand restore
would look great on a benchmark that never reads memory (resume returns
fast, ping returns fast) while real first-requests would stall faulting
pages back in. With the walk in the cycle, "resume + first response"
includes the cost of making memory usable, however the restore path
schedules that work: eager restore pays it during resume, lazy restore
would pay it during the walk — either way the total is in the tracked
numbers.
Also upgrades churn with `WRITE_MODE_OVERWRITE_ROTATE`: overwrite at a
per-key cursor that advances past each write and wraps, so repeated
churn walks the whole array over time instead of re-dirtying the same
prefix every cycle.
#### How it works
- `ReadRAM(key, size)` walks the first `size` bytes (suffixed string,
e.g. `"1Gi"`; empty walks the whole array) of a `WriteRAM` allocation,
one byte per 4KiB page — the cheapest touch that forces every page
resident. The response returns bytes walked plus an XOR checksum of the
sampled bytes so the reads are observable and can't be elided.
- Cycle order: resume → fill (once) → **walk** → churn → ping → suspend.
The walk runs *before* churn deliberately: it must read the memory as
restored, not pages churn just rewrote; churn then re-dirties after, so
the next snapshot still carries fresh pages.
- The walk reports as its own `GluttonReadRAM` stats row — it never
pollutes ping or resume latencies. Today (eager restore) it reads warm
memory in milliseconds; a jump in this row is the signal that restore
work got deferred onto the request path.
- Config travels the established channel: `--mem-read` in the suite's
locust `flags:` → `/boomer-config` → the Go worker, passed verbatim to
the wire; glutton is the only parser. Empty = disabled; the tracked
large-memory suites set it to the full target (walk everything —
strongest signal, simplest story). Existing suites unchanged.
#### Testing
- `go test -race` across `cmd/benchmarking/glutton` and
`internal/benchmarking/boomer/...`: PASS. New tests cover the walk's
byte count and checksum, missing-key/bad-size errors, cycle call order
(fill → read → churn with the right sizes and modes), rotate-mode cursor
wrap, disabled-by-default, and walk-before-fill as a no-op.
- Cluster verification (microvm, 1Gi target, full walk):
`main` does not compile at `ea3bdc32`:
```
cmd/ate-setup/internal/steps/publish.go:41:19: e.Ko undefined (type *Env has no field or method Ko)
```
Postsubmit `run-tests` and `govulncheck` are both red, and every package
under
`cmd/ate-setup/...` fails to build.
### How it happened
6101fe4 added `steps/publish.go`, which calls `Env.Ko()` to get a runner
to
build with. ea3bdc3 replaced `Env.Ko()` with `Env.imageResolver()`,
returning
an interface that the pre-built resolver can satisfy too. Both branched
from
`8968cb09` and merged twenty minutes apart. The files do not overlap, so
there
was no textual conflict, and neither branch ever contained the other's
change —
both were legitimately green.
### The fix
`PublishWorkerImages` needs to *build*, which `imageResolver`
deliberately
cannot do: `--image-repo` installs images someone else published, and
its tag
need not correspond to anything in this checkout. So building gets its
own
accessor, `koRunner`, which asserts the resolver back to `*ko.Runner`
and
returns a clear error under `--image-repo` rather than a nil
dereference.
### Worth a follow-up
`strict_required_status_checks_policy` is `false` on this repo, so a
branch is
not required to be up to date with `main` before merging. Turning it on
would
have caught this pre-merge, and will catch the next one.
Verified on Linux: `go build ./...` clean, `go test ./cmd/ate-setup/...`
green.
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>
Fixes#1387
Also tested manually on GKE cluster and kind cluster both existing build
on demand pathways and relying on prebuilt images from my own registry.
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
---------
Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>
A useful command to publish the worker images at current build, current
used in OSS upgrade flow.
Part of #1272
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
We should have a subcommand for ate-controller too, this is also useful
for upgrade.
Part of #1272
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #246.
Both ateom herders default the actor ingress flags to `0.0.0.0:443` and
`0.0.0.0:8443`, and the
worker Deployment passes those same two values explicitly — so the
command line on a running worker
reads IPv4-only. It never was: Go treats an unspecified IPv4 address as
a wildcard on a listen and
binds it dual-stack, so those sockets already served both families. This
spells all four `:443` and
`:8443` so the flag says what it does.
No behavior change. The egress listen address stays `0.0.0.0:15001`
until the actor datapath
carries v6. #1042 carried the same lines under its egress pin and is
closed as superseded.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Follow-up to the review comments on #1288 (merged before they could land
there).
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Changes:
-
`hack/third_party/csi-driver-host-path/deploy/csi-hostpath-testing.yaml`:
same fix as #1288 — the inline `watched_directory` was a no-op, so the
serving cert and trust bundle move to filesystem SDS (SPIFFE SAN pin
kept inline via `combined_validation_context`). SDS requires a node id
and this sidecar passes no `--service-node`, so the bootstrap gains a
`node:` stanza; the `docs/csi-deployment.md` example had the same gap
and gets one too.
- `overlay_test.go` now asserts `watched_directory` appears nowhere in
the emitted cluster block or the patched `envoy.yaml` (comments
excluded) and is present in each SDS resource file, per review.
- Corrected the test comment claiming the sdsmint e2e suite is skipped
in CI: the sdsmint variant is deployed in the MITM lanes; only the
`--experimental-additional-egress-extproc-service` injection has no e2e
coverage.
Verification: `go test ./cmd/ate-setup/...` passes; `envoy --mode
validate` (v1.34) passes on the reworked csi-hostpath bootstrap with
dummy PEMs mounted at the referenced paths.
Fixes#1368
The gVisor worker leaves `seccompProfile` unset and relies on the
runtime default. `runsc`'s gofer/boot child processes `pivot_root()`
while setting up the sandbox rootfs, and the default seccomp profile
denies `pivot_root` whatever capabilities the worker holds — the same
reason the microVM worker already declares `Unconfined`. On a cluster
that defaults pod seccomp to `RuntimeDefault` (e.g. via an admission
policy), the sandbox child dies during startup and `runsc create` fails
with `cannot read client sync file: waiting for sandbox to start: EOF`.
This declares `Unconfined` explicitly for the gVisor worker too,
mirroring the microVM branch. It relaxes the filter only on gVisor's own
control processes; the actor workload inside the sandbox remains
confined by the sentry's own seccomp.
> This change was prepared with AI assistance; I have reviewed and
tested it.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR is very large since it updates all existing demos and benchmark
workloads to use the new ActorTemplate substrate proto.
Please use the "Commits" tab to review individual commits.
Verifications done:
* Used this script: gpaste/5143788763348992 to verify that the change
from CRD -> proto are equivalent.
* The e2e tests are using the new susbtrate resources.
* Picked the parking demo to run e2e manually: gpaste/6193361380311040
Follow up on issue #1168 and the base PR #1215
In this PR:
**Actor read/lifecycle verbs — completes Actor end-to-end**
* `Get`/`Delete`/`Suspend`/`Pause`/`Resume` `ActorRequest`: dropped the
`+k8s:opaqueType` fence on the actor ref, added `+k8s:required` +
`+k8s:subfield(atespace)=+k8s:required`. Combined with the recursion
into `Validate_ObjectRef` this reproduces exactly what
`resources.ValidateObjectRef` enforced (ref required, atespace required,
name required, both DNS-1123 labels).
* `ListActorsRequest`: tagged to match `ListAtespacesRequest` (atespace
optional + short-name format, `page_size` minimum 1, `page_token` max
length 256).
* All six `validate*Request()` functions are now one-line calls to the
generated validators. With this, every RPC verb on `Actor` and
`Atespace` is fully declarative; `resources.ValidateObjectRef` has no
callers left in `actor.go`.
**Note:**`ListActorsRequest.page_token` now capped at 256 chars (parity
with `ListAtespaces`).
**Testing**
* Existing verb tests updated to the generated error shapes
(`field.ErrorMatcher` with origin matching).
* New coverage: `page_token` boundary cases for `ListActors`;
* `controlapi`, `functionaltest`, `actoridentity`, `store`, and `atepg`
suites all pass; `go vet` and `gofmt` clean.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
---------
Co-authored-by: Michelle Au <msau42@users.noreply.github.com>
Restore timing breakdown is the only unsampled per-actor latency record
we emit, and since traces are head-sampled at 1% on the data plane, and
actor identity is not available from metric labels on purpose.
This adds this to logs:
Before:
```json
{"msg":"Restore timing breakdown",
"actor":{"type":"*ateapipb.Actor","atespace":"demo","name":"counter-1"},
"download":310000000,"oci_unpack":50000000,"ateom_restore":60000000,"total":420000000}
```
After:
```json
{"msg":"Restore timing breakdown",
"ate.atespace":"ate-demo-counter-substrate","ate.actor.name":"join-probe",
"ate.actor.uid":"3a5b4e86-1a77-4da7-84c4-c0993d9c83ec",
"ate.template.atespace":"ate-demo-counter-substrate","ate.template.name":"counter",
"ate.snapshot.scope":"full","ate.snapshot.kind":"golden","ate.sandbox.class":"gvisor",
"ate.actor.restore.duration.volume_mount":0.000004583,
"ate.actor.restore.duration.manifest_fetch":0.006111636,
"ate.actor.restore.duration.sandbox_assets":0.000098958,
"ate.actor.restore.duration.download":0.025386544,
"ate.actor.restore.duration.oci_unpack":0.000954919,
"ate.actor.restore.duration.ateom_restore":0.147448298,
"ate.actor.restore.duration.total":0.181189689,
"trace_id":"c9d7ea9a5f90a5eb4399bfa0b966d5ac","span_id":"89280ba761137c8e"}
```
The duration keys are the `ate.actor.restore.duration` instrument's name
plus an `ate.snapshot.phase` value, so a log key and the histogram it
mirrors are literally the same identifier in source, which is also why
they're seconds.
Also adds `ateattr.ActorLogAttrs`, the component-log twin of
`ActorLogLabels`, held to the same key set by a test.
Verified on kind:
- the ateom Actor restoring record for the same actor carries the
identical `ate.actor.uid` and `trace_id`, this was impossible before
- `ate_actor_restore_duration_seconds_sum{phase="total"} = 0.181189689`,
matching the log to the nanosecond
- induced a download failure by deleting a snapshot object: record sets
`ate.failure.reason:` `FAILED_GET_EXTERNAL_OBJECT` and correctly omitted
`ateom_restore`; the metric for the same restore shows the same six
phases
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
A rolling upgrade needs two substrate versions running in one cluster,
split by node: `atelet` and `ateom` speak a node-local protocol with no
cross-version guarantees, so everything on a node has to come from one
build. Today nothing records which build a node runs, and the `atelet`
DaemonSet is a single fixed-name object, so a second version can only
replace the first in place, instead of our porposed node by node rolling
([design](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A)).
This PR is the basis for upgrade. Nodes and the `atelet` DaemonSet get
keyed by an `ate.dev/substrate-version` label whose value is the build
version stamped into the binaries.
### One derivation for the version label
`internal/versionlabel` turns the build version into its two forms in
one place:
| form | grammar | used for |
|---|---|---|
| label value | k8s label value | node labels, DaemonSet labels,
nodeSelectors |
| name suffix | DNS-1123 | the DaemonSet name `atelet-<suffix>` |
A version that is not a valid label value is rejected, because the label
has to match what the `ldflags` stamp put into the binaries through
MAKEFILE and `ko`. It's also exposed shell, so the install script can
use it.
### A DaemonSet per version
`atelet-<suffix>` as the name, the version label on metadata, selector,
and pod template, and a pod nodeSelector on the same label. Two versions
run side by side on disjoint old/new node sets.
### The install becomes version-aware
- `install-ate.sh` reads the version from the same `make ldflags` output
it stamps the binaries with, then fills the manifest placeholders.
- It labels the nodes that exist at install time, nodes come after it
carry no version label yet (so add notes in README). Re-running the
install is idempotent, this is the basis for upgrade runbook later.
- The `README` documents the invariant, how to read the installed
version off the DaemonSet, and that a node added later hosts no workers
until it carries the label. On GKE the node pool label is the birth
default for new nodes - added in `tools/setup-gcp/README.md`.
### Follow-ups
- `ate-setup` need same modification
- minimize the affect for daily developer (pin the `VERSION` as
`{USER}-dev` instead of from git
- the manual rolling-upgrade runbook.
Fix#1270
Ref [design
doc](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A)
Continues the declarative-validation (DV) migration from #1215, covering
the `ActorTemplate` resource, its full spec tree, the create path, and
read/delete verbs.
**Key Changes**
* **DV Tags & Hooks**: Added validation tags and custom hooks across the
`ActorTemplate` spec (`Metadata`, `SandboxConfig`, `SnapshotsConfig`,
`Container`, `Volume`, `Resources`).
* **Create Path**: Replaced hand-written validation with generated
validators in `CreateActorTemplate`. Custom rules (e.g., `on_commit ⊆
on_pause`) are now handled via custom hooks.
* **Read/Delete Verbs**: Converted `Get`, `List`, and `Delete` requests
to use DV.
* **State Updates**: Refactored `atepg` metadata setters to update
in-place.
* Regenerated apitool exemptions (documented 15 previously-exempt
fields).
**Deliberate Behavior Changes**
The gRPC path now strictly enforces CRD rules. Specific tightenings
include:
* Empty `worker_selector` is now rejected.
* `EnvVar.name` is strictly required.
* Negative values in optional enum fields are rejected (`minimum=1`).
* `page_token` is capped at 256 characters (matching other list
requests).
**Deliberately Not Done**
* **No update DV / status tags**: `ActorTemplates` are immutable to
clients. The reconciler updates the status directly against the store,
so tagging the status subtree or adding update validation isn't
necessary right now.
* **`config_name` matching `sandbox_class`**: Enforcing this requires
calling SandboxConfigLister (ServiceImpl), so it's left as a TODO.
**Testing**
* Added ~70 positive and negative test cases in the validator tables.
* Store-backed tests (`TestCreateActorTemplate*`,
`TestUpdateActorTemplateMetadata`, and the `atepg` suite) verified
against real Postgres via testcontainers.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This PR is large, I grouped changes to the following commits:
- 1d5d8f06: moves consumers off the CRD path: demos, ate-setup scripts
and the e2e suites address templates by the actor_template ref.
- e502cf60: Ate API changes: removes actor_template_namespace,
actor_template_name from Actor, ActorAssignment and ActorSnapshot in the
public API, drops the CRD conversion fallback.
- 8f6c1adb: atecontroller: removes the ActorTemplate CRD controller.
Once this PR is submitted, existing demos that still uses CRD will stop
working. Created https://github.com/agent-substrate/substrate/pull/1355
to update existing demos.
We want to be able to answer questions like:
- image cache hit ratio per node
- cold starts got slower on one node
but nothing we emit says which node anything is on, this PR fixes this.
Verified on a local kind cluster, where both atelet and ateom-gvisor now
show `k8s_node_name` on `target_info`, including ateom's, which arrives
through the OTLP relay.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Follow up on issue #1168 and the base PR #1215
**Overview:**
Continues the declarative validation (DV) migration from #1215 by
converting the `Worker` resource. This moves immutable-field enforcement
from the storage layer up to the service layer. With `WorkerPoolSyncer`
now using RPCs, nearly all write paths share the exact same validation.
**Key Changes:**
* **Schema (`ateapi.proto`):** Added full DV tags (required, format,
immutable) to `Worker` pod-coordinates and metadata.
`WorkerStatus.state` is now strictly bounded to its enum range.
* **Service Layer:** Replaced ~60 lines of hand-written validation with
1-line generated calls. `CreateWorker` and `UpdateWorker` now scrub
server-owned fields and enforce immutability *before* hitting the store.
* **Storage Layer:** Removed `store.CheckWorkerMutation`.
`atepg.UpdateWorker` is optimized to only clone metadata instead of the
whole worker. Immutability contract tests were moved to the service
layer.
* **Behavior Tweaks (Tightenings):**
* `page_token` is now capped at 256 chars.
* `DeleteOptions.uid` must be a valid UUID.
* Global-ref atespace violations now correctly return `Forbidden`
instead of `Invalid`.
* Immutability checks now natively cover the `WorkerPoolSyncer` path.
* **Testing:** Added comprehensive positive/negative cases for all newly
tagged fields, boundary cases, and moved the immutability test suite.
All suites pass cleanly.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#1361
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Archiving and restoring a durable dir runs inside the VM's pause window,
so its cost is actor downtime. Two mechanical inefficiencies dominated
it, both independent of how much data the dir actually holds.
A tar is a sequence of 512-byte blocks, and the stream was written
straight to the file, so every header, short file, and padding tail
became its own write(2): 120904 of them for a 30k-file tree, against 31
once a 1 MiB buffer sits in between. The reader is buffered for the same
reason.
The larger cost was allocation. io.Copy honours a WriterTo on src or a
ReaderFrom on dst before it looks at any buffer, and *os.File implements
both. Neither fast path can complete here -- the other end is a tar
stream, not a file, so sendfile/copy_file_range do not apply -- and the
generic fallback each one drops into allocates a fresh 32 KiB buffer.
That is one allocation per entry on both the archive and the restore
side: 965 MiB of garbage and 287 collections for a 30k-file tree, all of
it swept while the actor is frozen. copyPooled masks both interfaces so
io.CopyBuffer actually uses the pooled buffer it is handed.
Measured on a 30000-file, 30874624-byte tree (linux/arm64, 9 repeats):
before after
Create 274 ms 964.6 MiB 165 ms 28.6 MiB GC 287 -> 12
Extract 583 ms 979.1 MiB 490 ms 44.7 MiB GC 279 -> 19
Output is unchanged: the archive hashes identically to the one
origin/main produces (sha256 6f524c45...), and so does a re-archive of
the extracted tree. The existing round-trip tests for xattrs, device
nodes, FIFOs, ownership, and special mode bits all pass.
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Follow up on issue #1168 and the base PR #1215
#### Identity RPCs (`MintJWT` / `MintCert`)
* **Requests:** Added full declarative validation tags (required,
formatting, enum bounds) to all fields in `MintJWTRequest` and
`MintCertRequest`.
* **Handlers:** Validation now occurs immediately after authentication
(the `purpose != ATUNNEL` policy check remains in the handler).
#### Notes
* **Stricter Validation:** Empty `atespace`/`actor_name` can no longer
mint malformed JWT subjects. UIDs must now be valid UUIDs.
* **Better Errors:** `MintJWT` empty-audience error is now an
`InvalidArgument` (was `Unknown`) checked *before* disk I/O.
`MintCert`'s atespace-on-global-ref error is now typed `Forbidden` (was
`Invalid`).
#### Testing
* Added 20 new table-driven test cases (`TestValidateMintJWTRequest` /
`TestValidateMintCertRequest`), as these requests previously lacked
validation tests.
* Updated existing verb tests to match the new generated error shapes.
* All test suites (`controlapi`, `functionaltest`, etc.), `go vet`, and
`gofmt` pass cleanly.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Fixes#1066
- [x] Tests pass
- [x] ~~Appropriate changes to documentation are included in the PR~~ No
documentation changes needed.
## Summary
- Move actor-facing Envoy upstream mTLS credentials from inline file
`DataSource`s to SDS secrets served over ADS.
- Publish separate SDS secrets for the upstream podidentity client cert
and upstream trust bundle.
- Preserve SPIFFE URI SAN validation by using a combined validation
context: trust bundle via SDS, SAN matcher inline.
- Reuse the ADS config source helper for downstream TLS SDS references.
- Add tests covering upstream SDS references, snapshot secret
publication, and projected-volume rotation behavior.
---------
Signed-off-by: Sachin Kamboj <skamboj1@bloomberg.net>
* Renames the`ate.template.namespace` attribute to
`ate.template.atespace` and sources it from the actor's `actor_template`
ObjectRef instead of the legacy CRD namespace/name pair.
* Updated functional tests now to use substrate ActorTemplates and bind
actors through the `actor_template` ref.
The proc/sys/dev mountpoints were created with os.MkdirAll on paths
inside the container rootfs, whose lower is the actor's image and whose
upper is restored from the guest's own snapshot. A symlink planted at
one of those names sent the mkdir wherever it pointed on the worker pod,
created by ateom as root. Confine it to the rootfs with os.Root, in both
the merged-overlay and the reconstructed-from-image staging paths.
The bundle rootfs comes from an untrusted actor image, and the plain
os.MkdirAll/os.WriteFile pair followed symlinks out of it: an image with
/etc or /etc/resolv.conf pointing at a worker pod path had ateom, running
as root, truncate and overwrite that path. Confine the write to the
rootfs with os.Root and unlink any existing entry instead of truncating
through it.
Both Atelet and Ateom internal APIs consume actor template
namespace/name pair to telemetry. Updated them to use the new Substrate
ActorTemplate proto instead of the legacy CRD one.
We should not make workerImage optional as it provide explicit
information for what image workers use, removing it will need
`atecontroller` to inject workerImage and leads to unnecessary control
plane & dataplane wiring.
Fixes#861
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Guarding on `[ ! -d "$VENV_DIR" ]` treats an interrupted create (no
bin/activate) and an interpreter upgrade (dangling bin/python3) as a
good venv, so the script dies in `source venv/bin/activate` and the only
fix is knowing to delete the directory. Probe that the venv runs, and
rebuild with --clear to relink the interpreter.
Also install requirements unconditionally, which is cheap. The license
verifier skipped the install whenever the venv already existed, so it
passed without ever seeing a newly added dependency.
Fixes failure encountered by @ahmedtd. Not filing an issue because this
is pretty trivial.
AI-assisted.
Installing the control plane pulls roughly 570MB of third-party images
(postgres, prometheus, the otel collector, rustfs, envoy, jaeger) onto a
single node.
kubelet serializes image pulls by default, so they queue behind one
another and whichever workload lands at the back of the queue can miss
its readiness deadline.
e2e has failed repeatedly with postgres still in PodInitializing after
60s, its image not yet pulled, while everything else was already
running.
Sample failure:
https://github.com/agent-substrate/substrate/actions/runs/33258860897/job/99117220741
Note: This is after PR merge, and it's clearly unrelated to the PR.
AI-assisted.
ate-controller creates a WorkerPool's Deployment, so it does not exist
yet when the apply returns.
`kubectl rollout status` reads the object before it starts watching and
errors on a missing one instead of waiting, so whether these waits
worked at all came down to beating the controller by a round trip.
e2e lost that race by 178ms, failing the deploy with NotFound while the
pool's pods were already being created.
Fixes yet another flake I observed here:
https://github.com/agent-substrate/substrate/actions/runs/33275880827/job/99162382080?pr=1283
Envoy applies a route timeout to a CONNECT tunnel's whole lifetime, so
the 15s default capped every actor's outbound connection. The sdsmint
variant already disabled it; add the same line here, and a test over
both manifests so they cannot drift apart again.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Part of #246
The egress Envoy pinned `dns_lookup_family` to `V4_ONLY` in both the
dynamic
forward proxy filter and the dynamic forward proxy cluster. On an
IPv6-only
cluster every upstream connection failed: Envoy reported a DNS
resolution
failure and the actor got a 503. Seven sites across the two egress
manifests,
including the sdsmint variant, now use `ALL`, which returns both
families and
enables Happy Eyeballs.
This is a prerequisite for IPv6 egress, not the fix on its own. An
actor's
connection is redirected by nftables and atunnel recovers the original
destination with `getsockopt(SOL_IP, SO_ORIGINAL_DST)`, which returns
ENOENT
for a v6-redirected connection — so egress fails before Envoy is ever
asked to
resolve anything.
## Testing
`make verify` is clean. A new test walks the shipped manifests and
requires
`ALL` on every `dns_cache_config`, so a new egress variant cannot
reintroduce
the pin. It has already paid for itself: the seventh site arrived with
the
sdsmint MITM leg while this was in review, still pinned to `V4_ONLY`,
and the
test caught it on rebase.
Measured on an IPv6-only kind cluster with #911, #958, #979 and #753
applied.
Each value was deployed, Envoy restarted, and the live `/config_dump`
checked
before running the suites:
| `dns_lookup_family` | `TestActorEgress` / `TestActorEgressHTTPS` |
| --- | --- |
| `ALL` (shipped) | pass / pass, `code=200` to
`[2606:4700:10::ac42:93f3]:80` |
| `V4_ONLY` (before) | fail / fail, `code=503 flags=DF` |
| `AUTO` | pass / pass |
`AUTO` is not distinguishable from `ALL` on this path: the dynamic
forward
proxy is handed the IP literal that atunnel recovered, never a hostname,
so
`dns_lookup_family` only decides which literal families it accepts.
`ALL` is
chosen as the value that strands neither family.
The IPv4 lane is unaffected: `TestActorEgress` and
`TestActorEgressHTTPS` pass
with the change in place, across two runs of the standard e2e job.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
🤖 Generated with [Claude Code](https://claude.com/claude-code)
ateapi is a headless Service, so its DNS name resolves to one address
per replica and the client picks. This one asked for no policy, which
means gRPC's pick_first: it connects to whichever address answers first
and sends everything there for the life of the connection.
So a client talks to one replica however many are running, and adding
replicas moves no load. Measured with eight load generators against two
replicas, one took 97% of the traffic.
internal/ateapiauth, which atelet and the controllers dial through, has
asked for round_robin all along. This is the same policy for the clients
that do not.
Pulled out from #1266, I ran into this while doing some custom
benchmarking using ateclient, but this is already a bug for kubectl-ate,
demos, ...
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
AI-assisted.
Part of #207, following the plan discussed there with @juli4n and
@kannon92: plumbing plus the non invasive fixes first, with the existing
invasive findings excluded so new APIs do not regress (as @kannon92
suggested in
https://github.com/agent-substrate/substrate/issues/207#issuecomment-4921051580).
The linter itself was originally suggested by @BenTheElder in #188.
Plumbing:
- hack/tools/kube-api-linter module with the
golangci-lint-kube-api-linter tool, same pattern as the other tools.
- .golangci-kal.yaml config. The pre-existing findings from rules that
need Go API changes (nomaps, nonpointerstructs, nophase, optionalfields,
requiredfields, ssatags) are excluded rather than disabled, and each
exclusion rule names the fields it excuses, so the rules still apply to
new API files and to new fields on the existing types. Fixing them on
the existing types is the follow up in #207.
- hack/verify/kube-api-linter.sh runs it against ./pkg/api/... and is
picked up by hack/verify-all.sh in CI.
Fixes in pkg/api/v1alpha1 (doc and marker only, no Go API change):
- commentstart: field godocs now start with the serialized field name.
- conditions: Conditions moved to the first position in
ActorTemplateStatus and got the listType, listMapKey and patch markers.
- defaultorrequired: SandboxConfigSpec.SandboxClass had both a default
and required; now optional with omitempty, matching the same field on
WorkerPoolSpec and ActorTemplateSpec. The ValidatingAdmissionPolicy in
sandboxconfig-validation.yaml is unaffected since CRD defaulting runs
before admission, so spec.sandboxClass is always set by then.
- optionalorrequired: missing +optional markers added on
ActorTemplateStatus fields.
- defaults: configured preferredDefaultMarker to kubebuilder:default
since CRDs are generated with controller-gen.
Generated CRDs regenerated. Structural schema changes are only the
conditions listType/listMapKey and sandboxClass no longer in the
required list (it is defaulted, so behavior is the same); the rest is
description text.
Note: my earlier count of 50 issues in #207 was capped by the default
golangci-lint issue limit. With the cap removed the real total is 86;
the extra ones are requiredfields (20) and ssatags (4), both in the
excluded set above.
Test plan
- hack/verify/kube-api-linter.sh passes (0 issues).
- Exclusions verified both ways: dropping them brings the pre-existing
findings back, and a scratch field added to WorkerPoolSpec is still
reported (optionalfields), which the earlier per-file exclusion
swallowed.
- TestSandboxConfigValidation passes against the regenerated CRD.
- go build ./... and go test ./pkg/api/... ./cmd/atecontroller/... pass.
- gofmt, shellcheck and the regular golangci-lint on pkg/api are clean.
Remove `ActorTemplateStatus.sandbox_assets` along with the
`SandboxAssets` messages it referenced. Nothing ever wrote the field:
sandbox assets are resolved from the WorkerPool and SandboxConfig
objects at resume time and travel to atelet via ateletpb, so freezing
them into the template status never materialized.
This requires more thought, one option is to add it as a field of
GoldenSnapshotStatus in the future, and make GoldenSnapshotStatus a
repeated field to allow multiple goldens.
DNS lookups are case-insensitive (RFC 4343), so a client that resolves
`MyActor.MySpace.actors.resources.substrate.ate.dev` reaches the router
with that spelling preserved in `Host`/`:authority` (Go's http.Client
and `curl -H` both send the host as written). `ParseActorDNSName`
compared it byte for byte against a strictly lower-case suffix and name
pattern, so the request was rejected with a 404 by `invalidHostErr` even
though its DNS lookup had resolved to that very actor.
Fold the name to lower case before parsing. Actor and atespace names are
always lower case (`ResourceNameRegexPattern`), so a folded name either
addresses the same actor its lookup resolved to, or fails validation
exactly as before.
Haven Xia