With --image-repo the image tag becomes the substrate version, which
names the atelet DaemonSet and the node label. A tag written with its
digest, v1@sha256:..., is a valid image reference but not a valid label
value, so the install failed at the atelet step unless VERSION was also
set. The version is now the tag alone; the digest still pins the image.
Followup for #1391
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Two follow-ups to the pre-built image install.
Digest lookups authenticated with the docker config file alone. Someone
installing a published release onto GKE, whose Artifact Registry
credentials are
gcloud's rather than a credential helper wired into
`~/.docker/config.json`, got
a working build from source and a 401 from `--image-repo`. The lookup
keychain is
now `authn.NewMultiKeychain(authn.DefaultKeychain, google.Keychain)`,
the same
relative order ko uses. `DefaultKeychain` stays first, so
`~/.docker/config.json`
still wins wherever it has an entry, and `google.Keychain` returns
`Anonymous`
immediately for any non-Google registry, so this is a no-op for Docker
Hub, ghcr,
ECR, ACR and the Kind local registry. ECR and ACR would need
credential-helper
modules this repository does not depend on, so those still need a
`docker login`;
the doc comment no longer claims the lookup authenticates the way ko
does.
Separately, `pin` appended the looked-up digest unconditionally, so
`--image-tag v1@sha256:...` produced
`<repo>/<image>:v1@sha256:...@sha256:...`,
which no registry can parse. A tag that already carries a digest now
passes
through as written.
No dependency change: `pkg/v1/google` was already vendored, and `atelet`
already
uses the same package for its image pulls.
Verified against Artifact Registry with `DOCKER_CONFIG` pointed at an
empty
directory, which reproduces the affected setup — without this change the
lookup
fails with `DENIED: Unauthenticated request`, with it the tag resolves.
Also
installed on a Kind cluster from a local registry, where all five
control-plane
images came up pinned to the digests the registry published.
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>
> [!WARNING]
> Recreate PostgreSQL databases from earlier development builds.
## Summary
This change replaces startup schema setup with embedded, versioned SQL
migrations.
`ateapi` uses Goose to apply migrations before readiness. Goose stores
one ledger record for each applied migration.
Goose runs each migration and inserts its ledger record in one
PostgreSQL transaction.
Closes#901.
Based on this design:
https://docs.google.com/document/d/13ixDKRoAIFXeLxS-_1nikNcAy76ca8m0eVgobfib93E/edit?usp=sharing
## Migration behavior
`ateapi` gets a session advisory lock for the configured schema before
it applies pending migrations.
One replica applies migrations while other replicas wait. Goose reads
the ledger again after it gets the lock.
If a migration fails, PostgreSQL rolls back its SQL and ledger record.
Earlier successful migrations remain applied and recorded.
Kubernetes restarts the failed replica. The next startup resumes from
the first migration without a ledger record.
## Changes
- Add Goose and a per-migration ledger.
- Replace the initial up and down files with one transactional, up-only
migration.
- Keep migration 1 aligned with the current schema, including actor
egress policy storage.
- Remove existence guards and explicit transaction statements from the
baseline migration.
- Apply all pending migrations before `ateapi` becomes ready.
- Serialize each migration run with a PostgreSQL session advisory lock.
- Start without changes when the database schema is current or ahead.
- Reject application tables that do not have a migration ledger.
- Log the starting, current, and latest versions.
- Log the applied migration count and duration.
- Retry only initial database connection failures.
- Return schema and migration errors without a retry.
- Add `--postgres-schema` and `ATE_API_POSTGRES_SCHEMA`.
- Use `public` as the default PostgreSQL schema.
- Use the configured schema for the main and watch pools.
- Restrict outbox partition maintenance to the configured schema.
- Let the installer use an external PostgreSQL database.
- Add the migration design and recovery policy to the repository.
## Migration file policy
Migration files use sequential versions and contain exactly one Goose
`Up` section.
CI rejects down migrations, nontransactional migrations, environment
substitution, explicit transaction control, and `IF NOT EXISTS` guards.
Before the first stable v1 release, developers can change or squash
migrations. Developers must recreate databases after migration history
changes.
After that release, CI rejects changes or deletions against the latest
stable release tag that contains migrations.
Goose does not store migration checksums. The binary embeds each
migration file, and release-tag checks protect released migration
history.
## Compatibility
No release includes PostgreSQL support. The `v0.0.0` release predates
the PostgreSQL backend.
Users must recreate databases from earlier PostgreSQL development
builds.
Every committed migration prefix must work with the current and previous
`ateapi` releases. This rule supports rolling upgrades and temporary
binary rollback.
A binary rollback does not roll back the database schema.
## Testing
Tests cover:
- Fresh database migration.
- Concurrent startup.
- Advisory lock waits.
- Current and ahead database schemas.
- Rejection of application tables without a migration ledger.
- Atomic rollback of a failed migration.
- Retention of earlier successful migrations.
- Resume from the failed migration after restart.
- Configured schema isolation.
- Outbox partition isolation.
- Migration file policy checks.
- Stable release migration immutability.
Fixes#368 . Deletes ActorTemplate CRD and any references to it.
Note about atenet router: It had a k8sclient controller that monitors
ActorTemplate, but the results were not used. Deleted as well.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
#### What pr does
Makes resume measurements require the actor's memory to actually work:
adds `ReadRAM` — a glutton request that walks the working set (reads one
byte per 4KiB page across the requested size) before responding, plus a
`--mem-read` knob so the benchmark cycle performs that walk right after
every resume.
Today's cycle proves an actor is *reachable* after resume, not that its
memory is *usable*: the ping answers without touching the working set. A
real application must read its memory to serve requests. This matters
for where restore optimization is headed — a lazy/on-demand restore
would look great on a benchmark that never reads memory (resume returns
fast, ping returns fast) while real first-requests would stall faulting
pages back in. With the walk in the cycle, "resume + first response"
includes the cost of making memory usable, however the restore path
schedules that work: eager restore pays it during resume, lazy restore
would pay it during the walk — either way the total is in the tracked
numbers.
Also upgrades churn with `WRITE_MODE_OVERWRITE_ROTATE`: overwrite at a
per-key cursor that advances past each write and wraps, so repeated
churn walks the whole array over time instead of re-dirtying the same
prefix every cycle.
#### How it works
- `ReadRAM(key, size)` walks the first `size` bytes (suffixed string,
e.g. `"1Gi"`; empty walks the whole array) of a `WriteRAM` allocation,
one byte per 4KiB page — the cheapest touch that forces every page
resident. The response returns bytes walked plus an XOR checksum of the
sampled bytes so the reads are observable and can't be elided.
- Cycle order: resume → fill (once) → **walk** → churn → ping → suspend.
The walk runs *before* churn deliberately: it must read the memory as
restored, not pages churn just rewrote; churn then re-dirties after, so
the next snapshot still carries fresh pages.
- The walk reports as its own `GluttonReadRAM` stats row — it never
pollutes ping or resume latencies. Today (eager restore) it reads warm
memory in milliseconds; a jump in this row is the signal that restore
work got deferred onto the request path.
- Config travels the established channel: `--mem-read` in the suite's
locust `flags:` → `/boomer-config` → the Go worker, passed verbatim to
the wire; glutton is the only parser. Empty = disabled; the tracked
large-memory suites set it to the full target (walk everything —
strongest signal, simplest story). Existing suites unchanged.
#### Testing
- `go test -race` across `cmd/benchmarking/glutton` and
`internal/benchmarking/boomer/...`: PASS. New tests cover the walk's
byte count and checksum, missing-key/bad-size errors, cycle call order
(fill → read → churn with the right sizes and modes), rotate-mode cursor
wrap, disabled-by-default, and walk-before-fill as a no-op.
- Cluster verification (microvm, 1Gi target, full walk):
`main` does not compile at `ea3bdc32`:
```
cmd/ate-setup/internal/steps/publish.go:41:19: e.Ko undefined (type *Env has no field or method Ko)
```
Postsubmit `run-tests` and `govulncheck` are both red, and every package
under
`cmd/ate-setup/...` fails to build.
### How it happened
6101fe4 added `steps/publish.go`, which calls `Env.Ko()` to get a runner
to
build with. ea3bdc3 replaced `Env.Ko()` with `Env.imageResolver()`,
returning
an interface that the pre-built resolver can satisfy too. Both branched
from
`8968cb09` and merged twenty minutes apart. The files do not overlap, so
there
was no textual conflict, and neither branch ever contained the other's
change —
both were legitimately green.
### The fix
`PublishWorkerImages` needs to *build*, which `imageResolver`
deliberately
cannot do: `--image-repo` installs images someone else published, and
its tag
need not correspond to anything in this checkout. So building gets its
own
accessor, `koRunner`, which asserts the resolver back to `*ko.Runner`
and
returns a clear error under `--image-repo` rather than a nil
dereference.
### Worth a follow-up
`strict_required_status_checks_policy` is `false` on this repo, so a
branch is
not required to be up to date with `main` before merging. Turning it on
would
have caught this pre-merge, and will catch the next one.
Verified on Linux: `go build ./...` clean, `go test ./cmd/ate-setup/...`
green.
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>
Fixes#1387
Also tested manually on GKE cluster and kind cluster both existing build
on demand pathways and relying on prebuilt images from my own registry.
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
---------
Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>
A useful command to publish the worker images at current build, current
used in OSS upgrade flow.
Part of #1272
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
We should have a subcommand for ate-controller too, this is also useful
for upgrade.
Part of #1272
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Part of #246.
Both ateom herders default the actor ingress flags to `0.0.0.0:443` and
`0.0.0.0:8443`, and the
worker Deployment passes those same two values explicitly — so the
command line on a running worker
reads IPv4-only. It never was: Go treats an unspecified IPv4 address as
a wildcard on a listen and
binds it dual-stack, so those sockets already served both families. This
spells all four `:443` and
`:8443` so the flag says what it does.
No behavior change. The egress listen address stays `0.0.0.0:15001`
until the actor datapath
carries v6. #1042 carried the same lines under its egress pin and is
closed as superseded.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Follow-up to the review comments on #1288 (merged before they could land
there).
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Changes:
-
`hack/third_party/csi-driver-host-path/deploy/csi-hostpath-testing.yaml`:
same fix as #1288 — the inline `watched_directory` was a no-op, so the
serving cert and trust bundle move to filesystem SDS (SPIFFE SAN pin
kept inline via `combined_validation_context`). SDS requires a node id
and this sidecar passes no `--service-node`, so the bootstrap gains a
`node:` stanza; the `docs/csi-deployment.md` example had the same gap
and gets one too.
- `overlay_test.go` now asserts `watched_directory` appears nowhere in
the emitted cluster block or the patched `envoy.yaml` (comments
excluded) and is present in each SDS resource file, per review.
- Corrected the test comment claiming the sdsmint e2e suite is skipped
in CI: the sdsmint variant is deployed in the MITM lanes; only the
`--experimental-additional-egress-extproc-service` injection has no e2e
coverage.
Verification: `go test ./cmd/ate-setup/...` passes; `envoy --mode
validate` (v1.34) passes on the reworked csi-hostpath bootstrap with
dummy PEMs mounted at the referenced paths.
Fixes#1368
The gVisor worker leaves `seccompProfile` unset and relies on the
runtime default. `runsc`'s gofer/boot child processes `pivot_root()`
while setting up the sandbox rootfs, and the default seccomp profile
denies `pivot_root` whatever capabilities the worker holds — the same
reason the microVM worker already declares `Unconfined`. On a cluster
that defaults pod seccomp to `RuntimeDefault` (e.g. via an admission
policy), the sandbox child dies during startup and `runsc create` fails
with `cannot read client sync file: waiting for sandbox to start: EOF`.
This declares `Unconfined` explicitly for the gVisor worker too,
mirroring the microVM branch. It relaxes the filter only on gVisor's own
control processes; the actor workload inside the sandbox remains
confined by the sentry's own seccomp.
> This change was prepared with AI assistance; I have reviewed and
tested it.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR is very large since it updates all existing demos and benchmark
workloads to use the new ActorTemplate substrate proto.
Please use the "Commits" tab to review individual commits.
Verifications done:
* Used this script: gpaste/5143788763348992 to verify that the change
from CRD -> proto are equivalent.
* The e2e tests are using the new susbtrate resources.
* Picked the parking demo to run e2e manually: gpaste/6193361380311040
Follow up on issue #1168 and the base PR #1215
In this PR:
**Actor read/lifecycle verbs — completes Actor end-to-end**
* `Get`/`Delete`/`Suspend`/`Pause`/`Resume` `ActorRequest`: dropped the
`+k8s:opaqueType` fence on the actor ref, added `+k8s:required` +
`+k8s:subfield(atespace)=+k8s:required`. Combined with the recursion
into `Validate_ObjectRef` this reproduces exactly what
`resources.ValidateObjectRef` enforced (ref required, atespace required,
name required, both DNS-1123 labels).
* `ListActorsRequest`: tagged to match `ListAtespacesRequest` (atespace
optional + short-name format, `page_size` minimum 1, `page_token` max
length 256).
* All six `validate*Request()` functions are now one-line calls to the
generated validators. With this, every RPC verb on `Actor` and
`Atespace` is fully declarative; `resources.ValidateObjectRef` has no
callers left in `actor.go`.
**Note:**`ListActorsRequest.page_token` now capped at 256 chars (parity
with `ListAtespaces`).
**Testing**
* Existing verb tests updated to the generated error shapes
(`field.ErrorMatcher` with origin matching).
* New coverage: `page_token` boundary cases for `ListActors`;
* `controlapi`, `functionaltest`, `actoridentity`, `store`, and `atepg`
suites all pass; `go vet` and `gofmt` clean.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
---------
Co-authored-by: Michelle Au <msau42@users.noreply.github.com>
Restore timing breakdown is the only unsampled per-actor latency record
we emit, and since traces are head-sampled at 1% on the data plane, and
actor identity is not available from metric labels on purpose.
This adds this to logs:
Before:
```json
{"msg":"Restore timing breakdown",
"actor":{"type":"*ateapipb.Actor","atespace":"demo","name":"counter-1"},
"download":310000000,"oci_unpack":50000000,"ateom_restore":60000000,"total":420000000}
```
After:
```json
{"msg":"Restore timing breakdown",
"ate.atespace":"ate-demo-counter-substrate","ate.actor.name":"join-probe",
"ate.actor.uid":"3a5b4e86-1a77-4da7-84c4-c0993d9c83ec",
"ate.template.atespace":"ate-demo-counter-substrate","ate.template.name":"counter",
"ate.snapshot.scope":"full","ate.snapshot.kind":"golden","ate.sandbox.class":"gvisor",
"ate.actor.restore.duration.volume_mount":0.000004583,
"ate.actor.restore.duration.manifest_fetch":0.006111636,
"ate.actor.restore.duration.sandbox_assets":0.000098958,
"ate.actor.restore.duration.download":0.025386544,
"ate.actor.restore.duration.oci_unpack":0.000954919,
"ate.actor.restore.duration.ateom_restore":0.147448298,
"ate.actor.restore.duration.total":0.181189689,
"trace_id":"c9d7ea9a5f90a5eb4399bfa0b966d5ac","span_id":"89280ba761137c8e"}
```
The duration keys are the `ate.actor.restore.duration` instrument's name
plus an `ate.snapshot.phase` value, so a log key and the histogram it
mirrors are literally the same identifier in source, which is also why
they're seconds.
Also adds `ateattr.ActorLogAttrs`, the component-log twin of
`ActorLogLabels`, held to the same key set by a test.
Verified on kind:
- the ateom Actor restoring record for the same actor carries the
identical `ate.actor.uid` and `trace_id`, this was impossible before
- `ate_actor_restore_duration_seconds_sum{phase="total"} = 0.181189689`,
matching the log to the nanosecond
- induced a download failure by deleting a snapshot object: record sets
`ate.failure.reason:` `FAILED_GET_EXTERNAL_OBJECT` and correctly omitted
`ateom_restore`; the metric for the same restore shows the same six
phases
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Continues the declarative-validation (DV) migration from #1215, covering
the `ActorTemplate` resource, its full spec tree, the create path, and
read/delete verbs.
**Key Changes**
* **DV Tags & Hooks**: Added validation tags and custom hooks across the
`ActorTemplate` spec (`Metadata`, `SandboxConfig`, `SnapshotsConfig`,
`Container`, `Volume`, `Resources`).
* **Create Path**: Replaced hand-written validation with generated
validators in `CreateActorTemplate`. Custom rules (e.g., `on_commit ⊆
on_pause`) are now handled via custom hooks.
* **Read/Delete Verbs**: Converted `Get`, `List`, and `Delete` requests
to use DV.
* **State Updates**: Refactored `atepg` metadata setters to update
in-place.
* Regenerated apitool exemptions (documented 15 previously-exempt
fields).
**Deliberate Behavior Changes**
The gRPC path now strictly enforces CRD rules. Specific tightenings
include:
* Empty `worker_selector` is now rejected.
* `EnvVar.name` is strictly required.
* Negative values in optional enum fields are rejected (`minimum=1`).
* `page_token` is capped at 256 characters (matching other list
requests).
**Deliberately Not Done**
* **No update DV / status tags**: `ActorTemplates` are immutable to
clients. The reconciler updates the status directly against the store,
so tagging the status subtree or adding update validation isn't
necessary right now.
* **`config_name` matching `sandbox_class`**: Enforcing this requires
calling SandboxConfigLister (ServiceImpl), so it's left as a TODO.
**Testing**
* Added ~70 positive and negative test cases in the validator tables.
* Store-backed tests (`TestCreateActorTemplate*`,
`TestUpdateActorTemplateMetadata`, and the `atepg` suite) verified
against real Postgres via testcontainers.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This PR is large, I grouped changes to the following commits:
- 1d5d8f06: moves consumers off the CRD path: demos, ate-setup scripts
and the e2e suites address templates by the actor_template ref.
- e502cf60: Ate API changes: removes actor_template_namespace,
actor_template_name from Actor, ActorAssignment and ActorSnapshot in the
public API, drops the CRD conversion fallback.
- 8f6c1adb: atecontroller: removes the ActorTemplate CRD controller.
Once this PR is submitted, existing demos that still uses CRD will stop
working. Created https://github.com/agent-substrate/substrate/pull/1355
to update existing demos.
We want to be able to answer questions like:
- image cache hit ratio per node
- cold starts got slower on one node
but nothing we emit says which node anything is on, this PR fixes this.
Verified on a local kind cluster, where both atelet and ateom-gvisor now
show `k8s_node_name` on `target_info`, including ateom's, which arrives
through the OTLP relay.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Follow up on issue #1168 and the base PR #1215
**Overview:**
Continues the declarative validation (DV) migration from #1215 by
converting the `Worker` resource. This moves immutable-field enforcement
from the storage layer up to the service layer. With `WorkerPoolSyncer`
now using RPCs, nearly all write paths share the exact same validation.
**Key Changes:**
* **Schema (`ateapi.proto`):** Added full DV tags (required, format,
immutable) to `Worker` pod-coordinates and metadata.
`WorkerStatus.state` is now strictly bounded to its enum range.
* **Service Layer:** Replaced ~60 lines of hand-written validation with
1-line generated calls. `CreateWorker` and `UpdateWorker` now scrub
server-owned fields and enforce immutability *before* hitting the store.
* **Storage Layer:** Removed `store.CheckWorkerMutation`.
`atepg.UpdateWorker` is optimized to only clone metadata instead of the
whole worker. Immutability contract tests were moved to the service
layer.
* **Behavior Tweaks (Tightenings):**
* `page_token` is now capped at 256 chars.
* `DeleteOptions.uid` must be a valid UUID.
* Global-ref atespace violations now correctly return `Forbidden`
instead of `Invalid`.
* Immutability checks now natively cover the `WorkerPoolSyncer` path.
* **Testing:** Added comprehensive positive/negative cases for all newly
tagged fields, boundary cases, and moved the immutability test suite.
All suites pass cleanly.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Archiving and restoring a durable dir runs inside the VM's pause window,
so its cost is actor downtime. Two mechanical inefficiencies dominated
it, both independent of how much data the dir actually holds.
A tar is a sequence of 512-byte blocks, and the stream was written
straight to the file, so every header, short file, and padding tail
became its own write(2): 120904 of them for a 30k-file tree, against 31
once a 1 MiB buffer sits in between. The reader is buffered for the same
reason.
The larger cost was allocation. io.Copy honours a WriterTo on src or a
ReaderFrom on dst before it looks at any buffer, and *os.File implements
both. Neither fast path can complete here -- the other end is a tar
stream, not a file, so sendfile/copy_file_range do not apply -- and the
generic fallback each one drops into allocates a fresh 32 KiB buffer.
That is one allocation per entry on both the archive and the restore
side: 965 MiB of garbage and 287 collections for a 30k-file tree, all of
it swept while the actor is frozen. copyPooled masks both interfaces so
io.CopyBuffer actually uses the pooled buffer it is handed.
Measured on a 30000-file, 30874624-byte tree (linux/arm64, 9 repeats):
before after
Create 274 ms 964.6 MiB 165 ms 28.6 MiB GC 287 -> 12
Extract 583 ms 979.1 MiB 490 ms 44.7 MiB GC 279 -> 19
Output is unchanged: the archive hashes identically to the one
origin/main produces (sha256 6f524c45...), and so does a re-archive of
the extracted tree. The existing round-trip tests for xattrs, device
nodes, FIFOs, ownership, and special mode bits all pass.
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Follow up on issue #1168 and the base PR #1215
#### Identity RPCs (`MintJWT` / `MintCert`)
* **Requests:** Added full declarative validation tags (required,
formatting, enum bounds) to all fields in `MintJWTRequest` and
`MintCertRequest`.
* **Handlers:** Validation now occurs immediately after authentication
(the `purpose != ATUNNEL` policy check remains in the handler).
#### Notes
* **Stricter Validation:** Empty `atespace`/`actor_name` can no longer
mint malformed JWT subjects. UIDs must now be valid UUIDs.
* **Better Errors:** `MintJWT` empty-audience error is now an
`InvalidArgument` (was `Unknown`) checked *before* disk I/O.
`MintCert`'s atespace-on-global-ref error is now typed `Forbidden` (was
`Invalid`).
#### Testing
* Added 20 new table-driven test cases (`TestValidateMintJWTRequest` /
`TestValidateMintCertRequest`), as these requests previously lacked
validation tests.
* Updated existing verb tests to match the new generated error shapes.
* All test suites (`controlapi`, `functionaltest`, etc.), `go vet`, and
`gofmt` pass cleanly.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Fixes#1066
- [x] Tests pass
- [x] ~~Appropriate changes to documentation are included in the PR~~ No
documentation changes needed.
## Summary
- Move actor-facing Envoy upstream mTLS credentials from inline file
`DataSource`s to SDS secrets served over ADS.
- Publish separate SDS secrets for the upstream podidentity client cert
and upstream trust bundle.
- Preserve SPIFFE URI SAN validation by using a combined validation
context: trust bundle via SDS, SAN matcher inline.
- Reuse the ADS config source helper for downstream TLS SDS references.
- Add tests covering upstream SDS references, snapshot secret
publication, and projected-volume rotation behavior.
---------
Signed-off-by: Sachin Kamboj <skamboj1@bloomberg.net>
* Renames the`ate.template.namespace` attribute to
`ate.template.atespace` and sources it from the actor's `actor_template`
ObjectRef instead of the legacy CRD namespace/name pair.
* Updated functional tests now to use substrate ActorTemplates and bind
actors through the `actor_template` ref.
The proc/sys/dev mountpoints were created with os.MkdirAll on paths
inside the container rootfs, whose lower is the actor's image and whose
upper is restored from the guest's own snapshot. A symlink planted at
one of those names sent the mkdir wherever it pointed on the worker pod,
created by ateom as root. Confine it to the rootfs with os.Root, in both
the merged-overlay and the reconstructed-from-image staging paths.
The bundle rootfs comes from an untrusted actor image, and the plain
os.MkdirAll/os.WriteFile pair followed symlinks out of it: an image with
/etc or /etc/resolv.conf pointing at a worker pod path had ateom, running
as root, truncate and overwrite that path. Confine the write to the
rootfs with os.Root and unlink any existing entry instead of truncating
through it.
Both Atelet and Ateom internal APIs consume actor template
namespace/name pair to telemetry. Updated them to use the new Substrate
ActorTemplate proto instead of the legacy CRD one.
Envoy applies a route timeout to a CONNECT tunnel's whole lifetime, so
the 15s default capped every actor's outbound connection. The sdsmint
variant already disabled it; add the same line here, and a test over
both manifests so they cannot drift apart again.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Part of #246
The egress Envoy pinned `dns_lookup_family` to `V4_ONLY` in both the
dynamic
forward proxy filter and the dynamic forward proxy cluster. On an
IPv6-only
cluster every upstream connection failed: Envoy reported a DNS
resolution
failure and the actor got a 503. Seven sites across the two egress
manifests,
including the sdsmint variant, now use `ALL`, which returns both
families and
enables Happy Eyeballs.
This is a prerequisite for IPv6 egress, not the fix on its own. An
actor's
connection is redirected by nftables and atunnel recovers the original
destination with `getsockopt(SOL_IP, SO_ORIGINAL_DST)`, which returns
ENOENT
for a v6-redirected connection — so egress fails before Envoy is ever
asked to
resolve anything.
## Testing
`make verify` is clean. A new test walks the shipped manifests and
requires
`ALL` on every `dns_cache_config`, so a new egress variant cannot
reintroduce
the pin. It has already paid for itself: the seventh site arrived with
the
sdsmint MITM leg while this was in review, still pinned to `V4_ONLY`,
and the
test caught it on rebase.
Measured on an IPv6-only kind cluster with #911, #958, #979 and #753
applied.
Each value was deployed, Envoy restarted, and the live `/config_dump`
checked
before running the suites:
| `dns_lookup_family` | `TestActorEgress` / `TestActorEgressHTTPS` |
| --- | --- |
| `ALL` (shipped) | pass / pass, `code=200` to
`[2606:4700:10::ac42:93f3]:80` |
| `V4_ONLY` (before) | fail / fail, `code=503 flags=DF` |
| `AUTO` | pass / pass |
`AUTO` is not distinguishable from `ALL` on this path: the dynamic
forward
proxy is handed the IP literal that atunnel recovered, never a hostname,
so
`dns_lookup_family` only decides which literal families it accepts.
`ALL` is
chosen as the value that strands neither family.
The IPv4 lane is unaffected: `TestActorEgress` and
`TestActorEgressHTTPS` pass
with the change in place, across two runs of the standard e2e job.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Remove `ActorTemplateStatus.sandbox_assets` along with the
`SandboxAssets` messages it referenced. Nothing ever wrote the field:
sandbox assets are resolved from the WorkerPool and SandboxConfig
objects at resume time and travel to atelet via ateletpb, so freezing
them into the template status never materialized.
This requires more thought, one option is to add it as a field of
GoldenSnapshotStatus in the future, and make GoldenSnapshotStatus a
repeated field to allow multiple goldens.
DNS lookups are case-insensitive (RFC 4343), so a client that resolves
`MyActor.MySpace.actors.resources.substrate.ate.dev` reaches the router
with that spelling preserved in `Host`/`:authority` (Go's http.Client
and `curl -H` both send the host as written). `ParseActorDNSName`
compared it byte for byte against a strictly lower-case suffix and name
pattern, so the request was rejected with a 404 by `invalidHostErr` even
though its DNS lookup had resolved to that very actor.
Fold the name to lower case before parsing. Actor and atespace names are
always lower case (`ResourceNameRegexPattern`), so a folded name either
addresses the same actor its lookup resolved to, or fails validation
exactly as before.
Necessary for #1266, but functionally useful regardless.
This eliminates the MPL go-reap dependency in favor of our own
implementation tuned to our needs.
Our package has utilities for spawning child processes without holding a
RWMutex to avoid convoying lots of concurrent activations in the future.
Switches all e2e tests that uses counter demo to validate the new
substrate proto ActorTemplate.
Made some changes to make the e2e test pass:
* cmd/ateapi/internal/controlapi/template_reconciler.go - golden actors
still need the golden atespace, because the logic in suspend actor
relies on it to always take FULL snapshot regardless of the config.
(filed https://github.com/agent-substrate/substrate/issues/1299)
* internal/ateattr/ateattr.go - Added placeholder label for now, will
fix in a follow up PR to wire the metric reporting when using new
ActorTemplate substrate object.
I noticed my machine filling up with postgres containers ...
---
First:
storetest starts one PostgreSQL container per test binary in a sync.Once
and never stops it, so every package that uses it leaks a container per
run. Testcontainers' Ryuk reaper normally covers this, but it is not
guaranteed to be running, and nothing else stops the container.
Add Shutdown, plus a RunTests wrapper for packages whose TestMain does
nothing else, and call it from the three packages that use storetest.
atepg already terminates its own container this way.
----
Second:
testcontainers does not respect docker contexts, making it difficult to
use with [limactl](https://lima-vm.io/) etc.
See: https://github.com/testcontainers/testcontainers-go/issues/2607
We automate the workaround to set the host env to point at the socket
from the active docker context.
Envoy honors watched_directory only on SDS-delivered secrets; on the
egress gateway's inline file-based certs it was silently ignored, so
kubelet's projected-certificate rotation never reached Envoy and the
gateway eventually served an expired certificate.
Move the serving cert (both egress manifests) and the spliced extproc
cluster's client cert and trust bundle (Go and shell emitters) to
filesystem SDS, with the flag-derived SAN pin kept inline via a
combined validation context. Fix the csi-deployment doc example that
showed the same broken pattern, and add tests for the emitters.
Rename `WorkerPoolSpec.AteomImage`to `WorkerImage` and drop the
constraints so the field can be left unset.
This is the basis for let an empty workerImage lets the controller
inject a versioned default image chosen by the pool's sandbox class.
Part of #861
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
#### What this pr do
Re-dirties part of the glutton working set on every benchmark cycle, so
repeated suspends snapshot an actor whose memory is changing — like a
live application's — instead of a set that was filled once and never
touched again.
Each iteration (after the one-time fill), the GluttonUser sends one
`WriteRAM` request with `WRITE_MODE_OVERWRITE`, re-randomizing the first
`mem_churn` bytes of the working set in place. Overwrite mode already
existed on the glutton API; this PR adds no proto changes and no glutton
server changes — it is driver + config wiring only.
This is the follow-up from #1130's: the original self-driving memload
continuously re-dirtied its pages, and that property was dropped in the
move to API-driven fill. This restores it in the API-driven shape with a
dialable amount instead of the old all-or-nothing full pass.
#### Why it matters
A fill-once working set is static: every suspend after the first packs
up identical memory. If snapshotting ever gains incremental / dirty-page
optimizations, a static benchmark would measure almost nothing from
cycle two onward — and would score "upload nothing" as an infinite win.
With churn, every cycle carries a known amount of freshly-dirtied pages,
and the knob is sweepable (64Mi, 256Mi, …) so a future
differential-snapshot optimization can be demonstrated as "upload cost
scales with churn size, not total size."
Haven Xia