mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-12 10:35:06 +08:00
2eeee044e64761f02df7d2ea8a0924685c5d203d
144
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jeremy Alvis
|
f100c27286 |
ateapi: add versioned PostgreSQL schema migrations (#1196)
> [!WARNING] > Recreate PostgreSQL databases from earlier development builds. ## Summary This change replaces startup schema setup with embedded, versioned SQL migrations. `ateapi` uses Goose to apply migrations before readiness. Goose stores one ledger record for each applied migration. Goose runs each migration and inserts its ledger record in one PostgreSQL transaction. Closes #901. Based on this design: https://docs.google.com/document/d/13ixDKRoAIFXeLxS-_1nikNcAy76ca8m0eVgobfib93E/edit?usp=sharing ## Migration behavior `ateapi` gets a session advisory lock for the configured schema before it applies pending migrations. One replica applies migrations while other replicas wait. Goose reads the ledger again after it gets the lock. If a migration fails, PostgreSQL rolls back its SQL and ledger record. Earlier successful migrations remain applied and recorded. Kubernetes restarts the failed replica. The next startup resumes from the first migration without a ledger record. ## Changes - Add Goose and a per-migration ledger. - Replace the initial up and down files with one transactional, up-only migration. - Keep migration 1 aligned with the current schema, including actor egress policy storage. - Remove existence guards and explicit transaction statements from the baseline migration. - Apply all pending migrations before `ateapi` becomes ready. - Serialize each migration run with a PostgreSQL session advisory lock. - Start without changes when the database schema is current or ahead. - Reject application tables that do not have a migration ledger. - Log the starting, current, and latest versions. - Log the applied migration count and duration. - Retry only initial database connection failures. - Return schema and migration errors without a retry. - Add `--postgres-schema` and `ATE_API_POSTGRES_SCHEMA`. - Use `public` as the default PostgreSQL schema. - Use the configured schema for the main and watch pools. - Restrict outbox partition maintenance to the configured schema. - Let the installer use an external PostgreSQL database. - Add the migration design and recovery policy to the repository. ## Migration file policy Migration files use sequential versions and contain exactly one Goose `Up` section. CI rejects down migrations, nontransactional migrations, environment substitution, explicit transaction control, and `IF NOT EXISTS` guards. Before the first stable v1 release, developers can change or squash migrations. Developers must recreate databases after migration history changes. After that release, CI rejects changes or deletions against the latest stable release tag that contains migrations. Goose does not store migration checksums. The binary embeds each migration file, and release-tag checks protect released migration history. ## Compatibility No release includes PostgreSQL support. The `v0.0.0` release predates the PostgreSQL backend. Users must recreate databases from earlier PostgreSQL development builds. Every committed migration prefix must work with the current and previous `ateapi` releases. This rule supports rolling upgrades and temporary binary rollback. A binary rollback does not roll back the database schema. ## Testing Tests cover: - Fresh database migration. - Concurrent startup. - Advisory lock waits. - Current and ahead database schemas. - Rejection of application tables without a migration ledger. - Atomic rollback of a failed migration. - Retention of earlier successful migrations. - Resume from the failed migration after restart. - Configured schema isolation. - Outbox partition isolation. - Migration file policy checks. - Stable release migration immutability. |
||
|
Zoe Zhao
|
120a519606 |
Delete ActorTemplate CRD (#1376)
Fixes #368 . Deletes ActorTemplate CRD and any references to it. Note about atenet router: It had a k8sclient controller that monitors ActorTemplate, but the results were not used. Deleted as well. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Chuang Wang
|
07ebc56797 |
csi-hostpath: fix cert rotation in the testing Envoy config (#1336)
Follow-up to the review comments on #1288 (merged before they could land there). - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR Changes: - `hack/third_party/csi-driver-host-path/deploy/csi-hostpath-testing.yaml`: same fix as #1288 — the inline `watched_directory` was a no-op, so the serving cert and trust bundle move to filesystem SDS (SPIFFE SAN pin kept inline via `combined_validation_context`). SDS requires a node id and this sidecar passes no `--service-node`, so the bootstrap gains a `node:` stanza; the `docs/csi-deployment.md` example had the same gap and gets one too. - `overlay_test.go` now asserts `watched_directory` appears nowhere in the emitted cluster block or the patched `envoy.yaml` (comments excluded) and is present in each SDS resource file, per review. - Corrected the test comment claiming the sdsmint e2e suite is skipped in CI: the sdsmint variant is deployed in the MITM lanes; only the `--experimental-additional-egress-extproc-service` injection has no e2e coverage. Verification: `go test ./cmd/ate-setup/...` passes; `envoy --mode validate` (v1.34) passes on the reworked csi-hostpath bootstrap with dummy PEMs mounted at the referenced paths. |
||
|
Elijah Rodriguez
|
53800cd045 | Update CSI NFS server image to use kubernetes build | ||
|
Zoe Zhao
|
f6852b7754 |
Update existing demos and benchmark tests to use substrate ActorTemplate resource (#1355)
This PR is very large since it updates all existing demos and benchmark workloads to use the new ActorTemplate substrate proto. Please use the "Commits" tab to review individual commits. Verifications done: * Used this script: gpaste/5143788763348992 to verify that the change from CRD -> proto are equivalent. * The e2e tests are using the new susbtrate resources. * Picked the parking demo to run e2e manually: gpaste/6193361380311040 |
||
|
Joe Betz
|
3efac91283 |
api: delete the DebugClear RPC and the Debug service (#1346)
Fixes #999. |
||
|
Krisztian F
|
85d404a823 |
(atelet): make the restore timing log a joinable per-actor latency record (#1364)
Restore timing breakdown is the only unsampled per-actor latency record
we emit, and since traces are head-sampled at 1% on the data plane, and
actor identity is not available from metric labels on purpose.
This adds this to logs:
Before:
```json
{"msg":"Restore timing breakdown",
"actor":{"type":"*ateapipb.Actor","atespace":"demo","name":"counter-1"},
"download":310000000,"oci_unpack":50000000,"ateom_restore":60000000,"total":420000000}
```
After:
```json
{"msg":"Restore timing breakdown",
"ate.atespace":"ate-demo-counter-substrate","ate.actor.name":"join-probe",
"ate.actor.uid":"3a5b4e86-1a77-4da7-84c4-c0993d9c83ec",
"ate.template.atespace":"ate-demo-counter-substrate","ate.template.name":"counter",
"ate.snapshot.scope":"full","ate.snapshot.kind":"golden","ate.sandbox.class":"gvisor",
"ate.actor.restore.duration.volume_mount":0.000004583,
"ate.actor.restore.duration.manifest_fetch":0.006111636,
"ate.actor.restore.duration.sandbox_assets":0.000098958,
"ate.actor.restore.duration.download":0.025386544,
"ate.actor.restore.duration.oci_unpack":0.000954919,
"ate.actor.restore.duration.ateom_restore":0.147448298,
"ate.actor.restore.duration.total":0.181189689,
"trace_id":"c9d7ea9a5f90a5eb4399bfa0b966d5ac","span_id":"89280ba761137c8e"}
```
The duration keys are the `ate.actor.restore.duration` instrument's name
plus an `ate.snapshot.phase` value, so a log key and the histogram it
mirrors are literally the same identifier in source, which is also why
they're seconds.
Also adds `ateattr.ActorLogAttrs`, the component-log twin of
`ActorLogLabels`, held to the same key set by a test.
Verified on kind:
- the ateom Actor restoring record for the same actor carries the
identical `ate.actor.uid` and `trace_id`, this was impossible before
- `ate_actor_restore_duration_seconds_sum{phase="total"} = 0.181189689`,
matching the log to the nanosecond
- induced a download failure by deleting a snapshot object: record sets
`ate.failure.reason:` `FAILED_GET_EXTERNAL_OBJECT` and correctly omitted
`ateom_restore`; the metric for the same restore shows the same six
phases
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
|
||
|
Haven Xia
|
9df8ca6abe |
[Upgrade] Version the install: node labels and a per-version atelet DaemonSet (#1357)
A rolling upgrade needs two substrate versions running in one cluster, split by node: `atelet` and `ateom` speak a node-local protocol with no cross-version guarantees, so everything on a node has to come from one build. Today nothing records which build a node runs, and the `atelet` DaemonSet is a single fixed-name object, so a second version can only replace the first in place, instead of our porposed node by node rolling ([design](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A)). This PR is the basis for upgrade. Nodes and the `atelet` DaemonSet get keyed by an `ate.dev/substrate-version` label whose value is the build version stamped into the binaries. ### One derivation for the version label `internal/versionlabel` turns the build version into its two forms in one place: | form | grammar | used for | |---|---|---| | label value | k8s label value | node labels, DaemonSet labels, nodeSelectors | | name suffix | DNS-1123 | the DaemonSet name `atelet-<suffix>` | A version that is not a valid label value is rejected, because the label has to match what the `ldflags` stamp put into the binaries through MAKEFILE and `ko`. It's also exposed shell, so the install script can use it. ### A DaemonSet per version `atelet-<suffix>` as the name, the version label on metadata, selector, and pod template, and a pod nodeSelector on the same label. Two versions run side by side on disjoint old/new node sets. ### The install becomes version-aware - `install-ate.sh` reads the version from the same `make ldflags` output it stamps the binaries with, then fills the manifest placeholders. - It labels the nodes that exist at install time, nodes come after it carry no version label yet (so add notes in README). Re-running the install is idempotent, this is the basis for upgrade runbook later. - The `README` documents the invariant, how to read the installed version off the DaemonSet, and that a node added later hosts no workers until it carries the label. On GKE the node pool label is the birth default for new nodes - added in `tools/setup-gcp/README.md`. ### Follow-ups - `ate-setup` need same modification - minimize the affect for daily developer (pin the `VERSION` as `{USER}-dev` instead of from git - the manual rolling-upgrade runbook. Fix #1270 Ref [design doc](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A) |
||
|
Zoe Zhao
|
fe9013a4a2 |
Full cutover: Drop the k8s CRD ActorTemplate fields in the ate apiserver, and update e2e tests (#1353)
This PR is large, I grouped changes to the following commits: - |
||
|
Krisztian F
|
39df566dfa |
(otel): atelet and ateom telemetry now says which node it came from (#1363)
We want to be able to answer questions like: - image cache hit ratio per node - cold starts got slower on one node but nothing we emit says which node anything is on, this PR fixes this. Verified on a local kind cluster, where both atelet and ateom-gvisor now show `k8s_node_name` on `target_info`, including ateom's, which arrives through the OTLP relay. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Zoe Zhao
|
d4fc52b924 |
Report the actor template atespace in OTel tags (#1343)
* Renames the`ate.template.namespace` attribute to `ate.template.atespace` and sources it from the actor's `actor_template` ObjectRef instead of the legacy CRD namespace/name pair. * Updated functional tests now to use substrate ActorTemplates and bind actors through the `actor_template` ref. |
||
|
Haven Xia
|
35344a2c68 |
api: make WorkerPool workerImage required again (#1334)
We should not make workerImage optional as it provide explicit information for what image workers use, removing it will need `atecontroller` to inject workerImage and leads to unnecessary control plane & dataplane wiring. Fixes #861 > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR |
||
|
Tim Hockin
|
955a79f290 |
Add a doc on using DV to validate protobuf (#1327)
API reviewers should memorize this :) @eitanya @juli4n @BenTheElder @dberkov |
||
|
Chuang Wang
|
d909d69053 |
atenet-egress: deliver TLS certs via filesystem SDS so rotation works
Envoy honors watched_directory only on SDS-delivered secrets; on the egress gateway's inline file-based certs it was silently ignored, so kubelet's projected-certificate rotation never reached Envoy and the gateway eventually served an expired certificate. Move the serving cert (both egress manifests) and the spliced extproc cluster's client cert and trust bundle (Go and shell emitters) to filesystem SDS, with the flag-derived SAN pin kept inline via a combined validation context. Fix the csi-deployment doc example that showed the same broken pattern, and add tests for the emitters. |
||
|
Haven Xia
|
7face2c4d2 |
Rename the rest ateomImage to workerImage (#1298)
Some new file added also need the rename. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Haven Xia
|
bdf494999b |
api: rename ateomImage to workerImage and make it optional (#1210)
Rename `WorkerPoolSpec.AteomImage`to `WorkerImage` and drop the constraints so the field can be left unset. This is the basis for let an empty workerImage lets the controller inject a versioned default image chosen by the pool's sandbox class. Part of #861 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
nybidari
|
f45a0561c0 |
Bump default gvisor sandbox config to 2026-08-28 nightly (#1287)
Contains the gVisor fixes for #1090 #992 |
||
|
Jing
|
8eae7ed6a3 |
Emit a runtime-neutral OCI spec. (#1188)
atelet's spec was written for runsc: a hardcoded `runsc` hostname, CRI pause/sandbox annotations and `dev.gvisor.spec.mount.*` hints. The micro-VM shim compensated by rewriting every spec at runtime (ensureKataCompatibleSpec), swapping the whole mount set and rebuilding it from one hand-written builder per volume kind — so a volume kind atelet learned about reached the guest only once the shim was taught about it too. atelet now emits a spec that names no runtime, and each ateom shapes it for the runtime it drives. Whatever atelet adds therefore reaches both, and a bind the micro-VM shaper cannot place in the guest is an error rather than a silent drop. Fixes #709. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
hajiler
|
b4529fcdb7 |
docs: add guide for CSI based external volumes (#1072)
#232 Add documentation guide for CSI volumes in Substrate - [X ] Tests pass - [ X] Appropriate changes to documentation are included in the PR |
||
|
Benjamin Elder
|
cb72d60bdf |
ateom microvm: drop privileged: true, using atelet device plugin for KVM (#1254)
This is the companion to #496 In order to grant access to `/dev/kvm` we have to either: - use a device plugin - use a DRA driver - use an NRI plugin An NRI plugin is highly privileged in it's own right for all pods on the host and is difficult to ship portably at the moment. DRA is promising, but not enough functionality is GA yet at our current 1.35+ target. Device plugin fits reasonably well. We do wind up publishing an ~arbitrarily high limit, which has some cost in kubelet memory, but otherwise is relatively clean. This approach is what kata uses currently. Their device plugin is not available unbundled, and we anyhow have a per-node daemonset. atlet is taught to sniff if /dev/kvm appears on the host at all, so we can also stop using the manually labeled nodes for microVM class and instead schedule to the KVM + TUN devices on nodes that advertise them. Later we can migrate to device plugin by using I implemented that already, but I don't think it's worth merging at the moment. We would want consumable capacity to be on by default. We can migrate later without changing the pod spec by using `extendedResourceName`. https://github.com/agent-substrate/substrate/compare/main...BenTheElder:substrate:ateom-microvm-dra NOTE: I confirmed with upstream that device plugin is not going anywhere despite being "v1beta1", it's GA in all but name. It won't receive new features but we don't really need anyhow. We'll move to DRA down the line. --- By doing this, we can drop `privileged: true` from the uVM ateom pods. We can also drop the `ate.dev/sandboxClass` node label hacks, reducing friction to deploy. |
||
|
Jeff Luo
|
75c9b145fb |
docs: point at the new names of the verify scripts (#1236)
The rename of `hack/verify/verify-metrics.sh` to `hack/verify/metrics.sh` left three references to the old path behind. A reader who copies one of them gets a "No such file or directory" error. Point `AGENTS.md`, `docs/observability.md`, and the header comment of `docs/metrics/registry/metrics.yaml` at the new name. Docs only; no change to the script or to the registry data. A follow-up of https://github.com/agent-substrate/substrate/pull/1221 |
||
|
eliranw
|
afcd5cd718 |
Per-container CPU and memory limits for micro-VM actors (#859)
Per-container CPU and memory limits for micro-VM actors. First slice of #752. An `ActorTemplate` container can cap its own cpu and memory so it cannot starve or kill its siblings in the same actor. A container that exceeds its memory limit is OOM-killed on its own; the rest of the actor is unaffected. ```yaml sandboxClass: microvm containers: - name: trainer resources: limits: {memory: 1500Mi} - name: sidecar resources: limits: {memory: 256Mi, cpu: "0.2"} ``` #752 asks for per-container device selection as well. This PR does the cpu and memory half; devices are the remaining half, and when they land, GPU assignment should follow the same field rather than staying actor-wide as it is today. ## Why micro-VM only The two sandbox classes support opposite halves of #752. Micro-VM actors have a real guest kernel, so each container gets its own cgroup and the limits bind. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty (google/gvisor#190). Measured on a running actor, the workload container's cgroup reported `memory.current=0` while all 20 sandbox processes sat in the pause leaf. A template that sets `resources` with `sandboxClass: gvisor` is rejected at admission. ## How a limit travels `ActorTemplate` → CEL validation → `ate-api-server` resolves each `resource.Quantity` once → `ateletpb` → atelet writes OCI `linux.resources` → `ateom-microvm` merges kata's defaults and checks the guest envelope → `SpecToAgentPB` → kata agent → guest cgroup. The limit is carried as a standard OCI field rather than a substrate-private concept, so a runtime that gains per-container enforcement picks it up without new plumbing. ## Composition with #679 #679 sizes the sandbox itself from `ActorTemplate.spec.resources`: guest RAM and vCPUs for a micro-VM, the sentry for gVisor. This PR subdivides that sandbox. The two are different fields and complementary layers. They collided in one place. #679 applied the actor-level size to **every** container's OCI spec, overwriting the per-container limits atelet writes into the same field, so a container asking for 64Mi silently received the whole guest. This PR removes that call on the micro-VM path. A container now gets a cgroup limit only when it declares one; an undeclared container is bounded by guest RAM, which is the real ceiling. On micro-VM the stamp never reached the guest at all: `SpecToAgentPB` carried neither `Memory` nor `CPU.Quota`, so it stopped at the bundle's `config.json`. Running the pre-PR pipeline shows `Quota=50000 Memory=2Gi` on disk arriving as `Quota=0 Memory=<nil>` on the wire. gVisor is untouched. There the actor-level size is applied to the `pause` container, and since one sentry backs every container, that is the only cgroup that binds. Two consequences worth calling out for reviewers of #679: - The `sizing.SandboxSize` threaded into `buildActorContainers` and `ensureKataCompatibleSpec` had no remaining reader, so it and the `guestSize` helper are removed. `resolveGuestMemMiB` still sizes the VM and still rejects a declared limit too small to boot. - `internal/sizing`'s package doc claimed both runtimes shared `ApplyToOCISpec` for container cgroups. That is now gVisor-only, and the comments say so. No code in that file changed. `checkResourceEnvelope` runs after `resolveGuestMemMiB`, so it validates the per-container sum against the post-reserve guest rather than the SandboxConfig default. Its error now names `spec.resources.limits.memory` (or `.cpu`) when the actor declared a size, and `SandboxConfig` when it did not. `internal/e2e/suites/sizing` is unaffected: it asserts on `num_cpu` and `mem_total_bytes`, both of which come from VM sizing, and only logs the cgroup files. ## Verified on hardware Same template, run twice, one commit apart on a micro-VM actor: | | before | after | | :--- | :--- | :--- | | `hog_ovl/memory.max` | `max` | `67108864` (the declared 64Mi) | | `bystander_ovl/memory.max` | `max` | `max` | | hog allocates 128MB | survived | OOM-killed | | bystander | alive | alive | The bug in between: `SpecToAgentPB` converted only `Devices` and `CPU.Shares` out of `Linux.Resources`, so a memory limit reached the bundle's `config.json` and was dropped on the way to the agent. Every unit test passed and the on-disk spec was correct while the feature did nothing. ## Notes for review - `ContainerResources` deliberately does not reuse `corev1.ResourceRequirements`, which also carries `requests` and `claims`. There is no scheduler inside an actor to hint at, and for memory a soft request cannot express "must have this much to come back at all". #679 reached the same conclusion from the other direction and now rejects `spec.resources.requests` and `.claims` at admission. - `Limits` is a bounded named type. A `MaxProperties` marker on the field lands at the wrong schema level for a named map, and without a bound the CEL cost estimator rejects the whole schema. This CRD is close to its schema-wide CEL cost ceiling; a rule that iterates containers and parses quantities exceeds it by more than 100x and takes the existing rules down with it. - A cpu limit below `10m` is raised to `10m`: the kernel rejects a CFS quota under 1ms. - `SpecToAgentPB` drops a non-positive quota and a zero period rather than forwarding them. A non-positive quota means unlimited in OCI but was sent as a literal zero, which the guest applies as no CPU at all; a non-nil zero period overwrote the CFS default set for a live quota. Both now read a spec the way `cpuLimitMillis` does, so the envelope check and the conversion agree on the same input. - `mergeKataResources` fills the gaps kata's defaults cover rather than allowlisting known fields, so a field it does not recognise survives the merge. That holds only as far as the merge: `SpecToAgentPB` converts `Devices`, `Memory` and `CPU` only, so `Pids`, `BlockIO`, `HugepageLimits` and `Network` stop at the ttrpc boundary. Nothing sets them today. ## Known gaps - An OOM-killed container is not reported above the guest. The actor stays `STATUS_RUNNING` and nothing records which container died. Raised on #550. Note that #961 settled workload stats at the sandbox level and deliberately does not attribute per container, so surfacing which container died needs a separate signal: the guest cgroup's `memory.events`, not the stats path. - A container's writable rootfs is a guest tmpfs, so filesystem writes are charged to its memory limit and a container writing more than its limit to `/tmp` is OOM-killed. Documented in `api-guide.md`; separating the two budgets needs an `ephemeral-storage` field. - Over-subscription across containers is caught when the actor starts, not at apply time. An admission-time sum check is not implementable within the CEL cost budget. - The atelet hop that attaches resources to the bundle spec has no test: replacing `ctr.GetResources()` with `nil` leaves the whole suite green. Closing it needs an imagecache fixture for `prepareOCIDirectory`. - Restored actors inherit the golden's cgroups rather than applying their own spec. Correct today only because `ActorTemplateSpec` is immutable. - The GPU section of `api-guide.md` still says `ActorTemplate` has no per-container resource fields, which this PR makes false. Correcting it is left to the devices half of #752, which is what will change the GPU behaviour that paragraph describes. --- - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Eliran Wolff <eliranw@nvidia.com> |
||
|
Jeff Luo
|
27a9412c10 |
observability: add a metric registry that CI checks (#1097)
`docs/observability.md` was the only description of the metrics of Substrate, and nothing compared that text with the code. It listed 13 instruments; the code sends 21. The request-parking and actor resource-usage instruments were missing entirely. This change adds an [OpenTelemetry Weaver](https://github.com/open-telemetry/weaver) semantic convention registry. ### What is in it | Path | Content | |---|---| | `docs/metrics/registry/manifest.yaml` | The Weaver manifest: name and schema URL. | | `docs/metrics/registry/metrics.yaml` | 21 instruments and 12 attribute groups: every label, its permitted values, and its buckets. | | `docs/metrics/substrate.yaml` | The rules Weaver cannot express. | | `hack/verify/verify-metrics.sh` | The check. | Fixes #1093 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
|
3a3ec62ff3 |
fix: land the three root-caused flaky-test fixes (identity, parking, relay) (#1160)
Fixes #675, fixes #1100, fixes #1146; addresses the CI flake in #1106.
## Why this PR
Flaky tests are the single biggest drag on this repo's velocity right
now: the three flakes fixed here account for the majority of red CI runs
over the last 7 days (identity: 35 failures, parking: 32, relay: 9 —
from the flake dashboard's cross-PR analysis of ~630 runs). Every red
run costs a contributor a rebase-and-rerun cycle and costs reviewers
signal. **This PR consolidates the three root-caused, in-flight fixes
into one change to get CI green now and unblock the community — the goal
is velocity, not authorship.**
## Credit where it's due
All three fixes were root-caused and written by others; this PR adopts
them onto latest main with their tests, unchanged in substance. Each
commit carries a `Co-authored-by` trailer:
| Commit | Original PR | Author | Root cause |
|---|---|---|---|
| e2e: give each probe fixture its own worker pool | #1147 |
@orangeCatDeveloper | identity/egressmitm/imagevolume suites share one
`workload: probe` pool label; cross-suite selection under concurrent
suite processes dials workers that are not there |
| atenet: never cancel an in-flight resume at the park budget | #991 |
@omeryahud | the park budget doubled as the ResumeActor RPC deadline; a
mid-restore cancel strands a RESUMING actor on a live worker |
| atunnel: close the relay's both ends before returning | #1101 |
@orangeCatDeveloper | the relay closed both ends from a
`context.AfterFunc` goroutine the test never waits for |
@Stevenjin8's #1107 correctly diagnosed the ateom readiness race in
#1106; the control-plane readiness gap it targets remains real and open
— this PR only removes the e2e-fixture contention that makes it fire
constantly in CI.
If maintainers prefer to land the original PRs individually instead,
closing this one is completely fine — the point is that the fixes land
somewhere, soon.
## Evidence the flakes are actually fixed
**TestRelayIngressCancellationClosesBothSides (unit, `-race`):**
- Unpatched main, `-count=3000`: **83 failures (2.8%)** — matches the
2.9% observed across 308 CI runs this week
- This branch, `-count=10000`: **0 failures**
**TestRequestParking (park-budget cancellation):**
- The new `InFlightAttemptRunsToCompletion` and
`LateRetryableErrorIsBudgetExhaustion` unit tests (from #991) encode the
exact failure mode from #675 and pass under `go test -race -count=100
./cmd/atenet/internal/router/ingress/`
- The pre-fix behavior (budget cancelling the in-flight RPC) is
deterministically reproduced by the old test it replaces
**TestActorIdentity_AfterRestore_IsOwnID_NotGolden (probe pool
isolation):**
- Not reproducible outside CI (needs concurrent suite processes on a
contended kind node), so verified statically: `${FIXTURE_SUFFIX}` is
always `-<suite>` (internal/e2e/sandbox.go:189,201 — never empty),
`probe-sized` already uses its own label, and no other manifest or
selector references `workload: probe`. #1147's CI data shows all three
failure signatures (missing `ateom.sock`, `runsc restore` killed, router
502/503) trace to cross-suite pool sharing; per-suite labels make the
selector suite-local by construction
- The definitive check is this PR's own CI plus the flake dashboard's
7-day window after merge — I will report the post-merge rates on #1106
Also run: `go build ./...`, `go vet` and the full `-race` suites of both
touched packages — all green.
## What this PR deliberately does NOT fix
`TestActorEgressHTTPS` (#1050, 4.6% this week, below the 5% flake
threshold) has no root-caused fix yet — the 503 `upstream connect error`
path needs investigation in a live cluster. #1103 (@orangeCatDeveloper)
tightens the related `TestActorArbitraryPortAccess` assertion so those
503s stop passing silently; it should land after #1050's cause is fixed,
or it converts hidden flakiness into visible red.
## Update (post-CI investigation)
The first e2e runs failed on `TestRequestParking/ParkThenServed`
(micro-VM lane). Investigation showed this is the **pre-existing
dominant mode** of #675 — identical failures in main-era runs
32305728993 / 32397291519 / 32487958077 — not a regression: on micro-VM,
`SuspendActor` returns before the snapshot upload completes, so the
worker legitimately isn't free within the 5s park budget and the
router's 503 is correct behavior. #991 fixes the *other* (mid-restore
cancellation/stranding) mode. Commit
|
||
|
Julian Gutierrez Oschmann
|
fbe580e4f6 |
Fix resource names for ActorSnapshotTag methods. (#1167)
Also clarified the style-guide about the naming convention. Fixes #1163 |
||
|
Jet Chiang
|
60073ecd57 |
Replace ateredis with atepg (#940)
## Summary A follow up to #640 where we introduced PostgreSQL as an alternative storage backend, selected conditionally in ateapi. - Deleted ateredis, its tests, and its dependencies - Removed Redis backend selection and configuration so ateapi always connects to Postgres - Replaced Valkey resources with Postgres in the standard and Kind deployment paths and simplified install script - Replaced miniredis fixtures with isolated Postgres testcontainers and added centralized helpers for seeding resources - Renamed Redis-specific debug flush command to backend-neutral `debug-clear-store` in CLI - Updated comments and docs where applicable ## Benchmarking Extensive benchmarking have been performed to evaluate Redis vs Postgres, and results can be found in these two documents: - https://docs.google.com/document/d/10K0wB6aTeFkJCL4HN3NbLJCdFGoLYdhIcqkFnqFHkKc/edit?usp=sharing - https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing --------- Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io> |
||
|
Luiz Oliveira
|
31a2e3850a |
Simplify Update methods to do a whole object replace (#1108)
* Removed field_mask from the API * Added a new protoupdate package to handle replacing mutable fields. This makes sure that unknown fields in the server are not dropped by an update from a stale/old client. #1011 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Haven Xia
|
640654df47 |
kubectl-ate: add --container log filters for actor logs (#1067)
Part of #294 as #311 has been inactive for over a month. Example ``` kubectl ate logs actors test -a demo # default: all containers + lifecycle kubectl ate logs actors test -a demo -c counter # only specified containers ``` Scoped this down to --container only after discussing with @BenTheElder — we are not sure the naming of the container related logs, the supervisor output may end up as a describe/events-style like how k8s does? - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Max Thompson
|
074fd1c12b |
Add trustBundle as a SystemInfo volume data source (#941)
Part of #932 (PR 1 of 3). Adds the user-declarable trustBundle data source for SystemInfo volumes (#802) and the end-to-end proof that the projected anchors work against the MITM egress gateway. Live refresh for running actors (PR 2) and auto-injection (PR 3) come separately. What this adds A SystemInfo volume data source that projects the trust anchors of a named trust bundle to a PEM file: volumes: - name: trust systemInfo: dataSources: - trustBundle: name: egress-mitm.ate.dev path: egress-ca.pem Inspired by the Kubernetes clusterTrustBundle projected volume source, but source-neutral: the template names a bundle; where it's fetched from is a deployment concern, not part of the API. Design points - Resolution lives on the node. The wire carries only {name, path}; atelet resolves the name at write time through an informer-backed lister on ClusterTrustBundles and writes the sanitized PEM with the temp+rename discipline from #803 (find-paths safe). Contents refresh on every Run/Restore. ateapi is not involved, per review discussion — the same informer is what live refresh (PR 2) will hang off. - Allowlist in atelet, not the CRD schema. Today only egress-mitm.ate.dev (the egress gateway CA bundle, #823), mapped to the ClusterTrustBundle that atecontroller's EgressMITMTrustReconciler (#946) derives from the egress-mitm-ca-pool Secret. The signer-linked object name stays a backend detail; the future backend registry (#932) widens the allowlist without an API change. - The watch is scoped to the one backing object via a metadata.name field selector — this informer runs on every node, so an unfiltered watch would fan every ClusterTrustBundle in the cluster out to every atelet. RBAC can't express this (resourceNames doesn't apply to list/watch), so the field selector is the enforcement point. get/list/watch on clustertrustbundles moves to the atelet ClusterRole. - No availability probe. The informer registers unconditionally; a cluster that doesn't serve the feature-gated certificates.k8s.io/v1beta1 blocks atelet startup at cache sync, with the reflector errors naming the missing API (hack/create-kind-cluster.sh enables the gate). - Fail-closed. Unknown names, missing bundles, and unusable bundles fail actor start naming the bundle — an actor that declared a trust bundle must not start without one. - Kubelet-parity sanitization (internal/pemutil): CERTIFICATE blocks only, deduplicated, headers stripped, and anchors deliberately shuffled so consumers can't grow a dependence on order. - Schema note: dataSources MaxItems tightened 32→8 while adding the trustBundle member. Vacuous in practice (the old schema couldn't admit more than one entry), but flagged since it's ratchet-shaped. E2E — delivery and consumption Delivery (identity suite, both sandbox classes): provisions the egress-mitm-ca-pool Secret and drives the real #946 reconciler (writing the bundle directly isn't possible — the reconciler reverts hand-edits), asserts the projected file byte-exact, then rotates the pool across a suspend/resume to prove refresh-on-restore. Since the probe fixture is shared and fail-closed, e2e.DeployProbe itself ensures the bundle exists for whatever suite deploys it. Consumption (new egressmitm suite, both sandbox classes): deploys the sdsmint (MITM) egress gateway and proves an actor completes a TLS handshake with the gateway's per-SNI minted leaf using ONLY the projected anchors — plus a system-roots negative control that must fail. The pair is unambiguous in both directions: the positive can't pass under passthrough (the bundle holds no public CAs), and the negative can't fail under passthrough. CI: two steps appended to the existing e2e job after the standard lanes (the gateway swap is cluster-wide and breaks passthrough assumptions): --deploy-atenet --experimental-use-sdsmint redeploys only the atenet components, then the egressmitm suite runs once per sandbox class. Flake mitigation: the probe fixture pool drops from 3 workers to 2. Each suite deploys its own copy and drives one actor at a time, so the third worker per copy was idle memory multiplied across suites on the one-node CI cluster — pressure that has been killing sandboxes mid-test (runsc: signal: killed, a vanished ateom socket) on this PR and on main's identity suite. This reduces the pressure; right-sizing e2e concurrency or worker-pod QoS cluster-wide is follow-up material. Not in this PR - Live refresh for running actors (#932 PR 2) — until then, a running actor's file is the bundle as of its last Run/Restore, and correctness rests on overlap rotation by the bundle publisher. - Auto-injection of the egress trust volume (#932 PR 3). - Configurable backend registry (#932) — the allowlist is the seam it will replace. |
||
|
Sneha-at
|
b7080602c6 |
Allow deleting an actor from any state. (#788)
Fixes #643 This feature introduces the ability to delete Substrate actors from any lifecycle state (e.g., RUNNING, PAUSED, PENDING), rather than requiring them to be hibernated into SUSPENDED (or CRASHED) beforehand. This is crucial for cleaning up stuck or failed actors. #### 1. Control Plane & gRPC API (pkg/proto/ateapipb, ateapi) • DeleteActorRequest.any_state: Added a new boolean field any_state to DeleteActorRequest. • When false (default): enforces the existing behavior where only actors in ACTOR_STATE_SUSPENDED or ACTOR_STATE_CRASHED (or already ACTOR_STATE_DELETING) can be deleted. • When true: allows deleting an actor in any state (e.g., RUNNING, PAUSED). • Orchestrated Cleanup Workflow (workflow_delete.go): 1. Transitions actor state to ACTOR_STATE_DELETING. 2. Calls atelet.Terminate to stop live workloads on the node. 3. Detaches external volumes from the worker node (with fallback logic using actor status if the ActorTemplate was deleted). 4. Releases the assigned physical worker Pod back to the WorkerPool. 5. Deletes external storage volumes through the storage plugins. 6. Finalizes removal of the actor from the persistence store. #### 2. Node Herder Agent (atelet) • Terminate RPC (main.go): Added a new Terminate method to AteomHerder: • Calls ateom.TerminateWorkload on the target worker pod. • Unmounts external volumes mounted for the actor on the node. • Cleans up and resets actor runtime host directories (OCI bundles, checkpoints, pid files). #### 3. In-Pod Sandboxes (ateom-gvisor & ateom-microvm) • TerminateWorkload RPC: • gVisor (main.go): Stops and deletes active runsc containers, unmounts bundle rootfs overlays, and tears down the actor's interior network namespace. • Micro-VM (checkpoint.go): Shuts down the Cloud Hypervisor VMM, unmounts bundle rootfs overlays, and cleans up the network namespace. • Emits an "Actor terminated" lifecycle log and clears the active actor attribution so the worker can host new workloads. #### 4. CLI (kubectl-ate) • --any-state Flag (delete_actor.go): kubectl-ate delete actor <actor-name> -a <atespace> --any-state ────── ## Verification [x] Changes tested in local with ``` • go build ./...: Passed go test ./...: Passed (all unit and functional tests pass) make lint: Passed (no lint errors) ``` Detailed manual tests ``` ### Scenario 1: Force Delete a Running Actor with --any-state 1. Create the actor: kubectl ate create actor manual-test-1 --template=ate-demo-counter-microvm/counter-microvm -a demo 2. Resume the actor: kubectl ate resume actor manual-test-1 -a demo 3. Force delete the running actor: kubectl ate delete actor manual-test-1 -a demo --any-state actor "manual-test-1" deleted ``` ────── ### Scenario 2: Standard Delete a Suspended Actor ``` 1. Create the actor: kubectl ate create actor manual-test-2 --template=ate-demo-counter-microvm/counter-microvm -a demo 2. Resume the actor: kubectl ate resume actor manual-test-2 -a demo 3. Suspend the actor: kubectl ate suspend actor manual-test-2 -a demo 4. Standard delete the suspended actor: kubectl ate delete actor manual-test-2 -a demo actor "manual-test-2" deleted ``` ────── ### Unit & E2E Tests # Unit & Functional Tests go test ./cmd/ateapi/internal/controlapi -run "TestDeleteActorWorkflow|TestEnsureMarkedDeleting" # E2E Tests E2E_TEMPLATE_NAMESPACE=ate-demo-counter-microvm E2E_TEMPLATE_NAME=counter-microvm ./hack/run-e2e-kind.sh ./internal/e2e/suites/demo -run TestActorLifecycle ./hack/run-e2e-kind.sh ./internal/e2e/suites/demo -run TestForceDeleteActorWithExternalVolume ``` - [ x] Appropriate changes to documentation are included in the PR |
||
|
Youssuf Elshall
|
86a736c90c |
workerpool: propagate template metadata to workers (#1058)
Part of #212 > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR ## Summary - add bounded, validated labels and annotations to `WorkerPoolPodTemplate` - propagate that metadata to the generated Deployment and worker pod template - reserve the controller-owned `ate.dev/worker-pool` label - regenerate the WorkerPool CRD and deepcopy code - add API validation, controller tests, and documentation I chose intentionally to avoid exposing a complete `PodTemplateSpec` as per the discussion in the #212. |
||
|
Michael Taufen
|
bbf9d440c6 | Security status report SKILL (#680) | ||
|
Luiz Oliveira
|
fc02127bfe |
ateapi - Require preconditions in Update methods (#1014)
https://github.com/agent-substrate/substrate/issues/954 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Max ThompsonandTaahir Ahmed
|
5edb1af1fe |
feat: add systemInfo volume source with actorIdentity data source (#803)
Part of #802 (first PR: the actorIdentity data source; does not close the issue). ## What changed and why Adds a systemInfo volume source to ActorTemplate — a read-only volume whose files are generated by atelet on every Run/Restore, analogous to Kubernetes projected volumes. The initial data source, actorIdentity, writes the actor's own name to a configurable relative path: ``` spec: volumes: - name: system-info systemInfo: dataSources: # Part 1 (this PR): own-metadata projection, downwardAPI-style - actorMetadata: items: - field: name # enum: name | atespace | uid path: actor-name - field: atespace path: atespace - field: uid path: actor-uid containers: - name: main image: app@sha256:... volumeMounts: - name: system-info mountPath: /run/ate ``` Because the files are regenerated before the sandbox starts, they carry the resumed actor's own values regardless of what checkpointed state it boots from — the property the old hardcoded /run/ate identity mount provided, now as an explicit, extensible API that future data sources (identity JWTs, certificates — see #802) can slot into. **Behavior change:** the automatic /run/ate/actor-id mount is removed; actors must opt in by declaring the volume (the e2e identity probe in this PR is the reference example). ### Reviewer notes: - Over half the diff is vendored + generated code (cmd/atelet/internal/third_party/atomicwriter/, atelet.pb.go, zz_generated.deepcopy.go, the CRD manifest). The hand-written surface is ~700 lines. - System-info volume roots live under a new ActorPath/system-info/ host dir, deliberately separate from durable-dir/: the micro-VM durable machinery snapshots everything under the durable-dir root, and generated identity files must never be captured into snapshots. - Supports microVM as well as gVisor. - The e2e identity suite exercises the new API end-to-end with unchanged probe binary and assertions. It runs in the kind-cluster CI job (not run locally). ## Checklist - [x] Issue is linked above - [x] Tests pass locally (go test ./...) - [x] Root-gated tests pass if applicable (N/A — no root-gated packages touched) - [x] Documentation updated if behavior changed (docs/api-guide.md: SystemInfo Volumes section with example) --------- Co-authored-by: Taahir Ahmed <taahm@google.com> |
||
|
Noureldin
|
01c92e9da3 |
Allow actors to opt in/out of Linux capabilities (#795)
Fixes #744 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Maya Wang
|
c618baa593 |
docs: convention for integration repository structure and naming (#712)
## Summary Adds `docs/integration-repos.md`: where end-to-end integrations live, how their repositories are named, and how the fixes they need flow back into core. The convention in one line — trivial demos stay in the core repo, each non-trivial integration gets one dedicated repo under the `agent-substrate` org, and core gaps get closed by making core configurable with defaults unchanged rather than by patching it downstream. ## Why now We are about to create the first real, end-to-end integrations rather than counter-style demos: a code-execution sandbox, and an always-on agent. Both are large enough to need their own images, dependencies, and release cadence. Whichever repository gets created first will set the precedent for every one after it. This writes the convention down so that precedent is chosen deliberately instead of inherited by accident. ## What it covers - **Where code lives** — the core-repo/dedicated-repo split, the rough test for which side something falls on (API keys, external services, third-party accounts), and why this is a set of peer repos rather than a second org. - **Naming** — capability-named for general capabilities (`code-execution-sandbox`), integration-named for specific third-party products, named for the product rather than the vendor behind it. Plus what to avoid: over-broad names, names that clone a vendor's API or brand, and the redundant `-integration` suffix. - **Third-party names** — allowed descriptively, with a non-affiliation note in the repo README, and brand/policy edge cases cleared before the repo exists. - **Upstreaming** — the part with teeth for this repo. Integration repos that accumulate local patches against core bitrot, and the gap they work around stays invisible to everyone else. So: prefer making core behavior configurable with defaults unchanged. #487 and #465 are linked as illustrations of that pattern — this PR does not depend on either, and branches from `main`. - **Two worked examples** that validate the convention rather than just following it, including the third-party-name edge case. ## Review This was announced at the community meeting and circulated as a shared design doc with a 7-day review window, which has now closed. It synthesizes the `#integrations` thread discussion. Comment history: <https://docs.google.com/document/d/1Tb6u0b1XSvWrNpoyD4jdsQaJ58aAgDtQOM18uxujs-8/edit> This PR is the trimmed version: doc-review scaffolding — status block, reviewer list, self-link — is dropped, and only the durable convention is carried over. ## Left open Two questions are deliberately out of scope, called out in the doc rather than answered. Both are maintainer calls and neither blocks the first repositories: - Governance tiers — whether to distinguish "official" from "community" integrations with different review bars, as Home Assistant and Obsidian do. - Who creates integration repositories and grants per-integration maintainer access. ## Also in this PR - README gets an entry in the docs list, matching every other file in `docs/`. - `CONTRIBUTING.md` gets one sentence pointing there, since "where does my integration go?" is a question a contributor asks before opening a PR. Fixes #<issue_number_goes_here> > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Krisztian F
|
c3c6876985 |
(chore): consolidate logging to ateattr (#1035)
Actor logs used `ate.dev/actor_*` while spans and metrics use `ate.*` registry in internal/ateattr. This PR makes ateattr the single source of truth for everything telemetry-related. - Renamed the six actor log labels onto the registry: - `ate.atespace`, `ate.actor.name`, `ate.actor.uid`, `ate.template.namespace`, `ate.template.name`, `ate.actor.container.name` - Logs join traces now. Records set `trace_id`, `span_id` and `trace_flags`, so you can go from Actor restored to the resume that caused it. Our own lines only, not an actor's stdout: one goroutine forwards a whole container stream and can't know which request produced a given line. Per line correlation comes with #853. - Actors can't fake platform labels. They already couldn't overwrite ours, but they could invent new ones like `ate.tenant` that look platform issued downstream. Anything under `ate.` from an actor is now dropped. - Fixed the asymmetry that was actually left: actor supplied label values weren't stringified, and one non string value makes Cloud Logging discard the labels for that whole entry. - Note: the actor_uid bullet in the issue is stale, #841 fixed it earlier. Lifecycle records still set five labels rather than six, on purpose as they're about the actor, so no container produced them. Fixes #886 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: krisztianfekete <git@krisztianfekete.org> |
||
|
Julian Gutierrez Oschmann
|
4c1bd9d36b |
Add status fields to Substrate resources (#1025)
Add `status` fields to all Substrate resources that need it. Move server-owned fields under it. Fixes #1006 . |
||
|
ILLUM1N0X
|
2af5308b4d |
ateapi: return ResourceExhausted when no worker is free (#908)
ResumeActor returned `FailedPrecondition` both when the fleet had no free worker and when the actor was not in a resumable state. Only the former is fixable by suspending the actor to drop its node pinning, and a client driving that recovery could not tell the two apart. `assignWorkerAttempt()` now returns `ResourceExhausted` for `scheduling.ErrNoCapacity`. `FailedPrecondition` was the de-facto capacity signal, so three router consumers move with it: - `retryable()`: `ResourceExhausted` is parked, so saturation behaves as before. `FailedPrecondition` stays retryable on purpose. It no longer carries saturation, but it does cover states a concurrent operation can move the actor out of. - `mapResumeError()`: `ResourceExhausted` now preserves the gRPC description, keeping saturation at 503 as today. Its former 429 "rate limited" branch was written speculatively as nothing in substrate produced the code until now, and the fleet being full is not the caller sending too many requests. - `classifyOutcome()`: `no_capacity` moves to `ResourceExhausted`, and `FailedPrecondition` gets its own label so the capacity bucket stops collecting non-capacity errors. `TestResumeActor_RelocatesAfterSuspendFromPaused` covers the recovery flow this signal exists for: a PAUSED actor pinned to a full node cannot resume, and after a suspend clears the pinning it schedules onto a worker on another node. Fixes #660 - `make test` clean (except the pre-existing macOS-only `internal/atunnel` failures, untouched by this PR) - `make fmt` clean - `go vet ./cmd/...` clean |
||
|
Jeff LuoandKrisztian F
|
9bfc67e815 |
benchmarking: add observability benchmarking for the OTel collector (#749)
Part of #563 Substrate had no way to measure how much telemetry it sends. This PR adds one, and builds it from the parts the benchmark suite already has. The managed collector sends its self-metrics to Cloud Monitoring only, and gives no data for each source. Thus you cannot see which component makes the volume. A second collector, the telemetry meter, counts the spans and the datapoints by `service.name`. ## Contents - **`benchmarking/telemetry/meter.yaml`** — the meter. A tee: it counts the telemetry by service, then sends it on to the managed collector. Thus one run gives the volume for each service and the cost of the load together. - **`benchmarking/telemetry/README.md`** — the method: install, measure, read, remove. - **`hack/install-ate.sh --otlp-endpoint`** — sends all control plane telemetry to the meter. Each component reads the endpoint from the `ate-otel-config` ConfigMap through `envFrom`, and `ate-controller` copies the value to the ateom worker pods, thus one patch is sufficient for all of them. - **`benchmarking/workloads/deploy.sh --otlp-endpoint`** — the same for actor containers, which substrate configures separately. - **`benchmarking/monitoring.yaml`** — scrapes the meter and cAdvisor. - **`benchmarking/observability.md`** — prerequisites and the scenario ladder. - **`docs/dev/best-practices/otel-collector.md`** — the volume model and the size guidance, corrected for the `ParentBased` defaults from #711. ## Correction The `glutton` ActorTemplate had no `env`. Substrate puts no OTLP configuration in an actor container, thus the exporter used the SDK default `localhost:4317`, no process listens on that address in the sandbox, and substrate discarded each actor span with no error message. The template now sets `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES`. `ActorTemplate.spec` is immutable, thus `deploy.sh --deploy` deletes the templates before it applies them. ## How I verified this On a GKE cluster with managed OTel. The sequence: ```bash # 1. Start from a clean state. ./hack/install-ate.sh --delete-benchmarks ./hack/install-ate.sh --delete-ate-system # 2. Install substrate from this branch. ./hack/install-ate.sh --deploy-ate-system # 3. Install the meter and Prometheus. kubectl apply -f benchmarking/telemetry/meter.yaml kubectl apply -f benchmarking/monitoring.yaml # 4. Send the control plane to the meter. METER=http://telemetry-meter.benchmarking.svc.cluster.local:4317 ./hack/install-ate.sh --deploy-ate-system --otlp-endpoint "${METER}" # 5. Install the workloads and locust, with the actors on the meter also. ./benchmarking/deploy_locust.sh --deploy --worker-count 10 \ --otlp-endpoint "${METER}" # 6. Read the volume. kubectl port-forward -n benchmarking svc/prometheus 9090:9090 ``` ## Result The idle floor, from one measurement on 10 workers: | Service | datapoints/min | |---|---| | `ate-controller` | 234.8 | | `ateapi` | 54.3 | | `atelet` | 44.9 | | `atenet-router` | 10.4 | | `ateom-gvisor` | 5.2 | | `glutton-actor` | 4.2 | | **Total** | **353.7** | The `glutton-actor` row is the correction above. Before it, actor telemetry could not arrive at all. Positive control for the same window: `otelcol_receiver_accepted_metric_points` 10.5/s, and each `otelcol_receiver_refused_*` counter at zero. Thus the zero values in the table are true zeros, and not data that did not arrive. Two values differ from the earlier manual measurement. `ate-controller` is now the largest source, and `ateom-gvisor` is no longer zero. The new imagecache counters from #831 are a probable cause. This difference is the reason to keep results out of the repository. ### Traces Trace volume follows the root sample decision, not the rate of each component. `atelet` and `ateom-gvisor` sent zero spans in each run, because they inherit the decision that `ateapi` makes and start no trace of their own. At idle each component is near zero: a trace needs an operation, but a metric pushes on a timer. This run gives no span volume for a usual load. The actors crashed on suspend, and the failed retries made an artificial rate of 13.7k spans/s. That rate is not a measurement of substrate. It does show that the meter accepted the spans with no refusals. ## Follow-up A separate PR adds the ladder to `automation/tests.yaml` and writes the results as artifacts of each run. The open items are at the end of `observability.md`. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Co-authored-by: Krisztian F <103492698+krisztianfekete@users.noreply.github.com> |
||
|
Jeff Luo
|
6dd24af84a |
ateapi: pair ate.workerpool.namespace with ate.workerpool.name (#953)
ate.actor.crashes, ate.actor.lifecycle.operation.duration and ate.scheduler.assignment.duration named a WorkerPool by name alone. A WorkerPool is namespaced, so same-named pools in different namespaces merged into one series, and the three could not join the instruments that already carry both keys. ateattr.WorkerPoolAttributes now builds the pair, and omits both keys when no pool is assigned, so a crash before the actor reached a worker no longer reports an empty-string pool. This changes the series identity of the three instruments. Fixes #951 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Keith Mattix II
|
89b83a5c89 |
Support CONNECT in atenet router (#715)
added CONNECT support for atenet ingress to support arbitrary actor ports |
||
|
Chenyi Wang
|
c178f5b74e |
ateom: export telemetry through an atelet unix-socket OTLP relay (#809)
ateom runs inside the worker pod that hosts the actor, and exported OTLP straight to the collector over the pod's network. This adds a node-local relay: ateom pushes OTLP/gRPC over a unix socket that atelet serves and forwards to the collector, so a worker pod needs no network path of its own to export spans and metrics. Four things motivate it: - Blast radius. The pod runs untrusted agent code, so allowing it egress to the collector makes the collector reachable to anything that escapes the sandbox. A unix socket cannot leave the node. - Connection count. Worker pods are heavily oversubscribed; N ateoms per node each held their own collector connection. They collapse into atelet's single per-node one. - Interference. ateom transparently redirects actor egress to its own atunnel listener, and its own outbound traffic has to stay clear of the rules it installs. A unix socket is not IP traffic. - Shutdown loss. Teardown frees the actor's network and then the pod goes away, which is when the spans describing teardown are still queued in the batch processor. atelet outlives the worker pod. The relay forwards the OTLP request verbatim rather than decoding and re-exporting, so each ateom's own resource (service.name, service.instance.id) survives instead of being absorbed into atelet's. It is best-effort: an ateom that finds no socket at startup logs it and exports directly to OTEL_EXPORTER_OTLP_ENDPOINT as before, so this is a no-op for a cluster running an older atelet. atelet likewise declines to serve a relay when no collector is configured, since it would accept spans only to drop them. Both halves stay off with --otlp-relay-socket="". The socket lives in ateompath.BasePath, the hostPath already mounted at the same path into atelet and into every ateom pod, so no new volume or controller change is needed. Also includes: - End-to-end tests covering the full serverboot-to-collector path. - Observability documentation updates for Jaeger tracing. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Chenyi Wang
|
71b006309f |
Right-size actor sandboxes to declared ActorTemplate resource limits (#679)
Actors previously ran in sandboxes sized to the whole node; there was no way to declare how much CPU/memory a given actor should get. This adds an explicit, immutable sizing knob on the ActorTemplate and plumbs it into the sandbox's OCI spec for both the gVisor and micro-VM runtimes. **API** — `ActorTemplate.spec.resources` (`*corev1.ResourceRequirements`). The `limits` size the sandbox and are baked into the immutable spec; the CRD and generated code are regenerated accordingly. **internal/sizing** (shared by both runtimes) — new `SandboxSize` value (`FromLimits` / `VCPUs` / `ApplyToOCISpec`). `ApplyToOCISpec` writes CPU quota+period and the memory limit onto the OCI spec, and is a no-op when neither dimension is set, so 0 means "unconstrained". `VCPUs` rounds milliCPU up to whole vCPUs. **Plumbing** — ateapi reads the template limits (`actorResourceLimits` → `tmpl.Spec.Resources`) and supplies `CpuMilli`/`MemoryBytes` over the actor RPCs (ateapi → atelet → ateom); ateom applies them — gVisor via the cgroup leaf (`runsc --cpu-num-from-quota` provisions the sentry vCPU count), micro-VM via the guest VmConfig. The two fields are carried on the proto messages. **Scheduling** — worker capacity is taken from the WorkerPool's per-worker limits and advertised on the Worker; the scheduler only places an actor on a worker whose capacity >= the actor's declared limits. A missing worker or actor dimension is treated as unconstrained, so placement is never blocked by absent data. **Docs & demos** — document the model in `api-guide.md`; the counter, sandbox, and micro-VM demos declare actor limits, and their WorkerPool comments now describe the real model (worker limits size the worker pod + advertise scheduling capacity; the sandbox itself is sized by `ActorTemplate.spec.resources`). - [x] Tests pass (unit tests for sizing + scheduling; e2e suite in `internal/e2e/suites/sizing` resumes an actor and asserts, via the probe fixture's `/resources` endpoint, that the running sandbox observes the declared CPU/memory from the inside) - [x] Appropriate changes to documentation are included in the PR |
||
|
Eitan Yarmush
|
2b3a4715c6 |
Configurable JWT authentication to ateapi (#757)
added configurable JWT authentication to ateapi. |
||
|
Jeff Luo
|
1da82f3e6a |
imagecache: add ate.imagecache.requests hit and miss telemetry (#831) (#834)
Add the `ate.imagecache.requests` counter. **`ate.imagecache.outcome`** (new key in `internal/ateattr`) | Value | Meaning | |---|---| | `hit` | the node holds a complete image record: each layer directory that the record names is present | | `miss` | the lookup must pull | | `error` | the lookup failed; the only outcome that carries `error.type` | | `cancelled`, `timeout` | the caller gave up, so the cache is not at fault | A failed lookup is neither a hit nor a miss, so it gets its own outcome, as `no_free_worker` does on `ate.scheduler.outcome`. `cancelled` and `timeout` are outcomes for the same reason they are on `ate.router.outcome`. The hit ratio is therefore `hit / (hit + miss)`, with failures and abandoned lookups out of the denominator. **`error.type`** — set only on the `error` outcome. | Value | Meaning | |---|---| | `404`, `401`, `429`, ... | the registry rejected the request; its own HTTP status, reported verbatim | | `_OTHER` | the failure carries no status of its own | Fixes #831 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |
||
|
Lucky Abolorunke
|
bce9366b3d |
docs: add guide for running the microVM runtime locally (KVM / Lima) (#743)
### Why
The README Quickstart covers getting a local Substrate cluster running
with
the default gVisor runtime, but there's no documentation for the
micro-VM
runtime, where the friction can't be scripted away: it requires
`/dev/kvm`
(bare metal, nested virtualization, or Lima on macOS), and on Apple
Silicon
the Lima configuration and asset assembly are non-obvious. New
contributors
have to reverse-engineer the `hack/` scripts.
### What
Adds `docs/dev/microvm-local.md` — "Running the microVM runtime locally"
— a
focused guide covering only the microVM delta, based on notes from real
onboarding runs. General setup is deferred to the README Quickstart
(listed
as the guide's prerequisite) rather than duplicated:
- **Option A: Linux host with KVM** — verifying `/dev/kvm` and CPU virt
support, the rootless-Docker caveat for the KVM probe, cluster creation,
and the one-shot `run-microvm-demo-kind.sh` bring-up.
- **Option B: Apple Silicon macOS via Lima** — nested virtualization
with the
guest image pinned to Ubuntu 25.10 (until the kernel issue in the
default
image is fixed), `vzNAT` networking, and assembling the arm64 assets
inside
the Lima guest (`assemble.sh` requires a Linux host of the target arch).
- **Trying it out** — defers to the next steps the demo script prints
and
links the counter demo's micro-VM variant, instead of duplicating those
commands.
- **Troubleshooting** — symptom → root cause → fix entries actually hit
during onboarding (rootless-Docker KVM probe failures, the `aws` CLI
staging requirement until #804 lands, arm64 `virtiofsd` build deps, M1
lacking FEAT_NV2).
|
||
|
Zoe Zhao
|
4b3423c01a |
Remove the secretKeyRef env sources feature (#835)
This reverts the ActorTemplate valueFrom.secretKeyRef support added in #20 (issue #15). We don't want actors to have any access to secrets, they will be injected on the egress route instead. Removing this now so we don't need to copy secrets into substrate resources. |
||
|
AngelaandJeff Luo
|
d4f2cf1c51 |
feat: Add control plane telemetry - workerpool desired and ready workers gauges (#826)
Part of #564 (Part 4, Fixes #564) - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR ## Description This PR implements Part 4 of #564 by adding OpenTelemetry gauge instrumentation for `ate.workerpool.desired_workers` and `ate.workerpool.ready_workers` in `atecontroller`. ## Key Changes: * Added `ReadyReplicas int32` to `WorkerPoolStatus` with `+kubebuilder:printcolumn:name="Ready"` annotation in `pkg/api/v1alpha1/workerpool_types.go` and regenerated the CRD manifest. * Updated `WorkerPoolReconciler.syncStatus` to synchronize `dep.Status.ReadyReplicas` into `wp.Status.ReadyReplicas`. * Implemented `InitMetrics(meter)` registering `ate.workerpool.desired_workers` and `ate.workerpool.ready_workers` as OpenTelemetry Observable UpDownCounters (`{worker}`) with an asynchronous observer callback that samples controller-runtime's local informer cache (`r.Client.List`). * Labeled metric datapoints using centralized attributes `ateattr.WorkerPoolNamespaceKey` and `ateattr.WorkerPoolNameKey`. ## Testing * `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...` * `go test -buildvcs=false ./cmd/atenet/internal/router/...` * `make test` ## E2E Test * `./hack/create-kind-cluster.sh` * `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter` * `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...` --------- Co-authored-by: Jeff Luo <jeffluoo@google.com> |
||
|
Benjamin Elder
|
da8414bbd2 |
api: move the pause image from ActorTemplate to SandboxConfig (#848)
The pause image holds the sandbox's namespaces and runs no workload code. It is an implementation detail of the sandbox, not something actor authors pick, so it belongs with the sandbox binaries that already moved off the ActorTemplate onto the cluster-scoped SandboxConfig. It now travels with those binaries end to end: resolved from the pool's SandboxConfig, carried on ateletpb.SandboxAssets rather than WorkloadSpec, and recorded in the per-actor sandbox record so Checkpoint pins it into the snapshot manifest and Restore rebuilds the sandbox from the image the snapshot was taken with (the golden's on a DATA_ON_GOLDEN restore). A record without one is rejected outright rather than pulling an empty image. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR |