In github actions:
- We manage kind with a script, no need to separately install at a
possibly incorrect version
- Use go.mod to specify version, instead of repeating the version in
actions config
In the code:
- Upgrade go.mod to use latest go release (which should be reflected in
CI)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The example CoreDNS regex in `internal/dns/readme.md` was last updated
before two renames: the actor DNS suffix changed from `substrate.k8s.io`
to `substrate.ate.dev`, and the actor id format changed from UUID to a
generic DNS label. The authoritative source
(`internal/resources/actor.go` — `ActorIDRegexPattern` +
`ActorDNSSuffix`) and the corefile test
(`internal/dns/corefile_test.go`) both reflect the current shape; only
the readme drifted.
Pure documentation change; no code or test changes.
Reopening manually under my credentials, with the commit authored by me.
The original PR was drafted by an a4 fleet bot and tracked as #99
(closed because the bot account is not CLA-signed). It was reviewed and
**approved by @bowei** on commit f92de4d before being closed — linking
that PR for review continuity:
https://github.com/agent-substrate/substrate/pull/99
(First re-open attempt landed as #112 with a bot author because the
local CLI shim swapped my OAuth token for the bot's installation token
on the API call — closed and re-opened here under the correct identity.
Same branch, same single commit `1df9895a` authored as me.)
Co-authored-by: Alex Bulankou <11968377+AlexBulankou@users.noreply.github.com>
`sessionDiscovery` was removed from the `ActorTemplate` CRD when
Substrate adopted the Uniform DNS Mesh (`docs/api-guide.md` §Workload
Connectivity). Four references in
`benchmarking/workloads/manifests/{workloads,full_workloads}.yaml.tmpl`
were missed in that cleanup — strict Kubernetes API servers reject the
rendered manifests via `benchmarking/workloads/deploy.sh`.
Pure deletion; no functional change to the rendered ActorTemplates.
## Symptom
When `<namespace>:<pod-name>` exceeds 107 bytes, `ateom-gvisor` exits
with `bind: invalid argument` and the controller surfaces a generic
`DeadlineExceeded`.
## Cause
Linux limits unix-socket paths to 107 bytes (`sockaddr_un.sun_path`). A
pod's ReplicaSet hash can jiggle between 8 and 10 chars across rollouts,
so a demo that fits at 8 silently breaks at 10 with no clear error.
## Fix
Add `ValidateAteomSocketPath` (with unit tests) and call it from
`ateom-gvisor` before the bind, so the operator sees the real cause in
the pod log instead of a kernel EINVAL.
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
Today, if a worker pod that hosts an active actor is forcibly destroyed,
substrate does not migrate the actor. The actor's AteomPodName still
points at the dead pod and the router times out forwarding to a dead IP.
WorkerPoolSyncer's pod-informer DeleteFunc and soft-delete branch now
check whether the worker being removed is bound to an actor, and if so,
reset that actor to `STATUS_SUSPENDED` before the worker row goes away.
The new helper releaseActorOnDeadWorker reads the actor, only acts if
the actor still claims this worker (a concurrent SuspendActor or
DeleteActor that already advanced state is respected), clears the
pod-binding fields and InProgressSnapshot, preserves LastSnapshot, and
writes via version-checked UpdateActor. On ErrPersistenceRetry we drop
the attempt and let the next informer event retry — no separate lock, no
in-handler retry budget.
`STATUS_SUSPENDED` is reused rather than introducing a new state. The
post-orphan invariant is identical and reusing keeps the proto, printer,
dialer, and router contracts unchanged. The next request through atenet
triggers an implicit resume; findFreeWorker picks any free worker in the
pool; LastSnapshot is restored.
Test plan: `TestSyncer_DeleteBoundWorker_ClearsActor` covers RUNNING
actor → pod delete → SUSPENDED with cleared bind fields,
InProgressSnapshot dropped, LastSnapshot preserved. All existing tests
in `cmd/ateapi/internal/controlapi/` still pass.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Co-authored-by: Julian Gutierrez Oschmann <juliangut@google.com>
## Symptom
With `KUBECTL_CONTEXT` set in `.ate-dev-env.sh`, `./hack/install-ate.sh
--deploy-ate-system` aborts mid-deploy at the first `ko resolve`:
Error: unknown command "--context=..." for "ko resolve"
CRDs, the ate-system namespace, and podcertificate-controller have
already been applied by then, leaving the cluster half-installed.
## Cause
`run_ko` unconditionally appends `-- --context=$KUBECTL_CONTEXT` to
every ko invocation. Only `ko apply`, `ko create`, `ko delete`, `ko run`
have a kubectl phase that consumes args after `--`. Other subcommands
(`resolve`, `build`, `deps`, `login`, ...) parse it as a flag and reject
it.
## Fix
Only forward `--context` for the ko subcommands that actually delegate
to kubectl. `run_kubectl` already passes the context to direct kubectl
invocations, so the chained `ko resolve | run_kubectl apply -f -` flow
still targets the right cluster.
This PR refactors `ateom-gvisor` and `kubectl-ate` to adopt a
flat-merged logging architecture, eliminating server-side and
client-side field guessing:
1. **Cloud Logging**: `ateom-gvisor` parses actor JSON, injects actor
labels/time, and outputs the flat map directly. GKE Cloud Logging
built-in rules parse native severity/message fields automatically.
2. **CLI Logging**: Removed the `--raw` flag. `kubectl ate logs` strips
Substrate metadata and outputs clean, consistent JSON lines with the
`time` field prepended first.
<img width="3235" height="306" alt="image"
src="https://github.com/user-attachments/assets/4a6ca291-db8d-45dc-b840-43efa74739f0"
/>
3. **Documentation**: Updated `docs/observability.md`
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Three related cleanups in cmd/{ateapi,atelet,ateom-gvisor}/main.go:
1. atelet Run/Checkpoint/Restore handlers shared three blocks of
cut-and-paste (fetchRunsc + MkdirAll, errgroup over pause + application
containers calling prepareOCIDirectory, dial-ateom + NewAteomClient, the
ateletpb -> ateompb WorkloadSpec projection). Factored into
fetchRunscAndPrep, prepareOCIBundles, dialAteom, buildAteomWorkloadSpec,
plus a uploadIfExists for the optional pages.img / pages_meta.img upload
in Checkpoint.
2. ateapi main() was a 242-line wall of inlined startup steps;
decomposed into named helpers (loadFlagsFromEnv, logFlagValues,
connectRedis with buildRedisTLSConfig + pingRedisWithRetries,
newKubeClients, buildServerCreds) so the top-level flow reads as
orchestration.
3. ateapi, atelet, and ateom-gvisor each carried their own ~30-line
initTracing + ~35-line initMetrics + slog/metrics-server boilerplate
that differed only in ServiceName, sampler choice, and whether the OTLP
exporter was wired. Lifted to a new internal/serverboot package exposing
InitLogger, InitTracing (with TracingOptions.NoExporter for ateom-gvisor
which loses egress once eth0 moves into the gvisor netns), InitMetrics,
Fatal, ShutdownProvider, and StartMetricsServer.
No behaviour change. Verified with go vet, GOOS=linux go build, and go
test ./cmd/ateapi/internal/...
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Document the rationale behind each directory and guidelines on how to
decide where to put new code in a dedicated markdown file.
Update AGENTS.md with a summary and a reference to the new markdown
file.
save some time in CI
follow-up https://github.com/agent-substrate/substrate/pull/63
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The existing trigger is pull_request only, so when a PR merges no CI
runs against the resulting main-branch commit. A bad squash, a
post-rebase test failure, or a flake-masked regression isn't surfaced
until the next PR opens, at which point bisecting "is it my change or
main?" wastes contributor time.
Add a push trigger restricted to main so the same run-tests and e2e-test
jobs run against the merged commit. No new jobs, no new permissions, no
change to PR behavior.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Now the worker pool can be scaled via regular `kubectl scale` commands.
Use server-side-apply to apply changes.
Propagate the effective number of replicas from the status of the
managed deployment.
Fixes https://github.com/agent-substrate/substrate/issues/48
ateom-gvisor moves the pod's eth0 interface into the actor's interior
netns before invoking runsc run/restore. The current code assumes eth0
is in the pod netns at the start of every RunWorkload/RestoreWorkload.
That assumption breaks if a previous call failed mid-flight: eth0 is
left stranded in the interior netns, the next call errors with
"eth0: Link not found", and the worker pod is stuck until restart.
Add ensureEth0InPodNetns that moves eth0 back to the pod netns if it
is in the interior, and is a no-op otherwise. Call it at the top of
both entry points to recover from a previously-aborted call, and via
a deferred handler after the eth0-into-interior move to roll back on
errors in the current call. The success path is unchanged.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Debian 11 is EOL.
Also ensure images are built for also AArch 64
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
Co-authored-by: Julian Gutierrez Oschmann <juliangut@google.com>
`make build-demos` invoked `ko build ./cmd/demos/counter`, but the demo
sources live at `./demos/counter`. The target failed loudly when run,
but nothing in CI exercises it — `hack/install-ate-kind.sh
--deploy-demo-counter` (the path CI actually uses) goes through
`hack/install-demo-counter.sh` and doesn't touch the Makefile, so the
breakage went unnoticed.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Adds golangci-lint v2.12.2 using the per-tool-module pattern under
hack/tools/golangci-lint/. The verifier at hack/verify/golangci-lint.sh
is auto-discovered by hack/verify-all.sh, so the existing CI verify
step picks it up with no workflow YAML change.
Ships .golangci.yaml with the standard linter set, a make lint
target, and a verify wrapper that pins GOOS=linux so ateom-gvisor's
netlink references typecheck on macOS the same way they do on CI.
Adjustments to keep the first run clean: two dead-code deletions
((*runsc).cmdList, MemoryPullCache.gcsClient), two SA9003 empty
branches replaced with slog.Error / t.Logf, one S1008 return-bool
simplification, and inline //nolint for one intentional best-effort
ReleaseLock plus four audit-tracked SA1019 deprecations.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
We are just using tools modules (plural) to manage installing go
binaries with binned versions easily.
We are NOT trying to patch transitive dependencies in those tools,
versus just updating those tools themselves.
Add a helper script to update them as intended, and add and AGENTS.md to
tell agents to use the script.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR adds live-streaming log support (`--follow/-f`) to `kubectl-ate
logs actors`. Unlike static container logs, this implementation is
actor-aware and resilient to virtual actor suspensions and migrations:
Suspend Lifecycle:
<img width="3700" height="1856" alt="image"
src="https://github.com/user-attachments/assets/38568108-8229-40b0-afc7-d1d5ddc817e1"
/>
Resume Lifecycle:
<img width="3696" height="1530" alt="image"
src="https://github.com/user-attachments/assets/cf934608-62fe-4fb8-bd2c-5e6e7a0af776"
/>
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Fixes#55
I was following the demo but ran into the issues described in #55. The
command is failing because the curl command runs before the
port-forwarding is ready.
`install-ate-kind.sh --deploy-ate-system` installs the kind overlay
(rustfs and the `otel-system` namespace) via a kustomize build of
`manifests/ate-install/kind`, but the matching teardown only removed
`kind/rustfs.yaml`, and only when `ATE_INSTALL_RUSTFS=true` was set in
the environment.
Teardown now deletes the same kustomize set whenever
`ATE_INSTALL_KIND=true` (already exported by `install-ate-kind.sh`). The
GKE path is unchanged; `ATE_INSTALL_RUSTFS` is unused after this change.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
## Symptom
`atelet`'s `Checkpoint` RPC returns `err: nil` even when the GCS upload
it performs silently failed. The destination bucket stays empty (no
`checkpoint.img.zstd`, no `pages.img.zstd`), and every subsequent actor
`Resume` then fails with `storage: object doesn't exist`, which looks
like a Resume bug but is actually a checkpoint-upload bug.
## Root cause
[`internal/ategcs/ategcs.go:51`](https://github.com/agent-substrate/substrate/blob/main/internal/ategcs/ategcs.go#L51)
— `gcsClient.PutObject`:
```go
wc := g.client.Bucket(bucket).Object(object).NewWriter(ctx)
defer wc.Close()
_, err := io.Copy(wc, reader)
return err
```
For the GCS Go client, `io.Copy` only writes into a client-side buffer;
the actual HTTP PUT (and any API-level error like 403 from an OAuth
scope mismatch) is returned from `Writer.Close()`. The `defer
wc.Close()` here discards exactly that error, so PutObject returns `nil`
even when the upload was rejected.
## Fix
Return the error from `wc.Close()`. On `io.Copy` failure still
best-effort-close the writer but report the copy error, matching the
existing `slog.InfoContext("Dropped error ...")` pattern used elsewhere
in this file. Minimal, localised change; no API surface change.
## How I noticed this
Evaluating Substrate on a GKE Standard 1.35.3 cluster (with
`--enable-kubernetes-unstable-apis=certificates.k8s.io/v1beta1/{podcertificaterequests,clustertrustbundles}`)
where the node pool's OAuth scope was `devstorage.read_only`. That's my
misconfig — pods can't write to GCS — but the user-visible symptom was
confusing: `atelet.Checkpoint` reported success in ~282ms,
`gs://ate-snapshots-.../` was empty, and the first user-facing error
came from a completely unrelated actor's Resume call. Surfacing the
upload error at the checkpoint call site makes the misconfig immediately
diagnosable.
## Verification
- `GOOS=linux go build ./...` passes locally.
- No new dependencies, no new logging frameworks, single file touched.
Leaving as **draft** for maintainer eyes before marking ready — happy to
add a unit test against a fake `ObjectStorage` if you'd like, but didn't
want to bloat the diff before getting a signal on the approach.
We've already been managing other go binaries with tools modules.
Doing this limits the number of setup steps needed, and allows our
scripts to make assumptions about the kind version.
`kind` has minimal dependencies and should compile very quickly.
Label the registry as created by our script, so we know if we should
delete it.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Tim Hockin