Adds a dynamic `--log-level` flag (debug, info, warn, error;
case-insensitive) to
the server binaries, backed by a `slog.LevelVar` in `serverboot`.
Defaults to
`info` — no behavior change unless the flag is set.
- **serverboot**: the LevelVar behind `InitLogger`'s handler;
`SetLogLevel(string)`
to parse and set it (empty string = unset, a documented no-op);
`LogLevel()`
returns a `slog.Leveler` for binaries that build their own handler.
- **atelet, ateapi**: new pflag, fatal on an invalid value.
- **ateom-gvisor, ateom-microvm**: flag accepted by the binary; note
their
container args are built by the WorkerPool controller with no
pass-through yet,
so wiring it is a follow-up — and that follow-up must roll out only
after this
change's ateom images are deployed (older images reject unknown flags
and would
crashloop).
- **atenet (router + dns)**: replaces two pre-existing hand-rolled
`--log-level`
parsers that silently fell back to `info` on a bad value; both now use
`serverboot.SetLogLevel` and `serverboot.InitLogger`, which also adds
`ate.dev/trace-id` correlation to atenet logs.
- **Kind manifests**: atelet runs at `--log-level=debug` on kind
(dev/CI), so e2e
suites can assert on per-item log lines; production installs keep
`info`.
Motivation: groundwork for #463 Phase 2 observability — per-item GC log
lines can
ship at Debug and be enabled per-node instead of landing at Info.
**Testing**: unit tests pin the untouched default at exactly `info`,
dynamic
raise/lower, case-insensitivity, invalid-value rejection, and the
empty-string
no-op; `go test -race` green across serverboot, atenet, atelet, ateapi.
Fixes#564 (Part 1)
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Description
This PR implements the first part of #564 by extending metric
`atenet.router.route.duration` with singleflight activation state
(`ate.router.resume`) and outcome status labels (`ate.router.outcome`).
All telemetry attribute keys and label value sets are centralized in
`internal/ateattr` to ensure consistency across components.
## Key Changes:
* Added `resumed` boolean to `ResumeActorResponse` in `ateapi.proto`
indicating whether a cold activation workflow occurred
(`!state.WasRunning`).
* Updated `workflow_resume_test.go` to assert `resumed == true` on cold
activations and `resumed == false` for already-running actors.
* Extended `classifyOutcome(err)` in Envoy `ext_proc` to classify errors
into metric labels: `ok`, `cancelled`, `timeout`, `no_capacity`,
`lock_conflict`, `not_found`, `unavailable`, `rate_limited`, and
`resume_error`.
* Added explicit handling for `codes.Unavailable` (`"unavailable"`) and
`codes.ResourceExhausted` / `StatusCode_TooManyRequests`
(`"rate_limited"`).
* Consolidated metric label names on `atenet.router.route.duration`
using standard keys from `internal/ateattr`:
* `ate.template.namespace` (`TemplateNamespaceKey`)
* `ate.template.name` (`TemplateNameKey`)
* `ate.router.outcome` (`RouterOutcomeKey`)
* `ate.router.resume` (`RouterResumeKey`)
* Updated `manifests/ate-install/kind/kustomization.yaml` to export
`atenet-router` OTLP metrics to `opentelemetry-collector`.
## Testing
* `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...`
* `go test -buildvcs=false ./cmd/atenet/internal/router/...`
* `make test`
## E2E Test
* `./hack/create-kind-cluster.sh`
* `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter`
* `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
Part of #232
1. During CreateActor, the volumes to be created get instantiated in the
Actor API in PENDING status. The Actor is still instantiated as
SUSPENDED.
However, the CreateVolume operation is moved to the beginning of
ResumeActor. This is because CreateActor is intentionally designed to
not be idempotent. The downside is that the user won't know dependent
resource creation failed until the first resume.
2. Changes volume id to use actor uid instead of atespace+actorname and
adds "substrate" prefix
3. Adds a new DELETING status that gets persisted before we start to
delete volumes.
This shifts the existing DB DELETE precondition checks to the new
DELETING status. And now you can only delete the actor in DB from
DELETING status.
4. Refactor DeleteActor to use a workflow
In the future, we will also add a workflow that will do mandatory node
cleanup of volumes before starting to delete the volumes.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
`ate.workerpool.workers` tallies by pool name, state, and sandbox class,
but a WorkerPool is namespaced, so two pools that share a name sum into
a single series. This is reachable today: the counter demo and the
autoscaled-workerpool demo both create a pool named `counter`, and with
both installed the demo HPA scales on the other pool's assigned workers.
Adds `ate.workerpool.namespace` to the tally key and attributes, and
pins it in the demo's adapter query and HPA selector.
`syncWorkerToStore` checks `pod.Status.PodIP != ""` before
`DeletionTimestamp`, so a Terminating pod reporting no IP never reaches
`markWorkerDraining` and stays `STATE_ACTIVE`, schedulable for as long
as the pod lingers. Draining resolves the stored record by name and
never reads the IP, so this swaps the two guards.
The startup reconcile runs through the same function, so a pod already
Terminating when ate-api-server restarts is now marked too.
`markWorkerDraining` already runs on every update event for a
Terminating pod, so this adds no new call pattern.
Fixes#600
Sessions are no longer a concept in Substrate; Actor is the glossary
term. This completes the "s/Session/Actor" TODO that sat at the top of
ateapi.proto, and removes the TODO.
API surface:
service SessionIdentity -> ActorIdentity
MintJWTRequest.session_id -> actor_id
MintJWTResponse.session_jwt -> actor_jwt
MintCertRequest.session_id -> actor_id
MintCertResponse.session_certificates -> actor_certificates
Go packages:
cmd/ateapi/internal/sessionidentity -> actoridentity
cmd/ateapi/internal/sessionidjwt -> actoridjwt
Flags and cluster resources:
--session-id-jwt-pool -> --actor-id-jwt-pool
--session-id-ca-pool -> --actor-id-ca-pool
Secrets, volumes and mount paths renamed to match, in both
manifests/ate-install/ate-api-server.yaml and hack/install-ate.sh
(--create-session-id-ca-pool-secret -> --create-actor-id-ca-pool-secret).
Two credential identity values change with the rename:
JWT issuer https://broker.agentic-substrate-session-id-broker.svc
-> https://broker.agentic-substrate-actor-id-broker.svc
SPIFFE ID spiffe://substrate-session.local/app/../session/..
-> spiffe://substrate-actor.local/app/../actor/..
Tokens and certificates issued before this change will not validate
against the new issuer or trust domain.
BREAKING: the gRPC wire path moves from /ateapi.SessionIdentity/* to
/ateapi.ActorIdentity/*, and the Secrets must be recreated under their
new names before the new ate-api-server rolls out.
Part of [#23](https://github.com/agent-substrate/substrate/issues/23)
The overall plan is:
1) Trap SIGTERM in the ateom, and forward SIGTERM to all application
containers.
2) Once all application containers exit, ateom will exit itself.
3) The workerpool syncer will mark workers as 'DRAINING' when a pod
deletion is triggered (pod.DeletionTimestamp) to prevent actors from
being scheduled. On the pod Deleted event, the worker and actor (if it
was assigned) will be cleaned up.
This PR implements 3), there are 2 more PRs coming which will implement
1) and 2) for each ateom (gvisor and uVM).
This PR adds missing OpenTelemetry span attributes in UpdateActor and
corresponding tests.
- [X] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Add unit tests that exercise all the validation cases and assert on the
k8s `ErrorMatcher` instead of comparing raw error strings.
Moved most validations from functional tests to unit tests, and kept one
validation for each gRPC method to validate that handlers properly map
Go errors to gRPC statuses.
The pattern of passing a tuple of atespace and actor name together
is across the whole codebase. This simplifies both callers and callees
by leting them pass a single field that bundles both.
The ActorRef is the actor specific, typed,
in-process version of the ObjectRef we have in our gRPC API.
A ClusterIP Service load-balances per TCP connection, not per RPC,
but gRPC multiplexes many RPCs over one long-lived HTTP/2 connection,
so a client that dials once (e.g. ate-controller, atenet) gets pinned to
whichever single pod that connection was routed to, for its entire
lifetime.
Make the "api" Service headless so DNS returns one A record per ready
pod instead of a VIP, and switch clients to a "dns:///" dial target
with the round_robin load balancing policy so RPCs actually spread
across every replica.
The error is only returned on an optimistic-concurrency version
mismatch. Rename it for that behavior so other retryable errors don't
get folded into it (but should introduce their own error name).
Followup for post-merge comment in #535
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Refresh the actor when `AssignWorkerStep`'s write hits a version
conflict so that the workflow won't kept retry with a stale version
actor and never succeed.
AI assisted on testing.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Make the server trap SIGTERM and initiate a graceful shutdown process.
This involves reporting the Pod as not ready, and draining all in-flight
gRPC requests so they can complete.
Part of #181
Part of [#170](https://github.com/agent-substrate/substrate/issues/170)
Establishes mutual TLS between all ate system components, and updates
the certificate plumbing it depends on.
Main changes:
1. The atenet router now verifies ate apiserver' serving certificate,
and presents its client cert to ate apiserver. Previously the connection
used `InsecureSkipVerify`.
2. AteApi server verifies atelet's serving cert.
Minor bug fixes:
1. Prevent `servicednssigner` from signing a cert with no DNS SANs. Also
updated valkey cluster's cert configuration, because it was relying on
the cert with empty DNS.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Initial part of #232.
Extends the ActorTemplate API to add support for per-actor external
volumes and adds control plane and atelet hooks following the Actor
lifecycle.
Callouts to volume operations are abstracted with a Volume interface.
Right now, only a mock volume plugin for testing has been implemented,
but we will add CSI support next.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR - Not
going to update documentation until we add CSI support.
This PR fixes a minor not found error in redis and adds corresponding
tests.
- [X] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
AssignWorkerStep starts a background goroutine to release a stale
worker assignment. It builds its context from the request ctx, but
that ctx is cancelled right after the workflow returns (releaseLock
runs on defer). If the goroutine has not called UpdateWorker yet, the
call fails with a cancelled context and the worker keeps the stale
assignment forever. Nothing else clears it while the pod is alive.
Fix: use context.Background() for the release call, same pattern used
in cmd/atenet/internal/router/resumer.go.
Tested with go test ./cmd/ateapi/internal/controlapi/...
Should found it when writing #474..
And also correct an error in `workflow_resume.go`.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Instead of requiring callers to specify a fixed deadline for the lock,
let the store return a `Lock` object that performs auto-renewal in
background.
This fixes an issue when a workflow takes longer than the hard-coded
TTL, most commonly when atelet takes long times to initialize an actor
(e.g. pulling / untaring large OCI images).
Leases are taken for 30 seconds, but renewed every 10 seconds (TTL/3).
Removing that implicit TTL-based limit means a caller that sets no
deadline of its own (e.g. kubectl-ate) could otherwise hang a
Resume/Suspend/Pause indefinitely. MaxDeadlineUnaryInterceptor adds a
10-minute ceiling on every RPC as a hang-prevention backstop.
`discoverKeysForIssuer` now parses each JWK via `parseJWK` and **skips**
keys the verifier can't use (unsupported `kty`/curve, malformed params,
or no `kid`), failing only when no usable key remains — with distinct
errors for `empty JWKS` vs `no usable keys ... (N skipped)`. Matches the
go-oidc/jose convention: skip at discovery, fail only when a token's
`kid` matches no usable key. Selection stays by `kid` and signatures are
still verified, so skipping cannot enable forgery.
## Behavior note
A key with no `kid` is now skipped instead of failing the whole set.
Safe: tokens must carry a `kid` and match by exact `kid`, so a no-`kid`
key was never selectable anyway.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR — N/A
Part of the preferred split of #433, please check that out if you need
wider context.
This PR adds `ate.workerpool.workers` metric end to end so we have
coverage for workers by pool / state / sandbox class, emitted by ateapi.
Comes with the few attribute keys it actually uses, unit tests, and an
e2e suite that drives an actor lifecycle and checks the metric (plus the
two existing platform metrics) actually lands in the collector.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The JWKS EC branch only ever returned an error, so an issuer publishing
an EC key failed verification entirely (even for its RS256 tokens).
Parse P-256/384/521 and add the package's first tests.
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This will free up the worker to which the crashed actor was assigned, so
it can be picked up by other actors.
Minor fix:
* Added functional test to make sure a crashed actor can be deleted.
Fixed flakey integration test at head.
Broken test in
https://github.com/agent-substrate/substrate/actions/runs/29795626928/job/88526261507
was because, once a old version of worker has been picked and saved in
state.worker, retrying no longer picked a new, usable worker.
Updated the implementation to only update state.worker after having
successfully picked and persisted the worker.
Fixes#369
Add a CheckPrerequisite method to the WorkflowStep interface, called by
RunWorkflow after IsComplete returns false and before Execute, so that
each workflow validates its actor state-machine edge up front while
retried (reentrant) workflows still fast-forward past completed steps.
When the worker pod disappeared before Checkpoint was called, the
suspend workflow skipped the atelet call but still let finalize promote
the reserved InProgressSnapshot URI to LatestSnapshotInfo, overwriting
the last good snapshot with one that was never written. This could break
the next resume call of this actor.
Now the dangling-worker path crashes the actor via `crashActor` too, and
crashActor clears any reserved snapshot URI so `FinalizeSuspendedStep`
won't overwrite the last good snapshot.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
* Add `args` field in addition to `command`.
* Implement logic in atelet to resolve both argv from the actor template
fields and the OCI image config (`entrypoint` and `cmd` fields).
* Update demos to not hardcode command as now we honor the image
default, which makes it not necessary.
Fixes#456
This PR sets consistent, namespaced actor-identity set on the spans we
already emit:
- `ate.atespace`, `ate.actor.id`, `ate.actor.template.{name,namespace}`,
`ate.actor.version`
- ateapi: the RPC server span for create/resume/suspend/pause/delete
We do this, so platform traces are queryable by actor, atespace, and
template. Cross-hop correlation and per-tenant/per-template filtering,
all on traces (not TSDB labels, as discussed in #174). This is a
general, workload-neutral telemetry identity plumbing. We could add
gen_ai specific stuff on top of this later.
This is a resume:
<img width="1913" height="1010" alt="image"
src="https://github.com/user-attachments/assets/876cf689-a8ec-4f37-b035-3b9b1de87d59"
/>
**Notes:**
- Left as it was on purpose: metric attribute names and the stdout log
labels (`ate.dev/*`), but we should probably do this as well. Maybe in
this PR?
- Bikeshed is welcomed on the `ate.*` spelling before merge, renaming
span attrs later is painful.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
When the worker pod is gone from the DB during pause finalization,
nodeName stays empty.
The code still wrote NodeVmsWithLocalSnapshots: []string{""}, so on the
next ResumeActor the scheduler looked for a worker with node name "",
found none, and returned "no free workers available" permanently.
Fix: only set NodeVmsWithLocalSnapshots when nodeName is non-empty.
Aligns the internal ateletpb/ateompb services with the API style guide's
actor naming (field tags unchanged, so the wire format is compatible).
internal APIs keep dedicated identity fields (not the public ObjectRef).
Replace actor_id and atespace name with `metadata` field which
holds common fields.
This commit only changes the ate-apiserver to support this field, but
intentionally skips upstream / downstream systems (such as kubectl-ate,
atenet, atelet, etc) to keep the scope manageable. This will come in subsequent PRs.
Decouple the actor resumption scheduler from Kubernetes by avoiding
direct queries to WorkerPool CRDs. Instead, cache the worker pool's
labels and sandbox class on the Worker object in Redis during pod
synchronization, and update the scheduler to evaluate worker eligibility
using these cached fields.
Should have a timeout to make the fix in #353 non-blocking.
Originated from #379
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
igooch