Part of #1266
This is a draft of the core API + data store changes.
It's still a large PR, apologies.
The "as rows" commit could be split out, but this takes it to ~all of
the breaking changes we can't hide behind updating internals.
Same for the claimlock, but in both cases it seems these are worth
understanding when considering the API shape.
They're loadbearing for performance once we actually have multi-actor
workers.
This PR moves sandbox config selection from the WorkerPool to the
ActorTemplate.
The existing behavior is preserved while we are designing the upgrade:
sandbox config still cannot be updated once set (ActorTemplates are
create-only and `sandbox_config` is immutable).
For now the ActorTemplate still *requires* `sandbox_config.config_name`
— there is no resolution of the cluster default (`spec.default`). This
is temporary while we figure out the defaulting design.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The Create Cluster warning explained the beta-API creation-time
constraint but not which GKE versions work. Clarifies the two supported
configurations: **GKE 1.36 with the beta APIs enabled at cluster
creation**, or **GKE 1.37+** where `certificates.k8s.io/v1beta1` is
served by default (no beta enablement needed). Versions below 1.36 are
unsupported.
- [x] Tests pass (docs only)
- [x] Appropriate changes to documentation are included in the PR
---
Also updates `hack/ate-dev-env.sh.example`: it pinned
`1.35.5-gke.1163012` — below the supported floor and no longer offered,
so copying it verbatim failed cluster creation. Now pins the newest
offered 1.36 patch (`1.36.3-gke.1767000`) with a comment on how to pick
a fresh one when the pin ages out.
**Validated**: `setup-gcp create cluster` at `1.36.3-gke.1767000`
(us-west1-c) succeeded, and both `certificates.k8s.io/v1beta1` APIs
(`clustertrustbundles`, `podcertificaterequests`) are served on the
resulting cluster — `kubectl get clustertrustbundles` returns the
kube-apiserver-serving bundle. Throwaway cluster deleted after
verification.
---------
Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>
### Description
This PR introduces native, secure support for using Cloud SQL as the
PostgreSQL store backend for `ate-api-server`.
To ensure the highest level of security and ease of use in GCP
environments, this integration leverages the Cloud SQL Auth Proxy
sidecar with automatic IAM database authentication. This means transport
security (TLS 1.3 tunnel) is handled automatically, and database
sessions are authenticated using Workload Identity via short-lived OAuth
tokens, completely eliminating the need for database passwords.
### Key Changes
* **Cloud SQL Auth Proxy Sidecar:** Added
`manifests/ate-install/cloudsql-proxy-patch.yaml` to patch the sidecar
into the `ate-api-server` deployment when a Cloud SQL instance is
configured.
* **Automated Provisioning:** Extended `tools/setup-gcp` with a new
`cloudsql` command. This handles the idempotent creation of the Cloud
SQL instance, Google Service Accounts (GSA), IAM bindings, and Workload
Identity bindings.
* **Installation Script Updates:** Updated `hack/install-ate.sh` to
parse new environment variables (e.g.,
`ATE_API_POSTGRES_CLOUDSQL_INSTANCE`, `ATE_API_POSTGRES_CLOUDSQL_GSA`)
and correctly synthesize the passwordless DSN and ConfigMaps for the
proxy.
* **Security & Documentation:**
* Added extensive documentation in `tools/setup-gcp/cloud-sql.md`
covering provisioning, schema privileges, deployment, and database
scaling.
* Updated `docs/threat-model.md` to reflect the new Cloud SQL egress
flows and Auth Proxy tunnel mechanics.
* **Dependencies:** Vendored required Google API clients (`sqladmin/v1`,
`servicenetworking/v1`, `iam/v1`) for the GCP setup tool.
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
Three fixes from user feedback on the substrate-gke installer experience, consolidated per reviewer preference (replaces #1405, #1406, #1407):
**1. setup-gcp: enable `artifactregistry.googleapis.com` in bootstrap.** Bootstrap grants `roles/artifactregistry.reader` to the GKE node service account and the atelet workload-identity principal, and ko pushes the control-plane images through gcr.io's Artifact Registry backing — but `enableRequiredAPIs` never enabled the API, so a fresh project failed at image push/pull instead of step 1.
**2. teardown.sh: cover everything bootstrap creates, without a dev-env file.**
- Revoke atelet's project-level bindings (`roles/storage.objectAdmin`, `roles/artifactregistry.reader`), which `grant_atelet_permissions` adds but nothing removed.
- Delete the three Substrate monitoring dashboards, matched by the display names in `tools/setup-gcp/dashboards/`.
- Accept configuration from the environment when `.ate-dev-env.sh` is absent (missing variables are named specifically), so installers and one-liners can drive the script.
- Drop the kubectl cluster-admin precheck: every step talks to GCP, not the cluster, and it blocked tearing down a cluster that was already gone.
- `--all` now runs in the true reverse of bootstrap's setup order.
**3. commands.md: the `hack/install-demo-*.sh` scripts still exist.** The closing paragraph claimed they are gone, but `hack/install-ate.sh` still sources them and generates its `--deploy-demo-NAME` / `--delete-demo-NAME` flags from them.
> [!WARNING]
> Recreate PostgreSQL databases from earlier development builds.
## Summary
This change replaces startup schema setup with embedded, versioned SQL
migrations.
`ateapi` uses Goose to apply migrations before readiness. Goose stores
one ledger record for each applied migration.
Goose runs each migration and inserts its ledger record in one
PostgreSQL transaction.
Closes#901.
Based on this design:
https://docs.google.com/document/d/13ixDKRoAIFXeLxS-_1nikNcAy76ca8m0eVgobfib93E/edit?usp=sharing
## Migration behavior
`ateapi` gets a session advisory lock for the configured schema before
it applies pending migrations.
One replica applies migrations while other replicas wait. Goose reads
the ledger again after it gets the lock.
If a migration fails, PostgreSQL rolls back its SQL and ledger record.
Earlier successful migrations remain applied and recorded.
Kubernetes restarts the failed replica. The next startup resumes from
the first migration without a ledger record.
## Changes
- Add Goose and a per-migration ledger.
- Replace the initial up and down files with one transactional, up-only
migration.
- Keep migration 1 aligned with the current schema, including actor
egress policy storage.
- Remove existence guards and explicit transaction statements from the
baseline migration.
- Apply all pending migrations before `ateapi` becomes ready.
- Serialize each migration run with a PostgreSQL session advisory lock.
- Start without changes when the database schema is current or ahead.
- Reject application tables that do not have a migration ledger.
- Log the starting, current, and latest versions.
- Log the applied migration count and duration.
- Retry only initial database connection failures.
- Return schema and migration errors without a retry.
- Add `--postgres-schema` and `ATE_API_POSTGRES_SCHEMA`.
- Use `public` as the default PostgreSQL schema.
- Use the configured schema for the main and watch pools.
- Restrict outbox partition maintenance to the configured schema.
- Let the installer use an external PostgreSQL database.
- Add the migration design and recovery policy to the repository.
## Migration file policy
Migration files use sequential versions and contain exactly one Goose
`Up` section.
CI rejects down migrations, nontransactional migrations, environment
substitution, explicit transaction control, and `IF NOT EXISTS` guards.
Before the first stable v1 release, developers can change or squash
migrations. Developers must recreate databases after migration history
changes.
After that release, CI rejects changes or deletions against the latest
stable release tag that contains migrations.
Goose does not store migration checksums. The binary embeds each
migration file, and release-tag checks protect released migration
history.
## Compatibility
No release includes PostgreSQL support. The `v0.0.0` release predates
the PostgreSQL backend.
Users must recreate databases from earlier PostgreSQL development
builds.
Every committed migration prefix must work with the current and previous
`ateapi` releases. This rule supports rolling upgrades and temporary
binary rollback.
A binary rollback does not roll back the database schema.
## Testing
Tests cover:
- Fresh database migration.
- Concurrent startup.
- Advisory lock waits.
- Current and ahead database schemas.
- Rejection of application tables without a migration ledger.
- Atomic rollback of a failed migration.
- Retention of earlier successful migrations.
- Resume from the failed migration after restart.
- Configured schema isolation.
- Outbox partition isolation.
- Migration file policy checks.
- Stable release migration immutability.
Fixes#368 . Deletes ActorTemplate CRD and any references to it.
Note about atenet router: It had a k8sclient controller that monitors
ActorTemplate, but the results were not used. Deleted as well.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Follow-up to the review comments on #1288 (merged before they could land
there).
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Changes:
-
`hack/third_party/csi-driver-host-path/deploy/csi-hostpath-testing.yaml`:
same fix as #1288 — the inline `watched_directory` was a no-op, so the
serving cert and trust bundle move to filesystem SDS (SPIFFE SAN pin
kept inline via `combined_validation_context`). SDS requires a node id
and this sidecar passes no `--service-node`, so the bootstrap gains a
`node:` stanza; the `docs/csi-deployment.md` example had the same gap
and gets one too.
- `overlay_test.go` now asserts `watched_directory` appears nowhere in
the emitted cluster block or the patched `envoy.yaml` (comments
excluded) and is present in each SDS resource file, per review.
- Corrected the test comment claiming the sdsmint e2e suite is skipped
in CI: the sdsmint variant is deployed in the MITM lanes; only the
`--experimental-additional-egress-extproc-service` injection has no e2e
coverage.
Verification: `go test ./cmd/ate-setup/...` passes; `envoy --mode
validate` (v1.34) passes on the reworked csi-hostpath bootstrap with
dummy PEMs mounted at the referenced paths.
This PR is very large since it updates all existing demos and benchmark
workloads to use the new ActorTemplate substrate proto.
Please use the "Commits" tab to review individual commits.
Verifications done:
* Used this script: gpaste/5143788763348992 to verify that the change
from CRD -> proto are equivalent.
* The e2e tests are using the new susbtrate resources.
* Picked the parking demo to run e2e manually: gpaste/6193361380311040
A rolling upgrade needs two substrate versions running in one cluster,
split by node: `atelet` and `ateom` speak a node-local protocol with no
cross-version guarantees, so everything on a node has to come from one
build. Today nothing records which build a node runs, and the `atelet`
DaemonSet is a single fixed-name object, so a second version can only
replace the first in place, instead of our porposed node by node rolling
([design](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A)).
This PR is the basis for upgrade. Nodes and the `atelet` DaemonSet get
keyed by an `ate.dev/substrate-version` label whose value is the build
version stamped into the binaries.
### One derivation for the version label
`internal/versionlabel` turns the build version into its two forms in
one place:
| form | grammar | used for |
|---|---|---|
| label value | k8s label value | node labels, DaemonSet labels,
nodeSelectors |
| name suffix | DNS-1123 | the DaemonSet name `atelet-<suffix>` |
A version that is not a valid label value is rejected, because the label
has to match what the `ldflags` stamp put into the binaries through
MAKEFILE and `ko`. It's also exposed shell, so the install script can
use it.
### A DaemonSet per version
`atelet-<suffix>` as the name, the version label on metadata, selector,
and pod template, and a pod nodeSelector on the same label. Two versions
run side by side on disjoint old/new node sets.
### The install becomes version-aware
- `install-ate.sh` reads the version from the same `make ldflags` output
it stamps the binaries with, then fills the manifest placeholders.
- It labels the nodes that exist at install time, nodes come after it
carry no version label yet (so add notes in README). Re-running the
install is idempotent, this is the basis for upgrade runbook later.
- The `README` documents the invariant, how to read the installed
version off the DaemonSet, and that a node added later hosts no workers
until it carries the label. On GKE the node pool label is the birth
default for new nodes - added in `tools/setup-gcp/README.md`.
### Follow-ups
- `ate-setup` need same modification
- minimize the affect for daily developer (pin the `VERSION` as
`{USER}-dev` instead of from git
- the manual rolling-upgrade runbook.
Fix#1270
Ref [design
doc](https://docs.google.com/document/d/1JduAyGZFyqdNp4UhKiv0EN-is3iW5tf5BwGTWbt2ouI/edit?usp=sharing&resourcekey=0-MXG12QCkleIxhY7cOB7g6A)
This PR is large, I grouped changes to the following commits:
- 1d5d8f06: moves consumers off the CRD path: demos, ate-setup scripts
and the e2e suites address templates by the actor_template ref.
- e502cf60: Ate API changes: removes actor_template_namespace,
actor_template_name from Actor, ActorAssignment and ActorSnapshot in the
public API, drops the CRD conversion fallback.
- 8f6c1adb: atecontroller: removes the ActorTemplate CRD controller.
Once this PR is submitted, existing demos that still uses CRD will stop
working. Created https://github.com/agent-substrate/substrate/pull/1355
to update existing demos.
Guarding on `[ ! -d "$VENV_DIR" ]` treats an interrupted create (no
bin/activate) and an interpreter upgrade (dangling bin/python3) as a
good venv, so the script dies in `source venv/bin/activate` and the only
fix is knowing to delete the directory. Probe that the venv runs, and
rebuild with --clear to relink the interpreter.
Also install requirements unconditionally, which is cheap. The license
verifier skipped the install whenever the venv already existed, so it
passed without ever seeing a newly added dependency.
Fixes failure encountered by @ahmedtd. Not filing an issue because this
is pretty trivial.
AI-assisted.
Installing the control plane pulls roughly 570MB of third-party images
(postgres, prometheus, the otel collector, rustfs, envoy, jaeger) onto a
single node.
kubelet serializes image pulls by default, so they queue behind one
another and whichever workload lands at the back of the queue can miss
its readiness deadline.
e2e has failed repeatedly with postgres still in PodInitializing after
60s, its image not yet pulled, while everything else was already
running.
Sample failure:
https://github.com/agent-substrate/substrate/actions/runs/33258860897/job/99117220741
Note: This is after PR merge, and it's clearly unrelated to the PR.
AI-assisted.
ate-controller creates a WorkerPool's Deployment, so it does not exist
yet when the apply returns.
`kubectl rollout status` reads the object before it starts watching and
errors on a missing one instead of waiting, so whether these waits
worked at all came down to beating the controller by a round trip.
e2e lost that race by 178ms, failing the deploy with NotFound while the
pool's pods were already being created.
Fixes yet another flake I observed here:
https://github.com/agent-substrate/substrate/actions/runs/33275880827/job/99162382080?pr=1283
Part of #207, following the plan discussed there with @juli4n and
@kannon92: plumbing plus the non invasive fixes first, with the existing
invasive findings excluded so new APIs do not regress (as @kannon92
suggested in
https://github.com/agent-substrate/substrate/issues/207#issuecomment-4921051580).
The linter itself was originally suggested by @BenTheElder in #188.
Plumbing:
- hack/tools/kube-api-linter module with the
golangci-lint-kube-api-linter tool, same pattern as the other tools.
- .golangci-kal.yaml config. The pre-existing findings from rules that
need Go API changes (nomaps, nonpointerstructs, nophase, optionalfields,
requiredfields, ssatags) are excluded rather than disabled, and each
exclusion rule names the fields it excuses, so the rules still apply to
new API files and to new fields on the existing types. Fixing them on
the existing types is the follow up in #207.
- hack/verify/kube-api-linter.sh runs it against ./pkg/api/... and is
picked up by hack/verify-all.sh in CI.
Fixes in pkg/api/v1alpha1 (doc and marker only, no Go API change):
- commentstart: field godocs now start with the serialized field name.
- conditions: Conditions moved to the first position in
ActorTemplateStatus and got the listType, listMapKey and patch markers.
- defaultorrequired: SandboxConfigSpec.SandboxClass had both a default
and required; now optional with omitempty, matching the same field on
WorkerPoolSpec and ActorTemplateSpec. The ValidatingAdmissionPolicy in
sandboxconfig-validation.yaml is unaffected since CRD defaulting runs
before admission, so spec.sandboxClass is always set by then.
- optionalorrequired: missing +optional markers added on
ActorTemplateStatus fields.
- defaults: configured preferredDefaultMarker to kubebuilder:default
since CRDs are generated with controller-gen.
Generated CRDs regenerated. Structural schema changes are only the
conditions listType/listMapKey and sandboxClass no longer in the
required list (it is defaulted, so behavior is the same); the rest is
description text.
Note: my earlier count of 50 issues in #207 was capped by the default
golangci-lint issue limit. With the cap removed the real total is 86;
the extra ones are requiredfields (20) and ssatags (4), both in the
excluded set above.
Test plan
- hack/verify/kube-api-linter.sh passes (0 issues).
- Exclusions verified both ways: dropping them brings the pre-existing
findings back, and a scratch field added to WorkerPoolSpec is still
reported (optionalfields), which the earlier per-file exclusion
swallowed.
- TestSandboxConfigValidation passes against the regenerated CRD.
- go build ./... and go test ./pkg/api/... ./cmd/atecontroller/... pass.
- gofmt, shellcheck and the regular golangci-lint on pkg/api are clean.
Switches all e2e tests that uses counter demo to validate the new
substrate proto ActorTemplate.
Made some changes to make the e2e test pass:
* cmd/ateapi/internal/controlapi/template_reconciler.go - golden actors
still need the golden atespace, because the logic in suspend actor
relies on it to always take FULL snapshot regardless of the config.
(filed https://github.com/agent-substrate/substrate/issues/1299)
* internal/ateattr/ateattr.go - Added placeholder label for now, will
fix in a follow up PR to wire the metric reporting when using new
ActorTemplate substrate object.
Envoy honors watched_directory only on SDS-delivered secrets; on the
egress gateway's inline file-based certs it was silently ignored, so
kubelet's projected-certificate rotation never reached Envoy and the
gateway eventually served an expired certificate.
Move the serving cert (both egress manifests) and the spliced extproc
cluster's client cert and trust bundle (Go and shell emitters) to
filesystem SDS, with the flag-derived SAN pin kept inline via a
combined validation context. Fix the csi-deployment doc example that
showed the same broken pattern, and add tests for the emitters.
Flesh out the agentgateway implementation as a router and egress PEP.
This also changes the atunnel CONNECT port to do 8443. Most of this is
agentgateway contained, but the one other system we touch is localca;
agentgateway's MITM support requires the TLS cert and key in the secret
instead of the JSON pool (we don't rely on sdsmint).
- [X] Tests pass
- [X] Appropriate changes to documentation are included in the PR
---------
Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
This is the companion to #496
In order to grant access to `/dev/kvm` we have to either:
- use a device plugin
- use a DRA driver
- use an NRI plugin
An NRI plugin is highly privileged in it's own right for all pods on the
host and is difficult to ship portably at the moment.
DRA is promising, but not enough functionality is GA yet at our current
1.35+ target.
Device plugin fits reasonably well. We do wind up publishing an
~arbitrarily high limit, which has some cost in kubelet memory, but
otherwise is relatively clean.
This approach is what kata uses currently. Their device plugin is not
available unbundled, and we anyhow have a per-node daemonset.
atlet is taught to sniff if /dev/kvm appears on the host at all, so we
can also stop using the manually labeled nodes for microVM class and
instead schedule to the KVM + TUN devices on nodes that advertise them.
Later we can migrate to device plugin by using
I implemented that already, but I don't think it's worth merging at the
moment. We would want consumable capacity to be on by default. We can
migrate later without changing the pod spec by using
`extendedResourceName`.
https://github.com/agent-substrate/substrate/compare/main...BenTheElder:substrate:ateom-microvm-dra
NOTE: I confirmed with upstream that device plugin is not going anywhere
despite being "v1beta1", it's GA in all but name. It won't receive new
features but we don't really need anyhow. We'll move to DRA down the
line.
---
By doing this, we can drop `privileged: true` from the uVM ateom pods.
We can also drop the `ate.dev/sandboxClass` node label hacks, reducing
friction to deploy.
Pick up the latest kind release for local dev clusters.
Includes fix relevant to
https://github.com/agent-substrate/substrate/issues/1133 (kindnetd
memory)
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This commit fleshes out localca's rotation support by changing the
interface so that signing operations should always go through a pool.
All existing in-tree callers are converted over to this pattern. This
ensures that signing can properly continue as a CA pool is rotated,
without requiring any process restarts.
The pool is periodically reloaded from disk in the course of signing
operations, with the loaded data being cached for up to a minute.
Some code for storing the CA state using PEM files was removed because
it seemed to be dead.
Follow-on changes will give localjwt a similar treatment, and add
administrative commands for running rotations on pools.
The image reference is pinned by digest, so the tag is discarded on
pull. Building it from ${WEAVER_VERSION} suggested that bumping that one
variable moved the image, when a new digest is needed as well.
Write the tag literally instead, next to the digest, as
`hack/third_party/kubernetes/verify-shellcheck.sh` does.
A follow-up of
https://github.com/agent-substrate/substrate/pull/1097#discussion_r3857057767
## Summary
This change updates the boilerplate verification to accept either of the
approved copyright attribution forms:
- `Copyright YYYY Google LLC`
- `Copyright YYYY The Agent Substrate Authors`
## Why
The repo now allows either owner attribution, and the boilerplate
checker should enforce that both valid forms pass without rejecting the
other.
I retained the Google LLC so that we didn't have to re-copyright all the
existing files.
## Verification
- Ran the focused matcher assertions for both allowed strings and an
invalid holder.
- Ran `hack/verify/boilerplate.sh` successfully.
This mostly eliminates the need to hand-write validation code.
This PR is a long series of commits which add DV for most of Actor and
all of Atespace.
Here is a map to the commits:
* The first few take a dep on a new Kubernetes tag, import the code into
third_party, and apply a single patch. Because we have different Go
modules for tools, I had to do it twice. When that patch lands, we can
revert these commits, but that won't be until the 1.38 cycle in a few
months.
* The next commits slowly add DV support, so a human can review each of
them in a reasonable amount of time. The emphasis is on great test
cases.
* I made a bad choice early on as to where to generate the code into, so
I moved it. Rebasing on that was exceedingly hard, so I left it as a
move.
* This required changing update/go-generate -> update/codegen -- we need
to get the ordering of tools right, which `go generate` does not
guarantee.
* Then I added a "middle" layer called "ServiceImpl" between the RPC and
storage layers. This allows things like workflow to call the same
business logic as the RPC layer, including validation. Lots of test
fixes.
* Then I finished the Create() and Update() paths for Actor. Those
represent the "right" (or closest to) way to implement resources, and
tests for validation.
I strongly encourage reviewers to read it commit-by-commit. Rebasing
this is VERY tedious, so the sooner it lands or dies completely, the
better. Then we can start converting the rest.
`./hack/run-tool.sh validation-gen --docs` will produce some docs on the
tool and the available tags.
@laoj2 @juli4n @EItanya @HavenXia
@lalitc375 @yongruilin @jpbetz FYI
`docs/observability.md` was the only description of the metrics of
Substrate, and nothing compared that text with the code. It listed 13
instruments; the code sends 21. The request-parking and actor
resource-usage instruments were
missing entirely.
This change adds an [OpenTelemetry
Weaver](https://github.com/open-telemetry/weaver) semantic convention
registry.
### What is in it
| Path | Content |
|---|---|
| `docs/metrics/registry/manifest.yaml` | The Weaver manifest: name and
schema URL. |
| `docs/metrics/registry/metrics.yaml` | 21 instruments and 12 attribute
groups: every label, its permitted values, and its buckets. |
| `docs/metrics/substrate.yaml` | The rules Weaver cannot express. |
| `hack/verify/verify-metrics.sh` | The check. |
Fixes#1093
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
install-microvm-deps.sh decided whether to run assemble.sh by testing
only whether the five asset filenames existed in $OUT. Those names don't
change when a version pin moves, so an asset dir built before a bump
looks complete while holding the old bytes.
## Summary
A follow up to #640 where we introduced PostgreSQL as an alternative
storage backend, selected conditionally in ateapi.
- Deleted ateredis, its tests, and its dependencies
- Removed Redis backend selection and configuration so ateapi always
connects to Postgres
- Replaced Valkey resources with Postgres in the standard and Kind
deployment paths and simplified install script
- Replaced miniredis fixtures with isolated Postgres testcontainers and
added centralized helpers for seeding resources
- Renamed Redis-specific debug flush command to backend-neutral
`debug-clear-store` in CLI
- Updated comments and docs where applicable
## Benchmarking
Extensive benchmarking have been performed to evaluate Redis vs
Postgres, and results can be found in these two documents:
-
https://docs.google.com/document/d/10K0wB6aTeFkJCL4HN3NbLJCdFGoLYdhIcqkFnqFHkKc/edit?usp=sharing
-
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Secures network-based gRPC connections between the ate-api control plane
and CSI driver controller services using mutual TLS (mTLS) backed by
Substrate's Pod Identity.
Requests from ate-api to CSI driver controllers occur over the network,
as configured via the CSIDriverConfig custom resource. This PR extends
the CSIDriverConfig CRD to include a tls configuration section, which
leverages
Substrate's Pod Identity certificate infrastructure (managed by
podcertcontroller) to secure these
TCP connections using mTLS. Because standard CSI drivers typically
communicate over Unix domain sockets, a proxy sidecar (e.g., envoy) is
added to the CSI controller deployment to terminate TLS and forward
requests to the local CSI socket.
#232
> It's a good idea to open an issue first for discussion.
- [X ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
Run the networking suite a second time against the sdsmint (MITM)
egress gateway, so the egress path is covered on both gateway variants
rather than only on passthrough.
A part of #823
From
https://github.com/agent-substrate/substrate/actions/runs/32786529664/job/97619491954?pr=1155,
this change adds about 70 seconds to the `e2e-test` workflow:
- Step `Deploy MITM egress demo`: 8s
- Step `Run E2E tests (networking, MITM egress)`: 33s
- Step `Run E2E tests (networking, MITM egress, micro-VM)`: 30s
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This reverts commit 57c3e1927c.
Drops --experimental-additional-egress-extproc-service and the
#ATE_MITM_EXTPROC_FILTER / #ATE_MITM_EXTPROC_CLUSTER markers, so
atenet-egress-with-sdsmint.yaml renders exactly as committed again
and hack/experimental-additional-egress-extproc.sh goes away.
`--experimental-additional-egress-extproc-service=NS/SVC:PORT` makes the
sdsmint egress gateway consult an external authorization processor on
the decrypted leg, where the request is a hostname, method, and path
rather than the IP:port the CONNECT checkpoint sees. The actor's
identity crosses into that callout as the `dev.ate.actor` filter state,
so the processor gets both halves of the decision.
The wiring is generated rather than checked in:
atenet-egress-with-sdsmint.yaml carries inert `#ATE_MITM_EXTPROC_FILTER`
and `#ATE_MITM_EXTPROC_CLUSTER` markers that the installer substitutes
only when the flag is set, so an install without it renders exactly the
manifest as committed.
The flag names any Service, so the processor itself is out of scope
here.
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This allows for faster signing of certs for clusters with large numbers
of atelets.
It also allows configurable ready wait times to compensate for larger
reconciliation waits generally.
Removes the non-functional token/JWT mode for in-cluster ateapi clients.
Clients now always use mTLS certificates; related flags, install and
benchmark plumbing, tests, and overlays are deleted.
Validated with focused Go tests, shellcheck, and Kustomize renders.
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Benjamin Elder