This commit changes everything except HTTP(s) links.
github.com/agent-substrate/substrate -> go.ate.dev/substrate
I chose "go.ate.dev" for brevity, but "pkg.ate.dev" would be OK, too.
This is implemented as "CloudFlare Pages" on my personal account. If we
like it, we need to make it more "community operated".
This does not yet convert other repos.
This breaks a TON of open PRs, so we should consider if/when we want to
merge it.
/cc @BenTheElder @rakyll @EItanya
We should rely on the checks enforced by kube-apiserver. We don't need
to double-check whether the pod exists.
Upcoming e2e test changes will rely on the fact that non-Kubelet clients
are allowed to create arbitary PodCertificateRequests, even if the pod
or service account they reference does not exist. This will allow the
test code running in Github Actions (or on the developer's machine) to
authenticate to the router running within the cluster.
I went through a number of models and this seems to be the simplest and
lowest overall impact.
To keep it simple:
* All gates are disabled unless enabled
* Only "preview" and "ga" - no "alpha" or "beta"
* We only have one gate for now, called "Preview" (later we can break
this down, maybe)
* All gates can be activated by passing "*"
* Package internal/preview has an Init() function which gets passed a
list of gates to enable, which is stored as a package-global
* Code can call preview.IsEnabled(gatename)
Fixes#1690
@juli4n @BenTheElder @msau42
### Summary
Every replica receives every PodCertificateRequest, but only the replica
chosen by rendezvous hashing should sign it. The others return
`ErrNotAssigned` and were requeued with `AddRateLimited`, which treats a
normal condition as a failure:
- consumes the shared failure rate limiter (10 QPS bucket), and
- grows the per-item exponential backoff (up to 1000s).
This slows retries of genuinely assigned PCRs and delays takeover when a
replica dies (the PCRs it owned may already be deep in backoff on the
surviving replicas).
Unassigned PCRs are now requeued with
`AddAfter(rendezvous.LeaseDuration)`. Assignments only change when a
replica joins or its lease expires, so re-checking more often gains
nothing. `LeaseDuration` is exported from `rendezvous` for this purpose;
its value is unchanged.
A follow-up could make this event-driven (re-enqueue pending PCRs when
the active replica set changes) instead of polling.
### Testing
`go test ./cmd/podcertcontroller/...`
- [x] Tests pass
- [ ] Documentation updated (n/a)
## Summary
Part of: #2097
Bundled PostgreSQL currently accepts any client certificate signed by
the pod-identity CA without binding it to the requested database
username. Require certificate authentication with a common name matching
the login, using a dedicated PostgreSQL client CA.
- Add a pod-certificate signer restricted to `ate-system` pods using the
`ate-api-server` service account. It issues only the two permitted
identities: `substrate_owner_user` and `substrate_readwrite_user`.
- Project separate certificates for the owner and runtime connections.
Configure passwordless connection strings with server certificate
verification, and retain the existing owner/runtime role grants.
- Have `ate-setup` provision the CA and configure bundled PostgreSQL
automatically, including on an existing installation. Disable passwords
on the bundled application users and keep administrator access local to
the PostgreSQL pod.
- Reuse existing rotation support: new database connections from ateapi
reread projected credentials and trust roots, while the bundled
PostgreSQL TLS-reloader detects server-certificate or
client-trust-bundle changes every 60 seconds and requests a
configuration reload.
External PostgreSQL and Cloud SQL continue to use their configured
connection strings and authentication settings. Both application
certificates are available to the API-server process; separate
credentials preserve database role separation but do not isolate the
roles from a compromised API server.
## Validation
- Passed `go test -race ./cmd/podcertcontroller/... ./cmd/ate-setup/...
./cmd/ateapi/internal/store/atepg ./pkg/postgressetup
./internal/installdefaults` after rebasing onto updated main.
- Signer tests cover both allowed identities, unauthorized requests, pod
identity checks, certificate lifetimes, and trust bundles.
- Docker integration tests use the bundled PostgreSQL configuration and
verify successful passwordless connections, bootstrap/migrations, and
runtime operations. They reject mismatched usernames, untrusted
certificates, missing certificates, remote administrator access,
plaintext connections, and runtime access to owner privileges.
Fixes#2153
Unchanged ClusterTrustBundles were rewritten every 5 seconds. Now
skipped.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Relates to https://github.com/agent-substrate/substrate/issues/1819
This is the followup to #1829 which did PCR.
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This adds dual v1beta1 and v1 PCR support, favoring v1. We have seen
users on v1.37 without v1beta1 but with v1 seeing incomaptibility.
Given our usage of PCR is pretty scoped its not too bad to have the dual
version support.
Fixes #<issue_number_goes_here>
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
I've seen this test flake at least once due to (I think) the way we are
truncating the timestamp for the NotBefore checks.
Replace all the clock injection stuff with synctest. No failures after
100 runs using `stress`.
```
taahm@taahm1 /u/l/g/h/t/p/s/substrate (deflake-signer)> go test ./cmd/podcertcontroller/...
? github.com/agent-substrate/substrate/cmd/podcertcontroller [no test files]
ok github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/podcertificate 0.013s
ok github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/podidentitysigner 0.031s
? github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/rendezvous [no test files]
? github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/servicednssigner [no test files]
? github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/signercontroller [no test files]
```
```
taahm@taahm1 /u/l/g/h/t/p/s/substrate (deflake-signer) [1]> stress -count 100 go test ./cmd/podcertcontroller/...
5s: 24 runs so far, 0 failures, 8 active
10s: 48 runs so far, 0 failures, 8 active
15s: 63 runs so far, 0 failures, 8 active
20s: 78 runs so far, 0 failures, 8 active
25s: 89 runs so far, 0 failures, 8 active
27s: 100 runs total, 0 failures
```
Emit one "<component> starting" line right after the logger is up,
carrying the stamped version.
atecontroller also gains the --version flag it was missing.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Summary
The controller built its Kubernetes client from the default rest.Config,
whose client-side limiter (QPS=5, Burst=10) is shared by every worker of
both signers. Certificate issuance was therefore capped at roughly 5
PodCertificateRequests per second no matter the `--workers-per-signer`
value, and pods in a large rollout sat in FailedMount ("credential
bundle is not issued yet") for the duration: about 2000 seconds for a
10k-pod worker pool in the 1000-node load test.
This adds `--kube-api-qps` and `--kube-api-burst` and applies them to
the config on both the in-cluster and kubeconfig paths. Both default to
0, which leaves client-go's own defaults untouched — behavior only
changes where a deployment opts in. A follow-up PR will set values sized
for large rollouts in
`manifests/ate-install/pod-certificate-controller.yaml`.
## Test plan
- `go build` / `go vet` clean.
- Verified against the 1000-node load test (with raised limits set via
the flags): bundle issuance keeps pace with pod creation instead of
plateauing at ~4.5/sec.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
This commit fleshes out localca's rotation support by changing the
interface so that signing operations should always go through a pool.
All existing in-tree callers are converted over to this pattern. This
ensures that signing can properly continue as a CA pool is rotated,
without requiring any process restarts.
The pool is periodically reloaded from disk in the course of signing
operations, with the loaded data being cached for up to a minute.
Some code for storing the CA state using PEM files was removed because
it seemed to be dead.
Follow-on changes will give localjwt a similar treatment, and add
administrative commands for running rotations on pools.
## Summary
A follow up to #640 where we introduced PostgreSQL as an alternative
storage backend, selected conditionally in ateapi.
- Deleted ateredis, its tests, and its dependencies
- Removed Redis backend selection and configuration so ateapi always
connects to Postgres
- Replaced Valkey resources with Postgres in the standard and Kind
deployment paths and simplified install script
- Replaced miniredis fixtures with isolated Postgres testcontainers and
added centralized helpers for seeding resources
- Renamed Redis-specific debug flush command to backend-neutral
`debug-clear-store` in CLI
- Updated comments and docs where applicable
## Benchmarking
Extensive benchmarking have been performed to evaluate Redis vs
Postgres, and results can be found in these two documents:
-
https://docs.google.com/document/d/10K0wB6aTeFkJCL4HN3NbLJCdFGoLYdhIcqkFnqFHkKc/edit?usp=sharing
-
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Secures network-based gRPC connections between the ate-api control plane
and CSI driver controller services using mutual TLS (mTLS) backed by
Substrate's Pod Identity.
Requests from ate-api to CSI driver controllers occur over the network,
as configured via the CSIDriverConfig custom resource. This PR extends
the CSIDriverConfig CRD to include a tls configuration section, which
leverages
Substrate's Pod Identity certificate infrastructure (managed by
podcertcontroller) to secure these
TCP connections using mTLS. Because standard CSI drivers typically
communicate over Unix domain sockets, a proxy sidecar (e.g., envoy) is
added to the CSI controller deployment to terminate TLS and forward
requests to the local CSI socket.
#232
> It's a good idea to open an issue first for discussion.
- [X ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This allows for faster signing of certs for clusters with large numbers
of atelets.
It also allows configurable ready wait times to compensate for larger
reconciliation waits generally.
## Summary
The comment on Hasher states the implementation is based on FNV-1a, but
fnvhash used fnv.New64() which is FNV-1. Switch to fnv.New64a() to match
the documented algorithm and get better distribution.
## Why this is safe
The hash function is used solely for live work assignment across
controller replicas via rendezvous hashing. No hash values are persisted
anywhere.
Every worker pod now hosts an atunnel ingress server on :443 and an
atunnel egress listener, both long-lived, with per-activation
Activate/Deactivate bracketing every Run, Restore and Checkpoint.
The nftables rules ateom installs change accordingly:
* The pod-IP:80 -> actor-veth:80 DNAT is gone. Worker port 80 is no
longer an Actor ingress path; the only way in is the mTLS listener
on :443, which authorizes the caller and checks the Actor is the one
currently assigned to this worker.
* A prerouting REDIRECT sends Actor TCP egress to the local atunnel
egress listener, preserving SO_ORIGINAL_DST. It is only installed
when the activation carries an egress gateway address, which nothing
populates yet, so the masquerade path is unchanged and Actor egress
behaves exactly as before.
The ateom Run/Restore protos gain the two fields that arm that path:
egress_gateway_address, which decides whether the redirect is installed
at all, and actor_version, the Actor resource version ate-api observed
when assigning the worker, which atunnel asserts to the egress gateway
as a lower bound on trustworthy Actor metadata. Both are consumed here
and left unset. atelet and ate-api start populating them in the egress
gateway change, which is the point at which they mean anything, so this
change adds no new requirement to the atelet wire contract.
atecontroller gives worker pods the podidentity credential + trust
bundles (the atunnel server identity), the servicedns trust bundle, and
container port 443. podidentitysigner now issues certs with
ExtKeyUsageServerAuth as well as ClientAuth, without which the worker
cannot present its podidentity cert as a TLS server cert and the
gateway handshake fails.
Part of [#170](https://github.com/agent-substrate/substrate/issues/170)
Establishes mutual TLS between all ate system components, and updates
the certificate plumbing it depends on.
Main changes:
1. The atenet router now verifies ate apiserver' serving certificate,
and presents its client cert to ate apiserver. Previously the connection
used `InsecureSkipVerify`.
2. AteApi server verifies atelet's serving cert.
Minor bug fixes:
1. Prevent `servicednssigner` from signing a cert with no DNS SANs. Also
updated valkey cluster's cert configuration, because it was relying on
the cert with empty DNS.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Adds internal/version with three vars (Version, Commit, BuildDate) set
via -ldflags -X at build time. When unset (e.g. plain `go build`
without LDFLAGS), runtime/debug.ReadBuildInfo() populates them from the
embedded vcs info so `go install`, `go run`, and ko-built images
(buildvcs is on by default) still produce something meaningful.
Wired through:
- Makefile: VERSION / COMMIT / BUILD_DATE resolved from `git describe`
/ `git rev-parse` / `date`. LDFLAGS plumbed into every go-build and
ko-build target (build-images, build-atectl, build-atenet,
build-demos).
- hack/install-ate.sh: untouched here — ko apply / resolve don't
accept --ldflags, and ko's buildvcs embedding gives Commit +
BuildDate automatically through runtime/debug.ReadBuildInfo() in
the version package's init() fallback.
All six binaries respond to `--version`:
- kubectl-ate and atenet (cobra): rootCmd.Version = version.String()
- ateapi, atelet, ateom-gvisor, podcertcontroller (flag): explicit
`-version` bool that prints and exits.
Sample output:
kubectl-ate version 3b0a0d5 commit=3b0a0d5c2... \
built=2026-05-26T14:16:53Z darwin/arm64
Closes the "kubectl-ate --version has nothing to print" gap; gives ops
a way to tell at a glance which commit a deployed substrate component
came from.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
Tim Hockin