I've seen this test flake at least once due to (I think) the way we are
truncating the timestamp for the NotBefore checks.
Replace all the clock injection stuff with synctest. No failures after
100 runs using `stress`.
```
taahm@taahm1 /u/l/g/h/t/p/s/substrate (deflake-signer)> go test ./cmd/podcertcontroller/...
? github.com/agent-substrate/substrate/cmd/podcertcontroller [no test files]
ok github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/podcertificate 0.013s
ok github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/podidentitysigner 0.031s
? github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/rendezvous [no test files]
? github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/servicednssigner [no test files]
? github.com/agent-substrate/substrate/cmd/podcertcontroller/internal/signercontroller [no test files]
```
```
taahm@taahm1 /u/l/g/h/t/p/s/substrate (deflake-signer) [1]> stress -count 100 go test ./cmd/podcertcontroller/...
5s: 24 runs so far, 0 failures, 8 active
10s: 48 runs so far, 0 failures, 8 active
15s: 63 runs so far, 0 failures, 8 active
20s: 78 runs so far, 0 failures, 8 active
25s: 89 runs so far, 0 failures, 8 active
27s: 100 runs total, 0 failures
```
Emit one "<component> starting" line right after the logger is up,
carrying the stamped version.
atecontroller also gains the --version flag it was missing.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
## Summary
The controller built its Kubernetes client from the default rest.Config,
whose client-side limiter (QPS=5, Burst=10) is shared by every worker of
both signers. Certificate issuance was therefore capped at roughly 5
PodCertificateRequests per second no matter the `--workers-per-signer`
value, and pods in a large rollout sat in FailedMount ("credential
bundle is not issued yet") for the duration: about 2000 seconds for a
10k-pod worker pool in the 1000-node load test.
This adds `--kube-api-qps` and `--kube-api-burst` and applies them to
the config on both the in-cluster and kubeconfig paths. Both default to
0, which leaves client-go's own defaults untouched — behavior only
changes where a deployment opts in. A follow-up PR will set values sized
for large rollouts in
`manifests/ate-install/pod-certificate-controller.yaml`.
## Test plan
- `go build` / `go vet` clean.
- Verified against the 1000-node load test (with raised limits set via
the flags): bundle issuance keeps pace with pod creation instead of
plateauing at ~4.5/sec.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
This commit fleshes out localca's rotation support by changing the
interface so that signing operations should always go through a pool.
All existing in-tree callers are converted over to this pattern. This
ensures that signing can properly continue as a CA pool is rotated,
without requiring any process restarts.
The pool is periodically reloaded from disk in the course of signing
operations, with the loaded data being cached for up to a minute.
Some code for storing the CA state using PEM files was removed because
it seemed to be dead.
Follow-on changes will give localjwt a similar treatment, and add
administrative commands for running rotations on pools.
## Summary
A follow up to #640 where we introduced PostgreSQL as an alternative
storage backend, selected conditionally in ateapi.
- Deleted ateredis, its tests, and its dependencies
- Removed Redis backend selection and configuration so ateapi always
connects to Postgres
- Replaced Valkey resources with Postgres in the standard and Kind
deployment paths and simplified install script
- Replaced miniredis fixtures with isolated Postgres testcontainers and
added centralized helpers for seeding resources
- Renamed Redis-specific debug flush command to backend-neutral
`debug-clear-store` in CLI
- Updated comments and docs where applicable
## Benchmarking
Extensive benchmarking have been performed to evaluate Redis vs
Postgres, and results can be found in these two documents:
-
https://docs.google.com/document/d/10K0wB6aTeFkJCL4HN3NbLJCdFGoLYdhIcqkFnqFHkKc/edit?usp=sharing
-
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Secures network-based gRPC connections between the ate-api control plane
and CSI driver controller services using mutual TLS (mTLS) backed by
Substrate's Pod Identity.
Requests from ate-api to CSI driver controllers occur over the network,
as configured via the CSIDriverConfig custom resource. This PR extends
the CSIDriverConfig CRD to include a tls configuration section, which
leverages
Substrate's Pod Identity certificate infrastructure (managed by
podcertcontroller) to secure these
TCP connections using mTLS. Because standard CSI drivers typically
communicate over Unix domain sockets, a proxy sidecar (e.g., envoy) is
added to the CSI controller deployment to terminate TLS and forward
requests to the local CSI socket.
#232
> It's a good idea to open an issue first for discussion.
- [X ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR
This allows for faster signing of certs for clusters with large numbers
of atelets.
It also allows configurable ready wait times to compensate for larger
reconciliation waits generally.
## Summary
The comment on Hasher states the implementation is based on FNV-1a, but
fnvhash used fnv.New64() which is FNV-1. Switch to fnv.New64a() to match
the documented algorithm and get better distribution.
## Why this is safe
The hash function is used solely for live work assignment across
controller replicas via rendezvous hashing. No hash values are persisted
anywhere.
Every worker pod now hosts an atunnel ingress server on :443 and an
atunnel egress listener, both long-lived, with per-activation
Activate/Deactivate bracketing every Run, Restore and Checkpoint.
The nftables rules ateom installs change accordingly:
* The pod-IP:80 -> actor-veth:80 DNAT is gone. Worker port 80 is no
longer an Actor ingress path; the only way in is the mTLS listener
on :443, which authorizes the caller and checks the Actor is the one
currently assigned to this worker.
* A prerouting REDIRECT sends Actor TCP egress to the local atunnel
egress listener, preserving SO_ORIGINAL_DST. It is only installed
when the activation carries an egress gateway address, which nothing
populates yet, so the masquerade path is unchanged and Actor egress
behaves exactly as before.
The ateom Run/Restore protos gain the two fields that arm that path:
egress_gateway_address, which decides whether the redirect is installed
at all, and actor_version, the Actor resource version ate-api observed
when assigning the worker, which atunnel asserts to the egress gateway
as a lower bound on trustworthy Actor metadata. Both are consumed here
and left unset. atelet and ate-api start populating them in the egress
gateway change, which is the point at which they mean anything, so this
change adds no new requirement to the atelet wire contract.
atecontroller gives worker pods the podidentity credential + trust
bundles (the atunnel server identity), the servicedns trust bundle, and
container port 443. podidentitysigner now issues certs with
ExtKeyUsageServerAuth as well as ClientAuth, without which the worker
cannot present its podidentity cert as a TLS server cert and the
gateway handshake fails.
Part of [#170](https://github.com/agent-substrate/substrate/issues/170)
Establishes mutual TLS between all ate system components, and updates
the certificate plumbing it depends on.
Main changes:
1. The atenet router now verifies ate apiserver' serving certificate,
and presents its client cert to ate apiserver. Previously the connection
used `InsecureSkipVerify`.
2. AteApi server verifies atelet's serving cert.
Minor bug fixes:
1. Prevent `servicednssigner` from signing a cert with no DNS SANs. Also
updated valkey cluster's cert configuration, because it was relying on
the cert with empty DNS.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Adds internal/version with three vars (Version, Commit, BuildDate) set
via -ldflags -X at build time. When unset (e.g. plain `go build`
without LDFLAGS), runtime/debug.ReadBuildInfo() populates them from the
embedded vcs info so `go install`, `go run`, and ko-built images
(buildvcs is on by default) still produce something meaningful.
Wired through:
- Makefile: VERSION / COMMIT / BUILD_DATE resolved from `git describe`
/ `git rev-parse` / `date`. LDFLAGS plumbed into every go-build and
ko-build target (build-images, build-atectl, build-atenet,
build-demos).
- hack/install-ate.sh: untouched here — ko apply / resolve don't
accept --ldflags, and ko's buildvcs embedding gives Commit +
BuildDate automatically through runtime/debug.ReadBuildInfo() in
the version package's init() fallback.
All six binaries respond to `--version`:
- kubectl-ate and atenet (cobra): rootCmd.Version = version.String()
- ateapi, atelet, ateom-gvisor, podcertcontroller (flag): explicit
`-version` bool that prints and exits.
Sample output:
kubectl-ate version 3b0a0d5 commit=3b0a0d5c2... \
built=2026-05-26T14:16:53Z darwin/arm64
Closes the "kubectl-ate --version has nothing to print" gap; gives ops
a way to tell at a glance which commit a deployed substrate component
came from.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
Taahir Ahmed