## Summary
A follow up to #640 where we introduced PostgreSQL as an alternative
storage backend, selected conditionally in ateapi.
- Deleted ateredis, its tests, and its dependencies
- Removed Redis backend selection and configuration so ateapi always
connects to Postgres
- Replaced Valkey resources with Postgres in the standard and Kind
deployment paths and simplified install script
- Replaced miniredis fixtures with isolated Postgres testcontainers and
added centralized helpers for seeding resources
- Renamed Redis-specific debug flush command to backend-neutral
`debug-clear-store` in CLI
- Updated comments and docs where applicable
## Benchmarking
Extensive benchmarking have been performed to evaluate Redis vs
Postgres, and results can be found in these two documents:
-
https://docs.google.com/document/d/10K0wB6aTeFkJCL4HN3NbLJCdFGoLYdhIcqkFnqFHkKc/edit?usp=sharing
-
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
Adds Postgres as an alternative persistence backend for ateapi as
proposed in #731. Currently this is opt-in with flags in ateapi or use
the `--store-backend=postgres` option in the install script.
Slack discussion:
https://cloud-native.slack.com/archives/C0B6M3E2J3D/p1785183769252679
Performance benchmark results for Postgres vs Redis:
https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing
Benchmarks are not included in this PR, but you can reproduce the
results by following the doc above
Limitations and questions:
- This draft does not yet handle migrations, this might come as a follow
up
- It currently uses `testcontainers` for testing Postgres, but as a
result it adds a lot of dependencies
---------
Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
This PR addresses the cold-start timeout issue reported in #811 by
driving down the decompression/extraction cost of gVisor release
archives on nodes, adding structured logging, and supporting context
cancellation during extraction.
* Ran unit tests locally: `go test -v ./cmd/atelet/... -run
TestExtractTarArchive` (PASS).
* Benchmarked extraction formats (Bzip2: 19.05s baseline vs Gzip: 1.50s
vs Uncompressed Tar: 0.25s).
atecontroller had no OTel at all. Dev-mode zap logger, and its
controller-runtime metrics were only available on a :8080 that we don't
scrape.
After this PR, logs go through the shared slog handler (plus a
`--log-level` flag to match the other binaries), and
controller-runtime's Prometheus registry is bridged onto the OTLP reader
so the reconcile/workqueue metrics actually reach the collector.
Filtering these out is a pipeline responsibility. Also added otelgrpc to
the ateapi client, which was untraced.
This unblocks #564 the workperpool metrics, cc @Angelawork, @JeffLuoo:
there's a working `MeterProvider` to use for
`ate.workerpool.desired_workers`/`ready_workers`.
Couple of things to mention for review:
-`InitMetricsPushOnly`, not `InitMetrics`, even though we do serve
:8080. That port is controller-runtime's own private registry, not the
global one `serverboot.metricsMux` serves, so a pull reader there would
collect into something we never expose.
- Bridge is pinned to v0.68.0 to match otelgrpc. Wanted to go to
v0.70.0, but that requires otel/sdk/metric 1.45.0 and pulls the whole
SDK up with it (406 vendor files instead of 81). Happy to do that bump
separately.
- zap/zapr fall out of go.mod since atecontroller was the last importer.
- I included an OTel collector image bump from the early 2024 (!) one to
latest, which was breaking exposing native histograms
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
The router used to rewrite :authority to the actor's worker pod IP and let
the dynamic_forward_proxy cluster resolve it, reaching the actor over
plaintext pod-IP:80. The worker no longer DNATs pod-IP:80 to the actor, so
that path is gone.
Instead the ext_proc resolves the actor to its worker's atunnel ingress
address (IP:443) and puts it in x-ate-original-dst. An ORIGINAL_DST cluster
dials exactly that address, which leaves the request Host as the actor DNS
name -- atunnel needs it to authorize the active actor. The header is set
with OVERWRITE_IF_EXISTS_OR_ADD so a client-supplied value can never
influence the address Envoy dials.
The upstream hop is mTLS: the cluster presents the router's podidentity
credential bundle as its client cert and validates the atunnel server
against the podidentity trust bundle. Validation matches the SPIFFE URI SAN
prefix rather than the dialed pod IP, because the atunnel cert carries only
a spiffe:// URI SAN and Envoy's default SAN check against an ephemeral pod
IP would never match. atunnel in turn only accepts
spiffe://cluster.local/ns/ate-system/sa/atenet-router.
The dynamic_forward_proxy cluster, HTTP filter and DNS cache config are
removed along with the :authority rewrite.
The weekly govulncheck run on main is failing:
[GO-2026-6061](https://pkg.go.dev/vuln/GO-2026-6061) (vulnerabilities in
the xDS RBAC engine and the HTTP/2 transport server) in
google.golang.org/grpc v1.81.0.
Bump grpc to the fixed v1.82.
Part of [#170](https://github.com/agent-substrate/substrate/issues/170)
Establishes mutual TLS between all ate system components, and updates
the certificate plumbing it depends on.
Main changes:
1. The atenet router now verifies ate apiserver' serving certificate,
and presents its client cert to ate apiserver. Previously the connection
used `InsecureSkipVerify`.
2. AteApi server verifies atelet's serving cert.
Minor bug fixes:
1. Prevent `servicednssigner` from signing a cert with no DNS SANs. Also
updated valkey cluster's cert configuration, because it was relying on
the cert with empty DNS.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
Content-addressed pool of unpacked image layers, shared by every actor on
the node; actor rootfs becomes an overlayfs mount (cached layers as read-
only lowers, bundle-local upper) instead of a full re-untar per run.
atelet (no capabilities) pulls and unpacks; the privileged ateoms finalize
whiteouts and mount. Tag refs resolve via one HEAD and become cacheable;
pull memory is O(stream buffers); the cache survives restarts.
Phase 1 of #463. Fixes#437, #166, #228.
Validated: kind + GKE counter demos (gvisor and microvm), suspend/resume
(oci_unpack ~3ms vs ~15-20s), 411 SWE-bench-scale images pulled and
unpacked with 0 failures, root-gated unit tests for the privileged paths.
Known gaps: no GC yet (Phase 2, see internal/imagecache/README.md);
upgrade ordering — deploy new ateoms before/with the new atelet.
## Summary
Bumps `github.com/google/go-containerregistry` v0.21.5 → v0.21.7. Phase
0 of #463.
v0.21.6 fixed the memory spike in `mutate.Extract` described in #120
(upstream fix: google/go-containerregistry#2190, merged as
google/go-containerregistry@38d6e4087c): extraction is now refactored
into a per-layer `extractLayer` function, so each layer's decompressor
and HTTP response body are closed as soon as that layer is processed,
instead of all being held open via deferred `Close` calls until the
entire (multi-GB) extraction completes. atelet's pull path goes through
`memorypullcache.Fetch` → `mutate.Extract`, so this directly caps its
extract-time memory.
Note this is only interim relief: #437's unbounded cache retention is
untouched and is addressed by the redesign in #463.
## Changes
- `go-containerregistry` v0.21.5 → v0.21.7, plus required transitive
bumps: `docker/cli`, `klauspost/compress`, `golang.org/x/sync`,
`golang.org/x/sys`
- `go mod tidy` + `go mod vendor`; the
`containerd/stargz-snapshotter/estargz` vendor tree drops out (upstream
removed the estargz integration; nothing in substrate used it), along
with `vbatts/tar-split` and `mitchellh/go-homedir`
## Testing
- `go build ./...` passes
- Full `go test ./...` passes
- Verified the layer-close fix is present in the vendored
`pkg/v1/mutate/mutate.go`
Fixes#120🤖 Generated with [Claude Code](https://claude.com/claude-code)
go-toml/v2 parses the kata configuration.toml; ttrpc (and its log dependency)
backs the kata-agent client the micro-VM runtime drives. Includes their licenses.
Jet Chiang