mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-27 07:34:12 +08:00
ILLUM1N0X
fa9875f791
Part of #1827 Move external snapshot transfers in atelet and ate-api-server behind the `objectstoresnapshot.v1` plugin API, and ship a reference plugin for GCS and S3 that runs as a sidecar next to each. ## What changes | Commit | Change | |---|---| | objectstoreplugin: add object-store plugin and snapshot-plugin binary | `internal/objectstoreplugin` implements `NodeProvider` and `ControlProvider` on GCS/S3. `cmd/snapshot-plugin` serves one of them on a Unix socket (`node` or `control` mode). A `healthcheck` mode checks a running plugin's gRPC health service, for the sidecar's startup probe. | | atelet: transfer snapshots through NodeProvider | atelet fetches and uploads external snapshots through the node plugin instead of calling object storage. | | ateapi: clean up and copy snapshots through ControlProvider | ate-api-server deletes and copies external snapshots through the control plugin instead of its own storage client. | | manifests: run snapshot-plugin as atelet and ate-api-server sidecars | Native sidecars next to atelet (node mode) and ate-api-server (control mode), each sharing a pod-local emptyDir for the socket. Each sidecar has an exec `startupProbe` running `snapshot-plugin healthcheck` against its socket, so the main container starts only once the plugin serves. | ## Reviewing - **Moved code.** The node plugin's per-file upload and download are atelet's code moved as is, so it can be found in the old location on `main`. The control plugin wraps the existing `objectstore.DeletePrefix` and `CopyPrefix`. New code is limited to the gRPC glue, request validation, the startup health check, the binary, and the manifests. - **Unchanged object layout.** Files are stored as `<file>.zstd` with an uncompressed `manifest.json`, so snapshots written before and after this change are interchangeable. The manifest still goes last, in its own call, and remains the commit marker. - **Validation added at the plugin boundary.** The plugin now serves a socket API, so it checks every request: the snapshot URI, that the local directory is inside `--root` (`/var/lib/ate`), and that file names are base names with no duplicates. Every directory atelet passes today is under `/var/lib/ate`, and the file-name checks are the ones atelet already applied before transferring. - **A plugin that never comes up fails fast.** Clients dial with `WaitForReady`, so without a check the first transfer would hang. atelet and ate-api- server run `objectstoreplugin.WaitReady` (a gRPC health check) at startup with a one-minute bound and exit if the plugin is not serving. The sidecar's `startupProbe` runs that same health check by executing `snapshot-plugin healthcheck` inside the container, so the kubelet does not start atelet or ate-api-server until the plugin serves. Transfer RPCs get no fixed deadlines, since one would be wrong for large snapshots - **Shared backend selection.** atelet's `ATE_STORAGE_BACKEND` selection moves to `objectstorage.NewFromEnv` in `pkg/objectstorage`, along with `UsesS3` and `NewS3ClientFromEnv`, so atelet and both plugin modes select the backend the same way. These are new exported functions in `pkg/`. - **Package location.** The plugin lives in `internal/objectstoreplugin`, not `internal/plugins/`, because atelet and ate-api-server tests import its in-process clients (`objectstoreplugintest`) and it imports `internal/objectstore`. ## Not in this PR - `actor_jwt` is not set yet. Per-actor authorization is #777. ## Breaking change The object storage settings (`ATE_STORAGE_BACKEND`, `AWS_*`) move from the `ate-api-server` container to its snapshot-plugin` sidecar, which is now the only process in that pod that touches object storage for snapshots. ate-api-server exits at startup if `ATE_STORAGE_BACKEND` is still set on it. An install that patches these settings onto the `ate-api-server` container must patch the sidecar instead; see the kind overlay for an S3 example. ## Testing - `make test` and `make verify`. - New unit tests for the plugins (`internal/objectstoreplugin`), backend selection (`pkg/objectstorage`), and the shared validation helpers (`internal/resources`). - E2E on kind runs in CI (gVisor and micro-VM, S3 via rustfs). - E2E on GKE (GCS)