trending/substrate
1
0
Fork 0
mirror of https://github.com/agent-substrate/substrate.git synced 2026-11-12 10:35:06 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
2eeee044e64761f02df7d2ea8a0924685c5d203d
substrate/go.mod
T
Jeremy Alvis f100c27286 ateapi: add versioned PostgreSQL schema migrations (#1196) 
> [!WARNING]
> Recreate PostgreSQL databases from earlier development builds.

## Summary

This change replaces startup schema setup with embedded, versioned SQL
migrations.

`ateapi` uses Goose to apply migrations before readiness. Goose stores
one ledger record for each applied migration.

Goose runs each migration and inserts its ledger record in one
PostgreSQL transaction.

Closes #901.

Based on this design:
https://docs.google.com/document/d/13ixDKRoAIFXeLxS-_1nikNcAy76ca8m0eVgobfib93E/edit?usp=sharing

## Migration behavior

`ateapi` gets a session advisory lock for the configured schema before
it applies pending migrations.

One replica applies migrations while other replicas wait. Goose reads
the ledger again after it gets the lock.

If a migration fails, PostgreSQL rolls back its SQL and ledger record.
Earlier successful migrations remain applied and recorded.

Kubernetes restarts the failed replica. The next startup resumes from
the first migration without a ledger record.

## Changes

- Add Goose and a per-migration ledger.
- Replace the initial up and down files with one transactional, up-only
migration.
- Keep migration 1 aligned with the current schema, including actor
egress policy storage.
- Remove existence guards and explicit transaction statements from the
baseline migration.
- Apply all pending migrations before `ateapi` becomes ready.
- Serialize each migration run with a PostgreSQL session advisory lock.
- Start without changes when the database schema is current or ahead.
- Reject application tables that do not have a migration ledger.
- Log the starting, current, and latest versions.
- Log the applied migration count and duration.
- Retry only initial database connection failures.
- Return schema and migration errors without a retry.
- Add `--postgres-schema` and `ATE_API_POSTGRES_SCHEMA`.
- Use `public` as the default PostgreSQL schema.
- Use the configured schema for the main and watch pools.
- Restrict outbox partition maintenance to the configured schema.
- Let the installer use an external PostgreSQL database.
- Add the migration design and recovery policy to the repository.

## Migration file policy

Migration files use sequential versions and contain exactly one Goose
`Up` section.

CI rejects down migrations, nontransactional migrations, environment
substitution, explicit transaction control, and `IF NOT EXISTS` guards.

Before the first stable v1 release, developers can change or squash
migrations. Developers must recreate databases after migration history
changes.

After that release, CI rejects changes or deletions against the latest
stable release tag that contains migrations.

Goose does not store migration checksums. The binary embeds each
migration file, and release-tag checks protect released migration
history.

## Compatibility

No release includes PostgreSQL support. The `v0.0.0` release predates
the PostgreSQL backend.

Users must recreate databases from earlier PostgreSQL development
builds.

Every committed migration prefix must work with the current and previous
`ateapi` releases. This rule supports rolling upgrades and temporary
binary rollback.

A binary rollback does not roll back the database schema.

## Testing

Tests cover:

- Fresh database migration.
- Concurrent startup.
- Advisory lock waits.
- Current and ahead database schemas.
- Rejection of application tables without a migration ledger.
- Atomic rollback of a failed migration.
- Retention of earlier successful migrations.
- Resume from the failed migration after restart.
- Configured schema isolation.
- Outbox partition isolation.
- Migration file policy checks.
- Stable release migration immutability.
2026-09-02 10:56:19 -04:00

233 lines
12 KiB
AMPL
Raw Blame History

module github.com/agent-substrate/substrate
go 1.27.0
require (
cloud.google.com/go/compute/metadata v0.9.0
cloud.google.com/go/container v1.49.0
cloud.google.com/go/iam v1.9.0
cloud.google.com/go/monitoring v1.24.3
cloud.google.com/go/resourcemanager v1.13.0
cloud.google.com/go/serviceusage v1.14.0
cloud.google.com/go/storage v1.62.1
github.com/aws/aws-sdk-go-v2 v1.41.7
github.com/aws/aws-sdk-go-v2/config v1.32.17
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.10
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
github.com/container-storage-interface/spec v1.12.0
github.com/containerd/ttrpc v1.2.8
github.com/envoyproxy/go-control-plane v0.14.0
github.com/envoyproxy/go-control-plane/envoy v1.37.1-0.20260812071801-353463cc7248
github.com/fsnotify/fsnotify v1.9.0
github.com/go-logr/logr v1.4.4
github.com/google/go-cmp v0.7.0
github.com/google/go-containerregistry v0.21.7
github.com/google/nftables v0.3.0
github.com/google/uuid v1.6.0
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674
github.com/jackc/pgx/v5 v5.10.0
github.com/klauspost/compress v1.19.1
github.com/myzhan/boomer v1.6.1-0.20250711115830-8a2c7ce4c7b1
github.com/opencontainers/runtime-spec v1.3.0
github.com/pelletier/go-toml/v2 v2.4.0
github.com/pressly/goose/v3 v3.27.3
github.com/prometheus/client_golang v1.24.0
github.com/spf13/cobra v1.10.2
github.com/spf13/pflag v1.0.10
github.com/spiffe/go-spiffe/v2 v2.7.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0
github.com/vishvananda/netlink v1.3.1
github.com/vishvananda/netns v0.0.5
go.opentelemetry.io/contrib/bridges/prometheus v0.68.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.68.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0
go.opentelemetry.io/otel v1.44.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.43.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.43.0
go.opentelemetry.io/otel/exporters/prometheus v0.65.0
go.opentelemetry.io/otel/metric v1.44.0
go.opentelemetry.io/otel/sdk v1.44.0
go.opentelemetry.io/otel/sdk/metric v1.44.0
go.opentelemetry.io/otel/trace v1.44.0
go.opentelemetry.io/proto/otlp v1.11.0
golang.org/x/sync v0.22.0
golang.org/x/sys v0.47.0
google.golang.org/api v0.274.0
google.golang.org/genproto/googleapis/rpc v0.0.0-20260720211330-0afa2a65878a
google.golang.org/grpc v1.83.2
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af
k8s.io/api v0.37.0-rc.0
k8s.io/apiextensions-apiserver v0.36.1
k8s.io/apimachinery v0.37.0-rc.0
k8s.io/client-go v0.37.0-rc.0
k8s.io/kubelet v0.37.0-rc.0
k8s.io/metrics v0.36.1
k8s.io/utils v0.0.0-20260626114624-be93311217bd
sigs.k8s.io/controller-runtime v0.24.1
sigs.k8s.io/kustomize/api v0.21.1
sigs.k8s.io/kustomize/kyaml v0.21.1
sigs.k8s.io/yaml v1.6.0
)
require (
cel.dev/expr v0.25.2 // indirect
cloud.google.com/go v0.123.0 // indirect
cloud.google.com/go/auth v0.19.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/longrunning v0.9.0 // indirect
dario.cat/mergo v1.0.2 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.33.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.55.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.55.0 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/asaskevich/EventBus v0.0.0-20200907212545-49d423059eef // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.11 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.17 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 // indirect
github.com/aws/smithy-go v1.25.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/clipperhouse/uax29/v2 v2.6.0 // indirect
github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/cli v29.5.3+incompatible // indirect
github.com/docker/docker-credential-helpers v0.9.3 // indirect
github.com/docker/go-connections v0.7.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/ebitengine/purego v0.10.1 // indirect
github.com/emicklei/go-restful/v3 v3.13.0 // indirect
github.com/envoyproxy/go-control-plane/ratelimit v0.1.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.3.3 // indirect
github.com/evanphx/json-patch/v5 v5.9.11 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/fxamacker/cbor/v2 v2.9.1 // indirect
github.com/go-errors/errors v1.4.2 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-openapi/jsonpointer v1.0.0 // indirect
github.com/go-openapi/jsonreference v1.0.0 // indirect
github.com/go-openapi/swag v0.27.1 // indirect
github.com/go-openapi/swag/cmdutils v0.27.1 // indirect
github.com/go-openapi/swag/conv v0.27.1 // indirect
github.com/go-openapi/swag/fileutils v0.27.1 // indirect
github.com/go-openapi/swag/jsonutils v0.27.1 // indirect
github.com/go-openapi/swag/loading v0.27.1 // indirect
github.com/go-openapi/swag/mangling v0.27.1 // indirect
github.com/go-openapi/swag/netutils v0.27.1 // indirect
github.com/go-openapi/swag/pools v0.27.1 // indirect
github.com/go-openapi/swag/stringutils v0.27.1 // indirect
github.com/go-openapi/swag/typeutils v0.27.1 // indirect
github.com/go-openapi/swag/yamlutils v0.27.1 // indirect
github.com/google/gnostic-models v0.7.0 // indirect
github.com/google/pprof v0.0.0-20250602020802-c6617b811d0e // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.14 // indirect
github.com/googleapis/gax-go/v2 v2.21.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/lufia/plan9stats v0.0.0-20260330125221-c963978e514e // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.23 // indirect
github.com/mattn/go-runewidth v0.0.19 // indirect
github.com/mdlayher/netlink v1.7.3-0.20250113171957-fbb4dce95f42 // indirect
github.com/mdlayher/socket v0.5.0 // indirect
github.com/mfridman/interpolate v0.0.2 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/go-archive v0.3.0 // indirect
github.com/moby/moby/api v1.55.0 // indirect
github.com/moby/moby/client v0.5.0 // indirect
github.com/moby/patternmatcher v0.6.1 // indirect
github.com/moby/spdystream v0.5.1 // indirect
github.com/moby/sys/sequential v0.7.0 // indirect
github.com/moby/sys/user v0.4.1 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/myzhan/gomq v0.0.0-20220926014711-4eea0d4a1e75 // indirect
github.com/myzhan/gomq/zmtp v0.0.0-20220926014711-4eea0d4a1e75 // indirect
github.com/olekukonko/cat v0.0.0-20250911104152-50322a0618f6 // indirect
github.com/olekukonko/errors v1.2.0 // indirect
github.com/olekukonko/ll v0.1.6 // indirect
github.com/olekukonko/tablewriter v1.0.8 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.70.0 // indirect
github.com/prometheus/otlptranslator v1.0.0 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/rivo/uniseg v0.2.0 // indirect
github.com/sethvargo/go-retry v0.4.0 // indirect
github.com/shirou/gopsutil v3.21.11+incompatible // indirect
github.com/shirou/gopsutil/v4 v4.26.6 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/stretchr/testify v1.11.1 // indirect
github.com/testcontainers/testcontainers-go v0.44.0 // indirect
github.com/tklauser/go-sysconf v0.4.0 // indirect
github.com/tklauser/numcpus v0.12.0 // indirect
github.com/ugorji/go/codec v1.3.1 // indirect
github.com/x448/float16 v0.8.4 // indirect
github.com/xlab/treeprint v1.2.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zeromq/goczmq v4.1.0+incompatible // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.44.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.55.0 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/term v0.45.0 // indirect
golang.org/x/text v0.41.0 // indirect
golang.org/x/time v0.15.0 // indirect
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260720211330-0afa2a65878a // indirect
gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/klog/v2 v2.140.0 // indirect
k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad // indirect
k8s.io/streaming v0.37.0-rc.0 // indirect
sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect
sigs.k8s.io/randfill v1.0.0 // indirect
sigs.k8s.io/structured-merge-diff/v6 v6.4.2 // indirect
)
replace k8s.io/apimachinery => ./third_party/k8s.io/apimachinery
Reference in New Issue View Git Blame Copy Permalink