Jeremy Alvis
f100c27286
> [!WARNING] > Recreate PostgreSQL databases from earlier development builds. ## Summary This change replaces startup schema setup with embedded, versioned SQL migrations. `ateapi` uses Goose to apply migrations before readiness. Goose stores one ledger record for each applied migration. Goose runs each migration and inserts its ledger record in one PostgreSQL transaction. Closes #901. Based on this design: https://docs.google.com/document/d/13ixDKRoAIFXeLxS-_1nikNcAy76ca8m0eVgobfib93E/edit?usp=sharing ## Migration behavior `ateapi` gets a session advisory lock for the configured schema before it applies pending migrations. One replica applies migrations while other replicas wait. Goose reads the ledger again after it gets the lock. If a migration fails, PostgreSQL rolls back its SQL and ledger record. Earlier successful migrations remain applied and recorded. Kubernetes restarts the failed replica. The next startup resumes from the first migration without a ledger record. ## Changes - Add Goose and a per-migration ledger. - Replace the initial up and down files with one transactional, up-only migration. - Keep migration 1 aligned with the current schema, including actor egress policy storage. - Remove existence guards and explicit transaction statements from the baseline migration. - Apply all pending migrations before `ateapi` becomes ready. - Serialize each migration run with a PostgreSQL session advisory lock. - Start without changes when the database schema is current or ahead. - Reject application tables that do not have a migration ledger. - Log the starting, current, and latest versions. - Log the applied migration count and duration. - Retry only initial database connection failures. - Return schema and migration errors without a retry. - Add `--postgres-schema` and `ATE_API_POSTGRES_SCHEMA`. - Use `public` as the default PostgreSQL schema. - Use the configured schema for the main and watch pools. - Restrict outbox partition maintenance to the configured schema. - Let the installer use an external PostgreSQL database. - Add the migration design and recovery policy to the repository. ## Migration file policy Migration files use sequential versions and contain exactly one Goose `Up` section. CI rejects down migrations, nontransactional migrations, environment substitution, explicit transaction control, and `IF NOT EXISTS` guards. Before the first stable v1 release, developers can change or squash migrations. Developers must recreate databases after migration history changes. After that release, CI rejects changes or deletions against the latest stable release tag that contains migrations. Goose does not store migration checksums. The binary embeds each migration file, and release-tag checks protect released migration history. ## Compatibility No release includes PostgreSQL support. The `v0.0.0` release predates the PostgreSQL backend. Users must recreate databases from earlier PostgreSQL development builds. Every committed migration prefix must work with the current and previous `ateapi` releases. This rule supports rolling upgrades and temporary binary rollback. A binary rollback does not roll back the database schema. ## Testing Tests cover: - Fresh database migration. - Concurrent startup. - Advisory lock waits. - Current and ahead database schemas. - Rejection of application tables without a migration ledger. - Atomic rollback of a failed migration. - Retention of earlier successful migrations. - Resume from the failed migration after restart. - Configured schema isolation. - Outbox partition isolation. - Migration file policy checks. - Stable release migration immutability.
4.7 KiB
How to Contribute
We would love to accept your patches and contributions to this project. Before you spend a lot of time on a contribution, please review the following guidelines.
Before you begin
Sign our Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License Agreement (CLA). You (or your employer) retain the copyright to your contribution; this simply gives us permission to use and redistribute your contributions as part of the project.
If you or your current employer have already signed the Google CLA (even if it was for a different project), you probably don't need to do it again.
Visit https://cla.developers.google.com/ to see your current agreements or to sign a new one.
Review our Community Guidelines
This project follows Google's Open Source Community Guidelines.
Set up a local development environment
The Quickstart (Development) in the README
covers bringing up a local cluster with the default (gVisor) runtime. To run
the microVM runtime locally — which needs /dev/kvm, or Lima nested
virtualization on Apple Silicon — see
docs/dev/microvm-local.md.
Contribution process
This is a very new project, so we are still working out exactly how it is going to be developed. For now, we are focused on iterating quickly to find the right design and architecture. This has implications for contributors:
-
Things are moving quickly, so PRs may need to be rebased or updated frequently. Small PRs that are focused on a single issue or feature are easier to review and update than large PRs that touch many different parts of the codebase.
-
While we welcome new contributors, we are really focused on the minimal capabilities needed to make this project useful. Before you start a new contribution, please discuss it with us first (if there is an issue open, comment there and if not, open one). We want to make sure that your work is aligned with our near-term goals for the project and that we are not duplicating work that is already in flight.
-
PRs which are not aligned with our near-term goals may be closed without extensive review. We are not trying to be discouraging, but we need to make sure that we are focused on the most important work.
If you are building something that runs on Substrate rather than changing Substrate itself, see Integration Repositories for where that code should live.
Sizing PRs for review
We optimize PRs for easy review — large PRs get broken down, small PRs get merged.
- Large PRs: split huge changes into a series of smaller PRs, each a logically distinct feature. When the intermediate steps are not useful on their own, keep the change as one PR split into commits at logical break points, and preserve those commits on merge.
- Small and bulk PRs: if you find a typo, review the whole file and fix everything in one pass rather than sending the single edit. Group related typo, doc, and single-line cleanup fixes into one PR rather than opening several small ones for the same area. Maintainers may ask you to consolidate fragmented PRs into one, or close them in favor of a combined submission.
As a rough scale: S is under 30 changed lines, M under 100, L under 500, XL under 1000. Most PRs should be L or smaller; XL and above are candidates for breaking down.
Code Reviews
All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose.
All code changes should be accompanied by tests. We will not merge code that does not have tests, and we will not merge code that causes tests to fail.
Follow the PostgreSQL schema evolution rules for each application schema change.
Root-gated tests
Tests that need root (overlay mounts, mknod, trusted.* xattrs, ...) call
roottest.Require(t, ...) from internal/roottest as
their first statement. They skip in a plain go test ./...; CI reruns every
package whose tests import that package under sudo. To run them locally:
hack/run-root-tests.sh
New privileged tests only need the roottest.Require call — no CI changes.
Copyright Headers
Every file containing source code must include copyright and license information. This includes any JS/CSS files that you might be serving out to browsers. (This is to help well-intentioned people avoid accidental copying that doesn't comply with the license.)
Our standard headers for various filetypes can be found in ./hack/boilerplate.