Jeremy Alvis
41c5f01968
Fixes #997 ## Summary Give `ateapi` separate PostgreSQL connections for schema ownership and normal application access. Migrations and outbox partition maintenance use the owner connection, allowing application reads and writes to run with restricted permissions. - Each connection assumes its configured role with `SET ROLE`, keeping object ownership and grants separate from login credentials. - Configure owner and read/write connections through `ate-setup` and the installation manifests. - Default the schema to `substrate` and document provisioning for external PostgreSQL and Cloud SQL, including one IAM login with membership in both roles. - Keep bundled PostgreSQL using its existing `postgres` account and role for both pools. Restricted runtime access requires operator-provisioned roles. See `docs/postgres.md` for configuration details. ## Scope This is the base PR for two separate follow-ups: [file-based credentials](https://github.com/agent-substrate/substrate/pull/1979) and [automatic development database bootstrap](https://github.com/agent-substrate/substrate/pull/2025). Neither feature is included here. Operators provision external database roles, login memberships, schema ownership, and grants before startup. Table migrations still run automatically through the owner connection. The documented external database layout assumes a fresh database; this PR does not migrate existing table ownership or grants. Both connections remain inside `ateapi`, so this separates database permissions without isolating owner credentials from the application process. ## Validation - `env -u NO_COLOR make verify` passed. - PostgreSQL integration tests cover restricted runtime access, owner migrations and partition maintenance, and OpenFGA table permissions. - Installer tests cover bundled PostgreSQL, separate external logins, and Cloud SQL configuration.
4.8 KiB
Scaling the Cloud SQL store
Sizing guidance for running the ateapi PostgreSQL store (tools/setup-gcp/cloud-sql.md) at large actor counts and high request rates.
The provisioning defaults (db-custom-2-8192, 10 GB disk) suit development
and modest fleets. At large actor counts the store becomes I/O-bound: once
tables and indexes outgrow memory, uniform random reads fall out of cache and
point lookups pay persistent-disk latency (several milliseconds) instead of
microseconds. The knobs below address that, in order of leverage.
Instance shape
Set with --tier / --edition at create time, or gcloud sql instances patch later — edition/tier changes restart the instance.
- Memory is the primary lever: reads are served from cache until the working set (tables + indexes) outgrows RAM, then p50 degrades to disk latency.
- Once the dataset can't fit RAM on any tier, switch to
--edition=enterprise-pluswith--tier=db-perf-optimized-N-<vCPU>. The tool enables its local-SSD data cache, which extends the effective cache several times beyond RAM: reads that would miss to persistent disk are served from local SSD at a fraction of the latency.
Storage
Set with --storage-size at create time; it only grows afterwards.
- Persistent-disk IOPS and throughput scale with provisioned size — the disk is also the I/O knob. Pre-size to ~2× the expected dataset (records + indexes + WAL + bloat) rather than relying on auto-resize, which grows in small steps and stalls under bulk loads.
Connection sizing for a target throughput
Set the shared read/write pool limit with ATE_API_POSTGRES_POOL_MAX_CONNS at
deploy time. Substrate and the embedded OpenFGA server use the same pool.
Target active connections equal throughput multiplied by average query
latency:
connections ≈ QPS × mean latency in seconds
≈ 10,000 req/s × 0.006 s ≈ 60 active connections
Provision ~2× headroom for bursts. This setting does not affect the owner and
watch pools; their fixed limits add up to 5 connections per replica beyond the
read/write pool limit (e.g. 4 replicas with
ATE_API_POSTGRES_POOL_MAX_CONNS=32 can open up to
4 × (32 + 2 + 3) = 148 connections).
An undersized pool causes client-side queuing inside pgx rather than database errors.
Ensure total connections across all replicas stay within Cloud SQL's limit:
replicas × (pool_max_conns + 5) ≤ max_connections − slack (superuser, maintenance)
Exceeding
max_connections
does error (FATAL: sorry, too many clients already); raise the flag with gcloud sql instances patch --database-flags=…, remembering the list replaces all flags, so always
re-include cloudsql.iam_authentication=on. Going far beyond ~2× vCPUs in
active connections buys no throughput either — backends are OS processes,
and excess active ones just context-switch. Sweep around the formula's
number rather than maximizing.
Proxy sidecar resources
The proxy imposes no connection limit and adds
sub-millisecond latency, but it encrypts all database traffic, so its CPU
use scales with throughput. The patch
(manifests/ate-install/cloudsql/proxy-sidecar-patch.yaml) requests 100m — sized
for control-plane traffic. For sustained thousands of ops/s, raise the
sidecar's CPU request so node pressure cannot throttle it into becoming the
bottleneck. Connection churn has a separate ceiling: IAM database logins
are quota'd at 12,000/min per instance — irrelevant for steady pools, but a
simultaneous reconnect storm across very many replicas can brush it.
Managed Connection Pooling
Managed Connection Pooling
(Enterprise Plus only) is a server-side pooler
(gcloud sql instances patch <instance> --enable-connection-pooling) that
multiplexes up to max_client_connections (default 5,000) client
connections onto at most max_pool_size (default 50) backends per
database+user pair — the fix when very many ateapi replicas would otherwise
need thousands of real backends. Not usable with atepg's efficient
transaction mode today: the worker-watch path uses LISTEN, which
transaction pooling doesn't support (session mode works but forfeits most
of the multiplexing). If enabled, size max_pool_size to the same
formula's number and reserve ~15 server connections per vCPU in
max_connections for the pooler.
Beyond configuration
At billions of rows per table, vacuum duration and index maintenance on monolithic tables become the operational limit — partitioning the large tables is schema work, not a configuration change.