mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-10 10:13:50 +08:00
Jeremy Alvis
41c5f01968
Fixes #997 ## Summary Give `ateapi` separate PostgreSQL connections for schema ownership and normal application access. Migrations and outbox partition maintenance use the owner connection, allowing application reads and writes to run with restricted permissions. - Each connection assumes its configured role with `SET ROLE`, keeping object ownership and grants separate from login credentials. - Configure owner and read/write connections through `ate-setup` and the installation manifests. - Default the schema to `substrate` and document provisioning for external PostgreSQL and Cloud SQL, including one IAM login with membership in both roles. - Keep bundled PostgreSQL using its existing `postgres` account and role for both pools. Restricted runtime access requires operator-provisioned roles. See `docs/postgres.md` for configuration details. ## Scope This is the base PR for two separate follow-ups: [file-based credentials](https://github.com/agent-substrate/substrate/pull/1979) and [automatic development database bootstrap](https://github.com/agent-substrate/substrate/pull/2025). Neither feature is included here. Operators provision external database roles, login memberships, schema ownership, and grants before startup. Table migrations still run automatically through the owner connection. The documented external database layout assumes a fresh database; this PR does not migrate existing table ownership or grants. Both connections remain inside `ateapi`, so this separates database permissions without isolating owner credentials from the application process. ## Validation - `env -u NO_COLOR make verify` passed. - PostgreSQL integration tests cover restricted runtime access, owner migrations and partition maintenance, and OpenFGA table permissions. - Installer tests cover bundled PostgreSQL, separate external logins, and Cloud SQL configuration.
102 lines
4.8 KiB
Markdown
102 lines
4.8 KiB
Markdown
# Scaling the Cloud SQL store
|
||
|
||
Sizing guidance for running the ateapi PostgreSQL store
|
||
([tools/setup-gcp/cloud-sql.md](../../tools/setup-gcp/cloud-sql.md)) at large
|
||
actor counts and high request rates.
|
||
|
||
The provisioning defaults (`db-custom-2-8192`, 10 GB disk) suit development
|
||
and modest fleets. At large actor counts the store becomes I/O-bound: once
|
||
tables and indexes outgrow memory, uniform random reads fall out of cache and
|
||
point lookups pay persistent-disk latency (several milliseconds) instead of
|
||
microseconds. The knobs below address that, in order of leverage.
|
||
|
||
## Instance shape
|
||
|
||
Set with `--tier` / `--edition` at create time, or `gcloud sql instances
|
||
patch` later — edition/tier changes restart the instance.
|
||
|
||
- Memory is the primary lever: reads are served from cache until the working
|
||
set (tables + indexes) outgrows RAM, then p50 degrades to disk latency.
|
||
- Once the dataset can't fit RAM on any tier, switch to
|
||
[`--edition=enterprise-plus`](https://docs.cloud.google.com/sql/docs/postgres/editions-intro)
|
||
with `--tier=db-perf-optimized-N-<vCPU>`. The tool
|
||
enables its **local-SSD data cache**, which extends the effective cache
|
||
several times beyond RAM: reads that would miss to persistent disk are
|
||
served from local SSD at a fraction of the latency.
|
||
|
||
## Storage
|
||
|
||
Set with `--storage-size` at create time; it only grows afterwards.
|
||
|
||
- Persistent-disk IOPS and throughput scale with provisioned size — the disk
|
||
is also the I/O knob. Pre-size to ~2× the expected dataset (records +
|
||
indexes + WAL + bloat) rather than relying on auto-resize, which grows in
|
||
small steps and stalls under bulk loads.
|
||
|
||
## Connection sizing for a target throughput
|
||
|
||
Set the shared read/write pool limit with `ATE_API_POSTGRES_POOL_MAX_CONNS` at
|
||
deploy time. Substrate and the embedded OpenFGA server use the same pool.
|
||
Target active connections equal throughput multiplied by average query
|
||
latency:
|
||
|
||
```
|
||
connections ≈ QPS × mean latency in seconds
|
||
≈ 10,000 req/s × 0.006 s ≈ 60 active connections
|
||
```
|
||
|
||
Provision ~2× headroom for bursts. This setting does not affect the owner and
|
||
watch pools; their fixed limits add up to 5 connections per replica beyond the
|
||
read/write pool limit (e.g. 4 replicas with
|
||
`ATE_API_POSTGRES_POOL_MAX_CONNS=32` can open up to
|
||
4 × (32 + 2 + 3) = 148 connections).
|
||
An undersized pool causes client-side queuing inside `pgx` rather than database errors.
|
||
Ensure total connections across all replicas stay within Cloud SQL's limit:
|
||
|
||
```
|
||
replicas × (pool_max_conns + 5) ≤ max_connections − slack (superuser, maintenance)
|
||
```
|
||
|
||
Exceeding
|
||
[`max_connections`](https://docs.cloud.google.com/sql/docs/postgres/quotas#maximum_concurrent_connections)
|
||
does error (`FATAL: sorry, too many clients
|
||
already`); raise the flag with `gcloud sql instances patch
|
||
--database-flags=…`, remembering the list **replaces all flags**, so always
|
||
re-include `cloudsql.iam_authentication=on`. Going far beyond ~2× vCPUs in
|
||
*active* connections buys no throughput either — backends are OS processes,
|
||
and excess active ones just context-switch. Sweep around the formula's
|
||
number rather than maximizing.
|
||
|
||
## Proxy sidecar resources
|
||
|
||
The proxy imposes no connection limit and adds
|
||
sub-millisecond latency, but it encrypts all database traffic, so its CPU
|
||
use scales with throughput. The patch
|
||
(`manifests/ate-install/cloudsql/proxy-sidecar-patch.yaml`) requests `100m` — sized
|
||
for control-plane traffic. For sustained thousands of ops/s, raise the
|
||
sidecar's CPU request so node pressure cannot throttle it into becoming the
|
||
bottleneck. Connection *churn* has a separate ceiling: IAM database logins
|
||
are quota'd at 12,000/min per instance — irrelevant for steady pools, but a
|
||
simultaneous reconnect storm across very many replicas can brush it.
|
||
|
||
## Managed Connection Pooling
|
||
|
||
[Managed Connection Pooling](https://docs.cloud.google.com/sql/docs/postgres/managed-connection-pooling)
|
||
(Enterprise Plus only) is a server-side pooler
|
||
(`gcloud sql instances patch <instance> --enable-connection-pooling`) that
|
||
multiplexes up to `max_client_connections` (default 5,000) client
|
||
connections onto at most `max_pool_size` (default 50) backends per
|
||
database+user pair — the fix when very many ateapi replicas would otherwise
|
||
need thousands of real backends. Not usable with atepg's efficient
|
||
`transaction` mode today: the worker-watch path uses `LISTEN`, which
|
||
transaction pooling doesn't support (`session` mode works but forfeits most
|
||
of the multiplexing). If enabled, size `max_pool_size` to the same
|
||
formula's number and reserve ~15 server connections per vCPU in
|
||
`max_connections` for the pooler.
|
||
|
||
## Beyond configuration
|
||
|
||
At billions of rows per table, vacuum duration and index maintenance on
|
||
monolithic tables become the operational limit — partitioning the large
|
||
tables is schema work, not a configuration change.
|