mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-12 10:35:06 +08:00
shrutiyam-glitch
137b6fd3df
Fixes #920 partially. Replaces per-write pg_notify with a worker_changes outbox table written in the same transaction, and LISTEN with a 100ms polling watcher. #### Motivation * **Scalability Bottleneck**: `pg_notify` serializes the commits of all notifying transactions through a global lock held across the commit (including `fsync`). This artificially caps worker writes at ~600/s on Cloud SQL regardless of instance size, whereas our target is O(10K) worker updates/s. * **Payload Limits**: Bypasses the 8KB `NOTIFY` payload limit that previously caused writes to fail. * **Reliability**: A cursor-based polling watcher survives reconnects and failovers without missing events, which was a known flaw with the ephemeral `LISTEN` approach. *(Known Postgres pathology prior art: [Recall.ai](https://www.recall.ai/blog/postgres-listen-notify-does-not-scale), [DBOS](https://www.dbos.dev/blog/postgres-listen-notify-scalability)).* ### Performance Improvement WorkerUpdate @ 1,000 QPS , 1M workers (preloaded) — before vs after the change feed: | | p50 | p90 | p95 | p99 | |---|---|---|---|---| | Before (per-update pg_notify) | 40.3s |55.6s | 61.2s | 63.8s | | After (change-feed table) | 7.08 ms | 8.04 ms | 8.52 ms | 27.5 ms | #### Changes Made * **Schema**: Added transactional outbox table `worker_outbox`. * **Write Path**: Worker writes now append to the `worker_outbox` feed inside the same transaction instead of calling `pg_notify()`. * **Watch Path**: Replaced `LISTEN` in `WatchWorkers` with a polling watcher that queries the feed every 50ms. * **Cleanup**: Implemented a janitor process during polling to periodically delete old feed rows. * **Tests**: Updated atomicity tests to verify feed inserts instead of `pg_notify` payloads. For full architecture: https://docs.google.com/document/d/10K0wB6aTeFkJCL4HN3NbLJCdFGoLYdhIcqkFnqFHkKc/edit?tab=t.txqcjwvmhp3v - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR