mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-02 03:24:42 +08:00
Maya Wang
4dbc4e6b77
Partially addresses #1394 — the "spend the retry budget it configures for itself" half of that issue's expected behavior. Candidate fix 4 from the issue, done first because it is manifest-only. ## The problem, in one line `ate-api-server` configures itself a 60s store-connect budget and is killed by its own liveness probe at ~30s, halfway through it. ## Why the kill happens `:9090` is bound near the end of boot — after `connectStore` and, since #1196, after the goose migrations — so nothing answers `/healthz` for the whole of that window. The liveness probe was the only probe governing it: `initialDelaySeconds: 10` then probes at 10s, 20s and 30s, so the default `failureThreshold: 3` is met at ~30s. The events show `connect: connection refused`, not a probe returning failure — the port is closed, the process is fine. The kill then makes it worse than a plain restart. `connectStore(shutdownCtx)` roots the retry loop in the signal context, so SIGTERM cancels it mid-budget: the loop reports `context canceled` at attempt 15 of 30 rather than exhausting the budget, `serverboot.Fatal` exits, and CrashLoopBackOff holds the pod down well past the point where the store came back. A *running* replica rides out the same outage with `restarts=0` and serves again ~15s after the store returns. A booting replica was strictly less resilient than a running one, which is backwards. ## The change One `startupProbe`, 24 × 5s. Liveness is suspended until `/healthz` answers, so boot gets ~115s before the kubelet intervenes, against a ~60s connect budget. | | before | after | |---|---|---| | kubelet kills at | ~30s | ~115s | | connect attempts used | 15 of 30 | 30 of 30 | | failure reported as | `context canceled` | `connect to PostgreSQL after 30 attempts: …` | A dead process is still restarted, just on a budget sized for boot rather than one that never was. And because a startup probe also suspends the *readiness* probe, the pod now stays out of Service endpoints while booting instead of being latched Ready by the zero-value `Readiness` — a small move in the right direction for #1395 rather than against it, which is why this rather than simply raising `failureThreshold`. ## What this deliberately does not do - **It does not fix #1394.** The process still `Fatal`s once its own 60s budget is gone. The motivating scenario there is an ungraceful node loss, where GKE's PD force-detach alone is ~6 minutes; this raises tolerance from 30s to 60s, and only "retry for the life of the process, report through readiness" survives six minutes. That fix needs the reversible readiness predicate #1395 describes. - **It does not bound the migrations.** #1196 put goose on the boot critical path behind a `pg_advisory_xact_lock`, so with two replicas restarting together one waits on the other — inside the window where nothing is listening. On an empty schema that is milliseconds; on a real database during an upgrade it is not bounded by anything, and no probe budget can bound it. Ordering can. - **It does not change the boot ordering.** A slow boot and a dead process are still indistinguishable while the port is closed. Moving the health surface ahead of `connectStore` is candidate fix 1 and the natural follow-up. ## Testing `hack/verify-all.sh` passes (`metrics.sh` skipped locally — needs weaver or docker; no metrics change here). Note for anyone rebasing: `make test` currently fails on `TestExternalVolumeRenders` in `cmd/ate-setup/internal/demos/counter` on a clean `cbae8250`, unrelated to this change.