mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-10 10:13:50 +08:00
shrutiyam-glitch
5f64ba4c26
### Which issue(s) this PR is related to: Fixes #721 Required for System Upgrade flow (#473) ### What this PR does / why we need it: This PR implements graceful termination and zero-outage rolling updates for `atenet-router` , building on the readiness probe and graceful drain patterns established in #719 (`atelet`). Draining a two-container networking pod (`atenet-router` Go control-plane + `envoy` C++ proxy dataplane) introduces complex lifecycle interdependencies. This PR resolves the dual-container SIGTERM race, respects Envoy's `failClosed` `ext_proc` filter dependency, preserves parked requests riding out worker pool saturation, and accelerates idle deployments via an event-driven file handshake. #### 1. Event-Driven Dataplane Synchronization (`emptyDir` Marker File) Envoy fast-exits by default on SIGTERM. To keep Envoy alive while the Go control-plane orchestrates the drain, we configured an IPC handshake between containers via a pod-shared `emptyDir` volume mounted at `/var/run/atenet`: * **Go Router Container:** Removes any stale marker at startup, then writes `/var/run/atenet/drain-complete` when its shutdown sequence finishes. * **Envoy Container `preStop` Hook:** Runs `while [ ! -f /var/run/atenet/drain-complete ]; do sleep 0.5; done`. **Outcome:** Envoy exits as soon as — the drain is done, rather than wasting time in a fixed worst-case sleep. If the router crashes, Kubelet terminates the `preStop` hook at `terminationGracePeriodSeconds: 60`—slower cleanup, never a wedge. #### 2. The Multi-Container Shutdown Sequence (`drain.go`) Because Kubernetes issues SIGTERM to both containers at once, a coordination state machine ensures Envoy never drops a connection and `ext_proc` is never stopped prematurely: | Phase / (best-case ex.) Timeline | atenet-router | envoy | K8s / Service Status | | :--- | :--- | :--- | :--- | | **SIGTERM Sent**<br>($t=0\text{s}$) | Catches SIGTERM<br>• Flips `/readyz` $\rightarrow$ `503`.<br>• Starts 13s `drain-delay`. | Enters `lifecycle.preStop` hook:<br>`while [ ! -f .../drain-complete ]; do sleep 0.5; done`<br>• Kubelet holds SIGTERM back. | Deployment will recreate the pod.<br>EndpointSlice controller begins dropping Old Pod IP. | | **Propagation**<br>($t=0\text{s}$ – $t=13\text{s}$) | Keeps serving normally.<br>• `ext_proc` continues unparking/routing requests. | Runs normally inside `preStop` loop.<br>• Serves active TCP connections. | Service endpoint removal completes.<br>No new connections arrive at Old Pod. | | **Envoy drain**<br>($t=13\text{s}$) | Issues Envoy admin API calls:<br>• `/healthcheck/fail`<br>• `/drain_listeners?graceful&skip_exit`<br>• Polls `/stats` for active downstream connections. | Begins graceful listener drain:<br>• GOAWAY / `Connection: close` on established connections.<br>• In-flight requests keep running. | In-flight HTTP requests finish executing through Envoy. | | **ext_proc drain**<br>($t\approx18\text{s}$) | Active Envoy connections hit 0 (the poll exits early).<br>• Calls `extproc.GracefulStop()`.<br>• Parked request streams finish. | All downstream connections closed.<br>• Still waiting in the `preStop` loop. | All client HTTP responses delivered. | | **Handshake & Exit**<br>($t=18.5\text{s}$)| `ext_proc` drain finishes.<br>• Writes `drain-complete` marker.<br>• Hard-stops xDS (`Stop()`) & exits. | `preStop` loop detects marker file!<br>• `preStop` exits 0.<br>• Kubelet sends SIGTERM to Envoy.<br>• Envoy exits immediately. | Old Pod deleted cleanly at ~18.5s (well under the 60s budget). | Envoy ref: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining #### Testing & Verification 1. Unit Tests (`drain_test.go`): Added comprehensive unit tests covering: - Graceful completion of in-flight `ext_proc` streams within deadline. - Force-stopping straggler streams past `--drain-timeout`. - Envoy admin API interaction and connection polling. - Stale marker cleanup and marker file creation. 2. Local testing on kind cluster (Observations): Test script: `hack/verify-atenet-drain.sh` - Steady state: `/readyz=200`, `/healthz=200`; the instant the pod turned Terminating: `/readyz=503` while `/healthz=200` — `NotReady` but alive, for the whole drain. - The parked request survived the shutdown: fired at a busy 1-worker pool → parked → pod deleted while parked → worker freed → HTTP=200, total=1.89s, body hello from: 169.254.17.2 | preserved memory count: 1 | preserved file counter: 1 — park → resume → route, served by the Terminating pod during its drain-delay window. - Pod terminated 14s after deletion — the idle floor, confirmed across three runs: >13s drain-delay (sequence ran), ≪60s grace (marker handshake released Envoy's preStop; no `SIGKILL`). - Log sequence captured verbatim, drain-delay honored to the millisecond (17:34:41.706 → 17:34:54.707): >Shutdown signal received; draining >(+13.000s) Draining Envoy {window: 15s} >Envoy drained >Starting ext_proc drain >ext_proc drain completed within deadline >Drain-complete marker written {path: /var/run/atenet/drain-complete} >Shutdown complete - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR