Files
substrate/cmd/atecontroller
Michael Aspinwall 510761bfc7 Implement endpoint slice resolution for ate api connections for HA (#582)
Fixes #565

Replaces standard DNS resolution (`dns:///`) and the forced 1-minute
`MaxConnectionAge` reconnect workaround with a native, event-driven
Kubernetes gRPC resolver (`internal/k8sresolver`). The resolver uses
client-go's `SharedInformerFactory` and `EndpointSliceLister` to watch
`discoveryv1.EndpointSlice` resources in real time.

### Verification
Created a kind cluster to test changes by scaling the deployments up and
down and verified the new servers were connected to by clients.

#### 1. Kind Cluster Scale-Out & EndpointSlice Discovery
  Click to view kubectl scale and EndpointSlice output
    $ kubectl scale deployment/ate-api-server -n ate-system --replicas=4
    deployment.apps/ate-api-server scaled

    $ kubectl get pods -n ate-system -l app=ate-api-server
    NAME                              READY   STATUS    RESTARTS   AGE
    ate-api-server-6bcd9fddbb-gxxth   1/1     Running   0          3m8s
    ate-api-server-6bcd9fddbb-jgwbw   1/1     Running   0          3m8s
    ate-api-server-6bcd9fddbb-m8nbx   1/1     Running   0          7m20s
    ate-api-server-6bcd9fddbb-vwm2g   1/1     Running   0          7m20s

$ kubectl get endpointslices -n ate-system -l
kubernetes.io/service-name=api
    NAME        ADDRESSTYPE   PORTS   ENDPOINTS
api-dmzfc IPv4 443 10.244.0.15,10.244.0.19,10.244.0.31,10.244.0.32

#### 2. Proof of Active gRPC Load-Balancing Across All Replicas
```
    [pod/ate-api-server-6bcd9fddbb-gxxth] {"time":"2026-07-29T13:20:49Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
    [pod/ate-api-server-6bcd9fddbb-jgwbw] {"time":"2026-07-29T13:20:46Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
    [pod/ate-api-server-6bcd9fddbb-m8nbx] {"time":"2026-07-29T13:20:48Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
    [pod/ate-api-server-6bcd9fddbb-vwm2g] {"time":"2026-07-29T13:20:47Z","level":"INFO","msg":"Handle RPC","method":"/ateapi.Control/ListActors","principal":{"ID":"spiffe://cluster.local/ns/ate-system/sa/atenet-router"}}
```
2026-08-10 16:47:12 -04:00
..