mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-30 07:33:59 +08:00
0a770d9173
* feat(kubernetes): support HA gateway rebalancing Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(server): cache peer connections, tokens, and owner lookups Every forwarded relay rebuilt its setup from scratch: an owner lookup, a blocking read of the peer token, a TLS connect to the owning replica, and a TokenReview plus Pod GET on the receiving side. Sandbox service routing does this per HTTP request, so the apiserver calls scaled with traffic. Cache all of it on ServerState: - peer channels pooled per endpoint, so relays multiplex over one connection instead of redialing - peer tokens keyed by SHA-256, expiring at min(ttl, token exp) so a hit cannot accept an expired token - owner records for 3s against a 45s ownership TTL, still freshness checked before use Entries are evicted when a relay fails. Also raise HTTP/2 max_concurrent_streams to 1024, since pooling funnels every relay between two replicas onto one connection and hyper's default of 200 sits below the 256 pending-relay budget. Signed-off-by: divesh <dgude@nvidia.com> * perf(server): pool upstream connections for sandbox services Each HTTP request to a sandbox service opened its own supervisor relay, paying a new TCP connection and HTTP/1 handshake every time. Worse, it counted against the 32 in-flight relay cap, so a service handling more than 32 concurrent requests failed outright. Pool idle upstreams per endpoint and port, up to 8 each for 15s. Reuse is safe because the pool only returns a connection hyper reports as ready, and HTTP/1 cannot start a request until the previous body has drained. Upgrades are never pooled since they take the connection over, and a failed send evicts that endpoint. Pruning is bounded per key, with the full sweep limited to once per 30s. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): address HA gateway review findings (#3449) - Let a gateway own supervisor sessions without a peer endpoint. Requiring one whenever the store is PostgreSQL broke every single-instance PostgreSQL deployment, because no sandbox supervisor could connect. A cross-replica request to an owner that advertises no endpoint now fails immediately naming the cause, instead of retrying until the wait timeout. - Close a supervisor session on heartbeat only when another replica owns it, or after renewals fail for the ownership TTL. A database error no longer drops every session heartbeating during an outage. - Clamp owner record ages at zero so a skewed or corrupt stored timestamp cannot produce a negative age. - Bound the cross-object advisory lock with a lock timeout, so a stuck holder fails instead of blocking every mutation in the fleet. - Refuse to start when a peer endpoint is configured on a multi-replica backend but peer authentication is unavailable, and warn when a multi-replica backend has no peer endpoint at all. - Reject a plaintext peer endpoint when the gateway serves TLS. - Skip the sandbox watch poller on single-replica backends, where the local update bus already sees every write. - Rate-limit the peer owner cache sweep so an insert no longer scans the whole map under the lock. - Retry GET and HEAD on a pooled upstream the sandbox closed, instead of returning 502, and drop an emptied endpoint from the pool right away. - Document the gateway peer environment variables and the post-rollout ownership skew operators should expect. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): harden HA supervisor ownership Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> Co-authored-by: Divesh Chowdary <47188680+FrostGod@users.noreply.github.com>
153 lines
5.4 KiB
YAML
153 lines
5.4 KiB
YAML
name: Kubernetes E2E Test
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
image-tag:
|
|
description: "Image tag to test (typically the commit SHA)"
|
|
required: true
|
|
type: string
|
|
runner:
|
|
description: "GitHub Actions runner label"
|
|
required: false
|
|
type: string
|
|
default: "linux-amd64-cpu8"
|
|
checkout-ref:
|
|
description: "Git ref to check out for test inputs (defaults to the workflow SHA)"
|
|
required: false
|
|
type: string
|
|
default: ""
|
|
job-name:
|
|
description: "Display name for the Kubernetes e2e job"
|
|
required: false
|
|
type: string
|
|
default: "Kubernetes E2E (Rust smoke)"
|
|
extra-helm-values:
|
|
description: "Colon-separated Helm values files to layer on the Kubernetes e2e chart install"
|
|
required: false
|
|
type: string
|
|
default: ""
|
|
external-postgres-secret:
|
|
description: "Create an ephemeral external PostgreSQL fixture and write its URI to this Secret"
|
|
required: false
|
|
type: string
|
|
default: ""
|
|
agent-sandbox-version:
|
|
description: "Agent Sandbox release to install before OpenShell"
|
|
required: false
|
|
type: string
|
|
default: "v0.5.0"
|
|
e2e-task:
|
|
description: "mise task to run for the Kubernetes e2e job"
|
|
required: false
|
|
type: string
|
|
default: "e2e:kubernetes"
|
|
test-name:
|
|
description: "Rust e2e test target to run (sets OPENSHELL_E2E_KUBE_TEST)"
|
|
required: false
|
|
type: string
|
|
default: ""
|
|
kubernetes-features:
|
|
description: "Cargo feature list for the Kubernetes e2e crate (uses script defaults when empty)"
|
|
required: false
|
|
type: string
|
|
default: ""
|
|
use-envoy-gateway:
|
|
description: "Route the test through Envoy Gateway instead of a direct gateway Service port-forward"
|
|
required: false
|
|
type: boolean
|
|
default: false
|
|
mise-version:
|
|
description: "mise version to install on the bare Kubernetes e2e runner"
|
|
required: false
|
|
type: string
|
|
default: "v2026.9.9"
|
|
gateway-artifact:
|
|
description: "Optional prebuilt gateway artifact used to compose a local test image"
|
|
required: false
|
|
type: string
|
|
default: ""
|
|
conformance-artifact-prefix:
|
|
description: "Optional prebuilt conformance CLI artifact prefix (artifact suffix is <triple>)"
|
|
required: false
|
|
type: string
|
|
default: ""
|
|
external-driver-binary:
|
|
description: "Optional standalone compute driver binary used to compose a local test image"
|
|
required: false
|
|
type: string
|
|
default: ""
|
|
cluster-images:
|
|
description: "Space-separated published images to preload into kind"
|
|
required: false
|
|
type: string
|
|
default: "gateway sandbox supervisor"
|
|
|
|
permissions:
|
|
actions: read
|
|
contents: read
|
|
packages: read
|
|
|
|
jobs:
|
|
e2e-kubernetes:
|
|
name: ${{ inputs.job-name }}
|
|
# Bare runner: running kind-in-container hits nested-Docker / kubeconfig
|
|
# complications. The runner has Docker; mise installs helm, kubectl, and
|
|
# the Rust toolchain.
|
|
runs-on: ${{ inputs.runner }}
|
|
timeout-minutes: 60
|
|
permissions:
|
|
actions: read
|
|
contents: read
|
|
packages: read
|
|
env:
|
|
MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
# Keep bare-runner installs aligned with the project CI image.
|
|
MISE_VERSION: ${{ inputs.mise-version }}
|
|
KIND_CLUSTER_NAME: kube-e2e-${{ github.run_id }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs['checkout-ref'] || github.sha }}
|
|
|
|
- name: Use prebuilt OpenShell CLI
|
|
uses: ./.github/actions/setup-e2e-cli
|
|
with:
|
|
conformance-artifact-prefix: ${{ inputs.conformance-artifact-prefix }}
|
|
|
|
- name: Use prebuilt OpenShell gateway
|
|
if: inputs.gateway-artifact != ''
|
|
uses: ./.github/actions/setup-e2e-gateway
|
|
with:
|
|
artifact-name: ${{ inputs.gateway-artifact }}
|
|
|
|
- name: Use prebuilt external compute driver
|
|
if: inputs.external-driver-binary != ''
|
|
uses: ./.github/actions/setup-e2e-driver
|
|
with:
|
|
binary: ${{ inputs.external-driver-binary }}
|
|
|
|
- name: Set up kind cluster
|
|
uses: ./.github/actions/setup-e2e-kind
|
|
with:
|
|
cluster-name: ${{ env.KIND_CLUSTER_NAME }}
|
|
image-tag: ${{ inputs.image-tag }}
|
|
images: ${{ inputs.cluster-images }}
|
|
registry-username: ${{ github.actor }}
|
|
registry-password: ${{ secrets.GITHUB_TOKEN }}
|
|
mise-version: ${{ inputs.mise-version }}
|
|
|
|
- name: Run Kubernetes E2E
|
|
env:
|
|
AGENT_SANDBOX_VERSION: ${{ inputs.agent-sandbox-version }}
|
|
OPENSHELL_E2E_KUBE_CONTEXT: kind-${{ env.KIND_CLUSTER_NAME }}
|
|
OPENSHELL_E2E_KUBE_EXTRA_VALUES: ${{ inputs.extra-helm-values }}
|
|
OPENSHELL_E2E_KUBE_EXTERNAL_POSTGRES_SECRET: ${{ inputs.external-postgres-secret }}
|
|
OPENSHELL_E2E_KUBE_TEST: ${{ inputs.test-name }}
|
|
OPENSHELL_E2E_KUBERNETES_FEATURES: ${{ inputs.kubernetes-features }}
|
|
OPENSHELL_E2E_KUBE_USE_ENVOY: ${{ inputs.use-envoy-gateway && '1' || '0' }}
|
|
IMAGE_TAG: ${{ inputs.image-tag }}
|
|
OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell
|
|
E2E_TASK: ${{ inputs.e2e-task }}
|
|
run: mise run --no-deps --skip-deps "$E2E_TASK"
|