trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-21 06:11:31 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
pull-request/2784
OpenShell/.github/workflows/branch-e2e.yml
T
Taylor MutchDrew NewberrydiveshDivesh Chowdary
 0a770d9173 feat(kubernetes): support HA gateway rebalancing (#1868) 
* feat(kubernetes): support HA gateway rebalancing

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* perf(server): cache peer connections, tokens, and owner lookups

Every forwarded relay rebuilt its setup from scratch: an owner lookup, a
blocking read of the peer token, a TLS connect to the owning replica, and
a TokenReview plus Pod GET on the receiving side. Sandbox service routing
does this per HTTP request, so the apiserver calls scaled with traffic.

Cache all of it on ServerState:

- peer channels pooled per endpoint, so relays multiplex over one
  connection instead of redialing
- peer tokens keyed by SHA-256, expiring at min(ttl, token exp) so a hit
  cannot accept an expired token
- owner records for 3s against a 45s ownership TTL, still freshness
  checked before use

Entries are evicted when a relay fails. Also raise HTTP/2
max_concurrent_streams to 1024, since pooling funnels every relay between
two replicas onto one connection and hyper's default of 200 sits below
the 256 pending-relay budget.

Signed-off-by: divesh <dgude@nvidia.com>

* perf(server): pool upstream connections for sandbox services

Each HTTP request to a sandbox service opened its own supervisor relay,
paying a new TCP connection and HTTP/1 handshake every time. Worse, it
counted against the 32 in-flight relay cap, so a service handling more
than 32 concurrent requests failed outright.

Pool idle upstreams per endpoint and port, up to 8 each for 15s. Reuse is
safe because the pool only returns a connection hyper reports as ready,
and HTTP/1 cannot start a request until the previous body has drained.
Upgrades are never pooled since they take the connection over, and a
failed send evicts that endpoint. Pruning is bounded per key, with the
full sweep limited to once per 30s.

Signed-off-by: divesh <dgude@nvidia.com>

* fix(server): address HA gateway review findings (#3449)

- Let a gateway own supervisor sessions without a peer endpoint. Requiring
  one whenever the store is PostgreSQL broke every single-instance
  PostgreSQL deployment, because no sandbox supervisor could connect.
  A cross-replica request to an owner that advertises no endpoint now fails
  immediately naming the cause, instead of retrying until the wait timeout.
- Close a supervisor session on heartbeat only when another replica owns it,
  or after renewals fail for the ownership TTL. A database error no longer
  drops every session heartbeating during an outage.
- Clamp owner record ages at zero so a skewed or corrupt stored timestamp
  cannot produce a negative age.
- Bound the cross-object advisory lock with a lock timeout, so a stuck holder
  fails instead of blocking every mutation in the fleet.
- Refuse to start when a peer endpoint is configured on a multi-replica
  backend but peer authentication is unavailable, and warn when a
  multi-replica backend has no peer endpoint at all.
- Reject a plaintext peer endpoint when the gateway serves TLS.
- Skip the sandbox watch poller on single-replica backends, where the local
  update bus already sees every write.
- Rate-limit the peer owner cache sweep so an insert no longer scans the
  whole map under the lock.
- Retry GET and HEAD on a pooled upstream the sandbox closed, instead of
  returning 502, and drop an emptied endpoint from the pool right away.
- Document the gateway peer environment variables and the post-rollout
  ownership skew operators should expect.

Signed-off-by: divesh <dgude@nvidia.com>

* fix(server): harden HA supervisor ownership

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: divesh <dgude@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: divesh <dgude@nvidia.com>
Co-authored-by: Divesh Chowdary <47188680+FrostGod@users.noreply.github.com>
2026-09-21 21:22:12 +00:00

511 lines
19 KiB
YAML
Raw Permalink Blame History

name: Branch E2E Checks
on:
merge_group:
types: [checks_requested]
push:
branches:
- "pull-request/[0-9]+"
workflow_dispatch: {}
permissions: {}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
pr_metadata:
name: Resolve PR metadata
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: read
outputs:
should_run: ${{ steps.gate.outputs.should_run }}
run_integration: ${{ steps.labels.outputs.run_core_e2e }}
run_core_e2e: ${{ steps.labels.outputs.run_core_e2e }}
run_gpu_e2e: ${{ steps.labels.outputs.run_gpu_e2e }}
run_kubernetes_ha_e2e: ${{ steps.labels.outputs.run_kubernetes_ha_e2e }}
run_kubernetes_credential_drivers_e2e: ${{ steps.labels.outputs.run_kubernetes_credential_drivers_e2e }}
run_any_e2e: ${{ steps.labels.outputs.run_any_e2e }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- id: gate
uses: ./.github/actions/pr-gate
- id: labels
if: steps.gate.outputs.should_run == 'true'
env:
EVENT_NAME: ${{ github.event_name }}
LABELS_JSON: ${{ steps.gate.outputs.labels_json }}
shell: bash
run: |
set -euo pipefail
case "$EVENT_NAME" in
push)
run_core_e2e="$(jq -r 'index("test:e2e") != null' <<< "$LABELS_JSON")"
run_gpu_e2e="$(jq -r 'index("test:e2e-gpu") != null' <<< "$LABELS_JSON")"
run_kubernetes_ha_e2e=false
run_kubernetes_credential_drivers_e2e=false
;;
merge_group)
# Merge groups have no PR labels. When GPU E2E is required as documented
# in CI.md, skipping it leaves the gate pending until the queue times out
# and ejects the PR. HA stays off until stable.
run_core_e2e=true
run_gpu_e2e=true
run_kubernetes_ha_e2e=false
run_kubernetes_credential_drivers_e2e=false
;;
*)
run_core_e2e=true
run_gpu_e2e=true
run_kubernetes_ha_e2e=false
run_kubernetes_credential_drivers_e2e=false
;;
esac
if [ "$run_core_e2e" = "true" ] || [ "$run_gpu_e2e" = "true" ] || [ "$run_kubernetes_ha_e2e" = "true" ] || [ "$run_kubernetes_credential_drivers_e2e" = "true" ]; then
run_any_e2e=true
else
run_any_e2e=false
fi
{
echo "run_core_e2e=$run_core_e2e"
echo "run_gpu_e2e=$run_gpu_e2e"
echo "run_kubernetes_ha_e2e=$run_kubernetes_ha_e2e"
echo "run_kubernetes_credential_drivers_e2e=$run_kubernetes_credential_drivers_e2e"
echo "run_any_e2e=$run_any_e2e"
} >> "$GITHUB_OUTPUT"
version:
needs: [pr_metadata]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true'
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
cargo: ${{ steps.version.outputs.cargo }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Compute version
id: version
run: |
cargo="$(python3 tasks/scripts/release.py get-version --cargo)"
echo "cargo=$cargo" >> "$GITHUB_OUTPUT"
build-binaries:
needs: version
permissions:
contents: read
uses: ./.github/workflows/build-binaries.yml
with:
cargo-version: ${{ needs.version.outputs.cargo }}
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
build-gateway-plain:
name: openshell-gateway driver-free (${{ matrix.triple }})
needs: [pr_metadata, version]
if: needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
contents: read
strategy:
matrix:
include:
- triple: x86_64-unknown-linux-gnu
runner: linux-amd64-cpu8
- triple: aarch64-unknown-linux-gnu
runner: linux-arm64-cpu8
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: ${{ github.sha }}
- uses: ./.github/actions/setup-nix
with:
cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }}
- uses: ./.github/actions/build-rust-binary
with:
package: openshell-gateway
binary: openshell-gateway
artifact-name: openshell-gateway-plain-${{ matrix.triple }}
triple: ${{ matrix.triple }}
cargo-version: ${{ needs.version.outputs.cargo }}
extra-cargo-flags: --no-default-features
build-external-drivers:
name: ${{ matrix.binary }} (${{ matrix.triple }})
needs: [pr_metadata, version]
if: needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
contents: read
strategy:
matrix:
include:
- package: openshell-driver-docker
binary: openshell-driver-docker
triple: aarch64-unknown-linux-gnu
runner: linux-arm64-cpu8
- package: openshell-driver-podman
binary: openshell-driver-podman
triple: x86_64-unknown-linux-gnu
runner: linux-amd64-cpu8
- package: openshell-driver-kubernetes
binary: openshell-driver-kubernetes
triple: x86_64-unknown-linux-gnu
runner: linux-amd64-cpu8
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: ${{ github.sha }}
- uses: ./.github/actions/setup-nix
with:
cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }}
- uses: ./.github/actions/build-rust-binary
with:
package: ${{ matrix.package }}
binary: ${{ matrix.binary }}
triple: ${{ matrix.triple }}
cargo-version: ${{ needs.version.outputs.cargo }}
build-vm-driver:
needs: [pr_metadata, version, build-binaries]
if: needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
contents: read
uses: ./.github/workflows/build-vm-driver.yml
with:
cargo-version: ${{ needs.version.outputs.cargo }}
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
build-images:
needs: [pr_metadata, build-binaries]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true'
permissions:
contents: read
packages: write
uses: ./.github/workflows/build-images.yml
prepare-integration:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.run_integration == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/prepare-integration-inputs.yml
# Run driver-independent conformance tests.
conformance-integration:
needs: prepare-integration
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
with:
category: conformance
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"conformance"},
{"environment":"fedora-podman-rootful","installer":"binaries","testsuite":"conformance"},
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"conformance"}
]
# Run feature-specific integration tests:
feature-specific-integration:
needs: prepare-integration
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
with:
category: feature-specific
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"fedora-podman-rootful","installer":"binaries","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"provider-refresh"}
]
# Run driver-specific integration tests:
# TODO: This should be added as soon as we have driver-specific tests enabled in tmachine.
# driver-specific:
# needs: prepare-integration
# permissions:
# actions: read
# contents: read
# packages: read
# uses: ./.github/workflows/integration-runner.yml
# with:
# category: driver-specific
# source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
# integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
# test-matrix: >-
# [
# ]
docker-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-docker-test.yml
with:
image-tag: ${{ github.sha }}
runner: linux-arm64-cpu8
conformance-artifact-prefix: openshell-conformance
vm-e2e:
needs: [pr_metadata, build-binaries, build-vm-driver]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-vm-test.yml
with:
conformance-artifact-prefix: openshell-conformance
docker-external-driver-e2e:
needs: [pr_metadata, build-binaries, build-gateway-plain, build-external-drivers, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-docker-test.yml
with:
image-tag: ${{ github.sha }}
runner: linux-arm64-cpu8
gateway-artifact: openshell-gateway-plain-aarch64-unknown-linux-gnu
external-driver-binary: openshell-driver-docker
conformance-artifact-prefix: openshell-conformance
suite-matrix: >-
[{"suite":"external-driver","cmd":"mise run --no-deps --skip-deps e2e:docker:external-driver","apt_packages":"openssh-client","python_proto":false,"mcp":false}]
podman-external-driver-e2e:
needs: [pr_metadata, build-binaries, build-gateway-plain, build-external-drivers, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-podman-test.yml
with:
image-tag: ${{ github.sha }}
gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu
external-driver-binary: openshell-driver-podman
conformance-artifact-prefix: openshell-conformance
suite-matrix: >-
[{"suite":"external-driver","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:external-driver"}]
vm-external-driver-e2e:
needs: [pr_metadata, build-binaries, build-gateway-plain, build-vm-driver]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-vm-test.yml
with:
gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu
suite-name: external-driver
e2e-task: e2e:vm:external-driver
conformance-artifact-prefix: openshell-conformance
gpu-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_gpu_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-gpu-test.yaml
with:
image-tag: ${{ github.sha }}
conformance-artifact-prefix: openshell-conformance
kubernetes-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
strategy:
fail-fast: false
matrix:
include:
- agent_sandbox_api: v1beta1
agent_sandbox_version: v0.5.0
- agent_sandbox_api: v1alpha1
agent_sandbox_version: v0.4.6
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (Rust smoke, Agent Sandbox ${{ matrix.agent_sandbox_api }})
agent-sandbox-version: ${{ matrix.agent_sandbox_version }}
conformance-artifact-prefix: openshell-conformance
kubernetes-workspace-managed-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (workspace managed mode)
e2e-task: e2e:kubernetes:workspace-managed
conformance-artifact-prefix: openshell-conformance
kubernetes-external-driver-e2e:
needs: [pr_metadata, build-binaries, build-gateway-plain, build-external-drivers, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (external compute driver)
e2e-task: e2e:kubernetes:external-driver
gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu
external-driver-binary: openshell-driver-kubernetes
cluster-images: sandbox supervisor
conformance-artifact-prefix: openshell-conformance
kubernetes-workspace-operator-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (workspace operator mode)
e2e-task: e2e:kubernetes:workspace-operator
conformance-artifact-prefix: openshell-conformance
kubernetes-ha-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes HA E2E (Rust smoke)
extra-helm-values: deploy/helm/openshell/ci/values-high-availability.yaml
external-postgres-secret: openshell-ha-pg
test-name: kubernetes_ha_rebalancing
kubernetes-features: e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-ha
use-envoy-gateway: true
conformance-artifact-prefix: openshell-conformance
kubernetes-credential-drivers-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_credential_drivers_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes Credential Drivers E2E
e2e-task: e2e:kubernetes:credential-drivers
conformance-artifact-prefix: openshell-conformance
core-e2e-result:
name: Core E2E result
needs:
- pr_metadata
- conformance-integration
- feature-specific-integration
- docker-e2e
- vm-e2e
- docker-external-driver-e2e
- podman-external-driver-e2e
- vm-external-driver-e2e
- kubernetes-e2e
- kubernetes-external-driver-e2e
- kubernetes-workspace-managed-e2e
- kubernetes-workspace-operator-e2e
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/check-job-results
with:
results: ${{ toJSON(needs) }}
gpu-e2e-result:
name: GPU E2E result
needs: [pr_metadata, gpu-e2e]
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_gpu_e2e == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/check-job-results
with:
results: ${{ toJSON(needs) }}
kubernetes-ha-e2e-result:
name: Kubernetes HA E2E result
needs: [pr_metadata, kubernetes-ha-e2e]
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/check-job-results
with:
results: ${{ toJSON(needs) }}
kubernetes-credential-drivers-e2e-result:
name: Kubernetes Credential Drivers E2E result
needs: [pr_metadata, kubernetes-credential-drivers-e2e]
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_credential_drivers_e2e == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/check-job-results
with:
results: ${{ toJSON(needs) }}
Reference in New Issue View Git Blame Copy Permalink