Files
OpenShell/.github/workflows/branch-e2e.yml
Polite_realismandDrew Newberry d376c90755 test(podman): close rootful userns, resource-limit, and daemon-failure CI gaps (#3690)
* test(podman): run driver-podman userns suite against rootful Podman too

The driver-specific-integration job only ran the driver-podman testsuite
(default/auto/keep-id/private userns reference checks) against
fedora-podman-rootless, leaving rootful behavior for this scenario
unverified even though the compute driver auto-detects and explicitly
supports rootful Podman.

The default-userns-baseline and userns-profile playbooks hard-asserted a
rootless tmachine gateway user, so pointing them at a rootful environment
would have failed that assertion immediately rather than exercising
anything. They now detect rootful vs. rootless via the existing
tmachine_container_runtime role and branch the reference-capture user
accordingly, while keeping the captured reference file itself owned by
tmachine, since the archived test binary that reads it back always runs
unprivileged as tmachine regardless of daemon mode.

Signed-off-by: politerealism <burdcat17@gmail.com>

* test(podman): add real-daemon coverage for resource limits and daemon failure

Neither the Podman driver's resource-limit enforcement nor its behavior
when the Podman daemon is unreachable had any test coverage against a
real daemon; both were only exercised through unit tests against a
mocked Podman client.

podman_resource_limits.rs creates a sandbox with --cpu/--memory flags and
reads /sys/fs/cgroup/memory.max and cpu.max from inside the sandbox
itself, verifying the limit is actually enforced rather than just echoed
back by the template API. Expected values are cross-checked against the
driver's own parse_cpu_to_microseconds/parse_memory_to_bytes and against
a real local `podman run --cpus/--memory` container.

podman_preflight.rs spawns the standalone openshell-driver-podman binary
against a guaranteed-nonexistent Podman socket and asserts it exits
non-zero within its bounded retry window with an actionable error naming
the socket path, rather than hanging or failing silently.

Signed-off-by: politerealism <burdcat17@gmail.com>

* test(podman): make rootful userns and cgroup checks pass

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(podman): match lifecycle containers by isolation role label

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(podman): accept non-expiring bootstrap tokens

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: politerealism <burdcat17@gmail.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
2026-09-24 20:16:29 -07:00

528 lines
20 KiB
YAML

name: Branch E2E Checks
on:
merge_group:
types: [checks_requested]
push:
branches:
- "pull-request/[0-9]+"
workflow_dispatch: {}
permissions: {}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
pr_metadata:
name: Resolve PR metadata
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: read
outputs:
should_run: ${{ steps.gate.outputs.should_run }}
run_integration: ${{ steps.labels.outputs.run_core_e2e }}
run_core_e2e: ${{ steps.labels.outputs.run_core_e2e }}
run_gpu_e2e: ${{ steps.labels.outputs.run_gpu_e2e }}
run_kubernetes_ha_e2e: ${{ steps.labels.outputs.run_kubernetes_ha_e2e }}
run_kubernetes_credential_drivers_e2e: ${{ steps.labels.outputs.run_kubernetes_credential_drivers_e2e }}
run_any_e2e: ${{ steps.labels.outputs.run_any_e2e }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- id: gate
uses: ./.github/actions/pr-gate
- id: labels
if: steps.gate.outputs.should_run == 'true'
env:
EVENT_NAME: ${{ github.event_name }}
LABELS_JSON: ${{ steps.gate.outputs.labels_json }}
shell: bash
run: |
set -euo pipefail
case "$EVENT_NAME" in
push)
run_core_e2e="$(jq -r 'index("test:e2e") != null' <<< "$LABELS_JSON")"
run_gpu_e2e="$(jq -r 'index("test:e2e-gpu") != null' <<< "$LABELS_JSON")"
run_kubernetes_ha_e2e="$(jq -r 'index("test:e2e-kubernetes") != null' <<< "$LABELS_JSON")"
run_kubernetes_credential_drivers_e2e="$(jq -r 'index("test:e2e-kubernetes") != null' <<< "$LABELS_JSON")"
;;
merge_group)
# Merge groups have no PR labels. When GPU E2E is required as documented
# in CI.md, skipping it leaves the gate pending until the queue times out
# and ejects the PR. HA stays off until stable.
run_core_e2e=true
run_gpu_e2e=true
run_kubernetes_ha_e2e=false
run_kubernetes_credential_drivers_e2e=false
;;
*)
run_core_e2e=true
run_gpu_e2e=true
run_kubernetes_ha_e2e=false
run_kubernetes_credential_drivers_e2e=false
;;
esac
if [ "$run_core_e2e" = "true" ] || [ "$run_gpu_e2e" = "true" ] || [ "$run_kubernetes_ha_e2e" = "true" ] || [ "$run_kubernetes_credential_drivers_e2e" = "true" ]; then
run_any_e2e=true
else
run_any_e2e=false
fi
{
echo "run_core_e2e=$run_core_e2e"
echo "run_gpu_e2e=$run_gpu_e2e"
echo "run_kubernetes_ha_e2e=$run_kubernetes_ha_e2e"
echo "run_kubernetes_credential_drivers_e2e=$run_kubernetes_credential_drivers_e2e"
echo "run_any_e2e=$run_any_e2e"
} >> "$GITHUB_OUTPUT"
version:
needs: [pr_metadata]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true'
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
cargo: ${{ steps.version.outputs.cargo }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Compute version
id: version
run: |
cargo="$(python3 tasks/scripts/release.py get-version --cargo)"
echo "cargo=$cargo" >> "$GITHUB_OUTPUT"
build-binaries:
needs: version
permissions:
contents: read
uses: ./.github/workflows/build-binaries.yml
with:
cargo-version: ${{ needs.version.outputs.cargo }}
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
build-gateway-plain:
name: openshell-gateway driver-free (${{ matrix.triple }})
needs: [pr_metadata, version]
if: needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
contents: read
strategy:
matrix:
include:
- triple: x86_64-unknown-linux-gnu
runner: linux-amd64-cpu8
- triple: aarch64-unknown-linux-gnu
runner: linux-arm64-cpu8
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: ${{ github.sha }}
- uses: ./.github/actions/setup-nix
with:
cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }}
- uses: ./.github/actions/build-rust-binary
with:
package: openshell-gateway
binary: openshell-gateway
artifact-name: openshell-gateway-plain-${{ matrix.triple }}
triple: ${{ matrix.triple }}
cargo-version: ${{ needs.version.outputs.cargo }}
extra-cargo-flags: --no-default-features
build-external-drivers:
name: ${{ matrix.binary }} (${{ matrix.triple }})
needs: [pr_metadata, version]
if: needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
contents: read
strategy:
matrix:
include:
- package: openshell-driver-docker
binary: openshell-driver-docker
triple: aarch64-unknown-linux-gnu
runner: linux-arm64-cpu8
- package: openshell-driver-podman
binary: openshell-driver-podman
triple: x86_64-unknown-linux-gnu
runner: linux-amd64-cpu8
- package: openshell-driver-kubernetes
binary: openshell-driver-kubernetes
triple: x86_64-unknown-linux-gnu
runner: linux-amd64-cpu8
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: ${{ github.sha }}
- uses: ./.github/actions/setup-nix
with:
cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }}
- uses: ./.github/actions/build-rust-binary
with:
package: ${{ matrix.package }}
binary: ${{ matrix.binary }}
triple: ${{ matrix.triple }}
cargo-version: ${{ needs.version.outputs.cargo }}
build-vm-driver:
needs: [pr_metadata, version, build-binaries]
if: needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
contents: read
uses: ./.github/workflows/build-vm-driver.yml
with:
cargo-version: ${{ needs.version.outputs.cargo }}
secrets:
CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}
build-images:
needs: [pr_metadata, build-binaries]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true'
permissions:
contents: read
packages: write
uses: ./.github/workflows/build-images.yml
prepare-integration:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.run_integration == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/prepare-integration-inputs.yml
# Run driver-independent conformance tests.
conformance-integration:
needs: prepare-integration
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
with:
category: conformance
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"conformance"},
{"environment":"fedora-podman-rootful","installer":"binaries","testsuite":"conformance"},
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"conformance"}
]
# Run feature-specific integration tests:
feature-specific-integration:
needs: prepare-integration
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
with:
category: feature-specific
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"fedora-podman-rootful","installer":"binaries","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"provider-refresh"}
]
# Run driver-specific integration tests:
driver-specific-integration:
needs: prepare-integration
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/integration-runner.yml
with:
category: driver-specific
source-sha: ${{ needs.prepare-integration.outputs.source_sha }}
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"fedora-podman-rootful","installer":"binaries","testsuite":"driver-podman"},
{"environment":"fedora-podman-rootless","installer":"binaries","testsuite":"driver-podman"}
]
docker-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-docker-test.yml
with:
image-tag: ${{ github.sha }}
runner: linux-arm64-cpu8
conformance-artifact-prefix: openshell-conformance
podman-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-podman-test.yml
with:
image-tag: ${{ github.sha }}
conformance-artifact-prefix: openshell-conformance
suite-matrix: >-
[{"suite":"ci","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:ci"}]
vm-e2e:
needs: [pr_metadata, build-binaries, build-vm-driver]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-vm-test.yml
with:
conformance-artifact-prefix: openshell-conformance
docker-external-driver-e2e:
needs: [pr_metadata, build-binaries, build-gateway-plain, build-external-drivers, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-docker-test.yml
with:
image-tag: ${{ github.sha }}
runner: linux-arm64-cpu8
gateway-artifact: openshell-gateway-plain-aarch64-unknown-linux-gnu
external-driver-binary: openshell-driver-docker
conformance-artifact-prefix: openshell-conformance
suite-matrix: >-
[{"suite":"external-driver","cmd":"mise run --no-deps --skip-deps e2e:docker:external-driver","apt_packages":"openssh-client","python_proto":false,"mcp":false}]
podman-external-driver-e2e:
needs: [pr_metadata, build-binaries, build-gateway-plain, build-external-drivers, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-podman-test.yml
with:
image-tag: ${{ github.sha }}
gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu
external-driver-binary: openshell-driver-podman
conformance-artifact-prefix: openshell-conformance
suite-matrix: >-
[{"suite":"external-driver","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:external-driver"}]
vm-external-driver-e2e:
needs: [pr_metadata, build-binaries, build-gateway-plain, build-vm-driver]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-vm-test.yml
with:
gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu
suite-name: external-driver
e2e-task: e2e:vm:external-driver
conformance-artifact-prefix: openshell-conformance
gpu-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_gpu_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-gpu-test.yaml
with:
image-tag: ${{ github.sha }}
conformance-artifact-prefix: openshell-conformance
kubernetes-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
strategy:
fail-fast: false
matrix:
include:
- agent_sandbox_api: v1beta1
agent_sandbox_version: v1.0.3
- agent_sandbox_api: v1alpha1
agent_sandbox_version: v0.4.6
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (Rust smoke, Agent Sandbox ${{ matrix.agent_sandbox_api }})
agent-sandbox-version: ${{ matrix.agent_sandbox_version }}
conformance-artifact-prefix: openshell-conformance
kubernetes-workspace-managed-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (workspace managed mode)
e2e-task: e2e:kubernetes:workspace-managed
conformance-artifact-prefix: openshell-conformance
kubernetes-external-driver-e2e:
needs: [pr_metadata, build-binaries, build-gateway-plain, build-external-drivers, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (external compute driver)
e2e-task: e2e:kubernetes:external-driver
gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu
external-driver-binary: openshell-driver-kubernetes
cluster-images: sandbox supervisor
conformance-artifact-prefix: openshell-conformance
kubernetes-workspace-operator-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes E2E (workspace operator mode)
e2e-task: e2e:kubernetes:workspace-operator
conformance-artifact-prefix: openshell-conformance
kubernetes-ha-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes HA E2E (Rust smoke)
extra-helm-values: deploy/helm/openshell/ci/values-high-availability.yaml
external-postgres-secret: openshell-ha-pg
test-name: kubernetes_ha_rebalancing
kubernetes-features: e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-ha
use-envoy-gateway: true
conformance-artifact-prefix: openshell-conformance
kubernetes-credential-drivers-e2e:
needs: [pr_metadata, build-binaries, build-images]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_credential_drivers_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
job-name: Kubernetes Credential Drivers E2E
e2e-task: e2e:kubernetes:credential-drivers
conformance-artifact-prefix: openshell-conformance
core-e2e-result:
name: Core E2E result
needs:
- pr_metadata
- conformance-integration
- feature-specific-integration
- driver-specific-integration
- docker-e2e
- podman-e2e
- vm-e2e
- docker-external-driver-e2e
- podman-external-driver-e2e
- vm-external-driver-e2e
- kubernetes-e2e
- kubernetes-external-driver-e2e
- kubernetes-workspace-managed-e2e
- kubernetes-workspace-operator-e2e
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/check-job-results
with:
results: ${{ toJSON(needs) }}
gpu-e2e-result:
name: GPU E2E result
needs: [pr_metadata, gpu-e2e]
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_gpu_e2e == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/check-job-results
with:
results: ${{ toJSON(needs) }}
kubernetes-ha-e2e-result:
name: Kubernetes HA E2E result
needs: [pr_metadata, kubernetes-ha-e2e]
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/check-job-results
with:
results: ${{ toJSON(needs) }}
kubernetes-credential-drivers-e2e-result:
name: Kubernetes Credential Drivers E2E result
needs: [pr_metadata, kubernetes-credential-drivers-e2e]
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_credential_drivers_e2e == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/check-job-results
with:
results: ${{ toJSON(needs) }}