trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-11-03 08:17:44 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
ff1f8ff55ca7886c2612a41b964fbff2b7fe8cd2
OpenShell/tasks/test.toml
T
Matthew GrossmanandEvan Lezar 33138516ad feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine (#4773) 
* feat(tmachine): add the Session API and a test-runner package

Add a library API that boots an environment, runs commands in it with timeouts,
copies files in and out, collects diagnostics and tears it down, so a caller can
drive a machine without writing an Ansible playbook per suite. SSH and exec
timeouts and QEMU cleanup on SIGTERM keep a hung guest or a cancelled CI job from
leaving QEMU running.

Add the `openshell-test-runner` flake package, which runs tests/runner with the
toolchains and VM tools it needs.

The Session API and test-runner package are adapted from NVIDIA/OpenShell#4629.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* build(cargo): add a check that aligned workspaces lock shared crates like the root

`cargo --locked` only checks a lockfile against its own manifest. A workspace
outside the root that builds root crates through path dependencies, such as the
testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two
lockfiles can resolve a shared crate to different versions without any check
failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link
against the AWS-LC that Nix provides.

check_cargo_lockfile_alignment.py fails when an aligned workspace locks a
registry crate that the root lockfile also contains at a version the root
lockfile does not have. With --fix it re-locks the workspace starting from the
root lockfile. tests/testkit is the first aligned workspace.

The script and its tests come from #4629 (a37ebe917).

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(conformance): add a sandbox smoke suite built on testkit

Add the `sandbox` crate to the conformance workspace with one testkit test that
creates a sandbox, runs `echo` in it with exec, deletes it and checks it is gone.
It is deliberately small: it exists so the runner can run a real test in a
tmachine environment on CI before the existing suites move to testkit.

The workspace's other crate, the CLI conformance suite, keeps running as it does
today. The conformance workspace joins the lockfile alignment check, so its
lockfile is realigned with the root lockfile.

The test is adapted from the `sandbox-lifecycle` suite in #4629.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* feat(runner): add a test runner that runs a test family on tmachine

`openshell-test-runner` builds a family's packages (a workspace under
tests/suites/) on the host as a nextest archive for the guest's musl target,
boots one tmachine environment by name with the named installer through the
Session API, runs nextest in it and brings back the JUnit report and log. No
per-suite Ansible playbook is involved:

  nix run .#openshell-test-runner -- --environment ubuntu-docker-rootful \
    --installer deb --family conformance --package sandbox

The runner points the tests at the system under test with environment
variables: OPENSHELL_GATEWAY (the gateway the installer registered, `--gateway`,
default `openshell`) and OPENSHELL_BIN (the installed CLI). It waits for the
gateway with `testkit-ready` before the tests start. Nextest runs with
`--test-threads 1` and a slow-timeout that terminates a hung test.

A run passes if nextest succeeds and its JUnit report shows at least one test
and no failures. `--filter` narrows the tests with a nextest filterset, and
`--keep` leaves the VM up for debugging. Results from an earlier run are
removed first, and the guest's diagnostics are collected after a failure.

Which tests run in which environments is left to the caller for now; there are
no plans, environment registry or per-test requirements, and a broken test is
`#[ignore]`d.

Most of this comes from #4629, adapted to testkit finding the gateway the way the
CLI does instead of reading a target file.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(tests): lint and test the runner, tmachine and conformance suite

Run fmt, clippy and nextest for tests/tmachine and tests/runner, and fmt and
clippy for the new `sandbox` suite crate, wherever tests/testkit already runs
them: the mise tasks and Branch Checks. The suite's tests need a gateway, so they
run through `mise run e2e:testkit` against a local Docker gateway and, on CI,
through the test runner rather than the unit-test job.

The lockfile task and the Cargo lockfiles job also run the alignment check, and
its own tests run with `mise run test`.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(tests): run testkit suites on tmachine from a reusable E2E Suites workflow

Add a reusable workflow with an explicit matrix: each row names a tmachine
environment and installer, a family and package, and the runner label. Each
row runs `nix run .#openshell-test-runner` once. The jobs use the same
integration inputs as the existing tmachine lanes, cache the tmachine disks
keyed on the Ansible tree, upload the JUnit report and logs whether or not the
run passes, and an aggregator job reports the overall result.

The only row runs the conformance sandbox suite on amd64 Ubuntu with rootful
Docker and the Debian package, which CI can provision today. Branch E2E calls
the workflow when the test:e2e label is set. The job is not in the Core E2E
result's needs yet, so it starts out advisory. The existing conformance,
feature and driver lanes are unchanged.

Modeled on the E2E Suites workflow in #4629.

Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(tests): document the runner and the conformance suite

Describe how the runner runs a family in a tmachine environment, locally and
on CI, in TESTING.md, add a README for the sandbox suite, point the testkit
README at the runner, and add rows for tests/suites, tests/tmachine and
tests/runner to the project-structure tables in AGENTS.md and CONTRIBUTING.md.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(tests): simplify the test runner and tmachine

- Replace tmachine's global QEMU PID registry with until_signal, which
  drops the work on SIGTERM/SIGINT/SIGHUP so kill_on_drop stops QEMU.
  This also lets Ctrl-C end a --keep run.
- Decide a run's status from nextest's exit code (with --no-tests fail)
  instead of parsing JUnit; print the end of the nextest log on failure.
- Move gateway diagnostics from tmachine's Session into the runner, drop
  Session::run_playbooks, and make Session own its VM directly.
- Read TMACHINE_SSH_PORT once and reject invalid values; drop the racy
  port pre-check.
- Build the runner with Nix so tmachine's firmware paths are compiled in,
  and set ANSIBLE_CONFIG only in the wrapper.
- Copy the bundle directory in directly, fold lib.rs into main.rs, and
  make the gateway name required.
- Drop the semver hint from the lockfile alignment check and trim
  duplicated docs.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(tmachine): keep the SSH port check and name a run with no tests

Restore the check that the SSH forward port is free before QEMU starts. QEMU
reports a taken port only by exiting, so a layer build would otherwise run its
playbooks against the VM that already holds the port and cache the result.

Report nextest's no-tests exit code as "no test ran" instead of a bare exit
status.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(runner): keep logs from a timed-out run and gate on the E2E suites

nextest now writes its output to a file in the guest, and the runner copies
nextest.log and junit.xml out after every run, so a run that times out or
loses SSH still leaves a log. A failed JUnit copy no longer turns a test
failure into an error.

The runner exits with 1 when a test failed and 2 when the run could not
complete, and annotates the GitHub Actions run with which it was.

The E2E suites lane is now part of the Core E2E result. Its disk cache key
also covers tmachine's cache versions in setup.rs and install.rs.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* ci(e2e): push test runner builds to Cachix

E2E Suites set up Nix without a Cachix token, so every job rebuilt
openshell-test-runner from source. Accept an optional CACHIX_AUTH_TOKEN
secret, pass it to setup-nix, and pass it from Branch E2E, as Branch Checks
and the binary builds already do.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
2026-10-28 18:29:25 +00:00

356 lines
18 KiB
TOML
Raw Blame History

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# Test tasks (Rust + Python + TypeScript SDK)
[test]
description = "Run all tests (Rust + Python + TypeScript SDK)"
depends = [
"test:rust",
"test:python",
"sdk:ts:test",
"test:sbom",
"test:install-sh",
"test:build-env",
"test:postgres-runner",
"test:gateway-pull-policy",
"test:e2e-image-overrides",
"test:gateway-config",
"test:packaging-assets",
"test:qualification-summary",
"test:codex-security-release-range",
"test:codex-security-diagnostics",
"test:docs-website",
"test:docs-nav",
"test:maintainer-approval",
"test:cargo-lockfile-alignment",
]
["test:cargo-lockfile-alignment"]
description = "Test the Cargo lockfile alignment check"
run = "uv run --no-project --with pytest pytest tasks/scripts/check_cargo_lockfile_alignment_test.py"
["test:docs-website"]
description = "Test the docs-website sync script"
# --no-project skips installing the OpenShell package; --with supplies the test
# dependencies and the script's runtime dependency, which live outside the project env.
run = "uv run --no-project --with pytest --with pytest-asyncio --with pyyaml pytest tasks/scripts/sync_docs_website_test.py"
["test:docs-nav"]
description = "Test the docs navigation check"
run = "uv run --no-project --with pytest --with pytest-asyncio --with pyyaml pytest tasks/scripts/check_docs_nav_test.py"
["test:sbom"]
description = "Run SBOM tooling tests"
run = "uv run --no-project --with pytest pytest -o \"python_files=*_test.py\" deploy/sbom/"
hide = true
["test:install-sh"]
description = "Run focused install.sh shell tests"
run = "tasks/scripts/test-install-sh.sh"
run_windows = "echo Skipping test:install-sh: Linux glibc installer tests do not apply on Windows."
hide = true
["test:build-env"]
description = "Run build-env.sh helper shell tests"
run = "tasks/scripts/test-build-env.sh"
run_windows = "echo Skipping test:build-env: the Unix build-env.sh helper does not apply on Windows."
hide = true
["test:gateway-pull-policy"]
description = "Test development gateway image pull-policy normalization"
run = "tasks/scripts/test-gateway-pull-policy.sh"
run_windows = "echo Skipping test:gateway-pull-policy: Unix gateway scripts do not apply on Windows."
hide = true
["test:e2e-image-overrides"]
description = "Test E2E gateway and supervisor image override resolution"
run = "tasks/scripts/test-e2e-image-overrides.sh"
run_windows = "echo Skipping test:e2e-image-overrides: Unix E2E wrappers do not apply on Windows."
hide = true
["test:packaging-assets"]
description = "Run static packaging asset tests"
run = "tasks/scripts/test-packaging-assets.sh"
run_windows = "echo Skipping test:packaging-assets: Linux service and RPM assets do not apply on Windows."
hide = true
["test:qualification-summary"]
description = "Test release qualification summary generation"
run = "tasks/scripts/test-qualification-summary.sh"
run_windows = "echo Skipping test:qualification-summary: the release workflow runs on Unix hosts."
hide = true
["test:codex-security-release-range"]
description = "Test Codex Security release-range resolution"
run = "uv run --no-project --with pytest pytest -o \"python_files=*_test.py\" tasks/scripts/codex_security_range_test.py"
hide = true
["test:codex-security-diagnostics"]
description = "Test public Codex Security diagnostics and sensitive-data filtering"
run = "uv run --no-project --with pytest pytest tasks/scripts/codex_security_diagnostics_test.py"
hide = true
["test:maintainer-approval"]
description = "Test the maintainer approval gate and change-alert tools"
run = "uv run --no-project --with pytest pytest -o \"python_files=*_test.py\" tasks/scripts/check_maintainer_approval_test.py tasks/scripts/alert_maintainer_change_test.py"
hide = true
[e2e]
description = "Run all end-to-end tests (Rust + Python + MCP)"
depends = ["e2e:rust", "e2e:python", "e2e:mcp"]
["e2e:test"]
description = "Build the current checkout and run a named host or Nix test-guest E2E suite"
run = "e2e/run.sh"
["e2e:gpu"]
description = "Run Docker GPU end-to-end tests"
depends = ["e2e:docker:gpu"]
["e2e:workloads:build"]
description = "Build local GPU workload test images and manifest"
run = "bash tasks/scripts/e2e-gpu-build-images.sh"
["test:rust"]
description = "Run Rust tests"
depends = ["rust:lockfiles:check"]
env = { OPENSHELL_TELEMETRY_ENABLED = "false" }
run = [
# Prove the facade contract without the default backend's compiled features.
"cargo test -p openshell-crypto --no-default-features --test independent_backend",
# Run the workspace once without openshell-server so we can run that crate
# with test-only helpers enabled.
"cargo test --workspace --exclude openshell-server",
"cargo test -p openshell-server --features test-support",
"cargo nextest run --config-file .config/nextest.toml --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml",
"cargo nextest run --config-file .config/nextest.toml --manifest-path tests/testkit/Cargo.toml",
"cargo nextest run --config-file .config/nextest.toml --manifest-path tests/tmachine/Cargo.toml",
"cargo nextest run --config-file .config/nextest.toml --manifest-path tests/runner/Cargo.toml",
]
run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test-precommit native"
hide = true
["test:rust:postgres"]
description = "Run PostgreSQL-backed openshell-server tests against OPENSHELL_TEST_POSTGRES_URL or a disposable Docker/Podman PostgreSQL container"
run = "tasks/scripts/run-postgres-tests.sh"
run_windows = "echo Skipping test:rust:postgres: PostgreSQL integration tests need a Linux or macOS container engine."
["test:rust:postgres:bench"]
description = "Run the load-sensitive bench_postgres_* capacity checks that test:rust:postgres and CI skip"
env = { OPENSHELL_TEST_POSTGRES_PREFIX = "bench_postgres_" }
run = "tasks/scripts/run-postgres-tests.sh"
run_windows = "echo Skipping test:rust:postgres:bench: PostgreSQL integration tests need a Linux or macOS container engine."
["test:postgres-runner"]
description = "Test PostgreSQL test runner database and prefix selection without a database or container engine"
run = "tasks/scripts/test-postgres-test-runner.sh"
run_windows = "echo Skipping test:postgres-runner: the Unix PostgreSQL test runner does not apply on Windows."
hide = true
["test:python"]
description = "Run Python tests"
depends = ["python:proto"]
env = { UV_NO_SYNC = "1" }
run = "uv run pytest python/"
hide = true
["e2e:rust"]
description = "Run Rust CLI e2e tests against a Docker-backed gateway"
run = "e2e/rust/e2e-docker.sh"
["e2e:oci-image"]
description = "Run the shared OCI image feature suite against a Docker-backed gateway"
run = "OPENSHELL_TEST_CONTAINER_ENGINE=docker e2e/with-docker-gateway.sh cargo test --locked --manifest-path tests/suites/features/Cargo.toml -p openshell-test-feature-oci-image -- --test-threads 1 --nocapture"
["e2e:oci-image:podman"]
description = "Run the shared OCI image feature suite against a Podman-backed gateway"
run = "OPENSHELL_TEST_CONTAINER_ENGINE=podman e2e/with-podman-gateway.sh cargo test --locked --manifest-path tests/suites/features/Cargo.toml -p openshell-test-feature-oci-image -- --test-threads 1 --nocapture"
["e2e:testkit"]
description = "Run the testkit suites in tests/suites/conformance against a Docker-backed gateway"
run = "e2e/with-docker-gateway.sh cargo nextest run --locked --manifest-path tests/suites/conformance/Cargo.toml --workspace --exclude openshell-test-conformance-cli --test-threads 1 --no-fail-fast"
["e2e:rust:push"]
description = "Run CLI conformance and policy e2e tests against a Docker gateway that pushes configuration"
depends = ["e2e:conformance:build"]
env = { OPENSHELL_CONFIG_DELIVERY_MODE = "push", OPENSHELL_E2E_DOCKER_FEATURES = "e2e-config-push", OPENSHELL_E2E_DOCKER_TEST = "config_push live_policy_update policy_activation" }
run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh"
["e2e:workload:build"]
description = "Build the Docker E2E workload fixture"
run = "CONTAINER_ENGINE=docker bash tasks/scripts/e2e-build-workload.sh"
["e2e:websocket-conformance"]
description = "Run focused WebSocket conformance e2e tests against a Docker-backed gateway"
run = [
"CONTAINER_ENGINE=docker bash tasks/scripts/e2e-build-workload.sh",
"e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test websocket_conformance",
]
["e2e:mcp"]
description = "Run MCP conformance e2e scenarios against one Docker-backed gateway (static defaults for spec 2025-11-25; set OPENSHELL_MCP_CONFORMANCE_SCENARIOS for a focused subset)"
run = "bash e2e/mcp-conformance.sh"
["e2e:nodejs"]
description = "Alias for e2e:mcp"
depends = ["e2e:mcp"]
["e2e:python"]
description = "Run Python e2e tests against a Docker-backed gateway (E2E_PARALLEL=N or 'auto'; default 5)"
depends = ["python:proto"]
env = { UV_NO_SYNC = "1", PYTHONPATH = "python" }
run = [
"CONTAINER_ENGINE=docker bash tasks/scripts/e2e-build-workload.sh",
"OPENSHELL_E2E_DOCKER_SANDBOX_IMAGE=openshell/e2e-python:dev OPENSHELL_E2E_DOCKER_SANDBOX_IMAGE_PULL_POLICY=never e2e/with-docker-gateway.sh uv run pytest -o python_files='test_*.py *_test.py' -m 'not gpu' -n ${E2E_PARALLEL:-5} e2e/python",
]
["e2e:podman"]
description = "Run Rust CLI e2e tests against a Podman-backed gateway"
run = "e2e/rust/e2e-podman.sh"
["e2e:podman:ci"]
description = "Run the Podman Rust e2e targets enabled in required branch CI"
run = "OPENSHELL_E2E_PODMAN_TEST_SET=ci e2e/rust/e2e-podman.sh"
["e2e:oidc-pkce"]
description = "Run Linux browser PKCE and RBAC e2e tests against Keycloak and a Podman gateway"
run = [
"CONTAINER_RUNTIME=podman e2e/with-keycloak.sh env OPENSHELL_E2E_OIDC_GATEWAY=1 e2e/with-podman-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-oidc-pkce --test oidc_pkce",
]
["e2e:oidc-pkce:docker"]
description = "Run Linux browser PKCE and RBAC e2e tests against Keycloak and a Docker gateway"
run = [
"CONTAINER_RUNTIME=docker e2e/with-keycloak.sh env OPENSHELL_E2E_OIDC_GATEWAY=1 e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-oidc-pkce --test oidc_pkce",
]
["e2e:oidc-python:docker"]
description = "Run Python OIDC and workspace authorization e2e tests against Keycloak and a Docker gateway"
depends = ["python:proto"]
env = { UV_NO_SYNC = "1", PYTHONPATH = "python" }
run = [
"CONTAINER_RUNTIME=docker e2e/with-keycloak.sh env OPENSHELL_E2E_OIDC_GATEWAY=1 e2e/with-docker-gateway.sh uv run pytest -m 'not gpu' e2e/python/oidc",
]
["e2e:podman:gpu"]
description = "Run GPU e2e against a standalone gateway with the Podman compute driver"
env = { OPENSHELL_E2E_PODMAN_GPU = "1", OPENSHELL_E2E_PODMAN_TEST = "gpu", OPENSHELL_E2E_PODMAN_FEATURES = "e2e-podman-gpu" }
run = "e2e/rust/e2e-podman.sh"
["e2e:kubernetes"]
description = "Run Rust CLI e2e tests against an OpenShell gateway deployed on Kubernetes via Helm (set OPENSHELL_E2E_KUBE_CONTEXT to reuse a cluster; otherwise creates a local k3d cluster when k3d is installed; set OPENSHELL_E2E_KUBE_TEST=<name> to scope to one test)"
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:v1alpha1"]
description = "Run Kubernetes e2e against Agent Sandbox v1alpha1"
env = { AGENT_SANDBOX_VERSION = "v0.4.6" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:agent-sandbox-versions"]
description = "Run Kubernetes e2e against Agent Sandbox v1beta1 and v1alpha1"
run = [
"e2e/rust/e2e-kubernetes.sh",
"AGENT_SANDBOX_VERSION=v0.4.6 e2e/rust/e2e-kubernetes.sh",
]
["e2e:kubernetes:isolation"]
description = "Run Kubernetes e2e with the workload network fence and separate supervisor"
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:db"]
description = "Run Kubernetes e2e with all database backend scenarios (SQLite and external PostgreSQL with existingSecret)"
env = { OPENSHELL_E2E_KUBE_DB_SCENARIOS = "1" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:ha-rebalancing"]
description = "Run the Kubernetes HA rebalancing suite through Envoy against two gateway replicas and external PostgreSQL"
env = { OPENSHELL_E2E_KUBE_EXTERNAL_POSTGRES_SECRET = "openshell-ha-pg", OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-high-availability.yaml", OPENSHELL_E2E_KUBE_TEST = "kubernetes_ha_rebalancing", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-ha", OPENSHELL_E2E_KUBE_USE_ENVOY = "1" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:credential-drivers"]
description = "Run Kubernetes e2e for provider credential storage backed by Kubernetes Secrets and Vault"
env = { OPENSHELL_E2E_CREDENTIAL_DRIVERS = "1", OPENSHELL_E2E_KUBE_TEST = "credential_drivers", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-kubernetes,e2e-kubernetes-credential-drivers" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:workspace-managed"]
description = "Run Kubernetes e2e with managed workspace mode (auto-created per-workspace namespaces)"
env = { OPENSHELL_E2E_CREDENTIAL_DRIVERS = "1", OPENSHELL_E2E_CREDENTIAL_DRIVER = "kubernetes-secrets", OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-workspace-managed.yaml", OPENSHELL_E2E_KUBE_IMAGE_PULL_SECRET = "e2e-regcred", OPENSHELL_E2E_KUBE_TEST = "workspace_namespace_managed", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-kubernetes,e2e-kubernetes-workspace-managed" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:shared-supervisor"]
description = "Run Kubernetes e2e with a one-member shared supervisor pool for the gateway's sandboxes (gateway supervisor topology)"
env = { OPENSHELL_E2E_KUBE_TEST = "kubernetes_shared_supervisor", OPENSHELL_E2E_KUBE_TLS = "1", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-shared-supervisor" }
# Prepend the supervisor overlay to any caller overlays, such as a workspace
# mode. The shared supervisor requires gateway TLS, which the Skaffold values
# turn off.
run = 'OPENSHELL_E2E_KUBE_EXTRA_VALUES="deploy/helm/openshell/ci/values-supervisor.yaml:deploy/helm/openshell/ci/values-tls-port-forward.yaml${OPENSHELL_E2E_KUBE_EXTRA_VALUES:+:$OPENSHELL_E2E_KUBE_EXTRA_VALUES}" e2e/rust/e2e-kubernetes.sh'
["e2e:kubernetes:shared-supervisor-ha"]
description = "Run Kubernetes e2e that kills the owning member of a two-replica shared supervisor pool and checks its sandboxes are adopted, not restarted"
env = { OPENSHELL_E2E_KUBE_TEST = "kubernetes_shared_supervisor_ha", OPENSHELL_E2E_KUBE_TLS = "1", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-shared-supervisor-ha" }
# The shared supervisor requires gateway TLS, which the Skaffold values turn off.
run = 'OPENSHELL_E2E_KUBE_EXTRA_VALUES="deploy/helm/openshell/ci/values-supervisor-ha.yaml:deploy/helm/openshell/ci/values-tls-port-forward.yaml${OPENSHELL_E2E_KUBE_EXTRA_VALUES:+:$OPENSHELL_E2E_KUBE_EXTRA_VALUES}" e2e/rust/e2e-kubernetes.sh'
["e2e:kubernetes:shared-supervisor-scaling"]
description = "Run Kubernetes e2e that scales a one-member shared supervisor pool to two members and back to one with Helm, checking sandboxes are handed off, not restarted"
env = { OPENSHELL_E2E_KUBE_TEST = "kubernetes_shared_supervisor_scaling", OPENSHELL_E2E_KUBE_TLS = "1", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-shared-supervisor-scaling" }
# Its own release: the test upgrades it, so it never runs beside the abrupt-loss
# test. The shared supervisor requires gateway TLS, which the Skaffold values turn off.
run = 'OPENSHELL_E2E_KUBE_EXTRA_VALUES="deploy/helm/openshell/ci/values-supervisor-ha-scaling.yaml:deploy/helm/openshell/ci/values-tls-port-forward.yaml${OPENSHELL_E2E_KUBE_EXTRA_VALUES:+:$OPENSHELL_E2E_KUBE_EXTRA_VALUES}" e2e/rust/e2e-kubernetes.sh'
["e2e:kubernetes:workspace-operator"]
description = "Run Kubernetes e2e with operator workspace mode (pre-provisioned per-workspace namespaces)"
env = { OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-workspace-operator.yaml", OPENSHELL_E2E_KUBE_TEST = "workspace_namespace_operator", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-kubernetes,e2e-kubernetes-workspace-operator" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:vm"]
description = "Start openshell-gateway with the VM compute driver and run VM e2e tests"
run = "e2e/rust/e2e-vm.sh"
["e2e:gateway:no-compute-drivers"]
description = "Build and launch-check openshell-gateway without compiled compute drivers"
run = "bash e2e/no-compute-driver-gateway.sh"
["e2e:docker:external-driver"]
description = "Run Docker conformance with a driver-free gateway and external Docker driver binary"
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_DOCKER_FEATURES = "e2e-conformance-only" }
run = "e2e/rust/e2e-docker.sh"
["e2e:podman:external-driver"]
description = "Run Podman conformance with a driver-free gateway and external Podman driver binary"
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_PODMAN_FEATURES = "e2e-conformance-only" }
run = "e2e/rust/e2e-podman.sh"
["e2e:vm:external-driver"]
description = "Run VM E2E with a driver-free gateway and external VM driver binary"
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1" }
run = "e2e/rust/e2e-vm.sh"
["e2e:kubernetes:external-driver"]
description = "Run Kubernetes conformance with a driver-free gateway and external Kubernetes driver"
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_KUBE_BUILD_IMAGES = "1", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e-conformance-only" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:docker"]
description = "Run Docker conformance and Rust e2e tests against a standalone gateway"
run = "e2e/rust/e2e-docker.sh"
["e2e:mechanistic-existing-endpoint"]
description = "Run #2821 existing inspected-endpoint auto-approval regression"
run = [
"cargo build -p openshell-cli",
"e2e/with-docker-gateway.sh bash -lc 'target/debug/openshell settings set --global --key agent_policy_proposals_enabled --value true --yes && OPENSHELL_BIN=$PWD/target/debug/openshell bash e2e/policy-advisor/existing-endpoint-auto-approve.sh'",
]
["e2e:docker:gpu"]
description = "Run GPU e2e against a standalone gateway with the Docker compute driver"
env = { OPENSHELL_E2E_DOCKER_GPU = "1", OPENSHELL_E2E_DOCKER_TEST = "gpu", OPENSHELL_E2E_DOCKER_FEATURES = "e2e-docker-gpu" }
run = "e2e/rust/e2e-docker.sh"
["test:gateway-config"]
description = "Test generated local gateway TOML without starting a runtime"
run = "bash tasks/scripts/test-gateway-config.sh"
run_windows = "echo Skipping test:gateway-config: Unix gateway scripts do not apply on Windows."
hide = true
Reference in New Issue View Git Blame Copy Permalink