OpenShell/TESTING.md

Testing

Running Tests

mise run test          # Rust + Python unit tests
mise run e2e           # End-to-end tests (starts a Docker-backed gateway)
mise run ci            # Everything: lint, compile checks, and tests

Test Layout

crates/*/src/          # Inline #[cfg(test)] modules
crates/*/tests/        # Rust integration tests
python/openshell/      # Python unit tests (*_test.py suffix)
e2e/python/            # Python E2E tests (test_*.py prefix)
e2e/rust/              # Rust CLI E2E tests

Rust Tests

Unit tests live inline with #[cfg(test)] mod tests blocks. Integration tests go in crates/*/tests/ and are named *_integration.rs.

Use #[tokio::test] for anything async:

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn store_round_trip() {
        let store = Store::connect("sqlite::memory:").await.unwrap();
        store.put("sandbox", "abc", "my-sandbox", b"payload").await.unwrap();
        let record = store.get("sandbox", "abc").await.unwrap().unwrap();
        assert_eq!(record.payload, b"payload");
    }
}

Run Rust tests only:

mise run test:rust     # cargo test --workspace

Rust validation checks tracked Cargo lockfiles; run mise run rust:lockfiles:check to check them directly. If one is stale, refresh it with Cargo using its adjacent manifest, review the diff, and commit the update.

Native Windows validation

Use mise run --skip-tools pre-commit with the existing Rust/MSVC toolchain. Windows now checks tracked Cargo lockfiles through PowerShell rather than skipping them. The deterministic gateway parity task uses Git for Windows Bash, with temporary Python launchers confined to a unique checkout-owned directory.

mise run --skip-tools sdk:ts:ci selects the x64 Biome executable on Windows (including ARM64 hosts running it under emulation), resolves the protobuf plugin through its Windows .cmd launcher, and installs the matching locked ARM64 Rolldown binding when Node itself is ARM64. The helper preserves lockfile resolution; it must not upgrade unrelated test dependencies.

mise run --skip-tools go:ci retains race detection except on Windows ARM64, where Go does not support it. Windows token-file tests explicitly skip POSIX mode-bit assertions; those skips do not establish Windows ACL protection. Use a checkout with LF text files when running Unix-shell fixture checks.

Python Unit Tests

Python unit tests use the *_test.py suffix convention (not test_* prefix) and live alongside the source in python/openshell/. They use mock-based patterns with fake gRPC stubs:

def test_exec_python_serializes_callable_payload() -> None:
    stub = _FakeStub()
    client = _client_with_fake_stub(stub)

    def add(a: int, b: int) -> int:
        return a + b

    result = client.exec_python("sandbox-1", add, args=(2, 3))
    assert result.exit_code == 0

Run Python unit tests only:

mise run test:python   # uv run pytest python/

E2E Tests

E2E tests run against a live gateway. By default, mise run e2e starts an ephemeral standalone gateway with the Docker compute driver, runs the suite, and cleans it up afterward. To run the suite against an existing plaintext gateway, set OPENSHELL_GATEWAY_ENDPOINT:

OPENSHELL_GATEWAY_ENDPOINT=http://127.0.0.1:18080 mise run e2e

Raw endpoint mode is HTTP-only. Use a named gateway config when a gateway requires mTLS.

Python E2E (e2e/python/)

mise run e2e:python builds openshell/e2e-python:dev from e2e/python/Dockerfile.workload and selects it only for the test gateway. This Noble-based fixture supplies the sandbox user, Python tooling, Git, and a writable /sandbox/.venv. Its Python version comes from .python-version so cloudpickle code objects match the test runner. The production workload default remains the unmodified NVIDIA Ubuntu image.

The Rust Docker harness also selects this fixture for tests that need tools or the named user. Explicit --from or --template arguments and the default-image tests retain their own image selection. Docker sandbox and support-container fixtures use the fixed image name; no workload-image override is needed. Podman, VM, and Kubernetes retain their existing pinned, pullable fixture and image setup. Conformance-only runs do not need the Docker fixture. Build the Docker fixture separately with mise run e2e:workload:build.

Tests use the sandbox fixture from conftest.py to create real sandboxes:

def test_exec_returns_stdout(sandbox):
    with sandbox(delete_on_exit=True) as sb:
        result = sb.exec(["echo", "hello"])
        assert result.exit_code == 0
        assert "hello" in result.stdout

Sandbox.exec_python

exec_python serializes a Python callable with cloudpickle, sends it to the sandbox, and returns the result. Because cloudpickle serializes module-level functions by reference (which fails inside the sandbox), use one of these patterns:

Closures from factory functions:

def _make_adder():
    def add(a, b):
        return a + b
    return add

def test_addition(sandbox):
    with sandbox(delete_on_exit=True) as sb:
        result = sb.exec_python(_make_adder(), args=(2, 3))
        assert result.stdout.strip() == "5"

Bound methods on local classes:

def test_multiply(sandbox):
    class Calculator:
        def multiply(self, a, b):
            return a * b

    with sandbox(delete_on_exit=True) as sb:
        result = sb.exec_python(Calculator().multiply, args=(6, 7))
        assert result.stdout.strip() == "42"

Shared Fixtures (e2e/python/conftest.py)

Fixture	Scope	Purpose
sandbox_client	session	gRPC client connected to the active gateway
sandbox	function	Factory returning a Sandbox context manager

Rust CLI E2E (e2e/rust/)

Rust-based e2e tests that exercise the openshell CLI binary as a subprocess. They live in the openshell-e2e crate and use a shared harness for sandbox lifecycle management, output parsing, and cleanup.

Suites:

• Common suite (--features e2e) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests.
• CLI conformance (openshell-conformance) - the portable deployment smoke scenario plus focused tests for its reusable command runner.
• Driver suites (--features e2e-docker, e2e-podman, e2e-kubernetes, or e2e-vm) - CLI conformance plus the common and driver-specific coverage for the selected deployment.
• Docker suite (--features e2e-docker) - includes Docker-only coverage such as Dockerfile image builds, Docker preflight checks, and managed Docker gateway start.
• Docker GPU suite (--features e2e-docker-gpu) - Docker suite plus GPU sandbox smoke coverage.
• VM suite (--features e2e-vm) - runs e2e tests on a VM.
• Kubernetes credential-driver suite (--features e2e-kubernetes-credential-drivers) - targeted Kubernetes Secrets and Vault provider credential storage coverage.

GPU device-selection tests compare OpenShell sandboxes against a plain Docker or Podman container that requests --device nvidia.com/gpu=all. The probe image defaults to the image used by the gateway stage in deploy/docker/Dockerfile.images; set OPENSHELL_E2E_GPU_PROBE_IMAGE to override it. Per-device checks run only for NVIDIA CDI device IDs reported by the runtime's discovered devices list, so WSL2 hosts that expose only nvidia.com/gpu=all skip the index-based cases. Exact CDI device selection is passed through --driver-config-json with the active Docker or Podman driver key.

Run the Docker-backed Rust CLI e2e suite:

mise run e2e:docker

Run the minimal portable CLI conformance profile against the gateway selected in your OpenShell CLI configuration:

mise run e2e:cli-conformance

The gateway must already be installed, reachable, and selected before the task starts. The task does not provision a gateway or select a compute driver. Set OPENSHELL_BIN to test a prebuilt CLI; otherwise, the task builds the CLI from the current checkout.

The phase-1 scenario verifies the complete CLI-to-gateway-to-driver path without depending on how the gateway was installed or which driver is configured. It requires machine-readable gRPC status, creates a uniquely named detached sandbox with the configured default image, verifies the sandbox is Ready by finding its unique name in paginated JSON list output, executes echo with a run-specific marker, deletes the sandbox, and verifies that its name no longer appears. Driver suites enable the same profile instead of maintaining a separate smoke implementation. Sandbox lifecycle, label matrices, VM overlay, and TLS-key permission assertions remain regular E2E coverage.

Each invocation prints a ten-character run ID before creating resources. Conformance sandboxes use names such as ct-<run-id>-01. The runner tracks the exact name and uses it for cleanup; phase 1 does not add ownership labels.

The runner deletes owned resources after both success and failure. If the test process is interrupted before cleanup, locate leftovers without touching unrelated gateway state:

openshell sandbox list --output json
openshell sandbox delete <sandbox-name>

Gateway-backed Rust E2E tasks build the standalone conformance CLI, run its registered scenarios against the configured gateway, then run any lane-specific Rust tests that still apply. Run the Podman-backed Rust CLI e2e suite:

mise run e2e:podman

Run the VM-backed Rust CLI e2e suite:

mise run e2e:vm

Run the targeted Kubernetes credential-driver e2e suite. This deploys an OpenBao fixture for the Vault-compatible driver path and validates Kubernetes Secrets and Vault storage backends one at a time:

mise run e2e:kubernetes:credential-drivers

Kubernetes E2E (e2e/rust/e2e-kubernetes.sh)

Kubernetes e2e tests deploy an OpenShell gateway into a real Kubernetes cluster via Helm and run the Rust e2e suite against it. On vanilla Kubernetes the harness reaches the gateway through kubectl port-forward; on OpenShift it instead uses a passthrough Route secured with mandatory mTLS (see the OpenShift note below).

Run with an ephemeral k3d cluster (macOS; created and torn down automatically):

mise run e2e:kubernetes

Target an existing cluster (kind, k3d, or OpenShift):

OPENSHELL_E2E_KUBE_CONTEXT=my-context mise run e2e:kubernetes

Scope to a single test for local debugging:

OPENSHELL_E2E_KUBE_TEST=smoke mise run e2e:kubernetes

OpenShift: when the target cluster exposes the route.openshift.io API group, the harness automatically applies SCC-compatible Helm overrides, grants the required SCCs (privileged to openshell-sandbox, and anyuid to the PostgreSQL fixture for DB scenarios), and drives the gateway through a passthrough Route with mandatory mTLS instead of port-forward. No extra flags are needed, but oc must be installed and authenticated against the target cluster with permission to modify SCC bindings (oc adm policy add-scc-to-user) — the harness exits early if oc is missing. The SCC grants and extracted client mTLS material are removed during cleanup, including on failure or interrupt.

On a remote cluster, drop the e2e-host-gateway feature. Those tests rely on the sandbox-side host.openshell.internal alias reaching the machine running the tests, which is unreachable from pods on a remote cluster, so they fail. Left enabled, the host_gateway_alias suite fails because host.openshell.internal does not resolve inside the pod, so the gateway SSRF-denies the request (DNS resolution failed / ssrf_denied) — a networking property of remote pods, not a gateway or transport fault. Override OPENSHELL_E2E_KUBERNETES_FEATURES to exclude it:

OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
  OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
  mise run e2e:kubernetes

On an existing cluster the harness builds the CLI from your branch but pulls the published gateway/supervisor image (default tag latest). The CLI and the image can therefore be different versions. If tests fail because of this version difference — for example, sandbox tests fail with Pod exists with phase: Failed or connect-based tests stall because the deployed image predates a feature your branch CLI needs — set IMAGE_TAG to an image that matches your branch.

The latest tag lags to the last semver release, so it is often older than main. Two better choices:

• IMAGE_TAG=dev — a floating tag that tracks the latest main build. Good for an ad-hoc run when your branch is close to main HEAD. Because it floats, two runs on different days can pull different images, so it is not reproducible.
• Pin the exact commit your branch is based on — deterministic and immune to a floating tag moving. Published tags are the full 40-char git SHA (semver tags without a v prefix also exist but only for released versions):

OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
  OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
  IMAGE_TAG=$(git rev-parse "$(git merge-base HEAD upstream/main)") \
  mise run e2e:kubernetes

To pin a specific released version, use its semver tag without a v prefix (0.0.115, not v0.0.115):

OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \
  OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \
  IMAGE_TAG=0.0.115 \
  mise run e2e:kubernetes

A semver tag matches a released commit, which may be behind main; if your branch CLI needs a newer feature, pin the SHA of your branch's base instead.

Confirm a tag exists before relying on it (set TAG to the tag you plan to use):

TAG=0.0.115
skopeo inspect "docker://ghcr.io/nvidia/openshell/gateway:${TAG}"

IMAGE_TAG sets only the gateway/supervisor image; the CLI under test is always built from your branch. To validate against images from your exact commit instead, build and push them and point OPENSHELL_REGISTRY/IMAGE_TAG at them.

Available task variants:

Task	Purpose
e2e:kubernetes	Default Rust e2e against Helm-deployed gateway
e2e:kubernetes:db	All database backend scenarios (SQLite + external PostgreSQL)
e2e:kubernetes:sidecar	Supervisor sidecar topology overlay
e2e:kubernetes:credential-drivers	Kubernetes Secrets and Vault credential storage
e2e:kubernetes:workspace-managed	Managed workspace mode (auto-created namespaces)
e2e:kubernetes:workspace-operator	Operator workspace mode (pre-provisioned namespaces)
e2e:kubernetes:v1alpha1	Agent Sandbox v1alpha1 compatibility
e2e:kubernetes:external-driver	External Kubernetes driver sidecar

Kubernetes e2e environment variables:

Variable	Purpose
OPENSHELL_E2E_KUBE_CONTEXT	kubectl context for an existing cluster (skips k3d creation)
OPENSHELL_E2E_KUBE_TEST	Scope to a single test (e.g. smoke)
OPENSHELL_E2E_KUBE_EXTRA_VALUES	Colon-separated additional Helm values files
OPENSHELL_E2E_KUBERNETES_FEATURES	Cargo feature flags (default: e2e,e2e-host-gateway,e2e-kubernetes)
IMAGE_TAG	Gateway/supervisor image tag (default: latest for existing clusters)
OPENSHELL_REGISTRY	Image registry prefix (default: ghcr.io/nvidia/openshell)

Run a single test directly with cargo:

cargo test --manifest-path e2e/rust/Cargo.toml --features e2e --test sync

Run a single Docker-only test directly with cargo:

cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test custom_image

The harness (e2e/rust/src/harness/) provides:

Module	Purpose
binary	Builds and resolves the openshell binary from the workspace
container	Container-engine selection and support containers for proxy tests
gateway	Managed gateway restart controls for gateway-owned e2e runs
sandbox	SandboxGuard RAII type — creates sandboxes and deletes them on drop
output	ANSI stripping and field extraction from CLI output
port	wait_for_port() and find_free_port() for TCP testing

Environment Variables

Variable	Purpose
OPENSHELL_GATEWAY	Override active gateway name for E2E tests
OPENSHELL_GATEWAY_ENDPOINT	Run E2E tests against an existing plaintext HTTP gateway endpoint
OPENSHELL_E2E_DRIVER	Driver name exported by the e2e gateway wrapper (docker, podman, or vm)
OPENSHELL_E2E_CREDENTIAL_DRIVERS	Enables the Kubernetes credential-driver fixture path in e2e/with-kube-gateway.sh
OPENSHELL_E2E_KUBE_CONTEXT	kubectl context for Kubernetes e2e (skips ephemeral k3d)
OPENSHELL_E2E_KUBE_TEST	Scope Kubernetes e2e to a single test by name