Files
OpenShell/e2e/rust/e2e-vm.sh
T
Jesse Jaggars 2c31ac7e60 feat(network): support additional destination CAs (#3292)
* feat(network): support additional destination CAs

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): harden additional CA propagation

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): narrow additional CA review scope

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): harden additional CA rotation

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* feat(network): complete additional CA delivery

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(docker): remove stale runtime fixture field

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): adapt additional CA series to upstream main

Resolve rebase fallout against current upstream main after the series was
rebased 214 commits forward:

- follow upstream #3110: supervisor→gateway TLS is CA-only, so
  configure_gateway_tls and its isolation tests drop the client identity
- follow upstream tls_secret_names reduction to [String; 1] in the Podman
  driver while keeping the network trust bundle delivery
- port [openshell.supervisor.network] rendering into the
  openshell.gatewayConfigToml serializer in _toml.tpl
- adapt tests to upstream struct/function changes (DriverStartupContext,
  SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image)
- drop conformance-artifact-prefix plumbing removed upstream

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(network): adopt upstream additional CA bundle surface

Unify the additional destination CA series on the mechanism merged to
main in #3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile

The network trust startup tests asserted on raw substrings of gateway
diagnostics, but the CLI renderer wraps output at a fixed width with
continuation markers and hard-breaks long paths. Compare assertions on
whitespace- and continuation-squashed text so they hold regardless of
wrap position.

Format the e2e workspace (separate cargo workspace, not covered by the
root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock
for the libc dependency added to the network trust crate.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(podman): reuse TLS CA secret during stopped-pair trust replacement

The additional CA e2e never ran in CI before the first maintainer test
dispatch, and its local container inspection was broken in two ways: it
located the supervisor container through e2e network membership (the
supervisor is host-networked or runtime-bridged, so the filter matched
the CI job container instead), and it required the gateway mTLS CA as a
bind mount (Podman delivers it as a container secret when user
namespaces remap UIDs). Select the supervisor by managed labels and
accept the secret delivery form.

The lifecycle leg then exposed a driver bug: stopped-pair trust
replacement re-created the per-sandbox TLS CA secret, colliding with
the secret still owned by the renamed stopped pair. Mirror the token
and resolver secret handling: require and reuse the existing secret
during replacement instead of creating it.

Validated end-to-end with the Podman additional CA e2e lane.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(server): restore upstream recovery semantics for start-request failures

Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when
driver_start_request fails during the startup or lifecycle sweeps.
Gateway configuration problems (for example a missing launch signer)
must not permanently fail unrelated sandbox records; the sweep logs a
warning and skips them, matching the behavior before the additional CA
work. Sandbox-record failures surface through the normal start path.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(podman): refresh stale supervisors without replacing workloads

Replace the stopped-pair trust reconciliation with supervisor-only
recreation. The destination-trust generation label moves from the
workload to the supervisor container, and start_sandbox recreates only
the supervisor when its generation is missing or stale. The workload
container, its volumes, its secrets, and its workspace data are never
replaced: the additional CA bundle is mounted only into the supervisor.

Running sandboxes are never gated on trust labels, and recovery without
fresh launch credentials keeps the existing supervisor. The intentional
-removal fence machinery in the watcher existed only for workload
replacement and is removed with it.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(docker): drop workload replacement for destination-trust changes

Delete the stopped-workload trust reconciliation path and all of its
machinery: the trust-generation label on workload containers, the
generation match check in the start fence, the stopped-sandbox
replacement, the bounded workspace archive copy with its 512 MiB cap
and regular-file-only validation, and the boundary-state rollback
helpers.

No replacement machinery is needed: the trusted supervisor container
is already re-created from the gateway's current configuration on
every start via ensure_control_process_for_container, so stop/start
picks up the current additional CA bundle while the workload
container, its volumes, and its workspace data are never replaced.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs(configuration): drop reconciliation wording from additional CA lifecycle

The stop/start lifecycle description already matches the
supervisor-only refresh model; remove the two remaining references to
restart reconciliation of workload containers.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): port additional CA test fixtures to the crypto interface

Route the gateway TLS isolation test PKI, the destination CA source
fixtures, and the gateway startup trust tests through
openshell_crypto::pki now that rcgen access is centralized there:
generate_keypair/self_signed/signed_by replace the direct rcgen calls
and CertifiedKey field access follows the wrapper's shape.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(e2e): repair additional CA lane for Docker and non-container builds

The first maintainer test dispatch revealed two latent issues in the
additional CA e2e that no local run had exercised: the Podman gateway
CA secret helper referenced ContainerEngine without the container
feature gate, breaking Kubernetes and VM builds of the test, and the
Docker branch asserted a client ca.crt bind mount that the Docker
driver never creates. Gate the helper like its callers, and assert
Docker's actual delivery path instead: the gateway mTLS CA staged as
tls/ca.pem inside the read-only supervisor state volume, selected
through the OPENSHELL_TLS_CA environment pointer.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(helm): restore alphabetical README ordering for supervisor keys

The helm:docs check regenerates the chart README with helm-docs, which
orders rows alphabetically; a conflict resolution had placed
supervisor.network.additionalCaConfigMapName after the
supervisor.podSecurityContext rows. Regenerate with mise run
helm:docs.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* chore: refresh testkit lockfile for openshell-core libc dependency

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(driver-podman): stage operator CA bundle as driver-owned artifact

Standalone Podman drivers accepting --sandbox-additional-ca-bundle
previously kept the operator's source path as the trust artifact path.
The immutable-artifact verification performed on every create/start
requires exactly mode 0444, which an operator-owned source path cannot
guarantee, so a readable-but-writable source failed reconciliation.

Read the operator bundle with the same bounded, no-symlink validation,
then stage the normalized material into a driver-owned artifact inside a
private tempdir held for the driver process lifetime. The artifact is
created 0600, synced, and only then restricted to 0444 so no observer
ever sees a writable artifact. The operator source and its permissions
are never referenced after startup.

Addresses GATOR-291f43b2-02.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(driver-podman): pin retained workload image during supervisor refresh

Supervisor-only trust reconciliation re-entered
create_sandbox_with_existing_token, which re-pulled and re-resolved the
workload image tag even though the workload container is retained. If
the tag moved to a different image between stop and start, identity
resolution and child-environment construction would reflect the new
image while the retained workload still runs the original one, leaving
mode-0600 bootstrap files the retained user cannot read.

During a refresh with a retained workload, inspect the stopped container
and pin to the immutable image ID it was created from (Config.Image);
never pull or resolve the request tag. The requested reference is still
used for child metadata so the retained workload's environment is
byte-identical to its original launch.

Addresses GATOR-291f43b2-01.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): verify host-only VM trust delivery and drop curl dependency

The additional-CA lane's VM assertions read the destination CA out of the
guest overlay and a guest-side supervisor-argv marker — artifacts from
the guest-supervisor design that no longer exist now that the VM driver
stages the CA host-only and runs the supervisor on the host. Inspect the
driver-staged host artifact (digest-compared against the source), assert
the guest overlay contains no CA material at the old mount paths, and
keep the serial-console no-disclosure check.

The workload validation scripts also ran curl, which exists only in the
container-engine fixture image; the Kubernetes and VM lanes use the
uv-based workload image and died instantly with "canonical main process
already finished". Rewrite the main-process, exec, and lifecycle
validation scripts in python3 (present in every lane's workload image).
The network-policy binary identity is the resolved /proc/<pid>/exe
target, so the versioned python3.12 binaries are allowlisted rather
than the unversioned symlinks. The hostname mismatch check keeps its
semantics: same reachable fixture, mismatched TLS SNI, and the failure
must specifically be hostname verification.

Addresses GATOR-291f43b2-04.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps

The destination-trust ConfigMap validation compared the whole label map
for equality. A mutating admission webhook that attaches an
organizational label to the created ConfigMap made the create response
fail validation, and the retained ConfigMap then failed every subsequent
create/start through the existing-object path, blocking sandbox launches
despite the ownership labels, immutable flag, digest, and certificate
bytes remaining correct.

Validate the gateway-owned label pairs individually and allow unrelated
labels; the exact trust-material checks are unchanged. Covered by a
regression test with an admission-added label and a missing
gateway-owned label.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(podman): close supervisor-refresh rollback gaps

The supervisor-only trust refresh renamed the previous supervisor to a
random rollback name and then performed the token-secret lookup. A
transient lookup failure returned with the supervisor still renamed,
where a normal delete could never find it, and the rollback rename and
final rollback removal ignored their own failures, leaving orphans with
attached control and channel resources.

Complete every fallible preparation before the rename. On a failed
replacement, restore the canonical name; if the restore itself fails,
return an error naming the rollback container so the orphan stays
identifiable — a later retry still succeeds because the canonical name
is absent and the replacement path recreates a fresh supervisor. Treat
not-found during the final rollback removal as success and name the
orphan in both the log and the error on other removal failures; a retry
observes the fresh canonical supervisor and resumes the normal start
path.

Covered by failure-injection tests for each transaction boundary:
preparation failure before any rename, replacement failure restoring
the canonical name, and restore failure naming the orphaned rollback
container.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs: address destination-trust review corrections

The gateway-configuration RFC documented an additional_ca_cert_paths
list that the strict configuration parser never accepted; document the
additional_ca_bundle single-path setting the parser actually enforces.

The debug-openshell-cluster skill named the wrong generation annotation
(openshell.ai/network-supervisor-trust-generation) and described digest
argv and network-init canonicalization checks the implementation does
not perform. Correct it to openshell.ai/additional-ca-generation and
describe the actual delivery: the supervisor Pod mounts the immutable
ConfigMap read-only and receives --additional-ca-bundle for the mount
path, with the driver validating the digest annotation and exact bytes
per creation.

Classify the destination-trust generation hashes as deterministic
identity derivation rather than authenticity proofs, following the
openshell-crypto exclusions table: trust decisions come from the
gateway's normalization, immutable staging, exact-byte validation, and
TLS verification. Record the classification in the crypto README and at
each hashing site.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): reconcile additional CA tests with suspend/resume and auth changes

Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7
and 8, so the durable-snapshot field moves from 7 to 9; update the
wire-format test to assert the field-9 key (0x4a) and initialize the
new resume controls. Repair the cli test module after the auth opt-in
test merged alongside the environment-guard helpers, and initialize the
gateway driver-startup context's network-trust fields in the shared
supervision test.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(server): box start-recovery futures for clippy large-futures

The durable-snapshot field grows StartSandboxRequest enough that the
start-recovery match and the recovery-expiry test exceed the
large-futures limit under -D warnings. Box::pin both futures, matching
the pattern the recovery path used before the suspend/resume rework.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

---------

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
2026-10-27 18:31:11 +00:00

502 lines
19 KiB
Bash
Executable File

#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# Run the Rust e2e smoke test against an openshell-gateway running the
# standalone VM compute driver (`openshell-driver-vm`).
#
# Architecture:
# * `openshell-sandbox` runs inside a NIC-less guest.
# * The host `openshell-supervisor` connects over virtio-vsock and owns
# gateway registration, policy evaluation, DNS, and external networking.
# * Readiness is authoritative on the gateway: a sandbox's phase
# flips to `Ready` the moment `ConnectSupervisor` registers, and
# back to `Provisioning` when the session drops. The VM driver
# only reports `Error` conditions for dead launcher processes.
#
# Usage:
# mise run e2e:vm
#
# What the script does:
# 1. When no prebuilt VM driver is supplied, ensures the VM runtime
# (libkrun) and bundled sandbox/supervisor binaries are staged.
# 2. Builds `openshell-gateway`, `openshell-driver-vm`, the native host
# `openshell-supervisor` host control, and the `openshell` CLI with the
# embedded runtime as needed. When CI supplies OPENSHELL_GATEWAY_BIN,
# OPENSHELL_VM_DRIVER_BIN, OPENSHELL_VM_SUPERVISOR_BIN, or OPENSHELL_BIN,
# the matching prebuilt binary is reused instead of rebuilt.
# 3. On macOS, codesigns the VM driver (libkrun needs the
# `com.apple.security.hypervisor` entitlement).
# 4. Writes a per-run gateway config with `[openshell.drivers.vm]`
# settings, starts the gateway with `--config <run-state>/gateway.toml`
# on a random free port, waits for an authenticated gateway status
# request to succeed, then runs the selected Rust e2e tests.
# 5. Tears the gateway down and (on failure) preserves the gateway
# log and every VM serial console log for post-mortem.
#
# Prerequisites (handled automatically by this script for local VM-driver builds):
# - `mise run vm:setup` — downloads / builds the libkrun runtime.
# - `mise run vm:supervisor` — builds the bundled sandbox supervisor.
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
source "${ROOT}/e2e/support/gateway-common.sh"
# shellcheck source=e2e/support/conformance.sh
source "${ROOT}/e2e/support/conformance.sh"
COMPRESSED_DIR="${ROOT}/target/vm-runtime-compressed"
GATEWAY_BIN="${OPENSHELL_GATEWAY_BIN:-${ROOT}/target/debug/openshell-gateway}"
DRIVER_BIN="${OPENSHELL_VM_DRIVER_BIN:-${ROOT}/target/debug/openshell-driver-vm}"
CLI_BIN="${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}"
E2E_TEST_OVERRIDE="${OPENSHELL_E2E_VM_TEST:-}"
# Unset or empty selects Rust defaults; e2e-conformance-only skips Rust tests.
E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES:-e2e-vm}"
ADDITIONAL_CA_MODE="${OPENSHELL_E2E_ADDITIONAL_CA:-0}"
if [ "${E2E_TEST_OVERRIDE}" = "additional_ca" ]; then
ADDITIONAL_CA_MODE=1
E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES:-e2e-vm,e2e-additional-ca}"
fi
SANDBOX_IMAGE="${OPENSHELL_SANDBOX_IMAGE:-nvcr.io/nvidia/base/ubuntu:24.04}"
# The VM driver places `compute-driver.sock` under `[openshell.drivers.vm].state_dir`.
# AF_UNIX SUN_LEN is 104 bytes on macOS (108 on Linux), so paths anchored
# in the workspace's `target/` blow the limit on typical developer
# machines — e.g. a ~100-char `~/.superset/worktrees/.../target/...`
# prefix plus the `compute-driver.sock` leaf leaves no room. macOS'
# per-user `$TMPDIR` (`/var/folders/xx/.../T/`) can be 50+ chars too,
# so root state under `/tmp` unconditionally to keep UDS paths short.
STATE_DIR_ROOT="/tmp"
# Smoke test timeouts. First boot extracts the embedded libkrun runtime
# (~60-90MB of zstd per architecture) and prepares an ext4 root disk from the
# configured image. The guest then starts the sandbox supervisor directly; a cold
# microVM is typically ready within ~15s after image preparation.
GATEWAY_READY_TIMEOUT=60
SANDBOX_PROVISION_TIMEOUT=180
# ── Build prerequisites ──────────────────────────────────────────────
if [ -n "${RUSTC_WRAPPER:-}" ] && [ "${OPENSHELL_E2E_VM_ALLOW_RUSTC_WRAPPER:-0}" != "1" ]; then
echo "==> Building without RUSTC_WRAPPER=${RUSTC_WRAPPER} (set OPENSHELL_E2E_VM_ALLOW_RUSTC_WRAPPER=1 to keep it)"
unset RUSTC_WRAPPER
fi
if [ -z "${OPENSHELL_VM_DRIVER_BIN:-}" ]; then
mkdir -p "${COMPRESSED_DIR}"
if ! find "${COMPRESSED_DIR}" -maxdepth 1 -name 'libkrun*.zst' | grep -q .; then
echo "==> Preparing embedded VM runtime (mise run vm:setup)"
mise run vm:setup
fi
# Always rebuild the guest bundle so an e2e run cannot silently exercise a
# stale boundary binary after supervisor or isolation-interface changes.
echo "==> Building bundled VM supervisor (mise run vm:supervisor)"
mise run vm:supervisor
export OPENSHELL_VM_RUNTIME_COMPRESSED_DIR="${OPENSHELL_VM_RUNTIME_COMPRESSED_DIR:-${COMPRESSED_DIR}}"
else
echo "==> Skipping embedded VM runtime prep; prebuilt VM driver was supplied"
fi
build_packages=()
if [ -z "${OPENSHELL_GATEWAY_BIN:-}" ]; then
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
echo "==> Building driver-free openshell-gateway"
cargo build \
-p openshell-gateway --bin openshell-gateway \
--no-default-features --features telemetry
else
build_packages+=(-p openshell-gateway)
fi
else
echo "==> Using prebuilt openshell-gateway at ${GATEWAY_BIN}"
fi
if [ -z "${OPENSHELL_VM_DRIVER_BIN:-}" ]; then
build_packages+=(-p openshell-driver-vm)
else
echo "==> Using prebuilt openshell-driver-vm at ${DRIVER_BIN}"
fi
if [ -z "${OPENSHELL_VM_SUPERVISOR_BIN:-}" ]; then
# The VM driver prefers a native sibling `openshell-supervisor`. Build it
# explicitly so a stale target/debug binary cannot disagree with the
# freshly embedded guest sandbox protocol.
build_packages+=(-p openshell-supervisor)
else
echo "==> Using prebuilt VM host supervisor at ${OPENSHELL_VM_SUPERVISOR_BIN}"
fi
if [ -z "${OPENSHELL_BIN:-}" ]; then
build_packages+=(-p openshell-cli)
else
echo "==> Using prebuilt openshell CLI at ${CLI_BIN}"
fi
if [ "${#build_packages[@]}" -gt 0 ]; then
echo "==> Building ${build_packages[*]}"
cargo build "${build_packages[@]}"
fi
for pair in \
"openshell-gateway:${GATEWAY_BIN}" \
"openshell-driver-vm:${DRIVER_BIN}" \
"openshell CLI:${CLI_BIN}"; do
label="${pair%%:*}"
path="${pair#*:}"
if [ ! -x "${path}" ]; then
echo "ERROR: expected ${label} binary at ${path}" >&2
exit 1
fi
done
export OPENSHELL_BIN="${CLI_BIN}"
DRIVER_DIR="$(dirname "${DRIVER_BIN}")"
if [ "$(uname -s)" = "Darwin" ]; then
echo "==> Codesigning openshell-driver-vm (Hypervisor entitlement)"
codesign \
--entitlements "${ROOT}/crates/openshell-driver-vm/entitlements.plist" \
--force \
-s - \
"${DRIVER_BIN}"
fi
# ── Pick a random free host port for the gateway ─────────────────────
HOST_PORT="$(python3 -c 'import socket
s = socket.socket()
s.bind(("", 0))
print(s.getsockname()[1])
s.close()')"
# Per-run state dir so concurrent e2e runs don't collide on the UDS or
# sandbox state. The VM driver creates `<state_dir>/compute-driver.sock`
# and `<state_dir>/sandboxes/<id>/overlay.ext4` under here. Keep the
# basename short — see the SUN_LEN comment above.
RUN_STATE_DIR="${STATE_DIR_ROOT}/os-vm-e2e-${HOST_PORT}-$$"
mkdir -p "${RUN_STATE_DIR}"
# Keep every gateway-created artifact, including normalized additional CA
# material, under this invocation's disposable state root. Set this before
# starting either fixture or gateway so no host-global XDG fallback is used.
export XDG_CONFIG_HOME="${RUN_STATE_DIR}/config"
export XDG_DATA_HOME="${RUN_STATE_DIR}/data"
export XDG_STATE_HOME="${RUN_STATE_DIR}/state"
mkdir -p "${XDG_CONFIG_HOME}" "${XDG_DATA_HOME}" "${XDG_STATE_HOME}"
GATEWAY_LOG="$(mktemp /tmp/openshell-gateway-e2e.XXXXXX)"
GATEWAY_PID_FILE="${RUN_STATE_DIR}/gateway.pid"
GATEWAY_ARGS_FILE="${RUN_STATE_DIR}/gateway.args"
GATEWAY_CONFIG="${RUN_STATE_DIR}/gateway.toml"
GATEWAY_DB="${RUN_STATE_DIR}/gateway.db"
JWT_DIR="${RUN_STATE_DIR}/jwt"
PKI_DIR="${RUN_STATE_DIR}/pki"
GATEWAY_NAME="openshell-e2e-vm-${HOST_PORT}"
DRIVER_PID=""
DRIVER_LOG="${RUN_STATE_DIR}/vm-driver.log"
DRIVER_SOCKET="${RUN_STATE_DIR}/compute-driver.sock"
ADDITIONAL_CA_SERVER_PID=""
ADDITIONAL_CA_SERVER_LOG="${RUN_STATE_DIR}/additional-ca-server.log"
ADDITIONAL_CA_DIR="${RUN_STATE_DIR}/additional-ca"
ADDITIONAL_CA_PORT=""
# ── Cleanup (trap) ───────────────────────────────────────────────────
cleanup() {
local exit_code=$?
local gateway_pid="${GATEWAY_PID:-}"
if [ -f "${GATEWAY_PID_FILE:-}" ]; then
gateway_pid="$(cat "${GATEWAY_PID_FILE}" 2>/dev/null || true)"
fi
if [ -n "${gateway_pid}" ] && kill -0 "${gateway_pid}" 2>/dev/null; then
echo "Stopping openshell-gateway (pid ${gateway_pid})..."
# SIGTERM first; gateway drops ManagedDriverProcess which SIGKILLs
# the driver and removes the UDS. Wait briefly, then force-kill.
kill -TERM "${gateway_pid}" 2>/dev/null || true
for _ in 1 2 3 4 5 6 7 8 9 10; do
kill -0 "${gateway_pid}" 2>/dev/null || break
sleep 0.5
done
kill -KILL "${gateway_pid}" 2>/dev/null || true
wait "${gateway_pid}" 2>/dev/null || true
fi
e2e_stop_process "${DRIVER_PID}" "external VM compute driver"
e2e_stop_process "${ADDITIONAL_CA_SERVER_PID}" "additional CA HTTPS fixture"
# On failure, keep the VM console log for debugging. We deliberately
# print it instead of leaving it on disk because the state dir gets
# wiped on success.
if [ "${exit_code}" -ne 0 ]; then
echo "=== gateway log (preserved for debugging) ==="
cat "${GATEWAY_LOG}" 2>/dev/null || true
echo "=== end gateway log ==="
if [ -f "${DRIVER_LOG}" ]; then
echo "=== external VM compute driver log ==="
cat "${DRIVER_LOG}" 2>/dev/null || true
echo "=== end external VM compute driver log ==="
fi
local console
while IFS= read -r -d '' console; do
echo "=== VM console log: ${console} ==="
cat "${console}" 2>/dev/null || true
echo "=== end VM console log ==="
done < <(find "${RUN_STATE_DIR}/sandboxes" -name 'rootfs-console.log' -print0 2>/dev/null)
fi
rm -f "${GATEWAY_LOG}" 2>/dev/null || true
# Only wipe the per-run state dir on success. On failure, leave it for
# post-mortem (serial console logs and root disk images).
if [ "${exit_code}" -eq 0 ]; then
rm -rf "${RUN_STATE_DIR}" 2>/dev/null || true
else
echo "NOTE: preserving ${RUN_STATE_DIR} for debugging"
fi
}
trap cleanup EXIT
# ── Launch the gateway + VM driver ───────────────────────────────────
start_additional_ca_fixture() {
e2e_start_additional_ca_fixture \
"${ADDITIONAL_CA_DIR}" "${ADDITIONAL_CA_SERVER_LOG}" \
ADDITIONAL_CA_SERVER_PID ADDITIONAL_CA_PORT
}
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
echo "ERROR: additional CA e2e requires the gateway-managed VM driver." >&2
exit 2
fi
start_additional_ca_fixture
fi
echo "==> Starting openshell-gateway on 127.0.0.1:${HOST_PORT} (state: ${RUN_STATE_DIR})"
# Pin `driver_dir` to the workspace `target/debug/` so we always pick up
# the driver we just cargo-built. Without this, the gateway's
# `resolve_compute_driver_bin` fallback prefers
# `~/.local/libexec/openshell/openshell-driver-vm` when present,
# which silently shadows development builds — a subtle source of
# stale-binary bugs in e2e runs.
# `grpc_endpoint` is consumed by the host supervisor. The host alias is
# normalized to loopback while keeping package-managed certificate naming.
e2e_generate_gateway_jwt "${JWT_DIR}"
e2e_generate_pki "${GATEWAY_BIN}" "${PKI_DIR}"
cat >"${GATEWAY_CONFIG}" <<EOF
[openshell]
version = 2
[openshell.gateway]
bind_address = "127.0.0.1:${HOST_PORT}"
compute_driver = "vm"
guest_tls_ca = "${PKI_DIR}/ca.crt"
[openshell.gateway.tls]
cert_path = "${PKI_DIR}/server/tls.crt"
key_path = "${PKI_DIR}/server/tls.key"
client_ca_path = "${PKI_DIR}/ca.crt"
[openshell.gateway.mtls_auth]
enabled = true
[openshell.gateway.gateway_jwt]
signing_key_path = "${JWT_DIR}/signing.pem"
public_key_path = "${JWT_DIR}/public.pem"
kid_path = "${JWT_DIR}/kid"
gateway_id = "${GATEWAY_NAME}"
# Local VM e2e gateways exercise the single-player default: sandbox JWTs
# identify the supervisor and do not expire.
EOF
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
cat >>"${GATEWAY_CONFIG}" <<EOF
[openshell.supervisor.network]
additional_ca_bundle = "${ADDITIONAL_CA_DIR}/ca.crt"
EOF
fi
cat >>"${GATEWAY_CONFIG}" <<EOF
[openshell.drivers.vm]
EOF
# A rootless Linux libkrun refuses file creation on read-write bind mounts by
# a guest UID/GID other than its own, so vm_bind_mount runs the sandbox as the
# gateway user there.
VM_IDENTITY_ARGS=()
if [ "$(uname -s)" = "Linux" ] && [ "$(id -u)" != "0" ]; then
VM_IDENTITY_ARGS=(--sandbox-uid "$(id -u)" --sandbox-gid "$(id -g)")
fi
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
printf 'socket_path = "%s"\n' "${DRIVER_SOCKET}" >>"${GATEWAY_CONFIG}"
else
cat >>"${GATEWAY_CONFIG}" <<EOF
grpc_endpoint = "https://host.openshell.internal:${HOST_PORT}"
driver_dir = "${DRIVER_DIR}"
state_dir = "${RUN_STATE_DIR}"
enable_bind_mounts = true
EOF
if [ "${#VM_IDENTITY_ARGS[@]}" -gt 0 ]; then
printf 'sandbox_uid = %s\nsandbox_gid = %s\n' "$(id -u)" "$(id -g)" >>"${GATEWAY_CONFIG}"
fi
fi
# Host bind mounts are exercised by vm_bind_mount, mirroring the Docker and
# Podman e2e gateways. An external driver must acknowledge the same policy.
# Keep [openshell.drivers.vm] the last table: vm_corporate_proxy appends bare
# keys to it.
cat >>"${GATEWAY_CONFIG}" <<EOF
allow_driver_config = true
resource_admission = { enabled = false }
EOF
if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then
"${DRIVER_BIN}" \
--bind-socket "${DRIVER_SOCKET}" \
--allow-same-uid-peer \
--grpc-endpoint "https://host.openshell.internal:${HOST_PORT}" \
--default-image "${SANDBOX_IMAGE}" \
--state-dir "${RUN_STATE_DIR}" \
--guest-tls-ca "${PKI_DIR}/ca.crt" \
--admission-config-json '{"allow_driver_config":true,"resource_admission":{"enabled":false}}' \
--enable-bind-mounts \
${VM_IDENTITY_ARGS[@]+"${VM_IDENTITY_ARGS[@]}"} \
>"${DRIVER_LOG}" 2>&1 &
DRIVER_PID=$!
e2e_wait_for_socket \
"${DRIVER_SOCKET}" "${DRIVER_PID}" "external VM compute driver" 60
fi
GATEWAY_ARGS=(
--config "${GATEWAY_CONFIG}"
--db-url "sqlite:${GATEWAY_DB}?mode=rwc"
)
e2e_write_gateway_args_file "${GATEWAY_ARGS_FILE}" "${GATEWAY_ARGS[@]}"
"${GATEWAY_BIN}" "${GATEWAY_ARGS[@]}" \
>"${GATEWAY_LOG}" 2>&1 &
GATEWAY_PID=$!
printf '%s\n' "${GATEWAY_PID}" >"${GATEWAY_PID_FILE}"
# Register the gateway before polling so readiness exercises the same mTLS
# client path as the smoke tests.
CLI_GATEWAY_ENDPOINT="https://127.0.0.1:${HOST_PORT}"
e2e_register_mtls_gateway \
"${XDG_CONFIG_HOME}" \
"${GATEWAY_NAME}" \
"${CLI_GATEWAY_ENDPOINT}" \
"${HOST_PORT}" \
"${PKI_DIR}"
export OPENSHELL_GATEWAY_ENDPOINT="${CLI_GATEWAY_ENDPOINT}"
# ── Wait for gateway readiness ───────────────────────────────────────
#
# Poll the authenticated gRPC health path instead of coupling readiness to a
# particular gateway log message. The VM driver is spawned eagerly, while
# sandboxes are created on demand.
echo "==> Waiting for gateway readiness (timeout ${GATEWAY_READY_TIMEOUT}s)"
elapsed=0
last_status_output=""
while [ "${elapsed}" -lt "${GATEWAY_READY_TIMEOUT}" ]; do
if ! kill -0 "${GATEWAY_PID}" 2>/dev/null; then
echo "ERROR: openshell-gateway exited before becoming ready"
exit 1
fi
if last_status_output="$("${CLI_BIN}" status --output json 2>&1)" &&
printf '%s\n' "${last_status_output}" |
grep -Eq '"status"[[:space:]]*:[[:space:]]*"connected"'; then
echo "==> Gateway ready after ${elapsed}s"
break
fi
sleep 2
elapsed=$((elapsed + 2))
done
if [ "${elapsed}" -ge "${GATEWAY_READY_TIMEOUT}" ]; then
echo "ERROR: openshell-gateway did not become ready after ${GATEWAY_READY_TIMEOUT}s"
echo "=== last openshell status output ==="
if [ -n "${last_status_output}" ]; then
printf '%s\n' "${last_status_output}"
else
echo "<no output>"
fi
echo "=== end openshell status output ==="
exit 1
fi
# ── Run the smoke test ───────────────────────────────────────────────
#
# The CLI uses the raw endpoint but still resolves matching metadata so it
# can find the mTLS client bundle.
export OPENSHELL_E2E_DRIVER="vm"
export OPENSHELL_E2E_VM_STATE_DIR="${RUN_STATE_DIR}"
if [ "${ADDITIONAL_CA_MODE}" = "1" ]; then
additional_ca_artifact_dir="${XDG_STATE_HOME}/openshell/network-supervisor"
additional_ca_artifacts=("${additional_ca_artifact_dir}"/additional-ca-*.crt)
if [ "${#additional_ca_artifacts[@]}" -ne 1 ] || [ ! -f "${additional_ca_artifacts[0]}" ]; then
echo "ERROR: expected exactly one content-addressed additional CA artifact under XDG_STATE_HOME" >&2
exit 1
fi
# Both paths exist after gateway initialization; plain `realpath` is
# portable across GNU and BSD/macOS implementations.
state_root="$(realpath "${XDG_STATE_HOME}")"
artifact_path="$(realpath "${additional_ca_artifacts[0]}")"
case "${artifact_path}" in
"${state_root}"/*) ;;
*)
echo "ERROR: normalized additional CA artifact escaped XDG_STATE_HOME" >&2
exit 1
;;
esac
export OPENSHELL_E2E_ADDITIONAL_CA_ARTIFACT="${artifact_path}"
export OPENSHELL_E2E_GATEWAY_CONFIG="${GATEWAY_CONFIG}"
fi
e2e_export_gateway_restart_metadata \
"${GATEWAY_BIN}" \
"${GATEWAY_ARGS_FILE}" \
"${GATEWAY_LOG}" \
"${GATEWAY_PID_FILE}"
# The VM driver creates each sandbox VM from a cached read-only ext4 root disk
# plus a writable overlay disk. The guest's sandbox supervisor then initializes
# policy, netns, Landlock, and sshd. On a cold host this is ~15s after image
# preparation; allow 180s for slower CI runners.
export OPENSHELL_PROVISION_TIMEOUT="${SANDBOX_PROVISION_TIMEOUT}"
e2e_run_openshell_conformance "VM"
if [ "${E2E_FEATURES}" = "e2e-conformance-only" ]; then
echo "::notice::Skipping Rust e2e tests (e2e-conformance-only)"
exit 0
fi
# Seed the catalog once for the whole lane. The profiles live in the gateway
# for as long as it runs, and the import is create-only: a second import of
# the same directory fails with "custom provider profile '<id>' already
# exists", so this cannot move inside the per-target helper below.
e2e_import_example_provider_profiles "${CLI_BIN}" "${ROOT}" || exit 1
run_e2e_test() {
local test_target="$1"
shift
echo "==> Running e2e ${test_target} test (features: ${E2E_FEATURES}, endpoint: ${OPENSHELL_GATEWAY_ENDPOINT})"
cargo test \
--manifest-path "${ROOT}/e2e/rust/Cargo.toml" \
--features "${E2E_FEATURES}" \
--test "${test_target}" \
"$@" \
-- --nocapture
echo "==> ${test_target} test passed."
}
if [ -n "${E2E_TEST_OVERRIDE}" ]; then
run_e2e_test "${E2E_TEST_OVERRIDE}"
else
run_e2e_test ephemeral_cleanup
run_e2e_test host_gateway_alias
run_e2e_test vm_overlay
run_e2e_test vm_bind_mount
run_e2e_test vm_gateway_start
run_e2e_test vm_corporate_proxy
fi