Files
OpenShell/e2e/rust/tests/rootfs_tar.rs
Philippe Martin 118b250f01 feat(sandbox): support rootfs tar as --from source for VM driver (#2863)
* feat(sandbox): support rootfs tar as --from source for VM driver

Accept flat rootfs tar archives (.tar, .tar.gz, .tgz) via the --from
flag for VM-backed gateways. The CLI detects the archive extension,
validates that the gateway uses the VM compute driver, and passes the
tar path through driver_config. The VM driver copies the tar into its
staging area and feeds it into the existing rootfs extraction and ext4
disk creation pipeline, skipping the container image pull/export steps.

Closes #2175

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(sandbox): validate rootfs tar path at the VM driver boundary

The rootfs_tar_path field in driver_config was passed from the API
caller directly to tokio::fs::copy without validation. An authenticated
user bypassing the CLI could supply arbitrary host paths (e.g.
/dev/zero for disk exhaustion, or readable host files for data
exfiltration).

Introduce a trusted staging directory that the VM driver creates on
startup and advertises via GetCapabilities. The CLI now copies the tar
into the staging directory before creating the sandbox, and the driver
validates that the received path is a regular file inside the staging
root and within a configurable size limit (default 10 GiB) before any
I/O.

New VmDriverConfig options:
- rootfs_tar_staging_dir: override the staging directory
  (default: <state_dir>/rootfs-tar-staging)
- rootfs_tar_max_bytes: override the size limit (default: 10 GiB)

Addresses GATOR-28b5152e-01.

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(sandbox): request-scoped staging, size pre-check, and cleanup for rootfs tar

Tighten the rootfs tar staging flow to address the remaining GATOR-01
obligations:

- Request-scoped staging: the CLI creates a unique per-request
  subdirectory (req-<pid>) under the staging root instead of placing
  files directly in the shared directory. The driver enforces that the
  tar path is at depth 2 (staging_root/<subdir>/<file>), preventing
  cross-request path selection.

- Size pre-check: the driver advertises rootfs_tar_max_bytes via
  GetCapabilities. The CLI reads this limit and rejects oversized files
  before copying, avoiding disk exhaustion in the staging directory.

- Cleanup: the driver removes the request staging subdirectory after
  consuming the tar (on cache hit, copy success, or copy failure),
  ensuring staged data does not persist beyond the request.

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(vm): restore rootfs-tar sandboxes from persisted image identity

On restore or restart, the one-shot staged tar archive has already been
cleaned up. Reading the persisted image identity from the sandbox state
directory and resolving the cached disk path directly avoids re-accessing
the deleted staging path.

Addresses GATOR-168b9210-01.

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(cli): use random staging dirs and enforce byte limit during rootfs tar copy

Replace PID-based request staging directories with tempfile-generated
random names to prevent collisions and make paths unpredictable.
Replace bare tokio::fs::copy with a streaming copy loop that enforces
the advertised max_bytes limit during transfer, closing the TOCTOU gap
between the pre-copy size check and the actual copy.

Signed-off-by: Philippe Martin <phmartin@nvidia.com>
Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(sandbox): issue rootfs tar staging slots from the gateway

A caller could name any host path in `driver_config.vm.rootfs_tar_path`,
which the privileged VM driver then read. The CLI-side locality check did
not apply to direct API requests.

The gateway now owns staging. `BeginRootfsTarStaging` allocates a
request-scoped directory under the driver-advertised staging root and
returns an opaque single-use token; `CreateSandbox` carries the token, and
the gateway substitutes the path it allocated before dispatching to the
driver. `template.driver_config.<driver>.rootfs_tar_path` is rejected
outright in request validation, so a caller-supplied path never reaches
privileged I/O.

Tokens are bound to the issuing workspace and subject, consumed once, and
expire after 30 minutes. Outstanding slots are capped per caller and
overall, so one caller can neither exhaust the staging filesystem nor
starve others. An RAII guard reclaims the directory on every failure path
after consumption, and an age-gated sweep runs at startup and on each
reconcile pass for directories whose driver died before its own cleanup.

The token is stripped from the public sandbox before persistence: the
stored copy is returned verbatim by GetSandbox, ListSandboxes and
WatchSandbox to every member of the workspace.

Also fixes two defects this exposed:

- The CLI wrote `rootfs_tar_path` at the top level of `driver_config`, but
  the gateway forwards only `driver_config.<driver_name>`, silently
  dropping unmatched keys. The archive never reached the VM driver, so the
  documented `--from ./rootfs.tar` flow did not work at all. Config is now
  nested under `vm` and deep-merged, so a caller's existing VM settings
  survive instead of being clobbered by a shallow extend.

- Staging previously required `GetGatewayInfo`, which is restricted to
  `platform_admin`, making the feature unusable for ordinary users on any
  RBAC-enabled gateway. The new RPC matches CreateSandbox at
  `sandbox:write` / `workspace_role: user`.

`compute_driver.proto` is unchanged; the gateway reads the staging root
from the capabilities it already stores.

Refs #2175

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(vm): derive rootfs tar cache identity from archive contents

The prepared-disk cache key combined the archive's full path with an mtime
truncated to seconds, then mapped punctuation to `-`. Distinct paths such as
`/tmp/a/b.tar` and `/tmp/a-b.tar` collapsed onto the same key and reused each
other's disk, a rewrite within the same second kept stale contents, and a long
path could exceed filesystem component limits.

Identity is now a SHA-256 of the archive contents. This is also what makes the
cache work at all now that the gateway allocates a fresh staging directory per
request: a path-derived key would miss on every create.

The archive is hashed, the cache checked, and only on a miss copied — so a hit
skips writing a multi-gigabyte file. The copy is hashed as it is written and
rejected if the digest differs from the first pass, which closes the window
where the source changes during staging rather than approximating it with a
re-stat.

Refs #2175

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(vm): decompress gzip rootfs tar archives during staging

`--from` accepts `.tar.gz` and `.tgz`, but the driver staged whatever bytes
it was given and the guest image-prep VM extracts the staged file with a
plain `tar -xpf`. Compressed sources therefore depended on the guest tar
auto-detecting gzip, and the prepared disk was sized from the compressed
length, which is far too small for the expanded rootfs.

Staging now detects gzip from the archive's magic bytes -- the driver only
ever sees a gateway-issued staging path, never the caller's file name -- and
writes an uncompressed tar. The digest still covers the source bytes, so the
"archive changed while staging" check is unaffected, and expansion is bounded
by `rootfs_tar_max_bytes` so a compression bomb cannot fill the host disk.

`extract_rootfs_archive_to` sniffs gzip as well, so the host-side extraction
path matches.

Adds unit coverage for gzip staging, bounded expansion, and gzip extraction,
plus an e2e sandbox created from a gzip-compressed export.

Signed-off-by: Philippe Martin <phmartin@redhat.com>

---------

Signed-off-by: Philippe Martin <phmartin@redhat.com>
Signed-off-by: Philippe Martin <phmartin@nvidia.com>
2026-09-08 22:36:40 +00:00

159 lines
5.4 KiB
Rust

// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
#![cfg(feature = "e2e")]
//! E2E tests: create a sandbox from a flat rootfs tar archive, plain and
//! gzip-compressed.
//!
//! Prerequisites:
//! - A running VM-backed openshell gateway with a default sandbox image configured
//! - Docker daemon running (for image build + container export)
//! - The `openshell` binary (built automatically from the workspace)
use openshell_e2e::harness::container::ContainerEngine;
use openshell_e2e::harness::output::strip_ansi;
use openshell_e2e::harness::sandbox::SandboxGuard;
use std::path::{Path, PathBuf};
use std::process::Command;
const DOCKERFILE_CONTENT: &str = r#"FROM public.ecr.aws/docker/library/python:3.13-slim
# iproute2 is required for sandbox network namespace isolation.
RUN apt-get update && apt-get install -y --no-install-recommends iproute2 \
&& rm -rf /var/lib/apt/lists/*
# Create the sandbox user/group so the supervisor can switch to it.
RUN groupadd -g 1000660000 sandbox && \
useradd -m -u 1000660000 -g sandbox sandbox
RUN echo "rootfs-tar-e2e-marker" > /etc/marker.txt
CMD ["sleep", "infinity"]
"#;
const MARKER: &str = "rootfs-tar-e2e-marker";
/// Build a Docker image and export its filesystem as a flat rootfs tar.
///
/// `suffix` keeps the image tag and temporary container name unique so tests
/// exercising different archive formats can run concurrently.
fn export_rootfs_tar(engine: &ContainerEngine, tmpdir: &Path, suffix: &str) -> PathBuf {
let dockerfile_path = tmpdir.join("Dockerfile");
std::fs::write(&dockerfile_path, DOCKERFILE_CONTENT).expect("write Dockerfile");
let tag = format!(
"openshell/e2e-rootfs-tar-test-{suffix}:{}",
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
);
let build_output = engine
.command()
.args(["build", "-t", &tag, "-f"])
.arg(&dockerfile_path)
.arg(tmpdir)
.output()
.expect("spawn docker build");
assert!(
build_output.status.success(),
"docker build failed:\n{}",
String::from_utf8_lossy(&build_output.stderr)
);
// Create a temporary container and export its filesystem as a flat rootfs
// tar (equivalent to `docker export`).
let container_name = format!(
"openshell-e2e-rootfs-export-{suffix}-{}",
std::process::id()
);
let create_output = engine
.command()
.args(["create", "--name", &container_name, &tag])
.output()
.expect("spawn docker create");
assert!(
create_output.status.success(),
"docker create failed:\n{}",
String::from_utf8_lossy(&create_output.stderr)
);
let rootfs_tar_path = tmpdir.join("rootfs.tar");
let export_output = engine
.command()
.args(["export", "-o"])
.arg(&rootfs_tar_path)
.arg(&container_name)
.output()
.expect("spawn docker export");
assert!(
export_output.status.success(),
"docker export failed:\n{}",
String::from_utf8_lossy(&export_output.stderr)
);
// Clean up the temporary container and image.
let _ = engine.command().args(["rm", &container_name]).output();
let _ = engine.command().args(["rmi", &tag]).output();
rootfs_tar_path
}
/// Create a sandbox from `archive` and assert the marker baked into the image
/// shows up in its output.
async fn assert_sandbox_from_archive(archive: &Path) {
let archive_str = archive.to_str().expect("archive path is UTF-8");
let mut guard = SandboxGuard::create(&["--from", archive_str, "--", "cat", "/etc/marker.txt"])
.await
.expect("sandbox create from rootfs tar");
let clean_output = strip_ansi(&guard.create_output);
assert!(
clean_output.contains(MARKER),
"expected marker '{MARKER}' in sandbox output for {}:\n{clean_output}",
archive.display()
);
guard.cleanup().await;
}
/// Build a Docker image, export its filesystem as a flat rootfs tar, then
/// create a sandbox from that tar and verify it contains the expected marker.
#[tokio::test]
async fn sandbox_from_rootfs_tar() {
let engine = ContainerEngine::from_env().expect("container engine available");
let tmpdir = tempfile::tempdir().expect("create tmpdir");
let rootfs_tar_path = export_rootfs_tar(&engine, tmpdir.path(), "plain");
assert_sandbox_from_archive(&rootfs_tar_path).await;
}
/// The CLI advertises `.tar.gz` and `.tgz` sources, so a gzip-compressed
/// export has to reach the sandbox the same way a plain tar does.
#[tokio::test]
async fn sandbox_from_gzipped_rootfs_tar() {
let engine = ContainerEngine::from_env().expect("container engine available");
let tmpdir = tempfile::tempdir().expect("create tmpdir");
let rootfs_tar_path = export_rootfs_tar(&engine, tmpdir.path(), "gzip");
let gzipped_path = tmpdir.path().join("rootfs.tar.gz");
let gzipped = std::fs::File::create(&gzipped_path).expect("create gzip archive");
let gzip_status = Command::new("gzip")
.arg("-c")
.arg(&rootfs_tar_path)
.stdout(gzipped)
.status()
.expect("spawn gzip");
assert!(gzip_status.success(), "gzip failed: {gzip_status}");
std::fs::remove_file(&rootfs_tar_path).expect("remove uncompressed archive");
assert_sandbox_from_archive(&gzipped_path).await;
}