43 Commits
Author SHA1 Message Date
Florent BENOIT a4c848adbd feat(vm): add host bind mount support via virtiofs (libkrun) (#3463)
* feat(vm): add host bind mount support via virtiofs (libkrun)

Introduce host directory bind mounts for libkrun VM sandboxes using
the virtiofs4 API with simplified permission semantics. The gateway
accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML
table and forwards it to the VM driver subprocess.

Mounts are specified per-sandbox in driver_config.mounts[] with
source, target, optional read_only (default: read-only), and an
optional `type` that must be `bind`, so Docker and Podman bind entries
(without `selinux_label`) parse unchanged.

At sandbox validation the driver rejects mounts unless bind mounts
are enabled and resource admission is disabled. It rejects GPU and
other QEMU launches, sources with control characters, and more mounts
than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and
checks targets against reserved VM-internal directories and each other
(no nested targets). The QEMU check runs again once the launch backend
is resolved.

Host source directories are checked at provisioning, before image
preparation, so a briefly missing source fails provisioning visibly
instead of denying recovery of a persisted sandbox. Missing,
inaccessible, symlinked, non-directory, and unreadable sources are
rejected: libkrun does not follow a final symlink, and on macOS it
panics during guest boot when it cannot open the share root.

The driver generates virtiofs tags and always writes a guest-side
manifest into the overlay disk, together with the init drop-ins, so an
image-supplied manifest can never drive guest mounts. The launcher
parses its tab-separated --vm-mount arguments from the right, so a
source can never shift the tag or mode, and the driver logs each
shared directory at launch.

Guest init mounts the shares after every /sandbox ownership fixup
and the init drop-ins, so recursive chowns never reach host files.
It refuses targets that traverse symlinks in the image, hands newly
created mount point parents under /sandbox to the sandbox user, and
skips an empty manifest.

krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes
still boot sandboxes without mounts.

The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS
explicitly; the pinned libkrunfw base config already enables both.

A vm_bind_mount e2e test checks a read-write share round-trips data
with the host, accepts a new file, directory, and nested file that the
host then sees, and that a read-only share rejects writes. libkrun on
rootless Linux refuses file creation by a guest user whose UID/GID
differ from its own, so the VM e2e gateway runs the sandbox as the
gateway user there. It enables bind mounts like the Docker and Podman
e2e gateways.

The runtimes docs cover the mount limit, the symlink restriction, the
guest-kernel exposure, and the rootless Linux UID/GID note. The
debug-openshell-cluster skill lists VM bind mount checks.

Signed-off-by: Florent Benoit <fbenoit@redhat.com>

* fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime

The Nix vm-runtime package fetches the rolling vm-runtime release with
fixed hashes pinned on 2026-09-16. The release was rebuilt on
2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (#3451),
but the hashes were not updated, so CI kept substituting the
2026-09-15 runtime (libkrun v1.17.4) from the binary cache.

That libkrun lacks krun_add_virtiofs4, which VM bind mounts need.
Point the hashes at the current release assets.

Signed-off-by: Florent Benoit <fbenoit@redhat.com>

---------

Signed-off-by: Florent Benoit <fbenoit@redhat.com>
2026-10-16 07:56:32 +00:00
alangou ea74f2bf5e fix(vm): bind cached registry images to verified manifest digests (#4342)
Signed-off-by: Adrien Langou <alangou@nvidia.com>
2026-10-13 13:22:00 +00:00
Shiju 4188eabaf2 fix(vm): stop image workers before cleaning staging files (#4039)
Run image preparation in an owned worker process, reserve its process
identity until cleanup completes, and protect staging with leases so
cancellation and recovery cannot race with another preparation attempt.

Signed-off-by: Shiju <shiju@nvidia.com>
2026-10-03 20:03:40 +00:00
Drew Newberry 021400be8a refactor(auth): separate sandbox identity from TLS (#3110)
* refactor(auth): separate sandbox identity from TLS

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(auth): clarify gateway mTLS behavior

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(auth): include workspace scope in TLS authorization checks

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(e2e): bound service auth sandbox names for large PIDs

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-01 04:33:25 +00:00
Drew Newberry 9244868056 docs: refresh architecture and agent guides (#3705)
* docs: refresh architecture and agent guides

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs: describe updated security architecture neutrally

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs: highlight new isolation primitives

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(sandboxes): clarify how to disconnect

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs: align architecture and guides with current navigation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(extensibility): streamline extension authentication guidance

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-09-25 02:38:00 -07:00
Evan Lezar 6af5520f37 fix(vm): confine OCI layer application to the rootfs (#3550)
* refactor(vm): isolate OCI layer application

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(vm): confine OCI layer application

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(vm): require trusted bootstrap image

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(vm): validate bootstrap image in gateway

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(vm): configure bootstrap image in tracing test

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
2026-09-24 17:22:46 +00:00
Drew Newberry 1e34e8c576 fix(drivers): require admission labels for external resources (#3538)
* fix(drivers): require admission labels for external resources

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(drivers): address resource admission review findings

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(core): reserve driver-owned admission labels

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(core): clarify workspace admission label

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(drivers): clarify resource admission failures

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(e2e): configure resource admission fixtures

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): retry forbidden admission lookups

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(e2e): preserve external driver admission defaults

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-09-22 21:22:30 +00:00
Emilien Macchiands2cube cb6e88acb7 fix(vm): unpack registry images correctly and validate prepared disks (#3524)
* fix(vm): unpack registry images correctly and validate prepared disks

The registry image-prep path expected `umoci raw unpack` to produce a
bundle-style rootfs/ subdirectory, but it extracts the image filesystem
directly into the target. Every registry prep therefore failed after a
successful unpack. Guest init exit codes do not survive the libkrun
boundary, so the failure looked like success and the broken disk was
cached, making every later sandbox for that image fail with "prepared
image disk missing /image-rootfs".

VM E2E started hitting this after the bootstrap image moved to
nvcr.io/nvidia/base/ubuntu:24.04: `--from base` no longer matches the
bootstrap image, so it now goes through registry prep.

- Accept umoci's direct extraction layout in the guest prep script.
- Build the image rootfs under a partial directory and rename it to
  /image-rootfs only after every prep step succeeds.
- Check the prepared disk for /image-rootfs before caching it. On
  failure, leave the cache untouched and report the prep console tail.
- Size the prep disk to hold the payload and the unpacked rootfs at the
  same time. The community base image needs 1.40 GB + 3.32 GB, which
  did not fit in the old payload*3 + 512 MiB.

Fixes #2358

Co-authored-by: s2cube <26961336+s2cube@users.noreply.github.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* test(e2e): run tool-dependent VM tests from the community base image

The host_gateway_alias and vm_corporate_proxy workloads run curl and
python3. The VM driver now defaults to nvcr.io/nvidia/base/ubuntu:24.04,
which ships neither, so these tests fail in VM E2E with "command not
found". Request the community base image explicitly with `--from base`.

Docker, Podman, and Kubernetes E2E already default to that image, so
their behavior is unchanged.

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

---------

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Co-authored-by: s2cube <26961336+s2cube@users.noreply.github.com>
2026-09-21 13:25:02 -07:00
Drew Newberry dee4f98dda chore(vm): refresh runtime defaults and hardening (#3446)
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-09-21 10:19:30 +00:00
Drew Newberry c1f2e7189f feat(isolation): implement the RFC 0012 sandbox architecture (#2942)
* feat(isolation): add RFC 0012 backend contract

Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>

* refactor(isolation): name the interface crate explicitly

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): expose trusted host gateway

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(agents): inventory the MXC driver

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): add mediated DNS transport

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): tighten interface error and digest contracts

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(isolation): remove unrelated driver inventory

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): define capability-free launch contract

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): seal confirmed boundary state

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): validate confirmation for external backend implementations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): clarify mediated DNS identity

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(isolation): generalize loopback connector

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(isolation): unify typed network mediation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): bind launches to sandbox sessions

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(mxc): initialize extended sandbox status

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): add boundary protocol and Linux primitives

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): harden signals and separate process status from transport

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): validate remote confirmation through public contract

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): validate wire state and propagate snapshot failures

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(isolation): import owned agent specification explicitly

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(isolation): describe mediated DNS channel

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): bound mediation attach without nested retries

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(isolation): generalize loopback protocol

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): add transport-neutral session authentication

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(isolation): separate sandbox backend protocol

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): harden runtime boundary controls

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): add terminal boundary operation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): split supervisor and sandbox runtimes

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): harden boundary isolation and lifecycle ownership

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): reject private root redirects and adopt typed errors

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): preserve accept thread ownership on musl

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(sandbox): isolate credential probes from filtered threads

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): return retained exec exit status to independent waiters

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): bound network mediation and preserve socket authorization

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): bound control admission and retire stale mediation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci(e2e): select migrated drivers per stack layer

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): implement loopback connector

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(isolation): authenticate the Sandbox Protocol

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): rotate launch-scoped authentication

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): consume dedicated backend crate

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(sandbox): align topology session fixture

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): align projected bootstrap bundle

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(auth): validate refreshed credentials before rotation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): fail closed across supervisor disconnects

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): repair rebased sandbox CI

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* build(runtime): publish separate sandbox and supervisor images

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(config): configure the sandbox runtime image

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(ci): validate sandbox binary linkage

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(isolation): use backend and runtime terminology

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): use a scratch runtime image

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(ci): refresh schema and dependency policy

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): bind reconnects to supervisor process

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs: align runtime split operational guidance

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* chore(security): document Kubernetes runtime RBAC

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): enforce runtime lifecycle invariants

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(compute): identify sandbox start generations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(server): restore sandbox launch sessions

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): support authenticated runtime replacement

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(auth): bind sandbox session successors

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(auth): retry pending sandbox successors

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(vm): run the supervisor outside the guest workload

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(vm): use sandbox backend protocol

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): repair rebase integration

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): use unified build toolchain

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(vm): use sandbox runtime terminology

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(vm): own guest network bootstrap

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): expose guest init version

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): select native supervisor artifacts

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): guard guest init Linux symbols

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): scope Linux test imports

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): avoid guest interface casts

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): reconcile admitted sandbox identity

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): share resolved sandbox identity

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): surface host supervisor failures

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): include guest logs on supervisor exit

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): rotate and clean runtime generations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): make sandbox starts generation-aware

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): rotate restored sandbox sessions

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(vm): keep shared paths in the base layer

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): bind sandbox session lineage

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(docker): isolate workloads behind the host supervisor

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(docker): rotate launch-scoped authentication

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(docker): use sandbox backend protocol

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(docker): use host networking for supervisor

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): preserve host gateway alias resolution

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(docker): use separate sandbox and supervisor images

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): restore startup validation after rebase

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(docker): name the sandbox runtime directly

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): narrow supervisor CA runtime storage

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): close companion isolation gaps

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(docker): align mediated network expectations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(docker): exercise mediated network paths

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): attach supervisor to managed network

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): defer supervisor recovery until gateway is ready

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): make sandbox starts generation-aware

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): rotate restored sandbox sessions

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): preserve workloads during session rotation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(docker): remove unrelated configuration RFC changes

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): bind sandbox session lineage

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(kubernetes): add proxy-pod isolation topology

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(kubernetes): use sandbox backend protocol

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): use stable sandbox service authority

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(kubernetes): split sandbox and supervisor images

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): adapt proxy pods to current runtime APIs

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(kubernetes): describe the single runtime placement

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(kubernetes): simplify sandbox orchestration

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): validate deployment prerequisites

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): update Trivy Helm profile inventory

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(kubernetes): update Trivy scan inventory count

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): reuse preloaded runtime images in e2e

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): type and clean runtime resources

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): make sandbox restarts recoverable

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): rotate restored sandbox sessions

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): preserve supervisor egress

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): bind sandbox session lineage

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(podman): adopt isolated sandbox and supervisor containers

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): stage bootstrap archives at named volume destinations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(podman): rotate launch-scoped authentication

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(podman): use sandbox backend protocol

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(podman): use host networking for supervisor

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(podman): split sandbox and supervisor images

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): repair rebase integration

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(podman): name the sandbox runtime directly

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): provision supervisor CA runtime storage

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): address isolation review findings

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): inspect Debian supervisor provenance

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): use libpod-compatible tmpfs options

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): bind verified sandbox runtime binary

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): provide external driver data directory

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): start sandbox before joining user namespace

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): separate supervisor user namespace

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): make sandbox starts generation-aware

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): rotate restored sandbox sessions

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): bind sandbox session lineage

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* perf(isolation): add TCP and DNS benchmark harnesses

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(perf): align benchmark timing and supported protocols

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(perf): report TCP benchmark metrics accurately

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(perf): cancel failed worker startup

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): build matching local supervisor image

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(podman): make local sandbox smoke test runnable

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): wire local sandbox runtime image

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): narrow sandbox service RBAC

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(ci): validate split runtime artifacts

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): harden runtime session handling

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): add standalone network proxy role

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(rfc): remove implementation companion notes

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(vm): standardize runtime release name

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(vm): pin renamed runtime artifacts

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(auth): persist sandbox runtime identity

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(runtime): restore branch validation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(isolation): reconcile main after rebase

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(network): close unframed HTTP 1.0 responses

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* chore(isolation): preserve upstream OCSF updates

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(security): close credential and TLS replay paths

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(auth): make sandbox refresh retries idempotent

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-09-16 00:49:14 +00:00
Jesse JaggarsandDrew Newberry 02b664bb0d refactor(config): normalize and enforce gateway schema v2 (#2814)
* refactor(config): normalize compute driver field names

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(config): introduce canonical gateway fields

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(config): enforce gateway schema version 2

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(config): preserve compute driver runtime guarantees

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(config): address schema v2 review regressions

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(config): complete schema v2 migration safeguards

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(config): expand schema v2 regression coverage

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(config): add schema v2 parity manifest

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(config): correct parity manifest inventory

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs(config): record schema v2 intentional changes

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs(config): disposition schema v2 parity gaps

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): add dual schema parity harness

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): establish compute lifecycle parity baseline

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(config): preserve gateway option compatibility

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): record gateway option parity

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs(config): close gateway-wide parity gaps

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(podman): apply configured pids limit

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): validate Podman option parity

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): add Kubernetes option parity harness

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): record Kubernetes option parity

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): disposition VM parity lanes

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): add external driver parity lane

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(e2e): preserve external driver pull policy

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): attest parity artifacts and launches

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): require clean parity build sources

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): bind parity runtime artifacts

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(e2e): use isolated supervisor tags

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(e2e): qualify parity image tags

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(e2e): serve parity supervisor locally

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): isolate parity podman services

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): harden parity evidence provenance

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): pin parity sandbox artifacts

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): attest parity runtime inputs

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): bind parity runtime evidence

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): record compute boundary parity

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): disposition cross-cutting parity lanes

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(packaging): preflight gateway config upgrades

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(config): preserve rebase integration guarantees

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(ci): isolate temporary git signing config

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(config): update remaining schema v2 consumers

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(ci): provide e2fs tools to VM tests

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(config): align preflight with gateway startup

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(vm): preserve rootfs tar configuration

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* chore(config): adopt duration unit constructors

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(packaging): preflight RPM gateway config

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(config): address driver review findings

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(e2e): require fresh semantic parity evidence

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(docker): update tests for renamed sandbox label

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(gateway): preserve selective driver coverage after rebase

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
2026-09-11 05:00:24 +00:00
Evie Howard 6e6b3c8905 refactor(cli): remove local Dockerfile image builds (#3214)
Signed-off-by: Evie Howard <evhoward@redhat.com>
2026-09-09 13:28:17 +00:00
Philippe Martin 857af42a16 feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes (#3090)
* feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes

The corporate forward proxy machinery from #1792 is driver-agnostic and
already merged: openshell-supervisor-network implements CONNECT chaining,
NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and
openshell-sandbox exposes it as six argv-only flags. Podman gained the driver
half in #2245/#2512 and Kubernetes in #2633; the VM driver had none of it, so
VM sandboxes on proxy-only networks could not reach any destination requiring
the proxy even when policy allowed it.

The blocking piece was not proxy logic but delivery: the VM guest init script
runs as PID 1 and execs a fixed supervisor command line, and libkrun's
krun_set_exec receives an empty argv, so there was no channel for driver-owned
supervisor arguments. The supervisor's proxy flags deliberately have no
environment fallback, and build_guest_environment merges user-supplied
environment, so the guest env is not a safe transport either.

Add a driver-authored argument file, mirroring the existing init.d manifest:
the driver writes /opt/openshell/supervisor-args into the overlay upperdir on
every launch and the guest reads it verbatim, one argument per line, appending
it to every supervisor exec. It is written even when empty, which is what makes
the channel unforgeable -- the upperdir always shadows the read-only image
layer, so an image can neither supply its own arguments nor disable the
operator's by omitting the file. Because both launch backends exec the same
init script, this covers libkrun and QEMU without touching either.

A microVM has no bind mounts or container secrets, so the credential and CA
bundle are staged into the per-sandbox overlay the way the gateway JWT already
is: credential root-only at 0600, CA at 0644, both rewritten every launch so a
removed setting clears prior material, and both deleted with the sandbox state
directory. This places the credential at rest in the overlay image on the
gateway host, which differs from the Podman secret model and is documented as
an explicit security consideration.

Validation is fail-closed and shared: a new
openshell_core::driver_utils::validate_upstream_proxy_settings holds the
pairing rules the Podman driver established, and both the gateway and the
driver call it so an invalid table names the offending key instead of
surfacing as an opaque driver-readiness timeout.

Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback
is reachable only through host.openshell.internal; the guest to gateway
callback is unaffected.

Closes #3088

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(vm): bound the proxy CA read and scope the host-loopback recipe

Two review findings on the corporate forward proxy support for microVM
sandboxes.

The driver read the operator's proxy_ca_bundle with an unbounded fs::read and
accepted it on a substring match for the PEM BEGIN CERTIFICATE marker. A
special file such as /dev/zero therefore grew driver memory without bound on
every authorized sandbox create, and a PEM block holding invalid DER passed
the host check but contributes no trust anchor in the guest, so every
supervisor would fail after boot with an error attributed to the sandbox
rather than to the setting.

Move the read into openshell-core as read_upstream_proxy_ca_bundle_file: it
reuses the credential reader's bounded-read path (non-regular files rejected
on fstat, size capped, read bounded even if the file grows), then requires at
least one anchor that RootCertStore::add_parsable_certificates accepts. The
supervisor's own reader now delegates to it, so host acceptance and guest
acceptance are the same function and cannot drift.

The published host-loopback recipe was written for libkrun only. gvproxy NATs
host.openshell.internal to the gateway host's 127.0.0.1, but GPU sandboxes run
on the QEMU/TAP backend where that name resolves to the TAP host address and
the driver's own nftables input chain accepts only the gateway port from the
guest — no proxy on the gateway host is reachable there at any bind address,
so an operator following the generic recipe lost all proxy-required egress
while configuration validation succeeded.

Scope the recipe to libkrun in every reference and reject a gateway-host proxy
URL when a launch plan resolves to QEMU, naming the reason, instead of booting
a sandbox whose policy-approved CONNECTs all time out.

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(vm): match the QEMU proxy preflight to the selected TAP host

The gateway-host proxy guard added for the QEMU/TAP backend classified the
wrong set of addresses in both directions.

It ran at the top of configure_qemu_launch_plan, before the subnet allocation
that settles plan.host_ip, so it could not compare against the address the
guest actually reaches the host on. An operator pointing https_proxy at the
sandbox's own TAP host address, such as 10.0.128.1, passed the check, and the
driver's nftables input chain — which accepts only the gateway port from the
guest — then dropped every policy-approved CONNECT, which is exactly the
silent timeout the guard exists to prevent.

In the other direction it rejected 192.168.127.254 unconditionally. That
address is special only to libkrun/gvproxy; on QEMU/TAP it is an ordinary
address that may be routable through the guest's masqueraded egress, so the
guard refused a working configuration.

Run the check after the launch plan's network allocation, on both the
freshly-allocated and already-complete paths, and compare IP literals with
that sandbox's selected TAP host. Loopback literals, localhost, and the
documented host aliases that write_host_gateway_aliases seeds to the TAP host
still classify as the gateway host, and the failure names the address. The
gvproxy host-loopback constant returns to being a documentation anchor.

Signed-off-by: Philippe Martin <phmartin@redhat.com>

---------

Signed-off-by: Philippe Martin <phmartin@redhat.com>
2026-09-02 14:56:08 +00:00
Drew Newberry 9ca19e6c80 refactor(compute): decouple gateway driver composition (#2823)
* refactor(compute): decouple gateway driver composition

Move first-party composition and VM process ownership into openshell-gateway, leaving openshell-server backend-independent. Update packaging and build references with the new crate, simplify the compiled-driver boundary, and keep the driver-free gateway path buildable with bundled Z3 tooling.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(telemetry): bound compute driver categories

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(core): keep runtime transport generic

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(compute): complete server driver decoupling

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(compute): preserve driver integrations after rebase

Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>

* fix(compute): preserve docker tracing after decoupling

Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>

* fix(compute): preserve driver behavior after extraction

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(compute): remove MXC policy side channel

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(compute): separate policy delivery from readiness

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
2026-09-01 21:13:45 +00:00
krishicks 197b41371d fix(dev): harden local cluster and gateway startup (#2993)
* fix(helm): refresh kubeconfig for existing k3d clusters

Docker can recreate the k3d load balancer on a new API port. Start existing
clusters and prefer fresh k3d entries so create does not retain a stale
endpoint.

Signed-off-by: Kris Hicks <khicks@nvidia.com>

* fix(dev): conditionally enable local OTLP export

Probe port 4317 before adding OTLP configuration for the VM, Docker,
and Podman gateway tasks. Document the startup behavior and troubleshooting
for local collector availability.

Signed-off-by: Kris Hicks <khicks@nvidia.com>

---------

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-08-27 17:55:01 +00:00
krishicks f68867b869 feat(gateway): identify gateways in exported traces (#2647)
* feat(gateway): add installation name configuration

Add a first-class operator-assigned gateway name with TOML, CLI, environment,
and Helm configuration surfaces. Local gateways default to openshell, while
Helm defaults to the chart fullname; operators sharing a collector across
namespaces or clusters can set a globally distinct name.

Signed-off-by: Kris Hicks <khicks@nvidia.com>

* feat(gateway): identify gateways in exported traces

Attach the configured gateway installation name and compute driver to the
gateway OpenTelemetry resource so operators can filter traces from multiple
installations that share a collector. Forward the gateway name and OTLP
endpoint to managed external drivers so their distinct service resources carry
the same installation identity.

Keep service.name stable per process type, omit blank resource values, and
leave per-span operation names and request attributes unchanged.

Refs #2507

Signed-off-by: Kris Hicks <khicks@nvidia.com>

---------

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-08-27 17:27:34 +00:00
Drew Newberry ef296806f5 feat(sandbox): add canonical main process (#2726)
* feat(sandbox): add canonical main process

Closes #2710

Persist and supervise one canonical workload per sandbox, attach sandbox connect to its retained session, and make every unexpected main-process exit terminal.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(sandbox): simplify canonical main process contract

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): preserve legacy VM main compatibility

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): preserve main status across driver updates

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): satisfy macOS process lint

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): gate Linux exit acknowledgement publisher

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): simplify main process plumbing

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): make controlling tty ioctl portable

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): initialize canonical process environment

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* perf(supervisor): optimize retained main session

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(sandbox): detach main session on ctrl-c

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): use explicit main detach keys

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(dev): atomically stage Docker supervisor

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(test): align Docker main environment assertion

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(sdk): expose canonical main process fields

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-08-20 22:18:23 +00:00
Drew Newberry 7909fb5d0f refactor(compute): unify gateway restart reconciliation (#2743)
* refactor(compute): unify gateway restart reconciliation

Remove the Docker-specific gateway shutdown cleanup and reconcile persisted running intent through ComputeDriver::StartSandbox for Docker, Podman, and VM drivers. Explicitly stopped sandboxes remain stopped.

Refs #2417

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(compute): stop local sandboxes on shutdown

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(e2e): match managed Podman containers

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(compute): synchronize lifecycle sweeps

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-08-20 10:12:41 +00:00
Seth Jennings 0f8fad23c4 feat(sandbox): add stop and start operations (#2653)
* feat(sandbox): add suspend and resume operations

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(server): preserve lifecycle work after cancellation

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(server): reconcile ambiguous lifecycle outcomes

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(server): complete suspended session cleanup

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(vm): preserve suspension state on resume failure

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(server): retry retained lifecycle transitions

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(server): clean sessions after suspend reconciliation

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* test(sandbox): cover deleting suspended sandbox

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(kubernetes): preserve progressing sandbox suspension

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(kubernetes): bound suspend status polling

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(kubernetes): detect legacy sandbox suspension

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(tui): render suspended sandbox phases

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* refactor(sandbox): rename suspend and resume lifecycle

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* perf(server): clean stopped sessions on transition

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(kubernetes): fail fast on rejected stop

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(compute): fence stale restart lifecycle events

Signed-off-by: Seth Jennings <sjenning@redhat.com>

---------

Signed-off-by: Seth Jennings <sjenning@redhat.com>
2026-08-13 06:13:54 +00:00
krishicks 8328412959 feat(vm): export driver traces over OTLP (#2564)
Continue distributed traces across the gateway-to-driver process boundary and
export VM driver spans to the same OTLP/gRPC collector. The driver reports as
the distinct openshell-driver-vm service.

Updated the gateway architecture and configuration reference with a generic
external-driver forwarding contract.

Instrumented:

- Every RemoteComputeDriver RPC injects the active W3C trace context into
  tonic metadata. Managed VM readiness and runtime initialization give startup
  capability probes stable parent operations rather than isolated root spans.
- A tonic service layer creates fixed, low-cardinality server spans for every
  ComputeDriver RPC. New handlers inherit tracing automatically; failures record
  OpenTelemetry error status and the gRPC status code.
- Background provisioning remains attached to CreateSandbox after the RPC
  returns without extending the RPC span lifetime.
- Provisioning records image preparation, bootstrap image resolution, overlay
  preparation, lifecycle configuration, pre-launch hooks, guest preparation,
  and launcher spawn as child spans.
- VM startup reconciliation roots one trace for the persisted-sandbox scan,
  with per-sandbox restore and provision operations beneath it. The root remains
  open until all spawned restore tasks finish.
- Delete cleanup records its own child operation.

Design notes:

- The gateway forwards its configured OTLP endpoint to managed external
  drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits,
  headers, and transport tuning.
- The VM driver has its own tracer provider and service resource so trace
  backends preserve the service boundary.
- RPC operation names come from an explicit method mapping, keeping cardinality
  bounded without parsing the protobuf descriptor set at runtime.
- Propagation uses a remote SpanContext for spawned provisioning. This keeps
  one trace while allowing the CreateSandbox server span to finish when the
  RPC response is sent.
- Startup restoration is independent of gateway requests. It begins at the VM
  driver reconciliation span rather than attaching to an unrelated RPC.
- Existing tracing events remain on the logging path. The OpenTelemetry layer
  exports spans only and excludes the SDK exporter callsites to avoid recursive
  traces.
- Export configuration failures do not prevent the driver from serving, and
  buffered spans are drained during graceful shutdown.
- Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC
  outcomes without recording credentials, sandbox tokens, or request query
  parameters.

Refs #2507

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-08-03 20:52:22 +00:00
Drew Newberry d8010efe29 fix(vm): scope rootfs cache by openshell version (#1587)
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-05-27 13:00:47 -05:00
Russell Bryant c5d1d76d94 refactor(sandbox): replace iptables with nftables for network policy enforcement (#1401)
Migrate all sandbox and VM driver network policy enforcement from
iptables to nftables. nftables provides atomic ruleset loading, a
cleaner rule syntax, and is the standard netfilter interface in modern
kernels.

Sandbox bypass enforcement (openshell-sandbox):
- Replace iptables chain of individual rule insertions with a single
  atomic nftables ruleset load via nft -f
- New nft_ruleset module with pure functions for ruleset generation
  and unit tests
- Combine log and reject rules in one inet family table (handles both
  IPv4 and IPv6 in a single ruleset)
- Fall back to reject-only ruleset when kernel lacks nft_log support
- Enable net.netfilter.nf_log_all_netns so log rules work from
  non-init network namespaces
- Use temp file for nft ruleset loading instead of stdin for
  compatibility with minimal VM guest environments

VM TAP networking (openshell-driver-vm):
- Replace iptables NAT/forwarding rules with nftables equivalents
- New nft_ruleset module for TAP network rule generation with unit
  tests
- Atomic table-per-TAP-device lifecycle (create/destroy)
- Host-side rules provide NAT infrastructure and defense-in-depth
  isolation (input chain restricts VM to gateway port only, forward
  chain blocks unsolicited inbound); primary security enforcement
  happens inside the VM guest via the sandbox supervisor's own rules

VM init script:
- Load nft kernel modules at sandbox init
- Enable nf_log_all_netns sysctl for bypass detection logging

OCSF / docs:
- Update firewall rule engine references from iptables to nftables
- Document host firewall interaction model and two-layer enforcement
  architecture in VM driver README and compute drivers reference

Closes #1335

Signed-off-by: Russell Bryant <rbryant@redhat.com>
2026-05-18 16:37:17 -07:00
Drew Newberry f819f7dcb5 fix(vm): restore sandboxes after gateway restart (#1407) 2026-05-15 15:08:59 -07:00
Drew Newberry 910d3f09f2 feat(vm): boot sandboxes from ext4 root disks (#1263)
* feat(vm): boot sandboxes from ext4 root disks

* feat(vm): boot sandboxes with writable overlay disk

* fix(sandbox): improve sandbox provisioning behavior

* fix(vm): repair sandbox image provisioning

* fix(vm): clean up rebase fallout

* perf(vm): cache empty overlay images

* perf(vm): skip repeated sandbox ownership repair

* fix(vm): address lifecycle review issues

* fix(vm): repair gateway vm startup

* wip
2026-05-15 13:23:10 -07:00
Taylor MutchandDrew Newberry b61a98dbad feat(gateway): add TOML configuration file (RFC 0003) (#1317)
* feat(gateway): add TOML configuration file (RFC 0003)

Introduces an opt-in --config / OPENSHELL_GATEWAY_CONFIG flag that loads a
TOML file with gateway-wide settings and per-driver tables. Source
precedence is CLI > env > file > built-in default, implemented via clap's
ValueSource so existing flags and env vars keep their priority.

Driver crates (kubernetes, docker, podman, vm) now derive Deserialize on
their config structs. SupervisorSideloadMethod gains Deserialize with
kebab-case rename. A per-driver inheritance allowlist on the loader side
overlays [openshell.gateway] shared defaults (default_image,
supervisor_image, image_pull_policy, guest_tls_*, ssh_handshake_skew_secs,
client_tls_secret_name, host_gateway_ip, enable_user_namespaces) onto
each [openshell.drivers.<name>] table before deserialization.

The Helm chart renders a new gateway-config ConfigMap and mounts it at
/etc/openshell/gateway.toml. The migrated OPENSHELL_* env entries are
dropped from the StatefulSet — only the Secret-backed
OPENSHELL_SSH_HANDSHAKE_SECRET remains. database_url stays on --db-url.

Adds examples/gateway/gateway.example.toml and updates architecture/gateway.md
with the source precedence and inheritance rules.

* docs(gateway): drop ssh_handshake_skew_secs and ssh_handshake_secret from examples

Both fields are scheduled for removal. Remove the example values and the
env-only note so the gateway.toml example and the architecture doc stop
recommending settings that will not exist much longer.

* docs(rfc): correct OPENSHELL_CONFIG to OPENSHELL_GATEWAY_CONFIG in RFC 0003

* docs(gateway): add per-driver TOML example configurations

Adds focused single-driver examples next to the comprehensive
gateway.example.toml: kubernetes, docker, podman, and microvm. Each one
demonstrates the realistic settings for that driver plus how shared
[openshell.gateway] defaults inherit into the driver table.

A new unit test (`checked_in_examples_parse`) loads every example through
the config_file loader so schema drift fails CI rather than silently
shipping a broken example.

* refactor(gateway): drop image_pull_policy from shared inheritance

Kubernetes and Podman use mutually-incompatible vocabularies for the same
TOML key:

  - Kubernetes: `Always | IfNotPresent | Never` (free-form string passed
    verbatim to the K8s API).
  - Podman: `always | missing | never | newer` (strict lowercase enum
    deserialised into `ImagePullPolicy`).

No value means the same thing in both drivers. Sharing the key at
`[openshell.gateway]` scope and inheriting it into every active driver's
table meant any value safe for one driver was either wrong or silently
dropped for the other (`IfNotPresent` → `ImagePullPolicy::Missing` after
`.unwrap_or_default()`). Operators run one driver per gateway, so the
"shared default" never pays for itself.

Make `image_pull_policy` driver-local:

  - Remove the field from `GatewayFileSection` and from
    `inheritable_keys()` for both Kubernetes and Podman.
  - Drop the file→`RunArgs` merge for the gateway-scope key.
  - Stop unconditionally clobbering the driver value with
    `config.sandbox_image_pull_policy` in the runtime wiring — only apply
    the CLI/env override when it was set (and, for Podman, only when it
    parses into the lowercase enum).
  - Move the key under `[openshell.drivers.kubernetes]` and
    `[openshell.drivers.podman]` in every example, the RFC, the
    architecture doc, and the Helm-rendered gateway ConfigMap.

The supervisor pull policy follows the same shape: it is K8s-only and
moves into `[openshell.drivers.kubernetes]` alongside `image_pull_policy`
in the Helm template.

* fix(gateway): address review feedback on TOML configuration

Resolves the P1 and P2 issues raised in PR #1317:

- Helm gateway ConfigMap moves `grpc_endpoint` under
  `[openshell.drivers.kubernetes]` so the default install no longer fails
  the gateway's `deny_unknown_fields` schema check.
- `kubernetes_config_from_file` and `podman_config_from_file` only let
  the gateway-wide CLI/env `grpc_endpoint` overwrite the driver-table
  value when it was actually supplied, preserving file-only configs.
- Kubernetes driver default `image_pull_policy` is now empty (was Podman
  vocabulary "missing"), so default deployments let the Kubernetes API
  apply its own policy instead of being rejected.
- New `disable_tls` gateway field plumbs `.Values.server.disableTls`
  through the TOML ConfigMap instead of relying on env vars dropped from
  the StatefulSet.
- StatefulSet pod template now carries a `checksum/gateway-config`
  annotation so `helm upgrade` rolls pods when the ConfigMap changes.
- Auxiliary listener resolution preserves the full `SocketAddr` from
  `health_bind_address` / `metrics_bind_address`, so a loopback-pinned
  health port is not silently relocated onto the public bind address.
- `ssh_session_ttl_secs` from the file is now applied to `Config` (it
  was previously accepted by the loader but never read).

New regression coverage: cli-level merge tests for the new fields plus
helm-unittest assertions for the ConfigMap shape, checksum annotation,
and `disable_tls` rendering.

* docs(gateway): consolidate gateway TOML examples into docs reference

Replaces the per-driver example files under examples/gateway/ with a
single published reference page at docs/reference/gateway-config.mdx
covering source precedence, layout, the full example, and the four
per-driver examples (Kubernetes, Docker, Podman, microVM). Drew flagged
during PR #1317 review that the examples belong with the user-facing
docs rather than in a sibling examples/ directory.

The cross-references in architecture/gateway.md and RFC 0003 are updated
to point at the new docs page; the round-trip test in config_file.rs is
removed (schema coverage stays on the inline parses_full_example test
and per-field merge tests — doc-snippet drift belongs in a separate
docs-lint, not in a cross-tree Rust unit test).

* refactor(core): move DEFAULT_K8S_NAMESPACE into K8s driver

The constant is Kubernetes-specific (used only by KubernetesComputeConfig's
Default impl) and does not belong in openshell-core. Relocate it to the
driver crate that owns the K8s vocabulary; openshell-core retains only
truly cross-cutting defaults.

* refactor(core): move Podman bridge default into Podman driver

DEFAULT_NETWORK_NAME is Podman vocabulary, consumed only by the Podman
driver. Also drops the unused DEFAULT_IMAGE_PULL_POLICY constant.

* docs(auth): scrub remaining SSH handshake secret references

Sweeps the trailing mentions left after the rebase: the gateway
config-file module doc, the Helm gateway-config ConfigMap header,
the gateway-config.mdx env-only note, and the RPM systemd unit
comment for init-gateway-env.sh.

* docs(gateway): clarify OPENSHELL_GRPC_ENDPOINT applies to all drivers

The previous comment implied the callback endpoint was Kubernetes-only,
but the value is propagated to every compute driver (Kubernetes, Docker,
Podman, VM) and must be reachable from wherever the sandbox runs.

* refactor(gateway): move driver options into config (#1394)

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(e2e): regenerate gateway config via TOML for docker + podman harnesses

The gateway CLI flags moved into TOML config tables in 560550d2 (#1394),
which made every existing e2e/with-{docker,podman}-gateway.sh invocation
fail with "unexpected argument '--sandbox-namespace'" (and a long tail
of similar driver-specific options) before the gateway could even bind.

Replace the obsolete CLI flags with a synthesized
`[openshell.drivers.<driver>]` table written to `${STATE_DIR}/gateway.toml`
and passed via the new `--config` flag. Only the gateway-wide flags
that survived 560550d2 (bind-address, port, drivers, db-url, tls-*,
disable-tls, log-level, health-port) stay on the command line.

Both scripts get a small `toml_string` helper to properly TOML-quote
the values (the previous `%q` printf format produced bash-escape, not
TOML-escape). The Docker harness also corrects two field names that
diverged from the driver schema: `docker_network_name` →
`network_name`, and the supervisor binary/image plumbing now reads
through to `supervisor_bin` / `supervisor_image` in the same table.

The Podman harness drops `--ssh-gateway-port` (deleted in 560550d2 —
gRPC + SSH are multiplexed on the same port now) and substitutes
`network_name` + `gateway_port` for the obsolete `--sandbox-namespace`
(which the Podman driver never had as a typed field).

* fix(core): swap bind-only 0.0.0.0 SSH gateway host for cluster URL host

CLI's resolve_ssh_gateway treated 0.0.0.0 as a loopback "keep as-is"
when the cluster URL was also loopback, so the SSH proxy connected to
0.0.0.0:port. The unspecified address is never a valid connect target
and is not present in any TLS cert SAN, which produced BadCertificate
TLS handshake failures during `openshell sandbox create -- ...` in
docker/podman e2e (e.g. bypass_detection).

Resolution: when the server returns 0.0.0.0 or :: as the gateway host
and both endpoints are loopback, fall back to the cluster URL's host
(which the CLI is already using to reach the gateway, so it must
resolve and match the cert).

* fix(e2e): repair podman harness on macOS

Podman 5.x with the applehv/libkrun provider no longer creates the legacy
~/.local/share/containers/podman/machine/podman.sock symlink, and
`podman system service` is a Linux-only subcommand — the macOS client
delegates the API service to the VM. Both assumptions in the harness
were stale, so the script tried to start a temporary service that podman
rejected with "unknown flag: --time".

- Discover the macOS socket via `podman machine inspect` instead of the
  hardcoded path.
- On Darwin, fail fast with a "start podman machine" message rather than
  attempting the Linux-only `podman system service` fallback.
- Write socket_path into [openshell.drivers.podman] so the in-process
  driver picks up the discovered socket; the driver reads TOML only
  after the config refactor (560550d2), so OPENSHELL_PODMAN_SOCKET alone
  was no longer enough.

* fix(server): use clone_from for TLS client CA assignment

clippy 1.95.0 rejects assigning the result of `Clone::clone()` to an
existing variable under `-D warnings` (`assigning_clones`). Switch to
`clone_from(&...)` to satisfy the lint and avoid the redundant
allocation.

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
2026-05-15 12:43:48 -07:00
Russell Bryant 44e843ede3 feat(vm): fall back to Podman socket when Docker is unavailable (#1370)
* test(e2e): add bypass detection test for sandbox REJECT rules

Validates that direct TCP connections bypassing the HTTP CONNECT proxy
are rejected immediately (ECONNREFUSED) rather than hanging until a
network timeout. This is implementation-agnostic — it tests observable
kernel behavior regardless of whether rules are installed via iptables
or nftables.

* feat(vm): fall back to Podman socket when Docker is unavailable

The VM driver resolves sandbox images from a local container engine
before falling back to registry pulls. Previously it only checked
Docker. Now it tries the Podman socket as a fallback, since Podman
exposes a Docker-compatible API via bollard.

On Linux, enable the Podman API socket with:
  systemctl --user start podman.socket

Signed-off-by: Russell Bryant <russell.bryant@gmail.com>

* chore(vm): fix formatting

Signed-off-by: Russell Bryant <russell.bryant@gmail.com>

* refactor(vm): rename docker-specific identifiers to engine-neutral names

Rename resolve_local_docker_image → resolve_local_container_image,
local_docker_image_platform_mismatch → local_image_platform_mismatch,
and the local docker variable → engine since the function now supports
both Docker and Podman backends.

Signed-off-by: Russell Bryant <russell.bryant@gmail.com>

---------

Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
2026-05-14 22:19:03 -07:00
Drew Newberry 529be37fd4 chore(installer): promote package install script (#1261) 2026-05-08 17:04:15 -07:00
Drew Newberry 645b880513 feat(install): add rpm dev installer support (#1262)
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-05-08 09:45:30 -05:00
Drew Newberry 8ab5ee875b fix(vm): harden compute driver socket (#1248) 2026-05-07 20:03:16 -07:00
Drew Newberry 028763d4db refactor(vm): remove legacy openshell-vm crate (#1239)
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-05-07 10:50:20 -07:00
Drew Newberry 938005584a fix(packaging): let gateway auto-detect package driver (#1236)
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-05-07 09:13:43 -07:00
Drew Newberry cc2114e267 docs(architecture): reset subsystem docs (#1184) 2026-05-07 08:59:02 -07:00
Drew Newberry fe41d679a4 fix(installer): install release formula from Homebrew tap (#1222) 2026-05-06 18:42:46 -07:00
Drew Newberry 5949b2fa34 feat(installer): support macOS dev installs (#1183) 2026-05-06 15:50:07 -07:00
Drew Newberry 4d388d2677 ci(vm): cleanup vm build infra (#1186) 2026-05-06 08:58:02 -07:00
Drew Newberry 2e0afeabe1 feat(vm): derive guest rootfs from sandbox images (#957) 2026-05-03 23:23:30 -07:00
Drew Newberry 9751872b2d feat(release): add Debian package publishing (#1069) 2026-05-01 07:51:45 -07:00
vince-brisebois 2adddaa6b2 feat: Adding qemu vm driver support with GPU pass-through (#992)
* Adding qemu vm driver support with GPU pass-through

Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com>

* Add GPU rootfs variant, harden VFIO binding, and fix networking and supervisor reliability issues discovered during GPU VM bring-up.

Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com>

---------

Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com>
2026-04-29 09:18:27 -07:00
Piotr Mlocek df38d1f66f feat(ci): add Markdown and Mermaid linting (#933) 2026-04-24 11:27:02 -07:00
Piotr Mlocek d0a29b64eb fix(driver-vm): preflight supervisor cross-compile toolchain in start.sh (#931) 2026-04-23 08:27:27 -07:00
Drew Newberry 4483c860e0 feat(server,driver-vm,e2e): gateway-owned readiness + VM compute driver e2e (#901) 2026-04-22 16:42:35 -07:00
Drew Newberry b39f5aaa0c feat(install-vm): install gateway + vm driver, add --driver-dir resolution (#887) 2026-04-20 15:15:01 -07:00
Drew Newberry e4d6f92d9b feat(vm): add standalone libkrun compute driver (#858) 2026-04-17 08:50:57 -07:00