* fix(vm): flush guest filesystem before a planned sandbox stop
A planned stop of a MicroVM sandbox ended the VM before the guest wrote its dirty pages to disk, so files written shortly before the stop were missing after the next start.
The driver closed the host supervisor's parent-liveness pipe before it sent SIGTERM. The supervisor exits at once on that end-of-file, so it never ran its orderly shutdown: terminate the workload, then ask the guest to terminate. Signal the supervisor first, keep the pipe open until it has exited, and give it a grace period derived from the supervisor's own bounds and kept below the gateway's reclaim deadline.
In the guest runtime, syncfs the root filesystem after the process tree is terminated and before the termination is acknowledged, bounded to 3 seconds and best effort. Only the VM backend, which uses a vsock boundary listener, flushes; the container drivers share this runtime and are unchanged.
A sandbox whose main process has already exited is not covered.
Closes#4185
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): warn about unconfirmed flush after supervisor kill
Preserve the supervisor exit status so a planned stop warns when a signal
killed the supervisor and guest writeback was not confirmed. Keep normal
workload exit codes quiet, including 143 after a handled SIGTERM.
Extend the existing process tests to check the warning after a forced kill
and its absence after an orderly shutdown.
Refs #4185
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): satisfy shutdown timeout control-flow lint
Use if let for the single successful timeout pattern while preserving
the supervisor exit status and forced-kill error handling.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): preserve shutdown merge resolutions during rebase
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* docs(vm): explain planned stop durability and flush limits
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(snap): only enable gateway user service for one user
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* docs(snap): correct snap docs for the gateway user service
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): add documentation of global to per-user enablement migration
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(ocsf): report https scheme for TLS-terminated requests
Closes#4562
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(ocsf): use transport scheme on passthrough relay
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
---------
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(server): warn on dangling provider refs at startup
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(server): load providers concurrently at startup
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
---------
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
helm push uploads the chart manifest by digest, then fetches it back to
apply the version tag. GHCR sometimes returns 404 for a manifest it just
accepted, failing the push with 'failed to perform "Tag" on
destination: <digest>: not found'. This failed Release Dev on Oct 5 and
the v0.1.5-pre.2 Release Tag run after qualification passed.
Retry each chart push up to four times with a short backoff. A retry
finds the blobs and manifest already present and only applies the tag.
Closes#4577
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(release): name the failed stage and failing jobs in pre-release failure notices
The pre-release failure notification said only that the run failed.
Name the failed stage (build, publication, or release qualification),
link the tag, and list each failing job of the run attempt with a link.
Skip the aggregate Release Qualification job unless it is the only
failure, and still send the message if the jobs lookup fails.
Closes#4718
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(release): simplify the failure notice to a list of failed jobs
Drop the stage headline and the aggregate-job filtering. Keep the
existing message and append a link to each failed job.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(release): promote qualified prereleases to stable
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(release): preserve qualified images during stable publication
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(release): run shared release jobs with Nix
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(release): recheck latest before publishing shared channels
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(tmachine): link libstdc++ statically and verify static musl binaries
On macOS hosts, the aarch64 musl openshell-prover that
build-artifacts-binaries produced was not static. z3-sys asks for a
dynamic libstdc++ (-Wl,-Bdynamic -lstdc++), and the musl cross compiler
wrapper resolved libstdc++.so. The result was a static executable with a
DT_NEEDED entry for libstdc++.so.6 and no interpreter, which crashed
during C++ static initialization. The local Debian package from #4598
ships this prover without running it.
Build the musl binaries with CXXSTDLIB=static:-bundle=stdc++ so z3-sys
requests a static libstdc++. CXXSTDLIB applies to every target and
Darwin has no libstdc++, so set it only on this musl build rather than in
the shared toolchain environment. The bundled form, static=stdc++, fails
because rustc cannot find libstdc++.a on its own search path.
Run tasks/scripts/verify-static-binary.sh on the musl CLI, sandbox and
prover, as the release does, so a binary that is not fully static fails
the local build.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(tmachine): build local RPM artifacts
Add build-artifacts-rpm, which builds the CLI and gateway RPMs that
tmachine's rpm installer expects at artifacts/packages/rpm/openshell.rpm
and artifacts/packages/rpm/openshell-gateway.rpm. It runs
`packit build locally` with openshell.spec in the Fedora 44 image that
build-rpm.yml uses, for the host's Linux architecture, and packages the
binaries staged by build-artifacts-binaries. The package version defaults
to the development version that branch E2E CI computes.
The aggregate build-artifacts app runs the RPM step after the DEB step.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Evan Lezar <elezar@nvidia.com>
OCSF 1.8 requires type_id on the os object, but device.os carried only
a name. Events now report Linux (200), Windows (100), macOS (300), or
Other (99) with the platform name as the type label. The os object
schema is vendored so tests validate it.
Refs #2912
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Authentication events now carry the connection's immediate peer as
src_endpoint, or edge-tunnel for WebSocket-tunneled requests, and the
request's x-request-id as metadata.correlation_uid when it is a UUID.
Refs #2912
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling.
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: divesh <dgude@nvidia.com>
Co-authored-by: divesh <dgude@nvidia.com>
Authenticators now return the mechanism and a reason category with the
status they reject a request with, and the gateway router records both
in the Authentication event. Callers still receive the original status.
Refs #2912
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* perf(server): drop async_trait from mutation replay
Mutation is never used as a trait object, so the boxed futures from
#[tonic::async_trait] buy nothing. Declare its methods as
-> impl Future + Send instead. This keeps the Send guarantee, removes a
heap allocation per call, and cuts a cold openshell-server lib build from
~60s to ~36s.
Box the approve_all_draft_chunks future at its call sites, since it is no
longer boxed implicitly and trips clippy::large_futures.
Closes#4618
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* perf(supervisor): drop async_trait from generic-only gateway traits
StartupGateway, PolicyGatewayClient, and ReportClient are only used
through generic parameters, never as trait objects. Declare their methods
as -> impl Future + Send instead of boxing each future.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(ci): make conformance-only e2e selection explicit
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(ci): use mapped NodePort for kind gateway sessions
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(ci): waive SSH transport recovery only on Kubernetes
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(e2e): log a notice when runners skip Rust e2e tests
Emit a GitHub Actions notice from the Docker, Podman, Kubernetes, and VM
runners when e2e-conformance-only skips the Rust tests, so a narrowed lane
is visible in the job summary instead of silently reporting success.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(testing): move e2e conformance-only selector to shared section
The selector applies to all four driver runners, so document it with the
Rust CLI e2e suites instead of under the Kubernetes task variants.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Matthew Grossman <mgrossman@nvidia.com>
The endpoint-status startup sweep ran before the gateway installed its
Prometheus recorder, so the lock waits, timeouts, errors and hold times
it recorded went to the no-op recorder and only its warnings reached
the log. The sweep retries lock timeouts for up to three lock waits
before startup fails, and a slow sweep that still succeeds is what
operators want to see in the lock metrics.
When metrics are enabled, the recorder is now installed before startup
reconciliation, and the metrics listener still binds with the other
listeners. A new pod's first scrape therefore already holds the sweep's
samples, so they show in the raw counter and histogram count values,
not in rate() or increase(). A sweep that fails still exits before any
listener binds, so its timeouts remain log-only.
The startup lock-timeout test now waits for the sweep's timeout counter
instead of a fixed sleep and checks its timeout and wait samples, and
the debug skill says which startup timeouts reach the metrics and how
to read them.
Part of #4307Closes#4523
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Requests the gateway rejects before consulting any authenticator now emit
an OCSF Authentication [3002] event: no credential presented, mTLS
required without a client certificate, or an operator certificate
combined with an authorization header. Each event carries a reason
category in status_detail and never the presented credential. The
response returned to the caller is unchanged.
openshell-ocsf gains the Authentication class with user and service
objects, and a shared OcsfCapture test layer. Schema validation now
rejects attributes the class does not define and profile attributes
whose profile the event does not declare.
Refs #2912
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(vm): add host bind mount support via virtiofs (libkrun)
Introduce host directory bind mounts for libkrun VM sandboxes using
the virtiofs4 API with simplified permission semantics. The gateway
accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML
table and forwards it to the VM driver subprocess.
Mounts are specified per-sandbox in driver_config.mounts[] with
source, target, optional read_only (default: read-only), and an
optional `type` that must be `bind`, so Docker and Podman bind entries
(without `selinux_label`) parse unchanged.
At sandbox validation the driver rejects mounts unless bind mounts
are enabled and resource admission is disabled. It rejects GPU and
other QEMU launches, sources with control characters, and more mounts
than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and
checks targets against reserved VM-internal directories and each other
(no nested targets). The QEMU check runs again once the launch backend
is resolved.
Host source directories are checked at provisioning, before image
preparation, so a briefly missing source fails provisioning visibly
instead of denying recovery of a persisted sandbox. Missing,
inaccessible, symlinked, non-directory, and unreadable sources are
rejected: libkrun does not follow a final symlink, and on macOS it
panics during guest boot when it cannot open the share root.
The driver generates virtiofs tags and always writes a guest-side
manifest into the overlay disk, together with the init drop-ins, so an
image-supplied manifest can never drive guest mounts. The launcher
parses its tab-separated --vm-mount arguments from the right, so a
source can never shift the tag or mode, and the driver logs each
shared directory at launch.
Guest init mounts the shares after every /sandbox ownership fixup
and the init drop-ins, so recursive chowns never reach host files.
It refuses targets that traverse symlinks in the image, hands newly
created mount point parents under /sandbox to the sandbox user, and
skips an empty manifest.
krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes
still boot sandboxes without mounts.
The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS
explicitly; the pinned libkrunfw base config already enables both.
A vm_bind_mount e2e test checks a read-write share round-trips data
with the host, accepts a new file, directory, and nested file that the
host then sees, and that a read-only share rejects writes. libkrun on
rootless Linux refuses file creation by a guest user whose UID/GID
differ from its own, so the VM e2e gateway runs the sandbox as the
gateway user there. It enables bind mounts like the Docker and Podman
e2e gateways.
The runtimes docs cover the mount limit, the symlink restriction, the
guest-kernel exposure, and the rootless Linux UID/GID note. The
debug-openshell-cluster skill lists VM bind mount checks.
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
* fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime
The Nix vm-runtime package fetches the rolling vm-runtime release with
fixed hashes pinned on 2026-09-16. The release was rebuilt on
2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (#3451),
but the hashes were not updated, so CI kept substituting the
2026-09-15 runtime (libkrun v1.17.4) from the binary cache.
That libkrun lacks krun_add_virtiofs4, which VM bind mounts need.
Point the hashes at the current release assets.
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
---------
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
* feat(sdk/go)!: replace positional labels with functional Create options
Move labels and annotations from separate parameter positions into
functional options (WithLabels, WithAnnotations) on Create() and
CreateFromTemplate().
Before: Create(ctx, ws, name, spec, labels, CreateOptions{Annotations: ann})
After: Create(ctx, ws, name, spec, WithLabels(labels), WithAnnotations(ann))
This unifies two resource metadata fields that were inconsistently
placed (labels as positional, annotations in a struct) into a single
extensible options pattern matching the existing LogOption convention.
BREAKING CHANGE: Create() and CreateFromTemplate() signatures changed.
All callers must use WithLabels() and WithAnnotations() functional
options instead of positional labels and CreateOptions struct.
Closes#2807
Signed-off-by: Roland Huß <rhuss@redhat.com>
* fix(sdk/go): ignore nil Create options
Callers of the previous signature passed nil for the positional labels
parameter to create an unlabeled sandbox:
Create(ctx, ws, name, spec, nil)
That call still compiles against the new variadic signature, because an
untyped nil converts to CreateOption. ApplyCreateOptions then invoked it
and panicked with a nil pointer dereference, so the migration was unsafe
for existing consumers.
Skip nil options in ApplyCreateOptions and cover the legacy trailing-nil
call with regression tests against ApplyCreateOptions directly, the fake
client (Create and CreateFromTemplate), and the gRPC client.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* refactor(sdk/go): apply Create options through the shared options helper
Signed-off-by: Roland Huß <rhuss@redhat.com>
* docs(sdk/go): use WithLabels in CreateSandboxFromTemplate examples
Signed-off-by: Roland Huß <rhuss@redhat.com>
---------
Signed-off-by: Roland Huß <rhuss@redhat.com>
* fix(podman): trust an operator-supplied additional CA for direct egress
Since v0.1.2 the Podman supervisor runs as its own container, separate
from the workload image, so it can no longer inherit a private CA
baked into the workload's system trust store. Policy-allowed,
inspected HTTPS requests to destinations signed by that CA fail
upstream TLS establishment, and the only workaround is rebuilding the
supervisor image with the CA baked in.
Add a Podman-only additional_ca_bundle setting, independent of the
existing proxy_ca_bundle (which requires https_proxy), that bind-mounts
an operator PEM into the supervisor container and folds it into the
upstream trust store alongside the system CA bundle. Validated on the
gateway host at sandbox-create time, matching the existing corporate
proxy CA bundle's validation, so a misconfigured path or
certificate-free bundle fails closed immediately rather than producing
a failed sandbox once the supervisor container starts.
Scoped to Podman only, reusing the existing additive-bundle mechanism
without new gateway.toml schema, proto fields, or cross-driver
plumbing -- a narrower alternative to the in-flight cross-driver
implementation in #3292, which remains the intended long-term
direction.
Fixes#3781.
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(podman): keep additional CA mount on the supervisor
build_isolation_specs() keeps only trusted_mount() destinations on the
supervisor, and the additional CA path was not in that list. The bind
mount stayed on the workload while the supervisor received
--additional-ca-bundle for a file it did not have, so startup failed.
Add the path to trusted_mount() and cover the final supervisor and
workload specs with a test. Correct the docs and a code comment to say
the bundle is validated when the driver starts, and add Podman
troubleshooting guidance to the debug-openshell-cluster skill.
Signed-off-by: politerealism <burdcat17@gmail.com>
---------
Signed-off-by: politerealism <burdcat17@gmail.com>
* feat(providers): report bounded token-grant failures
Signed-off-by: Shiju <shiju@nvidia.com>
* test(providers): align policy metadata and TLS lockfile
Signed-off-by: Shiju <shiju@nvidia.com>
* test(providers): fix trusted grant recovery fixture
Signed-off-by: Shiju <shiju@nvidia.com>
* test(providers): enable denial API in recovery fixture
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(providers): simplify token-grant failure reporting
Remove duplication and dead paths from the token-grant failure
reporting without changing classification, the workload-visible
502 response, or the wire format.
- Build the gRPC failure envelope with tonic-types, store the typed
failure in EndpointFailure, and share one request-ID rule and one
endpoint-status authority check.
- Merge the gateway exchange client wrapper, drop forwarding
validators, and name the request field when a reported cause is
rejected.
- Inject fallback credentials only from live provider state and
delete the unread per-connection credential copy. Keep one 502
writer per path and restore the base token-cache handling.
- Check the token endpoint after the cache lookup, still before any
credential acquisition.
- Document 429 and 5xx classification as implemented, and drop tests
that repeat retained assertions through the same code path.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(providers): trim token-grant failure tests and supervisor API
Cut the token-grant failure tests to one test per distinct behavior
across core, gateway, supervisor, CLI and Go, and replace the 794-line
closure E2E with status assertions in the existing route-recovery E2E:
the refused grant appears in sandbox status with the response's
request ID, code, stage and upstream status, and the matching grant's
later success clears it.
- Pin the live provider snapshot with one helper used by inspected and
fallback forwarding, and inject selector-matched grants through
inject_for_selectors, which borrows the pinned context. Move the
test-only wrapper into test_support.
- Drop the 35-second wrappers around the gateway grant RPC. The shared
channel bounds the connection attempt and the gateway bounds its
token-service calls.
- Log the token service's OAuth error and error_description, without
control characters and limited to 256 characters each, with the
request ID in the gateway or supervisor log. Status and the workload
response still carry only the bounded cause.
- Keep one check that an untrusted token-service certificate cannot
assert a cause.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): restore test import and features lockfile
The injection tests named EndpointObservationSender through the parent
module, which no longer imports it. The features suite lockfile kept the
tonic TLS dependency that the removed E2E fixture added.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(providers): fix body-bound fixture width and table type
Pass the response-size width as a named argument; an implicitly captured
constant panics at runtime. Name the mutation type in the validation
table for Clippy.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(providers): pad exact-bound body without a format width
MAX_TOKEN_RESPONSE_BYTES exceeds u16::MAX, the largest format width, so
the padded fixture panicked. Pad with repeat_n as before.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(server): update public RPC schema hash after rebase
Main added the ListProvidersRequest label selector. The merged public
schema hash comes from the Branch Checks storage_proto test output.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): keep token-service error text out of grant logs
A token service can echo a submitted credential in its OAuth error or
error_description. Log only the classified failure code, HTTP status and
request ID, and cover the canary echo in a dedicated test binary.
Signed-off-by: Shiju <shiju@nvidia.com>
* docs(providers): tighten token grant failure documentation
Signed-off-by: Shiju <shiju@nvidia.com>
* docs(providers): condense token grant failure guidance
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* ci(release): notify Slack when a prerelease succeeds
Post a top-level message to the triage webhook when a prerelease completes, so recovery after failures is visible. The message omits the triage mention.
Closes#4604
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(release): link the prerelease tag in the success notification
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Simon Scatton