Move behavioral suites under e2e/suites and extract shared Rust tooling into e2e/support/rust. Organize conformance tests into one Cargo target per area with story modules containing the tests and assertions. Remove the separate scenario crate and registry, and split lifecycle coverage into independent stop/start and stopped-deletion stories.
Reuse shared helpers from the existing Rust harness, preserve artifact packaging and focused test selection, and include tooling and pure assertion tests in test:rust and branch checks. Update authoring guidance for required behavior and retained failures with explicit waivers.
Closes#4119
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Keep the expected draft type, pure assertion helper, and its three unit tests together in a dedicated scenario module before reorganizing the conformance suite. Preserve the policy behavior and assertion checks.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(sandbox): accept local connections natively on loopback-confined sockets
On kernels without SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV (RHEL 9 / RHCOS
5.14), the broker cannot safely write an accepted peer address into
workload memory, so accept/accept4 with a peer-address buffer failed with
EOPNOTSUPP. Static binaries and Go servers, which issue the raw syscall,
could not accept connections at all.
Move local acceptance into the kernel and replace per-accept inspection
with standing kernel confinement:
- Bind every broker-created TCP/UDP socket to the loopback device before
injection and verify the binding. Accepted sockets inherit it, so they
can neither receive routed ingress nor emit routed egress.
- Stop notifying accept/accept4 and remove the accept workers, the
SIGUSR2 accept-interrupt monitor, and the 64-accept ceiling.
- Continue getpeername natively for every descriptor except relayed
connections.
- Deny interface-selection socket options and MSG_FASTOPEN sends from
scalar syscall arguments, so confinement does not depend on the
capability state of the user namespace that owns the network namespace
and covers unregistered descriptors.
- Drop loopback-interface ingress on the non-loopback TCP control
listener before it listens, so a workload cannot reach it by
reconnecting a natively accepted socket.
- Mark descriptors above stdio close-on-exec in every workload pre_exec
path.
- Require an active confinement probe at qualification and carry it as
required authenticated audit evidence.
Document OpenShift 4.19 as the minimum release: RHCOS kernels for 4.16
through 4.18 are built without Landlock.
Closes#4058
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): close review gaps in native local accept confinement
- Reject a TCP boundary control listener on a loopback address,
including IPv4-mapped loopback. Production drivers bind the
unspecified address; a loopback listener would be reachable from
workload sockets the broker does not track.
- Extend the confinement probe to IPv6 and prove that an accepted
socket keeps its loopback binding after an AF_UNSPEC disconnect.
- Pin the accepted-peer contract with tests: a loopback-bound listener
admits only clients in the sandbox network namespace, and workload
sockets cannot bind a non-loopback source address.
- Correct the support matrix: accepted connections are bounded by
per-process descriptor limits, the runtime PID limit, and the sandbox
memory limit, not by a broker limit.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): replace unsafe socket calls with socket2 and rustix
Socket confinement now uses socket2 for device binding and the ingress
filter, and rustix for interface lookup and AF_UNSPEC disconnect, so the
module contains no unsafe code. The control-listener filter is no longer
locked: no safe API exposes SO_LOCK_FILTER, and the listener descriptor
never leaves the trusted sandbox process, which marks every descriptor
above stdio close-on-exec before running workload code.
Tests added for native accept use rustix and socket2 instead of raw libc
calls. rustix issues accept4 and getpeername as raw syscalls, so the
direct-syscall test keeps its meaning. The close-on-exec sweep test runs
in a re-executed test process instead of a forked child.
The close_range(CLOSE_RANGE_CLOEXEC) syscall in the pre_exec hook remains
the only unsafe added by this branch; neither rustix nor nix wraps it.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): allowlist workload socket families
The workload seccomp filter denied only AF_PACKET, AF_BLUETOOTH, and
AF_VSOCK, and the broker continued socket() for every non-INET family.
Several protocol families, including AF_RXRPC, AF_SMC, and AF_KCM, carry
traffic over kernel-owned sockets that the broker never creates and that
are not bound to loopback.
Allow only AF_UNIX, AF_NETLINK (still limited to NETLINK_ROUTE), and the
brokered AF_INET/AF_INET6 families, and restrict socketpair(2) to
AF_UNIX. Both decisions use the scalar domain argument. The broker
independently refuses non-INET families other than AF_UNIX and
AF_NETLINK with EAFNOSUPPORT.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): remove legacy read-only mode
After native local accept, only two broker paths still wrote into
workload memory: getpeername on relayed connections and the per-message
lengths of a first sendmmsg to the DNS relay. Legacy mode existed only to
refuse those writes on kernels without WAIT_KILLABLE_RECV, and the
getpeername refusal broke CPython TLS on RHEL 9.
The broker now never writes workload memory:
- getpeername is no longer mediated and reports the kernel peer on every
kernel; relayed connections report the loopback relay address.
- A first send to the DNS relay pins the broker's socket copy to the
relay and continues the syscall, so the kernel performs the send and
writes any per-message results.
Remove the listener mode, the task-memory write path and probe, and the
task_memory_write, cancellation, and task_memory_writes_disabled audit
evidence fields. WAIT_KILLABLE_RECV is still used when available and is
reported for diagnostics, but no mediation decision depends on it.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): drop WAIT_KILLABLE_RECV and make handlers restart-safe
The broker no longer writes workload memory, so killable notification
waits only reduced how often a signal restarts a notified syscall. RHEL 9
kernels never had them, so the handlers must tolerate restarts anyway.
Install the listener without the flag on every kernel instead of
special-casing newer ones.
Handlers now check that the notification is still live immediately
before each side effect (bind, connect of the retained socket, listen,
and the first DNS send), and answer a restarted operation the broker
already completed as the kernel would: a repeated TCP connect returns
EISCONN, repeating the same UDP association succeeds, a repeat of a
completed bind succeeds, and a failed relay reports its errno.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): finish teardown only when every workload descendant has exited
Teardown waited only for registered process groups to disappear. A root
is unregistered once reaped, so a descendant that ignored SIGTERM could
outlive it while termination reported success and never sent SIGKILL,
violating the bounded-termination requirement.
The sandbox now becomes a child subreaper when it is not PID 1, so
orphaned descendants stay in its tree and are reaped. Termination waits
until no live descendant remains, and every scanned process is signalled
through a pidfd after confirming its start time, so a reused PID is never
signalled.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep a frozen workload from resuming itself
Freezing stops every workload process with SIGSTOP. tgkill and
rt_tgsigqueueinfo were not mediated, and mediated kill, rt_sigqueueinfo,
and tkill passed SIGCONT through, so a workload process that was not yet
stopped could resume the others while the supervisor recovered.
Mediate tgkill and rt_tgsigqueueinfo like tkill: refuse targets in the
sandbox thread group, report a thread outside the named group as
missing, and continue otherwise. The boundary marks the broker frozen
before stopping the workload and clears it after resuming; while frozen,
workload requests to send SIGCONT fail with EPERM.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): send mediated DNS datagrams from the broker socket
The native UDP send path re-ran the workload's own sendmsg, so
per-message ancillary data such as IP_PKTINFO rode along; only the
loopback destination contained a routing override.
The broker now reads the datagram and sends it from its retained,
relay-connected socket, which it builds without ancillary data, so a
per-message override cannot redirect the packet. It writes nothing back
into workload memory. A send carrying control data is refused with
EOPNOTSUPP rather than silently stripped.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep sandbox control variables out of the workload environment
The canonical process inherits the sandbox environment so the image's
own ENV reaches the workload, then removed only a denylist of credential
variables. Other variables in the reserved OPENSHELL_ namespace, such as
the serialized user environment and the log level, still reached the
workload.
Remove every inherited OPENSHELL_ variable before applying the declared
environment, and restore OPENSHELL_SANDBOX=1. The image's ordinary ENV
and declared variables are unaffected; declared variables cannot use the
reserved namespace.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep /proc read-only for GPU workloads
GPU mode granted read-write access to all of /proc so CUDA's cuInit could
write thread names to /proc/<pid>/task/<tid>/comm.
Keep /proc read-only. Open mediation now serves a writable open of the
caller's own thread comm file: the broker opens it and injects the
descriptor, with no syscall continued. The kernel accepts a comm write
only from the target's own thread group, so a substituted path or reused
thread ID cannot rename another process's thread. Any other /proc write
is left to Landlock, which denies it.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep the broker socket for connected UDP DNS sockets
Sending DNS datagrams from the broker's socket required the broker to
keep its copy, but the connect paths to the relay and to loopback still
released it. glibc connects the resolver socket and then sends A and
AAAA together with sendmmsg, which is mediated, so resolution failed.
Release the copy on those connect paths only for TCP. Add a regression
test for connect followed by sendmmsg that fails fast rather than
hanging.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep slow loopback connects from stalling mediation
The loopback connect path held the registry lock while polling a TCP
connect for up to five seconds on the single notification dispatcher.
A workload connecting to a busy local listener stalled every other
mediated syscall, including opens and signals.
Connect a duplicate of the retained socket without holding the lock. A
nonblocking socket gets the native EINPROGRESS and the kernel completes
the handshake on the shared socket; a blocking socket waits on a bounded
worker thread.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): contain network broker handler panics
A panic in a notification handler unwound the single broker thread while
the health flag still read healthy, so every later blocked workload
syscall hung until the sandbox was killed.
Run each dispatch under catch_unwind: a panicking handler fails only that
syscall with EIO and the broker keeps mediating. If the broker thread
ever exits, mark it unhealthy so dependent operations fail closed instead
of blocking.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): reject ancillary data on DNS sends instead of broker-sending
Sending mediated DNS datagrams from the broker's own socket required
keeping a broker handle for each DNS socket's whole life, which regressed
real name resolution through reclaim and resource accounting that the
mock-based unit tests did not exercise.
Revert to continuing the kernel send, but refuse a send that carries
ancillary control data (msg_controllen != 0) at read time, so a
per-message routing override such as IP_PKTINFO cannot ride a mediated
DNS send. A loopback destination contains an override that races the
check. Full broker-side UDP mediation is left to a separate change with
deployment e2e.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): address review findings across the hardening changes
Fixes:
- The blocking local-connect worker no longer toggles O_NONBLOCK on the
open file it shares with the workload; it waits with a plain connect.
- tgkill and rt_tgsigqueueinfo are notified only when they send SIGCONT,
so ordinary thread signals such as Go preemption stay in the kernel.
The frozen flag is re-read immediately before delivery.
- A shell redirect to the caller's own thread comm file works again;
O_CREAT and O_TRUNC are no-ops there and O_EXCL returns EEXIST.
- The teardown scan is a linear walk and fails closed when /proc cannot
be read.
Simplifications:
- Remove tests that need infrastructure outside the repo or prove
nothing: the topology harness tests, the static-server benchmark, the
direct-syscall accept test, the comm rename test without Landlock, a
serde-default test, and the test-only panic hook in setsockopt.
- Drop the unused Failed connect outcome, the redundant read-back after
binding to loopback, redundant OPENSHELL_SANDBOX settings, and stale or
duplicated comments; reuse the reserved environment prefix constant.
- Tighten the support matrix and OpenShift wording.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(cli): wait for input when piped exec stdin is nonblocking
Processes that inherit the same stdin share one open file description,
so any of them can make it nonblocking for all. `sandbox exec` then read
no input yet, got EAGAIN, and failed with "Resource temporarily
unavailable (os error 11)". Parallel e2e tests inherit the runner's
stdin, which made credential_gating fail intermittently.
The stdin reader now blocks in poll() until input or end of file arrives
instead of treating EAGAIN as fatal, and it leaves the shared descriptor's
flags alone. The e2e exec helper also stops inheriting the runner's
stdin, since its commands take no input.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): fix macOS lint and older-glibc test linking
Import HashSet only in the Linux-only process scan, and call gettid
through the raw syscall in a test, since the libc wrapper needs glibc
2.30 or newer.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(sandbox): clarify socket peer address reporting
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
When the gateway exports OTLP traces, compute drivers pass the gateway's
endpoint to supervisors as OPENSHELL_OTLP_ENDPOINT, along with TRACEPARENT
from the operation that launched them. The Kubernetes, Docker, Podman, and
VM drivers all receive the endpoint from the gateway; driver TOML cannot
set it. On Podman Machine, the Podman driver points a loopback endpoint
at host.containers.internal, since supervisor loopback is the VM's.
The supervisor exports spans as openshell-supervisor, tagged with
the sandbox ID, and flushes them before exiting.
supervisor.startup joins the sandbox's creation trace and covers image
policy discovery, policy load, and boundary attach, confirm, agent start,
and access start. Supervisor calls to the gateway carry W3C trace context,
so the gateway's server spans nest under them.
Each egress connection emits supervisor.egress.connect with authorize,
resolve, and dial children. These spans use DEBUG level because every
outbound connection starts its own trace. OpenShell spans export at INFO,
or at the sandbox log level when it is debug or trace; spans from other
libraries export at INFO.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(providers): compose dynamic credential grants per request
Select the most-specific grant independently for each protected header,
preserving each credential's issuer, audience and cache identity. Resolve
all selected grants before rewriting the request, replace agent-supplied
header copies, and fail closed on collisions or acquisition failures.
Allow distinct-header compositions in gateway validation and redact raw
issuer errors. Cover independent cache entries, atomic TLS relay failure,
header replacement and concurrent request isolation; document the profile
contract.
Fixes#3320
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): correct dynamic grant CI checks
Use if-let for a grant result whose error payload is deliberately ignored,
and name the empty query map type in the regression fixture. Preserve grant
acquisition, failure redaction and request atomicity. Update the token-exchange
failure test to require the sanitized error instead of raw issuer text.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(supervisor): move backend setup behind the selected backend
Keep launch decoding, policy discovery and client construction with the
trusted backend while sharing supervisor admission and lifecycle handling.
Preserve VM policy identity validation and live credential handles.
Exercise alternate launch data through shared startup, readiness and
shutdown, including confirmation denial and resource cleanup.
Closes#4173
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(supervisor): resolve backend setup lint failures
Box shared startup futures to keep callers' async state small.
Use the private module boundary for backend setup visibility and keep
test helpers and defaults explicit without changing their inputs.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(supervisor): simplify selected backend startup
Signed-off-by: Shiju <shiju@nvidia.com>
* test(supervisor): keep imports before descriptor assertions
Signed-off-by: Shiju <shiju@nvidia.com>
* test(supervisor): preserve discovery failure startup coverage
Exercise backend discovery failure through shared startup before the
existing confirmation denial and successful shutdown cases. Assert that
startup preserves the error without launching a workload or readiness.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(podman): create managed workspace volumes owned by the workload identity
Podman now creates the managed /sandbox volume with uid/gid options for the
resolved workload identity, so the workload starts directly as that
identity. This fixes rootful sandboxes whose image USER or policy
run_as_user could not write to a root-owned /sandbox, and removes the
root-then-drop workspace chown start path.
Resource admission accepts the managed workspace volume when its options
match the workload container's final identity, or are empty for volumes
created by older gateways. The channel volume still requires empty options.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(podman): cover managed volume reuse and workspace access
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(podman): clarify managed volume creation and validation
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(podman): verify workspace access across user namespaces
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
* feat(server): add gateway capacity metrics and optional HPA
Expose per-replica supervisor sessions, pending relay capacity, relay
rejections and claim latency, and outbound peer request outcomes and
latency. Use bounded labels and Prometheus histograms for the new latency
metrics while preserving existing summary metrics.
Add an optional Helm HPA with external-database and resource validation,
conservative scale-down defaults, and support for custom metrics. Keep
certificate hook pods outside gateway workload selectors.
Document per-pod scraping, scaling limits, upgrade behavior, and
PostgreSQL connection sizing.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): unify routing metrics and clarify replica capacity
Combine local relay setup and outbound peer requests in one counter, labeled by operation, route, target, outcome, and gRPC status. Preserve peer latency metrics.
Rename the rejection reason from global_capacity to replica_capacity to reflect the per-replica relay budget. Keep capacity limits unchanged.
Update tests and documentation.
Signed-off-by: divesh <dgude@nvidia.com>
* fix(server): align routed request metric labels and outcomes
Count a local relay as successful only when its supervisor claims it, the
same event that answers a peer relay on the owner, and rename the outcomes
to local_error and remote_error so they say where an attempt failed. Label
the peer latency histogram by operation, like the routed request counter,
and rename the target label to relay_kind so it does not read as the
Prometheus scrape target.
Rename RelayCapacity.global to per_replica, drop the per-sandbox relay
capacity gauge, which no per-sandbox series can pair with, describe the
latency histogram as peer-only, and restore the note that unavailable
spikes are expected during rollouts.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
---------
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Signed-off-by: divesh <dgude@nvidia.com>
Co-authored-by: divesh <dgude@nvidia.com>
Closes#3502
Resolve trusted OCI runtime images with driver TOML, process environment, and compiled-default precedence across Docker, Podman, and Kubernetes.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(vm): validate launch credentials before preparation
Negotiate a launch-authentication requirement and reject incomplete gateway
configuration before driver validation, archive consumption or persistence.
Validate replacement VM credentials before stopping active compute and
prefer explicit signer configuration over local discovery.
Closes#3949. Part of #3955.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): validate saved launch credentials before restore side effects
Signed-off-by: Shiju <shiju@nvidia.com>
* chore(gateway): satisfy launch preflight lint checks
Signed-off-by: Shiju <shiju@nvidia.com>
* docs(gateway): complete the MicroVM launch-signing example
Include the gateway JWT paths in the standalone VM configuration and show
the output directory required by local certificate generation. Explain
the distinction between configuration preflight and launch-key validation.
Refs #3949. Part of #3955.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(vm): account for preparation state in launch credential fixture
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* feat(server): separate image preparation and admission deadlines
Give sandbox image preparation its own deadline and start the admission
deadline after preparation finishes. Persist both phases across gateway
restarts and show recovery guidance for the phase that expired.
Preserve the current service authorization schema and regenerate the Go
bindings with the preparation timestamps.
Fixes#3952
Related to #3955
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): simplify preparation timeout fallback selection
Use lazy Option fallbacks while preserving timeout messages and retained
sandbox behavior.
Signed-off-by: Shiju <shiju@nvidia.com>
* docs(server): clarify admission timer prerequisites
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(compute): enforce deadlines during initial sandbox create
Release stalled create operations after preparation expires and preserve
the timeout diagnosis across late driver results. Keep failed-create cleanup
bound to its original attempt so another replica can retry safely.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(server): satisfy deadline regression lints
Drop the create-error mutex guard before matching its cloned value and use
idiomatic iteration and timeout matching in the deadline fixtures.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(server): retain ownership of pending provisioning operations
Keep submitted create and start requests alive after caller cancellation,
monitor failure, or preparation timeout. Persist request ownership before
dispatch and retain staged uploads while the driver response is pending.
Require timeout cleanup to stop compute after driver settlement without
discarding an active cleanup claim. Fence late result handling against newer
operations, preserve failed-start recovery, and defer automatic restart while
another request owns the sandbox. Expose pending ownership in CLI JSON.
Add ordered multi-replica and cancellation regressions for the review findings.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(openshell): preserve tracing and accept ready create responses
Carry the request span into the detached provisioning worker so compute
driver calls remain attached to their parent trace after task handoff.
Update the compensation regression to require no backend DELETE when the
durable cleanup claim fails, matching the operation ownership requirement.
Accept the gateway's current Ready snapshot when a sandbox becomes ready
before CREATE returns. Do not require the client to observe an earlier
provisioning phase. Cover the Ready-only watch and command attachment.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(relay): close outbound relay stream when target closes first
Closes#3724
out_tx was cloned into the target-reading task, so the function's
own copy kept the outbound relay stream open until the client side
also ended. A target that closed a keep-alive connection never
reached the client as EOF, so reused connections hung forever.
Move out_tx into the target-reading task instead of cloning it, so
dropping it on target EOF ends the outbound stream right away.
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(relay): keep client uploads after target half-close
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
---------
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* feat(gateway): validate VM filesystem tools during preflight
Check required local VM tools through config preflight and share executable
resolution with VM image operations. Report selected paths and actionable
errors without creating gateway or sandbox state.
Bound probe output and execution time, and clean up probe descendants on
interruption. Preserve pure static validation and skip local tool checks
for remote driver endpoints and unrelated drivers.
Fixes#3951
Related to #3955
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(gateway): stabilize filesystem preflight checks
Combine identical filesystem-tool error arms and normalize rendered
diagnostics in command tests so terminal wrapping preserves assertions.
Describe driver TLS validation without depending on removed guest fields.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(gateway): serialize preflight fixture paths as TOML
Keep temporary paths quoted and escaped through the TOML serializer
instead of relying on Rust Debug formatting.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Run image preparation in an owned worker process, reserve its process
identity until cleanup completes, and protect staging with leases so
cancellation and recovery cannot race with another preparation attempt.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): enforce the configured workload identity
Reject conflicting policy users and groups before VM image preparation
and before guest attach or process startup changes state. Validate
supervisor policy updates against the protected VM workload identity.
Preserve the gateway CA transport and capability-free sandbox launcher.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(sandbox): clarify VM identity rejection fixtures
Name invalid user and group fixtures distinctly and move the final
workload identity into its group mismatch test.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(supervisor): align VM identity startup with current APIs
Pass the optional rejection-log key for VM identity failures and keep
generic startup-write regressions free of VM identity constraints.
Repair the call sites after the branch rebase so the identity and cleanup
proposals compile against the current startup helpers.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): restore inactive sandbox workload identity
Recover the persisted overlay owner before publishing stopped and terminal
sandboxes. Keep resources manageable when identity metadata is invalid.
Clarify fixed MicroVM ownership in policy-generation guidance.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(vm): flush identity fixture before restart
Persist the canonical identity file before readiness and report the observed
exec, canonical and file-owner identities before comparing them.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
The Docker driver passed a bare reference as CreateImageOptions.from_image
with no tag. The daemon interprets a tagless fromImage as a request for
every tag in the repository and pulls them all (issue #4029).
Normalize a pull reference by appending ':latest' when it has neither an
explicit tag nor a digest, matching 'docker pull' and the Podman driver.
Parsing inspects only the final path component so a registry port (e.g.
'registry:5000/team/app') is not mistaken for a tag and a digest-pinned
reference ('...@sha256:...') is left untouched. Applied at both pull
sites (pull_image and pull_runtime_image).
Signed-off-by: Udaya Tejas <udayatejas2004@gmail.com>
* feat(cli): stream non-TTY exec input before EOF
Add --stream-stdin using the existing interactive exec RPC without a PTY. Preserve separate output streams and enforce the existing 4 MiB cumulative input cap while forwarding input.
Require explicit clean stdin EOF and drain the response through its final gRPC status. Cover held-open input, limits, cancellation, trailers, and default finite-input behavior with subprocess and live sandbox regressions.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(cli): distinguish exec cancellation from stdin EOF
Treat transport termination and response cancellation as separate test observations. Verify explicit stdin EOF through the shared frame writer and cover cancellation in the pinned Tonic decoder.
Signed-off-by: Shiju <shiju@nvidia.com>
* style(cli): use lazy optional stdin dispatch
Signed-off-by: Shiju <shiju@nvidia.com>
* test(cli): use imported duration in stdin EOF regression
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): restore supervisor-backed GCP metadata discovery
Relay the reserved metadata endpoint to the supervisor and restore project, account, and placeholder token responses from live provider state. Cover Google SDK discovery and repeated refresh with provider E2E tests.
Closes#3860
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(providers): preserve default metadata account without an email
Use the default account identifier when the optional service account email is missing or empty. Cover repeated SDK refresh for missing, empty, and configured email values with distinct providers for parallel E2E execution.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test(sandbox): fix metadata relay lint and timeout
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
Since #2726 the canonical main process's stdout and stderr are captured
in pipes that feed only the in-memory replay buffer used by sandbox
connect. Agent output therefore never reaches the container's own stdout
and stderr, so it is missing from kubectl logs, docker logs, and podman
logs and from anything that collects container logs. Before #2726 the
entrypoint inherited the container's descriptors and its output appeared
there.
Copy the main process's output to the launcher's stdout and stderr in
addition to the replay buffer, restoring the earlier behavior:
- Output is copied byte for byte to the matching stream from a
forwarder thread per stream, after it is published to the replay
buffer. When the container runtime falls behind on a stream, that
stream's reader waits instead of dropping output, so backpressure
reaches the agent as it did with inherited descriptors, while the
other stream and attachments keep receiving output.
- Before the main process's exit is published, the output readers
finish and queued output is drained to the container log, so an
agent's final lines are not lost at shutdown. A 30 second deadline
covers both; when it expires, readers waiting on the container log
are released and drain the pipes into the replay buffer only, so a
stalled container log cannot block exit reporting.
- PTY-mode processes are not copied. The terminal stream carries escape
sequences and echoed input, and terminal commands never reached the
container log before #2726.
- Exec, SSH, and SFTP sessions are not copied.
Launcher log lines keep their existing format and remain in the
container's stderr. They are written as whole lines, and a newline is
inserted first when the agent left stderr mid-line, so launcher and
agent lines do not merge.
The Docker and VM drivers appended the tail of the workload's output to
failure messages: Docker the workload container's log, and the VM driver
the guest console, which carries the launcher's stdout and stderr. Those
messages land in the sandbox's Ready condition and in platform events
that the gateway republishes to the sandbox event stream. With agent
output in that log, those messages would carry arbitrary agent output,
including anything sensitive the agent prints, into gateway status and
events. The supervisor starts its health endpoint only after the agent
starts, so every Docker failure path could include agent output, and the
VM driver reports one whenever the VM or host supervisor exits. Forward
only the supervisor's log tail, matching the Podman driver, which reads
the workload log solely to match fixed launcher markers and never
forwards raw workload output. The workload's output remains available
through docker logs and the VM's rootfs-console.log.
Document where main process output appears in the logging docs and the
cluster debugging skill.
Closes#3928
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(policy): refresh pending proposals when the sandbox policy changes
Approving, removing, or undoing a rule, or updating the sandbox policy,
changes the inputs every other pending proposal was evaluated against.
Only proposals the new policy covered were reconciled; the rest kept
their old prover result and review token. The review surface
(GetDraftPolicy) therefore showed a stale evaluation, and the first
approval of the next proposal refreshed it and failed with
FAILED_PRECONDITION, so approving proposals one after another always
failed once.
Re-evaluate the remaining pending proposals at each policy change,
reusing the cached prover result unless the proposal's inputs changed.
Approval still rejects a review token that does not match the stored
evaluation, so a reviewer holding a pre-refresh evaluation must still
refetch it.
When a refresh does happen at approval time (inputs changed between
fetch and approve), the CLI now explains that the rule was re-evaluated
and how to review it, instead of printing the raw gRPC status.
Closes#3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* fix(policy): make pending proposal refresh race-safe and bounded
Store refreshed evaluations with a compare-and-swap: the store re-reads
the proposal, refuses when its rule name, proposed rule, or review token
changed since the evaluation read it, copies only the evaluation fields
onto the stored record, and updates only if the payload is still the one
it read. A refresh can no longer revert a concurrent edit or observation,
and the edit path uses the same guard against a concurrent refresh.
Bound each refresh to the 32 newest pending proposals; the rest keep the
approval-time recheck, which still refuses a stale review token. Operator
decisions (approve, approve-all, remove, undo) refresh before responding.
UpdateConfig, which holds the gateway-wide sandbox sync guard, and
agent-driven auto-approval refresh in a background task instead, one per
sandbox with later changes coalesced into a single rerun.
Refs #3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* docs(policy): describe proposal rechecks after approvals and approve-all
Explain that approving, removing, or undoing a rule rechecks the other
pending proposals so they can be approved one after another, when the
recheck is deferred or bounded, and what rule approve reports when a
proposal changed after it was listed. Show rule approve-all in Run Your
First Agent with its security-flag behavior.
Refs #3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* fix(policy): refresh pending proposals after a full policy replacement
A full policy UpdateConfig (openshell policy set) re-reads the latest
revision after its atomic write, finds the revision it just committed,
and returns before reaching the pending-proposal refresh at the end of
the handler. Pending proposals kept their stale evaluation, so rule get
showed the old candidate and the next approval failed with the refresh
precondition. Schedule the background refresh right after the commit.
Refs #3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
---------
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
With a non-terminal stdin, sandbox exec read stdin to EOF before it sent
the exec request. A pipe that never closes (CI runners, supervisors, agent
harnesses) blocked the CLI forever in read(2) without the gateway ever
seeing the request, and a slow producer delayed the command until EOF.
Collect piped stdin on a detached reader thread for at most 200 ms. Input
that reaches EOF within that window still travels in the single request
that older gateways need. If the pipe is still open, start the command
through the streaming RPC and forward the collected prefix plus the rest of
stdin as it arrives, closing remote stdin at EOF. The 4 MiB cap covers the
prefix and the streamed remainder together.
Closes#3993
Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com>
* fix(supervisor): wait for repair when the gateway refuses a startup policy write
Startup writes the sandbox policy to the gateway in two cases: it
uploads a discovered image policy when the gateway has none, and it
writes the policy back after adding the proxy baseline filesystem paths.
When the gateway refused either write with FAILED_PRECONDITION or
INVALID_ARGUMENT, for example because the policy binds a provider that
is not attached, startup treated the refusal as a permanent error and
the supervisor exited. The sandbox never reached the ConfigurationInvalid
repair state that other startup rejections use.
Report such a refusal as a configuration rejection carrying the
gateway's message, log it once per write and error code, and keep
polling, so attaching the provider or replacing the policy completes
startup. Other error codes keep their current handling: transient codes
are retried, and permission, not-found and authentication failures
still end startup.
Skip the baseline-path write-back while a global policy is active. The
gateway refuses every sandbox policy write in that state, so startup
exited whenever a global policy lacked a baseline path. The supervisor
now adds the paths to its own copy of the policy without saving a
revision.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(supervisor): stabilize startup refusal log capture
Keep a second tracing dispatcher alive while capturing startup refusal
logs. With only one dispatcher, a parallel test thread without a default
subscriber can cache Interest::never for the shared OCSF callsite after
the capture thread rebuilds the cache.
Preserve the exact log-count, diagnostic, configuration-generation and
repair assertions. Production startup behavior is unchanged.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(supervisor): reconcile stale startup rejection reports
Refetch desired configuration immediately when a rejection report is aborted because its generation changed. Preserve acknowledged rejection pacing and all other report error handling.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(supervisor): box startup repair race futures
Keep the repair regressions below the large-future lint threshold without changing their inputs, scheduling, or assertions.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Start driver cleanup after terminal finalization and retain disconnect fallback. Add detached success and failure e2e coverage across supervisor-based drivers.
Closes#3938
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* refactor(sandbox): remove unreachable root-side identity and workspace code
RFC 0012 moved the workload into its own capability-free container that
starts as the final sandbox identity. The sandbox no longer runs a root
supervisor that prepares the filesystem, rewrites account files, resolves
OCI USER entries, or drops privileges before launching the workload, so
that code had no production callers.
Remove the unreachable paths and their tests:
- prepare_filesystem / prepare_filesystem_with_identity, the /sandbox and
OCI workspace chown preparation, and the root-side workspace validation
(validate_oci_workspace and its privilege-dropped subprocess)
- the hidden validate-workspace subcommand
- drop_privileges / drop_privileges_with_identity, capability bounding set
clearing, validate_sandbox_user/group, and /etc/passwd and /etc/group
rewriting
- the sandbox-side OCI USER resolver (identity.rs) and
ResolvedProcessIdentity; the boundary now writes the driver-resolved
UID/GID into the policy directly
The workspace check that still runs inside the capability-free boundary
(validate_oci_workspace_as_effective_identity) is unchanged.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* chore(sandbox): remove unused capability dependency and refresh Landlock comments
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
log_response has always logged every gateway response at INFO, including
health probes and the GetSandboxConfig and provider-readiness polls each
supervisor makes. #3915 demoted the request spans for those polled paths
to DEBUG, which stripped the request{method path} prefix from the log
line at INFO but left the line itself, so the gateway log fills with
bare 'response status=200' lines several times per second.
Follow the span's level: polled requests log their response at DEBUG,
or WARN on a 5xx so probe and poll failures stay visible.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(network): refuse protocol upgrades on GraphQL endpoints
Refuse Upgrade headers before forwarding GraphQL-over-HTTP requests.
Share the protocol refusal table with JSON-RPC and MCP, and close
unexpected protocol switches before relaying frames.
Keep GraphQL-over-WebSocket inspection on separate WebSocket endpoints.
Cover upgrade refusal, audit mode, subscription handshakes, and ordinary
HTTP and WebSocket controls. Update the current policy documentation.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(network): refuse GraphQL upgrades before reading bodies
Validate the HTTP head and endpoint authority before upgrade refusal, then inspect ordinary GraphQL bodies. Preserve missing-authority credential rejection after body inspection.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Store operation spans and request spans for supervisor-polled RPCs
(GetSandboxConfig, ReportProviderReadiness) use DEBUG level, so the
default INFO filter no longer exports them. The provider credential
refresh worker opens its span only when a state has work.
Refs #2698
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* test(podman): move podman_preflight into driver-podman integration tests
podman_preflight verifies that openshell-driver-podman fails fast when
its Podman socket is unreachable. It only needs the standalone driver
binary, not a gateway, so it never fit the gateway-backed e2e-podman
harness it lived under and never ran anywhere in CI.
Move it into crates/openshell-driver-podman/tests/ as a plain Cargo
integration test. It now runs via the existing required workspace test
job with no special mise task, workflow step, or coverage exception.
Signed-off-by: politerealism <burdcat17@gmail.com>
* test(podman): make preflight diagnostics portable
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: politerealism <burdcat17@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
* test(policy): reproduce raw OPA loading gaps against the typed schema
The supervisor loads a sandbox policy in two ways: through the typed
schema (parse_sandbox_policy, then from_proto) or directly into OPA
(from_strings and from_files). The raw path fills in defaults where the
typed schema is strict, so the same policy text can produce a different
sandbox configuration, or load when it should be rejected.
Add two regression tests that fail on the current code:
- An empty filesystem_policy loads with include_workdir true through raw
OPA and false through the typed schema. An absent stanza gives true on
both paths and must keep doing so.
- Raw OPA accepts a string include_workdir, a non-string read_only entry,
an unknown Landlock compatibility and an explicit null json_rpc, with
or without a version key. The typed schema rejects each. Every case has
a valid twin that both paths must accept.
A follow-up change makes raw loading apply the typed schema's rules.
Refs #3092.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(policy): align raw OPA loading with typed settings
Validate raw filesystem, Landlock, and process settings with the canonical
authored schema before normalization. Preserve the absent filesystem
default while applying the present-stanza default, and canonicalize valid
Landlock enum representations before runtime evaluation.
Reject explicit null JSON-RPC options through the shared parser. Preserve
versionless and runtime OPA data, and keep rejected reloads from replacing
the active policy or advancing its generation.
Add raw-versus-typed, file-loader, and rejected-reload regressions and
document the local loading contract.
Refs #3092.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(policy): validate raw OPA settings and redact startup errors
Validate raw network fields through the authored schema before
normalization. Preserve custom Rego data and supported runtime forms.
Apply the shared filesystem path checks and non-root identity predicate
to raw static settings.
Discard authored Rego source and nested errors from static configuration
evaluation. Cover malformed inputs, valid controls, file loading, and
rejected reloads retaining active decisions and generation.
Refs #3092.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(policy): satisfy unit-returning assertion lint
Terminate the two error-assertion match arms with semicolons, as required
by Clippy. Preserve the existing checks and runtime behavior.
Refs #3092.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Evan Lezar