* feat(mxc): ETW->OCSF audit consumer + Windows OCSF JSONL parity (cp6 P1)
Add a Windows MXC ETW->OCSF audit trail in openshell-driver-mxc: a real-time
Sandboxing-provider ETW consumer that decodes events (TDH), attributes each to
an OpenShell sandbox_id, and maps them to OCSF (lifecycle 6002, config 5019,
process 1007, finding 2004).
cp6 Phase 1 - durable OCSF JSONL audit-file parity with Linux:
- openshell-ocsf: add emit_ocsf_event_routed (populates the event-bridge
thread-local AND stamps sandbox_id+message in one dispatch) plus public
set/clear_current_event; OS-aware device (Device::windows/for_current_os) so
device.os.name reflects the host instead of a hardcoded Linux stub.
- etw_consumer: emit via the routed emit (previously fired a bare info! that
never populated the bridge, so the structured event was dropped).
- openshell-server: install OcsfJsonlLayer over a synchronous daily-rotated
appender (durable under force-kill), gated by OPENSHELL_OCSF_JSON, path via
%PROGRAMDATA%\OpenShell\logs (override OPENSHELL_OCSF_LOG_DIR).
- device.hostname now resolves to the real gateway machine name.
Box-proven on 7F203-MXC-001: JSONL lines == shorthand OCSF rows, all valid
OCSF JSON, per-sandbox attribution intact, disabled state writes nothing.
Signed-off-by: Akber Raza <akberr@nvidia.com>
* feat(mxc): map remaining Sandboxing ETW events to OCSF
Close the last three ETW->OCSF gaps so the audit trail covers the full
set of events the Sandboxing provider emits (12/12):
- ProcessLaunched -> Process Activity [1007] "Launch" (confirmed start;
carries the real processId/threadId, the twin of CreateProcessInSandbox
which only has the request + command line).
- SandboxProxyConfigured -> Device Config State Change [5019] (the one
network-plane setup event; surfaces proxyPort, "no proxy" when 0).
- SandboxConsoleReferencePlumbed -> Device Config State Change [5019]
(console-handle plumbing).
map_config_state now handles the full config/hardening/setup family and
carries proxyPort/hasConsoleReference/creationFlags as unmapped fields.
Verified on 7F203-MXC-001: 11/12 event types emit OCSF without a proxy
(SandboxProxyConfigured requires proxy config to fire).
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc): seed ETW attribution under registry lock + Device tests
Address CodeRabbit review on !31:
- Prevent stale ETW attribution on a delete/launch race: register the
wxc-exec pid while holding the registry lock, and bail if the sandbox
entry is already gone. Previously the attribution key could be seeded
after `delete` had removed the sandbox, leaving a stale key that could
misroute later Sandboxing ETW events to a dead sandbox_id. Lock order
(registry -> attribution) matches the delete path, so no deadlock.
- Add unit tests for the new Device::windows and Device::for_current_os
constructors to harden Windows/Linux OCSF device parity.
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc-etw): buffer+replay racing events and harden attribution keys
Addresses two ETW->OCSF attribution review items (Shailendra #1, #2).
#2 early-event loss: ETW delivers the sandbox create/config burst the instant wxc-exec starts, which can beat the driver's register_launch (now under the registry lock post-Ready). process_event previously dropped anything unresolved, losing the racing burst. Add a bounded, time-bounded pending buffer (PENDING_MAX=4096, PENDING_TTL=5s): unresolved events are held and replayed once attribution lands, aged-out ones dropped. Consumer switched to a timed recv_timeout(200ms) so the buffer is re-driven after each event and on a tick. Emit path factored into shared emit_resolved().
#1 attribution collisions: a Windows PID is recycled after exit and a command line is commonly identical across sandboxes. register_launch now rebinds by_pid on reuse and clears the stale last_pid_sid hint (warns if the PID still pointed at a different, leaked sandbox); command line is held in by_cmd only while unique and demoted to a new ambiguous_cmds set on a second owner, so a duplicate command refuses to resolve rather than misroute.
Unit tests: buffer replay (direct + cross-link), buffer bound, PID-reuse rebind, duplicate-cmd non-resolution. Box-verified on 7F203-MXC-001 (5 sandboxes, identical cmd -> 5 isolated sandbox_ids, 50/50 OCSF/JSONL, BuffersLost=0).
Signed-off-by: Akber Raza <akberr@nvidia.com>
* docs(mxc-etw): note cmd_line is captured raw with no privacy filtering
Review item #3 (Shailendra): add a PRIVACY NOTE on map_process_launch stating cmd_line is copied verbatim into OCSF process.cmd_line with no redaction, so secrets/PII on a command line land unredacted in the durable audit trail (deliberate audit-fidelity trade-off; treat the log as sensitive). Redaction is owned by an upstream privacy layer, not this path; no general audit-output PII scrubber exists today (openshell_core::secrets [CREDENTIAL] redaction is scoped to the proxy HTTP-target logging, a separate egress path).
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc-etw): open ETW trace on caller thread so start_session reports real status
Review item #4 (Shailendra): start_session previously returned Ok(EtwSession) as soon as the pump thread was spawned, but OpenTraceW ran later inside that thread; if it failed we still handed back a live-looking session and logged 'consumer started' (silent failure = false audit coverage).
Split the two Win32 calls instead of adding a channel handshake (avoids any lost-wakeup/hang risk): the quick, synchronous OpenTraceW now runs on the caller thread (open_trace), and only the blocking ProcessTrace runs on the pump thread (run_trace). start_session returns Err if OpenTraceW fails (reclaiming the boxed Sender so the consumer disconnects, stopping the session, joining the consumer) and returns Ok/logs 'started' only once capture is genuinely open. Opened handle + LoggerName buffer + boxed Sender are carried to the pump via a Send OpenedTrace so they outlive ProcessTrace.
Box-verified on 7F203-MXC-001: consumer started=True, failed-to-start=False, 50 OCSF rows / 50 JSONL, BuffersLost=0 (no regression to capture/emit).
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc-etw): guard pending-event replay against PID recycling
CodeRabbit flagged that drain_resolved() re-resolved buffered events
against the live by_pid map, so if Windows recycled a wxc-exec PID within
PENDING_TTL a stale event from the dead sandbox could be emitted under the
new owner.
Stamp each by_pid registration with its Instant and add resolve_replay(),
used only on the buffered/replay path. It (a) never falls back to the
recycle-/ambiguity-prone by_cmd or last_pid_sid keys, and (b) trusts a PID
match only when the registration is not newer than the buffered event by
more than REPLAY_PID_GRACE (2s) - a recycled PID's registration lands well
outside that window, so the stale event ages out instead of misattributing.
The legitimate #2 seed race (registration lands ~immediately) still replays.
Adds unit tests for the recycle-refusal, in-grace acceptance, and
weak-fallback exclusion.
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc-etw): surface unexpected ProcessTrace termination (review #4)
start_session already returns Err on OpenTraceW failure (runs on the
caller thread since e41a7701), closing the first half of Shailendra's #4.
This closes the second half: ProcessTrace's result was discarded, so if
capture died mid-run the backend had no way to know.
Add a shared CaptureHealth (stopped/stopping/exit_code) between the pump
thread and EtwSession. run_trace now records ProcessTrace's WIN32_ERROR
and, when the pump returns without a deliberate stop, logs at ERROR that
MXC OCSF capture is no longer running. EtwSession::stop() sets `stopping`
before teardown so a normal shutdown isn't misreported, and
EtwSession::is_capture_alive() exposes the state for status/diagnostics.
Box-verified on 7F203-MXC-001: 5 sandboxes, 50 attributed OCSF rows,
JSONL parity 50/50, BuffersLost=0, clean start/stop (no false failure).
Signed-off-by: Akber Raza <akberr@nvidia.com>
* feat(mxc-ocsf): add ETW->OCSF audit-trail example kit; fix proxy-configured message
Add a runnable OCSF audit-trail example under examples/ (run-ocsf-audit.ps1,
mxc-ocsf-audit.toml, ocsf-audit.yaml, README) that spins up sandboxes with the
in-process ETW consumer and egress proxy on, emitting a full OCSF JSONL audit
trail across all four classes (6002/5019/1007/2004).
Fix SandboxProxyConfigured mapping to log "MXC sandbox proxy configured" instead
of a misleading "(no proxy)" when the provider reports proxyPort=0; the event's
presence already indicates proxy configuration. Verified on-box: 26 events, all
mapped ETW event types present.
Signed-off-by: Akber Raza <akberr@nvidia.com>
* feat(mxc-ocsf): clearer audit report + client-safe run-ocsf-audit.ps1
Improve the ETW to OCSF audit-trail example output and make it safe to ship.
Report:
- Add an event-type coverage count ("N of M expected event types fired");
the denominator auto-adjusts (8 with proxy on, 7 with -NoProxy).
- Split the checklist into expected event types vs anomaly findings
(ActivityError/FallbackError), which are reported separately and not
counted toward coverage (a clean run may emit none).
- Verdict is now coverage-based (all expected types must fire) instead of
the looser "at least 3 OCSF classes".
- Call out the absolute path to the durable OCSF JSONL log prominently.
Client-safety:
- Default -ShareOut to empty (no auto-copy); pass -ShareOut a UNC path to
opt in. Removes a hardcoded internal share path from a published example.
- Drop internal-team wording ("Hand that zip back for evaluation", "BUNDLE:")
in favor of neutral "Results bundle:".
- Update README-ocsf-audit.txt to match the opt-in -ShareOut behavior.
Verified on both MXC boxes: 7F203-MXC-001 (base-container) -> PASS, 8 of 8
event types, 26 OCSF events across 4 classes; 7F203-MXC-003 (AppContainer
fallback) -> reduced set as expected, clean output.
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc): configure OCSF audit workloads per sandbox
- remove unsupported gateway-scoped workload fields from the shipped MXC audit example.
- build the command, working directory, and filesystem grant from each run's ShareDir
- pass the workload through --driver-config-json.
- preserve the host CONNECT proxy configuration and conditional audit coverage for the future host_connect_proxy merge
- require the workload output when determining the audit verdict.
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(mxc): omit command arguments from OCSF audit logs
- record only the executable basename for MXC CreateProcessInSandbox audit events
- leave process.cmd_line unset so workload arguments cannot reach shorthand or JSONL logs
- cover tokens, passwords, signed URLs, and PII with a secret-leak regression test
- update the audit example, architecture guidance, and published logging documentation
- preserve ETW attribution and future host_connect_proxy enforcement behavior
Signed-off-by: Akber Raza <akberr@nvidia.com>
* feat(etw): enhance ETW session management with distinct naming for concurrent gateways
* fix(etw): bound the audit queue during overload
- replace the unbounded ETW callback channel with count- and byte-bounded buffering
- keep the ETW callback non-blocking and count records rejected during overload
- emit immediate, rate-limited warnings that identify resulting audit coverage gaps
- make the audit example fail when queue overload causes dropped ETW records
- cover stalled consumers, oversized events, and warning throttling with unit tests
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(etw): harden sandbox audit attribution
- remove command-line and persistent per-PID fallback keys from live and replay resolution
- retire the driver-owned wxc-exec PID before publishing child completion
- retain established identity, activity, and correlation-vector links only for the five-second late-event window
- prevent buffered records from crossing rapid PID retirement and reuse boundaries
- add resolver and lifecycle coverage and document the attribution trust boundary
Signed-off-by: Akber Raza <akberr@nvidia.com>
* chore(mxc): address rebase follow-ups
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(mxc): align OCSF audit example with driver config
- remove unsupported egress proxy settings
- stop requiring the unavailable proxy audit event
- update example documentation for supported event coverage
Signed-off-by: Akber Raza <akberr@nvidia.com>
* fix(etw): redact command-line secrets in DecodedEtwEvent summary
* fix(etw): enhance PID resolution and event attribution logic for ETW records
* fix(ocsf): restrict gateway-local JSONL sink to Windows/MXC path with opt-in configuration
* address rebase issues
* fix(mxc): address ETW audit review feedback
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(mxc): fail closed across ambiguous PID reuse
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(mxc): bind ETW attribution to process generation
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Akber Raza <akberr@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Jamie King <jamiek@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
* refactor(inference): remove managed inference routes
Closes#3172
Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(policy): preserve alternate upstream isolation
Restore the provider policy activation guard so legacy OpenAI and Anthropic providers configured for alternate base URLs do not grant egress to the built-in public vendor endpoints.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* feat(sandbox): support rootfs tar as --from source for VM driver
Accept flat rootfs tar archives (.tar, .tar.gz, .tgz) via the --from
flag for VM-backed gateways. The CLI detects the archive extension,
validates that the gateway uses the VM compute driver, and passes the
tar path through driver_config. The VM driver copies the tar into its
staging area and feeds it into the existing rootfs extraction and ext4
disk creation pipeline, skipping the container image pull/export steps.
Closes#2175
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox): validate rootfs tar path at the VM driver boundary
The rootfs_tar_path field in driver_config was passed from the API
caller directly to tokio::fs::copy without validation. An authenticated
user bypassing the CLI could supply arbitrary host paths (e.g.
/dev/zero for disk exhaustion, or readable host files for data
exfiltration).
Introduce a trusted staging directory that the VM driver creates on
startup and advertises via GetCapabilities. The CLI now copies the tar
into the staging directory before creating the sandbox, and the driver
validates that the received path is a regular file inside the staging
root and within a configurable size limit (default 10 GiB) before any
I/O.
New VmDriverConfig options:
- rootfs_tar_staging_dir: override the staging directory
(default: <state_dir>/rootfs-tar-staging)
- rootfs_tar_max_bytes: override the size limit (default: 10 GiB)
Addresses GATOR-28b5152e-01.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox): request-scoped staging, size pre-check, and cleanup for rootfs tar
Tighten the rootfs tar staging flow to address the remaining GATOR-01
obligations:
- Request-scoped staging: the CLI creates a unique per-request
subdirectory (req-<pid>) under the staging root instead of placing
files directly in the shared directory. The driver enforces that the
tar path is at depth 2 (staging_root/<subdir>/<file>), preventing
cross-request path selection.
- Size pre-check: the driver advertises rootfs_tar_max_bytes via
GetCapabilities. The CLI reads this limit and rejects oversized files
before copying, avoiding disk exhaustion in the staging directory.
- Cleanup: the driver removes the request staging subdirectory after
consuming the tar (on cache hit, copy success, or copy failure),
ensuring staged data does not persist beyond the request.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(vm): restore rootfs-tar sandboxes from persisted image identity
On restore or restart, the one-shot staged tar archive has already been
cleaned up. Reading the persisted image identity from the sandbox state
directory and resolving the cached disk path directly avoids re-accessing
the deleted staging path.
Addresses GATOR-168b9210-01.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(cli): use random staging dirs and enforce byte limit during rootfs tar copy
Replace PID-based request staging directories with tempfile-generated
random names to prevent collisions and make paths unpredictable.
Replace bare tokio::fs::copy with a streaming copy loop that enforces
the advertised max_bytes limit during transfer, closing the TOCTOU gap
between the pre-copy size check and the actual copy.
Signed-off-by: Philippe Martin <phmartin@nvidia.com>
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox): issue rootfs tar staging slots from the gateway
A caller could name any host path in `driver_config.vm.rootfs_tar_path`,
which the privileged VM driver then read. The CLI-side locality check did
not apply to direct API requests.
The gateway now owns staging. `BeginRootfsTarStaging` allocates a
request-scoped directory under the driver-advertised staging root and
returns an opaque single-use token; `CreateSandbox` carries the token, and
the gateway substitutes the path it allocated before dispatching to the
driver. `template.driver_config.<driver>.rootfs_tar_path` is rejected
outright in request validation, so a caller-supplied path never reaches
privileged I/O.
Tokens are bound to the issuing workspace and subject, consumed once, and
expire after 30 minutes. Outstanding slots are capped per caller and
overall, so one caller can neither exhaust the staging filesystem nor
starve others. An RAII guard reclaims the directory on every failure path
after consumption, and an age-gated sweep runs at startup and on each
reconcile pass for directories whose driver died before its own cleanup.
The token is stripped from the public sandbox before persistence: the
stored copy is returned verbatim by GetSandbox, ListSandboxes and
WatchSandbox to every member of the workspace.
Also fixes two defects this exposed:
- The CLI wrote `rootfs_tar_path` at the top level of `driver_config`, but
the gateway forwards only `driver_config.<driver_name>`, silently
dropping unmatched keys. The archive never reached the VM driver, so the
documented `--from ./rootfs.tar` flow did not work at all. Config is now
nested under `vm` and deep-merged, so a caller's existing VM settings
survive instead of being clobbered by a shallow extend.
- Staging previously required `GetGatewayInfo`, which is restricted to
`platform_admin`, making the feature unusable for ordinary users on any
RBAC-enabled gateway. The new RPC matches CreateSandbox at
`sandbox:write` / `workspace_role: user`.
`compute_driver.proto` is unchanged; the gateway reads the staging root
from the capabilities it already stores.
Refs #2175
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(vm): derive rootfs tar cache identity from archive contents
The prepared-disk cache key combined the archive's full path with an mtime
truncated to seconds, then mapped punctuation to `-`. Distinct paths such as
`/tmp/a/b.tar` and `/tmp/a-b.tar` collapsed onto the same key and reused each
other's disk, a rewrite within the same second kept stale contents, and a long
path could exceed filesystem component limits.
Identity is now a SHA-256 of the archive contents. This is also what makes the
cache work at all now that the gateway allocates a fresh staging directory per
request: a path-derived key would miss on every create.
The archive is hashed, the cache checked, and only on a miss copied — so a hit
skips writing a multi-gigabyte file. The copy is hashed as it is written and
rejected if the digest differs from the first pass, which closes the window
where the source changes during staging rather than approximating it with a
re-stat.
Refs #2175
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(vm): decompress gzip rootfs tar archives during staging
`--from` accepts `.tar.gz` and `.tgz`, but the driver staged whatever bytes
it was given and the guest image-prep VM extracts the staged file with a
plain `tar -xpf`. Compressed sources therefore depended on the guest tar
auto-detecting gzip, and the prepared disk was sized from the compressed
length, which is far too small for the expanded rootfs.
Staging now detects gzip from the archive's magic bytes -- the driver only
ever sees a gateway-issued staging path, never the caller's file name -- and
writes an uncompressed tar. The digest still covers the source bytes, so the
"archive changed while staging" check is unaffected, and expansion is bounded
by `rootfs_tar_max_bytes` so a compression bomb cannot fill the host disk.
`extract_rootfs_archive_to` sniffs gzip as well, so the host-side extraction
path matches.
Adds unit coverage for gzip staging, bounded expansion, and gzip extraction,
plus an e2e sandbox created from a gzip-compressed export.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
Signed-off-by: Philippe Martin <phmartin@nvidia.com>
* docs(gateway-config): fix stale community sandbox image path
Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
* docs(sandbox-image): purge remaining stale image references
Rebasing onto main surfaced four more instances of the same dead
ghcr.io/nvidia/openshell/sandbox path, introduced by commits merged
after this branch was opened: three test fixtures (driver-docker,
openshell-ocsf, compute::mod) and one user-facing default in the
SPIFFE token-exchange Podman demo README. Correct all four to
ghcr.io/nvidia/openshell-community/sandboxes/base, consistent with
the rest of this fix.
Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
---------
Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
Centralize compute-driver RPC descriptors, stream instrumentation, provider
routing, and standalone installation in openshell-otel. Use typed RPC
constants so gateway and in-process driver paths cannot panic on unknown
operation strings or repeat runtime method parsing.
Emit semantic-convention rpc.service and rpc.method attributes, preserve
trace context and resource identity across deployment modes, and route both
RPC boundary and backend crate spans to each selected driver provider. Leave
consumer-dropped watch spans unset while recording observed terminal status,
and avoid reboxing untraced external-driver streams.
Derive each driver tracing identity from Cargo package and crate metadata and
attach its descriptor to the compute-driver registration, keeping provider
selection and target routing tied to the registered implementation. Share
tracing setup and round-trip test support across Docker, Podman, Kubernetes,
and VM, and update the gateway tracing documentation.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes
The corporate forward proxy machinery from #1792 is driver-agnostic and
already merged: openshell-supervisor-network implements CONNECT chaining,
NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and
openshell-sandbox exposes it as six argv-only flags. Podman gained the driver
half in #2245/#2512 and Kubernetes in #2633; the VM driver had none of it, so
VM sandboxes on proxy-only networks could not reach any destination requiring
the proxy even when policy allowed it.
The blocking piece was not proxy logic but delivery: the VM guest init script
runs as PID 1 and execs a fixed supervisor command line, and libkrun's
krun_set_exec receives an empty argv, so there was no channel for driver-owned
supervisor arguments. The supervisor's proxy flags deliberately have no
environment fallback, and build_guest_environment merges user-supplied
environment, so the guest env is not a safe transport either.
Add a driver-authored argument file, mirroring the existing init.d manifest:
the driver writes /opt/openshell/supervisor-args into the overlay upperdir on
every launch and the guest reads it verbatim, one argument per line, appending
it to every supervisor exec. It is written even when empty, which is what makes
the channel unforgeable -- the upperdir always shadows the read-only image
layer, so an image can neither supply its own arguments nor disable the
operator's by omitting the file. Because both launch backends exec the same
init script, this covers libkrun and QEMU without touching either.
A microVM has no bind mounts or container secrets, so the credential and CA
bundle are staged into the per-sandbox overlay the way the gateway JWT already
is: credential root-only at 0600, CA at 0644, both rewritten every launch so a
removed setting clears prior material, and both deleted with the sandbox state
directory. This places the credential at rest in the overlay image on the
gateway host, which differs from the Podman secret model and is documented as
an explicit security consideration.
Validation is fail-closed and shared: a new
openshell_core::driver_utils::validate_upstream_proxy_settings holds the
pairing rules the Podman driver established, and both the gateway and the
driver call it so an invalid table names the offending key instead of
surfacing as an opaque driver-readiness timeout.
Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback
is reachable only through host.openshell.internal; the guest to gateway
callback is unaffected.
Closes#3088
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(vm): bound the proxy CA read and scope the host-loopback recipe
Two review findings on the corporate forward proxy support for microVM
sandboxes.
The driver read the operator's proxy_ca_bundle with an unbounded fs::read and
accepted it on a substring match for the PEM BEGIN CERTIFICATE marker. A
special file such as /dev/zero therefore grew driver memory without bound on
every authorized sandbox create, and a PEM block holding invalid DER passed
the host check but contributes no trust anchor in the guest, so every
supervisor would fail after boot with an error attributed to the sandbox
rather than to the setting.
Move the read into openshell-core as read_upstream_proxy_ca_bundle_file: it
reuses the credential reader's bounded-read path (non-regular files rejected
on fstat, size capped, read bounded even if the file grows), then requires at
least one anchor that RootCertStore::add_parsable_certificates accepts. The
supervisor's own reader now delegates to it, so host acceptance and guest
acceptance are the same function and cannot drift.
The published host-loopback recipe was written for libkrun only. gvproxy NATs
host.openshell.internal to the gateway host's 127.0.0.1, but GPU sandboxes run
on the QEMU/TAP backend where that name resolves to the TAP host address and
the driver's own nftables input chain accepts only the gateway port from the
guest — no proxy on the gateway host is reachable there at any bind address,
so an operator following the generic recipe lost all proxy-required egress
while configuration validation succeeded.
Scope the recipe to libkrun in every reference and reject a gateway-host proxy
URL when a launch plan resolves to QEMU, naming the reason, instead of booting
a sandbox whose policy-approved CONNECTs all time out.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(vm): match the QEMU proxy preflight to the selected TAP host
The gateway-host proxy guard added for the QEMU/TAP backend classified the
wrong set of addresses in both directions.
It ran at the top of configure_qemu_launch_plan, before the subnet allocation
that settles plan.host_ip, so it could not compare against the address the
guest actually reaches the host on. An operator pointing https_proxy at the
sandbox's own TAP host address, such as 10.0.128.1, passed the check, and the
driver's nftables input chain — which accepts only the gateway port from the
guest — then dropped every policy-approved CONNECT, which is exactly the
silent timeout the guard exists to prevent.
In the other direction it rejected 192.168.127.254 unconditionally. That
address is special only to libkrun/gvproxy; on QEMU/TAP it is an ordinary
address that may be routable through the guest's masqueraded egress, so the
guard refused a working configuration.
Run the check after the launch plan's network allocation, on both the
freshly-allocated and already-complete paths, and compare IP literals with
that sandbox's selected TAP host. Loopback literals, localhost, and the
documented host aliases that write_host_gateway_aliases seeds to the TAP host
still classify as the gateway host, and the failure names the address. The
gvproxy host-loopback constant returns to being a documentation anchor.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* feat(gateway): add installation name configuration
Add a first-class operator-assigned gateway name with TOML, CLI, environment,
and Helm configuration surfaces. Local gateways default to openshell, while
Helm defaults to the chart fullname; operators sharing a collector across
namespaces or clusters can set a globally distinct name.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(gateway): identify gateways in exported traces
Attach the configured gateway installation name and compute driver to the
gateway OpenTelemetry resource so operators can filter traces from multiple
installations that share a collector. Forward the gateway name and OTLP
endpoint to managed external drivers so their distinct service resources carry
the same installation identity.
Keep service.name stable per process type, omit blank resource values, and
leave per-span operation names and request attributes unchanged.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
---------
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Mirror the VM, Podman, and Docker driver tracing setup for Kubernetes.
Export standalone driver spans through OTLP/gRPC as the distinct
openshell-driver-kubernetes service, preserve gateway trace context, record
lifecycle operations and gRPC failures, and flush spans on shutdown.
Kubernetes currently runs in-process when selected as a built-in gateway
driver. Use the temporary server-boundary shim shared with Podman and Docker
so traces retain the shape they will have when Kubernetes moves to a
separate process. Move the common ComputeDriver RPC tracing layer into
openshell-otel to keep all drivers aligned.
Propagate the active W3C context through the controller-reserved Sandbox
annotation and enable Agent Sandbox OTLP export in the local k3s workflow.
This connects asynchronous controller reconciliation spans to the originating
OpenShell create trace.
Expose gateway OTLP configuration through Helm and add an Aspire collector
to the local k3s workflow. Extend helm:k3s:forward with OTLP ingest and trace
UI forwarding for Kubernetes and local container gateway development.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(core): fall back to podman CLI when no API socket responds
Auto-detection only checked well-known Podman socket paths, so a Podman
machine exposing its API socket at a non-standard location went
undetected. The symlink at a well-known path is not always present; it
varies by Podman version, machine provider, and platform.
Extend detect_podman_socket() to fall back to podman CLI discovery when
no well-known candidate responds: podman info --format json determines
whether the service is local or remote, and podman machine inspect
resolves the host-side forwarded socket for VM-backed machines. All
existing callers (driver auto-detection, the Podman driver, and the VM
driver's container-engine fallback) pick this up without change.
Select the machine backing the active Podman connection instead of the
first entry from podman machine inspect, honoring podman's connection
precedence: CONTAINER_CONNECTION, then CONTAINER_HOST (mapped to a
connection by URI), then the containers.conf default. An explicit
endpoint that maps to no known machine is left unresolved rather than
guessing an unrelated machine. When CONTAINER_HOST is an explicit unix://
socket, use that path directly since podman info connects through it.
Update the gateway config reference and the Podman driver README, which
described probe-only detection.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix(core): inspect podman machine by name and bound discovery probes
Address two startup defects in Podman socket auto-detection:
- Remote discovery ran `podman machine inspect` with no arguments, which
inspects only `podman-machine-default`. On a host whose default
connection is a different machine, selection fell back to the first
(wrong) entry, so the gateway could operate against the wrong backend.
Resolve the active machine first and inspect it by name, trying the
`-root`-stripped machine for rootful connections and returning None
when no machine can be mapped instead of substituting another.
- `podman info`, `podman machine inspect`, and `podman system connection
list` used unbounded `Command::output()`, so a stalled machine, SSH
connection, or helper could hang gateway startup indefinitely. Route
all three through a bounded runner that kills and reaps the child on a
documented deadline and returns None so detection can continue.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix(core): bound podman probe stdout drainage and kill descendants
The previous timeout only bounded waiting for the direct child to exit.
Draining stdout still called read_to_end, which waits for pipe EOF — a
Podman probe can leave a daemonized descendant (SSH multiplexer, gvproxy)
that inherited the stdout pipe, so drainage could wait indefinitely even
after the direct child exited, defeating the deadline.
Run each probe as its own process-group leader, cap stdout drainage by
the same deadline via a channel, and on expiry kill the whole process
group so descendants holding the pipe are terminated and EOF is reached.
Add a regression test where the direct child exits but a backgrounded
descendant keeps stdout open.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix(core): make podman probe deadline absolute against escaped descendants
The prior drainage timeout still ended in an untimed rx.recv() after
killing the process group. A descendant that escaped the probe's process
group (e.g. via setsid) while holding the stdout pipe would not be killed,
so read_to_end never saw EOF, the reader never sent, and that recv() could
block gateway startup indefinitely.
On drainage timeout, best-effort kill the process group and return None
immediately, abandoning the reader thread instead of waiting on it again.
The deadline now bounds the whole call regardless of what descendants do.
Add a regression test whose stdout-holding descendant escapes the process
group (`set -m`) and assert the call still returns promptly.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
---------
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* feat(sandbox,podman): trust corporate CA for https:// proxies and intercepted TLS
The corporate proxy chaining only accepted plain http:// proxy URLs, so
operators whose forward proxy terminates TLS with a private corporate CA
had no way to reach it, and TLS-intercepting proxies (mitmproxy, squid
ssl-bump) that re-sign tunneled server certificates broke every upstream
handshake after CONNECT.
The supervisor now accepts https:// proxy URLs: it wraps the connection to
the proxy in TLS before the CONNECT handshake, verifying the proxy
certificate against the built-in Mozilla roots, the system CA bundle, and an
optional operator corporate CA bundle. The upstream dial returns a
Plain/Tls stream enum consumed generically by the relay paths.
The corporate CA is delivered as a driver-supplied command-line argument
(--upstream-proxy-ca-bundle), never an environment variable, matching the
hardened proxy-config model where a sandbox image cannot influence the
operator's egress boundary. It is folded into the sandbox combined trust
bundle (write_ca_files) and the L7 upstream verification store
(build_upstream_client_config) at startup, so intercepted upstream
handshakes succeed and sandbox workloads trust the re-signed certificates.
Configuration is fail-closed: a CA bundle set without a proxy, or an
unreadable or certificate-free file, is fatal rather than silently
weakening the trust boundary.
The shared parse_upstream_proxy_url validator accepts https:// (recording
the scheme so the driver and supervisor agree), keeping the explicit-port
requirement. The Podman driver gains a proxy_ca_bundle operator setting
(TOML, --sandbox-proxy-ca-bundle, OPENSHELL_SANDBOX_PROXY_CA_BUNDLE) that
bind-mounts the host PEM read-only into the sandbox (a CA certificate is not
secret) and points --upstream-proxy-ca-bundle at it, with a create-time
readability check.
The standalone dev gateway task passes OPENSHELL_SANDBOX_PROXY_CA_BUNDLE
through to the generated podman config, so a local gateway can be pointed at
a TLS-intercepting proxy without hand-editing the regenerated TOML.
Refs #1792
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox): reject CA bundles with valid PEM framing but invalid X.509 DER
The proxy CA bundle validation counted PEM blocks that base64-decoded
successfully, but did not verify the decoded bytes were accepted as
trust anchors by RootCertStore. A bundle with syntactically valid PEM
framing but invalid DER would pass the startup check while contributing
zero usable anchors, causing opaque TLS failures at runtime instead of
a fail-closed startup error.
Validate decoded certificates through RootCertStore::add_parsable_certificates
and reject the bundle unless at least one is accepted.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox,podman): allow proxy auth without insecure acknowledgement for https:// proxies
For an https:// proxy the Proxy-Authorization credential travels inside
the verified TLS session, so the proxy_auth_allow_insecure
acknowledgement is unnecessary. Previously both http:// and https://
proxies required it, producing a misleading cleartext-risk diagnostic
for a path that is already encrypted.
Skip the requirement when the proxy URL uses https://; the
acknowledgement is still tolerated if set. Updated in both the
supervisor and Podman driver validation paths, with docs and tests.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix: format
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(kubernetes): use truly unsupported scheme in proxy validation test
https:// is now a supported proxy scheme after a13c4dce, so the
unsupported-scheme test must use a genuinely unsupported scheme.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(e2e): sign the https proxy fixture listener cert with a CA
The corporate-proxy E2E fixture served a single `openssl req -x509`
certificate as its TLS listener identity. OpenSSL marks that certificate
`basicConstraints: critical, CA:TRUE`, and rustls refuses a CA
certificate presented as an end-entity certificate (CaUsedAsEndEntity).
The supervisor's TLS handshake with the proxy therefore failed, the
upstream dial errored, and the workload's CONNECT was dropped without a
response, so podman_corporate_proxy_trusts_ca_bundle_for_https_proxy
failed on the approved destination while policy denial still worked.
Generate a corporate CA and a separate listener leaf signed by it, serve
the leaf chain, and publish only the CA as the bundle the supervisor
trusts. This is what an intercepting proxy actually presents, and it
exercises the corporate-CA trust path rather than pinning the listener
certificate itself.
Refs #1792
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
Mirror the VM and Podman driver tracing setup for Docker. Export Docker
driver spans through OTLP/gRPC as the distinct openshell-driver-docker
service, preserve gateway trace context, record lifecycle and asynchronous
provisioning operations, and report gRPC failures.
Docker currently runs in-process when selected as a built-in gateway driver.
Add the same temporary server-boundary shim used by Podman so traces retain
the shape they will have when Docker moves to a separate process. Generalize
the gateway provider selection for both in-process drivers and share the OTLP
collector fixture across Docker, Podman, and VM tracing tests.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(provider): add ability to request token exchange instead of client credentials as OAuth grant_type
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(proxy): add further tests for token exchange
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(provider): add runnable example for token exchange
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(e2e): cover Podman token exchange grants
Signed-off-by: Gordon Sim <gsim@redhat.com>
* refactor(oauth): extract duplicated functionality from server and supervisor
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(provider): evict nearest-to-expiry entry from intermediate token cache
Signed-off-by: Gordon Sim <gsim@redhat.com>
* doc(supervisor): add podman example for token exchange
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(provider): withhold token-exchange subject credentials
Signed-off-by: Gordon Sim <gsim@redhat.com>
---------
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(cli): support OIDC device authorization grant for headless login
Closes#2793
Add OAuth 2.0 Device Authorization Grant (RFC 8628) support to the OpenShell CLI's OIDC login flow. When running in a headless environment (OPENSHELL_NO_BROWSER=1) without a client secret configured, the CLI now uses the device code flow instead of the browser-based PKCE flow.
The device code flow:
- Requests a device code and user code from the IdP's device authorization endpoint
- Displays a verification URL and user code to the user
- Polls the token endpoint until the user completes authorization or the code expires
- Supports slow_down responses per RFC 8628 by increasing the polling interval
This implementation:
- Extends OidcDiscovery to optionally capture device_authorization_endpoint
- Adds oidc_device_code_flow function with proper error handling for all RFC 8628 error codes
- Updates gateway add and gateway login to dispatch to device flow when browser is suppressed
- Adds comprehensive unit tests for device flow structs and response parsing
- Updates gateway authentication documentation to describe the device code fallback
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* fix(cli): validate OIDC device token responses
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* fix(cli): add PKCE to OIDC device flow
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* docs(cli): document PKCE device flow
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
---------
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* refactor(driver): extract shared supervisor binary helpers
Move supervisor binary extraction, caching, and validation helpers from
the Docker driver into openshell-core::driver_utils so both Docker and
Podman drivers can reuse them.
Moved helpers: extract_first_tar_entry, write_cache_binary_atomic,
supervisor_cache_path, temp_extract_container_name, and
validate_linux_elf_binary.
The shared extract_first_tar_entry gains entry-type and empty-payload
checks that the Docker-local version lacked. supervisor_cache_path
takes a driver_subdir parameter so each driver caches under its own
namespace (docker-supervisor vs podman-supervisor).
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
* feat(driver-podman): add userns config
Add a `userns` option to the Podman compute driver that maps to
Podman's user namespace modes. The mode string is split on the first
colon into the API's `nsmode` and `value` fields so parameterized
values like `auto:size=65536` and `keep-id:uid=1000,gid=1000` are
forwarded correctly. When the mode is `auto`, the container spec
also sets `idmappings.AutoUserNs = true` as required by the API.
An allowlist validates the mode at startup: `auto` and `keep-id`
accept optional parameters; `host`, `private`, and `nomap` reject
them; everything else is an error.
Podman image volumes use overlay mounts internally and the kernel
does not support idmapped mounts on overlay (`mount_setattr` returns
EINVAL). When userns is configured (any mode except `host`), the
driver extracts the supervisor binary from the image to a host-side
cache and bind-mounts it instead of using an image volume.
Configurable via TOML `userns = "auto"`, CLI `--userns`, or
environment variable `OPENSHELL_PODMAN_USERNS`.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
---------
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Johnny Greco