mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-18 05:39:50 +08:00
pull-request/2941
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 VarshaandJohn Myers
|
0803c4aa4c |
refactor(policy)!: remove NetworkBinary harness field (#3222)
* refactor(policy)!: remove NetworkBinary harness field Closes #3054 Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(policy): preserve unknown fields during migration Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(policy): preserve advisor provenance during merge Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(policy): remove redundant network binary default Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): record harness schema migration Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Piotr Mlocek
|
3112e9cd3c |
fix(mxc): resolve Windows ETW clippy lints (#3270)
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> |
||
|
|
ae57979b03 |
feat(mxc): add Windows ETW-to-OCSF audit trail (#3015)
* feat(mxc): ETW->OCSF audit consumer + Windows OCSF JSONL parity (cp6 P1) Add a Windows MXC ETW->OCSF audit trail in openshell-driver-mxc: a real-time Sandboxing-provider ETW consumer that decodes events (TDH), attributes each to an OpenShell sandbox_id, and maps them to OCSF (lifecycle 6002, config 5019, process 1007, finding 2004). cp6 Phase 1 - durable OCSF JSONL audit-file parity with Linux: - openshell-ocsf: add emit_ocsf_event_routed (populates the event-bridge thread-local AND stamps sandbox_id+message in one dispatch) plus public set/clear_current_event; OS-aware device (Device::windows/for_current_os) so device.os.name reflects the host instead of a hardcoded Linux stub. - etw_consumer: emit via the routed emit (previously fired a bare info! that never populated the bridge, so the structured event was dropped). - openshell-server: install OcsfJsonlLayer over a synchronous daily-rotated appender (durable under force-kill), gated by OPENSHELL_OCSF_JSON, path via %PROGRAMDATA%\OpenShell\logs (override OPENSHELL_OCSF_LOG_DIR). - device.hostname now resolves to the real gateway machine name. Box-proven on 7F203-MXC-001: JSONL lines == shorthand OCSF rows, all valid OCSF JSON, per-sandbox attribution intact, disabled state writes nothing. Signed-off-by: Akber Raza <akberr@nvidia.com> * feat(mxc): map remaining Sandboxing ETW events to OCSF Close the last three ETW->OCSF gaps so the audit trail covers the full set of events the Sandboxing provider emits (12/12): - ProcessLaunched -> Process Activity [1007] "Launch" (confirmed start; carries the real processId/threadId, the twin of CreateProcessInSandbox which only has the request + command line). - SandboxProxyConfigured -> Device Config State Change [5019] (the one network-plane setup event; surfaces proxyPort, "no proxy" when 0). - SandboxConsoleReferencePlumbed -> Device Config State Change [5019] (console-handle plumbing). map_config_state now handles the full config/hardening/setup family and carries proxyPort/hasConsoleReference/creationFlags as unmapped fields. Verified on 7F203-MXC-001: 11/12 event types emit OCSF without a proxy (SandboxProxyConfigured requires proxy config to fire). Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc): seed ETW attribution under registry lock + Device tests Address CodeRabbit review on !31: - Prevent stale ETW attribution on a delete/launch race: register the wxc-exec pid while holding the registry lock, and bail if the sandbox entry is already gone. Previously the attribution key could be seeded after `delete` had removed the sandbox, leaving a stale key that could misroute later Sandboxing ETW events to a dead sandbox_id. Lock order (registry -> attribution) matches the delete path, so no deadlock. - Add unit tests for the new Device::windows and Device::for_current_os constructors to harden Windows/Linux OCSF device parity. Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc-etw): buffer+replay racing events and harden attribution keys Addresses two ETW->OCSF attribution review items (Shailendra #1, #2). #2 early-event loss: ETW delivers the sandbox create/config burst the instant wxc-exec starts, which can beat the driver's register_launch (now under the registry lock post-Ready). process_event previously dropped anything unresolved, losing the racing burst. Add a bounded, time-bounded pending buffer (PENDING_MAX=4096, PENDING_TTL=5s): unresolved events are held and replayed once attribution lands, aged-out ones dropped. Consumer switched to a timed recv_timeout(200ms) so the buffer is re-driven after each event and on a tick. Emit path factored into shared emit_resolved(). #1 attribution collisions: a Windows PID is recycled after exit and a command line is commonly identical across sandboxes. register_launch now rebinds by_pid on reuse and clears the stale last_pid_sid hint (warns if the PID still pointed at a different, leaked sandbox); command line is held in by_cmd only while unique and demoted to a new ambiguous_cmds set on a second owner, so a duplicate command refuses to resolve rather than misroute. Unit tests: buffer replay (direct + cross-link), buffer bound, PID-reuse rebind, duplicate-cmd non-resolution. Box-verified on 7F203-MXC-001 (5 sandboxes, identical cmd -> 5 isolated sandbox_ids, 50/50 OCSF/JSONL, BuffersLost=0). Signed-off-by: Akber Raza <akberr@nvidia.com> * docs(mxc-etw): note cmd_line is captured raw with no privacy filtering Review item #3 (Shailendra): add a PRIVACY NOTE on map_process_launch stating cmd_line is copied verbatim into OCSF process.cmd_line with no redaction, so secrets/PII on a command line land unredacted in the durable audit trail (deliberate audit-fidelity trade-off; treat the log as sensitive). Redaction is owned by an upstream privacy layer, not this path; no general audit-output PII scrubber exists today (openshell_core::secrets [CREDENTIAL] redaction is scoped to the proxy HTTP-target logging, a separate egress path). Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc-etw): open ETW trace on caller thread so start_session reports real status Review item #4 (Shailendra): start_session previously returned Ok(EtwSession) as soon as the pump thread was spawned, but OpenTraceW ran later inside that thread; if it failed we still handed back a live-looking session and logged 'consumer started' (silent failure = false audit coverage). Split the two Win32 calls instead of adding a channel handshake (avoids any lost-wakeup/hang risk): the quick, synchronous OpenTraceW now runs on the caller thread (open_trace), and only the blocking ProcessTrace runs on the pump thread (run_trace). start_session returns Err if OpenTraceW fails (reclaiming the boxed Sender so the consumer disconnects, stopping the session, joining the consumer) and returns Ok/logs 'started' only once capture is genuinely open. Opened handle + LoggerName buffer + boxed Sender are carried to the pump via a Send OpenedTrace so they outlive ProcessTrace. Box-verified on 7F203-MXC-001: consumer started=True, failed-to-start=False, 50 OCSF rows / 50 JSONL, BuffersLost=0 (no regression to capture/emit). Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc-etw): guard pending-event replay against PID recycling CodeRabbit flagged that drain_resolved() re-resolved buffered events against the live by_pid map, so if Windows recycled a wxc-exec PID within PENDING_TTL a stale event from the dead sandbox could be emitted under the new owner. Stamp each by_pid registration with its Instant and add resolve_replay(), used only on the buffered/replay path. It (a) never falls back to the recycle-/ambiguity-prone by_cmd or last_pid_sid keys, and (b) trusts a PID match only when the registration is not newer than the buffered event by more than REPLAY_PID_GRACE (2s) - a recycled PID's registration lands well outside that window, so the stale event ages out instead of misattributing. The legitimate #2 seed race (registration lands ~immediately) still replays. Adds unit tests for the recycle-refusal, in-grace acceptance, and weak-fallback exclusion. Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc-etw): surface unexpected ProcessTrace termination (review #4) start_session already returns Err on OpenTraceW failure (runs on the caller thread since e41a7701), closing the first half of Shailendra's #4. This closes the second half: ProcessTrace's result was discarded, so if capture died mid-run the backend had no way to know. Add a shared CaptureHealth (stopped/stopping/exit_code) between the pump thread and EtwSession. run_trace now records ProcessTrace's WIN32_ERROR and, when the pump returns without a deliberate stop, logs at ERROR that MXC OCSF capture is no longer running. EtwSession::stop() sets `stopping` before teardown so a normal shutdown isn't misreported, and EtwSession::is_capture_alive() exposes the state for status/diagnostics. Box-verified on 7F203-MXC-001: 5 sandboxes, 50 attributed OCSF rows, JSONL parity 50/50, BuffersLost=0, clean start/stop (no false failure). Signed-off-by: Akber Raza <akberr@nvidia.com> * feat(mxc-ocsf): add ETW->OCSF audit-trail example kit; fix proxy-configured message Add a runnable OCSF audit-trail example under examples/ (run-ocsf-audit.ps1, mxc-ocsf-audit.toml, ocsf-audit.yaml, README) that spins up sandboxes with the in-process ETW consumer and egress proxy on, emitting a full OCSF JSONL audit trail across all four classes (6002/5019/1007/2004). Fix SandboxProxyConfigured mapping to log "MXC sandbox proxy configured" instead of a misleading "(no proxy)" when the provider reports proxyPort=0; the event's presence already indicates proxy configuration. Verified on-box: 26 events, all mapped ETW event types present. Signed-off-by: Akber Raza <akberr@nvidia.com> * feat(mxc-ocsf): clearer audit report + client-safe run-ocsf-audit.ps1 Improve the ETW to OCSF audit-trail example output and make it safe to ship. Report: - Add an event-type coverage count ("N of M expected event types fired"); the denominator auto-adjusts (8 with proxy on, 7 with -NoProxy). - Split the checklist into expected event types vs anomaly findings (ActivityError/FallbackError), which are reported separately and not counted toward coverage (a clean run may emit none). - Verdict is now coverage-based (all expected types must fire) instead of the looser "at least 3 OCSF classes". - Call out the absolute path to the durable OCSF JSONL log prominently. Client-safety: - Default -ShareOut to empty (no auto-copy); pass -ShareOut a UNC path to opt in. Removes a hardcoded internal share path from a published example. - Drop internal-team wording ("Hand that zip back for evaluation", "BUNDLE:") in favor of neutral "Results bundle:". - Update README-ocsf-audit.txt to match the opt-in -ShareOut behavior. Verified on both MXC boxes: 7F203-MXC-001 (base-container) -> PASS, 8 of 8 event types, 26 OCSF events across 4 classes; 7F203-MXC-003 (AppContainer fallback) -> reduced set as expected, clean output. Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc): configure OCSF audit workloads per sandbox - remove unsupported gateway-scoped workload fields from the shipped MXC audit example. - build the command, working directory, and filesystem grant from each run's ShareDir - pass the workload through --driver-config-json. - preserve the host CONNECT proxy configuration and conditional audit coverage for the future host_connect_proxy merge - require the workload output when determining the audit verdict. Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(mxc): omit command arguments from OCSF audit logs - record only the executable basename for MXC CreateProcessInSandbox audit events - leave process.cmd_line unset so workload arguments cannot reach shorthand or JSONL logs - cover tokens, passwords, signed URLs, and PII with a secret-leak regression test - update the audit example, architecture guidance, and published logging documentation - preserve ETW attribution and future host_connect_proxy enforcement behavior Signed-off-by: Akber Raza <akberr@nvidia.com> * feat(etw): enhance ETW session management with distinct naming for concurrent gateways * fix(etw): bound the audit queue during overload - replace the unbounded ETW callback channel with count- and byte-bounded buffering - keep the ETW callback non-blocking and count records rejected during overload - emit immediate, rate-limited warnings that identify resulting audit coverage gaps - make the audit example fail when queue overload causes dropped ETW records - cover stalled consumers, oversized events, and warning throttling with unit tests Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(etw): harden sandbox audit attribution - remove command-line and persistent per-PID fallback keys from live and replay resolution - retire the driver-owned wxc-exec PID before publishing child completion - retain established identity, activity, and correlation-vector links only for the five-second late-event window - prevent buffered records from crossing rapid PID retirement and reuse boundaries - add resolver and lifecycle coverage and document the attribution trust boundary Signed-off-by: Akber Raza <akberr@nvidia.com> * chore(mxc): address rebase follow-ups Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(mxc): align OCSF audit example with driver config - remove unsupported egress proxy settings - stop requiring the unavailable proxy audit event - update example documentation for supported event coverage Signed-off-by: Akber Raza <akberr@nvidia.com> * fix(etw): redact command-line secrets in DecodedEtwEvent summary * fix(etw): enhance PID resolution and event attribution logic for ETW records * fix(ocsf): restrict gateway-local JSONL sink to Windows/MXC path with opt-in configuration * address rebase issues * fix(mxc): address ETW audit review feedback Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(mxc): fail closed across ambiguous PID reuse Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(mxc): bind ETW attribution to process generation Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Akber Raza <akberr@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Jamie King <jamiek@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Piotr Mlocek
|
ddc8bba967 |
ci(windows): add Windows MSVC CI jobs (#2738)
* fix(ci): preserve Windows Rust build cache Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): invalidate empty Windows caches Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * perf(ci): cache Windows builds with sccache Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): restore target directory caching Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * perf(ci): use prebuilt Z3 on Windows Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * perf(ci): layer sccache on Windows target cache Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): split PR checks from main validation Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): separate checks builds and cache seeding Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): simplify Windows build dependency Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): rely on Windows job dependency status Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): use valid opt-in Windows ARM runner Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): keep ARM64 validation local Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): install Clippy for Windows validation Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): focus platform lint coverage Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(licenses): explain bzip2 allowance Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): simplify workflow name Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(windows): allow async platform stub Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(network): make file fingerprints portable Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): lint supported deliverables Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): allow platform-gated lint Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * chore(ci): align Windows cache action with main Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): align Windows validation with prerequisites Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): pin Rust toolchain action Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): use enterprise-approved Windows actions Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(windows): restore strict MSVC validation Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): run Rust tests with nextest Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): normalize nextest lock provenance Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): add native arm64 validation Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): lock nextest for Windows ARM64 Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(windows): resolve duplicate MXC authentication method Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * test(conformance): use native absolute paths on Windows Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(windows): address MSVC review feedback Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): simplify cache key names Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): isolate Windows Rust toolchains for stable caches Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(ci): configure Rustup home in runner setup Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): surface sccache server write diagnostics Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): remove temporary cache diagnostics Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * ci(windows): address review feedback Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(windows): reconcile merged driver capabilities Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(deps): preserve AWS-LC-only lockfile Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * chore(deps): allow z3 prebuilt TLS wrapper Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> |
||
|
Philippe Martin
|
118b250f01 |
feat(sandbox): support rootfs tar as --from source for VM driver (#2863)
* feat(sandbox): support rootfs tar as --from source for VM driver Accept flat rootfs tar archives (.tar, .tar.gz, .tgz) via the --from flag for VM-backed gateways. The CLI detects the archive extension, validates that the gateway uses the VM compute driver, and passes the tar path through driver_config. The VM driver copies the tar into its staging area and feeds it into the existing rootfs extraction and ext4 disk creation pipeline, skipping the container image pull/export steps. Closes #2175 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox): validate rootfs tar path at the VM driver boundary The rootfs_tar_path field in driver_config was passed from the API caller directly to tokio::fs::copy without validation. An authenticated user bypassing the CLI could supply arbitrary host paths (e.g. /dev/zero for disk exhaustion, or readable host files for data exfiltration). Introduce a trusted staging directory that the VM driver creates on startup and advertises via GetCapabilities. The CLI now copies the tar into the staging directory before creating the sandbox, and the driver validates that the received path is a regular file inside the staging root and within a configurable size limit (default 10 GiB) before any I/O. New VmDriverConfig options: - rootfs_tar_staging_dir: override the staging directory (default: <state_dir>/rootfs-tar-staging) - rootfs_tar_max_bytes: override the size limit (default: 10 GiB) Addresses GATOR-28b5152e-01. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox): request-scoped staging, size pre-check, and cleanup for rootfs tar Tighten the rootfs tar staging flow to address the remaining GATOR-01 obligations: - Request-scoped staging: the CLI creates a unique per-request subdirectory (req-<pid>) under the staging root instead of placing files directly in the shared directory. The driver enforces that the tar path is at depth 2 (staging_root/<subdir>/<file>), preventing cross-request path selection. - Size pre-check: the driver advertises rootfs_tar_max_bytes via GetCapabilities. The CLI reads this limit and rejects oversized files before copying, avoiding disk exhaustion in the staging directory. - Cleanup: the driver removes the request staging subdirectory after consuming the tar (on cache hit, copy success, or copy failure), ensuring staged data does not persist beyond the request. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): restore rootfs-tar sandboxes from persisted image identity On restore or restart, the one-shot staged tar archive has already been cleaned up. Reading the persisted image identity from the sandbox state directory and resolving the cached disk path directly avoids re-accessing the deleted staging path. Addresses GATOR-168b9210-01. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(cli): use random staging dirs and enforce byte limit during rootfs tar copy Replace PID-based request staging directories with tempfile-generated random names to prevent collisions and make paths unpredictable. Replace bare tokio::fs::copy with a streaming copy loop that enforces the advertised max_bytes limit during transfer, closing the TOCTOU gap between the pre-copy size check and the actual copy. Signed-off-by: Philippe Martin <phmartin@nvidia.com> Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox): issue rootfs tar staging slots from the gateway A caller could name any host path in `driver_config.vm.rootfs_tar_path`, which the privileged VM driver then read. The CLI-side locality check did not apply to direct API requests. The gateway now owns staging. `BeginRootfsTarStaging` allocates a request-scoped directory under the driver-advertised staging root and returns an opaque single-use token; `CreateSandbox` carries the token, and the gateway substitutes the path it allocated before dispatching to the driver. `template.driver_config.<driver>.rootfs_tar_path` is rejected outright in request validation, so a caller-supplied path never reaches privileged I/O. Tokens are bound to the issuing workspace and subject, consumed once, and expire after 30 minutes. Outstanding slots are capped per caller and overall, so one caller can neither exhaust the staging filesystem nor starve others. An RAII guard reclaims the directory on every failure path after consumption, and an age-gated sweep runs at startup and on each reconcile pass for directories whose driver died before its own cleanup. The token is stripped from the public sandbox before persistence: the stored copy is returned verbatim by GetSandbox, ListSandboxes and WatchSandbox to every member of the workspace. Also fixes two defects this exposed: - The CLI wrote `rootfs_tar_path` at the top level of `driver_config`, but the gateway forwards only `driver_config.<driver_name>`, silently dropping unmatched keys. The archive never reached the VM driver, so the documented `--from ./rootfs.tar` flow did not work at all. Config is now nested under `vm` and deep-merged, so a caller's existing VM settings survive instead of being clobbered by a shallow extend. - Staging previously required `GetGatewayInfo`, which is restricted to `platform_admin`, making the feature unusable for ordinary users on any RBAC-enabled gateway. The new RPC matches CreateSandbox at `sandbox:write` / `workspace_role: user`. `compute_driver.proto` is unchanged; the gateway reads the staging root from the capabilities it already stores. Refs #2175 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): derive rootfs tar cache identity from archive contents The prepared-disk cache key combined the archive's full path with an mtime truncated to seconds, then mapped punctuation to `-`. Distinct paths such as `/tmp/a/b.tar` and `/tmp/a-b.tar` collapsed onto the same key and reused each other's disk, a rewrite within the same second kept stale contents, and a long path could exceed filesystem component limits. Identity is now a SHA-256 of the archive contents. This is also what makes the cache work at all now that the gateway allocates a fresh staging directory per request: a path-derived key would miss on every create. The archive is hashed, the cache checked, and only on a miss copied — so a hit skips writing a multi-gigabyte file. The copy is hashed as it is written and rejected if the digest differs from the first pass, which closes the window where the source changes during staging rather than approximating it with a re-stat. Refs #2175 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): decompress gzip rootfs tar archives during staging `--from` accepts `.tar.gz` and `.tgz`, but the driver staged whatever bytes it was given and the guest image-prep VM extracts the staged file with a plain `tar -xpf`. Compressed sources therefore depended on the guest tar auto-detecting gzip, and the prepared disk was sized from the compressed length, which is far too small for the expanded rootfs. Staging now detects gzip from the archive's magic bytes -- the driver only ever sees a gateway-issued staging path, never the caller's file name -- and writes an uncompressed tar. The digest still covers the source bytes, so the "archive changed while staging" check is unaffected, and expansion is bounded by `rootfs_tar_max_bytes` so a compression bomb cannot fill the host disk. `extract_rootfs_archive_to` sniffs gzip as well, so the host-side extraction path matches. Adds unit coverage for gzip staging, bounded expansion, and gzip extraction, plus an e2e sandbox created from a gzip-compressed export. Signed-off-by: Philippe Martin <phmartin@redhat.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> Signed-off-by: Philippe Martin <phmartin@nvidia.com> |
||
|
Evan Lezar
|
1510e2c5a7 |
feat(compute): advertise resource capabilities (#3010)
* refactor(driver-docker): group runtime GPU capabilities Signed-off-by: Evan Lezar <elezar@nvidia.com> * feat(compute): advertise resource capabilities Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(compute): describe resource capabilities Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(server): satisfy resource capability clippy lint Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Jeff MAURY
|
8719874f2a |
fix(driver-mxc): implement authenticate_sandbox trait method (#3158)
Closes #3155 Add the missing authenticate_sandbox method to the ComputeDriver impl for the MXC driver. The method returns Status::unimplemented, matching the Docker and Podman drivers, since the MXC driver advertises supports_sandbox_authentication: false. Signed-off-by: Jeff MAURY <jmaury@nvidia.com> Signed-off-by: Jeff MAURY <jmaury@redhat.com> |
||
|
Drew Newberry
|
9ca19e6c80 |
refactor(compute): decouple gateway driver composition (#2823)
* refactor(compute): decouple gateway driver composition Move first-party composition and VM process ownership into openshell-gateway, leaving openshell-server backend-independent. Update packaging and build references with the new crate, simplify the compiled-driver boundary, and keep the driver-free gateway path buildable with bundled Z3 tooling. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(telemetry): bound compute driver categories Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(core): keep runtime transport generic Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(compute): complete server driver decoupling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(compute): preserve driver integrations after rebase Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * fix(compute): preserve docker tracing after decoupling Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * fix(compute): preserve driver behavior after extraction Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(compute): remove MXC policy side channel Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(compute): separate policy delivery from readiness Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> |
||
|
|
bcd517bbe0 |
feat(driver-mxc): native Windows MXC compute driver + server wiring (#2721)
* feat(driver): add MXC compute driver for Windows isolation sessions Introduces the openshell-driver-mxc crate implementing ComputeDriver backed by Microsoft MXC isolation sessions (Windows only). Wires the new driver into the server's build_compute_runtime dispatch and adds the Mxc variant to ComputeDriverKind. Also adds a local protobuf-src stub (tools/protobuf-src-local) to unblock Windows builds that lack MSYS2/MinGW, and pins the zig Windows x64 toolchain in mise.lock. (cherry picked from commit 4f7012224efb18fbfeb47aa87e0cfd3f036f32f0) Signed-off-by: Jamie King <jamiek@nvidia.com> * wip(mxc): checkpoint hung-agent work (recon, policy_map embed, A1 wiring, demo artifacts) Safety checkpoint of uncommitted work from the background agent run that stalled mid-Step-7. Includes: mxc-driver-recon.md (Step 0.5), policy_map.rs (~876L embedded mapper), A1 policy-threading edits across driver.rs/policy.rs/mxc.rs/compute/mod.rs, and examples/ (demo.yaml + mxc-gateway.toml). Not yet verified to compile end-to-end; to be reorganized into the skill's Step 11 commit sequence. (cherry picked from commit 38e42c03870be3d10e984a54f17a3b61122ff510) Signed-off-by: Jamie King <jamiek@nvidia.com> * test(mxc): fix lifecycle and policy unit-test compile drift - Bring futures::StreamExt into scope for the watch-stream `.next()` call in driver::lifecycle_tests so the negative policy proof test compiles. - Bind a local `mapper` and drop the unused/deprecated NetworkBinary in the embedded-mapper network-policy rejection test. Signed-off-by: Jamie King <jamiek@nvidia.com> (cherry picked from commit 039b0baf98735ca672dae52be8d3af2417dc0c1a) Signed-off-by: Jamie King <jamiek@nvidia.com> * fix(mxc): downgrade missing sandbox_token to debug log The gateway mints `sandbox_token` only when a sandbox-JWT issuer is configured. There is no in-sandbox supervisor on MXC (supervisor-removal design — D1/D4), so no component ever consumes the token; requiring it on the driver side blocks the demo's `--disable-tls` smoke gateway with a spurious `invalid_argument`. Log the absence and proceed instead. Signed-off-by: Jamie King <jamiek@nvidia.com> (cherry picked from commit cea209797d0edcb1d152251e748900b0a63cca62) Signed-off-by: Jamie King <jamiek@nvidia.com> * fix(mxc): keep sandbox Ready after a successful one-shot agent exec monitor_exec demoted Ready->Error on exit 0 (reason ExecCompleted), so the positive demo (write hello.txt + exit) landed in Error phase. Keep Ready=True (reason AgentCompleted) on success; only non-zero exits go to ExecFailed. Tighten the positive lifecycle test to assert the terminal condition stays Ready=True/AgentCompleted. Verified live via gateway mock round-trip: phase now Provisioning->Ready with no demotion. (cherry picked from commit 54ab030f03ca0f83d0050d8dd843b633651684ad) Signed-off-by: Jamie King <jamiek@nvidia.com> * feat(mxc): add processContainer backend for default-deny enforcement Add a backend selector to the MXC driver (isolation_session default | process_container). process_container drives a one-shot AppContainer that is genuinely default-deny: a write to any ungranted path is denied by the OS, unlike isolation_session which is grant-only and cannot deny. The lifecycle forks on the flag - isolation_session keeps provision/start/exec, process_container runs a single ephemeral container via run_oneshot. Also: run-demo.ps1 gains -Backend and hardens the CLI register/create calls; docs corrected to state isolation_session does NOT deny out-of-policy writes and that the negative proof requires process_container. Verified end-to-end on a real demo box (gateway -> CLI -> driver -> MXC): in-policy write succeeds, out-of-policy write denied (PermissionDenied), OVERALL: PASS. (cherry picked from commit c6cde3860bbe1b8edb3147d3e840f6bf0ece32d8) Signed-off-by: Jamie King <jamiek@nvidia.com> * refactor(driver-mxc): embed policy mapper as a module; remove standalone crate Adopt the proto-based mapper (map_to_mxc) as the single source of truth, embedded in openshell-driver-mxc as a Windows-gated `policy_map` module. Rewire EmbeddedPolicyMapper to call it directly on the typed SandboxPolicy, deleting the serde_yaml proto->YAML bridge. Move the CLI to a windows-gated example and the parity tests into the crate; delete openshell-policy-mapper. - gate policy_map + seam Windows-only (MXC is Windows-only) - drop serde_yaml; add dev-deps openshell-policy, clap, anyhow - normalize mapped paths to Windows form in the seam, in one place - docs: add driver-mxc to AGENTS.md table; correct design doc section 17 test lane Signed-off-by: Giedrius Burachas <gburachas@nvidia.com> (cherry picked from commit f22f9c7a25b9a651c5c5cc73f62fb01c4d6c1a8d) Signed-off-by: Jamie King <jamiek@nvidia.com> * feat(driver-mxc): implement lossless split_policy for proxy-delegated egress Signed-off-by: Giedrius Burachas <gburachas@nvidia.com> (cherry picked from commit 96d6afa0e2dc6a1d54edd12c34a0ceb0a30dadd0) Signed-off-by: Jamie King <jamiek@nvidia.com> * feat(driver-mxc): implement Pattern-C governed-egress split through the policy seam - split_policy: SocketAddr proxy_redirect (replaces bare port), processcontainer containment guard naming MXC M1, version preserved in the trimmed proxy_policy, delegation reported as an info loss item - seam: MappedConfig carries trimmed_policy + proxy_addr; MapCtx.egress selects the split path; coarse path unchanged when egress is disabled - driver: [openshell.drivers.mxc] egress_proxy / egress_proxy_addr config, validated at create (isolation_session rejected until M1); lifecycle threads the redirect into provision and stores the trimmed policy per sandbox, emitting an EgressRedirect platform event - mxc: optional MxcNetwork block (defaultPolicy=block + proxy) in provision and one-shot configs; mock records configs for test assertions - tests: lossless-invariant suite over all example policies (validate + serialize round-trip), split lifecycle proof, M1 rejection; example gains --split --proxy-addr writing mxc-config.json / trimmed-policy.yaml / loss-report.json Signed-off-by: Giedrius Burachas <gburachas@nvidia.com> (cherry picked from commit 34d54ad9f25dc6034c3ba15668555ff0d22cddd8) Signed-off-by: Jamie King <jamiek@nvidia.com> * fix(driver-mxc): emit MXC network.proxy as {localhost: port} Verified against the real wxc-exec 0.6.0-alpha via --dry-run: MXC accepts only the {localhost: N} proxy shape (the form the design doc specifies) and rejects {host, port} with a parse error. Schema 0.6.0-alpha can express only a loopback port, so non-127.0.0.1 redirect addresses are now rejected: split_policy emits an error loss (no proxy block) and the driver refuses egress_proxy_addr values off 127.0.0.1. Per-sandbox attribution must use per-sandbox ports until the schema widens. Signed-off-by: Giedrius Burachas <gburachas@nvidia.com> (cherry picked from commit edde8d5434571fd5398409204fcf6862672c0793) Signed-off-by: Jamie King <jamiek@nvidia.com> * fix(driver-mxc): serialize isolation_session stop/deprovision as unit variants Empirical contract finding from the real test lane (build 26300.8553, wxc-exec 2026-06-10): the stop and deprovision experimental blocks are unit variants in the wxc-exec schema and must serialize as null; sending {} is rejected with malformed_request (invalid type: map, expected unit), while provision/start accept maps. The production invoker, the real-lane test, the probe script, and the e2e runner all sent {} - the driver could provision and run an agent but never stop or delete an isolation-session sandbox against this build. Pinned by a unit test. Signed-off-by: Giedrius Burachas <gburachas@nvidia.com> (cherry picked from commit 0df39ca0b22ebc21eb965b2b567a5b2cac26af32) Signed-off-by: Jamie King <jamiek@nvidia.com> * test(driver-mxc): add Tier-0 mapper coverage matrix with schema drift guard Three-quadrant, table-driven matrix (38 tests): mappable fields assert exact MXC output; every OpenShell field MXC cannot express asserts a loss item with the expected severity (and seam rejection on error); an empty policy asserts the restrictive default-deny posture for every MXC knob OpenShell does not control. The handled_fields_inventory drift guard serializes a fully-populated policy and compares its YAML keys against the mapper-handled field lists, so a new openshell-policy field fails the suite until consciously mapped, delegated, or reported as loss. Re-exports the policy seam types for integration tests; adds serde_yml, base64, serde_json as dev-dependencies. Signed-off-by: Giedrius Burachas <gburachas@nvidia.com> (cherry picked from commit 91807f984a3b16846e35d6ca0d5ec41057aafa3a) Signed-off-by: Jamie King <jamiek@nvidia.com> * feat(driver-mxc): inject agent_env into sandbox process.env Add MxcComputeConfig.agent_env: each entry is either KEY=VALUE (verbatim) or a bare KEY resolved from the gateway host environment at launch, keeping secrets (e.g. inference API keys) out of the config file. Wire it into the agent process so gateway-launched agents can authenticate to cloud endpoints (process.env was previously hardcoded empty). Unit-tested via resolve_agent_env_passthrough_and_host_lookup. Also add a gateway-driven cloud-inference (T1) test harness: mxc-inference.toml (agent_env + curl agent), inference.yaml policy, and run-inference-test.ps1 which starts the gateway, creates an isolation_session sandbox, runs an authenticated Nemotron call, and bundles redacted results. Documented agent_env in mxc-gateway.toml. Validated end-to-end on the test box (chat HTTP 200 + completion via the gateway). (cherry picked from commit 94d9e827b8b77e0af9dc654943ea8e7d8400cc9f) Signed-off-by: Jamie King <jamiek@nvidia.com> * test(driver-mxc): avoid unsafe env mutation in resolve_agent_env test Replace std::env::{set,remove}_var (unsafe + racy under parallel test execution in edition 2024) with a read-only PATH lookup. Preserves all three behaviors under test and drops the #[allow(unsafe_code)]. (cherry picked from commit ac5766eeab2db4e8cc6fcd8d8a97809edaf3df30) Signed-off-by: Jamie King <jamiek@nvidia.com> * fix(mxc): adapt MXC driver to current GitHub OpenShell API The MXC driver crate was authored on GitLab against an earlier proto/core API. Adapt it to the API on GitHub main: - build_capabilities_response no longer takes supports_interactive_session - DriverSandboxSpec.gpu (bool) is now resource_requirements; detect GPU via effective_driver_gpu_count(driver_gpu_requirements(..)) - DriverSandbox gained a `workspace` field - SandboxPolicy gained `network_middlewares`: pass it through the proxy split, emit a loss item on the coarse MXC path, and account for it in the mapper drift-guard test Verified: cargo check + 75 mock-based tests pass (lib 27, examples 10, policy_mapper_matrix 38). Signed-off-by: Jamie King <jamiek@nvidia.com> * feat(server): wire the MXC compute driver into the gateway on Windows Register openshell-driver-mxc as the Windows-only in-process compute backend so compute_driver = "mxc" resolves to a working runtime: - ComputeRuntime::new_mxc, adapted to the current 11-arg from_driver - mxc_policy_sink A1 side channel, staged in create_sandbox before dispatch - mxc_config_from_context loader and the Mxc dispatch arm (Windows constructs; other targets return an explicit "Windows-only" error) - Windows-gated openshell-driver-mxc dependency - Mxc arms for the telemetry, config-file required-fields, and CLI reserved-builtin matches to keep them exhaustive/correct Verified with cargo check --workspace --features openshell-prover/bundled-z3 on x86_64-pc-windows-msvc, stacked on PR #2496. Signed-off-by: Jamie King <jamiek@nvidia.com> * fix(driver-mxc): implement GetGatewayListenerRequirements for #2496 base Signed-off-by: Jamie King <jamiek@nvidia.com> * test(driver-mxc): add probe-gated real wxc-exec test lane (no mocks) - tests/wxc_exec_real.rs: ignored-by-default integration tests against a real wxc-exec. Six --dry-run contract tests run wherever the binary exists (they caught the network.proxy shape mismatch); enforcement tests (processcontainer default-deny positive/negative, isolation session lifecycle round trip with a deprovision drop-guard) probe the backend and SKIP with a recorded reason where it is not live. - examples/probe-mxc-host.ps1: classifies a host (OS build, --probe, per-backend trial) and emits a JSON capability verdict. - examples/run-mxc-e2e.ps1 + e2e-policies/: scenario runner generalizing run-demo.ps1 (fs-rw, fs-readonly, fs-default-deny-empty, network-policy-rejected) with PASS/FAIL/SKIP gating and a stale OPENSHELL_MXC_MOCK_WXC guard in real mode. - tasks/windows.toml: windows:test:mxc-real:x64, windows:e2e:mxc, windows:e2e:mxc:mock. Signed-off-by: Giedrius Burachas <gburachas@nvidia.com> (cherry picked from commit 49afafe892caded59c4df50a9b652011ad97f41c) Signed-off-by: Jamie King <jamiek@nvidia.com> * fix(driver-mxc): address PR review feedback Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs: defer public MXC documentation Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * fix(driver-mxc): build Windows capabilities response Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * fix(server): gate in-tree tracing on Windows Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * test(windows): fix cross-platform test assumptions Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * test(windows): verify process and PEM portably Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * test(windows): keep lifecycle command in policy Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> --------- Signed-off-by: Jamie King <jamiek@nvidia.com> Signed-off-by: Giedrius Burachas <gburachas@nvidia.com> Signed-off-by: Shailendra Singh <shailendras@nvidia.com> Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> Co-authored-by: Prashant K <pkhodade@nvidia.com> Co-authored-by: Giedrius Burachas <gburachas@nvidia.com> Co-authored-by: Shailendra Singh <shailendras@nvidia.com> Co-authored-by: Drew Newberry <385+drew@users.noreply.github.com> |