mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-20 06:00:06 +08:00
docker-gpu-support
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Adam Miller
|
d44d8a1e27 |
feat: Openshell driver podman (#904)
* feat(podman): add Podman compute driver for rootless sandbox management Adds openshell-driver-podman, a new compute driver that manages OpenShell sandboxes as rootless Podman containers via the Podman REST API over a Unix socket. Enables local workstation sandboxes without Kubernetes. Driver features: - Bridge networking with ephemeral host-port mapping for rootless SSH reachability - Named volumes for workspace storage, Podman native health checks, GPU via CDI - Supervisor binary sideloaded via image volume mount (BYOC-compatible) - SSH handshake secret injected via Podman secrets API (not plaintext env) - Typed ContainerSpec structs, input validation, and path-traversal guards - Cgroups v2 required; fails fast on v1 hosts - Bounded event stream buffer; watch stream reconnection handled by server watch_loop - Graceful shutdown and standalone driver binary with gRPC bridge Rootless-specific fixes: - Skip drop_privileges when user namespace lacks SETUID/SETGID/DAC_READ_SEARCH caps - Add /run/netns tmpfs mount for ip netns in rootless containers - Use secret_env map (not secrets array) for env-var injection in libpod API - Resolve SSH endpoint to 127.0.0.1:<host_port> instead of unreachable bridge IP Server/sandbox hardening: - Split loopback and link-local SSRF gates; Podman/VM drivers allow loopback - Close SSRF bypass in SSH tunnel Host path by resolving DNS before connecting - Prevent OPENSHELL_* env var override by user-supplied spec environment maps - Disable SQLite pool idle_timeout/max_lifetime for in-memory databases - Emit deleted_event on 404-during-inspect instead of regressing sandbox phase - Key delete cleanup by stable sandbox_id to survive container label drift CLI fixes: - Restore --name as a named flag on sandbox create (not positional) - Fix exec command arg parsing to not consume sandboxed-command flags - Propagate SSH verbosity via OPENSHELL_SSH_LOG_LEVEL Build tooling: - Add tasks/scripts/container-engine.sh: auto-detects Podman or Docker, exposes unified ce_* helpers; all build/cluster/VM scripts updated to use it - Add docker:build:supervisor mise task for standalone supervisor image - Add openshell-driver-podman to Dockerfile.images pre-fetch/build stages - Add e2e/rust/e2e-podman.sh and e2e:podman mise task for full lifecycle testing Signed-off-by: Adam Miller <admiller@redhat.com> * fix(driver-podman): derive grpc endpoint from server bind port When a user starts the gateway on a non-default port (e.g. --port 8081), sandbox containers were receiving OPENSHELL_ENDPOINT pointing at the default port 8080. The driver's auto-detection fallback read OPENSHELL_BIND_ADDRESS from the environment, which was stale or unset, and fell back to DEFAULT_SERVER_PORT. Add gateway_port to PodmanComputeConfig and thread config.bind_address.port() from the server into the driver so the fallback uses the actual listening port. Remove the OPENSHELL_BIND_ADDRESS env var read and the extract_port_from_bind_address helper which are no longer needed. Add --gateway-port / OPENSHELL_GATEWAY_PORT to the standalone driver binary for parity when the driver is run outside the embedded server path. Signed-off-by: Adam Miller <admiller@redhat.com> * fix(driver-podman): address PR feedback on env test safety and cluster DNS docs Replace hand-rolled unsafe TempEnvVar RAII guard with temp_env::with_vars and a static ENV_LOCK mutex, fixing a data race in parallel test execution. The prior safety comment incorrectly claimed Cargo runs tests single-threaded. Update debug-openshell-cluster skill to accurately document the DNS proxy strategy (setup_dns_proxy + public DNS fallback) and clarify the separation between cluster DNS and sandbox agent DNS enforcement. Signed-off-by: Adam Miller <admiller@redhat.com> * fix(e2e): resolve CI failures in auth timeout, test harness, and formatting - Short-circuit browser_auth_flow when OPENSHELL_NO_BROWSER=1 instead of waiting the full 120s AUTH_TIMEOUT for a callback that never arrives - Add timeout to SandboxGuard::create() and create_with_upload() to prevent indefinite hangs (matches create_keep() which already had one) - Add missing '--' separator in no_proxy test before command args - Add #![cfg(feature = "e2e")] gate to sandbox_lifecycle.rs - Run cargo fmt on openshell-driver-podman - Refine cluster DNS docs for Podman in debug-openshell-cluster skill Signed-off-by: Adam Miller <admiller@redhat.com> * refactor(server): remove allows_loopback_endpoints from ComputeRuntime SSRF protection is now handled at the network and proxy layers (openshell-core net.rs, openshell-sandbox proxy.rs) rather than requiring per-driver flags on ComputeRuntime. Update architecture docs to reflect supervisor relay SSH transport and add rootless networking deep-dive. Signed-off-by: Adam Miller <admiller@redhat.com> --------- Signed-off-by: Adam Miller <admiller@redhat.com> |
||
|
Piotr Mlocek
|
a6d45528c1 | feat(server,sandbox): supervisor-initiated SSH connect and exec over gRPC-multiplexed relay (#867) | ||
|
John T. Myers
|
28db08eeb8 |
fix(sandbox): disable child core dumps (#821)
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
John T. Myers
|
29a3b1cacd |
fix(sandbox): two-phase Landlock to fix privilege ordering and add enforcement tests (#810)
* fix(sandbox): add parent-side Landlock availability probe logging Landlock status was only logged from inside the pre_exec child process where the tracing/OCSF pipeline is non-functional after fork. This made Landlock failures completely invisible in sandbox logs. Add a probe_availability() function that issues the raw landlock_create_ruleset syscall to check kernel support, and call it from the parent process before fork in all three spawn paths (entrypoint, SSH PTY, SSH pipe). Uses std::sync::Once to emit exactly once per sandbox lifetime. WIP - addresses logging gap from #803. * test(e2e): add Landlock filesystem enforcement tests Verify Landlock availability logging and enforcement in e2e: - OCSF probe event appears in sandbox logs - Read-only paths block writes, allow reads - Read-write paths allow both - Paths outside policy are denied entirely - User-owned paths outside policy are still blocked (proves Landlock enforces independently of Unix DAC permissions) Requires Linux host with Landlock support (GitHub Actions runners, Docker Desktop linuxkit). Related to #803. * test(e2e): add xfail tests proving #803 privilege ordering bug Two strict xfail tests that demonstrate the root cause of #803: - PathFd::new() runs as uid 998 after drop_privileges, so root-only paths (mode 700) silently fail and Landlock degrades - When ALL paths fail, best_effort silently drops Landlock entirely These tests will pass after the two-phase Landlock fix (open PathFds as root before drop_privileges, restrict_self after). * fix(test): fix Landlock e2e tests based on test run results - Remove log-reading tests: the Landlock probe logs to the supervisor's container stdout, not the in-sandbox file appender at /var/log/openshell*.log* - Replace broken xfail test (checked for child-process log messages that never reach the file appender) with a stat()-based test that verifies /root is actually in the Landlock allowlist - Keep enforcement tests (all passing) and the root-only policy xfail test (correctly proves #803 bug) * fix(test): remove mixed-policy xfail test Landlock doesn't restrict stat(), so the test passed unexpectedly. In the mixed policy case where /root is silently skipped, Landlock still applies (using other paths) and blocks /root even harder (not in allowlist = denied). The observable security degradation only occurs when ALL paths fail, which the existing xfail test already covers. * fix(sandbox): two-phase Landlock to fix privilege ordering (#803) Split Landlock apply into prepare() and enforce(): - prepare() runs as root before drop_privileges: opens PathFds, creates ruleset, adds rules. Root-only paths (mode 700) now succeed instead of silently failing as uid 998. - enforce() runs after drop_privileges: calls restrict_self() which does not require root. This fixes the root cause of #803 where drop_privileges() ran before sandbox::apply(), causing PathFd::new() to fail on root-only paths. In best_effort mode this silently dropped all Landlock restrictions. The fix applies to all three spawn paths: entrypoint (process.rs), SSH PTY (ssh.rs), and SSH pipe exec (ssh.rs). Removes xfail marker from e2e test that now passes. * fix(sandbox): address PR review feedback - enforce() now respects best_effort: if restrict_self() fails and policy is best_effort, log and degrade instead of aborting startup - log_sandbox_readiness distinguishes best_effort (degraded) from hard_requirement (will fail) in OCSF messages |
||
|
John T. Myers
|
b7779bdefa |
feat(sandbox): integrate OCSF structured logging for sandbox events (#720)
* feat(sandbox): integrate OCSF structured logging for all sandbox events WIP: Replace ad-hoc tracing calls with OCSF event builders across all sandbox subsystems (network, SSH, process, filesystem, config, lifecycle). - Register ocsf_logging_enabled setting (defaults false) - Replace stdout/file fmt layers with OcsfShorthandLayer - Add conditional OcsfJsonlLayer for /var/log/openshell-ocsf.log - Update LogPushLayer to extract OCSF shorthand for gRPC push - Migrate ~106 log sites to OCSF builders (NetworkActivity, HttpActivity, SshActivity, ProcessActivity, DetectionFinding, ConfigStateChange, AppLifecycle) - Add openshell-ocsf to all Docker build contexts * fix(scripts): attach provider to all smoke test phases to avoid rate limits GitHub's unauthenticated API rate limit (60/hour) causes flaky 403s for Phases 1, 2, and 4. Fix by attaching the provider to all sandboxes and upgrading the Phase 1 policy to L7 so credential injection works. Phase 4 (tls:skip) cannot inject credentials by design, so relax the assertion to accept either 200 or 403 from upstream -- both prove the proxy forwarded the request. * fix(ocsf): remove timestamp from shorthand format to avoid double-timestamp The display layer (gateway logs, TUI, sandbox logs CLI) already prepends a timestamp. Having one in the shorthand output too produces redundant double-timestamps like: 15:49:11 sandbox INFO 15:49:11.649 I NET:OPEN ALLOWED ... Now the shorthand is just the severity + structured content: 15:49:11 sandbox INFO I NET:OPEN ALLOWED ... * refactor(ocsf): replace single-char severity with bracketed labels Replace cryptic single-character severity codes (I/L/M/H/C/F) with readable bracketed labels: [LOW], [MED], [HIGH], [CRIT], [FATAL]. Informational severity (the happy-path default) is omitted entirely to keep normal log output clean and avoid redundancy with the tracing-level INFO that the display layer already provides. Before: sandbox INFO I NET:OPEN ALLOWED ... After: sandbox INFO NET:OPEN ALLOWED ... Before: sandbox INFO M NET:OPEN DENIED ... After: sandbox INFO [MED] NET:OPEN DENIED ... * feat(sandbox): use OCSF level label for structured events in log push Set the level field to 'OCSF' instead of 'INFO' for OCSF events in the gRPC log push. This visually distinguishes structured OCSF events from plain tracing output in the TUI and CLI sandbox logs: sandbox OCSF NET:OPEN [INFO] ALLOWED python3(42) -> api.example.com:443 sandbox OCSF NET:OPEN [MED] DENIED python3(42) -> blocked.com:443 sandbox INFO Fetching sandbox policy via gRPC * fix(sandbox): convert new Landlock path-skip warning to OCSF PR #677 added a warn!() for inaccessible Landlock paths in best-effort mode. Convert to ConfigStateChangeBuilder with degraded state so it flows through the OCSF shorthand format consistently. * fix(sandbox): use rolling appender for OCSF JSONL file Match the main openshell.log rotation mechanics (daily, 3 files max) instead of a single unbounded append-only file. Prevents disk exhaustion when ocsf_logging_enabled is left on in long-running sandboxes. * fix(sandbox): address reviewer warnings for OCSF integration W1: Remove redundant 'OCSF' prefix from shorthand file layer — the class name (NET:OPEN, HTTP:GET) already identifies structured events and the LogPushLayer separately sets the level field. W2: Log a debug message when OCSF_CTX.set() is called a second time instead of silently discarding via let _. W3: Document the boundary between OCSF-migrated events and intentionally plain tracing calls (DEBUG/TRACE, transient, internal plumbing). W4: Migrate remaining iptables LOG rule failure warnings in netns.rs (IPv4 TCP/UDP, IPv6 TCP/UDP) to ConfigStateChangeBuilder for consistency with the IPv4 bypass rule failure already migrated. W5: Migrate malformed inference request warn to NetworkActivity with ActivityId::Refuse and SeverityId::Medium. W6: Use Medium severity for L7 deny decisions (both CONNECT tunnel and FORWARD proxy paths) to match the CONNECT deny severity pattern. Allows and audits remain Informational. * refactor(sandbox): rename ocsf_logging_enabled to ocsf_json_enabled The shorthand logs are already OCSF-structured events. The setting specifically controls the JSONL file export, so the name should reflect that: ocsf_json_enabled. * fix(ocsf): add timestamps to shorthand file layer output The OcsfShorthandLayer writes directly to the log file with no outer display layer to supply timestamps. Add a UTC timestamp prefix to every line so the file output matches what tracing::fmt used to provide. Before: CONFIG:VALIDATED [INFO] Validated 'sandbox' user exists in image After: 2026-04-01T15:49:11.649Z CONFIG:VALIDATED [INFO] Validated ... * fix(docker): touch openshell-ocsf source to invalidate cargo cache The supervisor-workspace stage touches sandbox and core sources to force recompilation over the rust-deps dummy stubs, but openshell-ocsf was missing. This caused the Docker cargo cache to use stale ocsf objects from the deps stage, preventing changes to the ocsf crate (like the timestamp fix) from appearing in the final binary. Also adds a shorthand layer test verifying timestamp output, and drafts the observability docs section. * fix(ocsf): add OCSF level prefix to file layer shorthand output Without a level prefix, OCSF events in the log file have no visual anchor at the position where standard tracing lines show INFO/WARN. This makes scanning the file harder since the eye has nothing consistent to lock onto after the timestamp. Before: 2026-04-01T04:04:13.065Z CONFIG:DISCOVERY [INFO] ... After: 2026-04-01T04:04:13.065Z OCSF CONFIG:DISCOVERY [INFO] ... * fix(ocsf): clean up shorthand formatting for listen and SSH events - Fix double space in NET:LISTEN, SSH:LISTEN, and other events where action is empty (e.g., 'NET:LISTEN [INFO] 10.200.0.1' -> 'NET:LISTEN [INFO] 10.200.0.1') - Add listen address to SSH:LISTEN event (was empty) - Downgrade SSH handshake intermediate steps (reading preface, verifying) from OCSF events to debug!() traces. Only the final verdict (accepted/denied) is an OCSF event now, reducing noise from 3 events to 1 per SSH connection. - Apply same spacing fix to HTTP shorthand for consistency. * docs(observability): update examples with OCSF prefix and formatting fixes Align doc examples with the deployed output: - Add OCSF level prefix to all shorthand examples in the log file - Show mixed OCSF + standard tracing in the file format section - Update listen events (no double space, SSH includes address) - Show one SSH:OPEN per connection instead of three - Update grep patterns to use 'OCSF NET:' etc. * docs(agents): add OCSF logging guidance to AGENTS.md Add a Sandbox Logging (OCSF) section to AGENTS.md so agents have in-context guidance for deciding whether new log emissions should use OCSF structured logging or plain tracing. Covers event class selection, severity guidelines, builder API usage, dual-emit pattern for security findings, and the no-secrets rule. Also adds openshell-ocsf to the Architecture Overview table. * fix: remove workflow files accidentally included during rebase These files were already merged to main in separate PRs. They got pulled into our branch during rebase conflict resolution for the deleted docs-preview-pr.yml file. * docs(observability): use sandbox connect instead of raw SSH Users access sandboxes via 'openshell sandbox connect', not direct SSH. * fix(docs): correct settings CLI syntax in OCSF JSON export page The settings CLI requires --key and --value named flags, not positional arguments. Also fix the per-sandbox form: the sandbox name is a positional argument, not a --sandbox flag. * fix(e2e): update log assertions for OCSF shorthand format The E2E tests asserted on the old tracing::fmt key=value format (action=allow, l7_decision=audit, FORWARD, L7_REQUEST, always-blocked). Update to match the new OCSF shorthand (ALLOWED/DENIED, HTTP:, NET:, engine:ssrf, policy:). * feat(sandbox): convert WebSocket upgrade log calls to OCSF PR #718 added two log calls for WebSocket upgrade handling: - 101 Switching Protocols info → NetworkActivity with Upgrade activity. This is a significant state change (L7 enforcement drops to raw relay). - Unsolicited 101 without client Upgrade header → DetectionFinding with High severity. A non-compliant upstream sending 101 without a client Upgrade request could be attempting to bypass L7 inspection. |
||
|
John T. Myers
|
ed74a19a61 |
fix(sandbox): track PTY state per SSH channel to fix terminal resize (#687)
Replace flat pty_master/input_sender/pty_request fields in SshHandler with a HashMap<ChannelId, ChannelState> so each channel tracks its own PTY resources independently. This fixes window_change_request resizing the wrong PTY when multiple channels are open simultaneously. Also fixes ioctl UB in set_winsize (pass &winsize not winsize by value) and adds warn! logging for unknown channels across all handlers. Resolves #543 |
||
|
John T. Myers
|
834f8aa184 |
fix: security hardening batch 1 (SEC-002 through SEC-010) (#548)
* fix(l7): reject ambiguous HTTP framing in REST proxy (SEC-009) Harden the L7 REST proxy HTTP parser against request smuggling: - Reject requests containing both Content-Length and Transfer-Encoding headers per RFC 7230 Section 3.3.3 (CL/TE ambiguity) - Replace String::from_utf8_lossy with strict UTF-8 validation to prevent interpretation gaps with upstream servers - Reject bare LF line endings (require CRLF per HTTP spec) - Validate HTTP version string (HTTP/1.0 or HTTP/1.1 only) * fix(server): harden shell_escape and command construction (SEC-002) Harden the gRPC exec handler against command injection via the structured-to-shell-string conversion: - Reject null bytes and newlines/carriage returns in shell_escape() - Add input validation in exec_sandbox: reject control characters in command args, env values, and workdir - Enforce size limits: max 1024 args, 32 KiB per arg/value, 4 KiB workdir, 256 KiB total assembled command string - Change shell_escape and build_remote_exec_command to return Result so callers must handle validation failures * fix(server): add command validation at SSH transport boundary (SEC-003) Add defense-in-depth validation in run_exec_with_russh before sending the command to the sandbox SSH server: - Reject null bytes in command string at transport boundary - Enforce max command length (256 KiB) at transport boundary - Enhance stream_exec_over_ssh logging with command length, stdin length, and truncated command preview for audit trail * fix(sandbox): validate port range and extend loopback check in SSH (SEC-007) Harden the SSH direct-tcpip channel handler: - Validate port_to_connect <= 65535 before u32-to-u16 cast to prevent port truncation (e.g., 65537 becoming port 1) - Replace string-literal loopback check with is_loopback_host() that covers the full 127.0.0.0/8 range, IPv4-mapped IPv6 (::ffff:127.x), bracketed IPv6, and case-insensitive localhost - Remove #[allow(clippy::cast_possible_truncation)] since the cast is now proven safe by the preceding range check * fix(sandbox): sanitize inference error messages returned to sandbox (SEC-008) Replace verbatim internal error strings in router_error_to_http with generic messages to prevent information leakage to sandboxed code. Upstream URLs, internal hostnames, TLS details, and file paths are no longer exposed. Full error context is still logged server-side at warn level by the caller for debugging. * fix(server): block internal IPs in SSH proxy target validation (SEC-006) Add IP validation in start_single_use_ssh_proxy to prevent SSRF if a sandbox status record were poisoned: - Resolve DNS before connecting and validate the resolved IP - Block loopback (127.0.0.0/8) and link-local (169.254.0.0/16, covers cloud metadata endpoint) addresses - Block IPv4-mapped IPv6 variants of the same ranges - Connect to the validated SocketAddr directly to prevent TOCTOU - Add debug logging of resolved target IP for audit * fix(sandbox): add resource limits to chunked body parser (SEC-010) Harden parse_chunked_body in the inference interception path: - Replace all unchecked +2 additions with checked_add for consistent overflow safety across all target architectures - Add MAX_CHUNKED_BODY (10 MiB) to cap decoded body size - Add MAX_CHUNK_COUNT (4096) to prevent CPU exhaustion via tiny chunks - Early-reject chunk sizes larger than remaining buffer space * fix(cli): double-escape command for SSH path, validate host and name (SEC-004) Harden doctor_exec against command injection in the SSH remote path: - Apply shell_escape to inner_cmd in the SSH path so it survives the double shell interpretation (SSH remote shell + sh -lc). This also fixes a correctness bug where multi-word commands were silently broken in the SSH path. - Add validate_gateway_name to reject shell metacharacters in gateway names before use in container_name - Add validate_ssh_host to reject metacharacters in remote_host loaded from metadata.json * fix(sandbox): add CIDR breadth warning and control-plane port blocklist (SEC-005) Defense-in-depth for the allowed_ips feature: - Log a warning when a CIDR entry has a prefix length < /16, as overly broad ranges may unintentionally expose control-plane services - Block K8s API (6443), etcd (2379/2380), and kubelet (10250/10255) ports unconditionally in resolve_and_check_allowed_ips, even when the resolved IP matches an allowed_ips entry * test(e2e): update assertion for sanitized inference error message (SEC-008) The SEC-008 fix changed the error message from 'no compatible route for source protocol ...' to 'no compatible inference route available'. Update the E2E assertion substring to match. |
||
|
Drew Newberry
|
f6ae1da12d | chore: remove remaining navigator and nemoclaw references (#279) | ||
|
Drew Newberry
|
fbd93a4632 | refactor: rename navigator- crate prefix to openshell- (#277) |