Change sandbox user UID from 1000 to 1000660000 in custom image examples and E2E tests.
Using a high UID range (1000000000+) prevents conflicts with host users when running
without user namespace remapping, where container UIDs map directly to host UIDs.
This resolves fork failures caused by RLIMIT_NPROC enforcement when the host user
already has many threads running.
Signed-off-by: Derek Carr <decarr@redhat.com>
The docker-cleanup.sh script failed when no containers were running because
grep -v returned exit code 1 on empty input, causing the script to abort
due to set -euo pipefail.
Add || true to the volume detection pipeline so the script succeeds when
there are no running containers (in_use_volumes will be empty, which is
the correct behavior).
Signed-off-by: Derek Carr <decarr@redhat.com>
#966 hard-coded `buildkitd-config: /etc/buildkit/buildkitd.toml` inside
the `driver: local` branch of the setup-buildx composite action. The only
caller using that driver is shadow-docker-build.yml, which runs inside
the ghcr.io/nvidia/openshell/ci:latest container — so the host-side
buildkitd.toml was invisible to docker/setup-buildx-action and every
matrix job failed at "Set up buildx".
Revert the hard-coded path and expose it as an opt-in input on the
action (empty default, passed through to both the remote and local
branches). Wire shadow-docker-build.yml to bind-mount /etc/buildkit
into the ci container and pass the path explicitly, so the action can
read the file from inside the container. Remote-driver callers are
unaffected (empty input is a no-op).
Signed-off-by: Jonas Toelke <jtoelke@nvidia.com>
* feat(docker): add BINARY_SOURCE selector for prebuilt Rust binaries
Signed-off-by: Jonas Toelke <jtoelke@nvidia.com>
* fix(docker): preserve exec bit on prebuilt binary COPY
Adds --chmod=755 to the COPY instructions in the scratch-based
prebuilt binary stages. Without this, binaries produced by PR 4a and
shuttled through actions/upload-artifact + download-artifact lose
their executable bit during the roundtrip, and the resulting image's
ENTRYPOINT fails at runtime.
Signed-off-by: Jonas Toelke <jtoelke@nvidia.com>
---------
Signed-off-by: Jonas Toelke <jtoelke@nvidia.com>
* feat(podman): add Podman compute driver for rootless sandbox management
Adds openshell-driver-podman, a new compute driver that manages OpenShell
sandboxes as rootless Podman containers via the Podman REST API over a
Unix socket. Enables local workstation sandboxes without Kubernetes.
Driver features:
- Bridge networking with ephemeral host-port mapping for rootless SSH reachability
- Named volumes for workspace storage, Podman native health checks, GPU via CDI
- Supervisor binary sideloaded via image volume mount (BYOC-compatible)
- SSH handshake secret injected via Podman secrets API (not plaintext env)
- Typed ContainerSpec structs, input validation, and path-traversal guards
- Cgroups v2 required; fails fast on v1 hosts
- Bounded event stream buffer; watch stream reconnection handled by server watch_loop
- Graceful shutdown and standalone driver binary with gRPC bridge
Rootless-specific fixes:
- Skip drop_privileges when user namespace lacks SETUID/SETGID/DAC_READ_SEARCH caps
- Add /run/netns tmpfs mount for ip netns in rootless containers
- Use secret_env map (not secrets array) for env-var injection in libpod API
- Resolve SSH endpoint to 127.0.0.1:<host_port> instead of unreachable bridge IP
Server/sandbox hardening:
- Split loopback and link-local SSRF gates; Podman/VM drivers allow loopback
- Close SSRF bypass in SSH tunnel Host path by resolving DNS before connecting
- Prevent OPENSHELL_* env var override by user-supplied spec environment maps
- Disable SQLite pool idle_timeout/max_lifetime for in-memory databases
- Emit deleted_event on 404-during-inspect instead of regressing sandbox phase
- Key delete cleanup by stable sandbox_id to survive container label drift
CLI fixes:
- Restore --name as a named flag on sandbox create (not positional)
- Fix exec command arg parsing to not consume sandboxed-command flags
- Propagate SSH verbosity via OPENSHELL_SSH_LOG_LEVEL
Build tooling:
- Add tasks/scripts/container-engine.sh: auto-detects Podman or Docker, exposes
unified ce_* helpers; all build/cluster/VM scripts updated to use it
- Add docker:build:supervisor mise task for standalone supervisor image
- Add openshell-driver-podman to Dockerfile.images pre-fetch/build stages
- Add e2e/rust/e2e-podman.sh and e2e:podman mise task for full lifecycle testing
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): derive grpc endpoint from server bind port
When a user starts the gateway on a non-default port (e.g. --port 8081),
sandbox containers were receiving OPENSHELL_ENDPOINT pointing at the
default port 8080. The driver's auto-detection fallback read
OPENSHELL_BIND_ADDRESS from the environment, which was stale or unset,
and fell back to DEFAULT_SERVER_PORT.
Add gateway_port to PodmanComputeConfig and thread config.bind_address.port()
from the server into the driver so the fallback uses the actual listening
port. Remove the OPENSHELL_BIND_ADDRESS env var read and the
extract_port_from_bind_address helper which are no longer needed.
Add --gateway-port / OPENSHELL_GATEWAY_PORT to the standalone driver
binary for parity when the driver is run outside the embedded server path.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): address PR feedback on env test safety and cluster DNS docs
Replace hand-rolled unsafe TempEnvVar RAII guard with temp_env::with_vars
and a static ENV_LOCK mutex, fixing a data race in parallel test execution.
The prior safety comment incorrectly claimed Cargo runs tests single-threaded.
Update debug-openshell-cluster skill to accurately document the DNS proxy
strategy (setup_dns_proxy + public DNS fallback) and clarify the separation
between cluster DNS and sandbox agent DNS enforcement.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(e2e): resolve CI failures in auth timeout, test harness, and formatting
- Short-circuit browser_auth_flow when OPENSHELL_NO_BROWSER=1 instead
of waiting the full 120s AUTH_TIMEOUT for a callback that never arrives
- Add timeout to SandboxGuard::create() and create_with_upload() to
prevent indefinite hangs (matches create_keep() which already had one)
- Add missing '--' separator in no_proxy test before command args
- Add #![cfg(feature = "e2e")] gate to sandbox_lifecycle.rs
- Run cargo fmt on openshell-driver-podman
- Refine cluster DNS docs for Podman in debug-openshell-cluster skill
Signed-off-by: Adam Miller <admiller@redhat.com>
* refactor(server): remove allows_loopback_endpoints from ComputeRuntime
SSRF protection is now handled at the network and proxy layers
(openshell-core net.rs, openshell-sandbox proxy.rs) rather than
requiring per-driver flags on ComputeRuntime. Update architecture
docs to reflect supervisor relay SSH transport and add rootless
networking deep-dive.
Signed-off-by: Adam Miller <admiller@redhat.com>
---------
Signed-off-by: Adam Miller <admiller@redhat.com>
Without a per-arch SCCACHE_GHA_VERSION, both matrix jobs
(linux-amd64-cpu8 and linux-arm64-cpu8) target the same GHA cache
key. Run 1 on 2026-04-24 showed this concretely: arm64 started
first and landed 575/1064 cache writes; amd64 started ~1 min later
and every one of its 1062 writes failed, leaving the cache with no
amd64 entries at all.
Partitioning by matrix.runner ensures each arch gets its own
cache namespace, letting us see real warm-cache behavior on the
subsequent Run 2-4 dispatches.
Signed-off-by: Jonas Toelke <jtoelke@nvidia.com>
Python runtime requires /dev/urandom access during initialization
to seed the hash randomizer. The _default_policy() in provider tests
was missing this path, causing exec_python tests to fail with:
'Fatal Python error: _Py_HashRandomization_Init: failed to get
random numbers to initialize Python'.
Add /dev/urandom to read_only paths to match the policy used in
test_sandbox_policy.py, allowing Python to initialize successfully.
Signed-off-by: Derek Carr <decarr@redhat.com>
Previously `openshell sandbox upload <local-dir> <remote-dir>` flattened
the local directory's contents into the remote destination, dropping the
source directory's basename. This diverged from the standard `scp -r` /
`cp -r` semantics that operators expect from any *nix-style transfer
tool, and could silently shadow files at the destination.
Now an upload of `foo` to `/sandbox/dest/` lands at `/sandbox/dest/foo/`,
matching `scp -r foo remote:/sandbox/dest/` and `cp -r foo /sandbox/dest/`.
Paths without a meaningful basename (`.`, `/`) preserve the legacy flat
extraction.
Closes#885
* feat(server): add Prometheus metrics infrastructure and gRPC/HTTP request metrics
Add metrics exposition via a dedicated server port (--metrics-port,
default disabled) following the same optional-listener pattern as the
health port. The metrics crate facade records counters and histograms
in the MultiplexedService hot path, and a PrometheusHandle renders them
at GET /metrics on the dedicated port.
Metrics added:
- openshell_server_grpc_requests_total (counter: method, code)
- openshell_server_grpc_request_duration_seconds (histogram: method, code)
- openshell_server_http_requests_total (counter: path, status)
- openshell_server_http_request_duration_seconds (histogram: path, status)
* feat(helm): add metrics port to openshell-server helm chart
Expose the Prometheus metrics endpoint in the helm chart by adding
service.metricsPort (default 9090) to values, the --metrics-port arg
and container port to the statefulset, and a metrics service port.
Set metricsPort to 0 to disable.
Most contributors cannot assign PRs or issues to themselves due to
repository settings. Remove the --assignee flag from skill templates
to prevent gh CLI errors.
Clears four High-severity container findings in the CI image, which
installs helm via mise.toml:
- CVE-2026-35204 / GHSA-vmx8-mqv2-9gmg (plugin path traversal)
- CVE-2026-35205 / GHSA-q5jf-9vfq-h4h7 (plugin .prov verification fails open)
Both are fixed in helm 4.1.4.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Ubuntu Noble's git links against libcurl-gnutls, which does not read
SSL_CERT_FILE. Without GIT_SSL_CAINFO, `git clone` over HTTPS fails
inside every community sandbox image with "server certificate
verification failed".
Add GIT_SSL_CAINFO to tls_env_vars pointing at the same combined CA
bundle already used by CURL_CA_BUNDLE / REQUESTS_CA_BUNDLE.
Fixes#790FixesNVIDIA/NemoClaw#2270FixesNVIDIA/NemoClaw#1828
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Closes#879
Resolve policy hostnames against the sandbox's /etc/hosts before DNS so Kubernetes hostAliases are visible to the proxy while keeping the existing allowed_ips SSRF enforcement semantics intact.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Closes#905
Skip proxy baseline read-write enrichment when a path is already explicitly configured as read-only, add regression coverage for the proto and local policy code paths, and document the precedence rule in the sandbox architecture docs.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
the transfer of very large sandbox images to containerd can
timeout depending on the size of the image and the speed of
the local host.
Made-with: Cursor
The 30s read_timeout on the shared kube client was killing the
long-lived watch streams during idle periods, causing a reconnect
cycle every 30 seconds. Use a separate client with no read_timeout
for watch_sandboxes so the streams stay open indefinitely.
Move /health, /healthz, and /readyz to a dedicated plaintext HTTP port
(default 8081) so Kubernetes probes work without mTLS client certificates.
- Add health_bind_address to Config with --health-port CLI arg
- Spawn standalone axum::serve for health_router on the health port
- Remove health routes from the main multiplexed HTTP router
- Update Helm statefulset probes from tcpSocket to httpGet on health port
- Fix cluster-healthcheck.sh to open/close TCP without sending data,
avoiding InvalidContentType TLS errors in the gateway log
Wrap gRPC and HTTP services with TraceLayer to log method, path,
status, and latency for every request. Health/readiness probe
endpoints are logged at debug level to reduce noise.
* fix(sandbox): canonicalize HTTP request-targets before L7 policy evaluation
The L7 REST proxy evaluated OPA path rules against the raw request-target
and forwarded the raw header block byte-for-byte to the upstream. The
upstream normalized `..`, `%2e%2e`, and `//` before dispatching, so a
compromised agent inside the sandbox could bypass any path-based allow
rule by prefixing the blocked path with an allowed one (e.g.
`GET /public/../secret` matches `/public/**` at the proxy but the
upstream serves `/secret`). The inference-routing detection had the same
normalization gap via `normalize_inference_path`, which only stripped
scheme+authority and preserved dot-segments verbatim.
- Add `crates/openshell-sandbox/src/l7/path.rs` with
`canonicalize_request_target`. Percent-decodes (with uppercase hex
re-emission), resolves `.` / `..` segments per RFC 3986 5.2.4,
collapses doubled slashes, strips trailing `;params`, rejects
fragments / raw non-ASCII / control bytes / encoded slashes (unless
explicitly enabled per-endpoint), and returns a `rewritten` flag so
callers know when to rewrite the outbound request line.
- Wire the canonicalizer into `rest.rs::parse_http_request`. The
canonical path is the `target` on `L7Request` that OPA sees, and
when the canonical form differs from the raw input the request line
is rebuilt in `raw_header` so the upstream dispatches on the exact
bytes the policy evaluated.
- Canonicalize the forward (plain HTTP proxy) path at
`proxy.rs`'s second L7 evaluation site. A non-canonical request-target
is rejected with 400 Bad Request and an OCSF `NetworkActivity` Fail
event rather than silently passed through.
- Replace the old `normalize_inference_path` body with a call to the
canonicalizer. On canonicalization error the raw path is returned
(so inference detection simply misses and the normal forward path
handles and rejects).
- Document the invariant in `sandbox-policy.rego`: `input.request.path`
is pre-canonicalized so rules must not attempt defensive matching
against `..` or `%2e%2e`.
- Architecture doc (`architecture/sandbox.md`) lists the new module.
- 24 new unit tests cover dot segments, percent-encoded dot segments,
double-slash collapse, encoded-slash reject/opt-in, null/control byte
rejection, legitimate percent-encoded bytes round-tripping, mixed-case
percent normalization, fragment rejection, absolute-form stripping,
empty / length-bounded / non-ASCII handling, and regression payloads
for the specific bypasses above.
Tracks OS-99.
* fix(sandbox): close forward-proxy parser-differential and wire allow_encoded_slash
Addresses two review findings on the L7 path canonicalization change.
1. `handle_forward_proxy` previously evaluated OPA on the canonical path
but wrote the raw path to the upstream via `rewrite_forward_request`,
leaving the parser-differential open for plain-HTTP forward-proxy
traffic even though it was closed for the L7 tunnel path. `path` is
now reassigned to the canonical form inside the L7 block before the
outbound write, so the bytes dispatched to the upstream match what
OPA evaluated. Covered by a new regression test against
`rewrite_forward_request`.
2. `allow_encoded_slash` is now wired through the YAML policy schema,
the proto `NetworkEndpoint` message, the Rego endpoint config, and
the `L7EndpointConfig` → `RestProvider` canonicalization options.
Operators can opt in per-endpoint for upstreams like GitLab that
embed `%2F` in namespaced resource paths; the default remains
strict. The `RestProvider` gains a constructor
`RestProvider::with_options` so `relay_rest` constructs a provider
with per-endpoint options while `relay_passthrough_with_credentials`
retains strict defaults.
Integration tests:
- `parse_http_request_canonicalizes_target_and_rewrites_raw_header`
drives a non-canonical request through the parser and asserts both
`L7Request.target` (what OPA evaluates) and `raw_header` (what the
upstream receives) are canonical.
- `parse_http_request_canonicalization_preserves_query_string`,
`parse_http_request_leaves_canonical_input_byte_for_byte`,
`parse_http_request_rejects_traversal_above_root`,
`parse_http_request_preserves_http_10_version_on_rewrite`.
- `parse_http_request_accepts_encoded_slash_when_endpoint_opts_in` /
`_rejects_encoded_slash_by_default` verify the
`allow_encoded_slash` gate.
- `test_rewrite_forward_request_uses_canonical_path_on_the_wire`
asserts the fix to the forward-proxy flow.
- `parse_l7_config_allow_encoded_slash_{defaults_false,opt_in}` verify
the YAML/Rego wiring.
John Myers