* feat(workspace): implement workspace model (Phase 1 of RFC 0011)
Implements workspace and membership model providing hard isolation
boundaries for multi-player OpenShell deployments.
Workspace CRUD with Kubernetes-style Terminating phase for graceful
deletion. All resources scoped by workspace via ObjectMeta. Membership
RPCs for workspace access control. Persistence migration shifts name
uniqueness to (object_type, workspace, name). Provider profiles support
platform and workspace scoping. Service routing uses workspace-prefixed
DNS labels. Inference routes renamed and workspace-scoped with
DeleteInferenceRoute RPC. Python SDK with WorkspaceClient, two-method
list pattern (workspace-scoped and for_all_workspaces), and workspace
parameter on all methods. CLI workspace flags, TUI workspace cycling.
K8s driver filters unmanaged CRs and uses delete preconditions. Podman
driver uses immutable container IDs. Label serialization fixed across
all put_if call sites.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(cli): delegate sandbox upload command to existing upload function
The standalone `sandbox upload` command reimplemented upload logic
inline with two bugs: it used `Path::exists()` which follows symlinks
(rejecting dangling symlinks), and it ran git-aware filtering on
symlink sources. The `run::sandbox_upload()` function already handles
both cases correctly via `sandbox_upload_plan()`. Replace the inline
logic with a call to the existing function.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(e2e): shorten sandbox names and fix test compatibility
Shorten the sandbox name in initial_sparse_policy_is_acknowledged_as_loaded
from 'e2e-2159-sparse-enrich' (22 chars) to 'e2e-sparse-enrich' (17 chars)
to comply with MAX_ROUTABLE_NAME_LEN (19 chars).
Also capture stderr in create_keep_with_args so future sandbox creation
failures include the actual CLI error instead of reporting empty output.
Signed-off-by: Derek Carr <decarr@redhat.com>
* test(workspace): add test coverage for workspace CRUD and persistence isolation
Add unit tests for workspace create happy path, get round-trip, get
not-found, get empty-name rejection, already-exists error, and
resolve_workspace not-found. Add persistence test proving cross-workspace
name uniqueness (same name in different workspaces produces separate
records). Add workspace name max-length boundary tests. Fix e2e harness
to include stderr in name-parse-failure error path. Align Python e2e
test_workspace_crud with try/finally pattern. Document provider profile
catalog workspace scoping gap in RFC 0011.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(examples): update examples for workspace model compatibility
Shorten sandbox names in demo scripts to fit the 19-character
MAX_ROUTABLE_NAME_LEN limit: policy-demo prefix to pd-, multi-agent
notepad derives a short SANDBOX_TAG from the run ID, governance
interceptor uses gs-PID-RANDOM. Update vscode-remote-sandbox.md SSH
host aliases from openshell-{name} to openshell-{name}.{workspace}
format.
Signed-off-by: Derek Carr <decarr@redhat.com>
* feat(sdk): add workspace-scoped client and workspace CRUD
Add WorkspaceScopedClient modeled after kube::Api::namespaced — captures
workspace once and injects it into every sandbox request. Add workspace
CRUD methods (create, get, list, delete) and list_sandboxes_all_workspaces
on OpenShellClient. Extend SandboxRef with workspace field and add
WorkspaceRef type. Include mock tests for all new operations.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(lint): resolve clippy warnings in workspace test assertions
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(docs): convert indented code blocks to fenced in RFC 0011
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(lint): resolve clippy warnings and apply cargo fmt across workspace
Auto-format with cargo fmt and fix clippy warnings exposed by the
reformat: unnecessary qualifications, map_unwrap_or, identical match
arms, unused variable prefix, dead code annotations, and let-unit-value
in e2e harness.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(workspace): address workspace scoping issues from review
- Add workspace field to settings JSON output (CLI)
- Skip Podman containers missing workspace label instead of defaulting
to empty string, matching K8s driver behavior
- Add resource_version to list_by_scope SELECT in both SQLite and
Postgres backends, with regression test
- Gate PolicyLocalContext proposal/lookup routes on workspace readiness,
returning 503 when workspace is not yet discovered
- Block sandbox and provider creation in TUI all-workspaces mode
- Clear workspace vectors in TUI reset_sandbox_state
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(workspace): make provider profile catalog workspace-aware
Thread workspace through snapshot_catalog so the
EffectiveProviderProfileCatalog enforces workspace boundaries on both
read and write paths. UserProviderProfileSource now loads platform-scoped
profiles (workspace "") plus the target workspace's profiles, preventing
cross-workspace duplicate profile ID collisions that previously caused
global catalog failures.
Update RFC 0011 to reflect catalog scoping is implemented in Phase 1
rather than deferred to future work.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(persistence): include workspace column in atomic policy revision INSERT
put_policy_revision_atomic omitted the workspace column from the INSERT
into the objects table in both SQLite and Postgres backends, causing
atomically-written policy revisions to lose their workspace association.
Add workspace field to AtomicPolicyRevisionWrite and thread it through
both backend INSERT statements, matching the non-atomic put_policy_revision
path which already included it.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(proxy): skip ancestor walk when socket owner is the entrypoint
collect_ancestor_identities walked the entire process tree above the
entrypoint when the connecting process was the entrypoint itself,
SHA256-hashing every ancestor binary (IDE, shell, container runtime).
On dev machines with large binaries in the ancestor chain this exceeded
the 30-second test timeout. When start_pid == stop_pid there are no
intermediate ancestors to verify, so return an empty list immediately.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(workspace): make provider profile catalog scope-aware
Allow the same profile ID at platform and workspace scopes by
introducing layered catalog entries where workspace profiles shadow
platform profiles. Add source and scope fields to the ProviderProfile
proto and CLI output. Migrate List/Get handlers to the catalog,
fixing divergence with runtime profile resolution.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(e2e): align podman e2e labels with centralized driver constants
The podman driver moved its container labels to the centralized
openshell.ai/ prefix, but the e2e test harness and cleanup script
still referenced the old openshell.sandbox-* keys, causing the
local_driver_token_restart test to fail on container lookup.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(e2e): align python profile isolation test with scope-aware catalog
Platform profiles are now visible in workspace listings as fallbacks
per the layered catalog design. Update the assertion to match.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(workspace): honor profile_workspace in runtime profile resolution
Runtime profile lookups now consult provider.profile_workspace via
get_type_profile_for_scope. Providers created with --global-profile
(profile_workspace="") resolve to the platform profile even when a
workspace profile shadows the same ID. All 6 runtime call sites
updated; type-only call sites remain scope-agnostic.
Signed-off-by: Derek Carr <decarr@redhat.com>
---------
Signed-off-by: Derek Carr <decarr@redhat.com>
* test(e2e): run VM suite in CI
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(ci): configure KVM permissions directly
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(e2e): flush VM overlay before restart
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs: simplify VM test documentation
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(e2e): include gateway resume in VM run
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Run cargo fmt for both the root workspace and the standalone e2e Rust workspace
from the rust format and format-check tasks. Apply rustfmt to the e2e sources
so the expanded formatting check passes.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(sandbox): acknowledge initial policy revision
The supervisor loaded and enforced a sandbox-scoped policy but never told
the gateway which revision it loaded. The policy poll loop seeded itself
with the initial revision's hash on its first poll, so `policy_changed`
was never true for that revision and `ReportPolicyStatus(LOADED)` — which
only ran in the hot-reload branch — was never called. The revision stayed
`Pending` and `current_policy_version` stayed 0 even though the sandbox was
`Ready` and the policy was effective. This was most visible with sparse
policies that get baseline-enriched into a new revision during startup.
After the OPA engine is constructed, report the exact sandbox revision the
supervisor loaded as LOADED, and seed the poll loop from that revision so
it is not re-reported. Report FAILED with the original construction error
if engine construction or conversion fails. Only sandbox-sourced revisions
(version > 0) whose canonical content matches the loaded policy are
acknowledged; global and local-file policies are untouched. Delivery uses
the shared bounded retry, is non-fatal on transient failure, and a pending
initial acknowledgement is delivered before any newer revision so policy
history is never reordered.
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
* feat(python): expose sandbox labels and selectors
The gateway protobuf and CLI already support request-level sandbox labels
(`CreateSandboxRequest.name`/`labels`) and selector-based listing
(`ListSandboxesRequest.label_selector`), but the public Python SDK dropped
them, so Python-created sandboxes could not be found via
`openshell sandbox list --selector ...`.
Add optional, source-compatible `name`/`labels` to `SandboxClient.create`,
`create_session`, and the high-level `Sandbox`, and `label_selector` to
`list`/`list_ids`. `SandboxRef` now carries the gateway labels as an
immutable mapping (default empty, so `SandboxRef(id, name, status)` still
works). Caller-provided label mappings are copied. Attaching the high-level
`Sandbox` to an existing sandbox rejects `name`/`labels` since creation
metadata cannot change on attach. Template labels remain a separate concept.
No protobuf changes are required.
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
* fix(python): keep SandboxRef hashable and copy high-level labels
Excluding the new immutable `labels` field from SandboxRef equality/hash
(`compare=False`) preserves the original (id, name, status) identity and keeps
the frozen dataclass hashable — a MappingProxyType field would otherwise make
`hash(SandboxRef(...))` raise. Also defensively copy caller-provided labels in
the high-level `Sandbox` so later caller mutation cannot change what is sent.
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
* fix(sandbox): bound initial-policy-ack retries
The poll loop retried a pending initial acknowledgement before processing any
newer revision, but retried unconditionally forever. A permanently
undeliverable ack (e.g. the revision was superseded before it could be
reported) would then stall all later policy hot-reloads and provider-env
refreshes. Cap the retries; after the bound, give up and resume normal polling
so the loop cannot livelock on a stuck acknowledgement.
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
* test(sandbox): add sparse-policy revision-2 acknowledgement e2e
Regression for #2159: create a sandbox with the network-only policy-advisor
fixture, which the supervisor enriches with baseline filesystem paths during
startup (creating revision 2, superseding revision 1). Assert the effective
policy reaches revision 2 and no revision remains Pending once the supervisor
acknowledges the load. Adds SandboxGuard::create_keep_with_args to create a
kept sandbox with an initial --policy.
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
* fix(ci): correct sandbox checks
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
* test(cli): serialize mTLS environment access
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
* fix(sandbox): address policy review feedback
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
* fix(sandbox): preserve exact policy acknowledgements
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
* fix(sandbox): preserve local policy overrides
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Kyle Zheng <kyzheng@nvidia.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Add policy schema, proto, provider profile, OPA, and L7 proxy support for
`protocol: json-rpc` and `protocol: mcp`. Generic JSON-RPC endpoints match
exact method names only, with `method: "*"` as the all-method sentinel;
wildcard/glob methods and params matchers are rejected.
Parse JSON-RPC request bodies and batches in the forward proxy, deny
response-shaped client frames, limit receive-stream GET allowance to MCP
endpoints, and redact params in decision logs. Preserve L7 rule params on the
proto load path so MCP `tools/call` tool filters behave like YAML-loaded
policies.
Add MCP conformance coverage, JSON-RPC L7 e2e coverage, and docs for the new
protocols and current matcher limitations.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Co-authored-by: ddurst <267424412+ddurst-nvidia@users.noreply.github.com>
Remove the production-crate dev-settings feature so e2e tests build the same gateway, CLI, and supervisor binaries that we ship. Move settings coverage from dummy test-only keys to the production ocsf_json_enabled setting.
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Prefer a single CDI-qualified device when Docker or Podman resolves the default GPU request to one GPU.
Allow nvidia.com/gpu=all only as a WSL2 all-only compatibility fallback, using Docker daemon info and Podman's /dev/dxg probe to identify that case.
Update driver docs, architecture notes, and GPU e2e coverage for the default selection behavior.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* feat(bootstrap): add system gateway registry for installer defaults
Adds a read-only installer-seeded gateway registry that the CLI consults after per-user gateway config. The registry uses the same layout as per-user config with `active_gateway` at the root and `gateways/<name>/metadata.json` beneath it. By default the system config root is `/etc/openshell`, while `OPENSHELL_SYSTEM_GATEWAY_DIR` remains available as an override for packages that need a different location. User-managed gateways continue to shadow installer entries on name collision.
Originally-authored-by: Mark Shuttleworth <mark@ubuntu.com>
Signed-off-by: Alex Lewontin <alex.lewontin@canonical.com>
* feat(cli): show gateway config source in list and term
Expose whether a gateway registration comes from user or system config in `openshell gateway list`, the TUI gateway pane, and list JSON output. The CLI also refuses to remove system-managed registrations and the smoke tests cover the new list output.
Signed-off-by: Alex Lewontin <alex.lewontin@canonical.com>
* fix(bootstrap): preserve user shadowing on invalid metadata
Signed-off-by: Alex Lewontin <alex.lewontin@canonical.com>
* fix(cli): keep system fallback out of rollback state
* docs(gateway): describe system config fallback layout
* test(bootstrap): cover system gateway last_sandbox persistence
* test(gateway): cover system-only removal rejection
* fix(bootstrap): validate gateway names before path joins
---------
Signed-off-by: Alex Lewontin <alex.lewontin@canonical.com>
* fix(cli): fall back to regular upload when git filtering excludes all files
When uploading a directory whose contents are entirely excluded by
.gitignore, git_sync_files returns an empty file list. The git-aware
upload path silently accepted this and printed "Upload complete" without
transferring any files. Now both call sites (direct CLI handler and
sandbox_upload_plan) check for an empty file list and fall through to the
regular, unfiltered upload path instead.
Closes#1778
Signed-off-by: Russell Bryant <rbryant@nvidia.com>
* test(e2e): add regression test for gitignored upload fallback (#1778)
Reproduces the reported workflow: download a file from a sandbox into a
local git repo where the target directory is gitignored, then re-upload
that directory without --no-git-ignore. Before the fix the upload
silently succeeded with zero files transferred.
Signed-off-by: Russell Bryant <rbryant@nvidia.com>
* fix(cli): warn on stderr when gitignore fallback triggers
Address PR review: print a notice when .gitignore filtering excludes
all files and the CLI falls back to unfiltered upload. Add a
GitFilteredEmpty variant to SandboxUploadPlan so the caller can
distinguish this case and emit a visible warning. Document the
default gitignore filtering and fallback behavior in manage-sandboxes.
* fix(cli): warn on stderr for direct upload gitignore fallback
The SandboxCommands::Upload path in main.rs had the same silent
fallback as sandbox_create. Restructure the git_sync_files check to
emit the same .gitignore warning before falling back to unfiltered
upload. Assert the warning in the E2E regression test.
* docs(sandboxes): fix misleading upload fallback guidance
Replace inaccurate "explicit file list" advice with actionable
alternatives: --no-git-ignore, uploading outside the work tree,
or git force-adding the files.
* fix(cli): collapse nested ifs to satisfy clippy::collapsible_if
* style(cli): apply rustfmt to main.rs
---------
Signed-off-by: Russell Bryant <rbryant@nvidia.com>
* feat(gateway): add readiness probe metrics and test-only store close
Emit Prometheus readiness metrics for database probes (healthy gauge and
outcome-labeled latency histogram) with coverage in health HTTP tests.
Restrict Store::close behind test support cfg to prevent accidental runtime
pool shutdown under live traffic.
Signed-off-by: Adrien Langou <alangou@nvidia.com>
* test(e2e): add simple e2e test with kubernetes to test /readyz
Signed-off-by: Adrien Langou <alangou@nvidia.com>
---------
Signed-off-by: Adrien Langou <alangou@nvidia.com>
* test(e2e): close Podman driver test coverage gaps
Add podman_gateway_resume test following the VM pattern (no
container-state assertions since Podman keeps containers running across
gateway restarts). Widen websocket_conformance feature gate from
e2e-docker to e2e-host-gateway so it runs on both Docker and Podman.
Fix e2e-podman.sh to default to the e2e-podman feature so Podman-
specific and host-gateway tests are actually included in Podman CI runs.
* refactor(e2e): extract shared CLI helpers from gateway_resume tests
Move run_cli, wait_for_healthy, sandbox_names, and
wait_for_sandbox_exec_contains into a new harness::cli module. All three
gateway_resume variants (Docker, Podman, VM) now import these shared
helpers instead of defining identical copies. Docker-specific container
introspection functions remain local to gateway_resume.rs.
Validates that direct TCP connections bypassing the HTTP CONNECT proxy
are rejected immediately (ECONNREFUSED) rather than hanging until a
network timeout. This is implementation-agnostic — it tests observable
kernel behavior regardless of whether rules are installed via iptables
or nftables.
* fix(cli): cp-style sandbox download and workspace-boundary check
`openshell sandbox download` produced a directory at the host destination
instead of writing the source bytes as a regular file, and did not refuse
sandbox-side paths that resolved outside `/sandbox` (only `creds.json` was
rejected, by filesystem policy rather than by an explicit check).
Mirrors the cp-style design from the upload-side fixes (#667 / #694 and
#885 / #952): destination resolution now respects trailing slashes and
existing-directory destinations, single-file sources land as regular files
via a staged extraction and atomic rename, and the function delegates the
SSH tar stream to a shared helper. Adds an explicit lexical
workspace-boundary check on the sandbox-side source — required on the
download side because filesystem policy alone is not a substitute for an
explicit check when crossing the trust boundary in the leak direction.
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
* fix(cli): canonicalise sandbox download path remotely and quote tar member
The lexical check in `validate_sandbox_source_path` clears any path whose
components stay under `/sandbox` after `.` and `..` are folded out, but it
cannot see symlinks. A workspace-relative symlink such as
`/sandbox/etc-link -> /etc` lets `openshell sandbox download <name>
/sandbox/etc-link/passwd /tmp/passwd` slip through, then `tar -C
/sandbox/etc-link passwd` follows the link on the sandbox side and
streams `/etc/passwd` back to the host.
Add `canonicalize_sandbox_source_path` which runs `realpath -e --` over
SSH and rejects the result unless it still equals `/sandbox` or starts
with `/sandbox/`. The lexical guard stays as a cheap fail-fast for
obviously-bad inputs, and the rest of the download flow now uses the
canonical path everywhere a tar invocation or probe might dereference a
symlink.
While editing the single-file branch, also add the `--` separator before
the basename in `tar cf - -C <parent> -- <name>`. Without it, a sandbox
file whose basename starts with `-` (e.g. `--checkpoint-action=...`) is
parsed by GNU tar as an option rather than a member, which at best
fails the wrap and at worst triggers tar option behaviour. The directory
branch passes literal `.` and is unaffected.
Cover both fixes with pure-function tests: `is_under_sandbox_workspace`
gets accept/reject pairs (the same predicate the post-realpath
validator uses) and the new `build_single_file_tar_cmd` helper asserts
the `--` separator and shell escaping survive any future refactor.
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
* chore(cli): drop redundant tempfile dev-dependency entry
`tempfile` was promoted from `[dev-dependencies]` to `[dependencies]`
once the cp-style single-file download path started using
`TempDir::new_in` from production code. The original `[dev-dependencies]`
line stayed behind by accident — entries under `[dependencies]` are
already available to test compilation, so the second declaration is
dead weight.
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
* test(cli): add E2E coverage for sandbox download paths
Adds four end-to-end tests for `openshell sandbox download` so the
happy path and both Codex regressions have explicit coverage:
* `sandbox_download_file_only` — seeds a single file via the new
`SandboxGuard::exec` harness helper and pulls it back, exercising
`sandbox_sync_down_file` without an upload phase.
* `sandbox_download_directory_only` — seeds a small directory tree and
pulls it recursively, exercising `sandbox_sync_down_directory`.
* `sandbox_download_rejects_symlinks_pointing_outside_workspace` —
regression for the high-severity finding. Plants three escape shapes
(`/sandbox/etc-link -> /etc`, `/sandbox/passwd-link -> /etc/passwd`,
and a path with a symlinked component `/sandbox/etc-link/passwd`) and
asserts all three are refused with a workspace-boundary error before
any tar streams. Also checks the host destination stays empty so a
silent leak would surface.
* `sandbox_download_handles_dash_leading_basename` — regression for the
medium-severity finding. Seeds a file named `--checkpoint-action=evil`
and verifies it downloads with byte-for-byte contents, proving the
`--` separator stops tar parsing the basename as an option.
The harness gains a single new method, `SandboxGuard::exec`, that wraps
`openshell sandbox exec --name <name> --no-tty -- <argv>`. Existing
tests preferred the upload path to populate sandbox state, but the
regression cases need to plant filesystem state the upload flow cannot
produce (symlinks, files whose basenames clap would otherwise reject).
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
* docs(cli): note cp-style destination and workspace boundary on download
`openshell sandbox download` now refuses any source path that resolves
outside `/sandbox` (lexically or through a symlink), and the
single-file branch follows `cp`-style placement rules. Document both
so users hitting the workspace-boundary error or the
"land at exact path vs land inside dir" decision can self-serve.
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
* test(cli): tolerate miette line-wrapping in symlink-rejection assertions
`sandbox_download_rejects_symlinks_pointing_outside_workspace` matched
the substring `"outside the sandbox workspace"` against the CLI's error
output. The check is correct in spirit but fragile in practice: miette
wraps long single-line errors across multiple lines at terminal width
and inserts a `│ ` continuation marker, so the substring breaks across
the wrap and the test fails even though the canonicalisation step
correctly refused the source.
Split the match into three short markers — `resolves to`, `outside the`,
`sandbox workspace` — each guaranteed to fit on one wrapped line. The
combined check is stricter than the original (it now requires evidence
that the remote `realpath -e` step actually ran, not just the boundary
refusal) without inheriting the wrap fragility.
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
---------
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
* wip
* refactor(docker): use native rust builds for split gateway/supervisor images
Drop the in-Docker BUILD_FROM_SOURCE path so both images consume only
prebuilt binaries staged natively via tasks/scripts/stage-prebuilt-binaries.sh.
This mirrors what CI does and reuses the host's cargo target cache and
sccache across rebuilds.
- Dockerfile.gateway: nvcr.io/nvidia/distroless/cc:v4.0.4 base (the 4.0.0
tag does not exist on nvcr.io; the registry uses a v prefix). GNU-linked
binary copied to /usr/local/bin.
- Dockerfile.supervisor: scratch base, static musl binary. Static linkage
lets the image stay scratch while still being executable as a Kubernetes
init container.
- skaffold.yaml: each artifact invokes tasks/scripts/docker-build-image.sh,
which stages the binary natively (cargo / cargo-zigbuild) and then builds
the image. Drops the cross-build.sh dependency from the supervisor build.
- seccomp.rs: add a local SYS_kexec_file_load constant for musl/aarch64.
libc 0.2.185 omits the symbol from its musl/aarch64 bindings, so the
supervisor's seccomp filter previously failed to compile for that target.
- architecture/build.md: describe the native-first pipeline and per-image
runtime choices.
Local validation: gateway image 101MB (was 194MB), supervisor image 21.7MB.
helm:skaffold:run deploys cleanly; the static musl supervisor binary runs
correctly in a non-glibc agent container.
* refactor(docker): tighten binary perms via --chown + 0550
Replace `COPY --chmod=755` with `COPY --chown=<user> --chmod=0550` in
the gateway and supervisor Dockerfiles. The binary is no longer
world-readable or world-executable; ownership is pinned to the runtime
user.
- Gateway uses `--chown=nvs:nvs` + `USER nvs:nvs`, matching the only
non-root user defined in `nvcr.io/nvidia/distroless/cc` (UID 1000) and
the Helm chart's `securityContext.runAsUser: 1000`, which overrides
the Dockerfile USER at runtime.
- Supervisor uses numeric `--chown=65534:65534` because the scratch base
has no `/etc/passwd` for name resolution. The supervisor image is
only consumed by the init-container copy-self path; the destination
pod's runAsUser governs execute access.
Validated by deploying to a local k3d cluster via `helm:skaffold:run`
and confirming the gateway StatefulSet reaches 1/1 Running.
* ci(gpu): repoint GPU probe image lookup at Dockerfile.gateway
The previous awk parsed `FROM <image> AS gateway` from the now-deleted
`Dockerfile.images`. The new `Dockerfile.gateway` uses an ARG with a
default (`ARG GATEWAY_BASE_IMAGE=nvcr.io/nvidia/distroless/cc:v4.0.4`)
and `FROM ${GATEWAY_BASE_IMAGE} AS gateway`, so the old script returns
nothing.
Parse the ARG default value directly so the GPU prerequisites check
keeps using the gateway base image as a `nvidia-smi` probe target.
* ci(gpu): pin GPU probe to nvcr.io/nvidia/base/ubuntu:noble
The previous probe parsed the gateway base image out of the Dockerfile,
relying on the fact that the gateway ran on `nvcr.io/nvidia/base/ubuntu`
and that NVIDIA Container Toolkit CDI injection would populate
`nvidia-smi` and the supporting libs at runtime. The new gateway base
(`nvcr.io/nvidia/distroless/cc`) lacks `ldconfig`, a populated
`/usr/bin`, and the broader filesystem layout CDI injection assumes,
so it cannot serve as a GPU probe.
Pin the probe image explicitly to the NVIDIA-managed Ubuntu base. The
probe is independent of the gateway runtime and survives future base
swaps.
* fix(e2e-gpu): pass GPU probe image via env, drop Dockerfile.images parse
The Rust e2e test `gpu_request_for_each_discovered_device_matches_plain_container`
was still parsing the deleted `Dockerfile.images` to derive its GPU probe
image, panicking with `No such file or directory` after this PR's
Dockerfile split.
Move the probe image to a single source of truth in the workflow
(`OPENSHELL_E2E_GPU_PROBE_IMAGE` env at the job level) and require the
e2e test to read it from there. No silent codebase default — the test
panics with a pointer at the workflow if the env is missing, so it
fails loudly rather than drifting from CI.
The prereq probe step in the workflow now consumes the same env, so
the probe image is declared exactly once.
* fix(docker): keep supervisor binary root-owned for rootless Podman
The Podman driver mounts the supervisor image read-only into the sandbox
container at /opt/openshell/bin and runs that container as UID 0, but
deliberately drops DAC_OVERRIDE for hardening (container.rs:419). With
--chown=65534:65534 --chmod=0550 the binary was r-xr-x--- owned by UID
65534, so the container's UID 0 fell into "other" with no read or exec
access and the supervisor crashed on start (ContainerExited code 1).
Docker and Kubernetes both retain DAC_OVERRIDE, so root could still
exec the file — which is why this regression only surfaced in the
Podman e2e job.
Drop --chown from the supervisor COPY so the binary stays root-owned.
Keep --chmod=0550: the security win was dropping world-execute, not
changing the owner. The chown bought nothing here because the container
is always UID 0 regardless of driver, but it actively broke the only
driver that drops DAC_OVERRIDE.
---------
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Add opt-in support for Kubernetes user namespace isolation on sandbox
pods. When enabled, container UID 0 maps to an unprivileged host UID
and capabilities become namespaced, providing defense-in-depth for the
supervisor process.
Configuration is two-layered: a cluster-wide default via
OPENSHELL_ENABLE_USER_NAMESPACES (default false) and a per-sandbox
override via the new `user_namespaces` field on SandboxTemplate.
When user namespaces are active, the pod security context is extended
with SETUID, SETGID, and DAC_READ_SEARCH capabilities to match the
bounding-set requirements inside a user namespace.
Introduces SandboxPodParams struct to replace long argument lists on
sandbox_to_k8s_spec and sandbox_template_to_k8s.
Validated end-to-end on OCP 4.22 (K8s 1.35.3, CRI-O 1.35, RHEL
CoreOS, kernel 5.14) with full SSH tunnel and non-identity UID mapping.
Change sandbox user UID from 1000 to 1000660000 in custom image examples and E2E tests.
Using a high UID range (1000000000+) prevents conflicts with host users when running
without user namespace remapping, where container UIDs map directly to host UIDs.
This resolves fork failures caused by RLIMIT_NPROC enforcement when the host user
already has many threads running.
Signed-off-by: Derek Carr <decarr@redhat.com>
* feat(podman): add Podman compute driver for rootless sandbox management
Adds openshell-driver-podman, a new compute driver that manages OpenShell
sandboxes as rootless Podman containers via the Podman REST API over a
Unix socket. Enables local workstation sandboxes without Kubernetes.
Driver features:
- Bridge networking with ephemeral host-port mapping for rootless SSH reachability
- Named volumes for workspace storage, Podman native health checks, GPU via CDI
- Supervisor binary sideloaded via image volume mount (BYOC-compatible)
- SSH handshake secret injected via Podman secrets API (not plaintext env)
- Typed ContainerSpec structs, input validation, and path-traversal guards
- Cgroups v2 required; fails fast on v1 hosts
- Bounded event stream buffer; watch stream reconnection handled by server watch_loop
- Graceful shutdown and standalone driver binary with gRPC bridge
Rootless-specific fixes:
- Skip drop_privileges when user namespace lacks SETUID/SETGID/DAC_READ_SEARCH caps
- Add /run/netns tmpfs mount for ip netns in rootless containers
- Use secret_env map (not secrets array) for env-var injection in libpod API
- Resolve SSH endpoint to 127.0.0.1:<host_port> instead of unreachable bridge IP
Server/sandbox hardening:
- Split loopback and link-local SSRF gates; Podman/VM drivers allow loopback
- Close SSRF bypass in SSH tunnel Host path by resolving DNS before connecting
- Prevent OPENSHELL_* env var override by user-supplied spec environment maps
- Disable SQLite pool idle_timeout/max_lifetime for in-memory databases
- Emit deleted_event on 404-during-inspect instead of regressing sandbox phase
- Key delete cleanup by stable sandbox_id to survive container label drift
CLI fixes:
- Restore --name as a named flag on sandbox create (not positional)
- Fix exec command arg parsing to not consume sandboxed-command flags
- Propagate SSH verbosity via OPENSHELL_SSH_LOG_LEVEL
Build tooling:
- Add tasks/scripts/container-engine.sh: auto-detects Podman or Docker, exposes
unified ce_* helpers; all build/cluster/VM scripts updated to use it
- Add docker:build:supervisor mise task for standalone supervisor image
- Add openshell-driver-podman to Dockerfile.images pre-fetch/build stages
- Add e2e/rust/e2e-podman.sh and e2e:podman mise task for full lifecycle testing
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): derive grpc endpoint from server bind port
When a user starts the gateway on a non-default port (e.g. --port 8081),
sandbox containers were receiving OPENSHELL_ENDPOINT pointing at the
default port 8080. The driver's auto-detection fallback read
OPENSHELL_BIND_ADDRESS from the environment, which was stale or unset,
and fell back to DEFAULT_SERVER_PORT.
Add gateway_port to PodmanComputeConfig and thread config.bind_address.port()
from the server into the driver so the fallback uses the actual listening
port. Remove the OPENSHELL_BIND_ADDRESS env var read and the
extract_port_from_bind_address helper which are no longer needed.
Add --gateway-port / OPENSHELL_GATEWAY_PORT to the standalone driver
binary for parity when the driver is run outside the embedded server path.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(driver-podman): address PR feedback on env test safety and cluster DNS docs
Replace hand-rolled unsafe TempEnvVar RAII guard with temp_env::with_vars
and a static ENV_LOCK mutex, fixing a data race in parallel test execution.
The prior safety comment incorrectly claimed Cargo runs tests single-threaded.
Update debug-openshell-cluster skill to accurately document the DNS proxy
strategy (setup_dns_proxy + public DNS fallback) and clarify the separation
between cluster DNS and sandbox agent DNS enforcement.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(e2e): resolve CI failures in auth timeout, test harness, and formatting
- Short-circuit browser_auth_flow when OPENSHELL_NO_BROWSER=1 instead
of waiting the full 120s AUTH_TIMEOUT for a callback that never arrives
- Add timeout to SandboxGuard::create() and create_with_upload() to
prevent indefinite hangs (matches create_keep() which already had one)
- Add missing '--' separator in no_proxy test before command args
- Add #![cfg(feature = "e2e")] gate to sandbox_lifecycle.rs
- Run cargo fmt on openshell-driver-podman
- Refine cluster DNS docs for Podman in debug-openshell-cluster skill
Signed-off-by: Adam Miller <admiller@redhat.com>
* refactor(server): remove allows_loopback_endpoints from ComputeRuntime
SSRF protection is now handled at the network and proxy layers
(openshell-core net.rs, openshell-sandbox proxy.rs) rather than
requiring per-driver flags on ComputeRuntime. Update architecture
docs to reflect supervisor relay SSH transport and add rootless
networking deep-dive.
Signed-off-by: Adam Miller <admiller@redhat.com>
---------
Signed-off-by: Adam Miller <admiller@redhat.com>
Remove test_live_policy_update_and_logs and
test_live_policy_update_from_empty_network_policies from the Python e2e
suite. Both used a manual 90s poll loop against GetSandboxPolicyStatus
that flaked in CI with 'Policy v2 was not loaded within 90s'.
Add e2e/rust/tests/live_policy_update.rs with two replacement tests
that exercise the same policy lifecycle (version bumping, hash
idempotency, policy list history) through the CLI using the built-in
--wait flag for reliable synchronization.
* feat(bootstrap): switch GPU injection to CDI where supported
Use an explicit CDI device request (driver="cdi", device_ids=["nvidia.com/gpu=all"])
when the Docker daemon reports CDI spec directories via GET /info (SystemInfo.CDISpecDirs).
This makes device injection declarative and decouples spec generation from consumption.
When the daemon reports no CDI spec directories, fall back to the legacy NVIDIA device
request (driver="nvidia", count=-1) which relies on the NVIDIA Container Runtime hook.
Failure modes for both paths are equivalent: a missing or stale NVIDIA Container Toolkit
installation will cause container start to fail.
CDI spec generation is out of scope for this change; specs are expected to be
pre-generated out-of-band, for example by the NVIDIA Container Toolkit.
---------
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
Drew Newberry