* feat(k8s): add namespace-per-workspace support (RFC 0011 Phase 3)
Implement three workspace namespace modes for the Kubernetes compute
driver: shared (default, preserves current single-namespace behavior),
managed (auto-creates/deletes namespaces per workspace), and operator
(pre-provisioned namespaces with dynamic discovery via label selector
or drop-in allowlist file).
Key changes:
- WorkspaceMode enum and namespace resolution in driver config
- Managed namespace lifecycle with ServiceAccount and OpenShift SCC
annotation propagation
- Cluster-wide sandbox CR watchers for managed/operator modes
- NamespaceValidator (Exact/Prefix/Allowlist) for SA token auth
- Workspace-aware credential secret storage
- Helm ClusterRole for multi-namespace RBAC
- Gateway config, architecture, and reference docs
Signed-off-by: Derek Carr <decarr@redhat.com>
* test(k8s): add e2e tests for workspace namespace modes
Add end-to-end tests for managed and operator workspace modes
introduced in RFC 0011 Phase 3. The managed mode tests verify
namespace creation with correct labels, ServiceAccount provisioning,
sandbox CR placement, and namespace survival with remaining sandboxes.
The operator mode tests verify rejection of unlabeled and nonexistent
namespaces. The positive operator path (sandbox in labeled namespace)
is known to fail due to an RBAC gap and will be addressed separately.
Also fixes Helm 4 compatibility: move SPDX license headers inside
conditional guards in 8 chart templates to prevent empty comment-only
documents, and fix a trailing whitespace trimmer in clusterrole.yaml
that concatenated the license header with apiVersion.
Adds cleanup sweep in with-kube-gateway.sh to remove managed and
operator namespaces before Helm uninstall, and mise tasks for running
each mode independently.
Signed-off-by: Derek Carr <decarr@redhat.com>
* feat(k8s): add operator namespace label watcher
Spawn a background kube::runtime::watcher in the K8s driver that
watches namespaces matching the configured label selector and populates
the OperatorNamespaceAllowlist at runtime. The driver owns the
allowlist and exposes its Arc so the server can share the same set with
the SA token authenticator.
create_sandbox now gates pod creation on the allowlist in operator
mode — workspaces whose namespace is not yet labeled are rejected at
resource render time rather than silently proceeding. Workspace
lifecycle itself is unaffected; only sandbox (resource) creation is
gated.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(k8s): harden operator mode and address review findings
Close the fail-open gap in operator mode when only
operator_namespace_file is configured: the allowlist is now created
unconditionally in operator mode (fail-closed from startup).
Implement the namespace file watcher using the notify crate, following
the TLS hot-reload pattern (parent-directory watch, 1s debounce,
ConfigMap symlink-swap safe). The file format is a JSON array of
namespace name strings.
Additional fixes from the 10-reviewer audit:
- Change allowlist rejection from InvalidArgument to FailedPrecondition
so callers know the request may succeed later once the namespace is
provisioned.
- NamespaceValidator::Allowlist now holds the OperatorNamespaceAllowlist
newtype instead of a raw Arc<RwLock<BTreeSet>>, eliminating silent
denial on RwLock poison.
- Verify LABEL_MANAGED_BY and LABEL_GATEWAY_ID ownership before
deleting a managed namespace.
- Replace fixed 5s sleep in operator e2e test with a 30s poll loop.
- Add Helm validation for workspaceMode values.
- Fix Helm README type column and description for operator fields.
- Add insert/remove methods to OperatorNamespaceAllowlist; label
watcher now uses them instead of reaching through shared().
- Reject configs with both operator_namespace_label and
operator_namespace_file set.
Signed-off-by: Derek Carr <decarr@redhat.com>
* feat(k8s): add workspace-level compute driver RPCs and harden RBAC
Decouple namespace lifecycle from sandbox lifecycle by adding
EnsureWorkspace/DeleteWorkspace RPCs to the ComputeDriver service.
Namespace creation now happens before credential storage and namespace
deletion happens on workspace delete, fixing credential storage in
managed workspace mode.
- Add EnsureWorkspace and DeleteWorkspace proto RPCs with
implementations across all compute drivers (K8s managed delegates to
ensure_namespace/delete_namespace_if_empty; others no-op)
- Wire ensure_workspace into provider create/update/refresh paths so
the namespace exists before the credential driver writes secrets
- Wire delete_workspace into workspace deletion for cleanup
- Remove delete_namespace_if_empty from sandbox deletion path
- Scope ClusterRole secrets access to non-shared workspace modes
- Add TODO for TLS cert hot-reload in sandbox gRPC client
- Harden e2e tests with control-plane sandbox resolution assertions
- Fix docker image save --platform flag for OCI index manifests
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(k8s): address re-review findings and add test coverage
- Use server-side apply for TLS secret sync (fixes second sandbox
creation failure when TLS is enabled)
- Scope gateway-ID label selector unconditionally across all workspace
modes (fixes operator reads/watches/deletes seeing foreign sandboxes)
- Validate operator allowlist in EnsureWorkspace and DeleteWorkspace
RPCs (prevents credential writes to namespaces outside the allowlist)
- Extend ClusterRole secrets patch+delete to all non-shared modes with
credential driver enabled (fixes operator credential storage RBAC)
- Validate namespace ownership on 409 conflict in ensure_namespace
(prevents adopting unowned namespaces in managed mode)
- Replace delete_namespace_if_empty with unconditional delete_namespace
letting Kubernetes cascade cleanup (fixes stuck terminating CRs)
- Strengthen NetworkPolicy TODO to cover both managed and operator modes
- Extract selector and ownership logic into testable free functions
- Add unit tests for gateway-ID selectors and namespace ownership
- Add Helm ClusterRole RBAC tests for operator credential driver
Signed-off-by: Derek Carr <decarr@redhat.com>
* ci(k8s): add workspace managed and operator mode e2e to CI
Wire the existing e2e:kubernetes:workspace-managed and
e2e:kubernetes:workspace-operator mise tasks into the branch-e2e
workflow so they run alongside the other core Kubernetes e2e suites.
Both are gated by run_core_e2e and included in the Core E2E result
gate.
Signed-off-by: Derek Carr <decarr@redhat.com>
* test(k8s): add e2e tests for workspace namespace modes
Add 7 new e2e tests covering workspace namespace lifecycle, TLS secret
copying, ownership conflict detection, DNS-1123 validation, operator
namespace preservation, and dynamic label watcher behavior. Fix async
sandbox deletion race condition in existing tests by polling sandbox
list instead of asserting immediately after delete.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(k8s): grant secrets/patch unconditionally and backfill gateway-id labels
Address two review findings:
1. RBAC: server-side apply (PATCH) is used for TLS secret sync in
multi-namespace modes, but the ClusterRole only granted patch when
the kubernetes-secrets credential driver was enabled. Grant patch
unconditionally for non-shared modes since TLS sync always needs it;
keep delete gated on the credential driver.
2. Upgrade safety: the new gateway-id label selector would orphan
legacy Sandbox CRs that predate its introduction. Add a startup
backfill in shared mode that patches any managed Sandbox CR missing
the gateway-id label before the driver begins serving requests.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(k8s): address workspace namespace review findings
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(k8s): address follow-up review findings
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(k8s): preserve workspace lookup after rebase
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(helm): allow managed secret creation
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(k8s): stop pods in workspace namespace
Signed-off-by: Derek Carr <decarr@redhat.com>
* test(k8s): scope pod deletion check to v1alpha1
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(k8s): address workspace namespace review findings
Signed-off-by: Derek Carr <decarr@redhat.com>
---------
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(core): trust public root CAs alongside the sandbox mTLS CA
The supervisor gRPC client only trusted the CA configured via
OPENSHELL_TLS_CA, since tonic ClientTlsConfig starts with an empty root
store unless with_native_roots()/with_webpki_roots() is also enabled.
Deployments where the gateway server certificate is issued by a public CA
(e.g. cert-manager against an ACME issuer) caused every supervisor
connection to fail the TLS handshake with "UnknownCA", since the sandbox
mTLS CA and the server cert issuer were no longer the same.
Enable both native and webpki roots in addition to the configured CA.
tonic root store is a union of all configured sources, so this does not
weaken verification for existing self-signed deployments. webpki-roots
(compiled in) is enabled alongside native-roots since the supervisor
binary may run in minimal sandbox images without a populated system CA
bundle.
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* feat(helm): support external cert-manager issuers and OpenShift Route passthrough
Add certManager.serverIssuerRef/clientIssuerRef so the gateway and mTLS
client certificates can be issued by a real Issuer/ClusterIssuer (e.g.
ACME) instead of only the chart built-in self-signed CA.
Add openshiftRoute template for exposing the gateway via a TLS
passthrough Route so the gateway keeps terminating its own TLS/mTLS.
The server Certificate excludes internal-only SANs (cluster-local,
localhost, loopback) when an external issuer is configured, since ACME
issuers reject those per CA/Browser Forum baseline requirements. A
template-time fail guard catches the misconfiguration at helm install
time rather than asynchronously at cert-manager issuance time.
Includes Helm unittest coverage for both issuerRef overrides and Route
rendering, plus a CI values overlay for lint coverage.
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* docs: document cert-manager external issuer and OpenShift Route
Update managing-certificates.mdx with the serverIssuerRef workflow and
install-time validation behavior. Add a production section to the
OpenShift guide covering passthrough Route with a real certificate.
Regenerate Helm README for new certManager and openshiftRoute values.
Sync debug-openshell-cluster skill with new troubleshooting steps for
ACME issuance failures and supervisor UnknownCA from mismatched CAs.
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* fix(helm,core): address PR review feedback on cert-manager external issuer
Addresses all five blocking review items from #2468:
1. Remove .with_native_roots() from supervisor gRPC client -- the
supervisor runs inside the user-selected sandbox image, so the
image CA bundle is not operator-controlled. Keep .with_webpki_roots()
(compiled-in, not user-controlled) alongside the configured CA.
2. Fail at render time when serverIssuerRef.name is set but
clientCaFromServerTlsSecret is still true. Add negative Helm test.
3. Remove clientIssuerRef -- changing only clientIssuerRef breaks both
directions because trust bundles are not modeled separately. Change
serverIssuerRef.kind default from ClusterIssuer to Issuer.
4. Add server.oidc.issuer and server.oidc.audience to the documented
OpenShift production Helm command. Add Access Control prerequisite.
5. Fail at render time when openshiftRoute.enabled and disableTls are
both true. Add negative Helm test.
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* fix(drivers): strip GATEWAY_TLS_SERVER_NAME from Docker and Podman env
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* fix(helm,drivers): guard default clientCaSecretName and add env-strip tests
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* feat(tls): SNI-based dual certificate for internal and external server TLS
Split the gateway server certificate into two: an internal cert issued by
the chart's own CA (for supervisor connections via cluster-local SANs) and
an external cert issued by an operator-configured Issuer such as ACME/Let's
Encrypt (for CLI and Route access via public SANs).
The gateway uses SNI-based certificate selection: connections whose SNI
hostname matches external_server_names receive the external cert; all
others (including those with no SNI) receive the internal cert.
Security improvement: remove .with_webpki_roots() from the supervisor
gRPC client so supervisors trust only the chart CA, closing a MITM vector
via publicly-trusted certificates in user-supplied container images.
Key changes:
- Add DualCertResolver with SNI-based cert selection and full test coverage
- Add external_cert_path, external_key_path, external_server_names to TlsConfig
- Validate partial external cert config (error on cert-without-key or vice versa)
- Validate empty external_server_names when external cert is configured
- Split cert-manager templates into internal + external Certificate resources
- Add Helm guards for misconfigured external issuer (empty serverDnsNames,
internal-only SANs with external issuer, conflicting clientCaFromServerTlsSecret)
- Update gateway-config.mdx, managing-certificates.mdx, openshift.mdx docs
- Update debug-openshell-cluster skill for dual-cert troubleshooting
Signed-off-by: Pi Agent <agent@openshell.local>
* fix(drivers): strip GATEWAY_TLS_SERVER_NAME in VM driver and correct comments
Add the same GATEWAY_TLS_SERVER_NAME environment stripping to the VM
compute driver that Docker, Podman, and Kubernetes drivers already
perform. Without this, a sandbox user on the VM driver could override
the TLS server name the supervisor verifies.
Fix stale comments in Docker and Podman drivers that referenced
'with WebPKI roots trusted' — WebPKI roots are explicitly not trusted
after the tls-webpki-roots removal.
Use tls-ring instead of bare channel for tonic in openshell-core so the
TLS API (ClientTlsConfig, Endpoint::tls_config) is available without
pulling in any root certificate store.
Signed-off-by: Pi Agent <agent@openshell.local>
* fix(tls,helm): wildcard SNI matching and Route host validation
Add RFC 6125 single-level wildcard matching to DualCertResolver so
external_server_names entries like *.example.com correctly match SNI
hostnames like gw.example.com. Previously only exact matches worked,
silently falling back to the internal cert for wildcard configurations.
Add a Helm fail guard in route.yaml that rejects openshiftRoute.host
values not listed in certManager.serverDnsNames when an external issuer
is configured — catches cert/route hostname mismatches at install time
instead of at TLS connect time.
Quote the host field in route.yaml for robustness.
Signed-off-by: Pi Agent <agent@openshell.local>
* fix(helm): address blocking review items — client-CA guard, wildcard Route, serverIssuerRef gate
1. Remove the obsolete guard rejecting serverIssuerRef + clientCaFromServerTlsSecret=true.
The internal server certificate is always signed by the chart CA (the same
CA that signs the client cert), so clientCaFromServerTlsSecret=true is
correct — its filtered ca.crt is exactly the right trust anchor. The old
workaround (mounting openshell-ca-tls directly) unnecessarily exposed the
CA private key to the gateway container. Remove the client-CA overrides
from docs, CI overlay, and production examples.
2. Route host validation now supports wildcard certificates per RFC 6125:
single-level wildcards like *.example.com match gateway.example.com but
not deep.sub.example.com. Require an explicit openshiftRoute.host when
an external issuer is configured — without one, OpenShift generates a
hostname absent from serverDnsNames.
3. Reject serverIssuerRef.name when certManager.enabled is false — the
external certificate, its Secret mount, and the gateway TLS config all
require cert-manager to be enabled.
Validated on ROSA (dev.dyee.p3) with branch-built images:
- Fresh install with letsencrypt-prod ClusterIssuer
- SNI dual-cert: external hostname served Let's Encrypt cert
- Supervisor mTLS via internal cert path: ConnectSupervisor accepted
- Client CA volume: filtered ca.crt from internal server secret (no key)
- CLI connected via Route + OIDC
Helm tests: 81 pass across 7 suites.
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
---------
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Signed-off-by: Pi Agent <agent@openshell.local>
* build(bazel): stage VM runtime bundle
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* build(bazel): add VM driver targets
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Continue distributed traces across the gateway-to-driver process boundary and
export VM driver spans to the same OTLP/gRPC collector. The driver reports as
the distinct openshell-driver-vm service.
Updated the gateway architecture and configuration reference with a generic
external-driver forwarding contract.
Instrumented:
- Every RemoteComputeDriver RPC injects the active W3C trace context into
tonic metadata. Managed VM readiness and runtime initialization give startup
capability probes stable parent operations rather than isolated root spans.
- A tonic service layer creates fixed, low-cardinality server spans for every
ComputeDriver RPC. New handlers inherit tracing automatically; failures record
OpenTelemetry error status and the gRPC status code.
- Background provisioning remains attached to CreateSandbox after the RPC
returns without extending the RPC span lifetime.
- Provisioning records image preparation, bootstrap image resolution, overlay
preparation, lifecycle configuration, pre-launch hooks, guest preparation,
and launcher spawn as child spans.
- VM startup reconciliation roots one trace for the persisted-sandbox scan,
with per-sandbox restore and provision operations beneath it. The root remains
open until all spawned restore tasks finish.
- Delete cleanup records its own child operation.
Design notes:
- The gateway forwards its configured OTLP endpoint to managed external
drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits,
headers, and transport tuning.
- The VM driver has its own tracer provider and service resource so trace
backends preserve the service boundary.
- RPC operation names come from an explicit method mapping, keeping cardinality
bounded without parsing the protobuf descriptor set at runtime.
- Propagation uses a remote SpanContext for spawned provisioning. This keeps
one trace while allowing the CreateSandbox server span to finish when the
RPC response is sent.
- Startup restoration is independent of gateway requests. It begins at the VM
driver reconciliation span rather than attaching to an unrelated RPC.
- Existing tracing events remain on the logging path. The OpenTelemetry layer
exports spans only and excludes the SDK exporter callsites to avoid recursive
traces.
- Export configuration failures do not prevent the driver from serving, and
buffered spans are drained during graceful shutdown.
- Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC
outcomes without recording credentials, sandbox tokens, or request query
parameters.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Remove temporary registry layer data before formatting the merged rootfs, preserve formatter execution failures over missing fallbacks, and validate required ext4 tools in rust-vm CI.
Closes#2423
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(driver-podman): resolve Podman socket via auto-detection
Align Podman socket selection with the existing Docker model: explicit
config wins, otherwise probe openshell-core for a responsive socket.
This also fixes the original HOME-unset panic, since resolution no
longer hardcodes a per-OS default path.
- add detect_podman_socket() in openshell-core, mirroring
detect_docker_socket
- PodmanComputeConfig.socket_path is now Option<PathBuf>, no default
- remove default_socket_path() (podman driver) and podman_socket_path()
(vm driver), both replaced by the shared detector
- update server env override and CLI for the new Option type
- add tests: responsive-candidate detection in openshell-core, and
config-error (not panic) when no socket is configured or reachable
* fix(driver-podman): address review feedback on socket resolution
Extract socket resolution into resolve_socket_path, taking the
detector as a parameter so tests do not depend on real env vars or
the host's actual Podman state. Replace the flaky env-mutating test
with three deterministic cases: explicit wins, detected is used when
absent, and neither source errors.
Fix the socket_path doc comment to describe it from a config user's
point of view, matching DockerComputeConfig's docstring. Drop a
comment that only made sense next to the Docker driver code.
Update the Podman README and gateway docs: they described a fixed
per-OS default path that no longer exists, replace with the actual
probe-then-fail behavior.
* docs(driver-podman): simplify socket default description
Previous wording was self-contradictory (says auto-detect on unset,
then lists the same var as a probed candidate) and omitted the Linux
/run/user/uid/podman/podman.sock candidate.
* feat(workspace): implement workspace model (Phase 1 of RFC 0011)
Implements workspace and membership model providing hard isolation
boundaries for multi-player OpenShell deployments.
Workspace CRUD with Kubernetes-style Terminating phase for graceful
deletion. All resources scoped by workspace via ObjectMeta. Membership
RPCs for workspace access control. Persistence migration shifts name
uniqueness to (object_type, workspace, name). Provider profiles support
platform and workspace scoping. Service routing uses workspace-prefixed
DNS labels. Inference routes renamed and workspace-scoped with
DeleteInferenceRoute RPC. Python SDK with WorkspaceClient, two-method
list pattern (workspace-scoped and for_all_workspaces), and workspace
parameter on all methods. CLI workspace flags, TUI workspace cycling.
K8s driver filters unmanaged CRs and uses delete preconditions. Podman
driver uses immutable container IDs. Label serialization fixed across
all put_if call sites.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(cli): delegate sandbox upload command to existing upload function
The standalone `sandbox upload` command reimplemented upload logic
inline with two bugs: it used `Path::exists()` which follows symlinks
(rejecting dangling symlinks), and it ran git-aware filtering on
symlink sources. The `run::sandbox_upload()` function already handles
both cases correctly via `sandbox_upload_plan()`. Replace the inline
logic with a call to the existing function.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(e2e): shorten sandbox names and fix test compatibility
Shorten the sandbox name in initial_sparse_policy_is_acknowledged_as_loaded
from 'e2e-2159-sparse-enrich' (22 chars) to 'e2e-sparse-enrich' (17 chars)
to comply with MAX_ROUTABLE_NAME_LEN (19 chars).
Also capture stderr in create_keep_with_args so future sandbox creation
failures include the actual CLI error instead of reporting empty output.
Signed-off-by: Derek Carr <decarr@redhat.com>
* test(workspace): add test coverage for workspace CRUD and persistence isolation
Add unit tests for workspace create happy path, get round-trip, get
not-found, get empty-name rejection, already-exists error, and
resolve_workspace not-found. Add persistence test proving cross-workspace
name uniqueness (same name in different workspaces produces separate
records). Add workspace name max-length boundary tests. Fix e2e harness
to include stderr in name-parse-failure error path. Align Python e2e
test_workspace_crud with try/finally pattern. Document provider profile
catalog workspace scoping gap in RFC 0011.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(examples): update examples for workspace model compatibility
Shorten sandbox names in demo scripts to fit the 19-character
MAX_ROUTABLE_NAME_LEN limit: policy-demo prefix to pd-, multi-agent
notepad derives a short SANDBOX_TAG from the run ID, governance
interceptor uses gs-PID-RANDOM. Update vscode-remote-sandbox.md SSH
host aliases from openshell-{name} to openshell-{name}.{workspace}
format.
Signed-off-by: Derek Carr <decarr@redhat.com>
* feat(sdk): add workspace-scoped client and workspace CRUD
Add WorkspaceScopedClient modeled after kube::Api::namespaced — captures
workspace once and injects it into every sandbox request. Add workspace
CRUD methods (create, get, list, delete) and list_sandboxes_all_workspaces
on OpenShellClient. Extend SandboxRef with workspace field and add
WorkspaceRef type. Include mock tests for all new operations.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(lint): resolve clippy warnings in workspace test assertions
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(docs): convert indented code blocks to fenced in RFC 0011
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(lint): resolve clippy warnings and apply cargo fmt across workspace
Auto-format with cargo fmt and fix clippy warnings exposed by the
reformat: unnecessary qualifications, map_unwrap_or, identical match
arms, unused variable prefix, dead code annotations, and let-unit-value
in e2e harness.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(workspace): address workspace scoping issues from review
- Add workspace field to settings JSON output (CLI)
- Skip Podman containers missing workspace label instead of defaulting
to empty string, matching K8s driver behavior
- Add resource_version to list_by_scope SELECT in both SQLite and
Postgres backends, with regression test
- Gate PolicyLocalContext proposal/lookup routes on workspace readiness,
returning 503 when workspace is not yet discovered
- Block sandbox and provider creation in TUI all-workspaces mode
- Clear workspace vectors in TUI reset_sandbox_state
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(workspace): make provider profile catalog workspace-aware
Thread workspace through snapshot_catalog so the
EffectiveProviderProfileCatalog enforces workspace boundaries on both
read and write paths. UserProviderProfileSource now loads platform-scoped
profiles (workspace "") plus the target workspace's profiles, preventing
cross-workspace duplicate profile ID collisions that previously caused
global catalog failures.
Update RFC 0011 to reflect catalog scoping is implemented in Phase 1
rather than deferred to future work.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(persistence): include workspace column in atomic policy revision INSERT
put_policy_revision_atomic omitted the workspace column from the INSERT
into the objects table in both SQLite and Postgres backends, causing
atomically-written policy revisions to lose their workspace association.
Add workspace field to AtomicPolicyRevisionWrite and thread it through
both backend INSERT statements, matching the non-atomic put_policy_revision
path which already included it.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(proxy): skip ancestor walk when socket owner is the entrypoint
collect_ancestor_identities walked the entire process tree above the
entrypoint when the connecting process was the entrypoint itself,
SHA256-hashing every ancestor binary (IDE, shell, container runtime).
On dev machines with large binaries in the ancestor chain this exceeded
the 30-second test timeout. When start_pid == stop_pid there are no
intermediate ancestors to verify, so return an empty list immediately.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(workspace): make provider profile catalog scope-aware
Allow the same profile ID at platform and workspace scopes by
introducing layered catalog entries where workspace profiles shadow
platform profiles. Add source and scope fields to the ProviderProfile
proto and CLI output. Migrate List/Get handlers to the catalog,
fixing divergence with runtime profile resolution.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(e2e): align podman e2e labels with centralized driver constants
The podman driver moved its container labels to the centralized
openshell.ai/ prefix, but the e2e test harness and cleanup script
still referenced the old openshell.sandbox-* keys, causing the
local_driver_token_restart test to fail on container lookup.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(e2e): align python profile isolation test with scope-aware catalog
Platform profiles are now visible in workspace listings as fallbacks
per the layered catalog design. Update the assertion to match.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(workspace): honor profile_workspace in runtime profile resolution
Runtime profile lookups now consult provider.profile_workspace via
get_type_profile_for_scope. Providers created with --global-profile
(profile_workspace="") resolve to the platform profile even when a
workspace profile shadows the same ID. All 6 runtime call sites
updated; type-only call sites remain scope-agnostic.
Signed-off-by: Derek Carr <decarr@redhat.com>
---------
Signed-off-by: Derek Carr <decarr@redhat.com>
* feat(policy): accept numeric UIDs in sandbox process identity validation
Allow run_as_user and run_as_group to be either the literal 'sandbox'
or a numeric UID/GID within [1000, 2_000_000_000]. This removes the
hard dependency on a baked-in 'sandbox' user in container images,
enabling compute drivers to inject resolved UIDs at sandbox creation.
Phase 1 of #1959.
Signed-off-by: Seth Jennings <sjenning@redhat.com>
* feat(supervisor): accept numeric UIDs for process identity dropping
Allow run_as_user and run_as_group to be numeric UIDs/GIDs, removing
the hard dependency on a baked-in 'sandbox' user in container images.
Changes:
- validate_sandbox_user(): accepts numeric UIDs without passwd lookup
(logs OCSF event); keeps passwd check for "sandbox" name; rejects
non-numeric non-sandbox strings that fail passwd lookup
- prepare_filesystem(): passes numeric UIDs/GIDs directly to chown()
instead of requiring a passwd entry
- drop_privileges(): resolves numeric UIDs/GIDs directly via UID::from_raw
/ Gid::from_raw; skips initgroups when target uid matches current euid;
uses guard conditions before setgid/setuid calls
- session_user_and_home(): falls back to ("{uid}", "/sandbox") for
numeric UIDs, avoiding a passwd lookup that will fail
Re-exports MIN_SANDBOX_UID and MAX_SANDBOX_UID from openshell-policy
so callers have consistent range constants.
Phase 2 of #1959.
Signed-off-by: Seth Jennings <sjenning@redhat.com>
* feat(driver-kubernetes): resolve sandbox UID/GID from config or OpenShift SCC annotations
Phase 3 of the numeric-UID plan: allow operators to specify explicit
sandbox_uid/sandbox_gid in Kubernetes driver config, auto-detect from
OpenShift SCC namespace annotations, and propagate resolved values to
supervisor container env vars and PVC init container securityContext.
Changes:
- Add sandbox_uid/sandbox_gid fields to KubernetesComputeConfig
- Add SANDBOX_UID/SANDBOX_GID env var constants to openshell-core
- Implement resolve_sandbox_identity() to fetch namespace annotations
and auto-detect OpenShift SCC UID ranges (sa.scc.uid-range)
- Pass resolved UID/GID through SandboxPodParams to pod spec builder
- Inject SANDBOX_UID/SANDBOX_GID env vars into supervisor container
- Update PVC init container securityContext with resolved UID/GID
instead of hard-coded root
- Add comprehensive unit tests for resolution logic and annotation
parsing (resolve_sandbox_uid, resolve_sandbox_gid, OpenShift SCC
annotation parsing)
Signed-off-by: Seth Jennings <sjenning@redhat.com>
* feat(driver-vm): add configurable sandbox UID/GID and update docs/examples
Phase 4 of the numeric-UID plan: replace hardcoded SANDBOX_UID (10001)
in VM rootfs preparation with configurable sandbox_uid/sandbox_gid fields.
Changes:
- Add sandbox_uid/sandbox_gid to VmDriverConfig with serde derives
- Pass resolved UID/GID through prepare_sandbox_rootfs_from_image_root
to ensure_sandbox_guest_user which writes /etc/passwd/group/gshadow
- Update BYOC Dockerfile: remove groupadd/useradd, document runtime UID
injection and the ability to skip baked-in sandbox user
- Update gateway-config.mdx: document sandbox_uid/sandbox_gid for both
Kubernetes (with OpenShift SCC autodetection) and VM drivers
- Update sandbox-compute-drivers.mdx: add Sandbox User Identity section
explaining numeric UID support across all compute drivers
- Update rootfs tests to use non-default UIDs, verify config passthrough
Signed-off-by: Seth Jennings <sjenning@redhat.com>
* code review changes
* fix(supervisor): harden tests for restricted CI container environments
Guard tests against CI-specific constraints: root without CAP_SETPCAP,
UIDs with no /etc/passwd entry, and restricted /proc access.
Signed-off-by: Seth Jennings <sjennings@nvidia.com>
Signed-off-by: Seth Jennings <sjenning@redhat.com>
---------
Signed-off-by: Seth Jennings <sjenning@redhat.com>
Signed-off-by: Seth Jennings <sjennings@nvidia.com>
* feat(gpu)!: add resource requirements
BREAKING CHANGE: SandboxSpec.gpu and DriverSandboxSpec.gpu were replaced with resource_requirements.gpu, changing protobuf field 9 from a bool to a message for both public and driver APIs.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(gpu): pass requirements through sandbox create
Pass the coupled GPU requirement object through the CLI sandbox_create boundary instead of splitting presence and count into separate arguments.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(gpu): pass requirements to timeout message
Pass ResourceRequirements into the provisioning timeout message helper so GPU hints are derived from the same nested request object used to create the sandbox.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(gpu): pass driver requirements through helpers
Thread Option<GpuResourceRequirements> through driver validation and rendering helpers instead of splitting GPU presence and count into separate arguments.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(gpu): validate exact device requests
Require exact driver GPU device lists to be tied to a GPU request, allow a single exact device to use the default countless request, and require explicit matching counts for multi-device lists.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Move duplicate in-container path constants from the Docker, Podman,
Kubernetes, and VM driver crates into openshell-core::driver_utils:
- TLS_CA_MOUNT_PATH, TLS_CERT_MOUNT_PATH, TLS_KEY_MOUNT_PATH,
SANDBOX_TOKEN_MOUNT_PATH — identical string literals that were
copy-pasted between openshell-driver-docker and
openshell-driver-podman.
- SUPERVISOR_CONTAINER_DIR (/opt/openshell/bin) and
SUPERVISOR_CONTAINER_BINARY (/opt/openshell/bin/openshell-sandbox) —
replaces three diverging local constants across docker, kubernetes,
podman, and vm-rootfs that referred to the same paths under different
names with different string values (the k8s constant was the directory;
the docker/vm constant was the full binary path).
Each driver crate now assigns its local constant from the canonical core
value, keeping existing names visible to tests without any call-site
churn. The openshell-core constants carry doc comments explaining when
to use the dir vs. the full binary path.
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* feat(telemetry): add build-time option to compile out telemetry
Gate anonymous telemetry emission behind a default-on `telemetry` Cargo
feature in openshell-core. The data model (enums, validation, emit_*/enabled*
signatures) stays always-compiled, while the endpoint, HTTP client, queue, and
emission code are feature-gated. With the feature off, enabled() returns false
and emit_* are no-ops, so dependent crates compile unchanged and no telemetry
endpoint, HTTP client, or emission code is included in the binary.
chrono and reqwest become optional dependencies of openshell-core, dropped from
its dependency graph when telemetry is disabled.
Thread the switch through the workspace: every crate depends on openshell-core
with default-features = false, and the default-on `telemetry` passthrough lives
on the binary crates that emit or collect telemetry (openshell-server,
openshell-sandbox, openshell-driver-vm). In-process drivers inherit it via
resolver v2 feature unification.
Build a telemetry-free binary with, e.g.:
cargo build --release -p openshell-server --no-default-features
The runtime OPENSHELL_TELEMETRY_ENABLED switch is unchanged for default builds.
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
* ci(telemetry): guard that telemetry can be compiled out
Add tasks/scripts/verify-telemetry-compiled-out.sh, which inspects a built
binary for telemetry markers (the telemetry endpoint host and client ID) that
exist only when emission code is compiled in. The rust:verify:telemetry-off
mise task builds the gateway with default features (positive control: markers
must be present, so the absent checks can never be silently vacuous) and with
--no-default-features (markers must be absent), and checks the
--no-default-features sandbox binary as well.
Wire the task into the Rust branch-checks job so a regression that reintroduces
telemetry code into a --no-default-features build fails CI.
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
---------
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
* feat(cli): add --env flag to sandbox create and exec
Add `--env KEY=VALUE` (repeatable) to `openshell sandbox create` and
`openshell sandbox exec`, wiring user-specified environment variables
into the existing `SandboxSpec.environment` and
`ExecSandboxRequest.environment` protobuf fields which were previously
only accessible via the SDK/gRPC API.
Fix a bug where user-specified environment variables set via
`SandboxSpec.environment` were silently dropped inside sandbox SSH
sessions. The SSH handler's `apply_child_env()` calls `env_clear()` to
strip supervisor-internal secrets before spawning user shells, but this
also wiped container-level env vars including user-specified ones. This
was introduced in 5fd4885a (2026-02-26, "feat(sandbox): VS Code
Remote-SSH support with platform detection fix") which added
`env_clear()` to fix a VS Code platform detection timeout caused by
leaked supervisor env vars.
The fix introduces an `OPENSHELL_USER_ENVIRONMENT` sidecar env var:
compute drivers (Docker, Podman, Kubernetes, VM) serialize the
user-specified environment as JSON into this variable at container
creation time, the sandbox supervisor deserializes it at startup, and
the SSH handler re-injects the values into child processes after
`env_clear()`. User-specified vars are applied at lowest priority so
proxy, TLS, and provider credentials still take precedence.
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
* fix(sandbox): address PR review feedback for env var injection
- VM driver: restructure build_guest_environment to apply user env
before required driver vars, preventing user-specified values from
overriding OPENSHELL_ENDPOINT, PATH, and other critical vars.
Matches the pattern already used by Docker/Podman/Kubernetes.
- All drivers: insert OPENSHELL_USER_ENVIRONMENT sidecar after
environment.extend(user_env) so a user-supplied key of the same
name cannot clobber the JSON payload the supervisor parses.
- Podman driver: fix template/spec env precedence — template env is
now applied first with spec overwriting, matching the other drivers
(spec is user-specified and should win over image defaults).
- CLI: add parse_env_pairs() that rejects OPENSHELL_* prefixed keys
with a clear error message. Both sandbox create and exec call sites
updated to use it.
- SSH handler: filter out OPENSHELL_* keys from user_environment
before injecting into child processes.
- Docs: add "Set Environment Variables" section to manage-sandboxes.mdx
documenting --env for both create and exec, per-command override
semantics, and the OPENSHELL_* reservation.
- Tests: add sandbox_create_env_rejects_reserved_prefix test.
- Fix rustfmt formatting in lib.rs, container.rs, driver.rs.
* fix(server): reject OPENSHELL_* env keys at API boundary
Add server-side validation to reject environment variable keys starting
with OPENSHELL_ in SandboxSpec.environment, SandboxTemplate.environment,
and ExecSandboxRequest.environment. This closes the gap where SDK/gRPC
callers (bypassing the CLI) could inject reserved keys that affect
supervisor-internal state.
The CLI already rejects these keys via parse_env_pairs, but the API
boundary is the authoritative enforcement point since all clients
(CLI, SDK, direct gRPC) pass through it.
* fix(server): validate env key names and control chars at API boundary
Add POSIX env key format validation (^[A-Za-z_][A-Za-z0-9_]*$) and
control character rejection for SandboxSpec.environment,
SandboxTemplate.environment, and ExecSandboxRequest.environment at the
gRPC validation layer. This catches invalid keys like "1BAD" or
"BAD-NAME" and values containing newlines at the API boundary rather
than letting them surface as driver-specific container/pod failures.
Also add matching CLI-side validation in parse_env_pairs so users get
immediate feedback. Update docs to say "OpenShell rejects" rather than
"the CLI rejects" since enforcement is now at both boundaries.
* fix(server): allow exec helper keys and add env size limit
Split exec env validation so OPENSHELL_PYFUNC_B64 (used by the Python
SDK's exec_python()) passes through while runtime-clobberable keys
like OPENSHELL_SANDBOX_ID are still rejected. Create-time validation
remains strict — all OPENSHELL_* keys are blocked.
Add a 256 KiB total serialized-size limit on create-time environment
maps. The drivers serialize the full map into a single
OPENSHELL_USER_ENVIRONMENT env var; capping the input at the API
boundary prevents driver/runtime-specific startup failures from
oversized env blocks.
Add template environment control-character test and exec helper key
acceptance test.
---------
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
Move three duplicated definitions into openshell-core so every
consumer has a single canonical source:
- format_bytes: identical 14-line function existed in docker,
kubernetes, and vm drivers. Moved to openshell-core::progress
where all three already imported from.
- DEFAULT_SANDBOX_PIDS_LIMIT: i64 constant (2048) duplicated in
docker driver and podman config. Moved to openshell-core::config
alongside other shared defaults. Podman re-exports it for
internal call-site compatibility.
- current_time_ms: secrets.rs in openshell-sandbox reimplemented
the same logic as openshell-core::time::now_ms(). Remove the
local copy and call now_ms() directly via the existing dep.
In the Rust ecosystem there's largely three ways to do system calls:
- raw libc
- nix
- rustix
Of the three, libc is almost all `unsafe` and really 95% of use
cases should be either nix or rustix. nix is the original one,
but after having looked at the code of both, I think rustix
is just better designed and organized. It's also reached 1.0,
whereas nix is still making semver-breaking changes (in fact
we're behind here in this project).
Now in practice, we have both *transitively* in the depchain
already, and that's true for quite a lot of projects.
But I think rustix is better, so let's add rustix as
a workspace dependency (process feature) and migrate
a few use cases to it - it's especially better than the raw
libc which is suprisingly widespread.
If we agree to do this, then many other calls can be ported.
Signed-off-by: Colin Walters <walters@verbum.org>
Migrate all sandbox and VM driver network policy enforcement from
iptables to nftables. nftables provides atomic ruleset loading, a
cleaner rule syntax, and is the standard netfilter interface in modern
kernels.
Sandbox bypass enforcement (openshell-sandbox):
- Replace iptables chain of individual rule insertions with a single
atomic nftables ruleset load via nft -f
- New nft_ruleset module with pure functions for ruleset generation
and unit tests
- Combine log and reject rules in one inet family table (handles both
IPv4 and IPv6 in a single ruleset)
- Fall back to reject-only ruleset when kernel lacks nft_log support
- Enable net.netfilter.nf_log_all_netns so log rules work from
non-init network namespaces
- Use temp file for nft ruleset loading instead of stdin for
compatibility with minimal VM guest environments
VM TAP networking (openshell-driver-vm):
- Replace iptables NAT/forwarding rules with nftables equivalents
- New nft_ruleset module for TAP network rule generation with unit
tests
- Atomic table-per-TAP-device lifecycle (create/destroy)
- Host-side rules provide NAT infrastructure and defense-in-depth
isolation (input chain restricts VM to gateway port only, forward
chain blocks unsolicited inbound); primary security enforcement
happens inside the VM guest via the sandbox supervisor's own rules
VM init script:
- Load nft kernel modules at sandbox init
- Enable nf_log_all_netns sysctl for bypass detection logging
OCSF / docs:
- Update firewall rule engine references from iptables to nftables
- Document host firewall interaction model and two-layer enforcement
architecture in VM driver README and compute drivers reference
Closes#1335
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* feat(gateway): add TOML configuration file (RFC 0003)
Introduces an opt-in --config / OPENSHELL_GATEWAY_CONFIG flag that loads a
TOML file with gateway-wide settings and per-driver tables. Source
precedence is CLI > env > file > built-in default, implemented via clap's
ValueSource so existing flags and env vars keep their priority.
Driver crates (kubernetes, docker, podman, vm) now derive Deserialize on
their config structs. SupervisorSideloadMethod gains Deserialize with
kebab-case rename. A per-driver inheritance allowlist on the loader side
overlays [openshell.gateway] shared defaults (default_image,
supervisor_image, image_pull_policy, guest_tls_*, ssh_handshake_skew_secs,
client_tls_secret_name, host_gateway_ip, enable_user_namespaces) onto
each [openshell.drivers.<name>] table before deserialization.
The Helm chart renders a new gateway-config ConfigMap and mounts it at
/etc/openshell/gateway.toml. The migrated OPENSHELL_* env entries are
dropped from the StatefulSet — only the Secret-backed
OPENSHELL_SSH_HANDSHAKE_SECRET remains. database_url stays on --db-url.
Adds examples/gateway/gateway.example.toml and updates architecture/gateway.md
with the source precedence and inheritance rules.
* docs(gateway): drop ssh_handshake_skew_secs and ssh_handshake_secret from examples
Both fields are scheduled for removal. Remove the example values and the
env-only note so the gateway.toml example and the architecture doc stop
recommending settings that will not exist much longer.
* docs(rfc): correct OPENSHELL_CONFIG to OPENSHELL_GATEWAY_CONFIG in RFC 0003
* docs(gateway): add per-driver TOML example configurations
Adds focused single-driver examples next to the comprehensive
gateway.example.toml: kubernetes, docker, podman, and microvm. Each one
demonstrates the realistic settings for that driver plus how shared
[openshell.gateway] defaults inherit into the driver table.
A new unit test (`checked_in_examples_parse`) loads every example through
the config_file loader so schema drift fails CI rather than silently
shipping a broken example.
* refactor(gateway): drop image_pull_policy from shared inheritance
Kubernetes and Podman use mutually-incompatible vocabularies for the same
TOML key:
- Kubernetes: `Always | IfNotPresent | Never` (free-form string passed
verbatim to the K8s API).
- Podman: `always | missing | never | newer` (strict lowercase enum
deserialised into `ImagePullPolicy`).
No value means the same thing in both drivers. Sharing the key at
`[openshell.gateway]` scope and inheriting it into every active driver's
table meant any value safe for one driver was either wrong or silently
dropped for the other (`IfNotPresent` → `ImagePullPolicy::Missing` after
`.unwrap_or_default()`). Operators run one driver per gateway, so the
"shared default" never pays for itself.
Make `image_pull_policy` driver-local:
- Remove the field from `GatewayFileSection` and from
`inheritable_keys()` for both Kubernetes and Podman.
- Drop the file→`RunArgs` merge for the gateway-scope key.
- Stop unconditionally clobbering the driver value with
`config.sandbox_image_pull_policy` in the runtime wiring — only apply
the CLI/env override when it was set (and, for Podman, only when it
parses into the lowercase enum).
- Move the key under `[openshell.drivers.kubernetes]` and
`[openshell.drivers.podman]` in every example, the RFC, the
architecture doc, and the Helm-rendered gateway ConfigMap.
The supervisor pull policy follows the same shape: it is K8s-only and
moves into `[openshell.drivers.kubernetes]` alongside `image_pull_policy`
in the Helm template.
* fix(gateway): address review feedback on TOML configuration
Resolves the P1 and P2 issues raised in PR #1317:
- Helm gateway ConfigMap moves `grpc_endpoint` under
`[openshell.drivers.kubernetes]` so the default install no longer fails
the gateway's `deny_unknown_fields` schema check.
- `kubernetes_config_from_file` and `podman_config_from_file` only let
the gateway-wide CLI/env `grpc_endpoint` overwrite the driver-table
value when it was actually supplied, preserving file-only configs.
- Kubernetes driver default `image_pull_policy` is now empty (was Podman
vocabulary "missing"), so default deployments let the Kubernetes API
apply its own policy instead of being rejected.
- New `disable_tls` gateway field plumbs `.Values.server.disableTls`
through the TOML ConfigMap instead of relying on env vars dropped from
the StatefulSet.
- StatefulSet pod template now carries a `checksum/gateway-config`
annotation so `helm upgrade` rolls pods when the ConfigMap changes.
- Auxiliary listener resolution preserves the full `SocketAddr` from
`health_bind_address` / `metrics_bind_address`, so a loopback-pinned
health port is not silently relocated onto the public bind address.
- `ssh_session_ttl_secs` from the file is now applied to `Config` (it
was previously accepted by the loader but never read).
New regression coverage: cli-level merge tests for the new fields plus
helm-unittest assertions for the ConfigMap shape, checksum annotation,
and `disable_tls` rendering.
* docs(gateway): consolidate gateway TOML examples into docs reference
Replaces the per-driver example files under examples/gateway/ with a
single published reference page at docs/reference/gateway-config.mdx
covering source precedence, layout, the full example, and the four
per-driver examples (Kubernetes, Docker, Podman, microVM). Drew flagged
during PR #1317 review that the examples belong with the user-facing
docs rather than in a sibling examples/ directory.
The cross-references in architecture/gateway.md and RFC 0003 are updated
to point at the new docs page; the round-trip test in config_file.rs is
removed (schema coverage stays on the inline parses_full_example test
and per-field merge tests — doc-snippet drift belongs in a separate
docs-lint, not in a cross-tree Rust unit test).
* refactor(core): move DEFAULT_K8S_NAMESPACE into K8s driver
The constant is Kubernetes-specific (used only by KubernetesComputeConfig's
Default impl) and does not belong in openshell-core. Relocate it to the
driver crate that owns the K8s vocabulary; openshell-core retains only
truly cross-cutting defaults.
* refactor(core): move Podman bridge default into Podman driver
DEFAULT_NETWORK_NAME is Podman vocabulary, consumed only by the Podman
driver. Also drops the unused DEFAULT_IMAGE_PULL_POLICY constant.
* docs(auth): scrub remaining SSH handshake secret references
Sweeps the trailing mentions left after the rebase: the gateway
config-file module doc, the Helm gateway-config ConfigMap header,
the gateway-config.mdx env-only note, and the RPM systemd unit
comment for init-gateway-env.sh.
* docs(gateway): clarify OPENSHELL_GRPC_ENDPOINT applies to all drivers
The previous comment implied the callback endpoint was Kubernetes-only,
but the value is propagated to every compute driver (Kubernetes, Docker,
Podman, VM) and must be reachable from wherever the sandbox runs.
* refactor(gateway): move driver options into config (#1394)
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(e2e): regenerate gateway config via TOML for docker + podman harnesses
The gateway CLI flags moved into TOML config tables in 560550d2 (#1394),
which made every existing e2e/with-{docker,podman}-gateway.sh invocation
fail with "unexpected argument '--sandbox-namespace'" (and a long tail
of similar driver-specific options) before the gateway could even bind.
Replace the obsolete CLI flags with a synthesized
`[openshell.drivers.<driver>]` table written to `${STATE_DIR}/gateway.toml`
and passed via the new `--config` flag. Only the gateway-wide flags
that survived 560550d2 (bind-address, port, drivers, db-url, tls-*,
disable-tls, log-level, health-port) stay on the command line.
Both scripts get a small `toml_string` helper to properly TOML-quote
the values (the previous `%q` printf format produced bash-escape, not
TOML-escape). The Docker harness also corrects two field names that
diverged from the driver schema: `docker_network_name` →
`network_name`, and the supervisor binary/image plumbing now reads
through to `supervisor_bin` / `supervisor_image` in the same table.
The Podman harness drops `--ssh-gateway-port` (deleted in 560550d2 —
gRPC + SSH are multiplexed on the same port now) and substitutes
`network_name` + `gateway_port` for the obsolete `--sandbox-namespace`
(which the Podman driver never had as a typed field).
* fix(core): swap bind-only 0.0.0.0 SSH gateway host for cluster URL host
CLI's resolve_ssh_gateway treated 0.0.0.0 as a loopback "keep as-is"
when the cluster URL was also loopback, so the SSH proxy connected to
0.0.0.0:port. The unspecified address is never a valid connect target
and is not present in any TLS cert SAN, which produced BadCertificate
TLS handshake failures during `openshell sandbox create -- ...` in
docker/podman e2e (e.g. bypass_detection).
Resolution: when the server returns 0.0.0.0 or :: as the gateway host
and both endpoints are loopback, fall back to the cluster URL's host
(which the CLI is already using to reach the gateway, so it must
resolve and match the cert).
* fix(e2e): repair podman harness on macOS
Podman 5.x with the applehv/libkrun provider no longer creates the legacy
~/.local/share/containers/podman/machine/podman.sock symlink, and
`podman system service` is a Linux-only subcommand — the macOS client
delegates the API service to the VM. Both assumptions in the harness
were stale, so the script tried to start a temporary service that podman
rejected with "unknown flag: --time".
- Discover the macOS socket via `podman machine inspect` instead of the
hardcoded path.
- On Darwin, fail fast with a "start podman machine" message rather than
attempting the Linux-only `podman system service` fallback.
- Write socket_path into [openshell.drivers.podman] so the in-process
driver picks up the discovered socket; the driver reads TOML only
after the config refactor (560550d2), so OPENSHELL_PODMAN_SOCKET alone
was no longer enough.
* fix(server): use clone_from for TLS client CA assignment
clippy 1.95.0 rejects assigning the result of `Clone::clone()` to an
existing variable under `-D warnings` (`assigning_clones`). Switch to
`clone_from(&...)` to satisfy the lint and avoid the redundant
allocation.
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Clippy's collapsible_if lint fails `mise run rust:lint` on main because
of the nested `if let Ok(..) { if ping().is_ok() { .. } }` pattern in
`connect_local_container_engine`. Introduced by 44e843ed
("feat(vm): fall back to Podman socket when Docker is unavailable",
#1370). Rewrite both blocks using `&&` let-chains.
* test(e2e): add bypass detection test for sandbox REJECT rules
Validates that direct TCP connections bypassing the HTTP CONNECT proxy
are rejected immediately (ECONNREFUSED) rather than hanging until a
network timeout. This is implementation-agnostic — it tests observable
kernel behavior regardless of whether rules are installed via iptables
or nftables.
* feat(vm): fall back to Podman socket when Docker is unavailable
The VM driver resolves sandbox images from a local container engine
before falling back to registry pulls. Previously it only checked
Docker. Now it tries the Podman socket as a fallback, since Podman
exposes a Docker-compatible API via bollard.
On Linux, enable the Podman API socket with:
systemctl --user start podman.socket
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
* chore(vm): fix formatting
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
* refactor(vm): rename docker-specific identifiers to engine-neutral names
Rename resolve_local_docker_image → resolve_local_container_image,
local_docker_image_platform_mismatch → local_image_platform_mismatch,
and the local docker variable → engine since the function now supports
both Docker and Podman backends.
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
---------
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
The nftables log statement requires CONFIG_NFT_LOG, which was not
included in the VM guest kernel config. Without it, nft rejects
rules containing the log keyword with "No such file or directory".
On Linux non-root hosts, virtiofs maps guest root to the host user,
so chown inside the VM is denied. This causes the init script to
abort under set -e, preventing the sandbox from starting.
Make the chown best-effort: if it fails, log a warning and continue.
The supervisor's own filesystem preparation handles ownership for the
paths that matter, so the sandbox works correctly without it.
This is a pragmatic workaround — anything that would break due to
the failed chown will still break, just later and with a clearer
error rather than a cryptic boot failure. In practice the sandbox
functions normally because the supervisor sets up its own working
directories.
Signed-off-by: Russell Bryant <russell.bryant@gmail.com>
* refactor!(auth): drop SSH handshake secret in favor of mTLS
The OPENSHELL_SSH_HANDSHAKE_SECRET / x-sandbox-secret mechanism was
misnamed: it does not authenticate SSH (which flows over the
RelayStream gRPC RPC and is gated by mTLS plus supervisor Unix-socket
permissions). It only gated a small set of sandbox-to-gateway
control-plane RPCs, and production deployments already enforce mTLS
on that channel — so the shared secret was redundant.
Replace the secret check with an mTLS-presence marker. Sandbox-class
methods (ReportPolicyStatus, PushSandboxLogs,
GetSandboxProviderEnvironment, SubmitPolicyAnalysis, GetSandboxConfig,
GetInferenceBundle) accept callers without a Bearer token; the gRPC
mTLS handshake is the trust boundary. Dual-auth methods treat
Bearer-present as full-scope CLI access and Bearer-absent as
sandbox-restricted scope via validate_sandbox_caller_update.
Drops the secret from all drivers (K8s, Podman, VM), the sandbox gRPC
interceptor, the Helm chart (values + pre-install hook + StatefulSet
env), the RPM bootstrap script, the man pages, and the
debug-openshell-cluster skill. Also removes the never-read
ssh_handshake_skew_secs flag and config field.
BREAKING CHANGE: --ssh-handshake-secret / OPENSHELL_SSH_HANDSHAKE_SECRET
and --ssh-handshake-skew-secs / OPENSHELL_SSH_HANDSHAKE_SKEW_SECS are
removed from the gateway, sandbox, and all driver binaries. The
openshell-ssh-handshake K8s Secret is no longer managed by the chart;
operators may delete the orphan. Deployments using
--disable-gateway-auth must enforce caller authentication at the
fronting proxy, since the gateway no longer validates a per-request
secret on sandbox-class methods.
Refs OS-174.
* docs(auth): scrub residual SSH handshake secret references
Sweep across docs, e2e scripts, the Podman driver README/NETWORKING
notes, the RPM/Helm/setup guides, the gateway man page, and RFC 0003
to remove instructions and examples that still referenced
OPENSHELL_SSH_HANDSHAKE_SECRET / --ssh-handshake-secret /
ssh_handshake_skew_secs. The mechanism is gone; nothing should still
suggest setting it. Negative-assertion regression tests are kept so
the env var cannot silently be re-introduced.
* test(drivers): drop SSH handshake secret negative-assertion tests
The supporting code, env vars, CLI flags, and config plumbing are
gone — these tests asserted absence of strings that no longer have
any path to being set. Remove the guards from the Docker, Podman,
Kubernetes, and VM driver test modules.
Consolidate shared constants and helpers into openshell-core to avoid
duplicate definitions scattered across crates. Adds OPENSHELL_SANDBOX
to the sandbox_env module so all drivers and the sandbox supervisor
reference it via a typed constant rather than string literals, matching
the existing pattern for SANDBOX_ID, ENDPOINT, and the other env vars.
When the VM driver extracts OCI image layers on a non-root host (e.g.
macOS UID 501), fs::copy and tar::unpack create files owned by the host
user. The sandbox user inside the VM then cannot write to its own home
directory.
Add a conditional chown in the VM init script that detects the mismatch
and fixes ownership before the supervisor starts. The check is skipped
when ownership is already correct (e.g. Linux-as-root extraction).
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
Derek Carr