mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-11-06 08:44:05 +08:00
feat(supervisor): port streamed configuration to RFC 0012
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
This commit is contained in:
Piotr Mlocek
@@ -52,6 +52,9 @@ The `audience` value must match the client ID configured in your identity provid
|
||||
| Value | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `server.oidc.issuer` | `""` | OIDC issuer URL. Empty disables OIDC. |
|
||||
| `server.oidc.dangerouslyAllowInsecureHttp` | `false` | Development-only acknowledgement for numeric-loopback HTTP. It does not allow cluster-service or remote HTTP issuers. |
|
||||
| `server.oidc.jwksAllowedOrigins` | `[]` | Additional trusted HTTPS origins allowed to serve JWKS. |
|
||||
| `server.oidc.caConfigMapName` | `""` | ConfigMap containing the private issuer CA in `ca.crt`. |
|
||||
| `server.oidc.audience` | `openshell-cli` | Expected `aud` claim in the JWT. |
|
||||
| `server.oidc.jwksTtl` | `3600` | JWKS key cache TTL in seconds. Must be greater than zero. |
|
||||
| `server.oidc.rolesClaim` | `""` | Dot-separated path to the roles array in JWT claims. |
|
||||
@@ -59,6 +62,11 @@ The `audience` value must match the client ID configured in your identity provid
|
||||
| `server.oidc.userRole` | `""` | Role name that grants standard user access. |
|
||||
| `server.oidc.scopesClaim` | `""` | Dot-separated path to the scopes array in JWT claims. |
|
||||
|
||||
The issuer must use HTTPS. The gateway rejects discovery and JWKS redirects,
|
||||
limits response sizes, requires a JSON media type, and rejects a `jwks_uri` on
|
||||
a different origin unless that origin appears in `jwksAllowedOrigins`. Use
|
||||
`server.oidc.caConfigMapName` for issuers signed by a private CA.
|
||||
|
||||
### Auth-only mode vs. RBAC mode
|
||||
|
||||
Leave both `adminRole` and `userRole` empty to use auth-only mode: any request with a valid JWT from the configured issuer is accepted, but no role distinction is enforced.
|
||||
|
||||
@@ -57,6 +57,7 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set grpcRoute.enabled=true \
|
||||
--set grpcRoute.gateway.create=true \
|
||||
--set grpcRoute.gateway.className=eg
|
||||
@@ -113,6 +114,7 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set grpcRoute.enabled=true \
|
||||
--set grpcRoute.gateway.create=true \
|
||||
--set grpcRoute.gateway.className=eg \
|
||||
|
||||
@@ -52,6 +52,7 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set certManager.enabled=true
|
||||
```
|
||||
|
||||
@@ -73,6 +74,7 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set certManager.enabled=true \
|
||||
--set certManager.serverIssuerRef.name=letsencrypt-prod \
|
||||
--set certManager.serverIssuerRef.kind=ClusterIssuer \
|
||||
|
||||
@@ -3,87 +3,62 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
title: "OpenShift"
|
||||
sidebar-title: "OpenShift"
|
||||
description: "Install the OpenShell Helm chart on OpenShift, including the SCC binding and chart overrides required by OpenShift's Security Context Constraints."
|
||||
description: "Install the OpenShell Helm chart on OpenShift with capability-free sandbox workloads."
|
||||
keywords: "Generative AI, Cybersecurity, Kubernetes, OpenShift, SCC, Security Context Constraints, Helm, Gateway, Installation"
|
||||
position: 6
|
||||
---
|
||||
|
||||
The Kubernetes driver resolves the UID range assigned to each OpenShift
|
||||
namespace and renders the sandbox and supervisor with a numeric non-root
|
||||
identity from that range. OpenShell does not require the `privileged` SCC or any
|
||||
added Linux capability.
|
||||
|
||||
<Warning>
|
||||
The OpenShift install path is experimental. It currently requires running sandbox pods under the `privileged` SCC and installing the gateway with TLS disabled. Use only for evaluation on a private network.
|
||||
Verify that the selected OpenShift runtime profile permits an unprivileged
|
||||
process to install a nested seccomp user-notification filter and use Landlock.
|
||||
OpenShell fails sandbox startup when either capability-free runtime probe fails.
|
||||
</Warning>
|
||||
|
||||
OpenShift's [Security Context Constraints](https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html) reject the chart's default pod security settings. Installing on OpenShift requires precreating the namespace, granting the `privileged` SCC to the sandbox service account, and overriding a few chart values so the cluster admission controller can assign UIDs and FS groups itself.
|
||||
|
||||
OpenShell installs sandbox nftables rules as individual commands. On OpenShift
|
||||
nodes where optional conntrack or packet log expressions are unavailable, those
|
||||
optional rules can fail without rolling back the required proxy bypass reject
|
||||
rules.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- OpenShift 4.x cluster with `oc` configured
|
||||
- Helm 3.x
|
||||
- [Agent Sandbox](/kubernetes/setup#install-agent-sandbox) controller and CRDs installed
|
||||
- OpenShift 4.x cluster with `oc` configured.
|
||||
- Helm 3.x.
|
||||
- [Agent Sandbox](/kubernetes/setup#install-agent-sandbox) controller and CRDs.
|
||||
- A CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces.
|
||||
|
||||
## Install
|
||||
## Install OpenShell
|
||||
|
||||
<Steps>
|
||||
|
||||
## Create the namespace
|
||||
|
||||
Pre-create the namespace so the SCC binding can be applied before the chart installs:
|
||||
Pre-create the namespace, then install the chart. Keep the default restricted
|
||||
security posture and acknowledge NetworkPolicy only after validating the CNI.
|
||||
|
||||
```shell
|
||||
oc create ns openshell
|
||||
```
|
||||
|
||||
## Grant the privileged SCC to sandbox pods
|
||||
|
||||
Sandbox pods run under the `openshell-sandbox` service account in the `openshell` namespace and require the `privileged` SCC:
|
||||
|
||||
```shell
|
||||
oc adm policy add-scc-to-user privileged -z openshell-sandbox -n openshell
|
||||
```
|
||||
|
||||
## Install the chart with OpenShift overrides
|
||||
|
||||
```shell
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set server.disableTls=true \
|
||||
--set podSecurityContext.fsGroup=null \
|
||||
--set securityContext.runAsUser=null
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
```
|
||||
|
||||
| Override | Reason |
|
||||
|---|---|
|
||||
| `server.disableTls=true` | Runs the gateway over plaintext HTTP for simpler evaluation. |
|
||||
| `podSecurityContext.fsGroup=null` / `securityContext.runAsUser=null` | Clear the chart's hardcoded UID and fsGroup so OpenShift's SCC admission can assign them. |
|
||||
The driver reads the namespace's `openshift.io/sa.scc.uid-range` annotation and
|
||||
uses the resulting UID/GID for the sandbox, agent, trusted init containers, and
|
||||
supervisor. Each container sets `allowPrivilegeEscalation: false`, drops all
|
||||
Linux capabilities, and uses `RuntimeDefault` seccomp.
|
||||
|
||||
## Wait for the gateway to be ready
|
||||
Wait for the gateway:
|
||||
|
||||
```shell
|
||||
oc -n openshell rollout status statefulset/openshell
|
||||
```
|
||||
|
||||
If you set `workload.kind=deployment`, use
|
||||
`oc -n openshell rollout status deployment/openshell` instead.
|
||||
If you set `workload.kind=deployment`, wait for `deployment/openshell` instead.
|
||||
|
||||
</Steps>
|
||||
## Connect to the Gateway
|
||||
|
||||
## Connect to the gateway
|
||||
|
||||
The gateway is now running over plaintext HTTP. Connect with `oc port-forward`:
|
||||
Forward the gateway port for local evaluation:
|
||||
|
||||
```shell
|
||||
oc -n openshell port-forward svc/openshell 8080:8080
|
||||
```
|
||||
|
||||
Register the gateway with the CLI:
|
||||
|
||||
```shell
|
||||
openshell gateway add http://127.0.0.1:8080 --local --name openshift
|
||||
openshell gateway add https://127.0.0.1:8080 --local --name openshift
|
||||
openshell status
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
title: "Kubernetes Sandbox Runtime"
|
||||
sidebar-title: "Sandbox runtime"
|
||||
description: "Understand how Kubernetes places and protects the sandbox runtime and supervisor."
|
||||
keywords: "Generative AI, Cybersecurity, Kubernetes, Sandboxing, Network Policy, Seccomp, Landlock"
|
||||
position: 2
|
||||
---
|
||||
|
||||
OpenShell runs each Kubernetes sandbox as two separately scheduled workloads.
|
||||
The sandbox owns the agent process. The supervisor owns gateway credentials,
|
||||
policy decisions, and upstream connections.
|
||||
|
||||
## Understand the Components
|
||||
|
||||
The Kubernetes driver always uses this placement:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Gateway[OpenShell gateway]
|
||||
Supervisor[Supervisor Pod]
|
||||
Sandbox[Sandbox workload Pod]
|
||||
Agent[Agent processes]
|
||||
External[External services]
|
||||
|
||||
Gateway <-->|JWT-authenticated session| Supervisor
|
||||
Supervisor <-->|TLS and bootstrap token| Sandbox
|
||||
Sandbox --> Agent
|
||||
Supervisor --> External
|
||||
```
|
||||
|
||||
`openshell-sandbox` runs as PID 1 in the workload container. It launches the
|
||||
agent, applies Landlock and child seccomp filters, identifies the process behind
|
||||
each network operation, and relays approved streams. `openshell-supervisor` runs
|
||||
in a directly managed Pod. It authenticates to the gateway, evaluates policy,
|
||||
handles L7 and provider transformations, and opens upstream connections.
|
||||
|
||||
Both containers run as the same namespace-resolved non-root UID and GID. Their
|
||||
Pod specs set `allowPrivilegeEscalation: false`, drop every Linux capability,
|
||||
and use `RuntimeDefault` seccomp. The sandbox installs an additional nested
|
||||
seccomp user-notification filter without requesting a capability. Startup fails
|
||||
closed if the runtime blocks the required seccomp or Landlock operations.
|
||||
|
||||
The supervisor Pod points directly to the Sandbox resource with a non-controller
|
||||
owner reference. Kubernetes therefore removes it when the Sandbox is deleted,
|
||||
while the Agent Sandbox controller remains the sole controller of the workload
|
||||
Pod.
|
||||
|
||||
## Enforce Network Isolation
|
||||
|
||||
The driver creates one workload fence per sandbox namespace before it releases
|
||||
any workload Pod:
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
openshell.ai/boundary-role: workload
|
||||
policyTypes: [Ingress, Egress]
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
openshell.ai/boundary-role: supervisor
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5500
|
||||
egress: []
|
||||
```
|
||||
|
||||
The empty egress list blocks direct DNS, gateway, node, metadata, and Internet
|
||||
connections from every OpenShell workload in the namespace. The ingress rule
|
||||
allows OpenShell supervisor Pods to reach sandbox TLS listeners. TLS, JWT
|
||||
claims, session generation, and recorded Pod UIDs—not the NetworkPolicy—bind a
|
||||
supervisor to its exact sandbox. Supervisors can reach cluster DNS, the gateway,
|
||||
and policy-approved upstream destinations unless another namespace policy
|
||||
restricts them.
|
||||
|
||||
Kubernetes policies are additive. Keep sandbox namespaces under administrative
|
||||
control so another principal cannot add permissive policies, create Pods with
|
||||
OpenShell labels, or read bootstrap Secrets. Set
|
||||
`supervisor.sandboxRuntime.networkPolicyEnforced: true` only after you verify that the
|
||||
cluster CNI enforces both ingress and egress policies for these namespaces.
|
||||
|
||||
## Bootstrap a Sandbox
|
||||
|
||||
The driver creates each sandbox generation in a fail-closed order:
|
||||
|
||||
1. Create and validate the workload egress fence.
|
||||
2. Create the Sandbox resource with a scheduling gate.
|
||||
3. Inspect the admitted Pod identity, security context, DNS settings, and
|
||||
generation-specific Secret reference.
|
||||
4. Create a gated supervisor Pod and separate immutable Secrets for sandbox and
|
||||
supervisor trust material.
|
||||
5. Remove both scheduling gates.
|
||||
6. Publish readiness only after the supervisor attaches, confirms enforcement,
|
||||
and registers the gateway relay.
|
||||
|
||||
The trusted sandbox init container copies its Secret into a memory-backed
|
||||
volume. The main container never mounts the projected Secret and removes the
|
||||
staged bootstrap before it launches untrusted code. The supervisor receives an
|
||||
audience-bound Kubernetes token, exchanges it for a sandbox-scoped JWT, and
|
||||
keeps gateway and provider credentials outside the workload Pod.
|
||||
|
||||
Stopping a sandbox removes both the workload and supervisor Pods. Starting it
|
||||
creates a new generation with new Secrets and a replacement supervisor Pod
|
||||
while preserving the workspace PVC. The namespace-wide workload fence remains
|
||||
in place across sandbox generations.
|
||||
|
||||
## Check Cluster Requirements
|
||||
|
||||
This architecture requires the following cluster behavior:
|
||||
|
||||
- Linux nodes and a container runtime that permits an unprivileged process to
|
||||
install a nested seccomp user-notification filter under `RuntimeDefault`.
|
||||
- Landlock enabled and usable by the non-root sandbox process.
|
||||
- A CNI that enforces `networking.k8s.io/v1` ingress and egress policies,
|
||||
including node-local and metadata destinations.
|
||||
- Support for Pod scheduling gates and the safe
|
||||
`net.ipv4.ip_unprivileged_port_start=0` sysctl.
|
||||
- Administrative control of sandbox namespaces and OpenShell role labels.
|
||||
|
||||
OpenShell actively probes the Linux primitives and validates the admitted Pod
|
||||
before starting the agent. Treat a failed probe or changed security posture as
|
||||
an unsupported runtime, not a degraded mode.
|
||||
+12
-17
@@ -85,7 +85,8 @@ Install from the OCI registry on GHCR. Replace `<version>` with the chart versio
|
||||
helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
```
|
||||
|
||||
To use the latest development build instead of a stable release:
|
||||
@@ -94,7 +95,8 @@ To use the latest development build instead of a stable release:
|
||||
helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version 0.0.0-dev \
|
||||
--namespace openshell
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
```
|
||||
|
||||
The chart automatically generates PKI secrets on first install using pre-install Helm hooks. No manual secret creation is required.
|
||||
@@ -109,6 +111,7 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set workspaceResources.enabled=false \
|
||||
--set server.sandboxNamespace=app-a
|
||||
|
||||
@@ -199,14 +202,12 @@ The most commonly changed values are:
|
||||
| `server.sandboxImage` | Default sandbox image used when a sandbox does not specify one. |
|
||||
| `server.sandboxImagePullSecrets` | Image pull secrets attached to sandbox pods. Referenced Secrets must exist in the sandbox namespace. |
|
||||
| `server.grpcEndpoint` | Endpoint that sandbox supervisors use to call back to the gateway. Must be reachable from inside the cluster. |
|
||||
| `server.appArmorProfile` | AppArmor profile requested for sandbox agent containers. Defaults to `Unconfined`. |
|
||||
| `server.disableTls` | Run the gateway over plaintext HTTP. Use only behind a trusted transport. |
|
||||
| `server.auth.allowUnauthenticatedUsers` | Accept user-facing calls without OIDC or mTLS credentials. Use only for trusted local development or a fully trusted access proxy. |
|
||||
| `server.enableLoopbackServiceHttp` | Enable local plaintext HTTP for loopback sandbox service URLs. Defaults to `true`. |
|
||||
| `pkiInitJob.serverDnsNames` / `certManager.serverDnsNames` | Additional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain. |
|
||||
| `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect based on cluster version: clusters running Kubernetes 1.35 or later use `image-volume` (ImageVolume GA in 1.36); older clusters use `init-container`. Set explicitly to `image-volume` on Kubernetes 1.33 or 1.34 with the ImageVolume feature gate enabled, or to `init-container` to force the legacy path on any version. |
|
||||
| `supervisor.topology` | Sandbox pod topology. Refer to [Topology](/kubernetes/topology). |
|
||||
| `supervisor.sidecar.proxyUid` | Non-root UID used when sidecar process/binary-aware network policy is disabled. The default binary-aware sidecar runs as UID 0 instead. The configured UID must not match the sandbox UID. |
|
||||
| `supervisor.sandboxRuntime.networkPolicyEnforced` | Required acknowledgement that the cluster CNI enforces ingress and egress `NetworkPolicy` in sandbox namespaces. |
|
||||
| `supervisor.sandboxRuntime.boundaryPort` | Non-privileged TLS port used between paired supervisor and sandbox Pods. |
|
||||
| `upstreamProxy` | Operator-owned corporate HTTP forward proxy for policy-approved TLS egress. Refer to [Configure a Corporate Upstream Proxy](#configure-a-corporate-upstream-proxy). |
|
||||
|
||||
Use a values file for repeatable deployments:
|
||||
@@ -216,16 +217,10 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--values my-values.yaml
|
||||
```
|
||||
|
||||
The chart defaults `server.appArmorProfile` to `Unconfined` because
|
||||
runtime/default AppArmor profiles can block the supervisor's network namespace
|
||||
mount setup on AppArmor-enabled nodes. Set `server.appArmorProfile` to an empty
|
||||
string to omit the field, `RuntimeDefault` to force the runtime default, or
|
||||
`Localhost/<profile-name>` when you load and manage a localhost profile on each
|
||||
node.
|
||||
|
||||
To use private sandbox images, create a `kubernetes.io/dockerconfigjson` Secret
|
||||
in the sandbox namespace and reference its name:
|
||||
|
||||
@@ -265,13 +260,12 @@ upstreamProxy:
|
||||
key: credentials
|
||||
authAllowInsecure: true
|
||||
|
||||
supervisor:
|
||||
topology: sidecar
|
||||
```
|
||||
|
||||
Use `authAllowInsecure: true` only when you accept that Basic authentication is cleartext on the connection to an `http://` proxy. The initial release supports `http://` proxy endpoints and TLS CONNECT egress. It does not support HTTPS-to-proxy, custom corporate CA bundles, or forwarding plain HTTP egress through the proxy.
|
||||
|
||||
Proxy credentials require `sidecar` topology. It mounts the credential only into the dedicated network supervisor container. OpenShell rejects credential Secrets with `combined` topology because Kubernetes `fsGroup` volume permission handling can make a shared credential mount readable by the sandbox group.
|
||||
The credential mounts only in the separately scheduled supervisor Pod. The
|
||||
sandbox workload cannot read it through its environment or volumes.
|
||||
|
||||
## RBAC
|
||||
|
||||
@@ -306,6 +300,7 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set serviceAccount.create=false \
|
||||
--set serviceAccount.name=my-existing-sa
|
||||
```
|
||||
@@ -321,7 +316,7 @@ The gateway exposes `/healthz` for process liveness and `/readyz` for dependency
|
||||
|
||||
## Next Steps
|
||||
|
||||
- To choose between combined and sidecar sandbox pods, refer to [Topology](/kubernetes/topology).
|
||||
- Kubernetes sandboxes use separate workload and directly managed supervisor Pods; refer to [Sandbox runtime](/kubernetes/sandbox-runtime).
|
||||
- To enable automatic certificate rotation with cert-manager, refer to [Managing Certificates](/kubernetes/managing-certificates).
|
||||
- To expose the gateway externally without port-forwarding, refer to [Ingress](/kubernetes/ingress).
|
||||
- To configure OIDC or reverse-proxy authentication, refer to [Access Control](/kubernetes/access-control).
|
||||
|
||||
@@ -1,249 +0,0 @@
|
||||
---
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
title: "Kubernetes Sandbox Topology"
|
||||
sidebar-title: "Topology"
|
||||
description: "Choose between combined and sidecar supervisor topology for Kubernetes sandbox pods."
|
||||
keywords: "Generative AI, Cybersecurity, Kubernetes, Sandboxing, Sidecar, Network Policy, RuntimeClass"
|
||||
position: 2
|
||||
---
|
||||
|
||||
Kubernetes sandbox pods can run the OpenShell supervisor in `combined` or
|
||||
`sidecar` topology. Choose the topology based on which controls you need inside
|
||||
the pod and how much privilege your cluster allows on the agent container.
|
||||
|
||||
## Choose a Topology
|
||||
|
||||
The default `combined` topology preserves the full OpenShell enforcement model.
|
||||
Use `sidecar` only when you accept network-focused enforcement in exchange for a
|
||||
lower-privilege agent container.
|
||||
|
||||
| Topology | Use when | Main tradeoff |
|
||||
|---|---|---|
|
||||
| `combined` | You need OpenShell network, filesystem, and process controls in the sandbox workload. | The agent container carries the Linux capabilities the supervisor needs. |
|
||||
| `sidecar` | You need the agent container to run as non-root without added Linux capabilities, and network policy is the primary control. | Privilege-dropping and supervisor mount isolation do not run in the agent container. |
|
||||
|
||||
## Privilege Model
|
||||
|
||||
The long-running container permissions differ by topology:
|
||||
|
||||
| Topology | Pod or container | UID/GID | Privilege escalation | Capabilities | Result |
|
||||
|---|---|---|---|---|---|
|
||||
| `combined` | Agent container, which also runs the supervisor | Not forced by topology | Not explicitly disabled by the driver | Adds `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, and `SYSLOG`; adds `SETUID`, `SETGID`, and `DAC_READ_SEARCH` when user namespaces are enabled | Full supervisor controls run in the agent container. |
|
||||
| `sidecar` | Agent container, process-only supervisor (`network-only`) | `sandbox_uid:sandbox_gid` | `false` | Drops `ALL` | Agent and workload run without added Linux capabilities. |
|
||||
| `sidecar` | Network supervisor sidecar, binary-aware mode (default) | `0:sandbox_gid` | `false` | Drops `ALL`; adds `SYS_PTRACE` and `DAC_READ_SEARCH` | Root sidecar inspects cross-UID workload `/proc` entries. The nftables fence exempts UID 0, so do not inject other root containers into these pods. |
|
||||
| `sidecar` | Network supervisor sidecar, endpoint/L7-only mode | `proxyUid:sandbox_gid` | `false` | Drops `ALL` | Non-root sidecar enforces endpoint and L7 policy without matching `policy.binaries`. |
|
||||
|
||||
Short-lived setup containers still have the permissions needed to prepare the
|
||||
pod:
|
||||
|
||||
| Topology | Setup container | UID/GID | Privilege escalation | Capabilities | Purpose |
|
||||
|---|---|---|---|---|---|
|
||||
| `combined` | Supervisor install init container | `0` | Not set | Not set | Copies the supervisor binary into the agent container volume. |
|
||||
| `sidecar` | Network init container | `0` | `false` | Drops `ALL`; adds `NET_ADMIN`, `NET_RAW`, `CHOWN`, and `FOWNER` | Installs pod-local nftables rules and prepares shared sidecar state. |
|
||||
|
||||
## Combined Topology
|
||||
|
||||
Combined topology is the original Kubernetes mode and remains the default. The
|
||||
agent container starts the OpenShell supervisor, and the supervisor launches the
|
||||
workload after applying sandbox setup.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
Sandbox["agents.x-k8s.io Sandbox"]
|
||||
|
||||
subgraph Pod["Sandbox pod"]
|
||||
subgraph Agent["agent container"]
|
||||
Supervisor["OpenShell supervisor<br/>network + process + filesystem"]
|
||||
Workload["Agent workload"]
|
||||
end
|
||||
end
|
||||
|
||||
Gateway["OpenShell Gateway"]
|
||||
External["External services"]
|
||||
|
||||
Sandbox --> Pod
|
||||
Supervisor --> Workload
|
||||
Supervisor -->|"gateway callback / SSH relay"| Gateway
|
||||
Supervisor -->|"policy-enforced egress"| External
|
||||
```
|
||||
|
||||
Combined topology keeps these controls in one supervisor path:
|
||||
|
||||
- Network endpoint and L7 policy enforcement.
|
||||
- Filesystem policy enforcement.
|
||||
- Process and binary identity checks.
|
||||
- Privilege drop into the sandbox user.
|
||||
- Gateway relay, SSH sessions, exec, and file sync.
|
||||
|
||||
Because the supervisor performs network namespace setup and process/filesystem
|
||||
controls from the agent container, Kubernetes grants that container elevated
|
||||
Linux capabilities. Use this mode when you need the complete OpenShell sandbox
|
||||
contract and your cluster policy permits those capabilities.
|
||||
|
||||
## Sidecar Topology
|
||||
|
||||
Sidecar topology splits the supervisor into a network sidecar and a
|
||||
low-privilege process supervisor in the agent container.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
Sandbox["agents.x-k8s.io Sandbox"]
|
||||
|
||||
subgraph Pod["Sandbox pod"]
|
||||
Init["network init container<br/>root setup capabilities"]
|
||||
State["shared state + TLS volumes"]
|
||||
NetNS["pod network namespace"]
|
||||
|
||||
subgraph Agent["agent container"]
|
||||
ProcessSupervisor["process supervisor<br/>network-only"]
|
||||
Workload["Agent workload"]
|
||||
end
|
||||
|
||||
NetworkSidecar["network supervisor sidecar<br/>UID 0 by default"]
|
||||
SshEndpoint["abstract SSH relay socket<br/>peer-PID authenticated"]
|
||||
end
|
||||
|
||||
Gateway["OpenShell Gateway"]
|
||||
External["External services"]
|
||||
|
||||
Sandbox --> Pod
|
||||
Init -->|"installs nftables rules"| NetNS
|
||||
ProcessSupervisor --> Workload
|
||||
Workload -->|"egress redirected on loopback"| NetworkSidecar
|
||||
NetworkSidecar -->|"gateway session + relays"| Gateway
|
||||
NetworkSidecar -->|"policy-enforced egress"| External
|
||||
NetworkSidecar -->|"control socket + proxy TLS"| State
|
||||
ProcessSupervisor -->|"bootstrap + updates"| State
|
||||
ProcessSupervisor --> SshEndpoint
|
||||
NetworkSidecar -->|"SSH relay"| SshEndpoint
|
||||
NetworkSidecar --- State
|
||||
```
|
||||
|
||||
The pod contains these OpenShell-managed pieces:
|
||||
|
||||
| Component | Runs as | Purpose |
|
||||
|---|---|---|
|
||||
| Network init container | Root with setup capabilities | Installs pod-level nftables rules and prepares shared sidecar state. |
|
||||
| Network sidecar | UID 0 by default; `supervisor.sidecar.proxyUid` when binary-aware policy is disabled | Runs the proxy, enforces network policy, owns gateway authentication and the gateway session, and serves local policy/provider state over the sidecar control socket. |
|
||||
| Agent container | Resolved sandbox UID/GID | Runs the process supervisor and launches the user workload. |
|
||||
|
||||
In this topology, the agent container defaults to `runAsNonRoot: true`,
|
||||
`allowPrivilegeEscalation: false`, and `capabilities.drop: ["ALL"]`. The
|
||||
default binary-aware network sidecar runs as UID 0, drops default Linux
|
||||
capabilities, and adds `SYS_PTRACE` plus `DAC_READ_SEARCH` for cross-UID workload
|
||||
process identity resolution. Setting
|
||||
`supervisor.sidecar.processBinaryAwareNetworkPolicy=false` runs the sidecar as
|
||||
the configured non-root `proxyUid`, omits both capabilities, and downgrades
|
||||
network policy to endpoint/L7 enforcement without binary matching. The root
|
||||
init container keeps the setup capabilities needed to configure pod networking.
|
||||
|
||||
Sidecar mode preserves gateway session behavior, including SSH connectivity,
|
||||
because the network sidecar owns the gateway session and bridges relay requests
|
||||
to a Linux abstract SSH socket owned by the process supervisor. The relay
|
||||
verifies the socket peer PID against the authenticated control connection, so
|
||||
the workload cannot replace the relay endpoint. The agent container does not get a
|
||||
gateway endpoint, gateway TLS material, or the sandbox bootstrap token in the
|
||||
default sidecar path.
|
||||
|
||||
<Warning>
|
||||
Sidecar mode runs the process supervisor in `network-only` mode. OpenShell still
|
||||
enforces network endpoint and L7 policy through the sidecar, and the process
|
||||
supervisor applies Landlock filesystem policy and child seccomp filters where
|
||||
the kernel/runtime supports them. The process supervisor does not perform
|
||||
root-to-sandbox privilege dropping because Kubernetes starts the container as
|
||||
the sandbox UID/GID, and it does not perform supervisor identity mount
|
||||
isolation because gateway credentials are not mounted into the agent container.
|
||||
Sidecar pods use `shareProcessNamespace: true` so the network sidecar can
|
||||
resolve workload process and binary identity through `/proc/<entrypoint-pid>`.
|
||||
</Warning>
|
||||
|
||||
## Credential Exposure
|
||||
|
||||
Sidecar topology keeps gateway credentials in the network sidecar. The agent
|
||||
container does not mount the projected ServiceAccount token used for sandbox
|
||||
token bootstrap, does not mount the sandbox client TLS secret, and does not get
|
||||
gateway callback environment variables.
|
||||
|
||||
The network sidecar serves the policy and workload-facing provider environment
|
||||
over a Unix control socket in the shared sidecar state volume. Before launching
|
||||
the workload, the process supervisor establishes the only accepted connection.
|
||||
The sidecar validates its UID, GID, and PID with peer credentials, unlinks the
|
||||
listener, derives the SSH target from trusted configuration, and rejects later
|
||||
clients. The connection receives bootstrap state and provider-environment
|
||||
updates after settings polls. If it closes, the network sidecar exits so
|
||||
Kubernetes recreates the one-client bootstrap listener, and the process
|
||||
supervisor exits so Kubernetes terminates the workload and restarts the agent
|
||||
container. This symmetric failure behavior prevents a surviving workload from
|
||||
claiming the new control listener after an isolated sidecar restart. Future
|
||||
child processes can see refreshed provider env without giving the agent
|
||||
container gateway authentication material. This does not mutate the environment
|
||||
of the already-running workload entrypoint. Use `combined` topology when you
|
||||
need the full single-supervisor enforcement path; use additional runtime
|
||||
isolation when you need a stronger container boundary around sidecar workloads.
|
||||
|
||||
## RuntimeClass Isolation
|
||||
|
||||
Sidecar topology has been validated with Kata Containers. It does not currently
|
||||
support gVisor because sidecar mode requires pod-local nftables setup, which
|
||||
gVisor does not provide to the init container. A supported sandboxed runtime
|
||||
strengthens the container boundary while OpenShell focuses on network policy
|
||||
enforcement from the sidecar.
|
||||
|
||||
Runtime classes do not re-enable the OpenShell privilege-drop or supervisor
|
||||
mount-isolation controls that sidecar mode relaxes. Use them as an additional
|
||||
workload boundary, not as a replacement for the combined topology's full
|
||||
supervisor controls.
|
||||
|
||||
You can set a default runtime class in the Kubernetes driver configuration or
|
||||
override it per sandbox with driver config:
|
||||
|
||||
```shell
|
||||
openshell sandbox create \
|
||||
--driver-config-json '{"kubernetes":{"pod":{"runtime_class_name":"kata-containers"}}}' \
|
||||
-- claude
|
||||
```
|
||||
|
||||
## Enable Sidecar Mode
|
||||
|
||||
For direct gateway TOML configuration, set the Kubernetes driver fields:
|
||||
|
||||
```toml
|
||||
[openshell.drivers.kubernetes]
|
||||
topology = "sidecar"
|
||||
|
||||
[openshell.drivers.kubernetes.sidecar]
|
||||
proxy_uid = 1337
|
||||
```
|
||||
|
||||
`proxy_uid` configures only the relaxed endpoint/L7-only sidecar. It must be at
|
||||
least `1000` and must not match the sandbox UID. The default binary-aware mode
|
||||
runs the sidecar as UID 0 instead. The network init container exempts the
|
||||
effective sidecar UID from proxy redirection so the sidecar can reach the
|
||||
gateway.
|
||||
|
||||
When the Helm chart renders `gateway.toml`, set the equivalent chart values:
|
||||
|
||||
```yaml
|
||||
supervisor:
|
||||
topology: sidecar
|
||||
sidecar:
|
||||
proxyUid: 1337
|
||||
processBinaryAwareNetworkPolicy: true
|
||||
```
|
||||
|
||||
Leave `topology` unset, or set it to `combined`, to keep the original
|
||||
single-container supervisor path. For Helm installs, leave
|
||||
`supervisor.topology` unset or set it to `combined`.
|
||||
|
||||
Set `supervisor.sidecar.processBinaryAwareNetworkPolicy=false` only when you
|
||||
accept downgrading sidecar network policy to endpoint/L7 enforcement without
|
||||
matching `policy.binaries`. This changes the sidecar from UID 0 to `proxyUid`
|
||||
and removes its `SYS_PTRACE` and `DAC_READ_SEARCH` capabilities, which are used
|
||||
for cross-UID `/proc` inspection.
|
||||
|
||||
## Next Steps
|
||||
|
||||
- To install OpenShell on Kubernetes, refer to [Setup](/kubernetes/setup).
|
||||
- To configure gateway authentication, refer to [Access Control](/kubernetes/access-control).
|
||||
- To review the driver fields, refer to [Gateway Configuration File](/reference/gateway-config).
|
||||
@@ -168,6 +168,10 @@ The `class_uid` field identifies the event type:
|
||||
| 5019 | Device Config State Change | `CONFIG:` |
|
||||
| 6002 | Application Lifecycle | `LIFECYCLE:` |
|
||||
|
||||
HTTP Activity records include request or response details. Rejections that have
|
||||
parsed a request, including early credential-binding failures, include safe
|
||||
method-only request context and the generated response.
|
||||
|
||||
## SIEM Schema Version Compatibility
|
||||
|
||||
OpenShell emits OCSF v1.8.0 events internally, but many SIEMs only support older schema versions. The `ocsf_schema_version` setting tells the JSONL layer to downgrade events before writing, stripping fields and profiles that don't exist in the target version.
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
title: "API Errors"
|
||||
sidebar-title: "API Errors"
|
||||
description: "Inspect structured gateway failures and choose an appropriate recovery action."
|
||||
keywords: "OpenShell, SDK, gRPC, API errors, retries"
|
||||
---
|
||||
|
||||
Gateway errors include a gRPC status code and a message. Shared request validation
|
||||
and concurrency checks also return standard protobuf details in
|
||||
`grpc-status-details-bin`. Use the status code and structured fields for decisions.
|
||||
Treat messages as explanations whose text can change.
|
||||
|
||||
## Structured details
|
||||
|
||||
The SDKs decode these standard detail types while retaining the original failure.
|
||||
Older gateways and checks that have not adopted structured details can return only
|
||||
a code and message. Missing details do not change the meaning of the status code.
|
||||
|
||||
| Detail | Meaning |
|
||||
|---|---|
|
||||
| `google.rpc.BadRequest` | `field_violations` identifies rejected fields and explains each violation. Shared sandbox, exec, provider size, and workspace selector checks supply these details. |
|
||||
| `google.rpc.ErrorInfo` | `reason`, `domain`, and `metadata` identify a failure without parsing its message. Gateway reasons use the `openshell.nvidia.com` domain. |
|
||||
| `google.rpc.RetryInfo` | `retry_delay` gives a minimum delay before an otherwise safe retry. Its presence does not guarantee that a mutation has not already committed. |
|
||||
|
||||
Recognized gateway reasons include the following.
|
||||
|
||||
| Reason | Code | Recovery |
|
||||
|---|---|---|
|
||||
| `INVALID_ARGUMENT` | `INVALID_ARGUMENT` | Correct the fields listed in `BadRequest`. |
|
||||
| `RESOURCE_VERSION_CONFLICT` | `ABORTED` | Read the resource again and construct a new conditional write. `metadata.recovery` is `REFRESH_STATE`; `current_resource_version` is included when known. |
|
||||
| `PROFILE_SOURCE_UNAVAILABLE` | `UNAVAILABLE` | Retry a profile snapshot read after at least the supplied delay. |
|
||||
|
||||
## Status and retry guidance
|
||||
|
||||
A timeout or disconnected transport can occur after a mutation commits. Do not
|
||||
automatically repeat creates, credential rotation, or command execution based
|
||||
only on a transient status. Retry a mutation only when its documented operation
|
||||
contract makes the repeated request safe. A correlation ID does not provide that
|
||||
guarantee.
|
||||
|
||||
| Status | Recovery |
|
||||
|---|---|
|
||||
| `INVALID_ARGUMENT`, `OUT_OF_RANGE` | Correct the request. |
|
||||
| `UNAUTHENTICATED` | Refresh or replace credentials before a new attempt. |
|
||||
| `PERMISSION_DENIED` | Obtain the required authorization. |
|
||||
| `NOT_FOUND` | Check the resource and workspace. Follow the operation's missing-resource contract. |
|
||||
| `ALREADY_EXISTS` | Inspect the existing resource before deciding whether it satisfies the request. |
|
||||
| `FAILED_PRECONDITION` | Resolve the reported state or configuration requirement. |
|
||||
| `ABORTED` | Read fresh state before retrying a conditional operation. |
|
||||
| `UNAVAILABLE`, `RESOURCE_EXHAUSTED` | A transient condition may recover. Apply backoff and any minimum retry delay only to a retry-safe operation. |
|
||||
| `DEADLINE_EXCEEDED`, `CANCELLED` | A mutation's outcome can be unknown. Cancellation does not imply rollback. |
|
||||
| `UNIMPLEMENTED` | Check gateway and SDK version compatibility. |
|
||||
| `INTERNAL`, `UNKNOWN`, `DATA_LOSS` | Preserve the status and correlation metadata for diagnosis. Do not blindly retry mutations. |
|
||||
|
||||
## SDK access
|
||||
|
||||
Each SDK exposes decoded fields and an escape hatch for complete transport data.
|
||||
Unknown details remain available through the original error even when the SDK
|
||||
does not recognize their message type.
|
||||
|
||||
| SDK | Decoded details | Original failure |
|
||||
|---|---|---|
|
||||
| Rust | `SdkError::error_details()`, `retry_delay()` | `SdkError::grpc_status()` returns the original `tonic::Status`, including details bytes and metadata. |
|
||||
| Go | `StatusError.FieldViolations`, `ErrorInfo`, `RetryDelay`, `GRPCCode` | `StatusError.Cause` retains the gRPC error. `status.FromError` can recover its details through error wrapping. |
|
||||
| TypeScript | `SdkError.fieldViolations`, `errorInfo`, `retryDelayMs`, `connectCode` | `SdkError.cause` retains the `ConnectError`, including all details and metadata. Use `fromConnect` for raw calls. |
|
||||
| Python | `GatewayError.field_violations`, `error_info`, `retry_delay` | `raw_error` retains the gRPC exception and metadata; `raw_status` retains the parsed envelope. Use `from_grpc_error` for raw calls. |
|
||||
|
||||
Retry delays are Rust and Go durations, TypeScript milliseconds, and Python
|
||||
seconds. Absent or invalid delay details do not produce a suggested delay.
|
||||
|
||||
## Migration
|
||||
|
||||
Existing error codes and human-readable messages remain available. Python curated
|
||||
clients now raise `GatewayError`, which remains a `grpc.RpcError`; existing
|
||||
`except grpc.RpcError` handlers continue to work. `GatewayError` is not a
|
||||
`grpc.Call`. If your handler checks that interface, inspect `raw_error` before
|
||||
checking the status. The Python SDK's deletion wait and managed-sandbox cleanup
|
||||
use the original call to recognize `NOT_FOUND`; other failures still propagate.
|
||||
Rust error variants now retain
|
||||
status fields; use `..` when destructuring variants that do not need those fields.
|
||||
Go and TypeScript add typed detail fields to their existing error types.
|
||||
|
||||
No SDK automatically retries a mutation as a result of decoding these details.
|
||||
@@ -90,6 +90,8 @@ The same settings are available through environment variables:
|
||||
| Environment variable | Purpose | Default |
|
||||
|---|---|---|
|
||||
| `OPENSHELL_OIDC_ISSUER` | OIDC issuer URL. The gateway discovers `/.well-known/openid-configuration` from this URL. | None |
|
||||
| `OPENSHELL_OIDC_DANGEROUSLY_ALLOW_INSECURE_HTTP` | Development-only acknowledgement for HTTP issuers on numeric loopback addresses. HTTP hostnames and non-loopback addresses remain rejected. | `false` |
|
||||
| `OPENSHELL_OIDC_JWKS_ALLOWED_ORIGINS` | Comma-separated additional HTTPS origins trusted to serve JWKS. | Empty |
|
||||
| `OPENSHELL_OIDC_AUDIENCE` | Expected JWT `aud` claim. | `openshell-cli` |
|
||||
| `OPENSHELL_OIDC_JWKS_TTL` | JWKS cache TTL in seconds. The gateway also refreshes on an unknown key ID. Must be greater than zero. | `3600` |
|
||||
| `OPENSHELL_OIDC_ROLES_CLAIM` | Dot-separated claim path containing roles. | `realm_access.roles` |
|
||||
@@ -104,12 +106,22 @@ server:
|
||||
oidc:
|
||||
issuer: https://idp.example.com/realms/openshell
|
||||
audience: openshell-cli
|
||||
# Only needed when discovery returns a deliberately separate JWKS origin.
|
||||
jwksAllowedOrigins:
|
||||
- https://keys.example.com
|
||||
rolesClaim: realm_access.roles
|
||||
adminRole: openshell-admin
|
||||
userRole: openshell-user
|
||||
scopesClaim: ""
|
||||
```
|
||||
|
||||
The gateway requires HTTPS for OIDC discovery and JWKS retrieval, rejects
|
||||
redirects, and trusts the issuer origin for JWKS by default. Add a
|
||||
`jwksAllowedOrigins` entry only when the identity provider intentionally serves
|
||||
keys from another trusted origin. For local tests, numeric-loopback HTTP also
|
||||
requires `dangerouslyAllowInsecureHttp: true`; this setting never permits an
|
||||
HTTP hostname or non-loopback address.
|
||||
|
||||
Register an OIDC gateway with the CLI:
|
||||
|
||||
```shell
|
||||
|
||||
@@ -22,7 +22,7 @@ Gateway CLI flag > gateway OPENSHELL_* env var > TOML file > built-in defa
|
||||
|
||||
## Supervisor connections and policy recovery
|
||||
|
||||
Current supervisors open `ConnectSupervisor` before initializing gateway-owned runtime state and require a complete configuration bootstrap before the gateway accepts the session as initialized. They apply later policy, settings, middleware, and provider snapshots from that stream without configuration fetch polling. If first-start image policy discovery or image-specific baseline enrichment changes policy, the supervisor commits that repair and reconnects before installing the fresh stream bootstrap; it does not initialize from the mutation response. The gateway gives bootstrap construction the same bounded 45-second window as other snapshot builds and rejects the connection when construction fails or times out. Immediately previous supervisors retain the Stage 1 compatibility behavior: the gateway waits at most one second for an optional bootstrap, accepts the session without it when necessary, and relies on supervisor polling for configuration.
|
||||
During supervisor reconnects, the gateway waits at most one second for the optional configuration bootstrap. If a credential backend stalls, the gateway accepts the session without that bootstrap so SSH, exec, and service connections can resume. Supervisor polling continues to supply configuration.
|
||||
|
||||
On startup, the gateway repairs missing legacy policy history and skips invalid stored policies. A valid global policy can still override an invalid local policy; otherwise, that sandbox's configuration reads continue to report the validation failure.
|
||||
|
||||
@@ -100,7 +100,7 @@ future version. To migrate an existing file:
|
||||
omit Docker or Podman `sandbox_pids_limit` to use OpenShell's default limit
|
||||
of 2048, and omit Podman `health_check_interval_secs` to disable health
|
||||
checks. Explicit zero values are invalid.
|
||||
7. Remove local Docker, Podman, or VM `grpc_endpoint` when the topology-derived
|
||||
7. Remove local Docker, Podman, or VM `grpc_endpoint` when the driver-derived
|
||||
callback is correct, or retain it as an explicit override. Kubernetes raw
|
||||
TOML requires an explicit endpoint; Helm derives one from the release's
|
||||
gateway Service. New VM root filesystems use an image-provided `sandbox`
|
||||
@@ -219,13 +219,15 @@ endpoint = "http://otel-collector.observability.svc:4317"
|
||||
service_name = "openshell-gateway"
|
||||
|
||||
[openshell.gateway.oidc]
|
||||
issuer = "https://idp.example.com/realms/openshell"
|
||||
audience = "openshell-cli"
|
||||
jwks_ttl_secs = 3600 # Must be greater than zero.
|
||||
roles_claim = "realm_access.roles"
|
||||
admin_role = "openshell-admin"
|
||||
user_role = "openshell-user"
|
||||
scopes_claim = ""
|
||||
issuer = "https://idp.example.com/realms/openshell"
|
||||
audience = "openshell-cli"
|
||||
jwks_ttl_secs = 3600 # Must be greater than zero.
|
||||
jwks_allowed_origins = ["https://keys.example.com"] # Default: issuer origin only.
|
||||
dangerously_allow_insecure_http = false # Development-only numeric-loopback escape hatch.
|
||||
roles_claim = "realm_access.roles"
|
||||
admin_role = "openshell-admin"
|
||||
user_role = "openshell-user"
|
||||
scopes_claim = ""
|
||||
|
||||
[[openshell.gateway.interceptors]]
|
||||
name = "quota"
|
||||
@@ -252,10 +254,10 @@ namespace = "openshell"
|
||||
# Required in raw TOML; Helm derives this from the gateway Service.
|
||||
grpc_endpoint = "https://openshell-gateway.openshell.svc:8080"
|
||||
default_image = "ghcr.io/nvidia/openshell/sandbox:latest"
|
||||
# Defaults to the gateway version. The gateway accepts its current internal
|
||||
# supervisor protocol revision, the immediately previous revision through
|
||||
# polling compatibility, and pre-handshake supervisors for one release. Other
|
||||
# revisions reject the session. Compatibility sessions emit gateway warnings.
|
||||
# Defaults to the gateway version. The current internal supervisor protocol
|
||||
# streams authoritative configuration snapshots. The immediately previous
|
||||
# revision remains polling-compatible, and pre-handshake supervisors remain
|
||||
# compatible for one release. Other revisions reject the session.
|
||||
# supervisor_image = "ghcr.io/nvidia/openshell/supervisor:<version>"
|
||||
client_tls_secret_name = "openshell-client-tls"
|
||||
service_account_name = "openshell-sandbox"
|
||||
@@ -450,7 +452,8 @@ For Vault instead:
|
||||
credential_drivers = ["vault"]
|
||||
|
||||
[openshell.credential_drivers.vault]
|
||||
address = "http://vault.vault.svc.cluster.local:8200"
|
||||
address = "https://vault.vault.svc.cluster.local:8200"
|
||||
ca_bundle = "/etc/openshell/vault/ca.pem"
|
||||
mount = "secret"
|
||||
kv_version = "2"
|
||||
auth_method = "kubernetes"
|
||||
@@ -470,7 +473,9 @@ server:
|
||||
|
||||
For `kubernetes-secrets`, `namespace` sets where OpenShell-managed provider Secret objects are stored. When omitted, the driver uses the in-cluster ServiceAccount namespace when available, otherwise `default`. The Helm chart creates a Role granting the gateway access to all Secrets in the credential namespace because OpenShell-managed Secret names are dynamic SHA-256 hashes that cannot be restricted with `resourceNames`. Deploy credential Secrets in a dedicated namespace (`server.credentialDrivers.kubernetesSecrets.namespace`) to limit the RBAC blast radius.
|
||||
|
||||
For `vault`, `address` points at the Vault service, `mount` and `kv_version` describe the KV engine where OpenShell-managed provider secrets are stored, and `auth_method = "kubernetes"` logs in with the gateway Pod's ServiceAccount token. For local or development validation, use `auth_method = "token_file"` with `token_path = "/path/to/token"`. Do not put literal Vault tokens in TOML.
|
||||
For `vault`, `address` points at the Vault service. Non-loopback endpoints must use HTTPS; plaintext HTTP is accepted only for `localhost` or an IP loopback address during local development. Vault requests do not follow redirects, preventing credentials from being replayed to a downgraded or substituted endpoint. HTTPS uses platform trust roots by default. Set `ca_bundle` to a certificate-only PEM bundle for a private Vault CA; the bundle augments platform roots and normal hostname verification remains enabled. `mount` and `kv_version` describe the KV engine where OpenShell-managed provider secrets are stored, and `auth_method = "kubernetes"` logs in with the gateway Pod's ServiceAccount token. For local or development validation, use `auth_method = "token_file"` with `token_path = "/path/to/token"`. Do not put literal Vault tokens in TOML.
|
||||
|
||||
For Helm deployments, set `server.credentialDrivers.vault.caConfigMapName` to a ConfigMap containing the private CA bundle under the `ca.crt` key. The chart mounts that key and renders `ca_bundle` automatically.
|
||||
|
||||
Provider records that already contain inline database credentials remain readable for upgrade compatibility. New provider create/update requests still submit credential values through the normal API, but the gateway stores those values through the active credential storage path and persists only handles. Before OpenShell 0.1.0, OpenShell does not automatically migrate inline refresh material or credential handles between drivers. Reconfigure refresh grants after an upgrade. Before changing credential drivers, remove affected credentials while the original driver is still available, then select the new driver and create them again. Do not run mixed gateway versions against the same refresh records.
|
||||
|
||||
@@ -501,7 +506,8 @@ transport = "uds"
|
||||
socket_path = "/run/openshell/credential-drivers/vault.sock"
|
||||
command = "/usr/libexec/openshell/openshell-driver-vault"
|
||||
args = [
|
||||
"--address", "http://vault.vault.svc.cluster.local:8200",
|
||||
"--address", "https://vault.vault.svc.cluster.local:8200",
|
||||
"--ca-bundle", "/etc/openshell/vault/ca.pem",
|
||||
"--auth-method", "kubernetes",
|
||||
"--role", "openshell-gateway",
|
||||
]
|
||||
@@ -554,20 +560,16 @@ service_account_name = "openshell-sandbox"
|
||||
default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
|
||||
image_pull_policy = "if_not_present"
|
||||
image_pull_secrets = ["regcred"]
|
||||
# Defaults to the gateway version. The gateway accepts its current internal
|
||||
# supervisor protocol revision, the immediately previous revision through
|
||||
# polling compatibility, and pre-handshake supervisors for one release. Other
|
||||
# revisions reject the session. Compatibility sessions emit gateway warnings.
|
||||
# Defaults to the gateway version; override to pin a specific build.
|
||||
# sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:<version>"
|
||||
sandbox_runtime_image_pull_policy = "if_not_present"
|
||||
# Defaults to the gateway version. Custom builds must match the gateway's
|
||||
# internal supervisor protocol revision; mismatched peers reject the session.
|
||||
# Supervisors from releases before the handshake existed remain compatible for
|
||||
# one release.
|
||||
# supervisor_image = "ghcr.io/nvidia/openshell/supervisor:<version>"
|
||||
supervisor_image_pull_policy = "if_not_present"
|
||||
|
||||
# Use the image volume on Kubernetes >= 1.35 (GA in 1.36); switch to "init-container"
|
||||
# on older clusters or where the ImageVolume feature gate is off.
|
||||
supervisor_sideload_method = "image-volume"
|
||||
# "combined" runs the existing single supervisor container with full process,
|
||||
# filesystem, and network enforcement in the agent container. "sidecar" moves
|
||||
# pod-level network enforcement and gateway session handling into a network sidecar.
|
||||
topology = "combined"
|
||||
# Optional corporate HTTP forward proxy for policy-approved TLS egress. The
|
||||
# sandbox workload cannot select or override these settings. Only http:// proxy
|
||||
# endpoints and TLS CONNECT traffic are supported; plain HTTP egress remains
|
||||
@@ -585,9 +587,7 @@ topology = "combined"
|
||||
# Secret prevents that Pod from starting; unreadable or malformed `user:pass`
|
||||
# content is validated fail-closed by the supervisor at startup and never
|
||||
# falls back to direct egress.
|
||||
# Proxy credential Secrets require `topology = "sidecar"`. Combined topology
|
||||
# shares its credential mount with the workload and can make it readable by the
|
||||
# sandbox group through Kubernetes `fsGroup` volume permission handling.
|
||||
# Proxy credentials mount only in the separately scheduled supervisor Pod.
|
||||
# Required with a credential Secret: Basic authentication to an http:// proxy
|
||||
# is cleartext on the connection to that proxy.
|
||||
# proxy_auth_allow_insecure = true
|
||||
@@ -602,7 +602,6 @@ ssh_socket_path = "/run/openshell/ssh.sock"
|
||||
client_tls_secret_name = "openshell-client-tls"
|
||||
host_gateway_ip = "10.0.0.1"
|
||||
enable_user_namespaces = false
|
||||
app_armor_profile = "Unconfined"
|
||||
workspace_default_storage_size = "10Gi"
|
||||
# Kubernetes StorageClass for the workspace PVC. Empty (default) omits the
|
||||
# field, using the cluster's default StorageClass. Set this on clusters with no
|
||||
@@ -637,19 +636,12 @@ enabled = true
|
||||
gateway_namespace = "openshell"
|
||||
gateway_pod_selector = { "app.kubernetes.io/name" = "openshell", "app.kubernetes.io/instance" = "openshell" }
|
||||
|
||||
[openshell.drivers.kubernetes.sidecar]
|
||||
# UID used by relaxed long-running network sidecars. Strict process/binary-aware
|
||||
# sidecars run as UID 0 so Kubernetes grants the required /proc inspection
|
||||
# capabilities into the effective set. In sidecar topology the network init
|
||||
# container installs nftables rules that exempt the effective sidecar UID, so
|
||||
# this dedicated infrastructure UID must remain at least 1000 and must not
|
||||
# match the sandbox workload UID.
|
||||
proxy_uid = 1337
|
||||
# Keep process/binary-aware network policy enabled in sidecar topology. Set
|
||||
# false to run the sidecar as proxy_uid, drop the sidecar's extra /proc
|
||||
# inspection capabilities, and enforce endpoint/L7 policy without matching
|
||||
# policy.binaries.
|
||||
process_binary_aware_network_policy = true
|
||||
[openshell.drivers.kubernetes.sandbox_runtime]
|
||||
# Required acknowledgement that the cluster CNI enforces NetworkPolicy and the
|
||||
# sandbox namespaces prevent untrusted policy, pod, label, and Secret changes.
|
||||
network_policy_enforced = true
|
||||
# TLS-protected boundary listener reached only by the paired control pod.
|
||||
boundary_port = 5500
|
||||
```
|
||||
|
||||
In managed workspace mode, the Kubernetes driver copies each explicitly named
|
||||
@@ -672,12 +664,12 @@ map Windows Sandboxing provider events into the gateway's OCSF stream.
|
||||
|
||||
```toml
|
||||
[openshell]
|
||||
version = 1
|
||||
version = 2
|
||||
|
||||
[openshell.gateway]
|
||||
bind_address = "127.0.0.1:17670"
|
||||
log_level = "info"
|
||||
compute_drivers = ["mxc"]
|
||||
compute_driver = "mxc"
|
||||
|
||||
[openshell.drivers.mxc]
|
||||
wxc_exec_path = "C:\\mxc-kit\\bin\\wxc-exec.exe"
|
||||
@@ -685,10 +677,6 @@ backend = "process_container"
|
||||
default_configuration_id = "composable"
|
||||
pc_least_privilege = false
|
||||
pc_capabilities = []
|
||||
# Pattern-C governed egress. The address is a loopback seed; the driver
|
||||
# allocates a unique ephemeral port for each sandbox.
|
||||
egress_proxy = false
|
||||
egress_proxy_addr = ""
|
||||
debug = false
|
||||
etw_audit = true
|
||||
```
|
||||
@@ -698,19 +686,6 @@ administrator or belong to the Windows Performance Log Users group. Workload
|
||||
commands and working directories remain sandbox-scoped and must be supplied in
|
||||
the `mxc` driver configuration when creating a sandbox.
|
||||
|
||||
Set `egress_proxy = true` with a loopback seed such as
|
||||
`egress_proxy_addr = "127.0.0.1:18080"` to enable the Windows Pattern-C split.
|
||||
MXC redirects sandbox traffic to a sandbox-unique port on the configured
|
||||
loopback IP, where the host CONNECT proxy enforces the trimmed network policy.
|
||||
The driver exposes the proxy's public CA bundle through a sandbox-unique
|
||||
internal filesystem grant so HTTPS clients can read the injected trust path;
|
||||
the CA private key remains in gateway memory.
|
||||
Governed egress requires `process_container` until MXC supports `network.proxy`
|
||||
for isolation sessions.
|
||||
MXC rejects policies containing `network_middlewares` before launch because
|
||||
this host-proxy path does not yet receive the gateway's built-in and remote
|
||||
middleware registry.
|
||||
|
||||
The driver records executable identity in process audit events and omits raw
|
||||
command arguments because they can contain credentials or personal data. See
|
||||
[OCSF JSON Export](/observability/ocsf-json-export) for durable Windows audit
|
||||
@@ -741,19 +716,17 @@ image_pull_policy = "if_not_present"
|
||||
# Value assigned to the openshell.sandbox_namespace label on sandbox containers.
|
||||
sandbox_label = "docker-dev"
|
||||
# Optional override. When omitted, the gateway derives
|
||||
# https://host.openshell.internal:<gateway-port> for this topology.
|
||||
# https://host.openshell.internal:<gateway-port> for this driver.
|
||||
grpc_endpoint = "https://host.openshell.internal:17670"
|
||||
# Skip the image-pull-and-extract step by pointing at a locally built binary.
|
||||
supervisor_bin = "/usr/local/libexec/openshell/openshell-sandbox"
|
||||
# When supervisor_bin is omitted, Docker extracts /openshell-sandbox from this image.
|
||||
# Defaults to the gateway version. The gateway accepts its current internal
|
||||
# supervisor protocol revision, the immediately previous revision through
|
||||
# polling compatibility, and pre-handshake supervisors for one release. Other
|
||||
# revisions reject the session. Compatibility sessions emit gateway warnings.
|
||||
# Workload-side runtime. Defaults to the gateway version.
|
||||
# sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:<version>"
|
||||
# Supervisor runtime defaults to the gateway version. The current internal
|
||||
# protocol streams authoritative configuration snapshots; the immediately
|
||||
# previous revision uses polling compatibility. Pre-handshake supervisors
|
||||
# remain compatible for one release. Other revisions reject the session.
|
||||
# supervisor_image = "ghcr.io/nvidia/openshell/supervisor:<version>"
|
||||
network_name = "openshell-docker"
|
||||
host_gateway_ip = "172.17.0.1"
|
||||
ssh_socket_path = "/run/openshell/ssh.sock"
|
||||
# Unsafe operator override. Host bind mounts, including Docker local-driver
|
||||
# bind-backed volumes, expose gateway-host paths inside sandboxes and can
|
||||
# negate OpenShell isolation and filesystem controls.
|
||||
@@ -780,7 +753,7 @@ Use `sandbox_label` for Docker configurations. The legacy
|
||||
|
||||
### Podman
|
||||
|
||||
Sandboxes run as Podman containers on a user-mode bridge network. The supervisor image is mounted read-only via Podman's `type=image` mount. Configure guest mTLS paths once under `[openshell.gateway]`; the gateway validates and injects the bundle into the selected local driver.
|
||||
Each Podman sandbox uses two containers. The workload container runs `openshell-sandbox` with `network=none`; the supervisor container runs on the host network and initiates policy-approved upstream connections. A private volume carries their authenticated Unix-domain socket. Configure guest mTLS paths once under `[openshell.gateway]`; the gateway validates and injects the bundle into the selected local driver.
|
||||
|
||||
```toml
|
||||
[openshell]
|
||||
@@ -813,10 +786,13 @@ network_name = "openshell"
|
||||
# host_gateway_ip = "192.168.127.254"
|
||||
ssh_socket_path = "/run/openshell/ssh.sock"
|
||||
stop_timeout_secs = 45
|
||||
# Defaults to the gateway version. The gateway accepts its current internal
|
||||
# supervisor protocol revision, the immediately previous revision through
|
||||
# polling compatibility, and pre-handshake supervisors for one release. Other
|
||||
# revisions reject the session. Compatibility sessions emit gateway warnings.
|
||||
# Statically linked workload-side runtime. Defaults to the gateway version.
|
||||
# sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:<version>"
|
||||
# Dynamically linked supervisor runtime defaults to the gateway version. The
|
||||
# current internal protocol streams authoritative configuration snapshots; the
|
||||
# immediately previous revision uses polling compatibility. Pre-handshake
|
||||
# supervisors remain compatible for one release. Other revisions reject the
|
||||
# session.
|
||||
# supervisor_image = "ghcr.io/nvidia/openshell/supervisor:<version>"
|
||||
# Unsafe operator override. Host bind mounts, including Podman local-driver
|
||||
# bind-backed volumes, expose gateway-host paths inside sandboxes and can
|
||||
@@ -919,18 +895,17 @@ health_check_interval_secs = 10
|
||||
# explicit container-reachable TCP endpoint, for provider token exchange.
|
||||
# provider_spiffe_workload_api_socket = "/run/spire/agent.sock"
|
||||
# provider_spiffe_workload_api_socket = "tcp:169.254.1.2:8081"
|
||||
# Omit app_armor_profile to preserve Podman's runtime-selected profile.
|
||||
# Set Unconfined only when the supervisor's mount setup requires it.
|
||||
# Omit app_armor_profile to preserve Podman's runtime-selected workload profile.
|
||||
# Explicit RuntimeDefault and Localhost/<name> require Podman to report
|
||||
# AppArmor support.
|
||||
# app_armor_profile = "Unconfined"
|
||||
# app_armor_profile = "RuntimeDefault"
|
||||
```
|
||||
|
||||
Use `ssh_socket_path` for Podman configurations. The legacy
|
||||
`sandbox_ssh_socket_path` key is rejected. When `app_armor_profile` is omitted,
|
||||
OpenShell sends no override and Podman applies its runtime-selected profile.
|
||||
Set `Unconfined` explicitly only when the deployment requires the supervisor's
|
||||
mount setup to bypass that profile.
|
||||
The setting applies to the workload container; the supervisor retains Podman's
|
||||
runtime-selected profile.
|
||||
|
||||
### MicroVM
|
||||
|
||||
@@ -956,7 +931,7 @@ state_dir = "/var/lib/openshell/vm"
|
||||
driver_dir = "/usr/local/libexec/openshell"
|
||||
default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
|
||||
# Optional override. When omitted, the gateway derives
|
||||
# https://host.openshell.internal:<gateway-port> for the VM topology.
|
||||
# https://host.openshell.internal:<gateway-port> for the VM driver.
|
||||
grpc_endpoint = "https://host.openshell.internal:17670"
|
||||
# Empty falls back to default_image.
|
||||
bootstrap_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
|
||||
@@ -979,35 +954,17 @@ overlay_disk_mib = 4096
|
||||
# any present-but-invalid value is rejected at gateway startup rather than
|
||||
# degrading to a direct dial. proxy_auth_file is a path on the gateway host.
|
||||
#
|
||||
# The sandbox cannot select or override these settings. They reach the guest
|
||||
# supervisor through a protected per-sandbox argument file the driver writes
|
||||
# into the overlay upperdir on every launch, so a sandbox image cannot supply
|
||||
# its own values or disable the operator's by baking a file at that path.
|
||||
# The sandbox cannot select or override these settings. The driver passes them
|
||||
# only to the host supervisor.
|
||||
#
|
||||
# Reachability: a proxy on the corporate network needs no special address and
|
||||
# works on every VM sandbox. The guest's callback to the gateway is unaffected
|
||||
# and never traverses the proxy.
|
||||
# works on every VM sandbox.
|
||||
#
|
||||
# A proxy on the gateway host itself is reachable only from libkrun-backed
|
||||
# (non-GPU) sandboxes: their egress leaves through gvproxy, which NATs
|
||||
# 192.168.127.254 to the host's 127.0.0.1, so address it as
|
||||
# http://host.openshell.internal:<port> rather than http://127.0.0.1:<port>.
|
||||
# GPU sandboxes run on the QEMU/TAP backend, which has no such NAT —
|
||||
# host.openshell.internal resolves to the TAP host address, and the driver's
|
||||
# nftables rules let the guest reach only the gateway port on the host. A
|
||||
# gateway-host proxy URL is therefore rejected when the sandbox launches on
|
||||
# QEMU, rather than timing out on every CONNECT; give GPU sandboxes a proxy
|
||||
# address routable from the guest's masqueraded egress.
|
||||
# A proxy on the gateway host is addressed as
|
||||
# http://host.openshell.internal:<port>. The host supervisor normalizes that
|
||||
# name to host loopback for both libkrun and QEMU sandboxes.
|
||||
#
|
||||
# Because a microVM has no bind mounts or container secrets, the driver stages
|
||||
# the credential and optional CA bundle into the per-sandbox overlay disk and
|
||||
# removes them with the sandbox. The credential is root-only; the non-secret CA
|
||||
# bundle is mode 0644. Both are referenced from the protected supervisor
|
||||
# argument file by fixed guest paths, never their gateway-host paths. The
|
||||
# credential is therefore at rest in that overlay image on the gateway host —
|
||||
# the same delivery the per-sandbox gateway token already uses, and a difference
|
||||
# from the Podman secret model worth noting when choosing where to keep proxy
|
||||
# credentials.
|
||||
# Credentials and private CA material remain host-side with the supervisor.
|
||||
# https_proxy = "http://host.openshell.internal:8080"
|
||||
# no_proxy = "10.0.0.0/8,.internal.example"
|
||||
# proxy_auth_file = "/etc/openshell/secrets/proxy-auth"
|
||||
|
||||
@@ -32,6 +32,8 @@ network_middlewares: { ... }
|
||||
| `network_policies` | map | No | Dynamic | Declares which binaries can reach which network endpoints. |
|
||||
| `network_middlewares` | map | No | Dynamic | Attaches ordered middleware by destination host; each implementation's manifest selects its supported HTTP and WebSocket operations. |
|
||||
|
||||
The YAML root and each present top-level section listed as an object or map must be mappings. Each named network policy or middleware entry must also be a mapping. The OPA loader rejects the first malformed container before normalization or access preset expansion, with a structural error that excludes authored keys and values.
|
||||
|
||||
Static fields are set at sandbox creation time. Changing them requires destroying and recreating the sandbox. Dynamic fields can be updated on a running sandbox with `openshell policy update` for incremental merges or `openshell policy set` for full replacement, and take effect without restarting.
|
||||
|
||||
## Version
|
||||
@@ -193,6 +195,7 @@ Each endpoint defines a reachable destination and optional inspection rules.
|
||||
|
||||
**Validation constraints:**
|
||||
|
||||
- When present, `endpoints`, `binaries`, `rules`, and `deny_rules` must be lists of objects, even when they contain only one entry. `null` cannot replace a list or an object entry.
|
||||
- `access` and `rules` are mutually exclusive; setting both is rejected.
|
||||
- `protocol: tcp` requires a valid DNS hostname. Hostless `allowed_ips`, IP-literal hosts, trailing-dot names, and malformed DNS selectors are rejected with a policy-validation error.
|
||||
- `protocol: tcp` requires at least one port and rejects L7-only fields, including `path`, `enforcement`, `access`, `rules`, `deny_rules`, request rewriting and credential signing fields, and GraphQL, JSON-RPC, or MCP options.
|
||||
|
||||
@@ -40,36 +40,9 @@ an exited canonical process remains a terminal sandbox result. Exit code zero
|
||||
produces `Completed`; a nonzero or signal-normalized exit produces `Error`
|
||||
with the exact exit code. Driver and supervisor failures remain `Error`.
|
||||
|
||||
## Build with Selected Compute Drivers
|
||||
|
||||
Source builds of `openshell-gateway` can include any subset of the Docker,
|
||||
Podman, Kubernetes, VM, and MXC drivers. Enable the corresponding
|
||||
`compute-driver-docker`, `compute-driver-podman`, `compute-driver-kubernetes`,
|
||||
`compute-driver-vm`, or `compute-driver-mxc` Cargo features. For example, build
|
||||
a Docker-only gateway with telemetry support:
|
||||
|
||||
```shell
|
||||
cargo build --release -p openshell-gateway --no-default-features --features telemetry,compute-driver-docker
|
||||
```
|
||||
|
||||
On Windows, select only MXC with:
|
||||
|
||||
```shell
|
||||
cargo build --release -p openshell-gateway --no-default-features --features telemetry,compute-driver-mxc,bundled-z3
|
||||
```
|
||||
|
||||
The default `in-tree-compute-drivers` feature retains the full platform driver
|
||||
set, including MXC on Windows. MXC links only on Windows. The other four
|
||||
features link drivers on non-Windows platforms; on Windows they install
|
||||
registrations that report the driver as unsupported. A build with no default
|
||||
features and no driver features connects to external drivers only.
|
||||
Auto-detection probes only compiled registrations.
|
||||
To select a driver omitted from a custom build, configure its external
|
||||
`socket_path` as described below.
|
||||
|
||||
## Configure a Compute Driver
|
||||
|
||||
Configure the compute driver on the gateway. Current releases accept one driver per gateway. Set the singular `compute_driver` key in the gateway TOML file:
|
||||
Configure the compute driver on the gateway. Current releases accept one driver per gateway. Set `compute_driver` in the gateway TOML file:
|
||||
|
||||
```toml
|
||||
[openshell.gateway]
|
||||
@@ -83,15 +56,13 @@ Non-reserved names select an extension driver and require a
|
||||
|
||||
When `compute_driver` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. Docker must respond on a known API socket. Podman first probes known API sockets and then asks the `podman` CLI for the active native or machine-backed socket. The VM driver is never auto-detected; configure it explicitly with `compute_driver = "vm"` or set `OPENSHELL_COMPUTE_DRIVER=vm` in the launch environment.
|
||||
|
||||
`compute_driver` accepts exactly one scalar driver name. The legacy `compute_drivers` list is rejected by schema version 2.
|
||||
|
||||
Common gateway options:
|
||||
|
||||
| Gateway TOML option | Description |
|
||||
|---|---|
|
||||
| `compute_driver = "<driver>"` | Select the compute driver. Built-in values are `docker`, `podman`, `kubernetes`, and `vm`; custom names require `[openshell.drivers.<name>].socket_path`. |
|
||||
|
||||
Set driver-specific values such as sandbox images, callback endpoints, network names, and VM sizing in the gateway TOML file. A TLS-enabled gateway-managed Docker, Podman, or VM driver requires a complete `guest_tls_ca`, `guest_tls_cert`, and `guest_tls_key` bundle in `[openshell.gateway]`; package-managed local TLS supplies it automatically. Driver tables reject those gateway-owned fields. Kubernetes projects guest TLS through a Secret instead. See the [Gateway Configuration File](./gateway-config) reference for the full schema and migration steps.
|
||||
Set driver-specific values such as sandbox images, callback endpoints, network names, TLS material, and VM sizing in the gateway TOML file. See the [Gateway Configuration File](./gateway-config) reference for the full `[openshell.drivers.<name>]` schema.
|
||||
|
||||
Extension drivers use the same `compute_driver.proto` gRPC surface as the
|
||||
managed VM driver. For an out-of-tree driver, choose a driver name and point
|
||||
@@ -110,8 +81,8 @@ socket path. The endpoint replaces normal driver construction for that name,
|
||||
including canonical built-in names:
|
||||
|
||||
```shell
|
||||
openshell-gateway --compute-driver kyma --compute-driver-socket /run/openshell/kyma.sock
|
||||
openshell-gateway --compute-driver docker --compute-driver-socket /run/openshell/docker.sock
|
||||
openshell-gateway --drivers kyma --compute-driver-socket /run/openshell/kyma.sock
|
||||
openshell-gateway --drivers docker --compute-driver-socket /run/openshell/docker.sock
|
||||
```
|
||||
|
||||
The gateway connects to the operator-provided endpoint; it does not provision
|
||||
@@ -186,7 +157,7 @@ the gateway. If the primary listener covers that address, the gateway reuses
|
||||
it and sandbox JWT authentication restricts the supervisor to its callback RPC
|
||||
allowlist. If the primary listener is not reachable through that address, the
|
||||
gateway creates an additional callback-only listener. Use the primary endpoint
|
||||
for CLI, administrator, health, reflection, provider management, and
|
||||
for CLI, administrator, health, reflection, inference-route management, and
|
||||
HTTP requests. A `PermissionDenied` response from an additional callback-only
|
||||
listener is expected for those requests. Do not broaden the primary listener
|
||||
to `0.0.0.0` solely to make sandbox callbacks reachable.
|
||||
@@ -195,7 +166,7 @@ to `0.0.0.0` solely to make sandbox callbacks reachable.
|
||||
|
||||
[Docker](https://www.docker.com/get-started/)-backed sandboxes run as containers on the gateway host. Use Docker for local development, single-machine gateways, and hosts that already use Docker Desktop or Docker Engine.
|
||||
|
||||
The gateway talks to the Docker daemon to create sandbox containers.
|
||||
The gateway talks to the Docker daemon to create sandbox containers. Docker is also required for local image builds from directories or Dockerfiles.
|
||||
|
||||
Docker Desktop and compatible macOS runtimes route `host.openshell.internal`
|
||||
through an IPv4 host-gateway alias. The gateway reuses an IPv4 primary listener
|
||||
@@ -204,7 +175,7 @@ that already covers loopback. Otherwise, the Docker driver requests a separate
|
||||
|
||||
For maintainer-level implementation details, refer to the [Docker driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-docker/README.md).
|
||||
|
||||
Select Docker with `compute_driver = "docker"` in `[openshell.gateway]`. Configure Docker driver values such as `socket_path`, `grpc_endpoint`, `network_name`, `sandbox_label`, `supervisor_bin`, `supervisor_image`, `image_pull_policy`, `ssh_socket_path`, and `sandbox_pids_limit` in `[openshell.drivers.docker]`. When `socket_path` is unset, the driver uses the same responsive local socket selected by auto-detection. An explicitly selected Docker driver falls back to `/var/run/docker.sock` when no candidate responds.
|
||||
Select Docker with `compute_driver = "docker"` in `[openshell.gateway]`. Configure Docker driver values such as `socket_path`, `grpc_endpoint`, `network_name`, `sandbox_runtime_image`, `supervisor_image`, `image_pull_policy`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.docker]`. The sandbox runtime image contains `/openshell-sandbox`; the supervisor image contains `/openshell-supervisor`. When `socket_path` is unset, the driver uses the same responsive local socket selected by auto-detection. An explicitly selected Docker driver falls back to `/var/run/docker.sock` when no candidate responds.
|
||||
|
||||
When operating `openshell-driver-docker` as an external driver, set
|
||||
`OPENSHELL_OTLP_ENDPOINT` to export its spans. The driver continues W3C trace
|
||||
@@ -271,7 +242,7 @@ OpenShell rejects mount `source`, `target`, and Docker volume `subpath` values
|
||||
with surrounding whitespace. OpenShell also rejects mount targets that replace
|
||||
the workspace root or container root, or contain or are contained by the
|
||||
configured SSH socket or reserved `/opt/openshell`, `/etc/openshell`,
|
||||
`/etc/openshell-tls`, `/run/openshell`, `/run/openshell-sidecar`, and network
|
||||
`/etc/openshell-tls`, `/run/openshell`, and network
|
||||
namespace roots. These checks do not make host bind mounts safe.
|
||||
|
||||
## Podman Driver
|
||||
@@ -280,29 +251,11 @@ namespace roots. These checks do not make host bind mounts safe.
|
||||
|
||||
The gateway talks to the Podman API socket. The Podman driver requires Podman 5.x, cgroups v2, rootless networking, and an active Podman user socket. When `socket_path` is not set, the driver probes known socket paths, then uses the `podman` CLI to resolve the active native or machine-backed connection. It fails to start if neither method finds a socket.
|
||||
|
||||
The agent workload uses `network=none`. Its trusted supervisor companion uses Podman's host network for gateway callbacks and policy-approved upstream connections.
|
||||
|
||||
For maintainer-level implementation details, refer to the [Podman driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/README.md) and [Podman networking notes](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/NETWORKING.md).
|
||||
|
||||
Select Podman with `compute_driver = "podman"` in `[openshell.gateway]`. Configure Podman driver values such as `socket_path`, `network_name`, `supervisor_image`, `stop_timeout_secs`, `image_pull_policy`, `grpc_endpoint`, `host_gateway_ip`, `ssh_socket_path`, and `sandbox_pids_limit` in `[openshell.drivers.podman]`.
|
||||
|
||||
### macOS Podman Socket Path
|
||||
|
||||
On macOS, Homebrew-installed Podman does not create the default socket path
|
||||
that the driver probes (`~/.local/share/containers/podman/machine/podman.sock`).
|
||||
The actual API socket lives under `/var/folders/` in a path that macOS can
|
||||
rotate after a reboot.
|
||||
|
||||
If the gateway fails with `Podman socket not found; is podman machine running?`
|
||||
while `podman machine list` shows a running machine, set the
|
||||
`OPENSHELL_PODMAN_SOCKET` environment variable to the dynamic socket path:
|
||||
|
||||
```shell
|
||||
export OPENSHELL_PODMAN_SOCKET="$(podman machine inspect --format '{{.ConnectionInfo.PodmanSocket.Path}}')"
|
||||
```
|
||||
|
||||
Add this to your shell profile or gateway launch environment so it resolves
|
||||
correctly after each reboot. Alternatively, set `socket_path` in
|
||||
`[openshell.drivers.podman]` to the current path, but note that the path may
|
||||
change when macOS rotates `/var/folders/`.
|
||||
Select Podman with `compute_driver = "podman"` in `[openshell.gateway]`. Configure Podman driver values such as `socket_path`, `network_name`, `sandbox_runtime_image`, `supervisor_image`, `stop_timeout_secs`, `image_pull_policy`, `grpc_endpoint`, `host_gateway_ip`, `ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.podman]`.
|
||||
|
||||
Podman sandboxes default to a 45-second graceful stop window before Podman escalates from `SIGTERM` to `SIGKILL`. Set `stop_timeout_secs` in gateway config, or `OPENSHELL_STOP_TIMEOUT` for the standalone driver, when a local runtime needs a different teardown window.
|
||||
|
||||
@@ -316,12 +269,6 @@ stopped sandboxes alone.
|
||||
|
||||
For proxy-required networks, the Podman driver also accepts the corporate egress proxy keys `https_proxy`, `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, and `proxy_connect_by_hostname`. The supervisor chains policy-approved TLS tunnels through the proxy with HTTP CONNECT instead of dialing destinations directly. See the [Gateway Configuration File](./gateway-config) reference for the full contract, including the cleartext-credential acknowledgement and the validated-IP CONNECT behavior.
|
||||
|
||||
Podman preserves its runtime-selected AppArmor profile when
|
||||
`app_armor_profile` is omitted. Set `Unconfined` explicitly only when the
|
||||
supervisor's mount setup requires it. Explicit `RuntimeDefault` and
|
||||
`Localhost/<profile>` selections fail startup when Podman reports that AppArmor
|
||||
is unavailable.
|
||||
|
||||
On macOS with `podman machine`, the driver uses gvproxy's host-loopback IP, `192.168.127.254`, for sandbox host aliases by default. Set `host_gateway_ip` only when your Podman machine uses a non-standard host-loopback address. On Linux, an empty `host_gateway_ip` keeps Podman's `host-gateway` resolver behavior. Direct local callbacks from rootless Podman require Podman to report the pasta network helper. Slirp4netns, other helpers, and Podman versions that do not report their helper require an explicitly remote `grpc_endpoint`; otherwise the gateway fails startup rather than leaving sandbox callbacks unreachable. Rootful Podman continues to use the configured network's bridge gateway address.
|
||||
|
||||
### Podman Driver Config Mounts
|
||||
@@ -411,16 +358,10 @@ compute_driver = "vm"
|
||||
|
||||
For a launch-time override, set `OPENSHELL_COMPUTE_DRIVER=vm` in the gateway environment and restart the service.
|
||||
|
||||
Configure VM driver values such as `grpc_endpoint`, `driver_dir`, `state_dir`, `default_image`, `bootstrap_image`, `vcpus`, `mem_mib`, `overlay_disk_mib`, and `krun_log_level` in `[openshell.drivers.vm]`. The VM `state_dir` stores overlay disks, console logs, runtime state, image-rootfs cache, and the private `run/compute-driver.sock` socket. The VM socket path is managed by the gateway and is not configurable through remote endpoint settings.
|
||||
Configure VM driver values such as `grpc_endpoint`, `driver_dir`, `state_dir`, `default_image`, `bootstrap_image`, `vcpus`, `mem_mib`, `overlay_disk_mib`, `krun_log_level`, and `guest_tls_*` in `[openshell.drivers.vm]`. The VM `state_dir` stores overlay disks, console logs, runtime state, image-rootfs cache, and the private `run/compute-driver.sock` socket. The VM socket path is managed by the gateway and is not configurable through remote endpoint settings.
|
||||
|
||||
The gateway starts `openshell-driver-vm` over a private Unix socket and passes its process ID so the driver can reject unexpected local clients. The driver's standalone TCP listener is disabled unless `--allow-unauthenticated-tcp` is set for local development.
|
||||
|
||||
Scripts that invoke the experimental standalone driver directly must use
|
||||
`--grpc-endpoint` and the `--upstream-proxy*` option family. Schema v2 removes
|
||||
the previous `--openshell-endpoint`, `--https-proxy`, `--no-proxy`, and
|
||||
`--proxy-*` spellings; gateway-managed deployments do not use those options
|
||||
directly.
|
||||
|
||||
### Local image resolution
|
||||
|
||||
The VM driver resolves sandbox images from a local container engine before falling back to registry pulls. It tries Docker first, then uses the same Podman socket discovery as the Podman driver. On Linux with Podman, enable the API socket so the driver can find local images:
|
||||
@@ -429,23 +370,28 @@ The VM driver resolves sandbox images from a local container engine before falli
|
||||
systemctl --user start podman.socket
|
||||
```
|
||||
|
||||
### Host Firewall
|
||||
### Network isolation
|
||||
|
||||
The VM driver creates nftables rules on the host for each sandbox VM's TAP network interface. These rules provide NAT for VM connectivity and defense-in-depth isolation: unsolicited inbound connections to the VM are dropped, and the VM can only reach the gateway port on the host. Primary security enforcement (proxy-only egress and bypass detection) is handled by the sandbox supervisor inside the VM guest.
|
||||
|
||||
On hosts with restrictive firewalls (e.g. firewalld), the host firewall may additionally block VM traffic that the driver's rules accept. If VM sandboxes cannot reach the network, verify that the host firewall allows forwarding and input for `vmtap-*` interfaces. See the [VM driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-vm/README.md#host-side-nftables-rules) for details.
|
||||
VM sandboxes boot without a virtual NIC. `openshell-sandbox` intercepts workload
|
||||
network syscalls inside the guest and carries mediated streams over virtio-vsock
|
||||
to the host `openshell-supervisor`, which owns DNS, policy evaluation, and
|
||||
external connections. The driver does not create TAP interfaces or host
|
||||
nftables rules.
|
||||
|
||||
### Corporate Proxy Egress
|
||||
|
||||
For proxy-required networks, the VM driver accepts the same corporate egress proxy keys as the Podman driver: `https_proxy`, `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, `proxy_connect_by_hostname`, and `proxy_ca_bundle`. The in-guest supervisor chains policy-approved TLS tunnels through the proxy with HTTP CONNECT instead of dialing destinations directly.
|
||||
For proxy-required networks, the VM driver accepts the same corporate egress proxy keys as the Podman driver: `https_proxy`, `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, `proxy_connect_by_hostname`, and `proxy_ca_bundle`. The host supervisor chains policy-approved TLS tunnels through the proxy with HTTP CONNECT instead of dialing destinations directly.
|
||||
|
||||
The settings reach the guest supervisor on its command line through a per-sandbox argument file the driver writes into the overlay upperdir on every launch, so a sandbox cannot select, alter, or disable the proxy from inside the guest — including through image `ENV`, the sandbox environment, or files baked into the image at the paths the driver uses.
|
||||
The settings reach the host supervisor through driver-owned arguments, so a sandbox cannot select, alter, or disable the proxy from inside the guest.
|
||||
|
||||
A proxy on the corporate network needs no special address and works on every VM sandbox. The guest's callback to the gateway never traverses the proxy.
|
||||
|
||||
A proxy on the gateway host itself works only for libkrun-backed (non-GPU) sandboxes, whose egress leaves through gvproxy: configure `https_proxy = "http://host.openshell.internal:<port>"` rather than a `127.0.0.1` URL, because gvproxy NATs that alias to the host's `127.0.0.1`. GPU sandboxes use the QEMU/TAP backend, where `host.openshell.internal` resolves to the TAP host address and the driver's [host firewall rules](#host-firewall) allow the guest to reach only the gateway port on the host. The driver rejects a gateway-host proxy URL when a sandbox launches on QEMU instead of letting every CONNECT time out, so give GPU sandboxes a proxy address routable from the guest's masqueraded egress.
|
||||
A proxy on the gateway host works for both libkrun and QEMU sandboxes. Configure
|
||||
`https_proxy = "http://host.openshell.internal:<port>"`; the host supervisor
|
||||
normalizes that name to host loopback.
|
||||
|
||||
Because a microVM has no bind mounts or container secrets, the driver stages the credential (root-only) and the CA bundle into the per-sandbox overlay disk and removes them with the sandbox. See the [Gateway Configuration File](./gateway-config) reference for the full contract, including the cleartext-credential acknowledgement and the validated-IP CONNECT behavior.
|
||||
The credential and private CA material stay with the host supervisor rather
|
||||
than being staged into the guest. See the [Gateway Configuration File](./gateway-config) reference for the full contract, including the cleartext-credential acknowledgement and the validated-IP CONNECT behavior.
|
||||
|
||||
## Kubernetes Driver
|
||||
|
||||
@@ -462,7 +408,7 @@ owner references or use the sandbox ServiceAccount. The operator namespace
|
||||
allowlist is a trust grant, not a tenant isolation mechanism.
|
||||
</Warning>
|
||||
|
||||
Helm deployments set Kubernetes driver values through the chart. Canonical TOML places `namespace`, `service_account_name`, and `enable_user_namespaces` in `[openshell.drivers.kubernetes]`; schema version 2 rejects their historical `[openshell.gateway]` locations.
|
||||
Helm deployments set Kubernetes driver values through the chart.
|
||||
|
||||
For maintainer-level implementation details, refer to the [Kubernetes driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-kubernetes/README.md).
|
||||
|
||||
@@ -470,27 +416,25 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA
|
||||
|---|---|---|
|
||||
| `compute_driver = "kubernetes"` | Not applicable | Select the Kubernetes compute driver. |
|
||||
| `[openshell.drivers.kubernetes].namespace` | `server.sandboxNamespace` | Set the namespace for sandbox resources. The Helm chart defaults to the release namespace when left empty. |
|
||||
| `[openshell.drivers.kubernetes].service_account_name` | `sandboxServiceAccount.name` | Set the Kubernetes service account assigned to sandbox pods and accepted by the Kubernetes driver's TokenReview bootstrap path. The Helm chart creates a dedicated sandbox service account by default. |
|
||||
| `[openshell.drivers.kubernetes].enable_user_namespaces` | `server.enableUserNamespaces` | Enable Kubernetes user namespaces for sandbox pods. |
|
||||
| `service_account_name` | `sandboxServiceAccount.name` | Set the Kubernetes service account assigned to sandbox pods and accepted by the Kubernetes driver's TokenReview bootstrap path. The Helm chart creates a dedicated sandbox service account by default. |
|
||||
| `default_image` | `server.sandboxImage` | Set the default sandbox image. |
|
||||
| `image_pull_policy` | `server.sandboxImagePullPolicy` | Set the canonical sandbox pull policy: `always`, `if_not_present`, or `never`. `newer` is Podman-only. |
|
||||
| `image_pull_policy` | `server.sandboxImagePullPolicy` | Set the Kubernetes image pull policy for sandbox pods. |
|
||||
| `image_pull_secrets` | `server.sandboxImagePullSecrets` | Attach Kubernetes image-pull Secrets to sandbox pods. Managed mode copies these explicitly named Secrets from the configured source namespace into each workspace namespace. In shared and operator modes, the Secrets must already exist in the sandbox namespace. |
|
||||
| `[managed_ssh_ingress]` | `networkPolicy.enabled` | In managed mode, create an SSH ingress policy in every workspace namespace. Helm configures the gateway namespace and pod selector automatically. Operator mode leaves namespace policy management to the platform operator. |
|
||||
| `grpc_endpoint` | `server.grpcEndpoint` | Set the gateway callback endpoint reachable from sandbox pods. Raw TOML and the standalone Kubernetes driver require an explicit endpoint because the sandbox namespace does not identify the gateway Service. Helm derives it from the release's gateway Service when the value is empty. |
|
||||
| `grpc_endpoint` | `server.grpcEndpoint` | Set the gateway callback endpoint reachable from sandbox pods. |
|
||||
| `client_tls_secret_name` | `server.tls.clientTlsSecretName` | Mount sandbox client TLS materials from a Kubernetes secret. |
|
||||
| `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Override the supervisor image that provides the `openshell-sandbox` binary. The default repository with an empty tag uses the version-pinned image built into the gateway. Changing the repository uses the effective gateway image tag, while setting a tag pins that version explicitly. |
|
||||
| `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the canonical supervisor pull policy: `always`, `if_not_present`, or `never`. `newer` is Podman-only. |
|
||||
| `supervisor_sideload_method` | `supervisor.sideloadMethod` | How the supervisor binary is delivered into sandbox pods. Leave empty to auto-detect from cluster version. Set to `image-volume` to mount the supervisor OCI image directly as a volume (requires Kubernetes 1.33+ with the ImageVolume feature gate; GA in 1.36), or `init-container` to copy it through an init container on older clusters. |
|
||||
| `topology` | `supervisor.topology` | Set `combined` for the default single supervisor path, or `sidecar` to move pod-level network enforcement and the gateway session into a dedicated sidecar. |
|
||||
| `sandbox_runtime_image` | `sandboxRuntime.image.repository` / `sandboxRuntime.image.tag` | Override the image that provides `openshell-sandbox`. The default repository with an empty tag uses the version pinned into the gateway. |
|
||||
| `sandbox_runtime_image_pull_policy` | `sandboxRuntime.image.pullPolicy` | Set the Kubernetes image pull policy for the sandbox runtime image. |
|
||||
| `supervisor_image` | `supervisor.image.repository` / `supervisor.image.tag` | Override the image that provides `openshell-supervisor`. The default repository with an empty tag uses the version pinned into the gateway. |
|
||||
| `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the Kubernetes image pull policy for the supervisor image. |
|
||||
| `sandbox_runtime.network_policy_enforced` | `supervisor.sandboxRuntime.networkPolicyEnforced` | Acknowledge that the cluster CNI enforces ingress and egress `NetworkPolicy` in sandbox namespaces. This must be `true`. |
|
||||
| `sandbox_runtime.boundary_port` | `supervisor.sandboxRuntime.boundaryPort` | Set the non-privileged TLS port used between the paired supervisor and sandbox Pods. |
|
||||
| `https_proxy` | `upstreamProxy.url` | Set the operator-owned `http://host:port` corporate forward proxy used for policy-approved TLS CONNECT egress. |
|
||||
| `no_proxy` | `upstreamProxy.noProxy` | Set destinations that bypass only the corporate proxy. OpenShell policy evaluation still applies. |
|
||||
| `proxy_auth_secret_name` | `upstreamProxy.authSecret.name` | Set the existing Secret name in the sandbox namespace that contains the proxy credential. Requires `sidecar` topology. |
|
||||
| `proxy_auth_secret_key` | `upstreamProxy.authSecret.key` | Set the Secret key containing the `user:pass` credential. Requires `sidecar` topology. |
|
||||
| `proxy_auth_secret_name` | `upstreamProxy.authSecret.name` | Set the existing Secret name in the sandbox namespace that contains the proxy credential. The Secret mounts only in the supervisor Pod. |
|
||||
| `proxy_auth_secret_key` | `upstreamProxy.authSecret.key` | Set the Secret key containing the `user:pass` credential. |
|
||||
| `proxy_auth_allow_insecure` | `upstreamProxy.authAllowInsecure` | Set `true` to acknowledge that Basic authentication to an HTTP proxy is cleartext. Required with a proxy credential Secret. |
|
||||
| `proxy_connect_by_hostname` | `upstreamProxy.connectByHostname` | Send hostnames rather than validated IPs in CONNECT requests. Use only when proxy ACLs require hostname targets. |
|
||||
| `sidecar.proxy_uid` | `supervisor.sidecar.proxyUid` | Dedicated UID of at least `1000` used by the relaxed sidecar when process/binary-aware network policy is disabled. It must not match the workload UID. The default binary-aware sidecar runs as UID 0. The network init container exempts the effective sidecar UID from proxy redirection. |
|
||||
| `sidecar.process_binary_aware_network_policy` | `supervisor.sidecar.processBinaryAwareNetworkPolicy` | Keep process/binary-aware network policy enabled in `sidecar` topology. The default runs the sidecar as UID 0 with `SYS_PTRACE` and `DAC_READ_SEARCH`. Set false to run as `proxy_uid`, drop both capabilities, and enforce endpoint/L7 policy without matching `policy.binaries`. |
|
||||
| `app_armor_profile` | `server.appArmorProfile` | Set the sandbox agent container's AppArmor profile. Helm defaults this to `Unconfined` so AppArmor-enabled nodes do not block supervisor network namespace setup. Set the Helm value to an empty string to omit the field, or use `RuntimeDefault` or `Localhost/<profile-name>` for operator-managed profiles. |
|
||||
| `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. |
|
||||
| `workspace_storage_class` | `server.workspaceStorageClass` | Set the `StorageClass` for the workspace PVC. Empty (default) omits `storageClassName` and uses the cluster's default `StorageClass`. Set this on clusters with no default `StorageClass`, otherwise the workspace PVC stays `Pending` and the sandbox never starts. |
|
||||
| `sa_token_ttl_secs` | `server.sandboxJwt.k8sSaTokenTtlSecs` | Set the projected ServiceAccount token TTL used for the bootstrap token exchange. |
|
||||
@@ -502,30 +446,17 @@ remain limited to the explicitly configured TLS and image-pull Secret names.
|
||||
The driver creates copies only in gateway-owned managed namespaces. Do not
|
||||
reuse the gateway ServiceAccount for unrelated workloads.
|
||||
|
||||
In `combined` topology, the agent container carries the Linux capabilities
|
||||
needed by the supervisor for network namespace setup, Landlock filesystem
|
||||
policy, process privilege changes, and network policy enforcement. In `sidecar`
|
||||
topology, the agent container runs as the resolved sandbox UID/GID with no added
|
||||
Linux capabilities. A root init container performs the nftables setup, and the
|
||||
long-running binary-aware sidecar runs as UID 0, drops default capabilities,
|
||||
and adds `SYS_PTRACE` plus `DAC_READ_SEARCH` for workload process identity
|
||||
resolution through shared `/proc`. The
|
||||
`sidecar.process_binary_aware_network_policy = false` setting runs it as the
|
||||
configured non-root `proxy_uid`, removes both capabilities, and relaxes network
|
||||
policy to endpoint/L7 matching only. The
|
||||
network sidecar owns gateway authentication and writes local policy/provider
|
||||
state to the process supervisor over a local control socket, so the agent
|
||||
container does not mount the sandbox bootstrap token or client TLS secret in
|
||||
the default sidecar path. The provider environment is refreshed by the network
|
||||
sidecar after settings polls and streamed to the process supervisor so future
|
||||
child processes can see updated provider env without gateway access in the
|
||||
agent container.
|
||||
Sidecar mode keeps gateway session and SSH behavior. The process supervisor
|
||||
applies Landlock filesystem policy and child seccomp filters where supported,
|
||||
but it does not perform root-to-sandbox privilege dropping or supervisor
|
||||
identity mount isolation. Network policy still runs in the sidecar, and sidecar
|
||||
pods set `shareProcessNamespace: true` so the network sidecar can resolve
|
||||
process/binary identity through `/proc/<entrypoint-pid>`.
|
||||
The Kubernetes driver always places the sandbox runtime in the workload Pod and
|
||||
the supervisor in a separate, directly managed Pod. The
|
||||
workload Pod runs `openshell-sandbox` as the same non-root UID/GID as the agent
|
||||
and requests no added Linux capabilities. The supervisor Pod runs
|
||||
`openshell-supervisor`, authenticates to the gateway with a sandbox JWT, and
|
||||
owns upstream connections. Both containers disable privilege escalation, drop
|
||||
all capabilities, and use `RuntimeDefault` seccomp. The sandbox adds a nested
|
||||
seccomp user-notification filter and Landlock restrictions before it launches
|
||||
the agent. One namespace-wide, empty-egress `NetworkPolicy` is the mandatory
|
||||
outer fence for all OpenShell workload Pods. It permits supervisor Pods to
|
||||
reach sandbox listeners; TLS and JWT identity enforce the exact pairing.
|
||||
|
||||
The Kubernetes driver creates namespaced `agents.x-k8s.io` `Sandbox` resources from the Kubernetes SIG Apps [agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) project. It detects the served Sandbox API at runtime, caches the selected API version for the gateway process, and uses `v1beta1` when available before falling back to `v1alpha1`, so supported Agent Sandbox installations work without version-specific operator configuration. The Agent Sandbox controller turns those resources into sandbox pods and related storage.
|
||||
|
||||
@@ -670,7 +601,7 @@ The resolved UID/GID appear in:
|
||||
|
||||
### VM Driver
|
||||
|
||||
The VM driver preserves an image-provided `sandbox` account when `sandbox_uid` and `sandbox_gid` are omitted. Images without that account use UID/GID `1000`. Explicit values in `[openshell.drivers.vm]` override the image account. Persisted overlays retain the UID/GID recorded when they were created. An unmarked overlay recovers identity from concrete overlay or prepared-image state, an explicit override, or the current image; the driver never assigns legacy `10001:10001` without persisted evidence.
|
||||
The VM driver injects the sandbox UID into the rootfs guest's `/etc/passwd`, `/etc/group`, and `/etc/gshadow` during rootfs preparation. Default UID is `10001`; configure `sandbox_uid` in `[openshell.drivers.vm]` to use a different value.
|
||||
|
||||
### Custom Images
|
||||
|
||||
|
||||
@@ -84,12 +84,18 @@ To override the default image references, use Helm values:
|
||||
|
||||
## Kernel Requirements
|
||||
|
||||
OpenShell enforces sandbox isolation through two Linux kernel security modules:
|
||||
The sandbox boundary requires the following Linux kernel facilities, including
|
||||
when it runs inside a container or microVM:
|
||||
|
||||
| Module | Requirement | Details |
|
||||
| -------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| [Landlock LSM](https://docs.kernel.org/security/landlock.html) | Recommended | Enforces filesystem access restrictions at the kernel level. The `best_effort` compatibility mode uses the highest Landlock ABI the host kernel supports. The `hard_requirement` mode fails sandbox creation if the required ABI is unavailable. |
|
||||
| seccomp | Required | Filters dangerous system calls. Available on all modern Linux kernels (3.17+). |
|
||||
| [Landlock LSM](https://docs.kernel.org/security/landlock.html) | Required | ABI 3 or newer, introduced in Linux 6.2, with Landlock enabled. The mandatory baseline protects private channel and bootstrap files, including against truncation. A filesystem policy's `best_effort` setting never disables this baseline. |
|
||||
| seccomp | Required | Nested user-notification filters and atomic `SECCOMP_IOCTL_NOTIF_ADDFD` with `SECCOMP_ADDFD_FLAG_SEND`, usable under the runtime's existing seccomp profile without added capabilities. The sandbox actively probes these operations before admitting the workload. |
|
||||
|
||||
A kernel version alone does not establish support. A disabled Landlock LSM or a
|
||||
runtime profile that blocks the required seccomp operations causes launch to
|
||||
fail closed. An upstream Linux 6.2 or newer kernel provides the required
|
||||
Landlock ABI; distribution backports must pass the same active qualification.
|
||||
|
||||
On macOS, these kernel modules run inside the Docker Desktop Linux VM, not on the host kernel.
|
||||
|
||||
|
||||
@@ -46,6 +46,17 @@ captures the command result, and deletes the sandbox after the command exits:
|
||||
openshell sandbox create --no-keep -- sh -c 'echo done; exit 0'
|
||||
```
|
||||
|
||||
Combine `--detach` and `--no-keep` for a background workload whose lifecycle
|
||||
belongs entirely to the gateway:
|
||||
|
||||
```shell
|
||||
openshell sandbox create --detach --no-keep -- ./worker
|
||||
```
|
||||
|
||||
The create command returns after the workload is ready. The gateway keeps the
|
||||
canonical process running without a host-side attachment and deletes the
|
||||
sandbox after that process exits.
|
||||
|
||||
`--upload` cannot yet be combined with a trailing main command because uploads
|
||||
finish after the canonical process starts. Create a scratch sandbox, upload the
|
||||
files, then launch the workload with `sandbox exec`, or build the files into the
|
||||
|
||||
@@ -90,6 +90,12 @@ to forward conversation text. Malformed placeholder candidates and candidates
|
||||
longer than 4096 wire bytes are denied. Authentication headers and HTTP trailers
|
||||
retain their credential restrictions.
|
||||
|
||||
When `openshell-sandbox` and the supervisor run separately, the supervisor mediates DNS by
|
||||
hostname. DNS sender identity is unavailable because native socket writes can
|
||||
come from a process that inherited the socket or replaced its executable.
|
||||
Resolving a name does not authorize a connection: the supervisor still checks
|
||||
the destination and calling binary when the workload opens TCP traffic.
|
||||
|
||||
## Supervisor Middleware
|
||||
|
||||
Supervisor middleware can inspect, deny, or replace admitted HTTP request bodies and client WebSocket text messages before provider credentials are injected. Middleware selection is independent of the `network_policies` rule that admitted the traffic: each keyed `network_middlewares` entry matches the destination host through `endpoints.include` and `endpoints.exclude`.
|
||||
|
||||
@@ -159,14 +159,20 @@ Paths listed in `read_only` receive read-only access.
|
||||
Paths listed in `read_write` receive full access.
|
||||
All other paths are inaccessible.
|
||||
|
||||
Landlock setup runs in two phases. The parent supervisor probes the kernel ABI and opens the configured path file descriptors before forking. The child then applies the ruleset with `restrict_self()` after privilege drop. At startup, OpenShell emits the selected ABI version and the applied read-only and read-write rule counts so you can confirm what the kernel accepted.
|
||||
The sandbox installs a mandatory Landlock baseline before running agent code.
|
||||
It requires ABI 3 or newer, introduced in Linux 6.2, so denied writes include
|
||||
file truncation. The baseline hides the private channel and bootstrap root from
|
||||
the agent by granting access only to explicitly opened, permitted root children;
|
||||
it never grants the filesystem root or follows a root-level symlink when
|
||||
constructing that allowlist. Startup fails if this baseline cannot be enforced.
|
||||
The configured filesystem policy adds restrictions on top of the baseline.
|
||||
|
||||
| Aspect | Detail |
|
||||
|---|---|
|
||||
| Default | `compatibility: best_effort`. Uses the highest kernel ABI available. Missing paths are skipped. If the kernel does not support Landlock or any configured path cannot be opened, the sandbox continues without those restrictions and emits a High-severity OCSF `DetectionFinding`. |
|
||||
| What you can change | Set `compatibility: hard_requirement` to abort sandbox startup if Landlock is unavailable or any configured path cannot be opened. |
|
||||
| Risk if relaxed | On kernels without Landlock (pre-5.13), or when all paths fail to open, the sandbox runs without kernel-level filesystem restrictions. The agent can access any file the process user can access. |
|
||||
| Recommendation | Use `best_effort` for development. Use `hard_requirement` in environments where any gap in filesystem isolation is unacceptable. Treat High-severity Landlock findings as a signal to investigate the host kernel or the image. Run on Ubuntu 22.04+ or any kernel 5.13+ for Landlock support. |
|
||||
| Default | The mandatory baseline always requires Landlock ABI 3 or newer. The additional filesystem policy defaults to `compatibility: best_effort`, which may skip unavailable policy paths and emits a High-severity OCSF `DetectionFinding` when that policy cannot be fully applied. |
|
||||
| What you can change | Set `compatibility: hard_requirement` to reject startup when the additional filesystem policy cannot be applied. Neither setting relaxes the private-file baseline. |
|
||||
| Risk if relaxed | A skipped additional policy can expose otherwise permitted image or workspace files to the agent. It cannot grant access to files excluded by the mandatory baseline. |
|
||||
| Recommendation | Require Linux 6.2 or newer with Landlock enabled, or a qualified distribution backport providing ABI 3. Use `hard_requirement` where all configured filesystem restrictions must apply. Investigate High-severity Landlock findings rather than treating them as a supported degraded security mode. |
|
||||
|
||||
### Read-Only vs Read-Write Paths
|
||||
|
||||
|
||||
Reference in New Issue
Block a user