OpenShell/docs/kubernetes/setup.mdx

---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: "Set Up OpenShell on Kubernetes"
sidebar-title: "Setup"
description: "Deploy the OpenShell gateway to a Kubernetes cluster using the official Helm chart from GHCR."
keywords: "Generative AI, Cybersecurity, Kubernetes, Helm, Gateway, Deployment, OCI, GHCR, Installation"
position: 1
---

<Warning>
Your cluster MUST use a CNI that enforces Kubernetes `NetworkPolicy` for both
ingress and egress in every sandbox namespace. OpenShell creates the policies,
but Kubernetes accepts them even when no CNI enforces them. Without enforcement,
sandbox workloads may reach the network directly and bypass supervisor policy.
Verify CNI support before installing OpenShell.
</Warning>

![OpenShell on Kubernetes: the gateway provisions a Sandbox resource, separate supervisor and workload Pods, a boundary Service, NetworkPolicy, and a workspace PVC.](../images/openshell-kubernetes-runtime.svg)

The gateway's Kubernetes compute driver creates a `Sandbox` resource and a
separate supervisor Pod. The Agent Sandbox controller creates the workload Pod
from that resource. A Service gives the supervisor a stable address for the
workload's authenticated TLS boundary, and a PVC preserves `/sandbox` across
Pod restarts. The gateway can run in the same namespace as these resources or
in a separate namespace.

The supervisor opens the boundary connection and maintains an outbound session
to the gateway. A `NetworkPolicy` allows only supervisor ingress to the
workload's boundary port and denies new workload-initiated connections. Agent
network requests return on the established channel; the supervisor evaluates
policy and opens approved upstream connections. For the trust boundaries and
request flow, see [Architecture](/about/architecture).

## Prerequisites

Make sure the following are in place before you install.

| Prerequisite | Required | Notes |
|---|---|---|
| Kubernetes 1.29+ with RBAC enabled | Yes | No additional notes. |
| CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces | Yes | Verify enforcement on your cluster. |
| Helm 3.x | Yes | No additional notes. |
| Agent Sandbox controller and CRDs | Yes | Install before the OpenShell chart. Refer to [Install Agent Sandbox](#install-agent-sandbox). |
| cert-manager | No | Refer to [Managing Certificates](/kubernetes/managing-certificates). Use cert-manager only if you prefer it over the built-in PKI job. |
| Kubernetes Gateway API | No | Refer to [Ingress](/kubernetes/ingress). Use it only for external access without port-forwarding. |

## Install Agent Sandbox

OpenShell uses the [Agent Sandbox](https://agent-sandbox.sigs.k8s.io) Kubernetes SIG project to provision sandbox pods. Install the Agent Sandbox controller and its CRDs on your cluster before installing the OpenShell Helm chart.

Apply the latest release manifest:

```shell
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/sandbox.yaml
```

This creates the `agent-sandbox-system` namespace, installs the `sandboxes.agents.x-k8s.io` CRD, and starts the controller.

The Helm chart checks for a supported Agent Sandbox API before it creates
gateway resources. This preflight is enabled by default. Disable it only for
offline `helm template` rendering, where Helm cannot discover cluster APIs:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --set agentSandbox.preflight.enabled=false
```

The chart does not install or upgrade the cluster-scoped Agent Sandbox CRDs or
controller.

<Note>
**Air-gapped clusters:** mirror the manifest above and the `registry.k8s.io/agent-sandbox/agent-sandbox-controller` image referenced inside it to your internal registry, then point the manifest's image reference at your mirror before applying. Mirror the OpenShell gateway, supervisor, and trusted sandbox runtime images, then set `global.image.registry` to the mirror. Configure the separate default workload sandbox image with `sandbox.image` when needed.
</Note>

Confirm the controller pod is running before proceeding:

```shell
kubectl -n agent-sandbox-system get pods
```

The controller pod should reach `Running` status within a few seconds. For cluster-specific setup instructions, including KinD and GKE walkthroughs, refer to the [Agent Sandbox getting started guide](https://agent-sandbox.sigs.k8s.io/docs/getting_started/).

### Upgrade Agent Sandbox

OpenShell detects the served Agent Sandbox `Sandbox` API when the Kubernetes gateway first needs it and caches that choice for the gateway process. If you upgrade Agent Sandbox in place, restart the OpenShell gateway after the Agent Sandbox controller and CRD rollout completes so the gateway can detect the served API versions again. Existing sandboxes keep running during the upgrade, and the restarted gateway can continue managing them.

## Install OpenShell

<Steps>

## Create the namespace

```shell
kubectl create namespace openshell
```

## Install the chart

Install from the OCI registry on GHCR. Replace `<version>` with the chart version you want to install.

```shell
helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell
```

To use the latest development build instead of a stable release:

```shell
helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version 0.0.0-dev \
  --namespace openshell
```

The chart automatically generates PKI secrets on first install using pre-install Helm hooks. No manual secret creation is required.

### Split gateway and workspace releases

For a platform-managed namespace, install the gateway without namespace-scoped
sandbox resources, then install the workspace chart in the sandbox namespace:

```shell
helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set workspaceResources.enabled=false \
  --set server.sandboxNamespace=app-a

helm upgrade --install openshell-workspace \
  oci://ghcr.io/nvidia/openshell/openshell-workspace \
  --version <version> \
  --namespace app-a \
  --set gateway.serviceAccount.name=openshell \
  --set gateway.serviceAccount.namespace=openshell
```

The workspace chart does not create the namespace or deploy a gateway. It owns
only the sandbox ServiceAccount, Role, RoleBinding, and NetworkPolicy in its
release namespace. For one pre-provisioned namespace, keep
`server.drivers.kubernetes.workspaceMode=shared` and set
`server.sandboxNamespace=app-a`. To map multiple workspaces to separately
provisioned namespaces, use `workspaceMode=operator`, configure exactly one of
`operatorNamespaceLabel` or `operatorNamespaceFile`, and install the workspace
chart in every allowlisted namespace. In operator mode, the workspace chart
Role is the gateway's only Secret permission grant in that namespace.

If you enable `server.drivers.kubernetes.allowDriverConfig` on the gateway,
also set `gateway.allowDriverConfig=true` on every workspace release. This
grants the namespace-scoped PVC metadata read needed to admit caller-selected
PVC mounts. Leave both values disabled when callers do not need driver config.

### Store provider credentials in Kubernetes Secrets

To store provider credentials as Kubernetes Secrets instead of in the gateway
database, enable the Kubernetes Secrets credential driver with a dedicated
namespace:

```shell
helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set server.credentialDrivers.kubernetesSecrets.enabled=true \
  --set server.credentialDrivers.kubernetesSecrets.namespace=openshell-credentials \
  --set server.credentialDrivers.kubernetesSecrets.createNamespace=true
```

The driver stores every credential in that namespace, in every workspace mode.
The gateway receives Secret permissions through a Role in that namespace only. For
the full set of options, refer to the
[gateway configuration reference](/how-it-works/gateways/configuration#credential-drivers).

## Wait for the gateway to be ready

```shell
kubectl -n openshell rollout status statefulset/openshell
```

If you set `workload.kind=deployment`, wait on the Deployment instead:

```shell
kubectl -n openshell rollout status deployment/openshell
```

## Connect to the gateway

For local evaluation, use a port-forward:

```shell
kubectl -n openshell port-forward svc/openshell 8080:8080
```

<Warning>
The port-forward is for local evaluation only. For shared environments, expose the gateway through your ingress controller or access proxy. Refer to [Ingress](/kubernetes/ingress) for an external access option.
</Warning>

## Install the TLS client bundle

The chart generates an mTLS bundle for transport security. Kubernetes deployments do not use that bundle as user authentication; configure OIDC or a trusted access proxy as described in [Access Control](/kubernetes/access-control). For local port-forwarded access, copy the generated bundle so the CLI can verify the gateway certificate:

```shell
mkdir -p ~/.config/openshell/gateways/k8s/mtls
kubectl -n openshell get secret openshell-client-tls \
  -o jsonpath='{.data.ca\.crt}'  | base64 -d > ~/.config/openshell/gateways/k8s/mtls/ca.crt
kubectl -n openshell get secret openshell-client-tls \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/k8s/mtls/tls.crt
kubectl -n openshell get secret openshell-client-tls \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/k8s/mtls/tls.key
```

The server certificate SANs include `localhost` and `127.0.0.1`, so hostname verification passes over the port-forward without extra flags.

## Register with the CLI

In another terminal, register the gateway with the user authentication mode you configured and verify it is reachable. For example, with OIDC:

```shell
openshell gateway add https://127.0.0.1:8080 --local --name k8s \
  --oidc-issuer https://your-idp.example.com/realms/openshell \
  --oidc-client-id openshell-cli
openshell status
```

</Steps>

## Configure Chart Values

The most commonly changed values are:

| Value | Purpose |
|---|---|
| `global.image.registry` / `global.image.tag` / `global.image.pullPolicy` | Shared registry, tag, and pull policy for the gateway, supervisor, and trusted sandbox runtime. Individual image settings take precedence. |
| `gateway.image.registry` / `gateway.image.repository` / `gateway.image.tag` / `gateway.image.digest` | Gateway container image. Defaults to `ghcr.io/nvidia/openshell/gateway:latest`. |
| `replicaCount` | Number of gateway replicas. Values above `1` require shared PostgreSQL through `server.externalDbSecret`. |
| `workload.kind` | Gateway workload controller. Use `statefulset` for SQLite or `deployment` with `server.externalDbSecret`. |
| `workload.allowMultiReplicaStatefulSet` | Allow `replicaCount > 1` with `workload.kind=statefulset`. Prefer Deployment for external database-backed multi-replica gateways. |
| `server.sandboxNamespace` | Namespace where sandbox pods are created. Defaults to the Helm release namespace when left empty. |
| `workspaceResources.enabled` | Create namespace-scoped sandbox prerequisites from the gateway chart. Disable when installing the workspace chart separately. |
| `server.externalDbSecret` | Secret containing a PostgreSQL connection URI in the `uri` key. Use when the database is managed outside the chart. |
| `server.telemetryEnabled` | Enable anonymous OpenShell telemetry from the gateway and its sandbox supervisors. Set to `false` to opt out. |
| `sandbox.image.repository` / `sandbox.image.tag` / `sandbox.image.digest` | Default sandbox image used when a sandbox does not specify one. |
| `sandboxRuntime.image.registry` / `sandboxRuntime.image.repository` / `sandboxRuntime.image.tag` / `sandboxRuntime.image.digest` | Trusted workload-side image that provides the `openshell-sandbox` binary. A digest takes precedence over the tag. |
| `supervisor.image.registry` / `supervisor.image.repository` / `supervisor.image.tag` / `supervisor.image.digest` | Trusted control-side supervisor image. |
| `server.sandboxImagePullSecrets` | Image pull secrets attached to sandbox pods. Referenced Secrets must exist in the sandbox namespace. |
| `server.grpcEndpoint` | Endpoint that sandbox supervisors use to call back to the gateway. Must be reachable from inside the cluster. |
| `server.disableTls` | Run the gateway over plaintext HTTP. Use only behind a trusted transport. |
| `server.auth.allowUnauthenticatedUsers` | Accept user-facing calls without OIDC or mTLS credentials. Use only for trusted local development or a fully trusted access proxy. |
| `server.enableLoopbackServiceHttp` | Enable local plaintext HTTP for loopback sandbox service URLs. Defaults to `true`. |
| `pkiInitJob.serverDnsNames` / `certManager.serverDnsNames` | Additional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain. |
| `supervisor.sandboxRuntime.boundaryPort` | Non-privileged TLS port used between paired supervisor and sandbox Pods. |
| `upstreamProxy` | Operator-owned corporate HTTP forward proxy for policy-approved TLS egress. Refer to [Configure a Corporate Upstream Proxy](#configure-a-corporate-upstream-proxy). |

Use a values file for repeatable deployments:

```shell
helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --values my-values.yaml
```

To use private sandbox images, create a `kubernetes.io/dockerconfigjson` Secret
in the sandbox namespace and reference its name:

```shell
kubectl -n openshell create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username="$REGISTRY_USER" \
  --docker-password="$REGISTRY_TOKEN"
```

```yaml
server:
  sandboxImage: registry.example.com/team/openshell-sandbox:latest
  sandboxImagePullSecrets:
    - name: regcred
```

## Configure a Corporate Upstream Proxy

Configure a corporate forward proxy when sandbox TLS egress cannot dial the Internet directly. OpenShell evaluates policy and SSRF checks before it opens an HTTP CONNECT tunnel through the proxy. The proxy URL is operator-owned configuration. Sandbox environment variables cannot select, replace, or bypass it.

Create the credential Secret in the sandbox namespace when the proxy requires Basic authentication. The Secret value uses the `user:pass` form.

```shell
kubectl -n openshell create secret generic corporate-proxy-auth \
  --from-literal=credentials="$PROXY_USER:$PROXY_PASSWORD"
```

Add the proxy settings to your Helm values file. Replace the DNS suffixes and CIDRs in `noProxy` with values for your cluster. `noProxy` bypasses only the corporate proxy. OpenShell policy evaluation still applies.

```yaml
upstreamProxy:
  url: http://proxy.corp.example:8080
  noProxy: .svc,.svc.cluster.local,10.96.0.0/12,10.244.0.0/16
  authSecret:
    name: corporate-proxy-auth
    key: credentials
  authAllowInsecure: true

```

Use `authAllowInsecure: true` only when you accept that Basic authentication is cleartext on the connection to an `http://` proxy. The initial release supports `http://` proxy endpoints and TLS CONNECT egress. It does not support HTTPS-to-proxy, custom corporate CA bundles, or forwarding plain HTTP egress through the proxy.

The credential mounts only in the separately scheduled supervisor Pod. The
sandbox workload cannot read it through its environment or volumes.

## RBAC

The chart creates the following RBAC resources in the release namespace:

| Resource | Scope | Name |
|---|---|---|
| ServiceAccount | Namespace | `openshell` |
| ServiceAccount | Namespace | `openshell-sandbox` (for sandbox pods) |
| Role + RoleBinding | Namespace | `openshell-sandbox` |
| ClusterRole + ClusterRoleBinding | Cluster | `openshell-node-reader-<release namespace>` |

Every other object the chart creates is namespaced. The `ClusterRole` and
`ClusterRoleBinding` in the last row are the only cluster-scoped objects.

When the Kubernetes Secrets credential driver is enabled, the chart also creates
an `openshell-credential-secrets` Role and RoleBinding in the credential
namespace. That Role grants `get`, `create`, `patch`, and `delete` on all Secrets
in the credential namespace, and is the gateway's only permission grant for provider
credential Secrets.

In managed and operator workspace modes, the chart creates an
`openshell-workspace-secret-source` Role and RoleBinding in the sandbox
namespace. That Role grants `get` on only the gateway client TLS Secret and, in
managed mode, the configured image-pull Secrets. The gateway stages their
contents into each sandbox runtime generation's Secrets in the workspace
namespace.

The namespaced Role covers sandbox lifecycle and identity:

| API Group | Resource | Verbs |
|---|---|---|
| `agents.x-k8s.io` | `sandboxes`, `sandboxes/status` | create, delete, get, list, patch, update, watch |
| `""` | `events` | get, list, watch |
| `""` | `pods` | get |

The ClusterRole grants node inspection and token validation:

| API Group | Resource | Verbs |
|---|---|---|
| `authentication.k8s.io` | `tokenreviews` | create |
| `""` | `nodes` | get, list, watch |

To use an existing ServiceAccount instead of creating one, set `serviceAccount.create=false` and supply its name:

```shell
helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  --set serviceAccount.create=false \
  --set serviceAccount.name=my-existing-sa
```

The ServiceAccount must already have the Role and ClusterRole bindings described above.

### Installing without cluster-admin

By default the release creates the `ClusterRole` and `ClusterRoleBinding`, so
the installer needs cluster-scoped permissions. When cluster-scoped RBAC is
owned by a different team, or the installer is a namespace-admin GitOps
controller, split the install into a cluster-admin step and a namespace-admin
step.

A cluster-admin applies the cluster-scoped objects once per gateway
ServiceAccount. Render them from the same values the release uses, so the rules
match the configured workspace mode and credential driver:

```shell
helm template openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  -f my-values.yaml \
  --set rbac.create=true \
  --set rbac.clusterScoped.create=true \
  --set agentSandbox.preflight.enabled=false \
  --show-only templates/clusterrole.yaml \
  --show-only templates/clusterrolebinding.yaml | kubectl apply -f -
```

A namespace-admin then installs and upgrades the chart with cluster-scoped
objects omitted. The release renders only namespaced objects and needs no
permission on `clusterroles` or `clusterrolebindings`:

```shell
helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  -f my-values.yaml \
  --set rbac.clusterScoped.create=false
```

The gateway ServiceAccount name and namespace are unchanged by this flag, so
the pre-created `ClusterRoleBinding` still binds the ServiceAccount the
namespaced release creates. The same holds with `serviceAccount.create=false`:
the binding subject follows `serviceAccount.name`, so pass the same values to
both steps.

#### Migrating an existing release

Helm deletes objects that leave a release manifest, so setting
`rbac.clusterScoped.create=false` on a release that already owns the
`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses
TokenReview until a cluster-admin re-applies them. Hand ownership over first, as
cluster-admin, so nothing is deleted:

```shell
kubectl annotate clusterrole "openshell-node-reader-<namespace>" \
  helm.sh/resource-policy=keep --overwrite
kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
  helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flag, and the cluster-admin
owns them from that point on. Fresh installs need no such step.

`rbac.clusterScoped.create` is independent of the workspace mode. `managed` and
`operator` modes change what the `ClusterRole` grants, but they never require a
namespace-admin to apply it. Re-run the cluster-admin step after changing any
value that affects the `ClusterRole` rules.

Set `rbac.clusterScoped.clusterRoleName` and
`rbac.clusterScoped.clusterRoleBindingName` when the cluster-admin owns the
naming.

#### Which flag the installer needs

In the default `shared` workspace mode the release also creates a namespaced
sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions.
Kubernetes forbids granting permissions you do not hold, and the built-in
`admin` ClusterRole does not cover that CRD, so an installer holding only
`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not
enough, and the install fails with `attempting to grant RBAC permissions not
currently held`.

| Workspace mode | Installer holds | Use |
|---|---|---|
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |

`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
is needed there.

With `rbac.create=false`, the cluster-admin applies the namespaced RBAC in the
same step by adding it to the render:

```shell
helm template openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version <version> \
  --namespace openshell \
  -f my-values.yaml \
  --set rbac.create=true \
  --set rbac.clusterScoped.create=true \
  --set agentSandbox.preflight.enabled=false \
  --show-only templates/clusterrole.yaml \
  --show-only templates/clusterrolebinding.yaml \
  --show-only templates/role.yaml \
  --show-only templates/rolebinding.yaml \
  --show-only templates/peer-role.yaml | kubectl apply -f -
```

To grant the installer the sandbox permissions instead, bind it to a Role
carrying the same rules as the chart's `openshell-sandbox` Role.

## Probes

The gateway exposes `/healthz` for process liveness and `/readyz` for dependency-aware readiness on the health port. The Helm chart wires both into Kubernetes probes:

- `startupProbe` and `livenessProbe` use `/healthz`.
- `readinessProbe` uses `/readyz`, which reflects the latest result of an in-process background database check.

Each sandbox supervisor Pod uses a `tcpSocket` readiness probe on port 5501. The supervisor accepts connections on that port only while its gateway session is ready. If you add a default-deny ingress policy to sandbox namespaces, allow TCP port 5501 from the node so kubelet can reach supervisor Pods.

## Next Steps

- To run multiple gateway replicas, refer to [High Availability](/kubernetes/high-availability).
- To enable automatic certificate rotation with cert-manager, refer to [Managing Certificates](/kubernetes/managing-certificates).
- To expose the gateway externally without port-forwarding, refer to [Ingress](/kubernetes/ingress).
- To configure OIDC or reverse-proxy authentication, refer to [Access Control](/kubernetes/access-control).
- To create your first sandbox, refer to [Manage Sandboxes](/how-it-works/sandboxes/overview).