OpenShell/deploy/helm/openshell/README.md.gotmpl

# OpenShell Helm Chart

<!--
This file is generated by helm-docs.
Edit README.md.gotmpl and values.yaml, then run `mise run helm:docs`.
-->

> **Experimental** - the Kubernetes deployment path is under active development. Expect rough edges and breaking changes.

This chart deploys the OpenShell gateway into a Kubernetes cluster. It is published as an OCI artifact to GHCR at `oci://ghcr.io/nvidia/openshell/helm-chart`.

By default, this chart also creates the namespace-scoped resources needed by
sandboxes. For a shared-gateway deployment, install it with
`workspaceResources.enabled=false`, then install the
`deploy/helm/openshell-workspace` chart in every pre-provisioned workspace
namespace. The gateway and workspace releases can then be upgraded and removed
independently. Use Kubernetes `operator` workspace mode when one gateway serves
multiple pre-provisioned workspace namespaces.

## Cluster-scoped vs namespaced objects

Most objects in this chart are namespaced and land in the release namespace.
Only two are cluster-scoped:

| Object | Default name |
| --- | --- |
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |

By default the release creates both, so an install by a cluster-admin is
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
split the install in two.

A cluster-admin applies the cluster-scoped objects once per gateway
ServiceAccount, rendered from the same values the release uses:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
  --namespace openshell -f my-values.yaml \
  --set rbac.create=true \
  --set rbac.clusterScoped.create=true \
  --set agentSandbox.preflight.enabled=false \
  --show-only templates/clusterrole.yaml \
  --show-only templates/clusterrolebinding.yaml | kubectl apply -f -
```

A namespace-admin then installs and upgrades the release with cluster-scoped
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
or the equivalent `--set`:

```shell
helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
  --namespace openshell -f my-values.yaml \
  --set rbac.clusterScoped.create=false
```

The gateway ServiceAccount name and namespace do not change, so the
pre-created `ClusterRoleBinding` keeps matching the release. This works with
`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows
`serviceAccount.name`, so render the admin step with the same values.

### Which flag the installer needs

In the default `shared` workspace mode the release also creates a namespaced
sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions.
Kubernetes forbids granting permissions you do not hold, and the built-in
`admin` ClusterRole does not cover that CRD, so an installer holding only
`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not
enough and the install fails with `attempting to grant RBAC permissions not
currently held`.

| Workspace mode | Installer holds | Use |
| --- | --- | --- |
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |

`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
is needed there.

With `rbac.create=false` the cluster-admin applies the namespaced RBAC too,
adding it to the same render:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
  --namespace openshell -f my-values.yaml \
  --set rbac.create=true \
  --set rbac.clusterScoped.create=true \
  --set agentSandbox.preflight.enabled=false \
  --show-only templates/clusterrole.yaml \
  --show-only templates/clusterrolebinding.yaml \
  --show-only templates/role.yaml \
  --show-only templates/rolebinding.yaml \
  --show-only templates/peer-role.yaml | kubectl apply -f -
```

To grant the installer the sandbox permissions instead, bind it to a Role
carrying the same rules as the chart's `openshell-sandbox` Role.

The certgen hook and credential driver RBAC keep their own flags
(`pkiInitJob.enabled` and
`server.credentialDrivers.kubernetesSecrets.rbac.create`).

### Migrating an existing release

Helm deletes objects that leave a release manifest, so setting
`rbac.clusterScoped.create=false` on a release that already owns the
`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses
TokenReview until a cluster-admin re-applies them. Hand ownership over first, as
cluster-admin, so nothing is deleted:

```shell
kubectl annotate clusterrole "openshell-node-reader-<namespace>" \
  helm.sh/resource-policy=keep --overwrite
kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
  helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flag, and the cluster-admin
owns them from that point on. Fresh installs need no such step.

`rbac.clusterScoped.create` is independent of
`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change
what the `ClusterRole` contains, but they never force the namespaced release to
apply it. Re-run the cluster-admin step after changing values that affect the
`ClusterRole` rules.


## Prerequisites

> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
> but Kubernetes accepts them even if no CNI enforces them. Without enforcement,
> sandbox workloads may connect directly and bypass supervisor network policy.
> Verify CNI support before installing OpenShell.

The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with:

```shell
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/sandbox.yaml
```

The chart does not install this cluster-scoped dependency. By default, it
fails before creating gateway resources when the cluster serves neither
supported Sandbox API (`agents.x-k8s.io/v1beta1` or
`agents.x-k8s.io/v1alpha1`). Disable the check with
`agentSandbox.preflight.enabled=false` for offline `helm template` rendering,
where Helm cannot discover cluster APIs.

## Install on Kubernetes

```shell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version>
```

## Install on OpenShift

See the full [OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift) for details. Quick start:

```shell
# Precreate the openshell namespace
oc create ns openshell

# Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> -n openshell \
  --set server.disableTls=true \
  --set podSecurityContext.fsGroup=null \
  --set securityContext.runAsUser=null
```

On OpenShift 4.22+, end-to-end TLS is supported via `BackendTLSPolicy`. See the
[OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift#end-to-end-tls-openshift-422) for details.

## Available versions

| Tag | Source | Notes |
| --- | --- | --- |
| `<semver>` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway, sandbox, and supervisor image versions. Recommended for production. |
| `<semver>-pre.N` (e.g. `0.1.0-pre.3`) | A specific prerelease candidate | Immutable candidate pin that tracks images with the same exact version. |
| `0.0.0-dev` | Latest commit on `main` | Floating tag, overwritten on every push. `appVersion` is `dev`, so images resolve to the `:dev` tag. |
| `0.0.0-dev.<commit-sha>` | A specific commit on `main` | Per-commit pin. Chart version and `appVersion` both use the full 40-character commit SHA, which matches the image tag pushed by CI. |

Prerelease and `dev` tags are intended for testing changes ahead of a release. Production deployments should pin to a stable tagged release.

## Configuration

See [`values.yaml`](values.yaml) for source defaults. Selected overlays:

- [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - gateway-only configuration
- [`ci/values-cert-manager.yaml`](ci/values-cert-manager.yaml) - cert-manager integration
- [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration
- [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing
- [`ci/values-autoscaling.yaml`](ci/values-autoscaling.yaml) - CI overlay for rendering the optional gateway HorizontalPodAutoscaler
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development

### Database backend

By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the
database is backed by a per-pod PVC:

```yaml
server:
  dbUrl: "sqlite:/var/openshell/openshell.db"
```

#### External PostgreSQL

Use external PostgreSQL when the gateway should connect to a database managed
outside this chart. The OpenShell chart does not deploy a database; install
PostgreSQL separately using the chart, operator, or managed service that fits
your environment, then pass the connection URI through a Secret.

Create a Secret containing the PostgreSQL connection URI if one does not
already exist:

```bash
kubectl create secret generic my-pg-credentials -n openshell \
  --from-literal=uri="postgresql://user:pass@host:5432/dbname"
```

Then install the chart pointing at that Secret:

```bash
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
  -n openshell \
  --set workload.kind=deployment \
  --set server.externalDbSecret=my-pg-credentials
```

Use `workload.kind=deployment` for external database-backed multi-replica
gateways. `workload.kind=statefulset` is still available for single-replica
SQLite installs and for operators who explicitly need StatefulSet identity or
storage semantics.

### Credential storage

By default, the chart uses the gateway's encrypted database credential storage.
The gateway writes encrypted provider credential envelopes to the OpenShell
database. The chart creates a retained Kubernetes Secret with the shared
key-encryption key and injects that key into every gateway pod, so the same
default works for single-replica and external database-backed HA deployments.

Use `kubernetes-secrets` or `vault` instead when credentials should live in a
cluster or external secret backend. Enabling one external credential driver
disables the default credential-storage key-encryption key Secret and env injection.

#### OpenShift

Append these flags to any of the PostgreSQL commands above for OpenShift:

```
--set server.disableTls=true \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null
```

### High availability

Set `replicaCount` above `1` only with `server.externalDbSecret`; the default
SQLite database is per pod and cannot coordinate multiple gateway replicas.
The chart creates a headless peer Service for gateway-to-gateway relay traffic.
StatefulSet pods use stable pod DNS names through that headless Service.
Deployment pods advertise their pod IP with `OPENSHELL_PEER_ENDPOINT`, because
Kubernetes does not assign stable per-pod DNS names to Deployment replicas.

Gateway peer traffic uses Kubernetes ServiceAccount identity. Each gateway pod
mounts a projected, pod-bound ServiceAccount token with audience
`openshell-gateway-peer`; receiving replicas validate that token with the
Kubernetes TokenReview API, verify the live pod UID and Helm selector labels,
and authorize only the internal `PeerRelay` RPC. The chart does not create or
accept a shared gateway peer Secret.

With gateway TLS enabled, peer calls use the chart CA and client TLS Secret for
server verification and mTLS. The client verifies the stable gateway Service
DNS name while connecting directly to the owning pod. Custom TLS Secrets must
include that Service DNS name in the server certificate and provide the CA and
client credentials configured by `server.tls`.

Set `autoscaling.enabled=true` to render an `autoscaling/v2`
HorizontalPodAutoscaler for the gateway workload. The Deployment or
StatefulSet then omits `spec.replicas`, and the chart applies its
multi-replica checks to `autoscaling.maxReplicas`. CPU and memory targets
need a matching `resources.requests` entry, or a `resources.limits` entry,
which Kubernetes copies into the request. Add custom metrics from a metrics
adapter with `autoscaling.metrics`. See the
[High Availability guide](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability)
for the metrics to scale and alert on.

Enabling autoscaling on an existing release removes `spec.replicas` from the
workload in that upgrade. Kubernetes can reset the workload to one replica
until the HPA scales it back to at least `autoscaling.minReplicas`, which
disconnects supervisor sessions from the other gateway pods; they reconnect to the remaining replica. Enable autoscaling
when you install the chart, or during a maintenance window. On an existing
release, upgrade with `--reset-then-reuse-values`, which needs Helm 3.14 or
later. With an older Helm, save the values with
`helm get values <release> -o yaml > values.yaml` and upgrade with
`--reset-values -f values.yaml`. `--reuse-values` keeps the
previous chart version's defaults, which lack the autoscaling values,
including the scale-down `behavior`, when that release predates them. For background, refer to
[Migrating Deployments and StatefulSets to horizontal autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#migrating-deployments-and-statefulsets-to-horizontal-autoscaling).
Disabling autoscaling renders `spec.replicas` from `replicaCount` again,
which defaults to 1, so set `replicaCount` to the replica count you want in
that upgrade.

## Secret bootstrap

By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs`
to create the gateway's server/client mTLS Secrets and sandbox JWT signing Secret.
The Job uses the gateway image itself, so air-gapped environments only need to
mirror that one image (no separate openssl/alpine sidecar).

When `certManager.enabled=true`, cert-manager owns the TLS Secrets and the chart
runs the same hook in JWT-only mode because cert-manager does not create the
sandbox JWT signing Secret. This precedence applies even if
`pkiInitJob.enabled` remains true. Set `pkiInitJob.enabled=false` only when an
external non-cert-manager TLS source manages TLS and you pre-create the sandbox
JWT signing Secret.

## SPIFFE/SPIRE provider token grants

Set `server.providerTokenGrants.spiffe.enabled=true` to let the gateway and
sandbox supervisors use SPIFFE JWT-SVIDs for dynamic provider token grants. The
chart keeps supervisor-to-gateway authentication on gateway-minted sandbox JWTs,
mounts the SPIFFE CSI socket into the gateway pod, exports
`OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET`, and passes the socket path to
the Kubernetes driver so sandbox pods can mount the same socket.

For local development, uncomment the SPIRE Helm releases in `skaffold.yaml` and
add `ci/values-spire.yaml` to the OpenShell release values files.

The gateway verifies supervisor JWT-SVIDs with JWT bundles fetched from the
SPIFFE Workload API, so this path does not require access to the SPIRE OIDC
discovery endpoint or its TLS CA.

{{ template "chart.valuesSection" . }}
{{ template "helm-docs.versionFooter" . }}