OpenShell/deploy/helm/openshell/README.md.gotmpl

# OpenShell Helm Chart

<!--
This file is generated by helm-docs.
Edit README.md.gotmpl and values.yaml, then run `mise run helm:docs`.
-->

> **Experimental** - the Kubernetes deployment path is under active development. Expect rough edges and breaking changes.

This chart deploys the OpenShell gateway into a Kubernetes cluster. It is published as an OCI artifact to GHCR at `oci://ghcr.io/nvidia/openshell/helm-chart`.

By default, this chart also creates the namespace-scoped resources needed by
sandboxes. For a shared-gateway deployment, install it with
`workspaceResources.enabled=false`, then install the
`deploy/helm/openshell-workspace` chart in every pre-provisioned workspace
namespace. The gateway and workspace releases can then be upgraded and removed
independently. Use Kubernetes `operator` workspace mode when one gateway serves
multiple pre-provisioned workspace namespaces.

## Certificate hook metadata

Use `pkiInitJob.podLabels` and `pkiInitJob.podAnnotations` to add metadata to the
Pod templates of the `<fullname>-certgen` and `<fullname>-certgen-backend-ca`
Jobs. These settings apply to both hooks when enabled and do not affect gateway
or sandbox Pods. Chart-owned Pod labels take precedence when custom keys
conflict.

For example, disable Istio sidecar injection for the hook Pods in an injected
namespace:

```yaml
pkiInitJob:
  podAnnotations:
    sidecar.istio.io/inject: "false"
```

## Cluster-scoped vs namespaced objects

Most objects in this chart are namespaced and land in the release namespace.
Only two are cluster-scoped:

| Object | Default name |
| --- | --- |
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |

By default the release creates both, so an install by a cluster-admin is
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
split the install in two.

A cluster-admin applies the cluster-scoped objects once per gateway
ServiceAccount, rendered from the same values the release uses:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
  --namespace openshell -f my-values.yaml \
  --set rbac.create=true \
  --set rbac.clusterScoped.create=true \
  --set agentSandbox.preflight.enabled=false \
  --show-only templates/clusterrole.yaml \
  --show-only templates/clusterrolebinding.yaml | kubectl apply -f -
```

A namespace-admin then installs and upgrades the release with cluster-scoped
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
or the equivalent `--set`:

```shell
helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
  --namespace openshell -f my-values.yaml \
  --set rbac.clusterScoped.create=false
```

The gateway ServiceAccount name and namespace do not change, so the
pre-created `ClusterRoleBinding` keeps matching the release. This works with
`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows
`serviceAccount.name`, so render the admin step with the same values.

### Which flag the installer needs

In the default `shared` workspace mode the release also creates a namespaced
sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions.
Kubernetes forbids granting permissions you do not hold, and the built-in
`admin` ClusterRole does not cover that CRD, so an installer holding only
`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not
enough and the install fails with `attempting to grant RBAC permissions not
currently held`.

| Workspace mode | Installer holds | Use |
| --- | --- | --- |
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |

`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
is needed there.

With `rbac.create=false` the cluster-admin applies the namespaced RBAC too,
adding it to the same render:

```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
  --namespace openshell -f my-values.yaml \
  --set rbac.create=true \
  --set rbac.clusterScoped.create=true \
  --set agentSandbox.preflight.enabled=false \
  --show-only templates/clusterrole.yaml \
  --show-only templates/clusterrolebinding.yaml \
  --show-only templates/role.yaml \
  --show-only templates/rolebinding.yaml \
  --show-only templates/peer-role.yaml | kubectl apply -f -
```

To grant the installer the sandbox permissions instead, bind it to a Role
carrying the same rules as the chart's `openshell-sandbox` Role.

The certgen hook and credential driver RBAC keep their own flags
(`pkiInitJob.enabled` and
`server.credentialDrivers.kubernetesSecrets.rbac.create`).

### Migrating an existing release

Helm deletes objects that leave a release manifest, so setting
`rbac.clusterScoped.create=false` on a release that already owns the
`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses
TokenReview until a cluster-admin re-applies them. Hand ownership over first, as
cluster-admin, so nothing is deleted:

```shell
kubectl annotate clusterrole "openshell-node-reader-<namespace>" \
  helm.sh/resource-policy=keep --overwrite
kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
  helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flag, and the cluster-admin
owns them from that point on. Fresh installs need no such step.

`rbac.clusterScoped.create` is independent of
`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change
what the `ClusterRole` contains, but they never force the namespaced release to
apply it. Re-run the cluster-admin step after changing values that affect the
`ClusterRole` rules.


## Prerequisites

> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
> but Kubernetes accepts them even if no CNI enforces them. Without enforcement,
> sandbox workloads may connect directly and bypass supervisor network policy.
> Verify CNI support before installing OpenShell.

The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with:

```shell
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/sandbox.yaml
```

The chart does not install this cluster-scoped dependency. By default, it
fails before creating gateway resources when the cluster serves neither
supported Sandbox API (`agents.x-k8s.io/v1beta1` or
`agents.x-k8s.io/v1alpha1`). Disable the check with
`agentSandbox.preflight.enabled=false` for offline `helm template` rendering,
where Helm cannot discover cluster APIs.

## Install on Kubernetes

```shell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version>
```

## Install on OpenShift

See the full [OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift) for details. Quick start:

```shell
# Precreate the openshell namespace
oc create ns openshell

# Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> -n openshell \
  --set server.disableTls=true \
  --set podSecurityContext.fsGroup=null \
  --set securityContext.runAsUser=null
```

On OpenShift 4.22+, end-to-end TLS is supported via `BackendTLSPolicy`. See the
[OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift#end-to-end-tls-openshift-422) for details.

## Available versions

| Tag | Source | Notes |
| --- | --- | --- |
| `<semver>` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway, sandbox, and supervisor image versions. Recommended for production. |
| `<semver>-pre.N` (e.g. `0.1.0-pre.3`) | A specific prerelease candidate | Immutable candidate pin that tracks images with the same exact version. |
| `0.0.0-dev` | Latest commit on `main` | Floating tag, overwritten on every push. `appVersion` is `dev`, so images resolve to the `:dev` tag. |
| `0.0.0-dev.<commit-sha>` | A specific commit on `main` | Per-commit pin. Chart version and `appVersion` both use the full 40-character commit SHA, which matches the image tag pushed by CI. |

Prerelease and `dev` tags are intended for testing changes ahead of a release. Production deployments should pin to a stable tagged release.

## Configuration

See [`values.yaml`](values.yaml) for source defaults. Selected overlays:

- [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - gateway-only configuration
- [`ci/values-cert-manager.yaml`](ci/values-cert-manager.yaml) - cert-manager integration
- [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration
- [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing
- [`ci/values-autoscaling.yaml`](ci/values-autoscaling.yaml) - CI overlay for rendering the optional gateway HorizontalPodAutoscaler
- [`ci/values-supervisor-autoscaling.yaml`](ci/values-supervisor-autoscaling.yaml) - CI overlay for rendering the optional shared supervisor memory HorizontalPodAutoscaler
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development

### Additional destination CAs

Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach
TLS destinations signed by a private CA:

```shell
kubectl create configmap private-destination-ca -n openshell \
  --from-file=ca.crt=/path/to/ca.crt
```

Then set the source ConfigMap name:

```yaml
supervisor:
  network:
    additionalCaConfigMapName: private-destination-ca
```

The chart mounts the source into the gateway and renders
`[openshell.supervisor.network].additional_ca_bundle`. The gateway validates
and normalizes the one certificate-only PEM bundle at startup. The Kubernetes
driver creates a separate managed ConfigMap in each target sandbox namespace
and mounts it only in the dedicated supervisor Pod at
`/etc/openshell/tls/additional/ca-bundle.pem`. That supervisor receives the
path through `--additional-ca-bundle` and folds the bundle into system roots.
The configured roots augment default destination trust. They do not configure
the corporate proxy, OIDC, or gateway callback mTLS trust.

Changing only the source ConfigMap does not roll or reload the gateway. Restart
the gateway, then recreate or restart affected sandboxes to use the new roots.
Removing `additionalCaConfigMapName` removes the global TOML setting; newly
created or recreated sandbox pods then have no additional-CA volume or mount.
Existing running sandbox supervisors retain their startup trust until they are
restarted or recreated.

The driver creates one immutable, content-addressed ConfigMap generation in
each target namespace. Names begin with
`openshell-additional-ca-` and include DNS-safe hashes of the effective
gateway ID and normalized bundle generation. Existing objects are accepted only
when their ownership labels, generation annotation, immutability, and exact
`ca.crt` data match; the driver never patches or updates them.

Kubernetes cannot restrict `create` by `resourceNames`, and `resourceNames`
cannot express the digest-bearing name prefix needed for `get`. Enabling this
feature therefore gives the gateway/driver service account namespace-wide
ConfigMap `get` and `create` permissions in shared mode. In managed/operator
modes, the chart's ClusterRoleBinding makes those permissions cluster-wide for
the bound service account. The chart does not grant ConfigMap `list`, `watch`,
`patch`, `update`, or `delete`. Use a dedicated gateway service account and
namespace-scoped bindings when this boundary matters. If you render the TOML
setting outside this Helm value, grant `get` and `create` in every target
sandbox namespace.

Disabling the setting intentionally does not delete driver-managed generations:
the gateway has no ConfigMap list/delete permissions, and an older object may
still be referenced by a running or stopped sandbox. An operator with separate
list/delete authority can inventory the gateway's generations by label:

```shell
kubectl -n <sandbox-namespace> get configmap \
  -l 'openshell.ai/managed-by=openshell,openshell.ai/gateway-id=<effective-gateway-id>'
```

Before deleting a named generation, verify that no Sandbox resource or pod
volume in that namespace references it. Delete only the confirmed unreferenced
name; removing a referenced generation can make a later pod start or reschedule
fail closed.

A configured CA can authenticate every policy-permitted TLS endpoint with a
matching certificate chain and hostname. Scope private CAs accordingly.

### Database backend

By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the
database is backed by a per-pod PVC:

```yaml
server:
  dbUrl: "sqlite:/var/openshell/openshell.db"
```

#### Data PVC storage

Size the gateway data claim, `openshell-data`, and select its StorageClass with
`server.dataStorageSize` and `server.dataStorageClass`. Omitting both
requests `1Gi` from the cluster's default StorageClass. Set
`server.dataStorageClass` on clusters with no default StorageClass,
otherwise the gateway PVC stays Pending. Both values are ignored under
`workload.kind=deployment`, which mounts no PVC.

```yaml
server:
  dataStorageSize: 20Gi
  dataStorageClass: fast-ssd
```

`volumeClaimTemplates` is immutable, so set both at install time. To grow an
existing claim, expand it in place when its StorageClass allows expansion, then
recreate the StatefulSet so its template matches:

```shell
kubectl -n openshell patch pvc openshell-data-openshell-0 \
  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
kubectl -n openshell delete statefulset openshell --cascade=orphan
helm upgrade openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <installed-version> \
  --reuse-values -n openshell --set server.dataStorageSize=20Gi
```

A recreated StatefulSet re-adopts existing claims unchanged, so changing
StorageClass means copying the data to a new claim.

#### External PostgreSQL

Use external PostgreSQL when the gateway should connect to a database managed
outside this chart. The OpenShell chart does not deploy a database; install
PostgreSQL separately using the chart, operator, or managed service that fits
your environment, then pass the connection URI through a Secret.

Create a Secret containing the PostgreSQL connection URI if one does not
already exist:

```bash
kubectl create secret generic my-pg-credentials -n openshell \
  --from-literal=uri="postgresql://user:pass@host:5432/dbname"
```

Then install the chart pointing at that Secret:

```bash
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
  -n openshell \
  --set workload.kind=deployment \
  --set server.externalDbSecret=my-pg-credentials
```

Use `workload.kind=deployment` for external database-backed multi-replica
gateways. `workload.kind=statefulset` is still available for single-replica
SQLite installs and for operators who explicitly need StatefulSet identity or
storage semantics.

### Credential storage

By default, the chart uses the gateway's encrypted database credential storage.
The gateway writes encrypted provider credential envelopes to the OpenShell
database. The chart creates a retained Kubernetes Secret with the shared
key-encryption key and injects that key into every gateway pod, so the same
default works for single-replica and external database-backed HA deployments.

Use `gatewayConfig` to select `kubernetes-secrets` or `vault` when credentials
should live in a cluster or external secret backend. Selecting an external
credential driver disables the default credential-storage key-encryption-key
Secret and environment injection. The map is rendered directly as gateway TOML,
so it uses the gateway's snake_case field names:

```yaml
gatewayConfig:
  openshell.gateway:
    credential_drivers:
      - vault
  openshell.credential_drivers.vault:
    address: https://vault.vault.svc.cluster.local:8200
    mount: secret
    kv_version: "2"
    auth_method: kubernetes
    role: openshell-gateway
```

> `gatewayConfig` must contain only non-secret values. Helm serializes unknown
> fields generically and cannot determine whether an arbitrary string, such as
> `api_token`, is confidential. Do not put passwords, tokens, private keys,
> database URLs, or other secret material in this map. Use Secret-backed
> environment variables, files, volumes, or gateway credential drivers instead.
> The chart rejects known unsafe forms such as `database_url`, inline URL
> credentials, and PEM private keys; it is not a general secret scanner.

For the Kubernetes Secret driver, use
`openshell.credential_drivers.kubernetes-secrets.namespace` in the same map.
The chart derives any required RBAC from the selected driver; use a dedicated
namespace to limit access to OpenShell-managed Secrets.

#### OpenShift

Append these flags to any of the PostgreSQL commands above for OpenShift:

```
--set server.disableTls=true \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null
```

### High availability

Set `replicaCount` above `1` only with `server.externalDbSecret`; the default
SQLite database is per pod and cannot coordinate multiple gateway replicas.
The chart creates a headless peer Service for gateway-to-gateway relay traffic.
StatefulSet pods use stable pod DNS names through that headless Service.
Deployment pods advertise their pod IP with `OPENSHELL_PEER_ENDPOINT`, because
Kubernetes does not assign stable per-pod DNS names to Deployment replicas.

Gateway peer traffic uses Kubernetes ServiceAccount identity. Each gateway pod
mounts a projected, pod-bound ServiceAccount token with audience
`openshell-gateway-peer`; receiving replicas validate that token with the
Kubernetes TokenReview API, verify the live pod UID and Helm selector labels,
and authorize only the internal `PeerRelay` RPC. The chart does not create or
accept a shared gateway peer Secret.

With gateway TLS enabled, peer calls use the chart CA and client TLS Secret for
server verification and mTLS. The client verifies the stable gateway Service
DNS name while connecting directly to the owning pod. Custom TLS Secrets must
include that Service DNS name in the server certificate and provide the CA and
client credentials configured by `server.tls`.

Set `autoscaling.enabled=true` to render an `autoscaling/v2`
HorizontalPodAutoscaler for the gateway workload. The Deployment or
StatefulSet then omits `spec.replicas`, and the chart applies its
multi-replica checks to `autoscaling.maxReplicas`. CPU and memory targets
need a matching `resources.requests` entry, or a `resources.limits` entry,
which Kubernetes copies into the request. Add custom metrics from a metrics
adapter with `autoscaling.metrics`. See the
[High Availability guide](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability)
for the metrics to scale and alert on.

Enabling autoscaling on an existing release removes `spec.replicas` from the
workload in that upgrade. Kubernetes can reset the workload to one replica
until the HPA scales it back to at least `autoscaling.minReplicas`, which
disconnects supervisor sessions from the other gateway pods; they reconnect to the remaining replica. Enable autoscaling
when you install the chart, or during a maintenance window. On an existing
release, upgrade with `--reset-then-reuse-values`, which needs Helm 3.14 or
later. With an older Helm, save the values with
`helm get values <release> -o yaml > values.yaml` and upgrade with
`--reset-values -f values.yaml`. `--reuse-values` keeps the
previous chart version's defaults, which lack the autoscaling values,
including the scale-down `behavior`, when that release predates them. For background, refer to
[Migrating Deployments and StatefulSets to horizontal autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#migrating-deployments-and-statefulsets-to-horizontal-autoscaling).
Disabling autoscaling renders `spec.replicas` from `replicaCount` again,
which defaults to 1, so set `replicaCount` to the replica count you want in
that upgrade.

Set `supervisor.autoscaling.enabled=true` with the shared supervisor topology
to autoscale its Deployment independently of the gateway. It uses average
memory per Pod (`supervisor.autoscaling.targetMemoryAverageValue`, default
`512Mi`) through the resource metrics API, usually provided by Metrics Server.
Defaults keep two to four members and remove at most one member every two
minutes after a five-minute stabilization window. The HPA owns replicas, so
`supervisor.replicas` is ignored. Enabling it on an existing Deployment has
the same temporary reset-to-one risk described above; pick up new defaults
with `--reset-then-reuse-values`. When disabling it, set `supervisor.replicas`
explicitly. Existing sandboxes stay on their owners when scaling up, and
memory-based scaling does not guarantee spare capacity for handoff. See the
[shared supervisor autoscaling guide](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability#autoscale-shared-supervisors-by-memory).

## Secret bootstrap

By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs`
to create the gateway's server/client mTLS Secrets and sandbox JWT signing Secret.
The Job uses the gateway image itself, so air-gapped environments only need to
mirror that one image (no separate openssl/alpine sidecar).

When `certManager.enabled=true`, cert-manager owns the TLS Secrets and the chart
runs the same hook in JWT-only mode because cert-manager does not create the
sandbox JWT signing Secret. This precedence applies even if
`pkiInitJob.enabled` remains true. Set `pkiInitJob.enabled=false` only when an
external non-cert-manager TLS source manages TLS and you pre-create the sandbox
JWT signing Secret.

## SPIFFE/SPIRE provider token grants

Set `gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path`
to let sandbox supervisors use SPIFFE JWT-SVIDs for dynamic provider token
grants. The chart keeps supervisor-to-gateway authentication on gateway-minted
sandbox JWTs and passes the configured socket path to the Kubernetes driver.

```yaml
gatewayConfig:
  openshell.drivers.kubernetes:
    provider_spiffe_workload_api_socket_path: /spiffe-workload-api/spire-agent.sock
```

For local development, uncomment the SPIRE Helm releases in `skaffold.yaml` and
add `ci/values-spire.yaml` to the OpenShell release values files.

The gateway verifies supervisor JWT-SVIDs with JWT bundles fetched from the
SPIFFE Workload API, so this path does not require access to the SPIRE OIDC
discovery endpoint or its TLS CA.

{{ template "chart.valuesSection" . }}
{{ template "helm-docs.versionFooter" . }}