trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-21 06:11:31 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
windows
OpenShell/deploy/helm/openshell/README.md
T
Emilien MacchidiveshBryce Wilkinson
 e30c938bb1 feat(server): scope mutation locks and bound lock waits (#4147) 
* test(server): add a local PostgreSQL test runner

The openshell-server tests that need a real PostgreSQL server are
ignored by default and had no shared way to run. The mutation-replay
test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every
contributor had to provision a database by hand, and the advisory-lock
tests that follow in this series need the same setup.

Add mise run test:rust:postgres. It runs every ignored postgres_* test
in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a
disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE
selects one) on the image pinned by the Kubernetes e2e fixture and
removes it on exit. Tests run one at a time because advisory locks are
database-wide. The runner always points the legacy replay variable at
the selected database, so an inherited URL cannot send that test to a
different server. test:postgres-runner checks that selection with a fake
cargo and needs no database or container engine. CI does not run the
PostgreSQL tests; TESTING.md documents the task.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* fix(server): hold the mutation guard while deleting a provider

DeleteProvider checked that no sandbox referenced the provider and then
deleted the record without holding the sandbox mutation guard. Sandbox
create and provider attach take that guard while they write provider
references, so one of them could add a reference after the attached
sandbox check passed, and the delete then removed a provider that a
sandbox spec still named.

Take the guard before the attached sandbox check, as provider create and
update already do, so the check and the delete run against a stable
set of sandbox references, and reject an empty name after authorization
but before the guard. A new test holds the guard, attaches the provider
while the delete waits, and asserts that the delete fails with
FailedPrecondition and leaves the provider in place.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* fix(server): release staged credentials when a refresh fails early

A provider credential refresh stages the minted values under new
credential handles before it commits the provider update, and deletes
those handles when validation or persistence fails. Two earlier returns
skipped that cleanup. The minted expiry was converted to a protobuf
timestamp only after staging, so an expiry outside the timestamp range
failed the refresh and left the staged values in the credential driver.
A failure to acquire the sandbox mutation guard returned the same way.

Convert the expiry before staging anything, so a bad value fails before
any handle exists, and delete the staged handles when the guard cannot
be acquired. A new test refreshes a stored credential with an expiry of
i64::MAX and asserts that the call fails, the provider keeps its
original handles, and the credential driver holds the same number of
values as before. The guard failure path has no test here, because the
SQLite guard used by unit tests cannot fail.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool

The cross-object mutation guard took its advisory-lock session from the
10-connection data pool and closed it on every release, so each guarded
mutation opened a new database connection, and lock holders competed
with their own critical sections for data connections. A lock wait that
hit the 10 second lock_timeout surfaced as INTERNAL.

Add a lazy, dedicated 4-connection lock pool and a new
persistence::mutation_lock module for the lock key, mode and set types.
Store::acquire_distributed_mutation_guard now takes a lock set and a
deadline, and each lock statement sets lock_timeout to the time left
before that deadline, so a cancelled acquisition never waits past it on
the server. Returned connections are scrubbed with
pg_advisory_unlock_all() and reused, and each return is bounded by 5
seconds so a stalled session cannot keep a pool permit.

A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a
new LockTimeout error, and so does a wait for a lock connection during
which every one was checked out at some point, or that started with less
than a second left, too little to open one. The API returns it as
UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry
delay, because the mutation lock was not acquired, so the request's
guarded writes did not run. Only the guard acquisition classifies 55P03;
any other 55P03 stays a database error. A lock connection that
PostgreSQL does not open by the deadline despite at least a second to do
so (refused, out of connection slots, starting up, or not answering, all
of which SQLx retries silently) is a database error, not a lock timeout.

Mutations still take the legacy global key exclusively, so old and new
replicas keep excluding each other during a rolling upgrade. Each
PostgreSQL-backed gateway pod can now open up to 14 database
connections (10 data plus 4 lock) instead of 10, and the chart's
autoscaling.maxReplicas comment says what its default of 4 needs. The
SSH host identity lock stays on a data connection, because sandbox
creation takes it while holding its mutation guard and a second
lock-pool connection there could exhaust the pool. The provider
credential refresh lock stays there too, so a refresh still progresses
while every lock connection is in use.

Part of #3528

Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* perf(server): partition mutation locks by workspace and sandbox

Every cross-object mutation, and every lifecycle and supervisor update,
took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory
lock. Unrelated sandboxes and providers therefore serialized across all
replicas, and one slow critical section delayed every other mutation in
the fleet.

Replace that lock with hierarchical intention locks on a global key, one
key per workspace and one key per sandbox, each held shared or
exclusive. Global settings and policy writers and platform-scope profile
writers hold the global key exclusively. Provider and workspace-scoped
profile writers, including DeleteProvider and credential refresh, hold
their workspace key exclusively. Sandbox writers and supervisor reports
hold their sandbox key exclusively under shared global and workspace
keys. Lifecycle, driver-watch and reconcile paths take only
process-local keys and keep relying on compare-and-swap across replicas.
Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each
guard takes its local keys and then its advisory locks in ascending
order under one 10 second deadline, and no path nests guards. The
settings mutex is removed, since the global and sandbox keys cover it.
Sandbox settings are keyed by sandbox name, which a sandbox key no
longer covers once another sandbox reuses the name, so a settings
update returns NOT_FOUND unless its sandbox still holds the name after
the read, and ABORTED if the settings row was recreated before the
write.

The global key keeps its legacy value, so older replicas, which hold it
exclusively for every mutation, still exclude new ones during a rolling
upgrade. Guards now also give up after 10 seconds on SQLite, and
provider refresh reports a guard timeout as UNAVAILABLE instead of
INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND.
Startup endpoint-status reconciliation still holds the global key for
its whole scan.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* fix(server): reconcile endpoint status on startup one sandbox at a time

Startup endpoint-status reconciliation held the global mutation key
exclusively for its whole scan. On PostgreSQL that blocked every guarded
mutation on every replica until the starting gateway had walked all
sandboxes, and the scan first waited behind every in-flight sandbox
mutation. It also paged with an offset, so sandboxes deleted mid-scan
shifted rows between pages, and a write that raced the scan (a Conflict
from a lifecycle update on another replica, or a sandbox deleted after
it was listed) failed gateway startup.

Walk the sandboxes with keyset paging instead, and reset each candidate
under its own sandbox-scoped guard, up to four at a time to match the
lock pool. A candidate with a fresh shared owner is skipped before any
guard is taken. Under the guard the reconciliation re-checks the owner,
re-reads the sandbox, skips it when it is gone or no longer carries
endpoint status, and writes against the version it just read. A Conflict
or a database error at write time, which is how a row deleted after the
re-read surfaces, is retried with a fresh read, at most five attempts
per sandbox. A lock timeout on the guard is retried the same way, so a
holder of the sandbox, its workspace or the global key that releases
within the retry window cannot stop a replica from starting. No
lock-timeout retry starts once the scan has run for three lock timeouts,
so a scan stuck behind a long holder fails within about 40 seconds,
inside the chart's 60-second startup probe, and logs the sandbox it gave
up on. Store::list_all_messages loses its only caller and is removed.

Reconciliation stays fail-closed: a holder that outlasts the retry
window, an owner lookup failure, or a sandbox that keeps changing still
fails startup, so stale endpoint success is never served. User-visible
change: a restarting gateway no longer stalls mutations of unrelated
sandboxes and providers on its peers while it reconciles.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* perf(server): skip the endpoint-status lock when a session was replaced

When a supervisor session ends, the gateway resets that sandbox's tool
server endpoint evidence unless a replacement session already owns
endpoint observation. The replacement check ran only after the reset
had read the sandbox and taken its sandbox mutation guard, so a reset
whose supervisor had already reconnected still queued behind other
mutations of that sandbox, and on PostgreSQL it used a lock-pool
connection, only to find it had nothing to do.

Check for a replacement first: a live local session, or a fresh owner
record on a peer. When one exists, return without reading the sandbox
or taking the guard. Otherwise take the guard and repeat the same check
under it, because a replacement's pre-acknowledgement reset runs under
the same sandbox key and resetting after it would wipe the
replacement's fresh evidence. Both checks share one helper, so they
cannot drift apart. When no replacement exists, the reset costs one
extra owner-row read.

This helps most when a replica notices a dead stream late, after the
supervisor has already reconnected elsewhere (a network partition or a
keepalive expiry).

Deferring to a replacement leaves the old session's evidence to the
replacement's pre-acknowledgement reset, which can time out on its
bounded mutation lock wait. That reset's failure path removed the
replacement and released its owner record but scheduled no
invalidation, so the old session's success evidence could outlive both
sessions. An old session that ended after its replacement registered
already deferred this way, and the unguarded check now makes every
disconnect that finds the replacement defer too. When the reset fails
while the replacement is still current, also schedule the retried
invalidation that follows a session end. That failure path keeps the
lifecycle demotion from abandon_session_setup, but runs it in the
spawned task before the retry and only while this session is still
current, so a timed-out reset returns promptly.

Part of #3528

Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* test(server): cover staged credential cleanup on a lock timeout

A provider credential refresh deletes its staged handles when it cannot
acquire the provider mutation guard, but that path had no test: the
SQLite guard used by unit tests could not fail. Mutation locks now
acquire under a deadline that tests can shorten, so the timeout path
can be exercised directly.

The new test shortens the lock deadline to 50 ms, holds the workspace
mutation guard, and refreshes a provider that already stores an AWS
access key. The refresh returns UNAVAILABLE with the
MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle,
which still resolves to the old key, and the credential driver holds
the same number of values as before the refresh.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* feat(server): export mutation lock wait and timeout metrics

A handler mutation guard can now wait up to 10 seconds for its keys and
then fail with UNAVAILABLE, but nothing showed operators how long guards
waited or how often they gave up, short of reading warning logs on each
replica.

Export openshell_server_mutation_lock_wait_seconds, a histogram of the
time one guard took to acquire all of its keys (local registry plus
PostgreSQL advisory locks), recorded on success, and
openshell_server_mutation_lock_timeouts_total, a counter of guard
acquisitions that timed out. Both carry a bounded scope label: global,
workspace or sandbox. The platform scope, whose workspace is empty,
counts as global. The histogram uses the same 1 ms to 15 s buckets as
the relay and peer histograms, and the timeout counter starts at 0 for
every scope, so rate() works on an idle replica. The timeout warning
now also logs the scope. A guard that fails for another reason, such as
a lock connection PostgreSQL did not open, is not counted as a timeout
and logs a "mutation lock acquisition failed" warning instead.

Lifecycle, driver-watch and reconcile paths take only process-local
locks and never time out, so they are not recorded.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* test(server): add PostgreSQL mutation lock integration tests

The scoped mutation locks were only unit-tested on SQLite, where the
database layer is a no-op. Add ignored postgres_* tests that run the
advisory-lock layer against a real PostgreSQL through mise run
test:rust:postgres, each in its own disposable schema (TestSchema).

persistence/mutation_lock_pg_tests.rs uses two stores on one database
as two replicas. It covers disjoint sandbox scopes holding at once, a
workspace key blocking only sandboxes in that workspace, the global key
blocking every scope, and exclusion in both directions against a raw
holder of the legacy global key, whose 55P03 stays a database error
outside a guard acquisition. It checks that timed-out and cancelled
acquisitions release the keys they already took, that a released lock
connection returns to the pool without residual locks, that a release
stalled on the network closes its session and the pool recovers, and
that the lock pool never grows past its size. It also checks that a
full lock pool times out as lock contention, while a lock connection
that PostgreSQL never answers fails as a database error.
compute/mutation_guard.rs runs the random scope mix across two
PostgreSQL runtimes and checks that a by-id guard waits for a workspace
writer on the other replica. The provider refresh lock test moves onto
the runner too, and checks that a refresh proceeds while every lock
connection is held.

A bench_postgres_* capacity check measures a paced burst of 500
reconnects 12 ms apart into one replica and fails when the p99 lock
wait reaches a fifth of the lock timeout. Machine load moves those
waits, so mise run test:rust:postgres skips it and mise run
test:rust:postgres:bench runs it alone, through a new
OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner.

The tests add test-only helpers: the lock pool's size and idle count
and the backend pid of a held lock session. Nothing changes outside
tests and the test runner.

Part of #3528

Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* ci: run the PostgreSQL-backed server tests

Branch Checks now runs the postgres_* tests through the same runner
developers use, on an x86_64 Linux runner with Docker. They cover lock
scope exclusion, interop with replicas still on the old global key,
cancellation, lock-pool bounds and connection failures, which no other
CI lane checks directly. The load-sensitive bench_postgres_* capacity
checks stay local.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* docs: document scoped mutation locks and lock-pool sizing

In the High Availability guide, size PostgreSQL for 14 connections per
gateway pod, 10 for data and 4 for the new lock pool, so three or more
replicas now exceed what the PostgreSQL default leaves, and keep
headroom for lock sessions that cancelled requests leave behind.
Explain which changes lock a sandbox, a workspace, or the whole fleet,
and that a lock wait over 10 seconds returns a retryable UNAVAILABLE
with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to
raise max_connections to the 14-per-pod size before upgrading, because
the rollout already runs new pods at that count, and warns that older
replicas keep serializing every mutation until the rollout finishes.
Monitor Capacity gains the lock metrics and their queries, and names
slow driver, credential, middleware, or profile source calls as another
cause of lock waits.

List the lock wait histogram and timeout counter, with their scope
label, on the Gateway Metrics page. The counter also counts background
work and leaves out lock connections that PostgreSQL does not open
although at least a second of the wait remained.

In the cluster debugging skill, replace the single advisory-lock
paragraph with the lock levels and pool, what each timeout detail
means, how a lock connection that PostgreSQL does not open shows up,
when terminating a lock backend is safe, a pg_locks query that shows
holders and waiters, and how to spot the global key. The per-pod greps
also read the timeout counter, and new rows cover lock timeouts and
lock connections that PostgreSQL does not open.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* feat(server): check mutation lock ordering in debug builds

The mutation lock ordering rules were prose only, so a nested guard on
a different key passed every test and could only deadlock under
contention. Debug builds now register each held guard with the Tokio
task (or the thread, outside a task) that acquired it, and panic before
waiting when a local key, a blocking lifecycle gate or the SSH identity
lock would break rules 1, 4 or 5. Startup endpoint-status
reconciliation and the tests that hold a guard while driving guarded
code run under lock_order::branch(), which marks deliberate concurrency
within one task. Release builds compile the check out.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* fix(server): do not hold gateway shutdown for the endpoint-status retry

When a replacement supervisor session failed its endpoint-status reset,
the spawned cleanup task kept the session lifetime guard through
retry_endpoint_status_after_supervisor_disconnect. That retry only ends
once the reset is durable, so a gateway shutdown during a database or
lock outage waited the full 10 s and reported ownership cleanup as
incomplete. The guard now covers only the lifecycle demotion, as in
finish_supervisor_session, and a SQLite test checks that shutdown
completes while the retry is still blocked.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* refactor(server): make the platform profile lock scope explicit

Platform-scope provider profile writers (import, update, delete) pass an
empty workspace on purpose and need X(global), because they validate
sandboxes in every workspace. MutationScope::profiles() now names that
escalation and picks Global for the platform scope, instead of relying
on MutationScope::Workspace("") to mean it. An empty workspace name now
trips a debug assertion, and release builds keep X(global) as a
fail-safe that over-locks rather than under-locks.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* perf(server): amortize local mutation lock table sweeps

LocalMutationLocks swept its whole table on every key lookup, so a
burst of N concurrent acquisitions on one replica cost O(N^2) under a
blocking mutex. The table is now swept only when a new key grows it
past the larger of 64 and twice the live entries left by the last
sweep, which makes acquisition amortized O(1). Released entries,
including those of cancelled waiters, stay bounded at about twice the
peak live set. The lifecycle gate registry keeps its per-call sweep.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* feat(server): make the database pool ceiling configurable

Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5
for on-disk SQLite. One pool is shared by every database-backed RPC, so
that number is also the gateway's ceiling on concurrent database work.
Once every connection is checked out, callers queue on acquire and sqlx
logs "time to acquire exceeded slow threshold"; sandbox creates then time
out and are retried, which adds load rather than shedding it.

The right ceiling is deployment-specific — it depends on how many gateway
replicas share the database and what max_connections the server itself
allows — so it cannot be a single number baked into the binary.

Expose it on the same config surface as the rest of the gateway's
settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the
TOML key database_max_connections, resolved in that precedence order.
Helm deployments set the key through gatewayConfig. Omitting it keeps
each backend's previous value, so existing deployments do not move.

Values below 1 are rejected rather than silently replaced by the default:
a zero pool would block every acquire, and quietly ignoring a typo would
reproduce the ceiling the operator is trying to lift. An in-memory SQLite
database stays pinned to one connection, since the database lives inside
that connection.

Refs: #2561
Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
(cherry picked from commit 33517b32bd)
[emacchi: dropped the server.dbMaxConnections Helm value; operators set
 the key through gatewayConfig]
Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* feat(server): make the mutation lock pool size configurable

The mutation lock pool was fixed at 4 connections per replica. That
number rests on one reconnect-burst bench rather than scale testing, so
a gateway that serves many more sandboxes needs a way to raise it
without a rebuild. Add --db-lock-max-connections,
OPENSHELL_DB_LOCK_MAX_CONNECTIONS and the database_lock_max_connections
gateway.toml key next to the data pool knob, resolved in the same order;
Helm deployments set it through gatewayConfig. The default stays 4,
zero is rejected, SQLite ignores it with a warning, and the gateway logs
both pool sizes at startup. PostgreSQL now needs at least 2 data
connections, because the SSH identity lock keeps one while it queries
the pool. The connection sizing docs, the HA guide formula and the
debug skill now count data plus lock connections.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* feat(server): export mutation lock errors and lock connections in use

A mutation lock acquisition that failed without timing out, such as a
lock connection PostgreSQL never opened or a failed lock statement,
only logged a warning, and readiness stays healthy because it pings the
data pool. Count those failures in
openshell_server_mutation_lock_errors_total{scope}, kept apart from
timeouts because they point at PostgreSQL capacity or connectivity
rather than contention. Export
openshell_server_mutation_lock_connections_in_use as a gauge slot held
by each checked-out lock connection, next to
openshell_server_mutation_lock_connections_capacity, which reads the
configured lock pool size; both exist only with PostgreSQL. The metrics
catalog, the HA guide's signals and PromQL examples, and the debug
skill cover the new series.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

* feat(server): time mutation lock holds and test a stalled holder

Nothing bounds how long a mutation guard is held, so a holder stuck in
a compute driver, credential backend or middleware call keeps its scope
locked while waiters time out. Each guard now records its hold in
openshell_server_mutation_lock_hold_seconds{scope} when it is released,
and logs a warning when the hold outlasted the lock wait timeout, so
operators can find the replica and scope behind those timeouts. A
PostgreSQL test with a stalled holder pins the current contract:
waiters on its sandbox, workspace and global scopes fail by their
deadline, other scopes proceed, the holder keeps its key, and a full
lock pool on one replica leaves the other working. The metrics catalog,
the HA guide and the debug skill describe the new signals, which
observe holds without bounding them.

Part of #3528

Signed-off-by: Emilien Macchi <emacchi@redhat.com>

---------

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Co-authored-by: divesh <dgude@nvidia.com>
Co-authored-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
2026-10-13 23:34:26 +00:00

568 lines
51 KiB
Markdown
Raw Permalink Blame History

# OpenShell Helm Chart
<!--
This file is generated by helm-docs.
Edit README.md.gotmpl and values.yaml, then run `mise run helm:docs`.
-->
> **Experimental** - the Kubernetes deployment path is under active development. Expect rough edges and breaking changes.
This chart deploys the OpenShell gateway into a Kubernetes cluster. It is published as an OCI artifact to GHCR at `oci://ghcr.io/nvidia/openshell/helm-chart`.
By default, this chart also creates the namespace-scoped resources needed by
sandboxes. For a shared-gateway deployment, install it with
`workspaceResources.enabled=false`, then install the
`deploy/helm/openshell-workspace` chart in every pre-provisioned workspace
namespace. The gateway and workspace releases can then be upgraded and removed
independently. Use Kubernetes `operator` workspace mode when one gateway serves
multiple pre-provisioned workspace namespaces.
## Cluster-scoped vs namespaced objects
Most objects in this chart are namespaced and land in the release namespace.
Only two are cluster-scoped:
| Object | Default name |
| --- | --- |
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |
By default the release creates both, so an install by a cluster-admin is
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
split the install in two.
A cluster-admin applies the cluster-scoped objects once per gateway
ServiceAccount, rendered from the same values the release uses:
```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.create=true \
--set rbac.clusterScoped.create=true \
--set agentSandbox.preflight.enabled=false \
--show-only templates/clusterrole.yaml \
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
```
A namespace-admin then installs and upgrades the release with cluster-scoped
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
or the equivalent `--set`:
```shell
helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.clusterScoped.create=false
```
The gateway ServiceAccount name and namespace do not change, so the
pre-created `ClusterRoleBinding` keeps matching the release. This works with
`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows
`serviceAccount.name`, so render the admin step with the same values.
### Which flag the installer needs
In the default `shared` workspace mode the release also creates a namespaced
sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions.
Kubernetes forbids granting permissions you do not hold, and the built-in
`admin` ClusterRole does not cover that CRD, so an installer holding only
`admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not
enough and the install fails with `attempting to grant RBAC permissions not
currently held`.
| Workspace mode | Installer holds | Use |
| --- | --- | --- |
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |
`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
is needed there.
With `rbac.create=false` the cluster-admin applies the namespaced RBAC too,
adding it to the same render:
```shell
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--namespace openshell -f my-values.yaml \
--set rbac.create=true \
--set rbac.clusterScoped.create=true \
--set agentSandbox.preflight.enabled=false \
--show-only templates/clusterrole.yaml \
--show-only templates/clusterrolebinding.yaml \
--show-only templates/role.yaml \
--show-only templates/rolebinding.yaml \
--show-only templates/peer-role.yaml | kubectl apply -f -
```
To grant the installer the sandbox permissions instead, bind it to a Role
carrying the same rules as the chart's `openshell-sandbox` Role.
The certgen hook and credential driver RBAC keep their own flags
(`pkiInitJob.enabled` and
`server.credentialDrivers.kubernetesSecrets.rbac.create`).
### Migrating an existing release
Helm deletes objects that leave a release manifest, so setting
`rbac.clusterScoped.create=false` on a release that already owns the
`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses
TokenReview until a cluster-admin re-applies them. Hand ownership over first, as
cluster-admin, so nothing is deleted:
```shell
kubectl annotate clusterrole "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
```
The objects then survive the upgrade that sets the flag, and the cluster-admin
owns them from that point on. Fresh installs need no such step.
`rbac.clusterScoped.create` is independent of
`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change
what the `ClusterRole` contains, but they never force the namespaced release to
apply it. Re-run the cluster-admin step after changing values that affect the
`ClusterRole` rules.
## Prerequisites
> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
> but Kubernetes accepts them even if no CNI enforces them. Without enforcement,
> sandbox workloads may connect directly and bypass supervisor network policy.
> Verify CNI support before installing OpenShell.
The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with:
```shell
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/sandbox.yaml
```
The chart does not install this cluster-scoped dependency. By default, it
fails before creating gateway resources when the cluster serves neither
supported Sandbox API (`agents.x-k8s.io/v1beta1` or
`agents.x-k8s.io/v1alpha1`). Disable the check with
`agentSandbox.preflight.enabled=false` for offline `helm template` rendering,
where Helm cannot discover cluster APIs.
## Install on Kubernetes
```shell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version>
```
## Install on OpenShift
See the full [OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift) for details. Quick start:
```shell
# Precreate the openshell namespace
oc create ns openshell
# Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> -n openshell \
--set server.disableTls=true \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null
```
On OpenShift 4.22+, end-to-end TLS is supported via `BackendTLSPolicy`. See the
[OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift#end-to-end-tls-openshift-422) for details.
## Available versions
| Tag | Source | Notes |
| --- | --- | --- |
| `<semver>` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway, sandbox, and supervisor image versions. Recommended for production. |
| `<semver>-pre.N` (e.g. `0.1.0-pre.3`) | A specific prerelease candidate | Immutable candidate pin that tracks images with the same exact version. |
| `0.0.0-dev` | Latest commit on `main` | Floating tag, overwritten on every push. `appVersion` is `dev`, so images resolve to the `:dev` tag. |
| `0.0.0-dev.<commit-sha>` | A specific commit on `main` | Per-commit pin. Chart version and `appVersion` both use the full 40-character commit SHA, which matches the image tag pushed by CI. |
Prerelease and `dev` tags are intended for testing changes ahead of a release. Production deployments should pin to a stable tagged release.
## Configuration
See [`values.yaml`](values.yaml) for source defaults. Selected overlays:
- [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - gateway-only configuration
- [`ci/values-cert-manager.yaml`](ci/values-cert-manager.yaml) - cert-manager integration
- [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration
- [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing
- [`ci/values-autoscaling.yaml`](ci/values-autoscaling.yaml) - CI overlay for rendering the optional gateway HorizontalPodAutoscaler
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development
### Database backend
By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the
database is backed by a per-pod PVC:
```yaml
server:
dbUrl: "sqlite:/var/openshell/openshell.db"
```
#### External PostgreSQL
Use external PostgreSQL when the gateway should connect to a database managed
outside this chart. The OpenShell chart does not deploy a database; install
PostgreSQL separately using the chart, operator, or managed service that fits
your environment, then pass the connection URI through a Secret.
Create a Secret containing the PostgreSQL connection URI if one does not
already exist:
```bash
kubectl create secret generic my-pg-credentials -n openshell \
--from-literal=uri="postgresql://user:pass@host:5432/dbname"
```
Then install the chart pointing at that Secret:
```bash
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
-n openshell \
--set workload.kind=deployment \
--set server.externalDbSecret=my-pg-credentials
```
Use `workload.kind=deployment` for external database-backed multi-replica
gateways. `workload.kind=statefulset` is still available for single-replica
SQLite installs and for operators who explicitly need StatefulSet identity or
storage semantics.
### Credential storage
By default, the chart uses the gateway's encrypted database credential storage.
The gateway writes encrypted provider credential envelopes to the OpenShell
database. The chart creates a retained Kubernetes Secret with the shared
key-encryption key and injects that key into every gateway pod, so the same
default works for single-replica and external database-backed HA deployments.
Use `gatewayConfig` to select `kubernetes-secrets` or `vault` when credentials
should live in a cluster or external secret backend. Selecting an external
credential driver disables the default credential-storage key-encryption-key
Secret and environment injection. The map is rendered directly as gateway TOML,
so it uses the gateway's snake_case field names:
```yaml
gatewayConfig:
openshell.gateway:
credential_drivers:
- vault
openshell.credential_drivers.vault:
address: https://vault.vault.svc.cluster.local:8200
mount: secret
kv_version: "2"
auth_method: kubernetes
role: openshell-gateway
```
> `gatewayConfig` must contain only non-secret values. Helm serializes unknown
> fields generically and cannot determine whether an arbitrary string, such as
> `api_token`, is confidential. Do not put passwords, tokens, private keys,
> database URLs, or other secret material in this map. Use Secret-backed
> environment variables, files, volumes, or gateway credential drivers instead.
> The chart rejects known unsafe forms such as `database_url`, inline URL
> credentials, and PEM private keys; it is not a general secret scanner.
For the Kubernetes Secret driver, use
`openshell.credential_drivers.kubernetes-secrets.namespace` in the same map.
The chart derives any required RBAC from the selected driver; use a dedicated
namespace to limit access to OpenShell-managed Secrets.
#### OpenShift
Append these flags to any of the PostgreSQL commands above for OpenShift:
```
--set server.disableTls=true \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null
```
### High availability
Set `replicaCount` above `1` only with `server.externalDbSecret`; the default
SQLite database is per pod and cannot coordinate multiple gateway replicas.
The chart creates a headless peer Service for gateway-to-gateway relay traffic.
StatefulSet pods use stable pod DNS names through that headless Service.
Deployment pods advertise their pod IP with `OPENSHELL_PEER_ENDPOINT`, because
Kubernetes does not assign stable per-pod DNS names to Deployment replicas.
Gateway peer traffic uses Kubernetes ServiceAccount identity. Each gateway pod
mounts a projected, pod-bound ServiceAccount token with audience
`openshell-gateway-peer`; receiving replicas validate that token with the
Kubernetes TokenReview API, verify the live pod UID and Helm selector labels,
and authorize only the internal `PeerRelay` RPC. The chart does not create or
accept a shared gateway peer Secret.
With gateway TLS enabled, peer calls use the chart CA and client TLS Secret for
server verification and mTLS. The client verifies the stable gateway Service
DNS name while connecting directly to the owning pod. Custom TLS Secrets must
include that Service DNS name in the server certificate and provide the CA and
client credentials configured by `server.tls`.
Set `autoscaling.enabled=true` to render an `autoscaling/v2`
HorizontalPodAutoscaler for the gateway workload. The Deployment or
StatefulSet then omits `spec.replicas`, and the chart applies its
multi-replica checks to `autoscaling.maxReplicas`. CPU and memory targets
need a matching `resources.requests` entry, or a `resources.limits` entry,
which Kubernetes copies into the request. Add custom metrics from a metrics
adapter with `autoscaling.metrics`. See the
[High Availability guide](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability)
for the metrics to scale and alert on.
Enabling autoscaling on an existing release removes `spec.replicas` from the
workload in that upgrade. Kubernetes can reset the workload to one replica
until the HPA scales it back to at least `autoscaling.minReplicas`, which
disconnects supervisor sessions from the other gateway pods; they reconnect to the remaining replica. Enable autoscaling
when you install the chart, or during a maintenance window. On an existing
release, upgrade with `--reset-then-reuse-values`, which needs Helm 3.14 or
later. With an older Helm, save the values with
`helm get values <release> -o yaml > values.yaml` and upgrade with
`--reset-values -f values.yaml`. `--reuse-values` keeps the
previous chart version's defaults, which lack the autoscaling values,
including the scale-down `behavior`, when that release predates them. For background, refer to
[Migrating Deployments and StatefulSets to horizontal autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#migrating-deployments-and-statefulsets-to-horizontal-autoscaling).
Disabling autoscaling renders `spec.replicas` from `replicaCount` again,
which defaults to 1, so set `replicaCount` to the replica count you want in
that upgrade.
## Secret bootstrap
By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs`
to create the gateway's server/client mTLS Secrets and sandbox JWT signing Secret.
The Job uses the gateway image itself, so air-gapped environments only need to
mirror that one image (no separate openssl/alpine sidecar).
When `certManager.enabled=true`, cert-manager owns the TLS Secrets and the chart
runs the same hook in JWT-only mode because cert-manager does not create the
sandbox JWT signing Secret. This precedence applies even if
`pkiInitJob.enabled` remains true. Set `pkiInitJob.enabled=false` only when an
external non-cert-manager TLS source manages TLS and you pre-create the sandbox
JWT signing Secret.
## SPIFFE/SPIRE provider token grants
Set `gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path`
to let sandbox supervisors use SPIFFE JWT-SVIDs for dynamic provider token
grants. The chart keeps supervisor-to-gateway authentication on gateway-minted
sandbox JWTs and passes the configured socket path to the Kubernetes driver.
```yaml
gatewayConfig:
openshell.drivers.kubernetes:
provider_spiffe_workload_api_socket_path: /spiffe-workload-api/spire-agent.sock
```
For local development, uncomment the SPIRE Helm releases in `skaffold.yaml` and
add `ci/values-spire.yaml` to the OpenShell release values files.
The gateway verifies supervisor JWT-SVIDs with JWT bundles fetched from the
SPIFFE Workload API, so this path does not require access to the SPIRE OIDC
discovery endpoint or its TLS CA.
## Values
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` | Affinity rules for the gateway pod. |
| agentSandbox.preflight.enabled | bool | `true` | Check the live cluster for a supported Agent Sandbox API before rendering gateway resources. Disable only for offline rendering and linting. |
| autoscaling.behavior | object | `{"scaleDown":{"policies":[{"periodSeconds":120,"type":"Pods","value":1}],"stabilizationWindowSeconds":300}}` | HPA scaling behavior. Scale-down hands the removed pod's supervisor sessions to the remaining replicas, so the default removes at most one replica every two minutes after a five-minute stabilization window. Helm merges maps: set autoscaling.behavior.scaleDown to null to drop the default. |
| autoscaling.enabled | bool | `false` | Render a HorizontalPodAutoscaler and stop rendering spec.replicas. |
| autoscaling.maxReplicas | int | `4` | Maximum gateway replicas. Each replica opens up to database_max_connections + database_lock_max_connections PostgreSQL connections (14 with the defaults; set them under gatewayConfig.openshell.gateway); size the database for rollouts at this count, as the High Availability guide describes. With a Deployment and the default pool sizes, the default of 4 needs `max_connections` of at least 126, more than the PostgreSQL default of 100. |
| autoscaling.metrics | list | `[]` | Additional autoscaling/v2 MetricSpec entries appended verbatim, such as Pods metrics served by prometheus-adapter. |
| autoscaling.minReplicas | int | `2` | Minimum gateway replicas. Use 2 or more to survive a pod failure. |
| autoscaling.targetCPUUtilizationPercentage | int | `80` | Target average CPU utilization, as a percentage of resources.requests.cpu. Set to null to disable. Requires resources.requests.cpu, or resources.limits.cpu, which Kubernetes copies into the request. |
| autoscaling.targetMemoryUtilizationPercentage | int | `nil` | Target average memory utilization, as a percentage of resources.requests.memory. Null disables it. Requires resources.requests.memory, or resources.limits.memory, which Kubernetes copies into the request. |
| certManager.caSecretName | string | `"openshell-ca-tls"` | Secret created for the intermediate CA (Certificate with isCA: true). |
| certManager.certificateDuration | string | `"8760h"` | Duration for cert-manager-issued certificates. |
| certManager.certificateRenewBefore | string | `"720h"` | Renewal window for cert-manager-issued certificates. |
| certManager.clientCaFromServerTlsSecret | bool | `true` | Mount gateway client CA from the internal server TLS secret's ca.crt. The internal server certificate is always signed by the chart CA — the same CA that signs the client (mTLS) certificate — so the default (true) is correct for all configurations, including when serverIssuerRef is set. Only set to false if you mount the client CA from a separate secret via server.tls.clientCaSecretName. |
| certManager.enabled | bool | `false` | Create cert-manager Issuer and Certificate resources. When enabled, cert-manager owns TLS and the chart runs a JWT-only certgen hook to create the sandbox JWT signing Secret that cert-manager does not manage. |
| certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. |
| certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. |
| certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. |
| credentialDrivers.vault.caConfigMapName | string | `""` | ConfigMap containing the private Vault/OpenBao CA certificate under the ca.crt key. Helm mounts it only when the Vault driver is selected. |
| fullnameOverride | string | `""` | Override the full generated resource name. |
| gateway.image.digest | string | `""` | Gateway image digest. When set, this takes precedence over tag. |
| gateway.image.pullPolicy | string | `nil` | Gateway image pull policy. Empty uses global.image.pullPolicy. |
| gateway.image.registry | string | `""` | Gateway image registry. Empty uses global.image.registry. |
| gateway.image.repository | string | `"openshell/gateway"` | Gateway image repository. |
| gateway.image.tag | string | `""` | Gateway image tag. Defaults to the chart appVersion when empty. |
| gatewayConfig | object | `{"openshell":{"version":2}}` | Non-secret gateway application configuration. Top-level keys name TOML tables and are rendered into the mounted gateway.toml file. Kubernetes resource inputs remain outside this map; template expressions derive the corresponding runtime values from their resource owner. |
| global.image.pullPolicy | string | `"IfNotPresent"` | Shared OpenShell image pull policy. Individual image pull policies take precedence. |
| global.image.registry | string | `"ghcr.io/nvidia"` | Shared OpenShell image registry. Individual image registries take precedence. |
| global.image.tag | string | `""` | Shared OpenShell image tag. Defaults to the chart appVersion when empty. |
| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to `<fullname>-backend-ca` when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. |
| grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. |
| grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (`<fullname>.<namespace>.svc.cluster.local`) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. |
| grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. |
| grpcRoute.gateway.className | string | `"eg"` | GatewayClass for a chart-created Gateway. Set this to the name exposed by your Gateway API implementation. Envoy Gateway uses "eg" in the OpenShell examples. Ignored when gateway.create is false. |
| grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. Use sectionName to select a specific listener when needed. |
| grpcRoute.gateway.listener.allowedRoutes | string | `"Same"` | "Same" restricts attached routes to the release namespace; "All" allows any namespace. |
| grpcRoute.gateway.listener.port | int | `80` | Listener port for the generated Gateway resource. Use 443 with protocol HTTPS. |
| grpcRoute.gateway.listener.protocol | string | `"HTTP"` | Listener protocol for the generated Gateway resource: HTTP or HTTPS. HTTPS terminates TLS at the Gateway listener; pair it with server.disableTls=true so the proxy forwards plaintext to the gateway pod, and use OIDC for client identity (the gateway never sees the client certificate). |
| grpcRoute.gateway.listener.tls.certificateRefs | list | `[]` | certificateRefs for the HTTPS listener. Required when protocol is HTTPS. Each entry needs a `name` pointing at a kubernetes.io/tls Secret in the Gateway's namespace. May reference a cert-manager-issued Secret or the existing openshell-server-tls Secret (its SANs must include the external hostname). |
| grpcRoute.gateway.name | string | `""` | Name of the Gateway resource. Defaults to `<fullname>`. Set a distinct name when the selected controller creates a proxy Service with the Gateway name. When gateway.create is false, set this to the name of the existing Gateway. |
| grpcRoute.gateway.namespace | string | `""` | Namespace of the Gateway referenced by the GRPCRoute parentRef. Defaults to the release namespace. A chart-created Gateway always resides in the release namespace. |
| grpcRoute.gateway.sectionName | string | `""` | Listener name on an existing Gateway. Leave empty to let Gateway API listener matching select compatible listeners. |
| grpcRoute.hostnames | list | `[]` | Hostnames the GRPCRoute matches on. Leave empty to match all hosts. |
| grpcRoute.replicaRouting.enabled | bool | `false` | Route requests carrying an `x-openshell-replica` header straight to that gateway replica. The CLI sets the header on long-lived sandbox connections (SSH, port forwards, exec) using the owner the gateway reports, so they skip the relay through a peer replica. Renders one Service per replica. Requires workload.kind=statefulset, workload.allowMultiReplicaStatefulSet=true, and at most 15 replicas. |
| imagePullSecrets | list | `[]` | Image pull secrets attached to gateway and helper pods. |
| nameOverride | string | `"openshell"` | Override the chart name used in generated resource names. |
| networkPolicy.enabled | bool | `true` | Restrict SSH ingress on sandbox pods to the gateway. In managed mode, the driver applies the equivalent policy to each workspace namespace. |
| nodeSelector | object | `{}` | Node selector for the gateway pod. |
| openshiftRoute.annotations | object | `{}` | Extra annotations on the Route (e.g. haproxy.router.openshift.io/*). |
| openshiftRoute.enabled | bool | `false` | Create an OpenShift Route with TLS passthrough. |
| openshiftRoute.host | string | `""` | Hostname for the Route. Must match a SAN on the gateway's server cert. |
| pkiInitJob.enabled | bool | `true` | Run a pre-install/pre-upgrade Job that creates gateway and client mTLS Secrets. When certManager.enabled=true, cert-manager owns TLS and this same hook runs in JWT-only mode even if pkiInitJob.enabled remains true. |
| pkiInitJob.failOnTimeout | bool | `true` | Fail the helm install/upgrade if cert-manager does not issue the certificate within the polling timeout. When true (default), the install fails immediately if the timeout is reached, providing clear feedback that BackendTLSPolicy is non-functional. When false, the hook succeeds with a warning and you can run `helm upgrade` after cert-manager issues the certificate to create the backend CA ConfigMap. If you set this to false and see "TLS error: Secret is not supplied by SDS" when connecting to the gateway, check if the TLS secret exists and run `helm upgrade` to create the ConfigMap. |
| pkiInitJob.serverDnsNames | list | `[]` | Extra DNS SANs to append to the server certificate. |
| pkiInitJob.serverIpAddresses | list | `[]` | Extra IP SANs to append to the server certificate. |
| pkiInitJob.timeoutSeconds | int | `120` | Maximum time in seconds for the certgen hook to poll for cert-manager certificates. When using cert-manager with BackendTLSPolicy, the hook polls for this many seconds waiting for the certificate to be issued, then creates the backend CA ConfigMap. The Job deadline is set to (timeoutSeconds + 30) to allow time for ConfigMap creation and cleanup. Increase this if cert-manager takes longer than 120 seconds to issue certificates. |
| podAnnotations | object | `{}` | Extra annotations to add to the gateway pod. |
| podLabels | object | `{}` | Extra labels to add to the gateway pod. |
| podLifecycle.terminationGracePeriodSeconds | int | `30` | Maximum time, in seconds, Kubernetes waits for the gateway to exit before killing it. The gateway exits as soon as shutdown completes; the limit covers supervisor session cleanup and draining queued OCSF records. |
| podSecurityContext.fsGroup | int | `1000` | fsGroup assigned to the gateway pod. |
| probes.liveness.failureThreshold | int | `3` | Liveness probe failure threshold before the container is restarted. |
| probes.liveness.initialDelaySeconds | int | `2` | Liveness probe initial delay, in seconds. |
| probes.liveness.periodSeconds | int | `5` | Liveness probe period, in seconds. |
| probes.liveness.timeoutSeconds | int | `1` | Liveness probe timeout, in seconds. |
| probes.readiness.failureThreshold | int | `3` | Readiness probe failure threshold before the pod is marked not ready. |
| probes.readiness.initialDelaySeconds | int | `1` | Readiness probe initial delay, in seconds. |
| probes.readiness.periodSeconds | int | `2` | Readiness probe period, in seconds. |
| probes.readiness.timeoutSeconds | int | `1` | Readiness probe timeout, in seconds. |
| probes.startup.failureThreshold | int | `30` | Startup probe failure threshold before the container is killed. |
| probes.startup.periodSeconds | int | `2` | Startup probe period, in seconds. |
| probes.startup.timeoutSeconds | int | `1` | Startup probe timeout, in seconds. |
| rbac.clusterScoped.clusterRoleBindingName | string | `""` | Name for the ClusterRoleBinding. Empty uses the `<fullname>-node-reader-<release namespace>` default. |
| rbac.clusterScoped.clusterRoleName | string | `""` | Name for the ClusterRole. Empty uses the `<fullname>-node-reader-<release namespace>` default. |
| rbac.clusterScoped.create | bool | `true` | Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable for a namespace-admin install where a cluster-admin applies them separately; the gateway ServiceAccount name and namespace are unchanged, so a pre-created ClusterRoleBinding still matches. |
| rbac.create | bool | `true` | Create the RBAC objects that grant the gateway ServiceAccount access. Disable to supply the namespaced sandbox and peer Role/RoleBinding and the cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook and credential driver RBAC keep their own flags. |
| replicaCount | int | `1` | Number of OpenShell gateway replicas. Values greater than 1 require server.externalDbSecret because the default SQLite backend is per pod. |
| resources | object | `{}` | Gateway pod resource requests and limits. |
| sandbox.image.digest | string | `""` | Sandbox image digest. When set, this takes precedence over tag. |
| sandbox.image.pullPolicy | string | `nil` | Sandbox image pull policy. Leave unset to use the Kubernetes image default. |
| sandbox.image.repository | string | `"nvcr.io/nvidia/base/ubuntu"` | Default standalone sandbox image repository. |
| sandbox.image.tag | string | `"24.04"` | Sandbox image tag. Defaults to latest when empty. |
| sandboxRuntime.image.digest | string | `""` | Sandbox runtime image digest. When set, this takes precedence over tag. |
| sandboxRuntime.image.pullPolicy | string | `nil` | Sandbox runtime image pull policy. Empty uses global.image.pullPolicy. |
| sandboxRuntime.image.registry | string | `""` | Sandbox runtime image registry. Empty uses global.image.registry. |
| sandboxRuntime.image.repository | string | `"openshell/sandbox"` | Sandbox runtime image repository. |
| sandboxRuntime.image.tag | string | `""` | Sandbox runtime image tag. Defaults to the chart appVersion when empty. |
| sandboxServiceAccount.annotations | object | `{}` | Annotations to add to the generated sandbox service account. |
| sandboxServiceAccount.create | bool | `true` | Create a service account for sandbox pods. |
| sandboxServiceAccount.name | string | `""` | Existing service account name for sandbox pods when sandboxServiceAccount.create is false. |
| securityContext.allowPrivilegeEscalation | bool | `false` | Whether the gateway container can gain additional privileges. |
| securityContext.capabilities.drop | list | `["ALL"]` | Linux capabilities dropped from the gateway container. |
| securityContext.runAsNonRoot | bool | `true` | Require the gateway container to run as a non-root user. |
| securityContext.runAsUser | int | `1000` | UID assigned to the gateway container. |
| server.auth.allowUnauthenticatedUsers | bool | `false` | UNSAFE: accept unauthenticated CLI/user requests as a local developer principal. Intended only for trusted local Skaffold/k3d development or a fully trusted fronting proxy. Leave false for shared or production clusters. |
| server.credentialDrivers.kubernetesSecrets.createNamespace | bool | `false` | Create the credential namespace. Requires a namespace other than the release namespace. The Namespace is retained on uninstall so stored credentials survive; an existing Namespace not owned by this release is left untouched. |
| server.credentialDrivers.kubernetesSecrets.enabled | bool | `false` | Enable the in-tree Kubernetes Secret credential driver. WARNING: The RBAC Role grants read/write access to ALL Secrets in the configured namespace. Use a dedicated namespace to limit blast radius. |
| server.credentialDrivers.kubernetesSecrets.namespace | string | `""` | Namespace where OpenShell-managed provider Secret objects are stored. Empty = Helm release namespace. A dedicated namespace is RECOMMENDED to isolate OpenShell-managed Secrets from other workloads. |
| server.credentialDrivers.kubernetesSecrets.rbac.create | bool | `true` | Create a Role/RoleBinding granting the gateway ServiceAccount read/write access to managed provider Secrets. |
| server.credentialDrivers.vault.address | string | `""` | Vault service base URL. Non-loopback endpoints must use HTTPS, for example https://vault.vault.svc.cluster.local:8200. |
| server.credentialDrivers.vault.authMethod | string | `"kubernetes"` | Authentication method. Use "kubernetes" in-cluster or "token_file" for local/dev validation. |
| server.credentialDrivers.vault.caConfigMapName | string | `""` | ConfigMap containing the private Vault CA certificate bundle in the ca.crt key. Leave empty to use platform trust roots. |
| server.credentialDrivers.vault.enabled | bool | `false` | Enable the in-tree Vault credential driver. |
| server.credentialDrivers.vault.kubernetesAuthMount | string | `"kubernetes"` | Vault Kubernetes auth mount. |
| server.credentialDrivers.vault.kvVersion | string | `"2"` | Default KV engine version. Use "1" or "2". |
| server.credentialDrivers.vault.mount | string | `"secret"` | Default KV mount name. |
| server.credentialDrivers.vault.role | string | `""` | Vault Kubernetes auth role when authMethod is kubernetes. |
| server.credentialDrivers.vault.serviceAccountTokenPath | string | `"/var/run/secrets/kubernetes.io/serviceaccount/token"` | ServiceAccount token path used for Kubernetes auth. |
| server.credentialDrivers.vault.timeoutSecs | string | `""` | HTTP request timeout in seconds. Empty = driver default. |
| server.credentialDrivers.vault.tokenPath | string | `""` | Mounted token file path when authMethod is token_file. |
| server.credentialStorage.existingSecret | string | `""` | Name of a pre-existing Secret containing the key-encryption key. When set, the chart does NOT generate a new Secret; it references this one instead. The Secret must contain a key named "key-encryption-key" with a base64-encoded 32-byte value. Required for GitOps workflows that render manifests with `helm template` (where `lookup` is unavailable). |
| server.dbUrl | string | `"sqlite:/var/openshell/openshell.db"` | Gateway database URL (used for the default SQLite backend). SQLite runs in WAL mode and needs a local block-backed volume, not NFS or other network filesystems. |
| server.defaultRuntimeClassName | string | `""` | Default Kubernetes runtimeClassName for sandbox pods. Applied when a CreateSandbox request does not specify one. Empty (default) = omit the field, using the cluster's default RuntimeClass. Set to a RuntimeClass name (e.g. "kata-containers", "nvidia") to apply it to all sandboxes that don't explicitly override it. |
| server.disableTls | bool | `false` | Disable TLS entirely - the server listens on plaintext HTTP. Set to true when a reverse proxy / tunnel terminates TLS at the edge. |
| server.drivers.kubernetes.allowDriverConfig | bool | `false` | Allow caller driver JSON; external resources still require approval. |
| server.drivers.kubernetes.operatorNamespaceFile | string | `""` | Path to a JSON file containing an array of namespace names allowed in operator mode. Hot-reloaded on change. |
| server.drivers.kubernetes.operatorNamespaceLabel | string | `""` | K8s label selector for namespace discovery in operator mode. The driver watches namespaces matching this label. |
| server.drivers.kubernetes.resourceAdmission.enabled | bool | `true` | Require operator approval labels on external sandbox attachments (GPU attachments exempt). |
| server.drivers.kubernetes.resourceAdmission.requiredLabels | string | `nil` | Replacement label map; null uses the built-in admission labels. Empty map is invalid when enabled. |
| server.drivers.kubernetes.workspaceMode | string | `"shared"` | How workspaces map to Kubernetes namespaces. "shared" (default): all sandboxes in a single namespace. "managed": auto-creates per-workspace namespaces. "operator": uses pre-provisioned namespaces. |
| server.enableLoopbackServiceHttp | bool | `true` | Enable plaintext HTTP routing for loopback sandbox service URLs on TLS-enabled gateways. |
| server.enableUserNamespaces | bool | `false` | Enable Kubernetes user namespace isolation (hostUsers: false) for sandbox pods. Requires Kubernetes 1.33+ with user namespace support available (beta through 1.35, GA in 1.36+), plus a supporting container runtime and Linux 5.12+. When enabled, container UID 0 maps to an unprivileged host UID and capabilities become namespaced. |
| server.enableWebsocketTunnel | bool | `false` | Enable the WebSocket tunnel used by CLI/SDK clients behind an authenticated edge proxy. Leave disabled for direct gateway installs. |
| server.externalDbSecret | string | `""` | Name of a pre-existing Opaque Secret containing a PostgreSQL connection URI (key: uri). When set, the gateway reads OPENSHELL_DB_URL from this Secret instead of using dbUrl. The Secret must contain a `uri` key, e.g. postgresql://user:pass@host:5432/dbname. |
| server.extraVolumeMounts | list | `[]` | Additional volume mounts for the gateway container. |
| server.extraVolumes | list | `[]` | Additional volumes for the gateway pod. |
| server.grpcEndpoint | string | `""` | gRPC endpoint sandboxes call back into the gateway. Leave empty to derive it from the chart fullname, release namespace, service port, and disableTls flag, for example https://openshell.openshell.svc.cluster.local:8080. Override only when sandboxes must reach the gateway via a different hostname (e.g. an external ingress or a host alias). |
| server.grpcRateLimit.requests | int | `0` | Maximum gRPC requests allowed per window. Must be positive (alongside windowSeconds) to enable rate limiting; 0 (default) disables it. |
| server.grpcRateLimit.windowSeconds | int | `0` | gRPC rate-limit window length in seconds. Must be positive (alongside requests) to enable rate limiting; 0 (default) disables it. |
| server.hostGatewayIP | string | `""` | Host gateway IP for sandbox pod hostAliases. When set, sandbox pods get hostAliases entries mapping host.docker.internal and host.openshell.internal to this IP, allowing them to reach services running on the Docker host. Auto-detected by the cluster entrypoint script. |
| server.logLevel | string | `"info"` | Gateway log level. |
| server.name | string | `""` | Operator-facing gateway name. Defaults to the chart fullname so all replicas in one installation share an identity. Set explicitly when one telemetry collector receives spans from multiple namespaces or clusters. |
| server.ocsfLog.enabled | bool | `false` | Write gateway OCSF events as JSONL. |
| server.ocsfLog.maxFiles | int | `7` | Rotated files retained when rotation is daily. |
| server.ocsfLog.path | string | `"/tmp/gateway-ocsf.jsonl"` | OCSF JSONL path. The default is writable in the gateway container but does not persist across restarts. To keep records, mount a volume with server.extraVolumes and server.extraVolumeMounts and set a path on it. When replicas share the volume, set subPathExpr: $(OPENSHELL_POD_NAME) on the mount so each replica writes its own file. |
| server.ocsfLog.queueCapacity | int | `10000` | Maximum records waiting for the file writer. |
| server.ocsfLog.queueMaxBytes | int | `16777216` | Maximum encoded bytes waiting for the file writer. |
| server.ocsfLog.rotation | string | `"daily"` | Rotate the active file daily in UTC, or never. |
| server.ocsfLog.schemaVersion | string | `""` | Optional OCSF downgrade target. Empty emits native OCSF 1.8.0. Supported values: "1.1", "1.3". |
| server.oidc.adminRole | string | `""` | Role name for admin access. Leave empty (with userRole also empty) for authentication-only mode. Both must be set or both empty. |
| server.oidc.audience | string | `"openshell-cli"` | Expected audience claim for the API resource server. This should match the server's --oidc-audience, NOT the CLI client ID. |
| server.oidc.caConfigMapName | string | `""` | Name of a ConfigMap containing a CA certificate bundle (key: ca.crt) for verifying the OIDC issuer's TLS certificate. Required when the issuer uses a non-public CA (e.g. OpenShift ingress, private PKI). |
| server.oidc.dangerouslyAllowInsecureHttp | bool | `false` | Development only: permit cleartext OIDC requests to numeric loopback addresses. This never permits HTTP to hostnames or non-loopback addresses. |
| server.oidc.issuer | string | `""` | OIDC issuer URL (e.g. https://keycloak.example.com/realms/openshell). |
| server.oidc.jwksAllowedOrigins | list | `[]` | Additional trusted HTTPS origins allowed to serve JWKS. The issuer origin is always allowed. Entries must not include a path or query. |
| server.oidc.jwksTtl | int | `3600` | JWKS key cache TTL in seconds. Must be greater than zero. |
| server.oidc.rolesClaim | string | `""` | Dot-separated path to the roles array in the JWT claims. Keycloak: "realm_access.roles", Entra ID: "roles", Okta: "groups". |
| server.oidc.scopesClaim | string | `""` | Dot-separated path to the scopes array in the JWT claims. |
| server.oidc.userRole | string | `""` | Role name for standard user access. |
| server.otlp.endpoint | string | `""` | OTLP/gRPC collector endpoint, conventionally using port 4317. |
| server.otlp.serviceName | string | `""` | Gateway OpenTelemetry service name. Empty uses openshell-gateway. |
| server.policyValidationFailureMode | string | `"fail_closed"` | Posture when a candidate sandbox policy fails validation. `fail_closed` deactivates the previous policy; `retain_last_valid` keeps it active. |
| server.providerTokenGrants.spiffe.enabled | bool | `false` | Mount the SPIFFE Workload API socket into gateway and sandbox pods for dynamic provider token grants. |
| server.providerTokenGrants.spiffe.workloadApiSocketPath | string | `"/spiffe-workload-api/spire-agent.sock"` | Path to the SPIFFE Workload API socket mounted into gateway and sandbox pods. |
| server.sandboxGid | string | `""` | GID for sandbox pods (`sandbox_gid`). Empty (default) = same as sandboxUid. Must be an integer between 1 and 4294967294. |
| server.sandboxImagePullSecrets | list | `[]` | Image pull secrets attached to sandbox pods. Referenced Secrets must exist in the sandbox namespace. |
| server.sandboxJwt.gatewayId | string | `""` | Stable gateway identity embedded in iss/aud of every minted token. Defaults to the release name so HA replicas share identity. |
| server.sandboxJwt.k8sSaTokenTtlSecs | int | `3600` | Lifetime (seconds) of the projected ServiceAccount token kubelet writes into each sandbox pod for the IssueSandboxToken bootstrap exchange. Kubelet enforces a minimum of 600s; the driver clamps values outside [600, 86400]. Default 3600 — generous, since the supervisor consumes the token within seconds of pod start. |
| server.sandboxJwt.secretDefaultMode | string | `""` | File mode for the mounted JWT signing key Secret. Default 0400 (owner-read only). Override to 0440 or 0444 if the container UID does not match the volume file owner. |
| server.sandboxJwt.signingSecretName | string | `""` | Name of the Opaque Secret holding the signing key material. Empty falls back to the chart fullname with "-jwt-keys" appended. |
| server.sandboxJwt.ttlSecs | int | `3600` | Token TTL in seconds. Defaults to 3600 (1h). |
| server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. |
| server.sandboxUid | string | `""` | UID for sandbox pods (`sandbox_uid`). Empty (default) = use the OpenShift SCC namespace annotation if present, otherwise the driver default. Must be an integer between 1 and 4294967294. |
| server.telemetryEnabled | bool | `true` | Enable anonymous OpenShell telemetry from the gateway and the sandbox supervisors it launches. |
| server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. |
| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Only used when enableMtls is true. Set to "" to disable client certificate verification for HTTPS-only mode. |
| server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. |
| server.tls.enableMtls | bool | `true` | Enable mTLS client certificate authentication. When false, the gateway runs HTTPS-only without requiring client certificates (use OIDC for auth instead). Must be false when using BackendTLSPolicy because ingress proxies cannot present client certificates to the backend. |
| server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). |
| server.workspaceStorageClass | string | `""` | Kubernetes StorageClass for the workspace PVC in sandbox pods. Empty (default) = omit storageClassName, using the cluster's default StorageClass. Set this on clusters with no default StorageClass, otherwise the workspace PVC stays Pending and the sandbox never starts. |
| service.healthPort | int | `8081` | Gateway health service port. |
| service.metricsPort | int | `9090` | Gateway metrics service port. |
| service.port | int | `8080` | Gateway gRPC/HTTP service port. |
| service.type | string | `"ClusterIP"` | Kubernetes Service type for the gateway. |
| serviceAccount.annotations | object | `{}` | Annotations to add to the generated service account. |
| serviceAccount.create | bool | `true` | Create a service account for the gateway. |
| serviceAccount.name | string | `""` | Existing service account name to use when serviceAccount.create is false. |
| supervisor.image.digest | string | `""` | Supervisor image digest. When set, this takes precedence over tag. |
| supervisor.image.pullPolicy | string | `nil` | Supervisor image pull policy. Empty uses global.image.pullPolicy. Prefer always, if_not_present, or never; the chart also accepts legacy Kubernetes spellings Always, IfNotPresent, and Never. |
| supervisor.image.registry | string | `""` | Supervisor image registry. Empty uses global.image.registry. |
| supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. |
| supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. |
| supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. |
| tolerations | list | `[]` | Tolerations for the gateway pod. |
| upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. |
| upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. |
| upstreamProxy.authSecret.key | string | `""` | Secret key containing the proxy credential. |
| upstreamProxy.authSecret.name | string | `""` | Existing Secret in the sandbox namespace containing a user:pass value. |
| upstreamProxy.caBundle.configMapName | string | `""` | ConfigMap in the release namespace holding the corporate proxy CA bundle. Required for an https:// proxy with a private CA, and for a TLS-intercepting proxy that re-signs upstream certificates. The gateway reads it and stages it into each sandbox's immutable supervisor bootstrap Secret, so the anchor stays in the gateway's trust domain rather than the workload namespace. Supply only the CA that signs your proxy's certificate, or that the proxy re-signs intercepted upstream certificates with. Public roots already come from the supervisor image and its TLS stack, so a full merged trust bundle (for example an OpenShift config.openshift.io/inject-trusted-cabundle ConfigMap) adds hundreds of kilobytes of duplicated roots and can exceed the sandbox boundary's control-frame budget. |
| upstreamProxy.caBundle.key | string | `"ca.crt"` | Key inside that ConfigMap. Change this only to reuse an existing ConfigMap whose key is not ca.crt. |
| upstreamProxy.connectByHostname | bool | `false` | Last-resort option for hostname-filtering proxy ACLs. It lets the proxy resolve CONNECT targets. |
| upstreamProxy.noProxy | string | `""` | Comma-separated destinations that bypass only the corporate proxy. |
| upstreamProxy.url | string | `""` | Proxy URL in http://host:port or https://host:port form. An https:// proxy whose certificate is not publicly trusted also needs caBundle below. |
| workload.allowMultiReplicaStatefulSet | bool | `false` | Allow replicaCount > 1 while rendering a StatefulSet. Prefer workload.kind=deployment for external database-backed multi-replica gateways; this override exists for operators who explicitly require StatefulSet identity or storage semantics. |
| workload.kind | string | `"statefulset"` | Gateway workload controller kind. Use `statefulset` for the default SQLite database, or `deployment` when server.externalDbSecret points at an external database. |
| workspaceResources.enabled | bool | `true` | Create the sandbox ServiceAccount, Role, RoleBinding, and NetworkPolicy from this chart. Disable for a gateway-only release. |
----------------------------------------------
Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)
Reference in New Issue View Git Blame Copy Permalink