mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-16 05:20:22 +08:00
* feat(credentials): add provider credential storage drivers Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(credentials): harden credential update handling Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(credentials): harden credential driver security, correctness, and performance Address review findings from the credential storage drivers PR: - Route additional_credentials through the driver on refresh to prevent silent data loss for multi-credential providers (e.g. AWS STS) - Clean up stored credential handles on CAS failure during refresh to prevent orphaned secrets in external backends - Enforce namespace validation in the Kubernetes Secrets driver to prevent cross-namespace credential access when allow_reference_namespace is not enabled - Cache Vault Kubernetes auth tokens with 80% TTL to avoid re-authenticating on every credential operation - Parallelize resolve_credentials in all three drivers using try_join_all for faster sandbox startup - Add existingSecret support for the KEK Secret to fix helm template/GitOps workflows where lookup returns empty and regenerates the key - Document RBAC blast radius for the Kubernetes Secrets credential driver and recommend a dedicated namespace Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): add optimistic concurrency, fix thundering herd, parallelize operations Use resourceVersion optimistic concurrency with retry loop for K8s Secret ownership checks to prevent TOCTOU races. Switch Vault token cache from RwLock to Mutex with double-check pattern to prevent thundering herd on cache miss. Parallelize credential store and delete operations across independent keys using try_join_all. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): handle partial failures, add delete retry, consolidate cleanup Replace try_join_all with join_all in credential store/delete operations to handle partial failures — successfully-stored handles are cleaned up when another key fails. Add retry loop with conflict detection to db-credstore delete_credential, matching the K8s driver pattern. Consolidate 4 manual cleanup_pre_stored_provider_credentials call sites into a single error handler using an async block. Remove inconsistent .trim() from db-credstore validate_handle_owner. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): fix retry loop guard and remove unprotected validation Remove attempt-count guard from 409/Aborted match arms in retry loops so the post-loop Status::aborted error is reachable after exhausting retries. Previously, last-attempt conflicts fell through to the catch-all error arm, producing misleading Status::unavailable errors. Remove duplicate validation calls that ran after prepare_provider_credential_update but outside the cleanup-protected async block, which would leak pre-stored handles on failure. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): add workspace/provider UUID to credential backend paths Include workspace and provider ID in credential backend object paths to ensure cross-workspace uniqueness and prevent credential collision (GATOR-1806c9be-01). - Updated credential driver proto to include workspace and provider_id fields - Modified Vault driver to include workspace/provider_id in managed_secret_path - Modified Kubernetes Secrets driver to include workspace/provider_id in credential_owner_id and managed_secret_name - Updated all credential runtime calls to pass workspace/provider_id - Updated tests to use the new signatures This prevents two workspaces sharing the same external credential store from colliding on provider names, which was a critical security issue (CWE-639). * fix(credentials): preserve provider-level expiration for handle-backed credentials Compute effective expiration from both provider and driver values using the earliest non-zero timestamp and skip expired values before insertion (GATOR-1806c9be-02). - Modified resolve_provider_handles to check provider credential_expires_at_ms - Skip expired credentials during resolution instead of returning them - Use effective expiration (min of provider and driver) in resolution results - Fix inference.rs to preserve earliest expiration when merging This ensures handle-backed credentials respect the same expiration semantics as inline credentials. * fix(credentials): stage refresh changes under new handles before validation Stage credential replacements under new immutable handles instead of reusing existing handles to prevent overwriting committed values before validation/CAS (GATOR-1806c9be-03). - Stage credentials with empty existing_handles map to force new handle creation - Validate and CAS before the new values are committed to backend storage - Delete old handles only after successful CAS - On CAS failure, delete only the newly staged handles - This prevents CWE-362/CWE-367 race conditions where failed refreshes could still modify or delete the active credential The fix ensures that a rejected refresh cannot modify the backend object still referenced by the committed provider record. * fix(credentials): add timeouts to credential driver RPCs Apply configured timeouts to both startup capability negotiation and runtime RPCs to prevent indefinite hangs (GATOR-1806c9be-05). - Add DEFAULT_CREDENTIAL_DRIVER_RPC_TIMEOUT_SECS constant (30s) - Apply timeout to GetCapabilities during startup connection - Apply timeout to all runtime RPCs (store, delete, resolve) - Use tokio::time::timeout to bound the entire GetCapabilities operation during startup, not just the socket connection - Return contextual deadline errors on timeout This prevents a faulty or overloaded driver from hanging gateway operations indefinitely. * fix(credentials): fix test to use consistent workspace/provider identity The Kubernetes auth Vault resolve test was constructing a managed path with test-workspace/test-provider-id but sending default/prov-123 in the request, causing validation to reject the request (GATOR-18e32351-01). - Update test to use test-workspace and test-provider-id in the request to match the logical_path construction - This ensures the test exercises the intended code path and validates Kubernetes auth resolution properly The test now passes and correctly validates identity enforcement. * fix(credentials): use unique staging ID for refresh to avoid overwrites Stage refresh replacements under genuinely distinct immutable handles using a unique staging ID to prevent overwriting committed values (GATOR-1806c9be-03). - Generate a unique staging ID using UUID for each refresh operation - Use this staging ID when storing credentials instead of the real provider ID - Pass the same staging ID during cleanup on failure to delete only staged objects - This ensures deterministic paths (Vault) and object names (K8s) don't collide with the committed provider's credentials The fix prevents failed refreshes from silently replacing active credentials or breaking providers by deleting still-referenced backend objects. * fix(credentials): wrap credential driver RPCs in local timeouts Add local tokio::time::timeout wrappers around credential driver RPCs to bound non-compliant or stalled UDS peers (GATOR-1806c9be-05). - Wrap StoreCredential, DeleteCredential, and ResolveCredentials in local timeouts - Return contextual deadline_exceeded errors when timeouts occur - Keep existing gRPC timeout metadata for compliant implementations - GetCapabilities during startup was already wrapped in previous commit This ensures a faulty local driver cannot hang gateway operations indefinitely, even if it accepts the connection but never responds to the RPC. * fix(credentials): preserve ownership for staged refreshes Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(credentials): bound startup capability probe Signed-off-by: Seth Jennings <sjenning@redhat.com> * test(provider): authenticate credential handler requests Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(ci): grant actions read to credential driver e2e Signed-off-by: Seth Jennings <sjenning@redhat.com> --------- Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> Signed-off-by: Seth Jennings <sjenning@redhat.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
153 lines
6.2 KiB
Go Template
153 lines
6.2 KiB
Go Template
# OpenShell Helm Chart
|
|
|
|
<!--
|
|
This file is generated by helm-docs.
|
|
Edit README.md.gotmpl and values.yaml, then run `mise run helm:docs`.
|
|
-->
|
|
|
|
> **Experimental** - the Kubernetes deployment path is under active development. Expect rough edges and breaking changes.
|
|
|
|
This chart deploys the OpenShell gateway into a Kubernetes cluster. It is published as an OCI artifact to GHCR at `oci://ghcr.io/nvidia/openshell/helm-chart`.
|
|
|
|
## Prerequisites
|
|
|
|
The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with:
|
|
|
|
```shell
|
|
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/manifest.yaml
|
|
```
|
|
|
|
## Install on Kubernetes
|
|
|
|
```shell
|
|
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version>
|
|
```
|
|
|
|
## Install on OpenShift
|
|
|
|
See the full [OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift) for details. Quick start:
|
|
|
|
```shell
|
|
# Precreate the openshell namespace so we can create the SCC cluster role
|
|
oc create ns openshell
|
|
|
|
# Sandboxes are deployed into the openshell namespace and use the openshell-sandbox service account
|
|
oc adm policy add-scc-to-user privileged -z openshell-sandbox -n openshell
|
|
|
|
# Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway
|
|
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> -n openshell \
|
|
--set server.disableTls=true \
|
|
--set podSecurityContext.fsGroup=null \
|
|
--set securityContext.runAsUser=null
|
|
```
|
|
|
|
## Available versions
|
|
|
|
| Tag | Source | Notes |
|
|
| --- | --- | --- |
|
|
| `<semver>` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway and supervisor image versions. Recommended for production. |
|
|
| `0.0.0-dev` | Latest commit on `main` | Floating tag, overwritten on every push. `appVersion` is `dev`, so images resolve to the `:dev` tag. |
|
|
| `0.0.0-dev.<commit-sha>` | A specific commit on `main` | Per-commit pin. Chart version and `appVersion` both use the full 40-character commit SHA, which matches the image tag pushed by CI. |
|
|
|
|
The `dev` tags are intended for testing changes ahead of a release. Production deployments should pin to a tagged release.
|
|
|
|
## Configuration
|
|
|
|
See [`values.yaml`](values.yaml) for source defaults. Selected overlays:
|
|
|
|
- [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - gateway-only configuration
|
|
- [`ci/values-cert-manager.yaml`](ci/values-cert-manager.yaml) - cert-manager integration
|
|
- [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration
|
|
- [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing
|
|
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
|
|
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development
|
|
|
|
### Database backend
|
|
|
|
By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the
|
|
database is backed by a per-pod PVC:
|
|
|
|
```yaml
|
|
server:
|
|
dbUrl: "sqlite:/var/openshell/openshell.db"
|
|
```
|
|
|
|
#### External PostgreSQL
|
|
|
|
Use external PostgreSQL when the gateway should connect to a database managed
|
|
outside this chart. The OpenShell chart does not deploy a database; install
|
|
PostgreSQL separately using the chart, operator, or managed service that fits
|
|
your environment, then pass the connection URI through a Secret.
|
|
|
|
Create a Secret containing the PostgreSQL connection URI if one does not
|
|
already exist:
|
|
|
|
```bash
|
|
kubectl create secret generic my-pg-credentials -n openshell \
|
|
--from-literal=uri="postgresql://user:pass@host:5432/dbname"
|
|
```
|
|
|
|
Then install the chart pointing at that Secret:
|
|
|
|
```bash
|
|
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
|
|
-n openshell \
|
|
--set workload.kind=deployment \
|
|
--set server.externalDbSecret=my-pg-credentials
|
|
```
|
|
|
|
Use `workload.kind=deployment` for external database-backed multi-replica
|
|
gateways. `workload.kind=statefulset` is still available for single-replica
|
|
SQLite installs and for operators who explicitly need StatefulSet identity or
|
|
storage semantics.
|
|
|
|
### Credential storage
|
|
|
|
By default, the chart uses the gateway's encrypted database credential storage.
|
|
The gateway writes encrypted provider credential envelopes to the OpenShell
|
|
database. The chart creates a retained Kubernetes Secret with the shared
|
|
key-encryption key and injects that key into every gateway pod, so the same
|
|
default works for single-replica and external database-backed HA deployments.
|
|
|
|
Use `kubernetes-secrets` or `vault` instead when credentials should live in a
|
|
cluster or external secret backend. Enabling one external credential driver
|
|
disables the default credential-storage key-encryption key Secret and env injection.
|
|
|
|
#### OpenShift
|
|
|
|
Append these flags to any of the PostgreSQL commands above for OpenShift:
|
|
|
|
```
|
|
--set server.disableTls=true \
|
|
--set podSecurityContext.fsGroup=null \
|
|
--set securityContext.runAsUser=null
|
|
```
|
|
|
|
## Secret bootstrap
|
|
|
|
By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs`
|
|
to create the gateway's server/client mTLS Secrets and sandbox JWT signing Secret.
|
|
The Job uses the gateway image itself, so air-gapped environments only need to
|
|
mirror that one image (no separate openssl/alpine sidecar).
|
|
|
|
When `certManager.enabled=true`, cert-manager owns the TLS Secrets and the chart
|
|
runs the same hook in JWT-only mode because cert-manager does not create the
|
|
sandbox JWT signing Secret. This precedence applies even if
|
|
`pkiInitJob.enabled` remains true. Set `pkiInitJob.enabled=false` only when an
|
|
external non-cert-manager TLS source manages TLS and you pre-create the sandbox
|
|
JWT signing Secret.
|
|
|
|
## SPIFFE/SPIRE provider token grants
|
|
|
|
Set `server.providerTokenGrants.spiffe.enabled=true` to let sandbox supervisors
|
|
use SPIFFE JWT-SVIDs for dynamic provider token grants. The chart keeps
|
|
supervisor-to-gateway authentication on gateway-minted sandbox JWTs and passes
|
|
the SPIFFE Workload API socket path to the Kubernetes driver so sandbox pods can
|
|
mount the SPIFFE CSI socket.
|
|
|
|
For local development, uncomment the SPIRE Helm releases in `skaffold.yaml` and
|
|
add `ci/values-spire.yaml` to the OpenShell release values files.
|
|
|
|
{{ template "chart.valuesSection" . }}
|
|
{{ template "helm-docs.versionFooter" . }}
|