mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-02 07:34:45 +08:00
fix(kubernetes): remove NetworkPolicy acknowledgement (#3677)
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
This commit is contained in:
@@ -109,7 +109,6 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--namespace openshell --create-namespace \
|
||||
--set server.disableTls=true \
|
||||
--set server.telemetryEnabled=false \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--wait --timeout 5m
|
||||
|
||||
kubectl wait --namespace openshell \
|
||||
|
||||
@@ -359,7 +359,6 @@ jobs:
|
||||
--set server.disableTls=true \
|
||||
--set server.auth.allowUnauthenticatedUsers=true \
|
||||
--set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}" \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--wait --timeout 5m
|
||||
|
||||
- name: Verify gateway pod is Ready
|
||||
|
||||
@@ -41,12 +41,14 @@ The installer installs the latest stable release by default. See [Prerelease and
|
||||
**Kubernetes installation:**
|
||||
|
||||
> **Experimental** — the Kubernetes deployment path is under active development. Expect rough edges and breaking changes.
|
||||
> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
|
||||
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
|
||||
> but Kubernetes does not verify that the CNI applies them.
|
||||
|
||||
Deploy the OpenShell gateway into a Kubernetes cluster from the OCI chart published to GHCR:
|
||||
|
||||
```bash
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart
|
||||
```
|
||||
|
||||
See [`deploy/helm/openshell/README.md`](deploy/helm/openshell/README.md) for available versions, dev tag conventions, and configuration.
|
||||
|
||||
@@ -301,7 +301,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
|
||||
|---|---|---|---|
|
||||
| Docker | Local development with Docker available. | Capability-free workload container. | Uses `network_mode=none`; a separate capability-free supervisor container mediates egress and access over a private daemon-local Unix socket volume. |
|
||||
| Podman | Existing rootless driver. | Container. | Not converted by this isolation stack. |
|
||||
| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Uses one namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace. |
|
||||
| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Always creates a namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace; the Kubernetes API does not attest policy enforcement. |
|
||||
| VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. |
|
||||
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
|
||||
|
||||
|
||||
@@ -81,9 +81,9 @@ and permits OpenShell supervisor Pods to reach the sandbox TLS port. The
|
||||
authenticated Sandbox Protocol binds each connection to the exact sandbox and
|
||||
supervisor Pod identities. Supervisors have normal egress for gateway, DNS,
|
||||
and policy-approved upstream connections unless an operator policy restricts
|
||||
them. Set
|
||||
`sandbox_runtime.network_policy_enforced = true` only after verifying that the cluster
|
||||
CNI enforces ingress and egress `NetworkPolicy` for sandbox namespaces.
|
||||
them. The cluster CNI must enforce ingress and egress `NetworkPolicy` for every
|
||||
sandbox namespace. Kubernetes accepts policy objects without confirming
|
||||
enforcement, so operators must verify CNI support before running sandboxes.
|
||||
|
||||
Each sandbox generation uses two immutable bootstrap Secrets. A trusted init
|
||||
container stages the sandbox bootstrap into memory, and the sandbox removes it
|
||||
|
||||
@@ -78,9 +78,6 @@ pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi";
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(default, deny_unknown_fields)]
|
||||
pub struct KubernetesSandboxRuntimeConfig {
|
||||
/// Explicit operator assertion that the cluster CNI enforces
|
||||
/// `networking.k8s.io/v1` `NetworkPolicy` for the sandbox namespaces.
|
||||
pub network_policy_enforced: bool,
|
||||
/// TCP port exposed by the workload boundary to its paired control pod.
|
||||
pub boundary_port: u16,
|
||||
}
|
||||
@@ -88,7 +85,6 @@ pub struct KubernetesSandboxRuntimeConfig {
|
||||
impl Default for KubernetesSandboxRuntimeConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
network_policy_enforced: false,
|
||||
boundary_port: 5500,
|
||||
}
|
||||
}
|
||||
@@ -96,12 +92,6 @@ impl Default for KubernetesSandboxRuntimeConfig {
|
||||
|
||||
impl KubernetesSandboxRuntimeConfig {
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if !self.network_policy_enforced {
|
||||
return Err(
|
||||
"sandbox_runtime.network_policy_enforced must be true after the operator has verified CNI NetworkPolicy enforcement"
|
||||
.to_string(),
|
||||
);
|
||||
}
|
||||
if self.boundary_port < 1024 {
|
||||
return Err("sandbox_runtime.boundary_port must be at least 1024".to_string());
|
||||
}
|
||||
@@ -887,19 +877,6 @@ mod tests {
|
||||
assert!(cfg.workspace_storage_class.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sandbox_runtime_requires_network_policy_enforcement_acknowledgement() {
|
||||
let mut cfg = KubernetesComputeConfig::default();
|
||||
assert!(
|
||||
cfg.validate_proxy_uid()
|
||||
.unwrap_err()
|
||||
.contains("network_policy_enforced")
|
||||
);
|
||||
|
||||
cfg.sandbox_runtime.network_policy_enforced = true;
|
||||
cfg.validate_proxy_uid().unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serde_rejects_sidecar_binary_identity_field() {
|
||||
let json = serde_json::json!({
|
||||
|
||||
@@ -128,13 +128,6 @@ struct Args {
|
||||
#[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY")]
|
||||
supervisor_image_pull_policy: Option<KubernetesImagePullPolicy>,
|
||||
|
||||
#[arg(
|
||||
long,
|
||||
env = "OPENSHELL_K8S_SANDBOX_RUNTIME_NETWORK_POLICY_ENFORCED",
|
||||
default_value_t = false
|
||||
)]
|
||||
sandbox_runtime_network_policy_enforced: bool,
|
||||
|
||||
#[arg(
|
||||
long,
|
||||
env = "OPENSHELL_K8S_SANDBOX_RUNTIME_BOUNDARY_PORT",
|
||||
@@ -270,7 +263,6 @@ async fn main() -> Result<()> {
|
||||
.unwrap_or_else(openshell_core::config::default_supervisor_image),
|
||||
supervisor_image_pull_policy: args.supervisor_image_pull_policy,
|
||||
sandbox_runtime: KubernetesSandboxRuntimeConfig {
|
||||
network_policy_enforced: args.sandbox_runtime_network_policy_enforced,
|
||||
boundary_port: args.sandbox_runtime_boundary_port,
|
||||
},
|
||||
https_proxy: args.https_proxy,
|
||||
|
||||
@@ -19,6 +19,12 @@ multiple pre-provisioned workspace namespaces.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
|
||||
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
|
||||
> but Kubernetes accepts them even if no CNI enforces them. Without enforcement,
|
||||
> sandbox workloads may connect directly and bypass supervisor network policy.
|
||||
> Verify CNI support before installing OpenShell.
|
||||
|
||||
The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with:
|
||||
|
||||
```shell
|
||||
@@ -35,8 +41,7 @@ where Helm cannot discover cluster APIs.
|
||||
## Install on Kubernetes
|
||||
|
||||
```shell
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version>
|
||||
```
|
||||
|
||||
## Install on OpenShift
|
||||
@@ -49,7 +54,6 @@ oc create ns openshell
|
||||
|
||||
# Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> -n openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set server.disableTls=true \
|
||||
--set podSecurityContext.fsGroup=null \
|
||||
--set securityContext.runAsUser=null
|
||||
@@ -110,7 +114,6 @@ Then install the chart pointing at that Secret:
|
||||
```bash
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
|
||||
-n openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set workload.kind=deployment \
|
||||
--set server.externalDbSecret=my-pg-credentials
|
||||
```
|
||||
@@ -352,7 +355,6 @@ discovery endpoint or its TLS CA.
|
||||
| supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. |
|
||||
| supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. |
|
||||
| supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. |
|
||||
| supervisor.sandboxRuntime.networkPolicyEnforced | bool | `false` | Required operator acknowledgement that the cluster CNI enforces NetworkPolicy. |
|
||||
| tolerations | list | `[]` | Tolerations for the gateway pod. |
|
||||
| upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. |
|
||||
| upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. |
|
||||
|
||||
@@ -19,6 +19,12 @@ multiple pre-provisioned workspace namespaces.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
|
||||
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
|
||||
> but Kubernetes accepts them even if no CNI enforces them. Without enforcement,
|
||||
> sandbox workloads may connect directly and bypass supervisor network policy.
|
||||
> Verify CNI support before installing OpenShell.
|
||||
|
||||
The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with:
|
||||
|
||||
```shell
|
||||
@@ -35,8 +41,7 @@ where Helm cannot discover cluster APIs.
|
||||
## Install on Kubernetes
|
||||
|
||||
```shell
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version>
|
||||
```
|
||||
|
||||
## Install on OpenShift
|
||||
@@ -49,7 +54,6 @@ oc create ns openshell
|
||||
|
||||
# Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> -n openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set server.disableTls=true \
|
||||
--set podSecurityContext.fsGroup=null \
|
||||
--set securityContext.runAsUser=null
|
||||
@@ -110,7 +114,6 @@ Then install the chart pointing at that Secret:
|
||||
```bash
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
|
||||
-n openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set workload.kind=deployment \
|
||||
--set server.externalDbSecret=my-pg-credentials
|
||||
```
|
||||
|
||||
@@ -8,8 +8,7 @@
|
||||
#
|
||||
# Then layer this file on top of values.yaml when deploying:
|
||||
# helm upgrade --install openshell . \
|
||||
# -f values.yaml -f ci/values-skaffold.yaml -f ci/values-keycloak.yaml \
|
||||
# --set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
# -f values.yaml -f ci/values-skaffold.yaml -f ci/values-keycloak.yaml
|
||||
#
|
||||
# Or add this file to skaffold.yaml valuesFiles for iterative dev.
|
||||
#
|
||||
|
||||
@@ -4,8 +4,7 @@
|
||||
# OpenShift SCC compatibility overlay. Removes the hardcoded runAsUser and
|
||||
# fsGroup so that OpenShift's restricted-v2 SCC can inject the namespace-
|
||||
# assigned UID/GID range. Layer after values.yaml:
|
||||
# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml \
|
||||
# --set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml
|
||||
#
|
||||
# The e2e Kubernetes harness applies this automatically when it detects an
|
||||
# OpenShift cluster (route.openshift.io API present).
|
||||
|
||||
@@ -20,4 +20,3 @@ supervisor:
|
||||
# The local k3s cluster created by `mise run helm:k3s:create` enables its
|
||||
# built-in NetworkPolicy controller for sandbox namespaces.
|
||||
sandboxRuntime:
|
||||
networkPolicyEnforced: true
|
||||
|
||||
@@ -232,7 +232,6 @@ data:
|
||||
gateway_pod_selector = { "app.kubernetes.io/name" = {{ include "openshell.name" . | quote }}, "app.kubernetes.io/instance" = {{ .Release.Name | quote }} }
|
||||
|
||||
[openshell.drivers.kubernetes.sandbox_runtime]
|
||||
network_policy_enforced = {{ .Values.supervisor.sandboxRuntime.networkPolicyEnforced }}
|
||||
boundary_port = {{ .Values.supervisor.sandboxRuntime.boundaryPort | default 5500 }}
|
||||
|
||||
{{- if not $credentialDrivers }}
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
{{- if not .Values.supervisor.sandboxRuntime.networkPolicyEnforced }}
|
||||
{{- fail "supervisor.sandboxRuntime.networkPolicyEnforced must be true after you verify that the cluster CNI enforces ingress and egress NetworkPolicy in every sandbox namespace" }}
|
||||
{{- end }}
|
||||
@@ -1,20 +0,0 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
suite: NetworkPolicy enforcement acknowledgement
|
||||
templates:
|
||||
- templates/network-policy-ack.yaml
|
||||
tests:
|
||||
- it: rejects an install without operator acknowledgement
|
||||
set:
|
||||
supervisor.sandboxRuntime.networkPolicyEnforced: false
|
||||
asserts:
|
||||
- failedTemplate:
|
||||
errorMessage: supervisor.sandboxRuntime.networkPolicyEnforced must be true after you verify that the cluster CNI enforces ingress and egress NetworkPolicy in every sandbox namespace
|
||||
|
||||
- it: accepts an acknowledged CNI
|
||||
set:
|
||||
supervisor.sandboxRuntime.networkPolicyEnforced: true
|
||||
asserts:
|
||||
- hasDocuments:
|
||||
count: 0
|
||||
@@ -70,8 +70,6 @@ supervisor:
|
||||
# -- Supervisor image digest. When set, this takes precedence over tag.
|
||||
digest: ""
|
||||
sandboxRuntime:
|
||||
# -- Required operator acknowledgement that the cluster CNI enforces NetworkPolicy.
|
||||
networkPolicyEnforced: false
|
||||
# -- Workload boundary TLS listener port.
|
||||
boundaryPort: 5500
|
||||
|
||||
|
||||
@@ -11,7 +11,6 @@ trap 'rm -rf "${work_dir}"' EXIT
|
||||
helm template openshell "${repo_root}/deploy/helm/openshell" \
|
||||
--namespace openshell \
|
||||
--set agentSandbox.preflight.enabled=false \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set workspaceResources.enabled=false \
|
||||
>"${work_dir}/gateway.yaml"
|
||||
|
||||
@@ -41,7 +40,6 @@ fi
|
||||
helm template openshell "${repo_root}/deploy/helm/openshell" \
|
||||
--namespace openshell \
|
||||
--set agentSandbox.preflight.enabled=false \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set-json workspaceResources=null \
|
||||
>"${work_dir}/legacy-reuse-values.yaml"
|
||||
|
||||
|
||||
@@ -90,7 +90,6 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--values values-ha.yaml \
|
||||
--wait
|
||||
```
|
||||
@@ -196,7 +195,6 @@ helm upgrade openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--values values-ha.yaml \
|
||||
--wait
|
||||
```
|
||||
|
||||
@@ -57,7 +57,6 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set grpcRoute.enabled=true \
|
||||
--set grpcRoute.gateway.create=true \
|
||||
--set grpcRoute.gateway.className=eg
|
||||
@@ -114,7 +113,6 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set grpcRoute.enabled=true \
|
||||
--set grpcRoute.gateway.create=true \
|
||||
--set grpcRoute.gateway.className=eg \
|
||||
|
||||
@@ -52,7 +52,6 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set certManager.enabled=true
|
||||
```
|
||||
|
||||
@@ -74,7 +73,6 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set certManager.enabled=true \
|
||||
--set certManager.serverIssuerRef.name=letsencrypt-prod \
|
||||
--set certManager.serverIssuerRef.kind=ClusterIssuer \
|
||||
|
||||
@@ -41,17 +41,23 @@ in the sandbox qualification output.
|
||||
- [Agent Sandbox](/kubernetes/setup#install-agent-sandbox) controller and CRDs.
|
||||
- A CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces.
|
||||
|
||||
<Warning>
|
||||
Your cluster MUST enforce ingress and egress `NetworkPolicy` in every sandbox
|
||||
namespace. OpenShell creates the policies, but Kubernetes does not verify that
|
||||
the CNI applies them. Without enforcement, sandbox workloads may bypass
|
||||
supervisor network policy through direct connections.
|
||||
</Warning>
|
||||
|
||||
## Install OpenShell
|
||||
|
||||
Pre-create the namespace, then install the chart. Keep the default restricted
|
||||
security posture and acknowledge NetworkPolicy only after validating the CNI.
|
||||
security posture.
|
||||
|
||||
```shell
|
||||
oc create ns openshell
|
||||
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
--namespace openshell
|
||||
```
|
||||
|
||||
The driver reads the namespace's `openshift.io/sa.scc.uid-range` annotation and
|
||||
|
||||
@@ -81,9 +81,10 @@ restricts them.
|
||||
|
||||
Kubernetes policies are additive. Keep sandbox namespaces under administrative
|
||||
control so another principal cannot add permissive policies, create Pods with
|
||||
OpenShell labels, or read bootstrap Secrets. Set
|
||||
`supervisor.sandboxRuntime.networkPolicyEnforced: true` only after you verify that the
|
||||
cluster CNI enforces both ingress and egress policies for these namespaces.
|
||||
OpenShell labels, or read bootstrap Secrets. The cluster CNI MUST enforce both
|
||||
ingress and egress `NetworkPolicy` in every sandbox namespace. Kubernetes accepts
|
||||
policy objects even when no CNI enforces them; OpenShell does not test traffic
|
||||
to verify enforcement.
|
||||
|
||||
## Bootstrap a Sandbox
|
||||
|
||||
|
||||
@@ -12,6 +12,14 @@ position: 1
|
||||
The OpenShell Helm chart is experimental and under active development. Templates, values, and defaults can change between releases. Do not use it in production.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
Your cluster MUST use a CNI that enforces Kubernetes `NetworkPolicy` for both
|
||||
ingress and egress in every sandbox namespace. OpenShell creates the policies,
|
||||
but Kubernetes accepts them even when no CNI enforces them. Without enforcement,
|
||||
sandbox workloads may reach the network directly and bypass supervisor policy.
|
||||
Verify CNI support before installing OpenShell.
|
||||
</Warning>
|
||||
|
||||
Use the Kubernetes deployment when the gateway should run on a shared cluster, in a cloud environment, or as part of team infrastructure. The Helm chart handles PKI bootstrap, RBAC, sandbox namespace setup, and the gateway workload. It uses a StatefulSet by default for the SQLite database, and can render a Deployment when `server.externalDbSecret` points at an external database.
|
||||
|
||||
## Prerequisites
|
||||
@@ -21,6 +29,7 @@ Make sure the following are in place before you install.
|
||||
| Prerequisite | Required | Notes |
|
||||
|---|---|---|
|
||||
| Kubernetes 1.29+ with RBAC enabled | Yes | No additional notes. |
|
||||
| CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces | Yes | Verify enforcement on your cluster. |
|
||||
| Helm 3.x | Yes | No additional notes. |
|
||||
| Agent Sandbox controller and CRDs | Yes | Install before the OpenShell chart. Refer to [Install Agent Sandbox](#install-agent-sandbox). |
|
||||
| cert-manager | No | Refer to [Managing Certificates](/kubernetes/managing-certificates). Use cert-manager only if you prefer it over the built-in PKI job. |
|
||||
@@ -85,8 +94,7 @@ Install from the OCI registry on GHCR. Replace `<version>` with the chart versio
|
||||
helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
--namespace openshell
|
||||
```
|
||||
|
||||
To use the latest development build instead of a stable release:
|
||||
@@ -95,8 +103,7 @@ To use the latest development build instead of a stable release:
|
||||
helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version 0.0.0-dev \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
--namespace openshell
|
||||
```
|
||||
|
||||
The chart automatically generates PKI secrets on first install using pre-install Helm hooks. No manual secret creation is required.
|
||||
@@ -111,7 +118,6 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set workspaceResources.enabled=false \
|
||||
--set server.sandboxNamespace=app-a
|
||||
|
||||
@@ -149,7 +155,6 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set server.credentialDrivers.kubernetesSecrets.enabled=true \
|
||||
--set server.credentialDrivers.kubernetesSecrets.namespace=openshell-credentials \
|
||||
--set server.credentialDrivers.kubernetesSecrets.createNamespace=true
|
||||
@@ -237,7 +242,6 @@ The most commonly changed values are:
|
||||
| `server.auth.allowUnauthenticatedUsers` | Accept user-facing calls without OIDC or mTLS credentials. Use only for trusted local development or a fully trusted access proxy. |
|
||||
| `server.enableLoopbackServiceHttp` | Enable local plaintext HTTP for loopback sandbox service URLs. Defaults to `true`. |
|
||||
| `pkiInitJob.serverDnsNames` / `certManager.serverDnsNames` | Additional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain. |
|
||||
| `supervisor.sandboxRuntime.networkPolicyEnforced` | Required acknowledgement that the cluster CNI enforces ingress and egress `NetworkPolicy` in sandbox namespaces. |
|
||||
| `supervisor.sandboxRuntime.boundaryPort` | Non-privileged TLS port used between paired supervisor and sandbox Pods. |
|
||||
| `upstreamProxy` | Operator-owned corporate HTTP forward proxy for policy-approved TLS egress. Refer to [Configure a Corporate Upstream Proxy](#configure-a-corporate-upstream-proxy). |
|
||||
|
||||
@@ -254,7 +258,6 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--values my-values.yaml
|
||||
```
|
||||
|
||||
@@ -350,7 +353,6 @@ helm upgrade --install openshell \
|
||||
oci://ghcr.io/nvidia/openshell/helm-chart \
|
||||
--version <version> \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set serviceAccount.create=false \
|
||||
--set serviceAccount.name=my-existing-sa
|
||||
```
|
||||
|
||||
@@ -791,13 +791,15 @@ gateway_namespace = "openshell"
|
||||
gateway_pod_selector = { "app.kubernetes.io/name" = "openshell", "app.kubernetes.io/instance" = "openshell" }
|
||||
|
||||
[openshell.drivers.kubernetes.sandbox_runtime]
|
||||
# Required acknowledgement that the cluster CNI enforces NetworkPolicy and the
|
||||
# sandbox namespaces prevent untrusted policy, pod, label, and Secret changes.
|
||||
network_policy_enforced = true
|
||||
# TLS-protected boundary listener reached only by the paired control pod.
|
||||
boundary_port = 5500
|
||||
```
|
||||
|
||||
The Kubernetes driver always creates the workload `NetworkPolicy` before
|
||||
releasing a sandbox Pod. Your cluster CNI must enforce ingress and egress
|
||||
`NetworkPolicy` in every sandbox namespace; the Kubernetes API does not verify
|
||||
enforcement.
|
||||
|
||||
In managed workspace mode, the Kubernetes driver copies each explicitly named
|
||||
`image_pull_secrets` Secret from `namespace` into the managed workspace
|
||||
namespace on sandbox creation. Shared and operator modes require the Secret to
|
||||
|
||||
@@ -442,6 +442,11 @@ than being staged into the guest. See the [Gateway Configuration File](./gateway
|
||||
Kubernetes-backed sandboxes run as pods in the configured sandbox namespace. Use Kubernetes for shared clusters, remote compute, GPU scheduling, and operator-managed environments.
|
||||
|
||||
<Warning>
|
||||
The cluster CNI MUST enforce ingress and egress `NetworkPolicy` in every sandbox
|
||||
namespace. OpenShell creates the workload policy, but the Kubernetes API cannot
|
||||
confirm enforcement. Without it, workload pods may connect directly and bypass
|
||||
supervisor network policy.
|
||||
|
||||
Kubernetes workspace namespaces are an administrative trust boundary. In
|
||||
shared and managed modes, only the OpenShell gateway and its trusted Agent
|
||||
Sandbox controller may administer Sandbox CRs, sandbox pods, or the configured
|
||||
@@ -471,7 +476,6 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA
|
||||
| `sandbox_runtime_image_pull_policy` | `sandboxRuntime.image.pullPolicy` | Set the Kubernetes image pull policy for the sandbox runtime image. |
|
||||
| `supervisor_image` | `supervisor.image.registry` / `supervisor.image.repository` / `supervisor.image.tag` / `supervisor.image.digest` | Override the image that provides `openshell-supervisor`. Individual image values take precedence over `global.image`; a digest takes precedence over the tag. |
|
||||
| `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the canonical supervisor pull policy: `always`, `if_not_present`, or `never`. `newer` is Podman-only. |
|
||||
| `sandbox_runtime.network_policy_enforced` | `supervisor.sandboxRuntime.networkPolicyEnforced` | Acknowledge that the cluster CNI enforces ingress and egress `NetworkPolicy` in sandbox namespaces. This must be `true`. |
|
||||
| `sandbox_runtime.boundary_port` | `supervisor.sandboxRuntime.boundaryPort` | Set the non-privileged TLS port used between the paired supervisor and sandbox Pods. |
|
||||
| `https_proxy` | `upstreamProxy.url` | Set the operator-owned `http://host:port` or `https://host:port` corporate forward proxy used for policy-approved TLS CONNECT egress. |
|
||||
| `no_proxy` | `upstreamProxy.noProxy` | Set destinations that bypass only the corporate proxy. OpenShell policy evaluation still applies. |
|
||||
|
||||
@@ -24,8 +24,7 @@ ENV OPENSHELL_COMPUTE_DRIVER=kubernetes \
|
||||
OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY=if_not_present \
|
||||
OPENSHELL_SANDBOX_RUNTIME_IMAGE=${SANDBOX_RUNTIME_IMAGE} \
|
||||
OPENSHELL_SANDBOX_RUNTIME_IMAGE_PULL_POLICY=if_not_present \
|
||||
OPENSHELL_SUPERVISOR_SIDELOAD_METHOD=init-container \
|
||||
OPENSHELL_K8S_SANDBOX_RUNTIME_NETWORK_POLICY_ENFORCED=true
|
||||
OPENSHELL_SUPERVISOR_SIDELOAD_METHOD=init-container
|
||||
|
||||
USER 1000:1000
|
||||
EXPOSE 8080
|
||||
|
||||
@@ -6,6 +6,7 @@ Deploy or register an OpenShell gateway, verify it is reachable, and run your fi
|
||||
|
||||
- OpenShell CLI installed (`openshell`)
|
||||
- A reachable gateway endpoint, or access to a Kubernetes cluster where you can install the Helm chart
|
||||
- For Kubernetes installs, a CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces
|
||||
|
||||
## Helm Deployment
|
||||
|
||||
@@ -15,7 +16,6 @@ Install the gateway into a Kubernetes cluster you manage:
|
||||
kubectl create namespace openshell
|
||||
helm upgrade --install openshell deploy/helm/openshell \
|
||||
--namespace openshell \
|
||||
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
|
||||
--set server.disableTls=true \
|
||||
--set service.type=ClusterIP
|
||||
```
|
||||
|
||||
@@ -708,9 +708,8 @@ kubectl -n <sandbox-namespace> get sandbox <sandbox-name> -o jsonpath='{.spec.te
|
||||
```
|
||||
|
||||
The Kubernetes driver creates a sandbox workload Pod and a separate, directly
|
||||
managed supervisor Pod. Helm must render
|
||||
`network_policy_enforced = true`. This is an explicit operator assertion that
|
||||
the cluster CNI enforces Kubernetes NetworkPolicy; the Kubernetes API cannot
|
||||
managed supervisor Pod. The cluster CNI must enforce ingress and egress
|
||||
Kubernetes NetworkPolicy in every sandbox namespace; the Kubernetes API cannot
|
||||
attest enforcement. Run sandboxes only in a trusted namespace
|
||||
where tenants cannot create Pods, copy OpenShell role labels, or read the
|
||||
bootstrap Secret.
|
||||
|
||||
+2
-2
@@ -39,12 +39,12 @@ run = """
|
||||
helm dependency build deploy/helm/openshell
|
||||
echo "--- helm lint: defaults ---"
|
||||
echo "values files: deploy/helm/openshell/values.yaml"
|
||||
helm lint deploy/helm/openshell --set agentSandbox.preflight.enabled=false --set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
helm lint deploy/helm/openshell --set agentSandbox.preflight.enabled=false
|
||||
for f in deploy/helm/openshell/ci/values-*.yaml; do
|
||||
variant=$(basename "$f" .yaml | sed 's/values-//')
|
||||
echo "--- helm lint: $variant ---"
|
||||
echo "values files: deploy/helm/openshell/values.yaml, $f"
|
||||
helm lint deploy/helm/openshell -f "$f" --set agentSandbox.preflight.enabled=false --set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
||||
helm lint deploy/helm/openshell -f "$f" --set agentSandbox.preflight.enabled=false
|
||||
done
|
||||
echo "--- helm lint: workspace defaults ---"
|
||||
helm lint deploy/helm/openshell-workspace
|
||||
|
||||
Reference in New Issue
Block a user