mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-19 05:50:41 +08:00
* fix(sandbox): fall back to plain seccomp listener when WAIT_KILLABLE_RECV is unavailable SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV was added in Linux 5.19. On older kernels (for example RHEL 9.x / 5.14 nodes such as RHCOS on OpenShift) the flag is rejected with EINVAL, which made the capability-free sandbox fail to start during the notification probe with "notification launcher disappeared". Install the notification listener with WAIT_KILLABLE_RECV when the kernel supports it and fall back to a plain NEW_LISTENER on EINVAL. The fallback listener records wait_killable_recv = false: its notification receive is uninterruptible, but the sandbox is otherwise fully functional. Signed-off-by: Akram <akram.benaissi@gmail.com> * fix(sandbox): address review — legacy read-only listener mode + cancellation invariant Follow-up to the PR review (GATOR-de00bfcc-01 / mrunalp): make the < 5.19 fallback cancellation-safe instead of racing broker writes. - Record an explicit ListenerMode (Killable vs LegacyReadOnly); add writes_disabled()/mode() and emit the selected mode in qualification output (seccomp_listener_mode). - Centralize task-memory output writes behind NotificationListener:: write_task_output; in LegacyReadOnly mode getpeername, accept/accept4 with a non-null address, and sendmmsg length write-backs fail closed with EOPNOTSUPP. accept with a null address, socket/connect/bind/listen/sendto/sendmsg keep working (copied inputs, scalar responses, atomic ADDFD_SEND). - Enforce the launch invariant `cancellation || task_memory_writes_disabled` in SandboxConfirmEvidence::validate() rather than dropping cancellation unconditionally; add task_memory_writes_disabled to SeccompEvidence. - Add per-path fail-closed tests (write_task_output, write_socket_addr, a real plain listener installed on a modern kernel) and confirmation-invariant tests. - Correct the flag-semantics comments and document both modes plus the reduced legacy syscall compatibility in architecture/sandbox.md. Signed-off-by: Akram <akram.benaissi@gmail.com> * docs: document legacy read-only sandbox mode on kernels before 5.19 Kernels without SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV (< 5.19, e.g. RHEL 9.x / RHCOS 5.14) run the sandbox in a legacy read-only cancellation mode where the broker fails closed with EOPNOTSUPP on the mediated operations that write results back into workload memory (getpeername, accept/accept4 with a non-null address, sendmmsg length write-backs). Document this observable behavior and its syscall limitations in the public Fern docs: the support-matrix kernel requirements and the OpenShift runtime guidance. Signed-off-by: Akram <akram.benaissi@gmail.com> * style(sandbox): satisfy rustfmt and clippy doc_markdown Match the pinned rustfmt (Rust 1.95.0) line-wrapping for the write_task_output call, and backtick `legacy_read_only` in the qualification-report doc comment so clippy::doc_markdown (-D warnings) passes. Signed-off-by: Akram <akram.benaissi@gmail.com> * fix(sandbox): migrate task_memory_writes_disabled into backend protocol Add the `task_memory_writes_disabled` field to `SeccompEvidence` in the backend protocol contract and relax the validation from requiring `cancellation` to accepting `cancellation || task_memory_writes_disabled`. This completes the rebase migration missed by the isolation-interface refactor: the sandbox reports this field but the backend struct lacked it, and the validator rejected every pre-5.19 legacy listener. Signed-off-by: Akram <akram.benaissi@gmail.com> --------- Signed-off-by: Akram <akram.benaissi@gmail.com>
244 lines
12 KiB
Plaintext
244 lines
12 KiB
Plaintext
---
|
|
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
title: "OpenShift"
|
|
sidebar-title: "OpenShift"
|
|
description: "Install the OpenShell Helm chart on OpenShift with capability-free sandbox workloads."
|
|
keywords: "Generative AI, Cybersecurity, Kubernetes, OpenShift, SCC, Security Context Constraints, Helm, Gateway, Installation"
|
|
position: 6
|
|
---
|
|
|
|
The Kubernetes driver resolves the UID range assigned to each OpenShift
|
|
namespace and renders the sandbox and supervisor with a numeric non-root
|
|
identity from that range. OpenShell does not require the `privileged` SCC or any
|
|
added Linux capability.
|
|
|
|
<Warning>
|
|
Verify that the selected OpenShift runtime profile permits an unprivileged
|
|
process to install a nested seccomp user-notification filter and use Landlock.
|
|
OpenShell fails sandbox startup when either capability-free runtime probe fails.
|
|
</Warning>
|
|
|
|
## Node kernel and legacy read-only mode
|
|
|
|
OpenShift nodes run RHCOS, which currently ships a RHEL 9.x kernel (5.14). That
|
|
kernel predates `SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` (Linux 5.19), so the
|
|
sandbox starts in a reduced **legacy read-only** cancellation mode. Isolation is
|
|
unchanged, but the broker fails closed with `EOPNOTSUPP` on the mediated
|
|
operations that write results back into workload memory — `getpeername`,
|
|
`accept`/`accept4` with a non-null peer-address argument, and `sendmmsg`
|
|
per-message length write-backs. Outbound-oriented workloads run unchanged;
|
|
server workloads that read the peer address on accept need a node kernel with
|
|
`WAIT_KILLABLE_RECV` (Linux 5.19+, or a distribution backport). See the
|
|
[support matrix](/reference/support-matrix#legacy-read-only-mode-kernels-before-linux-519)
|
|
for the full behavior; the selected mode is reported as `seccomp_listener_mode`
|
|
in the sandbox qualification output.
|
|
|
|
## Prerequisites
|
|
|
|
- OpenShift 4.x cluster with `oc` configured.
|
|
- Helm 3.x.
|
|
- [Agent Sandbox](/kubernetes/setup#install-agent-sandbox) controller and CRDs.
|
|
- A CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces.
|
|
|
|
## Install OpenShell
|
|
|
|
Pre-create the namespace, then install the chart. Keep the default restricted
|
|
security posture and acknowledge NetworkPolicy only after validating the CNI.
|
|
|
|
```shell
|
|
oc create ns openshell
|
|
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
|
|
--version <version> \
|
|
--namespace openshell \
|
|
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
|
|
```
|
|
|
|
The driver reads the namespace's `openshift.io/sa.scc.uid-range` annotation and
|
|
uses the resulting UID/GID for the sandbox, agent, trusted init containers, and
|
|
supervisor. Each container sets `allowPrivilegeEscalation: false`, drops all
|
|
Linux capabilities, and uses `RuntimeDefault` seccomp.
|
|
|
|
Wait for the gateway:
|
|
|
|
```shell
|
|
oc -n openshell rollout status statefulset/openshell
|
|
```
|
|
|
|
If you set `workload.kind=deployment`, wait for `deployment/openshell` instead.
|
|
|
|
## Connect to the Gateway
|
|
|
|
Forward the gateway port for local evaluation:
|
|
|
|
```shell
|
|
oc -n openshell port-forward svc/openshell 8080:8080
|
|
openshell gateway add https://127.0.0.1:8080 --local --name openshift
|
|
openshell status
|
|
```
|
|
|
|
## Options for end-to-end TLS
|
|
|
|
The steps above run the gateway over plaintext HTTP for quick evaluation. For production deployments, choose one of the approaches below based on your OpenShift version and preferences.
|
|
|
|
### End-to-end TLS using Gateway API and BackendTLSPolicy (OpenShift 4.22+)
|
|
|
|
OpenShift 4.22 and later support `BackendTLSPolicy` in the Gateway API, enabling end-to-end TLS between the OpenShift router and the OpenShell gateway pod. The traffic flow is:
|
|
|
|
```text
|
|
client → HTTPS → OpenShift Gateway (terminate TLS) → TLS (re-encrypt) → openshell gateway pod
|
|
```
|
|
|
|
This removes the requirement to run the gateway with `server.disableTls=true`. The OpenShift router terminates client-facing TLS at the listener and re-encrypts when connecting to the backend service, validating the backend's certificate against a CA you provide.
|
|
|
|
#### Prerequisites
|
|
|
|
- OpenShift 4.22+ cluster with the Gateway API enabled
|
|
- cert-manager installed (recommended) or the built-in pkiInitJob for server certificates
|
|
- A `GatewayClass` registered for the OpenShift gateway controller
|
|
|
|
#### Create the GatewayClass
|
|
|
|
If your cluster does not already have an OpenShift GatewayClass, create one:
|
|
|
|
```shell
|
|
oc apply -f - <<'EOF'
|
|
apiVersion: gateway.networking.k8s.io/v1
|
|
kind: GatewayClass
|
|
metadata:
|
|
name: openshift-default
|
|
spec:
|
|
controllerName: openshift.io/gateway-controller/v1
|
|
EOF
|
|
```
|
|
|
|
#### Create the Gateway
|
|
|
|
Create a Gateway resource in the `openshift-ingress` namespace. Replace `<external-hostname>` with your cluster's route hostname (typically a wildcard like `*.openshell-ingress-gw.example.com`):
|
|
|
|
```shell
|
|
oc apply -f - <<'EOF'
|
|
apiVersion: gateway.networking.k8s.io/v1
|
|
kind: Gateway
|
|
metadata:
|
|
name: openshell-gateway
|
|
namespace: openshift-ingress
|
|
spec:
|
|
gatewayClassName: openshift-default
|
|
listeners:
|
|
- name: grpc
|
|
hostname: "<external-hostname>"
|
|
port: 443
|
|
protocol: HTTPS
|
|
tls:
|
|
mode: Terminate
|
|
certificateRefs:
|
|
- name: <listener-tls-secret>
|
|
kind: Secret
|
|
allowedRoutes:
|
|
namespaces:
|
|
from: Selector
|
|
selector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: openshell
|
|
EOF
|
|
```
|
|
|
|
The listener TLS Secret should contain the certificate for the external hostname.
|
|
|
|
#### Install with e2e TLS
|
|
|
|
Install the chart with the GRPCRoute and BackendTLSPolicy enabled. The certgen hook automatically creates the backend CA ConfigMap from the generated PKI bundle:
|
|
|
|
```shell
|
|
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
|
|
--version <version> \
|
|
--namespace openshell \
|
|
--set podSecurityContext.fsGroup=null \
|
|
--set securityContext.runAsUser=null \
|
|
--set server.tls.enableMtls=false \
|
|
--set grpcRoute.enabled=true \
|
|
--set grpcRoute.gateway.name=openshell-gateway \
|
|
--set grpcRoute.gateway.namespace=openshift-ingress \
|
|
--set 'grpcRoute.hostnames[0]=gateway.example.com' \
|
|
--set grpcRoute.backendTLSPolicy.enabled=true \
|
|
--set server.oidc.issuer=https://keycloak.example.com/realms/openshell \
|
|
--set server.oidc.audience=openshell-cli
|
|
```
|
|
|
|
| Override | Reason |
|
|
|---|---|
|
|
| `podSecurityContext.fsGroup=null` / `securityContext.runAsUser=null` | Let OpenShift's SCC admission assign UIDs. |
|
|
| `server.tls.enableMtls=false` | Disable mTLS client certificate authentication. BackendTLSPolicy only validates the server certificate; the ingress proxy cannot present a client certificate to the backend. Use OIDC for authentication instead. |
|
|
| `grpcRoute.enabled=true` | Create a GRPCRoute pointing at the external Gateway. |
|
|
| `grpcRoute.gateway.name` / `namespace` | Reference the Gateway created above in `openshift-ingress`. |
|
|
| `grpcRoute.backendTLSPolicy.enabled=true` | Create a BackendTLSPolicy for TLS re-encryption to the gateway pod. The certgen hook auto-creates the backend CA ConfigMap. The Gateway proxy validates the backend certificate against the service FQDN, which is already in the default server certificate SANs. |
|
|
| `grpcRoute.hostnames` | External hostname for the GRPCRoute. This goes on the Gateway listener certificate, not the backend certificate. |
|
|
|
|
Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)).
|
|
|
|
**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are regular release objects, and a separate post-install/post-upgrade Job (`<release>-certgen-backend-ca`) polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases.
|
|
|
|
If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=<seconds>`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional.
|
|
|
|
#### Register over HTTPS
|
|
|
|
```shell
|
|
openshell gateway add https://gateway.example.com \
|
|
--name openshift \
|
|
--oidc-issuer https://keycloak.example.com/realms/openshell \
|
|
--oidc-client-id openshell-cli
|
|
openshell status
|
|
```
|
|
|
|
### End-to-end TLS using pass-through Route (all OpenShift versions)
|
|
|
|
For OpenShift versions prior to 4.22, or when you prefer Route-based ingress, cert-manager can issue the gateway's server certificate from a real Issuer or ClusterIssuer (for example, a LetsEncrypt/ACME issuer), and an OpenShift Route with TLS passthrough exposes it externally while the gateway keeps terminating its own TLS and mTLS.
|
|
|
|
Install cert-manager and configure a working `ClusterIssuer` first — see
|
|
[Managing Certificates](/kubernetes/managing-certificates) for the
|
|
`certManager.serverIssuerRef` details. Configure an OIDC provider as described
|
|
in [Access Control](/kubernetes/access-control) — remote gateways authenticate
|
|
CLI users via OIDC, not mTLS, so the gateway must know the OIDC issuer URL.
|
|
Install the chart with:
|
|
|
|
```shell
|
|
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
|
|
--version <version> \
|
|
--namespace openshell \
|
|
--set podSecurityContext.fsGroup=null \
|
|
--set securityContext.runAsUser=null \
|
|
--set server.disableTls=false \
|
|
--set certManager.enabled=true \
|
|
--set certManager.serverIssuerRef.name=letsencrypt-prod \
|
|
--set certManager.serverIssuerRef.kind=ClusterIssuer \
|
|
--set certManager.serverDnsNames[0]=gateway.example.com \
|
|
--set openshiftRoute.enabled=true \
|
|
--set openshiftRoute.host=gateway.example.com \
|
|
--set server.oidc.issuer=https://keycloak.example.com/realms/openshell \
|
|
--set server.oidc.audience=openshell-cli
|
|
```
|
|
|
|
| Override | Reason |
|
|
|---|---|
|
|
| `certManager.serverIssuerRef` | Creates a second server certificate from your Issuer or ClusterIssuer for external clients. The gateway uses SNI to present this cert for the external hostname while continuing to present the internal (chart CA) cert to supervisors. The internal certificate's `ca.crt` is the chart CA that also signed the client cert, so the default `clientCaFromServerTlsSecret=true` is correct. |
|
|
| `openshiftRoute.enabled` / `openshiftRoute.host` | Creates an OpenShift Route with TLS passthrough — the router forwards the encrypted connection by SNI without decrypting, so the gateway uses the SNI hostname to select the external certificate. |
|
|
| `server.oidc.issuer` / `server.oidc.audience` | Configures server-side OIDC validation. Without these, the gateway expects mTLS client certificates and rejects OIDC-only CLI connections. See [Access Control](/kubernetes/access-control). |
|
|
|
|
Register the gateway with the CLI over OIDC. Remote gateways authenticate CLI
|
|
users via OIDC, not mTLS — see [Access Control](/kubernetes/access-control):
|
|
|
|
```shell
|
|
openshell gateway add https://gateway.example.com \
|
|
--name openshift \
|
|
--oidc-issuer https://keycloak.example.com/realms/openshell \
|
|
--oidc-client-id openshell-cli
|
|
openshell gateway login openshift
|
|
```
|
|
|
|
## Next Steps
|
|
|
|
- For more on certificate provisioning modes, refer to [Managing Certificates](/kubernetes/managing-certificates).
|
|
- To expose the gateway externally through the Kubernetes Gateway API instead of a Route, refer to [Ingress](/kubernetes/ingress).
|
|
- To configure OIDC authentication, refer to [Access Control](/kubernetes/access-control).
|