Files
OpenShell/docs/kubernetes/openshift.mdx
Akram Ben Aissi 293fab75d4 fix(sandbox): support kernels < 5.19 via seccomp WAIT_KILLABLE_RECV fallback (#3420)
* fix(sandbox): fall back to plain seccomp listener when WAIT_KILLABLE_RECV is unavailable

SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV was added in Linux 5.19. On older
kernels (for example RHEL 9.x / 5.14 nodes such as RHCOS on OpenShift) the
flag is rejected with EINVAL, which made the capability-free sandbox fail to
start during the notification probe with "notification launcher disappeared".

Install the notification listener with WAIT_KILLABLE_RECV when the kernel
supports it and fall back to a plain NEW_LISTENER on EINVAL. The fallback
listener records wait_killable_recv = false: its notification receive is
uninterruptible, but the sandbox is otherwise fully functional.

Signed-off-by: Akram <akram.benaissi@gmail.com>

* fix(sandbox): address review — legacy read-only listener mode + cancellation invariant

Follow-up to the PR review (GATOR-de00bfcc-01 / mrunalp): make the < 5.19
fallback cancellation-safe instead of racing broker writes.

- Record an explicit ListenerMode (Killable vs LegacyReadOnly); add
  writes_disabled()/mode() and emit the selected mode in qualification output
  (seccomp_listener_mode).
- Centralize task-memory output writes behind NotificationListener::
  write_task_output; in LegacyReadOnly mode getpeername, accept/accept4 with a
  non-null address, and sendmmsg length write-backs fail closed with EOPNOTSUPP.
  accept with a null address, socket/connect/bind/listen/sendto/sendmsg keep
  working (copied inputs, scalar responses, atomic ADDFD_SEND).
- Enforce the launch invariant `cancellation || task_memory_writes_disabled` in
  SandboxConfirmEvidence::validate() rather than dropping cancellation
  unconditionally; add task_memory_writes_disabled to SeccompEvidence.
- Add per-path fail-closed tests (write_task_output, write_socket_addr, a real
  plain listener installed on a modern kernel) and confirmation-invariant tests.
- Correct the flag-semantics comments and document both modes plus the reduced
  legacy syscall compatibility in architecture/sandbox.md.

Signed-off-by: Akram <akram.benaissi@gmail.com>

* docs: document legacy read-only sandbox mode on kernels before 5.19

Kernels without SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV (< 5.19, e.g. RHEL 9.x /
RHCOS 5.14) run the sandbox in a legacy read-only cancellation mode where the
broker fails closed with EOPNOTSUPP on the mediated operations that write results
back into workload memory (getpeername, accept/accept4 with a non-null address,
sendmmsg length write-backs). Document this observable behavior and its syscall
limitations in the public Fern docs: the support-matrix kernel requirements and
the OpenShift runtime guidance.

Signed-off-by: Akram <akram.benaissi@gmail.com>

* style(sandbox): satisfy rustfmt and clippy doc_markdown

Match the pinned rustfmt (Rust 1.95.0) line-wrapping for the write_task_output
call, and backtick `legacy_read_only` in the qualification-report doc comment
so clippy::doc_markdown (-D warnings) passes.

Signed-off-by: Akram <akram.benaissi@gmail.com>

* fix(sandbox): migrate task_memory_writes_disabled into backend protocol

Add the `task_memory_writes_disabled` field to `SeccompEvidence` in the
backend protocol contract and relax the validation from requiring
`cancellation` to accepting `cancellation || task_memory_writes_disabled`.

This completes the rebase migration missed by the isolation-interface
refactor: the sandbox reports this field but the backend struct lacked
it, and the validator rejected every pre-5.19 legacy listener.

Signed-off-by: Akram <akram.benaissi@gmail.com>

---------

Signed-off-by: Akram <akram.benaissi@gmail.com>
2026-09-22 20:37:19 +00:00

244 lines
12 KiB
Plaintext

---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: "OpenShift"
sidebar-title: "OpenShift"
description: "Install the OpenShell Helm chart on OpenShift with capability-free sandbox workloads."
keywords: "Generative AI, Cybersecurity, Kubernetes, OpenShift, SCC, Security Context Constraints, Helm, Gateway, Installation"
position: 6
---
The Kubernetes driver resolves the UID range assigned to each OpenShift
namespace and renders the sandbox and supervisor with a numeric non-root
identity from that range. OpenShell does not require the `privileged` SCC or any
added Linux capability.
<Warning>
Verify that the selected OpenShift runtime profile permits an unprivileged
process to install a nested seccomp user-notification filter and use Landlock.
OpenShell fails sandbox startup when either capability-free runtime probe fails.
</Warning>
## Node kernel and legacy read-only mode
OpenShift nodes run RHCOS, which currently ships a RHEL 9.x kernel (5.14). That
kernel predates `SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV` (Linux 5.19), so the
sandbox starts in a reduced **legacy read-only** cancellation mode. Isolation is
unchanged, but the broker fails closed with `EOPNOTSUPP` on the mediated
operations that write results back into workload memory — `getpeername`,
`accept`/`accept4` with a non-null peer-address argument, and `sendmmsg`
per-message length write-backs. Outbound-oriented workloads run unchanged;
server workloads that read the peer address on accept need a node kernel with
`WAIT_KILLABLE_RECV` (Linux 5.19+, or a distribution backport). See the
[support matrix](/reference/support-matrix#legacy-read-only-mode-kernels-before-linux-519)
for the full behavior; the selected mode is reported as `seccomp_listener_mode`
in the sandbox qualification output.
## Prerequisites
- OpenShift 4.x cluster with `oc` configured.
- Helm 3.x.
- [Agent Sandbox](/kubernetes/setup#install-agent-sandbox) controller and CRDs.
- A CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces.
## Install OpenShell
Pre-create the namespace, then install the chart. Keep the default restricted
security posture and acknowledge NetworkPolicy only after validating the CNI.
```shell
oc create ns openshell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
```
The driver reads the namespace's `openshift.io/sa.scc.uid-range` annotation and
uses the resulting UID/GID for the sandbox, agent, trusted init containers, and
supervisor. Each container sets `allowPrivilegeEscalation: false`, drops all
Linux capabilities, and uses `RuntimeDefault` seccomp.
Wait for the gateway:
```shell
oc -n openshell rollout status statefulset/openshell
```
If you set `workload.kind=deployment`, wait for `deployment/openshell` instead.
## Connect to the Gateway
Forward the gateway port for local evaluation:
```shell
oc -n openshell port-forward svc/openshell 8080:8080
openshell gateway add https://127.0.0.1:8080 --local --name openshift
openshell status
```
## Options for end-to-end TLS
The steps above run the gateway over plaintext HTTP for quick evaluation. For production deployments, choose one of the approaches below based on your OpenShift version and preferences.
### End-to-end TLS using Gateway API and BackendTLSPolicy (OpenShift 4.22+)
OpenShift 4.22 and later support `BackendTLSPolicy` in the Gateway API, enabling end-to-end TLS between the OpenShift router and the OpenShell gateway pod. The traffic flow is:
```text
client → HTTPS → OpenShift Gateway (terminate TLS) → TLS (re-encrypt) → openshell gateway pod
```
This removes the requirement to run the gateway with `server.disableTls=true`. The OpenShift router terminates client-facing TLS at the listener and re-encrypts when connecting to the backend service, validating the backend's certificate against a CA you provide.
#### Prerequisites
- OpenShift 4.22+ cluster with the Gateway API enabled
- cert-manager installed (recommended) or the built-in pkiInitJob for server certificates
- A `GatewayClass` registered for the OpenShift gateway controller
#### Create the GatewayClass
If your cluster does not already have an OpenShift GatewayClass, create one:
```shell
oc apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: openshift-default
spec:
controllerName: openshift.io/gateway-controller/v1
EOF
```
#### Create the Gateway
Create a Gateway resource in the `openshift-ingress` namespace. Replace `<external-hostname>` with your cluster's route hostname (typically a wildcard like `*.openshell-ingress-gw.example.com`):
```shell
oc apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: openshell-gateway
namespace: openshift-ingress
spec:
gatewayClassName: openshift-default
listeners:
- name: grpc
hostname: "<external-hostname>"
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: <listener-tls-secret>
kind: Secret
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
kubernetes.io/metadata.name: openshell
EOF
```
The listener TLS Secret should contain the certificate for the external hostname.
#### Install with e2e TLS
Install the chart with the GRPCRoute and BackendTLSPolicy enabled. The certgen hook automatically creates the backend CA ConfigMap from the generated PKI bundle:
```shell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null \
--set server.tls.enableMtls=false \
--set grpcRoute.enabled=true \
--set grpcRoute.gateway.name=openshell-gateway \
--set grpcRoute.gateway.namespace=openshift-ingress \
--set 'grpcRoute.hostnames[0]=gateway.example.com' \
--set grpcRoute.backendTLSPolicy.enabled=true \
--set server.oidc.issuer=https://keycloak.example.com/realms/openshell \
--set server.oidc.audience=openshell-cli
```
| Override | Reason |
|---|---|
| `podSecurityContext.fsGroup=null` / `securityContext.runAsUser=null` | Let OpenShift's SCC admission assign UIDs. |
| `server.tls.enableMtls=false` | Disable mTLS client certificate authentication. BackendTLSPolicy only validates the server certificate; the ingress proxy cannot present a client certificate to the backend. Use OIDC for authentication instead. |
| `grpcRoute.enabled=true` | Create a GRPCRoute pointing at the external Gateway. |
| `grpcRoute.gateway.name` / `namespace` | Reference the Gateway created above in `openshift-ingress`. |
| `grpcRoute.backendTLSPolicy.enabled=true` | Create a BackendTLSPolicy for TLS re-encryption to the gateway pod. The certgen hook auto-creates the backend CA ConfigMap. The Gateway proxy validates the backend certificate against the service FQDN, which is already in the default server certificate SANs. |
| `grpcRoute.hostnames` | External hostname for the GRPCRoute. This goes on the Gateway listener certificate, not the backend certificate. |
Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)).
**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are regular release objects, and a separate post-install/post-upgrade Job (`<release>-certgen-backend-ca`) polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases.
If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=<seconds>`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional.
#### Register over HTTPS
```shell
openshell gateway add https://gateway.example.com \
--name openshift \
--oidc-issuer https://keycloak.example.com/realms/openshell \
--oidc-client-id openshell-cli
openshell status
```
### End-to-end TLS using pass-through Route (all OpenShift versions)
For OpenShift versions prior to 4.22, or when you prefer Route-based ingress, cert-manager can issue the gateway's server certificate from a real Issuer or ClusterIssuer (for example, a LetsEncrypt/ACME issuer), and an OpenShift Route with TLS passthrough exposes it externally while the gateway keeps terminating its own TLS and mTLS.
Install cert-manager and configure a working `ClusterIssuer` first — see
[Managing Certificates](/kubernetes/managing-certificates) for the
`certManager.serverIssuerRef` details. Configure an OIDC provider as described
in [Access Control](/kubernetes/access-control) — remote gateways authenticate
CLI users via OIDC, not mTLS, so the gateway must know the OIDC issuer URL.
Install the chart with:
```shell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null \
--set server.disableTls=false \
--set certManager.enabled=true \
--set certManager.serverIssuerRef.name=letsencrypt-prod \
--set certManager.serverIssuerRef.kind=ClusterIssuer \
--set certManager.serverDnsNames[0]=gateway.example.com \
--set openshiftRoute.enabled=true \
--set openshiftRoute.host=gateway.example.com \
--set server.oidc.issuer=https://keycloak.example.com/realms/openshell \
--set server.oidc.audience=openshell-cli
```
| Override | Reason |
|---|---|
| `certManager.serverIssuerRef` | Creates a second server certificate from your Issuer or ClusterIssuer for external clients. The gateway uses SNI to present this cert for the external hostname while continuing to present the internal (chart CA) cert to supervisors. The internal certificate's `ca.crt` is the chart CA that also signed the client cert, so the default `clientCaFromServerTlsSecret=true` is correct. |
| `openshiftRoute.enabled` / `openshiftRoute.host` | Creates an OpenShift Route with TLS passthrough — the router forwards the encrypted connection by SNI without decrypting, so the gateway uses the SNI hostname to select the external certificate. |
| `server.oidc.issuer` / `server.oidc.audience` | Configures server-side OIDC validation. Without these, the gateway expects mTLS client certificates and rejects OIDC-only CLI connections. See [Access Control](/kubernetes/access-control). |
Register the gateway with the CLI over OIDC. Remote gateways authenticate CLI
users via OIDC, not mTLS — see [Access Control](/kubernetes/access-control):
```shell
openshell gateway add https://gateway.example.com \
--name openshift \
--oidc-issuer https://keycloak.example.com/realms/openshell \
--oidc-client-id openshell-cli
openshell gateway login openshift
```
## Next Steps
- For more on certificate provisioning modes, refer to [Managing Certificates](/kubernetes/managing-certificates).
- To expose the gateway externally through the Kubernetes Gateway API instead of a Route, refer to [Ingress](/kubernetes/ingress).
- To configure OIDC authentication, refer to [Access Control](/kubernetes/access-control).