mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-14 04:59:28 +08:00
Jorge
519e5eb35f
* feat(e2e): make e2e:kubernetes work transparently on OpenShift
Running `mise run e2e:kubernetes` on OpenShift required manual namespace
creation, SCC grants, Helm value overrides, and cleanup. A separate
`e2e:openshift` task existed but only checked pod readiness without
running the Rust e2e test suite, and even with the suite wired up the
SSH-relay `sandbox connect` path stalled to the ready timeout because
`kubectl port-forward` cannot carry round-trip-heavy SSH over the
internet.
The harness now auto-detects OpenShift via the `route.openshift.io` API
group and, on OpenShift, both configures the cluster and switches the
gateway transport automatically:
- Drives the gateway through a passthrough OpenShift Route secured with
mandatory mTLS instead of port-forward, so the connect suites
(live_policy_update, port_forward, sync, connect-based
sandbox_lifecycle, settings_management) actually pass. Computes the
Route host from the cluster ingress domain, extracts client mTLS
material from the openshell-client-tls secret, waits for the Route to
serve mTLS, asserts a certless caller is rejected at the TLS
handshake, and registers an mTLS CLI gateway pointing at the Route.
- Applies an SCC-compatible Helm values overlay that removes hardcoded
runAsUser/fsGroup, letting OpenShift assign UIDs from the namespace
range.
- Grants the privileged SCC to openshell-sandbox before Helm install
and removes it during cleanup.
- Grants the anyuid SCC to the PostgreSQL fixture service account in
DB scenarios and removes it during cleanup.
- All oc commands use --context to target the correct cluster.
The OpenShift e2e overlay (ci/values-openshift-e2e.yaml) turns TLS back
on, enables the Route, promotes the cert-verified caller to a dev
principal, and forces `image.pullPolicy`/`supervisor.image.pullPolicy`
to Always so runs against the `latest` upstream image use it instead of
a stale copy cached on the cluster nodes. Every OpenShift branch is
gated on OPENSHIFT_DETECTED, so the vanilla-Kubernetes port-forward path
is unchanged.
The Helm template for podSecurityContext is wrapped with {{- with }} so
null values omit the block instead of rendering invalid YAML.
The separate e2e:openshift task and e2e-openshift.sh script are removed
since e2e:kubernetes now covers OpenShift.
TESTING.md is updated with Kubernetes e2e documentation including
OpenShift auto-detection, dropping the e2e-host-gateway feature on
remote clusters, pinning IMAGE_TAG when the CLI and image versions
differ, task variants, and environment variables.
The debug-openshell-cluster skill gains an OpenShift platform row and
two SCC failure patterns (gateway rejected over hardcoded runAsUser,
sandbox missing the privileged SCC) covering the SCC handling and
podSecurityContext behavior this change introduces.
Signed-off-by: Jorge Garcia Oncins <jgarciao@redhat.com>
* fix(e2e): harden OpenShift SCC cleanup, mTLS gate, and Route timeout
Track the anyuid SCC grant for the PostgreSQL fixture with a dedicated
OPENSHIFT_POSTGRES_SCC_GRANTED flag set before the fixture apply, so a
failed apply no longer leaks the binding; cleanup now revokes it whenever
the grant succeeded, independent of deploy state.
Validate the Route server cert in the certless security gate (curl
--cacert instead of -k) and classify curl's exit code so only a TLS
client-auth rejection (35/56) counts as the expected certless rejection;
an unrelated DNS/timeout/TLS failure now fails loudly instead of masking
a potential mTLS hole.
Raise the OpenShift Route timeout in the e2e overlay. The default HAProxy
Route timeout is 30s, which severed long-lived transfers (large sandbox
upload/download, SSH-relay `sandbox connect`) mid-stream and failed the
sync e2e tests. Set both haproxy.router.openshift.io/timeout and
timeout-tunnel to 300s: a passthrough Route proxies in TCP mode, so
timeout-tunnel governs the established tunnel while timeout covers the
pre-tunnel phase.
Document the OpenShift transport exception, oc prerequisites and SCC
grants, and make the skopeo tag-check example copy-safe in TESTING.md.
Signed-off-by: Jorge Garcia Oncins <jgarciao@redhat.com>
---------
Signed-off-by: Jorge Garcia Oncins <jgarciao@redhat.com>
54 lines
2.5 KiB
YAML
54 lines
2.5 KiB
YAML
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
|
|
# OpenShift overlay for the Kubernetes e2e harness.
|
|
#
|
|
# Bundles the OpenShift-specific settings the harness needs: the Route/mTLS
|
|
# transport, and an image pull policy that avoids stale cached images.
|
|
#
|
|
# Route/mTLS transport: on OpenShift, `kubectl port-forward` stalls the SSH-relay
|
|
# `sandbox connect` path (round-trip-heavy SSH over SPDY), so the harness drives
|
|
# the gateway through a passthrough OpenShift Route with mTLS instead. This
|
|
# overlay turns TLS back on (values-skaffold.yaml disables it), enables the Route,
|
|
# and promotes the cert-verified caller to a dev principal.
|
|
#
|
|
# Image pull policy: force `Always` so runs against the `latest` upstream image
|
|
# actually use it, instead of a stale copy cached on the cluster nodes.
|
|
#
|
|
# Layered by e2e/with-kube-gateway.sh AFTER ci/values-skaffold.yaml and
|
|
# ci/values-openshift-scc.yaml when an OpenShift cluster is detected. The harness
|
|
# supplies `openshiftRoute.host` and `pkiInitJob.serverDnsNames[0]` via --set at
|
|
# install time (both are the cluster-derived Route hostname).
|
|
#
|
|
# Security: this is NOT an open gateway. `server.tls.clientCaSecretName` defaults
|
|
# to `openshell-server-client-ca` and there is no OIDC, so `require_client_auth`
|
|
# is true and mTLS is MANDATORY at the TLS handshake — a caller with only the
|
|
# Route URL and no client certificate is rejected before any RPC. The passthrough
|
|
# Route terminates TLS at the gateway pod, so this holds end-to-end.
|
|
# `allowUnauthenticatedUsers` only promotes the already cert-verified caller to a
|
|
# dev principal at the app layer (mtls_auth is unsupported with the Kubernetes
|
|
# driver). Both are required together; the client certificate is the access gate.
|
|
image:
|
|
pullPolicy: Always
|
|
|
|
supervisor:
|
|
image:
|
|
pullPolicy: Always
|
|
|
|
server:
|
|
disableTls: false
|
|
auth:
|
|
allowUnauthenticatedUsers: true
|
|
|
|
openshiftRoute:
|
|
enabled: true
|
|
# host is supplied via --set at install time (cluster-derived Route hostname).
|
|
# The default HAProxy Route timeout is 30s, which severs long-lived transfers
|
|
# (large sandbox upload/download, SSH-relay `sandbox connect`) mid-stream. Raise
|
|
# both the connection timeout and the passthrough tunnel timeout so these paths
|
|
# survive. Passthrough Routes proxy in TCP mode, so timeout-tunnel governs the
|
|
# established tunnel while timeout covers the pre-tunnel phase.
|
|
annotations:
|
|
haproxy.router.openshift.io/timeout: 300s
|
|
haproxy.router.openshift.io/timeout-tunnel: 300s
|