mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-19 05:50:41 +08:00
Drew Newberry
27b0d3c102
Add --gpu flag to 'openshell gateway start' that enables NVIDIA GPU access inside the cluster container. When enabled, the Docker container receives GPU device requests via bollard, k3s auto-detects the NVIDIA runtime, and the NVIDIA k8s-device-plugin is deployed via HelmChart CR so Kubernetes workloads can request nvidia.com/gpu resources. - Add GPU device request to Docker container creation (bollard DeviceRequest) - Set GPU_ENABLED=true env var to gate GPU manifest deployment - Multi-stage Dockerfile build to bundle NVIDIA Container Toolkit binaries - Conditional GPU manifest copy in cluster entrypoint - HelmChart CR for nvidia-device-plugin v0.17.1 with GFD and NFD enabled - Thread gpu bool through DeployOptions, CLI flag, and run dispatch
101 lines
5.3 KiB
Docker
101 lines
5.3 KiB
Docker
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
|
|
# k3s cluster image with OpenShell Helm charts and manifests
|
|
#
|
|
# This image is based on k3s and includes:
|
|
# - Packaged OpenShell Helm chart
|
|
# - HelmChart CR for auto-deploying OpenShell
|
|
# - Custom entrypoint for DNS configuration in Docker environments
|
|
#
|
|
# Component images (openshell/server, openshell/sandbox) are
|
|
# pulled at runtime from the distribution registry rather than bundled as tarballs.
|
|
# Registry credentials are generated by the entrypoint script at container start.
|
|
#
|
|
# The helm charts are built by the docker:build:cluster mise task
|
|
# and placed in deploy/docker/.build/ before this Dockerfile is built.
|
|
|
|
# Tracked upstream vulns in rancher/k3s:v1.35.2-k3s1 (bundled Go dependencies):
|
|
# GHSA-pwhc-rpq9-4c8w containerd v2.1.5-k3s1 (local privesc via CRI dir perms;
|
|
# upstream patched in 2.1.5 -- may be scanner false positive
|
|
# from the -k3s1 suffix)
|
|
# GHSA-p436-gjf2-799p docker/cli v28.3.2 (Windows-only plugin path hijack; N/A)
|
|
# GHSA-9h8m-3fm2-qjrq otel/sdk v1.39.0 (macOS-only PATH hijack; N/A for Linux)
|
|
# CVE-2024-36623 docker/docker v25.0.8 (streamformatter race condition)
|
|
# Bump K3S_VERSION when a release with updated dependencies ships.
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Stage: nvidia-toolkit
|
|
# ---------------------------------------------------------------------------
|
|
# Install the NVIDIA Container Toolkit from the official Ubuntu packages and
|
|
# extract the binaries. These are copied into the final k3s image so that
|
|
# nvidia-container-runtime is on PATH and k3s auto-detects it as an
|
|
# alternative container runtime (creates the "nvidia" RuntimeClass).
|
|
# The binaries add ~15 MB to the image but allow GPU support to be activated
|
|
# at runtime via the --gpu flag without a separate image variant.
|
|
FROM ubuntu:24.04 AS nvidia-toolkit
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
gpg curl ca-certificates && \
|
|
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
|
|
| gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && \
|
|
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
|
|
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
|
|
| tee /etc/apt/sources.list.d/nvidia-container-toolkit.list && \
|
|
apt-get update && \
|
|
apt-get install -y --no-install-recommends nvidia-container-toolkit && \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Stage: final k3s image
|
|
# ---------------------------------------------------------------------------
|
|
ARG K3S_VERSION=v1.35.2-k3s1
|
|
FROM rancher/k3s:${K3S_VERSION}
|
|
|
|
# Copy NVIDIA Container Toolkit binaries from the build stage.
|
|
# k3s auto-detects nvidia-container-runtime on PATH and registers it as a
|
|
# containerd runtime + creates the "nvidia" RuntimeClass automatically.
|
|
COPY --from=nvidia-toolkit /usr/bin/nvidia-container-runtime /usr/bin/
|
|
COPY --from=nvidia-toolkit /usr/bin/nvidia-container-runtime-hook /usr/bin/
|
|
COPY --from=nvidia-toolkit /usr/bin/nvidia-container-cli /usr/bin/
|
|
COPY --from=nvidia-toolkit /usr/bin/nvidia-ctk /usr/bin/
|
|
# libnvidia-container shared libraries required by nvidia-container-cli
|
|
COPY --from=nvidia-toolkit /usr/lib/x86_64-linux-gnu/libnvidia-container*.so* /usr/lib/
|
|
|
|
# Create directories for manifests, charts, and configuration
|
|
RUN mkdir -p /var/lib/rancher/k3s/server/manifests \
|
|
/var/lib/rancher/k3s/server/static/charts \
|
|
/etc/rancher/k3s \
|
|
/opt/openshell/manifests \
|
|
/opt/openshell/charts \
|
|
/opt/openshell/gpu-manifests
|
|
|
|
# Copy entrypoint script that configures DNS for Docker environments
|
|
# This script detects the host gateway IP and configures CoreDNS to use it
|
|
COPY deploy/docker/cluster-entrypoint.sh /usr/local/bin/cluster-entrypoint.sh
|
|
RUN chmod +x /usr/local/bin/cluster-entrypoint.sh
|
|
|
|
# Copy healthcheck script that verifies cluster readiness
|
|
COPY deploy/docker/cluster-healthcheck.sh /usr/local/bin/cluster-healthcheck.sh
|
|
RUN chmod +x /usr/local/bin/cluster-healthcheck.sh
|
|
|
|
# Registry credentials for pulling component images at runtime are generated
|
|
# by the entrypoint script at /etc/rancher/k3s/registries.yaml.
|
|
|
|
# Copy packaged helm charts to a staging directory that won't be
|
|
# overwritten by the /var/lib/rancher/k3s volume mount. The entrypoint
|
|
# script copies them into the k3s static charts directory at container start.
|
|
COPY deploy/docker/.build/charts/*.tgz /opt/openshell/charts/
|
|
|
|
# Copy Kubernetes manifests to a persistent location that won't be overwritten by the volume mount.
|
|
# The bootstrap code will copy these to /var/lib/rancher/k3s/server/manifests/ after cluster start.
|
|
COPY deploy/kube/manifests/*.yaml /opt/openshell/manifests/
|
|
|
|
# Copy GPU-specific manifests (deployed conditionally by entrypoint when GPU_ENABLED=true)
|
|
COPY deploy/kube/gpu-manifests/*.yaml /opt/openshell/gpu-manifests/
|
|
|
|
# Use custom entrypoint that configures DNS before starting k3s
|
|
ENTRYPOINT ["/usr/local/bin/cluster-entrypoint.sh"]
|
|
|
|
HEALTHCHECK --interval=5s --timeout=5s --start-period=20s --retries=60 \
|
|
CMD ["/usr/local/bin/cluster-healthcheck.sh"]
|