trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-31 07:44:57 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
dd0dcffe7f65de8b81241e214effdf588ecf0b6a
OpenShell/deploy/docker/Dockerfile.images
T
Taylor Mutch b1cba79620 fix(sandbox): add copy-self subcommand for scratch-image init container (#1208) 
* fix(sandbox): add copy-self subcommand for scratch-image init container

The supervisor image is built FROM scratch and contains only the
openshell-sandbox binary. The Kubernetes side-load init container
previously invoked `sh -c "cp ..."`, which failed with
`executable file 'sh' not found in $PATH` because the scratch image
has no shell.

Add a `copy-self <DEST>` subcommand to the supervisor binary that
resolves its own executable, copies it to the destination path, and
sets mode 0755. Update the Kubernetes driver init container to invoke
this subcommand directly. Mirrors argoexec's emissary self-copy pattern.

* fix(docker): use ubuntu base for supervisor image

`FROM scratch` cannot exec the dynamically linked supervisor binary
(no `/lib/ld-linux-*.so.*`), so the Kubernetes side-load init container
hit `exec /openshell-sandbox: no such file or directory` even after
the `copy-self` subcommand replaced the broken `sh -c` command.

Switch the supervisor stage to `nvcr.io/nvidia/base/ubuntu:noble-20251013`
— the same base the gateway image already uses — so glibc and the
dynamic loader are present. Docker/Podman drivers continue to work
unchanged (they extract or volume-mount the binary into agent containers
that already have glibc); the Kubernetes init container now succeeds
end-to-end.

Verified on a k3d cluster: init container `openshell-supervisor-install`
exits 0 on first try, binary lands at `/opt/openshell/bin/openshell-sandbox`
(mode 0755), and the agent container reaches Ready.

* docs(docker): refresh supervisor target description

The file header still described the supervisor image as
`FROM scratch, binary only`. Update it to match the current
Ubuntu-base supervisor stage.
2026-05-06 13:02:26 -07:00

232 lines
9.8 KiB
Docker
Raw Blame History

# syntax=docker/dockerfile:1.4
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# Shared OpenShell image build graph.
#
# Targets:
# gateway Final gateway image
# supervisor Final supervisor image (Ubuntu base, supervisor binary)
# cluster Final cluster image
#
# Rust binaries are built natively before the image build and staged at:
# deploy/docker/.build/prebuilt-binaries/<arch>/openshell-{gateway,sandbox}
#
# For local dev (Skaffold), pass --build-arg BUILD_FROM_SOURCE=1 to compile
# binaries inside Docker instead. BuildKit only executes the selected binary
# staging stage, so missing prebuilt files do not cause a build failure.
# Pin by tag AND manifest-list digest to prevent silent upstream republishes
# from breaking the build. Update both when bumping k3s versions.
# To refresh: docker buildx imagetools inspect rancher/k3s:<tag> | head -3
ARG K3S_VERSION=v1.35.3-k3s1
ARG K3S_DIGEST=sha256:4607083d3cac07e1ccde7317297271d13ed5f60f35a78f33fcef84858a9f1d69
ARG K9S_VERSION=v0.50.18
ARG HELM_VERSION=v3.17.3
ARG NVIDIA_CONTAINER_TOOLKIT_VERSION=1.18.2-1
# Controls binary source: 0 = prebuilt (release), 1 = compile in Docker (local dev).
# Must be declared here (global scope) so it can be used in FROM instructions below.
ARG BUILD_FROM_SOURCE=0
# ---------------------------------------------------------------------------
# Optional in-Docker Rust build (BUILD_FROM_SOURCE=1, local dev only)
# ---------------------------------------------------------------------------
FROM rust:1.95.0-slim-bookworm AS rust-builder
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
cmake \
pkg-config \
libssl-dev \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
COPY Cargo.toml Cargo.lock ./
COPY crates/ crates/
COPY proto/ proto/
COPY providers/ providers/
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/build/target \
cargo build --release \
--features "openshell-core/dev-settings" \
--bin openshell-gateway \
--bin openshell-sandbox \
&& mkdir -p /build/out \
&& install -m 0755 target/release/openshell-gateway /build/out/openshell-gateway \
&& install -m 0755 target/release/openshell-sandbox /build/out/openshell-sandbox
# ---------------------------------------------------------------------------
# Per-arch binary stages
# ---------------------------------------------------------------------------
# Prebuilt path (release default, BUILD_FROM_SOURCE=0)
FROM scratch AS gateway-binary-0
ARG TARGETARCH
# --chmod=755 preserves the executable bit through actions/upload-artifact +
# download-artifact, which strip exec perms during the roundtrip.
COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-gateway /build/out/openshell-gateway
# Source-built path (local dev, BUILD_FROM_SOURCE=1)
FROM rust-builder AS gateway-binary-1
FROM gateway-binary-${BUILD_FROM_SOURCE} AS gateway-binary
# Prebuilt path (release default, BUILD_FROM_SOURCE=0)
FROM scratch AS supervisor-binary-0
ARG TARGETARCH
# --chmod=755 preserves the executable bit through actions/upload-artifact +
# download-artifact, which strip exec perms during the roundtrip.
COPY --chmod=755 deploy/docker/.build/prebuilt-binaries/${TARGETARCH}/openshell-sandbox /build/out/openshell-sandbox
# Source-built path (local dev, BUILD_FROM_SOURCE=1)
FROM rust-builder AS supervisor-binary-1
FROM supervisor-binary-${BUILD_FROM_SOURCE} AS supervisor-binary
# ---------------------------------------------------------------------------
# Final gateway image
# ---------------------------------------------------------------------------
FROM nvcr.io/nvidia/base/ubuntu:noble-20251013 AS gateway
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates && \
apt-get install -y --only-upgrade gpgv && \
rm -rf /var/lib/apt/lists/*
RUN useradd --create-home --user-group openshell
WORKDIR /app
COPY --from=gateway-binary /build/out/openshell-gateway /usr/local/bin/
RUN mkdir -p /build/crates/openshell-server
COPY --chmod=755 crates/openshell-server/migrations /build/crates/openshell-server/migrations
USER openshell
EXPOSE 8080
ENTRYPOINT ["openshell-gateway"]
CMD ["--bind-address", "0.0.0.0", "--port", "8080"]
# ---------------------------------------------------------------------------
# Final supervisor image
# ---------------------------------------------------------------------------
# Supervisor image based on the same NVIDIA Ubuntu base used by the gateway.
#
# Used by:
# - Docker driver: binary is extracted from the image and run inside the
# agent container.
# - Podman driver: image is mounted as an OCI volume at /opt/openshell/bin.
# - Kubernetes driver: image runs as an init container that invokes the
# binary's `copy-self` subcommand to seed an emptyDir volume.
#
# An Ubuntu base provides glibc and the dynamic loader needed to exec the
# dynamically linked binary. `FROM scratch` would be smaller but cannot run
# the binary, breaking the Kubernetes init-container path.
FROM nvcr.io/nvidia/base/ubuntu:noble-20251013 AS supervisor
COPY --from=supervisor-binary /build/out/openshell-sandbox /openshell-sandbox
# ---------------------------------------------------------------------------
# Cluster asset stages
# ---------------------------------------------------------------------------
FROM rancher/k3s:${K3S_VERSION}@${K3S_DIGEST} AS k3s
FROM ubuntu:24.04 AS k9s
ARG K9S_VERSION
ARG TARGETARCH
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates && \
curl -fsSL "https://github.com/derailed/k9s/releases/download/${K9S_VERSION}/k9s_Linux_${TARGETARCH}.tar.gz" \
| tar xz -C /tmp k9s && \
chmod +x /tmp/k9s && \
rm -rf /var/lib/apt/lists/*
FROM ubuntu:24.04 AS helm
ARG HELM_VERSION
ARG TARGETARCH
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates && \
curl -fsSL "https://get.helm.sh/helm-${HELM_VERSION}-linux-${TARGETARCH}.tar.gz" \
| tar xz --strip-components=1 -C /tmp "linux-${TARGETARCH}/helm" && \
chmod +x /tmp/helm && \
rm -rf /var/lib/apt/lists/*
FROM ubuntu:24.04 AS nvidia-container-toolkit
ARG NVIDIA_CONTAINER_TOOLKIT_VERSION
RUN apt-get update && apt-get install -y --no-install-recommends \
gpg curl ca-certificates && \
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && \
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| tee /etc/apt/sources.list.d/nvidia-container-toolkit.list && \
apt-get update && \
apt-get install -y --no-install-recommends \
"nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION}" && \
rm -rf /var/lib/apt/lists/*
# ---------------------------------------------------------------------------
# Final cluster image
# ---------------------------------------------------------------------------
FROM nvcr.io/nvidia/base/ubuntu:noble-20251013 AS cluster
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
iptables \
mount \
dnsutils \
&& apt-get install -y --only-upgrade gpgv \
&& rm -rf /var/lib/apt/lists/*
COPY --from=k3s /bin/ /bin/
COPY --from=k9s /tmp/k9s /usr/local/bin/k9s
COPY --from=helm /tmp/helm /usr/local/bin/helm
COPY --from=k3s /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/k3s-ca-certificates.crt
COPY --from=k3s /usr/share/zoneinfo/ /usr/share/zoneinfo/
ENV PATH="/var/lib/rancher/k3s/data/cni:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux" \
CRI_CONFIG_FILE="/var/lib/rancher/k3s/agent/etc/crictl.yaml"
COPY --from=nvidia-container-toolkit /usr/bin/nvidia-cdi-hook /usr/bin/
COPY --from=nvidia-container-toolkit /usr/bin/nvidia-container-runtime /usr/bin/
COPY --from=nvidia-container-toolkit /usr/bin/nvidia-ctk /usr/bin/
COPY --from=nvidia-container-toolkit /etc/nvidia-container-runtime /etc/nvidia-container-runtime
COPY --from=supervisor-binary /build/out/openshell-sandbox /opt/openshell/bin/openshell-sandbox
RUN mkdir -p /var/lib/rancher/k3s/server/manifests \
/var/lib/rancher/k3s/server/static/charts \
/etc/rancher/k3s \
/opt/openshell/manifests \
/opt/openshell/charts \
/opt/openshell/gpu-manifests \
/run/flannel
COPY deploy/docker/cluster-entrypoint.sh /usr/local/bin/cluster-entrypoint.sh
RUN chmod +x /usr/local/bin/cluster-entrypoint.sh
COPY deploy/docker/cluster-healthcheck.sh /usr/local/bin/cluster-healthcheck.sh
RUN chmod +x /usr/local/bin/cluster-healthcheck.sh
COPY deploy/docker/.build/charts/*.tgz /opt/openshell/charts/
# Only the core k3s auto-deploy manifests belong in the cluster image.
# Gateway API routing is optional and requires Envoy Gateway CRDs, so
# deploy/kube/manifests/envoy-gateway-openshell.yaml stays repo-local and is
# applied manually by `mise run helm:gateway:apply` when grpcRoute is enabled.
COPY deploy/kube/manifests/openshell-helmchart.yaml \
deploy/kube/manifests/agent-sandbox.yaml \
/opt/openshell/manifests/
COPY deploy/kube/gpu-manifests/*.yaml /opt/openshell/gpu-manifests/
ENTRYPOINT ["/usr/local/bin/cluster-entrypoint.sh"]
# Default to "server" so bare `docker run <image>` works without requiring
# the caller to pass a subcommand. The openshell CLI already passes
# ["server", "--disable=traefik", ...] as CMD; this default only affects
# manual `docker run` invocations that omit a command.
CMD ["server"]
HEALTHCHECK --interval=5s --timeout=5s --start-period=20s --retries=60 \
CMD ["/usr/local/bin/cluster-healthcheck.sh"]
Reference in New Issue View Git Blame Copy Permalink