Files
OpenShell/tasks/scripts/docker-build-image.sh
T
Taylor MutchandDrew Newberry ba77967fc4 refactor(docker): split gateway/supervisor Dockerfiles and use native rust builds (#1316)
* wip

* refactor(docker): use native rust builds for split gateway/supervisor images

Drop the in-Docker BUILD_FROM_SOURCE path so both images consume only
prebuilt binaries staged natively via tasks/scripts/stage-prebuilt-binaries.sh.
This mirrors what CI does and reuses the host's cargo target cache and
sccache across rebuilds.

- Dockerfile.gateway: nvcr.io/nvidia/distroless/cc:v4.0.4 base (the 4.0.0
  tag does not exist on nvcr.io; the registry uses a v prefix). GNU-linked
  binary copied to /usr/local/bin.
- Dockerfile.supervisor: scratch base, static musl binary. Static linkage
  lets the image stay scratch while still being executable as a Kubernetes
  init container.
- skaffold.yaml: each artifact invokes tasks/scripts/docker-build-image.sh,
  which stages the binary natively (cargo / cargo-zigbuild) and then builds
  the image. Drops the cross-build.sh dependency from the supervisor build.
- seccomp.rs: add a local SYS_kexec_file_load constant for musl/aarch64.
  libc 0.2.185 omits the symbol from its musl/aarch64 bindings, so the
  supervisor's seccomp filter previously failed to compile for that target.
- architecture/build.md: describe the native-first pipeline and per-image
  runtime choices.

Local validation: gateway image 101MB (was 194MB), supervisor image 21.7MB.
helm:skaffold:run deploys cleanly; the static musl supervisor binary runs
correctly in a non-glibc agent container.

* refactor(docker): tighten binary perms via --chown + 0550

Replace `COPY --chmod=755` with `COPY --chown=<user> --chmod=0550` in
the gateway and supervisor Dockerfiles. The binary is no longer
world-readable or world-executable; ownership is pinned to the runtime
user.

- Gateway uses `--chown=nvs:nvs` + `USER nvs:nvs`, matching the only
  non-root user defined in `nvcr.io/nvidia/distroless/cc` (UID 1000) and
  the Helm chart's `securityContext.runAsUser: 1000`, which overrides
  the Dockerfile USER at runtime.
- Supervisor uses numeric `--chown=65534:65534` because the scratch base
  has no `/etc/passwd` for name resolution. The supervisor image is
  only consumed by the init-container copy-self path; the destination
  pod's runAsUser governs execute access.

Validated by deploying to a local k3d cluster via `helm:skaffold:run`
and confirming the gateway StatefulSet reaches 1/1 Running.

* ci(gpu): repoint GPU probe image lookup at Dockerfile.gateway

The previous awk parsed `FROM <image> AS gateway` from the now-deleted
`Dockerfile.images`. The new `Dockerfile.gateway` uses an ARG with a
default (`ARG GATEWAY_BASE_IMAGE=nvcr.io/nvidia/distroless/cc:v4.0.4`)
and `FROM ${GATEWAY_BASE_IMAGE} AS gateway`, so the old script returns
nothing.

Parse the ARG default value directly so the GPU prerequisites check
keeps using the gateway base image as a `nvidia-smi` probe target.

* ci(gpu): pin GPU probe to nvcr.io/nvidia/base/ubuntu:noble

The previous probe parsed the gateway base image out of the Dockerfile,
relying on the fact that the gateway ran on `nvcr.io/nvidia/base/ubuntu`
and that NVIDIA Container Toolkit CDI injection would populate
`nvidia-smi` and the supporting libs at runtime. The new gateway base
(`nvcr.io/nvidia/distroless/cc`) lacks `ldconfig`, a populated
`/usr/bin`, and the broader filesystem layout CDI injection assumes,
so it cannot serve as a GPU probe.

Pin the probe image explicitly to the NVIDIA-managed Ubuntu base. The
probe is independent of the gateway runtime and survives future base
swaps.

* fix(e2e-gpu): pass GPU probe image via env, drop Dockerfile.images parse

The Rust e2e test `gpu_request_for_each_discovered_device_matches_plain_container`
was still parsing the deleted `Dockerfile.images` to derive its GPU probe
image, panicking with `No such file or directory` after this PR's
Dockerfile split.

Move the probe image to a single source of truth in the workflow
(`OPENSHELL_E2E_GPU_PROBE_IMAGE` env at the job level) and require the
e2e test to read it from there. No silent codebase default — the test
panics with a pointer at the workflow if the env is missing, so it
fails loudly rather than drifting from CI.

The prereq probe step in the workflow now consumes the same env, so
the probe image is declared exactly once.

* fix(docker): keep supervisor binary root-owned for rootless Podman

The Podman driver mounts the supervisor image read-only into the sandbox
container at /opt/openshell/bin and runs that container as UID 0, but
deliberately drops DAC_OVERRIDE for hardening (container.rs:419). With
--chown=65534:65534 --chmod=0550 the binary was r-xr-x--- owned by UID
65534, so the container's UID 0 fell into "other" with no read or exec
access and the supervisor crashed on start (ContainerExited code 1).

Docker and Kubernetes both retain DAC_OVERRIDE, so root could still
exec the file — which is why this regression only surfaced in the
Podman e2e job.

Drop --chown from the supervisor COPY so the binary stays root-owned.
Keep --chmod=0550: the security win was dropping world-execute, not
changing the owner. The chown bought nothing here because the container
is always UID 0 regardless of driver, but it actively broke the only
driver that drops DAC_OVERRIDE.

---------

Co-authored-by: Drew Newberry <anewberry@nvidia.com>
2026-05-12 16:05:49 -07:00

196 lines
4.8 KiB
Bash
Executable File

#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "${SCRIPT_DIR}/container-engine.sh"
normalize_arch() {
case "$1" in
x86_64|amd64) echo "amd64" ;;
aarch64|arm64) echo "arm64" ;;
*) echo "$1" ;;
esac
}
prebuilt_arches() {
if [[ -n "${DOCKER_PLATFORM:-}" ]]; then
local raw_platforms=${DOCKER_PLATFORM//[[:space:]]/}
local platform
IFS=',' read -r -a platforms <<< "${raw_platforms}"
for platform in "${platforms[@]}"; do
case "${platform}" in
linux/amd64) echo "amd64" ;;
linux/arm64) echo "arm64" ;;
*)
echo "Error: unsupported DOCKER_PLATFORM '${platform}'" >&2
echo "Supported platforms: linux/amd64, linux/arm64" >&2
exit 1
;;
esac
done
return
fi
normalize_arch "$(ce_info_arch)"
}
required_prebuilt_binaries() {
case "$1" in
gateway)
echo "openshell-gateway"
;;
supervisor|supervisor-sideload|supervisor-output)
echo "openshell-sandbox"
;;
esac
}
missing_prebuilt_paths() {
local target=$1
local arch
local binary
local path
mapfile -t arches < <(prebuilt_arches)
read -r -a binaries <<< "$(required_prebuilt_binaries "${target}")"
for arch in "${arches[@]}"; do
for binary in "${binaries[@]}"; do
path="deploy/docker/.build/prebuilt-binaries/${arch}/${binary}"
if [[ ! -f "${path}" ]]; then
echo "${path}"
fi
done
done
}
ensure_prebuilt_binaries() {
local target=$1
local missing
local arch
if [[ -z "${CI:-}" && "${PREBUILT_AUTO_STAGE:-1}" != "0" ]]; then
echo "Staging prebuilt Rust binaries for Docker target '${target}'..."
mapfile -t arches < <(prebuilt_arches)
for arch in "${arches[@]}"; do
PREBUILT_ARCH="${arch}" "${SCRIPT_DIR}/stage-prebuilt-binaries.sh" "${target}"
done
fi
missing="$(missing_prebuilt_paths "${target}")"
if [[ -n "${missing}" ]]; then
echo "Error: missing prebuilt Rust binaries required by Docker target '${target}':" >&2
printf ' %s\n' ${missing} >&2
echo "Stage binaries at deploy/docker/.build/prebuilt-binaries/<arch>/ before building." >&2
exit 1
fi
}
TARGET=${1:?"Usage: docker-build-image.sh <gateway|supervisor|supervisor-output> [extra-args...]"}
shift
IS_FINAL_IMAGE=0
IMAGE_NAME=""
DOCKER_TARGET=""
DOCKERFILE=""
case "${TARGET}" in
gateway)
IS_FINAL_IMAGE=1
IMAGE_NAME="openshell/gateway"
DOCKER_TARGET="gateway"
DOCKERFILE="deploy/docker/Dockerfile.gateway"
;;
supervisor)
IS_FINAL_IMAGE=1
IMAGE_NAME="openshell/supervisor"
DOCKER_TARGET="supervisor"
DOCKERFILE="deploy/docker/Dockerfile.supervisor"
;;
supervisor-output)
# Backward-compat alias: same as "supervisor".
IS_FINAL_IMAGE=1
IMAGE_NAME="openshell/supervisor"
DOCKER_TARGET="supervisor"
DOCKERFILE="deploy/docker/Dockerfile.supervisor"
;;
*)
echo "Error: unsupported target '${TARGET}'" >&2
exit 1
;;
esac
if [[ ! -f "${DOCKERFILE}" ]]; then
echo "Error: Dockerfile not found: ${DOCKERFILE}" >&2
exit 1
fi
if [[ -n "${IMAGE_REGISTRY:-}" && "${IS_FINAL_IMAGE}" == "1" ]]; then
IMAGE_NAME="${IMAGE_REGISTRY}/${IMAGE_NAME#openshell/}"
fi
IMAGE_TAG=${IMAGE_TAG:-dev}
DOCKER_BUILD_CACHE_DIR=${DOCKER_BUILD_CACHE_DIR:-.cache/buildkit}
CACHE_PATH="${DOCKER_BUILD_CACHE_DIR}/images"
mkdir -p "${CACHE_PATH}"
BUILDER_ARGS=()
if ce_is_docker; then
if [[ -n "${DOCKER_BUILDER:-}" ]]; then
BUILDER_ARGS=(--builder "${DOCKER_BUILDER}")
elif [[ -z "${DOCKER_PLATFORM:-}" && -z "${CI:-}" ]]; then
_ctx=$(ce_context_name)
BUILDER_ARGS=(--builder "${_ctx}")
fi
fi
CACHE_ARGS=()
if [[ -z "${CI:-}" ]]; then
if ce_is_docker; then
if ce_buildx_inspect ${BUILDER_ARGS[@]+"${BUILDER_ARGS[@]}"} 2>/dev/null | grep -q "Driver: docker-container"; then
CACHE_ARGS=(
--cache-from "type=local,src=${CACHE_PATH}"
--cache-to "type=local,dest=${CACHE_PATH},mode=max"
)
fi
fi
fi
ensure_prebuilt_binaries "${TARGET}"
TAG_ARGS=()
if [[ "${IS_FINAL_IMAGE}" == "1" ]]; then
TAG_ARGS=(-t "${IMAGE_NAME}:${IMAGE_TAG}")
fi
OUTPUT_ARGS=()
if [[ -n "${DOCKER_OUTPUT:-}" ]]; then
OUTPUT_ARGS=(--output "${DOCKER_OUTPUT}")
elif [[ "${IS_FINAL_IMAGE}" == "1" ]]; then
if [[ "${DOCKER_PUSH:-}" == "1" ]]; then
OUTPUT_ARGS=(--push)
elif [[ "${DOCKER_PLATFORM:-}" == *","* ]]; then
OUTPUT_ARGS=(--push)
else
OUTPUT_ARGS=(--load)
fi
else
echo "Error: DOCKER_OUTPUT must be set when building target '${TARGET}'" >&2
exit 1
fi
ce_build \
${BUILDER_ARGS[@]+"${BUILDER_ARGS[@]}"} \
${DOCKER_PLATFORM:+--platform ${DOCKER_PLATFORM}} \
${CACHE_ARGS[@]+"${CACHE_ARGS[@]}"} \
-f "${DOCKERFILE}" \
--target "${DOCKER_TARGET}" \
${TAG_ARGS[@]+"${TAG_ARGS[@]}"} \
--provenance=false \
"$@" \
${OUTPUT_ARGS[@]+"${OUTPUT_ARGS[@]}"} \
.