* ci(kubernetes): pin mise in e2e workflow
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* ci(kubernetes): mirror postgres image for ha e2e
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* ci(kubernetes): reuse e2e workflow for ha
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
---------
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
release-helm and tag-ghcr-release now depend on the release job.
This is to prevent a GHCR image or helm chart from being published when some
other aspect of the release fails.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(gateway): align package TLS bootstrap path
Closes#1593
Default package-managed gateway services to a stable local TLS directory and use that same value for certificate generation and runtime startup.
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* test(packaging): validate package asset paths exist
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* ci(e2e): pin mise in kubernetes job
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
---------
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Rootless Podman sandbox containers reach the host through pasta's local
connection bypass, which translates L2 frames to L4 host sockets. The
dev gateway script binds to 127.0.0.1 by default, which is not routable
through pasta. Auto-detect rootless mode and bind to 0.0.0.0 so sandbox
containers can connect to the gateway.
- Auto-detect rootless Podman in gateway.sh and export
OPENSHELL_BIND_ADDRESS=0.0.0.0 when not explicitly set
- Add e2e:podman:rootless mise task and CI matrix entry to validate
rootless Podman networking end-to-end
- CI creates a non-root user inside the privileged container to trigger
Podman's rootless code paths (pasta, user namespace isolation)
Signed-off-by: Naveen Malik <nmalik@redhat.com>
* ci: pin azure/setup-helm and helm/kind-action to commit SHAs
* chore(python): add py.typed marker for PEP 561 compliance
* ci: use full semver in pinned action version comments
Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
---------
Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
- Add publish-manifest input to docker-build.yml (default true); single-arch
branch callers set it false so the merge job is skipped and the shared
bare :SHA tag in GHCR is never written by branch workflows
- branch-kubernetes-e2e: retag :SHA-amd64 to :SHA before kind load so
Helm's image.tag matches what is loaded in kind containerd
- branch-e2e: pass image-tag as :SHA-arm64 to e2e-test so the arch-specific
GHCR tag is used directly without depending on the bare tag
- bare :SHA in GHCR is now written only by test-gpu.yml (multi-arch build),
eliminating the last-writer-wins race across concurrent workflows
The VM driver artifacts were already attested. Expand the attest step
to cover CLI, gateway, supervisor, deb, rpm, and wheel artifacts so all
release binaries can be verified with gh attestation verify.
Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
Adds a helm:test mise task that installs the helm-unittest plugin if
not present and runs chart unit tests under deploy/helm/openshell.
Installs the plugin into Dockerfile.ci so CI runs do not need to
download it each time.
Closes#1281
Signed-off-by: Mesut Oezdil <versusfinem@gmail.com>
* wip
* refactor(docker): use native rust builds for split gateway/supervisor images
Drop the in-Docker BUILD_FROM_SOURCE path so both images consume only
prebuilt binaries staged natively via tasks/scripts/stage-prebuilt-binaries.sh.
This mirrors what CI does and reuses the host's cargo target cache and
sccache across rebuilds.
- Dockerfile.gateway: nvcr.io/nvidia/distroless/cc:v4.0.4 base (the 4.0.0
tag does not exist on nvcr.io; the registry uses a v prefix). GNU-linked
binary copied to /usr/local/bin.
- Dockerfile.supervisor: scratch base, static musl binary. Static linkage
lets the image stay scratch while still being executable as a Kubernetes
init container.
- skaffold.yaml: each artifact invokes tasks/scripts/docker-build-image.sh,
which stages the binary natively (cargo / cargo-zigbuild) and then builds
the image. Drops the cross-build.sh dependency from the supervisor build.
- seccomp.rs: add a local SYS_kexec_file_load constant for musl/aarch64.
libc 0.2.185 omits the symbol from its musl/aarch64 bindings, so the
supervisor's seccomp filter previously failed to compile for that target.
- architecture/build.md: describe the native-first pipeline and per-image
runtime choices.
Local validation: gateway image 101MB (was 194MB), supervisor image 21.7MB.
helm:skaffold:run deploys cleanly; the static musl supervisor binary runs
correctly in a non-glibc agent container.
* refactor(docker): tighten binary perms via --chown + 0550
Replace `COPY --chmod=755` with `COPY --chown=<user> --chmod=0550` in
the gateway and supervisor Dockerfiles. The binary is no longer
world-readable or world-executable; ownership is pinned to the runtime
user.
- Gateway uses `--chown=nvs:nvs` + `USER nvs:nvs`, matching the only
non-root user defined in `nvcr.io/nvidia/distroless/cc` (UID 1000) and
the Helm chart's `securityContext.runAsUser: 1000`, which overrides
the Dockerfile USER at runtime.
- Supervisor uses numeric `--chown=65534:65534` because the scratch base
has no `/etc/passwd` for name resolution. The supervisor image is
only consumed by the init-container copy-self path; the destination
pod's runAsUser governs execute access.
Validated by deploying to a local k3d cluster via `helm:skaffold:run`
and confirming the gateway StatefulSet reaches 1/1 Running.
* ci(gpu): repoint GPU probe image lookup at Dockerfile.gateway
The previous awk parsed `FROM <image> AS gateway` from the now-deleted
`Dockerfile.images`. The new `Dockerfile.gateway` uses an ARG with a
default (`ARG GATEWAY_BASE_IMAGE=nvcr.io/nvidia/distroless/cc:v4.0.4`)
and `FROM ${GATEWAY_BASE_IMAGE} AS gateway`, so the old script returns
nothing.
Parse the ARG default value directly so the GPU prerequisites check
keeps using the gateway base image as a `nvidia-smi` probe target.
* ci(gpu): pin GPU probe to nvcr.io/nvidia/base/ubuntu:noble
The previous probe parsed the gateway base image out of the Dockerfile,
relying on the fact that the gateway ran on `nvcr.io/nvidia/base/ubuntu`
and that NVIDIA Container Toolkit CDI injection would populate
`nvidia-smi` and the supporting libs at runtime. The new gateway base
(`nvcr.io/nvidia/distroless/cc`) lacks `ldconfig`, a populated
`/usr/bin`, and the broader filesystem layout CDI injection assumes,
so it cannot serve as a GPU probe.
Pin the probe image explicitly to the NVIDIA-managed Ubuntu base. The
probe is independent of the gateway runtime and survives future base
swaps.
* fix(e2e-gpu): pass GPU probe image via env, drop Dockerfile.images parse
The Rust e2e test `gpu_request_for_each_discovered_device_matches_plain_container`
was still parsing the deleted `Dockerfile.images` to derive its GPU probe
image, panicking with `No such file or directory` after this PR's
Dockerfile split.
Move the probe image to a single source of truth in the workflow
(`OPENSHELL_E2E_GPU_PROBE_IMAGE` env at the job level) and require the
e2e test to read it from there. No silent codebase default — the test
panics with a pointer at the workflow if the env is missing, so it
fails loudly rather than drifting from CI.
The prereq probe step in the workflow now consumes the same env, so
the probe image is declared exactly once.
* fix(docker): keep supervisor binary root-owned for rootless Podman
The Podman driver mounts the supervisor image read-only into the sandbox
container at /opt/openshell/bin and runs that container as UID 0, but
deliberately drops DAC_OVERRIDE for hardening (container.rs:419). With
--chown=65534:65534 --chmod=0550 the binary was r-xr-x--- owned by UID
65534, so the container's UID 0 fell into "other" with no read or exec
access and the supervisor crashed on start (ContainerExited code 1).
Docker and Kubernetes both retain DAC_OVERRIDE, so root could still
exec the file — which is why this regression only surfaced in the
Podman e2e job.
Drop --chown from the supervisor COPY so the binary stays root-owned.
Keep --chmod=0550: the security win was dropping world-execute, not
changing the owner. The chown bought nothing here because the container
is always UID 0 regardless of driver, but it actively broke the only
driver that drops DAC_OVERRIDE.
---------
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
* test(helm): Add kube gateway e2e tests
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* ci(helm): add Branch Helm E2E workflow gated on test:e2e-helm
Adds a label-gated GitHub Actions workflow that exercises the Helm
chart end-to-end against the Rust e2e suite via `mise run e2e:helm`.
Pipeline:
- pr_metadata gates on the `test:e2e-helm` label via the pr-gate action.
- build-gateway / build-supervisor build and push Docker images using
the reusable docker-build.yml workflow.
- helm-e2e (bare runner): apt-installs z3 build deps so cargo can
compile the openshell-policy crate's z3-sys backend, creates a kind
cluster via helm/kind-action, materializes the kind kubeconfig at the
path mise's [env] block expects, side-loads the freshly built
gateway/supervisor images, applies
deploy/kube/manifests/agent-sandbox.yaml so the
sandboxes.agents.x-k8s.io CRD and reconciling StatefulSet are in
place, and finally runs `mise run e2e:helm`.
Also expands the `e2e:helm` task to run the full Rust e2e suite
(matching `e2e:podman`) instead of only the smoke test, with
OPENSHELL_E2E_KUBE_TEST as an opt-in single-test override for local
debugging.
Extends the e2e-label-help workflow so applying `test:e2e-helm` posts
the next-step hint pointing at this workflow.
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* test(helm-e2e): run host-gateway tests on the kube driver
Six tests previously skipped on the kubernetes driver — three in
host_gateway_alias.rs plus the forward_proxy_l7 + graphql_l7 cases —
relied on `host.openshell.internal` reaching either a host process or a
sibling docker container. The Helm chart already supports this via
`server.hostGatewayIP`, but the e2e wrapper never set it.
- with-kube-gateway.sh: auto-detect the host-routable IP (CoreDNS
`host.k3d.internal` first to handle Docker Desktop, docker network
gateway as a fallback for kind on Linux CI) and pass it to helm
install. Also import locally-built images into existing k3d clusters
and wait for namespace deletion to complete before exit.
- e2e/rust/Cargo.toml + e2e-helm.sh: add an `e2e-host-gateway` feature
that gates the three test files; docker and podman runs imply it,
helm runs opt in by default (overridable via
OPENSHELL_E2E_HELM_FEATURES for remote clusters where the test host
is unreachable from pods).
- Drop the `skip_if_kube` helper and its callers — feature gating now
decides whether the tests compile in.
Verified against a local k3d cluster: all six previously-skipped tests
pass, smoke regression intact.
* refactor(e2e): rename e2e:helm to e2e:kubernetes
The runner exercises the gateway on Kubernetes; helm is just the
deployment mechanism. Names now describe the target environment.
- mise task: `e2e:helm` -> `e2e:kubernetes`
- script: e2e/rust/e2e-helm.sh -> e2e/rust/e2e-kubernetes.sh
- env var: OPENSHELL_E2E_HELM_FEATURES -> OPENSHELL_E2E_KUBERNETES_FEATURES
- workflow: branch-helm-e2e.yml -> branch-kubernetes-e2e.yml
(display name "Branch Kubernetes E2E", job "kubernetes-e2e")
- PR gate label: test:e2e-helm -> test:e2e-kubernetes (e2e-label-help
hint workflow updated to match)
PR #1251 needs the new label applied; the old label can be removed.
* fix(e2e): prefer IPv4 host-gateway IP from dual-stack docker networks
kind dual-stacks its network; the wrapper's awk picked the first
non-empty gateway, which was the IPv6 entry (fc00:...). Sandbox pods
can't reach the test host's IPv4 listener through that, so the L7 and
host-gateway-alias tests failed in CI even though they passed locally
against k3d (where host.k3d.internal in CoreDNS short-circuits the
docker-network fallback).
Restrict the awk filter to IPv4 octets.
---------
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* ci(helm): add OCI chart release workflow
Publishes the Helm chart to ghcr.io/nvidia/openshell/helm-chart via helm push on every tag (versioned + :latest) and main push (:0.0.0-dev overwrite + :0.0.0-dev.<sha> per-commit pin). Renames Chart.yaml name to helm-chart to match the target OCI path.
* ci(helm): use full sha in dev pinned chart version
The container images pushed by docker-build.yml are tagged with the
full $GITHUB_SHA, not a 7-char prefix. Match the chart's pinned
version (0.0.0-dev.<full-sha>) so the chart tag and the image tag
the chart resolves to are identical.
* docs(readme): add helm chart install instructions
Document the OCI chart at ghcr.io/nvidia/openshell/helm-chart with
examples for tagged, floating dev, and SHA-pinned dev installs, and
flag the Kubernetes deployment path as experimental.
* docs(helm): split chart details into chart README
Move the dev tag conventions and configuration pointers into a new
README under deploy/helm/openshell/ and shorten the top-level README
section to the install command (no --version, defaults to latest
tagged semver) plus a link to the chart README.
Taylor Mutch