Install kubectl and helm into the training image (#2191)

docs/developer/ci/02-docker-build.md

@@ -27,12 +27,17 @@ The Dockerfile is the build recipe: it provides the cu13 defaults and emits one
 | `WHEELS_REPO`                                                                                          | prebuilt-wheels GitHub repo (`yueming-yuan/miles-wheels`)                                                                                                                                                                                                                                                                                           |
 | `WHEELS_TAG_X86` / `WHEELS_TAG_ARM64`                                                                  | the two **complete** wheels release tags selected by `TARGETARCH` and installed **verbatim**. cu13 uses the rolling `cu130-torch213-x86_64` / `cu130-torch213-aarch64` releases; cu12-x86 overrides `WHEELS_TAG_X86` with the rolling `cu129-x86_64` release                                                                                                                                                                                              |
 | `SGLANG_BRANCH` / `SGLANG_COMMIT`, `MEGATRON_REPO` / `MEGATRON_BRANCH` / `MEGATRON_COMMIT`, `MILES_COMMIT`, `SGL_ROUTER_*` | source pins for the layered repos; empty commit args follow their branch, while release builds provide exact SHAs |
+| `TARGETARCH`                                                                                           | set by buildx; also the one argument `docker/install-kube-tools.sh` takes                                                                                                                                                                                                                                                                           |
 
 
-**Output** — one `radixark/miles` image for the platform buildx targets: the SGLang base, then the Python dependencies declared in `requirements.txt`, Megatron-LM at its branch default or caller-pinned commit, Miles, and the prebuilt wheels (`sgl-router` among them). A multi-arch build is one `buildx` run executed once per platform — `TARGETARCH` differs each time, so each arch installs its own wheels — and buildx pushes the two as a single manifest.
+**Output** — one `radixark/miles` image for the platform buildx targets: the SGLang base, then the Python dependencies declared in `requirements.txt`, Megatron-LM at its branch default or caller-pinned commit, Miles, and the prebuilt wheels (`sgl-router` among them). It also carries `kubectl`, `helm` and `tmux`, which the k8s-native launch path drives the cluster with. A multi-arch build is one `buildx` run executed once per platform — `TARGETARCH` differs each time, so each arch installs its own wheels — and buildx pushes the two as a single manifest.
 
 `docker/Dockerfile.rocm` is the ROCm counterpart (build-args `GPU_ARCH`, a ROCm `SGLANG_IMAGE_TAG`, and a `WHEELS_TAG_ROCM` release from `XinyuJiangCMU/miles-wheels-rocm`). `rocm724-mi35x` uses the dated Python 3.12 base and ROCm 7.2.4 wheels from the `rocm724-gfx950-v0.5.20` release. It sets `APPLY_ROCR_VMMFIX=1` to install the point-release-matched ROCr VMM-pause fix; ROCm 10 has the fix upstream.
 
+### `docker/install-kube-tools.sh`
+
+`kubectl` and `helm` are not optional extras of a training image: the workbench installs releases with them and the launcher drives the run with them, so **every** image `docker/build.py` can build must carry them. The pinned versions and their checksum verification therefore live in one script that each training Dockerfile copies and runs, rather than in a `RUN` block per Dockerfile that only one of them ever got. Bumping either client is an edit to `docker/install-kube-tools.sh` alone.
+
 ## Build script
 
 `docker/build.py` builds and pushes the images. Select a build with `--variant` and a tag mode with `--image-tag {dev,latest,custom}`. The `VARIANTS` table is the source of truth for each variant's image, target platforms, Dockerfile, and default build-args. Repeatable `--build-arg KEY=VALUE` options are appended after those defaults, so an explicit caller override wins.

tests/fast/test_training_images.py

@@ -0,0 +1,58 @@
+import importlib.util
+import re
+import sys
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+BUILD_SCRIPT = REPO_ROOT / "docker" / "build.py"
+INSTALL_SCRIPT = REPO_ROOT / "docker" / "install-kube-tools.sh"
+DEFAULT_DOCKERFILE = "docker/Dockerfile"
+INSTALL_STEP = "install-kube-tools.sh"
+
+
+def build_module():
+    spec = importlib.util.spec_from_file_location("miles_docker_build", BUILD_SCRIPT)
+    module = importlib.util.module_from_spec(spec)
+
+    script_dir = str(BUILD_SCRIPT.parent)
+    sys.path.insert(0, script_dir)
+    try:
+        spec.loader.exec_module(module)
+    finally:
+        sys.path.remove(script_dir)
+
+    return module
+
+
+def training_dockerfiles() -> list[Path]:
+    paths = {config.get("dockerfile", DEFAULT_DOCKERFILE) for config in build_module().VARIANTS.values()}
+    return sorted(REPO_ROOT / path for path in paths)
+
+
+class TestClusterToolsInEveryTrainingImage:
+    @pytest.mark.parametrize("dockerfile", training_dockerfiles(), ids=lambda path: path.name)
+    def test_the_image_installs_the_kubernetes_clients(self, dockerfile):
+        """The workbench installs releases with them and the launcher drives the run with them."""
+        assert INSTALL_STEP in dockerfile.read_text(), f"{dockerfile.name} builds an image without kubectl and helm"
+
+    @pytest.mark.parametrize("dockerfile", training_dockerfiles(), ids=lambda path: path.name)
+    def test_the_image_takes_the_shared_step_rather_than_its_own_copy(self, dockerfile):
+        """Two hand-copied install blocks drift, and the one that drifts is the one nobody runs locally."""
+        text = dockerfile.read_text()
+
+        assert "dl.k8s.io" not in text
+        assert "get.helm.sh" not in text
+
+    def test_the_shared_step_pins_both_clients_and_checks_what_it_downloaded(self):
+        """An unpinned or unverified client is a silent cluster-wide behaviour change on the next rebuild."""
+        script = INSTALL_SCRIPT.read_text()
+
+        assert re.search(r'^KUBECTL_VERSION="v\d+\.\d+\.\d+"$', script, re.MULTILINE)
+        assert re.search(r'^HELM_VERSION="v\d+\.\d+\.\d+"$', script, re.MULTILINE)
+        assert script.count("sha256sum -c -") == 2
+
+    def test_every_variant_of_the_build_script_is_covered(self):
+        """A new variant pointing at a third Dockerfile must not quietly opt out of this check."""
+        assert len(training_dockerfiles()) == 2