diff --git a/architecture/custom-vm-runtime.md b/architecture/custom-vm-runtime.md index 0dbdaf010..7d0be284e 100644 --- a/architecture/custom-vm-runtime.md +++ b/architecture/custom-vm-runtime.md @@ -7,8 +7,7 @@ lightweight microVM with Apple Hypervisor.framework (macOS) or KVM (Linux). The is embedded inside `libkrunfw`, a companion library that packages a pre-built Linux kernel. The stock `libkrunfw` from Homebrew ships a minimal kernel without bridge, netfilter, or -conntrack support. This limits Kubernetes networking to a point-to-point (ptp) CNI -configuration without kube-proxy or service VIPs. +conntrack support. This is insufficient for Kubernetes pod networking. The custom libkrunfw runtime adds bridge CNI, iptables/nftables, and conntrack support to the VM kernel, enabling standard Kubernetes networking. @@ -31,33 +30,25 @@ Host (macOS/Linux) Guest VM ├── gateway-init.sh (PID 1) -│ ├── Detects kernel capabilities -│ ├── Selects network profile: bridge | legacy-vm-net -│ ├── Configures CNI +│ ├── Validates kernel capabilities (fail-fast) +│ ├── Configures bridge CNI │ └── Execs k3s server └── check-vm-capabilities.sh (diagnostics) ``` -## Network Profiles +## Network Profile -The VM init script auto-detects kernel capabilities and selects the appropriate -networking profile. This can be overridden via `OPENSHELL_VM_NET_PROFILE`. +The VM uses the bridge CNI profile, which requires a custom libkrunfw with bridge and +netfilter kernel support. The init script validates these capabilities at boot and fails +fast with an actionable error if they are missing. -### Bridge Profile (custom runtime) +### Bridge Profile - CNI: bridge plugin with `cni0` interface -- IP masquerade: enabled (iptables) -- kube-proxy: enabled +- IP masquerade: enabled (iptables-legacy via CNI bridge plugin) +- kube-proxy: enabled (nftables mode) - Service VIPs: functional (ClusterIP, NodePort) -- hostNetwork workarounds: removed - -### Legacy VM Net Profile (stock runtime) - -- CNI: ptp plugin (point-to-point) -- IP masquerade: disabled (no iptables) -- kube-proxy: disabled -- Service VIPs: not functional (direct IP routing only) -- hostNetwork workarounds: required for some pods +- hostNetwork workarounds: not required ## Runtime Provenance @@ -98,10 +89,12 @@ libkrunfw kernel: | Netfilter | `CONFIG_NETFILTER` | iptables/nftables framework | | Connection tracking | `CONFIG_NF_CONNTRACK` | NAT state tracking | | NAT | `CONFIG_NF_NAT` | Service VIP DNAT/SNAT | -| iptables | `CONFIG_IP_NF_IPTABLES` | kube-proxy iptables mode | -| nftables | `CONFIG_NF_TABLES` | kube-proxy nft mode (future) | +| iptables | `CONFIG_IP_NF_IPTABLES` | CNI bridge masquerade | +| nftables | `CONFIG_NF_TABLES` | kube-proxy nftables mode (primary) | | veth | `CONFIG_VETH` | Pod network namespace pairs | | IPVS | `CONFIG_IP_VS` | kube-proxy IPVS mode (optional) | +| Landlock | `CONFIG_SECURITY_LANDLOCK` | Filesystem sandboxing support | +| Seccomp filter | `CONFIG_SECCOMP_FILTER` | Syscall filtering support | ## Verification @@ -116,10 +109,9 @@ Two verification tools are provided: ## Rollout Strategy 1. Custom runtime support is opt-in via `OPENSHELL_VM_RUNTIME_SOURCE_DIR`. -2. Auto-detection selects the correct network profile at boot. -3. The stock runtime path (`legacy-vm-net`) remains the default until the custom - runtime is proven stable. -4. Rollback: unset the env var and re-bundle with stock libraries. +2. The init script validates kernel capabilities at boot and fails fast if missing. +3. Rollback: unset the env var and re-bundle with stock libraries (note: stock + libraries lack bridge/netfilter and pod networking will not work). ## Related Files diff --git a/crates/openshell-vm/runtime/README.md b/crates/openshell-vm/runtime/README.md index 891d1a235..e8289685e 100644 --- a/crates/openshell-vm/runtime/README.md +++ b/crates/openshell-vm/runtime/README.md @@ -9,11 +9,11 @@ The stock `libkrunfw` (from Homebrew) ships a kernel without bridge, netfilter, or conntrack support. This means the VM cannot: - Create `cni0` bridge interfaces (required by the bridge CNI plugin) -- Run kube-proxy (requires iptables/nftables) +- Run kube-proxy (requires nftables) - Route service VIP traffic (requires NAT/conntrack) The custom runtime builds libkrunfw with an additional kernel config fragment -that enables these features. +that enables these networking and sandboxing features. ## Directory Structure @@ -21,7 +21,7 @@ that enables these features. runtime/ build-custom-libkrunfw.sh # Build script for custom libkrunfw kernel/ - bridge-cni.config # Kernel config fragment (bridge + netfilter) + bridge-cni.config # Kernel config fragment (networking + sandboxing) ``` ## Building @@ -66,23 +66,16 @@ mise run vm:bundle-runtime mise run vm ``` -## Network Profiles +## Networking -The VM init script (`gateway-init.sh`) auto-detects the kernel capabilities -and selects the appropriate networking profile: +The VM uses bridge CNI for pod networking with nftables-mode kube-proxy for +service VIP / ClusterIP support. The kernel config fragment enables both +iptables (for CNI bridge masquerade) and nftables (for kube-proxy). -| Profile | Kernel | CNI | kube-proxy | Service VIPs | -|---------|--------|-----|------------|--------------| -| `bridge` | Custom (bridge+netfilter) | bridge CNI (`cni0`) | Enabled | Yes | -| `legacy-vm-net` | Stock (no netfilter) | ptp CNI | Disabled | No (direct IP) | - -To force a specific profile: - -```bash -# Inside the VM (set in gateway-init.sh env): -export OPENSHELL_VM_NET_PROFILE=bridge # Force bridge CNI -export OPENSHELL_VM_NET_PROFILE=legacy-vm-net # Force legacy ptp CNI -``` +k3s is started with `--kube-proxy-arg=proxy-mode=nftables` because the +bundled iptables binaries in k3s have revision-negotiation issues with the +libkrun kernel's xt_MARK module. nftables mode uses the kernel's nf_tables +subsystem directly and avoids this entirely. ## Runtime Provenance @@ -158,14 +151,17 @@ If this fails, you are running the stock runtime. Build and use the custom one. ### kube-proxy CrashLoopBackOff -The kernel does not have netfilter support. Verify: +kube-proxy runs in nftables mode. If it crashes, verify nftables support: ```bash # Inside VM: -iptables -L -n +nft list ruleset ``` -If this fails with "iptables not found" or "modprobe: can't change directory", -the kernel lacks CONFIG_NETFILTER. Use the custom runtime. +If this fails, the kernel may lack `CONFIG_NF_TABLES`. Use the custom runtime. + +Common errors: +- `unknown option "--xor-mark"`: kube-proxy is running in iptables mode instead + of nftables. Verify `--kube-proxy-arg=proxy-mode=nftables` is in the k3s args. ### Runtime mismatch after upgrade diff --git a/crates/openshell-vm/runtime/build-custom-libkrunfw.sh b/crates/openshell-vm/runtime/build-custom-libkrunfw.sh index ab2aa1bf2..a69fc0c13 100755 --- a/crates/openshell-vm/runtime/build-custom-libkrunfw.sh +++ b/crates/openshell-vm/runtime/build-custom-libkrunfw.sh @@ -152,6 +152,13 @@ REQUIRED=( CONFIG_IP_NF_IPTABLES CONFIG_IP_NF_FILTER CONFIG_IP_NF_NAT + CONFIG_NF_TABLES + CONFIG_NFT_NUMGEN + CONFIG_NFT_FIB_IPV4 + CONFIG_NFT_FIB_IPV6 + CONFIG_NFT_CT + CONFIG_NFT_NAT + CONFIG_NFT_MASQ CONFIG_VETH CONFIG_NET_NS ) @@ -195,103 +202,75 @@ fi echo " Kernel source dir: ${KERNEL_DIR_NAME}" if [ "$(uname -s)" = "Darwin" ]; then - # On macOS, the entire kernel build (extract, patch, config, compile) - # must happen inside a Linux container since the kernel build system - # requires a Linux toolchain. We mount the checkout, inject the - # config fragment, and produce kernel.c inside the container. + # On macOS, use krunvm to build the kernel inside a lightweight Linux VM. + # This matches the upstream libkrunfw build approach and avoids all the + # issues with Docker emulation and APFS filesystem limitations. + # + # Prerequisites: brew tap slp/krun && brew install krunvm - if ! command -v docker &>/dev/null; then - echo "ERROR: docker is required to build the kernel on macOS" >&2 + if ! command -v krunvm &>/dev/null; then + echo "ERROR: krunvm is required to build the kernel on macOS" >&2 + echo " Install with: brew tap slp/krun && brew install krunvm" >&2 exit 1 fi - echo "==> Building kernel inside Docker (macOS detected)..." + echo "==> Building kernel inside krunvm (macOS detected)..." - # Pre-build a Docker image with all build dependencies so the - # (potentially emulated) apt-get/pip work is cached across runs. - BUILDER_IMAGE="libkrunfw-builder:bookworm" - if ! docker image inspect "${BUILDER_IMAGE}" >/dev/null 2>&1; then - echo " Building Docker toolchain image (first time only)..." - docker build --platform linux/arm64 -t "${BUILDER_IMAGE}" - <<'DOCKERFILE' -FROM debian:bookworm-slim -RUN apt-get update -qq && \ - apt-get install -y -qq make gcc bc flex bison libelf-dev python3 \ - coreutils curl xz-utils patch libssl-dev cpio >/dev/null 2>&1 && \ - rm -rf /var/lib/apt/lists/* -DOCKERFILE + VM_NAME="libkrunfw-openshell" + + # Clean up any leftover VM from a previous failed run + krunvm delete "${VM_NAME}" 2>/dev/null || true + + # Copy the config fragment into the libkrunfw tree so the VM can see it. + # The merge hook (MERGE_HOOK) is already written there by the cat above. + cp -f "${KERNEL_CONFIG_FRAGMENT}" "${LIBKRUNFW_DIR}/openshell-bridge-cni.config" + + echo " Creating VM..." + # krunvm may print "The volume has been configured" on first use of a + # volume path and exit non-zero. Retry once if that happens. + if ! krunvm create fedora \ + --name "${VM_NAME}" \ + --cpus 4 \ + --mem 4096 \ + -v "${LIBKRUNFW_DIR}:/work" \ + -w /work; then + echo " Retrying VM creation..." + krunvm create fedora \ + --name "${VM_NAME}" \ + --cpus 4 \ + --mem 4096 \ + -v "${LIBKRUNFW_DIR}:/work" \ + -w /work fi - # Stage only the files the build needs into a temp dir so we - # avoid copying the enormous kernel source tree over virtiofs. - DOCKER_STAGE=$(mktemp -d) - trap 'rm -rf "${DOCKER_STAGE}"' EXIT + echo " Installing build dependencies..." + krunvm start "${VM_NAME}" /usr/bin/dnf -- install -y \ + 'dnf-command(builddep)' python3-pyelftools - echo " Staging build inputs..." - cp "${LIBKRUNFW_DIR}/Makefile" "${DOCKER_STAGE}/" - cp "${LIBKRUNFW_DIR}/bin2cbundle.py" "${DOCKER_STAGE}/" - # Copy base kernel config(s) and patches - for f in "${LIBKRUNFW_DIR}"/config-libkrunfw*; do - [ -f "$f" ] && cp "$f" "${DOCKER_STAGE}/" - done - if [ -d "${LIBKRUNFW_DIR}/patches" ]; then - cp -r "${LIBKRUNFW_DIR}/patches" "${DOCKER_STAGE}/patches" - fi - if [ -d "${LIBKRUNFW_DIR}/patches-tee" ]; then - cp -r "${LIBKRUNFW_DIR}/patches-tee" "${DOCKER_STAGE}/patches-tee" - fi - # Copy the cached tarball if present so we don't re-download - if [ -d "${LIBKRUNFW_DIR}/tarballs" ]; then - cp -r "${LIBKRUNFW_DIR}/tarballs" "${DOCKER_STAGE}/tarballs" - fi + krunvm start "${VM_NAME}" /usr/bin/dnf -- builddep -y kernel - docker run --rm \ - -v "${DOCKER_STAGE}:/work" \ - -v "${KERNEL_CONFIG_FRAGMENT}:/fragment/bridge-cni.config:ro" \ - -v "${MERGE_HOOK}:/fragment/merge-hook.sh:ro" \ - -w /build \ - --platform linux/arm64 \ - "${BUILDER_IMAGE}" \ - bash -c ' - set -euo pipefail + # Step 1: prepare kernel sources (download, extract, patch, base config) + echo " Preparing kernel sources..." + krunvm start "${VM_NAME}" /usr/bin/make -- "${KERNEL_DIR_NAME}" - KDIR="'"${KERNEL_DIR_NAME}"'" + # Step 2: merge the OpenShell config fragment + echo " Merging OpenShell kernel config fragment..." + krunvm start "${VM_NAME}" /usr/bin/bash -- \ + /work/openshell-kconfig-hook.sh "/work/${KERNEL_DIR_NAME}" /work/openshell-bridge-cni.config - # Copy staged inputs to container-local filesystem so tar - # extraction does not hit macOS APFS bind-mount limitations. - echo " Copying build inputs to container filesystem..." - cp -r /work/* /build/ + # Step 3: build the kernel and generate the C bundle + echo " Building kernel (this is the slow part)..." + krunvm start "${VM_NAME}" /usr/bin/make -- -j4 - # Patch bin2cbundle.py to make the elftools import lazy. - # We only use -t Image (raw path) which does not need elftools, - # but the top-level import would fail without pyelftools installed. - sed -i "s/^from elftools/# lazy: from elftools/" bin2cbundle.py + echo " Cleaning up VM..." + krunvm delete "${VM_NAME}" - # Step 1: prepare kernel sources (download, extract, patch, base config, olddefconfig) - echo " Preparing kernel sources..." - make "$KDIR" + # Clean up temp files from the libkrunfw tree + rm -f "${LIBKRUNFW_DIR}/openshell-bridge-cni.config" - # Step 2: merge the OpenShell config fragment - echo " Merging OpenShell kernel config fragment..." - bash /fragment/merge-hook.sh "/build/$KDIR" /fragment/bridge-cni.config - - # Step 3: build the kernel and generate the C bundle - echo " Building kernel (this is the slow part)..." - make -j"$(nproc)" "$KDIR"/arch/arm64/boot/Image - - echo " Generating kernel.c bundle..." - python3 bin2cbundle.py -t Image "$KDIR"/arch/arm64/boot/Image kernel.c - - # Copy artifacts back to the bind-mounted staging dir - echo " Copying artifacts back to host..." - cp /build/kernel.c /work/kernel.c - cp /build/"$KDIR"/.config /work/kernel.config - ' - - # Move artifacts from staging dir to the libkrunfw checkout - cp "${DOCKER_STAGE}/kernel.c" "${LIBKRUNFW_DIR}/kernel.c" - if [ -f "${DOCKER_STAGE}/kernel.config" ]; then - mkdir -p "${LIBKRUNFW_DIR}/${KERNEL_DIR_NAME}" - cp "${DOCKER_STAGE}/kernel.config" "${LIBKRUNFW_DIR}/${KERNEL_DIR_NAME}/.config" + if [ ! -f "${LIBKRUNFW_DIR}/kernel.c" ]; then + echo "ERROR: kernel.c was not produced — build failed" >&2 + exit 1 fi # Compile the shared library on the host (uses host cc for a .dylib) diff --git a/crates/openshell-vm/runtime/kernel/bridge-cni.config b/crates/openshell-vm/runtime/kernel/bridge-cni.config index 2e484b070..7b9610e30 100644 --- a/crates/openshell-vm/runtime/kernel/bridge-cni.config +++ b/crates/openshell-vm/runtime/kernel/bridge-cni.config @@ -35,11 +35,27 @@ CONFIG_NF_CONNTRACK_EVENTS=y CONFIG_NF_CONNTRACK_TIMEOUT=y CONFIG_NF_CONNTRACK_TIMESTAMP=y +# ── Netfilter xtables match modules (required by kube-proxy & kubelet) ─ +# kube-proxy uses xt_conntrack for stateful rules and xt_comment for +# labeling chains. Without these, iptables fails with: +# "Couldn't load match 'conntrack': No such file or directory" +CONFIG_NETFILTER_XTABLES=y +CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y +CONFIG_NETFILTER_XT_MATCH_COMMENT=y +CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y +CONFIG_NETFILTER_XT_MATCH_MARK=y +CONFIG_NETFILTER_XT_MATCH_STATISTIC=y +CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y +CONFIG_NETFILTER_XT_MATCH_RECENT=y +CONFIG_NETFILTER_XT_TARGET_MARK=y +CONFIG_NETFILTER_XT_TARGET_CONNMARK=y +CONFIG_NETFILTER_XT_MATCH_CONNMARK=y + # ── NAT (required for service VIP / DNAT / SNAT) ────────────────────── CONFIG_NF_NAT=y CONFIG_NF_NAT_MASQUERADE_IPV4=y -# ── iptables (kube-proxy iptables mode) ──────────────────────────────── +# ── iptables (CNI bridge masquerade + compat) ────────────────────────── CONFIG_IP_NF_IPTABLES=y CONFIG_IP_NF_FILTER=y CONFIG_IP_NF_NAT=y @@ -47,7 +63,9 @@ CONFIG_IP_NF_MANGLE=y CONFIG_IP_NF_TARGET_MASQUERADE=y CONFIG_IP_NF_TARGET_REJECT=y -# ── nftables (kube-proxy nft mode, future-proofing) ──────────────────── +# ── nftables (kube-proxy nftables mode — primary proxy backend) ───────── +# kube-proxy nftables proxier requires: numgen (random LB), fib (local +# address detection), counter, ct, nat, masq, reject, limit, redir. CONFIG_NF_TABLES=y CONFIG_NF_TABLES_INET=y CONFIG_NFT_CT=y @@ -55,7 +73,12 @@ CONFIG_NFT_NAT=y CONFIG_NFT_MASQ=y CONFIG_NFT_REJECT=y CONFIG_NFT_COMPAT=y -CONFIG_NFT_COUNTER=y +CONFIG_NFT_NUMGEN=y +CONFIG_NFT_FIB_IPV4=y +CONFIG_NFT_FIB_IPV6=y +CONFIG_NFT_LIMIT=y +CONFIG_NFT_REDIR=y +CONFIG_NFT_TPROXY=y # ── IP forwarding and routing (required for pod-to-pod) ──────────────── CONFIG_IP_ADVANCED_ROUTER=y @@ -90,3 +113,7 @@ CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_PIDS=y CONFIG_MEMCG=y + +# ── Security features required by the sandbox runtime ─────────────────── +CONFIG_SECURITY_LANDLOCK=y +CONFIG_SECCOMP_FILTER=y diff --git a/crates/openshell-vm/scripts/build-rootfs.sh b/crates/openshell-vm/scripts/build-rootfs.sh index d2fed06e0..4c4064977 100755 --- a/crates/openshell-vm/scripts/build-rootfs.sh +++ b/crates/openshell-vm/scripts/build-rootfs.sh @@ -65,7 +65,7 @@ docker rm -f "${INIT_CONTAINER_NAME}" 2>/dev/null || true echo "==> Building base image..." docker build --platform linux/arm64 -t "${BASE_IMAGE_TAG}" -f - . <<'DOCKERFILE' -FROM ubuntu:22.04 +FROM nvcr.io/nvidia/base/ubuntu:noble-20251013 RUN apt-get update && \ apt-get install -y --no-install-recommends \ ca-certificates \ diff --git a/crates/openshell-vm/scripts/gateway-init.sh b/crates/openshell-vm/scripts/gateway-init.sh index 0d6c92742..ab1b8c08c 100755 --- a/crates/openshell-vm/scripts/gateway-init.sh +++ b/crates/openshell-vm/scripts/gateway-init.sh @@ -153,14 +153,16 @@ find /run -name '*.sock' -delete 2>/dev/null || true # # The kine (SQLite) DB cleanup in build-rootfs.sh already removes stale # pod/sandbox records from k3s etcd, preventing kubelet from reconciling -# against stale sandboxes. Containerd's internal sandbox records in -# meta.db are harmless because the CRI plugin reconciles with kubelet -# on startup — any sandboxes unknown to kubelet are cleaned up gracefully -# without triggering SandboxChanged events. +# against stale sandboxes. CONTAINERD_DIR="/var/lib/rancher/k3s/agent/containerd" if [ -d "$CONTAINERD_DIR" ]; then # Remove runtime task state (stale shim PIDs, sockets from dead processes). rm -rf "${CONTAINERD_DIR}/io.containerd.runtime.v2.task" 2>/dev/null || true + # Remove sandbox controller shim state. Stale sandbox records cause + # containerd to reuse network namespaces from previous boots, which + # already have routes configured. The CNI bridge plugin then fails + # with "file exists" when adding the default route on retry. + rm -rf "${CONTAINERD_DIR}/io.containerd.sandbox.controller.v1.shim" 2>/dev/null || true # Clean stale ingest temp files from the content store. rm -rf "${CONTAINERD_DIR}/io.containerd.content.v1.content/ingest" 2>/dev/null || true mkdir -p "${CONTAINERD_DIR}/io.containerd.content.v1.content/ingest" @@ -171,6 +173,83 @@ rm -rf /run/k3s 2>/dev/null || true ts "stale artifacts cleaned" +# ── Clean stale CNI / pod networking state ────────────────────────────── +# The rootfs persists across VM restarts via virtio-fs. Previous pod +# sandboxes leave behind veth pairs, bridge routes, host-local IPAM +# allocations, and network namespaces. If not cleaned, the bridge CNI +# plugin fails with: +# "failed to add route ... file exists" +# because the default route via cni0 already exists from the prior boot, +# or a stale network namespace already has the route configured. + +# Tear down the CNI bridge and its associated routes. +if ip link show cni0 >/dev/null 2>&1; then + ip link set cni0 down 2>/dev/null || true + ip link delete cni0 2>/dev/null || true + ts "deleted stale cni0 bridge" +fi + +# Remove any leftover veth pairs (CNI bridge plugin creates vethXXXX). +for veth in $(ip -o link show type veth 2>/dev/null | awk -F': ' '{print $2}' | cut -d'@' -f1); do + ip link delete "$veth" 2>/dev/null || true +done + +# Flush host-local IPAM allocations so IPs can be reassigned cleanly. +rm -rf /var/lib/cni/networks 2>/dev/null || true +rm -rf /var/lib/cni/results 2>/dev/null || true + +# Flush any stale CNI-added routes for the pod CIDR. These can conflict +# with routes the bridge plugin tries to add on the next boot. +ip route flush 10.42.0.0/24 2>/dev/null || true + +# Clean up stale pod network namespaces from previous boots. Containerd +# creates named netns under /var/run/netns/ for each pod sandbox. If +# these persist across VM restarts, the CNI bridge plugin fails when +# adding routes because the stale netns already has the default route +# configured from the prior boot. Removing all named network namespaces +# forces containerd to create fresh ones. +if [ -d /var/run/netns ]; then + for ns in $(ip netns list 2>/dev/null | awk '{print $1}'); do + ip netns delete "$ns" 2>/dev/null || true + done +fi +# Also clean the netns bind-mount directory used by containerd/CRI. +# Containerd may use /run/netns/ or /var/run/netns/ (same via tmpfs). +rm -rf /run/netns/* 2>/dev/null || true +rm -rf /var/run/netns/* 2>/dev/null || true + +ts "stale CNI networking state cleaned" + +# ── Network profile detection ─────────────────────────────────────────── +# Detect early so manifest patching and k3s flags both use the same value. +# +# "bridge" is the only supported profile. It requires a custom libkrunfw +# with CONFIG_BRIDGE, CONFIG_NETFILTER, CONFIG_NF_NAT built in. If the +# kernel lacks these capabilities the VM cannot run pod networking and we +# fail fast with an actionable error. + +NET_PROFILE="bridge" + +ts "network profile: ${NET_PROFILE}" + +# Validate that the kernel actually has the required capabilities. +_caps_ok=true +if ! ip link add _cap_br0 type bridge 2>/dev/null; then + echo "ERROR: kernel lacks bridge support (CONFIG_BRIDGE). Use a custom libkrunfw." >&2 + _caps_ok=false +else + ip link del _cap_br0 2>/dev/null || true +fi +if [ ! -d /proc/sys/net/netfilter ] && [ ! -f /proc/sys/net/bridge/bridge-nf-call-iptables ]; then + echo "ERROR: kernel lacks netfilter support (CONFIG_NETFILTER). Use a custom libkrunfw." >&2 + _caps_ok=false +fi +if [ "$_caps_ok" = false ]; then + echo "FATAL: required kernel capabilities missing — cannot configure pod networking." >&2 + echo "See: architecture/custom-vm-runtime.md for build instructions." >&2 + exit 1 +fi + # ── Deploy bundled manifests (cold boot only) ─────────────────────────── # On pre-initialized rootfs, manifests are already in place from the # build-time k3s boot. Skip this entirely for fast startup. @@ -211,15 +290,13 @@ if [ -f "$HELMCHART" ]; then # Use pre-loaded images — don't pull from registry. sed -i 's|pullPolicy: Always|pullPolicy: IfNotPresent|' "$HELMCHART" - if [ "$NET_PROFILE" = "bridge" ]; then - # Bridge CNI: pods use normal pod networking, not hostNetwork. - sed -i 's|__HOST_NETWORK__|false|g' "$HELMCHART" - sed -i 's|__AUTOMOUNT_SA_TOKEN__|true|g' "$HELMCHART" - else - # Legacy: VM bootstrap runs without CNI bridge networking. - sed -i 's|__HOST_NETWORK__|true|g' "$HELMCHART" - sed -i 's|__AUTOMOUNT_SA_TOKEN__|false|g' "$HELMCHART" - fi + # Bridge CNI: pods use normal pod networking, not hostNetwork. + # The pre-init in build-rootfs.sh replaces __HOST_NETWORK__ with "true" + # for Docker container networking. At VM boot with bridge CNI we need + # to override it back to "false" so pods use the CNI bridge network. + sed -i 's|hostNetwork: true|hostNetwork: false|g' "$HELMCHART" + sed -i 's|__HOST_NETWORK__|false|g' "$HELMCHART" + sed -i 's|__AUTOMOUNT_SA_TOKEN__|true|g' "$HELMCHART" sed -i 's|__KUBECONFIG_HOST_PATH__|"/etc/rancher/k3s"|g' "$HELMCHART" sed -i 's|__PERSISTENCE_ENABLED__|false|g' "$HELMCHART" @@ -231,115 +308,34 @@ fi AGENT_MANIFEST="$K3S_MANIFESTS/agent-sandbox.yaml" if [ -f "$AGENT_MANIFEST" ]; then - if [ "$NET_PROFILE" = "bridge" ]; then - # Bridge CNI: agent-sandbox uses normal pod networking. - # kube-proxy is enabled so kubernetes.default.svc is reachable - # via ClusterIP — no need for KUBERNETES_SERVICE_HOST override. - sed -i '/hostNetwork: true/d' "$AGENT_MANIFEST" - sed -i '/dnsPolicy: ClusterFirstWithHostNet/d' "$AGENT_MANIFEST" - ts "agent-sandbox: using pod networking (bridge profile)" - else - # Legacy: keep agent-sandbox on pod networking to avoid host port - # clashes. Point in-cluster client traffic at the API server node - # IP because kube-proxy is disabled in VM mode. - sed -i '/hostNetwork: true/d' "$AGENT_MANIFEST" - sed -i '/dnsPolicy: ClusterFirstWithHostNet/d' "$AGENT_MANIFEST" - if ! grep -q 'metrics-bind-address=:8082' "$AGENT_MANIFEST"; then - sed -i 's|image: registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.1.0|image: registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.1.0\ - args:\ - - -metrics-bind-address=:8082\ - env:\ - - name: KUBERNETES_SERVICE_HOST\ - value: 192.168.127.2\ - - name: KUBERNETES_SERVICE_PORT\ - value: "6443"|g' "$AGENT_MANIFEST" - else - sed -i 's|value: 127.0.0.1|value: 192.168.127.2|g' "$AGENT_MANIFEST" - fi - if grep -q 'hostNetwork: true' "$AGENT_MANIFEST" \ - || grep -q 'ClusterFirstWithHostNet' "$AGENT_MANIFEST" \ - || ! grep -q 'KUBERNETES_SERVICE_HOST' "$AGENT_MANIFEST" \ - || ! grep -q 'metrics-bind-address=:8082' "$AGENT_MANIFEST"; then - echo "ERROR: failed to patch agent-sandbox manifest for VM networking constraints: $AGENT_MANIFEST" >&2 - exit 1 - fi - ts "agent-sandbox: patched for legacy-vm-net (API server override)" - fi + # Bridge CNI: agent-sandbox uses normal pod networking. + # kube-proxy is enabled so kubernetes.default.svc is reachable + # via ClusterIP — no need for KUBERNETES_SERVICE_HOST override. + sed -i '/hostNetwork: true/d' "$AGENT_MANIFEST" + sed -i '/dnsPolicy: ClusterFirstWithHostNet/d' "$AGENT_MANIFEST" + ts "agent-sandbox: using pod networking (bridge profile)" fi -# local-storage implies local-path-provisioner. In legacy mode it -# requires CNI bridge networking that is unavailable. In bridge mode -# it can work but we leave it disabled for now until validated. -if [ "$NET_PROFILE" != "bridge" ]; then - rm -f "$K3S_MANIFESTS/local-storage.yaml" 2>/dev/null || true -fi - -# ── CNI configuration ─────────────────────────────────────────────────── -# Two networking profiles: -# -# 1. "bridge" (default when kernel supports it): -# Uses the bridge CNI plugin with iptables masquerade. Requires -# CONFIG_BRIDGE, CONFIG_NETFILTER, CONFIG_NF_NAT in the VM kernel. -# This is the standard Kubernetes CNI path — compatible with -# kube-proxy, service VIPs, and portmap. -# -# 2. "legacy-vm-net" (fallback for stock libkrunfw without netfilter): -# Uses ptp CNI without iptables. No masquerade, no portmap. -# kube-proxy must be disabled. This is the original VM path. -# -# The profile is auto-detected from kernel capabilities but can be -# forced via OPENSHELL_VM_NET_PROFILE=bridge|legacy-vm-net. - -NET_PROFILE="${OPENSHELL_VM_NET_PROFILE:-auto}" - -detect_net_profile() { - # Check for bridge + netfilter kernel support. - # If we can create a bridge and the netfilter sysctl exists, use bridge CNI. - local has_bridge=false - local has_netfilter=false - - if ip link add _probe_br0 type bridge 2>/dev/null; then - ip link del _probe_br0 2>/dev/null || true - has_bridge=true - fi - - if [ -d /proc/sys/net/netfilter ] || [ -f /proc/sys/net/bridge/bridge-nf-call-iptables ]; then - has_netfilter=true - fi - - if [ "$has_bridge" = true ] && [ "$has_netfilter" = true ]; then - echo "bridge" - else - echo "legacy-vm-net" - fi -} - -if [ "$NET_PROFILE" = "auto" ]; then - NET_PROFILE=$(detect_net_profile) -fi - -ts "network profile: ${NET_PROFILE}" +# ── CNI configuration (bridge) ────────────────────────────────────────── +# Uses the bridge CNI plugin with iptables masquerade. Requires +# CONFIG_BRIDGE, CONFIG_NETFILTER, CONFIG_NF_NAT in the VM kernel +# (validated above at boot). kube-proxy uses nftables mode for service +# VIP routing. CNI_CONF_DIR="/etc/cni/net.d" CNI_BIN_DIR="/opt/cni/bin" mkdir -p "$CNI_CONF_DIR" "$CNI_BIN_DIR" -if [ "$NET_PROFILE" = "bridge" ]; then - # ── Bridge CNI (full Kubernetes networking) ───────────────────── - # This path requires a custom libkrunfw with bridge + netfilter - # kernel support. Creates a cni0 bridge, uses iptables masquerade, - # and is compatible with kube-proxy. +# Enable IP forwarding (required for masquerade). +echo 1 > /proc/sys/net/ipv4/ip_forward 2>/dev/null || true - # Enable IP forwarding (required for masquerade). - echo 1 > /proc/sys/net/ipv4/ip_forward 2>/dev/null || true +# Enable bridge netfilter call (required for CNI bridge masquerade to +# see bridged traffic). +if [ -f /proc/sys/net/bridge/bridge-nf-call-iptables ]; then + echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables 2>/dev/null || true +fi - # Enable bridge netfilter call (required for kube-proxy to see - # bridged traffic). - if [ -f /proc/sys/net/bridge/bridge-nf-call-iptables ]; then - echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables 2>/dev/null || true - fi - - cat > "$CNI_CONF_DIR/10-bridge.conflist" << 'CNICFG' +cat > "$CNI_CONF_DIR/10-bridge.conflist" << 'CNICFG' { "cniVersion": "1.0.0", "name": "bridge", @@ -353,8 +349,7 @@ if [ "$NET_PROFILE" = "bridge" ]; then "hairpinMode": true, "ipam": { "type": "host-local", - "ranges": [[{ "subnet": "10.42.0.0/24" }]], - "routes": [{ "dst": "0.0.0.0/0" }] + "ranges": [[{ "subnet": "10.42.0.0/24" }]] } }, { @@ -369,53 +364,19 @@ if [ "$NET_PROFILE" = "bridge" ]; then } CNICFG - # Remove any legacy ptp config. - rm -f "$CNI_CONF_DIR/10-ptp.conflist" 2>/dev/null || true +# Remove any stale legacy ptp config. +rm -f "$CNI_CONF_DIR/10-ptp.conflist" 2>/dev/null || true - ts "bridge CNI configured (cni0 + iptables masquerade)" -else - # ── Legacy ptp CNI (iptables-free) ───────────────────────────── - # The libkrun VM kernel has no netfilter/iptables support. Flannel's - # masquerade rules and kube-proxy both require iptables and crash - # without it. We disable both and use a simple ptp CNI with - # host-local IPAM instead. This avoids linux bridge requirements. - # - # ipMasq=false avoids any iptables calls in the plugin. - # portmap plugin removed — it requires iptables for DNAT rules. - - cat > "$CNI_CONF_DIR/10-ptp.conflist" << 'CNICFG' -{ - "cniVersion": "1.0.0", - "name": "ptp", - "plugins": [ - { - "type": "ptp", - "ipMasq": false, - "ipam": { - "type": "host-local", - "ranges": [[{ "subnet": "10.42.0.0/24" }]], - "routes": [{ "dst": "0.0.0.0/0" }] - } - }, - { - "type": "loopback" - } - ] -} -CNICFG - - # Remove any bridge config. - rm -f "$CNI_CONF_DIR/10-bridge.conflist" 2>/dev/null || true - - ts "ptp CNI configured (iptables-free, no linux bridge)" -fi +ts "bridge CNI configured (cni0 + iptables masquerade)" # Symlink k3s-bundled CNI binaries to the default containerd bin path. # k3s extracts its tools to /var/lib/rancher/k3s/data//bin/. +# Use -e (not -f) because k3s ships these as symlinks to a `cni` multicall +# binary. K3S_DATA_BIN=$(find /var/lib/rancher/k3s/data -maxdepth 2 -name bin -type d 2>/dev/null | head -1) if [ -n "$K3S_DATA_BIN" ]; then - for plugin in bridge ptp host-local loopback bandwidth portmap; do - [ -f "$K3S_DATA_BIN/$plugin" ] && ln -sf "$K3S_DATA_BIN/$plugin" "$CNI_BIN_DIR/$plugin" + for plugin in bridge host-local loopback bandwidth portmap; do + [ -e "$K3S_DATA_BIN/$plugin" ] && ln -sf "$K3S_DATA_BIN/$plugin" "$CNI_BIN_DIR/$plugin" done ts "CNI binaries linked from $K3S_DATA_BIN" else @@ -427,11 +388,15 @@ fi rm -f "/var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist" 2>/dev/null || true # ── Start k3s ────────────────────────────────────────────────────────── -# Flags tuned for fast single-node startup. The k3s flags vary depending -# on the network profile: +# Flags tuned for fast single-node startup. Bridge CNI handles pod +# networking; kube-proxy runs in nftables mode for service VIP / ClusterIP +# support. # -# bridge: kube-proxy enabled, flannel disabled (bridge CNI handles it) -# legacy-vm-net: kube-proxy disabled (no iptables), flannel disabled +# nftables mode: k3s bundles its own iptables binaries whose MARK target +# doesn't negotiate xt_MARK revision 2 correctly with the libkrun kernel, +# causing --xor-mark failures. nftables mode uses the kernel's nf_tables +# subsystem directly and sidesteps the issue entirely. The kernel is +# configured with CONFIG_NF_TABLES=y and related modules. K3S_ARGS=( --disable=traefik,servicelb,metrics-server,coredns @@ -443,19 +408,9 @@ K3S_ARGS=( --tls-san=localhost,127.0.0.1,10.0.2.15,192.168.127.2 --flannel-backend=none --snapshotter=native + --kube-proxy-arg=proxy-mode=nftables ) -if [ "$NET_PROFILE" = "bridge" ]; then - # With bridge CNI + iptables, kube-proxy can run. Don't disable it. - # local-storage can also work with bridge networking. - ts "starting k3s server (bridge profile — kube-proxy enabled)" -else - # Legacy: no iptables means no kube-proxy and no local-storage. - K3S_ARGS+=( - --disable=local-storage - --disable-kube-proxy - ) - ts "starting k3s server (legacy-vm-net profile — kube-proxy disabled)" -fi +ts "starting k3s server (bridge CNI + nftables kube-proxy)" exec /usr/local/bin/k3s server "${K3S_ARGS[@]}" diff --git a/crates/openshell-vm/scripts/verify-vm.sh b/crates/openshell-vm/scripts/verify-vm.sh index 31e65931f..a314301f1 100755 --- a/crates/openshell-vm/scripts/verify-vm.sh +++ b/crates/openshell-vm/scripts/verify-vm.sh @@ -157,24 +157,18 @@ check "kubernetes service has ClusterIP" "networking" \ "kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}' | grep -q ." # Check if bridge CNI is in use (cni0 bridge exists) -CNI_PROFILE="unknown" if kubectl exec -n openshell openshell-0 -- ip link show cni0 >/dev/null 2>&1; then - CNI_PROFILE="bridge" + echo " CNI profile detected: bridge" else - CNI_PROFILE="legacy-vm-net" + echo " WARNING: cni0 bridge not detected — bridge CNI may not be running yet" fi -echo " CNI profile detected: ${CNI_PROFILE}" -if [ "$CNI_PROFILE" = "bridge" ]; then - check "cni0 bridge exists in pod" "networking" \ - "kubectl exec -n openshell openshell-0 -- ip link show cni0 2>/dev/null" +check "cni0 bridge exists in pod" "networking" \ + "kubectl exec -n openshell openshell-0 -- ip link show cni0 2>/dev/null" - # With bridge CNI, kubernetes.default.svc should be reachable. - check "kubernetes.default.svc reachable from pod" "networking" \ - "kubectl exec -n openshell openshell-0 -- wget -q -O /dev/null --timeout=5 https://kubernetes.default.svc/healthz 2>/dev/null || kubectl exec -n openshell openshell-0 -- curl -sk --connect-timeout 5 https://kubernetes.default.svc/healthz 2>/dev/null" -else - echo " (skipping bridge-specific checks for legacy-vm-net profile)" -fi +# With bridge CNI, kubernetes.default.svc should be reachable. +check "kubernetes.default.svc reachable from pod" "networking" \ + "kubectl exec -n openshell openshell-0 -- wget -q -O /dev/null --timeout=5 https://kubernetes.default.svc/healthz 2>/dev/null || kubectl exec -n openshell openshell-0 -- curl -sk --connect-timeout 5 https://kubernetes.default.svc/healthz 2>/dev/null" check "no bridge creation errors in events" "networking" \ "! kubectl get events -A 2>/dev/null | grep -qi 'bridge.*fail\\|cni0.*error\\|FailedCreatePodSandBox.*bridge'" @@ -185,9 +179,6 @@ echo "" echo "[Host Connectivity]" -check "port 6443 (kube-apiserver) reachable" "host" \ - "timeout 5 bash -c 'echo > /dev/tcp/127.0.0.1/6443' 2>/dev/null || nc -z -w5 127.0.0.1 6443 2>/dev/null" - check "port 30051 (gateway service) reachable" "host" \ "timeout 5 bash -c 'echo > /dev/tcp/127.0.0.1/30051' 2>/dev/null || nc -z -w5 127.0.0.1 30051 2>/dev/null" diff --git a/crates/openshell-vm/src/lib.rs b/crates/openshell-vm/src/lib.rs index b14c234f4..e6bd5ce7c 100644 --- a/crates/openshell-vm/src/lib.rs +++ b/crates/openshell-vm/src/lib.rs @@ -146,8 +146,7 @@ impl VmConfig { /// /// Runs `/srv/gateway-init.sh` which mounts essential filesystems, /// deploys the OpenShell helm chart, and execs `k3s server`. - /// Exposes the Kubernetes API on port 6443 and the OpenShell - /// gateway (`NodePort`) on port 30051. + /// Exposes the OpenShell gateway on port 30051. pub fn gateway(rootfs: PathBuf) -> Self { Self { rootfs, @@ -162,10 +161,6 @@ impl VmConfig { ], workdir: "/".to_string(), port_map: vec![ - // Map host 6443 -> guest 6444 (real kube-apiserver). - // The k3s dynamiclistener on 6443 has TLS issues through - // port forwarding, so we go directly to the apiserver. - "6443:6444".to_string(), // Navigator server — with hostNetwork the server binds // directly to port 8080 on the VM's interface, bypassing // NodePort (which requires kube-proxy / iptables). @@ -369,6 +364,7 @@ fn raise_nofile_limit() { fn log_runtime_provenance(runtime_dir: &Path) { if let Some(prov) = ffi::runtime_provenance() { eprintln!("runtime: {}", runtime_dir.display()); + eprintln!(" libkrun: {}", prov.libkrun_path.display()); for krunfw in &prov.libkrunfw_paths { let name = krunfw .file_name() @@ -993,8 +989,6 @@ pub fn launch(config: &VmConfig) -> Result { match std::fs::read_to_string(&kubeconfig_src) { Ok(contents) => { - // The kubeconfig has server: https://127.0.0.1:6443 - // which is correct since we forward host:6443 -> guest:6444. if let Err(e) = std::fs::write(&dest, &contents) { eprintln!(" failed to write kubeconfig: {e}"); } else { diff --git a/tasks/scripts/bundle-vm-runtime.sh b/tasks/scripts/bundle-vm-runtime.sh index bc91e9b35..f391363ac 100755 --- a/tasks/scripts/bundle-vm-runtime.sh +++ b/tasks/scripts/bundle-vm-runtime.sh @@ -35,10 +35,19 @@ if [ -z "$GVPROXY_BIN" ]; then fi fi +# libkrun.dylib: prefer the custom runtime dir, fall back to Homebrew. +# libkrun is the VMM and does not need a custom build; only libkrunfw +# carries the custom kernel. LIBKRUN="${LIB_DIR}/libkrun.dylib" if [ ! -e "$LIBKRUN" ]; then - echo "libkrun not found at ${LIBKRUN}; set OPENSHELL_VM_RUNTIME_SOURCE_DIR" >&2 - exit 1 + BREW_PREFIX="${BREW_PREFIX:-$(brew --prefix 2>/dev/null || true)}" + if [ -n "$BREW_PREFIX" ] && [ -e "${BREW_PREFIX}/lib/libkrun.dylib" ]; then + LIBKRUN="${BREW_PREFIX}/lib/libkrun.dylib" + echo "using Homebrew libkrun at ${LIBKRUN}" + else + echo "libkrun not found at ${LIB_DIR}/libkrun.dylib or Homebrew; install libkrun or set OPENSHELL_VM_RUNTIME_SOURCE_DIR" >&2 + exit 1 + fi fi KRUNFW_FILES=()