fix(vm): resolve CNI bridge and nftables errors for reliable pod networking

Switch kube-proxy to nftables mode and add missing kernel config options
(NFT_NUMGEN, NFT_FIB_IPV4/6, NFT_LIMIT, NFT_REDIR, NFT_TPROXY) plus
xtables match modules required by CNI bridge masquerade. Add stale CNI
state cleanup on boot (cni0 bridge, veth pairs, IPAM allocations, pod
network namespaces, sandbox controller shim) to prevent 'route already
exists' errors from persistent rootfs. Remove dual bridge/legacy-vm-net
profile system in favor of bridge-only with fail-fast kernel validation.
Drop host-mapped port 6443 (kube-apiserver) since it is not needed for
normal gateway operation. Update bundle script to fall back to Homebrew
for libkrun.dylib (VMM) while still requiring custom libkrunfw (kernel).

architecture/custom-vm-runtime.md

@@ -7,8 +7,7 @@ lightweight microVM with Apple Hypervisor.framework (macOS) or KVM (Linux). The
 is embedded inside `libkrunfw`, a companion library that packages a pre-built Linux kernel.
 
 The stock `libkrunfw` from Homebrew ships a minimal kernel without bridge, netfilter, or
-conntrack support. This limits Kubernetes networking to a point-to-point (ptp) CNI
-configuration without kube-proxy or service VIPs.
+conntrack support. This is insufficient for Kubernetes pod networking.
 
 The custom libkrunfw runtime adds bridge CNI, iptables/nftables, and conntrack support to
 the VM kernel, enabling standard Kubernetes networking.
@@ -31,33 +30,25 @@ Host (macOS/Linux)
 
 Guest VM
 ├── gateway-init.sh (PID 1)
-│   ├── Detects kernel capabilities
-│   ├── Selects network profile: bridge | legacy-vm-net
-│   ├── Configures CNI
+│   ├── Validates kernel capabilities (fail-fast)
+│   ├── Configures bridge CNI
 │   └── Execs k3s server
 └── check-vm-capabilities.sh (diagnostics)
 ```
 
-## Network Profiles
+## Network Profile
 
-The VM init script auto-detects kernel capabilities and selects the appropriate
-networking profile. This can be overridden via `OPENSHELL_VM_NET_PROFILE`.
+The VM uses the bridge CNI profile, which requires a custom libkrunfw with bridge and
+netfilter kernel support. The init script validates these capabilities at boot and fails
+fast with an actionable error if they are missing.
 
-### Bridge Profile (custom runtime)
+### Bridge Profile
 
 - CNI: bridge plugin with `cni0` interface
-- IP masquerade: enabled (iptables)
-- kube-proxy: enabled
+- IP masquerade: enabled (iptables-legacy via CNI bridge plugin)
+- kube-proxy: enabled (nftables mode)
 - Service VIPs: functional (ClusterIP, NodePort)
-- hostNetwork workarounds: removed
-
-### Legacy VM Net Profile (stock runtime)
-
-- CNI: ptp plugin (point-to-point)
-- IP masquerade: disabled (no iptables)
-- kube-proxy: disabled
-- Service VIPs: not functional (direct IP routing only)
-- hostNetwork workarounds: required for some pods
+- hostNetwork workarounds: not required
 
 ## Runtime Provenance
 
@@ -98,10 +89,12 @@ libkrunfw kernel:
 | Netfilter | `CONFIG_NETFILTER` | iptables/nftables framework |
 | Connection tracking | `CONFIG_NF_CONNTRACK` | NAT state tracking |
 | NAT | `CONFIG_NF_NAT` | Service VIP DNAT/SNAT |
-| iptables | `CONFIG_IP_NF_IPTABLES` | kube-proxy iptables mode |
-| nftables | `CONFIG_NF_TABLES` | kube-proxy nft mode (future) |
+| iptables | `CONFIG_IP_NF_IPTABLES` | CNI bridge masquerade |
+| nftables | `CONFIG_NF_TABLES` | kube-proxy nftables mode (primary) |
 | veth | `CONFIG_VETH` | Pod network namespace pairs |
 | IPVS | `CONFIG_IP_VS` | kube-proxy IPVS mode (optional) |
+| Landlock | `CONFIG_SECURITY_LANDLOCK` | Filesystem sandboxing support |
+| Seccomp filter | `CONFIG_SECCOMP_FILTER` | Syscall filtering support |
 
 ## Verification
 
@@ -116,10 +109,9 @@ Two verification tools are provided:
 ## Rollout Strategy
 
 1. Custom runtime support is opt-in via `OPENSHELL_VM_RUNTIME_SOURCE_DIR`.
-2. Auto-detection selects the correct network profile at boot.
-3. The stock runtime path (`legacy-vm-net`) remains the default until the custom
-   runtime is proven stable.
-4. Rollback: unset the env var and re-bundle with stock libraries.
+2. The init script validates kernel capabilities at boot and fails fast if missing.
+3. Rollback: unset the env var and re-bundle with stock libraries (note: stock
+   libraries lack bridge/netfilter and pod networking will not work).
 
 ## Related Files
 

crates/openshell-vm/runtime/README.md

@@ -9,11 +9,11 @@ The stock `libkrunfw` (from Homebrew) ships a kernel without bridge, netfilter,
 or conntrack support. This means the VM cannot:
 
 - Create `cni0` bridge interfaces (required by the bridge CNI plugin)
-- Run kube-proxy (requires iptables/nftables)
+- Run kube-proxy (requires nftables)
 - Route service VIP traffic (requires NAT/conntrack)
 
 The custom runtime builds libkrunfw with an additional kernel config fragment
-that enables these features.
+that enables these networking and sandboxing features.
 
 ## Directory Structure
 
@@ -21,7 +21,7 @@ that enables these features.
 runtime/
   build-custom-libkrunfw.sh   # Build script for custom libkrunfw
   kernel/
-    bridge-cni.config          # Kernel config fragment (bridge + netfilter)
+    bridge-cni.config          # Kernel config fragment (networking + sandboxing)
 ```
 
 ## Building
@@ -66,23 +66,16 @@ mise run vm:bundle-runtime
 mise run vm
 ```
 
-## Network Profiles
+## Networking
 
-The VM init script (`gateway-init.sh`) auto-detects the kernel capabilities
-and selects the appropriate networking profile:
+The VM uses bridge CNI for pod networking with nftables-mode kube-proxy for
+service VIP / ClusterIP support. The kernel config fragment enables both
+iptables (for CNI bridge masquerade) and nftables (for kube-proxy).
 
-| Profile | Kernel | CNI | kube-proxy | Service VIPs |
-|---------|--------|-----|------------|--------------|
-| `bridge` | Custom (bridge+netfilter) | bridge CNI (`cni0`) | Enabled | Yes |
-| `legacy-vm-net` | Stock (no netfilter) | ptp CNI | Disabled | No (direct IP) |
-
-To force a specific profile:
-
-```bash
-# Inside the VM (set in gateway-init.sh env):
-export OPENSHELL_VM_NET_PROFILE=bridge      # Force bridge CNI
-export OPENSHELL_VM_NET_PROFILE=legacy-vm-net  # Force legacy ptp CNI
-```
+k3s is started with `--kube-proxy-arg=proxy-mode=nftables` because the
+bundled iptables binaries in k3s have revision-negotiation issues with the
+libkrun kernel's xt_MARK module. nftables mode uses the kernel's nf_tables
+subsystem directly and avoids this entirely.
 
 ## Runtime Provenance
 
@@ -158,14 +151,17 @@ If this fails, you are running the stock runtime. Build and use the custom one.
 
 ### kube-proxy CrashLoopBackOff
 
-The kernel does not have netfilter support. Verify:
+kube-proxy runs in nftables mode. If it crashes, verify nftables support:
 ```bash
 # Inside VM:
-iptables -L -n
+nft list ruleset
 ```
 
-If this fails with "iptables not found" or "modprobe: can't change directory",
-the kernel lacks CONFIG_NETFILTER. Use the custom runtime.
+If this fails, the kernel may lack `CONFIG_NF_TABLES`. Use the custom runtime.
+
+Common errors:
+- `unknown option "--xor-mark"`: kube-proxy is running in iptables mode instead
+  of nftables. Verify `--kube-proxy-arg=proxy-mode=nftables` is in the k3s args.
 
 ### Runtime mismatch after upgrade
 

crates/openshell-vm/runtime/build-custom-libkrunfw.sh

@@ -152,6 +152,13 @@ REQUIRED=(
     CONFIG_IP_NF_IPTABLES
     CONFIG_IP_NF_FILTER
     CONFIG_IP_NF_NAT
+    CONFIG_NF_TABLES
+    CONFIG_NFT_NUMGEN
+    CONFIG_NFT_FIB_IPV4
+    CONFIG_NFT_FIB_IPV6
+    CONFIG_NFT_CT
+    CONFIG_NFT_NAT
+    CONFIG_NFT_MASQ
     CONFIG_VETH
     CONFIG_NET_NS
 )
@@ -195,103 +202,75 @@ fi
 echo "  Kernel source dir: ${KERNEL_DIR_NAME}"
 
 if [ "$(uname -s)" = "Darwin" ]; then
-    # On macOS, the entire kernel build (extract, patch, config, compile)
-    # must happen inside a Linux container since the kernel build system
-    # requires a Linux toolchain. We mount the checkout, inject the
-    # config fragment, and produce kernel.c inside the container.
+    # On macOS, use krunvm to build the kernel inside a lightweight Linux VM.
+    # This matches the upstream libkrunfw build approach and avoids all the
+    # issues with Docker emulation and APFS filesystem limitations.
+    #
+    # Prerequisites: brew tap slp/krun && brew install krunvm
 
-    if ! command -v docker &>/dev/null; then
-        echo "ERROR: docker is required to build the kernel on macOS" >&2
+    if ! command -v krunvm &>/dev/null; then
+        echo "ERROR: krunvm is required to build the kernel on macOS" >&2
+        echo "  Install with: brew tap slp/krun && brew install krunvm" >&2
         exit 1
     fi
 
-    echo "==> Building kernel inside Docker (macOS detected)..."
+    echo "==> Building kernel inside krunvm (macOS detected)..."
 
-    # Pre-build a Docker image with all build dependencies so the
-    # (potentially emulated) apt-get/pip work is cached across runs.
-    BUILDER_IMAGE="libkrunfw-builder:bookworm"
-    if ! docker image inspect "${BUILDER_IMAGE}" >/dev/null 2>&1; then
-        echo "  Building Docker toolchain image (first time only)..."
-        docker build --platform linux/arm64 -t "${BUILDER_IMAGE}" - <<'DOCKERFILE'
-FROM debian:bookworm-slim
-RUN apt-get update -qq && \
-    apt-get install -y -qq make gcc bc flex bison libelf-dev python3 \
-        coreutils curl xz-utils patch libssl-dev cpio >/dev/null 2>&1 && \
-    rm -rf /var/lib/apt/lists/*
-DOCKERFILE
+    VM_NAME="libkrunfw-openshell"
+
+    # Clean up any leftover VM from a previous failed run
+    krunvm delete "${VM_NAME}" 2>/dev/null || true
+
+    # Copy the config fragment into the libkrunfw tree so the VM can see it.
+    # The merge hook (MERGE_HOOK) is already written there by the cat above.
+    cp -f "${KERNEL_CONFIG_FRAGMENT}" "${LIBKRUNFW_DIR}/openshell-bridge-cni.config"
+
+    echo "  Creating VM..."
+    # krunvm may print "The volume has been configured" on first use of a
+    # volume path and exit non-zero. Retry once if that happens.
+    if ! krunvm create fedora \
+        --name "${VM_NAME}" \
+        --cpus 4 \
+        --mem 4096 \
+        -v "${LIBKRUNFW_DIR}:/work" \
+        -w /work; then
+        echo "  Retrying VM creation..."
+        krunvm create fedora \
+            --name "${VM_NAME}" \
+            --cpus 4 \
+            --mem 4096 \
+            -v "${LIBKRUNFW_DIR}:/work" \
+            -w /work
     fi
 
-    # Stage only the files the build needs into a temp dir so we
-    # avoid copying the enormous kernel source tree over virtiofs.
-    DOCKER_STAGE=$(mktemp -d)
-    trap 'rm -rf "${DOCKER_STAGE}"' EXIT
+    echo "  Installing build dependencies..."
+    krunvm start "${VM_NAME}" /usr/bin/dnf -- install -y \
+        'dnf-command(builddep)' python3-pyelftools
 
-    echo "  Staging build inputs..."
-    cp "${LIBKRUNFW_DIR}/Makefile" "${DOCKER_STAGE}/"
-    cp "${LIBKRUNFW_DIR}/bin2cbundle.py" "${DOCKER_STAGE}/"
-    # Copy base kernel config(s) and patches
-    for f in "${LIBKRUNFW_DIR}"/config-libkrunfw*; do
-        [ -f "$f" ] && cp "$f" "${DOCKER_STAGE}/"
-    done
-    if [ -d "${LIBKRUNFW_DIR}/patches" ]; then
-        cp -r "${LIBKRUNFW_DIR}/patches" "${DOCKER_STAGE}/patches"
-    fi
-    if [ -d "${LIBKRUNFW_DIR}/patches-tee" ]; then
-        cp -r "${LIBKRUNFW_DIR}/patches-tee" "${DOCKER_STAGE}/patches-tee"
-    fi
-    # Copy the cached tarball if present so we don't re-download
-    if [ -d "${LIBKRUNFW_DIR}/tarballs" ]; then
-        cp -r "${LIBKRUNFW_DIR}/tarballs" "${DOCKER_STAGE}/tarballs"
-    fi
+    krunvm start "${VM_NAME}" /usr/bin/dnf -- builddep -y kernel
 
-    docker run --rm \
-        -v "${DOCKER_STAGE}:/work" \
-        -v "${KERNEL_CONFIG_FRAGMENT}:/fragment/bridge-cni.config:ro" \
-        -v "${MERGE_HOOK}:/fragment/merge-hook.sh:ro" \
-        -w /build \
-        --platform linux/arm64 \
-        "${BUILDER_IMAGE}" \
-        bash -c '
-            set -euo pipefail
+    # Step 1: prepare kernel sources (download, extract, patch, base config)
+    echo "  Preparing kernel sources..."
+    krunvm start "${VM_NAME}" /usr/bin/make -- "${KERNEL_DIR_NAME}"
 
-            KDIR="'"${KERNEL_DIR_NAME}"'"
+    # Step 2: merge the OpenShell config fragment
+    echo "  Merging OpenShell kernel config fragment..."
+    krunvm start "${VM_NAME}" /usr/bin/bash -- \
+        /work/openshell-kconfig-hook.sh "/work/${KERNEL_DIR_NAME}" /work/openshell-bridge-cni.config
 
-            # Copy staged inputs to container-local filesystem so tar
-            # extraction does not hit macOS APFS bind-mount limitations.
-            echo "  Copying build inputs to container filesystem..."
-            cp -r /work/* /build/
+    # Step 3: build the kernel and generate the C bundle
+    echo "  Building kernel (this is the slow part)..."
+    krunvm start "${VM_NAME}" /usr/bin/make -- -j4
 
-            # Patch bin2cbundle.py to make the elftools import lazy.
-            # We only use -t Image (raw path) which does not need elftools,
-            # but the top-level import would fail without pyelftools installed.
-            sed -i "s/^from elftools/# lazy: from elftools/" bin2cbundle.py
+    echo "  Cleaning up VM..."
+    krunvm delete "${VM_NAME}"
 
-            # Step 1: prepare kernel sources (download, extract, patch, base config, olddefconfig)
-            echo "  Preparing kernel sources..."
-            make "$KDIR"
+    # Clean up temp files from the libkrunfw tree
+    rm -f "${LIBKRUNFW_DIR}/openshell-bridge-cni.config"
 
-            # Step 2: merge the OpenShell config fragment
-            echo "  Merging OpenShell kernel config fragment..."
-            bash /fragment/merge-hook.sh "/build/$KDIR" /fragment/bridge-cni.config
-
-            # Step 3: build the kernel and generate the C bundle
-            echo "  Building kernel (this is the slow part)..."
-            make -j"$(nproc)" "$KDIR"/arch/arm64/boot/Image
-
-            echo "  Generating kernel.c bundle..."
-            python3 bin2cbundle.py -t Image "$KDIR"/arch/arm64/boot/Image kernel.c
-
-            # Copy artifacts back to the bind-mounted staging dir
-            echo "  Copying artifacts back to host..."
-            cp /build/kernel.c /work/kernel.c
-            cp /build/"$KDIR"/.config /work/kernel.config
-        '
-
-    # Move artifacts from staging dir to the libkrunfw checkout
-    cp "${DOCKER_STAGE}/kernel.c" "${LIBKRUNFW_DIR}/kernel.c"
-    if [ -f "${DOCKER_STAGE}/kernel.config" ]; then
-        mkdir -p "${LIBKRUNFW_DIR}/${KERNEL_DIR_NAME}"
-        cp "${DOCKER_STAGE}/kernel.config" "${LIBKRUNFW_DIR}/${KERNEL_DIR_NAME}/.config"
+    if [ ! -f "${LIBKRUNFW_DIR}/kernel.c" ]; then
+        echo "ERROR: kernel.c was not produced — build failed" >&2
+        exit 1
     fi
 
     # Compile the shared library on the host (uses host cc for a .dylib)

crates/openshell-vm/runtime/kernel/bridge-cni.config

@@ -35,11 +35,27 @@ CONFIG_NF_CONNTRACK_EVENTS=y
 CONFIG_NF_CONNTRACK_TIMEOUT=y
 CONFIG_NF_CONNTRACK_TIMESTAMP=y
 
+# ── Netfilter xtables match modules (required by kube-proxy & kubelet) ─
+# kube-proxy uses xt_conntrack for stateful rules and xt_comment for
+# labeling chains. Without these, iptables fails with:
+#   "Couldn't load match 'conntrack': No such file or directory"
+CONFIG_NETFILTER_XTABLES=y
+CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
+CONFIG_NETFILTER_XT_MATCH_COMMENT=y
+CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
+CONFIG_NETFILTER_XT_MATCH_MARK=y
+CONFIG_NETFILTER_XT_MATCH_STATISTIC=y
+CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
+CONFIG_NETFILTER_XT_MATCH_RECENT=y
+CONFIG_NETFILTER_XT_TARGET_MARK=y
+CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
+CONFIG_NETFILTER_XT_MATCH_CONNMARK=y
+
 # ── NAT (required for service VIP / DNAT / SNAT) ──────────────────────
 CONFIG_NF_NAT=y
 CONFIG_NF_NAT_MASQUERADE_IPV4=y
 
-# ── iptables (kube-proxy iptables mode) ────────────────────────────────
+# ── iptables (CNI bridge masquerade + compat) ──────────────────────────
 CONFIG_IP_NF_IPTABLES=y
 CONFIG_IP_NF_FILTER=y
 CONFIG_IP_NF_NAT=y
@@ -47,7 +63,9 @@ CONFIG_IP_NF_MANGLE=y
 CONFIG_IP_NF_TARGET_MASQUERADE=y
 CONFIG_IP_NF_TARGET_REJECT=y
 
-# ── nftables (kube-proxy nft mode, future-proofing) ────────────────────
+# ── nftables (kube-proxy nftables mode — primary proxy backend) ─────────
+# kube-proxy nftables proxier requires: numgen (random LB), fib (local
+# address detection), counter, ct, nat, masq, reject, limit, redir.
 CONFIG_NF_TABLES=y
 CONFIG_NF_TABLES_INET=y
 CONFIG_NFT_CT=y
@@ -55,7 +73,12 @@ CONFIG_NFT_NAT=y
 CONFIG_NFT_MASQ=y
 CONFIG_NFT_REJECT=y
 CONFIG_NFT_COMPAT=y
-CONFIG_NFT_COUNTER=y
+CONFIG_NFT_NUMGEN=y
+CONFIG_NFT_FIB_IPV4=y
+CONFIG_NFT_FIB_IPV6=y
+CONFIG_NFT_LIMIT=y
+CONFIG_NFT_REDIR=y
+CONFIG_NFT_TPROXY=y
 
 # ── IP forwarding and routing (required for pod-to-pod) ────────────────
 CONFIG_IP_ADVANCED_ROUTER=y
@@ -90,3 +113,7 @@ CONFIG_CGROUP_DEVICE=y
 CONFIG_CGROUP_CPUACCT=y
 CONFIG_CGROUP_PIDS=y
 CONFIG_MEMCG=y
+
+# ── Security features required by the sandbox runtime ───────────────────
+CONFIG_SECURITY_LANDLOCK=y
+CONFIG_SECCOMP_FILTER=y

crates/openshell-vm/scripts/build-rootfs.sh

@@ -65,7 +65,7 @@ docker rm -f "${INIT_CONTAINER_NAME}" 2>/dev/null || true
 
 echo "==> Building base image..."
 docker build --platform linux/arm64 -t "${BASE_IMAGE_TAG}" -f - . <<'DOCKERFILE'
-FROM ubuntu:22.04
+FROM nvcr.io/nvidia/base/ubuntu:noble-20251013
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates \

crates/openshell-vm/scripts/gateway-init.sh

@@ -153,14 +153,16 @@ find /run -name '*.sock' -delete 2>/dev/null || true
 #
 # The kine (SQLite) DB cleanup in build-rootfs.sh already removes stale
 # pod/sandbox records from k3s etcd, preventing kubelet from reconciling
-# against stale sandboxes. Containerd's internal sandbox records in
-# meta.db are harmless because the CRI plugin reconciles with kubelet
-# on startup — any sandboxes unknown to kubelet are cleaned up gracefully
-# without triggering SandboxChanged events.
+# against stale sandboxes.
 CONTAINERD_DIR="/var/lib/rancher/k3s/agent/containerd"
 if [ -d "$CONTAINERD_DIR" ]; then
     # Remove runtime task state (stale shim PIDs, sockets from dead processes).
     rm -rf "${CONTAINERD_DIR}/io.containerd.runtime.v2.task" 2>/dev/null || true
+    # Remove sandbox controller shim state. Stale sandbox records cause
+    # containerd to reuse network namespaces from previous boots, which
+    # already have routes configured. The CNI bridge plugin then fails
+    # with "file exists" when adding the default route on retry.
+    rm -rf "${CONTAINERD_DIR}/io.containerd.sandbox.controller.v1.shim" 2>/dev/null || true
     # Clean stale ingest temp files from the content store.
     rm -rf "${CONTAINERD_DIR}/io.containerd.content.v1.content/ingest" 2>/dev/null || true
     mkdir -p "${CONTAINERD_DIR}/io.containerd.content.v1.content/ingest"
@@ -171,6 +173,83 @@ rm -rf /run/k3s 2>/dev/null || true
 
 ts "stale artifacts cleaned"
 
+# ── Clean stale CNI / pod networking state ──────────────────────────────
+# The rootfs persists across VM restarts via virtio-fs. Previous pod
+# sandboxes leave behind veth pairs, bridge routes, host-local IPAM
+# allocations, and network namespaces. If not cleaned, the bridge CNI
+# plugin fails with:
+#   "failed to add route ... file exists"
+# because the default route via cni0 already exists from the prior boot,
+# or a stale network namespace already has the route configured.
+
+# Tear down the CNI bridge and its associated routes.
+if ip link show cni0 >/dev/null 2>&1; then
+    ip link set cni0 down 2>/dev/null || true
+    ip link delete cni0 2>/dev/null || true
+    ts "deleted stale cni0 bridge"
+fi
+
+# Remove any leftover veth pairs (CNI bridge plugin creates vethXXXX).
+for veth in $(ip -o link show type veth 2>/dev/null | awk -F': ' '{print $2}' | cut -d'@' -f1); do
+    ip link delete "$veth" 2>/dev/null || true
+done
+
+# Flush host-local IPAM allocations so IPs can be reassigned cleanly.
+rm -rf /var/lib/cni/networks 2>/dev/null || true
+rm -rf /var/lib/cni/results 2>/dev/null || true
+
+# Flush any stale CNI-added routes for the pod CIDR. These can conflict
+# with routes the bridge plugin tries to add on the next boot.
+ip route flush 10.42.0.0/24 2>/dev/null || true
+
+# Clean up stale pod network namespaces from previous boots. Containerd
+# creates named netns under /var/run/netns/ for each pod sandbox. If
+# these persist across VM restarts, the CNI bridge plugin fails when
+# adding routes because the stale netns already has the default route
+# configured from the prior boot. Removing all named network namespaces
+# forces containerd to create fresh ones.
+if [ -d /var/run/netns ]; then
+    for ns in $(ip netns list 2>/dev/null | awk '{print $1}'); do
+        ip netns delete "$ns" 2>/dev/null || true
+    done
+fi
+# Also clean the netns bind-mount directory used by containerd/CRI.
+# Containerd may use /run/netns/ or /var/run/netns/ (same via tmpfs).
+rm -rf /run/netns/* 2>/dev/null || true
+rm -rf /var/run/netns/* 2>/dev/null || true
+
+ts "stale CNI networking state cleaned"
+
+# ── Network profile detection ───────────────────────────────────────────
+# Detect early so manifest patching and k3s flags both use the same value.
+#
+# "bridge" is the only supported profile. It requires a custom libkrunfw
+# with CONFIG_BRIDGE, CONFIG_NETFILTER, CONFIG_NF_NAT built in. If the
+# kernel lacks these capabilities the VM cannot run pod networking and we
+# fail fast with an actionable error.
+
+NET_PROFILE="bridge"
+
+ts "network profile: ${NET_PROFILE}"
+
+# Validate that the kernel actually has the required capabilities.
+_caps_ok=true
+if ! ip link add _cap_br0 type bridge 2>/dev/null; then
+    echo "ERROR: kernel lacks bridge support (CONFIG_BRIDGE). Use a custom libkrunfw." >&2
+    _caps_ok=false
+else
+    ip link del _cap_br0 2>/dev/null || true
+fi
+if [ ! -d /proc/sys/net/netfilter ] && [ ! -f /proc/sys/net/bridge/bridge-nf-call-iptables ]; then
+    echo "ERROR: kernel lacks netfilter support (CONFIG_NETFILTER). Use a custom libkrunfw." >&2
+    _caps_ok=false
+fi
+if [ "$_caps_ok" = false ]; then
+    echo "FATAL: required kernel capabilities missing — cannot configure pod networking." >&2
+    echo "See: architecture/custom-vm-runtime.md for build instructions." >&2
+    exit 1
+fi
+
 # ── Deploy bundled manifests (cold boot only) ───────────────────────────
 # On pre-initialized rootfs, manifests are already in place from the
 # build-time k3s boot. Skip this entirely for fast startup.
@@ -211,15 +290,13 @@ if [ -f "$HELMCHART" ]; then
     # Use pre-loaded images — don't pull from registry.
     sed -i 's|pullPolicy: Always|pullPolicy: IfNotPresent|' "$HELMCHART"
 
-    if [ "$NET_PROFILE" = "bridge" ]; then
-        # Bridge CNI: pods use normal pod networking, not hostNetwork.
-        sed -i 's|__HOST_NETWORK__|false|g' "$HELMCHART"
-        sed -i 's|__AUTOMOUNT_SA_TOKEN__|true|g' "$HELMCHART"
-    else
-        # Legacy: VM bootstrap runs without CNI bridge networking.
-        sed -i 's|__HOST_NETWORK__|true|g' "$HELMCHART"
-        sed -i 's|__AUTOMOUNT_SA_TOKEN__|false|g' "$HELMCHART"
-    fi
+    # Bridge CNI: pods use normal pod networking, not hostNetwork.
+    # The pre-init in build-rootfs.sh replaces __HOST_NETWORK__ with "true"
+    # for Docker container networking. At VM boot with bridge CNI we need
+    # to override it back to "false" so pods use the CNI bridge network.
+    sed -i 's|hostNetwork: true|hostNetwork: false|g' "$HELMCHART"
+    sed -i 's|__HOST_NETWORK__|false|g' "$HELMCHART"
+    sed -i 's|__AUTOMOUNT_SA_TOKEN__|true|g' "$HELMCHART"
 
     sed -i 's|__KUBECONFIG_HOST_PATH__|"/etc/rancher/k3s"|g' "$HELMCHART"
     sed -i 's|__PERSISTENCE_ENABLED__|false|g' "$HELMCHART"
@@ -231,115 +308,34 @@ fi
 
 AGENT_MANIFEST="$K3S_MANIFESTS/agent-sandbox.yaml"
 if [ -f "$AGENT_MANIFEST" ]; then
-    if [ "$NET_PROFILE" = "bridge" ]; then
-        # Bridge CNI: agent-sandbox uses normal pod networking.
-        # kube-proxy is enabled so kubernetes.default.svc is reachable
-        # via ClusterIP — no need for KUBERNETES_SERVICE_HOST override.
-        sed -i '/hostNetwork: true/d' "$AGENT_MANIFEST"
-        sed -i '/dnsPolicy: ClusterFirstWithHostNet/d' "$AGENT_MANIFEST"
-        ts "agent-sandbox: using pod networking (bridge profile)"
-    else
-        # Legacy: keep agent-sandbox on pod networking to avoid host port
-        # clashes. Point in-cluster client traffic at the API server node
-        # IP because kube-proxy is disabled in VM mode.
-        sed -i '/hostNetwork: true/d' "$AGENT_MANIFEST"
-        sed -i '/dnsPolicy: ClusterFirstWithHostNet/d' "$AGENT_MANIFEST"
-        if ! grep -q 'metrics-bind-address=:8082' "$AGENT_MANIFEST"; then
-            sed -i 's|image: registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.1.0|image: registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.1.0\
-            args:\
-            - -metrics-bind-address=:8082\
-            env:\
-            - name: KUBERNETES_SERVICE_HOST\
-              value: 192.168.127.2\
-            - name: KUBERNETES_SERVICE_PORT\
-              value: "6443"|g' "$AGENT_MANIFEST"
-        else
-            sed -i 's|value: 127.0.0.1|value: 192.168.127.2|g' "$AGENT_MANIFEST"
-        fi
-        if grep -q 'hostNetwork: true' "$AGENT_MANIFEST" \
-            || grep -q 'ClusterFirstWithHostNet' "$AGENT_MANIFEST" \
-            || ! grep -q 'KUBERNETES_SERVICE_HOST' "$AGENT_MANIFEST" \
-            || ! grep -q 'metrics-bind-address=:8082' "$AGENT_MANIFEST"; then
-            echo "ERROR: failed to patch agent-sandbox manifest for VM networking constraints: $AGENT_MANIFEST" >&2
-            exit 1
-        fi
-        ts "agent-sandbox: patched for legacy-vm-net (API server override)"
-    fi
+    # Bridge CNI: agent-sandbox uses normal pod networking.
+    # kube-proxy is enabled so kubernetes.default.svc is reachable
+    # via ClusterIP — no need for KUBERNETES_SERVICE_HOST override.
+    sed -i '/hostNetwork: true/d' "$AGENT_MANIFEST"
+    sed -i '/dnsPolicy: ClusterFirstWithHostNet/d' "$AGENT_MANIFEST"
+    ts "agent-sandbox: using pod networking (bridge profile)"
 fi
 
-# local-storage implies local-path-provisioner. In legacy mode it
-# requires CNI bridge networking that is unavailable. In bridge mode
-# it can work but we leave it disabled for now until validated.
-if [ "$NET_PROFILE" != "bridge" ]; then
-    rm -f "$K3S_MANIFESTS/local-storage.yaml" 2>/dev/null || true
-fi
-
-# ── CNI configuration ───────────────────────────────────────────────────
-# Two networking profiles:
-#
-# 1. "bridge" (default when kernel supports it):
-#    Uses the bridge CNI plugin with iptables masquerade. Requires
-#    CONFIG_BRIDGE, CONFIG_NETFILTER, CONFIG_NF_NAT in the VM kernel.
-#    This is the standard Kubernetes CNI path — compatible with
-#    kube-proxy, service VIPs, and portmap.
-#
-# 2. "legacy-vm-net" (fallback for stock libkrunfw without netfilter):
-#    Uses ptp CNI without iptables. No masquerade, no portmap.
-#    kube-proxy must be disabled. This is the original VM path.
-#
-# The profile is auto-detected from kernel capabilities but can be
-# forced via OPENSHELL_VM_NET_PROFILE=bridge|legacy-vm-net.
-
-NET_PROFILE="${OPENSHELL_VM_NET_PROFILE:-auto}"
-
-detect_net_profile() {
-    # Check for bridge + netfilter kernel support.
-    # If we can create a bridge and the netfilter sysctl exists, use bridge CNI.
-    local has_bridge=false
-    local has_netfilter=false
-
-    if ip link add _probe_br0 type bridge 2>/dev/null; then
-        ip link del _probe_br0 2>/dev/null || true
-        has_bridge=true
-    fi
-
-    if [ -d /proc/sys/net/netfilter ] || [ -f /proc/sys/net/bridge/bridge-nf-call-iptables ]; then
-        has_netfilter=true
-    fi
-
-    if [ "$has_bridge" = true ] && [ "$has_netfilter" = true ]; then
-        echo "bridge"
-    else
-        echo "legacy-vm-net"
-    fi
-}
-
-if [ "$NET_PROFILE" = "auto" ]; then
-    NET_PROFILE=$(detect_net_profile)
-fi
-
-ts "network profile: ${NET_PROFILE}"
+# ── CNI configuration (bridge) ──────────────────────────────────────────
+# Uses the bridge CNI plugin with iptables masquerade. Requires
+# CONFIG_BRIDGE, CONFIG_NETFILTER, CONFIG_NF_NAT in the VM kernel
+# (validated above at boot). kube-proxy uses nftables mode for service
+# VIP routing.
 
 CNI_CONF_DIR="/etc/cni/net.d"
 CNI_BIN_DIR="/opt/cni/bin"
 mkdir -p "$CNI_CONF_DIR" "$CNI_BIN_DIR"
 
-if [ "$NET_PROFILE" = "bridge" ]; then
-    # ── Bridge CNI (full Kubernetes networking) ─────────────────────
-    # This path requires a custom libkrunfw with bridge + netfilter
-    # kernel support. Creates a cni0 bridge, uses iptables masquerade,
-    # and is compatible with kube-proxy.
+# Enable IP forwarding (required for masquerade).
+echo 1 > /proc/sys/net/ipv4/ip_forward 2>/dev/null || true
 
-    # Enable IP forwarding (required for masquerade).
-    echo 1 > /proc/sys/net/ipv4/ip_forward 2>/dev/null || true
+# Enable bridge netfilter call (required for CNI bridge masquerade to
+# see bridged traffic).
+if [ -f /proc/sys/net/bridge/bridge-nf-call-iptables ]; then
+    echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables 2>/dev/null || true
+fi
 
-    # Enable bridge netfilter call (required for kube-proxy to see
-    # bridged traffic).
-    if [ -f /proc/sys/net/bridge/bridge-nf-call-iptables ]; then
-        echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables 2>/dev/null || true
-    fi
-
-    cat > "$CNI_CONF_DIR/10-bridge.conflist" << 'CNICFG'
+cat > "$CNI_CONF_DIR/10-bridge.conflist" << 'CNICFG'
 {
   "cniVersion": "1.0.0",
   "name": "bridge",
@@ -353,8 +349,7 @@ if [ "$NET_PROFILE" = "bridge" ]; then
       "hairpinMode": true,
       "ipam": {
         "type": "host-local",
-        "ranges": [[{ "subnet": "10.42.0.0/24" }]],
-        "routes": [{ "dst": "0.0.0.0/0" }]
+        "ranges": [[{ "subnet": "10.42.0.0/24" }]]
       }
     },
     {
@@ -369,53 +364,19 @@ if [ "$NET_PROFILE" = "bridge" ]; then
 }
 CNICFG
 
-    # Remove any legacy ptp config.
-    rm -f "$CNI_CONF_DIR/10-ptp.conflist" 2>/dev/null || true
+# Remove any stale legacy ptp config.
+rm -f "$CNI_CONF_DIR/10-ptp.conflist" 2>/dev/null || true
 
-    ts "bridge CNI configured (cni0 + iptables masquerade)"
-else
-    # ── Legacy ptp CNI (iptables-free) ─────────────────────────────
-    # The libkrun VM kernel has no netfilter/iptables support. Flannel's
-    # masquerade rules and kube-proxy both require iptables and crash
-    # without it. We disable both and use a simple ptp CNI with
-    # host-local IPAM instead. This avoids linux bridge requirements.
-    #
-    # ipMasq=false avoids any iptables calls in the plugin.
-    # portmap plugin removed — it requires iptables for DNAT rules.
-
-    cat > "$CNI_CONF_DIR/10-ptp.conflist" << 'CNICFG'
-{
-  "cniVersion": "1.0.0",
-  "name": "ptp",
-  "plugins": [
-    {
-      "type": "ptp",
-      "ipMasq": false,
-      "ipam": {
-        "type": "host-local",
-        "ranges": [[{ "subnet": "10.42.0.0/24" }]],
-        "routes": [{ "dst": "0.0.0.0/0" }]
-      }
-    },
-    {
-      "type": "loopback"
-    }
-  ]
-}
-CNICFG
-
-    # Remove any bridge config.
-    rm -f "$CNI_CONF_DIR/10-bridge.conflist" 2>/dev/null || true
-
-    ts "ptp CNI configured (iptables-free, no linux bridge)"
-fi
+ts "bridge CNI configured (cni0 + iptables masquerade)"
 
 # Symlink k3s-bundled CNI binaries to the default containerd bin path.
 # k3s extracts its tools to /var/lib/rancher/k3s/data/<hash>/bin/.
+# Use -e (not -f) because k3s ships these as symlinks to a `cni` multicall
+# binary.
 K3S_DATA_BIN=$(find /var/lib/rancher/k3s/data -maxdepth 2 -name bin -type d 2>/dev/null | head -1)
 if [ -n "$K3S_DATA_BIN" ]; then
-    for plugin in bridge ptp host-local loopback bandwidth portmap; do
-        [ -f "$K3S_DATA_BIN/$plugin" ] && ln -sf "$K3S_DATA_BIN/$plugin" "$CNI_BIN_DIR/$plugin"
+    for plugin in bridge host-local loopback bandwidth portmap; do
+        [ -e "$K3S_DATA_BIN/$plugin" ] && ln -sf "$K3S_DATA_BIN/$plugin" "$CNI_BIN_DIR/$plugin"
     done
     ts "CNI binaries linked from $K3S_DATA_BIN"
 else
@@ -427,11 +388,15 @@ fi
 rm -f "/var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist" 2>/dev/null || true
 
 # ── Start k3s ──────────────────────────────────────────────────────────
-# Flags tuned for fast single-node startup. The k3s flags vary depending
-# on the network profile:
+# Flags tuned for fast single-node startup. Bridge CNI handles pod
+# networking; kube-proxy runs in nftables mode for service VIP / ClusterIP
+# support.
 #
-# bridge: kube-proxy enabled, flannel disabled (bridge CNI handles it)
-# legacy-vm-net: kube-proxy disabled (no iptables), flannel disabled
+# nftables mode: k3s bundles its own iptables binaries whose MARK target
+# doesn't negotiate xt_MARK revision 2 correctly with the libkrun kernel,
+# causing --xor-mark failures. nftables mode uses the kernel's nf_tables
+# subsystem directly and sidesteps the issue entirely. The kernel is
+# configured with CONFIG_NF_TABLES=y and related modules.
 
 K3S_ARGS=(
     --disable=traefik,servicelb,metrics-server,coredns
@@ -443,19 +408,9 @@ K3S_ARGS=(
     --tls-san=localhost,127.0.0.1,10.0.2.15,192.168.127.2
     --flannel-backend=none
     --snapshotter=native
+    --kube-proxy-arg=proxy-mode=nftables
 )
 
-if [ "$NET_PROFILE" = "bridge" ]; then
-    # With bridge CNI + iptables, kube-proxy can run. Don't disable it.
-    # local-storage can also work with bridge networking.
-    ts "starting k3s server (bridge profile — kube-proxy enabled)"
-else
-    # Legacy: no iptables means no kube-proxy and no local-storage.
-    K3S_ARGS+=(
-        --disable=local-storage
-        --disable-kube-proxy
-    )
-    ts "starting k3s server (legacy-vm-net profile — kube-proxy disabled)"
-fi
+ts "starting k3s server (bridge CNI + nftables kube-proxy)"
 
 exec /usr/local/bin/k3s server "${K3S_ARGS[@]}"

crates/openshell-vm/scripts/verify-vm.sh

@@ -157,24 +157,18 @@ check "kubernetes service has ClusterIP" "networking" \
     "kubectl get svc kubernetes -o jsonpath='{.spec.clusterIP}' | grep -q ."
 
 # Check if bridge CNI is in use (cni0 bridge exists)
-CNI_PROFILE="unknown"
 if kubectl exec -n openshell openshell-0 -- ip link show cni0 >/dev/null 2>&1; then
-    CNI_PROFILE="bridge"
+    echo "  CNI profile detected: bridge"
 else
-    CNI_PROFILE="legacy-vm-net"
+    echo "  WARNING: cni0 bridge not detected — bridge CNI may not be running yet"
 fi
-echo "  CNI profile detected: ${CNI_PROFILE}"
 
-if [ "$CNI_PROFILE" = "bridge" ]; then
-    check "cni0 bridge exists in pod" "networking" \
-        "kubectl exec -n openshell openshell-0 -- ip link show cni0 2>/dev/null"
+check "cni0 bridge exists in pod" "networking" \
+    "kubectl exec -n openshell openshell-0 -- ip link show cni0 2>/dev/null"
 
-    # With bridge CNI, kubernetes.default.svc should be reachable.
-    check "kubernetes.default.svc reachable from pod" "networking" \
-        "kubectl exec -n openshell openshell-0 -- wget -q -O /dev/null --timeout=5 https://kubernetes.default.svc/healthz 2>/dev/null || kubectl exec -n openshell openshell-0 -- curl -sk --connect-timeout 5 https://kubernetes.default.svc/healthz 2>/dev/null"
-else
-    echo "  (skipping bridge-specific checks for legacy-vm-net profile)"
-fi
+# With bridge CNI, kubernetes.default.svc should be reachable.
+check "kubernetes.default.svc reachable from pod" "networking" \
+    "kubectl exec -n openshell openshell-0 -- wget -q -O /dev/null --timeout=5 https://kubernetes.default.svc/healthz 2>/dev/null || kubectl exec -n openshell openshell-0 -- curl -sk --connect-timeout 5 https://kubernetes.default.svc/healthz 2>/dev/null"
 
 check "no bridge creation errors in events" "networking" \
     "! kubectl get events -A 2>/dev/null | grep -qi 'bridge.*fail\\|cni0.*error\\|FailedCreatePodSandBox.*bridge'"
@@ -185,9 +179,6 @@ echo ""
 
 echo "[Host Connectivity]"
 
-check "port 6443 (kube-apiserver) reachable" "host" \
-    "timeout 5 bash -c 'echo > /dev/tcp/127.0.0.1/6443' 2>/dev/null || nc -z -w5 127.0.0.1 6443 2>/dev/null"
-
 check "port 30051 (gateway service) reachable" "host" \
     "timeout 5 bash -c 'echo > /dev/tcp/127.0.0.1/30051' 2>/dev/null || nc -z -w5 127.0.0.1 30051 2>/dev/null"
 

crates/openshell-vm/src/lib.rs

@@ -146,8 +146,7 @@ impl VmConfig {
     ///
     /// Runs `/srv/gateway-init.sh` which mounts essential filesystems,
     /// deploys the OpenShell helm chart, and execs `k3s server`.
-    /// Exposes the Kubernetes API on port 6443 and the OpenShell
-    /// gateway (`NodePort`) on port 30051.
+    /// Exposes the OpenShell gateway on port 30051.
     pub fn gateway(rootfs: PathBuf) -> Self {
         Self {
             rootfs,
@@ -162,10 +161,6 @@ impl VmConfig {
             ],
             workdir: "/".to_string(),
             port_map: vec![
-                // Map host 6443 -> guest 6444 (real kube-apiserver).
-                // The k3s dynamiclistener on 6443 has TLS issues through
-                // port forwarding, so we go directly to the apiserver.
-                "6443:6444".to_string(),
                 // Navigator server — with hostNetwork the server binds
                 // directly to port 8080 on the VM's interface, bypassing
                 // NodePort (which requires kube-proxy / iptables).
@@ -369,6 +364,7 @@ fn raise_nofile_limit() {
 fn log_runtime_provenance(runtime_dir: &Path) {
     if let Some(prov) = ffi::runtime_provenance() {
         eprintln!("runtime: {}", runtime_dir.display());
+        eprintln!("  libkrun: {}", prov.libkrun_path.display());
         for krunfw in &prov.libkrunfw_paths {
             let name = krunfw
                 .file_name()
@@ -993,8 +989,6 @@ pub fn launch(config: &VmConfig) -> Result<i32, VmError> {
 
                     match std::fs::read_to_string(&kubeconfig_src) {
                         Ok(contents) => {
-                            // The kubeconfig has server: https://127.0.0.1:6443
-                            // which is correct since we forward host:6443 -> guest:6444.
                             if let Err(e) = std::fs::write(&dest, &contents) {
                                 eprintln!("  failed to write kubeconfig: {e}");
                             } else {

tasks/scripts/bundle-vm-runtime.sh

@@ -35,10 +35,19 @@ if [ -z "$GVPROXY_BIN" ]; then
   fi
 fi
 
+# libkrun.dylib: prefer the custom runtime dir, fall back to Homebrew.
+# libkrun is the VMM and does not need a custom build; only libkrunfw
+# carries the custom kernel.
 LIBKRUN="${LIB_DIR}/libkrun.dylib"
 if [ ! -e "$LIBKRUN" ]; then
-  echo "libkrun not found at ${LIBKRUN}; set OPENSHELL_VM_RUNTIME_SOURCE_DIR" >&2
-  exit 1
+  BREW_PREFIX="${BREW_PREFIX:-$(brew --prefix 2>/dev/null || true)}"
+  if [ -n "$BREW_PREFIX" ] && [ -e "${BREW_PREFIX}/lib/libkrun.dylib" ]; then
+    LIBKRUN="${BREW_PREFIX}/lib/libkrun.dylib"
+    echo "using Homebrew libkrun at ${LIBKRUN}"
+  else
+    echo "libkrun not found at ${LIB_DIR}/libkrun.dylib or Homebrew; install libkrun or set OPENSHELL_VM_RUNTIME_SOURCE_DIR" >&2
+    exit 1
+  fi
 fi
 
 KRUNFW_FILES=()