Two issues caused the gateway service readiness check to time out:
1. Port mapping mismatch: gvproxy mapped host:30051 → VM:8080, but with
bridge CNI the pod listens on 8080 inside its network namespace, not
on the VM's root namespace. Changed to 30051:30051 so traffic flows
through the NodePort service (kube-proxy nftables → pod:8080).
2. Pod cycling from helm upgrade: build-rootfs.sh pre-baked with
hostNetwork=true and automountServiceAccountToken=false, but
gateway-init.sh changed these at boot, triggering a HelmChart
reconcile that killed the pre-baked pod ~90s in. Aligned pre-bake
values (hostNetwork=false, automountServiceAccountToken=true) to
match runtime, eliminating the manifest delta.
Switch kube-proxy to nftables mode and add missing kernel config options
(NFT_NUMGEN, NFT_FIB_IPV4/6, NFT_LIMIT, NFT_REDIR, NFT_TPROXY) plus
xtables match modules required by CNI bridge masquerade. Add stale CNI
state cleanup on boot (cni0 bridge, veth pairs, IPAM allocations, pod
network namespaces, sandbox controller shim) to prevent 'route already
exists' errors from persistent rootfs. Remove dual bridge/legacy-vm-net
profile system in favor of bridge-only with fail-fast kernel validation.
Drop host-mapped port 6443 (kube-apiserver) since it is not needed for
normal gateway operation. Update bundle script to fall back to Homebrew
for libkrun.dylib (VMM) while still requiring custom libkrunfw (kernel).
Enable Kubernetes-compatible networking in the gateway microVM by
building a custom libkrunfw kernel with CONFIG_BRIDGE, CONFIG_NETFILTER,
CONFIG_NF_CONNTRACK, CONFIG_IP_NF_IPTABLES, and CONFIG_VETH compiled in.
Key changes:
- Docker-based kernel build pipeline for macOS (build-custom-libkrunfw.sh)
- Kernel config fragment enabling bridge/netfilter/conntrack/NAT/IPVS
- Feature-flagged bridge CNI with auto-detection fallback to legacy ptp
- Runtime provenance tracking (SHA-256, build metadata, manifest validation)
- VM capability checker and host-side verification matrix scripts
- Mise tasks: vm:build-custom-runtime, vm:verify, vm:check-capabilities
- Architecture and operator documentation
Stop deleting meta.db in gateway-init.sh and include the native
snapshotter, content store, and metadata DB in the rootfs built by
build-rootfs.sh. Without meta.db, containerd re-extracts all image
layers on every boot (~2 min for navigator/server on virtio-fs),
causing kubelet CreateContainer timeouts. Also replace the etcd-snapshot
approach with direct SQLite cleanup of the kine DB to remove stale
pod/event/lease records.
Enable full NemoClaw control plane deployment inside the libkrun
microVM so e2e tests can run against the VM instead of Docker.
Build-time (build-rootfs.sh):
- Package helm chart and inject into k3s static charts directory
- Copy HelmChart CR and agent-sandbox manifests into rootfs
- Pull and save arm64 container images as tarballs for airgap boot
Boot-time (gateway-init.sh):
- Enable flannel CNI (remove --flannel-backend=none and related flags)
- Deploy bundled manifests to k3s auto-deploy directory
- Patch HelmChart CR for VM context (pullPolicy, SSH placeholders)
- Ensure DNS fallback when DHCP doesn't configure resolv.conf
Post-boot (lib.rs):
- Wait for navigator namespace created by Helm controller
- Generate PKI and apply TLS secrets via host kubectl
- Store cluster metadata and mTLS creds for CLI/SDK access
- Set 'gateway' as active cluster for e2e test discovery
Also bump VM to 8GB RAM / 4 vCPUs, add port 30051 forwarding,
fix nemoclaw wrapper fingerprint to include navigator-vm crate,
and add test:e2e:vm mise task.
Add a new navigator-vm library crate that boots k3s inside a libkrun
microVM, accessible from the host via gvproxy port forwarding.
Key components:
- FFI bindings to libkrun C API (krun_create_ctx, krun_add_net_unixgram, etc.)
- VmConfig with gateway() preset for k3s and custom exec mode
- gvproxy integration: virtio-net via unixgram, DHCP, native HTTP port forwarding
- gateway-init.sh: PID 1 init script with DHCP via udhcpc, mounts, k3s exec
- build-rootfs.sh: builds Ubuntu 22.04 arm64 rootfs with k3s + busybox-static
- Kubeconfig auto-extraction to ~/.kube/gateway.yaml
- CLI integration as 'ncl gateway' with --exec, --port, --net flags
- macOS codesigning and DYLD_FALLBACK_LIBRARY_PATH in ncl wrapper
Drew Newberry