Drew Newberry
070bcca19c
Switch kube-proxy to nftables mode and add missing kernel config options (NFT_NUMGEN, NFT_FIB_IPV4/6, NFT_LIMIT, NFT_REDIR, NFT_TPROXY) plus xtables match modules required by CNI bridge masquerade. Add stale CNI state cleanup on boot (cni0 bridge, veth pairs, IPAM allocations, pod network namespaces, sandbox controller shim) to prevent 'route already exists' errors from persistent rootfs. Remove dual bridge/legacy-vm-net profile system in favor of bridge-only with fail-fast kernel validation. Drop host-mapped port 6443 (kube-apiserver) since it is not needed for normal gateway operation. Update bundle script to fall back to Homebrew for libkrun.dylib (VMM) while still requiring custom libkrunfw (kernel).
Custom libkrunfw Runtime
This directory contains the build infrastructure for a custom libkrunfw runtime
that enables bridge CNI and netfilter support in the OpenShell gateway VM.
Why
The stock libkrunfw (from Homebrew) ships a kernel without bridge, netfilter,
or conntrack support. This means the VM cannot:
- Create
cni0bridge interfaces (required by the bridge CNI plugin) - Run kube-proxy (requires nftables)
- Route service VIP traffic (requires NAT/conntrack)
The custom runtime builds libkrunfw with an additional kernel config fragment that enables these networking and sandboxing features.
Directory Structure
runtime/
build-custom-libkrunfw.sh # Build script for custom libkrunfw
kernel/
bridge-cni.config # Kernel config fragment (networking + sandboxing)
Building
Prerequisites
- Rust toolchain
- make, git, curl
- On macOS: Xcode command line tools and cross-compilation tools for aarch64
Quick Build
# Build custom libkrunfw (clones libkrunfw repo, applies config, builds)
./crates/openshell-vm/runtime/build-custom-libkrunfw.sh
# Or via mise task:
mise run vm:build-custom-runtime
Output
Build artifacts are placed in target/custom-runtime/:
target/custom-runtime/
libkrunfw.dylib # The custom library
libkrunfw.<version>.dylib # Version-suffixed copy
provenance.json # Build metadata (commit, hash, timestamp)
bridge-cni.config # The config fragment used
kernel.config # Full kernel .config (for debugging)
Using the Custom Runtime
# Point the bundle script at the custom build:
export OPENSHELL_VM_RUNTIME_SOURCE_DIR=target/custom-runtime
mise run vm:bundle-runtime
# Then boot the VM as usual:
mise run vm
Networking
The VM uses bridge CNI for pod networking with nftables-mode kube-proxy for service VIP / ClusterIP support. The kernel config fragment enables both iptables (for CNI bridge masquerade) and nftables (for kube-proxy).
k3s is started with --kube-proxy-arg=proxy-mode=nftables because the
bundled iptables binaries in k3s have revision-negotiation issues with the
libkrun kernel's xt_MARK module. nftables mode uses the kernel's nf_tables
subsystem directly and avoids this entirely.
Runtime Provenance
At VM boot, the gateway binary logs provenance information about the loaded runtime:
runtime: /path/to/gateway.runtime
libkrunfw: libkrunfw.dylib
sha256: a1b2c3d4e5f6...
type: custom (OpenShell-built)
libkrunfw-commit: abc1234
kernel-version: 6.6.30
build-timestamp: 2026-03-23T10:00:00Z
For stock runtimes:
runtime: /path/to/gateway.runtime
libkrunfw: libkrunfw.dylib
sha256: f6e5d4c3b2a1...
type: stock (system/homebrew)
Verification
Capability Check (inside VM)
# Run inside the VM to verify kernel capabilities:
/srv/check-vm-capabilities.sh
# JSON output for CI:
/srv/check-vm-capabilities.sh --json
Full Verification Matrix
# Run from the host with a running VM:
./crates/openshell-vm/scripts/verify-vm.sh
# Or via mise task:
mise run vm:verify
Rollback
To revert to the stock runtime:
# Unset the custom runtime source:
unset OPENSHELL_VM_RUNTIME_SOURCE_DIR
# Re-bundle with stock libraries:
mise run vm:bundle-runtime
# Boot — will auto-detect legacy-vm-net profile:
mise run vm
Troubleshooting
"FailedCreatePodSandBox" bridge errors
The kernel does not have bridge support. Verify:
# Inside VM:
ip link add test0 type bridge && echo "bridge OK" && ip link del test0
If this fails, you are running the stock runtime. Build and use the custom one.
kube-proxy CrashLoopBackOff
kube-proxy runs in nftables mode. If it crashes, verify nftables support:
# Inside VM:
nft list ruleset
If this fails, the kernel may lack CONFIG_NF_TABLES. Use the custom runtime.
Common errors:
unknown option "--xor-mark": kube-proxy is running in iptables mode instead of nftables. Verify--kube-proxy-arg=proxy-mode=nftablesis in the k3s args.
Runtime mismatch after upgrade
If libkrunfw is updated (e.g., via brew upgrade), the stock runtime may
change. Check provenance:
# Look for provenance info in VM boot output
grep "runtime:" ~/.local/share/openshell/gateway/console.log
Re-build the custom runtime if needed:
mise run vm:build-custom-runtime
mise run vm:bundle-runtime