mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-23 06:30:28 +08:00
184 lines
4.5 KiB
Markdown
184 lines
4.5 KiB
Markdown
# Custom libkrunfw Runtime
|
|
|
|
> Status: Experimental and work in progress (WIP). VM support is under active development and may change.
|
|
|
|
This directory contains the kernel config fragment for a custom `libkrunfw` runtime
|
|
that enables bridge CNI and netfilter support in the OpenShell gateway VM.
|
|
|
|
## Why
|
|
|
|
The stock `libkrunfw` (from Homebrew) ships a kernel without bridge, netfilter,
|
|
or conntrack support. This means the VM cannot:
|
|
|
|
- Create `cni0` bridge interfaces (required by the bridge CNI plugin)
|
|
- Run kube-proxy (requires nftables)
|
|
- Route service VIP traffic (requires NAT/conntrack)
|
|
|
|
The custom runtime builds libkrunfw with an additional kernel config fragment
|
|
that enables these networking and sandboxing features.
|
|
|
|
## Directory Structure
|
|
|
|
```text
|
|
runtime/
|
|
kernel/
|
|
openshell.kconfig # Kernel config fragment (networking + sandboxing)
|
|
```
|
|
|
|
## Build Pipeline
|
|
|
|
Each platform builds its own kernel and runtime natively.
|
|
|
|
```text
|
|
Linux ARM64: builds aarch64 kernel -> .so (parallel)
|
|
Linux AMD64: builds x86_64 kernel -> .so (parallel)
|
|
macOS ARM64: builds aarch64 kernel -> .dylib
|
|
```
|
|
|
|
### Build Scripts
|
|
|
|
| Script | Platform | What it does |
|
|
|--------|----------|-------------|
|
|
| `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw + libkrun from source |
|
|
| `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Builds libkrunfw + libkrun from source |
|
|
| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages runtime tarball (libs + gvproxy + provenance) |
|
|
|
|
### Quick Build (Linux)
|
|
|
|
```bash
|
|
# Build both libkrunfw and libkrun from source
|
|
tasks/scripts/vm/build-libkrun.sh
|
|
|
|
# Or build the full runtime from source via mise:
|
|
FROM_SOURCE=1 mise run vm:setup
|
|
```
|
|
|
|
### Quick Build (macOS)
|
|
|
|
```bash
|
|
# Download pre-built runtime (recommended, ~30s):
|
|
mise run vm:setup
|
|
|
|
# Or build from source:
|
|
FROM_SOURCE=1 mise run vm:setup
|
|
```
|
|
|
|
### Output
|
|
|
|
Build artifacts are placed in `target/libkrun-build/`:
|
|
|
|
```text
|
|
target/libkrun-build/
|
|
libkrun.so / libkrun.dylib # The VMM library
|
|
libkrunfw.so* / libkrunfw.dylib # Kernel firmware library
|
|
```
|
|
|
|
## Networking
|
|
|
|
The VM uses bridge CNI for pod networking with nftables-mode kube-proxy for
|
|
service VIP / ClusterIP support. The kernel config fragment enables both
|
|
iptables (for CNI bridge masquerade) and nftables (for kube-proxy).
|
|
|
|
k3s is started with `--kube-proxy-arg=proxy-mode=nftables` because the
|
|
bundled iptables binaries in k3s have revision-negotiation issues with the
|
|
libkrun kernel's xt_MARK module. nftables mode uses the kernel's nf_tables
|
|
subsystem directly and avoids this entirely.
|
|
|
|
## Runtime Provenance
|
|
|
|
At VM boot, the openshell-vm binary logs provenance information about the loaded
|
|
runtime:
|
|
|
|
```text
|
|
runtime: /path/to/openshell-vm.runtime
|
|
libkrunfw: libkrunfw.dylib
|
|
sha256: a1b2c3d4e5f6...
|
|
type: custom (OpenShell-built)
|
|
libkrunfw-commit: abc1234
|
|
kernel-version: 6.6.30
|
|
build-timestamp: 2026-03-23T10:00:00Z
|
|
```
|
|
|
|
For stock runtimes:
|
|
|
|
```text
|
|
runtime: /path/to/openshell-vm.runtime
|
|
libkrunfw: libkrunfw.dylib
|
|
sha256: f6e5d4c3b2a1...
|
|
type: stock (system/homebrew)
|
|
```
|
|
|
|
## Verification
|
|
|
|
### Capability Check (inside VM)
|
|
|
|
```bash
|
|
# Run inside the VM to verify kernel capabilities:
|
|
/srv/check-vm-capabilities.sh
|
|
|
|
# JSON output for CI:
|
|
/srv/check-vm-capabilities.sh --json
|
|
```
|
|
|
|
### Rollback
|
|
|
|
To revert to the stock runtime:
|
|
|
|
```bash
|
|
# Unset the custom runtime source:
|
|
unset OPENSHELL_VM_RUNTIME_SOURCE_DIR
|
|
|
|
# Re-download pre-built runtime and rebuild:
|
|
mise run vm:setup
|
|
mise run vm:build
|
|
|
|
# Boot:
|
|
mise run vm
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### "FailedCreatePodSandBox" bridge errors
|
|
|
|
The kernel does not have bridge support. Verify:
|
|
|
|
```bash
|
|
# Inside VM:
|
|
ip link add test0 type bridge && echo "bridge OK" && ip link del test0
|
|
```
|
|
|
|
If this fails, you are running the stock runtime. Build and use the custom one.
|
|
|
|
### kube-proxy CrashLoopBackOff
|
|
|
|
kube-proxy runs in nftables mode. If it crashes, verify nftables support:
|
|
|
|
```bash
|
|
# Inside VM:
|
|
nft list ruleset
|
|
```
|
|
|
|
If this fails, the kernel may lack `CONFIG_NF_TABLES`. Use the custom runtime.
|
|
|
|
Common errors:
|
|
|
|
- `unknown option "--xor-mark"`: kube-proxy is running in iptables mode instead
|
|
of nftables. Verify `--kube-proxy-arg=proxy-mode=nftables` is in the k3s args.
|
|
|
|
### Runtime mismatch after upgrade
|
|
|
|
If libkrunfw is updated (e.g., via `brew upgrade`), the stock runtime may
|
|
change. Check provenance:
|
|
|
|
```bash
|
|
# Look for provenance info in VM boot output
|
|
grep "runtime:" ~/.local/share/openshell/openshell-vm/console.log
|
|
```
|
|
|
|
Re-build the custom runtime if needed:
|
|
|
|
```bash
|
|
FROM_SOURCE=1 mise run vm:setup
|
|
mise run vm:build
|
|
```
|