Remove all kubectl calls from the host-side boot sequence, eliminating
the need to forward port 6443 (kube-apiserver) outside the VM.
Changes:
- wait_for_gateway_service: TCP probe only (30051), no kubectl pod check
- bootstrap_gateway: cold boot writes TLS secret manifests via virtio-fs
into k3s auto-deploy dir instead of kubectl apply
- bootstrap_gateway: warm boot skips namespace wait (TCP probe suffices)
- recover_stale_pods: removed entirely (gateway-init.sh already cleans
containerd runtime/sandbox state, CNI state, and network namespaces)
- Kubeconfig copy moved to best-effort post-readiness (for debugging)
- Port 6443 removed from gvproxy port_map
Removed functions: recover_stale_pods, wait_for_namespace,
apply_tls_secrets, kubectl_apply.
Net: -362 lines, +147 lines. No kubectl binary required on host.
The previous commit (070bcca1) dropped port 6443 from the gvproxy
port_map, breaking all host-side kubectl commands including the
readiness check and stale pod recovery. k3s runs the API server
with host networking so VM:6443 is directly reachable — restore
the 6443:6443 mapping alongside the 30051:30051 NodePort mapping.
Two issues caused the gateway service readiness check to time out:
1. Port mapping mismatch: gvproxy mapped host:30051 → VM:8080, but with
bridge CNI the pod listens on 8080 inside its network namespace, not
on the VM's root namespace. Changed to 30051:30051 so traffic flows
through the NodePort service (kube-proxy nftables → pod:8080).
2. Pod cycling from helm upgrade: build-rootfs.sh pre-baked with
hostNetwork=true and automountServiceAccountToken=false, but
gateway-init.sh changed these at boot, triggering a HelmChart
reconcile that killed the pre-baked pod ~90s in. Aligned pre-bake
values (hostNetwork=false, automountServiceAccountToken=true) to
match runtime, eliminating the manifest delta.
Switch kube-proxy to nftables mode and add missing kernel config options
(NFT_NUMGEN, NFT_FIB_IPV4/6, NFT_LIMIT, NFT_REDIR, NFT_TPROXY) plus
xtables match modules required by CNI bridge masquerade. Add stale CNI
state cleanup on boot (cni0 bridge, veth pairs, IPAM allocations, pod
network namespaces, sandbox controller shim) to prevent 'route already
exists' errors from persistent rootfs. Remove dual bridge/legacy-vm-net
profile system in favor of bridge-only with fail-fast kernel validation.
Drop host-mapped port 6443 (kube-apiserver) since it is not needed for
normal gateway operation. Update bundle script to fall back to Homebrew
for libkrun.dylib (VMM) while still requiring custom libkrunfw (kernel).
Enable Kubernetes-compatible networking in the gateway microVM by
building a custom libkrunfw kernel with CONFIG_BRIDGE, CONFIG_NETFILTER,
CONFIG_NF_CONNTRACK, CONFIG_IP_NF_IPTABLES, and CONFIG_VETH compiled in.
Key changes:
- Docker-based kernel build pipeline for macOS (build-custom-libkrunfw.sh)
- Kernel config fragment enabling bridge/netfilter/conntrack/NAT/IPVS
- Feature-flagged bridge CNI with auto-detection fallback to legacy ptp
- Runtime provenance tracking (SHA-256, build metadata, manifest validation)
- VM capability checker and host-side verification matrix scripts
- Mise tasks: vm:build-custom-runtime, vm:verify, vm:check-capabilities
- Architecture and operator documentation
openshell-vm links against libkrun which is only available on macOS
with Homebrew. Exclude it from cargo check, clippy, and test workspace
commands so CI passes on Linux runners.
Move orphaned integration test from crates/navigator-vm/ to
crates/openshell-vm/tests/ and update all navigator_bootstrap
references to openshell_bootstrap, including renamed types
(ClusterMetadata -> GatewayMetadata) and functions.
Two #[ignore] tests that require libkrun + pre-built rootfs:
- gateway_boots_and_service_becomes_reachable: starts the full gateway
and verifies the gRPC service on port 30051
- gateway_exec_runs_guest_command: runs /bin/true inside the VM via
--exec and checks the exit code
Move the gateway VM launching out of `nemoclaw gateway` into its own
`gateway` binary built from the navigator-vm crate. The nemoclaw CLI
no longer links against libkrun or requires macOS hypervisor codesigning.
Add scripts/bin/gateway wrapper (build + codesign + exec) and clean up
scripts/bin/nemoclaw to remove navigator-vm artifacts.
Stop deleting meta.db in gateway-init.sh and include the native
snapshotter, content store, and metadata DB in the rootfs built by
build-rootfs.sh. Without meta.db, containerd re-extracts all image
layers on every boot (~2 min for navigator/server on virtio-fs),
causing kubelet CreateContainer timeouts. Also replace the etcd-snapshot
approach with direct SQLite cleanup of the kine DB to remove stale
pod/event/lease records.
Enable full NemoClaw control plane deployment inside the libkrun
microVM so e2e tests can run against the VM instead of Docker.
Build-time (build-rootfs.sh):
- Package helm chart and inject into k3s static charts directory
- Copy HelmChart CR and agent-sandbox manifests into rootfs
- Pull and save arm64 container images as tarballs for airgap boot
Boot-time (gateway-init.sh):
- Enable flannel CNI (remove --flannel-backend=none and related flags)
- Deploy bundled manifests to k3s auto-deploy directory
- Patch HelmChart CR for VM context (pullPolicy, SSH placeholders)
- Ensure DNS fallback when DHCP doesn't configure resolv.conf
Post-boot (lib.rs):
- Wait for navigator namespace created by Helm controller
- Generate PKI and apply TLS secrets via host kubectl
- Store cluster metadata and mTLS creds for CLI/SDK access
- Set 'gateway' as active cluster for e2e test discovery
Also bump VM to 8GB RAM / 4 vCPUs, add port 30051 forwarding,
fix nemoclaw wrapper fingerprint to include navigator-vm crate,
and add test:e2e:vm mise task.
Add a new navigator-vm library crate that boots k3s inside a libkrun
microVM, accessible from the host via gvproxy port forwarding.
Key components:
- FFI bindings to libkrun C API (krun_create_ctx, krun_add_net_unixgram, etc.)
- VmConfig with gateway() preset for k3s and custom exec mode
- gvproxy integration: virtio-net via unixgram, DHCP, native HTTP port forwarding
- gateway-init.sh: PID 1 init script with DHCP via udhcpc, mounts, k3s exec
- build-rootfs.sh: builds Ubuntu 22.04 arm64 rootfs with k3s + busybox-static
- Kubeconfig auto-extraction to ~/.kube/gateway.yaml
- CLI integration as 'ncl gateway' with --exec, --port, --net flags
- macOS codesigning and DYLD_FALLBACK_LIBRARY_PATH in ncl wrapper
Docker Desktop 29.x defaults to private cgroupns which prevents k3s
kubelet from accessing cgroup v2 controllers (cpu, cpuset, memory,
pids, hugetlb). This causes ContainerManager to fail during startup.
Explicitly set cgroupns_mode to host, which is backwards compatible
with all Docker versions and matches what k3s-in-Docker tooling
(k3d) requires.
* fix(sandbox): prevent overread request smuggling in L7 REST parser
The HTTP/1.1 parser used a 1024-byte read buffer that could capture
bytes from a pipelined second request. Those overflow bytes were
forwarded upstream as body overflow without L7 policy evaluation,
enabling request smuggling that bypasses per-request method/path
enforcement.
Replace multi-byte read with byte-at-a-time read_u8 that stops exactly
at the CRLFCRLF header terminator. Add regression test proving two
pipelined requests are parsed independently.
Refs: #350
* fix(server): harden sandbox TLS secret volume permissions to 0400
Kubernetes secret volumes default to 0644, allowing the unprivileged
sandbox user to read the mTLS client private key via the Landlock
baseline /etc read set. A compromised sandbox could use the key to
impersonate the control-plane client.
Set defaultMode to 256 (octal 0400, owner-read only) on both the
default and custom pod template paths. The supervisor reads TLS
materials as root before forking, so this does not affect normal
operation.
Refs: #350
* fix(sandbox): stop using cmdline paths for binary policy matching
/proc/pid/cmdline is fully attacker-controlled (argv[0] can be set to
any string via execve) and had no integrity verification, unlike
exec.path and ancestors which are kernel-managed and get TOFU/SHA256
checks. The Rego policy used cmdline_paths as a grant-access signal,
allowing any sandboxed process to claim the identity of an allowed
binary and bypass network restrictions.
Remove the cmdline exact-match rule and exclude cmdline_paths from the
glob-match rule. Only exec.path and exec.ancestors (from /proc/pid/exe)
are now used for binary identity. cmdline_paths remain in the OPA input
for deny-reason diagnostics only.
Refs: #350
* fix(sandbox): reject symlink and non-dir read_write paths before chown
The supervisor runs as root and calls chown on each read_write path.
Since chown follows symlinks, a malicious container image could place a
symlink (e.g. /sandbox -> /etc/shadow) to trick the supervisor into
transferring ownership of arbitrary files to the sandbox user.
Add symlink_metadata (lstat) check before chown to reject symlinks and
non-directory entries. The TOCTOU window is not exploitable because no
untrusted child process has been forked yet at this point.
Refs: #350
* fix(server): redact provider credentials in gRPC CRUD responses
Provider CRUD RPCs (create, get, list, update) returned full Provider
objects including plaintext credentials (API keys, secrets). Any
authenticated client -- including sandbox workloads running untrusted
code -- could read credentials for all providers.
Add redact_provider_credentials helper that clears the credentials map
before returning. Internal server paths (inference routing, sandbox env
injection) read from the store directly and are unaffected. Update
tests to verify redaction and assert persistence via direct store reads.
Refs: #350
* fix(sandbox): enforce non-root fallback when process user unset
drop_privileges silently returned Ok(()) when both run_as_user and
run_as_group were None, even when running as root. In local/dev mode
policies are loaded from disk without passing through the server-side
ensure_sandbox_process_identity normalization, so child processes could
retain root and all capabilities (SYS_ADMIN, NET_ADMIN).
When running as root with no process identity configured, fall back to
sandbox:sandbox instead of no-oping. Non-root runtimes are unaffected.
Refs: #350
* fix(sandbox): deny forward proxy for L7-configured endpoints
The forward proxy path only performed L4 (endpoint) and allowed_ips
checks. If an endpoint had L7 rules (method/path restrictions), a
sandboxed process could bypass them by using HTTP_PROXY with plain
http:// requests instead of CONNECT tunneling, since L7 inspection
only runs in the CONNECT path.
Add a guard in handle_forward_proxy that queries the endpoint's L7
config and returns 403 if any L7 rules are present, forcing traffic
through the CONNECT path where per-request inspection happens.
Refs: #350
* test(e2e): add regression test for forward proxy L7 bypass
Verifies that the forward proxy path (plain http:// via HTTP_PROXY)
returns 403 for endpoints with L7 rules configured, preventing
sandboxed processes from bypassing per-request method/path enforcement
by avoiding the CONNECT tunnel.
Refs: #350
* fix: update tests for cmdline path and TLS volume changes
Update OPA tests to verify cmdline_paths no longer grant access
(regression tests for the security fix). Fix formatting in TLS
volume test.
Refs: #350
* test(e2e): update provider e2e tests for credential redaction
Provider CRUD gRPC responses no longer include credential values.
Update three e2e tests to assert credentials are empty in responses.
Provider functionality is verified by existing e2e tests that check
env var injection into sandboxes (which read from the store directly).
Refs: #350
* chore: reformat for Rust 1.94 assert! macro style
* fix(sandbox): make drop_privileges tests root-aware for CI
CI runs as root but has no 'sandbox' user. The security fix correctly
errors when running as root with no process identity and no fallback
user available -- this is the intended behavior (refuse to run as root).
Update tests to expect that error in root-without-sandbox-user
environments instead of unconditionally asserting Ok.
Refs: #350
* fix(sandbox): allow non-directory read_write entries like /dev/null
The symlink guard incorrectly rejected all non-directory entries.
Character devices like /dev/null are legitimate read_write paths
used in sandbox policies. Only reject symlinks, which are the
actual attack vector for the chown privilege escalation.
Refs: #350
* feat(sandbox): log connection attempts that bypass proxy path
Add iptables LOG + REJECT rules inside the sandbox network namespace to
detect and diagnose direct connection attempts that bypass the HTTP
CONNECT proxy. This provides two improvements:
1. Fast-fail UX: applications get immediate ECONNREFUSED instead of a
30-second timeout when they bypass the proxy
2. Diagnostics: a /dev/kmsg monitor emits structured BYPASS_DETECT
tracing events with destination, protocol, process identity, and
actionable hints
Both TCP and UDP bypass attempts are covered (UDP catches DNS bypass).
The feature degrades gracefully if iptables or /dev/kmsg are unavailable.
Closes#268
* chore: track .python-version to pin Python 3.13.12 for uv
The sandbox base image runs Python 3.13. A stale venv on 3.12 causes
all exec_python E2E tests to fail because cloudpickle bytecode is not
compatible across minor versions.
* fix(cluster): preserve hostGatewayIP across fast deploys
The fast deploy's helm upgrade was missing the hostGatewayIP value that
the bootstrap entrypoint injects into the HelmChart CR. This caused
host.openshell.internal hostAliases to be lost from the gateway pod
and sandbox pods after any fast deploy, breaking host gateway routing.
Read the IP from the HelmChart CR and pass it through to helm upgrade.
* wip: fix iptables path resolution, use dmesg for kmsg, add CAP_SYSLOG
* fix(sandbox): restore NetworkNamespace Drop impl, remove dead kmsg code
The Drop impl for NetworkNamespace was accidentally deleted during the
bypass detection refactor, which would cause network namespaces and
veth interfaces to leak on every sandbox shutdown.
Also removes dead kmsg volume/mount code (bypass monitor uses dmesg
instead of direct /dev/kmsg access) and removes an accidentally
committed session transcript file.
Closes#273
Verify inference endpoints synchronously on the server during set/update, expose a --no-verify escape hatch in the CLI and Python helper, and return actionable failures when validation does not pass.
The strip_headers list in send_backend_request included default_headers
names (e.g. anthropic-version), which removed the client's value. The
already_sent check then saw the original input still contained the header
and skipped injecting the route default. Neither value reached upstream.
Remove default_headers names from strip_headers so client values pass
through. The existing already_sent guard still injects defaults only when
the client omits them.
* feat(sandbox): inject host gateway hostAliases into sandbox pods
Sandbox pods running in the k3s cluster cannot resolve host.docker.internal
by default, preventing them from reaching services on the Docker host.
Detect the host gateway IP (default route) in the cluster entrypoint,
thread it through the Helm chart to the gateway server, and inject
hostAliases entries (host.docker.internal, host.openshell.internal)
into every sandbox pod spec. The injection is conditional -- when the
IP is empty (non-Docker deployments), no hostAliases are added.
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
Drew Newberry