Root cause of readyz timeout: gvproxy DHCP assigns guest IPs
nondeterministically (.2 or .3 depending on timing), but port
forwarding was hardcoded to .3. When the guest got .2, the health
check polled the wrong IP for 120s.
Fix: vm-init.sh now assigns 192.168.127.2 statically instead of using
DHCP. The CLI guest_ip constant matches. This eliminates the race.
Also fixes libkrun VMM warnings (virtio-fs passthrough symlink errors)
appearing on the parent terminal — fork_start() now redirects the
child's stderr to the console log file via dup2(). The warnings are
from macOS symlink resolution limits on Kubernetes ConfigMap mounts
and are non-fatal.
vm-init.sh is now always updated in the cached rootfs on every boot
(not just during initial extraction), so networking fixes take effect
without requiring rootfs re-extraction.
After fork_start(), the parent now:
- Polls https://localhost:<kube_port>/readyz with 2s intervals (120s timeout)
- Checks child PID is alive between polls (fast-fail on VM crash)
- Reads kubeconfig from rootfs via virtio-fs (host-visible)
- Rewrites server URL and cluster name, stores in standard location
- Prints kubectl usage instructions when ready
Always forwards kube API port (ephemeral if --kube-port not specified)
to enable health checking. Adds --name flag for kubeconfig context naming.
Boot reliability: vm-init.sh now retries k3s up to 3 times with cleanup
between attempts, handling the transient kine SQLite race condition on
tmpfs that occasionally crashes k3s on first boot.
Exports rewrite_kubeconfig and store_kubeconfig from navigator-bootstrap
for use by the gateway cluster command.
Introduce the navigator-gateway crate with safe Rust wrappers over the
libkrun C FFI, enabling lightweight microVM execution via Apple
Hypervisor.framework (macOS) or KVM (Linux).
Key components:
- navigator-gateway crate: KrunContextBuilder with RAII, ~15 FFI
bindings, support for virtio-fs, console redirect, TSI control,
and gvproxy virtio-net networking
- nav gateway run: ad-hoc microVM execution (direct enter model)
- nav gateway cluster: boots k3s inside a microVM with gvproxy
networking, automatic rootfs extraction from Docker images, and
port forwarding via gvproxy HTTP API
- vm-init.sh: guest bootstrap script with DHCP networking, noop CNI
plugin (kernel lacks bridge module), and tmpfs-backed k3s data dir
- macOS codesigning with com.apple.security.hypervisor entitlement
via auto-signing in scripts/bin/nav
- navigator-cli build.rs for libkrun/libkrunfw rpath resolution
Closes#34
## Summary
- Add `nav ssh-proxy` name mode (`--cluster`/`--name`) and `nav sandbox ssh-config` for VS Code Remote-SSH integration
- Fix VS Code Remote-SSH platform misdetection (was defaulting to Windows/PowerShell)
- Fix sandbox network policy for VS Code server connectivity
- Fix sandbox user shell configuration
## Problem
VS Code Remote-SSH was failing to connect to Navigator sandboxes due to three issues:
1. **Platform misdetection**: VS Code sends `uname -rsv` after a `ready:` marker and expects a response within ~1ms. Forced PTY allocation in `shell_request`, interactive bash (`-i`), and leaked supervisor env vars added enough latency that VS Code timed out and defaulted to "Platform: windows", then tried `powershell`.
2. **Broken network policy**: The `vscode` entry in `dev-sandbox-policy.yaml` was at the YAML top-level instead of nested under `network_policies:` (indentation bug). The VS Code server binary also needed a glob entry.
3. **No login shell**: The sandbox user had `/usr/sbin/nologin` as its shell.
## Changes
| File | Change |
|------|--------|
| `crates/navigator-sandbox/src/ssh.rs` | Remove forced PTY in `shell_request`, remove `-i` flag from pipe exec, add `env_clear()` + explicit `SHELL`/`PATH` to both shell spawn paths |
| `deploy/docker/sandbox/Dockerfile.base` | Change sandbox user shell to `/bin/bash`, set home to `/sandbox`, add `.bashrc`/`.profile` |
| `dev-sandbox-policy.yaml` | Fix `vscode:` indentation under `network_policies:`, add `/sandbox/.vscode-server/*` binary glob |
| `crates/navigator-cli/src/main.rs` | Add `--cluster`/`--name` to `SshProxy`, add `SshConfig`/`Policy`/`Logs` subcommands (rebase merge) |
| `crates/navigator-cli/src/ssh.rs` | Update `print_ssh_config` to use `--cluster` flag |
| `examples/vscode-remote-sandbox.md` | User-facing example and docs |
## How to use
```bash
# Create a persistent sandbox
nav sandbox create --keep my-sandbox
# Generate SSH config
nav sandbox ssh-config my-sandbox >> ~/.ssh/config
# Open VS Code
code --remote ssh-remote+nav-my-sandbox /sandbox
```
## Testing
- All 307 tests pass
- Clippy clean on modified crates
## Summary
- Track last deployed local changes and only redeploy new deltas in auto mode.
- Add per-component fingerprints (server/sandbox/helm) persisted to .cache/cluster-deploy-fast.state.
- Update docs for the new incremental deploy behavior.
```
mise run cluster
....
________________________________________________________
Executed in 75.21 secs fish external
usr time 578.03 millis 0.25 millis 577.78 millis
sys time 364.45 millis 1.22 millis 363.22 millis
```
with a change to `navigator-sandbox` (triggers rust recompile)
```
⟩ time mise run cluster:deploy
[cluster:deploy] $ build/scripts/cluster-deploy-fast.sh
Change detection took 0s
Fast deploy plan:
build server: 0
build sandbox: 1
helm upgrade: 1
...
________________________________________________________
Executed in 25.88 secs fish external
usr time 618.91 millis 0.24 millis 618.67 millis
sys time 557.71 millis 1.54 millis 556.18 millis
```
(most time is spent in docker build)
## Test Plan
- mise run pre-commit
Closes#48, #52
## Summary
- Replace the envoy-gateway-based TLS setup with inline PKI generation during cluster bootstrap, generating CA, server, and client certificates directly in the `navigator-bootstrap` crate
- Remove all envoy gateway Helm templates (`gateway.yaml`, `gatewayclass.yaml`, `grpcroute.yaml`, PKI job, traffic policies) and the `Dockerfile.pki-job`
- Add native mTLS support to the navigator server with `tokio-rustls`, mounting client TLS certs as volumes into sandbox pods
- Update cluster entrypoint, healthcheck, and deploy scripts to work with the new direct-TLS architecture
- Add TLS security e2e test and fix formatting/clippy warnings
## Test Plan
- All unit tests pass (`cargo test --workspace`)
- Clippy clean (`cargo clippy --workspace --all-targets`)
- Format clean (`cargo fmt --all -- --check`)
- Python tests pass (`uv run pytest python/`)
- Full `mise run pre-commit` passes
## Summary
- Fix L7 REST relay response framing so chunked HTTP responses are parsed by chunk-size + trailer termination, instead of waiting for socket close semantics.
- Prevent ~30s stalls observed in `gh auth status` when talking to `api.github.com` through the sandbox proxy.
- Keep detailed debug timing around response framing/chunk completion to make future networking investigations easier.
## Additional changes in this branch
- Adds sandbox policy generation skill docs under `.agent/skills/generate-sandbox-policy/`.
- Includes related sandbox config/image updates in `dev-sandbox-policy.yaml` and `deploy/docker/sandbox/Dockerfile.base`.
## Test plan
- `cargo test -p navigator-sandbox l7::rest::tests:: -- --nocapture`
- `cargo test -p navigator-sandbox -- --nocapture`
- `mise run pre-commit`
- Manual: rebuilt `navigator-sandbox:dev`, ran `gh auth status` in sandbox, and confirmed the previous long proxy delay no longer reproduces.
Closes#67
## Summary
Implements transparent inference interception and routing for sandboxed AI agents. The sandbox proxy intercepts outbound AI SDK calls (OpenAI, Anthropic) and reroutes them through the gateway to policy-controlled backends — enabling organizations to redirect inference traffic to local or self-hosted models without modifying agent code.
**Decision model** — a tri-state OPA evaluation for every CONNECT request:
1. Binary + endpoint explicitly allowed in `network_policies` → **allow** (pass through)
2. Not explicitly allowed + `inference.allowed_routes` configured → **inspect for inference** (TLS intercept, detect API patterns, route through gateway)
3. Otherwise → **deny**
No endpoint declarations or binary lists needed for inference routing. Just configure `inference.allowed_routes`.
## Key Changes
### Sandbox (interception)
- **OPA policy**: New `network_action` Rego rule with three outcomes (`allow`, `inspect_for_inference`, `deny`). New `NetworkAction` enum replaces `PolicyDecision.allowed` bool for the proxy's main decision path.
- **Proxy**: New `InspectForInference` path — TLS-terminates client, parses HTTP, detects inference API patterns (`POST /v1/chat/completions`, `/v1/completions`, `/v1/messages`), strips auth headers, forwards via gRPC.
- **New module**: `l7/inference.rs` — `InferenceApiPattern`, `detect_inference_pattern()`, HTTP request/response parsing.
- **gRPC client**: New `proxy_inference()` for sandbox→gateway forwarding.
- **Sandbox init**: Creates OPA engine when inference is configured, even without `network_policies`.
### Gateway (dispatch)
- **InferenceService**: `ProxyInference` RPC loads sandbox policy, resolves allowed routes, dispatches to router. Full CRUD for inference routes.
- **Proto**: `InferenceRoute`, `InferenceRouteSpec`, `ProxyInferenceRequest/Response`, Inference gRPC service.
### Router (backend proxying)
- **New crate**: `navigator-router` with `Router`, `proxy_with_candidates()`, protocol-based route selection, backend HTTP proxying with auth header rewriting.
- **Mock support** for testing (`mock://` scheme).
### CLI
- `nav inference create/update/delete/list` commands for route management.
### Python SDK
- Updated protobuf bindings. Removed old `inference.py` client (replaced by transparent interception).
### Documentation
- New `architecture/inference-routing.md` — end-to-end system documentation.
- Updated `architecture/sandbox.md` — proxy, OPA, and source index sections.
- Updated `architecture/README.md` — new subsystem overview and diagram.
## Addendum: Chunked Transfer Compatibility
This branch now also fixes intercepted SDK requests that send chunked request bodies:
- `inspect_for_inference` now accepts `Transfer-Encoding: chunked` and decodes chunked request bodies before forwarding to the gateway
- Removed the prior `411 Length Required` response for chunked intercepted requests
- Added request/response header sanitization for framing and hop-by-hop headers (`content-length`, `transfer-encoding`, `connection`, etc.) to keep forwarded requests and returned responses valid
- Added unit tests for chunked parsing and header sanitization
Note: this improves compatibility for streaming-style SDK request patterns; true token-by-token passthrough response streaming is still a separate follow-up.
## Minimal Policy for Inference Routing
```yaml
inference:
allowed_routes:
- local
```
Any outgoing connection from a binary not explicitly allowed in `network_policies` will be intercepted and checked for inference API patterns.
## Test Plan
- [x] `cargo test --workspace` — all tests pass
- [x] `mise run pre-commit` — all checks pass
- [x] E2E: OpenAI chat completions routed through gateway
- [x] E2E: Anthropic messages routed through gateway
- [x] E2E: Python OpenAI SDK from sandbox (`examples/inference/inference.py`)
- [x] E2E test: `e2e/python/test_inference_routing.py`
Closes#44
## Summary
- Add `nav sandbox image push --dockerfile <path>` command to build and push custom container images into the cluster
- Fix sandbox Docker build failure caused by OpenClaw installer TTY access
- Fix sandbox proxy crash when network namespace is unavailable (fall back to loopback)
- Add e2e test for the full custom image build and sandbox creation flow
- Rename and rewrite BYOC architecture doc to reflect current state
## Changes
### `navigator-bootstrap`
- New `build` module (`src/build.rs`) with `build_and_push_image()` public API
- Builds via bollard `Docker::build_image()`, streams output to caller
- `.dockerignore` support with glob matching, 8 unit tests
### `navigator-cli`
- Added `SandboxImageCommands::Push` subcommand with `--dockerfile`, `--tag`, `--context`, `--build-arg` flags
- Added `test:e2e:custom-image` mise task
### `navigator-sandbox`
- Proxy now falls back to `127.0.0.1:3128` when no network namespace or explicit bind address is available, instead of erroring out
### Docker / Infra
- Sandbox Dockerfile: replaced OpenClaw native installer (`curl | bash`) with `npm install -g openclaw` to avoid `/dev/tty` failures in Docker build
### Documentation
- Renamed `architecture/sandbox-byoc.md` to `sandbox-custom-containers.md`
- Rewrote doc to describe current behavior (CLI usage, push pipeline, supervisor behavior, design decisions) instead of listing implementation changes
### Tests
- Added `e2e/bash/test_sandbox_custom_image.sh` covering image push + sandbox create with custom image
## Summary
- add a Docker-based macOS arm64 wheel build path so darwin wheels can be produced from Linux CI hosts
- wire macOS wheel builds into release publish workflows (`publish:main` and `publish:tag`) alongside existing Linux wheel builds
- mirror `crazymax/osxcross:latest` into `$CI_REGISTRY_IMAGE/third_party/osxcross:latest` and use that mirrored image in CI publish jobs
## Scope Notes
- this MR focuses on shipping reliable macOS wheel build support first
- temporary MR-only smoke-job wiring was removed after validation so the branch reflects final intended CI behavior
- broader cache optimization work is tracked separately in issue #43
## Test Plan
- run `mise run pre-commit`
- validate `.gitlab-ci.yml` with `uv run --with pyyaml python -c 'import yaml, pathlib; yaml.safe_load(pathlib.Path(".gitlab-ci.yml").read_text())'`
## Summary
- Add `Provider` entity for managing 3p deps from a sandbox
- Add provider CRUD API/server persistence and new CLI workflows (`nav provider create/get/list/update/delete`), including `--from-existing` laptop discovery.
- Integrate providers into sandbox create flow: infer from command (`-- claude`), support repeatable `--provider <type>`, prompt before auto-create, and allow manual in-sandbox setup.
- Add a dedicated `navigator-providers` crate with per-provider modules and mockable discovery test helpers.
## Key UX Changes
- `nav sandbox create --provider gitlab -- claude`
- Missing provider prompt now asks before creating from local state.
- `nav provider list --names` for scripting/cleanup.
## Test Plan
- `mise run cluster:deploy`
- `mise run test:e2e:sandbox`
- `mise run pre-commit`
Closes#19Closes#22Closes#11
Closes#13
## Summary
- add Python sandbox execution APIs for command and callable workflows
- consolidate sandbox policy fixtures and expand e2e test coverage for policy and Python exec paths
- update CI/build config and images for sandbox e2e execution dependencies
## Test Plan
- mise run pre-commit
## Summary
- Add a "push" path for local development that exports Docker images from the host daemon and imports them directly into the k3s containerd runtime, eliminating the need for a local registry or remote image pulls during `mise run cluster`
- Add `mise run cluster:deploy` task for fast iterative rebuilds (~25s) against a running cluster without recreating it
- Update `cluster-entrypoint.sh` to patch HelmChart manifest with local image tags and `IfNotPresent` pull policy when push mode is active
## Details
Previously, after the local registry (k3d + skaffold) was removed, `mise run cluster` only pulled component images from the remote CloudFront registry. This meant local code changes required a remote publish cycle to test in-cluster.
This MR adds a push path that sits alongside the existing pull path:
- **`push.rs`** — Uses bollard to `export_images()` all 3 component images into a single tar (with shared layer dedup), uploads via Docker `put_archive` API into the cluster container, then runs `ctr images import` to load them into containerd
- **`docker.rs`** — Passes `IMAGE_TAG` and `IMAGE_PULL_POLICY` env vars into the cluster container when `NAVIGATOR_PUSH_IMAGES` is set
- **`lib.rs`** — Wires push into the deploy flow between `clean_stale_nodes()` and `wait_for_cluster_ready()`, guarded by `remote_opts.is_none() && NAVIGATOR_PUSH_IMAGES`
- **`cluster-entrypoint.sh`** — Patches the baked-in HelmChart manifest with sed to use the local image tag and pull policy before k3s starts
- **`cluster.toml`** — Updated `[cluster]` task to tag and push images; new `[cluster:deploy]` task for iterative deploys via `docker save | ctr import` + `helm upgrade`
The remote deploy path (`nav cluster admin deploy --remote`) is completely unaffected — the push logic only activates for local deploys with the env var set.
## Test Plan
- `mise run cluster` — end-to-end success, cluster healthy with local images (~168s)
- `mise run cluster:deploy` — iterative deploy success, helm upgrade + rollout (~43s)
- `mise run pre-commit` — all checks pass (fmt, clippy, 82 tests, python lint)
- Verified images present in containerd: `ctr -a /run/k3s/containerd/containerd.sock images ls | grep navigator/`
- Remote path verified safe via code analysis (guards prevent push logic from running)
The previous approach wrote the host.docker.internal IP to the k3s
resolv.conf, assuming it would serve DNS. On Docker Desktop, this IP
is reachable but doesn't run a DNS server, causing CoreDNS to fail
resolving external names and breaking the PKI job and Helm install.
The new approach discovers Docker's embedded DNS listener ports from
the DOCKER_OUTPUT iptables chain, then sets up DNAT rules to forward
DNS traffic from k3s pods (via PREROUTING) to Docker's actual DNS
resolver on 127.0.0.11. The container's eth0 IP is used as the
nameserver in the k3s resolv.conf since it's routable from all pods.
Drew Newberry