cluster-deploy-fast.sh ran helm and kubectl directly on the host,
which requires a local kubeconfig pointing at the k3s API server.
Since the cluster container does not expose port 6443, these commands
always fail with 'connection refused'.
Install helm in the cluster container image and run all helm/kubectl
commands via docker exec with KUBECONFIG pre-configured. The local
chart directory is copied into the container before helm upgrade.
The k9s COPY instruction was added without a corresponding build stage,
causing Docker to look for a nonexistent docker.io/library/k9s image.
Add a multi-arch stage that downloads the k9s binary from GitHub releases.
Add --gpu flag to 'openshell gateway start' that enables NVIDIA GPU
access inside the cluster container. When enabled, the Docker container
receives GPU device requests via bollard, k3s auto-detects the NVIDIA
runtime, and the NVIDIA k8s-device-plugin is deployed via HelmChart CR
so Kubernetes workloads can request nvidia.com/gpu resources.
- Add GPU device request to Docker container creation (bollard DeviceRequest)
- Set GPU_ENABLED=true env var to gate GPU manifest deployment
- Multi-stage Dockerfile build to bundle NVIDIA Container Toolkit binaries
- Conditional GPU manifest copy in cluster entrypoint
- HelmChart CR for nvidia-device-plugin v0.18.2 with GFD and NFD enabled
- Thread gpu bool through DeployOptions, CLI flag, and run dispatch
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
* refactor(docker): rename server image to gateway
Rename Dockerfile.server to Dockerfile.gateway and update all image
references from openshell/server to openshell/gateway across Helm
charts, Kubernetes manifests, mise tasks, build/deploy scripts, CI
workflows, and documentation.
The underlying Rust binary (navigator-server) is unchanged -- this
rename only affects the Docker image name and Dockerfile.
* fix: catch remaining server->gateway references in docs and comments
Migrate all production container runtime stages from Debian/Ubuntu
upstream to NVIDIA's hardened Ubuntu base image for supply chain
consistency.
- Dockerfile.ci: ubuntu:24.04 -> nvidia base
- Dockerfile.server: debian:bookworm-slim -> nvidia base (runtime stage)
- Dockerfile.base (sandbox): python:3.12-slim-bookworm -> nvidia base
with system Python 3.12 from Ubuntu Noble repos
- Dockerfile.cluster: convert from single-stage rancher/k3s to
multistage build extracting k3s artifacts onto nvidia base
Closes#244
npm 11 does not support `npm dedupe -g` in global mode, causing the
sandbox container build to fail with error code EDEDUPEGLOBAL.
The dedupe was intended to remove nested copies of vulnerable
dependencies, but npm 11 already handles global package installation
differently. The force-upgrade of tar and @hono/node-server already
ensures the patched versions are installed.
Pass the computed cargo version through Docker and cluster packaging paths so deployed clusters report the built artifact version while keeping latest and explicit version tags aligned.
The entrypoint script uses 'openssl rand -hex 32' to generate the SSH
handshake secret (added in e9f1071), but the rancher/k3s base image
does not include openssl. This causes the container to exit with code
127 immediately on startup.
Fixes#136
* fix(security): harden sandbox SSH with mandatory HMAC secret, NetworkPolicy, and nonce replay detection
Closes#25
- Make NEMOCLAW_SSH_HANDSHAKE_SECRET mandatory: server and sandbox both
refuse to start if the secret is empty/unset. Cluster deployments
auto-generate it via openssl rand in the entrypoint script.
- Add Kubernetes NetworkPolicy restricting sandbox port 2222 ingress to
the gateway pod only, preventing lateral movement from other cluster
workloads.
- Add NSSH1 nonce replay detection with a TTL-bounded cache, rejecting
replayed handshakes within the timestamp validity window.
- Add unit tests for verify_preface (valid, replay, expired, bad HMAC,
malformed) and env injection.
* fix(deploy): pass sshHandshakeSecret in fast deploy helm upgrade
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
The multi-arch publish job fails with 'extern location does not exist'
errors for version_check when BuildKit cache mounts contain stale
artifacts. The existing retry logic (cargo clean + retry) was
ineffective because:
1. cargo clean relies on its own metadata to decide what to remove,
which can itself be stale on BuildKit cache mounts shared across
different build profiles (debug vs release).
2. sccache remained enabled during the retry, so a corrupt memcached
cache entry would reproduce the same failure even after cleaning.
Replace cargo clean with rm -rf /build/target/* for a thorough wipe,
and unset RUSTC_WRAPPER on retry so cargo falls back to raw rustc.
Closes#34
## Summary
- Add `nav ssh-proxy` name mode (`--cluster`/`--name`) and `nav sandbox ssh-config` for VS Code Remote-SSH integration
- Fix VS Code Remote-SSH platform misdetection (was defaulting to Windows/PowerShell)
- Fix sandbox network policy for VS Code server connectivity
- Fix sandbox user shell configuration
## Problem
VS Code Remote-SSH was failing to connect to Navigator sandboxes due to three issues:
1. **Platform misdetection**: VS Code sends `uname -rsv` after a `ready:` marker and expects a response within ~1ms. Forced PTY allocation in `shell_request`, interactive bash (`-i`), and leaked supervisor env vars added enough latency that VS Code timed out and defaulted to "Platform: windows", then tried `powershell`.
2. **Broken network policy**: The `vscode` entry in `dev-sandbox-policy.yaml` was at the YAML top-level instead of nested under `network_policies:` (indentation bug). The VS Code server binary also needed a glob entry.
3. **No login shell**: The sandbox user had `/usr/sbin/nologin` as its shell.
## Changes
| File | Change |
|------|--------|
| `crates/navigator-sandbox/src/ssh.rs` | Remove forced PTY in `shell_request`, remove `-i` flag from pipe exec, add `env_clear()` + explicit `SHELL`/`PATH` to both shell spawn paths |
| `deploy/docker/sandbox/Dockerfile.base` | Change sandbox user shell to `/bin/bash`, set home to `/sandbox`, add `.bashrc`/`.profile` |
| `dev-sandbox-policy.yaml` | Fix `vscode:` indentation under `network_policies:`, add `/sandbox/.vscode-server/*` binary glob |
| `crates/navigator-cli/src/main.rs` | Add `--cluster`/`--name` to `SshProxy`, add `SshConfig`/`Policy`/`Logs` subcommands (rebase merge) |
| `crates/navigator-cli/src/ssh.rs` | Update `print_ssh_config` to use `--cluster` flag |
| `examples/vscode-remote-sandbox.md` | User-facing example and docs |
## How to use
```bash
# Create a persistent sandbox
nav sandbox create --keep my-sandbox
# Generate SSH config
nav sandbox ssh-config my-sandbox >> ~/.ssh/config
# Open VS Code
code --remote ssh-remote+nav-my-sandbox /sandbox
```
## Testing
- All 307 tests pass
- Clippy clean on modified crates
## Summary
- Track last deployed local changes and only redeploy new deltas in auto mode.
- Add per-component fingerprints (server/sandbox/helm) persisted to .cache/cluster-deploy-fast.state.
- Update docs for the new incremental deploy behavior.
```
mise run cluster
....
________________________________________________________
Executed in 75.21 secs fish external
usr time 578.03 millis 0.25 millis 577.78 millis
sys time 364.45 millis 1.22 millis 363.22 millis
```
with a change to `navigator-sandbox` (triggers rust recompile)
```
⟩ time mise run cluster:deploy
[cluster:deploy] $ build/scripts/cluster-deploy-fast.sh
Change detection took 0s
Fast deploy plan:
build server: 0
build sandbox: 1
helm upgrade: 1
...
________________________________________________________
Executed in 25.88 secs fish external
usr time 618.91 millis 0.24 millis 618.67 millis
sys time 557.71 millis 1.54 millis 556.18 millis
```
(most time is spent in docker build)
## Test Plan
- mise run pre-commit
Closes#48, #52
## Summary
- Replace the envoy-gateway-based TLS setup with inline PKI generation during cluster bootstrap, generating CA, server, and client certificates directly in the `navigator-bootstrap` crate
- Remove all envoy gateway Helm templates (`gateway.yaml`, `gatewayclass.yaml`, `grpcroute.yaml`, PKI job, traffic policies) and the `Dockerfile.pki-job`
- Add native mTLS support to the navigator server with `tokio-rustls`, mounting client TLS certs as volumes into sandbox pods
- Update cluster entrypoint, healthcheck, and deploy scripts to work with the new direct-TLS architecture
- Add TLS security e2e test and fix formatting/clippy warnings
## Test Plan
- All unit tests pass (`cargo test --workspace`)
- Clippy clean (`cargo clippy --workspace --all-targets`)
- Format clean (`cargo fmt --all -- --check`)
- Python tests pass (`uv run pytest python/`)
- Full `mise run pre-commit` passes
## Summary
- Fix L7 REST relay response framing so chunked HTTP responses are parsed by chunk-size + trailer termination, instead of waiting for socket close semantics.
- Prevent ~30s stalls observed in `gh auth status` when talking to `api.github.com` through the sandbox proxy.
- Keep detailed debug timing around response framing/chunk completion to make future networking investigations easier.
## Additional changes in this branch
- Adds sandbox policy generation skill docs under `.agent/skills/generate-sandbox-policy/`.
- Includes related sandbox config/image updates in `dev-sandbox-policy.yaml` and `deploy/docker/sandbox/Dockerfile.base`.
## Test plan
- `cargo test -p navigator-sandbox l7::rest::tests:: -- --nocapture`
- `cargo test -p navigator-sandbox -- --nocapture`
- `mise run pre-commit`
- Manual: rebuilt `navigator-sandbox:dev`, ran `gh auth status` in sandbox, and confirmed the previous long proxy delay no longer reproduces.
Closes#67
## Summary
Implements transparent inference interception and routing for sandboxed AI agents. The sandbox proxy intercepts outbound AI SDK calls (OpenAI, Anthropic) and reroutes them through the gateway to policy-controlled backends — enabling organizations to redirect inference traffic to local or self-hosted models without modifying agent code.
**Decision model** — a tri-state OPA evaluation for every CONNECT request:
1. Binary + endpoint explicitly allowed in `network_policies` → **allow** (pass through)
2. Not explicitly allowed + `inference.allowed_routes` configured → **inspect for inference** (TLS intercept, detect API patterns, route through gateway)
3. Otherwise → **deny**
No endpoint declarations or binary lists needed for inference routing. Just configure `inference.allowed_routes`.
## Key Changes
### Sandbox (interception)
- **OPA policy**: New `network_action` Rego rule with three outcomes (`allow`, `inspect_for_inference`, `deny`). New `NetworkAction` enum replaces `PolicyDecision.allowed` bool for the proxy's main decision path.
- **Proxy**: New `InspectForInference` path — TLS-terminates client, parses HTTP, detects inference API patterns (`POST /v1/chat/completions`, `/v1/completions`, `/v1/messages`), strips auth headers, forwards via gRPC.
- **New module**: `l7/inference.rs` — `InferenceApiPattern`, `detect_inference_pattern()`, HTTP request/response parsing.
- **gRPC client**: New `proxy_inference()` for sandbox→gateway forwarding.
- **Sandbox init**: Creates OPA engine when inference is configured, even without `network_policies`.
### Gateway (dispatch)
- **InferenceService**: `ProxyInference` RPC loads sandbox policy, resolves allowed routes, dispatches to router. Full CRUD for inference routes.
- **Proto**: `InferenceRoute`, `InferenceRouteSpec`, `ProxyInferenceRequest/Response`, Inference gRPC service.
### Router (backend proxying)
- **New crate**: `navigator-router` with `Router`, `proxy_with_candidates()`, protocol-based route selection, backend HTTP proxying with auth header rewriting.
- **Mock support** for testing (`mock://` scheme).
### CLI
- `nav inference create/update/delete/list` commands for route management.
### Python SDK
- Updated protobuf bindings. Removed old `inference.py` client (replaced by transparent interception).
### Documentation
- New `architecture/inference-routing.md` — end-to-end system documentation.
- Updated `architecture/sandbox.md` — proxy, OPA, and source index sections.
- Updated `architecture/README.md` — new subsystem overview and diagram.
## Addendum: Chunked Transfer Compatibility
This branch now also fixes intercepted SDK requests that send chunked request bodies:
- `inspect_for_inference` now accepts `Transfer-Encoding: chunked` and decodes chunked request bodies before forwarding to the gateway
- Removed the prior `411 Length Required` response for chunked intercepted requests
- Added request/response header sanitization for framing and hop-by-hop headers (`content-length`, `transfer-encoding`, `connection`, etc.) to keep forwarded requests and returned responses valid
- Added unit tests for chunked parsing and header sanitization
Note: this improves compatibility for streaming-style SDK request patterns; true token-by-token passthrough response streaming is still a separate follow-up.
## Minimal Policy for Inference Routing
```yaml
inference:
allowed_routes:
- local
```
Any outgoing connection from a binary not explicitly allowed in `network_policies` will be intercepted and checked for inference API patterns.
## Test Plan
- [x] `cargo test --workspace` — all tests pass
- [x] `mise run pre-commit` — all checks pass
- [x] E2E: OpenAI chat completions routed through gateway
- [x] E2E: Anthropic messages routed through gateway
- [x] E2E: Python OpenAI SDK from sandbox (`examples/inference/inference.py`)
- [x] E2E test: `e2e/python/test_inference_routing.py`
Closes#44
## Summary
- Add `nav sandbox image push --dockerfile <path>` command to build and push custom container images into the cluster
- Fix sandbox Docker build failure caused by OpenClaw installer TTY access
- Fix sandbox proxy crash when network namespace is unavailable (fall back to loopback)
- Add e2e test for the full custom image build and sandbox creation flow
- Rename and rewrite BYOC architecture doc to reflect current state
## Changes
### `navigator-bootstrap`
- New `build` module (`src/build.rs`) with `build_and_push_image()` public API
- Builds via bollard `Docker::build_image()`, streams output to caller
- `.dockerignore` support with glob matching, 8 unit tests
### `navigator-cli`
- Added `SandboxImageCommands::Push` subcommand with `--dockerfile`, `--tag`, `--context`, `--build-arg` flags
- Added `test:e2e:custom-image` mise task
### `navigator-sandbox`
- Proxy now falls back to `127.0.0.1:3128` when no network namespace or explicit bind address is available, instead of erroring out
### Docker / Infra
- Sandbox Dockerfile: replaced OpenClaw native installer (`curl | bash`) with `npm install -g openclaw` to avoid `/dev/tty` failures in Docker build
### Documentation
- Renamed `architecture/sandbox-byoc.md` to `sandbox-custom-containers.md`
- Rewrote doc to describe current behavior (CLI usage, push pipeline, supervisor behavior, design decisions) instead of listing implementation changes
### Tests
- Added `e2e/bash/test_sandbox_custom_image.sh` covering image push + sandbox create with custom image
## Summary
- add a Docker-based macOS arm64 wheel build path so darwin wheels can be produced from Linux CI hosts
- wire macOS wheel builds into release publish workflows (`publish:main` and `publish:tag`) alongside existing Linux wheel builds
- mirror `crazymax/osxcross:latest` into `$CI_REGISTRY_IMAGE/third_party/osxcross:latest` and use that mirrored image in CI publish jobs
## Scope Notes
- this MR focuses on shipping reliable macOS wheel build support first
- temporary MR-only smoke-job wiring was removed after validation so the branch reflects final intended CI behavior
- broader cache optimization work is tracked separately in issue #43
## Test Plan
- run `mise run pre-commit`
- validate `.gitlab-ci.yml` with `uv run --with pyyaml python -c 'import yaml, pathlib; yaml.safe_load(pathlib.Path(".gitlab-ci.yml").read_text())'`
## Summary
- Add `Provider` entity for managing 3p deps from a sandbox
- Add provider CRUD API/server persistence and new CLI workflows (`nav provider create/get/list/update/delete`), including `--from-existing` laptop discovery.
- Integrate providers into sandbox create flow: infer from command (`-- claude`), support repeatable `--provider <type>`, prompt before auto-create, and allow manual in-sandbox setup.
- Add a dedicated `navigator-providers` crate with per-provider modules and mockable discovery test helpers.
## Key UX Changes
- `nav sandbox create --provider gitlab -- claude`
- Missing provider prompt now asks before creating from local state.
- `nav provider list --names` for scripting/cleanup.
## Test Plan
- `mise run cluster:deploy`
- `mise run test:e2e:sandbox`
- `mise run pre-commit`
Closes#19Closes#22Closes#11
Drew Newberry