> **🏗️ build-from-issue-agent**
Closes#79
## Summary
Replaces the gateway-proxied inference model (`ProxyInference` RPC) with sandbox-local execution. The sandbox now resolves routes from either a standalone YAML route file or a cluster bundle fetched via the new `GetSandboxInferenceBundle` RPC, then forwards requests directly to inference backends using `navigator-router`.
### Benefits of removing the gRPC hop
Moving inference execution from the gateway to the sandbox eliminates the gRPC round-trip for every inference request:
- **No payload size limits**: Direct HTTP forwarding replaces protobuf-over-gRPC serialization.
- **Streaming-ready**: Preserves connection semantics for future SSE support (e.g., `stream: true`).
- **Lower latency**: Direct sandbox → backend path instead of sandbox → gateway → backend round-trip.
- **Reduced gateway load**: Gateway only serves the control-plane bundle delivery RPC.
- **Simpler error model**: HTTP status codes flow through directly without gRPC wrapping.
## Changes Made
- **Proto**: Removed `ProxyInference` RPC. Added `GetSandboxInferenceBundle` RPC for route bundle delivery.
- **Gateway**: Replaced `proxy_inference` with `get_sandbox_inference_bundle`. Removed `Router` and `navigator-router` dependency — gateway is now control-plane only for inference.
- **Router**: Switched config file format from TOML to YAML. Added `Debug` impl that redacts API keys.
- **Sandbox**: New `InferenceContext` with local `Router` + route cache. Routes loaded from `--inference-routes` file (standalone) or cluster bundle via gRPC, with 30s background refresh. New `--inference-routes` / `NAVIGATOR_INFERENCE_ROUTES` CLI arg.
- **Dev sandbox**: Added `inference-routes.yaml` at repo root with default NVIDIA NIM route. `mise run sandbox` now mounts it automatically and supports `-e` flag to forward env vars (e.g., `mise run sandbox`).
- **Example**: Added standalone example (`examples/inference/routes.yaml`) and rewrote README to cover both standalone and cluster workflows.
- **E2E tests**: Updated existing test names/docs. Added tests for Anthropic messages protocol and multi-route policy filtering.
## Tests Added
- **Unit (21 new):** `router_error_to_http` (6 variants), `load_from_file` YAML round-trip (5), `resolve_sandbox_inference_bundle` gRPC handler (6), `build_inference_context` route loading (4)
- **E2E (2 new):** Anthropic messages protocol routing, route filtering by `allowed_routes`
- **Total:** 188 unit tests passing across 3 crates
## Documentation Updated
- `architecture/gateway.md`: Removed router, updated Inference Service
- `architecture/sandbox.md`: Updated orchestration flow, InferenceContext, route loading
- `architecture/inference-routing.md`: Complete rewrite for sandbox-local architecture
## Verification
- [x] All unit tests passing (188 across 3 crates)
- [x] Pre-commit Rust checks passing (fmt, clippy, tests)
- [x] Architecture documentation updated
- [x] E2E tests updated with new test cases
- [x] Standalone example and documentation added
- [x] Default inference-routes.yaml with env passthrough for mise run sandbox
## Summary
- Track last deployed local changes and only redeploy new deltas in auto mode.
- Add per-component fingerprints (server/sandbox/helm) persisted to .cache/cluster-deploy-fast.state.
- Update docs for the new incremental deploy behavior.
```
mise run cluster
....
________________________________________________________
Executed in 75.21 secs fish external
usr time 578.03 millis 0.25 millis 577.78 millis
sys time 364.45 millis 1.22 millis 363.22 millis
```
with a change to `navigator-sandbox` (triggers rust recompile)
```
⟩ time mise run cluster:deploy
[cluster:deploy] $ build/scripts/cluster-deploy-fast.sh
Change detection took 0s
Fast deploy plan:
build server: 0
build sandbox: 1
helm upgrade: 1
...
________________________________________________________
Executed in 25.88 secs fish external
usr time 618.91 millis 0.24 millis 618.67 millis
sys time 557.71 millis 1.54 millis 556.18 millis
```
(most time is spent in docker build)
## Test Plan
- mise run pre-commit
## Summary
- Rewrites the README quickstart with accurate, tested setup steps (Docker prerequisite, venv/uv usage)
- Moves the "developing from source" instructions into CONTRIBUTING.md
- Removes the manual cluster deploy step from quickstart (cluster bootstraps automatically on sandbox create)
- Fixes empty bash array expansion in Docker build scripts that broke on macOS default bash 3.2 (`set -u` treats `"${EMPTY_ARRAY[@]}"` as unbound)
## Changes
- **README.md**: Replace placeholder quickstart with working install and sandbox create steps; remove manual cluster deploy step since it happens automatically
- **CONTRIBUTING.md**: Add "developing from source" section with mise setup instructions
- **build/scripts/docker-build-cluster.sh**: Use `${arr[@]+"${arr[@]}"}` pattern for safe empty array expansion
- **build/scripts/docker-build-component.sh**: Same fix
Closes#48, #52
## Summary
- Replace the envoy-gateway-based TLS setup with inline PKI generation during cluster bootstrap, generating CA, server, and client certificates directly in the `navigator-bootstrap` crate
- Remove all envoy gateway Helm templates (`gateway.yaml`, `gatewayclass.yaml`, `grpcroute.yaml`, PKI job, traffic policies) and the `Dockerfile.pki-job`
- Add native mTLS support to the navigator server with `tokio-rustls`, mounting client TLS certs as volumes into sandbox pods
- Update cluster entrypoint, healthcheck, and deploy scripts to work with the new direct-TLS architecture
- Add TLS security e2e test and fix formatting/clippy warnings
## Test Plan
- All unit tests pass (`cargo test --workspace`)
- Clippy clean (`cargo clippy --workspace --all-targets`)
- Format clean (`cargo fmt --all -- --check`)
- Python tests pass (`uv run pytest python/`)
- Full `mise run pre-commit` passes
Closes#20, #36, #46
## Summary
- Replace the external `rsync` dependency with built-in tar-over-SSH for file synchronization, using the `tar` crate to stream archives through the existing SSH proxy tunnel
- Add a new `nav sandbox sync` subcommand supporting bidirectional file transfer (`--up` to push local files, `--down` to pull sandbox files)
- Add sync workflow example in `examples/sync-files.md`
## Changes
| File | Change |
|---|---|
| `crates/navigator-cli/Cargo.toml` | Add `tar = "0.4"` dependency |
| `crates/navigator-cli/src/ssh.rs` | Remove `sandbox_rsync()`, add `sandbox_sync_up()`, `sandbox_sync_down()`, `sandbox_sync_up_files()` |
| `crates/navigator-cli/src/run.rs` | Add `sandbox_sync_command()` dispatcher, update re-exports, update `--sync` call site |
| `crates/navigator-cli/src/main.rs` | Add `Sync` variant to `SandboxCommands` with `--up`/`--down` flags |
| `architecture/sandbox-connect.md` | Rewrite File Sync section for tar-over-SSH and new command |
| `examples/sync-files.md` | New example walkthrough for sync workflows |
## New UX
```bash
# Push local files up to sandbox
nav sandbox sync my-sandbox --up ./src /sandbox/src
# Pull sandbox files down to local
nav sandbox sync my-sandbox --down /sandbox/output ./output
# Existing --sync flag on create still works (now uses tar internally)
nav sandbox create --sync -- python main.py
```
## Design
- **No new host dependencies**: `tar` crate handles archive creation/extraction in Rust; `ssh` is already required. The sandbox side uses the `tar` binary from the base image.
- **No new server-side infrastructure**: reuses the existing SSH tunnel, no new gRPC RPCs or gateway changes.
- **Bidirectional**: the same tar-pipe pattern works for both push and pull by reversing which side produces and which consumes the archive.
- **No compression** for v1 — the SSH tunnel is local-network; can add gzip via `flate2` later.
## Test Plan
- All existing tests pass (`mise run pre-commit` green: fmt, clippy, 206 Rust tests, 6 Python tests)
- Manual e2e testing required for actual sandbox sync operations
The sandbox Dockerfile was moved from deploy/docker/Dockerfile.sandbox to
deploy/docker/sandbox/Dockerfile.base but the multiarch publish script
was not updated, causing publish_main_artifacts to fail.
Closes#44
## Summary
- Add `nav sandbox image push --dockerfile <path>` command to build and push custom container images into the cluster
- Fix sandbox Docker build failure caused by OpenClaw installer TTY access
- Fix sandbox proxy crash when network namespace is unavailable (fall back to loopback)
- Add e2e test for the full custom image build and sandbox creation flow
- Rename and rewrite BYOC architecture doc to reflect current state
## Changes
### `navigator-bootstrap`
- New `build` module (`src/build.rs`) with `build_and_push_image()` public API
- Builds via bollard `Docker::build_image()`, streams output to caller
- `.dockerignore` support with glob matching, 8 unit tests
### `navigator-cli`
- Added `SandboxImageCommands::Push` subcommand with `--dockerfile`, `--tag`, `--context`, `--build-arg` flags
- Added `test:e2e:custom-image` mise task
### `navigator-sandbox`
- Proxy now falls back to `127.0.0.1:3128` when no network namespace or explicit bind address is available, instead of erroring out
### Docker / Infra
- Sandbox Dockerfile: replaced OpenClaw native installer (`curl | bash`) with `npm install -g openclaw` to avoid `/dev/tty` failures in Docker build
### Documentation
- Renamed `architecture/sandbox-byoc.md` to `sandbox-custom-containers.md`
- Rewrote doc to describe current behavior (CLI usage, push pipeline, supervisor behavior, design decisions) instead of listing implementation changes
### Tests
- Added `e2e/bash/test_sandbox_custom_image.sh` covering image push + sandbox create with custom image
## Summary
- add a Docker-based macOS arm64 wheel build path so darwin wheels can be produced from Linux CI hosts
- wire macOS wheel builds into release publish workflows (`publish:main` and `publish:tag`) alongside existing Linux wheel builds
- mirror `crazymax/osxcross:latest` into `$CI_REGISTRY_IMAGE/third_party/osxcross:latest` and use that mirrored image in CI publish jobs
## Scope Notes
- this MR focuses on shipping reliable macOS wheel build support first
- temporary MR-only smoke-job wiring was removed after validation so the branch reflects final intended CI behavior
- broader cache optimization work is tracked separately in issue #43
## Test Plan
- run `mise run pre-commit`
- validate `.gitlab-ci.yml` with `uv run --with pyyaml python -c 'import yaml, pathlib; yaml.safe_load(pathlib.Path(".gitlab-ci.yml").read_text())'`
## Summary
- Add `GetSandboxProviderEnvironment` gRPC endpoint so the sandbox supervisor can fetch provider credentials at runtime instead of embedding them in the pod spec
- Sandbox supervisor (`navigator-sandbox`) fetches credentials on startup and injects them as environment variables into both entrypoint processes and SSH shell sessions via `Command::env()`
- Remove credential injection from sandbox creation-time pod spec in `navigator-server`
- Update `architecture/providers.md` to document the full runtime credential injection flow, discovery engine details, trait definitions, and end-to-end diagram
## Changes
- **proto**: Add `GetSandboxProviderEnvironment` RPC and request/response messages, add `providers` field to `SandboxSpec`
- **navigator-server**: Implement `GetSandboxProviderEnvironment` handler and `resolve_provider_environment()` with env var key validation and first-wins dedup; remove old creation-time credential injection from `sandbox/mod.rs`
- **navigator-sandbox**: Fetch provider env via gRPC on startup (`grpc_client.rs`), thread `provider_env` HashMap through to `process.rs` (entrypoint) and `ssh.rs` (shell sessions), inject via `cmd.env()`
- **navigator-cli**: Adjust sandbox create flow to set provider names in `SandboxSpec.providers`
- **e2e**: Add `test_sandbox_providers.py` end-to-end tests
- **docs**: Rewrite `architecture/providers.md` with implementation details
## Test Plan
- Unit tests for `resolve_provider_environment` in `grpc.rs`
- Integration tests updated in `provider_commands_integration.rs`, `mtls_integration.rs`, `multiplex_integration.rs`, `multiplex_tls_integration.rs`
- New e2e test suite `test_sandbox_providers.py`
## Summary
- Add `Provider` entity for managing 3p deps from a sandbox
- Add provider CRUD API/server persistence and new CLI workflows (`nav provider create/get/list/update/delete`), including `--from-existing` laptop discovery.
- Integrate providers into sandbox create flow: infer from command (`-- claude`), support repeatable `--provider <type>`, prompt before auto-create, and allow manual in-sandbox setup.
- Add a dedicated `navigator-providers` crate with per-provider modules and mockable discovery test helpers.
## Key UX Changes
- `nav sandbox create --provider gitlab -- claude`
- Missing provider prompt now asks before creating from local state.
- `nav provider list --names` for scripting/cleanup.
## Test Plan
- `mise run cluster:deploy`
- `mise run test:e2e:sandbox`
- `mise run pre-commit`
Closes#19Closes#22Closes#11
## What changed
- Added tag-driven release flow in GitLab CI:
- new `release` stage with tag-only jobs
- `publish_tag_artifacts` publishes Docker + Python artifacts on `vX.Y.Z` tags
- `create_release_notes` generates release notes from conventional commits via `git-cliff` and creates a GitLab release via `glab`
- Updated main-branch image publishing to version-aware tagging:
- `publish_ecr_images` now runs `mise run publish:main`
- main publishes `:dev`, `:latest`, and a versioned dev tag
- Added release-oriented mise tasks:
- `publish:main`
- `publish:tag`
- `python:publish:macos` (manual macOS arm64 wheel publish)
- Switched Linux Python wheel builds to buildx:
- added `deploy/docker/Dockerfile.python-wheels`
- replaced old per-arch docker-run tasks with `python:build:multiarch`
- Added macOS arm64 wheel build path for local publishing:
- `python:build:macos` builds `aarch64-apple-darwin`
- intended to run locally on macOS after tag CI finishes
- Made Docker multiarch publish script tag-flexible:
- `TAG_LATEST` is no longer hardcoded in ECR mode
- supports `EXTRA_DOCKER_TAGS`
- applies extra tags to sandbox/server/pki-job/cluster images
- Moved release tooling to `build/scripts/release.py` and updated all mise references
- Removed obsolete Docker Artifactory env vars from `mise.toml`
## Release behavior
- **Main branch CI**
- Docker: `:dev`, `:latest`, and versioned dev tag
- **Tag CI (`vX.Y.Z`)**
- Docker: `:X.Y.Z` only (no `:latest`)
- Python: Linux wheels published from CI
- GitLab release notes created from conventional commits
- **Manual macOS step (after tagging)**
1. Checkout the tag locally on macOS
2. Run `mise run python:publish:macos`
## Validation
- `mise run python:lint`
- `mise run version:print`
- `mise run python:build:macos`
- `uv run python build/scripts/release.py --help`
Closes#13
## Summary
- add Python sandbox execution APIs for command and callable workflows
- consolidate sandbox policy fixtures and expand e2e test coverage for policy and Python exec paths
- update CI/build config and images for sandbox e2e execution dependencies
## Test Plan
- mise run pre-commit
Closes#26
## Summary
Consolidates sandbox policy data into `dev-sandbox-policy.yaml` and deletes `dev-sandbox-policy-data.rego`. The YAML file becomes the single source of truth for sandbox policy data across both paths:
- **`nav create sandbox`** (CLI reads YAML → proto → gRPC → server stores → sandbox fetches)
- **`mise run sandbox`** (sandbox reads YAML directly via regorus)
## Why YAML
The policy data file is user-facing — it's the interface users interact with to configure sandbox security policy (allowed endpoints, binary identities, filesystem access, etc.).
Keeping this as YAML rather than rego:
- **Decouples data from enforcement engine** — if we change from OPA/rego to a different policy engine, the user's data file stays the same
- **Still works with regorus** — regorus can natively load YAML data via its `yaml` feature, so there's no conversion step in the local dev path
- **Enables future indirection** — we can add a transformation layer between the YAML file and what the policy engine receives, without changing the user-facing format
## Changes
### Policy data (`dev-sandbox-policy.yaml`)
- Flat top-level structure: `filesystem_policy`, `landlock`, `process`, `inference`, `network_policies`
- Renamed `filesystem` → `filesystem_policy` to match rego/proto convention
- Deleted `dev-sandbox-policy-data.rego` (was the rego equivalent of this data)
### Rego rules (`dev-sandbox-policy.rego`)
- All data references updated: `data.sandbox.*` → `data.*` (dropped `sandbox.` prefix)
- `package navigator.sandbox` unchanged (that's the rego rule namespace, not the data path)
### Sandbox entrypoint (`navigator-sandbox`)
- `--rego-policy`/`--rego-data` → `--policy-rules`/`--policy-data`
- `NAVIGATOR_REGO_POLICY`/`NAVIGATOR_REGO_DATA` → `NAVIGATOR_POLICY_RULES`/`NAVIGATOR_POLICY_DATA`
- `from_files()`: loads YAML natively via `regorus::Value::from_yaml_file()`
- `from_strings()` and `reload()`: accept YAML data via `Value::from_yaml_str()`
- `proto_to_opa_data_json()`: emits data at root level (no `"sandbox"` wrapper)
- Enabled regorus `yaml` feature
### CLI (`navigator-cli`)
- `DevSandboxPolicyFile.filesystem` → `filesystem_policy`
### Tests
- All inline test data converted from rego/JSON to YAML
- `test_engine()` uses `from_strings(rules, yaml)` with the dev YAML file
### Documentation
- Updated `architecture/plans/sandbox-policy-refactor/plan.md`, `architecture/plans/opa/plan.md`, `architecture/sandbox.md`, `architecture/containers.md`
Closes#24
## Problem
SSH sessions into the sandbox were **not entering the network namespace** or receiving proxy environment variables. This meant every command run via SSH (the only user-facing path) had unrestricted internet access, completely bypassing OPA network policy enforcement.
The root cause: the SSH server was started **before** the network namespace and proxy were created in `lib.rs`, so it never received the netns fd or proxy URL.
Additionally, **gRPC inference from within the sandbox did not work at all** — even after fixing the netns, multiple issues prevented the Python SDK from reaching the navigator server through the CONNECT proxy.
## Changes
### Sandbox binary
**Core fix — reorder startup + thread netns through SSH:**
- `lib.rs`: Move netns + proxy creation before SSH server start. Compute `ssh_netns_fd` and `ssh_proxy_url`, pass them to `run_ssh_server()`.
- `ssh.rs`: Thread `netns_fd` and `proxy_url` through the full SSH call chain into `spawn_pty_shell()`. Set proxy env vars on the shell command. Call `setns(fd, CLONE_NEWNET)` in `install_pre_exec()`.
**Proxy — control plane allowlist + IPv6 socket lookup:**
- `proxy.rs`: Connections to the navigator endpoint (derived from `NAVIGATOR_ENDPOINT`) are always allowed without OPA evaluation, logged with `engine=control_plane`. This is infrastructure the sandbox needs to function, not a user-configurable policy.
- `procfs.rs`: Extended `parse_proc_net_tcp` to also check `/proc/<pid>/net/tcp6`. gRPC C-core uses `AF_INET6` sockets even for IPv4 connections, so its TCP entries were invisible to the proxy's identity resolver. Also fixed port parsing to use `rsplit_once(':')` for correct IPv6 address handling.
**Proxy env vars — lowercase variants for gRPC C-core:**
- `process.rs` + `ssh.rs`: Added lowercase `http_proxy`, `https_proxy`, `grpc_proxy` alongside uppercase. gRPC C-core (libgrpc) checks lowercase first and was ignoring the uppercase-only vars.
### Server
- `sandbox/mod.rs`: Added `CAP_SYS_PTRACE` to sandbox pod security context. Required for the proxy (root) to read `/proc/<pid>/fd/` of sandbox-user processes for binary identity resolution.
### Python SDK
- `inference.py`: Strip `http://`/`https://` scheme from endpoint before passing to `grpc.insecure_channel()`, which expects `host:port`. Default `endpoint` and `sandbox_id` from `NAVIGATOR_ENDPOINT` / `NAVIGATOR_SANDBOX_ID` env vars so `Inference()` works inside sandboxes with no arguments.
### Build / infra
- `ci.toml`: Added `--cap-add=SYS_PTRACE` to `mise run sandbox` to mirror the k8s pod capabilities.
### Documentation
- `architecture/sandbox.md`: Documented `CAP_SYS_PTRACE` requirement and the full set of proxy env vars (uppercase + lowercase).
## Testing
All 29 sandbox unit tests pass. `mise run pre-commit` passes (fmt, clippy, all workspace tests, python lint).
E2E verified on a live cluster:
| Test | Result |
|------|--------|
| Proxy env vars (6 vars, upper+lowercase) | PASS |
| Blocked endpoints (google, anthropic via curl) | PASS — all denied |
| **gRPC inference from SSH session** (`Inference()` with no args, env var defaults) | **PASS** |
| Proxy log: navigator requests show `engine=control_plane` | PASS |
| Proxy log: blocked requests show `engine=opa` with correct deny reasons | PASS |
## Summary
- Add a "push" path for local development that exports Docker images from the host daemon and imports them directly into the k3s containerd runtime, eliminating the need for a local registry or remote image pulls during `mise run cluster`
- Add `mise run cluster:deploy` task for fast iterative rebuilds (~25s) against a running cluster without recreating it
- Update `cluster-entrypoint.sh` to patch HelmChart manifest with local image tags and `IfNotPresent` pull policy when push mode is active
## Details
Previously, after the local registry (k3d + skaffold) was removed, `mise run cluster` only pulled component images from the remote CloudFront registry. This meant local code changes required a remote publish cycle to test in-cluster.
This MR adds a push path that sits alongside the existing pull path:
- **`push.rs`** — Uses bollard to `export_images()` all 3 component images into a single tar (with shared layer dedup), uploads via Docker `put_archive` API into the cluster container, then runs `ctr images import` to load them into containerd
- **`docker.rs`** — Passes `IMAGE_TAG` and `IMAGE_PULL_POLICY` env vars into the cluster container when `NAVIGATOR_PUSH_IMAGES` is set
- **`lib.rs`** — Wires push into the deploy flow between `clean_stale_nodes()` and `wait_for_cluster_ready()`, guarded by `remote_opts.is_none() && NAVIGATOR_PUSH_IMAGES`
- **`cluster-entrypoint.sh`** — Patches the baked-in HelmChart manifest with sed to use the local image tag and pull policy before k3s starts
- **`cluster.toml`** — Updated `[cluster]` task to tag and push images; new `[cluster:deploy]` task for iterative deploys via `docker save | ctr import` + `helm upgrade`
The remote deploy path (`nav cluster admin deploy --remote`) is completely unaffected — the push logic only activates for local deploys with the env var set.
## Test Plan
- `mise run cluster` — end-to-end success, cluster healthy with local images (~168s)
- `mise run cluster:deploy` — iterative deploy success, helm upgrade + rollout (~43s)
- `mise run pre-commit` — all checks pass (fmt, clippy, 82 tests, python lint)
- Verified images present in containerd: `ctr -a /run/k3s/containerd/containerd.sock images ls | grep navigator/`
- Remote path verified safe via code analysis (guards prevent push logic from running)
Drew Newberry