* fix(sandbox): add GPU device nodes and nvidia-persistenced to landlock baseline
Landlock READ_FILE/WRITE_FILE restricts open(2) on character device files
even when DAC permissions would otherwise allow it. GPU sandboxes need
/dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, /dev/nvidia-modeset,
and per-GPU /dev/nvidiaX nodes in the policy to allow NVML initialization.
Additionally, CDI bind-mounts /run/nvidia-persistenced/socket into the
container. NVML tries to connect to this socket at init time; if the
directory is not in the landlock policy, it receives EACCES (not
ECONNREFUSED), which causes NVML to abort with NVML_ERROR_INSUFFICIENT_PERMISSIONS
even though nvidia-persistenced is optional.
Both classes of paths are auto-added to the baseline when /dev/nvidiactl is
present. Per-GPU device nodes are enumerated at runtime to handle multi-GPU
configurations.
* feat(sandbox): add L7 query parameter matchers
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(sandbox): decode + as space in query params and validate glob syntax
Three improvements from PR #617 review:
1. Decode + as space in query string values per the
application/x-www-form-urlencoded convention. This matches Python's
urllib.parse, JavaScript's URLSearchParams, Go's url.ParseQuery, and
most HTTP frameworks. Literal + should be sent as %2B.
2. Add glob pattern syntax validation (warnings) for query matchers.
Checks for unclosed brackets and braces in glob/any patterns. These
are warnings (not errors) because OPA's glob.match is forgiving,
but they surface likely typos during policy loading.
3. Add missing test cases: empty query values, keys without values,
unicode after percent-decoding, empty query strings, and literal +
via %2B encoding.
* fix(sandbox): add missing query_params field in forward proxy L7 request info
* style(sandbox): fix formatting in proxy L7 query param parsing
---------
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* fix(sandbox,server): fix chunk merge duplicates and OPA variable collision with overlapping policies
Two related bugs triggered when a draft rule approval creates a second
policy entry for the same host:port:
1. merge_chunk_into_policy looked up existing rules by chunk.rule_name
(auto-generated as allow_{host}_{port}), which never matched the
user's original rule name. Now scans all network_policies entries
for a host:port endpoint match before falling back to insertion,
and merges allowed_ips into the existing endpoint.
2. The Rego allow_request rule and _matching_endpoint_configs
comprehension used 'some ep; ep := policy.endpoints[_]' which
caused regorus to error with 'duplicated definition of local
variable ep' when multiple policies covered the same host:port.
Refactored to isolate endpoint iteration inside helper functions
(_policy_allows_l7, _policy_endpoint_configs) so variables are
scoped per-policy evaluation.
Refs: #567
* test(e2e): add overlapping policy tests and update FWD-2 for implicit allowed_ips
- Update FWD-2 (test_forward_proxy_denied_without_allowed_ips ->
test_forward_proxy_allows_private_ip_host_without_allowed_ips):
literal IP host no longer requires explicit allowed_ips, expects 200.
- Add OVL-1: overlapping L4 policies for same host:port must not crash
OPA and should allow forward proxy connections.
- Add OVL-2: overlapping L7 policies for same host:port must not crash
OPA and should allow CONNECT tunnel establishment.
Refs: #567
* style: apply cargo fmt formatting
* test(e2e): update SSRF-3 and SSRF-6 for implicit allowed_ips behavior
SSRF-6: Private IP with literal IP host now gets implicit allowed_ips
from PR #570, so CONNECT returns 200 instead of 403.
SSRF-3: Loopback is still blocked but via the always-blocked path
(implicit allowed_ips is synthesized, then resolve_and_check_allowed_ips
catches it). Log message says 'always-blocked' instead of 'internal
address'.
* fix(e2e): use negative assertion for SSRF-6 when nothing listens on target port
When the SSRF check passes but nothing listens on the target port,
recv() returns empty bytes. Use 'assert 403 not in' (matching SSRF-4
pattern) instead of 'assert 200 in'.
* fix(e2e): update provider tests for redacted credential values
PR #569 changed credential redaction from clearing the map to
replacing values with 'REDACTED'. Update e2e assertions to expect
credential keys with REDACTED values instead of an empty map.
* feat(policy): support host wildcards and multi-port endpoints
Add glob-style host wildcards to endpoints[].host using OPA's
glob.match with "." as delimiter — *.example.com matches a single
DNS label, **.example.com matches across labels. Validation rejects
bare * and requires *. prefix; warns on broad patterns like *.com.
Add repeated uint32 ports field to NetworkEndpoint for multi-port
support. Backwards compatible: existing port scalar is normalized to
ports array. Both the proto-to-JSON and YAML-to-JSON conversion paths
emit a ports array; Rego always references endpoint.ports[_].
Fix OpaEngine::reload() to route through the full preprocessing
pipeline instead of bypassing L7 validation and port normalization.
Closes#359
* fix(policy): reject configs with both port and ports set
---------
Co-authored-by: johntmyers <johntmyers@users.noreply.github.com>
* feat(proxy): support plain HTTP forward proxy for private IP endpoints
Add forward proxy mode to the sandbox proxy so that standard HTTP
libraries (httpx, requests, etc.) work with HTTP_PROXY for plain HTTP
calls to private IP endpoints. Previously, non-CONNECT methods were
unconditionally rejected with 403.
The forward proxy path requires all three conditions to be met:
- OPA policy explicitly allows the destination
- The matched endpoint has allowed_ips configured
- All resolved IPs are RFC 1918 private
This ensures plain HTTP never reaches the public internet while enabling
seamless access to internal services without custom CONNECT tunnel code.
Implementation:
- parse_proxy_uri(): parses absolute-form URIs into components
- rewrite_forward_request(): rewrites to origin-form, strips hop-by-hop
headers, adds Via and Connection: close
- handle_forward_proxy(): full handler with OPA eval, SSRF checks,
private-IP gate, upstream connect, and bidirectional relay
- Updated dispatch in handle_tcp_connection to route non-CONNECT methods
Includes 14 unit tests and 6 E2E tests (FWD-1 through FWD-6).
CONNECT path remains completely untouched.
Closes#155
* fix(proxy): remove InspectForInference match arm removed by #146
The inference routing simplification in #146 reduced NetworkAction to
Allow/Deny, removing InspectForInference. Drop the dead match arm from
handle_forward_proxy.
* fix(sandbox): restore BestEffort as default Landlock compatibility
The Landlock V2 upgrade in #151 changed the default from BestEffort to
HardRequirement. This causes all proxy-mode sandboxes to crash with
Permission denied when the policy omits the landlock field, because the
child process gets locked to only /etc/navigator-tls and /sandbox.
Restore BestEffort as the default so policies without an explicit
landlock field degrade gracefully.
Fixes#161
* fix(sandbox): inject baseline filesystem paths for proxy-mode sandboxes
Proxy-mode sandboxes need baseline filesystem paths (/usr, /lib, /etc,
/app, /var/log read-only; /sandbox, /tmp read-write) for the child
process to function under Landlock. Without these, the child can't exec
binaries, resolve DNS, or load shared libraries.
The supervisor now enriches the policy with these baseline paths at
startup, covering both standalone (file) and gateway (gRPC) modes. For
gateway mode, the enriched policy is synced back so users see the
effective policy via 'nemoclaw sandbox get'.
The gateway validation is relaxed to allow additive filesystem changes
(new paths can be added, existing paths cannot be removed) to support
the supervisor's enrichment sync-back.
Includes 2 E2E tests: BFS-1 (missing filesystem_policy) and BFS-2
(incomplete filesystem_policy).
Fixes#161
* fix(e2e): update assertion for relaxed filesystem validation message
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* refactor(policy): consolidate duplicated YAML struct hierarchies into navigator-policy
Closes#96
Merge the Deserialize-only input structs and Serialize-only output structs
into a single set of types in navigator-policy that derive both Serialize
and Deserialize. This eliminates the duplicate PolicyYaml hierarchy in
navigator-cli and fixes three round-trip issues:
- filesystem_policy vs filesystem field name mismatch
- allowed_ips silently dropped on proto-to-YAML conversion
- network policy name field silently dropped on proto-to-YAML conversion
Also adds api_patterns support to the inference YAML schema and switches
network_policies from HashMap to BTreeMap for deterministic output ordering.
* fix(e2e): update non-CONNECT test assertion from 405 to 403
Align test_l4_non_connect_method_rejected with the proxy behavior
change in c06117e which intentionally returns 403 for non-CONNECT
requests.
---------
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
* feat(sandbox): allow egress to private IP space via allowed_ips policy field
Add an allowed_ips field to NetworkEndpoint that accepts CIDR notation and
exact IPs, enabling sandboxes to reach private IP space under policy control
while maintaining SSRF protections for loopback and link-local addresses.
Three modes are supported:
- Default (no allowed_ips): all private IPs blocked (existing behavior)
- Host + allowlist: domain must resolve to an IP in allowed_ips
- Hostless allowlist (no host): any domain allowed if resolved IP matches
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Closes#78
## What You Can Do
### Live policy updates
Update a running sandbox's network policy without recreating it:
```bash
nav sandbox policy set <sandbox> --policy new-policy.yaml --wait --timeout 60
```
The sandbox hot-reloads the policy within 30s (configurable). On failure, the previous policy stays active (last-known-good).
Idempotent — submitting the same policy twice returns the existing version:
```
✓ Policy version 3 submitted (hash: a1b2c3d4e5f6)
$ nav sandbox policy set test --policy same.yaml
· Policy unchanged (version 3, hash: a1b2c3d4e5f6)
```
### Policy history & inspection
```bash
nav sandbox policy list <sandbox> # version history with status
nav sandbox policy get <sandbox> # current policy metadata
nav sandbox policy get <sandbox> --full # print full policy as YAML
nav sandbox policy get <sandbox> --rev 2 --full # specific revision as YAML
```
### Sandbox logs
Stream logs from both the gateway and sandbox supervisor in one view:
```bash
nav sandbox logs <sandbox> # one-shot, last 2000 lines
nav sandbox logs <sandbox> --tail # live streaming
nav sandbox logs <sandbox> --source sandbox # supervisor logs only
nav sandbox logs <sandbox> --source gateway # gateway logs only
nav sandbox logs <sandbox> --level warn # warnings and errors only
nav sandbox logs <sandbox> --since 5m # last 5 minutes
nav sandbox logs <sandbox> --tail --source sandbox --level info
```
Each log line is tagged with its source and includes structured fields:
```
[1772055394.673] [sandbox] [INFO ] [navigator_sandbox::proxy] CONNECT action=allow dst_host=api.anthropic.com dst_port=443 policy=claude_code
[1772055061.005] [gateway] [INFO ] [navigator_server::grpc] GetSandboxPolicy served from policy history
```
---
## Implementation
### Proto changes
- 7 new RPCs: `UpdateSandboxPolicy`, `GetSandboxPolicyStatus`, `ListSandboxPolicies`, `ReportPolicyStatus`, `GetSandboxLogs`, `PushSandboxLogs` (client-streaming)
- `SandboxLogLine`: added `source` (gateway/sandbox), `fields` (structured key-value map)
- `WatchSandboxRequest`: added `log_since_ms`, `log_sources`, `log_min_level`
- `GetSandboxLogsRequest`: added `sources`, `min_level`
- `PolicyStatus` enum, `SandboxPolicyRevision` message
- `Sandbox.current_policy_version` field
### Server
- **Policy persistence**: New `sandbox_policies` table (SQLite + Postgres) with per-sandbox monotonic versions, status tracking, and policy hash
- **UpdateSandboxPolicy**: Validates static field immutability (filesystem/landlock/process), network mode consistency (Block↔Proxy), deterministic hash comparison for idempotent updates
- **Lazy backfill**: First `GetSandboxPolicy` call creates version 1 from `spec.policy` for existing sandboxes
- **Log broker**: `TracingLogBus::publish_external()` injects sandbox-pushed logs into the same broadcast channel + tail buffer (2000 lines). Server forces `source="sandbox"` and `sandbox_id` on all pushed logs
- **Source/level filtering**: Applied server-side in both `GetSandboxLogs` and `WatchSandbox` streams
- **Version supersession**: When a new version is loaded, all older pending+loaded versions are marked superseded
### Sandbox
- **`OpaEngine::reload_from_proto()`**: Full `from_proto()` pipeline (L7 validation, access preset expansion) with atomic engine swap. On failure, previous engine untouched (LKG)
- **Policy poll loop**: Background task polls every 30s (configurable via `NAVIGATOR_POLICY_POLL_INTERVAL_SECS`), reports status via `ReportPolicyStatus` RPC
- **`LogPushLayer`**: Tracing layer captures events at INFO+ (configurable via `NAVIGATOR_LOG_PUSH_LEVEL`), sends structured fields via `PushSandboxLogs` client-streaming RPC. Background task batches 50 lines / flushes every 500ms. Best-effort (drops on full channel, never blocks)
- **`CachedNavigatorClient`**: Persistent mTLS channel for both policy polling and log push
### Database migration
- `002_create_sandbox_policies.sql` (SQLite + Postgres)
## Tests
- **Unit**: 8 policy persistence tests (put/get/list/status/supersede/isolation)
- **Integration**: 4 test files updated with new RPC stubs
- **E2E**: `test_live_policy_update_and_logs` — full lifecycle: create → set same (unchanged) → push new → wait for load → verify connectivity → push same (unchanged) → verify history → fetch logs
## Documentation
- `architecture/sandbox.md`: Log streaming architecture, LogPushLayer, push task, server broker, source tagging, structured fields, CLI filtering, failure modes
- `architecture/security-policy.md`: Live update semantics, deterministic hashing, CLI filter flags, policy inspection
- `architecture/plans/issue-78-sandbox-log-streaming.md`: Design plan for log streaming
## Security
- Trust boundary documented: shared mTLS cert model (per-sandbox auth tracked in #80)
- Server forces `source="sandbox"` and `sandbox_id` on pushed logs (can't impersonate gateway or other sandboxes)
- Per-batch line cap (100) prevents flooding
Closes#67
## Summary
Implements transparent inference interception and routing for sandboxed AI agents. The sandbox proxy intercepts outbound AI SDK calls (OpenAI, Anthropic) and reroutes them through the gateway to policy-controlled backends — enabling organizations to redirect inference traffic to local or self-hosted models without modifying agent code.
**Decision model** — a tri-state OPA evaluation for every CONNECT request:
1. Binary + endpoint explicitly allowed in `network_policies` → **allow** (pass through)
2. Not explicitly allowed + `inference.allowed_routes` configured → **inspect for inference** (TLS intercept, detect API patterns, route through gateway)
3. Otherwise → **deny**
No endpoint declarations or binary lists needed for inference routing. Just configure `inference.allowed_routes`.
## Key Changes
### Sandbox (interception)
- **OPA policy**: New `network_action` Rego rule with three outcomes (`allow`, `inspect_for_inference`, `deny`). New `NetworkAction` enum replaces `PolicyDecision.allowed` bool for the proxy's main decision path.
- **Proxy**: New `InspectForInference` path — TLS-terminates client, parses HTTP, detects inference API patterns (`POST /v1/chat/completions`, `/v1/completions`, `/v1/messages`), strips auth headers, forwards via gRPC.
- **New module**: `l7/inference.rs` — `InferenceApiPattern`, `detect_inference_pattern()`, HTTP request/response parsing.
- **gRPC client**: New `proxy_inference()` for sandbox→gateway forwarding.
- **Sandbox init**: Creates OPA engine when inference is configured, even without `network_policies`.
### Gateway (dispatch)
- **InferenceService**: `ProxyInference` RPC loads sandbox policy, resolves allowed routes, dispatches to router. Full CRUD for inference routes.
- **Proto**: `InferenceRoute`, `InferenceRouteSpec`, `ProxyInferenceRequest/Response`, Inference gRPC service.
### Router (backend proxying)
- **New crate**: `navigator-router` with `Router`, `proxy_with_candidates()`, protocol-based route selection, backend HTTP proxying with auth header rewriting.
- **Mock support** for testing (`mock://` scheme).
### CLI
- `nav inference create/update/delete/list` commands for route management.
### Python SDK
- Updated protobuf bindings. Removed old `inference.py` client (replaced by transparent interception).
### Documentation
- New `architecture/inference-routing.md` — end-to-end system documentation.
- Updated `architecture/sandbox.md` — proxy, OPA, and source index sections.
- Updated `architecture/README.md` — new subsystem overview and diagram.
## Addendum: Chunked Transfer Compatibility
This branch now also fixes intercepted SDK requests that send chunked request bodies:
- `inspect_for_inference` now accepts `Transfer-Encoding: chunked` and decodes chunked request bodies before forwarding to the gateway
- Removed the prior `411 Length Required` response for chunked intercepted requests
- Added request/response header sanitization for framing and hop-by-hop headers (`content-length`, `transfer-encoding`, `connection`, etc.) to keep forwarded requests and returned responses valid
- Added unit tests for chunked parsing and header sanitization
Note: this improves compatibility for streaming-style SDK request patterns; true token-by-token passthrough response streaming is still a separate follow-up.
## Minimal Policy for Inference Routing
```yaml
inference:
allowed_routes:
- local
```
Any outgoing connection from a binary not explicitly allowed in `network_policies` will be intercepted and checked for inference API patterns.
## Test Plan
- [x] `cargo test --workspace` — all tests pass
- [x] `mise run pre-commit` — all checks pass
- [x] E2E: OpenAI chat completions routed through gateway
- [x] E2E: Anthropic messages routed through gateway
- [x] E2E: Python OpenAI SDK from sandbox (`examples/inference/inference.py`)
- [x] E2E test: `e2e/python/test_inference_routing.py`
Closes#44
## Summary
- Add `nav sandbox image push --dockerfile <path>` command to build and push custom container images into the cluster
- Fix sandbox Docker build failure caused by OpenClaw installer TTY access
- Fix sandbox proxy crash when network namespace is unavailable (fall back to loopback)
- Add e2e test for the full custom image build and sandbox creation flow
- Rename and rewrite BYOC architecture doc to reflect current state
## Changes
### `navigator-bootstrap`
- New `build` module (`src/build.rs`) with `build_and_push_image()` public API
- Builds via bollard `Docker::build_image()`, streams output to caller
- `.dockerignore` support with glob matching, 8 unit tests
### `navigator-cli`
- Added `SandboxImageCommands::Push` subcommand with `--dockerfile`, `--tag`, `--context`, `--build-arg` flags
- Added `test:e2e:custom-image` mise task
### `navigator-sandbox`
- Proxy now falls back to `127.0.0.1:3128` when no network namespace or explicit bind address is available, instead of erroring out
### Docker / Infra
- Sandbox Dockerfile: replaced OpenClaw native installer (`curl | bash`) with `npm install -g openclaw` to avoid `/dev/tty` failures in Docker build
### Documentation
- Renamed `architecture/sandbox-byoc.md` to `sandbox-custom-containers.md`
- Rewrote doc to describe current behavior (CLI usage, push pipeline, supervisor behavior, design decisions) instead of listing implementation changes
### Tests
- Added `e2e/bash/test_sandbox_custom_image.sh` covering image push + sandbox create with custom image
> **🔧 security-fix-agent**
Closes#62
## Security Fix
### Summary
The CONNECT proxy accepted hostnames from clients and connected to whatever IP they resolved to, with no validation against internal address ranges. While the OPA policy is default-deny, a misconfigured or overly permissive policy could allow SSRF to cloud metadata (169.254.169.254), localhost, or RFC1918 services. This fix adds DNS resolution before connecting and rejects any host that resolves to an internal IP.
### Severity Assessment
- **Impact:** Medium — if exploited, could reach cloud metadata (IAM creds), cluster-internal services, or host-local services
- **Exploitability:** Very low — requires OPA policy misconfiguration or DNS rebinding attack
- **Affected components:** `crates/navigator-sandbox/src/proxy.rs` — `handle_tcp_connection`
### Changes Made
- `crates/navigator-sandbox/src/proxy.rs`: Added `is_internal_ip()` helper that checks IPv4 loopback/private/link-local, IPv6 loopback/link-local, and IPv4-mapped IPv6. Added `resolve_and_reject_internal()` that resolves DNS and rejects internal IPs. Inserted check between OPA allow and `TcpStream::connect`, with control plane endpoints exempt.
- `architecture/security-policy.md`: Added SSRF Protection section with blocked ranges table and flow diagram
- `architecture/sandbox.md`: Updated proxy connection flow diagram and added SSRF protection subsection
- `architecture/README.md`: Added internal IP rejection step to proxy description
### Tests Added
- **Unit:** 17 tests in `proxy::tests` — covers IPv4 loopback/private/link-local, IPv6 loopback/link-local, IPv4-mapped IPv6, public IPs, DNS resolution of localhost/127.0.0.1/169.254.169.254, and DNS failure handling
- **Integration/E2E:** N/A — the proxy runs inside a Linux network namespace; unit tests for IP checking and DNS resolution cover the security boundary
### Documentation Updated
- `architecture/security-policy.md`: New SSRF Protection section with blocked IP ranges and Mermaid flowchart
- `architecture/sandbox.md`: Updated proxy flow diagram and added SSRF protection subsection
- `architecture/README.md`: Added step 4 to proxy description
### Verification
All 85 sandbox tests pass including 17 new proxy SSRF tests. Pre-commit (fmt, clippy, full test suite) passes clean with zero warnings.
Closes#13
## Summary
- add Python sandbox execution APIs for command and callable workflows
- consolidate sandbox policy fixtures and expand e2e test coverage for policy and Python exec paths
- update CI/build config and images for sandbox e2e execution dependencies
## Test Plan
- mise run pre-commit
Piotr Mlocek