mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-15 05:09:44 +08:00
vouched
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John T. Myersandjohntmyers
|
241e95dc39 |
feat(policy): support host wildcards and multi-port endpoints (#366)
* feat(policy): support host wildcards and multi-port endpoints Add glob-style host wildcards to endpoints[].host using OPA's glob.match with "." as delimiter — *.example.com matches a single DNS label, **.example.com matches across labels. Validation rejects bare * and requires *. prefix; warns on broad patterns like *.com. Add repeated uint32 ports field to NetworkEndpoint for multi-port support. Backwards compatible: existing port scalar is normalized to ports array. Both the proto-to-JSON and YAML-to-JSON conversion paths emit a ports array; Rego always references endpoint.ports[_]. Fix OpaEngine::reload() to route through the full preprocessing pipeline instead of bypassing L7 validation and port normalization. Closes #359 * fix(policy): reject configs with both port and ports set --------- Co-authored-by: johntmyers <johntmyers@users.noreply.github.com> |
||
|
Drew Newberry
|
fbd93a4632 | refactor: rename navigator- crate prefix to openshell- (#277) | ||
|
Drew Newberry
|
7b0a243304 | ci: remove sandbox docker build from publish and e2e workflows (#275) | ||
|
Drew Newberry
|
756950140c | refactor(python): rename navigator module to openshell and migrate config to gateway paths (#220) | ||
|
John T. MyersandJohn Myers
|
b8d873f68f |
feat(proxy): support plain HTTP forward proxy for private IP endpoints (#158)
* feat(proxy): support plain HTTP forward proxy for private IP endpoints Add forward proxy mode to the sandbox proxy so that standard HTTP libraries (httpx, requests, etc.) work with HTTP_PROXY for plain HTTP calls to private IP endpoints. Previously, non-CONNECT methods were unconditionally rejected with 403. The forward proxy path requires all three conditions to be met: - OPA policy explicitly allows the destination - The matched endpoint has allowed_ips configured - All resolved IPs are RFC 1918 private This ensures plain HTTP never reaches the public internet while enabling seamless access to internal services without custom CONNECT tunnel code. Implementation: - parse_proxy_uri(): parses absolute-form URIs into components - rewrite_forward_request(): rewrites to origin-form, strips hop-by-hop headers, adds Via and Connection: close - handle_forward_proxy(): full handler with OPA eval, SSRF checks, private-IP gate, upstream connect, and bidirectional relay - Updated dispatch in handle_tcp_connection to route non-CONNECT methods Includes 14 unit tests and 6 E2E tests (FWD-1 through FWD-6). CONNECT path remains completely untouched. Closes #155 * fix(proxy): remove InspectForInference match arm removed by #146 The inference routing simplification in #146 reduced NetworkAction to Allow/Deny, removing InspectForInference. Drop the dead match arm from handle_forward_proxy. * fix(sandbox): restore BestEffort as default Landlock compatibility The Landlock V2 upgrade in #151 changed the default from BestEffort to HardRequirement. This causes all proxy-mode sandboxes to crash with Permission denied when the policy omits the landlock field, because the child process gets locked to only /etc/navigator-tls and /sandbox. Restore BestEffort as the default so policies without an explicit landlock field degrade gracefully. Fixes #161 * fix(sandbox): inject baseline filesystem paths for proxy-mode sandboxes Proxy-mode sandboxes need baseline filesystem paths (/usr, /lib, /etc, /app, /var/log read-only; /sandbox, /tmp read-write) for the child process to function under Landlock. Without these, the child can't exec binaries, resolve DNS, or load shared libraries. The supervisor now enriches the policy with these baseline paths at startup, covering both standalone (file) and gateway (gRPC) modes. For gateway mode, the enriched policy is synced back so users see the effective policy via 'nemoclaw sandbox get'. The gateway validation is relaxed to allow additive filesystem changes (new paths can be added, existing paths cannot be removed) to support the supervisor's enrichment sync-back. Includes 2 E2E tests: BFS-1 (missing filesystem_policy) and BFS-2 (incomplete filesystem_policy). Fixes #161 * fix(e2e): update assertion for relaxed filesystem validation message --------- Co-authored-by: John Myers <johntmyers@users.noreply.github.com> |
||
|
John T. MyersandJohn Myers
|
af8fe4d1d5 |
refactor(policy): consolidate duplicated YAML struct hierarchies (#97)
* refactor(policy): consolidate duplicated YAML struct hierarchies into navigator-policy
Closes #96
Merge the Deserialize-only input structs and Serialize-only output structs
into a single set of types in navigator-policy that derive both Serialize
and Deserialize. This eliminates the duplicate PolicyYaml hierarchy in
navigator-cli and fixes three round-trip issues:
- filesystem_policy vs filesystem field name mismatch
- allowed_ips silently dropped on proto-to-YAML conversion
- network policy name field silently dropped on proto-to-YAML conversion
Also adds api_patterns support to the inference YAML schema and switches
network_policies from HashMap to BTreeMap for deterministic output ordering.
* fix(e2e): update non-CONNECT test assertion from 405 to 403
Align test_l4_non_connect_method_rejected with the proxy behavior
change in
|
||
|
Alexander Watson
|
1d7909cb38 |
chore: add open-source compliance files and SPDX headers (#71)
Add Apache 2.0 licensing, SPDX copyright headers on all source files, DCO enforcement, third-party notices, and CI enforcement. - LICENSE: Apache License 2.0 full text - DCO: Developer Certificate of Origin 1.1 - SPDX headers on all 176 source files (.rs, .py, .proto, .rego, .sh, .toml, .yaml, Dockerfiles) - scripts/update_license_headers.py: header management with --check mode - scripts/generate_third_party_notices.py: dependency license aggregation - THIRD-PARTY-NOTICES: generated listing of all Rust and Python deps - build/license.toml: mise tasks for license:check and license:update - CI: license-headers job in checks.yml, DCO check workflow - CONTRIBUTING.md: DCO sign-off requirement and license header docs - Cargo.toml: license changed to Apache-2.0, repository URL updated - pyproject.toml: license field added Closes #58 |
||
|
John T. MyersandJohn Myers
|
ff52643237 |
feat(sandbox): allow egress to private IP space via allowed_ips policy field (#60)
* feat(sandbox): allow egress to private IP space via allowed_ips policy field Add an allowed_ips field to NetworkEndpoint that accepts CIDR notation and exact IPs, enabling sandboxes to reach private IP space under policy control while maintaining SSRF protections for loopback and link-local addresses. Three modes are supported: - Default (no allowed_ips): all private IPs blocked (existing behavior) - Host + allowlist: domain must resolve to an IP in allowed_ips - Hostless allowlist (no host): any domain allowed if resolved IP matches Co-authored-by: John Myers <johntmyers@users.noreply.github.com> |
||
|
John Myers
|
757217f4bd |
feat(sandbox): support live policy updates, history, and policy-aware logs (!55)
Closes #78 ## What You Can Do ### Live policy updates Update a running sandbox's network policy without recreating it: ```bash nav sandbox policy set <sandbox> --policy new-policy.yaml --wait --timeout 60 ``` The sandbox hot-reloads the policy within 30s (configurable). On failure, the previous policy stays active (last-known-good). Idempotent — submitting the same policy twice returns the existing version: ``` ✓ Policy version 3 submitted (hash: a1b2c3d4e5f6) $ nav sandbox policy set test --policy same.yaml · Policy unchanged (version 3, hash: a1b2c3d4e5f6) ``` ### Policy history & inspection ```bash nav sandbox policy list <sandbox> # version history with status nav sandbox policy get <sandbox> # current policy metadata nav sandbox policy get <sandbox> --full # print full policy as YAML nav sandbox policy get <sandbox> --rev 2 --full # specific revision as YAML ``` ### Sandbox logs Stream logs from both the gateway and sandbox supervisor in one view: ```bash nav sandbox logs <sandbox> # one-shot, last 2000 lines nav sandbox logs <sandbox> --tail # live streaming nav sandbox logs <sandbox> --source sandbox # supervisor logs only nav sandbox logs <sandbox> --source gateway # gateway logs only nav sandbox logs <sandbox> --level warn # warnings and errors only nav sandbox logs <sandbox> --since 5m # last 5 minutes nav sandbox logs <sandbox> --tail --source sandbox --level info ``` Each log line is tagged with its source and includes structured fields: ``` [1772055394.673] [sandbox] [INFO ] [navigator_sandbox::proxy] CONNECT action=allow dst_host=api.anthropic.com dst_port=443 policy=claude_code [1772055061.005] [gateway] [INFO ] [navigator_server::grpc] GetSandboxPolicy served from policy history ``` --- ## Implementation ### Proto changes - 7 new RPCs: `UpdateSandboxPolicy`, `GetSandboxPolicyStatus`, `ListSandboxPolicies`, `ReportPolicyStatus`, `GetSandboxLogs`, `PushSandboxLogs` (client-streaming) - `SandboxLogLine`: added `source` (gateway/sandbox), `fields` (structured key-value map) - `WatchSandboxRequest`: added `log_since_ms`, `log_sources`, `log_min_level` - `GetSandboxLogsRequest`: added `sources`, `min_level` - `PolicyStatus` enum, `SandboxPolicyRevision` message - `Sandbox.current_policy_version` field ### Server - **Policy persistence**: New `sandbox_policies` table (SQLite + Postgres) with per-sandbox monotonic versions, status tracking, and policy hash - **UpdateSandboxPolicy**: Validates static field immutability (filesystem/landlock/process), network mode consistency (Block↔Proxy), deterministic hash comparison for idempotent updates - **Lazy backfill**: First `GetSandboxPolicy` call creates version 1 from `spec.policy` for existing sandboxes - **Log broker**: `TracingLogBus::publish_external()` injects sandbox-pushed logs into the same broadcast channel + tail buffer (2000 lines). Server forces `source="sandbox"` and `sandbox_id` on all pushed logs - **Source/level filtering**: Applied server-side in both `GetSandboxLogs` and `WatchSandbox` streams - **Version supersession**: When a new version is loaded, all older pending+loaded versions are marked superseded ### Sandbox - **`OpaEngine::reload_from_proto()`**: Full `from_proto()` pipeline (L7 validation, access preset expansion) with atomic engine swap. On failure, previous engine untouched (LKG) - **Policy poll loop**: Background task polls every 30s (configurable via `NAVIGATOR_POLICY_POLL_INTERVAL_SECS`), reports status via `ReportPolicyStatus` RPC - **`LogPushLayer`**: Tracing layer captures events at INFO+ (configurable via `NAVIGATOR_LOG_PUSH_LEVEL`), sends structured fields via `PushSandboxLogs` client-streaming RPC. Background task batches 50 lines / flushes every 500ms. Best-effort (drops on full channel, never blocks) - **`CachedNavigatorClient`**: Persistent mTLS channel for both policy polling and log push ### Database migration - `002_create_sandbox_policies.sql` (SQLite + Postgres) ## Tests - **Unit**: 8 policy persistence tests (put/get/list/status/supersede/isolation) - **Integration**: 4 test files updated with new RPC stubs - **E2E**: `test_live_policy_update_and_logs` — full lifecycle: create → set same (unchanged) → push new → wait for load → verify connectivity → push same (unchanged) → verify history → fetch logs ## Documentation - `architecture/sandbox.md`: Log streaming architecture, LogPushLayer, push task, server broker, source tagging, structured fields, CLI filtering, failure modes - `architecture/security-policy.md`: Live update semantics, deterministic hashing, CLI filter flags, policy inspection - `architecture/plans/issue-78-sandbox-log-streaming.md`: Design plan for log streaming ## Security - Trust boundary documented: shared mTLS cert model (per-sandbox auth tracked in #80) - Server forces `source="sandbox"` and `sandbox_id` on pushed logs (can't impersonate gateway or other sandboxes) - Per-batch line cap (100) prevents flooding |
||
|
Piotr Mlocek
|
34fd3cbf64 |
feat(inference): inference interception and routing (!38)
Closes #67 ## Summary Implements transparent inference interception and routing for sandboxed AI agents. The sandbox proxy intercepts outbound AI SDK calls (OpenAI, Anthropic) and reroutes them through the gateway to policy-controlled backends — enabling organizations to redirect inference traffic to local or self-hosted models without modifying agent code. **Decision model** — a tri-state OPA evaluation for every CONNECT request: 1. Binary + endpoint explicitly allowed in `network_policies` → **allow** (pass through) 2. Not explicitly allowed + `inference.allowed_routes` configured → **inspect for inference** (TLS intercept, detect API patterns, route through gateway) 3. Otherwise → **deny** No endpoint declarations or binary lists needed for inference routing. Just configure `inference.allowed_routes`. ## Key Changes ### Sandbox (interception) - **OPA policy**: New `network_action` Rego rule with three outcomes (`allow`, `inspect_for_inference`, `deny`). New `NetworkAction` enum replaces `PolicyDecision.allowed` bool for the proxy's main decision path. - **Proxy**: New `InspectForInference` path — TLS-terminates client, parses HTTP, detects inference API patterns (`POST /v1/chat/completions`, `/v1/completions`, `/v1/messages`), strips auth headers, forwards via gRPC. - **New module**: `l7/inference.rs` — `InferenceApiPattern`, `detect_inference_pattern()`, HTTP request/response parsing. - **gRPC client**: New `proxy_inference()` for sandbox→gateway forwarding. - **Sandbox init**: Creates OPA engine when inference is configured, even without `network_policies`. ### Gateway (dispatch) - **InferenceService**: `ProxyInference` RPC loads sandbox policy, resolves allowed routes, dispatches to router. Full CRUD for inference routes. - **Proto**: `InferenceRoute`, `InferenceRouteSpec`, `ProxyInferenceRequest/Response`, Inference gRPC service. ### Router (backend proxying) - **New crate**: `navigator-router` with `Router`, `proxy_with_candidates()`, protocol-based route selection, backend HTTP proxying with auth header rewriting. - **Mock support** for testing (`mock://` scheme). ### CLI - `nav inference create/update/delete/list` commands for route management. ### Python SDK - Updated protobuf bindings. Removed old `inference.py` client (replaced by transparent interception). ### Documentation - New `architecture/inference-routing.md` — end-to-end system documentation. - Updated `architecture/sandbox.md` — proxy, OPA, and source index sections. - Updated `architecture/README.md` — new subsystem overview and diagram. ## Addendum: Chunked Transfer Compatibility This branch now also fixes intercepted SDK requests that send chunked request bodies: - `inspect_for_inference` now accepts `Transfer-Encoding: chunked` and decodes chunked request bodies before forwarding to the gateway - Removed the prior `411 Length Required` response for chunked intercepted requests - Added request/response header sanitization for framing and hop-by-hop headers (`content-length`, `transfer-encoding`, `connection`, etc.) to keep forwarded requests and returned responses valid - Added unit tests for chunked parsing and header sanitization Note: this improves compatibility for streaming-style SDK request patterns; true token-by-token passthrough response streaming is still a separate follow-up. ## Minimal Policy for Inference Routing ```yaml inference: allowed_routes: - local ``` Any outgoing connection from a binary not explicitly allowed in `network_policies` will be intercepted and checked for inference API patterns. ## Test Plan - [x] `cargo test --workspace` — all tests pass - [x] `mise run pre-commit` — all checks pass - [x] E2E: OpenAI chat completions routed through gateway - [x] E2E: Anthropic messages routed through gateway - [x] E2E: Python OpenAI SDK from sandbox (`examples/inference/inference.py`) - [x] E2E test: `e2e/python/test_inference_routing.py` |
||
|
Drew Newberry
|
6dc97171ed |
feat(sandbox): add image build/push and fix cluster deploy (!34)
Closes #44 ## Summary - Add `nav sandbox image push --dockerfile <path>` command to build and push custom container images into the cluster - Fix sandbox Docker build failure caused by OpenClaw installer TTY access - Fix sandbox proxy crash when network namespace is unavailable (fall back to loopback) - Add e2e test for the full custom image build and sandbox creation flow - Rename and rewrite BYOC architecture doc to reflect current state ## Changes ### `navigator-bootstrap` - New `build` module (`src/build.rs`) with `build_and_push_image()` public API - Builds via bollard `Docker::build_image()`, streams output to caller - `.dockerignore` support with glob matching, 8 unit tests ### `navigator-cli` - Added `SandboxImageCommands::Push` subcommand with `--dockerfile`, `--tag`, `--context`, `--build-arg` flags - Added `test:e2e:custom-image` mise task ### `navigator-sandbox` - Proxy now falls back to `127.0.0.1:3128` when no network namespace or explicit bind address is available, instead of erroring out ### Docker / Infra - Sandbox Dockerfile: replaced OpenClaw native installer (`curl | bash`) with `npm install -g openclaw` to avoid `/dev/tty` failures in Docker build ### Documentation - Renamed `architecture/sandbox-byoc.md` to `sandbox-custom-containers.md` - Rewrote doc to describe current behavior (CLI usage, push pipeline, supervisor behavior, design decisions) instead of listing implementation changes ### Tests - Added `e2e/bash/test_sandbox_custom_image.sh` covering image push + sandbox create with custom image |
||
|
John Myers
|
82e6b81324 |
fix(security): reject CONNECT to internal IPs (SSRF defense-in-depth) (!37)
> **🔧 security-fix-agent** Closes #62 ## Security Fix ### Summary The CONNECT proxy accepted hostnames from clients and connected to whatever IP they resolved to, with no validation against internal address ranges. While the OPA policy is default-deny, a misconfigured or overly permissive policy could allow SSRF to cloud metadata (169.254.169.254), localhost, or RFC1918 services. This fix adds DNS resolution before connecting and rejects any host that resolves to an internal IP. ### Severity Assessment - **Impact:** Medium — if exploited, could reach cloud metadata (IAM creds), cluster-internal services, or host-local services - **Exploitability:** Very low — requires OPA policy misconfiguration or DNS rebinding attack - **Affected components:** `crates/navigator-sandbox/src/proxy.rs` — `handle_tcp_connection` ### Changes Made - `crates/navigator-sandbox/src/proxy.rs`: Added `is_internal_ip()` helper that checks IPv4 loopback/private/link-local, IPv6 loopback/link-local, and IPv4-mapped IPv6. Added `resolve_and_reject_internal()` that resolves DNS and rejects internal IPs. Inserted check between OPA allow and `TcpStream::connect`, with control plane endpoints exempt. - `architecture/security-policy.md`: Added SSRF Protection section with blocked ranges table and flow diagram - `architecture/sandbox.md`: Updated proxy connection flow diagram and added SSRF protection subsection - `architecture/README.md`: Added internal IP rejection step to proxy description ### Tests Added - **Unit:** 17 tests in `proxy::tests` — covers IPv4 loopback/private/link-local, IPv6 loopback/link-local, IPv4-mapped IPv6, public IPs, DNS resolution of localhost/127.0.0.1/169.254.169.254, and DNS failure handling - **Integration/E2E:** N/A — the proxy runs inside a Linux network namespace; unit tests for IP checking and DNS resolution cover the security boundary ### Documentation Updated - `architecture/security-policy.md`: New SSRF Protection section with blocked IP ranges and Mermaid flowchart - `architecture/sandbox.md`: Updated proxy flow diagram and added SSRF protection subsection - `architecture/README.md`: Added step 4 to proxy description ### Verification All 85 sandbox tests pass including 17 new proxy SSRF tests. Pre-commit (fmt, clippy, full test suite) passes clean with zero warnings. |
||
|
John Myers
|
f1439727b9 |
feat(sandbox): L7 protocol-aware inspection with TLS termination (!29)
## Summary Implements L7 (application-layer) protocol-aware policy enforcement for the sandbox proxy, enabling per-request allow/deny decisions based on HTTP method and path — not just host:port. ### Phase 1: HTTP/REST L7 Inspection (plaintext) - New `l7/` module with provider trait, REST parser, and relay loop - Parses HTTP/1.1 requests inside CONNECT tunnels, evaluates method+path against OPA/Rego policy - Supports `access` presets (`full`, `read-only`, `read-write`) and explicit `rules` with glob path matching - `enforcement` modes: `enforce` (deny with 403 JSON) vs `audit` (log only, forward traffic) - Structured logging for every L7 decision (`L7_REQUEST` with protocol, action, target, decision) ### Phase 2: MITM TLS Termination for HTTPS - Ephemeral CA generated at sandbox startup (`rcgen`) - Per-hostname leaf certificate cache (256 cap) for dynamic cert presentation - Client-side TLS termination (sandbox trusts ephemeral CA via `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, etc.) - Upstream TLS connection with `webpki-roots` verification - Endpoints with `tls: terminate` get full L7 inspection; others remain passthrough ### Additional - Benign TLS/connection errors (close_notify, handshake EOF, reset) downgraded from WARN to DEBUG - Generic `AsyncRead + AsyncWrite` bounds throughout L7 module (works with both `TcpStream` and `TlsStream`) - Proto schema extended with `protocol`, `tls`, `enforcement`, `access`, `rules` fields on `NetworkEndpoint` - CLI `nav sandbox run` wired with `--rego-policy`/`--rego-data` flags and L7 policy display ### E2E Test Coverage - 7 L4 tests: no-matching-policy deny, wildcard binary, binary-restricted, wrong port, cross-policy isolation, non-CONNECT 405, structured log fields - 7 L7 TLS tests: full access allow, read-only deny POST, audit mode, explicit path rules, CA trust store injection, deny response JSON format, structured log fields ## Key Files | Area | Files | |------|-------| | L7 core | `l7/mod.rs`, `l7/provider.rs`, `l7/relay.rs`, `l7/rest.rs` | | TLS termination | `l7/tls.rs` | | Proxy integration | `proxy.rs`, `lib.rs`, `main.rs` | | Env/process | `process.rs`, `ssh.rs` | | OPA policy | `opa.rs`, `dev-sandbox-policy.rego` | | Proto schema | `proto/sandbox.proto` | | CLI | `navigator-cli/src/run.rs` | | E2E tests | `e2e/python/test_sandbox_policy.py` | ## Test plan - [x] 67 unit tests pass (`cargo test --workspace`) - [x] `cargo clippy --workspace --all-targets` clean - [x] 16 e2e policy tests pass (`mise run test:e2e:sandbox`) - [x] Manual smoke test: sandbox with `tls: terminate` on `api.anthropic.com:443` Closes #25 |
||
|
Drew Newberry
|
d2f3ca71d4 |
feat(sandbox): add callable python exec API and refresh e2e coverage (!19)
Closes #13 ## Summary - add Python sandbox execution APIs for command and callable workflows - consolidate sandbox policy fixtures and expand e2e test coverage for policy and Python exec paths - update CI/build config and images for sandbox e2e execution dependencies ## Test Plan - mise run pre-commit |